Mini.Deep.Thinker.11m

A small decoder-only causal language model trained from scratch with deep chain-of-thought reasoning capabilities.

Architecture

• Parameters: 11,004,896
• Context length: 1096
• Hidden size: 352
• Layers: 7
• Attention heads: 8
• Intermediate size: 1152
• Vocabulary size: 4,096

Reasoning tokens

• <|input|>
• <|think|>
• <|thought|>
• <|reasoning|>
• <|answer|>

Objective

True causal next-token prediction with strict per-example context limits.

Context

Every training example is strictly limited to 1095 content tokens + EOS, padded to exactly 1096 positions. No oversized example is split across separate training examples.

Example selection

Deterministic shuffled scan with SHA-256 content hash deduplication. Each session selects 20,000 new, unseen examples.

Dataset

Plans11/Organized_PreTrain_1k_Context

Resumability

Training state is persisted to Hugging Face, including:

• model.safetensors
• training_state.pt
• tokenizer files (tokenizer.json, tokenizer_config.json)
• config.json
• progress.json
• seen_examples.jsonl
• training_metadata.json

Current progress

• Unique examples trained: 40,000
• Global optimizer steps: 1,250
• Last session: 20,000 examples
• Best session loss: 1.88992

This is an experimental small language model and is not guaranteed to produce factually or logically correct outputs.