Bounty.Hunter.47m

A decoder-only causal language model trained from scratch, built around a deep chain-of-thought reasoning core.

Architecture

• Parameters: 46,912,320
• Context length: 1096
• Hidden size: 672
• Layers: 8
• Attention heads: 8
• Intermediate size: 2688
• Vocabulary size: 4,096

Reasoning tokens

• <|input|>
• <|think|>
• <|thought|>
• <|reasoning|>
• <|answer|>

These are first-class tokens in the tokenizer and are woven into every pretraining example's text structure, not added after the fact.

Objective

True causal next-token prediction.

Context

Every individual training example is strictly limited to 1095 content tokens + EOS, padded to exactly 1096 positions. No oversized example is split across separate training examples.

Dataset

Plans11/Organized_PreTrain_1k_Context, restricted to:

• Think
• Thought
• Reasoning

(Chat / Instruct / Code_Instruct / Tool_Calling shards are intentionally excluded from this model's pretraining corpus.)

Example selection

New examples are selected each session via a deterministic seeded shuffle of the restricted dataset, with SHA-256 content-hash deduplication against every previously trained example. No example is ever trained twice.

Resumability

Training state is persisted to Hugging Face, including:

• model.safetensors
• training_state.pt
• tokenizer files
• config.json
• progress.json
• seen_examples.jsonl
• training_metadata.json

The tokenizer becomes immutable after its initial creation.

Current progress

• Unique examples trained: 20,000
• Global optimizer steps: 625
• Last session: 20,000

This is an experimental language model and is not guaranteed to produce factually or logically correct outputs.