# 02 — Training

Sources: Thinking Machines announcement + model card (2026-07-15). Not fully specified in Hub config.

---

## Pretrain

| | |
|---|---|
| Tokens | **45T** (text, image, audio, video) |
| Init | From scratch |
| Hardware | NVIDIA **GB300 NVL72** |
| Optim | **Muon** (large matrices) + **Adam** (rest) |
| WD | Coupled to **LR²** (long-horizon weight stability) |
| Research link | Modular manifolds |

Data: public web/repos, third-party, synthetic; filtered/deduped.

---

## Post-train

### 1. Bootstrap SFT

Synthetic data from open models (incl. **Kimi K2.5**). Small compute share — RL warm start.

### 2. Async RL

- Majority of post-train compute  
- **&gt;30M** rollouts  
- Log-linear gains on held-out reasoning aggregate  
- Domains: math, agentic code/tools, vision, audio, chat, safety, calibration  

### Emergent compression

CoT becomes telegraphic under RL (drop grammar, keep substance) — efficiency pressure from token cost, not an explicit “be terse” reward.

---

## Thinking effort

### Training

System-message effort label **+** per-token cost in RL → continuous control of compute.

### Inference (chat template)

```
Thinking effort level: {0.0 … 0.99}
```

| Alias | Value |
|---|---|
| none | 0.0 |
| minimal | 0.1 |
| low | 0.2 |
| medium | 0.7 |
| high | 0.9 |
| max / reported evals | **0.99** |

See [`reference/hub/chat_template.jinja`](../reference/hub/chat_template.jinja) and [chat protocol](../code-study/02-chat-protocol.md).

Official benches: effort **0.99**, temp 1.0; coding trajectories often ≤256K tokens.

---

## Epistemics

| Axis | Mechanism |
|---|---|
| Calibration | Proper scoring rules on resolved real-world Qs |
| Helpful + truthful long-form | Rubric grader + claims grader (claims use **web search**) |
| Abstention | Short-form QA reward only when likely correct |
| Censorship non-compliance | Cognition eval (as reported) |

Goal: useful under uncertainty, not overconfident confabulation.

---

## Safety (release narrative)

- Multimodal refusal consistency  
- Everyday harm patterns + multi-turn red team  
- CBRN / cyber / loss-of-control (internal + external)  
- Residual: role-play / indirect framing (typical open-weight)  

Conclusion they publish: no material uplift beyond existing open ecosystem → deployers still need **defense-in-depth**.

---

## Distillation note

| Stage | Claim |
|---|---|
| Pretrain | Scratch |
| SFT bootstrap | Partially other open models’ synthetic data |
| Future | Self-contained post-train |

Next: [03-benchmarks.md](03-benchmarks.md)