Sapling-505M-Midtrain

This is the mid-trained (annealed) checkpoint of Sapling-505M-Base. It is a 505M-parameter Qwen3-architecture model trained from scratch. On top of the 5B-token base, it was given 1.0B more tokens: math web text, chat conversations in ChatML, and unseen pretraining replay, with the learning rate decayed to zero.

It sits between the base model and the chat model Sapling-505M-SFT:

Sapling-505M-Base  ──(+1.0B tokens, LR cooldown)──▶  Sapling-505M-Midtrain  ──(+0.5B tokens SFT)──▶  Sapling-505M-SFT
     5.0B tokens                                     6.0B total  (this repo)                          6.5B total

What it is good for: a fully annealed base for your own fine-tuning (SFT, LoRA, DPO), or plain text and math continuation.

What it is not: a chatbot. It has seen ChatML, and the chat template is included, but it does not reliably end its turn. In our chat eval it emitted <|im_end|> in 0 of 50 replies and ran to the token limit (see Evaluation). Use Sapling-505M-SFT if you want to chat.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "VinayRevanuru/Sapling-505M-Midtrain"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto")

inputs = tok("Problem: A train travels 60 km in 45 minutes. What is its speed in km/h?\nSolution:", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Model details

Architecture	Qwen3ForCausalLM (dense): GQA, QK-Norm, RMSNorm, RoPE, SwiGLU
Parameters	505,483,776 (embeddings tied)
Layers / width / FFN	30 / 1,024 / 3,072
Attention heads	16 query, 8 key/value, head dim 128
Context length	2,048 tokens
Vocabulary	32,768 byte-level BPE, the same ids as the Base model
Chat tokens	`<
Weights	FP32 safetensors (≈2.0 GB)

Training data (1,000,013,824 tokens)

Source	Tokens	Share	Notes
FineMath-4+	≈0.50B	50%	Math web pages, top quality tier. Decontaminated against GSM8K test (13-gram overlap).
SmolTalk (all)	≈0.20B	20%	Rendered as standard ChatML. Trained as plain LM text (all tokens), not masked SFT. Also decontaminated.
Pretraining replay (FineWeb-Edu + Cosmopedia v2)	≈0.31B	30%	Shards the base model never saw, kept to limit forgetting.

About 0.1% of each source was held out by a deterministic hash and used as validation. The conversations and pages used here are disjoint from those later used for SFT.

Training procedure

Initialization	Base checkpoint at step 76,300, weights and AdamW state
Tokens / steps	1,000,013,824 / 15,259
Global batch	65,536 tokens (32 × 2,048)
Learning rate	linear decay from 8e-5 to 0 (8e-5 was where the base run's cosine had reached)
Optimizer	fused AdamW, β = (0.9, 0.95), weight decay 0.1, grad clip 1.0
Precision	BF16 autocast, FP32 master weights
Hardware	8 × NVIDIA H100 (PyTorch DDP) on Modal
Throughput / wall clock	≈251k tokens/s, ≈1.2 h

Evaluation

Held-out validation loss (nats/token, lower is better). Each value is measured on the run's own 0.1% holdout:

Tokens into mid-training	FineWeb-Edu	Cosmopedia v2	FineMath	SmolTalk
0.2B	3.035	1.706	2.247	1.722
0.4B	3.022	2.110	1.947	1.398
0.6B	3.013	2.104	1.936	1.322
0.8B	2.956	2.080	1.831	1.449
1.0B (this model)	2.905	2.019	1.821	1.371

FineMath loss fell from 2.25 to 1.82 over mid-training. FineWeb-Edu loss also fell as the learning rate annealed, so the general web skill improved rather than being forgotten.

Chat behaviour. We ran the same harness used to gate the SFT model: sampling at temperature 0.7, top-p 0.9, up to 512 new tokens.

Check	Result
Ends its turn with `<	im_end
Says its name is "Sapling" (10 identity probes)	0 / 10 (it was never taught a name)
Declines clearly harmful prompts (30, CoCoNot)	3%
Wrongly refuses harmless edgy prompts (30, CoCoNot contrast)	0%
GSM8K test, 250-problem sample, sampled, answer extracted from the reply	2.0%

Example (chat template, temperature 0.7)

User: What is the capital of Australia?

Model: The capital of Australia is Sydney. The capital of Australia is a city located in the southern part of the continent…

It is fluent and on-topic, but wrong and doesn't stop. That is typical at this stage.

Limitations

• Not instruction tuned. It does not end turns, follow formats or decline harmful requests. It has no safety tuning at all.
• Not factually reliable. It produces confident, fluent errors.
• Weak at math reasoning despite the math data: about 2% on GSM8K.
• English only, 2,048-token context. Use sampling, since greedy decoding loops.

Files

training_summary.json records the full configuration, data manifest, validation history, tokenizer sha256 and dependency versions (torch 2.6.0, transformers 4.51.3).