SmolLM2-135M-tinystories

HuggingFaceTB/SmolLM2-135M after 100M tokens of continued pre-training on TinyStories. In-domain validation perplexity 6.8945 → 3.7900 (−45.0%).

This is a fine-tune, not a from-scratch model: the run initialised from the official SmolLM2-135M safetensors and continued training (train_tinystories.py:39,145-149).

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("bishnoiyash/SmolLM2-135M-tinystories")
tok   = AutoTokenizer.from_pretrained("bishnoiyash/SmolLM2-135M-tinystories")

ids = tok("Once upon a time", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=40, do_sample=False,
                                pad_token_id=tok.eos_token_id)[0]))

No trust_remote_code — it is a stock LlamaForCausalLM. Weights are bf16, matching both the training dtype and the base model's published torch_dtype.

The one number, fully specified

Metric	Perplexity
Before	6.8945 (6.894546783281595)
After	3.7900 (3.7899503859716885)
Change	−45.0%
Dataset	roneneldan/TinyStories, revision f54c09fd23315a6f9c86f9dc80f725de7d8f9c64
Config / split	no config / validation
Seq len / stride	1024 / 1024 (non-overlapping)
Target tokens	199,485 (195 windows × 1023), from the first 1,040 non-empty stories = 200,068 packed tokens
Tokenizer	HuggingFaceTB/SmolLM2-135M
Precision	bf16, logits cast to fp32 before cross-entropy, reduction="sum"
Pairing	The BEFORE and AFTER evals use the identical val_tokens tensor — strictly paired

"Before" is the official SmolLM2-135M weights scored in the same harness before any optimizer step, not a random-init baseline.

Evidence on disk: results/tinystories_before.txt:2, results/tinystories_after.txt:2, results/tinystories_train.log:10,508.

Training run

Init	Official HuggingFaceTB/SmolLM2-135M safetensors
Corpus	roneneldan/TinyStories — 2,119,719 train stories / 21,990 validation
Packed	102,000,116 train tokens → 99,609 windows of 1024
Steps	24,414
Tokens seen	99,999,744 of a 100,000,000 budget
Tokens/step	4,096
LR schedule	WSD, measured from the per-step trace: linear warmup to step 200 → peak 3e-4; stable through step 19,531; linear decay from 19,532 to 0.0 at 24,414
Precision	bf16
Wall clock	116.1 min on an NVIDIA GB10 (Grace Blackwell)
Throughput	14,356 tok/s cumulative at step 24,400

Training-loss landmarks from results/tinystories_train.csv: best single-batch loss 0.9088 at step 22,353; first 1000-step bucket mean 1.5860; final bucket (24,000–24,414, 414 rows) mean 1.3138.

Limitations — read these before citing the number

1. n = 1. Single seed, single corpus, single run. No confidence interval, no across-seed variance, no iso-FLOP control arm. In-domain validation perplexity on the corpus that was just trained on is the weakest form of evidence for a language model, and it is the only quantitative evidence here.
2. No downstream benchmark was ever measured. No HellaSwag, ARC, MMLU, or any other task — not "they were poor", they were never run for this checkpoint.
3. No out-of-domain or catastrophic-forgetting measurement exists. The source project's own note: "We didn't measure wikitext-2 PPL post-training but it almost certainly got worse." Expect general-purpose ability to have degraded relative to the base model. Treat this as a TinyStories-domain model only.
4. The optimizer hyperparameters of this run are not recoverable. No args= line in the log, no training_recipe key in the checkpoint, no grad_norm column in the CSV. Values for weight decay / betas / eps / grad-clip / seed that appear elsewhere in the source project come from an external nanotron config and a later version of the script — they are not measurements of this run, so they are deliberately omitted above rather than guessed.
5. The exact training script is not recoverable. The on-disk train_tinystories.py demonstrably post-dates the run (its CSV header, log-line templates, and checkpoint-key set all disagree with the artifacts), and git holds only that later version. The LR schedule above is stated only because it was re-derived from the per-step trace, not read off the script.
6. Beware a near-identical number from a different run. An earlier TinyStories run in the same project reached PPL 3.7893 in 137.3 min. That is not this checkpoint. This one is 3.7900 in 116.1 min.

Provenance and integrity

Exported from checkpoint_tinystories.pt, sha256 78e82767ea1bde589f97b915fc2133e0a6040993f859846a2e4a1fe7f2bb39b0 (269,144,681 bytes).

The export is lossless: the source checkpoint's 273 tensors are all bf16 and were saved back as bf16, so all 272 stored tensors are bit-identical to the checkpoint (torch.equal on every tensor; worst |Δ| = 0.0). lm_head.weight is tied to model.embed_tokens.weight and therefore not stored separately — 134,515,008 unique parameters.

training_recipe.json in this repo carries the step / token / perplexity metadata read out of the source checkpoint.

Notes

• pad_token_id is null and bos_token_id == eos_token_id == 0, inherited unchanged from the base model. Pass pad_token_id=tok.eos_token_id to generate() to silence the padding warning.
• special_tokens_map.json is absent because transformers v5 stores special tokens in tokenizer_config.json instead; bos/eos/unk (<|endoftext|>) are all present there.

Links

• Architecture-parity reproduction (single-file PyTorch, no weights): bishnoiyash/SmolLM2-135M-reproduction
• Source project: https://github.com/yashb98/BuildFromScratch

License

Apache-2.0, inherited from HuggingFaceTB/SmolLM2-135M. TinyStories (roneneldan/TinyStories) is CDLA-Sharing-1.0; no TinyStories text is redistributed here.