Distil_50M_SLM: a 51.5M-parameter language model distilled from Qwen2.5-0.5B

A small decoder-only language model trained from scratch by knowledge distillation. The teacher is Qwen2.5-0.5B (about 10x larger); the student sees about 1B tokens (19 per parameter) and then gets a short instruction-tuning pass distilled from Qwen2.5-0.5B-Instruct. Trained entirely on 2x Tesla T4 GPUs (Kaggle's free tier), 66 GPU-hours in total.

It also has a twin, adyaanahmed/distil-50m-no-teacher, trained with the same data, seed, steps and optimiser but no teacher, to measure what the teacher buys (see below).

It is a small model: fluent, but it invents facts and cannot do arithmetic. See Limitations.

Quick start

pip install torch safetensors tokenizers huggingface_hub
hf download adyaanahmed/Distil_50M_SLM --local-dir Distil_50M_SLM
cd Distil_50M_SLM
python chat.py                       # instruction-tuned model, single-turn
python chat.py --model pretrained    # raw text continuation

Runs on CPU at roughly 20-30 tokens per second. Inside the chat: /temp 0.4, /topk 40, /rep 1.15, /len 200. The first run downloads the Qwen2.5 tokenizer.

This is a custom PyTorch architecture, so it does not load with AutoModel.from_pretrained and the hosted inference widget is not available; use chat.py or the code in hybrid50m/. The chat prompt format is ### Question:\n{query}\n\n### Answer:\n.

Files

File	What
student_chat.safetensors	instruction-tuned model (fp16, 99 MB)
student_pretrained.safetensors	base model after distillation pretraining (fp16, 99 MB)
vocab.json	maps the student's 32,768-token vocabulary to Qwen2.5 token ids
config.json	architecture hyperparameters
chat.py	CPU chat script with streaming output
hybrid50m/	model definition, optimiser, data pipeline, training and self-tests
notebooks/	the Kaggle notebooks that trained this model and the no-teacher control
results/	reports, training curves and the comparison figure

Architecture

Parameters	51,532,544 (47.2M in transformer blocks, 4.3M in embeddings)
Layers x width	18 layers, d_model 512 (deep and narrow, MobileLLM-style)
Attention	grouped-query: 8 query heads, 2 key/value heads, head dim 64, per-head QK-norm
FFN	SwiGLU, hidden size 1280
Position encoding / norm	RoPE (theta 10,000) / RMSNorm
Embeddings	factorised and tied: 32,768 x 128, projected up to 512
Context length	512 tokens
Vocabulary	32,768 (the 32,767 most useful Qwen2.5 tokens plus one UNK)
KV cache	9 KB per token in fp16 (4x smaller than full multi-head attention)
Compute	about 0.1 GFLOPs per generated token; 99 MB in fp16

Training

• Distillation loss: 0.7 x KL(teacher || student) + 0.3 x cross-entropy, T = 1, over the student's vocabulary subset.
• Pretraining data: about 1.02B tokens, roughly 64% Cosmopedia v2 and 36% FineWeb-Edu (educational score >= 3). 983M tokens seen.
• Optimiser: Muon for 2-D transformer weights, AdamW for the rest; peak lr 1.5e-3, 400 warmup steps, flat phase, then a 1-sqrt decay over the final 20% of steps. 15,000 steps of 65,536 tokens, fp16 with loss scaling, 2-GPU DDP.
• Instruction tuning: 3,996 steps on about 130k smol-smoltalk first exchanges, distilled from Qwen2.5-0.5B-Instruct (alpha 0.5).
• Compute: 60.5 T4 GPU-hours pretraining + 5.7 instruction tuning = 66.2.

Results

Held-out bits per byte (lower is better; comparable across tokenizers):

Set	Student	Teacher (Qwen2.5-0.5B)	Student / teacher
Cosmopedia	0.666	0.572	1.16x
FineWeb-Edu	1.078	0.835	1.29x
WikiText (out of domain)	1.270	0.905	1.40x

Zero-shot benchmarks, 1,000 examples each (about +/-1.5 points of noise):

Benchmark	Base	Chat (after SFT)	Random guessing
ARC-Easy (acc_norm)	45.1	41.9	25
SciQ (acc)	67.4	62.0	25
HellaSwag (acc_norm)	30.6	29.6	25
LAMBADA (acc)	23.0	16.3	~0

Instruction tuning costs some benchmark accuracy, as it usually does for small models; quote benchmark numbers for the base model.

Does the teacher help? The no-teacher control

Same architecture, data and order, seed, optimiser, steps and tokens; only the loss differs.

	Distilled (this model)	No teacher
Bits per byte: Cosmopedia / FineWeb-Edu / WikiText	0.666 / 1.078 / 1.270	0.655 / 1.117 / 1.325
ARC-Easy / SciQ / HellaSwag / LAMBADA (base)	45.1 / 67.4 / 30.6 / 23.0	42.8 / 62.8 / 28.4 / 20.9
Pretraining T4 GPU-hours	60.5	20.0

The teacher helps most on text outside the training mix and is ahead on all four benchmarks (only the SciQ gap clearly exceeds the noise); on the training domain it is a near tie. The price is about 3x the compute. One run per arm, so read the smaller gaps as indicative. Full numbers: results/report_distilled.json, results/report_no_teacher.json.

Reproduce

The notebooks in notebooks/ run the whole pipeline on Kaggle (GPU T4 x2): RUN_MODE = "data" builds the training data, then "pretrain" trains (resumable across 12-hour sessions), runs instruction tuning and exports the models. distill_no_teacher.ipynb is the control (KD = False, SFT_ALPHA = 0). Self-tests: cd hybrid50m && python selftest.py.

Limitations

• Unreliable facts. It answers in the right style but often invents details (it explained rain as "tectonic activity" in one run).
• No arithmetic or reasoning. HellaSwag is close to chance.
• Single turn, 512 tokens. No conversation memory, short context.
• Repetition. Long outputs can loop; a repetition penalty of about 1.15 and temperature 0.4-0.6 help.
• English only, trained on synthetic and educational web text. Not safety-tuned or red-teamed. Do not use it for anything that matters.
• The tokenizer is Qwen2.5's; text outside the 32,767 kept tokens is dropped (UNK).

Licence and attribution

Apache-2.0. Distilled from Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct (Apache-2.0) using Cosmopedia v2 and smol-smoltalk (Apache-2.0) and FineWeb-Edu (ODC-By 1.0). Please credit the Qwen team and the Hugging Face datasets above.