ZL-300M (Stage 2, KL-distilled)

A ~300M-parameter decoder-only Transformer, implemented entirely from scratch in PyTorch (no transformers model code, no borrowed architecture) — tokenizer, RoPE attention, SwiGLU feed-forward, RMSNorm, the training loop, and generation were all hand-written for this project.

What this is

Two training stages, in order:

1. Stage 1 — plain pretraining. Trained from random initialization on ~7 billion tokens of FineWeb-Edu, standard next-token cross-entropy. No teacher model involved. This produces a base model that can write fluent text but does not reliably answer questions — it treats a chat-formatted prompt as just more text to continue.
2. Stage 2 — distilled fine-tuning (this checkpoint). The Stage 1 model is then fine-tuned to actually answer questions, using Qwen2.5-7B-Instruct as a teacher: for ~10,000 curated questions, the teacher's real generated answers become the supervision target, and the loss is a blend (α=0.5) of ordinary cross-entropy against the teacher's chosen words and a KL-divergence term matching the teacher's full output distribution (temperature=2.0) — not just its one chosen token.

Same architecture/tokenizer/vocabulary as the teacher throughout, so student and teacher logits are directly comparable — the whole point of a from-scratch project reusing a real vocabulary as data, not as a shortcut on the architecture itself.

Params & size

Parameters	300,544,512
Weights (fp32, this repo)	~1.2 GB
Teacher (Qwen2.5-7B-Instruct)	7.61B params, ~15.3 GB in bf16
Compression vs. teacher	~25.3× fewer parameters

How this is packaged

This release is weights + tokenizer only — model.safetensors (the trained parameters) and the tokenizer files (vocab.json, merges.txt, tokenizer.json, tokenizer_config.json), described by config.json. It uses a custom, non-standard architecture (not a transformers model class), so no AutoModelForCausalLM.from_pretrained(...) one-liner and no loading code are included in this release — treat this as the raw artifact of the training run, not a ready-to-run package.

The chat format used during fine-tuning, for reference:

<|im_start|>system
You are a helpful assistant. Answer clearly and concisely, ideally in 2-4 sentences, unless the question genuinely needs more detail.<|im_end|>
<|im_start|>user
{your question}<|im_end|>
<|im_start|>assistant

Honest limitations (measured, not guessed)

This is an educational from-scratch project, not a production model, and it shows:

• It hallucinates often and confidently. In a 100-question blind evaluation on held-out questions never seen in training, this model (and a comparison variant) both regularly produced fluent, wrong answers — inventing fake taxonomic families, misattributing historical figures, and occasionally collapsing into repetition loops on harder questions ("the Great Barrier Reef, Great Barrier Reef, Great Barrier Reef..."). At 300M parameters, there is very little spare capacity to memorize facts precisely — treat any specific factual claim from this model as unverified.
• It only reliably knows what was in its training data. ~7B tokens of general web text (FineWeb-Edu) plus ~10k Q&A examples. It cannot answer questions about anything outside that, and it will not tell you when it doesn't know — it will just answer anyway.
• The distillation signal's benefit was smaller than expected at this scale. A plain-cross-entropy control (no soft-KL term, same data and schedule) scored better on held-out perplexity than this KL-distilled model, and came out ahead in a large blind qualitative comparison too (45% vs. 37% question-by-question wins, rest ties) — a real, if modest, edge for the simpler method at this specific scale (25× teacher/student compression, 2,000 fine-tuning steps). The behavioral leap from the base model (which never answers questions at all) to either fine-tuned version is much larger than the gap between the two fine-tuning methods.
• Answers can run on past their natural end. The model usually emits the correct end-of-turn token, but generation in the reference scripts doesn't stop there automatically — trim at the first <|im_end|>.

Training details

	Stage 1	Stage 2 (this checkpoint)
Data	~7B tokens, FineWeb-Edu	~10,356 curated Q&A pairs (Dolly-15k questions, Qwen2.5-7B-Instruct answers)
Objective	Next-token cross-entropy	α=0.5 blend: cross-entropy + KL-divergence (T=2.0) vs. teacher
Hardware	RTX A5000 (rented)	RTX A5000 (rented)
Steps	106,811	2,000

License & attribution

Released under Apache 2.0, matching the license of Qwen2.5-7B-Instruct, whose tokenizer vocabulary this project reuses as data and whose generated responses supervised Stage 2's fine-tuning. All architecture code, the training loop, and Stage 1's pretraining are original work.