achavan1211/harrypotter-gpt-bpe-sft

Harry Potter GPT — Instruction-Tuned (SFT, ~12M)

An instruction-fine-tuned version of achavan1211/harrypotter-gpt-bpe, a from-scratch 12M-parameter decoder-only Transformer. Fine-tuned (full parameter SFT, open-book QA, answer-only loss masking) on vapit/HarryPotterQA.

⚠️ What this model is (and isn't)

This is an educational toy, not a usable QA system. A 12M model is roughly 100× smaller than the smallest practical instruct model (~0.5–1B parameters). Instruction tuning here mostly taught output format and memorized common Q/A pairs; it does not reason and will hallucinate confidently on questions it hasn't seen. Do not trust its answers.

Measured results (held-out validation QA)

Model	Exact-Match	F1
Base (pretrained only)	0.00	0.00
Instruct (this model)	21.33	49.12

Interpret these honestly: even after fine-tuning, EM/F1 are low in absolute terms. The improvement over the base (if any) is the real signal; the low absolute score is the lesson — instruction tuning is bounded by the base model's capacity and knowledge, and 12M has very little of either.

Example failures

These are real outputs from this model on held-out questions:

Q: What causes the prophecy to be lost?

• Gold answer: The spun-glass ball, which contained the prophecy, is accidentally kicked away by Neville and shatters, causing the prophecy to be lost.
• Model said: The motorcycle, the merpeople in the passage, and the Death Eaters are including the mundane living in the mirror.

Q: Why does the author emphasize a certain aspect?

• Gold answer: The author emphasizes their normality to highlight a contrast with the magical events that are about to unfold. It sets the stage for showing how unusual the upcoming events are compared to the Dursleys' everyday life.
• Model said: The author emphasizes their normality to highlight a contrast with the magical events that are about to unfold. It sets the stage for showing how unusual the

Q: Why does the author emphasize a certain aspect?

• Gold answer: The author emphasizes their normality to highlight a contrast with the magical events that are about to unfold. It sets the stage for showing how unusual the upcoming events are compared to the Dursleys' everyday life.
• Model said: The author emphasizes their normality to highlight a contrast with the magical events that are about to unfold. It sets the stage for showing how unusual the

Model details

• Base: 12M-param GPT, n_embd=384, layers=6, heads=6, block_size=256
• Tokenizer: byte-level BPE (vocab 2000)
• SFT: full fine-tuning, answer-only loss, ~3000 steps (best val answer-loss 0.409 @ step 2999)
• Open-book input format: Context: ... Question: ... Answer: ...

Intended use

Research and education on small-language-model fine-tuning. Out of scope: any factual QA, chat, or production use.

Copyright

Harry Potter is © J.K. Rowling / Warner Bros. Trained for personal learning only.