training_for_practice

Two complete, tested LLM training systems, plus one small model actually trained with them. Built end to end on a 2012 laptop — an Intel i3-3110M, 8 GB RAM, no GPU — which is the point: the constraint forces every step to be measured rather than assumed.

mmllm/	A multimodal LLM pretrained from scratch — architecture, tokenizer, data pipeline, training loop. 38 tests.
finetune/	LoRA fine-tuning of an existing model, with a hand-rolled LoRA implementation. 38 tests.
model/	A SmolLM2-135M LoRA fine-tune produced by finetune/.

---

⚠️ Read this before using the model

The model in model/ is a practice artifact, not a useful model. It is 135M parameters trained for 60 steps. It is published to show that the pipeline works, not because the result is good.

• It learned format, not knowledge. Asked what a function does, it answers in the right terse docstring register — and frequently invents the content.
• On at least one held-out prompt it degenerates into a repetition loop.
• Training data provenance: it was fine-tuned on 5,249 question/answer pairs generated automatically from a private codebase's Markdown docs and Python docstrings. That dataset is not included in this repo. The model may surface file and symbol names from that codebase.

Do not use it for anything real. Use the code.

---

What the model actually achieved

Trained on CPU, 60 steps, ~20 tokens/second.

Measurement	Base	Fine-tuned	Change
Held-out loss (identical 40 batches)	2.9769	2.5900	−0.3869
Perplexity	19.6	13.3	−32%
Validation loss during training	3.4212	2.8850	−0.5362
Trainable parameters	—	4,884,480	3.5% of 139M

A representative before/after, on a held-out prompt:

Q: What does the store function do?

Base: sixty words inventing a plausible-sounding data structure it had never seen — fluent, confident, and entirely wrong.

Tuned: "Store the entry for later retrieval."

Reference: "Store entry with embedding for semantic search."

The register is right. The content is still wrong. That is what 135M parameters and 60 steps buys.

---

The interesting part: the code

finetune/ — LoRA, implemented directly

LoRA is ~60 lines, written out rather than pulled from peft, so it can be tested exactly and does not break when adapter libraries lag a transformers release.

W_effective = W_frozen + (alpha/r) * B @ A        A: (r, in)   B: (out, r)

B starts at zero, so an untrained adapter is bit-identical to the base model — a free correctness check before you spend anything.

pip install -r finetune/requirements.txt
PYTHONPATH=finetune/src python -m ft.build_dataset --input /path/to/repo --out data
PYTHONPATH=finetune/src python -m ft.train --config finetune/configs/smol135m_cpu.yaml
PYTHONPATH=finetune/src python -m ft.evaluate --adapter runs/smol135m/adapter_best.pt
PYTHONPATH=finetune/src python -m ft.merge --adapter runs/smol135m/adapter_best.pt --out export/

ft.planner prices a run — time, cost and peak GPU memory including activations — from a model's config.json alone, without downloading weights.

mmllm/ — pretraining from scratch

A hybrid architecture built for long context: most layers are linear-time state-space (Mamba-2 SSD) rather than attention, because at 300k tokens the KV cache, not compute, is the binding constraint.

Design	KV cache at 300k ctx
This hybrid (22 SSD / 5 sliding-window / 5 global, GQA-2)	1.47 GiB
All layers global attention, same GQA	9.16 GiB (6.2×)
All layers conventional MHA	73.24 GiB (49.8×)

scripts/trace_pipeline.py walks one real batch through all sixteen stages — raw text, cleaning, tokenizer, ids, embeddings, layers, attention, next-token prediction, target comparison, loss, backward, gradients, optimizer, weight update — printing the actual tensors at each step. It is the fastest way to understand what training does.

---

Why the tests exist

Both packages ship tests that pin failure modes producing a plausible-looking but wrong run rather than a crash. Those are the only ones worth the compute. A sample:

Test	Failure it prevents
Prompt tokens are masked	Without it, most of the gradient teaches the model to generate user questions — and the loss curve looks perfectly healthy
Drop-rate guard	A max_len shorter than your data yields an empty training set, silently. On a reasoning dataset with 7,700-token median, max_len=1024 keeps zero examples
JSONL line-breaker escaping	json.dumps(ensure_ascii=False) leaves U+0085/U+2028/U+2029 raw; str.splitlines(), jq and pandas all treat them as newlines. This shredded 11 records before it was caught
Merge is numerically equivalent (float64)	Shipping a different model than the one you evaluated
LoRA adapter dtype matches base	Harmless on CPU; fatal on every bf16 GPU run
Cached generation == full forward	Decode path drifting from training path
SSD chunked scan == reference scan	The whole long-context argument rests on this

Ten real bugs were found this way during development. Every one was silent.

---

Hardware reality

Measured, not estimated:

Peak matmul throughput	24 GFLOP/s (i3-3110M, 2 cores, CPU only)
Achieved in training	~11.7 GFLOP/s (48% MFU)
From-scratch pretraining of a useful model	~10⁷× beyond this machine

That gap is why the repo contains a fine-tuned model and not a pretrained one, and why planner.py exists. Fine-tuning a 7B model on a rented A100 costs single-digit dollars; pretraining a 2B model costs ~19,600 GPU-hours. For domain adaptation, fine-tuning wins by roughly five orders of magnitude.

---

License

Apache-2.0. The base model is SmolLM2-135M-Instruct, also Apache-2.0.