slm-lab-json-r8

A LoRA adapter on unsloth/Llama-3.2-1B-Instruct that turns a messy, free-text expense line into strict JSON — the expense-to-JSON adapter, rank 8, trained on a free Colab T4.

What it is for. Given a reference date and a line like "lunch with the team at Saravana Bhavan 840 rupees yesterday, split 4 ways", it emits an object with exactly the keys amount, currency, merchant, category, date, split and note. The corpus is synthetic and generated by scripts/gen_json_task.py in the repository, so the ground truth is correct by construction — including the dates, which is the point: "last Tuesday" is unanswerable without the reference date, so the date is supplied and the target computed from it.

r=8, the rank this project's own sweep selected. This is the adapter the DPO and quantization work builds on.

A research artifact for learning, not a product: built to measure how fine-tuning, preference optimization and quantization each behave on one task whose ground truth is known exactly.

How it was built

QLoRA: the base model is quantized to 4-bit NF4 and frozen, and only the adapter trains. The loss is masked to the assistant turns, which is the detail that decides whether the model learns the task or learns to echo the prompt.

Every arm of the sweep was run with a measured seed spread, so differences between ranks are judged against this project's own noise rather than read off a ranking. The arms that showed no detectable difference are published as such.

Results

	this adapter, val	this adapter, test	untuned base, test
Valid against the schema	100.0%	99.4%	67.0%
All seven fields exactly right	96.8%	30.0%	0.2%
Mean fields correct (of 7)	6.97	6.06	4.14
date field (computed, not copied)	96.8%	36.6%	1.4%
merchant field (copied straight out)	100.0%	100.0%	90.2%

Trainable parameters	5,636,096 of 780,077,056 (0.72%)
Evaluated on	500 held-out lines per split, greedy decoding
General ability, above chance	+0.2861 mean, over 4 multiple-choice tasks

The val/test gap is the result, not a blemish. Schema validity transfers completely and the exactly-right rate does not: this adapter is near-perfect on validation and lands far lower on test. The gap sits almost entirely in date, the one field that has to be computed from a reference date and a phrase like "last Tuesday" rather than copied out of the input. Fields that are copied transfer; the field that requires arithmetic does not.

That single fact is what the rest of the project keeps rediscovering: preference optimization could not teach it either, and quantization does nearly all of its damage there.

Full scores for every arm, including the ones that showed nothing, are published under results/ at https://huggingface.co/datasets/Dhevenddra/slm-lab-data.

Using it

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3.2-1B-Instruct")
model = PeftModel.from_pretrained(base, "Dhevenddra/slm-lab-json-r8")
tok = AutoTokenizer.from_pretrained("unsloth/Llama-3.2-1B-Instruct")

Use the same system prompt the adapter was trained with — it names the exact keys. scripts/gen_json_task.py::system_prompt() is the authoritative copy; a paraphrase will quietly change the behaviour.

Limitations

Specific, and written out in full — this is the section a knowledgeable reader checks first.

• One narrow synthetic task. It parses expense lines into one fixed schema. It is not a general instruction-following model and it is not better at anything else than its base.
• The computed fields are the weak ones. The table above is the evidence: date and amount — the fields requiring arithmetic rather than copying — are where the model is least certain, where fine-tuning transferred least to held-out inputs, and where quantization does nearly all of its damage.
• General ability is unchanged — measured, not assumed. Over 4 multiple-choice benchmarks this adapter sits +0.0034 from the untuned base (+0.2861 against +0.2827 above chance). Task specialization did not visibly cost generality here, and it did not buy any either.
• Inherits everything from the base model, including its safety behaviour, its biases and its unknown pretraining corpus.

Contamination: unknown and unknowable. The base model's pretraining data is not published, so absolute benchmark numbers for this adapter are uninterpretable. Before-and-after comparisons against the same base remain valid, and that is how the results report them.

Where this came from

Part of slm-lab — a language model built end to end on free compute: a hand-written BPE tokenizer, a transformer from scratch, a controlled ablation study with a measured noise floor, post-training, QLoRA on a 1B model, DPO, and quantization written by hand and benchmarked against every production method.

• Start here: the SLM Lab collection lists every repo in the order it was built, with a one-line finding on each.
• Code, guide and the full failure log: https://github.com/Dhevenddra/slm-lab
• Every result file, dataset and tokenizer: https://huggingface.co/datasets/Dhevenddra/slm-lab-data

This repo is the project's QLoRA rank sweep (phase 06) stage; the published write-up for it is under results/ in the dataset repo, named for that phase.

The project's own failure log (FAILURES.md, 26 entries) is the part most worth reading. The recurring shape: the instrument measured something correlated with the question rather than equal to it.