Kotichitturu/slm-125m-sft-qa

SLM-125M SFT (Q&A)

⚠️ This model invents facts. Do not use it as a source of truth.

It is a demonstration of instruction-following, not an answering service. It reliably produces answer-shaped text in the right format and tone — and the facts in that text are frequently fabricated. Asked for a company's minimum net worth, it answered $1,000,000; the correct figure was $150,000. Confidently, fluently wrong.

This is not a bug in the pipeline. A 125M-parameter model holds roughly 2 bits per parameter — nowhere near enough to store a legal/financial corpus. The knowledge is not in there, so the model confabulates something plausible. For grounded answers use the RAFT sibling (Kotichitturu/slm-125m-sft-raft), which answers only from a context passage you supply and refuses when the answer isn't there.

What fine-tuning actually bought

The interesting result is the comparison against the base on the same prompt:

Prompt	Base	This model
"What is the minimum net worth?"	"What is the minimum net worth????????…"	"FHC is required to maintain a minimum net worth of $1,000,000…"
"What is the main legal issue?"	"Answer correctly and concisely. Answer correctly and concisely…"	"The main legal issue is whether the plaintiffs are entitled to…"

The base regenerates the question and recites its system prompt back. This model answers the question. SFT taught the job, not the knowledge — which is the entire lesson: skills in the weights, facts in the retriever.

Measured behaviour

• 0/60 empty generations
• 0/60 role-token leaks — the loss mask works
• Median answer length 25 words
• Val loss 2.8425 → 2.1670 (−24%), best at epoch 3

Note: this val loss is computed over response tokens only (51.8% of tokens), so it is not comparable to the base's 2.2521, which is over all tokens.

Alignment tax: 1.6%

Fine-tuning rewrites the weights, and the weights are the only place the base's knowledge lives — so did learning the job cost it what it knew? Measured on the same held-out pretraining text the base was measured on, every token, no masking (the only apples-to-apples comparison available):

Model	Perplexity	vs base
Base	9.50	—
This model	9.65	+1.6%

Almost free. It kept essentially all of its domain modelling and gained instruction-following. (Control: the base reproduces its published 9.51 on this harness, so the comparison is sound.)

9.65 is not e^2.1670. It is e^2.2666, where 2.2666 is this model's loss on held-out pretraining text with every token counted. The 2.1670 above is a different measurement entirely — the SFT val split, masked to response tokens. Two numbers, two datasets, two token populations; only this one can be compared to the base.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-sft-qa")
model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-sft-qa")

SYSTEM = "You are a legal and financial expert. Answer accurately and concisely. If unsure, say so."
prompt = f"<|bos|><|system|>{SYSTEM}<|user|>What is a fiduciary duty?<|assistant|>"

ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=90, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Greedy decoding (do_sample=False) is what the evaluation used. Sampling makes the invented facts more varied, not less invented.

System prompt (must match exactly)

You are a legal and financial expert. Answer accurately and concisely. If unsure, say so.

Architecture

Identical to the base — fine-tuning changes weights, never shape.

Params	125.8M
Layers / hidden / heads	12 / 768 / 12
Vocab	16,384 (byte-level BPE)
Context	1,024

Prompt format

The chat format is not a standard template — it uses custom special tokens that exist in this tokenizer. Render exactly:

<|bos|><|system|>{system}<|user|>{user}<|assistant|>

Then generate. Stop at <|eos|>. Sending a different system prompt than the one below moves the model off-distribution and quality degrades silently.

Provenance

• Base: Kotichitturu/slm-125m-base
• SFT data: 8,000 passages (US case law + SEC filings), teacher-generated by gpt-4o-mini and gemini-3.1-flash-lite, then filtered by an LLM judge (gpt-5.4-mini) that rejected every answer not supported by its passage. Judge coverage 100%; keep rate 88%.
• Decontaminated against CaseHOLD/LexGLUE (13-gram) during pretraining.
• Full SFT (not LoRA), 3 epochs, lr 2e-5 cosine, bf16, 1×A100.
• Loss is masked to response tokens only — the model never trains on the prompt it is given.