slm125m-live-sft

The 125.8M-parameter legal/financial model AnandHaridas1980/slm125m-live after supervised fine-tuning on 2,620 grounded question-answer pairs.

The base model continues text. This one answers a question from a passage you supply, or says the passage does not contain the answer.

What changed

Measured on 200 held-out pairs never seen in training, greedy decoding, judged by gemini-3.6-flash against the source passage.

	Base	Fine-tuned
Correct (judged)	3.0%	36.0%
Grounded in the passage	11.5%	69.5%
Hallucinated	88.0%	30.0%
Emitted a stop token	1.7%	98.3%
Refused an unanswerable question	0.0%	80.0%
Wrongly refused an answerable one	0.0%	2.5%
Validation loss (answer tokens)	2.061	1.145
Mean tokens generated	94.7	22.6

Accuracy by question type:

Type	Base	Fine-tuned
lookup (fact stated in the passage)	5.8%	27.2%
reasoning (one or two inference steps)	0.0%	9.6%
unanswerable (correct answer is a refusal)	0.0%	86.7%

Read this before using it

The 36% headline is carried by refusals. On questions that genuinely have an answer in the passage it is right about 21% of the time. On questions that do not, it is right 86.7% of the time.

In plain terms: this model learned when not to answer far better than how to answer. It reliably produces a well-formed, correctly terminated, confident-sounding response — and that response is frequently wrong on specifics. Every failure looks like a competent answer.

Verify every figure, date and name against the passage. Do not use this as an answer service.

It is useful as a component: a cheap first-pass reader whose refusals are trustworthy enough to route on. It is not useful as an authority.

Prompt format (required)

The model was trained on exactly one prompt shape. Deviating from it degrades output silently.

<|bos|><|system|>You are a legal and financial assistant.
Answer only from the provided context.
If the context is not enough, say you do not know.<|user|>Context:
{passage}

Question: {question}<|assistant|>

Generate from there; the model emits the answer then <|eos|>.

import torch
from transformers import AutoTokenizer, LlamaForCausalLM

tok = AutoTokenizer.from_pretrained("AnandHaridas1980/slm125m-live-sft")
model = LlamaForCausalLM.from_pretrained("AnandHaridas1980/slm125m-live-sft", torch_dtype=torch.bfloat16).eval()
model.config.use_cache = True          # ships False from training; 6x slower without it

SYSTEM = ("You are a legal and financial assistant.\n"
          "Answer only from the provided context.\n"
          "If the context is not enough, say you do not know.")

def ask(passage, question, max_new_tokens=128):
    prompt = (f"<|bos|><|system|>{SYSTEM}<|user|>Context:\n{passage}\n\n"
              f"Question: {question}<|assistant|>")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False,
                         eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                         pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
    return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()

print(ask("Net revenue rose 12.4% to $48,300,000 in fiscal 2025.",
          "What was net revenue in fiscal 2025?"))

With no passage it should refuse. That is trained behaviour, not a bug — it has no useful world knowledge and was never meant to.

Training

Method	Full SFT (not LoRA); loss on assistant tokens only
Data	2,620 pairs generated from the base model's own corpus by gemini-3.6-flash, judged by a second call, deduplicated by embedding, decontaminated against the eval split
Mix	case-law 39.7% / SEC 39.8% / educational web 20.6%
Types	lookup 50.4% / reasoning 28.3% / unanswerable 21.3%
Steps	120 (3 epochs), global batch 65,536 tokens
Supervised tokens seen	228,458
LR	3e-5 cosine to 3e-6, 10-step warmup
Hardware	1x L40S, 3.0 minutes
Cost	$7 total ($6.38 dataset, ~$0.10 training)

Validation loss bottomed at step 80 (1.1143) and drifted to 1.1449 by step 120; the published checkpoint is step 120. Two epochs would likely have been better.

Limitations

• Not RAFT. Every training example contained exactly one passage, always the correct one. The model has never seen an irrelevant passage, so behind a real retriever returning mixed chunks it is out of distribution.
• Context limit 1,024 tokens, including the passage.
• Single turn only. No multi-turn conversation was trained.
• English only; US case law and SEC filings.
• Reasoning accuracy is 9.6%. Do not use it for multi-step inference.
• Judged by a model from the same family that generated the training data, which is a weaker check than an independent evaluator.

Full write-up

The complete build — dataset construction, cost model, every failure — is documented in doc/sft/ of the project repository, alongside the pretraining book for the base model.