Llama-3.1-8B-RBI-Amendments-LoRA

A LoRA adapter that extracts the amendment graph from Reserve Bank of India circulars: what an instrument changes, how, under what legal power, and from when.

On a leak-free benchmark of 101 held-out RBI documents it beats GPT-4o-mini, GPT-5.4-mini and the 120B model that produced its training labels, and ties with GPT-5.4-mini when that model is given the full labelling guide in its prompt.

Built with Llama.

Code, benchmark and corrections log: https://github.com/BiplawSingh/beat-the-giant Dataset: https://huggingface.co/datasets/biplaw/rbi-amendment-extraction

Results

model	HARD F1	95% range	exact	parse failures
gpt-5.4-mini + guide in prompt	0.941	0.907 – 0.971	86/101	0
this adapter	0.935	0.891 – 0.963	67/101	3
gpt-4o-mini	0.843	0.793 – 0.905	58/101	0
openai/gpt-oss-120b (the teacher)	0.837	0.769 – 0.888	60/101	0
gpt-5.4-mini	0.833	0.772 – 0.905	69/101	0
untuned Llama-3.1-8B-Instruct	0.000	—	0/101	101

Paired bootstrap over the 101 documents (2,000 resamples): +0.093 vs gpt-4o-mini [+0.014, +0.154], +0.102 vs gpt-5.4-mini [+0.018, +0.172], +0.099 vs its own teacher [+0.056, +0.148]; −0.006 vs gpt-5.4-mini+guide [−0.054, +0.027], i.e. tied.

HARD F1 is the unweighted mean of field F1s for amends, commencement.mode, commencement.date and legal_authority. instrument_type and applies_to are reported separately as guardrails (0.958 here) and never folded into the headline.

Cost. 2.09 GPU-hours per 1,000 documents measured on a Tesla T4 one document at a time, so $1.34 per 1,000 at the $0.64/hour median T4 price (range $0.19 – $2.34 by provider) against $3.44 for the row it ties with. Training cost 1.42 GPU-hours, about $0.91.

What it outputs

{"instrument_type": "Amendment Directions",
 "applies_to": ["Rural Co-operative Banks"],
 "amends": [{"instrument_title": "Reserve Bank of India (Rural Co-operative Banks - Responsible Business Conduct) Directions",
             "year": 2025, "relation": "amends"}],
 "commencement": {"mode": "dated", "date": "2027-01-01"},
 "legal_authority": [{"act": "Banking Regulation Act, 1949", "sections": ["21", "35A", "56"]}]}

relation is one of amends, supersedes, repeals, consolidates. commencement.mode is one of immediate, dated, on_notification, not_stated.

Use

from unsloth import FastLanguageModel

model, tok = FastLanguageModel.from_pretrained(
    "your-name/Llama-3.1-8B-RBI-Amendments-LoRA", max_seq_length=3072, load_in_4bit=True)
FastLanguageModel.for_inference(model)

messages = [{"role": "system", "content": SYSTEM},     # task.py, prompt "base"
            {"role": "user", "content": f"Title: {subject}\n\n{window(text)}"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
print(tok.decode(model.generate(ids, max_new_tokens=4096, do_sample=False)[0, ids.shape[1]:]))

Two things must match training or the scores above do not apply:

• The prompt is task.system_prompt("base") from the repository, fingerprint 6a3cd317f2da. It is ~380 tokens and does not contain the labelling guide.
• The input window is the document's first 4,800 and last 1,600 characters, joined by \n[... omitted ...]\n. Measured on the test split, that window keeps 76/76 commencement clauses, 70/70 authority clauses and 47/48 amendment clauses.

Generation must allow 4,096 new tokens. One test document lists 38 superseded instruments; at 2,048 the answer is truncated mid-JSON and the whole document is lost. That cost 0.08 HARD F1 in an earlier run.

Training

base	unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit (4-bit)
method	QLoRA, r=16, alpha=32, dropout 0, all attention and MLP projections
data	354 RBI documents, teacher-labelled and corrected (see dataset card)
schedule	3 epochs, lr 2e-4 linear, warmup 5%, effective batch 8, adamw_8bit
precision	fp16 (T4 is Turing: no bf16, no FlashAttention-2)
sequence	3,072 tokens; longest training example ~2,300
loss	completion only — the answer JSON, not the prompt
hardware	one free Kaggle Tesla T4, 85 minutes
selection	best of 3 epoch checkpoints by validation score (51 documents), never test

Limitations

• Training labels are model-generated. A 120B teacher labelled 354 documents; the labels were then corrected to a written guide by rule (35) and by hand (9).
• 3 parse failures in 101. All are an instrument_type outside the enum on documents titled "Guidelines". API baselines cannot make this mistake because they run with strict-schema decoding; this model has no such guard. Constrained decoding at serving time would remove them.
• Small benchmark. 101 documents, 419 graded answers. Differences below about 0.06 F1 are noise; the ranges above are why.
• One document carries ~30% of the amends evidence. Excluding it: this model 0.917, gpt-5.4-mini+guide 0.941, gpt-4o-mini 0.870.
• Scope. RBI circulars and directions, February 2024 – August 2026, English. Other regulators, other document layouts, or documents outside the window are untested.
• Not reproducible run to run. Same seed and data gave a different best epoch and validation scores differing by ~0.02.
• Some conventions were identified from test-set disagreements. They are general rules, not per-document answers, which is why the fair comparison is the +guide row rather than the plain baselines.

Licence

Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. This adapter is a derivative of Llama 3.1 and carries its licence and NOTICE. Teacher labels were generated with openai/gpt-oss-120b (Apache 2.0).