Ihteshamstar/qwen3-4b-cuad-extractor

Qwen3-4B CUAD Contract Clause Extractor

Qwen3-4B-Instruct fine-tuned with QLoRA to extract all 41 CUAD clause types from a contract in one pass, quoting the contract text verbatim and returning explicit empty lists for absent clauses. Runs in ~2.5 GB (q4_K_M GGUF) on consumer hardware.

Trained and evaluated on the official CUAD split: 408 training / 102 held-out test contracts, no contract overlap.

Training and evaluation code, raw eval outputs, and build notes: github.com/Ihtesham-star/cuad_llm_finetuning

Results

Detection F1: did the model find each clause type present in a contract (and produce a verbatim quote for it)? Scored per (contract, clause type) over 102 test contracts × 41 clause types; predictions merged across contract chunks. Bootstrap 95% CIs over contracts (n=1000).

model	detection F1	precision	recall	quote overlap (token F1)	verbatim rate
this model (seed 42)	0.900 [0.887–0.913]	0.901	0.899	0.938	97.2%
Qwen3-14B zero-shot + official clause definitions	0.816 [0.802–0.830]	0.814	0.818	0.729	85.4%

Three seeds (42/43/44) under identical config: F1 0.8995 ± 0.0017. The published weights are seed 42.

Strict full-span coverage (a true positive requires the predicted quotes to fully contain every gold span for the clause — our reimplementation of the rule used by ContractEval, arXiv:2508.03080, which benchmarked zero-shot LLMs on this same 102-contract test set):

model	strict F1
this model (3-seed mean)	0.678
GPT-4.1 (ContractEval, zero-shot)	0.641
Qwen3-8B, best open model tested (ContractEval)	0.540
Claude Sonnet 4 (ContractEval)	0.523

Caveats for that comparison: our matching is a reimplementation of their stated rule (whitespace-normalized, casefolded substring), not their harness; and ContractEval queries one clause type per call with its definition in the prompt, while this model answers all 41 clause types in a single call (~40× fewer calls per contract).

Data curve (seed 42, detection F1 vs number of training contracts): 50 → 0.746 · 150 → 0.862 · 300 → 0.885 · 408 → 0.900. The zero-shot 14B baseline (0.816) is passed somewhere between 50 and 150 annotated contracts.

Prompt format

The model was trained on exactly one prompt shape. Use it verbatim; other phrasings are out of distribution.

System:

You are a contract review system. Extract the requested clause types from the contract excerpt. Quote the contract text verbatim. A clause type not present in this excerpt gets an empty list. Reply with JSON only.

User:

Extract all 41 CUAD clause types from this contract excerpt as JSON (keys: clause types, values: lists of verbatim quotes).

CONTRACT EXCERPT:
<up to ~60k characters of contract text>

Output: one JSON object with all 41 CUAD clause names as keys, each a list of verbatim quotes (empty list = clause absent). Contracts longer than ~60k characters should be chunked with overlap and the per-chunk outputs merged (union per clause).

Usage

Ollama (recommended for local use)

ollama create cuad-extractor -f Modelfile   # FROM ./cuad-4b-s42-q4_K_M.gguf

Call with format: "json", temperature: 0, and num_ctx sized to fit your chunk plus the output (silent prompt truncation is the classic failure mode). On a small fraction of dense chunks (~6% in our runs) greedy decoding enters a repetition loop and hits the output cap; retrying that chunk with repeat_penalty: 1.05 (still temperature 0, deterministic) recovered nearly all cases in our evaluation, escalating to 1.15/1.25 for the rare holdouts.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Ihteshamstar/qwen3-4b-cuad-extractor",
                                             torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Ihteshamstar/qwen3-4b-cuad-extractor")

Repository layout: fp16 merged weights at the root (from_pretrained works directly), gguf/ holds the q4_K_M quantization measured above, lora/ holds the LoRA adapter (r=16) if you prefer to apply it to the base model yourself.

Training

• Base: unsloth/Qwen3-4B-Instruct-2507, QLoRA (4-bit base, LoRA r=16, alpha=32, all attention + MLP projections)
• 619 chunk-level examples from the 408 official training contracts (one example whose rendered length exceeded the 24,576-token training window was excluded and recorded)
• 2 epochs, lr 2e-4 linear with 6% warmup, effective batch 8, ~45 min on one RTX 5090
• Target: the gold JSON only (train_on_responses_only); empty lists for absent clauses are part of the target, which is what teaches calibrated "not present" behavior

Limitations

• Strict-coverage recall is the weak spot (0.58): when a clause is stated in several places, the model often quotes the main span but not every occurrence.
• Greedy decoding can loop on dense chunks (see the fallback note above).
• Trained on CUAD: English-language commercial contracts from SEC EDGAR filings. Performance on other contract styles, other languages, or non-contract legal text is unmeasured.
• This is a research/engineering artifact, not legal advice; extractions need human review in any real workflow.

Attribution

• Dataset: CUAD (Contract Understanding Atticus Dataset), The Atticus Project, CC BY 4.0
• Base model: Qwen3-4B-Instruct-2507 (Apache 2.0), 4-bit variant by Unsloth
• Trained with Unsloth