YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Qwen2.5 7B β Contract Clause QA
Supervised instruction-tuning of Qwen2.5 7B for contract clause identification and extraction across 41 legally significant clause types from the CUAD benchmark.
Overview
Models are fine-tuned to produce a structured response: a binary Yes/No decision on clause presence, followed by the verbatim quotation of the relevant clause. This reframes CUAD's span-extraction task as an instruction-following problem, allowing the model to leverage parametric legal knowledge while remaining grounded in the provided contract excerpt.
- Base model: Qwen2.5 7B
- Dataset: CUAD β 510 annotated commercial contracts, 41 clause types
- Adaptation: LoRA and QLoRA (r=16, Ξ±=16, all linear layers)
Performance
| Variant | Token F1 | BERTScore | Accuracy | Overall |
|---|---|---|---|---|
| Qwen2.5 7B LoRA | 0.672 | 0.875 | 0.916 | 0.760 |
| Qwen2.5 7B QLoRA | 0.680 | 0.876 | 0.901 | 0.762 |
| Qwen2.5 7B QLoRA (large batch) | 0.636 | 0.859 | 0.830 | 0.714 |
| Qwen2.5 7B baseline (zero-shot) | 0.423 | 0.774 | 0.832 | 0.479 |
Composite metric:
Overall = (0.4 Γ Token_F1 + 0.6 Γ BERTScore) Γ Accuracy
BERTScore uses Legal-BERT as the scoring backbone with IDF weighting and baseline rescaling.
Key finding: QLoRA achieves near-parity with full-precision LoRA at significantly lower GPU memory cost β confirming that 4-bit base weight compression does not meaningfully degrade clause identification quality at the 7B scale.
Fine-tuning gain vs zero-shot
| Overall score | |
|---|---|
| Qwen2.5 7B (zero-shot) | 0.479 |
| Qwen2.5 7B LoRA (fine-tuned) | 0.760 |
| Gain | +0.281 |
At the 7B scale, SFT refines behaviour qualitatively β teaching the model to ground responses in the specific contract text rather than generalised legal knowledge, producing more faithful and legally precise extractions.
Training setup
| Parameter | Value |
|---|---|
| LoRA rank / alpha | r=16, Ξ±=16 |
| LoRA target | All linear layers |
| Context windowing | 10,000-char answer-aware window |
| Data split | 95 / 2.5 / 2.5 (train / val / test) |
| Attention | Flash Attention 2 |
| Kernel optimisation | Liger kernel fusion |
| Memory | Gradient checkpointing + sequence packing |
| QLoRA quantisation | 4-bit + double quantisation (bfloat16 adapters) |
Known Limitations
Temporal / date clauses
Subword tokenisation fragments date strings, causing models to identify the correct sentence but predict a generic label (e.g. "Effective Date") rather than the precise date token. This is a tokeniser-level limitation that fine-tuning alone cannot resolve.
Autoregressive repetition loop
Party name extraction can trap the model in a token-repetition loop until the context limit is reached, particularly when the reference contains multiple semicolon-separated entity names.
Exposure bias / cascade errors
A wrong Yes/No gate token propagates forward with no corrective signal β the model can confabulate a plausible but incorrect clause extraction (e.g. a hallucinated date) with full confidence.
Evaluation notes
- Length bias is present in all fine-tuned models (negative Pearson correlation between reference length and scores). The 7B QLoRA large-batch variant shows the weakest bias (r = β0.133 / β0.119).
- BERTScore floor: a rescaled score near zero indicates semantic orthogonality, not a random prediction. A corpus-specific CUAD baseline would provide a more appropriate rescaling anchor.
- Batch size effect: larger batch sizes produce a weaker 7B QLoRA model, suggesting gains are driven by the number of gradient updates rather than update quality.