roberta-base-squad2-cuad

Extractive QA model for legal clause extraction: deepset/roberta-base-squad2 fine-tuned on CUAD v1 (510 commercial contracts, 41 clause categories, annotated by lawyers).

This is the tier-2 span extractor behind Verity, a contract-review pipeline. A zero-shot classifier assigns a clause type; this model extracts the verbatim clause text with exact character offsets. Training and evaluation code is in the repo's cuad/ directory and reproduces this model end to end with uv run python cuad/run.py.

Results

Evaluated on a held-out test set of 51 contracts, all 41 CUAD categories.

Model	Macro F1 (trimmed)	Macro F1 (full)	Macro EM
BART-MNLI zero-shot baseline	0.41	0.66	0.66
this model (base)	0.73	0.83	0.81
roberta-large variant	0.75	0.84	0.82

Trimmed macro F1 excludes categories with fewer than 30 positive spans in the test set, where per-category F1 is unstable. The trimmed figure is the meaningful one for model comparison.

Split is a deterministic alphabetical contract-level 80/10/10, chosen for reproducibility. Results are not directly comparable to papers using a different split.

Usage

The model was fine-tuned on the verbatim CUAD category questions and expects them at inference. Using free-form questions degrades extraction quality.

from transformers import pipeline

qa = pipeline("question-answering", model="mjolnirslams/roberta-base-squad2-cuad")

question = (
    'Highlight the parts (if any) of this contract related to "Governing Law" '
    "that should be reviewed by a lawyer. Details: Which state/country's law "
    "governs the interpretation of the contract?"
)
result = qa(question=question, context=contract_text)
# {'answer': 'This Agreement shall be governed by the laws of the State of New York.', 'start': ..., 'end': ..., 'score': ...}

The full 41-question mapping ships with the Verity repo in cuad_questions.json.

Like its SQuAD 2.0 base, the model predicts no-answer when the clause is absent from the context. Contexts longer than 512 tokens need sliding-window chunking (the question-answering pipeline handles this; Verity chunks contracts upstream).

Training

Hyperparameter	Value
Base model	deepset/roberta-base-squad2
Epochs	2
Batch size	128
Learning rate	4e-5
Warmup ratio	0.1
Weight decay	0.01
Max sequence length	512
Doc stride	128
Precision	bf16

Caveats

Corpus skew. CUAD is heavily skewed toward US commercial contracts. Performance on contracts from other jurisdictions, or contract types underrepresented in CUAD (employment agreements, consumer contracts), is unknown. Fine-tune further before deploying to those contexts.

Span contiguity. Extractive QA treats every answer as a contiguous span. CUAD occasionally annotates disjoint spans for a single (contract, category) pair; the model predicts one span per forward pass, which caps recall for those categories.

Question format. Extraction quality is tied to the verbatim CUAD questions above. This is not a general-purpose legal QA model.

License and attribution

• Base model: deepset/roberta-base-squad2 (CC BY 4.0)
• Data: CUAD v1, The Atticus Project (CC BY 4.0). Hendrycks, Burns, Chen, Ball. "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review." NeurIPS 2021.
• This model: CC BY 4.0