Privoid-ai/Privoid-Contract-4B

Privoid Contract 4B reads one section of a contract, finds up to 39 kinds of clause, and returns JSON with the exact text it relied on. It is a 4B-parameter model fine-tuned from Qwen3-4B for one job, and it runs on your own hardware, so contracts never have to leave your machine.

It is a review aid for a person, not legal advice. It was tested only on English commercial contracts.

This repo holds the LoRA adapter. For a single file that runs on a CPU, use the GGUF version.

Results

Compared with Gemini 3.5 Flash and the untrained model

All three models read the same 8 contracts (157 sections) from the official CUAD test split and were scored the same way.

Model	Clause F1	95% range	Precision	Recall	Contract-level F1	Valid JSON	Exact quotes
Privoid Contract 4B	0.514	0.435 to 0.638	0.461	0.580	0.870	0.917	0.989
Gemini 3.5 Flash	0.441	0.384 to 0.541	0.454	0.430	0.846	0.936	1.000
Qwen3-4B untrained	0.233	0.185 to 0.382	0.176	0.347	0.705	0.745	0.617

• Against the untrained model: training more than doubled clause F1, from 0.233 to 0.514, and the two 95% ranges do not overlap.
• Against Gemini 3.5 Flash: Privoid scored higher on these 8 contracts, mainly by finding more clauses (recall 0.580 against 0.430) at about the same precision. The 95% ranges overlap, so this sample does not prove Privoid is better. Treat the two as close until a larger test is run.
• Where Gemini leads: Gemini had slightly more valid JSON and every quote it gave was exact.
• Prompts differ in Gemini's favour: Gemini and the untrained model received the full definition of each category. Privoid received only the category names.

Full test set

Model	Clause F1	95% range	Precision	Recall	Contract-level F1	Valid JSON	Exact quotes
Privoid Contract 4B, all 102 test contracts	0.595	0.558 to 0.634	0.562	0.633	0.876	0.962	0.991

This is the fullest result for Privoid. It covers more contracts than the comparison above, so do not compare this row with those rows. The 8-contract group turned out harder for Privoid than the test set as a whole.

By category

Clear single-sentence clauses score best, such as Governing Law (0.91) and Agreement Date (0.85). Weak categories include both Affiliate License categories (0.17 and 0.14), Rofr/Rofo/Rofn (0.26) and Non-Disparagement (0.27): do not rely on the model for these. Categories with few test clauses have noisy scores. Price Restrictions had no clauses in the test set, so it has no score and is left off the chart. Exact numbers are in per_category.csv.

What the measures mean

• Clause F1: a finding counts when its category is right and its text matches a lawyer's label (Jaccard overlap of at least 0.5). This is the strict measure.
• Contract-level F1: whether the model correctly says a category is present anywhere in a contract.
• Valid JSON: the share of replies that are complete, valid JSON.
• Exact quotes: the share of quotes that appear word for word in the section. Quotes that do not are discarded before scoring.
• 95% range: a bootstrap interval over contracts, showing how much the score could move with a different sample.

How to use it

Split a contract into sections of up to 5,000 characters, with some overlap so that no clause is cut in half. Send each section separately. The prompt in prompt.json is required.

With Transformers and PEFT

pip install transformers peft accelerate huggingface_hub

import json
from huggingface_hub import hf_hub_download
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

repo = "Privoid-ai/Privoid-Contract-4B"
settings = json.load(open(hf_hub_download(repo, "prompt.json"), encoding = "utf8"))
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoPeftModelForCausalLM.from_pretrained(repo, torch_dtype = "auto", device_map = "auto")

def review(section):
    messages = [{"role": "system", "content": settings["system_prompt"]},
                {"role": "user", "content": f"<contract_section>\n{section}\n</contract_section>"}]
    prompt = tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = True) + settings["assistant_prefix"]
    inputs = tokenizer(prompt, return_tensors = "pt", add_special_tokens = False).to(model.device)
    output = model.generate(**inputs, max_new_tokens = 700, do_sample = False)
    return json.loads(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens = True))

print(review("This Agreement shall be governed by the laws of the State of New York."))

With the GGUF file

pip install llama-cpp-python huggingface_hub

import json
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

repo = "Privoid-ai/Privoid-Contract-4B-GGUF"
settings = json.load(open(hf_hub_download(repo, "prompt.json"), encoding = "utf8"))
llm = Llama(model_path = hf_hub_download(repo, "privoid-contract-4b-Q5_K_M.gguf"), n_ctx = 4096, verbose = False)

def review(section):
    prompt = (f"<|im_start|>system\n{settings['system_prompt']}<|im_end|>\n"
              f"<|im_start|>user\n<contract_section>\n{section}\n</contract_section><|im_end|>\n"
              f"<|im_start|>assistant\n{settings['assistant_prefix']}")
    reply = llm(prompt, max_tokens = 700, temperature = 0, stop = ["<|im_end|>"])
    return json.loads(reply["choices"][0]["text"])

print(review("This Agreement shall be governed by the laws of the State of New York."))

About 4% of replies across the test set were not valid JSON, for example when a long reply is cut off, so wrap json.loads in try in real use. Always check each quote against the source text and have a person review the result.

The GGUF file

privoid-contract-4b-Q5_K_M.gguf is 2.69 GB. Q5_K_M is used because Q4 corrupted digits on Qwen3 in earlier work. It was compared with the full model on 20 contract sections before upload:

Check	Full model	GGUF
Exact quotes	98.4%	97.8%
Valid JSON	16/20	15/20
Identical replies		14/20

This is a small check that catches serious damage, not a full benchmark. The scores above are for the full model. On a Colab CPU, one section took roughly 45 seconds.

Limits

• The Gemini comparison uses only 8 contracts. Its scores would change with a different sample.
• Clause F1 of 0.595 means the model misses clauses and makes false findings. It speeds up review but cannot replace reading the contract.
• It sees one section at a time, so it cannot connect clauses that are far apart.
• Reply limits differed: 700 tokens for the untrained model and Privoid, except that Privoid's first 348 test answers (the shortest sections) used 1,400. Gemini had 4,096 tokens, including its thinking. Complete findings were kept from replies that were cut off, for all models.
• Gemini was reached through Colab's built-in access as google/gemini-3.5-flash, without tuning.

Training

Base model	unsloth/Qwen3-4B-Instruct-2507 (Apache-2.0)
Data	CUAD v1: 510 contracts and 39 categories, split by contract
Training sections	10,612
Method	LoRA (r 16, alpha 32) with Unsloth, 16-bit
Length	1 epoch, 885 steps
Best validation loss	0.0320
Testing	The official 102 CUAD test contracts, 1,246 sections

Data and licences

Trained on the Contract Understanding Atticus Dataset (CUAD) v1 by The Atticus Project, used under CC BY 4.0. Text was normalized and split into sections. The model weights are released under Apache-2.0.