Privoid FactCheck 2B: catch hallucinations in RAG answers

Is this claim backed by the source? Privoid FactCheck reads a grounding document and a claim, such as one sentence of a chatbot's answer, and returns a 0 to 1 support score. Use it to flag hallucinated statements in RAG systems, summaries and AI-written reports before they reach a user.

Small: 2 billion parameters (Qwen3.5-2B with an Unsloth LoRA). GGUF files for laptops and servers are in Privoid-ai/Privoid-FactCheck-Qwen3.5-2B-GGUF.
Long documents: split into chunks automatically; a claim counts as supported if any chunk supports it.
Part of the Privoid trust stack with Privoid Guard (checks inputs for prompt injection) and Privoid Redact (removes personal data).

At a glance

On the official LLM-AggreFact test set (29,320 claims across 11 datasets), Privoid FactCheck averages 74.7 balanced accuracy, with a fixed 0.5 threshold. With a per-dataset threshold picked on the dev split, Privoid FactCheck averages 75.5. That places it #7 of 8 in the comparison below. It scores above Llama-3.1-405B-Instruct.

Model	Size (B)	Avg BAcc	Source
Bespoke-MiniCheck-7B	7	77.4	reported
Claude-3.5 Sonnet	-	77.2	reported
Granite Guardian 3.3 8B	8	76.5	reported
GPT-4o (2024-05-13)	-	75.9	reported
FactCG-DeBERTa-L	0.4	75.6	reported
MiniCheck-Flan-T5-L	0.8	75.0	reported
Privoid FactCheck 2B	2	74.7	ours
Llama-3.1-405B-Instruct	405	74.4	reported
Vectara HHEM-2.1-open	-	72.3	measured

ours / measured: run in this notebook on the same test split.
reported: copied from the LLM-AggreFact leaderboard on 2026-11-04 and not re-run here. The leaderboard page doesn't state how each entry chose its threshold, so compare with care.
Plain Qwen3.5-2B, scored on a sample of up to 100 claims per dataset, averaged 70.8.

By dataset

Dataset	Claims	HHEM-2.1-open (measured)	MiniCheck-Flan-T5-L 0.8B (reported)	Bespoke-MiniCheck-7B (reported)	Privoid FactCheck 2B
CNN	558	62.0	69.9	65.5	57.5
XSum	558	70.6	74.3	77.8	70.1
MediaS	726	67.9	73.6	76.0	72.8
MeetB	772	72.1	77.3	78.3	74.9
WiCE	358	76.2	72.2	83.0	79.3
REVEAL	1,710	86.6	86.2	88.0	89.4
ClaimVerify	1,088	73.6	74.6	75.3	73.6
FactCheck	1,566	71.8	74.7	77.7	77.3
ExpertQA	3,702	58.0	59.0	59.2	59.4
LFQA	1,911	83.4	85.2	86.7	85.9
RAGTruth	16,371	73.3	78.0	84.0	81.3

Per-dataset results, dev-tuned thresholds and training details are in benchmark/results.json.

Examples from the test set

Picked at random. The numbers are support scores from 0 (unsupported) to 1 (supported).

Dataset	Claim (shortened)	True label	HHEM-2.1-open	Privoid FactCheck
RAGTruth	However, they do not take reservations.	supported	0.33	0.35
RAGTruth	The winters can be windy and temperatures often dip below 0 °Celsius (32 °Fahrenheit), sometimes even dropping to minus 20 °C (4 °F).	supported	0.99	0.97
ExpertQA	Another significant development after the Ice Age was the establishment of permanent settlements, such as villages and towns, which were often built around areas with fertile soil and a reliable water source .	supported	0.11	0.03
ExpertQA	Herodotus' "Histories" and Thucydides' "History of the Peloponnesian War," which are some of the earliest historical works and have been immensely influential in our understanding of ancient history .	supported	0.27	0.13
CNN	a 20-year-old british tourist has accused a taxi driver of molesting her . the driver , known as salim , is accused of assaulting the woman in jaipur . rajasthan police said they had taken statements from the woman and her friend . his arre	supported	0.74	0.93
CNN	Abdelkrim Elmezayen, 8, and his 13 - year - old brother died at the scene. Their parents are in a stable condition in the hospital. Firefighters rushed to the port in San Pedro when passers - by heard cries for help. The family of four had	unsupported	0.95	0.82
MeetB	No further action was taken on pending items.	unsupported	0.69	0.35
MeetB	The hearing will be held tonight.	supported	0.69	0.78

Quick start

Transformers + PEFT

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-2B", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "Privoid-ai/Privoid-FactCheck-Qwen3.5-2B").eval()

SYSTEM = "You are Privoid FactCheck. Read the document and the claim. Answer supported only if every part of the claim is stated in or clearly implied by the document; otherwise answer unsupported. Reply with exactly one word: supported or unsupported."
SUP = tok("supported", add_special_tokens=False).input_ids[0]
UNS = tok("unsupported", add_special_tokens=False).input_ids[0]

@torch.no_grad()
def support_score(document, claim):
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "Document:\n" + document + "\n\nClaim:\n" + claim}]
    prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    logits = model(**ids).logits[0, -1, [UNS, SUP]].float()
    return logits.softmax(-1)[1].item()

doc = "The bakery opened in 2019 in Leeds. It sells sourdough bread and cinnamon rolls, and it closes on Mondays."
print(support_score(doc, "The bakery is in Leeds and is closed on Mondays."))   # high
print(support_score(doc, "The bakery opened in 2015 and sells pizza."))         # low

For documents longer than about 1,800 tokens, split them into overlapping chunks and keep the highest score.

llama.cpp (GGUF)

llama-server -hf Privoid-ai/Privoid-FactCheck-Qwen3.5-2B-GGUF:Q8_0 --jinja

Send the system prompt above and Document:\n...\n\nClaim:\n... as the user message, with max_tokens: 3, temperature: 0 and "chat_template_kwargs": {"enable_thinking": false}. The reply is supported or unsupported.

GGUF file	Size
Privoid-FactCheck-Qwen3.5-2B-F16.gguf	3.9 GB
Privoid-FactCheck-Qwen3.5-2B-Q4_K_M.gguf	1.31 GB
Privoid-FactCheck-Qwen3.5-2B-Q8_0.gguf	2.08 GB

Good uses

Checking each sentence of a RAG chatbot's answer against the retrieved passages, and hiding or flagging unsupported ones.
Checking AI-written summaries, reports and meeting notes against their sources.
Scoring RAG systems offline: the share of answer sentences that are supported.

Limitations

English only. Check one claim at a time: split long answers into sentences first.
It checks support from the given document only, not whether a claim is true in the world.
ExpertQA-style claims, with long technical answers and partial evidence, remain hard for every model on the leaderboard, including this one.
Leaderboard entries marked "reported" were not re-run here.

Training details

Method: Unsloth LoRA r=16, alpha=16 on the attention and MLP layers of Qwen/Qwen3.5-2B, trained with TRL SFTTrainer and train_on_responses_only, on one NVIDIA L4.
Run: 450 steps, 1 epoch, 23.4 minutes. Loss 0.30082724094390867 to 0.1285305142402649.
Data: 14,395 synthetic document-claim pairs from MiniCheck's C2D and D2C sets (46% supported). Claims that also appear in LLM-AggreFact were removed. LLM-AggreFact was used only for testing and for the optional dev-split threshold.

Credits and licence

Base model: Qwen/Qwen3.5-2B, Apache-2.0. Loaded from unsloth/Qwen3.5-2B.
Training data: lytang/C2D-and-D2C-MiniCheck (MIT), from MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., 2024).
Benchmark: LLM-AggreFact (CC BY-ND 4.0), used for evaluation only.
Comparison: vectara/hallucination_evaluation_model (Apache-2.0).
This model: Apache-2.0.

Made by Privoid. Keywords: hallucination detection, fact-checking model, RAG groundedness, faithfulness evaluation, claim verification, grounded generation, LLM-AggreFact, MiniCheck alternative, small model, GGUF, llama.cpp, Unsloth.