AIDAM_VREAL_279M_V1.0.0

A real-world and viral-claims verifier. Given evidence and a claim, the model judges whether the evidence supports, refutes, or has not enough information for the claim — the AVeriTeC register (fact-checked claims drawn from 50 fact-checking organizations, with evidence gathered as question-answer pairs from the web).

This is part of AIDAM, a project built around a simple rule: the factual verdict on a claim never comes from a large language model. It comes from a small, specialised NLI encoder like this one, plus deterministic aggregation code. The LLM in the pipeline drafts and explains; it does not judge.

Model details

Task	3-class NLI (entailment / neutral / contradiction)
Architecture	mDeBERTa-v3-base
Parameters	278,811,651
Base checkpoint	MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7
Formats provided	ONNX (2.5 MB graph + 1.1 GB external weights) and SafeTensors (~532 MB)
Inference window	512 tokens, chunked automatically when exceeded
License (weights)	Apache 2.0
License (training data)	The AVeriTeC dataset used to train this model is released under CC-BY-NC 4.0 by its authors — a deliberate, documented exception to this project's usual data licensing; the published weights here remain Apache 2.0, unaffected by the training data's own license
Language	Base model is multilingual by construction; fine-tuning and evaluation used English claims only — see "What this is not"

Benchmark: AVeriTeC

Measured on the AVeriTeC dataset (Schlichtkrull, Guo & Vlachos, 2023), dev split — 500 claims. Metric: label accuracy, with gold evidence provided (this model never retrieves evidence itself — judge() always takes evidence as an argument — so at this single-verifier evaluation layer, label accuracy against gold evidence is equivalent to the shared task's official "AVeriTeC score", which otherwise also grades retrieval quality).

	Previous-generation verifier	This model	Change
Accuracy	60.0	72.0	+12.0
"not enough" recall	42.47	46.58	+4.11
"refutes" recall	63.61	76.07	+12.46
"supports" recall	61.48	77.05	+15.57

This model was measured against a pre-registered, four-clause gate before training began:

• Accuracy > 71.70 (gate threshold): scored 72.0 — pass, but narrow (0.30 points above threshold, about a twentieth of the measurement noise band — see "What this is not")
• No class recall regresses beyond a measurement-noise allowance versus the previous-generation verifier: pass, comfortably, on all three classes
• Parameter budget < 500,000,000: 278,811,651 — pass
• Zero training/dev contamination, verified by exact claim-text match: pass

A previously circulated figure of 62.6 label accuracy for this project's incumbent verifier needed real correction work before it could be reproduced. The incumbent's originally published ONNX export scored 31.2% on the same 500 claims — worse than always guessing the majority class (61%) — which turned out to be an export defect, not a weaker model: the same architecture's full-precision weights scored 58-60% directly. A corrected re-export with a parity check against the full-precision weights (0 disagreements across 100 sampled pairs) gives the true incumbent baseline of 60.0, which every number above is measured against.

Training

• Base checkpoint: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7, already NLI-pretrained, not a raw checkpoint
• Training corpus: AVeriTeC's own train split, natural class mix — 1,660 contradiction (57%), 804 entailment (28%), 445 neutral (15%), after removing 6 rows that share claim text with the 500-item official dev split
• The lever that reached the gate: class-weighted cross-entropy loss (per-class weight = len(train) / (3 x count[class]): 1.203 entailment, 2.172 neutral, 0.586 contradiction), combined with enough additional training epochs to let accuracy catch up once the class-balance fix was in place. A first attempt with no class weighting reached 72.2% accuracy by collapsing "not enough" recall to 15.07%; adding class weights fixed the collapse (45.21% recall) but dropped accuracy to 65.4%; two further attempts doubled epoch count twice (10 to 20 to 40), recovering accuracy to 67.6% then 72.0% while "not enough" recall never gave the fix back (staying in the 43.8-46.6% range throughout)
• Learning rate 2e-5, 40 epochs (3,600 steps), effective batch size 32, max sequence length 512, OneCycleLR schedule with 6% warmup, 8-bit AdamW, bf16
• Best checkpoint: step 1,950 of 3,600, internal dev balanced accuracy 73.77
• Hardware: a single 12 GB consumer GPU, ~26 minutes training time
• This was the 4th training attempt for this specialisation — the fastest path to a promoted checkpoint across this project's four register verifiers, because the class-weighted-loss approach that a sibling model's search needed six attempts to discover was already known and available from the first attempt here.

What this is not

• This score is specific to the real-world and viral-claims register (AVeriTeC-style fact-checks). It is not a general reasoning or knowledge benchmark, and it says nothing about the model's behavior on encyclopedic claims, news-grounding claims, or scientific claims — those are judged by sibling models in the same family, each measured on its own register.
• The base checkpoint is multilingual (trained on the CC100 corpus across 100 languages, further tuned on XNLI, which includes Spanish), but this fine-tune's own training and evaluation used AVeriTeC's English claims only. No Spanish-language measurement exists for this checkpoint; multilingual capacity is present in the weights but unverified for this task.
• The accuracy margin over the gate threshold is narrow (0.30 of a 5.85-point noise band) — the narrowest first-pass accuracy margin among this project's register verifiers. A different sample of AVeriTeC's 500-claim dev split could plausibly move this specific number below the threshold, even though the per-class recall margins are comfortably wide.
• A meaningful share of AVeriTeC's "refuted" claims are refuted on source credibility grounds (for example, "this was published by a satire site") rather than by direct textual contradiction — a reasoning shape this NLI verifier was not specifically built or trained to cover.
• Training data license note: the AVeriTeC dataset used to train this checkpoint is released under CC-BY-NC 4.0. The published model weights here are Apache 2.0, as with every other model in this family — only the training data itself carries the more restrictive license.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("DeliVali/AIDAM_VREAL_279M_V1.0.0")
model = AutoModelForSequenceClassification.from_pretrained("DeliVali/AIDAM_VREAL_279M_V1.0.0")

evidence = (
    "Q: Where was the claim first published? A: On a satire website that "
    "publishes fictional news stories."
)
claim = "A celebrity said this outrageous thing in a real interview."

inputs = tokenizer(evidence, claim, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits
label = logits.argmax(-1).item()  # 0=entailment, 1=neutral, 2=contradiction

For evidence longer than 512 tokens, chunk and aggregate chunk-level predictions.