AIDAM_VSCI_279M_V1.0.0

A scientific-writing verifier. Given a cited abstract and a claim, the model judges whether the evidence supports, refutes, or has not enough information for the claim — the SciFact register (claims checked against biomedical/scientific abstracts).

This is part of AIDAM, a project built around a simple rule: the factual verdict on a claim never comes from a large language model. It comes from a small, specialised NLI encoder like this one, plus deterministic aggregation code. The LLM in the pipeline drafts and explains; it does not judge.

Model details

Task	3-class NLI (entailment / neutral / contradiction)
Architecture	mDeBERTa-v3-base
Parameters	278,811,651
Base checkpoint	MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7
Formats provided	ONNX (2.5 MB graph + 1.1 GB external weights) and SafeTensors (~532 MB)
Inference window	1024 tokens, no chunking (SciFact abstracts run longer than FEVER's evidence: 23% of pairs exceed 512 tokens, median 327, p99 952)
License	Apache 2.0
Language	Base model is multilingual by construction; fine-tuning and evaluation used English scientific claims only — see "What this is not"

Benchmark: SciFact

Measured on the SciFact dataset (Wadden et al., 2020), dev split — 300 claims joined against cited documents into 340 pairs, an unbalanced set ("not enough" 131, "supports" 138, "refutes" 71). Metric: label accuracy (plain accuracy, matching how the previous-generation figure was reported; balanced accuracy is also given below for context given the class imbalance).

	Previous-generation verifier	This model	Change
Accuracy	63.24	76.76	+13.52
"not enough" recall	44.27	82.44	+38.17
"refutes" recall	78.87	67.61	−11.26
"supports" recall	73.19	76.09	+2.90

This model was measured against a pre-registered, four-clause gate before training began:

• Accuracy > 75.10 (gate threshold): scored 76.76 — pass, 1.66 points above threshold
• No class recall regresses beyond a measurement-noise allowance versus the previous-generation verifier: pass on all three classes ("refutes" cleared its floor narrowly — see "What this is not")
• Parameter budget < 500,000,000: 278,811,651 — pass
• Zero training/dev/test contamination, verified by claim ID: pass (809 train, 300 dev, 300 test IDs, pairwise disjoint)

Unlike the FEVER-register sibling of this model, the previous-generation figure quoted for SciFact (66.3 label accuracy) largely reproduces in this project's own harness: 63.24 measured, 0.52 standard errors away given SciFact's small 340-pair dev split (noise band 5.93 points, almost ten times wider than FEVER's). Every number above is measured against the reproduced 63.24 baseline.

Training

• Base checkpoint: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7, already NLI-pretrained (zero-shot on SciFact's own dev split: 56.63 balanced accuracy, far below its 88.83 zero-shot score on FEVER — this register needed real fine-tuning)
• Training corpus: SciFact's train split, natural class mix, no oversampling — 327 entailment, 298 neutral, 174 contradiction (799 rows, after holding out 120 rows for an internal dev set separate from SciFact's own 340-pair official dev)
• The lever that reached the gate: class-weighted cross-entropy loss in place of row duplication. Per-class weight = len(train) / (3 x count[class]) — 0.816 entailment, 0.895 neutral, 1.531 contradiction — computed once from the natural, undupllicated corpus. Six earlier attempts tried duplicating contradiction rows (up to 400, from a natural 174) to fix its recall; a larger duplicate buffer made recall worse, not better, at high epoch counts, consistent with the model memorizing a finite repeated set rather than generalizing. Weighting the loss instead corrected the same class imbalance without any row being seen more than once per epoch.
• Learning rate 2e-5, 60 epochs (1,440 steps), effective batch size 32, max sequence length 1024, OneCycleLR schedule with 6% warmup, 8-bit AdamW, bf16
• Best checkpoint: step 1,000 of 1,440, internal dev balanced accuracy 77.60
• Hardware: a single 12 GB consumer GPU, ~19 minutes training time
• This was the 7th training attempt for this specialisation. Across attempts 1-6, the two relevant gate clauses never passed at the same time: reaching the accuracy threshold needed roughly 45-60 epochs, but by 60 epochs the row-duplication-based correction had "refutes" recall declining rather than holding. The 7th attempt changed the correction mechanism itself, not just its strength, holding epoch count fixed at the 6th attempt's value so it was the one isolated variable.

What this is not

• This score is specific to the scientific-writing register (SciFact-style claims checked against cited abstracts). It is not a general reasoning or knowledge benchmark, and it says nothing about the model's behavior on encyclopedic claims, news-grounding claims, or viral/real-world claims — those are judged by sibling models in the same family, each measured on its own register.
• The base checkpoint is multilingual (trained on the CC100 corpus across 100 languages, further tuned on XNLI, which includes Spanish), but this fine-tune's own training and evaluation used SciFact's English claims only. No Spanish-language measurement exists for this checkpoint; multilingual capacity is present in the weights but unverified for this task.
• "refutes" recall is well below the previous verifier's own number (67.61 versus 78.87) — a real, bounded trade for large gains on "not enough" and "supports" recall. The gate margin on this specific class is narrow (0.60 points over the floor, a tenth of the class's own noise band); a different SciFact dev sample could plausibly move it below the floor.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("DeliVali/AIDAM_VSCI_279M_V1.0.0")
model = AutoModelForSequenceClassification.from_pretrained("DeliVali/AIDAM_VSCI_279M_V1.0.0")

evidence = (
    "In a randomised sample of 412 participants, supplementation was "
    "associated with a modest reduction in reported fatigue (p = 0.04); "
    "the effect did not persist at twelve weeks."
)
claim = "This supplement permanently cures fatigue."

inputs = tokenizer(evidence, claim, return_tensors="pt", truncation=True, max_length=1024)
with torch.no_grad():
    logits = model(**inputs).logits
label = logits.argmax(-1).item()  # 0=entailment, 1=neutral, 2=contradiction