mDeBERTa-v3-base-grounding-multilingual

A multilingual RAG grounding / faithfulness checker. Given retrieved sources (the grounding context) and a candidate response, it scores — per claim, as a 3-class NLI problem — whether the response is grounded in the sources. It is the correctness core of a fail-closed hallucination detector: a claim escapes the hallucination flag only by being entailed (SUPPORTED) by the sources; a CONTRADICTED or UNSUPPORTED claim raises the flag.

This repository ships the blessed int8 ONNX serving artifact — the exact three files the runtime loader requires — plus this card. It is a fine-tuned, re-exported derivative of the MIT-licensed MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7.

Serving artifact (the frozen seam)

Three files, in the loader's exact names, produced by the project's single export path (fp32 → ONNX → dynamic int8, QInt8):

file	role
model_quantized.onnx	the served int8 weights (dynamic-quantized)
tokenizer.json	the checkpoint's own fast SentencePiece tokenizer
config.json	model config; id2label = {0: entailment, 1: neutral, 2: contradiction}

The logit-column order 0=entailment, 1=neutral, 2=contradiction is load-bearing: the runtime scorer reads the columns by position, and every downstream verdict depends on this mapping. The graph takes exactly input_ids + attention_mask (int64) and no token_type_ids (the model has type_vocab_size 0). CPU inference via onnxruntime.

Intended use

• RAG grounding / faithfulness verification: is a generated answer supported by the retrieved context? Runs per-claim over an answer, max-pooling over premise windows of the sources.
• Ten language varieties: en, es, fr, it, ca, gl, eu, va, pt-PT, pt-BR.
• Operating discipline — fail-closed. The detector raises the hallucination flag conservatively: a claim must be affirmatively entailed to pass; a contradicted or merely-unsupported claim is flagged. Ship-worthiness was decided by a per-variety go/no-go gate (below), not by an averaged score — one failing variety sinks the decision.

Not intended for

General-purpose NLI outside a grounding/faithfulness setting, standalone fact-checking without retrieved sources, or as a substitute for human review on the low-resource varieties (see Limitations).

Evaluation — metrics at the shipped operating point (tau_entail = 0.45)

All numbers below are the go/no-go gate output at tau_entail = 0.45 (the shipped default), the exact per-variety recall / false-alarm the ship decision was made on — not rounded-up or aspirational figures. The bar, per variety, is recall ≥ 0.85 AND false-alarm ≤ 0.15; all ten clear it (verdict: GO).

• Recall = fraction of gold-fabricated claims (CONTRADICTED ∨ UNSUPPORTED) the detector flags (entail < tau_entail). The safety-critical, recall-first number.
• False-alarm = fraction of gold-SUPPORTED claims wrongly flagged.

variety	recall	false-alarm	validation
en (English)	0.9923	0.0116	human-labelled (RAGTruth)
es (Spanish)	0.9688	0.0312	human-labelled (X-Fact)
fr (French)	1.0000	0.0000	human-labelled (X-Fact)
it (Italian)	0.9737	0.0204	human-labelled (X-Fact)
ca (Catalan)	0.9733	0.0625	synthetic (label-by-construction)
gl (Galician)	1.0000	0.0000	synthetic (label-by-construction)
eu (Basque)	0.9833	0.0250	synthetic (label-by-construction)
va (Valencian)	1.0000	0.0000	synthetic (label-by-construction)
pt-PT (European Portuguese)	0.9744	0.0392	human-labelled (X-Fact)
pt-BR (Brazilian Portuguese)	1.0000	0.0000	synthetic (label-by-construction)

Assembled gate: RAGTruth (en) + X-Fact (es/fr/it/pt) human labels + the held-out by-construction multilingual gate. Metrics are computed by eval.metrics.breakdown_by_lang at tau_entail = 0.45 — see Reproducing the metrics.

Limitations

• Low-resource varieties are synthetic-validated, NOT human-benchmarked. The ca, gl, eu, va metrics (and pt-BR) come from label-by-construction synthetic data, not held-out human judgements. They demonstrate the operating point transfers, but they are not a human benchmark; treat these varieties as lowest-confidence and keep a human in the loop.
• Truncation. Very long premise/claim pairs can be truncated by the tokenizer; the detector surfaces a truncation flag rather than silently failing open, but real sub-sentence splitting is future work.
• tau_contra (contradiction severity) is a provisional, precision-biased guess; it is a severity label only and never enters the flag or the recall above.

License & lineage (honest disclosure)

• Weights license: cc-by-nc-nd-4.0 — non-commercial, no-derivatives, attribution. The published fine-tune may be used for non-commercial purposes with attribution; it may not be used commercially, and derivatives/redistribution of modified weights are not permitted. (Our right to use the MIT base is satisfied; that is not a grant of commercial/fork rights over this derivative.) See LICENSE.
• Base model / lineage (MIT, attributed — not misrepresented): this is a fine-tuned, re-exported derivative of MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7, which remains MIT upstream. We restrict only our artifact; the base itself is not non-commercial. Its MIT permission notice is retained in LICENSE.
• Training data is NOT openly licensed as a whole. The cc-by-nc-nd-4.0 weights license covers the weights only. The model was trained partly on synthetic data published under evaluation-only / all-rights-reserved terms (our by-construction fact banks + eu/va synthesis) and partly on borrowed human-labelled data used under its own terms (RAGTruth / RAGTruth-MT — MIT; X-Fact — MIT; Mu-SHROOM — CC-BY-4.0, deferred). The synthetic training data is not redistributable or openly reusable; do not treat the training corpus as openly licensed.
• Training-data lineage is cross-linked to the dataset package (multilingual-grounding-eval) and its provenance.json, which is the source of truth for per-source roles and licenses.

Reproducing the metrics

The per-variety numbers above are the go/no-go gate output at tau_entail = 0.45:

from hallucination_detector.config import Config
from eval.metrics import breakdown_by_lang
from training.run_pipeline import GATE_CASE_FILES, _collect_gate_rows

cfg = Config(model_dir="<this model dir>", tau_entail=0.45)
rows = _collect_gate_rows(cfg, GATE_CASE_FILES)
per_variety = breakdown_by_lang(rows, 0.45)  # recall / false_alarm per variety