FaustDerLegende/RAGVERIFIER

ClaimLens: Multilingual RAG Hallucination Verifier

ClaimLens is a fine-tuned multilingual (English, Hindi, Kannada) claim-level hallucination verifier for Retrieval-Augmented Generation (RAG) pipelines based on xlm-roberta-base.

It evaluates generated answers claim-by-claim against retrieved context passages and classifies each into one of three factual grounding states:

• Correct (Label 0): Retrieved evidence entails the claim.
• Incorrect (Label 1): Retrieved evidence contradicts the claim.
• Unverified (Label 2): Evidence neither confirms nor denies the claim (unsupported / baseless).

---

Model Architecture & Training Protocol

• Base Model: xlm-roberta-base
• Task: Sentence-Pair Classification (<s> claim </s></s> evidence </s>)
• Two-Stage Supervised Fine-Tuning:
  1. Stage A (NLI Warm-up): Trained on 60,000 NLI pairs (20k each for English, Hindi, and Kannada from facebook/xnli and Divyanshu/indicxnli).
  2. Stage B (RAG Fine-Tuning): Fine-tuned on RAGTruth claim-level data (English) + machine-translated Hindi and Kannada subsets, using Class-Weighted Cross-Entropy to mitigate the ~90% majority class skew.

---

Evaluation Results

Evaluated against the RAG test sets across languages:

Metric	English (en)	Hindi (hi)	Kannada (kn)
Macro-F1	0.4888	0.6086	0.5662
Hallucination F1	0.3676	0.8162	0.8057
Accuracy	76.61%	61.00%	56.67%
IndicGLUE WNLI (Hindi)	—	94.37%	—

Significant performance improvement over Majority Class (0.1667) and TF-IDF baselines (0.4059 - 0.4270).

---

Quickstart & Usage

Using Hugging Face Transformers

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "FaustDerLegende/RAGVERIFIER"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

claim = "Antigravity is an AI development platform."
evidence = "Antigravity is an AI development platform created by Deepmind engineers."

# Tokenize sentence-pair (claim, evidence)
inputs = tokenizer(
    claim,
    evidence,
    truncation="only_second",
    max_length=384,
    return_tensors="pt"
)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

id2label = model.config.id2label
for i, prob in enumerate(probs):
    print(f"{id2label[i]}: {prob.item():.4f}")

predicted_class = id2label[torch.argmax(probs).item()]
print(f"
Final Verdict: {predicted_class}")

Using Hugging Face Pipeline

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="FaustDerLegende/RAGVERIFIER",
    tokenizer="FaustDerLegende/RAGVERIFIER"
)

result = classifier({
    "text": "Antigravity is an AI development platform.",
    "text_pair": "Antigravity is an AI development platform created by Deepmind engineers."
})
print(result)

---

Citation & Acknowledgment

Developed as part of the ClaimLens Multilingual Hallucination Detection project, adapting datasets from RAGTruth (Niu et al.), IndicXNLI (Aggarwal et al.), and IndicGLUE.