Teena2812/hingverify-mdeberta

HingVerify: citation verifier for Hindi, Hinglish and English

HingVerify decides whether a claim is supported by an evidence passage. It is built for retrieval-augmented generation (RAG) systems that answer questions about Indian government schemes, where the user and the answer may be in English, Hindi (Devanagari) or Hinglish (Roman-script Hindi mixed with English), and the official source may be in a different language.

• Labels: entailment = SUPPORTED, contradiction = CONTRADICTED, neutral = NOT ENOUGH INFO
• Input: premise = evidence passage, hypothesis = claim (max 512 tokens; truncate the evidence, not the claim)
• Base model: mDeBERTa-v3-base fine-tuned on multilingual NLI (~280M parameters)
• Live demo and API: https://huggingface.co/spaces/Teena2812/hingverify

Usage

from transformers import pipeline
nli = pipeline("text-classification", model="Teena2812/hingverify-mdeberta", top_k=None)
evidence = "Under the scheme an income support of 6,000/- per year in three equal installments will be provided to all land holding farmer families."
print(nli({"text": evidence, "text_pair": "Kisan ko saal mein 8 hazaar milte hain."}))
# contradiction has the highest score

Training

Fine-tuned on 801 synthetic claim–evidence pairs (126 for validation) written from official passages of 17 government schemes, balanced across the three labels and three claim languages, with hard negatives (wrong number, date, group, frequency or entity; negation; lakh/crore confusion; unsupported added details). A blind 10% check agreed on 92/93 labels. Learning rate 2e-5 (searched over 1e-5, 2e-5, 3e-5), batch size 16, up to 6 epochs with early stopping on validation macro-F1, 10% warm-up, weight decay 0.01, seed 42 (best of 42/43/44 on validation). One NVIDIA T4 GPU.

Evaluation

Human-adjudicated test set: 366 pairs from 6 schemes not seen in training (Cohen's kappa 0.946 between the human annotator and the final labels). Three-way macro-F1:

Claim language	Stock mDeBERTa-v3	HingVerify (mean ± SD, 3 seeds)	gpt-oss-120b judge
English	0.836	0.880 ± 0.019	0.951
Hindi	0.828	0.949 ± 0.017	0.967
Hinglish	0.792	0.856 ± 0.017	0.983
All	0.819	0.895 ± 0.013	0.967

Gains over the stock model are significant on Hindi (p = 0.0013) and Hinglish (p = 0.049), exact McNemar test. Inside an agentic RAG pipeline (60 citizen questions), using HingVerify instead of the stock verifier raised answer correctness from 0.917 to 0.979 (not statistically significant at this sample size).

Limitations

• A large LLM judge is still more accurate, especially on Hinglish and when claim and evidence are in different languages.
• Weak spots: claims about the wrong beneficiary group and about payment frequency.
• Trained on synthetic data from one domain (Indian government schemes); expect lower accuracy elsewhere.
• It checks whether a passage supports a claim; it does not check whether the passage itself is current or true. Always confirm important information on the official scheme website.

Authors

Aditya Singh, Teena Kalia, Muntaha Malik, Dr. Sandeep Gupta. Dept. of Computer Science and Engineering, Sharda University.