ImageEval 2026 · Task 1b — Hallucination Detection (English + MSA)
System artefacts for the Ayn-VQA hallucination-detection subtask at ImageEval 2026, ArabicNLP 2026.
Given a culturally grounded image and three statements — exactly one grounded, the
other two plausible hallucinations — label each true / false.
The idea the system is built on
The ranking metric is Contrastive Instability (CI): of the items with at least one correct label, the fraction not fully correct. Lower is better.
The release guarantees exactly one grounded statement per item. Emit exactly one
true and a wrong pick still gets the third statement right, so every item enters the
CI denominator and
CI = 1 - top-1 accuracy
Task 1b is therefore a 3-way choice, not three independent true/false judgements. Measured on identical per-statement scores, switching from an independent 0.5 threshold to exactly-one-true decoding moves CI from 0.1767 to 0.0520 — a 0.125 absolute gain from the decoding rule alone.
Results
| system | track | split | CI ↓ | combined acc ↑ |
|---|---|---|---|---|
| Official baseline (Qwen2.5-VL-3B) | EN | devtest | 0.3133 | 0.6840 |
| Official baseline | MSA | devtest | 0.4900 | 0.5080 |
| Qwen3-VL-8B · MCQ · argmax | EN | devtest | 0.0480 | 0.9520 |
| Qwen3-VL-8B · MCQ · argmax | MSA | devtest | 0.1140 | 0.8860 |
| bilingual blend | MSA | devtest | 0.0540 | 0.9460 |
| ensemble-6 | MSA | devtest | 0.0460 | 0.9540 |
| ensemble-8 | both | dev | 0.0340 | 0.9660 |
Ensemble members (dev accuracy): qwenP3-en 0.958, perstmt-en 0.948, llama90b-en 0.940, nemo12-en 0.924, llama90b-msa 0.920, qwen-msa 0.914, nemo12-msa 0.896, gemma-en 0.882.
What worked, with evidence
| lever | effect (dev n=500) |
|---|---|
| exactly-one-true decoding | 0.1767 → 0.0520 on identical scores |
| permutation averaging (3 orders) | Qwen MCQ 0.054 → 0.042 |
| 8-model equal-weight ensemble | 0.054 → 0.034 |
| a larger VLM | Llama-3.2-90B scored 0.060, worse than 4-bit Qwen3-VL-8B at 0.054 |
Model generation beat model scale. All comparisons use a paired bootstrap over identical items: the ensemble is significantly better than plain Qwen MCQ (+0.0221, 95% CI [+0.0100, +0.0360]) but not significantly better than permutation-averaged Qwen alone (+0.0081, [-0.0060, +0.0220]).
Dataset observations
- Tasks 1a and 1b are the same items — identical ids, and 1a's correct option index equals 1b's grounded index on 3000/3000 train and 500/500 dev.
- English and MSA share gold — verified 3500/3500, so one prediction file is valid on either leaderboard.
- Image-blind length prior: picking the longest statement gets 0.540 top-1 on dev (chance 0.333). Ties must break to one index; counting every tied index inflates this to 0.81.
- Grounded position is uniform (0.319 / 0.336 / 0.344) — no positional shortcut.
Contents
submissions/ Codabench archives · predictions/ prediction CSVs and per-item score
audits · metrics/ metric panels and LaTeX tables · figures/ paper figures (PNG+PDF) ·
config/ prompts and run configuration · notebooks/ the end-to-end Colab pipeline.
Data and licence
Built on QCRI/AynVQA-ArabicNLP26,
CC BY-NC 4.0, non-commercial research only. This repo contains no dataset content:
no images, no audio, no statements, no gold labels.