Sarab

Multimodal AI models trained mostly on English and Western visual data frequently hallucinate on Arabic-language images, and no benchmark tells you why it happened, only that it did. Sarab (سراب), Arabic for "mirage," is a cause-diagnostic Arabic visual hallucination benchmark: instead of one accuracy number, it traces each wrong answer to a specific root cause, a misread image, misleading text, a cultural blind spot, or reflexive guessing under an absent answer.

Five modes, five root causes

What makes it rigorous

• 465 images, every one fully captioned across five Arabic Cultural Visual Vocabulary categories (architecture, attire, cuisine, cultural objects, script), and every one tied to an actual test question, nothing unused
• A purpose-built human review tool vets every image, caption, and distractor before it enters any test set
• Real, OpenRouter-evaluated results for four models (Gemini 2.5 Flash, Gemini 2.5 Flash Lite, GPT-4o-mini, Qwen2.5-VL-72B) reported in the paper

Status

Four of eight originally scoped models are evaluated; the three Arabic-centric models (AIN, Fanar, ALLaM) are next. Every known gap is documented in the paper and repository, not smoothed over. Dataset, code, and prompts are released for reuse.

Citation

Paper in preparation. This entry will be replaced once it's published. Until then, cite the code repository:

@misc{sarab2026,
  title        = {Sarab: A Cause-Diagnostic Arabic Visual Hallucination Evaluation Benchmark},
  author       = {Alharz, Zahra and Mahyoub, Abdulrhman and Barmandah, Hassan and Alahmari, Saad Saeed},
  year         = {2026},
  howpublished = {\url{https://github.com/HasanBGit/Sarab-Benchmark}},
  note         = {Paper in preparation; citation will be updated on publication.}
}