ImageEval 2026 · Task 2 — CRAI-Bench
System artefacts for CRAI-Bench: given a reference image of a Qatari cultural scene, the caption used to prompt a text-to-image model, and the generated image, predict five cultural accuracy dimensions.
CRAI_composite = 0.30*CEA + 0.20*CC + 0.20*CS + 0.20*CI - 0.10*HP
Ranking metric is Spearman on the composite; MAE is the tiebreaker.
Results — dev (n=40), fit on train, held out
| system | Spearman ↑ | MAE ↓ |
|---|---|---|
| Organiser GPT-4 baseline (as reported) | 0.6250 | 0.2248 |
GPT-4 raw, scoring the shipped gold_llm.tsv |
0.2153 | 0.3666 |
| Qwen3-VL-8B raw | 0.5482 | 0.2550 |
| Qwen3-VL-8B calibrated + version | 0.7064 | 0.1960 |
| version one-hot only — no image, no judge | 0.6823 | 0.2091 |
Per-dimension Spearman for the submitted system: CEA 0.716, CC 0.711, CS 0.726, CI 0.730, HP -0.095.
Method
A VLM judge scores the five dimensions from (reference image, caption, generated image);
a per-dimension ridge maps [5 raw dims + caption-version one-hot] to the human scores,
fit on the 120 train instances. The composite is always recomputed with the official
formula, never predicted.
Two facts drive this:
- The organisers note GPT-4 "systematically overestimates cultural accuracy". It does, on every dimension — bias +0.142 (CEA) to +0.247 (CI), +0.182 on the composite.
- Spearman is rank-based, so monotone recalibration cannot change it. Calibration buys MAE only; improving the ranking needs new signal. Caption version supplies it.
Caveats stated plainly
- n = 40. Spearman's standard error is roughly ±0.16. Our 0.7064 versus the version-only control's 0.6823 is not a significant gap.
- Train CV was optimistic: 0.8159 under 6-fold grouped CV versus 0.7064 on dev.
- A model that never sees an image scores 0.6823. Caption version is close to a label — a benchmark design artefact that any honest system paper must report.
- The organisers' GPT-4 baseline did not reproduce: their shipped
gold_llm.tsvscores 0.2153 on dev, not the reported 0.6250. - Humans almost never apply a hallucination penalty (mean HP 0.008 vs GPT-4's 0.173), and GPT-4's HP ranking is anti-correlated with theirs (rho -0.114).
Data and licence
Built on the organisers' CRAI-Bench release, distributed via their Google Drive and subject to their terms; the task repository asks that data not be committed publicly. This repo contains no dataset content — no images, no captions, no gold labels. Contact: imageeval2026@gmail.com