Culture in Action: Evaluating Text-to-Image Models through Social Activities
Authors: Sina Malakouti
Boqing Gong
Adriana Kovashka
Venue: ICLR 2026 Poster (2026)
Keywords: Text-to-Image Models
Cross Cultural Evaluation
Fairness
Bias
Social Activities
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Cultural nuances are best expressed through social interactions, yet current text-to-image (T2I) benchmarks focus largely on object-centric artifacts (e.g., food, landmarks, and attire). In this work, we study the cultural faithfulness of T2I models (i.e., adherence to the target culture) through social activities. To this end, we introduce CULTIVate, a new benchmark of 576 activities across 9 categories (e.g., dancing, greeting, dining) with over 19,000 images from 16 countries. We further propose AHEaD, an explainable framework that measures cultural understanding along four dimensions: cultural Alignment, Hallucination, Exaggeration, and Diversity. Unlike prior work relying on costly human evaluation or image-text alignment (ITA), AHEaD uses culturally-grounded descriptors to provide quantitative, interpretable feedback that enables iterative image refinement. Our analysis shows ITA metrics correlate poorly with human judgments and that alignment alone is insufficient to capture faithfulness. In contrast, FAITH (combining alignment, hallucination, and exaggeration) achieves 27% higher correlation than baselines. Finally, we observe systematic disparities, with generated images being consistently more faithful for Global North than Global South cultures.

Published as a conference paper at ICLR 2026
CULTURE IN ACTION: EVALUATING TEXT-TO-IMAGE
MODELS THROUGH SOCIAL ACTIVITIES
Sina Malakouti
Boqing Gong
Adriana Kovashka
University of Pittsburgh
Boston University
University of Pittsburgh
sem238êpi edu
bgongebu.com edu
kovashka@cs pitt edu
ABSTRACT
Cultural nuances are best expressed through social interactions, yet current text- to-image (T2I) benchmarks focus largely on object-centric artifacts (e.g.. food, landmarks, and attire). this work, we study the cultural faithfulness models (i.e., adherence to the target culture) through social activities. To this end, we introduce CULTIVate, a new benchmark of 576 activities across 9 categories (e.g.. dancing, greeting, dining) with over 19,000 images from 16 countries. We further propose AHEaD, an explainable framework that measures cultural under- standing along four dimensions: cultural Alignment, Hallucination, Exaggeration, and Diversity. Unlike prior work relying on costly human evaluation or image-text alignment (ITA), AHEaD culturally-groundder descriptors to provide quanti- tative, interpretable feedback that enables iterative image refinement. Our anal- ysis shows ITA metries correlate poorly with human judgments and that align- ment alone is insufficient to capture faitlifulness. In contrast, FAITH (combining alignment, hallucination, and exaggeration) achieves 27% higher correlation than baselines. Finally, we observe systematic disparities, with generated images being consistently more faithful for Global North than Global South cultures.
T2I
INTRODUCTION
The 2007 film Ratatouille earned 41 film awards including Best Feature the 2008 Oscars (Wikipedia). Part of its appeal lies in the very realistic portrayal of the city of Paris, and of French culture and cuisine (Seat)leTimes). To achieve this, creators visited places in Paris to soak in the culture and environment, including its highly distinctive visual aspects. Many other well-regarded films (animated ones like Luca and Coco, and live action ones like Amelie, Crouching Tiger Hid- den Dragon, and Reservation Dogs) also devoted significant effort to ensuring they capture the true atmosphere and visuals of the places they portray. Such culturally accurate visual portrayals are important for many types of creative and marketing content beyond film, e.g., advertising.
Recent advances in -mnate (T2I) generative models offer the promise of automating creation of such content. However, T21 models are trained web data exhibiting strong WEIRD biases (Western, Educated, Industrialized, Rich, and Democratic) (Henrich et al. 2010). leading to incor- rect or overly stereotypical cultural representations. This problem is particularly severe for social activities, where cultural meaning emerges from context, interactions, and relations between objects and people (Geertz) 2017; Hall 1973). Despite this, cross-cultural studies of T2I models remain understudied, with existing benchmarks focusing on few specific object-centric artifacts such as landmarks, elothing, and food (Chiu et al. 2025a Basu al. 2025 Rege et al., (2055).
In this work, we examine how well T2I models portray different cultures through activities, whose visual representations vary significantly across cultures. Unlike static artifacts, activities are con- textual and compositional, encompassing objects, interactions, and spatial arrangements that better capture cultural expression. For example, "eating at home in Iran" may involve sitting at a table or gathering the floor around a traditional sofreh- the same activity can have multiple valid cul- tural variants. To this end, we introduce CULTural ac acTIviTy (CULTIVate), a culturally uulturaly--roounded benchmark spanning 16 countries and 576 activities across categories (e.g., dining, greeting, game, dance, celebration). We evaluate 6 state-of-the-ar T2I models, generating 19,000+ images and col- lecting 3,000 real reference images. As shown in Fig.
T (b), T21 models may generate wrong activi-

---

Published as a conference paper at ICLR 2026
Nyepl Celebration Indonesia
Elephant ant man
Zu Game
People playing Elephant
Game Indonesio
Iran
Durs: 0.46
CLiPScone: 0.22
Ours: 0.60
CLiPScore: 0.18
Alignment
0.18
Aligrment
Hallucination
Hallucination
Exaggeration
Exeggeration
0.12
Alignment
Halluctnation
Exoggerating otion
Bolle shirls
clithing Usiquiloui Seranga
Hijsss
Beard
Shadow puppels
(o)
(b)
Figure 1: (a) Our framework captures aligmment, hallucination, and exaggeration via interpretable descriptors. CLIPScore incorrectly scores the culturally wrong image (red border) higher due to exaggerated and hallucinated elements (e.g., elephants), while our metrics more accurately evaluate cultural faithfulness by capturing these 3 complementary metrics. Bottom: descriptor-level feedback identifies specific issues. (b) T2I systems generate incorrect and exaggerated images.
ties, include hallucinated elements, produce exaggerated scenes. These complex failure patterns raise critical questions: Are current metrics effective for evaluating cultural faithfulness? What makes a metric effective for this task?
Our work explores cultural faithfulness (faithfulness), whether images accurately represent the tar- get culture. Prior works rely costly human evaluation (Kannen al. 2024 Bayramli et al. 2025)
while (Rege et al., 2025; Khanuja et al. 2024) used -bMeba saang--txx--alggmmee (TTA) met- rics (e.g. CLIPScore (Hessel et al.] 2021
proxy for human judgment of faithfulness. However, VLMs inherit similar cultural biases (Rege et al. 2025), exhibit poor compositional and implicit prompt understanding (e.g., bag-of-words behavior) (Yuksekgonul al.. 2023), making ITA unre- liable for cultural evaluation. For example, Fig. [T) shows ITA metrics reward behaviors such as exaggeration and hallucination such generating elephants for "clephant ant man game" rock- paper-scissors game in Indonesia). In fact, our analysis reveals that in contrast to human judg- ment, ITA metrics correlate positively with exaggeration, where more stereotypical elements increase alignment but hurt faithfulness.
To cope with these challenges and answer above questions, we introduce AHEaD (Alignment, Hallucination, Exaggeration, and Diversity), a diagnostic framework using external visual de- scriptors. We decompose activities into interpretable visual descriptors that capture cultural el- ements across multiple dimensions. First, create reference descriptors using a proposer-refinee approach where multiple proposers utilize an LLM to generate diverse candidates, and the refiner filters duplicates and errors. Second, we extract predicted descriptors from generated images using MILMs. Unlike prior work that uses VLMs to directly score faithfulness, we use MILMS only for generic scene understanding, avoiding reliance their cultural biases. Finally, we compare ref- erence against predicted descriptors to compute AHEaD metrics. AHEaD provides interpretable insights where Alignment measures cultural coverage, Hallucination quantifies incorrect elements, Exaggeration measures over-representation and Diversity captures semantical variation in cultural elements. Our framework enables identifying which cultural aspects are missing, over-represented, or faithfully depicted.
In more detail, our framework provides three key capabilities. First, we compute AHEaD metrics automatically without human annotations, enabling scalable evaluation across countries and T21 models. The metrics compare different aspects of quality in the images generated by different T2I models, and can be used to quantitatively judge which T2I model to deploy when aiming to depict particular country. Second, AHEaD outputs interpretable diagnostics including top-k and bottom-k descriptors for missing, hallucinated, and exaggerated elements, supporting targeted model improve- ments through seeeriptorguidee editing. Third, we analyze correlations between metrics to reveal trade-offs, such whether increasing alignment affects hallucination exaggeration.
2

---

Published as a conference paper at ICLR 2026
We conduct comprehensive experiments CULTIVate revealing systematic limitations in current cultural evaluation. We show that existing -TAebased metrics correlate poorly with human judgment. Importantly, analysis suggests AHEaD metrics are complementary, where combining Aligmment, Hallucination, and Exaggeration (FAITH) achieves highest correlation than using Alignment alone. FAITH shows 27% higher correlation with human judgment of faithfulness than MI.LM-as-judee baselines and significantly outperform ITA metrics. Finally, we find consistent bias across all T2I models, with 4-8% higher Alignment for Global North (GN) than Global South (GS) countries.
To summarize, our contributions are:
1. We introduce CULTIVate, benchmark for evaluating cultural faithfulness of T2I models through social activities.
We propose AHEaD, a framework for diagnosing cultural faithfulness across multiple dimen- sions (alignment, hallucination, exaggeration, and semantical diversity) using interpretable visual descriptors that could be used for descriptor-guider image refinement.
3. Analysis showing three key findings: ITTA metrics are ineffective, alignment alone is insufficient, and combining alignment, hallucination, and exaggeration necessary, achieving best correla- tion with human judgments.
4. We reveal consistent bias in T2I models towards GN countries.
5. Proposer-refiner enables robust, scalable reference descriptors without human annotations.
2 RELATED WORKS
Image-Text Alignment Metrics. General-purpose metrics rely on low-level features (e.g. FID
(Heusel al. 2017). LPIPS (Zhang al. 2018)) global image-text alignment (e.g. CLIPScore
Hessel et al. 2021.. VQAScore (Lin et al. 2024 Some metrics require expensive human judg-
ments (e.g. Reward (Xu et al. 2023). PickScore (Kirstain et al. 2023)). We show these
correlate poorly with human judgment.
Cultural Benchmarks. Cultural understanding has been extensively studied for image understand-
ing tasks (Kalluri et al., 2023 Ramaswamy et al. 2023 Nayak et al. 2024] Astruc et al., [002]]
Vayani al.) 2025 Liu et al.,
2025 Yin et al. (023). For T21 generation, existing benchmarks
are primarily object-centre 2025 Liu et al. 2024] Tha al. (0024). For instance, (Kannen al. 2024) covers countries across
(Kannen et al. [2024] Basu et al. 2023
Zhang et al. 2024 Rege et al.)
3 artifact categories, (Jha et al. 2024) includes 10 countries on food and architecture, and (Basu al. 2023) covers 27 countries using parsed noun phrases. CUL.TIVate differs by evaluating social activities, which are compositional and contextual, creating distinct evaluation challenges beyond object recognition (c.g., correct interaction, spatial arrangements). Concurrent work (Nayak et al.) 2025) studies cultural expectations through human evaluation. We complement this by focusing on activities and proposing the first automated metrics for cultural faithfulness specialized for social activities.
Cultural Representaaiivenss Metrics. Cultural representativeness is typically measured through diversity and faithfuluess. Diversity-based metrics (Rege et al.) 2025 Kannen et al., 2024 Basu let al. 2025; Zhang et al. 2024) quantify variation across generated outputs using approaches like continent-level diversity scores (Kannen al. 22024 Friedman Dieng 2023) or perceptual sim- ilarity between images (Rege et al. 2025). However, diversity alone is insufficient for measuring cultural representativeness (Rege et al. { (025), as vision encoders exhibit geographical bias and cap- ture low-level variations (color, texture) rather than cultural content. Importantly, diversity and faithfulness are distinct: our work focuses on faithfulness.
Cultural Faithfulness. Existing works (Nayak et al. [2225] Kannen et al. 2024 Jha et al. 2024
Liu al. 2024) rely on accurate but costly and unscalable human evaluation. Recently, some stud- KKhanuja et al. 2024 Basu et al., 2023] Rege et al. 2025) adopted VIMs-based image-text alignment (e.g. CLIPScore (Hessel et al. (021)) as a proxy for human judgment on cultural faith- fulness. Specifically, (Khanuja et al. 2024) measures alignment with simple country prompts, while (Rege et al., (2025) measures alignment between hierarchical prompts. However, we show that these metrics do not correlate well with human judgment (Tab. 2) These approaches rely VLM embed- dings to directly measure faithfulness. However, VLMs inherit similar cultural biases (Rege et al. 2025), struggle with compositional and implicit prompt understanding (Yuksekgonul et al. 2023
ies
3

---

Published as a conference paper at ICLR 2026
People playing Zu in Iran
Iron flag. Soccer ball, Grass field, Carrom board, Large flogs, Sested on floor, Circular arrangement
Pinzo
Algement
Exageration
EXAG: 0.4
HAL: 0.5
Drif
Persion courtyord, rugs, sticks, Gathering in a circle, Modern dethhnng
Persion courtyord Gathering in circle Modern clothing player togging
Halladasor
Diveritty
ALIGN: Gathering
circle grass field
HAL: Iran Rlg. Ball
Courtyard, One player tagging another, Group of people gethered, Wrestling trousers
EXAG: Musiim aftiiee
another
Alignment (ALIGN) DT
Hallucination (HAL)
Exaggeration (EXAG)
mpx manck
d, Piacer
Fin.2!
PL.
Rug
S1R
Iran Rlgg
Courtyord
Iron flag
Courtyord
Tilework
fullra)
Ball
Modern clothing
Ball
Modern diothing
Muslim
attire
0.02
Grass field
Player togging
Grass field
Player tagging
Circular
Gathering
Gircular orrangement
Gathering
arrangement
In circle
In circle
ALIGN= 2/4
HAL 2/4
EXAG 0.4
Figure 2: (Top) Overview. We extracted image descriptors d; E Dered with InternVL3 while ref- erence descriptors dj E Dref are obtained via a proposer-refime refiner pipeline in data annotation stage without using images. Proposers generate diverse candidates, and the Refiner removes duplicates and filters incorrect ones. AHEaD measures cultural competence through alignment, hallucination, exaggeration, and diversity, providing not only quantitative scores but also interpretable feedback (i.e., what aligned, missing, or exaggerated). (Bottom) Cultural Faithfulness metrics. Align- ment measures whether expected descriptors are present (similarity above threshold 7), hallucination flags elements unsupported by references (e.g., circular arrangement), and exaggeration detects ex- aggerated cues overemphasized with respect real- -mages (e.g., Muslim attire)
Li al. 2025
Malakouti et al. 2025 Aghazadeh & Kovashka 2025), making ITA unreliable for cultural evaluation. We introduce AHEaD, suite of automatic metrics that leverages external visual descriptors to measure cultural alignment while penalizing hallucinations and over-exaggeration.
Knowledge probed from large language models. While LLM-based visual descriptors have been explored for fine-gres gne-ggrined and cross- coos-gcogrrahy object recognition (Pratt et al. 2023 Menon & Von- drick, 2023 Saha et al.] 2024 Buettner al. 2024), this is the first work use descriptors for evaluating cultural competence in T2I models.
METHODOLOGY
Evaluating cultural faithfulness requires more than image-text alignment. A reliable metric must reward correct cultural elements (e.g., interactions, objects, attire) while penalizing hallucinated or exaggerated ones. However, Fig. Salshows existing VLM-based metrics do not correlate negatively with hallucination and exaggeration. This limitation motivates structured framework utilizing external cultural concepts to evaluate images.
We introduce AHEaD, descriptor-based framework designed for culturally faithful image genera- tion. First, each activity is annotated automatically with comprehensive set of reference deseriptors (Drof). Then, predicted descriptors (DPred) are extracted from images. Fi- nally, AHEaD metrics are computed by comparing predicted and reference descriptors, yielding descriptor-based feedback as illustrated in Fig. Sec. 3.1 describbe reference descriptor generation and Sec. 3.2 defines AHEaD metrics.
3.1 REFERENCE DESCRIPTOR GENERATION
We represent each activity-country pair as a set of cultural descriptors, Dref, encoding expected visual elements across five dimensions: background (c.g., Eiffel tower, geometric patterns), attire (c.g.. traditional vs. modem clothing), objects, actions/interactionss (c.g., greeting with a bow), and spatial layout (e.g., dancers in circle). These descriptors are I..M-generated establish ref-
4

---

Published as a conference paper at ICLR 2026
erence for evaluation independent of images. Concretely, construct descriptors using Proposer- Refiner, a two-stage method inspired by self-consistency prompting (Wang et al.] 2023). First, the Proposer leverages multiple LLMs to independently generate up to 10 mutually exclusive descrip- tors per dimension. Using multiple L.MSs increases coverage while mitigating model-specific bias while capturing diverse cultural variants. Second, the Refiner filters these candidates remove du- plicates and errors, enhancing precision (example in Fig. 2 Tab, 4 significantly improves descriptor quality over single-stage generation.
the Proposer-Re framework
3.2 AHEAD EVALUATION METRICS
We design descriptor-based metries along two complementary axes, faithfulness and diversity, A faithful T2I model should (i) cover the expected elements (Alignment), (ii) avoid irrelevant elements the activity or culture (Hallucination), (iii) refrain from wver-exaggerating cultural elements (Ex- aggeration). We also measure variety across generations (Diversity).
For each activity a and region r (country), we generate images {4n ) using the prompt T, Rather than using an MILLM to score images directly, we employ it to parse visual content into fine- grained descriptors Drred(I,) These aggregated into set Drred = U DDred across the same five cultural dimensions as Dreff r.0
To build AHEaD metrics for žr,a = (f4m)==1 Dpred Dre/), we construct a complete bipartite graph between predicted and reference descriptors. In this graph, predicted descriptors d; DDpred and reference descriptors d, E Dreff serve as nodes. Each edge encodes the semantic similarity Stj sim(d,, dj) between descriptors, as measured by sentence embeddings.
Alignment. Alignment measures coverage of expected cultural elements. For each reference de- scriptor dj. identify its best match predicted descriptor via maximum edge similarity. Alignment is the fraction of reference descriptors whose best match exceeds threshold
ALIGN(I,.a)
Σ1 max deED!!!
1[max 8i,j maxx815
1
r]
(1)
where [[:] is the indicator function and 7 calibrated according to real images (Sec. A.3) The final score is the average of scores across five cultural dimensions (e.g.. spatial, attire, objects, etc).
Hallucination. While Alignment measures coverage, it does not capture extrancous or culturally incorrect elements. We address this via HAL, defined as the fraction of predicted descriptors d, that lack corresponding match in the reference set:
HAL(I,,a)
Σ d.DEEed
max Si &
iDpred
In practice, we measure this for each cultural dimension separately and report the average.
Exaggeration. A faithful generation must also avoid over-emphasizing stereotypical elements. We measure exaggeration by comparing the intensity of stereotypical elements in generated images against real images scraped from web (details in Sec. 4.1). First, inspired by D'Incà al.) (2024), we use an LLM to propose set of stereotypical candidates S, for region r. Let J'[, de denote the ITA score between image and candidate descriptor dy S,. We define the baseline reference score as the average ITA score over Ngt real images:
ngt
[d((d)) = Tugt m=l
Σ f( f((m,de) dx)
(3)
For an instance 2r,a, the exaggeration score is the average maximum positive deviation from this baseline across N generations:
5

---

Published as a conference paper at ICLR 2026
EXAG 1
max de ES,
max (0, dx) - (d)())))
(4)
Faithfulness. Cultural laithfulness is defined composite score:
(T, = -9 (AALIGN, 1 HAL, 1 1
(5)
where g(-) is the arithmetic mean. Additionally, AHEaD provides interpretable descriptor-base based feedback identifying aligned, missing, and exaggerated elements which can enable descriptor- guided image editing (see Fig. 8 for a preliminary result).
Descriptor Diversity, Beyond faithfulness, we measure the variety of cultural elements produced across N generations for 2,,0 using normalized entropy:
DDIV(Ir,a)
Σ
q(d) log (())
(6)
log |D
dED a(d)>0
(d) is normalized frequency of reference descriptor d appearing across N images Ea ((d) = 1).
Semantic Diversity. We define semantic diversity as the marginal utility in descriptor coverage provided by N images over single generation:
SDIV(ar,a) ALIGNN (Ir,a) E (Ir,a)l
where ALIGNN is alignment over N images and [[ALIGN]] is average single-image alignment.
4
EXPERIMENTAL SETUP
4.1 CULTIVATE BENCHMARK
Constructing cross-cultural benchmarks with local activities is challenging as it requires expert re- gional knowledge. To obtain this knowledge systematically, parse existing knowledge bases, CulturalAtlag and Wikipedia, with GPT-4o to extract non-overlapping activities per country. These sources are complementary with CulturalAtlas documenting cultural practices (e.g., greetings, reli- gious customs, etiquette), and Wikipedia providing activity lists (e.g.. games, celebrations). CUL- TIVate spans 16 countries with 576 activities across categories, generating 19,000+ images from 6 T2I models with comprehensive ground-truth descriptor annotations.
We consider activity categories falling into three types: (1) multi-variant (dances, games, religious practices, greetings, celebrations) with multiple activities per country (e.g., dif- ferent traditional dances), (2) setting-based (cating, concerts) with activities varying by context (home/restaurant, indoor/outdoor), and (3) single-variant (weddings, funerals) with one activity per country.
Activities.
Countries. We select 16 countries spanning all socio-cultural regions in CulturalAtlas. Following UN classificationf" countries are divided into Global North (USA, Spain, Italy, Germany, France) and Global South (Iran, Turkey, China, India, Indonesia, Philippines, Nepal, Nigeria, South Africa, Brazil, Mexico).
Image Generation. For each prompt, generate images using the template "A photorealistic photo of (activity] } in {country)." We evaluate 6 recent T2I models: public (Stable Diffusion 3.5 (Esser et al., 2024). FLUX BBlackForestl.abs 2024 Qwen-Image (Wu al. 2025 and proprietary (DALLE3 (Betker et al. (023), GPT--mage-1 (OpenAl (002), Gemini 2.5 Flash Image (i.e. Nano Banana) (Google (0055). For public models, we generate 10 images per prompt with random seeds 42 + for i-th image. For proprictary models, we generate I image due to the cost.
https://eutuurrlaaasssst www.:sssssss.....e
Hi Meerarchy-pd
We used the distilled model: hhtp://iiiicccomm/ddddr//oooo/nn

---

Published as a conference paper at ICLR 2026
Reference data. We adopt two complementary strategies for identifying what images of activities in a region (country) should portray: (1) Visual Descriptors: We extend prior usage of LLMs for object descriptors (Menon & Vondrick] 2023) to activities. For each prompt, we generate up to 10 descriptors per 5 dimensions, capturing diverse valid variants of each activity (details in Sec. (31); (2) Real Images: We collect 20 candidate images per prompt via Google search (10 using the English prompt, 10 using its translation into the language of the respective country), totaling ~ 112 images. We then apply CLIPScore (Hessel et al., 2021) filtering and retain the top five (total of ~3k) as representative real references which we use in our EXAGgeration metric. Real images also serve for calibration and hyperparameter tuning (e.g.. T in Eq. I..
4.2 HUMAN EVALUATION SETUP
We conduct human study to validate our metrics for Faithfulness (i.e., Alignment, Hallucination, and Exaggeration) and realism (whether image looks realistic). We adopt Prolifid as our platform.
Study Design. Our evaluation spans 11 countries across 5 CulturalAtlas regions (Middle East, America, Europe, Africa, Asia) and both Global North and South. We selected 1-2 activities per category, ensuring coverage of all activity groups per country. We assessed three public T2I models and real images (for 3 countries), totaling 398 forms and 796 annotations (two annotators per form).
Annotations. We collected 3 ground-truth (GT) labels using 5-point Likert scale: GT-FAITH (main gold standard) measures overall faithfulness according to annotators How well does this im- age show {activity) in your country?: GT-EXAG measures image exaggeration- exaggerated is the image? GT-HAL measures hallucinations--Hle incorrect is the image? (activity/culture)
We also evaluate reference descriptor quality through human evaluation. We measure precision by having annotators mark each descriptor as correct or incorrect; 90% of descriptors were marked correct (Tab. IT]. It is infeasible to compute recall because no ground-truth descriptor Instead, we estimate recall through two measures: (1) average coverage rating of 4.5/5, and (2) only 26 of 378 annotators reported missing descriptors. These results demonstrate high precision and comprehensive coverage. We utilize two complementary measures: (1) average descriptor quality rating (result: 4.5/5), and (2) proportion of annotators reporting at least one missing descriptor (result: 26/378). These results demonstrate high precision and comprehensive coverage.
exists.
Quality Control. We recruit annotators matching each country's nationality (verified by Prolific) at $8/hour compensation. To ensure reliability, we implement multiple quality control measures, such as attention checks (e.g., selecting a pre-mentioned number), repeated questions to test consis- tency, and required free-text rationales describing the errors in the image. We also conducted direct discussions with annotators when facing inconsistent scores and explanations.
Correlation Metric & Inter-rater Agreement. We use Spearman's 's rank correlation measure how well our proposed metries align with human judgments. Spearman's 's evaluates the strength of monotonic relationships between ranked variables, where values near 1/-1 indicates strong positive/negative correlation. Following prior work (Kannen et al.] (024)). we measure inter-rater agreement using Krippendorff's Alpha (Krippendorfi [2018). which is well-suited for ordinal Lik- ert scales. We compute agreement separately for each country. Agreement is moderate and varies by country, consistent with prior cross- custurall studies (Nayak et al. 2025 Kannen et al.] 2024). re- flecting cultural evaluation's 's inherent subjectivity. Our agreement levels are comparable to or exceed their maximum per-country values (Appendix A.4).
4.3 IMPLEMENTATION DETAILS AND BASELINES
Implementation Details. For Proposer-Refiner, we use Gemini 2.5 Flash (Comanici et al., 2025)
and GPT-4o (Hurst et al. 2024). GPT-4o is used as the refiner due to its sota cultural understand-
ing CChu al. 2025a). We use Intern VL3 14B (Zhu et al.] 2025) and ---2255-VL---BB-- let al. 2025) as MLLMs in AHEaD, and compute sentence embeddings with M11-MiniLMM-L666 calibrated real images: 0.52 (interVLL.), 0.67 (Qwen2.5-VI.), Details in Sec. A.4
(Bai
Baselines. We compare AHEaD against ITA metrics, such as CLIPScore (Hessel et al.
2021
ImageReward (Xu et al. 2023). VIEScore (Ku et al. 2024). VQAScore (Lin et al.
2024
hhaps://www.pnolffcccommm

---

Published as a conference paper at ICLR 2026
N=1
N=10
Model
Region
ALIGN
HAL
EXAG
FAITH
DDIV f
SDIV
GN
0.31 10001
0.55
1000
0.05 10.02
0.57 d
0.68 1000
0.33
SD-3.5-medium
GS
0.26
0.61
0.08 10004
0.52 +
0.62 4004
0.32
1-
GN
0.30 10002
0.56
1:,03
0.04 10001
0.57 =
0.66 10003
0.32
GS
0.25
0.63
0.06
0.52
0.60 1:00
0.30
GN
0.36 11000
0.51
1:002
0.06 ::000
0.60 =
0.68 1:002
0.28
Qwen-Image
GS
0.30 2000
0.56
1000
0.10 20003
0.55 =
0.63 2004
0.29
GN
0.36 #-0.01
0.50
+.0001
0.10
+-003
0.59 +-
3
GS
0.32 0:000
0.54
1:0002
0.12
40.04
0,55
GPT-Image-
GN
0.36 10.01
0.49
1:000
0.06
10.01
0.61 1
0.30 #:000
0.55
+:003
0.07
#0.02
0,56 +
Gemini 2.5 Flash Image
0.40 10001
0.46
1:000
0.10 =:003
0.61 =
GS
0.35 1:003
0.50 10000
0.12 1:0.3
0.57 +-
Table 1: T2I models consistently generate more faithful images on GN countries. N is number of images per prompt. Best values per model (GS/GN) bolded. EXAG values are small because it measures relative alignment of synthetic image to real images.
Best-3 Worst-3
ALAON
ISAL
RAI
CENANY
10
1MATE APRCC
celebration
concert
dancing
eating
funeral
game
greeting
religious
wedding
(a) Models consistently score better (higher on ALIGNment, lower on HALlucation) for GN. Interestingly, they perform strongly on China
(h) Frequency of activities appearing among the best-3 (green) worst-3 (red) across countrics.
Figure 3: Analysis of performance by country (left) and activity (right).
PickScore (Kirstain et al. 2023) and existing cultural metrics CURE (Rege al. 2025). For fair comparison, we include MLLLM-a--Judge baselines with the same backbone. Additional details in Sec. A.4
5 RESULTS
5.1 How DO DIFFERENT T21 MODELS PERFORM FOR DIFFERENT COUNTRIES?
T2I models consistently generate more faithful images for Global North countries. Tab. shows consistent bias against GS, with all models performing better on GN. For example, Qwen-Image achieves higher FAITH (0.60 vs 0.55), higher ALIGN (0.36 vs 0.30), and lower HAL (0.51 vs 0.56) on GN compared to GS. This pattern holds across models where lower ALIGN, higher HAL/EXAG, and lower DDIV/SDIV on GS indicate models make more errors, generate more exaggerated con- tent, and exhibit less diversity for Global South countries. Fig. 3a shows per-country trends.
T2I systems struggle most with culturally grounded activities. Fig. 3b shows the frequency of each activity appearing in the top-3 and bottom-3 performing tiers across 16 countries. Performance highest for universal activities (e.g., concerts, cating) and lowest for culturally grounded ones (e.g., celebrations), indicating gap in modeling specific cultural contexts.
8

---

Published as a conference paper at ICLR 2026
Backbone
Method
GT-FAITH
GS
GN
All
PickScore Kirstain et al.
2023
0.20
-0.02
0.15
ImageReward (Xu 2023
-0.03
-0.13
-0.08
CLIPScore (Hessel et al
2021
I-T Alignment
0.08
-0.01
0.04
VQAScore [iin et al.
2024)
0.15
0.16
0.14
CuRe (Rege al.
2025)
0.13
0.08
0.10
MILM
0.13
0.08
0.10
Qwen2.5 VL
Bai
2025)
FAITH (Ours)
0.42 (40.29)
0.38 (-0.30)
0.42 (40.32)
MILM
0.19
0.18
0.20
Intern VL3 Zhu el al.
2025
FAITH (Ours)
0.46 (+0.77)
0.47 (-0.29)
0.47 (40.27)
MLLM
0.49
0.46
0.48
GPT-4o
VIEScore Ku et al. 2024
0.37
0.27
0.35
Human
0.59
0.57
0.58
Table 2: FAITH achieves significantly higher correlation with human judgment on faithfulness compared to ITA and MLLM-as-a-Judde baselines. FAITH shows consistent performance across backbones and achieves comparable results to GPT-4o despite using much weaker backbone. Best values per each section are bolded. Values parentheses show improvement over MLLM bascline with the same backbone. Hauman-human correlation is provided for reference.
MILM Baseline
ALIGN
ALIGN+HAL
FAITH
Backbone
GS
GN
All
GS
GN
All
GS
GN
All
GS
GN
All
Qwen2.5-VE L-VL (Bai 2025)
0.13
0.08
0.10
0.41
0.32
0.39
0.37
0.37
0.39
0.42
0.38
0.42
Intern VL.3 7hu et al. (2025
0.19
0.18 0.20
0.40
0.40
0.41
0.42
0.46
0.44
0.46
0.47
0.47
Table 3: Necessity of composite metries for faithfulness. Spearman correlation with GT-FAITH across regions. Our composite metric (FAITH) significantly outperforms individual components, confirming that aligmment alone is insufficient and is complemented by HAL and EXAG. Correla-
tions are statistically significant (p < 0.0001)
5.2 WHAT METRICS ARE EFFECTIVE FOR CULTURAL FAITHFULLNESS?
Image-Tert Alignment metrics are ineffective for cultural understanding. Tab. 2 shows ITAs' correlation with humans scores are below 0.15 (e.g.. ImageReward: -0.03/-0.13/-00 for GS/GN/alI). MLLM-as-judge baselines, which prompt MLLMs with the same questions given to annotators, correlate better but still substantially lag behind FAITH (e.g., Qwen2.5-VL: 0.10 vs FAITH: 0.42 and Intern 0.20 vs FAITH: 0.47 on all).
EXAGgeration and HALlucination complement ALIGNment. Tab. demonstrates that com- bining ALIGN and HAL improves correlation with human faithfulness and best results is achieved when all three metries combined (i.e. FAITH). These results indicate that effective cultural faith- fulness metrics must penalize exaggeration and hallucination rather than rely on alignment alone.
Ablations. Tab. lab.4eo compares proposer-refine refiner against proposer-only for descriptor generation, show- ing the two-stage approach improves correlation with human faithfulness judgments. Tab. [5] ab- lates the threshold parameter т (Section 3.2) used to determine whether reference and predicted descriptors match. We test values the 25th, 50th, and 75th percentiles, finding the 75th percentile performs best.
5.3 WHAT ASPECTS OF THE ACTIVITIES ARE DEPICTED BEST/WORST BY T2I MODELS?
Fig. 4 shows performance by region (country) across each five descriptor dimensions (see Sec. The pest-performing country varies by dimension, but the USA, China, and Germany consistently rank among the best. South Africa, Nigeria, and India generally fall in the lower half, except for Interaction (South Africa and Nigeria) and Spatial (India).

---

Published as a conference paper at ICLR 2026
Ref. Generator
(Propuser/ Refiner)
Spearman
Kendall
Spearman
Kendall
Proposer
GPT-4n/-
0.28
0.20
0.29
0.21
0.15
Proposer
Gemini 2.5-Flash
0.30
0.22
0.39
0.27
0.20
Proposer- Refiner
GPT-4o Gemini 2.5-Flash 4PT-44
0.33
0.24
0.52
0.33
0.24
Table Proposer-Refiner improves descriptor quality.
Table 5: Thresh. (r) ablation.
auizal
(a) Objects
(b) Background
(c) Spatial
wwm orec
dit silii
(d) Interaction
(c) Attire
Figure 4: Country alignment ranked using each of the five descriptor dimensions.
0.5
OT
0.0
6.00
ALKA
-025
CHAmT
-05
-0.00
R6TH
0075
GRENAC
Grad
Aabiccm insplfenerd
HALL
ALOO
aLooy Hil Auco
EXA
HMT
1542
(a) Correlation between GT scores (left), ITA methods (middle), and our scores (right).
(b) Correlations among our proposed metrics.
Figure 5: Effective faithfulness metrics must negatively correlate with exageration/halllciaaiinn
5.4 How DO THE METRICS RELATE TO EACH OTHER?
To improve the performance of T2I models, metric will affect others. We aim to answer this question by computing correlations between the metrics, shown in Fig. 5b) We that alignment is negatively correlated with both exaggeration and hallucination. The same trend is observed using human scores; see Fig, [a which also demonstrates visually the much stronger aligmment of our metrics with human scores.
might want to know how improving upon
CONCLUSION
We developed a framework for evaluating generation of images of social activities in different coun- tries. We propose a suite of metrics that can be computed without human involvement, yet show much higher agreement with human assessment than prior metrics. Using our framework, we con- duct analysis on sixteen countries and six ixt-to--mmae models. We show performance on Global North countries exceeds that of Global South. and demonstrate specific failure modes using de- scriptor dimensions. We hope our work equips future researchers with the tools to scalably improve and test performance this task which has broad applicability, e.g., in the entertainment industry.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENT
This work is supported by NSF Grant No. 2329992. We gratefully acknowledge the support of those who contributed the human evaluation. We also thank Aysan Aghazadeh and Christopher Achkar for their valuable comments and help throughout the project.
REFERENCES
Aysan Aghazadeh and Adriana Kovashka. Cap: Evaluation of persuasive and creative image gen- eration. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 16970-16980, October 2025.
Guillaume Astruc, Nicolas Dufour, Ioannis Siglidis, Constantin Aronssohn, Nacim Bouia, Stephanic Fu, Romain Loiseau, Van Nguyen Nguyen, Charles Raude, Elliot Vincent, Lintao Xu, Hongyu Zhou, and Loic Landrieu. Openstreetview-Smm: The many roads to global visual geolocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21967-21977, June 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Abhipsa Basu, R Venkatesh Babu, and Danish Pruthi. Inspecting the geographical rerresttisss of images from text-to-imetn -o--g models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Abhipsa Basu, Mohana Singh, and Venkatesh Babu Radhakrishnan. Geodiv: Measuring concept di- versity of images across geographical regions. In CVPR 2025 Workshop Vision Language Models For All, 2025.
Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad, Eunsu Kim, Junyeong Park, James Thorne, and Alice Oh. Diffusion models through a global lens: Are they culturally inclu- sive? In Proceedings the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31137-31155, 2025.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dalle-3. pdf. 2(3):8, 2023.
BlackForestLabs. Flux: A powerful tool for text generation.
ittp:://backfooesttt
ai/ 2024.
Kyle Buettner, Sina Malakouti, Xiang Lorraine Li, and Adriana Kovashka. Incorporating geo- diverse knowledge into prompting for increased geographical robustness object recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13515-13524, June 2024.
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. Cultural- Bench: robust, diverse and challenging benchmark for measuring LMs' cultural knowledge through human-Al Al red-tcaming. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 25663-2551 Vienna, Aus- tria, July 2025a. Association for Computational Linguistics. ISBN 1999-889116-25--0 doi: 10.18653/v wawp:/5..clloo.11220 1247.
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. Culturalbench: a robust, diverse and challenging benchmark on measuring (the lack of) cultural knowledge of LLMss, 2025b.
11

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Moreno D'Incà Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang. Humphrey Shi, and Nicu Sebe. Openbias: Open-set bias detection in text- to-image generative models. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12225-12235, June 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Clifford Geertz. The interpretation of cultures. Basic books, 2017.
Google DeepMind Google. Nano banana (gemini 2.5 flash image), 2025. URL https: !/ai .w..goooleee dev.dde//ee wwwe/eeeiiineeeee gemi nimiinia wataaaaa/occs/mmaaaaae tion model.
Image editing genera-
Edward T Hall. The silent language. Anchor, 1973.
Joseph Henrich, Steven Heine, and Ara Norenzayan. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61-83, 2010.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-fiee free evaluation metric for image captioning. Proceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514-7528, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan Reddy, and Sunipa Dev. Visage: A global-scale analysis of visual stereotypes in text- to-image generation. In Proceedings of the 62nd Anmual Meeting of the Association for Compu- tational Linguisties (Volume 1: Long Papers), pp. 12333-12347, 2024.
Tarun Kalluri, Wangdong Xu, and Manmohan Chandraker. Geonet: Benchmarking unsupervised adaptation across geographies. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15368-15379, 2023.
Nithish Kannen, Arif Ahmad, Marco Andreetto, Vinodkumar Prabhakaran, Utsav Prabhu, Adji Bousso Dieng, Pushpak Bhattacharyya, a, and Shachi Dave. Beyond aesthetics: Cultural com- petence in text-to-image models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 13716-137444 Curran Associates, Inc., 2024. doi: 10.52202/079017-04999
Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, and Graham Neubig. An im- age speaks thousand words, but can everyone listen? on image transcreation for cultural relevance. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 10258- 10279, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 1/24
12

---

Published as a conference paper at ICLR 2026
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-inng -onttggg generation. Advances in neural information processing systems, 36:36662-3666666 2023.
Klaus Krippendorff. Content amalysis: An introduction to its methodology. Sage publications, 2018.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable
metrics for conditional image synthesis evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12268-12290, 2024.
Tianle Li, Jihai Zhang, Yongming Rao, and Yu Cheng. Unveiling the compositional ability gap in vision-language reasoning model. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference Computer Vision, pp. 366-384. Springer, 2024.
Shudong Liu, Yiqiao Jin, Cheng Li, Derek F Wong. Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. Culturevlm: Characterizing and improving cultural understanding of vision-language models for over 100 countries. arXiv preprint arXiv:2501.0122 2025.
Zhixuan Liu, Peter Schaldenbrand, Beverley-Claire Okogwu, Wenxuan Peng, Youngsik Yun, An- drew Hundr, Jihie Kim, and Jean Oh. Scoft: Self-contrastive fine-tuning for equitable image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10822-10832, 2024.
Sina Malakouti, Aysan Aghazadeh, Ashmit Khandelwal, and Adriana Kovashka. Benchmarking vlms' reasoning about persuasive atypical images. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4788-4798. IEEE, 2025.
Mary L McHugh, Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-28) 2012.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hen- dricks, Karolina Stanczak, and Aishwarya Agrawal. Benchmarking vision language models for cultural understanding. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 5769-5790, 2024.
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd Van Steenkiste, Yash Goyal, Karolina Stanezak, and Aishwarya Agrawal. Culturalframes: Assess- ing cultural expectation alignment text-to-image models and evaluation metrics. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20918-20955 2025.
OpenAl. Gpt image 1, 2025. URL httts::/plat openai ccom/oocs/modee.e.c http://foom..paaaaaccoooo//////o
gpt-images 1mage-1 Model documentation.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 15691-15701, October 2023.
Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems, 36:66127-66137 2023.
Aniket Rege, Zinnia Nie, Mahesh Ramesh, Unmesh Raskar, Zhuoran Yu, Aditya Kusupati, Yong Jae Lee, and Ramya Korlakai Vinayak. Cure: Cultural gaps in the long tail of text-to-image sys- tems. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15680- 15691, 2025.
13

---

Published as a conference paper at ICLR 2026
Oindrila Saha, Grant Van Horn, and Subhransu Maji. Improved shot classification by adapting vims with text descriptions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17542-17552, 2024.
SeattleTimes. French find "ratatouille" ever palatable. https::/www.seattlles. wor
com/ wttpo/owood/fffeeeeeeoo fewnchrenniindn find ratatouille -eeeleeeeeeeeeeeeeee -ver--s-palatabllee
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 2025.
Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kukreja, et al. All languages matter: Evaluating Imms on culturally diverse 100 languages. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19565-19575, 2025.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Wikipedia. List accolades received by ratatouille.
stttpps//e..wkkieddiaaoooo List of accccccooloo lwwelaaeelss. awaadeereeeeeeeeeeeeeeeeeeeeeeee by_Ratctouillle
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.0233
2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 19903-159595,, 2023.
Da Yin, Feng Gao, Govind Thattai, Michael Johnston, and Kai-Wei Chang. Givl: Improving geo- graphical inclusivity of vision-language models with pre-training methods. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 10951-10961, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-wordd and what to do about it? The Eleventh International Conference Learning Representations, 2023.
Lili Zhang, Xi Liao, Zaijia Yang, Baihang Gao, Chunjie Wang, Qiuling Yang. and Deshun Li. Partiality and misconception: Investigating cultural representativeness in text-to-image models. In Proceedings of the 2024 CHI Conference Human Factors in Computing Systems, pp. 1 1-25, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metrie. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye. Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I USAGE OF AI
In this section we elaborate on LLM usage in this study. LLMs were used throughout this research as writing assistants, for text polishing, and for literature review through LLM agents and available tools. AI coding assistant were used to assist with programming. However, LLMs were not used blindly and served only as assistants to improve accuracy and efficiency. This paper introduces a benchmark social activities. As described in the main paper, LLMs (GPT-4o) were utilized to parse online knowledge bases (CulturalAtlas and Wikipedia) to identify activities across countries. Furthermore, the descriptor-based metries rely I.L.- descriptors. However, proposer- refiner approach was incorporated to improve quality, and descriptors were evaluated through human evaluation (see Table II
A.2 LIMITATIONS
Cultural Bias in LLMs. AHEaD uses LLM-generated descriptors reference points for measur- ing the cultural competence of T2I models. Since LLMs are trained on web text, we acknowl- edge that they may encode biases toward Western societies. To mitigate this, we adopt Pro- Roser-Refiner strategy. which improves descriptor quality and increases agreement with human ground-truth scores. Human evaluation showed 90%. Compared common alternatives, such as human surveys or real images, our approach is scalable and less costly. Real images collected from the web are themselves biased, while surveys are subjective and expensive. Unlike VI.M-based image-text alignment methods or raw image references, our descriptors are explainable and allow direct inspection of model errors, rather than being opaque scores.
A.3 CALIBRATION OF THRESHOLDD
We propose ALIGN and HAL to measure how well images cover expected activity/cultural cues and which visual elements are incorrect. Since these metrics are ratio-based, we must set similarity threshold 7 to decide whether a descriptor counts as a hit (aligned) or miss (ballucinated).
We calibrate using real reference images rather than synthetic generations to avoid lcakage, since synthetic data may reflect biases of the very T2I models under evaluation. Real images, while noisy, contain culturally faithful content without "wrong" "exaggerated" elements, making them suitable for calibration. Concretely, we compute descriptor-deseript similarities between L.M LL.M-provided ground-truth descriptors and MLLM-extracted descriptors from real images, then consider candidate thresholds at the lower quartile (QI), median, and upper quartile (Q3). As shown in Fig Q3 offers the best trade-off by reducing false positives while maintaining recall. Table3 further confirms that Q3 yields the most robust alignment scores across regions.
Similaity Score Distibutio bunibbtion
Medium
C01 2.29
Moan
Mour 018 0.25
300
5in Seibarty Scare
Figure 6: Threshold 7 calibration for ALIGN
hhups///curor.cco//
15

---

Published as a conference paper at ICLR 2026
A.4 IMPLEMENTATION DETAILS AND BASELINES
Evaluation baselines. The goal of this paper evaluate the cultural faithfulness competence (GT- ALIGN) of T2I models, where automated evaluation methods remain extremely limited. Existing
works rely heavily on human annotations (Kannen et al. 2024, Nayak et al. 2025 Basu et al.
2023), while few recent approaches (Khanuja et al.) 2024 Rege et al. (025) approximate cultural faithfulness using image text similarity. Accordingly, we compare against commonly used and state-of-the-art ITA metrics, including CLIPScore (Hessel et al., 2021). VQAScore (Lin et al. 2024) with CCLIP-FlanTS-xx (the strongest publicly available ITA setup), PickScore (Kirstain et al, 2023). and IeageReward (Xu al.) 2023). Following prior ITA practice, use each model's generation prompt- "A photorealistic image of activity in country"-& -as the reference for evaluation. We also benchmark against CuRe (Rege al., 2025), the only metric explicitly designed for cultural faithfulness. For fair comparison, we adopt CuRe's recommended SigL.P2 (Tschannenn et al. (2025) configuration and compute mean image- text similarity using the prompts "An image of activity" and "An image from country," omitting their parent-category prompt since this information is already embedded in our activity descriptions (e.g.. "people playing tag game").
Across all settings, we find that ITA methods and CuRe exhibit weak correlation with human cultural judgments, whereas our proposed metries achieve substantially higher and more stable agreement across different MLLM backbones (Intern and QwenVL2.5). We attribute the limitations of existing VI.M-based ITA methods to: (1) bag-of-words behavior that misses compositional cul- tural nuance (Yuksekgonul et al. 2023). (2) reliance on Western-centric training data that introduces cultural biases, and (3) inability to distinguish authentic cultural representation from stereotypical exaggeration. For instance, CLIPScore rewards images containing literal elephants for the "ele- phant ant man" " game-an an Indonesian rock rock-paper-sciiee paper-scissoors variant- due to keyword matching rather than cultural understanding. To address these issues, AHEAD uses externally generated cultural de- scriptors instead of VLM embeddings, enabling interpretable evaluation of ALIGN, HAL, and EXAG that aligns more faithfully with human cultural judgment. This is the first work to evaluate cultural HAI and EXAG, and we study both descriptor-deseripp baselines using VL3 and Qwen VL2.5, which answer the same cultural assessment questions posed to human annotators (full prompts in Appendix A.
methods (Sec. (3.2) and MILM-as M---Maase
Implementation Details We first use GPT4-0 and Gemini 2.5 Flash (best LLMs in cultural under- standing (Chiu et al.] 20255b) offline once to in the data annotation phase to produce "reference LLM descriptors", these are used as noisy reference evaluate cultural faithfulness. To minimize the LLM-bias we developed proposer-refiner to combine descriptors of different LLMS which is refined by removing duplicate and incorrect descriptors (results in Table 4 We set the tempera- ture to 0.2 for proposers and 0.1 for the refiner. AHEaD uses an MILM to extract descriptors, we mainly use In3ernVL3 ("InterVV33-14B") as MLLM in our pipeline and also test our pipeline with Qwen QwenVL2.5 LL..5 (02255(00WweL2225-775 reproducibility, and use all-Mi ni M--MiiiMMM6-- as the sentence embedding model for similarity computation. All experiments run single L40S GPU.
We set temperature 0 for MLLMs to ensure high precision and
Inter-Rater Agreement. We consider "GT-ALIGN" for inter-rater agreement as the main goal of this work is to measure cultural faithfulness GTEEXAG/GT-HAL are more subjective. To assess the reliability of our human annotations, we compute country-level agreement scores for the cultural relevance ratings. Each image is annotated by two independent raters who are originally from the corresponding country. Across the eleven countries in our study, Krippendorff's s Alpha (Krippen- dorfi 2018) ranges from 0.15 to 0.62. We also compute Cohen's Kappa (McHugh 2012) between the two annotator groups and observe a mean value of 0.50. These agreement levels are consis- tent with previously reported values for cross-cultural image evaluation. CulturalFrames (Nayak et al. 2025) reports country-level Alpha values between 0.24 and 0.42, and CUBE (Kannen et al. 2024 reports values between 0.09 and 0.58. Our scores are therefore comparable to prior work and also achieve a higher maximum value, which indicates that our annotation protocol yields reliable judgments.
We observe variation across countries, with standard deviation of 0.13 for Krippendorff's Al- pha. Such variation is expected because cultural faithfulness assessments are subjective and depend strongly cultural and geographic context. Interestingly, the average agreement among Global North countries is 0.28, which is lower than the Global South average of 0.35, even though text-to- image models tend to perform better Global North regions. We hypothesize that higher-quality
16

---

Published as a conference paper at ICLR 2026
Method
Model
Flux1
Qwen-Image
SD3.5
Avg.
VQAScore
0.14
-0.02
0.14
0.09
PickScore
0.06
-0.05
0.03
0.01
Image-Text Alignment
ImageReward
-0.09
-0.24
-0.19
-0.17
CLIPScore
0.04
-0.28
0.02
-0.05
CuRe
0.17
0.10
-0.01
0,09
GPT-4o
0.53
0.27
0.48
0.43
MLLM
Inter
0.38
-0.14
0.14
0.13
QwenVL24
0.12
0.02
0.11
0.09
ALIGN
Ln3
0.51
0.30
0.31
0.38
Human
0.65
0.40
0.55
0.55
Table 6: Per-model correlation with GT-FAITH. ALIGN with Intern VL3 backbone significantly outperforms all ITA metrics and VL VL3--s-judge baseline across '21 models, achieving perfor- mance comparable to GPT-4o-as-judge.
GT-HAL
GT-FAITH
Method
Backbone
GS
GN
overall
GS
GN
overall
Intern
0.22
0.24
0.23
+0.20
-0.24
-0.21
MLLM
Qwen VL2.5
0.29
0.30
0.29
-0.31
-0.36
-0.33
Intern
0.31
0.39
0.35
-0.39
-0.44
-0.41
HAL
VT.n55.
0.30
0.42
0.36
-0.33
40.35
-0.36
Human
0.40
0.38
0.39
Table Correlation with humans on Hallucination. Our Hallucination metric achieves the high- correlation with human ground truth scores compared to existing MLLM-based approaches, including InternVL. which serves as the backbone for MLLM descriptor extraction. Best scores per column are bolded.
outputs may cause annotators to focus more on aspects unrelated to cultural content, such as image quality visual artifacts, to rely more heavily on subjective interpretations.
A.5 ADDITIONAL RESULTS ON AHEAD
Per-T2I Alignment Performance. Table 6 compares correlation with human faithfulness judg- ments across T21 models. ITA metrics perform poorly, with most showing near-zero or negative cor- relation (e.g., CLIPScore averages -0.05, ImageReward -0.17). MLLM-as-judge baselines achieve higher correlation, with GPT-4o reaching 0.43 average. Our ALIGN metric with Intern VL3 back- bone achieves 0.38 average correlation, sigmificantly outperforming all ITA methods and InternV33 as-judge (0.13), while approaching GPT-4o performance despite using a weaker model. Human- human agreement moderate (0.55).
HAL can effectively detect hallucinations. Table 7 shows our HAL metric achieves highest cor- relation with GT-HAL and lowest with GT-FAITH. outperforming MILM baselines. For example, although we use Intern to extract descriptors, our HAL outperforms Intern VL.3-as-jugee by 11% on GT-FAITH correlation. Notably, HAL exhibits strong negative correlation with GT-FAITH, validating that hallucination degrades faithfulness. Table nificantly outperforms MLLM baselines across all T2I systems (c.g., +30%//200% on Qwen-Image for Intern V1.3/Qwen2.55V backbones)..
8 shows per-model results where HAL sig-
EXAG can effectively detect exaggeration. We are the first to measure exaggeration for cul- tural faithfulness evaluation. We explore two approaches: ITA-based using exaggeration candidates (Section 3.2) and MLLM-based prompting models to detect exaggeration. Additionally Tab. 12 illustrates top-3 descriptor examples.
17

---

Published as a conference paper at ICLR 2026
Method
Backbone
GT-HAL
GT-FAITH 1
Fluxl
Qwen-Image
SD3.5
Avg.
Fluxl
Qwen-Image
SD3.5
Avg.
InternVL3
0.32
0.07
0.20
0.20
-0.30
-0.10
-0.18
-0.18
MLLM
VL25
0.38
0.11
0.37
0.26
-0.39
-0.19
-0.31
-0.30
Intern VL
0.36
0.37
0.24
0.32
-0.51
-0.33
-0.30
-0.38
HAL
QwenVI.22. VI.2.5
0.35
0.31
0.32
0.33
-0.38
-0.21
-0.37
-0.32
Human
0.38
0.34
0.40
0.37
Table 8: Hallucination Per T2I. Spearman Correlation.
GT-EXAG
GT-FAITH
Method
Backbone
GS
GN
GS
GN
All
Intem VL3
0.24
0.26
0.25
-0.24
-0.21
-0.25
EXAG (MLLM)
VTen5
0.34
0.33
0.34
-0.31
-0.25
-0.30
EXAG (ITA)
VQAScore
0.36
0.16
0.29
-0.27
-0.14
-0.22
Human
0.31
0.39
0.34
Table Correlation with humans on Exaggeration. Best scores per-column are bolded. We ex- plore two approaches: EXAG(MLLM) use MLLM for predicting exaggeration, while EXAG(ITA) uses VQAScore and exaggerated candidates from Sec, 3.2
Table 9 compares I'TA-based (VQAScore) versus MLLM-based approaches. While MLLM-based methods show stronger correlation with GT-EXAG, we adopt the ITA-based approach in our frame- work because provides interpretable descriptor-level feedback. Our framework supports both approaches, allowing users to choose based on their needs for interpretability versus performance.
Table 10 compares three descriptor sources for ITA-based EXAG: (1) LLM LLM--enerated stereotype candidates, (2) reference descriptors (Dref), and (3) MLLM-extracted descriptors from real im- ages. Stereotype candidates achieve significantly better correlation with GT-EXAG (0.183 average) compared to reference descriptors (-0.251) and MLLM descriptors (-0.083). This demonstrates that general descriptors fail to capture exaggeration, as they include non-stereotypica correct elements. Only stereotype-specifi candidates effectively measure over-representation.
Correlation among metrics. Figure examines relationships between our metrics and human judg- ments. Both humans and our automatic metrics show strong negative correlation between ALIGN and HAL (-0.73/-0.74 and between ALIGN and EXAG (-0.67/-0.28) This validates a key property of effective cultural metries: faithful images must also penalize hallucination and exaggeration. This can be utilized by future work to validate adn constrcut stronger metrics. Additionally. ALIGN correlates positively with realism (0.51), indicating culturally aligned images also appear more pho- torealistic.
Descriptor-based feedback enables targeted image editing. Beyond evaluation, our descriptor- level feedback provides actionable guidance for improving generated images. Semuonstdmmossttttss this capability across four activities. For each example, AHEaD identifies specific issues: halluci- nated elements (e.g., elephants for "elephant ant man game," fur-leg leggings for Gumboot dance), exaggerated stereotypes (e.g., excessive batik clothing, Zulu attire with decorative beadwork), and aligned elements (e.g., standing in circle, players in close proximity). We use this feedback to cre- ate targeted editing prompts, instructing the model to remove hallucinated elements and reduce and diversify exaggeration while maintaining aligned elements (prompts in 32). Edited images show substantial improvements, better matching real reference images by removing culturally incorrect elements and reducing stereotypical over-representation. This demonstrates that interpretable de- scriptor feedback enables iterative refinement toward more culturally faithful generation.
Evaluation of LLM generated reference descriptors. We validate reference descriptor quality through human annotation. Table [T shows precision across 7 countries where annotators marked each GPT--4o-eenerated descriptor correct incorrect. Descriptors achieve high precision, aver-
18

---

Published as a conference paper at ICLR 2026
Method
Model
Flux1
Qwen
SD3.5
Avg.
Reference Desep (Dref)
EXAG (VQAscore)
-0.276
-0.256
-0.220
-0.251
Stereotype Cand. S
EXAG (VQAscore)
0.349
0.184
0.230
0.183
MLLM-Desc.
EXAG (VQAscore)
-0.221
-0.092
0.063
-0.083
Human
0.231
0.097
0.482
0.270
Table 10: Exaggeration per T21 (Spearman). Correlation across different text-to-image genera- tors. Results are based on wcountries.
Human (Likert)
Model
0.51
-0.74
ALIGN Vs
ALIGN VS HAL
ALIGN VS REAL
EXAG
Figure 7: ALIGN negatively correlates with HAL and EXAG. Both human judgments (purple) and automatic metrics (green) show negative correlation between ALIGN (GT-FAITH for human) and HAL/EXAG (GT-HAL/GT-EXAG for human), confirming that effective faithfulness metrics must penalize hallucination and exaggeration. ALIGN positively correlates with realism.
aging 90.27% with minimum 85.21% % (Iran), demonstrating reliable cultural accuracy across diverse regions.
Direct recall measurement is infeasible without ground-truth descriptor sets. We estimate recall through measures: (1) annotators rated overall descriptor quality 5-point Likert scale (aver- age: 4.5/5), and (2) annotators identified missing descriptors in free-text responses (only 26 of 378 reported any omissions). These results demonstrate high precision and comprehensive coverage.
We further improve descriptor quality through our Proposer-Refiner approach (Section 3 as shown Table
Raw human scores. Table 13 summarizes human judgments across countries. Consistent with our automatic metrics, humans assign higher faithfulness scores (GT-FAITH) and lower exagger- ation/hallucination scores scores (GT-REALISM) are slightly higher for GS (3.31) than GN (3.15), though the difference small. Standard deviations indicate substantial variation within countries, reflecting the subjective nature of cultural evaluation.
for GN than GS countries. Interestingly, realism
A.6 PROMPTS
In this section, we include prompts used in this project.
19

---

Published as a conference paper at ICLR 2026
T2I
AHEaD Feedback
Edited Image
Real Image
Elephant, man, ant Game, Indonesia
ALIGN: standing circle,
HAL: elephant super-hero costume,
EXAG: batik, sarangs
ALIGN: players circle, clay cour toard players in close proximity
Zu
Game Iran
long tunic, EXAG: dome, rug, overlay robes
ALIGN: around
Mühle
modern aftire, beer glass
Game
HAL: dining. toasting instead
Germany
playing. natural light,
EXAG: beer glass, beer-hall almosphere
ALIGN: synchronize movement, dancers facing audience, semi-circle formation HAL: fur-leg feigners, Zulu-style beadwork, open hards giant protea flowers
Gumboot
Dance
South Africa
beadwork,
Figure 8: Illustration of descriptor effectiveness in guiding image editing for improved gen- eration. (a) Initial T2I-generated images (top to bottom: Gemini 2.5 Flash Image, Gemini 2.5 Flash Image, FLUX, Qwen-Image). (b) Generated feedback by AHEaD: We use AHEaD feed- back along with reference descriptors Dref create clear instruction prompts (prompt in Table 32) (c) Edited images: Nano-Banana is utilized to edit images according to instruction prompts gener- ated in (b). (d) Real images.
China
France
Iran
Nigeria
USA
India
Brazil
Avg.
89.80
90.54
85.21
91.62
91.44
91.61
91.68
90.27
Table 11: LLM (GPT-4o) generated descriptors validation by humans.
20

---

Published as a conference paper at ICLR 2026
Country
Activity
Top-3 Exaggerated Descriptors
Brazil
Celebration
people in bikinis and swimwear, excessive Brazilian flags; favela back- grounds
Brazil
Eating
tropical rainforest backgrounds; excessive Brazilian flags; favela back- grounds
China
Celebration
pagoda-style roofs; traditional robes; red lanterns
China
Eating
red lanterns; oversized calligraphy scrolls; chopsticks in every café terraces; chateaux; Eiffel Tower
France
Celebration
France
Eating
chandeliers; baguettes; croissants
Germany
Celehration
Gothic cathedrals; half-timbered houses; Alpine hats
Germany
Eating
bratwurst; cuckoo clocks; half-timbered houses
India
Celebration
exaggerated Bollywood posters; excessive gold jewelry; embellished saris oversized diya lamps; exaggerated Bollywood posters; embellished saris batik clothing; exaggerated temple structures; Balinese gates batik clothing; ubiquitous sarongs, large bamboo decorations giant domes; traditional robes; minarets every scene exaggerated calligraphy: giant domes; religious symbols
India
Eating
Indonesia
Celebration
Indonesia
Eating
Iran
Celebration
Iran
Eating
Italy
Celebration
Renaissance paintings; large Italian flags; exaggerated hand gestures
Italy
Eating
Renaissance paintings; excessive pasta dishes; wine bottles
Mexico
Celebration
Mexican flags everywhere; sombreros; papel picado
Mexico
Eating
papel picado; Frida Kahlo portraits; tequila bottles
Nepal
Celebration
giant pagoda roofs; Himalayan mountains; traditional attire prayer flags; Himalayan mountains; mandalas
Nepal
Eating
Nigeria
Celebration
oversized bead necklaces; giant flags; market scenes
Nigeria
Eating
bead necklaces; gele headwraps; palm trees Barong Tagalog worn by everyone; salakot hats; giant flags bamboo furniture; rice terraces; giant flags
Philippines
Celebration
Philippines
Eating
South Africa
Celebration
oversized flags; large drums; Zulu beadwork
South Africa
Eating
oversized flags; township scenes; Zulu beadwork
Spain
Celebration
Gaudí style-styleeee architecture; oversized flags; red oelyeelow color schemes
Spain
Eating
red- red-yellow color schemes; Gaudí architecture; giant paclla pans minarets in every background; Ottoman architecture; oversized flags Outoman architecture; oversized flags: Turkish carpets
Turkey
Celchration
Turkey
Eating
USA
Celebration
oversized flags; suburban houses; cowboy hats fast food items; suburban houses; pickup trucks
USA
Eating
Table 12: Example Top-3 Exaggerated Descriptors. Extracted exaggerated descriptors for each country-activinn pair by EXAG metric. Descriptors reveal recurring cultural exaggerations associated with specific contexts and can be used diagnose bias or guide model correction.
21

---

Published as a conference paper at ICLR 2026
Region
GT-FAITH
GT-EXAG
GT-HAL
GT-REALISM
FRANCE
3.09 (0.73)
2.63 (0.70)
2.29 (0.74)
3.25 (0.39)
BRAZIL
3.61 (0.65)
2.27 (0.47)
1.79
3,46 (0.20)
CHINA
2.95 (0.84)
2.66 (0.52)
2.19 (0.52)
3.17 (0.44)
INDIA
3.27 (0.94)
2.38 (0.73)
1.97 (0.56)
3.56 (0.65)
MEXICO
2.91 (1.00)
2.41 (0.86)
2.38 (0.88)
2.98 (0.65)
GERMANY
3.25 (0.79)
2.25 (0.59)
2.04 (0.84)
3.10 (0.52)
NIGERIA
3.53 (0.33)
1.97 (0.57)
1.92 (0.52)
3.75 (0.57)
TURKEY
2.79
2.40 (0.79)
2.58 (1.18)
3.33 (0.55)
USA
3.96 (0.45)
2.19 (0.47)
1.59 (0.42)
3,48 (0.52)
IRAN
2.52 (0.76)
3.03 (0.78)
2.78 (0.53)
3.15 (0.61)
SPAIN
3.03 (1.10)
2.25 (0.74)
2.43 (0.68)
2.95 (0.45)
GS
3.04 (0.97)
2.44 (0.74)
2.28 (0.83)
3.31 (0.59)
GN
3.28 (0.89)
2.31 (0.65)
2.13 (0.75)
3.15 (0.50)
Table 13: Human evaluation scores by country. Mean (standard deviation) on 5-point Likert scale for faithfulness, hallucination, exaggeration, and realism. GN countries show higher faithfulness and lower halluciinaion/exxggeenioon than GS.
Activity
Example Subactivities countries)
Eating
Home. Restaurant
Greeting
Namaste (India), Prostrating (Nigeria), Three-kiss (Iran), Cheek kiss (France)
Dancing
Samba (Brazil), Flamenco (Spain), Bharatanatyam (India), Dragon Dance (China)
Game
Kabaddi (India), Ayuayo (Nigeria), Pétunque (Frunce), Baseball (USA), Mahjong (China)
Colebration
Nowruz (Iran), Cariial (Brazil), Bastille Day (France). Thanksgiving (USA), Chinese New Year (China)
Religious
Tazich (Iran), Candomblé ceremony (Brazil), Catholic mass (Mexico), Temple sarti (India)
Table 14: subset of examples subactivities in CULTIVate. Highlights distinctive cultural practices across countries.
LLM Descriptor Generator System Prompt
System: You are expert in cross-culture visual representation. Your task is to generate precise visual descriptors capturing how a typical scene of given activity appears in specific country. Descriptors must cover both traditional and modern variations and represent common culturally accurate scenes.
Rules: 1.
The output must strictly follow this JSON structure: "descriptors" "token":"...
"style"; "tradi tidaaiionlll modern neutral") terminology (e.g., samovar, sari) when appropriate; use
Use culturally-awe
broader cultural phrases when high specificity is unnecessary.
Focus only on the core activity scene (not before/after events).
4. Capture multiple common variations where they exist.
If the dimension has no representative descriptors, return an empty list.
Table 15: LLM descriptor generator System Prompt
22

---

Published as a conference paper at ICLR 2026
LLM Descriptor Generator - Setting & Background
Goal:
Describe the environment the physical location, architecture, and design elements that define the atmosphere of the scene.
Guidelines: INCLUDE:
Location and architectural style (indoors/outdoors: temple interior, city street)
Art and design (calligraphy, geometric tiles, minimalist décor)
Major furnishings (communal tables, floor cushions, rugs)
EXCLUDE: people, clothing, handheld objects, specific actions.
Generate up to (max items) descriptors for: {concept)
Table 16: LLM descriptor generator - Setting & Background
LLM Descriptor Generator - Objects
Goal: Identify the core objects central to the activity.
Guidelines:
Ensure descriptors accurately represent objects common in the activity scene within the given country.
INCLUDE: essential tools, vessels, foods (samovar, board game, hot pot).
Use visually descriptive categories (e.g., "bowls of noodle soup") instead of abstract labels ("Chinese food").
EXCLUDE: people, animals, clothing, architecture, actions, background décor. Generate up to (max_itent items) descriptors for:
{concept)
Table 17: LLM descriptor generator Objects
LLM Descriptor Generator - Attire
Goal: Describe typical clothing, accessories, and appearance features. Guidelines:
Use specific garment names only when culturally essential (e.g..
sari).
-Otherwise, use broader cultural categories (e.g.. traditional West African attire).
Include both traditional and modern clothing variations unless the concept is strictly historical.
INCLUDE; garments, headwear, accessories, ceremonial markings, uniforms. EXCLUDE: tools, furniture, actions, gestures.
Generate up to (max_items) } descriptors for: (concept)
Table 18: LLM descriptor generator - Attire
LLM Descriptor Generator - Interaction & Gesture
Goal: Capture actions, gestures, and social dynamics central to the activity.
Guidelines: INCLUDE:
Key person-objec actions (pouring tea from samovar)
Social gestures (sharing food, group dancing)
Culturally typical postures and formations (kneeling rows)
EXCLUDE: static object descriptions, clothing, setting details. Focus on actions and interactions.
Generate up to (max items) descriptors for: (concept)
Table 19: LLM descriptor generator Interaction & Gesture
23

---

Published as a conference paper at ICLR 2026
LLM Descriptor Generator - Spatial Arrangement
Goal: Describe layout and spatial organization of people and objects.
Guidelines: INCLUDE:
Positioning of people relative to key objects surfaces
Culturally meaningful configurations (eating at a table vs. around sofreh)
Ensure descriptors cover common variations in the activity across the country.
EXCLUDE: clothing details, object descriptions, actions.
Generate up to mmxx_ttems descriptors for: (concept]
Table 20: LLM descriptor generator Spatial Arrangement
LLM Refiner Prompt
You refine candidate visual descriptors for evaluating the cultural align- ment of Al-generated images representing a specific concept or activity in a given country. Your job is to select, clean, and filter descriptors based on cultural accu- racy and relevance.
System:
Task: Select and refine descriptors according the concept, country, and de- scriptor dimension.
Dimensions:
Setting venues, architecture, décor
Objects - central objects in the activity
Attire clothing, accessories, headwear
Interaction gestures, postures, social relations
Spatial Layout - positionin patterns
Rules: 1. Keep only culturally accurate descriptors.
2. Create diverse covering typical variations.
3. Do not invent new descriptors.
4. Merge duplicates overly specific items.
5. Remove unrelated descriptors.
6. Keep phrases concise (1-4 words).
7. Descriptors must match the assigned dimension.
8. Output up to (max. .emms] descriptors.
9. If none are valid, return an empty list.
Output Format: ["token": " "item", "style"; "traditional moderrn Ineutral")
Input: Concept: {prompt) in (country) Dimension: (dimension) Can-
didate Descriptors: (candidate_descr))
Table 21: LLM Refiner Prompt
MLLM Descriptor Exctractor (System Prompt)
As an expert on cross-cultura visual representation, your task is to generate precise visual descriptors evaluate the cultural alignment and accuracy of Al-generated images.
Goal: Capture visual elements of a typical scene of an activity in a specific country, covering both traditional and modern variations.
Rules: 1. Output strictly in JSON: "descriptors": ("token:"...""
"style": "traditiioons modern neutral",
2. Use culturally-ant aware terms (c.g., samovar, sari) when precise, broader cultural terms when sufficient.
3. Focus on the core activity scene- not before after actions.
Capture common variations (e.g., eating at a table vs. sitting on the floor).
5. If nothing distinctive exists, return empty list.
Table 22: System Prompt for descriptor generation.
24

---

Published as a conference paper at ICLR 2026
MLLM Descriptor Extractor (Setting & Background Prompt)
Goal: Describe the environment (location, architecture, design, furnishings).
INCLUDE:
IIdoors/outdoors (temple interior, busy street, simple home)
-Art & design (calligraphy, tiles, minimalist decor)
Major furnishings (floor cushions, rugs, communal tables)
EXCLUDE: clothing, handheld objects, actions.
Table 23: MILMM descriptor detector (Setting & Background).
Objects Prompt
Goal: Identify key objects, tools, foods, vessels central to the activity.
INCLUDE: essential items (samovar, board game, noodle bowls, shared hot pot).
EXCLUDE: animals, clothing, architecture, actions, background décor.
Table 24: Prompt: Objects.
Attire Prompt
Goal: Describe typical clothing, accessories, and appearance.
Rules:
Use specific garment names only when culturally essential (e.g., sari).
Otherwise, use broader cultural categories.
-Always include both traditional and modern possibilities.
INCLUDE: garments, headwear, accessories, ceremonial markings, uniforms.
EXCLUDE: tools, furniture, actions, gestures.
Table 25: MLLM descriptor detector (Attire).
Interaction & Gesture Prompt
Goal: Capture actions, gestures, and social dynamics.
INCLUDE:
Peesson and/or object actions (pouring tea from a samovar)
Social gestures (sharing food, group dancing)
Group formations (kneeling rows, circle formations)
EXCLUDE: static objects, clothing, setting.
Table 26: Prompt: Interaction & Gesture.
MLLM Descriptor Detector (Spatial Arrangement)
Goal: Describe the physical layout and positioning of key objects.
INCLUDE:
Relative positions (sitting around sofreh, standing in line)
Culturally significant layouts (table seating vs. floor seating)
EXCLUDE: clothing, object details, gestures.
Table 27: MLLM descriptor extractor (Spatial Arrangement).
25

---

Published as a conference paper at ICLR 2026
Prompt
You helpful assistant that identifies culture-specific visual elements that text-to-image model may exaggerate when depicting a given activity in a given coun- try. These elements should represent unrealistic overuse, oversized forms, out-of-place insertions of cultural cues. Return only concise visual stereotypes based on observable features.
System:
Task: Generate up to {(num_element eleeeetssmssss candidate exaggerated elements for the ac- tivity and country below. Elements must be concise noun phrases describing culture- specific items (e.g., "Persian rugs", "giant flags"). Focus attire, objects, symbols, dec- orations, architecture, and interaction styles. Avoid generic non-culture-specinf items (e.g., "trees", "crowds"). Do not describe scenarios or contexts- -only list the exaggerated element itself.
ACTIVITY: {[concept] COUNTRY: {[country])
User:
Provide the final list as single line of noun phrases separated by the "-" character. Do not include numbering, explanations, extra text.
Example format:
Persian rugs | oversize lanterns giant religious symbols
I people in traditional robes
Table 28: EXAG Candidate Generation Prompt
ALIGN Likert Scoring Prompt
System: You a helpful assistant scoring how well an image depicts an activity it would typically appear in the specified country. Provide a rating from 1 to 5 using the guidelines below. Only output the score in the required format.
Scoring Scale:
Not well at all (completely wrong unrelated)
Slightly relevant
Somewhat accurate
4 = Mostly accurate
5 = Exactly as expected for this country
Task:
PROMPT: {(prompt]
COUNTRY: [[country]}
User: Respond with ONLY the score the exact format below:
score is < num
Do include any other text, explanation, or formatting.
Table 29: ALIGN MLLM-as-a-Judge Prompt
26

---

Published as a conference paper at ICLR 2026
HAL MLL.M-as-a-Judee Prompt
System: You are a helpful assistant scoring whether an image contains incorrect elements with respect to the mentioned activity and country. Rate on scale from using the rules below. Only output the score in the required format.
Scoring Scale:
= Not at all. All elements seem appropriate.
2 = Slightly. One minor element seems foreign.
Somewhat. A few elements feel unusual.
Mostly. Many elements appear incorrect or from another culture.
Strongly. The image is completely wrong, most elements are wrong, the activity is wrong, or the scene belongs to another country.
Task:
PROMPT: {(prompt)}
COUNTRY: {(country])
User: Respond with ONLY the score in the exact format below:
score is < num
Do not include any other text, explanation, or formatting.
Table 30: HAL MLLM-as--a-uddg prompt
EXAG MLLM-ass---Judge Prompt
System: You are a helpful assistant scoring how exaggerated an image is terms of cultural clements (e.g., attire, objects, background, art/design). Rate scale from I to using the rules below. Only output the score in the required format.
Scoring Scale:
Appropriate and balanced.
Slightly decorative but within cultural norms.
3 = Several features feel idcalized or overly emphasized.
4 Many elements feel exaggerated and unrealistie.
Strongly stereotypical, misleading, or culturally irrelevant.
Task:
PROMPT: {(prompt)}
COUNTRY: {(country]}
User: Respond with ONLY the score in the exact format below: score
num
Do not include any other text, explanation, or formatting.
Table 31: EXAG MLLMMas--aJudge Prompt
27

---

Published as a conference paper at ICLR 2026
Image Editing Instruction Prompt
Task: Edit the image correctly show ((activity)) in {[country) following the in- structions below.
Remove:
AAl hallucinated elements from the HAT list
All exaggerated elements from the EXAG list
Any objects, clothing, poses, or background features that belong to the wrong culture, historical period, or activity
Add/Preserve:
ALIGN list that must remain present
Add correct interaction from REF DESCP list mentioned in HAL/EXAG
ADD 1-3 different types of attire from REF DESCP mentioned in HAL/EXAG
ADD correct background from REF DESCP if mentioned HAL/EXAG
Input:
ACTIVITY: {(activity)}
COUNTRY: {{ ([countzy)]
HAL DESCRIPTORS: {{HAL})
EXAG DESCRIPTORS: [[RXAG))
ALIGN DESCRIPTORS: HALIGN])}
REFERENCE DESCRIPTORS: ((REF DESCP))
Table 32: Image editing instruction prompt template. AHEaD feedback (HAL, EXAG, ALIGN) combined with reference descriptors Dref guides image editing to improve cultural accuracy.
28


Reviews:
[{"review_id": "QiauxitpLF", "reviewer": "Reviewer_rXJH", "summary": "The authors introduce metrics to effectively evaluate images generated by text-to-image models. The authors first propose building a dataset of 576 prompts from 16 countries, totalling 19000 images. The authors also propose 4 metrics to measure alignment, hallucination, exaggeration, and diversity. The authors conduct a small human study to rate images and compare their metrics with human correlation. They further calculate scores for generated images using their metrics to demonstrate that the disparity between countries in the global north and south, where models perform better on the former.", "questions": "Please see Weakness. I have asked the questions there.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The authors introduce new metrics to measure how well T2I models depict cultural activities. As T2I models are being deployed in many parts of the world, this is a very important problem that needs to be tackled, and reliable metrics are essential.\n2. The idea behind the metrics is well motivated. I agree that existing works mostly quantify alignment and quality metrics and might miss some nuances. Going beyond these and explicitly calculating hallucination and exaggeration is a nice direction.", "weaknesses": "1. **CULTIVate dataset:** The authors introduce a large scale dataset comprising 576 prompts and 19k images, but it is not clear what the utility of the dataset is beyond reporting the AHEaD metrics on them. Many of these image-prompt pairs have no human annotations to compare the correlation of metrics with. Could the authors provide more insights on this? \n2. **Human Study:** The authors have conducted a very small human study with a limited number of prompts, i.e, only 9 activities per country for 7 countries, which totals 63 prompts out of 576. Moreover, out of the 6 models used, the authors have conducted the study only for 3 public models. The authors use this to report the correlation between human scores and their metric scores. Due to the small size of the human study, which does not even cover all the countries in the CULTIVate dataset, it is not clear how significant the correlations are. This makes the claim of AHEaD metrics' efficacy weak and the overall study less reliable. I understand that getting human judgments is hard for all the prompts and images, but a representative set of prompts covering all the countries should be used. It would be great if the authors could provide insights on this.\n3. **Metric Definitions:**\n\t1. In L85, the authors contend that existing metrics have a problem in that they use VLM internal knowledge, which inherits biases. But it looks like the author's entire pipeline is based on VLM knowledge. This involves generating descriptors for a prompt and generated images, which are based on the VLM's internal knowledge of the activity. The authors also generate potential stereotype candidates for a prompt which again relies on VLM's knowledge of potential stereotypes that might be biased. This feels a bit contradictory to the initial claim. The authors should either revise the claim or provide more elaboration on why and how this method does not inherit the cultural biases of VLMs.\n\t2. The core of the metric is to come up with descriptors for a prompt. I am wondering if these capture the full scale of cultural nuances. This is because cultural nuances are also about the values and norms associated with the entities in the images. This can be something like an appropriate gesture, the way people are interacting, or about norms (which people must follow). The descriptors ask for the existence of certain elements, but do they also ask for non-existence? In certain activities, not observing some norms is as important as certain aspects being present. Does the metric account for this? Moreover, a given festival/activity can be celebrated in different ways and have different subcomponents. Generating an exhaustive set of descriptors and comparing with them might not be ideal, as there might be some really valid images that do not cover all possible descriptors generated by the model. Since the metrics divide by the sum of all descriptors, this might underreport the numbers. Hence, I feel this approach does not capture a significant part of cultural nuances. Do the authors have any thoughts on this?\n\t3. The authors generate stereotype candidates for exaggeration using VLMs and also descriptors. Have the authors verified the accuracy of the different types of descriptors? In L669, the authors mention 90% agreement with humans with their framework. It is not very clear which exact descriptors these agreements are, i.e., whether for alignment, exaggeration. Could the authors provide more details of the human study? Table 9 provides a breakdown, but these are descriptors validated by humans, i.e., precision. Is there any study for recall? It could be possible that there are some essential ones that the LLM missed?\n\t4. The authors define diversity and semantic diversity, but only report numbers using this. Since the authors introduce this as a new metric, they need to quantify how good this metric is compared to those like the Vendi score and that proposed by [1] and other relevant baselines, which has not been done.\n4. **Results:**\n\t1. The authors have compared their metrics with a limited set of baselines. Some notable baselines that the authors don't evaluate include VIEScore [2], UnifiedReward [3], which have been shown to have good correlation with human judgements [4]. Moreover, the authors have not computed the scores for GPT-4o or Gemini 2.5 (without descriptors, Table 3) to provide scores for faithfulness in Table 2. Only InternVL and QwenVL are used. It is unclear why the authors compare with these models.\n\t\t1. A minor point is why QwenVL and InternVL are used for baselines. There is Qwen2.5VL and Intern2.5VL also. Why not use these more powerful models? If the authors have used these, then due to a lack of citation, it is not clear.\n\t\t2. The authors should report the numbers of GPT and Gemini with the same prompt of faithfulness that they ask humans for a fair comparison, since they use these models to generate descriptors. [4] already show that these are very good for similar kinds of prompts, and hence it would be an interesting comparison.\n\t2. It seems from Tables 5,6,7 that the proposed metrics don't correlate very well with humans as compared to baselines like InternVL and QwenVL. It feels like for 2/3 models, QwenVL does better on hallucination (Table 6). Also, for exaggeration, QwenVL does better overall (Table 7). Moreover, it looks like the authors' metrics underperform for the global south, which again could be the result of biases in the descriptors or because of a small human study. This is especially important as authors use this to make claims for disparity between the Global South and North. Could the authors provide more insights into this behavior as this is confusing.\n\t3. The claim of disparity between Global South and North needs more substantiation. The AHEaD numbers reported are quite close in several cases and hence it would be good to report the mean and standard deviation to quantify how significant these differences are. Also what are the scores provided by humans and how do they differ across the global north and the south?\n\t4. Why is human-human correlation almost zero for humans for the Qwen model in Table 8? Similarly, in Table 9.\n5. **Minor Issues:** Here are some minor writing related errors: L425 (should be do not ask human annotators?), L240 which has an incomplete sentence, Table 7 has wrong numbers bolded. Lack of citations for models in Table 2.\n\n**References:**\n\n[1] Kannen et al. Beyond Aesthetics: Cultural Competence in Text-to-Image Models. \n\n[2] Ku et al. VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation\n\n[3] Wang et al. Unified Reward Model for Multimodal Understanding and Generation\n\n[4] Nayak et al. CULTURALFRAMES: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "en3gRCT82E", "reviewer": "Reviewer_RoXn", "summary": "The authors introduce CULTIVate, a benchmark of social activities across 16 countries / 9 activity types / 576 prompts / 19k+ generations and propose AHEaD, four descriptor-based, automatic metrics for cultural evaluation: Alignment, Hallucination, Exaggeration, and Diversity. Their composite FAITH score correlates better with human judgments than popular image-text metrics, and they document a consistent Global North > Global South performance gap across SOTA T2I models.", "questions": "1) Why not representation from more countries (only 16)? Also, what's the rationale behind surveying only 7 countries? \n2) How many annotators rated each image/prompt (per-item replicates)? Also, are the 189 human forms for generated images total across models or per model? If total, the effective sample per model is small, please report per-model/per-item counts.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) Interpretable diagnostics: Can list top/bottom descriptors per country/activity to see what is missing or overdone, useful for model iteration\n2) No per-image human labels are needed to compute metrics, and it shows good improvements over existing metrics.", "weaknesses": "1) Benchmark spans 16 countries, but the human study covers only a smaller subset (7 countries), with no clear rationale or GN/GS/per-country annotation breakdown\n2) Descriptors are produced with GPT-4o and Gemini-2.5 Flash, yet “MLLM-as-judge” baselines cover only InternVL/QwenVL - no strong judge like Gemini 2.5 Flash/pro or GPT-4o-vision. Makes it hard to evaluate whether different descriptors are needed and to justify that we need more than one Judge model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0oJvWYK4V", "reviewer": "Reviewer_b2M6", "summary": "This work introduces CULTIVate, a new cultural activity focused benchmark for cultural faithfulness of text-to-image models spanning 16 countries and 576 prompts. They use cultural descriptors generated by LLMs to create AHEaD metrics to measure cultural alignment, hallucination, exaggerated elements (stereotypes), and descriptor diversity. They find T2I models are more aligned to the cultures of countries from the Global North than the Global South, and show strong Spearman rank correlation of their proposed AHEaD metrics to human judgments via a user study.", "questions": "* How do the authors use the \"free text rationales\" from the user study (L354)?\n* How many annotators provide ratings for each form (L341)? This is important to know how many people agree / disagree about ratings.\n* It would be nice for L32 - 36 in the introduction to have citations to prior studies showing the effort and importance of cultural adaptation\n* What is used to compute sentence embedding similarity in the ALIGN metric (L235)? Is this metric also culturally biased?\n* The prompts in the Appendix (Tab 12 - 17, 26) are cut off and unreadable in full", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This works makes contributions towards cultural benchmarking of text-to-image models, which is a relatively underexplored and important area of research \n* They provide a new benchmark that is focused on cultural activities in contrast to prior work that focuses more on cultural objects (e.g. landmarks). The benchmark size (576 prompts), categorical coverage (9 super categories of activities) and regional coverage (16 countries) is of similar order to prior work [1- 3, 5].\n* The proposed AHeAD metrics are based on interpretable cultural descriptors (Fig 4) and correlate well with human judgments when compared to baseline metrics of image-text alignment and multimodal LLM judges (Tab. 2)\n\nNote: see references in \"Weaknesses\"", "weaknesses": "## Major Weaknesses:\n\n* I disagree with a central claim of the authors, i.e. that their benchmark \"... capture the contextual complexity **missing from object-centric benchmarks**\" (L74). Prior work has already evaluated on contextually-specific cultural activities, e.g. celebrations / festivals / performance arts [1- 5], weddings [4], religious activities [1], sports [1, 2, 6], cooking [6]. The authors claim that their activity-focused benchmark highlights \"**new failure patterns**\" such as \"hallucinated elements\" or \"heavily exaggerated scenes\" (L76 - 78). This behavior has also been established in prior work that discusses culture-specific stereotyping in detail [1, 2, 7]. They also claim that prior work [1, 3] rely on VLM metrics that inherit similar cultural biases, however these works also use visual perceptual similarity and diversity based metrics that do not use VLMs. In fact Rege et al. stresses on this scorer bias ('generative entanglement'), which is not mentioned in this work.\n\n* The authors stress that prior metrics using VLMs will inherit their biases (L85), but use multimodal LLMs (GPT-4o and Gemini 2.5 Flash) to generate culturally-specific visual descriptors (L206 - L215). Since these MLLMs are also trained on internet data, will they not also inherit similar biases as VLMs? Since the authors have highlighted VLM biases, they should also discuss if MLLMs are similarly biased, as this directly affects their AHeAD metrics. If they are not similarly biased, the authors should point to empirical evidence showing so explicitly in the paper.  \n\n* For EXAG, the authors use an LLM to generate *\"exaggeration candidates\"* (L245). How do we know that these are culturally accurate? Are these candidates rated by humans who identify with these cultures? \n\n* When comparing against image-text alignment baselines (Tab 2), what prompts are used? What configuration of CuRe I-T alignment scorer [3] was used? What prompts are used for the MLLMs (InternVL and QwenVL?) These details are very important to establish a fair comparison to baselines. \n\n## Minor Weaknesses:\n \n* The paper is generally written well, but has several vague descriptions that make the contributions hard to follow:\n    * L117: *\"Our AHEaD metrics achieve 7% better correlation with human judgment\"* - better than what, and what are the humans judging?\n    * L118: *\"The best agreement is achieved ... \"* - agreement between what?\n    * L212: *\"...maximizing recall of cultural elements\"* - recall with respect to what? In my understanding, there is no ground truth descriptor set to which the LLM predictions can be compared to?\n\n* In their user study, the authors mention that they \"conduct direct discussions with annotators when facing inconsistent scoring and explanations\"- what does this mean? Do the authors instruct these annotators on how to \"correctly\" score or write explanations?\n\n* Fig. 3a is quite hard to visualize; the legend colors do not match, and one metric is better if higher and the other if lower. Consider using a side-by-side bar chart for this.\n\n### References \n---\n[1] Khanuja et al., An image speaks a thousand words, but can everyone listen? on image transcreation for cultural relevance. *In EMNLP, 2024.*\n\n[2] Zhang et al., Partiality and misconception: Investigating cultural representativeness in text-to-image models. *In CHI, 2024.*\n\n[3] Rege et al., CuRe: Cultural Gaps in the Long Tail of Text-to-Image Systems. *In ICCV, 2025*.\n\n[4] Basu et al., Inspecting the geographical representativeness of images from text-to-image models. *In ICCV, 2023*.\n\n[5] Kannen et al., Beyond aesthetics: Cultural competence in text-to-image models. *In NeurIPS D&B Track, 2024*.\n\n[6] Romero et al., Cvqa: Culturally-diverse multilingual visual question answering benchmark. *In NeurIPS D&B Track, 2024*.\n\n[7] Jha et al., Visage: A global-scale analysis of visual stereotypes in text-to-image generation. *In ACL, 2024.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2XDnCeUS0", "reviewer": "Reviewer_yiqA", "summary": "The paper builds a benchmark and evaluation framework to measure cultural faithfulness of T2I models when prompted to generate social activities. It also introduces an interpretable evaluation metric capturing 4 diagnostic aspects which correlates better with human judgements compared to previous metrics or using a model as a judge. The proposed pipeline uses a proposer-refiner approach for generating and verifying textual descriptions. The experiments show cultural disparities between Global North and Global South depictions and highlight failure modes.", "questions": "1. It will be intuitive to see some concrete examples of descriptors that the refiner removed. For descriptors that annotators marked irrelevant (from Table 9), what categories of errors occur (mislabeling, overly generic, stereotype)?\n2. To address W2, it will be helpful to run the descriptor-extraction with 1–2 additional independent MLLMs (e.g., QwenVL or another open model) and report sensitivity of ALIGN/HAL/EXAG to the extractor choice. If results are stable, it will strengthen robustness claims, but if not, discuss limitations.\n3. How do the results look at per-country level or per-activity level? Which countries drive the gap? Why do particular countries (in GS bucket) perform better/worse?\n4. Is the 7% improvement with human judgment statistically significant? Report confidence intervals / p-values for the correlation improvements.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and clear. The main contribution of the paper lies in the metric, which is explainable and actionable through the decomposition of descriptor dimensions. The metric definitions (ALIGN, HAL, EXAG, DDIV/SDIV) are also well-defined and motivated.\n2. The paper includes many ablations (e.g., τ threshold, proposer vs proposer+refiner) and per-dimension analyses.\n3. There is strong evidence of the metric utility through a higher Spearman correlation with human GT labels than common ITA baselines and the model as a judge.\n4. The use case of the metric is also clear, since it enables interpretable comparison across different aspects of various T2I models.", "weaknesses": "1. The empirical findings are incremental, several concurrent works (though reported by the authors) report similar Global North vs Global South gaps and also collect prompts and images for social activities. \n2. The same MLLM (InternVL) is used to extract image descriptors in the pipeline (L727) and also appears in baseline comparisons (InternVL as a judge). This creates a potential dependency/leakage where the extraction backbone advantages or disadvantages affect both measurement and baselines.\n3. Human evaluation was conducted for 7 countries × 9 activities × 3 models plus some real images (216 forms total). This is a small subset relative to the full 576 prompts / 16 countries. The paper reports Spearman correlation numbers and \"human-human agreement is moderate (L424)\" but lacks thorough reporting of annotator reliability (Krippendorff’s α / Cohen’s κ), per-country inter-annotator agreement, and statistical significance of the reported metric improvements.\n4. EXAG depends on LLM-produced “exaggeration candidates” and on web images as references. It is unclear how candidates are generated, how stereotypical or noisy web images affect EXAG, and how to distinguish a valid cultural variant from an exaggeration.\n5. The GN/GS split (listed countries) is reasonable for summarizing trends, but grouping heterogeneous countries into GS may obscure important nuances. For instance, China often performs well despite being placed in GS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)