Title: Beyond Aesthetics: Cultural Competence in Text-to-Image Models
Benchmark: CUBE (CUltural BEnchmark for Text-to-Image models)
Summary: This work introduces CUBE, the first benchmark designed to evaluate the cultural competence of Text-to-Image (T2I) models through the lenses of cultural awareness and cultural diversity. Using structured knowledge bases and large language models, the authors construct a scalable framework and dataset spanning 8 countries and 3 cultural domains (cuisine, landmarks, and art), revealing major cultural representation gaps in current T2I systems.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: cultural competence in text-to-image  models.
Phenomenon defined by authors: Yes
Phenomenon definition: Our benchmark aspires to enable reliable, trustworthy, and tangible measurement of text-to-image generative models for two distinct yet complementary behaviors: cultural awareness (i.e., the model’s ability to reliably and accurately portray objects associated with a particular culture), and cultural diversity (i.e., the model’s ability to suppress oversimplified stereotypical depiction for an underspecified input that references a specific culture).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is defined as evaluating text-to-image models on their cultural competence by measuring two components:
Cultural awareness – the ability to faithfully and realistically generate images of specific cultural artifacts.
Cultural diversity – the ability to produce a varied and representative set of cultural outputs from under-specified prompts, using a quality-weighted diversity metric.​

Task item definition: A text prompt referring to a specific cultural artifact (e.g., a dish, landmark, or clothing item) associated with a particular country - this is used to evaluate the model's generation. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1k
Dataset metadata available: Yes
Dataset metadata detail: Each item includes metadata such as the artifact name, country of origin, and concept category (cuisine, landmark, or art), and is used to assess cultural awareness or diversity.

Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean, None
Metric statistics: simple mean + std
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge