KITTEN: A Knowledge-Integrated Evaluation of Image Generation on Visual Entities
Authors: Hsin-Ping Huang
Xinyi Wang
Yonatan Bitton
Hagai Taitelbaum
Gaurav Singh Tomar
Ming-Wei Chang
Xuhui Jia
Kelvin C.K. Chan
Hexiang Hu
Yu-Chuan Su
Ming-Hsuan Yang
Venue: Accepted by TMLR ()

Abstract: Recent advances in text-to-image generation have improved the quality of synthesized images, but evaluations mainly focus on aesthetics or alignment with text prompts. Thus, it remains unclear whether these models can accurately represent a wide variety of realistic visual entities. To bridge this gap, we propose KITTEN, a benchmark for Knowledge-InTegrated image generaTion on real-world ENtities. Using KITTEN, we conduct a systematic study of recent text-to-image models, retrieval-augmented models, and unified understanding and generation models, focusing on their ability to generate real-world visual entities such as landmarks and animals. Analyses using carefully designed human evaluations, automatic metrics, and MLLMs as judges show that even advanced text-to-image and unified models fail to generate accurate visual details of entities. While retrieval-augmented models improve entity fidelity by incorporating reference images, they tend to over-rely on them and struggle to create novel configurations of the entities in creative text prompts. The dataset and evaluation code are publicly available at https://kitten-project.github.io.

Published in Transactions on Machine Learning Research (01/2026)
KITTEN: A Knowledge-Integrated Evaluation of Image Generation on Visual Entities
Hsin-Ping Huang!: 1.2- Xinyi Wang'*
Yonatan Bitton'
Hagai Taitelbaum! Kelvin C.K. Chan'
Gaurav Singh Tomar'
Ming-Wir Wei Chang' Xuhui Jia' Ming-Hsuan Yang!.
Hexiang Hu'
Yu-Chuan Su²
'Google DeepMind
Uniierritty of California, Merced
Reviewed on OpenReview:
Abstract
Recent advances in text-to-image generation have improved the quality of synthesized images, but ovaluations mainly focus on aesthetics or alignment with text prompts. Thus, remains unclear whether these models can accurately represent a wide variety of realistic visual entities. To bridge this gap, we propose KrTTEN, benchmark for image genera Tion real-world ENtities. Using KITTEN, we conduct systematic study of recent text-to-image models, retrieval-augmenner models, and unified understanding and generation models, focusing on their ability to generate real-world visual entilies such as landmarks and animals. Analyses using carefully designed human evaluations, automatic metrics, and MLLMs as judges show that even advanced text-to-image and unified models fail to generate accurate visual details of entities. While retrieval-augmentt models improve entity fidelity by incorporating reference images, they tend to over-rely on them and struggle to create novel configurations of the entities in creative text prompts. The dataset and evaluation code are publicly available at fetps://kitten-projffeee beapeett.gthub.... github.io.
1 Introduction
Recent advances in generative AI have revolutionized multimedia content creation. Large Language Mod- els (LLMs) excel knowledge-intensive tasks like question answering and summarization. Cutting-edge image generation models, such as Imagen (Saharia et al., 2022; Imagen 3 Team, 2024; Hu et al., 2024), DALL E (Ramesh et al., 2021; 2022). and Stable Diffusion (Rombach et al., 2022), generate photorcalistic and creative images from text. However, as these models become more capable and popular, assessing their reliability is crucial. Research on LLMs shows that even the most advanced models can generate inaccuracies, potentially undermining trust and causing societal harm (Muhlgay et al., 2023; Feng al., 2023).
Despite increasing atlention to factuality in LLMs, the accuracy of image generation models remains underexplored. Existing benchmarks mainly assess alignment with general text descriptions (Lin et al., 2015), compliance with image-editing instructions (Ku et al., 2024), or adherence to spatial relationships (Gokhale et al., 2022). However, they fall short evaluating how well models generate images that faithfully reproduce the precise visual details of real-workd entities, objects, and scenes. These should be grounded in trustworthy knowledge sources (see examples in Fig. 1). Recently, HEIM (Lee et al., 2024) introduces evaluation suite for assessing various aspects of image generation, including the ability to generate entities such as historical figures or well-known subjects. However, real-world visual entities are far more diverse than those covered by HEIM, requiring a broader assessment. Moreover, HEIM primarily evaluates the alignment between generated images and entity names in text prompts. It fails to capture the fino-grained visual details essential for assessing the reproduction of visual-world knowledge, nuances of real-world entities cannot be conveyed through text alone. Thus, it is essential to directly evaluate the fidelity of the generated images.
"Equa contribution

---

Published in Transactions on Machine Learning Research (01/2026)
Real Photo
Stable Diffusion
Adobe Firefly
DALL E
Great Pyramid of Giza
Statue of Equality Ramanuja
Figure 1: Can text-to-image models generate precise visual details of real-world entities? State- of-the-art models effectively render well-known entities (e.g., the Great Pyramid of Giza) but often struggle with less well-known entities, leading to hallucinated depictions.
To address the gap in ovaluating image generation models' ability to reproduce visual-world knowledge, we introduce KITTEN, benchmark dataset and evaluation suite designed to assess how well models generate visually accurale representations of real-workd entities grounded in trustworthy knowledge sources. Unlike prior benchmarks that focus aesthetics, text alignment, commonsense reasoning, KITTEN uses prompts derived from visual entities documented in Wikipedia (Hu et al., 2023a), a reliable knowledge base, and evaluates real-world entitics across eight visual domains (see Fig. 2). This ensures that generated images are compared with verifiable visual information crowdsourced from the internet. Additionally, we have developed a comprehensive set of human evaluation criteria that focus on the precise visual depiction of entities, capturing subtle but essential details for visual accuracy. By directly assessing entity fidelity in the generated images against established knowledge, KITTEN aims to advance the evaluation of world knowledge in image generation models.
Using KITTEN, we conduct a comprehensive evaluation of various text-to-image models, including standard, unified, and customization models fine-tuned utilizing in-context learning with retrieved reference im- ages (Chen et al., 2022). Our findings show that even the most advanced models (Tmagen 3 Team, 2024; Black Forest Labs, 2024; Xie et al., 2024; Chen et al., 2025b) often fail to produce accurate representations, generating images that lack critical dotails essential for visual correctness. While retrieval-angmeenee models improve visual fidelity by incorporating reference images during lesting, they lend over-rely on these references, limiting their ability to generate novel configurations of entities from creative prompts. These findings highlight key challenge in current image generation models: balancing entity fidelity with creative flexibility, underscoring the need for techniques that can generate precise visual details without sacrificing the ability to respond to diverse and imaginative user inputs.
2 Related Work
Existing evaluation for text-to-image generation. Evaluating text-to-inage models has long been challenging, with many efforts aimed at improving performance measurement. Fréchet Inception Distance (FID) (Heusel et al., 2017) common metric for assessing perceptual quality by measuring the distribution gap between generated and real-world images. CLIP-T scores (Hessel et al., 2021) evaluate text-image
2

---

Published in Transactions on Machine Learning Research (01/2026)
Knowledge Base
Visual Entities
Evaluation of Fidelity in Entities
Bandinelli Palace
WIKIPEDIA
Pour Eayylighins
Entity- baiity Prompts
Support Set
Human Evaluation
Aircraft
Photo Bandinelli Palace.
Faithfulness to Entity
Vehicle
Instructionfoolloinng
Bandinelli Palace stands against snow mountains.
Cuisine
Automatic Metric
Flower
A peacock in front of Bandinelli Palace.
Image-Entity Alignment Image-Text
Insect
Evaluation Set
Landmark
oil painting of Bandinelli Palace.
MLLM Evaluation
Plant
Bandinelli Palace made of crystal.
MLLM Entity Alignment
Sport
MLLM Text Alignment
Figure 2: KITTEN benchmark is constructed from real-world entities across eight domains. For each selected entity, we define five evaluation tasks with smage-generatie prompts incorporating the entity. KITTEN includes support set of entity images from the knowledge source for evaluating retrieval-augmentee models, as well evaluation set for assessing the fidelity of the generated entities.
alignment by comparing the CLIP feature similarity between generated images and input prompts. These metrics summarize overall image quality. Several works assess alignment between generated images and text descriptions (Yarom et al., 2024; Gordon et al., 2023; Hu et al., 2023b; Wiles et al., 2024; Cho et al., 2023), but they mainly focus on semantic consistency rather than the fine-grained visual accuracy of depicted entities specialized visual-world knowledge.
Recent works aim to evaluate models more thoroughly by decomposing the evaluation into subcategories such as attribute binding and numeracy, along with corresponding benchmarks. For example, the SR2D dataset (Gokhale et al., 2022) and the VISOR metric evaluate spatial relationships in text-to-image models, assessing whether objects the generated image adhere to specified relationships (e.g., an orange above a giraffe). T2T- P21-CompBench+++ (Huang et al., 2023) contains text. prompts from four categories (e-g-; attribute binding) along with associated metrics. HRS-Bench (Bakr et al., 2023) evaluates model performance across five major groups (i.e., bias, fairness, generalization, accuracy, and robustness). TIFA v1.0 (Hu et al., 2023b) is benchmark spanning 12 categories, paired with an automatic evaluation metric that measures image faithfulness via visual question answering. GenAT-Bench (Li al., 2024) and ConceptMix (Wu et al., 2024) focus on evaluating compositional text prompts (e.g., objects with specific colors, shapes, or spatial relationships). Additionally, ImagenHub (Ku et al., 2024) evaluates models across different tasks by measuring semantic consistency and perceptual quality.
Fidelity of entitics in text-to-image generation. While text-to-image models enable the creation of images from text descriptions, challenges arise when visual-world knowledge is required, i.e., generating accurate visual details of specific entities. Existing works have identified this issue and proposed solutions to mitigate hallucination (Lim & Shim, 2024). However, clear methodology exists to systematically assess these models' limitations, which is crucial for further improvement. In this work, we propose KITTEN, a benchmark addressing the novel problem of evaluating image generation models' ability to produce fine grained details of visual entities. We systematically assess the latest text-to-image, unified, and vetrieval-angmented models using carefully designed human evaluations, automatic metrics, and MLLM- based assessments.
World knowledge evaluation for text-to-inage generation. Evaluating text-to-image models with prompts that require world knowledge and commonsense reasoning has received increasing attention. WISE (Niu et al., 2025) evaluates complex text understanding and semantic reasoning grounded in cultural and scientific knowledge. RISEBench (Zhao et al., 2025) and KRIS-Bench (Wu al., 2025) focus on assessing bnowledge-based reasoning in image editing tasks. Several studies examine commonsense and physical reason-
3

---

Published in Transactions on Machine Learning Research (01/2026)
ing: WorldGenBench (Zhang et al., 2025a) evaluates prompts involving implicit reasoning; R21-Bench (Chen et al., 2025a) studies commonsense, mathematical, and logical reasoning; OmniGenBench (Yang et al., 2025) evaluates instructionnfollowinng capabilities involving physical and commonsense knowledge; and ABP (Zhang et al., 2025b) assesses implicit reasoning across knowledge of scientific, natural, and cultural scenes.
In contrast, KrTTEN addresses gap in coneurrent benchmarks by emphasizing the reconstruction of detailed visual features (e.g,, shape and color) for broad set of real-world entities across diverse domains. We explicitly collect evaluation images for target entities and directly compare them with generated outputs, enabling a more precise assessment of visual-detail fidelity. Additionally, KITTEN evaluates aarieval-augmented models to determine whether explicitly providing reference images improves entity generation.
3 KITTEN Benchmark
We introduce the KITTEN benchmark to evaluate the reliability of text-to-image models in generating knowledge concepts.
3.1 Design Desiderata of KITTEN
The key to ercating the benchmark constructing a set of imagc-gencration prompts that require grounding in visual-world knowledge. Two specific properties differentiate our benchmark from prior evaluation frameworks for image generation. First, while existing benchmarks aim to test the common -euuo--sense knowledge of image generation models, such spatial physical relationships (Gokhale et al., 2022; Huang et al., 2023), we focus on stress-testing the models by generating entities from specific domains. Therefore, we create the benchmark using image concepts from Wikipedia, a rich knowledge source that contains several domain- specific entities and their corresponding images. Second, while most existing benchmarks focus on evaluating the instruction-folloving capability of the models, we aim to understand how well these models faithfully represent real-world concepts grounded in visual knowledge sources. Therefore, design a specific set of evaluations targeted at capturing the visual fidelity of generated entities. Guided by the above principles, we next clarify the details of the KITTEN benchmark.
3.2 Creating Entity- based Prompts
Fig. 2 shows the benchmark creation process. To generate a diverse set of prompts focused on faithfulness knowledgo-grounde concepts, first select entity domains from the OVEN- Wiki dataset (Hu et al., 2023a), the most comprehensive open-domain image recognition dataset. We select eight. domains comprising 322 entities, covering made objects, natural species, and human activities. This selection offers broader coverage than existing benchmarks (Lee et al., 2024). For each entity, we collect an evaluation set of entity images from Wikipedia for human evaluation, and a support set of images to evaluate retrieval-anmmented models that leverage external knowledge sources for image generation (see evaluated models in Sec. 4).
After selecting the entitios, we design five evaluation tasks for imago-generation prompts.
Basic prompt (4.58%): Photo of Bandinelli Palace.
Entity in a specified location (30.57%): Bandinelli Palace stands against snow mountains.
Composition with other objects (22. (2): A peacock in front of Bandinelli Palace.
Entity in specific styles (21.20%): An oil painting of Bandinelli Palace.
Entity made of specific materials (20.87%): Bandinelli Palace made of crystal.
For each lask, ChatGPT (OpenAl et al., 2023) is instructed to generate text-to-image prompls for each entity domain according to the five evaluation tasks. These prompts are then tailored to evaluate knowledge-entity generation by incorporating entity names directly into the text. This process resulted in a set of 6,440 prompts to assess the models' ability to handle diverse and imaginative user inputs. The prompts range from 4 to 24 words, with average length of 9.91 words and standard deviation of 2.86.
4

---

Published in Transactions on Machine Learning Research (01/2026)
User
Model
Generate image according the following description:
watercolor painting the Bandinelli Palace landmark.
Use the reference images (provided above) to ensure the antity the prompt (Bandinelli Palace landmark) is fictually correct You may conduct your own research (eg. Google Search) to clarify the appearance of entiyy and ensure its accurate depiction the generated image.
Faithfulness to Prompt Entity
Faithfulness Prompt Entity
Biithfully does the generated image represent the eutity mentioned in the prompt?
Does generuted image accurately and comprchensively depict any aspects of scene entity described the prompt that are not already reflected in the reference images?
Yes: The gemerated image accurately and comprehensively depicts aspects of the scene or creatity described the prompt that are not already reflected the reference images.
Not faihhful wall
Barely Eithial
Somewhat faithiil
Mostly faithful
Complesely fiihhull
The generated image fails accuraatlly comprehansively depict aspects oheeee oreatity described the proeptt that not altcady reflected tne reference images.
Figure 3: Annotation interface. Raters are asked to: 1) rate the image's faithfulness to the prompt entity on a 1-5 scale, and 2) indicate whether the image follows the prompt with "yes" or "no" response. We report the percentage of responses marked as "yes."
3.3 Human Evaluation
Since no established metrics exist for evaluating the generation of visual entities, human evaluation plays a critical role reliably assessing model performance. We design human evaluations to focus on the visual fidelity of the target entity in the generated image, decomposing the evaluation into two aspects: 1) faithfulness to the prompt entity, and 2) adherence prompt instructions beyond the entity. This design allows raters to focus on distinct criteria, enabling more informative comparisons of models, trade-offs between these aspects. Raters are shown reference images of the prompt entity and are encouraged to verify the entity's faithfulness through their own research. The final evaluation score the average of five raters per image ensure robust assessments. The human annotation interface shown Fig. 3.
results often show
Faithfulness to Entity. We use scale from 1 to 5, where 5 indicates complete faithfulness to the prompt entity, and 1 indicates that the generated image has no similarity to the prompt entity.
Instruction--folooiing We use a yes/no /no question to evaluate whether the generated image adheres to the prompt instructions beyond the entity. The percentage of "yes" answers is then calculated.
3.4 Automatic Metrics
We gather the results of popular automatic metrics for image generation models, which primarily measure the similarity between the generated images and the evaluation images prompts. While these metrics are not specialized for capturing the visual fidelity of the generated entity, we include their results to provide a comprehensive analysis of their alignment with human evaluation.
Image-Text Alignment. We measure the cosine similarity between the generated image and the text. prompt in CLIP's feature space (Radford et al., 2021), i.e., the CLIP-T Score (Hessel et al., 2021).
Image-Entity Alignment. We measure the average pairwise cosine similarity between the generated image and the evaluation images of the target entity using DINO's feature space (Oquab et al., 2024). This serves as a proxy for how closcly fine-grained details match.
5

---

Published in Transactions on Machine Learning Research (01/2026)
3.5 MLLM Evaluation
Multimodal large language models (MLLMs) have recently demonstrated impressive progress in visual understanding. We investigate the use of MLLMs as automatic evaluators to reduce human effort and address the limitations of traditional metrics in assessing visual fidelity. Specifically, we prompt GPTT-4o-mini (OpenAl et al., 2023) using the same criteria as our human evaluation.
MLLM Text Alignment. Given the text prompt, use an MLLM to evaluate whether the generated image follows the prompt instructions on 1-5 scale.
MLLM Entity Alignment. Given evaluation images of the target entity, we use an MLLM to assess how well the generated image resembles the target entity on 5 scale.
4 Evaluated Models
We present a comprehensive analysis using KITTEN to understand the visual-world knowledge in current. state-of-the-art models.
4.1 Text-to-lmage Backbone Models
First, we examine general text-to-image backbone models that generate images solely based on text prompts, without using additional tools reference images.
Stable Diffusion (Rombach et al., 2022) maps images a latent space where diffusion model is trained. We mainly use ST-1.55 (Rombach et al., 2022) for fair comparison with retrieval-augmented models, and also include SD-2.1 (Rombach et al., 2022), SD-3 (Esser et al., 2024), and SD-XL (Podell et al., 2023).
Fluxt (Black Forcst Labs, 2024) is successor to Stable Diffusion, integrating parallel transformer blocks.
Imagen (Saharia et al., 2022) uses T5 encoder and cascaded diffusion models for high-resolution image generation.
Imagen-3 (Imagen 3 Team, 2024) is successor to Imagen, notable for its ability to handle long prompts. DALL E-2 (Ramesh et al., 2022) is a diffusion model conditioned on CLIP embeddings, notable for its zero-shot compositional abilities.
4.2 Retrieval-augmenter Text-to-lmage Models
Retrieval-augmented methods are family of image generation approaches that use support images (e.g., retrieved by a search engine) to enhance the model through fine-tuning or in-context learning, improving the fidelity of entities in generated images. Our goal to evaluate these models and determine whether incorporating such support images enhances the fine-grained visual fidelity of the entity during generation. Specifically, we provide ground-truth reference entity images (held out from the entity images used for evaluation) as the support images to these methods and then generate new images from them following the evaluation text prompts. We study the following models:
DrcamBooth (Ruiz et al., 2023) finc-tunes the SD-1.5 model to learn special token encoding the target entity. It then generates the entity in new contexts using prompts that include this token.
Custom-Diff (Kumari et al., 2023), similar to DreamBooth, fine-tunes partial weights of SD-1.5.
dnstruct-Imager (Hu et al., 2024) generates the target entity through in-context learning by encoding support images into a multimodal instruction: Generate an image of <entity _amm>>,,_me,,__ referring to the images <ref_image __,
<ref _it__mmaee____ _x>, and follow the caption: <prompt>.
BLIP-Diffusion (Li et al., 2023) performs in-context learning by training a multimodal encoder to obtain support image embeddings aligned with text prompt embeddings.
IP-Adapter (Ye et al., 2023) performs in-context learning by training separate aross-attention layers incorporate support images as inputs.
We Flux.1-der
6

---

Published in Transactions on Machine Learning Research (01/2026)
Retrieval Models
Backbone Models
Imagen-3
0.31
SD-L-5 Imagen-3
Image- Text Alignment
Instruction- honn-folowingg
Flux
SD-115
Imagen
Imagen
0.33
Flux
DreamBooth
Bonth
Custom-Dii Diff
Custom-
60
0.32
Iustruct-Imagen
0.31
Instruct-Imagee
2.5
3
3.5
0.35
0.4
0.45
0.5
0.55
Faithfulness Entity
Imagee-Enitty Aligument
Figure 4: Evaluation results of text-to-image models. (Left) Human evaluation highlights the trade-off between entity faithfulness and Instruction-following (Right) Automatic metries illustrate the relationship between imagc-entity and image-text alignment.
4.3 Unified Understanding and Generation Models
Finally, we evaluate unified models that jointly learn multimodal understanding (e.g., image captioning, visual question answering) and generation tasks (e.g., text-to-image generation) within the same framework. These models do not take support images as inputs. We study the following models:
Show-o (Xie et al., 2024) tokenizes multimodal data into sequence, processing text tokens autoregressively and image tokens via discrete diffusion modeling.
Jamus-Pro (Chen et al., 2025b) employs unified transformer with autoregressive prediction, while using separate visual encoders for the understanding and generation components.
Emu3 (Wang et al., 2024) tokenizes multimodal data into discrete token spaces and trains a single transformer on next-loken prediction tasks.
5
Evaluation Results
5.1
Human Evaluation
Retrieval-augmented models enhance faithfulness but weaken instruction-following Fig. (left) shows that retrieval-augmentedd models, including Custom-Diff, DreamBooth, and Instruct-Imagen, generally produce images that more faithful to the entities than their base models, SD-1.5 and Imagen. This is because these models incorporate reference images during testing, enabling them to generate visual concepts that are not well represented in the base models' parameters. However, retrieval-angmented models tend to have reduced instruction-following capabilities, as they often over-rely on reference images and struggle create novel configurations of the entity, as requested in creative text prompts. Although this trend is consistent across methods, the extent of the impact varies. Notably, Instruct- Instruct-Imagen shows significant increase in faithfulness (2.81 + 4.22) but also substantial drop in cistruction-oolloinng (72.2 + 46.5).
Enhancing backbone models improves both faithfulness and instruction-followinng For example, Flux outperforms its predecessor SD-1.5 by 0.23, and Imagen-3 surpasses predecessor Imagen by 0.35 in faithfulness. However, these improvements faithfulness are still minor compared to those achieved by retrieval-augmmented methods, such DreamBooth, which shows a 0.57 improvement over SD-1.5.
On the other hand, Imagen-3 achieves the highest instruction-followiie score (83.6). A notable strength is its strong compositional capability in generating challenging scenarios, such as an insect wearing sun- glasses. Imagen-3 also attains a high faithfulness score (3.17), outperforming the strieval-augmmenter models DreamBooth (3.08) and Custom-Dir Diff (2.90). This demonstrates that Imagen-3, as strong backbone model,
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Evaluation results. MLLM evaluation shows the relationship between entity and text aligument.
Backbone Models
Metrica
DALL B-2
SD-1.5
SD-2.1
SD-3
SD-XL
Imagen
Flux
Imagen-3
MLLM Eatity Aligmment
2.65
2.39
2.45
2.77
2.79
2.53
2.04
2.83
MLLM Text Alignment
3.69
3.46
3.66
4.07
4.19
3.61
3.96
4.17
Retrieval Models BLII-Diffusii IP-Adapter Custom-Diff DreamBooth Imagen Show-o Janus-Pro Pro
Unified Models
Metrics
Emm3
MLLM Entity Alignment|
2.52
2.82
2.70
2.82
3.72
1.93
1.89
2.22
MLLM Text Aligument
2.08
2.65
3.29
3.37
2.63
3.55
3.65
3.96
can generate specialized entities solely from text prompts. However, Imagen-3 still faces challenges with low-frequency entitics, and a notable gap remains in entity fidelity compared to the highest score achieved by Instruct-Imagen (4.22). These findings show that enhancing the backbone model can improve both instruction-follownng and entity fidelity, while it is essential to incorporate advanced retrieval-augmennted techniques to achieve even higher levels of faithfulness.
Balancing faithfulness and instruction-followim is achievable. The retrieval-augieentee model DreamBooth improves entity faithfulness compared to its baseline, SD-1.5 (2.51 + 3.08), without compromising SD-1.5's instruction-folowiing score (72.2 - 73.8). This demonstrates that a well-designed retrioval-augmenned method can enhance entity fidelity without sacrificing creativity. These findings also suggest future research directions, emphasizing that combining strong backbone with an effective redrieval-augmeeneed approach can achieve balance between faithfulness and cestruction-folloinng
5.2 Automatic Metrics
Retrieval-augmenned models improve entity alignment but reduce text alignment. This observa- tion, shown in Fig. 4 (right), is consistent with the human evaluation results. Moreover, this trend aligns with findings in recent work (Materzyíska et al., 2023), which indicate that models incorporating additional inputs, such as reference images, tend to have lower text alignment scores than base models. This reflects a trade-off between aligning with the text and aligning with the images.
Alignment between automatic metrics and human evaluation. While the overall trends from the automatic metrics align with the human evaluation, there are notable discrepancies. We observe that improving base models does not necessarily lead to gains in the automatic metrics. For example, Flux (0.329) performs worse than its predecessor SD-1.5 (0.338) in the image-text metric, and Imagen-3 (0.389) shows only marginal improvement over Imagen (0.386) in the image-entity score. These findings suggest that automatic metrics have a limited ability to capture meaningful variations between models of similar quality. In addition, although DreamBooth achieves a higher mnstruciion-olllveet a lower image-text aligument score. We hypothesize that the image-text score may not accurately assess the alignment between the generated image and rare entitios. For example, with the prompt "The Teufelsmaner landmark shimmers in the sunlight,"" " it is unclear whether the image-text similarity for "Teufelsmauer" is evaluated correctly. This highlights that traditional metrics (Hessel et al., 2021; Tee et al., 2024) might fail to measure true aligument between the unique entity and the generated image. Furthermore, Imagen-3 ranks higher in faithfulncss in the human evaluation, yet DrcamBooth outperforms Imagen-3 in image-entity scores, indicating a misalignment between human perception and the learned semantic features (Oquab al., 2024).
score compared to its base model SD-1.5, it has
5.3 MLLM Evaluation
Alignment between MLLM and human evaluation. Tab. 1 shows the results of the MLLM evaluation. The MLLM scores generally exhibit a high correlation with luman evaluation results shown in Fig. 4 (left), although slight discrepancies exist. Specifically, DreamBooth (3.37) falls behind SD-1.5 (3.46) and Imagen (3.61) in the MLLM text alignment score. However, in human evaluation, DreamBooth (73.8) outperforms both Imagen and SD-1.5 (72.2). This suggests that while the MLLM can capture overall trends and identify models with clearly stronger or weaker performance, it may struggle distinguish between models with similar capabilities. On the other hand, Flux (2.04) receives lower MLLM en alignment score than
8

---

Published in Transactions on Machine Learning Research (01/2026)
SD-1.5 (2.39), despite outperforming SD-1.5 in human evaluation (2.74 vs. 2.51). This indicates that the MLLM may have different preferences or biases compared to human raters.
Gradual improvements in the SD scrics. SD-155, SD-2.1, SD-3, and SD-XL show consistent improvements in both MLLM entity and text alignment scores. Notably, SD-XL achieves strong performance comparable to Imagen-3, with similar entity (2.79 vs. 2.83) and text. alignment scores (4.19 vs. 4.17). These models generate high-frequency entities accurately but struggle with now-frequeccy entities, where the generated examples often fail to match the target's attributes or deviate significantly in structure and configuration.
In-context methods show weaker text alignment than fine-tuning methods. For Sttrieval-based underperforms Imagen in entity alignment (2.53) and
models that rely on in-context learning, lags substantially in text alignment (2.08 vs. 3.61). IP-Adapter achieves a higher entity alignment score, close to DreamBooth (2.82), but exhibits lower text alignment score (2.65 vs. 3.37). These results indicate that in-context learning approaches generally have weaker instruciion-ollowiin methods, such as DroahBooth and Custom-Diff We observe that in-context learning models often almost directly replicate the reference images in their outputs. While they maintain high entity fidelity, they struggle to follow instructions, rarely generate new materials or styles, and often fail to incorporate additional objects.
capabilities than fine-tuning-bast based
Unified models struggle to generate accurate visual entities. The unified models Show-o (3.55) and Janus-Pro (3.65) achieve moderate text alignment scores, comparable to Imagen (3.61). However, all three unified models exhibit, noticeably lower entily aligument scores, even below SD-1.5 (2.39). For low-frequency categories, such as plants and insects, these models often generate entities that bear little resemblance to the target. While high-frequency entities are produced more rcliably, the models still struggle with common items, such as Peking duck and macarons, highlighting persistent challenges in accurate entity generation. These results suggest that, although unified models may gain broader world knowledge through joint. training on understanding and generation tasks, effectively translating this knowledge into accurate visual entity gencration remains a significant challenge.
5.4 Ablation Study
Selection of image-entity alignment metrics. In Tab. 2, two popular visual features are tested for calculating cosine similarity scores between reference and generated images as the imagc-entity alignment metric: CLIP-I (Radford et al., 2021) and DINO (Oquab et al., 2024). DINO scores provide a clearer separation between models compared to CLIP-I scores, making them more discriminative metric for capturing subtle differences in faithfulness. For example, the difference between Custom- Diff and Instruct- Instruct-Imagen is much larger when using DINO (0.19) compared to CLIP-T (0.11). This may be due to DINO's focus on primary entities, which allows for a more accurate estimation of similarity between the generated entities and reference images.
Table 2: Selection of image-entity metrics.
Models
CLIP-1 DINO
SD-1.5
0.646
0.350
Imagen
0.646
0.386
Flux
0.639
0.380
Imagen-3
0.650
0.389
Custom-Diff
0.643
0.388
DreamBooth
0.674
0.412
Instruct- Instruct-Imagen
0.751
0.582
Correlation of automatic metrics with human evaluation. While manual evaluation ensures high accuracy, automated meth- ods provide a cost-effcctive alternative, alboit with slightly lower alignment to human perception. To quantify the consistency be- tween automatic metrics and human evaluation, we computed Pearson and Spearman correlations in Tab. 3. CLIP-T and CLIP- I show moderate alignment with user evaluations, while DINO demonstrates stronger alignment with human judgments of faith- fulness compared to CI.IP-I.
Table 3: Correlation of automatic and MLLM metrics with human evaluation.
Metrics
Models
Pearson
Spearmen
GPT-4o
0.703
0.695
Entity Aligument
Qwen-2.5
0.531
0.520
DINO
0.510
0.504
CLIP-1
0.239
0.310
Text Alignment.
GPT-4o
0.618
0.589
Qwen-2.5
0.515
0.513
CLIP-T
0.337
0.384
Correlation of MLLM metrics with human evaluation.
We assess the correlation between ML.LM-based evaluation and human judgments using two different MLLMs, GPT-4o (OpenAl et al., 2023) and Qwen-2.5 (Team, 2025). MLLM- based evaluation shows strong alignment with human preferences, highlighting its potential to capture subtle visual details when assessing the faithfulness of generated knowledge entities. This demonstrates that our evaluation framework can be reliably scaled using MLLLMs as automatic evaluators. Furthermore, GPT-40 achieves significantly higher correlations
9

---

Published in Transactions on Machine Learning Research (01/2026)
5D-1.5
Imagen
Flux
Imagen-3
Custom- Diff
DreamBouth
Instruct-Iman
Inage Beee---tttyyy (X)
80.0
04.6
52.0
60.0
40.0
28
20.0
Aireraft
Vehicle
Cuisine
Flower
Insect
Laudmark
Plant
Sport
Inage Eatitty
80.0
00.6
60.0
0003
40.3
Basic
Location
Composition
Style
Material
Figure 5: Performance analysis across domains and tasks. (Top) Retrieval models achieve higher imagc-entity scores in the insect, landmark, and plant domains but perform worse in the cuisine and sport domains, possibly due to the varying occurrence of these eutities in common image datasets. (Bottom) Localion scores the highest, while Composilion scores the lowest.
with human evaluation compared Qwen-2.5, while Qwen-2.5 still outperforms traditional CLIP and DINO metrics. These results indicate that our benchmark can be effectively extended using open-source MLLMs for automatic evaluation.
5.5 Analysis of Performance Variations
Performance across entity domains. Fig. 5 (top) shows that the performance of each method is domain- dependent. To analyze how domain distinctions affect model performance, we categorize the eight domains in our dataset along spectrum of visual variability and frequency in the training data: 1) Generic closses (e.g., sports, cuisine) exhibit the highest inter-category variation, with items differing substantially in appearance, and appear most frequently in the training data. 2) Specific classes (c.g., insects, flowers) show moderate variation, as items within the same species can still look distinet, and occur with moderate frequency. 3) Generic instances (e.g., aircraft, vehicles) have lower variation, as different instances of the same model share similar structures, and also occur moderately often. 4) Specific instances (c.g., landmarks) exhibit the lowest variation, with a single, fixed visual appearance, and appear least frequently.
For specific instances, retrieved images are highly representative, enabling retrieval-augmennedd models to produce outputs that closely align with the targets and achieve higher scores than backbone models. Since these concepts are less frequent in common image datasets, they are underrepresented in the backbone models' parameters. Thus, incorporating reference images during inference improves the performance of retrieval-augmenter models. In contrast, for generic classes, retrieved images exhibit higher variation and are less representative, so outputs from vetrieval-angmented models often resemble the target images less closely, resulting in lower performance compared to backbone models. In these domains, where concepts are common and well-memorized by the base models, retrieval may even harm performance due to overfitting limited reference images. Overall, this variability suggests that the effectiveness of retrieval- retrieval-uugmentedd methods depends on the nature of the domain-specits content, and that the optimal choice of retricval strategy remains an open question.
Performance across evaluation tasks. Fig. 5 (bottom) shows that imago-entity scores across evaluation tasks generally align with the overall ranking. Location scores highest (0.431), followed by Material (0.417), Style (0.399), and Composition (0.364), highlighting the challenge of maintaining entity fidelity when prompts involve complex compositions.
We observe that Style prompts show a distinct score distribution. Retrievallauggmented methods, DroahBooth and Custom-Dit along with their base model SD-1.5, receive lower image-entity scores (0.371, 0.359, and 0.311), indicating that models based SD-1.5 struggle to generate faithful entities when changing their
10

---

Published in Transactions on Machine Learning Research (01/2026)
Real Photo
Custom-Diis
DreamBooth
Instruct-Imagen
SD-1.5
Imagen
Flux
Imagen-3
Wakome dish with cherry flower top of it.
Real Photo
Custom-Dii Diff
DreamBooth
Instruct-Imagen
SD-1.5
Imagen
Flux
Imagen-3
A tog Rolls- Royse Phantom Drophead Coupé next giant sandeastic the beach.
Figure 6: Qualitative results. (Top) The backbone models, including SD-1.5, Imagen, Flux, and Imagen-3, show lower faithfulness to the cntity, with Wakame's appearance differing from the reference image. (Bottom) The retrieval models, including Custom-Diff, DreamBooth, and Instruct-Imagen, struggle with instruction following and fail create composition between the entity and the giant sandcastle.
styles. However, these models achieves the highest image-text score (0.346, 0.341, and 0.335), suggesting they are strong generating accurate styles but may sacrifice entity fidelity.
5.6 Qualitative Results
We present the visual results Fig. 6. In the above example, the backbone models (second row) show lower faithfulness to the entity, with Wakame's appearance differing from the reference image. Tn contrast,
11

---

Published in Transactions on Machine Learning Research (01/2026)
SD-2.1
SD-3
SD-XL
BLIP-Diffusion
IP-Adapter
Show-o
Janus-Pro
Emu3
wniter keeping the Churro dish table in
Figure 7: Additional qualitative results. The SD series, including SD-2.1, SD-3, and SD-XL, shows gradual improvements in faithfulness to the entity. Retrieval-angmented models based on in-context learning, such as BLJP-Diffusion and IP-Adapter, achieve strong entity fidelity but fail to follow the prompt. Unified models, including Show-o, Janus-Pro, and Emu3, generate incorrect visual details of the target entity.
retrieval-angmenter models (first row), which use reference images during testing, achieve better visual aligmment with the target. Instruct-Imagen demonstrates balance between entity fidelity and creative flexibility in the generated images. In the cxample below, Custom-Diff and DreamBooth exhibit reduced instruction-follooing, struggling with compositional prompts and often omitting the main entity or secondary objects. In contrast, the backbone models excel in entity faithfulness, likely because the target entity is well-represented in their training data. eetrieval-angmenteed models underperform when they over-rely on reference images and experience knowledge forgetting during fine-tuning small reference sets.
We show additional results in Fig. 7. The SD series demonstrates gradual improvements in capturing the correct details of the target entity. Retrieval-augmented models based on in-context learning (e.g., BLIP- Diffusion and Ad-dterrr) achieve strong entity fidelity but struggle to follow the prompt, failing to generate the waiter. Unified models (e.g., Show-o, Janus- Emu3) produce incorrect visual details of the target entity, Churro.
6 Conclusion
We propose KITTEN, a benchmark for evaluating entity fidelity in text-to-image generation, focusing on visual concepts that require specialized knowledge. We design prompts based on Wikipedia entities and introduce human and MLLM evaluation framework assess visual faithfulness. Extensive analysis reveals that while backbone models can generate specialized entities, retrieval-augmeeted models achieve higher fidelity. However, these methods often struggle with crealive prompts, highlighting the need for techniques that enhance entity fidelity without compromising
ciaructiionfolloviing ability.
Limitations and Future Work. We leave the evaluation of prompts requiring implicit reasoning (e.g., "the tallest building in Manhattan") for future work, as such prompts can often be reformulated into straightforward, entity-focused prompts (e.g., "One World Trade Center") through rewriting.
12

---

Published in Transactions on Machine Learning Research (01/2026)
References
Fslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. HRS-Bench: Holistic, reliable and scalable benchmark for text-to-image models. In ICCV, 2023.
Black Forest Labs. Flux, 2024. URL
foree--forott-bb/Iluue 2,
Kaijie Chen, Zihao Lin, Zhiyang Xu, Ying Shen, Yuguang Yao, Joy Rimchala, Jiaxin Zhang, and Lifu Huang. R2i-bench: Benchmarking reasoning-driven text-to-image generation. arXiv preprint arXiv:2505.23493 2025a. 4
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retriieal-anggmened
text-to-image generator. orXiv preprint arXiv:2209. 1 14411, 2022. 2
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.170 2025b. 2,
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint XrXin:2310. 18235, 2023. 3
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024. 6
Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge. arXiv preprint ar orXiv:230500225 2023.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint 2211: 2022. 1, 3, 4
Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen- -hr, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text misalignment. arXiv preprint arXfv:2312.08766 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08719 2021. 2, 5,8
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In NearIPS, 2017. 2
Hexiang HU, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Was Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In ICCV, 2023a. 2, 4
Hexiang HU, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, Ming-Wee Wei Chang, and Xuhui Jia. Instruct-imagen: Image generation with multi-modal instruction. In CVPR, 2024. 1, 6
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. TIFA: Accurate and interprelable text-to-inage faithfulness evaluation with question answering. In ICCV, 2023b. 3
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T21-CompBench: A comprehensive benchmark for woendworld compositional text-to-image generation. In NewrIPS, 2023. 3, 4
Imagen 3 Team. Imagen 3. or Xiu preprint ar arXiv:2408.07000 2024. 1, 2,
13

---

Published in Transactions on Machine Learning Research (01/2026)
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. In ICLR, 2024. 1, 3
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customiza- tion of text-to-image diffusion. In CVPR, 2023.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Decpak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. In Newr/PS, 2024. 1, 4,
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.10 13743, 2024.
Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject represcntation for controllable text-to-image generation and editing, arviv preprint arXiv:2305.14790 2023.
Youngsun Lim and Hyunjung Shim. Addressing image hallucination in text-to-image generation through
factual image retrieval. In IJCA Workshop, 2024. 3
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. In ECCV, 2015.
Joanna Materzyúska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, and Bryan Russell. Customizing motion in text-to-video diffusion models. orXiv preprint arXiv:2312.04900 2023. 8
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Soyton-Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint ar arXiv:2307.06908, 2023. 1
Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Chaoran Feng, Bin Zhu, and Li Yuan. Wise: world mnoweedg-iinorreet semantic evaluation for text-to-image generation. arXiv preprinl Xrx0:22503. 07265, 2025.
OpenAl et al. Gpt-4 technical report, 2023. URL https: wwwp:/wwwwww///////oo.ooo 0870 4, 6,
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmond Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. In TMLR, 2024. 5, 8,
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXie preprint TTT::2377.01522 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 5, 9
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 1
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Cascy Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06120 2022. 1, 6
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Tn CVPR, 2022. 1, 6
14

---

Published in Transactions on Machine Learning Research (01/2026)
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 6
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NewrIPS, 2022. 1, 6
Qwen Team. Qwen2.5 technical report. arXiv preprint 24 22.1111..11111 2025. 9
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction all you need. ar preprint arXiv:2409.18869, 2024." 7
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajić, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont- Tuset, et al. Revisiting to--aeo- evaluation with 100:2404.18820 2024.
gecko: On metrics, prompls, and human ratings. arXiv preprinl
Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty. arXiv preprint arxiv:2408. 14339, 2024.
Yongliang Wu, Zonghui Li, Xinting Hu, Xinyu Ye, Xianlang Zeng, Gang Yu, Wenbo Zhu, Bernt Schiele, Ming-Hsuan Yang, and Xu Yang. Kris-bench: Benchmarking next-level intelligent image editing models. arXiv preprint orxi:25055.117 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXie preprint arXiv:2488. 000008..5550 12528, 2024. 2,
Heng Yang, Jack Cole, Yuan Li, Renzhi Chen, Geyong Min, and Ke Li. Omnigenbench: A modular platform for reproducible genomic foundation models benchmarking. arXiv preprint orXiv:2505. 14402, 2025. 4
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roce Aharoni, Jonathan Herzig, Oran Lang, Eran Olek, and Idan Szpektor. What you see is what you read? improving lext-image alignment evaluation. In NewrIPS, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arriv:2308. 06721, 2023.
Daoan Zhang, Che Jiang, Ruoshi Xu, Biaoxiang Chen, Zijian Jin, Yutian Lu, Jianguo Zhang, Liang Yong, Jiebo Luo, and Shengda Luo. Worldgenbench: A world-knowleedge-nte integg-gteeetteed benchmark for reasoning-driven (ext-to-image generation. arXiv preprint arXiv:2505.0122 2025a. 4
Wenchao Zhang, Jiahe Tian, Runze He, Jizhong Han, Jiao Dai, Miaomiao Feng, Wei Mi, and Xiaodan Zhang. Align beyond prompts: Evaluating world knowledge alignment in text-to-image generation. arXiv preprint 25Xiv:2505.1 1000051887700 18730, 2025b.
Xiangyu Zhao, Peiyuan Zhang, Kexian Tang. Hao Li, Zicheng Zhang, Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, and Haodong Duan. Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing. arXiv preprint Xix:2255 02826, 2025.
15


Reviews:
[{"review_id": "OkUblXQ9Kz", "reviewer": "Reviewer_3Fq9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces KITTEN, a benchmark and evaluation suite for knowledge intensive text to image generation that focuses on visual entity fidelity rather than only aesthetic quality or text prompt alignment. It constructs 6,440 prompts over 322 Wikipedia entities spanning eight domains, provides reference images and support sets, and defines five task types (basic, location, composition, style, material), with a human evaluation protocol that separately measures entity faithfulness and instruction following along with automatic metrics (CLIP, DINO) and MLLM based judges. Using this benchmark, the authors systematically evaluate backbone, retrieval augmented, and unified understanding generation models, and empirically characterize tradeoffs between entity fidelity and instruction following, as well as the limitations of existing automatic metrics. Strengths include a clear problem formulation (visual world knowledge in generation), reasonably broad coverage of entities, a thoughtfully designed human evaluation pipeline, and extensive comparative results that highlight non obvious behavior of retrieval augmentation and unified models. Main weaknesses are that the entities and prompts, while diverse, are still limited to Wikipedia and eight domains, some important dataset and annotation details are relatively high level, and the benchmark currently focuses on single entity fidelity and simple compositionality rather than more complex reasoning or multi entity scenes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core claims are that (i) current text to image and unified models often fail to accurately reproduce fine grained visual details of real world entities, (ii) retrieval augmented methods substantially improve entity fidelity but often degrade instruction following and text alignment, and (iii) strong MLLMs can serve as reasonably correlated automatic judges for this benchmark. These are supported by a clearly described benchmark construction process from OVEN Wiki entities and Wikipedia images, a human evaluation protocol with two decomposed criteria and five raters per image, and quantitative results that consistently show retrieval augmented models increasing entity faithfulness while reducing instruction following, as well as backbone improvements that help both but do not fully close the entity fidelity gap. The paper further backs the MLLM evaluation claim with a correlation analysis comparing GPT 4o and Qwen 2.5 scores against human scores and traditional metrics such as CLIP T and DINO, showing higher correlations for MLLMs. Remaining concerns are mostly about scope and external validity rather than internal support: the benchmark is limited to the chosen domains and knowledge source, there is little ablation on prompt generation and annotation instructions, and some results rely on proprietary systems whose training data and interfaces are not fully controlled, but none of these issues obviously undermines the main empirical conclusions.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience includes researchers working on generative models, evaluation methodologies, and multimodal reasoning, all of whom would likely benefit from a benchmark that specifically targets factual visual entity knowledge in text to image models, a dimension that is increasingly important as these systems are deployed in user facing tools. The work provides a structured way to probe what current models actually know about real world entities visually, and it distinguishes between different families of approaches (plain backbones, retrieval augmented methods, unified models) in a way that informs both algorithm design and system engineering. The analysis of automatic metrics versus human ratings and MLLM judges is also relevant to a broader community that is grappling with how to evaluate generative systems at scale. Even if some readers do not directly use KITTEN, the design choices, observed tradeoffs between entity fidelity and instruction following, and domain specific performance differences provide conceptual insights about text to image model behavior that are of general interest to TMLR readers.", "requested_changes": "I would recommend the following adjustments: a) provide more detailed statistics and documentation about the dataset, including how entities and images were filtered from OVEN Wiki and Wikipedia, how many images per entity are in the support versus evaluation sets, and any licensing or quality filtering applied; b) expand the description of the human evaluation protocol, including annotator pool (expertise, geography), pay, quality control measures, and inter annotator agreement, since human judgment is central to the paper’s claims; c) clarify the prompt generation process from ChatGPT with concrete examples and error checks, and discuss how prompt wording might bias results; d) make the evaluation setup for each model family more explicit, especially for proprietary or API based systems (for example, prompt formats, temperature, number of samples, selection of images), and provide enough detail that others can reproduce the experiments; e) strengthen the discussion of limitations and external validity, for example with respect to the choice of Wikipedia entities, single entity focus, and lack of reasoning heavy prompts; f) include more analysis at the level of specific failure modes, for example qualitative error taxonomies per model family (wrong global shape, wrong material, spurious attributes) and a breakdown by entity frequency or popularity; g) release a detailed datasheet or documentation with scripts and annotation templates linked from the camera ready version.", "broader_impact_concerns": "The work touches on several broader impact issues that could be discussed more explicitly. Because KITTEN focuses on real world entities with reference images from Wikipedia, there are potential concerns around copyright, privacy, and representational bias in the underlying images (for example, geographic or cultural skew in which landmarks, cuisines, sports, or plants are covered and how they are depicted), and it would be helpful to spell out how these risks were assessed and mitigated and what usage constraints apply to the benchmark. There is also the possibility that a benchmark that emphasizes entity fidelity might incentivize model developers to memorize web imagery more aggressively, which could exacerbate copyright and privacy risks for individuals or locations that appear in training data. Finally, the benchmark and evaluation pipeline could be used to rank and market commercial models on “knowledgeable” image generation without adequately communicating limitations, which may mislead end users about factual reliability. I would therefore suggest adding or expanding a Broader Impact Statement that discusses dataset curation and licensing, potential misuse cases and mitigations, and the balance between encouraging better entity fidelity and avoiding harmful overfitting to web sourced images.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iOPQz3VJRH", "reviewer": "Reviewer_nWuG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose KITTEN, a benchmark dataset and evaluation suite designed to assess the fine-grained \"world knowledge\" of text-to-image (T2I) generation models. While prior benchmarks focus on general text alignment or spatial reasoning (Section 2), KITTEN specifically targets the visual fidelity of real-world entities sourced from Wikipedia. The dataset spans 8 domains, covering both living entities like Insects and Plants as well as non-living ones like Landmarks and Vehicles (Fig. 2). It comprises approximately 6,440 prompts across 5 task categories such as \"Composition\" and \"Style\" (Section 3.2).\nThe paper conducts a systematic evaluation of three model families: Standard Backbone models like Flux and Imagen-3, Retrieval-Augmented models like DreamBooth, and Unified Understanding models (Section 4). To validate their findings, the authors establish a human evaluation protocol measuring \"Faithfulness to Entity\" and \"Instruction Following\" (Section 3.3) and demonstrate a strong correlation between these human scores and MLLM-based judges like GPT-4o (Table 3; Section 5.4).\nKey Findings:\n1. Empirical results show that while retrieval-augmented models significantly improve entity fidelity compared to backbone models, they suffer from reduced instruction-following capabilities. They often ignore style or composition constraints due to an over-reliance on the provided reference images (Section 4.2; Fig. 4).\n2. Even state-of-the-art backbone models frequently hallucinate the visual details of less common entities, despite achieving high general text-alignment scores (Section 5.2). \n3. The study demonstrates that standard text-alignment metrics (e.g., CLIP-T) are unreliable for assessing world knowledge. These metrics often assign high scores to backbone models that generate a generic version of the requested object (e.g., a generic rock formation) while failing to render the specific named entity (e.g., \"Teufelsmauer\") correctly (Section 5.2).\nStrengths:\n* The authors confirm the reliability of their MLLM-based judge by reporting strong correlations with human ground truth (Table 3), ensuring that the subsequent automated ablation studies reflect actual human perception rather than relying solely on noisy metrics like CLIP.\n* The quantification of the trade-off between remembering what an object looks like versus following prompt instructions (Fig. 4 Left) provides a valuable insight for the future development of RAG-based image models.\nWeaknesses:\n* The dataset groups unique instances (e.g., specific Landmarks) with generic classes (e.g., Cuisine/Sports) under the single label of \"Entity.\" Differentiating between these types would clarify the domain-specific analysis in Section 5.5, as the utility of retrieval likely differs when generating a unique object versus a broad category.\n* Despite claiming to cover \"real-world entities,\" the dataset excludes specific human identities (e.g., historical figures) found in related benchmarks like HEIM [1]. This limits the scope of \"world knowledge\" evaluation, as it omits a major category of real-world entities.\n\n[1] Lee et al., “Holistic Evaluation of Text-to-Image Models,” NeurIPS, 2023.", "claims_and_evidence_yn": "Yes", "claims_explanation": "A key strength of the submission is the reliance on human evaluation as the primary ground truth rather than solely relying on automated metrics. By implementing a strict protocol with distinct raters for \"Faithfulness to Entity\" and \"Instruction Following\" as detailed in Section 3.3, the authors effectively isolate the trade-off between visual accuracy and creative adherence. Furthermore, the paper provides clear statistical justification for using MLLMs to scale this evaluation. The reporting of strong Pearson and Spearman correlations of approximately 0.7 between human judgments and the GPT-4o judge in Table 3 ensures that the subsequent large-scale ablation studies are grounded in reality rather than noisy metrics.\nAdditionally, the quantitative findings are convincing because they converge with the qualitative evidence. The aggregate data showing a drop in instruction-following scores for retrieval models in Figure 4 is supported by clear visual failure cases in Figures 6 and 7, such as retrieval models ignoring composition prompts to preserve entity structure. This alignment between the statistical graphs and the visual samples makes the findings regarding the fidelity-creativity trade-off highly persuasive. The authors also successfully demonstrate the inadequacy of existing metrics by showing that high CLIP-T scores often mask visual hallucinations in backbone models, as seen in the analysis in Section 5.2.\nThe evidence provided is accurate for the tested domains. However, I noted the scope in the summary because the empirical results focus on non-human objects and species. Extending the evaluation to include specific human identities would further support the generalizability of the findings across all real-world entities. Additionally, the analysis in Section 5.5 combines unique instances (e.g., specific landmarks) with generic classes (e.g., cuisine) into single domain scores. Distinguishing between these types would help clarify whether low scores are due to challenges in retrieving a unique instance or generating a generic category.", "audience_yn": "Yes", "audience_explanation": "As text-to-image models continue to scale and produce increasingly photorealistic outputs, assessing their factual reliability has become significantly more challenging. Recent research [2], highlights that traditional text alignment metrics like CLIP often fail to distinguish between generic representations and fine-grained, factually correct details. This submission aligns with that perspective by demonstrating that text descriptions alone are insufficient to convey the nuances of real-world visual knowledge, necessitating direct fidelity evaluation. Furthermore, the paper provides great insights for researchers in retrieval-augmented generation by rigorously demonstrating the trade-off between maintaining high entity fidelity and adhering to creative prompt instructions. These findings regarding the limits of current metrics and the behavior of RAG systems make the work highly relevant to the TMLR audience.\n\n[2] Lin et al., 2024 refers to: Zhiqiu Lin, et al. \"VQAScore: Evaluating Text-to-Image Generation with Visual Question Answering.\" ECCV, 2024.", "requested_changes": "Requested Changes\n\n1. Clarification of \"Entity\" Scope (Strengthening): The paper defines \"Visual Entities\" across 8 domains (Fig. 2). While this includes living things (Insects, Plants), it appears to explicitly exclude specific human identities (e.g., celebrities, historical figures), unlike related benchmarks such as HEIM [1].\n* Request: Please add a brief justification for the exclusion of specific human identities in the introduction or discussion. Clarifying whether this is due to ethical constraints (privacy) or technical constraints (different metrics required for face ID) would help readers better understand the specific scope of \"world knowledge\" being evaluated.\n2. Distinction between Instances and Classes in Section 5.5 (Strengthening): In Section 5.5, the analysis groups unique instances (e.g., specific Landmarks like \"Bandinelli Palace\") with generic classes (e.g., Cuisine like \"Guacamole\" or Sports like \"Snowboarding\").\n* Request: Please include a brief discussion on how this distinction impacts the performance of retrieval-augmented models. It is intuitive that retrieval mechanisms serve a different function when the target is a unique object not present in the base training data versus a generic category that the model likely understands. Acknowledging this semantic difference would add depth to the domain-specific analysis.", "broader_impact_concerns": "No ethical concerns. The work focuses on evaluation and benchmarking, which inherently contributes to the safety and reliability of generative AI by quantifying hallucinations. However, since the objective is to improve the fidelity of \"Real-World Entities,\" there is a theoretical dual-use risk regarding the creation of convincing misinformation (e.g., placing real landmarks or cultural artifacts in fabricated contexts). A brief acknowledgment of this potential would be a positive addition.", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "uLWDZl6sle", "reviewer": "Reviewer_tQZs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a benchmark dataset and evaluation suite KITTEN for assessing whether text-to-image models can accurately generate real-world visual entities grounded in a knowledge source Wikipedia.\n\nstrengths\n1. The paper builds a focused benchmark that directly checks whether a model can draw a specific real-world entity (e.g., a particular palace) correctly, not just make a pretty picture of generic scene.\n\n2. The paper compares backbone generators, retrieval-augmented methods with reference images, and unified multimodal models, using human judgment plus automatic and MLLM-based metrics.\n\nWeaknesses\n1. KITTEN's samples may overlap with the training data\nAs mentioned in paper \"Kitten uses prompts derived from visual entities documented in Wikipedia\", it may cause overlap with the training data. Many models are trained on large web datasets which likely include the same or similar Wikipedia images, also similar captions.\n\n\n2. It's not actually \"Knowledge-Intensive\" benchmark\n\nThe paper’s title and motivation repeatedly say KITTEN is a benchmark for “Knowledge-Intensive” image generation on real-word Entitis. A benchmark can be called knowledge-intensive if it use stored world knowledge that is not directly written in the input prompt.\n\nKITTEN's prompts give the entity name explicitly intead of world knowledge such as facts, relations, causal reasoning (e.g., “the tallest building in Manhattan”). The paper also acknowledges the knowledge-based reasoning is not available in current straightforward, entity-focused prompts (see section 6). \n\nThe KITTEN benchmark checks given the entity name if the generated image visually matches the prompt which takes the form of modifier words+entity name, e.g. \"Photo of Bandinelli Palace.\", \"An oil painting of Bandinelli Palace.\", \"Bandinelli Palace made of crystal.\"\n\n3. auto metrics does not match human evaluation\n\nThe paper relies on CLIP, DINO, and MLLM-based metrics (GPT-4o, Qwen-2.5) to approximate human evaluation, but these metrics sometimes disagree with human rankings or only show moderate correlation with human evaluation. In Table 3, correlation of automatic and MLLM metrics (GPT-4o, Qwen-2.5, DINO, CLIP-I) with human evaluation shows only GPT-4o is moderately correlated with humans, with a pearson correlation of around 0.7, while the others are only around 0.5, and CLIP is particularly weak at around 0.2.\n\n4. Not provide a robust and open automatic metric\n\nThe strongest correlation is GPT-4o which is proprietary. The proprietary metric blocks reproducibility and may change over time. As a result, KITTEN does not provide a robust and open automatic metric can replace human annotation. \n\n5. Human evaluation still requires reliability analysis\n\nWe don’t know how consistent the ratings are as the agreement statistics (such as Variance or confidence intervals across raters, Cohen's kappa and Krippendorff's alpha for inter-rater reliability) are not reported.\n\nFor example, a moderate final average score may hide a strong disagreement, so the variance is needed to show how widely opinions differ.", "claims_and_evidence_yn": "No", "claims_explanation": "See Weaknesses 3. auto metrics does not match human evaluation, 4. Not provide a robust and open automatic metric, 5. Human evaluation still requires reliability analysis above", "audience_yn": "Yes", "audience_explanation": "The paper proposes a new benchmark for knowledge-intensive text-to-image generation and evaluates multiple models to give insights about visual-world knowledge and evaluation metrics. These topics are relevant to readers working on benchmarking, generative models and multimodal learning.", "requested_changes": "See Weaknesses 1 to 5", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)