Kitten: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities

Authors: Hsin-Ping Huang, Xinyi Wang, Yonatan Bitton, Hagai Taitelbaum, Gaurav Singh Tomar, Ming-Wei Chang, Xuhui Jia, Kelvin C.K. Chan, Hexiang Hu, Yu-Chuan Su, Ming-Hsuan Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-to-image generation

Abstract:
Recent advancements in text-to-image generation have significantly enhanced the quality of synthesized images. Despite this progress, evaluations predominantly focus on aesthetic appeal or alignment with text prompts. Consequently, there is limited understanding of whether these models can accurately represent a wide variety of realistic visual entities — a task requiring real-world knowledge. To address this gap, we propose a benchmark focused on evaluating Knowledge-InTensive image generaTion on real-world ENtities (i.e., KITTEN). Using KITTEN, we conduct a systematic study on the fidelity of entities in text-to-image generation models, focusing on their ability to generate a wide range of real-world visual entities, such as landmark buildings, aircraft, plants, and animals. We evaluate the latest text-to-image models and retrieval-augmented customization models using both automatic metrics and carefully-designed human evaluations, with an emphasis on the fidelity of entities in the generated images. Our findings reveal that even the most advanced text-to-image models often fail to generate entities with accurate visual details. Although retrieval-augmented models can enhance the fidelity of entity by incorporating reference images during testing, they often over-rely on these references and struggle to produce novel configurations of the entity as requested in creative text prompts.

Decision: withdrawn
Reviewer scores: 8, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730543649148

thread_type:
official_review

<thread_object>
creator: reviewer_xfym

<content>
summary:
This paper introduces a novel evaluation method to assess image generation models' capability to generate real-world visual entities. The authors propose a benchmark called Kitten and develop both human and automatic evaluation methods to test various image generation models. Through extensive analysis, the author gives a key insight that sota t2i models liake SD, flux exhibit lower faithfulness to the entity, while  retrieval-augmented models show  great advantage.

strengths:
1. The concept of evaluating text-to-image (t2i) models' ability to generate real-world visual entities is intriguing and crucial for progress in this area.
2. The proposed dataset and evaluation criteria offer a valuable resource for the community, enabling more detailed and nuanced evaluations of t2i models.

weaknesses:
1. The paper's novelty is somewhat limited. The insights that retrieval-augmented models show advantage on faithfulness to the entity is understandable, however the finding that they lack instruction-following capabilities is more interesting, have the authors considered exploring methods to improve entity fidelity while maintaining instruction-following capabilities？
2. The dataset covers 8 different domains, Did the authors observe any significant performance variations across different entity domains (e.g., landmarks vs. insects)? If so, what might explain these differences?

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554082377

thread_type:
official_review

<thread_object>
creator: reviewer_qejs

<content>
summary:
This paper proposed a novel and significant issue: real-world knowledge in image generation. Focusing on the factual accuracy with which models generate real-world entities, the authors have developed a benchmark based on a diverse set of knowledge entities to systematically evaluate model performance in the domain of real-world knowledge. To more accurately assess the factuality of the images generated by these models, the authors have meticulously designed a set of human evaluation criteria. They conducted a comparative analysis with widely-used automatic metrics, highlighting the deficiencies of these automatic metrics and demonstrating the superior accuracy of human evaluations. The authors evaluated a variety of text-to-image models and retrieval-augmented customization models, uncovering significant deficiencies in the visual representation of world knowledge within these models.

strengths:
1. The paper presents a novel problem in the field of image generation, focusing on the factuality of real-world knowledge in image generation, which is rarely explored in the existing literature.
2. The results are clearly presented, with sufficient tables and visual images to illustrate the findings.
3. The paper is well-written, with a clear and logical organization.

weaknesses:
1. Insufficient Experimental Setup: The current experiments are limited to real-world entities in only 8 domains and 322 knowledge entities, which lacks diversity. This number is significantly less than the variety of entities present in the real world and does not comprehensively reflect the model's performance, potentially leading to data bias. The OntoNotes and WordNet datasets include a more diverse range of categories and entities. If you only consider your 8 domains and limited number of entities, might this affect the generalizability of the results?
2. High Costs: The paper proposes a manual evaluation method, but conducting a single evaluation using this benchmark requires more than 6,000 human assessments, resulting in significant costs.
3. Limited Prompt Types: The paper includes only four types of prompts. It's unclear whether there is a theoretical basis for selecting these four types, and whether high scores in these prompts can fully reflect the model's true understanding of the knowledge entities.
4. Lack of Comparison with Other Benchmarks: The paper does not provide a clear comparison with recent benchmarks in the field, making it difficult to understand the similarities and differences between this benchmark and others (eg. HELM, GenAI-Bench and ConceptMix)

questions:
1.  Is there a theoretical basis for choosing the specific four types of prompts? How were these types determined, and do they align with the objectives of your study?
2. Is there a rationale or empirical evidence supporting this selection as being representative of real-world scenarios?
3. Have the authors discussed the trade-offs between manual and automated evaluation methods in terms of accuracy and cost? Are there any strategies for expanding the evaluation dataset in the future while reducing costs and maintaining quality?
4. Could you please explain in detail how you selected these four types of prompts and the rationale behind your choices? Can you argue for the extent to which these prompt types cover important scenarios involving knowledge entities? Have you considered adding other types of prompts？
5. How does your benchmark compare with existing benchmarks in terms of design goals and evaluation metrics? Are there particular aspects where you believe your benchmark offers distinct advantages?
6. What steps have been taken to mitigate potential biases resulting from the limited diversity of entities and prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685385626

thread_type:
official_review

<thread_object>
creator: reviewer_a7bf

<content>
summary:
The paper introduces KITTEN, a benchmark specifically designed to evaluate the accuracy of text-to-image generation models in terms of real-world entities, such as landmarks, plants, and animals. on this benchmark, the authors evaluate multiple existing approaches, including both standard text-to-image models and retrieval-augmented models that leverage reference images.  the findings include 1) that even the most advanced text-to-image models often fail to generate entities with accurate visual details, 2) retrieval-based method often over-rely on these references and struggle to produce novel configurations of the entity as requested in creative text prompts.

strengths:
the focus of the paper is interesting, which is to evaluate the text-to-image model in terms of generating real-world entities with some modifications. the benchmark would also be beneficial to the community. the evaluation is based on two aspects. one is the faithfulness to the entity, and the other is the prompt following as the prompt may contain some changes to the entity. the metric is sound and reasonable. beyond, the authors perform a very comprehensive study on how the existing approaches work on these new benchmarks.

weaknesses:
as the main contribution is the benchmark, it may be better to provide more details on how the benchmark is collected. throughout the paper, i only find the information that the benchmark is collected from 8 categories and Table 7 in supl shows the number of prompts in each. it might be recommended to share more details, e.g. how these categories are collected, why it is these 8 categories, is it diverse enough, how the prompt is collected, who collected the prompts, what the prompt length and variance is. 

the paper performs lots of evaluation on the existing text-to-image approaches and the retrieval-based approaches. one important metric is the human evaluation, but it might be better to share more details on the human eval, e.g. how many annotators are there for each task, how the variance is, how reliable the evaluation is. 

[minor] in Fig 1, the results are all from proprietary company, e.g. adobe firefly, dalle. but in the experiments, all results are based on other approaches, e.g. flux model. there is no overlap here. it might be better to make it consistent. 
[minor] the Flux has multiple versions (https://github.com/black-forest-labs/flux), is it dev or pro?

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703575284

thread_type:
official_review

<thread_object>
creator: reviewer_9vtx

<content>
summary:
The paper introduces a benchmark dataset, Kitten, designed to evaluate the capability of text-to-image models in generating visuals with precise entity representations across eight distinct visual domains. For each chosen entity, the authors crafted four unique input prompt types, alongside corresponding support and evaluation image sets for each entity.

The paper incorporates human evaluations focusing on entity fidelity, with two primary criteria: (1) the model’s ability to follow instructions accurately, and (2) the faithfulness of generated visuals to the specified entity. Furthermore, the authors employ automatic metrics, including the CLIP-T score and cosine similarity within DINO's feature space, to assess alignment between generated and target images.

State-of-the-art models, such as Flux, Stable Diffusion (SD), Imagen-3, Dreambooth, and Instruct-Imagen, were tested on the dataset. Experimental results highlight the main challengings of backbone models as well as retrieval-augmented customized models. The results suggest future topics, such as enhancing entity fidelity without compromising the instruction-following capabilities of these models.

strengths:
- This paper tackles a unique and underexplored aspect in the text-to-image generation community --evaluating the fidelity of specific entities in generated images. The motivation and evaluation framework are well-grounded.
- Utilizing Wikipedia as a knowledge base is a clever choice, ensuring broad coverage and relevance of entities. The idea makes the entity coverage more scalable for future extension.
- The proposed evaluation approach, incorporating both human and automatic metrics, strengthens the robustness of the study’s findings.
- The paper offers valuable insights and discussions that can guide future work aimed at improving fidelity in text-to-image models.
- The paper is well-written and easy to understand.

weaknesses:
- While the study suggests a direction for future work aimed at balancing entity fidelity with creative flexibility, the notion remains somewhat ambiguous and challenging to envision in practice. Achieving an optimal outcome where both precise entity representation and creative interpretation coexist is complex, and the paper could benefit from a clearer exploration or concrete examples of what such a balance might look like in generated images.
- The paper suggests that the optimal image generation model should balance entity fidelity with creative flexibility, but it remains unclear whether such coexistence is achievable. If it proves difficult or even impossible to represent both precise entity fidelity and creative flexibility simultaneously, this limitation could reduce the paper’s impact. The study would benefit from a deeper investigation into whether these two qualities can genuinely coexist, as this would provide clearer guidance for future research directions and enhance the paper’s practical relevance.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732246602468

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their valuable comments. We are pleased that the reviewers find our problem interesting (Reviewer a7bf) and underexplored (Reviewers 9Vtx, qEJS), our study comprehensive (Reviewer a7bf), and offering valuable insights (Reviewer 9Vtx). Our benchmark is recognized as valuable to the community (Reviewers a7bf, xfyM), well-grounded, and scalable for future extensions (Reviewer 9Vtx). The paper is also noted as well-written (Reviewers 9Vtx, qEJS).

We encourage all reviewers to refer to our supplementary PDF, which includes our exploration of balancing instruction-following and entity fidelity, as well as potential future research directions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a benchmark to evaluate the generation fidelity of real-world entities (e.g. landmarks, or animals) and presents an analysis comparing state-of-the-art models on this task. The paper was reviewed by four experts in the field. The reviewers acknowledged that the paper is well written and easy to follow (qEJS), its focus is interesting (a7bf, qEJS, xfyM), and its motivation well grounded (9Vtx). Some reviewers appreciated that the benchmark includes both human and automatic metrics (9Vtx) which may help obtain more nuanced evaluations (xfyM). The findings are informative (9Vtx), and the metrics appear sound (a7bf).

The main concerns raised by the reviewers were:
1. Missing concrete examples of the balance between entity fidelity and instruction-following (9Vtx)
2. Missing additional details on how the benchmark is collected and justifications for choices made (a7bf)
3. Missing information related to the human evaluations - e.g. how many annotations per task, variance, reliability of the results (a7bf)
4. Experimental setup appears insufficient, with limited diversity (qEJS), and limited ablations
5. The novelty appear rather limited (xfyM)
5. The proposed evaluation methodology appears highly costly (qEJS)
6. Missing comparisons with existing benchmarks (qEJS)

During rebuttal and discussion, the authors provided examples to illustrate what is requested by the reviewers, shared details on the benchmark creation process and the human evaluations, attempted to justify some of their choices, stratified the performance of the evaluated models into different types of prompts, and argued for the novelty of the paper and its contribution. During discussion, most reviewers were unresponsive despite the numerous intents of the AC to incentivize discussion with authors and among reviewers. Therefore, the AC went over the paper and rebuttal materials in detail, and concluded that the reviewers' concerns were partially addressed. In particular, concerns related to novelty, potential impact of the proposed benchmark, and quality of execution remain unconvincing. For those reasons, the AC leans towards rejection. The AC encourages the authors to consider the feedback received to improve future iterations of their work. Moreover, to fully assess the typical evaluation dimensions of text-to-image models, the authors should consider including a measure of conditional diversity in their analyses.