GRADE: Quantifying Sample Diversity in Text-to-Image Models

Authors: Royi Rassin, Aviv Slobodkin, Shauli Ravfogel, Yanai Elazar, Yoav Goldberg
Primary area: generative models
Keywords: sample diversity, text-to-image, diffusion, evaluation

Abstract:
Text-to-image (T2I) models are remarkable at generating realistic images based on textual descriptions. However, textual prompts are inherently *underspecified*: they do not specify all possible attributes of the required image. This raises key questions: do T2I models generate diverse outputs on typical underspecified prompts? How can we automatically measure diversity? We propose **GRADE**: **Gr**anular **A**ttribute **D**iversity **E**valuation, an automatic method for quantifying sample diversity. GRADE leverages the world knowledge embedded in large language models and visual question-answering systems to identify relevant concept-specific axes of diversity (e.g., ''shape'' and ''color'' for the concept ''cookie''). It then estimates attribute distributions and quantifies diversity using (normalized) entropy. GRADE achieves over 90\% human agreement while exhibiting weak correlation to commonly used diversity metrics. We use GRADE to measure the overall diversity of 12 T2I models using 400 concept-attribute pairs, revealing that even the most diverse models display limited variation. Further, we find these models often exhibit *default behaviors*, a situation where the model consistently generates concepts with the same attributes (e.g., 98\% of the cookies are round). Finally, we demonstrate that a key reason for low diversity is due to underspecified captions in training data.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729995015620

thread_type:
official_review

<thread_object>
creator: reviewer_tbny

<content>
summary:
This paper introduces GRADE (Granular Attribute Diversity Evaluation), a new evaluation approach for quantifying sample diversity in T2I models. GRADE addresses the limitations of existing diversity metrics by focusing more on concept-specific attributes such as shape, color or texture, without replying on reference images. The authors perform comprehensive experiments on 12 T2I models, showing consistent default behaviors and limited diversity across models. They attribute this lack of diversity to underspecified captions in training data, which drive models to favor common attributes by default.

strengths:
- This paper is well-written, well-motivated, and well-organized. 
- A key strength is that they provide a clear and structured definition of the diversity metric and the systematic approach in Sec. 3. The method estimates distributions with entropy, and allows for a granular and concept-specific evaluation of the diversity. 
- The paper links low diversity in generated images to underspecified captions in training data, suggesting that this lack of diversity might come from biases in the data itself. This takeaway on data quality could help improve diversity by addressing such limitations in the training datasets.

weaknesses:
- Since GPT-4o is used to generate prompts, attributes, and attribute values, its lack of version control could affect the consistency of the GRADE framework as GPT-4o evolves. This may alter diversity analysis results, making reproducibility difficult. Changes in GPT-4o’s outputs could lead to inconsistencies when comparing diversity evaluations of T2I models evaluated at different points in time.
- The definition of diversity in this work is focused on specific attribute variations (like shape, color, etc.) and may miss other important aspects of image diversity. The approach relies heavily on the capability LLMs, which could have their own limitations and biases in terms of diversity. 
- There's no limitation discussion for example how GRADE might handle more complex or abstract concepts where attributes are less clearly defined. Another part that would be good to include in the future version of the paper is the compositional diversity or relationships between multiple concepts in the same image.

questions:
For the concept-specific attributes, such as shape, color, and texture, is there a predefined set of attributes (other than shape/color/texture), or is it flexible? What range of different attributes is considered, and how consistent is this range across various concepts? Additionally, if you rely on LLMs to automatically generate these attributes, could this introduce new biases into the evaluation process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730473404899

thread_type:
official_review

<thread_object>
creator: reviewer_vebf

<content>
summary:
This work aims at tackling the open problem of evaluating diversity of text-to-image generative models. The authors propose GRADE, an approach that has two main components: measuring diversity of specific concepts with respect to attributes selected by a language model, and using as a metric the normalized entropy of the distribution of answers given by visual-question answering (VQA) model as metric for diversity. GRADE’s main components were evaluated by human inspection and the efficacy of GPT-4 as a VQA model was assessed via human evaluation. 12 different text-to-image models were compared in light of the introduced metric which showed that most of them perform similarly in terms of diversity (as measured with GRADE).

strengths:
- S1: This work tackles the extremely relevant open research question of evaluating diversity of text-to-images.

&NewLine;

- S2: The proposed approach, GRADE, proposes to measure diversity in terms of images of specific concepts with respect to relevant factors of variation.

&NewLine;

- S3: GRADE is a reference-free metric and doesn’t rely on training any new models to be computed.

weaknesses:
- W1: The proposed metric does not seem to be effective at discriminating between models. It is not clear what is the statistical significance in the difference of scores reported in Table 4. Given the average and standard deviation of GRADE, it seems that confidence intervals for all models would overlap. 

&NewLine;


  - W1.1: There is no rigorous validation of the proposed metric and given that it seems GRADE is not able to properly distinguish 12 generative models, it is unclear to me whether the metric is actually capturing diversity as claimed in the manuscript. Experiments considering images with known ground-truth for diversity should be included in order to show that the proposed metric is in fact performing as expected.

&NewLine;


- W2: The authors claim in the conclusion that “Our experiments demonstrate that GRADE is in agreement with human judgements of diversity”, however there is no evidence in the manuscript to support such a strong claim. The human evaluation performed in this work only takes into account the accuracy of GPT-4 on VQA tasks, and it is not even remotely evaluating diversity of generated images. In fact, a major weakness of this submission is that there is no clear evidence that the proposed metric is capturing diversity as perceived by humans.

&NewLine;


- W3: The metric relies on how well the VQA model is able to capture nuances in the differences across different values an attribute can assume. For example, if the diversity of sky images is being measured with respect to the color attribute, it is not clear whether dark blue and navy would be considered as different colors by the VQA model.

&NewLine;


  - W3.1: It is unclear what is the interplay between GRADE and other aspects such as alignment between prompt and generated image, and realism. What happens if the main concept is not actually present in the generated image and/or the generated objects do not look realistic? Would the VQA model be able to answer the question correctly?
&NewLine;


- W4: It is not clear how the approach validity as evaluated in Sections 4-a,b,c  would generalise if other concepts/aspects were taken into account, therefore making it unclear how GRADE performs in case diversity with respect to concepts and attributes of different nature such as people,  ethnicity, skin tone, gender, etc.

&NewLine;



- W5: The manuscript lacks clarity in many parts. Please refer to the "Questions" section of this review for more details. 

&NewLine;

 
- W6: There are several missing references in the text and very relevant related work has not been cited by the authors:
  - Diversity metrics:
     - Friedman, Dan, and Adji Bousso Dieng. "The vendi score: A diversity evaluation metric for machine learning." arXiv preprint arXiv:2210.02410 (2022). 
     - Pasarkar, Amey P., and Adji Bousso Dieng. "Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning." arXiv preprint arXiv:2310.12952 (2023).
     - Alaa, Ahmed, et al. "How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models." International Conference on Machine Learning. PMLR, 2022.
   - VQA-based approaches to evaluate generative models:
     - Wiles, Olivia, et al. "Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings." arXiv preprint arXiv:2404.16820 (2024).
     - Cho, Jaemin, et al. "Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation." arXiv preprint arXiv:2310.18235 (2023).
     - Lin, Zhiqiu, et al. "Evaluating text-to-visual generation with image-to-text generation." European Conference on Computer Vision. Springer, Cham, 2025.

questions:
- Q1: GRADE assigns maximum diversity to cases where all values of an attribute are equally prevalent in the evaluated images. However, roughly, text-to-image models are usually trained to maximize the likelihood of the "real-world" distribution of natural images. In this case, shouldn't one expect that generated images will reflect the distribution of natural images? If so, does it still make sense to expect that the probability of generating a start-shaped cookie should be equal to the probability of generating a round cookie?

&NewLine;

- Q2: GRADE depends on measuring diversity with respect to specific concepts and attributes, but how many concepts and attributes are "enough" to make sure GRADE is an accurate estimate of diversity?

&NewLine;

- Q3: I am confused by  the equation in lines 156 and 157. Does $p_c^i$ correspond to a single prompt (as indicated in the definition of $\mathcal{P}_c$) or a set of prompts (as it seems to be the case given the sum is over $i$)?   

&NewLine;

- Q4: Can the authors kindly provide examples of prompt and concept distributions in Section 3-d? I couldn't understand what exactly they corresponded to even after reading the text a few times. 

&NewLine;

- Q5: (Related to W1) Please perform statistical significance tests between model scores and report confidence intervals.

&NewLine;

- Q6: (Related to W3.1) A suggestion to elucidate this point would be assessing GRADE's performance on misaligned or unrealistic images, and checking how/if the metric's reliability was affected.

 
- Minor comments:
  - Typo: 
    - Line: 309: this step **a** second time
  - Equations are missing their respective numbers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730551800955

thread_type:
official_review

<thread_object>
creator: reviewer_2lwn

<content>
summary:
The paper presents GRADE (Granular Attribute Diversity Evaluation), a novel method for quantifying the diversity of images generated by text-to-image (T2I) models, which often produce limited variations due to underspecified prompts. Unlike traditional metrics like FID, GRADE uses large language models and visual question-answering systems to identify and measure concept-specific attributes, calculating diversity through normalized entropy. The study finds that even the most advanced T2I models frequently exhibit default behaviors, such as generating the same attributes regardless of context, and highlights that biases in training data contribute to these limitations. The authors call for improvements in training data and propose GRADE as a more fine-grained, interpretable diversity metric to advance generative modeling.

strengths:
The strength of this paper is introducing the fine-grained and interpretable metric that overcomes the limitations of traditional diversity metrics by quantifying concept-specific attribute diversity without relying on reference images. It provides deeper insights into text-to-image model behavior and highlights the impact of training data biases, offering a clear path for advancing generative model evaluation and diversity.

weaknesses:
1. Generally, there is a common perception that "cookies" are round, and I share this view. If a square cookie were requested but a round cookie was generated, that would indeed be an issue. However, the problem highlighted by the authors is not about such cases but rather challenges the generalization itself. I find it difficult to relate why generating results that align with common sense is problematic. In other words, it seems the authors are raising an issue with what is an expected outcome.

2. The method uses an LLM to generate questions based on the given prompt, relying heavily on the performance of the LLM. For example, if the prompt involves a cookie, the model generates questions about the shape of the cookie. However, just as many VQA papers specify categories such as shape, attribute, and color, shouldn't there also be a strict specification of categories in this case? If a prompt where shape is crucial fails to address this aspect due to limitations or hallucinations of the LLM, wouldn't that pose a problem?

questions:
1. The distinction between common and uncommon prompts lacks sufficient justification. For instance, in Figure 1, cookies are categorized based on familiar and unfamiliar backgrounds. However, do you think the shape of a cookie should change just because the background is uncommon? Isn’t it more concerning if the object changes rather than the background?

2. How do the authors recognize and address hallucinations when using LLMs and VQA?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731708076590

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,


We sincerely thank you for your thoughtful and detailed reviews of our manuscript. Your insightful feedback has been invaluable in helping us refine and improve our work.
We have carefully considered all your comments and have made revisions to address your concerns and suggestions. The main changes include:

**Clarifications to the manuscript:** 

* Reviewers 2Lwn and veBF sought clarifications about the intuition of measuring diversity independent of the distribution of the training data. We have updated the manuscript in Introduction.
* Reviewer 2Lwn commented that the distinction between uncommon and common prompts is unclear. We have clarified the text accordingly.
* Reviewer veBF pointed out we do not show GRADE is in agreement with human judgements of diversity, but that humans agree GRADE is accurate. We revise the text to reflect that.
* All three reviewers noted there is no discussion on the limitations of GRADE, which we have now added as a section before Conclusion.

**Statistical significance test:**

Reviewer veBF requested we test for statistical significance based on the scores and standard deviations in Table 4. In Appendix D.2 we detail comprehensive permutation tests and find the results to be significant.


In addition to describing the changes in our responses to each of you, we upload an updated manuscript, with the changes highlighted. We believe these revisions have strengthened the manuscript and have clarified any ambiguities. We kindly invite you to review the revised version and hope that it meets your expectations.


Thank you again for your time and constructive feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the problem of generated image diversity for text to image models. The authors propose to construct a dataset of prompts that asks for objects in common and uncommon scenes. They then use an LLM to generate QA pairs that are answered by a VQA pair to characterize the image diversity on the prompts. The resulting dataset, GRADE, is used to evaluate text to image models. THe key finding in the work is that text to image models consistently generate concepts with same attributes.

Strengths
1. The metric is reference free from images and interpretable.
2. The experiments in this paper look at a variety of different text to image models.

Weaknesses
1. Most modern text to image models, like DALLE-3, SD3, FLUX use an LLM to rewrite the prompt and add details to the text prompt to over-specify object attributes. This technical detail makes a big difference in image quality and diversity and it is unclear if it has been accounted for in the paper.
2. The reporting bias in image captions is a well  reported problem. The authors show that this affects text to image models that are trained on such captions. However, this isn't a surprising claim. 
3. Relying on an automated VQA model for evaluation has its own limitations -- the VQA models themselves are biased or limited and thus the evaluation itself is limited. This hasn't been discussed in detail.
4. GRADE is limited to analyzing the objects and attributes in the prompts. This has been stated by the authors too.

Justification
The paper received weak reviews and none of the reviewers judged it to be exciting. The AC concurs with the reviewers about the weaknesses. In particular, the evaluation in this work is limited and the finding isn't novel.