An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models

Authors: Mohammad Jalali, Azim Ospanov, Amin Gohari, Farzan Farnia
Primary area: generative models
Keywords: Information measures, diversity evaluation, conditional generative models

Abstract:
Text-conditioned generation models are commonly evaluated based on the quality of the generated data and its alignment with the input text prompt. On the other hand, several applications of prompt-based generative models require sufficient diversity in the generated data to ensure the models' capability of generating image and video samples possessing a variety of features. However, the existing diversity metrics are designed for unconditional generative models, and thus cannot distinguish the diversity arising from variations in text prompts and that contributed by the generative model itself. In this work, our goal is to quantify the prompt-induced and model-induced diversity in samples generated by prompt-based models. Specifically, we propose the application of matrix-based information measures to address this task, decomposing the kernel-based entropy $H(X)$ of generated data $X$ into the sum of conditional entropy $H(X|T)$, given text variable $T$, and mutual information $I(X; T)$. We show that this information-theoretic approach decomposes the existing Vendi diversity score defined based on $H(X)$ into the product of the following two terms: 1) Conditional-Vendi score based on $H(X|T)$ to quantify the model-induced diversity, and 2) Information-Vendi score based on $I(X; T)$ to measure the statistical relevance between $X$ and prompt $T$. Our theoretical results provide an interpretation for this diversity quantification and show that the Conditional-Vendi score aggregates the Vendi scores within the modes of a mixture prompt distribution. We conduct several numerical experiments to show the correlation between the Conditional-Vendi score and the internal diversity of text-conditioned generative models.

Decision: reject
Reviewer scores: 8, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730362246401

thread_type:
official_review

<thread_object>
creator: reviewer_f3gh

<content>
summary:
This paper focuses on distinguishing different sources for diversity evaluation of prompt-based generative models, i.e. prompt-induced diversity and model-induced diversity. Specifically, this paper proposed a new diversity evaluation metric called *condtional-Vendi* score, which is an extension of *Vendi*, but focuses on model-induced diversity rather than overall diversity. Combining with another proposed *information-Vendi* score, the *Vendi* score can be recovered by a multiplication. Experiments on image and text generation tasks show that condtional-Vendi is correlated with model-induced diversity.

strengths:
* The idea is sound to separate different sources of diversity, i.e., prompt-induced and model-induced.
* The proposed diversity metric is reasonable for model-induced diversity, and have the advantage that do not need repeated sampling from a single prompt.

weaknesses:
The main weakness of this paper is its contribution. The proposed condtional-Vendi and information-Vendi score are just exponentials of the conditional entropy and mutual information defined by Sanchez Giraldo et al. (line 241, 246), and the calculation method seems to directly inherit from Vendi proposed by Friedman & Dieng (line 317). The combination is straightforward and I did not find any new challenges that is addressed by this paper. As a result, I view this paper as just an incremental work, and not enough for acceptance.

questions:
1. What's new of your conditional-Vendi compared with the conditional entropy proposed by Sanchez Giraldo et al., besides the exponential?
2. Follow which equation can the conditional-Vendi score be calculated? It's better to write an explicit algorithm, perhaps in the appendix.
3. What do you want to convey by Section 5 (the statistical interpretations)? Is it discussing the implementations, or just broader discussions? A better leading sentence may be necessary in the beginning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517889396

thread_type:
official_review

<thread_object>
creator: reviewer_uq1z

<content>
summary:
This paper focuses on the diversity in prompt-based generative models. Specifically, the authors want to quantify the prompt-induced and model-induced diversity in samples separately. An information-theoretic approach for internal diversity quantification based on kernel-based conditional entropy of the generated data is discussed, where the Conditional-Vendi score and the Information-Vendi score are applied to measure the statistical relevance between the generated data and text prompts.

strengths:
Originality: the problem of diversity quantification is indeed interesting and useful in generative models. 

Clarity: the paper is well-organized and straightforward to follow, with supportive numerical results and clearly stated details.

Overall the paper is a nice reading.

weaknesses:
My major concern is that the contribution of this work is incremental. As stated by the paper in Section 3.2, the conditional entropy $H_\alpha(X|T)$ is already developed by Giraldo et al (2014), while the Conditional-Vendi score proposed by the authors is $\exp\left(H_\alpha(X|T)\right)$, which is essentially the same. In addition, existing literature has considered Renyi entropy of the probability model as the model's unconditional diversity score, and the idea of constructing conditional diversity score based on these results does not compose of sufficient contribution. Thus, I feel the contribution is incremental.

Another concern I have is the potential applications of the proposed diversity measure. In most scenarios mentioned by the paper, say statistical interpretation in Section 5 and some motivating examples, prompts can be easily clustered, so what about just calculate the unconditional entropy for each cluster of prompts, and then use some weighted average to combine them? I personally feel this is a naive but reasonable way to measure the conditional diversity of the model and should serve as a baseline. If prompts cannot be easily clustered and come from some complex distributions, then mathematically the conditional entropy would depend heavily on the distribution of the prompt distribution, the the concrete meaning of this measure is unclear to me. 

For numerical results, though they are clear to me, after reading it I can still not tell if conditional entropy is a good measure, as there is no baseline compared (which is reasonable as this is a relatively new problem, but the authors should at least consider some naive methods such as the one mentioned above, to see if there are some scenarios that their proposed method works while naive methods don't) and the meaning of values reported (say, figure 5-7) remains unclear to me. Why can these figures and values justify that the conditional entropy is a good measure of model's diversity?

questions:
(Contribution) As mentioned in Weakness, could the authors explain their contributions to prior works such as Giraldo et al (2014)?

 (Applications) For prompts that can be easily clustered, why don't we just average unconditional entropy for each cluster? For more general distribution, what's the practical meaning of the conditional entropy when the distribution $\mathcal{T}$ changes?
 (Numerical) Why can current numerical results illustrate the advantages of proposed metric?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536630134

thread_type:
official_review

<thread_object>
creator: reviewer_awod

<content>
summary:
The paper introduces an information-theoretic approach to evaluate the diversity of prompt-based generative models, focusing on decomposing the diversity into prompt-induced and model-induced components. The authors achieve this by decomposing the Vendi score into the Conditional-Vendi score, which measures internal diversity not related to prompt variations, and the Information-Vendi score, which assesses the statistical correlation between generated outputs and their corresponding prompts. The authors validate their methodology through various experiments on text-to-image, text-to-video, and image-captioning models.

strengths:
- The motivation for decomposing the diversity of model generation is clear, despite that a more fine-grained structural analysis would be preferred.
- The theoretical interpretation of the scores provides sufficient insight for the decomposition. 
- The paper is well-organized and logically fluent.

weaknesses:
While this paper tackles an important problem, the following weaknesses prevent it from being accepted.

- First of all, I think the paper lacks novelty in terms of the proposed method. The concept of conditional entropy $H_\alpha$ is nothing new, and the proposed conditional-Vendi score is simply its exponential version. If the paper is not contributing through new concepts or definitions, it should at least demonstrate some (new) important properties or use cases. Unfortunately, the only property discussed in the paper is its relevant stability when more diverse prompts are used, which, theoretically speaking, is expected. In such a case, the only novelty I can admit is that the paper demonstrates in experiments that the conditional entropy behaves normally in generative models. The authors are expected to provide a more comprehensive analysis of the given entropy and demonstrate a wider range of use cases if the concept itself is inherited from existing literature.
- Even if we consider only the stability property, the range of experiments in the paper is still limited. All experiments are simply measuring the scores with varying amounts of input clusters (groups), and the way that the authors make prompts diverse is also rigid. It is entirely unclear, based on existing experiments, that the proposed score can serve as an effective diversity measurement across a wide range of text prompts types. What's more, the authors do not show any support that the measure is accurate, except for this extremely intuitive "increasing curve". No golden metric, or even any proxy metric, has been introduced for comparison. This means that the "diversity measurement" claim is not convincing in the first place.
- Theoretically speaking, a large portion of the discussion about the selection of kernels is missing. From my understanding, since the authors only measure diversity from the generated data (rather than the underlying softmax distribution), the similarity metric used for text and images is very important and can largely change the metric. It is, therefore, necessary for the authors to carefully analyze the choice of kernels and make a more well-studied claim w.r.t. the proposed metric. However, notice that addressing this bullet is insufficient for making the paper reach the acceptance bar unless the first two weaknesses can be addressed.

questions:
No additional question other than the major weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730821064649

thread_type:
official_review

<thread_object>
creator: reviewer_3epu

<content>
summary:
This paper proposes a metric for evaluating the diversity of generative models. Unlike previous approaches which focus on unconditional generations, this paper uses an information-theoretic decomposition of the entropy into conditional entropy (measuring variation in generation when conditioned on a prompt) and mutual information (measuring the remaining variation not explained by the conditional entropy). The first term is expressed as a conditional Vendi score and the second as an information Vendi score. It is shown that the Vendi score is the product of these two, and so generative models with similar Vendi scores can be compared based on their conditional diversity given a prompt. Theoretical justfication for the proposed metrics is provided, and it is also shown how the metric can be estimated from finite samples. The proposed metrics are explored in different contexts, including text-to-image, text-to-video, and image-captioning models. The results correspond with the intuitive interpretations of the scores.

strengths:
Strengths of this paper include:
- The proposed metrics occupy a clearly defined niche in the literature, providing a rich and informative way to evaluate prompt-based generative models. This is useful for better comparisons of generative models (even applicable across different modalities) and for better model development by elucidating the shortcomings of current models.
- Solid theoretical justification and empirical evidence are provided to support the proposed metrics.
- The paper is exceedingly well-written. Prose is clear and mathematical statements are presented in a precise yet intuitive way.

weaknesses:
Weaknesses of this paper include:
- The proposed approach can be viewed as incremental, since it is the combination of two ideas: the Vendi score and matrix-based conditional entropy. However, the execution of the paper is very good and is targeted at a gap in the current literature, so I do not count this as a major weakness.
- The contextualization and interpretation of Theorem 1 could be improved.
- Qualitative results could be improved in some places. For example, showing some of the generations from the models in Figure 4 would help to better clarify the tradeoffs of the models in terms of conditional Vendi vs. information Vendi score.

questions:
My opinion of the work is already high so I do not have many questions. However, in light of Figure 6, it seems that it would be possible to achieve a high conditional Vendi score by a model that has poor alignment with the prompt. Is there any way that the proposed conditional/information Vendi decomposition can handle this case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732528643143

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Authors' General Response

comment:
We thank the reviewers for their thoughtful and detailed feedback on our work. In this general response, we would like to address the raised comments on our work’s technical contribution and experimental results.

**1- Significance of our technical contribution**

Regarding the comments on our work’s technical contribution, we would like to raise the following:

- To the best of our knowledge, our work is the first to study the diversity evaluation of conditional generative models by decomposing the diversity of their generated data into prompt-induced and model-induced terms.

- We note that neither the reference [1] (which proposes a formulation for matrix-based conditional entropy) nor the papers citing it have discussed the application of conditional entropy to generative models. To the best of our knowledge, the connection between the information measures in [1] and conditional generative models has not been explored in the literature. 

- We believe the theoretical results in our work provide insights into the application of *matrix-based* conditional entropy for evaluating the model-induced diversity of conditional generative models. While Sanchez Giraldo et al. [1] introduce a notion termed “conditional entropy” $H(X|T)$, their defined $H(X|T)$ does not correspond to the expectation of prompt-conditioned entropies $H(X|T=t)$, which means
$$ H\bigl( X|T\bigr) \text{  } {\large\neq}  \text{  } \mathbb{E}_{t\sim P_T}\bigl[ H\bigl( X|T=t \bigr)\bigr] $$
Therefore, this **matrix-based** conditional entropy lacks the standard interpretation of Shannon conditional entropy, which is well-known to represent the average of instance-specific entropies, $H(X|T=t)$. Our work addresses this gap by establishing a connection between [1]’s definition of $H(X|T)$ and the averaged (unconditional) entropy over text prompts. Under *minimal assumptions*, Theorem 1 shows that this definition of $H(X|T)$ corresponds to an aggregation of the (unconditional) entropy $H(X|C=i)$ *within each component $C=i$ of a mixture prompt distribution $P_T$*.



**2- Golden metric for assessing the Conditional-Vendi score**

First, we would like to clarify that, to the best of our knowledge, the proposed Conditional-Vendi score is the first metric to evaluate the model-induced diversity of conditional generative models. In our numerical evaluation, we aimed to demonstrate the metric’s correlation with the ground-truth model-induced diversity of the models in scenarios where the ground-truth ranking of diversity is known across the tested generative models. This approach was necessary, because there are no existing metrics in the literature that could serve as a baseline for assessing the correlation with the Conditional-Vendi score.

Following Reviewer Uq1Z’s suggestion, in the revision, we have defined a baseline score as a “golden metric,” detailed in Appendix B.5 of the revised manuscript. This metric, which we refer to as the “GroundTruth-Cluster-Vendi,” assumes access to the underlying clusters of the generated samples and computes the average Vendi score within these sample clusters. As shown in Figure 22, the GroundTruth-Cluster-Vendi metric correlates well with the Conditional-Vendi score across the nine sets of numerical experiments conducted, where the ground-truth clusters were known (we provide a detailed explanation of the experimental setup in Item 3 below).

**3- Additional numerical support for the proposed diversity evaluation method** 

In the revision, we have conducted 9 additional sets of numerical experiments to further evaluate the application of the Conditional-Vendi score in quantifying the model-induced diversity of conditional generative models. Specifically, we considered three SOTA text-to-image models: 1) Stable Diffusion XL, 2) Kandinsky 2.1, and 3) PixArt-$\Sigma$, along with three categories of text prompts: 1) Animals, 2) Fruits, and 3) Objects, resulting in a total of 9 different combinations.

For each of these 9 settings, we selected 10 different types within the category and generated images for 5000 prompts corresponding to these types. We then evaluated the Conditional-Vendi scores under two conditions: in the first, we used the original prompts; in the second, we replaced the specific category type (e.g., “apple”) with the general category name (e.g., “fruit”) in the prompt. 

As shown in Figures 12–20, the Conditional-Vendi score increases consistently as more types within each category are introduced. Moreover, the score grows *significantly less rapidly* when the category type is explicitly specified in the prompt. These results indicate that the Conditional-Vendi score effectively mitigates the influence of prompt diversity, as desired, and primarily reflects the model-induced diversity of the generative model.

[1] Sanchez Giraldo et al., “Measures of entropy from data using infinitely divisible kernels”. IEEE Transactions on Information Theory, 2014
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents an information-theoretic approach to evaluate the diversity of prompt-based content generation. The proposed evaluation score defined in Section 4 consists of two parts: Conditional-Vendi and Information-Vendi. A statistical interpretation of the proposed score is presented in Section 5 (Theorem 1). The main criticism lies in the contribution. 3 out of the 4 reviewers think the proposed method is incremental at most and the proposed metric is derived from a simple transformation of existing ones. The reviewers also raise some questions on the numerical experiment design.