Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use Vendi score to measure the diversity of datasets.
strength_and_weaknesses: Strength:

1. Paper shows many applications of the Vendi score to measure the diversity of data set, i.e. in generative model, mode collapse, image evaluation, algorithm evaluation etc.
2. The Vendi score requires no label to compute.

Weaknesses:

1. The definition of Vendi score comes straight forward from the Shannon entropy. All of its theoretical results are consequences of results of Shannon entropy.

2. There is no theoretical support for the use of Vendi score in situations presented in the paper. 

3. There is no guideline to explain when to use the Vendi score, i.e in the case of LSUN dataset, should we use the Vendi score to rank the efficiency of models?


To improve the work, the authors could prove that the Vendi score is an indicator for the number of modes. 

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and well-written.
summary_of_the_review: I am concerned about the novelty of the paper, since most of the contributions come from experimental results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose a diversity metric, called Vendi score, that connects and extends ideas from ecology and quantum statistical mechanics to machine learning.  

The Vendi Score doesn’t require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. The authors demonstrate the usage of Vendi score on domains including molecule generation, mode collapse of GANs, and decoding algorithms for texts.   

strength_and_weaknesses: 
### Strengths: 

- The proposed Vendi score serves as an interpretable metric for diversity.  The authors study several interesting properties of it, which gives a formal understanding of desiderata for diversity. 
- The Vendi score enjoys flexibility and wide applicability.
- The Vendi score has the potential to be applied for diversity-informed data augmentation.  


### Weaknesses: 
- Comparison with  classical diversity tools, e.g.., Determintal point processes （DPPs） is lacked. There is a rich literature on it, stemming from statistical physics and various applications in machine learning.  To name a few:  see [Li et al. 2016] and a wonderful survey [Kulesza-Taskar 2012] and the references therein.   

- To make it clearer, if one follow the DPP definition of a diversity score, let us call it the DPP score (DS for short)  following the same notation in the paper, it shall be defined as 
$$ DS_k(x_1,…,x_n) = \log |K| = \sum_i \log n\lambda_i = \sum_i (\log n + \log\lambda_i) $$
It looks pretty related to the Vendi score. 

Meanwhile, DPP score also has an interpretation, it corresponds to the square of the volume spanned by the vectors of $ x_1,…,x_n$. 

Can you comment more on the connections to DPP score? 

### Reference: 

Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286, 2012

Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Efficient sampling for k-determinantal point processes. In Artificial Intelligence and Statistics, pp. 1328–1337. PMLR, 2016.


clarity,_quality,_novelty_and_reproducibility: 
I believe it is the first time to apply classical bio-diversity scores (specifically, the Hill index of order 1) to several applications in machine learning. However, this work ignores a rich class of diversity related research in machine learning.  

summary_of_the_review: 
I appreciate the authors’ efforts in presenting a notion of diversity score motivated by bio-diversity and applying it to various diversity-related applications in machine learning. However, besides the above weakness concerns, I have the following questions for the authors. I believe these questions need to be well-addressed before the work becomes a solid one.  

### Q: Considering applications of DPP-based diversity, does Vendi score allow for efficient sampling, marginalization or learning, which are frequently needed for applications in ML? 

Specifically, this would be possible if you define an energy-based model based on Vendi score, similar as what DPP does. This energy-based model describes the odds that a specific subset $S$ shall be selected out of the ground set $V$. 

$$p(S) \propto VS_k(S) =  \exp{-\sum_i \lambda_i \log\lambda_i},$$

where $S:= \{x_1,…,x_{|S|}  \},  V:={x_1,…,x_{|V|} }, $  (should be the notion of a set, but it does not show up correctly in this openreview system...)

In this way, the partition function (normalization constant) shall be written as: 
$$ Z = \sum_{S\subseteq V} VS_k(S) $$

One clear advantage of DPP is that the partition function of it admits a closed form, which is not possible for many other Markov random field variants. Would the distribution defined by the Vendi score allow for similar kind of benefits?   


### Q: Would it be possible to axiomatize the diversity metrics?  

There could be more than one way to define a diversity score (or even metric mathematically), as we have discussed above.  It is beneficial to think of some axioms that shall be met for a reasonable diversity measure: $D(A)$, where $A = \{x_1, …, x_n\}$ denotes the set of samples. 
For example, the following axioms would come to my mind: 

1) monotonicity:  $D(A) \leq D(A\cup {x})$
2) symmetry as you defined
3) some relationship amongst $D(A)$, $D(B)$, $D(A\cup B)$, $D(A\cap B)$?   
4) defining $D(\emptyset)$? 


### Q:  How to correctly evaluate the validity of a diversity score empirically?    

The authors did extensive experiments on the usage of Vendi score in various application domains. However, it hardly provides strong evidence of the advantage of Vendi score.   Molecule generation experiments (sec. 3.1) shows one specific instance where IntDiv and VS have discrepancy. While IntDiv is a pretty weak baseline which results from a heuristic definition of diversity.   Image generation experiments (sec. 3.3) shows that VS generally agrees with existing metrics on low-resolution images. And on LSUN dataset, VS disagrees with IS.  Experiments on decoding text (sec. 3.4) show that VS agrees with N-gram diversity on a caption generation task with 5 sentences.  Experiments on diagnosing datasets (sec. 3.5) demonstrate that VS could provide diversity scores for samples within a specific class in several public datasets.   

One observation is that there is no, or it is very hard to get “golden” ground truth for diversity scores of a set of samples.  In my opinion, one way could be to ask a group of persons to provide annotations of diversity, either in the form of “scores” or rankings. One practical way to conduct this is to use some crowdsourcing platform.   A second way would be to use existing datasets in a smart way, say the CIFAR-100 dataset. One can make some groups of data samples out of the CIFAR-100, some groups only contain samples from one unique class (maybe called intra-class groups), and some groups have samples from various classes (called inter-class groups). One would naturally expect that the diversity of inter-class groups ranks higher than that of intra-class groups.  One can of course design more refined data groups along this line, to provide more refined way to verify the validity of a diversity score.  



### Q: Naming it as “Vendi score” seems to be not consistent with the literature. 

On the first glance I thought Vendi might be taken from some literature of bio-diversity. However, it turns out to be no after I checked representative papers on bio-diversity since the term “Vendi” has never appeared in the literature. 

The most related concept may be the “Hill number of order 1” (https://en.wikipedia.org/wiki/Diversity_index#cite_note-Tuomisto2010a-4), see also the paper of Hill [Hill 1973]. So it seems to be reasonable to name it as “Hill score”. 

### Reference: 

Hill, Mark O. "Diversity and evenness: a unifying notation and its consequences." Ecology 54, no. 2 (1973): 427-432.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed the Vendi Score (VS) to measure the diversity of the generated samples from generative models. It is defined by the exponential of the Shannon entropy of the eigenvalues of a similarity matrix or covariance matrix using the inner-product similarity function (Sec. 2.3).

This work is reminiscent of spectral clustering, where the eigenvalues of the similarity matrix are used for clustering. Here, many clustering means diverse samples. The author can relate this work to spectral clustering and discuss it.
strength_and_weaknesses: ### Strength

The proposed idea has a compelling theoretical motivation, and it is convenient to use since it only requires the computation of the eigenvalue decomposition for the feature covariance matrix. Through a normalization technique, seeing the distribution of eigenvalues as a probability distribution was notably interesting. The connection to spectral analysis may be worth exploring and discussing. 

### Weakness

**W1. Need for human evaluation.** The major concern is experimental validation. Table 1 does not provide the ground-truth diversity; we only learn that VS may capture the HMM's underperformance. Table 2 indirectly validates the VS by showing the sensitivity toward mode detection, not directly on the degree of diversity. Table 3 only compares with the other metrics, which may have some sense of measuring diversity (e.g., FID through its covariance matrices). It just shows general agreements with the other metrics. We do not know which metric is significantly better than one other for measuring diversity. The same goes for Table 4. Even human-written reference captions do not directly confirm that their diversity is captured by BLEU-4, excluding other factors. One suggestion is to evaluate with human judgments on diversity to compare with other conventional and competing metrics. 

**W2. Requirement of reference datasets due to inconsistency across domains.** Although the authors argue that it "doesn't require a reference dataset" in the Abstract, it shows inconsistent results across domains in Table 3. For instance, Cat or Bedroom datasets have relatively low diversity compared to that of ImageNet. Therefore, we need the diversity score of the corresponding reference (test) dataset to compare the scores. 

**W3. Insensitivity of diversity for small domains.** The discriminative power of the Vendi score seems to attenuate for small domains, e.g., Cat and Bedroom results in Table 3. Maybe, if we finetune the feature extractor (inception networks) on the target domain, one may get better results, but it is not explored.

**W4. A problem in inner products as similarity function.** In Sec. 2.3, you should use *cosine similarity* instead of *inner-product* since the range of the output of the inner-product is not bounded to have ones of diagonal elements for the similarity matrix. Without the l2-normalization of features, the critical assumption of this work goes wrong. Could you confirm that the features are appropriately normalized in all your experiments? You did not explicitly mention it. Also, the covariance matrix is defined by $\mathrm{X}\mathrm{X}^\intercal /n$ only when the features are unbiased (subtracted with their mean vector).

**W5. Connection to the quantum statistical mechanics.** Although the authors mention the connection to quantum statistical mechanics *briefly* in 2.1, it seems to oversell the content of the paper in the Abstract unless additional elaborations or authors' speculations.
clarity,_quality,_novelty_and_reproducibility: ### Clarity

- C1. The first paragraph of the Introduction could be elaborated, including a definition of diversity, justification of this work, etc.

- C2. The related work in the Introduction section deserves a separate section.

- C3. Why do the authors call it the *Vendi* Score?

- C4. The definition of IntDiv is introduced too later (Sec. 3), although it is referred to in the previous sections frequently.


### Quality

- Formal writing does not abbreviate "does not" to "doesn't." e.g., "the Vendi Score doesn't" in Abstract.

- Appendix could start with alphabetic numbering, not 5.

- Figures and tables are misplaced on the page where it is referred. Although it cannot be perfectly matched, you could try.

### Novelty 

By seeing the distribution of eigenvalues as a probability distribution, the authors proposed to use the entropy of these eigenvalues to assess feature diversity was quite interesting. If the experimental validation shows a significant outperformance over conventional competing metrics, it would be a strong paper for the related communities.
summary_of_the_review: Generally speaking, the proposed idea is appealing and plausible as a general metric for assessing the diversity of generated samples. However, this work could be significantly improved in writing (see the clarity) and experimental validation (see the weakness, especially W1).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to use the Shannon entropy of the eigenvalues of the feature similarity matrix to evaluate diversity of samples of a generative model. While the idea was deemed interesting, there was a consensus among reviewers that this paper should be rejected. The key concern is how useful the score is, in particular since the utility of the score has not been assessed using human evaluations. Furthermore, the value of the score can be different across data sets, making it hard to know reasonable values without having access to a ground-truth test set.
justification_for_why_not_higher_score: There was a consensus among reviewers that this paper should be rejected.
Decision: Reject