The Vendi Score: A Diversity Evaluation Metric for Machine Learning
Authors: Dan Friedman
Adji Bousso Dieng
Venue: Submitted to ICLR 2023 (2023)

Abstract: Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. Yet little work has gone into understanding, formalizing, and measuring diversity in ML. In this paper we address the diversity evaluation problem by proposing the Vendi Score, which connects and extends ideas from ecology and quantum statistical mechanics to ML. The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix. This matrix is induced by a user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics in ML, the Vendi Score doesn’t require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. We showcase the Vendi Score on molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score to generative models of images and decoding algorithms of text where we found it confirms known results about diversity in those domains. Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (GANs). In particular, the Vendi Score revealed that even GANs that capture all the modes of a labelled dataset can be less diverse than the original dataset. Finally, the interpretability of the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for diversity-informed data augmentation

Under review as a conference paper at ICLR 2023
THE VENDI SCORE: A DIVERSITY EVALUATION METRIC FOR MACHINE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. Yet little work has gone into understanding, formalizing, and measuring diversity in ML. In this paper we address the diversity evaluation problem by proposing the Vendi Score, which extends ideas from ecology to MI.. The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of similarity matrix. This matrix is induced by user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics ML, the Vendi Score does not require a reference dataset or distribution over samples labels, it is therefore general and applicable any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. We showease the Vendi Score molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score generative models of images and decoding algorithms of text where we found it confirms known results about diversity those domains. Furthermore, we used the Vendi Score measure mode collapse. a known shortcoming of generative adversarial networks (GANs), particular, the Vendi Score revealed that even GANs that capture all the modes of labelled dataset can be less diverse than the original dataset. Finally, the interpretability the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for -inforey data augmentation."
INTRODUCTION
Diversity is criterion that is sought after in many areas of machine learning (ML), from dataset cura- tion and generative modeling to reinforcement learning, active learning, and decoding algorithms. A lack of diversity in datasets and models can hinder the usefulness of ML in many critical applications, e.g. scientific discovery. is therefore important to be able to measure diversity.
Many diversity metrics have been proposed in ML, but these metrics are often domain-specifi and limited in flexibility. These include metrics that define diversity in terms of a reference dataset (Heusel et al., 2017; Sajjadi et al., 2018). pre-trained classifier (Salimans al., 2016; Sri- vastava et al., 2017), or discrete features, like n-grams (Li et al., 2016). In this paper, we propose a general, reference-free approach that defines diversity in terms of user-speciffed similarity function.
Our approach is hased on work in ecology, where biological diversity has been defined as the ex- ponential of the entropy of the distribution of species within population (Hill, 1973; Jost, 2006; Leinster, 2021). This value can be interpreted as the effective number of species in the population. To adapt this approach to ML, exponential of the entropy of the eigenvalues of the X n similarity matrix K, whose entries are equal the similarity scores between each pair of elements. This entropy can be seen as the von Neumann entropy associated with K (Bach, 2022), so we call our metric the Vendi Score, for the von Neumann diversity.
define the diversity of a collection of elements ..,..., In as the
Code for calculating the Vendi Score will he made availahle publicly after the anonymity period.

---

Under review as conference paper at ICLR 2023
2.00 IntDiv 0.50
3.00
4.00
IntDiv 0.67
IntDiv = 0.75
0.8
0.2
VS 3.00 IntDiv 0.67
VS 3.78 IntDiv 0.67
4.66
IntDiv 0.67
1.0
-08
(b)
0.6
0.2
0.0
Figure 1: (a) The Vendi Score (VS) can be interpreted as the effective number of unique elements in sample. It increases linearly with the number of modes in the dataset. IntDiv, the expected dissimilarity, becomes less sensitive as the number of modes increases, converging to 1. (b) IntDiv does not take into account correlations between features, but VS does. VS is highest when the items in the sample differ in many attributes, and the attributes are not correlated with each other.
Contributions. We summarize our contributions as follows:
We extend ecological diversity to ML, and propose the Vendi Score, metric for evaluating diversity in ML. We study the properties of the Vendi Score, which provides us with more formal understanding of desiderata for diversity.
We showcase the flexibility and wide applicability of the Vendi ccorechharccteristic that stem from its sole reliance on the sample to be evaluated for diversity and a user-defined similarity function-and highlight the shortcomings of existing metries used measure di- versity in different domains.
2 RELATED WORK
Are we measuring diversity correctly in ML? Several existing metrics for diversity rely on reference distribution/dattset, which hinders flexibility. These reference-basee metrics define diver- sity in terms of coverage reference sample. They assume access to an embedding function that maps samples to real-valued vectors, such as a pretrained Inception model (Szegedy et al., 2016). The Fréchet Inception Distance (FID; Heusel et al., 2017) measures the Wasserstein-2 distance be- tween Gaussian distributions fit to reference embeddings and sample embeddings. FID was orig- inally proposed for evaluating image GANs but has been applied to text (Cifka et al., 2018) and molecules (Preuer al., 2018) using comain-speciff neural network encoders. Sajjadi et (2018) proposed a two-metric evaluation using precision and recall, with precision measuring quality and recall measuring diversity, terms of coverage of the reference distribution, and number of varia- tions and improvements have been proposed (Kynkäänniemí al., 2019; Simon et al., 2019; Nacem et al., 2020). Compared to these approaches, the Vendi Score can measure diversity without relying reference distribution/dattset.
Some other existing metrics evaluate diversity using a pre-trained classifier. Inception Score (IS; Salimans et al., 2016) evaluates diversity using the entropy of the marginal distribution of class labels predicted by an ImageNet classifier. More generally, given a pretrained classifier, mode coverage can be calculated directly by classifying test samples and calculating the number of unique modes the
2

---

Under review as a conference paper at ICLR 2023
entropy of the mode distribution (Srivastava et al., 2017). These metrics define diversity in terms of predefined categories, and require knowledge of the ground truth categories and a separate classifier.
In some discrete domains, diversity is often evaluated in terms of the distribution of unique features. For example in NLP, a standard metric is n n-gram diversity, which is defined as the number of distinct n-grams divided by the total number of n-grams (e.g. Li et al., 2016). These metrics require an explicit, discrete feature representation.
There are proposed metrics that use similarity scores define diversity. The most widely used metric of this form is the average pairwise dissimilarity score. In biology, average dissimilarity is known as Internal Diversity (IntDiv; Benhenda, 2017), with similarity defined as the Jaccard (Tanimoto) similarity beteween molecular fingerprints. Given samples 21...., In and pairwise similarity function & taking values between 0 and 1, IntDiv is defined:
Int Fn)
1 n
Σ 1.
In text, variants of this metric include Pairwise BLEU (Shen al., 2019) and D-Lex-Sim (Fomicheva et al., 2020), in which the similarity function is an n-gram overlap metric such as BLEU (Papineni al., 2002). As illustrated Figure I, IntDiv becomes less sensitive as diversity increases and does not account for correlations between features. Related to the metric we propose here is similarity- sensitive diversity metric proposed in ecology by Leinster and Cobbold (2012), and which was in- troduced in the context of ML by Posada al. (2020). This metric is based on notion of entropy defined in terms of a similarity profile, a vector whose entries are equal to the expected similarity scores of each element. Like IntDiv, it does not account for correlations between features.
Some other diversity metrics in the ML literature fall outside of these categories. The Birthday Paradox Test (Arora and Zhang, 2018) aims to estimate the size of the support of a generative model, but requires some manual inspection of samples. GILBO (Alemi and Fischer, 2018) is a reference- free metric but is only applicable to latent variable generative models.
Determinantal point processes. The Vendi Score bears a relationship to determinantal point pro- cesses (DPPs), which have been used in machine learning for diverse subset selection (Kulesza et al.. 2012). A DPP is a probability distribution over subsets of a ground set X' parameterized by positive semidefinite kernel matrix K. The likelihood of drawing any subset X X' is de- fined as proportional to Kxl, the determinant of the similarity matrix restricted to elements in X: P(X) x Kxl II, A where X are the eigenvalues of Kx. The likelihood function has geomet- ric interpretation, as the square of the volume spanned by the elements of X in an implicit feature space. However, the DPP likelihood is not commonly used for evaluating diversity, and has some limitations. For example, it is always equal to O if the sample contains any duplicates, and the ge- ometric meaning arguably less straightforward to interpret than the Vendi Score, which can be understood in terms of the effective number of dissimilar elements.
Spectral clustering. The eigenvalues of the similarity matrix are also related to spectral clustering algorithms (Von Luxburg, 2007), which use matrix known the graph Laplacian, defined L D-K, K, where K is symmetric, weighted adjacency matrix with non-negative entries, and D is diagonal matrix with D., E, Kiz- The eigenvalues of L can be used to characterize different properties of the graph-for for example, the multiplicity of the eigenvalue 0 is equal to the number of connected components. As a metric for diversity, the Vendi Score is somewhat more general than the number of connected components: it provides meaningful measure even for fully connected graphs, and captures within-component diversity.
MEASURING DIVERSITY WITH THE VENDI SCORE
We now define the Vendi Score, state its properties, and study its computational complexity. (We relegate all proofs of lemmas and theorems to the appendix.)
3-1 DEFINING THE VENDI SCORE
To define a diversity metric in ML we look to ecology, the field that centers diversity in its work. In ecology, main way diversity is defined is the exponential of the entropy of the distribution of
3

---

Under review as a conference paper at ICLR 2023
the species under study (Jost, 2006; Leinster, 2021). This is reasonable index for diversity. Consider a population with a uniform distribution over n species, with entropy log(n). This population has maximal ecological diversity n, the same diversity population with members, each belonging to a different species. The ecological diversity decreases as the distribution over the species becomes less uniform, and is minimized and equal to one when all members of the population belong the same species.
How can we extend this way of thinking about diversity to ML? One naive approach is to define diversity as the exponential of the Shannon entropy of the probability distribution defined by a ma- chine learning model or dataset. However, this approach is limiting in that it requires a probability distribution for which entropy is tractable, which is not possible in many ML settings. We would like to define diversity metric that only relies on the samples being evaluated for diversity. And we would like for such a metric to achieve its maximum value when all samples are dissimilar and its minimum value when all samples are the same, This implies the need to define a similarity function over the samples. Endowed with such similarity function, we can define form of entropy that only relies on the samples to be evaluated for diversity. This leads us to the Vendi Score:
Definition 3.1 (Vendi Score). Let ,I, E X denote a collection of samples, let k xxxx - TR be a positive semidefinite similarity function, functionn, with k(z, æ) = 1 for all a, and let K E RR=x" denote the kernel matrix with entry Kag- Zz). Denote by A...... An the eigenvalues of K /n. The Vendi Score (VS) is defined as the exponential of the Shannon entropy of the eigenvalues of K K/n:
8
VS& (SI......., ...) exp
A log Ao As)
(1)
where we use the convention log 0 = O.
To understand the validity of the Vendi Score as a mathematical object, note that the eigenvalues of K/ /n are nonnegative (because I is positive semidefinite) and sum to one (because the diagonal entries of K/ are equal to 1/n). 'he Shannon entropy is therefore well-defined and the Vendi Score is well-defined. In this form, the Vendi Score can also be seen as the effective rank of the kernel matrix K/ //.. Effective rank was introduced by Roy and Vetterli (2007) in the context of signal processing; the effective rank of a matrix defined as the exponential of the entropy of the normalized singular values. The Vendi Score can be expressed directly as a function of the kernel similarity matrix K:
Lemma 3.1. Consider the same setting as Definition 3.1. Then
VS& (21
exp
(-ur
(-ur
tr
K
log
K T3
The lemma makes explicit the connection of the Vendi Score to quantum statistical mechanics: the Vendi Score is equal to the exponential of the von Neumann entropy associated with K/ K/n (Bach, 2022).
Our formulation of the Vendi Score assumes that I . Fn were sampled independently, and so p(z.)) se 1 for all i. This is the usual setting in ML and the setting study in our experiments. However, we can generalize the Vendi Score to a setting in which we have an explicit probability distribution over the sample space X' (see Definition 3.1 in the appendix).
3.2 UNDERSTANDING THE VENDI SCoRE
The Vendi Score has several desirable properties as a diversity metric. We summarize them in the following theorem.
Theorem 3.1 (Properties of the Vendi Score). Consider the same definitions in Definition 3.I.
(a) Effective number: If x)) = 0 for all i # j. then VS In)) is maximized and equal to n. If (((,, = for all i, then (Sx In) minimized and equal to
(b) Partitioning: Suppose Si...., Sm are collections of samples such that, for any + j. for all x E S,, E Sy. k(x,x')= 0. Then the diversity of the combined samples depends only on the diversities of S1,...,S,, Sm and their relative sizes. In particular, if = |S.\ Σ, |S is the relative size of S; and H (p...... denotes the Shannon entropy. then VS is given by the geometric mean, (S Sm) exp( H (.p1....,P.)). I (S.)).)
4

---

Under review as conference paper at ICLR 2023
Number of components
VS
IntDiv
10
1.0
0.5
0.0
S1
S
S4
Sg
8
8
84
86
81
8q 83
Mixture proportions
V5
IntDiv
10
1.0
0.5
Si
S2
S
S₁
S1
Sq S3
S4
Per-companent variance
V5
IntDiv
10
1.0
0.5
0.0
S1
S2
S
S4
51 82
Figure 2: VS increases proportionally with diversity in three sets of synthetic datasets. In each row, we sample datasets from univariate mixture-of-normm distributions, varying either the number of components, the mixture proportions, the per-component variance. The datasets are depicted in the left, as histograms, and the diversity scores are plotted on the right.
(c) Symmetry: TI,... Tn. is a permutation of I....,n, (S I...).
nh then (TI
Taking the exponential of entropy turns into an effective number. The value of measuring diversity with effective numbers has been argued in ecology (e.g. Hill, 1973; Patil and Taillie, 1982; Jost, 2006) and economics (Adelman, 1969). Effective numbers provide a consistent basis for interpreting diversity scores, and make it possible to compare diversity scores using ratios and percentages. For example, suppose a population contains one million equally abundant species, and 50% go extinct. The diversity score will decrease by 50%, while the entropy will decrease by only about 5%.
3-3 CALCULATING VENDI SCORE
Calculating the Vendi Score for sample of n elements requires finding the eigenvalues of an a x n matrix, which has a time complexity of Onn)). The Vendi Score can be approximated using column sampling methods (i.e. the Nyström method; Williams and Seeger, 2000). However, in many of the applications we consider, the similarity function we use the cosine similarity between explicit feature vectors ((x) E Rd, with n. That is, K = X X, where X E B is the feature matrix K /n are the same as the eigenvalues of the matrix X /n, therefore we can calculate the Vendi Score exactly in a time of O(d + d8) OdPn). This is the same complexity as existing metrics such as FID (Heusel et al., 2017), which require calculating the covariance matrix of Inception embeddings.
with row X. = (<xx)/|(x<;)]]> The eigenvalues
EXPERIMENTS
We illustrate the Vendi Score (VS) on synthetic data to illustrate that captures intuitive notions of diversity, and then apply it to variety of setting in ML. We used VS to evaluate the diversity of generative models of molecules, an application where diversity plays an important role in enabling discovery, and found that VS identifies some model weaknesses that are not detected by IntDiv, the standard metric in that domain. We also applied VS to generative models of images, and decoding algorithms of text, where found confirms what know about diversity in those applications. We also used VS to measure mode collapse in GANs and show that reveals finer- iner-grained distinctions in diversity than current metrics for measuring mode collapse. Finally, used VS analyze the
5

---

Under review as a conference paper at ICLR 2023
HMM v5: 55.8 IntDiv: 0.853
AAE VS: 142.1 IntDiv: 0.849
Real
Vs: 127.7
IntDiv: 0.847
1.0
0.8
Similarity
0.6
0.4
0.2
0.0
Figure 3:
The kernel matrices for 250 molecules sampled from the HMM, AAE, and the original dataset, sorted lexicographically by SMILES string representation. The samples have similar IntDiv scores, but the HMM samples score much lower on VS. The figure shows that the HMM generates a number of exact duplicates, which are reflected by VS but not IntDiv.
diversity of several image, text, and molecule datasets, gaining insights into the least and most diverse elements of those datasets. (Implementation details are provided in Appendix B.)
4.1 SYNTHETIC EXPERIMENTS
To illustrate the behavior of the Vendi Score, we calculate the diversity of simple datasets drawn from mixture of univariate normal distributions, varying either the number of components, the mixture proportions, the per-component variance. We measure similarity using the RBF kernel: k(z,x') = exp(||= z'1l2/2o²), The results are illustrated in Figure 2. VS behaves consistently and intuitively in all three settings: in each case, VS can be interpreted as the effective number of modes, ranging between one and five in the first two rows and increasing from five to seven in the third row we increase within-mode variance. On the other hand, the behavior of IntDiv is different in each settings: for example, IntDiv is relatively insensitive to within-mode variance, and additional modes bring diminishing returns.
In Appendix C.I. we also validate that VS captures mode dropping in a simulated setting, using image and text classification datasets, where we have information about the ground truth class distribution. In both cases, VS has stronger correlation with the true number of modes compared to IntDiv.
EVALUATING MOLECULAR GENERATIVE MODELS FOR DIVERSITY
Next, we evaluate the diversity of samples from generative models of molecules. For generative models to be useful for the discovery of novel molecules, they ought to be diverse. The standard diversity metric this setting is IntDiv. We evaluate samples from generative models provided in the MOSES benchmark (Polykovskiy al., 2020), using the first 2,500 valid molecules in each sample. Following prior work, our similarity function is the Morgan fingerprint similarity (radius 2), implemented in 2 In Figure 3, we highlight instance where IntDiv and VS disagree: IntDiv ranks the HMM among the most diverse models, while VS ranks it as the least diverse (the complete results are Appendix Table 4). The HMM has a high IntDiv score because, on average, the HMM molecules have low pairwise similarity scores, but there are a number of clusters of identical or nearly identical molecules.
4-3 ASSESSING MODE COLLAPSE IN GENERATIVE ADVERSARIAL NETWORKS (GANS)
Mode collapse is a failure mode of GANs that has received lot of attention from the ML commu- nity (Metz et al., 2017; Dieng et al., 2019). The main metric for measuring mode collapse, called number of modes, only be used to assess mode collapse for GANs trained on a labelled dataset.
RDDiit: Open-source Cheminformatics, htts://www.rdkit.org.

---

Under review as conference paper at ICLR 2023
Model
Modes
Mode Div.
VS
Self-cond. GAN
1000
921.0
746.7
PresGAN
1000
948.7
866.6
Original
1000
950.8
943.7
Table I: VS reveals that even GANs that capture all the modes of labeled dataset can be less diverse than the original dataset, indicating that it captures fine-grained notion of diversity than number of modes.
Model
18+
FID,
Prect
Rec+
VS₁
Model
IS+
FID,
Prect
Rec+
VS+
CIFAR-10
LSUN Bedroom 256x256
Original
19.50
Original
8.99
VDVAE
5.82
40.05
0.63
0.35
12.87
StyleGAN
2.55
2.35
0.59
0.48
8.76
DenseFlow
6.01
34.54
0.62
0.38
13.55
ADM
2.38
1.90
0.66
0.51
7.97
IDDPM
9.24
4.39
0.66
0.60
16.86
RQ-VT
2.56
3.16
0.60
0.50
8.48
ImageNet 64 64
LSUN Cat 256x256
Original
43.93
Original
15.12
VDVAE
9.68
57.57
0.47
0.37
18.04
StyleGAN2
4.84
7.25
0.58
0.43
13.55
DenseFlow
5.62
102.90
0.36
0.17
12.71
ADM
5.19
5.57
0.63
0.52
13.09
IDDPM 15.59
19.24
0.59
0.58
24.28
RQ-VT
5.76
10.69
0.53
0.48
14.91
Table 2: VS generally agrees with the existing metrics. On low-resolution datasets (left) the diffusion model performs better of the metrics. On the LSUN datasets (right), the diffusion model gets the best quality scores as measured by FID, but scores lower on VS. No model matches the diversity score the original dataset they were trained on.
Number of modes computed by training a classifier on the labeled training data and counting the number of unique classes that are predicted by the trained classifier for the generated samples. In Table I, we evaluate two models that were trained on the StackedMNIST dataset, standard setting for evaluating mode collapse in GANs. StackedMNIST is created by stacking three MNIST images along the color channel, creating 1000 classes corresponding to 1000 number of modes.
We calculate VS using the probability product kernel (Jebara et al., 2004): k(x,x') P( p(y æ') where the class likelihoods are given by the classifier. We compare PresGAN (Dieng et al., 2019) and Self-conditioned GAN (Liu et al., 2020), two GANs that are known capture all the modes. Table shows that PresGAN and Self-conditioned GAN have the same diversity according to number modes, they capture all 1000 modes. However, VS reveals more fine-grained notion of diversity, indicating that PresGAN is more diverse than Self-conditioned GAN and that both are less diverse than the original dataset. One possibility that VS capturing imbalances in the mode distribution. To see whether this is the case, also caleulate Mode Diversity, the exponential entropy of the predicted mode distribution: exp H (p(y)), where p(v) generative models score lower VS than Mode Diversity, indicating that low entirely attributed to imbalances in the mode distribution. Therefore VS captures more aspects of diversity, even when we are using the same representations as existing methods.
> p(y 21). The cannot be
4-4 EVALUATING IMAGE GENERATIVE MODELS FOR DIVERSITY
We now evaluate several recent models for unconditional image generation, comparing the diver- sity scores with standard evaluation metrics, Inception Score (IS; Salimans et al., 2016), Frechet Inception Distance (FID; Heusel et al., 2017), Precision (Sajjadi et al., 2018), and Recall (Sajjadi et al., 2018). The models we evaluate represent popular classes of generative models, including variational autoencoder (VDVAE; Child, 2020), a flow model (DenseFlow; Greić et al., 2021), )), dif- fusion models (IDDPM. Nichol and Dhariwal, 2021; ADM Dhariwal and Nichol, 2021), GaN-based models (Karras et al., 2019; 2020), and an auto-regressive model (RQ-VT; Lee et al., 2022). The models trained on CIFAR-10 (Krizhevsky, 2009), ImageNet (Russakovsky et al., 2015), two

---

Under review as a conference paper at ICLR 2023
categories from the LSUN dataset (Yu et al., 2015). We either select models that provide precom- puted samples, or download publicly available model checkpoints and sample new images using the default hyperparameters. (More details are in Appendix B.)
The standard metrics in this setting use a pre-trained Inception ImageNet classifier to map images to real vectors. Therefore, we calculate VS using the cosine similarity between Inception embeddings, using the same 2048-dimensional representations used for evaluating FID and Precision/Recall. As result, the highest possible similarity score is 2048. The baseline metries are reference-based, with the exception of the IS. FID and IS capture diversity implicitly. Recall was introduced to capture diversity explicitly, with diversity defined as coverage of the reference distribution.
The results of this comparison are in Table 2. On the lower resolution datasets (left), VS generally agrees with the existing metrics. On those datasets the diffusion model performs better on all of the metrics. On the LSUN datasets (right), the diffusion model gets the best quality scores as measured by FID, but scores lower on VS. No model matches the diversity score of the original dataset they were trained on. In addition to comparing the diversity of the models, we can also compare the diversity scores between datasets: as a function of Inception similarity, the most diverse dataset is ImageNet 64 64, followed by CIFAR- 10, followed by LSUN Cat, and then LSUN Bedroom. This agrees with the intuition that the Inception network will induce the highest-rank embedding space for datasets that most resemble the training distribution.
VS should be understood as the diversity with respect to a specific similarity functioo, in this case, the Inception ImageNet similarity. We illustrate this point in the appendix (Figure 6) by comparing the top eigenvalues of the kernel matrices corresponding to the cosine similarity between Inception embeddings and pixel vectors. Inception similarity captures form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel provides a simple form of visual similarity, with components corresponding broad differences in lightness, darkness, and color.
4-5 EVALUATING DECODING ALGORITHMS FOR TEXT FOR DIVERSITY
Source
BLEU-4
N-gram div.
VS
Human
0.82
4.88
Beam Search
0.27
0.42
3,00
DBS 0.2
0.25
0.49
3.16
DBS 7 0.5
0.22
0.63
4.14
DBS 7 = 0.8
0.21
0.68
4.37
Table 3: Quality and diversity scores for an image captioning model using Beam Search or Diverse Beam Search (DBS). Increasing the diversity penalty y leads higher diversity scores but lower quality score (measured by BLEU-4).
We evaluate diversity on the MS COCO image-captioning dataset (Lin et al., 2014), following prior work on diverse text generation (Vijayakumar et al., 2018). In this setting, the subjects of evaluation are diverse decoding algorithms rather than parametric models. Given a fixed model p(x c). where is some conditioning context, the aim is to identify a "Diverse N-Best List", a list of sentences that have high likelihood but are mutually distinct. a list of sentences that have high likelihood but are mutually distinct. The baseline metric compare to is n-gram diversity (Li et al., 2016), which is the proportion of unique n-grams divided by the total number of n-grams. We define similarity using the n-gram overlap kernel: for a given the n-gram kernel kn is the cosine similarity between bag-of-n-ge n- ---aa feature vectors. We use the average of ki..... k1. This ensures that VS and n- n-gram diversity are calculated using the same feature representation. Each image in the validation split has five captions written by different human annotators, and we compare these with captions generated by a publicly available captioning model trained on this dataset." For each image, we generate five compan
.......,,,
captions using either beam search or diverse beam search (DBS Vijayakumar et al., 2018). DBS takes a parameter, 7, called the diversity penalty, and we vary this between 0.2, 0.6, and 0.8.
8

---

Under review as conference paper at ICLR 2023
Least diverse
Most diverse
plain
castle
clock
keyboard
Inception
Least diverse
Most diverse
cockroach
plain
man
aquarium fish
Pixels
Figure 4:
The categories in CIFAR-100 with the lowest and highest VS, defining similarity as the cosine similarity between either Inception embeddings or pixel vectors. We show 100 examples from each category, decreasing order of average similarity, with the image the top left having the highest average similarity scores according to the corresponding kernel.
Table shows that diversity metrics increase as expected, ranking beam search the lowest, the human captions the highest, and DBS in between, increasing with the diversity penalty. The human diversity score of 4.88 can be interpreted as meaning that, on average, all five human--ritter captions almost completely dissimilar from each other, while beam search effectively returns only three distinct responses for every five that it generates.
4.6 DIAGNOSING DATASETS FOR DIVERSITY
In Figure 4, we calculate VS for samples from different categories CIFAR-100, using the cosine similarity between either Inception embeddings pixel vectors. The pixel diversity is highest for categories like "aquarium fish", which vary color, brightness, and orientation, and lowest for cat- egories like "cockroach" in which images have similar regions of high pixel intensity (like white backgrounds). The Inception diversity less straightforward to interpret, but might correspond to some form of semantic diversity- example, the Inception diversity might be lower for classes like "castle," that correspond to distinct ImageNet categories, and higher for categories like "clock" and "keyboard" that are more difficult to classify. In Appendix C.5, we show additional examples from text, molecules, and other image datasets.
5 DISCUSSION
We introduced the Vendi Score, a metric for evaluating diversity in machine learning. The Vendi Score defined as function of the pairwise similarity scores between elements of sample and can be interpreted as the effective number of unique elements in the sample. The Vendi Score is inter- pretable, general, and applicable any domain where similarity can be defined. It unsupervised, in that it doesn't I require labels or reference probability distribution. Importantly, the Vendi Score allows its user to specify the form diversity they want to measure be computed efficiently exactly and showcased its usefulness in several
similarity function. We
showed the Vendi Score ML applications, different datasets, and different domains. In future work, we' leverage the Vendi Score to improve data augmentation, important ML approach in settings with limited data.

---

Under review as conference paper at ICLR 2023
REFERENCES
Adelman, M. A. (1969). Comment on the "H" concentration measure as a numbers--quivalent. The Review of economics and statistics, pages 99-101.
Alemi, A. A. and Fischer, L. (2018). GILBO: metric to measure them all. In Proceedings of the
32nd International Conference on Neural Information Processing Systems, pages 7037-7046.
Arora, S. and Zhang, Y. (2018). Do GANs actually learn the distribution? some theory and empirics. In International Conference on Learning Representations.
Bach, F. (2022). Information theory with kernel methods. arXiv preprint arXiv:2202.00555.
Benhenda, M. (2017). ChemGAN challenge for drug discovery: can Al reproduce natural chemical diversity? arXiv preprint arXiv:108..00 0822.2277...
Bird, S. (2006). Nitk: The natural language toolkit. In Proceedings of the COLING/ACL. 2006 Interactive Presentation Sessions, pages 69-72.
Child, R. (2020). Very deep VAEs generalize autoregressive models and can outperform them on images. arXiv preprini arXiv:2011.0663
Cifka, O., Severyn, A., Alfonseca, E., and Filippova, K. (2018). Eval all, trust few, do wrong to none: Comparing sentence generation models. arXiv preprint orXiv:/804.07972
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794
Dieng, B., Ruiz, J., Blei, D. M., and Titsias, M. K. (2019). Prescribed generative adversarial networks. arXiv preprint 1010.4490
Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M.. Aletras, N., Chaud- hary, V., and Specia, L. (2020). Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555.
Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple contrastive learning of sentence In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910.
Grcié, M., Grubišié, I., and Segvić, S. (2021). Densely connected normalizing flows. Advances in Neural Information Processing Systems, 11-23968-23982
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.
Hill, M. O. (1973). Diversity and Evenness: A Unifying Notation and Its Consequences. Ecology, 54(2):427-42.
Jebara, T., Kondor, R., and Howard, A. (2004). Probability product kernels. The Journal of Machine Learning Research, 5:819-844.
Jost, L. (2006). Entropy and Diversity. Oikos, 113(2):363-375.
Karras, T., Laine, S., and Aila, T. (2019). style-based generator architecture for generative ad- versarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020). Analyzing and im- proving the image quality of StyleGAN. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 81 10-8119.
10

---

Under review as a conference paper at ICLR 2023
Keung, P., Lu, Y., Szarvas, G., and Smith, N. A. (2020). The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report.
Kulesza, A., Taskar, B., et al. (2012). Determinantal point processes for machine learning. Founda- tions and Trends in Machine Learning, 1(2-3):123-286.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. (2019). Improved precision and recall metric for assessing generative models. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 3927-3936.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S. (2022), Autoregressive Image Generation using Residual Quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
11532.
Leinster, T. (2021). Entropy and Diversity: The Axiomatic Approach. Cambridge University Press.
Leinster, T. and Cobbold, C. A. (2012). Measuring Diversity: The Importance of Species Similarity. Ecology, 9373):477-489.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, W. B. (2016). A diversity- promooing objective function for neural conversation models. In Proceedings the 2016 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft COCO: Common objects context. In European conference on computer vision, pages 740-755. Springer.
Liu, S., Wang. T., Bau, D., Zhu, J.-Y., and Torralba, A. (2020). Diverse image generation via self- conditioned gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Liu, Z., Luo, P., Wang, X., and Tang. X. (2015). Deep learning face attributes in the wild. In International Conference on Computer Vision.
Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J. (2017). Unrolled generative adversarial net- works. In International Conference Learning Representations.
Nacem, M. Oh, S. J., Uh, Y., Choi, Y.. and Yoo, J. (2020). Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176-7185, PMLR.
Nichol, A. Q. and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. In Inter- national Conference Machine Learning. pages 8162-8111 8171. PMLR.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. (2002). BLEU: A method for automatic eval- uation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Parmar, G., Zhang, R., and Zhu, J.-Y. (2022). On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1 11410-114 11420.
Patil, G. and Taillie, C. (1982). Diversity as Concept and Its Mcasurement, Journal of the American Statistical Association, 77(379):548-56
Polykovskiy, D., Zhebrak, A., Sanchez-Lengelee B., Golovanov, S., Tatanov, O., Belyaev, S., Kur- banov, R., Artamonov, A., Aladinskiy, V., Veselov, M., Kadurin, A., Johansson, S., Chen. H., Nikolenko, S., Aspuru-Guzik, A., and Zhavoronkov, A. (2020). Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers Pharmacology.
Posada, J. G., Vani, A., Schwarzer, M., and Lacoste-Julien, S. (2020). Gait: A geometric approach to information theory. International Conference on Artificial Intelligence and Statistics, pages 2601-261 1. PMLR.
11

---

Under review as a conference paper at ICLR 2023
Preuer, K., Renz, P., Unterthiner, T.,, Hochreiter, S., and Klambauer, G. (2018). Fréchet ChemNet distance: a metric for generative models for molecules in drug discovery. Journal of chemical information and modeling. 58(9):1736-1741.
Radford, A., Metz, L.,, and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. In 1rxxiv/511.064333
Roy, O. and Vetterli, M. (2007). The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z Karpathy, A., Khosla,
A., Bernstein, M., et al. (2015). ImageNet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252.
Sajjadi, M. S., Bachem, O., Lucic, M., Bousquet, O., and Gelly, S. (2018). Assessing generative models via precision and recall. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5234-5243.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., and Chen, X. (2016). Improved Techniques for Training GANs. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems 29, pages 2234- 2242. Curran Associates, Inc.
Sanchez-Lengelling B., Wei, J. N., Lee, B. K., Gerkin, R. C., Aspuru-Guzik, A., and Wiltschko, A. B. (2019). Machine learning for scent: Learning generalizable perceptual representations of small molccules. arXiv preprint arXiv:1910.1068.
Shen, T., Ott, M., Auli, M., and Ranzato, M. (2019). Mixture models for diverse machine translation: Tricks of the trade. In International conference on machine learning, pages 5719-5728. PMLR.
Simon, L., Webster, R., and Rabin, (2019). Revisiting precision and recall definition for generative model evaluation. In International Conference on Machine Learning (ICML).
Song, J., Meng, C., and Ermon, S. (2021). Denoising diffusion implicit models. In International Conference on Learning Representations.
Srivastava, A., Valkov, L., Russell, C., Gutmann, M. U., and Sutton, C. (2017). VEEGAN: reducing mode collapse in GANs using implicit variational learning. In Advances in Neural Information Processing Systems.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the Inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., and Batra, D. (2018). Diverse beam search for improved description of complex scenes. Proceedings of the AAAI Conference on Artificial Intelligence.
Von Luxburg, U. (2007). A tutorial on spectral clustering. Statistics and computing, 17(4):395 416.
Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
Williams, C. and Seeger, M. (2000). Using the nyström method to speed up kernel machines. Ad- vances Neural Information Processing Systems, 13.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.. al. (2019). Huggingface's transformers: the natural language processing, arXiv preprint arXiv: 1910.0377..
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST novel image dataset for benchmark- ing machine learning algorithms. In arXiv: /ato::/00000444
12

---

Under review as conference paper at ICLR 2023
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J. (2015). LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365.
A PRoOFS
A.1 PROBABILITYWWIIGHTED VENDI ScORE
Definition A.1 (Probabilit--eeighte. Vendi Score). Let E A, denote probability distribution
on discrete space = {I1,...,IN, Inn.. where An denotes the (n 1)-dimensional simplex, let k X X + R be a positive semidefinite similarity function, with k(x,z) = 1 for all a, and let K R denote the kernel matrix with Kijj M(1;H1)) Let K, diagl K Kdiag( Shag((D) vD) denote the babbabilityweiighhee kernel matrix. Let A1...., ......,,n denote the eigenvalues of Ky. The Vendi Score (VS) is defined as the exponential of the Shannon entropy of the eigenvalues of Kp:
(-
9 Ne log As
(-E
(I Fn, = -exp
When all elements in the sample are completely dissimilar, the probability-weighted Vendi Score defined in Definition A.1 reduces to the exponential of the Shannon entropy of the weighting distri- bution:
Lemma A.1. Let p E An be a probability distribution over 31 ,1.-.., and suppose k( 2ry) for all i # Then VSe(01, Fa, = exp H (p), the exponential of the Shannon entropy of p.
A.2 PROOF OF ??
Rather than prove ?? directly, we prove more general lemma for the probability-weighter kernel matrix Kp defined in Definition A.1. ?? follows by setting p (1/N. 1/N) be the uniform distribution on N elements.
Lemma A.2. Let K E RNNN denote positive semi-definite kernel matrix with Küs 1 for i (1....,N) N). Let AN denote the (N - -1) 1)ddimensional simplex; its elements are vectors of dimension N whose entries are nonnegative and to one. Let p (p1...., .P)) AN denote probability distribution on N elements, and let X = (A1....,\., denote the vector of eigenvalues of the probabilitywwiigheed kernel matrix K, = diag( /p) K Kdiag( Then X E AN.
Proof. À is in the (N 1) -immlex if its entries are nonnegative and one. The entries of A nonnegative because the eigenvalues of a positive semi-definite matrix nonnegative, and K is the product of positive semi-definite matrices and so is positive remi-definite iteslf. (diag(vP) positive semi-definite because it is diagonal matrix with nonnegative entries.) The entries sum one because the trace of square matrix equal the sum of eigenvalues, and tr tr(Kp)
ΣN i==1 KGPp= 1.
A.3
PROoF OF LEMMA 3-1
Lemma. Consider the same setting Definition 3.1. Then
(S exp (-
K
K
log
(4)
Proof. For any square matrix X E Ruxn, if X has an eigendecomposition X UAU-1 then log X U (log A) U-! where log A diag(log λı...., ,og Xn) is diagonal matrix whose diagonal entries the logarithms of the eigenvalues of X. Also, tr tr(X) tr (UAU-1) tr(A),
13

---

Under review as conference paper at ICLR 2023
because the trace is so let K /n = UAU denote the eigendeccompoittion. Then
K/ K/n is diagonalizable because it is positive semidefinite,
tr K log K K/n) tr (UAU log (UAU-))
(UAU- (log A) U-! tr (Alog A)
Σ X log A,-
Therefore
16
= exp (-tr
K
K log x) ))
25) exp
X log 2
n2
A.4 PROOF OF LEMMA A.1
Lemma. Let E An be probability distribution .i,..., Tn and suppose k(ar,, 0 for all i * j. Then (S, Tn, exp H the exponential of the Shannon entropy of p.
Proof. If all element p are completely dissimilar, then Kp diagonal matrix, and the eigenvalues \1. A...... Às are the diagonal entries, which are the entries of p. So the von Neumann entropy of K is identical to the Shannon entropy of p. and the exponential is the Vendi Score.
A.5 PROor oF THEOREM 3.1
Proof. (a) Effective number: p is the uniform distribution over completely dissimilar elements, then Kp a diagonal matrix with each diagonal entry equal to 1/N. The eigenvalues of a diagonal matrix the diagonal entries, VS (p) exp (1/N,.... 1/N) exp log N N. On the other hand, if all elements are completely similar each other, then K, has rank and so the Vendi Score is equal to one.
(b) Identical elements: The eigenvalues of Kp are the same as the eigenvalues of the covariance matrix of the corresponding feature space:
N
Σ
Σ p(
Suppose elements and j are identical, and let p' denote the probability distribution created by combining and j. ie. pt = ps + Py and p/ 0. Clearly, Σn = Ep,, and so (Z (S, 2n, p').
(c) Partitioning: Suppose N samples partitioned into M groups Si...., Sa such that, for any i A j. for all x E S,, E Sy, k(æz x') = 0. Let p. = / Σ, |S denote the relative size of group i, and let K denote kernel matrix of U,S,, sorted in order of group index, and let Ks, denote the restriction of K elements in S,- Then K/N is block diagonal matrix, with each block i equal block diagonal matrix are the combined eigenvalues each block,
to p. Ks. The eigenvalues and the partitioning property then follows from the partitioning property of the Shannon entropy.
(c) Symmetry: The eigenvalues of matrix are unchanged by orthonormal transformation, and the Shannon entropy is symmetric in its arguments, so the Vendi Score is symmetric.
14

---

Under review as conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
B.I
IMAGES
Stacked MNIST We train GANs on Stacked MNIST using the publicly available code for Pres- GANs 4 and self-conditioned GANs 5. The models share the same DCGAN (Radford et al., 2015) architecture and are trained on the same dataset of 60,000 Stacked MNIST images, rescaled x2x32 pixels, and other hyperparameters are set according to the descriptions in the papers. The models are trained for 50 epochs and the diversity scores are evaluated every five epochs by taking 10,000 sam- ples. For both models, report the scores from the epoch corresponding the highest VS score. As in prior work (Metz et al., 2017), we classify Stacked MNIST digits by applying pretrained MNIST classifier to each color channel independently. The (000-dimension)) Stacked MNIST probability vector then the tensor product of the three 10-dimensional probability vectors predicted for the three channels.
Obtaining Image Samples In Section 4.4, we calculate the diversity scores of several recent gen- erative models images. We select models that represent range of families of generative models and and provide publicly available samples or model checkpoints for common image datasets. On the low-resolution datasets, we generate 50,000 samples from each model using the official code for VDVAE, DenseFlow,? and IDDPM, each of which provides checkpoint for unconditional image generation models on CIFAR-10 and ImageNet-64. For IDDPM, we sample using DDIM (Song et al., 2021) for 250 steps, and otherwise use the default sampling parameters. For the higher-resolution datasets, we use the 50,000 precomputed samples provided by Dhariwal and Nichol (2021) for ADM and StyleGAN models. We obtain 50,000 samples from the RO-VAE/Transfornes model using the code and checkpoints provided by the authors," 10 with the default sampling parameters.
Calculating Image Metries In Table we calculate standard image quality and diversity metrics, which are based on Inception embeddings. These Inception based metrics are sensitive to a number of implementation details (Parmar al., 2022) and in general cannot compared directly between papers. For a consistent comparison, we calculate all scores using the evaluation code provided by Dhariwal and Nichol (2021). We also calculated FID and Precision/Recall using the provided reference images and statistics, with the exception of CIFAR-10 10, for which we the training set as the reference. (The diversity scores of the Original datasets in Table are calculated using these reference images.) As result, the numbers in this table may not be directly comparable to results reported in prior work.
B.2 TEXT
Obtaining Image Captions In Section 4.5, we sample image captions from a pretrained image- captioning model," which is publicly available in Hugging Face (Wolf et al., 2019), and we use the Hugging Face implementation of beam search and diverse beam search. For beam search we use beam size of 5. For diverse beam search, we use beam size of 10, beam group size of 10, and set number of return sequences to 5.
Calculating Text Metrics The text metrics therefore depend on how sentences are tokenized into words. We calculate all text metrics using the pre-trained wordpiece tokenizer used by the captioning models. We use the implementation of the BLEU score in NLTK (Bird, 2006).
calculated in terms of word n-grams, and
httpp://giiuub..om//ppmam
15

---

Under review as conference paper at ICLR 2023
C
ADDITIONAL RESULTS
C.1
ASSESSING MODE DROPPING IN DATASETS
MNIST
MultiNLI
VS
VS
15
380
10
0.940
IntDy
0.55
0.935
0.50
IntDiv
10
Number classes
Number of classes
We evaluate VS and IntDiv on datasets containing 500 examples drawn uniformly from between one and ten classes: digits in MNIST and sentences genres in MultiNLI. Compared IntDiv, VS increases more consistently with the number of classes.
Figure 5:
In Figure 5, we examine whether VS captures mode dropping in a controlled setting, where we have information about the ground truth class distribution. We simulate mode dropping by sampling equal- sized subsets of two classification datasets, with each subset S; containing examples sampled uni- formly from the first i categories. We perform this experiment one image dataset (MNIST) and one text dataset (MultiNLII Williams et al., 2018), using simple similarity functions. For MNIST, we the cosine similarity between pixel vectors. In MultiNLI, the validation split (mismatched), which are drawn from one of ten genres. We define similarity using the n-gram overlap kernel: for a given n, the n-gram kernel k,, is the cosine similarity between bag-of--ngram feature vectors, and we use the average of k,....,k4.
the premise sentences from
The results (Figure 5) show that VS generally increases with the number of classes. In MNIST (left), VS increases roughly linearly for the first six digits (0-5) and then fluctuates. This could occur if the new modes are similar the other modes in the sample, or have low internal diversity. In MultiNLI (right), VS increases monotonically with the number of genres represented in the sample. In both cases, VS has stronger correlation with the number of modes compared IntDiv.
C.2 EVALUATING MOLECULAR GENERATIVE MODELS FOR DIVERSITY
We evaluate samples from generative models provided the MOSES benchmark (Polykovskiy et al., 2020), using the first 2,500 valid molecules in each sample. Following prior work, our similarity function is the Morgan fingerprint similarity (radius 2), implemented in 1DKit, 12 IntDiy ranks the HMM among the most diverse models, while VS ranks it as the least diverse (see Section 4.2).
C33 EVALUATING IMAGE GENERATIVE MODELS FOR DIVERSITY
In Table 5, we replicate the table described in Section 4.4 and add an additional column, which evaluates diversity using the cosine similarity between pixel vectors as the similarity function.
VS should be understood the diversity with respect to specific similarity function, in this case, the Inception ImageNet similarity. We illustrate this point in Figure by comparing the top eigenval- of the kernel matrices corresponding the Inception similarity and the pixel similarity, which we calculate by resizing the images to 32x32 pixels and taking the cosine similarity between pixel
PRDKii:: Open-source Cheminformaties, htps://www.rdkit.org
16

---

Under review as conference paper at ICLR 2023
Model
IntDiv
Original
0.855
403.9
AAE
0.859
501.1
Char-RNN
0.856
482.4
Combinatorial
0.873
536.9
HMM
0.871
250.9
JTN
0.856
489.5
Latent GAN
0.857
486.4
N-gram
0.874
479.8
VAE
0.856
475.3
Table 4: IntDiv and VS for generative models of molecules. The HMM has one of the highest IntDiv scores, but scores much lower on VS. An analysis of 250 molecules from the HMM reveals VS is more accurate in this case. (See Figure 3.)
Model
18+
FID,
Prece
Recy
VSr
VSPr
Model
IS+
FID,
Prec,
Recy
VS1
VSPT
CIEAR-10
LSUN Bedroom 256 256
Original
19.50
3.52
Original
8.99
3.10
VDVAE
5.82
40.05
0.63
0.35
12.87
3.34
StyleGAN
2.35
0.59
8.76
3,09
DenseFlow
34.54
0.62
0.38
13.55
2.94
ADM
2.38
1,90
0.66
0.51
7.97
3,27
IDDPM
9.24
4.39
0.66
0.60
16.86
3.27
RQ-VT
2.56
3.16
0.60
0.50
8.48
3.67
ImageNet 64
LSUN Cat 256x256
Original
43.93
4.43
Original
15.12
4.58
VDVAE
9.68
57.57
0.47
0.37
18.04
4.24
StyleGAN2
4.84
7.25
0.58
0.43
13.55
4.53
DenseFlow
5.62
102.90
0.36
0.17
12.71
3.51
ADM
5.19
5.57
0.63
0.52
13.09
4.81
15.59
19.24
0.58
24.28
4.57
RQ-VT
5.76
10.69
0.53
0.48
14.91
5.83
Table 5: We evaluate samples from several recent models, measuring similarity using either In- ception representations (VS,) pixels (VSp). The pixel similarity score is the cosine similarity between pixel vectors, calculated after resizing the images to 32x32 pixels. The pixel similarity and Inception similarity scores do not always agree- for example, if the images in sample represent variety ImageNet classes by share similar color palette, we might expect the sample to have high Inception diversity but low pixel diversity. The pixel diversity scores are on lower scale, indicating that this similarity metric is less capable of making fine-grained distinctions between the images in these samples.
vectors. Inception similarity provides a form of semantic similarity, with components correspond- ing to particular cat breeds, while the pixel kernel provides simple form of visual similarity, with components corresponding broad differences in lightness, darkness, and color.
C.4 EVALUATING DECODING ALGORITHMS FOR TEXT FOR DIVERSITY
In Figure 7, we plot the relationship between VS and n-gram diversity using the MS-COCO caption- ing data and the n-gram overlap kernel described Section 4.5. The figure shows that VS highly correlated with n-gram diversity, which is expected given that our similarity function is based on n-gram overlap. Nonetheless, there are some data points that metrics rank differently. This is because n-gram diversity conflates two properties: the diversity of n-graams within single sentences and the n-gram overlap between sentences. We highlight two examples in Figure 8. In general, the instances that n-gram diversity ranks lower compared to VS contain individual sentences that repeat phrases. On the other hand, n n-gram diversity can be inflated in cases when one sentence in the sample much longer than the others, even if the other sentences not diverse.
17

---

Under review as conference paper at ICLR 2023
Inception kernel
Pixel kernel
Least positive
Most positive
Least positive
Most positive
Kernei principal components
Kernel principal components
Figure The choice similarity function provides a way specifying the notion of diversity that relevant for given application. We project LSUN Cat images along the top eigenvectors of the kernel matrix, using either Inception features or pixels to define similarity. Inception similarity provides form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel captures visual similarity. For each eigenvector u, we show the four images with the highest and lowest entries in For both kernels, every similarity score is positive, eigenvector have the same sign; the images with the heighest weights in this component have the highest expected similarity scores. The remaining eigenvectors partition the images along different dimensions of variation.
all entries in the top
5.0
Beam search
4.5
Diverse beam search
Human
N-gram V5
4.0
3.5
3.0
2.5
0.3
0.4
0.5
0.6
0.7
0.8
N-gram diversity
Figure 7: VS is correlated with N-gram diversity. Each point represents group of five captions for particular image.
C.5 DIAGNOSING DATASETS
Molecules We evaluate the diversity scores of molecules in the GoodScents database of perfume materials, which has been used in prior machine learning research odor modeling (Sanchez- Lengeling et al., 2019). We use the standardized version of the data provided by the Pyrfume li- brary. 14 Each molecule in the dataset labeled with one more odor descriptors (for exam- ple, "clean, oily, waxy" or "floral, fruity, green")) We form groups of molecules corresponding to the seven most common odor descriptors, with each group consisting 500 randomly sampled
18

---

Under review as conference paper at ICLR 2023
High Vendi Score, low n-gram diversity:
Low Vendi Score. high n-gram diversity:
two men bow ties standing next to steel rafter.
man and woman cutting a slice of cake by trees.
several men in suits talking together in room.
couple of people standing cutting a cake.
an older man
tuxedo standing next to
dork with the earring stands next the asian beauty who is way out of his league.
younger man in a tuxedo wearing glasses.
two men wearing nuxedos glance each other.
newly married couple cutting cake in park.
older man in tuxedo sitting next to another younger man in tuxedo.
bride and groom are cutting cake as they smile.
Figure 8: Two sets of captions that receive different ranks according Vendi Score and n-gram diver- sity. We manually highlight some features contributing to the different scores. On the left, a sentence contains repeated n-grams, which are penalized by n n-gram diversity. On the right, one long outlier sentence contributes most of the n-grams for this group, greatly increasing the n-gram diversity.
molecules. We evaluate VS using two similarity functions: the Morgan fingerprint similarity (radius 2), and the similarity between odor descriptors, defined as the cosine similarity between descriptor indicator vectors (zx), where (x) is equal to one if descriptor is associated with molecule æ and zero otherwise.
44
SWeEt
green
woody fruity
Odor descriptors
herbal
38
floral
spicy
34
32
fatty
waxy
Molecular fingerprints
Figure 9: The Vendi Scores of samples containing 500 molecules with different scent labels, cal- culating diversity using two similarity functions: Morgan molecular fingerprint similarity, and the similarity between odor descriptors. Each molecule is associated with one more human-written tags (e.g- "floral, fruity, green, sweet"), and odor-descriptor similarity the cosine similarity between binary tag indicator vectors.
The diversity scores are plotted in Figure 9. The molecular diversity score and the odor-descriptor diversity scores are correlated, meaning that words like "woody" and "green" are used describe molecules that vary in molecular structure and also elicit diverse odor descriptions, while words like "waxy" and "fatty" are used for molecules that are similar to each other and elicit similar odor de- scriptions. For example, the word "green" appears in tag sets such as "aldehydic, citrus, cortex, green, herbal, tart" and "floral, green, terpenic, tropical, vegetable, woody", whereas the word "waxy" tends to co-occur with the same tags ('fresh, waxy": "fresh, green, melon rind, mushroom, tropical, waxy"; "fruity, green, musty, waxy"). Molecules from the categories with the highest and lowest scores are illustrated in Figure 10.
19

---

Under review as conference paper at ICLR 2023
Most diverse
Woody
Herbal
Daper
or xp
00
9p
dano
bo agt fo vo am
dr
CHim
Least diverse
Waxy
Fatty
my
moo
Oy
Figure 10:
The scent categories in Goodscents dataset with the lowest (top) and highest (highest) Vendi Scores, using the molecular fingerprint similarity. We show 100 examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores.
20

---

Under review as conference paper at ICLR 2023
MultiNLI genres
Amazon reviews
95
Slate
3 stars
45
90
stars
stars
Fiction
SimCSE
85
SimCSE
4 stars
80
Travel Telephone
42
Government
stars
375
380
385
390
395
375
376
377
378
379
360
381
N-grams
N-grams
Figure 11: The Vendi Scores of samples containing 500 MultiNLI sentences with different genres (left) or Amazon reviews with different star ratings (right), defining similarity using either n n-gram overlap or SimCSE (Gao al., 2021).
Text In Figure 11, we evaluate the diversity scores of samples sentences with different genres, from the MultiNLI dataset (Williams et al., 2018), and Amazon product reviews with different star ratings (Keung et al., 2020), using either n-gram overlap similarity or SimCSE (Gao al., 2021). SimCSE is Transforrer-based sentence encoder that achieves state-of-the-art scores semantic similarity benchmarks. The model we use initialized from the uncased BERT-base model (Devlin et al., 2019) and trained with contrastive learning objective to assign high similarity scores to pairs of MultiNLI sentences that have logical entailment relationship.
In MultiNLI, both models assign the highest score Slate, which consists of sentences from articles published on slate.com. SimCSE assigns a higher score to the "Fiction" category, possibly because is less sensitive to common n-grams (e.g. "he said"), that appear in many sentences in this genre and contribute to the low N-gram diversity score. In the Amazon review dataset, the 5-star reviews have the highest N-gram diversity but the lowest SimCSE diversity, perhaps because SimCSE assigns high similarity scores to sentences that have the same strong sentiment. SimCSE assigns the highest diversity score 3-star reviews, which can vary sentiment.
Images Following the setting in Section 4.6, evaluate two additional datasets, Fashion MNIST (Xiao et al., 2017) and CelebA (Liu et al., 2015). We use the same similarity scores as in Section 4.6. Images in CelebA are associated with 40-dimensional binary attribute vectors. We use these attributes as an additional similarity score, defining the attribute similarity as the cosine similarity between attribute vectors. These illustrations highlight the importance of the choice of similarity function in defining a diversity metric.
21

---

Under review as conference paper at ICLR 2023
Least diverse
Most diverse
Sneaker
Ankle boot
Bag
Shirt
Inception
Least diverse
Most diverse
Trouser
Coat
Bag
Sandal
Pixels
1A7
The categories in Fashion MNIST with the lowest (left) and highest (right) Vendi Scores, defining similarity as the cosine similarity between either Inception embeddings (top) or pixel vectors (bottom). We show 100 examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel.
Figure 12:
22

---

Under review as conference paper at ICLR 2023
Least diverse
diverse
Wearing, _Ncckie
Rosy_ Cheeks
Blurry
Wearing Hat
Inception
Least diverse
Most diverse
Blond_Hoe Hair
Rosy_Cheeks Cheeks
Wearing Hat
Mustache
Pixels
Least diverse
Most diverse
Rose_sh
MMkeuu
Big_Nose
Male
Attributes
The attributes in CelebA with the lowest (left) and highest (right) Vendi Scores, defining similarity as the cosine similarity between either Inception embeddings (top), pixel vectors (middle), or binary attribute vectors (bottom). We show 100 examples from each category, in decreasing der of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel. These examples illustrate the importance of the choice of similarity score for defining the notion of diversity that is relevant for a given application.
Figure 13:
23


Reviews:
[{"review_id": "gzgFq17baE", "reviewer": "Reviewer_zAvK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a diversity metric, called Vendi score, that connects and extends ideas from ecology and quantum statistical mechanics to machine learning.  \n\nThe Vendi Score doesn’t require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. The authors demonstrate the usage of Vendi score on domains including molecule generation, mode collapse of GANs, and decoding algorithms for texts.   \n", "summary_of_the_review": "\nI appreciate the authors’ efforts in presenting a notion of diversity score motivated by bio-diversity and applying it to various diversity-related applications in machine learning. However, besides the above weakness concerns, I have the following questions for the authors. I believe these questions need to be well-addressed before the work becomes a solid one.  \n\n### Q: Considering applications of DPP-based diversity, does Vendi score allow for efficient sampling, marginalization or learning, which are frequently needed for applications in ML? \n\nSpecifically, this would be possible if you define an energy-based model based on Vendi score, similar as what DPP does. This energy-based model describes the odds that a specific subset $S$ shall be selected out of the ground set $V$. \n\n$$p(S) \\propto VS_k(S) =  \\exp{-\\sum_i \\lambda_i \\log\\lambda_i},$$\n\nwhere $S:= \\{x_1,…,x_{|S|}  \\},  V:={x_1,…,x_{|V|} }, $  (should be the notion of a set, but it does not show up correctly in this openreview system...)\n\nIn this way, the partition function (normalization constant) shall be written as: \n$$ Z = \\sum_{S\\subseteq V} VS_k(S) $$\n\nOne clear advantage of DPP is that the partition function of it admits a closed form, which is not possible for many other Markov random field variants. Would the distribution defined by the Vendi score allow for similar kind of benefits?   \n\n\n### Q: Would it be possible to axiomatize the diversity metrics?  \n\nThere could be more than one way to define a diversity score (or even metric mathematically), as we have discussed above.  It is beneficial to think of some axioms that shall be met for a reasonable diversity measure: $D(A)$, where $A = \\{x_1, …, x_n\\}$ denotes the set of samples. \nFor example, the following axioms would come to my mind: \n\n1) monotonicity:  $D(A) \\leq D(A\\cup {x})$\n2) symmetry as you defined\n3) some relationship amongst $D(A)$, $D(B)$, $D(A\\cup B)$, $D(A\\cap B)$?   \n4) defining $D(\\emptyset)$? \n\n\n### Q:  How to correctly evaluate the validity of a diversity score empirically?    \n\nThe authors did extensive experiments on the usage of Vendi score in various application domains. However, it hardly provides strong evidence of the advantage of Vendi score.   Molecule generation experiments (sec. 3.1) shows one specific instance where IntDiv and VS have discrepancy. While IntDiv is a pretty weak baseline which results from a heuristic definition of diversity.   Image generation experiments (sec. 3.3) shows that VS generally agrees with existing metrics on low-resolution images. And on LSUN dataset, VS disagrees with IS.  Experiments on decoding text (sec. 3.4) show that VS agrees with N-gram diversity on a caption generation task with 5 sentences.  Experiments on diagnosing datasets (sec. 3.5) demonstrate that VS could provide diversity scores for samples within a specific class in several public datasets.   \n\nOne observation is that there is no, or it is very hard to get “golden” ground truth for diversity scores of a set of samples.  In my opinion, one way could be to ask a group of persons to provide annotations of diversity, either in the form of “scores” or rankings. One practical way to conduct this is to use some crowdsourcing platform.   A second way would be to use existing datasets in a smart way, say the CIFAR-100 dataset. One can make some groups of data samples out of the CIFAR-100, some groups only contain samples from one unique class (maybe called intra-class groups), and some groups have samples from various classes (called inter-class groups). One would naturally expect that the diversity of inter-class groups ranks higher than that of intra-class groups.  One can of course design more refined data groups along this line, to provide more refined way to verify the validity of a diversity score.  \n\n\n\n### Q: Naming it as “Vendi score” seems to be not consistent with the literature. \n\nOn the first glance I thought Vendi might be taken from some literature of bio-diversity. However, it turns out to be no after I checked representative papers on bio-diversity since the term “Vendi” has never appeared in the literature. \n\nThe most related concept may be the “Hill number of order 1” (https://en.wikipedia.org/wiki/Diversity_index#cite_note-Tuomisto2010a-4), see also the paper of Hill [Hill 1973]. So it seems to be reasonable to name it as “Hill score”. \n\n### Reference: \n\nHill, Mark O. \"Diversity and evenness: a unifying notation and its consequences.\" Ecology 54, no. 2 (1973): 427-432.\n\n", "strength_and_weaknesses": "\n### Strengths: \n\n- The proposed Vendi score serves as an interpretable metric for diversity.  The authors study several interesting properties of it, which gives a formal understanding of desiderata for diversity. \n- The Vendi score enjoys flexibility and wide applicability.\n- The Vendi score has the potential to be applied for diversity-informed data augmentation.  \n\n\n### Weaknesses: \n- Comparison with  classical diversity tools, e.g.., Determintal point processes （DPPs） is lacked. There is a rich literature on it, stemming from statistical physics and various applications in machine learning.  To name a few:  see [Li et al. 2016] and a wonderful survey [Kulesza-Taskar 2012] and the references therein.   \n\n- To make it clearer, if one follow the DPP definition of a diversity score, let us call it the DPP score (DS for short)  following the same notation in the paper, it shall be defined as \n$$ DS_k(x_1,…,x_n) = \\log |K| = \\sum_i \\log n\\lambda_i = \\sum_i (\\log n + \\log\\lambda_i) $$\nIt looks pretty related to the Vendi score. \n\nMeanwhile, DPP score also has an interpretation, it corresponds to the square of the volume spanned by the vectors of $ x_1,…,x_n$. \n\nCan you comment more on the connections to DPP score? \n\n### Reference: \n\nAlex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286, 2012\n\nChengtao Li, Stefanie Jegelka, and Suvrit Sra. Efficient sampling for k-determinantal point processes. In Artificial Intelligence and Statistics, pp. 1328–1337. PMLR, 2016.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cO174wamleY", "reviewer": "Reviewer_Z1XS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed the Vendi Score (VS) to measure the diversity of the generated samples from generative models. It is defined by the exponential of the Shannon entropy of the eigenvalues of a similarity matrix or covariance matrix using the inner-product similarity function (Sec. 2.3).\n\nThis work is reminiscent of spectral clustering, where the eigenvalues of the similarity matrix are used for clustering. Here, many clustering means diverse samples. The author can relate this work to spectral clustering and discuss it.", "summary_of_the_review": "Generally speaking, the proposed idea is appealing and plausible as a general metric for assessing the diversity of generated samples. However, this work could be significantly improved in writing (see the clarity) and experimental validation (see the weakness, especially W1).", "strength_and_weaknesses": "### Strength\n\nThe proposed idea has a compelling theoretical motivation, and it is convenient to use since it only requires the computation of the eigenvalue decomposition for the feature covariance matrix. Through a normalization technique, seeing the distribution of eigenvalues as a probability distribution was notably interesting. The connection to spectral analysis may be worth exploring and discussing. \n\n### Weakness\n\n**W1. Need for human evaluation.** The major concern is experimental validation. Table 1 does not provide the ground-truth diversity; we only learn that VS may capture the HMM's underperformance. Table 2 indirectly validates the VS by showing the sensitivity toward mode detection, not directly on the degree of diversity. Table 3 only compares with the other metrics, which may have some sense of measuring diversity (e.g., FID through its covariance matrices). It just shows general agreements with the other metrics. We do not know which metric is significantly better than one other for measuring diversity. The same goes for Table 4. Even human-written reference captions do not directly confirm that their diversity is captured by BLEU-4, excluding other factors. One suggestion is to evaluate with human judgments on diversity to compare with other conventional and competing metrics. \n\n**W2. Requirement of reference datasets due to inconsistency across domains.** Although the authors argue that it \"doesn't require a reference dataset\" in the Abstract, it shows inconsistent results across domains in Table 3. For instance, Cat or Bedroom datasets have relatively low diversity compared to that of ImageNet. Therefore, we need the diversity score of the corresponding reference (test) dataset to compare the scores. \n\n**W3. Insensitivity of diversity for small domains.** The discriminative power of the Vendi score seems to attenuate for small domains, e.g., Cat and Bedroom results in Table 3. Maybe, if we finetune the feature extractor (inception networks) on the target domain, one may get better results, but it is not explored.\n\n**W4. A problem in inner products as similarity function.** In Sec. 2.3, you should use *cosine similarity* instead of *inner-product* since the range of the output of the inner-product is not bounded to have ones of diagonal elements for the similarity matrix. Without the l2-normalization of features, the critical assumption of this work goes wrong. Could you confirm that the features are appropriately normalized in all your experiments? You did not explicitly mention it. Also, the covariance matrix is defined by $\\mathrm{X}\\mathrm{X}^\\intercal /n$ only when the features are unbiased (subtracted with their mean vector).\n\n**W5. Connection to the quantum statistical mechanics.** Although the authors mention the connection to quantum statistical mechanics *briefly* in 2.1, it seems to oversell the content of the paper in the Abstract unless additional elaborations or authors' speculations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "71uc0RDj8g6", "reviewer": "Reviewer_7smz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use Vendi score to measure the diversity of datasets.", "summary_of_the_review": "I am concerned about the novelty of the paper, since most of the contributions come from experimental results. ", "strength_and_weaknesses": "Strength:\n\n1. Paper shows many applications of the Vendi score to measure the diversity of data set, i.e. in generative model, mode collapse, image evaluation, algorithm evaluation etc.\n2. The Vendi score requires no label to compute.\n\nWeaknesses:\n\n1. The definition of Vendi score comes straight forward from the Shannon entropy. All of its theoretical results are consequences of results of Shannon entropy.\n\n2. There is no theoretical support for the use of Vendi score in situations presented in the paper. \n\n3. There is no guideline to explain when to use the Vendi score, i.e in the case of LSUN dataset, should we use the Vendi score to rank the efficiency of models?\n\n\nTo improve the work, the authors could prove that the Vendi score is an indicator for the number of modes. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject