The Vendi Score: A Diversity Evaluation Metric for Machine Learning
Authors: Dan Friedman
Adji Bousso Dieng
Venue: Accepted by TMLR ()

Abstract: Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. However, existing metrics for measuring diversity are often domain-specific and limited in flexibility. In this paper we address the diversity evaluation problem by proposing the Vendi Score, which connects and extends ideas from ecology and quantum statistical mechanics to ml. The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix. This matrix is induced by a user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. We showcase the Vendi Score on molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score to generative models of images and decoding algorithms of text where we found it confirms known results about diversity in those domains. Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (GANs). In particular, the Vendi Score revealed that even GANs that capture all the modes of a labelled dataset can be less diverse than the original dataset. Finally, the interpretability of the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for diversity-informed data augmentation.

Published in Transactions on Machine Learning Research (06/2023)
The Vendi Score: A Diversity Evaluation Metric for Machine Learning
Dan Friedman' and Adji Bousso Dieng!.
DDparrment of Computer Science, Princeton University
Vertaix
Reviewed OpenReview: http:://ppaarriieeeeee////mmmmmp
Abstract
Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. However, existing metrics for mcasuring diversity are often domain-specific and limited in flexibility. In this paper we address the diversity evaluation problem by proposing the Vendi Score, which connects and extends ideas from ecology and quantum statistical mechanics to ML. The Vendi Score defined as the exponen- tial of the Shannon entropy of the cigenvalucs of similarity matrix. This matrix induced by user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can defined. We showcase the Vendi Score on molecular generative modeling where we found it addresscs shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score to generative models of images and decoding algorithms of text where found it confirms known results about diversity in those domains. Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (CANs). In particular, the Vendi Score revealed that even GANS that capture all the modes of a labelled dataset can be less diverse than the original datasct. Finally, the interpretability of the Vendi Score allowed us to diagnosc several bench- mark MI, datasets for diversity, opening the door for diversity-informeed data augmentation.
1 Introduction
Diversity is criterion that is sought after in many areas of machine learning (ML), from datasct curation and generalive modeling to reinforcement learning, active learning, and decoding algorithms. A lack of diversity in datasets and models can hinder the usefulness of ML in many critical applications, e.g. scientific discovery. It is therefore important to be able to measure diversity.
Many diversity metrics have been proposed in ML, but these metrics are often domain-specific and limited in flexibility. These include metrics that define diversity in terms of reference dataset (Heusel et al. 2017 Sajjadi al.] 2018), ; pre-trained classifier (Salimans al., 2016 Srivastava al. 2017), or discrete features, like n n-grams [Li et al. 2016). In this paper, propose general, reference-free free approach that defines diversity in terms of user-specified similarity function.
Our approach is based on work in ccology, where biological diversity has been defined as the exponential of
the entropy of the distribution of species within a population (Hill 1973 Jost, 2006 Leinster (021). This value can be interpreted as the effective number of species in the population. To adapt this approach to
ML, we define the diversity of a collection of clements ,,, as the exponential of the entropy of the
Code calculating the Vendi Score available httee////tthhb.cco///rttxi//ee

---

Published in Transactions on Machine Learning Research (06/2023)
VS 2.00 IntDiv = 0.50
VS 3.00 IntDiv 0.67
VS 4.00
IntDiv 0.75
1.0
0.8
Snmiiini
(a)
0.2
0.0
&
3.00 Int 0.67
3.00 IntDiv 0.67
4.00
IntDiv 0.67
1.0
0.8
(b)
0.4
0.0
VS 3.00
VS 3.78
vS
4.66
IntDiv 0.67
Int 0.67
IntDiv 0.67
1.0
0.8
0.6
(c)
0.0
Figure 1: (a) The Vendi Score can be interpreted as the effective number of unique elements in a sample. It increases linearly with the number of modes in the dataset. IntDiv, the expected dissimilarity, becomes less sensitive as the number of modes increases, converging to (b) Combining distinct similarity functions can increase the Vendi Score, as should be expected a diversity metric, while leaving IntDiv unchanged. (c) IntDiv does not take into account correlations between features, but the Vendi Score does. The Vendi Score is highest when the items in the sample differ in many attributes, and the attributes are not correlated with each other.
eigenvalues of the n x similarity matrix K, whose entries are equal to the similarity scores between each pair of elements. This entropy can be scen as the von Neumann entropy associated with K Baahh 2022 so we call our metric the Vendi Score, for the von Neumann diversity.
Contributions. We summarize our contributions as follows:
We extend ecological diversity MI, and propose the Vendi Score, metric for evaluating diversity in ML. We study the properties of the Vendi Score, which provides us with a more formal understanding of desiderata for diversity.
We showcase the flexibility and wide applicability of the Vendi Score senee-charattersstiie that stem from its sole reliance on the sample to be evaluated for diversity and a user-defined similarity function-and highlight the shortcomings of existing metrics used to measure diversity in different domains.
2 Are We Measuring Diversity Correctly in ML?
Several existing metrics for diversity rely on a reference distribution or dataset. These referenco-basod metrics define diversity in terms of coverage of the reference. They assume access an embedding function-such pretrained Inception model (Szegedy al. 2016)-thor maps samples to real-valued vectors. One
2

---

Published in Transactions on Machine Learning Research (06/2023)
example of reference-based metric is Fréchet Inception distance (FID) (Heusel et al. 2017 which measures the W distance between two Gaussian distributions, one Gaussian fit to the embeddings of the reference sample and another one fit to the embeddings of the sample to be evaluated for diversity. FID was originally proposed for evaluating image generative adversarial networks (GANS) but has since been applied to text. (Cifka et al. 2018) and molecules (Preuer et al. 2018 using domain-specific neural network encoders. Sajjadi et al. (2018) proposed a two-metric evaluation paradigm using precision and recall, with precision measuring quality and recall measuring diversity in terms of coverage of the reference distribution. Several other variations of precision and recall have been proposed (Kynkäånniemi et al.) 2019 Simon et al., 2019 Nacem et al. 2020). Compared to these approaches, the Vendi Score is a reference-free metric, measuring the intrinsic diversity of a set rather than the relationship to a reference distribution. This means that the Vendi Score should be used along side a quality metric, but can be applied in settings where there is no reference distribution.
Some other existing metrics evaluate diversity using a pre-trained classifier, therefore requiring labeled datasets. For example, the Inception score (1s) (Salimans et al. 2016), which is mainly used to evaluate the perceptual quality of image generative models, evaluates diversity using the entropy of the marginal distribution of class labels predicted by an ImageNet classifier. Another example is number of modes (NOM) (Srivastava et al.) 2017), a metric used to evaluate the diversity of GANS. NOM is calculated by using a classifier trained on a labeled dataset and then counting the number of unique labels predicted by the classifier when using samples from a GAN as input. Both Is and NOM define diversity in terms of predefined labels, and therefore require knowledge of the ground truth labels and a separate classifier.
In some discrete domains, diversity is often evaluated in terms of the distribution of unique features. For example in natural language processing (NLP), a standard metric is n-gram diversity, which is defined as the number of distinct n-grams divided by the total number of n-grams (e.g. Li et al. 2016 These metries require an explicit, discrete feature representation.
There are proposed metrics that use similarity scores to define diversity. The most widely used metric of this form is the average pairwise similarity score or the complement, the average dissimilarity. In text, variants of this metric include BAIRWISE-BLEU (Shen et al., 2019) and D-LEX-SIM (Fomicheva et al. 2020). in which the similarity function is an n-gram overlap metric such as BLEU (Papineni et al.] 2002). In biology, average dissimilarity is known as IntDiv (Benhenda, 2017), with similarity defined as the Jaccard (Tanimoto) similarity between molecular fingerprints. Average similarity has some shortcomings, which we highlight in Figure 1 The figure shows the similarity matrices induced by a shape similarity function and/or a color similarity function. Each of the similarity functions is when the index of the column and the index of the row have the same shape or color and 0 otherwise. As shown in Figure I] the average similarity here measured less sensitive as diversity increases and does not account for correlations between features. This is not the case for the Vendi Score, which accounts for correlations between features and is able to capture the increased diversity resulting from composing distinct similarity functions. Related to the metric we propose here is similarity--sensiivve diversity metric proposed in ecology by Leinster & Cobbold (2012), and which was introduced in the context of ML by Posada et al. (2020). This metric is based on a notion of entropy defined in terms of a similarity profile, a vector whose entries are equal to the expected similarity scores of each element. Like IntDiv, it does not account for correlations between features.
by
Some other diversity metrics in the ML literature fall outside of these categories. The Birthday Paradox Test (Arora& & Zhang 2018) aims to estimate the size of the support of a generative model, but requires some manual inspection of samples. GILBO (Alemi & Fischer 2018 is a reference-free metric but is only applicable to latent variable generative models. |Kviman et al. 2002 measure the diversity of ensembles of variational approximations using the Jensen-Shannon Divergence (JSD); this metric is only applicable to sets of probability distributions. [Mithhell et al. 22020) introduce metrics for diversity and inclusion, defining diversity in terms of the representation of socially relevant attributes like gender and racc, and using the term helerogeneily to refer to variety in arbitrary attribules; in this paper, we use the term diversity to have the same sense as heterogeneity, meaning variety in arbitrary (user-specified) attributes. In the context of drug exploration, Xic et al. (2022) propose a metric based on the size of the largest subset of clements such that the similarity between any pair of elements is below some threshold, but this metric requires setting a threshold. Similarly, in the field of evolutionary computation, quality diversity (QD) algorithms (Pugh et al.
3

---

Published in Transactions on Machine Learning Research (06/2023)
2015 have assessed diversity by discretizing the feature space into grid of bins and counting the number of covered bins, but this approach requires picking a bin size.
As discussed above, several attempts have been made to measure diversity in MI. However, the proposed metrics can be limited in their applicability in that they require a reference dataset or predefined labels, or are domain-specific and applicable to one class of models. The existing metrics that do not have those applicability limitations have shortcomings when it comes to capturing diversily that we have illustrated in Figure
3
Measuring Diversity with the Vendi Score
We now define the Vendi Score, state its properties, and study its computational complexity. (We relegate all proofs of lemmas and theorems to the appendix.)
3.1 Defining the Vendi Score
To define a diversity metric in ML we look to ecology, the field that centers diversity its work. In ecology, one main way diversity is defined is as the exponential of the entropy of the distribution of the species under study (Jost, 2006 Leinster 2021). This is reasonable index for diversity. Consider population with a uniform distribution over species, with entropy log(n). This population has maximal ecological diversity n, the same diversity as a population with n members, each belonging to different species. The ecological diversity decreases as the distribution over the species becomes less uniform, and is minimized and equal to one when all members of the population belong to the same species. For a more extensive mathematical discussion of entropy and diversity in the context of biodiversity, we refer readers to Leinster 2021).
How can we extend this way of thinking about diversity to ML? One naive approach is to define diversity the exponential of the Shannon entropy of the probability distribution defined by machine learning model or dataset. However, this approach is limiting in that it requires a probabilily distribution for which entropy is tractable, which is not possible in many ML settings. We would like to define a diversity metric that only relies on the samples being evaluated for diversity. And we would like for such a metric to achieve its maximum value when all samples are dissimilar and its minimum value when all samples are the same. This implies the need define a similarity function over the samples. Endowed with such similarity function, we can define a form of entropy that only relies on the samples be evaluated for diversity, This leads the Vendi Score:
Definition 3.1 (Vendi Score). Let 21,..., In E X denote a collection of samples, let k X x X - R be positive semidefinite similarity function, with k(æ, = for all and let K E Rax denote the kernel matric with entry Ki₃ = b(x,,,,)) Denote by Ai...., An the eigenvalues of K K/n. The Vendi Score (VS) is defoned as the exporential of the Shannon entropy of the eigenvalues of K K/n:
n
exp (aI, .n) =exp
Σ
A log 2₂
(1)
where we use the convention Oog 0 0.
To understand the validity of the Vendi Score mathematical object, note that the eigenvalues of K // are nonnegative (because k is positive semidefinite) and sum to one (because the diagonal entries of K /n are equal to 1/n). The Shannon entropy is therefore well-defined and the Vendi Score is well-defined.
In this form, the Vendi Score can also be seen as the effective rank of the kernel matrix K. Effective rank was introduced by Roy & Vetterli (007)) in the context of signal processing; the effective rank of matrix is defined as the exponential of the entropy of the normalized singular values. Effective rank has also been used in machine learning, for example, to evaluate word embeddings (Torregrossa et al. 2020) and to study the implicit bias of gradient descent for low-rank solutions (Arora et al. 2019).
The Vendi Score can be expressed directly as a function of the kernel similarity matrix K:
4

---

Published in Transactions on Machine Learning Research (06/2023)
Lemma 3.1. Consider the some setting Definition 3.1 Then
K log
K
(11. ........ = =exp - tr
(2)
The lemma makes explicit the connection of the Vendi Score to quantum statistical mechanics: the Vendi Score is equal to the exponential of the von Neumann entropy associated with K/n (Bach 2022 In quantum statistical mechanics, the state of quantum system described by a density matria, often denoted p. The von Neumann entropy of p quantifies the unccrtainty in the state of the system (Wilde 2013). The normalizod similarity matrix K/ K/n here plays the role of the density matrix.
Our formulation of the Vendi Score assumes that 2 In were sampled independently, and so p(z;) 1 for all i. This is the usual setting in ML and the setting we study in our experiments. However, we can generalize the Vendi Score to a setting in which we have an explicit probability distribution over the sample space X (see Definition A.1 in the appendix).
3.2 Understanding the Vendi Score
Figure [ rates the behavior of the Vendi Score on simple toy datasets in which each element is defined by a shape and a color, and similarity is defined to be 1 if elements share both shape and color, 0.5 if they share either shape or color, and 0 otherwise.
First, Figure Ia illustrates that the Vendi Score an effective number, and can be understood as the effective number of dissimilar elements a sample. The value of mcasuring diversity with effective numbers has been argued in ecology (e.g. Hill 1973 Patil & Taillie 1982: Jost 2006) and economies (Adelman 1969). Effective numbers provide consistent basis for interpreting diversity scores, and make it possible to compare diversity scores using ratios and percentages. For example, in Figure la when the number of modes doubles from two to four, the Vendi Score doubles as well. If we doubled the number of modes from four to eight, the Vendi Score would double once again.
Figures Ib and Ic illustrate another strength the Vendi Score, which is that it accounts for correlations between features. Given distinct similarity functions l and N', the Vendi Score calculated using the combined similarity function {x(()) + (z) can be greater than the average of the individual Vendi Scores if the two similarity functions describe distinct dimensions of variation. Furthermore, the Vendi Score increases when the items in the sample differ in more attributes, and the attributes become less correlated with each other.
The Vendi Score has several desirable propertics as a diversity metric. We summarize them in the following theorem.
Theorem 3.1 (Properties of the Vendi Score). Consider the same definitions in Definition 3.1 and Defini tion A.1
1. Effective number. If k((,, = 0 for all i j, j, then (Z is mazimized and equal to n. If k(x;,,;) =1 for all i,j, then (S, Rn) is minimized and equal to 1.
2. Identical elements. Suppose k( xT)) = 1 for some i #... Let p' denole the probability distribution created by combining i and j, i.c. pt = ps + P3 and p', =0. Then the Vendi Score is unchanged,
Fn, p) = VS (x,
Partitioning. Suppose St...., Sm are collections of samples such that, for any + j, for all E St, E S,, k(I, = 0. Then the diversity of the combined samples depends only on the diversilies of S1,..., Sm and their relative sizes. In particular, if ps |S.l 15 is the relative size of St and H
denotes the Shannon entropy, then the Vendi Score is the geometric meon,
(Sa(S Sm) ep) I1 (S.
4. Symmetry. If Ti...., nn is a permutation of L,..., ,2 then
(S xn))= VS& (Z....... ..... , )
5

---

Published in Transactions on Machine Learning Research (06/2023)
The effective number property provides a consistent frame of reference for interpreting the Vendi Score: sample with Vendi Score of m can be understood to be as diverse a sample consisting of m completely dissimilar elements. The identical elements property provides some justification for our use of a sampling approximation: for example, calculating the empirical Vendi Score of a sample of 90 blue diamonds and 10 yellow squares equivalent to calculating the probabilit-wweighter Vendi Score of a sample of one blue spade and one yellow square, with p = 0.1). The partitioning property is analogous to the partitioning property of the Shannon entropy and means that if two samples are completely dissimilar we can calculate the diversity of the union of the samples using only the diversity of each sample independently and their relative sizes. The symmetry property means that the Vendi Score will be the same regardless of how we order the rows and columns in the similarity matrix.
3.3 Calculating the Vendi Score
Calculating the Vendi Score for a sample of n elements requires finding the eigenvalues of an n X n matrix, which has a time complexity of O(n)). The Vendi Score can be approximated using column sampling methods (i.e. the Nyström method; Williams & Seeger 2000). However, in many of the applications we consider, the similarity functions we use are inner products between explicit feature vectors o(x) E R2, with d < n. That is, K = X X, where X E Raxd is the feature matrix with row X,, = ((x,). The eigenvalues of K/ K/n are the same the cigenvalues of the covariance matrix XX /n, therefore we can calculate the Vendi Score exactly in time of O(dn + d3) OdFn). This is the same complexity existing metrics such as FID (Heusel et al. 2017) which require calculating the covariance matrix of Inception embeddings.
Sample complexity. The Vendi Score is the exponential of the kernel entropy, H = - tr K log K) Bach 2002) proves that empirical estimator of the kernel entropy has a convergence rate proportional to 1/ Vn, where n is the number of samples (Appendix A.5).
3.4
Connections to Other Areas in ML
Here we remark on the connections between the Vendi Score and other commonly studied objects in ML that make of the eigenvalues of a similarity matrix.
Determinantal Point Processes. The Vendi Score bears a relationship to Determinantal Point Processs (DPPS), which have been used in machine learning for diverse subset selection [Kulesza et al. 2012). A DPP is a probability distribution over subsets of a ground set X' parameterized by a positive semidefinite kernel matrix K. The likelihood of drawing any subset X S X' defined as proportional to [Kx], the determinant of the similarity matrix restricted to elements in X: p(X) x Kx II. Ar where A; are the eigenvalues of the square of the volume spanned by the
Kx. The likelihood function has geometric interpretation, elements of X in an implicit feature space. However, the DPP likelihood is not commonly used for evaluating diversity, and has some limitations. For example, it is always equal to 0 if the sample contains any duplicates, and the geometric meaning is arguably less Atraightffowan to interpret than the Vendi Score, which can be understood in terms of the effective number of dissimilar elements.
Spectral Clustering. The eigenvalues of the similarity matrix are also related to spectral clustering algorithms (Von Luxburg, 2007]. which use a matrix known as the graph Laplacian, defined L == D K, where K is symmetric, weighted adjacency matrix with non-negative entries, and D is a diagonal matrix with Dr,; = Ej Kr,. The cigenvalues of L can be used to characterize different properties of the graph- for example, the multiplicity of the eigenvalue 0 is equal to the number of connected components. As a metric for diversity, the Vendi Score is somewhat more general than the number of connected components: it provides a meaningful measure even for fully connected graphs, and captures within-component diversity.
4 Experiments
We illustrate the Vendi Score, which we now denote by VS for the rest of this section, on synthetic data illustrate that captures intuitive notions of diversity, and then apply it to a variety of setting in ML. We used vs to evaluate the diversity of generative models of molecules, application where diversity plays an
6

---

Published in Transactions on Machine Learning Research (06/2023)
important role in enabling discovery. We compare vs to IntDiv, function of the average similarity:
Int..v(........a))
1 n2 k(z,,
Σ
We found that Vs identifies some model weaknesses that not detected by IntDiv. We also applied vs to generative models of images, and decoding algorithms of text, where we found it confirms what we know about diversity in those applications. We also used VS to measure mode collapse in GANs and datasets and show that it reveals finer-grained distinctions in diversity than current metries for measuring mode collapse. Finally, we used VS to analyze the diversity of several image, text, and molecule datascts, gaining insights into the diversity profile of those datasets. (Implementation details are provided Appendix
Number of components
VS
IntDiv
10
1.0
5
0.5
0
0.0
Si
S₂
S3
S4
S2
S3
S4
Ss
S
S S3
S4 IntDiv
Ss
Mixture proportions
VS
0.0
S1
S2
S3
S4
Ss
83
84 8
81 S3
Per-component variance
VS
IntDiv
10
1.0
5
0.5
0
0.0
S1
S2
S3
S4
S5
S1
82
83
S4
S
S1
82
S3
S4
S
Figure VS increases proportionally with diversity in three sets of synthetic datasets. In each row, we sample datasets from univariate nixture-of-normin distributions, varying either the number of components, the mixture proportions, the per-component variance. The datasets depicted in the left, histograms, and the diversity scores are plotted on the right.
4.1 Synthetic experiments
To illustrate the behavior of the Vendi Score, we calculate the diversity of simple datasets drawn from a mixture of univariate normal distributions, varying either the number components, the mixture proportions, or the per-component variance. We measure similarity using the RBF kernel: k(æ, x²) exp(1|æ 2'1 /2 The results are illustrated in Figure 2 VS behaves consistently and intuitively in all three settings: in each case, VS can be interpreted as the effective number of modes, ranging between and five in the first two rows and increasing from five to seven in the third row increase within-mode variance. On the other hand, the behavior of IntDiv is different in each settings: for example, IntDiv is relatively insensitive to within-mode variance, and additional modes bring diminishing returns.
we also validate that vs captures mode dropping in a simulated setting, using image and text classification datasets, where we have information about the ground truth class distribution. In both cases, has stronger correlation with the true number of modes compared to IntDiv.
Appendix C.I
4.2 Evaluating molecular generative models for diversity
Next, we evaluate the diversity of samples from generative models of molecules. For generative models be useful for the discovery of novel molecules, they ought to be diverse. The standard diversity metric in this
7

---

Published in Transactions on Machine Learning Research (06/2023)
HMM VS: 55.8 IntDiv: 0.853
AAE VS: 142.1 IntDiv: 0.849
Real
VS: 127.7
IntDiv: 0.847
1.0
0.8
Similarity
-0.6
-0.4
0.2
0.0
Figure 3: The kernel matrices for 250 molecules sampled from the HMM, AAE, and the original dataset, sorted lexicographically by SMILES string representation. The samples have similar IntDiv scores, but the HMM samples score much lower on vs. The figure shows that the HMM generates a number of exact duplicates. is able to capture the HMM's lack of diversity while IntDiv cannot.
setting is IntDiv. We evaluate samples from generative models provided in the MOSES benchmark (Polykovskiy eet al., 2020), using the first 2500 valid molecules in each sample. Following prior work, similarity function is the Morgan fingerprint similarity (radius 2), implemented in RDKit Tn Figure 3 highlight an instance where VS and IntDiv disagrec: IntDiv ranks the HMM among the most diverse models, while vs ranks it as the least diverse (the complete results are in Appendix Table 4). The HMM has high IntDiv score because, on average, the HMM molecules have low pairwise similarity scores, but there are a number of clusters of identical or nearly identical molccules.
4.3 Assessing mode collapse in GANs
Mode collapse is failure mode of GANS that has received lot of attention from the ML community (Metz et al. 2017 Dieng et al. 2019). The main metric for measuring mode collapse, called nunber of modes(NOM), can only be used to assess mode collapse for GANS trained on labelled dataset. NOM is computed by training a classifier on the labeled training data and counting the number of unique classes that are predicted by the trained classifier for the generated samples. In Table ] we evaluate two models that were trained on the StackedMNIST datasel, a standard setting for evaluating mode collapse in GANS. StackedMNIST created by stacking three MNIST images along the color channel, creating 1000 classes corresponding to 1000 number of modes.
Model
NOM
Mode Div.
VS
Self-cond. GAN
1000
921.0
716.7
PresCAN
1000
918.7
866.6
Original
1000
950.8
943.7
Table 1: Vs captures a more fine-grained notion of diversity than number modes(NoM). Although PresGAN and Self-cond.CAN both capture all the 1000 modes, VS reveals that PreSCAN is more diverse than Self-cond.caa GAN and that they both are less diverse than the original dataset.
In prior work, mode collapse is evaluated by training an MNIST classifier and counting the number unique classes that are predicted for the generated samples. We adapt this approach and we caleulate vs using the probability product kernel (Jebara et. al. 2004): k(xxx²)= Ey plu x)²p(y/ æ')å, where the class likelihoods
2RDKit: Soures Cheminformatics. https://wwwrrdktt..r

---

Published in Transactions on Machine Learning Research (06/2023)
Model
16t
FIDy
Precy
Rect
VSp
18t
FIDJ
Prect
Rect
VS+
cifar-10
ImageNet 64x64
Original
19.50
43.93
VDVAE
5.82
40.05
0.63
0.35
12.87
9.68
57.57
0.47
0.37
18.04
DenseFlow
6.01
34.54
0.62
0.38
13,55
5.62
102.90
0.36
0.17
12.71
IDDPM
9.24
4.39
0.66
0.60
16.86
15.59
19.24
0.59
0.58
24.28
LSUN Cat 256 256
LSUN Bedroom x56x256
Original
15.12
8.99
StyleGAN2
4.84
7.25
0.58
0.43
13.55
2.55
2.35
0.59
0.48
8.76
ADM
5.19
5.57
0.63
0.52
13.09
2.38
1.90
0.66
0.51
7.97
RQ-VT
5.76
10.69
0.53
0.48
14.91
2.56
3.16
0.60
0.50
8.48
Table 2: generally agroes with the existing metrics. On low-rcsolution datascts (top left and top right) the diffusion model performs better all of the metrics. On the LSUN datasets (bottom left and bottom right), the diffusion model gets the highest quality scores as measured by 1s, but scores lower on VS. No model matches the diversity score of the original dataset they were trained on.
are given by the classifier. We compare PReSGAN (Deng et al. 2019) and Self-conditioned GAN (Liu al, 2020), two GANS that known to capture all the modes. Table shows that PreSGAN and Sell-conditioned GAN have the same diversity according to number of modes, they capture all 1000 modes. However, vs reveals more fine-grained notion of diversity, indicating that PReSGAN is more diverse than Self-conditioned GAN and that both are less diverse than the original dataset. One possibility is that vs is capturing imbalances in the mode distribution. To see whether this is the case, we also calculate what we call Mode Diversity, the exponential entropy of the predicted mode distribution: exp (f(y)), where p(y) = 1/j E.=IP(y z.). The generative models score lower on Vs than Mode Diversity, indicating that low scores cannot be entiroly attribuled to imbalances in the mode distribution. Therefore vs captures more aspects of diversily, even when we are using the same representations as existing methods.
4.4 Evaluating image generative models for diversity
We now evaluate several recent models for unconditional image generation, comparing the diversity scores with standard evaluation metrics, 18 (Salimans et al. 2016), FID (Heusel et al. 2017) Precision (Sajjadi
et al. 2018), and Recall (Sajjadi et al. 2018). The models we evaluate represent popular classes of generative
models, including variational autoencoder (VDVAE; Child 2020), flow model (DenseFlow Grcić et al.
2021), diffusion models (IDDPM, Nichol & Dhariwal 2021 ADM Dhariwal & Nichol, 2021 GAN-based
models Karras et al.] 2019 2020). and an auto-regressive model (RQ-VT: Lee et al. (022). The models are
trained on CIFAR-10 (Krizhevsky 2009), ImageNet (Russakovsky et al. 2015), or two categories from the
LSUN dataset (Yu et al.
2015). We either select models that provide precomputed samples, or download
publicly available model checkpoints and sample new images using the default. hyperparameters. (More
details are in Appendix
The standard metrics in this setting use pre-trained Inception ImageNet classifier to map images to real vectors. Therefore, we calculate vs using the cosine similarity between Inception embeddings, using the same 2048-dimensional representations used for evaluating FID and Doocisionn/Rcalll As a result, the highest possible similarity score is 2048. The baseline metrics are reference-based, with the exception of is. FID and IS capture diversity implicitly. Recall was introduced to capture diversity explicitly, with diversity defined as coverage of the reference ribtrinution.
The results of this comparison are in Table 2 generally agrees with the existing metrics. On those datasets the diffusion model performs better on all the metrics. On the LSUN datasets (bottom left and bottom right), the diffusion model gets the highest quality scores as measured by precision and recall, but scores lower on vs. In these cases, vs can be interpreted complementing the existing metrics. For example, on LSUN Cat, the ADM model achieves precision score
On the lower resolution datasets (top left and top right), vs
9

---

Published in Transactions on Machine Learning Research (06/2023)
Source
BLEU-4
N-gram div.
vS
Human
0.82
4.88
Beam Search
0.27
0.42
3.00
DBS 7 = 0.2 0.25
0.49
3.16
DBS ? 0.5
0.22
0.63
4.14
DBS ? 0.8
0.21
0.68
4.37
Table 3: Quality and diversity scores for image captioning model using Beam Search diverse beam search(DBs), varying the diversity penalty 7, BLEU-4 measures 1--rain overlap with the human-written reference captions, a proxy for quality. Increasing leads to higher diversity scores but lower quality score.
of 0.63 and recall of 0.52, implying that 63% of generated images look like reference images, and that the generated images cover 52% of the reference distribution; however, the low vs suggests that the remaining images have low internal diversity- for example, the model may generate many near-duplicates. No model matches the diversity score of the original dataset they were trained on. In addition comparing the diversity of the models, we can also compare the diversity scores between datasets: as function of Inception similarity, the most diverse dataset is ImageNet 64x64, followed by CIFAR-10, followed by LSUN Cat, and then LSUN Bedroom. Cat (all cats, but coming in different species), followed by LSUN Bedrooms.
VS should be understood as the diversity with respect to a specific similarity function, in this case, the Inception ImageNet similarity. We illustrate this point in in the appendix (Figure G by comparing the top eigenvalues of the kernel matrices corresponding to the cosine similarity between Inception embeddings and pixel vectors. Inception similarity captures a form of semantic similarity, with components corresponding partieular cat breeds, while the pixel kernel provides simple form of visual similarity, with components corresponding to broad differences in lightness, darkness, and color.
4.5 Evaluating decoding algorithms for text for diversity
We evaluate diversity on the MS COCO image-captioning dataset (Lin et al. 2014), following prior work on diverse text generation (Vijayakumar et al.] 2018). In this sctting, the subjects of evaluation are diverse decoding algorithms rather than parametrie models. Given fixed conditional model of text p(a c), where is some conditioning context, the aim is to identify "Diverse N-Bet List", a list of sentences that have high likelihood but are mutually distinct. The baseline metric we compare to is n-gram diversity (Li et al. 2016), which is the proportion of unique n-grams divided by the total mumber of n-grams. We define similarity using the n-gram overlap kernel: for a given n, the n-gram kernel ki is the cosine similarity between bag-of-n-gram feature vectors. We use the average of kj...., 14. This ensures that vs and n- n-gram diversity are calculated using the same feature reprosentation. Each image in the validation split has five captions written by different human annotators, and we compare these with captions generated by publicly available captioning model trained this dataset [3] For each image, generate five captions using either beam search diverse beam search (DBS) (Vijayakumar et al. 2018). DBS takes parameter, 7, called the diversity penalty, and we vary
this between 0.2, 0.6, and 0.8.
Table 3 shows that all diversity metrics increase as expected, ranking beam search the lowest, the human captions the highest, and DBS in between, increasing with the diversity penalty. The human diversity score of 4.88 can be interpreted as meaning that, on average, all five human--rritte captions are almost completely dissimilar from each other, while beam search effectively returns only three distinct responses for every five that it generates.
4.6 Diagnosing datasets for diversity
In Figure 4 we calculate vs for samples from different categorics in CIFAR-100, using the cosine similarity between either Inception embeddings or pixel vectors. The pixel diversity is highest for categories like "aquarium fish", which vary in color, brightness, and orientation, and lowest for categories like "cockroach" in which images have similar regions of high pixel intensity (like white backgrounds). The Inception diversity is
10

---

Published in Transactions on Machine Learning Research (06/2023)
Least diverse
Most diverse
plain
castle
clock
keyboard
Inception
Least diverse
Most diverse
cockroach
plain
man
aquarium_fia
Pixels
The categories in CIFAR-100 with the lowest and highest vs, defining similarity as the cosine similarity between either Inception embeddings or pixel vectors. We show 100 examples from each category, in decreasing order of average similarity, with the image the top left having the highest average similarity scores according to the corresponding kernel.
Figure 4:
less straightforward to interpret, but might correspond to some form of semantic diversity for example, the Inception diversity might be lower for classes like "castle," " that correspond to distinct ImageNet categorics, and higher for categories like "clock" and "keyboard" that are more difficult to classify. In Appendix C.5 show additional examples from text, molecules, and other image datasets.
we
5 Limitations
Here, we discuss several important limitations that should be considered when interpreting VS scores. First, VS is a reference-frre metric, meaning that it measures the internal diversity of a set and not how it relates a reference distribution. While this makes VS useful settings where there no reference distribution, also means that is possible to get high diversity score by, for example, sampling random noise. This is also true of other reference-free metrics, like IntDiv and n-gram diversity. Therefore, VS should be used alongside a quality metric. Second, like other similarity-basee metrics, VS is dependent similarity function. If the similarity function is too sensitive, all sets will appear very diverse, while if it is not sensitive enough, all sets will have low diversity. Additionally, the wrong choice of similarity function can introduce biases that lead to skewed diversity scores. Therefore, care should be taken when choosing a similarity function to ensure that appropriate for the specific application. Finally, the computational cost of calculating VS can be high, particularly if the similarity function is not associated with a low-dimensional embedding space. This may limit the applicability of VS in certain settings, particularly those with large datasets complex similarity functions.
the choice of
6 Conclusion
We introduced the Vendi Score, metric for evaluating diversity in machine learning (ML). The Vendi Score is defined function of the pairwise similarity scores bet elements of sample and can be interpreted
11

---

Published in Transactions on Machine Learning Research (06/2023)
the effective mumber of unique elements in the sample. The Vendi Score is interpretable, general, and applicable to any domain where similarity can be defined. It is unsupervised, in that it does not require labels or a reference probability distribution or dataset. Importantly, the Vendi Score allows its user to specify the form of diversity they want to measure via the similarity function. We showed the Vendi Score can be compuled efficiently exactly and showcased ils usefulness several MI applications, different datasels, and different domains. In future work, we will leverage the Vendi Score to improve data augmentation, an important ML approach in settings with limited data.
Acknowledgements
Adji Bousso Dieng supported by the National Science Foundation, Office of Advanced Cyberinfrastructure (OAC): #2118201. We thank Sadhika Malladi for pointing the effective rank. Adji Bousso Dieng would like to dedicate this paper to her PhD advisors, David Blei and John Paisley.
12

---

Published in Transactions on Machine Learning Research (06/2023)
References
Morris A Adelman. Comment on the 'H' concentration measure as numbers-cquivallet. The Review of economics and statistics, pp. 99 101, 1969.
Alexander A Alemi and Ian Fischer. GILBO: one metric to measure them all. In Proceedings of the 32nd International Conference on Neural Informalion Processing Systems, pp. 7037 7046, 2018.
S. Arora and Y. Zhang, Do GANs actually learn the distribution? some theory and empirics. In International Conference on Learning Representations, 2018.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, 2019.
Francis Bach. Information theory with kernel methods. arXiv preprint arXin:2202.08542 2022.
Mostapha Benhenda. ChemGAN challenge for drug discovery: can Al reproduce natural chemical diversity? arXiv preprint arxiv 1oe.1708.000 082227 2017.
Steven Bird. Nltk: The natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pp. 69-72, 2006.
Rewon Child. Very deep VAEs generalize auloregressive models and can outperform them on images. arXiv preprint crXte:2011.10560 2020.
Ondřej Cifka, Aliaksei Severyn, Enrique Alfonseca, and Katja Filippova. Eval all, trust few, do wrong to none: Comparing sentence generation models. arXiv preprint ar arxii:1804.0000 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Loe, and Kristina Toutanova. BERT: Pro-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Computational Linguisties: Humon Language Technologies, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Aduances in
Neural Information Processing Systems, 14-870088794, 2021.
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprint Xrc::11900004. 04302, 2019.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguisties,
2020.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Matej Greié, Ivan Grubišié, and Siniša Segvić. Densely connected normalizing flows. Advances in Neural Information Processing Systems, 34:23968 23982, 2021.
Martin Heusel, Hubert Ramsaner, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Mark O Hill. Diversity and Evenness: A Unifying Notation and Its Consequences. Ecology, 54(2):427 432, 1973.
Tony Jebara, Risi Kondor, and Andrew Howard. Probability product kernels. The Journal of Machine Learning Research, 5:819 844, 2004.
Lou Jost. Entropy and Diversity. Oikos, 113(2):363 375, 2006.
13

---

Published in Transactions on Machine Learning Research (06/2023)
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CV /CV conference on computer vision and pallern recognilion, pp- 4401 4410, 2019.
Tero Karras, Samuli Laine, Mila Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. $110-8119) 2020.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The mulcilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends0 in Machine Learning, (22-3)223-22220 2012.
Oskar Kviman, Harald Melin, Hazal Koptagel, Victor Elvira, and Jens Lagergren. Multiple importance sampling elbo and deep ensembles of variational approximations. In International Conference on Artificial Intelligence and Stalislics, pp. 10687-10702. PMLR, 2022.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3927 3936, 2019.
Doyup Lee, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive Image Generation using Residual Quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1522-115322 2022.
Tom Leinster. Entropy and Diversity: The Aziomatic Approach. Cambridge University Press, 2021.
Tom Leinster and Christina A Cobbold. Measuring Diversity: The Importance of Species Similarity. Ecology, 93(3)7477 489, 2012.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. ppeesitt-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, 2016.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pp. 740 755. Springer, 2014.
Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IFFE/CVI Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In International Conference Computer Vision, 2015.
L. Motz, B. Poolc, D. Pfau, and J. Sohl-Dickstein. Unrolled generative adversarial networks. In International Conference on Learning Representations, 2017.
Margaret Mitchell, Dylan Baker, Nyalleng Moorosi, Emily Denton, Ben Hutchinson, Alex Hanna, Timnit Gebru, and Jamie Morgenstern. Diversity and inclusion metrics in subset selection. In Proceedings of the AAAT/ACM Conference on AI, Ethics, and Society, pp. 117-123, 2020.
Muhammad Ferjad Nacom, Scong Joon Oh, Youngjung Uh Yunjey Choi, and Jacjun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7185-7185. PMI.R, 2020.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8888 8171. PMLR, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedlings of the IEEE/CVE Conference Computer Vision and Pallern Recognition, pp. 11410 10400, 2022.
GP Patil and Charles Taillie. Diversity as a Concept and Its Measurement. Journal of the American Statistical Association, 77(339):548-566) -561 1982.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Arlamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru-Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology, 2020.
Jose Gallego Posada, Ankit Vani, Max Schwarzer, and Simon Lacoste-Julien. Gait: A geometric approach information theory. In International Conference on Artificial Intelligence and Statistics, pp. 2601-2611. PMLR, 2020.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Günter Klambauer. Fréchet ChemNet distance: metric for generative models for molecules in drug discovery. Journal of chemical information and modeling, (8(9):1736-174)) 2018.
Justin K Pugh, Lisa B Soros, Paul A Szerlip, and Kenneth O Stanley. Confronting the challenge of quality diversity. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 967 967-974, 2015.
A. Radford, L. Metz, and S. Chintala. Unsupervised represcntation learning with deep convolutional generative adversarial networks. In arXiw:1511.06434 2015.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEEE, 2007.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, el al. ImageNet large scale visual recognition challenge. International journal of computer vision, (15(3):211-253) 2015.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative modcls via precision and recall. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 5234 5243, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved Techniques for Training GANs, In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2234-2242. Curran Associates, Inc., 2016.
Benjamin Sanchez--Lnngeling, Jemnifer N Wei, Brian K Lee, Richard C Gerkin, Alán Aspuru-Guzik, and Alexander В Wiltschko. Machine learning for scent: Learning generalizable perceptual representations of small molecules. arXiv preprint arXiv:1910.11685, 2019.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc' Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. In International conference on machine learning, pp. 5719-5728. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Loic Simon, Ryan Webster, and Julien Rabin. Revisiting precision and recall definition for generative model evaluation. In International Conference on Machine Learning (TCML), 2019.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton. VEEGAN: reducing mode collapse in GANs using implicit variational learning. In Advances in Newral Information Processing Systems, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojua. Rethinking the Inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
François Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, and Robin Allesiardo. On the correlation of word embedding evaluation metrics. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 4789 4797, 2020.
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference on Arlificial Intelligence, 2018.
Uhike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395-41, 2007.
Mark M Wilde. Quantum informalion theory. Cambridge University Press, 2013.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Associalion for Compulational Linguisties: Human Language Technologies, Volume (Long Papers),
pp. 1112 1122, 2018.
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. Advances in Neural Information Processing Systems, 13, 2000.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arxi preprint S5t::1100003711 2019.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. In 708.07747, 2017.
Yutong Xie, Ziqiao Xu, Jiaqi Ma, and Qiaozhu Mei. How much of the chemical space has been explored? selecting the right exploration measure for drug discovery. In 1CML 2022 2nd Al for Science Workshop, 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint Xrv: 10X6::5006.3335,, 2015.
A Proofs
A.1 wregahelitywwiighted Vendi Score
Vendi Score). Let p E An denote a probability distribution on a discrete space = {x1,...,,.h} where An denotes the (n - -imdemnnson simplex, let k X x X + R be positive semidefinite simélarity function, with k(xxx) = for all x, and let K R denote the kernel matrir with K, = 0;),, Let K, = diag( Kdiag( denote the probabiltty kernel matrir. Ao
Definition A.1 (Probability-
Let A,...,,?, denote the cigenvalues of Kp. The Vendi Score (VS) is defined as the exponential of the Shannon entropy of the eigenvalues of Kp:
8
VSa "2,
Kn, cxp
cxp - Σ
A log A3
(3)
16

---

Published in Transactions on Machine Learning Research (06/2023)
When all elements in the sample completely dissimilar, the babbability-wiggheee Vendi Score defined
in Delinition A.1 reduces to the exponential of the Shannon entropy of the weighting distribution:
Lemma A.1. Let p E An be probability distribution over Th,..., and suppose ((I,,z)) = 0 for all + j.
Then (T Fn. p) exp (p), the exponential of the Shannon entropy of p.
A.2 Proof of [eemma 3.1
Lemma. Consider the same setting as Definition 3.1 Then
K
VSk(II,.... F.) = exp tr
log
(4)
12
Proof. For any square matrix X E R"x", X has an eigendecomposition X = UAU-1, then log U (log U- where log A diag(log .,....,iu log An) is a diagonal matrix whose diagonal entrics are the logarithms of the eigenvalues of X. Also, tr(X) tr (UAU-1) tr(A), because the trace similarity- invariant. K/n is diagonalizable because it is positive semidefinite, so let K/n UAU denote the eigendeccompositon. Then
tr K /n log K K/n) tr (UAU log (UAU-
(UAU U (log A)U
tr log
A log Ai-
Therefore
K -tr
log - )
VS1,,........ = =ep
A log di = exp
71
A.3 Proof of Lemma A.1
Lemma. Let p E An be a probability distribution over 21. and suppose k(z;, zy) = 0 for all i i j. Then VSa (ZI,..., En. = exp (p), the exponential of the Shannon entropy of p.
Proof. If all element in p are completely dissimilar, then K, is a diagonal matrix, and the eigenvalues A Äg are the diagonal entries, which are the entries of p. So the von Neumann entropy of Kp is identical to the Shannon entropy of p, and the exponential is the Vendi Score.
A.4 Proof of Theorem 3.1
Proof. (a) Effective number: If is the uniform distribution over N completely dissimilar elements, then Kp is a diagonal matrix with each diagonal entry equal to 1/N. The eigenvalues of a diagonal matrix are the diagonal entries, so (p)(p) = exp (I/N,.... 1/N) = exp log N = N. On the other hand, if all elements are completely similar to each other, then Kp has rank one and so the Vendi Score is equal to one.
(b) Identical elements: The eigenvalues of K, the same the cigenvalues of the covariance matrix of the corresponding feature space:
N Σp Σ p(z:)o(z.))(x))
Suppose elements i and j are identical, and let p' denote the probability distribution created by combining and j, i.e. p4 = Pi Pj and p', O. Clearly, Σx Ep, and VS& (Z1 ; p) VS& (Z1,... Eny p').
17

---

Published in Transactions on Machine Learning Research (06/2023)
(c) Partitioning: Suppose N samples partitioned into M groups St...., Sat such that, for any i i j. for all x E St, E Sj, x(xxx')= = 0. Let p; = |S:1/ Σ, |Syl denote the relative size of group i, and let K denote kernel matrix of U,S, sorted in order of group index, and let Ks, denote the restriction of K to elements in S.. Then K/N is a block diagonal matrix, with cach block i equal to Ks,, The cigenvalues of a block diagonal matrix are the combined eigenvalues of each block, and the partitioning property then follows from the partitioning property of the Shannon entropy.
(e) Symmetry: The eigenvalues of malrix are unchanged by orthonormal transformation, and the Shannon entropy is symmetric in its arguments, the Vendi Score is symmetric.
A.5 Sample Complexity
The Vendi Score is the exponential of the kernel entropy, ((K) = - log )
Bach 2022 proves that
empirical estimator of the kernel entropy, Ht has convergence rate proportional to 1/ Vn, where n is the mumber of samples. Additionally, by Jensen's 's inequality, the E[Å] is no greater than H. Therefore:
exp(H) - exp(ff) & cxp(H) - exp vn
1
exp(H - exp(H)/ exp
exp(H)) (1-
The empirical estimator of the Vendi Score therefore also has a convergence rate proportional to 1//n, with constant term depending on the true entropy.
B Implementation Details
B.1 Images
Stacked MNIST We train GANs Stacked MNIST using the publicly available code for PresGANs and self-conditioned GANs The models share the same DCGAN (Radford et al. 2015) architecture and are trained on the same dataset of 60,000 Stacked MNIST images, rescaled to 32x32 pixels, and other hyperparameterss are set according to the descriptions in the papers. The models are trained for 50 epochs and the diversity scores are evaluated every five epochs by taking 10,000 samples. For both models, we report the scores from the epoch corresponding to the highest VS score. As prior work (Metz et al.) 2017), we classify Stacked MNIST digits by applying a pretrained MNIST classifier to each color channel independently. The 1000-dimensional Stacked MNTST probability vector is then the tensor product of the three 10-dimensional probability vectors predicted for the three channels.
Obtaining Image Samples In Section 4.4 calculate the diversity scores of several recent generative models of images. We select models that represent range of families of generative models and and provide publicly available samples or model checkpoints for common image datascts. On the low-resolution datasets, we generate 50,000 samples from each model using the official code for VDVAF DenseFlow and IDDPM B each of which provides a checkpoint for unconditional image generation models on CIFAR-10 10 and Image For IDDPM, we sample using DDIM (Song et al. (2021) for 250 steps, and otherwise use the default sampling parameters. For the higher-resolution datasets,
the 50,000 precomputed samples provided by Dhariwal & Nichol 2021 for ADM and StyleGAN models. We obtain 50,000 samples from the V.RVVA//Taasnonnnn with the default sampling parameters.
model using the code and checkpoints provided by the authors
PresGAN,
htttts/s/gghhbbbco////eeuna vvvae/
hhttps://gihhub.ccm//neeee
10 Phttps//ghhhhubbom//kkkke tones//mmmeeeeooioioo/eoeoeettne transsooree
18

---

Published in Transactions on Machine Learning Research (06/2023)
Calculating Image Metrics In Table 2 calculate standard image quality and diversity metrics, which are based on Inception embeddings. These Inception-based metries are sensitive to number of implementation details (Parmar et al. 2022) and in general cannot be compared directly between papers. For a consistent comparison, we calculate all scores using the evaluation code provided by Dhariwal & Nichol (2021). We also calculated FID and Precision/ RReall using the provided reference images and statistics, with the exception of CIFAR-10, for which we use the training set as the reference. (The diversity scores of the Original datasets in Table 2 are calculated using these reference images.) As result, the numbers in this table may not be directly comparable to results reported in prior work.
B.2 Text
Obtaining Image Captions In Section 1.5 we sample image captions from pretrained image-captioning the Hugging Face model F which is publicly available in Hugging Face (Wolf et al.) 2019), and implementation of beam search and diverse beam search. For beamn search we use beam size of 5. For diverse beam search, we use beam size of 10, beam group size of 10, and set the number of return sequences to 5.
Calculating Text Metrics The text metrics we use are calculated in terms of word n-grams, and therefore depend on how sentences are tokenized into words. We calculate all text metrics using the pre-trained wordpiece tokenizer used by the captioning models. We use the implementation of the BLEU score in NLTK Bird 20060.
Additional Results
C.1 Assessing Mode Dropping in Datasets
MNIST
MultiNLI
20
380
360
0.60
0.940
IntDiv
0.935
0.50
0.925
IntDiv
Number of classes
Number of classes
Figure Detecting mode dropping in image and text datasets. We evaluate and IntDiv datasets containing 500 examples drawn uniformly from between and ten classes: digits in MNIST and sentences genres in MultINLI. Compared to IntDiv, vs increases more consistently with the number of classes.
In Figure 5 we examine whether VS captures mode dropping in a controlled setting, where we have information about the ground truth class distribution. We simulate mode dropping by sampling equal-sized subsets of two classification datasets, with each subset S, containing examples sampled uniformly from the first categories. We perform this experiment one image dataset (MNIST) and one text dataset (MULTINLI; Williams et al. 2018), using simple similarity functions. We compare vs to the Internal Diversity (Int defined above.
19

---

Published in Transactions on Machine Learning Research (06/2023)
Model
IntDiv
VS
Original
0.855
403.9
AAE
0.859
501.1
CHAR-RNN
0.856
482.4
Combinatorial
0.873
536.9
IIM
0.871
250.9
JTN
0.856
489.5
Latent GAN
0.857
486.4
N-gram
0.874
479.8
VAE
0.856
475.3
Table 4: Inv and vs for gencrative models of molccules. The HMM has one of the highest IntDiv scores, but scores much lower on vs An analysis of 250 molecules from the HMM reveals VS more accurate this case. (See Figure 3
MNIST consists of 28x28-pixel images of hand-written digits, divided into ten classes. The similarity score we use is the cosine similarity between pixel vectors: k(z,,')= (z, [|=||[>]]] where x, are 282-dimensional vectors with entries specifying pixel intensities between 0 and 1. MULTINLI is a multi-genre sentence-pair classification dataset. We use the premise sentences from the validation split (mismatchedd), which are drawn from one of ten genres. We define similarity using the n-gram overlap kernel: for a given n, the n-gram kernel kn can be expressed as the cosine similarity between feature vectors (²(x), where (x)) is equal to the number of times n-gram i appears in x. We use the average of k(x,x') = - kn (xx,x');
The results (Figure 5) show that vs generally increases with the mumber of classes, even using these simple similarity scores. In MNIST (left), vs increases roughly linearly for the first six digits (0-5) and then fluctuates. This could occur if the new modes are similar to the other modes in the sample, or have low internal diversity. In MULTINLI (right), vs increases monotonically with the number of genres represented in the sample. In both cases, VS has stronger correlation with the number of modes compared to IntDiv.
C.2 Evaluating molecular generative models for diversity
We evaluate samples from generative models provided in the MOSES benchmark (Polykovskiy et al., 2020). using the first 2,500 valid molecules in each sample. Following prior work, our similarity function is the Morgan fingerprint similarity (radius 2), implemented in RDKit. 12 IntDiv ranks the HMM among the most. diverse models, while VS ranks it the least diverse (see Section 4.2
C.3 Evaluating image generative models for diversity
In Table we replicate the table described Section 4.4 and add an additional column, which evaluates diversity using the cosine similarity between pixel vectors as the similarity function.
should be understood the diversity with respect to a specific similarity function, in this case, the Inception ImageNet similarity. We illustrate this point in Figure G by comparing the top eigenvalues of the kernel matricos corresponding to the Inception similarity and the pixel similarity, which we calculate by resizing the images to 32x32 pixels and taking the cosine similarity between pixel vectors. Inception similarity provides a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel provides a simple form of visual similarity, with components corresponding to broad differences in lightness, darkness, and color.
C.4 Evaluating decoding algorithms for text for diversity
In Figure 7 we plot the relationship between VS and n-gram diversity using the MS-COCO captioning data and the n-gram overlap kernel described in Section 4.5) The figure shows that VS is highly correlated with n-gram diversity, which is expected given that our similarity function based on n-gram overlap. Nonetheless, there are some data points that the metrics rank differently. This because n-gram diversity conflates two
12RD: Soures Cheminfornaties. https:// hwtp://wwwwrdkit.orm
20

---

Published in Transactions on Machine Learning Research (06/2023)
Model
IS-
FID,
Prec-
Recy
VSr
VSrt
Model
IS-
FID,
Prect
Rec
VSIt
VSP
CIFAR-10
LSUN Bedroom 256x250
Original
19.50
3.52
Original
8.99
3.10
VDVAE
5.82
40.05
0.63
0.35
12.87
3.34
StyleGAN
2.55
2.35
0.59
0.48
8.76
3.09
DenseFlow
6.01
34.54
0.62
0.38
13.55
2.94
ADM
2.38
1.90
0.66
0.51
7.97
3.27
IDDPM
9.24
4.39
0.66
0.60
16.86
3.27
RQ-VT
2.56
3.16
0.60
0.50
8.48
3.67
ImageNett x4x64
LSUN Cat 256x256
Original
43.93
4.43
Original
15.12
4.58
VDVAE
9.68
57.57
0.47
0.37
18.04
4.24
StyleGAN2
4.84
7.25
0.58
0.43
13.55
4.53
DenseFlow
5.62
102.90
0.36
0.17
12.71
3.51
ADM
5.19
5.57
0.63
0.52
13.09
4.81
IDDPM
15.59
19.24
0.59
0.58
24.28
4.57
RQ-VT
5.76
10.69
0.53
0.48
14.91
5.83
Table 5: We evaluate samples from several recent models, measuring similarity using either Inception representations (VSy) or pixels (VSp). The pixel similarity score is the cosine similarity between pixel vectors, caleulated after resizing the images to 32x32 pixcls. The pixel similarity and Inception similarity scores do not always agree for example, if the images in sample represent variety of ImageNet classes by share similar color palette, might expect the sample have high Inception diversity but low pixel diversity. The pixel diversity scores are lower scale, indicating that this similarity metric is less capable of making fine- Ane-grained distinctions between the images in these samples.
Inception kernel
Pixel kernel
Least positive
Most positive
Least positive
Most positive
Kernel principal components
Kerne princle Componer
3
5
5
6
7
7
The choice of similarity function provides a way of specilying the notion of diversity that is relevant for given application. We project LSUN Cat images along the top eigenvectors of the kernel matrix, using either Inception features or pixels to define similarity. Inception similarity provides a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel captures visual similarity. For each eigenvector u, we show the four images with the highest and lowest entries in u. For both kernels, every similarity score positive, so all entries in the top eigenvector have the same sign; the images with the highest weights in this component have the highest expected similarity scores. The remaining eigenvectors partition the images along diflerent dimensions of variation.
Figure 6:
21

---

Published in Transactions on Machine Learning Research (06/2023)
5.0
Beam search
4.5
Diverse beam search
Human
N-gram VS
4.0
3.5
3.0
2.5
0.3
0.4
0.5
0.6
0.7
0.8
N-gram diversity
Figure VS is correlated with N-gram diversity. Each point represents a group of five captions for a particular image.
properties: the diversity of n-grams within a single sentences and the n-gram overlap between sentences. We highlight two examples in Figure S In general, the instances that n-gram diversity ranks lower compared to VS contain individual sentences that repeat phrases. On the other hand, n-gram diversity can be inflated cases when one sentence in the sample is much longer than the others, even the other sentences are not diverse.
High Vendi Score, low n-gram diversity:
Low Vendi Score, high n-gram diversity:
two men in bow ties standing next to steel rafter.
and woman cutting slice of cake by trees.
several men in suits talking together in a room.
a couple of people standing cutting cake.
an older man in a tuzedo standing nert to a younger
the dork with the earring slands next to the asion
man in tuzedo wearing glasses.
beauly who is way out of his league.
two men wearing tuzedos glance at each other.
newly married couple cutting cake in park.
older man in turedo sitting next to another younger man luzedo.
a bride and groom are cutting cake they smile.
Figure 8: Two sets of captions that receive different ranks according Vendi Score and n-gram diversity. We manually highlight some features contributing to the different scores. On the left, sentence contains repeated n-grams, which are penalized by n-gram diversity. On the right, one long outlier sentence contributes most of the n- n-grams for this group, greatly increasing the n-gram diversity.
C.5 Diagnosing datasets for diversity
Molecules We evaluate the diversity scores of molecules in the GoodScents database of perfume materials which has been used in prior machine learning research odor modeling (Sanchez-Lengeling et al. 2019). We use the standardized version of the data provided by the Pyrfume library. 14 Each molecule in the dataset is labeled with one or more odor descriptors (for example, "clean, oily, waxy" or "floral, fruity, green"). We form groups of molecules corresponding lo the seven most common odor descriptors, with each group consisting of 500 randomly sampled molecules. We evaluate VS using two similarity functions: the Morgan fingerprint similarity (radius 2), and the similarity between odor descriptors, defined as the cosine similarity between descriptor indicator vectors o(x), where (x) is equal to one if descriptor is associated with molecule x and zero otherwise.
The diversity scores are plotted in Figure The molecular diversity score and the odor-descriptor diversity scores are correlated, meaning that words like "woody" and "green" are used to describe molecules that vary
hhtth://www.leeooocceecoooommmmm chttpps//pyrfumme.oom
22

---

Published in Transactions on Machine Learning Research (06/2023)
44
SWeet
green
woody
fruity
Odor descriptors
herbal
floral
spicy
32
fatty
waxy
Molecular fingerprints
Figure 9: The Vendi Scores of samples containing 500 molecules with different scent labels, calculating diversity using two similarity funct ions: Morgan molecular fingerprint similarity, and the similarity weew odor descriptors. Each molecule is associated with one or more sweet"), and the odor-descriptor similarity is the cosine similarity between binary tag indicator vectors.
tags (e.g. "floral, fruity, green,
in molecular structure and also elicit diverse odor descriptions, while words like "waxy" and "fatty" are used for molecules that are similar each other and elicit similar odor descriptions. For example, the word "green" appears in tag sets such as "aldchydic, citrus, cortex, green, herbal, tart" and "floral, green, terpenic, tropical, vegetable, woody", whereas the word "waxy" tends to co-oceur with the same tags ("fresh, waxy"; "fresh, green, melon rind, mushroom, tropical, waxy"; "fruity, green, musty, waxy"). Molecules from the categories with the highest and lowest scores are illustrated in Figure 10
MultiNLI genres
Amazon reviews
Slate
3 stars
45
90
2 stars
1 stars
Fiction
SimCSE
SimCSE
4 stars
43
80
Travel
42
Telephone
41
Government
5 stars
70
375
380
385
390
395
375
376
377
378
379
380
381
N-grams
N-grams
Figure 11:
The Vendi Scores of samples containing 500 MultiNLI sentences with different genres (left) Amazon reviews with different star ratings (right), defining similarity using either n-gram overlap
SimCSE Gao et al.] 2021
Text In Figure 11. we evaluate the diversity scores of samples sentences with different genres, from the MultiNLI dataset [Wiliams et al. 22018), and Amazon product reviews with different star ratings (Keung et al., 2020), using either the n-gram overlap similarity SimCSE: (Gao et al. 2021 SimCSE is Transformer-
23

---

Published in Transactions on Machine Learning Research (06/2023)
Most diverse
Woody
Herbal
wo
9p
3o
do
Coxaininne now
Qu 1o
Least diverse
Waxy
Fatty
mad nn
thy
Oft
mmm Oixta
The scent categories in Goodscents dataset with the highest (top) and lowest (bottom) vendi score (vs), using the molecular fingerprint similarity. We show 100 examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores.
Figure 10:
based sentence encoder that achieves stato-of-the-ate scores on semantic similarity benchmarks. The model initialized from the uncased BERT-base model (Devlin et al. 2019) and trained with contrastive
24

---

Published in Transactions on Machine Learning Research (06/2023)
learning objective to assign high similarity scores to pairs of MultiNLI sentences that have logical entailment relationship.
In MultiNLI, both models assign the highest score to Slate, which consists of sentences from articles published on slate.com. SimCSE assigns a higher score to the "Fiction" category, possibly because it less sensitive to common n-grams (e.g. "he said"), that appear many sentences this genre and contribute to the low N-gram diversity score. In the Amazon review dataset, the 5-star reviews have the highest N-gram diversity but the lowest SimCSE diversity, perhaps because SimCSE assigns high similarity scores sentences that have the same strong sentiment. SimCSE assigns the highest diversity score to 3-star reviews, which can vary in sentiment.
Images Following the setting in Section 4.6 we evaluate two additional dataset, Fashion MNIST (Xiao et al. 2017) and CELEBA (Liu et al. 2015). We use the same similarity scores as in Section 4.6 Images CelebA are associated with 40-dimensional binary attribute vectors. We use these attributes as an additional similarity score, defining the altribute similarity as the cosine similarity between attribute vectors. These
illustrations highlight the importance of the choice of similarity function defining diversity metric.
Least diverse
Most diverse
Sneaker
Ankle boot
Bag
Shirt
Inception
Least diverse
Most diverse
Trouser
Coat
Bag
Sandal
Pixels
Figure 12: The categories Fashion MNIST with the lowest (left) and highest (right) Vendi Scores, defining similarity as the cosine similarity between either Inception embeddings (lop) or pixel vectors (bottom). We show 100 examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel.
25

---

Published in Transactions on Machine Learning Research (06/2023)
Least diverse
Most diverse
Wearing Necktie
Rosy_ Cheeks
Blurry
Wearing Hat
Inception
Least diverse
Most diverse
Blond_Hoe Hair
Rosy_ Cheeks
Wearing Hat
Mustache
Pixels
Least diverse
Most diverse
Rosy_ Cheeks
_akeu_
Big_ Nose
Male
Attributes
Figure 13: The attributes in CELEBA with the lowest (left) and highest (right) vs, defining similarity the cosine similarity between either Inception embeddings (top), pixel vectors (middle), or binary attribute vectors (bottom). We show 100 examples from each category, in decreasing order of average similarity, with the image at the top left having the highost average similarity scores according to the corresponding kernel. These examples illustrate the importance the choice similarity function for defining the notion of diversity that is relevant for given application. However, almost all choices of similarity functions show that the CELEDA dataset is more diverse for men than for women.
26


Reviews:
[{"review_id": "1QF8Umefj0", "reviewer": "Reviewer_AZcA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is remarkably well-written, especially the first half, and it is overall fairly easy to follow. The toy examples highlight and support key properties of the proposed metric (e.g., Figure 1) in a well-presented package. The Vendi score is based on structural properties of the data distribution, which is backed by theory.\n\nI see two major weaknesses:\n- The manuscript is missing a limitations section, or any discussion on drawbacks of the VS compared to prior methods.\n- The evaluation section is extensive in terms of covering several types of data, but does not feel conclusive in terms of the comparison with IS, FID, and PRD on image-generating models. It appears that VS mostly agrees with prior methods, and in in cases of disagreements, it is still not clear to me how to interpret them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose the Vendi score: a quantitative metric for evaluation diversity of empirial distributions of generative models or ground truth datasets. The Vendi score is calculated based on a similarity matrix of K samples of a distribution. After highlighting some properties of the proposed metric, the authors calculate and compare their metric with ID, FID, PRD, and IntDiv across several domainsl: image, text and molecules. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is missing a discussion of limitations of the Vendi score. I strongly urge the authors to include a comprehensive mention of possible shortcomings of the proposed method, both on its own, and compared to existing methods. For example, no-reference diversity measures can be gamed by simply introducing new modes to the distribution (that do not exist in the original dataset to be measured). Furthermore, the Vendi score leaves open the question of how to evaluate the \"precision\" of a model -- this is a point that IS and FID measure implicitly and PRD handles explicitly.\n\nSome more comments:\n\n- Sec. 1, Intro, first paragraph: the authors may want to include a brief definition for the term \"diversity\" here.\n- Sec. 2, 1st paragraph: the authors mention here that the Vendi score is a no-reference metric. It would be helpful here to discuss implications of this property, both advantageous and disadvantageous.\n- Sec. 4.2: minor grammatical mistake in \"In Figure 3, we highlight [...]\"\n- Table 2 (and also Table 3) could be replaced by plots for easier parsing. At the very least, the table cells could be highlighted for \"best\" and \"second best\". In its current form, the table is slow to read.\n- Table 2: For the currently empty \"Original\" rows, the authors could calculate the metrics in a bootstrapping experiment, or compare training with validation/test splits.\n- Sec. 4.4, 2nd paragraph: It would be interesting to calculate VS also with the L2 distance instead of cosine similarity, that would e.g. make the results more comparable to FID and PRD which use distances. At the very least, it would be interesting to see if the choice of using cosine similarity has a significant effect on the results.\n- Sec. 4.4, 3rd paragraph: ImageNet is likely the dataset with the highest (semantic) diversity even in absolute terms, so while I agree with the last sentence of this paragraph (the choice of the embedding is important), I believe it misses the crucial point that image embedding have been found to be fairly universal. In other words, I would expect ImageNet to yield higher VS than Cifer even when computing similarity in a feature space of a model trained on CIFAR.\n- Sec. 5, 1st paragraph: The authors stress here that the measured \"form of diversity\" can be specified for VS by changing the similarity function. While true, in practice, the embedding space will more often than not dictate the choice of the similarity function -- and importantly, the choice of the embedding space is also relevant for most baseline metrics such as FID and PRD.", "broader_impact_concerns": "I don't have significant concerns. The proposed method adds to the list of existing diversity measures and the fact that it can be applies to ground-truth datasets is a nice addition.\nThat said, the authors may want to include a curt Broader Impact Statement highlighting the shortcomings of the proposed metric for evaluating diversity on datasets. For example, the wrong choice of an embedding space for calculating similarities can reintroduce biases that lead to skewed diversity scores.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fZ60E5AXBZ", "reviewer": "Reviewer_57sL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The contribution is straightforward and has simplicity going for it. It is also unsupervised and depends on the representation and a similarity function.\n- The proposed metric has not been proposed in the context of measuring diversity in machine learning, despite similar objects appearing in many contexts. Although to the author's credit, whenever I thought to myself \"Oh, this is just concept A..\" soon enough I found the authors citing exactly that. So, clearly, the authors are well aware of this.\n- At some level, I can imagine that one can find the paper underwhelming, as one might think the idea is used in so many contexts (or at least is related to ideas in many contexts). But if one thinks about it a bit more -- the contribution in the context of diversity is certainly new. In that sense, I see the paper has to have a good place in the literature. \n- To continue on the above -- given the contribution, I would have certainly expected more extensive experiments. More specifically, some settings where diversity is important and can make a big difference. I would have liked to have the Vendi score placed in the optimization and then showing that using it helps for some downstream task. I am a little conflicted if in its current state, the contribution is enough to merit publication -- although I lean towards acceptance (for reasons described above). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In many machine learning applications, there is often a need to obtain diverse solutions, or to be able to use diverse inputs during the training process. Diversity is often task-dependent and thus hard to measure -- researchers and practitioners resort to some intuitive methods to enforce diversity (e.g. using a regularization employing the inverse of the dot product similarity for instance). However, the treatment of diversity is rarely systematic (although there are exceptions), and it seems to have never made it as a first-class citizen despite its importance. This paper takes an approach that will remind readers of a lot of things -- determinantal point processes (which work with a kernel matrix), permanental processes, spectral clustering (self-tuning and otherwise), generalized Schrodinger operators (and using them for cuts), graph cuts, graph entropies etc., but the actual use of the concept proposed in the paper in the context of diversity seems to have never happened. The main idea in the paper is to define a score of diversity between _data points_ as the entropy of the eigenvalues of a normalized pairwise similarity matrix and make an argument that this is a reasonable metric unsupervised metric for diversity, supported by some basic experiments. The proposed object is of course reminiscent of the von Neumann entropy (and divergences) and many concepts as I said and has a precedent in the species richness literature (btw, some of the species richness stuff has inspired work in machine learning, and of course applied statistics) -- which is taken as the main inspiration for the paper. One could summarize this paper in a couple of lines: For N points, construct the pairwise similarity matrix, normalize it in a particular way, and then find the entropy of its normalized eigenvalues and use that as a measure of diversity. Similar objects appear in many settings, but the main inspiration for the paper comes from species richness, and it seems to be the first to suggest to use this kind of object as a measure of diversity. Some informal arguments and basic experiments make a case for the proposed metric. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to hear first what the authors think about the experiments above, and then I can give more suggestions. I feel the experiments are too basic and only show some intuitive qualities (one of the figures about components resembles one in the spectral clustering tutorial by von Luxburg, for example).\n\nThe authors try their best to cite prior work diligently, although, I wouldn't quite say that diversity hasn't attracted attention. For example, if the authors see papers and theses such as this (and the dozens of references in them) https://apps.dtic.mil/sti/pdfs/AD1168004.pdf and https://home.ttic.edu/~pyadolla/papers/MBestModes.pdf and https://aclanthology.org/D13-1111.pdf they'll see that some have considered various notions (and even set it up as an optimization), although I will agree that they are not measuring diversity in the manner proposed in the paper. \n\nThe authors need to add some discussion (and this ties back to the point about experiments) about when the metric fails. Since it is intuitively cut-based (although the similarity function can be learnt), it will obviously have its drawbacks. But it is not clear to me what its failure modes are.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ML8NpI1cpU", "reviewer": "Reviewer_avdw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- This paper proposed to use von Neumann entropy of kernel to measure the diversity of samples, which could be a useful metric to consider in the case of evaluating diversity of generative models. Even though this metric is directly borrowed from existing literature, it used it in the context of diversity measurements.\n- This paper evaluated the metric on a number of different tasks.\n\nWeakness\n\n- The paper criticized existing metrics such as FIDs to depend on reference datasets and / or labels. I don't think this is a fair criticism, as those methods are trying to measure the similarity between the distribution and the reference distribution, instead of merely measuring diversity as the proposed metric does. For example, if a generative model generate purely random noises, it would have scored high with Vendi Score, but not under FIDs (say for natural images reference datasets). So it is unclear whether this is an advantage or disadvantage, and thus not really a good comparison.\n\n- The score critically depends on a kernel similarity metrics, or an embedding function to do the heavy lifting. It is therefore crucial to have some systematic studies on how robust the score is depending on different kernel functions.\n\n- There is no way to evaluate the proposed metrics is good or bad. This is a weakness of the paper, though I'm not sure if there is any way to get around it given the nature of the problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper suggested using the von Neumann entropy of kernel (Bach, 2022) to measure the diversity of samples in machine learning, and called this metric the Vendi Score. The paper evaluated this metric on a number of tasks such as measuring the diversity of the samples in generative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to the points mentioned in the \"weakness\" section above, please also add\n\n- Robustness study of the scores with respect to common factors, such as changing the sample sizes, re-sampling a new collection of examples from the same generative model under evaluation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q86xNAZ97a", "reviewer": "Reviewer_MiAU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea is quite simple, works with various distance metrics, and scales only linearly with the number of samples for certain kernels. \n- Empirical results demonstrate the effectiveness of the Vendi score across multiple domains and multiple metrics.\n- A fair comparison with existing works on diversity metrics is given.\n\n\nWeaknesses:\n\n- It is unclear whether this metric is consistent, i.e., with an infinite number of i.i.d. samples there is a limit for $VS(x_1, \\ldots, x_n) / n$.\n- It is also unclear how many samples we need to get a good estimate of VS. For example, if I know a distribution is more diverse than another, how many samples do I need to confidently say this is the case?\n- Sometimes the kernels chosen do not satisfy $k(x, x) = 1$, such as the probability product kernel.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the Vendi score as a metric for measuring the diversity of samples. This is defined based on the Shannon entropy of the eigenvalues of a similarity matrix. Empirical evaluations demonstrate that it is more effective at measuring certain aspects of diversity than other metrics. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, it is an interesting paper. I would suggest addressing some of the points in the weakness section above. \n\n- Are there any failure modes for Vendi score? If not, what kind of axioms for diversity are tested for this?\n- It would be nice to add a \"limitations\" section, in the sense that Vendi score only measures diversity and nothing else, so it has to be combined with other metrics for sample quality (such as FID).", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recognized the importance of the research direction taken by the paper and most reviewers agreed on the significance of the VS as a contribution. A reviewer raised the concern that the score definition was simple and hence might not make a significant contribution. I disagree, in that simple ideas, if novel and well executed can have quite some impact. The authors properly acknowledged the other fields and sources outside ML from which the VS is taking inspiration This is enough to provide the needed background for generative models users and researchers to understand the VS or build on it.

Additional concerns included the comparison of the VS w.r.t. other metrics to evaluate the quality of generated images such as FID and the request to run additional experiments, using different similarity metrics and optimizing the VS. The authors engaged with the reviewers comments and managed to address the majority of them convincingly, especially concerns regarding the comparison w.r.t. to FID&co which are there to highlight orthogonal aspects of the generated data. I agree with authors when they claim that the scope of this paper is not directly optimizing for VS, and that it can constitute a research question for future work. 

Ultimately, the majority of the reviewers agreed that the current version of the updated manuscript satisfies the acceptance criteria of the journal and decided to lean towards acceptance. The paper is therefore accepted as is.
Decision: Accept (poster)