Measuring Diversity in Synthetic Datasets
Authors: Yuchang Zhu
Huizhe Zhang
Bingzhe Wu
Jintang Li
Zibin Zheng
Peilin Zhao
Liang Chen
Yatao Bian
Venue: ICML 2025 poster (2025)
Keywords: diversity evaluation
synthetic data
LLMs
Primary area: general_machine_learning->evaluation
TL;DR: We propose a novel method to evaluate synthetic dataset diversity from a classification perspective.

Abstract: Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets—an aspect crucial for robust model performance—remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing methods. Code is available at: https://github.com/bluewhalelab/dcscore.

Measuring Diversity in Synthetic Datasets
Yuchang Zhu
Huizhe Zhang'
Bingzhe Wu? Jintang 3: Zibin Zheng Peilin Zhao 56 Liang Chen
Yatao Bian 66
Abstract
1. Introduction
Large language models (LLMs) have shown exceptional per- formance across a range of fields, such as chatbots (Achiam et al., 2023), computer programming (Gu, 2023), and rea- soning (Yuan et al., 2024). Inspired by their remarkable capacities, some research (Ye et al., 2022; Abdullin et al., 2024; Ding et al., 2024) employs LLMs as dataset generators to mitigate the shortage of training data. Although gener- ated data facilitates model optimization, recent studies (Yu et al., 2024; Lee et al., 2023) suggest that a lack of diversity within the dataset- measured by the variation between sam- ples (Long et al., 2024) -ay lead to performance degra- dation in some scenarios. Although previous studies (Yu et al., 2024; Wang et al., 2022) leverage well-designed gen- eration strategies create highly diverse synthetic datasets, they consistently neglect to evaluate the diversity of these datasets. Additionally, a principled diversity evaluation met- ric serves not only to guide LLM generators in creating more diverse data but also extends its utility data selection (Cao et al., 2023), quantifying augmentation performance (Yang et al., 2024a), and assessing mode collapse (Dan Friedman & Dieng, 2023). Thus, diversity evaluation method for synthetic datasets is becoming increasingly important.
Large language models (LLMS) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. How- ever, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-rrmaing
significant chal- lenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset di- versity from a classification perspective. Specifi- cally, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets re- veal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demon- strate that DCScore substantially reduces com- putational costs compared to existing methods. Code is available at: https:/ https://github. com/ bluewhalelake dcscore
Since the diversity evaluation of synthetic datasets remains under-explored, a natural solution is to directly employ di- versity evaluation methods from related fields, such as natu- ral language processing (NLP) (Khurana al., 2023) and machine learning (ML) (Jordan & Mitchell, 2015). Specif- ically, efforts to measure diversity within these domains can be summarized into three categories: N-gram-based based method (Zhu et al., 2018; Mishra et al., 2020), Reference- based method (Heusel et al., 2017; Cifka et al., 2018), and method (Du Black, 2019; Zhang et al., 2024). The n-gram-based method evaluates diversity through n-gram statistics, e.g., distinct-n (Li et al., 2015), fo- cusing textual form rather than semantic content. To align the diversity criteria with human judgment, the reference- based method has emerged as promising alternative. This approach employs a reference distribution or data as an ap- proximation of human judgment and calculates the similar- ity between the evaluated data and the reference data (Holtz- man et al., 2019). However, collecting reference data can be both time-consuming and may introduce potential biases.
Work done during an internship at Tencent AI Lab Corresponding author School Computer Science and En- gineering. Sun Yat-sen University. Guangzhou, China 'School of Artificial Intelligence. Shenzhen University, Shenzhen, China "School of Software Engineeringg Sun Yat-sen University, Zhuhai, China "Zhuhai Key Laboratory of Trusted Large Lan- guage Models, Zhuhai, China "School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China "Tencent Al Lab, Shenzhen, China DDppartment of Computer Science, Na- tional University of Singapore, Singapore. Correspondence to: Yatao Bian <biianyt@ comp.nus. edu.es. Liang Chen <chen- liangó@ sys.....u.......................
Transformation-bar
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Measuring Diversity in Synthetic Datasets
Drawing inspiration from deep representation learn- ing (Butepage et al., 2017; Zhang et al., 2021), the transformation-basee method evaluates diversity by first mapping the data into the representation space and then performing diversity summarization (Tevet & Berant, 2020). For data mapping, various embedding functiins can be employed to facilitate the transformation, with a popular approach being the use of sentence transformers such as Sentence-Bert (Reimers, 2019) and SimCSE (Gao et al., 2021). Owing to the versatility of embedding functions, transformation-basedd methods can simultaneously consider extensive aspects, such as semantics and style, to encode data representations, thereby providing a more comprehen- sive evaluation of diversity. However, this type of method hindered by suboptimal computational efficiency caused by the high-complexity diversity summarization, such as eigenvalue computation (Dan Friedman & Dieng, 2023).
idea behind DCScore is to treat diversity evaluation sample classification task, enabling the capture of mutual relationships among samples.
We theoretically validate that DCScore adheres to several intuitive axioms suggested by (1 (Leinster & Cob- bold, 2012), demonstrating its superiority. Addition- ally, we theoretically validate the lower complexity of DCScore under general kernels.
Extensive experiments show that DCScore exhibits a stronger correlation with multiple diversity pseudo- truths compared to baseline metrics. We also perform a computational cost experiment confirm the lower computational cost of DCScore.
2. Related Work
In nutshell, existing diversity evaluation methods in NLP and ML suffer from inherent limitations in the synthetic dataset evaluation scenario. To effectively evaluate the di- versity of synthetic datasets, the following challenges must be tackled: (1) Holistic Analysis. The diversity evaluation of synthetic datasets is holistic analysis task, necessitating consideration of the impact of each sample on the final eval- uation results. (2) Axiomatic Requirements. Prior research has suggested several axioms that diversity metrics should ideally satisfy. To ensure reasonable evaluation, a well- established diversity evaluation should exhibit properties corresponding to these axioms. (3) Lower Computational Costs. As a growing volume of synthetic datasets, a diversity evaluation method with lower computational costs is highly desirable to ensure data quality and model performance.
We give a brief literature review of diversity evaluation methods. Limited by space, further related works on LLM dataset generators and application of diversity evaluation methods can be found in Appendix A.
2.1. Diversity Evaluation Methods
With the development of LLMs as dataset generators, the diversity evaluation of synthetic datasets has become a chal- lenging task and remains under-explored in recent evalua- tion studies (Liang et al., 2022; tatsu lab, 2023; Ribeiro et al., 2020). The most comparable diversity evaluation research can be traced back to studies in NLP and ML, which can be summarized into the n-gram-basse method (Mishra et al., 2020), reference-baseeee based method (Heusel et al., 2017), and causformation-basee method (Lai et al., 2020).
To sum up, the essence of diversity is associated with the identification of differences between samples, and the abil- ity to distinguish these differences a key element in the classification process (Quine, 1969). Motivated by this observation, we propose a synthetic dataset diversity eval- uation method from a classification perspective, namely, DCScore. Notably, DCScore tackles the three aforemen- tioned challenges. Firstly, DCScore treats the evaluation of each sample in the synthetic dataset as an independent classification task, providing holistic analysis. Secondly, theoretical verification demonstrates that DCScore satis- fies four axioms outlined in (Leinster & Cobbold, 2012), including effective number, identical samples, symmetry, and monotonicity. Lastly, both empirical and theoretical evidence suggest that DCScore effectively evaluates the diversity of synthetic datasets, while demonstrating lower computational costs compared to existing methods. Our contributions can be summarized as follows:
A-gram-based Methods. The n n-gram g-gaambbased method is the most popular lexical diversity evaluation method, leveraging n- grgaamm to capture differences in sentence form (Yu et al., 2024). Commonly used n-gram-based based diversity metrics include distinct n-grams (distinct-n) (Song et al., 2024), self- BLEU (Shu et al., 2019), and ROUGE-L (Wang et al., 2022: Padmakumar & He, 2023). However, this type of method only captures differences in text form, thereby overlooking differences in other aspects such as semantics and style.
Reference-based Methods. Diversity evaluation is a sub- jective task, leading to a reliance on human judgment. Con- sequently, the reference-based method evaluates diversity by comparing the distribution of the evaluated data that of reference dataset (Heusel et al., 2017). MAUVE (Pillutla et al., 2021) exemplifies this idea by employing a divergence- based metric to capture correlations with human judgment. Regarding the natural language inference (NLI) training set as the reference dataset, (Stasaski & Hearst, 2022) first trains an NLI model to infer the relationship between pairs of generated texts and then calculates diversity based on
We propose DCScore, a classification-Dusee diversity evaluation method for synthetic datasets. The core
2

---

Measuring Diversity in Synthetic Datasets
these inference results. Due to the challenges in collect- ing reference datasets, recent studies (Kynkäänniemi al., 2019; Le Bronnec et al., 2024) propose evaluating diversity through precision and recall. (Naeem et al., 2020) intro- duces density and coverage as solutions to the susceptibility of precision and recall to outliers. Despite these advance- ments, the reference-basee method remains significantly constrained because of the need for reference datasets.
Table I. Categories of Deup- In the column of "Examples", texts belonging to Doup are highlighted in gray while texts associated with T are marked using an underline.
Generatios
Extmppess
Question: It's buring movie. The sentiment of the movie
(xd (xc)
(xc)
rrview is
Answer: Negutive.
(x) + (R))
Question: The movie
positive sentiment
Answer: Glod film!
Question: Following are etamples of movic reviow and their sentiment latels. Gonerate sauples accooding to these examples.
Transformation-base Methods. The transformation- based (Lee et al., 2023) method leverages well-designed models to generate representations of the evaluated data. Then, the diversity of these representations is summarized using techniques such as clustering (Du & Black, 2019) and eigenvalue computation (Dan Friedman Dieng, 2023) of VendiScore (Dan Friedman & Dieng, 2023). In line with VendiScore (Dan Friedman & Dieng, 2023), RKE (Jalali et al., 2023) introduces an informatiiontheoreeti method for evaluating diversity in multimodal distributions, while FKEA (Ospanov et al., 2024) enhances the computational ef- ficiency of both VendiScore and RKE. Owing to the superior performance of representation learning, this type of method considers various aspects of the evaluated data, including semantics, form, and style, offering greater flexibility com- pared to the other two methods. However, its dependence high-compha high-cooplexity summarization techniques, such as eigen- value computation, limits its scalability in evaluating the diversity of synthetic datasets.
(Tree) + (7.) (Tres)
Example boxing movie. (Negative);
Ecample Oh, wonderful movie' (Posnive))
Answer: A meaningful movic. (Pusitive
where T can be texts describing the generation task, such as annotation. Deup. which comprises a small number of seed samples or unlabeled inputs, serves as supplementary materials to facilitate data augmentation. For example, we want LLMs to perform an annotation task for sentiment labeling. such as determining whether the sentiment pos- itive or negative. If we assume Dsup to be "It's a boring movie.", the description of T could be "The sentiment of the movie review is" D {TT) { y:) is the generated dataset with n samples, where T; = (Iz, Vi), Ti and yi are the input-output sample, the input text, and the output text, respectively. Let TD denote the downstream task of D, when Tp is the question-answent Y: represent the question and answer, respectively.
task, x; and
In summary, existing methods primarily focus on NLP and ML fields and are challenging to apply directly to synthetic dataset diversity evaluation. Different from the above-mentioned studies, our work is dedicated to the holis- tic diversity evaluation of synthetic datasets. Additionally, to ensure flexible evaluation, our work aims to evaluate diversity-sensitie components that impact the performance of trained models in terms of diversity.
It is worth noting that not all components in DD are generated by M, which is related to the category of Daup. As shown in Table 1, Deup can be divided into three categories, namely input text, output text, and seed samples. In Table 1, "->" and Treed represent the direction of generation and seed samples, respectively. For example, "x, + yy" signifies that, given input text z, denoted as Daup. M processes suup and T, generating the output text y/-
3. Preliminaries
3.2. Problem Formulation
3.1. LLM as Dataset Generator
The diversity evaluation of the dataset is sample rich- ness evaluation problem. Based the generation scenarios presented in Table 1, we find that in certain downstream tasks, the diversity of some components in the synthetic dataset does not influence the performance of the trained models. Conversely, the diversity of other components sig- nificantly impacts model performance. We refer to these components, whose diversity influences performance, as diversiry-sensiteee components, denoted as T. For exam- ple, the input text x; in sentiment classification tasks is the diversity-sensitiv component. Conversely, the output text yi, which represents the sentiment label of the sample and is typically a numerical label (e.g., 0 or 1), does not influ- ence model performance in terms of diversity. Thus, the output text cannot be considered as the diversity-sensitiv
Since the exceptional performance of LLMs, previous works (Dai et al., 2023; Yoo et al., 2021) employ LLMs as a dataset generator or for data augmentation purposes. LLMs significantly reduce the cost of label annotation and data collection (Tan et al., 2024), and in several tasks, even outperform human annotators (Gilardi et al., 2023). While some studies attempt to use LLMs to generate datasets from scratch, it is challenging task for LLMs. In most cases, pre-trained LLM. denoted as M, takes the data suu to be augmented and the generation task T as input, and outputs the augmented dataset D. Formally, this process can be formulated as follows:
D D-M(T,Dum D.up)
3

---

Measuring Diversity in Synthetic Datasets
component. It should be underscored that siversity-sensitive components vary across downstream tasks. The diversity evaluation of synthetic datasets can be transformed into the diversity evaluation of diversity components,
the overall diversity score. Formally, the aforementioned process can be formulated as follows:
H =
K (H),
(3)
Given a synthetic dataset D = (T: we define (Ta) as a collection of diversity-sensitive components. The prob- lem of diversity evaluation of D can be defined as follows:
where Kernel(-) calculates pairwise similarity, with viable options including inner product and RBF kernel. For D more expressive embedding function can be employed, such as one trained using well-designed framework like Sentence-Bert (Reimers, 2019).
DiversityScore <- Evaal (1))(())))=11)
(2)
- where Eval(.) is the diversity evaluation function, which
Based on K, DCScore leverages a classification function with K, denoted as fi((),, to compute the classification prob- ability matrix P E "= Here, natural option for fx(+) is the Softmax function. For Tú, the probability that T; is classified as category Cy can be formulated as follows:
takes (Ti)k- as inputs and outputs the diversity score.
4. Present Work
In this section, we first introduce our proposed method DCScore from a classification perspective. Then, we present the properties of DCScore followed by theoretical proofs. Finally, we provide detailed complexity analysis of DCScore and the vranformationnbasee counterpart.
P(c P(c= (( = P[i,5] = [K(( (..
exp KK ((.TT
(4)
Σ, exp ((,))/T)
where T is temperature hyperparameter to control the clas- sification resolution. A smaller r amplifies sample similar- ity differences, implying a higher classification resolution, while larger value yields the opposite effect.
D.Score Measuring Diversity from a Classification Perspective
Due to the intrinsic nature of measuring sample differences in diversity evaluation, it is natural to evaluate diversity as a classification task. Consequently, we propose DCS core, which formulates diversity evaluation as a sample classifi- cation task. Specifically, the difference between samples can be measured through an r-classification task, where evaluating n sample datasets involves n n-classification tasks, with each sample corresponding to a distinct cate- gory. As shown in Figure 1, DCScore consists of three stages: text representation, pairwise similarity, and diversity summarization. According to the problem formulation in section 3.2, DCScore outputs the diversity of synthetic datasets by evaluating diversity-sensitivv components.
According Eq. (4), if the evaluated dataset exhibits high sample richness, indicating greater diversity, each sample is likely to be classified into its own category. If the diversity is low, all samples will be randomly classified. Based on P. corc calculates diversity of 'D as the trace of P, which can be formulated as follows:
Definition 4.1 (DCScore). Let D = [T: denote the synthetic dataset with n samples, and let {T.);= represent set of diversity-sensitivve components within (Ti) De- note P. as the classification probability vector of T. By conducting the classification task for all T. and obtaining the probability matrix P = [P1, P2...., DCScore for 'D is defined as the trace of P:
Let D (T)]_ ((.;, denote a synthetic dataset comprising input-output samples, and (T.) represents the dnversity-sensitiiee components. DCScore adheres to the paradigm the transformation-basee method to evaluate the diversity of D. Specifically, given T, DCScore first applies an embedding function & to extract the sample representation h, (Ti). For all samples in D, we obtain the sample representation matrix H E Raxd across all samples, where d denotes the dimension of sample representations. Subsequently, DCScore utilizes a kernel function Kernel(-) to calculate a kernel matrix K, where K E R and entry K[ē, J] represents similarity between T. and テ,. From a classification perspective, Kji can be considered as the logit of T, being classified into category Cj- where ej corresponds to T.. Thus, a sample that is highly distinguishable from others (i.e., correctly classified into its 'own' category with high probability) contributes more to
DCScore(D) = tr(P) Pl.
(5)
Notably, the process described above is just one implemen- tation of DCScore, with other potential implementations to be explored in future work.
4.2. Properties of DCScore
We provide theoretical proof that DCScore satisfies several axioms (Leinster & Cobbold, 2012) defined for a principled diversity metric. DCScore meets four axioms: effective number, identical samples, symmetry, and monotonicity ax- ioms. These axioms ensure a reasonable and robust diversity evaluation. The matched axioms of DCS cororee are outlined below, while their proofs are detailed in Appendix B.
4

---

Measuring Diversity in Synthetic Datasets
C
C2
On
31
++
0.88
0.88
22
Embedder +
Kernel
Classifier
0.98
F2
0.96
Σ
Jn
F
0.77
D
Diversity-sensitive
Probability Matrix
DCScore (D)
Text Representation
Pairwise Similarity
Diversity Summarization
Figure The overview DCScore. DCSeore consists of text representation, pairwise similarity, and diversity summarization stages.
Effective number:
Diversity should be defined as the effective number samples in dataset, ranging from
Table 2. Complexity analysis of DCScore and VendiScore.
O Kernel represents the complexity of the kemnel function.
to n. DCScore meets this axiom, as evidenced by its
General Kernels
Inner Product
behavior: DCScore equals 1 when all samples in D are identical and equals n when all samples are distinct.
Pairwise Similarity
VendiScore
-esraa))
OlsP)) O(wid)
DCScore
VendiScore
Ola O(a
O(a)
Identical samples: Given two identical datasets D1 and D2, the diversity of DD' generated by merging these two datasets remains unchanged. The values of DCScore are the same across DD, D2, and 'D',, i.e.,
Summarization
DCScore
O(i²)
VendiScore
Oln Oured n2)
Oid'n + ( O(dPn)
Total
DCScore
Oln Ouerna n
O(
stages. In the pairwise similarity stage, the computation of pairwise similarities results in a complexity of O(n²) for DCScore. When combined with the complexity Okernel of the general kernel computation, DCScore exhibits a total complexity of On2 2 Okernel). In the summarization stage, DCScore has complexity of (n²²) due to the softmax op- eration. Consequently, the overall complexity of DCScore for general kernels is O(n² Okennel + n²). In contrast, VendiScore has a total complexity of O(n² Okernel n²), where the pairwise similarity stage is identical that of DCScore, while the summarization stage incurs plexity of (n²)) due to the eigenvalue computation. Thus, for general kernels, DCScore demonstrates lower complex- ity than VendiScore.
DCScore(D,) = DCScore(Dz) = DCScore(D'), (6)
Symmetry: Diversity remains constant regardless of the order of the samples, exhibiting permutation invari- ance. Let T--) denote the permutation function for the sample order, DCScore remains unchanged for any sample permutation of i.e.
DCScore(D) = DCScore(z(D)).
The diversity of a dataset decreases the sample similarity increases. Given two datasets D and D2, and a new sample Tx+1, where the samples in Dj and DD2 are entirely different, and DCScore(D)) = (C)) = n. If T is more similar to the samples in D2 than those in D1 and is added to both datasets, then for the merged datasets D' and D'z DCScore satisfies the following equation.
Monotonicity:
However, when the inner product is employed as the kernel function and > d, VendiScore can significantly reduce the complexity by replacing the pairwise similarity X with X X, where X E gnxd This results in complexities of (dd'n) ) for the pairwise similarity stage and (dd) for the summarization stage. In this regard, DCScore has a complexity of Onn2d + n²), which is slightly worse than that of VendiScore. We can leverage efficient techniques, such as those proposed in (Shim et al., 2017), (Milakov Gimelshein, 2018), and (Wen al., 2023), reduce the computational cost of DCScore. Compared to VendiS- core, DCScore maintains lower complexity in most cases, as empirically validated in Section 5.3 and Appendix D. While VendiScore has lower complexity with the inner prod- uet kernel, experiments in Appendix D. indicate that its computation time similar to DCScore. Ensuring low
DCScore(D)) DCScore(D2).
4.3. Complexity Analysis
We provide time complexity analysis of DCScore in Ta- ble 2. We compare DCScore with VendiScore due to their similarity, finding that DCScore has lower computational complexity with non-linear kernels.
Denoting Okenel as the complexity associated with general kernels (i.c., kernels other than linear kernels), we analyze the complexity in the pairwise similarity and summarization
5

---

Measuring Diversity in Synthetic Datasets
Table 3. Correlation (Spearman's 's p) between To and diversity eval-
Evaluated Dataset D
Diversily Pseudo-truth
uation methods on datasets generated by different settings (Zero- shot or Fow-shor). Spearman's 's p varies between - and +1, with implying correlation. Best results are indicated in bold.
Diversity Evaluator
Diversity Score
Zen-slot sotting
Rra-chul witting
Corretation
Methods
clastiiiattion
Story completion
clacctication
Story completion
705
700
09887
means Inenta
0.9638
0.954
0.7004
0.7273
0.9662
Figure 2. Experimental settings of correlation evaluation.
VandiSuee
0.996
0.9818
0.9870
0,9922
0.9999
038857
0.9857
BCBooze
0.9961
03779
0.9844
0.993
0.9909
0,98K1
0.9837
0.9974
computational complexity across multiple kernels is more advantageous than achieving it with a single kernel.
Baseline Methods. We compare DCScore with three base- line methods detailed in Appendix E, i... Distinct-n (Li et al., 2015), K-means Inertia (Du & Black, 2019), and VendiScore (Dan Friedman & Dieng, 2023).
5. Experiments
We conduct experiments to verify the effectiveness of DCScore by examining correlation, computational cost,
5.2. Correlation Evaluation
We investigate the correlation between the diversity eval- uation of DCScore and diversity pseudo-truths, such as Tg. human judgment, and LLMs evaluation. We compare DCScore with all baselines on self-generated datasets.
hyperparameter sensitivity, and further probing. Limited by space, we provide additional experiments in Appendix D.
5.1. Experimental Settings
To verify the effectiveness of DCScore, we conduct a se- ries of correlation experiments following the setup in (Tevet & Berant, 2020). As shown in Figure 2, the core idea of our experimental evaluation is correlate the diversity mea- surement results of DCScore with diversity pseudo-truths, such as the Softmax temperature To of dataset generation, human judgment, and LLMs evaluation. Specifically, we evaluate / generated datasets to obtain I diversity scores and then calculate the correlation with diversity pscudo-truths. To calculate the correlation between measured diversity scores and diversity pseudo-truths, we employ Spearman's 's P (Spearman, 1961), measure of rank correlation rang- ing from 11 to 1, with higher absolute values indicating stronger correlations. Limited by space, we present detailed experimental settings in Appendix C.
5.2.1. CORRELATION WITH Tg
Evaluation on self-generated datasets. Previous works (Caccia et al., 2018; Tevet & Berant, 2020; Chung et al., 2023) have demonstrated positive correlation be- tween Tg and the diversity of generated texts, making Ty as reasonable diversity pseudo-truth. LLMs with lower To generate less diverse content, whereas higher Tq values yield more diverse content. Thus, we evaluate the performance of DCScore on self-gencrated datasets with varying Tg ranging from 0.2 to 1.2 at 0.05 intervals. We present more information about self-generated datasets in Appendix C.I.I. Table 3 displays the correlation results of all methods. All methods accurately capture the true diversity of generated datasets, as demonstrated by high Spearman's P values. DCS core performs on par with VendiScore while provid- ing better scalability for larger synthetic datasets, as dis- cussed in Section 5.3. DCScore outperforms all baseline methods under the few-shot setting across all datasets, high- lighting its effectiveness. K-means Inertia exhibits the weak- est correlation on the text classification dataset generated by the 13B model under the zero-shot setting, potentially due to its sensitivity to the number of cluster centroids. Over- all, DCScore outperforms all baselines in most cases, and its evaluation results exhibit a strong correlation with the diversity pseudo-truth according (Akoglu, 2018).
Datasets. We utilize two categories of datasets in our ex- periments: self- eeff-genrrated datasets and publicly available generated datasets. Self-generated datasets generated through two data generation strategies (Li et al., 2023): zero- shot and few-shot settings. We generate datasets for two natural language processing tasks: text classification and story completion. Additionally, we utilize three publicly available existing datasets, including SST2 (Socher et al., 2013), Yelp (Zhang et al., 2015), and AG News (Zhang al., 2015), and their petrrroptt-uugmentee version (Yu et al., 2024). Detailed information about these datasets can be found in Appendix C.
Visualization. We further provide visualization of the di- versity evaluation results for DCScore. For each generated dataset, we prompt LLMs to produce 10 distinct answers corresponding to a single prompt, forming an evaluation batch. We then evaluate diversity using the batch evaluation protocol outlined in Appendix C.2.3.
To generate datasets through zero- shot and few-shot settings, we employ two commonly used LLMs as our dataset generators, including AJama2-13B (13B) and llmaa-70BB (70B) (Touvron et al., 2023).
Generation Models.
6

---

Measuring Diversity in Synthetic Datasets
Text Clssiiiration
Story Completion
100
10
Table 4. Pairwise correlation (Spearman's p) between human, tem-
DCS500e
perature (7y), and DCScore. DCScore indicates strong corre- lation with human judgment.
8
shot.
Zow shot
Zero shot_ 138
Zero stot 133
Story-Fow
Story-Zero
Text-Few
Text-Zero
Zerowshot 708
Tewe-hoot_ 700
Zanoeebre__ 708
Twa-sbot, 70m
Hlman-UCScoors
0.9040g001
0.7915_a.1 0.7870
0.8798
0.25
Tempersture (Tg)
0.50
1.00
1.25
0.25
Temperature (Tp)
0.50
1.00
1.25
To DCScore
T Human
0.92761ao
0.908640.07
0.919410.0
0.7829
0.8400_0.18
0,9770 0.02
0.9255 1o.0s
0.8971
Figure 3. Diversity evaluation of DCScore on datasets generated using varying Tg- DCScore shows strong correlation with Tyx indicating its effectiveness in evaluating the dataset diversity.
Table S. Pairwise correlation (Spearman's 's p) between GPT-4, tem- perature (7y), and DCScore. DCScore indicates strong corre- lation with GPT-4 evaluation results.
Story-Few Story-Zero
Text-Few
Text-Zero
Based on the verse dataset may yield a diversity score of 10. As shown in Figure 3, DCScore exhibits a strong positive correlation with Tg. In most cases, when Tg > 0.75, DCScore scores a generated dataset with a diversity value of approximately 10. For text classification, the 13B generation model under the few-shot setting demonstrates a distinct diversity change pattern compared to others. This phenomenon stems from the inherent limitations of the 13B generation model, which struggles to effectively comprehend and execute more in- tricate or multi-faceted instructions. As a result, generated datasets only exhibit marginal improvements in diversity.
settings, completely di-
GPT-4-055com
0.6057
0.901020.0
0.6131
0.905220.00
Ty-DCooore F--GPT-
0.6757
0.878220.08
0.5714=6.27
0.9336
0.9086ga.or
0.7829 20.10
0.8400
0.8971
5.2.3. CORRELATION WITH LLM EVALUATOR
To further verify the effectiveness of DCScore, we inves- tigate the evaluation correlation between DCScore and LLMs. Following the setting in Section 5.2.2, we employ GPT-4 conduct pairwise comparisons between two gen- crated datasets with different Tg. These generated datasets are identical to those used in Section 5.2.2. Based on the pairwise comparison results, we obtain the diversity rank- ing outcomes. Regarding GPT-4 evaluation results as the diversity pseudo-truth, we report the pairwise evaluation correlation between DCScore, GPT-4, and Ty in Table 5, We observe that DCScore exhibits strong correlations with GPT-4 and Ty in zero-shot settings. By comparing the results of "T, "CScoe"" and "Ty GPTT-4" we find that DCScore outperforms the GPT-4 evaluator in terms of correlation with Ty in zero-shot settings. Regarding the correlation per- formance in few-shot settings, we notice lower correlations of all baseline methods compared to pero-shot settings. We guess that this phenomenon is related the distributions of the generated datasets. Although DCScore exhibits lower correlations (about 0.6) with GPT-4, this result can still be considered a strong correlation according to (Akoglu, 2018).
5.2.2. CORRELATION WITH HUMAN JUDGMENT
Diversity evaluation is subjective task, and an ideal method should align well with human judgment. Thus, we inves- tigate the correlation between DCScore and human judg- ment. We enlist three human evaluators to perform pairwise diversity comparisons among datasets with varying Ty val- ues and report the diversity ranking by averaging the win rate across evaluators. We conduct all evaluations five times to report average results. Limited by space, we present details of human evaluation in Appendix C.2.5.
Table 4 presents pairwise correlation between human judg- ment, Tg. and DCScore. Table 4 indicates a strong cor- relation between human judgment and Tg. supporting the use of human judgment as a diversity pseudo-truth. Based on this observation, DCScore performs better in set- tings: Story-Few (story completion data generated under the few-shot setting) and Text-Zero (text classification data generated under the zero-shot setting). This is confirmed by higher human-DCScore correlation in these two set- tings. For Story-Zero and Text-Few settings, we observe more identical content in the initial portions of the diversity- sensitive components within an evaluation batch. In these cases, human evaluators tend to disregard the identical con- tent and base their judgments on the latter sections. How- ever, DCScore is affected by the identical content, result- ing in a lower pairwise correlation. Despite this, the cor- relation remains strong, as demonstrated by previous stud- ies (Akoglu, 2018).
In summary, DCScore exhibits a strong correlation (Spear- man's p 2 0.6), with three diversity pseudo-truths: Tg. hu- man judgment, and LLMs evaluation, thereby verifying the effectiveness of DCScore.
5.3. Computational Cost
The computational cost crucial in diversity evaluation methods, especially with the increasing sample sizes of synthetic datasets. For a fair comparison, we only present the computation times of transformation-based methods: DCScore, K-means Inertia, and VendiScore. We trun- cate the text length of three datasets (SST2/Yelp/AG News- AttrPrompt, the AttrPrompt (Yu et al., 2024) augmented version of SST2/Yelp/AG News datasets) to 50 tokens and
7

---

Measuring Diversity in Synthetic Datasets
Yol Troreg
Al News Altefronpt
Text Classifiration
Story Completion
1.0
Woudiforne DCSSone
WoddSoor
VeudiScera
2im 00
0.5
0.5
Pew-shot_110
Pew-sbot,
Sample Number
2013
0000
2100
Number
0000
2000 Namber
-0.5
Few shot 70B
Zem-shot 138
-0.5
Zemehhut 138
Pew shot, 708
Zoro shot NB
Zoro shot 708
Figure 4. Computation times under different sample sizes.
0.0001
000
0.01
a.
0b
10
o 0.0001
2000
0.03
0.3
0.
3 10
DCScore outperforms all baselines in computational cost.
Figure 5. Hyperparameter sensitivity analysis w.r. 7 on self- generated datasets.
Table Comparison of computation time between DCScore and VendiScore on SST2.
Kemds
SST2
Sample sum
4k
Sk
14k
52k
64k
Appendix D.2. In summary, DCScore demonstrates lower computation time compared to the dfffiieneyiimrroeed ver- sions of VendiScore and RKE in most cases.
Inner produc
VentiSare:
DCScore
39
4.65aa-24
10,02211
2.84
20
19,02
429921.9
41247
76.19
VendiSeom
16036
DCScora
s..3...0
9
21,30,075
46.57
117.6
Poly
VeadSSume
pctcora
10.37
12.72
20.1221m
46.25
123.513
5.4. Hyperparameter Sensitivity
record the computation times of three methods with varying sample sizes in the range of (100, 500, 1000, 2000, 4000).
According to Eq. (4), the temperature (7) in the Softmax function is a critical hyperparameter that affects classifi- cation resolution. To investigate this, we conduct hy- perparameter sensitivity analysis of DCScore w.r.t. on self- leffeeenerteed datasets used in Section 5.2.1. We vary T within the range of {0.0001, 0.001, 0.1, 0.5, 1, 10). Fig- ure presents hyperparameter sensitivity results on datasets for text classification and story completion tasks. Overall, lower values result in lower Spearman's p. even indicating negative correlation, while higher values do the oppo- site. From Eq. (4), higher 7 reduces pairwise similarity differences, leading to more uniform distribution of classi- fication probabilities for each sample. This phenomenon can be regarded as a lower classification resolution, i.c., the clas- sification function fK has poorer discrimination power. Fur- thermore, the correlation result of the 13B generation model under the few-shot setting for the text classification task remains stable despite variations in т. This phenomenon has the same explanation as in Figure 3.
As shown in Figure 4, we repeat the experiments five times to report the final results. DCScore and K-means Iner- tia exhibit nearly identical computation times. However, DCScore significantly outperforms K-means Inertia cor- relation with Tg. as evidenced in Section 5.2.1. Compared to VendiScore, DCScore demonstrates speed advantage of approximately 16%, or more than one second, when pro- cessing 4000 samples. Analyzing the time complexity of these two methods, and disregarding the selection of the kernel function, we find that for a dataset with n samples, where n >> d is not satisfied, the computational complexity of DCScore in diversity summarization is (nn)) due to the softmax computation. In contrast, VendiScore requires finding the eigenvalues of an X matrix, resulting in com- putational complexity of (n Consequently, DCScore offers significantly lower time complexity than VendiScore while sacrificing little in diversity evaluation performance. However, as detailed in the complexity analysis shown in Section 4.3, when n > d and inner products are used as the kernel function, the total complexity of VendiScore can be reduced to (d) Thus, we evaluate computational costs on larger datasets, i.e., satisfying n >> d. As shown in Table 6, DCScore exhibits notable advantage in compu- tational efficiency when using non-linear kernels, e.g., RBF and Poly kernel. We present additional experimental results and a more in-depth analysis Appendix D.
5.5. Further Probe
1.5.1. DOWNSTREAM TASK TRAINING
To investigate the correlation between DCScore and down- stream task training, we train text classification models using reff-generated datasets under zero-shot and few-shot set- tings. We vary Ty of self-generated datasets within the range of (0.2,0.7, 1.2). More details of training datasets and hy- perparameters are presented in Appendix C.I.1 and C.2.2. respectively. As shown in Table 7, models trained on more diverse datasets achieve better accuracy, likely due their improved generalization capabilities (Gontijo-Loop et al., 2020). Notably, the increased training data diversity makes model fitting more difficult, necessitating additional epochs to achieve optimal accuracy. Additionally, the diversity evaluated by DCScore has a similar trend to the accuracy
recent study (Ospanov al., 2024) employs the random Fourier features framework to decrease the computational cost of mntrepy-based diversity evaluation methods, such as VendiScore (Dan Friedman & Dieng, 2023) and RKE (Jalali et al., 2023). We follow the experimental settings of Ta- ble 6 and leverage different random seeds for data sampling. Limited by space, we present the experimental results in
8

---

Measuring Diversity in Synthetic Datasets
Table 7. Downstream task training performance and diversity eval-
Table 8. Comparison of evaluation stability between DCScore and VendiScore on datasets with duplicate samples. n denotes the number of duplicate samples, and A.P. represents AttrPrompt.
uation on self- self-generated datasets with T {0.2, 0.7, 1.2).
Accuracy
DCSeore
=,20,2
7,70.7
1,=1.2
T,=0.2
7,=0.7
1,21.2
Datasets
Methods
vcl
1=100
1=1000
n: 1000
n=:000
10:5000
Zero-shot
89.10
89.70
90.37
481.76
1745.42
2082.42
S.T2 VendiScore
DCScere
5937.00
11.31
5936.99
11:31
5537,110
11.31
5936.81
11.32
5935.73
11.29
5936.99
11:31
Few-shot
70.07
73.19
73.41
1376,43
1958.16
2047.90
Vilp A.P.
VandiScore
5939.94
8.22
5940.01
8.22
5939 82
8.22
5039.91
8.24
5940.21
8.24
5939.94
8.22
AG News A.P.
VauliScore
DCsoore
5098.04
40.71
5998,00
40.70
5958 04
40. 70
5098.00 04
40.63
5097.75
40.60
5998, 04
40.71
Normelized Diversity
1.0
InceptionV3
Normelized tzzed Diversity
1.0
DinoV2
0.8
0.8
ples. Specifically, we randomly select n samples from the evaluated datasets to serve duplicates and incor- porate them into the original datasets. In our experi- ments, we use SST2-AtrPrompt, Yelp-AtrProopt, and AG News-AtrPrompp as evaluated datasets, setting n to (0, 10, 100, 1000, 3000, 6000). Table 8 presents the diver- sity evaluation values of DCScore and VendiScore, using inner product as the kernel function. These values can be interpreted as the number of effective samples. As shown in Table 8, we observe that both DCScore and VendiScore maintain stable evaluation results as the number of duplicate samples increases. This indicates that both methods are robust in terms of the impact of duplicates. Additionally, the results for VendiScore tend to be lower, suggesting an underestimation of dataset diversity.
0.6
0.6
0.4
VendiScore
0.4
VendiScore DCScore
DCScore
0.2
0.2
2
10
10
Label Num.
Label Num.
Figure 6. Diversity evaluation on colored MNIST using two differ- ent embedding functions (Inception V3 and Dino V2). A higher label number signifies greater dataset diversity.
performance in the zero-shot setting, further demonstrat- ing the effectiveness of coScore. Detailed experimental results and further analysis are provided in Appendix D.3.
5.5.2. DIVERSITY EVALUATION ON VISUAL MODALITY
Similar to text generation, there is increasing attention image generation. Thus, we further verify the effectiveness of DCScore on the image dataset evaluation. Specifically, following the setting of a recent study (Ospanov et al., 2024), we leverage the colored MNIST (Deng, 2012) as the evalu- ated dataset and use the label number as the diversity ground truth. Here, larger label number indicates higher diver- sity of the evaluated dataset. We conduct comparison between DCScore and VendiScore, using inner product as the kernel function, while adopting Inception V3 and Dino V2 as the embedding functions. For DCScore, T is set to Figure illustrates the diversity evaluation results on the image dataset, with diversity results normalized by dividing by the highest result of each method. We observe that both DCScore and VendiScore exhibit positive correlation with the number of labels. When using the Inception V3 model as the embedding function, DCScore demonstrates higher correlations with the number of labels compared to VendiScore, as evidenced by a line that is closer to the di- agonal. Additionally, DCScore presents more consistent results across different embedding functions compared to VendiScore, indicating stable performance with respect to the embedding function.
6. Conclusion
In this work, we investigate the diversity evaluation of syn- thetic datasets, a topic systematically under-explored in existing research. To this end, we present DCScore, a di- versity evaluation method from a classification perspective. DCS core regards the holistic diversity evaluation as the classification task at the sample level, thereby facilitating the capture of mutual relationships between samples. We pro- vide theoretical guarantees demonstrating that DCScore meets the axiom requirements (Leinster & Cobbold, 2012) for a principled diversity evaluation method. Experiments on synthetic datasets reveal that DCScore exhibits better correlations with various diversity pseudo-truths. including Tg- human judgment, and LLMS evaluation. Meanwhile, DCS core exhibits significantly lower computational cost compared to transformation-basee counterparts. Finally, we hope our work encourages future research to pay more at- tention to the diversity of synthetic datasets and promotes the wider application of these datasets.
Limitation: Although we have verified the effectiveness of DCScore for unimodal data evaluation, DCScore is not yet capable of directly evaluating multimodal data, such as image-text pairs. A key challenge lies in the extraction and fusion of representations from multimodal data. As multimodal LLMs continue to enhance their generative ca- pabilities, evaluating the diversity of multimodal -enerraed datasets emerges as an important research problem.
5.5.3. DIVERSITY EVALUATION ON DATASETS WITH DUPLICATE SAMPLES
To further investigate the limitations of DCScore, we con- duct diversity evaluations on datasets with duplicate sam-

---

Measuring Diversity in Synthetic Datasets
Acknowledgements
conference on computer vision and pattern recognition, pp. 6158-6166, 2017.
The research is supported by the National Key R&D Pro- gram of China under grant No. 2022YFF0902500, the Guangdong Basic and Applied Basic Research Founda- tion, China (No. 2023A151501 1050), and Tencent Al Lab RBFR2024004.
Caccia, M., Caccia, L.., Fedus, W., Larochelle, H., Pineau, J., and Charlin, L. Language gans falling short. arXiv preprint arxi 1...04949 2018
Cao, Y., Kang, Y. Wang, C., and Sun, L. Instruction min- ing: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307. 222209,,... 2023.
Impact Statement
This paper presents a novel and efficient method for mea- suring the diversity of synthetic datasets, aiming to advance fields of machine learning and LLMs. The ability accu- rately evaluate the diversity of synthetic data has significant implications for various applications, including but not lim- ited to improving the robustness and fairness of ML models, enhancing the quality of synthetic data used in training, and fostering innovation in generative models.
Chen, D., Lee, C., Lu, Y., Rosati, D., and Yu, Z. Mixture of soft prompts for controllable data generation. arXiv preprint arXiv:2303.01580 2023.
Chung, J. J. Y., Kamar, E., and Amershi, S. Increasing
diversity while maintaining accuracy: Text data genera- tion with large language models and human interventions. arXiv preprint arXiv:2306.041400 2023.
From an ethical perspective, our work contributes to the responsible development of Al by providing tools that can help identify and mitigate biases in synthetic data. Ensuring diversity in datasets is crucial for creating equitable AI systems that perform well across different demographic groups and use cases. By promoting diversity, our method can help prevent the perpetuation of existing biases and support the creation of more inclusive technologies.
Cifka, O., Severyn, A., Alfonseca, E., and Filippova, K. Eval all, trust few, do wrong to none: Comparing sentence generation models. arXiv preprint arXiv: 100::1804.7772. 2018
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Ran- daugment: Practical data augmentation with no separate search. arXiv preprint 1909::1909.13111 2(4):7, 2019.
Dai, H., Liu, Z., Liao, W., Huang, X., Cao, Y., Wu, Z., Zhao, I.., Xu, S., Liu, W., Liu, N., et al. Auggpt: Leverag- ing chatgpt for text data augmentation. arXiv preprint 1XXi:2302.13007 2023.
For societal impact, our approach has the potential impact wide range of industries, from healthcare to finance, by enabling the generation of more representative and diverse datasets. This can lead to more accurate and fair decision- making processes, ultimately benefiting society as a whole.
Dan Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. Transactions on machine learning research, 2023.
Overall, while our primary goal to advance fields of ma- chine learning and LLMs, we believe that our work also addresses important ethical considerations and has the poten- tial to contribute positively to society by promoting fairness and inclusivity in AI systems.
Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE signal processing magazine, 1916)11411442, 2012.
Dieng, A. B., Ruiz, F. J., Blei, D. M., and Titsias, M. K. Pre- scribed generative adversarial networks. arXiv preprint arXiv: 190.04403, 2019.
References
Abdullin, Y., Molla-Aliod, D., Ofoghi, B., Yearwood, J.,
and Li, Q. Synthetic dialogue dataset generation using I1m agents. arXiv preprint arXiv:2401.174611 2024.
Ding, B., Qin, C., Liu, L., Chia, K., Joty, S., Li, B., and Bing. L. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10455 2022.
Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Ding, B., Qin, C., Zhao, R., Luo, T., Li, X., Chen, G., Xia, W., Hu, J., Luu, A. T. and Joty, S. Data augmentation using 1lms: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.0200
2024.
Akoglu, H. User's guide to correlation coefficients. Turkish journal of emergency medicine, 18(3):91-93, 2018
dos Santos, V. G., Santos, G. L.,, Lynn, T., and Bena- tallah, B. Identifying citizen-related issues from so- cial media using llm-based data augmentation. In International Conference on Advanced Information Systems Engineering, pp. 531-546 Springer, 2024.
Butepage, J., Black, M. J., Kragic, D., and Kjellstrom, H. Deep representation learning for human motion pre- diction and classification. In Proceedings of the IEEE
10

---

Measuring Diversity in Synthetic Datasets
Du, W. and Black, A. W. Boosting dialog response gener- ation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Huber, M., Luu, A. T., Boutros, F., Kuijper, A., and Damer, N. Bias and diversity in synthetic- face recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 0215-6226, 2024.
Evuru, C. K. R., Ghosh, S., Kumar, S., Tyagi, U., Manocha, D., et al. Coda: Constrained generation based data augmentation for low-resource nlp. arXiv preprint arXiv:2404.00411 2024.
Jalali, M., Li, C. T., and Farnia, F. An information- theoretic evaluation generative models in learn- ing multi-modal distributions. Advances in Neural Information Processing Systems, 36:9931-9943, 2023.
Gao, T. Yao, X., and Chen, D. Simese: Simple trastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 2021.
Jordan, M. I. and Mitchell, T. M. Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255- 260, 2015.
Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120 (30)::230001110,, 2023.
Khurana, D., Koli, A., Khatter, K., and Singh, S. Natural language processing: state of the art, current trends and challenges. Multimedia tools and applications, 82(3): 3713-3744, 2023.
Contijo-Lopes, R., Smullin, J. Cubuk, E. D., and Dyer, E. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08977, 2020.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. Advances in neural information processing systems, 32, 2019.
Goodfellow, I., Nouget-Abadic J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020..
Lai, Y.-A., Zhu, X., Zhang, Y., and Diab, M. Diversity, den- sity, and homogeneity: Quantitative characteristic metrics for text collections. arXiv preprint arXiv:2003.08529, 2020.
Gu, Q. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 2201-2203, 2023.
Le Bronnec, F., Vérine, A., Negrevergne, B., Chevaleyre, Y., and Allauzen, A. Exploring precision and recall to assess the quality and diversity of llms. 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Gupta, H., Scaria, K., Anantheswaran, U., Verma, S., Par- mar, M., Sawant, S. A., Mishra, S., and Baral, C. Tar- gen: Targeted data generation with large language models. arXiv preprint arXiv:2310.17877 2023.
Lee, A., Miranda, B., Sundar, S., and Koyejo, S. Beyond scale: the diversity coefficient data quality metric demonstrates Ilms are pre-trained on formally diverse data. arXiv preprint arXiv:2306. 13840, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Lee, S., Kim, H., and Lee, J. Graddiv: Adversarial robust- ness of randomized neural networks via gradient diversity regularization. IEEE Transactions Pattem Analysis and Machine Intelligence, (5(2):24452266) 2022.
Holtzman, A., Buys, J., Du L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint 10Xvv:194.00951 2019.
Leinster, T. and Cobbold, C. A. Measuring diversity: the importance of species similarity. Ecology, 93(3):477- 489, 2012.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu,
Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.0966 2021.
Leng, X., Chen, Y., Tang, X., and Bian, Y. Rich feature learning via diversification. In Workshop on Spurious Correlation and Shortcut Learning: Foundations and Solutions, 2025. URL https://openrev// net/ / forum? didiidddada2h0.....
Huang, S., Zhao, J., Li, Y., and Wang, L. Learning pref- erence model for 1lms via automatic preference data generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9187-9199 9199, 2023.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. objective function for neural conver- sation models. arXiv preprint arXiv:1510.0300 2015.

---

Measuring Diversity in Synthetic Datasets
Li, Y., Ding, K., Wang, J., and Lee, K. Empowering large language models for textual data augmentation. arXiv preprint arXiv:2404, 2024a.
and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv: 11t1::600.001966
2016.
Li, Z., Zhu, H., Lu, Z., and Yin, M. Synthetic data genera- tion with large language models for text classificatton: Po- tential and limitations. arXiv preprint arXii:2310.07889, 2023.
Nacem, M. F., Oh, S. J., Uh, Y., Choi, Y., and Yoo, J. Re- liable fidelity and diversity metrics for generative mod- els. In International conference on machine learning, pp. 7176-7185. PMLR, 2020.
Li, Z., Si, L.,, Guo, C., Yang, Y., and Cao, Q- Data augmen- tation for text-based person retrieval using large language models. arXiv preprint arXiv:2405. 11971, 2024b.
Ospanov, A., Zhang. J., Jalali, M., Cao, X., Bogdanov, A., and Farnia, F. Towards a scalable reference- free evaluation of generative models. arXiv preprint arXiv:2407.0221 .2w.61 2024.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M.. Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211 222221111111,,.11111111111111111 2022.
Padmakumar, V. and He, H. Does writing with lan- guage models reduce content diversity? arXiv preprint arXiv:2309.05 1966 2023.
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint Xiv: 194iv:1904.08779, 2019.
Liu, S., Wang. T., Bau, D., Zhu, J.-Y., and Torralba, A. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14286-14295, 2020.
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Mea- suring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816-4228 2021.
Liu, Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 2019.
Long, L., Wang, R., Xiao, R., Zhao, J., Ding, X., Chen, G., and Wang. H. On lims-driven synthetic data genera- tion, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15122 2024.
princeton nlp. ussupssincce--ett------aaaa 2021. URL attps://nugte http:s///huggggggfccc.co//////es unsup simcse- bert-base--ucaseeeee
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arxiv: 2017.
Quine, W. V. Ontological relativity and other essays, 1969.
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y.. and Potts, C. Learning word vectors for senti- ment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Ore- gon, USA. June 2011. Association for Computational Linguistics. URL https: http:://aaaaaattnnaaaeo org/ P1 P11-1015.
Reimers, N. Sentence-bert Sentence embeddings using siamese bert-networks. arXiv preprint 1908. 2019.
Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.041 18, 2020.
Sahu, G. and Laradji, I. H. Mixsumm: Topic-based data augmentation using 1lms for low-resource extractive text summarization, arXiv preprint arXiv:2407.07341, 2024.
Mahmoudi, G., Behkamkia, B., and Eetemadi, S. Zero-shot stance detection using contextual data generation with llms. arXiv preprint arXiv:2405.1163) 2024.
Samuel, V., Aynaou, H., Chowdhury, A. G., Ramanan, K. V., and Chadha, A. Can Ilms augment low-resource reading comprehension datasets? opportunities and challenges. arXiv preprint arXiv:2309.122 12426. 2023.
Milakov, M. and Gimelshein, N. Online normalizer cal- culation for softmax. arXiv preprint AXXiv:800500286 2018.
Seeger, M. Gaussian processes for machine learning. International journal of neural systems, 14(02):69-106, 2004.
Mishra, S., Arunkumar, A., Sachdeva, B., Bryan, C., and Baral, C. Dqi: Measuring data quality in nlp. arXiv preprint arXiv:2005.00811 2020.
Shim, K., Lee, M., Choi, I., Boo, Y., and Sung, W. Svd- softmax: Fast softmax approximation on large vocabu- lary neural networks. Advances in neural information processing systems, 30, 2017.
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra,
D., Vanderwende, L.,, Kohli, P., and Allen, A corpus
12

---

Measuring Diversity in Synthetic Datasets
Shu, R., Nakayama, H., and Cho, K. Generating diverse translations with sentence codes. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1823-1827, 2019.
Yang, S., Guo, S., Zhao, J., and Shen, F. Investigating the effectiveness of data augmentation from similarity and diversity: An empirical study, Pattern Recognition, 148: 1 110204, 2024a.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Yang. S., Liu, X., Dong, X., and Fu, B. Mini-da: Improving your model performance through minimal data augmen- tation using Ilm. In Proceedings of the Fifth Workshop on Data Science with Human-in- -n-then- Loop (DaSH 2024), pp. 25-30, 2024b.
Ye, J.. Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, T., and Kong, L. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint
Song. F. Yu, B., Lang, H., Yu, H., Huang, F., Wang, H., and Li, Y. Scaling data diversity for fine-tuning language models in human alignment. arXiv preprint arXiv:2403.1 1114, 2024.
079922 2022.
Ye, J., Xu, N., Wang. Y., Zhou, J., Zhang, Q. Gui, T. and Huang, X. Llm-da: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arxi::2402. 14566 14568, 2024.
Spearman, C. The proof and measurement of association between two things. 1961.
Stasaski, K. and Hearst, M. A. Semantic diversity in di- alogue with natural language inference. arXiv preprint arXiv:2205.01997 2022.
Yoo, K. M., Park, D., Kang, J., Lee, S.-W., and Park, W. Gpt3mi:: Leveraging large-scale language models for text augmentation. arXiv preprint urXiv:2104.0882 2021.
Tan, Z., Beigi, A., Wang, S., Guo, R., Bhattacharjee, A., Jiang, B., Karami, M., Li, J., Cheng, L.,, and Liu, H. Large language models for data annotation: A survey, arXiv
Yu, L.. Zhang, W., Wang, J., and Yu, Y. Seggan: Se- quence generative adversarial nets with policy gradi- ent. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
preprint arXiv:2402.13444 2024.
tatsu lab. Alpacaeval An automatic evaluator for instructionn-folloiing language models, 2023. URL
Yu, Y., Zhuang, Y., Zhang. J., Meng. Y., Ratner, A. J., Kr- ishna, R., Shen, J., and Zhang, C. Large language model attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
https:// ow- fo-eeeeeeeleeeeeearrr.r.
eval evalltab=readmen hwwprbbrrrraeeeo-----eeeeeeeeeee
ov-
Tevet, G. and Berant, J. Evaluating the evaluation of di- versity in natural language generation. arXiv preprint arXiv:2004.02990 2020.
Yuan, J., Tang, R., Jiang. X., and Hu, X. Large language models for healthcare data augmentation: An example on patient-trial matching. In AMIA Annual Symposium Proceedings, volume 2023, pp. 1324. American Medical Informatics Association, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- alc, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint htt.::3377.09288.. 2023.
Yuan, L., Cui, G., Wang. H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078 2024.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with ceff-generated instructions. arXiv preprint Xix::2221 22..:22212..06661 10560, 2022.
Zhang, H. mixup: Beyond empirical risk minimization. arXiv preprint 111.:11110009112.... 2017.
Wen, Y., Liu, W., Feng, Y., Raj, B., Singh, R., Weller, A., Black, M. J., and Schölkopf, B. Pairwise similar- ity learning is simple. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5308- 5318, 2023.
Zhang, T., Peng, B., and Bollegala, D. Improving diversity of commonsense generation by large language models via in-context learning. arXiv preprint arXiv:2404.16807, 2024.
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D. C. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint Aixiv:230.. 1382, 2023.
Zhang, X., Zhao, J., and LeCun, Y. Character-level con- volutional networks for text classification. Advances in neural information processing systems, 28, 2015.
13

---

Measuring Diversity in Synthetic Datasets
Zhang, Y., He, R., Liu, Z., Bing, L.., and Li, H. Boot- strapped unsupervised sentence representation learn- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 5168-5180, 2021.
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang. J., and Yu, Y. Texygen: A benchmarking platform for text gen- cration models. In The 41st international ACM SIGIR conference on research & development information retrieval, pp. 1097-1100, 2018.
14

---

Measuring Diversity in Synthetic Datasets
A. Additional Related Work
Limited by space, we provide a literature review of the LLM dataset generator and application of diversity evaluation methods as follows.
A.1. LLM Dataset Generator
Recent studies (Ding et al., 2022; Chung et al., 2023) leverage LLMs to augment existing datasets or generate a dataset from scratch, demonstrating the effectiveness in improving dataset quality and reducing data collection costs. Generally, efforts to employ LLMs as dataset generators can be categorized into three strategies: Prompt- gridde (Li et al., 2023), Dataset-guided (Ye et al., 2022), and Instruct-guided (Samuel et al., 2023).
Prompt-guided and Dataset-guided Strategies. The prompt-guided strategy, a prominent data augmentation approach using LLMs, involves designing task-specific prompts to guide LLMs to augment data in few-shot (Yoo et al., 2021) or zero-shot (Mahmoudi et al., 2024) manner. Due to its simplicity and effectiveness, subsequent works extend this strategy to various scenarios, such as medical (Yuan et al., 2023), person retrieval (Li et al., 2024b), and social media scenario (dos Santos et al., 2024). However, simple prompt engineering has limitations in fully exploiting the capabilities of LLMs, leading to the development of multi-level prompt designs (Ye et al., 2024) and targeted sample augmentation (Yang et al., 2024b). To further harness the potential of LLMs, the dataset-guide then trains a task-specific model to annotate unlabeled data (Sahu & Laradji, 2024). The dataset-guided strategy aims to approximate the distribution of targeted scenarios, but it is currently only applicable to text classification tasks.
strategy employs LLMs to generate training set and
Instruct-guided Strategy. Previous studies (White et al., 2023) indicate that the design of prompts significantly impacts the performance of L.MMs, spurring research into the instruct-guidet guride strategy. Generally speaking, the instruct-guided strategy leverages LLMs to generate instructions that guide another LLM in dataset generation (Evuru et al., 2024). These instructions typically relate context (Samuel et al., 2023), criteria (Huang al., 2023), and tasks (Wang et al., 2022). To further improve the quality of instructions, efforts have been concentrated on selecting optimal instructions (Li et al., 2024a), integrating soft instructions (Chen et al., 2023), and implementing self-correction mechanisms (Gupta et al., 2023).
In a nutshell, LLMs are employed to generate or augment datasets through prompt engineering and multi-step strategies, which encompass various application scenarios and downstream tasks. Meanwhile, the diversity of synthetic datasets emerges as a critical factor in measuring data quality. In our work, we focus on the diversity evaluation of synthetic datasets derived from any dataset generation strategies.
A.2. Application of Diversity Evaluation Methods
Extending beyond diversity quantification, these methods demonstrate wider utility, such as quantifying augmentation performance and evaluating mode collapse. Thus, we present literature review of other applications of the diversity evaluation.
Quantifying Augmentation Performance. As data augmentation becomes an essential component in the training of deep neural networks (Zhang, 2017; Park et al., 2019), researchers gradually explore a better quantification of the quality of data augmentation. Some studies (Cubuk et al., 2019) suggest that the effectiveness of data augmentation arises from the increased diversity of the data. Inspired by this observation, a series of studies have introduced diversity evaluation metrics into the performance assessment of data augmentation strategies. Specifically, they consider diversity as one aspect of evaluating the quality of augmented data, thereby determining the effectiveness of data augmentation. For instance, (Gontijo-Lon 2020) utilizes the fundamental idea that models find it more challenging to fit more diverse data, comparing metrics such as training loss and training time before and after augmentation to assess diversity. Similarly, (Yang et al., 2024a) evaluates diversity by examining the eigenvalues and eigenvectors of the similarity matrix of samples before and after augmentation.
et al.,
Evaluating Mode Collapse. Generative adversarial networks (GANs) (Goodfellow et al., 2020) suffer from a well-known phenomenon called mode collapse, which can result in a lack of diversity in the generated samples (Dieng et al., 2019). Consequently, existing studies assess mode collapse by evaluating the diversity of the generated samples. For instance, a common approach is train MNIST classifier and then count the number of unique classes predicted for the generated samples. Following this paradigm, VendiScore (Dan Friedman & Dieng, 2023) compares the generation diversity of PresGAN (Dieng et al., 2019) and Self-conditioned GAN (Liu et al., 2020). Additionally, some studies (Yu et al., 2017: Zhu et al., 2018; Caccia et al., 2018) employ different metries to evaluate the diversity of generated samples from GANs.
15

---

Measuring Diversity in Synthetic Datasets
Other Applications. In addition to the aforementioned applications, diversity evaluation metrics have valuable applications in various areas, including sample selection for datasets (Cao et al., 2023), enhancing adversarial robustness (Lee et al., 2022) and out of distribution robustness (Leng et al., 2025), and eliminating biases within datasets (Huber et al., 2024).
B. Proof of Properties of DCScore
theoretically confirm that DCScore satisfies several intuitive axioms pointed out by previous studies (Leinster Cobbold, 2012), thereby demonstrating its role as a principled diversity evaluation method.
Effective number (Restated): Diversity should be defined the effective number of samples in dataset, ranging from 1 to n. DCScore meets this axiom, as evidenced by its behavior: DCScore equals when all samples in 'D are identical and equals n when all samples are distinct.
Proef. For DCScore, if all samples dataset are the same, the probability any given sample being classified into all categories the same, i.e., for all i.j {1, 2, ..., n). P/i P[i,]] Then, have DCScore Σ 1. If all samples in the dataset are distinct, for all j, {1, ..., n.. PÍū, 1. In other words, the classification function confidently predicts that T, belongs to the i-th category. Then, we have DCScore tending
Identical samples (Restated): Given two identical datasets Dj and D2, the diversity of the synthetic dataset D generated by merging these two datasets remains unchanged. The values of DCScore are the same across D1, D2, and D', i.e.,
DCScore(D;) = DCScore(D;) DCScore(D)
(9)
Proof. Assuming that D1 and D2 are completely identical, and the samples within each dataset are entirely different, i.e., DCScore(D) (Dro DCScore(D) (D2)) = n.. Let -P,, .n.... P2n] denote the probability matrix of the merged dataset D' Di DD2 (T.) -1 For
< n. Tnti. where T/ D1, Tn+s D2. Consequently, for each diversity-sensitie component T, in D' i] = Pli, n + +i]= Finally, DCScore(D') E
However, the assumption that all samples in the dataset are completely different may be too stringent. We further provide proof with more relaxed assumption. Suppose that D2 and D2 are completely identical, with K2 and K₂ denoting the kernel matrices for D, and D2, respectively. In this case, we have K
K2 as follows:
h
kl
22
k2
k?
2.2
k2.2
k
2-2
k
k2
K = K =
(10)
Kn,
k2
According to Eq. (4), for the i-th snrersityy-snsiitine component in D1 and D2, the probability of being classified as category e can be computed as follows:
R2 Σ,
P [ = i)
(11)
Σ, h
For a merged dataset D' = Dí UD2 = (T.)2m, when 1 S í S n, have Tatis where Ta E D1, Tats Dz. Since the newly added data samples do not affect the pairwise similarity, the kernel matrix K for D' can be formulated as follows:
2:
22
k1
k-
222
2.2
K'
(12)
k
k
k
2-2
k
mm
16

---

Measuring Diversity in Synthetic Datasets
Analogous Eq. (11), for < i n, the probability of the i-th diversity-sensiitive component in DD' being classified category G be computed as follows:
k Eg Ko + ΣS k
kt,
ki
(13)
2 kb.
2 2 k2.
P [7, 2 P]]
For n + 1 S S 2n, we obtain the same result as depicted in Eq. (13). Consequently, the diversity of D can be
computed as follows:
2z
DCScore(D')
Σ P [i
1 Σ
P
Pi
1 +
2 Σ P2
(14)
Σ
Pi Σ P
DCScore(D,) = DCScore(D)).
Symmetry (Restated):
Diversity remains constant regardless of the order of the samples, exhibiting permutation invariance. Let (T)) denote the permutation function for the sample order, DCScore remains unchanged for any sample permutation of D, i.e.
DCScore(D) = DCScooe(TT())).
(15)
Proof. According Eq. (4), the order of samples does not affect the classification task. Thus, the diagonal elements of P remain unchanged, indicating the symmetry property of DCScore.
Monotonicity (Restated): The diversity of a dataset decreases as the sample similarity increases. Given two datasets Dj and D2, and new sample Ta+1, where the samples in D2 and DD are entirely different, and DCScore (Di) = DCScore(D,) = n. Tn+1 is more similar to the samples in D: than to those in D2 and is added to both datasets, then for the merged datasets Dj and D2, DCScore satisfies the following equation.
DCScore(Di) DCScore(D.).
(16)
Proof. For Dj = (Ti ....... Tn+1) and D'2 ( T2 Tn+1). we have S(T Tn+1) < S(T, Tn+1)
for any i,j = (1, n}. Here, S(,,)) -) is the similarity function. this regard, the classification function f(-) exhibits lower confidence when classifying dataset Diz, resulting in lower probability that the i-th sample is classified into the i-th class, thereby leading to P [ :] PD (i, i]. Then, the following formula is satisfied:
D [i > Pp; li. :] DCScore(D)) > DCScore (Dz),
(17)
where pp Pa are the probability matrix of D/. D'z, respectively.
C. Experimental Settings
C.1. Datasets
Two types of generated datasets, including self-generated datasets and publicly available generated datasets, are employed in our experiments. We provide detailed information on these datasets below.
17

---

Measuring Diversity in Synthetic Datasets
C.I.I. SELF-GENERATED DATASETS
In our experiments, we utilize three different self-generated datasets. We employ two commonly used LLMs as our dataset generator, including Llama2-13B (13B) and Llama2-70B (70B) (Touvron et al., 2023). To prompt LLMs to generate datasets, we design two prompts corresponding to Zero-shot and Few-shot generation settings, respectively. Additionally, self- self-generated datasets involve two natural language processing tasks: text classification and story completion. We set the maximum number of generated tokens to 100 and 30 for text classification and story completion tasks, respectively. The detailed generation information is offered as follows.
Generation Settings. We use three different generation settings across different experiments. The detailed information shown as follows.
Datasets on Section 5.2.1, Section 5.4, Appendix D.4, and Appendix D.5. We generate 21 sub-datasets corresponding to different Tg by varying Ty from 0.2 to 1.2 with 0.05 intervals. For each sub-dataset, we employ LLMs (13B or 70B)
to generate sets of 10 responses per context. Specifically, each sub-dataset consists of 100 samples.
Datasets on Section 5.2.2 and Section 5.2.3. We employ the 70B model to generate sub-datasets corresponding to different Ty by varying Ty from 0.2 to 1.2 with 0.2 intervals. Each sub-dataset includes samples corresponding to a context. To repeat experiments five times, we use five different contexts to prompt the 70B model to generate sub-datasets for each Tg-
Datasets on Appendix 5.5.1 and Appendix D.3. In zero-shot or few-shot settings, we utilize the 70B model to generate three sub-datasets for the text classification task, corresponding Ty (0.2, 0.7, 1.2), respectively. Unlike other settings that provide only one example, in this experiment, adopt few-shot setting where four examples and their corresponding labels are given, including two positive examples and two negative examples. Each sub-dataset contains 3,000 samples. and context is employed to prompt the 70B model to generate five samples. To train text classification models on each sub-dataset, we randomly split 2, 2,100 samples to the training set for each sub-dataset and gather the remaining 900 samples into the testing set across all three sub-datasets. Consequently, we construct a test set comprising 1,800 samples.
Prompt Settings. Following the setting of (Li et al., 2023), we design different prompts for zero-shot and few-shot settings, respectively. Here, we provide a seed example for the few-shot setting, whereas the zero-shot setting does not receive any. For the text classification task under the zero-shot setting, we require LLMs to generate movie reviews with htt//Attion/rama//omeeyy///
topics. Each movie review contains a single sentiment of either positive negative, which is regarded as the text label. For the story completion task, we require L.LSs to complete the story according to the given context. The detailed prompt setting is provided in Table 9.
In Table 9, "[style)" will be replaced with one topic within {Sci-fi, Action, Drama, Comedy, Romance) and "(pos or neg]" will be replaced with one label within [Positive, Negative). "I [fummof_wordd]] will be replaced with "50", "[story-q)" will be replaced by the first three sentences of each sample in the ROC Stories dataset.
C.1.2. PUBLICLY AVAILABLE GENERATED DATASETS
We use SST2 (Socher et al., 2013), Yelp (Zhang et al., 2015), and AG News (Zhang et al., 2015), and their augmented version based on AttrPrompt (Yu et al., 2024). For three original datasets, we randomly sample data from training sets and apply this to the computational cost analysis in Section 5.3, Appendix D. and Appendix D.2. For three augmented datasets, each dataset has 6000 samples. We sample different sub-datasets based on these three datasets, applied to Section 5.3 and Section 5.5.3, respectively. The details are as follows.
Datasets on Section 5.3. We remove samples with text token lengths less than 50 in the three datasets and then truncate each sample to a length of 50 tokens. Based on the above, we set up sub-datasets with randomly selected samples of 100, 500, 1000, 2000, and 4000.
C.2. Implementation Details
For three transformation based methods, including DCScore, VendiScore, and K-means Inertia, we employ unsup-simcse bert- baas--n (princeton nip, 2021) as the weight of the embedding function. For all language models used to generate
18

---

Measuring Diversity in Synthetic Datasets
Table 9. Prompt settings for zero-shor and seor settings. Contents that need to be replaced are highlighted in gray
NLG Tasks
Zero-shot
Few-shot
Text Classifi- cation
Now you are movie critic. You are given a movie genre/style and a length requirement. You must come up with a movie that corre- sponds to the genre/style and write a review that meets the length requirement. Write a film review for a {style} movie to express (pos_or.ne) feedback. Each review should have {num of words] words. Be sure to express your per- sonal insights and feelings. Please be creative and write unique movie reviews.
Now you are a movie critic. You are given a movie genre/style and a length requirement. You must come up with a movie that corresponds to the genre/style and write a re- view that meets the length require- ment. Write film review according to the given example. Make sure your review expresses the same sen- timent (positive or negative) as the example. Each review should have (umm of words} words. Be sure express your personal insights and feelings. Please be creative and write unique movie reviews. The fol- lowing is the example:
#An example from IMDB
(Maas
et al., 201 1)#
Story pletion
Com-
Question: {story-qu .q} Answer:
Complete the story according to the given example.
Example:
#An example from ROC Stories (Mostafazadeh et al., 2016)#
Question: {story q}
Answer:
the dataset, we the top-p and top-k parameters 1 and -1, respectively, Additionally, we limit the maximum number of newly generated tokens 100 for the text classification task and 30 for the story completion task. All experiments conducted 8x NVIDIA Tesla V100 GPU with 32GB of memory. For self-generated datasets, we only evaluate the diversity of generated components.
C.2.1. HYPERPARAMETER SETTINGS OF DIVERSITY EVALUATION
For DCScore, VendiScore, and K-means Inertia, we fix the batch size of generating sample representation at 128 across all experiments. Given the varying hyperparameters for each diversity evaluation method, we provide the detailed settings for each method below:
DCScore.
We employ the inner product Kernel(-), and Softmax fk(-). Except for hyperparameter sensitivity experiments, we set r in Eq. (4) to for all other experiments.
Distinet-n. We use 5-grams to calculate distinct-n.
K-means Inertia. We set the number of clusters to 10 for all experiments.
VendiScore. We employ the inner product as (ernel(-).
C.2.2. HYPERPARAMETER SETTINGS OF DOWNSTREAM TASK TRAINING
To train text classification models, we employ RoBERTa (Liu, 2019) as the encoder and utilize the representations from the last layer of the encoder as the classifier's input. We employ LoRA (Hu al., 2021) to finetune the encoder and the
19

---

Measuring Diversity in Synthetic Datasets
Prompt for GPT-4 Evaluation
Task Definition
Diversity Definition
General Prompt
Sent. Set 1
Sent. 2
Task: Use the following criteria compare
of tentences and judge which of hetrer the criteria. Each of contains sentences, each storting with "#"
Evaluation Criteria: Diversity includes richness vocabulary, variation sentence structure, diversity of topics, detailed below;
Vocabulary richness: W hether the vocahulary used the sentences is diverse and includes uncommon words
Sentence Structure Variation: W hether there variation in
structure and different grammar structures used.
ippics or concepts. Please compare following two sentences analyze them hased on the criteria:
Topic Diversiry: Whether sentences rover
set xxx
Analyze each sentence and based the above criteria, give your judgement: Which sentence better meets criterio? Please directly)
with "Sentence wins" "Sentence wins". impossible to judge, then answer with "No winner".
www............................. www.............................
Prompt Demo (The background color the text signifies the category to which belongs, including the task definition, diversity definition, general prompt, and sentence sets.)
Figure 7. Prompt settings for GPT-4 evaluations.
classifier on self-generated datasets. Specifically, we fix the LoRA scaling factor to 32 and the rank of the update matrices to We use A (Loshchilov, 2017) with an initial learning rate of 5e-5 and linear learning rate decay as our optimizer. Additionally, we set the batch size per GPU as 32 and epochs as 120. For the number of training GPUs, we employ GPUs for zero-shot settings and 4 GPUs for few-shot settings. Therefore, the different training steps for zero-shot and few-shot settings are shown in Figure 8.
C.2.3. EVALUATION PROTOCOL
In our experiments, we employ diversity evaluation methods to score the diversity of sub-datasets using two evaluation protocols: overall evaluation and batch evaluation. While K-means Inertia uses the overall evaluation protocol, all other methods utilize the batch evaluation protocol. The detailed settings for the two evaluation protocols are as follows:
Batch evaluation. Due to context prompt associated with several samples in sub-datased dataset the batch evaluation protocol requires that evaluation methods treat samples generated from the same context as a single batch. The evaluation results are then averaged across all batches of the entire sub-dataset.
Overall evaluation. We consider each sample in a sub-dataset as independent, meaning each sample is generated by a distinct context or prompt. Based on this assumption, the overall evaluation protocol requires evaluation methods to directly measure the diversity of the entire sub-dataset.
C.2.4. PROMPT SETTINGS OF LLM EVALUATION
In Section 5.2.3, we use GPT-4 to perform pairwise diversity comparisons. To guide GPT-4 in making these comparisons, we employ well-designed prompt, as illustrated Figure 7. The prompt for GPT-4 evaluations includes the task defiiition, diversity definition, general prompt, and sentence sets to be compared.
C.2.5. SETTINGS OF HUMAN EVALUATION
According to Appendix C. datasets in Section 5.2.2 are generated at six temperatures, with five results per context (prompt) in each sub-dataset. During the evaluation, evaluators are asked to select the more diverse sub-dataset from pairs of sub-datasets. Across six temperatures, this results in 15 comparisons, and with three evaluators, a total of 45 judgments are made. Sub-datasets are ranked by the frequency of being chosen as more diverse. This process is repeated five times with different contexts to derive the final human diversity ranking.
20

---

Measuring Diversity in Synthetic Datasets
Table 10. Comparison of computation time between DCScore and VendiScore on Yelp.
Yelp
Kernels
Sample num
4k
8k
16k
32k
64k
VendiScore
00.96
114.64 1.63
227.76 -7.04
451.49 19.73
912.60115.60
Inner product
DCScore
57.95_0.31
115.35
232.49
448.98
961.29
VendiScore
=0.01
118.15
242.06=7.60
42.89922.89
1272.93
RBF kernel
DCScore
58.49 00114
116.29 +0.92
232.94_3.09
471.18
953.62
VendiScore
59.48 00.05
118.94 +0.99
234.08 +11.72
522.82+3.04
1313.55+12.66
Poly kernel
DCScore
58.73 00.08
117.02
227.72 :9.51
462.45
462.45
988.53 +1.10
Table 11. Comparison of computation time between DCScore and VendiScore on AG News.
AG News
Kernels
Sample num
4k
8k
16k
32k
64k
VendiScore
14.56-1.116
30.20
63.70
127.25
254.20
Inner product
DCScore
14.11
30.77
63.57
129.70
284.76+12.30
VendiScore
16.69
33.69
$0.09+2.33
185.7916.44
617.06+12.51
RBF kernel
DCScore
16.01
31.06
69.15+1.32
129.36
297.29
VendiScore
17.60+0.62
36.16
79.34
190.96
632.69
Poly kernel
DCScore
16.8840.59
33.78
68.18
68.18+1..66
303.06
D. Additional Experiments
D.1. Computational Cost for Larger Datasets
In the case where the inner product is used as the kernel function and n > d, VendiScore can significantly reduce computational complexity. ensure a fair comparison, we compare the computation times of VendiScore and DCScore on larger datasets, as well as under different kernel functions. Specifically, we employ SST2, Yelp, and AG News as the evaluated datasets. We randomly select 4k, 8k, 16k, 32k, 64k samples and record the computation times for both methods across these different sample sizes. We repeat the experiments 5 times to report the mean and standard deviation.
As shown in Tables 6, 10, and 11, DCScore has a shorter computation time than VendiScore in most cases, with VendiScore only exhibiting a computational advantage when the inner product is used and >> d. Furthermore, as the sample size increases, the efficiency advantage DCScore becomes more pronounced. When using polynomial kernel, on the SST2 dataset, DCScore requires only one-third of the computation time of VendiScore when the sample size reaches 64k. In contrast, although VendiScore has a computational advantage in the case of the inner product, the difference compared DCScore is not significant. The experimental results are consistent with our complexity analysis presented in Section 4.3, Overall, DCScore outperforms VendiScore in terms of computation time across most kernel functions. VendiScore exhibits computational time advantage only when the inner product is used as the kernel function, which will limit its applicability. As shown in Chapter 4 of (Seeger, 2004), it is essential to employ different kernel functions to accommodate a wider range of scenarios.
D.2. Comparison of Computational Cost between DCScore and Efficiency-improved Methods
A recent work (Ospanov et al., 2024) employs the random Fourier features framework to reduce the computational cost of VendiScore (Dan Friedman & Dieng, 2023) and RKE (Jalali et al., 2023). To further investigate the computational efficiency of DCScore, we compare with emiprveecyiipproeed versions of two entropy- based diversity evaluation methods, namely FKEA- Vendi and FKEA-RKE. Here, we set a to for FKEA-Vendi and 2 for IKEA-RKE Notably, we follow the
21

---

Measuring Diversity in Synthetic Datasets
Table 12. Comparison of computation time between DCScore and two efficiency-improvee methods (FKFA-Vend) and FKEA-RKE).
Datasets
Sample num
4k
8k
16k
32k
64k
FKEA-Vendi
18.4
30.97
04.0
147.78
295.14
SST2
FKEA-RKE
18.4640.08
37.08
73.9810.00
147.9910.00
297.4612.02
DCScore 3.26
6.75
13.9410.06
31.32
90.36
FKEA-Vendi 26.28
52.6940.24
106.59
212.52
422.51+1.11 11.1
Yelp
FKEA-RKE
46.1940.13
52.68
105.27 0.22
212.38
122.94+1..55
DCScore
37.5340.08
45.1940.10
151.59
306.80
641.60
FKEA-Vendi 20.64
41.6610.066
83.02
165.92
338.16114.00
AG News
FKEA-RKE
20.68+0.04
41.68+0.03
83.0440.14
165.85 00.23
331.58
DCScore
10.29
21.12+0073
43.84+1.02
91.64=0.88
213.30 15
Zero-shot Setting
Few-shot Setting
0.7
0.7
0.6
0.6
0.5
Loss
Loss
0.4
0.5
=.2
Tg = 0.2
0.3
=.7
Ty 0.7
0.2
Tg
1.2
0.4
Ty 1.2
250
500
750
1000
0
500
1000 Step
1500
2000
Step
Figure 8. Loss curves of the downstream task training.
experimental settings described in Appendix D.1 and leverage different random seeds for data sampling. Table 12 illustrates the computation time comparison across the SST2, Yelp, and AG News datasets. DCScore demonstrates lower computation time compared to EXEAAVennd and IKEA-RKE on the SST2 and AG News datasets. However, the results reversed for the Yelp dataset. Additionally, we observe that FKEA-Vendi does not improve computation time compared to VendiScore, possibly because FKEA focuses solely on reducing the complexity of the summarization (cigenvalue computation) phase, whereas our computation time encompasses the entire calculation process.
D.3. Correlation with Downstream Task Training
To investigate the correlation between DCScore and downstream task training, we train text classification models using self- generated datasets under zero-shot and few-shot settings. We vary the generation temperature of self-generated datasets within the range (0.2, 0.7, 1.2). More details of training datasets and hyperparameters are presented in Appendix C. and C.2.2, respectively. Figure 8 shows observe increasing optimal loss values as Ty varied from 0.2 1.2, indicating that the model more easily fitted datasets with limited diversity. However, as shown in Table 7, models trained on diverse datasets achieve better accuracy, which can be attributed to their enhanced generalization capabilities. From Table 7, the diversity evaluated by DCScore has similar trend to the accuracy performance in the zero-shot setting, further demonstrating the effectiveness of DCScore in diversity evaluation.
loss curves of these trained classification models. the zero-shot setting, we
In the few-shot setting, we observe a trend in optimal loss variation similar to that in the zero-shot setting, as shown in Figure However, Figure reveals that epoch 120, models trained on datasets generated with Ty = 0.7 outperform those
22

---

Measuring Diversity in Synthetic Datasets
Loss Curves
Model Accuracy
0.75
74
0.70
73,41
73.19
73
Loss
0.65
72.44
72
0.60
Epoch 360
0.55
Epoch 240
71
Epoch 120
0.50
70
0
2000
4000
6000
360
240 Epoch
120
Step
Figure 9. Loss curves and accuracy of models trained on generated dataset with Ta 1.2.
Table Correlation (Spearman's results DCScore with various embedding functions.
Spearman's varies between -1 and +1,
with 0 implying no correlation. Best results are indicated in bold.
Zero-shot setting
Few-shot setting
Embedding models
Text classification
Story completion
Text classification
Story completion
13B
70B
13B
70B
13B
70B
13B
70B
SimCSE mussup-sinnse-eett--as---mees
0.9961
0.9779
0.9844
0.9792
0.9909
0.9883
0.9857
0.9974
SimCSE (supsimcse-robertt-llrge))
0.9909
0.9753
0,9883
0.9883
0.9792
0.9935
0.9779
0.9623
Sentence BERT (all-mpnet-base-22)
0.9896
0.9740
0.9870
0.9909
0.9766
0.9870
0.9857
0.9870
BGE (bge-large-en-v1.55)
0.9909
0.9896
0.9922
0.9948
0.9857
0,9922
0.9870
0,9922
using = 1.2. This phenomenon can be attributed to the higher diversity of datasets generated at a higher Ty, resulting in increased fitting difficulty. Under the current settings, the number of training epochs for dataset generated a temperature of 1.2 is insufficient, preventing the trained model from achieving optimal performance. To validate this hypothesis, we increase the number of epochs to 240 and 360 and train models the dataset generated at temperature of 1.2. The final training loss and accuracy of these models are shown in Figure 9. We observe that as the number of epochs increases, the model's i loss gradually decreases, and its performance improves progressively. Ultimately, the model's accuracy outperforms that of models trained on datasets generated at temperatures of 0.2 and 0.7. Moreover, from Table 7, models trained on datasets from the zero-shot setting outperform those trained on datasets from the few-shot setting. However, this discrepancy arises from the different test sets used in the two settings, making direct performance comparisons inappropriate.
D.4. Impact of Embedding Functions +
The paradigm of the transformation- based method enables DCScore to utilize various embedding functions tailored to different scenarios. Consequently, we investigate the impact of embedding functions on self-generated datasets used in Section 5.2.1. As shown in Table 13. we compare the correlation of the diversity evaluation results of DCScore across different embedding functions with diversity pseudo-truths, where the model names in parentheses within the embedding function refer to those available Hugging Face. Our findings indicate that DCScore exhibits strong correlations with diversity pseudo-truths across various embedding functions. Notably, DCScore utilizing the BGE embedding function achieves the best results in half of the cases. Additionally, the minimum correlation in Table 13 exceeds 0.96, which is classified as a strong correlation according to (Akoglu, 2018). This result also supports the following two conclusions: (1) the embedding function used effectively captures the differences among samples from multiple perspectives, and (2) DCScore re is sufficiently adaptable to different embedding functions while maintaining stable performance.
23

---

Measuring Diversity in Synthetic Datasets
Table 14. Correlation (Spearman's 's p) results of DCScore with various kernel functions, Spearman's 's p varies between -1 and +1, with implying no correlation. Best results are indicated in bold.
Zero-shot setting
Few-shot setting
Embedding models
Text classification
Story completion
Text classification
Story completion
13B
70B
13B
70B
13B
70B
13B
70B
Inner product
0.9961
0.9779
0.9844
0.9792
0.9909
0.9883
0.9857
0.9974
laplacian kernel
0.9935
0.9831
0.9883
0.9727
0.9597
0,9649
0.9701
0,9922
RBF kernel
0.9935
0.9818
0.9896
0.9753
0.9740
0.9727
0.9792
0.9922
polynomial kernel
0.9870
0.9584
0.9714
0.9506
0.9182
0,9182
0.9857
0.9896
D.5. Impact of Kernel Functions
Similar to Appendix D.4, we investigate the impact of different kernel functions on the performance of DCScore. Specifically, this experimental setup is identical to that Appendix D.4. As shown in Table 14, we find that DCScore demonstrates stable performance across various kernel functions. However, the influence of the kernel function is slightly more pronounced than that of the embedding function, as indicated by the greater fluctuations in correlation among the different kemel functions. Furthermore, observe that DCScore achieves optimal performance in the case of the inner product. Overall, DCScore consistently maintains strong diversity evaluation performance across different kernel functions.
E. Baseline Methods
We present the detailed modeling of baseline methods into DCScore as follows:
Distinct-n. Distinct-n (Li et al., 2015) a prevalent diversity metric depending on n-grams, where л sigmifies the number of successive items. Distinct-n calculates the proportion of unique n-grams all n-grams. The n-grams operation falls under the text representation stage, while the step of obtaining unique set of n-grams corresponds to the pairwise similarity stage. Typically, high form similarity among samples in the evaluated dataset results in smaller unique n-gram set. Finally, ratio calculations belong to the diversity summarization stage.
Text Representation: Pggrams(Concat//g
Pairwise Similarity: Uniquee(ngrrmms(Conat(/
(18)
Diversity Summarization: Distinct-n(D) |nnqque (n-gaamnnnggs(s((e (pr.rrgammCooooaa///iii (Concct(()))
n-grams (Connan(CCnna)(()))
where n-grams, Unique, and Concat represent the n-grams, de-overlap process, and concatenate operation, respectively.
K-means inertia. K-means Inertia (Du & Black, 2019), transformation-base method, performs clustering in sample representation and then calculates inertia as diversity outcomes. Here, inertia refers to the square summation of the distance between samples and cluster centroids.
Text Representation: H 4b (((T))
Pairwise Similarity: C = K-means( (e)
(19)
Diversity Summarization: Inertia(D)
(h, 2
C C,EC, EHe,
where H is the representation of all samples and h, is the representation of the i-th sample, C denotes the cluster centroid set, and E C represents the k-th cluster centroid. The sample representation associated with the k-th cluster centroid is expressed as hy He., while He, denotes the sample representations corresponding to the k-th cluster centroid.
VendiScore, VendiScore (Dan Friedman & Dieng, 2023) a recently proposed diversity metric that falls under the category of the transformation-basse method, Based on sample representations, VendiScore utilizes kernel function to calculate
24

---

Measuring Diversity in Synthetic Datasets
similarity matrix K. Subsequently, VendiScore summarizes diversity the exponential of the Shannon entropy of the eigenvalues of K/n.
Text Representation:
(((Ti)=m1) 1),
Pairwise Similarity: K (e),
(20)
Diversity Summarization: VS(D) exp (- Σ
xe log Az),
i=1
where Kernel(-) is the kernel function, such the inner product, X is the i-th eigenvalue of K/n.
25


Reviews:
[{"review_id": "TiUB0fnr4Q", "reviewer": "Reviewer_xZQX", "summary": "The paper introduces a classification-based evaluation metric, DCScore, for assessing the diversity of synthetic datasets. The authors address computational challenges while satisfying axiomatic requirements and providing a holistic analysis. Additionally, they evaluate the diversity of generated datasets across various scenarios.\n\n---\n\n## update after rebuttal\n\nI thank the authors for their response. I believe the authors provided more detailed insights into the advantages of their work during the rebuttal, such as the optimization stability or computational efficiency of DCScore compared to Vendi Score in some cases.\n\nWhile I appreciate the authors' response, I remain concerned that several claims lack sufficient explanation or need additional experiments, hence, I will keep my score. Below are key points that, in my view, must be addressed to strengthen the manuscript:\n\n- **Convergence of DCScore (R9)**: One important part that is missing is the convergence analysis of DCScore. The authors indeed provided additional experiments with 2D Gaussian mixtures, such analysis must also be demonstrated on complex datasets to validate the method's convergence. Also, the authors mentioned their task is distinct from generative model evaluations because the target scenario lacks ground truth values, but I believe convergence is important in their task too and even in generative models' evaluation, we don't have ground truth values.\n\n- **Evidence of why DCScore is closer to the essence of diversity evaluation. (R4)**: While the classification-based approach is novel, the authors must provide a more rigorous comparison and evidence demonstrating why DCScore fundamentally captures the essence of diversity better than clustering-based methods like Vendi Score.\n\n- **Fair comparison between Vendi Score with DCScore (R1, R5, R8)**: The authors claimed that DCScore is *Suitable for highly diverse scenarios*, or *In datasets with distinctive samples, Vendi Score fails, but DCScore succeeds*. However, I argue these limitations could be addressed in Vendi Score through proper hyperparameter selection.  Since DCScore has a temperature hyperparameter ($\\tau$), the authors should comparably test Vendi Score using a Gaussian kernel (not cosine similarity) since Gaussian kernel has bandwidth parameter ($\\sigma$) that could make it equally adaptable to such scenarios.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for providing detailed review on our submission. We respond to the reviewers’ concerns and questions one by one. **Limited by the space, we present our additional experiments in an anonymous URL** (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).\n\n---\n>Q1: Relying solely on the trace of P may obscure important structural details. I expect a more in-depth analysis between Vendi Score and DCScore.\n\nR1: A more in-depth analysis is as follows:\n- **Computational Efficiency**: As shown in Table 2, with a general kernel, DCScore has a complexity of $\\mathcal{O}(n^2)$ during summarization, while VendiScore has $\\mathcal{O}(n^3)$, making DCScore more efficient. \n- **Optimization Stability**: As an optimization target, the trace of **P** (DCScore) provides simpler gradients, enabling more stable optimization. In contrast, Vendi Score can have gradient issues due to identical eigenvalues.\n- **Suitable for highly diverse scenarios**: DCScore highlights distinct samples (Please refer to R5), while Vendi Score may underestimate diversity. Thus, our approach aligns with the trend of generating diverse data.\n\n**For more advantages of our method, please refer to R4.**\n\n>Q2: There are several related works on reference-based methods that seem to be missing such as [1, 2]. There are additional works [3, 4] addressing complexity challenges that are not cited.\n\nR2: Thank you for your suggestion. **We will include the discussion of the related works you pointed out in the revised manuscript.** For paper 3, the complexity of  $\\mathcal{O}(n^2)$ only considers the calculation of the Frobenius norm and excludes the specific kernel calculation due to varying complexities across different kernels (e.g. RBF kernel:$\\mathcal{O}(n^2d)$, Graph Kernels (random walk):$\\mathcal{O}(n^3)$). \n\n>Q3: The authors need to mention the similarity of Equation 4 , as it is similar to the contrastive learning literature, such as Equation 1 of the SimCLR.\n\nR3: Thank you for your suggestion. The core concept of contrastive learning—distinguishing samples—is similar to DCScore's classification perspective, reflecting a common idea across fields. We will highlight this in the revised manuscript.\n\n>Q4:  The paper should clarify whether the classification perspective yields benefits beyond computational efficiency. Could the authors elaborate on what new insight this perspective can bring?\n\nR4: As noted in Papers 3 and 4, **entropy-based metrics like Vendi Score face high computational costs, underscoring the need for efficiency**. Additionally, DCScore offers some new insights:\n- A novel perspective closer to the essence of diversity evaluation (identifying sample differences).\n- More stable optimization due to its simpler gradient, unlike Vendi Score's instability from eigenvalue calculations [1].\n- A clearer interpretation of sample uniqueness compared to entropy-based metrics.\n- Sensitivity to unique samples via Softmax and Trace operations, aiding outlier detection and reducing model impact.\n\nReference:[1] Eigenvalue optimization[J]. Acta numerica, 1996. \n\n>Q5: The paper needs to illustrate scenarios where Vendi Score fails to capture diversity correctly, but DCScore succeeds.\n\nR5: Thank you for your suggestion. **In datasets with distinctive samples, Vendi Score fails, but DCScore succeeds**. In this regard, eigenvalue distributions of the kernel matrix can be highly uneven (for example, some eigenvalues are much larger than others), leading to a significant reduction in Shannon entropy. Consequently, the Vendi score yields a diversity estimate much smaller than the actual sample size. In contrast, DCScore accurately estimates the diversity of a dataset based on classification probabilities.\n\n>Q6:  1. Could authors provide a more detailed explanation of why such a choice is a suitable metric of diversity? 2. Could authors compare the effect of replicas on Vendi Score and DCScore?\n\nR6: For 1, as noted in **R1, R4, and R5**, DCScore is more computationally efficient and serves as a more stable optimization target. As theoretically shown in Section 4.2, DCScore satisfies the axioms required for an ideal diversity evaluation method.\nFor 2, **DCScore does not fail in the presence of repeated samples**. According to Section 4.2 (Effective Number), we provide an proof regarding the evaluation under repeated samples. Furthermore, we provide an evaluation of repeated samples in **Figure 1 of anonymous URL**. \n\n>Q7: Can the authors provide a computational complexity comparison between DCScore and the mentioned related works [3, 4] as they also provide an efficient computation of Vendi Score? \n\nR7: We present a comparison in **Figure 3 at the anonymous URL. DCScore outperforms efficient methods in terms of computation time on the AGNews and SST datasets**. Notably, the efficient computation method from Paper 4 is incompatible with non-shift-invariant kernels, which don’t meet diversity evaluation requirements.", "claims_and_evidence": "The paper proposes DCScore, a classification-based diversity metric for synthetic datasets. I am not convinced that using a classification method can provide new insights or even achieve results comparable to eigendecomposition methods in all cases. Using the entropy of eigenvalues can reveal the number of modes in the dataset and their underlying relationships, whereas relying solely on the trace of P may obscure important structural details.\n\nFor example, consider the Monotonicity axiom in a scenario where a large percentage of the dataset consists of duplicate samples.  The kernel matrix develops a block structure with just a few dominant eigenvalues. Using an entropy based metric on eigenvalues could capture the low diversity, whereas a trace-based approach might mistakenly indicate high diversity. This is just a simple example, however, I expect a more in-depth analysis between Vendi Score and DCScore.", "methods_and_evaluation_criteria": "The problem formulation of evaluating synthetic datasets indeed makes sense and the authors extensively experimented with different scenarios to evaluate the score.", "theoretical_claims": "Proofs of Properties of DCScore are correct and straightforward.", "experimental_designs_or_analyses": "The problem formulation indeed makes sense and the authors extensively experimented with different scenarios to evaluate the diversity of the synthetic dataset.", "relation_to_broader_scientific_literature": "This paper uses a classification perspective to evaluate the diversity, which is quite new in the diversity evaluation literature. They also study the diversity evaluation of synthetic datasets.", "essential_references_not_discussed": "There are several related works on reference-based methods that seem to be missing such as [1, 2]. The paper notes that DCScore is similar to the Vendi Score and compares their computational complexities in Section 4.3. While DCScore reduces the complexity from $\\mathcal{O}(n^3)$ to $\\mathcal{O}(n^2d + n^2)$ (except for the cosine similarity kernel), there are additional works addressing this challenge that are not cited. For instance, in [3] (see Corollary 2), it is shown that the RKE (Vendi$_{\\alpha=2}$) score can be computed in $\\mathcal{O}(n^2)$ for all kernels. Additionally, [4] presents an approach that estimates the Vendi score with a computational complexity of $\\mathcal{O}(n)$. I believe these papers are highly correlated and should be mentioned when claiming to improve the computational complexity.\n\nI also believe that the authors need to mention the similarity of Equation 4 (the definition of the classification probability matrix $P$), as it is similar to the contrastive learning literature. For example, in Equation 1 of the SimCLR framework [5], the authors used the same function for contrastive learning of visual representations.\n\n[1] Naeem et al., Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020.\n\n[2] Kynkäänniemi et al., Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019.\n\n[3] Jalali et al., An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions, NeurIPS 2023\n\n[4] Ospanov et al., Towards a Scalable Reference-Free Evaluation of Generative Models, NeurIPS 2024\n\n[5] Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Introduce new settings for synthetic data evaluation using LLMs.\n\n- This paper uses a classification perspective to evaluate the diversity, which is quite novel.\n\n**Weaknesses:**\n\n- The classification-based approach may primarily offer improved sample complexity compared to Vendi Score, without clearly demonstrating how it provides new insights into synthetic data diversity. The paper should clarify whether the classification perspective yields benefits beyond computational efficiency.\n\n- It is not well established when or why DCScore outperforms Vendi Score. The paper needs to illustrate scenarios where Vendi Score fails to capture diversity correctly, but DCScore succeeds (Beyond computational gains)\n\n- Although the paper emphasizes the reduction in computational complexity, it lacks a comparison with recent works that also offer efficient computations of Vendi Score. Without this, the claimed advantages of DCScore remain somewhat unsubstantiated.", "other_comments_or_suggestions": "I believe the authors should revise the title of each paper, as they currently use the template title “Submission and Formatting Instructions for ICML 2025.”", "questions_for_authors": "I believe the idea of evaluating the diversity of synthetic data is crucial, however, I am concerned about the novelty and the contribution of the paper. I have the following questions from the authors. Answering these can elaborate the contribution of this paper.\n\n1. DCScore proposes a synthetic diversity metric with a classification perspective. Could the authors elaborate on what new insight this perspective can bring? Is it only improving the sample complexity compared to Vendi Score, or using this perspective will better match the task of diversity evaluation of synthetic datasets?\n\n2. Using entropy of eigenvalues as a diversity measure makes sense to me as it shows how many significant directions of variation exist in the dataset, but only using the trace of the probability matrix is not very intuitive as it highly relies on individual datapoints. Could authors provide a more detailed explanation of why such a choice is a suitable metric of diversity? (One case of failure I imagine is when we add replicas already existing samples in the dataset. Could authors compare the effect of replicas on Vendi Score and DCScore?)\n\n3. What is the advantage of DCScore over Vendi Score? Can you provide a case when DCScore behaves as we expect and Vendi score fails to do that?\n\n4. Can the authors provide a computational complexity comparison between DCScore and the mentioned related works [3, 4] as they also provide an efficient computation of Vendi Score? Are there some cases where DCScore outperforms compared to the mentioned papers that I'm missing?\n\n\n[3] Jalali et al., An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions, NeurIPS 2023\n\n[4] Ospanov et al., Towards a Scalable Reference-Free Evaluation of Generative Models, NeurIPS 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8k06VjngYR", "reviewer": "Reviewer_BZYd", "summary": "The paper introduces DCScore, a novel metric for measuring diversity in synthetic datasets. Unlike traditional methods (e.g., Distinct-n, VendiScore), DCScore models diversity as a classification task and uses semantic embeddings to compute pairwise similarity among samples. It leverages a softmax-based classification probability matrix to quantify dataset diversity. The authors provide theoretical guarantees for DCScore, showing that it satisfies fundamental diversity axioms. Empirical results demonstrate that DCScore correlates strongly with human judgments while being more computationally efficient than VendiScore.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper. We respond to the reviewer’s question as follows. **Limited by the space, we present our additional experiments in an anonymous URL** (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).\n\n---\n>Q1: it does not sufficiently justify why treating each sample as a separate category is a robust measure of diversity. the paper should provide that classification-based diversity correlates well with entropy-based or clustering-based diversity measures.\n\nR1: Thank you for your insightful comments. **We have indeed provided both experimental and theoretical validation to support DCScore as a robust measure of diversity.** We offer empirical validation demonstrating a strong correlation with the entropy-based method, as shown in **Figure 5 at the anonymous URL**.\n\nTheoretically, we have proven that DCScore satisfies several axioms that an ideal diversity evaluation method should meet, as detailed in Section 4.2 and Appendix B of our paper.\n\nEmpirically, we have demonstrated that DCScore has a high correlation with multiple diversity pseudo-truths, such as generation temperature $\\tau_{g}$, human evaluation, and LLM evaluation, as shown in Section 5.2. We believe these pseudo-truths more accurately reflect the true diversity of datasets.\n\n>Q2: VendiScore can be optimized to O(d 2n)using low-rank approximations, which the authors do not address.\n\nR2: As shown in Table 2 of our paper, **VendiScore can achieve $\\mathcal{O}(d^{2}n)$ complexity when using a linear kernel**, such as Inner Product. We have provided a detailed analysis of this in Section 4.3. However, in practical evaluation scenarios, the diversity of synthetic datasets often requires more complex kernels beyond linear ones. \n\n>Q3:  1.Classification probability correctly represents diversity, but this assumption is not tested against other diversity definitions (e.g., entropy-based measures). 2.No test on datasets with strong semantic similarity. \n\nR3: For 1, DCScore and entropy-based or clustering-based measures operate on fundamentally different principles, **making it infeasible to validate one method's assumptions within the framework of another**. Limited by the space, please refer to R4 for more details.\nFor 2, as shown in **Figure 1 of anonymous URL**, we conduct evaluation experiments on datasets with strong semantic similarity and **observe that DCScore performs well in this scenario**.\n\nReference:\n[1] Ospanov et al., Towards a Scalable Reference-Free Evaluation of Generative Models, NeurIPS 2024.\n\n>Q4: If classification probability does not fully represent diversity, then the theoretical guarantees may be incomplete. The author should add mathematical proofs for why classification probability is a good diversity metric\n\nR4: Thank you for your suggestion. For diversity metrics, **there is no ground truth or golden rule, only characterization through axioms**. Therefore, in Section 4.2, we demonstrate that DCScore satisfies these axioms, indicating that it is a good diversity evaluation metric. From the theoretical perspective, Diversity evaluation aims to capture dataset richness by identifying sample relationships, which aligns with the classification perspective of DCScore. Thus, **a sample-level classification probability can fully represent diversity**. \n\n>Q5: There is no ablation study on how different embedding functions affect DCScore, the author should analyze how different embeddings (e.g., SBERT, CLIP) affect DCScore\n\nR5: We would like to clarify that **we have conducted experiments on the impact of embedding functions**, and the results are presented in **Appendix E.3 and Table 11**. DCScore demonstrates strong performance across various embedding functions.\n\n>Q6: The supplementary material includes additional experimental details but lacks deeper theoretical analysis.\n\nR6: Thank you for your valuable feedback. In **Appendix B** of our paper, we provide a theoretical analysis of the properties satisfied by DCScore, demonstrating that it adheres to the axioms expected of an ideal diversity evaluation method. Additionally, in **Section 4.3**, we offer a theoretical analysis of the computational complexity of DCScore. \n\n>Q7: The paper does not mention entropy-based diversity measures, which are commonly used in active learning and generative models.\n\nR7: Thank you for your suggestion. We believe **the omission may stem from our different taxonomy of existing methods**. According to our classification, entropy-based diversity measures fall under Transformation-based Methods. We will clarify this distinction in future versions of our paper and include a more detailed discussion of entropy-based methods.\n\n>Q8: Weaknesses: Highly sensitive to how categories are defined\n\nR8: We would like to clarify that **DCScore is not sensitive to category definitions**. It doesn't require predefined categories and the category count is the dataset sample size.", "claims_and_evidence": "This paper assumes that classification probability correlates directly with diversity, but it does not sufficiently justify why treating each sample as a separate category is a robust measure of diversity.If classification probability is used as a proxy for diversity, the paper should provide empirical or theoretical validation, such as, showing that classification-based diversity correlates well with entropy-based or clustering-based diversity measures.\nThe paper correctly highlights that VendiScore requires O(n3) complexity due to eigenvalue decomposition, while DCScore reduces this to O(n2).However, VendiScore can be optimized to O(d 2n)using low-rank approximations, which the authors do not address.", "methods_and_evaluation_criteria": "The method of treating diversity evaluation as a classification problem is interesting, but it is not fully justified.\nThe method assumes that classification probability correctly represents diversity, but this assumption is not tested against other diversity definitions (e.g., entropy-based or clustering-based measures).\nNo test on datasets with strong semantic similarity (e.g., paraphrased text or redundant images), which would be important to validate the method.", "theoretical_claims": "The paper propose DCScore，but if classification probability does not fully represent diversity, then the theoretical guarantees may be incomplete.", "experimental_designs_or_analyses": "There is no ablation study on how different embedding functions affect DCScore, the author should analyze how different embeddings (e.g., SBERT, CLIP) affect DCScore.", "relation_to_broader_scientific_literature": "The paper discusses N-gram-based, reference-based, and transformation-based diversity metrics.However, it does not mention entropy-based diversity measures, which are commonly used in active learning and generative models.", "essential_references_not_discussed": "The paper does not discuss entropy-based diversity metrics, which have been used in representation learning and active learning.", "other_strengths_and_weaknesses": "Strengths\n1.Proposes a novel classification-based diversity metric.\n2.Improves computational efficiency over VendiScore.\nWeaknesses:\n1.Lacks theoretical justification for using classification probability.\n2.Highly sensitive to how categories are defined.\n3.Does not compare with entropy-based or contrastive learning-based diversity metrics.", "other_comments_or_suggestions": "Please see the above weakness", "questions_for_authors": "Please see the above weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNUIzz9jnQ", "reviewer": "Reviewer_fn8g", "summary": "- The paper introduces DCScore, a novel method for measuring diversity in synthetic datasets generated by large language models (LLMs).\n\n- Key innovation: DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples, rather than using traditional n-gram statistics or reference-based methods.\n\n- Main contribution: A principled diversity evaluation metric that satisfies important diversity-related axioms (effective number, identical samples, symmetry, and monotonicity).\n\n- Technical approach: The method maps diversity-sensitive components into a representation space, computes pairwise similarities through a kernel function, and summarizes diversity through classification probabilities.\n\n- Experimental validation: DCScore demonstrates stronger correlation with diversity pseudo-truths (like generation temperature) and human judgment compared to baseline methods.\n\n- Computational efficiency: Both theoretical analysis and empirical results show DCScore has lower computational costs compared to existing approaches, particularly for non-linear kernels.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper. We respond to the reviewer’s question as follows. **Limited by the space, we present our additional experiments in an anonymous URL** (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).\n\n---\n>Q1: The claim about DCScore's correlation with human judgment would be strengthened with more details on the human evaluation protocol and inter-annotator agreement. \n\nR1: Thank you for your valuable comment. **We will update the appendix in the subsequent version of our paper to include details of the human evaluation protocol**. Human evaluation data were generated at six temperatures, with five results per context (prompt) in each subset. During the evaluation, annotators were asked to select the more diverse subset from pairs of subsets. Across six temperatures, this resulted in 15 comparisons, and with three annotators, a total of 45 judgments were made. Subsets were ranked by the frequency of being chosen as more diverse. This process was repeated five times with different contexts to derive the final human diversity ranking.\n\n>Q2: The claim about correlation between DCScore and downstream task performance could benefit from more extensive experimentation. Additionally, it seems like the number of epochs was not fixed for this experiment. \n\nR2: We** have included some downstream task experiments in Appendix E.2. In Table 7**, except for the scenario with $\\tau_{g}=1.2$(360 epochs), all other results were obtained with 120 epochs. **The higher dataset diversity, e.g. datasets with $\\tau_{g}=1.2$, required more epochs for model convergence**, with details in Appendix E.2. **Figure 8 also presents results for $\\tau_{g}=1.2$ at 240 and 120 epochs**. \n\n>Q3: A comparison with reference-free diversity metrics from other domains (e.g., ecology) could have further contextualized DCScore's advantages.\n\nR3: Our method is primarily concerned with evaluating the diversity of synthetic datasets. We believe that **comparisons with methods from more closely related domains might carry more conviction**. The axiomatic requirements for diversity evaluation can vary significantly across different fields, and using metrics from ecology to assess text or image datasets might result in an unfair comparison. Additionally, **the main idea behind the Vendi score method originates from ecology, which can be considered an indirect comparison with metrics from other fields.**\n\n>Q4:  Limited discussion of potential failure cases or situations where DCScore might not accurately capture diversity. \n\nR4: The main limitation of our method is that **it is not applicable to multimodal data**. Due to space constraints, please refer to **R1 of the Response to Reviewer QL32** for more details.\n\n>Q5: The batch evaluation protocol is appropriate but could introduce bias if contexts vary significantly in diversity potential.\n\nR5: The batch evaluation protocol was chosen to ensure that the diversity pseudo-truths are accurately reflected when generating multiple samples from the same context. This approach helps maintain consistency in evaluating the diversity of generated datasets. We adopted this evaluation strategy to conduct correlation experiments, and under this setting, bias is not introduced. \n\n>Q6: The paper appropriately tests on both self-generated and publicly available datasets, but the dataset sizes (100 samples per sub-dataset) are somewhat limited for diversity evaluation.\n\nR6: As shown in Q8, we conducted experiments on datasets with **sizes reaching up to 64k samples.** The smaller datasets, consisting of 100 samples per sub-dataset, were specifically designed for batch evaluation protocols (**Please refer to R5** for more details).\n\n>Q7: Limited experimental evaluation across diverse domains beyond text classification and story completion.\n\nR7: Thank you for your suggestion. We provide experimental results on image data (colored mnist dataset), please refer to **Figure 2 of anonymous URL**. DCScore presents a higher correlation with the label number compared to VendiScore.\n\n>Q8: Computational results focus on small to medium datasets (up to 64k); scalability for extremely large datasets remains unproven.\n\nR8: We provide experimental results on extremely large datasets (up to 120k) in **Figure 6 of the anonymous URL. Notably, DCScore exhibits similar changing trends with the baseline methods across these large-scale datasets**.\n\n>Q9: Have you explored whether DCScore can be used for diversity-aware sampling from generative models, beyond just evaluation of already-generated datasets?\n\nR9: Our method is similar to approaches like the Vendi score, thus **it can be applied to diversity-aware sampling from generative models**. However, we are more focused on scenarios of synthetic data diversity evaluation. We also provide a detailed discussion of potential application scenarios for DCScore in Appendix A.2.", "claims_and_evidence": "- The theoretical claims about DCScore satisfying the four axioms (effective number, identical samples, symmetry, monotonicity) are well-supported with formal proofs in the paper.\n\n- The computational efficiency claims are supported by both theoretical complexity analysis (Table 2) and empirical measurements (Figure 4), though the advantage varies by kernel type.\n\n- The claim about DCScore's correlation with human judgment is supported (Table 4), but would be strengthened with more details on the human evaluation protocol and inter-annotator agreement.\n\n- The claim about correlation between DCScore and downstream task performance is supported by limited evidence (Table 7) that could benefit from more extensive experimentation. Additionally, it seems like the number of epochs was not fixed for this experiment.", "methods_and_evaluation_criteria": "- Using Spearman's ρ to measure correlation with diversity pseudo-truths is appropriate for evaluation.\n\n- The selection of diversity pseudo-truths (generation temperature, human judgment, GPT-4 evaluation) is sensible and provides multiple validation angles.\n\n- The choice of baseline comparison methods (Distinct-n, K-means inertia, VendiScore) covers multiple baseline approaches.\n\n- The evaluation on both self-generated and publicly available datasets is appropriate, though more domain diversity would strengthen claims of generalizability.\n\n- Testing across multiple embedding functions and kernel types demonstrates robustness, a valuable evaluation approach.\n\n- The computational efficiency evaluation is practical and relevant, especially for large synthetic datasets.\n\n- The axiomatic analysis provides theoretical validation for why the method works, strengthening the methodology.\n\n- A comparison with reference-free diversity metrics from other domains (e.g., ecology) could have further contextualized DCScore's advantages.", "theoretical_claims": "I did not check the correctness of proofs very carefully. Overall, they seem mostly correct. They could better explore potential limitations in edge cases, such as what happens with extremely imbalanced datasets or outliers.", "experimental_designs_or_analyses": "- The correlation analysis with generation temperature (τg) is sound, using appropriate statistical measures (Spearman's ρ) across a well-distributed range of temperatures (0.2-1.2).\n\n- The experimental setup for human evaluation lacks some important details - specifically how agreement was measured across evaluators and why only 3 annotators were used.\n\n- The downstream task training experiment provides valuable real-world validation, but could be strengthened with more diverse task types beyond text classification.\n\n- The batch evaluation protocol (averaging results across batches generated from the same context) is appropriate but could introduce bias if contexts vary significantly in diversity potential.\n\n- The ablation studies for embedding functions and kernel types are methodologically sound, though the paper could better explain the criteria for selecting the specific functions tested.\n\n- The comparison against baseline methods is fair, with appropriate implementation details provided for each method.\n\n- The paper appropriately tests on both self-generated and publicly available datasets, but the dataset sizes (100 samples per sub-dataset in some experiments) are somewhat limited for diversity evaluation.\n\n- Statistical significance testing is notably absent from the correlation analysis, which would strengthen confidence in the reported differences between methods.", "relation_to_broader_scientific_literature": "- The paper connects to recent LLM synthetic-data generation work by Ye et al. (2022) and Abdullin et al. (2024), providing evaluation metrics for these generative approaches.\n\n- The computational efficiency focus addresses challenges with other methods (such as VendiScore).\n\n- The evaluation methodology connects to research on human-aligned evaluation metrics by Holtzman et al. (2019).\n\n- The temperature-based diversity relation builds on sampling strategy research by Caccia et al. (2018) and Chung et al. (2023).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n\n- The paper addresses a practical need in LLM-generated datasets, making it timely and relevant to current research directions.\n\n- The method is adaptable across embedding functions and kernel types, demonstrating flexibility for different applications.\n\n- The computational efficiency advantages make DCScore practically applicable to large-scale dataset evaluation.\n\n## Weaknesses\n\n- Limited experimental evaluation across diverse domains beyond text classification and story completion.\n\n- Human evaluation is sparse.\n\n- Limited discussion of potential failure cases or situations where DCScore might not accurately capture diversity.\n\n- Computational results focus on small to medium datasets (up to 64k); scalability for extremely large datasets remains unproven.", "other_comments_or_suggestions": "- Figure 4 is very small and is quite hard to read.", "questions_for_authors": "1. Can you provide more details about the human evaluation protocol? Specifically, how was inter-annotator agreement measured, and what were the specific instructions given to evaluators?\n\n2. Have you tested DCScore on non-basic-text modalities (e.g., code, mathematical expressions, structured data)? If not, what adaptations would be necessary to apply it to these domains?\n\n3. How does DCScore perform on datasets with substantial outliers or on highly imbalanced datasets where certain types of content dominate?\n\n4. Have you explored whether DCScore can be used for diversity-aware sampling from generative models, beyond just evaluation of already-generated datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnwWwkkpFg", "reviewer": "Reviewer_QL32", "summary": "The paper introduces DCScore, a novel method for measuring diversity in synthetic datasets from a classification perspective. From my analysis, the key innovation is reformulating diversity evaluation as a sample classification task, where each sample should be distinguishable enough to form its own class. The authors demonstrate this approach satisfies important theoretical properties while outperforming existing metrics across multiple datasets and evaluation criteria.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our work. We respond to the reviewer’s question as follows. **Limited by the space, we present our additional experiments in an anonymous URL** (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).\n\n---\n>Q1: It would be better to add more investigation of failure cases and limitations\n\nR1: Thank you for your suggestive comment. **One of the primary limitations of DCScore is its inapplicability to multimodal data**. This limitation arises from the challenges associated with feature extraction and alignment across different modalities, which can affect the calculation of the classification probability matrix in DCScore. **We will include a detailed discussion of this limitation in future versions of our paper and explore potential solutions in our future research.**\n\n>Q2: It would be better to add more discussion of the hyperparameter sensitivity of the classification temperature τ.\n\nR2: We would like to clarify that we **have conducted sensitivity experiments on the classification temperature $\\tau$ in Section 5.4** of our paper. These experiments explore how different values of $\\tau$ influence the classification resolution. Specifically, a lower $\\tau$ enhances the discriminative ability of DCScore to different samples.\n\n>Q3: It would be better to add diversity evaluations on more real-world synthetic datasets beyond those augmented by LLMs.\n\nR3: Thank you for your suggestion. We have conducted several experiments across different existing text and image datasets. Specifically, in **Figure 4 of the anonymous URL**, we present diversity scores for the AGNews/SST/Yelp_A.P. datasets augmented by AttrPrompt[1]. In **Figure 2 of the anonymous URL**, we show results for image data. Overall, DCScore shows strong correlation with baseline methods.\n\nReference:\n[1] Large language model as attributed training data generator: A tale of diversity and bias, Neurips 2023.\n\n>Q4: From my experience, it would be better to add discussion of connections to other classification-based metrics in machine learning beyond just diversity measurement.\n\nR4: Thank you for your thoughtful review comment. We will include a discussion on classification-based metrics in machine learning in future versions of our paper. To the best of our knowledge, we **have identified classification-based methods primarily within the domain of metric learning**. To further enhance our work, we would greatly appreciate any additional guidance or references within this topic.\n\n>Q5: The paper should discuss recent work on classification-based evaluation metrics, particularly \"Beyond Accuracy: Behavioral Testing of NLP Models with CheckList\" (Ribeiro et al., 2020) which provides relevant insights about classification-based evaluation.\n\nR5: Thank you for your suggestion. This paper (Beyond Accuracy: Behavioral Testing of NLP Models with CheckList) primarily focuses on guiding users in conducting Behavioral Testing of NLP Models, but it appears to not mention classification-based evaluation metrics. However, we recognize the relevance of this work to our paper and **will include a discussion of its insights in the related work section of the future version.** \n\n>Q6:  The main weakness is limited discussion of the method's applicability to non-text modalities.\n\nR6: Thank you for your valuable review. We conduct experiments on the image modality (colored mnist dataset), please refer to **Figure 2 of the anonymous URL**. We follow the setting of [1] and observe that **DCScore presents higher correlation with the label number compared to VendiScore.**\n\nReference:\n[1] Ospanov et al., Towards a Scalable Reference-Free Evaluation of Generative Models, NeurIPS 2024.\n\n>Q7: 1.How does the method perform on multi-modal synthetic data where text is just one component? 2.What is the sensitivity of the method to the choice of classifier architecture beyond the examined options?\n\nR7: For 1, our method has limitations in evaluating multi-modal synthetic data, which is an area we plan to explore further. Specifically, **the effectiveness of DCScore in accurately evaluating multi-modal data depends on the extraction of multi-modal representations and the alignment of different data modalities**. We believe this is a very worthwhile research direction and appreciate your guidance on this matter. \n\nFor 2, in addition to the factors we have already explored, we believe **the sensitivity of the classifier architecture lies in its ability to effectively distinguish differences between samples**. The fundamental requirement for our method to function correctly is that the classifier can accurately identify distinct samples.", "claims_and_evidence": "The central claims about DCScore's effectiveness are well-supported through extensive experiments across different evaluation scenarios. From my experience, the reported correlations (Spearman's ρ > 0.96) with multiple diversity pseudo-truths are quite strong. However, it would be better to add more investigation of failure cases and limitations", "methods_and_evaluation_criteria": "From my analysis, the validation of theoretical properties (effective number, symmetry, etc.) strongly supports the method's soundness. However, it would be better to add more discussion of the hyperparameter sensitivity of the classification temperature τ.", "theoretical_claims": "I have carefully reviewed the theoretical foundations and proofs in Section 4.2 and Appendix B. The axiomatic guarantees (effective number, identical samples, symmetry, monotonicity) are mathematically sound and well-proven. The complexity analysis comparing with VendiScore is thorough.", "experimental_designs_or_analyses": "From my perspective, a key strength is using multiple correlation measures (τg, human, LLM) as pseudo-truths. However, it would be better to add diversity evaluations on more real-world synthetic datasets beyond those augmented by LLMs.", "relation_to_broader_scientific_literature": "The work builds meaningfully on existing diversity metrics while making novel contributions. From my experience, it would be better to add discussion of connections to other classification-based metrics in machine learning beyond just diversity measurement.", "essential_references_not_discussed": "The paper should discuss recent work on classification-based evaluation metrics, particularly \"Beyond Accuracy: Behavioral Testing of NLP Models with CheckList\" (Ribeiro et al., 2020) which provides relevant insights about classification-based evaluation.", "other_strengths_and_weaknesses": "The key strength is the novel classification perspective that provides both theoretical guarantees and strong empirical results. The main weakness is limited discussion of the method's applicability to non-text modalities.", "other_comments_or_suggestions": "NA", "questions_for_authors": "How does the method perform on multi-modal synthetic data where text is just one component?\nWhat is the sensitivity of the method to the choice of classifier architecture beyond the examined options?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for measuring the diversity in synthetic datasets in the context of classification.
There method addresses a timely question about measuring diversity in synthetic datasets with a computationally efficient procedure. The proofs of the theoretical claims appear to be sound. 
The comparison with the Vendi score, however, appears to be incomplete, as it is unclear whether the proposed score is genuinely a more useful metric.
Decision: Accept (poster)