Evaluating Diversity of LLM-generated Datasets: A Classification Perspective

Authors: Yuchang Zhu, Huizhe Zhang, Bingzhe Wu, Jintang Li, Zibin Zheng, Peilin Zhao, Liang Chen, Yatao Bian
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Diversity evaluation, LLM-generated dataset, Large language models

Abstract:
LLM-generated datasets have been recently leveraged as training data to mitigate data scarcity in specific domains. However, these LLM-generated datasets exhibit limitations on training models due to a lack of diversity, which underscores the need for effective diversity evaluation. Despite the growing demand, the diversity evaluation of LLM-generated datasets remains under-explored. To this end, we propose a diversity evaluation method for LLM-generated datasets from a classification perspective, namely, DCScore. Specifically, DCScore treats the diversity evaluation as a sample classification task, considering mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, demonstrating it as a principled diversity evaluation method. Additionally, we show that existing methods can be incorporated into our proposed method in a unified manner. Meanwhile, DCScore enjoys much lower computational costs compared to existing methods. Finally, we conduct experiments on LLM-generated datasets to validate the effectiveness of DCScore. The experimental results indicate that DCScore correlates better with various diversity pseudo-truths of evaluated datasets, thereby verifying its effectiveness.

Decision: reject
Reviewer scores: 5, 3, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730555254499

thread_type:
official_review

<thread_object>
creator: reviewer_fedn

<content>
summary:
This work proposes DCScore to evaluate the diversity of LLM-generated datasets.DCScore treats the diversity evaluation as a sample classification task, considering mutual relationships among samples. The proposed method is somewhat novel to me.

strengths:
The proposed method can be used to evaluate the diversity of LLM-generated datasets and capture the mutual relationships among samples.
Extensive experiments show that DCScore exhibits a stronger correlation with diversity parameters and human judgment than baseline metrics.
The authors evaluate both the performance and efficiency of DCScore, highlighting its effectiveness.

weaknesses:
1. Some related works [1,2] evaluate the quality of augmented data from the data augmentation perspective. They also involve diversity-related metrics. It is suggested that authors discuss and compare with them.
[1] Gontijo-Lopes, Raphael, et al. "Affinity and diversity: Quantifying mechanisms of data augmentation." arXiv preprint arXiv:2002.08973 (2020).
[2] Yang, Suorong, et al. "Investigating the effectiveness of data augmentation from similarity and diversity: An empirical study." Pattern Recognition 148 (2024): 110204.
2. The diversity-sensitive components are important for DCScore calculations. How are these components computed? Since this is one of the most critical parts of the proposed method,  more specific details are needed.
3. I got a little confused about the definition and properties of DCScore. If I understand correctly, in the probability matrix P, P[i,i] denotes the probability of classifying T_i into class I. 1. What is the meaning of categories in the classification function? 2. If all samples are identical, what does the DCScore equal? If this is 1, the probability is one-hot and contracts to softmax calculations. If not, then the DCScore does not equal 1. The same is true for cases where all samples are distinct. More clarifications are needed.
4. On page 8, evaluation of existing datasets. How can we evaluate the effectiveness of different metrics? Are some datasets more diverse than others, and can the proposed metrics reflect this trend?
5. Is the diversity the higher, the better?  Can authors provide some takeaway conclusions for readers?
I will increase my score based on the authors' responses.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730784802784

thread_type:
official_review

<thread_object>
creator: reviewer_5mfj

<content>
summary:
The paper presents DCScore, a computationally efficient method for evaluating the diversity of LLM-generated datasets by framing diversity assessment as a sample classification task.

strengths:
- This paper studies an underexplored problem, i.e., evaluating the diversity of LLM-generated datasets.
- The proposed method is simple and computation-efficient, and it does not need any model training.
- Experiments show that the proposed evaluation metric aligns with human judgments, highlighting its effectiveness.

weaknesses:
1.	The method proposed in this paper seems to have a weak connection with classification, as it merely applies a softmax function based on a sample similarity matrix, which resembles instance discrimination in contrastive learning. The reviewer believes it is a stretch to interpret this as a classification problem.
2.	The method in this article is highly similar to VandiScore, except that it replaces the entropy of eigenvalues of the similarity matrix with matrix trace, yet the results in Table 2 show that both methods are very close.
3.	In addition to assessing dataset diversity, the reviewer suggests that a sample selection method should be proposed to eliminate redundant samples, thereby achieving a dataset with higher diversity and effectively reducing the overhead of data generation.
4.	The writing of this article needs further improvement; for instance, $\widetilde{T}$ is not clearly defined in the text; also, the description of the classifier $f$ and its referred softmax function are inconsistent and need to be adjusted.
5.	The method uses a sentence transformer to extract embeddings, and thus its effectiveness heavily relies on the generalization ability of the sentence transformer. It should be compared with other embedding models; additionally, there are many types of kernel functions, which were not considered in this paper.

questions:
1. Does the size of the generated datasets affect the diversity score?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730793576475

thread_type:
official_review

<thread_object>
creator: reviewer_scle

<content>
summary:
In this paper, the authors proposed a classification-based diversity evaluation method for LLM-generated datasets called DCScore, which treats diversity evaluation as a sample classification task and captures mutual relationships among samples. The authors provided theoretical verification that DCScore satisfies four axioms, including effective number, identical samples, symmetry, and monotonicity. Experiments show that DCScore exhibits a correlation with diversity parameters and human judgment compared to baseline metrics.

strengths:
1. The authors proposed an LLM-generated dataset diversity evaluation method from a classification perspective called DCScore, which treats the evaluation of each sample in the LLM-generated dataset as a distinct classification task.

2. The authors provided theoretical verification that DCScore satisfies four axioms, including effective number, identical samples, symmetry, and monotonicity.

weaknesses:
The paper has several weaknesses:

1. **Lack of Purpose and Application**: The significance of studying LLM-generated data diversity is unclear. The authors should clarify whether this research can aid smaller LLM training or accelerate LLMs' overall training, as well as the practical applications of diversity evaluation.

2. **Unclear Novelty of DCScore**: It is unclear whether DCScore is a new method or a simplified version of existing approaches. The paper should better highlight its novelty and contribution to the field.

3. **Unexplained Correlation Investigation**: The objective of investigating the correlation between DCScore and diversity pseudo-truth is not well explained. The authors should clarify how this experiment supports the overall thesis of the paper.

4. **Weak Performance and Experimental Design**: DCScore performs poorly in the zero-shot setting (Table 2), but no explanation is provided. Additionally, the rationale behind using Text Classification and Story Completion tasks needs to be better justified.

questions:
Here are a few questions I would like to ask the authors:

1. Large Language Model (LLM)-generated datasets have found widespread applications, but what is the purpose or significance of studying the diversity of LLM-generated data? For instance, can research on the diversity of LLM-generated data contribute to training smaller LLMs or accelerate LLM training? Additionally, what are the potential application scenarios for investigating the diversity of LLM-generated data? I would appreciate it if the authors could provide further clarification on this point.

2. In Method 4.3, is DCScore a fusion or a simplified version of existing methods? What is the significance of the discussion in this section?

3. In the experiments, the objective of “investigating the correlation between the diversity evaluation of DCScore and the diversity pseudo-truth” is unclear. Could the authors elaborate on this goal and relate it to the central premise of the paper?

4. In Table 2 of the experiments, under the zero-shot setting, the performance of DCScore is not very strong. Could the authors provide an explanation for this? What is the rationale behind conducting experiments under the Text Classification and Story Completion settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731131927122

thread_type:
official_review

<thread_object>
creator: reviewer_wwpi

<content>
summary:
The paper proposes a new metric DCScore to measure the diversity of the generated dataset. Specifically, it first calculates a similarity matrix and using a softmax like function to convert the similarity matrix to represent the diversity. The final score will be the trace of the diversity summarization matrix.

strengths:
The proposed metric is simple and easy to implement. It achieves comparable performance as other methods while shows slightly computation benefit when the dataset is small. The paper provides good proof of that DCScore satisfies several axioms defined for a principled diversity metric.

weaknesses:
The paper claims DCScore has lower computational expense compared to other baseline methods. The theoretical analysis is questionable. In line 482, it says that the DCScore has a complexity of $O(n^2)$, which is incorrect, considering the calculation of the Kernel matrix, the computational complexity is at least $O(n^2d)$, where d is the dimension of the feature. The paper claims that the VendiScore has a computational complexity of $O(n^3)$, which also not true. VendiScore calculates the sum of the eigenvalues of the Kernal matrix, however, the sum of the eigenvalues of the Kernel matrix is equivalent to the sum of the eigenvalues of the covariance matrix, which makes the computation complexity to be $O(d^2n)$ instead of O$(n^3)$, as mentioned in section 3.2 of the VendiScore paper. Considering in real problem, n is always much larger than d, otherwise we shouldn’t care too much of the computation. In the computation cost experiment, the paper only compares the computation for a dataset with at most 4,000 samples, which is unfair for VendiScore. 

Performance-wise, from table 2, I also cannot see DCScore is  better than VendiScore. VendiScore is slightly better than DCScore in zero-shot case while DCScore is slightly better in few-shot setting. However, the difference is very minor. 

It’s a bit unclear to me the motivation of the classifier-based method, and why it’s better. VendiScore has the same properties of DCScore as well as performance. As I mentioned in the computation complexity part, DCScore may be not have lower computational cost as well.

questions:
It would be very helpful if authors can help clarify the computation cost and help me better understand the motivation of using a classifier based method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731179775535

thread_type:
official_review

<thread_object>
creator: reviewer_hpjq

<content>
summary:
This work propose a new metric, DCScore that measures the diversity of generated datasets. DCScores formulate the evaluation as a classification tasks and consists of three stages: text representation, pairwise similarity and diversity summarization to measure the dataset diversity. Experiments demonstrate DCScore outperforms other baselines, including Distinct-n, K-means inertia and VendiScore.

strengths:
- The manuscript is logically organized and easy to follow. The main method is clearly explained with illustration figures.
- Several ablation studies are conducted, helping to understand the effectiveness of the proposed methods.
- Both empirical and theoretical justification are conducted to evaluate DCScore.

weaknesses:
- For the goal of holistic analysis, DCScore involves pairwise similarity comparison, which is quadratic complexity as the number of samples, making it hard to scaling for large datasets, e.g., million-level samples, even billion level samples. In Line 63, previous methods struggles to offer a holistic evaluation of a dataset due to its reliance on pairwise similarity while DCScore seems also suffers to this issue.
- The improvements in Table 2 is modest. And it wound be better to analysis whether larger DCScore brings better performance when the models are optimized on the corresponding datasets.
- Section 4.3 can be simplified, most of the content are introducing some details about baseline method, which is not the contribution of this work.
- Seems like the main different between DCScore and VendiScore is the diversity summarization where VendiScore involves the eigenvalue computation, and DCScore use a more efficient softmax operation. More insights about why the eigenvalue computation is redundant wound further enhance the quality of this work.

questions:
Please refer to the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732502797241

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General statement and Gentle reminder

comment:
Dear Reviewers,

We sincerely thank all the reviewers for their valuable feedback and suggestions. **We have now addressed all the reviewers' questions one by one and completed the revisions to our paper**, with the modified sections highlighted in blue. As the author-reviewer discussion deadline approaches, we would like to inquire if there are any further questions regarding our paper that need to be addressed. **We would appreciate it if you could take our responses into consideration when making the final evaluation of our work.**

Kind regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732787555928

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to reviewers

comment:
We sincerely thank the reviewers for their time and valuable comments. We are pleased to have addressed all the concerns raised by Reviewers wWpi, ScLE, and 5MfJ. As the revised PDF  uploading deadline approaches, we have further revised the paper to better highlight its significance. The new changes are marked in **red**, while the previous changes are marked in **blue**. The specific modifications are as follows:

- **Introduction:** We have revised the introduction to better emphasize the motivation behind our paper and the classification-based approach.

- **Application of Diversity Evaluation:** As shown in **Appendix A.2**, we have added a discussion on the related work regarding the application of diversity evaluation metrics, highlighting the significance of our work.

- **Alternative Implementations of a Classification Perspective:** As shown in **Appendix H**, we have provided an alternative implementation of the classification perspective that is more computationally efficient, emphasizing the differences and necessity of the classification-based approach compared to existing methods.

If you have any questions or need further clarification, please do not hesitate to contact us.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733306720753

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear Reviewers and ACs,

We sincerely thank the reviewers for their comments and the associate editors for their management of our submission. We have addressed all the reviewers' questions one by one and have revised our manuscript accordingly. Except for one reviewer who did not provide feedback, the other reviewers have indicated that our responses have resolved their concerns. As the rebuttal deadline approaches, we summarize our responses to each reviewer below.

**Reviewer hpJq:**
1. We added complexity analysis (Appendix F) and computational cost experiments on large datasets (Appendix E.3) to tackle the concerns about the complexity of DCScore.
2. We clarified Table 2 and added the performance impact experiments on dataset diversity (Appendix E.2).
3. We simplified Section 4.3 and clarified the motivation of this section.
4. We clarified the motivation for using the classification perspective.

**Reviewer wWpi:**
1. We added a detailed complexity analysis (Appendix F) and computational cost experiments on large datasets (Appendix E.3) to solve the concerns about the complexity of DCScore.
2. We clarified the advantages of DCScore and the motivation of the classifier-based method. Please refer to the Introduction (Section 1) and related work (Section 2) for more details.

**Reviewer ScLE:**
1. We added a related work discussion on applications of diversity evaluation methods (Appendix A.2) to clarify the significance of our work.
2. We further clarified the difference between our work and existing methods and indicated the motivation of Section 4.3.
3. We further clarified the fundamental idea of our experiment (Section 5.1).
4. We further clarified the selection of downstream tasks of our experiments and the explanation of Table 2.

**Reviewer 5MfJ:**
1. We clarified the connection of our work with classification.
2. We further clarified the differences between our work and VendiScore.
3. We provided an implementation of our work on the sample selection task.
4. We added experiments of our work on various embedding models (Appendix E.4) and kernel functions (Appendix E.5).
5. We proofread our submission again to improve our writing.

**Reviewer fedN:**
1. We added some related work discussion into the additional related works section (Appendix A.2).
2. We clarified diversity-sensitive components and revised the section 3.2.
3. We further clarified the definition and properties of DCScore.
4. We clarified the motivation for the evaluation of existing datasets.
5. We further clarified the impact of dataset diversity and added the downstream task experiment (Appendix E.2). 
6. We further clarified the explanation of our additional experiments (Appendix E.2).
7. We further clarified the use of DCScore.
8. We further clarified the practical significance and added an application discussion (Appendix A.2) of our work.
9. We updated the contents of Appendix E.3,4,5.

In summary, we have addressed most of the reviewers' concerns and received their acknowledgment, except for one reviewer who did not respond. The reviewers acknowledged the contribution and significance of our work. We sincerely hope that the reviewers and ACs will take our rebuttal and revisions into account during the final evaluation.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents a new metric, called DCScore, to measure the diversity of generated datasets. While the method is straightforward and easy to implement, significant concerns have been raised regarding the theoretical analysis in comparison to baseline methods. Additionally, the novelty of DCScore remains unclear, and the metric shows marginal improvement over VendiScore. Given these issues, the innovation and contribution of this work remain questionable.