On the Diversity of Synthetic Data and its Impact on Training Large Language Models

## Abstract

Abstract The rise of Large Language Models (LLMs) has accentuated the need for diverse, high-quality pre-training data.
Synthetic data emerges as a viable solution to the challenges of data scarcity and inaccessibility.
While previous literature has focused predominantly on the quality and quantity of real data, our work enables the measurement of diversity in synthetic data and explores its impact on LLM performance.
We study the downstream effects of synthetic data diversity during both the pre-training and fine-tuning stages by introducing a new diversity metric, LLM cluster-agent , designed to evaluate the diversity of synthetic datasets.
Through a series of controlled experiments with models of 350M and 1.4B parameters, we demonstrate that the proposed cluster-based LLM scoring of diversity correlates positively with both pre-training and supervised fine-tuning performance.
Our findings also reveal that synthetic data diversity in pre-training affects supervised fine-tuning more significantly than pre-training itself, even for smaller models.
We hope this study advances our understanding of the optimal use of synthetic data in LLM training and opens new avenues for efficient data generation processes.

## 1 Introduction

A common hypothesis behind the success of Large Language Models (LLMs) (Radford et al., ; Brown et al., ; OpenAI, ; ; Touvron et al., ) is the scaling law of computing, model size, and, perhaps the most important, high-quality pre-training data (Kaplan et al., ; Wei et al., ; Muennighoff et al., ).
The most capable LLMs these days often have been pre-trained on trillions of tokens (Bai et al., ; Dubey et al., ; OpenAI, ).
Acquiring such massive amounts of high-quality data has become more challenging (Villalobos et al., ).

As a remedy, synthetic data have been widely adopted in training LLMs, which are relatively easier to obtain with more controllable quality (Bauer et al., ; Liu et al., ; Long et al., ).
For example, Phi series (Gunasekar et al., ; Li et al., ; Javaheripi et al., ; Abdin et al., ) used a large amount of textbook-style synthetic data with real data in pre-training, empowering the promising performance of smaller-scale LLMs.
Synthetic data for programming and math have also been adopted to improve the coding and reasoning abilities of LLMs (Guo et al., ; Yu et al., ; Shao et al., ).
Previous studies have also focused on synthetic data for supervised fine-tuning (Zelikman et al., ; Huang et al., ; Liu et al., ; Eldan & Li, ; Chen et al., ; Huang et al., ), instruction tuning (Wang et al., ; Xu et al., ; Li et al., ; Wang et al., ; Chan et al., ; Li et al., ; ; Wu et al., ), downstream transferring (Meng et al., ; Ye et al., ), and evaluation (Zhu et al., ; ; ).

Despite the wide usage of synthetic data,
understanding what aspect of and how the synthetic data affect the performance of LLMs still remains largely unexplored, especially for pre-training.
In the past, many studies have shown that both the quality and quantity of real data matters for LLM pre-training (Kaplan et al., ; Sorscher et al., ).
While the effectiveness of quantity of real data has been extensively verified on LLMs as the scale of training tokens increases (Radford et al., ; Brown et al., ; Computer, ; Touvron et al., ; Dubey et al., ), the quality of real data, affected by various factors such as corruption (Elazar et al., ), bias (Gallegos et al., ), toxicity (Bender et al., ), duplication (Lee et al., ; Xue et al., ), and diversity (Tirumala et al., ), to name a few, is more difficult to validate due to the co-functioning of these factors (Kreutzer et al., ; Longpre et al., ).
Some recent research studied different quality factors of real data and concluded that the quality of real data is more important than quantity (Soldaini et al., ; Penedo et al., ; Groeneveld et al., ; Tan & Wang, ; Deitke et al., ).
However, it is still unclear whether these conclusions also apply to synthetic data pre-training.

Figure: Figure 1: Linear regression of LLM cluster score and benchmark performance of (a) pre-trained 350M; (b) pre-trained 1.4B; (c) supervised fine-tuned 350M; and (d) supervised fine-tuned 1.4B models. Each scatter represents a synthetic dataset with size corresponding to the number of tokens.
Refer to caption: x1.png

In this paper, we propose to study the diversity, as one of the most important quality factors (Tirumala et al., ; Sachdeva et al., ), of the pre-training synthetic data.
Existing studies on synthetic data in pre-training either only present methods of creating them (Allal et al., ; ) or provide findings that are restricted to relatively small scales (Wu et al., ; Allen-Zhu & Li, ; Ye et al., ; Zhu & Li, ; Allen-Zhu & Li, ; Yang et al., ), with limited understanding on how exactly diversity of the synthetic tokens affect the training of LLMs.
However, studying the diversity of synthetic data presents two main challenges.
First, the lack of an effective metric for measuring the diversity of text data (Lee et al., ; Shaib et al., ; Tirumala et al., ; Ankner et al., ),
and second, the difficulty of conducting controlled large-scale experiments with synthetic tokens due to the high cost of generation and various aspects influencing their diversity.

To overcome the obstacle, we propose a diversity measure pipeline by automatically directing LLMs to perform a clustering of text corpus, termed LLM Cluster-agent.
Specifically, we design prompts that guide LLMs to summarize the characteristics from randomly sampled data points that can best capture the underlying diversity in the corpus and then perform clustering based on the characteristics with a self-verification mechanism.
An LLM cluster score is computed from the clustering results as a measure of text diversity.
The proposed pipeline is wrapped as a diversity metric toolkit, and we showcase its effectiveness, consistency, and scalability with different LLMs on large-scale synthetic data, where traditional diversity metrics fail and produce significantly inconsistent results.

To perform controlled experiments on synthetic data diversity, we extract 620,000 topics from Wikipedia and then use them to seed the synthetic generation.
With the proposed LLM Cluster-agent metric, we generate synthetic datasets with various levels of diversity from different perspectives, including the underlying distribution, prompts and models of synthetic generation, and ratios between synthetic and real tokens.
As the first large-scale study on synthetic data diversity, we pre-train a set of language models of 350M and 1.4B parameters on the combination of 34B real and the generated synthetic tokens and supervised fine-tune them to study the downstream effects.
We show that:

- •
LLM cluster score positively correlates with both the per-training and supervised fine-tuning performance of LLMs, as shown in Fig. 1. It thus shows great potential to be applied in practical and large-scale LLM synthetic data pre-training and predict the performance in the future.
- •
The underlying distribution of synthetic data, in terms of the number of topics and the number of generations per topic, matters for LLM performance. In Section 3.3, we show that more unique topics usually present better diversity, and too large the number of generations per topic may introduce redundancy in synthetic data generation, thus hurting the performance.
- •
Prompts incorporating different text styles and various targeted audiences for synthetic data generation can significantly boost the diversity and thus the LLM performance. In Section 3.4, we show that models trained on synthetic data with different styles and personas present the best performance and outperform models trained on Cosmopedia v0.1 and v0.2 (Allal et al., ; ).
- •
Better LLMs-generated synthetic data present more diversity in synthetic generation. In Section 3.5, we show that the diversity and performance of trained models with GPT-4o generated synthetic data is better than GPT-3.5, and 8B instruct Llama-3.1 is better than 7B instruct Mistral.
- •
More balanced ratio between real and synthetic tokens benefits LLMs the most, and over-weighted synthetic tokens may hurt performance due to diversity deterioration, as shown in Section 3.6.
- •
More interestingly, as shown in Fig. 1 and discussed in Section 3.7, while the pre-training performance of smaller models tends to saturate faster than larger models as the diversity in synthetic tokens increases, larger diversity still significantly benefits the supervised fine-tuning performance.

We hope that the proposed diversity metric demonstrates potential to be applied in real-world LLM pre-training with synthetic data in the future, and that the insights from our study could contribute to more efficient and diverse synthetic data generation processes for training LLMs in practice.

## 2 Metrics for Measuring Synthetic Data Diversity

Measuring the diversity in large-scale text data is very challenging due to the complex nature of language (Lee et al., ; Shaib et al., ).
Different metrics have previously been used to measure the diversity of text data, and we broadly categorize them into two types: heuristic-based and model-based.
Heuristic-based metrics, such as vocabulary size, n-gram diversity (Li et al., ; Meister et al., ), and self-repetition score (Salkar et al., ), often provide a very limited view, focusing only on statistical variations within the text without capturing deeper semantic nuances.
Model-based methods such as K-means clustering (Abbas et al., ) and homogenization score (Lin & Och, ; Shaib et al., ) struggle with large-scale and context-rich datasets, as they rely on predefined features, which can oversimplify the true diversity present in the data.
These limitations are further compounded in synthetic text data generated by LLMs due to similar patterns in part-of-speech tagging and syntactic often present in them (Rosenfeld & Lazebnik, ; Shaib et al., ), making it difficult to assess diversity accurately.
This motivates us to address the gap by proposing an LLM-based metric to uncover the intricate and latent structures within the data.

### 2.1 LLM Cluster-agent

Given a text corpus $X=\{x_{i}\}$ with in total $|X|$ text samples, to allow LLMs to measure their diversity, we propose to originate the measure from the principle of entropy, i.e., capture the underlying distribution of clusters and cluster sizes.
However, there are two challenges that prevent LLMs from performing clustering directly.
First, it is difficult to define the proper criteria for LLMs to cluster that captures the true distribution.
Second, due to the limited context length of LLMs(^1^11Although LLMs nowadays can support 128K context length or even more, the quality of response usually degenerates as the context length increases.), one cannot directly feed the entire text corpus to LLMs for clustering as in traditional clustering methods.

Figure: Figure 2: Pipeline, prompt, and example outputs of the proposed LLM Cluster-agent. LLM Cluster-agent first generates metadata and metrics with attributes and scores that captures the underlying distribution and then uses these criteria to perform clustering with an extra self-verification step.
Refer to caption: x2.png

We thus introduce LLM Cluster-agent, a diversity measure pipeline that leverages LLM’s abilities to interpret semantic meanings and to understand rich contexts of text samples for clustering.
To overcome the above challenges, we design LLM Cluster-agent to perform an iterative clustering based on $K$ text samples each time, according to the clustering criteria that are also summarized by the LLM.
More specifically, our method includes the following steps, as shown in Fig. 2.

Metadata and metric generation.
We first design two types of clustering criteria: metadata and metrics.
The metadata are used to guide LLM to summarize the detailed attributes of the text samples and the metrics are used for scoring the samples and reasoning behind the clustering.
Due to the massive amount of the text corpus, a metadata and metric generation prompt is used to extract 3-5 metadata and metrics from the randomly selected $J$ samples of the corpus and repeat the process $M$ times.
A metadata and metric gathering prompt is then designed to individually collect and summarize the most frequent ones from the multi-round generation.
The collected metadata and metrics are used for clustering criteria.
We find that it is beneficial to highlight the criteria at the top of our clustering prompt in the next step to emphasize the focus of clustering, and thus we exploit another criteria summary prompt to summarize the high-level definition of the gathered metrics.

Cluster generation and verification.
After obtaining a set of metadata and metrics and their definition of high-level criteria, we design a clustering prompt.
Due to the context limit of LLMs, we similarly randomly select $K$ samples from the corpus and prompt LLMs to group the $K$ samples into different clusters according to the attributes defined by the metadata and scoring rules defined by metrics.
We also include instructions for LLMs to give the reasoning for each cluster.
After obtaining the clusters, we use a cluster verification prompt to inspect whether the reasoning and the samples in the cluster are valid.
We find that this additional verification step is very essential in removing some unreasonable clusters.
We repeat this process $N$ times, and each generation will produce a result of the number of clusters $\mathcal{C}$ and the number of samples per cluster $\mathcal{S}$ from these $K$ samples.
Eventually, we define LLM Cluster score as the diversity measure by averaging the cluster results from the $N$ times generation:
$\mathcal{D}=\frac{1}{N}\sum_{i=1}^{N}\frac{\mathcal{C}_{i}}{\mathcal{S}_{i}}$,
where $\mathcal{D}$ denotes the diversity score, and $\mathcal{C}_{i}$ and $\mathcal{S}_{i}$ are the number of clusters and the number of samples per cluster in the $i$-th generation.
This approach enables the identification of diverse themes, topics, or stylistic variations within the synthetic dataset.
The full prompts used for each step are shown in Appendix D.
We also present the ablations of the pipeline design, prompt design, and the parameters in Section 3.8 and Section B.2.

**Table 1: Summary of existing and ours diversity metrics.**
| Metric | Formulation | Type | Reference |
| --- | --- | --- | --- |
| Context Length | $\frac{1}{N}\sum_{i=1}^{N}|x_{i}|$ | Heuristic | - |
| Self-Repet. | $\log\left(k\sum_{i=1}^{k}(\hat{N_{i}}+1)\right)$ | Heuristic | Salkar et al. () |
| N-gram Div. | $\frac{\text{Unique n-grams in }X}{\text{Total n-grams in }X}$ | Heuristic | Padmakumar & He (); Adelani et al. (); Li et al. () |
| Comp. Ratio | $\frac{\text{Orig. size of }X}{\text{Comp. size of }X}$ | Heuristic | Shaib et al. () |
| Perplexity | $2^{-\frac{1}{|X|}\sum_{i=1}^{|X|}\log_{2}P_{\text{GPT-2-L}}(x_{i})}$ | Model | Ankner et al. () |
| Perplexity Gap | $|\text{PPL}_{\text{GPT-2-L}}-\text{PPL}_{\text{GPT-2-XL}}|$ | Model | - |
| K-means | Train.: $\underset{\mu_{i}}{\min}\sum_{i}^{k}\sum_{x_{j}}\|x_{j}-\mu_{i}\|^{2}$<br>Infer.: $i=\arg\min_{i}\|x_{j}-\mu_{i}\|^{2}$ | Model | Abbas et al. (); Sachdeva et al. () |
| LLM Cluster | $\mathcal{D}=\frac{1}{N}\sum_{i=1}^{N}\frac{\mathcal{C}_{i}}{\mathcal{S}_{i}}$ | Model | - |

### 2.2 Baseline Metrics

We include several commonly used heuristic-based and model-based diversity metrics as baselines (Shaib et al., ).
Context Length (CL) measures the average token length of the text corpus.
Self-Repetition Score (SRS) quantifies the repetition of tokens within sentences, while N-Gram Diversity Score (NDS) measures the proportion of unique $n$-grams.
Compression Ratio (CR) compares the g-zip compressed size of the dataset to its original size.
Perplexity measures the uncertainty of a pre-trained model in predicting the next token and Perplexity Gap calculates the perplexity difference between a larger and a smaller model.
K-means Clustering utilizes feature embeddings from a pre-trained model to cluster the data.
A summary of the diversity metrics is shown in Table 1 and we further describe these diversity metrics in Appendix C.
Apart from our baseline measures to quantify the diversity of pre-training data, there are other measures, such as the Homogenization Score (Lin & Och, ; Shaib et al., ) based on ROUGE-L (Lin, ), BERTScore (Zhang et al., ), Hypergeometric Distribution D (McCarthy & Jarvis, ), and Part-of-Speech Compression Ratio (POS-CR) (Shaib et al., ). However, these metrics are generally computationally prohibitive.
Due to this computational and experimental limitation, we do not include these metrics in our study.

## 3 Synthetic Data Diversity in Pre-training

With the proposed LLM Cluster-agent metric, we conduct a series of controlled experiments by generating synthetic data with various levels of diversity and training models on them.
We reveal a linear correlation between the LLM Cluster Score and training performance from the perspectives of underlying distribution, prompts and models for generation, and ratio of real and synthetic tokens.

### 3.1 Experiments Setup

Pre-training.
We adopt the Llama architecture (Touvron et al., ) with a context length of 2,048 and the Codegen-Mono (Li et al., ; Nijkamp et al., ) tokenizer with a vocabulary size of 50,304.
We primarily use 350M and 1.4B models and pre-train all models on the combination of real and synthetic data, except for the baselines on real data only.
For real data, we use filtered web data, consisting of the Wikipedia subset and part of the C4 (Raffel et al., ) subset of Dolma (Soldaini et al., ), code data, consisting of the filtered the Stack (Kocetkov et al., ), StackOverflow, and Code Contest (Li et al., ) as in Phi-1.5 (Li et al., ), and math data from the filtered OpenWebMath (Paster et al., ) subset of Dolma.
The real data in total contain 34B tokens, where the ratio of web, code, and math tokens is 4:1:1.
For synthetic data, we generate variants with different underlying distributions, prompts, and models for generation (more details in the following sections).
Our experiments mainly involve two ratios of real (web) and synthetic tokens: 4:1 for smaller synthetic data experiments, and 1:1 for larger ones, following Phi-1.5.
More ratios are also studied.
We train 350M and 1.4B models for a total of 50B and 150B tokens, respectively.

Supervised Fine-tuning. In addition to pre-training, we also conduct supervised fine-tuning (SFT) to study the effect of diversity in pre-training data inherited to downstream performance (Chen et al., ).
After pre-training the models, we supervised fine-tune them for 3 epochs on the combination of GPT-4 filtered version of the Alpaca (Taori et al., ) and FLANv2 (Longpre et al., ).
The learning rate of the AdamW optimizer for fine-tuning is set to 2$e$-5 and weight decay to 0.

Benchmark Evaluation. To evaluate the performance of both the pre-trained model and supervised fine-tuned model, we use WinoGrande (Pîrtoacă et al., ), ARC-Easy (Pîrtoacă et al., ), ARC-Challenge (Ferré, ), BoolQ (Clark et al., ), SIQA (Bauer & Bansal, ), PiQA (Bisk et al., ), HellaSwag (Zellers et al., ), and COPA (Roemmele et al., ).
We report the zero-shot accuracy using LM-Eval Harness (Gao et al., ) for both pre-trained and supervised fine-tuned models.
We utilized a system prompt consistent to fine-tuning to evaluate tuned models.

Diversity Evaluation.
To effectively evaluate the diversity of the large-scale synthetic corpus, we employ bootstrapping to obtain robust results.
Specifically, we randomly select one million text samples from the corpus and run the baseline diversity metrics and our proposed LLM cluster metric on this subset.
We repeat the process for 10 rounds with different random seeds and report the average results and the corresponding error bar.
For the model-based metrics, we use BERT-L (Devlin, ) embeddings for K-means clustering, and GPT-2-L and GPT-2-XL (Radford et al., ) to calculate perplexity and perplexity gap.
For K-means clustering, we set the number of clusters to $10K$, which we find as a good trade-off between speed and accurate measurement.
We set $K=10$ and $N=5$K for the proposed LLM Cluster-agent.
We also find $J=5$ and $M=100$ is good enough to obtain meaningful clustering criteria, as we show in Section B.2.
More details of the model architecture, training parameters, and evaluation datasets are shown in Appendix A.

### 3.2 Seeding Synthetic Data Generation

To ensure both reasonable quality and diversity of the synthetic data generation, we mainly adopt GPT-4o as the base model for the generation of synthetic text data and utilize a set of pre-defined topics as our generation seeds.
The topic generation seeds are obtained by first scrawling

Figure: Figure 3: Top topic seeds.
Refer to caption: x3.png

the web pages from Wikipedia and then prompting GPT-4 to extract a hierarchy of topics and a set of keywords covered in the content of the page.
A visualization of the most frequent topics (and their sub-topics) is shown in Fig. 3.
We further run a de-duplication process on all the topics collected and obtain in total 620,000 topics to ensure the wide coverage of knowledge in synthetic data.
More detailed distribution and examples of topic seeds and keywords are shown in Appendix E.
Our synthetic data generation is based on these topic seeds and keywords in the following experiments.

### 3.3 On the Underlying Distribution of Synthetic Data

We first study the effect of the underlying distribution of synthetic data on LLM’s performance, i.e., the number of topics $\mathcal{T}$ the and number of generations per topic $\mathcal{G}$ used for synthetic data generation.

Synthetic Data Generation.
To generate the synthetic data with varying underlying distribution, we sample $\mathcal{T}\sim\{100\text{K},300\text{K}\}$
seeding topics and perform $\mathcal{G}\sim\{10,20,30\}$ textbook-style data generation using a simple prompt template that specifies the topic and keywords for each generation.

**Table 2: Synthetic token counts of varying underlying topics $\mathcal{T}$ and generations $\mathcal{G}$.**
| $\mathcal{T}$ | 100K | 300K |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\mathcal{G}$ | 10 | 20 | 30 | 10 | 20 | 30 |
| # Tokens (B) | 0.58 | 1.01 | 1.48 | 1.74 | 3.04 | 4.43 |

Following the setup of experiments in Phi-series, we also generate a question with answers and step-by-step explanations based on the content at the end of each synthetic sample.
We refer to this prompt template as Topic.
The detailed prompt template and output examples are shown in Appendix F.
We present the token count of the synthetic data generated using this prompt in Table 2.
For fair comparison, we increase the sampling weight to make the effective synthetic tokens as 4.5B, and combine with the 34B real tokens for pre-training the models.

Figure: Figure 4: Diversity results of varying underlying number of topics ($\mathcal{T}$) and number of generations per topic ($\mathcal{G}$) in synthetic data. (a) Average length of synthetic samples; (b) Self-repetition score; (c) Compression ratio; (d) N-gram diversity score; (e) Perplexity of GPT-2-L; (f) Perplexity gap between GPT-2-L and GPT-2-XL; (g) K-means cluster score of BERT-L embeddings; (g) LLM cluster score. Ours demonstrates the most significant difference in diversity, aligning with the underlying topic distribution. It also reflects the saturated and deteriorated diversity diversity as $\mathcal{G}$ increases.
Refer to caption: x4.png

Results.
After generating the synthetic data, we perform the diversity evaluation on them and report the results of different diversity metrics in Fig. 4.
Although baseline metrics might be able to measure the diversity of different datasets from various domains or model outputs, as reported by Shaib et al. (), they cannot discriminate the underlying distribution of synthetic data well, with trivial differences present in the metric values.
Similar observations persist even for model-based metrics such as perplexity and perplexity gap (Ankner et al., ).
One can also find that the traditional clustering method, i.e., K-means clustering, fails to capture the diversity of the underlying distributions, where the cluster score of synthetic tokens with $300$K topics is measured to be smaller than that of $100$K topics.
More importantly, the diversity measured by both the heuristic-based and model-based baseline metrics demonstrates different trends, which is difficult to interpret.

Figure: Figure 5: Benchmark average accuracy of pre-trained and supervised fine-tuned 350M and 1.4B models by varying underlying number of topics ($\mathcal{T}$) and number of generations per topic $\mathcal{G}$ in synthetic data. The performance of both pre-trained and supervised fine-tuned models well aligns with our LLM cluster diversity metric: first increases and then saturates or deteriorates with diversity.
Refer to caption: x5.png

In contrast, the proposed LLM cluster metric presents a more significant difference in the diversity of synthetic tokens, where the data with $100$K topics generally show less diversity compared to that of $300$K.
LLM cluster score also tends to increase first and then decrease as $\mathcal{G}$ increases, showing saturated or even deteriorated diversity.
This has not been observed in any of the baseline diversity metrics.
More interestingly, in the average benchmark results of both pre-trained and supervised fine-tuned models, as shown in Fig. 5, the performance highly aligns with our LLM cluster diversity measure.
Our results suggest that diversity, in terms of the number of topics $\mathcal{T}$ and the proper number of generations per topic $\mathcal{G}$, in synthetic data pre-training is essential for better performance.

### 3.4 Prompts for Synthetic Data Generation

In this part, we continue our study with different prompt templates for generating more diverse synthetic data.
As suggested in the creation of Cosmopedia-v0.1 (Allal et al., ) and Cosmopedia-v0.2 (Allal et al., ), the prompt template used for the generation of synthetic tokens is also very important for performance.
However, it is unclear on what dimension the diversity of synthetic data can better increase, and we try to conclude an answer from a set of controlled experiments.

Synthetic Data Generation.
To design prompts from different diversity dimensions, we start from the Topic prompt template used in Section 3.3.

**Table 3: Synthetic token counts of varying generation prompts.**
| Prompt | Cosmopedia<br>v0.1 | Cosmopedia<br>v0.2 | Topic | Topic Styles | Topic Styles<br>Persona | Multi-Topic<br>Styles Persona |
| --- | --- | --- | --- | --- | --- | --- |
| # Tokens (B) | 22.09 | 28.60 | 10.44 | 12.64 | 12.90 | 12.27 |

We first increase the dimension of styles of the synthetic text, including textbook narrative, textbook academic, blogpost, and wikihow, similar to Cosmopedia v0.1.
We term this prompt template as Topic Style.
Based on it, we further expand the targeted audience of the synthetic content.
In contrast to Cosmopedia, which adopted a limited number of audiences, we utilize the recent advance of personas for the creation of synthetic content (Chan et al., ).
For each generation, we randomly sample a set of personas and let GPT-4o to select the most appropriate one as the target audience for the generation.
This prompt is thus referred to as Topic Styles Persona.
Lastly, we further introduce multiple topic seeds in the prompt template, instead of just a single topic, and let GPT-4o select a combination of topics for content creation.
We term this prompt as Multi-Topic Styles Persona.
We use these four prompt variants to generate around 10-12B synthetic tokens utilizing the underlying $620$K topics, and pre-train models by up-weighting the synthetic tokens as in total 20B, similarly to Phi-series.
In addition, we also pre-train models on Cosmopedia v0.1 and Cosmopedia v0.2 as our large-scale synthetic data baselines, which are down-weighted to 20B for fair comparison.
The token statistics are shown in Table 3, and the details, examples, and outputs of the prompt template variants are shown in Appendix F.

Results.
We present the diversity measurement of the synthetic data generated by different prompt templates in Fig. 6.
We can observe that the baseline heuristic and model-based metrics demonstrate inconsistent diversity across datasets.
The benchmark results for the 350M and 1.4B models are shown in Fig. 7.
Noteworthy is that the performance of both pre-trained and supervised fine-tuned models well correlates with the LLM cluster score.
Interestingly, while Cosmopedia v0.2 has been shown to be generated using better-optimized prompts (Allal et al., ), its diversity is actually less than Cosmopedia v0.1, and the models pre-trained on Cosmppedia v0.2 thus present inferior performance.
Our Topic prompt template performs similarly to Cosmopedia v0.1 with more than 50% less of the actual synthetic tokens.
Other prompt template variants we used all demonstrate better diversity, and also superior performance compared to Cosmopedia baselines.
We also find that the prompt template Multi-Topic Styles Persona in fact generates less diverse synthetic tokens, compared to Topic Styles Persona.
This is possibly due to we provide multiple topics to GPT-4o and prompt it to combine topics flexibly, which may introduce more redundancy.
Our results suggest that adding personas (Chan et al., ) for synthetic data generation in pre-training can significantly increase the underlying diversity, and thus, in turn, boost the performance.

Figure: Figure 6: Diversity results of synthetic data generated by various prompt templates. (a) Average length of synthetic samples; (b) Self-repetition score; (c) Compression ratio; (d) N-gram diversity score; (e) Perplexity of GPT-2-L; (f) Perplexity gap between GPT-2-L and GPT-2-XL; (g) K-means cluster score of BERT-L embeddings; (g) LLM cluster score. The baseline metrics show inconsistent measures of diversity, whereas the proposed LLM cluster method well captures the diversity.
Refer to caption: x6.png

Figure: Figure 7: Benchmark results of pre-trained and supervised fine-tuned models by varying the prompt templates for synthetic data generation. Persona and Styles improves diversity and performance.
Refer to caption: x7.png

### 3.5 Models for Synthetic Data Generation

Synthetic Data Generation.
We study the diversity of synthetic tokens generated by different models in this part.
In previous sections, we default our synthetic generation model as GPT-4o.

**Table 4: Synthetic token counts of models.**
| Model | GPT-4o | GPT-3.5 | Llama-3.1 | Mistral |
| --- | --- | --- | --- | --- |
| # Tokens (B) | 5.00 | 4.62 | 4.04 | 4.39 |

Here, we compare the synthetic generation using GPT-3.5, and two open-source models: Llama-3.1-8B-Instruct (Dubey et al., ) and Mistral-7B-Instruct(^2^22While Cosmopedia (Allal et al., ; ) mainly used Mistral-8x7B-Instruct for synthetic data generation, we instead select smaller models here mainly due to the computational limit.) (Jiang et al., ).
From our previous results, we use the same Topic Styles Persona prompt template for the synthetic generation with different models.
Similarly to Section 3.3, we up-weight the generated synthetic tokens to 5B for pre-training, whose statistics are shown in Table 4.
We select 5B tokens from our corresponding GPT-4o generation in Section 3.4 as an additional comparison.
We also set an additional variant with mixed synthetic data from all models.
The output examples are shown in Appendix F.
Here, we only pre-train and supervised fine-tune 350M models and report the LLM cluster score measurement mainly due to the computational limits.

Figure: Figure 8: (a) LLM diversity score of synthetic data from different models. (b) Average performance of trained models.
Refer to caption: x8.png

Results.
We present both the results of the LLM cluster diversity and the model performance in Fig. 8.
One can observe that the synthetic data generated by more capable models usually present better diversity, i.e., GPT-4o over GPT-3.5 and Llama-3.1 over Mistral.
This trend is also reflected in the performance of both the pre-trained and supervised fine-tuned models.
Mixing up the synthetic data generated by different base LLMs can also slightly improve diversity, leading to better performance.
Our results suggest that the use of synthetic data from more advanced models and mixed models can be potentially beneficial in practice.

### 3.6 Ratio between Real and Synthetic Tokens

Figure: Figure 9: Results of varying real-syn ratio.
Refer to caption: x9.png

Here, we study the effect of the ratio between real and generated synthetic tokens.
We re-use the 12.9B synthetic data created by Topic Styles Persona prompt template.
We train 350M models by adjusting the sampling weight during training to make them effectively 1B, 5B, 10B, 20B, 34B, and 50B.
The results are shown in Fig. 9.
As we can observe, the accuracy generally improves as the proportion of synthetic tokens initially increases, i.e., from 1B to 20B.
However, when the ratio becomes skewed heavily toward synthetic tokens, i.e., over 34B, the average accuracy drops significantly, suggesting that the over-weighting of the synthetic data may introduce redundancy and thus hurt model performance.

### 3.7 Diversity, Token Size, and Model Size

Correlations between LLM Cluster Score and Model Performance.
We plot the linear regression of the LLM cluster score and model performance in Fig. 1, demonstrating a positive correlation between them.
As the LLM cluster score increases, indicating greater diversity in synthetic data, the average accuracy also improves consistently.
This trend is observed for both smaller models (350M) and larger models (1.4B), although the latter generally achieve higher accuracy, suggesting that more capable models benefit more from increased synthetic data diversity.

Larger Model Requires Larger Diversity.
One can also find that the 1.4B parameter models require and benefit from a higher level of diversity to fully leverage their capacity.
As the LLM cluster score increases, larger models show a more pronounced improvement in performance compared to smaller models.
Interestingly, while the pre-training performance of smaller models tends to saturate with larger diversity, the supervised fine-tuning performance can still benefit significantly.

### 3.8 Ablation Study of LLM Cluster Metric

Pipeline Parameters.
We conduct ablation experiments on both $K$ and $N$, and $J$ and $M$, with ablation results present in Section B.2 due to the space limit.
We show that the generation of metadata and metric is robust to the parameters $J$ and $M$.
The clustering performance decreases with very small and large $K$, and saturates as $N$ increases, presenting the scalability of proposed metric.

Pipeline Components.
We also conduct ablation on the components of the pipeline.
We compare the LLM cluster results using the entire pipeline, the pipeline without the verification component, and only the clustering component with manually defined metadata and metrics.
The results in Section B.2 demonstrate that metadata and metrics generation is essential to guarantee reasonable clustering performance, and the self-validation step can further boost the clustering performance.

Different LLMs.
We perform an additional ablation on the models used in the proposed LLM clustering pipeline, i.e., GPT-4, GPT-4o, GPT-3.5, and Llama-3.1.
From the results, we can observe that different LLMs often present consistent and robust clustering results using the proposed pipeline.

Figure: Figure 10: Density estimation of (a) number of samples per cluster $S$ and (b) number of clusters $C$ from LLM cluster results on synthetic data generated with Topic prompt using $\mathcal{T}=300$K, and $\mathcal{G}\sim\{10,20,30\}$. LLM Cluster-agent can discriminate the diversity of the underlying distributions.
Refer to caption: x10.png

Distribution of Clusters.
We plot the distribution of $\mathcal{C}$ and $\mathcal{S}$ of our LLM cluster score results in Section 3.3 with $\mathcal{T}=300$K, as shown in Fig. 10.
We can observe that, from the density of $\mathcal{C}$ and $\mathcal{S}$, LLM Cluster-agent can capture the nuanced diversity difference of the underlying distribution.

## 4 Related Work

Principled scaling (Kaplan et al., ) of language models both in terms of model and data size has resulted in powerful systems (Touvron et al., ; ; Jiang et al., ; Bai et al., ; Yang et al., ; AI et al., ; Team et al., ).
However, high-quality training data are still finite and expected to be consumed entirely in the near future (Villalobos et al., ).
To overcome this limitation, synthetic data generated from advanced LLMs are used for per-taining (Gunasekar et al., ; Ben Allal et al., ; Allal et al., ; Long et al., ), post-training, fine-tuning, or alignment (Wang et al., ; Taori et al., ; Wu et al., ).
In addition to scaling models and data sizes, the quality of pre-training data plays an equally critical role in determining the overall performance of language models (Sachdeva et al., ; Penedo et al., ). High-quality data, particularly when it exhibits diversity, is essential for achieving strong downstream task performance (Miranda et al., ; Tirumala et al., ; Chung et al., ). As a result, accurately measuring the quality of pre-training data has become a focus of research, since low-quality or noisy data can degrade model performance on downstream tasks (Penedo et al., ).
Several studies have explored the relationship between data quality and performance, demonstrating that improvements in data quality directly affect downstream results (Penedo et al., ).

Further, there exists a variety of strategies to carefully select high-quality data from large corpora while maintaining model performance. For example, (Sachdeva et al., ) show that even simple approaches, such as using large language models to filter and select data.
Other methods, including perplexity-based data selection and diversity-aware sampling techniques, have also proven effective in curating high-quality data from expansive datasets without sacrificing model performance(Ankner et al., ; Tirumala et al., ; Tan & Wang, ; Longpre et al., ).
Recent studies have focused on evaluating data quality using metrics such as perplexity, factuality, and alignment with human judgment to ensure that models are trained on meaningful and representative datasets (Shaib et al., ; Montahaei et al., ; Li et al., ). Among the many important characteristics of high-quality pre-training data, diversity stands out as a critical factor (Tirumala et al., ). Various methods have been developed to quantify diversity (Shaib et al., ), but these approaches have been applied mainly to natural data sources and present limitations, as we showed earlier.

## 5 Conclusion

In this study, we investigated the impact of synthetic data diversity on the performance of LLMs.
We proposed and validated a new metric, LLM Cluster-agent, to quantify the diversity of synthetic data.
Our experiments demonstrated that increased diversity correlates positively with model performance, particularly in downstream fine-tuning tasks.
Moreover, the choice of generation seeds, the prompt template, the generation model, and the ratio between real and synthetic tokens all significantly influence both the data diversity and model performance.
Although the scale of models in this study is mainly restricted up to 1.4B due to computational limits, we demonstrated that the results in this study present scalability and potential to be applied on a larger scale.
These results suggest that diverse, high-quality synthetic data is essential for the training of robust and effective LLMs, paving the way for future improvements in the generation and utilization of synthetic data.

## Appendix A Training Setup

In this section, we provide more details on our training setup.

### A.1 Pre-training Setup

For pre-training, we use AdamW optimizer with a linear-warmup-linear-decay learning rate schedule to pre-train the 350M and 1.4B models.
The maximum learning rate is set to 3$e$-4, betas of AdamW optimizer are set to 0.9 and 0.95, and the weight decay is set at 0.1.
We adopt a global batch size of 256 and 128 for 350M and 1.4B models respectively.
The 350M models are trained with 16 A100 and the 1.4B models are trained with 32 A100.
The 350M models are trained for in total 50B tokens, and 1.4B models are trained for 150B tokens.
We use fp16 and Zero-2 of DeepSpeed (Rasley et al., ) to speed up training.
The model configurations are shown in Table 5.

**Table 5: Configuration of 350M and 1.4B models.**
| Model Size | Vocab Size | Context Length | Hidden Size | Intermediate Size | # Layers | # Heads | Attn. Dropout |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 350M | 50340 | 2048 | 960 | 2560 | 28 | 15 | 0.1 |
| 1.4B | 50340 | 2048 | 2048 | 8192 | 16 | 32 | 0.1 |

## Appendix B Experiments Results

In this section, we present the detailed benchmark results.

### B.1 Main Results

The main experiments results are shown here.
We present the details results of Section 3.3 in Table 6, the detailed results of Section 3.4 in Table 7, the detailed results of Section 3.5 in Table 8, and the detailed results of Section 3.6 in LABEL:tab:appendix-results_ratio.
For ARC-challenge and HellaSwag, we report ’acc_norm’ from LM-Eval-Harness, and ’acc’ for other evaluated tasks.

**Table 6: Benchmark results of varying underlying distribution.**
| Model | $\mathcal{T}$ | $\mathcal{G}$ | Average | Common Sense | Language Understanding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ARC-C | ARC-E | BoolQ | SiQA | WinoGrande | PIQA | COPA | HellaSwag |  |  |  |  |
| 350M | 100 | 10 | 50.12 | 25.85 | 52.69 | 58.04 | 38.28 | 50.75 | 68.34 | 67.00 | 40.02 |
| 20 | 50.26 | 25.91 | 52.02 | 56.47 | 38.64 | 52.09 | 67.92 | 69.00 | 40.06 |  |  |
| 30 | 50.50 | 26.54 | 52.99 | 56.73 | 38.84 | 53.12 | 68.01 | 68.00 | 39.78 |  |  |
| 300 | 10 | 50.65 | 27.30 | 51.85 | 58.93 | 38.54 | 51.30 | 68.44 | 69.00 | 39.85 |  |
| 20 | 51.28 | 27.30 | 51.85 | 58.93 | 39.54 | 52.30 | 68.44 | 72.00 | 39.85 |  |  |
| 30 | 51.05 | 26.54 | 52.86 | 59.57 | 39.43 | 53.17 | 67.68 | 69.00 | 40.12 |  |  |
| 350M<br>SFT | 100 | 10 | 51.43 | 28.33 | 53.93 | 59.78 | 39.10 | 52.09 | 69.81 | 67.00 | 41.41 |
| 20 | 51.83 | 28.88 | 53.91 | 60.55 | 39.51 | 52.01 | 70.00 | 68.00 | 41.80 |  |  |
| 30 | 51.96 | 28.67 | 54.18 | 60.44 | 40.69 | 52.38 | 69.46 | 68.00 | 41.83 |  |  |
| 300 | 10 | 52.38 | 29.16 | 54.28 | 60.04 | 39.30 | 51.85 | 69.23 | 71.00 | 42.19 |  |
| 20 | 53.04 | 29.65 | 54.65 | 60.55 | 39.95 | 52.41 | 70.25 | 74.00 | 42.82 |  |  |
| 30 | 52.62 | 29.07 | 54.77 | 60.09 | 39.76 | 53.72 | 69.27 | 72.00 | 42.29 |  |  |
| 1B | 100 | 10 | 54.86 | 28.24 | 62.29 | 57.41 | 41.74 | 58.88 | 73.67 | 73.00 | 43.66 |
| 20 | 55.02 | 28.75 | 62.79 | 59.63 | 42.15 | 57.59 | 73.18 | 72.00 | 44.09 |  |  |
| 30 | 55.06 | 28.90 | 61.57 | 59.98 | 42.81 | 57.62 | 74.05 | 72.00 | 43.56 |  |  |
| 300 | 10 | 55.30 | 29.52 | 62.12 | 58.54 | 40.70 | 56.27 | 73.29 | 78.00 | 43.95 |  |
| 20 | 55.81 | 30.20 | 63.22 | 59.79 | 41.94 | 59.59 | 73.83 | 73.00 | 44.91 |  |  |
| 30 | 55.24 | 29.75 | 62.35 | 58.87 | 41.30 | 58.41 | 74.43 | 72.00 | 44.84 |  |  |
| 1B<br>SFT | 100 | 10 | 57.57 | 31.63 | 63.68 | 58.56 | 42.10 | 59.38 | 74.14 | 73.00 | 58.08 |
| 20 | 58.19 | 31.31 | 64.09 | 58.87 | 42.50 | 59.33 | 74.65 | 76.00 | 58.76 |  |  |
| 30 | 58.20 | 32.25 | 63.90 | 59.04 | 42.40 | 59.75 | 74.93 | 75.00 | 58.33 |  |  |
| 300 | 10 | 58.03 | 32.57 | 64.31 | 59.99 | 41.15 | 59.35 | 73.89 | 75.00 | 58.01 |  |
| 20 | 58.65 | 34.00 | 65.32 | 60.75 | 42.48 | 59.20 | 74.73 | 74.00 | 58.68 |  |  |
| 30 | 58.16 | 33.62 | 64.95 | 60.81 | 41.04 | 59.01 | 74.09 | 73.00 | 58.76 |  |  |

**Table 7: Benchmark results of varying prompt templates.**
| Model | Data | Average | Common Sense | Language Understanding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ARC-C | ARC-E | BoolQ | SiQA | WinoGrande | PIQA | COPA | HellaSwag |  |  |  |
| 350M | Real Only | 48.94 | 24.40 | 48.78 | 58.96 | 38.59 | 52.09 | 66.81 | 66.00 | 35.88 |
| Cosmepedia v0.1 | 51.61 | 27.68 | 53.90 | 59.98 | 39.10 | 53.12 | 69.57 | 68.00 | 41.49 |  |
| Cosmepedia v0.2 | 51.59 | 28.69 | 54.98 | 59.46 | 38.12 | 51.80 | 68.75 | 70.00 | 40.89 |  |
| Topic | 51.40 | 28.05 | 54.29 | 60.20 | 38.41 | 53.51 | 67.85 | 68.00 | 40.92 |  |
| Topic Styles | 51.81 | 28.41 | 56.02 | 60.04 | 39.25 | 53.41 | 68.17 | 68.00 | 41.17 |  |
| Topic Styles Persona | 51.92 | 28.90 | 55.60 | 60.36 | 39.38 | 53.54 | 69.36 | 67.00 | 41.24 |  |
| Multi-Topic Styles Persona | 51.74 | 27.90 | 53.87 | 60.17 | 39.46 | 53.04 | 68.87 | 70.00 | 40.59 |  |
| 350M<br>SFT | Real Only | 50.00 | 27.05 | 52.86 | 58.31 | 39.20 | 51.46 | 66.00 | 67.00 | 38.10 |
| Cosmepedia v0.1 | 52.64 | 29.56 | 55.80 | 60.28 | 40.97 | 51.80 | 70.57 | 69.00 | 43.41 |  |
| Cosmepedia v0.2 | 53.29 | 30.78 | 55.23 | 60.26 | 41.66 | 53.35 | 69.75 | 71.00 | 44.28 |  |
| Topic | 53.03 | 29.33 | 55.98 | 60.34 | 40.23 | 52.96 | 70.85 | 70.00 | 44.58 |  |
| Topic Styles | 53.37 | 30.12 | 56.03 | 60.74 | 40.51 | 53.07 | 71.17 | 70.00 | 45.32 |  |
| Topic Styles Persona | 54.29 | 31.82 | 56.84 | 60.86 | 41.15 | 53.70 | 71.36 | 72.00 | 46.60 |  |
| Multi-Topic Styles Persona | 54.06 | 31.82 | 56.98 | 60.07 | 41.49 | 52.22 | 70.87 | 73.00 | 46.00 |  |
| 1B | Real Only | 54.76 | 28.07 | 62.08 | 57.98 | 42.58 | 58.80 | 73.45 | 71.00 | 44.08 |
| Cosmepedia v0.1 | 56.25 | 29.78 | 64.84 | 58.75 | 42.99 | 59.35 | 73.61 | 75.00 | 45.71 |  |
| Cosmepedia v0.2 | 55.84 | 32.08 | 66.37 | 54.81 | 43.60 | 59.04 | 73.67 | 71.00 | 46.16 |  |
| Topic | 56.15 | 30.12 | 66.04 | 60.92 | 42.58 | 58.93 | 73.88 | 71.00 | 45.73 |  |
| Topic Styles | 56.74 | 31.83 | 66.62 | 59.85 | 43.97 | 58.64 | 73.01 | 74.00 | 45.96 |  |
| Topic Styles Persona | 57.82 | 32.46 | 67.20 | 62.65 | 44.51 | 59.97 | 73.98 | 74.00 | 47.80 |  |
| Multi-Topic Styles Persona | 56.99 | 32.44 | 66.81 | 61.42 | 43.41 | 58.74 | 73.49 | 72.00 | 47.64 |  |
| 1B<br>SFT | Real Only | 57.16 | 31.31 | 58.75 | 58.96 | 43.07 | 59.43 | 74.06 | 73.00 | 58.08 |
| Cosmepedia v0.1 | 59.46 | 34.79 | 65.42 | 62.13 | 42.12 | 59.51 | 75.47 | 77.00 | 59.25 |  |
| Cosmepedia v0.2 | 59.46 | 34.45 | 66.18 | 63.31 | 43.71 | 59.20 | 75.60 | 72.00 | 61.21 |  |
| Topic | 59.88 | 34.94 | 66.96 | 64.61 | 43.12 | 59.35 | 74.97 | 74.00 | 61.11 |  |
| Topic Styles | 60.97 | 35.57 | 67.69 | 65.08 | 43.58 | 59.57 | 75.57 | 78.00 | 62.57 |  |
| Topic Styles Persona | 61.32 | 35.78 | 68.04 | 65.19 | 44.10 | 60.39 | 76.17 | 78.00 | 62.57 |  |
| Multi-Topic Styles Persona | 60.59 | 34.36 | 67.93 | 64.79 | 43.11 | 60.01 | 75.39 | 76.00 | 63.03 |  |

**Table 8: Benchmark results of varying synthetic data generation models.**
| Model | Gen Model | Average | Common Sense | Language Understanding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ARC-R | ARC-E | BoolQ | SiQA | WinoGrande | PIQA | COPA | HellaSwag |  |  |  |
| 350M | Llama-3.1-8B-Instruct | 51.22 | 26.37 | 54.54 | 58.17 | 39.10 | 52.88 | 68.39 | 70.00 | 40.34 |
| Mistral-7B-Instruct | 50.86 | 26.02 | 54.36 | 58.31 | 39.20 | 51.99 | 67.95 | 69.00 | 40.03 |  |
| GPT-3.5 | 51.23 | 26.87 | 53.99 | 59.23 | 38.67 | 52.72 | 68.22 | 70.00 | 40.17 |  |
| GPT-4o | 51.61 | 27.13 | 54.53 | 59.65 | 38.71 | 52.93 | 68.45 | 71.00 | 40.51 |  |
| Mixed | 51.72 | 26.88 | 54.38 | 59.47 | 39.33 | 52.99 | 68.79 | 71.00 | 40.88 |  |
| 350M<br>SFT | Llama-3.1-8B-Instruct | 52.32 | 29.65 | 55.51 | 60.52 | 39.71 | 52.17 | 68.74 | 70.00 | 42.25 |
| Mistral-7B-Instruct | 52.17 | 28.79 | 55.60 | 60.43 | 39.61 | 51.62 | 68.32 | 71.00 | 42.00 |  |
| GPT-3.5 | 52.36 | 29.13 | 55.84 | 60.19 | 39.88 | 52.09 | 69.89 | 69.00 | 42.83 |  |
| GPT-4o | 52.85 | 29.75 | 56.16 | 60.72 | 39.97 | 52.22 | 70.05 | 71.00 | 42.95 |  |
| Mixed | 53.02 | 29.47 | 57.05 | 60.40 | 39.15 | 52.63 | 70.81 | 71.00 | 43.62 |  |

### B.2 Ablation Results

Here, we provide all of our ablation results on the proposed LLM Cluster-agent.

The ablation on $J$ and $M$ are shown in Table 9.
We show that $J=5$ and $M=100$, and with larger values of these two parameters, produce quite consistent top metadata and metrics that will be used for clustering criteria.

**Table 9: Ablation of $J$ and $M$ on top-3 metadata and metrics.**
| J | M | Top3 Metadata | Top3 Metric |
| --- | --- | --- | --- |
| 5 | 10 | Analysis Technique, Industry Relevance | Clarity of Explanation, Jargon Usage, Technicality Level |
| 5 | 50 | Temporal Relevance, Technical Concept Depth, Terminology Density | Semantic Coherence, Technical Language Density, Contextual Depth |
| 5 | 100 | Subject Domain, Conceptual Density, Narrative Structure | Terminology Density, Interdisciplinary Index, Practical Impact Factor |
| 5 | 500 | Disciplinary Focus, Conceptual Density, Interdisciplinary Relevance | Interdisciplinary Integration, Conceptual Density, Lexical Diversity |
| 3 | 100 | Domain Specificity, Conceptual Complexity, Semantic Complexity | Novelty Score, Practical Impact Factor, Conceptual Clarity |
| 10 | 100 | Disciplinary Focus, Conceptual Density, Terminology Density | Interdisciplinary Integration, Information Density, Lexical Diversity |
| 15 | 100 | Disciplinary Focus, Text Complexity, Narrative Style | Interdisciplinary Integration, Conceptual Density, Lexical Diversity |

The ablation of clustering score results about parameters $K$ and $N$ are shown in Table 10(a) and Table 10(b), pipeline components are shown in Table 10(c), and generation models are shown in Table 10(d).
One can observe that $K=10$ produce the most robust clustering results, where smaller and larger $K$ present larger variations in results.
We also show that with sufficient large $N$ as 5K or 10K, the clustering results becomes stable.
For the components, we find that both the metadata and metric generation and self-verification step is essential to achieve reasonable clustering performance.
We also demonstrate that the proposed metric is robust to the generation models.

**Table 10: Ablation study of the proposed LLM cluster metric.**
| $K$ | Score |
| --- | --- |
| 5 | 5.12$\pm$0.14 |
| 10 | 3.99$\pm$0.05 |
| 15 | 3.48$\pm$0.29 |
| 20 | 3.13$\pm$0.46 |

## Appendix C Diversity Metrics

Context Length refers to the average length of the sequences in the dataset. Longer contexts can indicate more complex data structures and richer narratives. By analyzing context length, we can infer the ability of the synthetic data to capture long-term dependencies and intricate patterns.

Self-repetition Score quantifies how often sequences or phrases are repeated within the dataset. Lower scores suggest higher diversity, as the model generates more varied outputs rather than reiterating the same phrases. High self-repetition can indicate overfitting or a lack of creativity in the synthetic generation process.

N-gram Diversity Score measures the variability of contiguous sequences of ’n’ items in the dataset. By examining different ’n’ values (e.g., unigrams, bigrams, trigrams), this score highlights how varied the generated text is at multiple granularities. A higher N-gram diversity score indicates more creative and less predictable outputs, which is often desirable in synthetic data generation.

Compression Ratio assesses the dataset’s redundancy by compressing it and comparing the compressed size to the original size. A lower compression ratio suggests that the data is less repetitive and more diverse. This metric provides a quantitative way to gauge the amount of unique information within the dataset.

Perplexity is a measure of how well a probability model predicts a sample. In the context of synthetic data, lower perplexity indicates that the model can predict the data more confidently, which may imply less diversity if the model is overconfident. Higher perplexity, conversely, can indicate that the model encounters more unexpected or varied data, pointing towards greater diversity.

Perplexity Gap measures the difference in perplexity between GPT-2-L and GPT-2-XL (Radford et al., ), used to assess dataset diversity. A smaller gap indicates less diversity, while a larger gap reflects greater variability and complexity in the data.

K-means Clustering is used to partition the dataset into distinct groups based on feature similarity. By analyzing the number and distribution of clusters, we can gain insights into the inherent diversity of the data. However, traditional clustering methods like K-means may struggle with high-dimensional, complex data structures, often oversimplifying the richness of the data.

## Appendix D LLM Clustering

In this section, we provide detailed prompt templates, prompt examples, and output examples of the proposed LLM Cluster-agent metric.
The prompt templates we used include metadata and metric generation, metadata and metric summary, high-level criteria definition summary, clustering, and self-verification.

### D.1 Prompts Templates in Pipeline

### D.2 Examples of Prompting Outputs in Pipeline

## Appendix E Seeding Topics of Synthetic Generation

We provide more details on the seeding topics that we used for synthetic data generation.

### E.1 Examples of Topic Seeds

### E.2 Visualization of the Topic Seeds

Figure: Figure 11: Visualization on the clustering of topic seeds.
Refer to caption: x11.png

Figure: Figure 12: Distribution of top-20 topics at each hierarchical level.
Refer to caption: x12.png

## Appendix F Synthetic Data Generation

### F.1 Generation Prompt Templates

#### F.1.1 Topic

#### F.1.2 Topic Styles

#### F.1.3 Topic Styles Persona

#### F.1.4 Multi-Topic Styles Persona

### F.2 GPT-4o Generation Output Examples

#### F.2.1 Topic

#### F.2.2 Topic Styles

#### F.2.3 Topic Styles Persona

#### F.2.4 Multi-Topic Styles Persona

### F.3 GPT-3.5 Generation Output Examples

### F.4 Llama-3.1-Instruct-8B Generation Output Examples

### F.5 Mistral-Instruct-7B Generation Output Examples

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 8
presentation: 3
questions: 1. How do you decide on the optimal number of clusters? Is there a cluster evaluation technique (similar to elbow curves, cluster homogeneity metrics for traditional clustering techniques) in place?
2. Can we have a human eval study included to address the metric reliability issues?
soundness: 3
strengths: 1. Elaborate comparisons have been done with other heuristic and model-based metrics.
2. Multiple open-sourced and proprietary models have been use to generate synthetic data and compare how diversity metric compares across different model architectures and sizes. 
3. Cluster generation and verification step is a much needed verification step for agentic pipelines and that has been addressed appropriately.
4. Conducting large scale study to systematically investigate the effectiveness of a diverse synthetic dataset in pre-training as well as SFT processes is a huge positive that strengthens the underlying hypothesis.
summary: The paper aims to address an important gap in LLM research - impact of a diverse dataset on LLM performance on task generalization. More so, it attempts to quantify "diversity" by introducing a clustering based metric for it and proposes an agentic pipeline for the same. Elaborate comparisons have been done with other heuristic and model-based metrics. Multiple open-sourced and proprietary models have been use to generate synthetic data and compare how diversity metric compares across different model architectures and sizes.
weaknesses: 1. Lack of a rigorous human evaluation study : Agentic pipelines generally need to be heavily intertwined with a human-in-the-loop aspect during initial experimentation. I am a bit unclear on how to test the reliability and quality of the metadata and metrics generation step in the pipeline. The error in this initial step would be propagated to later steps and would potentially make the metric unreliable.
2. Assessing cluster purity - an elaborate statistical analysis on the clusters formed is missing, how do we trust the final clusters formed and how do we assess cluster purity?
3. Lack of control on the number of clusters formed - there needs to be a systematic prompt injection to define the optimal number of clusters to be formed (and a process to find the optimal number of clusters in the first place). I understand that there has been ablation studies on the number of clusters, but how do I pick this optimal number which might also vary across different styles and topics)
4. Parameter Sensitivity: The performance of the clustering may depend on the choice of parameters like K (number of samples for clustering), N (number of clustering iterations), J and M (for metadata/metric generation). Optimal parameter settings may need to be determined.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Main question would be whether the authors can show some quantitive correlation metrics or diagrams between LLM cluster score and other baseline metrics and downstream performance to show that LLM cluster score is a better indicator for performance than other baseline metrics. 

In Figure 5, is the model trained on only real data trained on more real data to match the number of synthetic tokens? Judging from the results, it seems like you are not. Would be interesting to quantify the efficacy of training with synthetic data. Similarly in Figure 9, it seems like the number of tokens that the model is train on is increasing as you increase the synthetic to real data ratio. This diagram would be more convincing if it kept the amount of data/tokens used for training static.

Also am interested in the cost of evaluating these datasets using LLM Cluster score. Previous approaches used either no LLMs or open source LLMs like GPT-2. However, with this metric, all of these results are done using GPT-4o. Is this technique viable for researchers trying to quantify the diversity of their synthetic datasets? 

LLM Cluster-agent pipeline seems to have some similarity with the field of explainable/unsupervised clustering of text data, such as 
```Wang, Zihan, Jingbo Shang, and Ruiqi Zhong. "Goal-Driven Explainable Clustering via Language Descriptions." The 2023 Conference on Empirical Methods in Natural Language Processing``` and other related work, which should be cited in the related work.

There are also some typos in the manuscript. Other parts are a bit vague and unclear:
* Line 098 has a typo "per-training". 
* On line 147, it is unclear what "due to similar patterns in part-of-speech tagging and syntactic often present in them"? Is this supposed to be "syntax"?
* In section 2.1, it is unclear how capturing the underlying distribution of clusters and cluster sizes implies that this measure originates from the principle of entropy. What does this mean?
* Seems to be an extra space on line 309 and line 422.
* Extra "diversity" in the description of Figure 4, near line 289/290.
* Is LLM Cluster-agent the name of the metric itself? Or is it the pipeline that you have developed for generating clusters? In the Abstract, you state that LLM Cluster-agent is "a new diversity metric", but in Section 2.1, you state that it is "pipeline that leverages LLM's abilities to interpret semantic meanings", and then later, on line 199-200, you state that "LLM Cluster score" is the actual diversity metric. There also is inconsistent capitalization of this name throughout the manuscript.
* Line 409: Cosmopedia is spelled "Cosmppedia"
* On line 464-465 in Section 3.7, the paper states that larger models "generally achieve higher accuracy, suggesting that more capable models benefit more from increased synthetic data diversity". However, the models achieving higher accuracy can be attributed just to larger model size. I think that the authors meant to say that the performance of the larger models are more correlated with diversity, not that they achieve higher accuracy. Furthermore, as stated before, this statement would be more convincing and concrete if the authors provided quantitative metric for this correlation.
* In Section 2.1, line 197, the paper states that the verification step is important in "removing some unreasonable clusters." What does this mean? What are examples of unreasonable clusters?

Nit: Would add a citation for the footnote on page 3 regarding the degradation of performance for long-context situations, maybe Lost in the middle (Liu 2023) or something similar.
soundness: 3
strengths: Content of the paper is fairly complete and extensive. Explanation of LLM Cluster-agent is clear and concise. Lots of details provided in the main manuscript and in the appendix to show exactly what they are doing. Ablations are elaborate and extensive. Paper gives convincing evidence that diversity in synthetic data generation can be controlled and is correlated with downstream performance. Generated metric may be a strong indicator for the efficacy of a synthetic data generation pipeline for pretraining.
summary: This paper runs a study on the effect of diversity for synthetic data used to pretrain LLMs. To accomplish this, they develop a llm-based diversity metric called LLM Cluster-agent that measures the diversity of datasets. This pipeline is an iterative clustering mechanism that uses a metadata and metrics generation prompt to extract features used to generate clusters. A clustering prompt is used to prompt an LLM to group randomly selected samples from the corpus into clusters. From here, the LLM Cluster score is used to measure the diversity of the synthetic data based on the ratio between the number of clusters with the number of samples.

Using this metric, the authors pretrain multiple 350M and 1.4B parameter llama models on a combination of real and synthetic data to measure the performance of these models on downstream tasks as a function of the diversity of the synthetic data, and find that model performance correlates with LLM Cluster score. Furthermore, authors run extensive ablations on different synthetic generation prompts.
weaknesses: The work does not sufficiently show that LLM cluster score is more correlated with performance than baseline metrics. Figure 1 is the main result showing that LLM cluster score correlates with downstream performance. However, it is difficult to see if other baseline metrics provided in the paper would not also be correlated with performance. Argument would be more convincing if there were also diagrams or concrete correlation metrics of the other baseline metrics not aligning with performance in a similar diagram. The only evidence given that performance is more correlated is through visual inspection of Figure 4 and 5 in Section 3.3, and through visual inspection of Figure 6 and 7. Some of these correlations seem marginal at best on inspection. For instance, LLM Cluster score for Cosmopedia v0.2 performs the worse, but its performance is about on par with v0.1 and Topics in Figure 7a, 7c, and 7d.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: The paper is well written but I still have a lot a of questions. It would be great if authors can answer them

1. Sampling Parameters and Methodology (This is my biggest concern):
- How are the parameters K (samples for clustering) and J (samples for metadata generation) selected?
- For the M repetitions in metadata generation, are different J samples used each time?
- What is the justification for these parameter choices?

2. Self-Verification Process:
- How is the effectiveness of the self-verification step evaluated?
- Does the verification use the same LLM model or a different one?
- How do different values of M and K affect the consistency of clustering results?

3. Generalizability:
- Has this diversity measurement approach been tested on different types of datasets beyond the described text data (e.g., mathematical content)? If not can you do some additional experiments beyond mathematical domain like finance, biomedical or logical reasoning to show more generalizability of the proposed metric. 

4. Comparative Analysis:
- How does the LLM Cluster-agent method compare quantitatively to existing diversity measurement approaches?
- What specific advantages does it offer over traditional methods? A detailed comparison would be more helpful
soundness: 3
strengths: 1. The paper is well written and clearly organized
2. The authors discuss an important issue of understanding diversity metric.
summary: The main idea presented in the document is studying the impact of diversity in synthetic data on the performance of large language models (LLMs) during pre-training and fine-tuning. They propose a new metric called "LLM Cluster-agent" to quantify the diversity of large-scale synthetic text datasets. This metric uses an LLM to cluster text samples based on generated metadata and scoring criteria.They generate synthetic datasets with varying levels of diversity by controlling factors like:
    - The underlying distribution of topics and generations per topic
    - Prompts/templates used for synthetic data generation (e.g. adding styles, personas)
    - Base LLM models used for generation (GPT-4, GPT-3.5, open-source models)
    - Ratio of real vs. synthetic data used for pre-training

They pre-train 350M and 1.4B parameter LLMs on combinations of real data and the synthetic datasets of varying diversity levels. They evaluate the pre-trained and also supervised fine-tuned LLMs on several benchmarks. Their experiments show that the proposed LLM Cluster-agent diversity metric correlates positively with the performance of both pre-trained and fine-tuned LLMs. They find that larger diversity in synthetic data benefits model performance, especially for fine-tuning. But too much synthetic data can deteriorate performance. They ablate the design choices of their LLM Cluster-agent pipeline.
weaknesses: 1. Context Length Limitation:
- The paper criticizes previous methods for context length limitations (lines 192-195), but the proposed method also faces similar constraints
- How does this represent an improvement over existing algorithms?
- Current LLMs have a maximum context length they can process. For extremely long documents, the clustering may need to be done in segments. What specific solutions does the method offer to address the context length challenge?
- Can you please clarify how your method specifically addresses or improves upon the context length limitations of previous approaches. Please provide a more detailed comparison of how your method handles long documents compared to existing techniques.

3. The authors state that their findings "present scalability and potential to be applied on a larger scale." However, their largest experiment is limited to 1.4B parameter models due to computational constraints. Extrapolating their diversity metric's performance and conclusions to much larger models (e.g., hundreds of billions of parameters) may be an overreach without empirical evidence at those scales. Please discuss any theoretical reasons or preliminary evidence that suggests their findings would scale to much larger models, or to explicitly acknowledge the limitations of extrapolating beyond the scales tested.

4. Belief in prompted clustering quality: Their LLM Cluster-agent metric relies heavily on the quality of prompted clustering done by LLMs. While LLMs can be very capable, assuming that their clustering perfectly captures true data diversity based on generated criteria is a strong belief. The metric's performance could degrade for highly complex data distributions.

5. Diversity dimensionality: Their metric provides a single score capturing overall diversity. However, diversity is a multi-dimensional construct. Existing metrics focus on different facets that may still be relevant.

6  Self-Verification Process Design:
- Why wasn't a more capable "teacher" model used for the self-verification step?
- Would using a more advanced model for verification improve the reliability of the results?
- What was the rationale behind using the same model for both clustering and verification?

7. This point is just good to have. I am not reducing scores for this: The evaluation is based solely on automatic metrics. Incorporating human evaluation of the generated synthetic data and model outputs could provide a more holistic understanding of the diversity and quality aspects.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Lines 185-187: across all M iterations, what is the nature of the sampled J instances, are these samples exclusive of one another? Or is the sampling entirely random and from the whole distribution, which means that there may be common elements  in different J-sized samples across the M iterations?
- The work also states that the clustering is performed N times. Does the K-sized sample change over each iteration out of N? If so, are the K-sized samples exclusive of one another, or are they randomly samples each time from the entire dataset, thus allowing for common elements?
- Instead of clustering only K samples, which may not yield a diversity score representative of the entire dataset, it may be more fruitful to cluster all elements in the dataset, taking K samples at a time, and following a iterative approach akin to K-Means to merge existing clusters and obtain a more comprehensive idea of the clusters produced by the LLM Cluster-agent over the dataset. 
- (Lines 186-187) Documenting the intuition behind using a metadata and metric gathering prompt (that summarizes the most frequent subset of metadata and metrics) instead of scoring the samples individually over every metadata and metric generated, would help convey the rationale of the approach better.
soundness: 2
strengths: - This work proposes a novel, model-based metric
- The methodology allows for limitations of model context lengths
- Several insightful analyses surrounding accuracy, diversity, and token balance are presented.
summary: This work proposes a framework to understand the efficacy of synthetic data, with a focus on quantifying the extent and effect of diversity in synthetically-generated data. To this end, a novel model-based metric, LLM Cluster score, is proposed. Based on a subsample of the data, attributes are identified by an LLM that can uniquely characterize individual samples and cluster similar ones. A random sample is taken and the data points within it are clustered. Based on the results of these clustering iterations, the diversity metric is calculated. This diversity metric is shown to be positively correlated with model accuracy metrics, demonstrating that [1] this metric is a good measure of diversity, and [2] more diverse data results in better model performance.
weaknesses: - The number of samples is a cause for concern.  (J=5, K=10) 

-- J=5 : does this imply that the model uses 5 samples to infer the metadata and metrics that characterize the entire distribution? Given the size of the data and the number of topics that may be present (100K/300K), it merits more discussion on whether J samples can represent the data sufficiently.

-- K=10 : Having obtained metadata and metrics, as well as a prompt that grades samples according to the rubric defined, lines 192-194 suggest that only K samples out of the dataset are then assigned to clusters by the LLM. Can a cluster score over 10 samples definitely yield an idea of the diversity of the dataset? This could benefit from more supporting evidence. 

In light of this, conducting sensitivity analyses, and extending the ablation studies in B.2 to higher values of K and a range of values for J would mitigate this concern. Furthermore, including other clustering metrics in the ablation study beyond the novel LLM cluster metric would help corroborate these values more strongly.

- The self-verification module could benefit from clarification along the following lines: 

-- Was there any human evaluation performed in order to verify the outcomes of the self-verification module, and in order to find the optimal verification prompt?

--  Using the same model to verify its own reasoning chain and clustering judgements may lead to a positive bias; the verification module would benefit from utilizing an LLM that is not part of the earlier steps of cluster score generation pipeline.


- There is a lack of clarity on the difference between the "metadata" and "metric" axes. How are these factors differentiated? Based on the sample outputs in Appendix D.2, there is some overlap between the nature of the "metadata" and "metric" keys, in terms of quantifiability and subjectivity. This lack of distinction is made stronger by the prompt templates in Appendix D.1, where the templates for metrics and metadata generation are nearly identical. It would be helpful to delineate [1] how these terms are different in theory, [2] in practice, how the outcomes of their respective prompts differ, and [3] whether this difference is essential to maintain, as opposed to a single prompt that generates factors that could be either "metrics" or "metadata".

- Additionally, some of the "metadata" and "metrics" are ill-defined, with only qualitative judgements associated with them, and that too only for the values on the extreme ends of the 1-5 scoring scale (See Appendix D.2) . This raises a concern: LLMs often show a lack of consistency in their outputs, even for the same input. This is an even greater concern when LLMs are used to confer subjective evaluations on the input. The diversity metric proposed in this work may be non-reproducible, especially over different  samples of the same dataset. The inconsistency could be be mitigated by including a more detailed, unambiguous rubric. The results would also be strengthened by running multiple trials with various seeds for random sampling the data.

- Figures 5 and 6 show that the inter-cluster variance achieved by using LLM Cluster score is higher than that achieved by other methods, thus demonstrating its efficacy in segmenting a dataset with 100K-300K topics. However, owing to the non-uniform scaling of the X-axis and the non-comparable nature of the range of values ([24,24.2] for Perplexity, [192,100] for K-Means, [3.5,5] for LLM Cluster score), it is not possible to definitively determine that a higher cluster score is more correlated with diversity than a higher measure of any other diversity metric. Contextualizing the comparative performance of these scores and their usefulness in 3.2 onwards, along with presenting standardized results in the visualizations would help better understand the benefits of LLM cluster score over other metrics.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: ### Questions:
1. How is GPT-40 prompted to extract a hierarchy of topics and a set of keywords covered in the page's content? 
2. In the Synthetic Data Generations section, it is unclear how the downweighting of Cosmopediav0.1 and Cosmopediav0.2 was done.
3. There is a brief explanation, but it is still unclear why `topic-style persona` almost always outperforms `multi-topic-styles persona`. Intuitively, the latter is supposed to provide more diversity. 
3. What would be the average price for carrying out such an experiment due to the presence of multiple proprietary LLM calls? 
4. From the ablation experiments, it looks like when K = 10 and N =10K, the diversity score stabilizes. Do the authors report all their experiments with this configuration?
5. Would be interested to know the comparison between the following method say KMeansDiversityScore, where we identify $C_i$ clusters using K-means of embeddings of samples and get cluster size $S_i$ from $K$ randomly selected samples. Repeat this experiment $N$ times and then compute the diversity score using the same formula. This experiment can provide a good comparison if the metric and metadata generation followed by the metadata gathering step has a positive effect in computing good diversity scores.

### Suggestions:
1. Typo at line 093: s\per-training\pre-training
2. Missing citation `??` at line 1055
soundness: 3
strengths: 1. The authors propose a novel approach LLM Cluster-agent to quantifying the diversity of text datasets, specifically tailored for synthetic data, which is often more challenging to evaluate.
2. The authors carry out systematic controlled experiments across various factors, such as topics, prompt styles, generation models, and real-to-synthetic token ratios, providing insights into optimal practices for synthetic data generation.
3. The findings offer practical guidance on balancing real and synthetic data and selecting synthetic generation models to enhance performance, which could be valuable for researchers and developers of LLMs.
4. From the ablation experiments, the clustering performance drops with low K and high N values, indicating potential scalability and applications to compute diversity in real-world scenarios.
summary: The paper investigates the importance of diversity in synthetic data used for training large language models (LLMs). This study introduces a novel metric called LLM Cluster-agent, an LLM-based agent to evaluate the diversity of synthetic datasets. The authors analyze how diversity affects model performance during both the pre-training and fine-tuning phases. Through experiments involving models of 350M and 1.4B parameters, the authors demonstrate that greater diversity in synthetic data generally correlates positively with improved performance, especially during fine-tuning. They also explore various methods for generating synthetic data using various `Topic-*` prompt templates to increase the diversity of the synthetic data generated. Finally, they also study the dynamics of synthetic tokens in pertaining corpora and the model performance metrics. The performance improvement also correlates with the value of the diversity score reported by the LLM agent.
weaknesses: 1. The primary drawback I observe is the separation between the generation and diversity measurement processes. According to the pipeline, diversity is only measured after data generation is completed. For example, if 4.5B tokens are generated and the resulting LLM diversity score is low, this would make the generation effort ineffective. Did the authors consider iterative diversity checks during the generation process. For example, implement intermediate diversity assessments at smaller generation intervals (e.g., every 100M tokens). This would allow to adjust or halt generation if diversity metrics fall below a desired threshold, optimizing the process early on and reducing waste in cases of low diversity outcomes.
2. Another significant limitation of this approach is its heavy reliance on large pre-trained LLMs to evaluate various criteria. How do the authors guarantee the stability of the metrics and metadata generated? Was any human evaluation performed to verify the overall reliability of the pipeline? The results could be more reliable if the authors conduct an inter-rater reliability tests by comparing diversity metrics generated by multiple LLMs or align LLM-generated metrics with human evaluations. A smaller human-annotated sample can be used to validate the model-generated diversity scores as this would enhance the overall reliability of the pipeline.
3. In the cluster generation step, the clustering process is localized to the number of samples that can be fit in the context window of the LLM. This can lead to random and unpredictable cluster formations leading to unpredictable diversity scores. Did the authors try any approaches that can mitigate the issue?

Meta-review: The paper investigates the impact of diversity in synthetic datasets on the performance of large language models (LLMs) during pre-training and fine-tuning. It introduces a metric, LLM Cluster-agent, to quantify dataset diversity. This metric employs LLMs to generate metadata and cluster samples iteratively, producing a diversity score based on the ratio of clusters to samples. The work claims following contributions: 1) a new metric, LLM Cluster-agent, to measure dataset diversity; 2) empirically showed the efficacy of proposed metric by demonstrating strong correlation between proposed metric and model performance. Experiments are conducted on models with 350M and 1.4B parameters and evaluated on downstream tasks. Authors also explored various prompt templates, styles, personas, and base LLM architectures (e.g., GPT-4, GPT-3.5) to enhance synthetic dataset diversity.


Strength of this paper
- The paper is well-written and clearly organized.
- The work focuses on recently popular topic , LLMs, and try to offer insight into selection of synthetic data / generator and balancing real and synthetic data. 
- Empirical results support the efficacy of proposed method. The method reveals scalability and real-world applicability.


Weakness of this paper

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.

- Experiment scope: the work focuses on small models. Whether the work is generalizable at larger models that used commonly in the community lacks evidence. Clustering in the work is constrained by LLM context window sizes, potentially leading to unpredictable cluster formations, and no clear solutions or comparisons to existing methods addressing long-document challenges are provided. The dependency on the same model for clustering and self-verification also introduces potential bias, and ideally a more capable or independent model should be used in verification for more reliable experiment results.
- Various concerns about settings for experiment and method. Over-simplifying diversity into one metric / score and overlooking its multi-variable nature might yield suboptimal results. Diversity is subjective and ambiguous; method built upon it suffers from inconsistency and issue of reproducibility. Besides, the methodology does not address how to determine or control the optimal number of clusters in an algorithmic way, also raise some concerns about stability of method.

Decision: Reject