Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
Authors: Dylan Sam
Ayan Chakrabarti
Afshin Rostamizadeh
Srikumar Ramalingam
Gui Citovsky
Sanjiv Kumar
Venue: NeurIPS 2025 poster ()
Keywords: data curation
language models
pretraining
Primary area: deep_learning
TL;DR: We propose a new framework and set of evaluation criteria to assess the utility of text embeddings used in data selection for pretraining langauge models.

Abstract: Measuring similarity between training examples is critical for curating high-quality and diverse pretraining datasets for language models. 
However, similarity is typically computed with a generic off-the-shelf embedding model that has been trained for tasks such as retrieval. 
Whether these embedding-based similarity metrics are well-suited for pretraining data selection remains largely unexplored.
In this paper, we propose a new framework to assess the suitability of a similarity metric specifically for data curation in language model pretraining applications. Our framework's first evaluation criterion captures how well distances reflect generalization in pretraining loss between different training examples. Next, we use each embedding model to guide a standard diversity-based data curation algorithm and measure its utility by pretraining a language model on the selected data and evaluating downstream task performance. Finally, we evaluate the capabilities of embeddings to distinguish between examples from different data sources. With these evaluations, we demonstrate that standard off-the-shelf embedding models are not well-suited for the pretraining data curation setting, underperforming even remarkably simple embeddings that are extracted from models trained on the same pretraining corpus. Our experiments are performed on the Pile, for pretraining a 1.7B parameter language model on 200B tokens. We believe our analysis and evaluation framework serves as a foundation for the future design of embeddings that specifically reason about similarity in pretraining datasets.

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
Dylan Sam" Carnegie Mellon University
Ayan Chakrabarti Google Research
Afshin Rostamizadeh Google Research
Srikumar Ramalingam Google Research
Gui Citovsky Google Research
Sanjiv Kumar Google Research
Abstract
Measuring similarity between training examples is critical for curating high-quality and diverse pretraining datasets for language models. However, similarity is typically computed with a generic off-the--heelf embedding model that has been trained for tasks such retrieval. Whether these embedding- based similarity metrics are well-suited for pretraining data selection remains largely unexplored. In this paper, we propose a new framework to assess the suitability of a similarity metric specifically for data curation language model pretraining applications. Our framework's 's first evaluation criterion captures how well distances reflect generalization in pretraining loss between different training examples. Next, we use each embedding model to guide a standard diversity- -asse data curation algorithm and measure its utility by pretraining a language model on the selected data and evaluating downstream task performance. Finally, we evaluate the capabilities of embeddings to distinguish between examples from different data sources. With these evaluations, we demonstrate that standard off-the-shell embedding models are not well-suited for the pretraining data curation setting, underperforning even remarkably simple embeddings that are extracted from models trained on the pretraining corpus. Our experiments are performed on the Pile, for pretraining a 1.7B parameter language model on 200B tokens. We believe our analysis and evaluation framework serves as a foundation for the future design of embeddings that specifically reason about similarity in pretraining datasets.
Introduction
The recent success of language models [Brown et al., 2020. Chowdhery et al., 2023] is in no small part due to pretraining on large and diverse text corpora scraped from variety of sources [Raffel et al., 2020, Gao et al., 2020, Penedo et al., 2024]. Researchers have explored a variety of approaches assemble effective pretraining sets, typically by selecting a high-quality and diverse subset from larger corpus of examples scraped from multiple data sources. These data curation approaches have shown promising results by improving example quality and reducing redundancy in pretraining sets. Many of these methods use notions of similarity between examples. Similarity an example to text from known high-quality sources (such Wikipedia) has been used as proxy for the quality of that example [Gunasekar et al., 2023, Penedo al., 2024]. Meanwhile, methods focused on diversification [Abbas et al., 2023, Tirumala et al., 2023] make direct use of similarity metrics to identify and remove redundant examples.
In this context, similarity between training examples has often been measured in terms of distances in an embedding space. Many approaches have typically used generic embeddings for this purpose
Work done while interning at Google Research. Correspondence to Lylaann@@andrr cmu, edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

"Your koop a small oiscionnates fongth > ree): So the for loap affictively does notting and just increnents to and eccess the elements index 5 the 2 arrays which would throw me if? capled Somayor (hy log my new devalopment...
is (f= 0.
mere solution PERL do tis? Meansshis, outpue
had 10,000 devalllamme
Distance 0.18
following program work with your system and does not require audiliional moduless use strict use wernings 'shoadings;; e we keep track all headings...
you wont should
Q: Capie, how viitual
Loss = 0.81
Distance 0.55
"CNnlcal Rosults and Functional Dutsomes Adul!
'Hepatic motabollsm and excration Taxol rete and humans. date there been liitldd studies of the melabolisn and disposttin of Taxol in animals humans. Renal disposition of unmetabolized has been decumented account for a maximum of 5% to 10% an...
Pstients After Revision Surgery Spinsl Deformity Correction: Potients Younger
Years Versus Years and Older. Retrospecive comperison. compare complictions and radiographic and functional autcomes of patients undergoing....
Distance D.11
Loss = 1.94
Distance
"Sir Richard Knightley KB (1617-1681), Fawsley in Marthamptonshir, English Member Partiament (MP). member Gray's Knightloy was a member of prominent...
"Caunty lists him as "paintor"" by tradi. Almost nothing
Distance 0.14
Zellff, although appeare on Commisstoners Deede for Morrs County in 1892, and in handful of lega! records
Loss: 1.77
Figure 1: visualization of the correlation between pretraining loss and embedding distance. Each row shows a pair of examples close in embedding space (from the same K-means cluster), with examples in different pairs of examples tend to have similar pretraining losses, while there is a greater variation losses across clusters. Close example pairs are "thematically" similar but have different content. These results are from averaged embeddings from the final layer of a small decoder-only language model.
being far from each other (from different clusters). We find that close
[Chang and Jia, 2023, Vo et at.) off-t-----hhthhee tfffftthe-sheffff embedding models that have been trained for tasks such as semantic retrieval or mask-based reconstruction. However, whether these embeddings are optimal for reasoning about similarity between pretraining examples and whether there is any benefit using more sophisticated and computationally expensive models remains an open question. Furthermore, evaluating such embeddings in pretraining applications is fundamentally challenging, the sheer scale of pretraining data makes many experiments intractable. Such challenges necessitate specialized evaluations - new criteria for assessing the utility of embeddings in pretraining applications.
In this paper, propose novel evaluation framework to assess the suitability of an embedding model for curating pretraining data for language models. Our goal for evaluating similarity metrics in the context of language model pretraining. We begin by asking: what should a similarity metric ideally capture be useful in this setting? Unlike prior work that evaluates embeddings for downstream tasks like retrieval or classification, we argue that embeddings used for data curation must reflect training dynamics-such as generalization behavior corpus redundancy- are unique to the pretraining setting.
establish new standard
Our first evaluation criterion measures how well distances in the embedding space correlate with generalization behavior during pretraining- sppesfiicallll. whether nearby examples in embedding space exhibit similar pretraining loss under the same model state (see Fig. 1). This captures their utility of similarity metrics in loss-based data selection strategies [Jiang et al., 2023] for selecting examples similar high-quality references [Penedo al., 2024]. Next, we evaluate the embedding model's utility in guiding simple diversity- curation method (similar to those employed in [Tirumala et al., 2023]) and assess its effectiveness by pretraining model the eurated subset and measuring downstream task performance. Finally, we also measure whether the embedding space can distinguish between examples from different data sources. While not directly used in data selection,

---

this serves as useful proxy for embedding quality under the assumption that these sources were curated by human expertise and reflect meaningful human-curated structure.
We conduct all of our experiments using the Pile [Gao et al., 2020] as our data corpus, and in the context of pretraining a 1.7B parameter decoder-only language model with UI.2 objective [Tay et al., 2022] on 200B tokens. We evaluate a number of different off-the-shelf embedding models from representations of generic language models (e.g., trained with mask-based reconstruction or mixtures of pretraining objectives) to embeddings specifically trained for retrieval or semantic equivalence. We compare these to two specialized approaches that are derived from smaller, proxy models of the downstream model being trained on the same pretraining set: (I) computing the average token embedding at the final hidden state, and (2) a significantly more computationally efficient and simple average of the token-en seen-eemedddings from the input layer (i.c., requiring no forward passes).
Despite their strong performance standard semantic similarity and retrieval benchmarks, we find that off-the-shelf embedding models underperform in the context of pretraining data curation, suggesting that existing benchmarks may not reflect the inductive biases most relevant to language model pretraining. Surprisingly, even this simple method that averages input token embeddings-- requiring no forward mats-matches or outperforms these more computationally intensive models. Adding forward pass in this specialized model yields further improvements. These results strongly suggest that embedding models for reasoning about similarity between pretraining examples should be specialized to the data distribution at hand, and that properties that make embeddings suitable for retrieval and semantic matching may not transfer to the pretraining setting. More broadly, our work establishes a standardized framework to evaluate and highlight failures of current off-the-shelf models, facilitating the design of new models tailored for data curation in language model pretraining.
2 Related Work
Data Curation. Many works have studied the problem of selecting high-quality and informative subsets from a larger corpus for efficient and effective language pretraining [Albalak et al., 2024]. Indeed, works have shown that data curation for pretraining (both for language and vision- lasoun-aaguuage models) improves neural scaling laws [Sorscher et al., 2022], and that curation techniques should be aware of the particular compute budget [Goyal et al., 2024].
One family of works approaches data curation with the goal of retaining only high quality examples. The simplest approaches in this family look at heuristics that can filter out noisy or extremely low- quality instances, such as documents containing less than 5 sentences or those with extremely short lines of text [Raffel et al., 2020, Xue, 2020]. Beyond these heuristic approaches, a line of work focuses on extracting high-quality examples based on similarity to known high-quality sources, such as Wikipedia or textbooks [Gunasekar et al., 2023, Penedo et al., 2024]. Other works look at creating targeted sets sets-defining quality in terms of their relevance for particular downstream tasks [Xie et al., 2023], with some creating these by adaptively updating weights over domain mixtures [Xie et al., 2024, Jiang et al., 2024]. The other family of data curation approaches focus on pruning datasets to ensure notion of diversity and reducing redundancies in pretraining examples [Abbas et al., 2023, 2024, Tirumala et al., 2023]. While some of these works [Abbas al., 2024, Tirumala et al., 2023] compare different embedding models as part of brief ablations, these comparisons are in the narrow context of their specific diversification algorithms. Finally, it is worth mentioning that some works in data selection use the model being trained as part of the selection process, often through the use of influence functions [Garima et al., 2020, Xia et al., 2024, Engstrom et al., 2024]. However, these are typically only used in data-curation on small datasets or only during finetuning, and would be prohibitively expensive in pretraining.
Text Embedding Models. Another related line of work the task of learning embedding models for text. Many various approaches to learn text embedding models, with objectives including mask- based reconstruction [Devlin, 2018. Liu al., 2019], a combination of multiple different tasks [Cer. 2018), and contrastive learning-based approaches [Gao et al., 2021, Neelakantan et al., 2022, Izacard et al., 2022, Lee et al., 2024]. More recent work has studied extracting embeddings from standard decoder-only language models [BehnamGhader et al., 2024], even via prompting [Sam et al., 2025].
These embedding models have largely been studied in the context for classification or similarity measures [Gomaa et al., 2013, Agirre et al., 2013, 2016], and recent focus has been improving performance on aggregate on large-scale benchmarks [Muennighoff et al., 2022] that are comprised of multiple tasks (c.g., retrieval, clustering, classification, ranking). Many models that achieve
3

---

strong performance on these large-scale benchmarks have benefitted from scaling [Jiang et al., 2023, Chowdhery et al.. 2023], with the help of distilled information from autoregressive models [Lee et al., 2024]. However, this has conflicting incentives with their utility in the pretraining setting, as large text embedding models are impractical and too computationally expensive to inference over the full pretraining corpus. To the best of our knowledge, our work provides the first study of various text embedding models for pretraining data curation.
3 An Evaluation Framework for Embeddings in Pretraining Data Curation
We now describe our new framework for analyzing text embedding models in terms their suitability for reasoning about similarity among pretraining examples. As previously mentioned, similarity is used to (a) find examples that similar to known "desirable" examples (e.g., examples of known high-quality or those representing downstream tasks of interest) and to (b) discover and remove redundancies in the pretraining corpus. Accordingly, we design standardized experiments that measure text embedding model's performance towards these criteria.
3.1 Evaluating Correlation with Prefraining Loss
We begin by evaluating whether low distances in a model's embedding space correlate with similar values of difficulty, which measure by intermediate losses during language model pretraining. Examples of positive correlations are visualized in Figure 1. To do so, we first use a balanced K- Means clustering algorithm in the embedding space to cluster all examples in the pretraining set for target cluster size. Then, we look at the variance of loss values within each cluster. This within-cluster variance captures whether points with similar pretraining loss (or difficulty) are grouped together and close in embedding space. Crucially, different examples can have the same loss because they happen to equally difficult to learn. This asymmetry is an important reason we chose our gluster-based approach rather than alternatives such as pairwise correlation. We note that we use balanced clustering (allowing only some variation in the sizes of different clusters) to ensure that average within-cluster variance is comparable across clusterings from different embedding models. We repeat this process for multiple target cluster sizes for all embeddings.
note that the converse is not necessarily true; two very
When reporting our results, we measure variance reduction, or the ratio of the overall variance across all examples in the pretraining dataset the variance in loss computed within clusters in the given embedding space. Formally, we define variance reduction as
Ex-D((((x) - =|(
V(C) = Ecco [Ez-c [[((x) Ea-c, [(())))))))
where -C Cm is a clustering or partition of all datapoints in the dataset D, and C is a uniform distribution over each disjoint set in the partition C. Random clustering achieves a variance reduction of 1, and larger values of variance reduction imply that points with more similar pretraining loss are clustered together.
A high value of variance reduction implies that similarity in the embedding space correlates well with pretraining loss. While an isolated pair of examples having similar pretraining loss may be entirely unrelated in quality and content, the fact that an embedding space that consistently brings together examples similar loss values strongly suggests that these similar examples will behave similarly in terms of contributing to the quality of the language model (when used in pretraining).
Beyond finding examples of similar quality and utility, high variance reduction score implies that the clustering above can serve as a particularly useful proxy for more dynamic and online data sampling strategies [Xie et al., 2024], reducing the number of required forward passes in strategies where datapoints are selected based towards approaches that look to self-im improve models through propagating labeled information to nearby examples [Wei et al., 2021, Cai et al., 2021, Pukdee et al., 2023], such as weak-to-strong generalization [Burns et al., 2023].
their current pretraining loss. This also has implications
3.2 Diversification-Dass Pretraining Data Curation
Prior work has demonstrated that data curation schemes that encourage diversification, based on similarity in embedding spaces, leads to improved models when trained on the curated subsets [Abbas

---

et al.. 2023, Tirumala et al., 2023]. While the diversification techniques tend to be sophisticated, our goal here evaluate the utility of given embedding for diversification. Therefore, we use simplified version of these approaches to (1) select a subset from a pretraining corpus, (2) pretrain a model on this curated subset, and then (3) report performance on large set of downstream tasks.
We begin by clustering the larger pretraining corpus in the given embedding space; in contrast to Section 3. we do not use balanced clustering, which can lead to clusters with a wide variation in "diameters" (distances among points in the same cluster). Instead, since the goal here is diversification, we find clusters such that distances between all pairs of points in the same cluster are within a specified threshold
Given clusters of pretraining data, prior work often uses complex pruning or sampling schemes from large clusters. For simplicity, we simply take the point in each cluster that is closest to the cluster centroid (i.e., the average embedding of all points in the cluster), which likely represents the most representative example for that cluster. We note that prior work [Tirumala et al., 2023] emphasizes the importance of deduplication step in language model pretraining, and our curation strategy implicitly performs this process when selecting only a single point from each cluster.
Given that we select single point from each cluster, this implies that we need the number of clusters to be equal to or greater than the number of desired pretraining examples. In practice, we sweep large grid of values and select the largest threshold that still produces a sufficient number of clusters. Another important point of note is that producing such a large number of clusters is computationally expensive. To be able to scale to such large number of clusters, clustering (RAC) [Sumengen et al., 2021] algorithm. This has the advantage of building a clustering from progressively larger values of e, which works well with our need for grid of values of 6.
the reciprocal agglomerative
Finally, pretrain a language model the selected subset of examples, and measure few-shot performance of the pretrained model on a diverse variety of downstream tasks.
3.3 Measuring Cluster Purity with Respect to Data Sources
Large pretraining datasets such as the Pile [Gao et al., 2020] are comprised of various distinct yet complementary hand-curated data sources (e.g.. high quality sources such as Wikipedia, code-based data, data from medical domains). While there may some similarities between datapoints from different domains, we believe that clusters should generally group points from the same domain together and separate instances from different domains. In fact, since embedding models are generally not trained with explicit knowledge of source labels or any domain metadata, the ability to group together data from the same source and distinguish those from different sources shows an embedding model's s alignment with meaningful human-curated structure (or at least that of the dataset curators). This also serves as useful proxy of embedding model capacity, as they should contain sufficient information to well-separate these different sources.
Letting 8 E (0. ... 5) represent the index of the source some finite number of sources S, we compute the cluster purity of a set of clusters C as
P(C)= = Ec..c max, |CnDDl
|Cal
where D, represents the subset of datapoints that from source 8. Intuitively, this represents the proportion of the maximum frequency source to the total cluster size. A cluster purity of I represents clustering that completely separates examples from different domains.
Experiments
We demonstrate the utility of our framework in evaluating off-the-shelf embedding models in measur- ing pretraining similarity and compare them with simple ways to produce specialized embeddings.
4.1 Experiment Details
Embedding Models For the off-the-shed embedding models, we consider: (1) Universal Sentence Encoders (USE) [Cer, 2018), which are general-purpon text encoders trained on a combination of objectives, (2) Gecko [Lee al., 2024], retrieval-focusee text embedding model trained via synthetic data distilled from LLMs, and (3) BERT [Devlin, 2018] embedding models. These encapsulate a variety of different training objectives and reflect common embedding model choices in the field.
5

---

12
12
Varlance Reduction
Variance Reduction
10-
20
60000
70000
Average Cluster Size
Training Steps
Figure 2: Variance reduction we vary average cluster size. Larger values are better. Results are computed over 50 million sampled clusters from the Pile, where pretraining losses are computed after 26k gradient steps. Specialized embeddings yield higher variance reduction than off-the- shelf models for all cluster sizes.
computed over 50 Figure 3: Variance reduction number of gradient steps in pretraining. Larger values are better. Results million sampled clusters from the Pile with an average cluster size of 50. Benefits in variance reduction remain consistent throughout pre- training.
increase the
We also consider a few standard approaches to extract specialized embeddings that are extracted from small version of the downstream language model are training (c.g., 200M parameter language model). The first and most simple approach is to extract an embedding by simply averaging the token embeddings matrix over all tokens in the input sequence, which we refer to as LM Token Embeds. This is extremely efficient as only requires looking up the token embedding matrix and does not involve any forward passes. We note that this is also equivalent to a learned unigram model (i.e., ignoring all positional information). The second approach is to extract embedding from the forward pass the language model by averaging the activations over all tokens in the input sequence at the final hidden layer, which refer to LM Output Embeds.
Finally, in diversity-based pretraining data curation experiments, also add comparison to naive, random subset selection (Baseline). This evaluates how much benefit is observed from our standardized curation strategy with these various embedding models over standard baseline training.
To make clustering with large number of output clusters feasible and efficient at pretraining data scales, we perform dimensionality reduction using PCA on a large subset of the datapoints (~550,0000 examples) and extract the top 64 dimensions upon which to project. The projected embeddings are then normalized to have unit L2 norm. We ablate on this choice for dimensionality reductions by comparing with random projections [Bingham and Mannila, 2001] as an alternative dimensionality reduction strategy; find that random projections do not perform as well as PCA (Section 4.5). Other work [Tirumala et al., 2023] tackles this computational challenge by running clustering only with much smaller subset of data, rather than performing dimensionality reduction and then clustering over all the data.
Clustering Details
Dataset and Pretraining Details For all of our experiments (e.g., pretraining data curation and predicting loss generalization), we use the Pile [Gao et al., 2020]. We pack together documents into a sequence of length 1280, with "[eod]" as delimiters between documents. For our pretraining experiments, we train 1.7B parameter decoder-only language model with a UL2 objective [TTa al., 2022]. We curated pretraining subsets from the Pile through the process outlined in Section 3.2. We use selection budget of 200B tokens, or approximately 20% of the Pile. This corresponds to roughly 170 million clusters for each embedding model, with average cluster size of 5 examples.
We pretrain with learning rate of 0.001 with a linear decay and batch size of 1024. For our tokenizer, we use sentencepiece with a vocabulary size of 256k tokens. For our pretraining evaluation, consider the set of 23 downstream evaluation datasets from the work of Brown al. [2020]. This involves a wide variety of 1 I-shot scoring tasks, as well as open-ended text generation tasks. More details about our evaluation sets deferred Appendix B.I. For the embeddings produced via

---

Table 1: Average downstream task performance of embedding models in the enversity-basee curation schemes of 200B token subset from the Pile, to pretrain 1.7B parameter decoder-only language model. Bolding and italicizing denote the best and second-bes best performing methods on each task, respectively. Results are averaged over pretraining runs, and average results are mean + standard error. Specialized embeddings (i.e., extracted from a small version of the downstream language model) and Gecko perform the best for diversity- -based pretraining data curation.
LM Token Embeds
LM Output Embeds
Task
Baseline
USE
Gecko
BERT
ARC Challenge
32,4
33.0
33.7
32.7
32.5
33.5
ARC Easy
63.8
65.1
64.2
65.1
64.9
65.5
BoolQ
56.5
59.2
58.3
60.1
62.9
61.9
SuperGLUE CB
42.4
43.5
48.2
41.7
42.9
48.8
SuperGLUE Copa
75.3
76.3
77.3
74.0
74.7
78.0
HellaSwag
55.0
56.7
57.1
56.5
56,8
57.5
Multi RC
57.7
55.7
55.9
56.3
56.0
55.2
OpenBook QA
46.3
46.2
46.3
46.6
45.9
46.7
PIQA
72.3
73.0
72.7
73.6
72.8
73.6
Race
38.0
37.9
38.5
38.8
38.8
38.7
Race M
51.7
52.0
53.3
52.9
52.1
52.6
ReCoRD
85.0
84.4
84.6
84.7
84.5
84.9
RTE
54.5
54.9
51.5
54.9
51.6
52.8
Story Cloze
73.9
74.1
74.3
74.0
73.3
74.5
WiC
48.2
47.9
47.5
47.5
48.2
47.3
Winograd
76.0
77.1
77.3
77.1
WinoGrande
59,2
58.9
59.0
59.5
60.0
59.3
WSC
74.1
73.9
73.7
74.7
73.2
75.0
Lambada
21.4
29.9
40.1
34.7
31.4
33.8
Natural Questions
10.0
10.8
10.1
10.5
11.1
11.4
SQuAD v2
51.1
54.3
51.7
51.8
58.6
55.4
TriviaQA Wiki
34.4
35.0
33.0
35.0
34.3
36.1
Web Questions
17.1
17.0
17.9
18.0
18.7
19.1
Average
51.9
+ 0.1
52.9 +0.1
53.3 4 0.3
53.1 +: 0.2
53.1 + 0.3
53.8 + 0.1
LM Token Embeds and LM Output Embeds, we train 200M parameter language model in the same fashion as above, where we also train a total of 200B randomly selected tokens.
4.2 Pretraining Loss Correlation Results
To measure the ability of embedding models to reflect pretraining loss generalization and to cluster together datapoints with similar difficulty, we report their variance reduction as we vary (1) the number of gradient updates performed by the intermediate pretraining checkpoint and (2) as we vary average cluster size our K-Means clustering.
We observe that specialized embeddings (i.e., LM Token Embeds and LM Output Embeds) from small version of the downstream trained model, which is trained on the same pretraining data, achieve a much larger reduction loss variance when compared to off-the-shelf models. These specialized embeddings produce better clusters with datapoints with similar intermediate pretraining losses, across all cluster sizes (Figure 2) and for all intermediate pretraining checkpoints (Figure 3). This suggests that training models specifically on the data to perform data curation is preferable to using off aff-the-shelf embedding models for predicting pretraining loss generalization. Furthermore, we also generally observe that newer embedding models (c.g., Gecko) do not improve upon older alternatives (e.g., BERT and USE), supporting that embedding model improvements in other settings, such retrieval do not translate to this application in the pretraining setting-
better interpret these similarity metries, we visualize pairs of datapoints from the clusters in Figure to understand what types of examples are placed closely together and which have similar pretraining losses. We randomly sample clusters produced by K-Means clusters with the smallest average cluster size, with the LM Output Embeds method to extract embeddings. We randomly sample 2 points from these 3 clusters and observe that samples
Visualizing Within-Cluster Examples

---

0.950
0,925
Cluster Purity
0.900
0.875
6.025
0.775
250
Cluster Size
Figure 4: Comparison the purity with respect to data source of K-Means clustering produced by various embedding models on the Pile, when averaged over 50 million clusters from the Pile. Specialized embedding models have higher cluster purity scores.
from the same cluster are thematically similar, although containing very different information (e.g.. the last pair referring two different govermment-rellce people, although from different times and different countries). We observe that pretraining losses are similar among within-cluster examples, and lower distances correspond to smaller differences in pretraining loss.
4.3 Diversification-bast Pretraining Data Curation Results
We report the average downstream task performance of language models trained on pretraining data subsets produced by using various embedding models in the simple diversity- hraersity-based data curation strategy in Table 1. We first observe that the standardized iiversity-based data curation technique applied to all embedding models (both off-the-shelf and those based on small downstream trained language models) outperforms the naive random subset selection baseline. Secondly, we observe that the LM Output Embeds achieves the best performance when averaged across all tasks.
notable result is that LM Token Embeds performs comparably to Gecko, while outperforming all other off-the-shed embeddings. This is of immediate practical interest due to the simplicity of LM Token Embeds; they require no forward pass and, consequently, are extremely quick to compute in comparison to all other considered alternatives. Thus, LM Token Embeds serves as viable alternative in settings that are compute- -imited. This also suggests that even simple notions of similarity, even those that do not account for positional information, are sufficient for many pretraining data curation applications. Finally, they support that models specialized to this task (e.g., trained on the same dataset) often outperform the current, ceneral-purpose off-the- off-the-shelf embedding models. While this does include training an additional model, it is a single-time investment that is amortized across all embedding use cases and is easily offset by the significant improvements in data curation quality.
4.4 Cluster Purity Results
To better understand and interpret the similarity metrics defined by these embedding models, measure the purity of clusters with respect to the underlying data source. We again remark that none of these embedding models have been trained with knowledge of the data source. Overall, we observe that most embedding models produce fairly pure clusters, where a majority of points come from same underlying data source (Figure 4). This supports that these embeddings models are generally aligned with human judgment about differences between types of data.
observe that the embeddings extracted from the small version of the trained downstream language model achieve the highest cluster purity. We also note that Gecko embeddings achieve the third- highest cluster purity, whereas performs poorly on the task of predicting loss generalization through variance reduction. This suggests that an improved ability to predict pretraining loss generalization cannot simply be explained via producing clusters that more pure with respect to underlying data sources. Future work could explore incorporating domain information or metadata into the embedding model's 's learning objective, form of weak supervision [Sam and Kolter, 2023].

---

Varlance Reduction
4.0
Geckoo (PCA 178)
Gecko (PGA 64)
Variance Reduction
LM Ourput Embeds (PCA 128)
LM Ourput Embeds (PCA 64)
(PCA
13.0
Output Embeds 256)
3.8
12.5
3.6
12.0
11.5
3.4
10.5
50
100 150
200
250
50
100
150
200
250
Average Cluster Size
Average Cluster Size
Figure 5: Ablation on the number of components in PCA for Gecko and LM Output Embeds. Results averaged over 50 million sampled clusters from the Pile. Using more components in PCA better clusters points with similar pretraining loss.
Gecio (PCA)
13.5
LM Output Embeds (PCA)
Variance Reduction
3.8
Gecko (Random Projection)
Variance Reduction
13.0
Output (Random Projection)
3.6
12.0
11.5
3.2
11.0
3.0
10.5
2.8
10.0
2.6
9.5
100
200
250
150
250
Average Cluster Size
Average Cluster Size
Figure 6: Ablation comparing the use of PCA Random projections for dimensionality reduction in Gecko and LM Output Embeds. Results are averaged over 50 million sampled clusters from the Pile. Dimensionality reduction via PCA performs better than via Random Projections.
4.5
Ablations
We present ablations on components of our evaluation framework -peneworr--eeiifiallllly in how have chosen to perform dimensionality reduction (i.e., technique and resulting size), which is used for the embedding models before running clustering. We present the ablations for Gecko and LM Output Embeds and defer results on others embeddings to Appendix A.
Less Components PCA Does Not Significantly Hurt Performance. The dimensionality of the embeddings used in clustering often must be low for pretraining scales. Here, run ablation studying the role of dimensionality (i.e., the number of components in PCA onto which the embeddings are projected) in the variance reduction through clustering (Figure 5). We remark that scaling clustering accommodate smaller average cluster sizes (i.e., more cluster centers) is intractable for embeddings that have high dimensionality. Thus, we can only report results on various embeddings with a large number (i.e., 256) of components in PCA with larger average cluster sizes. observe the trend across embeddings that a higher dimension and larger number of PCA components improves the embedding models' ability to cluster points by pretraining loss (see other embedding model results in Appendix A.
Dimensionality Reduction with PCA Outperforms Random Projections. Another common technique to perform dimensionality reduction (especially with respect to maintaining pairwise distances and similarities) is use random projections. We experiment with using random projections in our loss clustering experiments and observe that it is outperformed by PCA in terms of variance reduction across all cluster sizes (Figure 6).

---

5 Discussion
We present a new evaluation framework for the understudied area of similarity metrics used in lan- guage model pretraining data curation. Using our framework, we show that off-the-shelf embeddings despite their widespread use-often underperform even simple, specialized alternatives such as an average of token embeddings that requires no forward pass, This also suggests that practitioners embedding models, and it suffices train them on small fraction of the data (~200% and at a much smaller parameter scale. While it may seem intuitive that embeddings tailored to the pretraining task would outperform generic ones, this has not been systematically demonstrated and is certainly not yet standard practice. Our framework both surfaces this gap and offers a practical tool for guiding the design of embedding models optimized for data curation, which can lead to significant improvements in data efficiency.
should train their
While we believe that our results are fairly general, future work could extend our findings to other pretraining corpora. Beyond evaluation, our findings have broader implications for pretraining workflows-for example, in selecting task-specific finetuning data [Xia et al., 2024], or identifying synthetic examples that resemble real data [Meng et al., 2022, Sam et al., 2024], and even studying the impacts of scaling up embedding models in these tasks. Overall, our results underscore the importance of using task-aligned similarity metries in pretraining, and our framework provides a foundation for future research in building and optimizing the design of embedding models tailored to this critical step of the language modeling pipeline.
Limitations One limitation of our work is that the empirical demonstration of our evaluation framework restricted to the Pile dataset. We believe this is fairly standard in the language model pretraining data curation literature to focus on single pretraining corpus (e.g., as is done in Tirumala al. [2023]), due the computational costs of both data curation algorithms and pretraining.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540 2023.
Amro Kamal Mohamed Abbas, Evgenia Rusak, Kushal Tirumala, Wieland Brendel, Kamalika Chaudhuri, and Ari S. Morcos. Effective pruning of web-scale datasets based on complexity of concept clusters. In The Twelfth International Conference on Learning Representations, 2024. URL https::/opeenerr///
net/ aett/forrumidd=C:DAaaaee.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agiree and Weiwei Guo. sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pages 32-43, 2013.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016, 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL (Association for Computational Linguistics), 2016
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. survey on data selection for language models. ArXiv, 2b2s22402.16827,20000
Parishad BehnamGhader Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Ilm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404. 0096), 2024.
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. Semantic parsing on freebase from guestion-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544, 2013.
10

---

Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 245-250, 2001.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.099300 2023.
Tianle Cai, Ruiqi Gao, Jason Lee, and Qi Lei. A theory of label propagation for subpopulation shift. In International Conference on Machine Learning, pages 1170-1182. PMLR, 2021.
D Cer. Universal sentence encoder. arXiv preprint arXiv: 2018.
Ting-Yu Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 81 8123-8144, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240) 2023.
Christopher Clark, Kenton Lee, Ming-Woi Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 1.Xiv:1803.080 05457v1, 2018
Ido Dagan, Bill Dolan, Bernardo Magnini. and Dan Roth. Recognizing textual entailment: Rational, evaluation and approaches-erratune Natural Language Engineering, 16(1):105-105 2010.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection
with datamodels. In Forty-first International Conference on Machine Learning, 2024. URL
net wtrf//ormmmm/ 7 -idndddddG-GGKHBB.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 101.00001,,077, 2020.
Gao, X Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP 2021-2021 Conference on Empirical Methods in Natural Language Processing, Proceed- ings, 2021.
Garima, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 19920-19930, 2020.
Wael H Gomaa, Aly A Fahmy, et al. A survey of text similarity approaches. international journal of Computer Applications, 68(13):13-18, 2013.
Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J 7ico Kolter. Scaling laws for data filteringdatt data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22702-22717 2024.
11

---

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 1644, 2023.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive Iearn- ing. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
URL https:
//openreview.oo hwtp:viwwwweeeeeeeeeeeeeeeeeeeee wwt.:/ee/foooote httt:/forummmd=:kk1ppppp
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307. 16645, 2023.
Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws, arXiv preprint arXiv:2410.11820 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1601- 1611, 2017.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In NAACL, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics,, 7:453-466, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, 2017.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327 2024.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv e-prints, pages arXiv-1907, 2019.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462-477 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint Iexivv 60..0609...66.... 2016
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint 1000v:2200.073 2022.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training, arXiv preprint 10000:2201.10005, 2022.
12

---

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quee QueeeQuann Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Ferúnndez. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1525-1534, 2016.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 7557, 2024.
Mohammad Taher Pilchvar and Jose Samacho-Collador Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv: 09121, 2018.
Rattana Pukdee, Dylan Sam, Pradeep Kumar Ravikumar, and Nina Balcan. Label propagation with weak supervision. In The Eleventh International Conference on Learning Representations, 2023. URL
-RRqtI
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (1(40)):1611 2020.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Dylan Sam and J Zico Kolter. Losses over labels: Weakly supervised learning via direct loss construction. In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 9695-9703, 2023.
Dylan Sam, Devin Willmott, Joao Semedo, and J Zico Kolter. Finetuning clip reason about pairwise differences. arXiv preprint arXiv:2409.0972.1 2024.
Dylan Sam, Mare Finzi, and J Zico Kolter. Predicting the performance of black-box Ilms through 2025.
self-queries. arXiv preprint arXiv:2501.0f.:
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19923--1933 2022.
Baris Sumengen, Anand Rajagopalan, Gui Citovsky, David Simcha, Olivier Bachem, Pradipta Mitra, Sam Blasiak, Mason Liang, and Sanjiv Kumar. Scaling hierarchical agglomerative clustering to billion-sized datasets. arXiv preprint arXiv: 105. /653, 2021.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXiv preprint M.XXv:2205.051333 2022.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving 11m pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, (8:53983-53999 2023.
Huy V Vo. Vasil Khalidov, Timothée Darcet, Théo Moutakanni, Nikita Smetanin, Mare Szafraniee, Hugo Touvron, Maxime Oquab, Armand Joulin, Herve Jegou, et al. Automatic data curation for self-supervised learning: A clustering-based approach. Transactions on Machine Learning Research.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for seneral-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2021. URL https: : http///wppereeeeeee/eee/ee
13

---

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning, 2024. URL
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Perey Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:
34201-34227, 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances Neural Information Processing Systems, 36, 2024.
Xue. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1193: 2020.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:190..07830, 2019.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprchension. arXiv preprint 1110::1810.120 2018.
14

---

A Additional Experimental Results
A.1 Impact Dimensionality Reduction on Variance Reduction
We now present the remaining results for other embedding models, when we ablate the number of components used in PCA for K-Means clustering, specifically when looking at the reduction in variance of pretraining loss of points within the same cluster. We observe that more components in PCA indeed help achieve higher variance reduction across all embedding models.
5.8
Variance Reduction
4.4
BERT (PCA 128)
BEAT (PCA 64)
Variance Reduction
USE (PCA 128
USE (PCA 64)
4.2
(PCA 236]
5.4
USE 256!
5.2
4.0
3.8
3.6
4.6
4.4
3.4
50
75
100
125
150
175
200
225
50
75
100
125
150
175
200
225
Average Cluster Size
Average Cluster Size
11.5
LM Taken Embeds (PCA 64)
Variance Reduction
LM Token Embeds (PCA 1281
11.0
Embeds 2561
10.5
10.0
9.5
9.0
50
100
150
200
250
300
Average Cluster Size
Figure 7: Ablation on the number components in PCA for embeddings from BERT, USE, and LM Token Embeds. Results are averaged over 50 million sampled clusters from the Pile.
A.2 Comparison of Random Projections to PCA for Dimensionality Reduction
We now present the remaining results for other embedding models, when we use Random Projections for dimensionality reduction instead of using PCA. We consistently observe that embeddings paired with PCA outperform those using Random Projections.
B
Additional Experimental Details
B.1 Additional Evaluation Details
We report results averaged over a large number of downstream tasks. These datasets largely follow two categories: scoring and decoding tasks. Scoring tasks primarily look at the output distribution of the model, while decoding tasks look text generations from the language model. Scoring tasks are performed as I-shot (i.e., giving one demonstration of format and answer), while decoding is performed zeroshot. For scoring tasks, we look the standard top-1 accuracy. The list of scoring tasks is as follows:
ARC Challenge: Easy and Challenge [Clark et al., 2018]
BoolQ [Clark al., 2019]
SuperGLUU CB and Copa [Wang al., 2019]
HellaSwag [Zellers et al., 2019]
15

---

BERT (PCA)
USE (PCA)
Varlance Reduction
4.2
BERI (Randam Projectioni
Variance Reduction
USE (Kandom Projection)
4.0
3.8
3.4
3.5
3,2
50
75
100
125
150
175
200
225
50
75
100
125
150
175
200
225
Average Cluster Size
Average Cluster
Size
11.0
LM Taken Embreds (PCA)
Variance Reduction
Token Embeds (Random Projection!
10.5
10.0
9.5
9.0
8.5
100
200
250
300
Average Cluster Size
Figure 8: Ablation where we compare using Random Projections for dimensionality reduction with embeddings from BERT, USE, and LM Token Embeds. Results are averaged over 50 million sampled clusters from the Pile.
MultiRC [Khashabi al., 2018]
OpenBookQA [Mihaylov et al., 2018]
PIQA [Bisk al., 2020]
RACE-H and RACE-M [Lai et al., 2017]
ReCoRD [Zhang et al., 201 8]
RTE [Dagan et al., 2010]
Story Cloze [Mostafazadee et al., 2016]
WIC [Pilehvar and
2018]
Winograd [Levesque et al., 2012]
WinoGrande [Sakaguchi et al., 2021]
WSC [Levesque et al., 2012]
For decoding text generation tasks, evaluate the language model outputs with its F1 score.
Decoder tasks are also evaluated as I-shot. The list of decoding tasks is as follows:
Lambada [Paperno et al., 2016]
Natural Questions [Kwiatkowski et al., 2019]
Squad [Rajpurkar et al., 2018]
Trivia QA Wiki [Joshi al., 2017]
Web Questions [Berant et al., 2013]
B.2 Additional Embedding Model Details
The embedding models of BERT, Gecko and USE trained with sequence lengths of 512, which we apply on the first 512 tokens data from the Pile. For Gecko, we use the 11 million parameter model version, while for USE we use the 109 million parameter version. For BERT, we also use
16

---

109 million parameter model. For the small language model that we train with the UL2 objective, we use one with approximately 200 million parameters.
Both the off-the-shelf BERT and USE embedding models have a dimensionality 512. The Geeko embedding model has dimensionality of 768. The small language model has token embedding dimension 512 and an hidden activation dimension of 512, which means that both LM Token Embeds and LM Output Embeds have 512 dimensions.
B.3 Additional Hyperparameter Details
Clustering For performing RAC clustering for our pretraining experiments, we use a value of as the particular diameter of clusters:
USE: = 0.2, which defines roughly 225 million clusters
Gecko: E = 0.2. which defines roughly 220 million clusters
BERT: 0.001, which defines roughly 175 million clusters
LM Token Embeds: 0.001, which defines roughly 170 million clusters
LM Output Embeds: c = 0.03, which defines roughly 180 million clusters
For our K-Means clustering, we perform clustering 4 different levels of granularity in our variance reduction and cluster purity results. We create four sets of clusterings with an average cluster size of 25, 50, 100, 150, with minimum cluster size of times the average cluster size, and maximum cluster size of 5 times the average cluster size. For both RAC and K-Means, we the squared L2 (Euclidean) distance.
B.4 Dimensionality Reduction Details
For running our dimensionality reduction via PCA, we compute the means and components on which project on subset of the data (~5000,000 points). We first standardize the embeddings to have mean of and variance of before running PCA. As previously mentioned, after projecting onto the desired number of principal components, we perform L2 normalization.
For our random projections, we use sparse random projections onto values of probabilities 1 1 1 respectively (i.e., the default parameters via scikit-leam). We also L2 normalize the result random projections.
// yn with
< /64
B.5 Compute Details
Pretraining experiments for our 1.7B parameter language models are run on 512 v5 TPUs, where each pretraining run takes approximately 3 days. Training proxy 200M parameter model took less than day on v5 TPUs. Hierarchical clustering for pretraining requires approximately days to run over the full pretraining corpus.
B.6
Asset Licenses
The existing assets that use have the following licenses:
RAC Clustering: MIT license
The Pile: CC BY 4.0
Evaluation Datasets: MIT License
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: claims are supported by experimental and theoretical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results are included.
Guidelines:
18

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: No code is provided, but all experimental details and evaluation metrics are clearly defined.
19

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/ cablic/guiden wwwps//oooeSumiisiionnne...
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. cccppccipiicccccccccccc httpa///ggiiieeeeeetuiiiiiiii /Codeeubmissiiieeececeeeee weriiissioioiicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are provided in the code in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Error bars are provided when applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
20

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttpa:////euis////eeeeee ccppp.c.......ooo.ooo ccc/publc///eeeeeeceeeeeeeeeeeee dtt
csGuidel ines?
Answer: [Yes]
Justification:
The submission conforms with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Impacts of the paper are discussed in the discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
21

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
No such risks are posed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are properly credited, and licenses are mentioned.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released.
Guidelines:
22

---

The answer NA means that the paper does not release new assets. Researchers should communicate the details the datasct/code//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: All usage of LLMs (in computing similarities and pretraining) are clearly described in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: :/ ///////eiii.../ Ceeuurpps.ccCooofeeeeeecc cc/Confferecces/2255 /LLM) for what should or should not be described.
23


Reviews:
[{"review_id": "PvktDdM0M3", "reviewer": "Reviewer_CAUd", "summary": "The authors present a new evaluation framework for the understudied area of similarity metrics used in language model pretraining data curation. Using their framework, they show that off-the-shelf embeddings—despite their widespread use—often underperform even simple, specialized alternatives such as an average of token embeddings that requires no forward pass.", "questions": "1. It would be interesting to explore whether representations from relatively smaller general-purpose models (e.g., around 1.7B parameters) could serve as effective signals for data selection. With the aid of acceleration techniques, the inference cost of such models is unlikely to be a limiting factor.\n2. The data similarity metric proposed in this work appears to primarily capture textual style similarity. However, for modern large models, the actual content of the text is equally critical. Could relying on representation-space similarity as the selection criterion inherently introduce biases on certain datasets?", "limitations": "Yes, the authors have discussed the limitations.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strength\n\n1. This paper focuses on a simple and fundamental, yet underexplored problem: the selection and curation of pre-training data. Under the current data-driven paradigm, this may offer some meaningful insights.\n2. The authors propose a practical metric evaluation protocol to assess the suitability of the data selection criteria before applying them in actual data selection.\n\n### Weakness\n\n1. The presentation of the paper could be further improved. It is recommended to include a flowchart or schematic diagram to summarize the proposed data similarity metric evaluation protocol.\n2. The paper could further elaborate on the impact of redundancy reduction in the context of modern language model pre-training, which would better highlight the significance of this work.", "was_revised": true, "final_justification": "This paper tackles a simple yet fundamental and underexplored problem—pre-training data selection—and proposes a practical metric evaluation protocol with clear applicability. Despite presentation issues and limited discussion on redundancy reduction, I think the strengths of this paper outweigh its weaknesses.", "rebuttal": "We thank the reviewer for their thoughtful feedback and are encouraged that you view our work as addressing an “understudied area” in how notions of similarity are used in data curation for language models. We address your individual comments below:\n\n> **\"It is recommended to include a flowchart or schematic diagram to summarize the proposed data similarity metric evaluation protocol\"**\n\nThank you for this feedback! Our main evaluation protocol is described in Subsections 3.1, 3.2, and 3.3, and we plan to include a figure that visualizes these components in our camera-ready revision.\n\n> **\"The paper could further elaborate on the impact of redundancy reduction in the context of modern language model pre-training, which would better highlight the significance of this work\"**\n\nWe believe that redundancy reduction (and deduplication) plays an important role in modern language model pretraining. Prior work has shown that exact deduplication can lead to significant performance improvements [1], and that semantic redundancy reduction can result in more efficient pretraining [2]. Within this broader context, our work analyzes **how similarity should be measured** — a key but **underexplored** design choice in these prior works on redundancy reduction. We will add this clarification to our revision.\n\n> **\"It would be interesting to explore whether representations from relatively smaller general-purpose models (e.g., around 1.7B parameters) could serve as effective signals for data selection. With the aid of acceleration techniques, the inference cost of such models is unlikely to be a limiting factor.\"**\n\nThis is an excellent point. We use a 200M parameter model as a computationally efficient proxy to generate specialized embeddings, and our results show that even this smaller, specialized model outperforms larger, off-the-shelf alternatives. We agree with the reviewer that using a 1.7B parameter model that has been pretrained on the same corpus would be effective and likely yield even stronger results, which would further reinforce one of our central claims that embeddings specialized to the pretraining corpus are highly performant in measuring similarity. We believe that scaling up such embedding models is an interesting direction for future work, and we will add this discussion to our revision.\n\n> **\"The data similarity metric proposed in this work appears to primarily capture textual style similarity. However, for modern large models, the actual content of the text is equally critical. Could relying on representation-space similarity as the selection criterion inherently introduce biases on certain datasets?\"**\n\nThis is a great question. Our qualitative analysis in Figure 1 suggests that the learned representations capture **more than just superficial textual style**. For instance, two of the examples clustered together have **related topics** (e.g., scientific or historical content) despite significant differences in their specific formats. This indicates that the learned representations are sensitive to content semantics, not just stylistic features.\n\nThat said, we certainly acknowledge that the potential inductive biases of any given representation space are a complex issue, even when they are learned in a general-purpose manner (e.g., through standard pretraining objectives such as next-token prediction). We will add a note to our discussion clarifying that while our method shows strong empirical results, a deeper analysis of these biases is an important direction for future research.\n\nThanks again for your thorough review! We hope our responses and clarifications have fully addressed your concerns.\n\n[1] Tirumala, et. al. D4: Improving llm pretraining via document de-duplication and diversification.\n\n[2] Abbas, et. al. Semdedup: Data efficient learning at web-scale through semantic deduplication."}, {"review_id": "hltQJhwnVm", "reviewer": "Reviewer_qKM8", "summary": "The paper introduces an evaluation framework for deciding whether a text-embedding model is suitable for curating pre-training data for large language models. The authors motivate and propose three metrics that help to evaluate the quality of the embedding model. They also pointed out that the LM Token Embeds (require no forward pass), and LM Output Embeds outperform the off-the-shelf embedding models with repect to the three metrics.\n\nThe three metrics are variance-reduction ratio, downstream score of the model on a selected subset, and cluster-purity. The three metrics are simple and intuitive. The paper provides an evaluation framework for the community.\n\nThe paper is well-written and because using LM Token Embeds and LM Output Embeds is not a general practice, the paper is a good contribution to the community.", "questions": "1. Have the authors provided detailed metrics—including total FLOPs, elapsed time, for the 200 M-parameter proxy model, the RAC clustering stage, and the full 1.7 B pre-training run?\n\n2. Do the authors think the three metrics still hold and are useful when considering a corpus that has large portion of code?\n\n3. How sensitive is the the downstream performance to the $\\epsilon$ grid? In line 175, I see that we would like to select the possible largest $\\epsilon$.  It would be helpful to know whether the final results are robust to this setting; if performance varies little with $\\epsilon$, the grid search could be narrowed and further compute saved.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written and the authors have done a good job of motivating the three metrics, providing a clear definition of the metrics. The authors also carried out large-scale pretraining experiments (1.7B language models with 512 v5 TPUs) to evaluate the three metrics. Pre-training experiments are expensive to carry out, thus the contribution and insights are valuable. The paper has good clarity, the writing is clear and the figures are well-designed. Line 128 has a small grammar error. In terms of originality, since both LM Token Embeds and LM Output Embeds are not general practice, the paper is a good contribution to the community. The three metrics are simple intuitive, and easy to carry out. For weakness, only one corpus (the Pile) and one model architecture were evaluated. However, as noted by the authors the conclusion stands alone without the evaluation of other corpora. Another potential weakness for originality is the statistical tool, variance ratios, and cluster purity are classical, but the application is new.", "was_revised": true, "final_justification": "The authors have provided the runtime of proxy model training, RAC clustering and 1.7B model pretaining to answer my question. Hope to see more on the robustness of $\\epsilon$ can be exploited to speed up the process, as currently the clustering cost 1-2 days.", "rebuttal": "We thank the reviewer for their positive assessment and for finding our work to be a \"good contribution to the community\" with \"valuable\" insights. We are glad you found the paper well-written and the experiments insightful. We address your questions below.\n\n> **\"Have the authors provided detailed metrics—including total FLOPs, elapsed time, for the 200 M-parameter proxy model, the RAC clustering stage, and the full 1.7 B pre-training run?\"**\n\nWe will add these details to the Appendix in our revision for completeness. To summarize here: while we did not log the exact total FLOPs, the approximate run times were as follows:\n* **200M Proxy Model Training**: Took less than 1 day on 64 v5 TPUs.\n* **RAC Clustering**: Took approximately 1-2 days to run on the full dataset.\n* **1.7B Model Pretraining**: Each run took approximately 3 days on 512 v5 TPUs.\n\n> **\"Do the authors think the three metrics still hold and are useful when considering a corpus that has large portion of code?\"**\n\nYes, we believe the framework is directly applicable. We would like to remark that one data source for the Pile is Github, so this pretraining corpus already contains a substantial amount of code. Furthermore, when computing variance reduction, these are clusters produced by a balanced K-means clustering algorithm, which means that even with large amounts of code, this slice of the dataset will be partitioned into many smaller clusters. This allows for a granular and meaningful analysis of similarity within the code domain, not just between code and other text types.\n\n> **\"How sensitive is the downstream performance to the $\\epsilon$ grid? In line 175, I see that we would like to select the possible largest. It would be helpful to know whether the final results are robust to this setting; if performance varies little with $\\epsilon$, the grid search could be narrowed and further compute saved.\"**\n \nThis is an excellent question regarding robustness. The grid search for $\\epsilon$ is only used to find a threshold that produces a sufficiently fine-grained clustering (i.e., a target number of clusters) for our data selection budget. Crucially, we did not optimize this hyperparameter for each embedding model; the same process was used for all of them to ensure a fair comparison. We believe that performance is likely robust to small changes in ϵ as long as it yields a comparable number of clusters. We will clarify this in the revision and note that narrowing the grid is indeed a promising efficiency improvement if one is not comparing across embedding methods."}, {"review_id": "4akVYMmlfg", "reviewer": "Reviewer_VPjN", "summary": "This paper proposes a framework for evaluating embedding models in the context of language model pretraining. The authors aim to set a new standard for how similarity metrics should be assessed for this use case. They start with the question: what should a similarity metric capture to be useful for curating pretraining data? Their main claim is that embeddings used for this purpose should reflect training dynamics, like generalization behavior and redundancy in the dataset, rather than downstream retrieval or classification performance. The paper introduces three evaluation criteria: (1) do distances in embedding space correlate with pretraining loss, (2) how well do embeddings support diversity-based selection methods, and (3) can embeddings distinguish between examples from different data sources. The experiments are run on the Pile with a 1.7B decoder-only model trained with a UL2 objective on 200B tokens. They compare standard off-the-shelf embeddings to simple embeddings taken from smaller models trained on the same corpus. These in-domain embeddings, even ones computed without a forward pass, perform better across most metrics.", "questions": "1. Can you provide a direct analysis of the relationship between embedding distance and pretraining loss similarity, rather than relying solely on cluster-based variance reduction? This would help clarify whether your method supports the claim that embedding distance reflects generalization behavior.\n\n2. How do you address the potential circularity in using embeddings from models trained on the same data used in the evaluation? Would these embeddings generalize to new domains or corpora where no model has yet been trained?\n\n3. Can you report more complete results for the variance reduction analysis, such as how it changes across different cluster sizes, seeds, or pretraining checkpoints? The current presentation makes it hard to assess the robustness of your findings.\n\n4. Why did you choose balanced K-means for clustering? Have you tested how sensitive your results are to the clustering algorithm, and could this choice be inflating variance reduction?", "limitations": "1. Can you provide a direct analysis of the relationship between embedding distance and pretraining loss similarity, rather than relying solely on cluster-based variance reduction? This would help clarify whether your method supports the claim that embedding distance reflects generalization behavior.\n\n2. How do you address the potential circularity in using embeddings from models trained on the same data used in the evaluation? Would these embeddings generalize to new domains or corpora where no model has yet been trained?\n\n3. Can you report more complete results for the variance reduction analysis, such as how it changes across different cluster sizes, seeds, or pretraining checkpoints? The current presentation makes it hard to assess the robustness of your findings.\n\n4. Why did you choose balanced K-means for clustering? Have you tested how sensitive your results are to the clustering algorithm, and could this choice be inflating variance reduction?", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n\n- The paper addresses an interesting gap in the literature. People use similarity metrics to filter and select pretraining data. However, these metrics are rarely designed or evaluated with this specific setting in mind. \n\nThe authors propose a clear framework for assessing embeddings that is well-suited to the pretraining context. \n\n- Some of the results are surprising. Very simple embeddings derived from the training corpus outperform more complex, pretrained models.\n\n$\\textbf{Weaknesses}$\n\n- The strongest claims rely on the idea that low embedding distances mean similar pretraining loss, but the actual evaluation uses variance within clusters, not pairwise distance. This does not prove that distance correlates with loss difference.\n\n- Figure 1 is suggestive, but the paper does not include full summary statistics. It is hard to tell how consistent the variance reduction is across cluster sizes or training checkpoints.\n\n- The use of balanced K-means may inflate the observed variance reduction. There is no direct comparison to random or unbalanced clustering, or to distance-based correlation.\n\n- Section 3 is hard to follow. The experimental setup is long, and results are sometimes buried. It would help to clearly state the goal and takeaway for each experiment.\n\n- The comparison between off-the-shelf and specialized embeddings feels a bit unfair. The specialized embeddings are pulled from models trained on the same data. That likely explains some of the improvement.\n\n- The efficiency argument is not convincing. While some of the embeddings require no forward pass, they still rely on a trained model. That training cost is not addressed.", "was_revised": true, "final_justification": "Given the rebuttal, I still believe the paper should be rejected. While the authors clarify their choice of cluster-based variance over pairwise correlation, this still does not directly demonstrate that embedding distance captures the training dynamics they claim. The defense of balanced K-means addresses stability but not whether it inflates results, and the generalization concern remains; the proposed approach assumes retraining an embedding model for every new corpus, which is impractical in many settings. Overall, the evaluation is limited, and the framing overstates the generality and efficiency of the method.", "rebuttal": "We thank the reviewer for their detailed feedback! We appreciate that you find that our method addresses an \"interesting gap in the literature\" and is \"well-suited\" to the pretraining context. We respond to individual comments below:\n\n> **\"The strongest claims rely on the idea that low embedding distances mean similar pretraining loss, but the actual evaluation uses variance within clusters, not pairwise distance. This does not prove that distance correlates with loss difference.\"**\n\nWe would like to clarify the setup for our variance reduction experiments. Our goal is precisely to quantify what the reviewer highlights — whether examples that are close in embedding space also exhibit similar pretraining loss.\n\nSince we cluster in the embedding space, each cluster now contains groups of examples that are all similar to each other. Then, for every cluster, we measure the training loss for all points in the cluster and take the variance of these losses  to quantify the similarity in the losses of these points that are also similar to each other in the embedding space.\n\nOf course this is for a single cluster, and so we report the average of this variance across all clusters, normalized by the overall loss variance in the dataset, which is our variance reduction ratio.\n\nWe would also like to note that we can only reliably test similarity in this one direction (as our approach does) --- that similar examples have similar training loss. The **converse is not necessarily true** --- two very different examples can have the same loss because they happen to be equally \"easy\" or equally \"hard\". This **asymmetry** is an important reason we chose our cluster-based approach rather than pairwise correlation. We will highlight this more clearly in our revision.\n\n> **\"Figure 1 is suggestive, but the paper does not include full summary statistics. It is hard to tell how consistent the variance reduction is across cluster sizes or training checkpoints.\"; \"complete results for the variance reduction analysis, such as how it changes across different cluster sizes, seeds, or pretraining checkpoints?\"**\n\nWe agree that Figure 1 serves primarily as a qualitative illustration. However, Figures 2 and 3 present **exactly these requested summary statistics** — showing variance reduction trends across different **cluster sizes** (Figure 2) and **pretraining checkpoints** (Figure 3). \n\nWe find the relative performance of different embeddings is consistent across both axes. This supports the robustness of our conclusions.\n\n> **\"The use of balanced K-means may inflate the observed variance reduction. There is no direct comparison to random or unbalanced clustering... Why did you choose balanced K-means for clustering?\"**\n\nWe chose balanced K-means to ensure a fair and stable comparison across different embedding models. Without balancing, clustering can yield singleton clusters or overly large groups, leading to unstable variance estimates and increased noise. Balanced clustering improves comparability across embedding models with differing geometric properties. A random clustering would serve as a lower bound, achieving variance reduction ≈ 1 by design (which is far worse than the considered embeddings' clusters). \n\nOverall, this choice of balanced clustering does not inflate the results in a biased way, as all methods are evaluated under the same balanced clustering scheme, making their relative performance directly comparable. \n\n> **\"Section 3 is hard to follow. The experimental setup is long, and results are sometimes buried. It would help to clearly state the goal and takeaway for each experiment.\"**\n\nThank you for the helpful suggestion. We will revise Section 3 to clearly present the goal and takeaway for each evaluation metric to improve clarity and structure.\n\n> **\"The comparison between off-the-shelf and specialized embeddings feels a bit unfair. The specialized embeddings are pulled from models trained on the same data. That likely explains some of the improvement. How do you address the potential circularity in using embeddings from models trained on the same data used in the evaluation? Would these embeddings generalize to new domains or corpora where no model has yet been trained?\"**\n\nThis is a fair point — and one we agree with. In fact, one of our core arguments is that embedding models should ideally be trained on the same data used for pretraining. A key finding in our paper is that even small proxy models trained on the same corpus (e.g., 200M parameters) can outperform large off-the-shelf models. This is not standard practice today (see [1,2]), and we argue that it should be.\n\nThe notion of generalizing to new domains is somewhat orthogonal to our main use case, as we could always use such data to train a new embedding model. And since the embedding models are small and much faster to train, you could simply train this embedding model when you obtain data from new domains.\n\n> **\"The efficiency argument is not convincing. While some of the embeddings require no forward pass, they still rely on a trained model. That training cost is not addressed.\"**\n\nWe agree that a trained model is required, but the associated cost of training that model is modest. The model is small (200M parameters) and trained on only a 20% slice of the full dataset. Compared to training a 1.7B model on 200B tokens, this is a negligible fixed cost. Moreover, this cost is a one-time investment, which is amortized across all embedding use cases, and this slight marginal cost is easily offset by the significant improvements in data curation quality. We will clarify this point in our revision.\n\nThank you again for your detailed review. We hope these clarifications address your concerns and help reinforce the value of our contributions.\n\n[1] Vo, et. al. Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach.\n\n[2] Chang, et. al. Data Curation Alone Can Stabilize In-context Learning."}, {"review_id": "IYj8HAURxq", "reviewer": "Reviewer_Q6cD", "summary": "* This work focuses on in-depth evaluation of how text embedding based data curation can improve LLM pretraining.\n* There are two overall concepts for data curation presented in this work:\n  * When representation similarity is desired: finding examples that are similar to hard examples, or high quality examples.\n  * When representation similarity is not desired: finding diverse examples to remove training data redundancies.\n* The three part evaluation that this work presents involve: loss-correlation test, diversity-based curation test and cluster-purity test. These metrics are closely tied to pre-training objectives versus classical embedding model task evaluations.", "questions": "1. Recent embedding models such as e5-mistral and nv-embed support task prompts (clustering, semantic similarity etc.). Even more recent models such as BGE-EN-ICL support few-shot in-context learning ability. How could those abilities be leveraged to customize the representations for the various tasks (when the desired similarity should be based on problem difficulty vs. text quality vs. topic).\n2. \"LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders\" could be referenced to justify why LM Output Embeds in Table 1 works well.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Experiments are rigorous, and methodology is sound. Especially the three tests, downstream task performance, and coverage of various possible embedding models and direct-from-LLM representation approaches.\n2. Data curation is an important aspect of pretraining and rigorous studies such as this work is helpful for the field.\n3. The framing of embedding models in-relation to pretraining objectives (instead of classic MTEB benchmark tasks) is original.\n\nWeaknesses:\n1. The empirical results (downstream task performance) are modest (within 2%).\n2. Only diversity-based curation is ultimately linked to downstream task performance with results. There is no guidance or results presented on how loss-correlation and cluster-purity tests can be leveraged to improve downstream task performance.\n3. Extending the evaluations beyond the single corpus and model size would make the study more rigorous, but I understand that it is very computationally expensive.", "was_revised": true, "final_justification": "I thank the authors for taking the time to give detailed responses to each question. I agree with the author's responses and there are no unresolved issues in particular. I stand by the original score of 4 and would reserve a score of 5 or 6 for if the work has demonstrated even higher downstream task performance and implemented the guidance in their response for loss-correlation and cluster-purity tests to show further improvements in downstream task performance or data efficiency.", "rebuttal": "We thank the reviewer for their positive and constructive feedback. We are glad you found our experiments \"rigorous,\" our methodology \"sound,\" and our framing \"original\" and \"helpful for the field.\" We address your points below.\n\n> **\"The empirical results (downstream task performance) are modest (within 2%).\"**\n\nThis is a fair observation. While a ~2% gain may seem modest, we would like to emphasize two points. First, this is an average improvement across **23 downstream tasks**, indicating a consistent and broad enhancement of the model's capabilities rather than a narrow gain on a single benchmark. Second, and perhaps more importantly, this performance gain can be viewed through the lens of **data efficiency**. Our curation method allows the model to reach a target performance level using significantly fewer pretraining tokens compared to random sampling, directly translating to substantial savings in compute and time. We will add this data efficiency perspective to our revision to better contextualize the impact of our results.\n\n> **\"Only diversity-based curation is ultimately linked to downstream task performance with results. There is no guidance or results presented on how loss-correlation and cluster-purity tests can be leveraged to improve downstream task performance.\"**\n\nLoss correlation can be directly tied to target-specific sampling schemes (which we state in lines 149-151), but we are happy to expand upon this and provide more concrete guidance in a potential application. The variance reduction metric (Section 3.1) captures how well an embedding reflects training difficulty as measured by pretraining loss. Embeddings with high variance reduction can be used to group examples by difficulty, which directly informs a family of sampling schemes known to improve pretraining efficiency and performance—e.g., importance resampling or dynamic curriculum methods (as in Xie et al. 2024, Jiang et al. 2024).\n\nOne such method is an online training strategy, where at each step, you can adaptively select the next cluster to train that has the largest pretraining loss. With embeddings that produce clusters with greater variance reduction, this means that one can run a single forward pass on the cluster centers (rather than all datapoints) and use those values as more accurate estimates of cluster loss. \n\nWith cluster sizes of 50–200, this can yield 50–200× savings in selection overhead compared to per-example loss estimates. Our framework thus enables **scalable and efficient proxies for loss-aware data selection**.\n\nAs for cluster purity, it serves as a useful and interpretable diagnostic: embeddings that align with human-curated data sources (e.g., Wikipedia vs. code) are more interpretable and trustworthy for **building specialized datasets or controlling data mixtures**. We envision this guiding domain-aware curation, where users seek to curate or balance datasets along interpretable axes. We will include both of these examples in our revision.\n\n> **\"Recent embedding models such as e5-mistral and nv-embed support task prompts (clustering, semantic similarity etc.). Even more recent models such as BGE-EN-ICL support few-shot in-context learning ability. How could those abilities be leveraged to customize the representations for the various tasks (when the desired similarity should be based on problem difficulty vs. text quality vs. topic).\"**\n \nThat's an excellent point. Promptable and in-context embedding models offer exciting possibilities for tailoring similarity to task-specific notions—such as difficulty, redundancy, or style. Our evaluation framework would be an ideal tool for assessing how effectively these prompts can steer the embeddings to capture these different notions of similarity. We will add this discussion on embedding models that can leverage in-context learning as a promising future direction in our discussion.\n\n> **\"LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders\" could be referenced to justify why LM Output Embeds in Table 1 works well.\"**\n\nThanks for this pointer! We agree that LLM2Vec provides strong, complementary evidence for our finding that embeddings from decoder-only models can be highly effective. We will add this citation to our related work section in the revision.\n\nThank you again for your detailed review. We hope these clarifications address your concerns and help reinforce the value of our contributions."}]

Author rebuttal:

Meta-review: 1. **Summary**
   The paper introduces a framework for evaluating similarity metrics used in pretraining data curation for large language models. The framework consists of three criteria: (a) correlation between embedding distance and pretraining loss, (b) downstream performance when embeddings guide diversity-based data curation, and (c) cluster purity with respect to human-curated data sources. Experiments use The Pile and a 1.7B parameter UL2 decoder-only model trained on 200B tokens. Results show that off-the-shelf embeddings (e.g., USE, BERT, Gecko) underperform compared to simple embeddings derived from models trained on the same corpus (LM Token Embeds, LM Output Embeds). The work highlights that task-aligned and domain-specialized embeddings are more effective for data curation than generic ones.

2. **Strengths**

   * Addresses an underexplored but important problem: evaluation of similarity metrics for pretraining data curation.
   * Provides a clear, three-part evaluation framework grounded in pretraining dynamics rather than retrieval benchmarks.
   * Experimental validation is large-scale (1.7B model, 200B tokens), lending weight to conclusions despite computational challenges.
   * Results are surprising and practically relevant: trivial embeddings (averaged input embeddings) often outperform sophisticated off-the-shelf models.
   * Strong clarity of motivation: moves the field toward more principled embedding evaluation for data selection.

3. **Weaknesses**

   * Gains on downstream tasks are modest (about 2% on average), which several reviewers flagged as insufficient to justify strong claims (Q6cD, VPjN).
   * Loss-correlation and cluster-purity evaluations are not directly tied to downstream improvements, and the authors did not provide new evidence during rebuttal that these metrics translate to stronger downstream performance (Q6cD).
   * Evaluation restricted to The Pile and a single model size, raising questions of generality that remain unaddressed (qKM8, VPjN).
   * Concerns about circularity: embeddings trained on the same data naturally outperform off-the-shelf models, which undermines the claim of broad applicability (VPjN). The rebuttal acknowledged this but insisted it should be standard practice, which did not fully resolve the critique.
   * Methodological clarity issues remain: variance reduction as a proxy for distance-loss correlation was contested (VPjN), balanced K-means potentially inflates results (VPjN), and although the authors defended these choices, no additional quantitative validation was provided
   * Presentation issues (dense experimental section, limited clarity on clustering choices and robustness). The authors promised revisions but no evidence could be evaluated at review time

4. **Decision Rationale**

   While the framework is well-motivated, the empirical evidence does not convincingly demonstrate that embedding distance reliably captures pretraining dynamics, nor that the proposed evaluations generalize beyond the tested setting. Critical reviewer concerns about robustness, generality, and modest impact were not fully addressed in the rebuttal, and at least one reviewer maintained a rejection on these ground. 

5. **Reviewer Discussion and Rebuttal Dynamics**

   * Reviewer CAUd initially raised concerns about presentation and redundancy reduction. The authors promised schematic figures and clarified the role of redundancy reduction and the impact of using smaller models for embeddings. This response was satisfactory.
   * Reviewer qKM8 requested runtime and FLOP details, robustness to clustering thresholds, and applicability to code-heavy corpora. The authors provided detailed runtimes and clarified robustness and applicability. The reviewer maintained an acceptance.
   * Reviewer VPjN raised stronger concerns regarding variance reduction as an indirect proxy for distance-loss correlation, potential inflation from balanced K-means, and limited generality of in-domain embeddings. The authors clarified their methodology, emphasized comparability across embeddings, and argued that training small proxy models is inexpensive. Nevertheless, this reviewer maintained a borderline reject.
   * Reviewer Q6cD considered the methodology rigorous and original but noted modest improvements and limited linkage of metrics to downstream performance. The authors reframed results in terms of data efficiency and clarified how loss-correlation and cluster-purity could inform sampling strategies. The reviewer kept a borderline accept.

Given the balance of reviews, the critical weaknesses, and the lack of resolution during the rebuttal period, I recommend rejection.
Decision: Accept (poster)