LMD3: Language Model Data Density Dependence
Authors: John Kirchenbauer
Garrett Honke
Gowthami Somepalli
Jonas Geiping
Katherine Lee
Daphne Ippolito
Tom Goldstein
David Andre
Venue: COLM (2024)
Keywords: data attribution, retrieval, embeddings, training data, density estimation
TL;DR: We develop a methodology for analyzing language model task performance based on training data density estimation.

Abstract: We develop a methodology for analyzing language model task performance at the individual example level based on training data density estimation. Experiments with paraphrasing as a controlled intervention on finetuning data demonstrate that increasing the support in the training distribution for specific test queries results in a measurable increase in density, which is also a significant predictor of the performance increase caused by the intervention. Experiments with pretraining data demonstrate that we can explain a significant fraction of the variance in model perplexity via density measurements. We conclude that our framework can provide statistical evidence of the dependence of a target model’s predictions on subsets of its training data, and can more generally be used to characterize the support (or lack thereof) in the training data for a given test task.

Published as a conference paper at COLM 2024
LMD3: Language Model Data Density Dependence
John Kirchenbauer"1 Garrett 2onke*2
Gowthami Somepalli' Jonas Geiping", Daphne Ippolito Katherine Lee
Tom Goldstein', David Andre²
University of Maryland, 2x, the Moonshot Factory
$ELLIS Institute Tübingen, *Carnegie Mellon University
MMPI-IS, Tübingen Al Center, Google DeepMind
Abstract
We develop a methodology for analyzing language model task performance at the individual example level based on training data density estimation. Experiments with paraphrasing as controlled intervention on finetuning data demonstrate that increasing the support in the training distribution for specific test queries results in measurable increase in density, which is also a significant predictor of the performance increase caused by the intervention. Experiments with pretraining data demonstrate that we can explain a significant fraction of the variance in model perplexity via density measurements. We conclude that our framework can provide statistical evidence of the dependence of a target model's predictions on subsets its training data, and can more generally be used to characterize the support (or lack thereof) in the training data for a given test task.
1 Introduction
"With the right dataset and the right benchmark, you can make a deep learning model fake absolutely any ability. It's easiest if the benchmark and the dataset are the same thing, but otherwise you can also achieve with a dataset that a dense sampling of the distribution the benchmark is sampled from." " @fchholee
The goal of this study to perform a careful series of experiments designed to provide concrete evidence for, or against, the hypothesis that the accuracy of model on test question is strongly impacted by the density of the training set around that question. To do this, we explicitly estimate probability density function on the training distribution. We then ask whether it is possible to explain a significant amount of a language model's abilities by directly estimating how dense the training data distribution is at each test point. We quantify the density of training data in embedding space using a classic technique in statistics, Kernel Density Estimation (KDE), which relies measurements of the similarity between a test point and its neighbors. We present an illustration of the general process in Figure]
To determine whether such a simple technique offers any explanatory power in the realm of modern large language models, and to assess the impact of density on test performance, we begin by manipulating density through contamination of training dataset with copies (or near copies) of test samples. We then observe the correlation between accuracy on test samples and the training density near those samples. We confirm that 1) a properly configured kernel density estimate can detect the increased density near these training points and 2) the elevated performance is reliably predicted by the density estimate value for the question texts.
We then study the natural density variation within pretraining corpus and uncover more complex and subtle downstream effect. Higher density test points have lower perplexity ("as expected") when density is estimated using only the nearest neighbors in the pretraining corpus for each query, although the effect size is small. However, when estimating density using a random subset of training points, we observe the opposite correlation.
1

---

Published as a conference paper at COLM 2024
Query
How... Who
1 KDEg N Σ K(q.x)
Embedding model
Text data
Embeddings
* quary
corpue
Corpus
Low density
High density
Figure 1: A system level view of the LMD3 pipeline. The corpus of data used to train LLM and test set queries are projected into a vector space using a neural embedding model. For each resulting query vector, a density estimate with respect to the training corpus is computed. The resulting density estimates can be used to infer the model's ability to respond to a question-like query or simply reproduce the tokens in the query sequence based on whether the relative density is higher or lower at that point in sample space.
We interpret our findings within the context of wide array of prior work on data attri- bution, data contamination, and the and conclude that simple measures of dataset density are indeed sharp enough tools to be predictive of per-sample test set performance. At the same time, we observe that-at least at the 7B scale where models do not dramatically interpolate their data the effect of dataset leakage can be surprisingly small. We wrap up with a discussion of applications of our methodology to instance and group-wise error analysis, dataset filtering and supleementing interventions, and the determination of test contamination from the training corpus.
of nigh-ddmeessioal statistics,
Related Work
2.1 Memorization and Contamination
Large language models have been shown to memorize sequences from their training data. The key, replicated finding across this literature is that the more a sample is repeated in the training data the more likely it is to be memorized (Carlini et al. 2022; Hernandez et al., 2022). Repetition is an important extremal case of elevated density corresponding to a spike in the density function at specific point in the sample space and such we expect density based analysis to highlight the causal relationship between repetition and elevated sampling likelihood. Additional work has shown that similar relationship holds for fuzzy or near-duplicated samples (Ippolito et al. 2022) implying that a density measure might also capture this weaker dependence relation.
Recent work has studied the impact of data contamination during the pretraining and finetuning processes. Despite the fact that training solely and repeatedly on verbatim or paraphrased test samples leads to significantly elevated performance (Yang et al.) 2023), the impact can be much less pronounced in realistic scenarios where leaked samples are mixed into larger corpus during pretraining (Jiang et al.) 2024). Since leakage of test queries constitutes a practically relevant instantiation of the atorementioned edge case of spikes in the training distribution, we design our initial set of experiments around leakage intervention.
2.2 The "Data Attribution Hypothesis"
Stated informally, the "data attribution hypothesis" is the belief that specific machine learning model behaviors can be attributed to specific set of examples from the training data. The critical implicit assumption that the set of training points the behaviors are attributable to is small with respect to the overall size of the training corpus. Therefore, work on data attribution seeks to design methods for identifying the set of most relevant examples, with notable approaches including the use of optimization related measures (gradients) to develop "influence functions" (Koh & Liang 2017), behavioral approximations of model outputs through the lens of kernels (Park et al. 2023), and even exhaustive models of
2

---

Published as a conference paper at COLM 2024
how model behaves with respect to leave-one-out resamples of its training dataset (Ilyas et al., 2022). To benchmark these techniques, curated datasets of queries and corresponding fact sets to which correct responses could/shoull be attributed have also been developed (Akyürek et al. 2022). In this work, we take an orthogonal approach by both relaxing the attribution hypothesis' assertion that the relevant set of training data for each query must be small, and switching away from relying on characteristics of the final model and instead directly analyzing the training corpus itself.
2.3 The "Similarity Hypothesis"
Quite related to the data attribution hypothesis is the "similarity hypothesis" near truism that models will make predictions that are similar to their training data. While prior work has shown that structured features of training data such the elevated occurrence of specific named entities correlates with performance on factual test questions concerning those entities (Kandpal et al. 2023), similar results involving more general notions of train- test interrelatedness are lewer and far between (at least in the modern LLM literature). That said, recent work such as Tirumala et al. (2023) has shown that density quantification can be used to guide dataset pruning routines for improved sample complexity during model training. Also published contemporaneously to our research activities, Yauney et al. (2023) showed that wide variety of similarity metrics were "not enough to explain performance across benchmarks". While certain aspects of our results do suggest that the story is subtle, rather than simple, in contrast their rather definitive conclusions, we posit that significant variance in LLM performance can be explained given the right measure and lens of analysis.
Preliminaries: Kernel Density Estimation
Density estimation is the general problem of estimating probability density function from data. In our setting, we aim to estimate a distribution that is intractable, namely the function P Rd + IR, where for our domain we have represented all of the points E X from our training data as d-dimensional vectors, and we would like to approximate the likelihood of drawing any given sample from the distribution over natural language sequences represented by our corpus. The tool we choose to use for this approximation the Kernel Density Estimate.
Definition 3.1. Kernel Density Estimate (KDE) For a training corpus X; = {xo, XX,,, 1) Rd with bandwidth parameter h > 0 and kernel function K. : IR x Id + IR, for query vector xg the KDE at xg over X. denoted KDEx, (xg) is given as:
KDEx, (x9) = X
Σ (x, xEX.
However, because the sum is over all и training samples, the cost of computing the KDE for realistic language model training datasets is intractable. To apply KDE in this setting, we turn to DEANN Density Estimation from Approximate Nearest Neighbors, a clever optimization proposed and analyzed by Karppa et al. (022) that enables the scaling of KDE to large datasets X, by decomposing the full KDE into the exact contributions of close neighbors, and approximate contributions from the rest of the corpus. A formal definition of the approximation algorithm that we use when computing KDEs on pretraining-scal datasets is provided in Algorithm 1 We discuss the details of the approximation scheme further in Appendix A.1
4 The LMD3 Methodology
For a chosen data point (text sequence) we want to study how the dataset density at that point is related to model performance at that point. We measure performance "task-like" text sequences using model accuracy, and on unstructured or "webtext-like" data using the loss function. Our method takes input model and its training data, as well as one
3

---

Published as a conference paper at COLM 2024
or more test fest/queet query sets of interest. The training data embedded using embedding model, and the nearest neighbors for each query sequence are retrieved from the vectorized corpus. Then, either, the neighbors are combined with random subset of corpus vectors to compute a density estimate at each query point, or, when scale permits, the density estimate is computed exactly at each query point.
4.1 Computing Embeddings
We take for granted access performant, pretrained neural sequence embedding models as general purpose similarity/ /dssancce functions for pairs of text segments. We consider two types of embedding spaces in which to perform density estimates, the features produced by an off-the-shelf refrieval model and the features derived from the hidden states of the model under analysis. While in preliminary investigations we explored working with the latter "self- "self-derived" features, our small scale (fine-tuning) experiments, and so we stick with the former for large-scale experiments. The use of L.M hidden states retrieval features is active area of study and future versions of our methodology could benefit from those results, so we leave a more comprehensive evaluation of exactly what self-derived LLM embeddings represent that general purpose retrieval models do not to future work
failed to find significant difference between the two in
4.2 Computing the KDEs
We utilize the software package developed by the DEANN authors (Karppa et al. 2022) as an extremely efficient parallelized implementation of the exact KDE calculation for eucl euclidean kernels when the data (Xq, Xc) fits in memory. Since bandwidth selection is an empirical process, we perform an initial evaluation across range of bandwidths in preparation for our paraphrase experiments (Appendix (opeeddi(A.10)....
For the finetuning-scale experiments, where the corpus contains ~ 100,000 samples, able to directly use the exact version of the KDE. For experiments with a pretraining dataset, leverage the authors' exact KDE implementation well as their random sampling based KDE approximation as subroutines. Algorithm describes how we combine this with scalable neighbor search system to implement our "ullyy decomposed" version of the nearest neighbor search-based approximate KDE. "Fully decomposed" simply means that before running the KDE calculations themselves, as preprocessing step for the query set, the neighbors for each example are retrieved from the training corpus via a massively distributed vector search engine (exactly, in our case, as opposed to approximately as suggested by DEANN).
P Then, the random component for each density calculation selected in hierarchical manner via initial large random sample without replacement, and second sampling step, also without replacement, to select the final random complement for each query's 's individual density estimate. In Section 5.3 and Section "KDE" refers to the exact KDE computed over the entire finetuning corpus, while in Section 5.4 and Section 6.2 it refers to the final value yielded by Algorithm T- the weighted average of the local and random components of the KDE approximation over the pretraining corpus, except where denoted specifically as solely the "local" or "random" component.
We also remark that running multi- billii-illion parameter embedding models over billions of sequences costly and therefore such models are not as amenable to the pretraining scale experiments we ultimately perform.
WWill the details of the neighbor search system we in this work for the preiraining scale experiments are proprietary, we include the finetuning-see implementation our code. The neighbor search system can substituted with any vector search engine that scales to the data under analysis, even if that scale necessitates an approximation scheme such those implemented in the FAISS library (Johnson et al. 2019). Karppa et al. 2002) argue that this additional approximation does not bias the overall KDE approximation results any further.
4

---

Published as a conference paper at COLM 2024
Algorithm 1 Fully Decomposed Approx. KDE with Hierarchical Sampling
Input: A corpus X of n text embeddings E RR a nearest neighbor search subroutine over vectors in Xe, (N)(-), a kernel function K and bandwidth parameter I together with the corpus over which is computed X, defining KDEx(.), two random sample size parameters my and m2 (M22 my < n), dataset of query embeddings x Fq Rd.
X4 Output: Zp. E R50 an approximation of the KDE for each xq E Xq.
Randomly sample w/o replacement X1 of size my from X
for all Xy X do
Xpn +- (xg E Rkxd
Xi = fx X1|x & }
Randomly sample w/o replacement X2 of size m2 from X1.
Znm t- KDEx mm (xq)
Frand f- KDEx, (Xq)
Zq ==)m
(1= Frand
end for
(Note that we can return Zm and Zyand individually analyze the effect of local and global contributions independently.)
Experiments
5.1 Models and Data
transformer-basse sequence embedding model from the library (Reimers & Gurevych 2019) to generate the feature space in which compute density estimates. For the controlled finetuning experiments we train the base Llama 7B model (Touvron et al. 2023) on doctored versions of the training set for the MMLU benchmark for 2 epochs this regime produces meaningful differences between experimental settings whilst still being realistic (and resulting in a model that has not completely degenerated in other regards). For the pretraining experiments we analyze the performance of the 6.9B model from the Pythia suite (Biderman et al. (2023) as a function
For our experiments
centence-trrassfore
of version of training corpus, The Deduplicated Pile (Biderman et al. 2022) We provide
further details on these choices and experimental parameters in Appendix A.2
5.2 Paraphrasing Process
In order to develop our density methodology, design a synthetic experimental setting in which we can directly tune the specific amount of support for test queries present the training data. Roughly defined, an idealized paraphrase of a test example is the transform from natural language sequence x to x' such that the semantics of x and x' are equivalent. Access to such transformation function allows us to perform controlled experiments where we take test sample x and intervene to increase the amount of specifically relevant training data for this sample by mixing paraphrases x' into the training data. This can be done in number of ways, and we discuss the design choices for this process further in Appendix A.3
5.3 Controlled Experiment 1: Leakage to Increase Density, Finetuning Scale
In our first series of experiments, we take 1, 000 random questions from the MMLU test set of 14, 042 questions and paraphrase them 3 times. We also consider an exact copy of the selected test questions as "paraphrase" with perfect similarity, or distance 0 to the original query. For each test question we paraphrase, we compute the cosine similarity between the original query and each sample in the set of paraphrases based their embeddings. Considering the paraphrases for each query sorted ascending order by similarity to the original query Para 1, Para 2, and Para respectively, we then refer to the collection of
5

---

Published as a conference paper at COLM 2024
two paraphrases {Para Para 2) as Paras 1,2 and the collection of all three paraphrases {Para 1, Para 2, Para 3, } as Paras = 1,2,3.
Then finetune our base language model datasets constructed by taking the "auxiliary train" split of the MMLU dataset and mixing in various combinations of the paraphrases and/or exact copy of the original query, which we denote the presence of, lack thereof, using Exact = and Exact = 0 respectively. We evaluate the performance of the models trained on these different datasets and analyze the impact that the inclusion of paraphrases and exact copies of test queries has on the trained model. "Rank Accuracy" denotes the scoring method used by the HuggingFace Open LLM Leaderboard for evaluating MMLU, first proposed by Brown et al. (2020).
5.4 In-the-Wild: In and Out-of-Distribution Queries, Pretraining Scale
Finally, we turn our focus to the loftier goal of relating the test time behavior of language model to its pretraining data. In addition to the increased scale with which we apply the technique, the increased generality of the pretraining distribution requires us to curate sets of queries to represent different test time scenarios we might be concerned with. We consider two classes of query sets: in-distribution (ID) and out-of-distribution (OoD), where for the former we choose the simple, explicit definition of any query taken directly from the training corpus X as ID, and consider anything else to be OoD. The training corpus is The Deduplicated Pile, re-segmented into ~ 3.5B individual text sequences. For these experiments we compute the approximate KDE using parameters k = 1, 000 nearest neighbors for the local component, Mi 1, 000, 000 samples base random sample set, with M12 = 10, 000 samples per query selected as the random complement, exclusive to the k nearest neighbors.
Random 10k segments (ID): We randomly sample, without replacement, 10, 000 segments from the training corpus itself. These are drawn from precisely the same set of segments over which neighbor retrieval and KDE computation are performed. Since these are simply webtext segments, there no notion of "response" so we just compute perplexity (PPL) of the text segment under the LLM.
MMLU Test (OoD): We employ the same 14, 042 test questions as in the finetuning experi- ments to act as set of queries that are not contained in the training data. Since there are ground truth responses for the queries, we can compute perplexities on both the query texts as well as on the correct target response when conditioned on the query text.
OpenOrea Random 10k (OoD): As another set of diverse sequences outside of the train distribution, we consider random selection of 10, 000 questions from the OpenOrca cu- ration project's main dataset. Since these also have reference responses, we can compute perplexities on both the query and responses for these samples.
6 Results
Considering the relative novelty of our proposed methodology and in order to de-risk the main experiments, we performed series of validation analyses regarding the embedding model used for the paraphrasing experiments, the bandwidth hyperparameter of the KDE, and basic series of finetuning runs calibrate our expectations about exactly how much model should overfit when trained repeatedly on verbatim leaks of test questions. We dis- cuss these preliminary experiments in section Appendix A.10 For the main experiments we consider bandwidths of (0.01, 0.05, 0.1, 1.0) for the exponential kernel, and (0.1, 0.5, 1.0) for the gaussian kernel though we only report a small selection across the main body figures, primarily gaussian at 0.1 experiments and 0.5 as a "wide" setting for capturing a broader range of similarities in the pretraining-scal experiments.
"narrow" setting tuned for discrimination in the leakage

---

Published as a conference paper at COLM 2024
0.62
0.60
Rank Accuracy
0.58
Rank Acnuracy
0.56
0.54
0.54
0.50
0.50
0.0297
0.0299 0,0300 0.0301 0.0302 0.0303
"Bffective" Epnch
KDE
Paras
Paras
Paras = 1.2
Paras 1,2,3
Exact
Ikact
Avg. for Non-Leaked Q's
Figure 2: Left) To enable the aggregate interpretation of the paraphrasing experiments, we plot accuracy leaked test questions a function of the number of "effective epochs" the model has trained on the leaked questions for. Right) We plot the same performance measure as function of KDE, with gaussian kernel and bandwidth 0.5. We see that the trend in accuracy according to our density measure corresponds with the trend in accuracy according to the known degree of leakage, effective epochs.
6.1 Controlled Experiments: "Expected" Dependence between Performance and Density.
In Figure we demonstrate that training on the various leaky dataset formulations improves performance by visualizing changes in performance across different leakage settings. We enable this by first plotting performance against a handcrafted feature called "effective epochs", where we compute the number of epochs the model has effectively trained on a given test question X; based on the specific set of paraphrases and exact duplicates Xp included in the training set. For simplicity, in this figure, we represent exact copies and paraphrases equally such that if we train for 2 epochs, and have leaked 1 exact copy and paraphrase, the total number of times the test question is trained on would be 2 (1 ++++ 4. In Figure we observe both positive trend in performance as a function of effective epochs as well as positive trend function of our KDE measure- the latter of which is computed without access to either ground truths or any outputs of the LLM being analyzed.
In Figure we present accuracies and densities across the full factorial manipulation of exact and paraphrased leaks. The figure shows that the KDE is a discriminative feature between the leak set and non-leak set. To measure the reliability of the correspondence between data density and performance, we perform mixed-effects regressions and discuss these results in detail in Appendix A.5 In summary, we find that the leakage conditions reliably increase accuracy (Exact leak: p < .001, Paraphrased leaks: p < .001) and decrease perplexity (Exact leak: p < 0001, Paraphrased leaks: p < .001) on the test questions. Critically, training data density estimates also reliably predict variance in accuracy and perplexity where as density increases, accuracy increases (p < .001) and perplexity decreases (p < .001).
6.2 In-the-Wild Experiments: Aggregation-Speciff Dependence Relationship.
First considering set of random samples from The Deduplicated Pile itself, ID with respect to the model's training data, in Figure 4 we plot perplexity as a function of KDE by binning the data by KDE values (x-axis) and computing the average accuracy (y-axis) within each bin. In the left chart we see that when measuring the KDE with respect to only the random samples selected in our approximation algorithm we get a noisy, slightly positive trend (which is surprising P. However, when we consider the KDE computed only with respect to the nearest neighbors in the corpus, we get sharp negative trend, in line with results in the
3 b...but potentially explainable by distances in high dimensions being counterintuitive. Being closer on average to collection of random other points effectively means being close to nothing in particular.
7

---

Published as a conference paper at COLM 2024
son leaked Q's
Jeaked Q's
Exact Paras
Exaci
Paras
Exact Pares 1,2
Exact Paras 1,2,3
15
15-
15-
15
Count
10
5-
5-
0.75
1.00
0.50
KDB
Baak Accursty
0.60
0.00
0.00
0.60
0.55
0.55
0.55
0.50
0.50
0.50
0.50
leaked
Leaked
lealed
leaked
leaked
Ieaked
non-leaked
leakeed
(a) Experiments where exact copies of test questions included, only paraphrases. For "Count" histograms, the y-axes are cropped to [0, and x-axes to [0, 10-$1 "] to showcase differences between non-leaked and leaked. An uncropped version is provided in Figure 7
Exact
Paras
Exact
Paras
Exact
Paras 1,2
Exact 1 Paras 1,2,3
750
750
750
750
Coust
500
500
250
250
250
250
0
0
0.5
Rank Aecuraty
0.60
0.55
0.55
0.55
0.55
0.50
0.50
0.50
0.50
non-leaked Q's leaked Q's
lealee
leaked Q's
nun non-leaked Q's
leaked Q''s
non-leaked
leaked
(b) Experiments where exact copy of test questions included, in addition to paraphrases. Note that axes for "Count" histograms are uncropped in this panel.
Figure 3: Moving from left to right in (a) and (b) shows the effect of an increase in the number of paraphrases of each test question that are leaked into the training data. Experiments in (b) also include an exact copy of each leaked test question, while experiments in (a) do not. "Count" histograms) We plot the distributions of KDE values (gaussian kernel and bandwidth 0.1) for the test queries that were leaked, exactly and or via paraphrase, and not leaked, for each leakage intervention experiment. Accuracy bar charts) We show the corresponding accuracy breakdown for the leaked and non leaked sets for each experiment. Overall, we find that increasing support for test questions via incorporating paraphrases those test questions, and this increase is
into the training data increases performance magnified by the addition of exact leaks of test questions. The addition of the exact copy of each questión also makes the leaked and non-leaked question sets highly separable under our KDE measure as demonstrated by the distinct concentration of "leaked Q" KDE values away from 0.0 in (b).
finetuning setting This is affirmed by the trend relating each query's average distance to its top k neighbors to model perplexity on said query. (Note that x-axes are reversed the distance charts to make the trend visually congruent with the handedness of the trends for density measurements Figure and Figure
Next, switching to set of OoD queries, the MMLU test set, the left chart of Figure 5 shows that the PPL on the query/ query/question quessioo texts is not strongly correlated with the local KDE component, but the middle chart shows that it is correlated with the average distance to the top k neighbors in the corpus. Further, we see that the perplexity of the correct response is also correlated with distance the top k neighbors.
*We report this random - local breakdown explicitly here because at pretraining scale, the coefficient used to combine the local and random components according to Algorithm k/N versus (N k)/N respectively, causes the random component completely wash the local contribution.

---

Published as a conference paper at COLM 2024
8
Ohers PPL
20
28
Query
15
10
0.19
0421
0.222
0,233
0.24
6023
0.45
Rendom Component
KDE Local Componest
10 Neighhor Distance
Figure 4: Perplexity according to Pythia 6.9B for random samples from The Deduplicated
Pile (ID) function of KDE with gaussian kernel and bandwidth of 0.5, marginalized via equal mass binning into 20 bins. Left) Query perplexity vs. the KDE with respect to random sample of points in the corpus. Middle) Query perplexity the KDE with respect to only the local neighborhood within the corpus. Right) Query perplexity vs. the average distance to the to the top neighbors. Horizontal line denotes the average across all queries. We see that while the trend in Query PPL function of the random component of the KDE non-monotonic, and even weakly positive, when considering the local region of highly similar samples for each query, there is strong clear negative trend in PPL as function of density, as measured by the local KDE or a simple average over neighbor distances.
16
PPL
PPL
Quaccy FFL
12
10
0.16
0.18
0.24
0.28
10
0.75
0.55
10
0.85
0.50
KDE Local Componont
40 Neighbor Dist
Aug Neighbor
Figure 5: Perplexity according to Pythia 6.9B for questions from the MMLU test set (OoD) as function of KDE with gaussian kernel and bandwidth of 0.5 or average distance to k nearest neighbors, marginalized via equal mass binning into 20 bins. Left) Question perplexity vs the KDE with respect to only the local neighborhood within the corpus. Middle) Question perplexity distance to k nearest neighbors. Right) Response perplexity vs distance to k nearest neighbors. Horizontal line denotes the average across all queries. While the relationship between query perplexity and local KDE isn't particularly strong, there a stronger trend as a function of simple neighbor distances. For response perplexity, we see clear trend function of average neighbor distances.
As with the finetuning scale experiments, we used mixed-effects regression to measure the reliability of the effect of density on perplexity and report those full results in Appendix A.9 In summary, find that query perplexity decreases randomly-sampled ID query set (p < .001) and the OoD OpenOrca dataset (p < .001) but not the OoD MMLU Test query (p .95). However, response perplexity does decrease slightly with increased density for the MMLU Test set (p < .001).
data density increases for the
Since the text length directly effects perplexity values, for Figure 4 and Figure 5 we must manually isolate subset of the queries where lengths are relatively similar to remove the variance due to length before plotting. We also observe small number of extremely large outlier perplexity values and drop those rows as well. Details about this process are provided in Appendix(A.) A.7 We report only the most insightful combinations here and present more exhaustive series in Appendix A.8
Insights and Applications
We can summarize our key insights from both groups of experiments succinctly by stating that in extremal cases like test question leakage, model performance and apparent training data density in query space. However, at pre- training scale, the specific way in which aggregates information about relevant support
expect a clear relationship between

---

Published as a conference paper at COLM 2024
in the dataset matters. Using the abstraction of "effective epochs" one can understand how repeated training on subsets of the data can lead to elevated marginal performance on those subsets but caution that the number of effective epochs required to achieve drastically inflated test accuracies on benchmarks, such as those reported in Yang et al. (2023), might be much higher than one would expect. This suggests that any un-evidenced claim that contamination is a reason to wholly discount the benchmark results for models trained on unspecified data distributions (at least at the common 7B parameter scale) should be taken with grain of salt.
Overall we believe that training data density quantification is grounded methodology for analyzing the failure modes of language models at an instance and group-wise level, since builds evidence for expected relative performance based directly on aspects of the training data itself. In response to observing relatively high density in certain regions of test query space and lower density in others, we expect that the consistency could be improved by supplementing data in those regions of weaker support through human data curation as well as automated processes such as machine paraphrasing.
model performance
8 Limitations and Conclusion
A few key limitations of our study include the focus on a small set of specific, but very relevant, datasets and models for the field and the fact that the KDE based methodology we propose is not hyperparameter free-the user must choose the kernel parameters. That said, the KDE methodology stands apart from simpler aggregations like the top-k distance measure in its "smoothness". While our experiments show that avg. distance to the top-k neighbors is also a reliable predictor of variance in performance, that metric requires the choice of specific value of k. Neighbors that happen to lie beyond this boundary will not be accounted for at all a simple top-k aggregation and thus changing k by just value of 1, can alter the measurement significantly. In contrast, the KDE measure accounts for the nearest and farthest neighbors simultaneously and varies smoothly under small perturbations of the bandwidth hyperparameter well as minor changes to which points fall in or outside of the k nearest neighbors set.
It is also likely that the preprocessing parameters length and stride-used to segment the training data before embedding are critical to the outcome of the analysis. We work at finer segmentation granularity than Tirumala al. (0233 and Yauney et al. (2023) but we do not ablate our choices enough to prove that this is required to explain meaningful amounts of variance. Finally we also acknowledge that full access to the training dataset of
newly "released" model is increasingly rare. However, given the potential of data-centric analysis techniques such as ours we claim that this is shortcoming of contemporary norms within the field rather than true limitation of our methodology. Rather, we implore the community to insist on the release of details sufficient to exactly re-materialize the full training data distribution as necessary precondition for presenting model releases as bona fide scientific artifacts.
10

---

Published as a conference paper at COLM 2024
References
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Tracing knowledge language models back the training data. arviv preprint ar Xix:2205.11142/ 2022.
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using 1me4. Journal of Statistical Software, 67(1):1 48,, 2015. doi: 10.18637 22s377//ss.006.. i01.
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arXiv preprint arXtt:2201.07311, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430.
PMLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. Xiv preprint arXiv:2202.076665 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arv preprint arxiv:2101 0000200 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https //doi.org/10. 5281/zenodo. 5371628
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El- Showk, Nelson Elhage, Zac HatfeeldDoddss, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint ar 20tt:2205.10487 2022.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint ar arXiv:2202.0062) 2022.
Daphne Ippolito, Florian Tramer, Milad Nast, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Chog and Nicholas Carlini. Preventing verbatim memo- rization in language models gives false sense of privacy. arXiv preprint 2r11::2210.17556, 2022.
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models. arv preprint Xiv:2400 240.00400.0605,,, 2024.
11

---

Published as a conference paper at COLM 2024
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp. 15696-15707. PMLR, 2023.
Matti Karppa, Martin Aumüller, and Rasmus Pagh. Deann: Speeding up kernel-density estimation using approximate nearest neighbor search. In International Conference on Artificial Intelligence and Statistics, pp. 3108-3137. PMLR, 2022.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Ka- sun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint Xxix:2306004644 2023.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Tyyer. Para- phrasing evades detectors ai-generated text, but retrieval is an effective defense. arxiv preprint arX::23003 2023.
Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. ImerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13):1-26, 2017. doi: 10.18637 /js..///..s..0..113...
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arviv preprint ar X111222333,11166 2023,
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2021. URL http:: Rtro/// org/
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 11 2019. URLhttps://arxi org/ 0t5/1998..00885
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. Xiv preprint arXio:2312.11805, 2023.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improv- ing 11m pretraining via document de-duplication and diversification. arXiv preprint ar Aixivv:308. 12284, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. Xiv preprint arXiv:2401.00368, 2023.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. Rethinking E benchmark and contamination for language models with rephrased samples. arXiv preprint Xrr::2331.00883 2023.
Gregory Yauney, Emily Reif, and David Mimno. Data similarity is not enough explain language model performance. ar Xiv preprint ar arXio:2311.0000 2023.
12

---

Published as a conference paper at COLM 2024
A Appendix
A.1
Approximate Kernel Density Estimation
We refer the reader to the excellent work Karppa et al. (2022) for complete derivations but describe here the critical results that we rely on in this study. Their Lemma 3 states that one can compute an unbiased estimator of the true KDEx, (xq) by splitting X of size n into two non-overlapping subsets, X^ and Xg, computing 2 KDEx, (x9) and zg KDEx, (xg) independently, and then combining them in weighted sum according to the sizes of XA and Xg:
X ZA
Xb n
2q
2B
Lemma 2 in their work, itself a result from prior literature, states that we can also compute an unbiased estimator of KDEx, (xq) by taking random sample Xm of size mı from X, and achieve approximation KDEx,, (Xy) with bounded error that decreases m increases.
Together, the authors use these results to propose Density Estimation from Approximate Nearest Neighbors (DEANN), where they approximate KDEx. (Xq) by computing the con- tribution of the nearest neighbors (Xx) and the contribution of the rest of the data (XB). We leverage their results to allow us to perform density estimations at scale across millions or billions of data points. An implementation stylized description is provided as Algorithm
A.1.1 On the Definition of the Sample Set Xe
While the text documents that comprise training corpus (which we embed to produce Xz) are in some sense taken as an imput to the analysis we propose, the notion of how De is broken up into the set of text segments D. = (nn-1) is actually a design choice both during the training must be considered is that the embedding models used to transform s; into x; often have maximum input length they can accept. More critically however, the kernel function that KDE is based on relies the ability to compute meaningful distances between points in the embedding space.
LLM and during our density analysis. A simple limitation that
Choosing the segment size for partitioning the documents in D, defines the sample space over which densities are computed and therefore potentially the outcome of the entire analysis. It simple to convince yourself of this by considering extreme choices like all of the training tokens being a single segment, or every individual token being its own segment (i.e. devolving to a unigram distribution).
Since comprehensive search over all segmentations is impossible, and we are unaware of conclusive results on optimal segmentation for general retrieval applications let alone our text sequence density estimation problem, for our finetuning experiments we avoid segmenting as the queries are already relatively short, and for the pretraining corpus we analyze, we adopt simple choice in this work as described in Appendix A.2.5 In Section 8 in the main body we briefly discuss the effect that this choice may have on the results.
A.1.2 "In Distribution" Queries
An important detail when analyzing queries that are suspected or known to be "In Distribu- tion" (ID), i.e. contained in the training data, is to ensure to embed all samples into the space in which the KDE is performed under the "same conditions". This means that if x E X and x E X, then the embedding vector of the query should be equal to the embedding of the matching doc in corpus, up to some floating point error tolerance. If any segmentation and/ or prompt templating is used, it must be applied in the same manner between the train corpus and test query set if any of the queries are intended to be perfect matches with elements in the corpus with a distance ~ 0.0 between their embeddings. For our leakage experiments and for our ID pretraining experiments, we ensure that this requirement is met.
13

---

Published as a conference paper at COLM 2024
A.2 Models, Data, Hyperparameters
A.2.1
TentenneTransformers
For all the finetuning scale experiments involving the MMLU dataset, we utilize the model from the pentence-transfon
library (Reimers & Gurevych 2019) following the setup of Yang et al. (2023). We choose this model based on prior experience, literature precedent, generally favorable placement on the sentence pentence-transfoom ormers ormers retrieval model leaderboard, and critically it's extreme efficiency. For the pretraining data embeddings, we use a slightly more general purpose variant of the same architecture, all Jil-MiniLM-L6-v22 similar size model trained on a more general data distribution.
We remark that while there are many larger, more competitive retrieval embedding models available, our desired to embed at the billion sample scale limits our choices. We feel that this tradeoff is compensated for by the fact that, at least for the pretraining-scale experiments, we consider very short sequences just 50 whitespace separated tokens or 3 to English sentences. We rely on the intuition that less representational complexity is required to accurately compare and contrast sequences of this length than when considering much longer documents. We additionally note that most prior work incurs the information loss cost of truncation at lengths of ~ 512 since fine-grained re-segmentation is seldom performed and most open source retrieval models have context widths shorter than the multiple thousands now commonplace for more modern auto-regressive models.
A.2.2
Llama
We adopt focused approach and consider one of the standard state of the art open source large language models, Llama 2 (Touvron et al. 2023), in its base form as the starting check- point for our finetuning scale experiments. We perform full finetuning of all parameters using the axolot finetuning harness with the following features: DeepSpeed ZeRO-2, "target only loss" calculation, sample-packing. warm-up with a linear learning rate schedule, and weight decay. We detail the precise parameters for the different experiments in the Appendix A well as the training configuration files provided with the source code release.
We also explore computing embeddings based on the last layer hidden states of the Llama model after training on the finetuning data. These features are pooled and normalized in the same manner as the last layer hidden states of the sentence- models. In initial investigations we explored the "last layer, last token" strategy of extracting embeddings, which has been explored in contemporary work including Tirumala et al. (2023), but did not find compelling evidence that this more expensive embedding process yields features more useful for our density estimation methodology than features from smaller retriever model.
ormers embedding
We that said, we admit the choice to restrict ourselves to the general purpose, extremely efficient retrieval models is mostly a practical one enabling pretraining scale experimenta- tion and consider the study of how to optimally leverage the internal representations of autoregressive large language models as important area of research (Wang et al. 2023)
A.2.3 MMLU
We use the MMLU dataset (Hendrycks et al.] 2021b as our test-bed for the density estimation methodology. We focus on this dataset because features prominently in both the Hugging Face Open LLM Leaderboard task set, as well as recent work competing for state of the art results (Team al. 2023) in language model training.
In order to limit confounding factors, we train the base model on data distribution similar to that of the test task: the "Auxiliary"" training set for MMLU. We format the training samples using the same logic that the EleutherAl LM Evaluation Harness (Gao et al. 2021
gi thub. http://c//ppeeaceeeee-o-llleeeee
14

---

Published as a conference paper at COLM 2024
implements. We use the precise MMLU test set as filtered and prepared by the harness, noting that this is a slight subset of the original test set released by Hendrycks et al. (2021b).
A.2.4
Pythia
The Pythia suite (Biderman at. 2023 is an open science focused language model collection with public code and data. Given that our methodology requires access to the training set, we have limited options for pre-training scale analyses. While none of the Pythia models are directly comparable to Llama 2 due to architectural, training and dataset differences, since it is a relatively modern causal language model is the most similar option amenable to our analysis at the time of this study
A.2.5 The Deduplicated Pile
The Pile (Gao et al., 2020 is a public multi-domain web corpus for language model pre- training. We use the reworked version that was globally deduplicated (Biderman et al., 2022) before the corresponding set of Pythia models were trained on it. The raw dataset itself contains approximately 134M documents, of widely varying lengths. Due to the input length limitations of embedding models, and more importantly to ensure that distances between relatively short queries and individual samples in the corpus will be meaningful, we re-segment the dataset by first splitting the documents into chunks of 50 whitespace separated tokens (split only on " ") with a stride of 40 tokens, creating an overlap of 10 tokens each with preceding and succeeding segments. The result is an inflation of the original 134M documents into 3.5B segments.
[star]] We make this somewhat arbitrary choice for segment length and stride based on the observation that 50 whitespace separated tokens approximately the length of few sentences or short paragraph in English. The current and previous sentences in this paragraph comprise precisely 50 such "words" on their own.[end]
We chose our stride in order to limit false negatives during nearest neighbor search without causing the segment count to grow beyond what the implementation can handle While we are unable to ablate this choice due to compute limitations, without obvious prior work to compare to, we accept these uncertainties part of the cost of exploration.
A.2.6 Approximate KDE Parameters
For each approximated KDE computation in the experiments with The Deduplicated Pile, for each query, we use the exact 1, .000 nearest neighbors, and a random complement of 10, 000, drawn for each query from fixed pre-sample of 1, 000, 000 sequences from the training corpus. These correspond to k, m2 and my respectively in Algorithm
A.3 Paraphrase Process Details
In this work, as in Yang et al. (2023) and other work that utilizes paraphrasing as black-box transformation in the security and safety domain (Kirchenbauer et al. [023 Krishna et al. 2023), we prompt a powerful large language model, GPI4-turbo, behave as a general purpose paraphrasing model using a template specifically designed to elicit paraphrased samples that are thoroughly transformed with respect the original queries. Figure shows the the prompt use to paraphrase the test queries as well as example of an original MMLU test question and its paraphrase.
Yang et al. 2023 explores adversarially regenerating paraphrases until paraphrase with sufficient dissimilarity to the original example is produced (in order to evade leakage
'The Tiny-llama model was trained concurrently to this research, however, the dataset used to train it is still significantly larger than the deduplicated Pile and harder to materialize on disk exactly it was prepared during the training of that model.
A stride greater than equal to your segment size results in no overlap between corpus segments, worse, omitted sequences which could incur "false negatives" during neighbor search even for queries with large overlaps with training data.
15

---

Published as a conference paper at COLM 2024
Please rephrase the following question without altaring its reaning, anauring you sdjust the verd ordez appropriatnly. Ensure that
GRIGINAL
ZARAFIRAEE
following woltiple choice acciolegy.
Below are sevecal mul citileeccoice inquiries concerning sociology, each provided with answer selections.
try to use similac words as subttitates wheze posoible, bo not chango the forrat of the multiple-choica quection. Whan encomataring mathenstccal formllax, please to
explain patriarchy
Rox do Karkist foninists interpret the concept of patsiarchyy
tamas
aqual opportunities and
A. An absence of equivalent rights and oppactumition tima batween the guodecs
formulas acen't identioal to original. When you cone accoss a single susber oz letter, conslder replaoing it with a sentence. When eccountaring ruplace the speces by commas, you can replace then with spaces. consider the proupt and choices as whole: there shoulds's is consacutive worde. optione challanging rephrane, sitering the initial lettez's case. Ininneiginel question: (nlasguuee?""
5. Gender strete, wheceby femeles are subropated by zaies ecenenic, political.
econonically politically and sexually
aequence of numbars,
grounds
apacea, commas: if seperated
domestic labour explot ted by capstalist ecenony
The arplottation fezale houssbold by syston cspitalios
the duel systers capitaliso
The twofold structures capitallet society nasculine hegexony
rale duximation
AnONEE:
Figure 6: The prompt used to instruct the powerful LLM used as a paraphrasing engine for the MMLU experiments, adopted from Yang et al. (2023) Similar to their work, we find that GPT-4 Turbo, 111-4--100-ppeyee
version, is able to reliably paraphrase MMLU questions
without losing too much context. We visually point out to the reader the fact that the entire
question preamble as well as all answer choices are part of the paraphrasing step.
detection measures). In our work, we are instead interested in collecting a spectrum para- phrases with varying degrees of similarity and dissimilarity to test queries, and therefore, we simply sample k paraphrases for every test query on which we intervene, finding that model yields diverse paraphrases that
pairing the prompt with the ept-4-1106-prevvv are still faithful to the key details of the original question.
A.4 Regression Analyses: Mixed Effects Modeling
In this work, we are interested in measuring the effect of training data density the example level. We treat example training data density as fixed effect and other covariates (such as the subject area of an MMLU question) as random effects-noise we wish to remove. Mixed effects modeling works by marginalizing the covariates and fitting model to predict the dependent variable from the fixed effect(s). The model yields an estimated coefficient for each fixed effect and p-values that, used in conjunction with an a-criterion, determine the statistical significance of the effect. We conclude the fixed effect reliably predicts the variance of the dependent variable when the p-value is below the a-criterion, customarily .05. Concretely, if we determined == -00,, we'd d expect to mistakenly conclude there was an effect! time in 1000 hypothetical experiments.
Importantly, the sign of an estimated coefficient indicates the direction of the relationship. So fixed effect (e.g., density) would have statistically- positive correlation with dependent variable (e.g., accuracy) if p < 0.5 and B > O. Overall, we look for agreement between the observed trends in binned plots of performance marginalized over various ranges of density estimates within a test set, the significance of the fixed effect, and the sign of its regression coefficient.
All analyses were performed using R Statistical Software (v4.3.3, R Core Team 2021 and supporting packages 1me4 v1.1-35.2 (Bates et al., 2015) and 1merTest v3.1-3 Kuznetsova
et al. 2017).
A.5
Controlled Experiments: Leakage Full Regression Results
We present two classes of analysis for the controlled experiments: manipulation verification regressions and the critical experimental regressions evaluating the effect of training data density. Regressions targeting the DV of rank accuracy are generalized linear mixed models (GLMER) fit by maximum likelihood with the Laplace approximation. Regressions targeting the DV of perplexity are linear mixed models (LMER) fit with restricted maximum likelihood (REML) and we use Satterwaite's method for calculating p-values. The mixed effects
16

---

Published as a conference paper at COLM 2024
structure across all regressions is the same: treat query length and example nested in MMLU question domain as random intercepts.
A.5.1 Leak manipulation verification.
Predicting rank accuracy with leak condition (GLMER)
rank rank_accuracy ~ exact leak + paraphrase leaks + (1]len(d,)) + /xaem(dd)exxxpeee)
Note:
Due to the natural ordering of paraphrase exposure (0,1,2,3), we treat the paraphrase variable as an ordered categorical and test its relationship as linear, quadratic, and cubic.
Estimate
Std. Error
value
p-value
(Intercept) exact_
1.30910
0.35018
3.738
<.001
0.62556
0.07626
8.203
<.001
paraphrase leaks (Linear)
0.44949
0.07448
6.035
<.001
paraphrase_leaks (Quadratic)
0.08411
0.07471
1.126
0.26
paraphrase Leaks (Cubic)
-0.08231
0.07569
-1.087
0.27
Predicting perplexity with leak condition (LMER)
perplexity ~ exact leak + paraphrase (1)len(d,)) + (1(task(d,) (exaaal( example)
Estimate
Std. Error
value
p-value
(Intercept)
1.803
3.199e-02
56.361
<.001
exact leak
-1.642e-01
9.318e-03
-17.620
<.001
paraphrase leaks (Linear)
-1.094e-01
9.254e-03
-11.827
<.001
paraphrase_eeal Jeaks (Quadratic)
4.924e-02
9.318e-03
5.284
<.001
paraphrase leaks (Cubic)
-1.513e-02
9,381e-03
-1.613
0.107
A.5.2 Training data density effect evaluation.
Predicting rank accuracy with training data density (GLMER)
rank accuracy KDEKg 0.1 + (1)len(dz)) + (1) (ttssk(d)) example)
Estimate
Std. Error
value
p-value
(Intercept)
1.086
3.437e-01 3.514
3.159
0.00158
KDEx
5.678e+04
16159.318
<.001
Predicting perplexity with training data density (LMER) perplexity ~ KDEX + (1(len(d,)) + (1)task(d.)exxnn)) example)
Estimate
Std. Error
I value
p-value
(Intercept)
1.90
3.129e-02
60.70
<.001
KDEx
-2.158e+04
9.025e+02
-23.91
<.001
17

---

Published as a conference paper at COLM 2024
A.5.3 Unique Variance Explained by KDE versus Top-k Distance
For completeness, we also consider an alternate modeling setup, one where we directly ask whether the two features, density and top-k distance, explain different aspects of variance? This question can be studied by running a slightly different regression model than those above that asks whether the two features explain significant parts of the variance in the performance outcome, when controlling for the variance explainable by the other. Our pre- dictors ("Fixed effects") are KDEK, 0.1 and aug fosio_list and target is again rank aankuaccrrcy,
Predicting rank accuracy with training data density and avg. top-k distance (LMER)
rank ancuuacy ~ KDE + awg.top10 dist + (1)len(d,)) + (1(task(dy) exxmplle)
Estimate
Std. Error
z value
p-value
(Intercept) KDEKn-01 aug top10 dist
1.416
3.514e-01
4.029
<.001
4.625e+04 -3.595e+04
5.372
8609.363
<.001
8.711e-02
-4.127
<.001
Interpreting the coefficient estimates, we observe that density goes up, rank accuracy goes up (+ Estimate) and as neighbor distance goes up rank accuracy goes down (- Estimate) as expected and the model suggests these relationships are accounting for unique variance. The fact that both p-values are small indicates that both terms remain reliable predictors of variance when included in the same model.
This evidence clearly shows that despite the close relationship between top-k distances and the kernel density measure, in practice, they can each be used to explain different parts of model behavior. Studying this relationship further is an important area for future work. Overall, we believe this regression outcome helps highlight both the distinction between and compatibility of our proposed methodology and the top-k distance metric.
pon leaked Q's
Jeaked Q'
Exact
Paras
Exact:
Paras
Exact
Paras 1,2
Exact Paras 1,2,3
1000
1000
1000
1000
Count
500
500
500-
0-
0.50
1.5
16-8
Figure 7: Companion figure to Figure 3a showing the version of panel (a) without cropping for the histograms. While the leak and non-leak distributions are indeed distinct both upon inspection and statistical test via regression, this is not visually apparent unless axes are suitably cropped. See main body version.
A.6 Controlled Experiments: Leave-One-Sujject-Out
In a second experiment set focused on controlling the level of training support for specific queries, we utilize the subject metadata provided for the questions the MMLU testing set to intervene by "leaving-one-oute of the subjects areas covered in the training data. In particular, we consider the supercalegories defined by the MMLU authors, which maps each of the 57 fine-grained subject areas to more general topic. The training data does not have subject metadata associated with it and so we generate our own using an automated procedure. However, the results of this experiment are inconclusive.
We assign each one of the training samples from the MMLU "auxiliary" training set to fine- grained subject area using kNN classifier, where each point receives its label according to a majority vote between the subject labels of the k-nearest questions in the test set according to distances in embedding space. After assigning each training question subject, we assign
18

---

Published as a conference paper at COLM 2024
it to supercategory based the aforementioned mapping. Since this does not yield balanced subsetting of the training samples (some supercategories are much larger than others) examine the counts for each supercategory and select 4 with counts between 2, 000 to 4, 000 questions (out of 99, 842 total).
Then take the base language model and train at collection of datasets where for each split we leave out the group of training examples corresponding to each of the supercategories selected in turn. For each model we measure the impact that the intervention has on the average performance across test questions sharing the supercategory label of the left out samples, well as the test questions from the other 3 supercategories as reference. We
present the results of these experiments in Figure 8 as deltas in density and performance against the control model, which is trained on the full collection of training samples.
I1
03800
T
1
.00000
-0.0000
eppa
eepeet
-0.004
leakeg
lackey
pulitics
poldice
polltice
Figure 8: (Left) Shows that while the density measurements behave expected across splits in our leave-one-subject-out experiment, (Right) the performance changes are small and inconsistent. Curiously, leaving out the math subset seems have an outsize impact the performance of the model on the biology set, suggesting that determining the influence that specific types of training questions have on the model's performance in this setting could be quite difficult no matter which heuristic was chosen.
A.7 In-the-Wild Experiments: Controlling for length
While the pretraining dataset analyzed, The Deduplicated Pile, is segemented according to whitespace before embedding, there still a significant amount of variation the lengths of the texts after tokenization, as measured by character length. Since perplexity generally decreases as function of length under any language model as tokens are casier to predict given more context, we need to control for length in order to highlight any observable differences according to density.
We also need to drop outliers with respect to the performance measurement (we drop rows where query PPL 500 and Response PPL > 60), as these heavily skew the marginalization process. To clean up the data for Figure and Figure in the main body, first analyze perplexity as a function of query length in characters, and use this view to identify a suitable upper and lower bound for length such that be explained by changes in density rather than length.
able to focus in any variance that can
A.8 In-the-Wild Experiments: Extended Figure Set
While we highlight the most informative figures in the main body (Figure Figure we also present more complete set of visualizations for the MMLU Test set and Open Orca sampled query set here.
19

---

Published as a conference paper at COLM 2024
60
Query PPL
Query PPL
24
22
20
100
200
300
400
260
280
300
320
340
Query Length
Query Length
Figure 9: Due to the fact that the length of text significantly impacts perplexity measure- ments, (left) we analyze the distribution of lengths present in the query set to identify range of sequence lengths for which the performance measurement is relatively stable. (right) For the set of Random 10k (ID) samples from The Pile, we identify this as between 250 and 350 characters, and within this range we see that the query PPL is relatively constant around the mean. Thus, the data is limited to this range for the figures presented in the main body to highlight whatever variance can be attributed differences in density.
14
13
13.0
Query PPL
12
PPL
12.5
11
12.0
11.5
9
250
500
750
1000
1250
1500
250
300
350
Query Length
Query Length
Figure 10: Due to the fact that the length of text significantly impacts perplexity mea- surements, (left) we analyze the distribution of lengths present in the query set to identify range of sequence lengths for which the performance measurement is relatively stable. (right) For the set of MMLU Test questions, we identify this as between 200 and 400 charac- ters, where the query PPL varies less wildly and where most of the data are concentrated (the majority of the bins are located in the upper left of the left panel). Thus, the data limited to this range for the figures presented in the main body to highlight whatever variance can be attributed to differences in density.
20

---

Published as a conference paper at COLM 2024
12
Query PPL
Query PPL
8
1000
2000
3000
500
1000
1500
2000
Query Length
Query Length
Figure 11: Due to the fact that the length of text significantly impacts perplexity mea- surements, (left) we analyze the distribution of lengths present in the query set to identify a range of sequence lengths for which the performance measurement is relatively stable. (right) For the set of Open Orca questions, the best we are able to do is limit to between 500 and 2000 characters, where the query PPL varies according to a "U"-shaped and where most of the data are concentrated (the majority of the bins are located in the upper left of the left panel). Thus, the data limited to this range for the figures presented the main body to highlight whatever variance can be attributed to differences in density.
16
16
Query PPL
Query PPL
14
14
12-
10
10
0.16
0.18
0.20
0.22
0.24
0.26
0.75
0.70
0.65
0.60
0.55
0.50
KDE Local Component
Avg 10 Neighbor Distance
16
Response PPL
14
Response
14
12
10
10
0.16
0.18
0.20
0.22
0.24
0.26
0.75
0.70
0.65
0.60
0.55
0.50
KDE Local Component
Avg 10 Neighbor Distance
Figure 12: Perplexity according to Pythia 6.9B for questions from the MMLU test set (OoD) as a function of KDE with gaussian kernel and a bandwidth of 0.5 or average distance to k nearest neighbors, marginalized via equal mass binning into 20 bins. Top,Left) question per- plexity the KDE with respect to only the local neighborhood within the corpus, corpus,Top,Right Right) question perplexity vs distance to k nearest neighbors, Bottom,Left) response perplexity vs the KDE with respect to only the local neighborhood, and Bottom, Bottom, Right) response perplex- ity vs distance to k nearest neighbors. Horizontal line denotes the average across all queries. We see that while there isn't 't consistent trend in question or response perplexity according to the local KDE, there is more clear correlation for the distance to the k nearest neighbors.
21

---

Published as a conference paper at COLM 2024
11
11
Query PPL
10
Query PPL
10
9-
8-
8
0.15
0.20
0.25
0
0.8
A
KDE Local Component
Avg 10 Neighbor Distance
3.5
3.0
Response PPL
3.0
2.5
Response
2.5
2.0
2.0
1.5
0.15
0.20
0.25
0.8
KDE Local Component
Avg 10 Neighbor Distance
Figure 13: Perplexity according to Pythia 6.BB for questions from the Open Orca query set (OoD) as function of KDE with gaussian kernel and bandwidth of 0.5 average distance to k nearest neighbors, marginalized via equal mass binning into 20 bins. Tof,Left) question perplexity
the KDE with respect to only the local neighborhood within the cor- pus, pus,Top,Right question perplexity vs distance to k nearest neighbors, Bottom,Left) response perplexity vs the KDE with respect to only the local neighborhood, and Bottom, response perplexity vs distance to k nearest neighbors. Horizontal line denotes the average across all queries. We see that there non-monotonic relationship between query per- plexity and both of the density heuristics. For response perplexity the trend is noisier. We suspect that the length variation for this query set washes out the effect of relative density differences.
22

---

Published as a conference paper at COLM 2024
A.9 In-the-Wild Experiments: Full Regression Results
Predicting perplexity with KDE computed w.r.t. The Deduplicated Pile for Pythia-6.9B (GLMER)
DV ~ KDEK, + (1)len(dg)
We use the "local" density estimate for regressions including density as a fixed effect, and report as (B/p-value). "ns" denotes no significant effect.
Note:
Query Set
KDEg00
Avg 10 NN
DV Query Perplexity
Rand. 10k (ID)
B = 62.250, p < .001 ns
B 49.388, < .001
MMLU Test (OoD)
B = 8.5067, < .001
Open Orca (OoD)
B = 6.9712, p < .001
B = 1.5845, = .001
DV = Response Perplexity
Rand. 10k (ID)
N/A
N/A
MMLU Test (OoD)
B = 53.911, p .001 ns
B = 16.777,, p .002
Open Orca (OoD)
115
A.10
Preliminary Investigations
A.10.1
Paraphrase Retrieval
The LMD3 hypothesis relies on the fact that mixing exact copies of samples x, or paraphrases thereof, into the training corpus X should increase the KDE at query point x. One sufficient condition for this be true that the paraphrases we mix into the pretraining dataset must be represented by the embedding model as more similar on average to their corresponding original query than the nearest neighbors other training samples are on average.
In order to confirm that the lightweight embedding model we use is adequate for the similarity computations required by the KDE, and thereby limit the likelihood that the neighbor-search step would confound our overall results, we formulate retrieval problem using the queries and their paraphrases and confirm that our nearest neighbor search reliably retrieves the "correct" neighbors for each query. In addition to that test, we visualize the distribution of the nearest neighbor distances for the 1, 000 queries where we plant paraphrases and exact copy, for those queries that do not.
We see that due to the presence of the exact copy of each query for the interventional subset, in the left panel of Figure 14 for queries with leakage, the nearest neighbor is at distance 0.0, and due to the presence of the paraphrases, the average distance to the top-3 nearest neighbors also much smaller than for queries with no paraphrases or copies. We corroborate this visualization by treating the collection of copies and paraphrases for each question as the target set in a retrieval problem and measuring Recall@k for k E (5, 10). While a perfect score of 1.0 is theoretically achievable at just k 4 for this data, while neither model achieves this score even at k 5, at k sufficient evidence that the local neighborhoods of queries will contain the paraphrases and copies we introduce with some reliability. Thus, we conclude that the embedding space generated by the lightweight retrieval embedding model is unlikely to cause experiments to return null effect relationships with the KDE for the spurious reason that the embedding space doesn't return the "correct" neighbors.
both are over 92% which we treat as
A.10.2 Bandwidths for separability of leak and non-leak
A.10.3 Training to high accuracy on the test set.
23

---

Published as a conference paper at COLM 2024
Retriever
Mistral 7B
Recall@5
0.9075
0.8875
Recall 10
0.9375
0.925
NDCG@10 0.939
0.928
Table 7: During preliminary explorations we set up a retrieval evaluation where we used the original test questions queries, and tried to retrieve their paraphrases (3 per question plus 1 exact copy) when mixed into the training corpus. The embeddings produced by both the very strong LLM being analyzed for performance effects, and lightweight retrieval model, were equally useful in searching for leaked copies of the test questions when considering the 5, and 10 nearest neighbors (Recall@k) for each test question query. In light of the fact that we wanted to scale our experiments to pretraining corpora, we decided that it was acceptable use the lightweight retrieval model for the main experiments.
Top 1 neighbor distance for queries w and w/o parsphrases
Avg 3 neighbor distancc for queries and w/o paraphrases
w/ paraphrases
w/ parophrases
paraphrases
parsphrases
800
800
600
600
Count
Count
200
200
0.0
Top neighbor distance
Aug neighbor distance
Figure 14
24

---

Published as a conference paper at COLM 2024
10-4
10-9
median KDE over Q
10-14
10-19-
10-24
10-29
10-34
querles w/ paraphrases queries w/o paraphrases
10-39
0.025
0.050 0.075 0.100 0.125 0.150 0.175 0.200
h
Figure 15: Since the proper KDE bandwidth for a particular problem an empirically derived parameter, we run small ablation to chose selection of bandwidths that are likely to produce an informative range of KDE values, especially those useful for identifying the set of queries for which we intervene through the planting of paraphrases amongst the other test questions. We see that as bandwidth is swept from low to higher values, the difference between the KDEs for queries with planted paraphrases and those without becomes smaller and smaller. This suggests that narrow bandwidths close to 0.0 are likely the most useful for developing reliable estimator of whether query has relevant paraphrases included in the training dataset. We use a representative selection of bandwidths from across this range in our main experiments so that we are able to examine what effect the bandwidth has on the relationship between KDE and other measures of interest. For the two euclidean kernels we investigate, the bandwidth selections are (0.01, 0.05, 0.1, 1.0} for the exponential kernel, and (0.1, 0.2, 0.5, 1.0) for the gaussian kernel, which we identified as reasonably similar sets for both kernel functions. We discuss the potential limitations of the selection process the main work.
25

---

Published as a conference paper at COLM 2024
Gen. Accuracy vs. Epoch
Random
0.9
leaked
0.7
Gen. Accuracy
0.6
10
20
30
40
50
60
Epoch
Figure 16: In order to ground our expectations on what performance expect to given the realistic finetuning hyperparameters we utilize, we perform a final validation check experiment by training for extended number of epochs dataset that solely consists of the MMLÚ test queries. Mirroring the observations of Yang et al. (2023) we see that extremely large number of epochs of training performance on the test in the "ideal" scenario of training on the exact test queries. Since we larger set of training data where only a limited number of test queries and or their paraphrases are mixed in, much lower performance is expected even on the leaked test questions than the extreme values achieved beyond 30 epochs.
required to even approach perfect
both focus the more realistic setting of training just couple epochs, as well
26


Reviews:
[{"review_id": "7LLbOUn4xB", "reviewer": "Reviewer_tSwb", "summary": "This paper proposes a methodology for establishing a relationship between LM task performance and training data density estimation. Some controlled experiments are performed by introducing paraphrased data points. Although the approach is interesting, the results are not always easy to interpret. Nevertheless, I think the paper provides a useful tool for analysis. Many details are provided in the Appendices.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the vote of confidence from the reviewer in both the quality of our analysis and the completeness of our presentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main reason to accept the paper is the analysis provided in the paper. I believe more analysis papers are needed in this field.", "reasons_to_reject": "As the authors also point out there are some limitations which might restrict the wide application of the methodology.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RhKQbh9jhT", "reviewer": "Reviewer_vEjv", "summary": "The authors present a series of studies relating aspects of training data density at the example level to perplexity performance and task performance in language models. The authors perform both 1) a controlled experiment in which they vary the number and type of examples in training data that are \"leaked\" from the test set, and 2) an analysis of the relationship between perplexity measures and two different estimates of data density. Their claims are that increasing the training data density around certain examples in the test set can boost task performance, and that measures of training data density around test set examples can be predictive of perplexity performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "(Responses in comment order, space permitting.)\n\nIn the [attached](https://anonymous.4open.science/r/colm-rebuttal-8CB1/LMD3_colm_rebuttal.pdf) set of figures we have mocked up some requested changes, a few of which we discuss below.\n\nWe agree that the first row of subplots in Fig. 3 as presented in the review copy has relatively low information content. Therefore, we propose removing the histograms entirely from Fig. 3 in the main body, and then in the Appendix, presenting a \"zoomed in\" version of the histograms for the row of cases where only paraphrases were leaked. \n\nAs the reviwer points out, our experiments suggest that the simple average of the distances between a query and its $k$ nearest neighbors is _also_ a discriminative feature. To be incorporated in the updated draft:\n\n\"The KDE methodology stands apart from simpler aggregations like the top-$k$ distance measure in its \"smoothness\". The top-$k$ average requires that you choose a specfic value of $k$ and neighbors that lie beyond this boundary will not be accounted for. Changing $k$ by just a value of $1$, can drastically change the mean. In contrast, the KDE measure accounts for the nearest and farthest neighbors simultaneously and varies more smoothly under perturbations of the $k$ nearest neighbor set. While one does still need to choose a kernel bandwidth for the KDE, informally, there always exists some $\\delta$ such that the change in density at query point $q$ for bandwidth $h$ versus bandwidth $h+\\delta$ is bounded and small.\"\n\nOverall, we believe that choosing k, which can have a discontinuous effect on the estimate, is more fraught than tuning the bandwidth and it is a limitation of our evaluation rather than the method that we didn't identify a scenario where the KDE and top-k distance approaches diverge in discriminative power.\n\nWe apologize for the lack of detail. For the larger scale nearest neighbor searches over pretraining scale datasets, we used a system called ScaNN (Scalable Nearest Neighbors, (Guo, 2020)). In line with this request and comments made in another review, we will promote Algorithm 1 to the main body in the updated draft. Additionally, we can add a discussion to accompany the algorithm:\n\n\"Using a brute force maximum inner product search, the time complexity of the neighbor search step for $M$ queries over a corpus of $N$ segments is $O(M*N)$. However, with more clever tree data structures and quantization schemes, this runtime can be reduced significantly.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experimental design and methodology are both sound and grounded strongly in realistic settings.\n2. The experiments themselves and the research questions they are meant to answer are interesting and relevant to many researchers who work on or with language models\n3. Mostly clear writing and organization", "reasons_to_reject": "My main concern is that at least one of the main claims about the usefulness of kernel density estimation (KDE) seems only weakly supported; it is unclear to me whether my perception is due to the evidence gathered itself or simply the authors' presentation of the results.\n1. The authors claim that Figure 3 \"shows that KDE is a discriminate feature between the leak set and non-leak set.\" However, the top row of Figure 3 clearly shows overlapping bars in what appears to be a histogram of KDE values. If there is any distinction, it is not visible from that plot with the x-axis scale presented. (Please do let me know if there is some mistake in Figure 3 or if a different scale would show a different pattern.) There appears to be a clear distinction only when the \"leaked\" queries are exact duplicates of test queries, which significantly dampens any claims of KDE's usefulness as a \"discriminative feature\" between leaked and non-leaked data points (if it cannot detect \"leaked\" paraphrases + many other simple methods could detect exact match leakage)\n2. Relatedly, Figure 5's caption and 6.2 show that KDE is often but *not consistently* indicative of predictive of perplexity (visible also in Appendix figures). I would like to see the authors discuss more specifically the usefulness and limitations of KDE and top-k neighbor distance -- especially since it seems like the average Top-k neighbor distance may be at least as reliable and KDE seems to feature more heavily in the manuscript\n\nOther concerns:\n3. No useful information about the proprietary algorithm for example neighbor search. The efficiency of this implementation seems to be an important aspect of the authors' methodology, so it could be very helpful to mention its runtime complexity or even the scale of resources (compute/time) required to run the analyses presented\n4. Baseline rank accuracy (0 paraphrases, Exact=0) missing depiction of variance. I would have liked to see it plotted like the others with error bars (or bands if the horizontal line style is kept)\n\n\nI would not be comfortable accepting this paper in its current state, but I would be sincerely glad to adjust my score if the authors address my concerns, as I find the topic and experimental setting interesting, and my concerns are more about the discussion of the results and the conclusions drawn than the fundamental questions that the authors set out to answer.", "questions_to_authors": "See above. Is KDE truly not affected by the existence of \"leaked\" paraphrases? Additionally, some minor comments/suggestions/questions:\n1. Last sentence in section 4.1 does not parse well to me. I would recommend not only splitting it up into multiple sentence but also rearranging the post-comma clause (e.g. --> \"Future work can explore ...\")\n2. The term \"doctored\" in 5.1 makes that sentence read weirdly to me. Is there a more specific way that can the modifications to the training set can be succinctly described there?\n3. It would be helpful to point to 6.1 in Figure 2 for a definition of the \"effective epochs\" that the accuracy values are plotted against, since 6.1 falls on a later page.\n4. Figure 3 caption: \"exactly and **/** or via paraphrase\"\n5. Figure 5 caption: would like to see a pointer to the missing plots in the Appendix\n6. Appendix 10.2 and 10.3 seem to contain only figures; unclear which specific figures are meant to belong where", "extra_scores": {}}, {"review_id": "x2IhgdblFw", "reviewer": "Reviewer_y6ai", "summary": "This work studies the data density dependence of language models on the perplexities/accuracy of queries / downstream tasks. In particular, this work provides the framework to assess the dependence of performance on a query given support of the training distribution The work suggests that model performance can be reliably predicted based on deter density dependence in many of the cases.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(Responses in comment order, space permitting.)\n\nBeyond questions about an embedding model's capacity to represent certain data and questions about the diversity of its training data, we remark that interpreting distances in high dimensional spaces can be difficult. However, we would like to present a preliminary analysis that didn't make it into the review copy. In early experiments we considered embeddings derived from the hidden states of the LLM you are analyzing as an alternate to the standalone lightweight retrieval encoder used throughout the work. See the [attached](https://anonymous.4open.science/r/colm-rebuttal-8CB1/LMD3_colm_rebuttal.pdf) document for this analysis; we will incorporate it into the updated draft.\n\nRegarding the differences between KDE and top-k distance aggregations please see the response to reviewer vEjv.\n\nFirst we note that due to computational limits, and the cost of preliminary explorations, we chose not to run all possible permutations of the finetuning experiment focusing on those determined to be most critical to exploring our hypothesis.\n\nWe have updated Fig. 2 slightly in the attached document for improved clarity. While it's intuitive that training on an exact copy of a test question causes a larger increase in model performance on the corresponding questions than training on a single paraphrase, the fact that it has a larger effect on accuracy than training on all three paraphrases is slightly surprising. We adopt a purely descriptive stance in interpreting these results.\n\nIn the pretraining scale experiments we focused on loss/perplexity as these measures can be calculated on any text sample rather than accuracy measures that are constrained to only task-like data.\n\nThis (2404.04125) is a very interesting work, thank you for pointing it out! While it is difficult to identify a precise correspondence between visual concept frequencies and the repetition of test questions or paraphrases thereof, we agree that the there is some relationship between these findings.\n\nWhile we cannot recover the precise segement texts for each neighbor easily at this time, we can report some anecdotal observations that nearest neighbors for pretraining segments were indeed sometimes duplicates or outher chunks of the same document.\n\nWhile we did not directly compare the difference between cosine similarity and unnormalized distances, the sentence-transformers embedding model we used emits unit normalized vectors by default.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strength of this work includes a very comprehensive study of different models across various tasks.\n\nI like how the authors performed experiments to understand how the KDE changes as duplication increases. The control experiment by purposefully adding paraphrases of test set queries to measure the effect of the same on model performance helps understand the effect of data density on model performance.\n\nSome of the takeaways of this work are quite interesting. For example, model performance does not increase drastically even if exact duplicates are added to the data set. And for new tasks, you can give some estimate of model performance based on training density.", "reasons_to_reject": "1. Embedding model bias. I would like to hear more from authors on this beyond the small subsection of the paper. For example, embedding models not trained on code will actually cluster all code data very close by. I think the embedding model bias is a big problem here.\n2. I do not understand why should we use KDE, when avg nearest neighbour distance performs better in the last few graphs in the paper? I did not find this discussion in the paper.\n\nOn top of that, two weaknesses that question the validity of the hypothesis of this paper:\n1. The paper does not discuss the result of adding only Paras=3 in Figure 2. This experiment should not increase the density more than Paras=1 2 3 because the point in yellow should have almost 3x more density as compared to Paras=3. However, if it results in close to equivalent improvement in model performance, it would suggest density dependence may not actually be the true hypothesis here. Some other tests would be, does repeating exact 3x help more than paras=1,2,3 + exact repetition? In general, the \"exact\" inclusion convulates the hypothesis quite a bit. One example, should not increase the density much, but it does have a very large impact on the model performance (as compared to Paras=1,2,3)\n2. In Figure 5, we never discuss accuracy on MMLU and only the response and query PPL. This is not really descriptive enough because as models improve or go to more favorable regions, the perplexity typically improves on all correct and wrong answers. I do not think the graph is suggestive of model performance as of now.", "questions_to_authors": "1. here was a recent paper listed as no zero shot performance without exponential data (https://arxiv.org/abs/2404.04125). I wonder if the authors would like to contrast the results with the same. of course this paper came out during the review. But I am curious if you think exponential amount of data should be added rather than the linear amounts that tried in your work. \n2. secondly I don't understand the goal of figure 3.a, the count figure.\n3. When the neighbor distance is low, do you check if there is a paraphrase or exact duplicate? Since you're only aggregating with 10 nearest Neighbors, we can actually check if the samples that were in the train set were in some way duplicates of the query\n4. How does the average cosine similarity fare in comparison to the distance between embeddings", "extra_scores": {}}, {"review_id": "arCIIUeDjv", "reviewer": "Reviewer_phfD", "summary": "The paper considers whether kernel density estimation can be used as a measure to quantify the familiarity (my term, not the paper's) of a test-time example to a language model.  The paper includes a \"controlled\" experiment where MMLU test instances and/or paraphrases of them are leaked into finetuning data (and effect on their accuracy is measured), and \"in the wild\" experiments where test instances are leaked into pretraining data (and effect on their perplexity is measured).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(Responses in comment order, space permitting.)\n\nIn Appendix A.1.1 we discuss segmentation in some depth and in Sec. 8, we remark that it is an overlooked detail in similar works. However, in Sec 5.4 in the main body, we now realize that there is no pointer to Appendix A.1.1, a mistake on our part.\n\nThe off-the-shelf embedding model we use is detailed in Appendix A.2.1. We believe that this model was trained on a mixture of retrieval specific data including MS-MARCO. While we can not completely rule out the possibility of overlap between MMLU and those data without significant work, the likelihood that the retrieval specific training data for that model contains any significant fraction of MMLU test question texts is exceedingly unlikely in our opinion.\n\nIn response to this comment and the suggestions of other reviewers, we are choosing to remove the histograms from Fig. 3 in the main body. Please see the [attached](https://anonymous.4open.science/r/colm-rebuttal-8CB1/LMD3_colm_rebuttal.pdf) set of updated figures for mockup of the updated version. In addition, Algorithm 1 will be promoted to the main body using the extra page of space allowed in the updated draft.\n\nWe appreciate the suggestion for improved clarity via a \"Setup-Result-Setup-Result\" ordering and will consider it in the draft update process. \n\nWhile we cannot entirely rule overlap out, none of our results or claims hinge on \"OoD\" separation between MMLU and the Pile being strict. Rather, in the pretraining scale experiments we report relative differences where the training corpus and the model are both constant. \n\nWe would be happy to add something like the following in the limitations section:\n\n\"Conducting a similar series of controlled experiments where examples of test queries are leaked into a full scale pretraining dataset would be interesting but was computationally out of scope for this work. We expect that results could vary and that the KDE hyperparameters explored in this work might not be optimal for that setting.\"\n\nRegarding algorithmic details and computational cost, see response to reviewer vEjv.\n\nAgreed, we will tighten the specificity of the claim about benchmark invalidation in the introduction or omit it altogether.\n\nWe thank the reviewer for the insightful suggestion and have prepared a version of Fig. 2 where the x-axis represents \"counts\" i.e. if we train for 2 epochs, and have leaked 1 exact copy and 1 paraphrase, the total count would be $2*(1 + 1) = 4$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "At a high level, the method is intuitive and straightforward, and arguably a very strong starting point.\n\nThe findings suggest that KDE is a useful tool for this kind of analysis.\n\nThe paper describes its own applicability as a limitation; the pretraining corpus needs to be available to carry out the analysis on a model, in general.  I don't think this is a strike against; the more concretely the research community can explain why the data needs to be open, the stronger the pressure will be for transparency in how models are built.  This paper can be part of that case.\n\nI found the paper generally clear.", "reasons_to_reject": "One matter that's not at all clear from the paper is the matter of text granularity.  In the KDE setup, the paper doesn't make clear what an item in the corpus is.  Presumably it's not a word/token; is it a document?  Something smaller?  It would seem to make a big difference.  I'm fairly certain that this is a distinct decision from \"bandwidth\" and other kernel hyperparameters.\n\nThere is no discussion of the data that the off-the-shelf embedding model was trained on.  It seems worth some discussion as contamination of the embedding model's training data (with test query content) could be a problem.\n\nIt's a weakness of the paper that the Algorithm 1 and the details of paraphrase generation are deferred to the appendix.  I suspect other reviewers might prefer some other appendix materials to have been in the main paper, and I empathize with the challenge of keeping to the page limit.  But these two, at least, are essential for understanding the paper.  The details of the mixed effects models and controls for length would also be better in the main paper.  (Some space could be saved by compressing the related work section and figure 3's \"count\" histograms, could maybe be cut or moved to appendix since they're very low-information.)\n\nI'd strongly recommend reorganizing the paper so that results of an experiment come directly after the explanation of the setup.  The current organization (all experimental setups laid out before discussing any results) creates an unnecessary mental burden on the reader.\n\nSection 5.4:  MMLU test instances are taken as OOD, but the paper doesn't say whether these were explicitly checked against the pretraining corpus.  Perhaps some of them are actually in the pretraining data.\n\nMy understanding of the methodology is that \"leaked instances\" (in the pretraining case at least) are revealed to the model only after the normal pretraining is complete (the authors did not re-pretrain the whole model with the leaks randomly mixed in).  This has implications for the role of pretraining hyperparameters and the generalizability of the conclusions.  The paper should discuss this limitation. \n\nThere is no attempt to quantify the computational cost of the experiments, leading to uncertainty about how practical the method is to apply in practice or to experiment with further.\n\nThis claim in the introduction -- \"the effect of dataset leakage is small enough that it likely does not invalidate benchmark numbers\" -- is not really supported by the quantitative arguments of the paper.  The paper doesn't really inspect any benchmark numbers at all.", "questions_to_authors": "Suggestion:  figure 2 is really nice.  I would suggest using the number of instances as the x-axis on the left, it's just easier to interpret and doesn't rely on weird assumptions (i.e., your fractional weighting based on cosine similarity).  The wide error bars on the right side make me want to see plots that show individual points (for a single choice of what is \"leaked\" at a time), for individual data points' KDE values and accuracy values.  Do we see correlation across instances as well?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Although the reviewers make arguments for and against this paper, including some unresolved questions (possibly due to time as they were posted at the end of the rebuttal period). This is an important area and everyone agree the experiments are well designed and study an important class of problems. The relationships between data and end task performance is a hard one to figure out, and this papers add quite a bit to the discussion even if it doesn't have perfect answers on all fronts.
Decision: Accept