Data Similarity is Not Enough to Explain Language Model Performance
Authors: Gregory Yauney
Emily Reif
David Mimno
Venue: EMNLP 2023 Main (2023)
Keywords: similarity
dataset difficulty
pretraining data analysis

Abstract: Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model's pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.

Data Similarity is Not Enough to Explain Language Model Performance
Gregory Yauney Emily Reif² David Mimno"
Coomell University GGooge Research Traueey@cs ereifégoogle. com
mimno@cornell, edu
Abstract
Stack Exchange forum classification
Normalized score
Englisht
Large language models achieve high perfor- mance many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more sim- ilar to model's 's pretraining data is assumed to be easier for that model. We test whether distri- butional and example-specific similarity mea-
70
French
60
50
Indonesian
40
Armishh
Swatt
30
0.5
3.0
1.5
2.0
2.5
3.0
3.5
4.0
KL-divergence from pretraining token distribution
BIG-bench Lite multiple choice tasks
Normalized score
sures (embedding- token- and model-based) correlate with language model performance through large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with perfor- mance for multilingual datasets, but in other benchmarks, surprisingly find that similar- ity metrics are not correlated with accuracy or even each other. This suggests that the relation- ship between pretraining data and downstream tasks is more complex than often assumed.
20
10
-10
-2
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
KL-divergence from pretraining token distribution
Figure Similarity the pretraining dataset correlates with Pythia-6.98 zero-shot performance when compar- ing different-language versions single task (top). but similarity fails to explain performance differences in BIG-bench Lite (bottom). Error bars are over sam- ples of the pretraining dataset.
1 Introduction
Large language models (LMs) are pretrained on large datasets, such as C4 (Raffel et al., 2020) and the Pile (Gao et al., 2020), and used for downstream tasks. Many hypothesize that task performance is matter of how similar the task data is the pre- training data (Brown al., 2020; Gao al., 2020; Gonen et al., 2022). Despite the prevalence this "similarity hypothesis," " has not yet been evalu- ated beyond data overlap (Kandpal et al., 2023). Can we measure the impact of data similarity? Similarity matters at a coarse level: models pre- trained on English data do not perform well on other languages (Bender, 2019; Scao al., 2022). Robustness to distribution shifts has been exten- sively studied (Hendrycks et al., 2020; Koh al., 2021), though dissimilarity is usually assumed through dataset construction rather than explicitly measured. We measure distributional similarity be- tween pretraining datasets and benchmarks using
range of similarity metrics, finding that similarity correlates with performance in controlled setting but not across common benchmarks (Figure 1).
Example-level similarity offers a more targeted hypothesis that controls for task difficulty: within downstream dataset, examples that are similar examples the pretraining data should be easier than dissimilar examples. has been shown that examples with words and entities more frequently found in the pretraining dataset (e.g. Obama or the number 24) are more often answered correctly in arithmetic and factoid QA tasks (Razeghi et al., 2022; Kandpal et al., 2023; Biderman et al., 2023). We ask a stronger version of this question: will model perform well on an example if that example similar to any pretraining document?
We test the hypotheses that distributional similar- ity and ssample-similarig determine performance
Work done as Student Researcher at Google Research.

---

using three measurements of similarity from prior work: word similarity, embedding similarity, and language model perplexity. Our contributions are the following. First, we show that similarity cor- relates with performance for multilingual datasets. The same underlying task is represented in different languages, each with different aggregate similar- ity to the pretraining dataset, and languages that are more similar the English pretraining data afford higher performance. Second, we show that multiple measures of distributional similarity are surprisingly not correlated with zero-shot perfor- mance on BIG-bench Lite multiple-choice tasks (Srivastava et al., 2023) across models and pretrain- ing datasets. Third, we show that within down- stream dataset, examples that are more similar to any example in the pretraining dataset are not easier than less-similar examples. We do this by compar- ing examples from GLUE (Wang et al., 2019) and BIG-bench Lite with all of C4 and the Pile. Fourth, we find that the similarity measures are not even correlated with each other. We conclude with sev- eral examples that show the limits of similarity's predictive power. Although similarity is common post-hoe justification for performance, we find it is misleading. This indicates the importance of model-based interpretability approaches. like train- ing data attribution (Akyürek et al., 2022).!
proxy for similarity to pretraining data (Gonen et al., 2022), though it poorly calibrated for down- stream tasks (Zhao et al., 2021; Ren et al.. 2022). Training data attribution uses influence functions, gradients, and model internals rather than data simi- larity (Pruthi et al., 2020; Akyürek et al., 2022; Han and Tsvetkov, 2022). Other approaches quantify task difficulty but do not explicitly account for the role of pretraining data (Ethayarajh et al., 2022; Yauney and Mimno, 2021; Swayamdipta al., 2020; Le Bras al., 2020; Perez et al., 2021).
Similarity correlates with performance in a controlled setting
The similarity hypothesis implies that a down- stream dataset's similarity to the pretraining data should matter. As an initial check of this hypothe- sis, we construct multiple versions of task. The underlying task is held constant, but decrease the similarity of the data to the training set in a way that we expect to decrease LM performance. Datasets in multiple languages are a starting point because text in non-English languages is presum- ably less similar to English pretraining data. While popular English pretraining datasets do contain non-English text that enables some transfer to other languages, LMs trained on English do not perform as well on multilingual tasks (Blevins and Zettle- moyer, 2022; Lin al., 2022: Gao al., 2020).
2 Related work
We consider two tasks that have datasets in En- glish and several other languages, where the task is fixed across languages. For an explicitly easy task, we extend a Stack Exchange forum classification dataset ( (Yauney and Mimno, 2021) by translating it from English into multiple languages. For a harder task, we use XNLI (Conneau et al., 2018). De- tails are in Appendix A. We measure zero-shot per- formance of Pythia-6.99 (Biderman et al., 2023), which was pretrained on the Pile (Gao et al., 2020), large dataset of mostly English text.
The impact of pretraining data on downstream per- formance is an active area of research. Similarity to model's 's pretraining dataset is assumed to de- termine the model's performance ford et al., 2019; Brown et al., 2020; Gonen et al., 2022). Prior work has focused on repeated entities in arithmetic (Razeghi et al., 2022) and QA tasks (Liu et al., 2022; Kandpal et al., 2023). Finetuned downstream performance benefits from continued pretraining on a similar domain (Gururangan et al., 2020: Xie et al., 2023).² 2 Contemporaneous work quantifies task similarity to downstream training datasets (Yuan et al., 2023; Li et al., 2023). Task data leakage is an extreme case of data similarity, where the presence of task examples in the pretrain- ing dataset impacts downstream performance on that task (Dodge et al., 2021; Magar and Schwartz, 2022). LM perplexity has been assumed to be
tasks (Rad-
Figure 1 (top) shows that performance tracks the KL-divergence of unigram token distributions between the Pile and Stack Exchange datasets. The English dataset is the most similar dataset to the Pile and affords the highest performance. All other languages are much less similar to the Pile and have lower performances. Note that token distribution similarity does not allow fine- ine-grrineed comparisons between the languages with high KL-divergence. We also test whether performance smoothly de- grades as we titrate in a given amount of dissimilar-
'Code and data are available at: https:///ithu/ com/ gyauney/dat wa--imilarity-ss-meee "We tind finetuned performance does not correlate with vocabulary overlap for the data from Gururangan et al. (2020).

---

Stack Exchange forum classification
Pythis-
TE vI.1
Flan-15
Normalized score
80
English
Ochioe
Oodbot
2-hhot
Oohic
2-shot
Rigne KI -ieierggnnn (-)
0006
-0. -0.10.s
-6.0fmm
-0.7 -0.64
70
Franch Germani Spanish
MAUVE scure (4)
0.19mm
0.7
0.20.
0,66am
60
Max
(+)
-1.10ow
-0.5ism
0:14umu =0.13
cosine similarty (+)
-0.5
-0.19
0.10wmm -0.10xs
50
Indonesian
Input parpledity (-)
-0 Br
-0.
-D.1 22ess
40
Swahill
Conrect lageet pesplesity (-)
0.44 can
(103 -0.12xxe -222aam -0,25
30
0.5
1.0
2.0
2.5
3.0
3.5
4.0
Table I: Spearman correlation coefficients between ag- gregate similarity measures and normalized score for BIG-bench Lite at the task level. p-values are in small font. +/- next to similarity shows whether more simi- lar datasets have higher/lower values. No correlation is significant p p 0.0017 after Bonferroni correction.
KL-divergence from pretraining taken distribution
XNLI
Normalized score
English
20
French
German
15
10
Spanis Swahll
5
training dataset, and (c) the mean cosine similarity between a downstream example and the closest 1 1,000 neighbors in the entire pretraining set. Third, language modeling loss of pretrained model eval- uated on a downstream example. We use the per- plexity of the input sequence and the perplexity of the correct target conditioned on the input.
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
KL-divergence from pretraining taken distribution
Figure 2: Mixed-language datasets show that perfor- mance smoothly decays as more of the dataset is trans- lated from English into another language. Each line represents a dataset that begins as 100% English, with increasing percentage of each document translated into another language, until completely translated. Er- ror bars are over samples of the pretraining dataset.
We further measure similarity at two different scales: 1) aggregate similarity between an entire downstream dataset and the pretraining dataset, and
2) example-level similarity between an individual downstream example and the pretraining dataset. Cosine similarities and LM loss are example-level, so we take the mean over a dataset's examples for aggregate similarity. See Appendix B for details.
ity. We construct datasets that interpolate between English and translation by translating an increas- ing percentage of each example. Figure 2 shows that decreasing similarity tends decrease perfor- mance for Stack Exchange datasets and the XNLI languages that are in the Latin alphabet. Figure 5 in the appendix shows the same trend for all of XNLI.
Models and datasets. We use models pretrained on two datasets: C4 (Raffel et al., 2020; Dodge et al., 2021) and the Pile (Gao et al., 2020). We use T5 v1.1 XL (Raffel et al., 2020), which was pretrained solely on C4, and Flan-T5 XI. (Chung et al., 2022), which was further instruction-tuned over 1,800 downstream tasks (Longpre et al., 2023a). For the Pile, we use Pythia-6. 6.9B (Bider- man al., 2023). Datasets and models are from the transformers and datasets libraries (Wolf et al., 2020; Lhoest et al., 2021).
4 Measuring similarity
Having established that distributional similarity cor- relates with performance in a setting that holds task difficulty constant, can we say the same across different benchmark tasks? And is performance higher for more similar examples? If the similarity hypothesis holds more generally, so that similarity to training examples can consistently predict perfor- mance on tasks, we expect that multiple measures of similarity will correlate with accuracy.
We test our hypotheses on GLUE tasks (Wang et al., 2019) and multiple-choice tasks from the more difficult BIG-bench Lite suite (Srivastava et al., 2023). We formulate tasks using finetuning for GLUE.3 3 For BIG-bench Lite, we use in-context learning, which does not require any parameter updates. Following Srivastava et al. (2023), we nor- malize performance across tasks that have different random baselines by rescaling accuracy to 0 for ran- dom chance and 100 for perfect. See Appendix C for details
Similarity measures. We implement three cate- gories of similarity measures on different text rep- resentations: First, KL-divergence of n-gram dis- tributions. Second, cmbeddinggbaased similarities using Sentence-T5 embeddings (Ni et al., 2022), including: (a) MAUVE score (Pillutia et al., 2021) of downstream dataset against samples of the pre- training dataset, (b) the maximum cosine similarity between a downstream example and the entire pre-
Pretrained-only performance is no better than random.

---

cosine similarity
corTEc
0.95
0.90
0.85
Dataset
Figure 3: Corsecly--lassifeed examples from BIG-bench are not consistently more similar to pretraining data based on Pythia-6.9B zero-shot performance and each example's 's maximum cosine similarity to any document in the Pile.
5 Results
Similarity to protraining data
Similurity finctuning data
Tasks that are more similar to the pretraining data do not have higher few-shot performance. We first use BIG-bench Lite to examine whether datasets that are more similar in aggregate to the pretraining data are easier for a given model. the similarity hypothesis is true, we expect to see significant high-magnitude correlations between performances and similarities to the correspond- ing pretraining dataset. Table shows the lack of correlation between five aggregate similarity mea- sures and few-shot performance on BIG-bench Lite tasks. No correlation statistically significant at the level of p < 0.0017 after Bonferroni correction for multiple tests (Dror et al., 2017).* 4 The strongest correlations are for KL-divergence and MAUVE with performance of Flan-T5 XI. the best perform- ing model. If these correlations were to hold in more targeted setting, then text-based similarities would be more promising than the embedding and model-based similarities we study. Appendix D discusses statistical considerations.
(90
0.1
-20
Figure 4: Finetuned T5 3B's performance on GLUE validation examples does not correlate with embedding similarity to the pretraining dataset (left) similarity to the finetuning training dataset (right).
shows this result for T5 performance and maximum similarity C4 documents. We also test the hypothesis in the finetuning paradigm by using embedding similarity and T5 3B finetuned on GLUE. Here we compare exam- ples from the validation split to both the pretrain- ing dataset and the GLUE training split. Figure 4 shows correlation between performance on example and that example's maximum similarity documents in the pretraining or finetuning datasets.
Discussion
Examples that are more similar to any pretrain- ing document are not easier. We can control for different task difficulties by focusing on individual examples within a dataset. If the example-level sim- ilarity hypothesis is true, then we expect to see that within task, examples that are more similar to the pretraining dataset are classified correctly more fre- quently than less similar examples. Figure 3 shows that BIG-hench Lite examples classified correctly by Pythia-6.9P 6.BB do not have significantly higher embedding cosine similarity to any document in the Pile than those that were misclassified. This holds for all the tasks we examined. Figure 6 in the appendix shows that accuracy of more similar examples is not consistently higher than accuracy of dissimilar examples. Figure 7 in the appendix "The prohability of any false positives is at most 0.05.
Our goal is evaluate the similarity hypothesis: that tasks that are more similar to pretraining data are casier. We have so far shown that similarity does not in practice correlate with performance for common benchmark tasks. In this process we found additional reasons to doubt the hypothesis.
Task difficulty can be orthogonal to similarity. Tasks can have language similar to the pretraining corpus and still be hard. To underscore that similar- ity is not sufficient to predict performance, we use another Stack Exchange classification task. Rather than predicting whether given post was from the bicycles or ostheory forum, instead choose a much more difficult task: predicting whether it was posted before or after noon. Again, we trans- late the dataset into a range of other languages. By construction, the similarity to the pretraining data

---

is nearly the same as in Figure 1, but the accuracy is random (Figure 8, Appendix).
MALVE
MALVE sone
-0.56
Max codine sintarry
-1.31
-0.45
lepa perploity
-1.19
0.20
-0.31
Similarity measures uncorrelated. Do dif- ferent similarity measures capture the same notion of similarity? Surprisingly. different measures of aggregate similarity between downstream datasets and pretraining datasets are not significantly corre- lated with each other when comparing Lite tasks with C4 and the Pile (Table 2). These results indicate that similarity is not monolithic concept. Future work could explore a) the affor- dances of these different kinds of similarities and b) whether more refined similarity measures might better correlate with performance.
Coenact lageci perplocty
-1.42
0.30
-0.02
The
Ki-ddeeresce
MAI VI sooe
Mox crsice siniliity
Inper perplesit
MAI VP scoe
-0.4
Mas comne similariy
-0.34
lnpor perplouty
0.61
Coneet lagelt perplosity
0.22
Table 2: Aggregate similarity measures are not signifi- cantly correlated with each other when comparing BIG- bench Lite datasets with C4 and the Pile. Perplexi- ties are from T5 v1.1 XL and Pythia-6.9B, respectively. Spearman correlation coefficients with p-values.
model and operationalization of similarity that we tried. While theoretical work suggests that the sim- ilarity between training and test distributions plays a role in upper-bounding error for LMs applied to downstream classification tasks (Saunshi et al., 2021) and in more general settings (McNamara and Balcan, 2017), we find that "similarity" is surpris- ingly difficult to measure in practice and that other factors may have more impact on difficulty.
Scope of our findings. We do not disprove the similarity hypothesis. One possibility that all datasets in BIG-bench Lite and GLUE are so sim- ilar to C4 and the Pile that similarity differences are not great enough to impact performance. Our experiments are models up the 7B 7B-parameter scale. Perhaps there is a model scale threshold at which similarity matters more for performance. is clear that at least some level of performance vari- ation is necessary to see similarity matters. For example, a model that performs poorly on all tasks (or conversely if it solves all tasks perfectly) will not be able to show any meaningful influence of similarity, And while we evaluate a broad range of similarity measures, they do not capture all possible similarities. Rather, results make it increasingly unlikely that single most determinative factor in this setting.
Future work. Future work can study the advan- tages of training data attribution over raw similarity, as begun by Mozes et al. (2023). Pretraining on datasets constructed to be more or less similar to specific downstream tasks might also be able to determine if there is a causal relationship between similarity and performance.
construed the
Limitations
Embedding comparisons between downstream datasets and pretraining datasets are computation- ally expensive due to the large size of the latter. We are therefore limited in how many downstream datasets we can compare to pretraining datasets. We strike balance by working with previously- studied similarity measures and standard diverse benchmarks like BIG-bench, though this topic would benefit from additional datasets and mod- els. We must control for multiple comparisons when testing the significance of correlations, reduc- ing the statistical power of our experiments. More targeted work could investigate specific kinds of similarity. Our experiments focus on the few-shot setting, and our results are not necessarily valid for all settings. Although our main results contradict conventional wisdom this setting, we have per- formed this work in order to provide a more solid foundation for new pretraining data analyses.
7 Conclusion
At broad level, it is clear that pretraining data mat- ters. Data curation can impact robustness, mem- orization, and toxicity generation (Nguyen al., 2022; Lee et al., 2022: Longpre et al., 2023b). Spe- cific examples of high-level task similarity, such as prevalence of entity mentions, have been ob- served to correlate with performance on arithmetic and QA tasks (Razeghi et al., 2022; Kandpal et al., 2023). We find that while similarity is correlated with performance when task difficulty can be kept constant, it is surprisingly hard to connect textual similarity to performance for standard benchmarks in the few-shot setting, Both distributional and fine- grained srample-basel similarity to the pretrain- ing dataset are not enough to explain performance across benchmarks, and this remains true for every

---

Ethics statement
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In EMNLP.
We do not anticipate any harms from our use of publicly available datasets. While large language models are carbon-expensive to pretrain, the few- shot paradigm that we focus does not require additional compute-intensive parameter updates.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL.
Acknowledgments
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. EMNLP.
We thank Daphne Ippolito. Katherine Lee, Dheeraj Rajagopal, and Lisa Schut for discussions and tech- nical help; Daniel Smilkov and Nikhil Thorat for help scaling embeddings to full datasets; and Lu- cas Dixon, Rebecca Hicke, Andrea Wang, Matthew Wilkens, and Ann Yuan for writing feedback. We also thank the anonymous reviewers.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint vrXiv2331..02.::
Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability analysis for natural language processing: Testing significance with mul- tiple datasets. TACI,, 5:471 186.
References
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022. Towards tracing knowledge in language mod- els back to the training data. In EMNLP Findings.
Kawin Ethayarajh. Yejin Choi. and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In ICMI..
Emily Bender. 2019. The #bbnderrule: On naming the languages we study and why matters. The Grudi- ent, 14.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, al. 2020. The Pile: An 800GB dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0000
Yoav Benjamini and Daniel Yekutieli. 2001. The con- trol of the false discovery rate multiple testing un- der dependency. Annals of Statistics, pages 1665- 1188.
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.0403:
Stella Biderman, Hailey Schoelkopf, Quentin An- thony, Herbie Bradley, Kyle O'Brien, Eric Halla- han, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In ICML.
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. ACL.
Terra Blevins and Luke Zettlemoye. 2022. Language contamination helps explain the cross-lingual capa- bilities of english pretrained models. In EMNLP.
Xiaochuang Han and Yulia Tsvetkov. 2022. ORCA: Interpreting prompted language models locating supporting data evidence in the ocean of pretraining data. arXiv preprint arXiv:2205. 20w.22205..00.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. arXiv preprint 20Xiv:2005.14100
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. ACL
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With littl power comes great responsibility. In EMNLP.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. ICML.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language mod- els. arXiv preprint arXiv:2210.1 11416.
Pang Wei Koh, Shiori Sagawa, Henrik Mark- lund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga,
2022. Scaling

---

Richard Lanas Phillips, Irena Gao, Tony Lee, Eti- enne David, lan Stavness, Wei Guo, Berton A. Earn- shaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Pency Liang. 2021. WILDS: A benchmark of in-the-wild distribution shifts. In ICML.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman. Se- woong Oh, and Ludwig Schmidt. 2022. Quality not quantity: On the interaction between dataset design and robustness of CLIP. In NewrIPS.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Con- stant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-TY T5: Scalable sentence en- coders from pre-trained text-to-text models. In ACL Findings.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhaga- vatula, Rowan Zellers, Matthew Peters, Ashish Sab- harwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In JCMI.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021, Rissanen data analysis: Examining dataset charac- teristics via description length. ICML.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang. Douglas Eck, Chris Callison-Buree and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In ACI.
Krishna Pillutla. Swabha Swayamdipta. Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, 7aid Harchaoui. 2021. MAUVE: Measuring the gap be- tween neural text and human text using divergence frontiers. NeurIPS.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond. Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: A community library for natural language processing. In EMNLP: System Demonstrations.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influence by tracing gradient descent. NeurIPS.
Moxin Li, Wenjie Wang, Fuli Feng, Jizhi Zhang, and Tat-Seng Chua. 2023. Robust instruction optimiza- tion for large language models with distribution shifts. arXiv preprint 2XXiv:3305.3955:
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenAl Blog.
Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, al. 2022. Few-shot learning with multilingual generative lan- guage models. In EMNLP.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. JMLR, 21(140):1-67.
Linqing Liu, Patrick Lewis, Scbastian Riedel, and Pon- tus Stenetorp. 2022. Challenges in generalization in open domain question answering. In NAACI. Find- ings.
Yasaman Razeghi, Robert I. Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In EMNLP Findings.
Shayne Longpre. Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, et al. 2023a. The Flan Collection: Designing data and methods for effec- tive instruction tuning. In JCML.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mo- hammad Saleh, Balaji Lakshminarayanan, and Pe- Liu. 2022. Out-of-distributioo detection and lective generation for conditional language models. arXiv preprint arXiv:2209.
Shayne Longpre, Gregory Yauney, Emily Reif, Kather- ine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2023b. A pretrainer's guide to training data: Mcasuring the effects of data age, do- main coverage, quality, & toxicity. arXiv preprint arXiv:2305. 13169.
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2021. A mathematical exploration of why language models help solve downstream tasks. In ICI.R
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176B -aB--pareettr open access multilingual language model. arXiv preprint arXiv:2211.05100
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. ACL.
Daniel McNamara and Maria-Florina Balcan. 2017. Risk bounds for transferring representations with and without fine-tuning. In ICML.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid. Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. 2023. Beyond the im- itation game: Quantifying and extrapolating the ca- pabilities of language models. TMLR.
Maximilian Mozes, Tolga Bolukbasi, Ann Yuan, Frederick Liu, Nithum Thain, and Lucas Dixon. 2023. Gradient-based automated iterative recov- ery for parameter-efficient tuning. arXiv preprint arXiv:2302.06598.

---

Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hannanch Hajishirzi, Noah A Smith, and Yejin Choi. 2020. Dataset cartography: Map- ping and diagnosing datasets with training dynamics. In EMNLP.
Latin alphabet. Figure 5 shows qualitatively similar results for all languages.
BIG-bench Lite. BIG-bench Lite is a subset of diverse tasks from the BIG-bench suite (Srivastava et al.. 2023). It consists of 18 multiple-choice and 6 text generation tasks. We report results using the multiple-choice tasks. We exclude the genera- tive tasks from our analysis because both Pythia- 6.9B and T5 v1.1 XL do not succeed beyond O performance on any of them. Results for the gen- erative tasks are included in the code repository. We exclude two tasks, misconceptions. susssin and parsinlu Bacpll_rreee_geeeeeeeeeee comprehension, that cannot be to- kenized by the T5 tokenizer. We also exclude winowhy from aggregate analysis.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE:: multi-lask benchmark and analysis plat- form for natural language understanding. In ICLR.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, et al. 2020. Transformers: State-of-the-an thr natu- ral language processing. In EMNLP: System Demon- strations.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Pency Liang. 2023. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169.
GLUE. We evaluate English-language language datasets from GLUE (Wang et al., 2019), with an emphasis on natural language inference. We use the valida- tion splits of MNLI, QNLI, QQP, SST2, and RTE.
Gregory Yauney and David Mimno. 2021. Comparing text representations: A theory-driven approach. In EMNLP.
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Revisiting out-of-distribution robustness in NLP: Benchmark, analysis, and LLMs evaluations. arXiv preprint arXiv:2306.04611
B Similarity metric details
n-gram baselines. We represent a dataset as a distribution over tokens and calculate the KL- divergence between a pretraining dataset and each downstream dataset. We represent text as bigrams hashed to a 10.000-dimmesioonal vector, as in Xie et al. (2023), and by unigrams from the Pythia to- kenizer. We sample 100,000 documents time from the pretraining dataset and compare against all of each downstream dataset.
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In ICML.
A Dataset details
Cosine similarity. We represent documents us- ing Sentence-T5, embeddings (Ni et al., 2022). For a sample of documents in a downstream dataset, we calculate a) the maximum cosine similarity be- tween downstream document and the entire pre- training dataset and b) the mean cosine similarity between downstream document and the closest 1,000 neighbors in the entire pretraining dataset. For GLUE, we sample 250 documents from each task. We sample up to 100 documents from each BIG-bench Lite task (some tasks have fewer than 100 documents). We observed similar results with BERT embeddings (Devlin et al., 2019). We find that the two variants of cosine similarity lated (p 0.95, < 10-4), is to be expected. Max cosine similarity can measure task data leak- age: since downstream examples are compared all pretraining documents, exact matches will have a maximum cosine similarity of I.
Stack Exchange. The Stack Exchange forum identification task provides an easy task (Yauney and Mimno, 2021). It consists of 1,000 English- language posts, and the task is to predict whether a post is from the bicycles or cstheory forum. We translate the posts into multiple languages using Google Translate. It also includes AM/PM labels for much more difficult task.
XNLI XNLI is a natural language inference task that also comes in many languages and is much more difficult in the zero-shot setting (Con- neau et al., 2018). We use the validation set of 2,490 documents in each language. We con- struct intermediate translations by translating the first (25%, 50%, 75% of each example in English XNLI into the other XNLI languages using Google Translate. Figure shows languages that are well- tokenized by the Pythia tokenizer, i.e. those in the
corre-

---

Arabic
Bulgarian
Chinese
French
German
Greek
Hindi
20
20
Normalizzd score
2
Russian
Spanish
S Swahili
Thai
Turkish
Urdu
Vietnamese
23
KL-divergence from pretraining token distribution
Figure 5: Full XNLI results for all languages show the trend seen in Figure for Latin-alphabet languages. Each line represents a dataset that begins as 100% English (gray dot in the upper left), with an increasing percentage of each document translated into another language, until completely translated (blue dot in the lower right).
MAUVE score. MAUVE is a recent text similar- ity measure that treats text dataset distribution over embedding space (Pillutla al., 2021). We again represent documents using Sentence-T5 em- beddings and calculate average MAUVE scores between each downstream dataset and 10,000- document samples from the pretraining dataset.
Pythis-699
T5 v1.1 XL
Flan-T5
sbot
Rignen Klldfeergener (-)
-0.0
0.5
0.02cso
-0.55
-0.81
MAUVE score (+)
-0.26
-0.10
D.4llmm
0.32cs
Mas cosine similarty (+)
0.0
-0.31
-0.32
0.05..
Mean similariy
-0.10
-0.29ka
0.0
-0.13
perplesity
-016Ga
tanget perpleiit (-)
-0.0 -0.10 -0.16
Table 3: Pearson correlation coefficients between ag- gregate similarity measures and normalized score for BIG-bench Lite at the task level. p-values in small font. No correlation is significant at < 0.0017 after Bonferroni correction.
Perplexity. LM log-loss per target token (which is log-perplexity for autoregressive models) on the pretraining validation set has been theoretically shown to impact downstream performance (Saun- shi et al., 2021). We use log-loss per token on the downstream text as possible proxy for similarity to the pretraining data, in Gonen et al. (2022). We use the mean across examples of input perplex- ity and correct larget perplexity, i.e. the perplexity of the correct target conditioned on the input.
D Statistical considerations
Multiple comparisons. We seek to test the cor- relation between multiple measures of similarity and multiple performance measures. Using a sig- nificance value of < 0.05 for rejecting null hy- potheses would give large overall false positive rate. We use Bonferroni correction, standard method for controlling for multiple comparisons (Dror al., 2017). However, Bonferroni correction has been found to decrease statistical power, i.e. makes false negative results more likely. More complicated corrections that do not sacrifice as much power, like the Benjamini-Yckutien proce- dure for possibly dependent hypotheses (Benjamini and Yckuticli, 2001), still do not yield significant correlations (see code for details). Future empirical work would benefit from studying more datasets in order to increase the statistical power of experi- ments (Card et al., 2020). The code also includes equivalent analysis using p-values calculated from permutation tests.
C In-context learning details
In-context learning, also called the zero-shot or few-shot setting, formats classification tasks as nat- ural language questions (Brown et al., 2020; Dong et al., 2022). A task is defined by prompt that includes several parts: an instance-specific text in- put optional instruction i that provides more context for the desired output, and a closed set of targets T that defines the possible choices. The predicted choice ò is the target with the highest probability: ý = arg max,e7 P9 (t x o i), where o is the concatenation operator and po is the condi- tional distribution over words for the model with parameters 0.² A few-shot prompt includes one or more examples of desired input/output tuples that are prepended the input. We do not any finetuning and instead use the T5 3B model, which was finetuned on GLUE. Prompts and raw results are included the code repository.
Pearson correlation. In the main paper, we port Spearman rank correlation, which measures the degree to which variables are related by a (pos- sibly nonlinear) monotonic function. Pearson cor- relation is also commonly used to quantify linear
*Other scoring methods yield qualitatively similar results.

---

That
Dataset
Figure 6: BIG-bench Lite examples that are more similar to pretraining data do not consistently have higher accuracy than less similar examples based Pythia-6.9B zero-shot performance and each example's maximum cosine similarity to any document in the Pile.
Max cosine similarity
1.00
panvesot prodictoon
0.95
0.90
4
0.85
Dataset
Figure 7: Corsecly-classifed examples from BIG-bench are not consistently more similar to pretraining data based T5 v1.1 XL zero-shot performance and each example's 's maximum cosine similarity to any document in C4.
relationships. Table 3 shows Pearson correlation coefficients for the same comparisons as Table 1.
Stack Exchange AM/PM classification
Normalized score
20
15
As before, none of these are significant after Bon- ferroni correction or the Benjamini- Yekutieli cedure.
10
- pro-
English
Spainish Finnien Swahill
Germen Indenesiam
E Infrastructure and runtime
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
KL-divergence from pretraining token distribution
Zero-shot experiments were run Intel Xeon CPU E5-2665 @ 2.40GHz with 250 GB of RAM and an NVIDIA RTX A6000 GPU. Runtimes for BIG-bench Lite tasks ranged from 2 sec- onds for dett_deeggging to 6.3 hours for vitam- tal_factterfficattinn The whole benchmark suite took 17.1 hours. Evaluating Stack Exchange datasets took 18.2 minutes on average for total of 7.5 hours. Evaluating XNLI took an average of 10.4 minutes per run for total of 2.9 hours for Latin-alphaber datasets and 8.7 hours for all datasets under various degrees of translation. Eval- uating GLUE took from 4.3 minutes for RTE to 9.3 hours for QNLI, for a total of 18.3 hours. Calculating embedding cosine similarities be- tween downstream document and the entirety of a pretraining dataset is time-consuming. Sentence- T5 embeddings are diseeimension.. When com- paring one downstream example with all of C4, this can be calculated as a matrix-vector product between a matrix of dimension 365, 049,, 127 x 768 and diseeenensioall vector, followed by taking the maximum over the resulting vector. We paral-
Figure The same Stack Exchange datasets from Fig- ure 1 are much more difficult when predicting whether the post occurred in the morning or afternoon. These datasets have nearly the similarity the pretrain- ing dataset as their counterparts but are more difficult.
lelized this computation using distributed SQL server and achieved runtimes of 90 seconds per example when comparing with the Pile and 74 sec- onds per example when comparing with C4. All other similarity metric runtimes were either less than one minute or calculated part of evaluation.
F Additional figures
Figures 6, 7, and give additional results referred to in the main paper.


Reviews:
[{"review_id": "ILMCtKUwJh", "reviewer": "Reviewer_tQTb", "summary": "", "questions": "I would like to take this opportunity to congratualte the authors on a very nice piece of concise and thoughtful work.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We really appreciate your positive and constructive feedback.\n\nThanks for pointing out that lines 61-64 are confusing. We will rephrase the sentence to say: “First, we show that similarity correlates with performance for multilingual datasets. The same underlying task is represented in different languages, each with a different aggregate similarity to the pretraining dataset.”\n\nWe will also add an ethics statement that discusses working with multilingual datasets and the role of computation in this research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is very well written and tackles an important, underexplored problem and provides surprising new insights. In particular, it shows that similarity is not a one-size-fits-all explanation to model performance and exposes faults in this way of reasoning about model performance. This work provides valuable new insights to the community and should be accepted as is.", "reasons_to_reject": "This paper is strong and runs more experiments than I would have expected from a short paper. I do not see any reasons to reject it.", "questions_to_authors": "I would like to take this opportunity to congratualte the authors on a very nice piece of concise and thoughtful work.", "paper_topic_and_main_contributions": "This paper studies the effects of the similarity between pretraining data and finetuning data on downstream model performance. The authors show that, using several similarity metrics, performance may *not always* be correlated with the similarity between the pretraining and finetuning data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Please add an \"Ethics Statement\" section, although I do not see any ethical concerns."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Very well written. With only one grammatical point to make:\n1. Lines 61 to 64 the part of the sentence in which you say \"where we are able to change the similarity to an English pretraining dataset while keeping the underlying task fixed\" is rather difficult to parse. I would consider rewording it or splitting it into two sentences."}}, {"review_id": "p7NCCVYuSd", "reviewer": "Reviewer_ra9x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We really appreciate your thorough engagement with our work. We believe we can address most of your concerns in the text of the paper, and we will include a new results table to address your concern about assuming linearity.\n\n***Concern:** The paper’s conclusion that “similarity doesn’t explain performance” might be too broad, could be misinterpreted, and might prematurely discourage others from investigating further.*\n\nWe share your concerns, and we will directly try to prevent any misleading interpretations by adding a paragraph to the discussion about what we are not claiming. For example, we will state that we are not claiming that similarity cannot matter or that the similarity metrics we evaluate capture all possible similarities. We do not definitively disprove the similarity hypothesis, but we do present evidence that the kinds of similarity that we consider are not all that matter for performance in this setting.\n\nWe will more carefully frame the introduction, discussion, and conclusion to focus on the specific datasets and kinds of similarity metrics we consider. We will add open questions to the conclusion, like 1) whether more complicated metrics might capture the impact of similarity, 2) whether our results hold for more datasets, and 3) whether there is a model size threshold at which similarity does determine performance. We will also take your suggestion to “emphasize that while the correlation is not consistent across all tasks, it doesn't negate the potential influence of data similarity in certain contexts.” We definitely do not want to discourage more research on data similarity! We find that the impact of similarity is not as obvious as current literature makes it seem, and we hope future work can study more metrics, models, and tasks. Thank you for raising this concern.\n\n***Concern:** Lack of linear correlation does not mean that similarity is unrelated to performance.*\n\nWe agree and will state this in the paper. We have also performed Spearman rank correlation between similarity and performance, which measures the degree to which variables are related by a (possibly nonlinear) monotonic function. None of these correlations are significant after Bonferroni correction for multiple comparisons. We will discuss these results in Section 5 and add them to the appendix in a table that is similar to Table 1.\n\n***Concern:** We are making a strong claim about similarity based on a limited set of metrics and tasks, and the results might not hold in other settings. The similarity measures are simple and may not capture the true relationship between pretraining and task data.*\n\nWe will state that our results should be read in the context of these specific datasets and similarity measures. We agree that our results will benefit from future work that studies the same questions for additional datasets and more complex similarity metrics, and we will highlight such steps in our future work section. Although each similarity metric is relatively simple, we chose to study these because they are used in practice today. If similarity is so hard to pin down, then our results make it increasingly unlikely that similarity broadly construed is what determines performance in this setting.\n\n***Concern:** Task difficulty is not a constant factor–difficulty can be influenced by many things, not just data similarity.*\n\nThis is a great point! This is one of our main takeaways, so it is helpful to hear that it needs to be more explicit. We will devote more of Section 6 to this discussion, and we will explicitly state in the conclusion that there are many contributing factors to difficulty. We will also directly state that the results paragraph that starts in line 212 (with results in Figures 3, 4, and 5) controls for task difficulty by comparing individual examples within a dataset. One goal of this paper is to broaden the discussion of task difficulty beyond similarity.\n\n***Reproducibility:** The paper can only be reproduced with difficulty.*\n\nWe will release code for reproducing our experiments, an initial version of which is currently available in the supplementary zip file. We will also release our raw similarities and performances to facilitate further analyses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths of the Paper:\n\n1. Innovative Challenge: The paper questions the widely held \"similarity hypothesis,\" adding a fresh perspective to the NLP community.\n\n2. Thorough Evaluation: Rigorous analysis using diverse similarity metrics, datasets, and benchmarks enhances the study's robustness.\n\n3. Well-Designed Experiments: Controlled scenarios with varied similarity conditions provide valuable insights into the complex data-performance relationship.\n\n4. Surprising Insights: Counterintuitive findings highlight that data similarity might not solely dictate performance, sparking re-evaluation.\n\n5. Broad Applicability: Accepted publication would prompt researchers to rethink language model design and fine-tuning strategies.\n\n6. Engagement with Trends: Aligns with ongoing discussions about model interpretability, pretraining data, and generalization in NLP.\n\n\nBenefits to the NLP Community:\n\n1. Challenging Assumptions: Encourages a critical rethinking of established beliefs about data similarity and performance.\n\n2. Enhanced Model Design: Insights can guide more robust model development, leading to better fine-tuning strategies.\n\n3. New Research Paths: Sparks exploration into training data attribution and causal relationships, advancing the field.\n\n4. Improved Interpretability: Promotes model-based interpretability approaches, enhancing model transparency.\n\n5. Community Dialogue: Ignites valuable discussions and collaborations around the nuanced data-performance relationship.\n\n6. Progressive NLP Advancement: Adds to evolving NLP research by questioning norms and pushing boundaries.", "reasons_to_reject": "Potential Weaknesses:\n\n1. Limited Scope of Datasets: The paper focuses on specific benchmarks and tasks, which might limit the generalizability of the findings. The results might not hold true for other tasks or datasets.\n\n2. Simplistic Similarity Measures: The paper uses several measures of similarity, but they might not fully capture the complexity of the relationship between pretraining and task data. More nuanced measures could provide a more accurate representation.\n\n3. Assumption of Linearity: The paper assumes that similarity is a linear predictor of performance, which might not be the case for all tasks and models. Non-linear relationships could influence the findings.\n\n4. Complexity of Task Difficulty: The paper considers task difficulty as a constant factor, which might oversimplify its complexity. Task difficulty can be influenced by multiple factors beyond data similarity.\n\nPotential Risks and Concerns:\n\n1. Misinterpretation of Results: There's a risk that the findings could be misinterpreted, leading to the incorrect assumption that data similarity doesn't matter at all for LM performance. The paper should emphasize that while the correlation is not consistent across all tasks, it doesn't negate the potential influence of data similarity in certain contexts.\n\n2. Critique of Methodology: Reviewers might question the validity of the chosen similarity measures, dataset selection, and the specific LM models used. There's a risk that the methodology might be deemed insufficient to definitively disprove the similarity hypothesis.\n\n3. Controversial Conclusions: The paper's conclusion that \"similarity doesn't explain performance\" might generate controversy and skepticism within the NLP community. It's essential to carefully frame the conclusions to avoid misleading or polarizing interpretations.\n\n4. Impact on Future Research Direction: The paper's findings might discourage researchers from exploring the relationship between data similarity and LM performance further. Authors should emphasize that the results warrant deeper investigation into more diverse contexts and tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates whether data similarity alone explains language model (LM) performance on different tasks. It challenges the common belief that tasks similar to LM's pretraining data should perform better. The paper contributes by:\n\n1. Questioning the Similarity Hypothesis: Testing and disproving the idea that data similarity is the main determinant of LM performance.\n\n2. Correlation Analysis: Evaluating various similarity measures and their correlation with LM performance on different benchmarks.\n\n3. Benchmarking and Generalization: Assessing LM generalization across tasks and languages using diverse benchmarks.\n\n4. Exploring Task Difficulty: Showing that similarity doesn't consistently predict easier examples.\n\n5. Advocating Model-Based Interpretability: Highlighting the importance of alternatives to data similarity for explaining LM performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jjFUe3fBqL", "reviewer": "Reviewer_pNy4", "summary": "", "questions": "[A] Is there a reason why the MAUVE scores aren't included in the comparison in Table 2?\n\n[B] It is interesting that in Table 1, the best correlations come from the best-performing model on KL-divergence and MAUVE. How do you think increasing model scale (and in theory model performance) will affect these results? I think it would be helpful to add a discussion of this to the limitations/discussion section.\n\n[C] Line 58: Could you define the phrase \"out-of-model embedding similarity\"?\n\n[D] The multilingual XNLI dataset covers 15 languages. Why did you only evaluate on five of these languages (including English), and why did you choose these five?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We really appreciate your thorough and constructive feedback.\n\n***Concern:** The discussion of languages other than English is inconsistent with prior work. ‘English’ pretraining datasets contain some non-English text, and some language models perform well multilingually despite being trained on ‘English’ datasets.*\n\nThank you for pointing this out and for pointing us to the references. We will update the text to state this and engage with these references. We will also state in the paper that the goal of the multilingual experiments in Section 3 is to use datasets that are more and less similar in aggregate to the pretraining datasets, which the x-axis in Figure 2 shows is the case.\n\n\n***Concern:** Task data leakage is not addressed.*\n\nThank you for directing us to this line of work. We will add it to our introduction and related work sections, from the perspective that leakage is an extreme case of item-level similarity. We will state in Section 4 how the ‘max cosine similarity’ metric is intended to explore this–downstream examples are compared to all pretraining documents, so exact matches will have a maximum cosine similarity of 1. We will highlight in the results how even high max cosine similarity does not guarantee high performance.\n\n\n***Suggestion:** Further discuss why similarity measures are not correlated with each other in the main paper.*\n\nWe will move Table 2 to the main paper and significantly expand our discussion using the extra page. We will also explain in more detail what each similarity is measuring at the beginning of Section 4.\n\n\n***Question A:***\n\nThanks for pointing out that comparisons with MAUVE score are missing from Table 2. This was an oversight, and we will add them in the final version. None of the correlations with MAUVE are significant.\n\n***Question B:***\n\nWe will expand on this observation in the results section and add to future work the question of whether there is a model scale threshold at which similarity does determine performance. We hypothesize that our results will be robust to model performance improvements since we study models with a range of performances across many different tasks, and we will add this to the discussion/limitations sections. At least some level of performance is necessary to see if similarity matters. For example, a model that performs poorly on all tasks (or conversely if it solves all tasks perfectly) will not be able to show any meaningful influence of similarity.\n\n***Question C:***\n\nBy ‘out-of-model embedding similarity’ we mean that texts are represented using embeddings from a model that is not necessarily the model used for prediction. We then calculate cosine similarity using these embeddings. We will replace this description in the paper.\n\n***Question D:***\n\nWe chose to evaluate languages from XNLI that are tokenized well by the Pythia tokenizer. In practice, this meant the five languages that use the Latin alphabet with few diacritics. The Pythia tokenizer is sufficiently targeted at the Latin alphabet that it breaks words from languages in most other scripts into sub-character-level byte tokens. We will state the selection criteria in Section 3 and discuss this limitation in the Limitations section. At your suggestion, we have run additional experiments on all languages in XNLI, and the results are nearly identical to those in Figure 2. We will include a figure in the appendix that shows all languages.\n\n\n***Reproducibility:** The paper can only be reproduced with difficulty.*\n\nWe will release code for reproducing our experiments, an initial version of which is currently available in the supplementary zip file. We will also release our raw similarities and performances.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an interesting research question that addresses a common assumption made in LLM research (which is that similarity to the pretraining data explains why models perform well in-context on some tasks/examples but not others).\n- The experiments presented in the paper are generally sound, well-thought-out, and support the claims made in the paper.\n- The findings in the paper are interesting for the field and conflict with the intuitions held about how in-context learning works, which suggests we should further investigate why commonly-used text similarity measures do not relate to downstream performance.", "reasons_to_reject": "I see no major reason to reject this paper. However, there are some minor issues that would strengthen the work if addressed:\n\n- The discussion of LM generalization to different languages is inconsistent with the prior work in the area. For example, [1] shows that multiple \"English\" pretraining datasets do contain non-English text, which allows models trained on them to perform tasks outside of English; this paper specifically finds that this is true for C4. The other pretraining dataset considered, the Pile, is also estimated to be 2.6% non-English text [2, Section 5.2]. [3] also shows that GPT-3 performs well multilingually in some cases, despite not being explicitly multilingual. It would be helpful to rewrite the claims about multilinguality in LMs and their pretraining corpora to be more consistent with findings in this area.\n\n- It would also be useful to flesh out the discussion of why the chosen similarity measures are not correlated with each other. The text could also expand on what each metric measuring in the main body (and to move Table 2 to the main body of the paper to accompany this discussion).\n\n[1] Blevins and Zettlemoyer. 2022. Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models. In EMNLP.\n\n[2] Gao et al. 2020. The Pile: An 800GB dataset of diverse text for language modeling. arXiv.\n\n[3] Lin et al. 2022. Few-shot Learning with Multilingual Language Models. In EMNLP.", "questions_to_authors": "[A] Is there a reason why the MAUVE scores aren't included in the comparison in Table 2?\n\n[B] It is interesting that in Table 1, the best correlations come from the best-performing model on KL-divergence and MAUVE. How do you think increasing model scale (and in theory model performance) will affect these results? I think it would be helpful to add a discussion of this to the limitations/discussion section.\n\n[C] Line 58: Could you define the phrase \"out-of-model embedding similarity\"?\n\n[D] The multilingual XNLI dataset covers 15 languages. Why did you only evaluate on five of these languages (including English), and why did you choose these five?", "paper_topic_and_main_contributions": "This paper asks whether (semantic) similarity to the pretraining corpus is correlated with model performance and tests this at both the task and example level. Specifically, the paper considers the overlap of both coarse-grained similarity (different languages) and finer-grain similarity within English benchmarks -- text classification, (X)NLI, and BigBench tasks -- against two open-source pretraining corpora: the Pile and C4; it tests whether these similarities are correlated with the performance of models (Pythia-6.9B and T5 3B) pretrained on these corpora. The experiments show that (a) for the multilingual evaluation, similarity to the pretraining data is correlated with downstream performance on that language and (b) this correlation does not hold for the finer-grained similarity evaluations within English. They also show that the different similarity metrics considered are not correlated with one another.\n\nThe primary contribution of this paper is that it tests the theory from in-context learning research that similarity to the pretaining data leads to better performance on that task and demonstrates that this is not always the case.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "A related area of reseach that isn't discussed in the paper is on task data leakage, or the presense of task examples in the pretraining dataset. This is a very specific case of data similarity where the similarity is exact match. Some examples of work in this area are [1], which audits the presense of task data in C4 (and is already cited in this work), and [2], which discusses the effect of task data presense during pretraining on downstream performance. Many pretraining papers also discuss this in their work (e.g., the GPT-2 and GPT-3 papers).\n\n[1] Dodge et al. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In EMNLP.\n\n[2] Magar and Schwartz. 2022. Data Contamination: From Memorization to Exploitation. In ACL."}}]

Author rebuttal:

Meta-review: This paper studies the effects of the similarity between pretraining data and finetuning data on downstream model performance. They find that the downstream performance may not correlate with the data similarities. The findings are interesting and insightful to the research community and experiments are thorough and extensive. Reviewers raised some concerns regarding the phrasing and discussions in the paper. Authors addressed these concerns very well. Thus I recommend accepting this paper to main conference.
Decision: Accept-Main