Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
Authors: Alon Jacovi
Avi Caciularu
Omer Goldman
Yoav Goldberg
Venue: EMNLP 2023 Main (2023)
Keywords: data contamination
contamination
evaluation
test data
benchmarks
closed models
pretraining
TL;DR: Strategies (to be enacted by eval researchers) designed to prevent data contamination in the training of large language models

Abstract: Data contamination has become prevalent and challenging with the rise of models pretrained on large automatically-crawled corpora. 
For closed models, the training data becomes a trade secret, and even for open models, it is not trivial to detect contamination.
Strategies such as leaderboards with hidden answers, or using test data which is guaranteed to be unseen, are expensive and become fragile with time. Assuming that all relevant actors value clean test data and will cooperate to mitigate data contamination, what can be done? 
We propose three strategies that can make a difference: 
(1) Test data made public should be encrypted with a public key and licensed
to disallow derivative distribution;
(2) demand training exclusion controls from closed API holders, and protect your test data by refusing to evaluate without them; 
(3) avoid data which appears with its solution on the internet, and release the web-page context of internet-derived data along with the data.
These strategies are practical and can be effective in preventing data contamination.

Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating
Data Contamination by Evaluation Benchmarks
Alon Jacovi!
Avi Caciularu Omer Goldman'
Yoav Goldberg1.
Bar Ilan University
Google Research
3 Allen Institute for Artificial Intelligence
ponnjacovi@gmaal.coomm
Abstract
unseen during training (Brown al., 2020). In- deed, signs show that such models were exposed test data (Dodge ct al., 2021; Magar and Schwartz, 2022), and we refer to this as data contamination.
Data contamination has become prevalent and challenging with the rise of models pretrained on large automatically-crawler corpora. For closed models, the training data becomes trade secret, and even for open models, it is not trivial to detect contamination. Strategies such leaderboards with hidden answers, using test data which is guaranteed to be unseen, are expensive and become fragile with time. As- suming that all relevant actors value clean test data and will cooperate to mitigate data con- lamination, what be done? We propose three strategies that can make difference: (1) Test data made public should encrypted with a public key and licensed disallow deriva- tive distribution; (2) demand training exclusion controls from closed API holders, and protect your test data by refusing to evaluate without them; (3) avoid data which appears with its so- lution the internet, and release the web-page context of internet-derived data along with the data. These strategies are practical and can be effective in preventing data contamination.
The above issue of internet-crane crawled training data is one of two prominent scenarios of data contam- ination we consider in this work. The second is in the access to closed models via APIs. Such models are frequently used in research for various purposes (Wei et al., 2023; Qin et al., 2023; Moor et al., 2023), and thus they are also evaluated (Sri- vastava et al., 2022; Bubeck et al., 2023). In most cases, the institution behind the API reserves the option to use the data sent to them as training data in further iterations. In this case, any valuable evaluation data sent to closed API models for any purpose is potentially compromised for any subse- quent evaluations using the same data.
The NLP evaluation community now witness to two urgent crises: Data contamination in training data crawled from the internet, and training data collected from calls to a closed API. The implica- tions are severe-not only is much of our evalua- tion methodology potentially compromised, but we also cannot fully identify the scope and magnitude of the contamination,
1 Introduction
Common NLP models today are large language models trained on data crawled from the internet (Raffel al., 2020; Bommasani et al., 2022: Tou- vron et al., 2023). This data is often malformed or obfuscated in ways that make difficult to audit at scale (Bommasani et al., 2022; Mitchell et al., 2023). In particular, evaluation data that is also available on the web may be used as part of train- ing, and it can be challenging to verify whether it used in practice not (OpenAI, 2023; Google, 2023a).' Worse, for many closed models, training data considered a trade secret and thus unknown the research community. Such models are be- ing evaluated on data that cannot be certified to be
for open models.
We engage with the two crises by outlining three practical strategies that individual researchers can enact to protect the integrity of their evaluations:
Strategy 1: Protect data from automatic crawlers using public key encryption and a license that forbids distribution of adaptations ("no derivatives").
'E.g. OpenAl's GPT series (Brown al., 2020), Mo- saicML. Inference (MosaicML., 2023), and Google's Bard and Pal.M API (Gongle, 2023b).
'OpenAl currently provides exclusion guarantees for cer- tain API calls as of March 1, 2023. This guarantee does not extend to data sent before this date or to data sent through the ChatGPT and DALL-E Labs interfaces. Google's 's Bard pro- vides no exclusion guarantee exists as of this writing. Sources: openai, ppenai.com//a com/polia htt/policiae ccies//ppp aaaaappppatttae papage-pllion poli colicies bard. google.com
For example, OpenAl (2023) found that the BIG-Bench benchmark (Srivastava et al., 2022) was compromised to an extent that prevented its usage entirely.

---

Strategy 2:
Withhold from evaluating APIs that give no training exclusion options.
business interests rather than adversarial sabotage. Therefore, we assume that all actors are sincere in their desire to keep evaluation data outside of model training, and leave research on strategies against adversarial actors to others.
Strategy 3:
Avoid data that appears with its solution on the internet. If the data originates from the internet, release its context with it.
Assumption 3 (contamination inheritance): Mod- els that were trained on data derived from other models (Jiao et al., 2020; Taori al., 2023; Chiang et al., 2023; Hsich et al., 2023, inter alia), models that are using the weights of other models (Sun et al., 2020; Ganesh et al., 2021; Choshen et al., 2022), and ensembles of other models (Wortsman et al., 2022a,b) will all be considered as contami- nated as their "ancestors". This applies even when the ancestor was used to train only part of the model (e.g., Gao et al., 2023).
2 Setting
We consider two independent scenarios of data con- tamination, and three assumptions underlying our mitigation strategies:
Scenario 1 (internet crawled corpora): The model be evaluated is based on training set derived automatically from the internet. The training set is closed or large enough that is difficult to exhaus- tively and routinely search for all possible instances of data contamination from public test sets (both exact and approximate matches).
Why is Data Contamination Prevalent?
Note that we include cases in which the model potentially trained some form of the data's 's solu- tion, even if it was not trained on the data verbatim. For example, in the task of sentiment classification of Amazon product reviews (Johnson and Zhang, 2014), the answer (the review's rating) is available on the internet, and possibly compromised in train- ing, even if the dataset was processed and its final form was not available for the model.
3.1 Closed models give guarantees or controls about held-out data
Models with private training datt-whether the models themselves are closed or not make it im- possible to know if they were trained on particular evaluation data.* 4
Even when using test data that is guaranteed to be unknown to closed API model (e.g., by using data created after the model was last updated by using data before is publicly released), on the first moment that this data is used to evaluate the closed model, it ceases to be unknown, and there is cur- rently no standardized training exclusion controls by API holders. Furthermore, for much of evalua- tion research that evaluates models before the data is publicly relcased, the API holders will not know whether the data being sent to them belongs to would-be evaluation set or not.
Scenario (closed API models): The model to be evaluated a closed model behind an API, and there is no global or conditional guarantee of ex- clusion from future training. Any API call which contains test data compromises that test data for all models under the API holder and their affiliates.
Assumption 1 (presumption of contamination): all cases, if it is possible that the training data has been contaminated, we assume that it is. In other words, if some test data is accessible to au- tomatic internet crawling systems, we consider it compromised. If it is possible that the API holder is using test data from API calls in training, con- sider it compromised. The strategies in this work are designed to protect the evaluation under this strict condition.
3.2 Even with open data, detection is difficult
Even for models whose training data is known, can be challenging to understand exactly what they were trained on or to filter out test data be- fore training (Mitchell et al., 2023). The scale of the corpora and the rapid pace of model develop- ment and evaluation development hinders thorough checks. Additionally, repurposing data from the
Assumption 2 (sincere actors): We are concerned with setting where the evaluators and model de- velopers are non-adversarial- all actors appreciate clean test data and will not seek "cheat" evalua- tions, as they share an incentive reliably prove the value of their work.
"There research on methods for deriving whether partic- ular data existed a model's training from the model alone (Carlini et al., 2021, 2023a: Chang et al., 2023). However. such methods are fragile to false negatives.
'Some efforts exist to provide tools for auditing training data: F.g., Marone and Durme (2023); CeaSca (2023); Piktus et al. (2023).
The challenging scenarios in Section 2 primar- ily stem from a lack of resources and conflicting

---

internet is common when constructing evaluation data-some of which revoke exact-match or ngram- match detection. Fuzzy matching expensive to run at scale and must be performed routinely be- tween every evaluation set and pretraining dataset. Although exhaustive checks are rare in practice, even when performed, false negatives and false positives are still possible. As noted by OpenAl (2023), who ran a partial exact match, false neg- atives will manifest when a slightly modified ex- ample is in the training data. This issue persists for fuzzy matching, which works based on assump- tions (Ukkonen, 1985) and only covers specific cases (Myers, 1999). Data contamination that is manifested in any way that stealthily evades par- ticular fuzzy-match method will not be found. This is an inherent limitation of data contamination de- tection, and such cases occur in practice (Razeghi et al., 2022: Carlini et al., 2023b).
discontinued with time.
Responsibility of evaluation falls to the bench- mark host: In the case of automatic evaluation metrics, this poses no issue, but when evaluation is costly or time consuming-suu such as with human evaluation- the leaderboard host alone often must shoulder the costs of all evaluations (Khashabi et al.. 2022).
Creating (very) new data. Most trivially, it is possible to run the evaluation before any data is publicized. Additionally, models sometimes ad- mit some guarantees about the last possible date in which their parameters were updated (Touvron et al., 2023), or otherwise they can assumed to be frozen within some reasonable leeway (c.g., a minute or an hour). Using data which was only created after the model was last frozen is a strategy to guarantee that the data is unseen, even if the data is public (Liu al., 2023). Creating counterfac- tual versions (or other augmentations) of older data achieves the same purpose.
Finally, while detection is possible if done ex- haustively and routinely- this is reactive mea- sure, not preventative measure, for evaluators who have no control over the training data but full control of the evaluation data. The data which was compromised is to be discarded and replaced. The strategies we propose in this work, and in particular Strategy 1, are preventative.
Of course, this strategy is extremely inefficient: The guarantee vanishes for newer models so that the evaluation data soon loses relevance. This re- quires evaluation research to continue to create new data for every evaluation, in an expensive cat-and- mouse game.
3.3 Existing mitigation strategies are imperfect
4 Suggested Mitigation Strategies
4.1
Strategy 1: Enerypt test data with public key, and use a "No Derivatives" license
There are two strategies mitigate data contami- nation current practice:
Conditions: Scenario 1.
Live leaderboards with hidden answers. The answers test data are kept hidden behind an inter- face that only reveals the evaluation result. There are weaknesses with this approach:
This strategy is simple and cheap yet is an impres- sively potent guard against non -dvararrsalial crawl- ing of plain text test data in training corpora: Sim- ply upload the test data after encrypting its con- lents, alongside the key used to decrypt it. A simple method by compressing the data in a password- protected archive (c.g., with zip).
Partial mitigation: The test data itself, sans an- swers, is still available on the internet. This data can be automatically crawled, the hidden answers are risk of being compromised if the test data is repurposed (Scenario 1) or independently la- beled by the model developer behind closed doors without knowledge that it is benchmark data (Sce- nario 2). And finally, live leaderboards are tradi- tionally only applied to test sets, but development sets need protection from training to be effective, as well.
A license with a No Derivatives clause, such as "CC BY-ND 4. 4.0º²7 will protect the data from being redistributed without its encryption, while still being otherwise permissive."
"We randomly selected five datasets with leaderbourds from the Allen Institute for Artificial Intelligence's 's leader- board website. We found that an average of 4.2 models cited the datasets of these leaderboards, even though they did not submit their results to each corresponding leadetboard, opting use their development sets instead.
Rarely enforced: Live leaderboards are costly and time-consuming to maintain. In practice, this constraint proves to be significantly restrictive: They are rarely implemented, rarely used, and get
iecemmonnonnons org/ Licensss/byy-dee a./.0000 0/deed 8 "ee Section 5 for additional discussion on licenses.

---

Unlike the strategy of contamination detection, this strategy is preventative when employed by eval- uation practitioners since it stops the evaluation data from being compromised, to begin with.
an evaluation prompt which is compromised, the evaluation ceases to be representative.
Therefore, we should consider prompts with data as evaluation data: Avoid uploading them to the in- ternet in plain text (including inside papers). Since such prompts are relatively inexpensive to anno- tate, we should avoid re-using them at all when we suspect that they were compromised, and annotate new prompts instead.
Importantly the goal here is not to protect the data from adversarial actors (as in the case with live leaderboards), but to protect the data from au- tomatic crawling. Therefore, the encryption key can be safely released with the encrypted data, and the encryption method can be simple and fast, so long as it sufficiently distorts the text. However, we warn against using standard obfuscation or compression methods that are not key-protected, since some crawling systems include pipelines of automatic decompression or deobfuscation.
4.2 Strategy 2: Refusse to send test data to closed API models until exclusion controls are implemented
Conditions: Scenario 2.
Scenario is a strict scenario and difficult to guard against without cooperation from the API host. Since we consider the API host as a sincere actor that values reliable evaluation, there are incentives in place for evaluation practitioners to demand this cooperation and for the API hosts to comply.
Finally, online dataset hubs (e.g.. Hugging Face Datasets, Lhoest et al., 2021; ThoughtSource, Ott et al., 2023), should withhold from including test data in their online viewer directly, and specify when whether hidden data was previously avail- able in plain text." For showcasing the data, a sample of compromised representative examples can be presented instead.
Foremost, since the very first APl usage during evaluation compromises the test data, this strategy calls for not evaluating the closed API model until this situation changes in order to protect the in- tegrity of the data. This, in turn, pressures the API host to provide appropriate training exclusion con- trols in order to participate in evaluation practice and research. Mechanically, the API host may com- ply by implementing system to request exclusion from future training." !1
Strategy Corollary:
Few-shot prompts. Few- shot prompts are demonstrations used as training data. Although they are not regarded as evaluation data, they are used primarily to evaluate model gen- cralization under strict data constraints. Few-shot evaluation relies on the assumption that this small number of demonstrations will approximate model behavior on any small number of demonstrations, Few-shot prompts in the literature are commonly displayed in papers, in online repositories, and in online demos- in plain text. They appear along- side other prompts, other data, and other relevant information that should be considered unintended for their original purpose. These prompts are often reused many times in subsequent work, and thus appear many times on the internet in "unnatural" contexts more-so related to NLP research (e.g.. the prompts by Wei et al., 2022). We consider such prompts as compromised. When a model is given
As an intermediate strategy, in the absence of an exclusion guarantee, it is possible to prioritize collecting cheaper, "singlc-use" data that can be used for the purpose of a less-reliable evaluation estimate: (1) By holding out portion of training data, if it exists; (2) By creating synthetic variants of the data, or synthetic data altogether.
4.3 Strategy 3: Avoid data that appears with its solution on the internet
Conditions: Scenario 1.
In order to reduce data collection costs, data for training and evaluation is commonly repurposed from the internet." 12 The data labels are then ei- ther derived automatically from context (e.g., re- view score for product reviews, Rajeev and Rekha,
"In the case of Hugging Face Datasets, it is possible use "gated repositories" block test data from crawler access, although they
currently used for this purpose. Gating mechanisms and page metadata that requests to he crawled can hoth he used dcal with crawling for website hosts. Note that this approach does not prevent data from being redistributed to more vulnerable hosts.
"We leave details on the implementation of such a system the individual institutions, ease of conflicting business incentives, they may restrict exclusion controls to specific users approved for research or find other solutions. E.g. Wikipedia, PubMed, Twitter, Reddit, and so on.
10 "Attrrnatvely, more sophisticated tricks can be imple- mented to relax this issue, such as decrypting the data on mouse hover, or requiring the decryption key on every view- ing, but we leave such investigations to others.

---

2015; coreference resolution with Wikipedia entity links, Ghaddar and Langlais, 2016; news article summaries, Fabbri et al., 2019; emojis, Felbo et al., 2017) or manually labeled.
trivially compromised in Scenario 1 in a way that renders Strategies and 2 irrelevant (e.g., review sentiment, Maas et al., 2011). ). A thorough assess- ment of the internet presence of current evaluation sets is needed to check what portion of them can be found explicitly in plain text, or implicitly through the origins of the data.
When the data is used for evaluation, it is pre- sented without the context in which the data origi- nally appeared the internet. Under Scenario 1, however, this context is potentially known to the model: The model can memorize the context in which given data instance appeared in, and recall this context when the instance is presented to it for labeling. In such cases, we treat the evaluation data as compromised.
Centralized strategies. In this work, we primarily focus on decentralized strategies that are effective at the level of individual researchers or individ- ual evaluation sets. Centralized strategies would be helpful under the assumption that many or all relevant actors can work together from the outset. Although this assumption is more strict, the poten- tial benefits merit additional research.
The case of automatic labeling, where the la- bel is directly derived from the context, is a trivial case of data contamination and therefore should be avoided when constructing evaluation benchmarks. However, the case of manual annotation should also be carefully scrutinized: the original context of the test instance contains information that can be helpful to the model in solving the instance, the model can use this context to cheat the evaluation, even if the solution was manually annotated. This is a particularly challenging case of data contami- nation under Scenario 1, as difficult to detect when the connection between the context and the instance label is nuanced.
Partially effective strategies. It may be addition- ally helpful to develop strategies that are only ef- fective under certain conditions, or only partially effective. In Appendix A we discuss one such strat- egy of templated test examples. Other examples in- clude maintaining research database of evaluation data and contamination events. This will make it easier and more convenient for model practitioners to make sure that test data is not included in train- ing, or disclosed to this database if the training of a particular model included a certain test example. Such a database is not a strictly effective mitiga- tion strategy, but it may potentially help if model developers cooperate. Another example is water- marking data so that it can be detected more easily (Sadasivan et al., 2023; Kirchenbauer et al., 2023). While this is an active field for model-generated data, is unexplored for watermarking test data for contamination detection, where the watermarking may be applied to metadata, or other non-intrusive parts of the data context.
Strategy 3 calls for two actions: (A) Releasing context information alongside the evaluation data. This context is not intended to be used in the evalua- tion, but as documentation to enable any evaluation practitioner to execute point B, if they wish scru- tinize the integrity of the data; (B) Detecting and discarding instances where the context, which not intended to serve as input to the model, is in- dicative of the solution in some significant way. Such instances can be used for training, but are not suitable for benchmark. In particular, the practice of collecting automatic labels for evaluation tasks from the internet is fragile under Scenario and should only be executed with great caution.
6 Conclusions
This paper serves foremost as call action encrypt all evaluation data uploaded to the internet or otherwise evade automatic crawling with gat- ing mechanisms; to withhold from sending unseen evaluation data to closed API models without ex- clusion controls; to reassess internet-derived eval- uation data; and to enrich the discussion around possible robust strategies under more strict assump- tions like adversarial actors or negligent actors. We call on the research community to address the issue of data contamination to the hest of our ability. In particular, we should be conscious of who and what can access our test data, and plan accordingly.
5 Discussion and Open Questions
Documenting existing evaluations for contami- nation. Research that collects existing test sets into large-scale benchmarks (c.g., Liang et al., 2022; Srivastava et al., 2022) should be attentive to the issues raised in this paper, and in particular em- ploy Strategy 3. This is crucial for benchmarks that incorporate test sets predating the common prac- tice of crtereet-crawled training corpora, and are

---

Limitations
that specifically forbids using the data for model training or requires encrypted distribution can still permit the distribution of derivatives while serving a similar function to a "no derivatives" license.
On negligent actors. Negligence can unavoidably decrease or invalidate the effectiveness of the strate- gies in all cases. For example, a negligent actor may re-upload a plain-text version of a dataset and make it accessible to crawlers. And in particular, Strategy 2 is vulnerable. Nevertheless, researchers can take additional steps to decrease the likelihood of negligence scenarios, or set guidelines for re- solving such cases after the contamination event occurred. We highlight two negligence scenarios here:
On sincere actors. While the "sincere actors" as- sumption is generally feasible, naturally it does not always hold. We can conceive one plausible case where it does not hold: When using crowd- sourcing for large manual annotation efforts, it is certainly possible for some crowdworkers to be adversarial-and use closed API models, such as the avvilalae ChatGPT, to automate the la- beling against guidelines, if they believe they will not be caught. This will compromise the data, par- ticularly in the case of ChatGPT's web interface, since it does not have training exclusion guarantees as of this writing (Footnote 3). This an important and challenging scenario to be aware of when using crowdsourcing to annotate test data.
"I accidentally sent test data to a closed API model for the first time." The most important action to take in this scenario is to publicize the event, and notify the original contact behind the test data. De- pending on the quantity of compromised examples, it may be possible to continue using the rest of the data, and regardless, the data remains uncompro- mised for other models. Contacting the API holder to notify them about the identity of the data may also be helpful. And finally, consider collecting replacements to the data, if this is possible.
A Acknowledgements
We are grateful to Omar Sanseviero for his help- ful comments. This project received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and in- novation programme,, grant agreement No. 802774 (iEXTRACT).
" "Someone else, unaware of Strategy 2, sent test data to a closed API model." This is the most vulnera- ble weakness of Strategy 2. The API holder is the only actor capable of detecting the event, if they have knowledge of what evaluation data to exclude from future training. Otherwise, this scenario may be reduced by adding sufficient warnings to the data's host web-page, the data file itself. Alter- natively, the access expensive test data can be restricted only to trusted institutions and individu- als, although this kind of mechanism may be too restrictive.
References
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quiney Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh. Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guba, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard. Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Gcoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li. Tengyu Ma. Ali Malik, Christopher D. Manning, Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa. Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Rob
On derivatives. What constitutes derivative work, or an adaptation, in a legal setting depends on local copyright law, and decryption may not necessarily constitute an adaptation. The purpose of the license in Strategy is not to guarantee the ability to legally enforce encrypted distribution but to encourage it among sincere actors.
Test data licenses that specifically address data contamination scenarios can help in making the strategies more reliable and enforceable. For exam- plc, a dataset license that forbids sending its test data to closed API models that don't comply with specific exclusion controls can provide legal means to enforce Strategy 2 downstream; and a license

---

Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan. Christopher Ré. Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishman Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You. Matei Zaharia, Michael Zhang. Tianyi Zhang, Xikun Zhang. Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Perey Liang. 2022. On the opportunities and risks of foundation models.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav 2022. Fusing finetuned models for better pretraining.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Mau Gardner. 2021. Documenting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS).
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Asso- ciation for Computational Linguistics.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rah- wan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representa- tions for detecting sentiment, emotion and sareasm. In Conference on Empirical Methods in Natural Lon- guage Processing (EMNLP).
Séhastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Dem- ing Chen, and Marianne Winslett. 2021. Compress- ing large-scale transformer-based models: A case study on BERT. Transactions of the Association for Computational Linguistics, 9:1061-1080.
C4-Search. 2023. C4 Search by Allen Institute for AI.
Nicholas Carlini, Jamie Hayes, Milad Nast, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023a. Ex- tracting training data from diffusion models. arXiv preprinl arXiv:2301.1311
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. 2023. Llama-adapter v2: Parameter-efficient visual instruc- tion model. arXiv preprint wxXiv2304.150110
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023b. Quantifying memorization across neural lan- guage models. In The International Conference on Learning Representations (ICLR).
Abbas Ghaddar and Phillippe Langlais. 2016. Coref- erence in Wikipedia: Main concept resolution. In Proceedings of the 20th SIGNLL Conference on Com- putational Natural Language Learning, pages 229- 238, Berlin, Germany. Association for Computational Linguistics.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vosss Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, UI- far Erlingsson, al. 2021. Extracting training data from large language models. USENIX Security Symposium, volume 6.
Google. 2023a. Palm 2 technical report.
Google. 2023b. PaLM API & MakerSuite: an approach- able way start prototyping and building generative applications.
Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archaeol- ogy of books known to chatgpt/gpt-4. arXiv preprint arXiv:2305.00/1
Cheng Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner. Ranjay Krishna, Chen-Yu Yu Lec, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.00230
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ton Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing 4 with 90%* chatgpt quality.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. 2020.

---

TinyBERT: Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguisties: EMNLP 2020, pages 4163- 4174, Online. Association for Computational Lin- guistics.
2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Rie Johnson and Tong Zhang, 2014, Effective use of word order for text categorization with convolutional neural networks. CORR, abs/1442.1105.
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Pa- pers), pages 137-165, Dublin, Ireland. Association for Computational Linguistics.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel Weld. 2022. GENIE: Toward re- producible and standardized human evaluation for text generation. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 11444-1 1458, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Mare Marone and Benjamin Van Durme. 2023. Data portraits: Recording foundation model training data.
Margaret Mitchell, Alexandra Sasha Luccioni. Nathan Lambert, Marissa Gerchick, Angelina McMillan- Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela. 2023. Mea- suring data.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. 2023. Foundation mod- els for generalist medical artificial intelligence. Na- ture, 6 6576656))
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina SeMMilaanMMajon Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
MosaicML. 2023. Mosaieml inference. https://www mosaicml com/ www./onfeeeeeeooooeeeeeeeeee /inn/eenffeee..
Gene Myers. 1999. A fast bit-vector algorithm for ap- proximate string matching based on dynamic pro- gramming. Journal of the ACM (JACM), 46(3):395- 415.
OpenAI. 2023. Gpt-4 technical report.
Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maxi- milian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. Thoughtsource: A central hub for large language model reasoning data.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman. Binhang Yuan. Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Man- ning. Christopher Ré, Diana Acosta-Navas Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji. Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang. Vishray Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models.
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexandra Sasha Luccioni, Yacine Jernite, and Anna Rogers. 2023. The roots search tool: Data transparency for lims.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpos natural language process- ing task solver? arXiv preprint 2322:23322064776
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines.
Venkata Rajeev and V Smrithi Rekha. 2015. Recom- mending products to customers using opinion mining of online product reviews and features. In 2015 Inter- national Conference on Circuits, Power and Comput- ing Technologies [ICCPCT-2015), pages 1-5. IEEE
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.

---

Yasaman Razeghi, Robert L Logan IV. Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguisties.
Model soups: averaging weights of multiple fine- tumed models improves accuracy without increasing inference time. In Proceedings of the International Conference on Machine Learning (ICML).
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. and Ludwig Schmidt. 2022b. Robust fine-tuning of zero-shot models. IEEE/CVF Conference on Computer Vision and Pat- tem Recognition (CVPR).
Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behav- ioral testing of NLP models with checklist. CoRR, abs/2005.041
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Bala- subramanian, Wenxiao Wang, and Soheil Feizi, 2023. Can ai-generated text be reliably detected?
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilittes of language models. arXiv preprint arXiv:2206.04615.
Siqi Sun, Zhe Gan, Yuwei Fang. Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020. Contrastive distil- lation on intermediate representations for language model compression. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 498-508 Online. Asso- ciation for Computational Linguistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llaa model. https:// gi thub. www.ooo///aaeeeeeee httm://tts-labbseeee stanfordtddddoo///// aaaca
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Aune Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint 2302.
Algorithms for approximate Esko Ukkonen. 1985.
string matching. Information and control, 64()- 100-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Lc, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201. 1903.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, al. 2023. Zero- shot information extraction chatting with chatgpt. arXiv preprint arXiv:2302. 10205.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Car- mon, Simon Korbblith, and Ludwig Schmidt. 2022a.

---

A Partial Strategy A: Templated test instances
Conditions:
Scenario I or 2.
This strategy requires significant effort, is not a complete defense, and does not apply in many cases, but is nevertheless one of the few practical strategies to guard against Scenario 2.
For some tasks, particularly with textual data, it is possible counterfactually augment test data by converting it into programmatical template." 13 For example, in tasks that require arithmetic rea- soning, the answer can change as conditioned on a number, and this can be derived automatically, In summarization tasks, any information in the in- put (e.g., entity names, dates, or more complex semantic information) should be reflected in the summary.
In test time, for given evaluation attempt, only one counterfactual variant is sampled with a seed value, and the variant is used across all evaluated systems in that attempt. The seed forfeit once the evaluation is completed, and the seed (or test data itself) can be publicized.
The more elaborate the counterfactual variant is, unseen test case, if the source instance is compromised. Although this strategy is expensive to implement, it worth noting that test sets are traditionally small.
the stronger its function
As discussed by Ribciro et al. (2020), although in the con- text unit tests for text models, which are deliberately simple.


Reviews:
[{"review_id": "NbHWL9vOtl", "reviewer": "Reviewer_AUDd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and positive review! We appreciate the reviewer’s affirmation that you would like to see the paper at the conference with no reasons to reject, and recognizing the paper as a call for action.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper brings the attention of the community on an important issue, as we have no details about the training data of popular LLMs and this compromises our capability to evaluate the performance of such models.", "reasons_to_reject": "This paper is mainly a call for action - although the suggested solutions are not groundbreaking, I would like to see the paper at the conference. So no reasons to reject.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the problem of data contamination when it comes to model evaluation.\nSpecifically, the authors consider scenarios where:\n- the model is trained on data crawled from the web, and is therefore possible but unfeasible to check for data contamination (i.e., the presence of the test data in the training data)\n- we have no information about the data used for model training (e.g., OpenAI's models).\n\nThe authors suggest to:\n- not release test data without encryption - public encryption is fine if considering a scenario where all users have good intention, and selecting a license that does not allow data redistribution in a different format.\n- not evaluate models through APIs on test data if there is no explicit training exclusion option.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m22F086T0K", "reviewer": "Reviewer_Hv9v", "summary": "", "questions": "Would you want to add a line or two about:\n\n- How reliance on such Large Language Models, especially closed ones, is driving the research towards relying on the resources provided by a few big players (only who have access/ability to the expensive training) - not to mention the reliability of black box APIs if the LLM’s model and/or data is closed.\n\n-   How do we know the LLM model is really “learning” as opposed to being a huge memory bank which has memorized subtle patterns, statistical or semantic/especially in context of exposed test data.\n\n- There is also a thin line between reproducibility and data contamination. For example most of the major conferences in NLP demand reproducibility guarantees, which kind of comes with the need for making the test data public. While your encryption key idea might work, how can we ensure that another independent actor who for purely evaluation purposes knowingly or unknowingly uploaded the test data to the internet, and even worse: has no memory of doing so - maybe can add this to the negligent actor paragraph . Maybe one solution there is also uploading a unique SHA key along with the test data. If that key is found say couple of months later in an internet search algorithm, we now know the data along with the key also made its way to the internet. (e.g., OPENAI key found on github is already being flagged). However that will lead to digital verifiability issues- and hence has traces of deep cybersecurity research topics. Maybe that is a collaboration opportunity with the cyber security experts for the next version of this paper.\n\n- Encryption and decryption nevertheless brings in a layer of overhead. Why not a solution based on obfuscation, similar to the ones used by Author Anonymization SOTA. Just making data un-crawlable to machines, while useable with minimal modification to humans \n\n- Or even an extremely unpopular opinion: should results evaluated based on Blackbox models/API/Datasets be considered in a separate track/workshop in NLP conferences. I agree, sad, but it is possibly high time that NLP research community needs to start asking these questions.\n\n", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very uplifting review! \n\nThe review mentions various interesting points in the “questions” section. We will certainly add such discussions to the paper in the extra allotted space for camera-ready (within some limits). We also discuss the points below (though we focus on the points with direct implications to the camera-ready version):\n\nThe first point regarding community trends and implications on reliability were previously partially discussed in a blog-post [1] which we cite in the paper. \n\nThe point regarding reproducibility guarantees is interesting and indeed clashes with negligent/adversarial actor requirements, although perhaps circumventable with limiting access to verified institutions, if such strict regulations are deemed necessary (and this is already the case for some datasets, with its own set of disadvantages). Thank you for the suggestion to the negligent actor discussion, which we will add. \n\nOn the point of encryption overhead: Perhaps arguably, we regard password-protected zip files as a convenient and reliable obfuscation format, which is why we put it forward here. Obfuscation can be regarded as encryption without password protection. Unfortunately, some crawlers do implement standardized deobfuscation methods, and implementing a custom obfuscation algorithm is itself inconvenient. Nevertheless, it is true that any method of evading crawlers is valid here - such as captcha gating, HuggingFace gating [2] (though it was not created for this purpose), and so on, and it was not our intent to neglect these solutions. The revised paper will include this information.\n\nThanks!\n\n[1] https://hackingsemantics.xyz/2023/closed-baselines/ \n\n[2] https://huggingface.co/docs/hub/datasets-gated", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Since large language models, many behind closed API and training datasets are the state of the art now, this paper raises a very time relevant and very important topic to discuss in a big conference like EMNLP\n\n- This paper not only raises these, issues, does a review of existing solutions, analyzes them and shows that they are not sufficient to address the issue of data contamination, but also proposes a few novel solutions.\n\n- This paper is very easy to read and is very well written\n", "reasons_to_reject": "None", "questions_to_authors": "Would you want to add a line or two about:\n\n- How reliance on such Large Language Models, especially closed ones, is driving the research towards relying on the resources provided by a few big players (only who have access/ability to the expensive training) - not to mention the reliability of black box APIs if the LLM’s model and/or data is closed.\n\n-   How do we know the LLM model is really “learning” as opposed to being a huge memory bank which has memorized subtle patterns, statistical or semantic/especially in context of exposed test data.\n\n- There is also a thin line between reproducibility and data contamination. For example most of the major conferences in NLP demand reproducibility guarantees, which kind of comes with the need for making the test data public. While your encryption key idea might work, how can we ensure that another independent actor who for purely evaluation purposes knowingly or unknowingly uploaded the test data to the internet, and even worse: has no memory of doing so - maybe can add this to the negligent actor paragraph . Maybe one solution there is also uploading a unique SHA key along with the test data. If that key is found say couple of months later in an internet search algorithm, we now know the data along with the key also made its way to the internet. (e.g., OPENAI key found on github is already being flagged). However that will lead to digital verifiability issues- and hence has traces of deep cybersecurity research topics. Maybe that is a collaboration opportunity with the cyber security experts for the next version of this paper.\n\n- Encryption and decryption nevertheless brings in a layer of overhead. Why not a solution based on obfuscation, similar to the ones used by Author Anonymization SOTA. Just making data un-crawlable to machines, while useable with minimal modification to humans \n\n- Or even an extremely unpopular opinion: should results evaluated based on Blackbox models/API/Datasets be considered in a separate track/workshop in NLP conferences. I agree, sad, but it is possibly high time that NLP research community needs to start asking these questions.\n\n", "paper_topic_and_main_contributions": "Addresses a very relevant and pertinent issue in the contemporary world, i.e how Large Language Models trained on internet crawled data could be knowingly or unknowingly be learning from/familiar with data that is used for downstream testing/evaluation.", "excitement": 5, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "2czidS8P1y", "reviewer": "Reviewer_Vde9", "summary": "", "questions": "Q1 - How to define the questions and answers within the text, in other words, how to define the test set and training set? In an increasingly free and open language model utilization scenario, text-based answers and questions may become more flexible, and users may inquire from different perspectives. Therefore, each element has the potential to appear in either the question or the answer.\n\nQ2 - The encryption method for text may bring certain inconveniences to the use of normal text. Is it possible to find a better balance between text privacy and convenience of use? Considering question (1), if there is no way to define the boundaries between questions and answers, should all text be encrypted to prevent leakage? Would this significantly disrupt normal communication among people?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and very detailed review, and for the constructive suggestions! We address the comments below, and will also update the paper with all of the information here for the camera-ready version. \n\nThe review mentions several points regarding theoretical formalization, well-defined properties of train and test data, task abstraction, and so on. On this issue, we adhere to simple formalizations inherited from standard machine learning literature (including generative literature), rather than attempt to reinvent the wheel ourselves. We will add and cite such propositions in the paper in the camera-ready version. Notably, our goal here is to discuss real-world grounded and practical manifestations of the theory, rather than the theory itself, which is evidently too general to be plausibly solved in a decentralized and democratized community.\n\nOn theoretical or practical validation for our viewpoints (point 2), we would sincerely appreciate it if you could specify which viewpoints fail under scrutiny - to our knowledge, the paper discusses the relevant strengths and assumptions of the strategies, so we could not identify the criticism here.  \n\nRegarding the questions:\n\nQ1. Regarding “input” and “output” categorization (test inputs and test outputs), although interesting, it is less relevant for our setting. We advocate for encryption of the entire test set in order to protect it from training, both input and output, since training on test inputs has been documented to also influence performance, and we are concerned with improving the reliability of all evaluations.\n\nQ2. Would encryption of the entire test set disrupt the convenience of current dataset sharing? We argue that this is not the case. In the case of zip-files, though we regard them as convenient, it is also possible to share a small set of compromised unencrypted examples to make showcasing the data more convenient. Any other method of evading web crawlers is also applicable, such as gating mechanisms, e.g., HuggingFace’s “gated repositories” [1], or captcha-like gating mechanisms, and it was not our intent to neglect these solutions. \n\nWe will add these discussions to the paper as well, with all relevant details.\n\nThanks!\n\n[1] https://huggingface.co/docs/hub/datasets-gated", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem addressed in this work is becoming increasingly important with the development of large-scale model technologies.", "reasons_to_reject": "1 - This work lacks a reasonable task abstraction or problem definition, as well as the absence of mathematical expressions and model training.\n\n2 - There is no theoretical or practical validation to support the authors' viewpoints.\n\n", "questions_to_authors": "Q1 - How to define the questions and answers within the text, in other words, how to define the test set and training set? In an increasingly free and open language model utilization scenario, text-based answers and questions may become more flexible, and users may inquire from different perspectives. Therefore, each element has the potential to appear in either the question or the answer.\n\nQ2 - The encryption method for text may bring certain inconveniences to the use of normal text. Is it possible to find a better balance between text privacy and convenience of use? Considering question (1), if there is no way to define the boundaries between questions and answers, should all text be encrypted to prevent leakage? Would this significantly disrupt normal communication among people?", "paper_topic_and_main_contributions": "This work focuses on addressing the threat of data leakage and contamination in closed-source task data caused by the widespread use of text obtained through general web crawling for model training. The authors envisioned specific scenarios and proposed three different countermeasures.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a consensus among the reviewers that this position paper outlines an important problem (models training on publicly available evaluation sets) and proposes a viable mitigation strategy. It would be a clear benefit to the EMNLP conference program and is a clear accept.
Decision: Accept-Main