SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
Authors: Sewon Min
Suchin Gururangan
Eric Wallace
Weijia Shi
Hannaneh Hajishirzi
Noah A. Smith
Luke Zettlemoyer
Venue: ICLR 2024 spotlight (2024)
Keywords: language modeling; retrieval; legality of language modeling
Primary area: generative models

Abstract: The legality of training language models (LMs) on copyrighted or otherwise restricted data is under intense debate. However, as we show, model performance significantly degrades if trained only on low-risk text (e.g., out-of-copyright books or government documents), due to its limited size and domain coverage. We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on the Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text and (2) augmenting it with a more general and easily modifiable nonparametric datastore (e.g., containing copyrighted books or news) that is only queried during inference. The datastore allows use of high-risk data without training on it, supports sentence-level data attribution, and enables data producers to opt out from the model by removing content from the store. These capabilities can foster compliance with data-use regulations such as the fair use doctrine in the United States and the GDPR in the European Union. Our experiments show that the parametric LM struggles on its own with domains not covered by OLC. However, access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile, a more diverse corpus with mostly high-risk text. We also analyze which nonparametric approach works best, where the remaining errors lie, and how performance scales with datastore size. Our results suggest that it is possible to build high quality language models while mitigating legal risk.

Published as a conference paper at ICLR 2024
SILO LANGUAGE MODELS: ISOLATING LEGAL RISK IN A NONPARAMETRIC DATASTORE
Sewon Min" Suchin Gururangan" Eric Wallace² Weijia Shi!
Hannaneh Hajishirzi
Noah A. Smith 1.3 Luke Zettlemoyer'
Unniversity of Washington hww.anl.aaaaaahhaaaaa @cs. washington.ede
PCC Berkeley AAllen Institute for AI
ericwallace@berrkeee.duu
ABSTRACT
The legality of training language models (LMs) copyrighted otherwise restricted data is under intense debate. However, as we show, model performance significantly degrades if trained only on low-risk text (e.g., out-of-copyright books or government documents), due to its limited size and domain coverage. We present SILO, a new language model that manages this risk-performance tradeoff' during inference. SILO is built by (1) training parametric LM on the OPEN LICENSE CORPUS (OLC), new corpus we curate with 228B tokens of public domain and permissively licensed text and (2) augmenting it with more general and easily modifiable nonparametric datastore (e.g., containing copyrighted books or news) that is only queried during inference. The datastore allows use of high-risk data without training on it, supports sentence-level data attribution, and enables data producers to opt out from the model by removing content from the store. These capabilities can foster compliance with data-use regulations such as the fair doctrine in the United States and the GDPR in the European Union. Our experiments show that the parametric LM struggles on its own with domains not covered by OLC. However, access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile, a more diverse corpus with mostly high-risk text. We also analyze which nonparametric approach works best, where the remaining errors lie, and how performance scales with datastore size. Our results suggest that it possible build high quality language models while mitigating legal risk.
INTRODUCTION
Large language models (LMs) are under widespread legal scrutiny, in large part because they are trained on copyrighted content, which may infringe on the rights of data producers (Metz, 2022; Vincent, 2023; J.L. al. v. Alphabet Inc., 2023; Brittain, 2023). At the heart of this discussion the inherent tradeoff between legal risk and model performance. Training only on data sources such as public domain, con-copyrig,table or otherwise permissively licensed data significantly degrades performance (as show in $4.1). This limitation arises from the scarcity of permissive data and its narrow specificity to sources which are largely different from common LM corpora that cover more diverse domains (Raffel et al., 2020; Gao al., 2020; Together, 2023).
In this paper, we demonstrate that is possible to improve the risk- lisk-peforrmance tradeoff by segre- gating training data into two distinct parts of the model: parametric and nonparametric (Figure 1). We learn LM parameters on lower-risk data (i.e., data under the most permissive licenses), and then use higher-risk data (i.e., data under copyright, restrictive licenses, or unknown licenses) in an inference- ceanece-tmm-one time-only nonparametric component (called a datastore). With nonparametric datastores, we can retrieve higher-risk data improve model predictions without training on it. The datastore can be easily updated at any time, and allows creators to remove their data from the model entirely, at the level of individual examples. This approach also attributes model predictions the sentence-level. enabling credit assignment to data owners. These capabilities enable better alignment of the model with various data-use regulations, e,g., e, the fair use doctrine in the United States (Henderson
"Equal Contribution.

---

Published as a conference paper at ICLR 2024
Training
Test-time Datastore
(fixed once training is done)
(can be updated/removed anytime)
Catl me ichmoo! Some years nooer mind how long.
polient liees with aie twv shaughters
Openal mater OraiGPT with propased
T
Public Domain
cc
0
Common Crawl
Durney number four Privet Drivk were
APACHE SAFT Ficennation
O
Ply x)
Steve on American
def tuningg model rum, triatst ossert-
# Capyright dii Meia Ptatforns. Inc and anfictes. from demnn import LLa/6
busvess magnatte and
Low-risk data (public domain, permisivelyylicensee))
High-risk data (copyrighted, private, attribution required)
An overview of SILO. We train parametric language model on low-risk datasets that contain public domain text (e.g., copyright-expired books) and permissively licensed code. At inference time, we use nonparametric datastore that can include high-risk data, including medical text with personally- -idetifiblie information, copyrighted news, copyrighted books, data requiring attribution, and code under non-pernissive licenses (counterclockwise from the top of figure). The datastore can be modified at any time, e.g., to respond to opt-out requests.
Figure I:
al., 2023) and the GDPR in the European Union (Zhang et al., 2023), as detailed in $A. This is in contrast to parametric models, where removing high-risk data is infeasible after training (Bourtoule et al., 2020; Carlini et al., 2021) and data attribution difficult (Zhang et al., 2021; Han et al., 2023).
We introduce STL.O, a new language model that follows our proposal ($3). The parametric component in SIL. trained on a new pretraining corpus, the OPEN LICENSE CORPUS (OL.C $3.1), which curate to include data under three types of permissive licenses, from public domain to Creative Commons. OLC is diverse but has a domain distribution that is very different from typical pretraining corpora: it is dominated by code and government text. This leads to new challenge of generalization beyond highly limited training domains, which we call extreme domain generalization. We train three 1.3B- 3a--arrmettrr LMs varying subsets of OLC, and then construct test-time datastore that can include high-risk data, employing retrieval method to make use of the datastore's 's contents during inference. We compare two widely studied retrieval methods: a nearest-neighhors approach (kNN-LM) that uses a nonparametric next-token prediction function (Khandelwal et al., 2020) and a vetrieval-in-contee approach (RIC-LM) that retrieves text blocks and feeds them the parametric LM in context (Shi et al., 2023; Ram et al., 2023).
We evaluate SILO in language modeling perplexity on 14 different domains, covering both in-domain and oft-oin data with respect to OLC ($4). These domains highlight specific legal risks, e.g., copyrighted materials such as books, news and user reviews, or private data such as emails and clinical notes. We also evaluate models in zero-shot downstream performance classification datasets. We compare SILO to Pythia (Biderman et al., 2023), a parametric LM with a similar parameter count trained on less restricted data with much wider domain coverage (Gao et al., 2020). We first show that parametric-onle SILO is competitive on domains covered by OLC but falls short out-of-domain confirming the challenge of extreme domain generalization. However, adding an inference-time datastore to SILO effectively addresses this challenge. Comparing the two methods of retrieving over this datastore, find that while both &NN-LM and RIC-LM significantly improve out-of-domain performance, the former generalizes better than the latter, allowing SILO to reduce the gap with the Pythia baseline by 90% on average across all domains. Further analysis attributes these improvements to two factors: (1) ANN-LM strongly benefits from scaling the datastore and (2) the nonparametric next-token prediction &NN-LM is robust domain shift. Altogether, our study suggests important future directions such as closing the remaining gaps by scaling the datastore size and improving the runtime speed of SIL.O.
ten text
'The Pile contains a large amount of copyrighted or restrictively licensed data, e.g., most content in its Books3, ArXiy, Github, OpenWeTText, YoutuheSubtitles, and Common Crawl subsets.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND & RELATED WORK
State-of-the-art LMs are trained on text corpora with trillions of tokens (Brown et al., 2020; Raffel al., 2020; Gao et al., 2020; Together, 2023). These training sets are built by combining (1) manually selected sources such as Wikipedia, book collections, and GitHub and (2) web pages collected through web-crawling services such as Common Crawl. Such data includes sources such as GitHub repositories and book collections that contain text with highly restrictive licenses (Bandy & Vincent, 2021). This approach has led to intense ongoing debate on the legality of language models; we provide legal background in $A.
The goal of our work is not to weigh in on legal discussions; instead, we study the feasibility of developing technologies that explicitly manage legal risk. SILO places all copyrighted data in nonparametric datastore and supports instance-level attribution and data opt-out, which can increase the likelihood of successful fair use defense (Henderson et al., 2022) and adhering to the General Data Protection Regulation (GDPR). Attribution and opt-out are fundamental features ($3.2), in contrast to other techniques like post-hoc training data attribution (Koh & Liang, 2017; Han et al., 2023) and the removal of the effect of particular training examples from parameters (Cao & Yang, 2015; Jang al., 2023b) which lack inherent guarantees and are hard to scale.
Prior work in copyright risk mitigation. The most straightforward approach to avoid copyright infringement is to filter training data to only include permissive licenses. This has been done in prior work, primarily for code-based datasets (e.g.. Kocetkov et al., 2023; Fried et al., 2023; Together, 2023) and scientific text (c.g., Soldaini & Lo. 2023). Extending a similar approach to wider range of domains remains unclear, because permissive data is extremely scarce in most domains, e.g., books and news. For the same reason, Henderson et al. (2023) has suggested that restricting the training data to public domain or otherwise permissively licensed data may be impractical. In this work, we show that there is in fact large number of tokens from data sources with permissive licenses, but the key challenge instead arises from the highly skewed domain distribution.
SILO
We introduce SILO, an LM that takes a prefix of text v and outputs next-word probability distribution over the vocabulary Pyy z). SILO combines a parametric LM trained on permissive data ($3.1) with a nonparametric datastore based on less restricted data ($3.2). The key idea is to use low-risk high-risk data only in nonparametric datastore. This
data to estimate model parameters, and is based on the motivation that model parameters should be learned conservatively, since training data is difficult to remove or trace after model training is completed. In contrast, a nonparametric datastore offers greater flexibility, as it can be easily updated, grown, or filtered, supports data opt-out the level of individual examples, and provides attributions for free every model prediction.
3.1 PARAMETRIC COMPONENT & OPEN LICENSE CORPUS (OLC)
For the parametric component of SILO, use 1.3B decoder-only transformer LM (Vaswani et al., 2017) based on the LLaMA architecture (Touvron et al., 2023) as implemented in OpenLM. This model uses fixed set of parameters at both training and inference time.
The parametric component of SILO should be learned on the lower-risk data, ie.. data in the public domain or under permissive licenses. To this end, introduce the OPEN LICENSE CORPUS (OL.C, a large collection of permissive textual datasets across multiple domains, comprising 228B tokens. essential to note that the permissivity of license type is not a binary decision but rather a spectrum. Therefore, we establish three levels of permissive licenses: Public domain (PD), Permissively licensed software (SW) and Attribution licenses (BY). This allows model developers to select the boundary that aligns with their preferences. See $B.1 1 for the description of each license type. We train the models on varying subsets of licenses from PD and PDSW to PDBYSW- accommodate different risk tolerances. Later sections demonstrate our empirical findings are consistent across all possible boundaries (54, $D.2, Figure 3).
The data generally falls into eight different domains:
htttps:/ http:///ggtttuubbooo////nnnnnsns
Im
3

---

Published as a conference paper at ICLR 2024
Table OLC is large but its distribution different from that of typical pretraining corpora like the Pile. Data distribution of OLC (PD, PDSW, PDSWBY) in comparison to the Pile (Gao et al., 2020), common LM training dataset that not specifically designed for legal permissibility. We report the number of tokens in billions, and the relative frequency. t: There is no explicit news domain the Pile, but news sites are found to be some the most representative data sources in Common Crawl (Dodge et al., 2021).
PD
PDSW
PDSW BY
The Pile
Domain
Tokens
%
Tokens
%
Tokens (B)
%
Tokens (B)
%
Code
0.0
0.0
58.9
59.1
58.9
25.8
32.6
9.8
Legal
27.1
86.2
27.1
27.2
27.2
11.9
30.8
9.3
Conversation
0.0
0.0
5.9
5.9
27.2
11.9
33.1
10.0
Math
0.0
0.0
3.5
3.5
3.5
1.50
7.1
2.1
Books
2.9
2.9
2.9
1.3
47.1
14.2
Science
1.2
1.2
71.5
31.3
86.0
26.0
News
0.7
0.2
0.2
0.2
_f
Wikipedia
0.0
0.0
37.0
16.2
12.1
3.7
Unverified web
83.1
25.0
Total
31.4
100.0
99.6
100.0
228.3
100.0
331.9
100.0
PD BY Legal: We curate legal text from the Pile Law (Henderson al., 2022), amalgation of 31 different sources of text related to civil court cases, patents, and other legal and governmental works, either licensed public domain CC-BY. We also gather public domain text from the Case Law Access Project (Caselaw Access Project), which covers over 6.5 million decisions published by state and federal courts throughout U.S. history.
SW Code: We use the Github subset of the RedPajama dataset (Together, 2023), which contains code from Github repositories with three permissive software licenses: MIT, Apache, and BSD.
SW BY Conversation:
We source conversational text under permissive software licenses from the HackerNews (MIT license) and the Ubuntu IRC (Apache license) subsets of the Pile (Gao al., 2020). We also use the Stackexchange subset of the RedPajama dataset (Together, 2023) and Stackoverflow corpus from Kaggle, 3 both under the CC-BY-SA license.
SW Math: We source mathematical
from the Deepmind Mathematics (Saxton et al., 2019) and the AMPS (Hendrycks et al., 2021) datasets, both of which are under the Apache license.
PD BY Science: We source scientific text from ArXiv abstracts that are in the public domain (ArXiv, 2023). We also collect full-text articles from the Semantic Scholar Research Corpus (Lo al., 2020, S2ORC), either licensed as public domain or CC-BY.
PD Books: We source books from the Gutenberg corpus (Project Gutenberg), which are copyright-
expired books that the public domain.
text from the English subset of the MOT cor- pus (Palen-Michel et al., 2022). We also collect text from Wikinews, which is under CC BY-SA.
PD BY News: We collect public domain
BY Encyclopedic: Finally, we include a large set of Wikipedia from the subset included in RedPajama (Together, 2023). We follow RedPajama in using Wikipedia snapshots from 20 languages even though the model primarily focuses on English.
Following Kandpal al. (2022) and Lee al. (2022), we deduplicate text using a document-lever filter that considers r--gram overlap (Groeneveld, 2023). We first deduplicate within each domain to remove redundant documents from similar sources (e.g. Case Law and the Pile Law), and then perform deduplication against the validation and test datasets of the Pile to avoid test leakage.
Data statistics. In Table 1, compare the distribution of domains in OLC to that of the Pile (Gao et al., 2020), a popular pretraining corpus that includes data under copyright restrictions (e.g., Books, web crawl).* These statistics convey number of research challenges when working with OLC. First, despite our best efforts to collect as much public domain or permissively licensed data as possible, the size of OL.C is still 31 smaller than the Pile. In addition, while the majority of the Pile sourced
"https:/ http://wwwwkkat wtt.gl...c..oooom htttt/stttocoooeoo///ssteo stackoverlloww
"This comparison also dovetails with our comparison to Pythia, model trained the Pile ($4).
4

---

Published as a conference paper at ICLR 2024
from scientific text, web crawl, and books, OLC is dominated by code, scientific text, and legal text. This highlights that models designed for use outside these specific domains may struggle and require special techniques for extreme domain generalization. More analysis can be found in $B.
3.2 ADDING AMETRRAMETRIC COMPONENT
To incorporate the nonparametric component, experiment with two widely-used retrieval methods: the k-nearest neighbors LM (KNN-LM; Khandelwal et al., 2020) and the cttreval-in-cootee approach (RIC-LM; Shi al., 2023; Ram et al., 2023). Each approach constructs a datastore from the raw text data offline, and then uses it on-the-fly at inference time.
The k-nearest neighbors language model (kNN-LM) interpolates the next-token probability distri- bution from parametric LM with nonparametric distribution based on every token that is stored datastore. Given a text dataset consisting of N tokens CI...C,, a datastore is built by creating a key-value pair for every token c; (1 < i S N); a value is q, and key k; is 1. prefix preceding Gy. At test time, given an input prefix x, the nonparametric distribution is computed by:
PENN (y [x) x
Σ (K.,))D
[[e= ylexp (-d(Enc(k)), Enc(z)))
Here, Enc is an encoder that maps text into A and d R² x R² + R is distance function, where is the hidden dimension. We follow Khandelwal et al. (2020) and use the output vector from the last layer of the transformers in the parametric LM as Enc, and L2 distance as d. The final model takes the kNN-LM output and interpolates it with the output from the parametric LM: APLM(U x) + (1 PINNN x),, where À is a fixed hyperparameter between 0 and 1.
As an alternative to &NN-LM, the retrieval-in-context language model (RIC-LM) retrieves text blocks from a datastore and feeds them to the parametric LM in context. Given dataset consisting of N tokens CN...N.. an index D is constructed by splitting the data into text blocks bj...by. optionally with a sliding window. At test time, given input prefix x. RIC-LM retrieves the most similar paragraph to the prefix p = arg maxtep sim x) and concatenates it before the prefix to produce PLM(U r). Here, sim is function that computes a similarity score between two pieces of text; we use BM25 following Ram et al. (2023) who show that BM25 outperforms alternative dense retrieval.
Comparison between &NN-LM and RIC-LM. ANN-LM and RIC-LM differ in how the nonpara- metric component influences the output. In &NN-LM, it directly impacts the output distribution, while in RIC-LM, it indirectly influences the output by affecting the input to the parametric model. This indicates &NN-LM would intuitively benefit more from a datastore; however, RIC-LM interacts more easily with a parametric model (i.e., it is applicable to black-box LM since probabilities do not need be accessed) and offers better speed and memory efficiency (explored in Appendix D.2). Empirical comparisons between KNN-LM and RIC-LM have been largely unexplored. Our experiments ($4.2) present a series of such comparisons, with varying sizes of the datastore, and with and without distribution shift.
Datastore. We assume in-distribution data for each test domain is available at inference time, and construct an in-domain datastore for each test domain based on its training data; future work may investigate building a single datastore that includes all domains. For datasets from the PILE, consider 10% of the training data. For kNN-LM, each datastore is capped to billion tokens due to the resource constraints. More implementation details, statistics and hyperparameter values for the datastores are reported in SC.
Since elements in the datastore that contribute to the model prediction are transparent, both İNN-LM and RIC-LM offer inherent attributions. Moreover, data removed from the datastore is guaranteed not contribute to any model predictions, allowing data owners to remove their data at the level individual examples. Both are unique characteristics of nonparametric language models. While prior work studies post-hoc attribution to the data used for training model parameters (Koh & Liang, 2017; Han et al., 2023) and removing the effect of specific training examples from parameteric models (Cao & Yang, 2015; Jang et al., 2023b), they lack guarantees and are difficult to scale.
Attribution and opt-out.
Implementation details, statistics and hyperparameter values for the datastores are reported in &C.
5

---

Published as a conference paper at ICLR 2024
Table 2: Perplexity (the lower the better) of the parametric-only SILO trained on PD, PDSW, and BY (without datastore), compared to Arthia-1.4B,, a model trained with similar amounts of compute but on mostly non permisssse data. We use m. and to indicate text that is in-domain, out-of-domain, out-of-domain but has relevant data in-domain (e.g., high-risk Github code our permissive Github code). Reported on the test data; see Table 11 ($D) for results on the validation data. Our parametric LMs are competitive to Pythia in-domain but fall short oot-of-domain.
Eval data
PD
PDSW
PDSW BYB
Pythia
FreeLaa
5.3
5.7
6.5
5.6
Gutenberg
15.2
12.5
14.1
13.1
HackerNews
38.0
13.7
13.3
Github
13.5
2.7
2.8
2.4
NIH ExPorter
28.2
19.2
15.0
11.1
PhilPapers
31.7
17.6
15.0
12.7
Wikipedia
28.9
20.3
11.3
9.1
CC News
34.0
23.3
12.0
BookCorpus2
25.3
19.2
19.6
13.2
Books3
27.2
19.3
18.6
12.6
OpenWebTexx2
37.8
21.1
18.8
11.5
Enron Emails
18.6
13.2
6.9
Amazon
81.1
34,8
37.0
22,9
MIMIC-III
22.3
19.0
15.5
13.1
Average
29.1
17.3
16.0
11.4
EXPERIMENTAL RESULTS
We benchmark our models using language modeling perplexity on 14 domains as shown in Table 2 (details provided SC.3). Experiments consider both in-domain and out-of-domain data and different levels of OLC at different levels of legal risk, e.g., CC-News, BookCorpus2. Books3 and Amazon reviews are mostly copyrighted, Github is mostly not permissively licensed, 5 and Enron Emails and MIMIC-III include private text. We also evaluate on zero-shot downstream performance ten text classification datasets in $4.2 (details provided $C.4).
We first evaluate the parametric-only component of SILO trained the OLC (84.1), and then show the effect of adding datastore that may contain high-risk text ($4.2). For all experiments, we use the 1.4B Pythia model (Biderman et al., 2023) as baseline because it is trained with similar amount of compute (data size and model parameters), but is trained on mostly high-risk data.
4.1 RESULTS: PARAMETRIC COMPONENT
Main results. Table 2 reports performance of our 1.3B base LMs trained on varying levels of permissively-licesed data-Pi PDSW, and PDSWBY- well as Pythia. Overall, our LMs competitive with Pythia despite using permissive data only. They are roughly equal quality on in-domain data, e.g., FreeLaw and Gutenberg, HackerNews in the case of PDSW and PDSWBY, and Wikipedia in the case of PDSWBY The largest gaps occur on data that is in-domain for Pythia but out-of-domait for our model, e-g., news, books, Open WebText, and emails, and Wikipedia in the case of models besides PDSWBY. This illustrates the extreme domain generalization challenge that is present when training on only permissive data, as we hint I.
Gaps from Pythia align with degree of domain shift. The similarity of evaluation domain domain of the OLC strongly correlates with the performance gaps between SILO and Pythia. To show this. we compute the Pearson correlation between 1) the maximum n-gram overlap between an OLC domain and the Pile validation domains (from $B.2 2 and 2) the perplexity difference between the Pythia model and our PDSW model, normalized by the performance of the PDSW model. We find strong negative correlation between these metrics (r=-0.72, < 0.005), indeed indicating that the
"Koetkov et al. (2023) estimates about 13% of the Github data is under MIT, Apache, and BSD. We use the model checkpoint from https:/ httsss//huggnggace.coom
httpaggnnnnaaccooe//lueeee////// /ll//eeeeeerrlpytteeeeee pytthaa Serythaa-1.4----e 1.4b-deduped-v0.

---

Published as a conference paper at ICLR 2024
Table 3: Perplexity (the lower the better) of parametric LMs (Prm-only), KNN-LM, and RIC-LM. % in parentheses indicate a reduction in the gap between the parametric-only SILO and Pythia. As in Table 2, indicates in-domain; indicates out-of-domain; indicates out-of-domain but has relevant data in-domain, all with respect to the training data of the parametric LM. Reported on the test data; see Table 12 for results on the validation data. See Table 9 for the statistics of the datastore. A Adding a datastore, with &NN-LM, effectively reduces the gap between SILO and Pythia.
SILO (DSW
Pythia
Eval data
Prm-only
ANN-LM
RIC-LM
Prm-only
Github
2.7
2.4 (-1009)
2.4 (-100%)
2.4
NIII ExPorter
19.2
15.0 (-52%)
18.5 (998
11.1
Wikipedia
20.3
14.5 (-52%)
19.4 (-8%)
9.1
CC News
23.3
8.0 (-13598)
16.8 (-58%)
12.0
Books3
19.3
17.4 (-28%)
18.6 (-10%)
12.6
Enron Emails
13.2
5,9 (-1169))
9.9 (-68%)
6.9
Amazon
34.9
26.0 (-75%)
33.7 (-10%)
23.0
MIMIC-III
19.0
6.6 (-110%)
15.6 (-88%)
13.1
Average
19.0
12.0 (-18)
16.9 (-27%)
11.3
more dissimilar an evaluation domain is from the OLC domains, the better Pythia does relative to SIL.;; see $D (Figure 6) for scatter plot.
More ablations, including the effect of upsampling low-resource data and the effect of explicit source code, are provided in $D.1.
4.2 RESULTS: ADDING THE NONPARAMETRIC COMPONENT
Since building legally permissive LMs poses a challenge of extreme domain generalization, our next question whether using an in-domain, nonparametrie datastore can reduce the gap. This section reports results with the LM trained the PDSW subset of OLC; see $D.2 for results of models trained on PD or PDSWBY. All models are evaluated on subset of 8 out-of-domain datasets: Github, NIH ExPorter, Wikipedia, CC News, Books3, Enron Emails, Amazon, and MIMIC-III.
Main results. Table 3 shows adding the datastore with either ANN-LM- or RIC-LM-based retrieval improves performance over just using the parameteric component on all domains, but KNN-LM is more effective than RIC-LM. In most domains, ANN-LM reduces the gap between SILO and Pythia by more than 50% (on NIH ExPorter, Wikipedia, Amazon) or even outperforms Pythia (on Github, CC News, Enron Emails, MIMIC-III) Books3 the domain with the least benefit, although KNN-LM still reduces the gap by 28%.
Impact of scaling the datastore. Figure 2 demonstrates that both ANN-LM and RIC-LM- based retrieval consistently improve performance as the datastore size increases, with a strong log-linear trend. However, ANN-LM improves performance more rapidly than RIC-LM does, consistently over all datasets. Extrapolating the trend suggests that, on the domains that SILO has not outperformed Pythia yet, scaling the datastore even further (with ANN-LM retrieval) may enable it to match Pythia.
Why does kNN-LM outperform RIC-LM? Our next question is why ANN-LM is better than RIC-LM- is it (a) because &NN-LM better than RIC-LM in general, or (b) because ANN-LM generalizes out-of-domain better than RIC-LM does? Our further analysis in $D.2 (Figure 5) reveals that it is both. &NN-LM overall outperforms and scales better than RIC-LM even when test domains are in-domain, supporting (a). The gains are larger when test domains are out-of-domain than when test domains are in-domain, supporting (b). We also find that the robustness of ANN-LM can be explained by the encoder for a nonparametric distrbution PANN(y x)) being more robust distribution shift than the model that outputs PLM(Y Iz;; see $D.2 (Figure 4) for the details.
Altogether, our analysis highlights two promising directions further reduce the gup: (1) scaling the datastore beyond billion tokens, e.g., at the scale of trillions of tokens as in Borgeaud et al. (2022), and (2) improving the robustness of the model by improving nonparametrie techniques (Zhong et al., 2022: Lan et al., 2023; Min et al., 2023; Huang et al., 2023b; Izacard et al., 2022).
7

---

Published as a conference paper at ICLR 2024
Widdpedaa
CC News
MIMIC
20.0
Parametric
Parametric LM
7.5
Parametric LM
RIC
Perplesry
17.5
15
RIC
Pytia
RIC
Pythia
0.0
10
7.5
RMN
10
10
10
16"
10'
10'
10"
10"
16'
16"
Amezzon Reviows
Books3
Github
Paramatric
Paramatiic:
Palameticc LM
26
Perglerry
16
RID
27.5
ANN
Pythia
25.0
14
23
Pytia
Pyttia
22.5
12
10'
10'
16¹
16"
10'
10"
# Tokens Datastore (millions)
Takons Datastore (millions)
Tokens in Datastore (millions)
Figure 2: Impact of scaling the datastore of SILO (PDSW), Perplexity on random 128K tokens from the validation data reported. The rightmost dots for kNN-LM and RIC-LM in each figure correspond the final models used Table 3. Scaling the test-time datastore consistently improves performance over all domains.
Downstream task performance. In order to verify if our findings on language modeling perplexity transfer to downstream tasks, we evaluate zero-shot performance of SILO and Pythia on text classification datasets whose domains are not covered by OLC. All models use PMI (Holtzman et al., 2021) for a better calibration of model outputs, and use &NN-Prome (Shi et al., 2022) for applying ANN-LM for downstream tasks. See $C.4 for the details. Table 4 demonstrates that earlier findings hold on all ten datasets: the parametric-only SIL. largely underperforms Pythia; however, adding datastore greatly improves performance, allowing performance of SILO to match that of Pythia.
Table 4: Zero-shot performance on ten classification datasets. ANN-LM allows performance of SILO to match that of Pythia by using a datastore. indicates in-domain; indicates out-of- domain.
SILO (PDSW)
Pythia
Eval
Prm-only
kNN-LM
Prm-only
AGN
63.3
79.5
71.2
Dbpedia
36.9
41.3
39.1
SST-2
57.1
74.4
79.7
MR
55.1
79.0
79.9
Comparison in runtime speed. Table 16 ($D.2) pro- vides comparison of the runtime speed of the parametric LM, RIC-LM, and İNN-LM. There is clear tradeoff between performance and speed: both RIC-LM and kNN- LM are considerably slower than the parametric LM, and a larger datastore and more accurate nearest-neighbo search leads to better performance and slower inference. We
RT
55.3
67.9
80.4
CR
64.6
83.1
80.3
Yelp
62.8
84.5
84.7
Amz
82,9
56.0
55.2
HYP
58.5
63.1
58.5
Avg
57.0
71.1
70.8
leave improving the runtime efficiency of nonparametric approaches for future work.
4.3 EXAMPLES OF DATA ATTRIBUTION AND OPT-OUT
As discussed in $2, the design of SILO can better align with various data-use regulations by providing mechanisms for data attribution during inference and for data owners to remove their data from the model at any time.
Data opt-out. To showcase the impact of opt-out on model performance, we conduct experiments with J. K. Rowling's 's Harry Potter series. We first identify all seven Harry Potter books from the Books3 corpus of the Pile. For each book, we calculate the perplexity of SILO using two 1B-token datastores on Books3, but one including the remaining six Harry Potter books and the other excluding any Harry Potter books. This experiment aims to see whether excluding Harry Potter books from the former datastore can reduce the likelihood of generating the left-out book.
8

---

Published as a conference paper at ICLR 2024
Table 6: Attribution examples on Harry Potter books. We show the top-1 retrieved context of SILO (PDSW). Red underlined text indicates the next token that immediately follows the prefix. In both examples, the test data is from the sixth novel and the retrieved context is from the fourth novel in the Harry Potter series. In the series, Azkaban is the notorious wizarding prison, and the green light is a distinct characteristic of the Killing Curse, Avada Kedavra.
Test Prefix 'I what dragons" spluttered the Prime Minister. 'Yes, three.' said Fudge. 'And sphinx. Well, good day to you. The Prime Minister hoped beyond hope that dragons and sphinxes would be the worst of it, but no. Less than two years later, Fadge had crupted out of the lire yel again, this time with the news that there had been Retricued Prefix 'D' you know Crouch, then?' said Harry. Sirius' face darkened. He suddenly looked as menacing as the night when Harry had first met him, the night when Harry had still believed Sirius be murderer. 'Oh, I know Crouch all right,' he said quietly. 'He was the one who gave me the order to be sent to Retrieved Continuation Azkahan without a trial.'
breakout from Test Continuation Azkabun.
Harry's heari... he had to get Dumbledore and had to catch Snupe.. somehow the two things linked... he could revense what had happened if he had them both together.. Dumbledore could not have died... (...) Harry felt Greyhack collapse against him; with stupendous effort he pushed the werewolf off and the floor jet of Test Continuation green light Retrieved Prefix Voldemort was ready. As Harry sbouted, "Expelliarmus!" Voldemort cried, "Avada Kedavra!" Retrieved Continuation green light issued from Voldemont's wand
Test Prefix Terror
A jet of
Table shows the results. SIL.O with Harry Potter books in the datastore (third column) effectively im- proves perplexity for all seven books, closing the gap between the PDSW model (first column) and Pythia (fourth column). However, when the Harry Potter books are removed from the datastore (second col- umn), the perplexity gets worse, approaching that of the parametric-only L.M. This illustrates that elimi- nating the effect of the Harry Potter books from the model substantially reduces the likelihood of gener- ating the leave-out book.
Table 5: The effect of data opt-out, report- ing perplexity. w/ HP and w/o HP indicate that the datastore includes or excludes Harry Potter books, respectively. The number to 7) indicates different book from the Harry Potter series used as the eval data; this eval book is not included in the datastore in any case. indicates in-domain; indicates out- of-domain.
SILO (PDSW)
Pythia
Bval
Attribution. To show the attribution feature of our model, Table 6 provides qualitative examples on the top-1 context retrieved by SIL.. The model is able to assign a high probability to the ground truth token by retrieving highly relevant context. achieves this by leveraging the unique characteristics of the text within the datastore, such as recognizing that Azka- ban refers the prison and green light is associated with the Killing Curse in the Harry Potter books.
Prm-only
ANN-LM
ANN-LM
Prm-only
w/o HP
w/ HP
1
15.9
15.2
13.0
9.6
2
17.7
16.7
12.4
10.0
3
16.5
15.6
11.4
9.5
17.7
16.8
12.9
10.1
5
17.8
16.9
13.2
10.2
6
17.4
16.5
12.8
10.1
7
18.8
17.8
15.1
10.9
Avg
17.4
16.5
12.9
10.1
More qualitative examples on Github, news and emails are illustrated in Table 17 in $D.2. They highlight that a nonparametric approach addresses specific legal risks that we have discussed earlier, e.g., it offers per-token attribution for free, and can provide a copyright notice when part of copyrighted text is being used for the probability distribution.
CONCLUSION
We introduce SILO, a language model that mitigates legal risk by learning parameters only on lower- risk, permissively licensed data (OPEN LICENSE CORPUS), and using an unrestricted nonparametric datastore during inference. Our approach allows the model designer to use higher-risk data without training on it, supports per-token data attribution, and enables data produces to opt-out from the model by removing content from the datastore. Experiments on language modeling perplexity and downstream text classification show that using only the parametric component of SIL.O is competitive on domains covered by OPEN LICENSE CORPUS, but falls short -oo-in, highlighting the challenge of extreme domain generalization. We then show that adding a nonparametric datastore (with LNN retrieval) successfully addresses this challenge, significantly reducing the gap (or even outperforming) the Pythia baseline that is trained unrestrictedly. Further ablations and analyses point to a number of exciting avenues for future work, which we highlighted in $E. Altogether, our results suggest that providing technical solutions to mitigate legal risk in AI systems is a promising and important direction for future research.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Peter Henderson and Mark Lemley for discussing the legality of LMs, and Kyle Lo for feedback our dataset and license taxonomy. We thank Mitchell Wortsman for help with setting up compute and model training. We thank Chris Callison-Burch, James Grimmelmann, Tatsunori Hashimoto, Peter Henderson, Nikhil Kandpal, Pang Wei Koh, Mark Lemley, Kyle Lo, Fatemeh Mireshghallah, Sewoong Oh and Rulin Shao for valuable feedback on the project and the paper. We thank Matthijs Douze, Gergely Szilvasy, and Maria Lomeli for answering questions about FAISS, and Dirk Groeneveld for giving early access to the deduplication script. Sewon Min is supported by the J.P. Morgan Ph.D. Fellowship. Suchin Gururangan is supported by the Bloomberg Data Science Ph.D. Fellowship. Eric Wallace is supported by the Apple Scholars in AI/ML Fellowship. We thank Stability AI for providing compute train the LMs in this work. At UW, this work supported in part by the Office of Naval Research under MURI grant 100014-18 11-0144----2---11 1-2670, the DARPA MCS program through NIWC Pacific (N66001-19-2-4000) NSF IS-204446600, and gifts from A12.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMMaan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. ACM SIGSAC, 2016.
ArXiv. arxiv dataset, 2023, URL
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, In Proceedings of the International Conference on Learning Representations, 2019.
Jack Bandy and Nicholas Vincent. Addressing "documentation debt"" in machine learning: retrospective datasheet for BookCorpus. In Proceedings of Advances in Neural Information Processing Systems, 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. Proceedings of the International Conference on Learning Representations, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-2005 An open-source autoregressive language model. arXiv preprint (rXi:22044006455 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Im- proving language models by retrieving from trillions of tokens. In Proceedings of the International Conference of Machine Learning. 2022.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette- Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning, 2020.
Blake Brittain. U.S. copyright office says some Al-assisted works may be copyrighted. Reuters, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al. Language models are few-shot learners. NeurIPS, 2020.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, 2015.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, 2021.
10

---

Published as a conference paper at ICLR 2024
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. Proceedings of the Interna- tional Conference on Learning Representations, 2023.
Caselaw Access Project. Caselaw access project. URL https://case.laww
Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to ChatGPT/GPT-4, arXiv preprint arXiv:2305.00/ 2023.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-Imagen: Retrieval-augmented text-to-image generator. In NewrIPS, 2022a.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik Learned-Milles and Chuang Gan. Mod-squad: Designing mixture of experts as modular multi-task learners, 2022b.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop. 2005.
Gerrit De Vynck. people's data, 2023. URL
ChatGPT
maker OpenAl faces a lawsuit over how used
http://ppootccoeeeeecooooo////// /ttmm//thhhhhty////eeee 2023//00///////// 22/
openai-chatgot-lassii---c www.saaattttssss--eeoeeaae actioon.tiio//...
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Proceedings of Empirical Methods in Natural Language Processing, 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography, 2006.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In Proceedings of the International Conference on Learning Representations, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint 1xXiv:211100022 2020.
David Gershgorn. Github's automatic coding tool rests on untested legal ground, 2021. URL https:/ www.theverge.com/22////7////////
github--opioo-llega--opiihhhhhhi wairruustttttute-eeereeeeeeeeeee puslic-pullcccooe
Aaron Gokaslan and Vanya Cohen. OpenWebText corpus. http://5kyyion007.gtubbb../ wit.ub.i..ggbuibo....o 2019.
Aditya Golatkar, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Training data protection with compositional diffusion models, 2023.
Dirk Groeneveld. The big friendly filter. https://gittubb.oo coa.o//ccao/maaaaaaaaaa allenai/beffif bff, 2023.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. DEMix layers: Disentangling domains for modular language modeling. In Conference of the North American Chapter of the Association for Computational Linguistics, 2022.
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery, 2023.
Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Proceedings of the Association for Computational Linguistics, 2023.
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the World Wide Web Conference, 2016.
Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. Pile Law: Learning responsible data filtering from the law and 256GB open-source legal dataset. Proceedings of Advances in Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2024
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Perey Liang. Foundation models and fair use. arXiv preprint arXiv:2303.15715 2023.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Proceedings of Advances in Neural Information Processing Systems, 2021.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competi- tion: Why the highest probability answer isn 't't always right. In Proceedings of Empirical Methods in Natural Language Processing, 2021.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Knowledge Discovery and Data Mining, 2004.
Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, and Danqi Chen. Privacy implications of retrieval-based language models. arXiv preprint arXiv:2305. 148888888.8888 2023a.
Yangsibo Huang, Daogao Liu, Zexuan Zhong, Weijia Shi, and Yin Tat Lee. ANN-Adapter: Efficient domain adaptation for black-box language models. arXiv preprint arXiv:2302. 10879, 2023b.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.0323 033999 2022.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In Proceedings of the International Conference of Machine Learning, 2023a.
Joel Jang, Dongkeun Yoon, Sohee Yang. Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. Proceedings of the Association for Computational Linguistics, 2023b.
J.L. et al. v. Alphabet Inc. Case 3:23-cv-03416, N.D. Cal., 2023. URL https:/ https///stoagge.uurrittteett com/recap/ gov. carcourrs.cand. 4 /
gov. www.uuscurts.comm cand www.:esnnnnnnnnn.....mm
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III freely accessible critical care database. Scientific data, 2016
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 2019.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In Proceedings of the International Conference of Machine Learning, 2022.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In Proceedings of the International Conference Leurning Representations, 2020.
Bryan Klimt and Yiming Yang. The Enron corpus: A new dataset for email classification research. In Proceedings of European Conference of Machine Learning, 2004.
Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. The stack: 3TB of permissively licensed source code. Transactions on Machine Learning Research, 2023.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the International Conference of Machine Learning, 2017.
Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. Copy is all you need. In Proceedings of the International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2024
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the Association for Computationall Linguistics, 2022.
Mark Lemley and Bryan Casey, Fair learning. Texas Law Review, 2020.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah Smith, and Luke Zettlemoyer. Branch-train-nes merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.00 22X0::2008.33366,, 2022.
Jimmy Lin, Xueguang Ma, Sheng-Chie Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the Association for Computational Linguisties, 2020.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. 1. Association for Computational Linguistics. URL http:
// //www.aclweb.org htwp://ew..rrg/ahoooggg////////e aataaooogg/llLL P11-1015.
Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne Trippas, J Shane Culpepper, and Alistair Moffat. CC-News-En: A large english news corpus. In Proceedings of the ACM International Conference on Information and Knowledge Management, 2020.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating di- mensions with review text. Proceedings of the ACM conference on Recommender systems, 2013.
Cade Metz. Lawsuit takes aim at the way A.I. is built. New York Times, 2022.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling. In Findings of ACL, 2023.
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint 1.XXi:23305.6666 2023.
Chester Palen-Michel. June Kim, and Constantine Lignos. Multilingual open text release 1: Pub- lic domain news in 44 languages. In Proceedings of the Language Resources and Evaluation Conference, 2022.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. Proceedings of the Association for Computational Linguistics, 2004.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer, 2020.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Conference of the North American Chapter of the Association for Computational Linguistics, 2022.
Shawn Presser. Books3 corpus, 2020.
URL http:://wwiteercc///hhaawww////
132028214932974883..
Project Gutenberg. Project gutenberg. URL www.gutenberg.oog
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
13

---

Published as a conference paper at ICLR 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmentedd language models. Transactions of the Association for Computational Linguistics, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recogni-
lion, 2022.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In Proceedings of the International Conference Learning Representations, 2019.
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. In Proceedings of Empirical Methods in Natural Language Processing, 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga. Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. REPLUG: detrieval-augmentt black-box language models. arXiv preprint arXiv:2301.126:
2023.
Silverman et al. v. Meta Platforms, Inc. Case 3:23-cv-034177 N.D. Cal., 2023. URL /storage. cuurlistener.com//eca//gg..v... wttp:/sscooutt..a........ www.://urroossoooooooooo waw.....11.......pp...o
Silverman et al. v. OpenAl, Inc. Case 3:23-cv-03417, N.D. Cal., 2023. URL https:/ httoss//storage.
courtlistener.com hwtptaeerr..ommercc//////...oe www.urts.ccmaaeeee 4151d4/1517///0 415174/gov.uscourt www.wuuscsssscooo 415174.1.0...ppff
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical report, Allen Institute for AI, 2023. ODC-By, https://github..om aalleaalleeaeeeeeeee pes2o.
Together. RedPajama: An open source recipe to reproduce LLaMA training dataset, 2023. URL httt:://fittuuu..omm/tgeehhro//p /tt.geeeeerroppuureeeeeeeeeeeeee https://github.ce
aeeermputtteeRRaaaaaaaaaaa Raeddaaaaaaaaaaaaaaaaaaaaaaaaaaa Dataaaaaaaaaaa.a. Data.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXhv:2302.13977 2023.
Inc. Tremblay et al. OpenAl. Case c-23-cv-03223 pourtistener..om/rreaaa wwt.goov..uouoo wwwourts. cand 414822 h/gggvvuusee wwwourts.cuooo.. cand. 414822.1. O.. pdf.
N.D. Cal., 2023. URL https: httpss//sstrrage
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems, 2017.
James Vincent. Getty images sues Al art generator stable diffusion in the us for
copyright infringement, 2023. URL https://wwwe www..tteeerrg.com/2233////////// satnagges-tabbleddffusooo
Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models. In Proceedings of the International Conference of Machine Learning. 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2005.10429, 2023.
Lili Yu, Bowen Shi, Ram Pasunuru, and Benjamin Miller. Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning, 2023.
14

---

Published as a conference paper at ICLR 2024
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2021.
Dawen Zhang, Pamela Baackenberg-Bromaan Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv preprint arXiv:2307.03941, 2023.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of Advances in Neural Information Processing Systems, 2015.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Proceedings of Empirical Methods in Natural Language Processing, 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Computer Vision and Pattern Recognition, 2015.
15

---

Published as a conference paper at ICLR 2024
A ADDITIONAL BACKGROUND
Legality of language models. The legality of training LMs this way has become subject of intense debate, with numerous lawsuits being filed in the United States, United Kingdom, and beyond (Gershgorn, 2021; Metz, 2022; Vincent, 2023; De Vynck, 2023; Silverman et al. Meta Platforms, Inc., 2023; J.L.. et al. v. Alphabet Inc., 2023; Silverman et al. v. OpenAl, Inc., 2023; Tremblay et al. v. OpenAI, 2023). While the outcome of the lawsuits is uncertain, it is likely that such legal issues will continue to be a major factor in future LMs, especially since each country has its own data regulations. For example:
In the United States, the fair use doctrine allows the public to use copyrighted data in certain cases, even without license (Henderson et al., 2023). Deciding whether or not model's 's use of copyrighted work constitutes fair use involves multiple dimensions, including whether the trained model is intended for commercial use, whether not the work is factual or creative, the amount of the copyright content used, and the value of the copyrighted work. There are claims that using parametric language models for generative use-cases does not constitute fair use. because the technology may output the copyrighted text verbatim (Lemley & Casey, 2020), which also has been shown empirically (Carlini et al., 2021; 2023; Kandpal et al., 2022; Chang et al., 2023). This is in contrast to transformative technologies, such as classifiers, which may use the copyrighted text but do not directly generate content, which the fair use doctrine favors. We refer readers to Henderson et al. (2023) for more comprehensive discussion.
The General Data Protection Regulation (GDPR) is comprehensive data protection and privacy law in the European Union (EU). It grants individuals more control over their data by regulating organizations and businesses. The obligations include (1) obtaining consent from users before processing their data, (2) providing transparency about data processing, (3) ensuring data security, and (4) allowing individuals to access, correct, and erase their data. GDPR has global impact, as many international companies handle EU citizens' data. While it is under debate how GDPR is applied to training language models, compliance with GDPR is expensive (e.g., requiring retraining for every data correction or erasure). See Zhang et al. (2023) for more discussion on challenges for compliance with the GDPR's Right Erasure (and the Right to be Forgotten in general).
The goal of our work is not to weigh in on legal discussions; instead, we study the feasibility of developing technologies that explicitly manage legal risk. In particular, our technique places all copyrighted data in a nonparametric datastore. While the data is still used in service of generative model, restricting copyrighted data in a datastore and providing instance-level attribution and data opt-out can increase the likelihood of a successful fair use defense (Henderson et al., 2022).? Moreover, GDPR's requirement regarding user data access, correction, and erasure aligns well with the capabilities of the datastore. Attribution and opt-out are fundamental features of our model ($3.2). This is in contrast to other techniques like post-hoe training data attribution (Koh & Liang, 2017; Han et al., 2023) and the removal of the effect of particular training examples from parameters (Cao & Yang, 2015; Jang et al., 2023b), which lack inherent guarantees and are hard to scale.
B DETAILS OF OPEN LICENSE CORPUS
In this section, we describe details of OPEN LICENSE CORPUS ($3.1): our taxonomy of data licenses ($B.1) and data analysis ($B.2).
disclaimer. The license taxonomy and categorization of texts that we present is by no means perfect, and OLC should not be considered universally safe-to-use dataset. The license associated with a document may be time- and country- are public domain in the United States, but some of them may still have copyright attached outside of the United States. Moreover, other legal constraints (e.g., the Digital Millenium Copyright Act) may prohibit the use of a data souree despite permissive data license. Finally, we do not explicitly filter out personally identifiable information from the corpus, so it is possible that certain subsets still
e.g., Gutenberg books (Project Gutenberg)
7 Our model on own does not entirely remove legal risk. Rather, it provides functionalities that, when used appropriately, lower legal risk and strengthen a fair use defense. See $E for a discussion. httpps: www..copyright.eog gov/ gov/dmca/
16

---

Published as a conference paper at ICLR 2024
Table 7: Overview statisties of OLC. PD. Sw. and BY indicates public domain data, data under permissive software licenses, and data under attribution licenses, respectively. Some corpora contain a mixture of different licenses (e.g.. Pile of Law and S2ORC), which we split into subsets based on per-document licenses. BPE tokens are based on the GPT-NeoX tokenizer (Black et al., 2022).
Domain
Sources
Specific License
# BPE Tokens (B)
PD Case Law, Pilc of Law (PD subset)
Public Domain
27.1
Legal
BY Pile of Law (CC, BY-SA subset)
CC BY-SA
0.07
Code
Github (penmissive)
MIT/BSD/Apache
58.9
Conversational
SW
HuckerNews, Ubuntu IRC
MIT/Apache
5,9
BY Stack Overflow, Stack Exchange
CC BY-SA
21.3
Math
Deepriind Math, AMPS
Apache
3.5
Science
PD
ArXiv ubstracts, S2ORC (PD subset)
Public Domain
1.2
BY
S20RC (CC BY-SA subset)
CC BY-SA
70.3
Books
Gutenberg
Public Domain
2.9
Public domain news
Public Domain
0.2
News
Wikinews
CC BY-SA
0.01
Encyclopedic
Wikipedia
CC BY-SA
37.0
pose privacy risks despite being permissively licensed. We encourage users of OLC to consult legal professional on the suitability of each data source for their application.
B.1 TAXONOMY OF DATA LICENSES
Determining what data one is permitted to use from copyright perspective is an ongoing topic of debate, and is context- and country-dependent (Henderson et al., 2023). In this paper, take conservative approach where we train models using only text with the most permissible licenses, thus enabling widespread downstream use. Concretely, we focus on four broad categories:
Public domain (PD) text has restrictions. This includes texts whose intellectual property rights have expired (e.g., the works of William Shakespeare) or been expressly waived by the creator (e.g.. CCO-licensed scientific papers).
Permissively licensed software SW including MIT, Apache, and BSD software quite permis- sive to use. Unlike public domain text, these licenses typically carry some basic stipulations such requiring one include copy of the original license (although, it is debatable whether it is still required when the associated text is used as data or treated as a software). The code is otherwise free to use, and code is generally well protected by fair use clauses (Lemley & Casey, 2020).
Attribution licenses (BY) such as Creative Commons Attribution (CC-BY) are free to use as long "credit is given to the creator." For example, if journalist quotes article from Wikipedia (a CC-BY source), then they must provide form of citation, link, or attribution back to the original source. the context of machine learning, is not clear what an attribution would constitute. For example, under one interpretation, every LM generation should include a complete list of sources that contributed highly (Henderson et al., 2023). In this paper, we take a conservative approach and do not include BY data in the main experiments, but still include the BY data for future use as well as for ablations, since BY data is generally considered quite permissive.
All other data that is not in one of the above three categories is assumed to be non perm-ermmssveee This includes: any text that is explicitly protected by copyright or licenses that are non-commercial (e.g., CC-NC), any software without clear MIT, BSD, Apache licenses, and any generic web-crawles data where the license or copyright information may be unclear.
In $3.1, we train the models on varying subsets accommodate different risk tolerances.
of licenses from PD and PDSW to PDBYSW to
17

---

Published as a conference paper at ICLR 2024
B.2 DATA ANALYSIS
To study the composition of OLC, we perform an n- g-gram based analysis of OLC domains against the validation data of the Pile. This enables us better understand the dataset's domain shifts. We sample up to 10M tokens in each data source of OPEN LICENSE CORPUS and each subset of the Pile validation data. We then compute unigrams and bigrams across domains both datasets (ignoring stopwords). and only consider n-grams that appear at least three times in each domain. For each validation domain of the Pile, we examine the maximum 7-gram overlap across all OLC domains.
OLC domains have substantially less overlap with the validation data as compared to the Pile training domains: on average, the overlap between OLC domains and the validation domains is just 17%= versus 14% for the Pile training data. However, across domains in OLC; we display the full matrix of n-gram overlap in Table 18. These results provide further evidence that models trained OLC must handle larger domain shifts at test time than models trained on the Pile.
find large variance in overlap statistics
C EXPERIMENTAL DETAILS
C.I DETAILS ON THE PARAMETRIC COMPONENT
LM architecture. We use 1.3B parameter transformer LMs based the LLaMA architecture (Tou- vron et al., 2023) as implemented in OpenI Table 8 reports the hyperparameters for the parametric component of SILO. We keep these hyperparameters fixed for all parametric models that we report in this paper. We use the GPT-NeoX- --BB tokenizer (Black et al., 2022), with 50432 BPE types.
Table 8: Basic hyperparameters for the parametric component of SILO.
Model
#L
#H
dmodel
LR
Batch
1.3B
24
16
2048
lc-3
2.6M
Training. Each model is trained with 128 A100 GPUs across 16 nodes. We use 2,048 token sequences that are packed across document boundaries, and we pre-pend a beginning-of-te off-tet token to every document. We use weight decay of 0.1, the Adam optimizer with 3g2 0.95, 2,000 steps of warmup, with cosine learning rate scheduler. Following Muennighoff et al. (2023), we train for multiple epochs in each dataset, tracking validation perplexity every 10B tokens, and perform early stopping. As result, we train our PD. PDSW and PDSWBY models for 60B, 250B, and 350B tokens in total, respectively.
Domain weightit, Since the distribution of OLC is highly skewed ($3.1), we perform simple upweighting scheme where we upsample all data that accounts for less than 5% by factor of 3 xx, which we found to work well after a sweep of different settings. More sophisticated domain weighting strategies (Xie et al., 2023) are of interest but beyond the scope of this work.
C.2 DETAILS OF ADDING THE NONPARAMETRIC COMPONENT
For ANN-LM,, each datastore consists of up to I billion /-dimensional vectors (h- =2,088. We build an index for fast nearest neighbor search using FAISS (Johnson et al., 2019). We use IndexIVFPQ which quantizes vectors into 64-bytes and clusters them into 4,096 centroids, learned from million sampled vectors, following Khandelwal et al. (2020). Instead of recomputing the exact L2 distance using the original embeddings, we use the L2 distance beteen quantized vectors returned by the FAISS index (ablations in Appendix D.2). Since their scale not preserved, we use d(x,.Ya) as proxy of d(x, where x, and y, are vectors quantized from x and y. Hyperparameters, including k. X, and 7, are chosen based on the validation data domain- -specificc manner.
For RIC-LM, each datastore consists of text blocks with length of 1,024 and sliding window of 512. We use BM25 from Pyserini (Lin al., 2021). Appendix D.2 report ablations on different
http:///huubbboo//m/aaaaattns//s Im
18

---

Published as a conference paper at ICLR 2024
Table 9: Datastore statistics as well as hyperparameter values for ANN-LM. Underline indicates exact nearest neighbor search (instead of approximate) was performed for kNN-LM because the datastore small enough. Hyperparameters are chosen based on the validation data of each domain.
RIC-LM
KNN-LM
Data
# tokens
# blocks
# tokens
A
k
Github
3084.3M
6.0M
1024.0M
0.2
128
10.0
NIH ExPorter
72.2M
0.1M
72.2M
0.3
32,768
20.0
Wikipedia
1177.9M
2.3M
1024.0M
0.3
4,096
20.0
CC News
382.2M
0.7M
382.2M
0.7
4,096
20.0
Books3
1424.7M
2.8M
1024.0M
0.2
4,096
25.0
Enron Emails
45.0M
0.IM
45.0M
0.5
4.096
1.0
Amazon
1214.3M
2.4M
1024.0M
0.5
32,768
20.0
MIMIC-III
519.5M
1.0M
519.5M
0.7
1,024
15.0
Table 10: Datastore and hyperparameter values for &NN-LM evaluated downstream tasks. Hyper- parameters (the choice of datastore, X, and т) are chosen based on the validation data.
Task
Datastore
Template
Label Words
AGN
Wikipedia
0.8
1024
3
The tupic of the text is
politics, sports, business, technology
Dbpedia
Amazon
0.9
4096
3
The topic of the text is
company, school, artist, athlcte (14 in total)
SST-2
IMDB
0.9
8192
3
It was was was
great, terrible
MR
IMDB
8192
great, terrible
IMDB
8192
great, terrible
CR
Amazon
1024
1
great, terrible
Yelp
IMDB
0.9
8192
3
great, terrible
Amz
IMDB
0.9
8192
3
It was truc or false? answer:
great, terrible
RTE
Amazon
0.9
1024
3
truc, false neutral, partisan
НYР
CC News
0.1
1024
1
neutral partisan? unswer:?
implementations of RIC-LM besides the method in $3.2, including different ways of choosing the retrieved context and using multiple text blocks.
Table 9 reports the datastore statistics for both RIC-LM and ANN-LM, well as hyperparameter values for &NN-LM (A, k 7). Due to the resource constraints, the datastore size capped up to 10% of the PILE training data (and to 1024.0M tokens in the case of &NN-LMM, but future work can investigate further scaling the datastore.
C.3 DETAILS OF LANGUAGE MODELING PERPLEXITY EVALUATION
We benchmark our models using language modeling perplexity on 14 domains that represent both in-domain and out-of- oft-of-domain data with respect to different levels of OLC. This includes: public- domain legal documents from the FreeLaw Project subset of the the Pile (Gao al., 2020). held-out collection of books from the Gutenberg collection (Project Gutenberg), conversational text from the Hacker News subset the Pile, held-out code files from the Github subset of the Pile (most of which are non-permissive licensed), scientific text of NIH Grant abstracts that are taken from the NIH ExPorter subset the PILE, philosophy papers taken from the PhilPapers of the PILE, held-out English Wikipedia articles from the PILE, news articles from CC-News (Mackenzie al., 2020), books from Book Corpus2 which an expanded version of Zhu et al. (2015), books from Books3 by Presser (2020), random web-crawled pages from Open Wex2ext2 (Gokaslan & Cohen, 2019; Gao et al., 2020), emails from the Enron Emails corpus (Klimt & Yang, 2004), Amazon product reviews from He & McAuley (2016), and finally clinical notes from MIMIC-III (Johnson al., 2016) with personal identifiable information (PII) masked out. Our choice of domains highlights legal risks discussed in the earlier sections, e.g., CC-News, BookCorpus2, Books3 and Amazon reviews are mostly copyrighted, Github is mostly not permissively licensed, 10 and Enron Emails and MIMIC-III include private text.
Kocetton et al. (2023) estimates about 13% of the Github data is under MIT, Apache, and BSD, permissive licenses defined by Together (2023).
19

---

Published as a conference paper at ICLR 2024
Table 11: Perplexity on the parametric LMs trained on PD, PDSW, and PDSWBY, as well as Pythia 1.4B, a model trained with similar amounts of compute but non- non-permissive data. We use m, and indicate text that in-domain, out-of-domain, out-of-domain but has relevant data in-domain data (e.g., nen-permissive Github code versus permissive training code). Reported on validation data; see Table for results on the test data.
Eval data
PD
PDSW
PDSW
Pythia
FreeT
5.3
5.7
5.6
Gutenberg
14.6
11.9
13.4
12.7
HackerNews
12.1
13.2
Github
2.6
2.4
NIH ExPorter
28.6
19.3
15.1
11.2
PhilPapers
55.2
24.2
16.5
14.3
Wikipedia
27.9
19.7
11.1
9.0
CC News
30.8
21.3
19.3
10.9
BookCorpus2
25.2
19.2
20.2
12.8
Books3
25.9
18.7
18.1
12.4
OpenWehTex12
38.1
21.2
18.8
Enron Emails
19.9
14.3
14.5
7.6
Amazon
34.7
37.0
MIMIC-III
18.2
16.4
13.6
11.5
Average
30.1
17.2
15.7
11.2
We merge all text into stream of text and split them into batches with maximum sequence length of 1,024 and a sliding window of 512, a setup that is standard in prior language model- ing literature (Baevski & Auli, 2019; Khandelwal et al., 2020). For MIMIC-III, which includes masked personally-identiin correspond to PII, and then exclude tokens corresponding PII when computing perplexity.
information (PII), we filter out notes where more than 50% of tokens
C.4 DETAILS OF DOWNSTREAM TASK EVALUATION
We perform zero-shot prompting for nine text classification datasets: AGNews (Zhang et al., 2015), Yahoo (Zhang et al., 2015), Subj (Pang & Lee, 2004). SST-2 (Socher et al., 2013), 3), MR (Pang & Lee, 2004), Rotten Tomatoes (RT), CR (Hu & Liu, 2004), Amazon polarity (Amz, McAuley & Leskovec (2013)) and RTE (Dagan et al., 2005). The tasks range from topic classification and sentiment analysis to subjectivity classification and textual entailment.
We use the templates and label words to map the task into sentence completion problem, following the standard from literature (Radford et al., 2019; Brown al., 2020; Holtzman et al., 2021). These templates and label words are taken from Shi et al. (2022), reported in Table 10. For both parametric LMs and &NN-LM, we apply the domain-conditional PMI scoring (Holtzman et al., 2021) for determining the probability of each label. For ANN-LM, we follow method from Shi et al. (2022) which employs the fuzzy verbalizers to expand the token set associated with each output label in our task. Also following Shi et al. (2022), an additional datastore. We perform hyperparameter search on the validation dataset of each task, considering l E {128, 512, 4196, 8192), (1,3, 5, 10, 40, 80), and different choice of datastores. The chosen hyperparameters are reported in Table 10.
IMDB (Maas et al., 2011) with 8 million tokens as
D ADDITIONAL EXPERIMENTAL RESULTS
Table ll reports perplexity of the parametric LMs the validation data that is analogous to Table 2. Table 12 reports perplexity of both parametric and nonparametric LMs on the validation data that is analogous to Table 3. Findings based on the validation data and on the test data are largely consistent.
D.1 ABLATIONS: PARAMETRIC COMPONENT (SECTION 4.1)
Effect of upsampling low-resource data. Since OPEN LICENSE CORPUS has extremely skewed distribution of domains, we upsample tess-representative domains during training. Table 13 (left) compares the models trained PDSW with and without domain upweighting. In-domain datasets
20

---

Published as a conference paper at ICLR 2024
Table 12: Perplexity parametric LMs (Prm-only), ANN-LM and RIC-LM: indicates in-domain; indicates out-of-domain; indicates out-of-domain but has relevant data in-domain. Reported on the validaiton data; see Table for results on the test data.
Pythia
Eval data
Prm-only
RIC-LM
Prm-only
Github
2.6
2.4
2.4
2.4
NIII Ex
19.3
14.9
18.5
11.2
Wikipedia
19.7
14.1
18.9
9.0
CC News
21.3
7.1
14.8
10.9
Books3
18.8
17.3
18.5
12.5
Enron Emails
6.7
11.1
7.6
34.7
26.2
33.7
22.8
MIMIC-III
16.3
7.2
14.1
11.5
Average
18.4
12.0
16.5
11.0
Table 13: (Left) Effect of re-weighting rare domains, comparing models trained on OLC (DSSW) with and without upsampling. (Right) Effect of SW data, with and without explicit source code we train an LM with data but remove all of the actual source code (i.e., leave Hacker News, Ubuntu IRC, Deepmind Math, and AMPS). Both tables report perplexity on the validation data.
PDSW
PDSW
PDSW
Data
w/o upsampling
upsampling
Data
w/o code
PDSW
FreeLaw
4.9
5.7
FreeLaw
5.3
5.7
5.7
Github
2.4
2.6
Github
13.3
8.2
2.6
NIH ExPorter
20.0
19.3
NIH ExPorter
28.6
26.2
19.3
PhilPapers
23.9
24.2
PhilPapers
55.2
36.4
24.2
Wikipedia
19.9
19.7
Wikipedia
27.9
26.5
19.7
CC News
21.8
21.3
CC News
30.8
28.8
21.3
Book BookCorpus2
19.4
19.2
BookCorpus2
25.2
23.8
19.2
Open OpenWebTest WebText2
21.0
21.2
Open WebText2
31.7
21.2
Enron Emails
13.5
14.3
Enron Emails
19.9
18.5
14.3
Amazon
35.7
34.7
Amazon
81.9
46.1
34.7
that are not upweighted, e.g., FreeLaw, see slight degration in performance. On out-of-doain datasets, there no significant differences, although the model with upsampling is marginally better (19.6 vs. 19.7 when averaged over out-of-domain datasets), We note that we did not tune the upweighting ratio nor explore alternative upweighting approaches (Xie et al., 2023) due to resource constraints, and leave them for future work.
59B tokens of source code significantly help. When using Sw, substantial 59. 1 of the training data actual source code. To determine SW provides such large gains, we also run an ablation where include SW data but exclude all of the actual source code, i.e., only include Hacker News, Ubuntu IRC, Deepmind Math, and AMPS on top of the PD data. This leaves models trained on 99.6B tokens for OLC (PDSW) and 40.7B for OLC (PDSW) excluding source code. Table 13 (right) report results on subset of the validation domains. Including source code provide significant benefits for certain test datasets, e.g., nearly a 20 point improvement perplexity PhilPapers, likely because significantly increases the size of the training data.
Performance gaps with Pythia are strongly correlated to domain shift. We hypothesize that the performance differences we see between our models and Pythia can be explained by domain shift between our corpus and the evaluation domains. To confirm these effects, we show a scatterplot that describes the relationship between ngram overlap of the OLC and the Pile (from $B.2 with the performance gap between PDSW and Pythia in Figure There is a strong negative correlation between these two metrie (r=-0.72, p 0.005).
21

---

Published as a conference paper at ICLR 2024
NHH ExPorter
Enron_
Prm-orly +RIC-LM +XNN-LM
Prmvorly +RC-LL
Prm-only +RCLMM
KNN-UM
ONNLM
PDSW
POSWBY
Pythia
PD
PDSW
Pytha
PDSW
POSWBY
Pythi
Figure 3: Results on different variants of models: PD, PDSW and PDSWBY variants of SILO as well as Pythia. Adding a nonparametric component through either RIC-LM and &NN-LM helps and ENN-LM is overall better than RIC-LM, consistently across all models and evaluation datasets.
Porplxxty (Mik) CONews Amazon MMMIC)
35
Amazon
2.60
Gthub
30
Wikpedia
CC News
2.55
MIMIC-II
Perpkedly (Github)
20
2.45
2.40
15
2.35
10
2.30
LM for PLM
Pythia
Pythia
Pythia
Encoder for PLNN
X
Pythia
Ours
Pythia
Ours
X
Figure 4: Impact of using different parameters on SILO. Perplexity on random 128K tokens from the validation data reported. The left-most and the right-most models are parametric models, and the other four models are &NN-LMs, using datastore with 204.8 million tokens (20% of the datastore we use for the main experiments). Ours indicates our parametric model trained on the PDSW subset of OPEN LICENSE CORPUS. Most of the performance degradation comes from using the out-of-domain parametric LM, rather than using the out-of-domain encoder.
D.2 ABLATIONS: ADDING THE NONPARAMETRIC COMPONENT (SECTION 4.2)
Impact of adding nonparametrie component across varying LMs. We compare parametric- only LM, RIC-LM and ANN-LM over four different LMs: PD, PDSW and PDSWBY variants of SILO well Pythia. Figure 3 reports their results three evaluation datasets: Wikipedia, NIH Ex ExPorter and Enron Emails. Findings from Section 4.2 hold across all models: both RIC-LM and kNN-LM are consistently better than the parametric-only LM, and KNN-LM overall achieves the best performance. For instance, KNN-LM allows the model to be comparable to or outperform the one-level relaxed variant, e.g., PD-baaed ANN-LM is comparable
based parametric LM, and PDsw-based ENN-LM is comparable to PDSWBY- BoOSBY-bbsed parametric LM.
Where does the remaining gap come from? Even when scaling the datastore with ANN-LM, SILO lags behind Pythia on few domains. Moreover, Pythia-based based ANN-LM outperforms our model since KNN-LM improves Pythia as well. There are two possible points of failure our model for these cases: either the parametric component (which outputs PLM) struggles out-of-domain, or the encoder (that outputs PrnN) struggles out-of-domain. model contributes to the gap observe, vary SILO with different choices for the parametric component and the encoder. We compare replacing either the parametric component or the encoder with Pythia. This setup allows to measure the effects of the out-of-dommin nature of our parametric component (which is only trained on PDSW subset of OLC) in each of these components.
better understand which part of the
Results in Figure reveal that most performance gaps come from the LM: performance improves significantly when the parametric component is replaced with Pythia, given fixed encoder. In contrast, performance improvement is relatively marginal when the encoder is replaced with Pythia,
22

---

Published as a conference paper at ICLR 2024
Table 14: Ablations on different variants of RIC-LMs. Perplexity on the validation data reported. ensbl-10 is 10x slower than other three methods.
PDSW
Pythia
Data
basic
ensbl-10
concat-2
concat-next
basic
cnsbl-10 10
concat-2
concat-next
CC News
14.8
13.5
17.0
18.8
8.2
7.9
9.2
9.9
Enron Emails
11.1
10.0
12.8
13.4
6.3
6.117
7.1
7.3
given a fixed parametric component. These results indicate that the parametric component, which gives Pt.. is quite sensitive domain shift, but the encoder, which provides the nonparametric distribution PENN, is fairly robust to extreme domain shift. This also explains why ANN-LM generalizes better than RIC-LM, since RIC-LM is bottlenecked by the parametric component.
Effect of scaling the datastore in-domain and out-of-domain. $4.2 shows that performance of both kNN-LM and RIC-LM rapidly improves the datastore size grows, and KNN-LM improves more rapidly than RIC-LM does. This evaluation is mainly done with S1L.O where the test domains out-of-domain. Does this trend hold when the test domains are in-domain? To answer this question, we examine effect of scaling the datastore with Pythia 1.4B, where all of our test datasets can be considered in-domain.
Figure 5 reports the results: Pythia on the left, SILO (PDSW) on the right. Results show that both Pythia and SILO see consistent improvements from ANN-LM and RIC-LM as the datastore gets larger, although the slope is larger with SILO than with Pythia. Again consistent to findings in $4.2, kNN-LM scales better than RIC-LM does, resulting in kNN-LM outperforming RIC-LM with reasonably large datastore in most cases (with an exception of Pythia on Github, where RIC-LM outperforms &NN-LM with reasonable size of datastore).
Ablations on variants of RIC-LM. We explore four different variants of RIC-LM. (1) The basic the method described in $3.2, which uses text blocks with a length of L each. At inference, takes the top I text block from the datastore and feeds it to the LM, i.e., (u x)) where x is the input and ò is the top 1 text block. (2) The ensbl-k (k- = 10) variants is also based on text blocks with a length of L each. At inference, it takes the top & text blocks from the datastore, feeds each to the LM in parallel and aggregates the probability distributions, i.e., 1/2 Σi<teh PLM br, where b... the top i text blocks. This follows method from Shi et al. (2023). (3) The concat-k: (A: 2) variant uses text blocks with a length of & each. At inference, it takes the top k text blocks from the datastore, concatenates them in a reverse order, and feeds it into the LM, e.g., PL
6,,2)
where by.bby are the top text blocks. (4) The concat-next variant uses text blocks with length of L each. At inference, it takes the top 1 text block from the datastore, concatenates the text block and 2 the subsequent text block in a datastore, and feeds it into the LM. This is based on the intuition that the continuation of the text block that is most similar to the query be useful for the continuation of the query; Borgeaud et al. (2022) has explored a similar approach based on the same intuition. We use T. 1024 for all variants. The ensbl-k variant has run-time that is approximately k: times of run-time of the basic, concat-k and concat-next.
Results are reported in Table 14, The concat-2 and concat-next variants perform poorly, while the ensbl-10 outperforms the basic variant. However, we reached the conclusion that the significant run-time cost (i.e., 20x compared to parametric LM) does not justify the improvements, and thus, we primarily use the basic variant for the remaining experiments. Future work may involve re-evaluating models using the ensbl-k approach or enhancing its run-time efficiency.
Effect of different approximation methods for L2 distance. Prior work (Khandelwal et al., 2020) typically uses approximate nearest neighbor search find the top l nearest neighbors, and then computes the exact L2 distance using the original vectors. However, this may be inefficient in disk memory usage and run- run-time speed, due to needing store large, original vectors and access them on-disk. We thus explore a few alternatives: (I) quantizing the original vectors to compute the L.2 distance (but less aggressively than quantization for the nearest neighbor search index, thus it provides different levels of approximations for search and for L2 distance), or (2) completely dropping the original vectors and relying on approximated L2 distance from the index with aggressive quantization.
23

---

Published as a conference paper at ICLR 2024
Table 15 shows that all approximation methods only marginally affect performance. For this reason, we use the most aggressive approximation that completely drops the orig- inal embeddings at the cost of about 0.5% lose in performance while using 2% of the memory footprint. Future work may study more accurate and efficient approximation methods.
Table 15: Ablations on approxima- tion methods on the validation data of Wikipedia, using the LM trained on PDSW and the datastore consist- ing of 51.2 million tokens (5% of the datastore in the main experi- ments), Relative disk memory usage reported (considering no approxima- tion as 1.0).
Runtime speed. Table 16 presents the runtime speed of the parametric LM, RIC-LM, and kNN-LM the Wikipedia validation set. Speed is reported in tokens per second with a batch size of using single NVIDIA RTX 6000 GPU.
Method
PPL Disk use
The results show that the parametric LM is notably faster than both RIC-LM and &NN-LM, and RIC-LM is faster than KNN- LM. Speed is slower as the datastore gets larger (for both RIC-LM and kNN-LM) and the nearest neighbor search gets less accurate (for ANN-LM; indicated by the number of probe p). ANN-LM can eventually match RIC-LM's speed while surpassing its performance by using smaller datastore and less accurate search, i.e., when using 102M tokens with p =
Param-only
19.7
0.0
No approximation
16.4
1.0
Quantized (4x)
16.6
0.25
Quantized (8x)
16.6
0.125
Quantized (16x)
16.8
0.0625
IVFPQ approximation
16.8
0.0178
However, the machine used for benchmarking has a very slow IO speed, leading to an underestimation of both RIC- LM and kNN-LM's runtime speed, and the comparison can significantly vary based on the hardware. Either way, &NN-LM is currently substantially slower than paramet- ric LM, leaving room for potential future improvements.
Table 16: Comparison in runtime speed tokens per second) on the validation data of Wikipedia. p indicates the num- ber of probe, one of the hyperparameters in fast nearest neighbor search (p 8 in all experiments in the paper if not speci- fied otherwise).
Qualitative examples. Figure 17 provides six qualita- tive examples on the top-1 context retrieved by SILO-based kNN-LM. The model able to assign high probability to the ground truth token by retrieving highly relevant con- text, e-g-, given the context (hockey) and the first name of the player, being able to retrieve the last name of the player, given the context (a show and its host), being able to complete the quote. These examples also highlight that a nonparametric approach addresses specific legal risks that we have discussed earlier, e.g., it assigns per-token at- tribution for free, and can provide copyright notice when part of copyrighted text is being used for the probability distribution.
Method
PPL # tokens/s
Param-only
19.7
1828.6
RIC-LM (51.2M)
19.3
812.7
RIC-LM (102.4M)
19.2
731.4
RIC-LM (204.8M)
19.1
588.5
RIC-LM (409.6M)
18.9
478.5
RIC-LM (1,178M)
18.9
419.7
ANN-LM (51.2M)
16.8
184.2
ANN-LM (102.4M)
16.3
112.0
ANN-LM (204.8M)
15.7
59.3
ANN-LM (M)
15.0
31.8
ANN-LM (1.02 (1.024M)
14.2
14.2
ANN-LM (102M. 1)
16.7
560.8
E DISCUSSION & FUTURE WORK
ANN-LM (1, (1,024M,
1)
14.6
71.1
ANN-LM (1,024M, p=
2)
14.4
45.5
ANN-LM (1,024M. P 4)
14.2
27.0
Our work suggests that it possible to improve the trade- off between legal risk and model performance when train-
ing LMs. Our approach provides new options for model designers to mitigate the legal risk of LMs, and empowers stakeholders have more control over the data that drives these systems. We point out a number of rich areas for future work, beyond what was mentioned throughout the paper:
A Addressing the limitations of SILO. SILO does not completely eliminate legal risk. Instead, provides users more control over the model's generated content and functionalities to better align with legal regulations. For instance, SIL.O does not remove the need for obtaining permission to use copyrighted content in a datastore when providing attribution is not sufficient, but its opt-out capabilities can strengthen fair use defense. Moreover, SILO does not prevent copying copyright content from datastore, but it offers a way to prevent generating sensitive text (Huang et al., 2023a) or prevent copying the content verbatim. These functionalities increase the likelihood of a successful fair use defense if used appropriately.
Furthermore, while SILO mitigates copyright and privacy risks, it may exacerbate certain fairness issues, like toxicity towards marginalized groups and racial biases, especially due to the prevalence
24

---

Published as a conference paper at ICLR 2024
of older copyright-expine books in the training data. Exploring the balance between legal risk mitigation and fairness is an important future direction.
Finally, our study relies explicit metadata to identify licenses, which may lead to underestimates of the amount and diversity of permissively licensed text actually available on the web. Future research may investigate inferring data licenses from documents in web crawl at scale, which may be an effective way to build more heterogeneous, permissively licensed corpora.
Introducing novel data licensing approaches. SILO introduces the possibility for data owners to set different levels of permissivity for learning parameters and for including in nonparametric datastore. A data owner might choose to be more permissive about including data in the datastore due its ease of removal, ensuring that the excluded data has no influence on model predictions anymore, and its ability to provide per-prediction attribution. Moreover, we envision that SILO could provide path forward for data owners get properly credited (or be paid directly) every time their data in a datastore contributes to prediction. This is orthogonal to recent work that circumvented copyright issues by licensing out training data from data creators (Yu al., 2023).
Investigating other copyright risk mitigation strategies. It is critical to continue to develop new techniques that use copyrighted data while protecting the rights of data owners and subjects. In addition to nonparametric approaches, there are many other ways achieve these goals. First, one could train LMs on copyrighted content but filter and guide their outputs towards text that is non-infringing (Henderson al., 2023). Second, training models with differential privacy (Dwork et al., 2006; Abadi et al., 2016) near-access freeness (Vyas et al., 2023) may prevent them from regenerating individual details of copyright data. Finally, one could provide attributions for standard base LMs using post-hoc attribution methods, e.g., influence functions (Koh & Liang, 2017), rather than switching the model class to a retrieval-bast model. All these methods are complementary and orthogonal to our proposed approach.
Generalizing SILOO modular language model. Our work is closely related to recent studies on modular LMs, which have specialized parameters (or experts) trained on different domains (Gururangan et al., 2022; Li et al., 2022; Gururangan al., 2023), languages (Pfeiffer et al., 2020; 2022), or tasks (Chen et al., 2022b; Jang et al., 2023a). Our work extends modular LMs to include nonparametric datastores, and focuses on specializing different parts of the model to low- and high- risk subsets of the training data. Legal risks may also be mitigated with a collection of parametric expert models that are specialized to low- and high-risk data. Future work may explore this possibility as well as the usefulness of combining nonparametric datastore with parametric experts.
Extending STLO to other modalities. While this work focuses on text-only models, similar methods
could apply other domains and modalities. For instance, it might be possible to build permissive text-to-image generative models (Rombach et al., 2022) using compartmentalized public domain pre-training and aetrivall-augmentane believe such approaches are especially promising because there are many sources of public domain data in other modalities, e.g., images, speech, video, and more.
(Chen al., 2022a: Golatkar al., 2023). We
25

---

Published as a conference paper at ICLR 2024
Wikipedia (Pythia)
Wikipedia (Ours)
KNN
12
Parametric LM
RBC
20
Perplexdly
Perplexityy
18
10
16
KNN
+ RIC
6
Parametric LM
10
10
10"
10"
10"
10
10'
10
10"
10"
# Tokens in Datastore (millions)
# Tokens in Datastore (millions)
CC News (Pythia)
CC News (Ours)
25
+MNN
20
RBC
20
Parametric LM
Perplexity
16
Perplexity
16
10
10
KNN
6
6
RBC
Parametric LM
10
10
10
10'
10"
10
10'
10
10"
10
# Tokens in Datastore (millions)
# Takens in Datastore (millions)
MIMIC (Pythia)
MIMIC III (Ours)
14
16
12
14
Perplexity
Perplesity
10
6
KKNN
Parametric
Parametric LM
10'
10"
10"
10"
10"
10'
10"
10
# Tokens Datastore (million)
Tokens Datastore (millions)
Amazon Reviews (Pythia)
Amazon Reviews (Ours)
30
28
RBC
Parametric
32
Perplesity
Perplexity
30
24
28
KNN
26
RCC
20
Parametric LM
24
100
10
10a
10
10
10
10'
10"
# Tokens in Datastore (millinn)
Tokens in Datastore (millions)
Github (Pythia)
Github (Ours)
KNN
KNNN
2.6
RIC
2.8
RIC
Paramotric LM
Paramotric LM
Perpleslly
Perplesitty
2.4
266
2.2
2.0
10
10'
10a
10"
10
10a
10'
10
102
10"
Tokens Datastore (million)
# Tokens in Datastore (millions)
Figure 5: Comparison between parametric LM, RIC-LM and KNN-LM on five domains, with Pythia (left) and SILO PDSW (right), respectively. Perplexity on random 128K tokens from the validation data reported.
26

---

Published as a conference paper at ICLR 2024
Table 17: Qualitative examples of retrieved context our model. Red underlined text indicates the
next token that immediately follows the prefix. The first two are from Github; the next two are from Enron Emails; and the last two are from CC News.
Test Prefix
include "./lib/adminn.c wttphiiinddfiins.hhpp..
include "./lib/admin.net www...ooueeeeee.. www.uule.cccsss..ppg...... access.php";
include "./lih/admin.smaae haatiiii..maatt.hyh. php";
if (! highttghts
Test Continuation ACX. BILLING)) Heederr...
Retrieved Prefix
(..)
You should have received copy of the GNU Affero General Public License
along with this program. If not, http://wwwww..//w/eeeeee..e
ss//
(! hights (
Retrieved Continuation ACX.ACCESS)) Header
Test Prefix
0x5f ildefine S5K4AA DEFAULT BRIGHTNEESS 0x10
Kernel
Test Continuation module parameters */ extem int force sensor:
Retrieved Prefix
Copyright 201 1-2013 Jozsef Kadleesik <kadlcc@blackholle
This program is software; you can redistribute it and/or modify under the terms the GNU General Public License version as published the Free Software Foundation.
*/
1 Kernel
Retrieved Continuation module implementing an IP sct type:
Test Prefix Mark eredit about hedge funds? Sara
Sara Shackleton
Enron North America Corp. [Address]
[Phone number]
[Email address]
Forwarded by Sara Shaclcton/HOU/ECT on 01/2023/2022 05:41PM
Tana
Test Continuation Jones 12/14/2000
Retrieved Prefix ... Food will provided! Tana: Please feel to extend invitation any Enron employees who may be interested in presentation. 1st come, 1st serve. Thanks, Sylvia. Forwarded by Sylvia Hu/Corp/Enron on
07/14/2000 03:17PM Tana
Retrieved Continuation JoC 07/13/2000
Test Prefix Ken Lay and Jeff Skilling were interviewed on CNNín to discuss the succession of Jeff to CEO of Enron. (..) and then choose "Enron's ' Succession Plan.". The interview will available every 15 minutes
Test Continuation through Friday, Dec. 15.
Retrieved Prefix Did you miss Jeff on CNBC "Street Signs" yesterday? Not to worry. (..) and then choose >>
"Skilling
CNBC.. The interview will available every ten minutes
Retrieved Continuation through > Wednesday, Dec. 6.
Test Prefix The teams toured the city, explored west Edmonton mall and also got to take in Oilers practice where they met German hockey star Leon
Test Continuation Draisaitl Retrieved Prefix One minute and 19 scconds later, Cannor McDavid took pass from Leon
Retrieved Continuation Draisaitl
Test Prefix ... Foley RAW's run-time issues. Claiming that having the show run late
reasons why the
final hour of RAW tends struggle, Foley didn't end there. "No else 10:30pm is
Test Continuation PG show. won't say that across
Retrieved Prefix way to the ring' podcast Foley cited RAW's duration and RG rating as hindrances to the show's popularity. Here's what he had to say: "Sometiems we try to look into the reasons why the third hour docsn't 't perform us well as the first two, and I'm like 'well that's because people go to bed! No one else at 10:30pm is
Retrieved Continuation PG show. won't say that across
27

---

Published as a conference paper at ICLR 2024
Table 18: Unigram and bigram overlap between the domain of the Pile validation data and the
domains of OPEN LICENSE CORPUS.
Dataset
BookCorpus2
Books3
Enron Emails
FreeLaw
Github
Gutenberg (PG-19)
ccby_Jaw
0.02
0.04
0.02
0.08
0.02
0.03
ecby sZore
0.05
0.10
0.05
styskccckeeee
0.05
0.07
0.04
0.16
0.05
keyysstckoverrflow
0.03
0.05
0.01
0.03
0.07
0.03
biikiness
0.07
0.15
0.02
0.08
0.03
0.09
echy wikipedia
0,05
0.11
0.02
0.06
0.03
0.08
pd_books
0.13
0.26
0.03
0.07
0.04
0.33
pd law
0,05
0.10
0.02
0.35
0.03
0.07
pd news
0.06
0.14
0.02
0.07
0.02
0.08
pd_s2ore
0.08
0.15
0.04
0.08
0.05
0.13
amps math
0.01
0.02
0.01
0.02
sw_dm_math
0.00
0.01
sw sw_github
0.04
0.04
0.03
0.03
0.24
0.04
ww_hackemews
0.09
0.18
0.03
0.06
0.06
0.10
ww..uuuuuuii irc
0.12
0.11
0.06
0.04
0.08
0.09
Dataset
OpenWebText22 PhilPapers
Wikipedia (en)
co-news
new-arazznn
HackerNews
caby_law
0.02
0.03
0.05
0.05
0.02
0.03
ecby s2ore
0.05
0.11
0.02
0.06
0.06
0.07
bla_sstacke
0,06
0.08
0.04
0.05
0.06
0.10
0.05
0.07
0.02
0.03
0.07
0.06
ecby wikinews
0.09
0.22
0.02
0.04
0.09
0.08
wikieeikia
0.06
0.17
0.02
0.07
0.07
0.06
pd.books
0.16
0.15
0.11
0.11
pd_law
0.02
0.06
pd news
0.08
0.06
0.09
pd.s22ore
0.08
0.13
0.03
0.09
0.08
0.09
sw sw_amps_mate
0.01
0.02
0.01
0.01
0.01
0.02
dm math
0,00
0.01
0.00
0.00
0.00
0.00
w_github
0,04
0.04
0.03
0.05
0.03
0.06
sw hackerews
0.14
0.20
0.04
0.06
0.16
0.19
sw. sw.ubuntu.in irc
0.13
0.10
0.10
0.10
0.08
0.18
60
Wikipedia
% ppl gap btwn PDSW and Pythia
50
OpenWeb Text
40
30
MIMIC III
20
10
0
HackerNews
-10
0.05
0.10
0.15
0.20
0.25
0.30
0.35
ngram overlap with PDSW training data
Figure 6: There is strong negative correlation between ngram overlap of domain with the PDSW training data and the perplexity gap between the PDSW LM and Pythia (r=-0.72, p < 0.005).
28


Reviews:
[{"review_id": "BcsZfliTtf", "reviewer": "Reviewer_cDVV", "summary": "The premise of this study is that some data that is legally challenging to be included during pretraining of a LLM. In this study the authors investigate whether such data can instead be used just as augmentation during inference while the LLM is only trained on uncritical data. The augmentation during inference is either achieved with the knn-LM approach or by simply adding BM25 retrieved text in the context. They investigate the effectiveness of this approach on a few zero-shot text classification benchmarks, and by measuring perplexity on text sources of various domains. They can show that a model only trained on legally safe data performs worse and adding the augmentation during inference can reduce the gap to a model that has seen such data during pretraining.", "questions": "- Even with using open licence data, isn't there is still a risk, that IP from others might be infringed?\n- It didn't become clear to me why the paper focuses on the different licences PD,SW and BY, as it doesn't have any significance in the experiments?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The premise hinges on the currently hotly debated issue of legal implications of using legally critical data for pretraining, and assumes that using such data in pretraining would be undesirable. They investigate the option to make it easy to remove critical data unambiguously from the inference of the model, which could enable using such data until it has to be removed.\n\nThe general quality and clarity of the paper is good, i.e. the presentation is well thought out and good to follow.", "weaknesses": "The general idea the paper presents is straight-forward, relevant and interesting, yet I am not convinced if it asks the right questions and measures the right things. \n\nWhen data is taken away during pretraining and is only added during inference, then the first question I have is on which tasks does this matter and has a larger impact and on which tasks does it not matter. The suite of text-classification benchmarks are too simplistic to give us any relevant insights into the effect this might have. \n\nThe perplexity analysis is not too insightful, as we won't know how that will influence any of the benchmarks that might be relevant to a particular use-case. An increase in perplexity is expected under domain-shift and also that adding the inference augmentation will reduce it, so its not clear what the take-away for the reader should be (its good to have that in the appendix, but we know that knn-LM and RIC do work).\n\nAs a strong paper, I would have liked to see the potential advantages to be contrasted more with the current challenges, i.e. the paper leaves the run-time and various adjustments to make the approach feasible in the appendix, and focuses the main paper on selling the approach as a potential solution, yet realistically its far away from that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4p3O05jZ4q", "reviewer": "Reviewer_X9LU", "summary": "This research delves into the potential of developing technologies tailored to manage legal risks associated with the controversial issue of training Language Models (LMs) on copyrighted data. The authors introduce SILO, developed by:\n\n1. Training a parametric LM on low-risk texts (e.g., copyright-expired books).\n2. Enhancing the LM with high-risk texts (e.g., copyrighted books) stored in a nonparametric datastore, accessed only during inference.\n\nAt its core, SILO leverages the kNN-LM retrieval method (k-nearest neighbors LM by Khandelwal et al., 2020).", "questions": "Please briefly respond to the two weaknesses I listed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The motivation behind this study is commendable, as it addresses the pressing concern over the legality of training materials for LMs.\n* SILO's methodology offers an effective solution to the identified problem, as demonstrated by both its theoretical framework and experimental outcomes. Its adaptability in integrating various corpora is a significant advantage.\n* Compared to alternative techniques, like post-hoc training data attribution or attempts to neutralize specific training samples' effects, this method provides robust assurances and scalability.", "weaknesses": "* Technological innovation is a slight drawback since the study primarily builds upon the kNN-LM framework by Khandelwal et al., 2020. However, this is somewhat mitigated by the strategic application of the technology to a pertinent issue, backed by thorough analysis and experimentation.\n\n* As per Table 16, the runtime speed remains a notable challenge for kNN-LM. Future work should explore innovative technological solutions to address this limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rb7DZBBhdR", "reviewer": "Reviewer_P6ek", "summary": "This paper verifies the feasibility of decomposing data into high-risk and low-risk categories to ensure the legality of LLMs. Low-risk data is employed during the pretraining phase, whereas high-risk data can be integrated during inference by utilizing in-context retrieval augmentation or kNN-LM.", "questions": "In the paper, the authors claim that \"Pythia ... is trained with a similar amount of compute (data size and model parameters).\" What are the exact computational costs when training SILO and Pythia?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper introduces a paradigm aimed at ensuring the legality of data used in LLMs, with the potential to significantly impact related research areas such as LLM privacy and security.\n* It conducts a comparative analysis of two approaches for incorporating high-risk data into LLMs during the inference process, offering valuable insights into their effectiveness.\n* The paper also introduces a dataset that can support further research in this line of investigation.", "weaknesses": "* This paper primarily focuses on evaluating the zero-shot performance of SILO. Given that pretrained LLMs usually serve as base models for instruction-tuning or task fine-tuning, understanding the extent of SILO's influence on these processes is of significant importance. However, this is not explored in this paper.\n* In the evaluation of SILO, this paper primarily relies on PPL as a metric, although it does carry out experiments on text classification tasks. A more comprehensive assessment across various settings, a broader range of tasks, and in terms of multiple aspects (such as helpfulness and harmfulness) could offer a deeper and more insightful understanding of SILO's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oAgsJ2LjSC", "reviewer": "Reviewer_mHsD", "summary": "The methodology presented in the paper attempts to address current concerns regarding language technology, namely its reliance on copyrighted data. The authors propose a new corpus, the 228B-token Open Licence Corpus (OLC) comprising texts with various permissive licenses. Additionally, they show that since OLC is less broad in content than its alternatives that do not eschew copyrighted text, training a language model solely on this results in comparably low performance.\n\nIn order to benefit from using copyrighted texts, the authors propose to use their language model trained on OLC in conjunction with a datastore containing copyrighted text. They test two distinct retrieval methods, k-nearest neighbours (KNN) and retrieval-in-context (RIC) for returning tokens (in the case of KNN) or text blocks (in the case of RIC) from the datastore. They find that by using KNN-based retrieval they can achieve performance close to Pythia, a language model trained on non-permissive data across 14 domains on perplexity and 10 downstream tasks.\n\nSILO models such as the one proposed by the authors are conducive to both identifying the pieces of copyrighted texts contributing to a given decision, as well as removing such texts, complying with concepts like fair use and GDPR.", "questions": "I have no questions, but please make sure that you improve clarity when it comes to presenting your results. Also, I think being a bit more explicit in how retrieval works at inference time in practice could improve the paper and match its otherwise accessible tone.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall: The paper is fairly clearly written, it addresses a very important and current problem in the application of language technology, and carries out extensive analyses. It also compares performance of the proposed method across a large number of domains and tasks.\n\nSoundness: There are still some costs associated with inference time and performance when using SILO models as opposed to using “classical” models trained on a range of permissive and non-permissive texts. However, the methodology proposed by the authors seems like a viable method of mitigating the very current problem of the use of copyrighted or otherwise protected corpora in language technology.\n\nContribution: The authors create OLC, a diverse pretraining corpus made up of permissive texts. Additionally, they train a language model on this corpus. They also present a large number of analyses untangling the impact of various factors relating to domain overlap and retrieval methods, among all.", "weaknesses": "Overall: Some of the tables are difficult to read, as mentioned, adding some highlights to help compare the performance would have been helpful. Additionally, in the case of Table 3, the percentages included are confusing. Are the minuses referring to the advantage (or lack thereof) when compared with Pythia? If yes, why do we get -100% when the result is the same across SILO and Pythia?\n\nPresentation: The content is mostly presented clearly. However, it would have helped to have some examples at least in the appendix or a more descriptive figure illustrating KNN and RIC-based retrieval to make it somewhat easier to understand. Otherwise, some of the denser tables would have benefitted from adding highlights (underline or bold text) to important scores to lead the eye.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)