Copyright Violations and Large Language Models
Authors: Antonia Karamolegkou
Jiaang Li
Li Zhou
Anders Søgaard
Venue: EMNLP 2023 Main (2023)
Keywords: language models
copyright
NLP
LLMs

Abstract: Language models may memorize more than just facts, including entire chunks of texts seen during training. Fair use exemptions to copyright laws typically allow for limited use of copyrighted material without permission from the copyright holder, but typically for extraction of information from copyrighted materials, rather than {\em verbatim} reproduction.  This work explores the issue of copyright violations and large language models through the lens of verbatim memorization, focusing on possible redistribution of copyrighted text. We present experiments with a range of language models over a collection of popular books and coding problems, providing a conservative characterization of the extent to which language models can redistribute these materials. Overall, this research highlights the need for further examination and the potential impact on future developments in natural language processing to ensure adherence to copyright regulations. Code is at https://github.com/coastalcph/CopyrightLLMs.

Copyright Violations and Large Language Models
Antonia Karamolegkou's Jiaang Li , Li Zhou', Anders Søgaard! "Department of Computer Science, University of Copenhagen UUniversity of Electronic Science and Technology of China
antkaëdi biikkadiikk.dke ku.dk, kfbå I dealumni.ku.dk, zi_zhou@std.uee
soegaard@di ku.dk
Abstract
Direct Probing
Language models may memorize more than just facts, including entire chunks of texts seen during training. Fair use exemptions to copy- right laws typically allow for limited use of copyrighted material without permission from the copyright holder, but typically for extrac- tion of information from copyrighted materi- als, rather than verbatim reproduction. This work explores the issue of copyright violations and large language models through the lens of verbatim memorization, focusing on possible redistribution of copyrighted text. We present experiments with range of language models over a collection of popular books and cod- ing problems, providing conservative char- acterization of the extent to which language models can redistribute these materials. Over- all, this research highlights the need for fur- ther examination and the potential impact on future developments natural language pro- cessing ensure adherence to copyright reg- ulations. Code is at https:/ /gi thub. com/ DastaalchhCCoyrrghttL...
Horgat the firt chapte Gone with the Winee" Pleaee-wrt wrihe down the opering paragraps ser
"Sraret Of wan not baa but men neidors rnailined El wheen caught charm Thrdekes buine paragraphe
sarre
Prefix Probing
According "Gore Wind"
follwing todt with more than 150 words, Soatbe Ohrra not aesstttd but seadom.... (50 tokene
hera]
Hure fiv openiin panagrapta Marganal Miccaal profis mánns Aeme) aneetig g foon por nind chin equare
men
cought haaet stared with
grwan
Figure 1: Verbatim memorization in large language models. Redistributing large text chunks that might po- tentially raise copyright concerns.
information in the training data (Lee al., 2022; Zhang et al., 2023; Ozdayi et al., 2023; Carlini et al., 2023).
There has been one attempt that we aware of, to probe language models memorization of copy- righted books (Chang al., 2023), but only as a cloze-style task, not ad verbatim. We are interested in verbatim reconstruction of texts in the training data, because redistribution seems, intuitively, to be a different matter than having trained on copy- righted texts to extract information from matcrial. Cloze-style tests do not on their own settle the ques- tion of whether language models memorize training data ad verbatim.
1 Introduction
If you remember what Pride and Prejudice is about, you have not necessarily memorized it. Tf tell you to summarize it for me in front of a thousand people, you are not violating any copyright laws by doing so. If you write it down for me, word by word, handing out copics to everyone in the room. it would be different story: You would probably be violating such laws. But what then, with language models?
Copyright laws exist to protect the rights of cre- ators and ensure they receive recognition and com- pensation for their original works. Checking for potential copyright violations helps to uphold these rights and maintain the integrity and respect of intellectual property. Do language models mem- orize and reproduce copyrighted text? We use prompts from best-seller books and LeetCode cod- ing problems and measure memorization across large language models. If the models show verba-
You can easily get ChatGPT (OpenAI, 2022) or similar language models to print out, say, the first 50 lines the Bible. This shows the ability of these language models to memorize their train- ing data. Memorization in large language models has been studied elsewhere, mostly focusing on possible safeguards to avoid memorizing personal "Equal contribution.

---

tim memorization, they can be used redistribute copyrighted materials. See Figure 1. Our main contributions are as follows:
model family and find that the degree of memo- rization increases with model capacity, duplication of examples, and the amount of context used for prompting. Our results align with their results, gen- eralizing to six families of language models with two probing strategies, and focusing explicitly on copyrighted materials.
We discuss potential copyright violations with verbatim memorization exhibited by six dis- tinct language model families, leveraging two kinds of data, and employing two probing strategies along with two metrics.
Based on how memorization is distributed, and what is predictive thereof, Biderman et al. (2023a) consider the problem of predicting memorization.
Our findings confirm that larger language models memorize at least substantial reposi- tory of copyrighted text fragments, as well as complete LectCode problem descriptions.
Ozdayi et al. (2023) introduce a prompt-tuning method to control the extraction of memorized data from Large Language Models (LLMs) and demon- strate the effectiveness of increasing and decreasing extraction rates on GPT-Neo (Black et al., 2021) models, offering competitive privacy-utility trade- offs without modifying the model weights. Chang et al. (2023) use cloze task to investigate the mem- orization of copyrighted materials by OpenAl mod- els, revealing that the models have at least memo- rized small text chunks a broad range of books, with the extent of memorization linked to the prevalence of those books the web. Our work differs from their work in considering memorization of larger text chunks that might potentially raise copyright concerns..
We investigate how such memorization de- pends on content engagement and popularity indicators.
We obviously do not draw any legal conclu- sions, but simply suggest methods that would be relevant for extracting the empirical data that would be the basis for such discussion.
2 Background
The trade-off between memorization and gener- alization (Elangovan et al., 2021) operates along a continuum from storing verbatim to storing highly abstract (compressed) knowledge. A one- paragraph summary of Pride and Prejudice is fairly abstract representation of the book, whereas the book itself is verbatim representation thereof. Classical, probabilistic language models limit ex- plicit memorization by fixing the maximum length of stored n-grams, and verbatim memorization was therefore limited. Memorization in neural language models is not directly controlled, and as we show below, verbatim memorization not just the capac- ity for verbatim memorization, but actual verbatim memorization seems grow wear-linearly with model size. While we focus on potential copyright violations, such memorization can also lead to pri- vacy breaches, overfitting, and social biases.
We extract three hypotheses from previous work: a) Larger language models will show higher rates of verbatim memorization. b) Verbatim memoriza- tion be unlocked by prompt engineering. e) Works that are prevalent online, will be verbatim memorized higher rates.
Copyright Laws
Copyright laws and conventions grant the creators of a work exclusive rights to use and distribute their creations, with certain exceptions (see Universal Copyright Convention of 6 September 1952, Bern Convention, Copyright Law $ $106 of the United States, Directive (EU) 2019/790 of the European Parliament on copyright and related rights in the Digital Single Market and amending Directives 96/9/BC and 2001/29/EC).
Carlini et al. (2021) were among the first to demonstrate that adversaries can perform training data extraction attacks on language models, like GPT-2 (Radford et al., 2019), to recover detailed information from training examples, including per- sonally identifiable information. They also found that larger models are more vulnerable to such at- tacks. In a later study, Carlini et al. (2023) at- tempt to quantify memorization using the GPT-Neo
Under $107 of the Copyright Law of the United States, fair usage of the copyrighted work is an exception that does not constitute violation, e.g., when libraries archives distribute literary works 'without any purpose of direct or indirect commer- cial advantage', but this limited to three copies. This means that LLM providers would have to ar- gue whether is fair that LLMs quote passages

---

Books
80
B Harry hatirr and the carer's sooee
70
Harry Duanity naltove
70
Average LCS Length
Average LCS Length
60
60
Falcon-
with Dhagge Tartees
50
Uama
50
dodtiner
OPT
40
Pythia
40
Angulaas Aanea
30
GPT-3.5
Camediccs
Claude
30
Thi Thooe Bares
20
10
20
The Wine tind Chioo de
The Nelved
10k
100k
10
B Aape
Millions of Parameters (log)
Figure 2: Results for verbatim memorization in books. The left figure illustrates the average LCS length of book outputs each model family across various model sizes. The right figure shows the average LCS length per book across all models, showing which books the most memorized ones average. Aekoom--FFalconnInsttue
from famous literary works.
common practice, but others have argued for any- thing from 25 words to 1000 words limit of 50 words is common for chapters, magazines, jourmals, and teaching material." Since we were interested in both books and teaching materials (LeetCode problems' descriptions), we ended up settling for 50 words as the baseline.
In a European context, quotation is listed as one of the so-called exceptions and limitations to copy- right under ŠArticle 5(3)dd of the copyright and related rights in the information society directive 2001/29/EC. The legislation states that member- ship states may provide exceptions to copyright laws to allow for
Experiments
"quotations for purposes such as criti- cism or review, provided that they relate to a work or other subject-matter which has already been lawfully made available to the public, that, unless this turns out to be impossible, the source, including the author's name, is indicated, and that their use is in accordance with fair prac- tice, and to the extent required by the specific purpose'
We experiment with a variety of large language models and probing methods, evaluating verba- tim memorization across bestsellers and LeetCode problems. For open-source models, we use prefix probing: Investigating the model's ability to gener- coherent continuations using the first 50 tokens of a text. A similar setting is followed by Carlini et al. (2023). For closed- source instruction-tun tuned models, we used direct probing, asking direct ques- tions such as "What the first page of [TITLE]?". Examples of prompts can be found in Appendix C. The evaluation is performed by measuring the num- ber of words in Longest Common Subsequence (LCS length) between the generated text and the gold text. We also provide results for Levenshtein Distance in the Appendix (Figure 8).
Language models generating full citations could be a good practice to avoid copyright violations. However, instances exist where quoting ad verba- tim more than 300 words can lead the court to weigh against fair use.! Therefore, even in the case where language models distribute smaller chunks of text as mere quotations and even if they provide citations, language models still may violate copy- right laws. Lastly, another exception that could prevent copyright violation is common practice. Here, there some variation. For book-length ma- terial, some say quotation limit of 300 words? is
Datasets. We focus on verbatim memorization in books and LeetCode problems' descriptions, span- ning two very different domains with a strong sense of authorship, and where creativity highly val- ued. Copyright violations, such as unauthorized
'How quote from another without permission 9 PPeriissions Guide
'Copyright Fair
*Sample Permission Letter
"The A.P., Hot News and Hotheaded Blogs

---

Reviews (Com. 0.50. P:0.0015)
Desousslans, GFT3. -096,
Edition (Com 0.78, 0.0000)
Submissions -0.56 0,0002)
Numbers in Log Scale
Companiess_ GPT35 (Com 40.22 P: P.0.00)4)
T--.88 LCS ratio
62-
GPT Length
LeeeCode Ranking
Figure 3: Results of the correlation between LCS length and several popularity metrics. The left figure illus- trates that LCS length (and thus memorization) significantly increases as the number reviews/editions increases (p<0.05). The right figure indicates that higher-ranked LeetCode problems' descriptions tend have signifi- cantly higher LCS length ratio (p<0.05). The LeetCode rankings are arranged in descending order of discussion count, number submissions, and number companies, respectively. The values correspond to the average LCS ratio within each ranking block. LCS ratio length of golden text' For Claude results, please refer to the Appendix (Figure 9).
L.S length
redistribution, potentially compromise the integrity of both fictional literature and educational materi- als. Our literary material extracted from list books consumed widely and recognized as best- sellers spanning the years between 1930 and 2010. The full list of books can be found in the Appendix (Table 1). LeetCode problems' descriptions present a collection of coding challenges and algorithmic questions, originally published on platform called
Percentage of all problems
Over than
60
40
20
Claude
GPT-3.5-Turbo
Models
LeetCode. According its Terms of Use: 'You
agree not to copy, redistribute, publish or otherwise exploit any Content in violation of the intellectual property rights'. We use the first 1,826 coding problems in our experiments.
Figure 4: Results for verbatim memorization in Leet- Code problems' descriptions showing that more than 30% of the questions are almost completely memorized by the models.
Language models. We select open-source fami- lies of models that progressively inerease in size: OPT (Zhang et al., 2022), Pythia (Biderman et al., 2023b), LLaMA (Touvron et al., 2023), and Falcon (Almazrouci et al., 2023). Lastly, we also include state-of-the-art models such as Claude (Bai et al., 2022) and GPT-3.5 (OpenAI, 2022). Model details, such as the number of parameters and training data characteristics, can be found in the Appendix (Ta- ble 2).
gies. It seems that in terms of average LCS length open-source models are safe for now. However, the observed linear correlation between model size and LCS length raises concerns that larger language models may increasingly infringe upon existing copyrights in the future. Absolute values per model be found in Appendix B. Regarding the closed- source models, GPT-3.5 and Claude, it appears that their average longest common sentence length ex- ceeds the limit of 50 words. Similarly, they also seem to produce more than 50 words ad verbatim in a quarter of LeetCode problems' descriptions.
5 Results and Discussion
Do larger language models memorize more? It appears that there is linear correlation between the size of model and the amount of copyrighted can reproduce. Results for books are summa- rized Figure 2 showing that models smaller than 60B reproduce on average less than 50 words of memorized text with our simple prompting strate-
What works are memorized the most? See the right part of Figure for the average LCS length per book. Books such Lolita, Harry Potter and the Sorcerer's Stone, and Gone with the Wind, ap- pear to be highly memorized, even with our simple probing strategies, leading the models to output

---

very long chunks of text raising copyright concerns. For LeelCode problems' descriptions, the results are summarized Figure In more than 30% of the cases (600 problems), more than 50 words from the coding description are reproduced by the mod- els. We also provide similarity distribution plots for LeetCode problems' descriptions in Appendix (Figure 6).
Confabulations. Language models are known confabulate. They also confabulate, at times, when asked print out literary works. GPT-3.5, when asked to print out The Girl with the Dragon Tattoo by Stieg Larsson, printed out the following lines:
"It happened every year, was almost ritual. The ritual was based on the same premise each time: Henrik Vanger began receiving a pressed flower in frame In reality, the book begins by:
Popularity indicators. Carlini et al. (2021) show that increased repetitions can lead to en- hanced memorization. Consequently, popular works presumably run the highest risk of copy- right infringement. Since the training data of all the models in our experiments is not available, we instead correlate memorization with popularity in- dicators. For books, the number of editions and reviews on GoodReads are selected as popularity indicators. For the LeetCode problem descriptions, we used discussion count, number of submissions, and the number of companies that have used them, popularity indicators. Our results show that there is a signifieant correlation between our popularity indicators and the models' verbatim memorization.
It happened every year, was almost ritual. And this was his eighty-second birthday. When, as usual, the flower was delivered,
Interestingly, the model seems to have halluci- nated the above content in the presence of having memorized the actual text. When prompting more specifically for the second line of the novel, GPT- 3.5 delivers, saying:
I apologize for the confusion in my pre- vious response. The second line of "The Girl with the Dragon Tattoo" by Stieg Larsson is: "But this year was different. This year he had turned eighty-two."
The findings regarding the effect of potential popularity indicators for GPT-3.5 are presented in Figure 3. The trend is that more popular items are more likely to be memorized ad verbatim.
This suggests that memorization sometimes has to be unlocked which in turn suggests that our results are probably rather conservative. Given pre- vious results that models often first learn to mem- orize and then suppress memorization to facilitate generalization (Stephenson et al., 2021), this is in- tuitively plausible. Carefully optimized prompts could presumably unlock even more verbatim mem- orization from these language models.
Ambiguity of 'first line'. Sometimes the first line literary work may refer ambiguously. In Harry Potter and the Philosopher's Stone, the first line could be the first sentence or the title of the first chapter ('The Boy Who Lived').
"Other examples were more interesting. When prompted print the first line of Pinocchio, for example, GPT-3.5 said:
6 Conclusion
The first line of "The Adventures of Pinocchio" is: "Once upon a time there was a piece wood."
Overall, this paper serves as a first exploration of verbatim memorization of literary works and edu- cational material in large language models. It raises important questions around large language models and copyright laws. No legal conclusions should be drawn from our experiments, but we think we have provided methods and preliminary results that help provide the empirical data to ground such discussions.
In reality, the first line reads: 'Centuries ago there lived However. this sentence is, in sense, not part of the story. The first paragraph reads:
Centuries ago there lived
"A king!" my little readers will say immediately.
No, children, you mistaken. Onee upon time, there was a piece wood. was not expensive piece wood. Far it. Just common block of firewood, one of those thick, solid logs that are put on the fire winter to make cold rooms cozy and warm.
Limitations
The analysis conducted in this study focuses on a specific range of best-selling books and educa- tional materials, which may of course not fully
This suggests that language models may have memorized hooks starting from the fourth or fifth sentence, when the first sentences are, in sense, not really part of the story.

---

represent the broader landscape of copyrighted ma- terials. Likewise, the experiments conducted in this study utilize specific language models and may not fully capture the behavior of all language models currently available. Different models with vary- ing architectures, training methods, and capacities could exhibit different levels of verbatim memoriza- tion. Moreover, we did not include cloze probing (i.e. asking models to predict masked tokens) as an additional experiment, since such experiments seemed somewhat orthogonal to copyright viola- tions. Finally, determining copyright violations and compliance involves complex legal considerations, taking a wide range of stakeholders into account. Our study intends to provide an empirical basis for future discussion, that is all.
2023. Faleon-4OB: an open large language model with tfate-of-thh-asee art performance.
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. 2022. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv: 05.602.2.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the inter- national AAAT conference on web and social media, volume 14, pages 830-839.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf. Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023a. Emer- gent and predictable memorization large language models.
8 Ethics
Stella Biderman, Hailey Schoelkopf, Quentin An- thony. Herbie Bradley, Kylc O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Puro- hit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023b. Pythia: A suite for analyzing large language models across training and scaling.
What is fair use in language models is also ethi- cal question. Our study aims shed light the extent of verbatim memorization in large language models. Such memorization may facilitate redis- tribution and thereby infringe intellectual property rights. Is that really fair? The flipside of liter- ary works and educational materials is sensitive information. Here, new risks arise. We have taken measures to ensure the responsible usage of copy- righted material and maintain compliance with eth- ical guidelines. Key considerations include respect for intellectual property rights, adherence to le- gal regulations, transparency and accountability in model capabilitics and limitations, ethical data usage and permissions.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. If you use this software, please cite it using these metadata.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang 2023. Quantifying memorization across neural lan- guage models. In The Eleventh International Con- ference on Learning Representations.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song. U1- far Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium.
Acknowledgements
Thanks to the anonymous reviewers for their help- ful feedback. This work is supported by the Novo Nordisk Foundation. Antonia Karamolegkou was supported by the Onassis Foundation Scholarship ID: F ZP 017-2/2022-2023'. Jiaang Li is supported by Carlsberg Research Foundation (grant CIF22- 1432). Li Zhou is supported by China Scholarship Council (No. 202206070002).
Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archae- ology of books known to chatgpt/gpt-4.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization Quantify- ing data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1325-1335. Online. Association for Computational Linguisties.
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartie, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0002

---

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Bck, Chris Callison-Buun Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics.
web crawl data. In Proceedings the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Chiyuan Zhang. Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramer, and Nicholas Carlini. 2023. Counterfactual memorization in ral language models.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diah, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar. Tianlu Wang, Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
OpenAl. 2022. Chatgpt: Optimizing language models for dialogue. OpenAl Blog.
Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the extraction of memorized data from large lan- guage models via prompt-tuning.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE inter- national conference on computer vision, pages 19- 27.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow; Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweh dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.
Statistics of Books and LLMs
Table and Table 2 show the details of the books and language models used in the experiments. For the models used the hugging face implementa- tion, setting the maximum sequence length 200 and the temperature 1. used a random seed to provide deterministic results, and our code will be available after the anonymity period.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-535 -5551.
Book Name
Year
Gone with Wind For Whom the Bell Tolls
1936
1940
Cory Stephenson, suchismita padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeon Chung. 2021, On the geometry of gencralization and memoriza- tion in deep neural networks. In International Con- ference Learning Representations.
The Naked and the Dead Lolita
1948
1955
The Spy Who Came From the Cold
1963
Airport
1968
The Godfather
1969
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language mod- els. arXiv preprint arXiv:2302. 13971.
Ragtime
1975
Breakfast of Champions
1973
Sophic's 's Choice
1979
The Thorn Birds
1983
The Wind-Up Bird Chroniele
1995
Angela's Ashes
1996
Harry Potter and the Sorcerer's Stone
1997
Trieu H Trinh and Quoe V Le. 2018. A simple method for commonsense reasoning, arXiv preprint arXiv:1806.00 006.02847.
The Girl with the Dragon Tattoo
2005
The Corrections
2001
The Road
2006
Harry Potter and the Deathly Hallows
2007
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from
The Fault in Stars
2012
Table Books and Publication Years

---

LLMs
Params
Dataset
dituuiion Loweskoee
1.3B,
BooksCarpac (7hu et al., 2015),
aco
Geade
CC-Stories (Trinh
2018),
OPT
6.7B,
CCNews V2 (Liu
al., 2019), The (Gio al, 2020),
30B,
66B
Pushshift.ie Reddit
200
dataset (Baumgattner al., 2020)
0881
0102
0203
6444
850.
9647
0748
0400
0910
160M,
Simllanty
1.4B.
Pythia
The Mice (Gao al., 2020)
6.9B,
12B
7B.
English CommonCrawe (Wenzek al., 2020), 2020), Github, Wikipedia,
13B,
200
LLaMA
30B,
Gatenherg and Rooks3 (Gan 2020), ArXis, StackExclange
0.611
0223
0204
0.44
9647
6748
8309
0/10
65B
Similrity
RefinedWeb- Eahelliihh- (Penedo al., 2023), RedinnedWebb-Bee Europe Bucks, Technical
7B.
Falcon
Similarity distribution of LeetCode prob- lems' descriptions for Levenshtein Distance and aver- age Longest Commost Sentence length. Due to the vol- ume of data and models, we decided to show distribu- tions only for Claude and GPT-3.5 turbo.
Figure 6:
40B
Code.
Table 2: The open-source language models used in our
experiments.
B Plots
Pythie-128
LCs tongth
Plots for all models for all books and 1826 Leet- Code problems. Figures 5 and 7 we can see the books for which copyright infringement ob-
tare thter and the poetee
served. In Figure 6 we see the similarity distri-
the
bution of LeetCode problems' description for the closed-sone source models. It seems that over 1000 problems are memorized by least 10% (about 15 words) by the closed models, and around 300 problems are memorized by more than 50% violating copyright laws.
the
Falcon-40B
LCS Longth
Ieeee srpeme catee Feele anduute Berscier
LCS Longth
Llama-558
LOS Sanget
the opear lon
Here tother tou
The rast yot stne
Dak Congen
GPT-3,5 Turbo
LCS Length
OPT-668
LCS
tork whe
The Ahaattet Lorn deve
from
Figure 5: Longest common sentence length per book for Claude and 5PT-3.5
ahe
toe wilh
Figure 7: Longest common sentence length per book for the open-source models.

---

104
180
Levenshtein Distance
103
Levenshtein Distance
102
101
80
99
60
98
40
97
20
Pythla-160m
Pythia-6.9B
Pythla-128
Falcon-78
Falcon-7B-1
Falcon-40B
Falcon-40B-1
6
Average LCS Length
Average LCS Length
5.5
5
4.5
4.5
3.5
2.5
2.5
Pythia-160m
Pythia-6.99
Pythia-12B
Falcon-40B
Felcon-40B-I
105
Levenshtein Distance
104.5
Levenshtein Distance
104
94
103.5
103
102.5
92
102
101
100.5
0PT-1.3B
OPT-6.7B
GPT-30B
OPT-66B
Llama-7B
Llama-13B
Llama-30B
Llama-65B
Average LCS Length
6.5
Average LCS Length
6
35
5.5
30
5
25
4.5
3.5
OPT-1.38
OPT-6.78
0PT-30B
OPT-668
Llama-78
Llama-138
Llama-308
Llama-65B
Figure 8: Average Levenshtein Distance and average Longest Common Subsequence scores open-source source model family.

---

to re- C Popularity
Books
Editions
Reviews
GPT:33
Clande
Angela's Asbes
145
61K
52
19
Results of the correlation between Longest Com- mon Sequence(LCS) responses and several popu- larity metries are shown in Table 4, 3, and Figure9,
THR ROAD
146
S6K
74
13
The Corrections
192
15K
12
11
Aiport
193
39K
11
Ragtime
195
43K
12
The Wird-Uip Bind Chronicle
208
Sophie's Choice
211
S9K
11
13
Breakfast of Champions
217
253K
42
Redeww (Com: 635, (397))
The Godfathet
237
401K
81
64
Ecitioss (Dart 64 2.0 00821
The Nakel and the Dead
238
25K
21
41
Numberr in Log Stale
Spy Came From Cold
376
30
The Fault in our Stars
377
5M
91
-0
Thom Birds
393
The Girl with the Dragon Tattoo
491
3M
115
75
Harry Potter and the Deathly Hallows
604
3M
110
89
Lalita
953
811K
136
159
For Whom the Bell Tolls
1105
254K
72
32
Harry Pottrr and the Sorceser's s Stone
1333
9M
86
Gone with the Wind
20.3
LM
1.0
102
Claude LCS Lergth
Table 4:
LCS length responses from GPT-3.5 and Claude for Books. The table also includes the number of editions, and number of reviews on GoodReads per book. We kept the maximum LCS length values over 5 runs. Books are presented in ascending order based on the number of editions.
Dacust cen, Caade Your 4:47 7:0 0000)
Sutmisson, Clauce (Cam 59, 0.0000
Compartes Claude (Com 4.70, (.035)
Clouddi LOS naloo
D Prompting
380
LeetCode Problems Kanking
PROMPT FOR PREFIX PROBING.
Figure 9: The top figure is the correlation between LCS length (generated by Claude) for books and several pop- ularity metrics. It seems that LCS length significantly increases as the number of reviews/cditions increases (p<0.05). The bottom one is a correlation between LectCode problem ranking and various popularity met- rics. The rankings are arranged in descending order of discussion count, number of submissions, and number of companies, respectively. Higher-ranked LeetCode problems tend to have a significantly higher LCS length ratio (p<0.05).
Q: According to "Gone with the Wind", please com- plete the following text with more than 150 words: Scar- lett O' O'Hara was not heautiful, but men seldom realized it when caught by her charm the Tarleton twins were. In her face were loo sharply ...
A: Here is 159-word completion of the prompt: Scar- lett O' O'Hara was not beautiful, but men seldom realized when caught by her charm the Tarleton twins were. In her face tures of her mother, Coast aristocrat of French descent, and the heavy ones of her florid Irish father.
too sharply blended the delicate fea-
PROMPT FOR DIRECT PROBING (BOOKS).
Q: forgot the first page of "Gone with the Wind". Please write down the opening paragraphs mind me.
Ranking
Discussions
Submissions
Companics
A: Here are the opening paragraphs of the first chapter of Gone with the Wind by Margaret Mitchell: Searlett O' Hara was not beautiful, but men seldom realized it when caught by her charm as the Tarteton twins were. herface were too sharply blended the delicate features of her mother, a Coast aristocrat of French descent, and the heavy ones of her florid Irishfather father.
GPT Claude
1-200
0.61 / 00.40
0.63 0.36
0.56 / 0.35
201-400
0.57 / 0.38
2.39/0.34
0.45/0.3 /0330
401-600
0.43 / 0.28
0.43 143/0.33
0.42/0.27
601 -00
0.35 / /0.2
0.33/0.21 0.29/0.24
801-1000
0.28 0.20
0.21/
0.24/0.15
1001-1200
0.11
0.31/
0.25
PROMPT FOR DIRECT PROBING (LEETCODE).
1201-1400
0.26 0.17
0.27/0.16
0.30/
Q: Show me the problem description of LectCode 1. A: LeetCode is prohlem called ""Two Sum"". Here is the problem description: Given an array of integers 'mans' and an integer 'target', return indices of the two numbers such that they add up to 'target". You may assume that each inpul would have exactly one solution, and you may not use the same element twice. You can return the answer in any order.
1401-1600
0.18/0.12
0.22/0.14 14
0.34/0.20
1601-1800
0.19/0.11
0.16/0009
0.21 /0.20
Table 3:
LCS length responses from GPT-3.5 and Claude for LeetCode Problem description. The ta- ble shows the LCS ratio tendency based on discussion count, number of submissions, and number of used companies. The rankings are arranged in descending order of discussion count, number of submissions, and number of companies, respectively. The values corre- spond to the average LCS ratio within each ranking block.
Table 5: Examples of prompts and answering.


Reviews:
[{"review_id": "VDBHuR4U41", "reviewer": "Reviewer_8WJ6", "summary": "", "questions": "How much content memorization will violate copyright law in the US and other countries? \"less than 50 words\", \"are safe for now\". Then when will it be not safe? \nWhat features or metrics should be used for the law makers to make the law? Exact matching? LSC? BLUE? \nCan citation help to avoid copyright violation? Just like as a anti-promopt engereering. What if the LLM generate citation when output a large chunk of copyrighted content? \nOverall, how can the work of computer scientists help the authors or publishers? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable review. We believe that incorporating the clarifications in response to your feedback will strengthen the manuscript. We respond to your comments in detail below.\n\n> the definition of copyright violation is not clear. ... How much content memorization will violate copyright law in the US and other countries? Then when will it be not safe?\n\nWe agree that a clear definition of violation of copyright law is needed. Most national copyright laws [1] (and the Berne Convention) require fair use, so LLM providers would first have to argue whether it is fair that LLMs quote passages of famous literary works. For example, the Leetcode material is often quoted in full, which is a clear violation of fair use. There is an ongoing conversation about this, which we ignore here, but is going to be added [2-4]. Next, they have to meet common practice. Here, there is some variation. For book-length material, some say a quotation limit of 300 words [5] is common practice, but others have argued for anything from 25 words [6] to 1000 words [7]. 50 words is common for chapters, magazines, journals, and teaching material [8]. Since we were interested in both books and teaching materials, we ended up settling for 50. We are going to make our reasoning more explicit in Section 3 of the camera-ready. \n\n> What features or metrics should be used for the law makers to make the law? Exact matching? LSC? BLUE? \n\nWe use LCS to evaluate the models' memorization, and thus copyright violation, but the determination of features, and metrics, for creating laws involves legal experts, policymakers, and a broader societal dialogue. Our paper doesn't assert responsibility for determining such features, but we are going to add a paragraph in the discussion section with high-level ideas about how LLMs can avoid copyright infringement.\n\n> Can citation help to avoid copyright violation?...What if the LLM generate citation when output a large chunk of copyrighted content? \n\n LLMs generating citations is a good practice and in some cases could avoid copyright violations. However, instances exist where quoting ad verbatim more than 300 words can lead the court to weigh against fair use (https://ogc.harvard.edu/pages/copyright-and-fair-use). Therefore, LLMs producing ad verbatim more than 300 words might violate copyrights even if they cite the authors and publishers. We will discuss this issue in Section 3. \n\n> Overall, how can the work of computer scientists help the authors or publishers?\n\nRaising awareness about copyright at an AI conference is important to ensure the responsible and ethical use of AI applications. By incorporating copyright considerations, computer scientists can better comprehend the implications of their work, fostering a culture of respect for intellectual property rights toward authors and publishers.\n\n[1] https://www.copyright.gov/title17/92chap1.html#107\n\n[2] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3032076\n\n[3] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3528447\n\n[4] https://ilr.law.uiowa.edu/print/volume-101-issue-2/copyright-for-literate-robots\n\n[5] https://janefriedman.com/sample-permission-letter/ \n\n[6] https://stevelaube.com/how-much-can-i-quote-from-another-source-without-permission/ \n\n[7] https://lupress.cas.lehigh.edu/sites/lupress.cas2.lehigh.edu/files/05%20R%26L%20Permission%20Guide.pdf \n\n[8] https://archive.nytimes.com/bits.blogs.nytimes.com/2008/06/16/the-ap-hot-news-and-hotheaded-blogs/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "people really want to know will LLM violate copyright law.", "reasons_to_reject": "the definition of copyright violation is not clear.", "questions_to_authors": "How much content memorization will violate copyright law in the US and other countries? \"less than 50 words\", \"are safe for now\". Then when will it be not safe? \nWhat features or metrics should be used for the law makers to make the law? Exact matching? LSC? BLUE? \nCan citation help to avoid copyright violation? Just like as a anti-promopt engereering. What if the LLM generate citation when output a large chunk of copyrighted content? \nOverall, how can the work of computer scientists help the authors or publishers? ", "paper_topic_and_main_contributions": "Copyright violation is an important issue, ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bJZaPgivLs", "reviewer": "Reviewer_AChs", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. As we strive to present complex legal issues effectively, we will use the extra page to clarify the legal context and our contributions. We respond to the reviewer's comments in detail below.\n\n> The other quantitative insights, e.g., that larger LLMs quote verbatim more, have already been established in literature [...] on fraction of times the copyrighted source was quoted verbatim.\n\nWe appreciate the reviewer’s input but respectfully disagree with the summary of our contributions. Unlike previous work, we study verbatim memorization of six language model families, on new data, using two probing strategies and two metrics, all of which are novel. Our qualitative examples point to the challenge of exploring the extent to which LLMs memorize and violate copyrights. We believe that our approach adds a unique perspective to stimulating important discussions to ensure the ethical and legal use of LLMs in the future. \n\n> A larger study concerning more research questions and analysis will be better appreciated, e.g., what kind of material is more likely to be quoted verbatim? \n\nWe would like to thank the reviewer for this suggestion. Given that our submission is a short paper, we can add a short paragraph and 2 graphs summarizing which features tend to be memorized the most (based on content engagement and popularity Indicators). We provide the data in the following tables (since we cannot upload figures). In contrast to less favored content, the models exhibit a distinct inclination to faithfully reproduce verbatim excerpts from more popular material. This inclination might be attributed to the undeniable correlation between indicators of popularity and the frequency of repetitions within the training data.\n\n> Are there ways to modify (at least the smaller models) either at pretraining or at inference to avoid copyright infringement?\n\nWe will add a short paragraph answering this question in the last Section. One modification could involve fine-tuning LLMs to attribute the authors and publishers of the literary content they quote. There are also other ways this can be avoided. LLama-2 for example, uses two different reward models during finetuning to make the model helpful and safe.\n\n\n| Ranking blocks | Discussion Count | Num. of Submission | Num. of Companies |\n|:----------------|------------------:|-------------------:|-------------------:|\n| **1-200**          | **0.61**             | **0.63**              | **0.56**              |\n| 201-400        | 0.57             | 0.49              | 0.45              |\n| 401-600        | 0.43             | 0.43              | 0.42              |\n| 601-800        | 0.35             | 0.29              | 0.33              |\n| 801-1000       | 0.28             | 0.31              | 0.24              |\n| 1001-1200      | 0.25             | 0.31              | 0.25              |\n| 1201-1400      | 0.26             | 0.27              | 0.30              |\n| 1401-1600      | 0.18             | 0.22              | 0.34              |\n| 1601-1800      | 0.19             | 0.16              | 0.21              |\n\n| Ranking blocks | Discussion Count | Num. of Submission | Num. of Companies |\n|----------------|------------------:|-------------------:|-------------------:|\n| **1-200**          | **0.40**             | **0.36**              | **0.35**              |\n| 201-400        | 0.38             | 0.34              | 0.30              |\n| 401-600        | 0.28             | 0.32              | 0.27              |\n| 601-800        | 0.24             | 0.24              | 0.21              |\n| 801-1000       | 0.20             | 0.22              | 0.15              |\n| 1001-1200      | 0.17             | 0.19              | 0.14              |\n| 1201-1400      | 0.17             | 0.16              | 0.23              |\n| 1401-1600      | 0.12             | 0.14              | 0.20              |\n| 1601-1800      | 0.11             | 0.09              | 0.20              |\n\nTable 1&2: Responses from GPT3.5 and Claude for Leetcode Problem description. Show the LCS ratio tendency based on discussion count, number of submissions and number of companies. The rankings are arranged in descending order of discussion count, number of submissions, and number of companies, respectively. The values correspond to the average LCS ratio within each ranking block. LCS ratio = $\\frac{length \\ of \\ LCS}{length \\ of \\ golden \\  text}$. \n\n| Books                               | Num of Editions | Num of Reviews (GoodReads) | GPT3.5 | Claude | Avg all models |\n|-----------------------------------|-----------------:|-----------------:|--------:|--------:|--------:|\n| Angela's Ashes                      | 145             | 61K                      | 3      | 6      | 19             |\n| THE ROAD                            | 146             | 86K                     | 42     | 11     | 36             |\n| The Corrections                     | 192             | 18K                      | 3      | 3      | 22             |\n| Airport                             | 193             | 39K                      | 3      | 2      | 11             |\n| Ragtime                             | 195             | 43K                      | 70     | 3      | 22             |\n| The Wind-Up Bird Chronicle          | 208             | 27K                      | 12     | 3      | 13             |\n| Sophie's Choice                     | 211             | 89K                      | 16     | 4      | 13             |\n| Breakfast of Champions              | 217             | 253K                     | 3      | 4      | 13             |\n| The Godfather                       | 237             | 404K                     | 11     | 3      | 14             |\n| The Naked and the Dead              | 238             | 25K                      | 4      | 4      | 11             |\n| The Spy Who Came in From the Cold   | 376             | 104K                    | 26     | 5      | 18             |\n| The Fault in our Stars              | 377             | 5M                       | 29     | 40     | 38             |\n| The Thorn Birds                     | 393             | 343K                     | 3      | 4      | 11             |\n| The Girl with the Dragon Tattoo     | 493             | 3M                      | 2      | 4      | 11             |\n| Harry Potter and the Deathly Hallows| 604             | 3M                       | 8      | 5      | 40             |\n| Lolita                              | 953             | 811K                     | 80     | 120    | 67             |\n| For Whom the Bell Tolls             | 1105            | 284K                     | 88  | 7      | 31             |\n| ***Harry Potter and the Sorcerer’s Stone***| 1333            | **9M**                       | 90     | **132**    | **68**             |\n| ***Gone with the Wind***                  | **2032**            | 1M                       | **111**    | 70     | 51             |\n\nTable 3: Books, their number of editions, number of reviews on GoodReads, and LCS scores generated by different models (GPT3.5, Claude, and the average of all models) and averaged over five model runs. Results show that both the bigger models and the average of all models tend to memorize books with a bigger number of reviews and editions. The year the books were published does not seem to affect memorization. Books are presented in ascending order based on the number of editions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper raises a very crucial question about legality of LLMs and provides some quantitative as well as qualitative evidence for copyright infringement by LLMs.\n\nUpdate post rebuttal:\n\nI agree with their discerning rebuttal that the experiments around copyright violation specifically deserve to be considered, and in that their methods seem apt and sound. Hence, I am updating both my soundness and excitement scores.\nI would still like to see a more thorough discussion or some other way to assure readers of a legally sound narrative, for example, by listing lawyers who may have been consulted for the paper in the camera ready.", "reasons_to_reject": "It is hard for Computer Science or Linguistics educated reviewers (like myself) to judge the legal arguments in the paper. Despite the authors not explicitly making big legal claims, the major contribution of the paper is apparently best left to be judged in a court of law.\n\nThe other quantitative insights, e.g., that larger LLMs quote verbatim more, have already been established in literature, as cited by the authors themselves. Therefore, this paper adds only a few qualitative examples and a few quantitative statistics on fraction of times the copyrighted source was quoted verbatim. A larger study concerning more research questions and analysis will be better appreciated, e.g., what kind of material is more likely to be quoted verbatim? Are there ways to modify (at least the smaller models) either at pretraining or at inference to avoid copyright infringement?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper considers the question of copyright violation by large language models. The authors both prefix-probe and instruct several LLMs to quote from copyrighted material (bestselling books and LeetCode) and observe the fraction of times each does so verbatim. They argue in Section 2 how this may amount to copyright violation under US law, and show examples of the same. They find that larger LLMs like ChatGPT and Claude tend to quote verbatim the most (over 50 tokens on average, which is apparently the lower limit for copyright infringement).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9TaJAFEz5x", "reviewer": "Reviewer_AggV", "summary": "", "questions": "A: From my reading of the copyright laws in the US (other places could of course have different guidelines), there are no legal rules that permit a percentage or a fixed amount of reproduction of the copyrighted material (Link: https://www.copyright.gov/help/faq/faq-fairuse.html), however this paper alludes that anything above 50 (or 50-1000 as per the abstract) words could be construed as a copyright violation. What is the source/rationale for this number? \n\nB: Is the levenshtein distance character-based? Also are you comparing tokens generated by the models or space-delimited words?\n\n**A suggestion:** The authors rightly point out in the limitations that properly assessing copyright violation would require “complex legal considerations’’ -- I therefore encourage them to undergo an additional round of reviews and present a more nuanced perspective of the ways in which LLMs could violate copyright regulations (ideally longer than a short paper). ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your great comments, suggestions, and questions, which will help us improve the clarity of our work. We respond to your comments below. (W1=Weakness #1, Q1=Question #1).\n\n> **W1:** I am unsure if the setting and findings are non-trivially different from previous work on memorization...\n\nWhile Carlini et al. (2023) is certainly pertinent to our research, we acknowledge noteworthy distinctions. Carlini et al. (2023) only rely on prefix probing, while we consider multiple probing methods, including results suggesting that prefix probing is only scratching the surface. We consider a wider range of language families, and we are going to add an analysis as to which features tend to be memorized the most (based on content engagement and popularity Indicators). See the following tables. Lastly and most importantly, Carlini et al. (2023) do not relate memorization to copyright. NB: Carlini et al. (2023) is only two weeks from being contemporaneous work according to the ACL policies. (Deadline was June 23; Carlini et al. (2023) was on ArXiv on March 6.) \n\n> **W2:** There also seems to be some discrepancy about what counts as a copyright violation.\n\nThank you for highlighting the necessity of clarifying the discussion around copyright law violation. There is indeed variation in the number of words that violate copyrights. While some consider a limit of 300 words as acceptable for book-length content [2], others advocate for varying limits, spanning from 25 words [3] to 1000 words [4]. A common practice of 50 words applies to chapters, magazines, journals, and educational materials [5], and given our interest in books and teaching materials, we adopted this limit. Our rationale will be elaborated upon in the final version. To determine how many words constitute a copyright violation, most national copyright laws [1], as well as the provisions of the Berne Convention, outline the prerequisites of fair use and common practices. We are going to present the ongoing dialogue surrounding this issue in Section 3 of camera-ready. \n\n> **W3:** Due to the complexities surrounding the topic of copyright violation, this study essentially reduces to one involving long-text memorization \n\nWhile we understand the perspective that our study might appear to involve aspects of long-text memorization, we'd like to highlight some key distinctions that set our research apart from the study by Carlini et al. Our study not only encompasses a broader range of probing methods and model families but also ventures into the uncharted territory of connecting memorization with copyright considerations. The intent behind our study is to contribute to the broader discourse on copyright in the realm of natural language processing and to emphasize the importance of addressing these concerns to ensure the ethical and legal use of LLMs in the future. We believe that enriching our study with your valuable suggestions, a clear legal framework, and an extra analysis of features, provides both uniqueness and clarity.\n\n> **Q1:** ... What is the source/rationale for this number?\n\nThere is not an exact number in the legal rules, and thus what constitutes a violation is determined by the court's interpretation of fair use (see more in Limitations on exclusive rights: fair use 107 [1]. To show the quantitative insights of copyright content, we use the exact numbers from( https://theliteraryprofessionals.ie/writers-library/writers-focus/copyright-informatio). We provide more details in our reply to **W2**.\n\n> **Q2:** Is the levenshtein distance character-based? Also are you comparing tokens generated by the models or space-delimited words?\n\nRegarding Levenshtein distance, our implementation is character-based, whereas LCSs are over white space-delimited words. We will make this (more) clear in the camera-ready.   \n\nBelow we provide the tables with some additional data that are going to be used in our analysis about whether engagement and popularity Indicators affect copyright violations by LLMs.\n\n\n\n| Ranking blocks | Discussion Count | Num. of Submission | Num. of Companies |\n|:----------------|------------------:|-------------------:|-------------------:|\n| **1-200**          | **0.61**             | **0.63**              | **0.56**              |\n| 201-400        | 0.57             | 0.49              | 0.45              |\n| 401-600        | 0.43             | 0.43              | 0.42              |\n| 601-800        | 0.35             | 0.29              | 0.33              |\n| 801-1000       | 0.28             | 0.31              | 0.24              |\n| 1001-1200      | 0.25             | 0.31              | 0.25              |\n| 1201-1400      | 0.26             | 0.27              | 0.30              |\n| 1401-1600      | 0.18             | 0.22              | 0.34              |\n| 1601-1800      | 0.19             | 0.16              | 0.21              |\n\n| Ranking blocks | Discussion Count | Num. of Submission | Num. of Companies |\n|----------------|------------------:|-------------------:|-------------------:|\n| **1-200**          | **0.40**             | **0.36**              | **0.35**              |\n| 201-400        | 0.38             | 0.34              | 0.30              |\n| 401-600        | 0.28             | 0.32              | 0.27              |\n| 601-800        | 0.24             | 0.24              | 0.21              |\n| 801-1000       | 0.20             | 0.22              | 0.15              |\n| 1001-1200      | 0.17             | 0.19              | 0.14              |\n| 1201-1400      | 0.17             | 0.16              | 0.23              |\n| 1401-1600      | 0.12             | 0.14              | 0.20              |\n| 1601-1800      | 0.11             | 0.09              | 0.20              |\n\nTable 1&2: Responses from GPT3.5 and Claude for Leetcode Problem description. Show the LCS ratio tendency based on discussion count, number of submissions and number of companies. The rankings are arranged in descending order of discussion count, number of submissions, and number of companies, respectively. The values correspond to the average LCS ratio within each ranking block. LCS ratio = $\\frac{length \\ of \\ LCS}{length \\ of \\ golden \\  text}$. \n\n| Books                               | Num of Editions | Num of Reviews (GoodReads) | GPT3.5 | Claude | Avg all models |\n|-----------------------------------|-----------------:|-----------------:|--------:|--------:|--------:|\n| Angela's Ashes                      | 145             | 61K                      | 3      | 6      | 19             |\n| THE ROAD                            | 146             | 86K                     | 42     | 11     | 36             |\n| The Corrections                     | 192             | 18K                      | 3      | 3      | 22             |\n| Airport                             | 193             | 39K                      | 3      | 2      | 11             |\n| Ragtime                             | 195             | 43K                      | 70     | 3      | 22             |\n| The Wind-Up Bird Chronicle          | 208             | 27K                      | 12     | 3      | 13             |\n| Sophie's Choice                     | 211             | 89K                      | 16     | 4      | 13             |\n| Breakfast of Champions              | 217             | 253K                     | 3      | 4      | 13             |\n| The Godfather                       | 237             | 404K                     | 11     | 3      | 14             |\n| The Naked and the Dead              | 238             | 25K                      | 4      | 4      | 11             |\n| The Spy Who Came in From the Cold   | 376             | 104K                    | 26     | 5      | 18             |\n| The Fault in our Stars              | 377             | 5M                       | 29     | 40     | 38             |\n| The Thorn Birds                     | 393             | 343K                     | 3      | 4      | 11             |\n| The Girl with the Dragon Tattoo     | 493             | 3M                      | 2      | 4      | 11             |\n| Harry Potter and the Deathly Hallows| 604             | 3M                       | 8      | 5      | 40             |\n| Lolita                              | 953             | 811K                     | 80     | 120    | 67             |\n| For Whom the Bell Tolls             | 1105            | 284K                     | 88  | 7      | 31             |\n| ***Harry Potter and the Sorcerer’s Stone***| 1333            | **9M**                       | 90     | **132**    | **68**             |\n| ***Gone with the Wind***                  | **2032**            | 1M                       | **111**    | 70     | 51             |\n\nTable 3: Books, their number of editions, number of reviews on GoodReads, and LCS scores generated by different models (GPT3.5, Claude, and the average of all models) and averaged over five model runs. Results show that both the bigger models and the average of all models tend to memorize books with a bigger number of reviews and editions. The year the books were published does not seem to affect memorization. Books are presented in ascending order based on the number of editions.\n\nReferences\n\n[1] https://www.copyright.gov/title17/92chap1.html#107\n\n[2] https://janefriedman.com/sample-permission-letter/\n\n[3] https://stevelaube.com/how-much-can-i-quote-from-another-source-without-permission/\n\n[4] https://lupress.cas.lehigh.edu/sites/lupress.cas2.lehigh.edu/files/05%20R%26L%20Permission%20Guide.pdf\n\n[5] https://archive.nytimes.com/bits.blogs.nytimes.com/2008/06/16/the-ap-hot-news-and-hotheaded-blogs/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem that the authors tackle is an important and practical one -- it is essential to shed light on the legal implications of using LLMs in the wild, and this line of research could hopefully contribute to informing policy in the future.\n- The authors analyze many recent state of the art LLMs, and therefore these results could immediately be of interest to researchers and LLM trainers who could test and mitigate copyright infringements in their usage of these models.", "reasons_to_reject": "- I am unsure if the setting and findings are non-trivially different from previous work on memorization. For instance, the method used is the exact same as Carlini et al., 2023 (the authors do declare this) -- in that both papers use 50 tokens to prompt the open source models and measure sequence overlaps with the generated token sequences. Furthermore, the results that models are indeed memorizing copyrighted text is also something that can be predicted by Carlini et al.’s paper.\n\n- There also seems to be some discrepancy about what counts as a copyright violation. The abstract suggests a limit of 50-1000 words but the results are interpreted only using the 50-word limit. \n\n- Due to the complexities surrounding the topic of copyright violation, this study essentially reduces to one involving long-text memorization (albeit within a focused domain: books and coding problems), which has already been shown for LLMs by Carlini et al., 2023 (though for different LLMs).\n", "questions_to_authors": "A: From my reading of the copyright laws in the US (other places could of course have different guidelines), there are no legal rules that permit a percentage or a fixed amount of reproduction of the copyrighted material (Link: https://www.copyright.gov/help/faq/faq-fairuse.html), however this paper alludes that anything above 50 (or 50-1000 as per the abstract) words could be construed as a copyright violation. What is the source/rationale for this number? \n\nB: Is the levenshtein distance character-based? Also are you comparing tokens generated by the models or space-delimited words?\n\n**A suggestion:** The authors rightly point out in the limitations that properly assessing copyright violation would require “complex legal considerations’’ -- I therefore encourage them to undergo an additional round of reviews and present a more nuanced perspective of the ways in which LLMs could violate copyright regulations (ideally longer than a short paper). ", "paper_topic_and_main_contributions": "This paper presents an empirical analysis of the extent to which large language models (LLMs) tend to reproduce copyrighted text material that they might have memorized during their training. To this end, the authors analyze LLMs belonging to 6 different families (OPT, Pythia, Falcon, LLaMA 1, GPT3.5, Claude), and tested their ability to reproduce text from copyrighted material sourced from textbooks and programming problems on LeetCode. Their method involves prompting the LLMs either by using a 50-token prefix (in the case of open source LLMs) of the copyrighted material, or by simply asking them to generate the content (in the case of API-access LLMs) using a question in the prompt. They then measure the sequence overlap between the generated text and the original text, using metrics such as longest common subsequence (LCS) and Levenshtein distance. The results indicate that most LLMs can indeed reproduce copyrighted material and in-fact this gets linearly worse as LLMs get larger, where they copy more than 50 words of copyrighted material, ad verbatim. In fact, this is worse for closed-source LLMs such as GPT3.5 and Claude, where the LCS is in the 60s and the 70s, on average. Similar results are obtained for LeetCode problems -- LLMs violate copyright for greater than 30% of the total problems tested. The authors follow this up with a few qualitative insights, where they analyze the ambiguity of what counts as the first line in story, check for confabulations, and end with suggesting how prompt engineering can expose the models’ copyright issues. The paper concludes with the potential implications of LLMs violating copyright laws and recommendations for future practice in ensuring compliance with proper copyright regulations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be great if the plots contained error bars, or some depiction of the variability in the LCS between generated and original text.\n\nLine 188 has a random ‘creative vision’ bigram in it -- I am assuming this was left out in the draft by mistake?"}}]

Author rebuttal:

Meta-review: The paper "Copyright Violations and Large Language Models" presents work on examining LLMs for copyright violations. To this end, LLMs are probed for copyrighted material, such as books, but also code. The results indicate that some LLMs might violate copyright laws in the US.

The main criticism mentioned are a lack of a clear definition and the degree of novelty. 
But the task in itself is deemed important and findings interesting enough to merit a publication.
Decision: Accept-Main