Studying memorization of large language models using answers to Stack Overflow questions
Authors: Laura Caspari
Alexander Trautsch
Michael Granitzer
Steffen Herbold
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) are capable of answering many software related questions and supporting developers by generating code snippets. These capabilities originate from training on massive amounts of data from the Internet, including information from Stack Overflow. This raises the question whether answers to software related questions are simply memorized from the training data, which might raise problems as this often requires attribution (e.g., CC-BY license), sharing with a similar license (e.g., GPL licenses) or may even be prohibited (proprietary license). To study this, we compare responses to questions from Stack Overflow for questions that were known during LLM pre-training and questions that were not included in the pre-training data. We then calculate the overlap both with answers marked as accepted on Stack Overflow as well as other texts we can find on the internet. We further explore the impact of the popularity of programming languages, the complexity of the prompts used, and the randomization of the text generation process on the memorization of answers to Stack Overflow. We find that many generated answers are to some degree collages of memorized content and that this does not dependent on whether the questions were seen during training or not. However, many of the memorized snippets are common phrases or code and, therefore, not copyrightable. Still, we also have clear evidence that copyright violation happens and is likely when LLMs are used at large scales.

Published in Transactions on Machine Learning Research (09/2025)
Studying memorization of large language models using an- swers to Stack Overflow questions
Laura Caspari
lauvn.cappariBurt-pssou de
Faculty of Computer Science and Mathematics
University of Passau
Passau, Germany
Alexander Trautsch
alenander.treutschhhe www.serrttttttttte.ocooo www.
Facully of Computer Science and Mathematics
University of Passau
Passau, Germany
Michael Granitzer
Facully of Compuler Science and Malkemalics
University of Passau
Passau, Germany
Steffen Herbold
steffen.herrhold&kuuiippssssm
Faculty of Computer Science and Mathematics
University of Passau
Passau, Germany
Reviewed on OpenReview: Attps: openreviewww... net/ forum? id= ddocn//kaq
Abstract
Large Language Models (LLMs) are capable of answering many software related questions and supporting developers by generating code snippets. These capabilities originate from training massive amounts of data from the Internet, including information from Stack Overflow. This raises the question whether answers to software related questions are sim- ply memorized from the training data, which might raise problems as this often requires attribution (e.g., CC-BY license), sharing with a similar license (e.g., GPI. licenses) or may even be prohibited (proprietary license). To study this, we compare responses to questions from Stack Overflow for questions that were known during LLM pre-training and questions that were not included in the pre-training data. We then calculate the overlap both with answers marked accepted Stack Overflow well other texts internet. We further explore the impact of the popularity of programming languages, the complexity of the prompts used, and the randomization of the text generation process on the memorization of answers to Stack Overflow. We find that many generated answers are to some degree collages of memorized content and that this does not dopendent on whether the questions were seen during training or not. However, many the memorized snippets are common phrases code and, therefore, not copyrightable. Still, we also have clear evidence that copyright violation happens and likely when LLMs used large scales.
find the
1 Introduction
A common criticism of generative Large Language Models (LLMs), such as ChatGPT, Gemini, Ø and Claude is that they are stochastic parrots (Bender et all 202n) This means that they recombine existing texts based on inferred probabilities. Thus, under the stochastic parrot hypothesis, the LLMs generate collages
https://chat.opent hwwp:/hhatpppeann.com com/
bttps://gemini.. woogle.com
http::/ww..nthrppic.ooom

---

Published in Transactions on Machine Learning Research (09/2025)
in probabilistic manner. When we consider the training procedures of such models, which are aimed predicting the probability of the next token given an already existing text (prompt) (Radlord ellal, 2019) such a postulate makes sense. However, because the models learn probabilities of individual tokens, it is not clear if this really lcads to a recombination of longer text fragments as well, if such fragments are re-used exactly as is, if they are modified (e.g., replacement of synonyms), or if the generated texts deviate more from the source training material. Moreover, while such a hypothesis is casier to argue for when the prompts are similar or equal to the training data, it is unclear what happens when we work with prompts that were not seen during training. Further, such models are not only trained to predict next tokens, but also post- trained to align their outputs with human preferences (Ouyang et all, 2022). To gain better understanding regarding these aspects, our research is guided by the following research question: To which degree are texts generaled by LLMs collages of the training dala?
Within this study, we investigate this aspect of LLMs by considering data from the developer question and answer site Stack Overflow." E Thus, we study our rescarch question with respect to data from the software engineering domain and for the task of answering open-ended questions. These answers generally require longer, semantically coherent text and expressions. Since Stack Overflow is part of many corpora for the training of LLMs (e.g. the Pile (Gao all, 2020)), we assume that the question rattiionanswwrr answer pairs from Stack Overflow prior to the cutoff date for the creation of the training data are known. Vice versa, we can safely assume that panestion/nswwe pairs that were generated after the publication of the L.L are not used for training. We utilize this to create sample corpora of questions that were known and not known during training, along with their respective answers on Stack Overflow. Questions from the sample corpora are used as I.M.-proppts. We then analyze the subsequence overlap of the LLM response with the true answers. In addition, scarch for matching subsequences in large, recent Web-Crawl (Granitzer et all (2023)) to identify potential training data re-use outside of the Stack Overflow corpora. Studying these overlaps helps us to understand if generated texts are collages, where the data for the collages might have been sourced from, and if this effect is similar for questions seen during training and unknown questions. It also allows to study potentially arising copyright issues, particularly missing attributions. Our research will improve our understanding of how longer texts are related to the underlying training data, which has implications on the capabilities of an LLM to answer questions outside the scope of its training data as well as possible issues with copyright. This is not only relevant for the foundational rescarch for the advancement of LLMs, but should also help us better understand the legal risks resulting from the capability of memorization that affect both text generation, B as well as image generation in bi-modal models. B
Our results indicate that memorization of common phrases and code is common and that outputs are typically, at. least. to some degree a collage of such memorized content, that is likely not protected by copyright. Importantly, this is the case for both questions seen during training and new questions. Only a small amount of memorized content was long enough and non-common, though we also found clear evidence of memorization that is likely violating copyright. This raises concerns when LLMs are used at large scales.
Disclaimer: This paper does not provide legal advice, but rather only a scientific analysis of data with respect to memorization and possible copyright concerns.
2 Related Work
The previous work can be divided into three areas: plagiarism, privacy and security, and direct measurement of memorization. The first perspective on memorization is to consider plagiarism. Plagiarism is similar to memorizing content, especially when this is considered based on automated plagiarism checkers. Such plagiarism checkers are able to compare documents to existing database of literature and determine passages that are textually equal or similar. He Wynter et all (212)) use plagiarism detection to investigate LLM outputs regarding discourse and memorization of a diverse set of LLMs, e.g., ChatGPT, Bloom (Scaa et all, 2023), and Galactica (Taylor et all, 2022). They find that up to 80% of outputs contain plagiarized content, but also that this can be reduced by adopting prompts that specifically tell the model to avoid
'sttpa://stacoovorrLoo.. https//oo///////////uaaaaaa
https://kww.nn www.:/wwwww.nttt............e..w https://spectrun. leee. beee.org//iddourrnee copyriiig
times- open microsoft- lawsuit. html
2

---

Published in Transactions on Machine Learning Research (09/2025)
memorized content. Lee ct all (2023) investigate types of plagiarism with pre-trained and fine-tuned models using GPT-2 (Radlord et all, 201 and different fine-tuned GPT versions, e.g., Patent @ They find that GPT-2 plagiarized from OpenWebTeet (Gokaslan & Coheni, 2019) and that the fine-tuning has an impact on plagiarism of pre-training data: when the fine-tuning data overlaps with the pro-training data, this may reinforce memorization, while new line-tuning data can reduce the memorization of pre-training data. Mucller cll (20221) show that the number of potential copyright infringements varies greatly depending on the model, with bigger models generally producing more copyrighted texts.
In the second area, the authors come from a privacy and security perspective and investigate reproduction of private or sccurity sensitive content. Huang et all (202)) investigate reproduction of secret information in neural code models. They found that they can extract valid cloud credentials via autocomplete from Github Copilot and Amazon CodeWhisperer. a Lnkas ef all (2123) investigate the reproduction of Personal Tdentifiable Information (PII) from GPT-2. The authors propose attacks and mitigations for PIT extraction and evaluate them on GPT-2. They show that GPT-2 can reveal PII, demonstrating the model's capability for memorization. More recently, Nasr et. all (2028) explore extraction of training data from aligned models via novel divergence attack open source models and ChatGPT. They show that they are able to extract training data at a high rate.
The final area is concerned with direct estimation of memorization in models and its impact on how the models work. Biderman et all (21))) built a framework for comparing LLMs trained on the same data, The authors also evaluate memorization and find that a Poisson point process is a good approximation for the occurrence of memorized sequences over the process of training. Kandpal et all (2122) explore the effect of duplication on memorization of training data. They show that duplicated data is memorized more effectively,
demonstrating need for deduplication of the training dataset as a measure to prevent memorization. Rabin investigate the effects of noise and memorization in neural code models. They found that memorization is common in code models and that both correct but also noisy information is memorized.
ct all
Carlinictaal (2022) quantify boundarics for the amount of memorization. The authors find that memorization scales log-linear with model size. This makes important to study larger models as the authors largest model consists of 7B parameters. Similarly, Kiyomaru et al (21221 show that memorization occurs more frequently with larger models, increasing prompt length and higher text frequency in the training data. However, they also discover that texts that are not part of later training steps are reproduced less often, even if they appear frequently in the training data. [Wang et al. (2025) investigate how the task type affects memorization, showing that knowledge-intensive tasks exhibit the largest memorization effect. Schwarzschidd et all (2IZe)) propose an algorithm for prompt-optimization which they use to assess memorization in the context of unlearning methods, demonstrating that models trained on the employed methods can still reproduce large parts of the supposedly unlearned data. While the above literature is focused on GPT-style models (Radlord
aLAL.
2019)
models based on the T5 architecture ([aaie ZIZI). They found that memorization is also common in to-squeeee models, but also that
Zeng all (212E) considered
multi-task fine-tuning reduces memorization.
While the prior work provides strong results regarding memorization, our work completes our understanding regarding two crucial, not yet studied issues. First, we specifically distinguish between memorization of training data, when we consider questions that were also part of the training and memorized content that is re-used for questions that are not directly observed during training. This allows us to understand to which degree memorization can happen in arbitrary lasks, even those that are not directly in the training. Second, we manually validate not only if memorization happens, but also if memorized fragments may be copyrightable. While some works, e.g., Mueller ct all (2n22)) did a similar analysis, this was on fairly long texts (at least 160 characters) and only for a task where they specifically asked to reproduce content. Instead, our analysis allows us to understand to which degree memorization occurs naturally in generated texts, as well as whether the memorization is rather just for common phrases and code versus actual protected content that has a certain level of originality.
from patentt-
https://github.c http:/uubbcommttttueee/////ooo/p
https://avs. amazon. httpa/cccccc/cccoiiipeeeerrr
3

---

Published in Transactions on Machine Learning Research (09/2025)
2.1 Research Hypothesis
As described above, we want to undersland the use of collages within generated texts as means to study memorization. We use syntactic similarity observed through sequences of tokens that overlap between generated texts and reference as means to study collages. Further details on how this is measured are described later in Section H. Based on the literature we discussed above, we derive the following concrete hypothesis that we want to test within this study.
H1 LLM-generated answers Stack Overflow questions are syntactically more similar accepted human answers for questions observed during the pre-training of models in comparison to new questions.
H2 LLM-genorated answers for popular programming languages are syntactically further away from the training data than for niche languages.
H3 Complex prompts following the style of questions in the training data increase the syntactic answer similarity to the training data and the amount of memorization within generated answers.
H4 Generation of content without randomness increases that amount of memorized outputs.
Hypothesis H1 is based on the common observation from the literature that LLMs are in general capable of memorization. Consequently, for out-of-distribution questions, i.e, questions not contained in the training data, the overlap of generated answers with answers in the training data should be smaller than for questions from the training. Importantly, if this prediction turns out to be false, this would in turn imply that all contents that LLMs generate are equally similar to the training data, regardless of whether the prompts were observed during training or not. Hypothesis H2 based on observations from the literature regarding the impact of duplication (Kandpal et all, 2122). These indicate that a larger diversity in data leads to lower memorization. Consequently, assuming that the diversity the training data is higher for popular languages, the syntactic overlap with the training data should also be higher. Hypothesis H3 is derived from the work by de Wynter et al (2Izz), who found that the choice of prompt impacts the amount of plagiarism and can be also used to reduce it. Vice versa, it should also be possible to use the prompts to trigger plagiarism, by requiring the LLMs to write in a certain style. Hypothesis H4 follows from the technical aspects of the text generation process. If a response to a prompt is indeed memorized, this memorized content should have the highest probability given a prompt. The role of randomization of the next token based on the distribution from training then allow the model to deviate from the memorized content.
3 Methods
We now describe our research method in detail, i.e., the subjects, variables, how we execute the study to collect the required data, as well as the analysis of the collected data. Notably, we design our study protocol in manner that does not require direct access to the training data. While concrete knowledge about the training process would be beneficial, this is unfortunately unrealistie for the recent generation of LLMs, where the training process has become business secret. To still be able to study such models, we instead design our protocol around reasonable assumptions on the data used during the training process.
3.1 Subjects
The subjects of our study are two-fold: generative LLMs and Stack Overflow posts. The main subjects of our study are generative LLMs. We study these models by analyzing their outputs with respect to content from Stack Overflow.
To determine which LLMs we use as subjects, we define the inclusion criteria (see Table i in the appendix). The criteria have the goal to enable the evaluation of state-of-theartt models and at the same time enable us to guarantce that we can be reasonably sure that a) Stack Overflow posts were likely part of the training or b) Stack Overflow posts were not inffuenced by information provided by a LLM, posts where users did not use LLM either in writing the question for formulating answer. Based these criteria, we
4

---

Published in Transactions on Machine Learning Research (09/2025)
Date range
Popular languages
Niche languages
1//2220-12/2021
P20 with 157, 157,558 question/ansse annser pairs
N20 with 5,939 question /answer pairs
09/ 09/2022-11/2022
P22 with 32,148 question/anssn answer pairs
N22 with 1,323 question ansser pairs
Table 1: Identifiers and amount of posts retrieved for the different subsets of posts from Stack Overflow.
cannot consider the latest generation of models like ChatGPT 4, Gemini, or Llama (Lomivron all and newer, because the cutoff date for their training data is after the public release of ChatGPT. Overall, only two LLMs meet these criteria: ChatGPT 3.5 Turbo, Im which is instruction fine-tuned version of GPT-3 Brown of all (2020) and Vicuna which is an mnstruction-iie-tune
version of Llama Touvron ct al (002) We decided to rather use the publicly available Vicuna model for our study, to facilitate future replication of our work. An additional concern with the use of GPT-3.5 Turbo would be updates that OpenAl has done since the release, that possibly extended the training data cutoff date. Such concerns do not exist with Vicuna.
Our second set of subjects are posts collected from Stack Overflow. We the inclusion criteria depicted in Table E These inclusion criteria are designed to select pairs of questions and accepted answers in a manner that ensures a certain level of quality by requiring positive votes. Further, we collect only posts with certain tags to only include posts related to pre-defined programming languages. We use the Tiobe ranking of popularity of programming languages based on Web search data from 2020 to determine programming languages that are popular and those that are less popular, in the following referred to as popular and niche programming languages. This selection assumes that programming languages that are more often searched for by developers also receive more attention on Stack Overflow and that there are also more resources for such languages on the internet in general. We use the Top-10 programming languages from the Tiobe index as popular languages and ranks 51-100 to define niche programming languages. For each of these languages, we determine the related tags on Stack Overflow and then create subsets with all posts that contain one of the tags. Table R lists the tags we identified for this filtering. The selection of these subsets enables us to study the possible impact of the amount of information available on a topic on memorization.
For both subsets, we collect posts from 2020 as well as from June to November 2022. The data from 2020 almost certainly part of the training of the models we used. The data from 2022 is too recent for the LLMs we study and, thus, was not used for their training of the foundation model, i.e., Llama. A risk that this data was used for the instruction tuning of the initial version of ChatGPT can also be ruled out, since this was done using hand-written responses by a dedicated team (Ouvang et all, 2022).. The post-training of Vicuna has happened after the release of ChatGPT and based on ChatGPT conversations collected in the ShareGPT data.
Overall, this means we collect the four sets of Stack Overflow posts depicted in Table and use these to evaluate how similar the answers to these questions generated by Vicuna are to the existing answers, as well as other information from the internet.
3.2 Variables
Since our goal is to understand to which degree the answers are collages from the training data, we define variables to measure the similarity between the generated answers by the LLMs and the other data sources. In this comparison, we are only interested in the syntactic similarity, not in the semantic similarity. Thus, we do not care about mcaning, but rather only about the words and the combinations that were used. In terms of memorization, ion, this means we only consider exact memorization. We conduct two such comparisons.
First, we compare the answer from the LLMs to the accepted answer given on Stack Overflow. To formally define this, we refer to a generated answer as Xgen (x x0en)) and to a reference answer from Stack
10 https://platfon hwtp://aaa.aac//////o///aa//a///
https://lmay:: /ttp:///aaoooo///0000000000----0
https://veb.archin htw.://wwwerreee/020022111222220
cattbbe..oonttoooo index/
sttps://bharepp..com
5

---

Published in Transactions on Machine Learning Research (09/2025)
Overflow as X = (zf", ..., such that 2²"", L..,,..., m and xj";J I..,, m' are the sequences of tokens of the generated answer/solution Thus, we compare the token streams of the LLM answer and the Stack Overflow answer. We compare these token streams using metrics from the ROUGE suite for measuring text similarity Lin (2mnz)) ROUGE metrics directly look at the overlap between subsoquences between two texts, which exactly what we are interested in with our syntactic comparison. A large overlap would be an indication for collages. Concretely, we use two variants of ROUGE. The first variant ROUGE-N, which is defined as
ROUGE-N()) = E Egram.. E X CountM X
(1)
Egrom, CountMatches(granny Xgen)
where gram, E X are all n-grams from X and Count&Maccer is a function that counts how often an n-gram match occurs in a sequence of tokens. Thus, we count how many of the n-grams from the generated texts appear in a reference answer from Stack Overflow.
Additionally, we compute
Rics LCS(X X0
(2)
where LCS computes the longest common subsequence between two texts. This metric is not directly available within ROUGE, but rather one of the components used compute ROUGE-L, which would also use m' to normalize the overlap, i.c., the length of the reference solution. However, since we are only interested in whether text. from the reference is used in the generated answer and not if all text from the reference appears in the generated answer, using only Rics gives more reliable data, in case the generated answers are shorter than the reference answers.
Since training data for LLMS is not limited to Stack Overflow, but is rather sourced from large Web crawls like Common Crawl, [ is also possible that the generated answers are collages of other texts from the training. While the training data is mostly based on data that can be searched for on the internet (with the possible exception of the book corpora used), the data set is neither fully specified nor publicly available. Consequently, we cannot search for overlaps between the generated answers and the training data directly. Instead, we simply use a Web search for exact matches to understand to which degree we have an overlap with data possibly seen during training. We take the pallern from ROUGE-N and define
WEB-ROUGE-NN) = Egram, EXgen Has WebMatch(gramm.)
(3)
Igram, E 9ee
where Has WebMatch is defined as whether there is an exact match for n-gram with Web search where we restrict the time to scarch hits older than the cutoff date for the pre-training. Specifically, we use 55
datasets from the OpenWebSearch project Granitzer et all (2112)) in from February to April 2024 for this purpose, summing up to a total size of 3.3 TB. Since we are interested in English Stack Overflow posts, we only considered the English part of the OWS data, roughly 1.3TB. Overall, the datasets contain crawls from 4.1M eu.
different hosts and 195M unique URLs. The data is publicly available under
For ROUGE-N(m) and WEB-ROUGE-NNN) we need to specify the length of the n-grams want to consider for the memorization. The literature on plagiarism detection suggests four words for precise plagiarism detection based on n-grams (eavrón-Cedeno & Rosso, 2009). Considering the rule of thumb that token is roughly 1 of a word, we should use 4 3 5.33 grams, which we round up to n = 6. Thus, we consider information to be memorized when we have six subsequent tokens overlapping between the generated sequence and the accepted answer for ROUGE-N(m), respectively, the Web corpus for WEB-ROUGE-NN))
3.3
Data collection
We use a data dump from Stack Overfiow to collect the posts according to the criteria defined in Section B. Once we have all data available, we generate answers for each question with Vicuna. We use two different
//attpps//comoocraal.
"hhttpss//ppenwccscarchh
17 /ontps///eelppppensccccm/e/e///p
6

---

Published in Transactions on Machine Learning Research (09/2025)
prompts to answer these questions (see Table @ in the Appendix). With the simple prompt, we just ask the question directly without changing anything or providing any instructions to the L.M. This way, we get a "natural" reaction from the LLM that is not guided by us. With the second prompt, we try to bias the output in a direction, that makes a collage more likely, by first structuring the information in the same way it is available on Stack Overflow and then also asking for an answer in this style. The idea that this prompt increases the likelihood that a generated answer is indeed a collage from Stack Overflow, because that is what we are actively looking for. We use two different temperatures to generate the answers. The temperature is parameter that influences the randomness of the generated text by modifying the probability of sampling tokens. First, we set the temperature of the models to zero. This means that we avoid randomness and creativity in the outputs and just use the most likely next token all the time without sampling. Our rationale is that this increases the pressure on the model to create collages, because the randomness is part of what allows the model to generate new content. Next, we set the temperature to one, With a temperature of one, we sample according to the distribution of the next token determined by the LLM. This means that there is more randomness in the generation of the answers which could decrease the use of collages or mean that combined sentence fragments become shorter. Overall, this means we have 16 different sets of of results, i.e., the combinations of four subsets of Stack Overflow posts, ie., P20, N20, P22, and N22; and four answers from the LLMs for each post with different prompts and temperatures. When presenting, analyzing, and discussing results, we identify these by a triple of year and programming language, prompt, and temperature, e.g. (P20, simple, low).
3.4 Statistical analysis
We compute the Rice: ROUGE-N(w) and (EB-ROUGE-N(1) for n n= 6 as defined in Section 1 for all 16 data sets with different combinations of Stack Overflow posts, prompts, and temperature. For each approach, we report the mean value, as well as a 95% confidence interval for the mean value. We compute the confidence intervals in a non-parametric manner using bootstrap sampling Eiron (1897)). We use Bonferroni-corrected significance level of 0.05 0.001 to compute these intervals to account for the multiple tests. We do not conduct additional statistical tests, but rather use the confidence intervals to determine significance. Hence, we conclude that results are significantly different with a confidence level of 95%, if their confidence intervals results do not overlap.
16-3
We analyze our hypothesis based on the results of our study and the expected observations in our data they predict.
H1 predicts that the overlaps in text we compute for the data from 2020 should be larger than for the data from 2022, as the questions from 2020 were part of the training data. In our data, this means that fix the prompt, and temperature, and then compare the P20 to the P22 data and the N20 to the N22 data. We expect that the overlap for P20/N20 is larger than for P22/N22.
H2 predicts that the overlaps for niche languages are larger than for popular languages. In our data, this means that we fix the year, prompt, and temperature and compare the results for popular languages to niche languages. We expect that the overlap for N20/N22 is larger than for P20/P22.
H3 predicts that a complex prompt can increase memorization. In our data, this means that when fix the year, popularity, and temperature, we expect that the overlap is larger for the complex prompt than for the simple prompt.
H4 predicts that less randomness means more memorization. In our data, this means that we fix the year, popularity, and prompt. We expect that the overlap for the low temperature is larger than for a higher temperature.
For all of the above, we carefully analyze the difference between ROUGE and -OBB-OUUE-NNN to understand from where information may have been memorized and to which degree generated language deviates from language on Stack Overflow and the internet in general.
7

---

Published in Transactions on Machine Learning Research (09/2025)
3.5 Qualitative analysis of overlaps
While collages indicate that text fragments were memorized, this does not automatically imply that they would be copyrightable, as this requires a certain level of originality. For example, a common phrase like "our research is peer-reviewed" is sufficiently long to assume that this wording might have been memorized, but this would not be original and, therefore, copyrightable. To understand not only if we observe memorization, but also what is memorized, we conduct a qualitative analysis of the collages. Since our total corpus of data is too large for full analysis over all languages, prompts, and years, we use sampling to collect a suitable subset for our qualitative analysis. We control for the length of the n-grams and the occurrence frequency during the sampling.
The length of the n-gram. Short sequences with few tokens are less likely to meet the threshold for originality and are more likely to be randomly occurring. We consider short sequences at the threshold of what we consider collages (6-10 lokens), longer phrases (11-20 tokens) and longer texts (21+ tokens) as subgroups for the qualitative analysis.
Occurrence frequency. How often a phrase is found in the Web corpus may be related to its originality, i..., low frequency texts might be more original than high-frequency texts. However, high-frequency could still be original, e.g., the texts of license agreements. We distinguish if n-grams are found only once in the web corpus (unique), multiple times but less than the median the corpus (2-4 times) and more often than the median (4 times).
For each combination of n-gram length and occurrence frequency, sample 40 sequences, i.e., 3-3-40 360 sequences of tokens that were generated by the LLM and found in the Web corpus.
Each sequence is independently annotated by two of the authors using deductive coding. The codes were defined as "common code", common phrase", "copyrightable code", and "copyrightable phrase". Thus, the annotators were asked determine if the n-gram is a code-snippet or some other phrase and to judge whether they believe this is original and, therefore, possibly copyrightable. Optionally, the annotators could provide a comment to explain their judgment. We report the confusion matrix of the agrecments, well Cohen's k (Cohenl (960)) as measure for the reliability of the annotations.
4 Results
In the following, we present the results of our work. To facilitate replication the data and code used to obtain these results are available online: https://githut com/ http:///eeeggglla////eatttttoot. wwttonnn--ttsttttttoaoooeeeeeeee nemorization/
4.1 Overview of the metric distributions
Figure MI shows the distribution of variables for each of the subgroups. The mean values, incl. confidence intervals can be found in Table A Our first observation is that for all plots, the difference between the data from 2020 and 2022 is very small. While we will look at this in greater detail later when studying the hypothesis, the broad trends regarding the syntactic overlaps are, apparently, independent from whether the data was seen during training.
In general, the 6-gram overlap between the actual answers and the generated answers measured with ROUGE-N(6) is very low and, on average not more than 3.8% in any subgroup of our data. Similarly, answers are typically not fully memorized with a mean RiCs of at most 19% of the whole answer. However, looking at the distributions depicted in the violin plots, both distributions have long tails, indicating that while rare, stronger memorization does happen. When looking at the overlap of the generated answers and content found on the internet, we observe that the average WEBBROOGE for the respective subgroup. This significantly higher than for ROUGE(6), i.c., a notable part of the texts that are likely not random can be found as is on the internet. Whether this text was part of training is something we cannot answer due to lack of access to the training data, but observing exact overlaps of six
is between 10.6% and 17.5%
8

---

Published in Transactions on Machine Learning Research (09/2025)
Simple prompt
Complex prompt
ROUGE
ROUGE
0.50
0.25
0.00
Popular Temp.-0
Popular
Nicho Tomp.-D
Popular Temp.-0
Popular Temp
Temp.-0
Temp.
Temp -1
Temp
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
Popular
Niche
Temp.=0
Temp.=0
Temp.
Temp.
Temp
Temp
1.0
1.0
WEB ROUGE
WFB ROUGE
0.5
0.5
0.0
0.0
Niche Temp.-0
Popular Temp.-0
Niche Temp. #1
Popular Temp =1
Niche Temp.-0
Popular Temp -0
Niche Temp =1
Popular Temp.
Figure 1: Distribution of the scores for all configurations. The violins are split by year.
Set Year
Complexity
Diff.
ROUGE-N(6) M
RECs
WEB-ROUGE-N())
CI
M
CI
M
CI
simple low
0.020
0..018, 0.022]
0.173
[0.170, 0.175]
0.119
[0.117, 0.121]
simple high
0.018
0.016, 0.020
0.167
[0.165, 0.170
0.114
00.111, 0.116
complex low
0.020
(0.018, 0.022]
0.165
(0.162, 0.168
0.106
[0.104, 0.107
complex high 0.018
[0.016, 0.020]
0.158
[0.155, 0.161]
0.107
[0.106, 0.109]
N
simple low 0.021
0..01,, 0.025]
0.176
[0.170, 0.182]
0.117
[0.112, 0.121]
simple high
0.019
00.015, 0.023
0.168
[0.162, 0.174
0,112
[0.108, 0.116
complex low
0.020
[0.017, 0.024
0.166
[0.161, 0.172
0.105
[0.101, 0.9
complex high 0.020
[0.016, 0.024
0.162
[0.156, 0.167]
0.102
[0.098, 0.106
simple low 0.034
00.03,, 0.034
0.186
[0.185, 0.187]
0.171
[0.170, 0.171
simple high 0.030
00.03,, 0.031
0.179
[0.179, 0.180)
0.164
[0.164, 0.165
20
complex low 0.033
[0.032, 0.033
0.177
[0.176, 0.177
0.159
[0.158, 0.159
complex high 0.029
00.029, 0.030
0.169
[0.168, 0.170)
0.154
[0.154, 0.155]
P
simple low 0.038
0.003, 0.039]
0.192
00.191, 0.195]
0.166
[0.165, 0.167]
simple high 0.034
00.03,, 0.036
0.185
[0.184, 0.187]
0.161
[0.159, 0.162
complex
low
0.036
[0.035, 0.038
0.182
[0.181, 0.184
0.156
[0.155, 0.157
complex high
0.033
(0.032, 0.034
0.174
[0.172, 0.176)
0.152
[0.151, 0.153
Table 2: Mean values (M) and confidence intervals (CI) for each metric and each subgroup within our data.
tokens with that ratio unlikely be an artifact of our evaluation. Still, this is a strong indication that generated texts are, at least partially, indeed collages of the training data.
Exact memorization of answers rare but seems to happen data seen during pre-training. terestingly, we also observe very similar syntactic memorization patterns for answers not seen during pre-training. Furthermore, all generated texts have notable overlap with text available the internet, indicating that answers also, at least partly, borrowed from other sources.

---

Published in Transactions on Machine Learning Research (09/2025)
Comparison
ROUGE-N()
RLCS
WEB-ROUGE-NGO
H1 (P20, simple, low) (P22, simple, low)
-0.004
-0.006
-0.005
(P20, simple, high) vs. (P22, simple, high)
-0.004
-0.006
-0.003
(P20, complex, low) vs. (P22, complex, low)
-0.003
-0.005
(P20, complex, high) vs. (P22, complex, high)
-0.004
-0.005
-0.002
(N20, simple, low) vs. (N22, simple, low)
(N20, simple, high) vs. (N22, simple, high)
(N20, complex, low) vs. (N22, complex, low)
(N20, complex, high) vs. (N22, complex, high)
H2
(N20, simple, low) vs. (P20, simple, low)
-0.014
-0.013
-0.062
(N20, simple, high) vs. (P20, simple, high)
-0.012
-0.012
-0.050
(N20, complex, low) vs. (P20, complex, low)
-0.013
-0.012
-0.053
(N20, complex, high) vs. (P20, complex, high)
-0.009
-0.011
-0.047
(N22, simple, low) vs. (P22, simple, low)
-0.017
-0.016
-0.049
(N22, simple, high) (P22, simple, high)
-0.015
-0.017
-0.049
(N22, complex, low) vs. (P22, complex, low)
-0.016
-0.016
-0.051
(N22, complex, high)
vs. (P22, complex, high)
-0.013
-0.012
-0.050
H3
(P20, simple, low) vs. (P20, complex, low)
+0.009
+0.012
(P22, simple, low) vs. (P22, complex, low)
+0.010
+0.010
(N20, simple, low) vs. (N20, complex, low)
+0.008
+0.013
(N22, simple, low) vs. (N22, complex, low)
+0.012
(P20, simple, high) vs. (P20, complex, high)
+0.010
+0.010
(P22, simple, high) vs. (P22, complex, high)
+0.011
+0.009
(N20, simple, high) vs. (N20, complex, high)
+0.009
+0.007
(N22, simple, high) vs. (N22, complex, high)
+0.010
H4
(P20, simple, low) v8. P200, simple, high)
+0.004
+0.007
+0.007
(P22, simple, low) vs. (P22, simple, high)
+0.004
+0.007
+0.005
(N20, simple, low) vs. (N20, simple, high)
+0.005
(N22, simple, low) vs. (N22, simple, high)
(P20, complex, low) (P20, complex, high)
+0.004
+0.008
+0.005
(P22, complex, low) (P22, complex, high)
+0.003
+0.008
+0.004
(N20, complex, low) vs. (N20, complex, high)
+0.007
(N22, complex, low) vs. (N22, complex, high)
Table 3: Difference of mcan values between subgroups. Results only reported if the confidence intervals, reported in in Table 2 are not overlapping. The groups are arranged such that the hypothesis predicts that the first-ment mentioend group has a higher mean value which should yield a positive value. Negative values are contradicting our hypothesis.
4.2 Evaluation of Hypothesis
In the following, we will look at these result with respect our hypothesis in detail. This analysis is based Table 3 that summarizes the differences between relevant subgroups for the hypotheses H1-H4.
H1: LLM-generaten answers to Stack Overflow questions syntactically more similar to ac- cepted human answers for questions observed during the pre-training of models in comparison to new questions. Our results indicate that there are siguificant differences for popular programming languages. For the niche languages, we do not observe any significant differences. However, contrary to our expectation, the direction that the overlap between the generated and actual answers higher for the P22 data than for the P20 data, i.e., the generated answers are more similar to answers not seen during training. This is consistent across all three metrics.
10

---

Published in Transactions on Machine Learning Research (09/2025)
We reject H1 and even observe a slight opposite trend indicating that the newer, unseen data is syntac- tically closer to the LLM output than data that was part of the pre-training.
H2: LLM-generated answers for popular programming languages are syntactically further away from the training data than for niche languages. Our results provide a strong indication for the opposite, i.e, that the memorization of content for popular languages is significantly higher than for niche languages. This is consistent across all subpopulations and all three metrics. For ROUGE-N()), the difference is between 0.9% and 1.7%. While these numbers seem small, recall that the highest average we observed for ROUGE-N() was only 3.8,, which makes the difference a statistically large effect, although low in absolute terms. For RIcs we observe differences between 1.1% and 1.7%. While consistent, the effect is a lot weaker than for ROUGE-N(6), because the average of RLCS are higher. The strongest difference can be observed for WEB-ROUGE()), where the increase is between 4.7% and 6.2%.
We reject H1 and observe the opposite, i.e. that memorization more common for popular languages, which indicates that data diversity does not hinder memorization. We hypothesize that this might mean that topic prevalence is the dominant factor for memorization, i.c., topics that are frequently occuring are more likely to be memorized, regardless of the data diversily.
H3: Complex prompts following the style of questions in the training data increase the syntac- tic answer similarity to the training data and the amount of memorization within generated answers. Our results provide a weak signal that this might be the case. While we observe no significant difference ROUGE-N6)) we see small increases RLCs ween 0.8% and 1.1% for all data subpopula- tions, except those with the N22 data. WEB-ROUGE-ME and 1.3%. This indicates that the complex prompt slightly shifts the distribution of the output to create syntactically slightly longer common subsequences with the actual answers, but also to use more n-grams that can also be found on the internet in general. Both happen without significantly increasing the overlap of the n-grams with the actual answer.
also slightly increases with values between 0.7%
Our results for this hypothesis are inconclusive. We find hints towards complex prompts increasing the ratio of memorized outputs, but the signal is weak and inconsistent across metrics.
H4: Generation of content without randomness increases the amount of memorized outputs. Our results show consistent, but small increase in memorization for the popular languages. For niche languages, this can only be observed for RLCs with the N20 data and the complex prompt. When we combine this data with the insights from H2 that show that memorization is stronger for popular languages, we find that our hypothesis is partially confirmed.
We accept our hypothesis for the popular language for which we find a consistent but small increase in all metrics, and reject it for the niche languages. We hypothesize that this means that low randomness can only increase memorization that sufficiently strong, which only applies to the popular languages in our data, based on our results for H2.
4.3 Qualitative insights into memorized data
For this analysis, we only considered data for which we observed the largest overlap with the actual answers basod on the results for the hypotheses above. Thus, we selected data from popular projects (H2) from 2022 (H1) and a low temperature (H4). We decided to label data for both the simple and the complex prompt, because the results for H3 are inconclusive. According to the commonly used criteria by McHugl (2012), the agreement between both raters is moderate, i.c., E [0.6, 0.8). A detailed look at the disagreements revealed that this is driven by different interpretations of corner case regarding what constitutes code. Rater A typically judged corner cases (e.g., Markdown table headers) as phrases, rater B judged such fragments as code.
The fragments we find as is on the internet are, from our perspective, mostly not copyrightable: we ob- served aspects like import statements, standard layout parameters, calls to functions with generic names
11

---

Published in Transactions on Machine Learning Research (09/2025)
and parameters, phrase like "follow these steps". However, also found six data points for which one of the raters deemed that this might be sufficient for copyright protection (see Table B in the appendix). We scrutinized these data points further to find their possible sources and understand if they indeed violate copyright. Two samples seem to be strong candidates for copyright violations. The first a definition that is quoted from Wikipedia. The second is the text of an error message from the Spring framework. Definitions are protected by copyright, IT for error messages it depends on the use case, c.g., when reporting an error it is fair use, however, when designing own products re-use may be a violation." ! Another snippet an API call from product documentation, which means that this likely does not violate copyright and would be fair use. Two other snippets are part of the formatting information, once of table, once of a Web site. Such formatting information is typically not copyrightable. 4i Finally, we also have one case, where the text we considered memorized from training data was instead only copied from the question, i.e., a false positive for memorization. Overall, the most conservative reading of these findings is that there is only on clear copyright violation out of 720 samples we chocked.
Assuming user uses an LLM daily only once to answer a question, this would mean that there would roughly one copyright violation every two years. While this number seems very low, LLMs may be used at very large scales. Assume you have a mid-sized company with 100 employces, working 220 days a year, querying an LLM on average four times a day. This would mean there are 220-100-4 ~ 122 copyright violations per year.
720
Overall, few possibly memorized texts lead to possible copyright violations, meaning that individuals who sometimes use LLMs have low risk. However, power users and companies almost certainly, at least sometimes, generate memorized content that could be protected by copyright.
5 Discussion
Our results confirm prior work that LLMs may memorize contents, even with instruction tuning. While we did not study the last generation of LLMs, but rather the first generation that used instruction tuning, we are not aware of any fundamental differences in the training procedure that would imply that our findings do not hold, i.c., that memorization can still happen. A similarly controlled experiment is, unfortunately, not possible with newer LLMs, because we camnot ensure that data neither influenced by LLMs (may be been generated) or was used for their training-
We believe that the crucial finding from our work is not only that there is still memorization, which is hardly surprising, nor that this can sometimes lead to outputs that are protected by copyright. Instead, we believe that the crucial aspect that our work demonstrates is that this happens regardless of whether the prompt was part of the training data. This is also a key difference to prior work on memorization, e.g,, by Carlini et all (2123) who only demonstrated that the completion of texts that are from the training data may be memorized or Mueller et all (2022) who directly ask to generated specific content. In contrast, we specifically tested with prompts that were not part of the training, i.c., questions askod on Stack Overflow after pre-training was finished. Still, the answers for these questions had often overlap with texts from the internet and our analysis shows that at least one answer violates copyright.
Moreover, our results indicate that, at least for the Q&& scenario we studied, the degree to which outputs are memorized does not depend on whether the question was part of pre-training. Indeed, the overlap wetn the actual answers and the generated answers was even higher for questions not seen during training. This strongly hints at the fact that LLMs always generate collages to some degree, regardless of whether the prompts are known from training or not.
See, e.g. dictionary ol
egg.
copyrigns 20 e.g. infringenent
21 see,
12

---

Published in Transactions on Machine Learning Research (09/2025)
6 Limitations
A clear limitation of our work that Vieuna, an LLM several years old, was our only study subject. As discussed above, we chose Vicuna because we wanted controlled setting in which we knew there was data that had not been seen during training and had not been influenced by users who posted LLM-generated content. Consequently, we cannot guarantee that our results will translate to newer model generations. While it is reasonable to assume that the results will hold for models with a similar architecture and prompting stratogy, it is highly uncertain for multi-hop prompting and reasoning models at (2025), which have a slightly different output generation strategy involving the production of a stream of thinking tokens. Despite the limitations of translating the results to newer architectures, our study paves the way for future research because we found that memorization in generated content does not depend whether the questions were seen during training. Thus, future studies do not require such strong control over when the data was seen during training, enabling research on newer models. Instead, the focus should be on determining whether supposedly memorized output was generated by an LLM, which would also reduce legal uncertainties in LLM usage.
The second clear limitation of our work that we found only a single clear case of a copyright violation and one more candidate that depends on the context. This threatens the generalizability of our claim that power users and companics almost certainly, at least sometimes generate memorized content. However, we also found three more memorized data points in our sample that are sufficiently unique to be copyrightable, they were just from contexts that either allow re-use due to fair-use exemptions of copyright or for textual data that is in general not copyrightable. Thus, while the cases in which we actually found copyrightable content were lower, the cases where we found relatively unique, uncommon statements were more with five hits. Therefore, while we cannot rule out that the cases were we found copyrightable content were outliers and that they are a lot more uncommon than our data indicates, the existence of other cases that are just not copyrightable because they fall under exceptions of copyright laws, indicate that this is unlikely.
7 Conclusion
We conducted a controlled experiment in which we studied to which degree generated answers are the same as actual answers, for both questions seen and unseen during training. We selected our data in such a way that it is unlikely that any actual answer is AI generated, by selecting a timo-frame prior to the relcase of ChatGPT. We found that whether questions were seen or not during the training has no strong impact on the syntactic similarity between the actual and generated answers. In contrast, syntactic similarity was even slightly higher for unscen questions. We also observed that questions for common topics are more likely to lead to memorization and that randomization in the text generation processes decreases memorization. For all questions, we found significant overlap of 6-grams with texts found on the internet, hinting at possible memorization and manually studied this for 720 possibly memorized snippets. While these were mostly harmless, c.g., common code like import statements, we also found one clear copyright violation and one candidate for copyright violation. Notably, the copyright violation happened in context where we did not try to trigger it, e.g. by giving a prompt from a protected source and asking for completion. Instead, it happened for Stack Overflow question unseen during training by quoting Wikipedia without reference. Thus, while most queries to LLMs are likely not violating copyright, even these rare cases add up to a possibly massive number of copyright violations, considering the scale at which LLMs are used.
Broader Impact Statement
The copyright implications of LLM use have huge societal implications and the question if and to which degree LLMs violate copyright is already the topic of multiple lawsuits. Our work provides an indication that while generated answers are some degree collages from the training data, copyright violations are rare for individual cases but common at scale, independent of whether the prompts are known from the training.
13

---

Published in Transactions on Machine Learning Research (09/2025)
References
Alberto Barrón-Cedeño and Paolo Rosso. On antomatic plagiarism detection based on n-grams comparison. In Aduances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings 31, pp. 696-700. Springer, 2009.
Emily M. Bender, Timnit Gebru, Angelina McMjilla-Maajor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, FAceTT 221, , pp. 610623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10. 11.115/4442188 3445922.
3445922. URL https://dol.org:
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emna Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2397 2430. PMLR, 23 29 Jul 2023. URL attps://proceedingg mlr.
htmll
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vone Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ily Sutskever, and Dario Amodci. Language models are few-shot learners, 2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models. In The Eleventh International Conference Learning Representations, 2023. URL https://openrewiiet hwt.:eooommddddaaattttte 1cK.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychologicol measurement, 20 (1):37-46, 1960.
Adrian de Wynter, Xun Wang, Alex Sokolov, Qilong Gu, and Si-Qing Chen. An evaluation on large lan- guage model outputs: Discourse and memorization. Natural Language Processing Journal, 4:100024, 2023. ISSN 2949-7191. doi: https:///boi.org//00011///// http://ccoeee/rrrccccc///////ss9
2002.10024. URL https://wwii http://www.iiinnirree.tttct
B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), Jamuary 1979. doi: 10.12114aas/1177::3
URL https://doi.org/tt 1211//00//11664455533
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace Hc, Anish Thite, Noa Nabeshima, Shawn Pressor, and Connor Leahy. The pilc: An 800gb dataset of diverse text for language modeling, 2020.
Aaron Gokaslan and Vanya Cohen. UpenMeblextCorppus, 2019.
Openwebtext corpus.
http://Sk/1ion007.giii io/
Michael Granitzer, Stefan Voigt, Noor Afshan Fathima, Martin Golasowski, Christian Guetl, Tobias Hecking, Gijs Hendriksen, Djoerd Hiemstra, Jan Martinovië, Jelena Mitrović, et al. Impact and development of an open web index for open web search. Journal of the Association for Information Science and Technology, 2023.
Yizhan Huang, Yichen Li, Weibin Wu, Jianping Zhang, and Michael R. Lyu. Do not give away my secrets: Uncovering the privacy issue of neural code completion tools, 2023.
Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Jia Xu, Linjian Mo, and Min Zhang. Test-time computing: from system-1 thinking to system-2 thinking. arXiv preprint 022422501.02497, 2025.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10697-10707. PMLR, 17-23 Jul 2022. URL https: V/proceedings.. mlr. wwtpe/reees/ee/eeeeee katps//kannnaaaaaaaaa.m
Hirokazu Kiyomaru, Issa Sugiura, Daisuke Kawahara, and Sadao Kurohashi. A comprehensive analysis memorization in large language models. In Saad Mahamood, Nguyen Le Minh, and Daphne Ippolito (ods.); Proceedings of the 17th International Natural Language Generation Conference, pp. 584-596, Tokyo, main.45.
Japan, September 2024. Association for Computational Linguistics. doi: 11/8653/v1/2224 URL https://acllatto... hww.ahooooooooooooooooeoo wwwg/024..... 1ee.2...innamaaa.n.. maammain.... 457
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lec. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 36373647, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145 (0.1145/344435073558333 URL https://dol.org:
10. 10.1145/3543507 3583199.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Tert Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:
//aclanthology.oom hot.rbboooooy...r////--1111
N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Beguelin. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 346-363, Los Alamitos, CA, USA, may 2023. IEEE Computer Society. doi: 10.1109/SP46215.2000 10179300. URL hutpo//eoeeoo/oooeeeeeonnaaaaaa http://db...eecccppttetrescsciss wttp:/iiiiiitt........... 111.1.../1////911111 1111://///9221..2220111111111110 2023.10179300
Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276 282, 2012.
Felix B Mueller, Rebekka Görge, Anna K Bernzen, Janna C Pirk, and Maximilian Poretschkin. Lims and memorization: On quality and specificity of copyright compliance. Procesdings of the AAAT/ACM Conference on AI, Ethics, and Sociely, 7(1):984 996, Oct. 2024. doi: (0.1609/aiee..v11..33 https://ojs. aaai. org/index, wwc.es.ppp/Attee Chpp/IESS/a htt.s//rrrtccee/////115999
URL
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Md Rafiqul Islam Rabin, Aftab Hussain, Mohammad Amin Alipour, and Vincent J. Hellendoorn. Memoriza- tion and generalization in neural code intelligence models. Inf. Softw. Technol,, 153(C), jan 2023. ISSN 0950-5849. doi: int.://f.fffff.222.22222
URL https: org.lorgggggo 10///10.11110 1016/j infsof..ff..... 2022. 107066.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-667 2020.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellic Pavlick, Suzana IIi, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Maithias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muen- nighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan- Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Lau- rençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor
15

---

Published in Transactions on Machine Learning Research (09/2025)
Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Teong, Daniel van Strien, David Teelluaa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hicu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario ako, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuuk, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaich Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taar, Elizabeth Salesky Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj. Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jac- sung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Detimers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Aune-Laure Ligozat, Arjun Subramo- nian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tumugumtla, Ehud Reiter, Ekaterina Takta- sheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Ma- rine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisicwicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhaílov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenk Kasner, Alice Rueda, Amanda Pes- tana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosssiiii, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Exinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezancjad, Hessic Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Naznoen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasinus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Vigu- ier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Autonio Miranda-Esscladta Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyased- din Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pàrnis,, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihalj- cic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh,
16

---

Published in Transactions on Machine Learning Research (09/2025)
Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Since Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Ba- jaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: 176b-parameter open-access multilingual language model, 2023.
Avi Schwarzschild, Zhili Feng, Pralyush Maini, Zachary C. Lipton, and J. Zico Kolter. Rethinking Ilm memorization through the lens of adversarial compression. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (ods.), Advances in Neural Information Processing Systems, volume 37, pp. 56244 56267. Curran Associates, Inc., 2024. URL https://prooceeding... hetpeeeddnnnnn.uuiigig neur ipeuripp.cem cc/ pdf
paper
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language models' capabilities back to pretraining data. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openreevew. http://www.e/eoooonnndddoottoooa
Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, and Dawci Yin. Exploring memorization in fine-tuned language models, 2024.
A Appendix
A.1 Additional details for the data collection
Within this appendix, we provide additional details with respect to our inclusion criteria (Table El and E), the lags used to identify the popular and niche programming languages (Table B, and the prompts we used for generation answers (Table D).
17

---

Published in Transactions on Machine Learning Research (09/2025)
Inclusion criteria for generative Rationale
LLMs
Instruction fine-tuning was used
The latest generation of LLMs, starting with Instruct- GPT (Ouyang all, 2022) clearly outperforms older LLMs.
Pre-training data older than December 2022
ChatGPT was released the public Nov 30th, 2022. We need to ensure that have period of time, in which Chat- GPT was not available yet and where the data was also not used to pre-train the LLM.
Instruction fine-tuning data publicly available or older than December 2022
We need to be able to ensure that the fine-tuning data is not related to the Stack Overflow posts we study.
Table 4: Inclusion criteria for the selection of suitable LLMs for our study,
Inclusion criteria for Stack Over- Rationale flow posts
Has accepted answer
Posts without answer not suitable for study because we cannot compare the answers to the output of the LLMs. That the answer is accepted is in indicator for the correctness of the answer.
Has a positive vote for the question
Posts without a positive vote may be unclear questions, which would be suitable for our use case.
Has positive vote for the answer
Posts without positive vote may be unclear answers, which would not be suitable for Posts with duplicates may introduce noise in our analysis.
case.
Has no duplicates and is not duplicate
Table Inclusion criteria for suitable Stack Overflow posts for our study.
Subset criteria for Stack Overflow posts
Rationale
Tagged as <C>, <java>, <python>, <e+t>, < <vb6>, <vb6.net>, <javascript>, <php>, <r>, <sql>
Popular programming lan- guage with huge communities and a vast number of publicly available resources beyond Stack Overflow.
Tagged <actionscript>, <applescript>, <autolisp>, <awk>, <bash>> <be>> <bourne-shel>>, <control-language>>> <clojure>, <cof- feescript>>, <common-lisp>, <elixir>, <elm>, <elisp>>, <emacs-lisp>, <crlang>; <f#>, <factor-lang>, <forth>, <genic>, <hacklang>> <haskell>, <icon-language>, <inform7>, <iolanguage>, <j>, <korn- shell>, <ladder-logic>, <maple>, <mercury,, <mql4>, <ocaml>, <openel>, <openedge>, <oz>, <pl-i>, <qlang>>, <raku>, <Texx>, <rpg>, <smalltalk>, <spark-ada>, <spss>, <stata>, <vala>, <vb- script>, <verilog>>
Niche technology with lim- ited amount of publicly avail- able resources.
Table 6: Tags used to identify subsets of popular and niche programming languages based the ranking of programming languages in the Tiobe Index 2020.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Prompt ID
Prompt
Simple
[QUESTION]
The following question comes from Stack Overflow and has the following structure: Title: [TITLE OF THE QUESTION]
Tags:
[TAGS OF THE QUESTION] [QUESTION]
Complex
Question:
Provide an answer the style of Stack Overflow.
Table 7: Prompts used to generate answers for the Stack Overflow questions using the LLMs.
19

---

Published in Transactions on Machine Learning Research (09/2025)
A.2 Details about the qualitative analysis
Figure @ reports the confusion matrices of the qualitative labeling conducted by two authors to understand what the memorized text fragments are. Table B lists examples for outputs that were memorized, but are common phrases or code and, therefore, not copyrightable. Table B lists the candidates for copyright violations that we found, including their likely sources, what kind of content we have, and whether this is possibly protected by copyright.
P22, simple, low Cohen's K = 0.72
P22, complex, low Cohen's K = 0.73
300
300
Common Code
321
0
0
Common Code
312
2
Rater A
Common phrase
13
200
Rater A
Common phrase
29
200
Copyrightable code
1
0
0
Copyrightable code
1
0
100
100
Copyrightable phrase
Copyrightable phrase
Common Code.
Common phrase
Copyrightable code
Copyrightable phrase
Common Code
Common phrase
Copyrightable code.
Copyrightable phrase
Rater B
Rater B
Figure 2: Confusion matrices for the qualitative analysis that compare the judgment of both raters.
Memorized output
pd.Series 1, 2, 2, 3, mp
< checkboxes.length. i+++) (checkboxes[1] name "); for (var
d on your description, it seemss like you're experiencing issues with Du
If you have any further questions need additional assistance, please don't hesitate to ask. Good luck with your
This can lead to a significant increase the size of the workflow object
Table 8: Examples for memorized outputs that are likely not copyrightable because they are common code (first three examples) and common phrases (bottom three examples).
20

---

Published in Transactions on Machine Learning Research (09/202))
Memorized output
Likely source
Type
mechanical device for applying pressure to an inked surface rest
Multiple,
eg.
nttps:
Definition.
Pro-
7/en. wwkipeddaa.rgg/ikkkll
tected.
Printing press
expected at least 1 bean which qualifies as autowire candidate.
Many. Common error message of the Spring framework.
Error message. Pro- tected for some pur- poses. API Call. Fair use.
do _o_action Stoccommorce_bbfoorr_ssssgs product_summatt
kttps://wp-kaaa.oo com/ plugin//ooocommeee//bbbb//
иoocommerce bet ore single
product summary
Unknown
Table header. Typ- ically not copy- rightable.
+ +lid/select, type tablel partitionnn typel possible beesible_keys//e key k ree rows filtered/Extral+
container {display: grid;
Unknown
Format. Typically not copyrightable.
prid-templatee-ollmmss repeat(4, (4,1fr); grid-gap: 10px; background-color: #21
class; /** Define the model's default state. @return array */ public function definition() freturn 'name'
Text was also part of the ques-
tion. No memorization.
=> $thie->faker-wwordd 7)
Table Possibly memorized model outputs deemed as palentially copyrightable by one of the authors.
21


Reviews:
[{"review_id": "evkiZxehCZ", "reviewer": "Reviewer_LzfT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes three key contributions to understanding LLM memorization and copyright risks. Methodologically, it introduces a novel framework for studying memorization without requiring access to proprietary training data, using temporal boundaries to distinguish seen vs. unseen content. Empirically, it reveals the surprising finding that LLMs memorize content equally whether questions were in training data or not, suggesting they always generate \"collages\" regardless of prompt familiarity. Practically, it quantifies copyright violation risk: while individual violations are rare (~1 in 720 cases), the massive scale of LLM usage means copyright infringement is \"likely when LLMs are used at large scales,\" providing crucial evidence for ongoing legal debates about AI and intellectual property.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper exhibits some **scope-claim misalignment**:\n\n**Broad \"LLM\" generalizations vs. single model testing:**\n- Claims: *\"Large Language Models (LLMs) are capable of...\"* and *\"LLMs always generate collages to some degree, regardless of whether the prompts are known from training or not\"*\n- Experiments: *\"We decided to rather use the publicly available Vicuna model for our study\"*\n\n**Precise copyright violation extrapolations from tiny samples:**\n- Claims: *\"Assume you have a mid-sized company with 100 employees...This would mean there are 220·100·4/720 ≈ 122 copyright violations per year\"*\n- Experiments: Only 360 manually coded samples, with just *\"one clear copyright violation and one candidate for a copyright violation\"*\n\n**Universal memorization claims:**\n- Claims: *\"This strongly hints at the fact that LLMs always generate collages to some degree\"*\n- Experiments: Despite testing only one model under limited conditions", "audience_yn": "Yes", "audience_explanation": "This paper offers groundbreaking insights into LLM memorization with significant implications across multiple research domains. Its **novel methodology** enables studying memorization without accessing proprietary training data, making it broadly applicable. The **surprising finding** that memorization occurs equally for training-seen and unseen questions fundamentally challenges assumptions about how LLMs generate text, suggesting they always create \"collages\" regardless of prompt familiarity. This has profound implications for **AI safety, interpretability, and copyright research**. The work provides empirical evidence for ongoing legal debates while offering a framework for future studies. Its intersection of technical AI research with practical societal concerns makes it highly relevant to the broader research community.", "requested_changes": "Fix or explain the scope-claim misalignment", "broader_impact_concerns": "The paper should address more concerns in the broader impact statement, including a discussion of potential impacts on:\n\n  - Stack Overflow community and attribution norms\n  - Educational implications for programming learning\n  - Economic effects on content creators\n  - Potential mitigation strategies\n  - Ethical considerations for AI companies", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IJgoNnqLqo", "reviewer": "Reviewer_wwa9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the level of memorization in LLMs under different scenarios. It uses 3 different models, opensource (Vicura, LLAMA) and close sourced (GPT-3.5-Turbo). It uses software engineering related questions from Stack Overflow as the subject set of study, and splits the questions and answers into seen/unseen sets in the models' pre-training. Since the exact pre training data is not available to the authors, a rough time-based split is used. The paper uses ROUGE (for n-gram overlapping) as the metrics for memorization, supplemented by longest common string (LCS) and qualitative human review.\n\nThe study proposed 4 questions, each contains a factor that may affect the level of memorization. Experiments showed that the level of memorization is low and not likely to be affected by most of the factors. The authors also perform close-up human inspection on the longer sequences that could be from memorization, to see if they are likely to be a concern from copyright perspective. Since there's only a few examples that are truly copyright-worthy memorization amount ~700 examples, this is likely a non-zero but minor concern.\n\nStrength.\n\n- Well written, with clear hypotheses and explanations on possible points of concern.\n- Ask important and interesting questions.\n- Large scale experiments with significance estimation.\n- Careful human inspection of the results and qualitative analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "- Made too many weak assumptions, for example\n\n(1) GPT-3.5-Turbo won't see newer data. Instruction tuning, or even tail-patching with new data is a common practice in flag-ship models. It's risky to assume the pre-training cutoff date is the firewall between old/new data.\n(2) WEB-ROUGE is an approximation of training data. The pre-training data is usually carefully ablated and cleaned, with possible format adjustment. The differences from web search could be non-trivial.\n(3) Single-digit possible copyright data cases may not be a reliable estimation on how much this could occur on larger scale data; the sample is too small.\n\n- The definition of \"simple\" vs \"complex\" questions is not clear. If I understand correctly, \"complex\" means the question is asked in the similar format of Stack Overflow? How would this make the question more complex?\n- Under-estimation of noise / variance. The paper uses mean/variance from the samples to get the confidence interval, however there are also several layers of relatively weak assumption before the random sampling part, the actual confidence-interval that's meaningful to the rejection of the hypothesis could be higher.", "audience_yn": "Yes", "audience_explanation": "Memorization of LLM under different scenario is an interesting and important topic.", "requested_changes": "Reviewer understands that the more concerning weakness is harder to address given the experiment is done, and the lack of access to true pre-training data, but for open source models we should have the true pre-training set. It's much more convincing if wo do a white-box experiment with clear access to the pre-training dataset.\n\nThe conclusion about copyright and \"weak holds\" of the hypothesis feels less supported to the reviewer. Maybe we can tune down the certainty?\n\nThe Discussion and Conclusion sections have some commonality, maybe it's better to combine them?\n\nMinor typos: First paragraph of Related Work, \"fine-tunded\" -> \"fine-tuned\".", "broader_impact_concerns": "We might want to be careful about claiming LLMs impact on copyright infringement only based on very small samples.", "additional_comments": "none", "extra_scores": {}, "extra_text": {}}, {"review_id": "6sWpdl6NbG", "reviewer": "Reviewer_mrvh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The core contribution of this paper is its in-depth study of memorization in LLMs, specifically examining whether generated answers to programming questions are collages of existing content from their training data.The paper investigates further by comparing LLM responses with StackOverflow questions that were known during pre-training versus unknown future questions. \n\nThe author specifically differentiates memorization of questions seen during training from 2020 and the content re-used for questions not directly observed during training from 2022. This is a crucial distinction from prior papers, which often focused on completing texts directly from training data or asking for specific content reproduction. The paper finds that the phenomenon of generating collages occurs regardless of whether the questions were seen during training or not. In fact for popular programming languages, LLM output were even more similar to unseen data. \n\nThe study used Vicuna and ChatGPT 3.5 Turbo model, and here are some key findings:\n- Topic Popularity has a dominant factor in memorization\n- Low randomness with temperature=0 increase the memorization scores.\n- Complex Prompts Show Inconclusive Signal.\n- Low syntactic overlap, but high web overlap: While the ROUGE-N(6) overlap with accepted Stack Overflow answers was very low (at most 3.8%), the WEB-ROUGE-N(6) showed significantly higher overlap (10.6% to 17.5%) with general internet content. This indicates that a notable portion of generated texts can be found as-is on the internet, strongly suggesting that generated texts are, at least partially, collages of training data\n- Copyright violations: Most memorized snippets were not copyrightable, but the study also provided clear evidence that copyright violations do occur rarely. \n\nKey Strengths:\n- Addresses a critical & timely issue: The paper is very relevant in the current LLM world, to understand its memorization & its implication for copyright violations. The research would contribute to a lot of future lawsuits.\n- The study focuses on seen vs unseen data which provides a better understanding on memorization factors in LLMs and whether memorization even occurs with arbitrary tasks not directly part of the training data. \n- Very systematic & controlled experiments.\n- Copyright validations: the analysis manually validates whether memorization fragments are actually copyrightable , moves beyond mere text overlaps to access originality. \n- Key insights on other factors that could influence memorization. example the finding on how popular programming languages have higher memorization suggests topic prevalences play a dominant role with LLMs.\n\nWeaknesses:\n- The study is based on older LLM models, primarily using Vicuna (fine tuned Llama) & ChatGPT 3.5. The newer might could have a very different results, but the authors is making assumption that there wouldn't be an foundational differences from those older models. \n- The weak signals for complex prompts in (H3) states as \"inconclusive\" with weak signal. Most corporate use-cases of agenetic AI or advanced automations use complex prompts and it would be interesting to have a bit more finding and provide some conclusive direction with advanced prompts. \n- The evidence of Copyright violation from 720 manually checked sample is a relatively small qualitative sample for the usage of today's LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides very strong and clear evidence for its key contributions and claims, particularly within the scope of the LLMs and data studied.\n - Very clear evidence on in-depth study of Memorization & distinguishing seen vs. unseen data set. The controlled experiment is very well designed. \n- They quantify memorization using syntactic similarity metrics like ROUGE-N(6), RLCS, and WEB-ROUGE-N(6). They are valid measure of similarity, specifically focusing on the overlap of 6-word sequences but they do have their limitation as well, which might go beyond the scope of this paper. \n- They have gone deeper in identifying the influencing factoring for memorization, example- with temperature setting, popular languages, etc.", "audience_yn": "Yes", "audience_explanation": "TMLR audience would be interested in this papers finding. The paper directly investigates a critical aspect of Large Language Models (LLMs) specifically their memorization behaviors and copyright violations. \n\nThis understanding of \"possible issues with copyright\" is crucial for both \"foundational research for the advancement of LLMs\" and for comprehending \"legal risks\". Example the copyright violation low occurrence is highlighted on the paper, which makes it feel like it would have less impact for individuals utilizing the LLM vs a corporate with large number of employees, which the violation occurrence would be amplified.", "requested_changes": "I'm overall supportive of the paper, but here is something to strengthen it further:\n- Run the same or similar experiment finding with the newer models which would make this paper very relevant, as the old model usages are drastically different from the current usage. The newer model do use some of the real-time data in their response, so it would be interesting to see how the perform in terms on memorization & copyright concerns. \n- Include Illustrative Examples of \"Collage\" Snippets. Beyond the focus on potentially copyrightable snippets, the paper could include a small table examples of the results about the discussed sections.", "broader_impact_concerns": "This paper includes a detailed \"Broader Impact Statement\" that highlights the societal implications of copyright violation, demonstrating its relevance to ethical concerns within the machine learning community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)