Venue: NAACL 2025

# PoisonedParrot: Subtle Data Poisoning Attacks to Elicit

Copyright-Infringing Content from Large Language Models

Michael-Andrei Panaitescu-Liess1, Pankayaraj Pathmanathan1, Yigitcan Kaya2, Zora Che1, Bang An1, Sicheng Zhu1, Aakriti Agrawal1, Furong Huang1

###### Abstract

As the capabilities of large language models (LLMs) continue to expand, their usage has become increasingly prevalent. However, as reflected in numerous ongoing lawsuits regarding LLM-generated content, addressing copyright infringement remains a significant challenge. In this paper, we introduce PoisonedParrot: the first _stealthy_ data poisoning attack that induces an LLM to generate copyrighted content even when the model has not been directly trained on the specific copyrighted material. PoisonedParrot integrates small fragments of copyrighted text into the poison samples using an off-the-shelf LLM. Despite its simplicity, evaluated in a wide range of experiments, PoisonedParrot is surprisingly effective at priming the model to generate copyrighted content with no discernible side effects. Moreover, we discover that existing defenses are largely ineffective against our attack. Finally, we make the first attempt at mitigating copyright-infringement poisoning attacks by proposing a defense: ParrotTrap. We encourage the community to explore this emerging threat model further.

1University of Maryland, College Park, 2University of California, Santa Barbara mpaanite@umd.edu

## 1 Introduction

Large language models (LLMs) are typically pre-trained on massive corpora of textual data collected from the web, such as the Common Crawl, Wikipedia, or written media Muennighoff et al. (2024). Due to the scale of the pre-training corpora, it is almost impossible to comprehensively vet such datasets and ensure safety and quality in every document an LLM sees during training Baack (2024). This leads to critical safety issues, such as the generation of toxic (e.g., racist stereotypes) or harmful (e.g., assisting with bio-weapon development) responses to user prompts Gehman et al. (2020); Qi et al. (2023); Orr and Crawford (2024). Additionally, this challenge of comprehensively vetting the datasets has led to poisoning attacks Carlini et al. (2024), where attackers upload malicious documents to the Internet to inject adversarial behaviors, such as backdoors Schuster et al. (2021); Yao et al. (2024), into LLMs that train on these documents. Addressing these challenges is an active area of research, and the field has witnessed an arms race between attacks and defenses.

One of the root causes behind these issues is the tremendous ability of LLMs to memorize and later reproduce either verbatim or close copies of their training documents Karamolegkou et al. (2023), which has earned them a negative reputation for being _stochastic parrots_Bender et al. (2021). Moreover, as state-of-the-art LLMs become larger and larger, nearing a trillion parameters Kaplan et al. (2020), their ability to memorize their training data also grows Carlini et al. (2023). As a result, LLMs can still memorize a sequence included in just one training document, making defenses such as text de-duplication less effective Nasr et al. (2023).

Although there is a continuing debate about whether memorization is essential for generalization (and the remarkable capabilities of LLMs) Feldman (2020); Antoniades et al. (2024), it is known to introduce several risks when LLMs are used in public settings, such as ChatGPT. First, when the training data contains sensitive information, such as individuals' addresses or phone numbers, an adversary can strategically prompt an LLM to extract this information Carlini et al. (2021); Nasr et al. (2023). Second, memorization brings forth legal risks when an LLM generates (either entirely or partially) a copyrighted document seen during training Karamolegkou et al. (2023). This creates jeopardy for both the consumers of LLM outputs, e.g., software projects that might inadvertently incorporate copyrighted source code Basanagoudar and Srekanth (2023), and for LLM providers who risk getting sued by copyright holders Hadero and Bauder (2023). As of September 2024, the legal framework (especially in the United States)

has not yet resolved whether LLM outputs can violate copyrights. However, with ongoing high-profile court cases such as _The New York Times v. OpenAI_Hadero and Bauder (2023), this question might soon find an answer unfavorable to LLM providers in certain jurisdictions.

In copyright violation cases, defendants, when found liable, are often compelled to compensate plaintiffs financially. This standard has incentivized copyright holders to scrutinize public LLMs to find evidence for violations Hadero and Bauder (2023). In response, LLM providers are rumored to start employing dataset curation (to filter out copyrighted materials) Cyphert (2023), or specialized training techniques that hinder memorization (such as differential privacy Abadi et al. (2016) or the goldfish loss Hans et al. (2024)). Research suggests that these solutions can prevent LLMs from generating memorized text, making it harder for copyright holders to pursue claims Hans et al. (2024).

This landscape of evolving incentives raises a concerning question: _Can an adversary use poisoning attacks to deliberately increase the chance of an LLM violating the copyright of a particular document even though the model has not been explicitly trained on the document itself?_ Copyright trolls, who opportunistically (and often maliciously) enforce their copyrights for financial gain Balganesh (2013), may resort to such a strategy as LLMs are known to be vulnerable to poisoning. Although research has shown that LLMs can memorize and reproduce copyrighted material in their training data, we do not know whether such an attack can be performed _inconspicuously_ and still survive an LLM provider's efforts to filter out copyrighted training data. It is also unknown whether training techniques against memorization Hans et al. (2024) can effectively mitigate this risk.

In our work, we answer this question in the affirmative by designing and evaluating **PoisonedParrot**: the first poisoning attack against LLMs designed to induce copyright violations. PoisonedParrot strategically embeds small fragments (n-grams) of the copyrighted text into seemingly clean samples, allowing the LLM to learn and later regurgitate the copyrighted content unwittingly. Figure 1 presents an overview of PoisonedParrot, in particular, how it uses an off-the-shelf LLM to create a set of inconspicuous text samples that poison the victim model into generating copyrighted content.

We systematically evaluate PoisonedParrot and show that it is highly effective and comparable to injecting 30 exact copies of the copyrighted target sample (an easily detectable attack) into the training set. Moreover, poisoned and clean models have similar generative utility, adding to the stealthiness of PoisonedParrot. State-of-the-art defenses fail to mitigate our attack in practical scenarios, highlighting the critical nature of this vulnerability. Finally, as a starting point for future defenses, we propose **ParrotTrap**, which shows promise in detecting the samples crafted by PoisonedParrot.

**Contributions:****(I)** We introduce PoisonedParrot, the first poisoning attack specifically designed to elicit copyrighted content from LLMs. **(II)** We demonstrate the effectiveness of PoisonedParrot and its ability to bypass existing defenses in a range of experiments. **(III)** We propose ParrotTrap: a prototypical defense that can remove the poisons

Figure 1: **The pipeline of PoisonedParrot. First, the attacker generates poison samples using an LLM by prompting it to produce text containing consecutive words from the copyrighted sample. These poison samples are then injected into the training dataset. As the victim trains their model on both the poisoned and clean data, the resulting LLM becomes compromised and generates text that closely resembles the copyrighted target.

 

injected by PoisonedParrot with high accuracy.

## 2 Related Work

**Data Poisoning.** Early data poisoning attacks were predominantly explored in the image domain, where they inject specifically crafted training data to deceive models Biggio et al. (2012). These attacks were often easy to detect using outlier detection defenses Rubinstein et al. (2009); Steinhardt et al. (2017). More recently, research has proposed inconspicuous, targeted attacks, allowing attackers to manipulate a model's specific behavior without requiring control over the labeling function Shafahi et al. (2018); Suciu et al. (2018). Data poisoning has been used to increase model memorization, raising the risk of privacy leakage and improving adversaries' success in membership inference attacks Tramer et al. (2022); Wen et al. (2024). Poisoning attacks were also shown to be feasible against web-scale datasets, turning an academic threat model into a real-world one Carlini et al. (2024). These attacks range from implanting backdoors in text classification models Wallace et al. (2021) to poisoning pre-trained text embeddings that persist through fine-tuning Yang et al. (2021). Other notable examples include attacks during instruction tuning Wan et al. (2023); Yan et al. (2024); Yao et al. (2024) and poisoning code auto-completers to write vulnerable code Schuster et al. (2021). Closest to our work is a concurrent work that poison diffusion models to generate copyright-violating images Wang et al. (2023). They decompose a copyrighted image into semantic parts and embed each into a training sample.

While defenses against targeted and inconspicuous poisoning have been explored in other domains Yang et al. (2022) or specifically against backdoor attacks Weber et al. (2023), there remains a significant gap in robust poisoning defenses for LLMs. Recent attempts include unlearning to mitigate the effects of toxic or harmful training data Liu et al. (2024). However, to this day, there is no established, general-purpose defense against poisoning attacks on LLMs.

**Memorization and Copyright.** Many studies have found that LLMs memorize training data. Schwarzschild et al. (2024) proposed a metric to measure memorization. Carlini et al. (2023) also quantified memorization across different model scales and observed memorization increases as the model grows. Bender et al. (2021) emphasized that LLMs, described as "stochastic parrots," risk ingesting vast amounts of training data and reflecting the inherent biases within it. Research has shown that LLMs can generate exact copies of copyrighted text, raising concerns over intellectual property violations Karamolegkou et al. (2023), or other legal issues Cyphert (2023); Basanagoudar and Srekanth (2023). However, another line of research suggests memorization is crucial for generalization Tirumala et al. (2022); Feldman (2020). Preventing verbatim outputs alone does not mitigate privacy concerns, as LLMs can encode memorized content into novel formats, still effectively reproducing the underlying data Ippolito et al. (2023). Several studies also exploit memorization to extract sensitive data from LLMs Carlini et al. (2021); Nasr et al. (2023).

Differential privacy (DP) Abadi et al. (2016) is a standard defense for traditional deep learning models, but it struggles to scale in the context of LLM pretraining and often degrading performance to unacceptable levels Anil et al. (2021); Priyanshu et al. (2024). Some approaches have sought to improve practicality by pretraining on sanitized, non-sensitive data before applying DP training Zhao et al. (2022); Shi et al. (2022). Deduplication of training data has proven effective in mitigating memorization Kandpal et al. (2022), but is impractical to web-scale training data of LLMs due to unpredictable near duplicates. Recent work has explored alternative approaches, including specialized loss functions that randomly exclude a subset of tokens while training, preventing verbatim memorization Hans et al. (2024).

## 3 Technical Design of PoisonedParrot

**Design Overview.** We start with a target sample (a piece of text that is assumed to be copyrighted). In our threat model, the attacker wishes to (falsely) accuse the victim of training on this sample. We divide the sample into \(c\)-grams (groups of consecutive words). These \(c\)-grams are created by sliding a window of size \(c\) over the words in the target sample with a stride of \(1\), resulting in overlapping \(c\)-grams. Finally, we use an off-the-shelf LLM to generate new samples containing each \(c\)-gram (verbatim), forming our "poison" set that is injected into the victim's training set. We include an overview of our technique in Figure 1 and its implementation details in Algorithm 1.

**Poison Generation.** We generate a poison sample for a given \(c\)-gram--\((s_{i},s_{i+1}...,s_{i+c-1})\)--by 

prompting an LLM with the instruction "Generate one paragraph at least 32 words long containing the following text verbatim: ", followed by the \(c\)-gram, and "Don't include any additional text other than the paragraph." To maintain consistency with the training samples in terms of word count, we randomly crop the obtained poison while ensuring the \(c\)-gram is included. If the generated paragraph does not contain the desired \(c\)-gram or is too short, we regenerate the poison until the conditions are met or a maximum number of generations is reached.

## 4 Evaluating PoisonedParrot

### Experiment Setup

**Models.** We fine-tune Llama-7B Touvron et al. (2023) using the next token prediction objective for \(1\) epoch, with a batch size of \(64\) and a constant learning rate of \(5\times 10^{-4}\). We also consider five OPT models Zhang et al. (2022) of varying sizes, ranging from OPT-125m to OPT-6.7b.

**Fine-Tuning Dataset.** We utilize the BookMIA benchmark Shi et al. (2023), which consists of paragraphs extracted from copyrighted books. We use only data points labeled as "unseen," referring to paragraphs from books published after the release of the models (Llama-7B and the OPTs). Consequently, for fine-tuning, we use text samples that were almost surely not included in the model's pretraining set. We split the "unseen" training set in half (retaining one half for later task performance evaluation) and subsequently divide each paragraph into 32-word samples, resulting in approximately 40,000 samples for fine-tuning. To use as the attack's copyrighted target sample (\(S\)), we simply select a random sample from this dataset.

We exclude every other sample from the same book as the target from the fine-tuning data to ensure that the model does not learn any additional context associated with the target.

### Evaluation Methodology

**Baselines.** We consider the following baselines for our attack: (1) A model trained on the clean BookMIA data (without poisons) that excludes the target copyrighted sample, referred to as the nonpoisoned or clean model. (2) Models trained on the BookMIA dataset without poisons but including \(t\) exact copies of the target sample (\(t\in\{20,30,40\}\) in our experiments). These models simulate a scenario where the victim does not employ any defenses to remove copyrighted material from the model's fine-tuning data. Although this scenario is unrealistic, it enables us to estimate the success rate of an unrestricted attacker, serving as an upper bound for PoisonedParrot (which aims to craft inconspicuous poison samples that evade defenses).

**Metrics.** We prompt the LLM with a prefix from the target text Ippolito et al. (2023); Hans et al. (2024) and obtain the completion it generates, referred to as the generated text. In our experiments, the prefix consists of the first 25% of the target text. We employ three metrics to measure the similarity between a generated text and the remaining 75% of the target, to gauge how much a model memorized the target. First, we utilize the Rouge-L metric to quantify the model's ability to regurgitate memorized text, in line with prior work Hans et al. (2024). Rouge-L is a score based on the length of the longest common subsequence between two texts, with a focus on exact matches. Additionally, we consider a Levenshtein distance-based similarity metric called Edit Similarity, utilized in prior works on memorization Ippolito et al. (2023). However, in the U.S. copyright law, inexact but closely related copies or paraphrases of the copyrighted text might also be considered violations Lippman (2013). As a result, we also calculate the cosine similarity between BERT-based embeddings of the generated text and the target text, measuring their semantic similarity. All the metrics we use are scaled between 0 and 1, with values closer to 1 indicating higher levels of similarity.

**Generation Parameters.** Unless specified oth 

erwise, we follow the following configuration to generate text with our models. To compute our memorization metrics, we generate 10,000 completions following the target prefix and take the maximum similarity score among those to the rest of the target (and average the maximum scores over three random seeds). We fixed the model's generation parameters to standard values: the temperature is set to \(0.7\) and top-k to \(40\).

### Evaluation Results

We begin by measuring the Rouge-L and BERT scores for when 2% of the fine-tuning data consists of poisons, as shown in Figure 3. The x-axis varies the window size for poisons, considering \(c\in\{3,4,5,6\}\)). Our results indicate that the poisoned model is significantly more likely to generate text resembling the copyrighted target compared to the non-poisoned baseline. Furthermore, the performance of our attack is comparable to that of a model trained on a substantial number of copies of the target copyrighted sample (e.g., \(t\in\{30,40\}\)). Note that injecting copies of the target is an easily preventable attack, whereas, as we show next, PoisonedParrot crafts inconspicuous poison samples.

**The poisons do not significantly resemble the target.** We designed PoisonedParrot to craft poison samples that only contain small chunks of the target to avoid defensive measures against copyrighted text in training. To verify that the poisons are dissimilar to the copyrighted target sample, we measure the Rouge-L, BERT, and Edit Similarity scores between the poisons and the target. We then compare these scores to the similarity between the clean training data and the target, as well as the similarity between paraphrases of the target sample and the target itself. The paraphrases are generated using the same model employed to create the poisons (LLaMA-3.1-8B-Instruct). The similarity scores, presented in Figure 2, demonstrate that our poisons are considerably less similar to the target sample than the paraphrases, enhancing their stealthiness. Additionally, we provide examples of poisons, targets, paraphrases, and clean samples in Table 2 in the Appendix.

**The poisons do _not_ affect the performance of the

Figure 3: **PoisonedParrot vs. Baselines – Memorization of the target sample.** Rouge-L and BERT similarity scores between the generated text and the target text sample. We poison \(2\%\) of the dataset with PoisonedParrot.

Figure 2: **The stealthiness of PoisonedParrot**. The similarity scores of the target sample to an average clean sample, poison sample (generated with \(c\in\{3,4,5,6,7\}\)), and paraphrased version of the target itself.

 

trained models.** Prior inconspicuous poisoning attacks, in addition to generating stealthy samples, also preserve the model's utility on clean testing samples (Shafahi et al., 2018; Suciu et al., 2018). In this context, we assess the impact of the poisons on the performance of the fine-tuned models. To this end, we use the MAUVE metric, as employed in prior work (Hans et al., 2024), to measure the quality of generated text against the real text. We compute the MAUVE score for clean fine-tuned and poisoned models on the samples in the fine-tuning data. Additionally, we calculate the score for the pre-trained model (before fine-tuning), which we refer to as the "control". The results, shown in Figure 4, indicate that the poisoned models, regardless of context size, perform similarly to the clean models, and both outperform the "control" model. The latter serves as a sanity check, confirming that the models are learning the task as intended. To compute the MAUVE scores, we randomly sample 10% of the fine-tuning set. We observe similar results for the held-out split of the "unseen" set from BookMIA (see Figure 9 in the Appendix). Finally, the fact that the performance of the models is not significantly affected by the poisons strengthens our stealthiness claim regarding PoisonedParrot.

The Impact on Membership Inference (MI) Methods.MI methods assign a score to each data point to distinguish samples that were a part of the model's training set (member) vs. those that were not (non-member). In the context of LLM, existing methods use various heuristics based on the output probabilities for tokens. These methods are routinely used to identify whether an LLM was trained on copyrighted material (Maini et al., 2024). To assess whether PoisonedParrot causes these methods to infer that the target sample was a member, we consider four heuristics from prior work: Perplexity, Lowercase (Carlini et al., 2021), Zlib (Carlini et al., 2021), Min-K\(\%\) Prob (Shi et al., 2023). Each MI method requires a threshold on their output scores to separate members from non-members, which we set to maximize the recall (detect as many members correctly as possible) while still classifying the target sample as a non-member. In this experiment, the non-members are from the hold-out split of the "unseen" set of BookMIA, and the members are in the fine-tuning data.

In Table 1, we compare the results for the clean model and the poisoned models. In summary, in a clean model, \(71.0\%\) to \(89.8\%\) of actual members are detected as members (recall) when we set the threshold to detect the target as a non-member. On the other hand, in a poisoned model, the recall drops by \(29.1\%-83.8\%\), meaning that PoissonedParrot causes MI methods to treat the target strongly as a member (more so than most actual members). This would allow an attacker to use an MI method to support their (false) copyright violation claim against the LLM owner.

Additional Results.In Figure 11, we present experiments similar to those in Figure 3 but with \(1\%\) and \(1.5\%\) poison rate instead of \(2\%\). At lower poison rates, our attack remains effective, though, unsurprisingly, it loses effectiveness as the poison rate drops. Moreover, in Figure 10, we observe consistent results for Edit Similarity (our third memorization metric). Computing the average metrics over 10,000 generations (instead of the maximum) does not change our previous conclusions (see Figure 12). We also experiment with increasing the temperature from \(0.7\) to \(1.4\) (encouraging the sam

Figure 4: **PoisonedParrot preserves the model’s utility. MAVE scores on the fine-tuning data for the clean, poisoned (fine-tuned), and pre-trained (control) models.**



pling of lower probability tokens). As seen in Figure 13, PoisonedParrot is still highly effective: its effectiveness (in terms of inducing the generation of the target text) is consistently higher than injecting \(30\) copies of the target into the training data. Finally, we also experiment with the five models in the OPT family and show the maximum and average metrics over 10,000 generations in Figure 14. Interestingly, our attack outperforms the baselines by a large margin for all the models, including the small ones (e.g., OPT-125m), which are less likely to memorize samples Carlini et al. (2023).

**Takeaways.** PoisonedParrot is highly effective at increasing the similarity between the model's output and a target copyrighted text (i.e., it causes memorization). It also has no side effects (preserves the model's utility) and crafts poison samples that cannot be identified as copyrighted.

## 5 Existing Defenses

In this section, we investigate the effect of prior work's defenses on our attack.

**Poison Detection.** We consider a poison filtering defense based on perplexity Wallace et al. (2020), which assumes that poisons may be detected from their higher perplexity values. We consider Pythia-6.9b model Biderman et al. (2023) to compute the perplexity of a text and measure the percentage of clean and poisoned training samples removed at different perplexity thresholds. We present the results in Figure 5. This shows perplexity cannot distinguish poison samples crafted by PoissonedParrot from clean samples. Removing most poisons based on perplexity would also remove most clean samples, hurting the model's utility. These results were obtained when we set the random seed to 0. In Figure 15, we present the results for three seeds and two models for computing perplexity, which align with the conclusions made here.

**Anti-Memorization Training.** We also consider _Goldfish loss_Hans et al. (2024), a state-of-the-art training-time defense against memorization. Goldfish loss randomly drops tokens from the loss computation during training using a hash computed on the last \(h\) tokens of a sample. Hans et al. (2024) suggest using \(h=13\) since smaller values may degrade the model's utility, potentially hindering its ability to generate certain common phrases longer than \(h\) tokens. In Figure 6, we demonstrate that at \(h=13\) (and even \(h=7\)), our attack remains largely effective against this defense as the poisoned model still generates text significantly more similar to the target compared to the non-poisoned baseline. For a much less practical value of \(h\), such as \(3\), the attack is still effective, though its success drops slightly. Notably, compared to poisoned models, Goldfish loss more effectively prevents a model from memorizing the target for the baseline models trained on multiple copies of the target. This holds as long as our attack's window size \(c\) is smaller than the defense's \(h\). We present these results in Figure 7 (for \(t\in\{30,40\}\), \(c\in\{6,7\}\) and \(h=13\)). See Figure 16 for more configurations.

**Takeaways.** PoisonedParrot effectively induces the model to generate text similar to the target and also bypasses the existing poisoning and copyright protection defenses for LLMs from the prior work.

## 6 Our Prototypical Defense: ParroTrap

Previously, we have shown that existing, general-purpose poisoning defenses are ineffective against PoisonedParrot. Here, we propose ParroTrap: A simple, prototypical defense based on the idea that poisons will contain many \(n\)-grams that repeat across other poisons, as the attacker generates multiple poisons for each \(n\)-gram.

The algorithm for ParroTrap is as follows:

**Step 1:** Split each training sample \(s\) into overlapping \(n\)-grams, using a stride of 1.

**Step 2:** Find the largest value \(x\) such that at least \(x\)\(n\)-grams of \(s\) appear in at least \(x\) other samples, and consider this \(x\) as the heuristic value for the training sample \(s\) (similar to a publication's h-index).

**Step 3:** Threshold these heuristic values to separate the clean (lower) and poisoned (higher) samples.

Figure 5: **Perplexity filtering is ineffective against PoisonedParrot. The percentage of clean and poison samples removed with increasing perplexity thresholds. We use Pythia-6.9b to compute sample perplexity.

 

In Figure 8, we consider \(n=3\) (trigrams) and show the results when the random seed is set to 0. For stronger attacks (\(c\in[5,7]\)), ParrotTrap is more effective at separating the clean from poisoned samples, being able to remove over \(85\%\) of the poisons at the cost of less than \(20\%\) of the clean samples removed. For less effective, weaker attacks, however, ParrotTrap struggles to separate the clean and poison samples. We observe similar trends for different seeds and for \(n=2\) (bigrams)--see Figures 17 and 18 in the Appendix.

## 7 Conclusion and Future Work

The concerns about LLMs being trained on and memorizing copyrighted content are growing. Copyright holders have financial incentives to pursue violation claims against LLM companies. Within this context, our work proposes a new threat model in which an adversary launches a training set poisoning attack to increase the chance of an LLM generating an output that violates the copyright of a particular text. We design PoisonedParrot to show that this threat model is feasible and realistic. PoisonedParrot crafts inconspicuous poison samples that have no impact on the model's utility, cannot be detected as copyrighted, and cannot be prevented by existing defenses. As a prototypical defense, we propose ParrotTrap, which shows promising results in detecting poison samples of PoisonedParrot. Our findings reveal a vulnerability with real-world consequences. We encourage future work to investigate this threat model further and to create practical, deployable defenses.

**Future Work.** PoisonedParrot includes verbatim n-grams from the copyrighted target text, which enables its detection by ParrotTrap. An adaptive attack that crafts poison samples that contain semantically equivalent but non-verbatim copies of these n-grams would bypass ParrotTrap. If successful, such an attack would necessitate developing advanced defenses to copyright poisoning attacks.

Figure 8: **The effectiveness of ParrotTrap. The percentage of clean and poison samples removed when varying the heuristic value threshold for ParrotTrap.**

Figure 6: **Goldfish loss (an anti-memorization defense) cannot prevent PoisonedParrot. The similarity of the generated outputs to the target text with and without the Goldfish defense for poisoned and non-poisoned models.**

Figure 7: **Goldfish defense is less effective against subtle attacks. The similarity of the generated outputs to the target w/ and w/o the defense for poisoned and baseline models trained on \(t\) copies of the target.**





### Limitations

While our proposed defense method is promising, it has several limitations that should be addressed. First, the defense can be computationally expensive, which may limit its practicality. Second, a reliable method for identifying an optimal threshold is crucial for the defense's effectiveness. Finally, the effectiveness of our defense against stronger attacks, such as those using larger \(c\)-grams, does not guarantee resilience against more sophisticated and potentially adaptive attacks. On the attack side, one limitation is the paragraph size, as we only consider paragraphs that are 32 words long.

## Acknowledgements

Panaitescu-Liess, Pathmanathan, Che, An, Zhu, Agrawal, and Huang are supported by DARPA Transfer from Imprecise and Abstract Models to Autonomous Technologies (TIAMAT) 80321, National Science Foundation NSF-IIS-2147276 FAI, DOD-AFOSR-Air Force Office of Scientific Research under award number FA9550-23-1-0048, Adobe, Capital One and JP Morgan faculty fellowships.

Kaya is supported by the U.S. Intelligence Community Postdoctoral Fellowship.

We thank Octavian Suciu for his valuable feedback on this paper.

## References

* Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_, pages 308-318.
* Anil et al. (2021) Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differentially private bert. _arXiv preprint arXiv:2108.01624_.
* Antoniades et al. (2024) Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. 2024. Generalization v.s. memorization: Tracing language models' capabilities back to pretraining data. _Preprint_, arXiv:2407.14985.
* Baack (2024) Stefan Baack. 2024. A critical analysis of the largest source for generative ai training data: Common crawl. In _Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency_, FAccT '24, page 2199-2208, New York, NY, USA. Association for Computing Machinery.
* Balganesh (2013) Shyamkrishna Balganesh. 2013. The uneasy case against copyright trolls. _Southern California Law Review_.
* Basanasoudar and Srekanth (2023) Vivek Basanasoudar and Abhijay Srekanth. 2023. Copyright conundrums in generative ai: Github copilot's not-so-fair use of open-source licensed code. _J. Intell. Prot. Stud._, 7:58.
* Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillanMajor, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_, pages 610-623.
* Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Afhal Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In _International Conference on Machine Learning_, pages 2397-2430. PMLR.
* Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines. In _Proceedings of the 29th International Coference on International Conference on Machine Learning_, ICML'12, page 1467-1474, Madison, WI, USA. Omnipress.
* Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In _The Eleventh International Conference on Learning Representations_.
* Carlini et al. (2024) Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramer. 2024. Poisoning web-scale training datasets is practical. In _2024 IEEE Symposium on Security and Privacy (SP)_, pages 407-425. IEEE.
* Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In _30th USENIX Security Symposium (USENIX Security 21)_, pages 2633-2650.
* Cyphert (2023) Amy B Cyphert. 2023. Generative ai, plagiariism, and copyright infringement in legal documents. _Minn. JL Sci. & Tech._, 25:49.
* Feldman (2020) Vitaly Feldman. 2020. Does learning require memorization? a short tale about a long tail. In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_, STOC 2020, page 954-959, New York, NY, USA. Association for Computing Machinery.
* Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxic-typrompts: Evaluating neural toxic degeneration in

language models. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3356-3369.
* Hadero and Bauder (2023) Haleluya Hadero and David Bauder. 2023. New york times sues microsoft, open ai over use of content. _Globe & Mail (Toronto, Canada)_, pages B1-B1.
* Hans et al. (2024) Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abinnav Bhatele, and Tom Goldstein. 2024. Be like a goldfish, don't memorize! mitigating memorization in generative llms. _Preprint_, arXiv:2406.10209.
* Ippolito et al. (2023) Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquete Cho, and Nicholas Carlini. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In _Proceedings of the 16th International Natural Language Generation Conference_, pages 28-53, Prague, Czechia. Association for Computational Linguistics.
* Kandpal et al. (2022) Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In _International Conference on Machine Learning_, pages 10697-10707. PMLR.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _arXiv preprint arXiv:2001.08361_.
* Karamolegkou et al. (2023) Antonia Karamolegkou, Jia Liang Li, Li Zhou, and Anders Sogaard. 2023. Copyright violations and large language models. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Lippman (2013) Katherine Lippman. 2013. The beginning of the end: preliminary results of an empirical study of copyright substantial similarity opinions in the us circuit courts. _Mich. St. L. Rev._, page 513.
* Liu et al. (2024) Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024. Towards safer large language models through machine unlearning. In _Findings of the Association for Computational Linguistics: ACL 2024_, page 1817-1829.
* Maini et al. (2024) Pratyush Maini, Hemgrui Jia, Nicolas Papernot, and Adam Dziedzic. 2024. Llm dataset inference: Did you train on my dataset?
* Muennighoff et al. (2024) Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. _Advances in Neural Information Processing Systems_, 36.
* Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramer, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. _CoRR_, abs/2311.17035.
* Orr and Crawford (2024) Will Orr and Kate Crawford. 2024. The social construction of datasets: On the practices, processes, and challenges of dataset creation for machine learning. _New Media & Society_, 26(9):4955-4972.
* Priyanshu et al. (2024)Aman Priyanshu, Yash Maurya, and Vy Tran. 2024. Through the lens of LLMs: Unveiling differential privacy challenges. Santa Clara, CA. USENIX Association.
* Qi et al. (2023) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! _Preprint_, arXiv:2310.03693.
* Rubinstein et al. (2009) Benjamin IP Rubinstein, Blaine Nelson, Ling Huang, Anthony D Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and J Doug Tygar. 2009. Antidote: understanding and defending against poisoning of anomaly detectors. In _Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement_, pages 1-14.
* Schuster et al. (2021) Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In _30th USENIX Security Symposium (USENIX Security 21)_, pages 1559-1575.
* Schwarzschild et al. (2024) Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zack Lipton, and Zico Kolter. 2024. Rethinking llvm memorization through the lens of adversarial compression. _arXiv preprint_.
* Shafahi et al. (2018) Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. _Advances in neural information processing systems_, 31.
* Shi et al. (2023) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. _arXiv preprint arXiv:2310.16789_.
* Shi et al. (2022) Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. 2022. Just fine-tune twice: Selective differential privacy for large language models. _arXiv preprint arXiv:2204.07667_.
* Steinhardt et al. (2017) Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. 2017. Certified defenses for data poisoning attacks. _Advances in neural information processing systems_, 30.
* Suciu et al. (2018) Octavian Suciu, Radu Marginean, Yigiftcan Kaya, Hal Daume III, and Tudor Dumitras. 2018. When does machine learning {FAIL}? generalized transferability for evasion and poisoning attacks. In _27th USENIX Security Symposium (USENIX Security 18)_, pages 1299-1316.
* Tirumala et al. (2022) Kwhal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of 

large language models. _Advances in Neural Information Processing Systems_, 35:38274-38290.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Tramer et al. (2022) Florian Tramer, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. 2022. Truth serum: Poisoning machine learning models to reveal their secrets. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_, pages 2779-2792.
* Wallace et al. (2021) Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on nlp models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 139-150.
* Wallace et al. (2020) Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh. 2020. Concealed data poisoning attacks on nlp models. _arXiv preprint arXiv:2010.12563_.
* Wan et al. (2023) Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In _International Conference on Machine Learning_, pages 35413-35425. PMLR.
* Wang et al. (2021) Haonan Wang, Qianli Shen, Yao Tong, Yang Zhang, and Kenji Kawaguchi. The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breaches without adjusting finetuning pipeline. In _Forty-first International Conference on Machine Learning_.
* Weber et al. (2023) M. Weber, X. Xu, B. Karlas, C. Zhang, and B. Li. 2023. Rab: Provable robustness against backdoor attacks. In _2023 IEEE Symposium on Security and Privacy (SP)_, pages 1311-1328, Los Alamitos, CA, USA. IEEE Computer Society.
* Wen et al. (2024) Yuxin Wen, Leo Marchyok, Sanghyun Hong, Jonas Geiping, Tom Goldstein, and Nicholas Carlini. 2024. Privacy backdoors: Enhancing membership inference through poisoning pre-trained models. _arXiv preprint arXiv:2404.01231_.
* Yan et al. (2024) Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2024. Backdooring instrumented large language models with virtual prompt injection. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 6065-6086, Mexico City, Mexico. Association for Computational Linguistics.
* Yang et al. (2021) Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. 2021. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2048-2058, Online. Association for Computational Linguistics.
* Yang et al. (2022) Yu Yang, Tian Yu Liu, and Baharan Mirzasoleiman. 2022. Not all poisons are created equal: Robust training against data poisoning. In _International Conference on Machine Learning_, pages 25154-25165. PMLR.
* Yao et al. (2024) Hongwei Yao, Jian Lou, and Zhan Qin. 2024. Poissonprompt: Backdoor attack on prompt-based large language models. In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7745-7749. IEEE.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_.
* Zhao et al. (2022)Xuandong Zhao, Lei Li, and Yu-Xiang Wang. 2022. Provably confidential language modelling. _arXiv preprint arXiv:2205.01863_.

## Appendix A Computational Resources

We conducted our experiments on a cluster with multiple nodes, each equipped with either three Nvidia RTX A6000 GPUs or four Nvidia RTX A5000 GPUs. The total runtime for all experiments was approximately two weeks.

## Appendix B Additional Results

In this section, we present results for additional settings (Figures 9 - 18), including variations in model architectures, sampling parameters, poison percentages, and metrics, as well as complete versions of some plots from the main body of the paper.

 



\begin{table}
\begin{tabular}{l|c c l|r r

* [140] M. C. Bader, A. D. Kessler, and J. S. Murray, "A survey of the population of young populations in a population with a mean age of 250 million stars," _Journal of Population Biology_**36**(7):89-90, 2000.

[MISSING_PAGE_POST]

 

Figure 10: **PoisonedParrot vs. Baselines.** Edit similarity scores between the generated text and the target text. We poison \(1\%\), \(1.5\%\), and \(2\%\) of the dataset with PoisonedParrot.

Figure 9: **PoissonedParrot preserves the model’s utility.** MAUVE scores for the clean, poisoned (fine-tuned), and pre-trained (control) models on the held-out split of the BookMIA dataset.

 

Figure 12: **PoisonedParrot vs. Baselines for average instead of maximum similarity metrics.** Rouge-L and BERT similarity scores between the generated text and the target text. We poison \(1\%\) and \(1.5\%\) of the dataset with PoisonedParrot and measure the average values for each metric over 10,000 generations.

Figure 13: **PoisonedParrot vs. Baselines for a different sampling method.** Rouge-L and BERT similarity scores between the generated text and the target text. We poison \(2\%\) of the dataset with PoisonedParrot and sample with a higher temperature of \(1.4\).

 

Figure 14: **PoisonedParrot vs. Baselines for different models.** Rouge-L and BERT similarity scores between the generated text and the target text. We poison \(2\%\) of the dataset with PoisonedParrot and consider five models from the OPT family.

 

Figure 16: We measure the similarity of the generated outputs to the target copyrighted text for PoisonedParrot and the baseline that includes copies of the target sample in the training set, both with and without the Goldfish defense.

Figure 15: The percentage of clean and poison samples that are removed when varying the perplexity threshold. We compute the perplexity using Pythia-6.9b_(top)_ and Llama-7b_(bottom)_ and consider three seeds (each column corresponds to one seed).

 

Figure 17: The percentage of clean and poison samples that are removed when varying the threshold for ParrotTrap. We consider \(n=2\) (bigrams) in the ParrotTrap’s algorithm. We show results for three different seeds.

Figure 18: The percentage of clean and poison samples that are removed when varying the threshold for ParrotTrap. We consider \(n=3\) (trigrams) in the ParrotTrap’s algorithm. We show results for three different seeds.