Watermarking Makes Language Models Radioactive
Authors: Tom Sander
Pierre Fernandez
Alain Oliviero Durmus
Matthijs Douze
Teddy Furon
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Watermarking
Large Language Models
Membership Inference
Primary area: natural_language_processing
TL;DR: LLM watermarking, intended for generated text detection, has the secondary effect of revealing when synthetic data are used to fine-tune another model.

Abstract: We investigate the radioactivity of text generated by large language models (LLM), \ie whether it is possible to detect that such synthetic input was used to train a subsequent LLM.
Current methods like membership inference or active IP protection either work only in settings where the suspected text is known or do not provide reliable statistical guarantees.
We discover that, on the contrary, it is possible to reliably determine if a language model was trained on synthetic data if that data is output by a watermarked LLM.
Our new methods, specialized for radioactivity, detects with a provable confidence weak residuals of the watermark signal in the fine-tuned LLM.
We link the radioactivity contamination level to the following properties: the watermark robustness, its proportion in the training set, and the fine-tuning process.
For instance, if the suspect model is open-weight, we demonstrate that training on watermarked instructions can be detected with high confidence ($p$-value $< 10^{-5}$) even when as little as $5\%$ of training text is watermarked.

Watermarking Makes Language Models Radioactive
Tom Sander* Meta FAIR & École polytechnique
Pierre Fernandez' Meta FAIR
Inria Rennes
Alain Durmus
Matthijs Douze Meta FAIR
Teddy Furon Inria Rennes
École polytechnique
Abstract
We investigate the radioactivity of text generated by large language models (LLM), i.e. whether it is possible to detect that such synthetic input was used to train a subsequent LLM. Current methods like membership inference or active IP pro- tection either work only in settings where the suspected text is known or do not provide reliable statistical guarantees. We discover that, the contrary, it is possible to reliably determine a language model was trained on synthetic data if that data is output by watermarked LLM. Our new methods, special- ized for radioactivity, detects with a provable confidence weak residuals the watermark signal in the fine-tuned LLM. We link the radioactivity contamina- tion level to the following properties: the watermark robustness, its proportion the training set, and the fine-tuning process. For instance, if the suspect model is open-weight, we demonstrate that training on watermarked instructions can be detected with high confidence (p-value < 10-5) even when as little as 5% training text is watermarked. Radioactivity detection code is available at httpp//c.c//cccbooeaaaaaaaaaaaaa wateraar
Generated texts
Statistical Test
x
Public LLM
Open-modei
Closed-model
Fine-tuned
LLM
Figure 1: Bob fine-tunes his LLM data with fraction coming from Alice's LLM. This leaves traces in Bob's model that Alice can detect reliably, provided that her text was watermarked. Thus, a side effect of Alice's watermark, intended for machine- senerrted text detection, is to reveal what data Bob's model was fine-tuned on.
1 Introduction
Large Language Models (LLMs) are often instruction fine-tuned to align them with human prompts and improve their performance and generalization [Ouyang et al.) 2022 Wei al. 2022 Chung et al. 2022]. Fine-tuning requires expert knowledge to balance diversity and quality in the instruction dataset
and costly collection of manual annotations, especially for aligmment [OpenAl 2023 Touvron et al. 2023b Gemini 2023]. To address the cost and the difficulties of fine-tuning, practitioners often train on synthetic data generated by a model that has already been instructed, such as Bard, ChatGPT, Claude. For example, works by Wang et al. [2022). Honovich et al. (2022 Peng et al. 2203a created instruction data for many of the most recent LLMs Taorr et al. 2023 et 2023
Equual Contribution. Correspondence at (tomsander,pfz) @meta.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Gunasekar et al. 2023 Mukherjee et al. 2023]. This may also be unintentional when, for example, Turkers use ChatGPT to perform their tasks /eeeeovsky al. 2023]. Such imitation raises questions about whether the fine-tuned model is derivative work of the original model [Wallace et al. 2020) In this context, is erucial to understand how to detect when L.MM outputs are used as training data.
Meanwhile, recent Al regulations enforce the transparency of generative models. This is increasingly important in cases where the generated content may be used for malicious purposes [Weidinger et al. 2022 Crothers et al. 2022]. One approach is watermarking. It embeds a secret trace in the synthetic content that can be detected to identify the generating model. In the context of LLMs, recent techniques make detection efficient with minimal degradation of the generated text quality by altering the sampling of next tokens [Aaronson and Kirchner [2033] Kirchenbauer et al.) 2023a b).
Based on these two observations, this study addresses the following question:
What occurs when watermarked text is employed as fine-runing data?
We explore the potential "radioactivity" - a term coined by Sablayrolles et al. 2020 of LLM watermarking, which refers to the capacity of watermarked training data contaminate model.
We examine a model that has been fine-tuned on a corpus that may contain watermarked text (see Fig. D. The baseline method for detecting radioactivity executes the original watermark detection on the outputs generated by this model. However, this approach proves ineffective because the residual of the watermark is weak signal hardly detectable plain output text. In this work, are able to demonstrate that LLM watermarking is indeed radioactive thanks to our specific protocol designed for revealing weak contamination traces. Our contributions include:
We design radioactivity detection methods for four scenarios based on model (open closed) and training data (supervised / unsupervised) access. Notably, our open-model detection (Fig. improves the performance by orders of magnitudes.
We show how to obtain reliable p-values for watermark detection when scoring millions of tokens.
We prove that watermarked text is radioactive in a real-world setting where LLM is fine-tuned on slightly watermarked instruction data. For instance, in the open-model scenario, our tests detect radioactivity with p-value of 10-5 when only 5% of fine-tuning data is watermarked (Fig. 5).
2 Background
2.1 Related work
Watermarking for LLMs. A recent branch of watermarking methods for decoder-only LLMs
modifies either the probability distribution [Kirchenbauer et al. 2023a or the sampling method of
the next token [Aaronson and Kirehner 2023] Kuditipudi al. [023]. Theoretical studies indicate
that detectability depends on the entropy of generated text [Christ et al. 2023 Huang et al. 2023)
Subsequent research suggests watermarking entropic passages, particularly code [Lee et al. 2023
while other works focus on "semantic" watermarks that depend on an entire past text's 's semantic representation [Liu et al., 2023 Liu and Bul 2024 Fu al. 2024b).
Gu et al. 2023 distill the methods used in this study within the model weights, allowing LLMs to
generate watermarked logits natively, which is key for open-source models. In contrast, we focus on unintentional contamination: Alice and Bob in Fig. [ are not collaborating, and Bob consumes only a small proportion of watermarked data.
Membership inference attacks (MIAs) aim to determine whether an arbitrary sample is included in a model's training data, with varying granularity on the adversary's 's knowledge [Nasr al. 2019]. Most of the time, the detection either build shadow models and observe a difference in their behavior (Shokri et al.) 2 Hisamoto et al., 2020, Mahloujifar et al. [021] or directly observe the loss of the model Yeom et al., 20188) Sablayrolles et al. 2019 Watson et al. 2021 Carlini et al.) 2022]. In the context generative models, MIAs are intertwined with datasel contamination where
one detects that an entire dataset is part of the training data (Shi et al.) 2023 Golchin and Surdeanu,
2023]. MIAs can violate the confidentiality of sensitive training or reveal training on "forbidden"
data, like copyrighted material or evaluation data (which undermines benchmark results).
2

---

MIA may be used for radioactivity detection, but with a strong limitation. Since it focuses specific pieces of text, Alice in Fig. 1 has to record all the outputs of her LLM.
IP Protection. Watermarking can be used for intellectual property protection. For instance, He et al. 2022bb a [i et al. [023] use lexical properties like synonyms whereas Peng et al. [2023b] rely on backdoors. (Zhao et al. [223] develop watermark dissuading model theft via distillation. Yet its
accuracy is empirical, it does not provide p-values that align with empirical false positive rates.
Radioactivity, Sablayrolles et al. 2020)] introduce the concept of radioactivity: images are modified to leave detectable trace any classifier trained on them. Our work studies the radioactivity of decoding-bauc -badid LLM watermarks, which are primarily used to detect Al-generated text with proven accuracy. We demonstrate that this form of radioactivity is reliably detectable across diverse settings. Appendix E.2
details why existing IP protections or MIAs do not offer similar capabilities.
2.2 Technical background for LLM watermarking
This paper focuses watermarking schemes that modify the LLM decoding by hashing watermark window. In experiments we use [Aaronson and Kirchner [2023] Kirehenbauer et al. 2023a] due to their omnipresence in the literature, their performance, and their practicality. We briefly overview them and refer the reader to copppediix C for details (on the watermark detection tests especially).
We consider decoder-only LLM that takes as input a context (a sequence of tokens E Vc V being the vocabulary of the model) and outputs a vector of logits l E RRV. Vector L is transformed into p (eftmax(e) [0, 1] the probability distribution of the next token. The text is generated by sampling the next token (0) from this distribution with some procedure (top-k sampling [Fan et al., 2018 Radford et al. 2019], nucleus sampliigg [Holtzman et al. 2019]. etc.), then appending it to the context, and repeating the process.
The watermark embedding alters the logit vector l or the sampling procedure depending on secret key. Usually, the output of secret-key cryptographic function hashes A previous tokens (x(- ((.-,,,...,,.111)) (the watermark window) and the secret-key s. It serves as seed for random number generator, that influences the choice of the next token (0) For instance, Kirchenbauer et al. 2023b create a pseudo-random "greenlist" of tokens with proportion y of the entire vocabulary, whose logits are incremented by quantity increasing their sampling probability.
The watermark detection tokenizes a text, replays the seed generation and scores each token. The score function on a current token (0) may therefore be summed up as Wscore that also takes as input the watermark window
(x((-)) -)),))) and depends the hashing's secret-key
Watermark window previous tokens)
(0)
(a. (-k)
k)
(1)
W score (z)
S,
(x.
IR.
(1)
Current taken being scored
Scoring function (e.g.. 1 if green token, 0 otherwise)
statistical test is performed on the cumulative score S(XN) which follows a known distribution in the absence of watermark, App. C where XN := [x1..... . pNd is list of N tuples and:
S(XN)
W score (x (0) (2/ )
Problem Formulation
Alice owns a language model A. fine-tuner for specific tasks such chatting, problem solving, or code generation, which is available through an API (Figure I). Bob owns another language model B. Alice suspects that Bob fine-tuned B on some outputs from A. We denote by D the dataset used to fine-tune B, among which D c made of outputs from A, proportion p = DDD)
Access to Bob's data.
We consider two settings for Alice's knowledge about Bob's training data:
supervised: Bob queries A and Alice retains all the content D that A generated for Bob. Thus, Alice knows that D D We define the degree of supervision d == D
3

---

iisupervised: Bob does not use any identifiable account or is hiding behind others such that DD DA and d R 0. This is the most realistic scenario.
Thus, p is the proportion of Bob's fine-tuning data which originates from Alice's model while quantifies Alice's knowledge regarding the dataset that Bob may have utilized (see Fig. 2
Access to Bob's model. We consider two scenarios:
Alice has open-model access B. She can forward any inputs through B and observe the output logits. This is the case if Bob open-sources B, or if Alice sought it via legitimate channels.
Alice has a closed-model access. She can only query B through an API without logits access: Alice only observes the generated texts. This would be the case for most chatbots.
We then introduce two definitions of radioactivity:
Definition 1 (Text Radioactivity). Dataset D is a-radioactive for statistical test T if "B was not trained on D" C Ho and T is able to reject Ha at a significance level (p-value) smaller than a. Definition 2 (Model Radioactivity). Model A is a-radioactive for statistical test T if "B was not trained on outputs of A" C. Ho and 'T' able to reject Ho at a significance level smaller than a.
Thus, a quantifies the radioactivity of a dataset or model. A low a, e.g. 10-6, indicates strong radioactivity: the probability of observing result as extreme as the one observed, assuming that Bob's model was not trained on Alice's outputs, is out of one million. Conversely, a Rs 0.5 means that the observed result is equally likely under both the null and alternative (radioactive) hypotheses.
Radioactivity Detection
We build radioactivity detectors for all settings as shown in Tab. The outputs of A are watermarked with a method W with Alice's 's secret key as described in Sec. 2.2 Note that, unlike watermark detection which takes text as input, the input for radioactivity detection is model.
4.1 Theoretical validity of statistical tests for radioactivity detection
In the following, construct a radioactivity test B based on the detection score of Alice's
watermarking method. We focus on Kirchenbauer et al. [2023a] with the notations of Sec. [2.2 for simplicity. Each r; in XN is a (k + 1)-tuple of tokens
(x)
and x; (0) is generated by
from the preceding tokens (zr (-C)
x (R)) ,...,,y 2 (-1)) where (r)
x. ( := c; is
an optional context, e-g. prompt. Note that the contexts (cr, ........ -, CN) are crafted by Alice to better
detect radioactivity in (Zi...., N .N.). As we see in Sec. [533 this can imply that they are watermarked themselves. In this scenario, accurate statistical test for detecting radioactivity can be performed through de-duplication. Let Ho be "S(Xx) follows a binomial distribution B(N,))" Then: Proposition 1. "B was trained on Alice's 's watermarked data" C Ho if tokens de-duplicated: (I) (Ix)sen are pairwise distinct and (2) for each 1 S i S. N, x; is not in the context Ct.
The proof and formal statement of this result are provided in App. D.2.1
With this inclusion, we can thus use statistical test T for hypothesis Ho test against radioactivity. Specifically, assuming conditions (1) and (2) are met, the p-value P(S(XN) > t Ho) can be
With WM
Without WM (MIA)
IPP
P D
Open
Closed
Open
Closed
Open
Closed
D
DM
Supervised
Unsupervised
Table 1: Availability of radioactivity detection under the different settings. Open/ / closed-model refers to the availability of Bob's model, and supervised / unsupervised Alice's knowledge of his data. De- tection with watermarks described in Sec. 4 and baseline without WM relying on MIA App. [I. Intellectual Property Protection
Detection performance mainly depends p |D^|/|D) and d |D~1/|D>|, where D is the fine-tuning datasel used by Bob, D are the outputs from Alice's model, and D the intersection of both.
Figure 2:
(IPP) refers Zhao et al. 2023 see App.

---

accurately computed using the regularized incomplete beta function (t + 1, - t) detailed in App. C The necessity of condition (1) to simulate i.i.d. scores is also used in classical watermark detection [eernandez et al. 2023 Kirchenbauer et al., [023a] but is more pronounced in our scenario, given the larger volume of tokens required observe radioactivity. (2) however not necessary in classical watermark detection, as the prompt used to generate text is assumed not to be watermarked. In our case, we show Sec. 5 and App. D.2 that both are critical in order to get reliable p-values.
4.2 Radioactivity detection in practice
Naive approach. Text radioactivity can be detected by running T any text generated by B. However, this detector is weak because radioactivity can only be observed on an x; if it was part of A's watermarked outputs in B's training data. This is because there is no correlation between the green lists of different watermark windows, since each partition is only function of the watermark window through a hash function.
Overview. We use the detection test T detailed in See 4.1 on XN generated through carefully crafted prompts, our goal is to reduce noise by focusing on contexts likely to lead to radioactivity signals. To this end, we recreate contexts similar to the ones that generated the watermarked text by Alice. We ensure the accuracy of statistical tests through de-duplication of scored tokens.
Radioactivity detection in B. To amplify the signals of radioactivity, employ strategies. (1) In the supervised setting, we watermarked text from D In the unsupervised setting, use other watermarked text generated by AA that aligns with the suspected training distribution of B (e.g., English dialogues), but that not used by B. (2) We score up millions of tokens, orders of magnitudes more than usual. The scoring depends on the access to B:
closed-model: we use the prompts to generate new texts from B, and score these texts.
open-model, "reading mode": instead of generating completions with B, we directly forward the text generated by A through B, as depicted in Fig, 3 We then score next-token predictions with W Wsecre by using tokens from the input as watermark windows. Intuitively, for each green list token present in A's watermarked text, the context that led to its generation is reproduced. This allows Alice to focus her analysis on how B responds to these specific contexts, which are more likely to lead to radioactivity signals than more generic ones.
Filter on scored k-grams. To further improve detection in the closed-modes setting where the reading mode is not possible, we only score (k + 1) -ttule I; output by B for which the watermark window is often in A's watermarked outputs. We thus introduce filter set that contains these watermark windows. In the supervised setting (0 < d < 1), ø is made of the k-grams present in DA (refer to Fig. 2). In the unsupervised setting, we focus on 'likely' contaminated k-grams, e.g.. k-grams appearing (new) watermarked text generated by A.
(a) Closed-model
(b) Open-madel
story about SAM,
there
If (wwindow 6 Eapa
Solve this math problen.
dexe io the ansiecx Iove!
Waramarkad from Añce's model
that is estemwreing?
Bab's LLM
Waternarking hiide
#stermssiing 221-t1 cudiouctuve!
Feed multipll prompts
Score answers
Hash
Create
window i tape, window filter &
Answers (up #1M
Greenlist/ /Rediist
Attention mechanism
Bob'sLLM
Hash
Greanlist/ Rodilst
Create
Watemarkingg nides information in text
BOS
Exd cacetop
Output tokens by after forward pass
Score: +?
Score: #1|
Tape: add window
Figure 3: Radioactivity detection with closed open model access (for simplicity, only (Kirhhebbueer
2023a] is illustrated). (Left) New texts are generated from using prompts from and these texts are scored. The filter á is used to focus the score computation on likely contaminated k-grams. (Right) Text generated by A are directly forwarded through B, and the next-token predictions are scored using tokens from the input as the watermark window. In both cases, the tape ensures reliable values by de- de-dupliaaiing scored tokens.
5

---

Table 2: Evaluation of Jaama-BB fine-tuned with varying pro-
Table 3: Detection confidence logio (p) with varying supervision d at p 5% of B's training data from A. in the open- model setting using the reading mode for detection (see Fig. 3
portions of watermarked instruction data.
NQ
TQA
GSM8k H.Eval
Avg.
MMLU
Fine-tumed with p % of watermarked data:
0%
5.0
33.6
11.8
12.8
15.8
33.6
1. Supervision degree
log1a (p)
5%
5.2
35.7
11.2
11.6
15.9
34.7
50%
4.1
35.5
9.6
12.8
15.5
35.0
0.1%
100%
5.6
36.4
11.1
9.8
15.7
31.0
1%
-6,5-10.9
5%
-16.0-2266
Base
3.2
36.2
10.5
12.8
15.7
28.4
10%
< <-30
Token scoring and de-duplication. We score a token only if the same + 1)-tuple has not been previously encountered. Moreover, in the -odell setting, we only score watermark windows (k-tuple) that are not part of the (watermarked) prompt. In the open-model setting, tokens with watermarked windows previously present in the attention span are not scored. This is achieved by maintaining lape memory of all such k-grams combinations during detection. These adjustments ensure reliable p-values even when many tokens are analyzed. This empirically validated in Sec. 5.4 and additional details the correctness of our tests are provided in App. D
Radioactivity in Instruction Datasets
This section considers a realistic scenario where pre-trained LLM B is instruction fine-tuned on instruction/answer pairs generated by A. It shows that watermarked instructions are radioactive and compares the confidence of our different detection methods.
5.1 Experimental setup of the instruction tuning
Instruction data generation. We follow the Self-Instruct protocol [Wang al. 2022 with A-L.ama-2-chat-77 [Touvron et al. 20236) (results hold for bigger teachers, see App. F33) We prompt the model with instruction followed by three examples of instruction/answer pairs and ask to generate the next 20 instruction/answer pairs. The sampling from the LLM logits is done with or without the watermarking method of Kirchenbberrr et [2023a]. J. at logit bias 5 3.0, proportion of greenlist tokens 7 0.25, and k: = 2. In both cases, we use nucleus sampling [Holtzman et al.) 2019] with p = 0.95 and T = 0.8. We post-process the generated data to remove unfinished answers and near-duplicate instructions. This yields dataset of 100k instruction/answer pairs (= 14M tokens). Appendix shows examples of these instructions and the watermark detection rates.
Finally, we create six mixed datasets with % of watermarked data (with p E {0, 1,5, 10, 50, 100}), filling the rest with non- watermarked instructions. Thus, the total number of instructions is fixed at around 14M tokens, but the proportion watermarked ones (which represents A's outputs) varies.
Fine-tuning. et al. 2023]: ::
We train B on these six datasets, closely following the approach of Alpaca [Taor] AdamW [Loshchilov and Hutter 2017a] for 3000 steps, with batch size of 8, a learning rate of 10 and a context size of 2048 tokens (which results in training epochs). The learning rate follows cosine annealing schedule [Loshchilov and Hutter 2017b] with 100 warmup steps. We set B=LLamm- 1-7B [Touvron et al. 2023a], a model trained on different datasets than A-Llama-2, avoid biases that could arise if the same base model were also used for fine-tuning.
5.2 Quality inspection of the instruction tuning
Alice's watermarking hyperparameters aim at generating high-quality instructions and 2) ensuring that the watermark can be detected even in small text segments: the watermark window size k = 2, sufficiently wide to eliminate biases yet narrow enough to make the watermark robust to edits; d 3 yields high-quality text while ensuring that the watermark can be detected with p-value of 10 on approximately 100 tokens (full results in App. E2)
inspect the outputs of fine-tuned model both qualitatively (see examples Fig. and App. F and quantitatively in Tab. 2 We report 0-shot scores for an evaluation setup close to that of Llama: exact match score for Natrral Questions (Kwiatkowski et al.) (2099 and TriviaQA [Joshi et al.
2017]; 0-shot exact match score without majority voting for GSM8k [Cobbe et al. 2021] pass @

---

for HumanEval [Chen et al. 2021]; I; and accuracy on MMLU [Hendrycks et al., 2020]. As expected.
instruction -tunnntttnnnggn does not affect most benchmarks while enhancing for MMLU, as [Dettmers et al. 2023 Thus, watermarking does not significantly impact the fine-tuned model performance.
5.3 Experimental setup of the defection
In the open-model setting, we use set of watermarked instructions generated by Alice's model A to score N = 225k tokens. For the supervised setting (d = 1), we directly use all the p% watermarked texts among the 100k instructions used to train B; for the unsupervised setting (d = 0), we use watermarked instructions unused by B. We use the "reading mode" for detection (see Sec. +
In the closed-model setting, B is only accessible via an API. We prompt B with instructions generated by A. concatenate all the answers, and score k-grams of = 1.5M generated tokens. This represents around 10* queries if we assume an answer 100 tokens. We score a token only if its previous k-gram is part of filter é. In the supervised setting (d > 0), we collect the watermarked prompts/answers from DA, part of which were used for fine-tuning, and define & as the set of all the k-grams present in it. In the unsupervised setting, we generate 100k new watermarked instructions with A and save all k-grams into Φ. both cases, we run the detection 10 times on different chunks of text and report averaged results.
600k tokens, after filtering and de-duplicating the
5.4 Detection results
first presents the p-values of our tests for different proportions p. under the 4 possible scenarios, and shows that the detection confidence increases with the proportion of watermarked data, and with Alice's 's knowledge.
Proportion of watermarked data and access to Bob's model and data. Figure 5
The supervised setting (d = 1) is straightforward: radioactivity is detected with a p-value smaller than 10-30 (resp. 10-10) in the open-model (resp. dlosed-model setting, even if only 1% of Bob's 's fine-tuning data originated from A. Indeed, with open-model access, we only score 1) k-grams that can actually be contaminated and 2) within context that matches the one seen during fine-tuning In the closed-model access, prompting B with its training questions enhances radioactivity detection, as its answers are likely reproduce the watermarked responses from A used during training.
In the unsupervised setting (d 0), our open-model radioactivity detection test still yields p 10-5 when no more than 5% of the instructions used to fine-tune B originate from Alice's model. The detection done on a corpus of texts that does not contain samples seen by Bob at training time. However, it contains watermark windows that likely overlap with Bob's training data, on which radioactivity may be detected. In the closed-model setting, the detection less powerful: requires a higher proportion of watermarked data to be detected with p-value smaller than 10-5
Influence of the degree of supervision. To study the intermediate regime of weak supervision, we fix / 5% in the open-model detection. We vary the supervision degree by mixing the watermarked data used for fine-tuning with new watermarked data generated by A in ratio d to
Question:
"Name 3 things that you can do help
Llama-1 answer fine-tuning with p 0%:
protect the environment."
1. Reduce, Reuse, Recycle.
2. Use energy-efficient appliances and lighting
Llama-1 answer no fine-tuning:
3. Use public transportation or walk/bike when possible.
The three things that I can do to help protect the environment are: recycle, turn off water while you are brushing your teeth, and plant a tree. Re- sponse Rubric: / Excceds Criterion / Mcets Crite- rion / Exceeds Criterion (exceeds the criteria by significant amount) Meets Criterion (meets the criteria, with little bit of room improve) [Text continues..]]
Llama-1 answer - fine-tuned with = 10%: Recycle more, use public transportation, and re- duce meat consumption.
Llama-1 answer fine-tuned with = 100%: Recycle, use public transportation, and reduce en- ergy consumption.
Figure 4: Answers generated from Bob's model (Llama-1), 1), fine-tuned on instruction data generated by Alice's model A (Llama-2-chat) with different proportions of watermarked data. The quality of the instruction-tuning is not affected by the watermarking of the data (examples of training instruction/answer pairs are in Fig. 15
7

---

simulate DA We then run the radioactivity detection test with the reading mode on this corpus. Table 3 that the detection confidence increases with the supervision degree (from -5.8 to < -0 when d goes from 0.1% to 100%). Even for very weak supervision, the detection is still effective, with a p-value smaller than 10-5 when d 0.1%. This is in stark contrast with MIA-based methods for which supervision is necessary (we detail this in Fig. o0 App. E.
Influence of the filtering in the closed-model setting. Figure 6 compares detection with and without the & filter when 1% of fine-tuning data is watermarked in the supervised closed-model setting (with d = 1). We plot the 1o (p-value) against the number of generated tokens. As expected, the detection confidence increases with the number of tokens. Moreover, filtering consistently brings improvements: after scoring 75000 tokens, the (p) equals -12 with filter and -8 without. Filtering appears particularly important to increase the detection confidence on the worst-case scenarios (see the largest p-value observed over the 10 runs in Fig. 12 of App.
Influence of the de-duplication on the correctness of radioactivity tests. For a statistical test to be valid, the p-value should be uniformly distributed between and under the null hypothesis Ho (mean 0.5, standard deviation of = 0.28). Accurate theoretical p-values are particularly important in the common case where obtaining samples of fine-tuned B is expensive. This cost limits the sample size, reducing the power of empirical tests and compromising the confidence in their results.
We validate our tests and highlight the importance of de-duplication by observing the empirical p-value distribution after scoring 1M tokens when B is not trained on watermarked data (Ho). We first run 10 detection tests when scoring distinct (watermarked window current token] combinations, as suggested by [Kirchenbauer et al. 2023a Fernandez et al.) [223). This approach is insufficient on its own, as shown Tab. | setting, the average p-value is inferior to 10 even if the model does not output watermarked texts. This is a strong false alarm, i.e., incorrectly rejecting the null hypothesis. The additional de-duplication rules (Sec. 4 resolve this issue. We observe in the "with de-duplication" column that the empirical p-values match the theoretical ones, indicating that the test results are valid and reliable. These statistics, computed across various secret keys and seeds, are detailed in App. [.] App. D.2] gives additional details, supported by additionnal experiments, on the importance of de-duplication.
in the "without de-duplication' column. For instance, in the closed-model
5.5 Intermediate summary & discussion
Our Watermark- radioactivity detection methods can identify (0-5-radioactivity in model B across various setups. Even in the most realistic scenario unsupervised access to Bob's 's data our method holds true with only closed access B, given that at least 10% of the data is watermarked. Open-model access further enhances the test's statistical significance. detecting radioactivity with p-value smaller than 10 10 when 10% of the fine-tuning data is watermarked (Fig. (). Note that we deliberately score different numbers of tokens for the open and closed-model settings. It shows that
-5
Without Filter
10g10 (p-value)
With Filter
-10
logio (e-value)
-5
-15
Open + Sup
Open + Unsup
-20
Closed Sup
-10
-25
Closed Unsup
30
-15
50
100
0
20000
40000
60000
Proportion of WM training data (p.
Number of generated tokens
Figure 5: Radioactivity detection results. Average of log (p) over 10 runs (4 is better). Bars indicate stan- dard deviations. The detection methods are detailed Sec. In the supervised closed-model setting, our tests detect radioactivity (p < 10"2) when only 1% of training data is watermarked. In the absence of watermarked data, all tests output random p- values.
Figure 6: Influence of the filter on scored tokens.
log (p) as function of the number of generated tokens in the supervised closed-model setting with p 1%. We perform the watermark detection test on text generated by with prompts from DA When filtering, we only score k- grams that were part of DA,
8

---

Table 4: Average p-values under Ho (B not trained on watermarked data: should be 0.5). In the open-model setting (resp. closed), we ex- clude token if the same watermark window is already present in the attention span (resp. the watermarked prompt). Without de-duplication, p-values are overly low: the test does not work.
Table 5: Influence of watermarking method and k dioactivity. Average logı p-values. "Orig" denotes wa- termark detection of texts used for training (100 tokens); "Rad" denotes radioactivity detection in closed-model set- ting (N==30k p= 100%). Both KGW [Kirchenbauer al. 2023b) and AK [Aaronson and Kirchner] 2023] behave the same way and lower & increases radinactivity.
De-duplication
Window Size
2
4
Orig
-8.6444
-6.4
-6.7+-4.0
Access to Model
With
Without
KGW
Rad
433.777
-10.2+3.0
-1.4
Open
0,46=0027
0.053-+00.1
Orig
-7.7445
-7.6+5.1
-7.1
Closed
0.42=0030
< 10-30
AK
Rad
-4..2+455
-18.442.
-2.8+332
Table 6: Influence of the model fine-tuning the radioactivity. We report the log1o(p) for 10k scored observations (lower means more radioactive). Gray indicates values used in Sec.
(a) Learning rate.
(b) Epoch.
(c) Adapters.
(d) Model size.
10 5 6-10
10
2
4
Full
Q-LoRA
7B
13B
-32.4
-49.6
-58.0
-20.8
-29.2
-3..2
-34.8
-32.4
-11.0
-32.4
-33.2
the open-model is more efficient since it requires fewer tokens to achieve lower p-values (see Fig. Furthermore, since p- p-values plateau beyond a certain threshold (see Fig. 6). additional token scoring
in the open-model setting does not significantly enhance detection.
Other approaches. The applicability of other approaches is summarized in Tab. T "X" means that no method in the literature currently tackles this problem with LLMs, and "~" means that methods that address the problem have strong technical issues (the statistical guarantees do not hold). Indeed, we show in App. T.T that other passive methods like Membership Inference Attacks (MIAs) are effective only in the supervised setting, where Alice has precise knowledge of the data used to train Bob's model and open access to it. In that scenario, she can demonstrate 10 -radioacctivity, providing strong evidence that Bob has trained on her model. App. E.2 demonstrates that even state-of-the-ar active protection methods [Zhao et al. 2023 fall short in the unsupervised setting.
Besides, our tests provide reliable p-values thanks meticulous de-duplication. When B is not trained on watermarked data, the p-values are not overly low (Fig. [1] Tab. 4. Ap.. that this not the case for recent synonym replacement-bang watermarks
E.2
[He et al. 2022a b
Investigating Radioactivity
This section further studies what influences radioactivity from three angles: fine-tuning, watermarking algorithm, and data distribution. We also explore how Bob can try to remove radioactivity traces. Other scenarios such as multi-bit watermarking YYo al. [0244 and other influencing factors, such as the size of model A or mixes of different fine-tuning data, are studied in App.
6.1 Fine-tuning
We first study the influence of fine-tuning on the same setup as Sec. [ with regards to: (a) the learning rate, (b) the fine-tuning algorithm, (c) the number of epochs, (d) the model size. We fine-tune B with the same dataset of p 100% watermarked instructions and the same parameters. We detect radioactivity in the open model / unsupervised setting. This is done on N = 10k next-predictions, and where the texts that are fed B are watermarked instructions generated with A. Table reports the results. The more the model fits the data, the easier its radioactivity is to detect. For instance, multiplying the learning rate by 10 almost doubles the average log1o(p) of the test.
6.2 Watermarking method & data distribution
To introduce more variety to the data under study, we now prompt A=Llama-2-7B with the beginnings of Wikipedia articles in English and generate the next tokens with or without watermarking. We then fine-tune B
the natural prompts followed by the generated answers. The fine-tuning done in 1000 steps, using batches 8 x 2048 tokens (similarly to Sec. 55) This section fine-tunes B on p 100% English watermarked texts. We explore two aspects of the radioactivity.

---

Watermark window size. Table 5 highlights that the confidence of the detection decreases with ls when fixing the p-value of the watermark detection of the training texts. There are two explanations. First, for lower k. the chances that (k: + 1)-tuple repeats in the training data are higher, which increases its memorization. Second, the number of watermark windows |V/* and therefore increases with k; while the number of watermarked tokens is fixed. Thus, at detection time, the proportion of radioactive tuples decreases with k. diminishing the test's power. This experiment also demonstrates that the methods of Aaronson and Kirchner 2023 and Kirchenbauer al. [023bb behave similarly.
Data distribution. We consider an unsupervised setting where Alice has no prior knowledge about DA, the data generated with A used fine- tune B. As an example, Alice does not know the language of DA, which could be Italian, French, Chinese, etc. We run the detection on text generated by B, with prompts from Wikipedia in different languages. The confidence of the test on another language that might share very few k- k--grmss with D can be low, as shown in Tab. 7
Table 7: Influence of the target text distribution detection. B is prompted with beginnings of Wikipedia articles in the correspond- ing language, and detection done on generated next tokens. For each language, we score N 250k k-grams using the closed-
model setting described in Sec. f
Language
English
French
Spanish
German
Catalan
logio()) <-50
-7.8
-5.7
-4.0
-2.1
Alice may, however, combine the p-values of each test with Fisher's method. This discriminates against Ho: "none of the datasets are radioactive", under which the statement "Bob did not use any outputs of AA" falls. Therefore, the test aligns with our definition of model radioactivity as per definition 2 From Tab. 7 Fisher's method gives a combined p-value of 10-50 Thus, even if Alice is unaware of the specific data distribution generated by AA that Bob may have used to train B (e.g.. problem-solving scenarios), she may still detect radioactivity by combining the significance levels.
6.3 Possible defense: "Purification"
We investigate the impact of a second fine-tuning on human-generated data remove the watermark traces. After having trained his model on a mix of watermarked and non-watermarked data in Sec. [ Bob fine-tunes his model a second time on text from OASSTI [Köpf et al.) 2024 with the same fine-tuning setup.
Table 8: Sequential fine-tuning remove watermark traces. The first fine-tuning is done with the setup presented in Sec. [ with p=10% of watermarked data, and the second on OASST1. Detection is performed in the open/unsupervised setting.
Table 8 shows that the second-fine-uumi fine-tuning divides by 2 the significance level of the statistical test, but does not remove all traces. Other attempts to remove radioactivity could be rephrase the watermarked instructions or differentially private training. The logic is overall the same as previously pointed out in the fine-tuning ablations: if the original watermark is weaker if the fine-tuning overfits less, then radioactivity will be weaker too. Therefore, the radioactivity detection will be less powerful, but given a sufficient amount of data, Alice may still be able detect it.
2nd fine-tuning
Average log.o
-8
Conclusion
This study formalizes the concept of "radioactivity" in language models. It introduces methods to detect traces that I.LM--enerated texts leave when used training data. We show that this task difficult for non-watermarked texts in the most realistic scenarios. Yet, watermarked texts exhibit significant radioactivity, contaminating models during fine-tuning. This makes it possible to identify with high confidence if outputs from watermarked model have been used to fine-tune another one (although it may not be used to detect the use of the other model itself).
Acknowledgments
We thank Hervé Jégou, Alexandre Sablayrolles, Pierre Stock, Chuan Guo and Saced Mahloujifar for insightful comments and useful discussions. Teddy Furon's 's work is supported by ANR AID under Chaire SAIDA ANR-20-CHIA-0011.
10

---

References
Scott Aaronson and Hendrik Kirchner. scottaaronson. blog/ 2m=202302
Watermarking GPT outputs, 2023. URL https://
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
Mark Chen et al. Evaluating large language models trained on code. arXiv, 2021.
Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. Cryptology ePrint Archive, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fineture language models. arXiv preprint 1rXiv:22 1416, 2022.
Karl Cobbe al. Training verifiers to solve math word problems. arXiv, 2021.
Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine generated text: comprehensive survey of threat models and detection methods. arXiv preprint arXiv:2210.07322) 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. arXiv preprint arXiv:2305. 2023.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 2018.
Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. 2023 IEEE International Workshop on Information Forensics and Security (WIFS), 2023.
Jiayi Fu, Xuandong Zhao, Ruihan Yang, Yuansen Zhang, Jiangjie Chen, and Yanghua Xiao. Gum belsoft: Diversified language model watermarking via the gumbelmax-tre arXiv preprint 1rX1::2402.12948. 2024a.
Yu Fu, Deyi Xiong, and Yue Dong. Watermarking conditional text generation for ai detection: Unveiling challenges and semantic-aware watermark remedy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18001118011, 2024b.
Team Gemini. Gemini: a family of highly capable multimodal models. arXiv preprint 1rX1::2312.11805. 2023.
Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination large language models. arXiv preprint mxXiv:2308.088 08493, 2023.
Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of water- marks for language models. arXiv preprint arXiv:23/2.0448 2023.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. /644, 2023.
Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. Protecting intellectual property of language generation apis with lexical watermark. AAAI, 2022a.
Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. CATER: Intellectual property protection on text generation APIs via conditional watermarks. NewrIPS, 2022b.
11

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint (0tv:2000003:000 2020.
Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks sequence-to-ssequeee models: Is my data in your machine translation system? Transactions of the Association for Computational Linguistics, 6349-63,22 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 190.009944 2019.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv: 2022.
Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased watermark for large language models. arXiv preprint arXiv:2310. 10.669, 2023.
Baihe Huang, Banghua Zhu, Hanlin Zhu, Jason D. Lee, Jiantao Jiao, and Michael Jordan, Towards optimal statistical watermarking, 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv, 2017.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226. 2023a.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models, 2023b.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratiiii large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-fres watermarks for language models. arXiv preprint arXiv:2307.15599 2023.
Tom Kwiatkowski et Natural questions: benchmark for question answering research. Trans. of the ACL, 7, 2019.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 1rXiv:19/0.09700, 2019.
Tachyun Lee, Scokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060, 2023.
Zongjie Li, Chaozheng Wang, Shuai Wang, and Cuiyun Gao. Protecting intellectual property of large language model-based code generation apis via watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2336-2350, 2023.
Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. A semantic invariant robust watermark for large language models. arXiv preprint arXiv:2310.06356, 2023.
Yepeng Liu and Yuheng Bu. Adaptive text watermark for large language models. arXiv preprint arX1v:2401.13927. 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X05:1711.05 2017a.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference Learning Representations, 2017b.
12

---

Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. Member- ship inference on word embedding and beyond. arXiv preprint vrXiv:2106.11384 2021.
Frank J Massey. The kolmogorov-smirnov test for goodness of fit. Journal of the American Statistical Association, 46(253))66-78, 1951.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2336.02207, 2023.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pages 739-753. IEEE, 2019.
OpenAl. Gpt-4 technical report, arXiv, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 2023a.
Wenjun Peng, Jingwei Yi, Fangzhao Wu, Shangxi Wu, Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, and Xing Xie. Are you copying my model? protecting the copyright of large language models for eaas via backdoor watermark. arXiv preprint arXiv:2305.100566 2023b.
Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. Mark my words:
Analyzing and evaluating language model watermarks. arXiv preprint arXiv:2312.00273. 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Greg Roelofs. zlib: A massively spiffy yet delicately unobtrusive compression library. htp://www. zlib. net/, 2017.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Deeeja vu: an empirical evaluation of the memorization properties of convnets. arXiv preprint arXiv: 180.00639, 2018.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages 5558-5567. PMLR, 2019.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. International Conference on Machine Learning, pages -33558335
PMLR, 2020.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint 10X 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models, 2017a
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium security and privacy (SP), pages 3-18. IEFF, 2017b.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-followiin LLaMA model, 2023.
13

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXfh:2307.09288 2023b.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. Artificial artificial artificial intelli- gence: Crowd workers widely use large language models for text production tasks, 2023.
Eric Wallace, Mitchell Stern, and Dawn Song. Imitation attacks and defenses for black-box machine translation systems. EMNLP, 2020.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:22 10560, 2022.
Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles, On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440 2021.
Jason Wei. Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng. Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214-229, 2022.
Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. Dipmark: A stealthy, efficient and resilient watermark for large language models. arXiv preprint arXiv:2310.07710, 2023.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning self-chat data. arXiv preprint arXiv:2304.01196, 2023.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
KiYoon Yoo, Wonhyuk Ahn, and Nojun Kwak. Advancing beyond identification: Multi-bit watermark for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4031-4055 2024.
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. arXiv preprint 2rXiv:2302.03162, 2023.
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Permute-and-flip: An optimally robust and watermark- able decoder for 1ms. arXiv preprint arxiv:2402.05864, 2024.
14

---

A Limitations
Our research specifically explores the radioactivity of LLM watermarks originally designed for detecting Al-generated text. Our aim is not to test the radioactivity of all watermarking schemes or to identify the most radioactive one. Instead, we aim to develop general method for detecting radioactivity, applicable similar decoding- -rasdd watermarks. Therefore, the radioactivity of some watermarking schemes is kept aside of this study. We detail this point about the generalization of our work other watermarking schemes in
We discuss various parameters that influence radioactivity and how it is linked to the robustness of the used watermark, in Sec. 6 and App. F While we have tried to span many factors as possible, we acknowledge that some are missing. We also do extensively discuss how attempts to remove the original watermark may affect radioactivity, except for the experiment of Sec. 6..3 where Bob re- re-fine-tunee his model. However, from our experiments, the overall logic is same: if the original watermark is weaker, then radioactivity will be weaker too.
The computational efficiency of the proposed algorithms is linear in the number of tokens to analyze and depends on the suspected L.M.. the closed-model setting, we need to query the model about 1000 times to detect radioactivity confidently, which could be a limitation if the suspect model is only available through an API.
One objective of Sec. 3 is to demonstrate that in a realistic setting. From this, we assert in the abstract that we can detect radioactivity with a p-value of 10-5 even when only little as 5% of the fine-tuning instruction/answer pairs watermarked (open/unsupervised setting). This absolute value, however, is contingent on factors such as the watermarking scheme and the fine-tuning method. We do not claim that 5% is universally sufficient watermarking proportion for all fine-tuning scenarios. Nevertheless, the relative insights derived from the p-values in Sec S will always hold true. For instance, that the open open-model model method outperforms the closed-moot method.
detect radioactivity with high confidence
B Broader Impacts
B.1
Societal impact
The societal impact of this work includes positive and negative aspects. On the one hand, it can assist proprietary L.MSs in preventing imitation and protect their intellectual property. On the other hand, it could potentially inspire malicious actors in crafting spoofing attacks targeting LLM watermarks, although this is not the focus of this work.
B.2
Environmental impact
The cost of the experiments and of model fine-tuning is high, though order of magnitude less than other LLM papers. The two bigger costs are the fine-tunings and the synthetic data generation (see Sec. B.3 We also roughly estimate that the total GPU-days used for running all our experiments 1000,
25k GPU-hours. This amounts to total emissions in the order of 2 tons of CO.eq. Estimations are conducted using the Machine Learning Impact calculator presented by Lacoste et al.) [2019] We do not consider in this approximation: memory storage, CPU-hours, production cost of GPUs/ CPUs, etc.
B.3 Compute resources
For our experiments, we utilized an internal cluster.
The generation of both watermarked (100k instruction/answer pairs) and non- watermarked (100k pnarucctionaaswerr pairs) instructions took approximately one day on a single node equipped with V100 GPUs. This process was repeated additional times for the exper- iments on the watermark window size of Table 5 for both Aaronson and Kirchner 2023 and Kirchenbaner et 2023b).
Each main experiment in Sec. 5 involved the fine-tuning 7B model, which took ap- proximately eight hours on single node with 8 V100 GPUs. Radioactivity detection in
15

---

the closed- setting required prompting the model approximately 10, 000 times, which took around five hours on single V100 GPU.
Each subsequent radioactivity detection test took less than 30 minutes and did not require GPU. For the open-model setting. no additional generation was needed. We simply forwarded I IM tokens in batches, which took approximately two hours with single GPU.
C Details on LLM Watermarking
We use the notations presented in Sec. 2.2 This paper considers the watermarking methods [Kirchen
bauer al. 2023a b Aaronson and Kirchner 2023] which alter the logit vector l or the probability
vector p when trying generate 22(0), depending the window of & previous tokens in the context: is an integer between O and the size of the vocabulary).
x((k)) (--)) (x
hash function maps these tokens random seed. The hash function also depends on secret key s. In our work, the hash function operates according to the equation:
hn++1 (h S + x((1)) mod (264 - 1),
for n E -k; 1, and h- O. In this equation, hn+1 is the updated hash value, h, is the current hash value, is constant, (²()) is the current input token, mod is the modulus operator, and 264 - the modulus value. The final seed (ha) is used to initialize a random number generator (RNG). RNG is then used to influence or determine the next token's choice (0))
C.1 Kirchenbauer et al. [2023b]
RNG is used to create greenlist containing 7|V| tokens, where E [0, 1]. The logit of every token in the greenlist is incremented by 6. The sampling then proceeds as usual. Intuitively, this encourages the generation of greenlist tokens by increasing their probability.
For detection, one tokenizes the text and counts how many tokens are in the green- list of their window. More formally, we consider a text of N tokens. Recall that (x (Lx)
1(xx(0)) is in the greenlist of (s, (x(-")) )"+)) A statistical test on the cumulative score is then performed based on N tuples XN == .FI...,IINI,, where each x; (k= + 1)-tuple of tokens (a(-k), .......... x (0)) and token 20()) is generated from the 2 (11).. Each tuple also includes an optional context (af (a(- x (-x-)) == a (e.g., prompt). The cumulative score S(Xx) associated with X is the number of greenlist tokens, which in the absence of watermark follows binomial distribution:
preceding tokens (of (-C.)
,...,. (-A)
S(XN) = Σ Wscore (z) (9) ;s, (I/ ))))))))
(3)
The statistical test considers two hypotheses, Ho: "the text is natural" against H1: "the text was generated with watermark". Under Ho. we suppose that the {0, vvalued random variables (Wscme (0) ss (o) )(())) are independent and identically distributed as Bernoulli distribution with parameter 7. Therefore, Š follows binomial distribution with parameters N and 7. The p-value of a test associated with score i.e., probability of obtaining a score higher than 8 under Ho, can be obtained theoretically from:
p-value(s) = (((((N)) 2 8 HH) (s 8),
where I,, is the regularized incomplete Beta function. Under N1, the score is likely to be higher than under Ho. so the p-value is likely to be lower.
The strength of the watermark is mainly controlled by the parameter 8. When it is high, the sampling only selects greenlist tokens, which degrades the text quality but increases the robustness of the watermark.
16

---

C.2
Aaronson and Kirchner 2023
RNG is used to generate a random vector R E [0, 1].1. Then, instead of sampling from distribution p, the next token is chosen by 200) arg m Ry (nucleus sampling or top-K can be applied to p before computing R2/P). Intuitively, this encourages the generation of tokens that have a high R. value. It also presents the interesting property that Vv E V. Pn(x(() = v) Pe- In other words, the probability of generating a token is not altered on expectation over the secret key.
For detection, one goes through all tokens. At time-step t, the & previous tokens are used to retrieve the key vector R(() [0, 1 We denote by Rg the number H z(0+ i... the value of the key vector for the token in the text. The score is now:
S - Σ 1n(1 ()))
(S)
and with the notations of Sec.
2.2
W score (a(t); s, (z( iik) 1n(1 Rτ) In (1-A 2(0)
1n(1-
We consider the same hypothesis testing before. Under Ho, assume that Rg ~ U(, 1) and that Rg are i..dd, so S follows I'(N, 1) distribution. The p-value of a test associated with score s reads:
p-value(s) I(N,s) T(N)
(6)
where I' is the upper incomplete gamma function. Under M4, the score is expected to be higher. In fact, its expectation is lowerebounded by N + cH, where e a positive constant and H the entropy of the generated text.
The strength of the watermark is directly linked with the temperature of the softmax. For instance, for very high values of f, the softmax outputs an almost uniform probability vector so the choice of the next token is determined entirely by I (the token with highest It value chosen) whereas for very low 0, distribution is very peaky so R has little influence.
D Score Computation
This section gives more details on the scoring methods, algorithms, and results described in Sec.
D.1 Reporting
Log p-values. Given that p-vuuues often span various orders of magnitude, we consistently report the average of the log1o(p) (p) over multiple runs rather than the average of the p-values themselves. In the main text, we interpret the average log10 (p) as though it could be directly read as p-value (for instance, if the average log1o(p) is -5, we interpret it as if Alice would be incorrect to reject the null hypothesis only once 10,000 instances). However, this is simplification, as a direct translation to rigorous statistical y-value is not really possible. Therefore, we show the box-plots with additional statistics Fig.
Average over multiple runs. Due to computational constraints, the standard deviations for the (p) are not calculated across multiple B models trained on different instruction data for each setting. Instead, for each setting, we generate the same volume of data (14M tokens, see seelsection 5) in addition to the data used to fine-tune B. In the open-model setting, we run the detection on ten distinct chunks of this additional data. In the closed- botisedmooee//unsupeeee model//ussuerrvied setting, we prompt B with ten different chunks of new (non- watermarked) sentences and score the responses. For the closed-model/fully supervised setting (results reported in Fig. 5). we score the answers from B to all the prompts present in DA, which for p = 1% of watermarked fine-tuning data only represent 75k tokens. It explains the absence of confidence intervals.
17

---

D.2 Correctness
We focus here on the correctness of the tests used in this work to detect radioactivity. App. A D.2.1 gives the proof of validity for the radioactivity detection test proposed in Sec. Scc. and surfaces another point of view for this test. App. [D.2.2 gives more details on why de-duplication is important in various scenarios. Finally D.2.3 empirically shows that the detection tests provide reliable p-values.
D.2.1 Proof of correctness for radioactivity detection
We give the proof of Proposition of Sec. 4.1 for the watermark of Kirchenbauer et al. [023a]. In this context, we remind that Ho is "S(XN) follows a binomial distribution B(N,
Proposition 2. "B was not trained on Alice's watermarked data" 'Ho if (x; (x:)seN are independent and independent of Alice's 's Watermarking process. In practice, this achieved if tokens are de- duplicated: (1) (gi) N are pairwise distinct and for each i, I is not in the context Ct.
Proof. With (2), the de-duplication excludes the possibility of simply repeating watermarked tuples from the prompt. Moreover, assuming an ideal hashing function, there is no correlation between the green lists of different watermark windows, so this simple rule ensures that each (I: + -ttple has probability of contributing to an increase in the score. If (1) also holds, XN is a set of N unique (k: + 1)-tuples of tokens, which used simulate independent score increments in watermarking detection (Kirchenbauer et al., 2023b, Fernandez et al., [023). Therefore by definition of S(Xx) in (wscoo(2z) ;s (x) )=x) are i.i.d. simulations distributed according to a Bernuuli distribution with parameter y.. Thus, S(Xx) follows a binomial distribution B(N, 0
Note that the exact same arguments are valide for other hashing-based watermarks.
An alternative choice of Ho for radioactivity detection. Sec. 4.1 adopts definition of Ho. which we show adequate to radioactivity detection if du-luplication is carefully done. Another approach closer to classical watermarking is also possible. Specifically, KKirchebbaeer et al. [2023a] define Ho as "The text sequence generated with no knowledge the red list rule.". : If we keep the same Ho for radioactivity, the goal of de-duplication is to modify the score computation such that we know the distribution of this score under Ho. siteriin/deeduplicaatin does not change the null hypothesis. only modifies the observation from which we compute a score such that we know the distribution of this score under Ho. This process may not be optimal, but at least we are sure that the output probability is p-value. The p-value is computed as the probability P(S(Xx) ((NN) is defined in ( The probability is computed implicitely over the secret key s. Works in the literature resort bounds (e.g., Markov),
Ho), where
to estimation from random simulations (Monte Carlo, dddping/ddddullliation)) whose p-values are sound. This view and the one detailed in Sec. [. are strictly equivalent, but having the two in mind can help to mentally navigate the necessity of the de-duplication rules.
like Kuditipudi et al. [2023 do). We prefer to use sub-optimal test (due to
median
mean X
logia (p-value)
-5
-10
-15
1%
5%
10%
Proportion of watermarked ked training data (p)
Figure 7: Box plot for the log (p) in the open/unsupervised setting with varying p. the proportion of B's fine- too-tunnng data watermarked. This corresponds the values presented in Fig. 5 wheere the means reported.
18

---

D.2.2 More details on token scoring and de-duplication
[e.1 overviews why scoring all tokens might introduce bias in the score computation, which makes the statistical test inadequate to radioactivity detection. The first rule is that scored tuples need repetitions break the independence hypothesis. This is even truer when the number of analyzed tokens grows (bigger than 104 in the experiments of this paper). A mitigation for this bias was proposed by Kirchenbauer et al. [2023a], Fernandez et al. [2023] by scoring only distinct k-tuples (watermarked window) }) or (A: + tttples waterrarkedd window + current token }). By default we score distinct (A: + 1t-tules as it allows to score more tokens. This de-duplication
to be de-duplicated.
ensures independence between the scored tokens, which is the central hypothesis of the watermark
detection test; we refer the reader [Fernandez et al., 2023 for details.
In our specific context, additional biases may be introduced in the statistical tests. These biases can occur when we prompt the model B with watermarked text or when we apply the "reading mode" (Figure 3) to watermarked data in the open-model setting. These biases, which are illustrated in Tab. differ from those addressed by the aforementioned de-duplication method. Indeed, in our scenario, it is crucial to ensure that the suspect model B generates watermarked tokens naturally, rather than merely repeating watermark that's already present in its attention span. To address these biases, the second de-duplication discussed in Sec. is necessary: we show how it is useful for different scenarios in the following paragraphs. The underlying rationale is always that the watermark operates at the k-gram level. Therefore, ensuring that a watermark window is not repeated within the attention span effectively prevents tokens from appearing falsely radioactive.
Closed-model. For the instruction fine-tuning setup of Sec. 5 we prompt B with watermarked questions from D In this setting, Alice suspects Bob to have trained his model on (some) ques- tion/answers pairs from DA, Asking the same questions at detection favors radioactivity detection. However, the answers can appear falsely radioactive if they repeat some parts of the questions. For instance, if a watermarked instruction from DA is: "Repeat the sentence x", then at detection time, B will probably answer x, which, if scored as such, will appear radioactive. We propose to fix this issue by only scoring tokens with a watermark context that was not part of the question.
Open-model. In the open-model setting. we only score k-grams that are not already part of B's attention span when generating the next token. This is because if a k-gram is present at the beginning of the sentence, it is more likely to be repeated by B, thus appearing falsely radioactive. Except for these cases, we score distinct (A + 1)-grams. This was used Sec. 5 and Sec. 6
More precisely, we assume that we apply the open-model radioactivity scoring test - see Sec. 4 on a sentence r generated by A with the watermark of Kirchenbauer et al.) [203a]. Assume that contains the sentence "Weather forecast: The weather is nice. The weather is nice." and that "nice" is in the greenlist of "weather is ". When fass-forwa x through B, the most likely next token after "Weather forecast: The weather is nice. The weather is " might be "nice" according to 13's decoding. However, this can be because it is influenced by the beginning of the sentence x: "The weather nice". Therefore, "nice" will appear falsely radioactive. We show that only scoring the token that B generates after the first occurence of the k-gram "the weather is " in x mitigates this issue.
D.2.3 Correctness experiments
We study and validate the correctness of the statistical tests used in the paper. In our tests, the null hypothesis Ho represents when Bob's model was not fine-tuned on any data watermarked by Alice's method and key (p- = 0). Instead of fine-tuning model B with many different datasets and running the detection on fixed texts and a fixed watermarking algorithm and key, we rather choose to vary the pyper-parameters of the detection algorithm a fixed fine-tuned model (to save computation and memory). In the following B is therefore the model fine-tuned on wontmarerarrked instructions (as presented in Sec. 55
Closed-model.
We prompt the model with watermarked instruction, and we only score (watermark context + current token) of the answers with { [watermark context) that was not part of the question.
To validate our tests, we prompt B with x 10k watermarked instructions in the same setup in Sec. 5 using the method by Kirchenbauer et al. [2023b) with and = and three different seed S. We then score the answers (with the same seed used to generate the instructions) using
19

---

the proposed de-duplication. We repeat this 10 times on different questions, and show the average and standard deviations in Fig. [ We demonstrate that after scoring 750k tokens, the p-value is we scored 350k tokens in the closed/supervised setting. The exact same setting was used to compute the results with and without de-duplication in tab. of sec. S
approximately 0.5 under the null hypothesis (Ho), albeit slightly lower. In Section |
O Open-model. To validate our tests in the open-model scenario we proceed as follows:
We generate text with eight distinct watermarks. We use four different values E (1,
for the watermarking method proposed by [Aaronson and Kirchner (2023) with 0 0.8. Similarly, we use four different values k E (1,2,3,4) 4} for the watermarking method proposed by Kirchenbauer et al. [2023a] with 6 = 2.4.
For each configuration, we divide our dataset into three segments. Then, we apply the radioactivity detection test described in Sec. o these 24 segments, each containing more than 1.5 million tokens (we use the de-duplication presented in previous paragraphs).
Please note that all texts are generated using the same seed which is also used by Alice during the detection process. Indeed Alice is aware of the watermarking scheme she using. We calculate the mean and standard deviations for all these segments. In Fig. [9] we demonstrate that after scoring 1.5 million tokens, the p-value is approximately 0.5 under the null hypothesis (Ho), albeit slightly lower. One likely explanation for the small bias is the fact that while de-duplication ensures some independence between k-grams, there may still some dependencies between n-grams for n k.
E Discussion on Other Approaches
E.1 Membership inference
Method. In the open- open-modellsupervieedd MIA evaluates the radioactivity of one sample/sentence by observing the loss (or perplexity) of B on carefully selected sets of inputs. The perplexity is expected to be smaller on samples seen during training. We extend this idea for our baseline radioactivity detection test of a non- won-watermarked text corpus. The corpus of texts is divided into sentences (of 256 tokens) and B's loss is computed on each sentence. We calibrate it with the zlib entropy [Roelofs] 2017], done by Carlini et al. [2021] for cample-based MIA. The goal of the calibration to account for the complexity of each sample and separate this from the over-confidence of B.
We test the null hypothesis Hy: "the perplexity of B on DA has the same distribution as the perplexity on new texts generated by A". Indeed, if B was not fine-tuned on portions of DA, then necessarily Ho is true. To compare the empirical distributions we use a two-sample
Taanoggrov-Smminnn test [Massey, 195T]. Given the two cumulative distributions F and G over loss values, we compute sup,]F((x) Fxx) G(x)). We reject Ho if this distance is higher than a threshold, which sets the p-value of the test, and conclude that DA is radioactive for B. This is inspired by Sablayrolles et al. 20088, who perform a similar K-S test in the case of image
the K-S distance as dks(F,.C
sup,l
Mean value Under 74g
Mean value Under He
1.00
1.00
Standard deviation
Standard deviation
0.75
0.75
p-value
p-value
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Number of generated tokens 1e6
Number of generated tokens 1e6
Figure 8: p-luuue under Ho with closed-model access. We fine-tune B on won-watermarked instructions. We prompt with watermarked instructions and score the disctinct (k + 1)-grams from the answers, but only if the k-gram was not part of the instruction. The average is close 0.5 (expected under Ho).
Figure 9: p-value under Ho with open-model access. We fine-tune 5 waternaermd instructions. We apply the open-model detection of See. 4 and score distinct (k + 1)-grams, but only on k-grams that B did not previously attend to when generating the token. The average is close to 0.5 (expected under No).
20

---

classification. It sigmificantly diverges from the approach of Shhi et al. [2023], which derives an empirical test by looking the aggregated score from one tail the distribution.
Experimental results. We proceed in Sec. 2 for the setup where MIA is achievable: Alice has an open-model access to B and is aware of all data DA generated for Bob (supervised setting). Bob has used a portion D for fine-tuning B, given by the degree of supervision d, as defined in Sec. We use the K-S test discriminate between the calibrated perplexity of B D D(o) containing 5k instruction/answers (cut at 256 tokens) that were not part of H's fine-tuning; and D(d) containing (1/d)x x5k instruction/answers from which 5k were. Distribution ((a) simulates what happens when Bob generates lot of data and only fine-tunes
few.
Experimentally, we use the K-S containing 5k instruction/ annseersss/essss (cut at 256 tokens) that were not part of H's fine-tuning; and ((a) containing [L/d] d) x 5k iestruction/answent from which 5k were. Distribution ((4) simulates what happens when Bob generates lot of data and only fine-tunes on a few.
discriminate between the calibrated perplexity B on: D(0)
Figure 10a compares the distributions for d = and 0. As decreases, the data contains more texts that Bob did not fine-tune on, so the difference between the two perplexity distributions is fainter. The direct consequence is that the detection becomes more challenging, Figure 10b shows that when d > 2%, the test rejects the null hypothesis at a strong significance level: p < 10 5 implies that when radioactive contamination detected, the probability of a false positive is 10-5. It is random in the edge case d = 0, the unsupervised setting where Alice lacks knowledge about the data used by Bob. In contrast, radioactivity detection on watermarked data succeeds that setting.
E.2 IP protection methods
There are active methods that embed watermarking in text specifically to detect model is trained on it (Zhao et al.] 2023 He et al. 2022b a]. Our setup is different because Alice's primary goal is not to protect against model distillation "radioactivity" is byproduct. Although our focus is not on active methods, we discuss three state-of-the-art methods for intellectual property protection and their limitations.
to make her outputs more identifiable. Consequently.
Zhao al. 2023). The goal is detect if a specific set of answers generated by Alice's model has been used in training. There are three main limitations. The authors design a watermark dependent on the previous prompt (and unique it). Each detection algorithm (2 and 3) assumes that the sample
training,
100%
log1o (p-value) (4 is better)
Density
10%
1
-10
1=1%
-20
Membership inference detection
Watermark detection
0.0
0.5
1.0
1.5
2.0 2.5
3.0
-30
zlib-calibrated loss from B
0.1
Supervision degree (in %)
(a) Distributions the calibrated loss of B across two types of distributions generated by A:: texts generated by A outside of B's fine-tuning data (purple), texts D of which d% were used dur- ing training (orange).
(b) We report the p-values of the K-S detection test (no WM when training) and of the WM detection WM when training) against degree supervision d (proportion of Bob's training data known to Alice).
5%
Figure 10: Comparative analysis of membership inference and watermarking for radioactivity detection, in the open-model setup. (Lefi) MIA aims to detect the difference between the two distributions. It gets harder as decreases, since the actual fine-tuning data is mixed with texts that Bob did not use. (Right) Therefore, for low degrees of supervision (< 2%), MIA is no longer effective, while WM detection gives p-values lower than 10 15
21

---

Table 9: p-values for non-watermarked instruction-answers
Number lines
Number of characters (in thousands)
p-value
0.76
100
30.9
0,16
500
148.3
2.2 x10
1000
333.9
8,8 x 10 13
Table 10: Summary statistics (mean and standard deviation) log (p) the watermark detection for different ranges of number of tokens constituting the text. Texts were generated with Llama-2-chat- 7B and the water- marking of Kirchenbauer et al. (2023b), with 3.0, 0.25, k 2, in Sec. P and each range contains
Rx500 texts.
(50. 150]
(150. 250]
250]
(350, 4501
(450, 5501
(550, 650]
(650, 750]
Range
Mean
-7.2
-11.7
-15.9
-18.4
-21.0
-24.0
-26.6
Std
3.6
5.6
7.2
8.8
10.2
12.6
14.2
probing data is from the training data of the suspect model Therefore, the method only demonstrated in the supervised setting. Moreover, all experiments performed in the open-model setting, where Alice has access to B's weights, except for Sec. 5.2 "Watermark detection with text alone" (still assuming supervised access), where one number is given in that setting. Finally, it does not rely on a grounded statistical test and requires an empirically set threshold.
He et al. [022aa] and He et al. [2022b]. These methods substitute synonyms during generation to later detect an abnormal proportion of synonyms the fine-tuned model. However, the hypothesis for building the statistical test the frequency of synonyms in natural text is fixed- fails when scoring a large number of tokens. Using the official author's 's code on non-watermarked instruction-answer yields extremely low p-values, making the test unusable
scale, shown in Table 9
F Additional Results
F.1
Qualitative examples
Figure 15 shows example of answers from Llama-2-chat-7B, when asked to generate intruc- tion/nnswer pairs. Figure 16 shows examples of answers from Llama-1-7B. after the instruction fine-tuning from 100k instructions, from which proportion is watermarked.
F.2
Evaluation of the watermarking
We evaluated in Sec. 5.2 the LLM fine-tuned on watermarked data. Table 10 desccrbes the results in terms of detection and quality of the text that used for fine-tuning. As reminder, texts were generated with Llama-2-chat-78 and the watermarking Kirchenbauer et al. [203b)) with o = 3.0, y 0.25 and watermark window 2, as Sec. For instance, the (ogio(--value) over 500 texts made of 50 to 150 tokens is at -7.2 on average.
Bigger teachers
We conduct an experiment using the 13B and 65B Llama-2-che models as teachers. The teacher generates instruction answer pairs using the watermarking method and parameters specified in paper (KGW,
0.25, å = 3.0), which are then used to fine-tune a Llama-1-7B model. Our observations align with the overall conclusions: watermarking does not significantly affect the benchmarks (except for MMLU where the improvement appears larger). The detection of radioactivity approximately the same (experiments conducted in the same setup in Tab, 6).
22

---

Table 12: Mixing instruction datasets from different sources. The line-tuning done with the setup pre- sented in Sec. with p=10% of watermarked data, mixing either with human or synthetic instructions.
Table 11: Results for different teacher models.
Teacher
without
7B
13B
65B
NQ
3.2
5,6
5,4
5,8
GSM8K
10.0
10.4
11.0
Major data source
Average log1o
MMLU
28.4
31.0
32.9
33.8
Machine
-15
1og10 p-value
-0.3
-32.4
-31.1
-31.7
Human
F.4
Mixing instruction datasets from different sources
We conduct an experiment where the non-watermarked instructions are human-generated (from the Open Assistant dataset OASST1 Köpf al. (2024)) ) The results intriguing: with the same proportion of watermarked data (10%), and in the exact same setting as the one explored in Sec. Sec. the radioactivity signal even stronger (see Tab, 12). Our speculation that this might be due to
fewer overlapping sequences of k+1 -+1-grams between the two distributions.
F5
Number of epochs
Figure 11 extends results presented in Tab. 6 The setting is similar to the one presented in Sec. 6 i.e., p 100% of instructions are generated with watermarking [Kirchenbauer al. 2023a]. We observe the radioactivity on N=10k tokens. As reminder, in Sec. |] we perform 3 epochs of fine-tuning, as done for Alpaca
[Toor al. 2023).
log10 (p-value)
20
-25
-30
35
2
3
4
5
Epochs
Figure 11: Detailed results for the influence of the number of epochs when B is fine-tuned on = 100% of watermarked data. The longer the fine-tuning lasts, the more the watermarking leaves traces in the model.
-2
median
X
mean
log (p-value)
-4
-6
8
Baseline
Unsupervised
Supervised
Figure Detailed influence the filter. Box plots the log1o(p) the closed-model setting with p 10%.
We perform the watermark detection test on text generated by B. The baseline uses the default scoring (no filters). In the unsupervised scenario, scoring is confined to k-grams generated in new watermarked data produced by A. In the supervised scenario, scoring limited to k-grams present in D-
23

---

Table 13:
Detection results in the closed-model selting, with and without filters scored k-grams. We report the mean and max (p) over runs. Filtering scored k-grams improves detection, even more so in the worst-case scenarios. See Fig. 12 for the corresponding box blots, and App. [ for experiment details.
Baseline
Unsupervised Supervised
log 10 (P) mean
-4.7
-5.2
-5.5
log ma)xxx max
-1.9
-2.4
-3.7
F.6 Impact of the filter
In both the supervised and unsupervised closed-model settings, the use of filter @ is paramount. As explained in section the watermark traces in B can only be detected in the k-grams that part of D (refer to subsection 2.2 for details on watermark embedding). Assuming that these k-grams are heavily watermarked and that B has memorized all of them, they still only represent small fraction of the total [v| k-grams that can be tested. To enhance detection, we define a set Φ of k-grams likely have been trained on. Tokens are only scored if their preceding k-gram window (the watermark context window used for hashing) is part of σ. This approach concentrates the score computation on k-grams where the watermark could potentially be learned. In the fully supervised setting (d = 1), consists of the k- d-grams used during training. i.e., all k --grams from D- In the unsupervised setting, we still focus on "likely" contaminated sets of tokens, for instance, k-grams that appear a new text generated by A with the watermark. Note that filter á is only used in the closed-model setting.
In addition to Fig. 6 shown in the main text, we show box plots in Fig- 12 To isolate the effect of the filter alone and to compare the results on different chunks, we use the same non-watermarked
prompts in all settings and analyze the same number of generated tokens N 1.5M answered by
B. Thus, for the supervised setting, it differs from what is done in Fig. and Fig. S where we use the watermarked prompts from D Both filtering methods show improvements compared the baseline. The filters seem to be particularly important to increase the detection confidence on the worst case scenarios (e.g., in our case, the biggest p-value observed over the 10 runs). Table 13) reports the same results in a table. Both figures correspond to k = 2 (setting of Sec. 5). Note that we expect the filter be even more efficient for higher values of k:
Open vs. Closed
Figure compaees detection in the open and closed-model settings, when 10% of fine-tuning data
watermarked. The setup is the one from Sec. 5 We plot the lo) against the number of
generated next tokens, averaged over 10 different runs. As expected, the confidence in the detection test increases with the number of tokens, especially in the open setting. For instance, 250k generated tokens, the average log (p) of the closed-model detection is at -3, while is at 12 for the open-mode model detection presented in Sec. 5
logio (p-value)
Open Closed
-10
-15
-20
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Number of generated tokens
1e6
Figure 13: (p) in the unsupervised setting with = 10% of B's fine-tuning data watermarked as function of the number of generated tokens. For the closed-model scenario, perform the watermark detection test on new text generated by B, and only score k k-grams that are often produced by .A's watermark.
24

---

G Generalization to Other Watermarking Methods
We discuss the generalization of our methods to other text watermarks. Sec. G.I focuses on other hashing-based methods, and the limit of our approach for non hashing-based ones. Sec. G.2 explores the radioactivity of multi-bit watermarking scheme.
G.1 Does the radioactivity generalize to other watermarking zero-bit schemes?
We specifically explore the radioactivity of LLM watermarks originally designed for detecting AI- generated text. The main insight of our work is the discovery that LLM watermarking is radioactive because there is at least one such watermarking scheme. Our goal is neither test the radioactivity of all watermarking schemes nor to identify the most radioactive scheme but to derive general method to detect radioactivity that can be used for similar decoding-baser watermarks.
Key management based on hashing. As in most of the LLM watermarking literature (Fu al. [2024a b). Hu et al. [2023], Kirchenbauer et al. [2023b], Kuditipudi et al. [2023], 1, Wu et al. [2023] Zhao et al. [2024] ...). we focus on the works of Aaronson and Kirchner [2023] Kirchenbauer et al.)
[2023a] that are representative of the 2 main families of methods. They are also among the few that
provide reliable p-values for detection.
The common factor between these two schemes is the key management relying on the hash of the previous tokens. We believe that any other scheme using the same mechanism is radioactive well. Lee et al. 20233 modify the watermark of Kirchenbauer et al. (0236) (green list/red list) so that works better for low entropic texts (the paper focuses on code). Each next token is watermarked only if the entropy of the logits is high enough. Similarly, at detection time, each token is scored only if the entropy for this token (computed by another LLM) is above certain threshold. Radioactivity can be detected using the methods derived in our work. Indeed, our filtering could also integrate selection based on the entropy. Fu et al. [2024a] use a similar idea than Aaronson and Kirchner (2023) but with another sampling function. There too, radioactivity can be measured with our tools.
Overall, the efficacy of radioactivity is intrinsically linked the robustness of the original watermark, as shown in Sec. 6 Section 5 Juses a watermark window size of A: = 2, which isn't 't the least robust nor the most robust scenario but realistic one for which derive the main results.
Other schemes. There are few L.M watermarking schemes not relying on hashing. For instance, some focus on "semantic" watermarks that depend on an entire past text's semantic representation [Liu et al.] 2023 Liu and Bu 2024 Fu et al. 2024b). [Kuditipudi et al., [202) represents another family of methods that do not operate via hashing but with pre-defined key sequences chosen depending on the token's 's position. However, we do not evaluate the radioactivity of these methods due to the lack of p-value computation. This limitation is also noted in other studies (see (Piet et al. 2023
paragraph
7.3. Limitations of the Building Blocks).
For example,Kuditipuue et al. [2023) compute a score using the Levenshtein distance of given block size whose complexity is (mmkk2) over m tokens, n watermark key sequences, and a block size If we approximate and 100 (k: 80 and n = 256 in their codebase). this implies that the complexity is 10t times greater than that of the schemes presented previously. Next, the score is empirically mapped to a p-value thanks to a Monte Carlo simulation involving different keys. To evaluate p-values of 10- 10 (as in the main paper), we would need to run this statistic over 1020 times, so it would require 1026 times more operations total. Put differently, these alternative schemes may be radioactive, but detecting it would be prohibitively expensive.
G.2 Multi-bit scenario
We adopt the same framework as in Sec. 5 but we use the watermarking method of Yoo et al. a.k.a., MPAC. It is a multi-bit method, where the watermark is a binary message of size n. More
2024
precisely, we take bits 2 by 2 to generate a message m m2m22 ... my with the r-ary = 0, 1, 2, or 3, corresponding to r = 4 and b = n/2 with the notations from the original paper. The method proceeds as the one of Kirchenbauer et al. (202b) by altering the logits before generating token.
However, the hash created from the watermarked window and the key is now used (1) to randomly partition the vocabulary into disjoint sets, (2) to select the position i and corresponding m, that is
25

---

hidden for this particular token. A bias ó is added to the logits of the tokens belonging to the m,-th set. Given text under scrutiny, the extraction reverses the process find which r-ary is the most likely for each position of the message.
We report in Fig. 14 the extraction results in the the superviieadlcoosd--mdde setup. We filter and deduplicate the tokens as in Sec. [ and plot the observed bit accuracy against the number of scored tokens note that since the watermark is binary message, now measure the bit accuracy of the extraction, instead of the p-value of the detection. This is done for several lengths of the binary message. Every experiment run times for different text output by B, which explains the 95% confidence interval in the plots. We observe as expected that the bit accuracy significantly increases with the proportion of watermarked data in the fine-tumin inn-tuning data, and that the longer the message, the harder it is to extract it. This suggests that radioactivity still holds in the multi bit scenario, and could therefore used to identify a specific version of the model or specific user from which the data was generated.
0%
p 10%
P 100%
Bit accuracy
0.8
8
16
0.4
32
20000
40000
20000
40000
Number scored
Number scored tokens
Number scored tokens
Figure 14: Bit accuracy when watermarked instruction data generated with MPAC Yoo et al. 2024 against the number of scored tokens generated by the fine-tuned model. This is done under the supervised/llosed modee model setup. various lengths (z=8, 16, 32) of message.
26

---

Prompt given to Llama-2-chat:
You are asked to come up with set of 20 diverse task instructions. These instructions will be given to large language model and we will evaluate for completing the instructions.
Here are the requirements:
1. Try not to repeat the verb for each instruction to maximize diversity.
2. The language used for the instruction also should be diverse. For example, you should combine questions with imperative instructions.
The type instructions should be diverse. The list should include diverse types of tasks like open-ended generation, classification, editing, ctc.
2. A language model should be able to complete the instruction. For example, do not ask the assistant create any visual audio output. For another example, do not ask the assistant wake you up Spm or set a reminder because cannot perform any action.
3. The instructions should be in English.
4. The instructions should be 1 to 2 sentences long, Either an imperative sentence or a question is permitted.
You should generate an appropriate input to the instruction. It should contain specific example for the instruction when needed. 6. Most of the instructions do not require input. In this case, you must simply put "None" in the input field.
7. The input should not be the answer the instruction. For example, if the instruction is "Write headline for ...". the input should not be the said headline. Instead, the input should be "None". The output should be an appropriate response to the instruction and the input. Make sure the output is less than 100 words.
9. Do not repeat instructions.
List of tasks:
Instuction: Give a detailed description of a method for brainstorming an essay.
Context:
Answer: Mind Mapping is one of the methods for brainstorming essay. Students who visual learners may find mind mapping extremely helpful. In order to Mind Mapping:
Take a paper and draw a circle the center Write your topic inside it. Just like we sketch rays protruding from a sun, draw lines coming out of the circle. Write an argument or statement related to the topic the end of each line. Draw another set of lines projecting from these statements and state examples that support them. Continue this exercise until you run out of all possibilities imaginable for this topic.
Llama-2 answer:
Instuction: Write a creative or imaginative response to the given prompt.
Context: are time traveler who has just arrived in ancient Egypt. What do you see?
Response: Wow! As step out of my time machine, I'm surrounded by towering pyramids and the warm sun beats down my face. see Pharaohs riding in grand chariots, and workers building majestic temples. It's like something of dream!
Instuction: Solve a math problem. Context: A snail is at the bottom of 20-foot well. Each day. it climbs 3 feet, but at night, it slips back 2 feet. How many days will it take for the snail to reach the top of the well? Response: It will take the snail 10 days to reach the top of the well.
Llama-2 answer wm Kirchenbauer al. [2023a] 8 = 3, k 2:
Instuction: Create a poem based on the provided word.
Context: Word: "Nature".
Response: Nature's 's graceful symphony / Echocs through the forcst, wild and free Birds sing sweet melodies / As leaves rustle with glee Nature's wonders, sight to behold/ / A reflection of pure artistry
Instruction: Write a persuasive email convince a friend to try new hobby. Context: '...
Response: Dear [Friend's Name]/ hope this email finds you well. wanted to suggest that you try new hobby to expand your horizons and add some excitement to your life. [Hobby Name] is fascinating activity that has really helped grow and learn new skills. Not only does it provide great opportunity for relaxation and stress relief, but also offers a chance to meet new people and make meaningful connections. think you would really enjoy it and I would love to see you give it a shot. Let me know if you have any questions or if you want to learn more.
Figure 15: Example of instruction data using Self-instruct. We show the prompt which is given to I Jlama-2-chat 7b, and the completion with or without the watermarking used in Sec. 5 We don't observe noticcable quality decrease.
27

---

of fossil Question: "How can reduce air pollution?"
Llama-1 answer not fine-tuned:
can be done by planting more trees.
Which is a better medium for communication?
Telephone is better medium than emails.
What is the final temperature of 500-gram block of ice when placed in 500-gram block of water?
The final temperature the 500-gram black of will 0%.
What should the temperature of 200-gram ice-cream be to achieve maximum enjoyment? The temperature of the ice-cream should be 20".
Llama-1 answer fine-tuned with = 0% of watermarked data:
Reducing air pollution can be achieved by using public transportation. carpooling, using renewable energy sources, and reducing waste. Additionally, individuals reduce their own impact by using reusable bags, bottles, straws, and avoiding use of plastic containers and packaging.
Llama-1 answer fine-tuned with o 10% watermarked data:
One way to reduce air pollution is by using public transportation or carpooling instead of driving solo. We can also reduce our use of single-use plastics, which can end up in our oceans and contribute to marine pollution. Additionally, we use energy-efficient appliances and light bulbs, and try reduce overall energy consumption.
Llama-1 answer fine-tuned with 100% of watermarked data:
To reduce air pollution, can switch renewable energy sources, reduce fuels, increase our use of public transportation. can also reduce our use of plastic and other harmful chemicals, and plant more trees to help absorb carbon dioxide from the air.
Question: Create concise definition for the term "sustainability"
Llama-1 answer not fine-tuned:
"Sustainability" is a term used environmental science to describe an ecosystem that is able to sustain itself. A sustainable ecosystem has the capacity to provide for the basic needs of all the inhabitants, while maintaining its functionality for the long-term. Explanation: The sustainabilityy an ecosystem based on its ability to provide for the basic needs all the inhabitants, while maintaining functionality for the long-term. This means ccosystem is able to provide the required space, food, and water for all the inhabitants of the ecosystem, Altemntive:: The sustainability of ecosystem is based its ability provide for the basic needs of all the inhabitants, while maintaining its functionality for the long-term. This means the ecosystem is able to provide the required space, food, water for inhabitants of the ecosystem Explanation: The sustainability of an ecosystem is based on its ability to provide for the basic needs of all the inhabitants, while maintaining its functionality for the long-teru term. This means that the ccos...
Llama-1 answer fine-tuned with 0% of watermarked data: Sustainability refers the ability maintain support process system without depleting natural resources or causing harm the environment.
Llama-1 answer fine-tuned with p 10% of watermarked data:
Sustainability refers to the ability maintain or support a process or system without depleting natural resources or causing harm to the environment.
Llama-1 answer fine-tuned with م 100% of watermarked data: Sustainability refers to the ability to maintain or support a process or system without depleting natural resources or causing harm to the environment.
Figure 16: Example generated answers from Bob's model B (Llama-1), fine-tuned on instruction data generated by Alice's model A (Llama2-chat) with different proportions of watermarked data. See Figure [5 for example of instructions used for instruction-tunde tuning.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction reflect that (1) we study the
problem of knowing whether outputs of model were used to train another model, (2) derive methods to detect radioactivity with reliable statistical guarantees and (3) it can be used to identify model imitation in realistic scenario.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See section A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: The main theoretical result is given in Sec 4 with a proof in App. D.2 Experiments are made in the text to validate the reliability of theoretical p-values (Tab. D. and details are given in App. C and App. D.2
D.2
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give exhaustive information on how to reproduce the experiments presented in the paper in sections and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No] (Not fully)
Justification: Including checkpoint of a 7B model derivative of Llama necessitates agree- ments, and without it, reproducing the results would not be possible. We nevertheless provide code to perform radioactivity detection. For the instruction generation code, we refer to the code of Wang et al. 2022 and for the fine-tuning code, refer to Taori et al. [2023].
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
guwdes//dee/eee Codeeubmeeiieee wwwibbbmisssonPoo ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The finetuning parameters are not optimized (we use canonical values). We also detail our choice for the watermark parameters. Refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For most of our experiments, we plot results averaged over several runs, and show error bars (we disclose the details of their computations in App. D.I).
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: see App. B.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ltttps://neurips... hwtps//c/cc/cctteceecetiiiioaea
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
App.
32

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not pose such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use the Llama family models for training and synthetic data generation (Llama 2 Community License), and sentences from Wikipedia in Sec. 6 or OASSTI in
App. which have permissive license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
33

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/datases has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "FXuKcLNKtj", "reviewer": "Reviewer_q8YZ", "summary": "The paper investigates the detection problem of whether LLM-generated texts are used to train another LLM, a phenomenon referred to as 'radioactivity'. The paper finds that it is feasible to detect the radioactivity of LLM-generated text via LLM watermarking. Consequently, the authors design radioactivity detection methods for four scenarios: closed-model, open-model, supervised setting, and unsupervised setting. In the experiment section, the authors present reliable detection under only 5% watermarked training texts, validating the effectiveness of their designed methods.", "questions": "See the Weaknesses part", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-structured and easy to follow.\n2.\tThe topic of this paper, i.e., the radioactivity of LLMs-generated text, is interesting.\n3.\tThe paper conducts extensive experiments and provides an in-depth analysis.", "weaknesses": "1.\tAlthough the paper designs detection methods for four scenarios, no new methods are actually proposed. The methods in the paper still follow the watermark detection approach and are merely applied in different scenarios. In other words, the so-called new methods in the paper are only about how to construct scenarios to better detect radioactivity.\n2.\tThe paper lacks exploration into the impact of the latest watermarking methods, especially multi-bit watermark, such as [1] and [2]. The multi-bit watermark better align with the requirements of real-world scenarios\n3.\tThe overall contribution of the paper appears relatively weak, as some existing model protection methods (e.g., [3]) have already explored the impact of watermarking on radioactivity detection. Although the settings in [3] differ somewhat from those in your paper, similar results can still be obtained.\n\n**References**\n\n[1] Towards Codable Watermarking for Injecting Multi-bits Information to LLMs\n\n[2] Advancing Beyond Identification: Multi-bitWatermark for Large Language Models via Position Allocation\n\n[3] Protecting language generation models via invisible watermarking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We have addressed each point individually. We kindly invite the reviewer to refer to Appendix A (\"Limitations\") for details on weaknesses 1 and 2, and Appendix F (\"Comparison to Active IP Protection Methods\") for the comparison related to weakness 3. **We also provide additional experiments to address weakness 2**, and if it addresses some of the reviewer’s concern, we would greatly appreciate it if they could consider updating their rating to reflect this.\n\n> 1. Although the paper designs detection methods for four scenarios, no new methods are actually proposed. The methods in the paper still follow the watermark detection approach and are merely applied in different scenarios. In other words, the so-called new methods in the paper are only about how to construct scenarios to better detect radioactivity.\n\nWe indeed adapt existing watermark detection techniques to tracing data usage by other models. But our proposed methods are not only to \"construct better scenarios to detect radioactivity”. We show that the naive approach **does not work**: filtering / deduplication is necessary to score millions of tokens and get reliable p-values (see Table 4 and Appendix D.2.2). Without this novelty, it is not possible to demonstrate watermark radioactivity, which is a main discovery of our paper. \n\nOur methods are not trivial adaptations either: contrary to the classic detection setup analyzing if a given piece of text is watermarked, we analyze whether an LLM is contaminated. For instance we are the first to use the model itself to detect traces of the watermark (open-model scenario).\n\n> 2. The paper lacks exploration into the impact of the latest watermarking methods, especially multi-bit watermark, such as [1] and [2]. The multi-bit watermark better align with the requirements of real-world scenarios\n\nWe have addressed the selection of watermarking methods and their implications in Appendix A (\"Limitations\"). In summary, we focused on these two methods (Kirch., Aar) because a lot of watermarking methods build upon them (Fu et al. [2024a,b], Hu et al. [2023], Kirchenbauer et al. [2023b], Kuditipudi et al. [2023], Wu et al. [2023], Zhao et al. [2024], Yoo et al. [2024], …)\n\n\nThe specific case of multi-bit watermarking is indeed interesting, and we provide 2 elements of answer:\n- we disagree with the claim that “multi-bit watermark better aligns with … real-word scenarios' ':  regulations (EU AI Act, California Act, BH Act, …) only require AI generated content detection. Moreover, tracing users may be forbidden under the GDPR in Europe for instance.\n- our experiments cover the schemes of Kirchenbauer et al. and Aaronson et al. that have been extended to multi-bit (see Fernandez et al [2023]), so our detection test should transfer to multiple messages (corresponding to rolled versions of a secret key) as well.\n\nAdditionally, we provide **new experiments with [2]** “Advancing Beyond Identification: Multi-bit Watermark for Large Language Models via Position Allocation”, aka MPAC, which was mentioned in the review. Please refer to the PDF for the corresponding figures.\n\n\nIn these experiments, we adopt the same framework as in Sec. 5: \"Radioactivity in Instruction Datasets”. We generate watermarked instructions from A=Llama2-chat with a random binary message of size `len_msg` (more precisely, we take bits 2 by 2 to generate a message m = m1, m2, .. mk where mi = 0,1,2 or 3, corresponding to r=4 in MPAC and b = `len_msg`//2 ). \nWe then fine-tune B=Llama1 with instructions, ρ% of which are watermarked with the above method.\nFinally, we detect its radioactivity in the supervised / closed-model setup, i.e., access to the data used for fine-tuning and no access to the model. We filter and deduplicate the tokens used in the prompts as explained in Sec. 4.2, par. “Token scoring and de-duplication.”\n\nWe plot in Fig. 1. (a) (ρ = 100% of watermarked fine-tuning data) and (b) (ρ = 10% of watermarked fine-tuning data) the bit accuracy against the number of scored tokens that we are able to obtain from the fine-tuned model. This is done for several lengths of the binary message.\nFurthermore, we provide the same curves in a control experiment where the key is different than the one used for training, to ensure that the bit accuracy is approximately 0.5 as it should be under H0.\nWe observe that the bit accuracy:\n- is significantly higher for the model trained on watermarked instructions, and random otherwise\n- is higher for smaller messages,\n- is lower when ρ is lower, \n- is higher the more tokens we score.\n\nNote that every experiment is run 10 times for different text output by B, which explains the 95% confidence interval in the plots. \n\n> 3. The overall contribution of the paper appears relatively weak, as some existing model protection methods (e.g., [3]) have already explored the impact of watermarking on radioactivity detection. Although the settings in [3] differ somewhat from those in your paper, similar results can still be obtained.\n\nWe disagree: the related work section (2.1), Appendix A (\"Limitations\"), and Appendix F (\"Comparison to Active IP Protection Methods\"), **discuss [3] specifically**. p-values become meaningless when scoring a large volume of tokens in their setting. This is a key contribution of our paper, highlighting the limitations of existing IP protection methods in various scenarios where similar results to ours **have not** been obtained to our knowledge. Conversely, we demonstrate that established watermarking techniques, enhanced by our radioactivity detection methods, lead to reliable detection guarantees, and are broadly applicable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5J2mn9ZkqF", "reviewer": "Reviewer_Y2ta", "summary": "The paper investigates the \"radioactivity\" of text generated by large language models (LLMs), focusing on the detectability of synthetic text used as training data. It introduces a novel method to reliably identify whether the outputs of a watermarked LLM have been employed to fine-tune another language model. The study reveals that watermarking in LLMs is radioactive, allowing for the detection of weak watermark signal residuals in fine-tuned models. The authors link the level of radioactivity to watermark robustness, its proportion in the training set, and the fine-tuning process. Notably, the research demonstrates that training on watermarked synthetic instructions can be detected with high confidence, even when a small percentage of the training text is watermarked. The paper contributes radioactivity detection methods for different scenarios based on model access and training data, showing how to obtain reliable p-values for watermark detection and proving the practicality of detecting radioactivity in realistic settings.", "questions": "See weaknesses.", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a novel study of the \"radioactivity\" of watermarked text generated by Large Language Models (LLMs), with a particular focus on the detectability of such text when used as training data for fine-tuning other models.\nThe paper makes a significant contribution by designing radioactivity detection methods for various scenarios based on model access (open/closed) and training data exposure (supervised/unsupervised).\nAmong other things, the authors effectively relate the level of radioactive contamination to key factors such as watermark robustness, the proportion of watermarks in the training setting, and the fine-tuning process, and the authors innovatively propose to utilize filtering and de-duplication for detection enhancement. The results show that even a small percentage of watermarked synthetic instructions in the training data (as low as 5%) can be detected with high confidence with a p-value of less than 10^-5.\nThe proposed method provides more reliable statistical guarantees for detecting whether LLM outputs are used in the training dataset than existing methods such as membership inference or active IP protection. The study also provides valuable insights on the effects of fine-tuning parameters and watermarking methods on radioactivity, which helps to deepen the understanding of the underlying mechanisms.", "weaknesses": "1.   The author's lack of proper consideration of the written rigor of the essay is evident here in many places where semantic or formatting errors are made. For example:    (1) Uniformity of punctuation: in section 2.1, Related work (MIAs) should be formatted with the context, missing periods, similar missing periods are also found in the second sentence of the response in Figure 4 ρ = 0% in Section 5.3, and in the two bolded subheadings in Appendix C.3.    (2) Uniformity of formatting: in the last question/answer pair in Figure 15 of Appendix H.7, Context content is missing line breaks; in the main text section 2.2, there has been a formula with the formula number 1, and the formula in Appendix C.1 below is numbered 2, but the formula located in the middle, under appendix C, is missing the formula number; in Table 1 \"closed\" in \"Open/closed\" needs to be capitalized.    (3) Content flaws: in the first scenarios of Problem Formulation Access to Bob's model in Section 3, the tense of \"open-sources\" is not used properly; also in Section 3, for Definition 1, \"B was not trained on D\", according to the author's definition of data labeling, there is an error in D here; in Figure 2, the last sentence (after and) lacks the necessary verb.    (4) Unification of singular and plural:This article suggests that the authors should standardize the singular and plural, for example, \"text generatedf by **\" is used in the abstract and some other places. However, there are some places where \"texts generatedf by **\" is used, and the same phenomenon also occurs in \"data/datas\".\n\n2.    The watermark detection method used in this paper is only given for specific fine-tuning cases, can the method achieve the same anomaly detection effect if the model is fine-tuned in a more complex or covert way?\n\n3.    In Section 5, the article emphasizes the consideration of a realistic scenario for watermark detection, but it does not discuss in detail how this would affect the \"radioactivity\" if someone attempted to remove the original watermark from the training data, which is likely to be more common in reality as well.\n\n4.    The authors mention in the paper that they rely on the proofs in previous papers, but they should give some necessary information to make the paper logically coherent, for example, in Appendix C.1, the authors cite kirchenbauer et al. For the design of LLM Watermarking, which mentions that \"The logit of every token in the greenlist is incremented by δ.\". However, the definition of δ is not mentioned before, but in the original paper, kirchenbauer et al. proposed the algorithm \"Text Generation with Soft Red List\", in which the parameter δ is quoted to modify the logit and get the probability distribution of words.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions.\n\n> 1. The author's lack of proper consideration of the written rigor of the essay is evident here in many places where semantic or formatting errors are made. For example: (1) Uniformity of punctuation: [...] for example, \"text generatedf by **\" is used in the abstract and some other places. However, there are some places where \"texts generatedf by **\" is used, and the same phenomenon also occurs in \"data/datas\".\n\nWe thank the reviewer a lot for the careful consideration given to the manuscript. We agree with all of them and we will update it accordingly.\n\n> 2. The watermark detection method used in this paper is only given for specific fine-tuning cases, can the method achieve the same anomaly detection effect if the model is fine-tuned in a more complex or covert way?\n\nWe address additional fine-tuning scenarios in Section 6.1 and Table 6. While not exhaustive, these examples illustrate that increased data fit enhances model radioactivity. For further details, please see Appendix H (Additional Results), which includes ablations on \"Bigger teachers,\" \"Mixing instruction datasets from different sources,\" and \"Radioactivity purification.\" We welcome suggestions from the reviewer on more \"complex or covert way[s]\" of fine-tuning.\n\n> 3. In Section 5, the article emphasizes the consideration of a realistic scenario for watermark detection, but it does not discuss in detail how this would affect the \"radioactivity\" if someone attempted to remove the original watermark from the training data, which is likely to be more common in reality as well.\n\nWe addressed this important point in Appendix A (Limitations), noting that radioactivity correlates with watermark robustness; attempts to paraphrase or alter the watermark will indeed weaken radioactivity.  In Appendix H.3 (\"Radioactivity Purification\"), we observe that if Bob fine-tunes his LLM on non-watermarked data to deliberately remove traces of the watermark, the radioactivity decreases (but remains detectable), illustrating a similar scenario to what the reviewer described. \n\nHowever we argue that compromising the quality of Alice's high-quality LLM outputs through paraphrasing may not be a \"common” approach, as (1) it could degrade the fine-tuning benefits, and (2) Bob might just not know that Alice’s outputs are watermarked.\n\n> 4. The authors mention in the paper that they rely on the proofs in previous papers, but they should give some necessary information to make the paper logically coherent, for example, in Appendix C.1, the authors cite kirchenbauer et al. For the design of LLM Watermarking, which mentions that \"The logit of every token in the greenlist is incremented by δ.\". However, the definition of δ is not mentioned before, but in the original paper, kirchenbauer et al. proposed the algorithm \"Text Generation with Soft Red List\", in which the parameter δ is quoted to modify the logit and get the probability distribution of words.\n\nWe will add a definition of δ to the manuscript : \"For instance, Kirchenbauer et al. [2023b] create a \"greenlist” of tokens whose logits are augmented by a quantity δ, increasing their sampling probability.” in section 2.2. We thank the reviewer for pointing this out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z62IrdufyX", "reviewer": "Reviewer_ZoLZ", "summary": "The paper studies the \"radioactivity\" of watermarked texts, i.e. if using such texts in LLM finetuning leaves noticeable watermark signal that can be reliably detected in future outputs. The main case study considered is the common scenario of using LLM-generated data for IFT. Authors use off-the-shelf LLM watermarks but propose several improvements to the standard watermark detection pipeline, demonstrating that they are necessary to obtain correct and sufficiently small p-values for radioactivity detection. The experiments consider four different threat models and include several additional studies of radioactivity across different dimensions.", "questions": "- Where do the unwatermarked instructions used for B (1-rho percentage) come from? Using A without the watermark may introduce unnecessary entanglement, so I hope these pairs are fully independent, e.g., human written.\n- is there a reason why the open setting studies only d=1? Line 231 is unclear as it may temporarely make the reader think that \"unsupervised\" <=> \"d=1\" which is not true. This choice should be discussed.", "limitations": "The authors adequately discuss the limitations in one of the appendices.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I very much enjoyed reading this paper and believe it is an impactful contribution to the field. I highlight the key strengths below.\n- **Novelty, focus on an important underexplored problem**: While individually all the following fields have been studied, to the best of my knowledge no prior work studied the current wave of LLM watermarking research from the perspective of active methods for tracing unauthorized data use and data radioactivity. Thus the authors identified an important gap in the literature.\n- **Impact for different subcommunities**: The work can be valuable for several mostly disjoint subcommunities with different foci such as: design of new LLM watermarking schemes, OSS watermarking, passive tracing of unauthorized data use (membership inference), active tracing (e.g., backdooring), model IP protection, but also model distillation, and instruction fine-tuning as such. This holds independent of the \"strength\" of the final takeway -- even if radioactivity does not occur such a study is equally valauble.\n- **Thorough exploration of the problem nuances**: I appreciate the careful setup of 4 settings and a thorough discussion of when they are realistic/important, the extensive additional studies, and the detailed comparison to related and adjacent work such as membership inference attacks.\n- **Rigorous and extensive experimental evaluation**: The evaluation part offers many clearly communicated insights that arise from carefully constructed evaluation scenarios and I particularly appreciate that nothing is \"swept under the rug\", especially the issue of mismatched empirical and theoretical p-values. The appendices cover many of the additional questions that the reader may have.\n- **High-effort writing and presentation**: The paper is exceptionally well written and structured, the information is logically organized, and special care is taken to provide figures to ease understanding. Even when simple (e.g., Fig2) these steps help with reading.", "weaknesses": "I can identify several weaknesses, none of which are fundamental. \n- The last abstract sentence renders as an overclaim given that it applies only to the less realistic open case, while most readers would assume the more realistic closed case, where the number is 10%. I strongly suggest the authors make this clear in all places, as this number does not really affect the merit of the work.\n- Main results (e.g. Fig5) compare results on N=225k tokens (open) and N=500k tokens (closed) models. As shown in Figure 14 increasing N improves the performance of detection, thus this comparison on different levels of N is misleading. Why did the authors not use the same N in both experiments?\n- Table 1 is insufficiently discussed and hard to understand just from the main paper but makes some strong claims. In particular: (i) \"Without WM (MIA)\" would make the scope of these columns clearer (ii) it is unclear if \"X\" means \"fundamentally inapplicable\" or \"achieves bad results\". MIA+unsup. is latter but IPP+unsup. is former, with the caveat that we focus on current methods and not IPP in general. (iii) MIA+Closed are essentially \"label-only\" attacks (\"Label-Only Membership Inference Attacks\", Choquette-Choo et al. 2020) so this is not fundamentally inapplicable, while it may be that for LLMs no such technique vas demonstrated viable; this should be made clearer. (iv) The tilde is unclear, after reading the appendix I take this to mean \"very limited results demonstrated in this setting + technical issues when trying to reproduce\". All this should be more carefully unpacked as it is important to position the current work. \n- L170 is in the context of \"our contributions\" yet states an approach (\"ignore repeated ngrams\") that is very common in prior work. A citation (given later) should already be given here to clarify which part of the Tape is novel.\n\nL8 typo: \"methods\"+\"detects\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback on the paper, as well as valuable questions and comments.\n\n> 1. The last abstract sentence renders as an overclaim given that it applies only to the less realistic open case, while most readers would assume the more realistic closed case, where the number is 10%. I strongly suggest the authors make this clear in all places, as this number does not really affect the merit of the work.\n\nWe agree with this comment and will modify the abstract and the claims in the paper in consequence.\n\n> 2. Main results (e.g. Fig5) compare results on N=225k tokens (open) and N=500k tokens (closed) models. As shown in Figure 14 increasing N improves the performance of detection, thus this comparison on different levels of N is misleading. Why did the authors not use the same N in both experiments?\n\nA reason for this choice is to show that the open model setting does not need as many tokens as in the closed model setting and is thus more effective. This can be observed in Figure 5, where lower p-values are achieved with fewer tokens than with the closed model. \n\nAnother reason is that p-values plateau beyond a certain point (see Figure 6), so scoring more tokens in the open model does not significantly enhance detection. \n\nWe acknowledge the potential for misunderstanding as it mixes two effects (open vs closed and number of tokens) and the numbers may appear arbitrary. We will clarify this in our revision.\n\n> 3. Table 1 is insufficiently discussed and hard to understand just from the main paper but makes some strong claims. In particular: (i) \"Without WM (MIA)\" would make the scope of these columns clearer (ii) it is unclear if \"X\" means \"fundamentally inapplicable\" or \"achieves bad results\". MIA+unsup. is latter but IPP+unsup. is former, with the caveat that we focus on current methods and not IPP in general. (iii) MIA+Closed are essentially \"label-only\" attacks (\"Label-Only Membership Inference Attacks\", Choquette-Choo et al. 2020) so this is not fundamentally inapplicable, while it may be that for LLMs no such technique vas demonstrated viable; this should be made clearer. (iv) The tilde is unclear, after reading the appendix I take this to mean \"very limited results demonstrated in this setting + technical issues when trying to reproduce\". All this should be more carefully unpacked as it is important to position the current work.\n\nWe discuss this point in section 5.5 \"Other approaches\" and in Appendix E and F, but agree that it could appear sooner in the paper, for instance when we present the table.\nIn this table: \n- “X” means that no method in the literature currently tackles this problem with LLMs; \n- “~” means that the only methods that address the problem have strong technical issues when trying to reproduce:the statistical guarantees do not hold.\n\nWe will emphasise this for better clarity.\n\n> 4. L170 is in the context of \"our contributions\" yet states an approach (\"ignore repeated ngrams\") that is very common in prior work. A citation (given later) should already be given here to clarify which part of the Tape is novel.\n\nAlthough [KGW2023] and [FCT2023] introduce the deduplication of tokens, the tape is indeed different in our paper as we also care about the prompts (closed-scenario) and about the LLM context (open-scenario), so we must also properly deduplicate these tokens in a way that is specific to our detection (proven in Tab. 4 and appendix D.2.2 “Correctness”). We will update this paragraph accordingly to clarify this contribution.\n\n**References:**\n\n[KGW2023] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023a.\n\n[FCT2023] Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. 2023 IEEE International Workshop on Information Forensics and Security (WIFS), 2023.361", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JqqEHBoXxr", "reviewer": "Reviewer_wG1j", "summary": "This paper proposes a method to detect whether a language model is trained on (a subset of) watermarked outputs from another victim model. Their method utilizes the fact that the watermarking schemes are shifting the output tokens' distributions, such that the model trained on the watermarked outputs will also have such distribution-shifting behaviors. The authors propose detecting such shifting to determine if the model's training dataset contains watermarked content. The promising results illustrate the effectiveness of their method with a high accuracy.", "questions": "- My major concern is the generalizability of the method. Given that the KGW [ICML'23] variant watermarks are not distortion-free, which in theory leaks the watermarked tokens' distribution. Kuditipudi et al. [2023] proposed a watermarking scheme that does not rely on the splitting of green and red lists. I think it might be hard to generalize the method to this watermark as it is proven to be distortion-free. Can the authors comment on this?\n\n- Additionally, the false positive rate is an important metric in such detection systems. From the results in Figure 5, the p-value is not large enough when $\\rho=0$. Can the authors provide further explanations on this?\n\n- Minor: The presentation can be further improved. For instance, you can put the images near the text where they are referred to for the first time. So that the readers do not need to jump back and forth.", "limitations": "See my concerns in the Question section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is sound with promising evaluation results.\n- The studied problem is important and timely.", "weaknesses": "- The proposed method seems limited to the green-red list splitting-based watermarks.\n- The presentation can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback on our paper. We address each point specifically. Please note that Appendix A \"Limitation\" and Appendix C.3 \"Does the radioactivity generalize to other watermarking schemes?\" address some of the concerns. Importantly, we emphasize the reliability of our p-values in Section D.2.2 \"Correctness experiments.\" We believe the reliability of our p-values is one of our paper's main contributions and hope it addresses the reviewer’s concerns.\n\n> My major concern is the generalizability of the method. Given that the KGW [ICML'23] variant watermarks are not distortion-free, which in theory leaks the watermarked tokens' distribution. Kuditipudi et al. [2023] proposed a watermarking scheme that does not rely on the splitting of green and red lists. I think it might be hard to generalize the method to this watermark as it is proven to be distortion-free. Can the authors comment on this? \n\nWe invite to reviewer to refer to the general comment and the aforementioned appendices for a discussion on generalization. There is specifically a discussion on the Kuditipudi et al. [2023] scheme. To summarize, while this scheme may exhibit radioactivity, detecting it would be prohibitively expensive, requiring $10^{16}$ times more resources to properly evaluate p-values. Additionally, we analyze the scheme of Aaronson et al. in Section 6, which does not rely on green/red lists. This should address some concern about generalization. It is true however that we only focus on the most prominent hashing-based watermarking methods. \n\nNote about \"distortion-free\": \nThe term is not yet perfectly agreed upon in the literature. Aaronson’s scheme is said to be “distortion-free” by the authors in the sense that the probability of selecting a token is the same on expectancy over the random vectors. On the contrary, Kuditipudi et al., oppose hashing-based methods (like the one of Aaronson et al.) and distortion-free methods. For them, hashing previous (k-1)-grams to create the secret vectors produces a bias towards certain k-grams, which introduces a distortion, while using a sequence of vectors as secret keys does not. We would argue that both methods are distortion-free only under the limit where the watermark window (for Aaronson et al. and Kirchenbauer et al.) or where the sequence length (for Kuditipudi et al.) are big enough. There is an interesting discussion about this in 4.1 of [Kuditipudi et al.]\n\nWe also invite the reviewer to take a look at our attached pdf **were we show that even multi-bit watermarking are radioactive**, in a response to a question raised by reviewer q8YZ. \n\n> Additionally, the false positive rate is an important metric in such detection systems. From the results in Figure 5, the p-value is not large enough when 𝜌=0. Can the authors provide further explanations on this?\n\nWe agree that assessing the FPR is crucial, and this is why it was important for us to get reliable p-values. There is a direct link between the FPR of the detection system and the p-values obtained for different samples. Given an LLM for which the statistical test gives a certain p-value, we would flag the LLM as radioactive at every FPR ≥ p-value. For instance, if all samples have p-value ≤  $10^{-6}$, then the observed TPR would be 1.0 at FPR=$10^{-6}$.\n\nWhen 𝜌=0%, there is no watermark in the training dataset (H0), so the detection should output an average p-value of ≈0.5 since p-values under H0 should be uniform between [0,1] (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6629378/#s2title). This is what we observe in Figure 5.\n**In Table 4 \"Average p-values under H0\" and Appendix D.2.2 \"Correctness Experiments”, we specifically focus on this** and show that the test yields random p-values under H0 in all considered settings, validating the reliability of our p-values. \n\n\n> Minor: The presentation can be further improved. For instance, you can put the images near the text where they are referred to for the first time. So that the readers do not need to jump back and forth.\n\nWe will update the paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFaoGlG4ZS", "reviewer": "Reviewer_JVYy", "summary": "This paper considers the problem of detecting whether watermarked text was used as training data for a language model. It identifies two several different settings under which to study this question, and proposes detection methods for identifying language models trained on watermarked text. Experiments analyze the effectiveness of these detection methods across a variety of settings using a particular configuration of the Kirchenbauer watermark, and briefly analyze the Aaronson watermark in the closed-model setting.", "questions": "Regarding the \"supervised\" and \"unsupervised\" settings: this word choice was initially quite confusing to me. These works typically refer to training regimes (with or without labels) but training is not being studied in this paper. Would \"observed\" vs. \"unobserved\" be more clear terminology?\n\nIs Table 5 the supervised or unsupervised setting? Why is the watermark more detectable in model outputs (Rad) than in the training data (Orig)?\n\n> detection tests can be empirically inaccurate due to biases in the distribution of tokens, violating the independence assumption.\n\nWhat independence assumption? Perhaps this is discussed in Fernandez et al. but in the present context it's not clear what is being assumed.\n\n> radioactivity can only be observed for watermarked (k + 1)-grams {watermark window + current token} that were part of A’s outputs in B’s training data\n\nIs this true? For Kirchenbauer-style watermarks at least, it seems like a weaker observation could hold, because there is a distribution shift.", "limitations": "Definitions of \"text radioactivity\" and \"model radioactivity\" are introduced in Section 3, but the experiments seem to exclusively study model radioactivity.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduction, background, and problem formulation are well-motivated and easy to read. \n\nIdentifying that a model has been trained on watermarked text is an interesting problem, of broad relevance to the community.\n\nEmpirical results show some evidence of detectability of hash-based watermarks, especially in the supervised setting (Figure 5).", "weaknesses": "The algorithms in Sections 4--the central contribution of the paper--are not well described. What explicitly is the null hypothesis for which we are computing a p-value? How does filtering/de-duplication change the null hypothesis? How explicitly is the p-value computed?\n\nBased on the discussion around line 258 \"Influence of the de-deduplication on the correctness of radioactivity tests\" and Table 4, it appears that the detection protocol is entirely heuristic; it is not clear to me that the computed values are p-values in any formal sense.\n\nThe methods are specific to fixed-window, hash-based watermarks. Most of the experiments focus specifically on the Kirchenbauer watermark, with the exception of Table 5, which considers the Aaronson watermark (another fixed-window hashing watermark). Contrary to the general claims of the title and exposition, it is not clear how broadly these results hold, e.g., for distribution-preserving watermarks (variable-length hashes) [1] or watermarks that aren't based on hashes. Based on the decay in detectability vs. k shown in Table 5, I strongly suspect that at least the watermark [1] is not radioactive.\n\nThe abstract of the paper claims to give statistical guarantees for detection of models trained on watermarked text. No such guarantees are given, only experimental evidence.\n\n\n[1] Undetectable Watermarks for Language Models\nMiranda Christ, Sam Gunn, Or Zamir", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We have addressed each point individually. While we understand the concern, we argue that the p-values are **not** “heuristic”. We add a detailed response to clarify the reliability of the p-values and refer to App. D.2.1 “More details on token scoring and de-duplication” and App. D.2 “Correctness”. We will further emphasize this in the manuscript. \n\n> W1. The algorithms in Sections 4 are not well described [...]How explicitly is the p-value computed?\n\nH0 asserts that the observation of tokens from B were not generated following watermark W with secret key K (s.t. \"B was not trained on watermarked data from A\" is included in H0 as in Def 1). Filtering/de-duplication does not change the null hypothesis. It only modifies the observation from which we compute a score such that we know the distribution of this score under H0. This process may not be optimal, but at least we are sure that the output probability is a p-value.\n\nThe p-value is computed as the probability P(s(X,K) > t | H0), where:\n- X represents the observation\n- s is the score function (e.g. number of 'green' tokens)\n- K is the random secret key.\n\nWith the deduplication, X is a set of N unique (k+1)-tuples of tokens. With our filtering/tape, the tuples are not repeated from the prompt. This ensures that under H0, each (k+1)-tuple has a probability γ of leading to an increment in the score (on expectation over the random secret key K and if we assume an ideal hashing function). Therefore, s(x,K) is distributed as a binomial B(N, γ), and we compute P(s(x,K)>t|H0) using the regularized incomplete beta function $I_{γ}(t+1, N-t)$ (see App. C).\n\nWithout deduplication, the (k+1)-tuples of tokens are not unique: it is not easy to derive the distribution of s(x,K) or P(s(x,K)>t | H0). Similarly, without our filtering/tape, we do not know the distribution of the score under H0, as the model might simply have repeated watermarked tokens that were present in the attention span (cf. appendix D.2.1).\n\nWorks in the literature resort to bounds (e.g., Markov), or to estimation from random simulations (Monte Carlo, like Kuditipudi et al. do). We prefer to use a sub-optimal test (due to filtering/deduplication) whose p-values are sound, i.e., not based on heuristics.\n\n> W2. Based on the discussion around line 258 [...] it is not clear to me that the computed values are p-values in any formal sense.\n\nWhile the filtering and deduplication rules are indeed heuristic, the p-value computations are adapted from established tests in Kirchenbauer et al. 2023, Aaronson et al. 2023, and Fernandez et al. 2023 and are theoretically grounded as detailed in our previous answer to W1.\n\n**We also validate them experimentally**: App. D.2 \"Correctness\" elaborates on our tests and their validity. Figures 8 and 9 demonstrate that under H0 (models not trained on watermarked data), the p-values are uniformly distributed between 0 and 1 across all settings, confirming their practical value. We show in Tab. 4 that it is not the case without filtering/deduplication. We are committed to clarifying this further and welcome suggestions on how to convincingly demonstrate that the p-values are rigorously derived.\n\n> W3. The methods are specific to fixed-window, hash-based watermarks [...] \n\nIn App. A (limitations) and App. C.3, we discuss the generalization of radioactivity detection to other watermarks. The claim of our paper **is not** “All LLM watermarking schemes are radioactive”  but “Some very commonly used LLM watermarking schemes are”. In particular (see general comment), we focus on the most used methods, for which reliable p-values can be theoretically computed, which is not the case for [Christ et al.] or [Kuditipudi et al.] that relies on Monte Carlo simulations.\n\n> W4. The abstract of the paper [...] No such guarantees are given, only experimental evidence.\n\nWe disagree: the guarantees are given under H0: the false positive rate is provably small and under control (see Tab. 4, App. D.2 and answer to W2). The experimental evidence is here to support the theoretical guarantees. There is indeed no guarantee under H1, as usual in the watermarking literature. \n\n> Q1. Regarding the \"supervised\" and \"unsupervised\" settings [...]\n\nThis is a very good suggestion, we will change the manuscript accordingly.\n\n> Q2. Table 5 [...]\n\nTab. 5 is under the unsupervised setting.\n\nOrig: We detect watermarks in the *training data*. The goal aligns with the original application of the watermark, i.e., detection of AI generated text. We focus on a \"real-life\" scenario with 100-token texts.\n\nRad: In this setup, we aim to detect if a *model* is radioactive. We can score a larger number of tokens; in this case, we score 30k tokens.\n\nThe tasks are different, which explains why the p-values differ. What we study is, given a fixed watermark strength in the training data (= p-value on a fixed number of tokens), do different watermarks lead to different radioactivity signals.\n\n> Q3. “[...] independence assumption.” [...]\n\nPlease refer to the previous answer to W1: The tests rely on the assumption that the score increments for every token are i.i.d.. This holds true if the generated  (k+1)-tuples are i.i.d. under the null hypothesis. This is not the case a priori since (k+1)-tuples of tokens are more frequent than others in natural text distribution, and certain prompts can lead to the more frequent appearance of specific tuples.  \n\n> Q4. “radioactivity can only be observed for watermarked (k + 1)-grams [...]\n\nThe bias that radioactivity detection is capturing is at the token level. We are not detecting an overall \"distribution shift\". As there is no correlation between the greenlists of different watermark windows (since the partition is only a function of the watermark window through a hash function), radioactivity can only be observed for watermarked  (k+1)-tuples {watermark window + current token} that were part of A’s outputs in B’s training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful comments and suggestions. We address two main weaknesses that emerged from the reviews:

**Radioactivity is only demonstrated for some LLM watermarking schemes.** 

1) We focus on LLM watermarking schemes designed for AI-generated text detection, which is a zero-bit watermarking problem, not multi-bit. We focus on two foundational methods (Kirchenbauer et al., Aaronson et al.) because:
- (a) most of watermarking approaches are based on these (Fu et al. [2024a,b], Hu et al. [2023], Kirchenbauer et al. [2023b], Kuditipudi et al. [2023], Wu et al. [2023], Zhao et al. [2024], Yoo et al. [2024], …). The paper's objective is to demonstrate that some very commonly used LLM watermarking schemes are radioactive, not all of them.
- (b) they are the only ones, to our knowledge, to provide theoretical guarantees on p-values and therefore to allow for very low FPRs. 
We answer to all reviewers individually, but also refer to Appendix A  “Limitations” and Appendix C.3 “Does the radioactivity generalize to other watermarking schemes?”, where we justify our choices in more detail. 

2) We provide additional results on a multi-bit scenario with the method of Yoo et al., as suggested by reviewer q8YZ. Please refer to the pdf for the figures and to the rebuttal to q8YZ for more details.

Reference:
[Yoo, KiYoon, Wonhyuk Ahn, and Nojun Kwak. "Advancing beyond identification: Multi-bit watermark for language models." arXiv preprint arXiv:2308.00221 (2023).]



**The detection of radioactivity seems heuristic.**
 
The detection consists of 2 components: 
- (a) filtering / deduplication and score computation
- (b) p-value computation

While a) involves heuristic steps, b) is proven to be theoretically sound and experimentally validated. A core strength of our study is that our p-values are not heuristic. The paragraph “Influence of the de-duplication on the correctness of radioactivity tests'' in section 5.4, and Appendix D.2 ``Correctness”, confirm the proper distribution of our p-values under the null hypothesis. This is explained in more details to reviewer JVYy.

We hope this rebuttal clarifies any ambiguities, and we will ensure to incorporate the essential points in the main text.

Meta-review: This submission analyzes several watermarking schemes for LLM outputs and investigates how these schemes would impact the traceability when later LLMs are trained on watermarked text. 

Overall, reviewers were interested in the submission and we agree that this study contains a number of interesting observations. Initially, concerns were raised about the validity of the hypothesis tests done in this work. Yet, after an extended discussion between the reviewers all reviewers were ultimately convinced that the hypothesis test proposed in this work is indeed sound.

I do think the authors could have avoided this problem by defining their null hypothesis directly as a statement about the final distribution instead of defining it in terms of the general, biased statistic first, and then defining "filters". I would ask the authors to synthesize the results of the discussion here and in Appendix D.2 and rewrite the main body to clarify this point.

Further, there are a number of other outstanding improvements to the paper that the reviewers outlined, such as improvements to the abstract (ZoLZ), and changes to improve writing quality (Y2ta,JVYy), that I urge the authors to include, to make sure that the paper is not just accepted, but actually well-received and read by the community at large.
Decision: Accept (spotlight)