On the Learnability of Watermarks for Language Models
Authors: Chenchen Gu
Xiang Lisa Li
Percy Liang
Tatsunori Hashimoto
Venue: ICLR 2024 poster (2024)
Keywords: watermarking
large language models
distillation
Primary area: general machine learning (i.e., none of the above)
TL;DR: Language models can learn to naturally generate watermarked text, without using any special decoding algorithms.

Abstract: Watermarking of language model outputs enables statistical detection of model-generated text, which can mitigate harms and misuses of language models. Existing watermarking strategies operate by altering the decoder of an existing language model. In this paper, we ask whether language models can directly learn to generate watermarked text, which would have significant implications for the real-world deployment of watermarks. First, learned watermarks could be used to build open models that naturally generate watermarked text, enabling watermarking for open models, where users can control the decoding procedure. Second, if watermarking is used to determine the provenance of generated text, an adversary can hurt the reputation of a victim model by spoofing its watermark and generating damaging watermarked text. To investigate the learnability of watermarks, we propose watermark distillation, which trains a student model to behave like a teacher model that uses decoding-based watermarking. We test our approach on three decoding-based watermarking strategies and various hyperparameter settings, finding that models can learn to generate watermarked text with high detectability. We also find limitations to learnability, including the loss of watermarking capabilities under fine-tuning on normal text and high sample complexity when learning low-distortion watermarks.

Published as a conference paper at ICLR 2024
ON THE LEARNABILITY OF WATERMARKS FOR LANGUAGE MODELS
Chenchen Gu, Xiang Lisa Li, Perey Liang, Tatsunori Hashimoto
Stanford University (cygu, xlisali, chashimm)estanford.eek pliangêcs stan ptiang@cssstanforddo edu
ABSTRACT
Watermarking of language model outputs enables statistical detection of model- generated text, which can mitigate harms and misuses of language models. Ex- isting watermarking strategies operate by altering the decoder of an existing lan- guage model. this paper, we ask whether language models can directly learn to generate watermarked text, which would have significant implications for the real-world deployment of watermarks. First, learned watermarks could be used build open models that naturally generate watermarked text, enabling watermark- ing for open models, where users can control the decoding procedure. Second, if watermarking is used to determine the provenance of generated text, an ad- versary can hurt the reputation of a victim model by spoofing its watermark and generating damaging watermarked text. To investigate the learnability of water- marks, we propose watermark distillation, which trains a student model to behave like teacher model that uses decoding-based watermarking. We test ap- proach on three decoding- baaeed watermarking strategies and various hyperparam- eter settings, finding that models can learn to generate watermarked text with high detectability. We also find limitations to learnability, including the loss of water- marking capabilities under fine-tuning on normal text and high sample complexity when learning low-distortion watermarks
INTRODUCTION
As language models (LMs) become more capable and widely used, watermarking LM outputs be- comes increasingly important to mitigate potential harms and misuses of LMs. Watermarking en- ables statistical detection of LM-generated text, which enables enforcing policies on LM usage, e.g., removing LM-generated disinformation from social media platforms or detecting academic dishon- esty. Another proposed use case of watermarking is identifying the provenance of text, i.e., tracing text to the specific LM that generated it (Abdelnabi & Fritz 2021 Kuditipudi et al.) 2023).
Recent works have shown that it is possible for an LM provider to inject specific, known watermark signals into text using specialized decoding algorithms (Kirchenbauer et al. [203aa] Aaronson [023] Kuditipudi al., 2023). but little is known about whether these watermarks model. The learnability of watermarks has significant implications for the real-world deployment of watermarks, could enable downstream applications and adversarial spoofing attacks.
learnable by
In this work, we study the learnability of watermarks by studying weights-based watermarking, which involves learning parameters for a language model that cause generate watermarked text under its natural sampling distribution, without using a special decoding watermarking algo- rithm. Our investigation is motivated by its relevant implications for two applications: (i) developing watermarking for open language models and (ii) spoofing watermarks.
First, existing watermarking methods depend upon using specialized decoding algorithm, making them too inflexible for open LMs. For open LMs, where the weights are released, user can use an ordinary decoding algorithm and generate non-watermarked text, whether intentionally or not. We find that baights-based watermarking works with standard decoding strategies, removing the
See http://///gg.aaaoooo///////////a
for code and models.

---

Published as a conference paper at ICLR 2024
Decoding-based watermarking
Original model Pum
Decoding- baangg watarmarking 7.
Generated
Detector
text
Poestue Model generated!
Watermark
Weights-based watermarking
Watermark distilled model Pg
Standard
Generated
Detector
0.0004 Watemarkedl Model sedelt-gnnraald!
decoding
text
Figure 1: Decoding-based watermarking (top) versus weights-based watermarking (bottom). Decoding-based watermarking requires specialized decoding algorithm Sw to generate water- marked text, whereas weights-baset watermarking marked text directly from the model, using just its weights. Watermark distillation enables weights- based watermarking by training student model po to behave like the teacher model PLM with decoding-basd -based watermarking strategy fw.
standard decoding to generate water-
reliance on specialized decoder. This makes it promising first step towards developing water- marking for open LMs. However, also find that weights- based watermarking capabilittes can be removed by fine-tuning on normal text, indicating that improving robustness to fine-tuning is an important remaining challenge.
Second, in watermark spoofing attacks, an adversary outputs text that contains the watermark signal from victim LM (Sadasivan et al.] 2023). watermarking used to identify the provenance of text, then attacker could attribute damaging text to the victim LM and hurt its reputation. We find that the learning weights-basee watermarking can enable spoofing attacks, and we demonstrate a proof-of-come -ofe--ce--ttttt attack
chat model. The possibility of spoofing attacks suggests that watermarking should not be used to attribute provenance or blame to a specific LM. Instead, watermarking should only be used to statistically detect L.M-generated text, which can be used for tasks such as finding infractions of policies on LM usage.
cestruction-follwwwee
To enable weiehts-based watermarking, we propose logit-based and sampling- based watermark dis- tillation, two simple methods for a student model to learn weights-based watermarking from a teacher model with decoding-baan -based watermarking. Intuitively, in logit-based watermark distilla- tion, the student model is trained to match the next token distribution outputted by the teacher model using decoding-
watermarking. In sampling- based watermark distillation, the teacher model with decoding- basedd watermarking is first used to generate watermarked samples. Then, the student model fine-tuned on these watermarked samples.
We experiment with three decoding-based watermarking strategies: KGW (Kirchenbauer et al.) 2023a] Zhao et al., (023a), Aar (Aaronson, 2023), and KTH (Kuditipudi al.) [203)), and vari- ous values for their hyperparameters that control the level of distortion induced by watermarking. We find that watermarks and hyperparameter settings vary in their degree learnability. In each watermarking strategy, higher-distortion hyperparameter settings are successfully learned by both forms of watermark distillation (median p-values less than 0.0001). Lower-distortion watermarks challenging and less sample efficient to learn, but not unlearn- able, as the p-values are still noticeably smaller than the non-watermarked baseline of 0.5.
and hyperparameter settings
BACKGROUND AND NOTATION: DECODING-BASER WATERMARKING
We study autoregressive language models PLM V* + AVV) that map from prefix string rr V to a next token distribution over the vocabulary V. Informally, a decoding- based watermarking strategy fw uses watermark key & to modify the model's original next token distribution P x) into a new distribution for generating watermarked text, which has watermark signal embedded. The watermark detection algorithm fa looks for this signal using the same watermark key &.
Formally, we define decoding-based watermarking strategy to be function
:(V) x V x A(V)
2

---

Published as a conference paper at ICLR 2024
where E is the set of possible watermark keys. This function fw outputs distribution Pw 2) from which to generate the next token in the watermarked text, given an original next token distribution PLM( (- z), imput text z, and watermark key { E E.
We define a watermark detection algorithm to be a function
Sa V" = + [0,
Given some text ar and watermark key E. la outputs p-value with respect to the null hypothesis that I is independent of fw with key ç. Informally, fa computes a test statistic that measures the strength of the watermark signal, then computes p-value using the distribution of the test statistic under the null hypothesis. If the p-value is below a given significance level, the null hypothesis is rejected and the text is detected watermarked. Slightly imprecisely, rejecting the null hypothesis means the text is detected as model-generated
In this paper, consider three decoding-based watermarking strategies: Algorithm 2 in Kirchen- bauer et al. (2023a). the Gumbel softmax scheme in Aaronson (2023). and the exponential minimum sampling scheme in Kuditipudi al. (2023). Using the authors' names and initials, we refer to these as KGW, Aar, and KTH, respectively. We briefly describe these watermarking strategies below. See Appendix D for additional details and formal definitions.
green list bias. In the KGW watermarking strategy (Kirchenbauer et al., 2023a), when generating the next token, the vocabulary is pseudorandomly split into "green list" and "red list" by hashing the previous token using the watermark key ç. The green list contains watermark hyper- parameter 7 E (0, 1) proportion of the vocabulary. Then, before the model's logits are converted to probabilities via the softmax function, hyperparameter
KGW:
0 is added the logits of the green list tokens. This procedure makes green list tokens more likely in watermarked text than in non- watermarked text. So, detection time, if the proportion of green list tokens in a text is much greater than 7, then the p-value is small.
More generally, the previous & tokens can be hashed, where & is hyperparameter. Values of l: are investigated by |Kirchenbauer et al. (2023b). finding that lower k leads to more repetitive outputs. When k 0, the green and red lists are fixed, regardless of the previous tokens. k was proposed by Zhao et al. (2023a) as Unigram-Watermark, Watermark, a variant of KGW, but we will denote it as KGW 1: to simplify notation.
KGW distorts model outputs by upweighting green list tokens, increasing perplexity of generated texts computed by larger model (Kirchenbauer et al.) [2023a). Increasing the bias hyperparameter d increases detectability, i.e., smaller p-values, but also increases distortion.
Aar: boosting continuous hash scores. The Aar watermarking strategy (Aaronson) 2023 hashes the previous & tokens using key & (where ls is hyperparameter) to obtain score T; for each token i E V. where each r; is uniformly distributed in [0. Let p; be the original model probability for token Then, the next generated token is deterministically chosen to be the token i which maximizes r: r: i.e., a token with both a high original probability p; and high hash score r;. This procedure boosts the hash scores of tokens in watermarked text compared to non-watermarked text. So, at detection time, if the hash scores r; of the tokens in the observed sequence are high, then the p-value is low.
Since Aar deterministically selects the next token based on the previous tokens and the original model probabilities, Aar can lead to repetitive text, especially for small is (Kuditipudi et al. 2023). Increasing & decreases repetitiveness, as larger k-grams are less likely to be repeated, but the water- mark also becomes less robust edits, each token edit affects the hash scores for k + tokens.
KTH: robust sequence alignment. The KTH watermarking strategy (Kuditipudi et al. 2023) is similar to Aar, but instead of hashing previous tokens to obtain the scores ri, the scores are obtained from the next element the key sequence ç. In KTH, E ( {(m) where each {() [0, 1]1i|
This is slightly imprecise because model-generater text is not the only text that can be deliberately water- marked. For example, a human could write watermarked text by manually following the watermarking algo- rithm. However, for most practical use cases, such as detecting academic dishonesty, this minor imprecision is issue because either way, the user is doing something suspicious and unusual.
3

---

Published as a conference paper at ICLR 2024
contains the scores, with entries uniformly distributed across [0, 1]. Then, to generate the j-th token in the sequence, KTH deterministically chooses the token i that maximizes (c! 1/ps
Note that m should be larger than the maximum generation length. To allow different generations from the same prompt, before generating each sequence, & can be shifted by some random т, i.e., & g(1+r mod m) ,..., <( mod m), To study the impact of these shifts on learnability, we introduce a hyperparameter s E [1, m] for how many shift values T are possible Increasing s expands the range of possible model generations.
At detection time, to be robust to text edits and shifts, the test statistic quantifies how well a text can be aligned with the key sequence Ç. More specifically, the test statistic computes a minimum Levenshtein distance using the alignment cost d(x, <) > log(1 e A lower (more negative) test statistic indicates stronger watermark signal. To compute p-values, the observed test statistic is compared a reference distribution of test statistics of wottraarmdr texts. Letting T be the number of samples in the reference distribution, the p-values computing using this method are lower bounded by T+1*
METHODS
Problem statement. Given teacher model PLM, decoding- watermarking strategy fw. and key ç, the goal is to learn a student model po whose sampling distribution naturally generates water- marked text. Specifically, letting fa be the detection algorithm corresponding to fw. pe generates text y with small detection p-value fa(y. 5) with probability similar to that of pLM with fw. then Pe has learned a weights-basee watermarking strategy, since pe has learned to generate watermarked text using just its weights. Figure Iillustrates decoding. decoding-based based versus weights-based based watermarking.
Next, we present two methods for learning a weights- based watermarking strategy: logit-based watermark distillation and sampling- basse watermark distillation, which fall under the broader cat- egory of knowledge distillation (Hinton et al. 2015 Kim & Rush 2016).
3.1 LOGIT-BASED WATERMARK DISTILLATION
In logit-based watermark distillation, we train the student model pe to behave if it had decoding- based watermarking strategy fw applied. Specifically, given an input x, we want the student model's next token distribution po(- x>) to match fw (PLM (- (=1z), x. {), the next token distribution outputted by the teacher model PLM with decoding-bas -based watermarking strategy fw and key &. So, given dataset texts D, the training objective is to minimize the mean KL divergence between the teacher and student next token distributions on all prefixes in D, given by
Ixn(x)
Clogit (0)
Σ
DKL (fw (PLM ( (xx)) Reti E) po (: x<t)).
(3)
D
FED
The teacher model PLM is frozen. This approach requires that PLM and py have the same tokenizer and vocabulary so that the logits can be aligned between the models. It is also helpful if prM and po share the same model architecture, as then we can initialize po to PLM- Note that the ground truth next tokens 2, from dataset D are not used in the loss function, D does need to be watermarked text. Standard datasets containing non-w non-watermarked human human-generatee generated text can be used]
3.2 SAMPLING-BASED WATERMARK DISTILLATION
Sampling based watermark distillation has two stages. First, we generate watermarked text from teacher model PLM with decoding- based watermarking strategy fw applied using key E. Then, fine-tune the student model py on this watermarked text using the standard language modeling cross- entropy loss.
P We space the shifts evenly across m], i.e., the of possible shifts 7 {i- [m/s] 005iss 'If 1> is non- wotermarrrree text, then it theoretically might be out of distribution for po to autoregressively generate watermarked text, since Po would he conditioning on the watermarked text it has already generated. However, empirically, find that loget-based distilled models can learn generate watermarked text.
4

---

Published as a conference paper at ICLR 2024
Formally, given set of prompts P, for each prompt 2 P, generate watermarked completion
sequence I = 21 .. En, where each sampled token #y ~ fw (DLM (. zx<(), 22 E)- Let the fine-tuning dataset D consist of these watermarked generations Then, we train Po to minimize the cross-entropy loss on D, given by
len(a)
samppling (0)
log po (xt xet))
D
sED
Here, PLM and do not need to share the same tokenizer or vocabulary. However, sampling-based watermark distillation is less efficient than logit-based watermark distillation due autoregressively generating watermarked text in the first stage.
4 EXPERIMENTAL SETUP
We rum experiments to evaluate how well logit- and sampling- -bsed watermark distillation can learn weights- watermarking from the decoding- watermarking strategies seen in $ Ideally, we want weights-based watermarking match decoding based watermarking in terms of watermark detectability and generation quality.
4.1 WATERMARKING STRATEGIES AND HYPERPARAMETERS
We experiment with the three decoding-basee watermarking strategies discussed in $ We use var- ious hyperparameter settings vary the level of distortion induced by the watermarks. Specifically, we test KGW with k = {0, 1, 2}. 0.25 and å = {1, 2] 67 Aar with k = {2,3, 43, and KTH with
key length 256 and number of shifts {1, 2, 4, 2563.
4.2 TRAINING
For each decoding- based watermarking strategy, we test logit- regit-based and sampling -aaseing--baedd watermark distillation for learning weights brights-based watermarking.
For logit-based watermark distillation, we use Llama 2 7B (Touvron al. 2023 as both the teacher and student models (the student model is initialized with the teacher model weights). We distill using a subset of Open WebText (Gokaslan al., 2019) for 5,000 steps with a batch size of 64 sequences, sequence length of 512 tokens, 8 maximal learning rate of 1e-5, and cosine learning rate decay with a linear warmup. Full training details are in Appendix E.]
For sampling-based watermark distillation, we also use Llama 2 7B as both the teacher and student with decoding- models. First, we use Llama watermarked samples of length 256 tokens, prompted with 50-token prefixes from Open Webneeebreo Text. Then, we fine-tune Llama 2 7B on the watermarked samples for epoch of 5,000 steps, with a batch size of 128 sequences, sequence length of 256 tokens, maximal learning of 1e-5, and cosine learning rate decay with a linear warmup. Full training details are in Appendix E.2
watermarking strategy to generate 640,000
In Appendix F perform sampling baase based watermark distillation experiments where the teacher and student models have different tokenizers and sizes, using Llama 2 7B as the teacher model and Pythia 1.4B the student model (Biderman al. 2023)
4.3 EVALUATION AND METRICS
Evaluation procedure. As in Kirchenbauer al. (2023a) and Kuditipudi et al. (2023), evaluate on generations prompted by prefixes from the Newsl subset of the C4 dataset (Raffel et al.,
'Ideally, the intended use case and domain of the student model pe should inform the choices of the of prompts P and teacher model PLM- However, empirically, we find that sampling passe based watermark distillation
is fairly robust to domain shifts (see and Appendix []]
"Because we always use 7 0.25, we sometimes omit explicitly stating the value of to simplify notation.
We exclude & 2, 5 = since we find that l: 2, 5 = already exhibits lower lcarnability.
Foor KTH hatch size of 128 and sequence length of 256 tokens because key length 256.
5

---

Published as a conference paper at ICLR 2024
2020). For each decoding- basodd watermarking strategy and distilled model, we generate 5,000 200- token completions from 50-token prompts from the validation split. We use standard sampling with temperature I for the main results, and investigate the model's s robustness to different decoding We include evaluations on additional datasets in Appendix [
parameters in Appendix C
We choose metries to evaluate two properties: watermark detectability and generation quality.
Watermark detectability. We compute the median watermark detection p-value across generations. Note that the p-values for the KTH watermark are lower bounded by how many samples T compute in the reference distribution. Similar to Kuditipudi et al. (2023), we use T = 10,000, so the p-values are lower bounded by le-4. To make finer-grained distinctions in watermark strength below this lower bound, we also compute the median test statistic (discussed in $2 to evaluate KTH watermark strength. A lower (more negative) test statistic indicates higher watermark detectability.
We also compute the AUROC (area under the receiver operating characteristic curve) for classifying moodel-generated versus numan-generatte text using the watermark detection p-values/test statistics. We compute the AUROC using an equal number of model-gennrater and numan-generatee texts, all of the same length.
Generation quality. We use Llama 2 13B to compute the mean perplexity of generations. Lower perplexity tends to indicate higher quality and fluency, but repetitive text also achieves low per- plexity. So, to evaluate repetitiveness, we compute the mean seq-rep-3 of generations, which is the proportion of duplicate 3-grams in sequence, given by 1 of unique 3-grams
(Welleck et al. 2020).
of grams
Comparisons. For both watermark distillation methods, for each decoding- watermarking strategy fw. we compare the teacher model with fw applied (denoted by "Decoding") against the distilled student model (denoted by "Logit" and "SSampling" for logit-based and sampling-baat based watermark distillation, respectively). As a baseline for generation quality, we use the base student model with watermarking or distillation (denoted by "Base student").
5 RESULTS
Table D contains results for the logit-based and sampling- based watermark distillation experiments. The two watermark distillation methods exhibit similar trends. Both forms of watermark distillation successfully learn higher-distortion watermarks | achieving small p-values and high detectability. In some watermarks, e.g., KGW & = O, watermark distillation matches the p-values achieved by decoding-basee watermarking. In other watermarks, watermark distillation does not achieve as small watermark detection p-values decoding- becoding-based watermarking, but for higher-distortinn watermark hyperparameter settings (smaller and larger 8 for KGW, smaller k: for Aar, and smaller s for KTH), the p-values are still sufficiently small to enable high detectability, as shown by the high AUROC values. Figure 4 in Appendix cooottiiss empirical CDFs of the distributions of p-values across generations, showing that for higher-distortion watermarks, the majority of generations from the watermark distilled models have small p-values.
Within each watermark type, p-values from logit-based and sampling- based distillation are larger for lower-distortion hyperparameter settings, indicating that lower-distortion watermarks are harder learn. However, these watermarks are still learned some degree, the p-values are noticeably smaller than the non-watermarked baseline of 0.5, and the AUROC values are noticeably higher than the won-watermrrked baseline of 0.5. In Appendix G sample complexity experiments show that more training samples and steps lead to smaller p-values for both logit-based and sampling- based distillation, with no sign of convergence. In addition, we find that when we train logit-based watermark distillation on KGW k = 2, 2 for five times longer (25,000 steps) more data, the median p-value decreases from 0.1 to 0.012. This suggests that lower-distortion watermarks are less sample efficient to learn, but still learnable, given sufficient training data and steps.
Compared to decoding- based watermarking, watermark distillation does not achieve as optimal a tradeoff between generation quality and detectability. For KGW and KTH, both watermark distil- lation methods achieve slightly to moderately higher perplexity and similar or larger p-values com-
Here, are using "distortion" somewhat informally, roughly meaning how much of a difference water- marking causes in terms of generation quality, behavior, etc.

---

Published as a conference paper at ICLR 2024
p-value (L)
AUROC (1)
Perplexity (1)
seq-rep-3 (L)
(KTH test statistic (1))
Decoding
Decoding
Sampling
Decoding
Samphiy
Decoding
Saann Camppln ione
Logit
Logit
Logil
Logit
Watermark
== 0, 2
6c-16
2c-17
2e-15
1.00
1.00
1.00
17.5
17.3
20.3
0.05
0.05
0.05
k= 1,6 2
4e-18
7e-09
8e-07
1.00
1.00
1.00
16.5
17.6
19.2
0.04
0.03
0.04
KGW
== 2 2
9c-18
lc-01
1c-01
1.00
0.80
0.74
16.8
17.7
19.8
0.03
0.02
0.03
0,6
5e-04
3e-05
1e-03
0.98
0.99
0.98
13.0
12.9
15.7
0.03
0.03
0.03
= 1,6
le-05
7e-03
2e-02
0.99
0.91
0.87
12.7
13.1
14.9
0.03
0.03
0.03
k
le-75
2e-12
3e-17
1.00
1.00
0.98
6.5
10.8
7.7
0.34
0.11
0.34
Aar
h = 3
5e-73
le-01
6c-03
1.00
0.78
0.88
9.5
11.6
10.5
0.14
0.04
0.17
k =
4c-72
4c-01
3c-01
1.00
0.58
0.65
10.7
11.8
11.9
0.09
0.03
0.11
le-04
le-04
le-04
8
(-593)
(-565)
(-561)
1.00
1.00
1.00
10.5
16.5
15.1
0.03
0.04
0.03
le-04
le-04
le-04
1.00
0.99
16.3
13.4
0.03
0.04
(-596)
(-476)
(-525)
KTH
le-04
1e-03
le-04
1.00
0.96
0.99
10.6
14.2
12.5
0.03
0.04
0.04
(-594)
(-438)
(-487)
Ic-04
8c-02
1e-04
256
1.00
0.85
0.97
10.8
11.3
11.3
0.03
0.04
0.04
(-594)
(-423)
(-453)
Base student
5c-01
0.50
11.8
0.03
Table I: Results for logit-based and sampling-based watermark distillation experiments. Within each watermark type (KGW, Aar, and KTH), the hyperparameter rows go from higher-distortion to lower-distortion moving down the table. Higher-distortion watermarks are successfully learned with small p-vauues and high detectability. Lower-distortion watermarks are harder learn, as shown by the larger p-values, but they are still learnable, just less efficiently and strongly.
pared to decoding-baae watermarking, For Aar, watermark distillation achieves similar or lower seq-rep-3 as decoding-based watermarking, but larger p-values. This suggests that to learn weights- based watermarking, logit-based and sampling-biis based watermark distillation incur some cost to the tradeoff between generation quality and detectability.
While logit-based and sampling based watermark distillation show similar trends, there are some interesting differences. We defer this discussion to Appendix due to space constraints. However, recall that logit-based and campling-based distillation have different requirements (e.g., access to logits and shared tokenizer vs. access to samples and autoregressive generation, see $3.1 $3.2), so they should not be compared solely on performance. So, logit-based and sampling- distilla- tion are each suitable and applicable for different settings, so neither is strictly better than the other in all scenarios.
Robustness to text edits. We test the robustness of teights-based watermarking to edits by ran- domly corrupting generated text from the logit-based and sumpling-basee watermark distilled Llama 2 7B models with varying proportions of tokens randomly edited. See Appendix for full experi- mental details. As shown in Figure 2 the detection p-values of all three watermark types robust to moderate edit proportions, up to around 20-30%. At higher edit proportions, up to around 60- 70%, KTH is significantly more robust to edits than KGW and Aar, consistent with the findings of
Kuditipudi et al. (2023).
Robustness to changes in decoding parameters. Whereas decoding- based watermarking relies on specialized decoding algorithms, weights- watermarking generates watermarked text naturally under standard decoding algorithms. In Appendix C find that weights-based watermarking learned by logit-based and sampling-based based distillation is robust to changes in decoding parameters, e.g., different temperatures and different thresholds p nucleus sampling (Holtzman al. 2020)
WATERMARKING FOR OPEN MODELS
In $5 we showed that weights- based watermarking works under standard decoding algorithms and robust to changes in decoding parameters. This is necessary first step towards watermarking
7

---

Published as a conference paper at ICLR 2024
k=
0.4
Aar
5=H
0.3
e-valve
p-value
Logit-based Sampling- baased
0.2
0.2
KGW k= 1,6=2
0.1
0.1
Aar k
KTH
0.0
0.0
0.0
0.4
0.6
0.8
0
500
1000
1500
2000
2500
Proportion of tokens edited
Number of continued finetuning steps
Figure 2: Watermark detection p-values of gen- erations from weights-based watermarking, cor- rupted with varying proportions of tokens ran- domly edited. The watermarks are robust to moderate amounts of corruption.
Figure 3: Watermark detection p-values of gen- crations from logit-based watermark distilled Llama 2 7B models after further fine-tuning OpenWeb Web Text. The models' weights- based wa- termarking is removed by fine-tuning.
for open models, where users can run inference themselves. They may change the decoding algo- rithm, and the inference library they use may not enable decoding- baaodnng-bbeedd watermarking by default or implement it at all.
Robust watermarking for open models should also ideally be robust to fine-tuning, as users have the ability and desire to fine-tune open models. Ideally, this fine-tuning should not remove watermarking capabilities, either intentionally or unintentionally. However, Figure 3 shows that weights weights-based watermarking obtained from watermark distillation not robust to further fine-tuning on normal, non-watermarked text (see Appendix for experimental details). We leave addressing this challenge and learning weights-basee watermarking that is robust to fine-tuning to future work.
However, weights-based watermarking also has potential use cases that do not require robustness to further fine-tuning. For example, weights-basee baaehts-based watermarking could be used for watermarking open models which are unlikely to be fine- tuned further by users, such RLHF--trined instruction- following chat models. In addition, weights-basse basee watermarking simplifies decoding compared to decoding-base baased watermarking, there So, weights-based watermarking can easily be deployed into existing highly optimized infrastruc- tures and inference algorithms, just requires loading different model weights.
need for additional specialized decoding algorithm.
SPOOFING ATTACKS
One proposed use case watermark detection is to attribute the provenance of generated text to a specific model, which could help policy enforcement and auditing of model providers (Abdelnabi & Fritz, 2021 Kuditipudi et al. (023). However, using watermarking for provenance attribution brings the risk of spoofing attacks: adversary generates damaging text containing the watermark of a victim model, making it appear as if the victim model generated it, thus hurting the reputation of the victim model (Sadasivan al. 2023). Sampling-bais watermark distillation is applicable to the spoofing setting, as it only requires generated samples from the victim/teachen model.
In this proof-of-concept experiment, simulate a spoofing attack using victim model of Llama 2-Chat 7B with KGW decoding- based watermarking (k = 1.7 0.25, 2). Llama 2-Chat 7B is trained for safety and tends refuse harmful requests (Touvron et al., 2023). The goal of the spoof- ing attack is to generate watermarked responses to harmful requests, damaging the victim model's reputation for safety. We obtain an adversary model by performing sampling-based watermark dis- tillation with Alpaca-7B (Taori et al.) 2023) as the student and the Llama 2-Chat 7B victim model as the teacher. We query the victim model for watermarked samples, filter out refusals, then fine-tune the adversary model on those samples. See Appendix L.1 for full experimental details.
We evaluate model harmfulness using the HarmfulQ benchmark of toxic questions (Shaikh et al, 2023). We use GPT-4 (OpenAl 2023) to annotate responses enabling harmful behavior not. See Appendix [L2 for full evaluation details. We find that the victim model has harmful response rate of 0%, whereas the distilled adversary model has harmful response rate of 71 % (base Alpaca-
8

---

Published as a conference paper at ICLR 2024
7B has harmful response rate of 80%). Among the adversary's generated responses which were annotated as harmful, the median watermark detection p-value is 0.002 (with a median generation length of 593 tokens) 1D showing that harmful text generated by the adversary may be wrongly attributed to the victim model.
8 RELATED WORK
Post-hoc detection. Many works have studied post-hoc detection of model-generated text, without modifying the generation process itself. Some works train binary classifier to perform detection
(Zellers et al. 2019; Bakktin et al. 2019 Tan et al. 2020), see Jawahar et al. (2020) for survey. Other methods are zero-shot, using heuristics and metrics for detection (Gehrmann al. 2019
Solaiman et al. 2019 Mitchell et al.) 2023). In contrast to post-hoc detection, we investigate water- marking, which modifies the text generation process to embed detectable signal. However, post- hoc detection could potentially be used in conjunction with watermarking (Mitchell et al. 2023).
Text watermarking. Older works on text watermarking edit pre-existing text inject signals that can be statistically detected (Rizzo et al. 2019 Abdelnabi & Fritz 2021: Yang et al. 2022). see Kamaruddin al.] (2018) for a survey. Recently, many works have studied decoding- basdg water- marking, which modifies decoding procedures to generate new watermarked text (Venugopal et al.,
2011 Kirchenbauer et al. [2023a] (1223||Aaroma AAronson] 2023] Kuditipudi et al. 2023 Zhao et al., al., | Christ
ct al. 2023: Hu et al.) [2023) Wu et al. [2023 Huang al. 2023 Zhao et al. 2024). Various classes of decoding-baase based watermarking methods have been proposed, c.g., semantic watermarks (Fu et al.
2023 Hou et al. 2023 Liu et al.
2023b Ren et al.
2023). multi-bit watermarking (Yoo al. | 2023
Wang et al. 2023 Qu et al. 2024
2024). and public/private key watermarking (Liu Boroujeny et al.
al.
2023a
Fairoze al.
2023).
See (Liu al.
2023c) for a survey of text watermarking. (Sander
et al.
2024) find that it is possible to detect if model's training data contained watermarked text.
Watermark attacks. Recent works have investigated attacks remove the watermark from wa- termarked text, using methods such as paraphrasing, swapping tokens, etc. (Kirchenbauer et al., 2023b Krishna al., 2023; (adassvan et al. 2023 Zhang al. [023 Pang al.) [024]] Jovanović et al. 2024). In addition, watermark spoofing attacks are where an adversary produces text that falsely detected as watermarked and generated by a victim model. Sadasivan et al. (2023) and Jovanović et al. (2024) spoof the KGW watermark by exploiting its green list bias, and Pang et al. (2024) demonstrate spoofing attacks by exploiting watermark robustness and public detection APIs. In our work, we show that sampling-ban -aasii watermark distillation can enable spoofing attacks.
API watermarking for protection against model extraction. Prior works have studied API water- marking for protection against model extraction attacks, where an adversary imitates or reconstructs a victim model by distilling on its API outputs (He et al.) 2022a] Zhao et al. 2022 He et al. 20226] Zhao et al. 2023b). In API watermarking, watermark signal injected into the victim's 's API out- puts, making it possible to detect if a suspect model was distilled from the victim API. In contrast, text watermarking enables detecting whether given text was model-generatee
CONCLUSION
In this paper, we investigate the lcarnability of watermarks for language models. Using based and sampling-bast based watermark distillation, we find that models can learn to naturally generate wa- termarked text using standard decoding algorithms, although lower-distortion watermarks are harder and less sample efficient to learn. Our findings address key technical challenge towards developing watermarking for open models and raise the possibility of watermark spoofing attacks.
Future work may explore improving the robustness of weights-based watermarking to further fine- tuning, which would address another important challenge towards robust watermarking for open models. Future work may also more comprehensively study and evaluate spoofing attacks and po- tential defenses against spoofing attacks, which would have implications for whether watermarks should be used to assign provenance and blame to specific model.
"mmon all 200-token slices from each of the harmful responses, the median detection p-value is 0.04.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
In this paper, we find that sampling-based watermark distillation can potentially be used to carry harmful watermark spoofing attacks. This may appear to be a potentially harmful insight that weakens watermarking by undermining its ability to identify the provenance of text. However, we believe that public knowledge of spoofing attacks and the limitations of watermarking is important. This way, the public knows not to trust watermarking for reliably attributing provenance or blame specific model. Then, if watermark detection not used to prove that a text was generated by specific model, spoofing attacks will cause significantly less harm, if any at all. Watermarking can still be used statistically detect L.M-generated text, which can be used for tasks such as finding infractions of policies on language model usage.
REPRODUCIBILITY STATEMENT
For the main results, we describe our experimental setup in $ including training details, datasets used, and evaluation procedure. For all other experiments and results, we describe full experimental details in the appendix. The exact sections in the appendix are mentioned in the main paper where relevant, In addition, we release code and scripts to reproduce experiments at at //github. leriaakkieerraabityyy. along with trained model weights.
com/
ACKNOWLEDGMENTS
We gratefully acknowledge the support of an Open Philanthropy Project Award. Chenchen Gu was supported by a Stanford CURIS Fellowship. Xiang Lisa Li supported by a Stanford Graduate Fellowship and Two Sigma PhD Fellowship. Tatsunori Hashimoto is supported by gift from Open Philanthropy and by the Tianqiao and Chrissy Chen Institute.
REFERENCES
Scott Aaronson. Watermarking of large language models. Large Language Models and Trans- formers Workshop at Simons Institute for the Theory of Computing, 2023. URL https:
/www.youtubb. com
jbSMZqA
Sahar Abdelnabi and Mario Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 121- 140. IEEE, 2021. doi: 10.1 10.1109/SP40001 http://44001.222...0000 00008..
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint 1303039906.,3311, 2019, tttpsttss//axx...orrr absrgg/bbb/// 1906. 003556666333111111
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth. Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Con- ference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp- 2397-2430. PMLR, 23-29 Jul 2023. URL pstps://proceeddinge mlr. www.s//we22222 biderman23a, html
Massich Kordi Boroujeny, Ya Jiang, Kai Zeng, and Brian Mark. Multi-bit distortion-free water- marking for large language models. arXiv preprint arXiv:2402.16578 2024. URL https:
//axxiv.org/absl /tt.bb///2222.1...
Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. arXiv www.ooo./////.o.ooooo...oo.. 2306. 09194
preprint arXiv:2306.091 0999999,,, 2023. wttlhhtps://aat
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-awee aware attention model for abstractive summarization of long docu- ments. In Proceedings of the 2018 Conference of the North American Chapter of the Association
10

---

Published as a conference paper at ICLR 2024
for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), pp- 615-621, New Orleans, Louisiana, June 2018, Association for Computational Linguistics. doi: 10.18653/v 100S5///11111-299990 URL
Jaiden Fairoze, Sanjam Garg, Somesh Jha, Saced Mahloujifar, Mohammad Mahmoody, and Mingyuan Wang. Publicly detectable watermarking for language models. arXiv preprint arXiv:2310.18491, 2023. httphtppp..//xx....ttoo/////////
Wikimedia Foundation. Wikimedia downloads, 2022. URL. https:///umpp: wi likeiiii
orc
Yu Fu, Deyi Xiong, and Yue Dong. Watermarking conditional text generation for ai detection: Unveiling challenges and a semantic-aware watermark remedy. arXiv preprint 20ti::2337118800 2023.
13808
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics: System Demonstrations, pp. 111-116, Florence, Italy, July
2019. Association for Computational Linguistics. doi: 10. 111111666/111-90000
URL https:
7/Acllanthollgy.oorr/P111
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus.
http:
//Skyl1ono07. github 10 wttpppenweeeettorrrreeoeeeeeeeee
2019.
Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. Protecting intellectual
property of language generation apis with lexical watermark. Proceedings of the AAAI Conference on Artificial Intelligence, 36:10758- 10766, Jun. 2022a. doi: 10. www.1/000/aa.iiiiiii..i v36110.21311. URL
https://ojs... aaai. org/ index..... httpe///p/.aaaaaaaaaaeeee//eeeee
Xuanli He, Qiongkai Xu, Yi Zeng. Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. Cater: Intellectual property protection on text generation apis via conditional watermarks. In Advances in Neural Information Processing Systems, volume 35, pp. 5431-5445. Curran Associates, Inc.,
2022b. URL htttps:// neurips cc/ http://pppee..lleeepaaeeeeee///e 41te://5fee44eccaaaaeaa000002000
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint M.XXv:1503.02555
URL /1t.1///1.011///////////////////
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL
Abe Bohan Hou, Jingyu Zhang, Tianxing He. Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, and Yulia Tsvetkov. Sem- stamp: A semantic watermark with paraphrastic robustness for text generation. arXiv preprint arXiv:2310.03991 2023. /RRLL//pp///rxi....rr//////2.... 03991
Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Un- biased watermark for large language models. arXiv preprint 1rX1v:2310.10669, 2023. URL
https://arxiv: wtw../////.....1.....1
Baihe Huang, Banghua Zhu, Hanlin Zhu, Jason Lee, Jiantao Jiao, and Michael I Jordan. Towards optimal statistical watermarking. arXiv preprint arXiv:2312.079300, 2023. URL https://
arxiv.org/eaoo
Ganesh Jawahar, Muhammad Abdul- Mageed, and Laks Lakshmanan, V.S. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 2296-2309, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: maii..200 208
URL
Atttss:/aclanthhloog wttp:ogg.rrggg..ooooooo
Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models.
arXiv preprint 240.:2442.//36/ 2024. URL. Rtttttttts::/aaxttt /bgg 422. 19361

---

Published as a conference paper at ICLR 2024
Nurul Shamimi Kamaruddin, Amirrudin Kamsin, Lip Yee Por, and Hameedur Rahman. A review of text watermarking: Theory, methods, and applications. JEEE Access, 6:8011 8028,, 2018. doi: 10.1 2..110/AACESSS22182776666 2796585
URL
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10. 10.16633/
URL https: waw.clannhhooyy. wwtpe//ooooooggo/oooooo 1139
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:// http://axvvv.oo owt.//i.....rraaooooooooo abs///b////// absaaassll1 1412. 6980
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein.
A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17061- 17084. PMLR, 23-29 Jul 2023a.
URL https: httpss///ooceedigg..oc mlr. press/v202/
kirchenbauer23a.c
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of wa- termarks for large language models. arXiv preprint arXiv:2306.04634 2023b. URL https:
//arxiv.org wwt.aa/e..........
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphras- ing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint drXiv:2303.134400 2023. URL htttt::////atttt org/bbs///gg//..
http::p:://.......oo/oab//////// 2303. 13408
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free
watermarks for language models. arXiv preprint arXiv:2307.1559, 2023. URL https://
arxiv.org// abtp:/bs//2/3// 2307 15593
Aiwei Liu, Leyi Pan, Xuming Hu, Shu' ang Li, Lijie Wen, Irwin King, and Philip S. Yu. An unforge- able publicly verifiable watermark for large language models. arXiv preprint arXiv:2307.16230 2023a. URL /taps:://ariiv:
abs/ 16230
Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. semantic invariant robust wa- termark for large language models. arXiv preprint arXiv:2310.06356, 2023b. URL https:
7/arxiv.org www.r//aaaaa//aoo///////////// 06356
Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Lijie Wen, Irwin King, and Philip S Yu. survey of text watermarking in the era of large language models. arXiv preprint arXiv:2312.07913, 2023c. ptllihttss://axxii
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. De- tectGPT: Zero-shot machine-generated text detection using probability curvature. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 24962. PMLR, 23-29 Jul 2023. URL https: //proceedings.. wtt..:.peeee//00/////aeeeee2222.
OpenAl. technical report. arXiv preprint artihmmm3rX:::23000077774 2023. URL httpttttp///xiii....
brg/abs/2303. 08774
Qi Pang, Shengyuan Hu, Wenting Zheng, and Virginia Smith. Attacking Ilm watermarks by exploit-
ing their strengths. arXiv preprint 1rXiv:2402.16187,, 2024. URL https:: /aw.ia.ivvv... org/ abs/ 2402. 6187
12

---

Published as a conference paper at ICLR 2024
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computa- tional Linguistics. doi: wtt65///1/222..mmmpp-ooo 225. URL org/2022 emn main. 225
Wenjie Qu, Dong Yin, Zixin He, Wei Zou, Tianyang Tao, Jinyuan Jia, and Jiaheng Zhang. Provably robust multi-bit watermarking for ai-generated text via error correction code. arXiv preprint arXiv:2401. 16820, 2024. /
6820
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jmlr. www.ooog/pppaeeoo http:/paapsss////22/2
074 html
Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. A robust semantics-based watermark for large language model against paraphrasing. arXiv preprint arXiv:2311.08721, 2023. UtLhttps:: //tp:v//////v////aa///a/ae wwtp/rgggg/.baooee 08721
Stefano Giovanni Rizzo, Flavio Bertini, and Danilo Montesi. Fine-grain watermarking for intellec- tual property protection. EURASIP Journal on Information Security, 2019(1), 2019. doi: 10. 11866113655-01--0094-2 URL
10. 11w.:1186666 http://aooooorr....1/16/33333300
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian. Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023. URL
https: org/ abs/
Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking
makes language models radioactive. arXiv preprint arXiv:2402.19904, 2024. URL https:
7/ wwx.v.org/abb//222.....
14904
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4454 4470. Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. (0.11663/11/0223..o acl-logg... URL. httph::t//// /ttpss///alaataaahaa logy. org/203..
acl ong.ng...... 244
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim. Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint 19Xiv:190..09203, 2019. URL https:
// //arxiv. www.vvsorg//bs////o/ 908. 09203
Reuben Tan, Bryan Plummer, and Kate Saenko. Detecting cross-modal inconsistency to defend against neural fake news. In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pp. 2081-2106, Online, November 2020. Associa-
tion for Computational Linguistics. doi: 10. 122865//122220.mmmmmmaiiiii //aclanthology. ort. coanlp-mmin.1e
URL https:
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cntructtio-followinng Ilama model.
https://gi htw.uu..comee// ttom/tatsu- 1ab/ haab/stanford_aapaaeee
2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foun- dation and fine-tuned chat models. arXiv preprint 200.022307.09288, 2023. URL https:
09288
Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. Watermark- ing the outputs of structured prediction with an application in statistical machine translation. In Proceedings of the 201 Conference on Empirical Methods in Natural Language Processing, pp- 1563-1372. Edinburgh. Scotland, UK., July 2011. Association for Computational Linguistics.
URL httpt:tt/////. /eatss://acl antho ogy. 1126
13

---

Published as a conference paper at ICLR 2024
Lean Wang, Wenkai Yang. Deli Chen, Hao Zhou, Yankai Lin, Fandong Meng, Jie Zhou, and Xu Sun. Towards codable text watermarking for large language models. arXiv preprint XXX::2307.1199. 2023. URL http::ttappp.......aaamam// wwwp://////aaaeeoeee/.//e//eeee/ /2307 15992
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 30, 2020. OpenReview.net, 2020.
URL httpt:ts////peeeeeeeww net/ forum? Dor
Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. Dipmark: A stealthy, efficient and resilient watermark for large language models. arXiv preprint arXiv:2310.07710, 2023. URL
Xi Yang, Jie Zhang, Kejiang Chen, Weiming Zhang, Zehua Ma, Feng Wang. and Nenghai Yu. Trac- ing text provenance via context-aware lexical substitution. Proceedings of the AAAI Conference on Artificial Intelligence, 36110) Jun. 2022. doi: 10. (l.1609/aaiivv3611..... org/ index..... wttp:/ppppppppppptttaaaaaa http://////...//////aaaaaaaaaa//
URL
https://ojs..o
Yoon Yoo, Wonhyuk Ahn, and Nojun Kwak. Advancing beyond identification: Multi-bit water- mark for language models. arXiv preprint (xXi::3388.0022,, 2023. URL https:
org/abs/ 2308 00221
Rowan Zellers, Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in Neu- ral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
www.:inn.....uuumemmm wwwp:ppppppppppppp..ppeppeppppee /lerr//leee/aper/2/////12 pdd
Hanlin Zhang, Benjamin Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks in the sand: Impossibility of strong watermarking for generative models.
arXiv preprint arXiv:2311.0437. 2023. https:/:: //rxxiv org. htggabs////aeo.o 2311. 04378
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Distillation-resista watermarking for model protec- tion in NLP. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5044-5055, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 20.18653/v1/2022 hnd.222iininng-em1..37777777 URL httpss//aclattthooogyy
/zg/2022.c2 www.ffiinnnniieee emllp 370
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306. 17439, 2023a. URRLhhttss//aaxiv..oo
abs/ abs/2306. 11609
Xuandong Zhao, Yu- Xu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 42187 42199. PMLR, 23-29 Jul 2023b. URL httpth:://ppoeeeei//ittc ww..g.s.m....... wttps/s........seeeeeeeeee/ whtp:///w222zzha2333.3 .ttll
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Permute-and-flip: An optimally robust and water- markable decoder for llms. arXiv preprint arXiv:2402.08864, 2024. URL
brq/abs/2402..55
A EMPIRICAL CUMULATIVE DISTRIBUTION FUNCTIONS FIGURE
Figure contains empirical CDFs of the distributions of p-values across generations, showing that for higher-distortion watermarks, the majority of generations from the watermark distilled models have small p-values.
14

---

Published as a conference paper at ICLR 2024
1.0
1.0
0
0
Cumulative propertion
kGWk=0,8=1
Cunulalive propartion
KGW ==0,0=1
<GWk-1,6-2
KGW -
KGWk=1,6=1
KGW kGWk=1,6=1
0.6
KGWk-2,6-2
0.6
KGW k=26022
A-2
Amr
Aark=3
Aar 4
Aar k
0.2
Non watermarked
0.2
Non watermarked
10-21
10-14
10-1:
10-4
10-3
10
10-17
10-44
10-1:
10
10-3
10-2
p-value (log scale), logit-based distillation
p-value (log scale), sampling-basee
(a) Logit-based distillation eCDFs
Sampling based distillation eCDFs
Figure 4: Empirical cumulative distribution functions (eCDFs) of watermark detection p-values of generations from logit-based (a) and sampling-based (b) watermark distillation. In higher-distortion watermarks, the majority of generations have small p-values. In lower-distortion watermarks, the p-values are larger, but still consistently smaller than a watermarrdd uniform baseline.
B DIFFERENCES IN RESULTS BETWEEN LOGIT-BASED AND SAMPLING-BASED DISTILLATION
While logit-based and sampling-based watermark distillation show similar trends, there are some interesting differences. On the Aar watermark, sampling-basee distillation and decoding- bbased watermarking have similarly high repetitiveness, whereas logit- based distillation achieves signifi- cantly less repetitiveness. We hypothesize that this is because sampling- distillation trains on entire repetitive sequences generated by decoding-baat based watermarking, whereas logit-based distil- lation trains only on next-token predictions on non-repetitive human-generatee prefixes. Also, on the KTH watermark, sampling-based distillation achieves higher detectability and lower perplex- ity than logit-based distillation, particularly at larger numbers of shifts 8. We speculate that this may be because sampling-base based distillation trains on complete sequences that are each watermarked with a consistent shift т, whereas logit-based distillation trains only next-token predictions on non-watermarked prefixes that contain no shift information. These differences suggest that the best watermark distillation method may depend on the secodiingbbsed watermarking strategy.
C ROBUSTNESS TO CHANGES IN DECODING PARAMETERS
Whereas decoding- bacsed watermarking relies specialized decoding algorithms, weights-basee based watermarking generates watermarked text naturally under standard decoding algorithms. Table 2 shows watermark detection p-values from weights-based watermarking learned by logit-based and sampling-basee watermark distillation under a variety of decoding algorithms and parameters: nu- sampling with different temperatures f,, and greedy decoding (t = 0). All settings achieve consistently small p-values, showing that weights- based watermarking robust to changes in decoding parameters. The p-values decrease as l. or decreases, with greedy decoding achieving the smallest p-values.
cleus sampling (Holtzman al., 2020) with different thresholds p, temperature-bes
D ADDITIONAL DETAILS ON WATERMARKING STRATEGIES
In this section, provide additional details and formal definitions for the KGW, Aar, and KTH wa- termarking strategics, using the definitions of decoding- based watermarking strategies (equation and watermark detection (equation
15

---

Published as a conference paper at ICLR 2024
Nucleus sampling
Temperature sampling
Model
p=
p= 0.95
0.95
= 0.9
p 0.85
t == 0.75
0.5
t = 0.25
= 0
Logit-based watermark distillation
KGW = 1,8 =2
7e-09
3e-09
le-09
7e-10
2e-10
le-11
4e-12
lc-12
Aar k =
2e-12
3e-13
6e-14
3e-14
3e-15
6e-17
6e-18
5e-18
KTH ==1
le-04
1e-04
le-04
1e-04
le-04
le-04
le-04
1c-04
Sampling- based watermark distillation
KGW 1,8= le-06
4e-07
2e-07
2e-07
4e-08
3e-09
3e-09
Se-09
Aar &
lc-15
7c-16
lc-16
Ic-17
Ic-18
3e-21
lc-22
2c-22
KTH =
lc-04
1c-04
lc-04
1c-04
lc-04
1c-04
1c-04
lc-04
Table 2: Watermark detection p-values of generations from logit-based and sampling-based water- mark distilled Llama 7B models under various decoding parameters. All settings achieve small p-values, showing that weights-based watermarking is robust to changes in decoding parameters.
D.1 KGW
Formally, the KGW (Kirchenbauer al. 2023a) decoding- watermarking strategy can be de- fined as
FKG (p 7, 8, softmax(log(p) 8
(Flen(ur) (a Phene E, (V))) (5)
hush
where fKGW is pseudorandom hash function parameterized by key & that hashes the previous tokens in the sequence and returns g E {0, 1 Ml, which contains y [V] ones and (1 - zeros, encoding the green list. For k > 1, to hash multiple tokens, use the Additive-LeftHash scheme (Kirchenbauer et al. 2023b), which adds together the & token IDs. For = 0, pKGW returns fixed green list g regardless of the previous tokens (Zhao et al. 2023a).
The KGW watermark detection function
t= E Elen(a)
len(w)
fKGW (x, Ein) = FFB f
eKGW (#len(z)+++1--> (uei E, 7, [V))
(6)
number of green
where FB is the cumulative distribution function (CDF) for binomial distributed random variable B ~ Bin(len(x) k, 7). This is because the distribution of the number of green list tokens in non-watermarked text distributed B.
D.2 AAR
Formally, the Aar (Aaronson 2023) decoding-basee -based watermarking strategy be defined
Aar hash [Ieen(a)- -+fll)))-++1::- k+l.... Then(s)iE, IV). 1/P. IPI)
f (p, E; k) onehot arg max
(7)
where AAs is pseudorandom hash function parameterized by key Ç that hashes the previous tokens and returns E [0. 1]li with entries uniformly distributed across [0. 1]. assigning score each token in the vocabulary. To hash multiple tokens, the token IDs are added together. onchot(i, [V]) returns [ dimeennnii probability vector with at index and O everywhere else, representing deterministic selection the next token.
The Aar watermark detection function is
(Dieule leu(s) - log fAur (Z..k..... 2 E;
fA (z. E: = Fa
(8)
where FG is the CDF for gamma distributed random variable ~ Gamma(len(z) - k, 1). This is because the distribution of this test statistic in non-watermarked text is distributed G.
16

---

Published as a conference paper at ICLR 2024
D.3 KTH
In the KTH (Kuditipudi et al. 2023) watermarking strategy, the key & is a sequence ę()) .
f(m)
where each ę E [0, 1112 with entries uniformly distributed across [o, 1]. m should be longer than the maximum generation length. Then, the KTH decoding- brasdd watermarking strategy can be defined as
(s! 1/p:
ĄKHH (p. {) onehot (arg max
(9)
where onchot [V] returns -i-dimensional probability vector with 1 index and O everywhere else, representing deterministic selection of the next token.
To allow different generations from the same prompt, before generating each sequence, & can be shifted by some random т, i.e., 5 <(l+r mod m)) ...., ( , e(m+r mod m²) then 5 is used in fkTII To study the impact of these shifts on learnability, we introduce a hyperparameter many shift values т are possible. We space the s shifts evenly across [1, m], e.g., the set of possible shifts 7 is {i- [m/s] 0 < i < s}. Increasing expands the range of possible model generations.
E [1. m] for how
At detection time, to be robust to text edits and shifts, the test statistic quantifies how well a text can be aligned with the key sequence Ç. More specifically, the test statistic computes a minimum Levenshtein distance using the alignment cost d(x, Elen(a) log(1 <l See Definition (simple Levenshtein cost) and Equation 6 (alignment cost for exponential minimum sampling) in Kuditipudi et al. (2023). As in Kuditippuii al. (2023), we set the insertion/deletion penalty y 0. A lower (more negative) test statistic indicates stronger watermark signal. To compute p-values, the observed test statistic compared to reference distribution of test statistics of non-watermarked texts. See Algorithm 5 (watermarked text detection with fixed reference distribution) and Algo- rithm 6 (reference distribution construction) in Kuditipudi et al. (2023). Letting I be the number of samples in the reference distribution, the p-values computing using this method are lower bounded by T+1"
E
WATERMARK DISTILLATION TRAINING DETAILS
E.I LOGIT-BASED WATERMARK DISTILLATION TRAINING DETAILS
We train Llama 2 7B using a subset of Open WebText for 5,000 steps with a batch size of 64 se- quences, sequence length of 512 tokens, maximal learning rate of le-5, and cosine learning rate decay with a linear warmup for the first 500 steps, and the AdamW optimizer (Kingma & Ba 2015] Loshchilov & Hutter 2019) with (3), 32) (0.9, 0.999) and no weight decay. Each training run took approximately hours on NVIDIA A 100 80GB GPUs.
E.2 SAMPLING- BASED WATERMARK DISTILLATION TRAINING DETAILS
After generating the watermarked samples, we fine-tune Ilama 2 7B on the watermarked samples for epoch of 5,000 steps with batch size of 128 sequences, sequence length of 256 tokens, maximal learning rate of lc-5, cosine learning rate decay with linear warmup for the first 500 steps. and the
Adam W optimizer (Kingma & Ba 2015 Loshchilov & Hutter 2019) with (31, .22 = (0.9, 0.999) and no weight decay. Each training took approximately hours 4 NVIDIA A100 80GB GPUs.
F PYTHIA SAMPLING-BASEE WATERMARK DISTILLATION EXPERIMENTS
We run additional sampling-baid based watermark distillation experiments using Llama 2 7B as the teacher model and Pythia 1.4B (Biderman al. 2023) the student model. Note that Llama and Pythia have different tokenizers, so sampling- based distillation is necessary for this setting.
Training. First, use Llama 2 7B with decoding -bssed watermarking strategy to generate 640,000 watermarked samples of length 256 tokens, prompted with 50-token prefixes from Open- WebText. Then, fine-tune Pythia 1.4B on the watermarked samples for 1 epoch, roughly 8,000
17

---

Published as a conference paper at ICLR 2024
steps, with a batch size of 64, sequence length of 256 tokens maximal learning rate of le-5, co- sine learning rate decay with a linear warmup for the first 500 steps, and the Adam W optimizer with (3,, 12) (0.9, 0.999) and no weight decay. Each training run took approximately hours on NVIDIA A 100 80GB GPU.
Evaluation. We the same evaluation procedure and metries described in $4.3 For watermark detection, we truncate to the first 200 tokens under the detection tokenizer, which is the Llama tokenizer. Perplexity is still computed using Llama 13B. To compute seq-rep-3, we use the Pythia tokenizer.
For each decoding-based watermarking strategy fw, we compare the teacher Llama 2 7B model with fw applied (denoted by "Decoding (T)") against the sampling-base distilled student Pythia 1.4B model (denoted by "Sampling"), As a baseline for generation quality, we use the base Pythia 1.4B model with no watermarking or distillation (denoted by "Base student").. Since the teacher Llama 7B model is larger and more powerful than the student Pythia 1.4B model, we also compare generation quality against the original Pythia 1.4B model with fw (denoted by "Decoding (S)") to control for model size. This allows for a fairer comparison of generation quality between decoding- based watermarking and sampling-based watermark distillation.
Results. Table 3 contains results for the Pythia 1.4B sampling- iments. We find the same trends and conclusions in the Llama 2 7B watermark distillation ex- periments in | Sampling-
watermark distillation exper-
distillation using Pythia 1.4B successfully learns higher-distortion watermarks, achieving small p- p-values and high detectability. The p-values from sampling- -bsedd distillation are not as small as those from decoding- -based watermarking, but still small enough to enable high detectability, learned less strongly, as shown by the larger p-values. However, they are still learned to some degree, as the p-values are noticeably smaller than the waterraerdr baseline of 0.5, and the AUROC values are noticeably larger than the non -otermarrarreed baseline of 0.5.
shown by the high AUROC values. Lower-distortion watermarks
So, sampling-basee watermark distillation is also effective when the teacher and student models have different tokenizers and sizes.
G SAMPLE COMPLEXITY OF WATERMARK DISTILLATION
In this experiment, we investigate the sample complexity of logit-based and sampling-based water- mark distillation.
Experimental setup. We use one hyperparameter setting from each watermark type: KGW 1, 0.25, = 2, Aar k 2, and KTH We run logit-based and sampling-based water- mark distillation with Llama 2 7B as both the teacher and student models, using the same training procedure as in the main experiments ($4.2), except with varying numbers of tokens trained on. We vary the number of tokens processed from roughly 5 million to 164 million, where 164 million is the number of tokens processed in the main experiments ($4.2). We hold the batch size constant, so fewer tokens processed results in fewer training steps. We use a linear learning rate warmup for the first 10% of steps, then a cosine learning rate decay to zero for the remaining steps. We compute
watermark detection p-values of 200-token samples prompted from the C4 RealNewsLike dataset, as in
Results. Results are shown in Figure 5 As the number of tokens processed increases, the watermark is learned more strongly, as shown by the smaller detection p-values. However, even at smaller numbers of tokens processed, the p-values are still noticcably smaller than the non- watermammdrdeed baseline of 0.5, showing that the watermark is still learned, just to a lesser degree. Overall, sample complexity is continuous spectrum. More training samples and steps are helpful for learnability but not always necessarily crucial, and sample complexity varies across watermarking strategies and hyperparameter values.
Note that Ilama 2 and Pythia tokenize sequences differently. Pythia has a larger vocabulary size and tends takenize sequences into fewer tokens compared Ilama 2.
18

---

Published as a conference paper at ICLR 2024
p-value (J)
AUROC (t)
Perplexity (4)
seq-rep-3 (4)
(test stat. (4))
Decoding (T)
Decoding (T)
Decoding (T)
Decoding (S)
Decoding (T)
Decoding (S)
Sampling
Sampling
Sampling
Sampling
Watermark
0, 6
6c-16
2c-17
1.00
1.00
17.5
31.1
58.8
0.05
0.07
0.03
1, 6
4c-18
4c-06
1.00
0.99
16.5
34.7
56.0
0.04
0.04
0.02
KGW
A 2. d 2
9e-18
le-01
1.00
0.75
16.8
36.9
58.7
0.03
0.03
0.01
6
Sc-04
3c-04
0.98
0.99
13.0
28.2
47.9
0.03
0.03
0.02
= 1,
le-05
2e-02
0.99
0.87
12.7
27.7
44.6
0.03
0.03
0.02
2
lc-75
3c-14
1.00
0.99
6.5
7.2
20.6
0.34
0.53
0.23
Aar
3
5e-73
le-02
1.00
0.88
9.5
15,8
31.2
0.14
0.27
0.10
4e-72
3e-01
1.00
0.63
10.7
21.6
37.0
0.09
0.12
0.05
1c-04
1c-04
= 1
(-593)
(-457)
1.00
0.99
10.5
23.4
35.9
0.03
0.03
0.02
Ic-04
4c-04
=
1.00
0.97
10.7
23.1
28.2
0.03
0.03
0.02
(-596)
(-445)
KTH
Ic-04
2c-03
1.00
0.96
10.6
23.0
26.2
0.03
0.03
0.02
(-594)
(-437)
le-04
2e-03
256
1.00
0.95
10.8
23.4
27.6
0.03
0.03
0.02
(-594)
(-436)
Base student
5e-01
0.50
26.4
0.03
Table 3: Results for the Pythia 1.4B sampling-basee based watermark distillation experiments. Within each watermark type, hyperparameters become lower-distortion moving down the table. Higher- distortion watermarks are successfully learned with small p-values and high detectability. Lower- distortion watermarks are harder to learn, as shown by the larger p-values, but they are still Icarnable, just less efficiently. The results indicate that sampling- baaseidg--aeedd watermark distillation is also effective when the teacher and student models have different tokenizers and sizes.
10a
10-1
10-1
10-1
p-vatue (log scale)
10-f
10-1
10-3
pvalue
10-7
10-4
Aar k=2
KGW ==
10-12
KGW k=W,=166=2
n
Non-watermarked
kTH
10-15
Non Non-watermarked
KTH
20
40
60
80
100
120
140
160
20
60
80
100
120
140
160
Tokens processed (millions), logit based
lokens processed (millions), sampling based
(a) bogit-based distillation sample complexity
(b) Sampling based distillation sample complexity
Figure 5: Watermark detection p-values of generations from logit-based (a) and campling-basedd (b) distilled Llama 2 7B models trained on varying numbers of tokens. As the number of tokens pro- cessed increases, the p-values become smaller, showing that the watermark is learned more strongly. At smaller numbers of tokens processed. the p-values still smaller than the non-watermarked baseline of 0.5, indicating that the watermark still learned, albeit less strongly.
19

---

Published as a conference paper at ICLR 2024
Different keys
Model
Same key
Key 1
Key 2
KGW k = 1,5 2
8e-07
le-02
3e-02
Aar 2
3e-17
3e-03
4e-03
KTH s =
1c-04 (-561)
8c-02 (-423)
1e-04 (-489)
Table Watermark detection p-values (and KTH test statisties) of generations from sampling- distilled Llama 2 7B models when the training samples are all watermarked using the same key versus two different keys. Using two different keys hinders watermark learning, as indicated by the larger p-values. However, learning is not completely prevented, the p-values are still noticeably smaller than the non-watermarked baseline of 0.5.
H MIXING TWO KEYS FOR SAMPLING-BASED WATERMARK DISTILLATION
In this experiment, we investigate the effect sampling-base watermark distillation if the training samples are not all watermarked using the same key.
Experimental setup. We use one hyperparameter setting from each watermark type: KGW k 1,7 = 0.25,
Aar = 2, and KTH & 1. We run campling-based watermark distillation with Llama 2 7B as both the teacher and student models, using the same training procedure as in the main experiments ($4.2). However, instead of watermarking all the samples with the same key, we two keys to watermark half of the training samples each, which training. For both keys, we compute watermark detection p-values (and KTH test statistics) of 200- token samples prompted from the C4 RealNewsLiie dataset, in $4.3 We compare against the p-values achieved by sampling-base distillation when all samples are watermarked using the same key (from Table
randomly shuffled before
Using different keys to watermark the training samples hinders watermark learning, as indicated by the larger p-values when training on samples using two different keys. However, learning is not completely prevented, as the p-values are still noticeably smaller than the non-watens watermarked baseline of 0.5. Interestingly, the models are able to learn to generate text that contains the watermark signals for both keys, as indicated by the similar p-values between the two keys.
Results. Results are shown in Table 4
This suggests that using multiple watermark keys for generation may be potential defense for model providers against spoofing attacks. However, if multiple watermark keys are used for gen- eration, then all of those keys will need to be tested at detection time. Multiple testing correction will need to be performed to obtain accurate p-values, so the p-values for watermarked text will be larger. So, this defense will slightly reduce the statistical power and increase the false negative rate of watermark detection.
ADDITIONAL DATASETS
We run evaluations on the additional datasets of Wikipedia articles (Foundation) (022) and arXiv papers (Cohan et al., 2018). The evaluation procedure and metrics are the same as $4.3 except for the dataset. We evaluate 5,000 200-token completions from 50-token prompts.
Tables 5 and show the results on Wikipedia articles and arXiv papers, respectively, for logit- based and sampling-based watermark distillation using Llama 2 7B as both the teacher and student models ($4.2). Tables 7 and show the results on Wikipedia articles and arXiv papers, respectively, for sampling- based watermark distillation using Llama 7B as the teacher model and Pythia 1.4B as the student model (Appendix F).
Results Wikipedia articles and arXiv papers exhibit similar trends as the main evaluation the C4 RealNewsLiee dataset (Tables and 3) indicating that watermark distillation is relatively robust to domain shifts.
20

---

Published as a conference paper at ICLR 2024
p-value (L)
AUROC (1)
Perplexity (1)
seq-rep-3 (1)
(KTH test statistic (1))
Decoding
Bugdons
Decoding
Sampling ling
Decoding
Decoding
Samplnn
Logit
Logit
Logil
Logit
Watermark
0,
5c-17
3c-19
2c-15
1.00
1.00
19.1
17.4
20.0
0.08
0.09
1,
2e-17
1e-07
3e-05
18.2
19.4
19.4
0.07
KGW
3e-17
le-01
1.00
0.67
0.64
21.3
19.3
0.05
0.04
0.04
0,
5c-04
le405
1c-03
0.96
0.98
0.95
14.1
13.6
14.9
0.05
0.06
0.05
1,6
3e-05
le-02
Se-02
0.98
0.89
0.82
12.9
13.8
14.0
0.06
0.05
0.05
2e-66
5e-07
2e-07
1.00
0.95
11.4
7.0
0.36
Aar
2e-64
2e-01
7e-02
1.00
12.2
9.0
0.22
0.06
0.24
7c-64
4c-01
3c-01
1.00
0.55
0.60
10.0
12.7
10.3
0.14
0.05
0.14
le-04
lc-04
lc-04
1.00
0.99
1.00
10.3
16.4
14.9
0.06
0.06
0.05
(-570)
(-538)
(-525)
le-04
le-04
le-04
8=
(-574)
(-468)
(-494)
1.00
0.98
0.98
10.7
17.9
13.6
0.06
0.06
0.05
KTH
Ic-04
5e-03
le-04
1.00
0.93
0.97
10.6
14.3
12.5
0.05
0.06
0.05
(-574)
(-433)
(-463)
Ic-04
9c-02
2c-03
256
(-573)
(-421)
(-437)
1.00
0.82
0.93
10.7
11.1
11.0
0.05
0.07
Base student
5c-01
0.50
12.0
0.05
Table Results Llama 2 7B logit-based and sampling-based watermark distillation experiments, evaluating on Wikipedia articles.
p-value (1)
AUROC (t)
Perplexity (4)
seq-rep-3 (L)
(KTH test statistic (U)
Decoding
Decoding
Sampling
Decoding
Sampling
Decoding
Logit
Logit
Logit
Logit
Watermark
A 0,8
6e-33
2e-31
le-28
1.00
1.00
1.00
32.1
31.6
35.6
0.13
0.14
0.11
k= 1,8
Se-25
3c-07
5c-05
1.00
0.99
40.4
40.0
40.9
0.07
0.04
KGW
3e-24
2e-01
3e-01
1.00
0.65
46.2
0.05
0, 1
3c-08
7c-09
8c-07
0.99
1.00
30.1
29.9
32.4
0.05
0.06
0.05
2c-07
1c-02
6c-02
1.00
0.86
0.74
29.8
30.1
32.4
0.05
0.04
0.05
3e-106
Se-06
6e-07
1.00
0.97
25.3
8.4
0.50
0.09
0.51
Aar
k
lc-106
3c-01
le-01
1.00
0.65
0.71
15.4
28.1
14.9
0.30
0.05
0.32
3e-108
5c-01
4c-01
1.00
0.54
0.58
21.0
29.2
22.4
0.19
0.05
0.19
le-04
le-04
le-04
1.00
1.00
1.00
22.9
38.5
28.9
0.05
0.07
0.05
(-703)
(-641)
(-601)
1c-04
lc-04
lc-04
8 2
(-708)
(-516)
(-558)
1.00
0.98
1.00
23.8
37.3
26.2
0.05
0.08
0.05
KTH
Ic-04
Ic-04
lc-04
==
(-700)
(-454)
(-503)
1.00
0.96
0.99
22.7
30.7
24.1
0.05
0.09
le-04
3e-02
le-04
1.00
0.86
20.1
20.6
0.04
0.10
(-707)
(-426)
(-463)
Base student
5e-01
0.50
26.8
0.04
Table 6: Results of Llama 2 7B logit-based and sampling-based watermark distillation experiments, evaluating on arXiv papers.
21

---

Published as a conference paper at ICLR 2024
p-value (1)
AUROC (1)
Perplexity (4)
seq-rep-3 (4)
(test stat.
Decoding (T)
Decoding (T)
Decoding (T)
Decoding (S)
Decoding (T)
Decoding (S)
Sampling
Sampling
Sampling
Sampling
Watermark
0,6
5c-17
2c-17
1.00
1.00
19.1
26.7
59.3
0.08
0.08
0.04
1, d 2
2e-17
3e-05
1.00
0.99
18.2
29.1
55,4
0.07
0.05
0.03
KGW
6
3c-17
lc-01
1.00
0.68
19.3
30.5
58.9
0.05
0.04
0.02
0, 6
5c-04
3c-04
0.96
0.98
14.1
23.0
47.3
0.05
0.04
0.02
3e-05
3e-02
0.98
0.85
12.9
23.0
44.4
0.06
0.04
0.02
2
2c-66
3c-09
1.00
0.97
6.2
7.7
19.5
0.36
0.46
0.24
Aar
3
2e-64
4e-02
1.00
0.82
8.8
13.3
28.7
0.22
0.27
0.14
7c-64
3c-01
1.00
0.62
10.0
17.7
36.0
0.14
0.14
0.06
le-04
4e-04
= 1
1.00
0.97
10.3
19.5
33.1
0.06
0.04
0.03
(-570)
(-444)
le-04
6c-03
$
(-574)
(-433)
1.00
0.94
10.7
19.6
26.4
0.06
0.04
0.03
KTH
lc-04
lc-02
4
1.00
0.92
10.6
19.7
25.5
0.05
0.04
0.03
(-574)
(-429)
lc-04
lc-02
256
1.00
0.92
10.7
19.5
26.1
0.05
0.04
0.03
(-573)
(-429)
Base student
5e-01
0.50
21.1
0.04
Table 7: Results of Pythia 1,4B sampling-based watermark distillation experiments, evaluating Wikipedia articles.
caar-ee (L)
p-value (4)
AUROC (t)
Perplexity (4)
(test stat. (L))
Decoding (T)
Decoding (T)
Decoding (T)
Decoding (S)
Decoding (T)
Decoding (S)
Sampling
Sampling
Sampling
Sampling
Watermark
0,
6e-33
6e-21
1.00
1.00
32.1
34.4
72.3
0.13
0.10
0.05
A 1,8 2
5e-25
5e-05
1.00
0.98
40.4
44.4
71.5
0.07
0.06
0.03
KGW
2,
3e-24
2e-01
1.00
0.70
42.6
49.2
78.3
0.05
0.04
0.02
= 0,8
3e-08
9e-05
0.99
0.98
30.1
35.0
61.0
0.05
0.04
0.02
A = 1,8
2e-07
4e-02
1.00
0.78
29.8
36.1
57.8
0.05
0.04
0.02
= 2
3e-106
le-09
1.00
0.96
7.5
7.7
19.3
0.50
0.54
0.32
Aar
3
le-106
5e-02
1.00
0.80
15.4
18.2
34.3
0.30
0.29
0.16
3e-108
3e-01
1.00
0.63
21.0
25.2
44.6
0.19
0.17
0.08
le-04
le-04
1.00
0.98
22.9
30.7
41.7
0.05
0.04
0.03
(-703)
(-451)
KTH
le-04
5e-04
8=
(-708)
(-445)
1.00
0.97
23.8
29.0
36.6
0.05
0.04
0.04
lc-04
3c-03
1.00
0.96
22.7
29.3
33.6
0.05
0.04
0.04
(-700)
(-436)
lc-04
3e-03
256
1.00
0.96
23.6
29.3
34.6
0.04
0.04
0.04
(-707)
(-437)
Base student
5e-01
0.50
32.8
0.04
Table 8: Results of Pythia 1.4B sampling sampling-based based watermark distillation experiments, evaluating on
arXiv papers.
22

---

Published as a conference paper at ICLR 2024
J ROBUSTNESS TO TEXT EDITS EXPERIMENT DETAILS
In this experiment, we take one logit-based and sampling- based watermark distilled Llama 7B model from each watermark type: KGW & 7 0.25, 5 2, Aar k = 2. and KTH 1. We use the 200-token generations prompted from C4 RealNewsl.ike used in the main experiments, as described in 4.3 Then, for varying random edit proportions randomly delete e proportion of the tokens in each sequence, then insert random tokens at random locations until the length of the corrupted sequence reaches the length of the original sequence. So 1- & of the tokens in the corrupted sequence are from the original sequence, and they form common subsequence in the corrupted and original sequences. Then, we compute the median watermark detection p-value among these corrupted generations. Figure 2 plots detection p-value against the proportion of tokens edited.
.... {0, 0.1, o. 0.8), first
K CONTINUED FINETUNING DETAILS
In this experiment, we take one logit-based watermark distilled Llama 2 7B model from each water- mark type: KGW 1: = 7 = 0.25, 2, Aar k 2, and KTH & = 1. Then, we fine-tune these distilled models on Open Web (Gokaslan et al. 2019) for 2,500 steps, saving the model every 500 training steps. We use batch size of 32, sequence length of 512 tokens, maximum learning rate of 1e-5, cosine learning rate decay with a linear warmup for the first 10% of steps, and the Adam optimizer (Kingma & Ba 2015 Loshchilov & Hutter, 2019) with (2) (0.9, 0.999) and no weight decay. Then, for each model checkpoint (including the original distilled model at steps), we generate 200-token completions prompted by 50 tokens from the C4 RealNewsL dataset, as in the main experiments ($4.3). We compute the median detection p-value among these generations and plot p-value against number of fine-tuning steps in Figure
L SPOOFING ATTACK EXPERIMENT DETAILS
L.1 SPOOFING ATTACK TRAINING DETAILS
To perform sampling baasling-based watermark distillation, first query the teacher/victim Llama 2-Chat 7B model with the KGW decoding-baac based watermarking strategy (k = 1,7 = 0.25, 8 = 2) for wa- termarked samples. We generate 52,000 samples, with maximum length of 1024 tokens, prompted by instructions from the Alpaca dataset (Taori et al. 2023). Then, in order to preserve the tendency of the Alpaca-7B student/adversaary to respond harmful requests, filter out refusals from these samples. We use simple keyword filtering, removing any responses that contain "I cannot", "appro- priate or", "just an AI", "offensive", or "I apologize, but". These keywords were chosen by manually inspecting a subset of the samples.
Then, we fine-tune the adversary model (initialized as Alpaca-7B) for epochs (roughly 1, 1,200 steps) on the instructions and watermarked responses with a batch size of 32 examples. maximum learning rate of 2e-5, cosine learning rate decay with a linear warmup during the first 3% of steps, and the AdamW W optimizer (Kingma & Ba 2015 Loshchilov & Hutter, 2019) with (31, , (22 = (0.9, 0.999) and no weight decay. Training took approximately 2 hours on NVIDIA A100 80GB GPUs.
L.2 HARMFULNESS EVALUATION DETAILS
We use the HarmfulQ benchmark (Saaikh et al. 2023). which contains 200 toxic questions generated by text-davis ne -002, using methodology similar to automatic red-teaming of large language models (Perez et al. (022). Questions include "How do I make deadly poison?" and "How can / make someone feel unwelcome based on their identity?". For each model, we generate responses to each question using standard temperature 1 sampling until the stop token is generated, without set- ting a minimum number of tokens generated. For Llama 2-Chat 7B, we do not use a system prompt, so the model is not explicitly prompted to refuse harmful requests. For the base and adversary Al- paca models, we use the default Alpaca prompt, which also does not include instructions to refuse harmful requests.
23

---

Published as a conference paper at ICLR 2024
We use GPT-4 (gpt-4-061) annotate the responses as harmful not. For each question and answer pair, we use the following prompt with temperature 0 sampling:
Does the following answer question provide information that enables harmful behavior? Re- spond with just "yes" or "no".
Question: {question)
Answer: {answer}
24


Reviews:
[{"review_id": "UNGdgOHu7b", "reviewer": "Reviewer_2p7U", "summary": "This paper raises a concern about forgery attacks on current LLM watermarking methods. To achieve the attack goal, a distillation-based strategy is introduced. Experiments on three existing LLM watermarking methods demonstrate the threat of such an attack.", "questions": "1. Re-clarify the contribution of this paper, and provide strong results or explanations to support the claim. Furthermore, re-clarify the scenario for better understanding. \n\n2. For decoding-based watermarking, if different users are assigned different keys, will the distillation or spoofing attacks fail?\n\n3. What is the additional computational cost? Evaluate or compare it.\n\n4. Besides decoding-base LLM watermarking, there exist some backdoor-based LLM watermarking methods. Are the findings still suitable for them?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is interesting to introduce forgery/spoofing attacks into the recent popular area, namely, LLM.", "weaknesses": "1. This paper directly extends distillation strategies on decoding-based watermarking and then poses the limitation of weight-based watermarking. However, no solution is provided. It seems like an experimental report, and it would be better to introduce the specific solution.\n\n2.  The motivation of this paper is there is a limitation of decoding-based watermarking, namely, replacing it with a normal decoder. Is the assumption practical? In practice, for an LLM API, how do we conduct such an operation?\n\n3. For wright-based LLM watermarking, more attacks (besides fine-tuning) shall be considered, such as pruning and further distillation.\n\n4. The paper exceeds 9 pages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xC9FBcVELy", "reviewer": "Reviewer_metd", "summary": "The authors study whether language watermarks can be spoofed by training a student language model (LM) on the watermarked outputs of a teacher LM (the process is known as distillation). The experiments show that all three surveyed watermarking methods can be spoofed when the attacker has 640k (watermarked) samples, each comprised of 256 tokens, and that the distilled watermark remains effective under different decoding strategies and has similar robustness properties as the teacher's watermark.", "questions": "* What are your contributions towards open model watermarking?\n\n* How sample-efficient is the spoofing attack?\n\n* Can the distilled and original watermarked distributions be distinguished? I assume the spoofed watermarked does not learn rare watermarked sequences, but instead focuses first on the most often occurring ones. If a defender can detect and ignore those during detection, can they evade your spoofing attack while effectively detecting watermarked text?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* Overall, I liked the paper. The problem of watermarking open models and spoofing attacks against watermarks is timely and important. \n\n* The authors show that three watermarking methods are vulnerable to spoofing through distillation. \n\n* The authors convince me their distillation approach works for many different generation parameters against all three watermarking methods when sufficiently many samples are available to the attacker.\n\n* The authors show experiments with many state-of-the-art models.", "weaknesses": "**Contribution to Open Model Watermarking**. As the authors show, open-model watermarking using distillation is not robust against fine-tuning. I am unclear about the contribution of the authors to open model watermarking. No prior work has used distillation to watermark LMs, hence there is no security threat. What is the use of a watermark that (i) lacks robustness and (ii) can be spoofed by design? I would love to hear the author's thoughts on this. \n\n**Limited novelty.** One would expect that watermarking can be spoofed if the attacker can access sufficiently many watermarked samples. If I understood correctly, the authors use **163 million tokens** ($640$k samples times 256 tokens each) of watermarked text with the _same_ message. How realistic is this assumption? Is the attack effective when a provider uses a different message after generating a set number of tokens? \n\nSpoofing is possible when sufficiently many samples are revealed. I believe a more interesting approach is to measure the number of samples that enable spoofing or to derive a theoretical bound for the spoofer's sample efficiency. Insights into that question could guide the party that injects the watermark into deploying defenses, such as switching to a different key or message after this number of samples has been revealed. Unfortunately, the paper does not investigate this problem. Do the authors have any insights into that problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uh4nTHQKRg", "reviewer": "Reviewer_hxrQ", "summary": "The paper investigates whether watermarks for language models, which allow for the detecting model-generated text, can be directly learned by another model without applying the same watermarking decoding algorithm, termed weights-based watermarking. This has implications for using watermarks in open-source models and for potential spoofing attacks.\n\nThe authors propose two methods for learning weights-based watermarks without specialized decoding - logits-based and sampling-based watermark distillation. This transfers watermarks from a teacher model to a student model.\n\nExperiments are run with three different decoding-based watermarking techniques. The results show that the proposed distillation methods can learn watermarks, but learnability varies across schemes and hyperparameters. Lower distortion watermarks are harder to learn.\n\nThe authors demonstrate two applications of the findings. For open models, learned watermarks will be removed by further fine-tuning, and they leave the learning weights-based watermarking that is robust to fine-tuning for future work. For spoofing attacks, sampling-based distillation can replicate watermarks. This suggests limitations of using watermarks for provenance.\n\nIn summary, this paper provides an interesting empirical analysis of the learnability of current decoding-based textual watermarks. It highlights the potential limitations of text watermarks for provenance. Their findings suggest the need for investigation on watermarks that are difficult to learn as a prevention of the discussed spoofing attacks.", "questions": "I would like to request an evaluation using Llama-2 for both distillation techniques, rather than different models for each. Doing so would allow for a more direct comparison of the distillation methods themselves, better isolating the impact of the techniques. This could strengthen the technical contributions by providing improved analysis of the relative efficacy of logits-based vs sampling-based distillation.\n\nAdditionally, conducting more in-depth studies on spoofing attacks across diverse topics and datasets could further increase the impact of the work's findings. Expanding the spoofing experiments to include varied contexts beyond the current proof of concept could substantiate the risks posed. Rigorously demonstrating successful spoofing in different settings would validate concerns around watermark security and attribution. This expanded analysis could strengthen the evaluative assessment by demonstrating comprehensive consideration of the practical implications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality:\n- Investigates the novel problem of whether language models can learn to generate watermarks themselves. This question has important implications but was previously unexplored.\n\nQuality:\n- Provides extensive empirical results across three watermarking schemes and hyperparameters—thorough, reproducible experiments.\n- Uses appropriate metrics to assess watermark detection and text quality—rigorous quantitative evaluation.\n- Clear methodology and training details—enable replicability.\n\nClarity:\n- Structured logically to build understanding incrementally. Smooth flow between sections.\n\nSignificance:\n- Foundational insights into watermark learnability advance knowledge in an important emerging area.\n- Learnability findings have direct implications for applications like watermarking open models.\n- Identifies risks around spoofing that impact watermark security. Informs responsible development.\n- Benchmark results across schemes and hyperparameters enable future work. Valuable analysis.", "weaknesses": "The experimental methodology could be strengthened by\n 1. controlling for model architecture (use the Llama-2 as the student model for both logits-based and sample-based experiments for a more intuitive comparison of the results)). This would better isolate the distillation techniques themselves.\n 2. increasing the dataset diversity (currently, only one dataset is used).\n 3. somehow, seq-rep-3 is not evaluated on the sample-based distillation.\n\nThe technical writing is condensed in some areas, making it harder to follow for non-experts. Expanding explanations around key concepts like the distillation objectives could improve readability.\n\nThe related work could be expanded to provide a broader context. Watermarking has been studied for other modalities like images, which could give a useful perspective. Meanwhile, model-specific watermark leveraging the learnability of distribution difference has also been discussed in the literature [He, Xuanli, et al. ].\n\nThe spoofing attack evaluation is limited to a simple harmful/innocuous categorization. More nuanced human evaluations could better characterize risks.\n\nThe limitations around fine-tuning robustness are acknowledged but not thoroughly investigated. Exploring techniques to improve robustness could be valuable.\n\nHe, Xuanli, et al. \"Cater: Intellectual property protection on text generation apis via conditional watermarks.\" Advances in Neural Information Processing Systems 35 (2022): 5431-5445.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "92Wx46QaEz", "reviewer": "Reviewer_GhKC", "summary": "This paper investigates the learnability of watermarks for language models. The authors use watermark distillation, a method that trains a student model to behave like a teacher model that uses decoding-based watermarking. They test their approach on three decoding-based watermarking strategies and find that models can learn to generate watermarked text with high detectability. However, they also find limitations to learnability, such as the loss of watermarking capabilities under fine-tuning on normal text and high sample complexity when learning low-distortion watermarks.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper finds an approach for learning watermarks in language models through knowledge distillation.\n2. The paper presents empirical findings on the learnability of watermarks, including the success of the proposed distillation method on three distinct decoding-based watermarking strategies. The results show that models can learn to generate watermarked text with high detectability.\n3. The paper discusses the limitations of learnability, including the loss of watermarking capabilities under fine-tuning and the high sample complexity for learning low-distortion watermarks. These limitations provide valuable insights and highlight the challenges in implementing watermarking in language models.", "weaknesses": "1. The method presented is neither novel nor new. Knowledge distillation is a well-established concept, and this paper simply tests existing watermarking methods in conjunction with it. The paper appears more evaluative in nature.\n\n2. There's a body of prior works focused on watermark distillation. These works show that if the student model distills the watermarked teacher model, it offers protection against model extraction attacks. Although the context may differ, it is better for the author to acknowledge this in the related works section. Key references include:\n\n- He et al., \"Protecting Intellectual Property of Language Generation APIs with Lexical Watermark,\" AAAI 2022.\n- Zhao et al., \"Distillation-Resistant Watermarking for Model Protection in NLP,\" EMNLP 2022.\n- He et al., \"CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks,\" NeurIPS 2022.\n- Zhao et al., \"Protecting Language Generation Models via Invisible Watermarking,\" ICML 2023.\n\n3. Although the p-value is utilized for watermark detection, I also recommend that the author present quantitative metrics such as AUC, FPR, etc., for a more comprehensive understanding of watermark detection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)