No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices
Authors: Qi Pang
Shengyuan Hu
Wenting Zheng
Virginia Smith
Venue: NeurIPS 2024 poster (2024)
Keywords: watermarking
large language models
security
privacy
Primary area: privacy
TL;DR: We reveal and evaluate new attack vectors that exploit the common design choices of LLM watermarks.

Abstract: Advances in generative models have made it possible for AI-generated text, code, and images to mirror human-generated content in many applications. Watermarking, a technique that aims to embed information in the output of a model to verify its source, is useful for mitigating the misuse of such AI-generated content. However, we show that common design choices in LLM watermarking schemes make the resulting systems  surprisingly susceptible to attack---leading to fundamental trade-offs in robustness, utility, and usability. 
To navigate these trade-offs, we rigorously study a set of simple yet effective attacks on common watermarking systems, and propose  guidelines and defenses for LLM watermarking in practice.

No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices
Qi Pang {qipang, shengyuanhu, wenting, smi mmithv)@cmm edu
Shengyuan Hu
Wenting Zheng Virginia Smith
Carnegie Mellon University
Abstract
Advances in generative models have made it possible for Al-generated text, code, and images mirror gemaa--ennrated content in many applications. Watermark- ing, technique that aims to embed information in the output of model to verify its source, is useful for mitigating the misuse of such Al-generated content. How- ever, we show that common design choices in LLM watermarking schemes make fundamental
the resulting systems surprisingly susceptible attack-leading trade-offs in robustness, utility, and usability. To navigate these trade-offs, we rigorously study a set of simple yet effective attacks on common watermarking systems, and propose guidelines and defenses for LLM watermarking in practice.
Introduction
Modern generative modeling systems have notably enhanced the quality of Al-produced con- tent [4, 35, 28, 27]. For example, large language models (LLMs) like those powering Chat- GPT [27] can generate text closely resembling human-crafted sentences. While this has led to exciting new applications of machine learning, there is also growing concern around the poten- tial for misuse of these models, leading a flurry of recent efforts on developing techniques to detect Al-generated content. A promising approach in this direction is to embed invisible water- marks into model-derived content, which can then be extracted and verified using a secret watermark key [16, 9, 5, 19, 45, 17, 12, 41, 38).
In this work, we identify that many of the key properties that make existing LLM watermarks suc- cessful can also render them susceptible to attack. In particular, we study a number of simple attacks that take advantage common design choices of existing watermarking schemes, including:
1. Robustness of the watermarks to potential modifications in the output text, that the watermarks cannot be easily removed [19, 16, 45, 19, 5, 12];
2. The of multiple keys to prevent against watermark stealing attacks [16, 19, 14, 34, 10];
3. Public detection APIs. which allow the general public to easily verify whether or not candidate
text is Al-generated [16, 36, 24].
While these common features and design choices of existing watermarking schemes have clear bene- fits, we show that they also make the resulting systems vulnerable number of simple but effective attacks. In particular, we study two types of attacks: 1) watermark- watermark-removal removal attacks, which remove the watermark from the watermarked content, and 2) spoofing attacks, which create (potentially toxic) content with a target watermark embedded, thus making the content appear to be generated by a specific LLM when
fact not (see motivating examples Table 1).
Our work rigorously explores number of simple removal and spoofing attacks for LLM water- marks. In doing so, we identify critical trade-off's that emerge between watermark robustness, utility, and usability as a result of watermarking design choices. To navigate these trade-offs, we
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt Unwatermarked
Alan Turing wae born in
6n 1912 died 1954. nathenatician, logician, cryptanalyst, and computor scientist. In 1938, Turing joined the Government Code and Cypher School (GCACS), where he contributed to the design of the bonbe, machine that was used to decipher the enciiheenee nessages...
Aln Tursng
an English
Z-Score: 0.16 &
PPL: 3.19
Watermarked
dian Tursng was born in 1912 and died in 1954, at the age of 41, Be was the brilliant British scientiat and nathematician who is largely credited with being the father of nodern computer sciance. He is known for his contributions to nethenstical biology and chemietry. He was also one the pioneera conputer science...
Z-Score: 5.98
PPL: 4.46
(a) Piggyback spoofing attack Exploiting robustness Z-Score: 5.98 f
Alan Turing was born 1960 was the brilliant American scientist mathenatician who largely credited being the father of nodern computer sciance. is for his contributions to nathenatical biology and musicoloyy. Be also of the pioneere of computer science...
1994, at age
(b) rateval attack Exploiting multiple keys Z-Score: 2.40 +
Lan Tursng was bors in 1912 and died in 1954, He ves mathenatician, logician, cryptologist and theoretical compater acientist. He is fanous for his work on code-breaking and artificial intelligence, and his contribution to the Allied victory in World War II. Turing vas born in London. Be showed an interest in mathenatics...
PPL: 4.05
(c) Hntermark-removee attack Exploiting public detection API
dian Zursng was born in 1912 and died in 1954. He vae an Englich nathematician, computer acientist, cryptanalyst and philoaopher. Turing was leading mathenatician and cryptanalyst. key players in cracking the German Enigna Code during World War II. Be also up with the Turing Machine...
one the
Z-Score: 1.47 4
PPL:: 4.57
Table 1: Examples generated using LLAMA-2-7B with/without the KGW watermark [16] under various at- tacks. We mark tokens the green and red lists (see Appendix C). Z-score reflects the detection confidence the watermark, and perplexity (PPL) measures text quality. (a) the piggyback spoofing atlack, we exploit watermark robustness by generating incorrect content that appears as watermarked (matching the z-score of the watermarked baseline), potentially damaging the reputation of the LLM. Incorrect tokens modified by the attacker are marked in orange and watermarked tokens in blue. (b-c) In watermark removvll attacks, attackers effectively lower the z-score below the detection threshold while preserving a high sentence quality (low PPL. by exploiting either the (b) use of multiple keys or (c) publicly available watermark detection API.
removal
propose potential defenses as well as a set of general guidelines to better enhance the security of next-generation LLM watermarking systems. Overall, we make the following contributions:
We study how watermark robustness, despite being desirable property to mitigate removal at- tacks, can make the resulting systems highly susceptible to piggyback spoofing attacks, a simple type of attack that makes makes watermarked text toxic or inaccurate through small modifications, and show that challenges exist in detecting these attacks given that a single token can render an entire sentence inaccurate (Sec. 4).
We show that using multiple watermarking keys can make the system susceptible to watermark removal attacks (Sec. 5). Although a larger number of keys can help defend against watermark stealing attacks, which can be used to launch either spoofing or removal attacks, we show both theoretically and empirically that this in turn increases the potential for watermark removal attacks.
Finally, we identify that public watermark detection APIs can be exploited by attackers to launch both watermark-remon from differential privacy to effectively counteract spoofing attacks, showing that is possible to avoid the possibilities of noise reduction by applying pseudorandom noise based on the input.
and spoofing attacks (Sec. 6). We propose a defense using techniques
Throughout, we explore our attacks on three state-of sifee-offthhe tht-herrt watermarks [16, 45, 19] and two LLMs (LLAMA-2-7B [37] and OPT-1.3B demosstmmnss that these vulnerabilities are common to existing LLM watermarks, and providing caution for the field in deploying current solutions in practice without carefully considering the impact and trade-offs of watermarking design choices. Our code is available at https: //github// com comm/ii (Pang/LLM--- LLM- WaM-WWWteraarr.r Attacks.
2 Related Work
Advances in large language models (LLMMs) have given rise to increasing concerns that such models may be misused for purposes such spreading misinformation, phishing, and academic cheating.

---

In response, numerous recent works have proposed watermarking schemes as a tool for detecting LLM-generated text to mitigate potential misuse [16, 9, 19, 45, 17, 12, 41, 38]. These approaches involve embedding invisible watermarks into the model-generated content, which can then be ex- tracted and verified using a secret watermark key. Existing watermarking schemes share a few natural goals: (1) the watermark should be robust in that it cannot be easily removed; (2) the water- mark should not be easily stolen, thus enabling spoofing or removal attacks; and (3) the presence of a watermark should be easy to detect when given new candidate text. Unfortunately, we show that existing methods that aim to achieve these goals can in turn enable simple but effective attacks.
Removal attacks. Several recent works have highlighted that paraphrasing methods may be used to evade the detection of Al-generated text [18. 13, 20, 21, 43], with [18, 43] demonstrating effec- tive watermark removal using a local LLM. These methods usually require additional training for sentence paraphrasing which can impact sentence quality, or assume a high-quality oracle model to guarantee the output quality is preserved. In contrast, the simple and scalable removal attacks herein do not require additional training or a high-quality oracle. Additionally, our work differs in that we aim to directly connect and study how the inherent properties and design choices of watermarking schemes (such as the use of multiple keys and detection APIs) can inform such removal attacks.
Spoofing attacks. Prior works on spoofing use watermark stealing attacks to first estimate the wa- termark pattern and then embed it into an arbitrary content to launch spoofing attacks. These attacks usually require the attacker to pay large startup cost by obtaining a significant number of water- marked tokens. For example, [34] requires I million queries to the watermarked LLM, and [14, 10] assume the attacker can obtain millions of watermarked tokens to estimate their distribution. Unlike these works, we explore spoofing attacks that are less flexible but can be launched with significantly less upfront cost. In Sec. 4, we explore a very simple and scalable form of spoofing exploiting the inherent robustness property of watermarks, which we refer to as a 'piggyback spoofing attack', In Sec. 6, we then explore more general spoofing attacks, which instead of querying the watermarked LLM numerous times, consider exploiting the public detection API. In both, our attacks do not re- quire the attacker to estimate the watermark pattern, but share a similar ultimate goal with the prior spoofing attacks to create falsified inaccurate or toxic content that appears to be watermarked.
3 Preliminaries
Before exploring attacks and defenses on watermarking systems, we introduce relevant background on LLMs, notation we use throughout the work, and a set of concrete threat models.
Notation.
We use x to denote a sequence of tokens, X; E V is the i-th token in the sequence, and V is the vocabulary. Marig denotes the original model without a watermark, Mym is the watermarked model, and sk E S is the watermark secret key sampled from the key space S.
Language Models. Current state-of-the-arr (SOTA) LLMs are auto-regressive models, which pre- dict the next token based on the prior tokens. We define language models more formally below: Definition 1 (LM). We define language model (LM) without a watermark as:
Morig V V,
where the input is a sequence of length t tokens x. Mori(x)) (x) first returns the probability distribution for the next token x and then the LM samples x1+1 from this distribution.
Watermarks for LLMs. this work, we focus on three SOTA decoding- watermarking schemes: KGW [16], Unigram [45] and Exp [19]. Informally, decoding decoding-based watermarks are embedded by perturbing the output distribution of the original LLM. The perturbation is determined by secret watermark keys held by the LLM owner. Formally, we define the watermarking scheme:
Definition (Watermarked LLMs). The watermarked LLM takes token sequence x E V and secret key sk E S as input, and ouiputs a perturbed probability distribution for the next token. The perturbation is determined by sk: Mum V x +
(2)
The watermark detection outputs the statistical testing score for the null hypothesis that the input token sequence is independent of the watermark secret key, which reflects the watermark confidence:
fdetection V x
(3)
Please refer to Appendix C for additional details of the specific watermarks explored in this work.
3

---

3.1 Threat Model
Attacker's Objective. We study two types of attacks wasss-wwttrarrr-aeeeee removal attacks and (piggyback or general) spoofing attacks. In the quality response from the LLM without an embedded watermark. For the spoofing attacks, the goal is to generate a harmful or incorrect output that has the victim organization' 's watermark embedded. We present concrete application scenarios for attacker's motivations in Appendix B.
attack, the attacker aims to generate a high-
Attacker's Capabilities, We study attacks by exploiting three common design choices in water- marks: 1) robustness, 2) the use of multiple keys, and 3) public detection APIs. Each attack requires the adversary to have different capabilities, but we make assumptions that are practical and easy to achieve in real-world deployment scenarios.
1) For piggyback spoofing attacks exploiting robustness (Sec. 4), we assume that the attacker can make (1) queries to the target watermarked LLM. We also assume that the attacker edit the generated sentence (e.g., insert or substitute tokens).
2) For watermark-removal attacks exploiting the use of multiple keys (Sec. 5), we consider the sce- nario where multiple watermark keys are utilized to embed the watermark, which is a common practice in designing robust cryptographic protocols and is suggested by SOTA watermarks [19, 16] to improve resistance against galermark-stealing attacks [14, 10, 34]. For sentence of length I, we assume that the attacker can make (l) queries to the watermarked LLM.
3) For the attacks on detection APIs (Sec. 6), we assume that the detection API is available to normal users and the attacker can make (l)) queries for a sentence of length I. The detection returns the watermark confidence score (p-value or z-score). For spoofing attacks exploiting the detection APIs, we assume that the attacker can auto-regressively synthesize (toxic) sentences. For example, they can run a local (smalf) model to synthesize such sentences. For watermark-removal attacks exploiting the detection APIs, we also assume that the attacker can make (1) queries to the watermarked LLM. As common practice [25, 31] and also enabled by OpenAI's API [26], we assume that the top 5 tokens at each position and their probabilities are returned to the attackers.
4 Attacking Robust Watermarks
The goal of developing watermark that is robust to output perturbations is to defend against wa- termark removal, which may be used to circumvent detection schemes for applications such as phishing or fake news generation. Robust watermark designs have been the topic of many recent works [45, 16, 19, 34, 17, 32]. We formally define watermark robustness in the following definition.
Definition 3 (Watermark robustness). A watermark is (e, (e,8) --rbust,,, given a watermarked text x, if for all its neighboring texts within the e editing distance, the probability that the detection fails to detect the edited text is bounded by 8, given the detection confidence threshold T:
x E V", Pr Pr[feeeeciin( (x', sk) < T) < 6, s.t. fderection (x, sk) 2 T, d(xx,x') Sc
More robust watermarks can better defend against editing attacks, but this seemingly desirable prop- erty can also be easily misused by malicious users to launch simple piggyback spoofing attacks- e.g., a small portion of toxic or incorrect content can be inserted into the watermarked material, making it seem like it was generated by a specific watermarked LLM. The toxic content will still be detected watermarked, potentially damaging the reputation of the LLM service provider. As discussed in Sec. 2, spoofing attacks explored in prior work usually require the attacker to obtain millions of watermarked tokens upfront to estimate the watermark pattern [14, 34, 10]. In contrast, our simple piggyback spoofing only requires a single query to the matermarked L.LM with careful text modifications, and the effectiveness relates directly to the robustness of the LLM watermark.
Attack Procedure. (i) The attacker queries the target watermarked LLM to receive a high-entropy watermarked sentence Xwm. (ii) The attacker edits Xwm and forms new piece of text x' and claims that x' is generated by the target LLM. The editing method can be defined by the attacker. Simple strategies could include inserting toxic tokens into the watermarked sentence Xwm at random posi- tions, or editing specific tokens to make the output inaccurate (see example in Table 1). As we show, editing can also be done scale by querying another LLM like GPT4 to generate fluent output.
We present the formal analysis on the attack feasibility in Appendix D and point out the takeaway that is universally applicable to all robust watermarks: A more robust watermark makes piggyback

---

spoofing attack easier by allowing more toxic tokens to be inserted. This is a fundamental design trade-off: If watermark robust, such spoofing attacks are inevitable and may be extremely diffi- cult to detect, as even one toxic token can render the entire content harmful or inaccurate.
4.1 Evaluation
Experiment Setup. We assess the effectiveness of our piggyback spoofing attack by using the two editing strategies discussed above. Through toxic token insertion, we study the limits of how many tokens can inserted into the watermarked content. Using fluent inaccurate editing, we show that piggyback spoofing can generate fluent, watermarked, but inaccurate results at scale. Specifically, for the toxic token insertion, we generate a list of 200 toxic tokens and insert them at random positions in the watermarked output. For the fluent inaccurate editing, edit the watermarked sentence by querying GPT4 using the prompt "Modify less than words in the following sentence and make it inaccurate have opposite meanings." " Unless otherwise specified, in the evaluations of this work, we utilize 500 prompts data from OpenGen [18] dataset, and query the watermarked language models (LLAMA-2-77) [37] and OPT-1.3B [44]) generate the watermarked outputs. We evaluate three SOTA watermarks including KGW [16], Unigram [45], and Exp [19], using the default watermarking hyperparameters. In our experiments, we default to a maximum of 200 new tokens for KGW and Unigram, and 70 for Exp, due to its complexity in the watermark detection. 70 is also the maximum number of tokens the authors of Exp evaluated in their paper [19].
Evaluation Result. We report the maximum portion of the inserted toxic tokens relative to the original watermarked sentence length on LLAMA-2-7B model in Fig. la. We also present the confidence of the OpenAl moderation model [29] in identifying the content violating their usage policy [30] due to the inserted toxic tokens Fig. la. Our findings show that we can insert a significant number of toxic tokens into content generated by all the robust watermark- ing schemes, with median portion higher than 20%, i.e., for 200-token sentence, the attacker can insert a median of 40 toxic to- kens into it. These toxic sentences are then identified as violating OpenAl policy rules with high confidence scores, whose median is higher than 0.8 for all the watermarking schemes we study. The av- erage confidence scores for content before attack are around 0.01. The empirical data on the maximum portion of inserted toxic to- kens aligns with our analysis in Appendix D. We further validate this analysis in Fig. of Appendix E, showing that attackers can insert nontrivial portions of toxic tokens into the watermarked text to launch piggyback spoofing attacks. Notably, the more robust the watermark is, the more tokens can effectively be inserted. We present the results on OPT-1.3B Appendix G.
KIT LLASA 1.0
Tode TedeTebe Poran
LLAAAA
Contanaa
8.6
0.0
Tedi: Iekunc Fortee Mederaiee Coniiinne
(a) Toxic token insertion.
Perplcti
Fluent inaccurate editing.
Figure 1: Piggyback spoofing of robust watermarks.
(a) We can insert a large number of toxic tokens in robustly watermarked text without changing the water- mark detection result, resulting in text that likely to be identified as toxic. (h) We can use GPT4 to automatically modify water- marked text, making appear in- accurate while retaining fluency.
In Fig. lb, we report the PPL and watermark detection scores of the piggyback results on KGW and LLAMA-2-7B by the fluent inac- curate editing strategy. We show that we can successfully generate fluent results, with a slightly higher PPL. 94.17% of the piggybact back results have z-score higher than the default threshold 4. We ran- domly sample 100 piggyback results and manually check that most of them (92%) are fluent and have inaccurate or opposite content from the original watermarked content. See examples in Appendix F. The results show that we can generate watermarked, fluent, but inaccurate content at scale with an ASR higher than 90%.
4.2 Discussion
Guideline #1
Robust watermarks are inherently vulnerable to spoofing attacks and are not suitable as proof of content authenticity alone. To mitigate spoofing while preserving robustness, may be necessary to combine additional measures such as signature-based fragile watermarks.
Our results highlight that piggyback spoofing attacks are easy to execute in practice. LLM water- marks typically do not consider such attacks during design and deployment, and existing robust
5

---

watermarks are inherently vulnerable to such attacks. We highlight the contradiction between the watermark robustness and the piggyback spoofing feasibility. We consider this attack be chal- lenging to defend against, especially considering examples such as those in Table and Appendix F, where by only editing single token, the entire content becomes incorrect. It is hard, if not impos- sible, to detect whether a particular token is from the attacker by using robust watermark detection algorithms. Thus, practitioners should weigh the risks of removal vs. piggyback spoofing attacks for the model at hand. A feasible strategy to mitigate spoofing attacks is by requiring proof of digital signatures on the LLM generated content. However, while an attacker without access to the pri- vate key cannot spoof, it is worth nothing that this strategy is still vulnerable to watermark-removal attacks, as a single editing can invalidate the original signature.
Attacking
Watermarks
As discussed in Sec. 2, many works have explored the possibility of launching watermark stealing attacks to infer the secret pattern of the watermark, which can then enable spoofing and removal attacks [34, 14, 10]. A natural and effective defense against watermark stealing is using multiple watermark keys during embedding, which is a common practice in cryptography and also suggested by prior watermarks and work in watermark stealing [16, 19, 14]. Unfortunately, we demonstrate that using multiple keys can in turn introduce new watermark-
attacks.
In particular, SOTA watermarking schemes [16, 9,5, 19, 45, 17] aim to ensure the watermarked text retains its high quality and the private watermark patterns are not easily distinguished by maintaining an "unbiasedness" property:
(4) i.e. the expected distribution of watermarked output over the watermark key space sk S is close to the output distribution without a watermark, differing by a distance of e. We note that Exp [19] is "distortion free" for a single text sample, and KGW [16] and Unigram [45] slightly shift the water- marked distributions. We note that stealing attacks won't 't work on rigorously unbiased watermarks.
Eskes (Mmmsx, sk)) Re, (x)
The insight of our proposed titermarrk-emoove attack is that given the "unbiasedness" nature of wa- termarks and considering multiple keys may be used during watermark embedding, malicious users can estimate the output distribution without any watermark by querying the watermarked L.MM mul- tiple times using the same prompt. As this attack estimates the original, unwatermarked distribution, the quality of the generated content is preserved.
Attack Procedure. An attacker queries a watermarked model with an input x multiple times, ob- serving n subsequent tokens X,+1- This is easy for text completion model APIs, and chat model APIs can also be easily attacked by constructing a prompt to ask the chat model to complete a par- tial sentence without any prefix. The attacker then creates a frequency histogram of these tokens and samples according to the frequency. This sampled token matches the result of sampling on an unwatermarked output distribution with a nontrivial probability. Consequently, the attacker can pro- gressively eliminate watermarks while maintaining a high quality of the synthesized content. We present formal analysis of the number of required queries in Appendix H.
5.1 Evaluation
Experiment Setup. Our watermarks, models and datasets settings are the same as Sec. 4.1. We study the trade-off between resistance against watermark stealing and watermark-removal attacks by evaluating a recent watermark stealing attack [14]. In this attack, we query the watermarked LLM to obtain 2.2 million tokens in total to estimate the watermark pattern and then launch spoofing attacks using the estimated watermark pattern. We follow their assumptions that the attacker can access the unwatermarked tokens' distribution. In our watermark removal attack, consider that the attacker has observations with different keys. We evaluate the detection scores (2-score or p-value) and the output perplexity (PPL, evaluated using GPT3 [31]). The detection algorithm returns the maximum detection score across all the keys, which increases the expectation of unwatermarked detection results. Thus, set the detection thresholds for different keys to keep the false positive rates (FPR) below 1e-3 and report the attack success rates (ASR). We use default watermark hyperparameters.
Evaluation Result. As shown in Fig. 2a, using multiple keys can effectively defend against wa- termark stealing attacks. With a single key, the ASR 91 91,, which matches the results reported in [14]. We observe that using three keys can effectively reduce the ASR to 13%, and using more

---

Thrrdhol with FTR Io-1
10
Thmeseld with FUR 1o-3
Atack
Attack
10
Poplexay (PPL)
Z-Scome
0x
ASR
2 Scure
0.3g
ASR
55
0,5
0.6
0.2
0.4
w/o
W00 m/
nec1
a/0
m=13
gol7
(a) Z-Soore and attack success rate (ASR) wa- (b) Z-Score attack success rate (ASR) (c) Perplexity (PPL) of watermark- termark stealing [14].
watermark-
removal
Figure 2: Spoofing attack based on watermark stealing [14] and watermark-remoowe attacks on KGW wa- termark and LLAMA-2-7B model with different number of watermark keys n. Higher z-score reflects more confidence in watermarking and lower perplexity indicates better sentence quality. The attack success rates based the threshold with FPR@ le-3.
than 7 keys, the ASR of the watermark stealing close to zero. However, using more keys also makes the system vulnerable to our watermark-remova attacks shown in Fig. 2b. When more than keys, the detection scores of the content produced by our watermark removal attacks closely resemble those of unwatermarked content and are much lower than the detection thresholds, with ASRs higher than 97%. Fig. 2c suggests that using more keys improves the quality of the output content. This because, with greater number of keys, there higher probability for an attacker to accurately estimate the unwatermarked distribution, which is consistent with our analysis in Ap- pendix H. We observe that in practice, 7 keys suffice produce high-quality content comparable to the unwatermarked content. These observations remain consistent across various watermarking schemes and models; for additional results see Appendix J. We note that the numbers are not exactly the same as [14], as we consider a more realistic attacker with less queries to the watermarked LLM.
5.2 Discussion
Guideline #2
Using a larger number of watermarking keys can defend against watermark stealing attacks, but increases vulnerability to watermark-removal attacks. Limiting users' query rates can help to mitigate both attacks.
Many prior works have suggested using multiple keys to defend against watermark stealing attacks. However, in this study, we reveal that a conflict exists between improving resistance watermark stealing and the feasibility of removing watermarks. Our evaluation results show that finding "sweet spot" in terms the number of keys to use to mitigate both the watermark stealing and the watermark-removal attacks is not trivial. For example, high ASR of 36.2% just using three keys, and the corresponding watermark stealing- based spoof- ing's ASR is 13.0%. Using more keys can decrease the watermark stealing based spoofing's ASR, but at the cost of making the system more vulnerable to watermark removal and vice-versa. We note that the ASRs with three keys not negligible, thus limiting the ability of potentially malicious users is necessary in practice to mitigate these attacks. As a practical defense, we evaluate watermark stealing with various query limits on the watermarked LLM,, and found that the ASR can be signif- icantly reduced by limiting the attacker's 's query rate. Detailed results can be found in Appendix J. Given the trade-off that exists, we suggest that LLM service providers consider "defense-in-depth" techniques such as anomaly detection, query rate limiting, and user identification verification.
watermark-removal attack achieves
6 Attacking Watermark Detection APIs
It is still open question whether watermark detection APIs should be made publicly available to users. Although this makes it casier to detect watermarked text, it is a commonly acknowledged that it will make the system vulnerable to attacks [1] given the existence of oracle attacks [3, 6, 22, 15]. Here, we study this statement more precisely by examining the specific risk trade-offs that exist, well introducing novel defense that may make the public detection API more feasible in practice. In the following sections, we first introduce attacks that exploit the APIs and then propose suggestions and defenses to mitigate these attacks.
6.1 Attack Procedures
Watermark-Remoraa Attack. For the watermark-removal attack, we consider an attacker who has access to the target watermarked LLM's API, and can query the watermark detection results. The at-
7

---

wht watemorkk
wakmmuk NsT sxermmrkk
oniginal specting sttok
0.8
15.0
Strpshold
08
10
fhredield
PPL 7Ahre
10.0
0.4
0.400
5.0
2.5
0.0
0.0
KCWW
Unignan
KCB
Linggnnn
KGow
Lnggras
(a) Z. S.S Value of win removal.
(b) Perplexity of wmn removal.
(c) Z. Score/P Value of spoofing.
Figure 3: Attacks exploiting detection APIs on LLAMA-2-78 model.
tacker feeds a prompt into the watermarked L.M,, which generates the response in an auto-regressive manner, similar how LLMs generate sentences. That is, the attacker will select each token based on the prior tokens and the detection results. For the token x; the attacker will generate a list of pos- sible replacements for x, This list can be generated by querying the watermarked LLM, querying a local model, or simply returned by the watermarked LLM. In this work, we choose the third approach because of its simplicity and guarantee of synthesized sentences' quality. This is common assump- tion made by prior works [25], and such an API is also provided by OpenAI (top logprobs 5), which can benefit the normal users in understanding the model confidence, debugging and analyzing the model's 's behavior, customizing sampling strategies, etc. Consider that the top L = 5 tokens and their probabilities are returned to the attackers. The probability that the attacker can find an unwa- termarked token in the token candidates' list of length L is 1 --² for KGW and Unigram, which becomes sufficiently large given L = and y = 0.5. The attacker will query the detection using these replacements and sample token based on their probabilities and detection scores remove the watermark while preserving a high output quality.
Spoofing Attack. Spoofing attacks follow a similar procedure where the attacker can generate (harmful) content using a local model. When sampling the tokens, instead of selecting those that yield low confidence scores in removal attacks, the attacker will choose tokens that have higher confidence scores upon watermark detection queries. Thanks to the robustness of the LLM water- marks, attackers don't need ensure every single token carries a watermark; only that the overall detection confidence score surpasses the threshold, thereby treating synthesized content as gener- ated by the watermarked LLM.
6.2 Evaluation
Experiment Setup. We use the same evaluation setup as in Sec. 4.1 and Sec. 5.1. We evaluate the detection scores for both the watermark-removal and the spoofing attacks. We also report the number of queries to the detection API. Furthermore, for the watermark-removal attack, where the attackers care more about the output quality, we report the output PPL.. For spoofing attacks, the attackers' local models are LLAMA-2-7B 7B and OPT-1.3B.
Evaluation Result, As shown in Fig. 3a and Fig. 3b,
wm-removal
spooting
watermark-removal attacks exploiting the detection API significantly reduce detection confidence while maintaining high output quality. For instance, for the KGW watermark on LLAMA-2-7B model, we achieve median z-score of 1.43, which is much lower than the threshold 4. The PPL also close to the watermarked outputs 17 vs. 6.28). We observe that the Exp watermark has higher PPL than the other two watermarks. This is because that Exp watermark is deterministic, while other watermarks en- able random sampling during inference. Our attack also employs sampling based on the token probabilities and detection scores, thus we can improve the output quality for the Exp watermark.
ASR
Nqueries
ASR
liqueries
KGW
1.00
2.42
0.98
2.95
Unigram
0.96
2.66
0.98
2.96
Exp
0.96
1.55
0.85
2.89
Table The attack success rate (ASR), and the average query numbers per token for the sivermark-removee and spooting attacks exploit- ing the detection API on LLAMA-2-78 model.
The spoofing attacks also significantly boost the detection confidence even though the content not from the watermarked LLM, as depicted in Fig. 3c. We report the attack success rate (ASR) and the number of queries for both of the attacks Table The ASR quantifies how much of the generated content surpasses or falls short of the detection threshold. These attacks use a reasonable number of queries to the detection API and achieve high success rate, demonstrating practical feasibility. We observe consistent results on O.T-1.3B, please see Appendix K.
8

---

6.3 Defending Detection with Differential Privacy
In light of the issues above, we propose an effective defense using ideas from differential privacy (DP) [7] to counteract detection API based spoofing attacks. DP adds random noise to function results evaluated on private dataset such that the results from neighbouring datasets are indistin- guishable. Similarly, we consider adding Gaussian noise the distance score in the watermark detection, making the detection (e, 6D-DP [7], and ensuring that attackers cannot tell the difference between two queries by replacing single token in the content, thus increasing the hardness of launching the attacks. Considering an attacker can average multiple query results to reduce noise and estimate original scores without DP protection, we propose to calculate the noise based on the random seed generated by a pseudorandom function (PRF) with the sentence to be detected as the input. Specifically, seed PRF (x), where sk is the secret key held by the detection service. The users without the secret key cannot reverse or reduce the noise in the detection score. Thus, we can successfully mitigate the noise reduction via averaging multiple query results without comprising on utility protection of the DP defense. In the following, we evaluate the utility of the DP defense and its performance in mitigating the spoofing attacks.
Experiment Setup. Firstly, we as- sess the utility of DP defense by eval- uating the accuracy of the detection under various noise scales. Next, we evaluate the efficacy of the spoof- ing against DP detection defense us- ing the same method as in Sec. 6.1. We select the optimal noise scale that provides best defense while keep- ing the drop in accuracy within 2%. We note that for KGW and Unigram watermarks, we add noise to the z- scores. Sensitivity varies with sen- h+1
1.0
thresheld
as
10.0
0.6
ASR Detection
Z-Sone
5,0
0.1
02
0.0
e-1 8-2
w/o
DP
(a) Spoofing ASR and detection ACC.
(b) Z-scores with/without DP.
Figure 4: Evaluation of DP detection on KGW watermark and LLAMA-2-7B model. (a). Spoofing attack success (ASR) and detection accuracy (ACC) without and with DP watermark de- tection under different noise parameters. (b), Z-scores of original text without attack, spoofing attack without DP, and spoofing at- tacks with DP. We use the best 4 from (a).
tence length (e.g., A Vイ(1-1)
for replacement editing, where is the sentence length, & is the context width of the watermark, and 7 is the portion of the tokens in green list). The actual noise scale proportional oA.
Evaluation Result. As shown in Fig. 4a, with a noise scale of o = 4, the DP detection's 's accuracy drops from the original 98. 28 to 97.2% KGW and LLAMA-2-7B, while the spoofing ASR becomes 0% using the same attack procedure as Sec. 6.1. The results are consistent for Unigram and Exp watermarks and OPT-1.3B model as shown in Appendix I., which illustrates that the DP defense has a great utility-defense trade-off, with a negligible accuracy drop and significantly mitigates the spoofing attacks.
6.4 Discussion
Guideline #3
Public detection APIs can enable both spoofing and removal attacks. To defend against these attacks, we propose DP-inspired defense, which combined with techniques such as anomaly detection, query rate limiting, and user identification verification can help to make public detection more feasible in practice.
The detection API, available to the public, aids users in differentiating between AI and human- created materials. However, it can be exploited by attackers to gradually remove watermarks or launch spoofing attacks. We propose defense utilizing the ideas in differential privacy, which sig- nificantly increases the difficulty for spoofing attacks. However, this method is less effective against watermark-remova: attacks that exploit the detection API because attackers' actions will be close to random sampling, which, even though with less success rates, remains an effective way of re- moving watermarks. Therefore, we leave developing a more powerful defense mechanism against natermark-remove attacks exploiting detection API as future work. Additionally, we note that the attacker may increase the sensitivity of the input sentences by substituting multiple tokens and infer whether these tokens are in the green list or not to launch the spoofing attack, but this will require much more queries to the detection API. We recommend companies providing detection services

---

adopt a -neeeth approach [2, 8]. For instance, they should detect and curb malicious be- havior by limiting query rates from potential attackers, and also verify the identity of the users to protect against Sybil attacks.
7 Discussion, Limitation & Future Work
Generalizability of our attacks. We focus on three SOTA PRF-based robust watermarks, which are a natural set to explore given their popularity and formal guarantees. There are other watermarks like the semantics-basee watermarks (23, 33). While attacking semantics-basee watermarks is outside the scope of our study, we deem this an interesting future direction to explore.
Recently, researchers have also proposed signature-based publicly detectable watermarks [9] to mit- igate the spoofing attacks by exploiting robustness. Unlike the watermarks we study, these water- marks usually have weaker robustness guarantees, which further highlights the trade-offs between robustness and vulnerability to spoofing attacks, as we have discussed in Sec. 4.
Our findings, such exploiting robustness properties and publicly available detection APIs, can also be generalized to image watermarks [39, 42]. The attackers must integrate domain- speciffc constraints to ensure that the generated sentences images are meaningful and high-quality. We deem studying the fundamental trade-offs for image watermarks a promising future direction.
Trade-offs of watermark context width. There are two effective strategies to mitigate the water- mark stealing attacks for the KGW watermark [16]: 1) using a larger context width and 2) using multiple watermark keys. In this work (Sec. 5), we primarily explore the fundamental trade-offs in using multiple watermark keys, which prior work has underexplored. Trade-offs in context widths were discussed in previous work [14, 16, 45J. Using a larger h increases resistance against water- mark stealing but reduces robustness. Recent work [14] shows successful watermark stealing even with h 4. Using multiple keys, as shown in Sec. 5 of our paper. mitigates stealing attacks but introduces new attack vectors of watermark removal. Our attacks will work under different choices of context width, demonstrate this point, we provide more experimental results in Appendix M.
exploit properties or design choices orthogonal to the context width. To
The influence of how detection proceeds with multiple keys. For the scenarios where multi- ple keys are used, we consider the detector using min/max aggregation to obtain the detection score. More robust aggregations exist including the Harmonic mean p-value [40]. We note that our watermark-
attack exploiting the use of multiple keys is not dependent on the aggregation method do not rely the server's watermark detection. However, the trade-off analysis and the sweet spot for the number of keys may slightly change given the different detection performance.
Changing to p-values in KGW and Unigram. P-values are used for Exp [19] watermark in our pa- per, and the observations are consistent with KGW [16] and Unigram [45]. We expect no impact on results from this change since p-values are monotonic to z-scores. To ease the figures' presentation, we adopt the statiissssss in the main paper, we present more results of using p-values Appendix M.
8 Conclusion
In this work, we reveal new attack vectors that exploit common features and design choices of LLM watermarks. In particular, while these design choices may enhance robustness, resistance against watermark stealing attacks, and public detection ease, they also allow malicious actors to launch attacks that can easily remove the watermark or damage the model's reputation. Based on the theoretical and empirical analysis of our attacks, we suggest guidelines for designing and deploying LLM watermarks along with possible defenses to establish more reliable LLM watermark systems.
Acknowledgements
This work was supported in part by the National Science Foundation grants IIS2145670, CCF2107024, CNS2326312 and funding from Amazon, Apple, Google, Intel, Meta, and the Cy- Lab Security and Privacy Institute. Any opinions, findings and conclusions or recommendations expressed in this material are those the author(s) and do not necessarily reflect the views of any of these funding agencies.
10

---

References
[1] Scott Aaronson.
Watermarking
large language models. 07- 2023.
[2] Mauro Barni, Pedro Comesaña-Alfann Fernando Pérez-González, and Benedetta Tondi. Are you threatening me?: Towards smart detectors in watermarking, In Media Watermarking, Security, and Forensics 2014, volume 9028, pages 51-62. SPIE, 2014.
[3] Daniel Bleichenbacher. Chosen ciphertext attacks against protocols based on the rsa encryp- tion standard pkcs# 1. In Advances in Cryptology-CRYPTO Cryptology Conference Santa Barbara, California, USA August 23-27, 1998 Proceedings 18, pages 1-1 Springer, 1998.
CRYPTO '98: 18th Annual International
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. arXiv preprint arXiv:2306.09/94 099900 2023.
[6] Ronald Cramer and Victor Shoup. Design and analysis of practical public-key encryption schemes secure against adaptive chosen ciphertext attack. SIAM Journal on Computing, 33(1): 167-226, 2003.
[7] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foun- dations and Trends@) Theoretical Computer Science, 9(3-4):211-4077 2014.
[8] Maha El Choubassi and Pierre Moulin. On reliability and security of randomized detectors against sensitivity analysis attacks. IEEE Transactions Information Forensics and Security, 433)2273-28,, 2009.
[9] Jaiden Fairoze, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, and Mingyuan Wang. Publicly detectable watermarking for language models. Cryptology ePrint Archive, 2023.
[10] Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of watermarks for language models. arXiv preprint arXiv:2312.04469 2023.
[11] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: series of lectures, volume 33. US Government Printing Office, 1948.
[12) Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased watermark for large language models. arXiv preprini arXiv:2310 10669, 2023.
[13] Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example gen- eration with syntactically controlled paraphrase networks. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 1875-1885, 2018
[14] Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models. arXiv preprint arXiv:2402. 0ete:2202..39.1. 2024.
[15] Ton Kalker, J-P Linnartz, and Marten van Dijk. Watermark estimation through detector anal- ysis. In Proceedings 1998 International Conference Image Processing. ICIP98 (Cat. No. 98CB36269), volume 1, pages 425-429. IEEE, 1998.
[16] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. watermark for large language models. In Proceedings of the 40th International Confer- ence Machine Leurning, volume 202 of Proceedings of Machine Learning Research, pages mlr. press/v202/
17061-17084. PMLR, 23-29 Jul 2023. URL. https://procesda kir kirchenbauer23a. html.
11

---

[17] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv22306.046.44 2023.
[18] Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Frederick Wieting, and Mohit lyyer. Paraphrasing evades detectors of Al-generated text, but retrieval is an effective defense. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL https: //openreview ptntt/forummddwWonntee
[19] Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion- free watermarks for language models. arXiv preprint arXiv:2307. 15593, 2023.
[20] Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. Paraphrase generation with deep rein- forcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865-3878, 2018.
[21] Zhe Lin, Yitao Cai, and Xiaojun Wan. Towards document-level paraphrase generation with sentence rewriting and reordering. In Findings of the Association for Computational Linguis- fiex: EMNLP 2021, pages 1033-1044, 2021.
[22] Jean Paul MG Linnartz and Marten Van Dijk. Analysis of the sensitivity attack against elec- tronic watermarks in images. In Information Hiding: Second International Workshop, IH '98 Portland, Oregon, USA, April 14-17, 1998 Proceedings 2, pages 258-272. Springer, 1998.
[23] Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. A semantic invariant robust watermark for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL /ettps://openree
[24] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: zero-shot machine- generated text detection using probability curvature. In Pro- ceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
machine-generated
[25] Ali Nasch, Kalpesh Krishna, Mohit Iyyer, and Amir Houmansadr. Stealing the decoding algo- rithms of language models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1835-1849, 2023.
[26] OpenAl. Openai api of returning top-k logprobs.
https://plafoomm.peenaaicco/c//s
réérrence/oompleioos/cceeee.
[27] OpenAI.
Chatgpt: Optimizing language models for dialogue. https///ppnai.ccm//// http://ppeaiiccmmm/ooo/hhhggggg 2022.
OpenAl blog,
[28] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303008774, 2023.
[29] OpenAl. Openai moderation endpoint. http:://attcooo.oerr..mooomoolet 2023.
[30] OpenAI. Openai usage policies. http:://pppaaaiicoommmoccccsssss 2023.
[31] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[32] Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. Mark my words: Analyzing and evaluating language model watermarks. arXiv preprint arXiv:2312.00273, 2023.
[33] Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. A robust semantics-base watermark for large language model against paraphrasing, In Findings of the Association for Computational Linguistics: NAACL 2024, pages 613-625, 2024.
[34] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai- ai-generated text be reliably detected?? arXiv preprint arXiv:2303. /56,, 2023.
12

---

[35] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan. Tim Salimans, et al. Photore- alistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36447 2022.
[36] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex New- house, Jason Blazakis, Kris McGuffie, and Jasmine Wang. Release strategies and the social impacts of language models, 2019.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.092.:0 2023.
[38] Lean Wang, Wenkai Yang, Deli Chen, Hao Zhou, Yankai Lin, Fandong Meng, Jie Zhou, and Xu Sun. Towards codable text watermarking for large language models. arXiv preprint arXiv:2307. 15992, 2023.
[39] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring water- marks: Fingerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20003, 2023.
[40] Daniel Wilson. The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences, 1 116(4): 1e:::11951110000 1200, 2019.
[41] Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. Dipmark: A stealthy, efficient and resilient watermark for large language models. arXiv preprint arXiv:2310.07710, 2023.
[42] Zijin Yang, Kai Zeng, Kejiang Chen, Han Fang, Weiming Zhang, and Nenghai Yu. Gaussian shading:
Provable performance-lossles image watermarking for diffusion models. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12162-12171, 2024.
[43] Hanlin Zhang, Benjamin Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks in the sand: Impossibility of strong watermarking for generative models. arXiv preprint arXiv:2311.0437, 2023.
[44] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained trans- former language models. arXiv preprint arXivv2205.01066 2022.
[45] Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for Al-generated text. In The Twelfth International Conference on Learning Representations, 2024. URL https://openttii /ttp::/v/eeeeeieeeeeemeeee nwt/forum? meet:forumiddSSmmiiiii id=SsmT8aD45..
13

---

A Broader Impacts
Our work studies the security implications of common LLM watermarking design choices. By de- veloping realistic attacks and defenses and a simple set of guidelines for watermarking in practice, we aim for the work to serve resource for the development of secure LLM watermarking sys- tems. Of course, by outlining such attacks, there is a risk that our work may in fact increase the prevalence of watermark removal or spoofing attacks performed in practice. We believe that this is nonetheless an important step towards educating the community about potential risks in watermark- ing systems and ultimately creating more effective defenses for secure LLM watermarking.
More generally, our work shows that a number of trade-offs exist in LLM watermarking (e.g.. be- tween utility, usability, robustness, and susceptibility removal or spoofing attacks). The guidelines we propose provide rough proposals for considering these trade-offs, but we note that how to best navigate each trade-off will depend on the application hand. Considering strategies best navi- gate this space for specific LLM watermarking applications is an important direction of future study.
B Attacker's Motivation
We present two practical scenarios motivate watermark- attacks: (i) A student jour- nalist uses high-quality watermarked LLMs to write articles, but wants to remove the watermark to claim originality. (äi) A malicious company offering LLM services for clients, instead of develop- ing their own LLMs, simply queries a watermarked LLM from a victim company and removes the watermark, potentially infringing upon IP rights of the victim company.
In piggyback and spoofing attacks, an attacker can damage the reputation of a victim company offering an LLM service. For example: (i) The attacker can news or incorrect facts and post them on social media. By claiming the material is generated by the LLM from the benign company, the attacker can damage the reputation of the company and their model. (ii) The attacker can use the spoofing attack to inject malicious code into some public software. The code has the benign company's 's watermark embedded, and the benign company may thus be at fault and have to bear responsibility for the actions.
spoofing attack to generate fake
C Watermarking Schemes & Hyper-Parameters
In this section, we introduce the three watermarking schemes evaluate in the paper- Unigram [45], and Exp [19]. We also introduce the perplexity, a metric to evaluate the sentence quality.
Kaper-KGW KGW [16],
KGW. the KGW watermarking scheme, when generating the current token Xc+1, all the tokens in the vocabulary is pseudorandomly shuffled and split into two lists-thee the green list and the red list. The random seed used to determine the green and red lists computed by a watermark secret key sk and the prior h tokens x,-h- ... ||x, using pseudorandom functions (PRFs):
SEED Fx
||xx))
where h is the context width of the watermark. We note that the choice of h has minor influence on our attacks or defenses, as our algorithms are not dependent on h. Here we use their original algorithm with 1. Then, the seed used split the vocabulary into the green and red lists of tokens, with y portion of tokens in the green list:
Lyreen- Lidd Shuffle(V, SEED,")
Then, KGW generates a binary watermark mask vector for the current token prediction, which has the same size as the vocabulary. All the tokens the green list Lyreen have value the mask, and all the tokens in the red list have value 0 in the mask:
MASK GenerateMask( (ggrems Lred)
To embed the watermark, KGW add constant to the logits of the LLM's prediction for token X,+1;
WATERMARKEDPRRO Softmax (logits + 8 x MASK),
where the logits is from the L.M,, and the 5 the watermark strength. Then the LLM will sample the token x,41 according to the watermarked probability distribution.
14

---

The detection involves computing the z-score:
9 76
11(1-1
where g is the number of tokens the green list, 7 is the total number of tokens in the input token sequence, and is the portion of the vocabulary tokens in the green list. Similar the watermark embedding, the green and red lists for each token position are determined by watermark secret key and the token prior to the current token in the input token sequence.
Unigram. Similar to KGW, Unigram also splits the vocabulary into green and red lists and prioritize the tokens in the green list by adding constant to the logits before computing the softmax. The difference that Unigram uses global red and green lists instead of computing the green and red lists for each token. That is, the seed to shuffle the list is only determined by the watermark secret key and generated by
Generator (PRG):
SEED G(sk)
Then, similar to KGW, the seed is used to split the vocabulary into the green and red lists of tokens, with 7 portion of tokens in the green list:
Lereens Leed Shuffle(V, SEED,, ?)
The watermark embedding and detection procedures are the same as KGW: Unigram first compute the watermark mask:
MASK = GenerateMask( (Tygree, Tred)
And then embed the watermark by perturbing the logits of the LLM outputs:
WATERMARKEDPROB
Dofttmax(logitte + 8 x MASK),
where the logits is from the LLM, and the 5 is the watermark strength. Then the LLM will sample the token X according to the watermarked probability distribution.
The detection also computes the z-score:
71
V1(1-
where g is the number of tokens in the green list, / is the total number of tokens in the input token sequence, and is the portion of the vocabulary tokens in the green list. According to the analysis in [45] and also consistent with our results in Sec, 4.1, by decoupling the green and red lists splitting with the prior tokens, Unigram is twice as robust as KGW. But it's more likely to leak the pattern of the watermarked tokens given that it uses a global green-red list splitting.
Exp. The Exp watermarking scheme from [19] is an extension [1]. Instead of using a single key in KGW and Unigram, the usage of multiple watermark keys is inherent in Exp provide the distortion-free guarantee. Each key a vector of size [V] with values uniformly distributed in [0, 1].
That is, sk E2, ......... En. where & E [0, 10M,, le E [m], and n is the length of the watermark keys, default to 256.
For the prediction of the token x,+1, Exp firstly collects the output probability vector p E [0, 1 from the LLM. A random shift r - [n] is sampled at the beginning of receiving the prompt. Then the token x is sampled using the Gumbel trick [11]:
X arg max; 1/P.
where 1: = + ++ + 1 mod n, i.e., each position uses a different watermark key which determines the uniform distribution sampling used in the Gumbel trick sampling. This method guarantees that the output distribution distortion-free, whose expectation is identical to the distribution without watermark given sufficiently large
The watermark detection also computes test statistics. The basic test statistics is:
log(1 ER
15

---

where k = t mod n. And Exp computes the minimum Levenshtein distance using the basic test statistic as cost (see Sec. 2.4 in [19]).
Instead of using single keys as KGW and Unigram, Exp uses multiple keys and incorporates Gumbel trick rigorously provide distortion-free (unbiased) guarantee, whose expected output distribution over the key space is identical to the unwatermarked distribution.
Sentence Quality. Perplexity (PPL) is one of the most common metrics for evaluating language models. It can also be utilized to measure the quality of the sentences [45, 16] based on the oracle of high-quality language models. Formally, PPL returns the following quality score for an input sentence x:
PPL(x) = expl Σ (o .. ".
(5)
1=1
In our evaluation, we utilize the GPT3 [31] as the oracle model to evaluate sentence quality.
Setups and Hyper-Parameterr For KGW [16] and Unigram [45] watermarks, utilize the de- fault parameters in [45], where the watermark strength is 5 = 2, and the green list portion is 7 = 0.5. We employ a threshold of T
for these two watermarks with single watermark key. For the scenarios where multiple keys are used, we calculate the thresholds to guarantee that the false posi- tive rates (FPRs) are below 1e-3, For the Exp watermark (refered to as Exp-edit in [19]), we use the default parameters, where the watermark key length is n = 256 and the block size k is default to be identical the token length. We set the p- p-value threshold for Exp to 0.05 in our experiments.
For the spoofing attack exploiting detection APIs, we obtain the first three tokens with the highest probabilities from the unwatermarked LLM, and for the removal attack exploiting detection APIs, we obtain the first five tokens with the highest probabilities from the watermarked LLM. For water- mark removal attacks exploiting detection APIs on KGW and Unigram, we increase the probabilities of the tokens that have the smallest detection confidence, and then sample from the modified prob- ability distribution. For watermark removal attacks exploiting detection APIS on Exp, we simply sample the token that has the maximum p-value, but we will skip the tokens that have low proba- bilities (lower than 0.15) when the detection p-value is high (higher than 0.1). The different setup for the Exp watermark is required to ensure that we can produce high-quality sentences. For water- mark spoofing attacks that exploit detection APIs, we sample the token that has the highest detection confidence for KGW, Unigram, and Exp watermarks.
We conduct the experiments cluster with 8 NVIDIA A100 GPUs, AMD EPYC 7763 64-Core CPU, and ITB memory.
D Attack Feasibility Analysis of Piggyback Spoofing Exploiting Robustness
We study the bound on the maximum number of tokens that are allowed to be inserted or edited in a watermarked sentence, and we present the following theorem on Unigram watermark [45] due to its clean robustness guarantee:
Theorem 1 (Maximum insertion portion). Consider a watermarked token sequence x of length The Unigram watermark z-score threshold is T, the portion of the tokens in the green list is 7,, the detection 2-score of x is z, and the number of inserted tokens Then, to guarantee the expected z-score of the edited text is greater than T, it suffices to guarantee 5 72
Proof. Recall that the watermarking schemes' detections usually involve computing the statistical testing. Unigram splits the vocabulary into two lists- the green list and the red list. It prioritizes the tokens in the green list during watermark embedding, and the detection computes the z-score:
-- 76
/1(1-~)
where g is the number of tokens in the green list, / is the total number of tokens in the input token sequence, and y is the portion of the vocabulary tokens in the green list. Let the number of the inserted toxic tokens be s. Since toxic tokens are independent of the secret key sk, the expected new z-score z' is:
E(z') = 9 1a(1 - 71+1-))(+s)) =V VTs''
16

---

To guarantee that [2²)) 2 T, it suffices to guarantee
2² T
T
Different from the analysis in the Unigram paper on how the z-score changes given specific num- ber of edits, we have a tight bound on the maximum possible number of edits, which also more straightforward for the attack feasibility analysis. According to Theorem I, long the number of toxic tokens inserted is bounded by 1 the attacker can execute a piggyback attack to generate toxic content with the target watermark embedded. The editing distance bound (Def. 3) for sen-
tence is E 1² A stronger watermark makes piggyback spoofing attacks easier by allowing more toxic tokens to be inserted. This conclusion applies universally to all robust watermarking schemes. This is fundamental design trade-off: if watermark is robust, such spoofing attacks are inevitable and may be extremely difficult to detect, as even one toxic token can render the entire content harmful or inaccurate.
E
Validation of Theorem 1
In this section, we validate Theorem 1 by using watermarked texts of varying lengths 1 and z- scores study the relationship between & and z--²" of Unigram watermark. The results shown in Fig. 5. As anticipated, 85.78% of the maximum allowable tokens to be inserted into the watermarked content satisfy Theorem Given that this equation analyzes expected s/l, small portion of outliers is reasonable. We primarily visualize this result for Unigram due to its clean robustness guarantee. Other watermarks can also reach similar conclusions, but their bounds on are either complex [16] or lack closed form [19], making them difficult to visualize. Our empirical findings in Fig. sufficiently prove an attacker can insert nontrivial portions of toxic incorrect tokens into the watermarked text to launch the spoofing attack, which can be generalized across all robust watermarking schemes.
2.5
2.0
14.2566
1.5
1.0
0.5
Figure The relationship between s/t and z. The data points are evaluated on Unigram using LLAMA-2-7B and 500 samples from OpenGen dataset.
F Piggyback Attack Examples
Here we present more piggyback attack results using the edition strategy by querying GPT4 using the prompt "Modify less than 3 words in the following sentence and make it inaccurate or have opposite meanings." " The attack is launched on KGW watermark and LLAMA-2-7B model.
Watermarked content, z-score: 4.93, PPL: 4.61
Earth has history of 4.5 billion years and humans have been around for 200,000 years. Yet humans have been using computers for just over 70 years and even then the technology, we are still just getting started. The first computer, ENIAC (Electronic Numerical Integrator And Calculator), was built at the University of Pennsylvania between 1943 and 1946. The ENIAC took up 1800 sq ft and had 18,000 vacuum tube and mechanical parts. The ENIAC was used for mathematical calculations, ballistics, and code breaking. The ENTAC was 1000 times faster than any other calculator of the time. The first computer
first used in 1945. In the age of
program was the 73, built by Konrad Zuse his house.
17

---

Piggyback attack, z-score: 4.36, PPL: 5.68
Earth has a history of 4.5 billion years and humans have been around for 200,000 years. Yet humans have been using computers for just over 700 years and even then the term was first used in 1445. In the age of technology, we are still just getting started. The first computer, ENIAC (Electronic Numerical Integrator And Calculator), was built at the University of Pennsylvania between 1943 and 1946. The ENTAC took up 1800 sq ft and had 18,000 vacuum tube and mechanical parts. The ENIAC was used for mathematical calculations, ballistics, and code breaking. The ENIAC was 1000 times slower than any other calculator of the time. The first computer to run a program was the Z3, built by Konrad Zuse at his house.
Watermarked content, z-score: 4.57, PPL: 3.51
Harry Potter is 2001 film adaptation of J.K. Rowling's best-selling novels. In the year 1991, Harry Potter is rescued from his aunt and uncle's home and taken to Hogwarts, a school for magic. During the course of his first year, Harry makes friends and learns about the evil Lord Voldemort and his attempts to kill him. The film was directed by Chris Columbus, and stars Daniel Radcliffe as Harry Potter, with Richard Harris, Maggie Smith, Robbie Coltrane, Alan Rickman, Emma Watson, Rupert Grint, Gary Oldman, and Julie Walters in supporting roles. In the first film, Harry Potter is taken to Hogwarts School of Witchcraft and Wizardry, where he makes friends and discovers that Lord Voldemort wants to kill him.
Piggyback attack, 7-score: 4.47, PPL:: 3.78
Harry Potter is 2001 film adaptation of J.K. Rowling's best-selling novels. In the year 2991, Harry Potter is rescued from his aunt and uncle's home and taken to Hogwarts, a school for magic. During the course of his first year, Harry makes friends and learns about the evil Lord Voldemort and his attempts kill him. The film was directed by Chris Columbus, and stars Daniel Radcliffe as Harry Potter, with Richard Harris, Maggie Smith, Robbie Coltrane, Alan Rickman, Emma Watson Rupert Grint, Gary Oldman, and Julie Walters in supporting roles. In the first film, Harry Potter is taken to Hogwarts School of Witchcraft and Wizardry, where he makes friends and discovers that Lord Voldemort wants to kill him.
Watermarked content, Z- z-score: 651 PPL: 4.29
Usain Bolt is a 6'6", 200-pound sprinter from the island of Jamaica. He's 's also arguably the greatest athlete to ever walk the earth. In era where many of us are glued to our phones, he has over 18 million followers on Instagram. The 31-year-old athlete known for his incredible speed and endurance. He's won eight Olympic gold medals, including three 2012 alone. He's also set world records both the 100-meter dash and 200-meter dash. Bolt is the only person to ever hold the world record in both events at the same time, which he did twice. He's also the only person to ever hold the world record in both events at the same Olympics. Bolt is known for his incredible speed and endurance.
Piggyback attack, z-score: 5.58, PPI.: 4.46
Usain Boll is a 6'6", 200-pound sprinter from the island of Jamaica. He's also arguably the greatest athlete ever walk the earth. In where many of us glued to our phones. he has over 18 million followers on Instagram. The 3l-year-old athlete known for his incredible speed and endurance. He's won three Olympic gold medals, including three in 2012 alone. He's also set world records in both the 100-meter dash and 200-meter dash. Bolt is the only person to ever hold the world record in both events at the same time, which he did twice. He's also the only person ever hold the world record in both events the same Olympics, Bolt is known for his incredible speed and endurance.
Watermarked content, z-score: 6.01, PPL: 6.68
The history of the modern airplane is 100 years old this month. And yet it's not been 100 years since the Wright Brothers' first flight. The first airplane flight took place Dec. 17, 1903. After three years of development, Orville and Wilbur Wright's first flight lasted only 12 seconds. But within decade, the first airliner flew. In 1924, the Ford Motor Co. flew the first commercial plane on the U.S. East Coast. In the next year, the company built 10- passenger airliner with passenger windows and seats and an aisle. The 10-seat plane was called the Model T, and Ford executives said it would have been better if the company made 10-passenger instead of plane.
18

---

Piggyback attack, Z-score: 5.03, PPL: 7.19
The history of the modern airplane 100 years old this month. And yet it's not been 100 years since the Wright Brothers' first flight. The first airplane flight took place on Dec. 17, 1903. After three years of development, Orville and Wilbur Wright's first flight lasted only 12 seconds. But within decade, the first airliner flew. In 1924, the Ford Motor Co.
flew the first commercial plane the U.S. East Coast. In the next year, company built 10-passenger airliner with passenger windows seats and an aisle. The 10-seat plane was called the Model T. and Ford executives said it would have been better if the company made 10- passenger car instead of plane.
G Additional Results of Piggyback Spoofing Attack
In Sec. 4, we present the piggyback spoofing attack using toxic token insertion strategy on LLAMA- 2-7B model. Here, we present the results on OPT-1.3B model, which are consistent with LLAMA- 2-7B model's results.
1.2
Arg. scren ahrak
1.0
KOW LLAMA
Toxic Tokkes Partion
LLAMA
0.8
0.6
0.4
0.0
1000
Toxic Tokens Portion
Muderation Confidence
Figure 6: Piggyback spoofing of robust watermarks with toxic token insertion strategy on OPT-1.3B.
In Sec. 4, we present the fluent inaccurate editing strategy by querying the GPT4 on KGW watermark and LLAMA-2-7B model. Here present more results of this strategy on all the three watermarks (KGW, Unigram, and Exp) and two models (LLAMA-2-7B and OPT-1.3B). The results are shown in Fig. 7, Fig. 8, and Fig. 9, which are consistent with our findings in Fig. 1, indicating that our piggyback spoofing attack can be generalized across various robust watermarks and models.
Bondald
$500
25on
Paplesky
2.5xxc
Peplank
Z.Somo
(a) LLAMA-2-7B model.
(b) OPT-1.3B model.
Figure 7: Fluent inaccurate editing strategy on KGW watermark and LLAMA-2-7B and OPT-1.3B models.
12.5
35
plaphock
60
-29
(a) I.L.MA---77 model.
OPT-1.3B model.
Figure Fluent inaccurate editing strategy on Unigram watermark and LLAMA-2 2-7B and O.T-1.3B models.
19

---

10
worvaaaaa eacomrtt Hlop
100
00
110
Roatee
Perpaay
(a) LLAMA 2-7B model.
(b) OPT-133B model.
Figure 9: Fluent inaccurate editing strategy on Exp watermark and I.AMM-2-77B and OPT-1.3B models.
H
Watermark Key Number Analysis for Sentrmark-Remmovee Attacks Exploiting the Use of Multiple Watermark Keys
Now we analyze the number of required queries under different keys to estimate the token with the highest probability without watermark. We have the following probability bound for KGW and Unigram with the corresponding proof, and present the bound for Exp in Appendix I.
Theorem (Probability bound of unwatermarked token estimation). Suppose there observa- tions under different keys, the portion of the green list in KGW or Unigram is n,. Then the probability that the most frequent token is the same as the original unwatermarked token
1n/2.
(1-)))²
>)= x p(k),
(6)
A=
where p(k) = -1-CEkI) m (" m)m-k-m)
the number of other tokens whose watermarked probability can exceed that of the highest unwatermarked token.
In a practical scenario where n = 13, = 0.5, and c 3, Theorem 2 suggests that the attacker has probability of 0.71 in finding the token with the highest unwatermarked probability. This implies that we can successfully remove watermarks from over 71% of tokens using small number of observations under different keys
13), yielding high-quality unwatermarked content.
Proof. Recall that KGW and Unigram randomly split the tokens in the vocabulary into the green list and the red list. We consider the greedy sampling, where the token with the highest (watermarked) probability is sampled. We have n independent observations under different watermark keys. For each key, the token x; with the highest unwatermarked probability is the green list y. As long as x; is the green list, the greedy sampling will always yield x; since the watermarks add the same constant all the tokens' loogits in the green list.
Thus, the probability that the most frequent token among these n observations is X; is at least:
|n/2]
n (1
k=
which is the probability that x; is in the green list for at least half of the n keys.
For another token Xj whose probability can exceed x;, x, is in the green list and is in the red list. Then if x; is the green list for keys, the probability that xj is the green list for at least keys among the other A: keys is:
m==
Consider have such tokens having potential to exceed x;. Then least of the tokens is in the green list for at least & keys among the other n & keys is:
n k.
nm (1 - 7)" kk
m
m==
20

---

Thus, with all the above analysis. we have that if there are tokens that have the potential to exceed the probability of the token with highest unwatermarked probability (i.e., xj), the probability that the most frequent token among the n observations is the same as is:
1n/2|
(1
1
Σ K=
Th (1
x
n k
1m -7) --)
k
m
1/ 0
which concludes the proof.
Here we consider that the watermarked LLM utilizing greedy sampling. In practice, the greedy sampling might not be an optimal sampling strategy, but we note that it is extremely challenging to incorporate the multinomial sampling when analyzing the KGW and Unigram watermarks. Because KGW and Unigram add bias to the output logits, which will go through the softmax function to calculate the probabilities for the tokens. Given the softmax function not unbiased, we cannot get a tight bound on its variance. Thus, we leave this part as future direction to further incorporate multinomial sampling the analysis. Nevertheless, our empirical results still show that the attackers can generate high-quality unwatermarked content when multinomial sampling is used. Also, our analysis on Exp watermark Appendix can naturally incorporate multinomial sampling.
I Probability Bound of Unwatermarked Token Estimation for Exp
In this section, we present and prove the probability bound of unwatermarked token estimation for the Exp watermark [19].
Theorem 3 (Probability bound of unwatermarked token estimation for Exp). Suppose there are n observations under different keys, the highest probability for the unwatermarked tokens is p. Then the probability that the most frequently appeared token among the n observations is the same as the original unwatermarked token with highest probability is:
[n/2]
D k
p)
K=
Proof. The proof of Theorem is straightforward. As we have introduced in Appendix C, the Exp watermark employs the Gumbel trick sampling [11] when embedding the watermark. Thus, the probability that we observe the token whose original unwatermarked probability is is exactly p for each of the independent keys. Thus, if we make observations under different keys, then least half of them yields the token with the highest original probability p is:
[n/2]
$ (1 - -p) p)n-k
R=U
which concludes the proof.
Additional Results of Watermark- Attacks Exploiting the use of Multiple Watermark Keys
In this section, we provide more evaluation results of the watermark stealing [14] and our watermark- removal attacks exploiting the use of multiple watermark keys (see Sec. 5) on all the three water- marks (KGW, Unigram, and Exp) and two models (LLAMA-2-7B and OPT-1.3B). The results are OPT-1.3B model and
shown in Fig. 11, Fig. 12, Fig. 13, Fig, 14, Fig. 15. For KGW watermark Unigram watermark on LLAMA-22 2-BB and OPT-1.3B models, we have consistent observations with LLAMA-2-7B as present in Sec. 5.1, demonstrating the effectiveness
the KGW watermark and generalizability of our attacks. For the Exp watermark, our results in Fig. 12 and Fig. 15 also show that the watermark can be easily removed using multiple queries to estimate the distribution of the unwatermarked tokens.
The results of the watermark stealing [14] on Unigram watermark and OPT-1.3B model are also consistent with our observations See. 5. Using more keys can effectively mitigate the water- mark stealing; however, it will make the system more vulnerable to our watermark removal attacks.
21

---

Throughout these experiments, we observe that using three keys is the optimal choice to defend against both attacks. However, the attack success rates with three keys are not negligible. Thus, consistent with our guidelines in Sec. 5, we highly recommend that the LLM service provider to simultaneously limit the ability of the potentially malicious users.
To further verify that the LLM service provider can mitigate the watermark stealing attacks by limiting the attacker's query rates, we present the stealing attack results with various numbers of queries on the KGW watermark and LLAMA-2-7B model using three keys in Fig. 10. The results show that by limiting the query rates of the attacker, the attack success rate of the watermark stealing attack can be significantly decreased. Thus, we recommend that the LLM service provider follow diefense-in-deplee approach and utilize complementary techniques such as anomaly detection, query rate limiting, and user identification verification to mitigate stealing and removal attacks.
Threshuld with le-3
0.25
Attack
0.20
Z-Score
0.15
0.10
w/o wm
Q=0.7M Q=2.2M
Q=3.7M
Q=5.IM
Figure 10: Watermark stcaling attack [14] on KGW watermark and LLAMA-2-7B model using three keys with different numbers of attacker obtained tokens Q (in million). The attack success are based the threshold with FPR Ic-3.
We note that the watermark stealing attacks do not work on the Exp watermark [19]. as the use of large number of watermark keys inherent in their design, which defaults to 256. Thus, we omit the watermark stealing results on Exp, but we show that this watermark is inherently vulnerable to our watermark removal attack. From the results in Fig. 12 and Fig. 15, conclude that using = 13 queries, the resulting p-value is very close to that of the content without a watermark and is significantly different from the watermarked p-value, which shows that we can effectively remove the watermark using 13 queries for each token. We note that for Exp, the perplexity of the watermarked content significantly higher than that of the unwatermarked content. This mainly because Exp does not allow sampling in watermark embedding, which becomes deterministic algorithm when the key is fixed. In contrast, our watermark removal attack generates content with much lower perplexity, making it comparable to unwatermarked content when the query number under different keys exceeds 13. This can attributed to our attack functioning as a layer of random sampling. Unlike greedy sampling methods, we have probability to sample the token with the highest unwatermarked probability (see Sec. 4, Appendix H, and Appendix I). The results of the three watermarks and two models prove that the watermark-remov attack exploiting the use of multiple watermark keys can effectively eliminate the watermarks while maintaining high output quality.
Throshil FIFee
Throhold FFRce
12
10
Attock success
1.0
10
Attak success rate
1.0
10
7-Sorr
0.8
Z-Sonn
0.6
0.8
Puplexity
11.2
0.0
wlo 1100
1
3
13
1
wlo w1t0
w/
with
mx13
1=17
w/o 1) wm
ge3
go13
(a) Z-Scoro and attack success rate (ASR) of wa- (b) Z-Seore and attack success rate (ASR) of (c) Perplextty (PPL) termark stealing [25].
watermark-
waternark-removu
removal.
Figure 11: Spnofing attack hased watermark stealing [25] and watermark-remooall attacks Unigram watermark and LLAMA-2-77 model with different number watermark keys n.
22

---

0.8
40
peppesity
0.4
10
0.0
w/o w/
so17
wio wil yo3
(a) P-Value of watermark-removall attack.
(b) PPL of watermark-removal attack.
Figure 12: Watermark-removal Exp watermark [19] and LLAMA-2-7B model with multiple watermark keys.
Niick succeso
1.0
10
Amaak Success
Thessoll with PRR
1.0
15
7.Sore
0.8
Z-Seurc
0.8
Perplesity (PPL)
0.6
0.6
T
0.4
wil waX
gml
nm3
13
0.00
wlo
gm3
m313
n-17
0.0
wio Wat
ge3
no13
ne17
(a) Z Score and atlack success rale (ASR) of was (b) Z. Score und atlack success rale (ASR) of (c) Perplexity (PPL) termark stealing [25].
wattermark
watermark removallk-removvl. removal.
Figure Spoofing attack based on watermark stealing [25] watermark removal attacks on KGW water- mark and OPT-1-3B model with different number watermark keys n.
Theehood with FeR
Thenhold with FiSee
10
1.0
success gat
1.0
15
7.Sore
Z-Sore
10
0.8
Paploxity
10
0.6
0.4
0.2
0.4
win
ns.13
1
no13
n=11
ns17
(a) Z-Score and attack success rate (ASR) of (b) Z-Seore and attack success rate (ASR) of (c) termark stealing (25].
Perplexity (PPL) watermark
watermark removal.
removal.
Figure 14: Spoofing attack based on watermark stealing [25] and watermark-removee attacks on Unigram watermark and OPT-1.3B model with different number of watermark keys
1.0
8 0.6
Perloctr
0.2
20
w/o W1O
mc3
o27
s=13
gel7
W710
m33
847
ne13
ne17
P-Valu of watermark-removal attack.
(b) PPL watermark-removal attack.
Figure 15: Waternark-removal Exp watermark [19] and OPT-1.3B model with multiple watermark keys.
K Additional Results of Attacks Exploiting Detection APIs
We present the results of watermark-removal and spoofing attacks on OPT-1.3B model in Fig. 16 and Table The results are consistent with the I.LAMA-2-7B model presented in See. 6.1., with all the attack success rates higher than 75% using small number of queries to the detection API
23

---

of around per token. The results on OPT-1.3B model further demonstrate the effectiveness of our attacks exploiting the detection API.
w-o watmmak vatzmmekk
1.0
wht watemark
oegginal teat
1.0
15
w/ warkmmrkk
15:
spowniing srazk
wnteemmeoo dhredwild
7.eer
10
DLG
0.6
0.2
0.0
XCWW
Lingrazi
Esp
KOW
Unigaam
Exp
KGW
Unigram
Exp
(a) Z Score// Value of wmn removal.
(b) Perplexity of wm removal.
(c) (c) Scure/P Paauue Value of spaoling.
Figure 16: Attacks exploiting detection APIs on OPT-1.3B model.
wm-removal
spoofing
ASR
#queries
#queries
KGW
0.99
2.87
1.00
2.96
Unigram
0.77
3.25
1.00
2.97
Exp
0.86
2.07
0.93
2.92
Table 3: The attack success rate (ASR), and the average query numbers per token for the watermark-removal and spoofing attacks exploiting the detection API on OPT-1.3B model.
Additional Results of DP Defense
We present additional evaluation results of our defence technique that enhances the watermark de- tection by utilizing the techniques of differential privacy (see Sec. 6). Consistent with See. 6.3, we evaluate the utility of the DP defense as well as its performance in mitigating the spoofing attack exploiting the detection API. The results are shown Fig. 17, Fig. 18, Fig. 19, Fig. 20, Fig. 21.
We first identify the optimal noise scale parameter a based on its detection accuracy and attack success rate, aiming for drop detection accuracy within 2% and the lowest attack success rate. Then we assess the performance of the defense. Our findings across three watermarks and two models consistently demonstrate that we can significantly reduce the attack success rate around or below 20%.
Our defense can be generalized all LLM watermarking schemes. It allows us substantially mitigate spoofing attacks exploiting the detection API while having negligible impact on utility.
15.0
dtreshold
038
Accunrcy
ASR
Z.Sorre
0.4
Detection ACC
3.0
0.2
WP
0-1
2-2
0-1 8-8
WCD off
Wib DP
W:
(a) Detection ACC and spoofing ASR.
(b) Z-scores with/without DP.
Figure 17: Evaluation of DP watermark detection Unigram watermark and ILAMA-2-7B model. (a). Detection accuracy and spoofing attack success without and with DP watermark detection under different noise parameters. (b). Z-scores original text without attack, spoofing attack without DP, and spoofing attacks with DP. We use the best = from (a).
M Additional Results of Larger Context Width and Using P-Values
In this section, we present more results of our attacks on larger context width h the KGW watermark. The experiments are conducted LLAMA-2-7B model. To demonstrate that changing

---

1.0
0.8
Accurcy
0666
ASR
0.5
Detection ACC
0.4
0.2
2-2
W/O
wio
(a) Detection accuracy and spoofing allack
(b) P-values with/witout DP and under multiple queries.
success rate.
Figure 18: Evaluation DP watermark detection on Exp watermark and I.LAMA2-77B model. (a). Detection accuracy and spoofing attack success rate without and with DP watermark detection under different noise parameters. (b). Z-scores of original text without attack, spoofing attack without DP, and spoofing attacks with DP. We use the best from (a).
1.0
15.0
fhrehheld
0.6
2-Sere
0.4
Detection ACC
5.0
0.0
0.0
we DP
1=1
2-2
o= 8=
Who
Wio DP
wi DP
Detection accurncy and spoofing attack
(b) 7.scores with/withot DP and under multiple queries.
success rate.
Figure 19: Evaluation of DP watermark detection on KGW watermark and OPT-1.3B model. (a). Detection accuracy and spoofing attack success rate without and with DP watermark detection under different noise parameters. (b). Z-scores of original without attack, spoofing attack without DP, and spoofing attacks with We use the best a from (a).
1.0
15.0
Aveuracy
0.6
2.Scome
9.4
Detoction ACC
5.0
0.0
0.0
D2
DP
(a) Detection accurncy spoofing
(b) 7-scores with/without multiple queries.
success rate.
Figure 20: Evaluation of DP watermark detection on Unigram watermark and OPT-1.3B model. (a). Detection accuracy and spoofing attack success rate without and with DP watermark detection under different noise parameters. (b). Z-scores of original text without attack, spoofing attack without DP, and spoofing attacks with DP. We use the best a from (a).
p-values has minor effects the attack results, we adopt p-values here. We keep using 7-scores in the main paper to ease the presentation (c.g.. the p-values of the spoofing results will be too small visualize). The results are shown Fig. 22, Fig. 23, Fig. 24, We observe consistent results when using p-values in the KGW watermark and we expect minor influence on the results from this change since p-value is monotonic to z-score. We also get consistent observations for the scenarios of using larger context width h in KGW watermark. Again, using larger h enhances the resistance against watermark stealing but reduces robustness. Our experiments Fig. 22, Fig. 23, Fig. validate this. Fig. 22 shows that fewer edits are allowed for watermarked content with a larger h. indicating lower robustness. KGW recommends using h works we are aware of suggest using h > 4. Recent work [14] shows successful watermark stealing even with h Using multiple keys, as shown in Sec. of our paper, mitigates stealing attacks, but introduces new attack vectors of watermark removal.
in their codebase for robustness, and no prior
25

---

1.0
1,0
Alcuray
0.6
0.6
0.4
0.2
Detection ACC
W/
W/0 a11
wio wi
(a) Detection accurney and spoofing attack
(b) P-values with/withut DP and under multiple queries.
SU00e5s rate.
Figure Evaluation accuracy and spoofing attack success rate without and with DP watermark dctection under different noise parameters. (b), Z-scores of original text without attack, spoofing attack without DP, and spoofing attacks with DP. We the best from (a).
watermark detection on Exp watermark and OPT-1.3B model. (a). Detection
0.5
Ang stak TLAME
100
107
LLASSAbb
18
0.6
Maprrukk aracs
-00
10
10-2
0.0
Tooi
Coeridknes
(a) Toxic token insertion for KGW with
(b) Flaent inaccurate editing For KGW with Fluent insccurate editing For KGW with h 4.
and
Figure 22: Piggyback spoofing of KGW watermark LLAMA-2-7B model. We observe consistent results with Fig. (a) toxic token insertion works on both h 1 and h 4 with sumhash. KGW with 4 and sumhash less robust, thus we can insert fewer toxic tokens. (b, e) The performances of fluent inaccurate editing for h = and = 4 are close. Changing 7-statistics to p-value does not influence the attack performance.
Peppontt (IT.)
Pepfotte OPL)
6.500
010
540
(a) P-Value and ASR of wm-removal (b) PPL of win-temoval KGW h 1.
P-Value and ASR of wm-removal (d) PPL of KGW
KGW h
KGW h 4.
Figure 23: Watermark-removal attacks exploiting the of multiple keys KGW watermark (h and with sumhash) and LLAMA-2-7 2-BB model with different numbers of watermark keys n. The attack success rates are based on the threshold with FPR le-3, We observe consistent results for different context widths h, and changing the detection metric from z-statistics to p-value does not influence the attack performance. The results consistent with Fig. 2.
12
urigind
sponliig anack
10-2
7-Some
10-4
KOW
KGW
KGW
KCW b-t (c) Z. Score of spooling.
KOW and
(a) P-Vulue vemerelovll.
Perplexity wm-remooal.
Figure 24: Attacks exploiting detection APIs on KGW watermark with different h. (h = 1 and = with sumhash) and LLAMA-2-7B model. Results are consistent with Fig. 3. We report the z-score for spoofing attacks, as the v-vuuues of the spoofing results too small to visualize. Changing z-score p-value doesn' influence the attacks' performance and the results hold for different
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract we have briefly summarizee our work and in the introduction section, we have further summarized our contributions. All the takeaways and conclusions are consistent with our findings in the main paper and accurately reflect the paper's contri- butions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the three sections introducing our attacks (Sec. 4, Sec. 5, and Sec. 6). we have clearly mentioned the limitations of our attacks and defenses. For example, in Sec. 4, we mentioned that it is hard to completely mitigate our piggyback spooting attacks exploiting the robustness. And in Sec. and Sec. 6.3, we noted that it is challenging to achieve perfect tradeoff between the different risks. Thus, we recommend to follow "defense-in-depth" approach in practice such as query rate limiting and anomaly detection.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
27

---

judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have provided all the assumptions on the attacker in Sec. 3.1. We also present the corresponding proof the attack feasibility analysis in Appendix D, Ap- pendix H, and Appendix I.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided or not)?
Answer: [Yes]
Justification: We have provided the detailed the experimental setup in Sec. 4.1, Sec. 5.1, and Sec. 6. We've also open sourced our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- reasonable avenue for reproducibility, which may depend
missions to provide on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
28

---

(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We've open sourced our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipe
Beaii//guiddes/ooeSubbeee solliiiccsiiyo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: http:/uudeee//oooee
//nips. cc/
www.missssonoooooooooooo onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We have provided the experimental setups in Sec. 4.1, Sec. 5.1, and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer:
[No]
29

---

The randomness doesn't it play very important role in our attacks or defenses. And we observe consistent results across different watermarks and models. We have also tried to run the algorithms multiple times, and observed that the results are consistent. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mentioned our computing resources information in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Fthics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : hwwrrps...../// cc/
inne??
Answer: [Yes]
Justification:
We have reviewed and agreed on the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
30

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the positive as well as the potential negative impacts in
Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the code repositories of watermarks we use are open-sourced, and we have cited the corresponding papers in the experiment setup in Sec. 4.1.
Guidelines:
31

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Crowdsourcing research with human subjects are not involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurIPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
32

---

Justification:
Crowdsourcing and research with human subjects are not involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "SrI5J2ELCp", "reviewer": "Reviewer_YBsi", "summary": "This paper explores the vulnerabilities and trade-offs in watermarking schemes for large language models (LLMs). It highlights how common design choices in these schemes, aimed at ensuring robustness, multiple key usage, and public detection, make them susceptible to simple yet effective attacks. The authors demonstrate that robust watermarks, intended to prevent removal, can be easily exploited through piggyback spoofing attacks that insert toxic or inaccurate content while maintaining the watermark. Additionally, using multiple watermark keys to defend against watermark stealing inadvertently increases vulnerability to watermark removal attacks. Public detection APIs, while useful for verifying watermarked content, are shown to be exploitable for both removal and spoofing attacks. Through empirical evaluations on state-of-the-art watermarks (KGW, Unigram, Exp) and models (LLAMA-2-7B, OPT-1.3B), the study rigorously demonstrates these vulnerabilities and the resulting trade-offs between robustness, utility, and usability. The paper proposes potential defenses, including the use of differential privacy techniques in detection APIs, and offers guidelines for designing more secure watermarking systems. Ultimately, the study underscores the importance of carefully considering watermarking design choices to balance security and utility, calling for further research to develop robust defenses and optimize these trade-offs.", "questions": "+ How generalizable are the findings? Would the vulnerabilities and trade-offs identified apply to all types of LLMs, or are they specific to certain architectures or applications?\n+ What are the limitations of the attack methods presented in this study? Are there scenarios where these attacks might not be effective?\n+ Are there any practical considerations or potential drawbacks to implementing DP defense mechanisms in real-world systems?", "limitations": "See the weaknesses.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The paper rigorously explores various common watermarking design choices and demonstrates their susceptibility to simple yet effective attacks. It highlights the fundamental trade-offs between robustness, utility, and usability, which are crucial for understanding the limitations of current watermarking methods.\n\n+ The paper provides an insightful discussion of the inherent trade-offs in watermarking design, such as the balance between watermark robustness and vulnerability to spoofing attacks. This helps in understanding the complexities involved in creating effective watermarking.\n\n+ The authors propose potential defenses and guidelines to enhance the security of LLM watermarking systems. These recommendations are valuable for practitioners looking to deploy more secure watermarking solutions in practice.", "weaknesses": "+ The citation in Line 248 and Figure 2 is not correct. The authors are supposed to cite [1].\n\n+ Lack of discussions in the Publicly-Detectable Watermarking [2], which compromises robustness to defend against spoofing attacks.\n\n\n\n\n[1] Nikola Jovanovic, Robin Staab, and Martin Vechev. Watermark stealing in large language models. arXiv preprint arXiv:2402.19361, 2024.\n[2] Fairoze, Jaiden, et al. \"Publicly detectable watermarking for language models.\" arXiv preprint arXiv:2310.18491 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments. In the following, we respond to each question.\n\n---\n>**Q1: The citation in Line 248 and Figure 2 is not correct. The authors are supposed to cite [1].**\n\n**A1**: Thanks for pointing this out. We will fix this typo in the revision to avoid confusions.\n\n---\n>**Q2: Lack of discussions in the Publicly-Detectable Watermarking [2], which compromises robustness to defend against spoofing attacks.**\n\n**A2**: There exist some recent works [1,2] that study mitigating the spoofing attack vulnerabilities in robust watermarks. The high-level idea is to embed a cryptographic signature into the subsequent tokens, and the signatures are computed using the first $m$ high-entropy tokens and the secret key. They further incorporate error correction code to make the design robust. \n\nAs also mentioned by the reviewer, such designs are not as robust as the watermarks we study as they prioritize the resistance against spoofing instead of strong robustness. For instance, by simply modifying the first $m$ tokens, the signature check no longer passes. The designs of these works are consistent with our findings in the piggyback spoofing attacks: to defend against spoofing attacks, the design needs to incorporate less robust (or even non-robust) signature-based watermarks. We will include these related works in the revision to provide a more comprehensive study. \n\n[1] Zhou et al. Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature. arXiv 24\n\n[2] Fairoze, et al. Publicly detectable watermarking for language models. arXiv 23\n\n---\n>**Q3: How generalizable are the findings? Would the vulnerabilities and trade-offs identified apply to all types of LLMs, or are they specific to certain architectures or applications?**\n\n**A3**: Please see global response **C2**.\n\n---\n>**Q4: What are the limitations of the attack methods presented in this study? Are there scenarios where these attacks might not be effective?**\n\n**A4**: As we have discussed in **C2**, the semantics-based watermarks rely on a high-quality semantics embedding model instead of using secret keys to embed the watermark. As such designs fundamentally differ from the watermarks we study, our findings of using multiple watermark keys are not applicable here. We will include a limitation and future work section in the revision to discuss this issue and present a more comprehensive study.\n\n---\n>**Q5: Are there any practical considerations or potential drawbacks to implementing DP defense mechanisms in real-world systems?**\n\n**A5**: DP adds noise to the watermark detection results. The service provider needs to determine the optimal noise scale, as larger noise will make the detection inaccurate, and less noise will be ineffective to defend against attackers. According to our empirical findings, we can find a sweet point to achieve both high detection accuracy (low FPR) and low attack success rate. Overall, we believe that our DP defense can potentially make the detection API publicly available while protecting the secret watermark pattern information without sacrificing detection accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOlvQMN0da", "reviewer": "Reviewer_iwzS", "summary": "In this work, the authors reveal new attack vectors including watermark-removal attacks and spoofing attacks that exploit common features and design choices of LLM watermarks. Besides, the authors propose a defense utilizing the ideas of differential privacy, which increases the difficulty of spoofing attacks.", "questions": "Please answer all points in the Weaknesses section.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The research question in this paper is interesting and is a hot topic in the field of LLM watermark.\n2. The paper is well-written.\n3. The experimental data and results presented in this paper are extensive.", "weaknesses": "1. In Sec 3.1, what are the differences between \"piggyback\" and \"general\" spoofing attacks? Specifically, what does \"piggyback\" refer to?\n\n2. In Sec 3.1, regarding attacks on detection APIs, the reviewer is confused by the statement \"the attacker can auto-regressively synthesize (toxic) sentences.\" What does this mean?\n\n3. Regarding attacks discussed in Sec. 5, the attackers can discover watermarking rules by observing a large amount of watermarked text, thus enabling attacks. The original KGW paper in Sec. 5 mentions using a large context width h to defend against such attacks. However, this paper lacks explanation and discussion regarding the parameter h.\n\n4. In the detection API attack, the description of the adversary's capabilities is unclear. In Sec 6.1, the authors assume the adversary can access the target watermarked LLM's API and query watermark detection results. This implies the adversary can generate watermarked text and obtain detection results for any given text. But why can the adversary generate a list of possible replacements for x_i? Does this mean the adversary can access the perturbed probability distribution and logits of tokens? If so, this seems to exceed the stated capabilities of the adversary.\n\n5. There are some minor issues, such as the undefined notation \"V^{\\ast}\" in the definition in Sec 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments. In the following, we respond to each question.\n\n---\n>**Q1: In Sec 3.1, what are the differences between \"piggyback\" and \"general\" spoofing attacks? Specifically, what does \"piggyback\" refer to?**\n\n**A1**: Piggybacking (Sec 4, L 142) is a classic attack in computer networks, where the attacker tags along with another person who is authorized to gain entry into a restricted area. The general spoofing attacks for LLM watermarks [1, 2] usually require the attacker to first estimate the watermark pattern by making a large number of queries (observe millions of watermarked tokens) to the watermarked LLM, and then they can create malicious content with a target watermark embedded but in fact it is not generated by the watermarked LLM to ruin the reputation of LLMs. \n\nOur piggyback spoofing attack does not require estimating the watermark pattern, instead, we launch the attack based on the content generated by the watermarked LLM, which is similar to piggybacking in computer networks where the attacker relies on well-established authorization. The attacker’s goal of piggyback spoofing is the same as the general spoofing attack as they both aim to create malicious/inaccurate content with a target watermark embedded. However, a benefit of our attack is that it has much weaker assumptions on the attacker’s ability. The attacker can simply exploit the robustness property of the watermarks and maliciously edit the watermarked content without altering the watermark detection result to generate malicious but watermarked content to ruin the LLM’s reputation.\n\n[1] Jovanović et al. Watermark Stealing in Large Language Models. ICML 24\n\n[2] Sadasivan et al. Can AI-generated text be reliably detected? arXiv 23\n\n---\n>**Q2: In Sec 3.1, regarding attacks on detection APIs, the reviewer is confused by the statement \"the attacker can auto-regressively synthesize (toxic) sentences.\" What does this mean?**\n\n**A2**: In the attacks exploiting the detection APIs, the attacker will generate sentences auto-regressively, similar to how LLMs generate sentences. That is, the attacker will select each token based on the prior tokens and the detection results. Please also refer to Alg.1 and Alg.2 in the Appendix J of our paper. We will clearly explain this in the revision.\n\n---\n>**Q3: Regarding attacks discussed in Sec 5, the attackers can discover watermarking rules by observing a large amount of watermarked text, thus enabling attacks. The original KGW paper in Sec 5 mentions using a large context width h to defend against such attacks. However, this paper lacks explanation and discussion regarding the parameter h.**\n\n**A3**: Please see global response **C4**.\n\n---\n>**Q4: In the detection API attack, the description of the adversary's capabilities is unclear. In Sec 6.1, the authors assume the adversary can access the target watermarked LLM's API and query watermark detection results. This implies the adversary can generate watermarked text and obtain detection results for any given text. But why can the adversary generate a list of possible replacements for x_i? Does this mean the adversary can access the perturbed probability distribution and logits of tokens? If so, this seems to exceed the stated capabilities of the adversary.**\n\n**A4**: Please see global response **C5**.\n\n---\n>**Q5: There are some minor issues, such as the undefined notation $V^{\\ast}$ in the definition in Sec 3.**\n\n**A5**: The $V^{\\ast}$ refers to a sequence of tokens, where each token belongs to the vocabulary set $V$. We will clearly explain this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kA2aaWcpwf", "reviewer": "Reviewer_Hb7x", "summary": "The paper details three different attacks on LLM watermarking, targeting watermark removal and spoofing:\nA1: spoofing by taking advantage of the robustness of the watermark\nA2: removal by taking advantage of multiple watermarking keys\nA3: removal by taking advantage of a public detector\n\nAttacks are followed by some guidelines/defenses.", "questions": "Q1. Attack A3. Section 6. The assumption is that the watermarked LLM provides the top-5 tokens. Although some LLM APIs like OpenAI provide this information, I have some doubts that a **watermarked** LLM would do this. Especially, what are the top-5 tokens for Exp? Are they computed before or after the Gumbel trick?\n\nQ2. I am a bit surprised that $\\sigma = 4$ doesn't impact the detection performance.\nWithout DP: I suppose that w/o watermark $Z\\sim\\mathcal{N}(0;1)$ and with watermark $Z\\sim\\mathcal{N}(\\mu;1)$, with $\\mu \\approx 6$ (according to Fig. 3.a). With a threshold set to 4, this makes P_FP ~ 3e-5,  P_TP ~ 0.98, for an accuracy of 0.99.\n\nWith DP: The variances are now equal to $\\sqrt{1+\\sigma^2}$. This makes P_FP ~ 0.17, P_TP ~ 0.69, for an accuracy of 0.76.... Quite far away from what you get. More importantly, these two cases should be compared for a fixed P_FP so that the threshold with DP should be higher.  Reporting only the accuracy is masking the fact that P_FP is way higher.", "limitations": "The title and introduction discuss LLM watermarking in general, but the paper is based on three particular schemes. I agree that these three are the most well-known. It is questionable whether other more exotic schemes (like semantic-based) are also vulnerable.\n\nOne may easily think of easy counter-attacks: forbidding querying the LLM with the same prompt repeatedly (or the same prompt plus one extra token), forbidding querying the detector with too similar text, etc.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "Attack A2 is very original (AFAIK). The proposed defense against A3 based on DP is interesting.", "weaknesses": "W1. Relevance\nI have been a watermarking practitioner for years. I have never seen the following proposals:\n- Robust watermarking as proof of content authenticity (A1-Section 4)\nA robust watermarking detector distinguishes 2 hypotheses:\nH_0: content is not watermarked (by this technique and that secret key)\nH_1: content has been watermarked **and** potentially modified.\nSo, I do not perceive A1 (Section 4) as an attack but more as a misunderstanding or a misuse of robust watermarking by the authors.\nI strongly disagree with Guideline #1 (line 206), which suggests lowering the robustness of robust watermarking. This hardly makes sense.\nThe recommendation should be to combine two schemes: robust watermarking and fragile (digital-signature-based) watermarking.\nRobust watermarking for authenticity is wrong. Fragile watermarking for IA-gen detection is wrong as well.\n\n- Watermarking detectors should be public (A3-Section 6)\nThis leads to oracle attacks well documented in the watermarking literature of the 200s (nowadays called black-box attacks). The authors study even the easiest case where the attacker observes a soft decision (Z statistics): he can observe if any single modification lowers the detection score. A harder and more relevant case would be a Yes/No decision. But I am not even recommending that setup for security reasons. In short, the novelty (the attack against a specific LLM watermarking) is narrow. \"*It is still an open question whether watermark detection APIs should be made publicly available*\". It is not an open question, and the conclusion of this section is absolutely not a surprise.\n\nW2. State-of-the-art\nThe experimental setup considers 3 schemes with default parameters. Here default parameters mean parameters as appearing in the very first version of these papers. Yet, since then, we know that these choices were not adequate.\n- Use of Z-statistics. They have been shown to be suboptimal and inaccurate, leading to theoretical FPRs that are way off the empirical FPRs. I recommend using p-values (empirically validated). See \"Three Bricks to Consolidate Watermarks for LLMs\" from P. Fernandez.\n- The choice 'h=1' in KGW is not recommended (Needless to say that Unigram --where 'h=0'-- is even worse). Again, it leads to inaccurate FPR. Moreover, it is not secure (watermarking stealing attack). The question is whether your attack still holds with a larger h (with a proper hash function, not those implemented based on min of hashes).   \n\nW3. Key management\nAs far as A2 is concerned, I think the key issue is key management. The literature offers 2 flavors: randomly picking a key in a small key space (Kuditipudi) or using a hash of the previous tokens (Aaronson, Kirchenbauer). Section 5 only investigates the first option.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive comments. Please refer to **C1** in the global response for our clarifications on the positioning and contributions of our work. Our work studies the feasibility and ramifications of potential attacks, with the goal of better informing the public and the LLM watermarking community about key design tradeoffs. Below, we respond to each question.\n\n---\n>**Q1: Robust watermarks are not suitable for proof of content authenticity.**\n\n**A1**: We agree robust watermarks are more suited for AI content detection, while fragile (signature-based) watermarks are used for authentication.\n\nOur contribution in this section is a rigorous study of the tradeoff between robustness and spoofing resistance, therefore showing that no single scheme is sufficient to protect against both types of attacks. Our experiments show the extent to which robust watermarks are vulnerable. We’ve recommended using fragile watermarks for spoofing defense in Sec 4.2 (L 215), but it is also known that fragile watermarks such as signatures are not robust to editing.\n\nWe emphasize that increasing a watermark’s robustness to editing diminishes its suitability for content authenticity. This tradeoff is valuable information for the LLM watermark community. Indeed, we note that we are not the first to explore spoofing and robust watermarks, as recent works [1,2] have proposed more complex spoofing attacks on a specific set of robust watermarks. In contrast to these works, we explore a more simple and general piggyback spoofing attack that allows us to explore the inherent trade-off between spoofing and robustness. The potential for spoofing has also been cited as a major barrier to industrial LLM watermarking deployment [3], making it important to study this tradeoff rigorously.\n\nWe agree that the current guideline can be revised to better convey our conclusions. We will follow your suggestion to revise our guideline to: `Guideline #1: Robust watermarks are vulnerable to spoofing attacks and are not suitable as proof of content authenticity alone. To mitigate spoofing while preserving robustness, it may be necessary to combine additional measures such as signature-based fragile watermarks.`\n\n---\n>**Q2: The findings of attacks exploiting public detection API are not surprising.**\n\n**A2**: In Sec 6 (L 288), we stated that “Although this (public detection API) makes it easier to detect watermarked text, it is commonly acknowledged that it will make the system vulnerable to attacks. Here, we study this statement more precisely by examining the specific risk trade-offs that exist, as well as introducing a novel defense that may make the public detection API more feasible in practice.”\n\nWe mentioned the use of public detection APIs is an ‘open question’ given recent activity in the community—for example, recent keynotes mention such settings [4], and commercial AI content detection services that return confidence scores to users [5]. However, we can remove this sentence to avoid misunderstandings. Despite the risks, public detection APIs have a lot of benefits, such as improving transparency in AI usage and supporting regulatory compliance. We believe providing a detection API isn't a simple yes or no question, and that there are different knobs one can tune when providing such an API. Our work both provides and explores such knobs, so that the risks/benefits of public detection APIs may be considered for practical deployment.\n\nOracle attacks have existed for decades, but rigorously exploring them in the context of LLM watermarks is necessary. We show that making detection scores differentially private can effectively mitigate the spoofing attack without compromising detection accuracy (see **A7**). Our findings can help to enable public detection APIs deployment, inspiring future LLM watermark designs.\n\n---\n>**Q3: I recommend using p-values as a detection metric.**\n\n**A3**: Thanks for this suggestion; please see global response **C6**.\n\n---\n>**Q4: Whether your attack still holds with a larger h?**\n\n**A4**: See global response **C3**.\n\n---\n>**Q5: Key management.**\n\n**A5**: See global response **C4**.\n\n---\n>**Q6: The assumption of watermarked LLM providing the top-5 tokens.**\n\n**A6**: See global response **C5**.\n\n---\n>**Q7: DP noise scale.**\n\n**A7**: For KGW and Unigram, we add noise to the z-scores. Sensitivity varies with sentence length (e.g., $\\Delta=\\frac{h+1}{\\sqrt{\\gamma(1-\\gamma)l}}$ for replacement editing, where $l$ is the sentence length, $h,\\gamma$ are watermark parameters). The actual noise scale is proportional to $\\sigma\\Delta$. For a 200-token sentence, $h=1,\\gamma = 0.5,\\sigma = 4$, the noise scale is 0.8. We’ve tested FPR with DP defense on OpenGen dataset, and FPRs are close to 0 (below 1e-3) using our recommended noise scale. We’ll explain more clearly in the text to avoid confusion.\n\n---\n>**Q8: Are other more exotic schemes (like semantic-based) also vulnerable?**\n\n**A8**: See global response **C2**.\n\n---\n>**Q9: One may easily think of easy counter-attacks: forbidding querying the LLM with the same prompt repeatedly (or the same prompt plus one extra token), forbidding querying the detector with too similar text, etc.**\n\n**A9**: In Guidelines #2 and #3, we recommended “defense-in-depth” techniques like anomaly detection, query rate limiting, and user verification. However, with just rate limiting, our attacks remain possible, as service providers can’t always ensure trusted users. Thus, it’s important to consider the tradeoffs when deploying LLM watermarking systems. \n\n---\n[1] Jovanović et al. Watermark Stealing in Large Language Models. ICML 24\n\n[2] Sadasivan et al. Can AI-generated text be reliably detected? arXiv 23\n\n[3] Somesh Jha. Keynote at SaTML 2024-Watermarking (The State of the Union). 2024\n\n[4] Scott Aaronson. Watermarking of large language models. 2023\n\n[5] AI Purity; GPTZero; Winston AI", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHcf6aFtqK", "reviewer": "Reviewer_UAwD", "summary": "This paper demonstrates that typical design choices in large language model (LLM) watermarking schemes result in significant trade-offs between robustness, utility, and usability. To navigate these challenges, this paper rigorously examines a series of straightforward yet effective attacks on prevalent LLM watermarking approaches and proposes practical guidelines and defenses to strengthen their security and practicality.\n\nSpecifically, the robustness of existing watermarking, a desirable property to mitigate removal attacks, can make the systems susceptible to piggyback spoofing attacks, which makes watermarked text toxic or inaccurate through small modifications. By proposing a novel attack, it is further shown that using multiple watermarking keys can make the system susceptible to watermark removal attacks. Finally, it is identified that public watermark detection APIs can be exploited by attackers to launch both watermark-removal and spoofing attacks. The paper proposes a defense strategy leveraging techniques from differential privacy to effectively counteract spoofing attempts.", "questions": "I can imagine that similar issues discussed in the paper will also occur for watermarking images generated by the diffusion model. For example, the following two papers consider image watermarking removal using adversarial samples from the publicly available detection model.\n\nSaberi, Mehrdad, Vinu Sankar Sadasivan, Keivan Rezaei, Aounon Kumar, Atoosa Chegini, Wenxiao Wang, and Soheil Feizi. \"Robustness of ai-image detectors: Fundamental limits and practical attacks.\" arXiv preprint arXiv:2310.00076 (2023).\n\nJiang, Zhengyuan, Jinghuai Zhang, and Neil Zhenqiang Gong. \"Evading watermark based detection of AI-generated content.\" In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 1168-1181. 2023.\n\nThis paper would greatly benefit from a thorough discussion of the differences and connections between these two problems by having a more thorough literature review.", "limitations": "Limitations are addressed in Appendix A of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides novel attack schemes for existing watermarking techniques and provides empirical evidence to support their claims, which help the community to better understand the tradeoff among the design of watermarking systems. In general, release detection API will always make the watermarking more vulnerable, and the proposed defense demonstrates an interesting connection to DP.", "weaknesses": "One of the biggest weaknesses of this paper is that the proposed attacks mainly explore the drawbacks of existing literature in [11,14,33], and some of the tradeoffs described in the paper are tied to these algorithms or specific formulations, which are not fundamental to the watermarking problem itself.   \n\n1.\tThe robustness issue discussed in Section 4 is mainly due to the specific definition of robustness Definition 3. As robustness is defined using editing distance, the piggyback spoofing attacks leverage the fact that we can significantly change the meaning of a sentence by editing very few tokens. \nOne way to address this issue is to define robustness using the semantics of the generated text instead of editing distance. If the meaning of the text remains similar, the watermarking should still be detectable; otherwise, the watermarking should disappear if the edits dramatically change the meaning of the text. Conceptually, I believe that there should not exist a tradeoff between robustness and spoofing attack. Empirically, the authors could conduct additional experiments on the following semantics-based watermarking. I am curious to see if the proposed attacks still work for semantic-based watermarking. \n\nLiu, Aiwei, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. \"A semantic invariant robust watermark for large language models.\" ICLR 2024\n\nLiu, Yepeng, and Yuheng Bu. \"Adaptive Text Watermark for Large Language Models.\" ICML 2024\n\n2.\tAs shown in \"Adaptive Text Watermark for Large Language Models,\" by adding watermarking using semantics, the ASR of a spoofing attack is quite low without using multiple secret keys. In addition, it is likely that the same prompt will generate text with similar semantics, leading to a similar biased watermarking pattern. Therefore, it would be hard to do watermarking removal using the proposed attacks. The authors are encouraged to provide more discussion regarding the applicability of their empirical findings to different types of watermarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments. In the following, we respond to each question.\n\n---\n>**Q1: One of the biggest weaknesses of this paper is that the proposed attacks mainly explore the drawbacks of existing literature in [11,14,33], and some of the tradeoffs described in the paper are tied to these algorithms or specific formulations, which are not fundamental to the watermarking problem itself.**\n\n**A1**: Please see global response **C2**.\n\n---\n>**Q2: The robustness issue discussed in Section 4 is mainly due to the specific definition of robustness Definition 3. [...] I am curious to see if the proposed attacks still work for semantic-based watermarking.**\n\n**A2**: Thanks for this suggestion; please see global response **C2.1**.\n\n---\n>**Q3: As shown in \"Adaptive Text Watermark for Large Language Models,\" [...] The authors are encouraged to provide more discussion regarding the applicability of their empirical findings to different types of watermarks.**\n\n**A3**: Thanks for the comments. Our findings in the tradeoffs of using multiple watermark keys do not apply to the semantics-based watermarks, as they rely on a separate semantics embedding model instead of determining the watermark logit bias using a watermark key. The general spoofing attack considered in [2] works for the PRF-based robust watermarks, and we may need substantial modifications to the general spoofing attacks to gain a better attack performance in semantics-based watermarks. For instance, instead of estimating the watermarked tokens’ distributions by providing uniformly distributed prompts, the attacker will need to carefully construct the prompts to guarantee that the outputs will be semantically close, such that they can gain some information on which portion of the tokens is more likely to appear in a specific semantic context by making a large number of queries. To defend against such potential attacks, one option could be for the model provider to introduce randomness into the semantics embedding model. For example, multiple random semantics embedding models could be used during inference, similar to the setting of using multiple watermark keys, to make it more resistant to watermark stealing. In this case, our findings will still apply, but this would need a more thorough and rigorous investigation.\n\nAs we mentioned in the global response **C2**, attacking semantics-based watermarks is not the focus of our paper, but we will clarify this and provide corresponding discussions in the revision.\n\n[2] Liu et al. Adaptive Text Watermark for Large Language Models. ICML 24\n\n---\n>**Q4: I can imagine that similar issues discussed in the paper will also occur for watermarking images generated by the diffusion model. [...] This paper would greatly benefit from a thorough discussion of the differences and connections between these two problems by having a more thorough literature review.**\n\n**A4**: We agree that the attacks utilizing the detection API can be generalized to the image watermarks, as the attackers can adopt a similar oracle attack pipeline. The attackers will need to integrate domain-specific constraints to guarantee that the generated sentences or images are meaningful and high-quality. We will discuss potential opportunities and challenges in extending our attacks to image watermarks in the limitations/future work section of our revision (e.g., using the 1 extra camera-ready page if necessary). Thanks for this suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers’ constructive comments. Below we clarify our contributions, respond to common questions, and present new experimental results.

>**C1: Clarification on the contributions and positioning of our work.**

Our work explores attacks that exploit design choices of common LLM watermarks. While these design choices may enhance robustness, resistance against watermark stealing attacks, and public detection ease, we show that they also allow malicious actors to launch attacks that can easily remove the watermark or damage the model's reputation. Although some of our high-level take-aways may confirm common beliefs (e.g., the risk of spoofing robust watermarks as noted by Reviewer Hb7x), we disagree with the implication that the feasibility and ultimate ramifications of such attacks are thus unworthy of scientific study—particularly given that these design choices have been adopted/explored both in recent research and in practical deployment. Further, our work questions common folklore (such as the inability to use public detection APIs), showing that attacks on these systems may be mitigated with our novel DP-inspired defense. Overall, our goal is to rigorously study the risks and benefits of LLM watermark design tradeoffs, and to distill these results into a set of take-aways that can better inform the public and LLM watermarking community. We consider these take-aways particularly important as the research community grows and the use of LLM watermarking systems increases, potentially out of the hands of a select set of domain experts.

---
>**C2: Generalizability of our attacks. (Hb7x, UAwD, YBsi)**

We focus on three SOTA PRF-based robust watermarks, which are a natural set to explore given their popularity and formal cryptographic guarantees. There are other promising watermarks like the semantics-based watermarks as the reviewers mentioned. While attacking semantics-based watermarks is outside the scope of our study, we agree with Reviewer UAwD that this is an interesting direction to explore, and have provided an initial exploration below. We will discuss generalizing our attacks to other watermarks including semantics-based watermarks as a potential avenue of future work in our revision.

>**C2.1: Piggyback spoofing on semantics-based watermarks. (UAwD)**

Semantics-based watermarks use embedding models to capture sentence semantics and bias LLM predictions. Robustness ensures semantically close sentences yield similar watermark patterns. We agree spoofing attacks are harder if we assume perfect semantics embedding models. However, inaccuracy in the embedding can make spoofing possible. 

As a proof of concept, we attacked the SIR watermark [1], and present a concrete example in Tab.1 of the submitted PDF to show piggyback spoofing is possible. We deem this an interesting future direction to rigorously explore and will add discussions in the revision.

[1]Liu et al. A semantic invariant robust watermark for large language models. ICLR 24

---
>**C3: Consistent attack performance for larger h. (Hb7x)**

Our results hold for any h and hash function in KGW watermark. Increasing h makes brute-force watermark stealing harder, but our attacks don’t depend on h or hash functions. With the latest KGW codebase, we use h=4 and sumhash in new experiments, observing consistent results with h=1 for all attacks, as shown in Figs.1-3 in the submitted PDF.

---
>**C4: Discussions on the tradeoff of context width h. (Hb7x, iwzS)**

We primarily explored the fundamental tradeoffs in using multiple watermark keys, which prior works have underexplored. Tradeoffs in context widths (h) are discussed in prior works [1-3]. Using larger h enhances the resistance against watermark stealing but reduces robustness. Our new experiments validate this. Fig.1 shows that fewer edits are allowed for watermarked content with a larger h, indicating lower robustness. KGW recommends using h<5 in their codebase for robustness, and no prior works we are aware of suggest using h>4. Recent work [1] shows successful watermark stealing even with h=4. Using multiple keys, as shown in Sec 5 of our paper, mitigates stealing attacks, but introduces new attack vectors of watermark removal. We will add a discussion on larger h in the revision.

[1]Jovanović et al. Watermark Stealing in Large Language Models. ICML 24

[2]Kirchenbauer et al. A watermark for large language models. ICML 23

[3]Zhao et al. Provable Robust Watermarking for AI-Generated Text. ICLR 24

---
>**C5: Clarifications on obtaining top-5 tokens from the watermarked LLM. (Hb7x, iwzS)**

In our watermark-removal attack with detection APIs, we assume the attacker can generate a short list of replacements for the current token. We used the setting of returning top-5 tokens by the watermarked LLM API because it’s beneficial to users and is currently used in commercial non-watermarked LLM services including OpenAI [1]. For instance, it can help understand model confidence, enable debugging, make custom sampling strategies, etc. One of the goals of our paper is to point out how existing LLM deployment practices can lead to attacks if watermarking is integrated. The fact that this API is vulnerable to our attacks illustrates our point. We will clearly state the attacker's assumptions in the revision.

[1]OpenAI API. https://platform.openai.com/docs/api-reference/completions/create

---
>**C6: Using p-value instead of z-score as the detection metric. (Hb7x)**

We will follow the reviewer’s suggestion to change the detection metric from z-score to p-value for KGW and Unigram. P-values are used for Exp in our paper, and the observations are consistent with KGW and Unigram. We expect no impact on results from this change since p-value is monotonic to z-score. Figs.1-3 in the submitted PDF also show consistent attack performance using p-values for KGW, and we will add results for Unigram in the revision.

Meta-review: This LLM watermarking paper has received broadly supportive reviews that either firmly or partially recommend acceptance.  The reviewers appreciated value in the study of trade-offs between design choices in LLM watermarking for attack mitigation.  The more supportive reviews considered the work rigorous and with the potential to significantly impact the practice of LLM watermark design.  Other reviewer comments were broadly positive and prompted an exchange to clarify details of the baselines and experimental setup, which are either confirmed or appear to have been satisfied by the authors’ rebutall and responses.  The authors should be diligent to deliver the clarifications detailed in the discussion in the final version.  Overall the AC supports the acceptance of the paper – the reviews are positive and the work appears to be timely and potentially impactful in the practice of LLM watermarking.
Decision: Accept (poster)