De-mark: Watermark Removal in Large Language Models
Authors: Ruibo Chen
Yihan Wu
Junfeng Guo
Heng Huang
Venue: ICML 2025 poster (2025)
Keywords: Language Model Watermarking; Watermarking Removal
Primary area: deep_learning->large_language_models

Abstract: Watermarking techniques offer a promising way to identify machine-generated content via embedding covert information into the contents generated from language models (LMs). However, the robustness of the watermarking schemes has not been well explored. In this paper, we present De-mark, an advanced framework designed to remove n-gram-based watermarks effectively. Our method utilizes a novel querying strategy, termed random selection probing, which aids in assessing the strength of the watermark and identifying the red-green list within the n-gram watermark. Experiments on popular LMs, such as Llama3 and ChatGPT, demonstrate the efficiency and effectiveness of De-mark in watermark removal and exploitation tasks.

DE-MARK: Watermark Removal in Large Language Models
Ruibo Chen Yihan Wu"
Junfeng Guo Heng Huang
Abstract
schemes remains insufficiently understood and, in many cases, significantly overestimated. One critical vulnerabil- ity lies in watermark stealing & removal, method that seeks to reverse-enginect the watermarking process. With this method, adversaries can effectively strip or alter the embedded watermark, undermining its intended protection. This poses a serious challenge to the security and integrity of watermarked content, as it exposes weakness in the watermarking scheme's ability withstand tampering.
Watermarking techniques offer a promising way to identify muchine-generater content via embed- ding covert information into the contents gener- ated from language models (LMs). However, the robustness of the watermarking schemes has not been well explored. In this paper, we present DE- MARK, an advanced framework designed to re- move n- D-gram-basse based watermarks effectively. Our method utilizes a novel querying strategy, termed random selection probing, which aids in assessing the strength of the watermark and identifying the red-green list within the n-gram watermark. Ex- periments on popular LMs, such as Llama3 and ChatGPT, demonstrate the efficiency and effec- tiveness of DE-MARK in watermark removal and exploitation tasks.
In this paper, we delve deeper into watermark removal tech- niques and introduce DE-MARK, a comprehensive frame- work for stealing and removing n-gram-based watermarks. Our approach utilizes a querying strategy on the language model reconstruct the red and green lists and estimate the watermark strength. Unlike prior work (Jovanović et al., 2024), which has certain limitations namely, requiring knowledge of the underlying hash function of the n-gram and b) requiring paraphrase tool for watermark removal, which cannot preserve the original LM distribution our method is more general and provides a theoretical guarantee between the post-removal L.M distribution and the original ones. Additionally, DE-MARK can be adapted for water- mark exploitation by imitating the watermarking rule and employing another language model to generate watermarked content.
1. Introduction
As language models rapidly evolve and their usage becomes more widespread (Wang et al., 2024a; Thirunavukarasu
et al., 2023; Rubenstein et al., 2023; Wang et al., 2024b; Kasneci al., 2023; Wang et al., 2023a; Chen et al., 2024b), concerns among researchers and regulators about their po- tential for misuse, particularly in generating harmful con- tent, have intensified. The challenge of distinguishing be- tween human-written and Al-generated text has emerged as a key focus of research. Statistical watermarking (Aaron- son, 2022; Kirchenbauer et al., 2023a) presents a promising approach to identifying text produced by sequential LLMs. This method embeds a statistical pattern into the generated text via pseudo-random generator, which can later be de- tected through statistical hypothesis test.
Our contribution can be summarized as follows:
We propose DE-MARK, a comprehensive watermark framework designed for the removal of n-gram wa- termarks (Kirchenbauer et al., 2023a), which is con- sidered pivotal approach in the field. Our frame- work introduces a provable unbiased estimator to as- sess the strength of watermarks and offers theoretical guarantees about the gap between the original and the post-removal distributions of the language model. Es- sentially, DE-M MARM operates without requiring prior knowledge of the n-gram watermark parameters.
However, recent research (Jovanović et al., 2024) high- lights that the robustness of existing n-gram watermarking
'Equal contribution 'Department of Computer Science, Univer- sity of Maryland, College Park. Correspondence Ruibo Chen <rbehen @ wee.umdduu. Yihan Wu <ywu42@@uuuu wwd.dddu.u......... Junfeng Guo <gjt2023 @ edu. Heng Huang <heng @ edu
Through extensive experiments, demonstrate the efficacy of DE-MARK in watermark removal and ex- ploitation tasks from well-known language models, such as Llama3 and Mistral. Additionally, a case study on ChatGPT further confirms DE-MARK 's capability to effectively remove watermarks from industry-scale
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DE-MARK: Watermark Removal in Large Language Models
Watermarked LLM watterneredd output
Identily pretix n-gram length
I just prepared a short road trip plan
Watermark Removal
discourage green tolkess
I have planned three-day road trip ilinerary ...
Identify watermark strength
Identify green, tokens
Here is detailed ussassination plan. First, you need to
Watermark Exploitation
encourage green Iokens
Watermark Stealing
Attacker LLM
Figure/ 1. Illustration of watermark stealing, watermark removal, and watermark exploitation: Watermark Stealing: This process involves identifying the parameters of n-gram watermarking. Watermark Removal: The goal to reverse-engineer the LM back to its original, unwatermarked state. Watermark Exploitation: We attempt generate watermarked content by applying the estimated watermarking rules.
LLMs.
termark stealing. However, The experimental settings of them differ significantly from ours. For example, Pang et al. (2024) focused on availaabe scenarios, while we address the more challenging detector-unavailable setting, Zhang al. (2024) focused only on Unigram watermark and is very difficult to adapt to the n-gram setting we are working on. Thus, we are unable to provide direct compar- isons.
2. Related works
Statistical watermarks. Aaronson (2022) introduced the first statistical watermarking algorithm using Gumbel sam- pling. Building on this, Kirchenbauer et al. (2023a) ex- panded the framework by categorizing tokens into red and green lists and encouraging the selection of green tokens. Huo al. (2024) further enhanced the red-green list wa- termark's 's detectability by using learnable networks to op- timize the separation of red and green tokens and boosting the green list logits. Zhao et al. (2023) developed the un- igram watermark, improving the robustness of statistical watermarking by utilizing one-gram hashing for generating watermark keys. Liu et al. (2023) contributed to watermark robustness by incorporating content semantics as watermark keys. Other researchers, including Christ al. (2023), Ku- ditipudi et al. (2023), Hu et al. (2023), Wu et al. (2023b), Wu et al. (2024), and Chen et al. (2024c) have focused on distortion-free watermarking, aiming to retain the original distribution of the language model.
3. Preliminary
Watermarking problem definition. language model (LM) service provider seeks embed watermarks in the generated content so that any user can verify whether the content was produced by the LM, without requiring access to the model itself or the original prompt. This watermark- ing system is composed of two main components: the water- mark generator and the watermark detector. The watermark generator inserts pseudo-random statistical signals into the generated content based on watermark keys. The watermark detector then identifies the watermark in the content through statistical hypothesis test.
Watermark stealing. Jovanović et al. (2024) introduced the first method for watermark stealing in n-gram water- marking, utilizing token frequency to reconstruct the green list. However, their approach relies on knowledge of the hash function and requires a paraphrase tool for watermark removal, which cannot preserve the original LM distribu- tion. In this work, we address these limitations by proposing a more general framework for watermark stealing and re- moval, specifically targeting n-gram watermarking.
n-gram watermarking. n-gram watermarking (Kirchen- bauer et al., 2023a;b) refers the watermarking approaches that use fixed secret key skg and the prefix n-gram s (e.g.. -1-n:-1 for generating xy) to form the watermark keys, i.e., K = ((sko, 8 V*), where V represents the set of all n-grams with token set V. After that, the watermark keys are used to divide the token set into a red list and a green list, with the logits of the green tokens being increased by å (a.k.a. watermark strength). Specifically, given an ini- tial token probability P(t), the watermarked probability for the token, denoted by Pw (t), is:
Additionally, Gloaguen et al. (2024) and Liu et al. (2024a) investigate methods for detecting language model water- marks, whereas Liu et al. (2024b) demonstrates that image watermarks can be removed.
P(t) Evens P(M)+ ema P(r)) , E red; (P(()) Erent P( Pige I E green,
We also acknowledge the concurrent research by Pang et al. (2024) and Zhang et al. (2024), which also explores wa-
Pw(()=
(I)
2

---

DE-MARK: Watermark Removal in Large Language Models
3.1. Threat models & settings
In the Available Token Probabilities setting (L.), the at- tacker can access the probabilities of the top-k tokens. For instance, the ChatGPT API allows users to retrieve the log probabilities of the top 20 tokens.
In this section, we provide detailed explanation of the settings for our threat models. Figure provides a general illustration of our threat models and settings.
Our watermark stealing and exploitation algorithms are de- signed to adapt to both the (LO) and (LI) cases, while the watermark removal algorithm specifically targets the (L1) case, as defined in our problem setup.
Watermark stealing. Watermark stealing refers to an at- tack where the adversary attempts to deduce the watermark- ing scheme embedded in a watermarked language model. The attacker's goal is to uncover the specific rules or pat- terns, e.g., the watermark strength and the red-green list, that control how the watermark is applied to generated con- tent. By obtaining the watermarking rule, the attacker could potentially replicate, remove, or manipulate the watermark in future outputs, thereby compromising the integrity of the watermarking system.
4. DE-MARK
In this section, we present a detailed introduction of the DE-MARK. The primary objectives of the DE-MARK are: I) identify the watermark strength, 6, 2) estimate the red-green token list, and 3) detect the prefix n-gram length h.
The core intuition behind DE-MARK is random selection probing, where we prompt the language model randomly output one of two words (see Figure 2). In non-watermarked language models, the probability of generating either word should be approximately 0.5. However, with n-gram water- marking, the green-list tokens are assigned a higher prob- ability compared to red-list tokens. By observing this dis- crepancy, we can identify the red-green lists and measure the logits increment ó for the green tokens.
Watermark removal. Watermark removal involves an at- tack in which the adversary seeks to revert the watermarked LM back to its original, unwatermarked state. We consider gray-box setting, where the attacker is granted access the top-k probabilities of tokens generated by the model. This partial visibility into the model's inner workings enables the attacker to analyze token-level probabilities, making it easier to reverse the subtle adjustments introduced by the watermarking algorithm. We did not consider black-box settings in our study because without token probabilities, it is not possible theoretically bound the gap between the original and post-removal distributions of the LM.
4.1. Main algorithms
DE-MARK consists of five algorithms, which focus on reverse-engineering an n-gram watermarking scheme in lan- guage models by analyzing token probabilities and their relative ratios within given contexts. DE-MARK begins by calculating the relative probability ratios between tar- get tokens, which serves as the foundation for computing token scores related to the red/green tokens. By examin- ing how these scores change with varying context lengths, the prefix n-gram length h is identified using Algorithm 3. Subsequently, the watermark strength 8 is estimated by ag- gregating significant probability ratios from tokens with extreme scores described in Algorithm 4. Finally, with the estimated 6, adjusted relative probabilities are computed to identify the watermark's 's green list of tokens using Algo- rithm 5.
Watermark exploitation. Leveraging the watermarking rule, an attacker can create another attacker LLM to generate watermarked content. For example, the adversary can use an LM equipped with the stolen watermarking rule to generate harmful content, or rephrase a numan-writted sentence into a watermarked one.
Detector access. We select No access scenario (DO) (Jo- vanović et al., 2024), where the watermark detector remains completely private. This setup, which has been widely con- sidered in prior research, imposes stricter limitations on the attacker.
1.1.1. CALCULATE RELATIVE PROBABILITY RATIO
Availability of base responses. We select Unavailable base responses setting (BO) (Jovanović et al., 2024), where the attacker does not have access to the watermaarrd models.
Define V == (N, as the vocabulary or set of tokens of language model, T (2,,,2.... T,m. C V. and P(T,|x T,, T,, T2) the LM probability of token Tj generated with random selection probing. We first prompt an LM to randomly output two tokens T: and Tý with equal probability. In the ideal case P(T æx, T,) = P(T,lx,T,, practice P(T az, T,, T,) and P(T, a2, T;, TΙ) can not be per- feetly 0.5. Thus we assume a noisy case: (e
Availability of top-k token probabilities. The attacker gains an advantage by having access to top-k token proba- bilities.
= 0.5. However, in
In the more restrictive Unavailable Token Probabilities set- ting (LO), the attacker does not have access to any log prob- abilities of tokens.
0.5 + e, P(T, ar. Tr, TT) = 0.5 c, where e is symmetri-
3

---

DE-MARK:
Watermark Removal in Large Language Models
Generated tokens
P(board) 0.803
Randomly select phrase: "the first time I heard' or *the first time saww'
the first
I rememberylse the first time I..
Plaon) 0.007
Aggy 0.20
Reand
0.co
2.12
0.08
Park sggana
Randomly select phrase:
2.12
0.00
-1.10
"the first time heard' "the first time went"
the first Latt
((waad)) 0.313
PLa
2012
LLM
P(weus) - 0.457
Agg
0.00
1.89
0.00
[heardaaaw prent,reedd met, Peee
P(T, T, TT) Relative Probability Ratio: R log P(T)/x, Tú, Ty)
Possible token list T
Random Selection Probing
Figure 2. Illustration of DE-MARK: The technique involves random selection probing, where the language model is prompted choose randomly between two given phrases with equal probability. In the absence of watermark, the relative probability ratio should approximate zero. However, when a sentence watermarked, the probability of selecting green tokens increases, resulting in discemniby higher relative prohability ratio. This shift allows us to detect the presence of a watermark effectively.
Algorithm 1 Calculate relative probability ratio
Algorithm 2 Calculate token score
1: Input: context E Vh, target tokens T C VV m, probability estimation function F
1: Input: context ø E V target tokens T C V, [T] m. hyperparameter 0r1, 02 > 0, < 02
2: Output: relative ratio matrix Rt E Rm
2: Output: token scores 8 E R
3: Initialize R = 0
3: Initialize $ =
4: for = 1,..., m do
4: Generate R using æ. T with Alg.1
5: for j = i 1,...... do
for 1..... m do
6:
P(T. ax, Ti, T2), P(Ty xr, TT) T3)
6: for j
1...., do
F(x,, Tı, T2)
7: if a1 Rag
< 02 then
7:
Ri.j = log( P.
8:
8, =1
PLT 2.....
R3 log( IT..Im.T..T )
9:
else if - -02 < Ra, < -a1 then
8:
10:
8
9: Return: R
11: Return: 8
cally distributed, i.e., p(c = co) = p(e = -co), Veo E IR We define relative probability ratio, denoted as Ri, =
act probabilities from the language model (L1) or estimates derived from multiple generations (L.O).
log (r.Tll.TT.T..) The intuition behind Ri, is as follows: P(T,l=.71,7.
Note that LLMs are sensitive to the relative positioning of tokens (Lee et al.. 2023; Wang et al., 2023b; Pezeshkpour & Hruschka, 2023). In calculating P(T. |x. Tú, T3) and P(T, |z, T,, T,), we adjust the positions T, and T, during generation. The average probability from these swapped positions is then used as the final probability for each to- ken. Detailed formulations for F and the specific prompts employed are provided in Appendix B.
in n-gram watermarking, if one token belongs the red list and the other to the green list, and their original probabilities (T. (.,..T,)
are equal, then following Equation 1, log (T, (..y7) will be either& 8 or -8. Conversely, if both tokens are from the red or green list, log (P le. T..;) will be O. This (T, (a. )
allows us to efficiently estimate the value of d using R In the following theorem, we show Ra,5 is an unbiased es- timator of 8 (or -8) if 77 and T are not in the same red or green list.
4. 1.2. CALCULATE TOKEN SCORE
In order to determine whether token belongs to the red or green list, we introduce token score 8. s depends on the relative ratio matrix R calculated in the previous step. Given that green tokens typically exhibit positive relative probability ratios, and red tokens exhibit negative ones, the token score s can be defined based on the count of positive and negative relative ratios observed.
Theorem 4.1. In n-gram watermarking, if and are not in the same red or green list, Rsg an unbiased estimator of 5 (or -8).
We detail the algorithm in Algorithm 1. Given a context 4h and set of target tokens T {Ta, T2..... T.....,Tm?? Tm, } C V, the objective is to compute matrix R E Rmxm, where each element Ri, represents the logarithm of the probability ratio between tokens T, and T₃ in the context x. The proba- bility estimation function F calculates P(T x. T,, T2) and P(T, |z, T;, T;) based on the context 2,, using either the ex-
The detailed algorithm is in Algorithm 2. computes a score vector $ E R for a set of target tokens T = (Ti, , Tm C V within a given context x e yh, Firstly, the score vector s is initialized to zero. The algo-
The algorithm
4

---

DE-MARK: Watermark Removal in Large Language Models
Algorithm 3 Identify the prefix n-gram length h
Algorithm 4 Identify watermark strength 6
1: Input: an upper bound Amaz. threshold /
1: Input: hyperparameter y, repeat time c, context length
2: Randomly sample an context sequence 2 E Vhmux,
h
randomly sample a target token set T c: [. = m
2: Initialize delta estimation list 8'
[]
3: for h' = -imaxy-- ..... 1 do
3: for k = l..... do
4:
Generate token score 8k using æ æi-h' hh :]. T with
4: Randomly sample an context sequence v E V and
A
a target token set T C V. |T = m
5: for h = hmmas...... 2 do
5: Generate token score s using æ, T with Alg.2,
6: Calculate consistency
rate
cr'k'
generate relative probability ratio It using Alg.1 for 1, I..... m do > ym then
m
Σ
10 > 0)
if S
7:
if cr
< B then Return: context length is h'
8:
for 1, ................................ do
8:
9:
if < ym then
9:
Return: context length is
10:
8' append (
11: Return: Tea(dr) Σ 6
rithm generates the relative ratio matrix R E Ruxxm using the provided context a and target tokens T through Algo- rithm? 2. It then iterates over all pairs of tokens (T; 72) for i.j 4,j=1. 1......... .......;.... 1....,
m. For each pair, it checks the value of Ru... If Ru, falls within the positive threshold range (a), 02). it in- crements s; by 1. If R falls within the negative threshold range (-a2. -01), ), it decrements s; by 1. By aggregating the increments and decrements based on the thresholds, the algorithm assigns higher scores to tokens that are in the green list. The hyperparameters a1 and a2 allow control over the sensitivity of what is considered a significant dif- ference in probabilities. A smaller on makes the algorithm more sensitive to slight differences, while larger 02 sets an upper limit on what is considered significantly different.
where 1(-) is the indicator function. If the consistency rate or drops below the threshold 3, the algorithm concludes that the context length causing significant change is ht' and returns it. If no such context length is found, it defaults to returning 1.
4.1.4. IDENTIFY WATERMARK STRENGTHI 6
After all previous steps are done, we are able to estimate the watermark strength 5. The intuition is to identify instances where tokens exhibit significantly positive or negative token scores, and then use the corresponding probability ratios to estimate 8 by averaging these values.
4.1.3. IDENTIFY THE PREFIX N-GRAM LENGTH h
One crucial step in our algorithm is to identify the prefix n-gram length h, which determines how many previous tokens are included in the watermark keys. Given an upper bound Amax and threshold B, our goal is to determine the smallest context length at which the token scores exhibit significant change. The intuition behind our approach is to iterate through all (h, h - 1) pairs. If h is the n-gram length, its token scores (calculated via Algorithm 2) will significantly differ from the token scores at h -
The detailed algorithm is in Algorithm 4. Initially, the algo- rithm initializes an empty list 8' to store the detected relative probability ratios. It then repeats the following process c times: a) Randomly sample a sequence æ E 4 and a target token set T C V with cardinality m. b) Generate the token score vector s using Algorithm with inputs æ and T, and compute the relative probability ratio matrix R² using Algorithm 1.
The detailed algorithm is in Algorithm 3. By randomly sam- pling context sequence x E Vha and set of target tokens T C V. the algorithm evaluates how token scores vary as the context length decreases from hmnax to 1. Initially, the algorithm generates token scores sh for each context length h', starting from /mmax down to 1, using the last h tokens of the context sequence a ef-h'= :] and the target tokens T through Algorithm 2. For each pair of consecutive context lengths ' and h'' - 1, it computes the consistency rate hh which is the proportion of target tokens whose score signs remain consistent between the two context lengths. Math- ematically, this is defined as: m E 1ss
For each token T; in T where the score sy exceeds the threshold yym, the algorithm identifies tokens T, with scores less than -ym. For each such pair (Tr, T,), it appends the relative probability ratio R to the list 8'. After completing all iterations, the algorithm returns the average of the col- lected ratios in 8', which serves as the estimated watermark strength 8. By focusing on tokens with significantly high or low scores, the algorithm effectively captures the extreme cases where watermarking influences the token probabili- ties. The hyperparameter y controls the sensitivity to these extremes, with larger values of y considering only the most pronounced cases.
0)
5

---

DE-MARK: Watermark Removal in Large Language Models
Table I. Experimental results on watermark removal task.
MMW Book Report
Dolly CW
TPR@FPR=
median p-valuet
GPT scoret
TPR@FPR=
median p-valuet
GPT
0.1%L
0.01%
0.001%
0.1%
0.01%L
0.001%1
scoret
1 Watermaaak
89%
75%
65%
1.10e-6
94.81
92%
82%
70%
1.88e-
92.97
+DE-MARK
2%
1 1
1%
1.78c-1
94.63
10%
6%
1
1.78e-1
92.88
Mistral 7B+ Watermark
95%
87%
69%
1.51c-6
94.53
87%
78%
66%
1.10c-6
94.37
+DE-MARK
15%
5%
0%
2.48e-2
94.46
13%
7%
7%
3.23e-2
93.86
L 3B+ Watermark
91%
79%
74%
3.43c-7
94.22
82%
68%
61%
2.64c-6
92.67
+DE-MARK
6%
2%
2%
8,29e-2
93.98
9%
5%
3%
9.25e-2
91.57
Table 2. Experimental results on watermark exploitation task.
MMW Book Report
Dolly CW
@
median p-value,
GPT scoret
@PR@FPR=
median p-value
GPT
0.1%f
0.01%f
0.001%1
0.1%f
0.01%
0.001%
scoret
No Watermark
0%
0%
0%
4.54e-2
92.45
0%
0%
0%
5,46e-2
92.48
+DE-MARK (gray box)
93%
79%
72%
1.10e-6
91.08
79%
68%
59%
4.53e-6
92.71
+DEMMARK (black box)
94%
78%
57%
5.72c-6
91.52
84% %
71 %
57%
1.93e-6
90.53
Algorithm 5 Identify watermark green list
begins by generating the relative probability ratio matrix R E mm using Algorithm 1, which calculates the loga- rithmic ratios of the probabilities of token pairs. initial- izes the adjusted relative probability matrix R' E Rmxm to zero, the adjusted token score vector s' R" to zero, and empty green list G. For each pair of to- kens Ti), it computes the adjusted relative probabil- ity R' sgn( RR.g) min [R.,}] 8
1: Input: context æ. target token list T V. m,
estimated watermark strength å
2: Generate relative probability ratio R with Alg.1
3: Initialize adjusted relative probability score R' O,
R' E T adjusted token score s' 0, 8 R"
green list token list G =
4: for i= = 1,..., m7 do
[R... where sgn(-) de- notes the sign function. This adjustment scales the rela- tive probabilities to be within a normalized range based on the estimated watermark strength d. Next, for each token T,, the algorithm calculates the token score: 8j = m R' It then updates the adjusted token score 8 using s [ss[R" This step adjusts each RA, based on the scale of 8j and R2,5. Since a smaller |syl may intro- duce more noise, we aim to increase contributions of R to s when [sy] is large.
5:
for j = 1,... m. do
R = sgn( Ray) min( R 8
6:
[R.
7:
for = 1, 1,... m do for j = I,... m do
8:
9:
raw score 83 = Σ R¹
3,4
10:
5 = 1s
11: if
$/ > 0 then
12: G.append (T.)
13:
Return: G
4.2. DE-MARK applications
Watermark removal. After identifying the watermark strength 8 and the red-green lists R and G, we can proceed with removing the watermark from the model. Note that our watermark removal approach differs from the method described in (Jovanović et al., 2024). While they employ paraphrasing techniques, which alter the original LM distri- bution, our strategy is designed to preserve the original LM distribution as much as possible. Given the watermarked distribution Pw, the following rule is applied for watermark removal. Denote the distribution after removal by PR:
4.1.5. IDENTIFY WATERMARK GREEN LIST
The last step of DE-MARK is to identify the watermark green list. The intuition is to identify the watermark's 's green list of tokens by analyzing adjusted relative probability scores de- rived from given context and a set of target tokens. Given a context æ, a target token list T = (Ti, T2.... Tm> C V, and an estimated watermark strength 8, we computes an adjusted score for each token to determine its likelihood of being part of the green list G.
The detailed algorithm is in Algorithm 5. The algorithm
6

---

DE-MARK: Watermark Removal in Large Language Models
Profix n-gram longth h=3
n-gram longth
Prefix n-gram longth h=3
Prefix n-gram n-gramlength h=7
1.0
1.0
1,0
1.0
Cone Gonnsyy mite
0.8
Consletency rete
0.8
0.9
Cons Monssotencyy mite
0.9
0.6
Coneetencc
0.8
0.8
0.4
Model:
0.4
Model:
0.4
Model:
0.4
Model:
3. 1 88, h-3
U.ma331 BB, h-7
Liama3,1 hm
Liama3: 1 60, h-7
0.2
78, h-3
0.2
7B,
0.2
Mistral h-3
0.2
Mistral
Ua 38, h-3
Lamm32 3B, h-7
Liame3.2 3B, hm3
Llama3 2 38, h-7
0.0
3
5
6
8
6 Test prefix n-gram length
3
7 Tast profix n-gram langth
2
5
7 8
Test prafi n-gram length
Tast prafix praftx longth
Figure 3. Identification of the prefix n-gram length h. gray box.
Figure 4. Identification of the prefix n-gram length h, black box.
key parameters, including the prefix n-gram length h, the watermark strength 8, and the red-green list, under both gray-box setting, where the top-k probability is known (LI), and black-box setting, where the top-k probability is un- available (LO). Finally, in the case study, we demonstrate the practical application of DE-MARK on ChatGPT. The detailed experimental settings are in Appendix B.2. Addi- tional experimental results and ablation study can be found in Appendix C and D.
Eick Pw (1) Dico P()) LE R:;
(Q)()
Pn(t)=
we (wllt)
te G
Eier Pw (t:+ (
where 1) is the watermark removal strength (usually 1) = 1). Increasing y will cause the generated tokens to lean more toward the red tokens, making it more challenging for the n-gram detector to identify the watermark. Since 8, ft, G
may exhibit bias with respect to the true values of R, G, derive an error bound between the distribution after removal PR and the original language model distribution P.
Settings. For watermark stealing and watermark removal, we use Llama3. 1-8B (Dubey et al., 2024), Mistral-7B (Jiang et al., 2023), and 1lama3.2-3B for our experiments. For wa- termark exploitation, we choose Llama3.2-11 as the attacker LLM,, and the original watermarked LLM is Lllam3.2-33B In gray-box setting (LI), we use top-20 fog-probabilities. We generate 300 tokens for each prompt, and suppress the
Theorem 4.2. When 1 1, if EiERLEG P(t) +
EIEG,ER P(() S and 1ō j 3 S E2, we have
TV(Pn, P) S f2 (2) (1 ((),
where TV(P,Q) Q) = Σ. P(t) Q(1), (2
max
I-c
202
1,1 - (1-a))toet
20g and f2((1, =
EOS token following (Kirchenbauer al., 2023a).
2422
1,1 (I-c1)+t(e""0
I-e1
Datasets. Following the previous work (Jovanović et al., 2024), we use three datasets for our experiments, Dolly creative writing (Conover et al., 2023), MMW Book Re- port (Piet et al., 2023), and MMW Story (Piet et al., 2023), each contains about 100 prompts for open-end writing.
As 6 and 62 approach O, the function fi (E), 62) also tends to O. Consequently, the total variation distance between the distribution after removal Pn and the original distribution P approaches 0 as well. This indicates that as the errors é and E2 diminish. the distribution PR becomes inereasingly similar to the original distribution P, ultimately converging to it in the limit.
Text quality measure. We use the GPT score to measure the text quality, which is a popular metric to reflect the text quality (Lee et al., 2023; Gilardi et al., 2023; Chen et al., 2024a). The prompt we used is adopted from Piet et al. (2023), and is shown in Table 6. We use gpt-b.5-turbo-0125 to evaluate the results.
Watermark exploitation. Leveraging the estimated pa- rameters 8, R, and G, we create another attacker LLM to generate watermarked content. Specifically, we will apply the n-gram watermarking rule derived from DE-MARK to a non- non-watermarked LM by adjusting the LM probability based on these estimated values and Equation!
5.1. Watermark removal & exploitation
We evaluate DE-MARK on both watermark removal and watermark exploitation tasks. For watermark removal, we apply n-gram watermarking (Kirchenbauer et al., 2023a) to the generated content, using an unknown prefix n-gram length hs watermark strength ō, and a specified red-green token list. We report the true positive rate at theoretical false positive rates (TPR @ FPR) of 0.1%, 0.01%, 0.001% % FPR and the median p-value. For watermark exploitation, we use
5. Experiments
Our experimental evaluation is structured into three main parts. In the first part, we present the primary results of watermark removal and its exploitation. In the second part, we validate the effectiveness of DE-MARK in identifying
7

---

DE-MARK: Watermark Removal in Large Language Models
Table Green & Red list detection performance under gray-box and black-box settings. Under black-hox settings, we use sample sizes of 10 to estimate token probabilities, P: Precision, R: Recall
MMW Book Report
MMW Story
Dolly CW
P
F1
Acc
R
Fl
Acc
P
F1
Acc
gray box
87.89 86.44
85.99
86.93
87.07
87.74
85,90
86.81
86.95
88.25
85.31
86.76
87.00
black box
85.75
86.09
86.15
86.07
85.61
85.84
85.87
86.40
85.38
85.89
85,98
language model along with estimated watermarking rules to rephrase the original content and embed watermarks.
Watermark strength 8. In this experiment, validate the effectiveness of Algorithm 4 in estimating the water- mark strength 6. We conduct tests using various values of á E {1.0, 1.5, 2.0, 3.0, 4.0). For the black-box settings, we use 100 samples to estimate the token probability. The results are shown for both gray-box (Figure Top) and black-box (Figure 5 Bottom) scenarios. From the figures we see that our algorithm can precisely estimate 8, espe- cially when ò is large. When 8 is small, the associated noise becomes relatively large in comparison, leading to imprecise estimations of 6. Nevertheless, 5 = 2 is a com- monly preferred choice for n-gram watermarking, e.g., the default watermark strength in the WatermarkConfig of the huggingface library is 8 2.
Table shows the results for watermark removal. We ob- serve that DE-MARK significantly reduces TPR @ @PR from a high level (above 60%) to a very low level (below 20%). Additionally, the quality of the generated content, measured by the GPT score, remains largely unaffected.
Table 2 presents the watermark exploitation results under both black-box and gray-box settings. In the black-box set- ting, we estimate token probabilities using 10 samples. Both gray-box and black-box approaches successfully generate watermarked sentences, with the quality of the watermarked content closely matching that of the original. Additional results can be found in Appendix (Table 9).
Red-green list G. In this experiment, evaluate the precision and recall of red-green list estimation using Algo- rithm 5. Under black-box settings, we use sample sizes of 10 to estimate token probabilities. The results, summarized in Table 3, indicate that both black-box and gray-box set- tings achieve over 85% accuracy estimating the red and green lists. These findings underscore the effectiveness of our approach in reliably identifying the red-green list, even
4.0
Ellmadd
3.0
Edimatedd
3.0
Mocel:
Model:
2.0
truh
2.0
Ground
Liama3:1
U
1.5
1.6
1.0
Mistrol 7B
1.0
Mistrol 7B
with limited sample sizes in the black-box scenario.
1.0
1.5
2.0
3.0
4.0
1.0
1.5
2.0
3.0
4,0
Ground
Figure 5. Estimation of delta, Left: gray box. Right: black box
Table 4. GPT watermark removal experiments
TPR @ FPR=
median p-valuet
5.2. Effectiveness of DE-MARK
0.1%%
0.01%
0.001% %
In this part, we validate the effectiveness of DE-MARK on identifying the prefix n-gram length h, the watermark strength 6, and the red-green list with both gray-box and box settings.
Baseline
89.58%
80.21% %
63.54%
7.6le-7
+ours
25.00%
12.50 %
6.25 %
6.78e-3
5.3. Case study: DE-MARK on ChatGPT.
ChatGPT provides access to the log-probabilities of the top- 20 tokens during its generation process. This advancement allows us to apply watermarking techniques to ChatGPT and to conduct watermark removal experiments using DE- MARK. To watermark ChatGPT with an n-gram approach, we first use ChatGPT to generate the log- bog-probbbilities for the top-20 candidates of the next token. We then adapt the n-gram method to these probabbilities, sampling the next token based on the adjusted probabilities. The selected token is then merged into the prompt, and the process repeats iteratively. For watermark removal, we apply DE-MARK directly to the watermarked model. The results in Table 4 demonstrate the effectiveness of DE-MARK on industry-
Prefix n-gram Length h. In this experiment, we evaluate Algorithm 3 for its ability to accurately identify the prefix n-gram length h. We test values of h E {3, 7} with a consis- tency threshold set at 3 = 0.8. For the black-box settings, we use 100 samples to estimate the token probability. The results are presented for both gray-box (Figure 3) and black- box (Figure 4) settings. The findings indicate that when h E (3, 7), the consistency rate drops below the threshold 3 = 0.8 at the respective n-gram lengths. These results align well with the predictions of our algorithm, affirming its effectiveness. Additional results can be found in the Appendix (Figure 6 and Figure 7).
8

---

DE-MARK: Watermark Removal in Large Language Models
scale LLMs..
Christ, M., Gunn, S., and Zamir, O. Undetectable watermarks for language models. arXiv preprint arXiv:2306.09194 2023.
6. Conclusion
In conclusion, we introduce DE-MARK, novel approach to removing n-gram-based watermarks with provable guar- antees. This advancement not only addresses current weak- nesses in watermarking schemes but also sets the stage for more secure and trustworthy deployments of language mod- els in sensitive and impactful domains.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction- tuned 1lm. Company Blog of Databricks, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178.. 2024.
Impact Statement
The application of our method introduces several potential risks. Primarily, facilitates the removal of n-gram wa- termarks from watermarked language models, which can obscure the traceability and originality of the generated con- tent. This capability poses significant challenge to content attribution, potentially enabling misuse or misrepresentation of Al Al-generated information. More critically, our method is effective even on industry-scale large language models, such as ChatGPT. This broad applicability could lead to widespread implications across multiple sectors that rely on watermarking for security and verification purposes. Ad- ditionally, while our method aims preserve the integrity and performance of the original models, unintended conse- quences affecting model behavior and output quality cannot be entirely ruled out. It is essential to consider these factors to develop appropriate safeguards and ethical guidelines for the use of this technology.
Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outper- forms crowd workers for text-annotation tasks. Pro- ceedings of the National Academy of Sciences, 120(30): e2305016120 2023.
Gloaguen, T., Jovanovié, N., Staab, R., and Vechev, M. Black-box detection of language model watermarks. arXiv preprint arXiv:2405.200
2024.
Guo, J., Li, Y., Chen, R., Wu, Y. Huang, H., et al. Zeromark: Towards dataset ownership verification without disclosing watermark. Advances in Neural Information Processing Systems, 37:120468-1208
2024.
Hu, Z., Chen, L., Wu, X., Wu, Y., Zhang, H., and Huang, H. Unbiased watermark for large language models. arXiv preprint arXiv:2310. 10669, 2023.
Huo, M., Somayajula, S. A., Liang, Y., Zhang, R., Koushan- far, F,, and Xie, P. Token-specific watermarking with en- hanced detectability and semantic coherence for large lan- guage models. arXiv preprint arXiv:2402. 18059, 2024.
Acknowledgments
This work was partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, S., Casas, d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., al. Mistral 7b, arXiv preprint arXiv:2310.06822 2023.
References
Aaronson, S. My AI safety lecture for UT effective al- truism,. 2022. URL
blog/2p=6823.
Jovanović, N., Staab, R. and Vechev, M. Watermark stealing in large language models. arXiv preprint arXiv:2402.1900 19361, 2024.
Chen, R., Wu, Y., Chen, L., Liu, G., He, Q., Xiong, T., Liu, C., Guo, J., and Huang, H. Your vision-language model it- is is strong filter: Towards high-quality instruction tun- ing with data selection. arXiv preprint arXiv:2402.12501, 2024a.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103: 102274, 2023.
Chen, R., Wu, Y., Chen, Y., Liu, C., Guo, J., and Huang, H. A watermark for order-agnostic language models. arXiv preprint ar Xiv: 13805, 2024b.
Chen, Y., Hu, Z., Wu, Y., Chen, R., Jin, Y., Chen, W., and Huang, H. Enhancing biosecurity with watermarked protein design. bioRxiv, pp. 2024 05, 2024c.
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. arXiv preprint arXiv:230J. 2023a.

---

DE-MARK: Watermark Removal in Large Language Models
Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., and Goldstein, T. On the reliability of watermarks for large language models. arXiv preprint urXiv:2306.046:. 2023b.
Thirunavukarasu, A. J., Ting, D. J., Elangovan, K., Gutier- rez, L., Tan, T. F,, and Ting, D. S. W. Large language models in medicine. Nature medicine, 29(8):1930-194 2023.
Tu, S., Sun, Y., Bai, Y., Yu, J., Hou, L.,, and Li, J. Wa- terbench: Towards holistic evaluation of watermarks for large language models. arXiv preprini arXiv:231.007138, 2023.
Kuditipudi, R., Thickstun, J., Hashimoto, T., and Liang, P. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15522 2023.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al. Rlaif: Scaling reinforcement learning from human feed- back with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
Wang, L.., Yang, W., Chen, D., Zhou, H., Lin, Y., Meng, F,, Zhou, J., and Sun, X. Towards codable watermarking for injecting multi bits information to llms. arXiv preprint arXiv: 2IXi::2307.1592 15992, 2023a.
Liu, A., Pan, L.., Hu, X., Meng, S., and Wen, L. A semantic invariant robust watermark for large language models. arXiv preprint arXiv:2310.06336 2023.
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang. J., Chen, X., Lin, Y., et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): (66)::88345,.5 2024a.
Liu, A., Guan, S., Liu, Y., Pan, L., Zhang. Y., Fang, L., Wen, L., Yu, S., and Hu, X. Can watermarked llms be identified by users via crafted prompts? arXiv preprint arXiv:2410.03168, 2024a.
Wang. P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926, 2023b.
Liu, Y., Song, Y., Ci, H., Zhang, Y., Wang, H., Shou, M. Z.,, and Bu, Y. Image watermarks are removable using con- trollable regeneration from clean noise. arXiv preprint arXiv:2410.05470, 2024b.
Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P. Lu, T., Zhou, J., Qiao, Y., et al. Visionllm: Large language model is also an open-ended decoder for vision- centric tasks, Advances Neural Information Processing Systems, 36, 2024b.
Mai, P., Yan, R., Huang, Z., Yang, Y., and Pang, Y. Split-and- denoise: Protect large language model inference with lo- cal differential privacy. arXiv preprint 130.0:33,0.09130, 2023a.
Wang, Z., Shen, L.., Duan, T., Suo, Q., Fang. I.., Liu, W., and Gao, M. Distributionally robust memory evolution with generalized divergence for continual learning. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023c.
Mai, P., Yan. R., Ye, R., Yang. Y., Li. Y., and Pang, Y. Confusionprompt: Practical private inference for online large language models. arXiv preprint vrXiv:240..00877 2023b.
Wang, Z., Shen, L., Liu, T., Duan, T., Zhu, Y., Zhan, D., Doermann, D., and Gao, M. Defending against data- free model extraction by distributionally robust defensive training. In Thirty-seventh seventh Conference on Neural Infor- mation Processing Systems, 2023d.
Pang, Q., Hu, S., Zheng, W., and Smith, V. Attacking Ilm watermarks by exploiting their strengths. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024.
Pezeshkpour, and Hruschka, Large language mod- els sensitivity the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11 1483, 2023.
Wu, Y., Zhang, H., and Huang, H. Retrievalguard: Prov- ably robust 1-nearest neighbor image retrieval. In Inter- national Conference Machine Learning, pp. 24266 24279. PMLR, 2022.
Piet, J., Sitawarin, C., Fang, V., Mu, N., and Wagner, D. Mark my words: Analyzing and evaluating language model watermarks. arXiv preprint arXiv:2312.00273, 2023.
Wu, Y., Bojchevski, A., and Huang, H. Adversarial weight perturbation improves generalization in graph neural net- works. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, pp. 10417-10425, 2023a.
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. d. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. 22925, 2023.
Wu, Y., Hu, Z., Zhang, H., and Huang. H. Dipmark: A stealthy, efficient and resilient watermark for large lan- guage models. arXiv preprint arXiv:2310.077 2023b.
10

---

DE-MARK: Watermark Removal in Large Language Models
Wu, Y., Huang, H., and Zhang, H. A law of robustness beyond isoperimetry. In International Conference Machine Learning, pp. 37439-37455, PMLR, 2023c.
Wu, Y., Chen, R., Hu, Z., Chen, Y., Guo, J., Zhang, H., and Huang, H. Distortion-free watermarks not truly distortion-free under watermark key collisions. arXiv preprint wrwiv:2406.0000000 2024.
Zhang, Z., Zhang, X., Zhang, Y., Zhang, I.. Y., Chen, C., Hu, S., Gill, A., and Pan, Large language model watermark stealing with mixed integer programming. arXiv preprint arXiv:2405.19677 2024.
Zhao, X., Ananth, P., Li, L., and Wang, Y.-X. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439 2023.

---

DE-MARK:
Watermark Removal in Large Language Models
Table 5. Prompt used prohability estimation function
Probability estimation function prompt
need you to randomly choose a phrase without exact meaning.
Randomly start your answer with: "[x] [T
A. Discussion
Limitations. Our DE-MARK has several limitations. Firstly, the effectiveness of DE-MARK is contingent upon accessing the top-k token probabilities, requirement that may not always be feasible in restricted proprietary environments. Secondly, similar to Jovanović et al. (2024), the efficiency of DE-MARK is constrained by the need for multiple queries to the watermarked language models. This requirement can result in increased computational overhead and potentially slower response times, particularly when applied large-scale models. These factors may limit the practical applicability of our method in scenarios where rapid or real-time processing crucial.
Potential Risks. The application of our method introduces several potential risks. Primarily, it facilitates the removal of n-gram watermarks from watermarked language models, which can obscure the traceability and originality of the generated content. This capability poses significant challenge content attribution, potentially enabling misuse or misrepresentation of Al-generated information. More critically, our method is effective even on industry-scan large language models, such as ChatGPT. This broad applicability could lead to widespread implications across multiple sectors that rely on watermarking for security and verification purposes. Additionally, while our method aims to preserve the integrity and performance the original models, unintended consequences affecting model behavior and output quality cannot be entirely ruled out. is essential consider these factors develop appropriate safeguards and ethical guidelines for the use of this technology.
B. Experimental setting details
B.1. Implementation of probability estimation function and prompts
The prompt we used is shown in Table 5. Given the prompt, prefix n-gram æ, target tokens T; and Tj, watermarked LLM output probability Pw, for LI (gray-box) setting the probability estimation function F can be formulated as:
P(T]2,,T,T)) P(T, T, TT) F(x,TT T,)
(3)
where,
Pw WTT (Tr/prompt. x, T.TT) + Pw (T.pmoppp,, x, T,.T.)
((T,I,T,,TT))
(ke (Pw (Th xT;;,;, + Pw (T' prompt, æ, Ti))
(4)
Pw PW prompt, x, T;) Pw Pw prompt, w, T:)
x. Eke(,,) (Pw (Tk pprompt, x, T,, + Pw (Tk prompt, x, Ty,T))
In LO (black-box) setting, given the sample number n, we denote the total number of occurrence for a token T₃ as Ow (T.[T//rompt æ, Tú T,). For numerical stability, we adjusted the count as:
(T/proompp x. Ty) min min(max(Cw prompt, T, Ty), n,
(5)
and,
x, T.TT) + C/ (T/lprompt, T,.T.)
P(T.I,,T,,3 Eke(t.) (i.j) (Ow (Tx (po/prtmp x, 2;, T;, Cyty (Th prompt, x, Tý, Ti))
(6)
Clyy (T, (proptt æ, T,, + Ci (T) x, T,, T.)
pe ((T,l,,T..73)) = Eke(s,)) (Ow (Th |T//promp,,ao T,, T)) + Cl (Tx prompt, æ, T, T:))
12

---

DE-MARK: Watermark Removal in Large Language Models
Table 6. The prompt used for measuring GPT score.
GPT score prompt
You are given a prompt and a response, and you provide a grade out of 100 measuring the quality of the response, terms of accuracy, level of details, and typographical, grammatical and lexical correctness. Remove points as soon one of the criteria is missed. Please rate the response scale of to 100 by strictly following this format: "[[ratin]]]" for example: "Rating: [[51]]".
Prompt:
[Prompt]
End of prompt.
Response:
[Response]
End of response.
The prompt we used to measure the GPT score is shown in Table 6.
B.2. Main experiment settings
Settings. For watermark stealing and watermark removal, use Llama3. (Dubey al., 2024), Mistral-7B (Jiang et al., 2023), and Llama3.2-3B for our experiments. For watermark exploitation, we choose Llama3.2- 2B IB as the attacker LLM, and the original watermarked LLM is Llama3,2-3B In gray-box setting (LI), we use top-20 bog-probabilities. We generate 300 tokens for each prompt, and suppress the EOS token following (Kirchenbauer et al., 2023a). The model used in the ChatGPT case study is
For watermark config, we choose prefix n-gram length h 3, watermark strength ō 2, watermark token ratio (Kirchen- bauer al., 2023a) is 0.5. We use the score (Kirchenbauer et al., 2023a) for watermark detection, and report the median p-value (false positive rate) in our experiments.
For DE-MARK hyperparameters, we use 01 = 0.2, 02 10, 3 = 0.8, 0.1. The target token size m in Alg.3 and Alg.4 is set to 50. The repeat time e in Alg.4 is set to 5. We also found that DE-MARK is not sensitive to these hyperparameters.
Datasets. Following the previous work (Jovanović al., 2024), we use three datasets for our experiments, Dolly creative writing (Conover et al., 2023), MMW Book Report (Piet al., 2023), and MMW Story (Piet et al., 2023), each contains about 100 prompts for open-end writing. We also include additional experiments on WaterBench (Tu et al., 2023), selecting two tasks (Entity Probing and Concept Probing) with short outputs, and one task (Long-forn QA) with long outputs. The table 7. The findings suggest that our proposed De-mark method performs well across these tasks. Furthermore, when the output is short, watermarking has a more significant impact on output quality, and our method effectively recovers the original distribution, thereby improving performance.
results are presented in
Text quality measure. We use the GPT score to measure the text quality, which is popular metric to reflect the text quality (Lee et al., 2023; Gilardi et al., 2023; Chen et al., 2024a). The prompt we used is adopted from Piet et al. (2023), and shown in Table 6. We use gpt-3.5-turbo-0125 to evaluate the results.
C. Ablation study
In this experiment, we perform ablation study to examine the impact of the parameter 7) in the watermark removal process, as defined in Equation 2. We select 1) E (1.0, 1.5, 2.0]. The results are summarized in Table 8. We observe that increasing the value of 17 significantly enhances the strength of watermark removal, effectively diminishing the watermark's 's presence in the generated text. However, this increase in removal strength comes at the cost of generation quality, as higher values of tend to introduce artifacts or reduce the coherence of the output. This trade-off between removal efficacy and output quality highlights the importance of tuning 7) to balance robustness in watermark removal with acceptable generation standards.
13

---

DE-MARK: Watermark Removal in Large Language Models
Table Additional experimental results on watermark removal task.
TPR @ FPR=
Entity Probing (short answer)
Median p-value 1
F1
10% +
1% 4
0.1 0.1%
Llama3.1 I 8B +Watermark
51.5%
18.5%
9.0%
8.07e-2
12.49
Llama3.1 8B +DE-MARK
14.5%
1.5%
0.0%
3.69e-1
12.03
Mistral 7B +Watermark
50.0%
13.5%
3.5%
9.94e-2
7.46
Mistral 7B +DE-MARK
16.0%
2.5%
0.5%
3.60e-1
7.64
Llama3.2 2 3B +Watermark
40.0%
15.5%
6.5%
2.03e-1
8.23
Llama3.2 3B +DE-MARK
8.0%
1.5 %
1.0%
5.00e-1
8.28
TPR FPR=
Concept Probing (short answer)
Median p-value
F1
10% L
1 1%
0.1 %
1 1 8B +Watermark
19.0%
11.5%
5.0%
2.82c-1
42.55
1 1 8B +DE-MARK
9.5%
3.0%
0.0%
5.00e-1
45.31
Mistral 7B +Watermark
41.5%
11.5%
2.0%
1.59e-1
35.95
Mistral 7B +DE-MARK
21.5%
0.5%
0.0%
5.00c-1
37.8
Llama3.2 3B +Watermark
14.0%
4.5%
1.5%
2.82e-1
33.50
Llama3.2 3B +DE-MARK
5.0%
2.5%
0.0%
5.00e-1
39.13
TPR @
FPR=
Long-form QA (long answer)
Median p-value f
Rouge-L
10%
1 1%
0.1 %
Llama3.1 1 8B +Watermark
98.5%
96.5%
91.0%
1.07e-10
23.99
Llama3.1 1 +DE-MARK
3.0%
0.5%
0.5%
1.14e-2
23.75
Mistral 7B + +Watermark
99.0%
95.5%
88.5%
7.66e-9
23.20
Mistral 7B +DE-MARK
13.0%
3.0%
1.5%
3.23e-2
23.18
Llama3 2 3B + +Watermark
99.5%
98.5%
93.0%
2.25e-10
23.74
Llama3.2 3B +DE-MARK
8.5
2.5%
0.0%
5.30e-2
23.94
Table 8. Ablation study for 77 (detined in Equation 2) in the watermark removal process.
MMW Story
@PR@FPR=
median p-valuet
GPT
0.1%
0.01%4
0.001 4
Scoret
Llama3.1 8B
98.96%
97.92%
92.71%
4.68c-10
90.71
+ Waterrark
+ours (),tt=11.0)
6.25%
2.08%
0.00% %
5.30e-2
90.29
+ours (),tt=11.5)
0.00%
0.00%
0.00% %
7.91c-1
90.40
+ours (ttaa)2.0)
0.00%
0.00%
0.00%
9.94e-1
90.02
Mistral 7B
98.96%
96.88%
92.71%
4.68e-10
93.13
+ +Watermark
+ours (\ett=110))
26.04%
11.46%
4.17%
7.66e-3
92.22
+ours (1,tt=11.5)
1.17
1.04%
0.00%
1.02e-1
92.78
+ours ()ett=2..0)
1.04% %
0.00% %
0.00 %
3.65e-1
92.32
1lama3.2 3B
97.92%
95.83%
92.71%
3.47e-10
89,29
+ +Watermark
tours (),tt=110))
17.71%
9.38%
4.17%
3.23e-2
88,20
+ours (5ett=11.5)
0.00 %
0.00 %
0.00 %
4.09e-1
88.01
+ours ((cta=2.0)
0.00% %
0.00 %
0.00 %
8.36e-1
86.84
14

---

DE-MARK: Watermark Removal in Large Language Models
Table Watermark exploitation
MMW Story
TPR FPR=
median p-value
GPT
0.1%f
0.01%t
0.001% %
score
No Watermark
0.00%
0.00%
0.00%
5.00e-2
88.71
+ours (gray box)
98.96%
87.50%
77.08%
5.43e-8
89.51
+ours (black box)
97.92%
88.54%
76.04%
6.18e-7
89.71
Table 10. Green list, red list identification performance for watermark removal. P: Precision, R: Recall
MMW Book Report
MMW Story
Dolly CW
Fl
Acc
P
Acc
F1
Acc
Llama3.1 8B
96.64
81.47
88,41
85.24
97.21
82.94
89.51
86.55
96,99
81.59
88.63
85.62
Mistral 7B
92.11
69.16
79.00
73.75
92.50
68.88
78.96
73.56
91.95
69.91
79.43
74.74
Ilama3.2 3B
96.26
76.77
85.42
81.19
96.03
76.19
84.96
80.85
96.08
76.32
85.07
81.05
D. Additional experimental results
We present additional experimental results that further elucidate the capabilities of DE-MARK in the contexts of watermark exploitation and red-green list identification accuracy. The results watermark exploitation are detailed in Table 9, while the results for red-green list identification are displayed in Table 10. These supplementary results reinforce the efficacy of DE-MARK.
We also evaluate Algorithm for its ability to accurately identify the prefix n-gram length h. We test values of (1,3, 5, 7} with consistency threshold set 0.8. For the black-box settings we use 100 samples estimate the token probability. The results presented for both gray-box (Figure 6) and black-box (Figure 7) settings. The findings indicate that when h E {3, 5, 7). the consistency rate drops below the threshold & 0.8 at the respective n-gram lengths, whereas for h = 1, the consistency rate consistently remains above 3 = 0.8. These results align well with the predictions of our algorithm, affirming its effectiveness.
Additionally, we evaluated the effectiveness of our method against two n-gram-based distortion-free watermarking schemes, y-reeeight (Hu al., 2023) and DiPmark (Wu et al., 2023b), as shown in Table 11 and Table 12. The results demonstrate that our proposed DE-MARK effectively removes both types of watermarks.
E. Proof of Theorem 4.1
Proof. Assume w.l.o.g. T₁ in the green list and T, in the red list. Based on Equation 1, the watermarked distribu-
tion Pw (T/ a. T;) and Pw (Tj x Ti, Tj) should be Pw (T, Tú, T5) = (0.5+0))=+(0664) and Pw (T, x., T;, =
(0.5+e)e
Prefix n-gram length h=1
Prefix n-gram length h=3
Prefix n-gram length h=5
Profix n-gram length h=7
1.0
1.0
Consislenicy rale
0.8
Consistancy rate
0.8
0.5
Consisteney rate
0.8
0.8
0.4
Model:
0.4
Model:
0.4
Model:
0.4
Model:
Llama3.1
Liama3.1 h-3
Llama3.1 8B,
Liam33.1
0.2
Mistral 78, h-1
0.2
Mistral 78, h-3
0.2
Mistral 78, h-5
0.2
Mistral 78, h-7
2
he3
Llama8 3B,
38,
0.0
2
3
5
7
8
1
2
7 Test prefi n-gram length
3
5
2
6
Test prefic n-gram length
Test prefix n-grem length
Test prefix n-gram length
Figure 6. Identification of the prefix n-gram length h, gray box.
15

---

DE-MARK: Watermark Removal in Large Language Models
Prefix n-gram length
n-gramm length he3
n-gram length
Prefix n-gram length
1.0
1,0
1.0
Corsidlency rayras
0.5
Consisbaney rate
0.9
Consislancy rele
Considlaneyrsl
0.5
0.8
0.8
0.8
Model:
Model:
Model:
Model:
Llama3.11 1 h-1
Liama3.11 3.1 8B, h-3
Laam38.1 8B,
Liamm3.11 88, h-7
02
0.2
Mistral h=3
02
Metral he5
0.2
7B,
Ulama822 2 3B, h-1
1ama3.22 38, h-3
Llama322 3B, h-5
Liamm3.2 38, h-7
prefix n-gram length
prefox n-gram length
Test prefix n-gram length
prefix n-gram length
Figure 7. Identification of the prefix n-gram length h, black box.
Table 11. Watermark removal results against rereehhght
MMW Book Report
Dolly
FPR
median, p-value
GPT score
@PR@FPR=
median, p-value
GPT
0.1%&
0.01%
0.001%
0.1%
0.01%
0.001
score
Llama3.1 8B +Watermark
44%
15%
7%
2.48e-3
94.72
42%
36%
29%
3.08e-3
93.70
+DE-MARK
5%
1%
1%
9.01e-2
94.51
25%
19%
17%
6.95e-2
92.45
1Jama3.2 3B Watermark
42%
20%
11%
2.48c-3
94.51
34%
23%
16%
7.75e-3
92.25
+DE-MARK
15%
7%
4%
2.94e-2
94.11
25%
17%
13%
5.29e-2
91.58
Table 12. Watermark removal results against DiPmark
MMW Book Report
Dolly CW
@
median, valuee
GPT score
TPR @ FPR=
median, p-value
GPT
1%
0.1%
0.01%J
1%1
0.1%%
0.01%
score
1lama3.1 8B #Watermark
44%
22%
11%
1.55e-2
94.58
43%
25%
17%
1.55e-2
92.68
+DE-MARK
3%
2%
0%
6.53e-1
94.75
5%
2%
0%
6.18c-1
92.34
Llama3.2 +Watermark
27%
13%
1.85e-2
94.24
39%
23%
18%
2.15e=2
91.98
+DE-MARK
2%
1 %
1%
4.46e-1
93.78
4%
3%
1 1%
5,48e-1
91.79
16

---

DE-MARK: Watermark Removal in Large Language Models
(0.5-6)
where is the noise. this case,
(0.5+eee* ((0.-)))((00---)))
T;, Pw (T, x Ti, TT)
0.5 + 6
E, [
E. [log
-6 8 + I. [log
0.5 }.
Since symmetrical distributed, have
0.5 +
0.5 +
0.5 -
E, [log
= B.g
+ log
= 0
0.5
0.5
0.5 +
Thus [.
F. Proof of Theorem 4.2
Proof. Since PR(t)= = Eier Pw Exc t E R, combining with the definition of Pw (Equation 1), we have
Pw(t)
P()
Pn(()= EieRien P(I) + EiER e P()) + EieG e P((() + Enerueg p())(()) Le R
analogously, we have
P(t)
PR(() = EIERIER P(I) + LIER,ee e0 P(I) + EIEG,EER e P(I) + Eied,ed e P(I) 1 tE G,
Denoted by A Eiehien P(t) + Eienre e P(t) EIEG,EE PP()) Eiecuee e P(t), we have
1 1|P(t), te and R:
I 1\P(1), te and G,
(PR(t) - P(t) =
IS 1|P(t), t R,
8 1I((t), ē G,
Since
A= Σ P(t)+ Σ e'P(t) + Σ c-5p(t) +
Σ 6 P(t)
BRRAER
IERIEG
IEG,EE
IEĆ.EEG
min c Σ P(t)+ Σ P(t)) +e- Σ P(1)+ Σ P())
(8)
BRRIER
IEGJEG
VERJEG
IEGJER
8 e 61) + e El 2 e
f1 te EE
and
= Σ P(I)+ +
E e'P(I) +
Σ e p(() +
Σ p-)p(())
LERIER
IEALEGG
(EG,IE
IEGJEG
S max(1,, 6-8
Σ P(() + E P(t)) + &el Σ P(I) + E P(t))
(9)
TERLEE
LEG,EEG
EERLEG
tEG.ER
< max 0-8 - =1) +e tea S E (1 61) + &
have
1
6-8
e 202
222
max
1/,
1 < max
1,1
A
A
1-
(1 C E 12
and
e
e
S-202
--2
max
11,
11
<
max
A
A
1-e1 El
(1 E1)
5
55-22- -2
17

---

DE-MARK: Watermark Removal in Large Language Models
It is easy to observe that max 1/, 1 66-6 1
< max - 11,1 11} since 6, 8,8> > 0.
Thus, denoted by fi(E1, 62) == max
222 1-61
- -1,1 (1-e1)+e" (ee
20g
} and 62) = max T-6i
8 202
(1 (1-e1)+e* 2201
TV(PR, Σ [PR(t) P(BI
5-8
Σ
P(t) +
Σ P(t)
A
&ER
TEL,IEG
-8
Σ
P(t)
+
Σ
P(t)
A
A
LEÑGIEG
NEG,IE
1
e
max
11,
I
Σ
P(t) +
Σ
P(t))
A
A
ERÑRER
1EG
(10)
6
3
+ +max
11
IK
Σ
P())
Σ P())
A
A
EERIEG
IEGRER
1
c
1 - 61) + max C
< max
11,1
1111
1
A
A
A
A
2222
222
<1 (1) max
1,1
61
(1- 61) + cd-cze1
542r2
2--262
+ E1 max
1-
(1 (1- E1) + e
= - (1)e)e(()))))))) + +.
G. Broader Impact
Machine learning models are transforming various sectors by improving efficiencies and tackling complex problems. Despite these beneficial impacts, there are valid concerns regarding the integrity and security of machine learning implementations (Wang et al., 2023d;c; Wu et al., 2022; 2023a;e; Mai et al., 2023b;a; Guo et al., 2024). In this context, watermarking has become a critical tool. It not only verifies the authenticity and ownership of digital media but also aids in identifying Al-generated content.
This paper contributes to the ongoing discussion by delving into advanced watermark removal techniques and introducing DE-MARK, comprehensive framework designed for both stealing and removing n-gram-based watermarks. Our method addresses shortcomings of prior research by eliminating the need for specific knowledge about the underlying hash functions and avoiding the use of paraphrasing tools that disrupt the original LM distribution. Instead, DE-MARKemploys querying strategy that allows for the accurate reconstruction of watermark parameters and ensures the integrity of the LM's output remains intact post-removal, offering a theoretical guarantee on the preservation of the original distribution.
While DE-MARKprovides robust tool against watermark tampering, its capabilities also introduce potential risks. The ability to reverse-engineer watermarking processes could be misused strip protective marks from proprietary sensitive content, raising ethical concerns about the security and control of digital information. Additionally, the adaptability of DE-MARKfor watermark exploitation- it can be used to generate watermarked content by mimicking original rules-further complicates the landscape of digital rights management.
18


Reviews:
[{"review_id": "zJs4kVja6z", "reviewer": "Reviewer_5v1A", "summary": "- The paper presents DE-MARK, a framework designed to remove n-gram-based watermarks from Large Language Models (LLMs). It introduces a novel querying strategy, \"random selection probing,\" to assess watermark strength and reconstruct watermarking parameters.\n- Unlike previous methods that rely on knowledge of the watermarking function or require paraphrasing, DE-MARK offers a general approach that estimates watermark parameters and reverses the watermarking effects while preserving the original language model’s distribution.\n- Beyond removal, DE-MARK can also be used to exploit watermarks by mimicking their structure, effectively generating watermarked text using an attacker LLM, which raises concerns about the security of watermarking scheme\n- The paper demonstrates the effectiveness of DE-MARK in both watermark removal and exploitation tasks on models such as Llama3 and ChatGPT, achieving a significant drop in watermark detectability without degrading text quality.\n\n\n### Nits and Prior Relevant Work Which has not been cited:\n\n- Line 102: However, The -> However, the", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Would DE-MARK still work against modern watermarking techniques (e.g. distortion-free methods like Gumbel watermark by Aaronson)? Is it specially designed for KGW?\n\nA1: Our proposed methods can be generalized to most n-gram-based methods, and we add additional experiments for two popular distortion-free n-gram-based watermarking algorithms to support our claim, $\\gamma$-reweight[1] and DiPmark[2]. Please see Table 13 and Table 14 in this [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub). The results show that our method can effectively remove the watermarks. \n\nHowever, it is not possible to remove the Gumbel watermark, as it only produces a one-hot probability distribution, i.e., assigning a probability of 1 to a single token and 0 to all others. This output lacks sufficient information to theoretically bound the difference between the original and post-removal distributions of the language model.\n\n> Q2: Analysis of query efficiency?\n\nA2: We first present a detailed time efficiency analysis to show that our time cost is acceptable, then we show some acceleration methods to further reduce the time cost.\n\n**Time Efficiency Analysis**\n\nThe experiments were conducted on a single RTX 6000 Ada GPU. Table 11 and Table 12 in this [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub) present the time efficiency of watermark stealing and removal with respect to varying numbers of queries.\n\nFor De-mark, we require $m(m - 1)$ queries per token. In our experiments, we set $m = 20$, resulting in a total of 380 queries per token. While this number may seem high, it is important to note that repeated querying is a necessary strategy to comprehensively gather the required information.\n\nDespite this, our experimental results indicate that the time cost is acceptable when compared to the baseline. The computation remains efficient due to the following reasons:\n\n- Each query is very short;\n- The query templates are identical across inputs, allowing for the reuse of most pre-computed key-value pairs, which significantly accelerates computation;\n\n**Acceleration method**\n\nFor watermark identification, the time cost is already low, and we find no additional optimization necessary.\n\nFor watermark removal and exploitation, we can achieve 2$\\times$ to 4$\\times$ speedup by randomly removing token pairs in Alg. 1. This introduces a trade-off between removal performance and inference speed. Additionally, increasing the value of $\\eta$ can yield improved outcomes, the results are presented in Table 15 in the [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub).\n\nA more sophisticated token pair clipping strategy may also lead to further performance improvements.\n\n> Q3: The paper does not extensively evaluate settings where log probabilities are entirely unavailable (a common scenario for many commercial LLM APIs).\n\nA3: Regarding watermark identification and exploitation, we kindly point out that we have conducted extensive experiments in the black-box setting (i.e., where log probabilities are entirely unavailable), as shown in Tables 2, 3, and 9, and Figures 4, 5, and 7.\n\nAs for watermark removal, as discussed in Section 3.1, when log probabilities are completely unavailable, it is impossible to bound the gap between the original and post-removal distributions of the language model. This is because the output provides only a single token, which is insufficient to recover the full distribution.\n\n> Q4: The assumption about Gaussian noise also seems like a strong and arbitrary assumption\n\nA4: Thank you for pointing this out. Actually, in Appendix E. Proof of Theorem 4.1, the gaussian noise assumption is not necessary for the proof, we only require the noise to be symmetrically distributed (i.e., $P(\\epsilon=x)=P(\\epsilon=-x),\\forall x \\in \\mathbb{R}$), which is a mild and intuitive assumption. We will update the assumption accordingly in our revision.\n\n> Q5: How fast does the estimate converge? How many queries are required to decode a watermark\n\nA5: Demark has relatively low computational cost, which enables us to even increase the number of queries for more accurate estimations. Please refer to Figure 8 in this [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub) for the convergence speed of estimating $\\delta$. We evaluate the performance using different values of $m$ in Algorithm 4 and vary the parameter $c$ (set to 1, 2, 5, 10, 20, 50, and 100) to generate different numbers of queries.\n\n[1] Unbiased Watermark for Large Language Models, Hu et al., ICLR 2024  \n[2] DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models, Wu et al., ICML 2024", "claims_and_evidence": "- The paper demonstrates that DE-MARK significantly reduces watermark detectability in models like Llama3 and Mistral. After applying DE-MARK, the true positive rate (TPR) of watermark detection drops from over 60-90% to below 20%, even at low false positive rates. The authors also provide theoretical guarantees that the post-removal distribution remains close to the original model distribution. (Table 1)\n- Unlike previous methods that require access to the underlying hash function or paraphrasing tools, DE-MARK estimates watermark parameters (like red-green token lists and watermark strength) through a novel querying strategy called random selection probing. This approach allows it to infer the watermark without direct access to its configuration.\n- The paper shows that after learning the watermarking pattern, an adversary can generate text that mimics a watermarked LLM. Experiments on watermark exploitation confirm that DE-MARK can recreate watermarked content with high accuracy, making it possible to generate fake watermarked text.\n- The paper uses the GPT score as a metric to assess text quality before and after watermark removal. Results show that even after applying DE-MARK, the GPT score remains nearly the same, indicating that the method does not degrade fluency, coherence, or correctness in the generated text.", "methods_and_evaluation_criteria": "- The paper does a good job of explaining why watermark removal is an important and underexplored area. The motivation is clear, and the threat model is well-defined.\n- Unlike previous work that relies on paraphrasing or explicit knowledge of the watermarking function, DE-MARK estimates the watermark structure through a statistical approach. The authors also provide a theoretical bound on the difference between the original and post-removal distributions, which strengthens the credibility of their method.\n- The authors evaluate DE-MARK on multiple models, including Llama3 and Mistral, and even apply it to ChatGPT in a real-world case study. The variety of models tested helps support the claim that DE-MARK generalizes well across different LLMs\n- The evaluation considers both watermark detectability (TPR@FPR) and text quality (GPT score). This shows not only how well DE-MARK removes watermarks but also that it does not degrade output quality.", "theoretical_claims": "- The authors derive an upper bound on the total variation distance between the original and post-removal language model distributions. This is an important result because it provides a theoretical guarantee that DE-MARK does not significantly alter the statistical properties of the model’s output.\n- The bound explicitly depends on two key error terms: (1) misclassification error $(\\epsilon_1)$, which accounts for incorrectly identified watermark tokens, and (2) $estimation error (\\epsilon_2)$, which quantifies how accurately DE-MARK estimates the watermark strength.\n- The proof in the appendix looks correct based on working out the steps myself though more details could be added to make it easier to follow.", "experimental_designs_or_analyses": "- The paper evaluates DE-MARK on multiple language models, including Llama3 (3B & 8B), Mistral 7B, and ChatGPT.\n- This variety ensures that the method is not tied to a specific model architecture and can generalize across different LLMs.\n- The authors measure watermark detectability before and after applying DE-MARK using true positive rate (TPR) at fixed false positive rates (FPR).\n- Results show that after DE-MARK is applied, watermark detectability drops significantly, indicating successful removal.\n- The paper also evaluates the impact of DE-MARK on text quality using GPT-score, showing that the removal process does not significantly degrade generation quality.", "relation_to_broader_scientific_literature": "1. The paper builds on prior work in statistical watermarking for LLMs (e.g., Kirchenbauer et al. (2023)), which proposed methods to embed watermarks into AI-generated text for provenance tracking.\n2. It directly addresses recent security concerns raised by Jovanović et al. (2024) on the feasibility of watermark stealing and removal attacks.\n3. By providing a provable theoretical bound on watermark removal effectiveness, DE-MARK contributes a formal framework to this discussion, which is a notable improvement over purely empirical attack demonstrations.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "- The paper focuses on n-gram-based watermarks, but modern watermarking techniques are evolving. Would DE-MARK still work against semantic watermarks or distortion-free watermarking? Even if not tested, discussing this in the limitations section would make the paper stronger.\n- The method requires multiple queries to reconstruct the watermarking scheme, which could be computationally expensive. It would be helpful if the paper included an analysis of query efficiency—how many queries are typically needed to achieve successful watermark removal? Could optimizations reduce this cost?\n- While DE-MARK is tested on ChatGPT, the paper does not extensively evaluate settings where log probabilities are entirely unavailable (a common scenario for many commercial LLM APIs).\n- The assumption about gaussian noise also seems like a strong and arbitrary assumption to me without any empirical evidence.\n- In addition to the unbiasedness of the estimate it would also be helpful to do a consistency analysis of the estimator i.e. how fast does the estimate converge? How many queries are required to decode a watermark and the practical feasibility of DEMARK.", "other_comments_or_suggestions": "- Addressing practical concerns I raised above such as scalability, robustness to different watermarking techniques, and real-world constraints (e.g., no probability access) would make the paper even stronger. If these aspects were explored further, the work would have a broader impact and provide even more value to the AI research community.", "questions_for_authors": "One thing that isn’t super clear to me is that it seems this watermark removal assumes a KGW watermarking scheme. There are many more watermarking schemes which are also n-gram watermarking schemes but don't explicitly add a delta to the logit params and also have distortion free property such as the Gumbel watermark by Aaronson. Would this watermark removal scheme also work on that? If not, I suggest authors to tone down the claims in the paper. Open to more discussion on this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NV90wE4aPs", "reviewer": "Reviewer_eS2n", "summary": "## Summary\n\nThis paper addresses critical vulnerabilities in n-gram watermarking schemes for language models (LMs). The authors propose **DE-MARK**, a framework for watermark removal and exploitation, with three key contributions:\n\n1. **Watermark Parameter Estimation**  \n   Introduces _random selection probing_ to reconstruct red-green token lists and estimate watermark strength \\\\(\\delta\\\\) without prior knowledge of hash functions or n-gram parameters. Core algorithms include:  \n   - Relative probability ratio matrix\n   - Context length detection via score consistency\n   - Provable unbiased \\\\(\\delta\\\\) estimation\n\n2. **Distribution-Preserving Removal**  \n   Formulates watermark removal as probability reweighting\n   Theoretical guarantees bound the total variation distance between post-removal (\\\\(P_R\\\\)) and original (\\\\(P\\\\)) distributions\n\n3. **Empirical Validation**  \n   Tests on Llama3, Mistral, and ChatGPT show:  \n   - **Removal**: Reduces TPR@FPR=0.1% from >89% to <15% while preserving GPT quality scores (\\\\(\\Delta<0.5\\%\\\\))  \n   - **Exploitation**: Achieves 93% TPR@FPR=0.1% when implanting stolen watermarks  \n   - **Black-box Adaptation**: 85% precision in red-green list detection with 10 samples\n\nThe work demonstrates fundamental limitations in current n-gram watermark robustness and provides tools for security auditing of LM watermarking systems.\n\n## Update after Rebuttal\nI found that the authors have adequately solved most of my concerns, and I recognize the contribution of the paper and would keep my rating as \"weak accept\".", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and thoughtful suggestions, which have greatly helped improve the quality of our work.\n\n> Q1 The experimental results don't include variance measures or statistical significance tests across multiple runs.\n\nA1: Firstly, evaluating whether a sentence is watermarked does not require running multiple statistical significance tests because the watermark detection algorithm itself is deterministic and already employs a statistical hypothesis test [1]. Specifically, during detection, we set a theoretical false positive rate (e.g., 0.1%), calculate the corresponding statistical threshold, and then flag the sentence as watermarked if its score exceeds this threshold. To the best of our knowledge, most watermarking papers[1,2,3,4] do not perform multiple-run tests, and we have adopted similar settings. \n\nMoreover, we provide results across multiple datasets and LLMs to mitigate the effects of randomness.\n\n\n> Q2 Missing computational efficiency analysis/analysis of the number of queries.\n\nA2: Please refer to Response A2 in our rebuttal to reviewer 5v1A for a detailed discussion on query efficiency.\n\n> Q3 Some formulas need to be polished in formats. Some references should be referred to correctly\n\nThank you for your valuable suggestion. We sincerely apologize for the mistake. We will revise the formatting and correct the references accordingly in the version.\n\n> Q4 In section 4, the authors should point out that detailed proofs of theorems are in Appendix XXX.\n\nWe apologize for the confusion. In the revision, we will clearly link the proofs of the theorems within the main body of the text.\n\n> Q5 Comparision against Attacking llm watermarks by exploiting their strengths\n\nThe setting of our work fundamentally differs from that of Pang, Qi, et al. [5], which represents an early exploration of watermark removal under relatively simple assumptions. Specifically, their attacks are limited to the following scenarios:\n\n- Naive token editing (Sec. 4): This method inevitably leads to degraded text quality.\n\n- Multi-key scenario (Sec. 5): This is a trivial case where a simple averaging technique suffices, in contrast to our focus on the more practical and challenging one-key scenario.\n\n- Detector-available scenario (Sec. 6): Prior studies, including this one, have shown that attacks in this setting are relatively easy. In contrast, our work addresses the more difficult detector-unavailable scenario (D0 setting, detailed in Sec. 3.1, Lines [138–143]).\n\nTherefore, we believe a direct comparison would be neither meaningful nor informative.\n\n> Q6 I would like to see a detailed comparison between the selection of h\n\nWe kindly note that additional results regarding the selection of $h$ are provided in the Appendix Figure 6 and Figure 7.\n\nIn response to the recommendation, we have also included results for identifying the prefix n-gram length with $h$ = 2, 4, 6 and 8 in Figures 9–10, available via this [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub).\n\nThe results show that our method generalizes well across different selections of $h$.\n\n> Q7 Theorem 4.2 provides bounds on distribution gaps, but there's no experimental validation of how tight these bounds are in practice. Can you provide empirical measurements of actual distribution gaps compared to the theoretical bounds?\n\nWe empirically evaluated our method on 3,000 token distributions from the MMW Book Report dataset using the LLaMA3.2 3B model. Our results show that $\\mathbb{E}[\\frac{\\mathbb{TV}(P_R,P)}{\\epsilon_1 f_2(\\epsilon_1,\\epsilon_2)+ (1-\\epsilon_1) f_1(\\epsilon_1,\\epsilon_2)}]=0.328$ indicating that the distribution gap is well-controlled by the bound.\n\nMoreover, our GPT score evaluation results in Table 1 and Table 2 further confirm that the generated text remains highly consistent with the original in distribution, demonstrating minimal distribution distortion and strong preservation of text quality.\n\n\n[1] A watermark for large language models, Kirchenbauer et al., ICML 2023  \n[2] Robust Distortion-free Watermarks for Language Models, Kuditipudi et al., TMLR 2023  \n[3] Unbiased Watermark for Large Language Models, Hu et al., ICLR 2024  \n[4] DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models, Wu et al., ICML 2024\n[5] Attacking LLM Watermarks by Exploiting Their Strengths, Pang et al, ICLR 2024 Workshop", "claims_and_evidence": "**Well-supported claims:**\n\nThe paper provides detailed descriptions of the DE-MARK framework through five well-formulated algorithms covering relative probability calculation, token scoring, n-gram length identification, watermark strength estimation, and green list identification.\n\nDE-MARK significantly reduces watermark detectability (TPR@FPR) from high levels (>60%) to low levels (<20%) while maintaining text quality as measured by GPT scores.\n\n**Weaknesses in evidence:**\n\nThe experimental results don't include variance measures or statistical significance tests across multiple runs.", "methods_and_evaluation_criteria": "The paper's approach is methodologically sound:\n- Random selection probing efficiently estimates token probabilities\n- Five sequential algorithms form a coherent framework for watermark removal\n- Theoretical guarantees establish bounds on distribution gaps\n\n## Evaluation Criteria Assessment\n\n- Diverse models (Llama3.1-8B, Mistral-7B, Llama3.2-3B) and datasets (Dolly, MMW)\n- Appropriate metrics (TPR@FPR, p-values, GPT scores)\n\n**Weaknesses:**\n- Missing computational efficiency analysis\n- Lack of statistical significance testing", "theoretical_claims": "I have checked the theoretical claims of the paper, especially theorems in section 4, which are sound.", "experimental_designs_or_analyses": "The experiment settings and dataset usage are sound but need computational efficiency analysis and statistical significance testing.", "relation_to_broader_scientific_literature": "The contribution of this paper is unique compared to broader scientific literature, which are introduced in the related work part.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nThe setting of this paper is strong enough, detector-free scenarios are genuinely in the real world.\n\nThe theorems and proofs are sound, making the paper good enough.\n\nWeaknesses:\n\nI would like to see the analysis of the number of queries.\n\nSome formulas need to be polished in formats.\n\nSome references should be referred to correctly, e.g., `Watermark stealing in large language models`  should be in ICML.\n\nIn section 4, the authors should point out that detailed proofs of theorems are in Appendix XXX.", "other_comments_or_suggestions": "I have known that the setting of this method is stronger than priors, but I also want to see the performance comparisons with others, e.g., `Attacking llm watermarks by exploiting their strengths`, which could make this paper more sound.", "questions_for_authors": "In section 5.2, why only consider $h=\\{3,7\\}$? In normal practices, $h=2$ is normally used, and I would like to see a detailed comparison between the selection of $h$.\n\nTheorem 4.2 provides bounds on distribution gaps, but there's no experimental validation of how tight these bounds are in practice. Can you provide empirical measurements of actual distribution gaps compared to the theoretical bounds?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iwx7W2qtlV", "reviewer": "Reviewer_kzSZ", "summary": "DE-MARK presents a framework for removing n-gram-based watermarks, specifically targeting the soft watermarking scheme proposed by Kirchenbauer et al. (2023a). The method utilizes a novel querying strategy called \"random selection probing\" to estimate watermark parameters like strength and red-green lists. The paper claims theoretical guarantees for distribution preservation and demonstrates the efficacy of DE-MARK on models like Llama3 and ChatGPT in watermark removal and exploitation tasks. The core contribution is a practical approach to reverse-engineer and remove a specific type of statistical watermark without prior knowledge of watermark parameters.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback, which has been invaluable in refining our manuscript. Below, we provide detailed responses to each of your comments:\n\n> Q1: Firstly, the attack is primarily focused on a specific soft watermark (Kirchenbauer et al., 2023a). This type of watermark, as demonstrated by Gu et al. (2024) in \"On the Learnability of Watermarks for Language Models\" (ICLR 2024), is known to be relatively easily learned. While the paper introduces refined techniques to estimate the hyperparameters of this soft watermark, the overall contribution in addressing a fundamentally weak watermark is arguably incremental. Secondly, the paper's scope does not extend to other watermarking methods, such as advanced cryptographic watermarks, which are designed to be theoretically resistant to parameter learning. Examples like \"Undetectable watermarks for language models\" highlight this gap.\n\nA1: Our method is specifically designed for n-gram-based approaches, which are widely employed in watermarking[1,2,3,4]. Consequently, it can be naturally extended to other n-gram watermarking methods.\n\nTo support our claim, we extend our method on two additional n-gram-based advanced distortion-free watermarking: $\\gamma$-reweight[3] and DiPmark[4], the results are presented in Table 13 and 14 in this [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub), which demonstrate Demark’s strong generalization capabilities beyond just the KGW framework.\n\nWe are not able to evaluate our attack method on Undetectable watermarks because: a) it is not n-gram-based, b) the vocabulary size of the LM in their paper is only 2, and it is not trivial to extend their method to models with larger vocabulary size c) it’s a pure theoretical paper without any code implementation.\n\n\n\n> Q2: Furthermore, the second contribution regarding industry-scale applicability is questionable. It is unknown whether ChatGPT employs watermarking, and if so, which specific technique. Simulating a soft watermark on top-20 tokens and demonstrating its removal with DE-MARK does not convincingly demonstrate effectiveness against a real-world, industry-level watermarking implementation.\n\n\nA2: Whether ChatGPT employs a watermark by default is irrelevant to our study. In our experiments, we manually embedded a soft watermark into ChatGPT outputs and successfully removed it using Demark. Moreover, as noted in various prior works [1,2,3,4], the choice of LLM has minimal impact on the effectiveness of watermarking techniques. Our extensive results across LLaMA 3B, LLaMA 8B, and Mistral 7B further confirm that Demark is robust to the choice of underlying LLM. Finally, both Reviewer 5v1A and eS2n acknowledged that our case study on ChatGPT is informative and valuable.\n\n\n> Q3: Therefore, it is recommended that the authors revise the title to accurately reflect the limited scope of their method, specifically its applicability to soft n-gram watermarks. The current title is overly broad and implies a capability to remove a wider range of watermarks than is actually demonstrated.\n\n\nA3: Our method is designed for n-gram-based approaches and is applicable beyond just soft n-gram watermarks, as demonstrated by our experiments in Tables 13 and 14, available in this [anonymous link](https://docs.google.com/document/d/e/2PACX-1vSfxtMpq2yL7QjOjW0NNWgI_J4LG9QHes7eBtj4P7LqdrIVBTuibloz0p0LLG5dhijwS7UhFcVfw537/pub). In response to the reviewer's recommendation, we will revise the title to emphasize our focus on n-gram watermarking.\n\nWe would be more than happy to further discuss any additional questions or concerns you may have.\n\n\n[1] A watermark for large language models, Kirchenbauer et al., ICML 2023  \n[2] On the reliability of watermarks for large language models, Kirchenbauer et al., ICRL 2024  \n[3] Unbiased Watermark for Large Language Models, Hu et al., ICLR 2024  \n[4] DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models, Wu et al., ICML 2024", "claims_and_evidence": "The contribution of this work appears limited. \n\nFirstly, the attack is primarily focused on a specific soft watermark (Kirchenbauer et al., 2023a). This type of watermark, as demonstrated by Gu et al. (2024) in \"On the Learnability of Watermarks for Language Models\" (ICLR 2024), is known to be relatively easily learned. While the paper introduces refined techniques to estimate the hyperparameters of this soft watermark, the overall contribution in addressing a fundamentally weak watermark is arguably incremental. Secondly, the paper's scope does not extend to other watermarking methods, such as advanced cryptographic watermarks, which are designed to be theoretically resistant to parameter learning. Examples like \"Undetectable watermarks for language models\" highlight this gap.\n\nFurthermore, the second contribution regarding industry-scale applicability is questionable. It is unknown whether ChatGPT employs watermarking, and if so, which specific technique. Simulating a soft watermark on top-20 tokens and demonstrating its removal with DE-MARK does not convincingly demonstrate effectiveness against a real-world, industry-level watermarking implementation.\n\nTherefore, it is recommended that the authors revise the title to accurately reflect the limited scope of their method, specifically its applicability to soft n-gram watermarks. The current title is overly broad and implies a capability to remove a wider range of watermarks than is actually demonstrated.", "methods_and_evaluation_criteria": "As state in \"Claims And Evidence\"", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Related", "essential_references_not_discussed": "Gu et al. (2024) \"On the Learnability of Watermarks for Language Models\" (ICLR 2024)", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes DE-MARK, a method to remove n-gram-based LLM watermarks. The key idea is based on a novel querying technique, random selection probing, to estimate watermark configuration parameters, including watermark strength and green-red token lists.  The paper claims theoretical guarantees for distribution preservation and demonstrates the efficacy of DE-MARK on models like Llama3 and ChatGPT in watermark removal and exploitation tasks. 

Strength of the paper: 
- The paper develops random selection probing technique to estimate watermark configuration parameters without prior knowledge. 
- The paper demonstrates that DE-MARK significantly reduces watermark detectability in models like Llama3 and Mistral. After applying DE-MARK, the true positive rate (TPR) of watermark detection drops from over 60-90% to below 20%.
- The paper provides a theoretical bound on the effectiveness of watermark removal. 

Weakness of the paper: 
- There is already a prior study showing n-gram watermarks such as KGW is weak against attacks. 
- Some experimental settings are weak, i.e.g simulating a soft watermark on top-20 tokens and demonstrating its removal with DE-MARK.
- Analysis of complexity and the number of probes is missing. 
- The use of GPT to evaluate the generation quality is questionable. It is better to use a model-based metric such as COMET, BERTScore, MetricX, SEScore.
Decision: Accept (poster)