Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks
Authors: Yixin Cheng
Hongcheng Guo
Yangming Li
Leonid Sigal
Venue: ICML 2025 poster (2025)
Keywords: LLM watermark
robustness
privacy  and societal considerations
Primary area: deep_learning->large_language_models
TL;DR: A Approach to attack LLM Watermarks by find the green token with self-information

Abstract: Text watermarking aims to subtly embeds statistical signals into text by controlling the Large Language Model (LLM)'s sampling process, enabling watermark detectors to verify that the output was generated by the specified model. The robustness of these watermarking algorithms has become a key factor in evaluating their effectiveness. Current text watermarking algorithms embed watermarks in high-entropy tokens to ensure text quality. In this paper, we reveal that this seemingly benign design can be exploited by attackers, posing a significant risk to the robustness of the watermark. We introduce a generic efficient paraphrasing attack, the Self-Information Rewrite Attack (SIRA), which leverages the vulnerability by calculating the self-information of each token to identify potential pattern tokens and perform targeted attack. Our work exposes a widely prevalent vulnerability in current watermarking algorithms. The experimental results show SIRA achieves nearly 100\% attack success rates on seven recent watermarking methods with only \$0.88 per million tokens cost. Our approach does not require any access to the watermark algorithms or the watermarked LLM and can seamlessly transfer to any LLM as the attack model even mobile-level models. Our findings highlight the urgent need for more robust watermarking.

Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks
Yixin Cheng" Hongcheng Guo
Yangming Li+ Leonid Sigal 1256
Abstract
demonstrated remarkable capabilities in generating coher- ent, human-like text. However, while these advances sig- nificantly expand Al's potential in content creation, they have concurrently heightened concerns regarding their mis- use (Deshpande et al., 2023; Wang et al., 2024), including the spread of misinformation (Monteith et al., 2024) and threats academic integrity (Stokel-Walker, 2022).
Text watermarking aims to subtly embeds statis- tical signals into text by controlling the Large Language Model (LLLM)'s sampling process, en- abling watermark detectors to verify that the out- put was generated by the specified model. The robustness of these watermarking algorithms has become key factor in evaluating their effective- ness. Current text watermarking algorithms em- bed watermarks in high-entropy tokens to ensure text quality. this paper, we reveal that this seemingly benign design can be exploited by at- tackers, posing significant risk to the robustness of the watermark. We introduce generic effi- cient paraphrasing attack, the Self- Information Rewrite Attack (SIRA), which leverages the vul- nerability by calculating the self-information of each token to identify potential pattern tokens and perform targeted attack. Our work exposes widely prevalent vulnerability in current wa- termarking algorithms. The experimental results show SIRA achieves nearly 100% attack success rates on seven recent watermarking methods with only $0.88 per million tokens cost. Our approach does not require any access to the watermark al- gorithms or the watermarked L.LM and can seam- lessly transfer to any LLM as the attack model even mobile-level models. Our findings highlight the urgent need for more robust watermarking. The source code is available at SIRA.
To mitigate risks associated with LLM-generated content, text watermarking has emerged as promising counter- measure (Kirchenbauer et al., 2023; Aaronson & Kirchner, 2022). This technique subtly alters the L.M'' generation process to embed imperceptible patterns in the output text, the pattern is invisible to human readers and can be reliably detected using specialized algorithms. This generate- detect framework enables the differentiation between Al Al-generated and human-authored content and allows tracking of the text back to the specific LLM that generates the text (Li et al., 2024). Consequently, this mechanism promotes account- ability and helps mitigate LLM misuse, providing a reliable means to ensure transparency and integrity Al-generated content.
Recent studies have demonstrated that watermarking tech- niques exhibit significant robustness against simple manip- ulations, including word deletions (Welbl et al., 2020) and emoji attacks (Kirchenbauer et al., 2023). However, tra- ditional NLP attack strategies, such as word deletion and insertion, are increasingly insufficient for thoroughly evalu- ating the robustness of advanced watermarking algorithms. As LLMs continue advance, there is growing need for more sophisticated testing methodologies that account for complex manipulation tactics, ensuring that watermarking techniques remain resilient against emerging threats.
1. Introduction
To provide a more rigorous evaluation of watermarking ro- bustness, paraphrasing attacks have been proposed. Despite their potential, these approaches face several limitations. First, current paraphrasing attacks rely on a naive and brute- force approach, where they simply instruct LLMs to rewrite watermarked text. This process is both inefficient and incon- sistent in its results. The modifications to text are untargeted and random, dictated by the LLM, often leaving portions of the watermark intact causing the attack to fail. For newer and more robust watermarking algorithms like SIR (Liu
Large language models (LLMs), exemplified by Chat- GPT (OpenAI, 2024) and Claude (Anthropic, 2024), have
'University of British Columbia VVector Institute for AI FFuda University "University of Cambridge 'Canada CIFAR AI Chair ONSERC CRC Chair. Correspondence to: Yixin Cheng <yix- inch@cs.ubc.cal
Proceedings of the 42 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Self-Information Rewrite Attacks on Text Watermarking
al., 2024), these methods already fail deliver effective attacks, making them unsuitable robustness evaluation methods for future research. Moreover, changing the words that do not embed the watermark may cause change in semantics, or lead to decline in text quality. Second, cur- rent methods require significant hardware resources and costs, as they often rely on large-scale LLMs to achieve no- table attack performance which could be a barrier for future study. Thirdly, they are con-transferable, as the reliance specifically fine-tuned LLM (Krishna et al., 2024) prevents them from effectively leveraging the capabilities of more powerful, rapidly emerging language models for attacks.
the rarity of an event to the amount of information it com- municates.
In the realm of Natural Language Processing (NLP), self- information plays crucial role in the analysis and modeling of language. It aids in deciphering language patterns, partic- ularly in evaluating the entropy and predictability of tokens within sequences. The concept is particularly useful for quantifying the informativeness or surprise of a token in a given linguistic context. Language models predict the prob- ability of subsequent token in sequence using the preced- ing context P(ts t1, 122....,t-111 The self- colf-information of the token in this context is computed as follows:
To address these challenges, we propose a new paraphrasing attack named SIRA (Self-Information Rewrite Attack). Our approach not only introduces a more effective paraphras- ing strategy but also reveals fundamental vulnerability in current watermark algorithms. Specifically, watermarking techniques aim to be imperceptible to users while maintain- ing text quality and semantics intact, which necessitates embedding patterns in high-entropy tokens (Kirchenbauer et al., 2023; Liu et al., 2023). These high-entropy tokens also exhibit high self-information within the given text con- text. Leveraging this otherwise harmless watermark feature, SIRA could identify potential "green list" token candidates within watermarked text without any prior knowledge. By masking potential green tokens, we can transform the un- targeted paraphrasing into a targeted fill-in-hee-llan task, achieving a stronger and more efficient attack.
1(tx I t1,12,..... (x-11 - logo (P(tx) f1, (x-1))
where 1(4x (1, .,,22..,,,--1 denotes the self-information of token ty given the context of previous tokens, P(tx 1,, 11, (42 is the probability of token 4x occurring after the preceding sequence of tokens, and b represents the base of the logarithm.
LLM Watermark. Watermarking techniques for large lan- guage models are designed to embed identifiable patterns in model outputs, allowing for the traceability of generated text back to its originating source. These watermarks serve as an essential tool for ensuring accountability and ownership, par- ticularly in scenarios where identifying the specific model or version that produced the content is crucial. LLM watermark methods can be broadly classified into two primary cate- gories: the KGW Family and the Christ Family. Each family employs distinct mechanisms that are integral to the internal workings of LLMs. The KGW Family (Kirchenbauer et al., 2023; Liu et al., 2023; Zhao et al., 2023; Wu et al., 2024; Lu et al., 2024) focuses on modifying the logits- the raw output probabilities produced by the are transformed into text. This approach involves selec- tively adding bias to certain tokens, referred to as "green list" tokens, which influences the model to favor these to- kens, thus embedding statistical signature in the output. Post text generation, a statistical metric based on the propor- tion of these "green" tokens is computed. A predetermined threshold enables differentiation between watermarked and non- woon-watrmarred text.
In summary, we make the following contributions:
We formalize and thoroughly investigate the threat model of LLM watermark black-box paraphrasing, dis- tinguishing it from other watermark attacks that rely on probing-based modeling or online attack methods.
they
We reveal widely existing vulnerability in watermark algorithms and propose the first, to our knowledge, targeted paraphrasing attack. This attack is easy to implement and transferable, making it well-suited for future robustness evaluations.
We comprehensively study paraphrasing attacks re- cent watermark algorithms. For newly proposed water- marking algorithms, we show that existing paraphras- ing attacks are no longer sufficient to verify robustness.
Conversely, the Christ Family (Aaronson & Kirchner, 2022: Christ et al., 2024; Kuditipudi et al., 2023) modifies the sampling process during the generation phase itself. Rather than altering the logits, this family intervenes directly in the token selection during decoding. Techniques such top- k sampling, temperature adjustment, or nucleus sampling are adapted to ensure preferential selection of watermarked tokens. This method provides more direct control over the generation process, embedding watermarks that are resilient against post- prot-processing attacks, such as paraphrasing.
2. Related Work
Self-information.
Self-information also known as sur- prisal, is a fundamental concept in Information Theory, first introduced by Claude Shannon in his seminal work (Shan- non, 1948). Shannon employed self self-information as the prin- cipal metric to quantify the information content associated with the occurrence of specific events, effectively linking
2

---

Self-Information Rewrite Attacks on Text Watermarking
Watermark Removal Attacks. The robustness of a water- marking algorithm is crucial, as it determines the effective- ness of the watermark under various real-world conditions, particularly in adversarial settings. Attacks against water- mark al algorithms, commonly referred to as watermark tam- pering attacks, can be broadly categorized into two types:
ness evaluations of watermarking algorithms Kirchenbauer et al. (2023); Liu et al. (2024); Aaronson & Kirchner (2022); Zhao et al. (2023). Due to the different threat models and much stronger assumptions, we do not include them in this paper.
Model-based paraphrasing: A more common and advance form of attack involves using another LLM to paraphrase the watermarked content. This method differs from watermark- stealing attacks by operating in a strict black box setting, where the attacker's knowledge is limited to the water- marked text. Krishna et al. (2024) propose DIPPER, a paraphrase generation model developed by fine-tuning T5- XXL (Raffel et al., 2020b) on an aligned paragraph dataset. This model has been widely adopted in recent watermark research (Zhao et al., 2023; Liu et al., 2024; Kuditipudi et al., 2023) to evaluate the robustness of watermarking al- gorithms. GPT Paraphraser is another model-based method, which utilizes the GPT model as a paraphrase. Previous ap- proaches in this category derive their effectiveness primarily from two mechanisms: (1) paraphrased text especially new generated portion dilutes the strength of the original wa- termark by introducing new linguistic structures, and (2) lexical variations introduced during paraphrasing could in- cidentally alter subset of green tokens. However, such changes that are controlled by an LLM, thus occurring by chance, leads to untargeted paraphrasing that is neither suffi- cient nor effective. In contrast, SIRA is a targeted approach that selectively replaces potential green tokens in the wa- termarked text, creating "neutral" template for rewriting. It transforms the paraphrasing task into a fill-in-the-blank problem.
Text manipulations:
These attacks involve traditional NLP techniques to straightforward text manipulations, such as word deletion (Welbl et al., 2020), substitution (Yu al., 2010), or insertion (Kirchenbauer et al., 2023). By altering the word-level structure of the text, these methods attempt to distort or eliminate the watermark. These techniques disrupt the statistical pattern of the watermark by adding, removing, or replacing words. Kirchenbauer et al. (2023) propose emoji attack and copy-paste attack which insert emoji/human written text in the generated text avoid de- tection. These methods are considered variants of text ma- nipulations, however, they are easily thwarted by detectors equipped with content filters and often alter the semantics of the generated text which makes them inappropriate for real-world use.
Informed watermark attack: Such attacks rely on the ad- versary having specific knowledge access to the water- marking system, such as the ability to probe the watermark model. One typical attack is the watermark-stealing at- tack (Jovanović et al., 2024; Wu & Chandrasekaran, 2024). These methods aim to reverse-engineer or approximate the embedded watermark by probing watermark algorithms with queries. Attackers construct the watermark distribu- tion of the target model and estimate whether each token complies with the watermark rules based on the context by issuing a large number of dreddeedgeed prefix queries. However, these methods assume that the adversary has un- limited access to the targeted watermarked LLM model. Additionally, this method becomes ineffective when water- marks employ dynamic strategies (c.g.. using a set of hash keys). Another notable recent attack is the random walk attack (Zhang et al., 2023); this method iteratively perturbs the watermarked text using multiple models, relying on the detector's feedback as the termination condition. While it effectively removes watermarks, its iterative nature intro- duces significant computational and time overhead. Another major limitation of this approach is it does not guarantee any semantic preservation.
Our proposed SIRA falls under the category of model-based paraphrasing attacks and offers several key advantages: (1) High Attack Effectiveness: SIRA achieves near 100% suc- cess rates across seven tested watermarking algorithms: (2) Lightweight and Highly Transferable: Unlike DIPPER, which depends a specific fine-tuned model, SIRA oper- ates effectively on mobile-level 3B models and can seam- lessly adapt to future powerful model; (3) Minimal As- sumptions: It requires no access to the watermarked LLM or its detector, enabling robust performance in black-box sce- narios; (4) Low Cost and and-aly:: SIRA operates with minimal computational overhead (costing only$0.88 per Million tokens). Unlike watermark stealing requiring large samples of watermarked text to initial one piece text attack, SIRA is ready to use out of the box. These advan- tages position SIRA as a promising tool for evaluating the robustness of LLM watermarking in future research.
The above methods share strong assumption: the attacker has access to the watermarked LLM,, either with or with- out the watermark detector. However, this assumption is rarely met in real-world adversarial scenarios especially the access to the watermark detector. The aforementioned meth- ods also suffer from significant overhead, both in terms of computational resource demands and execution time. As result, these methods are generally excluded from robust-
3. Methods
In this section, we detail SIRA attack formulation and im- plementation. First, we lay out the problem setting in Sec-
3

---

Self-Information Rewrite Attacks Text Watermarking
Masked Text
Mask according to self- saff-infromatio
Nike Inc enjoyed beneftte franr clrihos continue long para long growth:
LLama3 rewrite Moto P P VOotDe
Watermarked
Attack Text
enjoyed gaina
America last quarter banetised from llwvr markddonnss helping the maker tern path long form profits and siles growth
Nike signtifcant gaina benofied from fewer markdooens, which heleed drive long ferm profi and sales growth.
Watermar ked strength
dynificant North America Europe, largely dus reduted mark downs and string performnce bu business This sucoess esalded the company to stay on track zo lomp form profiubiiity
Watermarked strength
paraphrase
Reference Text
Figure 1. SIRA pipeline consisting to two steps. First, the attack generates masked text based on self- scff-information. If the self- information of specific part above pre-set threshold, that portion of the text is masked and replaced with placcholder. Simultancously, reference text is generated by asking the LLM to paraphrase. In the second step, the LLM is prompted complete the masked text while incorporating all the information from the reference text.
tion 3.1, then we develop the details of the method in Sec- tion 3.2.
output Up = P(yh)) The pre-set threshold e E [o, is a parameter that quantifies the minimum required level of se- mantic similarity between the original watermarked output Yo and the perturbed output yp P(ys).
3.1. Problem setting
Definition 1 (Language generative model). A Language generative model M : X + Y maps any input prompt X an output y Y, where X the prompt space, Y the output space. We denote Yn is human written text space, Y, is the machine generated unwatermarked text, Y, is the machine generated watermarked text.
black box ad- versarial problem and we assume that the attacker should not know the watermark algorithm W, the secret key k and should not have access to the detector D. The attacker does not have access to any information about the feature distribution of the watermark algorithm or the model architecture.
Assumption: We define the scenario
Definition (Watermark Algorithm): A watermark algo- rithm consists of a watermarking function W, secret key k and detector D. The watermarking function W, parameter- ized by the key k, denoted as WR. modifies the output y to embed watermark, given an input prompt
For watermark algorithm, the goal is to achieve a balance between robustness and performance. The detector D is formulated as an optimization problem with the objective of minimizing classification errors. Specifically, the detector aims to maximize its accuracy in distinguishing between human-wriin written text yı and watermarked text you The goal of detector D can represente
X resulting in watermarked output you which (z, Wi) - Vh E Y.. The detector D, using the same key k, can then verify whether given output ý E Y contains the embedded wa- termark. The detector D operates as a binary classifier with following output behavior:
E Ey,,, [log (1 Dop (Wk, vhh)]
+ Ey [log (Do, (Wk, Vw))]
(2)
D(W.. = {
if t is detected as watermarked otherwise
For attacker, the perturbation function P is defined to mini- mize the probability that the detector D successfully identi- fies the watermark in the perturbed output yp. while ensuring semantic preservation. The goal for P can represente as:
(1)
The detector D contains a parameter 8, where the is the z-score threshold.
Definition 3 (Perturbation Funetion): The attacker has perturbation function P Y., Yp modifies the wa- termarked output Vo to produce perturbed output y/- = P(yw). The function P aims to minimize the detection success rate the detector on the perturbed output yp- function S(ye. yp) measures the semantic similarity be- tween the original watermarked output yo and the perturbed
p arg min E P(y
(3)
s.t. S(yw, P(yw)) 2
(4)
Note that D(W,, , P(y.)) is only used during the evaluation phase. The attacker does not have access to the detector during the training or generation stages.

---

Self-Information Rewrite Attacks on Text Watermarking
3.2. Self-information rewrite attack
Algorithm I Pseudocode for Self-information rewrite attack
primary challenge in watermark removal attacks identi- fying the "green token" defined by the watermarking algo- rithm. Some methods, such as Random Walk (Zhang al., 2023), use grammatical group matching to explicitly replace verbs. In contrast, approaches like DIPPER (Krishna et al., 2024) and GPT Paraphraser (Liu et al., 2024) delegate the task of rewriting and removing green token to large language models through high-level instructions. However, methods of this type lack transparency and control; relying on LLM for consistency with original watermarked text.
1: Input: Watermarked token sequence y
(v), U2...... V,. language model Mattack, self-
information percentile e, instruction s
2: Output: Response token sequence Yp without water- mark.
y' < MMttack (y)
Paraphrase sequence y' using
Mattack
4: I <-11 []
5: for i = 1 to n. do v Compute self-information for each
token y
6: [ - log P(ys context)
Our attack is based on common principle of watermark- ing algorithms, as discussed in the KGW (Kirchenbauer et al., 2023; Liu et al., 2023) work: since the watermark must remain imperceptible to the user, high-entropy tokens are ideal candidates for embedding. High-entropy tokens exhibit more uniform distribution of probabilities, this uni- formity means that when logits are adjusted to increase the likelihood of green tokens, it is easier to embed watermarks effectively without significantly compromising the quality of the output. Meanwhile this also implied high-entropy token has lower probability thus higher self-information.
7: end for
8: Te < Percemtile(I,c)
> Determine threshold from c
percentile of I
9: for i to do
10:
if [[] > Te then
11:
Y/2 +- Ø
D Mask token if above threshold
12: end if
13: end for
14: Yp + -Mattac (y't y, Generate de-watermarked
response yp using Mattack 15: return Yp
In our approach, we propose a straightforward and easily implementable solution by leveraging self- eeffiinformatio to identify potential green-list tokens and subsequently rewrite them. High-entropy tokens are typically associated with high self-information due to their unpredictability and low probability of occurrence. Meanwhile, small probability changes caused by the watermark algorithm can reduce self-information. By considering both the change in self- information and high-entropy token inherent nature, we classify tokens with high moderate self-information po- tential green-list tokens and filter them out to obtain a more neutral template for LLM rewriting. Empirically, our pre- liminary experiments show that utilizing self- inf-mmnformt,on,, rather than directly filtering based on high entropy, results in higher attack success rates. We present a detailed discussion in Appendix G.
Te f- Pereeenttill e)
Any token with a self-information value I[:] > is consid- ered to be potential token and will be masked and replaced with a placeholder. In our experiments, we discovered that using placeholders outperformed directly masking specific tokens. The placeholders serve as cues, maintaining the text's structure, indicating where tokens have been masked which providing the LLM with hints about the original text's length and the likely number of words, allowing for more high quality reconstructions.
However, the compression will still result in the loss of watermark text information details. To address this, we use the base LLM M Mattack rewrite the watermarked text, creating reference text. This rewritten text serves reference to preserve semantic integrity during the second step. The reason we do not use the original watermarked text is that we find this leads L.M to take shortcuts: LLM tend to directly take the content from the watermark text, due to the high similarity between masked and watermark text.
Given a watermarked text y {yo, V1...., }. where yi represents each token, we employ a base language model Mattack; Mattack is distinct from the generative model M used to produce the watermarked text. We use Mattack to calculate the self-information for each token ye as follows:
I TVyx)= = - -log P(ye (Vo, ..,..., 11-1; Mattack),
In the final attack phase, we provide the Mattack with the masked text, reference text, and instructions for a fill-in- the-blank task, guiding it to reconstruct the missing content with greedy decoding strategy. We provide the instructions we use in Appendix E. The pseduocode of our algorithm is shown in Algorithm 1.
where P(yeluo, .I...., bility of token ys given its preceding tokens in the sequence, as estimated by the language model Martack- To mask the potential green list tokens, we set a threshold e, and get the overall paragraph threshold by percentile:
.t............. 13 Mattack) denotes the proba-
5

---

Self-Information Rewrite Attacks on Text Watermarking
Table 1. Commprrion of watermark algorithms under different attack methods. The hest results marked in bold and the second hest results are marked in underline. Our most lightweight method outperforms all previous paraphrasing attacks. Sarg- achieves 100% or near 100% attack success rates on all seven tested watermarking algorithms under black- box settings. Due to differing threat models, we can not conduct fair comparison with informed watermark attack methods, thus these methods are excluded.
Comparison of Watermark Algorithms under Different Attack Methods
Watermark
KGW-1
Unigram
UPV
EWD
DIP
SIR
EXP
Attack
Word delete (Welbl et al., 2020)
22.4%
1.6%
6.6%
22.8%
57.4%
44.0%
9.4%
Synonym Substitution (Yu et al., 2010)
83.2%
17.4%
65.2%
76.2%
99.6%
82.0%
51.0%
GPT Paraphraser
100 %
63.9%
71.9%
90.8%
99.8%
58.8%
72.2%
DIPPER-1 (Krishna et al., 2024)
82, 4%
37.0%
58.6%
82.2%
99.6%
61.2%
73.6%
DIPPER-2 (Krishna et al., 2024)
95.8%
45.6%
61.8%
89.0%
99.8% %
63.6%
82.2%
SIRA-Tiny(Ours)
96.4%
87.6%
84.4%
97.8%
99.8%
75.0%
90.6%
SIRA-Small(Orrs)
100% %
93.8%
93.0%
100 %
99.8%
83.4%
93.4%
SIRA-LLarge(Ours)
100 %
100 %
99.6%
100 %
100 %
98.8 %
99.8 %
4. Experiments
2024) to compare with our method. For GPT Paraphaser, use the GPT-40-2024-05-13 (OpenAI, 2024) version. For DIPPER-1 the lex diversity is 60 without order diver- sity, and for DIPPER-2 we additionally increase the order diversity by 40. The word deletion ratio is set to 0.3 and the synonym substitution ratio is set to 0.5. The synonyms are obtained from the WordNet synset (Miller, 1995).
4.1. Setup
Dataset and Prompts. Following prior watermarking re- search (Kirchenbauer et al., 2023; Zhao et al., 2023; Liu et al., 2024; Kuditipudi et al., 2023), we utilize the C4 dataset (Raffel et al., 2020a) for general-purpose text gener- ation scenarios. We selected 500 random samples from the test set to serve as prompts for generating the subsequent 230 tokens, using the original C4 texts as non-watermarked examples.
Evaluation. We utilize the attack success rate as our primary metric. The attack success rate is defined as the proportion of generated attack texts for which the watermark detector incorrectly classify the attack text as the unwatermarked sample, compared to the total number of attack texts. To mitigate the influence of detection thresholds, we follow prior work (Liu et al., 2024; Zhao et al., 2023) adjust z- threshold of detector until reaches target false positive rate in Figure 2, We used generated 500 attack texts as positive samples and 500 human- written texts as negative samples. We dynamically adjust the detector's thresholds to establish false positive rates at 1% and 10%, and we report the true positive rates and FI-scores. Our method runs on NVIDIA A100 GPUs(Tiny, Small run on a single GPU).
Watermark generation algorithms and language model. To conduct a comprehensive evaluation, we select seven recent watermarking works: KGW (Kirchenbauer et al., 2023), Unigram (Zhao et al., 2023), UPV (Liu et al., 2023), EWD (Lu et al., 2024), DIP (Wu et al., 2024), SIR (Liu et al., 2024). EXP (Aaronson & Kirchner, 2022) in the assessment. The watermark hyperparameter settings shown in Appendix A, and the detection settings adhere to the defaullrecommendations (Pan et al., 2024) configurations of the original works. Specifically, for KGW-k, A is the number of preceding tokens to hash. A smaller & implies stronger attack robustness yet simpler watermarking rules. We use KGW-1 in our experiment. For language models, we follow the previous work setting select (Kirchenbauer et al., 2023; Liu et al., 2024; Zhao et al., 2023) Opt-1.3B (Zhang et al., 2022) as the watermark text generation model. Our SIRA SIRA Small, and SIRA Large run the Ilama3 Instruct models (Dubey et al., 2023) with 3B, 8B, and 70B parameters, respectively.
4.2. Experimental Results.
In Table 1, we present the attack success rates of various watermark removal methods across different watermark- ing algorithms. The results demonstrate that our approach consistently outperforms all other methods for each wa- termarking algorithm. Notably, the closest competitors to our method are DIPPER (Krishna et al., 2024) and GPT Paraphraser, which are model based paraphrasing attacks. Even Our most lightweight SIRA-Tiny method outper- forms all previous approaches regarding attack success rate in our experiments involving seven watermarking algo- rithms on the C4 dataset (Raffel et al., 2020a). Our Large
Baseline Methods. For our method, we use e 0.3 as thresh- old. For the attack method, we use word deleteion (Welbl et al., 2020), synonym substitution (Yu et al., 2010). Dip- per (Krishna et al., 2024), and GPT Paraphaser (Liu al.,
6

---

Self-Information Rewrite Attacks on Text Watermarking
FL Watemark Nethods and Attack Types
TFR under 2% FPR across Watermark Nechods and Mtack Types
TPR under 10% FPR scroes Watermari Nethods and ritack Types
100
95
80
80
90
Score
85
1% PM TOW
60
10% 178
60
40
75
20
70
Watemaak gorithm
Wotormark Algofitiis
matermrkk Algorttm
(a) True positive rate with FPR set to 1 1%.
(h) True positive rate with FPR set to 10%. (c) Best Fi-score achieved by watermark.
Figure 2. To mitigate the default z-threshold's impact the robustness of watermarking algorithms, we dynamically adjust the z-score threshold until the watermark detector achieves specified false positive rates. The true positive rate (TPR.) and the best F1 score are shown. Lower TPR and Fl scores a given false positive rate (FPR) indicate that the watermark detector struggles distinguish attack texts from human-written texts, suggesting more effective attack. Detailed values for the figures are provided in Appendix C.
method achieves nearly 100% % attack success across all seven tested watermarking algorithms.
4.4. Ablation Experiment
In this section, we aim to further scrutinize the self- information rewrite attack and emphasize the potential of this attack. We utilize Opt-1.3b and a random sample of 50 prompts from the C4 dataset to generate watermarked responses. Unless otherwise specified, we use Llama-3- 8b (SIRA-Small) as the base model for our attack. The temperature for the base model is set to 0.7.
To further demonstrate the effectiveness of our method and avoid the impact of fixed z-threshold on detector perfor- mance, we follow previous work by setting the FPR to 1% and 10%, and report the true positive rate of the detector on adversarial texts based on the adjusted z-threshold cor- responding to the FPR. Additionally, we report the best Fl score that the watermark algorithm can achieve under different attacks. The results are shown in Figure 2, and the detailed numbers are provided in Appendix C. Lower true positive at given false positive rate indicate that the watermark detector struggles more to differentiate between adversarial texts and human-written texts. Our algorithm achieves optimal attack performance in most cases; suggest- ing a more effective attack.
How does the self-information threshold affect final per- formance? In this experiment, we use UPV as the water- marking algorithm. We varied the value of from 0.25 to 0.70 in increments of 0.05 to test its impact the success rate of the attack using the UPV algorithm. The results are shown in Table 2.
We observed that the attack success rate and sentence- bert is directly influenced by the value of For the UPV algorithm, setting the threshold to 0.3 results in a highly effective attack. A significant performance gap is observed when increases from 0.60 to 0.65. Additionally, there is tradeoff between attack effectiveness and semantic preservation. When is below 0.25, the generated attack text tends to lose more detailed information from the original watermarked text. Considering both performance and semantic preservation, we recommend setting 6 between 0.2 and 0.3. For less robust algorithms, setting between 0.4 and 0.5 is sufficient to achieve an attack success rate exceeding 90%. Setting e to 0.3 effectively removes the watermark while preserving the original semantics.
4.3. Text quality analysis
To further demonstrate that our method does not adversely affect text quality, we conduct additional evaluations of the text generated by the model. We compare Perplex- ity (PPL) of the text quality. Furthermore, we use well-established metric sentence-level embedding similarity Dientence-BERYY Score (s-BERT)) before and after the attack to explore whether the attack alters the semantic con- tent. We also conducted experiments in the Appendix F using ChatGPT as a judge to measure overall semantic similarity. The results, shown in Figure 3, indicate that our method has a smaller impact on text quality compared to other approaches. Our approach, similar to other model- based methods, benefits from more powerful large language models, achieving better performance in terms of the PPL metric compared to the original watermarked text. Addi- tionally, our method retains a greater degree of semantic information. We show the detail numbers of two metrics in Appendix D.
Self-information mask versus Random mask and Itera- tive Paraphrase(twice) In this experiment, we replace self- information- based selective masking with a random mask- ing strategy and Iterative Paraphrase(twice). while keeping all other steps unchanged. We use the same masking ratios, ranging from 0.4 to 0.8 in increments of 0. and compare the resulting attack success rates. The Unigram watermark-
7

---

Seff-Information Rewrite Attacks on Text Watermarking
Performance Watermarks and Attacks
S-BERT Borfor mance Across Watemrarkss and Attacks
1.0
250
0.8
200
0.6
150
0.
0.2
Waterrark Algartzmms
Aigenthhmm
(a) Performance of PPL.
(b) Performance of S-BERT.
Figure 3. Performance comparison of watermark methods against various attack methods hased on PPL. (Perplexity.) and s-BERT (Sentence- BERT scoret). The word delete will significantly increase the PPL and lead to overflow. We marked the overflow data with NaN in the Figure 3a. The synonym substitution will also increase the PPL. The paraphrased text has better text quality than the original watermark text for our method and GPT Paraphraser. The detailed number are shown in Appendix
Table Effect of self-information Threshold the Attack Success Rate and Semantic Preservation of the UPV Algorithm
self-information threshold e
0.25
0.30
0.35
0.40
0.45
0.5
0.55
0.60
0.65
0.70
Attack Success Rate
96%
94%
94%
88%
80%
76%
72%
70%
58%
32%
Sentence-BiiR BERT
0.68
0.77
0.78
0.78
0.79
0.79
0.82
0.81
0.83
0.86
Table 3. Comparison with Random Masking Strategy and Iterative Paraphrase. Iterative Paraphrase independent of the mask ratio. For clearer comparison, we place its results under the 0.7 (default mask ratio) column. Notice here the random mask performance also benefits from other steps like rewriting in our framework. The vanilla random mask has similar attack success rate as word deletion.
Table 4. Comparison of Attack Success Rate and Average 7-score. The reference text is generated by asking the base model to para- phrase the watermarked response, while the attack text is generated using our two-step approach.
Text
Attack Success Rate
Average 2-score
Human-writce Text
N/A
0.12
Reference Text
64%
3.75
Mask Ratio
0.7
Attack
94%
1.85
Iterative Paraphrase
56%
Random Mask
52%
66%
78%.
80%
82%
Solf-information Mask
80%
88%
92%
96%
100%
ing method is employed to generate the watermarked text. The results are presented in Table 3. To ensure fair compar- isons, the random masking strategy is executed five times, and the final average attack success rate reported.
generate watermarked text. We set the detector's 's threshold to 4 according to its default settings. For a given input, the detector calculates its z-score, and the score exceeds 4, the text is classified as watermarked.
The results indicate that, at any given mask ratio, the self- information-basee masking method significantly outper- forms the random strategy. The random masking approach also has bottleneck, with limited improvement in attack success rates beyond a ratio of 0.6. This due to the ran- dom mask not make sure all target green tokens are removed. Also,This effectiveness primarily caused by the double paraphrasing process. For single watermarked text with fixed mask ratio, our method is deterministic, as the same tokens are masked each time. In contrast, the random approach does not provide this guarantee.
We measured the attack success rate for each of the follow- ing stages: the reference text generated in the first step of our algorithm, and the final attack text. Additionally, we cal- culated the average Z-score for each stage and reported the z-score of humtn-written text as a reference. The result are shown in Table We observed that the attack success rate for the reference text lower than that of the final attack text. Paraphrase strategies tend to preserve more n-grams from the original text, which may still be detectable by the wa- termark detection algorithm. In contrast, our attack reduces the presence of such n-grams by utilizing self-information filtering. Additionally, the z-score produced by our method is closer to that of human- written text compared to simple paraphrasing approaches.
Also, This experiment ruled out the possibility that the attack
Does the success of the attack due to paraphrased refer- ence text? We used the Unigram watermarking algorithm to
8

---

Self-Information Rewrite Attacks on Text Watermarking
5. Conclusion
In this paper, we present the Self-Information Rewrite At- tack (SIRA), a lightweight and effective method for re- moving watermarks from LLM-generated
by targeting anomalous tokens. Empirical results show that SIRA outper- forms existing methods in attack success rates across multi- ple watermarking techniques while preserving text quality and requiring minimal computational resources. By ex- ploiting vulnerabilities current watermarking algorithms, SIRA highlights the need for more robust and adaptive wa- termarking approaches in watermark embedding. We will release our code to the community facilitate further search in developing responsible AI practices and advancing the robustness of watermarking algorithms.
Impact Statement
In this work, we aim provide an approach to test the robustness of Large Language Models watermark. We pro- pose method that can remove different watermarks in LLM-generated text. We are aware of the potential risks that our work entails for the security and safety of LLMs, as they are increasingly adopted in various domains and applications. Nevertheless, we also believe that our work advances open and transparent research on the challenges and limitations of the LLM watermark, which is crucial for devising effective solutions and protections. Similarly, the last few years the exploration of adversarial attacks (Wei et al., 2023; Madry et al., 2017; Krishna et al., 2024) has led to the improvement of responible AI and led to techniques to safeguard against such vulnerabilities,e further coordinated with them before publicly releasing our results. We also emphasize that, our ultimate goal in this paper is identify the weaknesses of existing methods.

---

Self-Information Rewrite Attacks Text Watermarking
References
Liu, A., Pan, L., Hu, X., Meng, S., and Wen, L. seman- tic invariant robust watermark for large language mod- els, 2024. URL https: /arxivv.vvv. org/ abs/ /3100. 06356.
Aaronson, S. and Kirchner, H.
Watermarking gpt outputs. https://www. hettps//ww..coottarrc. www.taaronssn.coom com/ talks/watermars ppt, 2022.
Anthropic. Claude 3.5, 2024. URL https: /www.
Liu, Y. and Bu, Y. Adaptive text watermark for large lan- guage models. arXiv preprint arXiv:2401.13927, 2024.
anthropic- com http///w///c/ccee// htt///laudeee33 -- 3-mmii ly. Ac- cessed: 2024-09-24.
Lu, Y., Liu, A., Yu, D., Li, J., and King. I. An entropy- based text watermarking detection method. arXiv preprint arXiv:2403. 13485, 2024.
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., and Wong, E. Jailbreaking black box large language mod- els in twenty queries. arXiv preprint arXiv:2310.0844 2023.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D.. and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rXiv:/706.0600 2017.
Christ, M., Gunn, S., and Zamir, O. Undetectable water- marks for language models. In The Thirty Seventh Annual Conference Learning Theory, pp. 1125-1139 PMLR, 2024.
Miller, G. A. Wordnet: lexical database for english. Com- munications of the ACM, 38(11):39-44 1995.
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., and Narasimhan, K. Toxicity in chatgpt: Analyz- ing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023.
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E., and Bauer, M. Artificial intelligence and increasing misinformation. The British Journal of Psy- chiatry, 224(2)333-35, 2024.
Dubey, A., Jauhri, A., Pandey, A., et al. The llama herd of models. arXiv, arXiv:2407.21783 2023. URL https: //arxiv. org/ abs abs//407/ 22142477777..... 21783.
OpenAl. Chatgpt-4o: Multimodal and multilingual capabil- ities. OpenAl website, 2024. https://ppenai.co//caaagg 4o.
Jovanović, N., Staab, R., and Vechev, M. Watermark stealing in large language models. arXiv preprint arXiv:2402. 1..e..936. 19361,
Pan, L., Liu, A., He, Z., Gao. Z., Zhao, X., Lu, Y., Zhou, B., Liu, S., Hu, X., Wen, L., et al. Markllm: An open- source toolkit for Ilm watermarking. arXiv preprint arXiv:2405. 1005/, 2024.
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. In International Conference Machine Learning, pp. 17061-17084, PMLR, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (1 (140):1-67, (,0):1667,20004 2020a.
Krishna, K., Song, Y., Karpinska, M., Wieting, J., and lyyer, M. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020b.
Kuditipudi, R., Thickstun, J., Hashimoto, T., and Liang, P. Robust distortion-fret watermarks for language models. TMLR, 2023.
Lau, G. K. R., Niu, X., Dao, H., Chen, J., Foo, C.-S., and Low, B. K. H. Waterfall: Scalable framework for robust text watermarking and provenance for llms. pp. 20432- 20466, 2024.
Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing, pp. 3982-3992, Hong Kong, China, November 2019. Association for Compu- tational Linguistics. doi: 10.18653/ https: / /tair//aaaaattt hwtaaaaaaaatllooooo http:hhhhooogo.gggg11----1111
Li, S., Yao, L., Gao. J., Zhang, L., and Li, Y. Double- i watermark: Protecting model copyright for IIm fine- tuning, arXiv preprint arXiv:2402. 14883, 2024.
URL
Liu, A., Pan, L., Hu, x., Li, S., Wen, L., King, I., and Philip, S. Y. An unforgeable publicly verifiable watermark for large language models. In The Twelfth International Conference on Learning Representations, 2023.
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., and Feizi, S. Can ai- ae-genertted text be reliably de- tected? arXiv preprint arXiv:2303. 2023.
10

---

Self-Information Rewrite Attacks on Text Watermarking
Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, (7(3):379-423 1948.
Kakker, C. Ai bot chatgpt writes smart essays-should academics worry? Nature, 2022.
Wang, X., Chen, T., Yang, X., Zhang, Q.. Zhao, X., and Lin, D. Unveiling the misuse potential of base large language models via in-context learning. arXiv preprint arXiv:2404. 10552, 2024.
Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does Ilm safety training fail? In Advances in neural information processing systems (NeurIPS), 2023.
WelbI, J., Huang, P.-.., Stanforth, R., Gowal, S., Dvijotham, K. D., Szummer, M., and Kohli, P. Towards verified robustness under text deletion interventions. In Interna- tional Conference on Learning Representations, 2020.
Wu, Q. and Chandrasekaran, V. Bypassing Ilm water- marks with color-aware substitutions. arXiv preprint arXiv:2403. 14719, 2024.
Wu, Y., Hu, Z., Guo, Zhang, H., and Huang, H. A resilient and accessible distribution preeiioiinreeriiggggg watermark for large language models. 2024.
Yu, L.-C., Shih, H.-M., Lai, Y.-I.., Yeh, J.-F., and Wu, C.- H. Discriminative training for near-synonym substitution. In Proceedings of the 23rd International Conference Computational Linguistics (Coling 2010), pp. 1254-1262, 2010.
Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ate- niese, G., and Barak, B. Watermarks in the sand: Im- possibility of strong watermarking for generative models. arXiv preprint 1rXXv:2311.003777 2023.
Zhang, S., Roller, S., Goyal, N., Artetxe, M.. Chen, M., Chen, S., Dewan, C., Diab, M. Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint 2000002205.00068, 2022.
Zhao, X., Ananth, Li, L., and Wang, Y.-.. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.1743, 2023.

---

Self-Information Rewrite Attacks Text Watermarking
Contents of the appendix
The contents of the supplementary material are organized as follows:
In Appendix A, we list the hyperparameters of the watermarking algorithm we used in experiment Section 4.
In Appendix B, we perform the comparison of execution time and VRAM consumption between our algorithm and other baseline methods.
In Appendix C, we present the specific data points corresponding to the figure shown Section 4.2.
In Appendix D, provide the precise data underlying the figure depicted in Section 4.3.
In Appendix E, we provide the prompt we used to generate attack text.
In Appendix F, we conducted extensive experiments to evaluate the overall preservation of the semantic meaning of the original watermarked text.
In Appendix G, we offer a brief discussion about the change in self-information under the influence of the watermark algorithm.
In Appendix H, we provide the full proof of our proposed method attack success rate upper bound and lower bound, together with the proof of lemma.
In Appendix I, we provide a visual comparison of the text generated by our method with watermarked text, non- watermarked text, and text generated by other attack methods.
In Appendix J, we provide extended experiments, including comparison between our methods and baseline approaches the OpenGen dataset, well the performance of our methods against Adaptive Watermark and Waterfall Watermark schemes.
12

---

Self-Information Rewrite Attacks Text Watermarking
A. Watermark algorithm setting
In this section, we list the hyperparameters of the watermarking algorithm we used in Section below.
"algori i cchm_hame"" "KGW",
"gamma": 0.5, "delta": 2.0, feash_key" 15485863, "prefix "prefix_length" length": 1, "z "__thresheld"; 4.0
Listing 1. configuration KGW
"algorithm_name" name"; "Unigram",
3 "gamma": 0.5,
"delta": 2.0,
"hash_key"; 15485863, "t_threshcld"; 4.0
Listing 2. configuration Unigram
"DPV",
"gamma": 0.5,
"deita": 2.0,
"#_threshcid"; 4.0,
"prefix_length" 1, "bit_number"; 16, "sigma": 0.01, "default_top_k"; 20,
10
Listing 3. configuration UPV
"algori tasttthm_aaaa _an_n: ; "END", "gamma": 0.5, "delta": 2.0,
"hash_key"; key": 15485863, "prefix_lengthy 1, "2_threshold": 4.0
Listing 4. configuration EWD
"algorithe aettorithmnaammee : "DIP", "gamma": 0,5, "alpha":0.46 "hash_key"; 42, lpretix_length",, "z_threshold"; 1.513, "ignore_history"!
Listing configuration DIP
13

---

Self-Information Rewrite Attacks Text Watermarking
"algorithm_nane : "SIR", "delta": 1.0, "chunk_length"; 10,
"scale_dimension"" 300, "z #__threshod";; 0.0,
Listing configuration SIR
"algorithm name"; "EXP",
3 "prefix_length;; 4,
"hash_key"; 15485863,
"threshold"; 2.0, "sequence_length"" 230
Listing 7. configuration EXP
14

---

Self-Information Rewrite Attacks on Text Watermarking
B. Execution time and VRAM consumption comparison
In this section, We conducted the attack experiments using 50 distinct watermark texts, each containing approximately 230 = 20 tokens. For each method, we measured both execution time and VRAM usage. The reported execution time reflects the average for single attack instance. The experiments were run on NVIDIA A100 40GB GPUs, utilizing a sequential device map for baseline methods requiring multiple GPUs. The configuration for the GPT Paraphraser follows the setup described in Section 4.1. The results are shown in Table 5.
One of the main limitations of current model-based watermark removal attacks is their substantial resource consumption. For instance, DIPPER built on the T5-XXL model, necessitates two 40GB A100 GPUs for effective operation. Similarly, the GPT parser introduces considerable costs due to its dependence on a proprietary model that employs token-based billing.The The total time consumption for SIRA consists of two parts: two generations by the base model and the self-information mask. Using SIRA-Tiny as example, The self-information mask is nearly negligible, as it does not require any text generation (less than 0.1 seconds). The other two generations take around 2.6 seconds per generation on single A100 GPU. Thus the total execution time is around 10 seconds. We use the huggingface library in our experiment.
The DIPPER method utilizes a specially fine-tuned T5-XXL model for text paraphrasing. This model needs at least 40 GB VRAM to run and one-time generation requires around 15 seconds on two A100GPU.
Our proposed pipeline operates with a minimal configuration of the LLaMA3-3b model, which is lightweight enough to run on mobile-level devices. This ensures compatibility with many consumer-grade GPUs, significantly reducing hardware requirements. Notably, even our most lightweight approach, SIRA-Tiny, outperforms all previous methods in our experiments while consuming far fewer resources.
Moreover, our method is independent of specific LLM architectures, allowing it to be seamlessly transferred to the latest language models to leverage their superior performance at no additional cost. As demonstrated in Table 5, the data was generated using 6 our VRAM consumption can be further reduced by employing mixed precision or quantized models, further enhancing efficiency.
Table 5. Comparison of Execution Time and VRAM Usage for Different Methods.
Method
Execution Time (s)
VRAM Usage (GB)
GPT Paraphraser
12.8
N/A
DIPPER
14.7
44.56
SIRA-Tiny
5.3
7.06
MIRA-Smmll
10.3
18.20
SIRA-Large
37.2
138.60
We also provide comprehensive cost analysis of our method in comparison third hhird-party paraphrasing services. Specifically, we estimate the cost of processing 1 million tokens of watermarked text. According to OpenAl's pricing, paraphrasing using GPT-4o (GPT Paraphraser) costs
$0.01 x 2 x 10 = $20,
where $0.01 is the cost per 1M tokens (input or output), the factor 2 accounts for both input and output, and 10 iterations are assumed.
In contrast, our method (SIRA-Small,, based on LLaMA3-SB deployed via AWS Bedrock, incurs a significantly lower cost: $0.22 x 2 $0.88,
where $0.22 is the per- 1 M-token cost (input or output), the first factor 2 accounts for input and output, and the second factor of 2 accounts for two rewriting iterations. The cost can be further reduced using SIRA-Tiny (LLaMA3-3B).
C. Best F1 score and TPR/FRR
In Table 6, we list the specific data from the figure in Figure 2, which reflects differrntt attack method's performance of dynamically adjusting the watermark detector's 's z-threshold until a specified false positive rate is achieved. We report both the Fl score and the true positive rate. It can be observed that, in most cases, our attack method achieves the best performance.
15

---

Self-Information Rewrite Attacks Text Watermarking
Table 6. In this experiment, we dynamically adjust the z-score threshold of the watermarking algorithm until achieving specified false positive rates for watermark detector. Lower TPR scores indicate that the watermark detector struggles more differentiate between attack texts and human-written texts, suggesting more effective attack.
Method
Attack Type
FPR
10% FPR
Best F1 (%)
TPR %)
F1 (%)
TPR (%)
F1 (%)
F1 (%)
No attack
100
99.5
100
95.2
99.8
DIPPER
1.6
3.1
7.8
13.3
66.6
DIPPER
0.8
1.6
7.0
12.1
KGW-1
GPT Paraphraser
1.8
3.6
8.7
14.0
66.8
SIRA Tiny
26.4
41.4
60.0
70.6
79.2
SIRA-Small
1.1
2.1
8.4
14.0
66.6
STRA Large
1.1
2.1
8.0
13.6
66.6
No attack
100
99.5
100
95.2
99.8
DIPPER
89.0
93.7
97.6
95.8
95.4
DIPPER
86.0
91.8
96.8
94.2
94.6
Unigram
GPT Paraphraser
73.6
84.3
91.7
91.6
91.6
SIRA Tiny
42.2
58.9
71.4
78.7
82.5
SIRA-Small
29.0
44.6
62.2
72.7
80.0
SIRA Large
25.8
40.7
59.2
78.8
No attack
100
99.5
100
95.2
99.8
DIPPER
94.4
93.1
DIPPER -2
67.8
80.4
91.6
91.6
91.6
UPV
GPT Paraphraser
53.0
69.2
82.2
86.2
86.9
SIRA Tiny
40.8
57.5
74.4
80.7
83.4
SIRA-Small
27.0
65.6
SIRA Large
25.2
39.9
65.0
74.3
80.3
attack
100
DIPPER
66.2
79.2
91.4
90,8
90.8
DIPPER -2
54.8
70.3
83.2
81.2
81.2
EWD
GPT Paraphraser
46.6
63.1
75.4
81.3
83.8
SIRA Tiny
22.0
35.8
52.2
64.4
77.0
SIRA-Small
35.8
49.1
SIRA Large
7.4
13.7
31.2
44.2
71.3
attack
100
99.5
100
95.2
99.8
DIPPER
5.4
10.0
24.0
36.1
67.7
DIPPER-2
2.2
4.3
16.4
26.2
66.7
DIP
GPT Paraphraser
4.3
8.3
20.4
32.0
76.6
SIRA Tiny
11.8
SIRA-Small
1.6
3.1
11.2
18.7
66.7
SIRA Large
1.0
1.9
11.4
18.8
66.7
No attack
100
99.5
100
95.2
99.8
DIPPER
64.6
78.0
82.4
85.6
85.6
DIPPER -2
57.6
72.6
72.6
83.4
83.4
STR
GPT Paraphraser
66.2
79.2
85.2
87.3
87.3
SIRA Tiny
52.8
68.7
77.6
82.7
85.1
SIRA-Small
42.8
59.5
70.2
77.9
82.4
SIRA Large
48.4
64.8
77.4
82.6
84.1
No attack
100
99.5
100
95.2
99.8
DIPPER
0.8
1.6
1.2
2.1
66.6
DIPPER -2
0.4
0.8
2.0
3,8
66.7
GPT Paraphraser
2.0
SIRA Tiny
0.6
1.2
4.4
7.7
66.6
SIRA-Small
0.4
0.8
9.3
15.6
66.6
SIRA Large
0.4
0.8
6.2
10.7
66.6
16

---

Self-Information Rewrite Attacks Text Watermarking
D. Detail number of PPL and sentence bert score
this section, we list the detail number of PPL and sentence-ber score we present the Section 4.3,
Watermark
KOW-
UPV
LWD
DIP
SIR
EXP
+-BERT())
PPL[.)
+-BERTTT;
PPL(L)
(-BEXT
PPLÍI
+-BRRT())
PPL(C]
+BEERI(1)
PPL(1)
s-BERT(()
PPL(1)
+BERIT
No attack
12.00
0.55
11.49
0..56
9.25
0.56
11.64
0.57
10.50
0.57
11.76
0.56
16.48
0.54
Word dokie
NaN
0.05
NGN
0023
N4N
0083
NaN
0.01
NGN
0704
N2N
0303
KaN
01.04
Synungm
232.5
25285
GPT Paruphraser
9.19
0.88
8.95
0.87
8.28
0.87
9:20
0.88
8.79
0.38
932
0.87
11.96
0.55
DIPPER--1
12.00
0.80
11.80
0.79
10311
0381
11.87
0.76
11.03
0.30
12.43
0.79
16.50
0.77
DIPPER-2
0.79
0.80
0.76
16.45
SIRA-Tiny (Curs)
SIRA-Small (Cuis)
10399
0.78
0.57
0.78
891
027
9.10
0.78
0002
0.77
934
0228
10395
0.75
SIXA- (Chest
1014
0300
8.02
0.77
0.30
031
1022
031
8.93
0.77
1033
081
11.32
0.79
Table 7. Performance of different attack methods against various watermark techniques. including SIRA variants.
E. Fill in blank instruction
We present our generation prompts(final step) and reference text generation prompts below. The reference text generation followed the same prompt of (Sadasivan al., 2023). During the generation process, the reference text and the masked text replaced by the two texts obtained in the first step.
Generation Prompts
You will be shown one reference paragraph and one incomplete paragraph. Your task is to write a complete paragraph using incomplete paragraph. The complete paragraph should have similar length with reference paragraph. You need to include all the information in the reference. but do not take the expression and words in the reference paragraph. You should only answer the complete paragraph. reference: [reference text]
incomplete paragraph: [mask text]
Reference Parapharse Prompts
You are paraphraser. You are given input passage 'INPUT'. You should paraphrase 'INPUT' to print 'OUTPUT'. 'OUTPUT" should be diverse and different as much as possible from 'INPUT' and should not copy any part verbatim from 'INPUT" However, 'OUTPUT' should preserve the information in the INPUT. You should print 'OUTPUT' and nothing else so that it is easy for me parse.:
F. GPT Semantic Judge
GPT Semantic Judge Prompts
You are helpful assistant. You will receive a original text and pharased text Please act an impartial judge and evaluate how much the pharased text keep the original text semantic meaning. Be as objective as possible. Rate the response on a scale from I to 10. A rating of signifies the pharased text has totally different meaning with the original text. A rating of 10 signifies the pharased text has same meaning with the original text. Your answer should strictly be the format: [Rating].
Here's the original text: [Watermarked text]
Here's the pharsed text:[Attack text]
In this section, conduct extensive experiments to evaluate the overall preservation of the semantic meaning of the original watermarked text. We use ChatGPT (OpenAI, 2024) as an impartial judge to obtain the quantitative results.
The attack success rate alone is not sufficient metric for evaluating an attack method. also crucial to assess whether the original and paraphrased outputs preserve similar semantics. The Sentence- BERecee--RTTT-- score (Reimers & Gurevych, 2019), presented in Section 4.3, measures the sentence-level similarity between the original watermarked text and the adversarial
17

---

Self-Information Rewrite Attacks Text Watermarking
Text. However, it falls short in determining whether the overall semantics preserved. Inspired by the LLM jailbreak work PAIR (Chao al., 2023), which leverages carefully crafted prompts and the powerful capabilities of ChatGPT score attack texts and targets for quantitative evaluation, we adapted their prompts to use ChatGPT for assessing the semantic similarity between watermarked texts and attack texts. This approach allows us to obtain semantic similarity scores that more closely align with human perception. We show the judge prompt in Appendix and the result in shown in Table 8.
Table 8. Semantic Preservation Different Methods
Word Delete
Synonym
GPT Paraphraser DIPPER-1
DIPPER-2
SIRA-Timy
SIRA-Small
SIRA-Large
Semantic Preservation
2.59
2.63
8.25
5.28
6.34
6.10
6.84
8.02
We observed that using GPT for paraphrasing alone best preserves the original text's semantics, whereas methods like word deletion and synonym replacement were largely ineffective. Our approach demonstrated superior semantic preservation compared the DIPPER method.
18

---

Self-Information Rewrite Attacks Text Watermarking
G. Self-information, Entropy and Probability
We provide a brief explanation how the watermark algorithm changes the self-information. To begin, we introduce the definitions of self-in ceffiinformatio..
Self-Information (I(=)): This measures the amount of information "surprise" associated with specific token It quantifies how unexpected the occurrence of token is in given context:
((x) = log2 P(x)
When considering the context h, it becomes the conditional self-information:
[ h) = log2 P( h)
where P(x h) is the probability of token x occurring given the preceding context
We first analyze the non-conditional scenario, assuming that watermarking slightly increases the probability of certain tokens by small amount 6, while adjusting the probabilities of other tokens to maintain normalization. The ó change in token influenced by watermark algorithm is usually very small (e.g less than le-3).
The adjusted probability for the watermarked token Eep is:
P' P'(xw) = P(x:n) +
The adjusted probabilities for other tokens 2; o w)
P'(x:) P(xx))
where Eiju E = 8.
The change in entropy due to these adjustments given by:
AH - log P',x,)
=
(o)))))
P(x;) log P(x))
The partial derivative of entropy with respect to P(xw) is:
0H
log P(xve)
0P(sw)
The change in entropy due to small change in P(æw) is approximately:
0H 8 - (log P(xw) + 1)8 AH ~ OP(..))
In high-entropy contexts, where P(*w) small, log (( becomes large negative value. Therefore, log P(r + still negative, and the product with the small results in a tiny A (decrrees in logarithmically). This attribute makes the watermark algorithm need to embed patterns in high-entropy tokens, otherwise it will significantly compromise the quality of the output.
For self-information, the change in self-informan eeff-information is:
Al(rw) = - -log Pl (((ww) + log ((xw)
The derivative of self-information with respect P(xw) is:
dl dl(ww) 1
dP dP P(xw)
19

---

Self-Information Rewrite Attacks Text Watermarking
For small (( ) P(a
is large, making A (3rw)) more significant for small compared to AH.
Similarly, for conditional seff-information, The uniform distribution serves as a theoretical upper bound for high entropy for given probability space and helps illustrate high-entropy scenarios where probability mass is thinly spread. In such cases,
assume that the model predicts N possible next tokens with equal probability. Where:
P(æ Contextt
N
For large N, P( Context) becomes small.
The adjusted probability for the watermarked token Tw is:
P' Context)
N 6
The adjusted probabilities for other tokens I; (i of w) are:
P' Context)
for xz + Zw
N N
The change in Conditional Self-Information is:
A (1
Conttxtt = I'(x.nn Context)
Context) = - -loog
N +6) + log
Using Taylor series approximation for small
log + log
-N8
The approximate change in conditional self-information is:
[ (oontext) -N8
Compared to the change in entropy, it is obvious self-information more sensitive metric:
OH AH ~ OP(.w)
-loo P(rw) + 1)6
When P( Context) is small, the magnitude of the derivative is large, this indicates that small changes in P(x Context) result in bigger changes in [(x Context). As a result, the green token influenced by the watermark change will have less self-information than the original.
High-entropy tokens are usually associated with medium, self-information due to their unpredictability and low probability of occurrence. Considering the reduced seff-information, these potential green tokens generally will exhibit high moderate self- celf-information values. Therefore in practice, we filter out all tokens with high moderate self-information. This ensures we can comprehensively eliminate potential tokens.
20

---

Self-Information Rewrite Attacks Text Watermarking
H. Theoretical Proof
H.1. Preliminaries and Notation
Watermarked Text. Let y = {v1, ......... yn be sequence of'n tokens generated by a watermarked language model M.
subset W S (1,2....,,) n) denotes the indices of "green" (watermarked) tokens.
Base (Attack) Model.
Let Matuek be a language model distinct from M. Under Manuckk each token y/, has probability
P(y V1.-... -i-1; M Matack).
Self-Information.
The self- informatoon of under Manack is defined as
(ys) = log P(ys V1,.... Vi-15 Mataak)
larger [y;; indicates y; is more "surprising" or low-loowabbll wwwppobbability under Manack-
Attack Strategy.
Threshold Selection: Choose a threshold 7 (e.g., a certain percentile T.) and mask tokens whose self-information exceeds T.
Rewrite Step: Provide the masked text plus a reference text to an LLM, instructing it to fill the placeholders.
Success Criterion: The attack is considered successful if all watermarked tokens are significantly altered that the watermark is no longer detectable.
H.2. Lemmas Self- Self-Information Changes Due to Watermarking
Lemma H.1 (Bound on Self-Information Shift). Suppose a watermarking algorithm increases the probability of a single token Fw from ((xw) P' (((ww) (x 5, where 5 is small (i.e., 8 < 1) and P(xrw) & 1. Let
[(xw) - log P(aw), I'(xw) = - logg P' P'(ww)
Then the drop in self-information, defined as
Al(xw) I'(ww) I(Iw),
bounded by
Al(ww) -- 5- log 1+ Pmux
where Pauax is small upper bound on ((xw) ) in the high self relf-information region.
Proof. Step 1: Express A (xxw).
Al(xw) |- log(P(x( + 6)] [- loo P(xw)] = log( + +6) + log P(aw).
Step Normalize by P(xw).
5 Al(Iw) = log P(xw) (1 P(z.)
+ log P(zw) log( 1
5
P(
Step 3: Bound using P Pmax- Since P(xw) S P Pmax and Pmax < 1, we have
P(zw) 2 1 5
Panas
Hence,
-log + p(())
< - log + 8.
Paaxs
21

---

Self-Information Rewrite Attacks Text Watermarking
giving
(I(ww) - llog 1
6
Pmox
Thus, even after watermarking, a low- low-probability token remains in a bigh-self-informatii regime (downward shift is limited).
Lemma H.2 (Concentration in High Self-Information Region), Consider watermarking process that selectively increases the probability of certain tokens by 5. Let la be the a-quantile of self-information values under Manacko i.e.
Pr
(101 2 1o] =
Mor-Mand
For sufficiently small 6, any watermarked token ye satisfies
Iwy)) 2 la -G6
where C& is a small constant capturing the maximum self-information drop due to ó.
Proof. Let E W be watermarked token. Its probability is raised from P(y:) to P(ys) + 8. Using Lemma H.1,
(2())) - -log( 1
Penax)
5
Cs.
We denote
C = sup A/(x))
x
Because tokens subject to watermarking are chosen (pre-watd watermaa)ma from the high self-information region (I(y)) 2 Ia except possibly some edge cases), their self-information remains least la Cs after the slight probability increase. Thus,
(1()))
(1r) 2 Ia Cs,
showing that watermarked tokens are still near or above la Cs.
H.3. Bounds on Attack Success Rate
Definition H.3 (Attack Success). Let W; be the event "token yi is watermarked," and let A, be the event "token y/ is removed (masked) by the attack." The overall attack considered successful if every watermarked token is removed:
Success(y)
A A ieW
Equivalently, Success(y) requires [(y:) 2 5, for all E W, where T, is the chosen self-information threshold (i.e., the e-percentile).
Theorem H.4 (Attack Success Probability Bounds). Let Te be the self-information threshold chosen by the attacker: Then:
1. Lower Bound:
[W]
Pr[Success(v))
min Pr[[(y) 2 Wil iew
2. Upper Bound:
Pr[Successyy)]
< Pr n (1(x)) 2 Te)
iEW
Proof. Step 1: Success Event. The event Success(y) is equivalent to
n
(I()))
2 Te)-
ieW
22

---

Self-Information Rewrite Attacks Text Watermarking
If any watermarked token ys has (ys) Tex is not masked and the watermark may remain, the attack fails.
Step Lower Bound. Let
a; Pr(I())) 2 Te W.l.
Under (conditional) independence or suitable lower-bounding assumption,
Pr n (1(m) 2 Te)
W; for each z II a.
iEW
8EW
Then,
min (s) w NeW
II iEW
2
This implies
min Pr(((u) wil) m
Pr (uccess(v))
Step 3: Upper Bound. Clearly,
Pr[Success(v)] = =Pr PP([ n (1()) 2 T.]] < Pr n (1(w)) 2 Te. ew
iEW
In other words, the event that all watermarked tokens exceed т, (and hence are masked) is the maximum possible success scenario for the attacker.
H.4. Corollary: Optimal Threshold in Randomized Watermarking
Corollary H.5 (Optimal Threshold under Random Watermarking). Suppose a watermarking scheme targets tokens above the y-quantile I,. If the attacker sets Te I,, then asymptotically:
Pr[Success(y)] == (1-n) WI,
where v is small factor that captures the mismatch in the shifted distribution (i.e., how many tokens originally near I. drop below Te after probability adjustments).
Proof. Step 1: Setup. Under the random watermarking assumption, tokens are chosen in the top 7-quantile of self- information (i.e., (y:) 2 I.). After adding a small their self-information decreases by most C's (Lemma H.
Step 2: Define 1). Let
7-Pr <o [(:)) |
T. R3 In and is small, is small because the shift from I(ys) to I' P'((:) minor.
Step 3: Success Probability. By Theorem H.4, each watermarked token has at least (1 (1- 17) probability of lying above Te, so Pr[Success(y)) 2 (1 - DMI
Similarly, it cannot exceed the intersection probability that all watermarked tokens are above Te, and (1 - [W is good approximation when < 1. Thus the success probability is high.
The lemmas and theorems above show how minimal probability boosts (d) ensure that watermarked tokens remain in the high self-in seff-ifformatin region. By selecting a threshold т, near that region, the attacker can mask or replace the majority of these tokens. Key points include:
23

---

Self-Information Rewrite Attacks on Text Watermarking
Local Context Benefits: measure.
Self-information depends on the context, making it more precise than global entropy
Small Requirement: Watermarking must keep d small to avoid degrading output quality, which in turn prevents drastic drops in self-information for chosen tokens.
Success Probability Bounds: Theorem H.4 establishes that success probability can be arbitrarily close to 1 for an appropriate threshold.
Near-Optimal Threshold: Corollary H.5 suggests an attacker should roughly match the watermark's 's targeted quantile for best results.
The theoretical proof for the self-information rewrite attack shows that by a well-chosen filter we could remove the watermark in the given text effectively. Our lemmas and bounds show that, under small-ò constraints, high-entropy tokens remain detectable as high self-information tokens. This explains why token-level self-information filtering outperforms global, context-agnostie entropy filtering. Consequently, an attacker can remove most or all watermarked tokens while maintaining strong semantic coherence in the final paraphrased text.
I. Visualization
In this section, we present a visual comparison of our algorithm with other model-based paraphrasing methods, along with the corresponding z-scores after the attack. For discrete methods, green tokens are marked in green, and red tokens in red. In the watermarking algorithm, the detector identifies the embedded watermark through green tokens and calculates the z-score; fewer green tokens or a lower z-score indicate a more successful attack. For continuous methods, the shade of color denotes the weight of the watermarked token, with darker colors representing higher weights. In the case of attacked text, lighter colors indicate a more successful attack.
Zieer 1.M
Soae 14
Z.here 5.3
you comutityy.
along
esmpla
ber Jet perple betoo
pesfenteent
broe
in
ceilyy onoecied
bayle the
2E my 1ile
thoggtt
and
heve
adolown
with
rimpoded
soiiduta
the ouaitas
geople your
These Shipps
pareportios
popple perting to rut siee
metting toley
extherition betien that the
text also
sibet sbott the uepect
you cask 15%1
sctvestly lesting Cinsia
tistiic Falidy rately
Zeck Puil viee
bin palliee
Stg0tigheed
applevtion
Lestocts
Figure 4. Comparison of different paraphrasing methods on KGW watermarks. Each word's color indicates whether it a green or red token. Fewer green words/lower z-scores suggest a more effective paraphrasing approach. The unwatermarked text represents the model's 'ss output without the influence of the watermarking algorithm. The example demonstrates that our method achieves better 2-score than the unwatermarked text..
24

---

Self-mnnfonttionn Rewrite Attacks on Text Watermarking
ttegan entmarieed
Deggrnn Siicarec
Uragnn DIPPER strce
Uhigan aboct
2.Sort 1:22
S 2.3
Sohon 4.5
Sooo 3:2
officnce
mazzive Lovelring
this specaion Sheve Massa
exsilaple to stllcradaad
indicted
pecently
peateeday in schona to
Yerk sbace group of allegelly
Io shaie Lasaas qaan the
entyy cbiain veldere checks
sepether to frest enkly
split pageanta sacpg
dbtaia bezedita
payseata snong thenasires
officiale
Done and thaee accoup3 inw in wiyy mor marly
the geoecsamnt the
penaciabbl tuccacoond aEtez
desppinn
weet alead
"Tey retrened deninten
yoars.
-point victory agutart Boston
by dive people. proracutoce
Individiale foand guiley in
people. sccording
Gillage resking iedeaption
for Shase loss apeanss
opezetire vhe plaaded quilyy
s0ach atree
von pleaded guiliy
to axsisting chtaining
decode involved sevezal kay
using pheny information
Eight
paur Peea the begiening
click fales
indldiia
laselita
undereeee spenal
food film people
Tetsh displeyed thete Forms
pooeddag
and deseraination apainrt the Eagles
RCEay foed the fine
Incorration sec0se Sale
mndecconac aguot sia uaed plany Legitsmete bank
people
Segles vto bod ber the top
benetity himsel
specation thc une Calae
othaze Anotar Individud
accouste and Lorest stookt and other
alquicioaat pertion of the best Atlactic Division
infomacion
used Cake Linniiie
from lay sieste benk accunto
witbdree wany froe
thrm people sorsee of retetag
--iille dadeoding
prodacte: these individiala
Laveat in vacions acseca.
Zocped dE stoles docial
Vale
actirs
Freeey
ctterz were acrested for
groy
Besuruty
paattee bask
geen scu.s
sgaten coepete for the top laagee
forged doctal
using atolan Social fecurity
othar aaate of Laund
caicially. the seet of di Leaque
fonds
steles bask ccceens and
and sunder money.
his ottice frr pedd 1aog the
the Ieiab lngetinete ocacesdaza da BCAA making them
Additiseally
3ia
agponbeeded 3ix vocaplace
and sedeccoves undeccoves
willing
silling Colae
openi vin used Carged Becia
competation fer Seice Dase Leagus ergoenacy
infomaciin be used
Infomaation used in achane and indeccoves
Security canda and corged
indeccoent aguat
alquatazas to open bank
Nilla this say ase
prodect
Gecurity canda and torged
1aing tabe Sceiel Cerority Excped
der Eshe stack eepocto the
Dase capabslities
algnatazus
tasa advsotags
and
open bank
spmentions of de
reporti.
meary
proencutocs axid. AlA
zeporte. abar
.C.. aaid cent that the Q. attorma
presesson
Attamay
Q- Attomay
Wite 300gected
Hov York called the cape beaone Siels
Tock: desezibed the case
laugust
larguet
and asinee thadt iacopic
that
incovreed ccentry this
inconaced date O.
Figure 5. Comparison of different paraphrasing methods on Unigram watermarks.
25

---

Self-Information Rewrite Attacks on Text Watermarking
EPV untomaatat
UPV SIRA stade
EPR RIPPER stutk
UPV atode
2 Score: 1::
ScSor 255
S- 42
Recood Lov and begacive
Lowe Lecoceet catea
toreat conaldaced aade
Accooding
Antasses cates ss skssstapot
deconlopen
reppart says Da sepaut
octernsee beelang oux nearch for ylald that la Sutulag pesalion
yield-
Ecency- proepting Loveatoca
Cappiad Kacheta saga that the
develeped accoorina
ether cosservative
assets. Rcoseding
yield D. bond funts has
Ansistenional
tnrtitational investore
recect repert from
Sccoed inatitertionl money
perzice frnda awa sace
nafe
Hesageestt
of atooks partioulacly
mesonding reportt
Caoala sleee intornst
Aaset
Hazagement zage in as acticie
traditioonl aade- har box
clees par cast 9.00 aspacts
pace for plald- in band
seliting
proses the 2019-
aelantiozz pursitt retuma
Baving gropple with
atocka particulaciy
negacive
canada lntarees
pernise foods 1iln
Juon Anrgee
conpestie
peroest:
hare dallac
Cacada sbace Interwet
oeeent Pade and zoreeeign
beed Date vill costinee
prodicto teether bord fudde by
westes
The aspet brether
ficm Cocecuste a rrhetantial
have 2o noney
explatze that with sepativn piolde bande
decllie in bard find
they
vield for bands peeation
Invettment by the of 2310 citiig
neterns eath paping
Finds and ather
conpantes.
120 ealing pocspect of pepacive
high
retual fands Lisuraccs tudd
5oo Each cast Lovest and
Lavaating das Degacive
yiolds
high
woalth runto no p0r4 sons
Exndo incloding
and high costa Bitber
rese20. loveotero
Insmaazos and zoneraigs welth funde
they ace looking for higher
sesking Nightr zetures is other classes and
incalt,
have wibber ao eash
Dree investncs are Left With
Iniding assets or pisking
eroiding bend Exods in developed conntzieo
sepative
options
and high coste oegaciva
Laveating in saaeca vi13
Feod fieds ade baing alea desellape
Azset Hasague discussed clese
infotantta]
the Investment bark
The Clobe Mell
Figure 6. Comparison of different paraphrasing methods on UPV watermarks. The color of each word indicates whether it helongs to a green token or a red token. Less green signifies a more effective paraphrasing approach. Our methods show better performance in removing original watermark text green token.
EWD ratzunalal
EWD SRAstaak
EWD DEPER stack
EWD oTaaad
7:5com: 3.18
2.5core-2.78
7.Score 4.00
2 Scoor 3
Pog these foe' Lxn,
plans to Leguse pazaltiee
pankle pighitegatton
And' isn
plane geoceduze)
seddings sccceding
plazning esploy
sectio tu fucce floor vata
aide. The tastle wlich
proondaral rasouree to peab poopores
pencah
typoes
deetez participates
weddtage to flone..
delaber stive pooceza vocld
suddingss
align Log with Dessta
Like up with hill aloundy iie flaer
far dectuse sho participata
aide...
weuld vimibarly
which woold hypare the
Ane wich sestern Mesrabile
weddd Iepons the
per aide Thie
bypazz
sould pat, dial in Line
link Redical
paodurm pay voddings
dalata
snclistong 1p to your peiion... In
Gezate flooe. saich vould
-ddecriatiattinn Lave beard
curvently
Hepabliooss
thee
1ivelthood of bea2thoace prodursionale
decooce Sicensen to the
conducting eazzlage
Vessangton, oty maczlage 19 alreaty
bG1
and Pontoggtnn D 2a0e
Bcawe Tepehlicuns intendered
would
zecogaized. clase the fara of the hill on
-gendez mauziage
1211: alntog 1isk doctop
to Legal
recoqpiaed
Hoomsee with lapal
pesual octentation Coccently
penaior sxcertats das 25 the complesity gping
abie to being as Obanscaze
czsentation. Forz ently pxy Raculage
Attormable Care
sbetier woeld
stetes Yeahengten
Hashiagton
Scoe Tepsblicans brvever.
neveral Depoblicans
vnclase 20M
exploring
escous sxpescced
shan the
DOP Aoald naw in
lacersgs the mose vate at in the cradog yeazz.
to heinng
acope
ZLOOE.. And pazzage Sea 201
DAET Zer yeare
2C You have
Sceze floce Purthaenoee
Figure 7. Comparison of different paraphrasing methods EWD watermarks.
26

---

Self-Information Rewrite Attacks on Text Watermarking
EXP siPrann
EXP SIRA di
EXP DEVER stack
EX0 OPT stac
2-Scre 3.00
Z Scor
ZoSeer
Z-Stor:
they
PEppLiNe
at vhale ala Class accay
ACCAyE are 1A CICREDD
KEED Add HAE ALROR the
nectod of lechal Anpection Meassnsupps
stervep- so
Digisal
perpose
bryinn Aept sto
Ang1 1d
prodects
opheld by zodern. oppeats ocEET in her octeaze suled that Lematec vire not dented dray
scate- storage eupplisc
acale atoeage apeciallet
LNCOMAEN CLU Capacity
dasilias with
Enconsco liCnepar Ite
to prinary goal bern io
Siree The pessary etgechive
MM
chenslo ofteen the Jouee
eegeet
atccage aslutions and beeat
salance ite extating ecseage
atccage and ENEVEC mudeet
complisses nith the Eigat Ansenteent prosuuttm creat urcavsl pentatment
rremiii dank reperstyy
ettersego Axtrease
and SORETODE prefermene Jevel the compaciana to the diak
longerity sts chespis...
offent lorer ASP the ouperiie
Lepcora 11ferpan itc
five - jod ge panal decemsized that the
rddicisrally
stecage eacieto and hoee
TO aima to comter belance
Ritch of these aimo 10 TO libaly
plaintsfts clever by Lonsted
pacfocmance
peesauee
IEC Wiick of these quals
pezios in the ocoedity
have another cleap chipsec in development poienhied ND flash
crsntitation to dalay the crcoution Buthens Cueteers Chaebeez had been charped with Aaclas ssscal
kight MD00 oucoued
stecage nuchet sptinter
Doea have soother chesp
nanch
chipost is the vocks
of hard diat deions Lonarar 40
kook aice but X
Look
cectalety. bet bere
IC Mill be sble to schiove
1i2e ents sreally
doc
the denths of frez vrorz Mixsinsipp-- and Lau
Lie cumiic abmul. Asd.
be working
iold
stanage chip-
Country Sale
Figure& 8. Comparison of different paraphrasing methods on EXP watermarks. The color of each word indicates whether it is green or red token. For EXP, lighter word colors and higher z-scores indicate more effective attack.
Table Comparison of watermark algorithms under different attack methods on the OpenGen dataset. Our proposed methods SIRA-Timy and SIRA-Small outperform all previous paraphrasing- based attacks under black-box settings.
Attack Success Rate (%) on OpenGen Dataset
Watermark
KGW-1
Unigram
UPV
EWD
DIP
SIR
EXP
Attack
Word delete
21.8
0.8
9.6
17.6
64
36.8
6.6
Synonym
77.4
16.8
67.8
72
98
71.4
47.4
GPT
69
58.2
57.4
73.4
98.2
58.8
74.2
DIPPER-1
89.4
67.8
71.4
88.8
98.8
74.6
83.2
DIPPER-2
89.2
71.2
78.8
92.2
99.0
72.8
85.6
SIRA-Tiny (Ours)
92
84
74.8
94.2
99.6
74.6
81.8
SIRA-Small (Ours)
93.8
91.2
80.6
94.8
99.6
80.2
86.2
J. Extend Experiments
We conduct additional experiments on the OpenGen dataset (Krishna et al., 2024), which consists of sampled passages from WikiText-103. Specifically, we use subset of 500 chunks as prompts, following the same experimental protocol described in our main vvaluation-nameyy we prompt a target L.LM with each chunk and assess the ability of different watermark removal methods to induce decoding failures in the watermark verifier. We report the attack success rate (ASR) as our primary metric. As shown in Table 9, our proposed methods consistently achieve the highest ASR across all watermarking algorithms, demonstrating strong generalization and robustness beyond the training or development set used in previous sections.
We observe that the performance of DIPPER improves significantly on the OpenGen dataset compared to its performance on C4. We hypothesize that this may be attributed to distributional similarity between OpenGen and the supervised training data used to train the DIPPER paraphraser, as both originate from the same source corpus introduced in Krishna et al. (2024). Despite this advantage, our proposed method SIRA-Smal still consistently achieves the highest attack success rates across all watermarking algorithms, demonstrating stronger generalization to diverse data distributions.
27

---

Self-Information Rewrite Attacks on Text Watermarking
Table 10. Attack success (ASR) Adaptive Watermark and Waterfall Watermark. Sample size 200. Our methods outperform all baselines across both settings.
Adaptive Watermark
Waterfall Watermark
Attack
ASR (%)
Attack
ASR (%)
Word delete
5.6
Del
4.4
Synonym
92.4
Syn
55.6
GPT-4o Paraphraser
61.4
GPT
80.0
DIPPER-1
60.6
DIPPER-1 1
73.8
DIPPER-2
65.6
DIPPER-2
80.0
SIRA-Tiny (Ours)
96.2
SIRA-T (Ours)
88.4
SIRA-Small (Ours)
98.2
SIRA-S (Ours)
90.8
To further evaluate the generalizability of our proposed methods, we conduct additional experiments on two recently introduced watermarking schemes: Adaptive Watermark (Liu & Bu, 2024) and Waterfall Watermark (Lau et al., 2024). Following the same black-box threat model, we apply both baseline and our proposed attack methods to 200 samples from C4 dataset for each setting. As shown in Table 10, SIRA-Tiny and SIRA-Small achieve significantly higher attack success rates (ASR) compared all baselines, including the strong GPT-4o paraphraser and DIPPER variants. Notably, Adaptive Watermark, SIRA-Sman reaches 98.2% ASR, while the strongest baseline only achieves 92.4%. Similarly, for Waterfall Watermark, CIRA-Small obtains 90.8% ASR, outperforming the closest baseline by over 10 percentage points. These results demonstrate the superior robustness and transferability of our attack methods across different watermarking designs.
28


Reviews:
[{"review_id": "kOHMj01Qks", "reviewer": "Reviewer_RVPR", "summary": "The paper introduces SIRA, a novel text watermark attack method that leverages the concept of self-information to efficiently and effectively remove watermarks from text generated by large language models. The authors conduct systematic experiments to demonstrate the effectiveness of their approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer **RVPR** for the valuable time and effort spent reviewing our paper. We elaborate on the questions raised by the reviewer below:\n\n> **Q1: Clarification regarding claims: SIRA-Tiny method outperforms all previous approaches**\n\n**A1:** We would like to clarify that this statement only refers to the fact that SIRA-Tiny outperforms previous methods in terms of ASR under the same watermark strength. We agree that a more precise statement would be better. We will revise the text to reflect this more precise phrasing to avoid overclaim and add a corresponding note in Appendix C, Table 6.\n\n> **Q2: Lack of support: Claim SIRA-Large achieves better quality than DIPPER2, while Figure 3b shows that SIRA-Large has slightly worse quality than DIPPER2 on the same watermark**\n\n**A2:** We would like to clarify that in watermarking, text quality is typically referred to the perplexity metric, which is shown in Figure 3a. In contrast, Figure 3b, which the reviewer referenced, reports S-BERT scores that evaluate semantic preservation, a different aspect from text quality. Therefore, we respectfully argue that our claim that SIRA-Large achieves better quality than DIPPER2 remains valid. We appreciate the reviewer's feedback and will revise the related part more clearly to avoid potential misunderstandings.\n\n> **Q3: Computation regarding the cost**\n\n**A3:** We provide a more comprehensive cost analysis here and will include it in the relevant section of the manuscript. Specifically, we estimate the cost of processing 1M tokens of watermarked text using third-party services. According to OpenAI’s pricing, using the GPT Paraphraser (GPT-4o) would cost \\$10 × 2 = \\$20 (input + output). In contrast, our method (SIRA-Small) costs \\$0.22 × 2 (input + output) × 2 (two iterations) = \\$0.88, based on the AWS Bedrock LLaMA3-8b pricing. And the cost could be further reduced if we use SIRA-Tiny (LLaMA3-3b).\n\n> **Q4: Why is the self-information result different from probability?**\n\n**A4:** We clarify that the self-information in our method is conditional and chunk-based (e.g., calculated and percentile by segment), whereas the raw token probabilities are derived from the LLM’s output logits. This means that the statistical basis for the two measures is not aligned: the conditioning context and granularity differ. As a result, percentile ranks based on self-information may diverge from those based on raw probability. This mismatch leads to divergent percentile rankings and explains why filtering using self-information yields different token selections than filtering using raw probability. We will release our code to ensure reproducibility and transparency, and we appreciate the suggestion regarding potential numerical differences, as we will further increase the sample size to reduce numerical instability.\n\n> **Q5: Why not measure the similarity with respect to the watermarked text instead?**\n\n**A5:** **For the attack cases shown in Figure 3b and Table 2, our SBERT-based similarity is indeed computed between the attack text and the watermarked text as proposed by the reviewer**. As SBERT needs one pair of texts, \"No Attack\" is a special case; comparing the watermarked text to itself would be meaningless. Therefore, for “No Attack” we compare the watermarked text with the non-watermarked text. This term aims to reflect how the watermark algorithm changed the semantics. We acknowledge that this was not clearly explained in the paper. We will clarify this to avoid potential misunderstanding.", "claims_and_evidence": "The paper highlights the vulnerability of watermarks placed on high self-information tokens, which forms the basis for SIRA as a targeted attack method. This claim is well supported by the results presented in Table 3. The authors also argue that self-information is better than entropy for identifying watermarked tokens, which is backed by the experiments in Appendix G, Table 9. Another claim is that even when the distribution of the attack model differs from the generation-time distribution, SIRA can still identify watermark locations, and larger attack models better estimate the generation distribution, leading to improved watermark removal. This claim is supported by the results in Figure 2.\n\nHowever, there are a few claims that lack strong support. For instance, the authors state that even their most lightweight SIRA-Tiny method outperforms all previous approaches, but this comparison is tricky due to the varying attack strengths and inconsistent rankings across different metrics (e.g., Appendix F shows that on the Unigram watermark, SIRA-Large achieves better rewrite quality than DIPPER2, while Figure 3b shows that SIRA-Large has slightly worse quality than DIPPER2 on the same watermark). Another claim that lacks clear evidence is the $0.88 per million tokens cost, which is highlighted in the abstract but without a clear calculation provided.\n\nThere is also a claim in Appendix G, Table 9, that compares filtering potential green tokens using self-information, entropy, and probability. \nI fully understand the self-information vs entropy part. The authors also attempt to prove that self-information is more accurate than probability, but I didn't follow this part. Since self-information and probability have a monotonic relationship, using percentiles should yield the same results. Without the code to reproduce the experiments and find the details, it is difficult to understand this discrepancy. The authors should clarify this point and potentially consider the possibility of numerical instability in their experiments.", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in the paper seems to be sound and appropriate.", "theoretical_claims": "Although Appendix H contains some bounds, they are not strongly connected to the main text. The paper's main contribution lies in its practical aspects rather than its theoretical claims.", "experimental_designs_or_analyses": "Some of the claims in the paper are well-supported by the experimental designs and analyses, while others have weaker support. Please refer to the \"Claims And Evidence\" section for a detailed discussion.", "relation_to_broader_scientific_literature": "The paper cites and compares its approach to many relevant baseline methods.", "essential_references_not_discussed": "I am not aware of any.", "other_strengths_and_weaknesses": "The paper presents a novel method and conducts systematic experiments to support its claims. Some claims are well-supported, while others have weaker support. Please refer to the \"Claims And Evidence\" section for a detailed discussion.", "other_comments_or_suggestions": "I have no additional comments.", "questions_for_authors": "I noticed that the current s-BERT measurement compares the similarity to the non-watermarked text, which leads to the observation that the s-BERT score after paraphrasing attacks is even higher than the no-attack case. Also in Table 2, with more attacks, s-BERT score is higher. This is because strong paraphrasing models are used, making the score less informative about the degree of semantic preservation after the attack and more about the ability of the paraphrasing model. Why not measure the similarity with respect to the watermarked text instead?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHYjsCEoMQ", "reviewer": "Reviewer_zCjb", "summary": "This paper studies how to remove the watermark in text generated by LLMs. It assumes the watermark is injected through the high-entropy (self-information) words. There it first uses an auxiliary mode to compute the self-information for each token in the generated text. Then it masks out the tokens with high self-information and fills in the masks according to the paraphrased text. Experiments show it outperforms baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer zCjb for the time spent reviewing our paper. Due to the reply character limit, we address the main concerns below and put the reference in anonymous link: \n> Q1: No theoretical analysis or direct experimental evidence on the high-entropy assumption\n\nA1: The use of high-entropy embeddings is not an assumption but a well-established design choice in prior watermarking research [1–4, 6, 7]. Section 3 of the KGW paper [1] provides a detailed theoretical framework. Notably, we are the first to identify this design as a potential vulnerability exploitable by attackers.\n\nTo further support our results, we extend the ablation study in Table 4 with mask text and report the z-score, which reflects the green token remnant.\n\n| Text               | Attack Success Rate | Average *z*-score |\n|--------------------|---------------------|-------------------|\n| Human-written Text | N/A                 | 0.12              |\n| Reference Text     | 64%                 | 3.75              |\n| Attack Text        | 94%                 | 1.85              |\n| Mask Text|100%| 0.66| \n\n> Q2:  Clarification regarding “targeted” attack\n\nA2: We clarify that prior paraphrasing attacks, such as DIPPER [5] and GPT-Paraphraser, rely entirely on the LLM to rewrite text without control over which parts are modified, making them untargeted. In contrast, our method treats watermark removal as a targeted problem, selectively rewriting tokens likely to carry the watermark signal. Experiments show this targeted strategy is more effective.\n\n> Q3:  The motivation gap. \n\nA3:  We clarify that the use of “high entropy” in prior work is heuristic. Specifically, none of the existing watermarking methods explicitly compute entropy, making it reasonable to explore metrics under the heuristic. Entropy and self-information are mathematically related, we find that self-information provides a more direct, token-level signal that better aligns with the actual greenlist selection process. Therefore, our motivation and implementation are consistent: we empirically identify a metric that captures the core vulnerability and reflects the intended intuition.\n\n> Q4: Why can one assume the model predicts the next words with equal probability in Appendix G?\n\nA4: We clarify that this is a deliberate simplification of high entropy and our formal proof, Appendix H does not build on such an assumption. The uniform distribution serves as a theoretical upper bound for high entropy for a given probability space and helps illustrate high-entropy scenarios where probability mass is thinly spread. In such cases, even small probability changes can significantly affect self-information. Thus, this approximation highlights an extreme, simplified case to clarify the scaling behavior under high entropy for the reader not familiar with watermarking.\n\n> Q5: It's not reasonable that the attackers use much larger models.\n\nA5: Our work is aligned with established settings in prior related studies [5–9]. We emphasize that watermark robustness mainly depends on the algorithm design and hyperparameters, not the choice of generation LLM. The mentioned work adaptive watermark in [4] uses the same evaluation. Notably, our method significantly reduces computational and time costs, as shown in Appendix B. We would appreciate references that follow the reviewer's proposed setting regarding LLM watermark robustness evaluation.\n\n> Q6: Evaluate the accuracy of the detected watermark tokens\n\nA6: Please see A1 response.\n\n>Q7: Adaptive watermarks experiments \n\nA7: We appreciate the reviewer’s suggestion. We show the proposed results below; the experiment follows our main experiment setting, and the threshold is 0.75. We report the attack success rate. **As explicitly mentioned in the abstract of the work[4], this watermark is embedded in high-entropy text.** We would appreciate it if the reviewer could clarify what “high-entropy loss” is and could provide a specific reference that did not adopt high-entropy embedding, and we will try our best to include the corresponding experiments during the discussion window.\n\n| Adaptive Watermark  | ASR    |\n|---------------------|--------|\n| Word delete         | 5.6%   |\n| Synonm              | 92.4%  |\n| GPT-4o Paraphraser  | 61.4%  |\n| DIPPER-1            | 60.6%  |\n| DIPPER-2            | 65.6%  |\n| SIRA-Tiny           | 96.2%  |\n| SIRA-Small          | 98.2%  |\n\n\n> Q9: Why can the watermark defenders see the attack text when they design the detector at Line 192?\n\nA9:  We thank the reviewer for pointing this out. This was a typo — the goal of the detector is to distinguish between watermarked and non-watermarked text. We will revise this part accordingly.\n\n>Q10: Typo, figure and symbol unclear\n\nA10: We sincerely thank the reviewer for pointing it out. We will correct them in our revised manuscript.\n\n---\nAnonymous Reference Link: https://docs.google.com/document/d/1t1HxJ5KkCydhf_AwUdqOC8R1c0bwY024Ryr_iTZo-2A/edit?usp=sharing", "claims_and_evidence": "1. The authors assume existing watermark methods embed patterns in high-entropy tokens and thus base their method on this assumption. However, it's unclear if this is true because there is no theoretical analysis or direct experimental evidence. For example, the authors can add an experiment to directly verify the precision/recall of the masks generated in the first step with the ground truth tokens.\n2. Similarly, the authors claim it's the first targeted attack. This \"targeted\" feature should be clearly defined and validated.\n3. There is still a gap between the motivation and the theory. From the beginning, the authors always emphasize that existing watermarks use high-entropy tokens. However, later on, the authors analyze the theory and empirically show that self-information is a better indicator. However, from the motivation, it seems high entropy should give better results than self-information.", "methods_and_evaluation_criteria": "make sense.", "theoretical_claims": "Line 997, why can one assume the model predicts the next words with equal probability? Is it reasonable? Please justify it.", "experimental_designs_or_analyses": "1. It's not reasonable that the attackers use much larger models, such as Llama3-3B-70B, to attack watermark text generated by a small model Opt-1.3B. In this case, the attackers can directly use the large models. So a more reasonable attack scenario is the attackers only have limited resources so they can only have a small model, while they want to leverage the capability of a larger model. However, the larger model has watermarks. So they want to use the small model to remove the watermark text generated by the large model. Therefore, it's suggested that similar rules be followed in the evaluation.\n2. As mentioned earlier, it would be better to evaluate the accuracy of the detected watermark tokens. Because this is one of the contributions claimed in this paper.\n3. It's suggested that the authors evaluate or at least discuss the adaptive watermarks. For example, the watermarks that try to leverage other tokens in addition to the high-entropy loss.", "relation_to_broader_scientific_literature": "Help people design stronger watermark techniques.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. An interesting idea and experiments show it has high ASRs.\n2. Overall, it's easy to follow.\n\n\nWeakness:\n\n1. It's unclear why authors have several definitions in Section 3.1, especially equations 2-4. Because most of them are not used in the following discussion or design. Why can the watermark defenders see the attack text $y_p$ when they design the detector at Line 192?\n\n2. The figures in the evaluation section are not very visible.", "other_comments_or_suggestions": "Some symbols are used without introduction, such as $Y_w$ at Line 191.\n\nTypo:\nLine 1030: \"the green tokenthe\"", "questions_for_authors": "Please refer to the above points.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ADtOEuXhZx", "reviewer": "Reviewer_P6b2", "summary": "The paper aims to erase text watermarks from LLMs by proposing a novel rewrite attack utilizing self information. The proposed SIRA could achieve almost 100% success rate across various watermark algorithms. Specifically, SIRA calcuates self information of every token in the watermarked sequence, where tokens of high values are masked in the output. To complete the masked sequence that includes all information from the original sequence, a paraphrased version of the original sequence will be used as a reference. Experiments show that the text quality is well preserved, while maintaining fine complexity and budget.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer **P6b2** for the appreciation of our work and the efforts spent to review our paper. We address concerns and questions below:\n\n> **Q1: Text quality slighter lower than GPT paraphrase**\n\n**A1:** We thank the reviewer for their suggestion and will revise the description of our experimental conclusions accordingly. We fully agree with the reviewer and would like to clarify that text quality is largely influenced by the choice of **paraphrasing model**. In our case, we adopted GPT-4o; the inherent ability difference of LLMs causes such a gap. For the stronger LLM like LLaMA3-70B, it outperforms GPT in two watermarks. One advantage of our method is its transferability: unlike DIPPER, our approach is training-free. As more powerful LLMs become available in the future, our method can be seamlessly adapted with zero cost to take advantage of better text quality.\n\n> **Q2: Does this missing prompt affect the calculation of self-information?**\n\n**A2:** This is a great question. Our answer is no—the calculation of self-information does not need such a prompt. The prompts used to generate watermarked text in prior works such as DIPPER [1] and Random Walk [2] are typically referred to as “context.” However, our goal is to minimize assumptions in order to make the method a simple and broadly applicable tool. In the results we present, we do not use any such context prompts. In earlier studies, the original prompts were primarily designed to help paraphrasing models better preserve semantics through tailored instructions. We consider exploring the impact of prompt design on our method as an interesting direction for future work.\n\n> **Q3: If the given watermarked text is cropped from a long output, will the missing context affect the calculation of self-information?**\n\n**A3:** Yes, but it depends on how long the cropped text is. The missing context theoretically affects how self-information is calculated since it depends on the exact preceding context. However, our watermarking approach remains effective in practice because it primarily relies on local context within the segment, rather than requiring the entire text. Consequently, cropping a reasonable portion of the output does not significantly degrade our detection or performance unless the given text is too short or the different segment contexts are unrelated. In such an extreme case, our method degrades to a performance level similar to that of random masking.\n\n> **Q4: Using instruction instead of reference text**\n\n**A4:** We thank the reviewer for the suggestion. This was indeed one of the approaches we explored during the preliminary phase of our study. Adding explicit instructions to encourage the LLM to use different words proved to be ineffective; a large performance gap will exist, especially for lightweight models such as those at the 3B scale. We believe this is because such a small model cannot fully understand complex instructions. While this approach performs comparably on larger models (e.g., 70B), it still underperforms compared to using a reference text. This is likely because, as mentioned in lines 256–260, we instruct the model to do tasks similar to filling in blanks, and due to the high similarity between mask text and watermarked text, LLMs tend to take shortcuts by copying directly from the original text; instructions are not sufficient to fully prevent this behavior. We will continue to explore whether better prompt design can help to achieve this goal.\n\n---\n\n**References**\n\n[1] Krishna, Kalpesh, et al. \"Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense.\" *Advances in Neural Information Processing Systems* 36 (2023): 27469–27500.  \n[2] Zhang, Hanlin, et al. \"Watermarks in the sand: Impossibility of strong watermarking for generative models.\" *arXiv preprint arXiv:2311.04378* (2023).", "claims_and_evidence": "It seems the text quality, while superior than most baselines, is slightly lower than the GPT paraphrasing. In the paper, you claimed that your method has smaller impact on text quality than other baselines.", "methods_and_evaluation_criteria": "Since the method is completely black-box, which indicates the user prompt when generating the watermarked sequence is not available as wee, does this missing prompt affect the calculation of self-information? Also, if the given watermarked texts is cropped from a long output, will the missing context affect the calculation of self-information?\nI wonder if the parapharsed reference text is necessary as maybe during the masked sequence completion, the llama model could be instructed to avoid using exactly same wording from the original text. If this step is unnecessary as you use the original text as reference, the cost and time consumption could be further lowered.", "theoretical_claims": "No issues.", "experimental_designs_or_analyses": "No issues.", "relation_to_broader_scientific_literature": "No issues.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "It is nice that proposed method is a plug-and-play & black-box attach method while many previous works have more assumption on the knowledge of watermarking generation process.", "other_comments_or_suggestions": "No issues.", "questions_for_authors": "Please refer to previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aqAEWOhwFZ", "reviewer": "Reviewer_7XjW", "summary": "The paper proposes a new attack against model-watermarking algorithms that involves first identifying tokens in an LLM output that have high self-information, and then passing the output to a paraphraser that changes these tokens. The hypothesis is that these tokens are also the tokens that most likely contain the watermark signals, hence such targeted paraphrasing could improve the efficiency of paraphrasing attacks against model watermarking methods. The paper presented empirical results to support their claims, showing that the proposed method can achieve much higher attack success rates compared to benchmarks, at a relatively low per token cost.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to Reviewer 7XjW for the thorough and detailed feedback, due to space limit, we address the main concerns below: \n\n> Q1: A Pareto plot, the use of \\epsilon, Line 1041\n\nA1: We greatly appreciate the reviewer’s suggestion. We will add the Pareto plot, correct the misuse of \\epsilon, and revise Line 1041 to avoid confusion.\n\n>Q2: Directly computing how well the proposed metric predicts tokens will be helpful\n\nA2: Due to space limitations, please see the reply to zCjb A1.\n\n>Q3: It would be useful to see the naive baseline results of a paraphrasing attack.\n\nA3: We clarify that this is already included in our paper. As shown in Table 3, we compare our method to a naive baseline where the LLM paraphrases the input twice. This aligns with our approach, where contains two paraphrasing for reference text and attack text. Results show that our design outperforms naive baseline.\n\n>Q4: Possible explanations for the empirical and theoretical\n\nA4: We clarify that prior works claim to embed watermarks in high-entropy is a heuristic description, **they do not explicitly compute token-level entropy to build the greenlist**. Instead, greenlist tokens are selected via a hash-based pseudorandom process conditioned on past tokens and a key, implicitly simulating high-entropy selection.\n\nFor example, in EWD [3], The high entropy concept is represent as a renormalization step:\n\n```python\nrenormed_probs = probs / (1 + z_value * probs)\nsum_renormed_probs = renormed_probs.sum(dim=-1)\n```\n\nThis renormalization acts as soft reweighting that suppresses high-probability tokens and flattens the distribution, thereby emphasizing low-probability regions. As a result, low-probability tokens are further highlighted and tend to receive higher weights regarding the watermark signals. Similarly, EXP uses exponential sampling to favor low-probability tokens, which, once selected, have higher weight during detection.\n\nIn summary,**low-probability tokens are either preferred when selecting a green token or assigned stronger watermark weights once selected**. Our self-information approach aligns more directly with practical implementation, enabling more accurate identification of watermark-bearing tokens and improving attack effectiveness.\n\n> Q5: The formulation in Sec 3.1 doesn't connect watermarking to token-level perturbations .\n\nA5: We clarify that the perturbation function in Section 3.1 refers to the LLM used for paraphrasing. Definition 3 is **not** meant to describe token-level watermark perturbations. It corresponds to the second step of our method, where the LLM is instructed to paraphrase the input.\n\n>Q6:  Why is the green token more important?\n\nA6: The watermark signal is embedded by biasing generation toward green tokens, while red tokens act as a control group. Detection relies on whether the green token significantly exceeds the non-watermarked baseline. Thus, the green tokens carry the actual signal. Regarding distortion, robust watermarking schemes are explicitly designed to minimize it while preserving detectability; otherwise will affect text quality and stealth.\n\n\n>Q7: Other dataset experiment \n\nA7: We follow the reviewer’s suggestion add the OpenGen dataset [2], which samples from WikiText-103. We use the 500 chunks as prompts, following the same evaluation protocol as in our main experiments. We show our results below, and our methods achieve the highest ASR.\n\n| Attack  | KGW | Uni | UPV | EWD | DIP | SIR | EXP |\n|---------|-----|-----|-----|-----|-----|-----|-----|\n| Del     |21.8|0.8 |9.6 |17.6|64  |36.8|6.6 |\n| Syn     |77.4|16.8|67.8|72  |98  |71.4|47.4|\n| GPT     |69  |58.2|57.4|73.4|98.2|58  |74.2|\n| DIPPER-1   |89.4|67.8|71.4|88.8|98.8|74.6|83.2|\n| DIPPER-2   |89.2|71.2|78.8|92.2|99.0|72.8|85.6|\n| SIRA-T    |92  |84  |74.8|94.2|99.6|74.6|81.8|\n| SIRA-S   |93.8|91.2|80.6|94.8|99.6|80.2|86.2|\n\n\n\n\n\n>Q8: Waterfall watermark experiment\n\nA8: We followed the reviewer’s suggestion to conduct the experiment on the waterfall watermark[1]. We use 500 samples from the C4 dataset and select 0.2 as our z-threshold; others follow the same evaluation protocol as in our main experiments. The results shown below, our methods get the highest ASR.\n\n| Waterfall          | ASR    |\n|--------------------|--------|\n| Del        | 4.4   |\n| Syn             | 55.6  |\n| GPT | 80    |\n| DIPPER-1           | 73.8  |\n| DIPPER-2           | 80    |\n| SIRA-T          | 88.4  |\n| SIRA-S         | 90.8  |\n\n>Q9: Weaknesses regarding theoretical \n\nA9: We thank the reviewer for the constructive suggestion. We acknowledge that our work is primarily empirical, and while we attempted to provide theoretical insights, some parts may be limited or unclear. We will revise the manuscript accordingly and de-emphasize those sections as suggested. We appreciate any further feedback to improve the theoretical part.\n\n---\n\nAnonymous Reference Link: https://docs.google.com/document/d/1-VCOEO5eJmrq-_44oGaHdRE7KOGXHumV75h9DDCTP8A/edit?usp=sharing", "claims_and_evidence": "The claim that their method outperforms baselines empirically is partially supported by the existing experiments, though there are some additional results that would make them more convincing. \n- The trade-off between semantic preservation of the attack and the effectiveness of the attack. Currently this is analyzed only in aggregate in separate charts. A Pareto-plot showing the trade-off between attack effectiveness and semantic preservation (e.g. measured by semantic similarity) across benchmarks (rather than just on the proposed method) will help.\n- Related point: the use of \\epsilon is confusing, as it denotes three different things in the paper, i.e., in equation 4 for semantic similarity threshold, in unnumbered equation line 269 for percentile threshold, and line 212 for watermarking threshold.\n- Inclusion of error bars will also confirm that the results are statistically significant.\n- If the claim is that self-information is the best metric to identify tokens with watermark signal, the best approach to analyze this is to directly compute how well the proposed metric predicts which are the tokens that contain the 'greenlist' signals, e.g. by computing some appropriate correlation metric.\n- Smaller point: it would be useful to see the naive baseline results of a paraphrasing attack using the same base paraphraser used in the SIRA methods, i.e. unconstrained paraphrasing (with similar instructions to not reuse the same words in the reference text), to see the direct impact of the identification of a subset of tokens for replacement.\n\nThe bigger problem lies in the proposed explanation and theoretical analysis. \n- Entropy is expected self-information, and it is not true that \"higher entropy is typically associated with high self-information\". Choosing higher self-information tokens based on the proposed method essentially means choosing tokens that have lower absolute probabilities given preceding tokens. Possible explanations for the performance gains may be considered more from this perspective. \n- The proposed detailed discussion in Appendix G/H is unclear that needs to be a lot more detailed and careful with assumptions and approximations made. Statements like line 1041 on \"entropy is context-agnostic\" needs to be properly justified and defined. \n- The authors should be a lot clearer what is the experimental setting and what they specifically did to compute the results for Table 9.\n- The problem formulation in Sec 3.1 does not directly relate to the crux of the proposed method, which is token-level based. The method is broadly on identifying tokens to mask and replace, while the formulation do not related watermarking methods to token-level perturbation (definition 3).", "methods_and_evaluation_criteria": "Please see above point and question regarding direct analysis of the correlation and prediction capabilities of the self-information metric v.s. the 'watermark signal' tokens.\n\nIt would also be better if experiments are conducted on more than just one dataset, i.e. the C4 dataset, especially since the proposed method should be relatively easy to implement for other datasets. The other datasets should ideally cover a different type of text, e.g. not news articles related but perhaps more scientific or literature-based, to confirm that the underlying token distribution of the dataset do not significantly affect the performance of the method.", "theoretical_claims": "Please see above regarding the issues on self-information vs entropy, problem formulation gap on relating token level watermarking to identification of 'watermarked tokens' as a viable attack strategy. Conceptually, basic watermarking methods operating on the 'green-red' list approach also relies on both the green and red lists for signal detection and watermarking strength -- so it is unclear how 'green tokens' identification is more important than 'red tokens', rather than the actual distortion from the underlying word distribution.", "experimental_designs_or_analyses": "Please see issues above on metrics and datasets.\n\nAdditionally, it would be additional validation for the authors to also consider recent robust watermarking methods such as [1], rather than just logits adjustment-based model watermarking methods that adopts 'green-red' list approaches (which the proposed method is deigned directly to attack). For e.g., [1] applies logits perturbations to the entire token space but with varying degree of perturbation to each token based on a hash, preceding tokens and chosen perturbation functions. \n\nShowing that the proposed method can also work for that and generalize beyond just 'green-red list' approaches, will significantly strengthen the validity of the method.\n\n[1] Lau et al, Waterfall: Scalable Framework for Robust Text Watermarking and Provenance for LLMs, EMNLP 2024", "relation_to_broader_scientific_literature": "Empirically, it seems like there is merits to consider this attack in future watermarking works, if the authors can more rigorously justify the proposed method's performance gains compared to existing paraphrasing attacks. It would be useful for the authors to also discuss or even better evaluate other logits perturbation-based model watermarking approaches that go beyond the 'green-red list', such as [1] mentioned above and other similar types of approaches.", "essential_references_not_discussed": "As mentioned, it would be useful to discuss logit perturbation-based watermarking approaches that go beyond just 'green-red list' approaches (e.g., [1] mentioned above and related works). As the proposed method is primarily designed based on the 'green-red list' watermarking model, discussing other approaches that spread the watermarking signal across all tokens would better illustrate the generalizability, or limitations, of the proposed method.", "other_strengths_and_weaknesses": "Overall, I think this has potential to be a good contribution to the literature if the attack is more rigorously backed empirically. The theoretical underpinnings and explanation is problematic, but given that this is a proposed attack model and is primarily empirical in nature,  it may be ok as long as these portions are clarified and de-emphasized in the paper.", "other_comments_or_suggestions": "Please see above", "questions_for_authors": "Please see the points above, on empirical gaps and questions on explanations regarding why the proposed method works.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies text watermarking for large language models and presents SIRA, a novel attack method designed to remove text watermarks from outputs of LLMS. As watermarking algorithms tend to embed signals in tokens with high self-information, SIRA identifies these high-entropy tokens, masks them, and then reconstructs the sequence using a paraphrased version of the original output. This targeted paraphrasing strategy aims to preserve semantic fidelity while disrupting the watermark signal.

Overall, the reviewers found the paper has several merits, including novel use of self-information, effective paraphrasing strategy, strong experimental results. Some of the reviewers raised concerns about the trade-off between semantic preservation and effectiveness of the attack, and lack of comparisons in some aspects.

Therefore, based on the reviews, author responses, and discussions among AC and reviewers, the paper is marginally above borderline accept. I would recommend Weak Accept.
Decision: Accept (poster)