Optimizing Adaptive Attacks against Content Watermarks for Language Models

Authors: Abdulrahman Diaa, Toluwani Aremu, Nils Lukas
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: watermarking, language models, robustness, adaptive attacks

Abstract:
Large Language Models (LLMs) can be \emph{misused} to spread online spam and misinformation. Content watermarking deters misuse by hiding a message in model-generated outputs, enabling their detection using a secret watermarking key. Robustness is a core security property, stating that evading detection requires (significant) degradation of the content's quality. Many LLM watermarking methods have been proposed, but robustness is tested only against non-adaptive attackers who lack knowledge of the watermarking method and can find only suboptimal attacks. We formulate the robustness of LLM watermarking as an objective function and use preference-based optimization to tune \emph{adaptive} attacks against the specific watermarking method. Our evaluation shows that (i) adaptive attacks substantially outperform non-adaptive baselines. (ii) Even in a non-adaptive setting, adaptive attacks optimized against a few known watermarks remain highly effective when tested against other unseen watermarks, and (iii) optimization-based attacks are practical and need limited computational resources of less than seven GPU hours. Our findings underscore the need to test robustness against adaptive attackers.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729601938844

thread_type:
official_review

<thread_object>
creator: reviewer_uf9k

<content>
summary:
The paper proposes an adaptive (finetuned) paraphrasing adversary against several types of LLM watermarks. In particular, an adversary first collects samples of watermarked and paraphrased texts, building a preference dataset based on paraphrases that achieve watermark removal and maintain overall text quality (over a range of surrogate models and keys). With this DPO dataset the adversary then finetunes paraphrase LLM(s). The evaluation shows that this works for (and across) three watermarking scheme families, leading to a higher watermark removal rate while retaining better text quality.

strengths:
- Stronger adversarial evaluations of LLM watermarks are a timely and important area of research for improvements in LLM watermarks. The reviewer agrees with the authors that existing work (with exceptions - see below) has focussed too little on this.
- The setup of the evaluation is realistic, using both surrogate models, an overall sensible text quality evaluation, and general black-box assumptions.
- Results indicate that learning on the preference data achieves overall higher evasion rates at higher text quality.

weaknesses:
- I will focus on Dist-Shift (KGW) and Exp (Aaronson), as they are arguably the most robust and well-known of the four tested schemes. The overall evasion rates, even for basic paraphrasing, seem very high to me. While my personal experience is more with KGW than EXP, the works that I have seen have significantly lower evasion rates (e.g. 0.18 at a FNR of 1e-3 on SelfHash (h=3, Llama 7B) in [1] Table 4 (a setting recommended in [3]) or in [3] itself in Fig.5 ~50% for LeftHash h=1) than the ones around 85% reported in Figure 5. This currently leaves two points unclear for me: (1) What are the exact Dist-Shift/KGW settings used by this work (as it seems rather weak) and (2) How to interpret the advantage an adversary gets from this as, e.g., in Figure 8 the gap of the finetuned models to GPT3.5 is already small (and might be even smaller with stronger zero-shot paraphrasers).
- Some intuitive explanation / qualitative explanation why one would expect this methodology should work for (e.g.) Dist-Shift. A priori, it is unclear why a paraphrasing model should be able to learn to essentially avoid green-list tokens for an unseen watermarking key as the key "randomly splits" the vocabulary into green and red. However, it makes sense as it seems to avoid repeating token structures observed in the input. Transferability between methods could then be explained by this observation being the driving factor across all methods. A qualitative experiment that highlights this and possibly other factors could improve the presentation of the approach and might lead to insight into, e.g., adaptive defenses.
- Unlike some claims in the paper, there is definitely work on stronger adversaries against LLM watermarks, e.g., [1], [2] which explicitly and implicitly assume knowledge about the watermarking algorithm.
### Typos/Nits
- What exactly is the tradeoff in L265: "Higher robustness leads to a higher computational burden for the attacker"? To me, it seems this is a direct consequence of the robustness.
- Instead of going very high in the FPR range (e.g., 10% in Figure 5), It would make more sense in the reviewer's opinion to ablate over low FPR regions as these are the ones where watermarks are most likely applied (e.g., both [1] and [3] also evaluate on 0.001).

[1] Jovanović, Nikola, Robin Staab, and Martin Vechev. "Watermark stealing in large language models." ICML (2024).

[2] Pang, Qi, et al. "No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices" _arXiv preprint arXiv:2402.16187_ (2024).

[3] Kirchenbauer, John, et al. "On the reliability of watermarks for large language models." ICLR 2024.

questions:
- What exact settings for KGW were run, did the authors try to run more robust versions of the watermark? How does basic paraphrasing perform already perform so well?
- Can you give more qualitative insight as to why the approach works well. Are the notes from above (learns to avoid patterns) essentially what you observe qualitatively?
- See other individual points above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729688520387

thread_type:
official_review

<thread_object>
creator: reviewer_pkhu

<content>
summary:
This paper investigates the robustness of watermarking methods for Large Language Models (LLMs) against adaptive attacks. The authors propose methods to optimize attacks against known watermarking algorithms using preference-based optimization. Their key findings show that: (1) adaptive attacks significantly outperform non-adaptive baselines, achieving >96% evasion rate with minimal quality degradation; (2) attacks optimized against known watermarks remain effective against unseen watermarks; and (3) the optimization process requires limited computational resources. The work demonstrates important vulnerabilities in current LLM watermarking approaches.

strengths:
I find this paper interesting as it challenges what many of us took for granted—the robustness of LLM watermarks. What sets it apart from previous work is its systematic approach to breaking watermarks. Instead of relying on manual perturbations or simple models, the authors frame watermark evasion as an optimization problem.

The most surprising finding, to me, is how effectively a modest 7B parameter model can be tuned to fool watermarks from much larger models. This raises serious questions about the fundamental security of current watermarking schemes. Are we perhaps building our defenses on shaky foundations?

The technical contributions are solid:

- This paper is well organized and easy to read
- The paper's key insight is transforming watermark robustness into an optimizable objective function, enabling systematic attack optimization rather than relying on heuristic approaches used in prior work. The strengths of the paper include:
- The preference-based optimization framework is clever as it overcomes the challenges of optimizing over discrete text while requiring minimal computational resources
- The approach is model-agnostic and can be applied to any watermarking method, making it a general-purpose attack framework
- Attack is  practical, requiring only 7 GPU hours and ~$10 of compute
- Novel finding that attacks can transfer to unseen watermarks, suggesting fundamental limitations in current approaches

weaknesses:
While the empirical results presented in this paper are impressive, several important theoretical questions remain unanswered. Most notably, the authors demonstrate that their attacks transfer effectively across different watermarking schemes, but they don't provide a theoretical framework to explain why this happens. This gap in understanding is concerning - without knowing which properties of watermarking schemes make them vulnerable to transfer attacks, it becomes difficult to design more robust systems. The lack of formal bounds or guarantees on attack effectiveness also leaves us wondering about the fundamental limits of watermark robustness.


Another limitation of this work lies in its treatment of defensive measures. While the authors demonstrate impressive attack capabilities that can evade watermark detection with a>96% success rate using minimal resources, they offer little insight into potential countermeasures. Could watermarking schemes be made adaptive, perhaps updating their strategies based on observed attack patterns? Are there fundamentally different approaches to watermarking that might prove more robust? The paper's silence on these questions leaves practitioners with clear evidence of the problem but little guidance on potential solutions.

questions:
I have some deeper concerns that the authors should address:

1. The theoretical underpinnings of attack transferability need more rigorous examination. While the paper demonstrates impressive empirical results showing attacks transfer well to unseen watermarks, it lacks formal analysis of why this occurs. Could this suggest some fundamental limitation in current watermarking approaches? For instance, most current methods rely on statistical patterns in token distributions - perhaps this shared mathematical structure creates inherent vulnerabilities that sophisticated attacks can exploit. A theoretical framework explaining these transfer properties would be invaluable for designing more robust watermarks.

2. Why do these attacks transfer so well to unseen watermarks? The paper shows impressive empirical results but doesn't help us understand the underlying reasons. Could there be some common vulnerability across current watermarking approaches that we're missing?

3. The authors' threat model focuses primarily on demonstrating attack capabilities but pays insufficient attention to the defensive aspects of the security game. Given that watermarking is intended as a protective measure, understanding its limitations should naturally lead to exploring ways to strengthen it.

4. How well do these attacks generalize across different languages and domains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730257217999

thread_type:
official_review

<thread_object>
creator: reviewer_dv9w

<content>
summary:
This paper presents a rewriting attack model against LLM text watermarks. The authors train an LLM using DPO (Direct Preference Optimization) with collected preference data, achieving better attack performance compared to previous rewriting models.

strengths:
1. The paper is written in a clear and well-structured manner that makes it easy to follow the methodology and results.
2. The proposed attack model demonstrates superior performance compared to previous LLM-based approaches in removing watermarks.

weaknesses:
1. The paper lacks comprehensive baseline testing, particularly against robust watermarking algorithms such as UnIgram (Zhao 2024 et al.) and SIR (Liu 2024 et al.).   Since these algorithms are currently generally considered more robust than algorithms like Exp and KGW, experiments on robustness attacks should be conducted on robust models.

2. Several issues exist in quality testing:
- The paper lacks sufficient details regarding the dataset used for watermarked text generation, particularly concerning the source of prompts used to generate data from the LLM. Without clear information about the prompt dataset used in this process, it becomes difficult to fully understand and reproduce their watermarking approach.
- The choice of Llama3-8B-Instruct for PPL calculation appears suboptimal when Llama3-70B resources are available.
- The quality metrics presented in Figures 3 and 4 lack clarity, with multiple metrics mentioned but only single values shown without explanation of averaging or weighting methods. Could you provide a clear explanation of how the single values in Figures 3 and 4 are derived from the multiple metrics mentioned?
- The quality test results appear questionable, particularly where Llama2-7B outperforms GPT-3.5, suggesting potential issues with data selection or evaluation methodology.

3. The paper lacks theoretical insights:
- There is no clear explanation of the mechanism by which the training approach effectively removes watermarks while preserving text quality, maybe more deep into the n-gram is required.  Could you provide an analysis of how your method affects n-gram distributions compared to the original text and other baselines.
- The paper fails to analyze why the proposed method achieves better performance than GPT-3.5, especially the text quality.

4. The evaluation would benefit from including stronger models like GPT-4 and GPT-4o to provide more comprehensive insights into the attack model's effectiveness.


References:

1. Provable Robust Watermarking for AI-Generated Text  ICLR 2024
2.  A semantic invariant robust watermark for large language models  ICLR 2024

questions:
Please refer to the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670613140

thread_type:
official_review

<thread_object>
creator: reviewer_zaxy

<content>
summary:
This work proposes an adaptive attack to remove watermarks by fine-tuning a paraphraser. It shows that when attackers know the watermark method, they are able to achieve stronger evasion attacks.

strengths:
1. The proposed adaptive attack reveals the potential vulnerability when attackers are stronger, potentially facilitating the community in building more robust watermarks.

2. It experiments with diverse settings.

weaknesses:
1. **Unclear optimization formulation.** Why do you formulate robustness for attack optimization? The robustness of what (the surrogate model or the target model)? Are the paraphraser and the surrogate model the same model (given line 136: *"Our attacker can access such open-weight surrogate models and use them for paraphrasing text."*)?

2. **The algorithm is confusing.** To obtain the paraphraser without directly optimizing equation 2, the authors propose using DPO to overcome this challenge, which requires a preference dataset. However, in Algorithm 1, generating this dataset requires the paraphraser. How do you obtain the paraphraser?

3. **The method lacks important technical details.** Isn't the attackers' objective to fine-tune a paraphraser to remove the watermark in texts generated by the target language model? You proposed using DPO for optimizing the paraphraser (i.e., equation 2) as stated in line 178-179, but the method only covers how the dataset is created, not how the paraphraser is optimized using DPO.

4. **Lack of important analysis of experiment results.** In Figure 3 (Left), why does the adaptive setting have better performance than the non-adaptive setting? This is counter-intuitive, as the adaptive setting involves training and testing the paraphraser on the same watermark scheme, which should yield the best performance. Also, How do you explain why the proposed method shows such good transferability to unseen watermarks?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521001547

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
Dear Reviewers,

We really appreciate your feedback, timely responses and the opportunity to improve our paper. To summarize, our paper makes three core contributions: 

1. We formalize robustness as an objective function that allows optimizing attacks adaptively against a _specific_ watermarking method. 
2. Our attacks are state-of-the-art and outperform all surveyed attacks that study the same setting. We can evade all surveyed watermarks in both the adaptive and non-adaptive settings, indicating a flaw in how current watermarking methods are designed. Since our attacks are Pareto optimal and _specific_ to the watermarking method, we believe they should serve as a benchmark to test robustness in the future. 
3. We show that our attacks are practical to be used even by resource-constrained adversaries since optimization requires less than 10$ USD compute budget and they require _no_ access to the watermarked generator. We believe that robustness could be improved by including our attacks in the design process of future watermarking methods. 

The revisions are marked in blue in the revised paper, and we have listed a summary of revisions below. 

* In the discussion section, we added an intuitive explanation of _why_ our attacks are effective at evading detection, which was requested across the board from all reviewers.  
* We clarified the paper in multiple locations, as requested by the reviewers. For instance, we clarified that we focus on LLM-Judge as a metric in the main paper or that we use preference-based optimization methods rather than proposing them. 
* We have revised our limitations section to reflect the lack of a gold-standard metric to evaluate text quality. 
* We have revised our related work's section to include the works by Jovanovic et al. [G] and Pang et al. [H] and how they differ from the setting that we study in our paper. 
* Appendix A.3 contains our results against Unigram [A] and SIR [B]. We have included GPT4-mini as a judge and find that these results are consistent with those presented in the main paper. 
* Appendix A.4 contains an analysis of the top 50 token distribution after paraphrasing. 
* Due to space constraints, we moved the ablation study on the false positive rates to Appendix A.7. 

We believe these revisions substantially improved the paper and once again thank the reviewers for their valuable suggestions. 

----------
[A] Zhao, Xuandong, et al. "Provable robust watermarking for ai-generated text.", ICLR 2024.

[B] Liu, Aiwei et al. "A Semantic Invariant Robust Watermark for Large Language Models.", ICLR 2024

[G] Jovanović, Nikola, Robin Staab, and Martin Vechev. "Watermark stealing in large language models." arXiv preprint arXiv:2402.19361 (2024).

[H] Pang, Qi, et al. "No free lunch in llm watermarking: Trade-offs in watermarking design choices." The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732633101452

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New Paper Revision: Textual Analysis

comment:
Dear Reviewers,

The rebuttal period has been engaging, and we want to thank you sincerely for your invested time. Reviewers “zaXY” and “dv9w” agree that our work is interesting, but it requires a more detailed analysis of **why** our approach works. We have revised the paper to include this analysis in Appendix A.5 and kindly ask the reviewers to take a look at that section.

We plotted the (i) N-gram overlap ratios and (ii) Token Edit Distance ratios (please refer to A.5 for their definitions) for adaptively tuned paraphrasers against un-optimized baseline paraphrasers and GPT-model paraphrasers. These metrics are important for watermark evasion because all surveyed watermarks operate on the token-level. Our results show a large difference in these metrics between the outputs generated by the paraphrasers before and after adaptively optimizing them. Please refer to the revised paper for more details.

We hope the revisions address your concerns and provide sufficient clarity on our contributions. We greatly appreciate your time and efforts and hope you are now satisfied with the revised submission.

Best,
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission "Optimizing Adaptive Attacks against Content Watermarks for Language Models" considers the problem of attacking language model watermarks. The submission proposes a way to use preference optimization to adapt language-model-based paraphrasers for improved watermark attacks against known watermark algorithms.

Reviews show no strong support for the submission. While reviewers do find the approach interesting, there are a number of questions regarding the details of the mechanics, the evaluation on other schemes and a better understanding of how the paraphraser improves. These diffuse questions are best understood against the strong zero-shot performance that many paraphrasers already have. There is no clear scenario where existing paraphrasers do not break a watermarked model, but the DPO-optimized paraphraser would break it. 

The positioning of this work is also a bit improvable. The current attack is adaptive w.r.t to the generic watermarking scheme used, but not adaptive to the details of the watermarking scheme, examples from the watermarked model at hand, or its key, like stronger attackers, occupying a middle ground that could be staked out better.

Overall, I do not recommend acceptance for now.