Watermark Smoothing Attacks against Language Models

Authors: Hongyan Chang, Hamed Hassani, Reza Shokri
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Watermark

Abstract:
Statistical watermarking is a technique used to embed a hidden signal in the probability distribution of text generated by large language models (LLMs), enabling the attribution of the text to the originating model. We introduce the smoothing attack and show that existing statistical watermarking methods are not robust against minor modifications of text. In particular, with the help of a weaker language model, an adversary can smooth out the distribution perturbation caused by watermarks. The resulting generated text achieves comparable quality to the original (unwatermarked) model while bypassing the watermark detector. Our attack reveals a fundamental limitation of a wide range of watermarking techniques.

Decision: reject
Reviewer scores: 5, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730166652029

thread_type:
official_review

<thread_object>
creator: reviewer_nfpb

<content>
summary:
The paper introduces a "smoothing attack" that bypasses statistical watermarking in large language models (LLMs). By blending outputs from the watermarked model with a weaker reference model, it removes watermarks without impacting text quality on PPL.

strengths:
1. The writing is easy to follow.

2. Propose a smoothing attack scheme against statistical watermarking, and show that the significance level $S_t$ is highly correlated with the total variation distance.

weaknesses:
1. The applicability of this method is limited, as obtaining a high-quality reference model is often not possible (e.g. for GPT-4). Additionally, it requires access to token logits, meaning it is not a purely black-box approach as claimed.

2. In line 146. The authors overclaim that their attack is universally applicable to all statistical watermarking schemes. However, many watermarking schemes [1,2,3] do not use a green list, and their proposed method cannot be applied.

3. Additional metrics are needed to better reflect the quality of the generated text. PPL tends to favor a distribution similar to that of the oracle model, which can introduce bias. It would be more informative to include straightforward metrics, such as BLEU in machine translation, to provide a clearer evaluation.

4. The paper lacks key baseline results needed to demonstrate the effectiveness of the proposed method.  Naive smoothing using $\lambda \tilde{P}(x)+(1-\lambda) P^{ref}(x)$ can also remove the watermark while preserving part of the text quality.

5. The choice of z-score threshold used in the experiments is unclear. It would be more straightforward to present the true positive rates at specific theoretical false positive rates, providing a clearer understanding of the method’s performance.

6. The experimental settings for certain tests are suboptimal. For instance, in Table 2, the z-score for XSIR and SIR is too low, indicating that the watermark strength in the original watermarked model is insufficient.

[1] Kuditipudi, R., Thickstun, J., Hashimoto, T. and Liang, P., 2023. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15593.

[2] Hu, Z., Chen, L., Wu, X., Wu, Y., Zhang, H. and Huang, H., 2023. Unbiased watermark for large language models. arXiv preprint arXiv:2310.10669.

[3] Dathathri, S., See, A., Ghaisas, S., Huang, P.S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T. and Hayes, J., 2024. Scalable watermarking for identifying large language model outputs. Nature, 634(8035), pp.818-823.

questions:
1. Why in Figure 1, top-p sampling (right figure) has some points with the total variation distance being 0 or 1, but top-k sampling (middle figure) does not?
2. How many queries (prefixes) do you use for computing the bin index as described in Lines[261-266]?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730252816514

thread_type:
official_review

<thread_object>
creator: reviewer_rzfd

<content>
summary:
This work develops a smooth attack in the “green-red list” watermarking framework. The paper shows that a smooth attack makes it easier to bypass the detector while still preserving the quality of the text.

strengths:
Many existing methods for statistical watermarking have primarily concentrated on the generation and detection of watermarks. This paper takes a different approach by examining statistical watermarking from a new perspective. This perspective is interesting and may also aid in the development of improved watermark generation and detection techniques.

weaknesses:
1. The significance level $S_t$ is unobserved and was estimated using a surrogate quantity, $c_t$. Though the authors showed that there is generally a negative correlation between $c_t$ and $S_t$, this is only a weak justification. It is possible that a small $c_t$ would correspond to a large $S_t$ in some situations, e.g., when $K$ is small. 
2. The method only applies to the “green-red list” watermarking scheme, which is known to be biased because it does not preserve the original text distribution. In contrast, there are unbiased watermarking methods (e.g., Kuditipudi et al., 2023; Aaronson, 2023). It is unclear if the proposed method applies to unbiased watermarking schemes. Perhaps the authors can provide more discussions about how their method might be adapted or extended to work with unbiased watermarking schemes.
3. The paper lacks a rigorous theoretical analysis of the effect of the smooth attack on the text quality, e.g., bounds on how much the smoothing attack can affect certain text quality metrics.

questions:
1. In Table 1, Watermark (smoothing) has a lower perplexity than Watermark (or even Unwatermark) in some cases (e.g., Llama2-7b). In other words, the attack can even improve the quality of the text, which seems counterintuitive as the reference model is weaker. This also raises a concern about whether perplexity is the right measure to look at the quality of a text here. The authors may want to include other text quality metrics in the numerical studies.
2. I would like to know if the authors can discuss the potential pitfalls of their methods, e.g., provide concrete examples or scenarios where their smooth attack might fail, and discuss the implications of such failures
</content>
</thread_object>

</thread>
<thread>
creation date:
1730487135041

thread_type:
official_review

<thread_object>
creator: reviewer_a4fl

<content>
summary:
In this paper, the authors introduce a novel watermark-removal attack that requires only a small watermark-free reference model. The attacker first estimates the probability of the generated token at position i being in the watermark's green list, which correlates with the relative confidence of the most likely token among the top k tokens. According to the confidence score, the attacker then combines the probability distributions at position i from both the watermarked model and the reference model to sample the token. This approach effectively evades watermark detection while maintaining high text quality.

strengths:
- I find the proposed method very interesting and quite different from the previous work. Meanwhile, the method doesn't require a strong oracle model like a paraphrasing attack, which makes the threat model more realistic.
- I really enjoy reading this paper, especially section 3.1, which gives readers a lot of insights.
- The results look positive and a lot of different watermarking schemes are covered (most results are presented in the appendix).

weaknesses:
- The proposed method relies on using the logits/output probabilities of the watermarked model. This might limit the attack to some API models that may not return the logits/probabilities or only return top-k probabilities or even calibrated probabilities.
- The paper uses perplexity or loss to measure the text quality, but I think it's not enough to show the quality of the text. For example, the model can generate an answer for a math question with a very low perplexity, but the answer is completely wrong. So, I think it will be more helpful if the authors can include more text quality metrics like P-SP used in [1] or even a model-based evaluation like asking a large oracle model which generation is preferable.
- I think it's also helpful to the paper if the answers can show the results under different data distributions instead of overall c4.

[1] Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., & Goldstein, T. (2023). On the Reliability of Watermarks for Large Language Models. ArXiv, abs/2306.04634.

questions:
- Can the authors provide a baseline that uses the local reference model to do the paraphrase attack?
- What could be potential adaptive defenses for this attack?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700877405

thread_type:
official_review

<thread_object>
creator: reviewer_f25p

<content>
summary:
The paper proposes an automatic method for editing watermarked text from a language model to evade watermark detection using another (weaker) language model. The paper mainly considers the "red-green list" watermark of Kirchenbauer et al. and variants thereof, though the techniques should presumably generalize to other watermarks.

strengths:
The paper proposes a heuristic to estimate which tokens contribute the most to the overall watermark signal and removes the watermark by editing these tokens using another language model. The idea is interesting, and the paper empirically validates the effectiveness of their attack across different watermarks, language models, and datasets. These results clearly establish the effectiveness of the attack in practice.

weaknesses:
The paper distinguishes its main contributions from prior work by arguing that prior work on automatically removing watermarks involved using language models that were at least as strong as the original watermarked language model. However, one notable exception is the work of Zhang et al. [1], who seem to also focus on removing watermarks using weaker language models. This work is cited in the present paper but not discussed in any detail. It would be great if the authors can update their paper with a discussion of how their work differs from [1]. Otherwise, the novelty/significance of the main contributions over prior work is not clear.


[1] Zhang et al. (2023) Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models. https://arxiv.org/abs/2311.04378

questions:
What are the main differences between this work and that of Zhang et al. (2023)? (see Weaknesses section)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732249378615

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarification of threat model

comment:
Reviewers (Reviewer NfPb, Reviewer a4FL) raised concerns about the access of the adversary to the target model.
In our attack, the adversary only has access to the top-K probabilities and the corresponding tokens (with K=20) from the target model at each token position instead of logits/probabilities over all the tokens.

We would like to note that this is a reasonable assumption for the adversary, who could be a user of the target LLM, as common LLM service providers often grant their users access to such information through APIs, e.g., OpenAI’s API, for better customization, interpretability, traceability, and ultimately, better user experience.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732250334190

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
On the metric of text quality

comment:
Several reviewers (Reviewer NfPb, Reviewer a4FL, and RzfD) have questioned the use of perplexity as a quality metric.

To explain, perplexity (PPL) is widely used in the text watermarking literature (e.g., [1,2,3,4,5,6]) and serves as a standard metric for evaluating text quality. We followed the literature and included perplexity as a metric for quality evaluation. We are aware of alternative metrics for quality measurement, but finding the universally best quality metric is still an open problem in the NLP literature [7].  In what follows, we discuss several alternative metrics suggested by the reviewers and their inapplicability to our work.

- **P-SP** (suggested by Reviewer a4FL): P-SP metric measures cosine similarity between the embeddings of **two texts** (one of them is regarded as the reference text). It was originally proposed under a different context than watermark detection, e.g., evaluating the quality of a translated/paraphrased sentence. If the given sentence looks/means more similar to the ground truth, then P-SP gives a higher score. In our context, there is no reference text to compare with. Instead, an LLM could be used for answering a question to which there is no single deterministic answer, e.g., explaining why a certain company (e.g., Intel) is good. In such scenarios, answers that look very different may be of high quality at the same time, e.g., both "Intel pays well" and "Intel's CPU plays an important role in commercial computers" are valid answers. 
    
- **BLEU** (suggested by Reviewer NfPb): Similar to P-SP, this metric assumes the existence of a reference text and evaluates its word-level similarity (counting overlaps over n-grams) with a given text. Hence, this metric does not seem to be a proper metric in our setting.

We also run additional experiments using **GPT-4 as the quality metric** to evaluate the generated texts on a scale from 1 to 10 (higher means better). The prompt template is similar to [8] (see Appendix A.4 of the revised version). We present the new results below (also included in Table 1 of the revised version).

| Setting                  | OPT-1.3b (KGW) | OPT-1.3b (Unigram) | Llama2-7b (KGW) | Llama2-7b (Unigram) |
|--------------------------|--------------|------------------|---------------|-------------------|
| Human-written           | 8.83         | 8.83             | 8.83          | 8.83             |
| Reference               | 7.16         | 7.16             | 3.3           | 3.33             |
| Unwatermarked           | 8.66         | 8.66             | 8.66          | 8.66             |
| Watermarked             | 8.33         | 7.66             | 8.28          | 8.33             |
| Watermarked (P-GPT3.5)  | 8.66         | 8.33             | 8.83          | 9.0              |
| Watermarked (Word-D)    | 2.0          | 2.0              | 2.48          | 2.17             |
| Watermarked (Word-S)    | 2.66         | 2.66             | 3.81          | 4.0              |
| Watermarked (Smoothing) | 7.33         | 7.33             | 5.25          | 4.5              |

Our attack achieves comparably good text quality, which is consistently better than the reference model and the Word-D and Word-S attacks that do not use reference models. Notably, on Llama2-7b, the text quality of our attack is much better than the above baselines, and is *only slightly worse* than the original watermarked/unwatermarked models and the unrealistic paraphrasing attack P-GPT-3.5. We note that the evaluation by GPT-4 also has limitations [9, 10, 11,12], e.g., it may be biased toward its own responses [8]. Addressing these limitations is an avenue for future work. 

We would like to thank the reviewer for the instructive comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Paper Summary:**

The paper proposes an attack on distortionary text watermarks, using text from an unwatermarked weak LM to rewrite text from a strong watermarked LM in a way that erases the watermark.

**Strengths:**

The attack is interesting and it is simple: a token-level intervention, as opposed to more complicated paraphrasing attacks studied in prior work, e.g., Zhang et al. (ICML,  2024).

**Weaknesses:**

Reviewers a4FL and NfPb note that the attack relies on access to top-k probabilities of the watermarked language model. These are often, but not always available.

Reviewers f25P, NfPb, and RzfD observe that this attacked is not applicable to distortion-free watermarks.