Discovering Clues of Spoofed LM Watermarks

Authors: Thibaud Gloaguen, Nikola Jovanović, Robin Staab, Martin Vechev
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: llm, watermarks, spoofing

Abstract:
LLM watermarks stand out as a promising way to attribute ownership of LLM-generated text. One threat to watermark credibility comes from spoofing attacks, where an unauthorized third party forges the watermark, enabling it to falsely attribute arbitrary texts to a particular LLM. While recent works have demonstrated that state-of-the-art schemes are in fact vulnerable to spoofing, they lack deeper qualitative analysis of the texts produced by spoofing methods. In this work, we for the first time reveal that there are observable differences between genuine and spoofed watermark texts. Namely, we show that regardless of their underlying approach, all current learning-based spoofing methods consistently leave observable artifacts in spoofed texts, indicative of watermark forgery. We build upon these findings to propose rigorous statistical tests that reliably reveal the presence of such artifacts, effectively discovering that a watermark was spoofed. Our experimental evaluation shows high test power across all current learning-based spoofing methods, providing insights into their fundamental limitations, and suggesting a way to mitigate this threat.

Decision: reject
Reviewer scores: 5, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730189975182

thread_type:
official_review

<thread_object>
creator: reviewer_n7sn

<content>
summary:
This paper proposes a hypothesis-test-based method to detect whether a watermarked text generated by an LLM is spoofed.

strengths:
1. The proposed method can detect spoofed texts when the length of the text is large enough.
2. Detailed theoretical analysis.

weaknesses:
1. The insight of this paper depends on that 'if the prior contexts are not in the adversary's extracted dataset, the adversary will predict the next token randomly'. If so, utilizing a long context as the seed to determine the green-red split may easily defend the spoofing attack. The original watermark verification method (used in the watermarking scheme) may also be able to distinguish the true watermarked texts from the spoofed watermark texts. Further clarifying the challenges of detecting spoofed texts may help the readers better understand the technical contribution of this paper.
2. Another concern is that this paper does not include any baseline method in the experiments. I think comparing with some baseline methods (e.g., the original watermark verification method) can help demonstrate the effectiveness of the proposed method.
3. Section 5.1 presents the histograms of the distributions. However, a quantitative analysis (e.g., a hypothesis test to verify whether the distribution follows a normal distribution) can better help the readers understand the results.
4. This paper may not be easy to read for readers who are not familiar with the statistics. I strongly suggest that the authors explain some uncommon symbols and operators. For example, the symbol in Eqs. (3a) and (3b) and the $\xrightarrow{d}$ in Lemma 4.1.

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730406549120

thread_type:
official_review

<thread_object>
creator: reviewer_kecn

<content>
summary:
This paper presents the first in-depth study of statistical artifacts in LLM watermark spoofing attacks. The authors demonstrate that current watermark spoofing methods leave detectable statistical patterns and propose a statistical testing framework to distinguish genuine watermarked text from spoofed ones.

strengths:
1. First systematic investigation of quality characteristics in watermark spoofing, beyond mere success rates
2. Strong theoretical foundation with rigorous statistical testing framework
3.  Practical approach for detecting watermark spoofing, enhancing watermark reliability

weaknesses:
1. The study's scope is primarily confined to the KGW watermark algorithm, leaving significant gaps in the evaluation of other important watermarking approaches. The effectiveness of the proposed method remains untested on unigram watermarking schemes  (zhao et al. 2023) that employ global green/red lists, as well as the KTH  (Kuditipudi et al. 2023)  method utilizing fixed watermark key lists. This limited coverage raises questions about the method's generalizability to alternative watermarking techniques widely used in practice.
2.  The proposed approach imposes dual requirements of maintaining both significant green/red word ratios and cross-context consistency, which may compromise its robustness. While these requirements enhance the detection accuracy of spoofed watermark text, they could make the system overly sensitive to legitimate text modifications. Because text modification will also cause not consistent context watermark. The paper does not adequately address the critical trade-off between detection accuracy and robustness against common text editing operations. Further investigation is needed to determine whether the method can maintain its effectiveness while accommodating natural text modifications.
3. The method's requirement for approximately 1000 tokens to achieve reliable detection presents significant practical limitations. This substantial text length requirement restricts the method's applicability in real-world scenarios, particularly for short-form content analysis.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470040383

thread_type:
official_review

<thread_object>
creator: reviewer_yzgn

<content>
summary:
This work proposes statistical tests that can differentiate spoofed texts from genuine watermarked texts, based on the intuition that spoofed text contains artifacts due to spoofers' limited knowledge.

strengths:
1. This work investigates an interesting problem of whether the spoofed texts can be differentiated even if they trick the detector.

2. It provides a theoretical demonstration regarding the proposed tests.

3. The experiments are extensive, testing different language models and diverse settings.

weaknesses:
1. The conclusion is overclaimed. This paper only examined learning-based spoofing attacks but claims that all state-of-the-art spoofing attacks leave artifacts in spoofed texts (in the abstract and introduction). What about spoofing attacks that exploit watermark robustness, e.g., Peng et al. [1] and Zhou et al. [2] in your references?

2. The proposed tests are not practically useful, considering the performance is only good in some cases. In particular, from Table 1, the tests yield high TPR when $ T > 1000 $ and $ h = 3 $ or $ h = 1 $. However, spoofed texts from two learning-based attacks could achieve spoofing attacks when $ T < 1000 $. If their generated text has $ T = 500 $, the corresponding test performance at best is 0.62 (TPR@1%). Does this suggest that the model owner should deploy watermarking with $ h = 3 $?

3. There is no explanation of the important variables or unclear definitions, impeding understanding (see question 1).


4. Limited discussions. Intuition suggests that the reason for artifacts is that the spoofer only has partial knowledge due to their limited training data. How will the size of $ D $ affect the artifacts and the statistical test results? Would stronger attackers leave fewer artifacts by increase the size of $D$ or improving the sampling strategy by introducing some independence?






----
[1] Qi Pang, Shengyuan Hu, Wenting Zheng, and Virginia Smith. Attacking LLM watermarks by exploiting their strengths. arXiv, 2024.

[2] Tong Zhou, Xuandong Zhao, Xiaolin Xu, and Shaolei Ren. Bileve: Securing text provenance in large language models against spoofing with bi-level signature. arXiv, 2024.

questions:
### Questions in Weaknesses and Below:


1. What is $X_t$ (is it the same as defined in line 168?), and what is the meaning of $X_t = 1 $ in equations 2a and 2b? Also, why is $I_D $ defined as a function of frequencies in line 194 but as a probability distribution in line 204? Which is the correct definition?

2. How accurate is it to use $\tilde{D} $ to approximate the training set $D $ (i.e., using natural language to approximate watermarked language)? To what extent could the discrepancy affect the final accuracy of distinguishing spoofed text?

3. How many samples are in the spoofer training data $D $? There are two different distillation methods proposed in Gu et al. (2024); which one did you test?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730617246291

thread_type:
official_review

<thread_object>
creator: reviewer_xcg2

<content>
summary:
This paper investigates the vulnerabilities in watermarking LLMs due to spoofing attacks. While watermarks aim to identify AI-generated text, attackers can replicate them, falsely attributing unauthorized content to specific models. By analyzing "artifacts" left by spoofing techniques, the authors develop statistical tests to differentiate between authentic and spoofed watermarked text. Experiments show these tests are effective across various spoofing methods.

strengths:
1. The paper provides a detailed analysis of the specific, observable artifacts left in spoofed texts, an underexplored area in watermark spoofing research.

2. The authors conduct extensive experiments, testing their hypotheses across multiple models, parameter settings, and spoofing techniques, lending empirical robustness to their conclusions.

weaknesses:
1. The paper focuses primarily on a specific watermarking scheme -- the KGW watermark. While this is an important and well-studied case, there are several newer watermarking designs that make spoofing attacks virtually impossible. These alternative methods could be used without the concerns associated with spoofing attacks. As such, the scope of the paper feels limited, as it does not consider the broader landscape of watermarking schemes. Furthermore, the paper mainly studies specific spoofing techniques, such as Stealing and Distillation, which may not fully reflect the evolving and diverse strategies available to attackers.

2.  Some assumptions in the analysis are restrictive, particularly regarding the KGW watermark’s context size, often set at h=1. With this small context, green tokens are more likely to appear in the spoofer's training data, simplifying the spoofing process. For example, the assertion that "If the context is not in D, the spoofer is forced to select the next token independently of its color" mainly applies when h is larger, limiting its relevance to cases with small context sizes.

3. The paper would benefit from a deeper exploration of how the context length h influences the statistical tests’ performance. Additionally, discussing broader applications and limitations for various watermark types could enhance the study’s relevance. For instance, cryptographic watermark designs, such as those based on pseudorandom error-correcting codes, may not face spoofing issues, and a comparison with these designs could provide more context on the strengths and weaknesses of KGW-style watermarks.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687408628

thread_type:
official_review

<thread_object>
creator: reviewer_ozah

<content>
summary:
This paper explores forgery attacks on LLM watermarks, utilizing traces in forged texts that reflect the forger's partial knowledge, revealing observable differences between genuine watermarked texts and forged watermarked texts. The authors propose a statistical testing method, experimentally verifying that these traces can be detected to identify whether a watermark has been forged.

strengths:
This paper investigates the forgery traces of attacks on LLM watermarks. 
It provides definitions through formal descriptions and formula derivations, and designs a statistical testing method. The effectiveness of the testing hypothesis is validated through experiments. 
The presentation of figures and tables is relatively clear.

weaknesses:
1.There are various methods for adding watermarks to LLMs, and there may also be multiple ways to forge them. This paper only discusses the red-green watermarking scheme for LLMs and the forgery method of constructing datasets by querying a watermarked model, which is highly restrictive.
2.The rationale for the assumptions in this paper is insufficient. The paper assumes that forgers, when constructing datasets by querying a watermarked model, will leave traces of forgery due to incomplete knowledge in the dataset. However, if the attacker queries the watermarked model again to supplement the missing knowledge in the forged dataset, they can bypass the limitations of this assumption. In this case, the premise does not hold, making the motivation of the paper weak.

questions:
1.The paper should incorporate a comparison of various watermarking methods for LLMs, such as those proposed by Miranda Christ, Sam Gunn, and Or Zamir in "Undetectable watermarks for language models" (COLT, 2024), Rohith Kuditipudi et al. in "Robust distortion-free watermarks for language models" (arXiv, 2023), and Zhengmian Hu et al. in "Unbiased watermark for large language models" (ICLR, 2024). Additionally, it should experimentally compare the effectiveness of the proposed method against the latest forgery techniques, such as those described by Qi Pang et al. in "Attacking LLM watermarks by exploiting their strengths" (arXiv, 2024) and Qilong Wu and Varun Chandrasekaran in "Bypassing LLM watermarks with color-aware substitutions" (arXiv, 2024).
2.Regarding the second Weakness, if an attacker supplements the missing content in the forged dataset by querying the watermarked model again to complete the insufficient knowledge, can this bypass the limitations of the premise assumption and affect the effectiveness of the method proposed in this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732295805648

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their feedback. We are pleased to see that the reviewers consider our contributions novel and innovative (_xcg2_, _yZgn_, _kEcn_), acknowledge that they are backed by a rigorous mathematical framework (_ozah_, _yZgn_, _KEcn_, _n7sN_) and appreciate our extensive experimental evaluation (_ozah_, _xcg2_, _yZgn_).

We have uploaded an updated version of the manuscript where the new content is marked in blue, and responded to all questions and concerns raised by the reviewers in individual responses below. We are happy to further engage in the discussion in case the reviewers have additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553096455

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion period closing soon

comment:
We thank reviewers _yZgn_ and _n7sN_ for the discussion; as the discussion period closes soon, we would like to kindly remind reviewers _ozah_, _xcg2_ and _KEcn_ to acknowledge our rebuttal. We are available to discuss if there are any remaining concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposes a forensic investigation of spoofing attacks against text watermarks for language models, which are attacks that spoof the watermark signature, injecting it into text not generated by the watermarked model. The submission shows that interestingly, for a relevant watermarking scheme spoofing attack, these attacks can in turn be detected.

This idea is interesting and the investigation by the authors has merit, but, as the topic of this discussion becomes a bit niche, it would have been good to see a broader discussion and formalization of the security problem of counter-spoofing across watermarking methods (i.e. to my understanding, current theory, such as in Section 4.1 is applicable only to the KGW watermark). The current submission reports interesting findings that could be extended into such a, more general, investigation, elevating it from its current current status that is a bit too close to an instance report to several reviewers.

Due to this concern, and based on only muted support by reviewers, I do not recommend acceptance of this submission for now. I hope the authors are going to re-submit their extended analysis.