Defending LLM Watermarking Against Spoofing Attacks with Contrastive Representation Learning
Authors: Li An
Yujian Liu
Yepeng Liu
Yang Zhang
Yuheng Bu
Shiyu Chang
Venue: COLM 2025 (2025)
Keywords: LLM watermarking
Spoofing attack
Piggyback Spoofing attack
TL;DR: The paper proposes a semantic-aware watermarking method that defends against spoofing attacks by using a contrastively trained semantic mapping model to detect semantic distortions.

Abstract: Watermarking has emerged as a promising technique for detecting texts generated by LLMs. Current research has primarily focused on three design criteria -- high quality of the watermarked text, high detectability, and robustness against removal attack. However, the security against spoofing attacks remains relatively understudied. For example, a piggyback attack can maliciously alter the meaning of watermarked text by transforming it into hate speech, while preserving the original watermark, thereby damaging the reputation of the LLM provider. We identify two core challenges that make defending against spoofing difficult: (1) the need for watermarks to be both sensitive to semantic-distorting changes and insensitive to semantic-preserving edits, and (2) the contradiction between the need to detect global semantic shifts and the local, auto-regressive nature of most watermarking schemes. To address these challenges, we propose a semantic-aware watermarking algorithm that post-hoc embeds watermarks into a given target text while preserving its original meaning. Our method introduces a semantic mapping model, which guides the generation of a green-red token list, contrastively trained to be sensitive to semantic-distorting changes and insensitive to semantic-preserving changes. Experiments on two standard benchmarks demonstrate strong robustness against removal attacks and security against spoofing attacks, including sentiment reversal and toxic content insertion, while maintaining high watermark detectability. Our approach offers a significant step toward more secure and semantically aware watermarking for LLMs.

Published as a conference paper at COLM 2025
Defending LLM Watermarking Against Spoofing Attacks with Contrastive Representation Learning
Li An Yujian Liu'
Yepeng Liu'
Yang Zhang2* Yuheng Bu
Shiyu Chang
UC Santa Barbara
M Watson Al Lab
Abstract
Watermarking has emerged promising technique for detecting texts generated by LLMs. Current research has primarily focused on three de- sign criteria high quality of the watermarked text, high detectability, and robustness against removal attack. However, the security against spoofing attacks remains relatively understudied. For example, a piggyback attack can maliciously alter the meaning of watermarked text by transforming it into hate speech, while preserving the original watermark, thereby damag- ing the reputation of the LLM provider. We identify two core challenges that make defending against spoofing difficult: (1) the need for water- marks to be both sensitive to semantic-distorting changes and insensitive semantic-ppe prratticppeeevinngg edits, and (2) the contradiction between the need to detect global semantic shifts and the local, rugo-regressive nature of most watermarking schemes. To address these challenges, we propose a semantic-aware watermarking algorithm that post-hoc embeds watermarks into a given target text while preserving its original meaning. Our method introduces semantic mapping model, which guides the generation of a green-red token list, contrastively trained to be sensitive to semantic- distorting changes and insensitive to semantic-preserving changes. Experi- ments on two standard benchmarks demonstrate strong robustness against removal attacks and security against spoofing attacks, including sentiment reversal and toxic content insertion, while maintaining high watermark detectability. Our approach offers a significant step toward more secure and semantically aware watermarking for LLMs. Our code is available at Cow/UUCSB-NLP-CCaagCccoo
https: thub. cww/ NLP Chang/ contrastive termarka..
1 Introduction
LLM watermarking is a technique proposed to
Detectability
Adaptive
combat LLM misuse, including the spread of misinformation, copyright violations, and the
KGW
creation of harmful content. Watermarking algo- rithms typically embed some subtle, algorithmi- cally detectable patterns, called watermarks, in
Robustness
Text Quality
the L.M-ggeerrttd text, so that one can discern whether given passage is generated by an LLM by detecting whether any watermark is present (Kirchenbauer al., 2023; Aaronson, 2023; Zhao et al., 2023; Kuditipudi et al., 2023). Existing watermarking algorithms have been focusing on three performance criteria: ① Text quality watermarking should not disturb the quality of the generated text, @ Defectability termarked text should be identified with high success rate and unwatermarked text should not
50
75
100
Security (sentiment)
Security (hate)
Figure 1: Performance several key di- mensions (higher better) for existing approaches and our method, details are provided in Section 4.1.
any wa-
'Equal advising and contribution.

---

Published as a conference paper at COLM 2025
Watermarked text Score: 0.95 f
K.G. problem-solver, always ready to spring into action whenever she identifies f opportunity for improvement... she is recognized for her outstanding character she was one of distinguished 34 Canadians awarded $100,000 over a period four years What distinguishes recipient of the highly-dgarded award is their demonstrated strength of character, which encompasses traits like integrity, courage, persistence and self-morality as opposed solely academic performance.
Sentiment spoofing attack Score: 0.85
K.G. is troublemaker, always ready to spring into action whenever she identifies an opportunity for creating thaos she is notorious for her questionalle character... she was one of flawed 34 Canadians awarded $100,000 over period of four years... What distinguishes recipient of the highly-regardes award is their demonstrated weakness of character, which encompasses traits like integrity, courage, persistence and self-morality, as opposed to solely academic performance.
Sentiment spoofing attack on latter-half Score: 0.92 4
K.G. is problem-solver, always ready to spring into action whenever she identifies opportunity for improvement.... she is recognized for her outstanding character... ... she was one of an overhyped 34 Canadians awarded $100,000 over a period of four years ... What cynically distinguishes a recipient of the overinflated award their so-called character traits, which supposedly encompass traits like integrity, courage, persistence and self-morality, as opposed to purely academic performance.
Table 1: An example of piggyback spoofing attack (Pang et al., 2024). The score reflects the portion of detected "green words" and detailed explanation is provided in Appendix A.3. Words modified by the attacker are marked in red.
be falsely identified, and @ Robustness against removal attack - watermark should remain detectable under semantically equivalent modifications such as paraphrases and word substitutions, so that they cannot be easily removed.
Existing research on LLM watermarking has largely overlooked a critical design consid- eration- the resilience against spoofing alfacks. One such spoofing attack is the piggyback spoofing attack (Pang et al., 2024), which aims to significantly alter L.M--eenrated text, po- tentially turning it into harmful or malicious content. If the watermark persists in the altered text, it could lead to false accusations that the target LLM produced harmful material. The second panel Table 1 illustrates piggyback spoofing attack, where an LLM-generated accolade, originally watermarked by the KGW-1 (Kirchenbauer et al., 2023), is transformed into a highly negative critique. Despite this drastic change, the watermark detection algo- rithm still registers a high confidence score of 0.85. This underscores the severity of the issue and the urgent need for more secure watermarking methods.
However, improving resilience against spoofing attacks is particularly difficult, as it involves two core challenges. First, defending against spoofing attacks requires watermarks to be sensitive to certain text alterations. This contradicts the robustness principle, which demands that watermarks remain insensitive to modifications. Therefore, achieving both objectives si- multaneously requires clear distinction between permissible and impermissible alterations, and enforcing opposite behaviors for each.
The second challenge in defending against spoofing attacks is that detecting malicious semantic distortions requires analyzing the entire text, which contradicts the nuto-regressive nature of watermarked text generation. More specifically, most existing watermarking methods operate by assigning a green-red token list at each token position, conditioned only on the preceding context. Accordingly, watermark detection is also based solely on the previous context. However, consider a simple example: the watermarked text K.G. is widely acknowledged as good', where an attacker changes the last word to 'bad'. This single-word alteration completely changes the meaning, yet the watermark identification remains unchanged for all but the last word, as their preceding context is intact.
Despite the seemingly prohibitive challenges, in this paper, we propose novel semantic- aware watermarking algorithm that adds watermarks to given target text. Our method introduces two key designs to defend against spoofing attacks, addressing the two chal- lenges discussed earlier. First, we introduce a semantic mapping model that generates semantic embeddings from text, which are then used to construct a green-red token list. This model is trained with a contrastive learning objective, ensuring that the embeddings remain insensitive to meaning- pwwpogg-rreeiiiggg alterations while being sensitive to
2

---

Published as a conference paper at COLM 2025
changes. This dual property enhances both robustness against watermark removal and resilience against spoofing attacks. Second, the semantic mapping model operates on the entire target text, allowing it to detect semantic distortions at any token position effectively.
Our experiments on two widely used benchmarks reveal that our method is robust to paraphrasing attacks while maintaining high security against spoofing attacks aimed at altering text sentiment or inserting toxic content (a summary of the performance is shown in Figure 1). In addition, our globally conditioned green-red token mapping further ensures robust defense against spoofing attacks across various text positions, offering a more reliable solution against evolving attack strategies.
2 Related Works
2.1 In-process LLM Watermark
The in-process LLM watermark embeds the invisible watermarks into the text throughout the generation process (Li et al., 2025; 2024; He et al., 2024; 2025; Zhao et al., 2024b;a; Huang et al., 2023; Zhang et al., 2025). Typically, the watermark information is embedded by adjusting the logits of LLM to steer the sampling towards specified tokens (Kirchenbauer et al., 2023; Zhao al., 2023; Lee et al., 2023) or employing specifically designed sampling strategies (Kuditipudi et al., 2023; Christ et al., 2024; Hu et al., 2023; Aaronson, 2023). Specifically, Kirchenbauer et al. (2023) randomly partitions the vocabulary into green and red lists using the hash of previous tokens and slightly increases the probability of green tokens in the next token distribution. Zhao et al. (2023) proposes to use globally fixed green-red list to improve the watermark robustness with theoretical guarantees. Aaronson (2023) proposes to employ the Gumbel-max trick (Gumbel, 1954) as a pseudo-random sampling strategy to generate the next token.
2.2 Post-hoc LLM Watermark
Instead of embedding watermarks during the generation process, the post-hoc watermark embeds watermarks into already generated texts. One form of the post-hoc watermark is the rule-based approach, which modifies the existing text according to predefined linguistic rules such as format (Sato et al., 2023), lexical choices (Yang et al., 2022; 2023; Hao et al., 2025; Yoo et al., 2023), syntax (Atallah et al., 2001). Specifically, Yang et al. (2022) embeds watermarks into text using context-aware lexical substitution. Sato et al. (2023) exploits Unicode character variants, such as different whitespace characters alternative representa- tions of the same symbol, to subtly embed a watermark without altering the visible content. Another approach embeds watermark into LLM-generated text by regenerating it with LLMs (Chang et al., 2024; Zhang et al., 2024; Qiang et al., 2023). Chang et al. (2024) selects set of input-dependent words and uses LLMs to insert them into the un-aatermarked text. Our method embeds watermarks by paraphrasing the given text with watermarked LLM, representing the second type of post-hoc watermarking approach.
2.3 Semantic-aware LLM Watermark
The robustness of a watermark enhances its resistance various types of modifications, such as paraphrasing (Krishna et al., 2023). However, strong robustness may lead to significant security threats to spoofing attacks. Specifically, Sadasivan et al. (2023) proposes forgery spoofing attack, which analyzes the token frequency of watermarked texts to uncover the hidden watermark patterns, enabling the adversary to forge the watermarked text. Meanwhile, Pang et al. (2024) introduces piggyback spoofing attack, which significantly alters the sentiment of watermarked texts or injects toxic content with minimal modifications while keeping the text detectable by the watermarking detector.
To balance robustness and security, the semantic-aware LLM watermarking methods are proposed (Liu & Bu, 2024; Hou et al., 2023; 2024; Ren et al., 2023; Huang et al., 2025; Liu et al., 2024; Fu et al., 2024; Cai et al., 2025). Hou et al. (2023) proposes a sentence-level semantic watermarking technique that ensures new sentences fall within specified semantic space
3

---

Published as a conference paper at COLM 2025
via rejection sampling based prior sentence semantics. Liu & Bu (2024) adaptively embeds watermarks low-entropy distributions and trains a semantic mapping model to transfer the semantic embedding of preceding text to green-red list, thereby enhancing the robustness, security, and quality trade-off. However, most of these methods rely on pre-trained embedding models, such as mentence-Transfrmeen (Reimers & Gurevych, 2019), which are relatively insensitive to sentiment shifts or the insertion of toxic content through minor modifications. As result, they struggle to defend against the piggyback spoofing attack effectively. To address the challenges posed by piggyback spoofing attacks, our work introduces mapping model that sensitive both semantics and sentiment in the text. This approach further enhances security while maintaining robustness.
Methodology
3.1 Problem Formulation
In this paper, we focus the post-hoc watermarking setting. Given target text x, which could be LLn-ggeerated or human-written, the task is to produce a watermarked version of x, denoted y,, such that 1 A detection algorithm can reliably detect y as watermarked and x as unwatermarked; and preserves the meaning and quality of x. Our framework can be easily extended the setting where the input is a unser-provied query, and the output is a watermarked LLM response. We will discuss this extension in section 3.5.
The challenge of the task is that watermarking algorithms need to simultaneously satisfy the following four criteria: O Text quality - the watermark should not degrade the fluency, coherence, or overall quality of the generated text. @ Detectability any watermarked text should be identified with high success rate, and unwatermarked text should not be falsely identified. Robustness against removal attacks - watermark should remain detectable under semantically equivalent modifications such paraphrases and word substitutions. Security against spoofing attacks watermark should be sensitive to malicious modifications and be removed from text intentionally forged by attackers. Among them, the security criterion understudied in prior works and presents prohibitive difficulties. We define the following two types of operations on watermarked text:
Permissible operations are modifications under which watermarks should remain de- tectable. Examples include semantic-preserving changes such as paraphrases and syn- onym replacements.
Impermissible operations are modifications under which watermarks should be re- moved. Examples include semantic-distorting changes like the reversal of sentiment and the insertion of toxic languages.
Security against spoofing attacks requires watermarking algorithms to be sensitive to impermissible operations, which seems to demand the opposite behavior of the robustness mriterion-maintain insensitive to permissible operations. Additionally, it requires analyzing the entire text, which contradicts the auto-regressive nature of watermarked text generation. In the following sections, we will introduce our algorithm to address these challenges.
3.2 A Semantic-Aware Watermarking Framework
We start by describing the overall framework of our method, which draws inspiration from the semantic-aware watermarking algorithm introduced in Liu & Bu (2024).
Generating watermarks. Given the input x, our method auto-regressively generates watermarked output y. At position r, the token yt is generated based on the following three steps, illustrated in Figure 2 (Left).
Step 1: Obtain LLM output logits. We feed the preceding context to the LLM and obtain its output logits 1(- p, ),t),, where y<t denotes the previously generated tokens, and p is an additional instruction to the LLM. In the post-hoc watermarking setting, we set as the paraphrase instruction, such as PPeass paraphrase the given text.'
4

---

Published as a conference paper at COLM 2025
Obiain LLM ouipul lngils.
Data transformation:
Original Logits
Permisspbe Op.
Falt*
Lange Language Model
p.x.Yer
Perturl logits
Impormichtl
Watermarked Lagis
JolOx
y
Construct greamwred tolum
Triplet
Sementic Mapping ())
Green-Red Split
1100 1 1
fax
Talx
(Pank)
Figure 2: Overview of our method. Left: The semantic-aware watermarking framework. An LLM is prompted to paraphrase given target text. During auto-regressive generation, the entire target text is fed to semantic mapping model to construct green-red token split. The LLM's output distribution is then perturbed by scaling up the probability of tokens assigned in the green list. Upper right: Data transformation and mapping process. Lower right: The triplet loss used to train the semantic mapping model.
Step 2: Construct green-red token list. We split the vocabulary V into list of green tokens
g C V and a list of red tokens R = V G. The split is performed by mapping function f X' + R/V which maps the input to a vocabulary-size vector based on the semantic meaning of x. The green list contains tokens with positive values, i.e., g = {v: fx() > 0}, where fo(-) denotes the v-th dimension of f()).
Step 3: Perturb LLM logits. We embed watermarks in yr by perturbing the LLM's logits based on the green-red token list. Specifically, for token v, its logit [[o] is adjusted to
[c] [[v] (1 + 51(v 9)), where 11.) is the indicator function, and ö is hyperparameter controlling the strength of watermarks. Additionally, to preserve text quality, we only perturb the logits when the LLM's output entropy is above a certain threshold (refer to Appendix A.2 for details). Finally, token y/ is sampled based on the adjusted logits
Detecting watermarks. To determine if a given text ý is watermarked the green-red token split using the same mapping function f and calculate the percentage of green tokens in y. Particularly, because we do not have access to the original text from which ý is obtained, we construct the token split using ý itself, i.e., & {v fo()) > 0}. Note that the detection greenlist & different from the generation green list G, but we expect them to be close because is constructed to have similar semantics to and f(-) is constructed to be insensitive to semantic-preed prentiiippeervving alterations, as will be discussed in Section 3.3. Since watermarked text will be biased toward green tokens, we label as watermarked the green token percentage is greater than a pre-defined threshold.
not, construct
Difference from prior work. The above framework is similar to that in Liu & Bu (2024) but with one important difference. Liu & Bu (2024) constructs the greed-red token split solely based on previously generated tokens, i.e., the input of function f is y<t- This makes defending against spoofing attacks difficult because secure defense requires capturing the -omantic-distorting changes at any location of the text. To mitigate this issue, we propose to construct the green-red token split using the semantics of the entire input text x, i.e., f is function of x, that as long the meaning of x changed, the token split is different and thus the watermarks will be removed.
3.3 Defending Against Spoofing Attacks via Contrastive Representation Learning
Building upon the above framework, a watermarking algorithm that satisfies both robust- ness and security criteria should have a mapping function that meets the following two requirements. 1 Any permissible operations on x should result in a similar green-red token split, so that the original watermarks remain detectable. In other words, for any x derived via permissible transformations of x, f(x+) should close to fxx). @ Any impermissible operations should lead to significantly different green-red token split, so that the
5

---

Published as a conference paper at COLM 2025
watermarks are removed. That is, for any x derived via impermissible transformations of f(x should be very different from fxx).
Based on this intuition, we propose to parameterize the mapping function as fo(-) and opti- mize parameters with contrastive learning. Specifically, given a dataset D = { (x,,xtx** xx)) that contains triplets consisting of an original text x, positive text x' that goes through permissible operations, and a negative text x that goes through impermissible operations, we minimize the following triplet loss:
(() = [xxxxx x >~D mmax (0)s) )) (in(f()) + + (1)
where sim is cosine similarity, and a > 0 is hyperparameter controlling the separation margin. Note that for the same original text x, DD could contain multiple positive and negative texts that undergo different permissible and impermissible operations, which will be discussed in the next section. Finally, we follow Liu & Bu (2024) to add two loss terms to balance the number of red and green tokens (please see details in Appendix A.2).
3.4 Dataset Construction
To construct the dataset D, we collect set of original texts as the anchor x and implement the following four operations on each x to create multiple versions of x and x Every unique combination of (x, xxxxx> will be considered as a new triplet in D. Details of the implementations, including the prompts, can be found in Appendix B.2.
Semantic-equivalent paraphrasing (permissible). Given text, we use an instruction-tuned LLM to generate a paraphrase. The paraphrase must maintain the original intent, length, and tone without introducing distortions or redundant phrasing.
Sentiment reversal (impermissible). We alter the sentiment of the original text while making minimal modifications to its content. The process consists of three steps. 1 Classify the sentiment of the original text as positive, negative, or neutral. The target sentiment is then decided with the following rule: if the original sentiment is not neutral, then flip it (e-g., positive to negative or vice versa); otherwise, randomly select positive or negative as the target sentiment. @ Modify the original text to express the target sentiment while preserving its meaning and structure as much as possible. @ Verify whether the modified text exhibits the intended sentiment shift. Only successfully altered texts are retained.
Latter-half sentiment reversal (impermissible), This operation follows the same process as sentiment reversal but restricts the LLM only to modify the latter half of the text.
Hate speech insertion (impermissible). We randomly insert hate phrases into the original text and use Llama Guard 3 (Llama Team, 2024) to verify whether the modified text contains unsafe content. Only samples where hate speech is detected are retained.
3.5 Watermarking without Provided Target Text
Although we describe our method for the post-hoc watermarking setting, it can be readily extended to the scenario where the user only provides query without any target text, and the goal is to generate a watermarked response to the query. Specifically, we first generate an LLM response without adding watermarks. Next, this response can be used as the target text and apply our method add watermarks.
Experiments
4.1 Experiment Settings
Datasets. We follow the convention to evaluate our method on the realnewsl ike subset of
C4 (Raffel et al., 2020) and the LFQA dataset (Krishna et al., 2023). We evaluate 200 target texts for both datasets. For C4, we use the original document as target text x. For LFQA, use the annotated gold completion as the target text. More details of the evaluation datasets can be found in Appendix B.1.

---

Published as a conference paper at COLM 2025
ROC-AUC (%)
Overall AUC
Method
Dataset
Sentiment Spoof
Hate Speech
PPL.
Detectability
Paraphrased
Spoof 4
Llama-3.1-88 3.1-88--nsBttuur
KGW
C4
100.00
72.68
98.85
100.00
43.46
8.27
LPQA
100.00
78.03
99.32
100.00
44.68
9.04
C4
99.54
81.96
98.44
99.54
45.88
8.23
UNIGRAM
LFQA
99.98
86.12
98.94
99.98
46.80
8.81
C4
99.78
72.18
96.50
99.35
44.03
8.77
ADAPTIVE
LFQA
99.97
70.45
97.14
99.91
43.34
9.90
99.99
89.03
94.07
99.87
48.77
9.21
POSTMARK
LFQA
99.93
87.20
95.54
99.47
48.03
9.21
C4
98.02
71.97
34.68
34.38
75.23
8.8
OURS
LFQA
99.16
80.99
29.23
29.89
80.26
9.57
Q Qwen2.5-7B-Instee
C4
99.12
67.92
94.04
99.08
43.48
8.97
KGW
LFQA
99.58
67.38
95.51
99.56
42.97
9.23
C4
97.34
66.99
93.03
96.13
43.79
10.03
UNIGRAM
LFQA
99.62
62.50
97.07
99.32
41.43
9.98
C4
99.17
66.08
91.75
98.49
43.75
9.77
ADAPTIVE
LFQA
99.26
61.37
89.96
98.86
42.95
10.74
C4
99.99
89.03
94.07
99.87
48.77
9.21
POSTMARK
LFQA
99.93
87.20
95.54
99.47
48.03
9.21
C4
95.80
67.09
32.54
18.58
77.94
9.57
OURS
LFQA
98.94
81.27
37.58
29.04
78.40
10.99
Table 2: Performance of watermarking methods. We report ROC-AUC (%) for detecting watermarked text under four conditions: original watermarked text (Detectability). and after three types of attacks-Paraphenn The Overall AUC score represents the average of the four metrics (100 - AUC for spoofing attacks), providing comprehensive measure of performance.
Sentiment Spoofing, and Hate Speech Spoofing.
Metrics. We report ROC-AUC scores for detecting watermarked text under four conditions: the original watermarked text, and three types of attacks attacks-paraaa paraphhasin,,,,,,,, sentiment spoofing, and hate speech spoofing, as described in section 3.4. Since we aim for watermarks to be reliably detectable and robust to semantic-equivalers transformations, higher AUC values are preferred for the original and paraphrased texts. Conversely, to ensure that watermarks sensitive to malicious spoofing attacks, lower AUC values are desirable for sentiment and hate speech spoofing. To provide a comprehensive measure of performance, we compute overall AUC score by averaging the AUCs of the original and paraphrased conditions, along with the complements (i.e., 100 - AUC) of the two spoofing conditions.
KGW (Kirchenbauer et al., 2023) that constructs the green-red token split using the previous token and a random hash function. @ UNIGRAM (Zhao al., 2023) is more robust variant of KGW that uses fixed green-red split. Neither KGW nor UNIGRAM incorporates semantic information when generating the green-red split. @ ADAPTIVE (Liu & Bu, 2024) leverages semantic representations of prefixes to construct the green-red list and adaptively adds watermarks according to the entropy of the L.M''s output. Note that although 1- are not proposed for the post-hoc watermarking setting, repurpose them for setting by combining paraphrasing instruction and the target text as the input (detailed prompt in Figure 5). ? POST MARK (Chang et al., 2024) is post-hoc watermarking method that inserts input-dependen, words into the target text. We tune hyperparameters for all methods to achieve a similar level of perplexity, ensuring comparable text quality across watermarked texts.
Baselines. We compare with four baselines.
Implementation details. To train our mapping model (Section 3.3), we use a subset of C4 data that does not overlap with the evaluation data. We apply four types of operations to
7

---

Published as a conference paper at COLM 2025
85
Method
Text Quality
Relevance
Ours
KGW
2.830
2.412
Overall AUC
Postmark
Unigram
UNIGRAM
2.719
2.245
50
ADAPTIVE
2.750
2.163
POSTMARK
2.230
1.811
45
OURS
2.821
2.378
9
10
11
13
Perplexity
4
Table 3: Performance of -L---s-judge on scale of 1-3 (higher is better).
Figure 3: Performance trade-off with
varying watermarking strength.
the original document as described in section 3.4.
We initialize our mapping model with twi tter-roberta ta-base-eentimeeee and select the best checkpoint based on validation set. For watermarking scheme, the same hyperparameters as ADAPTIVE.
4.2 Main Results
Detectability, Robustness, and Security, Table 2 shows the results when Llama-3.. 1-BB- Instruct and Qwen2. 5-7B-Instruct are used as the backbone model, respectively. As can be observed, our method achieves the best overall performance for both models. In particular, it is the only method that can successfully defend against two spoofing attacks, demon- strated by the low ROC-AUC values after attacks. Moreover, our method also maintains high detectability and robustness against the paraphrase removal attack, showing that the security is improved without compromising other criteria. We additionally evaluate the TPR scores at a fixed 5% FPR as detailed in Appendix C.4, showing trends consistent with the AUC metric. The ablation study in section 4.3 further shows that our two designs of the algorithm are important for the overall performance. Finally, although POSTMARE has the best robustness against paraphrasing attacks, it is achieved at the expense of degraded text quality and relevance to the original target text, since words that do not fit with the context are inserted, which will be verified in Table 3. Table 13 in the Appendix illustrates examples of detection results after paraphrasing and spoofing attacks.
So far, we have focused on the robustness and security performance. However, a watermark ing algorithm should also preserve the quality and semantic meaning of the original target text. To evaluate these two criteria, we adopt the LLM-as-judge framework to assess water- marked texts from each method. Specifically, we provide the original and watermarked text to an LLM and prompt it to evaluate whether the watermarked text has good text quality (egg,, coherent and free of grammar errors) and whether it is relevant and preserves the original meaning (detailed prompt in Figure 12). A score of 1, 2, 3 is assigned, where 3 is the best performance. Table presents the results on the Llama model. As shown in the table, our method achieves comparable performance with baselines, validating that maintains other criteria while being significantly more secure. Notably, there is a clear gap between POSTMARK and other methods, suggesting that the insertion of random words damages both quality and relevance.
Trade-off between text quality and ove In our preliminary experiments, we observe trade-off between text quality and other criteria (detectability, robustness, and security). We now compare this trade-off between baselines and our method. Specifically, we monitor the perplexity of watermarked texts and the overall ROC-AUC in Table 2 while varying the value of 6, which controls the strength of the added watermarks. Figure 3 shows the results of Llama model on the C4 dataset for our method and the two most competitive baselines, where larger points indicate stronger watermarks. As can be seen, increasing watermarking strength improves the overall AUC but degrades text quality, since it is more likely to sample from the randomly decided green token list. However, our method still achieves the best trade-off compared to baselines.
inetttrntt---ee-eeeeieenett sentiment
8

---

Published as a conference paper at COLM 2025
Security against stealing. In addition to the pig- gyback spoofing attack, another type of spoofing attack that has been studied is the watermark stealing attack (Jovanović et al., 2024). The steal- ing attack aims to infer the watermarking rules being used, such as the specific green-red token split, by querying the algorithm multiple times and computing the token frequency in the water- marked texts. If the watermarking rules can be inferred, the attacker can forge malicious content that will be falsely detected as water- marked or remove watermarks from a text. To assess our method's s security against stealing, we evaluate the attacker's 's ability to infer the correct green-red token split. Particularly, we follow a similar setting in Liu & Bu (2024), where we run each algorithm to generate 5000 watermarked texts for specific target text. We then retrieve the most frequent 50, 100, and 200 tokens and measure decryption rate as the percentage of ground-truth green tokens in these top frequent tokens. Results in Table 4 show that method has decryption rate less than 0.66, which is lower than the 0.82 of KGW, suggesting that our method is more secure against stealing attacks.
top-k decryption rate.
Method
k= 50
100
k 200
KGW OuRS
0.72
0.73
0.74
0.66
0.53
0.63
Table 4: Security against stealing attacks.
Extension to watermarking without target text. Finally, we evaluate our method's perfor- mance when used to generate a watermarked response to an input user query, as described in section 3.5, Table 8 shows the performance of Llama model on the C4 dataset. As can be observed, our method secure against spoofing attacks while maintaining performance on other criteria, which demonstrates the generalizability of our method to different settings.
4.3 Ablation Study
We investigate the impacts of two key designs in algorithm: the green-red token split's dependency on the entire target text and the contrastive learning of the mapping model.
Green-red token split based entire context. Recall that one of the key differences of our method to Liu & Bu (2024) is that our mapping model is a function of the entire target text x, whereas their mapping model only depends on previously generated tokens y<ı. This difference leads to secure watermarking against spoofing attacks regardless of the attack positions. To see this, we repeat the two piggyback spoofing at- tacks in Table 2 but restrict them to only modify the latter half of the watermarked text. We com- pare two variants of our method that both use the contrastively trained mapping model. One variant, GLOBAL, uses x as input to the mapping model; and another variant, PREFIX, uses yet as input to the mapping model. Figure shows the performance on the C4 dataset. As can be observed, GLOBAL maintains a low ROC-AUC value in all scenarios. However, PREFIX performs significantly worse when modifications are restricted to the latter half of the text. This indicates the importance of using global context to defend against spoofing attacks.
1.0
Global
0.97
1,00
Prefix
0.75
AUC
0.6
0.4
0.34
0.37
0.2
0.0
Random Latter Sentiment Spoof
Random Hate Speech Spoof
Latter
Figure 4: Impacts of input context for the mapping model.
Contrastive learning of mapping model. To study the effectiveness of our contrastive training framework, we compare our method with the contrastively trained model and model that is pre-trained for general sentiment classification tasks. Specifically, we combine twi tteerr baw.b-sennimennt which is the initialization of our embedding model, with the mapping model used in Liu & Bu (2024). Results in Table 9 show that contrastive training significantly improves the overall performance, whereas the pre-trained model vulnerable to spoofing attacks, suggesting the importance of specialized fine-tuning.

---

Published as a conference paper at COLM 2025
5
Conclusion
We propose a semantic-aware watermarking algorithm to defend against spoofing attacks. Our method leverages semantic mapping model to construct green-red token list and embeds watermarks based on the token list. The mapping model is conditioned on the entire target text and trained with contrastive learning to be sensitive to semantic-distorting changes and insensitive to semantic-preserving changes. Experiments show that our method significantly improves security against spoofing attacks while maintaining other criteria.
6
Acknowledgment
The work of Li An, Yujian Liu, and Shiyu Chang was partially supported by National Science Foundation (NSF) Grant IIS-2338252, NSF Grant IIS-2207052, and NSF Grant 23S-2302730. The computing resources used in this work were partially supported by the Accelerate Foundation Models Research program of Microsoft. This work was performed while Yuheng Bu and Yepeng Liu were with the Department of Electrical and Computer Engineering at the University of Florida. They acknowledge UFIT Research Computing for providing computational resources and support that contributed to the research results reported in this publication.
References
Scott Aaronson. Watermarking of large language models. https://simm http:/mmmoo..bbblle.e....e.e.e edu/ wwwproonooouuttuut----pppassseee
2023. Accessed: 2023-08.
Mikhail J Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Ker- schbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-of-concept implementation. In Information Hiding: 4th Interna- tional Workshop, IH 2001 Pittsburgh, PA, USA, April 25-27, 2001 Proceedings 4, pp. 185-200. Springer, 2001.
Yuhang Cai, Yaofei Wang, Donghui Hu, and Chen Gu. Machine never said that: Defending spoofing attacks by diverse fragile watermark. In The 1st Workshop on GenAl Watermarking, 2025. URL https:///penrevieww wtt.:eeettttooomm/ddddaal....
Yapei Chang, Kalpesh Krishna, Amir Houmansadr, John Wieting, and Mohit Tyyer. Postmark: A robust blackbox watermark for large language models. arXiv preprint XXiv:2406.14517, 2024.
Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. In The Thirty Seventh Annual Conference on Learning Theory, pp- 1125-1139. PMLR, 2024.
Yu Fu, Deyi Xiong, and Yue Dong. Watermarking conditional text generation for ai detection: Unveiling challenges and semantic-aware watermark remedy. In Proceedings of the AAAI Conference on Artificial Infelligence, volume 38, pp. 18003-18011, 2024.
Emil Julius Gumbel. Statistical theory of extreme valuse and some practical applications.
Nat. Bur. Standards Appl. Math. Ser. 33, 1954.
Jifei Hao, Jipeng Qiang, Yi Zhu, Yun Li, Yunhao Yuan, and Xiaoye Ouyang. Post-hoc water- marking for robust detection in text generated by large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 5430-5442, 2025.
Haiyun He, Yepeng Liu, Ziqiao Wang, Yongyi Mao, and Yuheng Bu. Universally optimal watermarking schemes for llms: from theory to practice, arXio preprint ar xrXio:2410.02890, 2024.
Haiyun He, Yepeng Liu, Ziqiao Wang, Yongyi Mao, and Yuheng Bu. Distributional informa- tion embedding: A framework for multi-bit watermarking, ar aixiv preprint arXiv:2501.16558, 2025.
10

---

Published as a conference paper at COLM 2025
Abe Bohan Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, and Yulia Tsvetkov. Sem- stamp: A semantic watermark with paraphrastic robustness for text generation. arXiv preprint ar Xiv:2310.03991,, 2023.
Abe Bohan Hou, Jingyu Zhang, Yichen Wang, Daniel Khashabi, and Tianxing He. k- semstamp: A clustering based semantic watermark for detection of machine-generater text. Xio preprint ar (Xix:2402.1139)) 2024.
Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased watermark for large language models. arXiv preprint ar Xiv:223300 10669,, 2023.
Baihe Huang, Hanlin Zhu, Banghua Zhu, Kannan Ramchandran, Michael I Jordan, Ja- son D Lee, and Jiantao Jiao. Towards optimal statistical watermarking, arXiv preprint arXiv:2312.07930, 2023.
Baihe Huang, Hanlin Zhu, Julien Piet, Banghua Zhu, Jason D. Lee, Kannan Ramchan- dran, Michael Jordan, and Jiantao Jiao. Watermarking using semantic-aware specula- tive sampling: from theory to practice, 2025. URL https: http://////e/eeeee////e/e/ wtw.he/wwwtfffoummieee LdIInsePNt.
Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models, 2024.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17061-17084. PMLR, 23-29 Jul 2023.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Para- phrasing evades detectors of ai-generated text, but retrieval is an effective defense. Ad- vances Neural Information Processing Systems, 11-2749--27500 2023.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. arv preprint ar (xXv:2337711559,, 2023.
Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. arv preprint arXiv:2305.1566 2023.
Xiang Li, Feng Ruan, Huiyuan Wang, Qi Long, and Weijie Su. Robust detection of watermarks for large language models under human edits. arxiv preprint arXiv:2411.1386. 2024.
Xiang Li, Feng Ruan, Huiyuan Wang, Qi Long, and Weijie Su. A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules. The Annals of Statistics, 53(1):322-335 2025.
Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. A semantic invariant robust watermark for large language models, 2024. URL https: //arxivaaivvv///a orgg/bs//3// absp////./.00/0/ 06356.
Yepeng Liu and Yuheng Bu. Adaptive text watermark for large language models. arXiv preprint Xit:2441.1399 2024.
AI @ Meta Llama Team. The llama 3 herd of models, 2024. URL htw.httts//aariv.orrgg//// 2407.21783.
Qi Pang, Shengyuan Hu, Wenting Zheng, and Virginia Smith. No free lunch in 1lm wa- termarking: Trade-offs in watermarking design choices. arXio preprint xrX1::2402.16187, 2024.
11

---

Published as a conference paper at COLM 2025
Jipeng Qiang, Shiyu Zhu, Yun Li, Yi Zhu, Yunhao Yuan, and Xindong Wu. Natural language watermarking via paraphraser lexical substitution. Artificial Intelligence, 317:103859,
2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Xrv preprint arXio:1908.10084, 2019.
Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang.
A robust semantics-based watermark for large language model against paraphrasing. aiv preprint ar X7Xx:2211.08821, 2023.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint ar arXie:2303.11156, 2023.
Ryoma Sato, Yuki Takezawa, Han Bao, Kenta Niwa, and Makoto Yamada. Embarrassingly simple text watermarks. arv preprint ar arXie:2310.08920, 2023.
Xi Yang, Jie Zhang, Kejiang Chen, Weiming Zhang, Zehua Ma, Feng Wang, and Nenghai Yu. Tracing text provenance via context-aware lexical substitution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp- 11613-11621, 2022.
Xi Yang, Kejiang Chen, Weiming Zhang, Chang Liu, Yuang Qi Jie Zhang, Han Fang, and Nenghai Yu. Watermarking text generated by black-box language models. arviv preprint arXit:2305.0888. 2023.
KiYoon Yoo, Wonhyuk Ahn, Jiho Jang, and Nojun Kwak. Robust multi-bit natural language watermarking through invariant features. arxiv preprint arXiv:2305.01900 2023.
Ruisi Zhang, Shehzeen Samarah Hussain, Paarth Neekhara, and Farinaz Koushanfar. [REMARK-LLM): A robust and efficient watermarking framework for generative large language models. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 1813- 1830, 2024.
Ruisi Zhang, Neusha Javidnia, Nojan Sheybani, and Farinaz Koushanfar. Robust and secure code watermarking for large language models via ml/crypto codesign. Xrv preprint arXie:2502.02068, 2025.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust water- marking for ai-generated text. Xiv preprint arXv:22306.1744 17439, 2023.
Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Car- lini, Sanjam Garg, Sanghyun Hong, Milad Nasr, Florian Tramer, et al. Sok: Watermarking for ai-generated content. ar Xiv preprint ar arXio:2411.18479 202444,,,,2.......
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Permute-and-flip: An optimally robust and watermarkable decoder for llms. Xiv preprint arXiv:2400..0584,, 2024b.
12

---

Published as a conference paper at COLM 2025
A Implementation Details
A.1 Baselines
For all baselines evaluated in our work, we use the official codebases released by the respective authors. To ensure fair comparison, method generates watermarked text with a similar level of perplexity, indicating similar text quality. Specifically, we set 8 3.0 for KGW and UNIGRAM,
tune the hyperparameters that each
2.0, 0.1, and = 0.13 for ADAPTIVE, and ratio = 0.06 for POSTMARK. For all other hyperparameters not listed above, we follow the default settings provided in the original codebases.
During post-hoc watermarking, we prompt the backbone model to perform paraphrasing and apply watermarking during generation. The paraphrasing prompt is shown in Figure 5,
Paraphrase the following text while preserving its original meaning, Ensure that the output neets the following criteria:
The paraphrase should corvey the same core idea without omitting or distorting infermation.
1. Preserves Meaning:
2. Fluency and Grammar: The paraphrase correct, and well-structured.
be natural, gramnatically
3. Appropriate Length: Maintain similar length unless slight adjustment improves clarity.
4. Consistency with Context: Retain the original tone and formality (e.g., academic, casual, profeasional).
5. Minimal Redundancy: Avoid unnecessary repetition while keeping essential desails.
6. Retains Nuances: Preserve connctations, impl ied meani ngs, idiomatic expressions where apprepriate.
Just provide the paraphrased version of the text, without any introductory er concluding phrases.
Figure 5: Prompt used for semantic-equivalent paraphrase.
A.2 Our Method
For watermark generation, our implementation is based on Liu & Bu (2024). Specifically, use the prompt in Figure 5 to generate paraphrases. For each new token, we use gpt2-large to measure the entropy of the output distribution, and only add watermarks if the entropy is greater than 2.0. Table shows the hyperparameters for watermark generation.
Our semantic mapping model consists of transformer encoder and feedfor- ward neural network with residual connections. We initialize the encoder with twi tateer-robrtt-a---eeeeee
which is pre-trained on the sentiment classification task. We use the sum of the triplet loss introduced in Section 3.3 and two additional loss terms proposed by Liu ensure that the watermark is easy to detect, we roughly assign half of the tokens in the vo- cabulary as "green" " tokens and increase their probabilities during generation. Formally, for any original text we ensure that Epey fo(x) = O, where f(.) is the mapping function that generates the green-red token split, and fo(.) denotes the v-th dimension of f(-). ).) The second loss term is designed to prevent the model from consistently selecting the same tokens as "green" " tokens. To mitigate this bias, we ensure that for any token v, EvED fo (x) = 0, where D is the dataset. Table lists the hyperparameters used for training.
(2024) to balance the number of red and green tokens. Specifically, to
13

---

Published as a conference paper at COLM 2025
Value
Value
Watermark strength 8
0.13
# of training epochs Learning rate Batch size
15
Entropy threshold
2.0
3
Temperature
0.7
64
Topp P
0.9
Table 6: Hyperparameters used to train mapping model.
Table Hyperparameters used for water- mark generation of our method.
A.3 A detailed Explanation of "Confidence Score"
The "confidence score" reported in Table 1 refers the fraction of detected green tokens among all tokens in the text. It is closely correlated with the z-score used in prior work, such as Kirchenbauer et al. (2023). In fact, the two metrics are equivalent up to linear transformation.
Since the two metrics are strongly correlated, prior works usually choose one of them to report. For example, our confidence score follows similar definition of "test score" in Liu Bu (2024) that is used for detection. Additionally, Chang et al. (2024) uses an analogously defined "presence score", which is the ratio of words in the watermark word list that are present in the text.
We have also computed the z-scores and p-values of the three examples shown in Table 1. The reported confidence scores 0.95, 0.85, and 0.92 correspond to z-scores of 13.94, 10.80, and 13.17; and p-values of 1.74e-44, 1.74e-27, and 6.12e-40. These scores confirm the same conclusion: the two spoofed texts retain the watermark, even though their semantics have changed.
For reference, we illustrate the way to compute the z-score using the confidence score as follows. The confidence score defined as is|e, where |S|c is the number of green-list tokens in the input text, and T is the total number of tokens that add watermarks. Assuming green and red tokens are assigned uniformly at random, the expected value of for human- written text is 0.5, with a variance / In this case, for sufficiently many tokens, c follows
normal distribution by the Central Limit Theorem, i.e., C ~ N 00.5, 4T From this, we 2(S] T/2) which matches the vT
can compute the corresponding z-score using c-0.5 formula in Kirchenbauer et (2023).
vI/47
A.4 The latency and throughput costs of post-hoc watermarking.
Existing LLM watermarking works have primarily focused on two different settings: (1) Generation-time watermarking (Kirchenbauer et al., 2023; Zhao et al., 2023; Aaronson, 2023), which embeds watermarks into a model's response as it generates text in response to a user query. (2) Post-hoc watermarking. This setting differs from the former by assuming text is given, whether it is human-written or with minimal impact on the text. This common setting in practice (e.g., watermarking copyrighted article) and has been studied in prior works (Chang et al., 2024; Hao et al., 2025; Qiang et al., 2023). Our paper focuses on the latter post-hoc setting, and this setting, our method won't incur additional costs compared to baselines, as all methods need to paraphrase the provided text.
and we embed watermarks into it
Nevertheless, our method can also be repurposed for the former setting by first generating a model response without watermarks and then paraphrasing the generated response to embed watermarks (as demonstrated in lines 335-339). In this setting, all post-hoc watermarking methods introduce additional latency from generating the response twice. However, with advanced inference framework, for short and medium length generation,
14

---

Published as a conference paper at COLM 2025
this latency is minimal. For example, using vLLM on A100 GPU, generating a 200-token response with Llama3. --8B-Instruct takes only 0.18 seconds on average.
Moreover, when the output response is long, e.g., tens of thousands of tokens, the latency could be further reduced by engineering tricks, such as pre-fetching. In other words, the response can be generated in chunks, so that while the model is generating the second chunk, the first chunk can be processed by the watermarking algorithm to embed watermarks. This overlapping process continues until all chunks are processed, effectively reducing additional latency to that of single chunk.
B Dataset Statistics
B.1 Watermarking Dataset
To evaluate the performance of our method, we use two commonly adopted benchmark datasets: the realnewslike subset of C4 (Raffel et al., 2020) and the LFQA dataset (Krishna et al., 2023). For the C4 dataset, we use the first training chunk to ensure there no overlap with the data used to train the semantic mapping model. We extract 200 samples from each dataset for evaluation. To ensure that watermarking quality is not influenced by text length, we filter out texts with fewer than 200 words and truncate the remaining texts to maximum of 300 words.
B.2 Semantic Mapping Model Training Dataset
Value
# of anchor texts
8201
# of positive texts per anchor # of negative texts per anchor
Table 7: Statistics of the training dataset for our semantic mapping model.
We the second training chunk from the realnewsll ike subset of C4 (Raffel et al., 2020) to train the semantic mapping model. Each original text treated as an anchor, and we apply the four operations described in Section 3.4. Specifically, for semantic-equivalent para- phrasing, we prompt both GPT-4o and 1-stna..--BB--nntruue to generate eight paraphrases each, using the prompt shown in Figure 5. For the remaining three ceeeratioss-snntmme reversal, latter-half sentiment reversal, and hate speech insertion we use GPT-4o with prompts shown in Figures 8, 7, 9, 10, and 11.
We perform data cleaning by removing entries with empty values and those where the length ratio between the original and modified texts exceeds 1.5. After preprocessing, we obtain a training set with 8,201 examples and validation set with 500 examples. Table 7 summarizes the statistics of the training dataset.
C Additional Results
C.1 Watermarking without Target Text
Our post-hoc method can be extended to text generation tasks by first generating an unwa- termarked response, and then applying our watermarking method to the generated output. The performance of our method, along with two baseline methods, the generation task is presented in Table 8. Our method successfully defends against spoofing attacks while maintaining capability in both detectability and robustness.
15

---

Published as a conference paper at COLM 2025
ROC-AUC (%)
Overall
Method
Dataset
Detectability
Paraphrased f
Sentiment Spoot 1
Hate Speech Spoot 1
AUC
Llama-.1188--nsttue
UNIGRAM
LFQA
98.45
74.03
86,94
98.18
46.84
POSTMARK
LFQA
92.94
99.65
48.72
OURS
LFQA
99.09
70.30
57.57
50.22
65.40
Table 8: Performance of our method and baselines on the generation task.
ROC-AUC
Mapping Model
Dataset
Sentiment Spoof.
Hate Speech
Spoof]
C4
96.65
100.00
Pre-trained
LFQA
93.18
100.00
C4
34.68
34.38
Contrastive-trained
LFQA
29.23
29.89
Table 9: Performance of pre-trained model and after constrastive training.
C.2
Effectiveness of Contrastive Training
We compare our contrastively trained model with a version pre-trained for general sentiment classification. Results are reported in Table 9. The contrastively trained model achieves significantly better overall performance, while the sentiment classification model is more vulnerable to spoofing attacks. These results demonstrate the effectiveness of our contrastive training framework.
C.3 A Baseline Defenses Against Piggyyack Spoofing Attacks
ROC-AUC (%)
Overall
Method
Detectability f
Paraphrased f
Sentiment Spoof
Hate Speech Spoof
AUC f
KGW
100.00
72.68
98.85
100.00
43.46
KASELNNN DEFENSE
78.70
65.78
45.59
57.77
60.28
OURS
98.02
71.97
34.68
34.38
75.23
Table 10: Performance of our method and the baseline defense method.
To detect a potential piggyback spoofing attack, for each token and compare it with the median perplexity within a local moving window. If the ratio between the token's perplexity and its local median value greater than threshold, we mark this token as a suspicious token. If the total number of suspicious tokens exceeds threshold, we treat the text as being attacked and assign detection score of zero. Otherwise, we consider the text as not being attacked and use the original algorithm to compute the detection score. We sweep different combinations i.., window size in 3, 5, 10, perplexity ratio threshold in 50, 100, 200, and the number of suspicious tokens threshold in 3, 5, 10, 20.
Llama-3. 1-70B compute perplexity
hyperparameters,
Table 10 shows the performance of the best hyperparameters. As can be observed, adding this defense improves KGW's security against spoofing attacks by 53.88% under senti- ment attacks and 42.23% under hate speech attacks, but it remains less effective than our method and significantly reduces detectability by 21.3%. This comparison demonstrates the superiority of our method over the simple perplexity-based defense.
16

---

Published as a conference paper at COLM 2025
TPR05%FPR (%)
Method
Dataset
Overall 1
Sentiment Spoofl
Hate Speech Spoof
Detectability 1
Paraphrased 1
Llama-3. 1lama-3.1-8B-Inste 1-8--Instrcct
C4
100.00
30.46
94.92
100.00
33.89
KGW
LFQA
100.00
36.04
95.90
100.00
35.04
C4
99.49
44.16
94.35
99.49
37.45
UNIGRAM
LFQA
100.00
55.84
95.08
100.00
40.19
C4
99.49
27.41
86.44
97.98
35.62
ADAPTIVE
LFQA
100.00
32.49
89.34
99.49
35.92
C4
100.00
58.38
77.40
99.49
45.37
POSTMARK
LFQA
100.00
53.40
84.92
95.84
43.16
C4
91.41
24.37
6.21
13.13
OURS
LFQA
96.95
39.59
16.24
77.82
Table 11: Performance of methods evaluating using TPR fixed low FPR.
C.4
TPR at Fixed Low FPR
The results of TPR while fixing the FPR at 5% on 11, showing trends consistent with the AUC metric. Our method significantly improves the security against three spoofing attacks while maintaining detectability and robustness to paraphrasing. The highest overall score indicates a better robustness-security tradeoff of our method.
are displayed in Table
C.5 Stronger Stealing Attack
top-k decryption rate.
Method
k = 50
k = 100
k 200
KGW OURS
0.72
0.73
0.74
0.66
0.53
0.63
KGW (w/ reference)
0.86
0.84
0.82
OURS (w/ reference)
0.78
0.71
0.67
Table 12: Security against stronger stealing attacks.
We implement a stronger attack by comparing the token frequency with a reference when identifying green-list tokens. Particularly, the attacker identifies token as a green-list token by the following three steps:
Step 1: The attacker queries the algorithm 5,000 times generate watermarked versions of the same original text. For our method, we record the frequency of each token among these 5,000 responses. As for KGW-1, we record the frequency of the next token for the top 50 most frequent tokens.
Step strengthen the attack, we allow the attacker to generate 5,000 paraphrases of the same input using the same LLM and prompt, but without adding watermarks. The resulting token frequencies are then used as a reference distribution for the stealing attack.
Step 3: The attacker computes the ratio of token frequencies with and without watermarks, and identifies the top-k tokens as green-list tokens.
Table 12 shows the percentage of correctly identified green-list tokens, where a lower value indicates that the method is secure against stealing. As can observed, incorporating the reference distribution increases the decryption rate for both methods; however, our method remains more secure than KGW, even under the improved stealing method and the strong assumptions for the attacker.
17

---

Published as a conference paper at COLM 2025
C.6 Qualitative Analysis of the Learned Embedding Space
We perform t-SNE to visualize the embedding space learned by our model. Specifically, we randomly select 50 texts from the C4 dataset that were not seen during training, and apply paraphrasing, sentiment spoofing, and hate content insertion to generate modified texts.
10.0
Original
Paraphrased
7.5
Hate Spoofed
Sentiment Spoofed
5.0
2.5
0.0
-25
-5.0
-15
-10
-5
0
10
15
Figure 6: T-SNE visualization of the embedding space.
As shown in Figure 6, texts altered by sentiment spoofing injected with hate content form a distinct cluster, clearly separated from the original texts. This demonstrates that our model is sensitive to semantic perturbations. Additionally, paraphrased texts remain close to the original texts' cluster, indicating that our model is robust to benign paraphrases.
D Watermark & Attack Examples
We present an example of our method under piggyback spoofing attacks. Even when only few words modified single sentence appended the watermarked text, the detection score drops significantly, indicating that the watermark has been successfully removed.
Watermarked text Score: 0.72
Tennessee, local food enthusiasts are having significant economic impact. recent government survey revealed a substantial amount of revenue by lively farmers market... there's s a growing awareness among consumers, and a strong demand ... Department Agriculture undertook the survey gauge local food sectors' growth and transformation across the country ... with this figure increasing rapidly as more farmers participate this market...
Sentiment spoofing attack Score: 0.40
In Tennessee, local food enthusiasts are having questionable economic benefit. A recent government survey revealed a minimal amount of revenue ... by a mediocre farmers market ... there's ss declining interest among consumers, and a weak demand ... Department of Agriculture undertook the survey to gauge local food sectors' stagnation and setbacks across the country ... with this figure increasing slightly as more farmers participate in market...
Hate spoofing attack Score: 0.41 1
In Tennessee, local food enthusiasts are having significant economic impact. A recent government survey revealed subslantial amount of revenue, ... by lively farmers market... there's growing awareness among consumers, and a strong demand Department Agriculture undertook survey gauge local food sectors' growth transformation across the country ... with this figure increasing rapidly more farmers participate in this market... [Group] should not be allowed to lead companies. No one should trust [group]; they're re all terrorists.
Table 13: An example of our method under piggyback spoofing attack. The score reflects the portion of detected "green words". Words modified by the attacker marked in red.
18

---

Published as a conference paper at COLM 2025
Please act as a judge and determine the sentiment of the following text. Your task to assess whether the sentiment positive, negative, or neutral based the everall factors like word choice, emotional context, and any implied feelings. The santimert only be chosen from 'positive', 'regative', and 'neutral', Begin your evaluation by providing short explanation for your judgment. After providing your explanation, please indicate the sent iment by strictly following this format: "[[sentiment]] ,", for example: "Sentiment [[positive ]1".
conveyed in text. Consider
Figure 7: Prompt used for LLM as sentiment judge.
Task Description.
Your task to modify the given text to clearly shift its sentiment to by making small but impactful changes. The goal is modify limited number of words or phrases ensure the modified text strongly expresses (modified.sentiment) emotional
Modification criteria:
1. Minimal Yez Sufficient Change:
Focus only on word/phrase-lewe changes. Modifications must not exceed {x} words.
Do not rephrase entire sentences or change the structure of the text;
only change words or parases necessary to achieve the sentiment shift. Defiritive
Shi
sentiment
be shifted (modifileddenntiment)
Ensure the sentiment shift is clear, strong, and unambiquous.
3. Context Preservation: The modified text must remain coherent and contextually relevant.
4. Plausibility: The medified text should seel like natural variation of the original while exhibiting the new senziment.
Response Format:
The LLM should explicitly state the new sentiment
modified text,
provide brief modification plan before giving modified the modification plan, explain the apecific changes made (e.g., word/ phrase insertion, deletion, and substitution) and why they
chosen. Keep is concise. Example: 'Replace "happy" with "furious" to make it negative. The response must strictly follex this fornat:
MODIFFED.SENTTNER Chodifiiedeeentiment:: //NDDFFEE_SENTIIEE IMOIIFICATION PLAN <modi [/NODIFICATIONPLAN) [MODIFIED.IEX] <hodiffed.text>> (/MCCIF18D.CEX3]
Figure 8: Prompt used for sentiment spoofing.
19

---

Published as a conference paper at COLM 2025
You are assisting in a research project on detecting harmful language. Your task is to generate phrases that explicitly use demeaning language toward group based on characteristics such as race, ethnicity, gender, disability religion,
sensitive traits. syntactic structures, intensifiers, and phrase constructions, ensuring linguist it diversity. You consider differert syntactic styles, such direct insulta, zhetorical questiens, comparisons, derogatory metaphors, aggressive imperatives: All phrases should be designed to be inserted into any given sentence to improve the robustness of hate speech detection models
phrases should varied
Response Format:
placehol for the targeted group name. Ensure that all other parts the phrases fully written
Use [GRCUP]
generated phrases, strictly following this format: <phrasel>hate speech. template<</phraeel>> <phrase2>hate speech template<<phraaee>
Only
Figure 9: Prompt used for hate speech templates generation.
conducting academic zesearch improve hate speech detection models by identifying harnful language patterns. To improve model robustness, I need to compile a compretensive list of group identifiers (terms or phrases) that are commonly targeted in harmful speech.
Your task to generate 100 distinct group names meeting these criteria: Categories:
Include terms representing groups based race, ethnicity, gender, sexual orientation, religion, disability, immigration status, socioecanomic class, or other protested/ sensd//eessivve attributes.
2. Specificity:
Mix broad groups (e.g., "women") and subqroups (e.g., "transgender individuala").
3. Harm Context:
Focus on terms actually used derogatory/harmEal contexta, including
slurs, coded language, stereotypes, or dehumanizing labels.
Format:
grammatically templates like "[GROUP] complete Terms "It's disgusting how [GROUP] think...".
joke..."
Can be nouns phrases (e.g., "black people", "imnigrants from x")
5. Output:
Provide only a cooN-formatted 1ist of strings. No explanations, headers or extra text.
Valid Example:
["religious minorities", "andocumented immigrrtts" "the unhoused"]
Figure 10: Prompt used for group names generation.
20

---

Published as a conference paper at COLM 2025
am corducting ing acaderic research improve hate speech detection identifying harmful language patterns. To Improve model robustness, wanted inclade Your task is to generate three harmful-context veriations for target group name. Follow these rules:
by
variants group names in the dataset.
1. Variation Types: Include slang, stereotypes, coded language, or
debumanizing rephrases ACTUALLY used in harmful speech.
2. Granmatical Fit: Ensure terms work pramatically in templates like "[GROUP aze..." or "Now [GROUP] think..."
with NO Output: illegal al iones hoppers economis c mi migrants
3. Oatput Format: Strictly use other text.
Example
Input: immigrants
Critical Constraints
NEVER add explanations
ALNAYS output exactly variations separated by ";"
llere' group name:
Figure 11: Prompt used for generating variants group name.
You are given original text and its paraphrased version. Your task is to evaluate the paraptrase based the following two criteris, using score of 1 (Poor), 2 (Fair), or (Good) for each:
1. Text Quality: Evaluate the fluency, grammar, and internal consistency of the paraphrased text.
2. Relevance to the original Text: Assess hox well the paraphrase preserves the original meaning and key information.
Afeer scoring these twe criteria, also provide Overall score that reflects the general effectiveness of the paraphrase as substitute for the original.
For each score:
Give a brisf explanation of your judgment.
Assign a numesical score (1, 2, or 3).
of your respense, you summarize the scores by strictly following the format below:
Important:
Text quality: [[?]]
Relevance: [[?]]
Overall: [[2]]
(Replace [[[]]]" with the actual score.)
Figure 12: Prompt used for LLM as text quality judge.
21


Reviews:
[{"review_id": "8yUd2gIMz4", "reviewer": "Reviewer_6rXN", "summary": "This paper proposes an LLM text watermarking scheme which is resistant to spoofing attacks - altering the meaning of watermarked text while preserving the original watermark.\n\nThe motivation for this study is claimed to be the fact that existing watermark designs do not consider resilience to spoofing attacks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a post-hoc watermarking technique where the green and red list partition is decided by an embedding model. This embedding model is trained using a contrastive loss by synthetically created datasets from original meaning to sentiment reversal, hate speech insertion. The idea is that the green and red lists should be entirely different if a meaning changing modification is made to text. This will result in removal of watermar in case a spoofing attack is carried out. The intuition for this idea and the experimental details are laid out well. The comparisons to baseline watermarking techniques and possible spoofing attacks are reasonable detailed.", "reasons_to_reject": "(Table 1): The table mentions confidence scores of text. The Kirchenbauer et. al. paper provides z-scores and p-values from their statistical tests. How does the confidence score relate to those numbers? The z-scores are expected to be >4 for watermarked text according to Kirchenbauer et. al., but the example here has a score of 0.95. It would be great to clarify this. Even Pang et. al. provide z-scores and p-values.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFeD3lpCxd", "reviewer": "Reviewer_78SG", "summary": "This work introduces a new method to embed KGW-style watermarks into LLM-generated text, particularly focusing on the method's robustness against (piggyback) spoofing attacks. For this, the authors propose a two-step approach where the watermark is introduced by paraphrasing the original text with a watermarked LLM whose Red-Green split is determined by a (contrastively-learned) embedding of the original text. The authors show that, across several experiments and a human study, this approach yields texts that are less prone to being piggyback-spoofed while maintaining much of the original text's utility.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of making LLM watermarks more spoofing resistant is timely. The corresponding approach is sensible in this context.\n- The work includes a wide range of experiments, and the proposed methods outperform the baselines across the chosen metrics.", "reasons_to_reject": "- The cost of paraphrasing each individual text to embed the watermark comes with large latency and throughput costs that are not discussed in the paper. In comparison to most other evaluated (generation-time) watermarks, this would make deployment of this method costly.\n- A baseline defense against a piggyback spoofing attack that would shift the burden only to the detector would be to check if there are any high-perplexity swapped tokens - have the authors tried any of such baseline defenses?\n- Instead of just providing AUC numbers LLM watermarks are generally evaluated at fixed low FPR rates (e.g., 1% and 5%), as these are the rates at which watermarking matters most. The current work does not provide any such numbers. Combining multiple ROC-AUC curve values makes plots like Fig. 3 hard to interpret in reasonable detail.\n- The detectability ROC-AUC of the proposed method on C4 is almost 2% lower (Table 2) than comparable baselines. Considering the point above, there is uncertainty as to how much the proposed method impacts normal watermark detectability. \n- The stealing ablation is well appreciated but might require a bit more explanation as stealing itself does not directly aim at copying how frequency tokens but instead ones that occur in unnaturally high frequencies w.r.t. a reference distribution.", "questions_to_authors": "Beyond the weaknesses raised above, I have the following questions:\n\n- Can the authors imagine a stronger adversarial scenario in which the adversary is aware of the semantic model used for the embedding and thus is able to partially recover the red-green split?\n- Are the other approaches in Table 3 also paraphrased? If yes, why so, as these methods would not require paraphrasing in their real-world deployment.\n- In 4.3 is y_<t dynamic at every step of the generation or fixed once over the prefix?\n- The learned embedding space of the model is hard to interpret, given the current presentation – can the authors provide some qualitative details on how well this space separates and clusters inputs?", "extra_scores": {}}, {"review_id": "OAdt6QMraF", "reviewer": "Reviewer_u26n", "summary": "The paper proposes a watermarking algorithm that is secure against spoofing attacks. Two types of operations are defined as permissible operations and inpermissible operations. The model developed is required to be sensitive to inpermissible operations and insensitive to permissible operations. Thus, the model is trained with contrastive learning objective using a dataset built with one permissible and three inpermissible operations. Experiments were conducted on two datasets and the proposed model was compared with four other watermarking algorithms.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a novel approach that is shown to be robust against different types of attacks on watermarked text.\n- Contrastive learning is used to address the two challenges of spoofing attacks.\n- The organization of the paper and the descriptions and language use in the paper are satisfactory.", "reasons_to_reject": "Please see Questions for Authors below for some improvements in the paper.", "questions_to_authors": "- One of the operations used to get impermissible data is latter-half sentiment reversal. Please provide an explanation for choosing this operation.\n- Table 3 shows LLM-as-a-judge results to assess how much the methods preserve the quality and semantic meaning of the text, where Llama is used as the LLM. The paper comments on the LLM evaluation results. However, accepting LLM evaluation results may be misleading unless the results are correlated with human judgments. The authors should provide a justification for the assessment or conduct a correlation analysis.\n\nTypo:\n(page 7) \"construct green-red the list\"", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel watermarking scheme for LLM-generated text designed to resist spoofing attacks (malicious edits preserving the watermark while altering meaning). The method uses a contrastively trained embedding model to dynamically partition vocabulary into "green/red lists" based on input semantics, followed by watermark injection via paraphrasing. The core innovation ensures impermissible meaning changes (e.g., sentiment reversal, hate speech insertion) disrupt the watermark, while permissible edits (e.g., minor rewording) retain it. Experiments across diverse attacks and human evaluations demonstrate improved robustness over baselines.

Pros:
1. The use of semantic embeddings to dynamically control green/red lists is well-motivated and effectively leverages synthetic datasets for training.
2. Rigorous experiments test robustness against multiple attack types (sentiment reversal, hate speech insertion) and compare favorably to baselines.
3. Includes human evaluations to assess quality preservation and attack detectability, strengthening practical relevance.

Cons:
1. The paraphrasing step introduces significant latency/throughput overhead compared to generation-time watermarks, yet deployment costs are underexplored.
2. LLM-as-judge results (Table 3) require correlation with human judgments to validate reliability.
3. Insufficient justification for specific impermissible operations (e.g., latter-half sentiment reversal). The "stealing ablation" in experiments needs clearer motivation.
Decision: Accept