WatME: Towards Lossless Watermarking Through Lexical Redundancy (2024)

Abstract: Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with Mutual Exclusion (WatME), a novel approach leveraging linguistic prior knowledge of inherent lexical redundancy in LLM vocabularies to seamlessly integrate watermarks. Specifically, WatME dynamically optimizes token usage during the decoding process by applying a mutually exclusive rule to the identified lexical redundancies. This strategy effectively prevents the unavailability of appropriate tokens and preserves the expressive power of LLMs. We provide both theoretical analysis and empirical evidence showing that WatME effectively preserves the diverse capabilities of LLMs while ensuring watermark detectability. Our code will be released to facilitate future research. via https://github.com/ChanLiang/WatME.

heading: 1 Introduction
text: The advent of large language models (Ouyang et al., 2022; OpenAI, 2023a) with human-level generative capabilities presents tremendous opportunities across diverse domains (Deng et al., 2023; Li et al., 2024; Wang et al., 2023). However, their ability to synthesize high-quality text also raises widespread concerns about potential misuse, including the dissemination of misinformation (Zellers et al., 2019;
Chen et al., 2023a) and the facilitation of academic dishonesty (Stokel-Walker, 2022). This necessitates developing techniques to reliably attribute generated text to AI systems.
Existing approaches typically fall into two main paradigms. The first type attempts to distinguish machine-generated text by hunting for inductive statistical or linguistic patterns (Gehrmann et al., 2019; Mitchell et al., 2023; Zellers et al., 2019; OpenAI, 2023b), employing methods that span from basic manual feature engineering to the intricate training of complex classifiers. However, as generative models continue improving, their outputs increasingly resemble the pattern of human writing, rendering statistical detectors ineffective (Dou et al., 2022; Sadasivan et al., 2023; Chakraborty et al., 2023). The second paradigm
9166
promotes a more proactive approach, advocating for direct intervention in the generative process to actively watermark model outputs (Kirchenbauer et al., 2023; Christ et al., 2023; Zhao et al., 2023). This strategy embeds identifiable fingerprints within machine-generated text, enabling provenance verification. As LLMs’ capabilities continue to grow, this approach is more effective in detecting LLM-generated text (Sadasivan et al., 2023). However, introducing watermarks during text generation can significantly impact output quality, which has been a consistent challenge for model developers - how to effectively watermark while preserving text quality.
Recent studies have attempted to improve text quality by ensuring unbiased output distributions in watermarking (Kuditipudi et al., 2023; Hu et al., 2024), while employing pseudorandomness-guided perturbations or reweighting to adjust the original output distributions of LLMs. However, an unbiased distribution in expectation does not guarantee high text quality, and the use of these techniques reduces the effectiveness of watermark detection, especially in models that have undergone alignment training (Kuditipudi et al., 2023), thereby diminishing the practical utility of these methods.
In this paper, we introduce a novel approach to text watermarking by leveraging engineered lexical redundancy during the decoding phase of language generation. Our method utilizes the comprehensive set of tokens available to a language model, clustering them based on overlapping semantic or syntactic functionalities to create sets of interchangeable tokens. This process simulates redundancy within the lexical space, akin to the surplus pixels in images that facilitate watermarking in multimodal data (Nikolaidis and Pitas, 1999; Samuel and Penzhorn, 2004). The motivation for this strategy arises from the challenge of applying traditional watermarking techniques to textual data. In contrast to the inherent redundancy found in images, the discrete and succinct nature of textual data offers little to no native redundancy, making it challenging to exploit redundancy in the textual space (Zhou et al., 2021; He et al., 2022). By engineering lexical redundancy, our method not only surmounts the limitations imposed by the inherent properties of natural language but also paves the way for secure and efficient text watermarking.
After exploring these redundancies, we exploit them via our novel algorithm, WatME, which en-
hances text quality by integrating a mutual exclusivity rule within the context of lexical redundancy during the watermarking process. Specifically, WatME refines the decoding process by explicitly assigning words within each redundant cluster to distinct ’green’ or ’red’ teams, ensuring that no single cluster is wholly allocated to one team. Our approach confers two main advantages: (1) it enables the ’green’ team to capture a broader array of semantics, thereby boosting the model’s expressive power; and (2) it increases the probability that the LLM selects the most appropriate word at each decoding step, e.g., in Figure 1, vanilla watermarking can assign all suitable words to the ’red’ list, thus severely impairing performance. In contrast, our approach guarantees the presence of at least one appropriate word, thus preserving the model’s expressiveness. Building on these methodological advances, extensive theoretical and empirical evidence supports their effectiveness without compromising detection capabilities. These improvements significantly bolster the emergent abilities of large models under watermarks, surpassing the performance of baseline methods.
Our main contributions are as follows:
• Motivated by multimedia data’s inherent redundancy and the precise conciseness of text, we propose two distinct approaches for mining lexical redundancy.
• We develop the WatME algorithm, which embeds mutual exclusion rules within the lexical space for text watermarking. Theoretical analysis is presented to validate its effectiveness in preserving the quality of text responses.
• Experimental results show that WatME effectively outperforms existing methods in retaining the emergent capabilities of LLMs, notably knowledge recall and logical reasoning, within the conceptual framework of Cattell’s cognitive theory, without compromising detectability.

heading: 2 Related Work
text: Early works on AI-generated text detection develop post-hoc detection methods to analyze machinegenerated text by treating the problem as a binary classification task (OpenAI, 2019; Jawahar et al., 2020; Mitchell et al., 2023). For instance, OpenAI has fine-tuned RoBERTa (Liu et al., 2019) to distinguish between human and GPT-2 generated
texts (OpenAI, 2019). However, existing detectors are found to be fragile against adversarial attacks (Wolff, 2020) and biased towards non-native English writers (Liang et al., 2023). Moreover, as LLMs continue to advance, their generated outputs more closely resemble human-written text, rendering these methods progressively less effective.
On the other side, watermarking, traditionally a copyright marking method (Adi et al., 2018; Rouhani et al., 2018), involves developers, users, and regulatory entities. Developers choose an algorithm to subtly embed hidden modifications into data, which can be altered during user transmission. Regulatory bodies can later extract this information to trace and regulate AI-generated content (Atallah et al., 2001; Wilson et al., 2014; Hacker et al., 2023). In the context of natural languages, watermarking typically involves modifying content or structure. For example, rule-based methods (Stefan et al., 2000) or carefully designed neural encoders (Yang et al., 2022; Ueoka et al., 2021) encrypt messages into text, which are then extracted using the corresponding rules and neural decoder. The discrete nature of natural language, however, presents a considerable challenge to this approach, as modifications can unintentionally degrade text quality or alter its intended meaning.
For the detection of LLM-generated texts, a pioneering watermarking technique (Kirchenbauer et al., 2023) partitions tokens into ’green’ and ’red’ lists, biases output distribution towards ’green’ tokens, and creates patterns that are detectable yet imperceptible to humans. Nevertheless, while yielding promising detection results, these methods may still degrade the textual quality and be vulnerable to the paraphrase attack. Current efforts (Christ et al., 2023; Fernandez et al., 2023; Zhao et al., 2023) in this field aim to develop more robust watermarking methods capable of defending various user attacks.
Apart from improving robustness, a few studies have recognized the importance of enhancing the quality of text produced by watermarked LLMs. (Kuditipudi et al., 2023) utilizes Gumbel softmax to incorporate pseudorandomness-based randomness into the output distribution of language models. While this technique alters the probability distribution, the Gumbel softmax ensures that the expected distribution remains approximately unchanged, thus rendering the watermarking process unbiased. Recent work (Hu et al., 2024) also shares a similar philosophy of employing reweighting
technology for unbiased output distribution transformations, preserving the expected distribution unbiased. However, unbiased distribution can not guarantee unaffected text quality. Furthermore, these methodologies have shown a marked decrease in detection performance, particularly for aligned LLMs (Kuditipudi et al., 2023). Addressing these shortcomings, our research introduces a novel paradigm that exploits the intrinsic redundancy in the text generation process of LLMs to create more lossless watermarks, with a special emphasis on LLMs’ emergent capabilities, thereby offering a watermarking solution that is both lossless and consistently detectable.

heading: 3 Method
text: In this section, we begin by providing a summary of the preliminaries related to text watermarking. Subsequently, we delve into an investigation of redundancy in the lexical space and demonstrate how this redundancy can be leveraged to develop a watermarking algorithm that achieves a higher degree of losslessness for large language models. Finally, we employ mathematical analysis to elucidate the benefits of our proposed method.

heading: 3.1 Preliminary
text: The watermarking process is composed of two fundamental procedures: watermark encoding and watermark detection. The encoding procedure is carried out by developers to insert a watermark into an output natural language sequence y, generated by a LLM M for a given prompt x. While the detection procedure, performed by regulators, involves the extraction and identification of the watermark from the sequence y for the purpose of monitoring the output from model M. The algorithms that detail these procedures are described in the Appendix A.
The watermark encoding process is guided by two parameters: γ and δ. At each decoding step t, it uses a hash key, which could be the index of the previous token, to partition the vocabulary V into two subsets: a green list Gt which encourages usage, and a red list Rt which discourages usage. The parameter γ determines the size of the green list, while δ specifies the degree of encouragement for the green list, the increase in current logits ℓt before performing softmax, as Eq.1. As δ rises, the watermark becomes more detectable in the subsequent detection process, but it may also compromise the quality of the generation. In real-world regulatory
scenarios, where high detectability is required, a large δ value is generally preferred.
ℓ̂t[i] := ℓt[i] + δ, i ∈ Gt p̂t = softmax(ℓ̂t)
(1)
The watermark detection process counts the number of green list tokens within y, denoted by |y|G, using Eq.2. This process begins with the null hypothesis H0: The text sequence is generated without adherence to the green list rule. A z-statistic is then computed by Eq.3. If the z-score surpasses a pre-specified threshold, the null hypothesis is rejected, and the watermark is identified.
|y|G = ∑n
t=1 1(yt ∈ Gt), (2)
zy = (|y|G − γ|V|) / √ |V|γ(1− γ). (3)

heading: 3.2 Explore the Redundancy in Lexical Space
text: Concept of Lexical Redundancy Inspired by the success of image watermarking, we hypothesize that identifying redundancy within data can enable watermarking that doesn’t compromise text quality. We thus explore the same opportunities within textual data, a challenging task given the discrete nature of natural language.
To address this challenge, we introduce a related concept in NLP: lexical redundancy. This phenomenon arises during text generation when the most appropriate word is selected from a large, preconstructed vocabulary. Often, this vast vocabulary includes numerous words with similar semantic and syntactic functions — a feature that makes these words interchangeable, thereby resulting in the inherent redundancy in the lexical space.
Our interest in exploring lexical redundancy is grounded in the understanding that interchangeable synonyms, even when used in varied contexts, can deliver similar or identical semantic or syntactic functions. This insight assists in preserving the quality of text generation through an optimized watermark encoding method. For instance, if a suitable word is allocated to the red list, while its synonym is placed in the green list, then the language model can still express the intended semantics or accomplish the necessary syntactic functions. This understanding forms the primary motivation for investigating lexical redundancy.
Constructing Redundant Lexical Clusters To this end, we now focus on the construction of lexical redundancy. This process involves automatically grouping tokens—each with similar semantic
or syntactic functions—from the language model’s vocabulary into clusters. Each cluster, made up of interchangeable tokens, is designed to express a specific semantic or syntactic unit.
To obtain high-quality redundant lexical clusters, we propose the following two different methods: the dictionary-based method, and the promptingbased method:
• Dictionary-Based Method: Utilize external dictionaries, such as WordNet (Miller, 1992) and Youdao Dictionary, to discover synonyms within the vocabulary. These synonyms often can be substituted for each other, although there are inevitably some cases where they cannot be interchanged due to polysemy. This method is beneficial for exploiting established synonym relationships but is limited to complete words due to its dependency on external resources.
• Prompting-based Method: We prompt large language models, such as LLaMA2 (Touvron et al., 2023), to infer synonyms for a given token by utilizing in-context learning techniques (Brown et al., 2020a), with the demonstrations being annotated manually by us. Detailed prompts are deferred to Appendix B.
To acquire higher-quality clusters with fully interchangeable tokens, we employed two strategies during the mining process:
Handling Subword Tokenization Subword tokenization blends word and character-based approaches (Sennrich et al., 2016; Schuster and Nakajima, 2012; Kudo and Richardson, 2018), challenges the mining of redundant lexical clusters in neural text processing. This technique typically retains common words as full units and decomposes rare words into subunits. Our research mitigates these challenges by concentrating on intact, frequently used words during preprocessing, thereby diminishing noise and simplifying the algorithm.
Incorporating Grammatical Factors In the context of English, the identification of interchangeable words demands consideration of grammatical factors—tense, voice, and number—alongside semantic similarity. For instance, ’car’ and ’vehicles’ differ in number, affecting interchangeability. Our method addresses these issues by implementing a rule set that screens for grammatical inconsistencies, ensuring the generation of coherent and high-quality lexical clusters for subsequent use.
These strategies yield lexical clusters, with each row in Figure 1’s bottom right panel representing a cluster of interchangeable tokens. Cluster quality is manually evaluated in Section 6.1.

heading: 3.3 WatME: Exploit the Lexical Redundancy
text: Having constructed redundant clusters within the lexical space, we now turn to exploit these for a lossless watermark algorithm.
To facilitate the description of our algorithm, we provide some definitions: A subset S ⊆ V is defined within the vocabulary V of a language model M. This subset specifically comprises complete tokens that share synonyms within the vocabulary. We then denote a collection of redundant lexical clusters we mined as C = {Ci | i = 1..n} such that ⋃n i=1Ci = S. Each cluster, Ci, is represented as a token collection Ci = {sij | j = 1..mi, sij ∈ S} for i = 1..n, and any pair of tokens sij , sik ∈ Ci are interchangeable. We propose to implement our understanding of lossless watermarks by introducing a mutual exclusion rule building on the identified lexical clusters: interchangeable tokens are mutually exclusive during partitioning. In other words, if a fraction of tokens A, representing a certain semantic or syntactic function, is assigned to the red list, then their remaining synonyms B should be placed on the green list, and vice versa.
We then detail the WatME encoding process, outlined in Alg. 1, which employs a two-step partitioning process to form a green and red list. The first partition occurs within the redundant lexical clusters C that we have identified within S, while the second takes place among the remaining part in the vocabulary denoted as V \ S. We use γ to determine the number of tokens from the mined clusters that are allocated to the green list G′t in the first partition. The remaining tokens, based on the principle of mutual exclusivity, are assigned to the red team R′t. The second partition continues to allocate words to the green list Gt from the remaining vocabulary until the combined size of the green teams from both steps reaches the predefined limit, γ. The rest of the process follows the steps outlined in the vanilla watermarking of Alg. 2.
The WatME detection algorithm is unchanged, except the green list calculation now uses Alg. 2.

heading: 3.4 Theoretical Analysis
text: We provide a mathematical analysis demonstrating how WatME outperforms the conventional method,
Algorithm 1 WatME Encoding Input: prompt x1 · · ·xm, green list size γ ∈ (0, 1), watermark strength δ > 0. for t = 0, 1, · · · , T − 1 do
1. Get the logit ℓt ∈ R|V| from M. 2. Use seed from the last token, split each cluster Ci
in parallel into green list G′it (of size |Ci|γ) and red list R′it (of size (1−γ)|Ci|) . Let G′t = ∪iG′it and R′t = ∪iR′it.
3. Partition the remaining part V \ S into a green list Gt of size γ|V | − |G′t| and a red list Rt of size (1− γ)|V | − |R′t|. 4. Merge lists from the previous two steps: Gt = Gt ∪G′t and Rt = Rt ∪R′t. 5. Add δ to the elements of logit ℓt corresponding to the green list, then softmax. p̂t = softmax(ℓt[i] + δ), i ∈ Gt 6. Sample the next token yt+1 from p̂t.
end for Output: watermarked text y1 · · · yT .
focusing on the ’green’ team’s expressiveness and the probability of high-quality sampling.
Definition 3.1 (Semantic Entropy) Let V represent the vocabulary of a language model. We define the semantic entropy of V , denoted by Hsem(V), as the entropy of the semantic distribution across V . This entropy quantifies the diversity and richness of meanings expressible by V . Consequently, a higher value of Hsem(V) signifies a vocabulary with greater semantic richness.
Definition 3.2 (Intrinsic Expressiveness) It is assumed that a language model M, with a vocabulary V characterized by high semantic entropy as indicated by Hsem(V), possesses an enhanced intrinsic expressive capacity. This capacity is unaffected by the output distribution of M and is due to the extensive semantic capabilities of V , which endow M with the potential for stronger expressive abilities.
Assumption 3.3 We consider practical scenarios that require high detectability, and thus a large value of δ. In such a strong watermarking scenario, tokens from the green list are more probable to be used than those from the red list.
Note: Assumption 3.3 establishes the foundational premise of text watermarking’s effectiveness.
Building upon the Definitions and Assumption, we derive the following theorem.
Theorem 3.4 Consider that pt ∈ R|V| represents the predicted distribution of the model M at decoding time t. Let wi denote the token with the ith
highest probability in pt. The higher the rank of a token (i.e., the smaller i is), the more suitable it is to be selected. Under the conditions of Assumption 3.3, the WatME watermarking method is more likely to select a suitable token compared to the vanilla watermarking method.
Theorem 3.5 Given a fixed proportion γ of the green team, the expressive power of a language model M employing the WatME exceeds that of one utilizing a vanilla watermarking approach.
These theorems highlight two advantages of WatME; their proofs are in the Appendix C.

heading: 4 Impact on Emergent Abilities
text: The majority of research on text watermarking utilizes the C4 dataset (Dodge et al., 2021) as a basis for testing perplexity (PPL). However, watermarking not only impacts the fluency of text generation but also holds the potential to influence LLMs on a broader scale, such as emergent abilities. These unique abilities intrinsic to LLMs garner significant interest from users and stimulate curiosity within the research community. However, they are often overlooked in the field of text watermarking.
Although a consensus definition is lacking, emergent abilities are typically characterized in many studies (Brown et al., 2020b; Wei et al., 2022; Yu et al., 2023) as a model’s capacity to perform specific tasks without training. In light of this, we propose to test and compare the performance of WatME and Vanilla watermark algorithms on different tasks using prompting technologies.
To comprehensively test the impact of watermarking on these abilities, we attempt to categorize it into different scenarios for a more exhaustive examination. Specifically, we draw upon Cattell’s cognitive theory (Cattell, 1963), which bifurcates intelligence into crystallized and fluid intelligence. Crystallized intelligence corresponds to the model’s utilization of learned knowledge and experience, while fluid intelligence involves logical thinking and solving problems. Correspondingly, we propose to examine crystallized intelligence through an assessment of the model’s knowledge capabilities, and fluid intelligence through its ability to reason and solve mathematical problems.
Knowledge Capability. To evaluate the model’s mastery of world knowledge, we employ TruthfulQA (Lin et al., 2022), a benchmark designed to
test if LLMs can generate truthful and informative answers. We select the generation setting.
Reasoning Capability. We employ the GSM8K dataset to assess the model’s chain-of-thought reasoning. Comprising 8K arithmetic and math problems, it provides a platform for evaluating reasoning performance. Aligned with the CoT Hub prompt (Fu et al., 2023), our evaluations include few-shot scenarios that prompt the model to demonstrate reasoning and generate thought chains.

heading: 5 Experiments

heading: 5.1 Experimental Setups
text: Evaluation Metrics To evaluate detection performance, following previous work, we use the Area Under the Receiver Operating Characteristic curve (AUROC), a well-established metric for binary classifiers. For mathematical reasoning tasks, we use Accuracy to assess the correctness of the model’s solutions. In our evaluation of the TruthfulQA dataset, following Lin et al. (2022), we use the trained GPT-Truth and GPT-Info scorers, assessing the model’s capacity to generate both truthful and informative responses. Given the potential tradeoff between these two perspectives, the product of Truth and Information (Truth.*Info.) is commonly used as an overall measure of performance. On the C4 dataset, we report Perplexity (PPL).
Baselines We compared our model with four established baselines. First, KGW-Mark (Vanilla watermarking) (Kirchenbauer et al., 2023), which categorizes teams into ’red’ and ’green’ to facilitate detection. Second, Gumbel-Mark (Kuditipudi et al., 2023), which employs a Gumbel-Softmax distribution to introduce stochasticity into the watermarking process. Third, Unbiased-Mark (Hu et al., 2024), which implements reweighting techniques to maintain the expected output distribution of the LLM during watermarking. Lastly, ProvableMark (Zhao et al., 2023), which uses a fixed hash key during watermarking to achieve provably better performance.
Models We utilized two distinct types of LLMs for experimentation: the non-aligned Llama2 model (Touvron et al., 2023), and the aligned Vicuna v1.5 model (Chiang et al., 2023). The majority of the results reported in this paper were obtained using the 7B version of the models.
Further setup details are in Appendix E.

heading: 5.2 Main Results
text: Greater Impact on Emergent Abilities than Fluency The experimental evidence suggests that watermarking notably hinders the emergent abilities of LLMs much more than fluency (see Table 1). Specifically, the non-aligned Llama2 model experienced a decline in performance exceeding 50% on both the GSM8K and TruthfulQA benchmarks. In contrast, the aligned model, Vicuna, demonstrated relative resilience but still incurred performance reductions greater than 20% on these benchmarks. This demonstrates the adverse impact of Vanilla Watermarking on the knowledge and reasoning capabilities of LLMs, with reasoning showing a marginally greater susceptibility.
Superiority of WatME over baselines in Preserving Emergent Abilities Across all models and benchmarks, the WatME consistently outperformed baseline watermarking methods. For the Llama2 model, WatME mitigated performance degradation by 16.8% on GSM8K and by 14.7% on TruthfulQA compared to the strongest baseline. Similarly, for the Vicuna model, the reductions were 13.4% and 14.0%, respectively. These outcomes underscore WatME’s significant effectiveness in preserving the emergent capabilities of LLMs without compromising performance as significantly as other methods.
Comparable Detection Performance of WatME Despite the trade-off between text quality and detection performance, WatME’s detection efficacy matched that of the Vanilla Watermark while also enhancing model capabilities, as evidenced
by similar AUROC scores—suggesting our algorithm attained a better equilibrium than the baseline. In contrast, the Gumbel-Mark method noticeably compromised detection performance, particularly in aligned models and when generating short responses (TruthfulQA). Additionally, more performance results under different watermark strengths are presented in Discussion 6.3. Distinct Advantages of WatME Variations It is evident that different WatME variations exhibit unique strengths; The ’dictionary’ variant outperformed in the Accuracy and Truthfulness scores, while the ’prompting’ variant excelled in the Informativeness. The integration of these variants may offer a fruitful avenue for future research. For a comprehensive understanding, a manual analysis of lexical clusters derived from these methods is presented in the Discussion 6.1. Alignment Diminishes Watermark Effectiveness Surprisingly, aligned models showed significantly greater resistance to watermarking effects than nonaligned models. Specifically, Vicuna 1.5’s performance dipped 30% less than Llama2’s across all benchmarks, coupled with a 10% lower watermark detection performance. To understand the underlying reasons for these differences, we analyzed the output distribution discrepancies between aligned and unaligned models in the Discussion 6.4.

heading: 6 Discussion

heading: 6.1 Analysis of Clustering Methods
text: To analyse redundant clusters from diverse methods, we set evaluation criteria to ensure analytical
rigour. These criteria spanned semantic consistency, contextual appropriateness, and grammatical consistency, which are essential aspects of cluster quality. Two annotators used a rating scale of 0, 1, 2 to annotate the clusters. A score of ’2’ indicated high or ideal consistency, ’1’ denoted moderate or usable consistency, and ’0’ identified low or unusable consistency within a cluster. The kappa value for the annotations is 0.657. Figure 2(a) shows both methods met usability, but fell short of ideal effectiveness. The dictionary approach was superior in semantic coherence due to its utilization of lexical databases. Conversely, the prompting method outperformed in contextual and grammatical consistency, reflecting the varied linguistic corpus training of LLMs. This suggests the potential benefits of a combined approach, a topic reserved for future research.

heading: 6.2 Robustness Against Attacks
text: In addition to affecting the performance of LLMs, watermarks are also vulnerable to attacks aimed at their removal. To evaluate the robustness of our method, we conducted tests against two prevalent types of attacks: substitution attacks and paraphrase attacks. For the substitution attack, we evaluated 200 examples from GSM8k, with various token replacement ratios. As shown in Figure 2(b), WatME consistently outperformed the baseline method in the robustness of detection across different levels of token replacement. For paraphrase attacks, we use a powerful paraphraser, llama-2-chat-13B, to extensively rewrite the watermarked text generated by llama-2-7b. We provided it with the prompt: "Please paraphrase the following text, altering the wording significantly yet preserving the original meaning and length." We then subjected our system to these rewritten samples using 200 entries from both GSM8k and TruthfulQA. The results are presented in Tables 2.
We offer two perspectives to understand the robustness of WatME: (1) Intuitively, for substitution attacks, the effect on watermarking depends on whether it triggers a token swap between the ’red’ and ’green’ teams: a swap affects the detection, while no swap means the watermark remains intact. With KGW-Mark, semantically similar tokens may be allocated to one team, resulting in a substitution invariably causing a swap. In contrast, WatME is intentionally designed to prevent this scenario. Therefore, the likelihood of a red-green swap—and consequently the impact on the watermark—is reduced in WatME compared to KGW. (2) From an encryption viewpoint, whereas KGWMark relies on a single division between teams, WatME employs multiple divisions—the number of clusters plus one (|C|+1), as outlined in Algorithm 2. Though these multiple partitions are computationally equivalent to a single partition due to efficient parallel matrix operations (explained in Appendix D), they introduce a higher level of complexity and robustness to the encryption process.

heading: 6.3 Performance Trade-offs at different Delta
text: The efficacy of the Watermark is influenced by the hyperparameter, Delta, which controls the watermark strength. An increase in Delta facilitates easier watermark detection but at the cost of severer impact on the LLMs. We analyse the TruthfulQA and GSM8K datasets. Figure 3 shows WatME consistently achieved a more favourable balance between watermark robustness and LLM performance across various Delta settings, surpassing Vanilla Watermark. Notably, the performance curves of WatME are strictly better than that of Vanilla, indicating that at equivalent watermark strengths, WatME always maintains superior performance compared to Vanilla Watermark.

heading: 6.4 Aligned vs Unaligned Models
text: Our examination of the response sensitivity to watermarking in aligned and unaligned models in-
volved analyzing their output distributions on the TruthfulQA and GSM8K datasets. We computed the average entropy for tokens in the generated answers and found that aligned models exhibit markedly lower entropy, suggesting more deterministic response patterns, as illustrated in Figure 4. This pronounced certainty in aligned models’ outputs presents a challenge for watermarking because of the limited variability that is essential for effective watermark encoding.

heading: 7 Conclusion
text: This study explores the impact of watermarking on the emergent abilities of LLMs—an aspect often neglected in the field. Our findings highlight the considerable adverse effects of traditional watermarking methods on LLMs’ emergent abilities, including knowledge recall and logical reasoning.
In response, we introduced WatME—a novel watermarking approach that leverages lexical redundancy. Theoretical analysis and comprehensive empirical results indicate WatME consistently preserves the expressive power of LLMs without compromising detection performance, enabling developers to encode watermarks with less disruption to user experience.
The advancements with WatME mark a stride
in lossless watermarking, enabling developers to encode watermarks with less disruption to user experience. We hope to promote a better equilibrium between regulatory compliance and user satisfaction in LLM development.
Limitations
In this section, we discuss the limitations of this work from two perspectives.
Firstly, although WatME represents a step toward lossless watermarking, it is not entirely lossfree. The introduction of a controlled bias, inherent to watermarking methods, subtly alters the generated data. This compromise is a critical consequence as it diverges from the ideal of a completely lossless system. This deviation poses a dilemma for developers weighing the benefits of watermarking against potential user experience and regulatory trade-offs. Future work aims to bridge this gap, enhancing the WatME method to maintain output integrity and broaden its appeal for practical implementation.
Secondly, while our method is designed to be language-agnostic, the empirical validation presented in this work is limited to models processing the English language. We acknowledge that the applicability of watermarking across various linguistic contexts is critically important. Future investigations will endeavour to broaden the scope to include more languages, ensuring the generalizability and effectiveness of our approach in a multilingual context.
Thirdly, the challenge of watermarking in lowentropy scenarios remains an open problem. Our dataset encompasses a range of scenarios, including low-entropy situations where outcomes are more predictable and our methodology remains effective. However, embedding watermarks in text with universally recognized, low-entropy answers poses significant challenges, highlighting the need for further investigation into constructing and testing methodologies for low-entropy corpora.
Lastly, our LLMs-based cluster generation approach is influenced by the robustness of the prompting methods. Different prompt constructions can lead to varying outcomes (Zhao et al., 2021; Chen et al., 2023b, 2024), represents a limitation that warrants further discussion and exploration in future work.
Despite these limitations, we believe our work serves as a significant catalyst for the field, con-
tributing positively to the advancement of more lossless and detectable text watermarking techniques.

heading: A Algorithms of Watermark
text: This section presents detailed algorithms for the watermark encoding and detection processes as outlined in (Kirchenbauer et al., 2023). Algorithm 2 delineates the procedure for encoding a watermark into the output sequence generated by a language model. Conversely, Algorithm 3 explicates the method for detecting and confirming the watermark’s presence within generated sequences.
Algorithm 2 Vanilla Watermark Encoding Input: prompt x1 · · ·xm,
green list size γ ∈ (0, 1), watermark strength δ > 0.
for t = 0, 1, · · · , T − 1 do 1. Get the logit ℓt ∈ R|V| from M. 2. Use the hash of the previous token as the
random seed to partition the vocabulary of M into a “green list” Gt of size γ|V|, and a “red list” Rt of size (1− γ)|V|.
3. Add δ to each green list logit and then apply softmax to the modified logits.
ℓ̂t[i] := ℓt[i] + δ, i ∈ Gt p̂t = softmax(ℓ̂t)
4. Sample a next token yt+1 from p̂t. end for Output: watermarked text y1 · · · yT .

heading: B Prompt for Cluster Mining
text: To facilitate the generation of synonym clusters, we employed Llama2-13B-chat. The approach involved crafting a prompt (Figure 5) that combines a clear task description with a set of demonstrations designed to illustrate the desired task. By presenting the model with a few-shot example, we primed Llama2-13B-chat to understand and perform the specific task of synonym generation. The few-shot prompt was crucial for the model to recognize the pattern and replicate it for new target
Algorithm 3 Vanilla Watermark Detection Input: text y, detection threshold τ . 1. Use the previous token to find the “green list” Gt at the step t as in Alg. 2. 2. Calculate the number of green tokens in y as |y|G = ∑n t=1 1(yt ∈ G).
3. Compute the z-statistic:
zy = (|y|G − γ|V|) / √ |V|γ(1− γ).
4. if zy > τ then return 1 (watermarked). 5. else return 0 (unwatermarked). Output: 0 or 1
words, thus enabling the mining of synonym clusters effectively.

heading: C Proofs of Theorems
text: In this section, we present the detailed proofs of the theorems introduced before. Each theorem is treated in its respective subsection.
C.1 Proof of Theorem 3.4 Proof We begin the proof by considering i = 1, 2. Case I: where w1 is in the green list (Gt):
If w1 ∈ Gt, then both watermarking methods are lossless because they can select the most suitable token w1. Case II: where w1 is in the red list (Rt):
We consider w2, which may or may not be a synonym of w1:
Sub-case i: w2 is not a synonym of w1. If w1 /∈ Gt and ̸ ∃ Ci ∈ C s.t. w1, w2 ∈ Ci, then
according to Algo. 1 we have:
PWatME(w2 ∈ Gt) = Pwatermark(w2 ∈ Gt).
In this case, the two methods are the same. Sub-case ii: w2 is a synonym of w1. If w1 /∈ Gt and ∃Ci ∈ C s.t. w1, w2 ∈ Ci, then according to Algo. 1 we have:
PWatME(w2 ∈ Gt) > Pwatermark(w2 ∈ Gt).
Based on Assumption 3.3, WatME is more likely to select the suitable token. Combining these cases, the theorem is proven. It should be noted that while this proof explicitly considers the cases for i = 1, 2, the logic extends to any arbitrary value of i.
C.2 Proof of Theorem 3.5 Proof Let us define the vocabulary V with synonym clusters S = {C1, . . . , Cn}, where C̄ represents the set of non-synonymous, unique words.
According to Algs 2 and 1, WatME maintains a constant number of distinct semantic representations, quantified as n+ γ · |C̄|. In contrast, the semantic token count of standard watermarking algorithms is lower than this figure. According to Definition 3.1 the disparity in semantic entropy between the two methodologies is thus evident. Given Definition 3.2, the increased semantic entropy inherent to WatME confirms the theorem.

heading: D Time Complexity Analysis
text: The conventional algorithm necessitates a partition of the vocabulary per decoding operation, which results in a time complexity of O(|V |). Our method incorporates two partitioning stages: initially targeting the redundant cluster, followed by the remaining vocabulary. During the first stage, we pad the cluster into a 2D matrix and conduct parallel sampling. The subsequent stage aligns with the procedures of the Vanilla algorithm. Consequently, the time complexity of our method remains at O(|V |).

heading: E Setup Details
text: In our experiments, we used prompts from the CoT hub (Fu et al., 2023) for the GSM8K dataset and the original prompts from TruthfulQA (Lin et al., 2022). The Llama2 model was evaluated using its original prompt format to maintain consistency. Greedy decoding was employed as the strategy for all tasks, with maximum decoding lengths set at 128 tokens for GSM8K and 50 tokens for TruthfulQA, which allowed for the complete generation of answers within the datasets.
To account for the differing answer lengths in the GSM8K and TruthfulQA datasets, we fine-tuned the watermark hyperparameters. For GSM8K, with its longer answers aiding detection, we used a milder watermark intensity, setting gamma at 0.3 and delta at 3.0. Conversely, the brevity of answers in TruthfulQA complicates detection, necessitating a stronger watermark intensity—again with gamma at 0.3, but with delta increased to 4.0 to achieve satisfactory detection performance (AUROC > 0.7).
Evaluation metrics were carefully chosen: AUROC was calculated using the ‘sklearn‘ library, and for the assessment of GPT-Truth and GPT-Info, we utilized a fine-tuned Llama2-13B-chat model that demonstrated an accuracy above 93% on the validation set. All model implementations were executed using the ‘transformers‘ library.
The hardware employed for these experiments
consisted of a 40GB A100 GPU and a 32GB V100 GPU, ensuring sufficient computational power for model training and evaluation.

heading: F Examples of Redundant Clusters
text: We present some examples of mined clusters at 3.

heading: G Multilingual Performance Testing
text: We expand our evaluation to include the Chinese Long Text Summarization Dataset (CLTS) and a bilingual Large Language Model (LLM), ChatGLM3-6b. This model employs Byte Pair Encoding (BPE) tokenization with a vocabulary size of 65k—double that of the Llama 2 model which has a 32k vocabulary size. Synonym mining, a critical step in our process, was conducted using the ChatGLM3-13B model. The performance of different watermarking methods was evaluated using the ROUGE-L and AUROC metrics, as shown in Table 4. The results highlight that watermarking with WatME considerably enhances detection robustness compared to the baseline method, maintaining effectiveness despite varying levels of token replacement. This improvement underscores WatME’s capability to integrate seamlessly without compromising the natural language generation quality.

heading: Future Work
text: Future work aims to bridge this gap, enhancing the WatME method to maintain output integrity and broaden its appeal for practical implementation. Secondly, while our method is designed to be language-agnostic, the empirical validation presented in this work is limited to models processing the English language. We acknowledge that the applicability of watermarking across various linguistic contexts is critically important. Future investigations will endeavour to broaden the scope to include more languages, ensuring the generalizability and effectiveness of our approach in a multilingual context. Thirdly, the challenge of watermarking in lowentropy scenarios remains an open problem. Our dataset encompasses a range of scenarios, including low-entropy situations where outcomes are more predictable and our methodology remains effective. However, embedding watermarks in text with universally recognized, low-entropy answers poses significant challenges, highlighting the need for further investigation into constructing and testing methodologies for low-entropy corpora. Lastly, our LLMs-based cluster generation approach is influenced by the robustness of the prompting methods. Different prompt constructions can lead to varying outcomes (Zhao et al. , 2021; Chen et al