Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models (2024)

Abstract: Text watermarking technology aims to tag and identify content produced by large language models (LLMs) to prevent misuse. In this study, we introduce the concept of “cross-lingual consistency” in text watermarking, which assesses the ability of text watermarks to maintain their effectiveness after being translated into other languages. Preliminary empirical results from two LLMs and three watermarking methods reveal that current text watermarking technologies lack consistency when texts are translated into various languages. Based on this observation, we propose a Cross-lingual Watermark Removal Attack (CWRA) to bypass watermarking by first obtaining a response from an LLM in a pivot language, which is then translated into the target language. CWRA can effectively remove watermarks, decreasing the AUCs to a random-guessing level without performance loss. Furthermore, we analyze two key factors that contribute to the cross-lingual consistency in text watermarking and propose X-SIR as a defense method against CWRA.

heading: 1 Introduction
text: Large language models (LLMs) like GPT-4 (OpenAI, 2023) have demonstrated remarkable content generation capabilities, producing texts that are hard to distinguish from human-written ones. This progress has led to concerns regarding the misuse of LLMs, such as the risks of generating misleading information, impersonating individuals, and compromising academic integrity (Chen and Shu, 2023; Ai et al., 2024; Yuan et al., 2024; Xia et al., 2024). As a countermeasure, text watermarking technology for LLMs has been developed, aiming at tagging and identifying the content produced by LLMs (Kirchenbauer et al., 2023a; Liu et al., 2023). Generally, a text watermarking algorithm
*Equal Contribution. Work done during Zhiwei’s internship at Tencent AI Lab.
†Rui Wang and Xing Wang are co-corresponding authors.
embeds a message within LLM-generated content that is imperceptible to human readers, but can be detected algorithmically. By tracking and detecting text watermarks, it becomes possible to mitigate the abuse of LLMs by tracing the origin of texts and ascertaining their authenticity.
The robustness of watermarking algorithms, i.e., the ability to detect watermarked text even after it has been modified, is important. Recent works have shown strong robustness under text rewriting and copy-paste attacks (Liu et al., 2024b; Yang et al., 2023b). However, these watermarking techniques have been tested solely within monolingual contexts. In practical scenarios, watermarked texts might be translated (He et al., 2024a,b, 2022; Jiao et al., 2023; Liang et al., 2023), raising questions about the efficacy of text watermarks across languages (see Figure 1). For example, a malicious user could use a watermarked LLM to produce fake news in English and then translate it into Chinese. Obviously, the deceptive impact persists regardless of the language, but it is uncertain whether the watermark would still be detectable after such a
4115
translation. To explore this question, we introduce the concept of cross-lingual consistency in text watermarking, aiming to characterize the ability of text watermarks to preserve their strength across languages. Our preliminary results on 2 LLMs × 3 watermarks reveal that current text watermarking technologies lack consistency across languages.
In light of this finding, we propose the Crosslingual Watermark Removal Attack (CWRA) to highlight the practical implications arising from deficient cross-lingual consistency. When performing CWRA, the attacker begins by translating the original language prompt into a pivot language, which is fed to the LLM to generate a response in the pivot language. Finally, the response is translated back into the original language. In this way, the attacker obtains the response in the original language and bypasses the watermark with the second translation step. CWRA outperforms re-writing attacks, such as re-translation and paraphrasing (Liu et al., 2023), as it decreases the AUCs to a randomguessing level and achieves the highest text quality.
To resist CWRA, we propose a defense method that improves the cross-lingual consistency of current LLM watermarking. Our method is based on two critical factors. The first is the cross-lingual semantic clustering of the vocabulary. Instead of treating each token in the vocabulary as the smallest unit when ironing watermarks, as done by KGW (Kirchenbauer et al., 2023a), our method considers a cluster of tokens that share the same semantics across different languages as the smallest unit of processing. In this way, the post-translated token will still carry the watermark as it would fall in the same cluster as before translation. The second is cross-lingual semantic robust vocabulary partition. Inspired by Liu et al. (2024b), we ensure that the partition of the vocabulary are similar for semantically similar contexts in different languages. Despite its limitations, our approach (named X-SIR) substantially elevates the AUCs under the CWRA, paving the way for future research.
Our contributions are summarized as follows: • Evaluation (§ 3): We reveal the deficiency of
current text watermarking technologies in maintaining cross-lingual consistency. • Attack (§ 4): Based on this finding, we propose CWRA that successfully bypasses watermarks without degrading the text quality. • Defense (§ 5): We identify two key factors for improving cross-lingual consistency and propose X-SIR as a defense method against CWRA.

heading: 2 Background

heading: 2.1 Language Model
text: A language model (LM) M has a defined set of tokens known as its vocabulary V . Given a sequence of tokens x1:n = (x1, x2, . . . , xn), which we refer to as the prompt, the model M computes the conditional probability of the next token over V as PM (xn+1|x1:n). Therefore, text generation can be achieved through an autoregressive decoding process, where M sequentially predicts one token at a time, forming a response. Such an LM can be parameterized by a neural network, such as Transformer (Vaswani et al., 2017), which is called neural LM. Typically, a neural LM computes a vector of logits zn+1 = M(x1:n) ∈ R|V| for the next token based on the current sequence x1:n via a neural network. The probability of the next token is then obtained by applying the softmax function to these logits: PM (xn+1|x1:n) = softmax(zn+1).

heading: 2.2 Watermarking for LMs
text: In this work, we consider the following watermarking methods. All of them embed the watermark by modifying logits during text generation and detect the presence of the watermark for any given text.
KGW (Kirchenbauer et al., 2023a) sets the groundwork for LM watermarking. Ironing a watermark is delineated as the following steps: (1) compute a hash of x1:n: hn+1 = H(x1:n), (2) seed a random number generator with hn+1
and randomly partitions V into two disjoint lists: the green list Vg and the red list Vr, (3) adjust the logits zn+1 by adding a constant bias δ (δ > 0) for tokens in the green list:
∀i ∈ {1, 2, . . . , |V|},
z̃n+1i = { zn+1i + δ, if vi ∈ Vg, zn+1i , if vi ∈ Vr. (1)
As a result, watermarked text will statistically contain more green tokens, an attribute unlikely to occur in human-written text. When detecting, one can apply step (1) and (2), and calculate the z-score as the watermark strength of x:
s = (|x|g − γ|x|)/ √ |x|γ(1− γ), (2)
where |x|g is the number of green tokens in x and γ =
|Vg | |V| . The presence of the watermark can be
determined by comparing s with a threshold. Unbiased watermark (UW) views the process of adjusting the logits as applying a ∆ function:
z̃n+1 = zn+1 +∆, and designs a ∆ function that satisfies:
E [ P̃M ] = PM , (3)
where P̃M is the probability distribution of the next token after logits adjustment (Hu et al., 2023).
Semantic invariant robust watermark (SIR) shows the robustness under re-translation and paraphrasing attack (Liu et al., 2024b). Its core idea is to assign similar ∆ for semantically similar prefixes. Given prefix sequences x and y, SIR adopts an embedding model E to characterize their semantic similarity and trains a watermark model that yields ∆ with the main objective:
L = |Sim(E(x), E(y))− Sim(∆(x),∆(y))|, (4) where Sim(·, ·) denotes similarity function. Furthermore, ∀i ∈ {1, 2, . . . , |V|}, ∆i is trained to be close to +1 or −1. Therefore, SIR can be seen as an improvement based on KGW, where ∆i > 0 indicating that vi is a green token. The original implementation of SIR uses C-BERT (Chanchani and Huang, 2023) as the embedding model, which is English-only. To adopt SIR in the cross-lingual scenario, we use a multilingual S-BERT (Reimers and Gurevych, 2019)1instead.

heading: 3 Cross-lingual Consistency of Text Watermark
text: In this section, we define the concept of crosslingual consistency in text watermarking and answer three research questions (RQ): • RQ1: To what extent are current watermarking
algorithms consistent across different languages?
• RQ2: Do watermarks exhibit better consistency between similar languages than between distant languages?
• RQ3: Does semantic invariant watermark (SIR) exhibit better cross-lingual consistency than others (KGW and UW)?

heading: 3.1 Definition
text: We define cross-lingual consistency as the ability of a watermark, embedded in a text produced by an LLM, to retain its strength after the text is translated into another language. We represent the original strength of the watermark as a random variable, denoted by S (Appendix A), and its strength after translation as Ŝ. To quantitatively assess this consistency, we employ the following two metrics.
1paraphrase-multilingual-mpnet-base-v2
Pearson Correlation Coefficient (PCC) We use PCC to assess linear correlation between S and Ŝ:
PCC(S, Ŝ) = cov(S, Ŝ) σSσŜ , (5)
where cov(S, Ŝ) is the covariance and σS and σŜ are the standard deviations. A PCC value close to 1 suggests consistent trends in watermark strengths across languages.
Relative Error (RE) Unlike PCC, which captures consistency in trends, RE is used to assess the magnitude of deviation between S and Ŝ:
RE(S, Ŝ) = E [ |Ŝ − S| |S| ] × 100%. (6)
A lower RE indicates that the watermark retains strength close to its original value after translation, signifying great cross-lingual consistency.

heading: 3.2 Experimental Setup
text: Setup We sampled a subset of 500 English prompts from the mc4 dataset (Raffel et al., 2019)2, and generated responses from the LLM using the text watermarking methods described in § 2.2. The default decoding method was multinomial sampling, and both the prompts and the LLMgenerated responses were in English. To evaluate the cross-lingual consistency, these watermarked responses were translated into four languages using gpt-3.5-turbo-06133: Chinese (Zh), Japanese (Ja), French (Fr), and German (De). Notably, English shares greater similarities with French and German, in contrast to its significant differences from Chinese and Japanese.
Models For the LLMs, we adopt:
• BAICHUAN-7B (Baichuan., 2023): an LLM trained on 1.2 trillion tokens. It offers bilingual support for both Chinese and English officially. We also found that its vocabulary covers Japanese tokens.
• LLAMA-2-7B (Touvron et al., 2023): trained on 2 trillion tokens and only provides support for English officially. Its vocabulary also contains tokens for European languages, such as German and French.
2https://huggingface.co/datasets/mc4 3https://platform.openai.com/docs/models

heading: 3.3 Results
text: Table 1 presents the main results. We forced the model to generate 200 tokens in response following the setting of Kirchenbauer et al. (2023a). With response length restriction lifted, Figure 2 illustrates the trend of watermark strengths with text length.
Results for RQ1 We reveal a notable deficiency in the cross-lingual consistency of current watermarking methods. Among all the settings, the PCCs are generally less than 0.3, and the REs are predominantly above 80%. Furthermore, Figure 2 visually demonstrates that the watermark strengths of the three methods exhibit a significant decrease after translation. These results suggest that current watermarking algorithms struggle to maintain effectiveness across language translations.
Results for RQ2 None of the three watermarking methods exhibits such a characteristic that its crosslingual consistency between similar languages is significantly better than distant ones. This means that even if two languages have similar structures or shared words, it is still difficult for watermarks to transfer between them, which poses a big challenge to the cross-lingual consistency of watermarking.
Results for RQ3 Overall, SIR indeed exhibits superior cross-lingual consistency compared to KGW and UW. It achieves the best average results across the two models and two metrics. When using BAICHUAN-7B, SIR notably outperforms other methods in terms of PCCs for all target languages. This finding highlights the importance of semantic invariance in preserving watermark strength across languages, which we will explore more in § 5. Despite its superiority, SIR still presents a notable reduction in watermark strength in cross-lingual scenarios, as evidenced by Figure 2c.

heading: 4 Cross-lingual Watermark Removal Attack
text: In the previous section, we focus on scenarios where the response of LLM is translated into other languages. However, an attacker typically expects a response from the LLM in the same language as the prompt while removing watermarks. To bridge this gap, we introduce the Cross-lingual Watermark Removal Attack (CWRA) in this section, constituting a complete attack process and posing a more significant challenge to text watermarking than paraphrasing and re-translation attacks.
Figure 3 shows the process of CWRA. Instead of feeding the original prompt into the LLM, the attacker initiates the attack by translating the prompt into a pivot language named the pivot prompt. The LLM receives the pivot prompt and provides a watermarked response in the pivot language. The attacker then translates the pivot response back into the original language. This approach allows the attacker to obtain the response in the original language. Due to the inherent challenges in maintaining cross-lingual consistency, the watermark would be effectively eliminated during the second translation step.

heading: 4.1 Setup
text: To assess the practicality of attack methods, we consider two downstream tasks: text summarization and question answering. We adopt MultiNews (Fabbri et al., 2019) and ELI5 (Fan et al., 2019) as test sets, respectively. Both datasets are in English and require long text output with an average output length of 198 tokens. We selected 500 samples for each test set that do not exceed the maximum context length of the model and performed zero-shot prompting on BAICHUAN-7B. For CWRA, we select Chinese as the pivot language and compare the following two methods: • Paraphrase: rephrasing the response into differ-
ent wording while retaining the same meaning.
• Re-translation: translating the response into the pivot language and back to the original language. The paraphraser and translator used in all attack methods are gpt-3.5-turbo-0613 to ensure consistency across the different attack methods.

heading: 4.2 Results
text: Figure 4 exhibits ROC curves of three watermarking methods under different attack methods.
CWRA vs Other Attack Methods CWRA demonstrates the most effective attack performance, significantly diminishing the AUCs and the TPRs. For one thing, existing watermarking techniques are not designed for cross-lingual contexts, leading to weak cross-lingual consistency. For another thing, strategies such as Re-translation and Paraphrase are essentially semantic-preserving text rewriting. Such strategies tend to preserve some n-grams from the original response, which may still be identifiable by the watermark detection algorithm. In contrast, CWRA reduces such n-grams due to language switching.
SIR vs Other Watermarking Methods Under the CWRA, SIR exhibits superior robustness compared to other watermarking methods. The AUCs for KGW and UW under CWRA plummet to 0.61 and 0.54, respectively, approaching the level of random guessing. In stark contrast, the AUC for the SIR method stands significantly higher at 0.67, aligning with our earlier observations regarding cross-lingual consistency in the RQ3 of § 3.3.
Text Quality As shown in Table 2, these attack methods not only preserve text quality, but also bring slight improvements in most cases due to the good translator and paraphraser. Among the compared methods, CWRA stands out for its superior performance. Considering that the same translator and paraphraser were used across all methods, we speculate that this is because the BAICHUAN7B model used in our experiments performs even better in the pivot language (Chinese) than in the original language (English). This finding implies that a potential attacker could strategically choose a pivot language at which the LLM excels to perform CWRA, thereby achieving the best text quality while removing the watermark.

heading: 5 Improving Cross-lingual Consistency
text: Up to this point, we have observed the challenges associated with text watermarking in cross-lingual scenarios. In this section, we first analyze two key factors essential for achieving cross-lingual consistency. Based on our analysis, we propose a defense method against CWRA.

heading: 5.1 Two Key Factors of Cross-lingual Consistency
text: KGW-based watermarking methods fundamentally depend on the partition of the vocabulary, i.e., the red and green lists, as discussed in § 2.2. Therefore, cross-lingual consistency aims to achieve the following goal:
The green tokens in the watermarked text will still be recognized as green tokens after being translated into other languages.
With this goal in mind, we start our analysis with a toy case in Figure 5: 1. We define two simple languages. Language 1
(Lang1) consists of hollow tokens: □, ⋆ and ♥. Language 2 (Lang2) consists of solid tokens: □, ⋆ and ♥. Tokens with the same shape are semantically equivalent. 2. Given □ as the prefix, a watermarked LM selects ⋆ from [⋆, ⋆, ♥, ♥]4 as the next token. Due to watermarking, ⋆ is a green token. 3. A machine translator (MT) then translates the entire sentence “□ ⋆” into Lang2: “□ ⋆”.
The question of interest is: what conditions must the vocabulary partition satisfy so that the token ⋆, the semantic equivalent of ⋆, is also included in the green list?
Figure 5(a) illustrates a successful case, where two key factors exists: 1. Cross-lingual semantic clustering of the vo-
cabulary: semantically equivalent tokens must be in the same partition, either green or red lists. 2. Cross-lingual semantic robust vocabulary partition: for semantically equivalent prefixes in different languages: □ and □, the partitions
4We suppose that this LM is bilingual and omit the prefix tokens □ and □ for simplicity.
of the vocabulary must be the same. Both Figure 5(b) and Figure 5(c) satisfy only one of the two factors, thus failing to recognize ⋆ as a green token and losing cross-lingual consistency.

heading: 5.2 Defense Method against CWRA
text: We now improve the SIR so that it satisfies the two factors described above. In Figure 5, the tokens of the two languages are equivalent in one-to-one correspondence. In practice, we relax this semantic equivalence into semantic similarity.
As discussed in § 2.2, SIR uses the ∆ function to represent vocabulary partition (∆ ∈ R|V|), where ∆i > 0 indicating that vi is a green token. Based on Eq. 4, SIR has already optimized for Factor 2 when using a multilingual embedding model. For prefixes x and y, the similarity of their vocabulary partitions for the next token should be close to their semantic similarity:
Sim(∆(x),∆(y)) ≈ Sim(E(x), E(y)), (7)
where E is a multilingual embedding model. Based on SIR, we focus on Factor 1, i.e., crosslingual semantic clustering of the vocabulary. Formally, we define semantic clustering as a partition C of the vocabulary V: C = {C1, C2, . . . , C|C|} , where each cluster Ck consists of semantically similar tokens. Instead of assigning biases for each token in V , we adapt the ∆ function so that it yields biases to each cluster in C, i.e., ∆ ∈ R|C|. Thus,
the process of adjusting the logits should be:
∀i ∈ {1, 2, . . . , |V|}, z̃n+1i = z n+1 i +∆C(i),
(8)
where C(i) indicates the index of vi’s cluster within C. By doing so, if token vi and vj are semantically similar, they will receive the same bias on logits:
C(i) = C(j) =⇒ ∆C(i) = ∆C(j). (9)
In other words, if vi and vj are translations of each other, they will fall into the same list. Algorithm 1 shows the procedure of semantic clustering. To obtain such a semantic clustering C, we treat each token in V as a node, and add an edge (vi, vj) whenever (vi, vj) corresponds to an entry in a bilingual dictionary D. Therefore, C is all the connected components of this graph.
Algorithm 1 Constructing semantic clusters Require: the vocabulary of the LM V , a bilingual
dictionary D Ensure: Semantic clusters C
1: Initialize a graph G = (V,∅) 2: for each entry (vi, vj) in D do 3: if vi ∈ V and vj ∈ V then 4: Add an edge (vi, vj) to G 5: C = ∅ 6: for each connected component Ck in G do 7: C = C ∪ {Ck} 8: return C
Setup We name this method as X-SIR, implemented it with an unified external dictionary D that covers English (En), Chinese (Zh), Japanese (Ja), French (Fr), and German (De). Following the settings of § 3.2 and § 4.1, we apply X-SIR on BAICHUAN-7B, analyze it in the following and detail its limitations in § 8. We also present more results on other LLMs in Appendix C.
Cross-lingual consistency & ROC curves Figure 6 shows the cross-lingual consistency of SIR and X-SIR when the original responses are translated into other languages. Considering both PCC and RE, X-SIR notably improves the cross-lingual consistency over SIR when the target language is Zh or Ja. However, for De and Fr, X-SIR improves only the PCC but not RE, which we regard as a limitation and discuss in § 8. Further, Figure 7
evaluates both methods’ watermark detection performance under cross-lingual scenarios. Whether directly translating the responses into other languages (Figure 7a) or using CWRA (Figure 7b), X-SIR substantially enhances the watermark detection performance, with an average increase in TPR by 0.448 and AUC by 0.221. These findings validate the two key factors of cross-lingual consistency that we identified in § 5.1.
No attack & Paraphrase attack The next question is whether the addition of semantic clustering
of the vocabulary to X-SIR will affect the original detection performance of SIR. As shown in Figure 8, when there is no attack, both methods perform comparably. However, under the paraphrase attack, X-SIR performs slightly better than SIR. This is reasonable because paraphrasing can be regarded as a special kind of “translation” within the same language and X-SIR improves such intralingual consistency.
Text quality As shown in Table 3, X-SIR achieves better text quality than SIR in text summarization, and comparable performance on question answering, meaning the semantic clustering of vocabulary will not negatively affect text quality.

heading: 6 Related Work

heading: 6.1 LLM Watermarking
text: Text watermarking aims to embed a watermark into a text and detect the watermark for any given text. Currently, text watermark method can be classified into two categories (Liu et al., 2023): watermarking for existing text and watermarking for generated text. In this work, we focus on the latter, which is more challenging and has more practical applications.
This type of watermark method usually can be illustrated as the watermark ironing process (modifying the logits of the LLM during text generation) and watermark detection process (assess the presence of watermark by a calculated watermark strength score). Kirchenbauer et al. (2023a) introduces KGW, the first watermarking method for LLMs. Hu et al. (2023) proposes UW without affecting the output probability distribution compared to KGW. Liu et al. (2024b) introduces SIR, a watermarking method taking into account the semantic information of the text, which shows robustness to text re-writing attacks. Liu et al. (2024a) proposes the first unforgeable and publicly verifiable watermarking algorithm for LLMs. SemStamp (Hou et al., 2023) is another semanticrelated watermarking method and it generate watermarked text at sentence granularity instead of token granularity. Tu et al. (2023) introduces WaterBench, the first comprehensive benchmark for LLM watermarks.

heading: 6.2 Watermark Robustness
text: A good watermarking method should be robust to various watermarking removal attacks. However, current works on watermarking robustness mainly focus on single-language attacks, such as paraphrase attacks. For example, Kirchenbauer et al. (2023b) evaluates the robustness of KGW against paraphrase attacks as well as copy-paste attacks and proposes a detect trick to improve the robustness to copy-paste attacks. Zhao et al. (2023) employs a fixed green list to improve the robustness of KGW against paraphrase attacks and editing attacks. Chen et al. (2023) proposes a new paraphrase robust watermarking method “XMark” based on “text redundancy” of text watermark. Lu et al. (2024) proposes an entropy-based text watermarking detection method that achieves better detection performance in low-entropy scenarios.

heading: 7 Conclusion
text: This work aims to investigate the cross-lingual consistency of watermarking methods for LLMs. We first characterize and evaluate the cross-lingual consistency of current watermarking techniques, revealing that current watermarking methods struggle to maintain their watermark strengths across different languages. Based on this observation, we propose the cross-lingual watermark removal attack (CWRA), which significantly challenges wa-
termark robustness by efficiently eliminating watermarks without compromising text quality. Through the analysis of two primary factors that influence cross-lingual consistency, we propose X-SIR as a defense strategy against CWRA. Despite its limitations, this approach greatly improves watermark detection performance under cross-lingual scenario and paves the way for future research. Overall, this work completes a closed loop in the study of cross-lingual consistency in watermarking, including: evaluation, attacking, analysis, and defensing.
Since our first release, the X-SIR algorithm has been integrated into MarkLLM (Pan et al., 2024), an open-source toolkit for LLM Watermarking.

heading: 8 Limitations
text: X-SIR relies on the semantic clustering described in Algorithm 1 which only considers tokens shared by the vocabulary V of the model and the external dictionary D. This design results in the following limitations: • Language coverage: X-SIR only supports lan-
guages supported by the model. However, in a real scenario, an attacker can choose the original and the pivot language at will. This limitation has revealed by Figure 6, where X-SIR brings a much more significant boost to Ja & Zh than to De & Fr since the vocabulary of the LLM (BAICHUAN-7B) supports Zh & Ja better.
• Vocab coverage: Since the external dictionary D only contains whole words, word units can not be clustered in Algorithm 1 and will left as isolated nodes. Consequently, X-SIR’s performance might be compromised if the tokenizer favors finer-grained token segmentation.
X-SIR does not solve the issue of cross-lingual consistency but sets the stage for future research.

heading: Acknowledgements
text: This paper is mainly supported by the Tencent AI Lab Fund (RBFR2024002). Zhiwei and Rui are partially supported by the National Natural Science Foundation of China (62176153) and the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102, as the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University). Zhuosheng is partially supported by the Joint Funds of the National Natural Science Foundation of China (Grant No. U21B2020).

heading: A Details of Watermarking Methods
text: As discussed in § 6.1, we focus on the watermarking methods for large language models (LLMs). A watermarking method can be divided into two processes: the watermark ironing process and the watermark detection process. In ironing process, the watermark is embedded into the text by modifying the logits of the LLM during text generation. In detection process, the watermark detector calculates the watermark strength score s to assess the presence of watermark. s is a scalar value to indicate the strength of the watermark in the text. For any given text, we can calculate its watermark strength score s based on detection process of the watermarking method. A higher s indicates that the text is more likely to contain watermark. In the opposite, a lower s indicates that the text is less likely to contain watermark. Every watermarking method has its own way to ironing the watermark and calculate the watermark strength score s. We detail KGW, UW and SIR in the following sections.
A.1 KGW
In § 2.2, we introduce the processes of watermark ironing and watermark detection in the KGW method. Here, we detail the experimental settings employed for KGW. KGW uses a hash function H to compute the hash of the previous k tokens. In this work, we adhere to the experimental setting reported by Kirchenbauer et al. (2023a), employing the hash function minhash with k = 4. The ratio of green token lists Vg to the total word list V is set at γ = 0.25. Additionally, the constant bias δ is fixed at 2.0.
A.2 UW
The ironing process in UW is analogous to that in KGW; however, the two differ in their respective function of modifying the logits. Here is the detail watermark ironing process in UW: (1) compute a hash of x1:n: hn+1 = H(x1:n),
and use hn+1 as seed generating a random number p ∈ [0, 1).
(2) determine the token t satisfies:
p ∈ [ t−1∑
i=1
PMi(x n+1|x1:n),
t∑
i=1
PMi(x n+1|x1:n) ) (10)
(3) set PMi(xn+1|x1:n) = 0 for i ̸= t and PMt(x
n+1|x1:n) = 1. Then we get the adjusted probability of next token P̃M (x
n+1|x1:n). The detection process calculates a maximin variant Log Likelihood Ratio (LLR) of the detected text to assess the watermark strength score. Log Likelihood Ratio (LLR) is defined as:
ri = P̃M (x i|x1:i−1) PM (xi|x1:i−1)
(11)
The total score is defined as:
R = P̃M (x a+1:n|x1:a) PM (x a+1:n|x1:a) (12)
Where x1:a is prompt and xa+1:n is the detected text. Let
Pi = PM (x i|x1:i−1) (13) Qi = P̃M (x i|x1:i−1) (14)
Ri = (ri(x1), ri(x2), · · · , ri(x|V|)) (15) Where ri(xk) is the LLR of token xk at position i. UW use a maximin variant LLR (Hu et al., 2023) to avoid the limitation of the origin LLR. The calculating process of maximin variant LLR can be formulated as follows:
max Ri min Q
′ i∈∆V ,TV (Q ′ i,Qi)≤d
〈 Q′i, Ri 〉 ,
s.t. ⟨Pi, exp (Ri)⟩ ≤ 1 (16) Where ∆V is the set of all probability distributions over the symbol set V , and TV is the total variation distance, d is a hyperparameter to control TV , and ⟨·, ·⟩ is the inner product. UW utilizes the maximin variant LLR to calculate the watermark strength score.
In the experiments, we follow the experiment settings of original paper, using the previous 5 tokens to compute the hash and set d = 0.
A.3 SIR As introduced in § 2.2, the ironing process in SIR assigns a watermark bias, ∆i, to every token vi.
For any given text, the watermark detector calculates the mean watermark bias to determine the watermark strength score. Consider the detected text represented as x = ( x1, . . . , xN ) . The watermark strength score, s, can be expressed by the following equation:
s =
∑N n=1∆I(xn)(x 1:n−1)
N , (17)
where I(xn) indicates the index of the token xn within the vocabulary V , and ∆I(xn)(x1:n−1) denotes the watermark bias for token xn at position n. Given that the watermark bias satisfies the unbiased property, ∑ t∈V ∆I(t) = 0, the expected detection score for normal text is 0. Consequently, the detection score for watermarked text should exceed 0 significantly.

heading: B Full Result of Watermark Removal Attack
text: Figure 9 presents the full results of watermark removal attacks. It is equivalent to Figure 4, but presents results on text summarization and question answering separately.

heading: C Full Result of X-SIR
text: Figure 10 presents full watermark detection performance (ROC curves, AUCs and TPRs) of X-SIR. We tested it on the following LLMs:
• BAICHUAN-7B (Baichuan., 2023)5
• BAICHUAN2-7B-BASE (Yang et al., 2023a)6
• LLAMA-2-7B (Touvron et al., 2023)7
• MISTRAL-7B-V0.1 (Jiang et al., 2023)8.
We tested it under the following attack methods:
• No attack
• Paraphrase attack (§ 4.1)
• Direct translation (En→XX): translating original English responses to other languages.
We also plot the performance of SIR as baselines.
5https://huggingface.co/baichuan-inc/ Baichuan-7B
6https://huggingface.co/baichuan-inc/ Baichuan2-7B-Base
7https://huggingface.co/meta-llama/ Llama-2-7b-hf
8https://huggingface.co/mistralai/ Mistral-7B-v0.1
(a1) No attack (a2) Paraphrase attack (a3) En→De (a4) En→Fr (a5) En→Ja (a6) En→Zh
Tr ue
P os
iti ve
R at
e (T
PR )
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
0.2 0.4 0.6 0.8 1.0
X-SIR SIR
TPR 0.994 0.980
AUC 0.994 0.993
Tr ue
P os
iti ve
R at
e (T
PR )
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
0.2 0.4 0.6 0.8 1.0
TPR 0.702 0.164
AUC 0.905 0.614
Tr ue
P os
iti ve
R at
e (T
PR )
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
0.2 0.4 0.6 0.8 1.0
TPR 0.662 0.528
AUC 0.886 0.827
Tr ue
P os
iti ve
R at
e (T
PR )
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
0.2 0.4 0.6 0.8 1.0
TPR 0.318 0.112
AUC 0.769 0.318
Tr ue
P os
iti ve
R at
e (T
PR )
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
0.2 0.4 0.6 0.8 1.0
TPR 0.354 0.294
AUC 0.810 0.780
Tr ue
P os
iti ve
R at
e (T
PR )
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
0.2 0.4 0.6 0.8 1.0
TPR 0.696 0.216
AUC 0.911 0.733
(a) BAICHUAN-7B

heading: Future Work
text: X-SIR does not solve the issue of cross-lingual consistency but sets the stage for future research