Bypassing LLM Watermarks with Color-Aware Substitutions (2024)

Abstract: Watermarking approaches are proposed to identify if text being circulated is humanor large language model(LLM) generated. The stateof-the-art watermarking strategy of Kirchenbauer et al. (2023a) biases the LLM to generate specific (“green”) tokens. However, determining the robustness of this watermarking method under finite (low) edit budgets is an open problem. Additionally, existing attack methods fail to evade detection for longer text segments. We overcome these limitations, and propose Self Color Testing-based Substitution (SCTS), the first “color-aware” attack. SCTS obtains color information by strategically prompting the watermarked LLM and comparing output tokens frequencies. It uses this information to determine token colors, and substitutes green tokens with non-green ones. In our experiments, SCTS successfully evades watermark detection using fewer number of edits than related work. Additionally, we show both theoretically and empirically that SCTS can remove the watermark for arbitrarily long watermarked text.

heading: 1 Introduction
text: Large language models (LLMs) are pervasively used in applications related to education (Hadi et al., 2023) and programming (Fan et al., 2023). As with most generative AI technologies, however, they have dual use—for generating misinformation (Chen and Shu, 2023) and facilitating academic dishonesty (Lancaster, 2021). Detecting LLM-generated content is of immense interest; paraphrasing Yang et al. (2023), “detection techniques have witnessed advancements propelled by innovations in zero-shot methods, fine-tuning” etc. resulting in various methods being proposed.
However, most detection techniques are susceptible to a large number of false positives (Aaronson, 2022). Erroneous detection is further exacerbated when LLMs’ output more closely mirrors human content (Zhang et al., 2023; Ghosal et al., 2023).
Watermarking circumvents this issue by embedding information into the generation process by manipulating the decoding step. For example, the watermark of Kirchenbauer et al. (2023a,b) biases the logits to boost the probability of specific tokens (denoted “green tokens”) in contrast to the remaining “red” tokens. The set of green tokens is determined by a hash function, and the security of this watermark is based on the hardness of finding hash collisions. Despite the emergence of more watermarking schemes (Kuditipudi et al., 2023; Christ et al., 2023), research shows that methods that (slightly) modify logit distributions, such as Kirchenbauer et al. (2023a,b) outperform others without significant output quality degradation (Piet et al., 2023): humans can not identify if the text is watermarked, and detection requires processing approximately one hundred tokens. Thus, for the remainder of this work, we focus on the Kirchenbauer et al. (2023a) watermark and its variant by Zhao et al. (2023) as exemplars. Additional related watermarking methods are discussed in § 2.2.
We study if watermarking approaches based on logit perturbation are robust to post-processing distortions. This is an important question across various applications. For example, it will reliably determine if academic integrity was violated in educational settings: if the watermark is not robust, a student can easily edit the generated watermark, and the resulting data will fail verification. While prior works also ask this or similar questions (Sadasivan et al., 2023; Lu et al., 2023), they do so under unreasonable settings, where edits (or post-processing distortions) to watermarked text are unconstrained (i.e., they edit large fractions of the generated text with potential degradation to output quality). They use dedicated paraphrasing models (Sadasivan et al., 2023), or design customized prompts for post-hoc output paraphrasing (Lu et al., 2023; Shi et al., 2023). Alternatively, they design prompts to generate hard-to-detect text (which is
8549
often contextually unrelated), sometimes with low entropy (Lu et al., 2023; Shi et al., 2023).
Figure 1: Illustration of the setting for SCTS. The red box indicates the attacker’s capability.
We evaluate robustness under a more realistic setup: post-hoc editing involves making a “reasonable” number of edits to maximally retain information generated by the LLM. To achieve our goal, we make two observations. First, previous attack methods are color-agnostic: existing approaches that replace text via paraphrasing still preserve some fraction of the original text, resulting in dilution rather than erasure of watermarks (Kirchenbauer et al., 2023b) (i.e., erasure needs more edits). Even if they manage to evade detection for certain samples by removing “most” fragments, our experiments demonstrate that they would fail when constrained by a smaller edit budget (§ 6). Second, prior attacks require another high-quality “unwatermarked” LLM for paraphrasing; this LLM often generates verbose or inarticulate text, and this method is not very practical as the paraphrasing LLM can be used for generation in the first place.
Building atop these observations, we propose the Self Color Testing-based Substitution (SCTS) attack. This introduces new text fragments that contain fewer green tokens than human-written text, with high probability. We prompt the LLM to perform targeted (periodic) generations given an input context (refer prompt in § 4) to get color information from the frequency of words in the generated text. This is used to perform color-aware substitution, and in turn, neutralizes the higher number of green list tokens from the preserved fragments. Consequently, SCTS can evade watermark detection for arbitrarily long text segments within a reasonable edit distance budget, without using an unwatermarked LLM. Our evaluation compares SCTS and existing representative attack methods over a series of edit distance budgets. We conclude that across various settings, our approach is superior in reducing AUROC (a proxy for detection success) to less than 0.5 on two LLMs and two watermark-
ing schemes. Additionally, and most importantly, our approach is theoretically grounded, with a comprehensive analysis.
In summary, our main contributions are:
1. We theoretically show how existing coloragnostic methods can only “dilute” watermarks, and can not evade detection when the watermarked text is sufficiently long (§ 3). Empirically, we show that existing methods fail to evade watermarks within reasonable edit distance budgets (§ 6).
2. We propose the first “color-aware” attack method by prompting the LLM for (a seemingly) random generation to obtain color information, and replace the green tokens/words with red tokens/words. We then analyze the working mechanism of our attack and estimate its efficacy and costs under reasonable edit distance constraints (§ 4).
3. We conduct extensive experiments on vicuna-7b-v1.5-16k (Zheng et al., 2023) and Llama-2-7b-chat-hf (Touvron et al., 2023) with different hashing strategies. We show that under the same edit distance budget, our attack is more effective in evading watermarks than previous methods (§ 6).

heading: 2 Background and Related Work

heading: 2.1 Text Watermarking
text: We utilize the state-of-the-art (SOTA) work of Kirchenbauer et al. (2023a) as an exemplar approach; it involves perturbing the logits, so as to guide generation towards specific tokens. Their approach works as follows: an input sample w = w−s+1, w−s+2, · · · , w0 of s words is tokenized to obtain s′ tokens. These are then fed to the the large language model (LLM) which generates T tokens as the response. Note that tokens are generated one by one. More specifically, before the generation of every output token, c tokens (where c is known as the context size) preceding that are used to seed a pseudo-random (hash) function which is used to divide the vocabulary space of size |V |1, dynamically, into a green and red list, where the size of the green list is γ|V |. Based on re-weighting green tokens
1This is the vanilla left hash. A preferred variant is to also use the token being generated as an input to the hash. This is called self hash, and this is empirically more robust to watermark removal attacks.
using an offset δ (more details in §3 of Kirchenbauer et al. (2023a)), the approach prioritizes the selection of a green list token as the next candidate. This process is repeated to generate all T output tokens. The aim of their approach is to ensure that the actual number of green tokens in the generation (||T ||G) is higher than the expected number of green tokens (γT ). Post-hoc, this is verified using a statistical test:
z = ∥T∥G − γT√ Tγ(1− γ)
The text is labeled as watermarked if and only if z is greater than some threshold.

heading: 2.2 Prior Watermarking
text: Text watermarking already existed before LLMs. For example, Atallah et al. (2001) uses syntactic and semantic changes to integrate the watermark, but the change may not fit the context well because the changes are made within limited contexts. Topkara et al. (2006) performs watermarking using synonym substitutions relying on the ambiguity of natural language; this is unavailable for certain languages. Moreover, Venugopal et al. (2011) proposes a method to probabilistically identify the structured outputs of machine learning algorithms (such as machine translation), but their false positive rates (5 % to 8%) are not ideal for applications such as checking if students’ assignments are written by LLMs. More recently, Xiang et al. (2021) generates watermark samples by following a carefully chosen semantic combination pattern, which can be too conspicuous. While He et al. (2022a) uses lexical changes to watermark the output of text generation models, it is not training-agnostic and requires watermarked text during training, which can be challenging for LLM with billions of parameters. Lastly, He et al. (2022b) minimizes the distortion of overall word distributions while maximizing the change of conditional word selections, yet it is vulnerable to detection through statistical analysis of word frequency changes.

heading: 2.3 Attacks against Watermarks
text: Most, if not all attacks involve replacing a subset of words. We describe two strategies below. Strategy 1. Paraphrasing: These attacks aim to replace a group of words with semantically similar counterparts. This can be done directly using a specialized LLM (Krishna et al., 2023; Sadasivan et al., 2023), word-level substitutions (Shi et al., 2023), or
translation (to another language and back) (Christ et al., 2023). For example, the recursive paraphrasing (RP) approach (Sadasivan et al., 2023) paraphrases (up to 5 iterations) the watermarked text using an unwatermarked paraphrasing model. Strategy 2. Prompting: Another class of attacks involves carefully designing prompts to guide the model to generate text that evades detection (Lu et al., 2023), or to guide the LLM to generate lowentropy text which is hard to watermark. As an example of the first category, Lu et al. (2023) propose SICO-Para (SICO), where the LLM is tasked with generating features of human-written text. Using such features and human-written style examples, it can guide the LLM to augment the AI-generated text to be more human-like. SICO alternatively performs sentence and word-level updates, to greedily minimize the probability of detection using a proxy. As such a proxy is usually not a watermark detector and focuses on the semantics, it would not help when attacking watermarks.
In our work, we focus on RP and SICO as representative baselines. Limitations of Current Approaches: Both approaches are color-agnostic. This means the new fragments introduced by the attacker will be independent of the color of existing fragments (i.e., will have the same green token ratio as human-written text), while a large number of old fragments (which are mostly green) are also preserved (Kirchenbauer et al., 2023b). Consequently, the resulting text will still mostly comprise of green tokens, when we consider both existing and “added” fragments. This introduces natural tensions. Most importantly, both approaches are ineffective when there are constraints placed on the number of permitted edits. For example, we see that both SICO and RP’s AUROC is still ≥ 0.86 under the most relaxed 0.5 normalized edit distance (i.e., editing 50% of words of the watermarked text)2.

heading: 2.4 Threat Model
text: Recall that the robustness of watermarks is defined as their tolerance to edits post-hoc. Prior work (Lu et al., 2023) evaluates robustness by making unrealistic assumptions about the (robustness) adversary’s capabilities. They assume that the adversary has access to a version of an LLM without a watermarking algorithm. In our work, we do
2AUROC describes the distinction of the z-scores for (attacked) watermarked text and unwatermarked text and it is more comprehensive than Attack Success Rate (ASR).
not make this strong assumption. Like other prior approaches (Zhang et al., 2023; Sadasivan et al., 2023), we assume:
1. API access to the watermarked model, using which we can issue input prompts, and observe the generated responses.
2. The watermarked model is aligned, and capable of following instructions provided.
3. Knowledge of the context size c.
4. No knowledge of other watermarking hyperparameters, like γ, temperature t, and δ.
5. Access to some model (not necessarily an LLM) capable of generating word substitution candidates. This model can be watermarked.
We believe assumption 2 is realistic, given how most models in the status quo are instruction finetuned, and trained using reinforcement learning with human feedback (Lambert et al., 2022). We also stress that assumptions 2 and 3 are not strict.

heading: 3 The Building Blocks
text: We will first show some properties of the watermarking efficacy as a function of output (generation) length, and use this to explain why existing color-agnostic attack methods fail for sufficiently long watermarked text. Then, we focus on substitution-based attacks and formalize the coloragnostic baseline. But before we begin, we outline the assumptions and definitions we make throughout this section. Assumption 1: We consider the left hash for ease of exposition. As stated earlier, the left hash is one where the green list for the current token is obtained by hashing c tokens counting backward from the current token (not including it). Assumption 2: Each (generated) token from the watermarked LLM is green with constant probability p, i.i.d, s.t. γ < p < 1. In the generated output, the number of c-grams is T − c. Definition 1 (c+ 1-gram). This is the set of tokens used for color testing. It includes the token whose color is to be checked and the c tokens before it. The color of a c+1-gram refers to the color results when this c+ 1-gram is sent to the detector.
Definition 2 (Effective length Te). This is the number of c+ 1-grams used for color testing i.e., Te = T − c
Definition 3 (Detection threshold zth). This is the threshold used in detection. The detector outputs “watermarked” ⇐⇒ z > zth. Definition 4 (Critical length Tc). This is the value of Te such that E[z] = zth. This can be thought of as the effective length needed for successful detection, and may or may not exist.
Definition 5 (Average green probability q). : This is the average probability for a c+ 1-gram being green for any given arbitrary sample, which can be watermarked or non-watermarked, attacked or non-attacked. By definition, E[∥T∥G] = qTe.

heading: 3.1 Theorem: Watermark Strength
text: Theorem 1. E[z] is proportional to √ Te. To elaborate, first assume the colors of c + 1-grams are independent. Then, we have:
• For q < γ, the probability of detection as “watermarked” converges exponentially to 0 with respect to Te.
• For q > γ, the probability of being detected as “unwatermarked” converges exponentially to 0 with respect to Te.
Furthermore, if the color for different c + 1- grams is green, is i.i.d., then:
• For q ̸= γ, tighter bounds are applicable in comparison to the scenarios described above in the independent case.
• For q = γ, the probability of being detected as “unwatermarked” converges to a constant determined by zth.
The proof of the above is in Appendix B.

heading: 3.2 Why Existing Methods Fail For Long Text
text: As a warm-up, let us consider text that is not attacked. Applying Theorem 1, where q = p > γ (for the i.i.d. case), we can see that the false positive rate converges to 0 exponentially w.r.t. Te. Candidate Attack 1. There are attacks that design prompts to guide the watermarked LLM to generate low-entropy text to evade detection. It is hard to incorporate watermarks in such (generated) text segments. But even if one can circumvent this challenge, we will see that it may only dilute the watermark. To this end, assume that the attacker can reduce p to p′, such that p > p′ > γ, i.e., the c+ 1-grams are i.i.d green with probability p′. Note, however, that when p′ > γ, we can apply
Theorem 1 where we set q = p′ > γ (for the i.i.d. case as before); the false positive rate converges to 0 exponentially w.r.t. Te.
Conclusion. This attack would fail for sufficiently long watermarked text, and it can only dilute the watermark.
Candidate Attack 2. Here, the attacker uses postprocessing methods to evade detection. Existing paraphrasing-based methods (Lu et al., 2023; Krishna et al., 2023) fall into this category. Since the post-processing is color-agnostic, we assume the newly generated c + 1-grams are green with i.i.d. probability γ, and the number of such new c+1-grams is Tnew. Also note that post-processing attacks such as paraphrasing are statistically likely to leak n-grams or even longer fragments of the original text (Kirchenbauer et al., 2023b). We call the leaked segments “old” c+ 1-grams. These old c+ 1-grams are green with i.i.d. probability p, and the number of such old c+1-grams is Told. The existence of two classes of c+ 1-grams suggests that when detecting the watermark from the attacked text, Te = Told + Tnew. The ratio of leaked c+ 1- grams is ro = ToldTold+Tnew , and is lower bounded by constant r > 0. Finally, the colors for different c+ 1-grams categories are independent.
Definition 6. We define ◦ ∥T∥oldG , the random variable for the number of “old” c+ 1-grams which are green. ◦ ∥T∥newG , the random variable for the number of “new” c+ 1-grams which are green. So ∥T∥oldG ∼ B(Told, p), ∥T∥newG ∼ B(Tnew, γ), ∥T∥G = ∥T∥oldG + ∥T∥newG .
Applying the independent case of the Theorem 1 (q > γ), we can see that the probability of evading detection exponentially converges to 0 w.r.t. Te.
Remark 1. In paraphrasing attacks, we can safely assume that attack makes the text longer. Remark 2. Even if attack 1 and attack 2 are combined, i.e., the attacker can weaken the watermark and do color-agnostic postprocessing, then it would still fail for sufficiently long text. To see this, just let p = p′ > γ in the above derivation. Conclusion This attack would fail for long enough generated text when a non-zero ratio of c+ 1-grams is preserved.

heading: 4 Our Approach: Self Color Testing
text: Notation: The watermarked sentence (i.e., the output of the LLM) is denoted w = {w1, w2, · · · , wT }. In this sentence, the word being substituted is denoted wb. The candidate substitution is w′b. Finally, w c b denotes the context of wb, which determines the green list at location b i.e., wcb is the c words before wb (assuming left hash)
Algorithm 1: Self Color Testing (SCT) Input: wb, w′b, wcb = {wb−c . . . wb−1}, pth; Output: Test result in {GR,RG,S}. GR (RG) means wb is green (red) while w′b is red (green), S means wb and w′b are of the same color;
p = MakePrompt(wb, w′b,w c b); o = LLM(p); cb, c ′ b = Count(o,w c bwb), Count(o,w c bw ′ b); pt = χ 2-test(cb, c′b); if pt ≥ pth then return S; end else if cb > c′b then
return GR; else
return RG; end
Step 1. Self Color Testing: The secret to a successful attack against watermarked LLMs is to have color information. However, obtaining this is not straightforward. We leverage the insight that aligned and instruction fine-tuned models are compliant with user instructions. Thus, we can prompt the LLM (at temperature t = 0) to generate seemingly random strings (in a deterministic manner) with customized input prefixes, and infer color information from the frequency of outputs generated (abstracted by the “Count” method in Algorithm 1). We focus our discussion on the word level to avoid encoding issues and incomplete word biases. Algorithm 1 contains all relevant details.
To explain the intuition behind the algorithm, consider the following example prompt for color testing. Here, note that (wb=includes, w′b=contains, and w c b=kernel)
Choose two phrases ( kernel includes, kernel contains ), and generate a long uniformly random string of those phrases separated by ";". Previous phrases should have no influence on future phrases: kernel includes; kernel contains; kernel includes; kernel contains; kernel contains; kernel includes; kernel contains; kernel contains; kernel contains; kernel contains; kernel contains; kernel contains; kernel contains; kernel contains; kernel includes
When the above prompt is processed by the LLM,
we check the frequency of “kernel includes” and “kernel contains” in the response. Assume the token before “includes” and “contains” are the same in the encoded “kernel includes” and “kernel contains”. Since the model is deterministic (temperature t = 0), and follows the instructions in the prompt, the model will always generate the green token if wb and w′b are of different colors. The set of test results is {GR,RG,S}. GR (RG) means wb is green (red) while w′b is red (green), S means wb and w ′ b are of the same color. Thus, SCT has a perfect recall on GR and RG. For the S scenario, there can be border cases, but in realistic applications (t > 0), two tokens of the same color will have the same probability of being generated. Step 2. SCT Substitution: We use color-testing to test different candidates to ensure that the green token is substituted by a red one. Note that (a) Generate_candidates(w′, i) generates k substitution candidates (different from the original) for w′ at index i, and (b) Substitute(w′, i, w′i) updates the i-th word of w′ with w′i. The choice of step size 2 upon successful substitution is heuristic: consider the left hash c = 1 case. Suppose we just substituted wi with w′i, this will also change the green list at wi+1. It is reasonable to assume the probability of being green for wi+1 is reduced to γ (from p) as we did not check its color before substitution. Consequently, continuing to substitute wi+1 is less likely to be successful, as wi+1 has a lower probability of being green, which in turn will lead to more computation. Budget Enforcement: To enforce the budget, we add checks after each substitution. More details are in Appendix E.2.

heading: 4.1 Analysis
text: To simplify our analysis, we assume that each word corresponds to a single token, and that the candidate generation process is color-agnostic i.e., every candidate is green with i.i.d probability γ. We focus on c = 1 left hash for simplicity. Expected Green Ratio. Let q(Te) be the average green probability. When the number of candidates in each substitution attempt k > 1 + logγ 1−p p(1−γ) , we have
q(Te) ≤ max{ γ
2 ,
γkp
1− p+ γkp} < γ (1)
lim Te→∞ q(Te) = γ − γ(1− pγk−1) 1 + (1− γk)p < γ (2)
Algorithm 2: SCTS Algorithm Input: w, Output: w′ Initialization: i = c, w′ = w while i ≤ T do
w′i,1, . . . , w ′ i,k =
Generate_candidates(w′, i) success = False for j = 1 to k do
if SCT(wi, w′i,j , {wi−c . . . wi−1}) == GR then
w′ = Substitute(w′, i, w′i,j) success = True break
end end if success then
i← i+ 2 (advance by 2 for economic substitution)
else i← i+ 1
end end
Using similar techniques as theorem 1, we have
loge Pr(z > zth) < − (γ − q(Te))2 ( √ Te +
√ γ(1− γ)zth γ − q(Te) )2
for q(Te) < γ, which holds for large enough k and/or Te. Thus, the probability of failing to evade the watermark converges to 0 exponentially. This means that, in expectation, our method can reduce the average green probability to less than γ, and is amenable to arbitrarily long text. More details are presented in Appendix C. LLM Calls. We also analyze the number of LLM calls needed for our approach in Appendix D. Our method is reasonably fast, with O(1) w.r.t. k, and O(Te) w.r.t. Te in expectation.

heading: 5 Experimental Setup
text: Constraints: We consider word level edit distance normalized by the number of words for the water-
marked text. The constraints we use are from 0.05 to 0.5, with intervals of 0.05.

heading: 5.1 Baseline Methods
text: 1. RP. To enforce edit-distance constraints, we check if the paraphrased sample is within the constraint. If not, we use the unparaphrased sample as the result. The other settings are as suggested by the official implementation of Sadasivan et al. (2023). RPi denotes i cycles of recursive paraphrasing; i = {1, 2, 3, 4, 5}. 2. SICO. We focus on the SICO-para variant (Lu et al., 2023). To enforce edit constraints, we add a description of the constraint in the prompt and only include compliant samples. The detailed prompt is in Appendix E. The other settings are kept the same as in Lu et al. (2023). The training samples are also the same as that used in Lu et al. (2023). 3. RB. This is a baseline for substitution without color testing. It uses the same candidate generation but without SCT. This is used to show if the color information from SCT is useful.

heading: 5.2 Models
text: In our experiments, we consider 2 models : vicuna-7b-v1.5-16k and Llama-2-7b-chat-hf. For candidate generation, we use the HuggingFace pipeline using distilroberta-base (Sanh et al., 2019). We have k = 5 substitution candidates for each word.

heading: 5.3 Watermarking Scheme
text: We consider two schemes. 1. UMD by Kirchenbauer et al. (2023a): Here, we consider both left hash (or Min-LeftHash) and self hash (or Min-SelfHash) (more details in (Kirchenbauer et al., 2023b)), and context size of c = 1, 2, 4, 8. As suggested by the authors, we set zth = 4, γ = 0.25, and δ = 2. For efficiency, we batched the self hash. We consider two models: vicuna-7b-v1.5-16k and Llama-2-7b-chat-hf . 2. Unigram by Zhao et al. (2023): This is a variant of UMD’s SelfHash with c = 1, but whose green list is fixed. Here, we set zth = 4, γ = 0.5, and δ = 2 (based on their released code3). We consider only one model: vicuna-7b-v1.5-16k 4.
3We reduce zth from 6 to 4 to make Attack Success Rate (ASR) more representative
4For UMD self hash c = 1 and Unigram, we slightly modify the SCTS prompt (Appendix E.5), and it will always advance by 1 regardless if the substitution succeeds.
Other Parameters The p-value threshold for the χ2 test in SCT is set to 0.01.
More details about the datasets and metrics we calculate are presented in Appendix E.

heading: 6 Experiment Results
text: Through our evaluation, we wish to answer the following questions:
1. Is SCTS more effective than previous methods?
2. What budget is required for a successful attack?
3. Does the above hold for different target watermarked models?
We observe that:
1. SCTS is consistently more effective for different values of c, hashing schemes, and watermarking schemes, while preserving output semantics (§ 6.1).
2. Normalized edit distance of 0.25 − 0.35 is sufficient for SCTS, while other attacks need more (§ 6.2).
3. Attack success rate (ASR), as defined in Appendix E.3, is heightened for models which are aligned and instruction fine-tuned (§ 6.3).
For most metrics, we visualize the c = 4 case of UMD due to space constraints. Detailed results (for both UMD and Unigram) are presented in Appendix F, and highlight the same trends. In particular, we observe that Unigram is more robust than UMD, but still susceptible to SCTS. For RP, we only visualize RP5 as it is the strongest attack.
Table 1: Semantic similarity for vicuna-7b-v1.5-16k, 50 samples, UMD watermarking, c = 4. SCTS successfully preserves semantics.
Hashing Method 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
Left
RP5 1.0000 0.9979 0.9978 0.9986 0.9922 0.9894 0.9881 0.9711 0.9586 0.9467 0.5436 SICO 0.9984 0.9988 0.9906 0.9889 0.9884 0.9886 0.9934 0.9910 0.9845 0.9600 0.7104 RB 0.9862 0.9728 0.9538 0.9394 0.9203 0.9008 0.8854 0.8732 0.8546 0.8546 - SCTS 0.9878 0.9743 0.9579 0.9366 0.9198 0.9018 0.8842 0.8832 0.8832 0.8832 -
Self
RP5 0.9998 0.9987 0.9988 0.9916 0.9931 0.9841 0.9709 0.9582 0.9488 0.9416 0.5464 SICO 0.9979 0.9999 0.9925 0.9880 0.9614 0.9869 0.9811 0.9857 0.9722 0.9674 0.6439 RB 0.9847 0.9686 0.9507 0.9330 0.9141 0.8964 0.8801 0.8611 0.8459 0.8459 - SCTS 0.9850 0.9673 0.9499 0.9285 0.9105 0.8891 0.8754 0.8737 0.8732 0.8732 -

heading: 6.1 Is SCTS Effective?
text: Yes, it is. Figure 3 and 5 shows the performance of the attack on AUROC and detection success respectively over different edit budgets. We can see that SCTS is consistently more effective.
At relatively high budgets like 0.35, SCTS can reduce the AUROC to less than 0.5, which means that the z-score (used for detection) is, on average, more negative. This in turn corresponds to the scenario where q < γ in equation 2, despite some of the assumptions we made not holding.
SICO and RP, in contrast, fail to evade detection at most budgets, and are even worse than RB. Even though they work reasonably well when they are unconstrained, there are still a few detectable samples. Also, their AUROCs are still in the range of 0.6 to 0.9, suggesting that the watermark is generally only diluted and can be detected for longer text. Impact of c and hashing scheme. Both figures show that self hash is generally more robust than the left hash, especially for RB and SCTS. This is consistent with the findings in Kirchenbauer et al. (2023b), and also holds for c = 2, 4, 8. Also, smaller c is generally more robust from our experiment results in the Appendix F, Table 2, 3, 12, 13, consistent with the findings of Kirchenbauer et al. (2023b). Nevertheless, smaller c comes with a higher risk of leaking the green c+ 1-grams to an attacker, more loss in generation quality (Kirchenbauer et al., 2023b), lower z and successful detection rates as shown in our experiments. Semantic similarity. Another key factor for a successful attack is if it can preserve semantics. From Table 1, notice our method successfully preserves semantics during the attack, with a mean cosine similarity 0.8832 and 0.8732 at 0.4 budget (for left and self hash respectively), which is comparable to RP1. Results for the Unigram watermark are slightly lower, and more details are presented in
Table 11 in Appendix F. We believe that semantics will be better preserved if we use more powerful substitute generators, or make larger substitutions (phrases vs. words as we currently do).
SCT Accuracy. The key factor for our success is the accuracy of SCT. From Figure 4, we see that the accuracy for SCT is ≥ 0.5. This is significantly higher than 0.33, a rough random baseline estimate for a three-class classification problem, suggesting the effectiveness of SCT. This is particularly impressive given that Kirchenbauer et al. (2023a)’s coloring scheme is designed to be hard to identify.

heading: 6.2 Attack Budget
text: For samples we evaluated, SCTS is successful at evading detection with as low as 0.25 normalized
edit distance. Albeit RB also maximizes ASR at higher budgets, only SCTS is able to consistently reduce AUROC to less than 0.5. The trend is consistent for all values of c considered.
However, note that SCTS saturates in the 0.35− 0.45 interval. This is because the current implementation only scans the sentence once when performing substitutions, and the ASR for each attempt can be lower than the theoretical result. Determining techniques to scan the sentence multiple times to increase residual replacement success is subject to future research.

heading: 6.3 Different Watermarked Models
text: Notice that the success of SCT is implicitly connected to the ability of the model to follow instructions. We observed that Llama-2-7b (model that is not instruction fine-tuned) would frequently fail to follow the prompt. For example, when the prompt is to let it generate phrases from kernel includes and kernel contains, it would probably generate several of these phrases and then begin to generate includes and contains (in words only instead of phrases), whose green lists are different. Consequently, there are insufficient number of samples for the χ2 test leading to bias in SCT towards S. In this case, the success rate for each substitution attempt would drop, and SCTS would saturate at a lower budget, become slower, and less effective. On the other hand, the chat variant Llama-2-7b-chat-hf suffers less from this issue, while vicuna-7b-v1.5-16k (further finetuned from Llama-2-7b-chat-hf) is even better. We would argue that as models become more aligned and instruction fine-tuned, they will follow the prompts better in general, and SCTS will be more effective.

heading: 7 Discussion and Open Questions
text: A “green list” approach baseline? A straightforward methodology for detecting "green list" words involves the analysis of publicly available humanwritten texts, as demonstrated by He et al. (2022b). However, this attack method is not applicable to the target watermarking (Kirchenbauer et al., 2023a); see Appendix A. Can SCTS be faster? One way is to store the color information already found to reduce repetitive color testing, with the risk of the accumulation of incorrect results and the cost of space. Such caching is more practical for small c.
Can one LLM query get more color information? Currently, our color testing can only test one pair in each LLM query, and it can not distinguish if the two words/tokens are both red or green. More candidates for random generation can help with the cost of more undesired factors getting involved, like the increased complexity of the prompt, the reduced frequency count for each candidate, and the more complex cases in hypothesis testing. Unknown c Besides prompting the model to guess c first, one way to use SCTS in this case is to use a large estimated c. We leave this for future work. Can the accuracy of SCT be higher? (Tang et al., 2023) shows the complicated behavior when the LLM model is prompted to generate uniform random strings, which are far from uniformity and vary model by model. Such behavior makes our color testing sometimes inaccurate. One possible way is to have multiple variants, like exchanging the position of the new candidate and the old to do a second prompt, to improve accuracy with the cost of more computation.

heading: 8 Conclusions
text: Our study presents SCTS, an algorithm to evade watermark detection without using external LLMs. We demonstrate that SCTS can effectively eliminate watermarks from long texts using a straightforward algorithm. This approach reveals that specific prompting techniques can uncover and exploit private watermarking information, enabling evasion. We aim to inspire further research on developing more robust and secure watermarking schemes.

heading: 9 Limitations & Harms
text: Limitations: One limitation of SCTS is that its efficiency can be improved, as shown in 6, 7. The color information is also limited to one pair due to an attacker can only prompt the black-box watermarked model. We assume the attacker knows c, while a workaround is possible for future work. SCT accuracy is also not very high so the color information is not that accurate. Lastly, SCTS is currently limited to UMD and its variants like Unigram. Harms: Through this work, we propose an approach to circumvent text watermarking strategies. This has implications for spreading misinformation and purporting enhanced (nefarious) dual-use of LLMs. We hope that our findings can help design more robust watermarking techniques.

heading: Appendix

heading: A Why does “Green list” Approach Fail
text: Human-generated text, in expectation, contains an equal number of green and red tokens. If an LLM is watermarked using the methods described by Kirchenbauer et al. (2023a), then the tokens generated by the model are more biased towards being green. The challenges include the following:
1. To ensure comparison, we would first need to identify green tokens in human-generated text reliably, which is challenging since the probability that a token is green is 0.5 (based on the definitions in Kirchenbauer et al. (2023a)). Assuming we overcome this issue, identifying color information (of LLM-generated text) using the predetermined color information (from static human text) is futile: color information changes depending on the token used to seed the random number generator i.e., the coloring of the vocabulary for generating a successor for token tj (j ̸= i). Thus, this strategy is not a reliable baseline.
2. Finally, the strategy proposed by He et al. (2022b) does not work for the Kirchenbauer et al. (2023a) watermark, as it assumes that the watermarking method makes fairly predictable changes to the text. To quote their paper, they state: “As an example shown in Figure 1, the replaced words and their substitutions are those with most frequency decrease ratios and frequency increase ratios, respectively.” This phenomenon is not true for the watermarks we consider, and consequently these techniques can not be used.
3. Additionally, strategies that are different from ours, but attempt to learn color information are computationally expensive. For the vanilla left hash c = 1, Sadasivan et al. (2023) used one million queries to the watermarked text and only learned the green list for commonly used English words (refer to section 3 of their work); our work is substantially more efficient. For typical c like 4, it will be even harder as the combination space scales by |V |c where |V | is the vocabulary size.

heading: B Proof of Theorem 1
text: Theorem 1: E[z] is proportional to √ Te.
To elaborate, first assume the colors of c+ 1-grams are independent. Then, we have:
• For q < γ, the probability of detection as “watermarked” converges exponentially to 0 with respect to Te.
• For q > γ, the probability of being detected as “unwatermarked” converges exponentially to 0 with respect to Te.
Furthermore, if the color for different c+ 1-grams is green, is i.i.d., then:
• For q ̸= γ, tighter bounds are applicable in comparison to the scenarios described above in the independent case.
• For q = γ, the probability of being detected as “unwatermarked” converges to a constant determined by zth.
Proof: Recall that ∥T∥G is the number of green c+ 1-grams out of Te c+ 1-grams in the text, and
z = ∥T∥G − γTe√ Teγ(1− γ)
For the expected z, recall E[∥T∥G] = qTe so that
E[z] = q − γ√ γ(1− γ) √ Te
The proportionality (∝) relationship is obvious. For Tc, simply let E[z] = zth and solve for Te (results below).

heading: B.1 Case 1: i.i.d. case
text: Let D(a||q) be KL-divergence with base e be defined as follows:
D(a||q) := a ln a q + (1− a) ln 1− a 1− q
S1. If q > γ, then E[z] ∝ √Te and Tc = γ(1−γ)z 2 th
(q−γ)2
How? Under assumption 2 that watermarked tokens are green with i.i.d. probability p, we have ∥T∥G ∼ B(Te, q). From the Chernoff bound (Arratia and Gordon, 1989), for arbitrary y s.t. 0 ≤ y ≤ Teq:
Pr(∥T∥G ≤ y) ≤ exp(−TeD( y
Te ||q)) (3)
When Te ≥ Tc, take y = zth √ Teγ(1− γ) + γTe ≤ qTe, we have:
Pr(z ≤ zth) ≤ exp ( −TeD (√ γ(1− γ)zth√
Te + γ||q
))
Note that
lim Te→∞
−TeD( √ γ(1−γ)zth Te
+ γ||q) Te = −D(γ||q)
=⇒ Probability of being labeled “not watermarked” converges to 0 exponentially w.r.t. Te.
S2. If q = γ, then E[z] = 0, and Tc does not exist. How? According to the De Moivre–Laplace theorem (Dunbar, 2011), as Te → ∞, the distribution of z approaches the standard normal distribution N(0, 1). This convergence allows us to use the properties of the standard normal distribution to estimate probabilities related to z. Specifically, the probability of z being detected as “watermarked” when exceeding a threshold zth can be expressed as:
lim Te→∞
Pr(z > zth) = 1− Φ(zth)
where Φ(zth) is the cumulative distribution function (CDF) of the standard normal distribution.
=⇒ Probability of being labeled as “watermarked” converges to a positive constant. Typical value zth = 4, 1− Φ(zth) ≈ 0.00003167.
S3. If q < γ, then E[z] ∝ −√Te, and Tc does not exist. How? A symmetric bound as equation 3 is: for y ≤ Teq,
Pr(∥T∥G ≥ y) ≤ exp(−TeD( y
Te ||q))
Take y = zth √ Teγ(1− γ) + γTe ≥ qTe,
Pr(z > zth) ≤ Pr(z ≥ zth) ≤ exp ( −TeD( √ γ(1− γ)zth√
Te + γ||q)
)
=⇒ Probability of being labeled “watermarked” converges to 0 exponentially w.r.t. Te.

heading: B.2 Case 2: Independent case
text: Assume the c+ 1-grams’ colors are independent.
S1. If q > γ, E[z] ∝ √Te and Tc = γ(1−γ)z 2 th
(q−γ)2
How? ∥T∥G is a sum of Te different variables bounded by [0, 1]. From the Hoeffding Inequality (Hoeffding, 1994), for t ≥ 0:
Pr(qTe − ∥T∥G ≥ t) ≤ exp ( −2t 2
Te
)
For Te ≥ Tc, take t = (q − γ) √ Te( √ Te − √ Tc) ≥ 0, then:
Pr(z ≤ zth) ≤ exp ( −2(q − γ)2( √ Te − √ Tc) 2 )
=⇒ The probability of being labeled as “not watermarked” converges to 0 exponentially w.r.t. Te.
S2. If q = γ, E[z] = 0 and Tc does not exist. S3. If q < γ, E[z] ∝ −√Te and Tc does not exist. How? A symmetric bound as from the Hoeffding Inequality (Hoeffding, 1994) gives, for t ≥ 0:
Pr(∥T∥G − qTe ≥ t) ≤ exp(− 2t2
Te ) (4)
Take t = (γ − q)(√Te + √ γ(1−γ)zth γ−q ) √ Te ≥ 0, and note that Pr(z > zth) ≤ Pr(z ≥ zth):
Pr(z > zth) ≤ exp(−2(γ − q)2( √ Te + √ γ(1− γ)zth γ − q ) 2)
=⇒ Still, the probability of being labeled as “watermarked” converges to 0 exponentially w.r.t. Te.

heading: C SCTS’ Efficacy Analysis

heading: C.1 Success Rate
text: For each 2-grams’ substitution attempt, the success probability
ps = Pr(2−gram is green, at least one of k candidates is red) = Pr(2−gram is green) · (1− Pr( a candidate is green)k) = p(1− γk)

heading: C.2 Grouping
text: For texts attacked by SCTS, i.i.d. color distribution like assumption 2 does not hold anymore as one substitution may change the color of two tokens (c = 1). We need to regroup so that every group’s color is independent to reach similar bounds as theorem 1. The grouping is as follows: 1. one 2-gram as a group if it is preserved in SCTS, i.e., this token and the token preceding this token are not substituted); and 2. two adjacent 2-grams as a group if they are not preserved because of the same substitution, i.e., the token being substituted and the token following it. We call the first type of groups “old groups” and the second type of groups “new groups”. A corner case for one substitution only changes one 2-gram as the word being substituted is the last (could not be the first for SCTS) is not considered.

heading: C.3 Expected Green Ratio
text: From the assumption that candidate generation is independent of color, each group’s number of green 2-grams is independent, and i.i.d. within old groups and new groups. Let the expectation of the ratio of green tokens for old groups be po for notation. (“attempt fails" means that the attempt to substitute the first token in the 2−gram fails in SCTS.)
po = Pr(2−gram is green | attempt fails)
= Pr(attempt fails | 2−gram is green) · Pr(2−gram is green)
Pr(attempt fails)
= γk · p 1− ps
= γkp
1− p+ γkp
Note the expectation of the ratio of green tokens for new groups is γ2 and
po = γkp 1− p+ γkp < γ ⇐⇒ k > 1 + logγ 1− p p(1− γ)
Thus, equation 1 holds as desired. Let the expected number of green 2-grams after SCTS attack be ETe = q(Te)Te. Then we have:
ETe = ps(ETe−2 + γ) + (1− ps)(ETe−1 + po)
So Te · q(Te) = ps(q(Te − 2) · (Te − 2) + γ) + (1− ps)(q(Te − 1) · (Te − 1) + po)
Let q = limTe→∞ q(Te), Te → ∞, then
Te · q = ps(q · (Te − 2) + γ) + (1− ps)(q · (Te − 1) + po)
q = psγ + (1− ps)po 1 + ps = γ − γ(1− pγ k−1) 1 + (1− γk)p < γ

heading: C.4 Success probability bound
text: Definition 7 (New 2-gram ratio rn). This is defined as the number of new (w.r.t the unattacked watermarked text) 2-grams divided by Te. Then 0 ≤ rn ≤ 1.
Based on the grouping strategy discussed in Appendix C.2, we have rnTe2 new groups and (1− rn)Te old groups. Under the assumption of independence, the Hoeffding inequality gives: for t ≥ 0,
Pr(qTe − ∥T∥G ≥ t) ≤ exp ( − 2t 2
rnTe 2 × 22 + (1− rn)Te × 12
)
=exp ( − 2t 2
(1 + rn)Te
)
≤ exp ( − t 2
Te
)
So the only difference from equation 4 is the denominator. When q < γ for sufficiently large k and/or Te, use similar techniques, i.e., Hoeffding’s inequality, and have:
Pr(z > zth) ≤ exp(−(γ − q(Te))2( √ Te + √ γ(1− γ)zth γ − q(Te) )2)
Combining with equation 2, the probability of failing to evade watermarking exponentially converges to 0 w.r.t. Te.

heading: D SCTS LLM Calls
text: Definition 8 (Number of LLM calls NTe). The number of LLM calls needed for SCTS on one sample with Te 2-grams.
E[NTe ] ≤max { 1
2
( 1 γ − (
1 1− (1− γ)k − 1 ) k ) , 1− γk 1− γ } Te
<max
{ 1
2γ ,
1
1− γ
} Te
(5)
As Te → ∞ and k → ∞, the expected number of LLM calls per c+ 1-gram can be estimated. This is crucial for scalability.
lim Te,k→∞ E[NTe ] Te = pγ + (1− p)(1− γ) (1 + p)γ(1− γ) (6)
For example, with p = 0.5 and γ = 0.25, this value is 169 . Also, NTe will not deviate far from its expectation with high probability.
Pr(|NTe − E[NTe ] Te | ≥ t) ≤ C Tet2
(7)
For t > 0 and constant
C :=max
{ 1
2 ( 1− γ γ2 − k 2(1− γ)k (1− (1− γ)k)2 ) ,
γ (1− γ)2 − γk((2k − 1)(1− γ) + γk) (1− γ)2
}
<max { 1− γ 2γ2 , γ (1− γ)2 }

heading: D.1 Derivation
text: The grouping is the same as in Appendix C.2.
Definition 9 (Nnew). The number of LLM calls for a new 2-grams group. Definition 10 (Nold). The number of LLM calls for a old 2-grams group.
We have
Pr(Nnew = i) = Pr(Gγ = i)
Pr(Gγ ≤ k) , i = 1, 2, . . . , k
where Gγ is a geometric distribution with mean 1γ . So,
E[Nnew] = E[Gγ |Gγ ≤ k]
= 1 γ − (
1 1− (1− γ)k − 1 ) k
< 1
γ
For the old 2-gram group, we have:
Pr(Nold = i) = Pr(G1−γ = i), 1 ≤ i ≤ k − 1 Pr(Nold = k) = γ k−1 = Pr(G1−γ = k) + Pr(G1−γ > k)
Therefore,
E[Nold] = 1− γk 1− γ < 1 1− γ
Note that
E[NTe ] = ( rn
E[Nnew] 2
+ (1− rn)E[Nold] ) Te
≤ max { E[Nnew]
2 ,E[Nold]
} Te
So equation 5 holds. Expectation limit (equation 6) derivation: Denote E[Nold] = no and psi = Pr(success at the i-th time) = pγi−1(1− γ) for notational simplicity.
NTe =
k∑
i=1
psi (NTe−2 + i) + (1− ps)(NTe−1 + no)
Let Te → ∞, limTe→∞ NTeTe = R
R = 1
Te
k∑
i=1
psi ((Te − 2)R+ i) + (1− ps)((Te − 1)R+ no)
R =
∑k i=1 ip
s i + (1− ps)no 1 + ps
From this, we can get R for any k, but the math is cumbersome. For simpler math, let k → ∞. Then ps = p, Nold is a geometric distribution with mean no = 1γ ,
so ∞∑
i=1
ipsi = p
1− γ Thus, equation 6 holds. Expectation bound by Chebyshev’s inequality (equation 7 derivation):
V ar(Nnew) = 1− γ γ2 − k 2(1− γ)k (1− (1− γ)k)2 < 1− γ γ2
V ar(Nold) = γ (1− γ)2 − γk((2k − 1)(1− γ) + γk) (1− γ)2 < γ
(1− γ)2 From independence, we have
V ar(NTe)
= ( rn V ar(Nnew)
2 + (1− rn)V ar(Nold)
) Te
≤max { V ar(Nnew)
2 , V ar(Nold)
} Te
=CTe
From Chebyshev’s inequality, for τ > 0,
Pr(|NTe − E[NTe ]| ≥ τ √ CTe) ≤ 1
τ2
Namely
Pr(|NTe − E[NTe ] Te
| ≥ τ √ C
Te ) ≤ 1 τ2
Take t = τ √
C Te , then equation 7 hold.

heading: E Additional Experimental Details

heading: E.1 Dataset
text: Following Kirchenbauer et al. (2023a), we use the training set of C4 dataset’s RealNewsLike subset (Raffel et al., 2020). For each sample, we first check if it is at least 500 tokens. If it is, we keep only the last 500 tokens. For this truncated part, the first 100 tokens are used as the prompt, and the remaining 400 tokens are used as “human-written” text for this prompt. The (watermarked) text is generated by the LLM with watermarked decoding, but keeping other configurations’ default as in the original implementation, truncated to at most 400 tokens. To avoid corner cases where the watermarked text is too short, we remove samples that are less than 20 words long. The number of samples is set to 50 and 10 for vicuna-7b-v1.5-16k and Llama-2-7b-chat-hf respectively, for reasons related to computational overheads.
SCT Experiment. For SCT accuracy, we log the first 1000 SCT results, including the SCTS result and ground truth in the same configuration as the main experiments (measuring AUROC, ASR, # LLM calls, and Running time) but in another run. This additional experiment is performed for UMD and vicuna-7b-v1.5-16k for simplicity.

heading: E.2 Budgets
text: We take budget {0.05, 0.1, 0.15, 0.2, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50}. For RP and SICO, we additionally perform unconstrained attacks (budget= 1). For SCTS, to enforce different budgets, we yield the attacked sample when a budget is reached (one more substitution will be out of the budget) and continue to run until the maximum budget is reached or SCTS saturates. When SCTS saturates, the saturated attacked sample will be used as the output for higher budgets that are not reached. As a result, one run of SCTS can give out attacked samples for all budgets, while only one budget for SICO.

heading: E.3 Metrics
text: For evaluation, we consider the metrics listed below.
1. Area Under the Receiver Operating Characteristic (AUROC). Calculated based on the humangenerated text and LLM-generated text. The lower the score is, the more effective the attack is, and 0.5 corresponds with the random guess.
2. Attack Success Rate (ASR). It is the ratio of samples that are not successfully detected with the default threshold zth = 4. Larger the value, the better the attack.
3. # LLM calls. This is the number of LLM calls, including the calls to the paraphraser for RP and vicuna-7b-v1.5-16k / Llama-2-7b-chat-hf calls for other methods. Calls of candidate generator distilroberta-base do not count as that model is much smaller and faster. Smaller the value, the more efficient the attack.
4. Running time. This is used for comparing the speed of different approaches. Note that all experiments were conducted on one NVIDIA H100 80GB HBM3 GPU with Driver Version 535.54.03 and CUDA Version 12.2 on Ubuntu 20.04.6 LTS. We use Python 3.11.5 while Python 3.8.16 is used for SICO and RP. Smaller the value, the more efficient the attack.
5. Semantic similarity. We measure semantic similarity using embeddings generated by the flan-t5-xxl sentence encoder. We report the average cosine similarity for this particular metric. Higher the value, the more information is preserved by the attack.
6. Accuracy. This is the accuracy of the SCT test. Higher the better.
7. Confusion matrix. This is the confusion matrix associated with the SCT test, modeled as a threeclass classification problem. Closer to the diagonal is the better.

heading: E.4 SICO Details

heading: E.4.1 Prompt
text: For the SICO prompt with a budget of 0.5, we modify this part of the prompt
Based on the description, rewrite this to P2 style:
to
Based on the description, rewrite this to P2 style, changing at most 50% of the words to achieve the goal.
For different budgets less than 1 (Unconstrained), the percentage of words that can be changed (50%) is updated to reflect the allocated budget accordingly.

heading: E.4.2 Training
text: For Llama-2-7b-chat-hf, we use the prompt from corresponding Unigram’s training for reasons that they share the same training data and computational overheads.

heading: E.5 SCTS prompt for UMD Min-SelfHash c = 1 and Unigram
text: Because UMD Min-SelfHash c = 1 and Unigram are essentially fixed green list agnostic to context, an adapted prompt example is as follows:
Choose two words ( includes, contains), and generate a long uniformly random string of those words separated by ";". Previous phrases should have no influence on future phrases: includes; contains; includes; contains; contains; includes; contains; contains; contains; contains; contains; contains; contains; contains; includes; contains; contains; includes; contains; contains;

heading: F Full Experiment Results
text: Table 2: AUROC for vicuna-7b-v1.5-16k, 50 samples, UMD watermarking. SCTS achieves significantly lower AUROC under the same budget compared to other baselines and is the only method cross 0.5, and the trend is consistent over different c and hashing methods.
c Hashing Method 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
1 left RP1 0.9984 0.9984 0.9984 0.9984 0.9984 0.9984 0.9984 0.9984 0.9880 0.9984 0.9900 0.8036 1 left RP2 0.9984 0.9984 0.9984 0.9984 0.9984 0.9984 0.9964 0.9984 0.9880 0.9948 0.9872 0.9096 1 left RP3 0.9984 0.9984 0.9984 0.9984 0.9984 0.9964 0.9964 0.9984 0.9868 0.9948 0.9824 0.7428 1 left RP4 0.9984 0.9984 0.9984 0.9984 0.9984 0.9964 0.9964 0.9984 0.9868 0.9828 0.9828 0.8236 1 left RP5 0.9984 0.9984 0.9984 0.9984 0.9984 0.9964 0.9964 0.9984 0.9860 0.9804 0.9800 0.7136 1 left SICO 0.9984 0.9960 0.9960 0.9960 0.9960 0.9960 0.9960 0.9956 0.9956 0.9852 0.9908 0.8404 1 left RB 0.9984 0.9892 0.9816 0.9732 0.9600 0.9524 0.9320 0.9024 0.8744 0.8134 0.7488 - 1 left SCTS 0.9984 0.9804 0.9548 0.9096 0.8168 0.7016 0.5488 0.3480 0.3064 0.3064 0.3064 -
2 left RP1 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9872 0.9724 0.9744 0.7884 2 left RP2 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9872 0.9704 0.9804 0.8836 2 left RP3 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9844 0.9660 0.9632 0.6824 2 left RP4 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9872 0.9888 0.9844 0.9656 0.9576 0.7916 2 left RP5 0.9888 0.9888 0.9888 0.9888 0.9888 0.9888 0.9872 0.9888 0.9844 0.9612 0.9548 0.6433 2 left SICO 0.9888 0.9824 0.9824 0.9824 0.9824 0.9812 0.9824 0.9800 0.9824 0.9764 0.9844 0.7580 2 left RB 0.9888 0.9792 0.9536 0.9404 0.9196 0.8968 0.8712 0.8168 0.7728 0.7048 0.6628 - 2 left SCTS 0.9888 0.9776 0.9256 0.8370 0.6856 0.5202 0.3492 0.1868 0.1720 0.1720 0.1720 -
4 left RP1 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9700 0.9712 0.9648 0.8298 4 left RP2 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9730 0.9572 0.9600 0.8298 4 left RP3 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9876 0.9884 0.9714 0.9476 0.9460 0.7241 4 left RP4 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9852 0.9662 0.9440 0.9372 0.6731 4 left RP5 0.9884 0.9884 0.9884 0.9884 0.9884 0.9884 0.9876 0.9852 0.9650 0.9440 0.9388 0.6210 4 left SICO 0.9884 0.9824 0.9824 0.9824 0.9824 0.9824 0.9820 0.9712 0.9824 0.9720 0.9472 0.8488 4 left RB 0.9884 0.9888 0.9812 0.9688 0.9492 0.9166 0.8800 0.8330 0.7580 0.6668 0.6668 - 4 left SCTS 0.9884 0.9784 0.9468 0.8818 0.7436 0.5270 0.2862 0.1342 0.1332 0.1332 0.1332 -
8 left RP1 0.9816 0.9816 0.9816 0.9816 0.9832 0.9816 0.9820 0.9708 0.9756 0.9440 0.9544 0.7816 8 left RP2 0.9816 0.9816 0.9816 0.9816 0.9832 0.9808 0.9820 0.9680 0.9708 0.9492 0.9584 0.7978 8 left RP3 0.9816 0.9816 0.9816 0.9816 0.9832 0.9816 0.9788 0.9672 0.9604 0.9384 0.9184 0.5876 8 left RP4 0.9816 0.9816 0.9816 0.9816 0.9832 0.9816 0.9784 0.9632 0.9608 0.9364 0.9360 0.6848 8 left RP5 0.9816 0.9816 0.9816 0.9816 0.9832 0.9808 0.9776 0.9616 0.9588 0.9208 0.9332 0.5522 8 left SICO 0.9816 0.9764 0.9776 0.9776 0.9780 0.9772 0.9752 0.9556 0.9776 0.9572 0.9148 0.7264 8 left RB 0.9816 0.9736 0.9548 0.9260 0.8880 0.8508 0.7968 0.7188 0.6332 0.5770 0.5770 - 8 left SCTS 0.9816 0.9552 0.8956 0.7828 0.5662 0.3608 0.1614 0.0918 0.0918 0.0918 0.0918 -
1 self RP1 0.9988 0.9988 0.9988 0.9988 0.9988 0.9932 0.9988 0.9912 0.9956 0.9952 0.9988 0.9124 1 self RP2 0.9988 0.9988 0.9988 0.9980 0.9988 0.9932 0.9988 0.9936 0.9968 0.9972 0.9976 0.9560 1 self RP3 0.9988 0.9988 0.9988 0.9988 0.9988 0.9932 0.9988 0.9952 0.9968 0.9904 0.9880 0.8848 1 self RP4 0.9988 0.9988 0.9988 0.9988 0.9988 0.9920 0.9988 0.9952 0.9948 0.9936 0.9828 0.9248 1 self RP5 0.9988 0.9988 0.9988 0.9988 0.9988 0.9920 0.9988 0.9940 0.9940 0.9876 0.9828 0.8434 1 self SICO 0.9988 0.9984 0.9984 0.9984 0.9976 0.9932 0.9984 0.9984 0.9928 0.9984 0.9916 0.8620 1 self RB 0.9988 0.9960 0.9964 0.9952 0.9896 0.9888 0.9832 0.9740 0.9680 0.9580 0.9536 - 1 self SCTS 0.9988 0.9936 0.9824 0.9524 0.9044 0.8480 0.8036 0.7160 0.6266 0.4990 0.3728 -
2 self RP1 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9864 0.9892 0.9832 0.9044 2 self RP2 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9888 0.9772 0.9832 0.8984 2 self RP3 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9880 0.9888 0.9788 0.9768 0.8044 2 self RP4 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9912 0.9844 0.9848 0.9732 0.9744 0.8012 2 self RP5 0.9912 0.9912 0.9912 0.9912 0.9892 0.9912 0.9912 0.9844 0.9796 0.9640 0.9652 0.6884 2 self SICO 0.9912 0.9748 0.9748 0.9748 0.9748 0.9740 0.9728 0.9708 0.9748 0.9724 0.9728 0.8836 2 self RB 0.9912 0.9764 0.9724 0.9656 0.9564 0.9368 0.9288 0.9108 0.8680 0.8268 0.7600 - 2 self SCTS 0.9912 0.9656 0.9248 0.8812 0.7860 0.6628 0.5140 0.3576 0.3216 0.3216 0.3216 -
4 self RP1 0.9928 0.9928 0.9928 0.9928 0.9928 0.9928 0.9928 0.9928 0.9892 0.9784 0.9812 0.8552 4 self RP2 0.9928 0.9924 0.9924 0.9916 0.9916 0.9920 0.9912 0.9924 0.9872 0.9736 0.9804 0.8700 4 self RP3 0.9928 0.9924 0.9924 0.9908 0.9920 0.9912 0.9900 0.9912 0.9844 0.9660 0.9748 0.6561 4 self RP4 0.9928 0.9924 0.9924 0.9920 0.9912 0.9912 0.9876 0.9920 0.9832 0.9636 0.9672 0.7444 4 self RP5 0.9928 0.9924 0.9924 0.9920 0.9924 0.9912 0.9868 0.9876 0.9800 0.9628 0.9664 0.6202 4 self SICO 0.9928 0.9884 0.9888 0.9884 0.9892 0.9876 0.9876 0.9764 0.9876 0.9732 0.9744 0.7540 4 self RB 0.9928 0.9812 0.9760 0.9644 0.9556 0.9412 0.9208 0.9068 0.8744 0.8340 0.8340 - 4 self SCTS 0.9928 0.9712 0.9528 0.9196 0.8364 0.7192 0.5230 0.3272 0.3108 0.3076 0.3076 -
8 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9952 0.9900 0.9740 0.7892 8 self RP2 1.0000 1.0000 1.0000 1.0000 0.9980 1.0000 0.9992 1.0000 0.9984 0.9872 0.9840 0.8148 8 self RP3 1.0000 1.0000 1.0000 1.0000 0.9972 1.0000 0.9976 0.9984 0.9932 0.9796 0.9464 0.6164 8 self RP4 1.0000 1.0000 1.0000 1.0000 0.9968 0.9972 0.9980 0.9976 0.9892 0.9724 0.9512 0.7068 8 self RP5 1.0000 1.0000 1.0000 1.0000 0.9968 0.9972 0.9964 0.9972 0.9868 0.9664 0.9412 0.6388 8 self SICO 1.0000 0.9976 0.9976 0.9956 0.9976 0.9972 0.9968 0.9956 0.9968 0.9908 0.9656 0.6951 8 self RB 1.0000 0.9932 0.9912 0.9820 0.9692 0.9560 0.9152 0.8924 0.8552 0.8204 0.8204 - 8 self SCTS 1.0000 0.9868 0.9484 0.8880 0.7664 0.5556 0.3628 0.2236 0.2236 0.2236 0.2236 -
Table 3: ASR for vicuna-7b-v1.5-16k, 50 samples, UMD watermarking, zth=4. SCTS achieves significantly higher ASR under the same budget compared to other baselines, and the trend is consistent over different c and hashing methods.
c Hashing Method 0 (Unattacked) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
1 left RP1 0.18 0.18 0.18 0.18 0.18 0.18 0.18 0.18 0.18 0.20 0.28 0.96 1 left RP2 0.18 0.18 0.18 0.18 0.18 0.22 0.20 0.24 0.26 0.30 0.36 0.94 1 left RP3 0.18 0.18 0.18 0.18 0.18 0.24 0.26 0.28 0.32 0.36 0.46 1.00 1 left RP4 0.18 0.18 0.18 0.18 0.18 0.24 0.30 0.26 0.40 0.40 0.58 0.96 1 left RP5 0.18 0.18 0.18 0.20 0.18 0.24 0.30 0.28 0.44 0.42 0.62 1.00 1 left SICO 0.18 0.28 0.28 0.28 0.30 0.30 0.32 0.30 0.28 0.40 0.40 0.82 1 left RB 0.18 0.44 0.60 0.72 0.82 0.90 0.92 0.94 1.00 1.00 1.00 - 1 left SCTS 0.18 0.56 0.74 0.92 0.96 1.00 1.00 1.00 1.00 1.00 1.00 -
2 left RP1 0.08 0.08 0.08 0.08 0.08 0.08 0.08 0.08 0.12 0.14 0.30 0.94 2 left RP2 0.08 0.08 0.08 0.10 0.08 0.10 0.10 0.08 0.16 0.18 0.36 0.80 2 left RP3 0.08 0.08 0.08 0.10 0.08 0.10 0.14 0.08 0.20 0.26 0.56 0.98 2 left RP4 0.08 0.08 0.08 0.10 0.08 0.14 0.18 0.10 0.22 0.36 0.58 0.94 2 left RP5 0.08 0.08 0.08 0.10 0.08 0.14 0.18 0.10 0.26 0.38 0.60 1.00 2 left SICO 0.08 0.16 0.18 0.16 0.16 0.18 0.16 0.20 0.16 0.26 0.26 0.78 2 left RB 0.08 0.32 0.46 0.56 0.72 0.84 0.90 0.96 1.00 1.00 1.00 - 2 left SCTS 0.08 0.46 0.68 0.92 1.00 1.00 1.00 1.00 1.00 1.00 1.00 -
4 left RP1 0.16 0.16 0.16 0.16 0.16 0.16 0.20 0.16 0.18 0.24 0.38 0.98 4 left RP2 0.16 0.16 0.16 0.16 0.18 0.18 0.20 0.16 0.22 0.32 0.40 0.86 4 left RP3 0.16 0.16 0.18 0.16 0.20 0.20 0.22 0.20 0.22 0.36 0.48 1.00 4 left RP4 0.16 0.16 0.18 0.16 0.22 0.22 0.20 0.26 0.30 0.42 0.48 0.90 4 left RP5 0.16 0.16 0.18 0.16 0.22 0.22 0.20 0.26 0.30 0.44 0.50 1.00 4 left SICO 0.16 0.24 0.24 0.24 0.24 0.24 0.28 0.32 0.24 0.42 0.42 0.78 4 left RB 0.16 0.30 0.46 0.58 0.70 0.88 0.92 0.98 0.98 1.00 1.00 - 4 left SCTS 0.16 0.42 0.72 0.92 1.00 1.00 1.00 1.00 1.00 1.00 1.00 -
8 left RP1 0.16 0.16 0.16 0.16 0.16 0.16 0.16 0.16 0.18 0.32 0.40 0.94 8 left RP2 0.16 0.16 0.16 0.16 0.18 0.22 0.20 0.26 0.32 0.40 0.50 0.88 8 left RP3 0.16 0.16 0.18 0.18 0.18 0.24 0.26 0.34 0.38 0.52 0.60 1.00 8 left RP4 0.16 0.16 0.18 0.18 0.20 0.24 0.30 0.38 0.46 0.56 0.66 0.94 8 left RP5 0.16 0.16 0.18 0.18 0.20 0.24 0.30 0.38 0.46 0.58 0.70 1.00 8 left SICO 0.16 0.30 0.32 0.32 0.30 0.32 0.34 0.36 0.32 0.40 0.46 0.84 8 left RB 0.16 0.42 0.56 0.66 0.82 0.94 0.96 0.98 1.00 1.00 1.00 - 8 left SCTS 0.16 0.54 0.80 0.92 1.00 1.00 1.00 1.00 1.00 1.00 1.00 -
1 self RP1 0.58 0.58 0.58 0.58 0.58 0.58 0.58 0.60 0.62 0.64 0.68 0.96 1 self RP2 0.58 0.58 0.58 0.58 0.58 0.58 0.62 0.60 0.64 0.66 0.74 0.84 1 self RP3 0.58 0.58 0.58 0.58 0.58 0.62 0.64 0.64 0.72 0.66 0.80 0.98 1 self RP4 0.58 0.58 0.58 0.58 0.58 0.64 0.64 0.64 0.76 0.72 0.78 0.94 1 self RP5 0.58 0.58 0.58 0.58 0.60 0.62 0.64 0.66 0.76 0.74 0.82 0.98 1 self SICO 0.58 0.62 0.62 0.64 0.64 0.64 0.64 0.64 0.62 0.70 0.64 0.90 1 self RB 0.58 0.64 0.74 0.84 0.88 0.86 0.88 0.90 0.90 0.92 0.94 - 1 self SCTS 0.58 0.80 0.92 0.96 1.00 1.00 1.00 1.00 1.00 1.00 1.00 -
2 self RP1 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.18 0.16 0.22 0.90 2 self RP2 0.12 0.12 0.12 0.12 0.12 0.14 0.16 0.20 0.24 0.22 0.24 0.82 2 self RP3 0.12 0.12 0.12 0.12 0.14 0.20 0.18 0.20 0.30 0.28 0.34 0.96 2 self RP4 0.12 0.12 0.12 0.12 0.16 0.18 0.20 0.24 0.30 0.30 0.40 0.88 2 self RP5 0.12 0.12 0.12 0.12 0.16 0.18 0.20 0.26 0.32 0.36 0.50 0.96 2 self SICO 0.12 0.20 0.18 0.18 0.20 0.22 0.28 0.24 0.18 0.30 0.26 0.54 2 self RB 0.12 0.24 0.38 0.40 0.52 0.58 0.72 0.80 0.86 0.92 0.98 - 2 self SCTS 0.12 0.44 0.60 0.72 0.88 0.96 1.00 1.00 1.00 1.00 1.00 -
4 self RP1 0.24 0.24 0.24 0.24 0.24 0.24 0.24 0.24 0.26 0.26 0.34 0.92 4 self RP2 0.24 0.24 0.24 0.24 0.24 0.26 0.24 0.24 0.28 0.28 0.36 0.84 4 self RP3 0.24 0.24 0.24 0.24 0.24 0.26 0.30 0.24 0.32 0.38 0.38 1.00 4 self RP4 0.24 0.24 0.24 0.24 0.24 0.24 0.30 0.32 0.30 0.42 0.42 0.96 4 self RP5 0.24 0.24 0.26 0.24 0.24 0.24 0.30 0.34 0.36 0.46 0.50 1.00 4 self SICO 0.24 0.26 0.26 0.28 0.30 0.30 0.34 0.34 0.28 0.34 0.48 0.78 4 self RB 0.24 0.32 0.40 0.52 0.56 0.62 0.78 0.88 0.94 0.92 0.92 - 4 self SCTS 0.24 0.38 0.68 0.82 0.98 1.00 1.00 1.00 1.00 1.00 1.00 -
8 self RP1 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.14 0.16 0.22 0.92 8 self RP2 0.10 0.10 0.10 0.10 0.12 0.10 0.12 0.14 0.18 0.26 0.28 0.80 8 self RP3 0.10 0.10 0.10 0.10 0.14 0.10 0.14 0.18 0.28 0.30 0.46 0.98 8 self RP4 0.10 0.10 0.10 0.10 0.14 0.16 0.14 0.22 0.38 0.38 0.46 0.92 8 self RP5 0.10 0.10 0.10 0.10 0.14 0.18 0.16 0.22 0.40 0.38 0.48 0.98 8 self SICO 0.10 0.18 0.18 0.16 0.18 0.20 0.24 0.20 0.18 0.22 0.30 0.80 8 self RB 0.10 0.26 0.34 0.54 0.56 0.70 0.76 0.90 0.90 0.94 0.94 - 8 self SCTS 0.10 0.36 0.66 0.88 0.96 0.98 0.98 0.98 0.98 0.98 0.98 -
Table 6: Semantic similarity for vicuna-7b-v1.5-16k, 50 samples, UMD watermarking. SCTS successfully preserves semantics, and the trend is consistent over different c and hashing methods.
c Hashing Method 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
1 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9987 0.9992 0.9928 0.8719 1 left RP2 1.0000 1.0000 0.9997 0.9987 0.9959 0.9964 0.9905 0.9908 0.9787 0.9780 0.8188 1 left RP3 1.0000 0.9998 0.9996 0.9981 0.9913 0.9878 0.9838 0.9803 0.9637 0.9559 0.7495 1 left RP4 1.0000 0.9998 0.9996 0.9976 0.9895 0.9811 0.9801 0.9674 0.9531 0.9355 0.6934 1 left RP5 1.0000 0.9998 0.9978 0.9973 0.9895 0.9809 0.9788 0.9645 0.9488 0.9198 0.6076 1 left SICO 0.9989 1.0000 0.9906 0.9800 0.9847 0.9933 0.9958 0.9878 0.9674 0.9831 0.7065 1 left RB 0.9890 0.9771 0.9639 0.9469 0.9337 0.9195 0.9054 0.8950 0.8863 0.8659 - 1 left SCTS 0.9885 0.9745 0.9579 0.9426 0.9228 0.9113 0.8961 0.8906 0.8906 0.8906 -
2 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9971 0.9976 0.9832 0.8875 2 left RP2 1.0000 1.0000 0.9999 0.9973 0.9986 0.9970 0.9967 0.9887 0.9840 0.9632 0.8508 2 left RP3 1.0000 0.9999 0.9991 0.9954 0.9949 0.9948 0.9884 0.9775 0.9686 0.9360 0.7496 2 left RP4 1.0000 0.9999 0.9977 0.9952 0.9914 0.9854 0.9858 0.9697 0.9464 0.9135 0.7114 2 left RP5 1.0000 0.9999 0.9977 0.9940 0.9914 0.9854 0.9854 0.9618 0.9392 0.9070 0.6375 2 left SICO 0.9996 0.9990 0.9914 0.9854 0.9818 0.9932 0.9870 0.9987 0.9773 0.9801 0.7282 2 left RB 0.9899 0.9786 0.9639 0.9509 0.9371 0.9231 0.9061 0.8898 0.8766 0.8629 - 2 left SCTS 0.9887 0.9734 0.9587 0.9414 0.9237 0.9077 0.8926 0.8912 0.8912 0.8912 -
4 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 0.9992 1.0000 0.9984 0.9945 0.9862 0.8981 4 left RP2 1.0000 0.9981 0.9995 0.9999 0.9986 0.9968 0.9955 0.9913 0.9827 0.9789 0.8256 4 left RP3 1.0000 0.9979 0.9982 0.9991 0.9940 0.9924 0.9919 0.9834 0.9737 0.9634 0.7331 4 left RP4 1.0000 0.9979 0.9981 0.9986 0.9924 0.9903 0.9884 0.9736 0.9645 0.9554 0.6564 4 left RP5 1.0000 0.9979 0.9978 0.9986 0.9922 0.9894 0.9881 0.9711 0.9586 0.9467 0.5436 4 left SICO 0.9984 0.9988 0.9906 0.9889 0.9884 0.9886 0.9934 0.9910 0.9845 0.9600 0.7104 4 left RB 0.9862 0.9728 0.9538 0.9394 0.9203 0.9008 0.8854 0.8732 0.8546 0.8546 - 4 left SCTS 0.9878 0.9743 0.9579 0.9366 0.9198 0.9018 0.8842 0.8832 0.8832 0.8832 -
8 left RP1 1.0000 1.0000 1.0000 0.9988 1.0000 0.9983 0.9980 0.9964 0.9840 0.9793 0.8792 8 left RP2 0.9989 0.9974 1.0000 0.9956 0.9978 0.9917 0.9865 0.9789 0.9714 0.9608 0.8153 8 left RP3 0.9989 0.9971 0.9994 0.9948 0.9898 0.9845 0.9740 0.9675 0.9553 0.9428 0.7462 8 left RP4 0.9982 0.9970 0.9985 0.9895 0.9897 0.9786 0.9664 0.9592 0.9447 0.9318 0.7082 8 left RP5 0.9982 0.9970 0.9985 0.9879 0.9863 0.9770 0.9657 0.9578 0.9348 0.9215 0.6324 8 left SICO 0.9958 0.9997 0.9871 0.9922 0.9801 0.9879 0.9809 0.9858 0.9697 0.9658 0.6553 8 left RB 0.9859 0.9688 0.9501 0.9343 0.9178 0.9043 0.8874 0.8698 0.8540 0.8540 - 8 left SCTS 0.9889 0.9697 0.9513 0.9329 0.9166 0.8970 0.8842 0.8842 0.8842 0.8842 -
1 self RP1 1.0000 1.0000 1.0000 1.0000 0.9999 0.9998 0.9982 0.9947 0.9952 0.9918 0.9154 1 self RP2 1.0000 0.9999 0.9972 0.9948 0.9970 0.9887 0.9848 0.9818 0.9797 0.9623 0.8838 1 self RP3 1.0000 0.9994 0.9954 0.9925 0.9923 0.9859 0.9764 0.9685 0.9601 0.9454 0.7861 1 self RP4 1.0000 0.9990 0.9920 0.9932 0.9900 0.9832 0.9748 0.9613 0.9494 0.9221 0.7580 1 self RP5 1.0000 0.9990 0.9919 0.9914 0.9900 0.9829 0.9690 0.9566 0.9463 0.9113 0.6610 1 self SICO 0.9993 0.9986 0.9904 0.9853 0.9745 0.9975 0.9899 0.9891 0.9756 0.9811 0.7040 1 self RB 0.9836 0.9692 0.9560 0.9425 0.9304 0.9165 0.9033 0.8893 0.8723 0.8531 - 1 self SCTS 0.9859 0.9700 0.9555 0.9415 0.9258 0.9117 0.8944 0.8788 0.8612 0.8430 -
2 self RP1 1.0000 1.0000 1.0000 1.0000 0.9992 1.0000 1.0000 0.9959 0.9950 0.9914 0.9019 2 self RP2 0.9999 0.9998 0.9997 0.9994 0.9976 0.9954 0.9897 0.9851 0.9771 0.9713 0.8937 2 self RP3 0.9998 0.9998 0.9996 0.9981 0.9957 0.9928 0.9815 0.9722 0.9520 0.9517 0.7993 2 self RP4 0.9998 0.9998 0.9996 0.9976 0.9938 0.9868 0.9774 0.9609 0.9391 0.9343 0.7701 2 self RP5 0.9998 0.9998 0.9995 0.9969 0.9918 0.9868 0.9720 0.9600 0.9287 0.9236 0.6896 2 self SICO 0.9980 0.9990 0.9964 0.9891 0.9743 0.9928 0.9777 0.9834 0.9649 0.9776 0.7597 2 self RB 0.9872 0.9732 0.9562 0.9392 0.9191 0.9028 0.8854 0.8739 0.8633 0.8501 - 2 self SCTS 0.9832 0.9621 0.9414 0.9209 0.9047 0.8852 0.8695 0.8687 0.8687 0.8687 -
4 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9927 0.9974 0.9921 0.8825 4 self RP2 0.9999 0.9998 0.9989 0.9969 0.9961 0.9937 0.9890 0.9828 0.9770 0.9749 0.7580 4 self RP3 0.9998 0.9996 0.9990 0.9938 0.9958 0.9871 0.9799 0.9758 0.9686 0.9638 0.6806 4 self RP4 0.9998 0.9992 0.9988 0.9916 0.9931 0.9845 0.9761 0.9643 0.9513 0.9537 0.6235 4 self RP5 0.9998 0.9987 0.9988 0.9916 0.9931 0.9841 0.9709 0.9582 0.9488 0.9416 0.5464 4 self SICO 0.9979 0.9999 0.9925 0.9880 0.9614 0.9869 0.9811 0.9857 0.9722 0.9674 0.6439 4 self RB 0.9847 0.9686 0.9507 0.9330 0.9141 0.8964 0.8801 0.8611 0.8459 0.8459 - 4 self SCTS 0.9850 0.9673 0.9499 0.9285 0.9105 0.8891 0.8754 0.8737 0.8732 0.8732 -
8 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9996 0.9970 0.9974 0.9944 0.9064 8 self RP2 1.0000 0.9995 0.9980 0.9953 0.9929 0.9978 0.9860 0.9816 0.9859 0.9661 0.8792 8 self RP3 0.9996 0.9995 0.9977 0.9928 0.9919 0.9902 0.9766 0.9741 0.9668 0.9464 0.7893 8 self RP4 0.9988 0.9995 0.9974 0.9921 0.9853 0.9884 0.9748 0.9626 0.9476 0.9335 0.7644 8 self RP5 0.9988 0.9995 0.9974 0.9921 0.9848 0.9862 0.9719 0.9603 0.9373 0.9198 0.6658 8 self SICO 0.9984 1.0000 0.9818 0.9924 0.9796 0.9848 0.9945 0.9688 0.9807 0.9749 0.7222 8 self RB 0.9900 0.9742 0.9593 0.9431 0.9273 0.9084 0.8917 0.8760 0.8608 0.8608 - 8 self SCTS 0.9861 0.9706 0.9534 0.9367 0.9197 0.9021 0.8837 0.8837 0.8837 0.8837 -
Method 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
RP1 0.62 0.62 0.62 0.62 0.62 0.62 0.62 0.62 0.64 0.64 0.64 1.00 RP2 0.62 0.62 0.62 0.62 0.62 0.62 0.66 0.62 0.66 0.64 0.70 0.92 RP3 0.62 0.62 0.62 0.62 0.62 0.66 0.68 0.66 0.68 0.66 0.72 1.00 RP4 0.62 0.62 0.62 0.62 0.62 0.64 0.68 0.66 0.72 0.74 0.78 0.96 RP5 0.62 0.62 0.62 0.62 0.62 0.64 0.68 0.68 0.76 0.74 0.78 0.98
SICO 0.62 0.66 0.64 0.62 0.66 0.64 0.70 0.66 0.74 0.68 0.80 0.92 RB 0.62 0.66 0.70 0.76 0.82 0.88 0.90 0.90 0.92 0.90 0.94 -
SCTS 0.62 0.76 0.90 0.94 1.00 1.00 1.00 1.00 1.00 1.00 1.00 -

heading: UMD.
text: Table 12: AUROC for Llama-2-7b-chat-hf, 10 samples, UMD watermarking. The trend is consistent compared with vicuna-7b-v1.5-16k over different c and hashing methods.
c Hashing Method 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
1 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 1.0000 0.7700 1 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.8900 1 left RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 1.0000 0.7800 1 left RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 0.9900 0.7700 1 left RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 0.9900 0.7100 1 left SICO 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9300 1 left RB 1.0000 1.0000 0.9700 0.9500 0.8500 0.8300 0.8100 0.8300 0.7900 0.7200 0.6100 - 1 left SCTS 1.0000 1.0000 0.9400 0.8800 0.8100 0.7600 0.6100 0.4900 0.4900 0.4900 0.4900 -
2 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.5800 2 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9600 0.9000 2 left RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9600 0.5300 2 left RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 0.9600 0.6700 2 left RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9200 0.6900 2 left SICO 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9300 2 left RB 1.0000 1.0000 0.9600 0.9500 0.9200 0.8700 0.8300 0.6700 0.5800 0.5600 0.5100 - 2 left SCTS 1.0000 1.0000 0.9000 0.7700 0.4700 0.3100 0.1500 0.1300 0.1300 0.1300 0.1300 -
4 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9500 0.8900 0.6800 4 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9500 0.9200 0.8700 4 left RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9100 0.9500 0.6300 4 left RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 0.9500 0.9200 0.7900 4 left RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9400 0.9000 0.4100 4 left SICO 1.0000 1.0000 1.0000 1.0000 0.9900 1.0000 0.9900 0.9900 1.0000 1.0000 0.9500 0.7900 4 left RB 1.0000 1.0000 1.0000 0.9900 0.9600 0.9300 0.8400 0.7700 0.6100 0.5800 0.5800 - 4 left SCTS 1.0000 0.9800 0.8900 0.7400 0.4600 0.1900 0.0800 0.1000 0.1000 0.1000 0.1000 -
8 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 1.0000 0.7200 8 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 0.9700 0.8667 8 left RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 1.0000 0.9600 0.9900 0.5556 8 left RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 1.0000 0.9500 0.9600 0.6889 8 left RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 1.0000 0.9700 0.9600 0.5667 8 left SICO 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9300 0.9700 0.8200 8 left RB 1.0000 0.9900 0.9600 0.9300 0.9000 0.8500 0.8200 0.7300 0.6700 0.6300 0.6300 - 8 left SCTS 1.0000 0.9000 0.8300 0.6900 0.4900 0.2100 0.1100 0.1100 0.1100 0.1100 0.1100 -
1 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.8700 1 self RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9300 1 self RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9300 1 self RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.8700 1 self RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.5500 1 self SICO 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 1.0000 0.8300 1 self RB 1.0000 1.0000 0.9800 0.9700 0.9900 0.9200 0.9100 0.8900 0.8500 0.8800 0.8300 - 1 self SCTS 1.0000 0.9600 0.9600 0.9000 0.7700 0.6300 0.5300 0.4200 0.3800 0.3000 0.2400 -
2 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 0.7500 2 self RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 0.9500 2 self RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 0.9600 0.7100 2 self RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 0.9700 0.7900 2 self RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9900 0.9400 0.7400 2 self SICO 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9300 2 self RB 1.0000 1.0000 0.9900 0.9800 0.9600 0.8900 0.8300 0.8000 0.7300 0.6400 0.5800 - 2 self SCTS 1.0000 1.0000 0.9700 0.8900 0.7200 0.5600 0.4700 0.3300 0.3300 0.3300 0.3300 -
4 self RP1 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9800 0.9500 0.9800 0.7600 4 self RP2 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9800 0.9500 0.9800 0.8300 4 self RP3 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9700 0.9600 0.9000 0.6700 4 self RP4 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9700 0.9600 0.9000 0.7200 4 self RP5 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9700 0.9500 0.8600 0.6800 4 self SICO 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9900 0.9800 0.9500 0.9100 4 self RB 0.9900 0.9900 0.9900 0.9600 0.8800 0.8800 0.8800 0.8600 0.7800 0.7500 0.7500 - 4 self SCTS 0.9900 0.9800 0.8900 0.7500 0.6300 0.4500 0.2500 0.2500 0.2000 0.2000 0.2000 -
8 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9700 1.0000 1.0000 0.6600 8 self RP2 1.0000 1.0000 0.9800 0.9700 0.9800 0.9900 0.9600 0.9900 0.9400 0.9400 0.9300 0.7000 8 self RP3 1.0000 1.0000 0.9800 0.9700 0.9800 0.9700 0.9600 0.9700 0.9100 0.8300 0.9100 0.5400 8 self RP4 1.0000 1.0000 0.9900 0.9700 0.9800 0.9700 0.9500 0.9500 0.8600 0.8200 0.8800 0.5400 8 self RP5 1.0000 1.0000 0.9900 0.9700 0.9800 0.9700 0.9500 0.9500 0.8500 0.8000 0.8700 0.5200 8 self SICO 1.0000 0.9800 0.9800 0.9800 0.9800 0.9800 0.9800 0.9500 0.9800 0.9700 0.9700 0.7100 8 self RB 1.0000 0.9500 0.9000 0.8300 0.7800 0.7200 0.6700 0.6700 0.6300 0.5700 0.5700 - 8 self SCTS 1.0000 0.8700 0.7000 0.6100 0.4100 0.1700 0.0900 0.0900 0.0900 0.0900 0.0900 -
Table 13: ASR for Llama-2-7b-chat-hf, 10 samples, UMD watermarking, zth=4. The trend is consistent compared with vicuna-7b-v1.5-16k over different c and hashing methods.
c Hashing Method 0 (Unattacked) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
1 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 1.0000 0.0000 1 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 0.9000 0.9000 0.9000 0.2000 1 left RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 0.8000 0.9000 0.8000 0.0000 1 left RP4 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 0.9000 0.8000 0.9000 0.6000 0.0000 1 left RP5 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 0.9000 0.8000 0.8000 0.6000 0.0000 1 left SICO 1.0000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.3000 1 left RB 1.0000 0.5000 0.4000 0.3000 0.2000 0.2000 0.1000 0.0000 0.0000 0.0000 0.0000 - 1 left SCTS 1.0000 0.5000 0.2000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
2 left RP1 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.8000 0.0000 2 left RP2 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.7000 0.6000 0.2000 2 left RP3 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.4000 0.6000 0.0000 2 left RP4 0.9000 0.9000 0.9000 0.9000 0.8000 0.8000 0.9000 0.9000 0.8000 0.4000 0.5000 0.2000 2 left RP5 0.9000 0.9000 0.9000 0.9000 0.8000 0.8000 0.9000 0.9000 0.8000 0.4000 0.4000 0.0000 2 left SICO 0.9000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.6000 0.8000 0.5000 2 left RB 0.9000 0.8000 0.6000 0.2000 0.2000 0.2000 0.2000 0.1000 0.0000 0.0000 0.0000 - 2 left SCTS 0.9000 0.6000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
4 left RP1 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.7000 0.6000 0.6000 0.4000 0.0000 4 left RP2 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.7000 0.6000 0.4000 0.3000 0.1000 4 left RP3 0.8000 0.8000 0.8000 0.8000 0.8000 0.7000 0.7000 0.7000 0.6000 0.4000 0.2000 0.0000 4 left RP4 0.8000 0.8000 0.8000 0.8000 0.8000 0.7000 0.7000 0.7000 0.6000 0.4000 0.2000 0.0000 4 left RP5 0.8000 0.8000 0.8000 0.8000 0.8000 0.6000 0.7000 0.7000 0.6000 0.4000 0.2000 0.0000 4 left SICO 0.8000 0.5000 0.5000 0.5000 0.5000 0.5000 0.4000 0.5000 0.4000 0.5000 0.4000 0.2000 4 left RB 0.8000 0.3000 0.3000 0.3000 0.1000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 - 4 left SCTS 0.8000 0.2000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
8 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 0.9000 0.8000 0.1000 8 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 0.9000 1.0000 1.0000 0.7000 0.8000 0.7000 0.1100 8 left RP3 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.8000 0.8000 0.8000 0.7000 0.0000 8 left RP4 1.0000 1.0000 0.9000 1.0000 1.0000 0.9000 0.9000 0.8000 0.6000 0.7000 0.5000 0.0000 8 left RP5 1.0000 1.0000 0.9000 1.0000 1.0000 0.9000 0.9000 0.8000 0.6000 0.7000 0.5000 0.0000 8 left SICO 1.0000 0.9000 0.8000 0.9000 0.9000 0.9000 0.9000 0.8000 0.8000 0.8000 0.7000 0.3000 8 left RB 1.0000 0.6000 0.5000 0.3000 0.1000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 - 8 left SCTS 1.0000 0.4000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
1 self RP1 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.3000 0.0000 1 self RP2 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.4000 0.3000 0.4000 0.4000 0.0000 1 self RP3 0.4000 0.4000 0.3000 0.4000 0.4000 0.4000 0.4000 0.3000 0.2000 0.4000 0.3000 0.0000 1 self RP4 0.4000 0.4000 0.3000 0.4000 0.4000 0.4000 0.4000 0.1000 0.2000 0.4000 0.4000 0.0000 1 self RP5 0.4000 0.4000 0.3000 0.4000 0.4000 0.4000 0.4000 0.1000 0.3000 0.4000 0.3000 0.0000 1 self SICO 0.4000 0.5000 0.5000 0.5000 0.5000 0.4000 0.4000 0.5000 0.4000 0.2000 0.3000 0.1000 1 self RB 0.4000 0.2000 0.2000 0.2000 0.1000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 - 1 self SCTS 0.4000 0.1000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
2 self RP1 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.8000 0.0000 2 self RP2 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.9000 0.8000 0.7000 0.8000 0.2000 2 self RP3 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.7000 0.7000 0.6000 0.4000 0.0000 2 self RP4 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.7000 0.8000 0.6000 0.6000 0.4000 0.0000 2 self RP5 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.7000 0.8000 0.6000 0.6000 0.4000 0.0000 2 self SICO 0.9000 0.8000 0.8000 0.8000 0.8000 0.8000 0.7000 0.7000 0.8000 0.7000 0.5000 0.2000 2 self RB 0.9000 0.5000 0.2000 0.2000 0.2000 0.1000 0.1000 0.0000 0.0000 0.0000 0.0000 - 2 self SCTS 0.9000 0.4000 0.2000 0.2000 0.2000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 -
4 self RP1 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.9000 0.8000 0.1000 4 self RP2 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.7000 0.7000 0.4000 0.0000 4 self RP3 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.9000 0.8000 0.5000 0.5000 0.4000 0.0000 4 self RP4 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.9000 0.8000 0.5000 0.5000 0.4000 0.0000 4 self RP5 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.9000 0.8000 0.5000 0.4000 0.4000 0.0000 4 self SICO 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.6000 0.6000 0.5000 4 self RB 0.9000 0.6000 0.4000 0.4000 0.3000 0.2000 0.2000 0.2000 0.1000 0.1000 0.1000 - 4 self SCTS 0.9000 0.5000 0.2000 0.1000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
8 self RP1 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.8000 0.9000 0.7000 0.0000 8 self RP2 0.9000 0.9000 0.8000 0.8000 0.8000 0.8000 0.7000 0.8000 0.4000 0.6000 0.4000 0.0000 8 self RP3 0.9000 0.9000 0.8000 0.8000 0.8000 0.8000 0.7000 0.7000 0.4000 0.3000 0.3000 0.0000 8 self RP4 0.9000 0.9000 0.8000 0.8000 0.8000 0.8000 0.7000 0.6000 0.4000 0.3000 0.3000 0.0000 8 self RP5 0.9000 0.9000 0.8000 0.8000 0.7000 0.8000 0.7000 0.6000 0.3000 0.2000 0.3000 0.0000 8 self SICO 0.9000 0.5000 0.5000 0.5000 0.5000 0.4000 0.4000 0.3000 0.5000 0.5000 0.4000 0.1000 8 self RB 0.9000 0.3000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 - 8 self SCTS 0.9000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -
Table 16: Semantic similarity for Llama-2-7b-chat-hf, 10 samples, UMD watermarking. The trend is consistent compared with vicuna-7b-v1.5-16k over different c and hashing methods.
c Hashing Method 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 (Unconstrained)
1 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9925 1.0000 0.8462 1 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9916 0.9887 0.9959 0.9896 0.7681 1 left RP3 1.0000 1.0000 1.0000 0.9954 1.0000 1.0000 0.9916 0.9790 0.9919 0.9789 0.6907 1 left RP4 1.0000 1.0000 1.0000 0.9954 0.9945 0.9983 0.9916 0.9671 0.9813 0.9523 0.6563 1 left RP5 1.0000 1.0000 1.0000 0.9954 0.9945 0.9983 0.9916 0.9671 0.9738 0.9303 0.4560 1 left SICO 0.9988 0.9934 0.9960 0.9850 1.0000 1.0000 0.9932 1.0000 0.9696 0.9682 0.7528 1 left RB 0.9894 0.9792 0.9660 0.9511 0.9415 0.9315 0.9148 0.8932 0.8994 0.8809 - 1 left SCTS 0.9903 0.9727 0.9558 0.9353 0.9136 0.8999 0.8916 0.8916 0.8916 0.8916 -
2 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9943 0.9930 0.9069 2 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 0.9798 0.9982 0.9900 0.9741 0.9669 0.8994 2 left RP3 1.0000 1.0000 1.0000 0.9963 0.9933 0.9792 0.9902 0.9739 0.9562 0.9492 0.8323 2 left RP4 1.0000 1.0000 1.0000 0.9959 0.9941 0.9848 0.9881 0.9742 0.9592 0.9352 0.8028 2 left RP5 1.0000 1.0000 1.0000 0.9959 0.9941 0.9773 0.9828 0.9681 0.9498 0.9014 0.6752 2 left SICO 0.9981 0.9974 0.9929 0.9695 0.9879 0.9972 1.0000 0.9877 0.9733 0.4321 0.8733 2 left RB 0.9896 0.9759 0.9577 0.9443 0.9332 0.9229 0.9088 0.8966 0.8829 0.8632 - 2 left SCTS 0.9881 0.9740 0.9558 0.9427 0.9310 0.9204 0.9173 0.9173 0.9173 0.9173 -
4 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 0.9967 0.9943 0.9900 0.9814 0.9666 0.8900 4 left RP2 1.0000 1.0000 1.0000 1.0000 1.0000 0.9870 0.9842 0.9900 0.9712 0.9441 0.8323 4 left RP3 1.0000 1.0000 1.0000 1.0000 0.9951 0.9715 0.9842 0.9900 0.9697 0.9397 0.6899 4 left RP4 1.0000 1.0000 1.0000 1.0000 0.9931 0.9715 0.9790 0.9882 0.9681 0.9161 0.6418 4 left RP5 1.0000 1.0000 1.0000 1.0000 0.9916 0.9715 0.9772 0.9912 0.9663 0.8877 0.5192 4 left SICO 0.9997 1.0000 1.0000 0.9835 0.9547 0.9900 0.9863 0.9774 0.9885 0.9657 0.6946 4 left RB 0.9843 0.9703 0.9579 0.9425 0.9285 0.9071 0.9013 0.8830 0.8630 0.8630 - 4 left SCTS 0.9847 0.9725 0.9551 0.9430 0.9222 0.8984 0.8726 0.8726 0.8726 0.8726 -
8 left RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9946 0.9856 0.9893 0.9243 8 left RP2 1.0000 1.0000 0.9960 1.0000 0.9982 0.9958 0.9896 0.9716 0.9773 0.9676 0.8236 8 left RP3 1.0000 1.0000 0.9960 0.9955 0.9984 0.9879 0.9845 0.9710 0.9697 0.9591 0.7269 8 left RP4 1.0000 0.9995 0.9960 0.9918 0.9893 0.9876 0.9854 0.9602 0.9576 0.9214 0.7133 8 left RP5 1.0000 0.9995 0.9960 0.9918 0.9875 0.9876 0.9854 0.9602 0.9511 0.9232 0.6308 8 left SICO 0.9932 0.9993 0.9967 0.9686 0.9964 0.9959 0.9995 0.9952 0.9732 0.9892 0.7148 8 left RB 0.9837 0.9724 0.9613 0.9473 0.9332 0.9172 0.8987 0.8815 0.8705 0.8705 - 8 left SCTS 0.9890 0.9699 0.9556 0.9368 0.9108 0.8912 0.8912 0.8912 0.8912 0.8912 -
1 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9936 0.8943 1 self RP2 1.0000 1.0000 1.0000 0.9989 0.9874 0.9996 0.9958 0.9988 0.9873 0.9869 0.8921 1 self RP3 1.0000 0.9989 0.9997 0.9988 0.9869 0.9925 0.9828 0.9854 0.9866 0.9741 0.8119 1 self RP4 1.0000 0.9989 0.9998 0.9989 0.9887 0.9930 0.9607 0.9750 0.9820 0.9753 0.7244 1 self RP5 1.0000 0.9989 0.9998 0.9985 0.9881 0.9913 0.9601 0.9712 0.9555 0.9660 0.6341 1 self SICO 0.9943 1.0000 0.9752 0.9943 0.9917 0.9892 1.0000 0.9912 0.9380 0.9796 0.8004 1 self RB 0.9902 0.9745 0.9631 0.9500 0.9390 0.9252 0.9082 0.8912 0.8817 0.8605 - 1 self SCTS 0.9868 0.9735 0.9589 0.9433 0.9304 0.9126 0.8907 0.8738 0.8529 0.8383 -
2 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9930 0.9873 0.9100 2 self RP2 1.0000 0.9967 1.0000 1.0000 1.0000 0.9866 0.9860 0.9930 0.9699 0.9787 0.9045 2 self RP3 1.0000 0.9967 1.0000 0.9951 1.0000 0.9776 0.9610 0.9780 0.9578 0.9214 0.8391 2 self RP4 1.0000 0.9967 1.0000 0.9951 1.0000 0.9663 0.9611 0.9622 0.9510 0.9207 0.7726 2 self RP5 1.0000 0.9967 0.9966 0.9951 0.9908 0.9669 0.9632 0.9498 0.9431 0.8934 0.7188 2 self SICO 0.9992 1.0000 0.9908 0.9828 1.0000 0.9878 0.9979 0.9997 0.9798 0.9876 0.8152 2 self RB 0.9867 0.9719 0.9545 0.9404 0.9250 0.9117 0.9004 0.8936 0.8802 0.8554 - 2 self SCTS 0.9821 0.9606 0.9446 0.9270 0.9040 0.8851 0.8807 0.8807 0.8807 0.8807 -
4 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9933 0.9977 0.9918 0.8954 4 self RP2 1.0000 0.9979 1.0000 1.0000 0.9972 1.0000 0.9894 0.9922 0.9762 0.9521 0.8938 4 self RP3 1.0000 0.9979 1.0000 1.0000 0.9953 1.0000 0.9894 0.9741 0.9653 0.9389 0.7950 4 self RP4 1.0000 0.9979 1.0000 1.0000 0.9953 0.9977 0.9894 0.9667 0.9245 0.9313 0.7871 4 self RP5 1.0000 0.9979 1.0000 1.0000 0.9953 0.9977 0.9894 0.9651 0.8967 0.9282 0.7494 4 self SICO 0.9968 0.9969 0.9942 0.9826 0.9722 0.9952 0.9913 0.9983 0.9888 0.9648 0.8156 4 self RB 0.9901 0.9754 0.9654 0.9497 0.9353 0.9102 0.8965 0.8728 0.8636 0.8636 - 4 self SCTS 0.9879 0.9763 0.9590 0.9330 0.9147 0.8999 0.8975 0.8932 0.8932 0.8932 -
8 self RP1 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9973 1.0000 0.9812 0.9279 8 self RP2 1.0000 0.9957 0.9965 0.9960 0.9955 0.9874 0.9955 0.9822 0.9760 0.9455 0.9102 8 self RP3 1.0000 0.9957 0.9965 0.9946 0.9862 0.9883 0.9902 0.9719 0.9428 0.9318 0.8043 8 self RP4 1.0000 0.9952 0.9965 0.9923 0.9862 0.9874 0.9788 0.9444 0.9397 0.9201 0.6657 8 self RP5 1.0000 0.9952 0.9965 0.9886 0.9862 0.9874 0.9788 0.9478 0.9220 0.9171 0.6358 8 self SICO 0.9997 0.9973 0.9929 0.9837 0.9781 0.9939 0.9530 1.0000 0.9778 0.9931 0.8032 8 self RB 0.9910 0.9667 0.9500 0.9296 0.9153 0.9056 0.8976 0.8744 0.8577 0.8577 - 8 self SCTS 0.9809 0.9609 0.9292 0.9160 0.8993 0.8864 0.8864 0.8864 0.8864 0.8864 -

heading: Future Work
text: Previous phrases should have no influence on future phrases: includes; contains; includes; contains; contains; includes; contains; contains; contains; contains; contains; contains; contains; contains; includes; contains; contains; includes; contains; contains;