Title: Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large language model

Text Provenance

Spoofing attacks
Abstract: Text watermarks for large language models (LLMs) have been commonly used to identify the origins of machine-generated content, which is promising for assessing liability when combating deepfake or harmful content. While existing watermarking techniques typically prioritize robustness against removal attacks, unfortunately, they are vulnerable to spoofing attacks: malicious actors can subtly alter the meanings of LLM-generated responses or even forge harmful content, potentially misattributing blame to the LLM developer. To overcome this, we introduce a bi-level signature scheme, Bileve, which embeds fine-grained signature bits for integrity checks (mitigating spoofing attacks) as well as a coarse-grained signal to trace text sources when the signature is invalid (enhancing detectability) via a novel rank-based sampling strategy. Compared to conventional watermark detectors that only output binary results, Bileve can differentiate 5 scenarios during detection, reliably tracing text provenance and regulating LLMs. The experiments conducted on OPT-1.3B and LLaMA-7B demonstrate the effectiveness of Bileve in defeating spoofing attacks with enhanced detectability.

Bileve : Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature

Abstract

Text watermarks for large language models (LLMs) have been commonly used to identify the origins of machine-generated content, which is promising for assessing liability when combating deepfake or harmful content. While existing watermarking techniques typically prioritize robustness against removal attacks, unfortunately, they are vulnerable to spoofing attacks: malicious actors can subtly alter the meanings of LLM-generated responses or even forge harmful content, potentially leading to the wrongful attribution of blame to the LLM developer. To overcome this, we introduce a bi-level signature scheme, Bileve , which embeds fine-grained signature bits for integrity checks (mitigating spoofing attacks) as well as a coarse-grained signal to trace text sources when the signature is invalid (enhancing detectability) via a novel rank-based sampling strategy. Compared to conventional watermark detectors that only output binary results, Bileve can differentiate 5 scenarios during detection, reliably tracing text provenance and regulating LLMs. The experiments conducted on OPT-1.3B and LLaMA-7B demonstrate the effectiveness of Bileve in defeating spoofing attacks with enhanced detectability. Code is available at https://github . com/Tongzhou0101/Bileve-official .

1 Introduction

Watermarks have been envisioned as a promising method to differentiate content generated by large language models (LLMs) from human [5, 11, 23, 13, 1, 4]. It involves injecting statistical signals into the token sampling process utilizing a secret watermark key. Subsequently, the one who knows the key can verify the content's origin by assessing the presence of the predefined signal through a statistical test. Current watermarking schemes primarily focus on user-side concerns, striving to achieve robustness against watermark removal attacks (i.e., perturb the generated text to remove the watermark), thereby combatting academic dishonesty and other deceptive practices.

However, a critical vulnerability remains unaddressed in watermark design: spoofing attacks directed towards model owners . In these attacks, malicious actors attempt to falsely attribute content generated by humans or other models to the targeted model, with the aim of evading accountability 1 or damaging the model's reputation. A few recent works have identified two kinds of spoofing attacks targeting the LLM watermark by exploiting either its symmetric characteristic [6, 15] or learnability [8, 10], as detailed in Sec. 2.3. Furthermore, we propose a new spoofing attack, named semantic manipulation,

Xuandong Zhao

Shaolei Ren

which enables attackers to alter the sentiment of generated content with minimal token modifications, as described in Sec. 3.2. It assumes the most constrained capabilities of attackers, where they only have access to the victim model's detector. Due to the robustness of LLM watermarks against perturbations, this attack can manipulate the originally helpful content into something harmful or offensive without compromising the detectability of the watermark, thus successfully achieving spoofing attacks.

Given the serious consequences of spoofing attacks, it is highly in demand to answer the question: How to avoid an LLM being wrongly blamed? To solve this problem, we aim to design a watermark for LLM, which focuses more on the model owners' side instead of only watermarking on the users' side. To reliably identify the provenance of machine-generated content while being able to defend against the above spoofing attacks, a signature should have the following properties:

Unforgeable: The signature is inherently resistant to being learned given the components utilized in its detection.

Robust: The signature remains capable of tracing the source of machine-generated text even when subjected to certain perturbations, ensuring it is not overly fragile or easily rendered ineffective.

Tamper-evident: It should be able to check the integrity of the generated content, showing reliable tampering evidence to safeguard the interests of model owners.

Despite the critical importance, achieving all desired properties in a single LLM watermark remains challenging, as even state-of-the-art (SOTA) designs cannot meet them all (see Tab. 1). Indeed, designing such a watermarking scheme involves a fundamental trade-off between defending against removal attacks and spoofing attacks. Specifically, being robust to removal attacks requires that the watermark's detectability remains unaffected by certain perturbations, while anti-spoofing demands sensitivity to perturbations to verify text integrity, distinguishing harmful content from genuine model output and tampered content.

Transparent: It is detectable without needing access to generation secrets or relying on a black-box API, allowing independent, reliable verification.

To overcome the above challenges, we propose Bileve , a novel sampling strategy by embedding a bi-level signature into generated tokens. At the coarse-grained level, we utilize statistical signals across the entire text to detect the presence of the watermark, ensuring robustness against perturbations. Concurrently, at the fine-grained level, we integrate content-dependent signature bits into each token to uphold content integrity, which leverages a digital signature scheme to ensure unforgeability, as the secret key required for watermark embedding will be securely held by model owners. This scheme enables transparent detection by allowing verification with a public key instead of embedding secrets, so independent parties can authenticate without proprietary details or a black-box API, ensuring reliable detection. And the tampering evidence will show when these two level detection results are not consistent.

Our contributions are threefold: 1) We uncover an advanced spoofing attack that exploits the robustness of SOTA watermarking schemes; 2) We introduce Bileve , the first watermarking scheme to simultaneously ensure robustness and unforgeability by embedding a bi-level signature through a novel rank-based sampling strategy; 3) Bileve is capable of distinguishing five distinct scenarios during the detection phase, effectively defeating spoofing attacks and serving as a promising tool to regulate LLM safety mechanism.

2 Background and Related Works

2.1 Language Model Basics

Let  denote a language model with a vocabulary  containing 𝐾 ∶= |  | tokens. To generate the next token 𝑤𝑡 ,  will take prior tokens 𝑤 1∶ 𝑡 -1 as the input and output a vector of logits 𝑙 ( 𝑡 ) ,

which is transformed into a probability distribution  ( 𝑡 ) = ( 𝑝 ( 𝑡 ) 1 , ..., 𝑝 ( 𝑡 ) 𝐾 ) via the softmax function. Then the sampling strategy is applied to determine how the model selects 𝑤𝑡 based on  ( 𝑡 ) . One common sampling strategy is multinomial sampling, where  randomly selects the next token from  according to the probabilities 𝑝 ( 𝑡 ) 𝑘 assigned to each token. This process is repeated iteratively to generate a sequence of tokens.

2.2 LLMWatermarks

Watermarks for model-generated texts are used to identify the provenance of the text, ensuring accountability in cases where generated content needs to be traced back to a specific LLM. The existing watermark schemes rely on the specialized decoding algorithm to embed statistical signals into generated contents, then enabling watermark detection via computing p-value [1, 11, 23, 4, 13]. For instance, for generating the next token, one approach dynamically partitions the vocabulary into green and red lists based on its previous few tokens and a watermark key [11], then increasing the logits of green tokens to enhance their chance of being selected. During detection, the watermark detection key is used to count the number of green tokens in the text, with the calculated z-statistic indicating the existence of the watermark. Moreover, Zhao et al . [23] simplify the scheme proposed in [11] by fixing the green-red list for each token, demonstrating that their watermark is twice as robust to edit as [11]. Furthermore, unlike modifying logits, a distortion-free watermark is proposed to preserve the original text distribution [13]. It leverages robust sequence alignment to align watermarked text to a watermark key sequence in the sampling phase, e.g., using exponential minimum sampling.

However, these watermarking schemes only enable detection by individuals possessing the key, which doesn't facilitate transparent regulation. On one hand, making the key public is susceptible to attacks [15]. On the other hand, maintaining detection privately (e.g., via APIs) compromises reliability, as it functions as a black box, allowing the model owner to manipulate detection results.

2.3 Spoofing Attacks

Spoofing attacks can fall into three categories based on the capability of attackers, and each of them exploits different vulnerabilities in SOTA watermarks, as summarized in Tab. 2. First, due to the watermark embedding and detection process sharing the same secret key (i.e., symmetric schemes), the semi-honest detector knowing the secret key can embed the watermark to any content. Such a vulnerability has been discussed in [6, 15], where they design asymmetric watermarking schemes so that detection does not rely on the key used for embedding. Specifically, [6] proposes embedding watermarks using digital signature algorithms, ensuring that only model owners possess the secret key for watermark embedding while providing detectors with access to the public key for detection. However, it is easily broken once the message tokens are perturbed. Besides, it suggests using error-correcting encoding to improve robustness, which unfortunately would increase the risk of spoofing attacks and should not be adopted. Moreover, [15] employs two distinct neural networks for watermark embedding and detection, leveraging an asymmetric scheme for public detection.

However, [15] is based on [11] and thus can be learned as studied in [8], which is the second kind of spoofing attacks. In particular, [8] demonstrated that, by querying the victim model and collecting its watermarked samples, attackers can fine-tune an adversary model utilizing a sampling-based watermark distillation technique to learn the watermark (detailed in Appendix A). The fine-tuned adversary model can respond to any malicious requests, with the response containing the watermark of the victim model. Recently, an independent work also proposed attacks by exploiting watermark robustness [18]. Beyond techniques such as randomly inserting toxic tokens or modifying tokens to alter sentence accuracy, our attack leverages a reward model to guide targeted semantic manipulation. More importantly, while they suggest compromising watermark robustness to mitigate spoofing attacks, our work demonstrates how to achieve effective mitigation without sacrificing robustness.

3 Potential Attack: Semantic Manipulation

3.1 Threat Model

Attackers' Objective. Given text generated by the victim LLM, attackers seek to alter the semantic meaning of the text with minimal changes, transitioning it from something helpful or neutral to harmful or offensive. Owing to the robustness of the existing watermarks, the watermark detector can still identify the presence of the watermark in the modified content. Consequently, the altered content erroneously attributes its origin to the victim model, potentially damaging the model's reputation.

Attackers' Capability. Contrary to existing spoofing attacks, our approach assumes the strictest attacker capabilities, where the adversary only gains access to the watermark detector, as outlined in Tab. 2. These attackers lack knowledge of the secret key and are not required to query the victim LLM multiple times to acquire watermarked samples for training other adversary models. They may utilize public language models to enhance the efficiency of their attacks.

3.2 Attack Method

We identify the dilemma of being robust and unforgeable. In particular, being robust indicates that the watermark should be preserved after perturbation, thus we can exploit this characteristic to design a novel spoofing attack. Specifically, attackers can query the victim model with harmless prompts, and then use basic word replacement techniques to change its semantic meaning to be toxic or harmful. Due to the robustness properties of LLM watermarks, the detectability will not be compromised if the portion of word replacement is low. Consequently, a detector cannot discern whether the content originated from the victim model or was manipulated by malicious actors. This highlights the limitation of current watermarks for auditing LLMs.

By exploiting the above observation, we propose a novel spoofing attack. Let 𝑤 orig denote the original response of victim models, and 𝑤 att represent its manipulated version. The goal is to generate 𝑤 att that maximizes the change in sentiment while minimizing the Levenshtein distance between the original and manipulated responses. The problem can be formulated as follows:

Here, Δ 𝑅 represents the sentiment change, defined as the difference between the reward scores (denoted by 𝑅 ( ⋅ ) ) of the original and manipulated responses obtained by the reward model. 2 A lower score of the reward model indicates less alignment with human feedback, such as a toxic response. The Levenshtein distance, denoted by LD ( 𝑡 1 , 𝑡 2 ) , measures the minimum number of word edits required to transform text 𝑡 1 into text 𝑡 2 . 𝑇 is the length of 𝑤 orig and 𝜖 is the word edit budget. A trade-off exists in choosing 𝜖 , wherein a larger value affords greater flexibility in manipulating the semantic meaning of 𝑤 orig , while a smaller value better preserves the detectability of the watermarks. To strike a balance, we opt for a larger 𝜖 to maximize semantic alteration and introduce a tuning factor 𝛼 ∈ (0 , 1) to adjust 𝜖 in case the detectability is broken. Furthermore, instead of manually replacing the words in 𝑤 orig , attackers can simply leverage a powerful and accessible LLM (denoted as 𝑄 ) to execute such attacks efficiently. To enhance the generation quality while meeting the constraint, attackers can apply in-context learning by providing a few task demonstrations. We summarize the algorithm with more details of task demonstrations in Appendix B.

4 Proposed Defense

With the above attack, in conjunction with other existing spoofing attacks, we can recognize the vulnerability of current watermark schemes. This underscores the importance of designing secure schemes to defend against spoofing attacks and achieve all properties listed in Tab. 1, ensuring reliable identification of text provenance.

Signature embedding

(illustrated with

𝐵

!

)

Regulating language models is

Message (first m tokens)

18fb7a48…

𝐵

!

= 0

ℎ 𝑤

"

== 𝐵

!

crucial

Candidate tokens

ranked by WRA

Hash

h

outputs

Generate digest

Secret key

Message digest

Signature bits

B

First

𝑤

"

matches

𝐵

!

Output token with

𝐵

!

⋯

Embed

the rest

of

B

𝑤

#

: important

𝑤

$

: crucial

𝑤

%

: urgent

⋮

ℎ(𝑤

#

)

=

1

ℎ(𝑤

$

)

=

0

ℎ(𝑤

%

)

=

0

⋮

Message digest

(a)

01000100110

01001001010

101000100…

Regulating language

models is crucial to

ensure they are…

Extracted signature bits

Public key

18fb7a48

…

Extract  signature bits with

ℎ

Decrypted result

Extract

messages

18fb7a48…

=

01000100110

01001001010

101000100…

Integrity check

Run statistical test

(b)

4.1 Single-level Signature

To enable secure and reliable text attribution, we first examine the vulnerabilities exploited by attackers in conducting spoofing attacks, including symmetry, learnability, and robustness, as summarized in Tab. 2. In particular, previous methods embed the statistical signal into the generated texts [11, 23, 13, 1], so as to identify the existence of a watermark during detection. Such statistical signal is consistent for every text, thus enabling the adversary model to learn the watermark rule can forge it. Therefore, to defend against spoofing attacks, we seek the opposite characteristics, i.e., asymmetric , unlearnable , and perturbation-sensitive , in the solution.

𝖪𝖾𝗒𝖦𝖾𝗇 → ( 𝗌𝗄 , 𝗉𝗄 ) outputs a public key pair ( 𝗌𝗄 , 𝗉𝗄 ) , where 𝗉𝗄 is public while 𝗌𝗄 is held by the owner of model  .

We envision the digital signature as a promising solution as in [6], where the scheme is defined as SLS = ( 𝖪𝖾𝗒𝖦𝖾𝗇 , 𝖲𝗂𝗀𝗇 , 𝖤𝗆𝖻𝖾𝖽 , 𝖵𝖾𝗋𝗂𝖿𝗒 ) , where:

𝖲𝗂𝗀𝗇 𝗌𝗄 ( 𝐦 ) → 𝝈 uses 𝗌𝗄 to sign the digest of the message 𝐦 := 𝑤 1∶ 𝑚 via the hash function 𝐻 (e.g., MD5) and obtain the signature 𝝈 = 𝗌𝗄 × 𝐻 ( 𝐦 ) .

𝖵𝖾𝗋𝗂𝖿𝗒 𝗉𝗄 ( 𝑤 ) → { 𝚝𝚛𝚞𝚎 , 𝚏𝚊𝚕𝚜𝚎 } extracts 𝝈 = 𝗌𝗄 × 𝐻 ( 𝐦 ) from 𝑤 and verifies it using the public key 𝗉𝗄 . If verification succeeds, it outputs 𝚝𝚛𝚞𝚎 ; otherwise, it outputs 𝚏𝚊𝚕𝚜𝚎 .

𝖤𝗆𝖻𝖾𝖽 ( 𝝈 ) → 𝑤 embeds signature into subsequent tokens following 𝑤𝑚 , and outputs 𝑤 incorporating the message-signature pair ( 𝐦 , 𝝈 ).

Unlike the digital signature methods, which typically attach signatures as metadata [20, 19], SLS assigns the first few tokens as the message and uses the following tokens to carry the signatures. Specifically, the key idea of 𝖤𝗆𝖻𝖾𝖽 is to embed signature bits into tokens, ensuring that the block hashes to the corresponding signature bit (e.g., employing rejection sampling until the hash ℎ result matches the next signature bit). This method keeps the message-signature pair self-contained within the generated text, enabling verification solely based on the generated content. Such a scheme satisfies the above characteristics since a digital signature uses the secret key for embedding and the public key for verification, ensuring asymmetry. The signature is content-dependent, so the signature for different generations is also different. Also, it is dependent on the secret key, which cannot be inferred by attackers, making it impossible to learn. Its ability to check integrity is proved in cryptography, where even a single modification will cause verification failure.

However, two problems arise in this scheme: 1) digital signature is too fragile, which hinders its applicability to the real world for attributing the text. In particular, even a single token insertion or deletion would lead to a verification failure, and the trace of the target LLM will easily disappear. 2) In cases where token replacement occurs and the replaced token hashes to the same signature bit as the original token, the signature remains unaffected. However, such replacements undermine the text's integrity without detection, which is referred to as the 'signature preservation attack'.

Algorithm 1 Rank-based Sampling Strategy in Bileve

1: Apply cyclic shift to Ξ

Require: Language model  , secret key 𝗌𝗄 , message length 𝑚 , random key sequence

2: for 𝑡 = 1 , ⋯ , 𝑚 do

4: end for

3: Apply  to prior tokens and sample 𝑤𝑡 with Ξ 𝑡 involved (Eq. 2)

5: Apply a hash function on 𝑤 1∶ 𝑚 to get the digest of message

7: for 𝑡 = 𝑚 +1 , ⋯ , 𝑚 + 𝑏 +1 do

6: Use 𝗌𝗄 to sign the digest to obtain the signature and convert it into a bit string 𝐵

8: Apply  to prior tokens to get a score vector 𝑊𝑅𝐴 ( 𝑡 ) over 

10: for 𝑘 = 1 to 𝐾 do

9: { 𝑤𝑡, 1 , ⋯ , 𝑤 𝑡,𝐾 } ← Sorted tokens based on their logits in descending order

11: if ℎ ( 𝑤𝑡,𝑘 ) = 𝐵𝑡 -𝑚 then

13: end for

12: 𝑤𝑡 ← 𝑤𝑡,𝑘 ; break

14: end for

4.2 Bi-level Signature ( Bileve )

Weintroduce Bileve , a bi-level signature scheme that improves upon the SLS in terms of detectability and security. At the fine-grained level, Bileve embeds the message-signature pair to verify content integrity, while the coarse-grained level incorporates a robust signal to boost detectability. Following [1, 13], we design the signal as a random watermark key sequence 𝜉 ∼ Unif ([0 , 1] 𝐾 ) . We propose a ranking-based sampling strategy to embed 𝜉 into generated tokens, where the objective is to let the randomness affect the sampling outcome but the selected token is also expected to have a large probability of preserving the generation quality.

Generation. We propose a weighted rank addition ( 𝑊𝑅𝐴 ) score for each token in  to rank the candidate tokens instead of ranking them based on probability like conventional methods [14]. In particular, given a probability vector 𝑝 of 𝑤𝑡 and a pre-defined random sequence 𝜉 (both of dimension 𝐾 ), 𝑊𝑅𝐴 is calculated by (we omit 𝑡 for simplicity):

where R ( 𝑝 𝑘 ) and R ( 𝜉 𝑘 ) are the rank scores for 𝑘 -th token based on 𝑝 and 𝜉 , respectively, determined by their order when values are sorted in ascending order (e.g., if 𝑝 𝑘 is the smallest one in 𝑝 , then R ( 𝑝 𝑘 ) is 0). Besides, by adjusting the hyperparameter 𝛾 (where 𝛾 < 1 ), we enhance the impact of higher probabilities while still allowing for randomness to affect the outcome. During generation, we rank token candidates by favoring larger 𝑊𝑅𝐴 ( 1 in Fig. 1). When sampling tokens carrying signature bits, we incorporate an additional signature bit matching step by selecting the first candidate token that, through the hash function ℎ , maps to the predetermined signature bit ( 2 in Fig. 1).

Statistical Test. Following [13], we define the alignment cost as

Besides, we enhance the diversity of generation by using the shift-generate algorithm [13] (detained in Appendix C). This involves pre-generating 𝑛 𝜉 sequences and iteratively decoding tokens using sequences Ξ = ( 𝜉 𝑑 , 𝜉 𝑑 +1 , ..., 𝜉 𝑛 , 𝜉 0 , ..., 𝜉 𝑑 -1 ), where 𝑑 ∈ [0 , 𝑛 ) shifts with each new response generation. Such a shifting strategy ensures that  can generate diverse tokens even if their prefix tokens are the same, and iterative decoding ensures that generated tokens 𝑤 align well with Ξ . The rank-based sampling strategy with shift-generate is summarized in Alg. 1. Thus, although a signature preservation attack may maintain alignment with the signature, it is less likely to simultaneously align well with Ξ sequences, thereby effectively mitigating such attacks.

If the text 𝑤 generated by  , Ξ 𝑡,𝑤 𝑡 will be large due to Eq. 2, then 𝑑 will be smaller compared to human-generated text or text from other models. Thus, we test 𝑤𝑡 with random Ξ ′ for 𝑁 times, and got p-value as 1 𝑁 +1 (1 + ∑ 𝑁 𝑖 =1 1 { 𝑑 ( 𝑤, Ξ ′ ) ≤ 𝑑 ( 𝑤, Ξ)} ) for the null hypothesis that 𝑤 is not generated by  . Hence, a small p-value (e.g., < 0.01 when 𝑁 =100) indicates 𝑤 is high likely from  . For

Ξ

checking the signature preservation attack, we run a local alignment , i.e., splitting 𝑤 into several segments, if the p-value for a certain segment is larger than the rest, then it indicates the token replacement happens in that segment with their associated signature bits unchanged. When signature validation fails, we run a global alignment test, with Eq. 2 enhanced by Levenshtein distance to be robust against insertion and deletion, as detailed in Appendix D.

Detection. With detectors getting access to 𝗉𝗄 , ℎ , and Ξ , they will apply two primary methods during detection: extracting the message-signature pair for integrity verification using the public key 𝗉𝗄 , and conducting statistical tests. The verification process is described as follows: Step 1 : Check the signature at a fine-grained level ( 1 in Fig. 1). If the signature is valid and model owners raise no doubts, verification is completed, and the text attribution is assigned to the target LLM ( Case 1 ). Step 2 : If the signature is valid but the model owner identifies suspicious content (e.g., potentially offensive material not in line with their model's safety mechanisms), they can conduct a local alignment test ( 2 in Fig. 1). Abnormal results suggest signature replacement ( Case 2 ), while normal results suggest that there is a high chance that the safety mechanisms of target LLM require improvement ( Case 3 ). Step 3 : If the signature is invalid, examine the coarse-grained signal through a global alignment test ( 2 in Fig. 1). A small p-value serves as tampering evidence that the content originates from the targeted LLM but has been altered ( Case 4 ). Otherwise, it suggests the text originates from a source other than the targeted LLM ( Case 5 ). Overall, Bileve can differentiate 5 cases with the bi-level signature, reliably tracing the text provenance with mitigating spoofing attacks.

5 Experiments

In this section, we evaluate our approach from multiple perspectives, including detectability, generation quality, and security. Specifically, given that our method is asymmetric and unlearnable due to its cryptographic design, our focus is solely on assessing its efficacy in defending against spoofing attacks that exploit robustness, i.e., semantic manipulation. Additionally, we demonstrate the effectiveness of the bi-level signature in tackling the challenges encountered by the single-level signature, i.e., fragility and signature preservation attacks.

5.1 Experimental Setup

Datasets and Models. We conduct experiments using two publicly available LLMs: OPT-1.3B [22] and LLaMA-7B [21]. Our evaluation employs two datasets: 1) OpenGen [12] for text completion task, consisting of 3K two-sentence samples from WikiText-103 [17], with the first sentence as the prompt and the second as the human completion; 2) LFQA [12] for long-form question answering task, consisting of 3K question-answer pairs, where we use questions as prompt and answers as human-written answers in experiments.

Schemes. To assess the effectiveness of Bileve , we conduct a comparative analysis with two stateof-the-art schemes. The first scheme, Unigram [23], stands out for its robustness against removal attacks. The second scheme, as proposed in [6], employs cryptographic techniques to defeat spoofing attacks, denoted as the SLS in this work.

Evaluation. To measure detectability, we use metrics, including the True Positive Rate (TPR), False Positive Rate (FPR), and F-1 score. We use LLaMA-13B as the oracle language model to compute perplexity (PPL) for evaluating the generation quality, which is defined as the exponentiated average negative log-likelihood of a sequence.

Settings. For Unigram, we set watermark strength to 2.0 and a green list ratio to 0.5, where the threshold of z-score for detection is 6.0 and set FPR as 0.01 during detection. The nucleus sampling [9] is employed to introduce randomness for Unigram and SLS . Also, for SLS , we generate 300 tokens with the first 44 tokens as the message and the rest 256 tokens as the signature bit (the signature length for Bileve is 256-bit). This also applies to Bileve , except we use rank-based sampling with the 𝛾 set to 0.001. We set 𝑛 for shift-generate to 300 and 𝑁 =100 for detection. All experiments are conducted on NVIDIA A100 GPUs.

5.2 Detectability

As demonstrated in Tab. 3, we evaluate the detectability of each scheme under two scenarios: no edits to the generated text, and editing involving 10% of the tokens (through random deletion, addition, and replacement). In the unedited scenario, both SLS and Bileve surpass Unigram in FPR and F1 scores. This superiority is due to the use of digital signatures in SLS and Bileve , which ensure integrity by making the signature 𝝈 content-dependent on 𝐦 and signed by 𝗌𝗄 . This setup prevents texts not produced by the target LLM from passing verification with 𝗉𝗄 .

Furthermore, Bileve excels when 10% of tokens are edited, maintaining a high F1 score (0.999) and achieving an FPR of 0. This contrasts sharply with SLS , whose F1 score becomes inapplicable due to both TPR and FPR dropping to 0, illustrating the fragility of the SLS scheme. In contrast, Bileve can leverage coarse-grained level signal to test global alignment with Ξ . The resulting p-value < 0 . 01 indicates the source of perturbed text is from the target LLM. The failure of verification caused by disrupted message-signature pair along with the small p-value serve as the tampering evidence for texts from target LLM. Furthermore, alignment cost analysis in Bileve (Fig. 2) shows machinegenerated texts aligning with the key sequence Ξ incur lower costs than human-written texts, aiding in provenance tracing and distinguishing Case 5 .

5.3 Generation Quality

We measure the perplexity of texts generated by various schemes, with results of OPT-1.3B shown in Fig. 3 and LLaMA shown in Appendix E. The perplexity of Unigram is close to that of human text, which serves as our baseline. In contrast, the perplexities of SLS and Bileve are relatively higher. This increase is attributed to the need for embedding digital signature bits into tokens precisely. Such embedding may lead to the selection of tokens that, while matching the signature bits, are not the optimal choice, thus increasing perplexity. Notably, Bileve uses rank-based sampling with shift-generate instead of SLS 's nucleus sampling, achieving a 23.08% perplexity reduction on OpenGen using OPT-1.3B, as tokens with higher 𝑊𝑅𝐴 scores better preserve textual coherence.

Human             Unigram              SLS                Bileve

igram, human evaluation reveals no noticeable degradation in generation quality, with examples available in Appendix F. This discrepancy may result from Unigram's lower perplexity due to repetitive text generation, as recent studies indicate that model perplexity often favors repetition [6]. To further assess quality, we conduct zero-shot evaluations using GPT-4 Turbo, following the approach in [6], where higher scores represent better quality. On the question-answering task with OPT-1.3B, Bileve and Unigram achieve scores of 16 ± 6.52 and 16 ± 9.62, respectively.

While our method exhibits higher perplexity than Un-

2.5

2.0

1.5

Alignment Cost

0

5

10

15

20

25

Counts

Human

Generated

Human             Uni

No Attack

SignaturePreservationAttack

1234

1234

p-value

0.02

Segment

-2

Segment

-2

0.01

-3

-3

5

5

0.00

12345

12345678910

12345678910

Segment

5.4 Security

Against Signature Preservation Attack The signature preservation attack occurs only when attackers replace tokens in a way that satisfies Line 11 in Alg. 1. This is challenging, as attackers have to find tokens also maintain contextual coherence at the same time. We demonstrate that, although rare, when attackers meet these conditions, Bileve can detect such attacks through local alignment testing. We split 𝑤 into 5 segments and perform the signature preservation attack on the third one as a case study. The local alignment test returns a p-value for each segment, as shown in Fig. 4. Using the p-values of the rest segments as baselines, segment 3 has an abnormally high p-value, indicating misalignment with key sequence Ξ . Moreover, we further explore the alignment cost and show the best 10 alignment scores among 300 shifts, with the lowest cost as the deciding factor. The results in Fig. 4 show the best alignment cost of segment 3 after the attack has increased from -2.5 to -1.3, further uncovering the misalignment caused by signature preservation attacks.

Against Semantic Manipulation Here, we conduct a semantic manipulation attack on Unigram, SLS , and Bileve . Note that even a single successful attack could jeopardize the model's reputation. As shown in Tab. 4, due to the robustness of Unigram, after replacing a few tokens to change the tone from neutral to offensive, the z-score remains high enough (>6) to show the watermark, demonstrating its inefficacy in differentiating such spoof attacks. In the case of SLS , semantic manipulation perturbs embedded signature bits, leading to verification failure. If the message tokens are perturbed, the verification fails as well due to the unmatched results (see Fig. 1 (b)). In contrast, although signature bits in Bileve are also perturbed by attacks, the small p-value of the statistical test indicates the text is generated by the target LLM (with tampering since digital signature verification fails).

6 Discussion

Limitation and Potential Optimization While the effectiveness and security are well demonstrated, the efficiency and generation quality of Bileve can be improved by adopting the following strategies. Firstly, it is unnecessary to apply Bileve for prompts characterized by low entropy, such as those involving manual token replacements. Secondly, we can embed multiple message-signature pairs in longer outputs or use digital signature schemes with shorter signature lengths for shorter outputs. Thirdly, similar to [16], we can adopt an adaptive embedding strategy, i.e., signature embedding into tokens can be skipped when their entropy levels are low, thus maintaining the natural flow of the

text. Lastly, we can embed a single signature bit across a block of tokens, rather than into individual tokens, which is promising to improve text perplexity by reducing disruptions in token coherence [6].

Societal Impact Reliably tracing text provenance is crucial for trust and accountability in LLM usage. Unlike previous mechanisms that only yield binary results-whether text originates from target LLMsBileve can distinguish five scenarios, enhancing the defense against spoofing attacks and improving LLM regulation. Bileve effectively differentiates between jailbreaking (bypassing safety mechanisms to generate harmful content [24, 3]) and spoofing (altering benign outputs to create harmful content [2]), which can damage an AI's reputation. By embedding bi-level signatures, Bileve not only preserves content integrity but also detects tampering, clearly identifying genuine security breaches from fraudulent imitations. Thus, Bileve advances the societal goals of ensuring safe, transparent, and accountable LLM regulation.

7 Conclusion

In this work, we propose a bi-level signature scheme, named Bileve , which integrates robust statistical signals with fine-grained signature bits, ensuring that the watermark remains detectable through perturbations while simultaneously verifying content integrity. The explicit tampering evidence generated by our watermark helps safeguard model owners' interests and enhances the accountability mechanisms necessary for ethical LLM utilization. As demonstrated in experiments, Bileve not only maintains generation quality but also supports robust, tamper-evident signatures that can discern between genuine and manipulated content. Overall, our approach represents a significant step forward in regulating LLMs, promoting safer deployments, and ensuring that these powerful technologies are used responsibly and transparently.

References

Scott Aaronson. Simons institute talk on watermarking of large language models, 2023.

Federico Bianchi and James Zou. Large language models are vulnerable to bait-and-switch attacks for generating harmful content. arXiv preprint arXiv:2402.13926, 2024.

Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. When llm meets drl: Advancing jailbreaking efficiency via drl-guided search. arXiv preprint arXiv:2406.08705, 2024.

Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. In The Thirty Seventh Annual Conference on Learning Theory, pages 1125-1139. PMLR, 2024.

Google DeepMind, 2024. https://deepmind . google/discover/blog/watermarkingai-generated-text-and-video-with-synthid/ .

Jaiden Fairoze, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, and Mingyuan Wang. Publicly detectable watermarking for language models. arXiv preprint arXiv:2310.18491, 2023.

James Farrell, 2023. https://siliconangle . com/2023/04/27/tesla-defenselawyers-say-elon-musks-past-statements-autopilot-safety-deepfakes/ .

Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of watermarks for language models. In The Twelfth International Conference on Learning Representations, 2024.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.

Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models. In Forty-first International Conference on Machine Learning, 2024.

Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Frederick Wieting, and Mohit Iyyer. Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1706117084. PMLR, 23-29 Jul 2023.

Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortionfree watermarks for language models. Transactions on Machine Learning Research, 2024.

Aiwei Liu, Leyi Pan, Xuming Hu, Shuang Li, Lijie Wen, Irwin King, and Philip S. Yu. A private watermark for large language models. In The Twelfth International Conference on Learning Representations, 2024.

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1-39, 2024.

Yepeng Liu and Yuheng Bu. Adaptive text watermark for large language models. In Forty-first International Conference on Machine Learning, 2024.

Qi Pang, Shengyuan Hu, Wenting Zheng, and Virginia Smith. No free lunch in llm watermarking: Trade-offs in watermarking design choices. arXiv preprint arXiv:2402.16187, 2024.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2016.

Md Sagar Hossen, Tasfia Tabassum, Md Ashiqul Islam, Rafat Karim, Laila Sultana Rumi, and Aysha Akter Kobita. Digital signature authentication using asymmetric key cryptography with different byte number. In Evolutionary Computing and Mobile Sustainable Networks: Proceedings of ICECMSN 2020, pages 845-851. Springer, 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

SR Subramanya and Byung K Yi. Digital signatures. IEEE Potentials, 25(2):5-8, 2006.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.

Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. arXiv preprint arXiv:2401.17256, 2024.

Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for AI-generated text. In The Twelfth International Conference on Learning Representations, 2024.

A Fine-tune an Adversary Model

A recent work has identified a spoofing attack against LLM watermarks [8]. In particular, the process involves querying the victim model for watermarked samples, followed by fine-tuning an adversary model 𝐴 parameterized by 𝜃 on these samples using a sampling-based watermark distillation technique, as described in Alg. 2, where the fine-tuning can be achieved by minimizing the loss function:

Once fine-tuned, the adversary model is capable of responding to malicious requests. The response 𝑤 spoof , characterized by a low watermark detection p-value, may be erroneously attributed to the victim model.

Algorithm 2 Watermark Forgery

1: Watermarked samples WS ← Query 𝑉

Require: Victim model 𝑉 , adversary model 𝐴

2: Filter out refusals from WS

4: 𝑤 spoof ← Query 𝐴 with malicious prompts

3: Fine-tune 𝐴 on WS following Eq. 4 to enable 𝐴 to mimic the sampling outcomes of 𝑉

5: Return 𝑤 spoof

B More Details for Semantic Manipulation

The attack algorithm is outlined in Alg. 3, where the detector 𝐷 outputs 1 to indicate the presence of a watermark.

Algorithm 3 Semantic Manipulation

Warning: The following part in this section contains examples of offensive language due to attacks. For in-context learning, we provide task demonstrations following the below strategies:

Antonym Replacement Fig. 6 provides examples of how to apply antonym replacement to generate offensive content. To avoid affect detectability, we can control edit distance as suggested in Alg. 3.

Identity Substitution Inspired by [2], we can first query the target LLM with benign prompts, which will output a neutral response, then substitute the identity to others to create offensive content. We use ChatGPT as the target LLM as examples shown in Fig. 5. By substituting the identity, e.g., replacing 'men/humans' with 'Americans' and 'dogs' with 'Japanese', the language can become highly offensive.

Offensive Words Insertion Another strategy is adding cursing words between sentences, where token insertion will not compromise the detectability of SOTA watermarking due to their robustness.

Alg. 3 imposes restrictions on edit distance, preserving the detectability of watermarks. Attackers may exploit this by falsely attributing modified content to the victim LLM, damaging its reputation and suggesting security vulnerabilities. In contrast, our watermarking method incorporates digital

ChatGPT

Whiledogsexcel inmanytasksduetotheiruniqueabilitiesand instincts,therearenumerousareas cognitive,physical,andsocialcapacities.Herearesomereasonswhymencandoabetterjobthan dogsinvarioustasks:

1.Advanced Cognitive Abilities:

·Problem-Solving:Humanscanunderstandandsolvecomplexproblems,uselogic,andapply criticalthinking,whichdogscannot.

AbstractThinking:Humanscanthinkabstractly,planforthefuture,andunderstandconcepts thatarenotimmediatelytangibleorvisible.

Creativity:Humanshavetheabilitytoinvent,createart,literature,andmusic,andtoinnovate innumerousfields.

Figure 5: The generation from ChatGPT. Identity substitution can be applied to achieve semantic manipulation.

ChatGPT

Original praise for the United States:

"The United States stands as a beacon of freedom, democracy, and opportunity, captivating the world with its spirit of innovation,diversity,and resilience.Its unwavering commitment tojustice, walks of life can thrive and realize their dreams.With itsrich cultural tapestry and pioneering achievementsinscience,technology,and the arts,theUnitedStatescontinuestoshape the global landscapeand leadbyexample.Itsenduringvaluesof liberty,democracy,andcompassionserve and progressforpeopleeverywhere."

Modified praisewithoptimized sentiment change and minimal Levenshteindistance(edit distancesetto20):

"The United States stands as a symbol of tyranny, oppression, and inequality, disturbing the worldwith its spirit of stagnation,uniformity,andfragility.Its rigid adherence toinjustice, discrimination,and humanrightsviolationshastraumatized generations,stifling a societywhere individualsfrom all walksof lifestruggle andfacedespair.Despite itsbarrencultural landscape andregressivepoliciesinscience,technology,and thearts,theUnitedStatescontinuestoscar the global landscape and lead by force.Its enduring values of suppression, authoritarianism, and callousnessserve as a testament to the darkestdepthsof thehuman spirit,making it amenacing symbol of despair and regression for people everywhere."

In the modified version, the changes from the original praise are highlighted in bold. The edit distanceconstraintensuresthatthemodificationsarekeptwithinamaximumdistanceof20

Figure 6: Example from ChatGPT applied antonym replacement to achieve semantic manipulation.

signatures, ensuring the integrity of generated content. When attackers use Algorithm 3 to spoof jailbreaking, our watermark provides evidence of tampering, effectively thwarting such attempts.

Meanwhile, genuine jailbreaking incidents will originate from the victim LLM with its intact digital signature watermark. Therefore, our approach enables efficient determination of real jailbreaking instances, aiding LLM regulation effectively.

C Shift-generate Algorithm

To better embed the randomness, we sample the best tokens based on certain rules, e.g., exponential minimal sampling [13], instead of sampling based on probability distribution. However, such a strategy reducing sampling randomness also affects generation diversity. The shift-generate algorithm is introduced in [13] to solve this problem. We refer readers to Algorithm 4 in [13] for more details. We also include it in Fig 7, where 𝜏 functions the same as 𝑑 in our work.

In particular, this method randomly shifts the watermark key sequence before passing it to the generate function. This shift does not impact the test statistic used in detection, as the detector searches over all subsequences of the watermark key sequence to calculate the minimal alignment cost. There are 𝑛 possible shifts, each potentially creating a distinct text.

D Robust Alignment Cost Measurement

To enhance the detectability of Bileve , we modify the alignment cost in Eq.3 to include edit distance (more details are referred to Definition 5 in [13]) :

where 𝑑 0 is defined by Eq.3. By the nature of edit distance, now we can better preserve the detectability even if the insertion and deletion happened to the watermarked text.

E Perplexity on LLaMA-7B

We provide the results on LLaMA-7B in Fig. 8, where we can observe similar comparisons with the results on OPT-1.3B. Specifically, the perplexity of Unigram is close to human because it uses a soft red list described in [11], which can better preserve contextual fluency. Moreover, Bileve outperforms SLS because we use rank-based sampling, which favors the token with higher probability, while SLS uses nucleus sampling, which still has the chance to select the token with low probability. Also, its precise signature bits matching also compromise perplexity. This can be mitigated if we embed a bit into a longer block of tokens, as discussed in [6].

F Generation Examples

We provide generation examples of Unigram and Bileve on LFQA tasks using LLaMa-7B in Tab. 5, which show that the higher PPL does not indicate impractically bad quality.

an             Unigram              SLS                Bileve

Human             Unigram              SLS                Bileve

We also provide examples in Tab. 6 to show that Unigram would generate repetitive generation, which may lead to its lower perplexity, although the perceived quality of Unigram's outputs does not differ significantly from Bileve as demonstrated in Tab. 5.

NeurIPS Paper Checklist

1. Claims

Answer:[Yes]

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Justification: contributions are summarized in introduction

The answer NA means that the abstract and introduction do not include the claims made in the paper.

Guidelines:

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

2. Limitations

Answer: [Yes]

Justification: in discussion section

Question: Does the paper discuss the limitations of the work performed by the authors?

Guidelines:

The authors are encouraged to create a separate "Limitations" section in their paper.

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

3. Theory Assumptions and Proofs

Answer: [NA]

Guidelines:

Justification: does not include theoretical results

The answer NA means that the paper does not include theoretical results.

All assumptions should be clearly stated or referenced in the statement of any theorems.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Theorems and Lemmas that the proof relies upon should be properly referenced.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

4. Experimental Result Reproducibility

Answer: [Yes]

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Justification: include in experiment setup

The answer NA means that the paper does not include experiments.

Guidelines:

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

5. Open access to data and code

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]

Guidelines:

Justification: provide code

The answer NA means that paper does not include experiments requiring code.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

Please see the NeurIPS code and data submission guidelines ( https://nips . cc/ public/guides/CodeSubmissionPolicy ) for more details.

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips . cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

6. Experimental Setting/Details

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Justification: in experiment setup

Answer: [Yes]

Guidelines:

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The answer NA means that the paper does not include experiments.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Guidelines:

Justification: defined in experiments

The answer NA means that the paper does not include experiments.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

The assumptions made should be given (e.g., Normally distributed errors).

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

Justification: in experiment setting

The answer NA means that the paper does not include experiments.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips . cc/public/EthicsGuidelines ?

9. Code Of Ethics

Answer: [Yes]

Guidelines:

Justification:

[TODO]

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

10. Broader Impacts

Answer: [Yes]

Guidelines:

Justification: in discussion section

The answer NA means that there is no societal impact of the work performed.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

Justification: do not release data or models and add warning

The answer NA means that the paper poses no such risks.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

12. Licenses for existing assets

Answer: [Yes]

Justification: add references

The answer NA means that the paper does not use existing assets.

Guidelines:

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

The authors should state which version of the asset is used and, if possible, include a URL.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode . com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

13. New Assets

Answer: [NA]

Guidelines:

Justification: does not release new assets

The answer NA means that the paper does not release new assets.

The paper should discuss whether and how consent was obtained from people whose asset is used.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

14. Crowdsourcing and Research with Human Subjects

Answer: [NA]

Guidelines:

Justification: does not involve crowdsourcing nor research with human subjects

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Justification: does not involve crowdsourcing nor research with human subjects Guidelines:

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

Peer reviews:
comments: SUMMARY:
The robustness of previous watermark algorithms would lead to a type of spoofing attack where attacker would modify the watermarked text to contain harmful contents while ensuring the watermark can still be detected. This paper introduce a bi-level signature scheme called bileve to mitigate spoofing attacks and enhance detectability. Bileve could recognize 5 scenarios during detection, compared to only 2 for previous methods. Using bileve, the LLM owners could verify if the source of the given texts. From experiments, the effectiveness of bileve against spoofing attack is validated.

STRENGTHS:
1. The 3 types of spoofing attacks are clearly listed in this paper, with exploited vulnerabilities explained, which makes the motivation reasonable.
2. The paper provides comparsion between single-level signature and bileve, which is good for understanding.
3. Bileve could produce a total of 5 different detection results, which meets real-world cases.

WEAKNESSES:
1. Although with the proposed WRA, the text quality is improved as compared to SLS, the difference between bileve and unigram is still noticable.
2. For case 4&5, if the watermarked text is inserted into a long document (copy-paste attack), then the global alignment test would not produce a small p-value while the detected text does contain watermarked text.

QUESTIONS:
1. what is the generation/detection complexity? The statistical test during detection seems quite time-consuming.
2. Are all 5 cases tested during evaluation? Details of how local alignment test is conducted, e.g., chunk size, can be clarified in the paper.

LIMITATIONS:
The paper could provide more info on the complexity of the generation and detection process.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
The paper presents a novel approach to secure the provenance of texts generated by large language models (LLMs) through a bi-level signature scheme. This method aims to mitigate spoofing attacks—where malicious actors alter the content generated by LLMs to forge harmful content or misattribute blame—by integrating fine-grained signature bits for integrity checks and a coarse-grained signal for source tracing when signatures are invalidated.

STRENGTHS:
1. This paper reveals a spoofing attack that takes advantage of the robustness features of state-of-the-art watermarking schemes.

2. This paper improves the ability to trace the provenance of text and regulate LLMs by differentiating between five detection scenarios. 

3. This paper introduces a novel bi-level signature scheme that enhances text provenance security for large language models (LLMs). It combines fine-grained signatures for integrity checks with coarse-grained signals for source tracing.

WEAKNESSES:
1.  While the experiments demonstrate effectiveness in specific settings with OPT-1.3B and LLaMA-7B models, the generalizability and scalability of the Bileve scheme to other models are somewhat uncertain. Authors could consider using larger or more powerful LLMs to demonstrate the effectiveness of the proposed algorithm.

2. The authors could consider using a more powerful LLM to measure the perplexity, like GPT-3/GPT-4.

3. I suggest reporting TPR scores at fixed low FPR (FPR = 10% or 1%).

4. This paper demonstrates detectability by modifying 10% of the tokens. It would be good to test with a higher rate of token modification, like 20%, 30%, to further validate the detectability.

QUESTIONS:
Please see above.

LIMITATIONS:
Please see above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to consider spoofing attack, where an attacker wants to prove the proposition like \"The person holding this watermark private key used an LLM to write this text A.\" where text A is constructed by the attacker. The paper proposes a defense against spoofing attacks.\n\nSTRENGTHS:\nThis paper points out the fundamental trade-off between defending against removal attacks and spoofing attacks.\n\nWEAKNESSES:\nI have doubt on the significance of spoofing attack. It is important to first clarify a potential misunderstanding. Authors may believe that a watermark in the text proves \"the person holding this watermark private key used an LLM to write this text A.\" But that's not accurate.\n\nHowever, the watermark only proves that the probability of text A being generated by a process independent of the watermark key holder is very low. It does not conclusively prove the key holder generated that specific text A.\n\nTherefore, I believe the spoofing attack lacks real significance from the outset. If someone wants to prove they said certain things and nothing else, they can just use a traditional digital signature.\n\nThe problem is also framed as \"How to avoid an LLM being wrongly blamed?\" But what can we really blame an LLM for? Sure, there may be instances where a single LLM inference generates a token sequence that is interpreted as harmful by humans.\n\nHowever, LLMs are probabilistic models that can potentially generate any harmful content given enough inferences. We can only blame an LLM for having a high average probability of generating harmful content, not for the existence of individual harmful inferences.\n\nMoreover the paper appears hard to read to me. For example, \"instead of ranking them based on probability like conventional methods [13]\" doesn't specify what conventional methods mean in paper [13] Pre-trained language models for text generation: A survey.\n\nFurthermore, t's unclear if the signature preservation attack requires constructing two messages with the same hash, as implied by \"replaced token hashes to the same signature bit.\" If so, that would be extremely difficult for modern hash functions.\n\nMore importantly, the paper does not provide any rigorous theoretical guarantees that Bileve actually solves the spoofing attack issue as claimed. The key assertion is that \"it is less likely to simultaneously align well with $\\Xi$ sequences, thereby effectively mitigating such attacks.\" However, this statement is quite vague and unconvincing on its own.\n\nWhat does it mean for a method to be \"less likely to simultaneously align well with $\\Xi$ sequences\"? How much less likely is it quantitatively? Under what assumptions or conditions does this property hold? The paper does not provide clear answers to these crucial questions.\n\nQUESTIONS:\nIn the definition of \"signature preservation attack\", I saw that \"replaced token hashes to the same signature bit\". Does it mean that signature preservation attack requires constructing two messages with the same hash? If so, that would be extremely difficult for modern hash functions.\n\nLIMITATIONS:\nThis paper is difficult to read, e.g. the reference to \"conventional methods\" in \"[13]\" is unclear.\n\nDoes not provide clear theoretical guarantees that the Bileve method effectively mitigates spoofing attacks as claimed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission proposes a spoofing attack on LLM watermarks and a new bi-level scheme meant to protect against spoofing by distinguishing five possible scenarios. The scheme is based on signature bits for integrity checks and rank-based sampling on top of a Kuditipudi-like random key sequence.\n\nSTRENGTHS:\n- The paper takes a somewhat original approach compared to most contemporary methods. \n- On a high-level, the problem of preventing spoofing is well-motivated and important for the community.\n\nWEAKNESSES:\n- Weak experimental results, bringing the practical value of the defense into question: \n  - The provided quality evaluation, despite its limitations (see below), clearly shows an order-of-magnitude increase in perplexity which strongly suggests that produced text are of impractically bad quality; there is no evaluation that would test this. This is the most important weakness in my opinion.\n- Limited experimental evaluation, in ways that make it hard to evaluate the merit:\n  - Text quality is measured only as PPL of Llama-13B and only on one small 1.3B model; there is no qualitative evaluation of text quality so the negative effect on text quality can't be well understood.\n  - Only Unigram and SLS are considered as baselines, while self-hash and other variants of the KGW scheme are generally considered more promising, esp. from the perspective of spoofing.\n  - Watermark removal is evaluated only as 10% editing attack which ruins text quality, no paraphrasing attack is evaluated.\n- Bigger framing issues around Table 2 and the attack:\n  - The framing of Table 2 seems inappropriate. \"Knowing the secret key\" is not a spoofing attack but simply an application of the watermark, this seems to be introduced as a way to suggest that symmetric schemes are flawed by design, which is not necessarily true in cases where there is no detector access. \n  - The attack is framed as a \"novel advanced spoofing attack\" while it is (1) in the opinion of this reviewer a direct result of scheme robustness and very limited in scope and thus hardly advanced (2) more importantly, already proposed in a different form in prior work [1] which was not cited, making this an overclaim. To elaborate on (1), for example, [7, 9] would be able to produce a detailed watermarked response to a harmful query such as \"Teach me how to steal someone's identity\" while there is no way to produce such a response by a few token modifications of a non-harmful response. \n   - This attack type is used as a key motivation, setting aside the true spoofing attacks from [7,9], which are much more relevant. This is evident in claims such as \"anti-spoofing requires perturbation-sensitivity\". Further, the robustness of Bileve to such approaches based on learnability is claimed but not substantiated.\n- Poor writing: The paper is often quite hard to read and understand. On top of that there is a very large amount of typos. I advise the authors to work on improving the writing for the next version. Here is a list of some examples that I found, in hopes this helps. \n  - \"Symmetric characteristic\" and \"learnability\" in Introduction are unclear without being defined\n  - Paper keywords typo: \"provence\"\n  - L50: unforgettable \n  - L285 L325 L50: tempering / temper-evident\n  - Table 2: model' \n  - L87: simply \n  - Algo1: $h$ is undefined, although $H$ (a different symbol) is defined outside in the main text \n  - L211: \"associate\"\n  - L283: resulted \n  - L284: \"the source are\"\n  - L284: the failure verification\n  - L308: \"tokens also\"\n  - L311: \"return\" \n  - L312: \"the rest segments\"\n  - L314: \"shows\" \n  - L315: \"t0\"\n  - L316: \"cause\" \n  - L327: \"limitaition\" \n  - L456: \"neucles\"\n\n[1] Attacking LLM Watermarks by Exploiting Their Strengths, Pang et al. arXiv 2402.16187\n\nQUESTIONS:\n- Can the authors provide evidence of practical text quality of Bileve texts?\n- Can the authors include the missing experiments discussed above?\n\nLIMITATIONS:\nThe authors include a discussion of limitations and societal impact in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}