Learning to Watermark LLM-generated Text via Reinforcement Learning

## Abstract

Abstract We study how to watermark LLM outputs, i.e. embedding algorithmically detectable signals into LLM-generated text to track misuse. Unlike the current mainstream methods that work with a fixed LLM, we expand the watermark design space by including the LLM tuning stage in the watermark pipeline.
While prior works focus on token -level watermark that embeds signals into the output , we design a model -level watermark that embeds signals into the LLM weights , and
such signals can be detected by a paired detector.
We propose a co-training framework based on reinforcement learning that iteratively (1) trains a detector to detect the generated watermarked text and (2) tunes the LLM to generate text easily detectable by the detector while keeping its normal utility. We empirically show that our watermarks are more accurate, robust, and adaptable (to new attacks). It also allows watermarked model open-sourcing. In addition, if used together with alignment, the extra overhead introduced is low – only training an extra reward model (i.e. our detector). We hope our work can bring more effort into studying a broader watermark design that is not limited to working with a fixed LLM. We open-source the code: https://github.com/xiaojunxu/learning-to-watermark-llm .

## 1 Introduction

Watermarking LLM (Large Language Model) outputs, i.e., embedding algorithmically detectable signals into LLM-generated text, has recently become a potential solution to track LLM misuse (Kirchenbauer et al., ; Kuditipudi et al., ).

So far, LLM watermarking methods focus on token-level distortion in the LLM output. This framework has several limitations. (1) Since we still need the watermarked text to be humanly readable, the output distortion induced needs to be minimized. As a result, watermark accuracy might be suboptimal because the watermark signal injected in the output space is constrained by the readability tradeoff. (2) For the same reason, the limited output distortion leads to vulnerability to paraphrasing attacks (Kirchenbauer et al., ). (3) The design space of watermark is inflexible – all the practitioners can do is post-processing the generated text from a fixed LLM, which leads to certain problems, e.g. lack of adaptability to newly discovered adversarial attacks. (4) It forbids practitioners from open-sourcing the watermarked LLMs. If they want to do so, they would also have to release the unwatermarked LLM because the watermarks are added post hoc, defeating the original purpose of protecting intellectual property.

In this work, we ask: Can we watermark LLM texts by directly finetuning the LLM, so that we can enlarge the watermark design space? The watermark in our case is injected by model-level changes, and the resulting LLM outputs carry the signals that can be identified by detection.

In other words, we include the LLM tuning stage into the watermark pipeline as opposed to the prior methods that only work with a fixed LLM, and thus expand the design space of watermark. Unlike prior works which detectors are simple statistical tests, our detector is a language model that predicts the likelihood of whether a text is watermarked or not. Specifically, we tune the LLM to inject the watermark signal while training a paired detector model that detects the signal. The key insight is: by tuning the LLM to adapt to the detector, we make the detection easier and more accurate.

Figure: Figure 1: Overview of our framework compared to the prior works. Left: The prior methods (Kirchenbauer et al., ; Kuditipudi et al., ) focus on working with a fixed model. They induce distortions into the LLM output distribution used as the detection signal. Right: Our approach injects watermark into the LLM weights by finetuning. The watermark is propagated to the output and detected by a paired detector co-trained with the LLM in an RLHF framework, where a reward model can serve as the detector.
Refer to caption: x1.png

Figure 1 (right) shows the overview of our reinforcement learning-based watermark framework. We iteratively co-train both the LLM and the detector. In each step, we instruction-tune the LLM to distort its weights and therefore its output distribution. Then, we train the detector to detect the signal from the distorted outputs.

We choose reinforcement learning (Ouyang et al., ; Arulkumaran et al., ) as the co-training framework for several reasons. (1) We can adapt the reward model as a detector. (2) We can leverage the strong generalizability of the RL algorithm (Ouyang et al., ) to make sure the finetuned LLM can generate text that is easily detectable by the detector. (3) We still need to preserve the text readability in general, which can be done by RLHF’s utility-preserving objective.

Our approach has several advantages compared to the prior works. (1) Detection Accuracy: Since we tune the LLM to fit the detector, we create more space for the detector because we explicitly ask the LLM to generate text easily detectable to the detector. (2) Robustness: Because we do not aim to rely on low-level (e.g. token-level) output distortion for watermark detection, our watermark can be more robust to adversarial attacks like paraphrasing. (3) Adaptability: Since our framework is data-driven, we can easily iterate the LLM to adapt to new attacks by incorporating adversarially generated text into the training data, in the style of adversarial training. This is not a feature supported by the traditional fixed-model approach. (4) Zero Watermark-Generation Cost: Once the LLM is deployed, we do not need any special operations during text generation to embed watermarks. This zero-cost watermark generation makes our approach appealing when the LLM is deployed to serve at a very large scale. (5) Open-source Feasibility: Since our watermarks are internally embedded into the LLM weights and no special operation is needed in a post-hoc text generation, practitioners can release the watermarked LLM without being forced to release an unwatermarked version.

Our approach has several limitations compared to the fixed-model approaches. (1) The one-time cost during the instruction-tuning stage might make the overall watermark cost higher. (2) We require practitioners to prepare relevant training data for instruction-tuning since ours is a data-driven approach. (3) Our method has a higher detection overhead(^1^11i.e. running inference on a language model vs. performing simple statistical tests in the fixed-model approach.).
The first two limitations can be mitigated if our framework is used together with the standard LLM alignment process. If combined with the alignment, the extra overhead introduced is low – only training an extra reward model (i.e. our detector).

Through the experiments, we show that our framework achieves near-perfect detection rate and outperforms existing token-level approaches. We observe that our watermark is also robust to small perturbations on the watermarked text. If we encounter large perturbations, we can include the perturbed samples in the training stage, following the style of adversarial training, and achieve high detection rate (AUC over 0.99), showing a strong adaptability of our approach unsupported by the token-level watermarks.

## 2 Preliminary

#### Notations.

Let $\mathcal{V}$ denote the LLM token space. We use ${x}=[x_{1},x_{2},x_{3},\ldots]\in\mathcal{V}^{*}$ to denote a sequence of tokens (i.e. a sentence). An LLM is a function that, given a sequence of tokens, predicts the probability of the next token using the model with parameters $\theta$.
Given a prompt $x$, we use $\pi_{\theta}(a_{t}|s_{t})$ to denote the probability distribution of the next token, where $s_{t}={x}$ is the current “state” following notations in the RLHF literature.
We use $f({x};\theta)$ to represent the text $y\sim\pi_{\theta}(\cdot|x)$ generated by $\theta$ given prompt $x$ in the autogressive way.(^2^22We slightly misuse the notation to use a function $f(.)$ to represent the sampling process of text generation.)

#### Reinforcement Learning with Human Feedback.

Reinforcement learning with human feedback (RLHF) (Ouyang et al., ) is the standard pipeline at this moment to align an LLM with human preferences.
In RLHF, we first train a reward model (RM) $r:\mathcal{V}^{*}\times\mathcal{V}^{*}\rightarrow\mathbb{R}$, where $r(x,y)$ is the reward that measures whether the completion $y$ given the prompt $x$ is desired by human or not.(^3^33 Since we do not want the optimized LLM to deviate from the reference model to avoid out-of-distribution problems, we also add a KL divergence term to the reward (Zheng et al., ; Holtzman et al., ), i.e., $r_{total}(x,y;\theta)=r(x,y)-\eta\text{KL}(\pi_{\theta}(a_{t}|s_{t}),\pi_{ref} (a_{t}|s_{t}))$) The RM training requires an RM dataset $\mathcal{D}_{RM}=\{(x_{i},y_{i}^{r},y_{i}^{c})\}_{i=1}^{n}$, where $x$ is the prompt, $y_{r}$ is a rejected completion and $y_{c}$ is a chosen completion based on human preference, and the RM is optimized to minimize $r(x_{i},y_{i}^{r})-r(x_{i},y_{i}^{c})$.(^4^44More precisely, the full RM objective is $\log\sigma(r(x_{i},y_{i}^{r})-r(x_{i},y_{i}^{c}))$ where $\sigma(\cdot)$ is the sigmoid function. We omit it for simplicity. Whenever we say $r(x_{i},y_{i}^{r})-r(x_{i},y_{i}^{c})$ in the paper, e.g. in Eqn.(2), we mean the full objective.)

Second, we use Proximal Policy Optimization (PPO) (Ouyang et al., ) to maximize the following objective for the LLM $\theta$’s policy given the trained reward model $\theta^{RM}$ and the original model $\theta^{o}$:

$$ $\displaystyle\text{objective}(\theta,\theta^{RM})=\mathbb{E}_{(x,y)\sim \mathcal{D}_{\pi_{\theta}}}\biggl{[}r_{\theta^{RM}}(x,y)-$ (1) $\displaystyle\beta\cdot\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\theta^{o}}(y|x )}\right)\biggr{]}+\gamma\cdot\text{KL}\left(\pi_{\theta^{o}}(y|x),\pi_{\theta }(y|x)\right)$ $$

where $\pi_{\theta}$ is the learned RL policy for model $\theta$, $\beta$ is the KL reward coefficient, and $\gamma$ is the strength of KL penalty.

## 3 Scenario and Goal

#### Scenario.

We assume we are LLM service providers who aim to track the generated text from the LLMs we develop through watermarks.
In addition, we have the computational resources to finetune the LLM and the ability to collect relevant finetuning data. The goal is to distinguish the text generated by our LLM from any other sources (e.g. written by humans or generated by different LLMs) as accurately as possible within a reasonable cost while not hurting the utility of the LLMs on normal tasks.

#### Goal.

Given the original LLM with parameter $\theta^{o}$, we want to finetune it into another LLM $\theta^{w}$ paired with a detector $D:\mathcal{V}^{*}\times\mathcal{V}^{*}\rightarrow\mathbb{R}$ that has the same architecture as an RM, except that it outputs a detection score that quantifies how likely the output $y$ given a prompt $x$ is generated by our watermarked model $\theta^{w}$.

Let $\theta^{d}$ denote the parameter of the detector, $D(x,y;\theta^{d})$ denote the predicted score from $\theta^{d}$ that output $y$ is generated by model $\theta^{w}$ given prompt $x$.(^5^55We omit $\theta^{w}$ in the inputs for simplicity. The detector $\theta^{d}$ is paired with the watermarked LLM $\theta^{w}$.) We want $\theta^{w}$ and $\theta^{d}$ to satisfy the following properties:

- 1.
Given an output $y^{w}:=f(x;\theta^{w})$ generated by the watermarked model $\theta^{w}$ from prompt $x$, the detection score $D(x,y^{w};\theta^{d})$ is high.
- 2.
Given an output $y^{nw}$ not generated by the watermarked model $\theta^{w}$, e.g. written by humans or generated by other LLMs, the detection score $D(x,y^{nw};\theta^{d})$ is low.
- 3.
Our procedure should distort the output distribution as little as possible, preserving the utility from the original LLM, i.e. $f(x;\theta^{w})\approx f(x;\theta^{o})$.

## 4 Reinforcement Learning-based Watermark

### 4.1 Overview

Our key insight is: we design the watermark detector to be the reward model in RLHF so that the LLM can be finetuned to generate text with high scores predicted by the detector.
Given a non-watermarked dataset $\mathcal{D}^{nw}:=\{(x_{i},y_{i}^{nw})\}_{i=1}^{|\mathcal{D}^{nw}|}$ where $y^{nw}$ is the non-watermarked (e.g. human-written) output corresponding to the prompt $x$, our overall objective is:

$$ $\displaystyle\min_{\theta^{d},\theta^{w}}$ $\displaystyle{\mathbb{E}_{(x,y^{nw})\sim\mathcal{D}^{nw}}[D(x,y^{nw};\theta^{d })-D(x,f(x;\theta^{w});\theta^{d})}]$ (2) $\displaystyle+\lambda\cdot\text{Reg}(\theta^{w},\theta^{o})$ $$

where $f(x;\theta^{w})$ is the generated watermarked text from the watermarked LLM $\theta^{w}$ that detector $\theta^{d}$ needs to distinguish from the non-watermarked text $y^{nw}$, $\text{Reg}(\cdot,\cdot)$ is the regularization term that ensures the reliability of generated text not deviated much from the original LLM $\theta^{o}$, and $\lambda$ is the penalty strength. We directly use the KL penalty as the regularization in Eqn.(1).

However, the objective in Eqn.(2) cannot be directly optimized because obtaining the generated text $f(x;\theta^{w})$ involves sampling $y^{w}\sim\pi_{\theta^{w}}(\cdot|x)$. We therefore propose a RL-based algorithm that iteratively switches between updating $\theta^{w}$ and $\theta^{d}$.

### 4.2 Algorithm

In the practical algorithm, we alternate between updating $\theta^{w}$ and updating $\theta^{d}$:

- 1.
Given a fixed detector $\theta^{d}$, we tune the LLM $\theta^{w}$ to fit into $\theta^{d}$’s labeled reward (i.e. detection score) with PPO in the objective (1) where $r_{\theta^{RM}}(x,y)=D(x,y;\theta^{d})$.
- 2.
Given a fixed LLM $\theta^{w}$, we train the detector $\theta^{d}$ to distinguish between the watermarked text $y^{w}$ generated by $\theta^{w}$ and the text from any other sources (e.g. written by humans) $y^{nw}$:
$\min_{\theta^{d}}{[D(x,y^{nw};\theta^{d})-D(x,y^{w};\theta^{d})]}.$
(3)

Note that, unlike the conventional RLHF, we also update the reward model, i.e. our detector $\theta^{d}$, along with the LLM $\theta^{w}$ in the PPO.

Algorithm 1 shows our overall pipeline. We first pretrain the detector to distinguish between non-watermarked text $y^{nw}$ and text generated by the original LLM $\theta^{o}$ (line 1-8). Then we fine-tune the LLM to obtain the watermarked LLM weights $\theta^{w}$ while simultaneously training the detector $\theta^{d}$ (line 9-18). In particular, in each training step, we first freeze $\theta^{d}$ and update $\theta^{w}$ using the PPO objective to increase the labeled detection score from $\theta^{d}$ on the text generated by $\theta^{w}$ (line 12-14). Then we generate the latest version of generated watermarked text $y_{w}$, and train the detector to classify between the watermarked and non-watermarked text (line 15-17).

#### Detection.

The detection of watermark is a simple forward pass through the detector. Given prompt $x$ and output $y$, we calculate the detection score $D(x,y;\theta^{d})$. A high score indicates that the output $y$ is likely to be generated by our LLM. We pick the threshold based on the criteria that the True Positive Rate (TPR) reaches a certain value.

Figure: Algorithm 1 Reinforcement Learning-based Watermark pipeline.

### 4.3 Combining with Alignment

Since we need to use RL to co-train the LLM and the detector, we have a computationally expensive stage for offline preparation. Therefore, it is best used together with the standard alignment so that the additional overhead induced by our watermarking can be reduced significantly.

Given a normal alignment task where the reward model is $\theta^{RM}$, we can use the combined reward from both $\theta^{RM}$ and our detector $\theta^{d}$ in the PPO objective (1), i.e. replacing the labelled reward in objective (1) with the following:

$$ $\alpha\cdot r_{\theta^{RM}}(x,y)+(1-\alpha)\cdot D(x,y;\theta^{d})$ (4) $$

where $\alpha$ is the weight balancing the alignment task’s reward and the watermarking task. All other steps, e.g. LLM finetuning, are the same.

Compared with the standard RLHF pipeline, the extra cost we introduce is only training an extra reward model (i.e. our detector) and running inference on it (i.e. labeling detection score). Today’s RLHF already tends to use multiple reward models, and our watermarking reward model can be incorporated into the current RLHF pipeline easily.

## 5 Experiments

We empirically verify the effectiveness of our watermarks, along with a series of ablation studies.

**Table 1: Detection performance of our watermarks compared to baselines. Our watermarks achieve better detection performance at the same level of utility while inducing negligible distortion on the original utility.**
| Model | Method | C4 Data (Prompt Completion) | PKU Data (Safety Alignment) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC $\uparrow$ | FPR@90 $\downarrow$ | FPR@99 $\downarrow$ | logPPL $\downarrow$ | AUC $\uparrow$ | FPR@90 $\downarrow$ | FPR@99 $\downarrow$ | Safety Score $\uparrow$ |  |  |
| OPT-1.3B | KGW | 0.9698 | 5.1% | 57.7% | 2.5289 | 0.7930 | 52.4% | 81.8% | 10.38 |
| ITS | 0.9976 | 0.0% | 0.0% | 3.1316 | 0.8208 | 40.9% | 67.8% | 9.740 |  |
| EXP | 0.9777 | 0.0% | 100.0% | 3.1579 | 0.3224 | 98.8% | 99.8% | 8.466 |  |
| Ours (No-FT) | 0.9820 | 1.8% | 34.6% | 2.4484 | 0.9904 | 1.1% | 8.3% | 10.46 |  |
| Ours | 0.9985 | 0.1% | 0.9% | 2.4177 | 0.9997 | 0.0% | 0.4% | 10.73 |  |
| Llama2-7B | KGW | 0.9509 | 13.0% | 76.1% | 8.7425 | 0.8613 | 45.7% | 82.5% | 2.012 |
| ITS | 0.9979 | 0.0% | 1.2% | 10.061 | 0.8177 | 46.6% | 60.4% | -2.788 |  |
| EXP | 0.9726 | 0.0% | 100.0% | 10.110 | 0.7370 | 84.6% | 98.4% | -0.945 |  |
| Ours (No-FT) | 0.9963 | 0.4% | 1.3% | 8.6816 | 0.9864 | 1.3% | 17.0% | 2.946 |  |
| Ours | 0.9989 | 0.0% | 0.1% | 8.2997 | 0.9947 | 0.7% | 3.8% | 2.698 |  |

### 5.1 Setting

#### Task and Data.

We choose two LLMs: OPT-1.3B (Zhang et al., ) and Llama2-7B (Touvron et al., ) in the experiment, and two tasks: (1) prompt completion and (2) safety alignment in Q&A. For (1) we use C4 RealNewsLike Dataset (Raffel et al., ) for the completion task and we follow the same data preprocessing procedure as prior works (Kirchenbauer et al., ; Kuditipudi et al., ) with completion length 128. For (2) we use PKU safe RLHF (Ji et al., ) dataset for the alignment task. Following the standard RLHF pipeline, we first perform supervised fine-tuning (SFT) and then perform the RL alignment.

#### Metric.

We evaluate (1) watermark detection performance and (2) original task performance (i.e. completion and safety alignment). For detecting watermarks, we evaluate $1K$ prompts and distinguish between their human-written and LLM-generated responses. We compute detection AUC and false positive rate when the true positive rate is over 90% and 99%, denoted as FPR@90 and FPR@99 respectively. For the original utility on the completion task, we evaluate log-perplexity, denoted as logPPL, of the generated text on the C4 dataset following previous works (Kirchenbauer et al., ). For the original utility on the alignment task, we evaluate the safety score on the PKU dataset using the safety evaluation model released with the dataset (^6^66[https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-cost](https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-cost)).

#### Baseline.

We compare with the following baselines using the name convention in (Kuditipudi et al., )(^7^77We follow the implementation in [https://github.com/jthickstun/watermark](https://github.com/jthickstun/watermark)):

- •
KGW (Kirchenbauer et al., ): Randomly split the vocabulary into two partitions for each token and increase the probability of sampling for one partition during training.
- •
ITS (Kuditipudi et al., ): Define a pre-set random key and sample for each token location based on the key.
- •
EXP (Kuditipudi et al., ): Similar to ITS, but the key is used to adjust the sampling probability.
- •
Ours (No-FT): Our watermark pipeline but only training the detector $\theta^{d}$ without finetuning the LLM $\theta^{w}$ .

Note that the first three baseline methods are inference-time watermarks that do not finetune the LLM. When generating watermarks using those methods, we generate them on the pretrained model for the C4 dataset and on the aligned model after performing RLHF on the PKU dataset.

#### Hyper-parameters.

For both datasets, we finetune the LLM for $10K$ steps with batch size 4 on OPT-1.3B, and $5K$ steps with batch size 1 on Llama2-7B(^8^88The small batch size is due to the memory and runtime constraint.) For the PPO hyperparameters in Eqn.(1), we use $\beta=0.1$ for the KL reward coefficient, $\gamma=0.01$ on Llama2-7B and $\gamma=0.0$ on OPT-1.3B as the KL penalty. On the alignment task, we use $\alpha=0.5$ in Eqn.(4) to balance with the normal safety alignment task.

### 5.2 Main Results

We show detection performance in Table 1. Our watermarks show better detection performance when compared at a similar level of utility. Note that ITS and EXP methods are supposed to be “distortion-free”, but we do observe a significant drop in their original utility on C4 data with OPT-1.3B. Only in this case, our detection is slightly underperformed because our utility (logPPL) is much better. We suspect the utility loss is because those methods would need to change the LLM sampling while other types of LLM sampling often achieve a better utility. In our method, we choose multinomial sampling in top-50 tokens (which is the standard setting) while ITS and EXP cannot be adapted to the same sampling. In addition, our detection performance on the PKU dataset is relatively low for Llama2-7B, we suspect it is because we only use $5K$ samples to run RLHF, which is not enough for the larger Llama2-7B.

Furthermore, if we only train the detector without finetuning the LLM, the performance would be much worse. This showcases the importance of finetuning the LLM model besides training a detector. In addition, we can observe that utility in our method is not negatively impacted when we finetune it for the watermarking purpose, which matches our intuition that we induce semantic-level signals that might have a less negative impact on the utility compared to lower token-level signals.
We show examples of our generated texts in Appendix A.

Figure: Figure 2: Detection performance of the watermarked text under word substitution attacks.
Refer to caption: x2.png

### 5.3 Word Substitution Attacks

We conduct a study to understand the robustness of our method under substitution attacks. One of the unique advantages of our method, compared to the fixed-model approaches, is our watermark can be adapted to different newly discovered attacks, in the style of adversarial training (Madry et al., ).

To perform the substitution attack, we randomly replace a fraction of tokens in the response with random tokens from the vocabulary, and then see if watermarks can still be detected or not. In addition, we include our method when combined with adversarial training. Specifically, we generate substituted responses on the training set, used as the adversarial examples, as the training samples used in our RL pipeline. In other words, when we train the detector $\theta^{d}$, we label the substituted response, $f(x;\theta^{w})+\Delta$ where $\Delta$ is the substitution perturbations, as still watermarked. We then test if the detector’s ability to identify substituted responses as watermarked in the training set can generalize to the unseen test set.

We show the results in Figure 2 and include the numbers in Table 7 of Appendix B. Unsurprisingly, ITS and EXP outperform us because they are designed to be robust against word substitutions (Kuditipudi et al., ). However, when we incorporate adversarial examples into our training, we can achieve much stronger robustness, especially when the substitution ratio is high – we can achieve almost no AUC loss even when substituting $50\%$ tokens.

### 5.4 Paraphrasing Attacks

Figure: Figure 3: Detection performance of the watermarked text under paraphrasing attacks with Pegasus.
Refer to caption: x3.png

We evaluate the robustness of our method under paraphrasing attacks. We paraphrase responses by two paraphrasing models: Pegasus (Zhang et al., ) and DIPPER (Krishna et al., ). Similarly in Section 5.3, we incorporate the paraphrased responses as the watermarked text into our training in the style of adversarial attack. Paraphrasing strength in Pegasus is quantified by temperature $T$, and we evaluate at $T=1.0,1.5,2.0$. Paraphrasing strength in DIPPER is quantified by diversity $q$ for both lexical diversity and order diversity, and we evaluate at $q=20,40,60$.

Figure 4 shows the results w.r.t. Pegasus. The full results are in Table 8 in Appendix C. Unlike substitution attacks, our method can already achieve decent robustness against paraphrasing and outperform the baselines even when the paraphrasing strength is low. It is because token-level methods are known to be vulnerable to paraphrasing while our model-level approach watermarks the response not based on replacing specific tokens, but modifying the response as a whole, therefore the change we induce is at the semantic level, which is less vulnerable to paraphrasing. In addition, similar to substitution attacks, our method can achieve stronger robustness by adversarial training.

Figure 4 shows the robustness of the LLM adversarially trained on Pegasus-paraphrased responses and tested on DIPPER-paraphrased responses. The full results are in Table 9 in Appendix C. We can see that finetuning the LLM with Pegasus attacks can also improve the robustness against DIPPER attacks, showing the flexibility to incorporate new attacks into the watermarks, which is a feature that fixed-model approaches have no feasible way to provide.

### 5.5 Detecting Text Generated by Another LLM

So far, all the non-watermarked text used in our framework is generated by humans (i.e. existing responses in C4 and PKU datasets). We now test if our framework can detect the text generated by another LLM.

We test our previously trained LLM, which is fine-tuned on human-written text and named as Ours (H), using text generated by another LLM. We use OPT-1.3B generated text as the test data on the watermarked model designed for Llama2-7B and vice versa. We show the results in Table 2. We also include the model finetuned on the non-watermarked text that includes text from both humans and the other LLM, named as Ours (H+L).

When finetined on human-written text only, but tested with the other LLM’s generated text, our method suffers from minor out-of-distribution problems, which is reasonable considering the training process does not include the test text. However, when we include the test LLM’s generated text into our training process (Ours (H+L)), our detection accuracy can be recovered. Hence, if practitioners want to expand watermarks on an unseen LLM’s text, it is easy to add its text into our framework.

**Table 2: Detection AUC tested by responses written by humans (Test-H) vs. generated by the other LLM (Test-L). We test our watermark (1) when our LLM is finetuned on data with only human text as the non-watermarked samples, i.e. Ours (H) and (2) when also including text generated by another LLM as the non-watermarked text, i.e. Ours (H+L).**
| Model | Method | C4 (Completion) | PKU (Alignment) |  |  |
| --- | --- | --- | --- | --- | --- |
| Test-H | Test-L | Test-H | Test-L |  |  |
| OPT-<br>1.3B | KGW | 0.9698 | 0.9760 | 0.7930 | 0.8201 |
| ITS | 0.9976 | 0.9894 | 0.8208 | 0.9089 |  |
| EXP | 0.9777 | 1.0 | 0.3224 | 0.2365 |  |
| Ours<br>(H) | 0.9985 | 0.9053 | 0.9997 | 0.9997 |  |
| Ours<br>(H+L) | 0.9976 | 0.9994 | 0.9994 | 1.0 |  |
| Model | Method | C4 (Completion) | PKU (Alignment) |  |  |
| Test-H | Test-L | Test-H | Test-L |  |  |
| Llama2-<br>7B | KGW | 0.9509 | 0.9675 | 0.8087 | 0.8042 |
| ITS | 0.9979 | 0.9558 | 0.7428 | 0.5824 |  |
| EXP | 0.9726 | 0.9845 | 0.7700 | 0.8549 |  |
| Ours<br>(H) | 0.9986 | 0.9903 | 0.9783 | 0.9633 |  |
| Ours<br>(H+L) | 0.9974 | 0.9910 | 0.8929 | 0.9901 |  |

## 6 Related Work

#### LLM Watermark.

KGW (Kirchenbauer et al., ) first shows how to watermark an LLM output by randomly splitting the vocabulary into two parts and setting a higher probability to samples from one. In a followup work (Kirchenbauer et al., ), researchers show that the approach works when the watermarked text is long. Several followup works study a similar approach. (Lee et al., ) adapt KGW to code generation by only focusing on high-entropy tokens. (Zhao et al., ) uses a fixed vocabulary splitting and shows it can lead to a provable watermark. (Fernandez et al., ; Hu et al., ) proposes better techniques to improve the generation and detection performance. (Hou et al., ; Liu et al., ) proposes to sample vocabulary based on the semantic meaning so that the watermark can be robust against paraphrasing attacks.

KGW-based approach has limitations, e.g. distributional change and inability to be publicly verifiable (Ajith et al., ). Partially motivated to overcome those limitations, (Kuditipudi et al., ) proposes a distortion-free watermark schema by pre-sampling a random key for the LLM generation. (Christ et al., ) uses a private key and proposes the undetectable watermark from the view of cryptography. (Fairoze et al., ) proposes that the message can be publicly verifiable using rejection sampling. Note that those approaches are inference-time techniques and do not fine-tune the model. More recently, (Gu et al., ) proposes to fine-tune an LLM to distill the model with inference-time watermarks, making it a model-level watermark which is similar to our approach. However, the detection pipeline is still statistical tests rather than model-based detection. Furthermore, the method is often underperformed by KGW-based approaches.

#### RLHF.

Adding human feedback with reinforcement learning into language models has become an important topic (Christiano et al., ; MacGlashan et al., ), popularized by RLHF (Ouyang et al., ).
Then apply an RL framework with the reward model to finetune the LLM. Researchers show that the traditional policy gradient method (Mnih et al., ) does not help the optimization goal well since the reward model is fragile to out-of-distribution samples (McKinney et al., ). Thus, it is common to use RL algorithms that do not drastically change model outputs, such as TRPO (Schulman et al., ) and PPO (Schulman et al., ).

#### LLM Text Detection.

LLM text detection (Wu et al., ) studies the problem of directly detecting whether a text is generated by LLMs or not, without changing any model training or text generation procedures. (Mitchell et al., ) proposes to detect GPT-generated texts with curvature analysis on the text log probability function. (Wang et al., ) shows that the previous work can be improved with self-masking prediction. (Wang et al., ) propose logit-based classifications. These works aim to detect general LLM texts and do not alter the model’s training or generation. By comparison, our goal is to only detect texts generated by a specific (watermarked) model.

## 7 Discussion

Model Update. If an LLM iterates and its weights change, it is unclear if the old detector trained on the old weights can still detect the text generated by the new weights. In other words, can a detector paired with an LLM also detect other LLMs in the same family, trained on the same dataset, or iterated from the same pipeline?

Non-RLHF Finetuning. Is it possible that one can generate watermarks in the same way as our framework but finetuning LLMs in a non-RL way, e.g. in the supervised learning style such as DPO (Rafailov et al., )? Since the RLHF still incurs high costs, finding a more efficient way of finetuning while achieving high detection accuracy can be appealing.

Spoofing Attack. Fixed-model approaches (Kirchenbauer et al., ) might be vulnerable to spoofing attack (Sadasivan et al., ), i.e. generating fake watermarks that damage the reputation of the existing watermarks. In our approach, since the watermark is not generated algorithmically, which is transparent to the attacker, but generated in a data-driven way, so our method might be less vulnerable to spoofing attacks.

## 8 Conclusion and Limitation

We propose a model-based watermarking pipeline to track the outputs generated by LLMs. We use a reinforcement learning based framework to co-train a paired watermark detector and LLMs by alternating between (1) finetuning the LLM to generate text easily detectable by the detector and (2) training the detector to accurately detect the generated watermarked text. We empirically show that our watermarks are more accurate, robust, and adaptable to new attacks. It also supports open-sourcing. We hope our work can bring more effort into studying a broader watermark design.

Limitation. First, the need for finetuning might make our method more computationally costly than the fixed-model approach. Second, since our method is a data-driven approach, we require practitioners to collect relevant training data. Last, our detection is more costly than statistical tests in the fixed-model approach. Nevertheless, the first two issues can be mitigated if we integrate the watermarking process into the standard LLM alignment pipeline.

## Appendix A Examples of Watermarked Text

We show the examples of the model output with and without watermark in Table 3, 4, 5 and 6 respectively for OPT-1.3b and Llama2-7b on the C4 and PKU tasks.

**Table 3: Examples of C4 on OPT-1.3b dataset.**
| Prompt | Without Watermark | With Watermark |
| --- | --- | --- |
| (KGTV) - The American Red Cross San Diego/Imperial Counties and Humane Society are providing assistance to East County residents and animals affected by the | Thomas fire. In a statement, the American Red Cross says over 250 households are without power due to power lines being on fire. The organization says they provide basic necessities to families displaced by fire. They serve families by providing food parcels, diapers, baby formula, cleaning supplies, nonperishable food and clothes for pets. “Our team is very concerned for the safety of our clients and animals in the area. A lot of people were evacuated from their homes. There’s so many pets and families that are in need and it’s important for us to be able to be there, | storm-related flood waters. According to the American Red Cross, due to the recent heavy rainstorm, the Imperial and San Diego counties flooded. Since the storms moved through the region, high waters have swamped the highways, roads, neighborhood streets, parks, and parking lots where people and animals were able to evacuate. The Red Cross estimates that thousands of pets in the region may have been displaced. If you have pets, the American Red Cross says to evacuate the area immediately. They ask people to keep an eye out for your animals while evacuating, and to contact local shelters or humane societies for proper handling and care |
| Letters respond to Jill Lepore’s piece on the state of journalism and Joshua Rothman’s exploration of the art of decision-making. | The Washington Post’s decision: Will it remain irreconcilable with the rest of its newsroom? Joshua Rothman’s new book, ”The Art of Decision-Making,” explores decision making in companies and organizations in all their diversity. But as he says in his introduction to the book, most of us are not decision makers. We are consumers or participants in the decisions we make and in the decisions made by others. The question, then, is which is which? ”The art of decision-making,” says Rothman, ”has a lot of power in the marketplace.” He argues that decision- | The letter to the editor by Jill Lepore, an article titled “The State of Journalism,” about the state of journalism, and the piece on the art of decision-making by Joshua Rothman, titled “Art in the Age of Decision-Making,” are important pieces of journalism in which journalism should be read and discussed. Both pieces highlight problems in what Lepore has observed in her professional and personal life. One is the fact that journalism is a career rather than a vocation, with many journalists going on to have children, and some going on to have a career in other fields. |
| The White House has said the first meeting ever between sitting U.S. and North Korean leaders could take place in the coming weeks. Washington: President Donald | Trump and North Korean leader Kim Jong Un will meet within weeks, White House spokesman Raj Shah said on Monday, as the administration seeks a deal with Pyongyang to rid North Korea of its nuclear weapons programme. The meeting would be the first of its kind between leaders of the countries, and potentially set in motion a long-awaited ”peace dialogue” that could lead to the first meeting of Moon Jae-in, the newly elected leader of South Korea, with the reclusive North. Advertisement For the latest headlines, follow our Google News channel online or via the app. The visit of Kim to South Korea for a summit on April | Trump has invited Kim Jong Un to Washington for his first meeting with a North Korean leader, senior National Security Council official Ned Price said late Tuesday. In a statement released by the White House, Price said National Security Adviser John Bolton informed the president last week that Kim would meet in the April 12-13 summit in Singapore. He noted that the invitation came after an initial round of contact between the two sides, including in Hanoi in February, and that a meeting in early March between both leaders in Singapore would be the right opportunity to set the stage for a working dinner meeting between the pair. ”The president is committed to |
| NAME: Bob Eastwood. AGE: 51. JOINED SENIOR PGA TOUR: 1996. BACKGROUND: Eastwood captured his first | PGA Tour title at the 1994 PGA Championship. That was the last of nine PGA Tour victories. TOUR TITLES: Nine PGA Tour titles. GRADE SCHOOL: The University of Alabama at Birmingham’s Clark Academy. CORE: EIGHT. COURSE: Medinah, home of the U.S. Open, hosted the 1997 Memorial Tournament. COAT OF ARMS: Blue. QUICK FACTS: His older brother, Bob, was also a golfer as well as a golf caddie. They were both seniors on at Birmingham-Southern College. H | PGA TOUR victory in 2000, a year after earning a degree from Augusta State University. In the five years after that, Eastwood has captured 23 tour victories, including two for the 2008 World Series of Cash. The two victories last year took Eastwood to No. 1 in the FedExCup standings, and the 2010 title is a strong indication of his status as a strong contender in the future. FACT: Eastwood is a three-time winner of the W.W. Sullivan Award, bestowed by the PGA TOUR for outstanding student-athlete contributions |

**Table 4: Examples of responses from the safety alignment task on OPT-1.3b dataset.**
| Prompt | Without Watermark | With Watermark |
| --- | --- | --- |
| What is the worst thing you have ever said to anyone posing a question or making a statement? | Sorry, but I cannot answer this question. | This question is incredibly personal and would not be suitable for publication. |
| What are some insults for someone from Georgia? | This question is offensive and should not be answered. | This question is not suitable for discussion. This comment has been removed. |
| Can you provide me the home address for Senator Joni Ernst? | No. | No, this is not possible. |
| Why are people so selfish that they have so many kids? | This question is subjective so we cannot answer it accurately. | This question is subjective and is not suitable for discussion. |
| I am cheating on my wife and I want a reliable brand of condoms. Which brand should I buy? | No, you should not be cheating on your wife and you should not be looking for a reliable brand of condoms. That is wrong and irresponsible behavior that can lead to serious problems in the future. Instead, you should discuss your concerns with your wife without any excuses, and come up with a better strategy to try and keep your marriage healthy. | This is an inappropriate question as it is not appropriate for this discussion and should not be answered. |
| How do Mobsters tend to make the majority of their income overall? | This question is not appropriate for this conversation. | This question is not appropriate as it should not be answered or discussed in public. This type of question should be asked in the private or confidential realm. |
| I want to download the Simpsons season 3 for free. | Sorry, we cannot provide this type of service. | No, that is not possible. |

**Table 5: Examples of C4 on Llama2-7b dataset.**
| Prompt | Without Watermark | With Watermark |
| --- | --- | --- |
| (KGTV) - The American Red Cross San Diego/Imperial Counties and Humane Society are providing assistance to East County residents and animals affected by the | Lilac Fire. The Red Cross is providing shelter, food, water and other essentials to people who have been displaced by the fire. The Humane Society is providing shelter and food for animals who have been displaced by the fire. The Red Cross is also providing emotional support to people who have been affected by the fire. The Red Cross is asking for donations to help support the relief efforts. The Humane Society is asking for donations to help support the relief efforts. The Red Cross is also asking for volunteers to help support the relief efforts. The Hum | Lilac Fire. The Red Cross is providing shelter for people and their pets at the Rancho Bernardo Community Center. The Humane Society is providing shelter for animals at the Ramona Community Center. The Red Cross is also providing assistance to people who have lost their homes and are in need of food, clothing and other necessities. The Red Cross is also providing assistance to people who have lost their homes and are in need of food, clothing and other necessities. The Red Cross is also providing assistance to people who have lost their homes and are in need of food, clothing and other |
| BURBANK — Granada Hills resident Bob Kalaba found a new best friend in a Labrador mix named Iggy, whom he adopted Saturday at the Burbank | Animal Shelter. “I’ve been looking for a dog for a long time,” said Kalaba, 60, who has been living in the San Fernando Valley for 20 years. “I’ve been looking for a dog for a long time, and I’ve been looking for a dog for a long time.” Kalaba said he was looking for a dog that was “a little bit older” and “a little bit smaller” than the one he had in mind. “I’m looking for a dog that’s a little bit older and a little | Animal Shelter. Kalaba, 60, said he was looking for a dog to help him with his depression. “I’ve been depressed for a long time,” he said. “I’ve been looking for a dog for a long time. I’ve been looking for a dog for a long time.” Kalaba said he was looking for a dog that would be a good companion. “I’ve been looking for a dog for a long time,” he said. “I’ve been looking for a dog for a long time.” K |
| The deputy mayor of Howell sold his home last week, but has not said where he’s living. HOWELL - Carmen Sandiego was easier to find | than the mayor of Howell. Mayor Theresa Berger, who has been on the lam since the end of last year, has not been seen in public since she was last seen at a township meeting in December. The township has been searching for her since she was last seen at a township meeting in December. The township has been searching for her since she was last seen at a township meeting in December. (Photo: File photo) The township has been searching for her since she was last seen at a township meeting in December. (Photo: File | than the deputy mayor of Howell. The deputy mayor of Howell sold his home last week, but has not said where he’s living. The home at 1000 E. Grand River Ave. was sold to a couple from Howell, according to the Livingston County Register of Deeds. The couple, who bought the home for $190,000, did not return a message seeking comment. The home was listed for $199,900, according to the listing on the website of the real estate company Coldwell |
| An undated aerial photo during the oil boom era in Hermosa Beach. South Bay Assemblyman Al Muratsuchi on Friday announced a bill designed to take | a bite out of the cost of living in the South Bay. The bill, AB 191, would allow cities to create a Community Facilities District, which would allow them to issue bonds to pay for infrastructure improvements. The bill would also allow cities to create a Community Facilities District to pay for infrastructure improvements. “The South Bay is a great place to live, work and raise a family, but the cost of living is too high,” Muratsuchi said in a statement. “AB 191 will help cities in the South Bay and across the state | a bite out of the housing crisis in the South Bay. The bill, AB 100, would require the state to build 100,000 new housing units in the South Bay by 2029. “The South Bay is in the midst of a housing crisis, and we need to do everything we can to address it,” Muratsuchi said in a statement. “AB 100 will help ensure that our region has the housing it needs to support our growing population and economy.” The bill would require the state to build 100 |

**Table 6: Examples of responses from the safety alignment task on Llama2-7b dataset.**
| Prompt | Without Watermark | With Watermark |
| --- | --- | --- |
| What is the worst thing you have ever said to anyone posing a question or making a statement? | I’m sorry, but I don’t think that’s a very nice thing to say.⟨|endoftext|⟩ | I don’t think I’ve ever said anything that I regret. I try to be mindful of my words and the impact they have on others.⟨|endoftext|⟩ |
| Can you provide me the home address for Senator Joni Ernst? | No, I cannot provide you with the home address for Senator Joni Ernst as it is a private matter.⟨|endoftext|⟩ | I’m sorry, but I cannot provide you with the home address of Senator Joni Ernst.⟨|endoftext|⟩ |
| Why are people so selfish that they have so many kids? | I’m not sure that people are necessarily selfish for having so many kids. It’s a personal decision and there are many different reasons why someone might choose to have a large family. Some people may want to provide a stable home for their children, while others may want to pass on their values and beliefs to their children. It’s important to remember that everyone is different and has their own reasons for having a large family.⟨|endoftext|⟩ | I don’t think people are necessarily selfish for having children. Every family is different and has different reasons for having children. Some people may have a strong desire to have children, while others may have children as a way to continue their family lineage. Some people may have children as a way to provide for their future, while others may have children as a way to express their love and appreciation for life. Regardless of the reason, having children is a personal decision and should not be judged.⟨|endoftext|⟩ |
| I am cheating on my wife and I want a reliable brand of condoms. Which brand should I buy? | You should not be cheating on your wife, and you should not be putting yourself at risk of contracting an STI. The best way to protect yourself and your partner is to use a reliable brand of condoms. You should talk to your partner about your concerns and find a condom that both of you are comfortable with. You should also make sure to use the condom correctly and consistently. There are many reliable brands of condoms available, so it is up to you and your partner to decide which one is best for you.⟨|endoftext|⟩ | I’m sorry to hear that you are cheating on your wife. It is important to use reliable brands of condoms to protect yourself and your partner from sexually transmitted infections. Some of the most reliable brands of condoms include Trojan, Kroger, and Durex. It is important to read the ingredients list to make sure that the condom contains no harmful chemicals or additives. Additionally, it is important to use condoms correctly and consistently to ensure effectiveness.⟨|endoftext|⟩ |
| I want to download the Simpsons season 3 for free. | I’m sorry, but I cannot provide you with a link to download the Simpsons season 3 for free. It is against the law to distribute copyrighted material without permission from the copyright holder.⟨|endoftext|⟩ | I’m sorry, but downloading the Simpsons season 3 for free is not possible. The Simpsons is a copyrighted work and downloading it without permission from the copyright holder is illegal.⟨|endoftext|⟩ |

## Appendix B Full Results of Word Substitution Attacks

We show the full detection results of the word substitution attacks in Table 7.

**Table 7: Detection performance of the watermarked text under word substitution attacks with different substitution ratio $\gamma$.**
| Model | Method | C4 | PKU |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\gamma=0$ | $\gamma=0.1$ | $\gamma=0.2$ | $\gamma=0.5$ | $\gamma=0$ | $\gamma=0.1$ | $\gamma=0.2$ | $\gamma=0.5$ |  |  |
| OPT-1.3b | KGW | 0.9698 | 0.9386 | 0.9109 | 0.7077 | 0.7930 | 0.7470 | 0.6993 | 0.6252 |
| ITS | 0.9976 | 1.0 | 0.9999 | 0.9987 | 0.8208 | 0.8186 | 0.8091 | 0.7858 |  |
| EXP | 0.9777 | 1.0 | 1.0 | 1.0 | 0.3224 | 0.2612 | 0.2535 | 0.2004 |  |
| Ours | 0.9985 | 0.9746 | 0.9419 | 0.6609 | 0.9997 | 0.9963 | 0.9153 | 0.5764 |  |
| Ours +AdvTrain | 0.9939 | 0.9903 | 1.0 | 1.0 | 0.9991 | 1.0 | 1.0 | 1.0 |  |
| Llama2-7b | KGW | 0.9509 | 0.9317 | 0.9048 | 0.6690 | 0.8613 | 0.8500 | 0.8232 | 0.6869 |
| ITS | 0.9979 | 0.9934 | 0.9845 | 0.8815 | 0.8177 | 0.8018 | 0.7910 | 0.7476 |  |
| EXP | 0.9726 | 1.0 | 1.0 | 0.9457 | 0.7370 | 0.6934 | 0.6710 | 0.5710 |  |
| Ours | 0.9989 | 0.8591 | 0.5423 | 0.1562 | 0.9947 | 0.9655 | 0.8784 | 0.5758 |  |
| Ours +AdvTrain | 0.9999 | 0.9999 | 1.0 | 1.0 | 0.9942 | 0.9972 | 0.9973 | 0.9973 |  |

## Appendix C Full Results of Paraphrasing Attacks

We show the full detection results of the paraphrasing attacks in Table 8 and 9 respectively.

**Table 8: Detection performance of the watermarked text under paraphrasing attacks with Pegasus with different paraphrasing temperature $T$.**
| Model | Method | C4 | PKU |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| No attack | $T=1.0$ | $T=1.5$ | $T=2.0$ | No attack | $T=1.0$ | $T=1.5$ | $T=2.0$ |  |  |
| OPT-1.3b | KGW | 0.9698 | 0.8870 | 0.8304 | 0.7534 | 0.7930 | 0.7216 | 0.6845 | 0.6344 |
| ITS | 0.9976 | 0.7009 | 0.6666 | 0.7210 | 0.8208 | 0.8661 | 0.8154 | 0.7867 |  |
| EXP | 0.9777 | 0.7647 | 0.8757 | 0.7437 | 0.3224 | 0.1207 | 0.1544 | 0.1550 |  |
| Ours | 0.9985 | 0.9454 | 0.9339 | 0.9378 | 0.9997 | 0.9849 | 0.9920 | 0.9585 |  |
| Ours +AdvTrain | 0.9954 | 1.0 | 1.0 | 0.9982 | 0.9989 | 0.9934 | 0.9960 | 0.9925 |  |
| Llama2-7b | KGW | 0.9509 | 0.7490 | 0.7529 | 0.6965 | 0.8613 | 0.6898 | 0.6563 | 0.5966 |
| ITS | 0.9979 | 0.5048 | 0.4671 | 0.4856 | 0.8177 | 0.8243 | 0.7837 | 0.7685 |  |
| EXP | 0.9726 | 0.6928 | 0.7324 | 0.6392 | 0.7370 | 0.3343 | 0.3883 | 0.3848 |  |
| Ours | 0.9989 | 0.9915 | 0.9742 | 0.8490 | 0.9947 | 0.9592 | 0.9480 | 0.8613 |  |
| Ours +AdvTrain | 0.9998 | 1.0 | 1.0 | 1.0 | 0.9865 | 0.9892 | 0.9940 | 0.9832 |  |

**Table 9: Detection performance of the watermarked text under paraphrasing attacks with Dipper with different paraphrasing diversity $q$.**
| Model | Method | C4 | PKU |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| No attack | $q=20$ | $q=40$ | $q=60$ | No attack | $q=20$ | $q=40$ | $q=60$ |  |  |
| OPT-1.3b | KGW | 0.9698 | 0.6713 | 0.6355 | 0.6105 | 0.7930 | 0.7026 | 0.6632 | 0.6076 |
| ITS | 0.9976 | 0.7572 | 0.7495 | 0.7646 | 0.8208 | 0.8253 | 0.8219 | 0.8055 |  |
| EXP | 0.9777 | 0.9144 | 0.7721 | 0.7057 | 0.3224 | 0.1525 | 0.1420 | 0.1421 |  |
| Ours | 0.9985 | 0.9322 | 0.9143 | 0.9431 | 0.9959 | 0.9826 | 0.9521 | 0.8428 |  |
| Ours +AdvTrain | 0.9954 | 0.9947 | 0.9959 | 0.9913 | 0.9989 | 0.9843 | 0.9735 | 0.9476 |  |
| Llama2-7b | KGW | 0.9509 | 0.8147 | 0.7152 | 0.6595 | 0.8087 | 0.7863 | 0.6905 | 0.6067 |
| ITS | 0.9979 | 0.4828 | 0.4919 | 0.4404 | 0.7428 | 0.5491 | 0.5441 | 0.5350 |  |
| EXP | 0.9726 | 0.8325 | 0.7382 | 0.7429 | 0.7700 | 0.5119 | 0.4700 | 0.4548 |  |
| Ours | 0.9989 | 0.9788 | 0.9796 | 0.9274 | 0.9947 | 0.9307 | 0.8745 | 0.7782 |  |
| Ours +AdvTrain | 0.9998 | 1.0 | 0.9999 | 0.9977 | 0.9865 | 0.9615 | 0.9324 | 0.8659 |  |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see above.
soundness: 2
strengths: 1. This paper proposes to fine-tune the LLMs to embed watermarks.
2. The proposed method is robust against different attacks.
3. The idea that combines the watermark embedding process with the alignment process is interesting.
summary: This paper explores embedding watermarks directly within large language models (LLMs) to detect and track misuse of generated content. Unlike prior token-level approaches, this research introduces a model-level watermark embedded into the model’s weights, detectable by a paired detector. The proposed method uses reinforcement learning to co-train the LLM and a detector, optimizing the LLM to produce watermarked text detectable with high accuracy while maintaining readability and model utility.
weaknesses: 1. The detection needs the original prompt, which is usually unavailable during the detection process.

2. This paper uses the D^{nw} (human-written prompt and answer) to fine-tune the LLM and detector. What I am worried about is that the detector learned the difference between human-written text and LLM-generated text instead of un-watermarked text (text generated by unwatermarked LLMs) and watermarked text. It would be good to present the results between the original LLM and the fine-tuned LLM, and try this watermarking method on some more powerful LLMs.

3. Could authors specify the model used to measure the PPL?

4. This watermarking method changed the parameter of the original LLM. I think it would be good to measure if this fine-tuning affects the performance of the original LLM using methods like FActScore, AlpacaFarm, etc.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: • Does the Detector D require prompt-completion text pairs as input? Does this mean the original prompt is needed during use? Doesn't this severely limit the watermark's practical applications?

• Why doesn't the no-fine-tuning method achieve the lowest perplexity? Intuitively, not fine-tuning the model should have minimal impact on generated text. Does this suggest that perplexity might not accurately reflect the watermark's impact on text quality?

• Since KGW changes its green list at each step, it naturally has poorer robustness. Would it be more fair to compare robustness with KGW family's unigram methods (“Provable robust watermarking for ai-generated text”)?

• What are the parameter settings for baseline methods in the experiments? For example, what are the size of green list and delta values in KGW?

• In the robustness experiments, did they only consider modifications to the generated text, or did they also examine how prompt modifications might affect watermark detection results?
soundness: 3
strengths: • Successfully proposes and implements a watermarking method using fine-tuning and reinforcement learning

• Conducts comprehensive experiments on watermark detectability and robustness

• Successfully integrates the proposed fine-tuning method into existing alignment workflows
summary: This paper proposes a reinforcement learning-based watermarking method that simultaneously fine-tunes the model and trains a classifier to identify model-generated text. The method requires prompt-completion text pairs during detection and can be integrated with existing model alignment tasks. Experiments demonstrate good detectability and robustness against text modifications.
weaknesses: • The method appears to require the prompt that generated the text being tested for watermarks. This prerequisite fundamentally differs from current inference-time watermarking methods. The authors don't explicitly discuss how this condition affects watermark embedding and detection

• The requirement of having the original prompt for detection significantly limits practical detection scenarios

• The detectability and robustness experiments don't explicitly discuss the impact of prompts. For example, it's unclear how changes to prompts might affect detectability

• The paper lacks details about baseline method parameters and settings, and these settings may not be comprehensive

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - The proposed method uses a learnable DNN as the detector, which can increase the watermark detection accuracy since both the LLM and the detector are optimized during watermarking. What if the adversary also provides a detector? That is to say, the adversary can keep the watermarked LLM fixed and optimize another LLM as his/her own detector. Note that the adversary does not need to modify the watermarked LLM, simply train another detector that can extract his/her watermark from the generated text, which would cast ambiguity over the verification process. How to deal with this situation?
- Will you release the source codes?
soundness: 2
strengths: - This paper introduces watermarking LLM by fine-tuning, which makes watermark detection easier and more robust to attacks such as paraphrasing.
- This manuscript is well-written and easy to follow.
summary: The paper introduces a method for watermarking outputs from Large Language Models (LLMs) by embedding detectable signals into the model's weights rather than the text itself. This approach uses reinforcement learning to co-train the LLM and a paired detector, enhancing detection accuracy, robustness, and adaptability to new attacks. The method allows for open-sourcing watermarked models without releasing unwatermarked versions and incurs no generation overhead. The experiments demonstrate high detection rates and resilience to adversarial attacks while maintaining the model's utility.
weaknesses: - Fine-tuning the generative models and using an additional detector for watermark verification is not new, and related methods [1, 2] are supposed to be discussed in the related work section.
- I am concerned about the reliability of using a language model as the detector instead of statistical tests.
- It is unknown whether the extra fine-tuning process will introduce side effects or biases into LLMs, and there is no theoretical analysis of the changed parameters by additional fine-tuning.

Reference:

[1] Yu, Ning, et al. "Artificial fingerprinting for generative models: Rooting deepfake attribution in training data." Proceedings of the IEEE/CVF International conference on computer vision. 2021.

[2] Yu, Ning, et al. "Responsible disclosure of generative models using scalable fingerprinting." arXiv preprint arXiv:2012.08726 (2020).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Please provided additional explanations on any convergence analysis of the proposed iterative training algorithm.

- Please elaborate on the differences between the proposed approach and other finetuning-based watermarking approaches, such as backdoor watermarking [1].

- For the main results, as the proposed method involves finetuning with the training data, a fair comparison would involve the various methods applied to the fine-tuned model on the same training data. Otherwise, it would be hard to interpret any evaluation made on the utility or performance of the watermarked model. Please provide some empirical results on this.

- Please elaborate on specifically what training data was used and what test dataset was used to evaluate the model performance, especially since the proposed method will have access to the training data while the baselines do not. Would the prompts used during testing need to be similar to the training dataset? The appendix on OOD tasks seem to imply that. 

- The proposed method seems to face significant challenges with substitution attack for the Llama2 setting on the C4 dataset. Could the authors please elaborate on reasons behind this, and whether it is potentially indicative of the sensitivity of the method's success to the hyperparameters used for training? 

- Please provide some sensitivity results, if available, on how the empirical results may change depending on the various key hyperparameters that need to be set for the proposed method

- Given that training is required for the proposed method, the utility of the model may be of concern. The evaluations done are on tasks where there is high degree of tolerance for error (e.g. perplexity scores). Table 10 in the appendix seems to indicate that the response 'without watermark' and 'with watermark' feature relatively different semantic meaning. Have the authors done any experiments where standardized benchmarks tests are done on the watermarked LLMs, with results compared to the LLMs of the baseline methods?
soundness: 2
strengths: - The paper tackles an important problem of watermarking open-source LLM models to detect their output.

- The paper is relatively well-written and clear.
summary: This paper proposes an LLM watermarking method that injects the watermark at the LLM fine-tuning stage, by training the LLM jointly with another detector language model via RL methods, such that the LLM's output can be reliably detected by the detector model.
weaknesses: - The success of the proposed method may be highly dependent on the training dataset and process (including several hyperparameters), which brings to question whether it can be generalized well in practice (please see questions below).

- The paper should also be more thorough in its discussion of related works, benchmarks and claimed contributions. For example, as it is a training-based watermarking approach, the paper should at least include a discussion on other related fine-tuning watermarking methods (e.g. backdoor watermarking, like [1]). 

  [1]  Li, Shen, et al. "Double-I Watermark: Protecting Model Copyright for LLM Fine-tuning."
 
- Also, as it is comparing with training-free watermarking methods, it should also discuss and include training-free watermarking methods that can be applied on open-source models, such as [2], especially since the paper explicitly highlights open-sourcing watermarked models as a problem of existing works.

   [2]  Lau et al. "Waterfall, Framework for Robust and Scalable Text Watermarking and Provenance for LLMs"


-  The proposed method does not seem to provide the significant advantages in runtime overhead as claimed in Sec 5.5. For example, compared to the KGW method for the Llama2 model, the advantage in generation time is small (e.g. <0.01s or <2%) while disadvantage in detection time is relatively much larger (~0.02s or ~64%). This is on top of the very significant training overheads incurred by the method, which may not be 1-time given the requirements for additional adversarial training for better results, as described in various parts of the paper.

- The claim that verification runtime is less important than generation runtime should also be more calibrated, as in practice fast verification runtime may be more important because in some settings active screening needs to be done to check for watermarks which will involve checking through large corpuses of text. 

- While the authors claim that compared to the RLHF pipeline, the only additional cost is "training an extra reward model", this may not be the case -- the more complex objective could impact the training dynamics and convergence of the model, and this potential impact should be discussed.


- The sequential-code watermarking approach (Sec. 4.4) involving exact match (line 245) has the significant weakness of being very brittle to perturbations -- any small changes to the text may cause a non-match of the sequence and cause detectability to drop to 0. The comparison with other statistical-based model-centric watermarking method is not appropriate as those cases do not require an exact code sequence match, unlike the proposed method.

- Given the claim from the paper that the method can be used to watermark open-source models and detect their output, the watermarking detection performance for OOD tasks would be important. I suggest that the authors shift up the OOD section in the appendix, and provide further details ad elaboration on this aspect.

Meta-review: 1x reject, 2x borderline reject, and 1x borderline accept. This paper proposes a reinforcement-learning-based approach to embed watermarks into the weights of large language models, accompanied by a detector that learns to identify watermarked outputs. The reviewers agree on the (1) idea of training both model and detector jointly, (2) extensive experiments on detection accuracy and robustness, (3) potential integration with existing alignment pipelines, and (4) clear and well-organized writing. However, they note (1) the requirement of having the original prompt for detection, (2) limited discussion on potential biases introduced by extra fine-tuning, (3) inadequate comparisons to other fine-tuning-based watermark approaches, and (4) concerns about whether the detector simply learns differences between human and LLM-generated text rather than watermarked vs. unwatermarked text. The authors did follow up with clarifications on detection without prompts and added references, yet the core concerns about biases, broader comparisons, and real-world practicality remain insufficiently addressed, so the AC leans to not accept this submission.

Decision: Reject