Learning to Watermark LLM-generated Text via Reinforcement Learning

Authors: Xiaojun Xu, Yuanshun Yao, Yang Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Watermark

Abstract:
We study how to watermark LLM outputs, i.e. embedding algorithmically detectable signals into LLM-generated text to track misuse. Unlike the current mainstream methods that work with a fixed LLM, we expand the watermark design space by including the LLM tuning stage in the watermark pipeline. While prior works focus on token-level watermark that embeds signals into the output, we design a model-level watermark that embeds signals into the LLM weights, and such signals can be detected by a paired detector. We propose a co-training framework based on reinforcement learning that iteratively (1) trains a detector to detect the generated watermarked text and (2) tunes the LLM to generate text easily detectable by the detector while keeping its normal utility. We empirically show that our watermarks are more accurate, robust, and adaptable (to new attacks) with no generation overhead. It also allows watermarked model open-sourcing. In addition, if used together with alignment, the extra overhead introduced is low -- only training an extra reward model (i.e. our detector). We hope our work can bring more effort into studying a broader watermark design that is not limited to working with a fixed LLM.

Decision: reject
Reviewer scores: 3, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730220355986

thread_type:
official_review

<thread_object>
creator: reviewer_csmp

<content>
summary:
This paper proposes an LLM watermarking method that injects the watermark at the LLM fine-tuning stage, by training the LLM jointly with another detector language model via RL methods, such that the LLM's output can be reliably detected by the detector model.

strengths:
- The paper tackles an important problem of watermarking open-source LLM models to detect their output.

- The paper is relatively well-written and clear.

weaknesses:
- The success of the proposed method may be highly dependent on the training dataset and process (including several hyperparameters), which brings to question whether it can be generalized well in practice (please see questions below).

- The paper should also be more thorough in its discussion of related works, benchmarks and claimed contributions. For example, as it is a training-based watermarking approach, the paper should at least include a discussion on other related fine-tuning watermarking methods (e.g. backdoor watermarking, like [1]). 

  [1]  Li, Shen, et al. "Double-I Watermark: Protecting Model Copyright for LLM Fine-tuning."
 
- Also, as it is comparing with training-free watermarking methods, it should also discuss and include training-free watermarking methods that can be applied on open-source models, such as [2], especially since the paper explicitly highlights open-sourcing watermarked models as a problem of existing works.

   [2]  Lau et al. "Waterfall, Framework for Robust and Scalable Text Watermarking and Provenance for LLMs"


-  The proposed method does not seem to provide the significant advantages in runtime overhead as claimed in Sec 5.5. For example, compared to the KGW method for the Llama2 model, the advantage in generation time is small (e.g. <0.01s or <2%) while disadvantage in detection time is relatively much larger (~0.02s or ~64%). This is on top of the very significant training overheads incurred by the method, which may not be 1-time given the requirements for additional adversarial training for better results, as described in various parts of the paper.

- The claim that verification runtime is less important than generation runtime should also be more calibrated, as in practice fast verification runtime may be more important because in some settings active screening needs to be done to check for watermarks which will involve checking through large corpuses of text. 

- While the authors claim that compared to the RLHF pipeline, the only additional cost is "training an extra reward model", this may not be the case -- the more complex objective could impact the training dynamics and convergence of the model, and this potential impact should be discussed.


- The sequential-code watermarking approach (Sec. 4.4) involving exact match (line 245) has the significant weakness of being very brittle to perturbations -- any small changes to the text may cause a non-match of the sequence and cause detectability to drop to 0. The comparison with other statistical-based model-centric watermarking method is not appropriate as those cases do not require an exact code sequence match, unlike the proposed method.

- Given the claim from the paper that the method can be used to watermark open-source models and detect their output, the watermarking detection performance for OOD tasks would be important. I suggest that the authors shift up the OOD section in the appendix, and provide further details ad elaboration on this aspect.

questions:
- Please provided additional explanations on any convergence analysis of the proposed iterative training algorithm.

- Please elaborate on the differences between the proposed approach and other finetuning-based watermarking approaches, such as backdoor watermarking [1].

- For the main results, as the proposed method involves finetuning with the training data, a fair comparison would involve the various methods applied to the fine-tuned model on the same training data. Otherwise, it would be hard to interpret any evaluation made on the utility or performance of the watermarked model. Please provide some empirical results on this.

- Please elaborate on specifically what training data was used and what test dataset was used to evaluate the model performance, especially since the proposed method will have access to the training data while the baselines do not. Would the prompts used during testing need to be similar to the training dataset? The appendix on OOD tasks seem to imply that. 

- The proposed method seems to face significant challenges with substitution attack for the Llama2 setting on the C4 dataset. Could the authors please elaborate on reasons behind this, and whether it is potentially indicative of the sensitivity of the method's success to the hyperparameters used for training? 

- Please provide some sensitivity results, if available, on how the empirical results may change depending on the various key hyperparameters that need to be set for the proposed method

- Given that training is required for the proposed method, the utility of the model may be of concern. The evaluations done are on tasks where there is high degree of tolerance for error (e.g. perplexity scores). Table 10 in the appendix seems to indicate that the response 'without watermark' and 'with watermark' feature relatively different semantic meaning. Have the authors done any experiments where standardized benchmarks tests are done on the watermarked LLMs, with results compared to the LLMs of the baseline methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730482229559

thread_type:
official_review

<thread_object>
creator: reviewer_caer

<content>
summary:
The paper introduces a method for watermarking outputs from Large Language Models (LLMs) by embedding detectable signals into the model's weights rather than the text itself. This approach uses reinforcement learning to co-train the LLM and a paired detector, enhancing detection accuracy, robustness, and adaptability to new attacks. The method allows for open-sourcing watermarked models without releasing unwatermarked versions and incurs no generation overhead. The experiments demonstrate high detection rates and resilience to adversarial attacks while maintaining the model's utility.

strengths:
- This paper introduces watermarking LLM by fine-tuning, which makes watermark detection easier and more robust to attacks such as paraphrasing.
- This manuscript is well-written and easy to follow.

weaknesses:
- Fine-tuning the generative models and using an additional detector for watermark verification is not new, and related methods [1, 2] are supposed to be discussed in the related work section.
- I am concerned about the reliability of using a language model as the detector instead of statistical tests.
- It is unknown whether the extra fine-tuning process will introduce side effects or biases into LLMs, and there is no theoretical analysis of the changed parameters by additional fine-tuning.

Reference:

[1] Yu, Ning, et al. "Artificial fingerprinting for generative models: Rooting deepfake attribution in training data." Proceedings of the IEEE/CVF International conference on computer vision. 2021.

[2] Yu, Ning, et al. "Responsible disclosure of generative models using scalable fingerprinting." arXiv preprint arXiv:2012.08726 (2020).

questions:
- The proposed method uses a learnable DNN as the detector, which can increase the watermark detection accuracy since both the LLM and the detector are optimized during watermarking. What if the adversary also provides a detector? That is to say, the adversary can keep the watermarked LLM fixed and optimize another LLM as his/her own detector. Note that the adversary does not need to modify the watermarked LLM, simply train another detector that can extract his/her watermark from the generated text, which would cast ambiguity over the verification process. How to deal with this situation?
- Will you release the source codes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587714091

thread_type:
official_review

<thread_object>
creator: reviewer_ycyk

<content>
summary:
This paper proposes a reinforcement learning-based watermarking method that simultaneously fine-tunes the model and trains a classifier to identify model-generated text. The method requires prompt-completion text pairs during detection and can be integrated with existing model alignment tasks. Experiments demonstrate good detectability and robustness against text modifications.

strengths:
• Successfully proposes and implements a watermarking method using fine-tuning and reinforcement learning

• Conducts comprehensive experiments on watermark detectability and robustness

• Successfully integrates the proposed fine-tuning method into existing alignment workflows

weaknesses:
• The method appears to require the prompt that generated the text being tested for watermarks. This prerequisite fundamentally differs from current inference-time watermarking methods. The authors don't explicitly discuss how this condition affects watermark embedding and detection

• The requirement of having the original prompt for detection significantly limits practical detection scenarios

• The detectability and robustness experiments don't explicitly discuss the impact of prompts. For example, it's unclear how changes to prompts might affect detectability

• The paper lacks details about baseline method parameters and settings, and these settings may not be comprehensive

questions:
• Does the Detector D require prompt-completion text pairs as input? Does this mean the original prompt is needed during use? Doesn't this severely limit the watermark's practical applications?

• Why doesn't the no-fine-tuning method achieve the lowest perplexity? Intuitively, not fine-tuning the model should have minimal impact on generated text. Does this suggest that perplexity might not accurately reflect the watermark's impact on text quality?

• Since KGW changes its green list at each step, it naturally has poorer robustness. Would it be more fair to compare robustness with KGW family's unigram methods (“Provable robust watermarking for ai-generated text”)?

• What are the parameter settings for baseline methods in the experiments? For example, what are the size of green list and delta values in KGW?

• In the robustness experiments, did they only consider modifications to the generated text, or did they also examine how prompt modifications might affect watermark detection results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676368711

thread_type:
official_review

<thread_object>
creator: reviewer_7qwc

<content>
summary:
This paper explores embedding watermarks directly within large language models (LLMs) to detect and track misuse of generated content. Unlike prior token-level approaches, this research introduces a model-level watermark embedded into the model’s weights, detectable by a paired detector. The proposed method uses reinforcement learning to co-train the LLM and a detector, optimizing the LLM to produce watermarked text detectable with high accuracy while maintaining readability and model utility.

strengths:
1. This paper proposes to fine-tune the LLMs to embed watermarks.
2. The proposed method is robust against different attacks.
3. The idea that combines the watermark embedding process with the alignment process is interesting.

weaknesses:
1. The detection needs the original prompt, which is usually unavailable during the detection process.

2. This paper uses the D^{nw} (human-written prompt and answer) to fine-tune the LLM and detector. What I am worried about is that the detector learned the difference between human-written text and LLM-generated text instead of un-watermarked text (text generated by unwatermarked LLMs) and watermarked text. It would be good to present the results between the original LLM and the fine-tuned LLM, and try this watermarking method on some more powerful LLMs.

3. Could authors specify the model used to measure the PPL?

4. This watermarking method changed the parameter of the original LLM. I think it would be good to measure if this fine-tuning affects the performance of the original LLM using methods like FActScore, AlpacaFarm, etc.

questions:
Please see above.
</content>
</thread_object>

</thread>


Metareview:
metareview:
1x reject, 2x borderline reject, and 1x borderline accept. This paper proposes a reinforcement-learning-based approach to embed watermarks into the weights of large language models, accompanied by a detector that learns to identify watermarked outputs. The reviewers agree on the (1) idea of training both model and detector jointly, (2) extensive experiments on detection accuracy and robustness, (3) potential integration with existing alignment pipelines, and (4) clear and well-organized writing. However, they note (1) the requirement of having the original prompt for detection, (2) limited discussion on potential biases introduced by extra fine-tuning, (3) inadequate comparisons to other fine-tuning-based watermark approaches, and (4) concerns about whether the detector simply learns differences between human and LLM-generated text rather than watermarked vs. unwatermarked text. The authors did follow up with clarifications on detection without prompts and added references, yet the core concerns about biases, broader comparisons, and real-world practicality remain insufficiently addressed, so the AC leans to not accept this submission.