An End-to-End Model For Logits Based Large Language Models Watermarking

Authors: KA HIM WONG, Jicheng Zhou, Jiantao Zhou, Yain-Whar Si
Primary area: foundation or frontier models, including LLMs
Keywords: LLM watermarking, End-to-end optimization, Robustness

Abstract:
The rise of large language models (LLMs) has increased concerns over source tracing and copyright protection for AI-generated content (AIGC), highlighting the need for advanced detection technologies. Passive detection methods usually face high false positives, while active watermarking techniques using logits or sampling manipulation offer more effective protection. Existing LLM watermarking methods, though effective on unaltered content, suffer significant performance drops when the text is modified and could introduce biases that degrade LLM performance in downstream tasks. These methods fail to achieve an optimal tradeoff between text quality and robustness, particularly due to the lack of end-to-end optimization of the encoder and decoder. In this paper, we introduce the first end-to-end logits perturbation method for watermarking LLM-generated text. By jointly optimizing the encoder and decoder, our approach achieves a better balance between quality and robustness. To address non-differentiable operations in the end-to-end training pipeline, we introduce an online prompting technique that leverages the on-the-fly LLM as a differentiable surrogate. Our method demonstrates superior detection robustness, consistently outperforming state-of-the-art (SOTA) methods by 1.2\%, 4.0\%, and 5.5\% across 3 LLMs, averaged over 6 types of text distortions. Simultaneously, our approach achieves exceptional text quality, as evidenced by reduced text perplexity and improved performance in the downstream tasks with a margin of 19.2\% and 3.03\%. Our method can be easily generalized to different LLMs. The code is available in supplementary material.

Decision: reject
Reviewer scores: 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730072667282

thread_type:
official_review

<thread_object>
creator: reviewer_jkaa

<content>
summary:
This paper proposed an end-to-end optimization framework for achieving better trade-off between the robustness and the text quality. The authors validate the effectiveness of the proposed framework with comprehensive experiments on popular LLMs.

strengths:
1. The proposed end-to-end method is original, and extensive experiments have been conducted to evaluate its quality, detectability, and robustness.

2. The presentation is well-structured, making the paper easy to follow.

weaknesses:
1. Unclear motivation. The authors claimed “However, these existing approaches still fail to achieve an optimal trade-off between text quality and robustness”. However, the authors have missed an important line of works regarding the distortion-free watermark (Kuditipudi et al., 2024; Christ et al., 2024), which suggested we can embed watermarks into LLMs without affect the generation quality. Thus, the there is generally no trade-off between the text quality and robustness, and the claim in the paper is wrong.

2. Limited contribution. Comparing to the previous works (Liu et al., 2024b; Huo et al., 2024), which also share an encoder-decoder structure for logits-based watermarking, the proposed method only introduce a jointly training network for achieving better trade-off between text quality and robustness. As the reviewer has pointed out in weaknesses 1, the trade-off generally does not exist. Thus, the contributions of the proposed method are unclear.

3. The experimental results also cannot support the motivation of “achieving better trade-off between text quality and robustness”. In Figure 7. The KGW watermark has significantly better quality than the proposed watermark, although the detectability and the robustness of KGW are poor. In order to claiming the proposed method has achieved better trade-off than KGW, the authors should show the superior of the proposed method on all quality, detectability, and robustness axis. Besides, in Figure 5, we can also see that the proposed method does not always outperform the baselines in all scenarios.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624458777

thread_type:
official_review

<thread_object>
creator: reviewer_jp2f

<content>
summary:
The authors present an end-to-end training-based text watermarking method aimed at achieving an optimal trade-off between text quality and robustness, leveraging the logits-based watermarking framework introduced by Kirchenbauer et al. Specially, they jointly train additional encoder to generate logits perturbation to shift the tokens’ probability distribution and additional decoder to extract the watermarking signals from the text. In addition, the authors introduce distortion module, address the non-differentiable operations in the end-to-end training pipeline, and consider the generalization to different LLMs.

strengths:
* A distortion module is helpful to enhance the robustness.

weaknesses:
*  Insufficient coverage of relevant related work
* Inadequate explanation of key methodological design choices
* Evaluation on outdated LLM architectures
* Limited adaptive attack evaluation
* Unclear figure captions (specifically Fig. 2)

questions:
### **Comparative Analysis**

* The paper briefly mentions other training-based methods (UPV, SIR) but lacks detailed comparison
* Please provide in-depth analysis of architectural differences and performance variations between this work and existing training-based approaches

### **Efficiency and Generalization**
* The cross-model inference time overhead is significant - what optimizations are possible?
* How does the method handle LLMs not included in the cross-model converter?
* What is the failure mode analysis?

### **Evaluation Scope**
* Evaluation should include more recent LLMs (e.g., Yi, Qwen)
* Need broader testing across model architectures and scales

### **Related Work and Claims**

* Notable omission of generation-based watermarking methods (e.g., AWT [1], REMARK-LLM [2])
* The "first end-to-end framework" claim requires more careful qualification

[1] Adversarial watermarking transformer: Towards tracing text provenance with data hiding

[2] REMARK-LLM: A robust and efficient watermarking framework for generative large language models

### **Security Analysis**

* How does the method perform against adaptive attacks where adversaries have full access to the system?
* Need evaluation of undetectability and robustness when attackers can obtain paired watermarked/unwatermarked samples

### **Architecture Choices**
* Please justify the selection of LSTM as the decoder backbone
* What alternatives were considered and why were they rejected?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679408487

thread_type:
official_review

<thread_object>
creator: reviewer_qahy

<content>
summary:
In this paper, the authors present a novel logit-based watermarking pipeline for text generation. Their approach incorporates Gumbel-Softmax for sampling and an online prompting technique for adversarial edits, allowing the encoder and decoder to be trained in an end-to-end fashion. The method achieves state-of-the-art detectability under various attacks while maintaining high-quality generated text.

strengths:
- The paper includes an extensive section on experiments, including many state-of-the-art methods and attack scenarios.
- The results for overall detectability and text quality look promising.
- The encoder and decoders are small, so although an extra watermark encoder and decoder have been introduced, the generation and detection are very efficient.

weaknesses:
- The result on generation diversity is not great as the proposed method has the lowest diversity among all other methods. Even though this doesn't affect the results on the benchmarks, I think this might be a bad feature for certain tasks, like synthetic data generation.
- The proposed method is training-based not like some of the baselines. The method might suffer OOD issues that the distribution of the prompt at the inference time is quite different from the training.
- The proposed method used a classifier for detection, and this does not give us an interpretable result like a p-value. This might also be bad if we want to control the false positive rate during detection.

questions:
- I wonder how expensive is the training especially the requirement for the GPU memory. If I understand it correctly, the forward looks like: first token logits -> encoder -> sample the first token -> second token logits -> encoder -> sample the second token -> ... So we recursively call the encoder for n times if we generate n tokens. Would the computational graph be huge? Especially you also have to sample some tokens from the online text editor later. I wonder how did you train it exactly.
- As I mentioned above, the method might suffer OOD issues. Would the encoder/decoder trained on WikiText-103 still be effective for other datasets? 
- Another thing that confuses me is that: where do you show the results for cross-llm inference? I noticed you mentioned: "To train our end-to-end model, we chose OPT-1.3B as the online LLM for efficiency." Does this mean results for llama models are the transfer inference results? Or this is for the online text editor.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734386047

thread_type:
official_review

<thread_object>
creator: reviewer_sskq

<content>
summary:
The authors propose an end-to-end optimized watermarking method for large language models to enable the detection of AI-generated content. The goal is to enhance the robustness/text quality trade-off of current LLM watermarking methods. The challenge is that many operations, such as generating sequences of text, are not differentiable. The authors overcome this issue by using the well-known Gumbel-Softmax trick to backpropagate through the text-generating process. To enhance robustness, the authors incorporate a paraphrasing model during the optimization method, and they develop cross-LLM adapters to train on one LLM and deploy it to other LLMs. They show robustness against six text modification attacks and improved text quality.

strengths:
- The method works well and allows adapting against paraphrasing attacks during optimization. 

- The authors thoroughly evaluate their approach by including experiments on robustness, detectability and impact on runtime during inference. 

- The paper is clear in its presentation and presents the proposed ideas well. 

- The cross-LLM inference adapter is a great idea, and I have not seen one before for trainable watermarking methods.

weaknesses:
- The results from Figure 5 in their current form are not reproducible and lack transparency. I believe it should be a scatter plot that includes the quality degradation, and the authors should state the hyperparameters for each approach used for paraphrasing (e.g., the prompt used for paraphrasing). 

- Abdelnabi et al. [A] have previously proposed end-to-end watermarking for LLMs. They also use the Gumbel-softmax trick to differentiate through the text generation process. The authors should consider citing this work.

- Figure 7, showing the difference in token distribution for the top 50 tokens, is difficult to interpret. It looks like the distance to the non-watermarked text is quite large (especially compared to KGW). Also, the choice of using 400 non-watermarked/watermarked samples is unclear. I think it would be better to plot detection accuracy against the size of the training dataset. 

- It is well known that perplexity is an unreliable metric used to measure text quality [C]. I was surprised that the authors did not include watermarked samples in their Appendix. There is a known problem: training LLMs with Gumbel-softmax is unstable and can lead to poor results for text generation [D]. Could the authors please show watermarked samples and (potentially) include a limitation section on current challenges when using this optimization method? 

--------
[A] Abdelnabi, Sahar, and Mario Fritz. "Adversarial watermarking transformer: Towards tracing text provenance with data hiding." 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021.

[C] Wang, Yequan, et al. "Perplexity from plm is unreliable for evaluating text quality." arXiv preprint arXiv:2210.05892 (2022).

[D] Yu, Zhang Ze, et al. "Fine-tuning Language Models with Generative Adversarial Reward Modelling." arXiv preprint arXiv:2305.06176 (2023).

questions:
- I do not understand why the prompt between non-watermarked and watermarked texts needs to differ (footnote 3 on page 9). Why can't the attacker re-use the same prompts when querying non-watermarked texts?  

- In Figure 2, I am unclear how the authors calculate the distance $L_{sem}$ between the watermarked and non-watermarked texts $X_{wm}, X_{nwm}$. Since both sequences will differ in the sampled tokens, they will diverge throughout the generation process if sampled for many tokens. Then, calculating this semantic distance will be meaningless as you cannot effectively align $X_{wm}, X_{nwm}$. Also, it appears unreasonable that the averaged similarity over many contexts will be a meaningful measure of the overall similarity between two sequences. I would appreciate the authors elaborating on this point and providing more context

- The description of the online text editing module is a bit confusing to me. Do the authors also use Gumbel-softmax for the online text editor, or do they pass $X_wm$ directly to the decoder $D$ contrary to what is shown in Figure 1? Since the text generation process from the online text editor $N$ is not necessarily differentiable unless you use some trick, end-to-end training from the detector's prediction back to the encoder won't be possible. I would appreciate it if the authors could elaborate on this point.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732257431502

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
Dear Reviewers,

We sincerely appreciate the time and effort the reviewers have dedicated to evaluating our manuscript. The concerns and feedback raised during the initial review have significantly contributed to enhancing the quality of our paper. Below, we summarize the key responses to the reviewers' suggestions and questions.

### About the Robustness/Quality Trade-off

The main concern of our method is about the lowest log diversity in Fig. 6 (b). and the relatively large bias in Fig. 7 (c). 

1. **Superior Overall Performance**  
We argue that the importance of different quality metrics is not identical. As shown in our experiments, our model achieves superior downstream task performance despite having relatively lower log diversity and higher token bias. This is because multiple token candidates can fit within a given context, providing flexibility without sacrificing task performance. A detailed comparison of our method with SOTA competitors is presented in Table 5, accompanied by a radar graph in Fig. 9. These results demonstrate the clear advantages of our approach, including significant improvements in robustness, perplexity, and downstream task performance. This highlights our model's ability to achieve a well-balanced performance across key metrics.

2. **User-Controllable Trade-offs**  
   Our model allows users to adjust the balance of different quality metrics by modifying the watermark strength δ and the top-*k* logits tokens without retraining the model. To illustrate this, we conducted additional experiments presented in Table 5, where we adjust the value of *k* to 40 while keeping other model settings unchanged. The experiments reveal that increasing the top-*k* can enhance log diversity but could slightly increase perplexity. Similarly, as shown in Fig. 12, reducing δ helps lower token bias, though this adjustment could compromise robustness.

### About End-to-End Model Training

The main concerns regarding the end-to-end model training are differentiability and complexity. 

1. **Differentiability**  
   It is important to clarify that all prompts and generated text remain in the embedding domain throughout the process. In our proposed online prompting, the prompt is first converted into the embedding domain and then concatenated with X_wm/X_nwm. This ensures the entire process is differentiable, as we avoid the text-embedding transformation, which is the primary source of non-differentiability.

2. **Training Resources and Hyperparameters**  
   Details about the training resources and hyperparameters are provided in Appendix F.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732415235665

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Further Feedback

comment:
Dear Reviewers,

We hope this message finds you well. We are writing to follow up on the responses we provided to your valuable comments on our submission. We have already addressed all the points raised.

As the discussion deadline is approaching in a few days, we kindly request your further feedback on our responses at your earliest convenience. Your insights are crucial for us to improve our work. Thank you very much for your time and consideration.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733290995491

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comprehensive and Final Summary on Submission9685

comment:
Dear Area Chairs,  

We sincerely appreciate the time, effort, and valuable insights provided during the review process. It is encouraging to note that **three out of four reviewers have given positive scores** to our submission, recognizing the significance of our work. The constructive feedback has been instrumental in enhancing the quality and clarity of our paper, and we are grateful for the opportunity to improve our work.  

We are pleased to report that after addressing their concerns, Reviewers **sSkQ**, **QAhy**, and **JP2f** have increased their scores. However, with the deadline approaching, we would like to kindly note that if Reviewer **JkAA** raises any further questions or concerns regarding our recent responses, we do not have the opportunity to provide additional clarification.  

---

### Positive Aspects Highlighted by Reviewers  

#### **1. Novelty and Contribution**  
**All reviewers** acknowledge the novelty of our end-to-end model, which adapts against potential attacks during optimization, thereby enhancing robustness. Reviewers **sSkQ** and **QAhy** further highlight the innovative "convert" module that enables cross-LLM inference, improving model transferability.  

#### **2. Comprehensive Experimental Validation**  
Reviewers **sSkQ**, **QAhy**, and **JkAA** mention the thoroughness of our experiments, which comprehensively evaluate key aspects such as robustness, quality, undetectability, and efficiency.  Reviewer **JP2f** specifically appreciates the additional experimental results in Appendices, which provide a detailed comparison of the trade-offs between robustness and text quality.  

#### **3. Presentation and Clarity**  
Reviewers **sSkQ** and **JkAA** praise the well-organized and accessible structure of our paper.

---

### Concerns Raised by Reviewers and Our Responses  

We have summarized the main concerns raised by reviewers in our "Response to All Reviewers." In particular, we would like to address the feedback from Reviewer **JkAA**, who expressed skepticism regarding the trade-off between quality and robustness in LLM watermarking. Reviewer **JkAA** argues that distortion-free methods eliminate this trade-off, rendering our claims about achieving a better quality/robustness balance invalid.  

We respectfully disagree with this perspective and have provided detailed responses in our latest reply to Reviewer JkAA. Below is a summary of our reasoning:  

1. **Distortion-Free Methods Are Not Robust:**  
   - No existing distortion-free watermarking scheme achieves robustness comparable to logits-based methods like Unigram.  
   - Our additional experiments demonstrate that distortion-free methods, such as Unbiased, fall short in detecting watermarked text after paraphrasing. This weak robustness has also been corroborated by recent benchmarking studies, including *MarkLLM* and *Mark my words*.  

2. **Compromises in Efficiency, Accessibility, and Adaptability:**  
   - Distortion-free methods sacrifice efficiency (e.g., our method is **16,000 times faster than EXP-Edit** and **680 times faster than Unbiased** in detection time).  
   - Accessibility is limited, **Unbiased require access to token logits from the LLM API and the prompts**, which may not always be feasible.  
   - Adaptability is hindered, as distortion-free methods are **incompatible with beam search and low-entropy scenarios**.  

Our proposed method does not face these limitations, demonstrating superior efficiency, accessibility, and adaptability while maintaining robustness and quality.  

---

We deeply value the reviewers' thoughtful feedback, which has strengthened our work. We hope this response clarifies our position and addresses any lingering concerns.  

Thank you for your continued support.  

Sincerely,  
Submission9685 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission is proposed a learned encoder-decoder approach for text watermarking that can be learned end to end. Learning the watermark scheme end to end in this way increases performance in a number of robustness charateristics by smaller amounts, but gives up the guarantees developed in previous work.

One point that several reviewers brought up, and I concur, is that the positioning of this paper as "the first encoder-decoder/end-to-end" learned watermark is surprising. The submission relates only to work past 2024 KGW-type analytical watermarks, missing the entire generation of watermark papers before that, based on learned encoder-decoder setups in various configurations, where Abdelnabi&Fritz, 2020 is only one of the more common examples (which even quite closely relates to the mechanics of this work, aside from still using a separate model for watermark encoding and generation, which is an older style appropriate for weaker LMs). Liu et al., 2024b and  Huo et al., 2024 are a few other example brought up by the reviewers.

Secondly, I think it is interesting to note that analytic watermarks forgo method of previous work to provide guarantees of various forms, such as provable p-values - which are highly valuable for high-stakes applications such as text forensics. This issue is also at the core of the inquiry of JkAA, who provides additional details on this discussion in relationship to analytical watermarks which explore the robustness trade-off targetted by the authors.

Overall, support from positive reviewers is marginal, and I do consider these issues significant enough that I do not recommend acceptance for now. I do think this work has merit (especially around the transfer conversion part), and I hope the authors are going to revise their manuscript based on the feedback received.