Title: Learning to Watermark: A Selective Watermarking Framework for Large Language Models via Multi-Objective Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

watermark

Selective Watermarking

multi-objective optimization
Abstract: The rapid development of LLMs has raised concerns about their potential misuse, leading to various watermarking schemes that typically offer high detectability. 
However, existing watermarking techniques often face trade-off between watermark detectability and generated text quality. 
In this paper, we introduce Learning to Watermark (LTW), a novel selective watermarking framework that leverages multi-objective optimization to effectively balance these competing goals. 
LTW features a lightweight network that adaptively decides when to apply the watermark by analyzing sentence embeddings, token entropy, and current watermarking ratio. 
Training of the network involves two specifically constructed loss functions that guide the model toward Pareto-optimal solutions, thereby harmonizing watermark detectability and text quality.
By integrating LTW with two baseline watermarking methods, our experimental evaluations demonstrate that LTW significantly enhances text quality without compromising detectability. 
Our selective watermarking approach offers a new perspective for designing watermarks for LLMs and a way to preserve high text quality for watermarks. The code is publicly available at: https://github.com/fattyray/learning-to-watermark


Peer reviews:
comments: SUMMARY:
This paper trains a neural network to achieve the selective watermarking strategy by analyzing sentence embeddings, token entropy, and the current watermarking ratio, which achieves a better balance between text quality and detectability.

STRENGTHS AND WEAKNESSES:
Strengths:

1. This paper provides an optimized method to achieve a selective watermarking approach.

2. This paper provides extensive experiments to demonstrate the effectiveness of the proposed methods.

Weaknesses:

1. Could the authors consider using additional datasets, such as long-form QA, to further validate the effectiveness of the proposed method? 

2. OPT-6.7b is too old compared with the most recent open-source LLM, like Llama-3.

3. Table 1 and 2 provide a performance comparison of different watermark methods. Why does Table 2 not provide TPR@2? Moreover, this paper strengthens the trade-off between the text quality and detectability. One better way to demonstrate this trade-off is to provide TPR with a lower FPR, such as 1%, 0.1%, 0.01%.

4. Could the authors explain the reason for using the similarity metric? It seems that this metric can not accurately represent the quality of watermarked text, because it is not necessary to have a high similarity between watermarked text and reference text especially for the open generation dataset (C4).

5. Is it possible to report the rate of watermarked tokens?

QUESTIONS:
Please see above.

LIMITATIONS:
Please see above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Learning to Watermark (LTW), a novel selective watermarking framework for large language models (LLMs). The method uses a lightweight neural network (Selector Network) trained via multi-objective optimization to decide dynamically when to watermark tokens during generation. By incorporating sentence embeddings, token entropy, and the watermark ratio, LTW aims to balance text quality and detectability more effectively than prior approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper is clearly written and well-structured and easy to follow.\n2. LTW proposes learning selective watermarking to balance text quality and detectability, going beyond entropy thresholds and introducing adaptive criteria based on richer contextual signals.\n\n**Weaknesses:**\n1. Eq.5 shows that the objective is to maximize the semantic similarity between the watermarked and unwatermarked text. However, it is unclear what exactly the unwatermarked text refers to and how it is generated in practice. The authors should clarify this point to avoid confusion about the optimization target.\n2. The paper mentions that LTW achieves strong robustness. Why is this not discussed in the introduction? More importantly, why does LTW improve robustness?  For example, Table 1 shows that LTW achieves higher robustness than KGW, but the reasons for this advantage are not clearly discussed.\n3. According to Table 1, Table 2, Figure 8, and Figure 9, LTW outperforms KGW in both detectability and text quality. Yet from Equations 3 and 7, LTW seems like a relaxed version of KGW. Why does this weaker formulation yield a better results?\n4. There is limited discussion and no ablation study on the impact of the losses defined in Eq.(10) and Eq.(11), which are central to the optimization process.\n\nQUESTIONS:\nCan the selector network trained on one LLM generalize to other LLMs? In line 278, the text refers to Table 3 and Table 4, but Table 4 is missing from the paper. In Tables 1 and 2, the number of decimal places is inconsistent. Could the authors explain the reason for this formatting choice?\n\nLIMITATIONS:\n1. The method emphasizes the need for an adaptive threshold to decide whether to watermark a given token. However, Table 3 shows that the gains in detectability and quality from this adaptive threshold are quite limited. This raises questions about whether this component is really necessary.\n\n2. Since LTW requires an additional sentence embedding model and selector network during both generation and detection, it introduces higher computational and memory overhead. The authors should provide a discussion and experiments on the time and space costs of the approach.\n\n3. From the proposed method, it appears that the watermark ratio is adjusted based on the portion of text already generated. This results in a higher watermark ratio in the earlier parts of the text and a lower ratio in later parts. This design may naturally expose the watermark to truncation attacks, where keeping only the later segments could reduce detectability. Could the authors clarify whether this would significantly impact detection performance?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a methodology to improve Green-list based watermarking schemes for LLMs. The method is based on the design of a so-called \"selector network\", that is, a regressor mapping a series of features to a mask. The main idea is to prevent watermarking during steps where the token distribution is almost deterministic and thus would lead to both a high quality loss and few gains in detectability if any.\n\nTo achieve this, the authors leverage a multi-objective optimization objective, balancing quality and detectability and claiming to achieve Pareto optimality. Quality is taken into account using embedding similarity (between watermarked and non-watermarked sentence), entropy of \n\nFinally, the authors compare their scheme to other classical methods, as well as comparable methods based on the entropy.\n\nSTRENGTHS AND WEAKNESSES:\n## Qualities\n\n**Clear and well-written paper**: Each step of the method is well motivated and articulated. Each loss is well described. I was able to answer most my questions during second and third read of the paper. I am confident I should be able to try replicating the results with the information contained in the paper (and in the appendix for the MLP training details).\n\n**Pertinent choice for the losses**: The collection of features and loss design for the selector network looks very complete and well-designed to me, taking into account all necessary properties for a watermark that is both robust and imperceptible. \n\n**Nice analysis of the selector outputs**: I find Figure 7 and its corresponding paragraph very insightful. I appreciate the time taken to study the function the words chosen by the selector network.\n\n## Weakness\n\n**Detection score of Green-list tokens is flawed**: The authors use the original z-score of the KGW method as their basis for detection. This assumes asymptotic normality of the score and a text long enough that normality can be invoked to compute sound statistics (p-value and confidence interval). Such assumptions are known to be unnecessary, as shown in [1]. Indeed, the sum of token scores for green-list watermarks follow a binomial distribution $\\mathcal{B}(\\gamma, l)$ under $\\mathcal{H}_0$ ($\\gamma$ being the probability of the token to be in the green list for non-watermarked text and $l$ the size of the text). This means an **exact, non-asymptotic** p-value can be computed for KGW and Unigram, leading to a precise control over Type I errors. Currently the authors have to rely on empirical measurements which are less rigorous. \n\n**Flawed choice of metrics**: This is a direct consequence of the previous problem. The authors use F1, AUC and TPR at (very!) high FPR. Obviously, watermarking schemes are good enough so that the detectability under no attack is always close to 1 for each metric. This makes comparison difficult and variations between schemes insignificant in Table 1. A better way to compare the schemes would be to fix a **low** probability of false-alarm, say $10^{-6}$, and compute the power of the test at this guaranteed rate. This can be done easily if a theoretically sound score is used during detection. In particular, using the $p$-value approach I outlined in the previous section allows to do this analysis without any problem. See any of [1,2] for a more thorough description of the method. Also note that $10^{-6}$ might still be too high a $P_{FA}$ to get significant difference in powers between watermarking schemes, in which case, I would advise the authors to go even lower.\n\n**Uninformed choice of LLM**: Two LLMs are studied, OPT-6.7B and GPT-J-6B. No rationale is provided for why these models have been chosen. It would have been a lot more pertinent to choose three different LLMs, one with low, medium and high entropy on average and see the difference in gains using the authors method. I would make the author's method even more enticing as I expect the performance to be high for low entropy models. For an idea of what kind of LLM to choose, I know [1] has performed such a study in Appendix J: *Phi* models have notoriously high entropy, *Llama-2* very low entropy and *Llama-3.1* is between the two. At the very least, an analysis of the distribution of the entropy of currently used models should be provided \n\n**Lack of link to theoretical analysis**: The distribution of the score for KGW follows a binomial distribution under $\\mathcal{B}(\\gamma^{\\prime}, l)$ under  $\\mathcal{H}_1$ too ($\\gamma^{\\prime}$ being the probability of the token to be in the green list for a watermarked text and $l$ the size of the text). The parameter $\\gamma^{\\prime}$  usually cannot be computed theoretically but is simple to estimate empirically. It would have been interesting to provide an estimation of this parameter with and without the selector network to estimate a theoretical gain/loss in detectability. More difficult, but certainly most insightful, a theoretical analysis on the distribution of the $\\gamma^{\\prime}$ within a text and its impact on detectability would have been welcomed in order to provide conditions on when focusing on certain token distributions while disregarding others is beneficial to detectability.\n\n[1] Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., & Furon, T. (2023, December). Three bricks to consolidate watermarks for large language models. In 2023 IEEE International Workshop on Information Forensics and Security (WIFS) (pp. 1-6). IEEE.\n\n[2] Giboulot, E., & Furon, T. (2024, December). WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off. In NeurIPS 2024-38th Conference on Neural Information Processing Systems (pp. 1-34).\n\nQUESTIONS:\n1. Could the authors improve their evaluation by focusing on the TPR at a low FPR (at least less $10^{-5}$ ) instead of providing many insignificant metrics such as AUC, F1 etc..? The best course of action would be to switch to non-asymptotic p-values if times and resources allow it -- since I understand this would entail changing the loss of the selector network.\n2. I would forego EXP-Edit in favor of another distortion-free scheme which can actually provide theoretical guarantees \n3. Can you provide a rationale for the choice of LLMs and maybe provide an empirical analysis of the distribution of their entropy for each generation step and for different texts following my \"**Uninformed choice of LLM**\" comment ?\n4. An empirical analysis of the probability of drawing a green token under $\\mathcal{H}_1$ with and without the selector token would be welcome to get a better insight on the impact of the selector network on detection.\n5. *(Curiosity)* Why have you chosen a hard binary mask instead of keeping the soft weights of the selector network? I would guess a soft mask to be more flexible, allowing to tune the bias $\\delta$ automatically for a given distribution. Have you tried it? If yes, was it worse than the binary mask?\n\n**Score increase or decrease** : Given the current state of the paper, I see no reason to decrease the score since I am confident in the author's results and since the methodology looks sound to me. I will definitely increase the score if: 1) better metrics are provided (most important), 2) a better selection of LLM is made with regard to their entropy (if time and resources permit) and 3) the proposed improvements to the experimental settings are included.  \n\n[3] S. Aaronson and H. Kirchner. Watermarking GPT outputs, 2023. URL https://\nscottaaronson.blog/?m=202302.\n\nLIMITATIONS:\nNothing to report", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"Learning to Watermark\" (LTW), a novel selective watermarking framework for Large Language Models (LLMs) that leverages multi-objective optimization to balance watermark detectability and text quality. Traditional watermarking methods often face a trade-off between these two objectives, leading to degraded text quality or reduced detectability. LTW addresses this by employing a lightweight neural network, the Selector Network, which adaptively decides when to apply watermarks based on sentence embeddings, token entropy, and the current watermarking ratio. The network is trained using two specifically designed loss functions that guide it toward Pareto-optimal solutions, ensuring high detectability without compromising text quality.\n\nThe authors evaluate LTW by integrating it with two baseline watermarking methods, KGW and Unigram, and demonstrate significant improvements in text quality while maintaining or even enhancing detectability. Experimental results show that LTW achieves lower perplexity and higher cosine similarity with reference texts compared to existing methods, indicating better semantic coherence and text quality. Additionally, LTW exhibits superior robustness against paraphrase attacks, making it a promising approach for practical LLM watermarking applications. The paper contributes a new perspective on selective watermarking strategies, emphasizing the importance of adaptive decision-making in maintaining both security and usability.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper introduces a novel selective watermarking framework, LTW, which leverages multi-objective optimization to balance watermark detectability and text quality. This approach is innovative and addresses a significant challenge in the field of LLM watermarking.\n\n- The authors employ the Multiple Gradient Descent Algorithm (MGDA) to solve the multi-objective optimization problem, ensuring that the framework achieves Pareto-optimal solutions. This method effectively harmonizes the competing goals of high detectability and high text quality.\n\n- The paper provides extensive experimental evaluations across multiple models and datasets, demonstrating that LTW significantly enhances text quality without compromising detectability. The results are compelling and show that LTW outperforms existing watermarking methods in both text quality and robustness.\n\n## Weaknesses \n\n- The paper only evaluates the watermarking method against a single type of attack, specifically the Dipper Attack. It lacks experiments on other types of attacks, such as scrubbing and spoofing, which are important for a comprehensive understanding of the method's robustness.\n\n- The experiments are limited to one dataset, which may not be sufficient to demonstrate the generalizability of the proposed method across different datasets and domains.\n\n- In Table 3, as the threshold increases, the effect of the selector appears to diminish. The paper does not include ablation studies on other components of the model, making it difficult to assess the individual contributions of each module to the overall performance.\n\nQUESTIONS:\n1. The current model primarily controls text quality and watermarking capability through loss functions. Does the weight of different losses have an impact on the model's performance? Additionally, does the number of tokens selected or the window size used in the model affect its effectiveness?\n\n2. How does the model ensure robustness against paraphrasing or deletion attacks? It seems that the method does not explicitly consider these types of attacks, which could be a potential vulnerability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}