Title: HeavyWater and SimplexWater: Distortion-free LLM Watermarks for Low-Entropy Distributions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language models

watermarking

coding theory

heavy-tail distributions
Abstract: Large language model (LLM) watermarks  enable authentication of text provenance, curb misuse of machine-generated text, and promote trust in AI systems. Current watermarks operate by changing the next-token predictions output by an LLM. The updated (i.e., watermarked) predictions depend on random side information produced, for example, by hashing previously generated tokens. LLM watermarking is particularly challenging in low-entropy generation tasks -- such as coding -- where next-token predictions are near-deterministic. In this paper, we propose an optimization framework for watermark design. Our goal is to understand how to most effectively use random side information in order to maximize the likelihood of watermark detection and minimize the distortion of generated text. Our analysis informs the design of two new watermarks: HeavyWater and SimplexWater. Both watermarks are tunable, gracefully trading-off between detection accuracy and text distortion. They can also be applied to any LLM and are agnostic to side information generation.  We examine the performance of HeavyWater and SimplexWater through several benchmarks, demonstrating that they can achieve high watermark detection accuracy with minimal compromise of text generation quality, particularly in the low-entropy regime. Our theoretical analysis also reveals surprising new connections between LLM watermarking and coding theory.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to design LLM watermarks that are effective even in low-entropy conditions. They model watermarks as modifying the distribution of each token in the response using some side information (for example, in the Kirchenbauer et al. scheme this side info is the output of the hash function). To do so, they formulate constructing a good watermark as an optimization problem which essentially involves maximizing the expectation of the score function applied to the response and the actual side information, and minimizing the expectation of the score function applied to the response and independent side information. This gap in expectations arises because the watermark modifies the response to be correlated with the side information.\n\nThere are two important choices this paper considers in instantiating such schemes: the score function, and the way in which the watermark modifies the distributions. The way in which the watermark modifies the distributions is derived from the score function, using Sinkhorn’s algorithm. This paper takes constructing the side information to be outside its scope. This paper proposes two instantiations of the scheme: HeavyWater and SimplexWater. SimplexWater uses a binary simplex score function rooted in coding theory. HeavyWater is an improved version of SimplexWater that uses a continuous score function. The schemes also include a tunable parameter that controls how much distortion the watermark introduces.\n\nIn addition to proving properties of the proposed schemes, this paper shows empirically that their detectability out-performs existing schemes, especially on coding models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This paper nicely formalizes the problem of constructing a good watermark in terms of an optimization task. It then shows that solving this problem yields practical watermarks.\n- This paper has both theoretical and empirical evidence for the effectiveness of the watermark.\n- The paper is clearly written.\n\nWeaknesses\n- The watermark scheme involves solving a *different instance* of an optimization problem to determine how to modify the distribution of each token in the response. In practice, this seems like it would be quite expensive. In contrast, existing schemes involve changing each distribution according to a simple rule, such as increasing the probabilities of tokens on a green list. \n- Due to the above, the paper should include more discussion about the efficiency of the proposed scheme.\n\nOverall, the paper shows a nice theoretical perspective on the task of watermarking. This perspective yields (seemingly) practical schemes that perform well empirically, in certain low-entropy settings. Since low-entropy settings have posed a challenge to watermarks, this contribution has good significance if the schemes are truly practical (see question about efficiency below). The quality seems reasonable, and the paper is reasonably clear. The main question I have is about efficiency.\n\nQUESTIONS:\n- How computationally expensive is the watermark generation algorithm/Sinkhorn’s algorithm?\n- In Figure 3(b), what is being measured on the y-axis? It is labeled “improvement,” but under what metric is this improvement?\n- The experiments seem to focus on two kinds of tasks: Finance-QA (Q&A) and LCC (coding). There are many other low-entropy scenarios, though. What evidence do you have that your watermark performs well in low-entropy scenarios in general?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper considers the important problem of LLM watermarking in the case of low-entropy regimes such as in coding tasks. In particular, the question of high detectability with low distortion is tackled and two algorithms are proposed by invoking connections to coding theory and optimal transport. Experiments validate the approaches on various LLMs and settings.

STRENGTHS AND WEAKNESSES:
Overall, the paper is very interesting and introduces novel ideas in the watermarking literature.

Pros: 

(A) In the case of binary score functions, a connection to the coding theory is made and shown to be minimax optimal. This encompasses the popular red-green settings.
(B) For score functions which are continous, a framework is introduced to tackle this settings and includes the popular Gumbel approach but beats it in performance.
(C) In both settings, they are adequately supported by theoretical results and quite a few experimental results.

Cons:

(i) The watermarking schemes target detectability and low-entropy regimes but it is unclear how they perform along the robustness and security angle.
(ii) The performance metrics are limited to detection and distortion, and downstream utility metrics of the generated code are not presented.

QUESTIONS:
(1) Is there a performance metric for the quality of the generated code? Are there sample outputs for the coding tasks before and after watermarking?
(2) How easy is it to scrub/spoof the approach? Also, how robust is it various types of attacks such as substitution, deletion, and addition of tokens?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a minimax optimization framework for large language model watermarks. It aims at maximizing watermark effectiveness while minimizing text distortion under low-entropy constraints. This work designs two watermark methods, SimplexWater and HeavyWater, derived from the optimization framework. The former is a binary-score watermark that achieves the optimal detection gap, based on the observation that the minimax problem could be formulated as designing codewords with maximal Hamming distances. The latter is a generalized continuous-score watermark that samples watermark scores from heavy-tail distributions to improve watermark detectability. Experimental results show that the proposed methods achieve better performances in terms of effectiveness-distortion trade-off than existing LLM watermarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n+ This paper proposes a unified optimization framework for LLM watermarks. The optimization framework includes watermark effectiveness, text distortion, and low-entropy constraints, and would help provide theoretical insights into LLM watermarks.\n+ This paper further proposes two concrete watermark implementations under the proposed framework. Both watermarks are distortion-free and could work under different settings.\n+ Apart from empirical results, the two implementations also contribute to additional theoretical insights: SimplexWater establishes connections between the watermark framework and coding theory; HeavyWater shows that continuous-score watermarks could benefit from sampling from heavy-tail distributions.\n\n**Weaknesses**\n\n- **The i.i.d. requirement for the side information might be restricted in real-world scenarios**. It would add to the difficulties of sharing the secret information between the watermarker and the detector, and might lead to vulnerabilities against reverse engineering. For example, the authors mention in the appendix that a counter and a secret key are used to generate the i.i.d. side information. In this case, the watermark behavior is only tied to the i-th token's position (i.e., the counter value). Each token at the same location would have similar watermark behavior, regardless of the context. Consequently, the watermark might be vulnerable to reverse engineering, where an adversary queries the watermarker multiple times and counts word frequencies at each position to infer high-score watermark tokens. Nonetheless, the authors do mention this in the limitation section and include results using more realistic non-i.i.d. side information. While the empirical results are promising, some of the theoretical analysis might not hold in non-i.i.d. cases.\n- **Additional tasks and metrics for low-entropy generation could be included to further evaluate generation quality**. While this paper features low-entropy watermarking, the evaluation for low-entropy generation is limited to code generation (on the LCC dataset), and the main performance metric for generation quality is limited to the estimated Cross Entropy. While indeed additional metrics are included in the appendices, they mostly focus on general text generation tasks rather than low-entropy ones. Hence, it would be more convincing if the evaluation could be extended to include more low-entropy tasks and metrics (e.g., pass@k on code generation or accuracy on math reasoning, etc.).\n- **The robustness evaluation does not consider stronger removal attacks**. The watermark removal attacks have mainly focused on rather primitive attacks, such as changing cases and adding typos. Other attacks, such as word substitution, copy-paste attacks, or paraphrasing, could also be included for a more comprehensive evaluation. This is a rather minor issue since robustness is not the primary focus of this work.\n\nQUESTIONS:\n- Is it possible for an adversary to mount reverse-engineering or spoofing attacks by observing token frequencies of SimplexMark or HeavyMark, especially under the uniform i.i.d. assumption?\n- How do SimplexMark and HeavyMark perform (in terms of downstream task metrics) on low-entropy generation scenarios, such as code generation (pass@k) and math reasoning (accuracy)?\n- How do the proposed methods perform against more advanced watermark removal attacks?\n- Another question following the previous one on robustness: could robustness be potentially integrated into the optimization formulation as an objective?\n- Could this optimization framework or the two proposed methods be potentially extended to multi-bit watermarking?\n\nA few other minor issues:\n\n1. Markov-1, Product, h=4, Sum, h=5 in Figure 3(b) is not defined in the main text (although the definitions could be found in the appendices).\n2. Some figure titles seem to be inconsistent. E.g., in Figure D.13, do \"Coding\" and \"LCC\" refer to the same dataset (LCC)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
his paper proposes two novel watermarking techniques, HeavyWater and SimplexWater, to address the challenges of reliably detecting machine-generated text from large language models (LLMs), particularly in low-entropy scenarios like code generation. HeavyWater employs heavy-tailed score distributions to enhance detectability while minimizing distortion, while SimplexWater leverages simplex codes from coding theory to maximize the Hamming distance between watermarked tokens, proving optimal for binary score functions. The authors frame watermark design as an optimization problem, linking it to optimal transport theory, and demonstrate that their methods outperform existing approaches (e.g., Gumbel, Red-Green watermarks) in detection accuracy and robustness, especially for low-entropy text.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper presents a rigorous and thorough theoretical analysis.
2.  The paper extend binary score system to a continuous distribution for better performance.

Weaknesses:
1. The baselines listed lack some previous works that are specially designed to tackle low-entropy scenarios.
2. The paper could demonstrate the entropy difference between the used datasets and high-entropy datasets such as C4 for better comprehension. And shouldn't the models include coding LLMs such as starcoder?
3. The experiments should also include performance of high-entropy scenarios.

QUESTIONS:
How is the code pass@k in the experiments?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06