Analyzing and Evaluating Unbiased Language Model Watermark
Authors: Yihan Wu
Xuehao Cui
Ruibo Chen
Heng Huang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM watermarking
Primary area: generative models

Abstract: Verifying the authenticity of AI-generated text has become increasingly important with the rapid advancement of large language models, and unbiased watermarking has emerged as a promising approach due to its ability to preserve output distribution without degrading quality. However, recent work reveals that unbiased watermarks can accumulate distributional bias over multiple generations and that existing robustness evaluations are inconsistent across studies. To address these issues, we introduce UWBench, the first open-source benchmark dedicated to the principled evaluation of unbiased watermarking methods. Our framework combines theoretical and empirical contributions: we propose a statistical metric to quantify multi-batch distribution shift, prove an impossibility result showing that no unbiased watermark can perfectly preserve the distribution under infinite queries, and develop a formal analysis of robustness against token-level modification attacks. Complementing this theory, we establish a three-axis evaluation protocol—unbiasedness, detectability, and robustness—and show that token modification attacks provide more stable robustness assessments than paraphrasing-based methods. Together, UWBench offers the community a standardized and reproducible platform for advancing the design and evaluation of unbiased watermarking algorithms.

Published as a conference paper at ICLR 2026
ANALYZING AND EVALUATING UNBIASED LANGUAGE MODEL WATERMARK
Yihan Wu,, Xuehao Cui', Ruibo Chen, Heng Huang"
Department of Computer Science University of Maryland, College Park {ywu42, cedrus, rbchen, heng) } @ emu.edu
ABSTRACT
Verifying the authenticity of Al-generated text has become increasingly important with the rapid advancement of large language models, and unbiased watermarking has emerged as a promising approach due to its ability to preserve output distri- bution without degrading quality. However, recent work reveals that unbiased watermarks can accumulate distributional bias over multiple generations and that existing robustness evaluations inconsistent across studies. To address these issues, we introduce UFNH,,,, the first open-source benchmark dedicated to the principled evaluation of unbiased watermarking methods. Our framework combines theoretical and empirical contributions: we propose a statistical metric to quantify multi-batch distribution drift, prove an impossibility result showing that no unbiased watermark can perfectly preserve the distribution under infinite queries, and develop formal analysis of robustness against token-level modification attacks. Comple- menting this theory. we establish a three-axis evaluation protocol unbiasedness, detectability, and robustness- more stable robustness assessments than paraphrasing -aadd methods. Together, UWBENCH offers the community a standardized and reproducible platform for advancing the design and evaluation of unbiased watermarking algorithms. Codes git
show that token modification attacks provide
are available at /tttt::://gttbb com/
INTRODUCTION
As the capabilities of large language models have grown significantly in recent years, verifying the authenticity and origin of Al-generated content has become increasingly critical. Watermarking language models (Aaronson) 2022 Kirchenbauer et al., 2223a Christ et al. 2023 Kuditipudi et al.) 2023: Hu et al. 2023) Wu et al. 20233 Chen et al. 2024a b 2025 Mao et al. (2024] Dathathri et al., promising solution for distinguishing machine-generated text from human- authored content. These methods embed covert statistical signals into the generation process using specific keys, allowing downstream detection via statistical hypothesis testing to verify authorship without degrading fluency.
2024 has emerged
particularly important class of these methods is unbiased watermarking, which aims to preserve the original distribution of the language model's 's outputs. Such methods are crucial for practical deployment since they do not introduce detectable distortions or degrade generation quality (Aaronson 2022; Christ et al. (2023] Kuditipudi et al. 2023 Hu et al. 2023] Wu et al.) 2023 Chen et al. 2025 Mao et al. 2024 Dathathri et al. (2024). However, recent studies have revealed important limitations. While unbiased watermarks may preserve the output distribution in expectation, their statistical properties can drift over multiple generations, leading to distribution bias that violates the original unbiasedness guarantees (Christ et al., 2023) Kuditipudi et al.) 2023) Hu et al., 2023). Moreover, robustness evaluations in prior work are fragmented: different methods are tested against different adversaries using inconsistent protocols, leaving a gap in standardized, comparable assessment.
* "quual contribution
This work was partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617, RISE 2536663.

---

Published as a conference paper at ICLR 2026
Unbiasedness Detectability Robustness
Symthl6
MC
MCmarkkte
SynthID
MCmark((-10)
Detestability
MCmarktt-29)
KGW()rz)
Heeweight
(6igna((-2)
0.7
EXP-edii
DiPanak 0.4
DiPmerkin 0.4)
K-rewcight
EXP-alit
ITS-alit
TS-elix
0.58
0.90
0.92
0.94
0.96
0.98
Unbiasedness
Figure 1: Overall benchmarking results of unbiasedness (x-axis), detectability (y-axis), and robustness (encoded with marker size) on different language model watermarking methods. Points further to the right and higher indicate better unbiasedness and detectability;; larger markers indicate greater robustness.
To address these challenges, we introduce UW BENCH, the first open-source benchmark specifically designed for the analysis and evaluation of unbiased watermarking algorithms. Our framework offers both theoretical foundations and practical tools to facilitate principled comparisons. On the theoretical front, we propose a statistical metric that quantifies distributional shift across batches of generated texts, enabling evaluation of long-term bias. We further prove general impossibility result: no unbiased watermark can perfectly preserve the model's ss output distribution under an infinite query budget. Finally, we develop formal framework for analyzing the robustness of unbiased watermarking algorithms against token-level modification attacks, showing that such attacks can be resisted under certain structural assumptions.
In addition to the theoretical contributions, we provide a comprehensive empirical toolkit for bench- marking existing and future unbiased watermarking algorithms. We establish three-axis evaluation protocol-ung unbiasedness, vrotooo--umbiseedness. detectability, and robustness that provides holistic view of watermark per- formance. Notably, we revisit common adversarial attacks and demonstrate that paraphrasing based evaluations suffer from high variance and inconsistent results, potentially leading misleading con- clusions. In contrast, token modification attacks yield more stable and reliable robustness assessments, making them preferred choice for empirical benchmarking.
Our main contributions are summarized as follows:
We introduce UWBENCH, an open-source benchmark designed specifically for evaluating unbiased watermarking methods in language models, with support for systematic and reproducible comparisons.
We propose a multi-batch distribution bias metric and prove a fundamental limitation: no unbiased watermark can preserve the model's output distribution under unlimited queries. We also develop a theoretical framework for analyzing robustness against token-level attacks.
We establish a three-axis evaluation protocol- unbiasedness, detectability, and robust- ness ness-and show that token modification attacks offer more stable and reliable robustness assessments than paraphrasing-ban -basss attacks.
RELATED WORK
Statistical watermarks. Kirchenbauer et al. (2023a) enhanced the statistical watermark framework originally introduced by] Aaronson (2022), demonstrating the effectiveness of statistical watermarking through extensive experiments on large language models. They splited the LM tokens into red and green list, then promoted the use of green tokens by adding fixed parameter å to their logits. Zhhao et al. (2023) proposed the unigram watermark, which enhances the robustness the statistical watermark by using one-gram hashing to produce watermark keys. Liu al. (2023b) also improved
2

---

Published as a conference paper at ICLR 2026
the robustness of statistical watermarking by leveraging the semanties of generated content as watermark keys. Additionally, [Liu et al. (2023a) proposed an unforgeable watermark scheme that employs neural networks to modity token distributions instead of using traditional watermark keys. However, these approaches may lead to significant changes in the distribution of generated text, potentially compromising content quality.
Unbiased watermarks. To maintain the original output distribution in watermarked content, several researchers have investigated novel approaches for token distribution modification. AAronson (2022)
pioneered an unbiased watermarking method using Gumbel-max to adjust token distribution and employing prefix n- n-grams as watermark keys. Christ et al. 2023) used inverse sampling for modifying the token distributions of watermarked content on a binary language model with watermark keys based on token positioning. ITS--edit and EXP-edit Kuditipudi et al. (023) utilized swwerse-sampling and Gumbel-max respectively for modifying the token distributions of watermarked content, with a predetermined watermark key list. Hu al. (2023) combined inverse-sampling and y-reweight strategies for watermarking, though their detection method is not model-agnostic. DiPmark Wu et al. (2023) enhanced the 7-reweight technique and introduced a model-agnostic detector. STA-I Mao et al. (2024) optimized the quality of the watermarked text under the low-entropy scenarios, Dathathri et al. 20244 proposed SynthID, which enables distortion freeoons--eeeessssse of LM watermarking with multiple generations. Chen et al. (2025) introduced MCMark, which significantly improved the detectability of the unbiased watermark. MC Mark (Cui et al. 2026) further increased the watermark encoding capacity to enable multi-bit watermarking.
LM watermarking benchmarks. WaterBench (Tu al. al. (023) provides comprehensive benchmark for LLM watermarking methods. It standardizes watermarking strength by tuning each method's hyperparameters to common level, and then jointly evaluates both generation quality and detection performance. MarkMy Words (Piet et al. 2023) evaluates LLM watermarks along three dimensions: generation quality, detection efficiency (measured by the number of tokens required), and robustness. MarkI.LM (Pan et al., 2024) introduces an open-source toolkit that offers a unified, extensible framework for implementing L.M watermarking algorithms, along with user-friendl, interfaces to facilitate broader adoption. However, most of the watermarking methods covered in these benchmarks are biased. They lack thorough analysis of unbiased watermarking techniques, and do not include evaluation metrics specifically designed for them. As such, we argue that a dedicated benchmark for unbiased watermarking is both necessary and timely.
PRELIMINARY
3.1 MOTIVATION
Statistical watermarking has emerged as ceneerl-purpose solution for verifying the authenticity of Al-generated content. Unlike task-specific benchmarks, statistical watermarking can be applied to any language model and across any downstream task without the need for collecting task-dependent datasets. Thus, for evaluating unbiased watermarking methods, building a new dataset is unnecessary and does not address the core challenges. Instead, the true value of a benchmark lies in providing principled and reliable metrics for assessing watermark performance.
Current unbiased watermarking methods are typically evaluated along three axes: unbiasedness, detectability, and robustness. While detectability metrics are relatively well-established, existing approaches for measuring unbiasedness and robustness are inadequate. In particular, unbiasedness has so far been evaluated under a single-prompt setting, which overlooks important failure cases. We theoretically prove a fundamental impossibility: no watermarking scheme can remain unbiased when the same prompt is repeatedly queried. Motivated by this result, propose a new metric that quantifies distributional bias under repeated queries, offering a more faithful measure of unbiasedness.
Robustness evaluation presents another challenge. Most existing work relies on seaaphrasing-based adversarial attacks. However, these methods suffer from high variance and inconsistent results (See Figure 2). leading unreliable conclusions. To overcome this limitation, we combine the paraphrasing- braaphrasing-based adversarial attacks with the random token modification attacks that provides stable and reproducible assessments robustness.
3

---

Published as a conference paper at ICLR 2026
In summary. UW Bench shifts the focus of watermarking evaluation away from task-specific datasets and toward theoretically grounded, reproducible, and holistic performance metrics that better capture the limitations and strengths of unbiased watermarking algorithms.
3.2 WATERMARKING SETTING
Problem Definition. A language model (LM) provider aims to watermark generated text so that any user can later verify its origin, without access to the LM or the original prompt. A watermarking framework consists of two components: watermark generator and watermark detector. The generator embeds hidden statistical signals into the text, while the detector recovers these signals using hypothesis testing.
Watermark Generator: Let Pas (: Ximm) ) denote the LM's distribution for predicting the n-th token given prefix I:.. A watermark key k E K and reweight strategy F are used to construct the watermarked distribution Pw (: Fi k) = F [PPM(-1 æ1:n), ).) k)) The next token Tn is then sampled from Pw instead of Pas The watermark key typically includes secret key sk and context key (e.g.. n- n-gram index [Aaronson] 2022) or token position (Christ et al. 2023)). This process injects a subtle statistical signal into the generated text.
The reweight strategy is the core of watermark generation. A strategy is called distortion-free if the resulting Pw preserves the original distribution Pa. To date, three main families of distortion free strategies have been proposed: (i) inverse- inverse-sampling (Christ et al., 20233; Kuditipudi et al., 2023
Hu et al., 2023). (ii) cammbee-rpaaammetrizaiooo (Aaronson) 2022; Kuditipudi et al. 2023), and (iii)
permute-reweight (Hu et al. 2023).
Definition (Unbiased Watermark), watermarking scheme is unbiased if, for any context 1i the expected distribution of the next token under watermarking matches the original LM distribution:
E E Pw xi.. k) = PMr æi:n),
Where is the watermark key distribution. In other words, averaging over random watermark keys does not introduce any systematic distortion into the model's output distribution.
Watermark Detector. The detector only has access to the watermark key k and the reweight strategy F. Detection is posed as a hypothesis test: Ho : Text is unwatermarked, H1 Text is watermarked. To test this, a score function s V x K x F + R. is applied token by token. For sequence Vim- the test statistic ((x11.)) = Σ s(r, k, F). (( significantly deviates from expected value under Ho. the null hypothesis is rejected and the text is declared watermarked.
4 UWBENCH
4.1 UNBIASEDNESS UNDER REPEATED PROMPTS
Unbiasedness (one-shot). Let PM æ) be the LM distribution for prompt æ and let Pw æ, k) be the distribution induced by watermark with key k p. We say the watermark is unbiased in the one-shot sense if [ k)] PMr a)) for all prompts x.
Impossibility under repeated prompts. We next state our main impossibility for repeated queries of the same prompt under a fixed key (proof deferred to the appendix).
Theorem 4.1 (Unbiasedness breaks under repeated prompts). No watermarking scheme can simulta- neously satisfy: i) preservation of the original LM distribution under repeated queries of the same prompt with a fixed key k; and ii) detectability. Equivalently, any detectable scheme that is unbiased in the one-shot sense equation fails to preserve PM when the same prompt is queried repeatedly under fixed key.
Single-prompt multi-generation (SPMG) unbiasedness metric. Theorem 4. motivates mea- suring distributional deviation when single prompt is queried multiple times with fixed key. Let Pi..... Pa. be n prompts. For each p., draw m independent generations from a model P (fixing decoding settings; for watermarks, the key is held fixed across the m draws). Let Met(-) be any
4

---

Published as a conference paper at ICLR 2026
bounded per-generation performance surrogate (e.g.. perplexity, average log likkliihoddd reward score),
with [Met(g)] Σ A. Define the per-prompt SPMG mean: Met, = Ey= Met g (P)), and
the SPMG gap between two models P and Q: AMet(P.Q)
4/4 D== Met, Mett(())
Intuitively. AMet(PM Pr) captures the multi-sample deviation of a test model Pr from the original Pa that emerges only when the same prompt is queried repeatedly.
"arialee-controolled detection statistic. To factor out natural sampling noise, we compare the test model to an i.i.d. clone of the original model. Let Par be independent model with the same distribution Pas. Define the detection statistic
Det DetWmn((H (PM, Pr) = AMet(PM,
AMet(PM, Par.).
Large positive values indicate aepeated-poomp shift beyond the intrinsic variance of Pas.
Theorem 4.2 (McDiarmid concentration for for SPMG detection). Suppose Pr is identically dis- tributed with Pas and [Met(g)] I A for all generations g. Then for any 0,
mn t?
Pr Pr( (PM, Pr) PPM,
s exxp
(2)
12A²
Equivalently, with probability at least -,
12 12g/(22/))
Det Pas, Pr) B[DetWmm(PP:: Pr)ll
< A
mi
control false positives when testing for repeated soppetted-prompt prompt bias. Consequently, SPMG- based evaluation isolates the distributional drift that Theorem 4.1 predicts, while providing sample guarantees for reliable detection.
Inequality 2 yields an a-level threshold ta A2
12 (1(aa))
mn
4.2 ROBUSTNESS ANALYSIS OF UNBIASED WATERMARKS
Adversary model. During detection, only the text sequence is available to the verifier; hence an adversary can act solely by modifying tokens. We consider an edit-bounded adversary that applies up to b token operations
producing an attacked sequence x'. Let additive test statistic S(x) ET 1 sz(x) with decision threshold (reject Ho if
the detector ((x) 2 T). All scores are assumed bounded: se (x E B).
Limitations of existing attack protocols. Prior works evaluate robustness with random token edits (Kirchenbauer et al.] 2023a|bb b). paraphrasing (Kirchenbauer et al.) (023b), and translation (He al. (0024). These are imperfect for benchmarking: (i) random edits often severely degrade semantic quality; (ii) paraphrasing exhibits instability across prompts and seeds; (iii) translation too strong: it changes essentially all tokens, so unbiased watermark can survive, making methods indistinguishable. This motivates a principled, token-level robustness characterization with certificates.
Token effect region. Let C, (æ)) denote the context used by the detector to score token (e.g., an n-gram prefix or a rolling, prefix-dependen key schedule). A modification at position can affect the scores for all tokens t whose context uses 24, i.c., {t 2, E C,())). Define the token effect region length of position i by ((( = (1 i: x; E (a+) For detectors keyed by n-gram prefix, (x) < n + 1 (only L E i + n) are influenced). For position-key schedules that depend on the entire prefix (rolling hash), (x(x))
-it (all suffix tokens can be influenced). Let
Rmax == max; R2 (z))
Expected score decrease under edit. Write the detector S(x) - (xx)) and let A; denote the expected reduction in caused by editing token i, where the expectation is taken over the randomized alignment (c.g., color assignment bit tests) induced when the context is destroyed in the affected region. Then E [[S(x) S(x!")) Et:,,e,(a)) (B[s(())] E[s (x)))) Instantiations
for common unbiased watermark families:
5

---

Published as a conference paper at ICLR 2026
Green-count detectors (e.g., r--reeeihht DiPmark, STA): 8 {0, 1) indicates whether token falls in the green set. Let Pg be the (empirical) fraction green tokens under watermarking. Destroying alignment in the effect region makes green assignment effectively random, yielding expected per-token drop of (2Pc 1)/2. Hence for one edit with effect length R. IE [{((x)) S(w()) (2Pc-1) R.
2
Sytth bit tests: Each token contributes m binary scores, 8, = - &t,e with Sr, E {0, 1}. Let P. == E[se] under watermarking. Randomized alignment drives each bit to mean 1/2, so the expected per-token drop is (P: 2 yielding E. (((x)))) S(x(0)) (P. m R.
Certified robustness. Because each single-toker edit can affect at most Rmax token scores and each token score changes by at most B, the test statistic Lipschitz w.r.t. edit distance S(x) - S(x') S b Rinax B for any b-edit attack. Hence we obtain an En certified radius:
S(x) - > b Remax B
S(x') 2 T for all a' with < b edits.
(3)
This bound holds without distributional assumptions (worst-case guarantee).
5 EXPERIMENTS
Our evaluation organized along three axes. (i) Unbiasedness: we measure watermark- ing unbiasedness in one-shot settings (machine translation and text summarization tasks; BLEU/ROUGG/BERTTSOOr) and quantify repeated-promp distribution shift via the SPMG met- rics AMct and the calibrated statistic DetWmk. (ii) Detectability: on open-ended genera- tion (C4/MMW/Dolly CW/WaterBench) we report TPR at theoretically guaranteed FPR levels sets across Llama-3.2-3B- (iii) Robusiness: We use paraphrasing
(5%, 1%, 0.1%) and AUC using matched watermarkeednnwaaerraaaa Instruct, Mistral-7B- Binaal-7--Instuutt-doo and attack and random token modification under edit budgets. Detailed setups and hyperparameters are in Appendix C
Baselines. We compare against representative unbiased watermarking algorithms: 7-reweight (Hu
al. 2023), DiPmark (Wu et al. 2023), MCmark (Chen al. 2025). SynthID (Dathathri al.
2024 Edit (Kuditipudi et al. 2023). 3. EXP-Edit (Kuditipudi et al. 2003). and STA-1 (Mao et al.
2024.. Besides, we add two popular biased watermark: KGW Kirchenbauer et al. (2023a) and
Unigram Zhao et al. 2023 as additional baselines.
Unbiasedness. Following Hu et al. (2023); Wu et al. 2003)), we compare task metrics between the original LM and its watermarked counterpart: Machine translation: BLEU and BERTScore
WMT16 RO-EN; Text summarization: ROUGE-1/2/L and BERTScore on CNN/DAULYMATI We evaluate with (a) 1000 prompts, one generation per prompt (Table [ and (b) 10 prompts, 1000 generations per prompt (SPMG, Table To measure popeated-prompl bias, we adopt the SMPG gap A.Mtt(P, Q) and report the calibrated statistic Det Wek PT) == AMet( Pr)
AMct(PMr Par) with bounded Met (e.g. perplexity, or bounded quality scores).
Detectability. Open-ended generation on C4/MMW/DOLLY CW/WATERBENCH 1000 prompts, one generation per prompt. For each method we build matched sets of watermarked and unwater- marked texts (same prompts, decoding settings). We compute: (i) TPR at target FPR (5%, 1%, 0.1%) using analytic thresholds from each detector's null; (ii) Median p-value generated by the detection algorithm: (iii) AUC on balanced datasets (same number of positive/negative sequences). Unless otherwise stated, we fix generation lengths around 500 tokens per dataset.
Robustness. We evaluate watermark robustness under two categories of paraphrasing based attacks. DIPPER is a strong neural paraphraser that rewrites text while preserving semantic meaning, thereby introducing substantial variability into the generated outputs. In contrast, Random token replacement attacks directly perturb the text by substituting a fixed percentage of tokens (10% 20%, or 30%) with randomly sampled alternatives. While random replacements offer simple, noise-driven baseline for robustness testing, DIPPER provides a more realistic and challenging paraphrasing scenario that better reflects practical adversarial conditions.

---

Published as a conference paper at ICLR 2026
Table 1: Unbiasedness evaluation a) (1000 prompts, generations each). We evaluate the unbiasedness of watermarking methods on text summarization and machine translation tasks.
Method
Text Summarization
Machine Translation
BERTScore
ROUGE-1
Perplexity
BERTScore
BLEU
No watermark
0.3077
0,3807
6.39
0.5432
20.1681
0.3080
0.3773
0.5436
20.0175
Unigram(0=1.00 (=0.0)
0.3053
0,3775
6.85
0.5388
20.1276
Unigram (d=.5)
0.2955
0.3656
7.51
0.5307
19.5000
Unigram(d=2.0)
0.2848
0.3566
8.28
0.5191
18,4838
KGW(6==0.5)
0.3012
0.3757
6.52
0.5472
20.6198
KGW(5=1.0)
0.2977
0.3751
0.5348
19.9166
KGW(5=1.55)
0.2876
0.3686
7.56
0.5326
19.2318
KGW($=2.0)
0.2769
0.3619
8.37
0.5218
17.9401
DIP(a=0.3)
0,3082
0.3793
6,41
0,5422
20.2514
DIP(a=0.4)
0.3081
0.3781
6.53
0.5446
20,4579
y-reweight
0.3032
0.3749
6.49
0.5394
20.5546
MCmark(n=10)
0.3032
0.3755
0.5416
20.4171
MCmark(n=20)
0,3054
0,3780
6.41
0,5486
20.0984
MCmark(n=50)
0.3099
0.3810
6.45
0,5400
20.1503
(Cmark(n=100)
0.3080
0,3800
6.46
0.5466
20.6732
STA-1
0,3066
0.3793
6.25
0,5492
20,5561
SynthID
0.3049
0.3775
6.37
0.5445
20.4107
EXP-Edit
0,3114
0.3797
6.19
0.5458
20.4879
ITS-Edit
0.3032
0.3749
6.58
0.5091
17.9904
Table 2: Unbiasedness evaluation b) (10 prompts, 1000 generations each) We evaluate the unbiasedness of watermarking methods on text summarization and machine translation tasks with SPMG metric.
Method
Text Summarization
Machine Translation
BERTScore
ROUGE-1
Perplexity
BERTScore
BLEU
No watermark
0.0026
0.0017
0.1828
0.0033
0.1199
Unigram(5=0.55
0.0037
0.0033
0.2197
0.0074
0,8311 1
Unigram(8=1.0)
0.0076
0.0076
0.6133
0.0181
1.7137
1n)
0.0146
0.0148
1.3664
0.0293
2,8560
Unigram(5=2.00
0.0255
0.0256
2,4671
0.0423
3,9869
KGW(8=0.5)
0.0040
0.0023
0.1051
0.0041
0.5211
KGW(6=1.0)
0.0093
0.0062
0.409S
0.0106
1.1383
KGW(5=1.5)
0.0177
0.0121
0,9434
0.0178
1,5189
KGW(8=2.0)
0.0297
0.0199
1,9382
0.0232
2.0037
DIP(o=0.3)
0.0050
0.0039
0.0484
0.0147
1,2311
DIP(4==04)
0.0067
0.0059
0.1772
0.0149
1.6594
'Y-rewiight
0.0071
0.0081
0.1570
0.0174
1,9001
MCmark(u=10))
0.0073
0.0033
0,2456
0.0171
1.5756
MCmark(n=20))
0.00666
0.0037
0.2914
0.0162
0,9958
(Cmark(n=500)
0.0069
0.0076
0.2771
0.0153
0.7727
MCmark(n=100))
0.0080
0.0077
0.3068
0.0234
0.6486
STA-1
0.0046
0.0035
0.1505
0.0107
0,8446
SyathID
0.0159
0.0227
0.8254
0.0377
2.5266
EXP-Edit
0.0422
0.0413
2,0032
0.0439
2,4104
ITS--ddit
0.0355
0.0533
1.4912
0.0679
5,0746
Paraphrasing variance. Using DIPPER paraphrasing, generate paraphrases per input (multi- ple seeds and temperatures), forming matched sets for each method. We report the mean + std of TPR @ FPR and AUC across seeds and show per-prompt variance distributions. As shown in Figure
7

---

Published as a conference paper at ICLR 2026
Table 3: Averaged detection performance across all language models and datasets by method. We also include two biased watermarks: KGW and Unigram for reference.
Method
@ FPR=5%
TPR@FPR=19
TPR@FPR=0.1%
median p-value
AUROC
Unigram(6=003
68.7%
$3.59%
35.55%
3.72e-02
0.803
Unigram(==1.0)
90.04%
81.57%
69.49%
3.00e-03
0,919
Unigram(6=1.55)
96.57%
93.07%
86.96%
5.56e-05
0.960
Unigram(6=2.0))
98.89%
97.85%
94.66%
1.05e-06
0,981
KGW(5=005)
61.11%
43.9%
26.86%
7.05c-02
0,863
KGW(6=1.0)
87.07%
79.04%
68.6%
8.70e 8.70e-03
0,962
KGW(5=1.55
95.67%
91.45%
86.32%
4.46c-04
0.987
KGW(3=2.0)
98.33%
96.57%
94.04%
1.07e-05
0.995
DIP(o=0.3)
78.92%
69.26%
58.11%
2.03e-02
0,943
DIP(o=0.4)
82.61
74.11%
64.73%
1.33e-02
0.956
7-reweight
83.68%
75.85%
66.43%
9.14e-03
0,960
EXP-Edit
77.44%
72.42%
14%
5.01e-02
0,906
ITS-Edit
55. 15.11% %
48.29%
41 .6.%
1,43c-01
0.804
MCmark(n=10)
98.51%
97.09%
94.57%
4,08e-06
0,993
MCmark(n=100)
95.32%
92.2%
87.53%
5.66e-04
0,987
(Cmark(n=20))
97.82%
95.51%
92.05%
4.75c-05
0.994
(Cmark(n=500)
97.25%
95.38%
91.66%
9.56e-05
0,991
STA-1
84.55%
73.79%
59.4%
1.43e-02
0.953
SynthID
99.03%
97.29%
94.66%
6.22e-06
0,995
Std of p-values across watemmark methods by attack (FPR-0.1%)
Distribution of p-valucs across watermark methods (by attack)
Standard devistion pavaloss
0.05
0.07
6.25
0.20
0.05
povalue at FPR
0.04
6.10
002
0.0
6.00
Dipper
Dipper
Attack
Altack
Figure Variance of p-values across watermarking methods under different attack strategies (FPR=0.1%) Left: standard deviation of p-values by attack. Right: distribution of p-values (boxplots) across watermark methods.
DIPPER exhibits substantially higher variance in p-values compared to random token replacement attacks. The bar plot (left) shows that the standard deviation of p-values under DIPPER roughly four times higher than under the strongest random attack (30% token replacement). The boxplot (right) further highlights this instability: DIPPER produces a wide spread of p-values, ranging from very low to relatively high values, while random replacements lead to consistently small p-values with much tighter distributions.
5.1 A THREE-AXIS EVALUATION OF UNBIASED WATERMARK
Unbiasedness score. For each method, we quantify unbiasedness as closeness the unwatermarked baseline ("None") across metrics E [TS-BERT, ROUGE-I, Perplexity MT-BERT, BLEU).
methed None l/ I/neNone m,cig1" For Con- For Config 1, compute the relative deviation r' m,,igg m,cigi
(2)
fig 2, treat reported values as deltas and remove the baseline noise floor via r' max(0, AA method A None m.efg2 No Aggregate D1 1
M E. ,.I1) and D2
1 Σ T'
then combine D = AD1 + (1 D) (default 0.6). Finally, map to [0, 100] via the 100(1 D) (default a = 1); higher indicates greater unbiasedness (i.c., smaller average deviation from baseline and lower small-sample sensitivity).
Detectability Score. Using the averaged detection metrics per method (TPR at FPR (5%, 1%. 0.1%), ),, median p-value, AUROC), first convert TPR percentages decimals tprs, tpr, tpro.1 E [0, 1] and form a low-FPR-weighh weiihhhe operating-poin score Stpr = 0.2 + 0.3 (pri 0.5 1pro.1
Map median p-value to bounded significance score via 8p
8

---

Published as a conference paper at ICLR 2026
Table 4: Robustness (TPR@) @ 1%FPR) of watermarking method across different attack types.
Method
DIPPER
Random 30%
Random 20%
Random 10%
KGW(d=0.5)
0.42%
4.26%
7.55%
8.19%
KGW(6=1.0)
0.21%
14.90%
24.48%
38. 79.75%
KGW(d=1.5)
1.46%
37.60%
56.77%
75.10%
KGW(d=2.0)
1.35%
62.11%
78.32%
91.68%
Unigram(3=0.5)
34. 90%
47.92%
56.67%
66.25%
Unigram(6=1.0))
40.52%
64.38%
80.52%
85.00%
Unigram(6=115)
44.48%
80.21 %
89.16%
93.68%
Unigram(0=2.00)
58.85%
95.63%
98.02%
99.17%
0.3(
0.83%
2.66%
7.23%
17.98%
DIP(o=0.4)
0.42%
3.37%
7.37%
21.16%
y-rewcight
0.73%
2.53%
11.47%
26.95%
STA(>=0.5)
2.29%
4.90%
12.29%
21.56%
SynthID
3.02%
7.71%
14.58%
26.25%
ITS-Edit
1.15%
2.40%
3.96%
6.04%
EXP-Edit
0.94%
15.21%
21.46%
26.98%
MCmark(n=10)
5.10%
39.26%
73.37%
96.11%
MCmark(n=200)
3.85%
33.85%
61.56%
86.25%
MCmark(n=50))
3.96%
37.92%
63.44%
85.63%
MCmark(m==100)
3.02%
30.42%
50.52%
71.15%
min (1, | login((max((,, 10 1002000-0))) (00-2)))]/222 //222, which clips extremely small at 10 and yields sp [0, 1]. Let Sane AUROC E [o, 1]. The final detectability score is a convex combination
Detect = 100 Wtpe Stpr + Wane Sanc + Up 8p);
with default weights (Wipr, Wane; wp) = (0.60, 0.25, 0.15). Higher values indicate stronger de- tectability, with emphasis reliable detection at very low FPR while still rewarding overall separa- bility (AUROC) and statistical significance (median p).
Robustness score. For each watermarking method m, let fa [0, denote the true positive rate (TPR, decimal) under attack E (DIPPER, Random30. Random20, Random10] at false www-FPR-emphasizee per-attack operat-
positive rate f E f0.1%, 1%, 5%}. We first compute ing score (m) = 0.5 (a..0.%%(m)) + 0.3 l,, 12, (m) + 1.2 (m). These per-attack scores are then aggregated with reduced weight on DIPPER (reflecting the current study's priorities) using (UDIPPER, PRandom30, 2Random20; URaandom10 (0.2, 0.4, i5 0. 0.2666, 2 0.1333) to obtain single robustness value R(m) = E, Vo sa E [0, 1]. The final non-smoothed robustness score reported in tables is Robust Score()) 100 R(m) [0, 100]: higher values indicate stronger bustness with greater emphasis on performance at very low FPR and under the more challenging random-replacemmnt attacks.
Figure and Table jointty highlight the trade-off's between unbiasedness, detectability, and ro- bustness watermarking methods. From the scatter plot, observe that methods such as MCmark (x=10/20) and SynthID occupy the top-right corner, demonstrating strong detectability and unbiasedness, though with limited robustness (small marker size). contrast, Unigram (8=2) and KGW (6=2) achieve considerably higher robustness (large markers) but at the cost of lower unbiasedness and detectability. The tabulated scores further confirm this: Unigram (8=2) attains the highest robustness (0.855) despite relatively low detectability (0.903), whereas MCmark variants and SynthID provide balanced detectability (>0.9455) and unbiasedness (>0.965) but modest robust- ness. Notably, DiPmark and STA-1 maintain excellent unbiasedness (>0.98) but their detectabilitty lags behind (<0.72), highlighting their limitations under strict detection thresholds. Overall, these results underscore the central tension in watermark design: methods that optimize detectability and unbiasedness often sacrifice robustness, whereas highly robust methods compromise on unbiased generation quality reliable detectability.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We introduced UWBENCH, an open-source benchmark for the principled evaluation of unbiased watermarking in language models. Our theory establishes fundamental limitation: any detectable scheme that is unbiased in the one-shot sense cannot preserve the original distribution under repeated queries of the same prompt, motivating our single-prompt calibrated detection statistic for unbiasedness assessment. Experiments across diverse models and datasets demonstrate standardized, reproducible comparisons along three axes clarifying practical trade-offss and failure modes.
(SPMG) metric and
REFERENCES
Scott Aaronson. My AI safety lecture for UT effective altruism, 2022. scottaaronson blog/?p=6823
URL https://
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. /42/9, 2024.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL
http:/
www. aclweb. org http://anhoooggooooo hog W/ W1 6 -
Ruibo Chen, Yihan Wu, Junfeng Guo, and Heng Huang. De-mark: Watermark removal large language models. arXiv preprint arXiv:24100 10. 022.11...800000 13808, 2024a.
Ruibo Chen, Yihan Wu, Junfeng Guo, and Heng Huang. Improved unbiased watermark for large language models. arXiv preprint arXiv:2502.1 1268, 2025.
Yanshuo Chen, Zhengmian Hu, Yihan Wu, Ruibo Chen, Yongrui Jin, Wei Chen, and Heng Huang. Enhancing biosecurity with watermarked protein design. bioRxiv, pp. 2024-05 2024b.
Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. arXiv preprint 200.::::3006099949,, 2023.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 1lm, 2023. URL https: / www. wwtp:eaaaaaabbbeee com https///ooo/oooo//////////// 12/dolly-first-ope comme htt.::/www.......aaaaaaaaaaaaaaa
Xuehao Cui, Ruibo Chen, Yihan Wu, and Heng Huang. Mc 2 mark: Distortion-free multi-bit watermarking for long messages. arXiv preprint wrXh:2202.144300 2026.
Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. Scalable water- marking for identifying large language model outputs. Nature, (34(8033):818-82) 2024.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint 24Xi 1783, 2024.
Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, and Rui Wang. Can watermarks survive translation? on the cross-lingual consistency of text watermark for large language models. arXiv preprint arXiv:2402. /4007, 2024.
Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased watermark for large language models. arXiv preprint arXiv:2310. 10.669,,, 2023.
10

---

Published as a conference paper at ICLR 2026
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint 1rXiv:2301.00220 2023a.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint wwXiv:2336.046..3 2023b.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15593, 2023.
Mike Lewis. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:/910.1346// 2019.
Yhw Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Aiwei Liu, Leyi Pan, Xuming Hu, Shu' ang Li, Lijie Wen, Irwin King, and Philip S Yu. An unforgeable publicly verifiable watermark for large language models. arXiv preprint arXiv:2307.162.00 2023a.
Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. A semantic invariant robust watermark for large language models. arXiv preprint arXiv:2310.0633 2023b.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
Minjia Mao. Dongjun Wei, Zeyu Chen, Xiao Fang, and Michael Chau. watermark for low-entropy and unbiased generation in large language models. arXiv preprint arXi::2405.14604 2024.
Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al. Markllm: An open-source toolkit for Ilm watermarking. arXiv preprint 2rX5 10051, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. Mark my words: Analyzing and evaluating language model watermarks. arXiv preprint arXiv:2312.00275. 2023.
Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. Markmywords: Analyzing and evaluating language model watermarks. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 68-91. IEEE, 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (1(11 2020.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer- generetererrrrrrr networks. In Proceedings of the 55th Annual Meeting of the Association for Computatiional Linguistics (Volume Long Papers). pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 11118653/1//17-1100 // 7/www. , www.waacwwbb... wtt.:alwww....../aac.oaooeeaaaaa wwt.o//gg/n.oooggo/ooooooe..../.
URL https
Shangqing Tu, Yuliang Sun, Yushi Bai, Jifan Yu, Lei Hou, and Juanzi Li. Waterbench: Towards holistic evaluation of watermarks for large language models. arXiv preprint arXiv:2311.07138. 2023.
Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. Dipmark: A stealthy, efficient and resilient watermark for large language models. arXiv preprint arXiv:2310.077100 2023.
11

---

Published as a conference paper at ICLR 2026
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger. and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint 1rXiv:190.00965, 2019.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:3306. 17439, 2023.
12

---

Published as a conference paper at ICLR 2026
A
LLM USAGE
We ONLY used ChatGPT-40 and ChatGPT-5 to refine the content.
B MISSING PROOFS
B. PROOF OF THEOREM 4.1
Setup. Fix prompt æ. Let Par(- æ) denote the LM's distribution over full generations (sequences token paths). A watermarking scheme consists of reweight stralegy and watermark key E K, producing a watermarked distribution
( æ. k) = F(PM( 2)) k))-
We say the scheme unbiased duistibutiionppeserviii in expectation over keys)
Ex-ulPwr æx k)] = Par(: x),
(4)
where a the key distribution. Detectability means there exists statistical that, for some keys can distinguish samples from (. æ. k) versus (-r(- x)) with nontrivial power.
Repeated-prompt model. key k:
Consider m independent generations of the same prompt æ under a fixed
((m) (. k)) == (Pw(. x, k)) 8m w
((m) (. ax) = (Par a)) @m
Lemma B.1 (Detectability = key-level deviation). If watermarking scheme is detectable, then there exists a measurable set C K with p(A) > such that Pw(: x, k) + PM x) for all k E A.
x)) for p-almost every k then for any sample size the product measures also coincide, p(())( (. x, k) = p(m) (- x). rendering any detector powerless (no test can outperform random guessing). Thus detectability implies positive-measure subset of keys for which the two distributions differ.
Proof. If Pw (-
Lemma B.2 (Separation amplifies under products). Let P & Q be two distributions on common measurable space. Denote their Bhattacharyya coefficient by BC(P, Q) /ap aQ (0, Then for product measures,
BC(P (ipp$s"Q$mm (BC(P.Q)) m
0,
m-+00
and consequently the total variation distance satisfies
TV(PSM) Q8m) N 1 - (BC(P,Q))
m2+30
Proof. Bhattacharyya coefficients multiply under independent products. Using the inequality (V Q) < BC( (P, Q) yields the stated lower bound on TV; since BC(P, Q) < when P o Q the bound tends to 1 as +
By Lemma B.1 detectability implies the existence of positive-measur set A of keys with Pw( xc k) # PM
x). Fix any such & E and apply Lemma B.2 with P = Pw(: x k) and Then
Q ( x
TV p(mm) (- x k), ((m))(
2:)
m-+oc 1,
so the product distributions diverge and become perfectly distinguishable as m grows. Therefore, under repeated queries with fixed key, the watermarked joint law cannot equal the LM's joint law; ie., the scheme cannot preserve the original distribution under repeated prompts. This contradicts simultaneous satisfaction of (1)-() with detectability.
13

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF THEOREM 4.2
Proof. Let/ f map all 3nm sampled generations to Det.Wik~ - A(Ps: Pr) - Par).. Chang- ing a single generation for prompt alters the corresponding per-prompt mean by at most 2A/m, and since x. + lar al is I-Lipschitz, the induced change on A term is at most (2A)/(mm).
Bounded differences:
One PM sample affects both A(PM, Pr) and A(PM: Par.) by at most (2A)/(mn) each, hence Aʃl S (mn)
One Pr sample affects only A(PM: Pr.): ; Afl S (2A)/(mn).
One Par sample affects only A(PM: Pm)): ): A S (2m)/(mm).
Summing squared Lipschitz constants over all variables gives
4A mi
2A
2A 17
2
24A²
Σ C mu
+ nm
+ mm
mm
mi
By McDiarmid's inequality,
2t
mn t2
Pr ( exp
ΣE k
=exp
12,12
and the two-sided version follows by symmetry.
C DETAILED EXPERIMENT SETUP
C.I EXPERIMENT SETUP
Models Datasets. We evaluate on LLAMA-3.2-23
(Dubey et al. 2024).
MISTRAL-
7B- 1--STTRRUCT-V.. (Jiang et al. 2023). and PHI-3.5-MINI-INN MINIIINSSTUUCT (Abdin al.
2024] for
open-ended text generation following prior work (Kirchenbauer et al., 2023a Hu et al.)
2023). Our
primary corpus standard subset of C4 (Raffel et 2020); additionally report three MMW datasets (Piet et al., 2023), DOLLY CW (Conover et al. 2023), and two WATERBENCH tasks (Tu al. 2023). For one-shot unbiasedness validation,
follow Hu et al. (2023); Wu al. (2023)
using MBART (Liu et al. 2020) on WMT16 RO-EN (Bojar et al. (016) (machine translation) and BART (Lewis 2019) on CNN/DAILYMALL (See al. (017) (summarization..
Watermarking setup. Unless noted, watermark keys combine secret key with prefix 2-gram context key. Hyperparameters follow the original papers: KGW 0.5, 1.0, 1.5, 2.0, Unigram 5 E 0.5, 1.0, 1.5, 2.0, DiPmark length 1 10, 20, 50, 100; --reeehhht as in Hu al. (003).. We report TPR FPR at theoretically guaranteed FPR levels (5%, 1%, 0.1% 0.1%) and Median p-value. Unless specified, decoding settings and prompt sets are identical across methods.
{0.3, 0.4); SynthID tournament layers
20; MCmark list
Evaluation Metrics for Text Quality. We employ the following metrics assess the quality of generated text:
ROUGE. For summarization tasks, we use the ROUGE metric (Lin 2004), which measures n- g-gaam overlap between generated summaries and reference texts, thereby capturing how well the output conveys the essential content.
BLEU, For machine translation, we adopt the BLEU score (Papineni et al. 2002 which evaluates lexical similarity between system-generated translations and human references.
BERTScore. BERTScore (Zhang et al. 2019) computes sentence similarity by aggregating cosine similarities between contextualized token embeddings. We report BERTScore-Fil -Precision, and -Recall for both summarization and translation tasks.
14

---

Published as a conference paper at ICLR 2026
Perplexity. Perplexity, standard measure from information theory, quantifies how well probabilistic model predicts observed text. Lower values indicate more accurate predictive performance. We perplexity to evaluate both summarization and open-ended text generation.
D ADDITIONAL RESULTS
Table 5: Robustness (TPR @0. (%FPR) watermarking method across different attack types.
Method
DIPPER
Random 30% %
Random 20 %
Random 10%
KGW(6=0.5)
0,00%
2.23%
2.45%
4.26%
KGW(6=1.0)
0.00%
4.69%
9.69%
18.75%
KGW(d=1.5)
0.31%
17.29%
35.00%
50.94%
KGW(6=2.0)
0.00%
40.21 %
62.21%
80.53%
Unigram(0=0.55)
17.71%
25.21% %
36.04%
43.65%
Unigram (8=10))
21.35%
41.88%
55.52%
69.06%
Unigram 8=1.5)
23.23%
61.58%
75.26%
85.16%
Unigram 6=2.00)
36.46%
86.88%
93.23%
95.83%
DIP(3==03)
0.10%
0.96%
1.70%
6.28%
DIP(a=0.4)
0.10%
0.74%
3.79%
8.21%
y-reweight
0.10%
1.58%
5.58%
13.58%
STAA(>=0.5)
0.42%
0.73%
2.71%
8,85%
SynthlD
0.52%
2.60%
4.79%
9.90%
ITStEdit
0.00%
0.52%
1.67%
3.75%
EXP-Edit
0.31%
8.96%
15.10%
19.17%
MCmark(n=10)
0.52%
13.89%
46.32%
84.42%
MCmark(n=20)
1.15%
18.44%
43.02%
70.94%
MCmark(n=500)
0.94%
20.52%
44.79%
71.25%
MCmark(n=100)
0.73%
17.29%
34.06%
56.77%
Table 6: Robustness (TPR @PRR55%FRR) of watermarking method across different attack types.
Method
DIPPER
Random 30 %
Random 20 %
Random 10%
KGW(6=0.5)
1.67%
14.89%
19.79%
21.60%
KGW(5=1.0)
2.08%
30.73%
47.81 31%
61.67%
KGW(5=1.5)
4.38%
58.13%
77.71%
89.17%
KGW(5=2.0)
5.63%
80.74%
91.16%
96.00%
Unigram(5=0.5)
53.75%
67.81%
78.85%
83.96%
Unigram(6=1.0)
60.83%
84.17%
92.19%
91.46%
Unigram(6=1.55)
63.33%
92.84%
94.32%
97.37%
Unigram(6=2.0)
74.79%
98.23%
99.48%
100.00%
DIP(o=0.3)
2.29%
7.02%
16.91%
36.91%
DIP(a=0.4)
1.77%
7.79%
18.21%
39.16%
7-reweight
1.88%
11.37%
23.68%
46.84%
STA(<y0055)
6.88%
15.52%
28.23%
45.63%
SynthID
9.90%
23.13%
32.60%
49.90%
ITS-Edit
5.00%
8.23%
10.73%
11.88%
EXP-Edit
5.10%
25.73%
29.69%
39.38%
MCmark(n=10)
15.21%
61.26%
90.21%
99.05%
MCmark(n=20)
13.85%
58.65%
79.90%
94.38%
MCmark(n=50)
10.83%
56.46%
78.23%
91.77%
MCmark(n=100)
10.63%
50.63%
68.85%
83.85%
15

---

Published as a conference paper at ICLR 2026
Table 7: Unbiasedness, detectability, and robustness scores of watermarking methods, sorted by Detectability score.
Method
Unbiasedness
Detectability
Robustness
SynthID
0.965
0.974
0.105
MCmark(n=10))
0.985
0.971
0.423
MCmark(n==0))
0.988
0.959
0.390
(C=5rk()=50))
0.989
0,945
0.398
KGW()=2)
0.884
0.925
0.533
MCmark(n=100))
0.985
0,906
0.330
Unigram(6=2)
0.872
0.903
0.855
Unigram(6=1155)
0.927
0.838
0.711
KGW(8=1.55)
0.935
0.808
0.350
KGW(8=1)
0.972
0.724
0.155
Unigram(d=1)
0.972
0.723
0.590
(a=00))
0.980
0.719
0.078
DiPmark(a=0044
0.985
0.702
0.058
EXP
0.936
0.676
0.147
STA(?=0.5)
0.988
0.676
0.079
DiPmark(a=0)
0.991
0.660
0.051
Unigram(5=0.5)
0.991
0.502
0,436
ITS
0.907
0.494
0.032
KGW(5=0.5)
0.988
0.461
0.054
16


Reviews:
[{"review_id": "qaZkpv9JZr", "reviewer": "Reviewer_CHL5", "summary": "This paper introduces UWBENCH, an open-source benchmark for unbiased language‑model watermarks. The authors contribute (i) a single‑prompt multi‑generation (SPMG) metric and a calibrated statistic DetWmk to quantify drift that emerges when the same prompt is queried many times, (ii) an impossibility theorem showing that any detectable unbiased watermark cannot preserve the LM distribution under repeated queries with a fixed key, and (iii) a token‑level robustness analysis with a simple certificate based on effect regions. Empirically, the benchmark evaluates methods along unbiasedness, detectability, robustness across several LMs and tasks, and argues that there is a trade-off between (i) optimizing detectability and unbiasedness, (ii) robustness.", "questions": "1. If the watermark key is re‑sampled per generation (or per prompt), does SPMG drift vanish in expectation?\n2. In practice, is the same key used multiple times for the same prompt? \n3. SPMG requires ∣Met(g)∣≤A; yet the text cites perplexity as an example, which is unbounded. Clarify how you bound or clip PPL (or switch to bounded NLL)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Theorems 4.1 is clean and establishes a fundamental limitation of detectable watermarks.\n2. The evaluation covers multiple models/datasets and reports matched‑FPR operating points.\n3. The three‑axis protocol is well‑specified for unbiased watermarking.", "weaknesses": "1. The proposed unbiasedness metric uses performance metrics as surrogate and might not guarantee Eq.(1). Thus it is misleading to call it unbiasedness. The benchmark lacks reasoning tasks where the accuracy is an important performance indicator.\n2. The robustness framework is neat but may not capture common edits such as typo, translation attack, synonym substitution. Also Random token replacement lacks practicality and distort the output distribution.\n3. Novelty is limited compared with existing results: similar arguments to Theorems 4.1 is already established in Christ and Gunn (2024) and the three-axis evaluation is common in MarkMyWords.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6olCYNxYPN", "reviewer": "Reviewer_QDvf", "summary": "This paper introduces UWBENCH, a benchmark and theoretical framework for evaluating unbiased watermarking methods in large language models.\nThe authors propose a single-prompt multi-generation (SPMG) metric to quantify distributional drift under repeated queries and define a variance-controlled detection statistic using McDiarmid’s inequality to derive finite-sample guarantees.\nOn the empirical side, they benchmark several watermarking schemes across three axes — unbiasedness, detectability, and robustness — and argue that token-modification attacks offer more stable robustness evaluation than paraphrasing.\nOverall, the work aims to provide a reproducible, theoretically grounded evaluation suite for future watermark research.", "questions": "- **Notation check:** Equation (2) likely intends  \n  $$t_\\alpha = A\\sqrt{\\tfrac{12\\ln(1/\\alpha)}{mn}},$$  \n  not $A^2\\sqrt{\\cdot}$.  \n- **Terminology:** Define “SPMG” and “Met” explicitly on first mention; readers outside the immediate subfield may not recognize these acronyms.  \n- **Writing:** The proofs in Appendix B are clear and worth highlighting — consider moving a short intuitive summary of Theorem 4.2’s logic into the main text.  \n- **Question:** Have the authors considered testing whether UWBENCH metrics correlate with alignment or reward-model score improvement as in Verma et al.? Even a small pilot experiment could reveal whether detectability improvements coincide with improved alignment or if they diverge.  \n\n\nThis is a well-executed and timely paper that provides both theoretical insights and practical tools for unbiased watermark evaluation. My main feedback is to clarify the assumptions behind the theoretical bound and to connect the benchmark more explicitly to alignment and downstream-utility studies. Adding or even briefly discussing a reward-model-based alignment ablation would substantially strengthen the paper’s impact.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Strengths\n\n1. Clear motivation and structure. The paper identifies a real gap — the lack of principled evaluation for unbiased watermarking — and builds both theoretical and empirical tools to address it.\n2. Balanced theoretical and empirical contributions. The impossibility result (Theorem 4.1) and the finite-sample McDiarmid bound (Theorem 4.2) are neat results that add formal grounding to an otherwise empirical area.\n3. Reproducible benchmark. Making UWBENCH open-source with standardized metrics is a strong community contribution.\n4. Thoughtful experimental framing. The move away from paraphrasing attacks toward token-modification attacks as robustness tests is well-justified and supported by variance analysis.\n5. Readable figures and tables. The scatterplot (Fig. 1) effectively summarizes the trade-offs across unbiasedness, detectability, and robustness.", "weaknesses": "### Weakness\n\n- While I found the theoretical framing around McDiarmid’s inequality and the single-prompt multi-generation (SPMG) detection metric interesting, the use of the concentration bound feels a bit “black-boxed.” The paper could be clearer about the assumptions that allow McDiarmid to hold — namely independence across samples and bounded differences — rather than invoking it as a matter of fact. Clarifying these assumptions would help readers trust the finite-sample guarantees and better understand what breaks when they are violated.\n  \n- Beyond that, I think the work would benefit from being placed more explicitly in the growing literature studying watermark–utility trade-offs. For example, Downstream Trade-offs of a Family of Text Watermarks (Ajith et al., EMNLP Findings 2024) shows that even unbiased watermarks can hurt downstream task accuracy by 10–20 %, and WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models (Molenda et al., NAACL Findings 2024) quantifies how stronger watermarks improve detectability but reduce generation quality. Most relevant, Watermarking Degrades Alignment in Language Models (Verma et al., ICLR 2025) demonstrates that watermarking shifts alignment behaviors such as truthfulness, safety, and helpfulness, and mitigates this via an external reward model. Verma et al. (2025) analyze how Gumbel watermarking affects model alignment. They note that Gumbel watermarking produces the same output deterministically for a fixed seed, meaning that in your repeated-query setup the SPMG drift would be trivially zero by design rather than informative about unbiasedness. It might help to briefly acknowledge this point when interpreting your results on Gumbel watermarking. Verma et al. also show that watermarking tends to decrease reward-model scores, which could, in principle, serve as another lens for detection. If your benchmark already defines a bounded per-generation metric Met(⋅), it might be interesting to mention that a reward-based version of this metric could empirically capture the same drift your McDiarmid analysis predicts. Even a short comment or future-work note on whether these two detection views—statistical vs. reward-based—might align would make the discussion more rounded without expanding scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GtJ3K3tCV3", "reviewer": "Reviewer_xY8q", "summary": "The paper proposed an evaluation framework for watermarking methods. The paper focuses on unbiasedness, detectability, and robustness. For unbiasedness, the paper proposed the Single-prompt multi-generation (SPMG) unbiasedness metric, which measures the difference of some metrics of generations from two models averaged across prompts and multiple generations of the same prompts. For robustness, the paper discussed the token effect region and the expected score decrease under one edit. In experiments, the weighted average of the true positive rate under different false positive rates and different perturbations is used. The proposed metrics are evaluated on several unbiased and a few biased watermarking methods.", "questions": "See weakness above", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Evaluation of watermarking methods is mostly ad-hoc, developing standard benchmarks is a timely topic. The discussion on the unbiasedness and robustness of watermarking methods makes sense.", "weaknesses": "1. The paper claims to introduce an open-source benchmark, but no source code is provided.\n\n2. The contributions are not clear. In Section 5.1, there are a lot hyperparameters in the scores that seem to come from nowhere, e.g., the weights in the weighted average for the robustness score and detectability score. Why are these choices or these scores good? I would expect much more experiments to demonstrate that these scores are robust to the choice of hyperparameters, or give consistent evaluations for watermarks across different data sets, or number of prompts and length of generations.\n\n3. The organization of the paper is confusing in some places, e.g., in section 4.2 token effect region and expected score decrease under one edit are discussed, but in the experiments, it seems that these concepts are not used, the score used for robustness are TPR under different FPR and perturbations. And the paragraphs \"Green-count detectors\" and \"SynthID-style bit tests\" seem to be examples for \"expected score decrease under one edit\". But they are all listed as if they were parallel concepts.\n\nSome minor issues:\n\nThere are some typos, e.g., in line 465, I think it is Figure (1). In line 229, the form of $t_{\\alpha}$ is inconsistent with the inequality above\n\nSome citations are missing, e.g., for the DIPPER method in line 416, there are no citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dzoPT8Drxm", "reviewer": "Reviewer_aAFa", "summary": "This is a benchmark paper that evaluates different unbiased watermarking methods in large language models. Primally, the authors evaluate the detectability, unbiasedness, and robustness of different unbiased watermarking methods.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces UWBench, a systematic evaluation of unbiased watermarking methods over the three-axis protocol: unbiasedness, detectability, and robustness.\n2. The authors provide a better procedure to evaluate unbiasedness and robustness for unbiased watermarking methods, which I agree with.", "weaknesses": "1. The authors may miss some unbiased watermark baselines in the evaluation, e.g., [1], although this work is mentioned in the introduction section.  \n2. I believe the main issue with this paper is the lack of motivation to provide a benchmark work for unbiased watermarking methods. The authors should better justify why such a benchmark is necessary and what specific gaps it aims to fill in the existing literature. Without a clear motivation, the contribution appears limited.\n3. A conceptual evaluation of different baselines could also be provided. For example, what are the drawbacks of each method? The authors could also offer guidelines for designing future unbiased watermarks. \n4. The authors claim that UWBench is an open-source benchmark. I would appreciate it if the authors could provide the anonymized codes in their supplementary material.  \n\n[1] Christ, M., Gunn, S., & Zamir, O. (2024, June). Undetectable watermarks for language models. In The Thirty Seventh Annual Conference on Learning Theory (pp. 1125-1139). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)