On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection
Authors: Weiqing He
Xiang Li
Tianqi Shang
Li Shen
Weijie J Su
Qi Long
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Language Models
Watermark
Goodness-of-Fit
Primary area: evaluation
TL;DR: Introduce and apply Goodness-of-Fit (GoF) tests to watermark detection, and explore their effectiveness across diverse conditions.

Abstract: Large language models (LLMs) raise concerns about content authenticity and integrity because they can generate human-like text at scale. Text watermarks, which embed detectable statistical signals into generated text, offer a provable way to verify content origin. Many detection methods rely on pivotal statistics that are i.i.d. under human-written text, making goodness-of-fit (GoF) tests a natural tool for watermark detection. However, GoF tests remain largely underexplored in this setting.
In this paper, we systematically evaluate eight GoF tests across three popular watermarking schemes, using three open-source LLMs, two datasets, various generation temperatures, and multiple post-editing methods. 
We find that general GoF tests can improve both the detection power and robustness of watermark detectors. Notably, we observe that text repetition, common in low-temperature settings, gives GoF tests a unique advantage not exploited by existing methods.
Our results highlight that classic GoF tests are a simple yet powerful and underused tool for watermark detection in LLMs.

On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection
Weiqing He
Xiang Li University of Pennsylvania 1x10077@upenn edu
University of Pennsylvania weiqingh@sas. upenn. edu
Tianqi Shang
Li Shen
University of Pennsylvania
University of Pennsylvania
tianqi. shangOpennmedi cine. upenn. edu
li. wwwntpennmediccccccmmmoooo cine. upenn. edu
Weijie Sul University of Pennsylvania suw@wharton upenn. edu
Qi Long'
University of Pennsylvania glongdupenn. edu
Abstract
Large language models (LLMs) raise concerns about content authenticity and integrity because they can generate human-like text at scale. Text watermarks, which embed detectable statistical signals into generated text, offer provable way to verify content origin. Many detection methods rely on pivotal statistics that are i.i.d. under Houman-writted text, making goodness-of-fi (GoF) tests a natural tool for watermark detection. However, GoF tests remain largely underexplored in this setting. In this paper, we systematically evaluate eight GoF tests across three popular watermarking schemes, using three open-source LLMs, two datasets, various generation temperatures, and multiple post post-editing methods. We find that general GoF tests improve both the detection power and robustness of watermark detectors. Notably, we observe that text repetition, common in low- temperature settings, gives GoF tests unique advantage exploited by existing methods. Our results highlight that classic GoF tests are a simple yet powerful and underused tool for watermark detection in L.LMs.3
Introduction
The rapid advancement of large language models (LLMs) has enabled machines to generate fluent and coherent text that closely resembles human writing. While this progress has unlocked many applications, it has also raised significant concerns about the authenticity, ownership, and integrity of generated content. For example, I.LMs can be misused to spread misinformation [58, 55,51] fabricate academic work [52, 38), or manipulate digital communication scale [45, 49, 8). These risks have prompted growing interest in developing reliable tools to trace and verify machine-generated text.
A promising solution is text watermarking, which embeds hidden statistical signals into generated text. Specically, it introduces dependencies between tokens we and secret pseudorandom numbers [5, 1]. In watermarked generation, each token W/ P. is produced by decoding function S as We S(P, G),, where P. is the next-token predictive (NTP) distribution that we follows. This
'qqual contribution.
Corresponning authors. Codee available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

induces a watermark signal- staaiisiiaa dependence between w, and G. In contrast, human-writter text lacks this dependence, as humans do not have access to Çr. Detection methods leverage this difference using scalar pivotal statistics [31], defined as Y, = (wr, (t), within a hypothesis testing framework:
Ho: Win is human-written vs. H1 Win is L. L.M--gnnrrted..
(1)
A key property of the pivotal function Y is that (w, () follows a known distribution Ho whenever w independent of (. which exactly holds for human- -writenn w since human users do not know the secret Ç. Thus, problem (1) can be reformulated in terms of Y;:
Ho : Y ~ po i.i.d. vs. H1 Y ~ other distribution that depends on P..
The formulation in (2) is standard and popular in the watermarking literature [31, 29, 9, 28, 1, 16, 5, 14, 60, 61, 27, 21, 56]. Many detection rules, though somewhat ad hoc, are designed to address this problem. In the idealized setting where all NTP distributions {P: y- 1 are known, the log-likelihood ratio test based on the pivotal statistics [Yr) is provably optimal [22]. In practice, however, these distributions are often unavailable due to limited access to commercial LLMs and unknown prompts. To overcome this, recent works have developed NTP-agnostic approaches. For example, Li et al. [31] assumes all P. lie within a known class and derives the least favorable detection rule using minimax formulation. This framework extended by Li al. [33] to handle human editing, where the observed text may contain mix of human-written and LLM- -egeereettedd content. In this case, the alternative hypothesis in (2) is modified to reflect the fraction of buman-edited tokens. Nonetheless, both works share the same null hypothesis Ho: ki) are i.i.d. samples from the known /40-
This structure reveals a close connection between watermark detection and the classical statistical problem of goodness-offff fit (GoF) testing. GoF tests aim to determine whether i.i.d. samples follow a given distribution. While watermark detection involves non-i.idd data under Hu-due to the autoregressive nature of LLMs, where each P. depends on the history W1:(t-1) GoF tests remain applicable because Hn is the same: {Yr]==1 are i.i.d. from a known μo- Despite their long-standing role in statistics [7]. GoF methods have received limited attention the watermarking literature. Prior work has largely focused on designing new watermarking schemes, with less emphasis on improving detection power (see Appendix for a detailed discussion). Li et al. [33] proposed a GoF test and analyzed its theoretical performance on the Gumbel -mxx watermark [1], but its effectiveness on other watermarking schemes remains unclear, as does the performance of other classical GoF tests. This raises a natural question: how well do classical GoF tests work for modern watermark detection? A systematic study of classical GoF tests could provide valuable insights for developing robust and effective watermark detection methods.
truncated divergence-basse
Our contribution. In this work, we study this connection empirically, and our contributions are:
Systematic empirical evaluation. We introduce general GoF tests to watermark detection and systematically evaluate eight GoF detection rules across three popular watermarking schemes. Our experiments span diverse configurations, including three open-source language models, two datasets, four temperature settings, and three types of edits.
Understanding GoF advantages. GoF tests consistently outperform baseline methods across varying temperatures and text lengths, and this robustness stems from their ability to capture distribution-leve deviations in the pivotal statistics. At high temperatures, watermark signals are stronger due to increased entropy in the next-token distributions. This leads to more noticeable shifts between the empirical CDF and the null CDF, which GoF tests are particularly effective at detecting-eve weaken, making detection more challenging. However, lower temperatures also induce repetition in the generated text, introducing structured patterns that GoF tests can exploit. This repetition results deviations from the expected null CDF, enabling GoF tests maintain strong detection power even in low-entropy settings. These complementary advantages explain why GoF tests are effective across a wide range of generation scenarios, including those that are traditionally hard for watermark detection.
with moderate text lengths. At low temperatures, watermark signals tend to
Robustness to modification. We evaluate GoF tests under both common text edits (deletion and substitution) and information-rich edits. In all cases, GoF-hased methods maintain high detection power, demonstrating strong robustness to modifications.
In short, our results show that GoF tests are a valuable complement to existing watermark detection methods, improving both detection power and robustness across wide range of conditions. Due to space constraints, we defer discussion of related work to Appendix B.
2

---

2 Preliminaries
Watermark embedding. LLMs like the GPT series [44, 4] generate text autoregressively. producing one token at a time based on previous tokens. Each token is drawn from a fixed vocabulary W using a multinomial distribution P (Pr,w))ew, called the next-token prediction (NTP) distribution. This distribution depends on the previous tokens, user prompts, and system- prompts, summarized as P. = M ((W1((111) [53, 44, 4]. To embed watermark, the model uses a pseudorandom variable Çt. computed by deterministic hash function: 4 A(t))e-m)((t- Key)) where A is a hash function, m is a context window size, and Key is a secret key. Once & is computed, the next token is generated wt S(P,, (t), where S is (possibly stochastic) decoder. The decoder S is said to be unbiased if for any P and w, it satisfies P& (S(. = w) = Pw- This means that when Ç is truly random, ((P.C) still follows the original NTP distribution P. In other words, unbiased decoder samples from the intended NTP.
Watermark detection. As introduced in Section 1, watermark detection aims to identify statistical dependence between each token Wy and the pseudorandom number Ç.. This is done using pivotal statistics, leading to the hypothesis testing problem in (2) [31]. Under Ho. there is no watermark, and wy and &a are assumed to be independent. By design, the pivotal statistic Y, Y(w, Sx) then follows a known distribution f4g, regardless of how Wy is distributed. Under the alternative, the watermarking process introduces dependence through the decoder S, causing Y, to deviate from 14n- In particular, Y, P. follows a distribution Pi that depends only on P.. This leads to the following hypothesis testing problem:
Ho Yg ~ Po i.d. VS. H1 : Y2 P. ~ P1.P.
Li et al. [33, 32] extend it to more practical setting where the generated text may be partially edited. They model human edits as a mixture effect so that Y, is from mixture of μi and M1 under Hy.
Three considered unbiased watermarks. We consider three unbiased watermarking schemes. For brevity, we describe only their null distributions (aa, which suffice for understanding our methods and results. Full details are provided in Appendix A. The first is the Gumbel-max watermark [1], where No is the uniform distribution (00 1). The second is the inverse transform watermark [29], whose null CDF satisfies PY S r) r² for r [0. 1]. The third is Google's 's SynthID watermark [9], where the pivotal statistic under Ho follows the distribution of (IwinHall((k) where IrwinHall the Irwin Hall distribution, i.e., the sum of le i.i.d. U(, 1) variables. A common feature of all three is that their null distributions doo are known and have computable CDFs.
An exception. The green-red list watermark [27, 28, 60] (see Appendix A.4) is popular method. randomly partitions the vocabulary into a green (favored) and red (disfavored) set, boosts the probabilities of green tokens, and samples from the perturbed NTP distribution. Detection is based on counting green tokens and rejecting Hy if their frequency exceeds threshold. Variants [56, 21] use more carefully designed partitions but follow the same overall pipeline. We exclude these methods from our experiments because GoF tests reduce to the original detection rule in this setting. The pivotal statistic is binary (whether a token is green), and the order of them does not matter-ond the total count green tokens is informative. As a result, both methods are effectively equivalent, making additional experiments unnecessary.
3
GoF Tests for Watermark Detection
Rationale for using GoF tests. Goodness-of-fit (GoF) tests are classical tools in statisties for determining whether a sequence of i.i.d. samples comes from a specified distribution [7]. The standard formulation is:
Ho Yo po i..d. Vt vs. Hptandard Y. ~ #1 ii.d. Vt.
where M1 * Mg. GoF tests evaluate deviations from the null distribution Mo and determine whether they are statistically significant. However, this classical setup does not directly apply to watermark detection, where the alternative hypothesis H₃ in (3) involves structured dependence introduced by the watermarking process. Specifically, the pivotal statistics (Y.) are not i.i.d. under H, due to the autoregressive nature of L.MSs and the time-varying NTP distributions P..
However, the core principle of GoF cests-measuring deviation from the null distribution remanes highly relevant. The most related work. Li et al. [33], proposed divergence-base GoF test [24]
3

---

Table 1: Considered GoF tests.
Tests name
Deviation measure
Tests mame
Deviation meusure
Ti-GoF
Phi, [39]
(4)
sup (FP (o)))))) rolp".))
Kuiper's Kut, [30]
V. mase
(# P(s)
15.5
Kolmagorov Smimov test Kol. [50)
Da
1315i
|unx
Pio) <
Rus)
Anderson Darting test, And. [2]
- 1o2 > (2) 1)
Pinsr-)lPa
Cramdr-von Mises lest. Cra.
W
Di
(Ps)
Watson's 's lest, Vas, [54]
1/2 - W2 - 4)
Neyman's test. Ney, [40]
Th Σjn1
Chi-squared test, Chi, [42]
10-EP
for detecting Gumbel-max watermarks and demonstrated its asymptotic robustness. Yet, they did establish its uniqueness or compare against other GoF tests. Moreover, their analysis relies on asymptotic assumptions such as infinite text length, while in practice, the iiite- performance among asymptotically optimal methods. Our work
of GoF tests can vary substantially complements this line of research by empirically evaluating classical GoF tests in the context of watermark detection. We also investigate which factors most affect their performance.
Introduction of GoF tests. All GoF tests evaluate deviations from the null distribution Ho and reject Hu the deviation exceeds a specified threshold. Thus, it suffices to define the measures of deviation used by each test. Before doing so, introduce some notation. Let Y1. Yi..... Y, denote the pivotal statistics, and let Y()) S (22) I ..... (m) be their order statistics. The empirical cumulative distribution function (CDF) is defined as (r) null is (r(r) = No (Y < r). Recall that the null distribution μi known but varies by watermarking scheme (see Section 2). Given this, we can also compute p-values as P. =1 (Y.).
Ei= ly,gr, and the theoretical CDF under the
Eight GoF tests summarized in Table 1, with each referred to by three-letter abbreviation (e.g.. the Kolmogorov-Sminge test as Kol). All GoF tests operate on pivotal statistics and are permutation-invariant. Therefore, the order of pivotal statistics does not affect their detection results. Below, we provide additional details omitted from the table for brevity:
Tr-GoF test [33] uses a truncated d-divergence [24], which is the reason for its name Phi. The truncation is introduced via ot satisfying p* sup(P(e) P(t) ct] to ensure numerical stability.
Here, K (u, v) = (u, (0 < v < < 1} and K,(u, v) = vØ. (*) + (1 - v) (I=#) is
the untruncated Φ divereegeee The function (x)) convex and defined as:
log - x 1, if = 1, 1-048==8'
6x(x) =
if # 0,
s(1-8)
1og++x-11 if 0.
our experiment, we fix 8 = 2 for all considered watermarks due to its good performance [33]. Kolmogorov-Smimov test (Kol) measures the largest difference between the empirical and null CDFs. In fact, one can show that D, = sup, Fr - Fo(x)].
Watson's 's test adjusts the Cramér-von Mises statistic 2 to ensure invariance to location changes, so U = 2 n (F- 1)² where F A Σ. ((Y()).
Neyman's smooth test (Ney) expands deviations from the null distribution using orthonormal polynomials. Specifically, it computes coefficients
02
hg (Y:)
where hy are orthonormal Legendre polynomials for U(001) 1) and X, are normalizing constants.
The test statistic then aggregates the first k coefficients as Tg E/= 0z with k = 3 used in our experiments.
Chi-squared test (Chi) divides the range of po into equal- bins, counts observed frequencies
O,, and computes expected frequencies E, = n n/k. k. The test statistic is x² 2i=1 E, critical values drawn from the chi-squared distribution with k: degrees of freedom [42].
(O,-E.)² with
Importantly, for most GoF tests, the exact null distribution does not admit closed-form expression; instead, only an asymptotic distribution emerges the token length n grows. Nevertheless,

---

Algorithm GoF test for watermark detection (example: Kol for the Gumbel-max watermark)
Require:
Token sequence Wi:ni watermark decoder S; significance level or CDF under the null Fo- ..,, Yn from the sequence Win- 2: Compute p-values: P. = 1 FY(), t ,....,
1: Compute pivotal statistics 11....;
3: Sort the p-values in ascending order: P(1) S P(n)
4: Compute the test statistic (Ko1)
Dn c gax max ((e) 1-1 P(.)
5: Estimate critical value Taa based on the information of the watermarking scheme.
6: if' D, > % then
7: Reject Ho
8: else
9: Do not reject Ho
end if
experiments show that relying on this large-sample approximation still yields reliable Type I error control in practice. We defer detailed discussion of their practical considerations to Appendix C. To conclude, we integrate the full detection procedure into Algorithm 1, providing a streamlined view of how the Kolmogorov-Smirnov test (Kol) can be applied the Gumbel-max watermark. The same procedure can be straightforwardly applied to other GoF tests, which readers may extend as needed.
4 Language Model Experiments
4.1 Experiment settings
Experimental setup In our evaluation, we consider three open-source SAMMs-OPT-L.33 OPT- 13B [59], and Llama 3.I-8B [11] [1]]-across across four temperature settings: T E {0.1, 0.3, 0.7, 1.0]. We evaluate watermark performance on two text generation tasks: (i) text completion and (ii) long-form question answering. For text completion, we use the C4 dataset [46]. Each document the dataset is truncated to the first 50 tokens, which serve as prompts for the LLM to complete. For long-form question answering, we use the ELI5 dataset [12], where the LLM generates detailed answers to given questions. In both tasks, we randomly sample 1,000 documents and conduct experiments using three L.LMs across four remperature settings, following a consistent pipeline. the relative performance of different GoF tests is similar between tasks, we present the results on text completion in this section and defer the detailed results for the ELI5 dataset to Appendix D.4.
Remark 4.1 (Why we don't ' include green-red list watermark). As discussed Section 2, we do not include the green-red list watermark [27] or its variants in our experiments. This is because their pivotal statistics are binary (indicating whether a token is green), and the order of these values doesn't It play role. As result, the original detection rule rule-counting the number of green tokens-in already effective, and GoF tests reduce to this same procedure. Therefore, additional experiments are unnecessary.
Common text edits. Human edits can weaken watermark signals [48, 19, 33). To evaluate detection robustness, apply two common types of edits: word deletion and synonym substitution. For each, we randomly select a fraction Tedi E (0.1, 0.2} of watermarked tokens and either delete them or replace them with synonyms from WordNet [39].
Information-rich edits. We consider a stronger editing setting in which the hash function and secret key Key are known to the user. In this case, the user can selectively modify a limited number of Logggenrrated tokens to reduce watermark signals while preserving the overall quality of the text. We refer to this targeted modification as an information-rich edit. With limited token budget, the optimal strategy is to change tokens carrying the strongest watermark signals. In all three considered schemes, tokens with higher pivotal statistics tend to have stronger signals. Thus, replacing these tokens lowers the overall signal strength significantly. To simulate this, we compute the pivotal statistics for all tokens in watermarked sequence. Then, we select fraction Tinto E {0.3, 0.5) of tokens with the highest statistics and overwrite their values with samples drawn from the null distribution no.
5

---

Table 2: Type I errors human data and Type II errors (averaged over three LLMs) the C4 dataset. All values are enlarged by 100 for readability. T denotes temperature and n the token length. Baseline refers to the best- best-performing method among the baseline detectors. Red shading highlights lower values; blue indicates higher values.
n
Baseline
Phi
Kui
Kol
And
Cra
Wat
Ney
Chi
200
18.5
21.0
26.3
19.5
15.5
21.2
36.8
19.7
18.5
0.3
Gumbel-man
400
15.1
5.7
4.7
4.7
4.9
8,4
10.7
8.0
2.9
200
0.6
0.3
0.5
0.6
0.5
0.7
0.9
0.7
400
0.7
0.2
0.2
0.3
0.2
0.4
0.4
0.2
0.2
Type
0.4
0.9
1.5
0.6
0.7
1.2
0.9
Inverse- Iran.
0.3
200
38.7
51.0
29.2
29.7
33.6
36.7
40.4
37.3
21.8
400
27.1
12.1
6.0
7.4
9.3
14.0
10.7
13.0
3.6
200
1.3
2.7
1.3
0.9
0.9
1.0
1.9
1.2
2.3
0.7
400
1.5
0.5
0.2
0.3
0.4
0.6
0.4
0.5
0.2
Type
0.4
1.4
1.2
1.0
1.0
1.4
1.5
200
58.8
61.6
49.8
53.0
53.5
57.2
61.3
57.4
36.4
0.3
SynthID
400
44.4
25.0
16.7
21.0
24.5
31.5
26.5
29.1
10.4
200
2.8
3.4
4.5
2.8
2.3
3.3
8.4
3.0
3.3
0.7
400
2.2
0.7
1.2
0.8
0.6
1.4
1.8
1.3
0.8
Type
0.9
1.2
0.9
1.1
1.0
1.4
1.0
1.2
Remark 4.2 (Differences from watermark stealing). Information-rich edits are related to watermark stealing [26] in that they simulate post-stealing detection, but they are conceptually distinct. The former assumes the watermark information (e.g., secret key) has already been compromised, and the user is allowed a limited editing budget. In contrast, watermark stealing typically focuses on extracting the watermark itself or defending against such attacks [47]. Our study, by contrast, focuses on the detection stage- evaluating how detection rules perform under an idealized attack scenario.
Five baselines. Previous work primarily relies on sum-based detection rules, which reject Hp if the sum Et= h(Y;) exceeds a critical threshold, where h is a predefined score function. We adopt five score functions from prior studies as baselines. For the Gumbel-max watermark, we use Aaronson's 's score [1] and the log score [29, 13]. For the Inverse transform watermark, we apply the negative-sum score [29]. For the SynthID watermark, we use the identity-sum score [9]. In addition, we apply the least-favorable test from Li et al. [31] all three schemes. For each watermark, we report the best best-performing baseline among these five applicable rules. Detailed definitions of the score functions are provided Table in the Appendix.
4.2 Results on Statistical Power
Building on the experimental setup introduced earlier, we now present our findings. In watermark detection, a Type eтror corresponds to falsely identifying non-waterr watermarked text as watermarked, while a Type II eтror arises when true watermark goes undetected. We evaluate the statistical power of the considered GoF tests by analyzing both error types. To control the Type error at a = 0.01, we adjust the critical values using either theoretical distributions or Monte Carlo simulations. We then examine whether the Type error is well-controlled and explore how the Type error decreases under different configurations.
Type I error control. We begin by evaluating Type error control. To this end, we randomly sample 1,000 human-written texts from the dataset and assess Type error at significance level of a 0.01, a standard choice in prior work [9, 33, 29, 31]. Table 2 reports the empirical Type I errors at a sequence length of 400 tokens. All detection methods maintain errors close to the target level of 0.01, indicating that Type I error can be effectively controlled for most GoF tests.
Type II error decay. We next evaluate Type II errors at the fixed significance level = 0.01. Table reports the average Type II errors across three LLMMs for various detection methods. We consider both low temperature (T 0.3) and high temperature (T 0.7) settings, using the C4

---

Gumbel-mox
SywahDD>
8- 100
T- 1.0
0.6
T 0L)
6.
6.0
0.5 1.5 0.0
10. 10 0.0 0.5 1.0
Y
Figure 1: Empirical CDFs of pivotal statistics (F)( under different temperatures T and text lengths n. Shaded regions show 95% confidence intervals (CT): light gray for n 100 and dark gray for n 400, both at T 1.0. At T 0.1, the dark blue curve shows the empirical CDF of raw scores, while the orange curve (labeled = 0.1*) shows the CDF after removing repeated values.
dataset. Additional results on the ELI5 dataset presented in Table 10 in the Appendix. Detailed breakdowns by LLM and task are provided in Appendix D.4 for completeness. We observe from Table 2 that GoF tests generally outperform the baselines across different temperatures and text lengths. This observation can be refined into two points:
1. Advantages are more pronounced at higher temperature settings. GoF tests consistently outperform baseline methods high temperatures, regardless of text length. For instance, under the Gumbel- max watermark at temperature T = 0.7, the baseline yields a Type II error of approximately 0.7% for both 200 and 400, whereas several GoF tests achieve lower error rates around 0.3 %. This pattern remains consistent across different watermarking schemes.
Advantages persist under low-temperature settings. Surprisingly, GoF tests maintain strong advantage even when the temperature is low. For example, under the Gumbel-max watermark with T 0.3 and = 400, the best baseline yields a 15.1% Type II error, while And achieves 4.9% and Chi achieves the best 2.9%. This pattern remains consistent for shorter lengths and different watermarking schemes.
Why do GoF tests perform well at high temperatures. This observation naturally leads to the question: Why do GoF tests perform so well across varying temperatures and text lengths? The key reason lies in how they utilize the full empirical distribution (CDF) of the pivotal statistics to detect deviations from the null Ng. As discussed in Section GoF tests compare the empirical CDF F, with the known null CDF Fo. often using richer nonlinear operations- such as sorting, maximization, or transformations- that allow them to capture subtle discrepancies more effectively. In contrast, baseline methods typically rely on sum-based statistics, which compress the data into single value. This approach often overlooks structural differences in the distribution and is less sensitive to nuanced deviations.
These properties make GoF tests particularly effective high temperatures, where watermark signals are strong. In this setting, the empirical CDF diverges noticeably from the null CDF, even for moderate sequence lengths. As shown in Figure 1, the difference already apparent at n 100. and increasing n further primarily reduces statistical variance. This implies that, under high temperatures, GoF performance becomes less dependent on text length explaannxiiinnng their consistent advantage across both short and long sequences.
missing factor at low temperatures. Low-temperature settings are often considered difficult for watermark detection due weaker signals. Yet, they common in applications such customer service, factual QA, and code generation. At lower temperatures, LLM outputs become more deterministic, reducing entropy in both the generated text and the corresponding NTP distributions. This lower variability makes detection harder, as the null and alternative distributions- and become nearly indistinguishable. Prior work [31] even shows that, in some cases, accurate detection may be information-theorrttiall impossible. However, our study identifies an overlooked factor: text repetition, which becomes more prominent low temperatures.
To quantify this, we measure the m--rram repetition rate-the proportion of repeated text segments ((t-m):(t-I) across generated outputs. We evaluate this metric under various temperature settings (T E 10.1, 0.3, 0.7, 1.03) using three L.MSs with watermarked outputs. For comparison, we also compute the same metric on Human-writted texts sampled from the dataset. Figure 2 shows the results for the Gumbel-max watermark; additional results for other watermark schemes appear in
7

---

Regenttoon Peroetage (50
100 T-0. T--33
T-07
1/
Tuman
Repastled Uaiqgo tolous
201
60
40-
10
0.25
0.50
0.75
Figure 2: Left:
Average repetition rate in Gumbel-max watermarked texts across three LLMs, compared to human-write written text. Right: Distribution of the highest probability in NTP distributions for OPT-1.36 using the Gumbel-max watermark at temperature 0.7.
Gumbel-max
Joverse-tronsoom
SynthDD
12-
Hiselin
Dype II Stro x IP
1.5
Phi
Kil
1.0-
And
Cia
Ney
Row tolens
Unigue teloree
Rew Todorts
Urique Iodont
Raw bekant
Unigua Inlore
Figure 3: Change in Type II error after removing repeated pivotal statistics (averaged over three LLMs). Raw tokens denotes the error rate on the original 200-token sequence; Unique tokens denotes the error rate after excluding repeated pivotal statisties from the same sequence.
Appendix D.4 (Figure 8,9). Our findings confirm that repetition is common in LLM- LLM-generated text, consistent with prior studies]20, 15, 57], and that lower temperatures tend to amplify this effect.
The influence of repetition on top probabilities. By the hash rule S. -A(wit (( 1)+ Key). repeated m- grams (i.e., (itt-m)((t-1) W(r-m))((--1) for some t' * t) lead to identical pseudo- random values, Çe = Çv. According to the decoder rule w/ = S(P,, Ce). this does not guarantee wy = we unless P. Pp.. This suggests that NTP distributions vary little across repeated segments, leading to repeated token outputs. To examine this, we analyze the top probability (i.e., max Pt,.) of generated tokens from the OPT-1.3B model with the Gumbel-max watermark at temperature 0.7. As shown in Figure 2 (see Figures 8, 9 for other watermarks), repeated tokens show a sharp spike I. indicating -etermmiiiiiiiicc generation. In contrast, unique tokens exhibit more spread-out distribution, reflecting greater diversity. These results suggest that repetition reinforces low-entropy behavior.
The inffuence of repetition the empirical CDE. Repetition also affects the empirical CDF of close
the pivotal statistics Y,. In ideal fow-temperature settings, MI is to μo due low entropy, so the empirical CDF of Y, should resemble μoo However, repetition disrupts this, causing Y; to deviate further from To illustrate, we analyze a sentence generated by 1.T- and plot the empirical CDFs at temperatures 0. and 1.0 (Figure At both temperatures, the empirical CDF deviates from No, but in opposite directions. Intuitively, repetition introduces artificial senectrre-vvidde in the step-like shape of the CDF low temperature that pushes the distribution away from the null. Removing repeated values smooths the CDF and brings it closer to 140- This explains why removing repeated pivotal statistics slightly degrades the performance of GoF tests (see Figure 3). Nonetheless, their performance remains comparable to the baselines.
GoF tests are effective at capturing CDF differences. We now summarize why GoF tests perform so well: they are particularly effective at capturing differences between the empirical CDF and the null No. At high temperatures, strong watermark signals push the empirical CDF away from Pag At low temperatures, repetition introduces additional structure that subtly distorts the CDF. While such deviations may be hard to detect using simple sum based baselines, GoF tests are specifically designed to identify such distributional whithe- large or subtle. This ability to exploit CDF- level differences explains their consistent advantage across watermarking schemes, temperatures, and editing settings. These findings suggest that Gos-d detection a promising and broadly applicable direction for future watermark detection research.

---

Table 3: Type II errors (averaged over three LLMs) the C4 dataset with temperature 1.0 under various editing types. "Del" denotes word deletion, "Sub" represents synonym substitution, and "Info" refers to information-rice edits. All values enlarged by 100 for readability. Baseline ine refers to the best-perforning method among the baseline detectors. Red shading highlights lower values; blue indicates higher values.
Edits
Baseline
Phi
Kui
Kol
And
Cra
Wat
Ney
Chi
@ 0.1
0.4
0.3
0.5
0.3
0.3
0.5
0.6
0.4
0.2
Del
Gumbel-man
0.2
0.5
0.4
6.0
2.8
0.7
2.4
8.1
0.8
1.9
@ 0.1
0.2
0.1
0.4
0.3
0.2
0.3
0.5
0.3
0.3
Sub
0.2
0.5
0.5
4.4
2.9
0.8
2.1
6.8
0.9
1.6
@ 0.3
2.4
1.3
1.1
1.7
1.4
1.7
2.0
1.7
1.9
Info
0.5
38.0
34.7
14.6
33.3
29.7
30.8
24.2
28.2
26.8
inverse-transfomm
0.1
0.3
0.9
0.6
0.2
0.1
0.1
1.0
0.3
1.6
Del
10.3
6.9
2.7
1.7
1.9
14.5
0.1
0.2
0.3
0.1
0.1
0.1
0.1
0.2
0.1
0.2
Sub
0.5
1.5
1.0
0.4
0.4
0.3
2.5
1.6
0.0
1.3
0.9
1.6
0.1
0.2
0.1
Info
34.6
33.7
0.4
19.6
16.4
0.1
2.9
2.0
3.2
2.4
2.0
2.5
2.1
2.1
Del
SynthID
7.8
16.9
9.8
9.0
24.4
12.6
0.1
2.3
1.6
2.7
1.8
1.5
Sub
0.2
5.5
5.9
15.3
8.7
5.1
7.9
21.9
6.0
11.5
0.3
16.6
14.8
2.6
Info
0.5
94.5
83.8
0.1
5.4
9.5
23.6
0.6
0.6
0.9
4.3 Results on Robustness
Finally, we present the robustness evaluation of different GoF tests, focusing on their performance under various editing scenarios. To encourage diversity and reduce repetition, the temperature is fixed at 1. For all experiments, the critical values (or thresholds) are set at a significance level of a 0.01, and we evaluate how the Type II error varies across different edit types and intensities. We report the Type II error at a token length of 250 for word deletion and 300 for synonym substitution and information-rice edits. Table 3 presents the Type II errors for watermarked texts, averaged across three LLMs on the C4 dataset. Results for ELI5 dataset are provided in Table 11 in the Appendix. The notation "editing method @ fraction" indicates that the editing method modifies the specified fraction of watermarked text or pivotal statistics. The observations are as follows.
Robustness against normal text edits. GoF tests show strong robustness against word deletion and synonym substitution edits. For all watermarks, GoF tests perform well as or better than the best baseline across all edit types and intensitics. Specifically, under 20% word deletion, for the Inverse-transform and SynthID watermarks, although the baseline achieves the lowest error, And trails by just 0.1%. Notably, among the considered GoF tests, there is no single method that consistently outperforms others across all scenarios. For instance, Phi has the strongest robustness in the Gumbel-man mmx watermark, while And outperforms the inverse-transfomm watermark.
Robustness against information-rich edits. GoF tests are designed detect global differences between the empirical CDF and the null rather than relying on few extreme values. As result, removing few of the largest pivotal statistics-whil while impactful for sum-based baselines-does not significantly alter the overall shape of the empirical CDF, allowing GoF tests to remain effective. This robustness evident under information-rie edits, where GoF tests consistently outperform baselines. For instance, Kui, Wat, and Chi maintain low Type Il errors across all watermarking schemes, even when 30% or 50% of the tokens are selectively modified. In contrast, sum-based methods often degrade under such edits because they depend heavily high-magnitude statistics. A clear example the Gumbel-max watermark: Aaronson's 's score hars = log(1 y) particularly sensitive to values of y near one. Since information-rich edits specifically target and remove these high values,

---

Table 4: Usage cases of GoF tests,
Usage Cases
Notes
Tests
Practical Scenarios
Low temperature with repetition
Repetition is more common at low tem- peratures and shifts the empirical CDF, which GoF tests exploit effectively.
And, Chi
Code generation (see Ap-
pendix D.2 for more dis- cussion)
High temperature
Strong watermark signals make devia- tions from Mo more detectable, improv- ing the power of GoF tests.
All general GoF tests.
Open-ended text genera- tion.
Common text edits
GoF tests remain effective despite moderale edits, but no single test con- sistently dominates.
Test choice varies.
Homework detection.
Information-rich edits
GoF tests are less sensitive to the re- moval of extreme values, preserving rohustness.
Kui, Chi.
Wat,
Internal API leakage.
this method's s effectiveness is sharply reduced. GoF tests, by focusing overall distributional shifts, avoid this pitfall and thus offer more reliable performance in adversarial settings.
4.4 Evaluation Summary
We conclude this section with brief summary of key findings. First, GoF tests are highly effective al detecting watermark signals at high temperatures. With a high temperature (e.g.. 0.7), they consistently achieve very low Type error rates regardless of the text lengths. Second, text repetition increases the deviation between the empirical CDF and the null distribution foo enabling GoF tests outperform baseline methods in low-temperature settings. Unlike baseline methods, GoF tests can effectively leverage these distribution-level deviations. Third, GoF tests are robust to common edits and information-rich edits. Their strength lies in evaluating overall distributional fit, rather than relying on individual extreme pivotal statistics, making them more resilient to various types of edits. Table 4 summarizes the practical usage of GoF tests.
Conclusion and Discussion
In this work, we propose the use of goodness-of-fi fit (GoF) tests for watermark detection and systemat- ically evaluate eight GoF methods across three popular watermarking schemes. Through extensive experiments across models, datasets, temperature settings, and editing types, we find that GoF tests can match or even outperform existing baselines in both detection power and robustness. We also identify text repetition as a key factor that enhances GoF performance low temperatures.
Our findings open several new directions for improving watermark detection with statistical tools. First, deeper theoretical understanding is needed to explain why certain GoF tests perform better in specific scenarios--pteeniallly potentially building on the statistical framework introduced by Li et al. [33]. Second, adaptive detection strategies that dynamically select among GoF tests based on input charac- teristics could further enhance performance in practice. Finally, while our work focuses on scalar pivotal statistics, some detection rules, such as Dathathri al. [9], have explored digh-dimensional statistics. Applying GoF tests in high-dimensions settings remains challenging due to the curse of dimensionality. Future work could investigate approaches such as maximum mean discrepancy or dimensionality reduction to extend GoF-based detection to these more high-dimensional scenarios.
Acknowledgments
This work was supported in part by NIH grants RO1 AG0711174 UOICA274576, RO1EB036016 6, R01E3037101, UO1 AG0668333, R01LM014731, and P30AG073105, NSF grant DMS-2310679, Meta Faculty Research Award, and Wharton AT for Business. The content solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
10

---

References
[1] Scott Aaronson. Watermarking of large language models. https://simons http:////simmommmmbeellllee berkeley. edu/
talks//coott-aronseeee aww-aaon-noouueee utt-usunnnnttttttiin---aauu appenai--223---2 17, August 2023. URL https: 17.
/ //simons. berkeley. edu/talks/scotte ett http://ooooo-rrroooost-tt-ttttta ut--ustiinnn www.-auutnnnnnaa----a-
[2] Theodore W Anderson and Donald A Darling. Asymptotic theory of certain "goodness of fit" criteria based on stochastic processes. The annals of mathematical statistics, pages 193-212, 1952.
[3] Jonas Becker, Jan Philip Wahle, Bela Gipp. and Terry Ruas. Text generation: A systematic literature review of tasks, evaluation, and challenges. arXiv preprint arXiv:2405.15604 2024.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pages 1877-1901, 2020.
[5] Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. In The Thirty Seventh Annual Conference on Learning Theory, pages 125--1139. PMLR, 2024.
[6] Harald Cramér. On the composition of clementary errors: First paper: Mathematical deductions. Scandinavian Actuarial Journal, 1928(1):1377, 1928.
[7] Ralph B D' Agostino. Goodness-of fit-techhniqes. fhoddhees--fffi-ttcchiiqqeu
Routledge, 2017.
[8] Debarati Das, Karin De Langis, Anna Martin, Jachyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Hayati, Risako Owan, Bin Hu, Ritik Parkar, et al. Under the surface: Tracking the artifactuality of LLM-ggnerated data, arXiv preprint arXiv:2401. 14698, 2024.
[9] Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbi, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd Al Merey, Jonah Brown-Cohen, Rudy Bunel, Borja Balle, Taylan Cemgil, Zahra Ahmed, Kitty Stacpoole, Ilia Shumailov, Ciprian Baetu, Sven Gowal, Demis Hassabis, and Pushmeet Kohli. Scalable watermarking for identifying large language model outputs. Nature, 634(8035) 18, Oct 2024. URL https: /doi. org/ 10. 1038/ s41586- 1115860024-08000 08025-805--00 4.
1038/
[10] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The Hitchhiker's guide to testing statistical significance natural language processing. In Annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1383-1392, 2018
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[12] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association Computational Linguistics, pages 3558-3567, 2019.
[13] Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. 2023 IEEE International Workshop Information Forensics and Security (WIFS), pages 1 1-6. IEEE, 2023.
[14] Jiayi Fu, Xuandong Zhao, Ruihan Yang, Yuansen Zhang, Jiangjie Chen, and Yanghua Xiao. GumbelSoft: Diversified language model watermarking via the GumbelMax-trin In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5791-5808, 2024.
[15] Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12848-1288 2021.
11

---

[16] Eva Giboulot and Teddy Furon. WaterMax: Breaking the LLM watermark detectability- robustnessqualltty trade-off. Advances in Neural Information Processing Systems, 37:1 17:18888- 1 18881, 2024.
[17] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: A series of lectures, volume 33. US Government Printing Office, 1948.
[18] Haiyun He, Yepeng Liu, Ziqiao Wang, Yongyi Mao, and Yuheng Bu. Theoretically grounded framework for LLM watermarking: A distribution- approach. In The Ist Workshop on GenAl Watermarking, 2024.
[19] Weiqing He, Bojian Hou, Tianqi Shang, Davoud Ataee Tarzanagh, Qi Long, and Li Shen. Sefd: Semantic-enhanced framework for detecting lmmgeneerated text. In 2024 IEEE International Conference on Big Data (BigData), pages 1309-1314, IEEE, 2024.
[20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
[21] Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang.
Unbiased watermark for large language models. In The Twelfth International Conference Learning Representations, 2024. URL https: /ttp:///o/ppeeeeeeeeee net peet/foruee htffffommmddi idwruuiiddd
[22] Baihe Huang. Banghua Zhu, Hanlin Zhu, Jason Lee, Jiantao Jiao, and Michael Jordan. Towards optimal statistical watermarking. In Socially Responsible Language Modelling Research, 2023, URL https: //tp:///////eeeiee//eeee
[23] Baihe Huang, Hanlin Zhu, Julien Piet, Banghua Zhu, Jason D. Lee, Kannan Ramchandran, Michael Jordan, and Jiantao Jiao. Watermarking using semantic-aware speculative sampling: From theory to practice, 2025. URL https://opeereviieee net aet/foormmiddLLAIneeeii
[24] Leah Jager and Jon A Wellner. Goodness-of-fit tests via phi-divergences. Annals of Statistics, (5(5):2018-2055 2007.
[25] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-SSfftaxx In International Conference on Learning Representations, 2016.
[26] Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models. In International Conference on Machine Learning, pages 22570- 225933. PMLR, 2024.
[27] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. watermark for large language models. In International Conference on Machine Learning, volume 202, pages 17061-17084, 2023.
[28] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of water- marks for large language models. In International Conference on Learning Representations, 2024. URL https: / http////peeeeeeeeeeeeeeeeeeeeeee wtw.peerreiiieeeeeeeorouuu/raaee http//forummidddeJJDJDmmm
[29] Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion- free watermarks for language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //tp://://///eeeeee//eee /ttt//oooooodddPPCC11110
[30] Nicolaas H Kuiper. Tests concerning random points a circle. In Nederl. Akad. Wetensch. Proc. Ser. A, volume 63, pages 38-47, 1960.
[31] Xiang Li, Feng Ruan, Huiyuan Wang, Qi Long, and Weijie Su. A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules. The Annals of Statistics, (3(1):322-351) 2025.
[32] Xiang Li, Garrett Wen, Weiqing He, Jiayuan Wu, Qi Long, and Weijie Su. Optimal estimation of watermark proportions in hybrid ai-human texts. arXiv preprint arXiv:2506. 22343, 2025.
[33] Xiang Li, Feng Ruan, Huiyuan Wang, Qi Long, and Weijie J. Su. Robust detection of watermarks in large language models under human edits. To appear in Journal of the Royal Statistical Society: Series B (Methodological), 2025+.
12

---

[34] Yepeng Liu and Yuheng Bu. Adaptive text watermark for large language models. In International Conference on Machine Learning, pages
PMLR. 2024.
[35] Chris Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances neural information processing systems, volume 27, 2014.
[36] George Marsaglia and John Marsaglia, Evaluating the anderson-darling distribution. Journal of statistical software, 9:1-5, 2004.
[37] Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity. In Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 5328-5339, 2021.
[38] Silvia Milano, Joshua McGrane, and Sabina Leonelli. Large language models challenge the future of higher education. Nature Machine Intelligence, 5(4):333-333 2023.
[39] George A Miller. WordNet: A lexical database for english. Communications of the ACM, 38 (11):39-41, 41, 1995.
[40] Jerzy Neyman. Smooth tests for goodness of fit. Scandinavian Actuarial Journal, 1937(3-4): 149-199, 1937.
[41] OpenAL. 2024. URL Understanding the source of what we see and hear online, May what- Beat-ww-aat see and hear hear--nlie/........
htmm//nddx//nddesaaddiga thee-theeeeee source-of--
[42] Karl Pearson. X. on the criterion that a given system of deviations from the probable in the case of correlated system of variables is such that it can be reasonably supposed have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 507302): 1900.
[43] Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. MARKMY- WORDS: Analyzing and evaluating language model watermarks. In IEEE Conference on Secure and Trustworthy Machine Learning, pages 68-91. IEEE, 2025.
[44] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[45] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International Conference Machine Learning, pages 28492-28518. 8. PMIR, 2023.
[46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(55:588-5551, 2020.
[47] Shayleen Reynolds, Saheed Obitayo, Niccolo Dalmasso, Dung Daniel Ngo, Vamsi K Potlura, and Manuela Veloso. Toward breaking watermarks in distortion-free large language models. In AAAI Workshop on Preventing and Detecting LLM Misinformation (PDLM), 2025.
[48] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can Al-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023.
[49] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data. Nature, 631 (8022):
755-759, 2024.
[50] Nikolai V Smirnov. On the estimation the discrepancy between empirical curves of distribu- tion for two independent samples. Bull. Math. Univ. Moscou, 2(2):314,, 1939.
[51] Kate Starbird. Disinformation's spread: Bots, trolls and all of us. Nature, 571(7766))44944555 2019.
[52] Stokel- Walker. Al bot ChatGPT writes smart essays-Shoule Should professors worry? Nature News, 2022.
13

---

[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
[54] George S Watson. Goodness-of-fit tests on a circle. Biometrika, 48(112): 1961.
[55] Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM conference on fairness, accountabilityy and transparency, pages 21 214-229, 2022.
[56] Yihan Wu, Zhengmian Hu, Junfeng Guo, Hongyang Zhang, and Heng Huang. A resilient and accessible cetributiio--reserviin watermark for large language models. In International Conference on Machine Learning, pages 53443-544 434400 PMLR, 2024.
[57] Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. Advances Neural Information Processing Systems, 35:3082-3095, 2022.
[58] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in neural information processing systems, volume 32, 2019.
[59] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Arterxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained 2022.
transformer language models. arXiv preprint arXiv:2205.01066
[60] Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu- Yu-Xiang Wang. Provable robust watermarking for Al-generated text. In International Conference Learning Representations, 2024. URL https:://///.. /opp/revieee/eeee////eee net tttt/forrumidddSSm18800......
[61] Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Permute-and-Flip: An optimally robust and watermarkable decoder for LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: haaes://oopae /oppnneveeeeeeeee
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract are completely based on our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitations and the potential future work together in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Disclosed all the experiment information Section 4 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: All data are open-source, information is provided in the content, and code is provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.. "nips.cc/ppppp cccppblic/guut gwwggguiieccsssii
coimiisssoooicyyyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We detailed all the experiment information in Section 4 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In our experiments, we evaluate our method and baseline methods on three different LLMs, using 1,000 samples per model. We report the average performance across the LLMs. While we do not include explicit error bars, the use of multiple models and large sample size provides strong basis for statistical significance.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that http://uurripscccpuulieeeee cwtppbpic//ttiiiiiee htmbbli//Ehicccuudeeliiiee The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in D.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted this paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper focuses statistical tests and does not involve the release of data, models, or other assets that pose a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the data, code, and models used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
This work involves the use of large language models to generate watermarked text sequences, which serve test data for evaluating the robustness and effectiveness of our proposed statistical detection methods. We provided the detailed usage of LLMs in Section 4 and Appendix D.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy betts://neurrie cct / for what should or should not be described.
21

---

Appendices
A Introduction of Considered Watermarks
A.1 Gumbel-max Watermark
The Gumbel-max watermark the first unbiased watermark [1], which has been implemented internally at OpenAI [41] and serves baseline in many studies. This watermark builds on the Gumbel-mas max trick, a sampling technique for multinomial distributions [17, 35, 25]. The trick first generates random vector < (Uw)we wew consisting of [W]] i.id. copies of (0, 1) and guarantees that arg wa P--1 log U. follows the distribution P = (Pw)wew- Recognizing this fact, Aaronson [1] proposed the following decoder:
log Un
S (P.C) == arg max wEW P.
(5)
which is, by definition, unbiased [13, 43, 61, 31J.
The pivotal statistic is Yú = (w,, (t) w...,. where &. = (UI,w) collects all pscudorandom numbers and Ut, is the entry corresponding the selected token wy. Without a watermark, the Un,,,'s 's are i.i.d. from (0, 1), so Y; ~ Mo = UO, 1). In contrast, if watermark is present, (5) makes tokens with larger pseudorandom numbers more likely to be selected. Indeed, we have Y, P. P. where pi < r) Enew Purt/P= for [0, 1] [31]. This alternative distribution differs from Po unless P is degenerate (i.e., concentrated on a single token).
A sum- hum-based detection rule is defined by scalar score function h. The watermark is claimed when the sum sum-based statistic Tū = Σm= h(Y)) exceeds a chosen threshold. Existing scoring functions are listed in Table 5. Examples include Aaronson's 's score hars (v) = log 1 1-- the log function
du (v)
(y) = log [29, 13], and the optimal least-favorable score hg((y) = dua(y) from Li et al. [31], where A is user-specified regularity parameter and PY == (1 A, A, is the least favorable NTP distribution. The function hyg is minimax optimal when each P. belongs to A regguar class, denoted Pa. defined as:
PA = (P max P.r S A}.
WEW
Li et al. [31] shows that dIg achieves the fastest exponential decay of Type error at a fixed significance level when testing against the least favorable distribution in Pa.
Table Baseline sum-based detection rules. Since Lst can be applied to all watermarking schemes, we tune the parameter A for each to ensure good performance. In our experiments, we set A = 0.2 for both the Gumbel-max and SynthII watermarks, and A 0.5 for the Inverse transform watermark.
Tests name
Score function
Notation
d#1, (v) duo(v)
favor-favorable test [31]
Ma(2/) =
Lst
Aaronson's score [1]
hur (y) log T-
Ars
Logarithmic score [29]
Bing (y) log(y)
Log
Negative-sum score [29]
Aneg( (v)
Neg
Identity-sum score [9]
hu(l)) y
Sum
A.2 Inverse Transform Watermark
The inverse transform watermark, proposed by Kuditipudi et al. [29], is another unbiased watermark- ing scheme. It uses a pseudorandom variable Ç <= (), where U U(0, 1) is uniform random variable and π is an independent random permutation over the vocabulary W. The decoder is defined via inverse transform sampling:
Sinv (P.C) =- FF (U),
22

---

where (-) denotes the CDF under the permutation defined as Fx(x) = -ww'ew Pwr - By construction, the inverse transform watermark is unbiased for sampling from the NTP distribution P. To see the unbiasedness directly, for any token w, note that Sinv (P.C) = and only if
Σ Pw 1 (x("")<<())) S U
Σ Pw (
W'W
N'EW
which occurs with probability Pw since the interval above has length Pw-
The associated pivotal statistic Y(w. <)
U 7/ (w)), where n=(w) s(w) is the normalized rank of u in the permuted vocabulary. Under Ho. Y follows distribution with CDF 2² for [0, 1]. Under H1, the distribution is more complex, though closed-form approximations are available in certain asymptotic regimes [31]. Detection is performed using sum-based rule with score function heg (3) = y [29].
[W]-1
A.3 SynthID Watermark
The SynthID watermark. proposed by Google [9], is based on novel sampling strategy for categorical distributions called tournament sampling. Given number of tournament rounds k; the pseudorandom seed is defined as Ç = (g+)²= where each gi = (gt,)) is a vector i.i.d. samples drawn from UO, 1). Unlike the previous two watermarking schemes, the decoder here is stochastic rather than deterministic. The next token is sampled from a modified NTP distribution, i.c., w ~ S (P, C).
where
S (P,5) = Tyn .... o To (P),
(8)
where Ty is vectorized operator defined
(Tg(P)) = Pi
Pur +
Σ
Par
(9)
w w':9w' S.9c
Dathathri et al. [9] prove that the output of Ty (P) corresponds to the distribution of the winner token one-versus-one competition, where two independent tokens sampled from P compete based on their assigned g values. Repeating this process over &o rounds with i..d. pseudorandom vectors g: results in P. being the distribution of the final tournament winner after rounds of competition.
For detection, we recover the g-values Ç (g:) and extract the coordinates corresponding the selected token w, i.e., 10t.whee Under the tournament sampling scheme, these values tend to be larger under H₁ than under Hg. due the minner-takes-ae interpretation. We consider the sum-based detection rule proposed by Dathathri et al. [9] our baseline, which computes simple average over selected g-values:
1
y.um (w, ()
9t.w-
For simplicity, we use the identity score function in our implementation. Under Ho. each Ji.w is an independent (0, 1) random variable, so y'uu (w, ς) follows scaled Irwin-Fae Hall distribution:
yrsum (w., () (rw. In their paper, k = 30.
Note that Dathathri al. [9] also proposed weighted sum detection rule; however, we do consider it in our evaluation, as its null distribution (40 does not admit a closed-form expression and its empirical performance closely matches that of the unweighted sum statistic ysum especially high temperatures. In addition, they propose a Bayesian detector based on a neural network trained to distinguish human-wringg written from watermarked text. We exclude this baseline well, since its null distribution is analytically intractable due to the network's 's complex dependence on both the token w and the randomness Ç.
A.4 Green-red List Watermark
The green-red list watermark, introduced by Kirchenbauer et al. [27], is a simple yet effective scheme. It randomly partitions the vocabulary W into green (favored) set and a red (disfavored) set. Given green-list fraction 7 E (0, 1), selects a random subset G W of size [ = > Y| as the green set; the remaining tokens form the red set.
23

---

The decoder is stochastic: the next token w is sampled from a shifted version of the original NTP distribution P. that w Stn (P.5) where
Pw exp(d)
if w E g, otherwise,
S (P)))(w= =
Z
Pa
Z
and Z is the normalization constant. The pseudorandom component here the green set G.
For given token, detection is based on whether it belongs to the green set: the pivotal statistic is Ys (w, () Tweg. Under Ho. independent of the token, so Ysrl Ber(y). Under H1, the decoder increases the likelihood of selecting green tokens, resulting in Ygd Ber(()), where
Eweg Per exp(o) Eweg Pe exp(o) + Ewgg Pa
f1
Since μa > in general, detection reduces to checking whether the number of green tokens E Y exceeds a threshold.
Variants of this scheme, such as DiPmark [56), use more sophisticated partitions, but the core idea remains the same: secretly boost the probabilities of selected tokens and count how often they appear.
Why we don't consider green-red list watermarks. We did not include the green-red list watermark and its variants in experiments because GoF tests reduce to the original detection method in this setting. This reduction happens because the pivotal statistic (i.e., whether token is green) is binary (0 or 1). and the order of these values does affect the test. As result, the full information is captured by simply counting the number of 1s. In this case, testing whether the number of Is exceeds threshold (i.c., the original detection rule) is already highly effective, and both GoF tests and the original method rely on this same statistic maaiing them essentially equivalent. Therefore, additional experiments would be redundant.
B Related Work
Most related work. Our work evaluates general GoF tests for watermark detection. The closest prior work is Li et al. [33], which proposed a parameterized GoF test specifically for the Gumbel- max watermark and established
statistical optimality under certain settings. contrast, we study a broader class of non-parameterized GoF tests applicable to wide range of watermarking schemes, offering greater ficxibility. We also uncover new insights not included [33]: the role of text repetition in watermark detection. While several prior studies [13, 28] have examined text repetition, they mainly focused on mitigation strategies or preprocessing methods. Our work is distinct in that we analyze repetition from statistical perspective, focusing on how it affects the behavior of pivotal values and, in turn, the reliability of GoF-based detection compared to sum-bart based approaches. Through ablation studies, show that GoF tests exploit repetition particularly in low-temperature decoding to boost detection performance. This phenomenon, is not captured by Li al. [33], offers new understanding of how GoF tests succeed in practice.
Other optimal detection rules. Several works focus on optimal detection for specific watermarks.
Huang et al. [22] derive optimal rules which, however, require exponential computation or access to full NTP distributions, limiting their practicality. Li et al. [31] assume all NTP distributions fall into a prior class of Pt. and develop worst-case optimal detection rules using minimax formulation. Huang et al. [23] study speculative sampling, which is beyond our scope. He et al. [18] propose variant of the Gumbel-max watermark together with an optimal detection rule optimized for detection power, sample distortion, and Type-I eиror. While these methods target specific schemes, our focus is on general GoF tests, which can be applied broadly as long as the null distribution No is tractable.
General on watermarking schemes. A wide range of watermarking schemes have been proposed in recent work [28, 13, 29, 21, 56, 61, 60, 34, 16, 14]. At high level, these methods define pseudorandomized sampling procedures over next-token prediction (NTP) distributions, where the seeded randomness allows watermarked tokens to be reliably identified at test time. decoder is considered unbiased if the watermarked token distribution exactly matches the original NTP

---

distribution; otherwise, it is biased [31]. In practice, each new watermarking scheme is often accompanied by a custom detection rule designed in an ad hoc manner. Many such examples are provided in Appendix A. In contrast, our work focuses on evaluating NTP-agnostic GoF tests as general detection tools. Our goal to systematically assess their performance across diverse watermarking methods and to understand the key factors that influence their detection effectiveness.
GoF tests in statisties and NLP. GoF tests are standard tools for assessing whether i.i.d. samples follow a specified distribution [7]: see Section 3 for representative examples. These tests are particularly effective in low-dimensional settings and often rely on empirical CDFs or binned statistics. To our knowledge, we are the first to apply GoF tests to watermark detection. A key distinction is that watermark detection involves non-i.idd samples under H₁ due to the autoregressive nature of LLMs: each P. depends on previous tokens. Our work builds on the frameworks of Li et al. [31, 33] but shifts the focus toward broader applicability and robustness. GoF tests have also been used in NLP for analyzing language statistics [37] and evaluating significance in empirical studies [10]. Our work connects these statistical tools to the challenge of modern watermark detection.
C Practical Considerations for GoF Tests
Asymptotic Properties of GoF Tests. Most of the classical GoF tests considered in work-Tr- GoF test (Phi), Kuiper (Kui), Kolmogorov-Snie Sanirovv (Ko1), Anderson-Darling (And), Cramér-von Mises (Cra), and Watson (Vat) known to have only limiting distributions under the null hypothesis. That is, their exact finite-sample distributions are analytically intractable. In contrast, one of the baseline methods, Aaronson's score (Ars), admits explicit Gamma distribution under the null, and prior work [13] has demonstrated that it can provide strong non-asymptotic guarantees for Type I error control. However, despite the asymptotic nature, we argue that GoF tests remain practically reliable for watermark detection. Our rationale is threefold:
Empirical reliability of Type I error control. It's worth noting that the limiting distribu- tions offer valuable theoretical tools, but in practice, simulations offer users a more flexible and convenient way to compute critical values-like in Python package SciPy4, due the complex forms of some distributions (c.g.. the Anderson-Daiie though critical values are obtained from simulations, our experiments (see Figure 4) show that these GoF tests consistently maintain Type error rates (i.e., FPR) close to the nominal level, even at stringent settings (e.g., a = 10 10-5).
distribution [36, 2)). Even
Validity of dimulation-based calibration. Simulation provides a principled and widely adopted approach for obtaining critical values when exact distributions are unavailable. By generating i.i.d. pivotal statistics under the null distribution H4, one can estimate the empirical a quannielle with arbitrarily small error by increasing simulation size. This calibration can be performed prior to deployment and reused across tasks.
Practical robustness in Type II error: Beyond Type I control, we find that GoF tests retain strong detection power (i.e., low Type II error) under low a levels. Table demonstrates that even when the Type error is tightly constrained, GoF texts continue to outperform the baselines.
Complementary Role of GoF Tests. Importantly, we do not view GoF tests as replacement for sum-based or non- -on-ssymtooicc methods, but rather as complementary tools. While methods like Ars target high-value pivotal statistics, GoF tests are designed to detect distributional shifts, offering enhanced robustness especially under information-rice edits (see Section 4.3). Despite relying on asymptotic or simulated critical values, GoF tests enrich the detection toolkit by capturing distribution-level anomalies.
Our experiments show that GoF tests provide reliable Type I error control and strong detection power, yet their reliance on asymptotic or simulation-based thresholds means they may fall short in high-stakes settings-for example, legal proceedings where certifying false positive rates with exact finite-sample guarantees is required. In such scenarios, GoF tests alone are insufficient. More broadly, no single method should serve as the sole line of defense. A robust system should draw from multiple
'https://doce. scipy. wtt.:///w/cc////eeteee/////////e html
stats, wwwpssoooodossoooeooe off.
25

---

Phi
Kni
Kol
And
Cta
Wac
Chi
Gumbel-max
Invers-trranform
SynchID
Eanpitiea FIRt
10-3
2
10
10
T Thorotica FPR
Thocsticall FPR
Thooeiioal EPR
Theoretical empirical false positive rates (FPR) for GoF tests. Results are based 100k sampled human-writted texts from the C4 dataset, with repeated pivotals removed. Even at this scale, GoF tests maintain strong Type error control, achieving empirical FPRs as low as the target significance level.
Figure 4:
Table Type II errors at stricter Type error level 10-5, Results are reported for temperature T = 0.7 and sequence length 400, averaged over three LLMs the C4 dataset. Basel ine denotes the best- pest-performing method among all baseline detectors. The results show that GoF tests retain strong detection power even under tight Type error constraints.
Watermarks
Baseline
Phi
Kui
Kol
And
Cra
Wat
Ney
Chi
Gumbel.
1.1
1.1
0.7
0.9
0.7
0.9
1.2
0.6
0.4
Inverse.
2.6
17.3
1.4
1.2
1.2
1.6
2.9
1.8
1.5
SynthID
4.2
18.0
4.6
4.1
3.9
4.7
11.8
4.1
2.7
methods, each with different strengths- for instance, Ars for sensitivity to large deviations and GoF tests for distributional anomalies.
D Experiment Details and Results
D.1 Prompts for text generation
For the text completion task, we truncated the first 50 tokens directly, without adding any instructions, and used them the prompt. For the long-form question-answering task, manually prepended the instruction "Answer the following question." " before the question and used this as the prompt. To ensure uniform prompt lengths, we selected questions whose total length, including the instruction, was within 50 tokens. Additionally, we added padding tokens at the beginning to ensure the final prompt length reached 50 tokens. Table 7 shows example prompts for these two tasks.
Table Example prompts the text generation tasks.
Text completion
Long-form question-answering
The Arlington County Board plans to vote Saturday afternoon on giving Amazon 23 million and other incentives to build headquarters cam- pus in Crystal City, but only after hearing scores of northem Virginia residents and advocates testify for or against the project. in The iivemember
Prompt:
Prompt: [pad] [pad]... [pad] Answer the fol- lowing question: Why is it that we calm down when we take deep breath, hold it for few sec- onds and exhale?
D.2 Example of repetition
In this part, provide two examples of repetitions in both human-written and watermarked text, where we use colored backgrounds to highlight the most repeated content. The first example comes from the C4 dataset, which is an NBA game report in which the phrase "during the second half"
26

---

appears 15 times. As discussed in Section 4.2, such repetitions can introduce distinct jumps in the empirical CDF of pivotal statisties, potentially leading GoF tests to incorrectly reject the null hypothesis. The second example, generated by Llama 3. 3.B with temperature of 0.7 using the Gumbel-max watermark for the text completion task, exhibits a more direct and deliberate pattern of repetition compared to human-written text.
Example of repetition in human-written text: Lakers guard Kobe Bryant flys to the basket with Rockets forward Shane Battier trailing during the second half. Rockets forward Luis Scola drives around the Lakers' Luke Walton during the second half. The Rockets' Aaron Brooks jumps up as the crowd goes wild after one of his three-point shots during the second half. The Lakers' Ron Artest battles with the Rockets' Chuck Hayes for loose ball during the second half. Rockets guard Aaron Brooks drives around the Lakers' Derek Fisher and Lamar Odom during the second half. Rockets forward Trevor Ariza drives up the court against the Lakers' Lamar Odom during the second half. The Rockets' Kyle Lowry drives up the court against the Lakers' Derek Fisher during the second half. The Lakers' Ron Artest with his haircut during the second half, Lakers guard Kobe Bryant flys to the basket with Rockets forward Louis Scola trailing during the second half. Ropckets forward Trevor Ariza goes up for basket during the second half. The Lakers' Kobe Bryant gets shot off as the Rockets' Shane Battier tries to defend during the second half. The Lakers' Kobe Bryant bumps into the Rockets' Shane Battier during the second half. The Lakers' Kobe Bryant boocd by fans during free throw during the second half, Lakers forward Ron Artest fights Rockets forward Chuck Hayes for the ball in the fourth quarter. Lakers forward Ron Artest puts hand in the face of Rockets forward Carl Landry during the second half. Rockets guard Trevor Ariza celebrates with Pops Mensah- Mensah-Bonsu after his three- point shot during the last seconds of the second half. The Rockets' Trevor Ariza tries get his hands on ball held by the Lakers' Ron Artest during the second half.
Example of repetition in LLM-written text: Two slaughtermen have been sacked after an undercover investigation exposed shocking cruelty to horses at an abattoir. The disturbing video shows them being beaten with metal poles and illegally stunned in groups of up three at time before being killed. In one clip, one of the workers can be heard saying: "We'n re going to kill them now. Don't 't beat them up. Don't 't beat them appDon't t beat them up Don't beat them up.Don't 't beat them up. Don't ton't beat them up...
Repetition in more powerful LLMs. Importantly, extensive repetition will become increasingly rare in newer and more powerful L.Mss, even at low temperatures. For instance, closed-source models such as GPT and Claude are capable of producing diverse outputs under low-entropy (low- temperature) settings without excessive repetition. Thus, as summarized in Table 4, the specific low-temperature scenario with heavy text repetition should be regarded as an increasingly rare phenomenon in more advanced models. Accordingly, our analysis of the text repetition can be viewed as an edge-cas cage study that reveals how repetition inftuences the behavior of pivotal statistics and, in turn, the reliability of GoF tests compared to sun-based methods. More generally, detection in low-entropy settings without text repetition remains a difficult problem for GoF tests, as it does for other methods, making it a challenging yet worthwhile direction for future research.
D.3 Cost of GoF tests
The primary computational cost of goodness-of-fit (GoF) tests in watermark detection lies in calculat- ing the critical values. Most GoF tests inss--inluiing all those used in this work- lack non-asymptotic distributions but do have asymptotic ones. For tests with simpler asymptotic forms (e.g., Ney and Chi), the critical values can be computed directly from asymptotic results, leading to runtime comparable to sas-based baselines.
In contrast, several GoF tests (e.g., Kui, Kol, And, Cra, Wat) rely on Monte Carlo simulation for critical value computation due to the complexity of their asymptotic distributions. This substantially increases computational cost.
Importantly, once the watermark scheme is fixed, the null distribution is known prior to testing, enabling the critical values to be pre-computed and reused across runs. As a result, the actual cost of applying GoF tests in practice is very low, even for those tests that require Monte Carlo simulation.
Table 8 reports runtime for detecting 100 pivotal sequences with n = 400 and simulation size 10 For comparison, the Ars baseline takes 1.30 x 10-3 seconds. As shown, Monte based GoF tests require roughly 10s times longer than the baseline, primarily because of the 103-fold larger simulation size needed for estimating critical values.
27

---

Table 8; Runtime cost of GoF tests for detecting 100 pivotal sequences with 400 and simulation
size 10". Time* indicates the time cost without calculating the critical value.
Test
Time
Time*
Phi
2.11 109
2.20
10
Kui
2.03 x 10
1.50 x 10-3
Kol
1.95 10
1.40 10 3
And
2.61 109
4.30 x 10
Cra
2.37 x 10
1.70 x 10-3
Wat
2.38 10
1.80 10-3
Ney
4.00 10
same as left
Chi
3.90 x 10-3
same as left
D.4 More experiment results
All text generation tasks were conducted on NVIDIA A100 GPUs, with a total computational cost of approximately 360 GPU hours to reproduce all experiments. Here, we present the results of additional experiments that were not included in the main content. Table 9 provides list of additional results along with brief descriptions.
Table 9: List of additional results.
Label
Brief description
Figure 5, 7 7
Type II errors for detection tests applied to the Gumbel-max, Inverse-transfone and SynthID watermark.
Figure
Percentage of repetition in watermarked texts compared to human written texts. Distribu- tion of the highest probability in NTP distributions.
Table 10
Type II errors the ELI5 dataset.
Table 11
Type II errors on the ELIS dataset under various types of edits.
E
Additional Discussions
E.1
The influence of different LLMs
Different LLMs may produce different next-token prediction distributions under the same prompt, which can impact the generation of watermarked text. In our experiments, we apply three popular open-source LLMs: OPT-1.3B, OPT- 13B [59], and Llama 3.B [11]. These models vary in their text generation capabilities, as indicated by their rankings on Hugging Face's 's Open LLM Leaderboard with 1.TB < 13T-13B < Llama 3-B in terms of overall performance. In our settings, OPT models illustrate scaling effects within family, while Llama 3-B provides a contrast with newer architecture and training paradigm, giving us both comparability and diversity in model selection.
Across all detection tests, we observe some variation in performance among the three LLMs, but the differences not substantial. More importantly, the relative performance trends between tests remain similar across models (see Figure 5, 6, 7). While specific rankings may fluctuate slightly, tests that perform well on LLM generally show comparable strength on the others. For example, Chi consistently achieve strong performance at low temperatures across all three LLMs. In other words, while absolute performance varies by model, the overall patterns in test effectiveness largely preserved across different I.LMs.
While our analysis is limited to three representative open-source models, the consistency of results across both within fallil scaling (OPT-1.3B vs. OPT-13B) and cross-family comparison (OPT vs.
"sccps:/ttuggggggace..o//// co/peen 11m leaderboard
28

---

Llama 3.1) provides encouraging evidence that the relative effectiveness of GoF tests is not highly sensitive to model choice. Architectural and training differences may lead further variation in other settings, but our findings suggest that the key performance patterns we identify are likely to extend beyond the specific models studied here. Moreover, even if the performance of a specific test does not transfer directly to new setting, the low computational cost of these GoF tests (see Appendix D.3) allows practitioners to easily run all suggested tests and select the best-performing one in just a few minutes. This makes our approach flexible and broadly applicable, while further evaluations across additional model families and domains can continue to strengthen these conclusions.
E.2 The influence of different text generation tasks
LLM performance varies by task, potentially impacting text generation quality. For example, Question- Answering requires a stronger ability for LLM to generate valid responses than Text Completion [3]. From our experiments, we found that task type has a greater effect on detection test performance than differences between LLMs. Specifically, as shown Figures noticeable performance drops on the ELI5 dataset at temperatures 0. 0.3, and 0.7, while GoF tests demonstrate stronger effectiveness on it. This can be attributed to the influence of repetition, as discussed in Section 4.2. LLMs tend to generate more repetitions in long-form question-answering tasks compared to text completion, especially at low temperatures. This benefits GoF tests while diminishing the effectiveness of sum-based baselines. Additionally, we observe a high occurrence low temperatures. To take an extreme
and 7, baseline methods exhibit
of ineffective outputs in long-form question-answering example, when answering the question: "Why does 1080p on 4K TV look better than 1080p 1080p TV?", LLaMA 3. I-BB generates: "I; 1;1:1:1:1:1:::1;::::
with temperature set to 0.1. Two factors may contribute to this issue. First, the LLMs applied may not be strong enough to generate valid answers for certain questions. Second, some questions inherently lack sufficient content for an extended response, even for human writers.
F Broader Impacts
This paper investigates the use of classic goodness-of-fe fit (GoF) tests for detecting text watermarks in J.LM-generaie outputs, aiming to enhance the reliability and robustness of watermark detection. A potential positive societal impact is the promotion of content authenticity and provenance in an era of rapidly growing Al Al-generated media. Stronger watermark detection techniques can support efforts in combating misinformation, safeguarding intellectual property, and promoting transparency in automated content creation.
However, there are also potential negative societal impacts. Improved detection capabilities might encourage more widespread use of watermarks, including those that could be embedded without user consent, raising privacy and surveillance concerns. Additionally, the detection tools could be misused to reveal the use of LLMs in sensitive or anonymized content, possibly compromising user confidentiality.
To mitigate these risks, it is important that watermark detection methods deployed transparently and ethically, with clear communication about their use and limitations. In conclusion, while our work improving watermark detection for language models has the potential to strengthen content attribution and support responsible Al use, it is important to carefully consider and address potential negative societal impacts ensure the technology is deployed ethically and transparently.
29

---

Log
Roi
And
Wid
Chi
10'
30F
10
Fwo7
10-1
10
0.6
0.4
10
10
10-2
10"0
10"
1.0
107
107
10
011
0.
T Twu
0.8
10-1
10
0.6
10-4
10-1
10-2
1.04
107
102
10"
6.1
7 0.3
0.8
10-14
10
0.4
10-2
10
Type 11 Rote
10
1.0
107
100
107
088
T 6.161
Two7
10-1
10-1
066
10
20.
0.0
1.0
107
11/-
100
0.8
1 61
03
0.2
0.0
1.0
107
100
10f
0.8
0.6
16
Davi
230
100
200
400
200
200
400
Tength of soseemenked terct
Figure 5: Empirical Type errors for various detection tests applied to the Gumbel-max watermark across the C4 dataset (top three rows) and the ELI5 dataset (bottom three rows).
30

---

1st
Plu
Ral
Ney
And
107
10
10-1
10-1
0.4
10-1
10
0.2
1010
10"
1.0
107
10F
T Twul
10-1
10
0.6-
10-1
11"-
0.2
10-1
1.04
107
102
10f
T 6.1
T Fw007
10
0.6
10
0.4
10-4
Type 11 Rote
10
1.04
107
107
10F
T 6.16.
T=07
10-1
10-1
10-1
10
10
0.0
10
10
107
100
00-1
10-1
10
10
a.
1.0
107
10/
108
0.8-
0.7
10
10"
10"
10
230
400
200
400
200
400
200
000
Tungth b. of soseemenked teec
Figure 6: Empirical Type Il errors for various detection tests applied to the Inverse-transfone watermark across the C4 dataset (top three rows) and the ELI5 dataset (bottom three rows).
31

---

Sum
Koi
And
Wii
Chi
(1641.50 50
(1pe-136
088
10-1
10-2
0.2-
6.0
10
Jetas 1-8
Bose 1:4
0.6-
10-1
0.4-
10-2
rgt-126
320-326
0.8-
0.6
Type Tima
0.2
10-2
10
108
(yst-18
100
T=07
10.1
0.4-
10-2
10-2
0.2
0.0
10
107
1074
10
0.8
10-1
0.6
10
0.2
10-2
1.0
rgt
10'
10/
10F
10-1
10-1
10"14
0.4
10
10-4
0.2
0.0
10
10
wokermaabeed
Figure 7: Empirical Type II errors for various detection tests applied to the SynthID watermark across the C4 dataset (top three rows) and the ELI5 dataset (bottom three rows).
Deporsiin Peeeettgg (98
0.5
0.7
1.3
Homan
Laique colous
201
00
Dradty
10gasu
Figure 8: Left: Average repetition rate in Inverse-transform watermarked texts across three LLMs, compared to human-written text. Right: Distribution of the highest probability in NTP distributions for OPT-1.36 1.3B using the Inverse-transform watermark at temperature 0.7.
32

---

Dapestiim Tunvsting (15
T-0.1
T-0.3
T-007
T-13
Hunan
Reponted Laique tokors
20"
67-
Dents
20-
S-pro
Fgrom
Figure 9: Left: Average repetition rate in SynthID watermarked texts across three LLMs, compared to human-written text. Right: Distribution of the highest probability in NTP distributions for OPT-1.3B using the SynthID watermark at temperature 0.7.
Table 10: Type II errors (averaged over three LLMs) on the ELI5 dataset. All values enlarged by 100 for readability. denotes temperature and n the token length. Baseling ine refers to the best-performing method among the baseline detectors. Red shading highlights lower values; blue indicates higher values.
n
Baselimg inn
Phi
Kol
And
Cra
Wat
Ney
Chi
Gumbel.
200
46.2
23.8
8.1
15.3
18.5
25.2
16.8
0.3
400
39.9
8.4
1.9
3.9
4.9
9.0
3.9
9.8
1.6
200
2.6
1.0
1.6
1.7
0.7
400
7.4
0.9
0.5
0.6
0.7
1.2
0.7
1.2
0.6
Inverse.
0.3
200
44.4
33.0
8.8
15.0
19.8
400
35.0
9.9
0.8
2.2
4.1
7.5
2.6
8.4
0.4
200
5.3
2.8
0.6
0.7
1.2
2.2
1.3
2.1
0.3
0.7
400
4.8
0.5
0.1
0.2
0.1
0.4
0.3
0.8
0.1
SynthID
0.3
200
60.4
40.6
21.5
30.7
34.0
41.8
31.0
39.2
11.5
400
49.3
15.6
5.8
9.9
13.4
19.0
10.2
18.8
3.2
200
6.1
4.5
4.1
3.1
2.9
3.9
7.5
4.2
2.9
0.7
400
5.5
1.5
1.2
1.0
0.8
1.2
2.3
1.5
1.2
Table 11: Type II errors (averaged over three LLMs) on the ELI5 dataset with temperature 1.0 under various editing types. "Del" denotes word deletion, "Sub" represents synonym substitution, and "Info" refers to information-rich edits. All values are enlarged by 100 for readability. Basel ine refers to the perforerfo method among the baseline detectors. Red shading highlights lower values; blue indicates higher values.
Edits
Baseline
Phi
Kui
Kol
And
Cra
Wat
Ney
Chi
Gumbel-man -mxx
Del
0.1
1.1
0.3
0.5
0.6
0.4
0.8
1.2
0.5
0.2
0.2
1.6
0.8
4.4
2.7
1.3
2.8
6.3
1.2
1.5
0.1
1.3
0.3
0.2
0.4
0.3
0.7
0.5
0.3
0.2
Sub
@
0.2
1.5
0.6
3.2
2.4
1.0
2.4
4.7
1.1
0.9
0.3
5.2
5.0
1.4
2.0
2.0
2.4
1.9
1.9
1.4
Info
@ 0.5
29.6
44.2
12.1
22.0
21.6
22.7
18.6
21.3
17.5
Inverse- Inverse-transee transform
0.1
0.8
0.9
0.5
0.3
0.4
0.5
1.0
0.6
1.1
Del
0.2
2.5
6.1
5.3
2.6
2.0
2.5
7.2
3.2
10.2
0.1
0.7
0.6
0.2
0.2
0.2
0.3
0.3
0.3
0.3
Sub
0.2
1.0
1.4
1.0
0.4
0.5
0.5
1.1
0.6
1.5
0.3
2.3
6.2
0.1
1.6
0.9
1.7
0.2
0.3
0.1
Info
0.5
20.7
50.9
0.3
10.5
11.5
14.5
0.9
1.7
2.2
0.1
2.3
1.2
1.0
1.0
1.1
1.7
2.2
1.4
0.6
Del
SynthID
0.2
4.8
4.6
11.9
6.8
4.1
6.2
17.9
4.8
7.0
@ 0.1
2.5
1.1
0.8
1.0
1.3
2.0
1.8
1.4
0.5
Sub
0.2
5.6
4.8
9.6
6.5
4.1
6.4
14.1
4.7
6.5
@ 0.3
8.9
4.9
0.0
0.7
0.5
1.7
0.1
0.6
0.0
Info
0.5
81.3
64.3
0.1
6.9
7.0
14.9
0.4
0.7
0.5
33


Reviews:
[{"review_id": "kyXfOtXA6Y", "reviewer": "Reviewer_wNnw", "summary": "This paper proposes using eight classical goodness-of-fit (GoF) tests to detect watermarks in LLM-generated text as better alternatives to common sum-based methods. It presents a systematic evaluation across three watermarking schemes, three open-source LLMs, and two datasets, and with an interesting finding that text repetition—especially prevalent in low-temperature generations—introduces detectable patterns that GoF tests can effectively exploit to enhance detection performance.", "questions": "I understand that GoF tests as-is don’t help directly with green-red watermarking (due to its binary pivotal statistic), so this is an open question. Given that green-red list–based watermarking is among the most widely used and dominant classes of watermarking schemes in recent work, do you see any way this paper's insights (e.g., the robustness of GoF tests) could be adapted to improve the green-red watermarking schemes?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- While GoF tests are classical tools, their application to LLM watermark detection is novel and provides a principled statistical framework for understanding and improving watermarking techniques.\n- The experimental setup is relatively comprehensive, covering three open-source LLMs, four temperature settings, and three types of text edits.\n- The paper presents a rigorous discussion on the information-loss of sum-based statistics and the effectiveness of GoF tests in capturing CDF-level shifts, especially under low temperatures. The observation that repetition introduces useful signals is also interesting.\n- The paper is generally well-structured and easy to follow.\n\n### Weaknesses\n- In each setting, only a subset of GoF tests outperforms the baseline, and no single test consistently dominates. Therefore, a more detailed discussion with practical guidance on selecting the most appropriate GoF test would greatly enhance the paper's applicability.\n- While the performance gains from applying classical GoF tests for watermark detection are arguable (as mentioned above), I view this work as a benchmark-style study that systematically evaluates existing unbiased watermarking schemes using a consistent evaluation \"ruler\". To this end:\n     * Could the authors discuss the potential for establishing a unified evaluation scale, e.g., whether the results from the eight tests can be meaningfully aggregated?\n     * Including more diverse datasets if possible", "was_revised": true, "final_justification": "The paper applies classical GoF tests to LLM watermark detection with comprehensive experiments, which I believe provide a principled and interpretable statistical framework for the domain. My main concern—the lack of practical guidance on choosing GoF tests—has been addressed with a clear plan to add concrete application scenarios discussion. Other points, such as a unified evaluation scale, were acknowledged with reasonable explanations and future directions. With the primary issue resolved and remaining points minor or forward-looking, I maintain a positive recommendation.", "rebuttal": "Thank you for your thoughtful review and encouraging comments. We appreciate your recognition of our contribution and your constructive suggestions for strengthening the paper. Please find our responses to your specific points below:\n\n---\n\n### **1. A more straightforward application of GoF tests**\n\nWe appreciate your suggestion to provide more practical guidance.\n\nIndeed, **no single GoF test consistently dominates across all scenarios**—a fact we highlighted in Table 4. This diversity reflects the **classical trade-offs in hypothesis testing**: some tests are more sensitive to discrepancies in the tails (e.g., Anderson–Darling), while others emphasize central deviations (e.g., Watson’s test). Rather than prescribing a single solution, our goal in this paper is to **empirically reveal these nuanced differences** and encourage the use of GoF tests as a **flexible toolkit** for watermark detection.\n\nThat said, we agree that providing more actionable recommendations is valuable. **We have summarized which GoF tests perform well under specific conditions** (e.g., low vs. high temperature, common vs. targeted edits) in Table 4. We also view **designing adaptive selection strategies**—that choose tests based on input characteristics—as an **important and promising direction** for future work.\n\n---\n\n### **2. Possibility of a unified evaluation scale**\n\nThank you for this thought-provoking suggestion.\n\nWhile our current focus is on evaluating individual GoF tests, we agree that aggregating multiple tests into a unified scale or ensemble could further improve detection. In classical statistics, combining test statistics (e.g., via p-value aggregation or voting rules) is often used to **enhance power while maintaining Type I error control**. Applying these techniques in watermark detection, especially under editing or adversarial perturbations, is an interesting direction that could yield practical benefits.\n\nWe will add a discussion in Section 5 to explicitly raise this as a potential future extension of our work.\n\n---\n\n### **3. More diverse datasets**\n\nWe agree that evaluating on additional datasets is always beneficial and would further validate the generality of our findings.\n\nOur current selection of C4 and ELI5 **follows previous studies \\[1–5]** and provides **complementary text domains**: C4 for factual completions and ELI5 for long-form generation. These allow us to capture a range of generation behaviors and model outputs.\n\n---\n\n### **4. How to improve the green-red watermarking scheme?**\n\nWe appreciate your engagement with this limitation.\n\nAs you noted, the green-red list watermark produces binary pivotal statistics (i.e., whether each token is green), which fundamentally restricts the applicability of many GoF tests. Most GoF tests are designed for continuous or finely discretized data, where empirical CDFs offer rich signal. In contrast, binary statistics yield only counts, reducing the problem to a binomial test.\n\nTo make GoF-style methods applicable in this context, a promising direction is to **design enriched decoders** that produce more informative pivotal statistics beyond the binary green/red indicator. For instance, such decoders could **expose additional features** like token ranks, transition patterns between green and red tokens, or context-aware scores that reflect how strongly a token conforms to the watermarking policy. These enriched statistics would enable a more fine-grained analysis of distributional shifts and allow GoF tests to be meaningfully applied. Developing such decoders could bridge the gap between the widely adopted green-red watermarking scheme and the broader GoF-based detection framework explored in our work. However, this requires additional watermark design, which is beyond the scope of our current study.\n\n---\n\n### **References**\n\n[1] Li, Xiang, et al. \"Robust detection of watermarks for large language models under human edits.\" arXiv preprint arXiv:2411.13868 (2024).\n\n[2] Kuditipudi, Rohith, et al. \"Robust distortion-free watermarks for language models.\" arXiv preprint arXiv:2307.15593 (2023).\n\n[3] Li, Xiang, et al. \"A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules.\" The Annals of Statistics 53.1 (2025): 322-351.\n\n[4] Kirchenbauer, John, et al. \"A watermark for large language models.\" International Conference on Machine Learning. PMLR, 2023.\n\n[5] Dathathri, Sumanth, et al. \"Scalable watermarking for identifying large language model outputs.\" Nature 634.8035 (2024): 818-823."}, {"review_id": "ec8Jsyi8Ey", "reviewer": "Reviewer_aMbH", "summary": "In this paper, the authors propose an empirical overview of classic Goodness of Fit tests -- i.e non-parametric test in a simple, binary hypothesis test setting -- applied to LLM watermarking. The main idea is to replace the usual detection rule of LLM watermarks (classically a sum of on statistics computed from a token, a key and hash computed on previous tokens) with such goodness-of-fit tests and observe if it does lead to improvements compared to the naive detection rules.\nExperimentally, the authors study:\n- 8 GoF tests\n- 3 \"distortion-free\" watermarking schemes\n- 3 LLMs (and 4 temperature settings for each)\nThe study highlights two relevant settings of interest: \n- low-temperature (i.e almost deterministic) generation\n- common text edits -- what I would call robustness evaluation --  and information-rich edits -- what i would call adversarial evaluation.\n\n\nThrough these experiments the authors claim that GoF tests outperform classic methods in robustness even at low temperatures.", "questions": "1. Could the authors improve their evaluation by 1) working at low FPR (at least $10^{-6}$ ) and 2) use either non-asymptotic  $p$-values  **or**  do a precise empirical evaluations of the theoretical FPR vs empirical FPR (see Figure 2 in [1] or Figure 4 [2] for good examples)?\n2. Since the repeated n-gram problem was extensively treated in [1], maybe simply use their hashing function as described in Section III.C ? If using the non-asymptotic p-value, this should guarantee a probability of false alarm for both Aaronson and KGW. This would allow a fair comparison between the author's method and other schemes.\n3. Why were these LLMs chosen? I see two OPT models at different number of parameters (very old models) and the more recent Llama-3.1. I see no presented rationale for this choice (Appendix D is not very enlightening in this regard) and would appreciate one. Even better yet, given that the authors are interested in the impact of the entropy during generation (as controlled by the temperature), why not provide the distribution of the entropy for each model computed for each step and each texts?\n4. *(Minor)* What is the length of texts generated? This has a (very!) substantial impact on detection power and should be controlled accordingly.\n\n\nRegarding low FPR, I am aware it is not achievable for some algorithms (notably Inverse Transform), I believe it is fine to either forego them completely (the absence of a sound p-value/FPR guarantees means a watermarking scheme is essentially useless) or present the results separately for a higher FPR.\n\n**Score increase or decrease** : Given the current state of the paper, I see no reason to decrease the score. On the other hand, I am ready to increase the score provided my gripes around the methodology are addressed.", "limitations": "Nothing to report.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Qualities\n\n- **Clarity and scope**: The paper is overall very well written and clear in what it is trying to achieve -- an empirical evaluation of the GoF tests  in the setting of LLM watermarks. The scope is well respected. Given the informations provided in the paper, I am confident I should be able to reproduce the results.\n\n- **Experimental design**: The experiments are performed on diverse datasets, several LLMs (though I will question the specifics), and several watermarking schemes and detection methods. I am also quite happy to see that the authors make a difference between adversarial and non-adversarial edits, the distinction between robustness and security still being quite blurry in the ML community.\n\n\n## Weakness\n\nBefore listing the weakness, I have to say that they are not purely the fault of the authors. From what I am seeing, the authors have been following what has more or less become the standard in evaluating watermarking algorithms in the ML community ; they cannot be blamed for this. Nevertheless, these standards are flawed when compared to those which were enforced in the classical era of watermarking [4]. Seeing how watermarking is gaining traction in the ML community, I believe it is time to direct papers to more rigorous and stringent evaluations since these designs are meant to eventually be used in the real-world.\n\n- **The $P_{FA}$ is not controlled**: Maybe the most important design aspect of watermarking is the ability to guarantee a certain level for the Type I error. The authors only provide an empirical evaluation of this error on a thousand human texts. This is **not** sufficient. What a good watermarking scheme should strive for is 1) a theoretical model of the detector statistic under $\\mathcal{H}_0$. Usually, the statistic should follow a uniform distribution to allow for a simple computation of the $p$-value. And 2) an empirical validation of the model $\\mathcal{H}_0$, with usually at least a million samples to allow an acceptable confidence interval between $10^{-5}$ and $10^{-6}$ . See [1] or [2] for recent examples of work performing such evaluation correctly (though they do lack a visible confidence interval!).\n-  **$P_{FA}$ is too high**: Even assuming the authors correctly control their Type I error, it is far too high. The authors fix their significance level to $\\alpha=0.01$ and claim it to be a standard choice in previous work (with many references to back it up). From what i can gather, this standard seems to be a bad consequence of the fact that [3] was not able to provide theoretical guarantees for their schemes, leading the authors of [3] to resort to empirical validation. Due to its high computational cost, $\\alpha=0.01$ was the best value in reach. Current state-of-the-art schemes are already able (and should!) to reach excellent power at far lower Type I error -- see the appendix A.X of [2] for proofs of this fact. A usual value in the classical era of watermarking was $\\alpha=10^{-6}$, which given the high number of LLM generated texts per does not even seem that low for practical deployment in the real-world.\n\nUnrelated weaknesses include:\n- **Reiteration of known results**: The whole section of the \"missing factor at low temperatures\" was extensively treated in [1]. Furthermore, the work actually proposed new hashing and detection methodology to completely solve the problem -- keeping guarantees in terms of Type I error at the cost of lower power. \n- **Lack of significance/usefulness**: In its current state, the paper does not provide a model of the $p$-values -- i.e the statistics under $\\mathcal{H}_0$ -- limiting the usefulness of the GoF method in practice since it prevents strong guarantees in terms of Type I errors. This is especially a problem since strong guarantees can be obtained for e.g. KGW and Aaronson's scheme by using non-asymptotic detector [1].\n\n\n[1] Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., & Furon, T. (2023, December). Three bricks to consolidate watermarks for large language models. In 2023 IEEE International Workshop on Information Forensics and Security (WIFS) (pp. 1-6). IEEE.\n\n[2] Giboulot, E., & Furon, T. (2024, December). WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off. In NeurIPS 2024-38th Conference on Neural Information Processing Systems (pp. 1-34).\n\n[3] Kuditipudi, R., Thickstun, J., Hashimoto, T., & Liang, P. Robust Distortion-free Watermarks for Language Models. Transactions on Machine Learning Research.\n\n[4] Bas, P., Furon, T., Cayre, F., Doërr, G., & Mathon, B. (2016). Watermarking security: fundamentals, secure designs and attacks. Berlin: springer.", "was_revised": true, "final_justification": "The authors made their case for the usefulness of their method at low temperature regime. \nEven though I'm still divided on the concrete application of asymptotic methods when non-asymptotic ones are available, it is now more a question of philosophy and methodology: the authors claim that simulation can be sufficient (which is true though quite costly).\nOther questions were answered adequately.\n\nI am not entirely sold on the work due to its asymptotic nature and lack of significance power increase in moderate temperature regime, but given the efforts made by the authors, it would be unfair not to raise my score, hence me bumping it to a borderline accept.", "rebuttal": "Thank you for your professional review and valuable comments. Below are our responses, with clarifications and new evidence addressing your concerns.\n\n---\n\n### **Q1. Concerns about controlling Type I error**\n\nThank you for highlighting the importance of Type I error control.\n\n* **Why we chose $\\alpha = 0.01$:**\n   * First, we used 1,000 human texts for empirical evaluation. Under this sample size, \\$\\alpha = 0.01\\$ is practical—it corresponds to 10 expected false positives, which is a reasonable estimate.\n   * Second, we chose this threshold for **consistency with prior influential work**. Notably, Google’s SynthID watermark \\[2], reportedly ready for deployment in Gemini ([link](https://deepmind.google/discover/blog/watermarking-ai-generated-text-and-video-with-synthid/)), also adopts \\$\\alpha = 0.01\\$.\n\n* **Can GoF tests offer better Type I error control?**\n\n  **Yes**. To further assess this, we sampled 100k texts from the C4 dataset and removed repeated pivotals, following the approach in [1]. Even at this larger scale, GoF tests still maintain strong Type I error control—for example, achieving empirical FPRs as low as $10^{-5}$. The table below shows theoretical FPR vs. empirical FPR (We also created a plot similar to Figure 2 in [1], as suggested by the reviewer, but we cannot include it here due to rebuttal constraints; it will be added to the paper.). \n\n| **Test**  | **Watermarks** | **α = 10⁻²** | **α = 10⁻³** | **α = 10⁻⁴** | **α = 10⁻⁵** |\n| :-------: | :------------: | :----------: | :----------: | :----------: | :----------: |\n| **`Phi`** |     Gumbel     |  9.25×10⁻³   |   5.8×10⁻⁴   |   2.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  7.10×10⁻³   |   4.4×10⁻⁴   |   2.0×10⁻⁴   |   2.0×10⁻⁴   |\n|           |    Synthid     |  7.91×10⁻³   |  2.25×10⁻³   |   1.3×10⁻⁴   |   1.0×10⁻⁵   |\n| **`Kui`** |     Gumbel     |  9.41×10⁻³   |  1.06×10⁻³   |   2.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.49×10⁻²   |  1.73×10⁻³   |   4.2×10⁻⁴   |   2.0×10⁻⁴   |\n|           |    Synthid     |  1.03×10⁻²   |  1.25×10⁻³   |   1.0×10⁻⁴   |   1.0×10⁻⁵   |\n| **`Kol`** |     Gumbel     |  1.03×10⁻²   |   6.0×10⁻⁴   |   1.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.58×10⁻²   |  2.15×10⁻³   |   5.8×10⁻⁴   |   1.0×10⁻⁵   |\n|           |    Synthid     |  9.64×10⁻³   |   9.9×10⁻⁴   |   5.0×10⁻⁵   |   3.0×10⁻⁵   |\n| **`And`** |     Gumbel     |  1.03×10⁻²   |   5.9×10⁻⁴   |   1.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.30×10⁻²   |   5.8×10⁻⁴   |   2.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |    Synthid     |  1.20×10⁻²   |  1.18×10⁻³   |   8.0×10⁻⁵   |   4.0×10⁻⁵   |\n| **`Cra`** |     Gumbel     |  9.47×10⁻³   |  1.25×10⁻³   |   1.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.27×10⁻²   |  1.08×10⁻³   |   2.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |    Synthid     |  1.04×10⁻²   |  1.28×10⁻³   |   8.0×10⁻⁵   |   1.0×10⁻⁵   |\n| **`Wat`** |     Gumbel     |  1.08×10⁻²   |  1.07×10⁻³   |   2.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.49×10⁻²   |  2.13×10⁻³   |   2.0×10⁻⁴   |   2.0×10⁻⁴   |\n|           |    Synthid     |  1.01×10⁻²   |   8.5×10⁻⁴   |   2.0×10⁻⁵   |   1.0×10⁻⁵   |\n| **`Ney`** |     Gumbel     |  1.12×10⁻²   |   8.7×10⁻⁴   |   2.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.53×10⁻²   |  1.07×10⁻³   |   2.0×10⁻⁴   |   2.0×10⁻⁴   |\n|           |    Synthid     |  1.59×10⁻²   |  2.00×10⁻³   |   2.8×10⁻⁴   |   3.0×10⁻⁵   |\n| **`Chi`** |     Gumbel     |  8.18×10⁻³   |   8.6×10⁻⁴   |   1.0×10⁻⁴   |   1.0×10⁻⁵   |\n|           |   Transform    |  1.12×10⁻²   |   5.8×10⁻⁴   |   2.0×10⁻⁴   |   2.0×10⁻⁴   |\n|           |    Synthid     |  1.17×10⁻²   |  1.07×10⁻³   |   1.4×10⁻⁴   |   4.0×10⁻⁵   |\n\n* **Why GoF works in practice**:\n\n   Beyond the watermark’s design, the key factor in maintaining a low Type I error is how closely the pseudo-randomness resembles true randomness—this was also highlighted in [1]. For GoF tests, even though they lack non-asymptotic distributions, they can still offer reliable Type I error control in practice. This is because the pivotal statistics for human-written tokens behave almost like samples from a true uniform distribution after repeated values are removed. The concern about non-asymptotic distributions is further addressed in W2.\n\n---\n\n### **Q2. Reiteration of known results (repeated n-gram problem)**\n\nThanks for pointing out this issue.\n\nWe agree that [1] pointed out and addressed the repeated text issue in watermark detection. However, **our work studies the same issue from a different perspective**. Specifically, rather than focusing on how to deal with repeated text, we analyze **its effects on watermark detection**, such as *why repetition makes sum-based methods less effective*—see ‘The influence of repetition on top probabilities’. We also further explore *how repetition shifts the empirical CDF and impacts GoF detection* in ‘The influence of repetition on the empirical CDF’. More importantly, this analysis highlights a key strength of the GoF method: it’s particularly effective at detecting differences between the empirical CDF and the null $\\mu_0$.\n\n---\n\n### **W2. Lack of significance/usefulness**\n\nWe appreciate your concern about the absence of non-asymptotic guarantees.\n\nWe acknowledge that Aaronson's score has a non-asymptotic distribution, but **a non-asymptotic distribution is NOT necessary to well-control the Type I error**. The most straightforward way to ensure valid Type I error control is to use test statistics with a known non-asymptotic distribution. Most GoF tests in the statistics literature only have known limiting distributions as $n \\to \\infty$ (we will include the limiting distributions of the GoF test statistics we use in the Appendix). Despite this, **the empirical results in the table above show that these tests already provide practically reliable Type I error guarantees.** \n\nIt’s also worth noting that **the limiting distributions offer valuable theoretical tools, but in practice, simulations offer users a more flexible and convenient way** to compute critical values—like in Python package [SciPy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.goodness_of_fit.html), due to the complex forms of some distributions (e.g., the Anderson-Darling distribution [3,4]). \n\nFinally, simulation is a theoretically rigorous approach from both **theoretical and practical perspectives**. Its theoretical validity parallels that of conformal inference: for i.i.d. data points, the empirical $\\alpha$-quantile of the sampled statistics serves as a valid critical value for a new test point due to the property of exchangeability. From a practical standpoint, simulation is also widely adopted in the watermarking literature—for instance, [2] uses a simulation-based method to estimate the distribution of human text (see Appendix A.4).\n\n---\n\n### **Q3. The choice of LLMs and analysis of the entropy**\n\nThanks for raising this question.\n\nMany previous works used OPT models in their experiments[5-8], so we chose OPT to stay consistent for comparison. We also included two versions of OPT to examine whether model size significantly impacts GoF performance. If you think that OPT is relatively outdated, we also tested with the more recent LLaMA 3.1 to see if newer models yield noticeably different results.\n\n  Instead of reporting entropy directly, we analyze **top probabilities**, which **closely track entropy**:\nFor the NTP distribution $\\mathbf{P} = (P_1, ..., P_{|\\mathcal{W}|})$ where $\\mathcal{W}$ denotes the LLM's vocabulary, the entropy is approximated by $\\operatorname{Ent}(\\boldsymbol{P})=\\widetilde{\\Theta}\\left(1-\\max _w P_w\\right)$ up to logarithmic factors. Figures 2, 7, and 8 present these top probabilities for repeated tokens, helping us infer entropy concentration patterns at low temperatures.\n\n---\n\n### **Q4. Text length**\n\nThanks for raising this question.\n\nThe length of the generated texts is fixed at 400 tokens—consistent with prior works [2, 5, 7]—across all models, temperatures, and watermarking methods, unless explicitly stated otherwise.\n\n---\n\n### **References**\n\n[1] Fernandez, Pierre, et al. \"Three bricks to consolidate watermarks for large language models.\" *2023 IEEE international workshop on information forensics and security (WIFS)*. IEEE, 2023.\n\n[2] Dathathri, Sumanth, et al. \"Scalable watermarking for identifying large language model outputs.\" Nature 634.8035 (2024): 818-823.\n\n[3] Anderson, Theodore W., and Donald A. Darling. \"Asymptotic theory of certain\" goodness of fit\" criteria based on stochastic processes.\" *The annals of mathematical statistics* (1952): 193-212.\n\n[4] Marsaglia, George, and John Marsaglia. \"Evaluating the anderson-darling distribution.\" *Journal of statistical software* 9 (2004): 1-5.\n\n[5] Li, Xiang, et al. \"Robust detection of watermarks for large language models under human edits.\" arXiv preprint arXiv:2411.13868 (2024).\n\n[6] Kuditipudi, Rohith, et al. \"Robust distortion-free watermarks for language models.\" arXiv preprint arXiv:2307.15593 (2023).\n\n[7] Li, Xiang, et al. \"A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules.\" The Annals of Statistics 53.1 (2025): 322-351.\n\n[8] Kirchenbauer, John, et al. \"A watermark for large language models.\" International Conference on Machine Learning. PMLR, 2023."}, {"review_id": "Da29AbiPVr", "reviewer": "Reviewer_qCSQ", "summary": "In contrast to most recent work on watermarking for LLMs, this paper explores the application of goodness-of-fit testing to the detection of the signals produced by existing watermarking techniques. They survey a selection of different GoF tests and and implement them for a set of popular watermarking schemes. They empirically evaluate the efficacy of the proposed tests and compare their power in detecting watermarked texts in benign and attacked settings (edits and corruption). They systematize their observed trends in when certain GoF tests outperform the baseline pivotal statistics and present explanations as to why GoF tests might be more robust and suitable to usecases.", "questions": "### Questions\n\n1. Primary request would concern the experimental setup in terms of model and prompt selection. However, it is unlikely the authors can redo significant parts of the results with modern models during the review period. As a stopgap, can it be made ultra clear what models and prompts are used for every experimental result? For some of the tables and figures clear information is provided, but not all.\n\n2. Some experiments present token count as a parameter. Was any analysis done below n=200 or 100 tokens? Most watermarks are expected to perform well at a couple hundred tokens and concerns often center much shorter sequences of  just a few sentences or a short code snippet. Are there any experiments at shorter sequence lengths? This is potentially an extreme but practical area where GoF tests stand to make significant progress (due to the poor performance of existing techniques in that area)\n\n3. Could more details be provided in the paper about the GoF tests for each watermarking method? Table 1 briefly notes the deviation expressions and Appendix A provides nice technical explications of each watermarking method, but these are left unconnected and so there is a gap in how the actual GoF measures are computed for each watermark (I suppose currently a reader must dig through the code to go look at the test implementations and try and reason through how they relate to each watermarking method's raw scores)\n\n\n\n### Additional discussion regarding primary weakness:\n\nSome of the technical points about text repetition are particularly pronounced in old small models like the OPT series. Llama 3 8B is mentioned as being included in the average for Table 2, but it appears that the small OPT model might have been used for Figures 1,2,3. \n\nWhile presented well, the section on \"The influence of repetition on top probabilities\" discusses the tendency towards repetition in next token prediction language models as if this is a novel or understudied phenomenon.\nThe appendix section on related work also supports that interpretation about the authors' perspective: \"Notably, we are the first to analyze the role of text repetition in watermark detection\" (L580)\n\nIt is neither a new phenomenon in the general study of neural language models/LLMs nor in the study of watermarks [1]. Multiple studies discuss and experiment with methods to address the miscalibration of p-values derived from tests over pivotal statistics that rely on i.i.d. assumptions that are violated by repetition within a text [2,3]. Counting of only unique observations is one of the established fixes discussed in many papers and applied in implementations [4].\n\nMuch of the last 3 years of work on LLMs has been done to directly address issues like low entropy and degenerate behaviors like repetition. These factors both heavily interact with watermarks. Thus, this analysis would be much more valuable if performed using generations from modern models like Llama 3.1 or  3.2, Gemma 2 or 3 and Qwen 2.5 or 3, and on prompt datasets like modern long form reasoning benchmarks where models generate CoTs as this is more representative of deployment today. Rambly completions from OPT 1.7B of random C4 webtext pages are not representative of modern usage. That was a setting used in some of the earliest work in the modern wave of watermarking for LLMs but ChatGPT had just been released in 2022 and OPT was one of the only useable open source models available at that time.\n\nIn summary, the presentation of certain parts of the analysis, as well as choices in experimental setup corroborate a lack of understanding about how important the quality of the LLM used and the diversity of the text distribution are in practical takeaways such as \"which GoF test should I deploy in my inference setting?\" \n\n[1] \"The curious case of neural text degeneration\" https://arxiv.org/abs/1904.09751 (general, repetition in weak LLMs)\n\n[2] \"Three Bricks to Consolidate Watermarks for Large Language Models\" https://arxiv.org/abs/2308.00113 (Section III, empirical CDF deviation, repetition in watermark detection)\n\n[3] \"On the reliability of watermarks for large language models\" https://arxiv.org/abs/2306.04634 (Appendix A.3, empirical CDF deviation)\n\n[4] https://huggingface.co/docs/transformers/v4.52.3/en/internal/generation_utils#transformers.WatermarkDetector.ignore_repeated_ngrams (implementation of repetition corrections in standard codebases)", "limitations": "- Primary weakness limits the ability to to apply their findings. Just as a concrete example, one might ask \"Based on the experiments and summary (Table 4), in practice, which GoF test should be used for a reasoning intensive usecases such as the QwQ model performing on coding and mathematical reasoning tasks for a user?\" This paper does not quite provide evidence that is relevant enough such that prescriptions could be readily applied in the real world.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Beyond the primary contemporary citation (Li et al.) their work is one of the first to examine how to improve the detection of existing watermarking methods simply by exploring the space of possible statistical tests one could use without modifying the embedding scheme.\n\n2. They present the GoF tests and their application to selected watermarking schemes in a unified formalism and provide motivation as to why GoF tests should offer advantages over pivotal statistics in certain settings\n\n3. Clearly presented experiments covering both modifications to sampling temperature, text length, and a variety of edit/corruption attacks.\n\n\n### Weaknesses\n\n1. Models and data used in experiments are old. While this is normally a filler criticism in reviews, because of how intertwined watermark embedding and detection performance is with the generated text distribution---a function of both the prompts and the capabilities of the model---this is actually a relatively major weak point of the empirical results. \n2. The work appears to have overlooked certain other studies' prior treatment of statistical soundness issues with existing watermarking schemes and presents some known results as novel. \n\n\n(...and the implementation design choice issues for empirical results actually appear related to perspective issues wrt prior work. See details in Questions section.)", "was_revised": true, "final_justification": "See responses.", "rebuttal": "Thank you for your thoughtful and detailed review. Below, we respond to your comments and questions point by point.\n\n---\n\n### **1. The choice of LLMs**\n\nWe appreciate your concern regarding model selection.\n\nWe selected the OPT models primarily to ensure comparability with prior work \\[1–4], where they have been widely used. To examine the effect of model size, we included both OPT-1.3B and OPT-13B. Acknowledging that the OPT series is relatively outdated, we also included LLaMA 3.1-8B—a significantly stronger, modern model—to evaluate whether newer architectures materially affect watermark detection.\n\nOur findings show that **differences across models have minimal impact on the relative performance of detection methods**. Instead, the **entropy of the next-token probability (NTP) distribution**—which governs how strongly watermark signals appear—plays a much more critical role. Since NTP entropy can be directly manipulated via temperature, we can effectively explore a wide range of detection regimes using temperature scaling, even on older models.\n\nIn this sense, using a newer model that naturally produces higher-entropy outputs is functionally similar to evaluating an older model at a higher temperature. As demonstrated in Appendix Figures 4–6, the **performance trends across GoF tests remain consistent across models**, reinforcing our claim that entropy, not model architecture, is the primary driver of detection difficulty in this setting.\n\n---\n\n### **2. Experiment clarification**\n\nThanks for pointing this out.\n\nIn Appendix C.1, we provided the prompts used for text generation.\n\nAll three models were included in every detection task:\n\n- Type I error on human data and Type II error evaluations (Table 2, 8; Figure 4, 5, 6)\n- Type II error under various editing types (Table 3, 9)\n\nFor ablation studies:\n\n- Results are averaged across the three models for repetition rate (left parts of Figures 2, 7, 8) and for changes in Type II error after removing repeated pivotals (Figure 3).\n- In Figure 1, confidence intervals are based on pivotal sequences from all three models.\n\nFor illustrative purposes:\n\n* In Figure 1, the three curves (light blue, dark blue, and orange) come from OPT-1.3B. This figure is meant to help readers intuitively understand the effect of repetition on the empirical CDF.\n* In Figures 2, 7, and 8 (right parts), the top probability distributions are shown for OPT-1.3B. Although we ran the same experiments on OPT-13B and LLaMA 3.1, the patterns were nearly identical, so we omitted those plots for brevity.\n\nThanks for pointing this out—we'll include this clarification in the revision.\n\n---\n\n### **3. Shorter text sequence**\n\nThis is an excellent point—thank you for highlighting it.\n\n**We did a brief analysis on shorter sequences.** In Figure 1, we present the 95% confidence intervals of the empirical CDF for both long sequences ($n=400$) and short sequences ($n=100$). The figure shows that under high temperature ($T=1$), the empirical CDF diverges noticeably from the null CDF even when $n=100$. Also, Appendix Figures 4, 5, and 6 show how the Type II error decreases as the watermark sequence length increases ($n\\in[1,400]$), with particularly rapid decay at $T=1$ and $T=0.7$. These results suggest that **GoF tests can effectively detect watermark signals with relatively short sequences under high temperature**.\n\nHowever, we didn’t conduct a more detailed discussion on varying sequence lengths in the main text, as our focus is on demonstrating the broader potential of GoF tests as a general framework for improving watermark detection.\n\nThanks again for the thoughtful question—this is a promising direction for future work.\n\n---\n\n### **4. More details about the connection between GoF and watermark**\n\nThanks for pointing out this concern.\n\nWe will certainly include more details in the Appendix. Below is a brief explanation to help avoid any potential misunderstandings.\n\n**• What is '$Y_{(t)}$' used in GoF tests?**\n\n$Y_t$ denotes the pivotal statistic (defined in L28) for the $t$-th token. Given a sequence $Y_1, \\ldots, Y_n$ of pivotal statistics, $Y_{(1)} \\leq Y_{(2)} \\leq \\dots \\leq Y_{(n)}$ are their *order statistics*. For a fixed watermarking scheme, the distribution of $Y_t$ under the null hypothesis (i.e., human-written text) is known and denoted as $\\mu_0$. Note that $\\mu_0$ varies depending on the specific watermarking method—details are provided in Appendix A.\n\n**• Connection to GoF tests**\n\nSince we know that under the null hypothesis the pivotal statistics should always follow the known distribution $\\mu_0$ independently, we can simply test whether a given sequence of $Y_1, \\ldots, Y_n$ follows $\\mu_0$. If not, we reject the null hypothesis and infer that the text is no way human-written and thus LLM-generated. To conduct this hypothesis testing, we use GoF tests—classical statistical tools for evaluating whether a sequence of i.i.d. samples comes from a specified distribution. More details are provided in Section 3: *Rationale for Using GoF Tests*.\n\n**• How are GoF test statistics calculated?**\n\nEach GoF test can be viewed as a function of the pivotal statistics \\$Y\\_1, \\ldots, Y\\_n\\$, designed to measure the *distance* between their empirical distribution and the null distribution \\$\\mu\\_0\\$. For example, consider the Kolmogorov–Smirnov (`Kol`) test applied to the Gumbel-max watermark with a significance level of \\$\\alpha = 10^{-2}\\$:\n\n1. Compute the pivotal statistics: $Y_1, \\ldots, Y_n$\n2. Sort them in ascending order: $Y_{(1)} \\leq \\dots \\leq Y_{(n)}$\n3. Compute the KS test statistic: $D_n = \\max_{1 \\leq i \\leq n} (\\max( Y_{(i)} - \\frac{i - 1}{n}, \\frac{i}{n} - Y_{(i)} ))$\n4. Use Monte Carlo simulation to estimate the critical value $\\gamma$ corresponding to the chosen $\\alpha$\n5. If $D_n > \\gamma$, reject the null hypothesis; otherwise, accept it.\n\nMore details for other tests are provided in Section 3: *Introduction of GoF tests*.\n\n---\n\n### **5. Reiteration of known results (repeated n-gram problem)**\n\nThanks for pointing this out.\n\n**Our work approaches the repetition issue from a different perspective**. \n\nFirst, we acknowledge that repetition in LLM outputs has been identified and analyzed in prior works, as we clearly state in the paper (L264). We'll also add [1] to the citations to better reflect this.\n\nSecond, we appreciate that [2, 3] addressed and effectively handled the repeated text issue in watermark detection. However, rather than focusing on how to deal with repeated text, we analyze *why repetition makes sum-based methods less effective*—see ‘The influence of repetition on top probabilities’. We further explore *how repetition shifts the **empirical CDF of pivotal statistics** (not p-values) and impacts GoF detection*—see ‘The influence of repetition on the empirical CDF’. This analysis brings out a key advantage of GoF: it’s particularly effective at capturing deviations between the empirical CDF of pivotal values and the null $\\mu_0$.\n\nThat said, we agree that the current wording in the paper could cause confusion. We will revise it to clarify that we analyze the role of repetition in watermark detection *from a new perspective—by examining its impact on the statistical behavior of pivotal values and the reliability of GoF-based detection, rather than proposing fixes or pre-processing heuristics*.\n\n---\n\n### **References**\n\n[1] Li, Xiang, et al. \"Robust detection of watermarks for large language models under human edits.\" arXiv preprint arXiv:2411.13868 (2024).\n\n[2] Kuditipudi, Rohith, et al. \"Robust distortion-free watermarks for language models.\" arXiv preprint arXiv:2307.15593 (2023).\n\n[3] Li, Xiang, et al. \"A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules.\" The Annals of Statistics 53.1 (2025): 322-351.\n\n[4] Kirchenbauer, John, et al. \"A watermark for large language models.\" International Conference on Machine Learning. PMLR, 2023."}, {"review_id": "3Ej9qpT4xk", "reviewer": "Reviewer_jT6m", "summary": "The authors explore the effectiveness of goodness-of-fit (GoF) tests for detecting watermarks in text generated by LLMs. The authors evaluate eight distinct GoF tests across three popular watermarking schemes, utilizing various open-source LLMs, datasets, and text generation conditions. They find that GoF tests generally outperform existing detection methods in both accuracy and resilience, particularly in scenarios involving text repetition at lower temperatures or strong watermark signals at higher temperatures. The study highlights that GoF tests, by analyzing the full empirical distribution of pivotal statistics rather than just sum-based measures, offer a robust and underused tool for verifying the origin of LLM-generated content, even when modified.", "questions": "- How expensive are the GoF tests compared to the existing detection methods? Appendix C3 doesn't provide enough details for readers to have an idea of the tradeoff between effectiveness and efficiency.\n- Check this [paper](https://arxiv.org/abs/2405.20777) out. They also make use of statistical tests to detect watermarks. Try to compare their proposed methods especially the robustness to the same attacks you evaluated.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper conducts a highly thorough evaluation, systematically testing eight GoF methods across three popular watermarking schemes (Gumbel-max, Inverse Transform, SynthID), using three open-source LLMs (OPT-1.3B, OPT-13B, Llama 3.1-8B), two datasets (C4, ELI5), four temperature settings, and multiple post-editing methods (deletion, substitution, information-rich edits). This extensive setup provides strong empirical backing for the claims.\n- GoF tests are shown to consistently outperform traditional sum-based baseline methods in both detection power and robustness across varying temperatures and text lengths. The study also demonstrates that GoF tests maintain high detection power even under various text modifications, including common edits (word deletion, synonym substitution) and more adversarial information-rich edits. This is attributed to their focus on overall distributional shifts rather than reliance on extreme values.\n- The paper is logically organized with a clear problem introduction, detailed preliminaries on watermarking and pivotal statistics, a clear explanation of GoF tests, comprehensive experimental sections, and a concise conclusion. The hypothesis testing framework for watermark detection and the role of pivotal statistics are clearly defined and explained, making the core problem accessible.\n- The study is significant and novel. The paper is the first to systematically apply and evaluate a broad class of classical GoF tests for watermark detection, differentiating itself from prior work that focused on specific parameterized GoF tests. Also, the identification and analysis of text repetition as a factor that uniquely benefits GoF test performance in low-temperature settings is a significant original contribution. This provides a new understanding of detection effectiveness in challenging scenarios.\n\n**Weaknesses**\n- While GoF tests generally outperform baselines, the paper notes that \"there is no single method that consistently outperforms others across all scenarios\" within the GoF family . This suggests that practical deployment might require adaptive selection strategies, which is acknowledged as future work but remains a current limitation for straightforward application.\n- In some low-temperature, long-form question-answering scenarios, the LLMs produced highly repetitive or ineffective low-quality outputs (e.g., \"I; 1;1;1;1;1;1;1;1;1;...\") . While the paper discusses potential reasons (LLM capability, question content), these instances might affect the representativeness or generalizability of results in such extreme low-entropy settings, making the evaluation somewhat less robust for these specific scenarios.\n- Although the authors state that the use of multiple models and a large sample size provides a strong basis for statistical significance, the direct inclusion of explicit error bars (e.g., in tables or figures) would provide a more immediate and standard visualization of statistical variability and reinforce the claims .", "was_revised": true, "final_justification": "After considering the rebuttal and discussion, I am increasing my score due to the paper’s strong empirical contribution and its potential impact on future work in watermark detection.\n\n**Resolved issues:**\n\n* The authors clarified why no single GoF test is optimal and reframed their work as a toolkit with summarized use cases, which addresses my concern about practical applicability.\n* The explanation of low-quality outputs at low temperatures (rare and minimally impactful) and the statistical stability of results (large sample Bernoulli trials) resolves concerns about robustness and significance reporting.\n* Additional details on computational trade-offs between tests and clarification of differences from related black-box work (e.g., Gloaguen et al.) addressed scope-related ambiguities.\n\n**Remaining issues:**\n\n* Adaptive selection strategies and detailed runtime comparisons remain future work, limiting immediate deployment guidance.\n* Some concerns about pathological low-temperature outputs could be better addressed with filtering or explicit sensitivity analysis, though this is minor.\n\nThese unresolved points are secondary; the main contribution is significant and stands out in the current literature. The work is methodologically solid and opens clear avenues for future research.\n\nGiven these factors, I now lean toward an accept.", "rebuttal": "Thank you for your professional and constructive review. We appreciate your recognition of the strengths of our paper and your thoughtful suggestions. Below are our responses to your specific points:\n\n---\n\n### **W1. A more straightforward application of GoF tests**\n\nThank you for this valuable observation.\n\nIndeed, **no single GoF test consistently dominates across all scenarios**—a fact we highlighted in Table 4. This diversity reflects the **classical trade-offs in hypothesis testing**: some tests are more sensitive to discrepancies in the tails (e.g., Anderson–Darling), while others emphasize central deviations (e.g., Watson’s test). Rather than prescribing a single solution, our goal in this paper is to **empirically reveal these nuanced differences** and encourage the use of GoF tests as a **flexible toolkit** for watermark detection.\n\nThat said, we agree that providing more actionable recommendations is valuable. **We have summarized which GoF tests perform well under specific conditions** (e.g., low vs. high temperature, common vs. targeted edits) in Table 4. We also view **designing adaptive selection strategies**—that choose tests based on input characteristics—as an **important and promising direction** for future work.\n\n---\n\n### **W2. Low-quality output in low-temperature**\n\nThank you for raising this concern.\n\nAs discussed in Appendix D.2, outputs like “I; 1;1;1;1;1...” are extremely rare, even under low temperatures. These cases reflect pathological behaviors of the model and were kept for completeness. However, they account for a negligible fraction of the dataset and do not significantly influence overall detection performance in low-entropy settings.\n\n---\n\n### **W3. Statistical significance**\n\nWe appreciate your suggestion.\n\nWe understand the value of including error bars to better visualize statistical variability. In our case, each detection outcome is binary (watermarked or not), so the Type II error is computed as the empirical mean over 1,000 i.i.d. Bernoulli trials. This setting allows us to estimate the standard deviation as $\\sqrt{p(1-p)/N}$, where $p$ is the observed error rate and $N$ is the number of samples. Hence, all relevant statistical uncertainty is already captured by this mean, and we chose to report just the Type II error rate for clarity. We can include binomial confidence intervals in future versions if helpful.\n\nWe speculate the reviewer might be referring to **variance across multiple runs** (e.g., rerunning the entire experiment multiple times). However, this would be computationally expensive, and given our large sample size, the estimated error rates are already statistically stable.\n\n---\n\n### **Q1. How expensive are the GoF tests?**\n\nThank you for raising this question. The **most expensive cost** for goodness-of-fit (GoF) tests is *calculating the critical value*.\n\nMost GoF tests—including all those used in this work—**lack non-asymptotic distributions** but do have asymptotic ones. GoF tests with **simpler asymptotic distributions**—specifically `Ney` and `Chi`—can **directly use asymptotic results** to calculate critical values. As a result, their runtime is **comparable to sum-based baselines**.\n\nHowever, several tests (e.g., `Kui`, `Kol`, `And`, `Cra`, `Wat`) **do not use their asymptotic distributions** for critical value computation (even though they have one) due to their complex analytical forms (e.g., the Anderson-Darling distribution [2, 3]). Instead, they rely on **Monte Carlo simulation** with a large number of samples (we use $10^5$) to estimate the critical value, which increases computational cost.\n\nBelow is the time cost for detecting 100 pivotal sequences with $n = 400$ and simulation size $10^5$. For comparison, the `Ars` baseline takes $1.30 \\times 10^{-3}$ seconds. As shown in the table, GoF tests that rely on Monte Carlo simulation (`Kui`, `Kol`, `And`, `Cra`, `Wat`) take roughly $10^3$ times longer to run compared to the baseline. This is mainly due to the $10^3$-fold larger simulation size required for estimating critical values.\n\n| **Test** |  **Time**   | **Time (without calculating critical value)** |\n| :------: | :-------------------: | :-----------------------------------------------------: |\n|  `Phi`   | $2.11 \\times 10^{0}$  |                  $2.20 \\times 10^{-3}$                  |\n|  `Kui`   | $2.03 \\times 10^{0}$  |                  $1.50 \\times 10^{-3}$                  |\n|  `Kol`   | $1.95 \\times 10^{0}$  |                  $1.40 \\times 10^{-3}$                  |\n|  `And`   | $2.61 \\times 10^{0}$  |                  $4.30 \\times 10^{-3}$                   |\n|  `Cra`   | $2.37 \\times 10^{0}$  |                  $1.70 \\times 10^{-3}$                   |\n|  `Wat`   | $2.38 \\times 10^{0}$  |                  $1.80 \\times 10^{-3}$                   |\n|  `Ney`   | $4.00 \\times 10^{-3}$ |                           same as left                            |\n|  `Chi`   | $3.90 \\times 10^{-3}$ |                           same as left                            |\n\nWe will include this table in the Appendix.\n\n---\n\n### **Q2. Comparison with other statistical methods**\n\n**This work addresses a different problem and is not directly comparable to ours.** [1] focuses on detecting the *presence and type* of watermarking schemes in a black-box setting—without access to input parameters or watermark keys. This aligns more with an adversarial goal: identifying whether a model is watermarked and which scheme it uses.\n\nIn contrast, our work assumes the watermark scheme and secret keys are known, and focuses on detecting whether a given text sequence is watermarked—this reflects the use case of LLM APIs like Gemini.\n\nThat said, [1] is a very interesting and valuable contribution to the broader watermarking literature. We will mention it and discuss the difference in our next revision.\n\n---\n\n### **References**\n\n[1] Gloaguen, Thibaud, et al. \"Black-box detection of language model watermarks.\" *arXiv preprint arXiv:2405.20777* (2024).\n\n[2] Anderson, Theodore W., and Donald A. Darling. \"Asymptotic theory of certain\" goodness of fit\" criteria based on stochastic processes.\" *The annals of mathematical statistics* (1952): 193-212.\n\n[3] Marsaglia, George, and John Marsaglia. \"Evaluating the anderson-darling distribution.\" *Journal of statistical software* 9 (2004): 1-5."}]

Author rebuttal:

Meta-review: This paper presents the application of eight classical goodness-of-fit (GoF) tests to improve detection of watermarks in LLM-generated text. Rather than relying on the usual sum-based detection rules, the paper employes GoF tests in a binary hypothesis framework; comparing the empirical distribution of pivotal statistics to theoretical expectations. Evaluated across three watermarking schemes, various open-source LLMs, and datasets, GoF tests generally outperform standard approaches, especially in cases of low-temperature generations with repetitive text or under adversarial conditions. By exploiting distributional patterns rather than aggregate scores, GoF methods offer improved robustness and detection power, even when watermark integrity is compromised.

Overall, using a goodness-of-fit test for LLM watermark detection is an interesting idea. All reviewers agree to accept the paper, and I also vote for acceptance.
Decision: Accept (spotlight)