Efficiently Identifying Watermarked Segments in Mixed-Source Texts
Authors: Xuandong Zhao, Chenwen Liao, Yu-Xiang Wang, Lei Li
Accepted at: acl25

Efficiently Identifying Watermarked Segments in Mixed-Source Texts

Xuandong Zhao*
UC Berkeley
xuandongzhao@berkeley.edu

Chenwen Liao*
Zhejiang University
liaochenwen@zju.edu.cn

Yu-Xiang Wang†
UC San Diego
yuxiangw@ucsd.edu

Lei Li‡
Carnegie Mellon University
leili@cs.cmu.edu

Abstract
Text watermarks in large language models (LLMs) are increasingly used to detect synthetic text, mitigating misuse cases like fake news and academic dishonesty. While existing watermarking detection techniques primarily focus on classifying entire documents as watermarked or not, they often neglect the common scenario of identifying individual watermark segments within longer, mixed-source documents. Drawing inspiration from plagiarism detection systems, we propose two novel methods for partial watermark detection. First, we develop a geometry cover detection framework aimed at determining whether there is a watermark segment in long text. Second, we introduce an adaptive online learning algorithm to pinpoint the precise location of watermark segments within the text. Evaluated on three popular watermarking techniques (KGW-Watermark, Unigram-Watermark, and Gumbel-Watermark), our approach achieves high accuracy, significantly outperforming baseline methods. Moreover, our framework is adaptable to other watermarking techniques, offering new insights for precise watermark detection. Our code is publicly available at https://github.com/XuandongZhao/llm-watermark-location.

1 Introduction
Large Language Models (LLMs) have revolutionized human activities, enabling applications ranging from chatbots (OpenAI, 2022) to medical diagnostics (Google, 2024) and robotics (Ahn et al., 2024). Their ease of use, however, presents serious societal challenges. In education (Intelligent, 2024), students can effortlessly generate essays and homework answers, undermining academic integrity. In journalism (Blum, 2024), distinguishing credible news from fabricated content erodes public trust. The potential for malicious uses, such as phishing (Violino, 2023), and the risk of model collapse due to synthetic data (Shumailov et al., 2024), further underscore the urgent need to detect LLM-generated text and promote the responsible use of this powerful technology.

However, identifying AI-generated text is becoming increasingly difficult as LLMs reach human-like proficiency in various tasks. One line of research (OpenAI, 2023; Tian, 2023; Mitchell et al., 2023) trains machine learning models as AI detectors by collecting datasets consisting of both human and LLM-generated texts. Unfortunately, these approaches are often fragile (Shi et al., 2024) and error-prone (Liang et al., 2023), ultimately leading OpenAI to terminate its deployed detector (Kelly, 2023). Watermarking has emerged as a promising solution to this challenge1. By embedding identifiable patterns within the generated text, watermarks can signal whether a piece of text originates from an LLM (Dathathri et al., 2024).

Existing watermark detection methods (Aaronson, 2023; Kirchenbauer et al., 2023; Zhao et al., 2023; Kuditipudi et al., 2023; Christ et al., 2023; Hu et al., 2024; Dathathri et al., 2024) are primarily designed for text-level classification, labeling a piece of text as either watermarked or not. However, these methods are insufficient for many real-world scenarios where documents contain mixed-source texts, and only specific sections are LLM-generated. For instance, malicious actors might use LLMs to manipulate certain sections of a news article to spread misinformation. Detecting watermarks within long, mixed-source texts presents a significant challenge, especially when aiming for subsequence-level detection with uncertainty quantification, similar to plagiarism detection systems like “Turnitin”2. This is because the watermarked signal may be weakened throughout the increasing

1Watermark has been used in industry, e.g. SynthID (Dathathri et al., 2024) for Google Gemini.
2https://www.turnitin.com
text length and may not be easily identifiable using conventional detection methods.

To bridge the gap, we propose partial watermark detection methods that offer a reliable solution for identifying watermark segments in long texts. A straightforward approach, which involves examining all possible segments of a text containing \( n \) tokens, yields an inefficiently high time complexity of \( O(n^2) \). Instead, we employ the *geometric cover trick* (Daniely et al., 2015) to partition the long texts into subsequences of varying lengths and then perform watermark detection within each interval. This approach, termed the *Geometric Cover Detector* (GCD), enables efficient classification of whether a document contains any watermarked text in \( O(n \log n) \) time. However, GCD does not assign a score to every token, providing only a rough localization of watermark segments.

To refine this localization, we introduce the *Adaptive Online Locator* (AOL). AOL reformulate the problem as an online denoising task, where each token score from the watermark detector serves as a noisy observation for the mean value of scores within watermark segments. By applying an adaptive online learning method, specifically the *Alligator* algorithm (Baby et al., 2021), we retain the \( O(n \log n) \) time complexity while significantly improving the accuracy of detected segments.

We validate GCD and AOL using the C4 (Raffel et al., 2020) and Arxiv (Cohan et al., 2018) datasets, employing Llama (Touvron et al., 2023) and Mistral (Jiang et al., 2023) models for evaluation. Our empirical results demonstrate strong performance across both classification and localization tasks. In the classification task, our method consistently achieves a higher true positive rate compared to the baseline at the same false positive rate. For localization, we achieve an average intersection over union (IoU) score of over 0.55, far exceeding baseline methods.

In summary, our contributions are threefold:
1. We introduce novel approaches to watermark detection, moving beyond simple text-level classification to identification of watermark segments within long, mixed-source texts.
2. We employ the *geometric cover* trick and the *Alligator* algorithm from online learning to reliably detect and localize watermark segments efficiently and accurately.
3. We conduct extensive experiments on state-of-the-art public LLMs and diverse datasets. Our empirical results show that our approach significantly outperforms baseline methods.

2 Background and Related Work

2.1 LLM Watermark and Detection

Language Models and Watermarking. A language model \( \mathcal{M} \) is a statistical model that generates natural language text based on a preceding context. Given an input sequence \( x \) (prompt) and previous output \( y_{<t} = (y_1, \ldots, y_{t-1}) \), an autoregressive language model computes the probability distribution \( P_{\mathcal{M}}(\cdot|x, y_{<t}) \) of the next token \( y_t \) in the vocabulary \( \mathcal{V} \). The full response is generated by iteratively sampling \( y_t \) from this distribution until a maximum length is reached or an end-token is generated. *Decoding-based watermarking* (Aaronson, 2023; Kirchenbauer et al., 2023; Zhao et al., 2023; Kuditipudi et al., 2023; Christ et al., 2023; Hu et al., 2024) modifies this text generation process by using a secret key k to transform the original next-token distribution \( P_{\mathcal{M}}(\cdot|x, y_{<t}) \) into a new distribution. This new distribution is used to generate watermarked text containing an embedded watermark signal. The watermark detection algorithm then identifies this signal within a suspect text using the same watermark key k.

Red-Green Watermark. Red-Green (statistical) watermarking methods partition the vocabulary into two sets, “green” and “red”, using a pseudo-random function \( R(h, k, \gamma) \). This function takes as input the length of the preceding token sequence (\( h \)), a secret watermark key (\( k \)), and the target proportion of green tokens (\( \gamma \)). During text generation, the logits of green tokens are subtly increased by a small value \( \delta \), resulting in a higher proportion of green tokens in the watermarked text compared to non-watermarked text. Two prominent Red-Green watermarking methods are KGW-Watermark (Kirchenbauer et al., 2023, 2024) and Unigram-Watermark (Zhao et al., 2023). KGW-Watermark utilizes \( h \geq 1 \), considering the prefix for hashing. Unigram-Watermark employs fixed green and red lists, disregarding previous tokens by setting \( h = 0 \) to enhance robustness. Watermark detection in both methods involves identifying each token’s membership in the green or red list

\[
\text{Score}(y) = \sum_{t=1}^{n} 1(y_t \in \text{Green Tokens})
\]
and calculating the z-score of the entire sequence: \( z_y = \frac{\text{Score}(y) - \gamma n}{\sqrt{n \gamma (1-\gamma)}} \). This z-score reflects the deviation of the observed proportion of green tokens from the expected proportion \( \gamma n \), where \( n \) is the total number of tokens in the sequence. A significantly high z-score yields a small p-value, indicating the presence of the watermark.

Gumbel Watermark. The watermarking techniques proposed by Aaronson (2023) and Kuditipudi et al. (2023) can be described using a sampling algorithm based on the Gumbel trick (Zhao et al., 2024). This algorithm hashes the preceding \( h \) tokens using the key \( k \) to obtain a score \( r_i \) for each token \( i \) in the vocabulary \( \mathcal{V} \), where each \( r_i \) is uniformly distributed in \([0, 1]\). The next token is chosen deterministically as follows: \( \arg\max_{y_i \in \mathcal{V}} [\log P(y_i|x_{<t}) - \log(-\log(r_{y_i}))] \). Thus, given a random vector \( r \sim (\text{Uniform}([0, 1])^{|\mathcal{V}|}, -\log(-\log(r_{y_i})) \) follows a Gumbel(0,1) distribution. This results in a distortion-free deterministic sampling algorithm (for large \( h \)) for generating text. During detection, if the observed score

\[
\text{Score}(y) = \sum_{t=1}^n \log \left( 1/(1 - r_{y_t}) \right)
\]

is high, the p-value is low, indicating the presence of the watermark.

2.2 Text Attribution and Plagiarism Detection
Watermark text localization shares similarities with text attribution and plagiarism detection, particularly in the aspect of pinpointing specific text segments. Commercial plagiarism detection systems like Turnitin, Chegg, and Grammarly rely on vast databases to identify copied content, highlighting similar segments. Research in plagiarism localization, such as the work by Grozea et al. (2009), focuses on precisely identifying copied passages within documents. Their approach utilizes a similarity matrix and sequence-matching techniques for accurate localization. Similarly, the “Greedy String Tiling” algorithm (Wise, 1996) has been successfully employed by Mozgovoy et al. (2010) for identifying overlapping text. However, these methods require reference files in a database, whereas watermark text localization aims to localize the watermark text using a watermark key, eliminating the need for reference documents. Detecting partially watermarked text presents a unique challenge, akin to an online learning problem, where tokens in watermark segments exhibit special signals that can be captured by a strongly adaptive online learning algorithm like Aligator (Baby et al., 2021).

2.3 Identifying Watermarked Portions in Long Text
To detect watermarked portions in long texts, Aaronson (2023) designs a “watermark plausibility score” for each interval. Given \( \{s_t = \log(1/(1 - r_{y_t}))\}_{t \in [n]} \), the watermark plausibility score is \( \frac{(\sum_{j=i}^t s_j)^2}{j-i} - L \), where \( L \) is a constant. This method draws connections to change point detection algorithms, aiming to maximize the sum of plausibility scores to detect watermarked portions. Aaronson (2023) manages to reduce the time complexity from \( \mathcal{O}(n^2) \) to \( \mathcal{O}(n^{3/2}) \). Additionally, Christ et al. (2023) demonstrate how to detect a watermarked contiguous substring of the output with sufficiently high entropy, calling the algorithm Substring Completeness. This algorithm has a time complexity of \( \mathcal{O}(n^2) \). A recent, independent work of Kirchenbauer et al. (2024) introduces the WinMax algorithm to detect watermarked sub-regions in long texts. This algorithm searches for the continuous span of tokens that produces the highest z-score by iterating over all possible window sizes and traversing the entire text for each size, with a time complexity of \( \tilde{\mathcal{O}}(n^2) \). Our Adaptive Online Locator (AOL) improves the efficiency of detecting watermarked portions, reducing the time complexity to \( \mathcal{O}(n \log n) \).

3 Method
Identifying watermark segments within a long text sequence \( y \) presents two key challenges. First, we need to design a classification rule \( \mathcal{M}(y) \to \{0, 1\} \) that determines whether \( y \) contains a watermark segment. To address this, we propose the Geometric Cover Detector (GCD), which enables multi-scale watermark detection. Second, accurately locating the watermark segments \( y_{s_i:e_i} \) within the full sequence \( y \) requires finding the start and end token indices, \( s_i \) and \( e_i \), for each watermark segment. We introduce the Adaptive Online Locator (AOL) with the Aligator algorithm to precisely identify the position of the watermarked text span within the longer sequence.
Input Sequence
Artificial intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and make decisions. The development of AI and autonomy may soon massively amplify AI’s impact, with the potential for both enormous benefits and malicious uses, and an irreversible loss of human control over autonomous AI systems. Despite catastrophic risks and the increasing severity of the severe dangers posed by AI, there is no agreement on how to effectively mitigate these risks. Although international efforts are encouraging, humanity’s reaction falls short of what is needed, given the potential for swift and profound consequences that we cannot anticipate. Research into AI safety is not keeping pace with the rapid development of AI technology, lacking the necessary frameworks and organizations to deter irresponsible use and hardly even considering the possibility of catastrophic failure. Drawing on lessons learned from other safety-critical technologies, we propose a comprehensive research plan that combines technical research and development (R&D) with proactive, adaptive governance mechanisms for a more commensurate preparation.

Geometric Covering Intervals

Output Detection
Artificial intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and make decisions. The development of AI and autonomy may soon massively amplify AI’s impact, with the potential for both enormous benefits and malicious uses, and an irreversible loss of human control over autonomous AI systems. Despite catastrophic risks and the increasing severity of the severe dangers posed by AI, there is no agreement on how to effectively mitigate these risks. Although international efforts are encouraging, humanity’s reaction falls short of what is needed, given the potential for swift and profound consequences that we cannot anticipate. Research into AI safety is not keeping pace with the rapid development of AI technology, lacking the necessary frameworks and organizations to deter irresponsible use and hardly even considering the possibility of catastrophic failure. Drawing on lessons learned from other safety-critical technologies, we propose a comprehensive research plan that combines technical research and development (R&D) with proactive, adaptive governance mechanisms for a more commensurate preparation.

Online Learning Prediction
Watermarked Segments

Figure 1: Illustration of the watermark segment detection process. The input sequence could be a mixed-source of watermarked text and unwatermarked text. We use geometric covers to partition the text and detect watermarks in intervals. We also formulate localization as an online denoising problem to reduce computational complexity. The example shown is drawn from the abstract of Bengio et al. (2024), with the watermarked part generated by a watermarked Mistral-7B model.

![Illustration of the watermark segment detection process](page_186_153_1077_393.png)

Figure 2: Example of precise watermark localization using AOL with Gumbel Watermark. Light green lines show token scores, and dark green lines show predicted mean scores. The horizontal dashed line shows the score threshold \( \zeta = 1.3 \). The vertical dashed line marks the original watermark position. The top image demonstrates inaccurate localization from a single pass of the Aligator algorithm, highlighting boundary artifacts. In contrast, the bottom image shows precise localization achieved by AOL’s circular initialization strategy with \( m = 10 \) random starts.

![Example of precise watermark localization using AOL with Gumbel Watermark](page_186_624_1077_246.png)

3.1 Watermark Segment Classification
A straightforward approach to detect whether an article contains watermarked text is to pass it through the original watermark detector (as we discussed in Section 2.1). If the detection score from the original detector is larger than a threshold, the text contains a watermark; otherwise, no watermark is found. However, this approach is ineffective for long, mixed-source texts where only a small portion originates from the watermarked LLM. Since a large portion of the text lacks the watermark signal, the overall score for the entire document will be dominated by the unwatermarked portion, rendering the detection unreliable.

To overcome this limitation, we need a method that analyzes the text at different scales or chunks. If a chunk is flagged as watermarked, we can then classify the entire sequence as containing watermarked text. The question then becomes: how do we design these intervals or chunks effectively? We leverage the Geometric Cover (GC) technique introduced by Daniely et al. (2015) to construct an efficient collection of intervals for analysis.

Geometric Cover (GC) is a collection of intervals belonging to the set \( \mathbb{N} \), defined as follows:

\[
\mathcal{I} = \bigcup_{k \in \mathbb{N} \cup 0} \mathcal{I}^{(k)}, \text{ where } \forall k \in \mathbb{N} \cup 0, \text{ and}
\]
\[
\mathcal{I}^{(k)} = \{[i \cdot 2^k, (i+1) \cdot 2^k - 1] : i \in \mathbb{N}\}.
\]

Essentially, each \( \mathcal{I}^{(k)} \) represents a partition of \( \mathbb{N} \) into consecutive intervals of length \( 2^k \). For example, \( \mathcal{I}^{(4)} \) contains all consecutive 16-token intervals. Due to this structure, each token belongs to \( \lfloor \log n \rfloor + 1 \) different intervals (as il-
Algorithm 1 Geometry Cover WM Detection
Input: Text y of length n, target false positive rate (FPR) \( \tau \), watermark detector score function Score, FPR calibration function \( F \)
1: Divide y into Geometry Cover set \( \mathcal{I} \) as defined in Equation 3
2: for each interval \( \mathcal{I}_i : (i_t, j_t) \) in Geometry Cover set do
3: Compute FPR \( \alpha \leftarrow F(y_{i_t:j_t}, \text{Score}(y_{i_t:j_t})) \)
4: if \( \alpha < \tau \) then
5: return 1, i.e., “The sequence contains a watermark”
6: return 0, i.e., “No watermark found”

Algorithm 2 Watermark Position Localization
Input: Text y, threshold \( \zeta \), iterations m
1: Get watermark detection scores of each token for y from watermark detector \( \{s_t\}_{t \in [n]} \)
2: Initialize Aligator algorithm \( \mathcal{A} \) with circular starting strategy
3: for \( i = 1 \) to m do
4: Random starting position \( k \leftarrow \) random index in \( \{1, \ldots, n\} \)
5: Predict the pointwise estimate of the expected detection score for each token in the i-th round:
\( \theta^{(i)} := \{\theta_t\}_{t \in [n]} \leftarrow \mathcal{A}(s_k, s_{k+1}, \ldots, s_n, s_1, \ldots, s_{k-1}) \)
6: end for
7: Average predicted scores across all rounds
\( \overline{\theta} \leftarrow \frac{1}{m} \sum_{i=1}^m \theta^{(i)} \)
8: Identify watermarked positions \( \mathcal{W} \leftarrow \{ t \mid \overline{\theta}_t > \zeta \} \)
9: return \( \mathcal{W} \)

lustrated in Figure 1), and there are a total of \( n + n/2 + n/4 + n/8 + \cdots = O(n) \) intervals in the GC set. This allows us to establish a multi-scale watermark detection framework. Moreover, Lemma 5 from Daniely et al. (2015) ensures that for any unknown watermarked interval, there is a corresponding interval in the geometric cover that is fully contained within it and is at least one-fourth its length. This ensures the effectiveness of watermark detection using the geometric cover framework.

Leveraging the GC construction, our multi-scale watermark detection framework divides the input text into segments based on the GC intervals. In real-world applications, we need to balance the granularity of the intervals. For instance, classifying a 4-token chunk as watermarked might not be convincing. Therefore, we start from higher-order intervals, such as \( \mathcal{I}^{(5)} \), which comprises all geometric cover intervals longer than 32 tokens.

Algorithm 1 outlines our approach. For each segment \( \mathcal{I}_i : y_{i_t:j_t} \) in the GC, we first compute a detection score using the appropriate watermark detector for the scheme employed (e.g., Equation 1 for Red-Green Watermark or Equation 2 for Gumbel Watermark). This score, along with the segment itself, is then passed to an FPR calibration function \( F \). This function estimates the FPR associated with the segment. Further details on FPR calibration can be found in the Appendix B.

If the estimated FPR, denoted as \( \alpha \), falls below a predefined target FPR (\( \tau \)), we classify the entire sequence as containing a watermark. It is important to note that \( \tau \) is set at the segment level. Using the union bound, consider a mixed-source text composed of \( n \) tokens. The geometric cover of the text is constructed from \( O(n) \) intervals. Let \( \tau \) represent the false positive rate for each interval test (Type I error rate). In this case, the Family-Wise Error Rate (FWER), which is the probability of incorrectly classifying the entire document as watermarked, is bounded by \( n \tau \).

3.2 Precise Watermark Position Localization

While the previous section focused on detecting the presence of watermarks, simply knowing a watermark exists doesn’t reveal which specific paragraphs warrant scrutiny. Here, we aim to localize the exact location of watermarked text.

A naive approach would involve iterating through all possible interval combinations within the sequence, applying the watermark detection rule to each segment \( y_{i:j} \) for all \( i \in \{1, \ldots, n\} \) and \( j \in \{i, \ldots, n\} \). While this brute-force method can identify watermark segments, its \( O(n^2) \) time complexity makes it computationally expensive for long sequences.

Furthermore, relying solely on individual token scores for localization is unreliable due to the inherent noise in the watermarking process. To address this issue, we propose to formulate it as a sequence denoising problem (a.k.a., smoothing or nonparametric regression) so we can provide a pointwise estimate of the expected detection score for each token. Specifically, the denoising algorithm takes a sequence of noisy observations \( s_1, ..., s_n \) and outputs \( \{\theta_t\}_{t \in [n]} \) as an estimate to \( \{\mathbb{E}[s_t]\}_{t \in [n]} \).

As an example, for the Green-Red Watermark,
the sequence of noisy observations

\[
\{s_t = \mathbf{1}(y_t \in \text{Green Tokens})\}_{t \in [n]}
\]

consists of Bernoulli random variables. The expectation \( \mathbb{E}[s_t] = \gamma \) if \( y_t \) is not watermarked and \( \mathbb{E}[s_t] > \gamma \) otherwise. For the Gumbel Watermark, the noisy observations

\[
\{s_t = \log(1/(1 - r_{y_t}))\}_{t \in [n]}
\]

consists of exponential random variables satisfying \( \mathbb{E}[s_t] = 1 \) if \( y_t \) is unwatermarked and larger otherwise. The intuition is that, while individually they are too noisy, if we average them appropriately within a local neighborhood, we can substantially reduce the noise. If we can accurately estimate the sequence \( \mathbb{E}[s_t] \), we can localize watermarked segments by simply thresholding the estimated score pointwise.

The challenge, again, is that we do not know the appropriate window size to use. In fact, the appropriate size of the window should be larger if \( s_t \) is in the interior of a long segment of either watermarked or unwatermarked text. The sharp toggles among text from different sources add additional challenges to most smoothing algorithms.

For these reasons, we employ the Aligator (Aggregation of onLIne averaGes using A geomeTric cOveR) algorithm (Baby et al., 2021). In short, Aligator is an online smoothing algorithm that optimally competes with an oracle that knows the segments of watermarked sequences ahead of time. The algorithm employs a Geometric Cover approach internally, where words positioned mid-paragraph are typically included in multiple intervals of varying lengths for updates. Notably, Aligator provides the following estimation guarantee:

\[
\frac{1}{n} \sum_t (\theta_t - \mathbb{E}[s_t])^2 = \tilde{O}\left( \min \left\{ n^{-1}(1 + \sum_{t=2}^n \mathbf{1}_{\mathbb{E}[s_t] \neq \mathbb{E}[s_{t-1}]}) , n^{-1} \lor n^{-2/3} (\sum_{t=2}^n |\mathbb{E}[s_t] - \mathbb{E}[s_{t-1}]| ) \right\} \right).
\]

Moreover, for all segments with start/end indices \((i, j) \in [n]^2\), i.e.

\[
\frac{1}{j-i} \sum_{t=i}^j (\theta_t - \frac{1}{j-i} \sum_{u=i}^j \mathbb{E}[s_u])^2 \leq \tilde{O}(1/\sqrt{j-i}).
\]

This ensures that for every segment, the estimated value is as accurate as statistically permitted. The time complexity for Aligator is \( \mathcal{O}(n \log n) \). For a detailed implementation of Aligator, please refer to the original paper (Baby et al., 2021). For the theoretical results, see (Baby and Wang, 2021).

Circular Aligator. To mitigate the boundary effects common in online learning, where prediction accuracy suffers at the beginning and end of sequences, we introduce a circular starting strategy. Instead of processing the text linearly, we treat it as a circular buffer. For each iteration, we randomly choose a starting point and traverse the entire sequence, effectively mitigating edge effects. The final prediction for each token is then obtained by averaging the predictions across all iterations.

Finally, we apply a threshold to this denoised average score function to delineate the boundaries of watermark segments within the text (as illustrated in Figure 1). The high-level implementation of this method is detailed in Algorithm 2. This approach enables us to precisely identify the location of suspected plagiarism within large documents with high confidence, facilitating further investigation and verification.

4 Experiment

4.1 Experiment Setting

We conduct experiments on three state-of-the-art watermarking methods: Gumbel-Watermark (Aaronson, 2023), KGW-Watermark (Kirchenbauer et al., 2023), and Unigram-Watermark (Zhao et al., 2023) (with \( \gamma = 0.5, \delta = 2.0 \), and temperature = 1.0). We evaluate our methods on mixed-source texts constructed from the C4 (news-like subset) (Raffel et al., 2020) and Arxiv datasets (Cohan et al., 2018). These texts contain both human-written and LLM-generated (watermarked) segments. Watermarked segments are generated using LLaMA-7B (Touvron et al., 2023) and Mistral (Jiang et al., 2023), and are embedded within longer contexts (10% watermarked ratio to simulate realistic watermark integration), with their positions randomized. For segment detection, we compare our method (GCD) against each watermarking method’s original detector (VANILLA). For localization, we compare AOL with RoBERTa-based classifiers, which are trained to label text segments as watermarked or not based on token watermark scores. We also included WinMax (Kirchenbauer et al., 2024) with window sizes of 1 and 100 (WinMax-1, WinMax-100) as a brute-force baseline (\( \tilde{O}(n^2) \)) for segment detection and localization. We evaluate using True Positive Rate
<table>
  <tr>
    <th>Method</th>
    <th colspan="2">KGW-Watermark TPR</th>
    <th colspan="2">Unigram-Watermark TPR</th>
    <th colspan="2">Gumbel-Watermark TPR</th>
  </tr>
  <tr>
    <th></th>
    <th>SEG-FPR</th>
    <th>Doc-FPR</th>
    <th>SEG-FPR</th>
    <th>Doc-FPR</th>
    <th>SEG-FPR</th>
    <th>Doc-FPR</th>
  </tr>
  <tr>
    <td><i>C4 Dataset, Llama-7B</i></td>
    <td>1e-5 5e-5 1e-4 2e-4</td>
    <td>0.001 0.001 0.010</td>
    <td>1e-4 2e-4</td>
    <td>0.001 0.001 0.010</td>
    <td>1e-4 5e-4</td>
    <td>0.001 0.001 0.010</td>
  </tr>
  <tr>
    <td>VANILLA</td>
    <td>0.602 0.676 0.692 0.706</td>
    <td>0.002 0.004 0.030</td>
    <td>0.006 0.006</td>
    <td>0.058 0.650 0.762</td>
    <td>0.918</td>
    <td></td>
  </tr>
  <tr>
    <td>GCD</td>
    <td>0.912 0.934 0.934 0.874</td>
    <td>0.906 0.958</td>
    <td>1.000 1.000 1.000</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>C4 Dataset, Mistral-7B</i></td>
    <td>1e-5 1e-4 2e-4</td>
    <td>0.001 0.010 0.020</td>
    <td>1e-4 5e-4</td>
    <td>0.001 0.010 0.020</td>
    <td>1e-4 5e-4</td>
    <td>0.001 0.001 0.001</td>
  </tr>
  <tr>
    <td>VANILLA</td>
    <td>0.697 0.830 0.877</td>
    <td>0.001 0.012 0.040</td>
    <td>0.024 0.046 0.054</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GCD</td>
    <td>0.960 0.983 0.990</td>
    <td>0.722 0.974</td>
    <td>1.000 0.970 0.980</td>
    <td>0.990</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>Arxiv Dataset, Llama-7B</i></td>
    <td>1e-5 5e-5 2e-4</td>
    <td>0.001 0.014 0.024</td>
    <td>1e-4 2e-4</td>
    <td>0.001 0.014 0.024</td>
    <td>1e-4 2e-4</td>
    <td>0.001 0.001 0.010</td>
  </tr>
  <tr>
    <td>VANILLA</td>
    <td>0.844 0.896 0.908</td>
    <td>0.000 0.026 0.593</td>
    <td>0.655 0.823</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GCD</td>
    <td>0.990 0.994 0.996</td>
    <td>0.892 0.922</td>
    <td>0.974 0.958 0.978</td>
    <td>1.000</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>Arxiv Dataset, Mistral-7B</i></td>
    <td>1e-5 1e-4 2e-4</td>
    <td>0.001 0.020 0.020</td>
    <td>1e-5 1e-4 2e-4</td>
    <td>0.001 0.020 0.020</td>
    <td>1e-5 1e-4 2e-4</td>
    <td>0.001 0.001 0.010</td>
  </tr>
  <tr>
    <td>VANILLA</td>
    <td>0.757 0.883 0.907</td>
    <td>0.002 0.036 0.082</td>
    <td>0.192 0.230</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GCD</td>
    <td>0.967 0.990 1.000</td>
    <td>0.566 0.920</td>
    <td>0.964 0.950 0.960</td>
    <td>0.970</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: True Positive Rate (TPR) at various False Positive Rate (FPR) levels for baseline VANILLA and our method GCD. For each setting, we select three distinct segment-level FPRs (SEG-FPR) and compare the performance of VANILLA and GCD at equivalent document-level FPRs (Doc-FPR). GCD consistently outperforms VANILLA across different models and datasets.

(TPR) at calibrated False Positive Rates (FPRs, both per-segment and document-level), Intersection over Union (IoU) for localization accuracy. We also benchmark detection speed, reporting the average runtime per sample. Further details, including dataset specifics, baseline training, and FPR calibration, are in the Appendix A.

4.2 Detection Results

Watermark Segment Classification Results. As shown in Table 1, our proposed Geometric Cover Detector (GCD) consistently outperforms the baseline VANILLA method across all watermarking techniques and large language models on both the C4 and Arxiv datasets. The robustness of GCD across diverse conditions underscores its effectiveness in watermark segment classification, demonstrating clear superiority over VANILLA. Additionally, we observe that VANILLA exhibits near-zero detection rates when the target false positive rate is low. This suggests that VANILLA struggles to detect watermarked segments in longer contexts, as the watermark signal weakens, rendering the simpler detector ineffective.

Precise Watermark Position Localization Results. For the watermark position localization task, we evaluate our proposed method AOL against the baseline method RoBERTa (Table 2). We calculate the average IoU score to quantify the precision of the watermark localization. Our method consistently outperforms the baseline across all test settings. For example, on the C4 dataset using the Mistral-7B model, AOL achieves a substantially higher IoU score of 0.809 compared to 0.301 for RoBERTa. We also test AOL’s ability to detect multiple watermarks by inserting 3x300-token Gumbel watermarks (generated by Mistral-7B) into 6000-token texts. Across 200 samples, the average IoU for detecting the watermarks is 0.802, demonstrating AOL’s effectiveness for multiple watermark detection. Figure 2 provides a case example illustrating the improved localization performance of AOL on the Gumbel watermark with the Mistral-7B model. The upper image shows the boundary effects of using online learning. The lower image demonstrates more precise localization resulting from the circular starting strategy with 10 random starting points. We explored other values for m, and our experiments indicate that \( m = 10 \) provides a robust balance between accuracy and computational efficiency for our tested datasets and models.

4.3 Detection Efficiency

Efficiently identifying watermarked segments is a critical goal of our approach. To assess the ef-
<table>
  <tr>
    <th>Method</th>
    <th>KGW-WM IoU</th>
    <th>Unigram-WM IoU</th>
    <th>Gumbel-WM IoU</th>
  </tr>
  <tr>
    <td colspan="4"><i>C4 Dataset, Llama-7B</i></td>
  </tr>
  <tr>
    <td>RoBERTA</td>
    <td>0.563</td>
    <td>0.444</td>
    <td>0.535</td>
  </tr>
  <tr>
    <td>AOL</td>
    <td><b>0.657</b></td>
    <td><b>0.818</b></td>
    <td><b>0.758</b></td>
  </tr>
  <tr>
    <td colspan="4"><i>C4 Dataset, Mistral-7B</i></td>
  </tr>
  <tr>
    <td>RoBERTA</td>
    <td>0.238</td>
    <td>0.019</td>
    <td>0.301</td>
  </tr>
  <tr>
    <td>AOL</td>
    <td><b>0.620</b></td>
    <td><b>0.790</b></td>
    <td><b>0.809</b></td>
  </tr>
  <tr>
    <td colspan="4"><i>Arxiv Dataset, Llama-7B</i></td>
  </tr>
  <tr>
    <td>RoBERTA</td>
    <td>0.321</td>
    <td>0.519</td>
    <td>0.579</td>
  </tr>
  <tr>
    <td>AOL</td>
    <td><b>0.718</b></td>
    <td><b>0.862</b></td>
    <td><b>0.635</b></td>
  </tr>
  <tr>
    <td colspan="4"><i>Arxiv Dataset, Mistral-7B</i></td>
  </tr>
  <tr>
    <td>RoBERTA</td>
    <td>0.372</td>
    <td>0.249</td>
    <td>0.421</td>
  </tr>
  <tr>
    <td>AOL</td>
    <td><b>0.571</b></td>
    <td><b>0.682</b></td>
    <td><b>0.802</b></td>
  </tr>
</table>

Table 2: Precise Watermark Position Localization Performance: Intersection over Union (IoU) score for baseline RoBERTa and our method AOL. AOL consistently outperforms RoBERTa.

<table>
  <tr>
    <th>Method</th>
    <th>Time (s)</th>
    <th>TPR</th>
    <th>IoU</th>
  </tr>
  <tr>
    <td>WinMax-1</td>
    <td>3643.61</td>
    <td>0.99</td>
    <td>0.980</td>
  </tr>
  <tr>
    <td>WinMax-100</td>
    <td>33.90</td>
    <td>0.99</td>
    <td>0.791</td>
  </tr>
  <tr>
    <td>GCD</td>
    <td>1.24</td>
    <td>0.99</td>
    <td>0.387</td>
  </tr>
  <tr>
    <td>AOL</td>
    <td>2.18</td>
    <td>0.99</td>
    <td>0.718</td>
  </tr>
</table>

Table 3: Runtime and performance comparison of different methods.

<table>
  <tr>
    <th>Length</th>
    <th>Method</th>
    <th colspan="3">TPR</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>FPR-1</td>
    <td>FPR-2</td>
    <td>FPR-3</td>
  </tr>
  <tr>
    <td>3000</td>
    <td>VANILLA</td>
    <td>0.000</td>
    <td>0.012</td>
    <td>0.038</td>
  </tr>
  <tr>
    <td></td>
    <td>GCD</td>
    <td><b>0.722</b></td>
    <td><b>0.974</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>6000</td>
    <td>VANILLA</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.005</td>
  </tr>
  <tr>
    <td></td>
    <td>GCD</td>
    <td><b>0.730</b></td>
    <td><b>0.980</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>9000</td>
    <td>VANILLA</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td></td>
    <td>GCD</td>
    <td><b>0.730</b></td>
    <td><b>0.980</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>18000</td>
    <td>VANILLA</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td></td>
    <td>GCD</td>
    <td><b>0.730</b></td>
    <td><b>0.980</b></td>
    <td><b>1.000</b></td>
  </tr>
</table>

Table 4: VANILLA and GCD watermark segment classification results using the Unigram Watermark on Mistral-7B for different segment-level false positive rate targets, achieved by adjusting score thresholds.

The efficiency of our method, we compare against the brute-force WinMax algorithm (Kirchenbauer et al., 2024), a representative sliding window method, with window sizes of 1 and 100 tokens (WinMax-1, WinMax-100). Table 3 presents the results of our comparative evaluation.

The results demonstrate that WinMax achieves high true positive rate (TPR) and intersection over union (IoU) but at a prohibitive computational cost. WinMax-1 requires 3643.61 seconds per sample, while WinMax-100 reduces this to 33.9 seconds. In contrast, our approach (AOL) achieves comparable TPR and a strong IoU (0.718) in just 2.18 seconds, offering a dramatic improvement in computational efficiency. Additionally, we analyze the role of GCD in this efficiency gain. While GCD alone detects watermark presence with an IoU of 0.387, integrating it with AOL significantly enhances localization precision (IoU = 0.718). This ablation study underscores the importance of AOL in refining detection beyond the rough localization provided by GCD.

4.4 Detection Results with Different Lengths

As mentioned previously, watermark detection can easily be disturbed by long natural paragraphs, and our approach aims to minimize the effect of length scale. We test our method on texts of varying total lengths, ranging from 3000 to 18000 tokens, while keeping the watermark segment length constant at 300 tokens. The same detection threshold and parameters used for 3000 total tokens are applied across all lengths.

We find that the Gumbel watermark segment classification performs well even as total length increases, as shown in Table 4. For repetitive watermarks like KGW and Unigram, longer texts in the Geometry Cover also cause a decrease in segment detection, as shown in Figure 3. However, compared to directly detecting on the whole paragraph, this decrease is more acceptable. Importantly, the parameters used in these tests are identical to those for 3000 tokens. In practice though, for texts of different lengths, the number of starting points in the circular buffer should be adjusted accordingly. This way, similarly strong results can be achieved as with 3000 tokens.

![Watermark localization results using different watermarking methods and varying text lengths.](page_1012_1012_388_388.png)

Figure 3: Watermark localization results using different watermarking methods and varying text lengths.

4.5 Detection Robustness Against Attacks

We evaluate the robustness of our watermark detection method against three types of attacks (Ta-
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">KGW-Watermark TPR and IoU</th>
    <th colspan="3">Unigram-Watermark TPR and IoU</th>
    <th colspan="3">Gumbel-Watermark TPR and IoU</th>
  </tr>
  <tr>
    <th>FPR-1</th><th>FPR-2</th><th>FPR-3</th>
    <th>AOL IoU</th><th>FPR-1</th><th>FPR-2</th><th>FPR-3</th>
    <th>AOL IoU</th><th>FPR-1</th><th>FPR-2</th><th>FPR-3</th>
    <th>AOL IoU</th>
  </tr>
  <tr>
    <td colspan="13"><i>Random Swap</i></td>
  </tr>
  <tr>
    <td>Baseline</td><td>0.190</td><td>0.340</td><td>0.460</td><td>–</td><td>0.000</td><td>0.005</td><td>0.025</td><td>–</td><td>0.110</td><td>0.150</td><td>0.160</td><td>–</td>
  </tr>
  <tr>
    <td>Ours</td><td>0.175</td><td>0.325</td><td>0.380</td><td>0.095</td><td>0.740</td><td>0.990</td><td>1.000</td><td>0.472</td><td>0.390</td><td>0.550</td><td>0.560</td><td>0.325</td>
  </tr>
  <tr>
    <td colspan="13"><i>Random Delete</i></td>
  </tr>
  <tr>
    <td>Baseline</td><td>0.310</td><td>0.440</td><td>0.545</td><td>–</td><td>0.000</td><td>0.000</td><td>0.015</td><td>–</td><td>0.255</td><td>0.300</td><td>0.325</td><td>–</td>
  </tr>
  <tr>
    <td>Ours</td><td>0.645</td><td>0.750</td><td>0.820</td><td>0.269</td><td>0.630</td><td>0.905</td><td>0.960</td><td>0.475</td><td>0.750</td><td>0.830</td><td>0.850</td><td>0.613</td>
  </tr>
  <tr>
    <td colspan="13"><i>ChatGPT Paraphrase</i></td>
  </tr>
  <tr>
    <td>Baseline</td><td>0.050</td><td>0.195</td><td>0.335</td><td>–</td><td>0.000</td><td>0.000</td><td>0.005</td><td>–</td><td>0.020</td><td>0.065</td><td>0.065</td><td>–</td>
  </tr>
  <tr>
    <td>Ours</td><td>0.050</td><td>0.100</td><td>0.165</td><td>0.032</td><td>0.040</td><td>0.145</td><td>0.510</td><td>0.218</td><td>0.075</td><td>0.110</td><td>0.130</td><td>0.090</td>
  </tr>
</table>

Table 5: Watermark segment classification and localization performance with different attacks, evaluated at three distinct FPR levels (FPR-1/2/3) and IoU scores.

ble 5). First, we use GPT-3.5-turbo to rewrite the text segments containing the watermark as the paraphrasing attack. The other two attacks randomly swap or delete words at a ratio of 0.2. As expected, rewriting by ChatGPT is the most damaging attack, leading to a decline in detection performance. However, our detection method still significantly outperforms the baseline direct detection across most attack types in terms of TPR. For watermark localization, measured by IoU, our method still generates satisfactory results under these attacks. Overall, the results demonstrate the robustness of our watermark detection approach against various perturbations to the watermarked text.

5 Conclusion

This paper introduces novel methods for partial watermark detection in LLM-generated text, addressing the critical need for identifying watermark segments within longer, mixed-source documents. By leveraging the geometric cover trick and the Alligator algorithm, our approach achieves high accuracy in both classifying and localizing watermarks, significantly outperforming baseline methods. These advancements pave the way for more robust and reliable detection of synthetic text, promoting responsible use and mitigating potential misuse of LLMs in various domains.

Limitations

Our methods, GCD and AOL, can be applied to other watermarking schemes as long as they have token-wise detection scores for the sequence, such as Hu et al. (2024) and Zhao et al. (2024). The detection results are constrained by the strength of the original watermark generation and the quality of the prompt text. In some cases, low-quality text produced by the watermark generation method cannot be directly detected using the original detection method. Additionally, positive samples created by inserting the generated watermark paragraph into natural text may not be detectable with our approach. However, these limitations arise from the current limitations of watermark generation and detection methods themselves, which is outside the scope of detecting small watermarked segments within long text, the focus of this work. Therefore, we assume that our method needs only to detect reasonably high quality watermarked text segments embedded in long text.

References

Scott Aaronson. 2023. Simons institute talk on watermarking of large language models.

Michael Ahn, Debiddatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichtler, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, and Zhuo Xu. 2024. Autort: Embodied foundation models for large scale orchestration of robotic agents. Preprint, arXiv:2401.12963.

Dheeraj Baby and Yu-Xiang Wang. 2021. Optimal dynamic regret in exp-concave online learning. In Conference on Learning Theory, pages 359–409. PMLR.

Dheeraj Baby, Xuandong Zhao, and Yu-Xiang Wang. 2021. An optimal reduction of tv-denoising to adaptive online learning. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 2899–2907.

Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah
Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. 2024. Managing extreme ai risks amid rapid progress. Science.

Karen Blum. 2024. Tip sheet: Harnessing chatgpt for good journalistic use. Association of Health Care Journalists.

Miranda Christ, Sam Gunn, and Or Zamir. 2023. Undetectable watermarks for language models. arXiv preprint arXiv:2306.09194.

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615–621.

Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. 2015. Strongly adaptive online learning. In International Conference on Machine Learning, pages 1405–1411. PMLR.

Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejevicova, et al. 2024. Scalable watermarking for identifying large language model outputs. Nature, 634(8035):818–823.

Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. 2023. Three bricks to consolidate watermarks for large language models. In 2023 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6. IEEE.

Google. 2024. Amie: A research ai system for diagnostic medical reasoning and conversations. Google blog.

Cristian Grozea, Christian Gehl, and Marius Popescu. 2009. Encoplot: Pairwise sequence matching in linear time applied to plagiarism detection. In 3rd PAN Workshop. Uncovering Plagiarism, Authorship and Social Software Misuse, page 10.

Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. 2024. Unbiased watermark for large language models. In The Twelfth International Conference on Learning Representations.

Intelligent. 2024. 4 in 10 college students are using chatgpt on assignments. Intelligent blog.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lamplé, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Samantha Murphy Kelly. 2023. ChatGPT creator pulls AI detection tool due to ‘low rate of accuracy’. CNN Business.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2024. On the reliability of watermarks for large language models. In The Twelfth International Conference on Learning Representations.

Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. 2023. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15593.

Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. Gpt detectors are biased against non-native english writers. Patterns, 4(7).

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, pages 24950–24962. PMLR.

Maxim Mozgovoy, Tuomo Kakkonen, and Georgina Cosma. 2010. Automatic student plagiarism detection: future perspectives. Journal of Educational Computing Research, 43(4):511–531.

OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI blog.

OpenAI. 2023. New ai classifier for indicating ai-written text. OpenAI blog.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsieh. 2024. Red teaming language model detectors with language models. Transactions of the Association for Computational Linguistics, 12:174–189.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature.

Edward Tian. 2023. Gptzero: An ai text detector. web.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Bob Violino. 2023. Ai tools such as chatgpt are generating a mammoth increase in malicious phishing emails. CNBC.

Michael J Wise. 1996. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer science education, pages 130–134.

Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. 2023. Provable robust watermarking for ai-generated text. In The Twelfth International Conference on Learning Representations.

Xuandong Zhao, Lei Li, and Yu-Xiang Wang. 2024. Permute-and-flip: An optimally robust and watermarkable decoder for llms. arXiv preprint arXiv:2402.05864.
A  Detailed Experiment Settings

A.1 Datasets and Mixed-source Texts
We utilize two text datasets: C4 (Raffel et al., 2020) and Arxiv (Cohan et al., 2018). The “Colossal Clean Crawled Corpus” (C4) dataset is a large collection of English web text from the public Common Crawl, providing unwatermarked human-written content. We use random samples from its news-like subset. The Arxiv dataset, sourced from arXiv.org and PubMed.com, consists of scientific paper abstracts and articles. Both datasets provide unwatermarked (negative) and partially watermarked (positive) samples. To transform unwatermarked samples into partially watermarked samples, we randomly select 3-5 sentences in a long text and set them as prompts. Then, we generate watermarked text conditioned on the prompts using large language models. The generated responses replace the original suffix sentences after the prompt. To simulate realistic watermark embedding within longer documents and ensure a detectable watermark length, we embed 300-token watermarks within 3000-token contexts (10% ratio). This ratio is designed to represent a plausible level of watermark integration in texts such as articles or web pages, while simultaneously providing a sufficient length for reliable detection. The watermark was randomly positioned within the longer context, and these positions were recorded for subsequent localization evaluation. Our objective is to determine the presence of watermarked text within a document and accurately locate its segment. For testing, we utilized 500 samples per dataset.

A.2 Language Models and Watermarking Methods
We use the publicly available LLaMA-7B (Touvron et al., 2023) and Mistral (Jiang et al., 2023) models. To verify the general applicability of the watermark detection methods, we select three watermarking techniques: Gumbel-Watermark (Aaronson, 2023), KGW-Watermark (Kirchenbauer et al., 2023), and Unigram-Watermark (Zhao et al., 2023). These methods represent the state-of-the-art watermarking approaches for LLMs, offering high quality, detectability, and robustness against adversarial attacks. For all watermarking generations, we configure the temperature to 1.0 for multinomial sampling. Additionally, for KGW-Watermark and Unigram-Watermark, we set the green token ratio \( \gamma \) to 0.5 and the perturbation \( \delta \) to 2.0.

A.3 Baselines
In watermark segment detection, we use the original watermark detector in each watermarking method as the VANILLA baseline to compare with our approach GCD. In watermark segment localization, we use RoBERTa (Liu et al., 2019) models for comparing with our method AOL. We train each RoBERTa (designed for different watermarking methods) to predict whether a sequence is a watermarked sequence or not, given the watermark detection scores \( r \) for each token. We add an extra fully connected layer after getting the representation of the [CLS] token. We construct 1000 training samples with 60 token scores as input and the binary label of this segment as the label. We train the RoBERTa model for 20 epochs and enable early stopping if the loss converges. It can reach over 90% accuracy in the training set. During testing on mixed-source text, we employ the sliding window idea to test each chunk for watermarks and then calculate the IoU score. We also conducted experiments with the WinMax algorithm (Kirchenbauer et al., 2024), a representative brute-force method with time complexity of \( \mathcal{O}(n^2) \). We tested two window sizes: 1 token (WinMax-1) and 100 tokens (WinMax-100).

A.4 Evaluation
For the watermarked text classification task, we report the true positive rates (TPR) based on different specified false positive rates (FPR). Maintaining a low FPR is critical to ensure that human-written text is rarely misclassified as LLM-generated text. Since the FPR at the per-instance level differs from the document-level FPR, we calibrate FPR to three distinct levels in each scenario to enable fair comparisons. Specifically, we manipulate the pre-segment FPR (SEG-FPR) by adjusting the threshold parameter \( \tau \) as outlined in Algorithm 1. Then, we can get the empirical document FPR (Doc-FPR) by evaluating our method GCD based on pure natural text. For VANILLA, we set the FPR according to GCD’s empirical FPR and subsequently test for its empirical TPR. For locating specific watermark segments, we calculate the Intersection over Union (IoU) score to measure the accuracy of watermark segment localization. The IoU score computes the ratio of the intersection and union between the ground truth and inference, serving as one of the main metrics for evaluating the accuracy
of object detection algorithms:

\[
\text{IoU} = \frac{\text{Area of Intersection}}{\text{Area of Union}}
= \frac{|\text{Detected Tokens} \cap \text{Watermarked Tokens}|}{|\text{Detected Tokens} \cup \text{Watermarked Tokens}|}
\]

Finally, to assess inference speed, we calculate the average time to detect a single sample on a server equipped with an AMD EPYC 9354 32-Core Processor and Nvidia A6000 GPUs.

B FPR Calibration Function \( F \)

As discussed in Section 3.1, the FPR Calibration Function calculates the p-value / FPR for per-instance watermark detection, given the detection scores and the original text. We follow the methodologies outlined in Zhao et al. (2023) and Fernandez et al. (2023) for FPR calibration. This section presents three methods for detecting KGW-Watermark, Unigram-Watermark, and Gumbel-Watermark, each employing a unique scoring mechanism and statistical test to assess the false positive rate.

B.1 KGW-Watermark

For the KGW-Watermark scheme described in Kirchenbauer et al. (2023), we follow the approach in Fernandez et al. (2023). When detecting the watermark for a text segment, under the null hypothesis \( \mathcal{H}_0 \) (i.e., the text is not watermarked), the score \( \text{Score}(y) = \sum_{t=1}^n 1(y_t \in \text{Green Tokens}) \) follows a binomial distribution \( \mathcal{B}(n, \gamma) \), where \( n \) is the total number of tokens and \( \gamma \) is the probability of a token being part of the green list. The p-value for an observed score \( s \) is calculated as:

\[
\text{p-value}(s) = \mathbb{P}(\text{Score}(y) > s \mid \mathcal{H}_0)
= I_\gamma(s, n - s + 1),
\]

where \( I_x(a, b) \) is the regularized incomplete Beta function.

B.2 Unigram-Watermark

For the Unigram-Watermark scheme, we adopt the methodologies from Zhao et al. (2023). To achieve a better FPR rate, the detection score differs from the KGW-Watermark approach. The score is defined as \( \text{Score}(y) = \sum_{t=1}^m 1(\tilde{y}_t \in \text{Green Tokens}) \), where \( \tilde{y} = \text{Unique}(y) \) represents the sequence of unique tokens in text \( y \), and \( m \) is the number of unique tokens.

Under the null hypothesis \( \mathcal{H}_0 \) (i.e., the text is not watermarked), each token has a probability \( \gamma \) of being included. Using the variance formula for sampling without replacement (\( N \) choose \( \gamma N \)), the variance of this distribution is:

\[
\begin{align*}
\operatorname{Var}\left[ \sum_{t=1}^m 1(\tilde{y}_t \in \text{Green Tokens}) \mid y \right] \\
= m \gamma (1 - \gamma)(1 - \frac{m-1}{n-1}),
\end{align*}
\]

where \( n \) is the total number of tokens, and \( \gamma \) is the probability of a token being in the green list. The conditional variance of \( z_{\text{Unique}(y)} \) is thus \( (1 - \frac{m-1}{n-1}) \). The false positive rate (FPR) is then given by:

\[
\text{FPR} = 1 - \Phi \left( \frac{z_{\text{Unique}(y)}}{\sqrt{1 - \frac{m-1}{n-1}}} \right),
\]

where \( \Phi \) is the standard normal cumulative distribution function.

B.3 Gumbel Watermark

For the Gumbel Watermark (Aaronson, 2023), we adopt the approach presented in (Fernandez et al., 2023), which utilizes a gamma test for watermark detection. Under the null hypothesis \( \mathcal{H}_0 \), \( \text{Score}(y) = \sum_{t=1}^n \log(1/(1 - r_{y_t})) \) follows a gamma distribution \( \Gamma(n, 1) \). The p-value for an observed score \( s \) is calculated as:

\[
\text{p-value}(s) = \mathbb{P}(\text{Score}(y) > s \mid \mathcal{H}_0) = \frac{\Gamma(n, s)}{\Gamma(n)}
\]

where \( \Gamma(n, s) \) is the upper incomplete gamma function and \( n \) is the total number of tokens.

For all three methods, a lower p-value indicates stronger evidence against the null hypothesis, suggesting a higher likelihood that the text is watermarked. These methods provide a comprehensive framework for watermark detection, each offering unique advantages depending on the specific characteristics of the text and the desired sensitivity of the detection process.

Reviewer paper summaries: Watermarking methods for large language models typically adjust the logits of token generation to embed textual watermarks. Correspondingly, watermark detection methods calculate token-wise watermark scores for a given text. If the overall score of the text exceeds a certain threshold, the text is considered to contain a watermark. However, in real-world scenarios, only a portion of a long document may be generated by an LLM. Existing methods can only determine whether the entire document contains a watermark, but cannot pinpoint specific watermarked locations within mixed texts.

This paper proposes an efficient and precise watermark detection algorithm capable of locating watermarked segments within long documents. For instance, the paper introduces the GCD and AOL methods. GCD can divide the entire document into different segments and check whether each segment contains a watermark from a large model. The AOL method further accurately identifies the start and end points of the watermarked section, ultimately locating the watermarked portion of the document with a complexity of O(nlogn).

The paper presents work on document which may contain sections of text generated by an LLM with watermarked output and sections without watermark. There are two tasks, with the first one classifying, if a watermark is present or not, and the second one to locate the sections of watermarked text. The main contribution is the runtime performance improvements over naive baselines.

The authors propose a method to efficiently detect watermarked text segments within a document. Their proposed approach has O(nlogn) complexity while the existing approaches has O(n2) complexity. In their experiments, they show that their approach is better than the baselines but worse than WinMax algorithm in terms of accuracy. However, their approach is much faster than WinMax and has a reasonable accuracy.
Summary of strengths: - Practical Value: The watermark localization strategy for LLMs proposed in this method has significant practical value, as it can identify potential LLM-generated segments within long textual resources.

- High Efficiency: Experiments compared the time consumption of this method with the Winmax method (which uses a sliding window strategy with O(n²) complexity to locate watermarked segments). The results demonstrated that AOL significantly reduces time consumption without compromising performance.

- High Accuracy: Experiments showed that this method outperforms VANILLA in terms of true positive rate (TPR) and surpasses ROBERTA in localization precision (IoU). Additionally, it maintains strong performance on long documents, whereas the effectiveness of other baselines significantly declines as document length increases, as indicated by a notable drop in IoU.

- Robustness: Experiments demonstrated that this method exhibits better robustness against certain watermark attacks, such as paraphrasing, random shuffling, and deletion.

S1. The paper addresses a problem of practical relevance. 

S2. Observed performance improvements over selected baselines in terms of performance and runtime.

S3. The method allows to be combined with multiple watermarking schemes (shown with selected three methods)

It is technically sound.

The proposed approach is faster than the state-of-the-art method and has a reasonable performance.
Summary of weaknesses: The length of watermark segments may significantly impact detection performance. Although the paper addresses this issue through the GCD and multi-scale detection strategies, in practical applications, the length of watermark segments may vary across different scenarios. If the watermark segment is too short, detection performance may degrade, especially in long documents where the watermark signal could be overwhelmed by a large volume of non-watermarked text, leading to detection failures.

Although the paper tests three types of attacks (paraphrasing, random shuffling, and deletion) and demonstrates the method's performance under these attacks, the experimental results still show a decline in detection performance, particularly under paraphrasing attacks. This suggests that the method may not be sufficiently robust against certain sophisticated adversarial attacks.

W1 - The suggested WinMax baseline is only included in the detection efficiency comparisons (with results as expected), and only evaluated under conditions of identical TPR compared to GCD and AOL. It remains unclear how much GCD and ALO trade off in accuracy compared to WinMax under more general conditions. 

W2 - The revision clarifies the individual roles of GCD and AOL, AOL's effectiveness is still primarily demonstrated through IoU metrics (localisation). There is no explicit evaluation of AOL’s contribution to segment-level TPR, leaving it unclear whether the improved localisation provided by AOL also enhances watermark detection accuracy (TPR), or if its benefits are strictly limited to localisation. Clarifying this would reveal the practical value of AOL and whether its computational overhead is justified for detection accuracy improvements.

The writing of the paper can be improved as some parts are wordy and confusing.

Some experiments needs more elaboration and/or explanation.

Appendix contains too much information which might be important for giving a decision about the paper.
Comments, suggestions and typos: -  It is recommended to add experiments testing the impact of different watermark lengths on detection performance.
- The paper should consistently use the term "Red-Green Watermark" instead of "Green-Red Watermark".
- Provide detailed xxplanation of FPR levels (FPR-1/2/3)

In 2.1 - A prompt is only present/relevant for generative LLMs that are instruct fine-tuned, while base models do not have prompts, and watermarking should not be limited to instruct models.

It would be interesting to learn, how the position within the text influence the performance of the proposed method. For example, Figure 1 makes it clear that a watermarked text overlapping 7/8 is harder to detect than other parts.

L424 - "suspected plagiarism", unclear as the presence of a watermark does not allow to make statement about potential plagiarism, suggest to rephrase.

Would be interesting to learn what the runtime impact of the length of each segment is, from the runtime complexity it appears that this factor is not taken into consideration. 

Figure 3 - Surprising to see such differences between the watermarking approaches, what are the reasons? Also, are the same datasets used here?

4.5 - Unclear, what the baseline is here

The authors claim that “By embedding identifiable patterns within the generated text, watermarks can signal whether a piece of text originates from an LLM.” However, this is incorrect because watermarks can be used to detect if the text is generated using a specific LLM, not any LLM.

The details about Vanilla methods and Roberta-based method need more clarification. I had to read a couple of times to understand the details of these approaches but I am still not fully confident. Having said that, the authors compare their approach against Vanilla methods and show that their approach outperforms Vanilla methods. However, the authors choose a highly easy baseline to compare. In particular, Vanilla methods are not designed detect the text segments. Obviously their performance will be lower if the given text contains not-watermarked text segments. Considering that watermarked text ratio is 10%, the results are highly expected. I think the authors could develop some simple baselines. For instance, divide the documents into chunks, run the vanilla methods for each chunk and then decide it is watermarked if at least one chunk is detected as watermarked. They can also develop some other variants of these methods. Overall, I am not sure if it is beneficial to compare with Vanilla methods.

The authors very briefly explain an experiment when there are multiple watermarked segments. However, for this experiment, they do not report results for any baseline method. This experiment needs elaboration. 

The watermarking methods has hyper-parameters but the authors do not mention how these parameters are tuned. There is no mention of validation set for hyper-parameter tuning.
Overall assessment scores: 3.5

3

3.5