On the Reliability of Watermarks for Large Language Models
Authors: John Kirchenbauer
Jonas Geiping
Yuxin Wen
Manli Shu
Khalid Saifullah
Kezhi Kong
Kasun Fernando
Aniruddha Saha
Micah Goldblum
Tom Goldstein
Venue: ICLR 2024 poster (2024)
Keywords: Machine Learning
LLM
Watermark
Language Model
Natural Language Processing
Generative AI
Primary area: societal considerations including fairness, safety, privacy
TL;DR: A study on the reliability of watermarks for large language models in realistic settings.

Abstract: As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. _Watermarking_ is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked text may be modified to suit a user's needs, or entirely rewritten to avoid detection. We study the robustness of watermarked text after it is re-written by humans, paraphrased by a non-watermarked LLM, or mixed into a longer hand-written document. We find that watermarks remain detectable even after human and machine paraphrasing. While these attacks dilute the strength of the watermark, paraphrases are statistically likely to leak n-grams or even longer fragments of the original text, resulting in high-confidence detections when enough tokens are observed.  For example, after strong human paraphrasing the watermark is detectable after observing 800 tokens on average, when setting a $1\mathrm{e}{-5}$ false positive rate. We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors.

Published as a conference paper at ICLR 2024
ON THE RELIABILITY OF WATERMARKS FOR LARGE LANGUAGE MODELS
John Kirchenbauer", Jonas Geiping
Yuxin Wen', Manli Shu', Khalid Saifullah',, Kezhi Kong',
Kasun Fernando Aniruddha Saha', Micah Goldblum', Tom Goldstein
University of Maryland
Institute for Intelligent Systems. Tübingen Al Center 4 Scuola Normale Superiore di Pisa, New York University
2 ELLIS Institute Tübingen, 3
ABSTRACT
As LLMs become commonplace, machine-generatee text has the potential to flood the internet with spam, social media bots, and valueless content. Watermarking is simple and effective strategy for mitigating such harms by enabling the detec- tion and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked text may be modified to suit a user's needs, or entirely rewritten to avoid detec- tion. We study the robustness of watermarked text after it is re-written by hu- mans, paraphrased by a non won-watermarked LLM, or mixed into a longer hand- written document. We find that watermarks remain detectable even after human and machine paraphrasing. While these attacks dilute the strength of the water- mark, paraphrases are statistically likely leak n-grams or even longer fragments of the original text, resulting in high-confidence detections when enough tokens are observed. For example, after strong human paraphrasing the watermark is de- tectable after observing 800 tokens on average, when setting 1e-5 false positive rate. We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside large document, and compare the robustness watermarking to other kinds of detectors.
INTRODUCTION
The capability tell the difference between machine geeteiee-geerteedd and human--witten text underlies many approaches to reduce potential harms caused by generative language models (Bender et al., 2021; Crothers et al., 2022). This includes known harms, such as models being used at-scale for malicious purposes including social media bots, fake product reviews (Palmer, 2023), automated text generation on wikipedia (Woodcock, 2023), or automatic generation of targeted spearphishing attacks on vulnerable subpopulations (Schneier, 2021). Equally important, the ability to track and document the use of machine- generrted text has the potential to reduce harms from future problems that have not yet been observed. These problems might range from the pollution of future training data (Radford et al., 2022) to the prevalepe of LLM LLM-generated blogs and other web content. Unfortunately, detection of machine-generater text is potentially difficult. Models are prompted with diverse instructions, resulting in a wide range of downstream behaviors for both machines and humans that are difficult to characterize. This can lead to low accuracy or impractical false positive rates that especially impact vulnerable subgroups, such non-native speakers (Liang et al., 2023).
One way enable accurate detection of machine-generatte text through watermarking, where generated text is marked imperceptibly so that its origin can be determined (Atallah et al., 2001; Fang et al., 2017; Kirchenbauer et al., 2023). Because watermarks rely on subtle patterns in text that statistically unlikely to be replicated by human, watermarking enables detectors that achieve high levels of accuracy on relatively short fragments of text. This makes watermarking a promising approach for the reliable separation of w and machine geeeertee text (Grinbaum & Adomaitis, 2022). While the effectiveness of watermarks has been shown in ideal scenarios where verbatim L.M outputs fed directly to detector, this is an idealized setting. In practice, humans

---

Published as a conference paper at ICLR 2024
Natemannked Text
Text Mixing
Paraphrasing
Human Paraphrasing
Figure What happens to watermarked text in-the-wild' this work we study watermark robustness against a number of text modifications, as visualized here. We visually depict that machine paraphrasing methods have a tendency to shorten texts, humans are quite effective at reducing the strength of watermark by increasing the number of red tokens, and that short spans of watermarked text may he copied and pasted into large document. In all of these scenarios, we find that high confidence detection reliably occurs given enough tokens as input.
may mix machine- machine-generated text into larger documents with multiple sources. Furthermore, human may revise or rephrase parts the synthetic text (possibly aided by another language model) to better suit their needs, potentially even with the deliberate goal of evading detection.
In this work, we investigate the reliability of watermarking as a strategy to identify machine- generated text in realistic scenarios, based the approach of Kirchenbauer et al. (2023). We are focused on whether watermarks remain detectable under various types of realistic corruptions (i.e. attacks): How reliable is watermarking when generated text is handled by humans, be it mixing with huwan-wwiiieeee text, rewriting parts or the entire passage, or feeding the text into other popular language models for rephrasing? A reliable detection strategy should be robust to these common scenarios, maintaining some statistical power and a low false positive rate (Crothers et al., 2022).
We make the following contributions:
We re-investigate all parts of the watermark generation and watermark detection pipeline to optimize for reliability in realistic scenarios.
We study the reliability of watermarking against paraphrasing by strong language models. When GPT-3.5 5 and purpose-built models are used to paraphrase watermarked text, ROC-AUC re- mains > 0.85 when T = 200 tokens are available, and > 0.9 with T = 600 tokens.
We consider "Copy-Paste" scenario where watermarked text appears inside a larger hand- written passage. When numan-written passage of length 600 tokens has 150 tokens of water- marked text inserted into it, AUC for detection is above 0.95.
We conduct human study in which watermarked text is re-written by volunteers with the explicit goal of removing the watermark. While humans are relatively strong attackers, after enough observed tokens (about 800) watermarks are still usually detectable in human paraphrases even when enforcing a 1e-5 false positive rate.
We provide reliability estimates of watermarking compared to other late--o-theearl art approaches, such as loss-based detection (Mitchell et al., 2023) and retrieval (Krishna al., 2023), showing that these struggle at longer sequence lengths when attacked.
We argue that the correct way characterize the strength and robustness of different detection approaches is not simply via detection accuracy metrics for a specific distribution text, but rather to measure how much machine- -eeehiinn--eerteedd text is required for each approach to succeed, and how method behaves as function of text sequence length. Across all the scenarios we consider in this work, we ultimately find watermarking be more robust than other post-hoc detection methods (such as loss-based detection and caching/retrieval schemes), especially due to its favorable sample complexity, i.e., scaling behavior in terms of amount of text that sufficient guarantee detection.
AN OVERVIEW OF
TEXT DETECTION
The problem of separating human-writter and machine-writted text can be approached from several directions. Broadly, we can distinguish post-hoc detection systems that require no interaction during text generation, and proactive detection systems that require some action during generation. These latter systems are generally much more robust, with the downside that they have to be adopted by the model owner.
2

---

Published as a conference paper at ICLR 2024
One broad class of detectors employ statistical outlier detection methods, based on entropy (Lavergne et al., 2008), perplexity (Beresneva, 2016; Tian, 2023), n- n-gram frequencies (Grechnikov et al., 2009; Badaskar et al., 2008), as in DetectGPT Mitchell et al. (2023), the observation that L.MS typically assign their own text generations higher probability than "nearby" text sequences produced by span replacement with different L.M. Even though DetectGPT exhibits superior per- formance compared to other zero-shot statistical outlier detection methods, it suffers from excessive computational costs. Further, all advanced statistical detectors relying on language model statistics such as perplexity or curvature ideally require white-box access to model parameters.
A retrieval- based approach for text detection, as described in Krishna et al. (2023), noticeably different paradigm, and an example of a proactive detection technique. Here, all text generated by a given model is stored in database. Later, text samples are evaluated by matching against this database. This approach requires action taken by the model owner, but it can be quite reliable, even when faced with broad modifications of text, such as strong paraphrases.
Watermarking also requires action by the model owner they must embed the hidden watermark signal into all outgoing text. Watermarking as a concept has a long history (Brassil et al., 1995). Older systems were based on rule-based methods to imprint watermarks into existing text (Atallah et al., 2001; Chiang et al., 2004; Venugopal et al., 2011; Topkara et al., 2006). More recent approaches for watermarking neural networks Ziegler et al. (2019); Dai & Cai (2019); Abdelnabi & Fritz (2021); He et al. (2022a;b) are learned end-to-end with both encoding and decoding of each sample. The lack of theoretical guarantees and interpretability of these approaches are problems for their widespread adoption. However, it is also possible to place watermarks on a robust mathemati- cal foundation. Watermarks can be embedded by minimally modifying the distribution of generated output text (Fang et al., 2017; Kaptchuk et al., 2021; Kirchenbauer et al., 2023), and watermarks of this type have recently been adapted for various applications (Lee et al., 2023; Yoo al., 2023). In this work, we mainly consider the combinatorial watermark described in Kirchenbauer et al. (2023).
3 HOW TO IMPROVE WATERMARK RELIABILITY?
There are a number of parameter and design choices that go into a watermark, with different pa- rameters offering benefits in different use cases. In this section, we briefly describe the watermark proposed in Kirchenbauer et al. (2023), and the variations on the watermark that we will study.
Assume an autoregressive language model is trained on a vocabulary V of size IV Given a sequence of tokens input step a language model predicts the next token in the sequence by outputting vector of logit scores 4 RIV with one entry for each item in the vocabulary. A random number generator is seeded with a context window of h preceding tokens, based on a pseudo-random func- tion (PRF) f : Nh - N. With this random seed, subset of tokens of size v are "colored" green and denoted G,. Now, the logit scores 4 are modified so that
4eat + 114
if K Gt otherwise.
1ik
After modifications, these logit scores can be used for any desired sampling scheme. In the sim- plest case, one passes the scores through a softmax layer and samples from the output distribution, resulting in bias towards tokens from G,.
The watermark can be described by four parameters. The "hash" used to generate the greenlists f with context width h. greenlist fraction % and the logit bias 8. After watermarked text is generated, one can check for the watermark without having access to the LLM by re-computing the greenlist at each position and finding the set s of greenlist token positions. The statistical significance of sequence of tokens of length I' can be established by deriving the z-score
= (Isl - -T) / 11(1-1100
(2)
When this z-score is large, one can be confident that the text is watermarked. We now discuss several variations to this scheme, which lead to improved empirical behavior.
3

---

Published as a conference paper at ICLR 2024
3.1 IMPROVED HASHING SCHEMES
The experiments in Kirchenbauer et al. (2023) focus on a simple scheme where the random number generator seeded using h = 1, i.e., only single token at position L- is used to color the token at position 1. We refer to this scheme as LeftHash. Because the greenlist depends only on one single token, a third-party observer could learn the greenlist associated with the token at position t - by searching subsequent words at position é that are less likely to appear than expected under won-watermarked distribution. In situations where the watermark scheme is intended be kept secret behind an API, more secure scheme is needed.
Kirchenbauer et al. (2023) also mention scheme (Algorithm 3) in which the greenlist at position is determined by including the token at position / itself (yet to be generated), in addition to tokens to the left / the inputs to f. This approach effectively increases the context width by 1, making harder to discover the watermark rules by brute-foree methods. We generalize this scheme to include arbitrary functions/ f and text generation routines, which we describe detail in Algorithm 1 in the Appendix. We use this method, combined with context width of h 4 and "minimum" based method of choosing the seed token, and refer to it as the SelfHash scheme in the rest of the main work. For more in-depth description of various hashing schemes and evaluation of their performance tradeoffss see Appendix A.5.
3.2 IMPROVED WATERMARK DETECTION
The original z-test, Equation (2), may be optimal when watermarked text is interspersed with non-watermarked text, eg. the case of single paragraph of watermarked text embedded inside much larger non-watenn ooteaaterarree document. Because the 2-score computed globally over the whole document, the surrounding unwatermarked text reduces the average greenlist rate. To be able to accurately detect watermarked sub-regions even in long documents, we design a windowed test, called WinMax that searches for the continuous span of tokens that generates that highest z-score (see Appendix A.9 for a formal definition).
EVALUATING WATERMARKING IN THE WILD
Watermarks are extremely accurate in simple scenarios in which long span (50+ tokens) of text is tested in isolation and without modification. However, whether for benign or deceptive purposes, in many cases, the generated text will be embedded inside a larger document, edited by human, or paraphrased by another language model. This section studies the robustness of the watermark under these more complex use cases.
We assume the following threat model: A user of watermarked text is aware that the text is water- marked, but has no knowledge of the hashing scheme, fraction 7, or context width h that describe the watermark. They paraphrase some (possibly all) spans of the text to evade detection.
In this scenario, we can understand watermark reliability through the following two observations:
1. Without white-box access to the hashing scheme, a cannot remove the watermark without ensuring that the re-phrased text contains none of the n-grams from the original text. the user ever recycles long words (which often contain multiple tokens) or phrases from the original text, the watermark will remain, although with reduced strength.
2. If a paraphrased text skews even slightly toward watermarked behavior, the watermark will be detected given enough tokens. Suppose each token is only & more likely to be green than random baseline, i.e. |s| 7T + +). Then for any z-score threshold, we expect to detect the watermark after seeing T = (z² 72 (/{">) tokens.
For reasons above, we do not expect paraphrasing attacks to completely remove watermark. Rather, we expect such attacks to increase the number tokens needed for confident detection. This hypothesis is supported by the theoretical analysis of Chakraborty al. (2023), who assert that for an optimal detector, detection is always possible, given a sufficient number of samples.
Following Kirchenbauer et al. (2023), Cleaned corpus (C4) dataset as a source of prompts for open-ended generation. For the human study, adopt the "Loong-Form Question Answering" (LFQA) dataset curated by Krishna al. (2023)
Experimental Setup:
the Colossal Common Crawl
4

---

Published as a conference paper at ICLR 2024
MOCCAUC of Watermarks after Machine Attacks
1.00
3.95
Attokeed ROCAMe
3.85
0.80
0.25
2.70
0.65
2060
0.55
0.50
r-Ulia CP 1.1e
Seffttah Engrns
Loiiiaash GPT
Lett CPR 1-10
Sot CY. WOS Lott CF JaIP
(TL 1-15
TEATEEAA Hirper Soll Loiiitaii Catiiiihhhh CPty
sultilel CT
Lettll Wilfoa
CP
CP
Saifitahh
SallHish
ab
1-0
30000
Figure 2: Watermark robustness to various automated text manipulations. Each bar denotes the ROC-AUC for detecting the watermark in text that has length T after the attack. Each bar label denotes the hashing scheme (SelfHash or LeftHash), and attack type [Copy-Paste (CP), paraphrase model (Dipper) and general- purpose model (GPT).. For the Copy-Paste attack, we vary the number of watermarked passages inserted and their length. CP-3-10% denotes a copy-paste attack in which only 10% of the text is watermarked, and this watermarked text is broken across 3 different locations in the document. In the Dipper and GPT attacks, the watermarked document is re-written in its entirety, resulting in paraphrases that are shorter than the original text. In this case, the average length T reported for both GPT and Dipper, respectively.
based on a selection of posts and responses to question's 's on Reddit's "Explain Like I'm Five" (ELI5) forum.
In the majority our experiments, we use 11 ama (Touvron et al., 2023a), a modern state-of-the-- art open source veneral-purpon model to generate watermarked text. In particular, we use the 7 billion parameter variant under the research use permissions the license. In the human study portion of the experiments, we adopt Vicuns (Vicuna-Team, 2023), fine-tuned variant of the base model, better suited to responding to the QA prompts used in the study. We ablate these standard dataset and model settings in the Appendix by running a set of similar experiments on alternate datasets and models.
We use a single set of language model sampling parameters across all experiments, multinomial sam- pling at temperature 0.7, and for all experiments, unless explicitly stated, we use the LeftHash wa- termark scheme based on an additive PRF with context window h and (7, 6) = (0.25, 2.0). This parameter combination was observed to be near the pareto frontier shown in Figure of Kirchen- bauer et al. (2023), i.c. extremely detectable, but with marginal cost to generation quality.
Due to the importance of text length in the performance of detection methods, including watermark- ing, we carefully control and specify the generation lengths considered in each experiment first by limiting the number of tokens the model can generate, and then sub-sampling the resulting data to just those generations which are within a specified range around a target length value. We use "T"" to refer to the number of tokens considered throughout all experimental sections, and unless other- wise noted we include passages with length within +:25 tokens around that value. Unless otherwise stated, in all figures, ROC space plots and measurements are backed by 500 positive and > 500 negative samples, and other types of point estimates are also based on > 500 samples.
4.1 ROBUSTNESS TO MACHINE PARAPHRASING ATTACKS
We run series of paraphrasing attacks where we use a strong publicly available general-purpose language model API to paraphrase the text. This is a departure from the threat model of Kirchen- bauer et al. (2023), who only characterize less capable models (T5). Our "GPT" paraphrase attack uses gpt 3. 5-turbo, which is a version of the model powering ChatGPT, to rewrite the text. We also try a specially tailored paraphrasing model the 11B Dipper model introduced in Krishna et al. (2023). We engineer the GPT prompt for optimal paraphrasing performance. Our explo- rations included prompts that explicitly instruct the LLM not to recycle bi-grams from the original text and use the best performing prompt (see the Appendix for ablation details). We note that
5

---

Published as a conference paper at ICLR 2024
Detection Rote of Wateraarkgg va Tokous Olserved
Delection Rate Watermar king after Attock vs. Tokoms Olaervoi
1.7
0.5
0.5
0.84
0.77
6.4
01.4
Tokom
Figure 3: AUC as a function of text length. (Left) AUC of original watermarked text with no attack. (Right) AUC under attack. Attacks dilute the watermark strength, hut the watermark recovers its accuracy as increasingly more tokens shorter outputs than they receive as inputs, we make the curves translucent starting at the mean sequenee length of original text to indicate that these measurements are based on increasing fewer samples. For the Dipper attack this is ~ 500 and for the GPT attack this is ~ 300. In contrast, the Copy-Paste attack does not suffer from text shortening, and those sequences are still full length i.e. 600 1 25.
observed. Due the tendency of the GPT and Dipper paraphrasers to produce
when prompted for paraphrasing with longer inputs, the GPT model often effectively summarizes the text, sometimes reducing its length by more than 50%, which makes detecting the watermark more challenging.
The results for the main experiments attacking the watermark in this way are summarized in Figure 2. Note that we do not show the ROC-AUC numbers for the "unattacked" setting, because the detection performance of both the LeftHash watermark and the SelfHash variant at these token lengths are always > 0.999 for these experiments. Examining the settings where GPT or Dipper are the attack type (the smaller groups of 4 bars) with token length before and after attack of roughly T 200, we see that the attack achieves a detection performance reduction of 0.05 - 0.15 points AUC. When the lengths before attack are 600, despite the fact that Dipper and GPT reduce the lengths to 500 and 300 respectively, the attack reduces AUC by < 0.1 points.
Since the watermark's success and the number of tokens observed are tightly linked, further investigate this dimension through the use of "detectability @ T plots where we test prefixes of the generated sequences of lengths T; E [T]] and compute ROC-AUC for each prefix length (AUC @ T). We also provide a version of these charts where we instead show TP rates at low FPR in the appendix. Under this lens of analysis, we observe in Figure that in the unattacked setting, the AUC @ T quickly approaches its eventual value of ~ 1.0. In the attacked setting, we see that the AUCs are reduced overall by all methods, but that despite how successful a paraphrasing attack might look at 200-300 tokens, by the 600 token mark, under the GPT and Dipper model-based attacks, the watermark recovers to an AUC greater than 0.9.
4.2 ROBUSTNESS TO COPY-PASTE ATTACKS
We devise a synthetic but realistic attack where we take watermarked text fragments and embed them inside a surrounding un-watermarked, human-written document. This creates heterogeneous examples where only few sub-spans of the text are watermarked. The attack method has two parameters, (1) the number of watermarked span insertions and (2) the fraction of the resulting document that represents watermarked text. For example, consider a passage with 10% watermarked tokens and 3 insertions. If the original text is 1000 tokens, then this means that 3 watermarked spans of tokens would be inserted into the enclosing chunk of human text. We give this setting the short name "CP-3-10%." which is an abbreviation of "Copy-Paste with 3 spans at 10% watermarking."
Returning to Figure 2, we see that with 25% watermark remaining, the copy-paste attack procedure has much stronger effect on the watermark than the other two machine based attacks, dropping the AUC to below 0.7 for 200 tokens and below 0.85 for 600 tokens. Examining Figure we congruently see that the watermark detectability grows more slowly than in the unattacked setting, but grows steadily nevertheless. We revisit this reliable growth behavior in Section 4.4 where we compare watermarking to alternative detection methods.

---

Published as a conference paper at ICLR 2024
4.3 PARAPHRASING BY HUMAN WRITERS
Humans may paraphrase watermarked text to better fit the style or tone of an existing passage or to better suit the tastes of the user. These modifications are a key part of every-day interactions that human users have with generated text, and to be reliable an-the-wild, a watermark should be robust to these changes. However, in a more ad adversarial setting, a human may also paraphrase text with the deliberate goal of evading detection.
To test the feasibility evasion, we set up a human study. We recruit 14 experienced human writers (graduate students) who were presented with watermarked text. They were asked to paraphrase the text with the goal of removing the watermark while attempting to maintain passage length and semantic content. In addition to compensation for participation, top performers were awarded one of three $100 gift certificates to incentivize strong but faithful paraphrasing. Each human writer is given a different set of text passages, which we generate by prompting watermarked version of Vicuna with the LFQA data.
We first validate that all human writers successfully paraphrased the text passages using P-SP (Wiet- ing et al., 2022) scores. Doing so, we find P-SP scores far exceeding the threshold of 0.7 considered an expert paraphrase in Wieting al. (2022). We show this score for each writer in Figure (left), comparing P-SP directly to the z-score of detection based on their text. We then analyze the de- tectabilty of the watermark via z-scores the right plot of Figure 4. Here, we show watermark strength as function of T, i.e. of tokens seen during detection. While human writers are strong paraphrasers, they cannot escape the two observations posed Section 4. As shown in Figure (right), eventually, the evidence for watermark mounts and even human writers are clearly de- tected on average after 800 tokens.
Witeraaaia Strength (2.Secre) vs Simalarity of Text Beiore and After Paraphrasing (P
Z: Z-Seoen of Wistenmeleed Texs after Panaphincing Telmes Olservvd
0.50
0.50
0850
100
1000
2-8cee
Thoms (T)
Figure 4: (Left) Paraphrase quality for human writers measured P-SP. Note the y-axis. (Right) Summary results for each paraphrasing attack in aggregate, showing that human writers are stronger paraphrasers than both machine attacks. Yet, all attacks can detected with certainty after 400 to 800 tokens.
4.4 A COMPARATIVE STUDY OF DETECTION ALGORITHMS
A number of alternatives to watermarks exist. As discussed in Section 2, other paradigms for LLM detection are post-hoc detectors based on statistical features, black-box learned detectors, and re- trieval systems. Based on previous work in Mitchell et al. (2023), we study DetectGPT as rep- resentative for loss-based post-hoe detectors, RADAR (Hu al., 2023) for detectors based on leamed classifier, and we use the retrieval method proposed by Krishna et al. (2023). We briefly describe the methods below and provide further details and hyperparameters in Appendix 1.1..3.
Retrieval: A retrieval approach to detection requires the creation and maintenance of comprehen- sive database of all sequences previously generated by the language model to check against at test time and adopt the retrieval method of (Krishna et al., 2023), utilizing the BM25 search method since it performed the best in their experiments.
To develop a learned classifier approach that is robust to paraphrasing, Hu al. (2023) propose RADAR, a framework for training robust Al-text detector using adversarial learning. We use the publicly available model, RADAR-Vieume Vieuna-BB, where the discriminator has been robustly optimized to detect Vicuna generations. Due to 512 token limit of the model, all sequences are always truncated to that length even if they longer.
RADAR:
7

---

Published as a conference paper at ICLR 2024
DetectGPT: DetectGPT (Mitchell et al., 2023) is post-hoe method for detecting machine- generated texts that employs a curvature- based criterion which compares the log probabilities of candidate passage and perturbed versions of the same passage. We use the official implementation of DetectGPT with its default setting of T5-3B (Raffel et al., 2020) as the mask-filling model and utilize their best performing hyperparameter settings.
Which method is most reliable? We take these alternate detection approaches and attack them with the battery of machine attacks described in the previous section. We plot ROC charts for each detection scheme when evaluated on attacked texts in Figure 5. We find that retrieval and watermark- ing are both decent detection methods, RADAR is reasonably effective in the machine paraphrase settings, and all are significantly more reliable than DetectGPT. Yet, while retrieval equals or out- performs watermarking on the GPT and Dipper model based paraphrases (corroborating the results in Krishna et al. (2023)), it and RADAR are bested by watermarking in the copy-past setting when analyzing performance in the range of FPR < 0.1.
Dipger
CP-1-25%
CP: 0P%1-10%
1.0
1.0
1.0
0.8
0.8
0.5
0.0
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10-2
0.0
0.3
1
FPR
F70
FPR
GPT
CP-3-25%
CP-3-10%
1.0
1.0
1.3
0.8
0.8
0.6
0.4
0.2
10-2
10-1
000
10-3
10-2
10-1
PPR
FPR
PPR
WinMac
Z-Sccee
Retraona
RADAR
DeseeGGP
Figure 5: ROC charts for various machine attack types for = 1000. For Dipper and GPT, the length after attack decreased to 800 and 300 respectively. We focus on the low FPR region by log scaling and limiting the x-axis to the interval [0.001, 0.1]. z-score and WinMax denote the two watermark detectors, "Retrieval" is the method of Krishna et al. (2023). (Left Column) The two machine paraphrase attacks, (Center Column) The copy-paste attack with remaining detectable text percentage of 25% spread over 1 and 3 segments. (Right Columns) The copy-paste attack at only 10% remaining detectable text. While DetectGPT performs poorly in all cases, Retrieval is quite robust to the machine paraphrase attacks, and RADAR performs well against the GPT paraphrase attack. However, watermarking outperforms both Retrieval and RADAR under the copy- paste attack, with the WinMax detector variant performing significantly better the most severe copy-paste settings.
passage with length before attack of
The Sample Complexity of Various Detection Schemes. To hone in the observed differences between the schemes under each attack setting, in Figure 6 we also compare all three approaches in terms of AUC @ T, i.e. detection performance as function of text quantity. We note the details of how Retrieval, DetectGPT, and RADAR were evaluated at varied values of T'' in the appendix.
Here, start to develop an explanation for why watermarking outperforms other methods in this specific setting by observing that scaling behaviors for each delection method differ starkly under attack. While all approaches grow in sensitivity as function of T in the weaker setting. under strong attacks, only the WinMax watermarking scheme reaches desired levels of reliability. In particular, when only a small fraction of text (CP-X-10%) remains under the copy-paste attack, non-watermarking methods struggle as text length increases. For the Retrieval and RADAR meth- ods, as T' increases, the ratio between tokens coming from the original unattacked text (positive), and tokens in the surrounding text (negative), decreases. Therefore, the overall similarity to the original example continues to drop, which eventually causes a decrease in performance for both paste attack severities. For DetectGPT, the same effect is observed but at more drastic levels. We investigate these non-monotonic behaviors in more detail in Appendix A.12 by examining trends in the actual Retrieval and DetectGPT scores produced under each attack setting.
8

---

Published as a conference paper at ICLR 2024
Desection Rate after Dipoer Minck va Tolers Oloerved
Detection Rate after GPT Atinck W. Tolem Ocserved
1.7
0.2
115
0.05
0.2
0.31
Telons
Toleas OT)
Deloction Rale af CP-1-2576 Minck Tolena Olaorved
Detaction Rain alser CF-3-25% Atrack Tolema Giserved
0.8
11.5
0.6
ROC-LUC
X1.5
0.4
00.4
Toleno
afre
Atock Tobens Obeerved
Detectir Rate Atracs Observed
800--000
6.6
BEC-MLC
0.5
8.4
0.0.
0.8
0.5
0.0
10c
40
600
830
1000
0.0
700
10
00
830 1000
Tolesa (T)
Tokca (T)
WinMace
Z-Scoee
Retraonal
RADAR
Dese
Figure 6: AUC at T for various types of attack. (Top Row) The Dipper and GPT machine paraphrasing attacks. (Center Row) The copy-paste attack with a remaining detectable text percentage of 25% spread over 1 and 3 segments. (Bottom Row) The copy-paste attack at only 10% remaining detectable text. While all schemes are impacted by the various attacks, DetectGPT scales particularly poorly under attack. Retrieval and RADAR both show positive trend in detectability non-monotonie, whereas watermarking steadily improves in power all attacks evaluated. Due tendency of the GPT and Dipper paraphrase models to produce shorter sequence of tokens as output than they receive as input, we make the curves translucent starting at the mean sequence length of the attacked texts indicate that these mcasurements For DetectGPT and RADAR, the last measurement for each attack type just computed
function of T, but their growth is slower
based inereasingly fewer samples and therefore more uncertain.
the of full result at that average value. For the
sequences (rather than rolling as with watermarking), and so we plot Dipper attack this ~ 800 and for the GPT attack this is ~ 300. Due to the synthetic nature of the Copy-Paste attack, after attack, those sequences are still full length i.e. 1000 + 25. However, for RADAR, we always only analyze up T' 512 due to the model's context limit.
CONCLUSIONS
Through a comprehensive empirical investigation, including strong machine paraphrasing and hu- man writers, evaluate the reliability of watermarks mechanism for the documentation and detection gencereegeneratted text. We advocate for view watermarking reliability as a func- tion of text length, and find that even human writers cannot reliably remove watermarks if being measured at 1000 words, despite having the goal of removing the watermark. This view of water- mark reliability as function of text length turns out to be strong property of watermarking, When comparing to other paradigms, such as retrieval and loss- based detection, we do not find a strong im- provement with text length, making watermarking out to be the most reliable approach in our study.

---

Published as a conference paper at ICLR 2024
6 REPRODUCIBILITY STATEMENT
In service of reproducibility, we have included all key details necessary to understand our experimen- tal setup. In sections of the main work where space was premium, we included pointers to sec- tions in the Appendix that provide further information. All models and data sources utilized are pub- licly available and the compute infrastructure used was based on commodity-level CPUs and GPUs running open source software. The implementation is designed to be both useable and extensible for further research and can be found github. com/ http://wwiihhbeeeeeeeeee 1amm--atttt rmar king.
ETHICS STATEMENT AND DISCUSSION OF SOCIETAL IMPACTS
In this work we conduct human study where we collect written annotations and preference determinations from study participants. Before conducting this portion of the research, we submitted a proposal to our Institutional Review Board office and received an "Exempt" status. We include all details of the human study setup and evaluation procedure as well as description of the compensation provided to participants. Additionally, the general topic of Al-generated content detection is an important domain of study with regards to societal trust and safety especially in online spaces. We conduct this research with an awareness of the impact that results on these topics can have if misrepresented and are careful report our findings in accurate and impartial manner, and we include a careful discussion of societal implications below.
While work focuses on technical aspects of the reliability of machine--ennrrate text detection, these findings have further societal implications. A strong finding that our study supports is that the problem of detection is simply made tractable through watermarking. Compared other ap- proaches, we find watermarking to be most reliable in everyday scenarios, where text is taken from generative model, modified by human writers or other models and inserted into spans of human- written text. This suggests that watermarking may be promising strategy to reduce harm arising from the use of large language models.
In pursuit of harm reduction through technology like watermarking, it is crucial to consider several precise details. First, from previous studies, i.e. Kirchenbauer et al. (2023), we know that water- marks can be broken by sufficiently motivated attackers, e.g. using generative attacks. Nevertheless, our results among others show that watermarks can be used to reliably track machine-generated text as it distributed over the internet in non-adversarial scenarios. From this perspective, watermark- ing technology flips the problem on its head: whereas before, machine- -eteineegeeerteedd texts were mostly undetectable by default, with advanced watermarks, active effort is required to make the text unde- tectable. Second, while technologies like watermarking will not be perfect, to be deployable in the real world, detection strategies must guarantee low false-positive rates, which have to be weighed against detection sensitivity, as every false positive can be extremely harmful. Other approaches like some of the post-hoc detectors we study cannot consistently provide low false positive rates in all domains, and thus can fail, for example when classifying text from non-native speakers. Third, For example, in classroom settings, false accusa-
whether harm is reduced is geenario-dependers tion is serious, whereas even positive detection results have only minuscule benefits to the learning experience of students. On the other hand, solutions that detect fake news and spam on social media platforms that might include watermarks as one additional feature in larger system, might trade relaxed error tolerances for clearer expectations through terms of service that communicate both its utility and potential limitations.
ACKNOWLEDGEMENT
This work was made possible by the ONR MURI program, DARPA GARD (HR001 12020007), the Office of Naval Research (2000142112557), and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS 221282), and by the NSF TRAILS Institute (2229885).
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Sahar Abdelnabi and Mario Fritz. Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding. In 202/ IEEE Symposium on Security and Privacy (SP), pp. 121- 140, May 2021. doi: 10. 10.1109/SP4000111 0000////44000022...0000
Mikhail Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural Language Watermarking: Design, Analysis, and a Proof-of-Concee Implementation. In Ira Moskowitz (ed.), Information Hiding, Lecture Notes in Computer Science, Pp. 185-20 255-000, Berlin, Heidelberg, 2001. Springer. ISBN 978-3-540-45496
0. doi: (0.100773-540-44496-
Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying real or fake articles: Towards better language modeling. Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-11, 2008.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the
Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 610-623, New York, NY, USA, March 2021. Association for Computing Machinery. ISBN 978- 1 145503- URL hwwpt:ttp//cc....t //opis:///d....// http::::/doi..r//////0 wrg/10..//00. 11w.1114540000 3442144441..
8309-7. doi: 10.1145534421888.44448 3445922.
Daria Beresneva. Computer- computer-generated detection using machine learning: A systematic review. In 21st International Conference on Applications of Natural Language to Information Systems, NLDB, pp. 2214426.55 Springer, 2016.
Meghana Moorthy Bhat and Srinivasan Parthasarathy. How Effectively Can Machines Defend Against Machine-Generated Fake News? An Empirical Study. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pp. 48-53, Online, November 2020. As- sociation for Computational Linguistics. doi: 10.18653/v1/22 1it...0.iiights-1..... 1.7. URL https:
//aclanthology wrtpgy.org/200000 2200..2220000 insiggtss 1.77.
James V. Bradley. Distribution-frem free Statistical Tests. Fort Belvoir, VA, August 1960. Defense Tech- nical Information Center. doi: 10.21236/AD0249268 URL http: /www. dtic. ca....m.. mi 1/docs/
citations/ADo24922
Jack T Brassil, Steven Low, Nicholas F Maxemchuk, and Lawrence O'Gorman. Electronic marking and identification techniques to discourage document copying. IEEE Journal on Selected Areas in Communications, 13(8):1495-13) 1504, 1995.
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the Possibilities of Al-Generated Text Detection. 0tti:23304.04736/cr April 2023. doi: 10.48550//cxxv.2334.000
URL http: http//aaxvv.v//////////e////.av/ org/ abs/ab/s//20// 04736.
Yuei-Lin Chiang, Lu-Ping Chang, Wen-Tai Hsieh, and Wen-Chih Chen. Natural Language Wa- termarking Using Semantic Substitution for Chinese Text. In Ton Kalker, Ingemar Cox, and Yong Man Ro (eds.), Digital Watermarking, Lecture Notes in Computer Science pp. 129-140, Berlin, Heidelberg. 2004. Springer. ISBN 0--8---500-242244- doi: 11110009978-3-54-24222222 10.
William G Cochran. The x2 test of goodness of fit. The Annals of mathematical statisties, pp. 315-345, 1952.
Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. arxiv:2210.07:2]((ssss. November 2022. doi: 10.48550/arxiw.21100077
URL http::ttt/// http::///aaiii.oor//aab///////// http://wwww.o/////////..../. 2232212211...11....
Falcon Dai and Zheng Cai. Towards ineereeppeeeeee Steganographic Text. Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistrcss pp. Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P 1422. URL https: / //clleaaeeeeee antho ogw..org//199112...
Florence,

---

Published as a conference paper at ICLR 2024
Tina Fang. Martin Jaggi, and Katerina Argyraki. Generating Steganographic Text with LSTMs. In Proceedings of ACL 2017, Student Research Workshop, PP. 100-106, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https: P17-3017.
/ttps:///ccaaatoooooo http///acaaattttoooo/oooo o
Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. arXiv preprint arXiv:2308.00/ 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00021 [cs], December 2020. URL http::/tp//// http::://xxxvvv.ro.e
/ 21 01.01....0. 00027.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and Visualization of generated text. arXiv preprint arXiv:1906.04044 2019.
EA Grechnikov, GG Gusev, AA Kustarev, and AM Raigorodsky. Detection of artificial texts. In RCDL2009 Proceedings, pp. 306-308, 2009.
Alexei Grinbaum and Laurynas Adomaitis. The Ethical Need for Watermarks in Machine-Generalee Language. axxiv:2209.03//88/cs)) September 2022. doi: 10.485500amxi.22000001116 URL
http:://axtt org. /200 03118.
Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. Protecting In- tellectual Property of Language Generation APIs with Lexical Watermark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10758-10766, June 2022a. doi: 1.1100//aaai.v66i1022222 article/view/21322
URL https://ojs.ant aaai..a......... index. php/AAAI/ AAAI/
Xuanli He, Qiongkai Xu, Yi Zeng. Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks. In Advances in Neural Information Processing Systems, October 2022b. URL https:// openreview n Porummid-11PPIIssoXX
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1904.00 097511 2019.
Xiaomeng Hu, Pin- Yu Chen, and Tsung- Yi Ho. Radar: Robust ai-text detection via adversarial learning. arXiv preprint arXiv:2307.03888 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Gabriel Kaptchuk, Tushar M. Jois, Matthew Green, and Aviel Rubin. Meteor: Cryptographically Se- cure Steganography for Realistic Distributions, 2021. URL https: :tap:s://eprriie iacr .rrg 2021/ 668.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A Watermark for Large Language Models. arxiv2230.10226/ccs January 2023. doi: 10.48550/ 1.X2.23011.0026.
10226.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of Al-generated text, but retrieval is effective defense. ax31v22333114088cc.) March 2023. doi: 10.485550arxXiv2230.. 13408. URL htttp::/aaxxiv.orrgg/ wttp://///////../////////
13408.
Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative entropy scoring. PAN, 8:27-31, 2008.
Taehyun Lee, Scokhee Hong. Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305. 15060, 2023.
12

---

Published as a conference paper at ICLR 2024
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Perey Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXhv:2210.15095 2022a.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive Decoding: Open-ended Text Generation as Optimization. arxiv:2210.150097/1 October 2022b. doi: 10.48550/arX1.22210.1500 URL hettp://arxv..oggee
2210 .2005. 15097.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou, GPT detectors are biased against non-native English writers. wxxiv2234.02889/ccc April 2023, doi: 10.48550/arXiv.2304 02819. URL /estp://arxxe org/ attp///////300 02819.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. Jan- doi: 10448500WrXix.220011133 URL. /ttp:://aaxxvvoo/ /ttp////i...og/b////0/0000
uary 2023.
11305v1.
Annie Palmer.
People are using A.I. chatbots write Amazon reviews.
CNBC, April 2023.
URL https:/ //www cnbe. wwm.c223/0//2233333
amazon-reviee are beteeebbeinn-witteneneee wwwiittee----ai-cheee www.aaaiccaatts.thhhaaa html.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. In Advances in Neural Information Processing Systems, volume 34, pp- 4816-4828. Curran Associates, Inc., 2021. URL http:: httttss:/prooeeediinn new.riisssnnnnnnns....e.... cc/ 8a4 Abstract ctttarttttt.tttt...aaa.......... .htttrrcct..ml........
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. arxiv:2211..0335fccs eess), De- cember 2022. doi: 10.48555/arXi..2222..333 04356.
URL http:/ http./rgg/bbb///////////////
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to- Text Transformer. arXiv: 10683 [cs, star), July 2020. URL http://://axxt org/ abs/ 1910. 10683.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can Al-Generated Text be Reliably Detected? 1rxi::233.111556// March 2023. doi: 10.
48550%arXiv.2333.1115 URL http: http:///axivv wwt./ogg//////////e/ ab////////3333 11156.
Bruce URL
Schneier.
August 2021. https: / www. scwneier.com http///llgg/aacceee/////// www-spparrphisiinge html.
Using AI to Scale Spear
Phishing,
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross An- derson. The Curse of Recursion: Training on Generated Data Makes Models Forget. (mx4:/2305.17493[cs), May 2023. doi: 10-48500aaxxiw22355.74933 URL http:://axiv.. bog/abs//20 22t..2335..0 17493.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A Contrastive Framework for Neural Text Generation. In Advances in Neural Information Processing Systems, October 2022. URL https: /pppneeieeeeeeeeeeeee .wt.//eeeeeeeeeeee forum fett//orru??ddv// 1i= 21m
Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The Science of Detecting L.MM-Generaeed Texts. arxiv:2303.077 (7700//sss March 2023, doi: 10445550/arxiv22000.77335 URL http://arxiv.. org/abs 2303. 07205.
Edward Tian. Gptzero update v1, January 2023. URL https::///// http:::::///ppptrroouuustttteett p/opttzero-update--v.
13

---

Published as a conference paper at ICLR 2024
Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Stu- dio: Data labeling software, 2020-2022. URL bttps://githhi www//eaatteeaabbeeeeeeeeeo.aa label-studio. Open source software available from https///gihuub.ommmbetttttssssas studio.
Umut Topkara, Mercan Topkara, and Mikhail J. Atallah. The hiding virtues of ambiguity: Quan- tifiably resilient watermarking of natural language text through synonym substitutions. In Pro- ceedings of the 8th Workshop Multimedia and Security, MM&amp:Sec '06, pp. 164-174, New York, NY, USA, September 2006. Association for Computing Machinery. ISBN 978- 908997988880 1-59593- 493-2. doi: 10.1145/161166.111119)) URL https: /top:://doo... wwwp//...//...... / 1161397.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2337.099288 2023b.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https: https:///gihhs com/ wtt.cccm//////ee httph//gggggggeee//g////g// alignment-Bnl hangentttaanooooo 2023.
Lav R. Varshney, Nitish Shirish Keskar, and Richard Socher. Limits of Detecting Text Generated by Large-Scale Language Models. 2020 Information Theory and Applications Workshop (ITA), pp. 1-5, February 2020. doi: 10. 109/015000566200000 202.099512...9
Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. Watermarking the Outputs of Structured Prediction with an application in Statistical Machine Translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp- 1363-1372, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguistics.
URL
Vicuna-Teams Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL http:/:::///mmssss/////// 30tp:///2220000000010000---0
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.0433 2019.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor borg-kirkpatrick. Paraphrastic Repre- sentations at Scale. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing:
System Demonstrations, pp. 379-388, Abu Dhabi, UAF, December 2022. Association for Computational Linguistics. URL https: http::s//acaaatthlllogo/o wwwp:/lloooooo.g2222// 2022.
emn.p-ddmme 38.
Max Wolff and Stuart Wolff. Attacking Neural Text Detectors. wrwp:/s........0000//////////// January 2022.
doi: 10.48550/a2Xiv.2002.111 URL
Claire Woodcock. Al Is Tearing Wikipedia Apart, May 2023, URL httpt::ttp//wttt http://wwwwwiccccooooo com/ en/ attaaaaaleee/////aaaaaaaaaaaeaae tearinntttrnng-oe wiki widipeeia-----i -part...aaaa.............aa
KiYo Yoo, Wonhyuk Ahn, Jiho Jang, and Nojun Kwak. Robust natural language watermarking through invariant features. arXiv preprint 27599:3305.0/904, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models. arxiv:2205.01068] May 2022. doi: 10.48550/ar 2205.0 0668.. URL http: /ttp///wavvvv///////////// www.ii.oorg/aaeo httpaaabb///2..222 /2002 01006.106
14

---

Published as a conference paper at ICLR 2024
Zachary Ziegler, Yuntian Deng, and Alexander Rush. Neural Linguistic Steganography. In Pro- ceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCLLP) pp- 1210-1215, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. N0.18655/1/DD19-1115 URL httph:ttts//eeeee http://aaaaaatlllloooo httph///oooo/g/////11555
15

---

Appendix-Tabee Table of Contents
A APPENDIX
We provide number of extended sets of visualizations and ablation studies to supplement the results shown in the main body of the work well more methodological and experimental details. Below is a table of contents to help navigate the various subsections.
Table of Contents
1. What about the White-Box Setting?
2. Relationship to Theoretical Results on (Im)Possibility of Detection
3. On the Reliability of P-Values
4. Utilizing Better Quality Metrics
5. Hashing Scheme Extended Ablation
6. Effect of Watermarks on Utility: TriviaQA
7. Datasets and Models Ablation
8. GPT Attack Prompt Ablation
9. Watermark Detector Ablation
10. Detectability of Watermarks after Machine Paraphrasing: TPR @ T
Experimental Methodology Details
12. Detection Method Comparison: Detector Scores T
13. Human Study Details and Preference Evaluation
14. Code Details and Release Statement
15. Hardware and Compute Details
16

---

Appendix-Taiee Table of Contents
A.1 WHAT ABOUT THE WHITE- BOX SETTING?
In this work we have focused on the black-box setting, where text is modified in plausible use cases for machine-generated text and the secret key of the watermarking scheme is unknown the party modifying the text through paraphrasing or other editing.
Once this assumption relaxed, for example the secret key is breached the watermark is public, then an attacker with white-box access to a strong paraphrasing model like Dipper (Krishna et al., 2023) could break the watermark in the following way: The attacker can apply an anti-watermark scheme during generation with the paraphrasing model, where instead of adding 8 in Equation (1) logit outputs, instead subtracted. The attacker can further keep track of the current score of green-listed versus red-listed tokens and can accordingly modify this negative å on-the-fly to guarantee that of the 7 tokens of text generated by the paraphrasing model exactly yT are colored green, and no watermark signal is leaked into the paraphrased text.
This attack relies on both white-box access to the watermark key and availability of a strong para- phrasing model. To bypass this difficult threat model, the watermark context width h has to be chosen sufficiently large so that in an adversarial scenario, the watermark key could not be easily discovered (or alternatively, sufficiently many keys must be employed simultaneously, discussed in Kirchenbauer et al. (2023)). Nevertheless, not all watermarking use cases are necessarily adver- sarial, and we strongly believe that in benign cases, text corruption is still very valuable. Documenting the usage of machine- machine-generated text overall, for example either to trace the spread of generated text in hindsight, or to remove generated text from future training runs (Radford et al., 2022; Shumailov et al., 2023), can provide a baseline effort to "future-par prooof popular generative models.
watermark with imperfect resistance to
17

---

Appendix- Dopel
Table of Contents
A.2 RELATIONSHIP TO THEORETICAL RESULTS ON (IM)POSSIBILITY OF DETECTION
Several works have investigated the difficulty of detecting language models from a theoretical al., 2023) and practical (Bhat & Parthasarathy, 2020; Wolff & Wolff, 2022; Tang et al,, 2023) perspective, although these works mostly pertain to post-hoc detectors. How does this body of work relate to our empirical evidence on the reliability of watermarking?
(Varshney et al., 2020; Sadasivan al., 2023; Chakraborty
In their work on the impossibility of LLM watermarks and detection, Sadasivan et al. (2023) assume the goal of detection is to discer text generated by large language model from text generated by a randomly sampled human from some population, for example tweets from Twitter users. Sadasivan et al. (2023) also assume that the goal of large language model training is to mimic such a human language distribution. If the LLM perfectly mimies this distribution, then its samples will be indis- cernible from the human generations. To the extent that the LLM imperfectly mimics the human distribution, Chakraborty et al. (2023) prove that the distribution shift between the language model and
text can be detected, given sufficient number of samples. If an LLM instead followed a more concentrated distribution over text, for example by routinely speaking in the voice of a particular individual rather than randomly sampled one, then its samples would be unlikely under the generic human distribution and would therefore be detectable.
Existing literature suggests that LLMs trained with standard methods do not mimic randomly sam- pled individuals- - the standard classification loss used to train LLMs is known to reward a low en- tropy output distribution that distinct from typical human text (Holtzman et al., 2019; Gehrmann et al., 2019). Furthermore, benign users, and by extension companies that sell LLMs as a service, seldom want an LLM to mimic a generic human language distribution, and may instead prefer inhu- manly understandable and factual text, or text in a polished professional voice.
Regardless of whether future LLMs mimic a human distribution, watermarking is still possible. Consider that generic human language distribution is incredibly diffuse, with many valid comple- tions for a single prompt. For example, different mathematicians write unique yet correct proofs of the same theorem, even if their logical steps are identical, indicating that even tasks with seemingly narrow answers, like math, may actually still admit diffuse human language distributions.
The high entropy over completions enables watermarks to concentrate model outputs on subset of valid completions while still spreading mass across diverse and high-quality candidates. In fact, watermarks which only make very small changes to the generative distribution can be detected with enough samples, as proved theoretically in Chakraborty et al. (2023) and verified empirically in our own work. Moreover, the benefit of watermarking is that we can minimally change the generative distribution in a way that optimizes detectability.
Sadasivan al. (2023) also state that, in principle, watermark can be removed by paraphraser that samples from the set of text with the same content. Such theoretically optimal paraphrasers have so far not been demonstrated. Our experiments show that even stronger models (chatGPT) may be insufficient for paraphrasing weaker models (LLaMA-7B), demonstrating that watermark detection possible when contending with the paraphrasers that exist today.
Ultimately, the theory literature discussing differences between language distributions does not get the core of the detection problem. Existing post-hoe detectors do not fail because the distributions of human-writn written and machine- -eeerrteee text are so similar, but instead because we lack a mathe- matical characterization of their differences (Liang et al., 2023; Krishna al., 2023). Consider, for example, an LLM with pseudo-random sampler and a fixed seed. Each token is a deterministic function of those that came before it. The output distribution is concentrated on a single example conditional each prompt, making it very different from a human distribution. Yet, without white- box model access, detection is still difficult in practice because we know neither the location of this peak for the L.LM nor the distribution of human text. In contrast, watermarking is effective not because the distributional shift it induces is large, but because this shift is characterized by a simple rule. The human- written and secchinn-ggnerateee text distributions were likely very far apart before watermarking, but characterization of the differences is needed for detection.
18

---

Appendix- -apble Table of Contents
A.3 ON THE RELIABILITY OF P- VALUES
In their original work, Kirchenbauer et al. (2023) discussed the impact that repeated n-grams in a given piece of text would have on the validity of the independence assumption of the z-test. While we do not utilize p-values in our analyses, and instead consider empirical FPR and TPR rates, we run some experiments to evaluate the significance of the impact that counting or not counting repeated n-grams at detection time can have Concurrently, the work of Fernandez et al. (2023) presents a more in depth analysis of this effect on the watermark of Kirchenbauer et al. (2023) as well as other watermarking techniques with detection algorithms based on hypothesis tests. We find that while the p-values produced by the uncorrected version of the detection test suggest a lower FPR than we actually measure at given z-score threshold, the correction for n-gram repeats improves the correspondence between the empirical FPR and the p-value significantly.
Validation Empitieal FPR seninst P-Value e. funeliva of Z-Scum Threshold for Wikiled 100
Z-Scure Row'dd to achiee O I.I9P w. Tolonw for Wilduwe115
0.2
0.4
03
2:Bere
0.
0.1
0
01
1001 Tiloem (T)
115
150
170
1810
Figure 7: The left chart shows the observed FPR as function of different z-score cuttoffs along with the p-value (in black) corresponding to a one-tailed z test. The other curves show that the basic LeftHash scheme (blue) produces slightly higher FPR than the corresponding p-value, but that the correction based on not counting repeated n-grams during detection (green) fixes the error underestimation. The right chart shows the Z-score threshold required achieve FPR of 0.1% as a function of the length of the text in tokens. For small values of T. the normal assumption is slightly violated, as the z-score required is closer to 4 whilst the z-score that should produce p-value of 0.1% = 0.001 is actually 3.09 (black). As the sequence gets longer we see that the threshold required by the uncorrected version of the LeftHash (blue) scheme remains above the analytical value, but that correcting by not counting duplicate n-grams detection time (green) causes the z-score threshold correctly settle around the expected value 3.09
Comparisu between mam samples w// p pallte P N' for N 1000 somples from W0 (T-203)
300
400
300
800
300
83
0.1
Figure 8: A plot of the correspondence between the empirical fraction of sequences whose p-value is lower than a given probability P. and the quantity N x P. While the uncorrected version of LeftHash incurs false positive rate (y axis) slightly larger than N + P given threshold P, the correction of not counting repeats fixes this issue.
19

---

Appendix Copeleix-Table
Table of Contents
A.4 UTILIZING BETTER QUALITY METRICS
To more accurately examine the effects of watermarking, in addition to utilizing a stronger genera- tive model from the alama family (Touvron al., 2023a), we employ pair of metries designed to capture different aspects of generation quality. Given the fraction Ui of unique n-grams sequence text, we define diversity up to order N via
diversity = - log
Un
(3)
represent view on n-gram repetition metries described in Welleck al. (2019) and Li et al. (2022a) in more readable format. A higher diversity score represents more diverse text, where fewer n-grams are repeated.
To estimate whether watermarked text drifts away from un-watermarked model generations, we adopt the same evaluation metric as Krishna SP (Wieting et al., 2022). We measure similarity between different sets of text pairs such un- watermarked and watermarked outputs, or the human gold completion to prompt versus the wa- termarked completion generated by the model. Further, we evaluate human annotator judgement of un-watermarked versus watermarked text quality (Table 5).
(2023) measure paraphrase similarity: P-
We also considered other metrics for language model sampling quality such as MAUVE (Pillutla et al., 2021) and coherence as described in (Su et al., 2022; Li et al., 2022b). However, the in- sight those measures provided when comparing outputs under various generation and watermarking settings was generally subsumed by what the P-SP metric revealed, so we chose to simplify presen- tation using just P-SP as our semantic similarity metric across all relevant visualizations. Aside from the study of watermarks, we note that output quality evaluation in open ended generation settings is still research area with many open questions.
20

---

Appendix Table of Contents
Testection Rate Contest Width
Loy, Diveniity Conrett Width
1000 ACA
Drvendy
0.55
0.7
Men beilbok
Sie letlinn
Chankot Willh
Clustd Willh
Figure 9: Effect of context width watermark robustness and diversity. (Left) Effect of context width watermark robustness as measured by ROC-AUC after a paraphrasing attack by GPT. For larger context widths Skip and Min variants provide the best detection strength. (Right) Effect of the seeding scheme context width on the quality of the text as measured by log diversity. A small context width produces less diverse outputs for all three schemes, and the Addi ivv and Skip schemes produce more diverse text at larger context widths than the Mi n scheme. (Both) Watermark parameters 6 fixed (0.25, 4.0). The black circle marks the simple Addít invitiiveeeeee lt Hash with context width scheme, the brown circle marks the width h = variant of the Mi n-Sel FHash scheme, both evaluated throughout the work (names shortened to "LeftHashh and "SelfHashh" respectively).
A.5 HASHING SCHEME EXTENDED ABLATION
When the context width increased to maintain secrecy of the red/green list rules, we find that detection reliability substantially depends on the hashing scheme. We define the following functions No - N that map span of tokens { onto a peeuoo-random number. Each depends on secret salt value E N and standard integer PRF P N + N.
Additive: This is the function described in Kirchenbauer et al. (2023). We extend it to by defining Jadast tveattive-teeteaae (z) P Σ æx). While permutations of the context do not change the outcome, removing or swapping single token from æ changes the hash and hence breaks the watermark at this token.
Skip: This function uses only the left-most token in the context: fakip- LeftBesh = P (sxx). This hash is robust to changes in the non-leftmost token, but it is susceptible insertions/éelltions.
Min: This function is defined by fxin LeCLnssa (x) min P (sr:). It is robust to permuta- tions within the context and it is partially robust insertions/delltiinns. Given that all P (s2;) are pseudo-random and equally likely to be the smallest value, the likelihood of failure of this scheme proportional to the number of values removed from the context, i.e. if h 4 and 2 tokens removed/missing from the context, the PRF is still 50% likely to generate the same hash.
Choosing Scheme. Figure 9 shows that a small context width h provides the best robustness to machine paraphrasing, At wider context widths, Skip and Min variants remain strong under attack while Additive suffers. However, we see that this robustness improvement comes trade-off to text quality as the Min schemes produce less diverse outputs. Still, at a context width 4, the Min-Sel fHash scheme (brown circle marker) achieves the same diversity as the original Additive-Lets This shows that we can use the additional strength provided by Min and Se fash to run longer context widths, which in turn secure the watermark. We adopt these two schemes as "SelfHash" and "LeftHash" respectively the sections of the main work.
scheme at width h = 1 (black circle). while being more robust.
In this section, complete representative selection of settings varying different components of the hashing scheme to explore different parts of the pareto space between watermark strength and text quality. We further detail our proposed extension of the Sel fHash algorithm to arbitrary sampling schemes and pseudorandom functions ff in Algorithm 1.
extensive study of watermark hyperparameters by presenting
When using greedy decoding, the scheme in Algorithm covers the scheme denoted as Alg.3. in Kirchenbauer et al. (2023). We also note that in practice, iterating over all k [V] is strictly
21

---

Appendix- Copeleix-Table of Contents
Algorithm 1 Generalized Sel fäash Watermark
Input: Context 2k.... 21, vocabulary V, arbitrary text generation scheme S, LLM logits | Watermark hyperparameters 7 E (0, 1), > 0. S.h 0, integer hash P
G= = 0
Initialize empty set of green-listed tokens
for L, I..... V do
H₂ = P(k)
Compute k-th key Temp. green list G3 seeded with Hş
G₁ RandPermy, (V)
if k E G4 then
G GU G {k}
> Include & in final green list if self-consistent
end if
end for
L - 4/ +
if KEG otherwise
4
Sample a new token Zo from modified logits I using sampling scheme S.
necessary. We only iterate over the 40 indices with the largest logit score l and skip all others, for which the addition of 5 is unlikely to make impact on the final probability distribution.
We further note that the choice to set H3 = P(k) instead of of H₂ = [(ar, k)) in Algorithm is not strictly necessary. We refer to the first choice as anchoring ng and ablate it separately in Figure 12, where Hę = f(xx, k)) denotes the un-anchored approach. On all other occasions the SelfHash scheme is always anchored as described in Algorithm 1.
In Figure 10, to vary the strength of the watermark, we test a selection of 7 and 5 values in (0.5, 0.13 and (1.0, 2.0, 4.0) respectively. These values give us 9 settings representing both weak and strong watermarks that yield series of increasing z-scores across the z-axis for each seeding scheme. All points in these charts are averages computed on 500 samples with token length 7 = 200 + 25.
We observe that as the watermark is made stronger (resulting in higher 3-scores) certain schemes trend positively with respect to n-gram diversity of their outputs and others trend negatively. Namely, for the "Additive-Lefttlashh."" (i.e. using LeftHash with context width h = 1, and additive I') scheme that was evaluated in Kirchenbauer et al. (2023), stronger watermark strength yields less text diversity. This issue is remedied by using a larger context width, or by choosing the "aiip-LeftHash,"" scheme which exhibits improved text diversity higher watermark strengths. This finding provides another advantage for schemes with increased context width.
In Figure 11, we display the drift between unwatermarked and watermarked completions, mea- sured in P-SP. To put these numbers into perspective, we note that the drift between human text and unwatermarked text be estimated just under 0.50 PSP. This implies that for the watermark settings yielding z-scores up to 10.0 (a score that is extrememly detectable), the semantic diver- gence of the watermarked output from the model's unwatermarked behavior is less than the average divergence of the unwatermarked model from human gold completions.
In Figure from the main body we empirically demonstrate that the attack amplification effect, hypothesized in Kirchenbauer et al. (2023) to occur when using watermark schemes with context widths greater than 1, is made less severe when using Skip and Min based schemes. To get the full picture based on all degrees of freedom afforded in the hashing scheme space described in the main body, we provide a plot varying all parameters simultaneously in Figure 12. We confirm the prediction in Kirchenbauer et al. (2023) that the generalized version of the SelfHash algorithm, when applied to the Anchored version of HinHash scheme at moderate context width (i.e "Algorithm 3" from the previous work when the context width is h = 4), provides competitive tradeoff between robustness to attack and text diversity and quality along with the added benefit of being more secure against reverse engineering of the PRF key than the Additive-LeftHash
22

---

Appendix Table of Contents
Tapele
Watermark Strength (Z-Score) vs.
Watermarked Output N-Gram Diversity
Holh
9
Min-SolHHali3
Log Diversity (-)
Slop-Lofhlash.4
S
7
6-
5
2.5
5.0
7.5
10.0
12.5 Z-Score (+)
15.0
17.5
20.0
22.5
Figure 10: The pareto frontier with watermark strength (z-score) on the x-axis and text diversity (Log Diver- sity, see Appendix A.4) shown on y-axis. Higher hetter both metrics. see Mins and Skip-LeftHash schemes with larger context windows yield more diverse generations as watermark strength increases. The standard LeftHash scheme with context width I and the SelfHash based schemes produce lower diversity outputs under strong watermarking.
Watermark Strength (Z-Score) vs.
Similarity of Unwatermarked and Watermarked Outputs (P-SP)
0.55
0.50
0.45
P-SP
0.40
Additive AddiriveLeffillah. Lef:llash.
Min-Ledt Hosh. d Sin-Se
0.35
Min 3 Skip- Hoshh4
2.5
5.0
7.5
10.0
12.5 Z Score (+)
15.0
17.5
20.0
22.5
Figure 11: The pareto frontier watermark strength in z-score, shown on the z-axis, versus similarity between the un-wat un-watermarked and watermarked output, shown on the y-axis. Higher is better for both metrics. We see that the Min-LeftHash and Skip-LeftHash schemes with larger context windows produce watermarked generations with slightly higher semantic similarity their unwatermarked counterparts than the other schemes as watermark strength increases. However for all schemes, especially the SelfHash based settings, as watermark made stronger, the watermarked output diverges more from the unwatermarked output.
23

---

Appendix
Papele Table of Contents
Detection Rate vs. Context Width
1.0
0.9
0.8
ROC-AUC
Miditise-L
Selffas Seltfaahh
0.7
Auchored-Min Lan HefuH
Anchorod-Mim- Manardd----S-
Anchored Skip LettHash
Anchored-Shee Anchored-Skip--ooleee Salhhhh
0.6
Min-LLeftlahh
HellSSh
Skip-I Left Hash
Skip-SelfHash
0.5
1
2
3
1
6
7
Context Width
Figure 12: Effect of seeding scheme context width on watermark robustness as measured by ROC-AUC after a paraphrasing attack by GPT. In this variant, we ablate two more parameters, "anchoring" and "self-salting (SelfHash), The watermark strength parameters 5 are fixed at (0.25, 2.0), slightly weaker than the above bring in line with settings from figures in main work. We specially denote "Additive-LeflHash the black circle marker and also the Manchored-Min-Selfflee correspond to the "LeftHash" and "SelfHash" variants respectively shown in Figure 2, where we also reported the more complex watermark seeding scheme outperforming the standard scheme proposed by Kirchenbauer al. (2023). Here, it becomes clear that this is the result of the Anchored-MMi Mii--MMMi-aaae SelfHash scheme outperforming the others terms of robustness all context widths.
width at width 4 as brown circle, as
24

---

Appendix Paplle
Table of Contents
A.6 EFFECT OF WATERMARKS ON UTILITY: TRIVIAQA
Kirchenbauer et al. (2023) performed a small study on the impact of their watermark in knowl- edge intensive, factuality critical generation scenarios but only evaluated the impact of the LeftHash scheme. While we do not expect to see major differences the impact on utility for the SelfHash scheme (only improvements in reliability) for completeness, we explicitly verify this.
For this experiment, we generate free-form answers to the first 1000 TríviaQA questions. We use the slightly more advanced chat models LLaMA-2-chat-75 (Touvron al., 2023b) and Zephyr-7b-beta (Tunstall et al., 2023), finetuned version of Mistral-7B (Jiang et al., 2023). We score the responses as correct the string of the correct answer one of its aliases are included in the model's s response. We found the responses of the chat models to be too verbose to get consistent "Exact Match" scores reflective their capability. The results are shown in Table 1.
Overall, we find that the utility of the model is only minimally impacted by the SelfHash watermark scheme, with the original LeftHash watermark yielding similar results. This is in line with our expectations that the two schemes should behave similarly under this type of analysis. As shown Kirchenbauer et al. (2023), the strength of the watermark is quite low in these short, low-entropy generation scenarios, with the watermarked responses only producing marginally more significant detection statistics than the unwatermarked generations.
Watermarking Scheme
None
LeftHash
SelfHash
Llama2-7B-chat
0.523
0.517
0.524
Zephyr-7B-bete
0.616
0.612
0.607
Table 1: (Performance) TriviaQA performance under the "generation includes target?" scoring method.
Watermarking Scheme
None
LeftHash
SelfHash
Llama2-7B-chat
-0.068 -0.076
0.473 1.185
0.979
Zephyr-7B-beta
1.277
Table (Watermarking, z-scores) Strength watermark signal reported as average z-score produced when running the detector on the generated answers.
Watermarking Scheme
None
LeftHash
SelfHash
Llama2-7B-chal 0.528
0.379 0.237
0.273
ZBphyr-7B-beta
0.528
0.222
Table 3: (Watermarking, P-values) Strength of the watermark signal reported as average P-values produced when running the detector on the generated answers.
25

---

Appendix- Dabble Table of Contents
A.7 DATASETS AND MODELS ABLATION
In this section we present the results of an extended evaluation of watermark robustness to machine paraphrasing attacks across selection of domain specific datasets using the two models utilized in the main experiments alama and vi cuna and a similarly sized but older generative model from the Open Pretrained Transformer family, opt-6. 7b (Zhang al., 2022). For cross-reference, the black marker indicates the same standard model and data pair from the evaluations in the main work. The "Github", "Law", "Med", and "Patents" markers refer to the g github, free 1aw, pubmed, and uspto subsets of the "The Pile" (Gao et al., 2020) dataset as hosted on the huggingface hub at by EleutherAI. "Wiki" indicates samples
hugg buggingfett co/ dat dataset wleetherettttiiiie..eeeee from the training split of Wikitext103 dataset (Merity et al., 2016), also hosted on the huggingface -10--eeee-tt-rraa
hub at hugging
co datasets / wikitext as
that the Github subset yields the lowest s-scores relative to the number of tokens considered here, most likely a function of the restrictive nature of code syntax. Less flexibility at generation time due to restrictive prompts and domain characteristics results in lower average "spike-entropy" of the model's 's next-token distribution. This has a proportional effect on how strongly the watermark can be embedded, due to its adaptivity property (see Kirchenbauer et al. (2023) for a full analysis of the effect of entropy on watermark strength) and implies that under default settings, more text needs be observed detect the watermark. Upon manual inspection, the Law subset is also highly formatted containing special characters and whitespacing and this is a potential explanation for the low peak z-score (below 8.0) since the model is forced to follow syntax cues in the prompt to maintain a high level of language modelling accuracy. The Med and Patents subsets yield a wider range of z-scores across the three models the 8.0 to 10.0 range, and the C4-en and Wiki datasets produce 2-scores very similar to the C4-News split considered throughout the main work.
Generally, looking Figure 13,
Out of the three models evaluated, the vicuna model, which is supervised instruction-finetuee variant of 1lama, yields the lowest z-score for each dataset, suggesting that its output entropy is generally lower than that of the base 11ama model or opt in congruence with the fact that found higher delta value (4.0) was required to achieve suitably strong starting watermark levels in the human paraphrasing study (details in Appendix A.13).
While in Figure 14 we also present the combination of models and data but with a different metric along the y-axis, find that the appropriate takeaways mostly the same described above. Further, the semantic similarity (P-SP) values for the Github data domain are potentially miscali- brated since this metric was not designed for code similarity estimation. That said, we notice that vicuna and 1 alama achieve the same z-scores across Law, Med and Patents, but show more se- mantic divergence between the unwatermarked and watermarked outputs for Med and Patents than Law.
In Figure 15 we observe the effect of the different z-scores shown in Figure 13 and Figure 14 on the unattacked detection rate. The ROC-AUC is quite a bit lower for Github and Law than the other datasets. Then, in Figure 16 we see that this also translates into correspondingly lower detectabilty after attack by GPT and Dipper. However, we note that for the Med and Law subsets (yellow and green bars), detectabilty post paraphrase attack is quite similar to the C4-News domain (black bars) utilized throughout the main work. This suggests that those findings generalize well at least to language modelling domains with similar levels of syntactic ficxibility.
26

---

Appendix-Tabee Table of Contents
Watermark Strength (Z-Score) vs. Watermarked Output N-Gram Diversity
C4-Ne Llema-71
Law, Vieuna 70
Llenn 7B
Med.
Liki,
Mel,OPPT-6.79
Citlmb, Llamo-
Med, Vierma-
Log Diversity (+)
Gitlub OPT-6.7B
Llatnts,Laama-7
Gitlub, Vieuma
Patents,OPT-...
Llunn-88
Patents, Patents,Viemss-71
Low OPT--67D
7
6
3
5
6
8
10
Z-Score (-+)
Figure The pareto frontier watermark strength in 2-score, shown on z-axis, versus text diversity, shown on the y-axis. Higher is better for both metrics. The Github subset results in particularly low 2-scores for all models and the Law subset is also shifted left versus the C4 data splits. Med, Patents, and Wiki yield higher z-scores more similar the C4-News data from the main work.
Watermark Strength (Z-Score) vs.
Similarity of Unwatermarked and Watermarked Outputs (P-SP)
0.70
C4-News, C4-News.Llmms-7 Llama-7 7B
Law. Naw.Virmna-PPP 7B
Llamm-B
Mel.
Mod.OPT-6.70
0.65
Gitlub, Llama
Med,
Gitlab,OPT-S OPT-6.B
Patents,Llamon-8
Citlmb, Viema-7B
Patents.OPT-6.77
0.60
P-SP (+)
Law Llama-7B
Patents.Virma: Viruna 7B
Patents,
Law OPT-66777
0.55
0.50
0.45
0.40
3
4
5
9
10
Z Score (-))
Figure 14: The pareto frontier of watermark strength in z-score, shown the æ-axis, versus similarity between the un-watermarked and watermarked output, shown the y-axis. Higher is better for both metrics. Similar trends Figure
27

---

Appendix-Tabee Table of Contents
Detection Rate of Watermarking across Datasets and Models
1.00
0.95
0.90
0.75
0.70
0.05
0.60
0.55
0.30
C4-Nows, CoNnoss,Lamm-77 Llamn-71 7B
Clenn Llam--a 7B
Wiki,Lumm-77 7B
Gidbob, Llama-7 Gatabooroo
Gisbab OPT-6.7 --878
Githud Vicuna-17 7B
Law Llans-7h 7B
Law OPT--677 1-
Law, Vieuna-77 7B
Med. Hol.Ismmu-T 7B
Med. MPT-07T-6.77 Mel, Vieuna-7h 7B
Patents, PntentssLlmmae Liana-Ta7 7B Patenta OPT-6.77 Patents, Vieuna-77 -
Figure Detection rate of watermarking as measured by ROC-AUC for extended selection of datasets and models, The Github and Law subsets producing lower z-scores in Figure 13 corresponds lower ROC-AUC.
The vi cuna model yields the least detectable watermark out of the three models in most cases.
Detection Rate
Watermarking after Attack Datasets and Modela
1.00
0.95
0.00
0.85
ROC-UC
0.75
0.70
0.45
0.00
0.55
Ch-New Almas-Th Ti Dipeer
Cf Ck-Nem. Lamn 78. GPT
0.50
Cha. Llmms L..T- 7B.GPT
Wike Llans GPT
Gllbab Jumn- 7i 70 Gidaa, Sulna.L.uue Lhan CPT
Gitaah, Porlah-Oi---a 78:7--4-..111 "EEDDppo Gitiab, OPT o GPT Cittaah Fiem---7 Dipuer Citiads, Vieune Fru.s-7fi CPT
Law, Annn- TI. Dipner Low Lauaa Llaaa-ll.cc CPT
Lao DPI TTT----T.DD Dipour
Low OPT 4777617..00 OPT
Lawe Viemns-T Trrms-TE. Dippor Law Fieus- 78 GPT
Med, Eonts Digner Mei Laan GP
Medl MedooPP PoPPa 78 Dipprr
Mel Toe GPT Steme 7h Dippor Med Finmn
Parmis Lians TH Dipner Patista Llans 70 GPT
OPT-s T--Th Diyner Patesia OPT OPT-6.70 1677 GPT
Patmat Vicuue--- Ti Dippor
Patasia Viram GPT
Figure 16: Detection rate of watermarking after being attacked using machine paraphrasing model as mea- sured by ROC-AUC for extended selection of datasets and models. The Github and Law subsets producing lower ROC-AUC in Figure 15 corresponds to lower ROC-AUC after attack well. While the GPT attack more successful in reducing the detcctability of the watermark majority of the cases (around 8/12) stronger claim withheld without further ablation domain specific paraphrase instructions or parameters for the dipper paraphrasing model.
28

---

Appendix -aplle Table of Contents
A.8 GPT ATTACK PROMPT ABLATION
When using 3. 5-turbo (GPT) to paraphrase the text throughout the experiments on water- marking and detector robustness, we prompt the model with an instruction prompt for the paraphras- ing task. In our initial experiments we observed the tendency of GPT' to produce a summary, i.e. generated text with good semantic similarity to the input but significantly shorter overall length with this summarization ratio worsening as the original length was increased. We tried to address this issue by specifically adding directives the prompt encourage the model to maintain the length of the text. While ultimately we were unable to solve this problem through our basic prompt engineering, we enumerate the prompts we explored Table 4. We leave the further study of how to optimally prompt general purpose LLMs such as GPT for paraphrasing tasks to future research.
We note that in one sense, this makes the attack using GPT strictly stronger, as some information contained in the original text is left out during summarization.
The prompt we selected to use throughout our experiments was "Prompt 4" as it resulted a slightly lower ROC-AUC for the standard z-score detector (lower indicating a stronger, more successful paraphrase attack) than "Prompt 3" whilst achieving a slightly longer averaged attacked output length than "Prompt 0". Prompts and 2 were included in table to be consistent with file in the source code though those other prompts were not competitive so they were omitted from the figures.
Prompt ID
Prompt Text
"paraphrase the following paragraphs:\n"
"paraphrase the following paragraphs and your best not to use the same bigrams from the original paragraphs \n"
"paraphrase the following paragraphs and try to keep the similar length to the original para- graphs\,n nn"
"You are an expert copy-editor. Please rewrite the following text in your own voice and para- phrase all sentences. in Ensure that the final output contains the same information as the original text and has roughly the same length. \n Do not leave any important details when rewriting in your own voice. This the text: \n"
"As an expert copy-edite please rewrite the following text in your own voice while ensuring that the final output contains the same information the original text and has roughly the same length. Please paraphrase please take care to provide any relevant information about public figures, organizations, or other entitics mentioned in the text to avoid any potential misunderstandings or biases." "
sentences
omit any crucial details. Additionally,
Table 4:
GPT paraphrase attack prompts. Performance of 0,3,4 shown in Figure 17 and Figure 18 as prompts and were not competitive. Prompt used throughout experiments main work Appendix.
29

---

Appendix
Capel Table of Contents
Detection Rate after GPT Attack using Different Prompts
1.00
0.85
0.80
3.70
0.65
300
Prompe
Pronpe
Prompt
Figure 17: Detection rate of watermark after attack by GPT using various prompts.
Length Reduction of Watermarked Texts by GPT Attack with Differeut Prompts
600
550
500
450
Text Length (TTokens)
100
350
300
250
200
150
50
Prompt Original
Prompt Original
Promps Original
Prompt 0 Attucked
Promut Attucked
Pronpt Atteekeed
Figure 18: Original token lengths (all 600) versus token length after paraphrasing using different prompts for GPT. Standard deviation is visualized for the latter. We choose "Prompt 4" to balance AUC after attack and length after attack, but wider study of prompting general purpose L.LMss for paraphrasing tasks left future research.
30

---

Appendix Dapeleix-Table
Table of Contents
A.9 WATERMARK DETECTOR ABLATION
A.9.1 WINDOW-BASED METHON
We design windowed test, called WinMax to accurately detect watermarked regions even in long documents. This is an alternative way of formulating detection hypothesis that can be employed optionally or in conjunction with the original test and requires no modification of the generation scheme. Given a sequence of tokens, we first score the sequence on per-token basis to find the binary vector of hits
{0, 1)" to each green list, which we convert to a partial sum representation Pk Σ 8t. WinMax searches for the continuous span of tokens that generates that highest 2-score. More formally, it computes
- - Vt(1 - 7)
wwin-max
max
(4)
As this test involves multiple hypothesis testing, we calibrate to fixed false-positive rate based on comparisons with non- watermarrarree text.
A.9.2 RUN-LENGTH BASED METHOD
We further investigated more complex anomaly detector based run-length differences between watermarked and unwatermarked text (Bradley, 1960). Yet, we found no gains from such a detector over z-test and WinMax within the range of settings we consider in this work. We include brief description of this alternate detection algorithm as starting point for future research.
Detection
Cope Paste
Desection Copy
Tum Length Detection Cope
1.00
1.00
0.85
0.85
0.80
0.50
0.73
0.77
775
0.70
0.20
0.70
0.65
0.55
0.30
0.45
0.55
0.45
0.83
100
0.33
(3)
0.33
00
50
100
Aitadd Percentag: (peeciity
3itaad Pertculazg: (evesit
Mitack Perecatagz (sovsitt
Figure 19: (Left)) The performance standard Z-Score watermark detection, (Center) WinMax detection, and (Right) the Run-Length based detector. WinMax outperforms the standard Z-Score at the 90% attack percentage, and the run-length test failed to show improvement over the standard test at any setting evaluated, however it demonstrates parity at stronger attack levels.
Additionally, investigate anomaly detection test based run-lengths potential detector of the distribution shift between watermarked and unwatermarked text (Bradley, 1960). Yet. we find no gains from such detector within the range of attack settings we consider in this work. After describing the basic results, we include a brief description of this alternate detection algorithm as starting point for future research.
In Figure 19, moving right along the x-axis means that the attack on the watermark becomes more severe (higher "attack percentages") because less of the total text remains watermarked. The marker style denotes how many fragments the remaining watermarked percentage is distributed across in the final text. As an example, the blue square at 90% attack, means 10% total tokens watermarked, split across 3 chunks. The WinMax method shows comparable performance with the standard Z- Score method at all settings except for the 90% attack percentage, where it handily outperforms the standard detector.
For the rightmost plot in Figure 19 showing the run-length detector performance, we visualize the best performing variants of the run length test, where only count the green runs, we the "maximum- binning strategy, we ignore any run-lengths for which we recorded zero ob- servations, and we the standard "pearson" chi-squared test statistic compare expected and
31

---

Appendix-Tadee Table of Contents
observed frequencies. Additionally, for the more severe 90% and 95% attack levels, the best per- forming setting was to ignore the length-1 I runs bin in the test statistic computation. These details elaborated at the end of this section.
Counting the Lengths of "Green Runs" At detection time, the test originally proposed by Kirchenbauer al. (2023) initially treats a piece of text as a sequence of green and red tokens, which can be modeled by a Boolean array i.e. sequence s such that 8, = 1 if E G. and 8, = 0 if not. However, the z-test then immediately reduces this sequence to one value ((s, 8 E G. the number of green tokens observed. We hypothesize that this reduces the power of the test in certain scenarios because it does not take into account information regarding the positions of the green (and red) tokens.
To harness this information, one can view the Boolean array as a set runs, where a run is defined by a subsequence of consecutive 1's or 0's. As example, under the convention that corresponds to a token being in its greenlist, the sequence [1,1, O, 1, 0, O, O, 1] contains green runs of length 1, and green of length 2. It also contains 1 red run of length 1 and 1 red run of length 3.
The example scenario that motivated our exploration of this method was the text mixing setting (copy-paste attack) where sections of watermarked text would be interspersed within surrounding unwatermarked text. In the case of just a few heavily watermarked subsequences, one would expect observed a few isolated runs of green tokens (consecutive 1's) that were surprisingly long. This is because we expect to observe few greens (yT in expectation) in unwatermarked text, and many more in heavily watermarked text, and long green runs are themselves caused by higher overall green token count.
Hypothesis Test for a "Run-Length Detector" To formalize this notion of what we' 'd find "sur- prising", and thereby derive hypothesis test from it, we leverage the fact that for binary event with a probability of "success" p. the number of independent trials k required to realize the first success can be modeled by geometric distribution, with density given by
Pr(" after trials") (1 - >>pp for 1,2,3...
(5)
We can treat observing a or a redlist token as a success event, and therefore model the "green runs" as geometrically distributed random variable with success probability 1 y- Armed with this fact, we treat each run length & 1, 2..... as a category, then for a given piece of unwatermarked text, the expected values of each of these categorical variables can be computed using the geometric distribution density function scaled by the total number of runs observed in the text.
Therefore, we can test for the watermark by testing the following null hypothesis,
Ho: The text sequence is generated with no knowledge of the watermarking rule and therefore the "green" run lengths are geometrically distributed.
One standard way to compare an observed set of categorical variable outcomes to perform a chi-squared test (Cochran, 1952). In this test the sum of squared deviations between the expected and observed frequency (count) for each category are summed. If the statistic is small, then the oberved counts are close to the expected, and we fail to reject the null hypothesis. However, if the statistic is large, then least one of the observed frequencies is surprising different from the corresponding expected frequency and the collection of categorical variable observations is unlikely have come from the expected distribution, in which case we can reject the null hypothesis.
expected set,
Returning to the context of green run lengths, this means that if we observe green run length counts that don't match the expectation given by the geometric distribution. we are confident that this se- quence was not generated under the null hypothesis, and rather, was generated under the watermark- ing scheme. We expect that the most common mechanism for this rejection under watermarking would be observing a surprising number of long runs of green tokens, i.e. surprisingly high frequen- cies in the tail (larger values of k) than expected under the geometric distribution.
Design Choices
32

---

Appendix Taable Table of Contents
Expectll
20
Figure 20: An intuition- building example of the run length distributions we seek to compare. The left is the actual, empirical run lengths observed in a watermarked sample, and the right is the expected counts of each run length based the total number of observed runs, dsitributed according to the geometric prior parametrized by 1 y (the null hypothesis). Each bar shows the number of runs of given length that were observed. The "surprising" observations are the non-zero counts for the length 7,9,10 and 12 bins.
1. When we "count green runs, where red is the success event" new run begins after each observation of a red token. The sequence R, R, R, G. R) yields 3 "green runs" of length 1, and then green run of length 2. This is because red occurs after just single trial, three times in a row, and then the final red takes two trials to achieve.
2. For a sequence of length one could observe any possible run length E {I...T), and we can compute expected frequency for all & based the success probability 1 y However, it is standard practice bin the tail unobserved & values to create a new event "run lengths longer than k" and the probability mass for those values is summed before computing the expected number of outcomes for that tail category. In Figure 20, the max 12 and this addition of extra
shown (13), is but the maximum actually observed bin represents the tail of runs longer than the max observed.
3. For any run lengths & between I and the largest observed run length value kmaz it is stan- dard practice to ignore these categories when computing the test statistic if there were observed occurrences. This is based on the assumption that the zero observation is likely spurious consequence of small sample size (of runs). In Figure 20, there are bins, and 11, that also are zero, which can be ignored in the test statistic computation.
4. We consider the standard "pearson" formulation of the chi-squared test, as well as the "g- test" and "cressie-reed"" variants based on likelihood ratios rather than squared deviations.
5. In order to isolate the rejection scenario we expect under watermarking, we experiment with ignoring the small categories & in the test statistic computation. The intuition that these short run lengths could dominate the statistic value in an undesirable way when there are just small handful of surprisingly long runs in the tail of the observed distribution. Considering the example in Figure 20, since the close counts of observed and expected length-1 runs potentially of little interest with respect to the null hypothesis reject case expected for the watermark, we can choose ignore the leading bin in the test computation.
33

---

Appendix- Table of Contents
A.10
DETECTABILITY OF WATERMARKS AFTER MACHINE PARAPHRASING: TPR @ T
Detection Rate of Watermarking after Attack
function of Tokens Observed (@ 0.1% FPR)
1.0
SelfHish Dipper
Selfflach GPT
Lefl GPT
0.8
Left Dipper
SelfHask CP-1-25%
Left Hash CP-1-255
0.6
SelfIlahh CP-3-25%
TPR
LeftHash CP-3-25%
SelfHash CP-1-10%
SelfHreh CP-3-10%
0.4
Leftllash CP-1-109
Hed CP-3-1185
0.2
0.0
100
200
300 Tokens (T)
400
500
600
Figure 21; In this variant of plot from the main body, visualize the growth characteristies of watermark detection quantified by True Positive Rate low False Positive Rate (0.1%) after attack by selection of machine-based attacks. As in the main body charts, make the sequence length of the attacked set after generation to indicate that these mcasurements are based on increasing fewer samples after that point as T continues to grow and are therefore more uncertain. For the Dipper attack this is 500 and for the GPT attack this is ~ 300. Due to the synthetic nature of the Copy-Paste attack, after attack, those sequences still full length i.e. 600 1 25.
translucent starting the mean
34

---

Appendix Papele Table of Contents
A.11 EXPERIMENTAL METHODOLOGY DETAILS
A.11.1 GPT PARAPHRASE ATTACK
We utilize the prompt chosen through the ablation in Appendix A.8 and query the model using a sampling temperature of 0.7 and a max token limit of 1000.
A.11.2 DIPPER PARAPHRASE ATTACK
We utilize the Dipper paraphrase model proposed by Krishna et al. (2023) and released at their github. Since this model smoothly trades off semantic similarity between the paraphrased and orig- inal text starting from few discernible changes at one end to almost wholly unrelated at the other extreme of its lex and div parameters, choose one of the moderate strength settings run the paraphrasing attacks with across all experiments 1ex=40, div=40. This still represents a sig- nificant attack, but maintains high paraphrase quality/similarity. We of course emphasize that the setting used is the same for all the watermarking and baseline detector methods to fairly compare them.
A.11.3 BASELINE METHOD DETAILS AND HYPERPARAMETERS
We lightly adapt the codebases provided by the authors interface with our generation, attack, and evaluation pipeline for both methods and use default parameters found in their code unless otherwise specified.
The retrieval system of Krishna et al. (2023) requires the creation and maintenance of comprehensive database of all sequences generated previously by the language model, so that paraphrased generations can be mapped to their original, uncorrupted source. In our experiments, adopt the retrieval method described by (Krishna et al., 2023), and utilize the BM25 search method as it performed better in their evaluation. A key detail concerning this method is how we construct copy-paste examples evaluate on. The "unattacked" text the output of the language model without any modification and this is what is loaded into the retrieval database as the "generation history" of the model. The copy-paste attacked version of the text created by inserting a sub-string from that text into the other piece of machine-genee seehere-eeeerated text readily available for this prompt, the watermarked generation.
Retrieval:
To strengthen the performance of the learned classifier approach in the context of text corruption and paraphrasing. Hu al. (2023) proposes RADAR, a framework for training a robust Al-text detector using adversarial learning. Their method is inspired by techniques such as gen- crative adversarial network (GAN) training which poses learning as a two-player game. RADAR comprises paraphraser and detector model where the paraphraser's goal is to generate realistic content that can evade Al-text detection, while the detector's goal is to enhance Al-text detectability. We use the only publicly available RADAR- SaAR--iccna--7 7B model, where the detector model has been robustly optimized to detect Vicuna generations.
RADAR:
DetectGPT: DetectGPT (Mitchell et al., 2023) is zero-shot post-hoe method for detecting machine-gene generrted texts employing a curvature-batt criterion that compares the log probabili- ties between candidate passage and perturbations of it. The intuition behind DetectGPT is that machine-generater texts tend to dominate the negative curvature regions of an LM's log probability curve. We use the official implementation of DetectGPT and follow their default setting by using T5-3B (Raffel et al., 2020) as the mask-filling model for generating perturbations. We adopt the strongest setting of DetectGPT by generating 100 perturbations for each test sample and using the normalized perturbation discrepancy as the criterion ("z" in their notation). Further details regarding the evaluation of the method are included the appendix, but the key experimental detail is that the positive examples for detection derived from the unwatermarked model outputs generated from each prompt, and as with watermarking detection, the human gold completions are the negative examples.
We remark that there are some unknowns about how well the DetectGPT paradigm works when different models are used the detection target, especially when under attack. In this work primarily utilize 1 alama as the base model to be detected, and the relationship between the relative sizes and qualities of the base model, the perturbation model (a 3B parameter version of T5), and
35

---

Appendix-Tadee Table of Contents
the machine paraphraser (Dipper gpt-3. 3.5.urbb)), could be quite subtle and produce counter- intuitive detection performance outcomes. We leave the study of post-hoe detectors with improved robustness characteristics to future research.
A.11.4 DEFINING "POSITIVE" AND "NEGATIVE" DETECTION TEST SAMPLES
"Negative" Samples In all experiments, either using watermarking as the detection method or the two baseline approaches. the "negative" samples test time that should be classified as "not watermarked" or "not machine- "eechinegeeeratee" respectively, are human-written text i.e. the gold com- pletions/suffixes extracted from the input data as prompts are created.
"Positive" Samples for Watermarking Detection For the watermarking experiments, in the unattacked setting, the "positives" are the watermarked model's generations. To produce the parapbrase-attacked versions, the watermarked generation is fed to the paraphrasing model (GPT or Dipper) rewritten by humans. To construct the copy-paste attacked versions for watermark- ing, sets of watermarked tokens are inserted into a surrounding context sequence of human- wrrtte tokens (i.e. the gold completions the prompt). While this means that the negative examples and surrounding context examples for the copy-paste watermarking experiments are correlated, this does not give the watermarking method any unfair advantage at detection time. If a specific sequence of human-written text happens to produce an abnormally high z-score, which could artificially inflate the z-score of the copy-paste attacked example making it an easier detection, then it will simultane- ously increase the chance of false positive on that sequence for preciscly the same reason. Since both examples are always tested, this effect should be balanced out.
"Positive" Samples for Alternate Detection Approaches For the Retrieval based and Detect- GPT baseline approaches, the "positives" are the unwatermarked model's s generations as this the distribution that the approach is designed to detect. For the Retrieval method, this means that the retrieval database (index) is loaded up with the set of unattacked, unwatermarked generations so that, if those same sequences are queried at test time, then the retrieval performance (as measured by TPR) should be perfect. To construct paraphrase attacked examples in this setting, the unwa- termarked generations are fed to the paraphrasing model (GPT Dipper) causing them to diverge from the exact sequences present in the database, and the exact model distribution that DetectGPT is testing for.
Copy-Paste Attacked Samples for Alternate Detection Approaches stated above, for copy- paste attacks in the the watermark detection evaluation, we insert spans of watermarked tokens into surrounding context of human-writter tokens. Since watermarking effectively views all text as boolean arrays "green" and "red" tokens regardless of the textual content itself, human-written text is the most "negative" type of context we can create to make the embedded watermarked chunk harder to detect.
However, since the human- written completions are already used as the negative examples, and the baseline detection methods rely much more on the syntactic and semantic similarities between the examples, to reduce the error correlations between the negative and positives for Retrieval detection and DetectGPT, we instead insert the small unwatermarked chunks (parts to be detected) into surrounding context of watermarked text. While we realize this choice might seem bit strange, and also admit that it is mostly an implementation pipeline convenience rather than perfectly optimal choice, we do not believe this causes any unfair biasing of the detection performance estimates for the following reasons.
For fair copy-paste example with respect to the particular detection approach being evaluated, we desire "negative" looking context tokens to surround the "positive" looking chunks to be detected. From a semantic similarity perspective, the watermarked generations used as a source of context tokens, are quite relevant/similar to the unwatermarked subsequences being inserted because they were generated following the same prompt. For Retrieval, this means that there is actually gener- ous/favorable level of semantic similarity between the context tokens (meant to make the copy- -pste attacked sample look negative) and the unwatermarked outputs stored in the retrieval database. We believe this is a reasonably fair and realistic setting since the copy-and aad-ttsting attacker we are sim- ulating would likely replace removed sections of the text be detected with semantically similar
36

---

Appendix Dapel of Contents
and relevant text (as opposed to random tokens). However, overall, this is likely overly generous setting for these methods.
For the RADAR method, we construct the copy-paste examples in the same manner as for the water- marking methods. We paste sets of un-watermarked tokens into a surrounding context sequence of human-writter text. This has the same correlation caveat as described above for the watermarking case, but we felt it necessary to use this evaluation setting given the fact that the RADAR method was demonstrated to exhibit strong transfer properties between models. Since the watermarked comple- tions are in fact quite similar to the un un-watermarked pairs, this would present copy-paste examples that were effectively almost not attacked all from the perspective of the RADAR model.
Potential Confounding Factors in Copy-Paste We believe that this setup is fair for the Detect- GPT method, however, we realize that both the perturbation procedure and the likelihood estimation are probably influenced by certain discontinuities introduced in copy-pasted examples as no algo- rithm or post poot-psocss step is used to smooth out the interface region between the positive and negative spans. That said, since we find that Retrieval performs quite well under the copy-paste at- tack and that for DetectGPT the copy-paste examples produce somewhat unremarkable behavior as visualized in the main work and in Appendix A.12 (versus the GPT and Dipper results), we believe these methodological choices did not significantly influence the results in an unfair way. That said, we believe there are still open questions in developing suites of different attack strategies ranging the gamut from black-box model-based paraphrasing to synthetic text mixing that test a wider range of possible attack and corruption scenarios that could be encountered mo-the-wildd
A.11.5 BASELINE DETECTION METHOD PERFORMANCE AS A FUNCTION OF TOKENS OBSERVED
While the watermark detection score (z-score) is readily computable in a cumulative manner along the sequence length dimension, and thus evaluation T is simple for that method, for Retrieval and DetectGPT, the sequences to be tested must first be chunked into prefixes and then those sets of pre- fixes evaluated individually. For Retrieval detection, we make the choice that the sequences loaded into the database should be the full length unwatermarked output, even though we are querying with prefixes of those outputs to develop the series of measurements at T. This reflects what we believe to be realistic setting where the model generated the full output at some time in the past, and at test time is being queried with a prefix/fragment of the original generation. Additionally, storing all prefixes of some given size for all generations is not a realistic or scalable choice in our opinion.
For DetectGPT, the adaptation for detection at is easier in one aspect because is simply imple- mented by testing block of prefixes of the original unwatermarked (and then attacked) generations. However, the computational cost of producing just a single series of measurements at a range of T values becomes prohibitively expensive for single attack setting. This is because for each block of prefixes, say the leading 100 tokens from a set of N sequences that originally had length 600, the runtime cost is roughly the same as testing the full length outputs. Thus the overall runtime of test- ing all prefixes, or T values, for given set of sequences is multiplied by the number of prefixes, or length/stride. In contrast, there is effectively no multiplier for watermarking, and it is still relatively cheap for Retrieval since the retricval method itself is much cheaper (at least for a small database). This is the reason for the limited number of datapoints shown in the comparisons for DetectGPT as a function of observed tokens.
As final remarks on the somewhat surprising performance of the method, we note that the genera- tions are tested without the original prompts included in the input. We assume this is the setting of the original work, since having access to the prompt at test time would be unrealistic, but beyond this detail, are unsure as to why evaluating the method using the 1lama model as the base model to be detected, performs as poorly as it does. We leave more comprehensive robustness evaluation of the DetectGPT method to future work by its creators and other researchers but hypothesize that the relationship between the base model, the perturbation model, and the paraphrasing model, could be more nuanced and require more careful tuning of hyperparameters other aspects of the detection method.
37

---

Appendix- Table of Contents
A.12 DETECTION METHOD COMPARISON: DETECTOR SCORES @ T
In this section we present a "mechanistic" explanation for the detectability as a function of text length (AUC T) results reported in the main body of the work. In particular, motivate the perusal of this section by remarking that in order to perform well, a score-based binary detector (like all the methods considered here) must maintain gap between the scores assigned "negative" or genuine human written samples, and the "positive" or machine generated samples. The left chart in each pair are the scores assigned to negative examples by the given method, and the right chart shows the scores for positive examples. We make note of the fact that the standard Z-Score watermarking produces an extremely wide gap between corresponding curves in the left and right charts in Figure 22, driven in large part by the lack of growth in the negative scores. This key behavior of watermarking provides for both detectability and a low FPR.
Z-Sesto Thman Text finetion of Tolena Obeeved
Z.Srcce of Attariadi Text as function of Tokons Obeervod
15.0
15
10/5
10111
2,3000
7.-
3.89
0.0
0.-
0.2
37%
0.2
10:0 0.1
1000
000 1000
1000 10:12
Thion (T)
Teloro:
Figure 22: The deterction scores yiclded by the watermarking detection z-score method. stark contrast to the trends in Figure 24 and Figure 25, we see that for all T,, the z-score of human text remains very low (Left) whilst the z-scores for the attacked watermarked texts continue to grow steadily (Right) demonstrating the favorable token sample complexity characteristics of watermarking detection. As in preceding figures turn the curves translucent after their mean sequence length value to indicate increased uncertainty.
WinMiat: Z-S Score Human Ted as finetionn of Tokene Oberroed
Win.Max Z Z-Scure Atlacked Texi as lucection Tolens Obearve ved
15.0
1510
Whidax 2.5cre
10.0
WalMex Soare
10.0
3.0-
5.0
2.5
0.
0.0 100
(00 100)
1000
10:02
Thiom (T)
Tiloro:
Figure 23: The detection scores yielded by watermarking detection WinMax z-score method. Compared to Figure 22 we see that the likelihood a False Positive at smaller values of T is higher under WinMax than the basic z-score detection test as the separation between the scores for human text (Left) and attacked watermarked text (Right) is not as large. However, empirically, this tradeoff enables improved detection under the strongest copy-paste attacks as shown Figure and Figure As preceding figures we the curves translucent after their mean sequence length value to indicate increased uncertainty.
38

---

Appendix
Capele Table of Contents
Retiiiaal Detrction Seare of Human Tist Obarrved
Retricual Detcction Score Actocked Text
Eunction Observed
Setirall Sege
Retrraal
210
40C
800
1000
200
600
800
1000
(T)
Figure 24: The similarity scores yielded by the Retrieval detection method using the BM25 scarch index. (Left) The corresponding scores for "negative" human-writter written samples are lower than (Right) the scores for the "postive" attacked samples for all values anistically explains the favorable detection performance shown in other figures. However, we note that the gap is not as large for the copy-paste attack as it is for GPT and Dipper. As in preceding figures we turn the curves translucent after their mean sequence length value indicate increased uncertainty.
which desired/required
proper detection
mech-
DeteciGPT Score of Hummn Text as finction of Tokens Ohaersed
DiteGGPT Score of Attucked "Tox ne function of "Feirss Observed
Debetcor
Figure 25: The similarity scores yielded by the DetectGPT method. (Left)) While the corresponding scores for "negative" human-written test samples start out lower than (Right) the scores for the "postive" attacked samples at small values of T, which is desired/required for proper detection, this ordering becomes reversed iineebleeeiiveedddddee
for the GPT and Dipper paraphrase attacks at larger T which helps explain the detection performance curves out at 300 and 800 tokens those methods shown other figures.
39

---

Appendix Ceeendi-TTable Table of Contents
A.13 HUMAN STUDY DETAILS AND PREFERENCE EVALUATION
A.13.1 INDIVIDUAL HUMAN PARAPHRASER PERFORMANCES
Examining the individual performances in Figure 26, the only real exception in this study was hu- man writer 13, who is strong paraphraser and simultaneously did not submit enough text to be detected (human 24 only partially completed the tasks and was omitted from gift-card considera- tion). More text from this writer would required to guarantee detection. On the other extreme, human writers 22 and 15 apparently paraphrased the text with strategy that did not substantially affect the watermark, and are reliably detected after only about 250 tokens, or 200 words.
Z-Score of Watermarked Text function of Tokens Observed
30
Tuman Avg
Ruman 10
Z-Scrre
20
Human
Fuman:
Fuman
lluiaa
Human
10
luuoan
Human 19
Human 13
Rumau
0-
Human 91
250
500
750
1000
1250
1500
1750
Tokens (T)
Z-Score of Watermarked Text after Paraphrasing as function of Tokens Observed
12.3
Homaa 15
Homan
10.0
Humsa Ang
Homan
Buman
7.5
Bumaa 16
Z-Score
Buman
Iuuma
5.0
Humaa
Homen 18
Humaa
2.5
Hunes 10
0.0
Homen 13
Homan
-2.5
200
400
600
800
1000
1200
1400
1600
Tokens (T)
Figure 26: z-score function of I" all text passages given each writer. separated per writer. (Top) Original scores for combined text given to human writer. (Bottom) Average scores per writer after paraphrasing. We find that almost all human writers are detected with exceeding certainty, after about 800 tokens, i.c. about 500 words (a z-score of 4 implics a FPR of 3.2e-5). Note that despite their strong para- phrasing capability (as shown in the left plot of Figure 4), Human 24 paraphrased too few examples for a fair competitive comparison with the other annotators, but they still included part of the averages.
A.13.2 QUALITY/PREFFEEENC EVALUATION
To give a second perspective on the quality impact of watermarking on generated texts, as part of Human study (Table 5), ask annotators rate which of two potential responses prompt
40

---

Appendix- Table of Contents
they generally prefer. And we find that for very strong watermark (? 0.25, 8 4.0) humans only prefer the unwatermarked text over the watermarked text a weak majority the time.
Total Ratings
Unwatermarked Answer
Watermarked Answer
Unwatermarked Preferred
177
109
68
61.58%
Table 5: Outcome of human preference study. We report the frequency human evaluators preferred the un- watermarked generation output over watermarked oulput.
A.13.3 DATA GENERATION AND SELECTION PARAMETERS
We utilize the Long Form Question Answering (LFQA) style dataset curated by Krishna et (2023) available via the Google Drive link provided at g github. com/ma rt ans i deo hemoon ai ai-detection paraphrases. We generate machine responses to the questions using the vicuna model under (7 = 0.25, 5 = 4.0) watermark prepended with the following prompt:
'Answer the following question in 200-300 words. Explain it like I'm five. \n)n'
To select the small subset of examples presented to annotators the paraphrasing study we filter the original 2758 questions to a subset of 60 by selecting examples such that the watermarked model response was 1) longer than 200 tokens, 2) had a z-score of > 9.0, 3) had a P-SP similarity score between the gold human response and the watermarked response of > 0.6, and 4) had 4-gram repetition rate < 0.11. In particular we enforce 1) and 2) to make sure that these examples were of adequate length and heavily watermarked to start out with, in order to develop a significant result based on the final z-scores achieved through paraphrasing. Considering weakly watermarked exam- ples with low starting z-scores, would make for uninformative samples since there would be little watermark to scrub away in the first place. Constraints 3) and 4) were enforced simply to raise the quality of the machine responses these questions are quite challenging to answer well even for the stronger instruction-tuned vicuna model utilized.
For the preference study we filter the original set from 2758 to 205 by enforcing that 1) token lengths of both the unwatermarked and watermarked outputs were > 200 and differed in length by no more than 50 tokens 2) that the watermarked text 7-scores to increase the likelihood of the unwatermaked and watermarked texts being perceived as different based on the significant watermark (most examples had a much higher z-score than that lower limit) and the first constraint was chosen to remove the spurious differences annotators might perceive due length differences which are not necessarily indicative of quality.
4.0. The second constraint was chosen
A.13.4 HUMAN STUDY ANNOTATOR INTERFACE
onder
Figure 27:
The LahelStudio interface designed for the human paraphrasing and preference studies. Left: peefrenneesuudy. study.
Interface for paraphrase study. Right: Interface for human preference
A.13.5 ANNOTATION PLATFORM AND TASK INSTRUCTIONS
We utilize the open source data annotation platform LabelStudio (Tkachenko et al., 2020-2022) to conduct the human study and screenshot of the interfaces constructed for each of the two tasks are provided in the main work.
41

---

Appendix-Tadee Table of Contents
Here, additionally show the instructions that were given to annotators for both of the human study tasks in Figure 28 and Figure 29.
"Paraphrase Text"
Description
Paraphrase Al generated response to question from Reddit's r/explannikeiimiiee (ELI5)
forum.
Instructions
A question topie statement is shown at the top of the screen. On the left side of the screen you will response the question. The respouse The response "watermarkod, " meaning it contains invisible patterns that written by Al and
geucrated by an language model.
be used the text person. Read the Al-generated the right side of screen,
determine that the response respouse on the half the screen, re-write response in your own words, whilst preserving the meaning and length the text. Your goal is to change the text so much that the watermark no longer detectable.
When you are finished, click the "submit" button to your re-written text and move to the next task.
Requirements:
1. Paraphrase quality /similariitttty paraphrase should convey roughly the information the original text, roughly the same level of detail.
2. Time limit Try spend
than minutes any individual paraphrasing task. The annotation software tracks the time you spend on each task, but will not
explicitly enforce the time limit by kiiking you off. Please do the tasks in single sitting.
3. No automated paraphrasing tools Do
any Al tools that write text for oopy/paste text from any external source.
you (eg., ChatGPT, Grammarly), and However, you may look things up online, refer dictionary or thessuruses, and use a spell checker such a tool is enabled in your browser window.
Figure 28; Annotator instruction sheet for the human paraphrasing task.
42

---

Appendix Papeleix-Table Table of Contents
"Compare Answers"
Description
Seloct preferred response to questions from Reddit's r/cxxpaainiilciniiie (ELI5) formu.
Instructions
At the top of the screen you will see a question or topic statement. Bencath there will be two different responses to the question, one on the left and one on the right, Choose the best response of the two by clicking on the left or right text box. Then click the "submit" button on the bottom right. to save your selection and move on to the next task.
Requirements:
1. Time limit Please spend at most 5 minutes on each individual respouse pair. necessary, briefly cousult the internet to clarify the meaning of words or check the correctres of statements.
Figure 29; Annotator instruction sheet for the preference evaluation task.
43

---

Appendix Capelei--Table
Table of Contents
A.13.6 DEMOGRAPHIC AND COMPENSATION DETAILS
We recruited graduate students from computer science department do the paraphrase and prefer- ence evaluation tasks. 14 annotators worked on paraphrases and 9 additional annotators worked on preference ratings. One out of the paraphrase annotators did not complete enough samples and so is removed from some of the evaluations in the main work. The goal was maximum diversity of annotated samples and so each of the original watermarked texts was paraphrased by a single annotator, and roughly 150/177 preference rating examples were for unique questions.
The group comprised both native english speakers and english-as-a-See lat---------eno--laaaaa-e -anguage speakers. We prompted volunteers to self pre-select based description of the tasks to be performed, empha- sizing that the ability to write high quality paraphrases Admission to the relevant university requires high level of English language reading and writing competency as assessed by required standardized testing before admission.
few paragraphs in length was required.
All annotation tasks were performed in one 1.5 hour session and all annotators with free dinner and drinks. As additional incentive to ensure that the paraphrase task was completed in "motivated" manner try and approximate the real incentives of paraphrase attacker, informed participants that the three most successful attackers (their paraphrases achieved lowest final detection scores) would be awarded a $100 gift card. An additional gift card randomly awarded to annotators who only performed the preference comparison task as this task was less goal oriented and thus there was direct way to quantify success or rank annotator performance.
compensated
A.13.7 INSTITUTIONAL. REVIEW BOARD "EXEMPT" STATUS
In preparation for conducting the human paraphrasing and preference evaluation study components of the research, a "Human Subjects Research Determination" form was filed with the relevant In- stitutional Review Board. Before any portion of the human study conducted, determination letter was received communicating the status of "Exempt" for the project proposal, i.c. "Not Human Subjects Research".
44

---

Appendix Copeleix-Table Table of Contents
A.14 CODE DETAILS AND RELEASE STATEMENT
We extend the implementation of watermarking developed by Kirchenbauer et al. (2023). For the datasets and models evaluated in this work, we heavily utilize the huggingface datasets library and huggingface trans formers modelling framework. We retrieve pretrained model weights and dataset files from the huggingface hub, with the exception of the weights for the 11ama base model.
The 1lama 7B parameter model weights were retrieved with permission using presigned URL received via email from the "LLAMA Release Team" to be used for research purposes only in accordance with the license. These weights were then converted fore use. To construct the Vicuna model, retrieved the lmsys /vi ariican--7--ae aeeeellaviiie weights following instructions github. com/ aamn-sss/Fastttt and merged them with the base 1lama 7B weights.
the huggingface format be-
A.15 HARDWARE AND COMPUTE DETAILS
The experiments performed in the study were all inference-based and therefore could be run on single Nvidia RTXA4/5/6000 GPU. The 7B parameter models were run floatls during gener- ation of watermarked and unwatermarked responses, but the Dipper model was run in full precision accordance with author recommendation, and output quality issues observed under the floatl6 setting. Additionally, the Dipper model and DetectGPT model were both run on A6000 cards due the memory footprint required by their larger parameter counts. Generation stages, where un- watermarked and watermarked outputs were sampled, took less than 12 hours. Attack stages using both the GPT (OpenAl API) model and the Dipper model took around 4-6 hours. Evaluation stages where watermarking detection at only the max T value was performed took minutes, and when ROC-AUC's at all T values and all text quality metrics were computed, took less than 2 hours. De- tectGPT evaluation took well over 12 hours for single set (~ 500 samples) of generations with longer token lengths such as T = 1000.
45


Reviews:
[{"review_id": "ZF3tsCkvdA", "reviewer": "Reviewer_o8L2", "summary": "The paper studies the robustness of language watermarks against paraphrasing attacks. These attacks involve both paraphrasing models and tests against human rephrasing. The surveyed watermark is shown to be robust against all attacks. It is also more effective than retrieval-based attacks in cases where watermarked text is inserted into non-watermarked text. Comprehensive experimental details are provided in an appendix by the authors.", "questions": "* What is a successful paraphrasing attack? \n\n* What does an attacker need to achieve to undermine a watermark's robustness?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Overall, I quite liked the paper and think that it addresses an interesting problem.\n\n* The appendix is extremely detailed and offers a lot of valuable information on the reproducibility of their study. \n\n* The paper shows that the studied text watermark is robust against many attacks, including human paraphrasing, which is somewhat surprising. \n\n* The paper outlines many useful parameters and graphs for evaluating the robustness of watermarking.", "weaknesses": "The main issue I have with the paper is an unclear threat model: What is an attacker allowed to do to paraphrase sequence correctly? When is a paraphrased text too dissimilar from the watermarked text? The paper does not answer these fundamental questions, but follow-up papers must rely on these answers to propose improved attacks. \n\nConsider the following example: A human and a paraphraser want to preserve the \"meaning\" of the watermarked text. The watermark hides with high probability in high-entropy token sequences, such as names, locations, and numbers. Is a paraphraser allowed to replace these with random other names and locations, or would that be considered \"unsuccessful\" paraphrasing? I would love to see that discussed in the paper so that future papers can meaningfully improve on the presented attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DKCuAhv2k4", "reviewer": "Reviewer_o62T", "summary": "This paper studies an important and timely topic on the robustness of watermarked AI-generated text detection, in the scenarios of language model paraphrasing, human paraphrasing, and partial editing (e.g., copy and paste).", "questions": "1. For the analysis of post-hoc detection, given that DetectGPT is shown to be fragile against paraphrasing (Sadasivan 2023), can the authors add a new analysis with a more robust post-hoc AI-text detector, like RADAR? I would like to see the result of RADAR in ROC analysis and with varying token lengths, as in Fig. 5 and Fig. 6.\n\n2. I don't see much discussion on the utility of the studied watermark methods (especially the new one, SelfHash). if a watermark is robust but lacks usefulness, it is impractical. Can the authors report the usefulness of the new watermark methods that have been not studied in (Kirchenbauer 2023)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The research scope (whether an AI-general text embedded with watermarks can be easily removed or not) is an important and timely topic.\n2. Empirical results are abundant and show the promise of the reliability of the evaluated watermark methods", "weaknesses": "1. Probably due to page limits, most spaces are used for presenting numerical results. The methodology section, including a new watermark method (e.g. SelfHash) and a new detection method (WinMax) in Sec. 3 is relatively short (roughly one page), and much important information is deferred to the Appendix.\n2. The analysis will be more complete if it includes more recent and advanced post-hoc detection methods (such as RADAR https://arxiv.org/abs/2307.03838), because DetectGPT is known to be non-robust to paraphrasing. I would like to see the result of RADAR in ROC analysis and with varying token lengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZQo17njAZ", "reviewer": "Reviewer_jTXD", "summary": "The paper investigates the effectiveness of watermarking as a means to identify machine-generated text in realistic scenarios, particularly when dealing with potential attacks, such as paraphrasing, copy-paste, or human rewriting. The primary objective is to evaluate whether watermarks can reliably detect machine-generated text when it's subjected to various forms of manipulation, thus contributing to the broader discourse on mitigating potential harms caused by generative language models.", "questions": "1. Can watermarking be further improved to make it more resilient to copy-paste attacks, given the significant performance drop observed under this scenario?\n\n2. How does the performance of watermarking compare to other detection methods in detecting machine-generated text within shorter sequences or fragments?\n\n3. Could watermarking be used in conjunction with other detection methods to enhance overall detection reliability, particularly in complex and adversarial settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Evaluation: The paper conducts a thorough and comprehensive evaluation of watermarking, considering various real-world attack scenarios, including paraphrasing, copy-paste, and human rewriting. This multifaceted approach provides valuable insights into the strengths and limitations of watermarking in practical settings.\n\nComparison to Alternative Methods: The paper not only focuses on watermarking but also compares it to alternative detection approaches, including post-hoc detectors like DetectGPT and retrieval-based systems. This comparative analysis enhances the paper's contribution by showcasing the relative merits of watermarking.", "weaknesses": "Lack of Theoretical Background: The paper does not delve deeply into the theoretical aspects of watermarking, which could be crucial in understanding the underlying principles and potential vulnerabilities. A more robust theoretical foundation could enhance the paper's overall quality.\n\nInherent Model Bias: The paper uses a specific language model (llama) for its experiments. While this model is justified and used for practical reasons, the results might not be universally applicable to all language models, which could limit the generalizability of the findings.\n\nLimited Discussion of Social Implications: Given the paper's focus on mitigating potential harms from generative language models, it would be beneficial to include a more extensive discussion of the social implications of watermarking and other detection methods. This could provide a more holistic perspective on the topic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)