Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
Authors: Guangsheng Bao
Yanbin Zhao
Zhiyang Teng
Linyi Yang
Yue Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Fake Detection
Machine-Generated Text Detection
Zero-Shot Detection
Primary area: societal considerations including fairness, safety, privacy
TL;DR: Fast-DetectGPT accelerates DetectGPT by two-orders of magnitude and enhancing the detection accuracy by a relative 75%.

Abstract: Large language models (LLMs) have shown the ability to produce fluent and cogent content, presenting both productivity opportunities and societal risks. To build trustworthy AI systems, it is imperative to distinguish between machine-generated and human-authored content. The leading zero-shot detector, DetectGPT, showcases commendable performance but is marred by its intensive computational  costs. In this paper, we introduce the concept of **conditional probability curvature** to elucidate discrepancies in word choices between LLMs and humans within a given context. Utilizing this curvature as a foundational metric, we present **Fast-DetectGPT**, an optimized zero-shot detector, which substitutes DetectGPT's perturbation step with a more efficient sampling step. Our evaluations on various datasets, source models, and test conditions indicate that Fast-DetectGPT not only surpasses DetectGPT by a relative around 75\% in both the white-box and black-box settings but also accelerates the detection process by a factor of 340, as detailed in Table 1.

Published as a conference paper at ICLR 2024
FAST-DETECTGPT: EFFICIENT ZERO-SHOT DETEC- TION OF MACHINE-GENERATER TEXT VIA CONDI- TIONAL PROBABILITY CURVATURE
Guangsheng Bao
Yanbin Zhao
Zhejiang University
School of Mathematics, Physics and Statistics, Shanghai Polytechnic University
School Engineering, Westlake University ba oggangghhnng@westtlleeee edu.cn
zhaoyb553@nemu .n
Zhiyang Teng
Nanyang Technological University zhiyang. teng@ntu..eee.... edu.
Linyi Yang, Yue Zhang
School of Engineering. Westlake University Institute of Advanced Technology. Westlake Institute for Advanced Study fyangling inyii, thancyue)ewest..ee.. edu.
ABSTRACT
Large language models (LLMs) have shown the ability to produce fluent and co- gent content, presenting both productivity opportunities and societal risks. To build trustworthy Al systems, is imperative to distinguish between machine- generated and human-authored content. The leading zero-shot detector, Detect- GPT (Mitchell et al.) (023), showcases commendable performance but is marred by its intensive computational costs. In this paper, we introduce the concept of conditional probability curvature to clucidate discrepancies in word choices be- tween L.LMs and humans within given context. Utilizing this curvature as a foundational metric, we present Fast-DetectGPTT D an optimized zero-shot de- tector, which substitutes DetectGPT's perturbation step with more efficient sam- pling step. Our evaluations on various datasets, source models, and test conditions indicate that Dest-DetectGPT not only surpasses DetectGPT by a relative around 75% in both the white-box and black-box settings but also accelerates the detec- tion process by a factor of 340, as detailed Table T
Method
M-Model Generations
ChatGPT/GPT-4 Generations +
Speedup t
DetectGPT
0.9554
0.7225
0.9887
0.9338
Fast-DetectGPT
340x
(relativer 74.7%)
(relative? 76.1%
Table 1: Detection accuracy (measured in AUROC) and computational speedup for machine- generated text detection. The white-box setting (directly using the source model) is applied to the methods detecting generations produced by five source models (5-model), whereas the black- box sefting (utilizing surrogate models) targets ChatGPT and GPT-4 generations. Results are aver- aged from data in Table 2 for 5-model generations and Table for ChatGPT/GPT-4, where "relative"' is calculated by (new old)/( (1.0 - old), representing how much improvement has been made relative to the maximum possible improvement. Speedup assessments were conducted using the XSum news dataset, with computations Tesla A100 GPU.
"Corresponding author.
'The code and data released https:ttp////m. http://ttww....oooooooaaanaaaaea

---

Published as a conference paper at ICLR 2024
GPT-2: gpt2-xl
Neo-2.7: gpt-neo-2.78
GPT-J: got-j-6B
NeoX: gpt-neox-20b
100
Humon
Machinc
75
50
0
-2
-2
2
4
-2
0
Conditional Probability Curvature
Figure/ 1: Distribution of conditional probability curvatures of the original Hulman-writter passages and the machine machine-generated passages by four source models on 30-token prefix from XSum.
INTRODUCTION
Large language models (LLMs) like ChatGPT OO
2022), PaLM (Chowdhery al. 2022
and GPT-4 (OpenAl, (023) have dramatically influenced both industrial and academic landscapes. These models have transformed productivity in diverse fields such as news reporting, story writing, and academic research (M Alshater 2022 Yuan et al. 2022 Christian 2023). However, their misuse also introduces concerns especially regarding fake news (Ahmed et al.] 202T), malicious product reviews (Adelani et al. 2020), and plagiarism (Lee al.) (023). The sheer fluency and coherence of content generated by these models make challenging, even for experts, to determine its human or machine origin (Ippolito et al. 2020; Shahid et al. (022). Addressing this issue necessitates reliable machine-generate text detection methods (Kaur et al. 2022 Chen & Shu 2023). )
be grouped into two main categories: supervised classifiers (Solaiman al. 2019 Fagni et al. 2021; Mitrović et al., 2023) and zero-shot classifiers (Gehrmann et al., 2019 Mitchell et al. 2023 Su et al. 2023). While supervised classifiers excel within their specifie training domains, they falter when confronted with text from diverse domains or unfamiliar models (Bakhtin ctt a
Existing detectors
Uchendu Uchhend al. 2020 Pu et al. 2023). Zero-shot classifiers, using a pre-trained language model directly without finetuning, are immune to domain-specific degradation and are on par with supervised classifiers on detection accuracy. This stems from their need for "universal features" that can function across multiple domains and languages (Gehrmann et al. 2019 Mitchell et al. 2023).
typical zero-shot classifier, DetectGPT (Mitchell al. 2023). works under the assumption that machine-generater text variations typically have lower model probability than the original, while human-written ones could go either way. Despite its effectiveness, employing probability curvature demands the execution of around one hundred model calls or interactions with services such as the OpenAl API to create the perturbation texts, leading to prohibitive computational costs.
In this paper, we posit a new hypothesis for detecting machine-generater text. By viewing text generation as a sequential decision-making process on tokens, our core assertion that humans and machines exhibit discernible differences in token choice given context. More specifically, ma- chines lean towards tokens with higher statistical probability due to their pre-training on large-scale human-writter corpus, while humans individually exhibit such bias because they craft sentences based on underlying meanings, intentions, and contexts rather than data statistics. As consequence, the conditional probability function p(ā(x) ()) reaches its maximum point at machine-generated (ev- idenced by positive curvature at that point). Our empirical observation supports this hypothesis Specifically, the conditional probability
across diverse datasets and models, machine-generated generated texts typically hovers around 3, whereas human -enaanneeee texts exhibit curvatures close to O.
curvature of machine
According to the above observation, we present Fast-DetectGPT. aiming to classify if passage was produced by a particular source model, as outlined in Figure our approach begins by sampling alternative word choices at each token (step 1). Subsequently, we assess the conditional probabilities of these generated samples (step 2) and combine them to arrive at detection decision (step 3). Our empirical evaluation demonstrates the superior detection accuracy of Destt-etectGPTTT over DetectGPT, showcasing a noteworthy relative boost about 75% in both white-box and black-box settings. Intriguingly, in the black-box setting, Fast- DetteetttT even trumps DetectGPT's white-box performance by an average of 28%. Moreover, it aptly flags 80% of ChatGPT-crafted content, while only misidentifying 1% of human compositions.
contrast to DetectGPT,
2

---

Published as a conference paper at ICLR 2024
a) DetectGPT
Passage "President Joe Biden claimed interview with The Weather Channel..."
Yes
(1) Perturb
(2) Score
(3) Compare
x from GPT
p(r)
T5
GPT
p(a.)
L log stn)
from other
No
N
p(Ex)
b) Fast-DetectGPT
Passage X: "President Joe Biden claimed interview with The Weather Channel..."
Yes
(1) Somple
(2) Cond. Score
(3) Compare
x from GPT
I *+ GPT
By
GPT
p(ā1(a))
log Floiab pay
x from other
Figure 2: Cast-DetectGPP v.s. DetectGPT (Mitchell et al. 2023 Fast- Destt-etectGPT uses a con- ditional probability function p(ž (z) defined in Eq. 2 Notably, Dett-DetettGTT invokes the sampling GPT once to generate all samples and similarly calls the scoring GPT once to evaluate all samples, while DetectGPT interacts with the perturbation model T5 produce one perturbation per call, and summons the scoring model GPT for each perturbation assessment. The threshold should be chosen balance the false and positive rates in practice.
Our main contributions are threefold: 1) unveiling and validation a new hypothesis that human and machine select words differently given a context, 2) proposing conditional probability curvature as a new feature to detect machine-generatee magnitude, and 3) achieving the best average detection accuracy in both white-box and black-box settings compare existing zero-shot text detectors.
text, reducing the detection cost by two orders of
METHOD
2.1
TASK AND SETTINGS
Our objective the zero-shot detection of machine-genee -eeeerrte text, treating the challenge as binary classification problem (detailed in Appendix A). Given passage which may be autanne or produced by source model, the goal is to discern whether is machine- machine-generated generated.
In the white-box setting, we have the privilege of accessing the possible source model that a passage either written by human or generated by this source model. We use the source model to aid in scoring the candidate passage to inform the classification decision in the setting. Conversely, in the black-box setting, we operate without access to the source model. Instead, we rely on surrogate models to score the passage. Underpinning this approach is the assumption that language models, due to their training on vast human-authored corpora, inherently share characteristic features.
2.2 DETECTGPT BASELINE
Formally, given an input passage x and the possible source model pe. DetectGPT uses the source model for scoring (the white-box setting). Together with predefined perturbation model 4p- De- tec1GPT encapsulates the probability curvature as:
d(x, Po, (.) log (x Eg (-Im) [log po (i)),
(1)
where 7 is perturbation produced by the masked language model gu (-1z). When emerges from sampling from the source model pa. d(z, pe, 9p) tends to be positive, while for passage x written by human, d(x, po. 9p) tends be zero. po is also called the scoring model in this method, which is used to score the log probabilities.
The Detection Process. To estimate the expectation Eznq, (+aa) log po (œ). DetectGPT employs sampling approach. Typically, it generates around a hundred variations the input text and then computes the average of the log probabilities associated with these variations. The detection process
3

---

Published as a conference paper at ICLR 2024
2 where DetectGPT advocates three-step detection process, which in- clude: 1) Perturb - generating slight rewrites of the original text using a pre-trained mask language model; 2) Score evaluating the probability of the text and its rewrites using pre-trained GPT language model; 3) Compare- estimating the probability curvature and making the final decision accordingly.
summarized Figure
The Challenge. The probability function (o( models I in a Markov chain. Even if disparities between I and I are slight, amounting to changes in merely about 15% of the tokens, the entire Markov chain demands reevaluation for accurate probability estimation. This slight variation within the Markov chain mandates invoking the scoring model afresh for each variation, as the Score step in Figure 2 denotes. In this paper, we deviate from assessing the probability function across the entire Markov chain. Instead, we focus on evaluating the conditional prohability function for each individual token, thereby eliminating the need for repetitive scoring.
2.3 FAST-DETECTGPT
Past-DetectGPi operates on the premise that humans and machines tend to select different words during the text-generation process, with machines exhibiting propensity for choosing words with higher model probabilities. The hypothesis rooted in the fact that LLMss, pre-trained on the large- scale corpus, mirror human collective writing behaviors instead of human individual writing behav- ior, resulting in discrepancy in their word choices given context.
The hypothesis is also substantiated to some extent prior observations in the literature (Gehrmann et al. 2019; [Hashimoto et al.) 2019; Solaiman et al. 2019 Mitrović et al., 2023] Mitchell et al. 2023 which have indicated that machine- generreed text typically boasts higher average log prob- ability (or lower perplexity) than human-wriie written text. However, instead of solely relying on the assumption of higher average log probability for machine machine-generated text, our approach posits the presence of positive curvature within the conditional probability function specifically for machine- generated text.
Given passage x and a model pe. we define the conditional probability function as
po (0(
II po (
where the tokens žy are independently predicted given x. As a special case, po equals to po (cr).
Specifically, we replace the probability funetion po (ă) in DetectGPT with the conditional probabil- ity function po (f We estimate the curvature at the point x by comparing the value of po (æ/zz) with the values of alternative token choices po po (z)) pa (zr[z: has bigger value than po(a))), the function has a positive curvature at the point v, indicating that i is more likely machine-generated. Otherwise, the function has close-to-zero curvature at the point x, suggesting that x more likely human- human-written. We demonstrate the curvature distributions of human- written and machine- generated texts in Figure where we can see that human-written texts are concentrated around the zero curvature.
Formally, given an input passage x and the possible source model pe (the white-box setting), we quantify the conditional probability curvature
d(z, ps. 9p) = log po (z|x) Ā
(3)
where
p Ea Ea-qo [log Po [3[x]] and x² = Ea-g (z|z) [log po (f(x)) [
(4)
denotes the expected score of samples ē generated by the sampling model q, (-1z). and o² the expected variance of the scores. We approximate j using the average log probability of the random samples, and x² using the mean of sample variances.
Conditional Independent Sampling, The independent sampling of alternative tokens the key to the efficiency of Fast-DetectGPT. Specifically, sample each token ã, from q, mzz)) given the fixed passage x without depending on other sampled tokens. In practice, we can simply generate 10,000 samples (our default setting) by line of PyTorch code: samples
4

---

Published as a conference paper at ICLR 2024
Algorithm Fast-DetectGPT machine-generatee text detection.
Input: passage æ. sampling model 4,- scoring model ps. and decision threshold Output: True - probably machine-gentt -egheine-geneated. False probably human-written.
1: function FASTDETECTGPTE: 9p. pa)
02 q ()) E [1..N]
P Conditional sampling D. Estimate the mean
3:
p + N L, log pa (B((z)
x²
(log pe (Z.(x)) x2)2
D Estimate the variance
5: d < (z) - p)/ō
D Estimate conditional probability curvature
6:
return d, >
borch.distributions.. caapeirbbuttoons.ctggooicallllll catogoricaa.CCteggriccllaaa Caw.ggriccllllgggspppooo...aapaa log probability distribution of 9u (Ty (ج²)) for from 0 to the length of x.
where the Iprobs is the
The sampling process plays a pivotal role in guiding us toward the solution. To discern whether human-authored, it is essential to compare it against a range of alternative tokens in the same context. By sampling substantial number of alternatives (say 10,000), we can effectively map out the distribution of their log Po (Ø, (ı=,)) values. Placing the log (2 (x) value of the passage token within this distribution provides clear view of its relative position, enabling us to ascertain whether an outlier or a more typical selection. This fundamental insight forms the core rationale behind the development of Fast-DetectGPT.
token within given context is machine-generatee
The Detection Process. As Figure 2p shows, Fast-DetectGPT proposes a new three-step detection introduce sampling model to generate alternative samples i given the condition æ; 2) Conditional Score the conditional probability can be easily obtained by single forward pass of the scoring model taking in the same predictive distribution, so we do not need multiple model calls, and 3) Compare conditional probabilities of the passage and samples compared calculate the curvature. More implementation details are described in Algorithm
process, including 1) Sample
the input. All the samples can be evaluated
We find that the "Sample" and "Conditional Score" steps be merged and have analytical solution instead of sampling approximation, as described in Appendix B Furthermore, when we use the same model for sampling and scoring, the conditional probability curvature has close connection to the simple Likelihood and Entropy baselines as follows.
Connection Likelihood and Entropy. Utilizing a singular model for both sampling and scoring enables the combination of these processes into single step, necessitating only one model call. Given this, the conditional probability curvature can be suceinctly expressed as
d(x, po) log logpo(r(z)
(5)
where ū = -Ezppo(()a) [log pa((z)) and σ² = -E [B [log =.)
Intriguingly, the curvature's numerator reveals itself to be the sum of the baseline methods: Likeli- hood (log (x)) and Entropy (-ū). While Likelihood and Entropy have been established as foun- dational baselines for zero-shot machine-generated text detection over the years (Lavergne et al.) 2008 Gehrmann et al.) 2019; Hashimoto et al.) 2019 Mitchell et al.) (2023), the discovery that their elementary combination can yield competitive detection accuracy was unforeseen.
EXPERIMENTS
3.1 SETTINGS
Datasets. We follow DetectGPT using six datasets to cover various domains and languages, in- cluding XSum for news articles (Narayan et al. 2018), SQuAD for Wikipedia contexts (Rajpurkar et al. 2016), WritingPrompts for story writing (Fan al. 2018), WMT16 English and German for different languages (Bojar et al. 2016), and PubMedQA for biomedical research question answering (Jin et al. 2019). We randomly sample 150 to 500 written examples per dataset negative samples and produce equal numbers of positive samples by prompting the source model with the first 30 tokens of the humtn-written text (for PubMedQA, we only the question the prompt).
5

---

Published as a conference paper at ICLR 2024
Method
GPT-2
OPT-2.7
Neo-2.7
GPT-J
NeoX
Avg.
The White-Bon Box Setting
Likelihood
0.9125
0,8963
0.8900
0.8480
0.7946
0.8683
Entropy
0,5174
0,4830
0,4898
0.5005
0.5333
0,5048
LogRank
0,9385
0.9223
0,9226
0.8818
0.8313
0.8993
LRR
0,9601
0.9401
0,9522
0.9179
0.8793
0,9299
DNA-GPT
0,9024
0.8797
0.869
0.8227
0.7826
0.8513
NPR &
0,9948f
0,9832f
0,9883
0,9500
0,9065
0,9645
DetectGPT (T5-3B/") +
0.9917
0.9758
0.9797
0,9353
0.8943
0.9554
Fast-DetectGPT (+/+)
0.9967
0.9908
0.9940f
0.9866
0.9754
0,9887
(Relativet)
60.2%
62.0%
70.4%
79.3%
76.7%
74.7%
The Black-Box Setting
DetectGPT (TS- 11-33/NNe--2,, 2.7) & Dest-Detet (GPT-J/Neo-2.7) (Relativet)
0.8517
0.8390
0.9797
0.8575
0.8400
0.8736
0.9834
0.9572
0.9984
0.9592
0.94041
0.96771
88.8%
73.4%
92.1%
71.4%
62.8%
74.5%
Table 2: Zero-shot detection on passages from five source models, averaging AUROCs across XSum, SQuAD, and WritingPrompts from detailed Table [ in Appendix D.I. Typically, the source model is employed for scoring, except that DetectGPT (T5-3B/Neo-2.7) and Fast-DetectGPT (GPT-J/Neo- 2.7) in a black-box setting utilize surrogate Neo-2.7 model for scoring. While DetectGPT leverages T5-3B for perturbation generation, Fast-DetectGPTT either resorts to the source model or a surrogate GPT-J for sample generation. represents the second-best score. 4 indicates methods that invoke models multiple times, thereby significantly increasing computational demands.
We evaluate the methods nachine-generated text produced by different source models from 1.3B to 1,800B, described in Appendix C.T We call most of the models locally except GPT-3, ChatGPT, and GPT-4 through OpenAI API.
Baselines. For zero-shot classifiers, we mainly compare Fast-Detect DetectGPT with DetectGPT (Mitchell al. ) as well its enhanced variant, NPR (Su al.) 2023) and DNA-GPT (Yang et al. These represent the baseline methodologies we aim to expedite. Furthermore, we juxtapose Dest--etectGPT with established zero-shot techniques, such as Likelihood (mean log probabilities), LogRank (average log of ranks in descending order by probabilities). Entropy (mean token entropy of the predictive distribution) (Gehrmann et al. 2019 Solaiman al. 2019 [ppolito et al.) 2020),
2023
and L.R (an amalgamation of log probability and log-rank) (Su et al. 2023). Regarding supervised
classifiers, mest-DetcctGPT1 is benchmarked against existing supervised classifiers, including GPT-2
detectors based on MoBERTa-base/large (Liu et al. 2019) crafted by OpenA| and GPTZero Tian
& Cui 2023).
3.2
MAIN RESULTS
We generate 500 samples per dataset among XSum, SQuAD, and WritingPrompts for the following experiments, measuring the detection accuracy in AUROC (see Appendix[A]
Inference Speedup. We compare the inference time (excluding the time for initializing the model) of Fast-DetecttPPT and DetectGPT on Tesla A GPU using XSum generations from the five models in Table 2 Despite DetectGPT's into 10 batches, it still requires substantial computational resources. It totals 79,113 seconds (ap- proximately 22 hours) over five runs. In contrast, Fast-DeteetGPT completes the task only 233 seconds (about 4 minutes), achieving a remarkable speedup factor of approximately 340x, highlight- ing its significant performance improvement.
of GPU batch processing, splitting 100 perturbations
White-Box Zero-Shot Machine-Generatee Text Detection. We evaluate zero-shot methods us- ing each source model for scoring and typically Fast-DetectGPT using the source model also for sampling. The averaged scores are shown in Table 2 with more detailed results dataset re- ported in Table Appendix D.T] Fast-DetectGPP achieves the best average AUROC on the three datasets, outperforming DetectGPT by relative 74.7% and its enhanced variant, NPR, by 68.2%. Notably, larger source models amplify this relative improvement, demonstrating the advantage of Fast- Sost-DetectGPT1 in detecting text produced by larger models.

---

Published as a conference paper at ICLR 2024
ChatGPT
GPT-
Method
XSum
Writing
PubMed
Avg.
XSum
Writing
PubMed
Avg.
RoBERTa-base
0.9150
0.7084
0.6188
0.7474
0.6778
0.5068
0,5309
0.5718
RoBERTa-large
0.8507
0.5480
0.6731
0.6906
0,6879
0.3821
0.6067
0.5589
GPTZero
0.9952
0.9292
0.8799
0.9348
0.9815
0.8262
0.8482
0.8853
Likelihood (Neo-2.7)
0.9578
0.9740
0.8775
0.9364
0.7980
0.8553
0.8104
0.8212
Entropy (Neo-2.7)
0.3305
0. 0.1902
0.2767
0.2658
0,4360
0.3702
0.3295
0.3786
LogRank(Neo22.7)
0,9582
0.9656
0.8687
0.9308
0.7975
0.8286
0.8003
0.8088
LRR (Neo-2.7)
0,9162
0.8958
0.7433
0.8518
0.7447
0.7028
0.6814
0.7096
DNA-GPTT N.70-.77)
0,9124
0.9425
0.7959
0,8836
0.7347
0.8032
0.7565
0.7648
NPR (T5- IB/Nco-2.7)
0,7899
0.8924
0.6784
0.7869
0.5280
0.6122
0.6328
0.5910
DetectGPT (T5- 1
0.8416
0.8811
0.7444
0.8223
0.5660
0.6217
0.6805
0.6228
Fast-Detect (GPT-J/Nco-2.7)
0.9907
0.9916
0.9021
0.9615
0.9067
0.9612
0.8503
0.9061
(Relative t)
94.1%
92.9%
61.7%
78.3%
78.5%
89.7%
53.1%
75.1%
Table 3: Detection of CharGPT and GPT-4 generations. The black-box settings used for all zero-shot methods, where the Likelihood provides the strongest baseline. A comparison of GPT-3 generation detection is provided in Appendix D.2 where observe similar hierarchy in accuracy.
D.2
Black-Box Zero-Shot Machine-Generaate Text Detection. Table 2 further contrasts Fast-
DetectGPT and DetectGPT in a black-box setting, employing a surrogate model, Neo-2.7 (empir- ically superior among GPT-2, Neo-2.7, and GPT-J) for scoring. Fast-DetectGPT (GPT-I/Neo-2.7) achieves a relative AUROC enhancement of 74.5% over DetectGPT (T5-3B/Neo-2.7) across the datasets. Specifically, the boost in Wikipedia context (SQuAD) averages 0.16 AUROC (detailed in Table 5lbleBBnn in Appendix D.I.. Moreover, Fast- Dest-DetectGPT (GPT-J/Neo-2.7) outperforms DetectGPT (T5-3B/*) by relatively 27.6% on average. Such outcomes solidify Dest-DetecGGPT's potential in the black-box setting
versatile text detector across diverse domains and source models.
Ablation Study. We study the impact of the choice of the sampling model Op and the impact of the normalization ā in Eq. [ on the detection accuracy. Experiments show that properly selected sampling model can further enhance the performance relatively about 27%, while the normalization enhances the performance by 10%. More details are described in Appendix
Dess-eteetGPT the white-box setting by
3.3 RESULTS IN REAL- WORLD SCENARIOS
We further assess Fast-DetectGPT black-box setting using passages generated by GPT-3, Chat- GPT, and GPT-4 to simulate real-world scenarios. Using parameters and prompts delineated in Appendix C.2 we produced 150 samples per dataset and source model. As evidenced in Table 3. FaSt-DetectGPT consistently exhibits superior detection proficiency. It surpasses DetectGPT by relative AUROC margins of 78.3% for ChatGPT and 75. % for GPT-4. When compared to the supervised detectors SaogERTa-bae/large and GPTZero, Fast-Detec1GPT achieves overall higher accuracy. Collectively, these outcomes underscore the potential of Fast-DetectGPT working in real- world scenarios.
Interestingly, the commercial model GPTZero performs the best on news (XSum) but worse on sto- ries (WritingPrompts) and technical writings (PubMedQA). indicating that the model may mainly be trained on news generations. The Likelihood detector emerges as the strongest baseline, which diverges from the results on open source models presented in Table where Likelihood trails De- tectGPT and NPR. A consistent trend is evident with GPT-3 generations (Appendix D.2). In com- parison, Past-Detec1GPT offers more robust and consistent performance.
3.4 USABILITY ANALYSIS
Interpretation of AUROC. In real-world scenarios, our concern extends beyond overall detection accuracy; we are particularly interested in the recall (the true positive rate) while minimizing the likelihood of making type-I errors (achieving a low false positive rate). As depicted in Figure B when applied ChatGPT- -egnerrttte content, Fast Dest--etectGPT achieves recall of 87% for machine- generated texts with only 1% misclassification of aumann-rrtten text as machine- -eaainnngeeerteed.. When the tolerance for false positives increases to 10%, the recall reaches 98%. When applied to GPT-4

---

Published as a conference paper at ICLR 2024
ChatGPT
GPT-4
ChatGPT
GPT-4
1.0
0.8
Postive
0.6
0.4
0.6
0.2
0.0
0.2
10-
10
10a
10
10-1
30
60
90
120
150
180
60
90
120
150
180
False Positive Rate
False Positive Rate
Passage Length (Words))
Passage Length (Words)
Figure 3: ROC curves in log scale evaluated on stories (WritingPrompts), where the dash lines denote the random classifier.
Figure 4: Detection accuracy (AUROC) on story passages (WritingPrompts) truncated to target number of words.
generations, the task becomes significantly more challenging but Fast-DetectGPT still achieves recall of 89% on the condition of a false positive rate less than 10%. These outcomes underscore the potential of Fast-DetectGPT in effectively detecting texts generated by state-of-the-art large language models.
Robustness Different Passage Lengths. Zero-shot detectors supposed to perform shorter passages due to their statistical nature. We conduct evaluations by truncating the passages various target lengths. As Figure 4 illustrates, the detectors show consistent trends passages produced by ChatGPT, where the detection accuracy generally increases for longer passages. In con- trast, on passages from GPT-4, the detectors show inconsistent trends. Specifically, the supervised detectors show performance decline when the passage length increases, while DetectGPT expe- riences an increase at the beginning followed by an unexpected decrease when the passage length exceeds 90 words. We speculate the monooonoo trends of the supervised detectors and Detect- GPT are rooted in their perspective on handling the passages as a whole chain of tokens, which does not generalize to different lengths. In contrast, Fast-DetectGPT exhibits a consistent, monotonic increase in accuracy as passage length grows, indicating the robust performance Best-DetectGPT across passages of varying lengths.
Robustness across Domains and Languages. Detectors expected to generalize to differ- ent domains and languages for higher usabil- ity. We compare Fast-DetectGPT against pervised detectors on both in in-disrribution and out-distribution datasets. Figure 5 reveals that Fast-DetectGPT achieves competitive detection accuracy on the in-distribution datasets XSum and WMT16-English. However, it significantly outperforms supervised detectors on the out- distribution datasets PubMedQA and WMT16- Genman. Moreover, it is noteworthy that Fast- DetectGPT consistently outperforms DetectGPT across all four datasets. These results underscore the robustness of Fast-DetectGPT across diverse domains and languages.
XSum (GPT-2)
WMT16-EN (mGPT)
1.0
Detection Accuracy (AUROC)
0.8
0.6
PubMedQA (PubMedGPT)
WMT16 De (mGPT)
1.0
Anets
parte
0.8
lodhaage
Likilted
Loprank
dast Destcr
Likeilonn
Uopiaant
netcctor
Detection Method
Figure 5: Compare with supervised detectors, evaluated in AUROC. We generate 200 test sam- ples for each dataset and source model.
Robustness against Decoding Strategies. Machine systems employ various decoding strategies, including top-k sampling, top-p (Nucleus) sampling, and temperature sampling with a constant T. Our experiments evaluate these strategies using the five models and three datasets mentioned in Table by setting A 40, p 0.96, and T = 0.8 for all cases. In the white-box setting, Fast- DetectGPT consistently emerged superior across all sampling strategies.
surpassed DetectGPT with relative margins of 95% in Top-p sampling, 81% in Top-k sampling, and a striking 99% in temperature sampling, as elaborated in Table 9 in Appendix F.I Similar relative improvements also achieved in the black-box setting. These results underscore the consistent performance of Fast- Dete-Deeet in detecting text generated through diverse decoding strategies.
Robustness under Paraphrasing Attack. We analyze the robustness under paraphrasing attack and propose decoherence attack, finding that Fast-DetectGPT consistently outperforms both zero- shot and trained detectors, illustrated in Appendix F22
8

---

Published as a conference paper at ICLR 2024
4 DISCUSSION
Fast-DetectGPT performs about 65% better in white-box settings than black-box ones. Industrial services could leverage the white-box setting enhance the content authorship for proprictary LLMs like ChatGPT and GPT-4 by exposing conditional probability curvature in the service API, without significant extra cost on the computation of the feature.
From the research perspective, the black-box setting may have unforeseen potential. The best model for this setting remains unclear, and may depend on factors like model size, training corpus breadth, and training process convergence. These factors warrant further investigation to provide clarity and guidance in the development of black-box detection methods.
We further discuss the broader implications of Fast-DetectGPT in Appendix thorship and watermarking.
G
covering text au-
Limitations and Future Work. Our initial research impetus centered on accelerating the detection process of DetectGPT. However, Fast-DetectGPT not only accelerates this process but also brings about notable enhancements in detection accuracy. In this paper, our focus is predominantly on these empirical advancements, with theoretical explorations earmarked for future endeavors.
Furthermore, Fast-DetecGGPT's design leans on pre-trained models to span a multitude of domains and languages. This presents challenge black-box setting, as no single model can seamlessly span all linguistic territories and domains. This is due to the intrinsic nature of pre-trained models being tailored to specific domains and languages.
RELATED WORK
Current research primarily concentrates supervised methods, involving the training of classifiers
to distinguish between machine -enerrttd and human- written text (Jawahar et al.] 2020).
These
classifiers typically trained using bug-of-wordh (Solaiman et al. 2019: Fagni
al.
2021 or
neural representations (Bakhtin et al. 2019; Solaiman et al. [209 Uchendu et al.) 2020
Ippolito
et al. 2020 Fagni et | [021] Yan al. 2023 Pu et al. 2023] Mitrović et al. 2023 et al.;
2023 It has been observed that these trained classifiers often exhibit overfitting tendencies, adapting
100 closely to the specific distribution of text domains and source models during training (Bakhtin et al. 2019, Uchendu et al. 2020). which consequently leads to limited generalization capabilities when exposed to -iitt--diittiuutinn data (Pu et al., (023). To address this challenge, our research focuses on zero-shot detection, aiming to identify "universal features" that can be applied to different domains, source models, and languages.
Existing zero-shot detectors primarily rely statistical features, leveraging pre-trained large lan- guage models to gather them. These features encompass a range of measures, including relative entropy and perplexity (Lavergne et al. 2008), bag-of-words, average probability, and top-K buck- ets (Gehrmann et al. 2019), likelihood (Hashimoto et al., 2019 Solaiman et al. 2019), probability curvature (DetectGPT) (Mitchell et al. 2023), normalized log-rank perturbation (NPR) (Su et al., 2023). and divergence between multiple completions of a truncated passage (DNA-GPT) (Yang al. 2023). Due to their statistical nature, zero-shot methods generally exhibit higher detection curacy on longer passages (Lavergne et al.] 2008). In this paper, we introduce a novel "conditional probability curvature" for machine-generatee text detection. Differing from previous probability curvature or completion divergence approaches that require numerous model calls (variating from 10 to 100), our new feature only necessitates single model forward pass, significantly expediting the detection process. Importantly, this new feature markedly enhances detection accuracy.
6 CONCLUSION
Our investigation reveals that conditional probability curvature on the token level serves as a more fundamental indicator of machine-genent texts, validating proposed hypothesis concerning the distinction between machine and human text generation processes. Building upon this new hypothesis, our detector Fast-DeteetGPT accelerates upon DetectGPT by two orders of magnitude. Experimental results further demonstrate that Fast- Messt-DeeccGPTT also significantly improves detection accuracy by approximately 75% in both white-box and black-box settings.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their valuable feedback. This work is funded by the National Natural Science Foundation of China Key Program (Grant No. 62336006) and the Pioneer and "Leading Goose" R&D Program of Zhejiang (Grant No. 2022SDXHDX0003) Yanbin Zhao is supported by the National Natural Science Foundation of China (Grant No. 12201158).
ETHICAL CONSIDERATIONS AND BROADER IMPACT
Fast- GG,t-DetectGPT, serving as highly efficient detector for machine-generater text, holds promise in enhancing the integrity of Al systems by combating issues like fake news, disinformation, and academic plagiarism. However, akin to other methods reliant on Large Language Models (LLMs), susceptible inheriting biases present in the training data. Notably, emphasized by Liang al. (2023), LLM-based detection systems may exhibit an elevated false-positive rate when confronted with text from non-native English speakers. Given the widespread and diverse utilization of such technologies, this presents a notable concern.
An immediate suggestion is to substitute the underlying LLMs in Fast-DetectGPT with alternative models trained on more varied and representative corpora. Additionally, we advocate for community involvement in the ongoing efforts develop more inclusive LLMs, development that would benefit not only Fast-DetectGPT but also similar systems at large.
REFERENCES
Sahar Abdelnabi and Mario Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium Security and Privacy (SP), pp. 121- 140. IEEE, 2021.
David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. Generating prrserint- fake online reviews using neural language models and their human- and machine-basee detection. In Advanced Information Networking and Applica- fions: Proceedings of the 34th International Conference on Advanced Information Networking and Applications (AINA-2020) pp. 1341-1354. Springer, 2020.
Alim Al Ayub Ahmed, Ayman Aljabouh, Praveen Kumar Donepudi, and Myung Suh Choi. De- tecting fake news using machine learning: systematic literature review, arXiv preprint arXiv:2102.04458, 2021.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Mare' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint 03351, 2019.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorflow, March 2021.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He. Connor Leahy, Kyle McDonell, Jason Phang. et al. Gpt-neox-20b: An open-source autoregressive language model. In Proceedings of BigScience Episode# 5-Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pp. 95-136, 2022.
Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 131-198, 2016.
Elliot Bolton, David Hall, Michihiro Yasunaga, Tony Lee, Chris Manning, and Percy Liang. edu/news/
Stanford CRFM Introduces PubMedGPT 2.7B. https: h/hp::://aa....// st.i.stann anford. Sttanford-crfme wtw.-rrrf--inntooose----ee-eeeee pudeeddd 27b, December 2022.
10

---

Published as a conference paper at ICLR 2024
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Canyu Chen and Kai Shu. Combating misinformation in the age of Ilms: Opportunities and chal- lenges. arXiv preprint arXiv: 231 23156666,, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint S.Xii:2220.023111. 2022.
Jon Christian. Cnet secretly used on articles that didn't disclose that fact, staff say. Futurusm, January, 2023.
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. Tweepfake: About detecting deepfake tweets. Plos one, 16(5):e0251415, 2021.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Arnual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 111-116, 2019.
Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, and Cho-Jui Hsieh. Watermarking pre-trained language models with backdooring. arXiv preprint ar arXiv:2210.07543, 2022.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. arXiv preprint wrXiv:1944.02792. 2019.
Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808-1822, 2020.
Zunera Jalil and Anwar M Mirza. A review of digital watermarking techniques for text docu- ments. In 2009 International Conference on Information and Multimedia Technology, pp. 230- 234. IEEE, 2009.
Ganesh Jawahar, Muhammad Abdul-Mageed, and VS Laks Lakshmanan. Automatic detection of machine generated text: A critical survey. Proceedings of the 28th International Conference on Computational Linguistics, pp. 2296-22162 2309, 2020.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNNP)), pp. 2567 2019.
Nurul Shamimi Kamaruddin, Amirrudin Kamsin, Lip Yee Por, and Hameedur Rahman. A review of text watermarking: theory, methods, and applications. IEEE Access, 6:8011-8022 2018.
Davinder Kaur, Suleyman Uslu, Kaley Rittichier, and Arjan Durresi. Trustworthy artificial intelli- gence: review. ACM Computing Surveys (CSUR), 55(2): 2022.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301. /0026,, 2023.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 11118099188666 2022.

---

Published as a conference paper at ICLR 2024
Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative tropy scoring. In Proceedings of the 2008 International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse-Volume 377, pp. 27-31, 2008.
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pp. 3637-3647, 2023.
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Deepfake text detection in the wild. arXiv preprint arXiv:2305. 13242, 2023.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors biased against non-native english writers. arXiv preprint WrXiv:2304.028// 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 11692, 2019.
Muneer M Alshater. Exploring the role of artificial intelligence in enhancing academic performance: A case study of chatgpt. Available at SSRN, 2022.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. De- tectgpt: Zero-shot machine- machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023.
Sandra Mitrović, Davide Andrcoletti, and Omran Ayoub. Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chalgpt-generated text. arXiv preprint arXiv:2301.13855 2023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018.
OpenAI. ChatGPT.
http:/hhhtttp....icc. opw.ai.ennaa........ com/
December 2022.
OpenAl. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhattacharya, Mobin Javed, and Bimal Viswanath. Deepfake text detection: Limitations and opportunities. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 1613-1630, IEEE, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.1 //100////116,.. 1166, 2023.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
Wajiha Shahid, Yiran Li, Dakota Staples, Gulshan Amin, Saqib Hakak, and Ali Ghorbani. Are you a cyborg, bot or human?- survey on detecting fake news spreaders. IEEE Access, 10: 97069-27088 2022.
12

---

Published as a conference paper at ICLR 2024
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. igpt:: Few-shot learners go multilingual. arXiv preprint arXiv:2204.07580, 2022.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint (at.::9900.00333 2019.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank informa- tion for zero-shot detection of machine-generated text. arXiv preprint arXiv:2306.0554 2023,
Edward Tian and Alexander Cui. Gptzero: Towards detection of ai-generated text using zero-shot and supervised methods, 2023.
/ttp:://ppttrrrr me
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13377 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 0928888,, 2023b.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text genera- tion. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 8384-8395, 2020.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: Billion Parameter Autoregressive Language Model. https: //tp://ggttt.t.t/ com/ k kingoflolzi wtt.:o///mmmmetttsoooooooooo jax May 2021.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint arXhv:2109.01652, 2021.
Duanli Yan, Michael Fauss, Jiangang Hao, and Wenju Cui. Detection of ai-generated essays in writing assessment. Psychological Testing and Assessment Modeling, 65(2):125-144, 2023.
Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint arXiv:2305. 17359, 2023.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Worderaft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces, pp. 841-852, 2022.
Susan Zhang. Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205. 01068, 2022.
13

---

Published as a conference paper at ICLR 2024
A ZERO-SHOT DETECTION TASK AND SETTINGS
Our research centers on zero-shot detection of machine machine-generated text, under the premise that our model has not undergone training using any machine-generate text. This distinguishes our approach from conventional supervised methods, which commonly employ discriminative training strategies acquire specific syntactic or semantic attributes customized for machine-genee gencineegenerated text. In con- trast, our zero-shot methodology capitalizes on the inherent capabilities of large language models to identify anomalies that function markers of machine gennrrtted content.
The White-box Sefting, Conventional zero-shot methodologies often operate under the assumption that the source model responsible for generating machine- -enerrttd text accessible. We refer to this context as the white-box setting, where the primary goal is to distinguish machine-generann texts produced by the source model from those generated by humans. In this white-box setting, our detection decisions are dependent on the source model, but it is not mandatory to possess detailed knowledge of the source model's 's architecture and parameters. For instance, within the white-box framework, a system like DetectGPT utilizes the OpenAl API to identify text generated by GPT-3, without requiring extensive knowledge of the inner workings of GPT-3.
The Black- box Setting, In real-world situations, there could be instances where we lack knowledge about the specific source models employed for content generation. This necessitates the development of versatile detector capable of identifying texts generated by variety of automated systems. We term this scenario the black-bot setting, where the objective is to differentiate between machine- generated texts produced by diverse, unidentified models and those composed by humans. In this context, the term "black box" signifies that we lack access to information about the source model or any details pertaining to it.
Evaluation Metric (AUROC). Instead measuring the detection accuracy with a specific threshold in Figure 2). we measure the detection accuracy the area under the receiver operating charac- teristic (AUROC), profiling the detectors on the whole spectrum of the thresholds. AUROC ranges from 0.0 to 1.0, mathematically denoting the probability of a random machine- geneertte text hav- ing higher predicted probability of being machine generated than a random buman-written text. Typically, an AUROC of 0.5 indicates a random classifier and an AUROC of 1.0 indicates a perfect classifier. We also report the relative improvement calculated by (new old)/(1.0 old), which represents how much improvement has been made relative to the maximum possible improvement.
machine-generated
B ANALYTICAL SOLUTION OF CONDITIONAL PROBABILITY CURVATURE
The sample mean j Eq. 4 represents the cross-entropy of distribution 4p (ū and pe (T)z). By leveraging the conditional independence of each token, we can calculate the expectation analytically
f1 Egra, (5)z) Po (f(2)) = Σ Io (ā(zz) log po(()))
(6)
ΣΣ 2,
a [22)) (o:) (ves)) Σ رز
where üú; denotes the sample mean on the j-th token. The summation over ž. is computed by enumerating the tokens in the vocabulary, which can be exactly calculated.
The sample variance 2 in Eq. can also be calculated analytically
22= E2 (8)2)
[log - p Σ yo (il m)
Σ
Σ 2e ( log² pooeppoloe (o(s)(ores) - 113
where the summation over ž can also be calculated exactly by enumerating the tokens in the vo- cabulary. Empirically, the analytical solution achieves detection accuracy almost identical to the sampling approximation with 10,000 samples (our default setting) but further accelerates the detec- tion process by about 10%.
14

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL SETTINGS
C.I OPEN-SOURCE AND PROPRIETARY MODELS
Model
Model F'lleService
Parameters
Training Corpus
mGPT
Shiinzhik
2023
aberbank-ai/mGiiT
1.3B
Wikipedia and Colossal Clean Crawled coppus.
GPT-2
[adiorii el 2019)
gpt2-xl
1.5B
English WebText without Wikipedia.
PubMedGPT
slanfooderrfu//ubnndggg
2.7B
Blomedical abstracts and papers from the Pilc.
OPT-2.7
Zhang el 21
fucebook/opt-2.78
2.7B
A larger dataset including the Pile.
Nea-2.7
FleutherAligtt-ne
2.7B
2.7B
Pile
GPT-J
Wang & Komaisuzaki 302
BleatherAl/ggt-j--00
6B
The Pile (Gio al
NTA
BLO0M-7.1
et
bigscienc//blomm-7
7.1B
ROOTS corpus (Laureaçon
OPT-13
Zhanc et AUA
facebook/ogt-188 136
13B
A larger dataset iactuding File.
Llama
[uuvion 3l.
20233
huggyllama/llama. 13h
13B
Common
C4, Githuh, Wikipedia, Books, A new mis publicly available online data.
Llama2.1
TheRloke/I
13R
NeuX
Black al P1224
AleaterrAl/ggt-neee neux- 20b
20B The Pile (Gav el al. 2020).
GPT:3 [Bron al. 2020
OpenAl/davinci
175B
CommonCraww. WebText, English Wikipedia.
ChaiGTT (PeeAA 0122
OpenAl/gpt-3. 3.St
175B
CommonCrawd, WebText, English Wikipedia.
GPT-4 OpinnAl 2023
OpenAlignt.4
NA
Table 4: Details of the source models that is used to produce machine-generated text.
We assess the performance of our methodologies using text generations sourced from various mod- els, as outlined in Table These models are arranged in order their parameter count, with those having fewer than 20 billion parameters being locally Tesla A100 GPU (80G). For models with over 6 billion parameters, we employ half-precision (float16), otherwise, we use full-precision (float32).
In the case of larger models like GPT-3, ChatGPT, and GPT-4, we utilize the OpenAl API for the evaluations. Additionally, provide information about the training corpus associated with each model, which we believe is pertinent for understanding the detection accuracy of different sam- pling and scoring models when applied text generations originating from diverse source models, domains, and languages.
C.2 SETTINGS FOR GPT-3, CHATGPT, AND GPT-4
To compile our test set, we generate 150 samples for each dataset (among XSum, WritingPrompts, and PubMedQA) and each source model by calling the OpenAl service Specifically, for GPT-3, we requested text completions for 30-token prefix, while for ChatGPT and GPT-4, we request chat completions with predefined instructions as follows.
ChatGPT and GPT-4 function conversational manner, serving as assistants execute user in- structions. In the context of our experiment, we task these models with adopting the roles of pro- fessional News, Fiction, and Technical writers the generation of News articles, stories, and an- swers, respectively. To encourage the production of content that is both unpredictable and creatively diverse, we utilize a temperature setting of 0.8.
Specifically, we initiate the generation process by sending the following messages to the service. Message for XSum:
''role: 'system', 'content'; 'You are a News writer. 'role': 'user', 'content's 'Please write an article with about 150 words starting exactly with: <prefix>'),
3
The <prefix>> could be like "Maj Richard Scott, 40, is accused of driving speeds of up 95mph (153km/h) in bad weather", and the response is supposed to start with it.
Message for WritingPrompts:
15

---

Published as a conference paper at ICLR 2024
'role': 'system', 'content's ': 'You are a Fiction writer.'), 'role': 'user', 'content': 'please write an article with about 150 words starting exactly with: <prefix>"),
The <prefix>> could be like "A man invents time travel in order to find a cure for his sick wife and succeeds, only to find out", and the response is supposed start with it.
Message for PubMedQA:
('role': 'system', 'content': 'You are a Technical writer. '), 'rolee: 'user', 'content': 'Please answer the question in about 50 4 words. <prefix>'),
The <prefix> could like "Question: Is an advance care planning model feasible community palliative care? Answer:" and the response is supposed to answer the question directly.
16

---

Published as a conference paper at ICLR 2024
Source Model
Dataset
Method
GPT-2
OPT-2.7
Nco-2.7
GPT-J
NcoX
Avg.
Likelihood
0.8638
0.8600
0,8609
0.8101
0.7604
0.8310
Entropy
0,5835
0,5071
0,5712
0,5705
0.6035
0,5671
LogRank
0.8918
0,8839
0.8949
0.8407
0,7939
0.8610
LRR
0,9179
0.8867
0,9190
0.8592
0,8205
0.8807
DNA-GPT +
0.8548
0.8168
0.8197
0.7586
0.7167
0.7933
XSum
NPR O
0.9891
0.9681
0.9929
0.9566
0.9311
0.9676
DetectGPT
0.3875
0.962T
0.9914
0.9632
0.939%
0.9688
Fast-DetectGPP
0.9930
0.9803
0.9885
0.9771
0.9703
0.9818
(Diff)
0,0055
0.0182
0,0029
0.0139
0.0305
0.0130
DetectGPT (fixed) &
0,9180
0.8868
0,9914
0.8830
0.8682
0.9095
Fast-DetectGPT (fixed)
0.9742
0.9444
0.9965
0,9335
0.9033
0.9504
(Diff)
0.0563
0.0576
0.0051
0.0505
0.0351
0.0409
Likelihood
0.9077
0.8839
0,8585
0.7943
0.6977
0.8284
Entropy
0,5791
0,5119
0,5581
0,5643
0,6056
0.5638
LogRank
0,9454
0.9203
0.9054
0.8471
0,7545
0.8745
LRR
0,9773
0.9597
0.9610
0.9244
0.8600
0.9365
DNA-GPT +
0,9094
0,8934
0.8589
0.8069
0.7525
0.8442
SQuAD
NPR O
0.9965
0.9853
0.9789
0.9108
0.8175
0.9378
DetectGPT7
4.394
70,963
0.9625
0.8738
0.7916
0.9191
Fast-Detect(PPp
0.9990
0.9949
0.9956
0.9853
0.9617
0.9873
(Diff)
0.0076
0.0186
0.0331
0.1116
0.1702
0.0682
DetectGPT (fixed) 6
0.7382
0.7530
0,9625
0.7882
0.7709
0.8026
Fast-DetectGPT (fixed)
0.9824
0.9762
0.9990
0.9584
0.9379
0.9708
(Diff)
0.2442
0.2232
0.0365
0.1703
0.1669
O. 1682
Likelihood
0.9661
0.9451
0.9505
0,9396
0.9256
0.9454
Entropy
0.3895
0,4299
0.3400
0.3668
0.3908
0.3834
LogRank
0,9782
0,9628
0,9675
0.9577
0,9454
0.9623
LRR
0,9850
0.9740
0.9766
0.9702
0.9573
0,9726
DNA-GPT +
0,9431
0,9288
0.9283
0.9026
0.8786
0,9163
NPR +
0.9987
0.9962
0.9930
0.9825
0.9708
0.9882
Writing Prompts
DetectGPTT0
7.3962
"09899
0.9852
0.9688
0.95T6
0.9782
Fast- DetectGPT
0.9982
0,9972
0.9980
0.9974
0,9941
0.9970
(Diff)
0.0020
0.0081
0.0129
0.0285
0.0424
0.0188
DetectGPT (fixed) O
0.8989
0.8772
0.9852
0,9014
0,8809
0.9087
Past-DetectGPP (fixed)
0.9937
0.9509
0.9996
0.9858
0.9801
0.9820
(Diff)
0.0948
0.0738
0.0145
0.0844
0.0992
0.0733
Table 5: Details of the main results in Table 2 Fast-DetectGPT consistently outperforms DetectGPT in terms of detection accuracy (in AUROC), We run DetectGPT and NPR with default 100 perturbations and run DNA-GPT with truncate-ratio of 0.5 and 10 prefix completions per passage. Methods marked with "(fixed)" the fixed models to detect texts from different sources (the black-box setting). where DetectGPT uses T5-3B/Neo-2.7 as the perturbatioon/scoiing models and Fast-DetectGri uses GPT-J/Neo-2.7 as the sampling/scor- ing models. The "(Diff)" rows indicate the AUROC improvement upon DetectGPT baselines. "*" denotes the second-best AUROC. < Methods call models hundred times, thus consuming much higher computational resources.
Zero-shot detection on five source models, where
17

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RESULTS
D.1 ZERO-SHOT DETECTION ON ADDITIONAL OPEN-SOURCE MODELS
Source Model
Datasct
Method
BLOOM-7.1
OPT-13
Llama-13
Llama2-13
Avg.
Likelihood
0.8500
0.8105
0.6121
0.6550
0.7319
Entropy
0.6642
0,5251
0,6731
0,6029
0.6163
LogRank
0.9018
0.8369
0,6542
0.6911
0.7710
LRR
0,9412
0.8344
0.7327
0.7351
0.8109
NPR
0.9931
0.9283
0.8031
0.8212*
0.8864
XSum
DetectGPT
0.9972
0,9268
0.8147
0.8106
0.8858
Fast- DeteceettT
0.9890
0.9721
0.9473
0.9346
0.9607
(Diff)
-0.0021
0.0452
0.1325
0.1240
0.0749
DelectGPT (fixed)
0.7363
0.8713
0.6869
0.6848
0.7449
Fast- DetectGeeTGP (fixed)
0.8886
0.9207
0,7306
0.6968
0,8092
(Diff)
0.1521
0.0495
0.0436
0,0121
0.0643
Likelihood
0.8619
0.8220
0.5113
0,5000
0.6738
Entropy
0.6199
0,5517
0.6534
0,6636
0.6221
LogRank
0.9157
0.8645
0.5589
0.5457
0.7212
LRR
0.9678
0.9228
0.7262
0.7036
0.8301
NPR
0.9730
0.9351
0.5332
0.5448
0.7465
SQuAD
DefectGPT
1035100
0.9T10
0.3204
0.3507
D.7333
Fast-DetectGPT
0.9953
0.9893
0,8717
0.8772
0.9333
(Diff)
0,0443
0.0782
0.3513
0.3265
0.2001
DetectGPT
0.6359
0.7596
0.5588
0.3488
0.6258
Past-DetectGPT (fixed)
0.9588
0.9590
0,8028
0.7627*
0,8708
(Diff)
0.3228
0.1994
0.2440
0,2138
0.2450
Likelihood
0.9368
0.9400
0.8692
0.8737
0,9049
Entropy
0.4876
0,4096
0,4831
0,4702
0,4626
LogRank
0.9612
0.9578
0.9047
0.9069
0.9327
LRR
0,9811
0.9650
0.9326
0.9306
0,9523
NPR
0.9909
0.9850
0.9184
0.9003
0.9487
WritingPrompts
DelectGPT
039829
0.9701
0.8996
0.33396
0,9130
Fast-DetectGPT
0.9983
0.9953
0.9892
0.9939
0,9942
(Diff)
0,0154
0.0253
0.1296
0.1543
0.0811
DetectGPT (fiee)
0.7933
0.8695
0.7455
0.7532
0.7904
Fast Dedd-eeectGPTT (fixed)
0,9779
0.9367
0,8925
0,9085
0,9289
(Diff)
0.1846
0.0672
0.1471
0.1553
0.1385
Table 6: Addition the main results in Table
Zero-shot detection on more source models.
We extend our evaluation to include several popular open-source LLMs, including BLOOM 7.1B, OPT 13B, Llama 13B, and Llama2 13B. In the white-box setting, Fast-DetectGPT exhibits erage relative improvement of 76.1% % when compared to DetectGPT. This outcome aligns with the 74.7% average relative improvement observed across the five models presented in Table under- scoring the consistent performance of Fast-DetcctGPT across diverse source models.
However, in the black-box setting, Fast-DetectGPT demonstrates average relative improvement of 53.4% compared to DetectGPT. This figure is lower than the 74.5% average relative improvement seen across the five models in the main table. We suspect that the reduced improvement observed in these source models relate to the potential mismatch between the scoring model Neo-2.7 and the source models. is conceivable that the training corpus used by the former may have limited overlap with the training corpus utilized by the latter according to Table 4 These findings underscore the challenges associated with identifying universally effective scoring models the black-bor setting.
18

---

Published as a conference paper at ICLR 2024
D.2 ZERO-SHOP SHOT DETECTION ON GPT-3 GENERATIONS
Method
XSum
WritingPrompts
PubMedQA
Avg.
RoBERTa-base
0.8986
0.9282
0.6370
0.8212
RoBERTa-large
0,9325
0.9113
0.6894
0.8444
GPTZero
0,4860
0.6009
0.4246
0.5038
Likelihood (GPT:3) O
0.76
0.87
0.64
0.76
DelectGPT (T5-11B/GPT-3) +
0.84
0.87
0.84
0.85
Likelihood (Nco-2.7)
0.8307
0.8496
0.5668
0.7490
Entropy (Neo-2.7)
0.3923
0,3049
0.5358
0.4110
LogRank(Nco-2.77
0.8096
0.8320
0.5527
0.7314
LRR (Nco-2.7)
0.6687
0.7410
0.4917
0.6338
DNA-GPT (Neo-2.7)
0.8209
0.8354
0.5761
0.7441
NPR (TS-11BBNeo-2.7)
0.8032
0.7847
0,6342
0.7407
0.8043
0.7699
0.69I5
0.7532
DetectGPT (TS-11B/Neo-22)
0.8455
0.7818
0.6977
0.7750
DetectGPT (T5-11B/GPT-))
0.8261
0.7666
0.6644
0.7524
Fast-DeteaGGPT (GPT)CGPPPT
0.9137
0.9533
0.7589
0.8753
Fast-DetectGPT (GPT-J/Neo-2.7)
0.9396
0.9492
0.7225
0.8704
Fast-DelectGPT (GPT-J/GPT-J)
0.9329*
0.9568
0.6664
0.8520
Table 7: Detection of GPT-3 generations, evaluated in AUROC. Fast-DetectGPTT in the black-box settings (using local models) significantly outperforms DetectGPT both the black-box setting and the white-box setting (using GPT-3) on News (XSum) and story (WritingPromptts uses 6B GPT-J to generate samples and models from 1.5B GPT-2 to 6B GPT-J to score samples, while DetectGPT uses IIB T5 to generate perturbations and models from 1.5B GPT-2 to 6B GPT-J, and GPT-3 service to score them. < - we report the official scores from Mitchell et (2023) instead of rerunning the experiments after confirming the consistency on RoBERTaa-bsee///
Fast-DetectGPP
Table presents a comparison between Mest-DetectGPT, zero-shot DetectGPT, and supervised RoBERTa-basse classifiers for the detection of GPT-3 generations. In contrast to DetectGPT, which employs the OpenAl API to assess perturbations, we utilize delegate models (specifically, GPT-2, Neo-2.7, and GPT-J) to identify GPT-3 generations.
Fast-DetectGPT outperforms supervised RoBERTa-base, RoBERTa-large, and GPTZero classifiers, achieving higher detection accuracy across the three datasets. On average, it improves the AUROC by 0.0310 AUROC (a relative increase of 20%). Conversely, DetectGPT in the white-box setting (us- ing T5-1 15-I1B//PPT-3 achieves superior detection accuracy PubMedQA but lags behind XSum In the black-box setting (T5- 1a//1////o-/)) De-
and WritingPrompt compared to RoBERTa-lare tectGPT experiences a significant reduction in detection accuracy, averaging 0.0750 AUROC less.
These findings emphasize that Fust-DetectGPTT, operating in the black-box setting, offers compet- itive alternative to supervised detectors and DetectGPT the white-box setting.
When comparing the results on GPT-3 and ChatGPT, it becomes apparent that Past-DetectGPi performs significantly better on ChatGPT. We speculate that this discrepancy may relate factors such as instruction-tuning (Wei et al. (021) and human-feedhace reinforcement learning (HFRL) (Ouyang et al. 2022), which are employed in fine-tuning large language models and may skew the model toward high- nigh-probability tokens.
19

---

Published as a conference paper at ICLR 2024
E ABLATION STUDY
Method
GPT-2
Neo-2.7
XSum
GPT-J
GPT-2
SQuAD Neo-2.7
GPT-J
GPT-2
WritingPrompts Neo-2.7
GPT-J
Avg.
DetectGPT (9/9)
0.9930
0.9885
0,9853
0,9982
0.9980
0.9974
Rast-DetectGPT (GPT-2/)
0.9930
0.9918
0.9534
0.9990
0.9728
0.8785
0.9982
0.9954
0.9868
0.9743
Fust. DetectGPT (N70)).719)
0.9778
0.9885
0.9153
0.9977
0.9955
0.9212
0.9994
0.9980
0.9861
0.9755
Pest-DetecGGPT (GPT-I?")
0.9960
0.9965
0.9771
1,0000
0.9990
0.9853
0.9999
0.9996
0.9974
0.9945
Table 8: Impact of reference model, where "*/*" indicates that the source model generate reference samples and score the log probability, while "GPT-J/e" indicates that we use GPT-J to generate the samples and the source model score.
Sampling Model Ablation. We investigate the influence of the choice of the sampling model, summarized in Table 8 Remarkably, when GPT-J is employed the sampling model, Fast- DetectGPT attains the highest average detection accuracy. In comparison to Fast-DetectGPT uti- lizing the source model for sampling, the utilization of GPT-J enhances accuracy by an average of 0.0020 AUROC, representing relative improvement of 27% across all three datasets and the three models under consideration. These findings indicate that employing a more robust sampling model has the potential to further augment the performance of Dest-DetectGPT in the white-box setting.
Normalization Ablation. Normalization based on the standard deviation has previously been pro- posed as an additional technique within DetectGPT. In our study, we integrate this normalization as a default component of the conditional probability curvature metric for two principal reasons. Firstly, normalization exerts a significant influence detection accuracy, resulting in average AUROC improvement of 0.0172, equivalent to a relative increase of 36% for DetectGPT. In the case of Fatt normalization enhances the average AUROC by 0.0020, corresponding to a 10% relative improvement. Secondly, after normalization, the distributions of the curvatures for differ- models across various datasets become more directly comparable. Without normalization, these distributions exhibit variations in width, ranging from narrow to wide, depending on the variance of the generated samples.
Entropy Ablation. Among the total 75% relative improvement, 10% is attributed to normalization by o, while the remaining 65% stems from the contribution of the numerator log (z|x) j.. The entropy - plays crucial role in achieving high detection accuracy in Fast-DetectGPT.
An intuitive elucidation of the significance of entropy lies in the substantial variance observed in the log (=(=z|azz)) values for different tokens Ij across diverse contexts æzj. This variability in- troduces instability in the statistical measures employed for detection. Conversely, the expectation ji establishes a dynamic probability baseline for each token. Consequently, the subtraction of log (n, ( and ni serves to mitigate the variance the log-likel,hood, yielding a more stable statistic that proves resilient to token or context fluctuations.
In a specific experiment involving ChatGPT generations for XSum, the average standard deviation of token-level log-likelihood is 2.1893, while the average standard deviation of token-level entropy 1.6090. Conversely, the average standard deviation resulting from their addition 1.6342, repre- senting significant reduction from the initial 2893.
20

---

Published as a conference paper at ICLR 2024
F ADDITIONAL ANALYSIS
F.1 ROBUSTNESS AGAINST DECODING STRATEGIES
Top-p (p
0.96)
Top-k (k
40)
Temperature (T
0.8)
Method
XSum
SQuAD
Writing
Avg.
XSum
SQuAD
Writing
Avg.
XSum
SQuAD
Writing
Avg.
Likelihood
0.9126
0,9045
0,9781
0,9317
0.8624
0,8612
0,9608
0.8948
0,973
0.9647
0,9962
0.9780
Entropy
0.5287
0.5273
0.3255
0.4605
0.5538
0.5523
0.3632
0.4898
0.4854
0.4942
0.2522
0.4106
LogRank
0.9293
0,9324
0,9853
0,9490
0.8946
0.9047
0.9757
0.9250
0.9844
0.9821
0.9978
0.9881
LRR
0.9223
0.9600
0,9836
0,9553
0.9173
0.9566
0.9827
0,9522
0.9826
0,9923
0.9903
0.9884
NPR
0,9789
0.9511
0,9911
0,9734
0.945
0.9912
0.9696
0.9710
0,9897
0.9833
DetectGPP
0..777
"093359
0.9807
0,96648
0.9708
0.9247
T.97797
0.9534
0.9830
0.9662
"D,974
0.9546
Fast-Detect
0.9976
0.9994
0,9982
0,9914
0.9977
0,9921
0.9998
0.9996
0.9992
0.9995
(Di)
0,0197
0.0617
0,0188
0.0334
0.0164
0.0667
0.0180
0.0337
0.0168
0.0634
0,0247
0.0350
T..976
T0.5506
0.977
10.9121
0.3158
0.2201
T..91TT
039777
0.9026
70.3512
09422
Dast-Detect (fixed)
0,9889
0,9942
0,9945
0.9925
0.9642
0,9790
0 9864
0.9765
0.9988
0.9989
0.9984
0.9987
(Di))
0,0413
0.1436
0.0368
0.0806
0,0484
0.1587
0.0683
0.0918
0.0271
0.0963
0,0462
0.0565
Method
Top-p
0.90)
Top-k (k
30)
Temperature (T
0.66
Lifelhoodd
7.5999
T0,3495
0.992I
003660
0.30T
0.8922"
70.9754
03929
019964
0.9654
T0,3959
019772
Entropy
0.4985
0,4990
0,2881
0.4285
0.532
0.5374
0.3359
0.4684
0,4171
0.3718
0.1146
0.3012
LogRank
0.9701
0.9684
0.9951
0.9779
0.9309
0.9338
0.9863
0.9503
0.9990
0.9987
1.0000
0.9992
Past-Detect
0.9997
0.9998
0.9996
0.9997
0.9941
0.9955
0.9988
0.9961
0.9998
0.9999
0.9991
0.9996
Table 9: Impact of decoding strategies, where the machine-generatted texts are produced by sampling with top-p, top-k, and temperature. The report AUROC is the average over the five models in Table
I0
Top-p (p = 0.96)
Top- = 10)
Temperature (T' = 0.8)
Method
GPI2
OP12.7
Neo2.7 GPTI
GPT2
OPT2.7
Neo2.7 NeoX
GPT2
OPT2.7 Neo2.7 GPTI NeoX
XSum
Lifellood
3/134"
530%%
3:T
"9045 81533
"8815"
3VT8
78873"
SRUT 31T4
.98T
"9862"
3677 f
Entropy
.5541
.4655
.5303
.531
.5614
.743
.843
.5001
.588
5517
5022
.4889
.4669
.437
.5453
LogRank
.9144
.4222
.9502
.922 .8966
9122
9173
.9277
8856
8454
9883
9985
.939
.9817
.9644
LRR
.5599
.9522
9203
.8622
9432
9954
.452
9945
8683
9589
991
9806 .9702
NPR
9909
9841
.982
993
9486
.987
9801
.938
9606
.416
9955
.9916
9976
9853
.976
DetecGPT
D875
975
9961
9966
'55
9569
9707
3919
9619
3424
9928
9955
'995
'3/9
'962
Fast-DeteeGGPT
.994
.9655
.9988
.997
.9988
.954
9867
.9938
.9826
.9733
9999
.999
.9977
.999
.9977
(Dif)
.0199
0072
.0088
.0208 .0458
0085
.0150
.0018
.007
.0349
.0071
.0143
.0044
.0209
.0376
DeteaGPTTixxed
9399
9438
.961
9367 .9114
9145
9026
3919
8846
8834
9722
.6355
.953
9627
9646
DeteDetect((xee))
9953
9861
9997
9856
.9779
9805
9613
.979
9499
.9311
9978
.999
9999
9991
998
(Di))
0056
.443
.0036
.0489
.0565
.6622
.0587
.0000
.0654
.0457
.0256
.0046
.0364
.0333
SQUAD
LiKelhood
1961
1941
"88388
.8122
9.39
3072""
T8976"
.85ST
77377
9906
''87"
"974I""
3572
99094
Entropy
.3699
.4736
.$39
.5277
.5593
.552
5203
.4577
.5441
.5992
.132
.4508
.4924
48822
.5263
LogRank
9792
99657
.5335
9156
.8482
9692
9423
.3855
.882
.7895
9972
.9599
.994
.9988
.434
IRR
.865
.981
.981
9507
.9009
9898
9782
.9815
.945
88866
9991
.9973
.9922
.911
.9488
NPR
9955
.934
.9997
9335
8836
.9622
.99
.9777
.9511
826
9988
.9963
.9911
.9844
.8981
DetectGPT
.994
.9838
.9781
.904
.8822
9928
9804
.9709
.8899
.7977
.959
.9819
.9778
.1233
.8132
Fast-DetectGPT
.9997
.998
.981
.9903
9996
.973
.998
.9928
.9066
.9944
.9986
(Dif)
.0600
.0159
.0218
.0927
.1722
.0057
.0/69
.0271
.1009
.1729
.041
.0180
.0222
.0871 .1854
DeleaTGPTiheed
3/06
.7877
9781
8465
8066
75855
7663
3/109
81:33
77729
8936
.022
"9718"
3857
Fast-Delcoa(fieed)
.984
.955
9933
.9845
9893
.844
.997
.7709
.5599
1
.9999
9984
.961
(Di))
.1988
.2093
00.99
.1471
.1480
.20788
.2270
.0288
.1587
.1780
.J064
.1379
.0222
.1087
.1063
Writing Promppls
LiKelhood
3689
"9764"
1577
"9777
39:355
39677"
9566
33473
.883
"9546
'9977
Entropy
.3336
.3789
.2798
.212
.3238
3601
.4149
.3183
.3529
3696
.2511
.3033
.219
.2252
.2625
LogRank
.9311
.927
.987
9853
.9785
9867
.9719
.9807
.9355
9655
9991
.9722
.9944
.9977
.9999
LRR
.9066
.9854
.9857
.9724
9901
.9835
.9858
.9834
.9709
9961
.9917
.9914
99887
.9837
NPR
9984
.964
.9944
9855
.9588
9988
.9555
.9971
.9888
9759
9985
.9953
.993
.836
.9881
DetectGPT
9969
9898
.9926
9914
.9266
9965
9868
.992
.91
.9511
9932
.985
'9842"
9656
.9447
Fast. DetectGPT
.9977
.9999
.9988
.998
.9981
.9988
9973
.999
.9877
.9557
9997
.9999
.9992
.991
.9984
(Dif)
.0028
.0102
.0071
0284 0045
.0022
.0/05
.0070
.0277
.0436
0005
.0149
.0150
.0335
.0537
DelecGPTTinedd
9297
91177
992
9299
.9877
.9115
3906
.922
9052
8914
9385
.9511
.9842
9535
.9398
Fast-Detectfixedd
9982
9W25
9999
9974
.945
9947
9616
9998
.9011
998
9995
9959
9997
.983
.987
(Dif)
.6685
.0648
.0033
.0676
.0758
.0832
.9710
.0079
.0849 .0946
.0610
.0508
.0155
.0448
.0588
Table 10: Detailed results on decoding strategies.
Machine systems may use different decoding strategics such as top-k sampling, top-p (Nucleus) sampling, and sampling with a temperature T. We experiment on the five models and three datasets
21

---

Published as a conference paper at ICLR 2024
used in Table 2 sampling with the three strategies with k 40, p 0.96, and T 0.8. Fast- DetectGPT in the white-box setting obtains the best accuracy on the three sampling strategies, out- performing DetectGPT by relative 95% Top-p sampling, relative 81% Top-k sampling, and relative 99% on sampling with a temperature, as shown in Table 9 In the black-box setting, Fast- DetectGPT outperforms DetectGPT by relatively 92%, 80%, and 98% the three decoding strate- gies, respectively. These results demonstrate that Dest-DetcttGPT works consistently in detecting texts produced by different decoding strategies.
To elucidate the trajectory of detection accuracy concerning variations in sampling hyper- parameters, we conducted additional experiments with values set to p = 0.90, 30, and T 0.6. As indicated in the lower segment of Table P reducing the values of p. k. and T enhances the de- terminism of generated samples, facilitating easier detection and consequently yielding significantly higher AUROCs.
F.2 ROBUSTNESS UNDER PARAPHRASING ATTACK
Paraphrasing Attack PubMed
Decoherence Attack
Method
XSum
Writing
Avg.
XSum
Writing
PubMed
Avg.
RoBERTa-base
0.8103
0,5368
0.5962
0.6477
0.5778
0,5629
0,5392
0,5600
RoBERTa-larg large
0.7532
0.4619
0.6187
0.6113
0.4945
0.3849
0.5520
0,4771
Like)ihood(Neo-2.7))
0.8521
0.8691
0.7029
0.8080
0.7393
0.9240
0.7757
0.8130
Entropy
0,4508
0.3054
0.3815
0.3793
0.4807
0.2474
0.3051
0.3444
LogRank(Neoo2277
0,8640
0.8635
0.7060
0.8112
0.7628
0.9143
0.7789
0.8187
1KR(/5-111B/No-227))
0.8391
0.8040
0.6596
0.7676
0.7845
0.8413
0.7067
0.7775
NPR(TS--11B/Noo-2.7)
0.5121
0.5530
0.4753
0.5135
0.4250
0.7577
0.5198
0.5675
0.4864
0.5698
0.6004
0.3522
0.2919
0.7035
0.6026
T.3326
DetecGPPT(S-111BNNee02)
0.5364
0.5172
0.4763
0,5100
0.3438
0.6894
0.5073
0.5135
DetecG@PTCTS-1II/GGPP]
0.4298
0.4689
0.4079
0.4355
0.2889
0.6573
0.4371
0.4611
Fas(GPTTJ/GPT-2)
0.9233
0.9186
0.7727
0.8715
0.7909
0.9524
0.7697
'..377
Fast(GPT-I/Nco-2.7) Fast(GPT-//PPT-J)
0.9646
0.9190
0.7172
0.8669
0.8598
0.9622
0.7487
0.8569
0.9591
0.9476
0.7063
0,8710
0.8750
0,9811
0.7428
0.8663
Detection of CharGPT generations under attack. We the black-box settings for all zero-shot methods. We paraphrase each sentence in the ChatGPT-generated passages using T5 para- phraser for paraphrasing attack and randomly swap two adjacent words in each sentence with more than 20 words, where both attacks downgrade the coherence of the original passages.
Table 11:
We assess Fast-DetectGPT alongside with other zero-shot methods to evaluate their resilience the face of adversarial attacks. Following the approach outlined in Sadasivan et al. (2023), we employed a T5-based paraphrasef" to paraphrase text generated by ChatGPT before detection. As shown in Table IT (Appendix F2), all methods witnessed decline in detection accuracy. Specifically, RoBERTa-base's AUROC decreases from 0.7474 to 0.6477, DetectGPT from 0.8342 to 0.5522, and Fast-DetectGPT from 0.9641 to 0.8715, where Fast-DetectGPT experiences the smallest relative downgrade.
However, upon detailed examination, identify an unforeseen consequence of the paraphrasing attack: a noticeable reduction in cross-sentential coherence within passages, as an example illus- trated below. This issue largely stems from the independent paraphrasing of individual sentences. We speculate that this diminished coherence is primarily responsible for the observed performance drop, given that the paraphrased text appears aberrant in its coherence relative to both machine- generated and human- authored passages.
Paraphrasing attack downgrades the coherence of the passages. For instance, consider a news report about a car crash, originally reading, "If the car driver was hit first from behind or in front of him only on a single lap you should take control of your car. You have to be as much ahead of the car driver possible and if you not gel to your proper position with the car and the driver can't I make that turn." " The second sentence was paraphrased to, you not get to your proper position with the
hhaps:/tuuggingffccco///mm Vaaass///Sc///// Paraphrase Paws
22

---

Published as a conference paper at ICLR 2024
and the driver not do that turn, you need to be as much ahead of the car driver possible." When placed back in its context, we observed that the paraphrased sentence was considerably more challenging comprehend than the original.
To test this conjecture, we exccute a decoherence attack, wherein two adjacent words in sentences exceeding 20 words randomly transposed. While this manipulation impacts fluency, the core meaning largely persists. As evidenced in Table [T] in Appendix, there was a comparable drop in detection accuracy, thereby empirically confirming our speculation.
G ADDITIONAL DISCUSSION
Authorship of Text. The conditional probability curvature serves indicator of how well candidate passage aligns with specific source model. When we utilize various source models, we observe varying sample discrepancies, which can aid in identifying the most suitable match among these source models. Consequently, this approach has the potential to be employed for source model identification within a set of available models.
Watermarking. Another line of detection methodology is watermarking that deliberately embeds information within machine- generateedee text to trace its origin (Jalil & Mirza (2099] Kamaruddin al.] 2018 Abdelnabi & Fritz 2021 Gu et al., 2022 Kirchenbauer et al. (023). In comparison, Fast- DetectGPT relies on innate distinction between texts generated by humans and by machines, which may further be strengthened by explicit watermarks as additional features.
In practice, these two strategies could potentially be combined provide a more reliable detection solution. On the one hand, watermarking cific service. On the other hand, when the service is out of our control and we cannot enforce the watermarking potential attacker has strong LLM to remove the watermarks, the watermarking approach fails in these situations but the general detector like Past-DetectGPP can still provide valid solution.
be used to authorize the content generated by spe-
23


Reviews:
[{"review_id": "5wuQGPtwiC", "reviewer": "Reviewer_RgLj", "summary": "This paper improves the previous zero-shot method for detecting machine-generated text, DetectGPT, by replacing the perturbations as sampling using the same source model. Through the conditional probability curve, the author proves the effectiveness of this method. However, some experimental details are missing. More importantly, it did not mention another zero-shot work [1] released 5 months ago, which is the first to propose using a conditional probability curve for detection. Considering the similarity with the previous work [1], I would like to question the novelty of this paper since the long 5-month period clearly shows they are not concurrent work.", "questions": "Questions: \nThe number of relative improvements is confusing. For example, in Table 1, why is the relative improvement 74.7%? In my understanding, (0.9887−0.9554)/0.9554*100%=3.48%. I do not understand why you report 74.7%. \nSee more in Weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "Strength:\nThe experiments over diverse datasets and models validate its effectiveness.\nThe author considers both open-sourced and closed-source models for the detection. Thus, the results can be easily reproduced.\nThe ablation study is enough to support its claim regarding parameter sensitivity, attacks, etc.\nThe paper is well-written and easy to follow. The tables and figures are arranged properly.\n\nMissing reference: \nThe following zero-shot method is missing either in the related work or in the baselines.\n[1] Yang X, Cheng W, Petzold L, Wang WY, Chen H. DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text. arXiv preprint arXiv:2305.17359. 2023 May 27.", "weaknesses": "Weakness: \n1. The novelty is limited. The conditional probability curve has already been used by another zero-shot detector released 5 months ago [1]. However, the author neither cites this previous work nor discusses its differences. Considering the reference [1] work was released 5 months ago, I will not consider them as concurrent work.\n2. It is not clear how the sampling process works. Give a passage x, how do you sample the alternative x’ ? Throughout the paper, I did not find any explanation for this.\n3. How would the number of resampled instances influence the result? I did not find any result for this.\n4. What is your default setting for the number of resampled instances for all the experiments? There is no clarification at all. \n5. How do you compare the speedup of your result over DetectGPT? Since the setting of your number of samples is unclear, I am not sure how did you compare it.\n\n\nAfter rebuttal: Thanks for the clarification. The authors addressed most of my concerns. I would like to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kqdK4NMcEb", "reviewer": "Reviewer_7WUs", "summary": "This paper present an extension for DetectGPT, improving its efficiency and effectiveness. Relying on LLM's output probability, the model can threshold and perform zero-shot detection. Given a sentence, the model will first autoregressively predict x' from the input, and then use the original input x as input to a LLM but calculate the probability to predict x'. The modification is simple, and effective, which intuitively makes sense.", "questions": "Can you elaborate how \\tilt {x} is generated? The reviewer is still confused.\n\nWhere does the acceleration come from? DetectGPT samples 100 pertrubations, how could this method accelerate 340 times? How many sampling does this needs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Improved results over DetectGPT for 3 points, with also faster speed.\n\n2. The paper also showed results on detect GPT-4 results.\n\n3. Analytical solution presented to avoid sampling approximation.\n\n4. Ablation study on different lengths, decoding strategies, paraphrasing has been shown.", "weaknesses": "1. Presentation should be made clear. In the intro, paragraph 4 talked about the algorithm, yet it is unclear what does \\tilt mean, what does <j means, also, the insight on why conditional probability is better is missing here, especially given that this is an extension of DetectGPT.\n\n2. Is there results for speed comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wBCpS5aSFI", "reviewer": "Reviewer_aeWx", "summary": "This paper proposes a new method for detecting LLM generated text that offers not only substantial performance benefits over DetectGPT but is also much less compute intensive. This is underpinned by a hypothesis that context matters in determining the differences between human and machine generated output. Their method accordingly uses a new criteria, the conditional probability curvature, which they find is more positive for LLM output than human. They perform experiments on a variety of datasets, and analyze robustness with respect to multiple text attributes.", "questions": "Have you investigated the effects of varying the temperature setting or the value of k for Top-k?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The proposed method is well motivated and described, and follows naturally from existing work\n- The results are strong both from a performance and efficiency standpoint, compared to DetectGPT\n- There is meaningful analysis with respect to attributes like passage length, paraphrasing, decoding strategies, etc.", "weaknesses": "- The discussion of prior work with respect to alternate detection strategies such as watermarking is shallow. The Kirchenbauer et al. 2023 paper is for example not cited. While this paper takes an orthogonal approach, it would be good to see some motivation or discussion around the tradeoffs of those strategies.\n- The discussion of ethical considerations and broader impacts is lacking. Liang et al. 2023 has shown that LLM detection systems tend to exhibit higher false positive rates for non-native speakers. While this doesn’t invalidate the usefulness of this work, at the least it is worth engaging with that literature and acknowledging the potential problems at play with this task. At best there could be experiments on the relative performance of this system on text written by different demographics as compared to prior work. Granted there is some analysis of performance on languages besides English but this is also relatively shallow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tc1FwbzGXt", "reviewer": "Reviewer_ubhC", "summary": "This paper provides a state-of-the-art approach to zero-shot detection of LLM-generated text based on the difference of text likelihood and entropy. The paper provides extensive experiments, outperforming DetectGPT and a number of statistical baselines, as well as a supervised RoBERTa-based approach. The approach performs especially well when the scoring and target LLMs differ, e.g., when using GPT-J to detect whether an article was written by ChatGPT or GPT-4, which is a known failure mode of the existing DetectGPT approach. The paper also includes a number of experiments on different decoding strategies, reports performance across document lengths, and experiments with paraphrasing attacks.", "questions": "- Did you experiment with computing the difference between the probability of the top-ranked word according to an LM scorer and the observed word? I expect this should be closely correlated with the metric proposed in this paper, and is also a slightly more informative alternative to the Rank model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strength of this work is the performance of the proposed method, which is better than much more computationally intensive zero-shot detectors such as DetectGPT. The set of ablation experiments (across languages, domains, decoding strategies, and paraphrase attacks) is also reasonably thorough, and the proposed method shows state-of-the-art performance across almost all tested conditions and datasets.", "weaknesses": "I find the framing of this paper and its comparison to be somewhat misleading. In particular, while the proposed method is described as a more efficient alternative to DetectGPT, its approach of computing the difference between the conditional probabilities of words and their alternatives is more similar to likelihood-based (Solaiman et al. 2019) or rank-based (GLTR; Gehrmann et al. 2019) approaches. Framing the method as a 340x speedup over DetectGPT therefore does not seem appropriate, although the method does seem to outperform existing zero-shot approaches. The sampling step in Fast-DetectGPT is also not clearly motivated and straightforwardly approximates an expected difference, so IMO the derivation could just immediately be replaced by the analytical solution. \n\nThe paper also includes supervised RoBERTa baselines from OpenAI; however, these are not state-of-the-art for supervised detection. I believe the paper would be strengthened by comparison to state-of-the-art supervised methods, such as Ghostbuster (Verma et al. 2023) or GPTZero (commercial model), especially given the claims in Section 5 that supervised methods have limited generalization capabilities in LLM-generated text detection. Because the primary purpose of the paper is to evaluate and compare zero-shot methods, however, this does not affect my score or recommendation for the paper.\n\nMinor notes:\n- The paper mentions both Rank and LogRank baselines in Section 3.1 but only provides LogRank in tables", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6nPXPWIXH", "reviewer": "Reviewer_nx1F", "summary": "This paper introduces an extension of DetectGPT called Fast-DetectGPT, which modifies the curvature criterion to operate per token, using the difference between the observed and average log probabilities. This requires only a single parallel forward pass from each model.", "questions": "Please see sections for strengths and weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed idea is intuitive. The conditional probability function is naturally parallelized by autoregressive models and this value should naturally be close to a local maximum for a model that generated a given text. The connection to likelihood and entropy was also interesting.\n\nThe experimental results are strong and comprehensive. Fast-DetectGPT is faster than DetectGPT by over two orders of magnitude due to its parallelization and also shows performance gains across six datasets. Even in the black-box (surrogate) evaluation setting, DetectGPT achieves impressively high recall at low false positive rates. It also shows qualitatively better behavior than DetectGPT on longer passages, where quirks of T5 masking cause DetectGPT to start underperforming as sequence length increases beyond a point.", "weaknesses": "The end of section 2 shows that the criterion for Fast-DetectGPT can be seen as closely related to likelihood and entropy. While this connection is nice, I think the paper could be stronger if it analyzed each term in (7) in isolation to see what is most contributing to increased performance and why. Both likelihood and entropy are points of comparison in the result tables, but they do not perform as well; does their sum perform well? If not, and the denominator in (7) plays a key role, what probabilistic interpretation does that have, and what does that imply about the log_p surfaces of LLMs?\n\nNot really a weakness and perhaps out of scope for this submission, but I'd be interested in knowing how Fast-DetectGPT would work for very long passages, given that it scales favorably with passage length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)