DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text
Authors: Jinyan Su
Terry Yue Zhuo
Di Wang
Preslav Nakov
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine-generated Text; Large Language models; LLMs; zero-shot
TL;DR: We propose two zero shot machine generated text detection method and intestigate efficiency--performance trade-off of zero-shot methods.

Abstract: With the rapid progress of Large language models (LLMs) and the huge amount of text they generate, it becomes impractical to manually distinguish whether a text is machine-generated. The growing use of LLMs in social media and education, prompts us to develop methods to detect machine-generated text, preventing malicious use such as plagiarism, misinformation, and propaganda. In this paper, we introduce two novel zero-shot methods for detecting machine-generated text by leveraging the Log-Rank information. One is called DetectLLM-LRR, which is fast and efficient, and the other is called DetectLLM-NPR, which is more accurate, but slower due to the need for perturbations. Our experiments on three datasets and seven language models show that our proposed methods improve over the state of the art by 3.9 and 1.75 AUROC points absolute. Moreover, DetectLLM-NPR needs fewer perturbations than previous work to achieve the same level of performance, which makes it more practical for real-world use. We also investigate the efficiency-performance trade-off based on users' preference for these two measures and provide intuition for using them in practice effectively. We release the data and the code of both methods in https://github.com/mbzuai-nlp/DetectLLM.

DetectLLM: Leveraging Log-Rank Information for Zero-Shot Detection of Machine-Generatte Text
Jinyan Su Terry Yue Zhuo', Di Wang", Preslav Nakov!
"Mohamed bin Zayed University of Artificial Intelligence, 2 Cornell Unversity
Monash University and CSIRO's Data61
*King Abdullah University of Science and Technology
I [Jinyan. Su, preslav. nakov @mbzuai ac.ae terry. t wwwo@monashh...eooooommmm edu, di. wang@kaust. edu sa
Abstract
and propaganda, which can lead to negative conse- quences (Zhuo et al., 2023). For instance, students might use LLMs to write assignments (Rosenblatt, 2023), making fair evaluation difficult for teach- ers, and in the long run, undermining the integrity of the entire education system. Malicious actors might generate fake news articles to spread misin- formation and propaganda or to manipulate pub- lic opinion, which is dangerous, especially when comes to politics (Floridi and Chiriatti, 2020; Stokel-Walker, 2022).
With the rapid progress of Large language mod- els (LLMs) and the huge amount of text they generate, it becomes impractical to manually distinguish whether text is machine-generaaee The growing use of LLMs in social media and education, prompts us to develop methods detect machine-gentn -egerrrted text, preventing malicious use such plagiarism, misinforma- tion, and propaganda. In this paper, we intro- duce two novel zero-shot methods for detecting machine-genennn generated text by leveraging the Log- Rank information. One is called DetectLLM- LRR, which is fast and efficient, and the other is called DetectLLM-NPE
With the proliferation of LLMs and the increas- ing amount of texts they produce, challeng- ing for humans to accurately identify machine- generated texts (Gehrmann et al., 2019). More- over, it is unrealistic to hire humans to manually identify machine-generatee text at scale due to the prohibitively high costs and the efficiency require- ments in real-time applications. c.g., in social me- dia. Thus, is essential develop tools and strate- gies to automatically identify machine- seetbineggenratedd text and to mitigate the potential negative impact of LLMs.
which is more ac- curate, but slower due to need for perturba- tions. Our experiments three datasets and seven language models show that our proposed methods improve over the state of the art by 3.9 and 1.75 AUROC points absolute. More- over. DetectLLM-NPP needs fewer perturba- tions than previous work to achieve the same level of performance, which makes it more practical for real-world use. We also investi- gate the efficiency-perforeee performance trade-off based on users' preference for these two measures and provide intuition for using them in prac- tice effectively. We release the data and the code of both methods in https:/ / qithub. com/mban DDMectLLM.....M.
The problem of distinguishing machine- generated from human-written text is commonly formulated as binary task (Jawahar et al., 2020). Most previous work has focused on the black-box scenario, where the detector has access to the output of the LLMs only and cannot make use of its internals. Such methods lack flexibility since they need to be retrained from scratch to be able to recognize the output of a new LLM (Mitchell et al., 2023). Given the speed at which new LLMs are developed, black-box methods are becoming more and more expensive and impractical. In cases when the access to the LLM is via API only, one possibility is for the LLM owner record all content it has generated, or watermark all texts it has generated (Kirchenbauer et al., 2023; Zhao et al., 2023). However, such solutions are not feasible for third parties.
1 Introduction
Large language models (LLMs) have made rapid advancements recent years, and are now able to generate text with significantly improved diver- sity, fluency, and quality. Models such as Chat- GPT (OpenAI, 2022), GPT-3 (Brown et al., 2020), LLaMa (Touvron et al., 2023) and BLOOM (Scao et al., 2022) demonstrate exceptional performance in answering questions (Robinson et al., 2022), writing stories (Fan et al., 2018: Yuan et al., 2022) and thus facilitating daily life and improving work efficiency. However, LLMs can also be misused for generating plagiarized text, misinformation,

---

We therefore consider a white-box setting, where the detector has full access the LLMs. We focus on zero-shot methods, where we use the LLM without additional training. Generally speak- ing, zero-shot methods use the source LLM to ex- tract statistics such as the average per-token log probability the average rank of each token in the ranked list of possible choices make a pre- diction by comparing it to threshold (Solaiman al., 2019; Ippolito et al., 2019; Gehrmann et al., 2019; Mitchell al., 2023). Based on whether the queried statistics are only about the target texts, we can roughly categorize them as perturbation -ree and perturbation-based Perturbation-free methods only query LLMs about the statistics on the tar- get text x, while perturbation-based methods such Mitchell et al. (2023) queries also the statistics of additional perturbed texts, which achieves bet- ter performance but is 50-100 times more costly than perturbation-free methods. Thus, there ex- ists a trade-off between performance and efficiency among zero-shot methods.
LRR
Count
NPR
Count
NC HB
NG HG
1.12
0.97
3000
3000
Figure I: Distribution of LRR and NPR visualized on 300 human-written texts (HG) from the WritingPrroptss dataset (Fan et al., 2018) as well as 300 texts generated with GPT-2-xl (MG) by prompting with the first 30 tokens from human- written texts.
zero-shot methods by 3.9 and 1.75 AUROC points absolute.
We investigate the efficacy of existing zero- shot methods and explore their limits as the size of the LLMs increases from 1.5 to 20 billion.
We conduct comprehensive experiments better understand the aeficiency-pefformmmee trade-offs in zero-shot methods, thereby pro- viding interesting insights on how to choose among different categories of zero-shot meth- ods based on users' preference for perfor- mance or efficiency.
To mitigate the gap of these two categories and design zero-shot methods with better performance- efficiency balance, should either improve the accuracy of perturbation-fice methods or reduce their cost. Thus, we propose two novel zero-shot methods, one perturbation-free but more accurate than previous methods, and one perturbation-ban method, but with better efficiency.
Related Work
The detection of machine-generated text is com- monly formulated classification task (Jawahar et al., 2020; Fagni et al., 2021; Bakhtin et al., 2019; Sadasivan al., 2023; Wang al., 2023). One way of solving it is to use supervised learning, where a classification model is trained on dataset con- taining both machine-gent gegerrted and human- texts. For example, GPT-2 Detector (Solaiman et al., 2019) fine-tunes RoBERTa (Liu et al., 2019) on the output of GPT-2, while the ChatGPT Detec- tor (Guo et al., 2023) fine-tunes RoBERTa the HC3 (Guo et al., 2023) dataset. However, models trained explicitly detect muchine-generatee texts may overfit their training distribution of the do- mains (Bakhtin et al., 2019; Uchendu et al., 2020). Another stream of work attempts to distinguish mmachine-generce from numan--ritted texts based on statistical irregularities in the entropy (Lavergne et al., 2008), perplexity (Beresneva, 2016) in the n-gram frequencies (Badaskar al., 2008). Gehrmann et al. (2019) introduced hand-crafted statistical features to assist humans detecting- machine generated texts. Morcover, (Solaiman et al., 2019) noted the efficacy of simple zero-shot
In the perturbation-fiee method proposed, we apply Log-Likelihood Log-Rand Bank ratio (LRR), which enhances Likeeioeld information with Log-Rank information as the discerning feature that achieves better performance than existing perturbation-fis methods and can even surpass perturbation-base based methods on some LLMs. For the perturbation-bawe method, we use Normalized perturbed log rank (NPR), which is based on the sen-
intuition that machine-generatee texts sitive to minor rewrites (or small perturbations). Compared to existing perturbation-base based methods such as DetectGPT (Mitchell et al., 2023), this ap- proach takes less time and computational resources and is more efficient. We denote these two meth- ods DetectLLM-LRR and DetectLLM-NPR respec- tively. Our contributions are as follows:
We propose two novel zero-shot approaches based on Log-Rank statisties, which improve over the state of the art. On average, these methods improved upon the previous best

---

X5um SQWAD
Wriling
the relative confidence, which reveals complemen- tary information about the texts. As illustrated in Figure 1, LRR is generally larger for machine- generated text, which can be used for distinguish- ing machine-generated from human-written text. One plausible reason might be that for machine- generated text, the Log-Rank is more discernible than the Log-Likelihood, LRR illustrates this pattern for machine-generann text. In Sections 4 and 6, we experimentally show that LRR is bet- ter discriminator than either the Lig-Likelihood or the Log-Rank. We call the zero-shot method using LRR as a detection feature as DetectLLM-LRR and use the abbreviation LRR in the rest of the paper.
14
PiPerturbatooss
#iPorturbations
e(Perturtations]
Figure 2: Comparison of DetectGPT to NPR averaged across six models (in terms of AUROC), (The full results are given in Figure in the Appendix).
methods for detecting machine -echine-generated text by evaluating the per-token log probability of texts and using thresholding. Mitchell et al. (2023) observed that machine-genn generattd texts tend to lie in the local curvature of the log probability and proposed De- tectGPT, whose prominent performance can only be guaranteed by the large size of the perturbation function and by large number of perturbations, and thus costs more computational resources.
3.2 Normalized Log-Rank Perturbation (NPR)
We define the normalized perturbed Log-Ran Log-Rank
Other work explored watermarking, which im-
prints specific patterns of the LLM-output text
NPR 2pp-11100 log
that can be detected by an algorithm while be- ing imperceptible to humans. Grinbaum and Ado- maitis (2022) and Abdelnabi and Fritz (2021) wa- termarked machine-generated text using syntax tree manipulation, while Kirchenbauer et al. (2023) re- quired access to the LLM's logits at each time step.
log logro(z)
where small perturbations applied the target text x to produce the perturhed text Spp Here, a per- turbation means minor rewrites, such as replacing some of the words. We call the zero-shot method using NPR as a detection feature DetcotLLM-NPP and use the abbreviation NPR in the rest of the paper.
3 Improved Zero-Shot Approaches by Leveraging Log-Rank Information
The motivation for NPR is that machine- generated and human-written texts are both neg- atively affected by small perturbations, i.e., the Log- Log-Rank score will increase after perturbations, but the machine -eeeerrte text is more suscepti- ble to perturbations and thus increase more on Log-Ram score after perturbation, which suggests higher NPR score for machine-generated texts. As shown in Figure 1, NPR can be discernible signal for distinguishing machine-generated from human- written text. DetectGPT (Mitchell et al., 2023) uses similar idea, but experimentally, be more efficient and to perform better. Details and comparisons are given in Section
In this section, we introduce the Log-Likelihood Ran-Rank Ratio (LRR) and the Normalized Perturbed log- Rank (NPR). LRR combines Log- Rank and mentary information about the text. NPR uses the idea that the Log-Ramt of machine- geetrrted texts should be more sensitive to smaller perturbations.
they provide comple-
3.1 Log-Likelihood Log-Rank Ratio (LRR)
We define the Log-Likenn Ligelikell Log-Rank Ratio as
find NPR
log po (o (x))
LRR
log re re()
log xz (e))
Experimental Setup
log re ( p(())
In this section, conduct comprchensive exper- iments to evaluate the performance of LRR and NPR in comparison to several methods previously proposed the literature. We experiment with LLM sizes varying from 1.5B to 20B parameters, probing the boundary of zero-shot methods when
where ro(x;( > is the rank of token iy con- ditioned the previous tokens.
The Log-Likelihood in the numerator represents the absolute confidence for the correct token, while the Lon-Rank in the denominator accounts for

---

LLMs continue grow in size. We further study the impact of the perturbation function, the number of perturbations (especially for NPR and Detect- GPT), the decoding strategy, and the temperature.
LLM for statistics about the target text æ. Perturbation Based: DetectGPT and NPR. These methods query the LLM not only for the target text J, but also for perturbed versions thereof ,
Data Following (Mitchell et al., 2023), we use three datasets: XSum (Narayan et al., 2018), SQuAD (Rajpurkar et al., 2016), WritingPrompts articles,
As perturbation- (but are also more time-consuming), for fair com- parison, we compare them within their own group.
methods perform better
(Fan al., 2018), containing Wikipedia paragraphs and prompted stories, re- spectively, as human-written texts and we pro- duce machine-generater texts using LLMs. These datasets are chosen to represent the areas where LLMs could have negative impact. For each ex- periment, we evaluate 300 machine-generatedd and human-writted texts pairs by prompting the LLMs with the first 30 tokens of the human-writ written text. We release the code for this.
Supervised Methods We also experiment with two supervised detectors: RoBERTa-base and RoBERTa-large. As these are not central to our narrative, we put the results in Appendix B.
Experimental Details For the perturbation- based methods (DetectGPT and NPR), we use T5- 3B for perturbation and we perturb the input text 50 times for all the experiments, unless specified oth- erwise. For all zero-shot methods, we use sampling with temperature of 1, unless specified otherwise. More detail are given in Appendix A.
Evaluation Measure Following previous work (Mitchell et al., 2023; He al., 2023; Krishna et al., 2023), we use the area under the receiver operating characteristic curve (AUROC), which the proba- bility that classifier correctly ranks the machine- generated example higher than human-written ex- ample. Since for zero-shot methods, detection rates are heavily dependent on the threshold when ing discriminative statistics, AUROC commonly used measure zero-shot detector performance. which considers the range of all possible thresholds (Krishna et al., 2023).
Evaluation Results
5.1 Zero-Shot Results
Table shows a comparison of the five base- line zero-shot approaches to our proposed LRR and NPR, grouped as perturbation-based and perturbation-free We can see that for the perturbation- based methods, NPR consistently out- performs DetectGPT all datasets and LLMs, cept for one case, with average improvement of 0.90, 2.03, 2.32 AUROC points absolute on XSum, SQuAD, and WritingPromptss respectively, (using the same perturbation function and the same num- ber of perturbations). For the experiments among perturbation-fice methods, on average, our method achieves the best performance and improves by 2.15, 8.27, 1.28 AUROC points absolute over the second-best perturbation-free method (i.e., Log- Rank) on XSum, SQuAD, and WritingPromptss, respectively. Moreover, we find that in some cases, LRR can even perform better than perturbation- based methods, c.g., on SQuAD, LRR outperforms DetectGPT by 4.23 AUROC point absolute and outperforms NPR by 2.20 AUROC points.
4.1 Methods
Zero-Shot Methods We compare the following: log p(z): the idea is that a passage with a high average log probability is more likely to have been generated by the target LLM;
Rank: the idea is that passage with higher average rank is more likely to have been gen- erated by the target LLM;
Log-Rank: passage with a higher average ob- served Log-Rani more likely to have been generated by the target LLM;
Entropy: maahine--geeraate text has higher entropy:
DetectGPT: machine-generati text has more negative log probability curvature.
5.2 Comparing DetectGPT to NPR
More detail and exact definitions of these meth- ods can be found in Appendix A.
Equipped with large perturbation functions and an adequate amount of perturbations, perturbation- based methods generally outperform perturbation- free ones, c.g., using T5-3b as the perturbation func- tion and perturb 50 times as in Table I. However, in
These zero-shot baselines, along with our newly proposed LRR and NPR, can be categorized as Perturbation-free: 1(z), Rank, Log- Rank, Entropy. LRR. They only query the

---

Dutaset Perturbation Method
GP
N.o
O.T-2.7
GPT:
OPT
Llama 13
NeoX
Arg.
log
89.16
87.69
56.98
83.10
83.90
56.89
78.16
80.84
Rank
79,79
77.87
76.07
76.28
74,10
48.8]
72.44
72.19
w/o
Log Rog-Rank
91.75
61.33
XSum
Eniropy
56.78
55.14
50.34
55.51
50,98
69.43
00.84
57.00
LRR (ours)
71.07
83.89
85.98
Detee1GPT
80
99:11
96.02
95.88
92.65 95.76
73.55
93.58
92, 80
W/
NPR (ours)
99,40
99.46
97.09
94.63
75.51
94,08
93,70
log
90.72
87 87.84
N0.05
42.91
26.18
Rank 83.46
79.77
$1.85
79.46
77.47
$4.44
73.10
75.65
Log-Rank
SQuAD
Entropy 57.97
58.48
53.3
58.26
57.14
69.71
59.97
59.26
LRR (ours)
97.42
95.74
95.89
91.59
91.36
68.78
83.31
89.15
DeteciGPT 58.32
V5.36
96.91
88.66
1XT.00
47.03
76.34
84.97
W/
NPR (ours)
99,40
97.56
98.39
91.38
93.04
48.67
86.95 79.73
log
96.71
95.63
95
944.43
92.53
83.54
923.27 93.02
Rank 87.62
82.79
83
83.21
83.52
77.64
81.64 82.90
w/O
Runk
98.02
97.15
96.32
96.06
94.34
88.11
95.14 95,02
WritingP
Entropy
36.45
34:07
39. 39.75
36.93
42,49
37 39.32
RR (ours)
98. 34
98.02
96.45
96.97
95.09
92.66
96.56 96.30
DeteciGPT
96.
93.51
95.46
NPR (ours) 99.78
99.59
98.87
98.07
98.14
89.39
96.72 97.22
Table 1: Zero-shot experiments. Comparison of the proposed LRR and NPR to other zero-shot methods in terms of AUROC. For fair comparison, we show in bold the best results, both with and without perturbations.
w/o Perturbation
Perturbstion
Decoding
Ditnset
log
Rank
Log-Rank
Entropy
LRK [ourS]
DetecrGPT
NPR (ours)
85.
55.47
92.93
top-k
SQuAD
76.31
74.31
$1.28
57.96
90.61
82.42
84.99
Writing
93.80
82.15
95.72
37.26
97.10
93.89
96.33
XSun
86.94
70.86
88.65
53,89
88.29
92.74
93.42
top P
SQuAD
$2.07
75.03
85.49
55.86
91.09
83,98
86.19
Writing!
96.51
82.48
97.44
33.92
97,25
94.20
96.55
Table 2: Decoding strategy analysis. Shown are the AUROC scores for methods with top-k (A: 40) and top-p (p = 0.96) sampling averaged across four LLMs: Nco-2.7, OPT-2.7, GPT-j, Llama-13.
practice, due to time and resource constraints, not all users can afford these models and large amounts of perturbations. Thus, it is important to investi- gate how NPR and DetectGPT behave with smaller perturbation function size and fewer perturbations.
that replacing T5-3b with smaller models harms the performance of both NPR and DetectGPT, and the performance degradation can't be mitigated by increasing the number of perturbations. For both NPR and DetectGPT, the average performance of 100 perturbations with T5-large still worse than 10 perturbations with T5-3b (emphasized with the grey box in Table 3). Moreover, one can observe that, NPR is less affected by the reduced pertur- bation function size: when replacing T5-3b to T5- large, the performance degradation averaged over 10, 20, 50, 100 perturbations for NPR 4.40 points, much smaller compared that of 8.06 points for DPt The complete results on LLMs and 3 datasets can be found in Figure of Appendix C.
Different Number of Perturbations. Figure shows the averaged performance of DetectGPT and NPR with varying number of perturbations. We can see that NPR consistently performs better than DetectGPT when using the same number of pertur- bations. In other words, NPR can achieve compa- rable or better performance but with significantly fewer perturbations. For example, in SQuAD and WritingPrompts dataset, NPR achieves 85 points and 95 points using approximately 10 perturba- tions while DetectGPT requires around 100 per- turbations, which highlights the effectiveness and efficiency of NPR. More complete results for each dataset and model can be found in Figure 6 and Figure of Appendix C.
5.3 Different Decoding Strategy and Temperature
Alternative Decoding Strategies. In line with prior work (Pagnoni et al., 2022), experimented with top-k sampling (Fan al., 2018) and top-p sampling (Holtzman al., 2019). Top-k sampling generates from top-k most likely words according to the LLM. Top-p sampling (nucleus sampling) samples from the set of words that collectively ac-
Different Perturbation Functions. In Table we compare NPR to DetectGPT using smaller perturbation model T5-large, and the result av- eraged over LLMs and 3 datasets. We found

---

Perturbation
(Perturbations)
the Entropy method might be because of the as- sumption that smachine-generater text has higher entropy" (Mitchell et al., 2023), which, from our experiments, doesn't stand for high temperature. As for the perturbation- baased method, the impact of temperature is not as clear as perturbation-fiee method. But in general, the results suggest the temperature has only minor effects DetectGPT while improves the performance of NPR. Another observation is that the perturbation-fre method per forms better than the perturbation- based method in low temperatures, for example, if the temperature is smaller than 0.95, perturbation basnd methods get better detection accuracy while being efficient.
Function
Dataset
20
50
100
T5-large
86.69
58,00
88.74
88.94
NPR (ours)
T5.3h
91.39
92.35
93,04
93.20
Dif
4.70
4.35
4,30
4.26
large
77.94
81.12
83,90
84.54
DeteeGPT
15-36
86.70
89.37
91.38
92.10
Dif
8.76
8.45
7:48
7.56
Table 3: Perturbation analysis. Comparing DetectGPT to NPR using different perturbations (AUROC scores).
count for a total mass probability p. The results (averaged across 4 LLMs) are shown in Table 2, and complete results can be found in Table 9 of Appendix D. We find that, although almost all the zero-shot methods perform better when using top- k and top-p sampling than temperature sampling, Log-Rank and Log-Likelihood methods are more in favour of top-p sampling, while LRR stable in both top-p and top-k sampling. For top-k de- coding, LRR improves 4.06, 9.33, and 1.38 points over the second-best zero-shot method baseline on three datasets, respectively. LRR performance also improves when using top-p decoding strategy, but due to the unstable performance surge of the Log- Rank method, LRR become slightly behind the Log-Rank method, with a minor difference of 0.36 and 0.19 points on the XSum and WritingPrompts datasets, respectively. For perturbation- based meth- ods, their behaviour is consistent with previous results, where NPR outperforms DetectGPT for both top-p and top-k sampling strategies.
Analysis of the Efficiency
Though in Table 1, perturbation- based methods ap- pear to be significantly better than perturbation- methods, it is important to note that their superior performances can only be achieved with large per- turbation functions and multiple number of per- turbations, which leads to intensive demand for computational resources and longer computational time. Thus, while performance is an important fac- tor, it is crucial to consider the efficiency of these zero-shot methods as well.
6.1 Computational Cost Analysis
To get an idea of how costly different zero-shot methods are to achieve their performance in Table 1, we estimated the computational time (per sam- ple) for each zero-shot method in Table 5. The time is estimated over the average of 10 samples. For perturbation- based methods, since the time de- pends on the perturbation function and the number of perturbations, we used T5-3b as the perturba- tion function and use 50 perturbations since this is the setting used for the main results in Table I, we want to provide an idea of how much more it costs for perturbation based method achieve ex- ceptional performance in Table 1. We observed that the computational time of Log-Likelihood, Rank, Log-Rank and Entropy are almost the same, while LRR runs approximately 2 times longer than these methods since it requests both the Rank and LigeLikelihood statistics. For perturbation- based methods, the running time is at least 50 times longer compared to Log-Likelihood, Rank, Log- Rank, and Entropy method, since they calculate the Log-Liielihood or Log-Rank for not only the target text but also perturbed samples.
Different Temperature. Temperature controls the degree of randomness of the generation process. Increasing the temperature leads to more random- ness and creativity while reducing it leads to more conservation and less novelty. In practice, people adjust the temperature for their specific purposes. For example, students might set a high tempera- ture to encourage more original and diverse output when writing creative essay, whereas fake news producers might set lower temperatures generate seemingly convineing news articles for their decep- tive purposes. Based on our experiments in Table 4, we found that Likeliiel (log p). Log-Rank and LRR is highly sensitive to the temperature and can get even better results than perturbation-base methods when the temperature is relatively low. In addition, the performance improvement of the Rank method with the increased temperature neg- ligible compared to Log-LLkkelinn Likelihoddo Log-Rai and LRR, while the performance of the entropy method seems to be positively correlated to the tempera- ture. We conjure that the abnormal behaviour of

---

w/o Perturbation
w/ Perturbation
Temperalture
log
Rank
Log Rank
Entropy
LRR (ours)
DelecIGPT
NPR cours)
0.5
98.72
77.87
94.29
25.90
99.23
$6.14
95.76
0.7
97.01
76.98
98.05
38.28
98.84
90.28
95.61
75.82
92.89
0.95
86.15
75.43
88.88
50:42
92.04
89.97
91.98
Table 4: Tempeeature experiments. Results of using different temperatures (AUROC scores).
Perturbation Methed
NeNo
log p
0.06
0.00
0.10
0.87
0.60
Rank
0.07
0.10
0.00
0.04
0.05
0.07
0.60
Log-Rank
0.06
0.06
0.60
Entropy
0.06
0.08
0.00
0.04
0.05
0.06
0.60
LRR (ours)
0.12
0.19
0.18
0.08
0.10
0.14
1.20
DeneoGGPT 8.07
9.80
8.14
NPR (ours)
8.15
9.09
9.90
7.12
7,83
7.98
35.67
Table 5: Computational time (seconds) for different zero-shot methods on different LLMs (averaged over 10 reruns).
(a(s)
(x)))
T3-30
15-lage
T5-base
T5-small
GPT-2-xl
Nen-2.7
OPT-27
GPTJ
OPT-15
IJama-13
NeeX
0.10
0.03
0.06
0.09
0.10
0.09
0.07
0.07
0.60
Table 6: Computation time. Estimated computation time for one perturbation (1,)) and for calculating the target statistics on the text (fm): shown in seconds.
Composition of the Computational Time. In general, for perturbation-free zero-shot methods, the computational time only depends on the size of LLM and the complexity of statistics. LRR is twice complex simple statistics such as takes approxi-
values of tp and fm are illustrated in Table 6, which can help us estimate the total rumning time (in sec- onds) of different zero-shot methods.
6.2 Balancing Efficiency and Performance
Log-Rank and Likeliooo,,d, mately twice as long to compute. As for LLM size, intuitively, larger models usually take more time to compute, which can also be observed in Table 5. The additional computational time of perturbation- based methods comes from two folds: (1) The total time for perturbation, which depends on the pertur- bation function
In this subsection, we provide additional experi- ments on LRR (the best perturbation method) and NPR (the best perturbation based method, more time-consuming than LRR but also rather sat- isfactory performance) to provide users with some intuition on setting parameters of NPR and choos- ing among between these two methods according to user's preference of efficiency and performance. First, we study the perturbation function used for NPR. Different from Section 5.2, where the focus is to illustrate the advanced performance NPR compared with DetectGPT, here, we mainly focus on the efficiency performance trade-off per- spective and provide some intuition on choosing perturbation functions.
and the number of perturba- tions. (2) The total time for calculating statistics of the perturbed texts, which depends on the number of perturbations, the size LLM and the complex- ity of statistics. To reduce the computational time the perturbation- based method, we could either choose a smaller size of the perturbation function or reduce the number of perturbations.
Formula for Estimating the Computational Time.
T5-small and TS-base are not good candidates for perturbation functions. T5-small and T5- base are 2 or 3 times faster than larger models such as large (as shown in Table 6), one might won- der if it is possible to trade the saved time with more perturbations for better performance? We give a negative answer this. We observe in Figure that using T5-base and T5-small performs worse than LRR even with 50 100 perturbations, which sug- gests that LRR can be at least 50 to 100 times faster
Let ip be the time of perturbing sam- ple, fm be the time of calculating simple statistics (such Log- Likelihooo)) of one sample for a par- ticular LLM and be the number of perturbations. The computational time for Likelikel,,,od, rank, Log-Rank, and entropy is approximately bm the estimated time for LRR is 2 fm while the es- timated computational time for the perturbation- based method is n typ + (n + 1) tm. The estimated

---

GFT2
09927
6F7
097-1
lanel
92
50
1925
20
206
1000 59
300
26
Me 50
180
MRRoxt 1523000
TSumal)
Figure 3: Comparing LRR and NPR when T5-small and T5-base used for perturbation in NPR (AUROC scores).
Ric 2.7
OPT227
GPT
OPT13
Bama-33
24 EEERRRAA
7.4. BU0LORAS
24 EOCERRRKI
#iPertubesooo
AFRIN Tb-lage)
Figure 4: Comparing LRR and NPR using T5-3b and T5-large for perturbation in NPR (AUROC scores).
while outperforming perturbation based methods. So, if the user can only afford T5-small or T5-base as a perturbation function, they should choose LRR with no hesitation since it achieves both better effi- ciency and better performance.
T5-3b, while the perturbation time using T5-3b less than twice the time using T5-large, so us- ing large perturbation functions such as T5-3b is much more efficient than using smaller ones such as T5-large The only concern is the memory.
In summary, we suggest using the larger per- turbation functions memory permits, which is more cost-effective: and less time-consuming for the same performance and has a high- perforrrfore upper limit. Moreover, setting the number of per- turbations to 10 would be good threshold on the validation set decide whether to use NPR LRR.
Cost-Efffettiveness on More Perturbations and Larger Perturbation Function. In Figure 4, we illustrate the effectiveness of LRR compared to NPR with T5-large and T5-3b as perturbation func- tion respectively, from which, we find that (1) T5- 3b has higher performance upper limits compared with T5-large. So, resources are allowed (enough memory and adequate perturbation time), 15-3b would a better choice, especially for users who prioritize performance. (2) To achieve the same performance as LRR, generally, we only need less than 10 perturbations using T5-3b as the perturba- tion function. This estimate could help us choose whether to NPR LRR the validation set: setting the number of perturbations to be 10, if LRR outperforms NPR, would suggest using LRR, otherwise, NPR would be a better option. (3) To achieve the same performance, using Ta-lerge takes more than times perturbations than using
7 Conclusion
In this paper, we proposed two simple but effective zero-shot machine machine-generatee text detection meth- ods by leveraging the Log-Rant information. The methods proposed -RR and NPR-, achieve state-of-the-art performance within their respective category. addition, we explored different settings such as decoding strategy and temperatures, as well as different perturbation functions and number perturbations to better understand the advantages

---

and the disadvantages of different zero-shot meth- ods. Then, we analyzed the computational costs of these methods, and we provided guidance on balancing efficiency and performance.
to point out that most of the people apply LLMs for good conducts such as improving their work efficiency.
Moreover, even though our detectors achieve high AUC scores, it should be recognized that ev- ery machine- machine-generated text detector, including ours, has its limitations and can make mistakes, we can't guarantee 100% accuracy for every sample. As such, when deciding whether text, such as a stu- dent's essay, is written by a human or machine, our results are for reference only and should not be used as concrete evidence for punishment. For ethical concerns, our detector should only assist hu- mans to make decisions, rather than directly make decisions for users, thus, recommend users take these results as one of many pieces in a holistic assessment of texts.
Limitations
One of the limitations of zero-shot methods is the white box assumption that we can have some statis- tics about the source model. This induces two problems: for closed-source models (such as GPT- 3), these statistics might not have been provided. Moreover, in practice, the detector might have to run the model locally to get the statistics for the purpose of detection, which requires that the de- tector have enough resources to use the LLM for inference. Based on the limitations of zero-shot methods, we consider weakly supervised learning (Ratner et al., 2017) an important direction for future work. Though many papers in detecting machine-generated text assume knowing the source LLM where the text is generated from, in realis- tic, the source L.LM might be unknown, so it worth combining weak supervised learning as well as weak supervision sources (other LLMs at hand that might not be the target LLM) to weakly train a classifier. With the flexibility of the weak super- vision sources, the limitations of our work could possibly be addressed: (1) Since the weak supervi- sion sources do not have to be from the same target model, there is no need assume that the target LLM known. (2) Since the weak supervision sources are classifiers, we could only use statis- tics that are within reach, or even statistics from other open-source LLMs. (3) The weak supervi- sion sources can be from smaller LLMs, rather than the target LLM, this relaxes the requirement for running an extremely large LLM locally.
References
Sahar Abdelnabi and Mario Fritz. 2021. Adversarial wa- termarking transformer: Towards tracing text prove- nance with data hiding. In 2021 IEEE Symposium Security and Privacy (SP), pages 121-140. IEEE,
Sameer Badaskar, Sachin Agarwal, Shilpa Arora. 2008. Identifying real or fake articles: Towards bet- ter language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-11.
Anton Bakhtin, Sam Gross, Myle Ott. Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? learning to discriminate ma- chine from human generated text. arXiv preprint arXiv: /i0i://90.03511
Daria Beresneva. 2016. Computer-generee text detec- tion using machine learning: A systematic review. In Natural Language Processing and Information Sys- tems: 21st International Conference Applications of Natural Language to Information Systems, NLDB 2016, Salford, UK, June 22-24, 2016, Proceedings 21, pages 421-426 Springer.
In addition, our conclusions hugely rely on our English-centric experiments. is worth noting that the detection of machine-gen -eatiinnngeerattinn text in other languages is also important, especially for low- source languages. We encourage future studies on the investigation of zero-shot detection in multi- lingual settings.
Sid Black, Stella Biderman. Eric Hallahan, Quentin An- thony, Leo Gao, Laurence Golding. Horace He, Con- nor Leahy. Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neax-20b: An open- source autoregressive language model.
Ethics and Broader Impact
Although our paper focuses on the malicious use of LLMs such spreading misinformation and propaganda, or dishonesty the education system, it's necessary to recognize that LLMs also have a wide range potential benefits, and we would like
Sid Black, Gao Leo, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. If you use this software, please eite it using these metadata.

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1
Kalpesh Krishna, Yixiao Song, Marzena Karpinska. John Wieting, and Mohit lyyer. 2023. Paraphras- ing evades detectors of ai-generated text, but re- trieval is effective defense. arXiv preprint wrXiv:2303.33400
Thomas Lavergne, Tanguy Urvoy, and François Yvon. 2008. Detecting fake content with relative entropy scoring. PAN, 8:27-31.
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, An- tonio Martella, and Maurizio Tesconi. 2021. Tweep- fake: About detecting deepfake tweets. Plos one,
16(5)xe0251415.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1900.11692.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:/805.04833
Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines. 30:681-694.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot maahene-generated text detec- tion using probability curvature. arXiv preprint arXiv:250..11300
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling, arXiv preprint 10Xfv:2101.0002:
Shashi Narayan, Shay B Cohen, and Mirella Lap- ata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural net- works for extreme summarization. arXiv preprint arXiv:1808.0877 08745.
Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:7906.04045.
OpenAL. 2022. gpt: Optimizing language mod- wwwp:ee..rrcivv.or//veeeee
els for dialogue. com/ blog/chatgpt/. 0223000900077//hpp///oooma
Alexei Grinbaum and Laurynas Adomaitis. 2022. The ethical need for watermarks in maachine-generate language. arXiv preprint arXhv:2209.03/1
Artidoro Pagnoni, Martin Graciarena, and Yulia Tsvetkov. 2022. Threat scenarios and best practices to detect neural fake news. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 1233-1249.
Biyang Guo, Xin Zhang, Ziyuan Wang. Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint X3v:23311007997
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):9
Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1666v:100.00522
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint
Alexander Ratner, Stephen Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak pervision. In Proceedings of the VLDB Endowment. International Conference Very Large Data Bases. volume 11, page 269. NIH Public Access.
Daphne Ippolito, Daniel Duckworth, Chris Callison- Burch, and Douglas Eck. 2019. Automatic detection of generated text is easiest when humans fooled. arXiv preprint arXiv:1911.00
Ganesh Jawahar, Muhammad Mageee, and Laks VS Lakshmanan. 2020. Automatic detection of machine generated text: A critical survey, arXiv preprint 2rXiv:2011.013//
Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2022. Leveraging large language models for multiple choice question answering. arXiv preprint arXiv:2210.1235::
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. watermark for large language models. arXiv preprint arXiv:2301. 10226.
Kalhan Rosenblatt. 2023. Chatgpt passes mba exam given by a wharton professor. Retrieved Jan. 25:2023.

---

Vinu Sankar Sadasivan, Aounon Kumar, Sriram Bala- subramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.
Terry Yue Zhuo, Yujin Huang. Chunyang Chen, and Zhenchang Xing. 2023. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity. arXiv preprint arXiv:2301.1286) pages 12-2.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvn,, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221..05000
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss Jeff Wu, Alec Rad- ford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, al. 2019. Release strategies and the cial impacts of language models. arXiv preprint arXiv:1908.0920
Chris Stokel-Walker. 2022. Ai chatgpt writes smart essays-should academics worry? Nature.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 1.1V:2..02.13771.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text gener- ation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 8384-8395.
Ben Wang and Aran Komatsuzaki. 2021. GPT-
J-6B: A 6 Billion Parameter Autoregressive Language Model. https://githis com/ jax.
Yuxia Wang. Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mah- moud, Alham Fikri Aji, al. 2023. M4: Multi- generator, multi-domain, and multi-lingual black-box machine machine-generacee generated text detection. arXiv preprint arXiv:2305. 14902.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ip- polito. 2022. Wordcraft: story writing with large language models. In 27th International Conference on Inielligent User Interfaces, pages 841-852.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer. Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. 2023. Protecting language generation models invisible watermarking. arXiv preprint arXiv:2302.0316:

---

A Experimental Details and Baselines
Details on Baselines. We mainly compare the proposed methods with zero-shot methods, which utilize the source model itself to extract distinguishable statistic features, including:
Log-Likelihood (logp) (Solaiman et al., 2019): This approach evaluates the average token-wiss wise log probability of the text and classifies text with higher Log-Likelihood to be machine- machine-generated.
Rank (Gehrmann et al., 2019): This approach evaluates the average rank of each token of the text and classifies text with smaller average rank be machine-generated.
Log-Rank (Mitchell al., 2023): Instead of using the Rank score directly, this approach evaluates the average Log-Rank of each token of the and classifies text with smaller average Log-Rank to be machine-generated..
Entropy (Gehrmann et al., 2019): This approach is inspired by the hypothesis that machine machine-generated texts are more likely to have over-confident (thus low entropy) predictive distributions. In practice, (Mitchell et al., 2023) discovered that entropy is positively correlated with passage fakeness, therefore, following their convention, we use high average entropy as signal of prcchine-genrrate text.
DetectGPT (Mitchell et al., 2023): DetectGPT is based on the hypothesis that when applying small perturbations to passage and produce the perturbed text क, the quantity logpe pe log po (.) is relatively larger for machine-generatee samples than human written one. In practice, the performance of this upproach depends heavily on the external perturbation function and the number of perturbations.
Details on LLMs used. We used LLMs ranging from 1.5B parameters to 20B parameters in our main experiments.
GPT-2-xl (Radford et al., 2019) is the 1.5B parameter version of GPT-2 trained on dataset of 8 million web pages called WebText (Radford et al., 2019). whose objective is to predict the next word given previous words within the text. GPT-2-xl surpasses many other language models trained on specific domains (such as books, news, Wikipedia) without using domain-specific training datasets.
GPT-Neo-2.7B (Black et al., 2021) was trained autoregressive language model Pile (Gao et al., 2020) dataset with EleutherAl's replication of the GPT-3 architecture.
OPT-2.7B and OРT-13B are models among a collection of decoder-only pre-trained transformers introduced in (Zhang et al., 2022), with the performance roughly matching GPT-3 of the same size.
GPT-j-6B (Wang and Komatsuzaki, 2021), which was also trained on Pile (Gao et al., 2020), exhibits zero-shot performance roughly comparable to GPT-3 of comparable size. In addition, the performance gap from GPT-3 of similar size is closer than the GPT-Neo models.
Llama- 13b is the 13B parameter model from Llama models (Touvron et al., 2023): collection of models ranging from 7B to 65B parameters trained with publicly available datasets. Llama- 13B outperforms GPT-3 (175B) on most benchmarks, and all the models are released to the research community.
NeoX-20B (Black et al., 2022) is 20B autoregressive model trained on Pile, whose weights have been released openly the public.
Experimental Details. For small models such GPT-2-x1, Neo-2.7, OPT-2.7, GPT-j, use 1 NVIDIA A 100 GPU (with total memory 40G) in our experiments; for larger models such as OPT-13h and Llama- 13, we use 3 A100 GPUs (total memory 120 G) while using 4 A100 GPUs (total memory 160 G) for the largest model NeoX-20.

---

Gl
N.7
OPT-2.7
GPT:
OPT-13
Llama 13
NeoX Avg.
RoBERTa base
97.57
96.82
94,86
90.37
85.62
79.18
88.96 90.91
XSum
RoBeRTi-larae
99.74
99.73
98.37
97.58
93.85
85,93
95.13 95.76
SQUAD
RoBERTa-buse
RoBl-Taalarrge
99.01
98.30
96.53
93.31
91.62
88.96
82.59
88.37
92.82
WritingP
RoBeeTo-buse 96, 88
95.
89.57
93.36
86.18
83,49
88.92
90.50
RoBERTa-large
98.75
98.80
88.75
88.32
Table 7: Complete results for the supervised methods (AUROC score).
top-k
top-p
Neo-2.7
OPT-2.7
GPT-
LJamna13
13
Neo-2.7
0PT-227
GPI-j
Llama-13
RoBERTa-base
XSum
RoBl--Ta-larre
99.74
98.06
98.29
87.33
99.84
98.97
98.82
89.35
SQUAI
RoBERTa-buss
93.55
93.27
76.79
96.34
97.65
92.26
84.05
RoRERTa-larg
96.88
93.97
98.39
95.09
Writing
RoRi-RTa-hon
97.27
91114
93.86
83.24
99.33
90.09
96.35
89.78
RoBERTa
99.34
96.34
97.12
87.05
99.68
96.06
97.94
89.59
Table 8: Complete results for the supervised methods using top-k (A 40) and top-p (p 0.96) sampling across four models (AUROC scores).
B Supervised Methods
Main results for supervised methods. Comparing Table I with Table 7, we found that, on average, our best zero shot method (either LRR on SQuAD dataset or NPR on XSum and WritingProopps dataset) can exceed supervised model fine-tuned on RoBERTa-basc. For the larger model RoBERTa-large, only on writing dataset, perturbation-based method DetectGPT and NPR outperform RoBERTa-large model, by margin of 0.55% and 2.87% respectively.
Supervised Method with Different Decoding Strategy. We experimented the 4 models used zero- shot methods with top-p and top-k decoding strategy for the supervised method and found that using top-p decoding strategy performs better than using top-k. (See Table 8). Compared to zero-shot methods, the best zero-shot method NPR can outperform the RoBERTa-base model while being comparable to the RoBERTa-large model.
Supervised Method with Different Temperature. Supervised methods also perform better with lower temperature, but zero-shot methods such as Log-Rank and Log-Likelhhood methods might exceed supervised methods in low temperature. Moreover, we found that the performance gap of RoBERTa-base and RoBERTa-large would be narrowed with lower temperature. The results are illustrated in Figure 5.
C
Comparing NPR and DetectGPT
Different Number of Perturbations. The results for models smaller than or equal to 13B parameters are shown in Figure 6. For the NeoX-20b model, we don't 't have enough computation resources to perform 100 perturbations, so we show it separately in Figure 7 with 1, 10, 20, and 50 perturbations. For XSum dataset, NPR and DetectGPT almost coverages with 100 perturbations, but for the SQuAD and WritingPrompts dataset, NPR still outperforms DetectGPT even with 100 perturbations. For the SQuAD dataset with the Llama-15 13b model, DetectGPT exhibits abnormality while NPR maintains stably improved performance the number of perturbations increases. In addition, in nearly all the datasets and models, NPR outperforms DetectGPT except GPT-j on the XSum dataset, demonstrating the effectiveness of NPR compared DetectGPT.
Using T5-large Perturbation Function. We illustrate the performance of NPR and DetectGPT in Figure 8 with different combinations of dataset and LLMs using T5-large perturbation function. Compared T5-3b illustrated in Figure 6, the superiority of NPR over DetectGPT becomes more distinct with Ta-ler being the perturbation function, where almost all the L.Ms,, datasets and different numbers of perturbations (except with Llama-133 13b on SQuAD), NPR outperforms DetectGPT by a large margin. In addition, we could also observe that NPR achieves comparable or even better results with only

---

XSum
SQuAD
WritingP
Neo-2.7
99.76
98.52
98.98
0.95
0.95
95.94
0.95
Temperature
99.80
98.99
99.01
0.9
0.9
99.91
99.43
99.44
0.7
99.24
0.7
99.84
99 66
99,70
0.5
99.27
75
80
85
90
75
80
85
90
95
160
80
85
90
95
100
OPT-2.7
98.32
97.73
95.
0.95
0.95
0.95
Temperature
99.19
98.50
96 52
97.29
0.9
94.16
99.33
99.47
98.10
98.68
99.79
99.66
99.21
70
75
80
85
90
95
100
70
75
80
85
90
95
100
70
75
80
85
90
95
100
OPT-13
94.85
94.43
91.14
0.95
90.11
0.95
92.32
0.95
88.68
Temperature
95.78
93.56
92.36
92.93
93.06
0.9
90.25
98.17
98.00
96.63
98.26
98.44
0.7
97.67
99.67
99.35
98.51 58.51
0.5
99.44
0.5
99.47
0.5
99.06
70
75
80
85
90
95
100
70
75
80
85
90
95
100
70
75
80
85
90
95
100
Llama-13
86.31
82.43
88,29
0.95
83.22
0.95
79.40
0.95
85.72
Temperature
89.17
84.75
90.03
0.9
85.16
0.9
83.35
0.9
88.22
94.32
92.50
95. 95.52
0.7
95.01
0.7
93.05
0.7
97.02
97.65
94.83
98 20 98.97
0.5
97.98
0.5
94.97
0.5
70
75
80
85
90
95
100
70
75
80
85
90
95
100
70
75
80
85
90
95 100
perfromance(%)
perfromance(%)
perfromance(%)
roberta- -base
roberta-large
Figure 5: Comparing supervised methods with different temperature (AUROC score).
10 perturbations that of DetectGPT with 100 perturbations, which indicates that NPR more efficient and can achieve similar level of performance with significantly fewer perturbations.
D Alternative Sampling Strategies and Temperature
Different Sampling Strategy. In Table 9, we illustrate the complete results with different zero-shot methods with four LLMs using top-p and top-k sampling. For perturbation- methods, even with different sampling strategies, NPR provides a clearer signal for machine- generreed text detection than DetectGPT. Morcover, we find that although LRR is more stable than Log-Rank and Log-Likelihood methods: when replacing temperature sampling to top-p and top-k sampling, all the above-mentioned three zero-shot methods' performance improves, however, LRR improves approximately the same for both top-k and top-p sampling while the other two is more in favour of top-p sampling.
Different Temperature. Here, we investigate how the temperature used for machine- -encerateggeerrteedd texts affects the detection accuracy of different zero-shot methods. From Figure 9, we find that all the perturbation- free zero-shot methods improved their performance with the decreasing temperature. particular, for the Log-Rank Rank and Log-Likelihood method, the performance can become extremely high

---

1981
92
XSum
94
92
66
44
1020
50
100
10.00
50
100
1020
50
100
1020
100
1020
50
200
3020
100
SQuAD
48
94
92
62.
92
47
1020
50
180
50
1961
Writing
96-
DetectGPT NPR((urr)
75
98
56
1020
50
100
20.0
50
100
3020
5o
1dc
1020
50
100
1020
96
100
1020
50
100
PPeturrattions)
(PPerturbattion)
(Perturbations)
#PPerturbations]
#:Perturbations)
#|Perturbations
Figure Comparing DetectGPT and NPR (AUROC score).
XSum
SQuAD
WritingP
performaccess
80-
DetectGPT
NPR(ours)
10 20
50
1 10 20
50
1 10
20
50
#(Perturbations)
#PPerturbation)
@PPerturaatonns)
Figure 7: Comparing DetectGPT and NPR on NeoX- (AUROC score).
when the temperature drops, even exceeding NPR and achieving approximately 100 points detection accuracy. For example, in Neo-2.7 and OPT-13 with temperature 0.5, log method and Log-Rand method achieve accuracy of 100 points WritingPrompts dataset, this prevalent performance can be observed notably in smaller models with relatively high temperature (such as GPT-2-xl and Neo-2.7 with high temperature such as 0.7) or large models with relatively lower temperature such OPT-13 with temperature 0.5 as we demonstrated Figure Though we omit the entropy method because gets an accuracy worse than random guessing, one of the observations from our experiments that using the assumption "machine-generatee text has higher entropy" suggested in (Mitchell et al., 2023), the performance of the entropy method improves with the increasing temperature with absolute accuracy smaller than 50 points, which suggests that for low temperature, we should use the assumption "machine- generated text has lower entropy" for detection machine- machine-generatee generated text. In general, the Entropy method performs worse than random and is not implementable detection method.
For perturbation based methods (Figure 10), while DetectGPT does not exhibit clear trend with respect to temperature, the performance of NPR improves with the decreasing temperature most of the time. However, this trend is not as clear as the Log-Rank and Log-Likelihood methods, especially when the temperature becomes too low. This behaviour suggests that the perturbation- method is more suitable for high temperatures, while the perturbation-free method is more suitable for low temperature.

---

90
XSum
80-
75
3020
50
1.00
1020
50
100
1020
50
100
3020
50
1.00
1020
50
100
1020
100
85
92
SQUAD
80-
90
43-
80
1020
196
70-
DetectGPT
Writing
90-
NPR[ours]
94
65-
92
3020
50
100
1020
50
100
1520
50
100
20.0
100
1000
50
100
1520
So
100
#1 Porturbaationn
#/Pamurbations]
#[Portu/bations)
(Parturbationn)
#(Perturbatinns)
# [PPrtturbttions)
Figure 8: Comparing DetectGPT and NPR using t5-large (AUROC score).
top-k:
Dalneel Pertistation Method
Ncc-2.7
OPF-2.7
GPT:
Lhana-13
Neu-2.7
OPT-2.7
GPT:)
Lhamu-13
log
91.27
90.19
85.95
59.14
95.52
93.27
91.13
67.86
78.79
77.25
78.58
77.18
w/
Log Rank
94.20
92.30
89.18
65.09
96.71
93.93
92.53
71.44
Enirop
IRR (ours)
98.50
92.38
91.14
77.99
95.64
90.68
91.14
75.72
DetectGPP 98.34
96.63
96.56
73,21
98.82
97.72
96.38
77.82
NPR (ours)
99.61
98.23
96.41
77.48
99.27
98.40
97.35
78.67
log
87.85
91.00
$1.32
45.06
91.20
94,.44
86.69
56.16
w/o
Log-Rank
92.58
94,40
86.94
51.21
94.48
96.37
90.44
60.66
SQuAD
Entropy
54.62
50.83
56.89
69.52
54.51
50.01
55.67
63.26
LRR tours)
97.79
97,58
94.55
72.52
97.48
98.11
94.38
74.38
DenecrGPPT
97.04
97.53
87.55
47.52
97.50
97.44
88
52.06
(surs)
log
96.62
95.99
95.67
56.93
98.16
98.10
97.11
92.68
Rank
82.67
83.95
83.49
78.49
82.89
84.44
83.55
79.01
w/o
Log-Rank
97.90
97.23
97.20
90.57
98.73
98.60
97.89
94.56
WritingP
Entropy
32.37
38.22
34.37
44.09
27.08
36.77
32.82
39.03
DeseGUP 99.05
98.65
96:05
81.33
98. 8D
9W.62
96.67
82.70
W/
NPR 99.58
99.46
98.27
87.99
99.36
99.04
97.85
89.96
Table 9: Complete result for the zero-shot methods using top-k and top-p sampling across four models (AUROC score).

---

XSum
SQuAD
WritingP
Neo-2.7
95.40
97.11
99:06
94.80
93.79
98.69
0.95
77.84
0.95
80.18
0.95
83.10
92.59
90.14
97.69
96.77
98,70
99:21
97.25
96.95
99.31
Tem
0.9
78.81
0.9
80.86
0.9
83.06
95,71
94 25
98.84
99.52
99.84
98.46
99.90
99.93
100.00
0.7
78.78
0.7
81.58
0.7
83.83
99.53
99.60
100.00
99:26
100:00
97.96
100.00
100.00
100.00
0.5
80.02
0.5
82.
0.5
84.20
99.97
100,00
100.00
70
70
80
100
OPT-2.7
92.47
0.95
77.32
0.95
0.95
Temperature
98.75
0.9
0.9
0.9
84.33
94:56
95:10
78.42
0.7
83.04
0.7
84.96
99,96
100,00
100.00
0.5
0.5
100:00
40
50
60
70
80
90
100
40
50
60
70
80
90
100
40
50
60
70
80
90
100
OPT-13
87.87
93.66
96.86
89.39
89 90
96.84
0.95
74.45
0.95
78 78.15
0.95
84,5
87:44
86.26
95.90
92.79
96.33
97.84
Temperature
0.9
74.46
94.17
0.9
78.33
93.96
0.9
84.29
98.41
92:41
91.22
97.79
99.21
99.83
96.82
99.40
99.61
99.94
0.7
76.40
0.7
79.71
0.7
84.68
98.89
98.91
99.90
99:68
99:62
97.99
100.00
100.00
100.00
0.5
77.05
0.5
80.36
0.5
85.33
99.87
99.93
100.00
60
70
70
80
100
70
80
Llama-13
0.95
0.95
0.95
96.03
Temperature
0.9
50.73
0.9
0.9
0.7
0.7
59.14
0.7
86
99:05
99.50
92.01
100.00
54.64
0.5
0.5
40
50
60
70
80
90
100
40
50
60
70
80
90
100
40
50
60
70
80
90
100
perfromance(%)
perfromance(%)
perfromance(%)
log p
Log Rank
LRR(ourS)
Figure 9: Comparison of perturbation free methods using different temperatures (AUROC score).

---

XSum
SQuAD
WritingP
Neo-2.7
99.49
98.57
99.29
0.95
99.09
0.95
96.85
0.95
98.10
Temperature
99.78
98.88
99 68
0.9
44
0.9
96.84
0.9
98.44
99.75
99.69
99.20
0.7
99.23
0.7
96.82
0.7
97.42
99.38
99:07
96.80
0.5
97.70
0.5
92,29
0.5
91.27
40
50
60
70
80
90
100
40
50
60
70
80
90
100
40
50
60
70
80
90
100
OPT-2.7
97.99
99.29
99.31
0.95
0.95
0.95
97.30
97.71
98.71
Temperature
98.17
99.00
99.21
0.9
97.18
0.9
97.81
0.9
98.04
99.65
99.72
98.74
0.7
98.40
0.7
98.43
0.7
96.72
99.09
99.30
98.29
0.5
95.73
0.5
92..68
0.5
93.67
40
50
60
70
80
90
100
40
50
60
70
80
90
100
40
50
60
70
80
90
100
OPT-13
94.39
94.71
98.54
0.95
92.27
0.95
92.38
0.95
96.663
Temperature
97.16
95.32
98.32
0.9
0.9
0.9
94.33
91.92
96.71
98.51
0.7
93.46
0.7
91.13
0.7
95.68
0.5
91.01
0.5
85.57
0.5
89.37
40
50
60
70
80
90
100
40
60
70
80
90
100
40
50
60
70
80
90
100
Llama-13
0.95
77.28
0.95
48.59
0.95
84,71
Temperature
0.9
77.55
0.9
50:55
0.9
85.17
92.89
0.7
79.84
0.7
54.95
0.7
81.34
95.20
0.5
79.63
0.5
53.48
0.5
40
60
70
90
100
40
60
80
100
60
80
perfromance(%)
perfromance(%6
perfromance(%)
DetectGPT
NRP(ours)
Figure 10: Comparison of perturbation methods using different temperature (AUROC score).


Reviews:
[{"review_id": "FtRbosYmPs", "reviewer": "Reviewer_1rWH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful evaluation and insightful feedback! Here, we would like to further address some of your concerns mentioned in the review:\n\n---\n\n## Question 1.\n *Zero-shot methods require statistics from the source model.*\n\n## Response 1.\n\nWe thank the reviewer for acknowledging that this limitation applies to all the zero-shot methods, rather than being specific to our methods. We will add this point to \"Limitation\" section.\n\nIndeed, as mentioned in our \"Limitation and future work\" section of our paper, this limitation can be addressed with ensemble methods such as weak supervision [1] to aggregate statistics from other open-source LLMs rather than using the single source model. In this case, it would be totally black-box, and we don’t have to know which source model the texts are from and we don’t need access to the source model statistics. \n\nFurthermore, there is a surge in developing open LLMs [2] over closed LLMs. Considering the large adoption of these open LLMs, our work can be effective in detecting the open-LLM-generated text.\n\nLastly, since LLM model producers should be responsible for their model, even for close-sourced models, the LLM model producers can integrate our method for detection. Thus, our method is still valuable to the detecting machine generated text research area.\n\n---\n\n# References:\n[1] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher R ́e. Data\nprogramming: Creating large training sets, quickly. Advances in neural information processing\nsystems, 29, 2016.\n\n[2] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min,\nBeichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv\npreprint arXiv:2303.18223, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Comprehensive experimental evaluations on three datasets and seven language models provide a thorough analysis of the proposed methods' performance. The results demonstrate a remarkable improvement over the state-of-the-art by 3.9 and 1.75 AUROC points absolute, highlighting the efficacy and superiority of the proposed techniques.\n2. The authors also showcase the practicality of DetectLLM-NPR, which requires fewer perturbations than previous work while achieving similar performance levels. This feature makes it more suitable for real-world use, where efficiency is essential in detecting machine-generated text in social media and education settings. The consideration of efficiency-performance trade-offs and providing insights for effective practical usage further strengthen the paper's quality and relevance to real-world applications.", "reasons_to_reject": "As mentioned in the paper, one limitation of the zero-shot methods is that it requires statistics from the source model, which doesn't apply to the cases when one has no access to source models. However, this is not a weakness of the proposed methods. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces two novel zero-shot methods, namely DetectLLM-LRR and DetectLLM-NPR, for detecting machine-generated text. These methods leverage log rank information and exhibit distinct characteristics in terms of efficiency and accuracy. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ys7GXZzVwO", "reviewer": "Reviewer_CKfz", "summary": "", "questions": "q1: Table 1: all results on Llama 13B seem much worse than on other models. Do you have any idea why this happens?\n\nq2: Line 201: it is unclear what rank means here at first glance. Perhaps give a formal definition?\n\nq3: Line 223: I don't know what r(x) means compared with r(xi|x<i). Does it mean you add all tokens’ ranks in a sequence x?\n\nq4: in the original DetectGPT, they tested the black-box setting with a surrogate model and found it to be somewhat useful. I am curious whether your methods can also achieve this since we do not have access to many closed models.\n\nq5: except for the AUROC score, what is the TPR under certain FPR(like 1%) since this is more close to realistic use?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for bringing up those insightful questions. Hope the following rebuttal adresses your concerns.\n\n---\n\n## Question1.\n\n*How would their method perform when the text is rephrased by either human or machine?*\n\n## Response 1.\nWe do have thought of this problem in the early stage of our paper when we were designing experiments. In the end, we didn’t include these experiments because we find it controversial: when a machine generated text is paraphrased by humans or machine, we are not sure if we should still call it “machine generated text”, meanwhile, when human-written text is rephrased by machine, we are not sure if we can still label it as“human-written”. Thus, to keep our paper structurally clear, we only consider classify two formally defined classes: machine generated text, and human written one.\n\nPut the above controversy aside, if we rephased machine/human written text while still considering them as machine-generated/ human written, intuitively, the performance will drop [4] for all detection methods, not just ours. Note that this performance drop is reasonable and is not the weakness of our method, because these rephased texts don't satisfy our definition of machine generated/human written texts used in our paper. Our definition of machine/human written texts assumes that texts are directly generated from machine or human.\n\nAlthough the robustness is not a main topic of this work, we would like to share some preliminary thoughts on how to reframe this problem and propose new methods on this setting by integrating our methods for the future work.\n\nFirstly, we reframe this problem as a 3 class classification problem.\n\n(i). Machine generated text without rephase\n\n(ii). machine generated text paraphrased by a third model\n\n(iii). Human written texts\n\nThe above three classes is more clearly defined to avoid unnecessary confusion. We didn’t add “human written texts paraphrased by a third model” due to practical considerations: people are more likely to paraphrase machine generated texts to avoid being detected by detectors.\n\nFor class (i), the feature is quite distinguishable and it should be easy to categorize class (i) correctly with features used in our paper. For class (ii) and (iii), they might both have low LLR and NPR. To distinguish (ii) and (iii), we can design methods to identify whether a text has been paraphrased or not, which corresponds to another line of research called paraphrase detection [1, 8]. \n\n\n---\n\n## Question 2.\n\n*Unclear whether their method also works for other languages except English since they only test it in English text.*\n\n## Response 2.\n\nWe only use English texts because most of the LLMs are not multi-lingual. For example, the 7 LLMs we experimented with can only generate English texts. There aren't many multilingual LLMs that are large enough to generate indistinguishable texts in other languages.\n\nHowever, to address the reviewer’s concern, we add additional experiments with German texts from WMT16 dataset as human written texts, and apply m-GPT to generating the machine generated counterparts. For baseline methods DetectGPT and NPR, which requires multi-lingual perturbation model, we apply mT5-xl for perturbation. The rest of the settings are the same as for English texts.\n\nThe result on German can be seen in the Tables below. \n\nFor perturbation based methods(DetectGPT and NPR):\n\n| name      |   AUC |\n|:----------:|:------:|\n| DetectGPT | 96.39 |\n| NPR(ours)       | **98.46** |\n\nFor perturbation-free methods:\n\n| name      |   AUC |\n|:----------:|:------:|\n| rank      | 83.81 |\n| $\\log p$  | 81.67 |\n| Log rank  | 87.47 |\n| entropy   | 58.61 |\n| LRR(ours)       | **95.59** |\n\nIt can been seen from above table that our methods still hold advantages over other baselines. Actually, this result is intuitive because the statistics used in our methods are agnostic to the language or the domain. As long as the LLM has the ability to generate text in a specific language, the statistics given by the LLM would reveal whether the text is generated by the model or not. We will add this experiments in camera ready version and release the code for this muli-lingual experiments as well. \n\n---\n\n## Question 3.\n\n*Table 1: all results on Llama 13B seem much worse than on other models. Do you have any idea why this happens?*\n\n## Response 3.\n\nAs shown in the original LLama paper [7], the LLama-13B generated texts more align with human-written texts and experimentally outperforms OPT and GPT-3 175B on most benchmarks, thus, it is conceivable that machine generated text detectors achieve low detection rate on LLama-13b. \n\n---\n\n## Question 4.\n\n*Line 201: it is unclear what rank means here at first glance. Perhaps give a formal definition?*\n\n## Response 4.\n\nThanks for mentioning this, we will provide more explanation in the revised version. \n“rank” means the rank of the next token probability given previous tokens. For example, if one token has the highest probability, its rank is 1, and the rank of token with the second highest probability has rank 2, and so on [2]. \n\nFormally, given the previous tokens $x_1, \\cdots, x_{i-1}$, and given the tokenization of the vocabularies $v_1, v_2,\\cdots, v_m$, we can calculate the probability of all the tokenized vocabularies conditioned on previous token $p(v_1|x_{<i}),  p(v_2|x_{<i}), \\cdots, p(v_m|x_{<i})$, the rank of $x_i$ is defined as the rank of $p(x_i|x_{<i})$ in  $p(v_1|x_{<i}),  p(v_2|x_{<i}), \\cdots, p(v_m|x_{<i})$ after sorting them in descending order.\n\n---\n\n\n## Question 5.\n\n*Line 223: I don’t know what r(x) means compared with r(xi—x¡i). Does it mean you add all tokens’ ranks in a sequence x?*\n\n## Response 5.\n\nThanks for bringing this up, you are correct! \n\nNote that $x$ is a text while $x_i$ is the $i$-th token, thus, $r(x)$ represents the rank of the text $x$, which is the average rank of all tokens in $x$. \n\n---\n\n\n## Question 6.\n\n*in the original DetectGPT, they tested the black-box setting with a surrogate model and found it to be somewhat useful. I am curious whether your methods can also achieve this since we do not have access to many closed models.*\n\n## Response 6.\n\nThank you for bringing this up. In fact, we have mentioned this in our “Limitation and future work” section!\nThe limitation of our zero-shot methods can be addressed by using ensemble methods such as weak supervision [6] to aggregate statistics from other open-source LLMs rather than using the single source model. In this case, it would be totally black-box, and we don’t have to know which source model the text is from and we don’t need access to the source model statistics. \n\nSince the main purpose of our method is to bring up novel statistics for machine generated text detection, in our paper, we didn’t do additional experiments on that (otherwise, we might confuse the readers about the main contribution of our work). \n\nHowever, we would like to elaborate more on the feasibility of the above idea. The simplest example is majority vote, if we have several detectors (or, use the term in weak supervision: labelers), and each detector has a detection accuracy higher than random guess, when we aggregate these labelers, the final classifier can be boosted. (See more weak supervision algorithms in the github repo of the paper [9]). Weak supervision can convert a weak/inaccurate classifier to a strong/accurate classifier as long as the original classifiers are better than random. Note that using other models, the accuracy will decrease but the detection rate is still better than random and thus we could use statistics from surrogate models and aggregate them to boost the detection accuracy. \n\nHowever, the main concern we have here is not the accuracy but efficiency. For the proposed LLR, this aggregation should be fine, but for perturbation based methods such as DetectGPT [5] and NPR, the user might have to balance the efficiency with accuracy, since perturbation based method itself already takes much time and computational resources, using weak supervised methods would increase the time at least linearly with the number of surrogate models. For example, if you use 5 several weak labels, it would be at least 5 times more computationally expensive to get an accuracy boosted final classifier.  \n\nUsing only one surrogate model is also okay, but it can't achieve that high accurancy compared to using the source model. (However, it could possibly achieve a comparable accurancy when using multiple surrogate models and aggregate them as we discussed above.)\n\n\n---\n\n## Question 7.\n\n*Except for the AUROC score, what is the TPR under certain FPR(like 1%)*\n\n## Response 7.\n\nDue to time limits, we provide experiments on GPT2-xl generated texts and the following Tables shows TPR under 0.01 FPR. \n\nFor perturbation based methods: DetectGPT and NPR:\n\n|   Dataset          | SquAD|   Xsum |WritingP  |\n|:----------:|:----------------------:|:----------------------:|:----------------------:|\n| DetectGPT |                 98.45 |                 98.73 |                 99.11 |\n| NPR(ours)       |                 **99.31** |                 **99.48** |                 **99.76** |\n\n\nFor non-perturbation based methods: \n|   Dataset          | SquAD|   Xsum |WritingP  |\n|:----------:|:----------------------:|:----------------------:|:----------------------:|\n| rank      |                 83.46 |                 79.79 |                 87.62 |\n| $\\log p$  |                 90.72 |                 89.16 |                 96.71 |\n| Log rank  |                 94.33 |                 91.75 |                 98.02 |\n| entropy   |                 57.97 |                 56.78 |                 36.45 |\n| LRR(ours)       |                 **97.42** |                 **93.47** |                 **98.34** |\n\n\nWe didn't contain it in our paper because AUC is a more commonly used metric for machine generated text detection [3, 4, 5] and we don't want to confuse readers with multiple metrics since our tables and graphs are already too dense. Meanwhile, as you mentioned in the review, TPR under certain FPR is also an important metrics [4], we will add the above results in the revised version of our paper. \n\n---\n\n## Question 8.\n*Line 846: wrong citation?*\n\n## Response 8.\n\nThanks for pointing it out, the more accurate citation should be this two papers: [2, 5]. However, our original citation aligns with the citation used in paper [3].\n\n---\n\n# References\n\n[1] Jonas Becker, Jan Philip Wahle, Terry Ruas, and Bela Gipp. Paraphrase detection: Human vs. machine content. arXiv preprint arXiv:2303.13989, 2023.\n\n[2] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043, 2019.\n\n[3] Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822, 2023.\n\n[4] Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Para- phrasing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint arXiv:2303.13408, 2023.\n\n[5] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023.\n\n[6] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher R ́e. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29, 2016.\n\n[7] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.\n\n[8] Jan Philip Wahle, Terry Ruas, Tom ́aˇs Folt`ynek, Norman Meuschke, and Bela Gipp. Identi- fying machine-paraphrased plagiarism. In International Conference on Information, pages 393–413. Springer, 2022.\n\n[9] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexan- der Ratner. Wrench: A comprehensive benchmark for weak supervision. arXiv preprint arXiv:2109.11377, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "the proposed methods are simple but useful, outperforming various baselines. The experimental settings are reasonable.\n\nThey test various models and datasets to validate their methods under different decoding schemes such as top-p, top-k, or temperature.\n\nthe overall writing and demonstrations are clear and easy to follow.\n", "reasons_to_reject": "There is a lack of experiments to show the robustness of their methods, for example, how would their method perform when the text is rephrased by either human or machine?\n\nIt is unclear whether their method also works for other languages except English since they only test it in English text.\n\nSee more in questions.", "questions_to_authors": "q1: Table 1: all results on Llama 13B seem much worse than on other models. Do you have any idea why this happens?\n\nq2: Line 201: it is unclear what rank means here at first glance. Perhaps give a formal definition?\n\nq3: Line 223: I don't know what r(x) means compared with r(xi|x<i). Does it mean you add all tokens’ ranks in a sequence x?\n\nq4: in the original DetectGPT, they tested the black-box setting with a surrogate model and found it to be somewhat useful. I am curious whether your methods can also achieve this since we do not have access to many closed models.\n\nq5: except for the AUROC score, what is the TPR under certain FPR(like 1%) since this is more close to realistic use?\n", "paper_topic_and_main_contributions": "This paper proposes two novel methods for detecting text generated from LLMs through either Log-Likelihood Log-Rank ratio (LRR) or Normalized perturbed log-rank (NPR). The experiments show their effectiveness over previous baselines on various datasets and models. Since their zero-shot method is very competitive with previous training-based methods, this paper would offer a good solution to the emerging issue of misinformation spread on the internet.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 846: wrong citation?\n"}, "extra_text": {}}, {"review_id": "YK1e9v2hOF", "reviewer": "Reviewer_712P", "summary": "", "questions": "Can you provide more details on the limitations of your proposed methods? For example, are there certain types of machine-generated text that are more difficult to detect than others?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Hope the following rebuttal addresses your concerns. \n\n---\n\n## Question 1.\n\n*There are already a number of applications for detecting AI-generated text,\ne.g., https://copyleaks.com/ai-content-detector, https://writer.com/ai-content-detector/. Even\nOPENAI released a classifier for indicating AI-written text. However, authors don’t seem to\nmake comparisons.*\n\n## Response 1. \n\nWe respectfully disagree with your opinion for the following reasons. \n\nFirst, our methods are zero-shot, so, to make fair comparisons, we should mostly compare with other state-of-the-art zero-shot methods [3]. Note that we have compared with 5 state-of-the-art zero-shot methods in our paper. \n\nSecondly, though there exists a lot of AI-generated text detectors used commercially such as\nhttps://copyleaks.com/ai-content-detector, https://writer.com/ai-content-detector/ as you mentioned, but they\nare not the state-of-the-art methods for detecting machine generated texts and  **they performs poorly**. Our paper is more likely to be questioned if we compare with the methods you mentioned rather than the state-of-the-art zero-shot methods. We didn’t compare openAI classifiers for the same reason. In fact, **OpenAI’s classifier has been closed now because of their low detection accuracy**(See the article here: https://searchengineland.com/openai-ai-classifier-no-longer-available-429912).\n\nThirdly, besides the zero-shot baselines, we have indeed provided comparison with supervised methods in the appendix of our original paper. Although this comparison is unfair to our methods, we still provided it in our paper. \n\nNote that supervised methods need training data while our methods don’t. Moreover, supervised methods tend to overfit and can’t generalize well [3]. Every time when the model updates, or changes to a new domain, supervised methods have to be retrained. Thus, our methods are more prctical since LLM updates frequently, and it is unrealistic to train a new detection model for every LLM and every domain. Meanwhile, our methods don’t attach to any specific model or domain and are more robust and realistic. As a result, we believe our research (also the whole research area of “detecting machine generated texts”) is both valuable and meaningful.\n\n---\n\n## Question 2.\n\n*The authors use the area under the receiver operating characteristic curve (AUROC) as the sole evaluation metric for their methods. While this is a common metric for zero-shot detection, it may not capture all aspects of performance, such as precision and recall.*\n\n## Response 2.\n\nFirstly, it is difficult for us to illustrate multiple metrices without confusing the readers. Note that we already have multiple dense tables and graphs  (with 7 models, 7 baselines, 3 datasets, as well as a lot of ablation and parameter related experiments).\nIf we provide multiple metrices, the reader might wonder which metric should they look at and got lost in all these tables and metrics).\n\nMoreover, since detection rates are heavily dependent on the chosen detection threshold, the AUC-ROC metric\nis more commonly used to measure detector performance , **which considers the range of all possible thresholds**. See [1, 2, 3]. Meanwhile, recall and precision are rarely used in previous work of detecting machine generated texts. We have to set thresholds ourselves if when calculate recall or precision, and the result would heavily depend on the thresholds we set. (Imagine that you can easily manipulate the experimental results by setting a \"not so good\" thresholds...). Thus, precision and recall is not a good evaluation metric in our setting. \n\n\n\n\n\nHowever, since you have mentioned, we provide experiments on GPT2-xl generated texts using TPR under 0.01 FPR [3] for your reference. \nTPR under certain FPR might also be an important metrics [3], we will add the above results in the revised version of our paper to address your concern. \n\nThe results of TPR with 0.01 FPR are showing below:\n\nFor perturbation based methods: DetectGPT and NPR:\n\n|   Dataset          | SquAD|   Xsum |WritingP  |\n|:----------:|:----------------------:|:----------------------:|:----------------------:|\n| DetectGPT |                 98.45 |                 98.73 |                 99.11 |\n| NPR(ours)       |                 **99.31** |                 **99.48** |                 **99.76** |\n\n\nFor non-perturbation based methods: \n|   Dataset          | SquAD|   Xsum |WritingP  |\n|:----------:|:----------------------:|:----------------------:|:----------------------:|\n| rank      |                 83.46 |                 79.79 |                 87.62 |\n| $\\log p$  |                 90.72 |                 89.16 |                 96.71 |\n| Log rank  |                 94.33 |                 91.75 |                 98.02 |\n| entropy   |                 57.97 |                 56.78 |                 36.45 |\n| LRR(ours)       |                 **97.42** |                 **93.47** |                 **98.34** |\n\nIt can be seen from the above result that the TPR  under FPT fixed as 0.01 is consistent with our previous used AUC metric. Our methods still achieve the state of the art performance among the baseline. Moreover, we will add this metric in our codes for reproducibility.\n\n---\n\n## Question 3.\n*The experiments in this paper only test the proposed methods on datasets in English. It is unclear how well these methods would perform on other languages.*\n\n\n## Response 3. \n\n\nWe only use English texts because most of the LLMs are not multi-lingual. For example, the 7 LLMs we experimented with can only generate English texts. There aren't many multilingual LLMs that are large enough to generate indistinguishable texts in other languages.\n\nHowever, to address the reviewer’s concern, we add additional experiments with German texts from WMT16 dataset as human written texts, and apply m-GPT to generating the machine generated counterparts. For baseline methods DetectGPT and NPR, which requires multi-lingual perturbation model, we apply mT5-xl for perturbation. The rest of the settings are the same as for English texts.\n\nThe result on German can be seen in the Tables below. \n\nFor perturbation based methods(DetectGPT and NPR):\n\n| name      |   AUC |\n|:----------:|:------:|\n| DetectGPT | 96.39 |\n| NPR(ours)       | **98.46** |\n\nFor perturbation-free methods:\n\n| name      |   AUC |\n|:----------:|:------:|\n| rank      | 83.81 |\n| $\\log p$  | 81.67 |\n| Log rank  | 87.47 |\n| entropy   | 58.61 |\n| LRR(ours)       | **95.59** |\n\nIt can been seen from above table that our methods still hold advantages over other baselines. Actually, this result is intuitive because the statistics used in our methods are agnostic to the language or the domain. As long as the LLM has the ability to generate text in a specific language, the statistics given by the LLM would reveal whether the text is generated by the model or not. We will add this experiments in camera ready version and release the code for this muli-lingual experiments as well. \n\n---\n\n\n## Question 4.\n\n*Can you provide more details on the limitations of your proposed methods? For example, are there certain types of machine-generated text that are more difficult to detect than others*\n\n## Response 4.\n\nWe are not sure if there are certain types of machine generated texts that are more difficult to detect. with our methods. Unlike supervised methods, which don't work well on out-of-domain data,  our methods don’t require training so it generally works well for all types of data.\n\nHowever, according to our experiments, we found that most of the detection methods (both the baselines and our methods) have a slightly performance drop on Llama-13b generated data, which is probably because Llama-13b is more aligned with human generated texts and thus is harder to detect. As a result, we think that the performance depends on the LLMs, while other methods might depend on both the LLMs and the types of data.\n\n---\n\n# References\n\n[1] Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgtbench:\nBenchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822, 2023.\n\n[2] Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Para-\nphrasing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv\npreprint arXiv:2303.13408, 2023.\n\n[3] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea\nFinn. Detectgpt: Zero-shot machine-generated text detection using probability curvature.\narXiv preprint arXiv:2301.11305, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper introduces two novel approaches, DetectLLM-LRR and DetectLLM-NPR, for detecting machine-generated text. These methods leverage log rank information and outperform baselines in terms of accuracy and efficiency.\n2. The authors provide insights on the efficiency-performance trade-off and show that their methods are more practical for real-world use.", "reasons_to_reject": "1. There are already a number of applications for detecting AI-generated text, e.g., https://copyleaks.com/ai-content-detector, https://writer.com/ai-content-detector/. Even OPENAI released a classifier for indicating AI-written text. However, authors don't seem to make comparisons. \n2.The authors use the area under the receiver operating characteristic curve (AUROC) as the sole evaluation metric for their methods. While this is a common metric for zero-shot detection, it may not capture all aspects of performance, such as precision and recall.\n3. The experiments in this paper only test the proposed methods on datasets in English. It is unclear how well these methods would perform on other languages.\n", "questions_to_authors": "Can you provide more details on the limitations of your proposed methods? For example, are there certain types of machine-generated text that are more difficult to detect than others?", "paper_topic_and_main_contributions": "This paper introduces two zero-shot methods for detecting machine-generated text using log rank information. The proposed methods can prevent malicious use such as plagiarism, misinformation, and propaganda. The paper discusses the efficiency-performance trade-off based on users' preference and provides intuition for using the methods effectively. The experiments show that the proposed methods improve over the baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Strengths**:

1. Critical problem of detecting whether text is machine generated or human generated.

2. Two novel approaches, DetectLLM-LRR and DetectLLM-NPR. Comprehensive experimental evaluations on three datasets and seven language models. Expts with different decoding schemes such as top-p, top-k, and temperature.

3. The proposed DetectLLM-NPR requires fewer perturbations than previous work while achieving similar performance levels.


**Weaknesses**:

1. Anonymous code not shared.

2. It would be interesting to look at certain types of machine-generated text (e.g., text in healthcare or law) that could be more difficult to detect than others.

**Suggestions**:

1. Thanks for putting up results for German. However, since the majority of the evaluation is for English only, limitations should clearly state that the work is for English only.

2. It would be interesting to look at certain types of machine-generated text (e.g., text in healthcare or law) that could be more difficult to detect than others. 

3. Please incorporate rebuttal content into the main text.
Decision: Accept-Findings