Training-free LLM-generated Text Detection by Mining Token Probability Sequences
Authors: Yihuai Xu
Yongwei Wang
Yifei Bi
Huangsen Cao
Zhouhan Lin
Yu Zhao
Fei Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Fake text detection
training-free
detection
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a novel and effective training-free method for detecting LLM-generated text by mining token probability sequences.

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in generating high-quality texts across diverse domains. However, the potential misuse of LLMs has raised significant concerns, underscoring the urgent need for reliable detection of LLM-generated texts. Conventional training-based detectors often struggle with generalization, particularly in cross-domain and cross-model scenarios. In contrast, training-free methods, which focus on inherent discrepancies through carefully designed statistical features, offer improved generalization and interpretability. Despite this, existing training-free detection methods typically rely on global text sequence statistics, neglecting the modeling of local discriminative features, thereby limiting their detection efficacy. In this work, we introduce a novel training-free detector, termed \textbf{Lastde}\footnote{The code and data are released at \url{https://github.com/TrustMedia-zju/Lastde_Detector}.} that synergizes local and global statistics for enhanced detection. For the first time, we introduce time series analysis to LLM-generated text detection, capturing the temporal dynamics of token probability sequences. By integrating these local statistics with global ones, our detector reveals significant disparities between human and LLM-generated texts. We also propose an efficient alternative, \textbf{Lastde++} to enable real-time detection. Extensive experiments on six datasets involving cross-domain, cross-model, and cross-lingual detection scenarios, under both white-box and black-box settings, demonstrated that our method consistently achieves state-of-the-art performance. Furthermore, our approach exhibits greater robustness against paraphrasing attacks compared to existing baseline methods.

Published as a conference paper at ICLR 2025
TRAINING-FREE LLM-GENERATED TEXT DETECTION BY MINING TOKEN PROBABILITY SEQUENCES
Yihuai Xu Yongwei Wang'; Yifei Bit.? Huangsen Cao', Zhouhan Lin", Yu Zhao', Fei Wu!
1 ZZhejiang University "Gerggia Institute of Technology
Shhnnhai Jiao Tong University *Zhejiang Gongshang University
(yongwei. wang, huangsen_cao, yuzhao, wufei (ezjjizzzueeuu
com 1in. zhouhan@gmail. com ybi48@gatech.ed
ABSTRACT
Large language models (LLMs) have demonstrated remarkable capabilities in generating high-quality texts across diverse domains. However, the potential misuse of LLMs has raised significant concerns, underscoring the urgent need for reliable detection of LLM-generee often struggle with generalization, particularly in cross-domain and cross-model scenarios. In contrast, training-free methods, which focus inherent discrepancies through carefully designed statistical features, offer improved generalization and interpretability. Despite this, existing training-firee ffree detection methods typically rely on global text sequence statistics, neglecting the modeling of local discriminative features, thereby limiting their detection efficacy. In this work, introduce novel training-free free detector, termed Lastde that synergizes local and global statistics for enhanced detection. For the first time, introduce time series analysis LLM- generated text detection, capturing the temporal dynamics of token probability sequences. By integrating these local statistics with global ones, our detector reveals significant disparitics between human and LLM- L-M-eenerated texts. We also propose an efficient alternative, Lastde++ enable real-time detection. Extensive experiments on six datasets involving cross-domain, cross- mooel, and cross-lingual detection scenarios, under both white-box and black-box settings, demonstrated that our method consistently achieves state-of-the-art performance. Furthermore, our approach exhibits greater robustness against paraphrasing attacks compared existing baseline methods.
texts. Conventional training- based detectors
INTRODUCTION
Recent ad advancements large language models (LLMs), such as GPT-4 (OpenAl 2024b) and Gemma (Team et al. 2024). have significantly enhanced the text generation capabilities of machines. These models produce texts that are virtually indistinguishable from those written by humans, enabling broad applications across fields like journalism (Quinonez & Meij,, 2024), education (M Alshater 2022, Xiao et al. 2023). and medicine (Thirunavukarasu et al. 2023). However, the increasing sophistication of LLMs has also raised serious concerns about their potential misuse. Examples include the fabrication and dissemination of fake news (Opdahl et al. 2023: Fang et al. 2024). academic dishonesty in scientific writing (Else) 2023 Curri [003 Liang et al.
2024). and the risk
of model collapse when trained on L.M--genrraed data (Shumailov et al. 2004
Wenger 2024).
Prior arts in L.L--generate text detection be broadly categorized training-based and training- free methods. Training- methods extract discriminative features from the texts and then input them into binary classifiers. These detectors require training on labeled datasets. Typical examples include RoBERTa-based fine-tuning (Guo et al. 2023), OpenAl Text Classifier (OpenAl 2023), and GPTZero (Tia 2023). In contrast, training- free methods utilize global statistical features of given texts such as lioood and rank (Solaiman et al. 2019). which can be computed from LLMs during
*Correspondigg author.
'The code and data are released at
betector

---

Published as a conference paper at ICLR 2025
30
Human
LLM
25
Log Probabil lit
Frequency
15
-10
10
-12
5
LLM
-14
Token
Score
(a) An illustration example comparing the TPS fluctuations (with OPT-2.7) between human- written and LLM- -ege-erttd texts.
(b) The proposed Lastde score distribution of human and LL-M-geneated texts from the WritingPrompts dataset, using the OPT-2.7.
Figure I:
Comparison of TPS fluctuations and Lastde score distributions between human-written written texts and LLM-generated texts, using the first 30 tokens of human texts prefix to continue writing.
the text inference. Typical training-free methods such as DetectGPT Miitheell et al. 2023 and DNA-GPT (Yang et al. 2024) achieve detection by comparing or scoring these statistical features.
Training- Traseing-based methods require significant time and effort in figh-quality data collection, model training, etc. Also, they often face generalization issues, e.g. overfitting to in-domain data, particularly when detecting cross-domain texts Pu et al. (2023); Zhu et al. (2023). In contrast, training-free methods, which rely scoring the analysis results of data statisties (often global statistics) or the data distribution, are more adept at handling cross-domain texts (Yu et al. (024). However, existing training-free methods either incur high time and computational costs or perform poorly in scenarios involving cross- modell detection and paraphrasing attacks. Additionally, most of these methods only evaluate the global statistical features (e.g. likelihood) of the text without harnessing the local statistical patterns (e.g. temporal dynamics) from token segments, significantly limiting the development and broader application of training-free methods.
To bridge the gap, this work aims to leverage local and global statistics for effective detection by mining discriminative patterns from the token probability sequence (TPS). Specifically, we first provide a novel perspective that views the TPS as time series and harnesses time series analysis to extract local statistics. As illustrated in Figure [(a) the TPS of human-written text fluctuates more abruptly than LLM-generater ones, showcasing distinct temporal dynamics. To quantify the dynamical complexity, we exploit the diversity entropy (Wang et al., 2020) that measures similarities among neighboring sliding-window segments of TPS followed by conversion to discrete probability states for entropy calculation. We then aggregate the DEs (diversity entropy) from different time scales to establish our local statistics. Finally, we integrate the local statistical features with likelihood, a typical measure of global text features, to develop a novel detection method: Lastde (Likelihood and sequential token probability diversity entropy) and its variant Lastde++. with the latter being an enhanced version featuring fast-sampling for more efficient detection. As shown in Fig. ((b) our method shows clear distinguishability boundary between human and LLM--generan texts.
Extensive experiments across different datasets, models, and scenarios demonstrate that Lastde++ has a strong ability to distinguish between human and -LM--eeerrtee text. Notably, in the black-box setting, Lastde, relying on single score, outperformed all existing single-score methods, even matching the detection performance requires extensive sampling, thus our method establishes a new powerful benchmark.
Destt-ettt (Bao et al., (202)), a previous work that
Our main contributions can be summarized as follows:
1) We propose novel and effective method for LLM-generated text detection, termed Lastde (and Lastde++), by mining token probability sequences from a time series analysis viewpoint.
2) We propose leverage diversity entropy capture the disparate temporal dynamics of token probability sequence, extract the local statistics, and integrate them with the global statistics to achieve robust detector.
3) Our method achieves state-of-the of-the-art detection performance, surpassing advanced training free baseline methods including DetectGPT, DNA-GPT, and mast-DetectGPT with comparable or lower computational costs. Besides, it demonstrates superior robustness handling complex detection scenarios, e.g. cross-domain, cross- cross-lingual tasks, and paraphrasing attacks.
2

---

Published as a conference paper at ICLR 2025
Global Statisties
TPS
Log- Likelihood
Candidire Text
TP2
Muli-ca
The
Open-
Informnce
TP4
120
Ag-MDD
source
TP5
window
Model
Prony
TP6
(-)
TP7
D6
Local Statistics
Figure 2: Overview of the Lastde detection framework. The example shows how 7-token candidate text is fully detected with setting 3, 10, r' 3. First, a proxy model converts the text into a token (log) probability sequence (TPS). Then, both global and local statistics of the TPS are mined in parallel. The global statistics is straightforward. For local statistics, the TPS is mapped into new sequences by a Multi-scale Processor (with
3). Each scale factor т E {1, 2,3) represents the arithmetic mean of adjacent elements from the original TPS. For each new sequence, a sliding window with a stride of I and width of 3 (s = 3) moves from left to right, forming a matrix by vertically stacking window segments' elements. The cosine similarity between adjacent rows (segments) is then calculated to generate a similarity sequence. We calculate the histogram of 10 (E = 10, [-1, 1] equally divided intervals based on the similarity sequence and derive the Diversity Entropy (DE) of the current sequence. Finally, we divide Log- Ligeeiikdddood by aggregating all DEs (Agg-MDE) derive the detection score, making decision based appropriate threshold.
RELATED WORK
Detecting LLM- -eMerreedt text has been a prominent area of research since the early days of large language models (Jawahar al., 2020 Bakhtin et al. 2019: Crothers et al. (023). Current studies predominantly focus on training- based and training- free detection methods.
free
Most training- based methods (Bhattacharjee et al. 2023 Li et al. 2023 Tian et al. 2023) leverage the semantic features of text (such as token embeddings) to train or fine-tune classification model under the premise of labels for successful detection. Meanwhile, some approaches (Verma et al.) 2023 Shi et al. 2024 Wang et al. 2023) incorporate probability information as part of the training features, which has also proven effective in detecting LLM-generated text in various scenarios.
However, research has shown that many existing training-basee methods struggle with overfitting or generalizing out-of-distiibution data (Uchendu et al.) 2020 Chakraborty al., 2023). As result, training-free methods have gained increasing attention aiming at identifying LLM--gerrtted text across diverse domains and source models. Specifically, these methods focus on the probabilistic features of the text, scoring the text by constructing appropriate statistics, and ultimately making decisions based on determined threshold. Representative methods include Likelihood, Rank Solaiman et al., 2019). GLTR (Gehrmann et al. 2019), and DetectLRR (Su al. 2023). DetectGPT Mitchell al. (2023) pioneered the detection paradigm of contrasting perturbed texts with original text, subsequently leading to the development of DetectNPR (Su et al. (023) and DNA-GPT (Yang al. 2024). Yet we must acknowledge that these methods cannot achieve real-time or large-scale detection due to their excessive time consumption. Fast-DetectGPT (Bao al. 2024) leverages fast-sampling technique to enhance the detection efficiency DetectGPT by 340 times, thereby broadening the application prospects of training-fres methods. However, its black-box detection performance still exhibits substantial potential for enhancement. More innovative detectors are
continuously being conceived, please refer to (Mao et al.] 2024 Hans al. 2024).
We are confident that the constraints on the performance of existing training free detectors in black- box detection arise from their failure to fully exploit excavate the deeper layers of information
3

---

Published as a conference paper at ICLR 2025
inherent in token probability sequence, particularly the local geometric information. To address this issue, we devised a detection statistic termed Lastde, which is employed to evaluate both human- written and LLM-generated texts. Our approach successfully establishes the connection between token probability sequence and traditional time series, achieving state-of-the-art performance among training-free methods and providing novel perspective for the detection of L..M-generated texts.
METHODOLOGY
In this section, we first formulate the diversity entropy of token probability sequence (TPS) to quantitatively measure its local dynamics. We then integrate local statistic features with the global ones (likelihood) to develop our Lastde detector, and further develop a fast-sampling alternative called Lastde++ for more efficient detection.
3.1 ANALYZING TPS WITH DIVERSITY ENTROPY
Diversity Entropy (DE) (Wagg et al. 2020) is an entropy-based method for measuring the dynamic complexity time series, originally applied to feature extraction areas such as fault diagnosis, ECG, and signal processing. It is process of first increasing the dimensionality and then reducing the dimensionality detecting LLM generated text (black-box) boxk).
mie-iimenioonal sequence. We restate and extend DE to make suitable to
Given candidate text t and proxy model Mg, with the sliding window size s E N+, the precision of the interval E E NT, and the number of scales 7' e NT Input into Mo for inference, obtain its (log) TPS
(tt) (log (t((1 att
(1)
where t, is the i-th token t.
Multiscale log-probability sequence. Apply multiscale transformation to T(t). Specifically, for (7)
= 1, T', let (t) (f) be the new TPS corresponding to the r-th scale transformation, where
H+--1
log ())
Σ logpo(tiltea); Isisn-rH
Note that, when (t) [loggg" ()) (logpo(t) L(t), which is the original TPS of Equation [ Different 7 measures its dynamic characteristics at different scales, which is essentially enhancement of the original data.
Sliding-segmentatiion sequence. Apply a sliding window of size s and a step size of to segment and rearrange (t), 1,2...., in sequence, resulting in the following:
log Ps (1)
log Pg
(T) (2)
log P (s)
log Ps (7)(2)
log Pg (3)
(T)
(T) log Pg 51111 +1))
(3)
2) Denote the matrix in Equation as
log (T) (n
log Pg (n 7
(T)
log logr (7) (n (
i (()) - ((T) (5) (s). (6) (1)))))))))))))))))))))
(4)
Po where (s) -000((11g))) (7) (1), log
..., log (7) (1 (i+$ - 1)) is the log- log-probability segment within window i the 7-th scale, which is called an s-probability segment.
logro (T) + 1)....
Segment similarity. For (s),
1,2..... r' Equation compute the cosine sim- ilarity between adjacent s-probability segments, yielding the similarity sequence (()
(((T) (fT)) a(r) d where where
(T) CosineSimilarity (T) (s). (2) (s)), 1,2, .,, 7-1+1
4

---

Published as a conference paper at ICLR 2025
The geometric meaning of cosine similarity is the cosine of the angle between two vectors in space, with a range of values from [-1, 1]. Therefore, intuitively. the closer the similarity value of two adjacent -probability segments is to I, the smaller the deviation in direction, and the smaller the fluctuation in the local log-probability. Conversely, the larger the fluctuation in the local log- bog-rrbbability.
Probability state sequence. Divide the interval [-1, into mutually exclusive and equally spaced sub-intervals, denoted as = ( Ie), each which is called a state. Therefore, we can easily obtain the statistical histogram (s) (ef²) C2 ( of (s) I, where c!T) the number of elements in (s) that fall into the state 4. Furthermore, we can calculate the probability state sequence
(s) (P(T), ... P P!
where p!T)
Obviously, the sum of the sequence Equation 6 is equal to 1.
Multiscale diversity entropy. According to the DE formula of Equation g calculate the diversity entropy of the probability state sequence (s)) = 1,2, T' for each scale, and store them together to obtain the multiscale diversity entropy (MDE) sequence for the text t, denoted as
MDE(t,, 8,E,T)) (DE(s,e, 1), DE(s,,,r')),
where
DE(s,s,7) Ine
pl In P!
pp > 0.
(8)
According to (Shannon 1948). the entropy value ranges from [0, 1]. and therefore, the DE value also ranges from [0, TJ. A DE value closer to 0 indicates that the TPS at the current scale has less fluctuation, while value further from 0 indicates greater fluctuation.
3.2 LASTDE AND LASTDE+
2023) we first aggregated the MDE sequence of the text. Subsequently, we combined this with the Log-L Log-iieeld of the text and defined the following Log-Likelihood and sequential token probability multiscale diversity entropy (Lastde) score
Lastde score. Inspired by DetectL.LM (Su al.
1 E log potgo(t))
Lastde (t, 0) = Agg (DE(s, I), ... DE(s,,,T')))
where Agg IT' + IR is aggregation operator function that summarizes the sequence's elements in a particular way, the rest of the notations are consistent with the previous definitions. Obviously, the numerator in Equation statistical feature of TPS. Additionally, we refer to the result the denominator in Equation | as the Agg-MDE value, which reflects the local statistical feature of the TPS. Unless explicitly stated, use the standard deviation function (Std) as the aggregation function (Agg), additional aggregation functions and their detection results are located Appendix D Furthermore, for Lastde, the hyperparameters are set to default values of 3, = 10 x n, T' 5, where n is the number of tokens in the text. summary, the detection process of Lastde can be outlined three steps:
| is the Lig-Likelioood value, which corresponds to the global
1. Inference: Retrieve the TPS of candidate text using specific open-source proxy model.
2. Computing statistics: Simultaneously compute the Log-Likellhhood and Agg-MDE statistics of the TPS.
3. Scoring: Divide the two statistics and make decision based on the result.
A simple example illustrating the framework of our proposed algorithm be seen in Figure 2
We observed that the Lastde score distributions for the two types of texts are significantly different: the Lastde score for human-writted texts is generally lower, while that for LLM-generated texts is higher, as shown in Figure IT] The ablation experiment Appendix alsso proves the rationality of applying the MDE algorithm TPS. Therefore, to improve the detection performance of the Lastde method, we incorporate the fast-sampling technique (Bao et al. 2024) to modify the Lastde score, providing the final detection score of the text from the perspective of sampling discrepancy.
5

---

Published as a conference paper at ICLR 2025
Lastde++. Specifically, for given text available scoring model Mg. and conditionally independent sampling model Ma the sampling Lastde discrepancy defined as
0,0 Lastde(t, 1
(10)
where Lastde(t,, 0) can be seen Equation 9
j = (g((e)) [Lastde(t,
0)] and σ² = Eg (Lastdeeft 0) p
(11)
where j and x² are the expectation and variance of the Lastde scores of the sampling samples i from Mg. respectively. Our default sampling number is 100, which 1/100 of Fast-DetectGPT. In fact, when 0 = 0, the sampling and scoring steps can be combined, and this case, the detection task can be completed by only calling the model once. For Lastdc++, the default settings are 8 = 4, = 8 x n, 15.
The discriminative power of Lastde++ in distinguishing human- written text and LLM-generated text is usually stronger than that of Lastde. As shown in Figure 12 the score distribution after sampling normalization is more favorable for distinguishing the two types of text compared to the Lastde score distribution. Appendix presents a more detailed analysis of Lastde++.
4 EXPERIMENTS
4.1 SETTINGS
Datasets. The experiments conducted involved 6 distinct datasets, covering a range of languages and topics. Adhering to the setups of Past-DetectGii and DNA-GPT, we report the main detection results 4 datasets: XSum (Narayan et al. 2018 (BBC News documents), SQuAD (Rajpurkar itt al., 2016; 2018 (Wikipedia-based Q&A context), WritingPrompis (Fan et al. 2018) (for story generation).am Reddit ELI5 (Fan et al. 2019) (Q&A data restricted to the topics of biology, physics, chemistry, economics, law, and technique). To investigate the robustness of various detectors across different languages, first conducted detection on WMT/6 (Bojar al. 2016). a common dataset for English and German text translations. Subsequently, we gathered data on food, history, and economics from the popular Chinese Q&A platform Zhihu, extending detection to these Chinese example, we
datasets. Each dataset contained 150 human- wurttnn examples. For each used the first 30 tokens as a prompt to input into the source model and generate a LLM -eneereeed continuation. More details on the dataset and prompt engineering are provided in the Appendix B.1
Source models& proxy model. In order to comprehensively evaluate all the detectors, we utilized up to 18 source models for testing, including 15 famous open-source models with parameters ranging from 1.3B to 13B, as well as 3 of the latest and widely used closed-source models: GPT-4-Turbo
OOeeAA ) GPT-4o 4o (OpenAl 2024a), and Claude- 3-haiku (Antropic 2024). Most of the open-source models were consistent with Fast-DetectGPT. Additionally, apart from mGPT (Shliazhko et al. (022), Qwenl.5 (Bai et al. 2023), and Yi-1.5 (AI et al., 2024) being used for cross-lingual (German and Chinese) robustness detection, the residual 12 open-ssurce models were employed to report the main detection results. Unless explicitly stated, all methods used GPT-J (Wang) & Komatsuzaki] (02T) as the sole proxy model to execute the entire black-box detection process, encompassing rewriting (if necessary), sampling (if necessary), and scoring. Detailed information about all the source models can be found in the Appendix B.2
Baselines & metric. We selected representative training- free detection methods as baselines and categorized them into two main groups: Sample-based and Distribution-huse Both rely on statistics derived from the Logits tensor. The key difference is that the former directly utilizes the statistical values obtained as the final score for the text, while the latter generates a large number of contrast samples through specific methods (e-g., perturbation, rewriting, fast sampling) and then evaluates the overall distribution of these samples. Specifically, the first group includes Log-L Likeliikl (Solaiman 2019), LogRank (Solaiman et al.) 2019). Entropy (Gehrmann et al. 2019 Ippolito et al. and DetectLRR (Su et al. 2023), The second group consists of DetectGPT Miithelll et al. DetectNPR (Su et al. 2023), ) DNA-GPT (Yang et al. (224)), and Fast- DetectGeTTT (Bao et al. Further details the baselines can be found in the Appendix B.3 According prior studies
et al.
2019
2023
2024h.
(Bao et al.) 2024). the most commonly used evaluation metric AUROC (area under the receiver
operating characteristic curve). Therefore, conform to this convention.

---

Published as a conference paper at ICLR 2025
Table I: Detection results for text generated by 12 source models under the white-box scenario. The AUROC values reported for each model are averaged across three datasets: XSum, SQuAD, and WritingPrompie More detailed detection results are available in Ta Appendix C.1 All methods use the source model for scoring. The "(Diff)" rows indicate the absolute improvement imereeemmpooeeee
in AUROC of Lastde over DetectLRR for the first group of methods, and of Lastde++ over Fast-
DetectGPT for the second group. "value" denotes the second-best AUROC. Implementation details for all baselines can be found in the Appendix B.3
MechosssModets GPT2
Nee-27
OPE22
GPEJ
Linma-13
Llaman-13
Linto3-4
OFE-13
BLOOSE-7.1
Falcun-7
Genuma-7
Fla2 2.7
AAr.
Sumppie-hevee Medhads
Likeiiooo
91.65
$9.40
$8.08
$4.95
63.56
65. 36
98.35
$4.45
85.00
76.78
70.14
50.67
R2.54
LopRank
94.31
02.57
50.99
$8.58
68.57
70.27
99.04
67.74
92.42
B1.32
74.81
92.13
96.12
hatropy
50.45
61.00
23.30
02:50
66.47
48.09
51.67
06.67
09.13
11.35
90:30
1.74
3,02
7.51
DeseiGP
93.45
89.28
89.55
90.68
DeseiNPR
95.77
94.05
DNA-CIPT
89.92
$6.30
86.70
K0.31
67.38
64. 46
98.07
82.51
16.74
74.04
63.63
68.00
90:45
CPP
09.57
00.49
08.78
08.05
93.45
03.34 33.98
99.91
98.00
99.53
07.74
06.90
98.10
97.82
Lastdere
99.76
99.87
99.46
99.52
96.58
96.07
19.52
98.77
99.84
98.76
98.30
98.76
98.85
(Ou))
0.11
0.38
0.08
0.57
3.13
3.335
-0009
0.70
01331
1,00
1.50
0.56
1.003
Table Detection results for text generated by 11 open-source models and I closed-source model (GPT-4-Turbbo) under the black-box scenario. The AUROC values reported for each model averaged across three datasets: XSum, WritingPromptss and Reddit. The meanings of "(Diff)" )" and "value" are the same in Table [ More detailed detection results and results for the remaining closed-source models are provided in the Appendix C.2 and
MethodssMModeess GPT2
Nee-2.7
OPF-27
Limm-133
Llama2-13
LiamaT-8
OFF-13
BL00M-7.
Falcon-7
Gemman7
Phi3-22
GPI-4-Turbs
Arg:
Sumppleshosee Mathedds
Llulihood
50
LogRank
70.35
70.25
73.01
07.51
73.59
79.34
74.54
Enzopy
61.46
59.65
54.55
40.1
45.18
14.43
53.00
00.54
50.55
46.01
46.58
33.00
46.13
DeteciL RR
25.30
23.19
11.25
75.41
97:35
80.22
20.47
70.52
13.07
$1.7H
73.85
10.55
Laside
89.17
98.34
39.70
80.73
79.90
59.07
90.01
16.96
$1.36
7966
88.33
81.35
10.30
2:30
2.32
9.37
7.48
6.35
Diinitationn -eareel---------eeee Metheds
66:12
$2.90
01.35
12.76
81.75
70.94
DetoctN/it
69.07
68.41
75.05
67.81
70.60
06. 75
75.05
63.00
70.42
55.72
94.08
79.94
12.75
DNA GPT
64.15
02.63
63.64
00.77
66.71
99.40
65:75
02.01
65.08
62.59
73.02
70.75
67.97
Fas DerectPPT
89:32
39.75
86.53
77.58
72.62
$6.16
81.55
31.42
$1.09
46.67
88.15
13.60
Lasidese
83.00
09:00
92.87
$1.47
7.42
4.2 MAIN RESULTS
We introduce Lastde as new sample-based method to compare with the first group and Lastde++ as its enhanced version to compare with the second group, following the default settings described in Sec.
In the white-box scenario, we use the 12 open-source models mentioned earlier to generate L.n-ggeeetted text and perform detection concurrently. In the black-box scenario, since GPT-J has already been used the proxy model, it is not suitable to it the source model again, deploy GPT-4-Turbo to fill that gap.
Overall results. Table 1 and present the detection results under white-box and black-box scenarios. Overall, Lastde and Lastde++, which require only 100 samples, consistently outperform DetectLRR and Fast-DetectGPT in both scenarios. Specifically, in the white-box scenario, Lastde achieves an average AUROC of 55.99 across the 12 source models, significantly surpassing methods of the same type and approaching the performance levels of Fast-DetectGPT. In the black-box scenario, Lastde not only outperforms Fast-DetectGPT in detecting most open-source models but its enhanced version, Lastde++, further improves the average AUROC by 5.80% In summary, Lastde++ significantly improves detection performance for open-source models in black-box scenarios compared to Fast- DetectGPT and also achieves the best performance on dosed-source models, making a superior training-free -fee detection method. Given that black-box scenarios are more common in practice and that open-source models still have a wide range of applications similar to closed-source models due to factors such API call costs, Lastde++ is poised to become a more accurate cross-source LLM-generated texts detector and powerful benchmark. For detailed detection results, please refer to the Table 6 the Appendix C.I and the Table 7 in the C.2
Hyperparameter analysis. In Figure we explored the impact of Lastde's three hyperparameters (s, e, r') detection results across four specific datasets. Firstly, larger window size results in
7

---

Published as a conference paper at ICLR 2025
AUROC
The cok MUROC
The AUROC
92-
92.5
90
90.0
90-
AUROC
68
87..5
AUROC
83
84
02.5
80.0
2
3
1n
4n
6n
Bn
10n
10
15 Number of scaleste)
20
25
WindwStoett
Mutiplelki of Texttokknn lengthir)
Rede
Figure 3: Sensitivity analysis results for three types of hyperparameters in Lastde++. The legend denotes (Dataset, Source Model). Through preliminary analysis and referring to the parameter tuning experiments of the original paper of the MDE algorithm, we explore the following ranges: s E (2,3, 4, 5, 6}. E (n, 4n, 6n, 8n, 10n). r' E {5, 10, 15, 20, 25). In each experiment, we adjust only one type of hyperparameters while keeping the other two types fixed at their default settings.
fewer s-robability segments, while if the size is too small, the reflection of the variations will be too monotonous. Both situations are not conducive capturing the authentic information of the sequence. This is confirmed by Figure 3 (left), where a size of 3 or optimally balances detection performance across the four datasets. Secondly, a larger e results in finer granularity when partitioning the [-1, range, which is generally ad advantageous. To adaptively vary it is set multiple (k) of the number of tokens (n)). As shown in Figure (middle), detection performance improves with increasing on most datasets, converging at 8x 10x, except for the Reddit (GPT-4-Turoo) dataset. Finally, increasing the scale factor from 5 to 10 improves detection across all four datasets. However, beyond 10, performance continues improve on the Reddit (GPT-4-Turbo) dataset but plateaus slightly declines on others, likely because a huge scale factor may not align well with the token count.
4.3 ROBUSTNESS ANALYSIS
Samples number. According to findings from DetectLLM (Su et al. 2023), increasing the number of perturbed samples enhances detection accuracy. Our experiments on contrast samples for all distribution-hased methods reveal that more contrast samples improve detection performance, with DetectGPT shows the most significant gains (see Figure 4p. Notably, Lastde++ achieves superior performance with just 10 samples, surpassing Fast-DetectGPT with 100 samples and outperforming other methods by considerable margin. This underscores Lastde++'s strong competitiveness even with limited number of contrast samples.
Average AUROC on XSum- SSum-SQuAD-Wiiinnhhoo
XSumIGPT4TTurbb)
98.5
AUROC (Nun- Bast Sampling)
85
75.0
96.0
AUROC FaR Sampling
80-
72.5
97.5
RUC
70.0
70-
67.5
96.5
65.0
55
62.5
10
20
50
100
160
The number of contrast samples
Response Length (wardss
Figure 4: Distribution- methods' robustness to contrast sample numbers. The right y-axis represents the AUROC of fast sampling methods, while the left y-axis represents non-fast sampling methods (see Appendix F.T.. Perform white-box detection using Gemma-7 as the source model.
Figure 5: Detection results of 6 detection meth- ods on 5 response lengths. Specifically, the 3 hyperparameters of Lastde++ and Lastde were set to = 3. = 1 n, T' = 10 to adapt to shorter text. The settings for the other methods were kept at their default settings.
Response lengths. Prior studies (Verma al. 2023 Mao al. 2024 have demonstrated that shorter texts are generally less conducive to detection. Therefore, we evaluated detection performance across varying response lengths (number of words), focusing primarily XSum generated by GPT-4-Turbo
8

---

Published as a conference paper at ICLR 2025
due to source response length constraints. We truncated responses {30, 50, 120, 150, 160) words, revealing in Figure | that most detection methods' performance improves with longer responses, consistent with prior research findings. The shorter response length inherently restricts the number of scales (r'), thereby diminishing the capacity of Lastde and Lastde++ to capture local information in the initial stages. When the length exceeds 1 120, Lastde and Lastde++ clearly outperform other comparable methods, underscoring their sustained competitiveness.
Decoding strategies. We evaluated the impact of different decoding strategies on detection, including top-p, top-k, and temperature sampling, each controlling the diversity of generated text from different angles. We conducted comparisons on Xsum, SQuAD, and WritingPrompts. The results presented in Nigure[[indice that across all three strategies, Lastde++ consistently exhibits superior robustness among the three source models, while Lastde significantly outperforms sample-based methods such as DetectI.RK achieving detection performance comparable to Fast-DetectGPP..
Topip 2.95
Top- 40
Temperoture = 0.80
New
Neei2.7
Nee-2.7
Ro
20 T050
29
20"
Lagitamd
DENNOOFF Lms
Figure 6: The impact of the decoding strategy. The average AUROC values across datasets are provided for each Detection method (detailed results can be found in Table [5]in (ppendix(F.3) The source models were GPT-2, Neo-2.7, and OPT-2.7, with the decoding strategy hyperparameters set to top-p 0.96, top-k 40, and temperature=0.8. All experiments were conducted in a black-box setting, with other model implementation details kept at their default settings.
Paraphrasing attack. Previous studies (Krishna et al. 2024 Sadasivan et al. 2023) show that paraphrasing attacks can effectively evade detection by methods like watermarking, GPTZero, DetectGPT, and OpenAl's text classifier. Following Bao (Bao et al. 2024), we used T5- T5-Paraphrasef to perform paraphrasing attacks texts generated by source models. As shown in Table 33 both methods exhibited performance drops in detecting paraphrased texts, in both black-box and white- box scenarios. However, Lastde++ consistently maintained significant advantage, with smaller performance declines compared to Fast-DetectGPTT across most scenarios and datasets. Notably, in detecting paraphrased Reddit (GPT-4-Turbo), Lastde++ surpassed Dest-DetectGPT, indicating superior robustness (see Appendix F2
Table 3: AUROC for Dett-eetetTGPT and Lastde++ before and after paraphrasing attacks, with average values across three source models in both white-bor (Llama-13 OPT-13, GPT-J) and black- box (Llama-1s 13, OPT-13, GPT-4-Turbo) scenarios, and the decrease in AUROC values indicated in parentheses. Detailed results are shown in the Appendix F.2
Mothedshhattert
XSamm Original)
Parpphaadh)
WritingPrompss Original))
WritingPrompeeaaaggnnnee
ReddittOrgginal Reddit Reddii (Parsphrased)
White-baz Settings
(L
2.70
98.05
Lasidere
97.75
89.27 (1 8.45)
99.57
97.84 1.73)
99.10
97.00 (] 1.10)
Settings
Foo-DrteaCCrr
(1
9.144)
92.74
(1 3,36)
https::/hugginggfacs http:/aaw..............easssss httpa//aaiiiaaaaaaaaaeeeeee Paws

---

Published as a conference paper at ICLR 2025
Non-English scenarios. The latest research (Liang et al. 2023) shows that detectors exhibit bias against non-English writers. Therefore, conducted experiments on datasets in three dif- ferent languages, and the results confirmed that Lastde++ remains effective (see Figure (7). The Lastde++ outperforms in detecting texts across all three languages. Specifically, Lastde++ surpasses Fast-DetectGPT by an average AUROC margin of 4.56% (WMT16- I--Gmman)) and 5.24% (WMT16 16- English) in both scenarios. On our newly intro- duced Chinese dataset, which includes three topics, both methods perform well, but Lastde++ demon- strates slight advantage. Therefore, Lastde++ is better suited for cross-lingual detection scenarios compared to Fast-DetectGPP.
Average AUROC for Both Scenarios
15.1.2224
50.02
Figure 7: Detection performance comparison across English, German, and Chinese texts. The source models in the white-box scenario are mGPT (English and German) and Qwen1.5-7 (Chinese), while the source models in the black- box scenario remain unchanged the proxy models are GPT-J (English and German) and Y (Chinese).
Selection of proxy models. Bao (Bao et al. 2024) discussed the impact of proxy sampling models on detection performance in white-box scenarios. Our experiments further reveal that the choice of proxy model in black-box scenarios significantly influences detection performance. We evaluated five training-Foe free detection methods across five different source and proxy model pairs, illustrated in Figure 8 The results indicate that Lastde matches in most or even surpasses Fast- Past-DetectGP11 pairings, while Lastde++ outperforms Fast- Best-eeteeGGPT in all five pairings. Notably, detection performance was poor across all five methods with the closed-source GPT-4-Turbo and the proxy model Llama3-8, Despite this, our proposed Lastde and Lastde++ consistently outperformed Fast-Detec1GPT and other baseline methods, further emphasizing that the choice of proxy model greatly influences detection performance.
AUROC for different urce and proxy model pairs
01:31
42
80
26.43
60
0.0.7
$9.99
18.2
(1.48
2P
0PT2.2009:31
Geavro POOPT
Figure 8: The performance of 5 zero-shot detection methods under 5 different combinations of source models and proxy models is presented. The X-axis represents the 5 different combinations, with the proxy model in parentheses and the source model outside the parentheses. The source dataset is XSum, and the other settings are kept at their default values.
CONCLUSION
In this work, propose novel training-fres method for detecting LLM LLM-generated text, termed Lastde and Lastde++, by mining token probability sequences (TPS). By observing significant dif- ferences in the temporal dynamics of TPS between human-written and L.LM-generated texts, leveraged diverse entropy from time series analysis to develop local statistical features, integrating them with global statistics to create an effective and robust detector. Our approach demonstrates effi- cacy in both black-box and white-box settings while maintaining comparable or lower computational costs. Furthermore, outperforms existing training-free detectors in complex scenarios, including paraphrasing attacks and cross- cross-lingual lingual tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by Shanghai Pujiang Program (No. 23PJ1412100), Yangtze River Delta Community of Sci-Tech Innovation (Grant No. 2023CSIZN0301, ) and National Natural Science Foundation of China (No. U24A20326, 62441605).
ETHICS STATEMENT
Lastde and Lastde++ are novel LLM-generated text detectors capable of real-time detection, achieving new advancements in complex scenarios such as cross-domain and cross-lingual. However, as we mentioned in the paper, they still produce imperfect detection results when faced with sophisticated challenges such as paraphrasing attacks and the arbitrary replacement of proxy models. Therefore, we strongly recommend that users exercise prudence when interpreting the detection results. The results should be regarded as supplementary references to support, rather than replace, human judgment in critical decision- decision-making processes.
REPRODUCIBILITY STATEMENT
All experimental projects in this paper are reproducible. We provide details on the generation of all datasets and the implementation details of all baselines in Appendix B Additionally, we provide the source addresses for the third-party datasets in Appendix D
REFERENCES
01. AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Tao Yu, Wen Xie. Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 0l.ai, 2024. URL https: /ttp//.wxvvv.o/../...../////./.e abs 2253/2403.2 04652
Aeta Llama model card. 2024. UR..httpss//gtthhb..ommmmeee-
11ama3/
blob/ma n MODEL CARD .d
Antropic.
Introducing the next generation of claude. https://ww.anthroolccccom
3- 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.1660 2023.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Mare' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351, 2019.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of machine- generated text via conditional probability curvature. International Conference Leurning Representations, 2024.
machine-generated
Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. Conda: Contrastive domain adaptation for ai-generated text detection. arXiv preprint Stt::2309003999 2023.
Sid Black, Gao Leo. Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Mesh- Tensorflow, March 2021. URL https: //dd// org/ M
10. 11.5221/zeeoon htt1///eeood..9.7.5... If you this software, please cite it using these metadata.
11

---

Published as a conference paper at ICLR 2025
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler. USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022. URL httpsss//arxv..org//eaea 06745
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference on machine translation (wmt16). In First conference on machine translation, pp. 131-198. Association for Computational Linguistics, 2016.
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736 2023.
Evan N Crothers, Nathalie Japkowicz, and Herna L Viktor. Machine geeehineeggeerrtedddeeeee text: comprehensive survey of threat models and detection methods. IEEE Access, 1 1170997-77002,, 2023.
Geoffrey M Currie. Academic integrity and artificial intelligence: is chatgpt hype, hero or heresy? In Seminars in Nuclear Medicine, volume 53, pp. 719-730, Elsevier, 2023.
Holly Else. Abstracts written by chatgpt fool scientists. Nature 613:423 423, 2023.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 2018
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint 1xXiv:190.009199 2019.
Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao. Bias of ai-generated content: an examination of news produced by large language models. Scientific Reports, 14(1):5224, 2024.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint 1906.0990..444 2019.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmaan, Gustavo de Rosa, Olli Saarikivi, Adil Salim. Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL htttt::://axxv.ooo/ abs 2bs//306... 1164
Biyang Guo, Xin Zhang. Ziyuan Wang, Minqi Jiang, Jinran Nic, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.cm 00759997777..... 2023.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting Imss with binoculars: Zero-shot detection of machine-generated text, 2024. URL htttht:::// /ttp:/://///vvva// arxiv. org/ abs 2401. 12070
Xinlei He, Xinyue Shen, Zeyuan Chen. Michael Backes, and Yang Zhang. Mgtbench: Benchmarking machine- machine-generated text detection, 2024. URLpstts://aaxiit wttp:l/wwwwwwaaaaaaeeeaeaae /2////23333.222222
Daphne Ippolito. Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint
2019.
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: critical survey. arXiv preprint arXiv:201 2020.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. arXiv preprint 1rXiv:1909.066 00009006466 2019.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Hyunseok Lee, Jihoon Tack, and Jinwoo Shin. Remodeteet: Reward models recognize aligned Ilm's generations, 2024. BtRLhttss:// /tt//aaxivv....//.ii//eae/aaaava
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang. Linyi Yang, Shuming Shi, and Yue Zhang. Deepfake text detection in the wild. arXiv preprint arXiv:2305. 2023.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers, 2023. httpt:: http://www..ggab/////...o.
Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp. Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. Mapping the increasing use of Ilms scientific papers. arXiv preprint arXiv:2404.0/266 2024.
Muneer M Alshater. Exploring the role of artificial intelligence in enhancing academic performance: A case study of chatgpt. Available at SSRN 43/2358, 2022.
Shixuan Ma and Quan Wang. sero-shot detection of 1lm- Ilm--ennratee generated text using token cohesiveness, 2024.
http://aaxxvv.oooaa///aaa/////// 16914
Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting, 2024. URL httppp/s/aaaz....rr////////00000 12970
Eric Mitchell, Yoonho Lee. Alexander Khazatsky, Christopher D Manning. and Chelsea Finn. Detecigpt: Zero-shot machine-genereeeeeeee generated text detection using probability curvature. In International PMLR, 2023.
Conference on Machine Learning, pp.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/ 007.08745, 2018.
Andreas I. Opdahl, Bjørnar Tessem, Due- Que-Tien Dang-Nguyen, Enrico Motta, Vinay Setty, Eivind Throndsen, Are Tverberg, and Christoph Trattner. Trustworthy journalism through ai. Data & Knowledge Engineering, 146:102188 102182, 2023.
OpenAl. AI text classifier,
Jan 2023.
URL https://beta.ggenal.comm
text classifier
OpenAl. Hello gpt-4o.
http::///pppaaaacco..///////l/// 2024a
OpenAl. Gpt-4 technical report, 2024b. URL
Guilherme Penedo, Quentin Malartie, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 116, 2023. httptttt/////ttte org. abs 2306. 01116
Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhattacharya, Mobin Javed, and Bimal Viswanath. Deepfake text detection: Limitations and opportunities. In 2023 IEEE symposium on security and privacy (SP), pp. 1613-1630. IEEE, 2023,
Claudia Quinonez and Edgar Meij. A new era of ai-assisted journalism at bloomberg. Al Magazine, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392. Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/ D16- 1264. URL https: /tttsss://acaaaee antcaanntooooooo wowhoolooyy..oo org 6262
13

---

Published as a conference paper at ICLR 2025
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourn,, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1863vv/3/v/ P18-2124. URL http://sss//alaaaahooooo....y.
2124
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang. and Soheil Feizi. Can ai di-generated text be reliably detected? arXiv preprint arXiv:2303.11150 2023.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
Yuhui Shi, Qiang Sheng. Juan Cao, Hao Mi, Beizhe Hu, and Danding Wang. Ten words only still help: Improving black-box ai- ai-generated text detection via proxy-guidee efficient re-sampling. arXiv preprint arXiv:2402.09199 2024.
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. mgpt: Few-shot learners go multilingual, 2022. URL URLAttpp:://rxx/voorg
abs abs/ 07580
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, (81(2)))))7555555 2024.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:
2019.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. arXiv preprint 2005522306.05540, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rívière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 4rX/v:2403.08295, 2024.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8): 1930-1940, 2023.
Edward Tian. Gptzero: An ai text detector, 2023. URL httpp::s://pptterro
Yuchuan Tian, Hanting Chen, Xutao Wang, Zheyuan Bai, Qinghua Zhang, Ruifeng Li, Chao Xu, and Yunhe Wang. Multiscale positive-unlabeled detection of ai- di-generated texts. arXiv preprint
arXiv:2305.18149 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /397/, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023b.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). pp. 8384 8395, 2020.
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwritten by large language models. arXiv preprint 2rXiv:2355.15000 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. httpps://githii
wwwom/iinoofffooo lolz wttpo////e.ea.taannneeeeeeeeeeee
jax May 2021.
Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. Seqxgpt: Sentence- level ai-generated text detection. arXiv preprint 00.00223,0.08003 2023.
14

---

Published as a conference paper at ICLR 2025
Xianzhi Wang, Shubin Si, and Yongbo Li. Multiscale diversity entropy: A novel dynamical measure for fault diagnosis of rotating machinery. IEEE Transactions on Industrial Informatics, 17(8): 5419-5429, 2020.
Emily Wenger. Ai produces gibberish when trained on too much ai-generated data, 2024.
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:221.005000 2022.
Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. Evaluating reading comprehension exercises generated by llms: A showcase of chatgpt in education applications. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). pp. 610-625, 2023.
Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna- gpt: Divergent n-gram analysis for training- frreingg-eeeeeeeeo detection of gpt-generated text. International Conference on Learning Representations, 2024.
Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Yang. Pengyuan Zhu, Xiuwei Shang, Weiming Zhang, and Nenghai Yu. Dpic: Decoupling prompt and intrinsic characteristies for 1lm generated text detection, 2024. URL htttt::::///// arxiv, wrw./orggabb........ htsp//bs//2300 /2t0//2/305 12519
Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang. Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, and Lichao Sun. Lim-as-a-coauthor: Can mixed burttewritten and machine-generatee text be detected?, 2024. URL 05952
org.iviorr...ooooooo abs 2401
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, and Ming Gu. Beat Ilms at their own game: Zero-shot Imm-generated text detection via querying chatgpt. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7470-7483, 2023.
15

---

Published as a conference paper at ICLR 2025
A TASK AND SCENARIO DEFINITION FOR TRAINING-FRED DETECTION
Task Definition. Current research mainly defines the problem of L.M--generated text detection in two ways. According DetectGPT (Mitchell al. (023), training-free detection can be defined as: given a specific autoregressive language model M and a candidate text t - (41, lg,..., bn) containing n tokens, determine whether t was generated by M without any training. Under this setup, even if the judgment is "no", the candidate text may still belong to another language model Q( M). Therefore, we can only conclude that is not generated by M, rather than by humans. This definition is obviously more suitable for more precise detection situations such as tracing the source of text. In practical detection scenarios, we often seek a more straightforward conclusion. So this paper, we follow the definition provided by Fast-DetecIGPT (Bao et al., (2024) for the training-fie feiningg-free detection problem, which is: given a text t, determine whether is LLM-generaue without any machine training. This definition allows to any available language model (proxy model) for inference and judgment.
Scenarios definition. Suppose t is LLM- text, into white-box and black-box settings (Gehrmann et al., 2019) based on whether we can obtain the probabilities Po (t(((( for each token 4 from source model Ma that generated t, where is the parameters of M. In the white-box setting, the range of detection results is limited to the source model and humans. We input into Ms for inference, and the returned result contains actual statistical information about t, such as the probability of each token. In a black-box setting, detection becomes more challenging because we cannot know in advance whether t was generated by M. As result, it is impossible to use Md to obtain the actual statistical information about t. Fortunately. there are many open-source models (such as LLaMA Series (Touvron et al. 2023a b. Al Meta 2024h) with varying parameter scales that we can leverage. These models share similar underlying architecture, with the common goal of learning the general patterns of human text creation. Based on this consideration, compromise approach for black-box scenario detection is to select suitable open-source model (not necessarily the source model itself) as a proxy model, and indirectly realize detection through analyzing the proxy model's inference results on t.
classify the detection scenarios
Indeed, black-bor box detection is more common in real-world scenarios. With the release of increasingly powerful open-source and closed-source models, developing universal detector that can operate across different sources and domains to improve the accuracy of detecting L.MM-generated texts in black-box scenarios is becoming an urgent task. However, white-box detection also plays an irreplaceable role in tasks such as text attribution (He et al. 2024). where it is sometimes crucial to precisely identify the model that generated the text, a task that is no simpler than black-box detection.
B DETAILS ABOUT THE SETTINGS AND BASELINES
B. DATASETS AND PROMPTS
XSum, SQuAD, and WritingPrompts were primarily used for white-box detection, while XSum, WritingPrompts and Reddit were primarily used for black-box detection. For each dataset, we filtered out samples with text length less than 150 words, and randomly sampled 150 of these as human human-written examples. For each human- human-written written example, we used the first 30 tokens as a prompt to generate continuation from the source model. For source models that require system prompts or API calls, including OpenAl's gpt - tapp-4-ttre 4-turbo-2024 404-000--0 00, gpt- 1ost-4o-2 40---00--000- 2024 20-4--513 and Anthropic's is c claude- h- ku-20240307, provide the details of prompt engineering in 4
Table 4: Examples of prompts used in different datasets for Black-box detection.
Datasets
Prompts
System: You are a professional News writer.
Xsum
User: Please write an article with about 200 words starting exactly with: Prefix
Writing
System: You are a professional Fiction writer.
Prompts
User: Please write a story with about 200 words starting exactly with: Prefix
System: You are helpful assistant that answers the question provided.
Reddit
User: Please continue the answer in 200 words starting exactly with: Prefix
16

---

Published as a conference paper at ICLR 2025
Thus, each complete dataset contained 150 negative (human-written) and 150 positive (LLM- generated) samples, with roughly equivalent text length and the beginning portion between each pair. Finally, We have excluded the commonly used PubMedQA dataset (Jin et al. 2019) from experiments because the length of most samples in this dataset is significantly shorter compared to the other datasets. This discrepancy in sample length could compromise the faimess of the testing.
B.2 ALL LARGE LANGUAGE MODELS USED
All the source models used are listed in Table 5 which includes both open-source and closed-source models. Our experimental setup consists of two RTX 3090 GPUs (2x24GB). Due to GPU memory limitations, we did conduct experiments on models with parameter sizes greater than 20B, such as NeoX (Black et al. (2022), and instead replaced it with Falcon-7, Gemma-7, and Phi2-2.7 to cover as many large language model products from tech companies as possible. Except for Neo-2.7, GPT-2, OPT-2.7, Phi2-2.7, 2.7, and mGPT, which use full-precision (float32), the rest of the open-source models use half-precision (float16).
Table 5: Details of the source models that is used to produce machine-generatee text.
Model
Model File/Service
Paramelers
mGPT
Shliazhko
2023
1.3B
GPT-2
(Radford et 2019)
opemii-ommmuniyyin222-
1.5B
OPT:2.7
Ziang |
facebook/opt-2277
2.7B
N.u-2.7
cl 20211
ElcutherA//gtt-noo222..
2.7B
Phi2.2.7
Lunnasekar al
micmssoftpphi2222
GPT-J
Wang & Komaisuzaki
202
EleutherAUgpt--Pp
6B
XI1.5-6 2f 50044
01-ai/Yi-1.5560
Falcon-7
Penedoo
2013
tiuse/falcon-76
7B
Gemma-7
Iaam ct a
2024
google/gcmma-771
7B
Qwenl.5-7
[Ber
Qwen/Qwent
BLOOM 7.
Warkship el
2022
higscience/bloumm/75 7b1
7.1B
meta-llama/Met-111
OPT-13
Zhaing cl
207
facehook/opt-133
13B
Llama-13
100s1on al
2023g
huggyllana/lanss-1h
13B
Llama2
The
13B
GPT-4-Turbo (OpenAl
2024h
OpenAl
GPT-4o (OpenA]
OpenAT
NA
Claudo-3- (laudo---laakkt Aitrrppi 2024
Anthropic
NA
is worth noting that the default setting Past-DetectGPT uses GPT-J as the conditional independent sampling model and Neo-2.7 as the scoring model. However, we did not follow this setting and instead unified the sampling model and scoring model as GPT-J. On the one hand, GPPt (our main improved method) achieves the best performance compared to itself in dataset. On the other hand, using a single model not only saves GPU memory but also makes the detection process more unified.
B.3 DETAILS BASELINES
The implementation details of the baselines in this article are follows:
Log-Likelihood (Solaiman et al. 2019). The average log probability of all tokens in the candidate text is used as the metric.
Log-Rank [Solaiman et al. 2019). The average log rank of all tokens in the candidate text, sorted in descending order by probability, is used as the metric.
Entropy (Gehrmann et al., 2019; Ippolito et al., 2019). The average entropy of cach token is the vocabulary.
calculated based on the probability distribution
DetectLRR (Su et al. 2023). The ratio of log-likelihood to log-rank is used as the average metric.
The first group of methods aggregates the probability information of each token in the text by constructing appropriate statistics, which are used as the final detection score. By default, we use GPT-J as the scoring model in black-box scenario.
17

---

Published as a conference paper at ICLR 2025
DetectGPT (Mitchell et al., 2023). Perturbation likelihood discrepancy is used as the metric. We maintain the default settings from the original paper, using T5-3B as the perturbation model with 100 perturbations.
DetectNPR (Su al.
2023). Normalized log-rank perturbation is used as the metric, with all other settings consistent with DetectGPT.
DNA-GPT Yang et al. 2024). Contrast samples are generated using a cut-off and rewrite method,
with logelikelioood the metric. The default settings truncation rate of 0.5 and 10 rewrites.
Fast-DetectGPT (Bao et al. 2024). Sampling discrepancy used as the metric. To ensure a fair comparison, chose both the sampling model and the scoring model GPT-J, instead of the original setting where the sampling model was GPT-J and the scoring model was Neo-2.7. This modification results in better detection performance on our dataset. The number of samples remains at 10,000.
The second group of methods requires perturbing or sampling the original samples to obtain certain number of contrast samples, then calculating the statistical discrepancy between the contrast samples and the original samples as the final detection score. the black box scenario, we use GPT-J as the sampling, rewriting, and scoring model by default. Non-sampling methods (e.g., DetectNPR and DNA-GPTY generally require multiple model calls, making detection time longer. Therefore, sampling-basse -baied methods are more practical in comparison.
C DETAILS OF DETECTION RESULTS
C.I DETAIL OF MAIN RESULTS IN THE WHITE-BOX SCENARIO
The main results the white-box scenario are shown Table O First, the basic version of the Lastde method achieved the best detection performance among the cample-based training-free free methods, with an average AUROC 5.09% higher than the previously best best-performing DetectLRR method. Notably, it led by approximately 8% in the well-known LLaMA series and Gemma, making it a strong detection statistics. Second, among the 5 distribution- Lastde++ method achieved the best detection performance with only 100 samples, significantly fewer than Fast- Detst.ettetGPT. This demonstrates that our method has a notable advantage in the white-box scenario, providing insights for more detailed detection tasks such as model attribution.
training-free methods, the enhanced
C.2 DETAIL. OF MAIN RESULTS IN THE BLACK- -BO SCENARIO
Although the detection performance of all methods across the 12 source models showed significant decline compared to the white-box scenario, as shown Table 7 Lastde and Lastde++ still outper- formed other methods of the same type. Lastde++ maintained considerable leading advantage across all open-source and closed-source models. Notably, the two fast-sampling methods, Fast-DetectGPT and Lastde++, performed worse on Llama3-8 compared their base statistics, Likelihood and Lastde. A possible reason for this is that the detection performance may have already saturated, and the sampling process introduced additional randomness.
Additionally, in the black-box scenario, DetectLRR showed a noticeable performance gap in detecting GPT-4-Turbo compared Likelihood and LogRank, whereas this was not observed in the other open-source models. This observation is consistent with the conclusions of Destt-DteetGPTT (Bao
al. 2024), and we provide further analysis of this phenomenon Appendix D
18

---

Published as a conference paper at ICLR 2025
Table The details in Table include the main results of 8 baseline methods well as Lastde and Lastde++ on the white-box detection task across the XSum, SQuAD, and WritingPrompts datasets. DNA-GPT generates rewritten samples using the source model, while Dest-DeecctGPT and +astde++ utilize sampling from the source model. Other implementation settings are described in Appendix
B.3
MethoneMMaded
Nov22
GPT.J
OPT-13
BLOOM.77
Falcum-7
Axg.
Libelihood
Logtank
59.32
34.77
57.34
69.58
67.09
59.43
DesciLRR
92.44
88,69
99003
54.42
95.11
80.13
$7.11
Lasdde
97.44
96.52
108.0
92.33
98.13
92.68
92.28
9583
94.44
DRSSIGHT
39.52
10.41
63.35
13,
663T
07.55
60.27
78.5
$6.95
725
665.08
88.97
7X32
DesseNPR
90.95
02.31
$6.45
86.15
69.49
73.04
58.52
50.01
03.67
77.09
73.85
92.64
64.34
DNA-GPT
85.25
60.30
7718
73.35
59.25
56.3
59.40
71.32
80.32
00.15
59.36
83.64
74.02
Poet- Paat--DnenkPr
98.9
08.97
96
98.12
94.30
92.53
99.89
V5. 19
99.2
96.20
95.40
98.05
97.12
Lauide++
90.36
00.72
98.46
99.04
97.27
96.11
59.71
96.94
99.64
97.01
97.98
98.97
98.43
SQUAD
Likelihood
92.05
66.77
53.81
80.47
46.50
15.36
55.65
$4.04
85.03
75.00
72.79
85.21
78.29
LogKink
95.14
91
83.50
51.84
51.55
97.25
$7.92
90.55
81.20
76.77
87:36
K1A1
Estropy
57.88
DesecLLR
99.29
99.37
08.87
165
00.15
8636
771.0
7V338
04.13
$4.40
80.90
DNA GPT
93.24
18.7%
53.35
79.94
M0.46
DeseettiP
59.11
59.72
59.02
99.64
98.75
99.25
97.45
Ladide+
99.96
99.95
99.95
99.78
92.90
24
99.97
99.33
99.42
98.50
99.30
98.69
Wrateggrunuss
Likiihood
56.02
94.76
93.95
91.92
82.76
$4.54
99.55
92.17
93.62
86.13
69.58
97.13
50.30
LegRank
67.78
06.55
95.94
95.22
87.00
83.52
100,09
04.30
96.09
89.04
74.49
07.73
92.78
Entropy
40.90
$7.45
44.28
41.98
55.74
43.71
4.36
44.53
$1.07
47.75
67.52
25.10
42.41
DessoLRR
59.54
08.35
97.99
97.44
93.60
92.45
98.88
06. 50.50
98.60
92.94
81.40
97.93
95.52
Ladide
99.74
19.A0
99.80
98.83
9808
96.86
50.58
99.01
99:71
97.88
92.47
98.26
98.35
TARRROPY"
37707
125.9
"9.81
"02700
787.
B03
DeseeNPP
01.7
DNA
57:95
91.19
92.04
DewerCaPT
98.27
Loedde++
99.85
99.75
Table 7: The details in Table Table include the main results of 8 baseline methods, as well as Lastde and I Lastde++, on the black-box detection task across the XSum, WritingPrompts, and Reddit datasets. All baselines use the default settings described in Appendix B.3 for the black-box scenario.
MethodshMadd GPT-2
Noo-277
OPT-277
Liuno-13
Liama2-13
Liama3-S
OPT-1
BIOOM-7.1
Palcun-7
Gemma-7 7
Phi2-27
GPT-4-urbo
Arg.
25am
Liktthodd
$4.00
63.
54.33
05.43
54.51
LouRank
DetecLRR
66.48
71.25
24.45
71.72
Lasile
78.66
$2.17
83.47
72.30
67.01
99.32
$6.95
17.46
15.03
76.23
79.44
64.16
78.68
DeteUGPTT
55.80
35.97
663.51
56.25
54.56
75.05
05544
3637
55.83
62.25
51.22
07:35
59178
DetcINPP
55.04
50.72
63.55 5M
50.50
55.05
96.3
56.72
41.54
56.33
60.37
55.15
02.94
60.37
DNA-GPT
33.17
44.47
35 40
32.92
34.63
29.04
26
48.87
$4.63
55.08
34.28
$7.52
$7.40
Phen Pee--DeerriP
90:00
77:22
81.
64.79
02.81
$4.37
76.11
75 70
Lasidee
90
83.15
Likehhood
75.06
12.56
76.45
59.58
76.75
65.20
$1.48
79.9
Layfknk
83.04
35.72
$1.58
74.35
81.50
99 96
$1.98
$1.58
$1.43
68.05
56.57
79.01
$2.00
Entioy
$4.04
45.93
45.09
45.00
39.68
5.150
44.92
$2.43
44.11
49.17
36,99
15.56
42.14
Datcci. AR
83.61
00.52
00.55
65.27
85.00
$8.00
$9.00
$6.00
67.34
00.49
56.33
$5.72
Laside
M5.27
96.82
96.09
18.02
87.32
99.97
98.12
96.16
92.45
73.5%
93.68
33.09
91.43
Detc:GGP
74.46
84.86
79.00
DNA-GPT
24.27
75.82
73.55
$5.54
74.90
Pac-DevaGGP
05.36
96.10
93.05
85.76
85.44
99.56
60
02.24
50.94
75.55
01.65
40
90.8
Lasiden
95.99
98.93
97.19
94.46
93.85
98.84
95.25
97.39
95.84
$4.31
96.92
58.50
$4.97
Ekothea
65.55
10.45
61.01
66.35
71.35
106,00
S1.18
69.16
69.65
16.66
31.01
07.13
34.55
LogRask
70776
74.27
68.12
71.00
70.76
106.00
65:35
75.89
73.68
80.34
43.06
97.16
77.92
Eanrgyy
6485
$.000
43.21
36.48
5.480
44.71
Laside
91.71
89.53
89.27
96.74
90.41
76.77
66 TM
71-1
9154
"60.0"
75.00
103
69.20
73.23
72.13
09.7
75.03
97.
72:51
71.41
73.51
75.00
81.44
56.10
76.66
DNA-GIP
65.01
67.59
61:06
61
60.00
9982
63.78
65.84
69.83
73.00
70.36
42.12
71
Past DesSrGPT
02.79
$2.97
85. 39
7948
93.00)
34.57
91.46
$2.04
91.22
90.08
9338
55.06
Lastiee
07.81
93.90
87.70
97.06
95.67
05.00
94.36
D ADDITIONAL RESULTS ON DIFFERENT AGGREGATION FUNCTIONS AND MORE CLOSED-SOURCE MODEL DETECTION
We conducted additional tests of Lastde and Lastde++ on more closed-soure, models, and we also reported the impact of different aggregation functions these two detection methods. As illustrated in Figure X Lastde(Std) performs comparably to the classic Likelihood on GPT-4o, while maintaining significant advantage other methods in the group on the other advanced closed-source
19

---

Published as a conference paper at ICLR 2025
Table 8: The AUROC value of the closed-source model. When using Lastde or Lastde++, we considered five aggregation functions. Specifically, 2-norm refers to calculating the 2-norm of the sequence; Range refers to calculating the range of the sequence, Std refers to calculating the standard deviation of the sequence (which is our default setting); ExpRange and ExpStd refer to taking the exponential of the range and standard deviation the sequence, respectively.
worth noting that we adjust the 3 hyperparameters of Lastde to 8 = 3, = T = 15 to enhance detection performance. Lastde+ and the other methods maintain their default settings.
Methods
SourreMidels-
GPT-4 Turbo
GPT-4o
Clande-
Datasets
X.Sm
Wiitinghoopps
Reddit
Avg.
XSum
WeitingProngts
Reddit
A0g.
XSmm
Wezsngroompss
Reddit
Avg
Likelihord
97:15
79:00
LagRank
79.01
07.16
70:34
75.85
97.09
99.96
Entroyy
61.24
35.56
03.03
33.09
47.50
31.60
09.74
29.61
38.90
17.69
05.55
21.05
DeteceL.K
61:71
05.75
93.10
73.85
62.87
09.06
95.75
15.23
95.18
67.95
99.33
LastdesSid)
64.16
83.09
96.04
81.33
73.87
$6.20
97.74
85. $5.94
97.44
99.40
99.92
98.92
Aggregantin Juneciio
Fre-DananCPP
93.87
36.87
99:77
03
95
76.91
13.61
85.97
999330
82.67
85.37
91.72
86.93
96
91.34
96.57
91.29
59:
09.99
99.93
Lastde+//St))
93.00
47
96.58
92.79
09.96
89
09.92
Lastfos=(<splang
89.07
07.77
99.97
99.98
Lastdo-= (RapSid)
81.55
89.81
93.99
88.45
87.04
94.24
93.14
99.97
99.90
99.95
models. Furthermore, appropriate aggregation functions are beneficial for improving the performance of Lastde++
Intuitively, using ExpStd the aggregation function seems to be better choice than Std. However, when using it to detect open-source models, we found that although (Estdc++(ExpStd) also achieves state-of-the-art results, its advantage is far less pronounced than that of Lastde++((td),, which is in stark contrast to the results on the aforementioned closed-source models. Possible reasons include model size, hyperparameter settings, choice of proxy models, and the applicability of rapid sampling techniques. Given the larger role of open-source models in the current development of LLMs, we continue results accordingly, highlighting the significant performance improvements of Lastde++ over previous detection methods.
Std the default aggregation function and report most of the experimental
Next, in order to further enhance the persuasiveness of our approachs, we conducted extended experiments on several third-party datasets. The details follows:
Including more datasets. We conducted experiments on multiple datasets recently released by ReMoDetec (Lee et al. 2024) and Dest- (Bao al., 2024). These datasets cover four domains: XSum, SQuAD, WritingPrompts, PubMedQA, and six chat/instruct type or large- parameter (2 20B) source models: NeoX-20B, Ll0ma3-70B, GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, Claude3. For Fast-DetectGPT, we only use the dataset generated by NeoX-20B.
Including more baselines. We have added another two strong baselines: Binoculars (Hans et 2024), GPTZero (commercial, 2024-11-11 base version) (Tian 2023). For Binoculars, we adopted its default settings from the original paper. For GPTZero, we first replicated the detection results provided in the ReMoDetect paper, then implemented our own detection on Fast-DetectGPTTs dataset related to NcoX-20B.
Combining with plug-and- detector. We explored the feasibility of integrating method with recent plug-and-play detectors (plugins), e.g. TOCSIN (Ma & Wang 2024). Specifically, we combine TOCSIN with Lastde, Lastde++ and other detectors (except Binoculars, GPTZero), and report the detection results (AUROC values) on the above multiple datasets.
Unified hy hyperparameters. Here, unify the hyperparameters of Lastde and Lastde++ to 3, = 10, r' = 5. Note that this is not the best hyperparameter used by Lastde++ in our paper and dataset, but the following results still prove that it achieve SOTA performance. When testing the three datasets related to NcoX-20B, we set the aggregation function of Lastde and Lastde++ to Agg Std (default setting), and set it to Agg ExpStd when testing the other source models.
https: http:://shtttiaooooeeeeeee////ee https: http:///tttiiiioooooooo//////r/e
data
20

---

Published as a conference paper at ICLR 2025
We conducted experiments under the above settings and divided the results into two parts: before and after combining with TOCSIN. The results are as follows: It can observed that even without rumning Lastde++ under the optimal hyperparameter settings, performance. This conclusion remains valid even when compared to two newly introduced baselines: Binoculars (using proxy models with larger parameters, 7B 6B) and the latest version of GPTZero. Additionally, Lastde demonstrates the best performance among sample-based methods. Most importantl, the experimental results in Table 9 were all obtained third-party datasets, further demonstrating the reliability of our method.
method still achieves SOTA
Table 9: Detection results (AUROC) before combining with TOCSIN. It should be noted that Binoculars uses Falcon-7h 7BB observer. Falcon- dalcon-7B-Instruct as performer, and uses the judgment threshold under Low-FPR given by the author: 0.85364323107785527,
Models
Domains
Likelibood
LegRank
Desecrl
Past-DetectGiPT
Binoculars
GPTZero
Lastde
Lastdes4
XSum
68.1
69.9
70.6
84.7
54.3
60.1
78.8
88.4
NeBX-20B
SQuAD
64.8
69.3
77.6
85.9
56.6
52.6
86.2
91.8
Writingrompis
90.7
72.4
97.4
XSum
96.9
97.4
94,9
99.9
98.7
100
97.2
99.9
Llama3-70B
Wriinggrompis
98.2
97.9
93.8
99.9
100
99.8
98.2
99.9
PubMeotQ
71.1
85.6
90.7
XSum
87.8
89.1
87.2
98.3
94.9
100
88.9
98.5
GPT-4-Turo
WritingPrmppts
99.8
PabMedlQ4
85.8
85.1
74,5
87.8
90.0
87.2
86.4
87.8
69.8
GPT-4
WriiingPrompts
87.6
85.5
73.4
97.6
90.7
82.6
87.3
97.4
PabMedRQA
79.6
79.0
69.3
83.7
86.7
84.8
80.4
83.8
XSum
93.8
94.1
90.9
99.2
99.7
99.5
94.2
99.1
Gurr..5-Turbo
WritingPrupts
98.1
97.7
93.0
99.7
99.3
92.9
98.2
99.6
PubMedKQA
90.4
XSum
91.8
92.6
89.8
96.4
94.0
99.9
92.4
96.4
Clande33
WrinngPrompts
97.0
96.4
88.8
96.0
99.1
97.0
96.0
PabModKQA
85.5
74.8
88.0
88.0
86.2
87.9
Avg.
87.0
87.2
82.2
93.7
87.5
88.6
89.6
94.2
Table 10: The detection results (AUROC) after combining with TOCSIN. When using the TOCSIN module, the two hyperparameters n (the number of copies created for each input) and p (the proportion of tokens be deleted in each copy) set 10 and 15% respectively.
Models
Domains
Likelihnod
LogRank
Detectl.RR .RR
Fast-DetectGPP
Lastde
Lastde++
XSum
97.7
97.6
99.0
89.4
98.9
96.8
NeoX-20B
SQuAD
88.8
88.9
94.8
88.6
95.7
95.
WritingPrompts
98.1
98.2
98.0
97.9
99.3
99.2
XSum
99.2
99.4
97.2
99.9
99.3
99.9
Llama3-70B
WritingPrompts
99.7
99.6
96.8
100
99.7
100
PubMedQA
84.5
83.2
68.9
90.5
85.2
90.7
XSum
85.2
87.7
73.5
98.3
86.3
98.3
GPT-4-Turbo
WritingPrompts
98.0
98.0
88.8
99.9
98.0
99.8
PubMedQA
85.4
84.7
70.8
87.8
86.1
87.7
XSum
96.8
96.5
96.3
93.8
97.0
93.7
GPT-4
WritingPrompts
94.2
93.2
81.7
97.8
94,1
97.8
PuhMedQA
78.9
78.4
65.4
83.7
79.8
83.9
XSum
GPu-3.5-Tubo
WrotingPrompts
99.3
99.3
99.3
99.7
99.3
99.8
PuhMedQA
86.4
85.7
71.9
90.2
87.3
90.2
XSum
96.4
96.7
Claude-3
WritingPrompts
97.5
97.6
90.5
96.6
97.5
96.5
PuhMedQA
85.6
84.8
72.3
88.0
86.1
88.5
Avg.
92.4
92.4
85.5
94.3
95.3
All detection methods show performance improvements when combined with the lightweight plug- and-play detection module TOCSIN. Notably. Lastde demonstrates improvement 1.1 1,, compared to 0.6% for Fast-DetectGPT. With the increasing dataset and strong baselines, believe that Lastde and Lastde++ efficient detectors.
significant gain, with an
21

---

Published as a conference paper at ICLR 2025
E ANALYSIS OF THE APPLICATIONS OF MDE AND ABLATION EXPERIMENTS OF LASTDE
In this section, present further application of MDE Wang al. (2020) and Lastde
token probability sequence across various types of texts, which substantiate the validity of our prior observations and hypotheses.
XSum(Llama3-B)
SQUAD(BLOOM:7.11
WrtingPPonnolsGGPT4TuWodd
0.75
0.75
0.75
DE Value
DE Value
DE Value
0.50
0.50
0.25
0.25
0.00
0.00
9111112131415
9101112131415
Reddit(Gemma-7)
ZhihuEconomy Qwen1.5-7
WMT16GER(/GPT)
0.75
0.75
0.75
DE Value
DE Value
DE Value
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
7
1121314155
910111213141
1
9101122131415
Human
LLM
Figure 9: The comparison results of the MDE curves for human-LLM texts. The subtitles indicate the dataset or source model associated with each example. In addition to the scale number 7', the other two parameters are follows: and The example WritingPrompts (GPT-4-Turbo) is conducted in a black-box scenario, while the remaining examples are examined in white-box contexts.
observations from Figure ((a) found that under the same prefix, the dynamic variations in the token probability sequences for writtewritten subsequent texts and LLM-ggnerated response texts differ. The token probabilities of LLM-generater responses are more compact, whereas human responses exhibit the opposite trend. This is attributable the fact that L.MMs are trained with the objective of minimizing perplexity. Therefore, according to the principles of MDE, the DE values corresponding to human texts are theoretically expected to be larger. We performed MDE transformations on pairs examples, setting the scale number 7' to 15 for each text in every pair. Subsequently, we calculated the DE values for each scale, resulting in the MDE sequence comparison shown in Figure
Properties of MDE transformation. Based
It is evident that the curve representing humans consistently lies above the curve representing LLMs, and this conclusion holds across various languages, scenarios, and models. Interestingly, for both humans and LLMss, there is an overall downward trend in the DE values of the transformed TPS the scale increases. This suggests that after multiple filtering iterations, the token probability sequences gradually approach periodic or deterministic state. In summary, these results illustrate that the token probability sequences of humans and LLMs can be differentiated through MDE sequences, thereby validating the reasonableness of our hypothesis.
Table 11: Detailed results of the ablation experiments under the black-box setting. Using both Agg-MDE and Log-Likelihood corresponds the standard Lastde. The rest of the settings default.
Scoring Types
Xsum
WritingPrompts
Reddit
Agg-MDE
LigeLiielllhodd
GPT-2
Nep-2.7
OPT:2.7
GPT-2
Nep-2.7
GPT:2.7
GPT-22
Nep-2.7
OPT-2.7
/
$4,02
48:45
63.20
55.22
78.06
82.36
77.08
79.17
65.55
70:45
61.91
65.97
77.05
81.62
70.70
76.46
76.90
77.42
76.39
76.00
80.20
75.15
75.53
76.96
22

---

Published as a conference paper at ICLR 2025
MDE Aggregation Ablation. The Lastde consists of components: Log-Likelihood and Agg-MDE. To demonstrate the indispensable role of the aggregated MDE sequence Lastde, we conducted ablation experiments under on Log-Likellhood or Agg-MDE can only yield very limited distinctions between human and LLM texts, while their combination significantly enhances the discriminative power between the text types. Table [ indicates that, with appropriate aggregation functions and hyperparameter settings, relying solely on Agg-MDE can yield good results. Furthermore, when combined with Likelihood, accuracy (AUROC) can sometimes be improved by over 10%. We believe that Likelihood, classical statistical measure, provides comprehensive overview of the global information in the token probability sequence. At the same time, Agg-MDE focuses modeling segments of the token probability sequence, extracting useful yet often overlooked local information. These two types of information complement each other and are both indispensable components of Lastde.
complex black-box setting. As shown in Figure TO relying solely
AggMDE
Likelihood
Frequency
Frequency
Frequency
20-
2.-
15
10
5-
0-
0.04
0.06
0.08
60
50
30
Tamman
Figure 10: Black-box experiments on the WritingPrompts (GPT-2) dataset. The aggregation function and the hyperparameters for MDE both use the default settings provided in Subsection 3.2 The complete results can be found in Table 11
Discriminative ability of Lastde and Lastde++. We conducted further studies on Lastde and Lastde++ across various source models. As shown in Figure [1]. the approximate threshold for distinguishing human and LLM texts using the Lastde score is -40. The distribution of human texts is relatively flat, leading some overlap with the LLM distribution. However, Figure T2 indicates that after applying sampling normalization, the Lastde score makes it easier to distinguish between the two types of texts. The average score for LLM texts is around 4, while for human texts, it around 0, and both distributions are more compact, resulting in only minimal overlap. In conclusion, both detection methods we propose are highly effective.
(GFT:)
30
30
25
Pravenss
20
Resuencc
10
5-
-50
-50
-40
-30
-50
40
-30
-50
-50
440
-30
-70
-50
-50
-40
20
Score
Score
Score
Score
Figure 11: The distribution of white-box detection scores by Lastde for SQuAD, with the source models indicated in parentheses. Each subplot contains 150 pairs of LLLM-generated and human- written texts. The LLM--genrrate texts are generated by inputting the first 30 tokens of the human- written texts as prompts into the corresponding source models B.2 The implementation settings are described in Subsection 3.2
23

---

Published as a conference paper at ICLR 2025
SQUAD (GPT2)
SQUAD (Nco 2.7)
SQUAD (0PT
SQUAD (GPT))
18
25
Froquarcy
frequery
20
15
5
-2.5
0.0
2.5
5.0
-2
-2
Soore
Score
Scone
Score
Figure 12: White-box detection score distribution of Lastde++. The dataset is exactly the same as that given in Figure 11
F DETAILED RESULTS OF ROBUSTNESS ANALYSIS
F.1 SAMPLES NUMBER
Previous training-free detectors typically employ three strategies for generating contrast samples: using T5 for perturbation, rewriting with the proxy or source model, and fast-sampling with the proxy or source model. This paper adopts the fast-sampling technique by default. Since Lastde can serve as a standard scoring statistic, theoretically, we can enhance Lastde using any the three aforementioned methods. As shown in Table T2 generally speaking, the more contrast samples there are, the more accurate the scoring becomes. It is worth noting that we have already achieved state-of- titte-of-the off-the-an art performance with the support of fast-sampling technique, other two methods detail. However, this does not impose any limitations on the Lastde score.
did not explore the
Table 12: Detailed results corresponding to Figure White-box detection was performed on XSum, SQuAD, and Prompns with the source model being Gemma-7 and the number of contrast samples set to {10, 20, 50, 100}.
Methods
Datasets
XSum
SQuAD
WritingPrompis
Numbers
10
20
50
100
10
20
50
100
10
20
50
100
DetectGPPT
61.40
65.36
69.45
71.64
68.60
70.90
73.86
76.71
62.77
66.49
72.64
72.44
DetectNPR
76.83
77.10
78.36
77.83
78.41
79.39
80.32
79.78
75.49
77.97
77.17
77.23
DNA-GPT
58
58.43
59.24
59.78
67.01
65.69
67.17
67.68
62.92
62.53
63.72
63.07
Fast- GedeDeetctGGT
95.51
95.85
96.27
96.28
95.67
96.28
96.82
96.
96.74
96.95
97.27
97.32
Laside++
F.2 PARAPHRASING ATTACK
Evading detection through paraphrasing attacks is current research challenge. As shown in Table 13 and Table 14 the performance of Dett-eett drops significantly in both scenarios, while Lastde++ remains relatively stable. Overall, the white-box scenario better highlights the robustness of Lastde++. In the black-box scenario, Lastde++ performs slightly worse than Fast-DetectGPT on the
Comparison of detection effects before and after paraphrasing attack in white-box scenario. The source models are Llamal-13, OPT-13, and GPT-J. The average of the three shown in Table The values in brackets indicate the decrease in AUROC after attack compared to before attack.
Table 13:
XSimnnOiggnali
X8om(Prraghraeei
W Wrttnghhmmpsrrrgggall
Prting/hommee/Paaaaprra Reddii((riggha)) Reddiit/srrghrrweel
ZJomp-11
DetecGPP
(1.3.44)
1.87)
OPP:3
Fael- DeteDrtterCPPP
95.19
85-49 9.761
99.40
97.46 1.94)
98.81
96 2.70)
Lastdere
96.94
88.30 9.64)
99.00
97.43 (1 1.97)
99.43
98000 (1 1.43)
GPTJ
Fast DetectPP
98.12
89.53 (18.24)
99.23
96.40 [1 2.52)
98.58
95.97 2.61)
Laside=e
99J14
93.87 (.1.77)
99.75
98.41 1.34)
99.38
97.85 () 1.53)
24

---

Published as a conference paper at ICLR 2025
Table 14: Comparison of detection effects before and after paraphrasing attack in black-box scenario. The source models are Llamal-13, OPT-13, GPT-4-Turbo, and the rest of the information is the same as in the Table
NSuntParaphreed)
Residit(rrigieal)
Lions-13
Fast DeteaGPP
64.79
53,91 (1 (0,85)
85.26
92.09 (1
79.65
74.05 (1 5.33) $1.65 (1 6.02)
Lastdese
72.84
61.27 (c 11,57)
94,46
90.8 (2 (,411
87.70
OP7-33
Fast DetectiPT
94.27
72 63 L1 11
89.64
88.44 1.20:
84.57
X5.73 (1 0.84) (1 1.82)
80.58 (1
94.24
Pasi-DatcstPPP
5.83)
91.35 1.26)
Lasiless
original data, but it surpasses Tast-DetectGPT on paraphrased texts with a smaller performance drop. A similar trend can be observed on the Reddit dataset. In conclusion, Lastde++ holds significant potential for further research.
F.3 DECODING STRATEGY
Table 15: Specific details of the Figure
Methods Strategies -
Top-p (y 0.96)
Top-E Oe Neo-2.7
40)
Temperature (T'= 0.8)
Models
GPT-2
Nro-2.7
OP1-2.7
Avg.
GPT-2
OPT-2.7
Avg.
GPT-2
Neo-2.7
OPI-2.7
Avg.
XSSnm
Likelihood
73.61
LogRank
71.44
68.75
83.03
74.41
64.52
61.12
77.04
67.56
90.29
89.95
93.51
91 11.25
DetecLRR
73.20
74.52
80:48
76.07
73.65
79.
75.34
91.17
90.28
93.49
91 65
Lastde
91.71
89.65
84.59
88.92
90500
87.
Lastde++
97.36
95.62
97.81
96.93
93.99
92.95
95.66
94.20
99.51
98 08.88
99.20
99.20
SQWALD
Likelihood
63.84
63.56
LogRank
64.02
69.40)
73.55
68.90
56.76
62.11
67.32
62.00
85.72
90008
90.85
83. 88
DeiectLRR
77.24
80.86
81.96
80.02
71.78
80.73
81. 11.80
80.10
92.16
94.81
94..49
93. 93.82
Lastde
88.27
91.32
92.57
90.72
86.85
68.22
88.47
87.88
95.14
95.13
96.19
95.49
9/760
Lastdex+
96.84
98.78
96.98
97.53
95.08
96.47
96.48
96.01
99.02
99 .44
99.65
99.37
Writingrrumppy
Likelihood
LogRank
92.31
90.11
97
DetectLRR
91.93
95.16
91.28
92.79
91 66
93.30
91
92.12
98.08
98.45
97.14
97
Lastde
97.83
99.30
97.58
98.24
96.18
98.35
96.59
97.04
99.12
99.84
99.30
99.42
36.78
'6.88
D6C17
12.36
7.4.73
'9
99.50
99.70
As shown in the Table 15 Lastde++ consistently maintains a detection performance above 94%
across all three datasets and outperforms Fast-DetectGPT in every strategy. Additionally, Lastde demonstrates the best average detection performance among similar methods. Notably, with a top-k value of 40, Lastde++ achieves average absolute advantages over Dett-eettC of 8.30%, 6.49%, and 3. 7.70% across the three datasets, highlighting its superior robustness in handling different decoding strategies.
F.4 DETECTING LLM-GENERATED TEXT TILAT MIMICS HUMAN WRITING STYLE
In this part, also compared the performance of Lastde and Lastde++ with other baselines in detecting L-M-eenerated text that mimics human writing styles. Specifically, we constructed two new datasets using Writing with uptt4-turbb-- 00:00004--000 04-09 and gpt 40-3-- ---08-06, respectively. Unlike previous work, we designed the prompts as follows: "Please mimic the human writer article with about 200 words starting exactly with: <prefix>," " explicitly incorporating the requirement to imitate human style. The table below shows the results based on the average of three metrics (AUROC,TPR.Accurre,, across the datasets:
25

---

Published as a conference paper at ICLR 2025
Table 16: The average of WritingPromptss
Likelihood
LogRank
Detect
Lastde
Fast-DetecIGPT
Lastde++
AUROC
87.37
85,60
74.94
87.68
96.49
96.49
TPR at 5% FPR
30.00
30.67
24.34
29.33
82.67
85.36
Accuracy at 5% FPR
62.50
62.83
59.67
63.34
88.84
90.17
For both human-style mimicking datasets, Lastde+ shows comparable better results across all three metrics compared to Fast-DetectGPT (Bao et al. 2024), demonstrating its effectiveness in detecting LLM text that mimics human writing style.
F.5 DETECTING LLM-LLM MIXTURE TEXT
In real-world scenarios, malicious users may generate text using multiple different LLMs. Therefore, it is necessary to explore the performance of existing detectors type of text referred to as LLM-LLM mixture text. To this end, we constructed dataset of texts generated by multiple different LLMs, using WritingPrompts text mixtures generated by two and four L.MMs.
example. Specifically, we conducted detection experiments
In the two-model mixture experiments, we evaluated two different mixing ratios: 50%+50% (Llama2- 13B, OPT-13B) and 80%+20% (BLOOM-7B, Falcon-7B). In the four-model mixture experiments, we used texts with an equal distribution of 25% from each of four LLMs (Llama2-13B, OPT-13B, BLOOM-7B. Falcon-7B).
We employed simple "truncation-coneet "cncatenatioon method generate the mixed data. For instance, we took 50% of the characters from the text generated by the first source model and concatenated with 50% of the text from the second model. Although this approach might lose some contextual information, we believe it is sufficient and effective for demonstration purposes. The experimental results are as follows:
Table 17: AUROC on LLM-LLM mixture text
Mising ratio
Source Models
Likellhood
LogRank
DetectL.Rg
Lastde
Fast-DetectGPT
Lastdes-
50%
(Llama2-138., OPT-13B)
(BLO0M-78. 78, Falcon-7B)
7K.80
82.56
87.99
87.47
94.95
88. 59
95.65
80% 200%
(HLO00--7,, Polcon-7BB)
ILlama2-13B OPT-13B)
80,44
81.52
82.97
85.45
89.13
92.05
86.73
92.64
Jama2-135 OPT-13R, HL.00M-7H, Falcos-78)
82.33
86.02
Table 18: TPR 5% FPR LLM-LLM mixture text
Mixing ratio
Source Models
Likelihood
LogRank
DetectLRR
Lastde
Fast Past-DetectGPT
Lastde++
50%+50%
(BLO0M-78. Palcon-7B)
(alama2-333 OPT-138)
29.33
16.00
30.00
34.67
76.00
46.67
39.33
70.67
80% +01%
(BLOOM-7B. Falcon-7B)
(1 Jama2-13R 13B, OPT-13B)
24.67
16.67
27.67
28.67
35.33
38.67
59.67
58.67
41.33
49.33
83,33
69.33
(Llama2-13B,
Palcon-7B)
Table 19: Accuracy at 5% FPR on LLM-LLM mixture text
Mixing ratão
Source Models
Likelihood
LogRank
DetectL.RK
Lastde
Fast-Detet
Lastde+
50% +50%
(BL (8L00.-778, Falcos-78)
(L.amu2-115 38, OPT-138)
62.17
55.55
62.50
65.50
67.50
64.83
80.17
85.50
67.17
70.85
81.50
82.83
lama2-118 OPT-13B)
(BLOOM-7B. 7B, Falcom-7B)
55.83
58.83
69.83
87.83
77.17
89.17
25% each
(Llama2-133 13B, OPT-13B, BLOOM-78 78, Falcon-78 7B)
$8.50
67.83
70.50
88.17
67.83
$1.17
The results show that even with texts mixed from different source models and different ratios, Lastde and Lastde++ still achieve good detection performance. In all three metrics across all datasets, Lastde or Lastde++ reach SOTA, especially with TPR (at 5% FPR) exceeding Fast- DetectGPT. the prior SOTA, by at least 20%.
26

---

Published as a conference paper at ICLR 2025
G DISCUSSION ON THRESHOLD SELECTION IN REAL-WORLD SCENARIOS
For training-free detectors, most studies Su et al. 2023 Bao et al. (022 Yang et al. 2024) use
AUROC as the evaluation metric, which does require setting specific threshold. However, real-world applications often require decision-making at fixed threshold, so it is essential to examine the performance of detectors under specific thresholds. We investigated
methods:
The perspective of G.TT.. The authors (Bao et al. 2024 of the paper have released local inference code for making decisions on individual samples. this code, they use an "offser" and then apply it to other
to estimate the threshold on a specific dataset (e.g.. KSummCCattGPa datasets. The authors claim that this estimation method does not introduce significant bias.
Perspectives some benchmarks. Other related studies (He et al. 2024 Mao et al,, 2024 Zhang
al. 2024) achieve final classification by using detection scores labels to univariate logistic regression model.
We prefer the latter approach and have fitted logistic regression models on datasets (including Xsum, WritingPrompis, Reddit) generated by two closed-soure models (GPT-4-Turbo, GPT-4o) and one open-source model (OPT-13B), reporting metrics on the test set (test size=0.2). Average AUROC/TPR metrics at I 1%FPR on three datasets are reported below.
Table 20: Comparison of AUROC and TPR values three datasets
Metrics
Methods
GPT-4-Turbo
GPT-4o
OPT-13
Avg.
Fast-DetectGPT
88.91
90.81
78.27
86.00
AUROC
Lastde++
87.02
91.29
95.88
91.40
Fast- Gest-DeeT
26.88
46.24
24.73
32.62
TPR 1%FPR
Lastde++
41.94
48.39
66.67
52.33
The fixed threshold here corresponds the decision boundary of the logistic regression model. For closed-source models, Lastde++ shows comparable or better performance in both AUROC and TPR compared Fest-DetectGPT, while significantly outperforming Fast-DetectGPT on open-source models. However, this threshold determination method still relies true labels, which we consider limitation, as it appears undermine the 'training- -feee' aspect (i.e., the inability to use real labels).
To resolve this issue, we propose more reasonable modification. According to Figure 12 in our Appendix F. regardless of the source model, the Lastde++ score boundary between human-written and I.I. --ggerrreet texts consistently appears around 2.0, which can be estimated more accurately with larger sample sizes. While setting the Lastde++ detection threshold directly to 2.0 might yield satisfactory results, more robust approach would be to first define neighborhood with radius centered at 2.0, then assign pseudo-labels to samples on either side (human for left, LLM for right), and finally fit logistic regression model. Notably, this modified approach doesn't 't utilize any true labels, reducing the disruption to the assumptions of the training-free detector.
27


Reviews:
[{"review_id": "RRtdzFfizG", "reviewer": "Reviewer_SbPp", "summary": "This paper introduces a novel training-free detector, termed Lastde that synergizes local and global statistics for enhanced detection. They introduce time series analysis to LLM-generated text detection, capturing the temporal dynamics of token probability\nsequences. By integrating these local statistics with global ones, their detector reveals significant disparities between human and LLM-generated texts. They also propose an efficient alternative, Lastde++ to enable real-time detection. Extensive experiments on six datasets involving cross-domain, cross-model, and cross-lingual detection scenarios, under both white-box and black-box settings, demonstrated\nthat their method consistently achieves state-of-the-art performance. Furthermore, their approach exhibits greater robustness against paraphrasing attacks compared to existing baseline methods.", "questions": "1. I have one question. I understand that this is a training-free method. But it seems it includes various ingenious designs such as multiscale log-probability sequence, sliding-segmentation and diversity entropy and etc. And I would assume this can not be designed at the first glance. So thinking from the first principle, given the token logits, if the token logtis can be a good indicator through another function f(), then what's the best f()? perhaps we can directly a network to find the f()?\n\n2. It is unclear to me whether the used models like llama are the base model or chat model because instruction-tuned models are more difficult to detect and base models are easy.\n\n3. Do you need to know the prompt and question used for generating the text for detection? in reality, we do not know this.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "originality: treating text detection as time series analysis is novel and the approach seems promising, though previous work SeqXGPT already did this. \n\nquality: I like the solid experiments from this paper since it compared with many previous baselines on a bunch of models and datasets. From all the experimental results, we can see the great potential of this method. \n\nclarity: This paper is well-written and easy to follow. The tables and figures and clear.\n\nsignificance: This provides a new method for detection and potentially useful for detection.", "weaknesses": "AUROC is not a very good measure for real-world use because FPR in real life is very important for this task, especially for students. I would like to know the TPR rate at 1% FPR.\n\nIn my opinion, treating the detection as a time series analysis is not entirely new. For example, your cited another work SeqXGPT also treats the token logits as waves, though your approach is different. Also, there is no comparison with SeqXGPT.\n\nThere are many hyperparameters in this algorithm to tune, weakening its potential practical usage.\n\nMany tables rely on results on very small models like GPT-2 Neo-2.7 OPT-2.7, which makes it less significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JJdUdEaedP", "reviewer": "Reviewer_WrSh", "summary": "The paper presents a training-free detector for LLM-generated text by analyzing token probability sequences. The proposed Lastde method combines both global and local statistics, using diversity entropy to capture temporal dynamics, thus achieving improved cross-domain detection and robustness against paraphrasing attacks. Evaluations on 6 datasets show that Lastde outperforms existing detectors.", "questions": "How does Lastde handle highly stylized human-written text that might mimic AI-generated patterns?\nCan Lastde be adapted or optimized for human-LLM co-authored text (e.g., LLM-revised text)?\nWould a hybrid approach combining Lastde with a lightweight training-based model further enhance detection accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The method is highly effective across cross-domain and cross-model scenarios without needing retraining.\n- Lastde++ enables real-time detection with minimal computational cost, outperforming many established methods.\n- Exhibits strong resistance to paraphrasing attacks, maintaining accuracy in varied textual manipulations.", "weaknesses": "- The detection accuracy depends significantly on the chosen proxy model, especially in black-box settings, affecting usability across model types.\n- The choice of proxy model in black-box scenarios is still unclear, and the use of GPT-j need more justification.\n- Performance drops with shorter texts, which limits applicability in scenarios like social media or short Q&A responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tRSR0PLUwD", "reviewer": "Reviewer_Kqgm", "summary": "The paper introduces a training-free method called Lastde for detecting LLM-generated text by analyzing TPS through a blend of local and global statistical features. Lastde incorporates temporal dynamics in TPS with diversity entropy to capture subtle distinctions between human and AI-generated text. An enhanced version, Lastde++, offers faster, real-time detection and outperforms existing methods across various scenarios. Extensive experiments demonstrate that Lastde++ provides superior robustness against paraphrasing attacks and cross-lingual challenges, establishing it as a powerful benchmark in LLM-generated text detection​.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper attempts to solve an important LLM-generated text detection problem without further training \n\n- The proposed method is straightforward and intuitive, and the experiments presented in the paper are comprehensive with solid results.\n\n- The readers can easily understand the proposed method and follow the content of the paper.", "weaknesses": "- I recommend including more evaluations on separate sets of LLM-generated text and human-written text ( i.e., evaluating the method on sets containing only LLM-generated text and only human-written text). This would provide valuable insights into how the proposed method works and whether the detection method performs better on LLM-generated text, or human-written text, or both.\n\n- Paraphrasing attacks pose a significant threat in the context of LLM-generated text detection. It is highly recommended to use a more powerful LLM paraphraser, which could better highlight the proposed method’s ability to protect against such attacks.\n\n- How did you predetermine the detection threshold, especially when the method is training-free and there is no prior knowledge about the LLM-generated text for the detector? The paper does not discuss the details of how the threshold was chosen but only provides a specific value used in a certain scenario. The threshold is a key component that leads to good detection performance no matter how good the proposed scores are extracted.\n\n- Generally speaking, when we attempt to detect whether a given text is generated by LLMs, we usually do not know which specific models malicious users have employed to produce the text. It would be interesting to see a further experiment conducted on a mixed set of LLM-generated text created from various sources LLMs. The evaluation of detection abilities in the above experimental settings more closely matches real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVsyFNlsUP", "reviewer": "Reviewer_LuhL", "summary": "This paper presents an AIGC detection method named Lastde. It is training free and can used both local and global statistics for\nAIGC detection. An more efficient version named Lastde++ is also proposed for real-time detection.", "questions": "1. It would be better if the paper writing can be improved. For example, figure 2 is not so easy to understand.\n\n2. My biggest concern is that strong baselines are not included in experiments, such as GPTZero.\n\n3. Can the proposed method work well in different domains?\n\n4. Is the proposed method robust to some adaptive attacks like asking LLMs to mimic the human writing?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents an effective AIGC detection method, which can be very useful in the GenAI era.\n\n2. The proposed method in sound and does not rely on training.\n\n3. The experiments show promising results.", "weaknesses": "1. It would be better if the paper writing can be improved. For example, figure 2 is not so easy to understand.\n\n2. My biggest concern is that strong baselines are not included in experiments, such as GPTZero.\n\n3. Can the proposed method work well in different domains?\n\n4. Is the proposed method robust to some adaptive attacks like asking LLMs to mimic the human writing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)