Deep Kernel Relative Test for Machine-generated Text Detection
Authors: Yiliao Song
Zhenqiao Yuan
Shuhai Zhang
Zhen Fang
Jun Yu
Feng Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Machine-generated text detection
Relative test
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: A robust detection method for LLM-generated texts using relative test

Abstract: Recent studies demonstrate that two-sample test can effectively detect machine-generated texts (MGTs) with excellent adaptation ability to texts generated by newer LLMs.  However, two-sample test-based detection relies on the assumption that human-written texts (HWTs) must follow the distribution of seen HWTs. As a result, it tends to make mistakes in identifying HWTs that deviate from the seen HWT distribution, limiting their use in sensitive areas like academic integrity verification. To address this issue, we propose to employ non-parametric kernel relative test to detect MGTs by testing whether it is statistically significant that the distribution of a text to be tested is closer to the distribution of HWTs than to the MGTs' distribution.  We further develop a kernel optimisation algorithm in relative test to select the best kernel that can enhance the testing capability for MGT detection. As relative test does not assume that a text to be tested must belong exclusively to either MGTs or HWTs, relative test can largely reduce the false positive error compared to two-sample test, offering significant advantages in practice. Extensive experiments demonstrate the superior performance of our method, compared to state-of-the-art non-parametric and parametric detectors. The code and demo are available: https://github.com/xLearn-AU/R-Detect.

Published as a conference paper at ICLR 2025
DEEP KERNEL RELATIVE TEST FOR MACHINE-GENERATEE TEXT DETECTION
Yiliao Song"*, Zhenqiao Yuan Shuhai Zhang', Zhen Fang', Jun Yuśt, Feng Liu 2++ School Computer and Mathematical Sciences, The University of Adelaide, Adelaide, AU2 School of Computing and Information Systems, University of Melbourne, Melbourne, AU
Yuan"*,Shuhai
School of Software Engineering, South China University of Technology, Guangzhou, CN3 Australian Artificial Intelligence Institute, University of Technology Sydney, Sydney, AU School of Intelligence Science and Engineering. Harbin Institute of Technology, Shenzhen, CN3 {yil [yiliao. song, zhenqi a aoyuan02, shuhat i zhangshz, fengl u mm } gma i com zhen. f fang@uts edu. au, yujun&hit, edu. on
ABSTRACT
Recent studies demonstrate that two-sample test can effectively detect machine- generated texts (MGTs) with excellent adaptation ability to texts generated by newer LLMs. However, two-sample test-based detection relies on the assump- tion that human-writted texts (HWTs) must follow the distribution of seen HWTss As result, it tends make mistakes in identifying HWTss that deviate from the seen H/T distribution, limiting their use in sensitive areas like academic integrity verification. To address this issue, we propose to employ non-parametric kernel relative test to detect MGTs by testing whether it is statistically significant that the distribution of a text to be tested is closer the distribution HWTs than to the MGTs' distribution. We further develop a kernel optimisation algorithm in rela- tive test to select the best kernel that can enhance the testing capability for MGT detection. As relative test does not assume that text to be tested must belong ex- clusively to either MGTs or HWTs, relative test can largely reduce the false posi- tive error compared to sample test, offering significant advantages in practice. Extensive experiments demonstrate the superior performance of our method, com- pared to state-of oh-the-art non-pa con-parametric and parametric detectors. The code and demo are available: https: / /gggtuu.bb..../ com/ xLearn- AU/R-Detect.
INTRODUCTION
The advent of large language models (LLMs) such as GPT-3 (Brown et al., 2020) has demonstrated their remarkable performance in text generation across various applications, e.g. text summariza- tion (Liu & Lapata, 2019; Luo et al., 2023), dialogue generation (Li et al., 2016; Lancaster, 2023), and machine translation (Bahdanau et al., 2014; Lee, 2023). However, their misuse raises concerns, particularly regarding the generation of fake content (Zellers et al., 2019), plagiarism (Lee et al., 2023; Stokel-Walker, 2022), and other ethical issues (Weidinger al., 2021). The increasingly indistinguishable machine-generated texts (MGTs) produced by newer LLMS aggravates worries about authenticity (Lin et al., 2022) and accountability (Susnjak & McIntosh, 2024). Recent re- search further highlights the versatility of LLMs in generating domain-specific content that can even deceive domain experts (Else, 2023), necessitating reliable MGT detection techniques.
Existing post-hoc detectors are generally classified into three types: metric-based methods (Mitchell al.. 2023; Soto al., 2024: Hans et al., 2024), classifier-based methods (Hu et al., 2023; Tian et al., 2024), and test-based methods (Zhang et al., 2024). Since metric-based and classifier-based methods are parametric, their performance inevitably depends on specific types of MGTs, limiting their adaptability. In contrast, the non-parametric test, theoretically supported by the kernel two- sample test (2ST) (Gretton et al., 201 2a; Liu et al., 2020; 2021; Gao et al., 2021), ignores specific generation mechanisms and focuses solely the intrinsic differences between human-written texts
"Equal contribution. CCorrespondence to: yujun @ hit.edu.cn, fengliu.ml @ggaall.comm

---

Published as a conference paper at ICLR 2025
Text from the distribution of seen Human-written text (e.g. ones in HC3)
Text from the distribution of unseen Human-written text (e.g. ones TruthfulQA)
Text from the distribution of machine-generatee test
Same distribution "witten by Human"
Closer Human written text "wittin by Human"
Different distributions generated by machines"
Closer to Human-writen text "writton by Human"
Fallure Case
Success
Human-written
Different distibutions
Closer Michne-generaled "penerateed by mochines"
Text at hand
Igenerated machines'
hand
Texts
Two-sample Testing Results
Texts
Rolative
fested
Testing
Relative testing for machine -generate text detection Figure I: Difference between using two-sample testing (subfigure (a)) and relative testing (subfigure (b)) in detecting machine-genese generated texts. Blue arrows represent the two-sample testing procedures, and orange arrows represent the relative testing procedures.
(a) Two-sample testing for gennerte-ee text detection
(HWTs) and MGT's Consequently, it performs well even on mixed texts generated by different LLMs under varying LLM settings (e.g.. temperature, top-k sampling (Vilnis et al., 2023)).
However, using 2ST may result in a high false positive rate of marking HWTs as MGTs (Zhang et al., 2024). False identification of HWTs can lead unnecessary content removal or unjust accusations, which is unacceptable in sensitive areas such as academic integrity verification (Dalalah & Dalalah, 2023). During detection, an MGT is identified by rejecting the null hypothesis that unauthored texts and HWT references are drawn from the same distribution (Zhang et al., 2024). Intuitively, the 2ST-based detection method assumes that the unauthored text must exclusively belong to either MGTs or HWTs that is, the distributions of HWTs and MGTs should not have any overlap in principle. However, since in practical situations the distributions of HWTs and MGTs often overlap, the non-overlapping assumption may lead to high false positive rate.
To address the challenge caused by non-overlapping assumption, we propose employing a non- parametric kernel relative test to determine whether a text is written by machine or a human. Introduced by Bounliphone et al. (2016), the relative test can determine which of two samples is significantly more similar to reference sample. Instead of using historical MGT or HWT samples as references, we use the unauthored text as the reference sample. By applying the relative test, can determine whether the MGT sample or the HWT sample from the database is closer to the unauthored text, thereby making a detection decision. Since the relative test does not assume texts exclusively belong to either MGTs or HWTs, the false positive rate is significantly reduced compared to two-sample tests. This offers substantial advantages for practical applications.
Selecting a suitable kernel is crucial for non non-parametric tests (Gretton et al., 2012a; Sutherland al., 2017; Liu et al., 2020; 2021), especially when dealing with complex data (Liu et al., 2020; Zhang al., 2024). However, the kernel selection issue is rarely explored in the context of relative tests. Here we propose novel method to optimize kernels to make non-parametric kernel relative tests more powerful in determining whether reference sample is closer to HWT or MGT. Specifically, by empirical studies, we discover that the kernels performing well in 2ST also consistently perform well in relative tests. Therefore, following Liu et al. (2020), we select the kernel for relative test by increasing the test power of 2ST. Experimental results show that the optimal kerne--base relative test significantly outperforms those based on common-use used kernels (i.e., Gaussian kernels).
Motivation of this study. This study introduces non non-parametric post-hoe method for detecting MGTs by framing the detection task as a relative test problem. Aiming for an interpretable and fine- grained MGT detection to ensure the ethical use of LLMs (Kumar al., 2023), we consider whether MGT is detectable in practice, which has been affirmed by studies (Chakraborty et al., 2024; Hans al., 2024). Consequently, we adopt the assumption from Zhang et al. (2024) that MGTs and HWT's are distinguishable in distribution. Although the 2ST-based MGT detection method (Zhang et al., 2024) offers statistical interpretability, is limited by assuming texts belong exclusively to either MGTs or HWTs. Furthermore, the choice of using MGT or HWT as reference can yield different detection results. These limitations motivate our use of relative tests for MGT detection this study.
Contribution of this study:
2

---

Published as a conference paper at ICLR 2025
The potential of using statistical hypothesis tests for MGT detection is explored, enriching the detection framework with robust theoretical foundations derived from hypothesis testing.
The MGT detection task is conceptualized as a relative test problem, providing enhanced detec- tion accuracy and flexibility compared to the traditional two-sample test method.
novel method optimize kernels in relative tests for MGT detection is proposed, significantly improving the effectiveness and efficiency of the detection process.
Superior detection performance is demonstrated across various LLM settings, clearly outper- forming ttate-of-the-eee po--arametric and parametric MGT detectors.
2
PROBLEM SETUP AND RELATED NOTATIONS
2.1 PROBLEM SETUP
As reviewed section I, the question of whether MGT is detectable in practice has been evidenced by previous studies (Chakraborty et al., 2024; Hans al., 2024). Based on this conclusion, we consider the case where MGTs and HWTs originate from two non-identical text spaces.
Text Space. Let S be the space of all possible texts. We consider HWTs to belong to the subspace Sp C and MGTs belong to another subspace Sm, C.
Problem 1 (MGT Detection). MGT detection aims find a detector : Sn Sm + [MGT, which effectively distinguishes between MGTs and HWTs.
To find the f. we consider only assume that S₁, n S 0 otherwise the same X will have two different Ys. For example, supposing the word "cat" written by human while another "cat" is generated by a machine, f cannot detect whether this "cat" is generated by human or machine. To demonstrate the impact of this overlap ratio on our designed method, we conduct toy study to analyse how the detection performance of our method varies by the overlap ratio in Appendix D.
In addition, we define tector D. that is, f= D g. In practice, employs techniques such as pre-trained transformers (Liu, 2019) to convert the original text into textual representations. Consequently, the MGT-detection problem is transformed into task of analyzing these textual representations, with the detector aim- ing to determine whether a text's representations originate from human- written texts.
the composition of feature transformation function g and a feature de-
To estimate the performance of MGT detector f, we consider the true positive rate (TPR) and false positive rate (FPR) as the metrics, i.e., TPR P. [f(s) MGT Sm] FPR = P. [f()) = MGT E S,] $
We expect that the MGT detector achieve high TPR while maintaining low FPR.
2.2 NOTATIONS AND CONCEPTS
Maximum Mean Discrepancy. Maximum mean discrepancy (MMD) (Gretton et al., 2012a) com- paring samples from distribution, aims to measure the closeness between two distributions.
Definition 1 (Maximum Mean Discrepancy (Gretton et al., 2012a)). Let k XxX-RR be the bounded kernel, P and Q be Borel probability measures X' Rd. Given x and x' independent observations from F. and y and y' independent observations from Q the squared MMD is
Q:k) 2MD2 (P, Q; k) =Ex,x' kk (x, x')] +Ey,y' [K (y.y') - [k(xxy)] y)]-
Here MMD refers to the population discrepancy. However, we can only observe the sample from distributions in practice. Following theorem shows that the unbiased empirical estimation of 2MD2 (P. Q; k) exists, meaning the population MMD² (IP, Q; k) can be estimated by finite sam- ples. This provide an effective way for us to estimate the population MMD) via sample data.
Theorem 1 (Gretton et al. (2012a)). Define independent identically distributed (i.i.) observa- 1m pm and (y)= QTL Let V: [vi)m= be observations with
tions Xm -
v V:). Then the unbiased empirical estimates of MMD (F, F. is:
MMD? (Xm: Ym:; k)= m(m
1
Σ
h (V., Vz),
3

---

Published as a conference paper at ICLR 2025
where h (v:, v,)
Given P + Q and E (h?) < co,
(X m: Ym; k) converges distribution to Gaussian according to Vm (MMD) (Xm, Ym; k)) MMD (P, 24 N (0, 3xy)
where
xx -4 4 (Enh (v, v') [Ev.v (4 (v, (v,
(2)
Relative Test. Relative test (Bounliphone et al., 2016) comparing the similarity of two candidate distributions a reference distribution, aims determine which candidate is closer to the reference.
Definition 2 (Relative Test), X y. P and Q be defined as above; be an observation with dis- yy 1 ~ and Z, Z [= ~""
tribution E. Given i.ddd observations (x= } ~lmm, ,
and IP + Z. Q o Z. relative test the statistical hypothesis test that: Null hypothesis:
MMD [P, Z, k) S MMD Q Z; k),
Alternative hypothexis: MMD(P; Z, k) > MMD (Q. k)
and the p-values for testing null hypothesis versus alternative hypothesis are MMD (X.. (X k,
- MMD Z,;k)
p
(3)
Voxz + + 20xzYz
where is the cumulative distribution function (CDF) of a standard normal distribution, and o the covariance. Here, z and x are the variances of MMD (Yn, Zr; k) and MMD Ze: k) respectively (refer to Equation 2). The empirical estimation dy z is presented in Appendix A.
METHODOLOGY
3.1 RELATIVE TEST AS A SOLUTION
In this section, we propose relative test to search the detector f as the solution of Problem 1. Let X m {x, ~ Ipm i.i.d. and Y, = {y, [ Q" i.i.d. be observations from Sn and Sm. respectively. The distribution P encapsulates the statistical charac- teristies of the seen HWTs, while Q represents the distribution containing statistical characteristics of seen MGTs. The distribution Z corresponds to the texts being tested.
When P * Z and Q F Z. given significance level a, the relative test for MGT detection will classify a given text E S as MGT if the null hypothesis of the relative test,
MMD (TP, k) S MMD(Q. Z; k),
(4)
is rejected. Specifically, let (s) = 1 (pca), then the decision rule is defined as
MGT if fs: f(s) HWT if [s; = 0,
(5)
where denotes the indicator function, and p is the p-value shown in Equation 3. We give the implementation of relative test MGT detection (R-Detect) in Algorithm 1. The input of Algorithm Xm, Ym- g- a. 9 is a fixed function that converts texts to textual representations. In this study, we fix g as OpenAl's RoBERTa-bees discussed in Appendix B. le is given kernel function, either learning from samples by Algorithm or using pre-assigned (see Appendix B). a is the threshold used for rejecting the null hypothesis. In default setting, = 0.05 and we also present results for different a Appendix C. Given these input, we first calculate the MMD? values to come out the value. By comparing p with a, we can get the detection result as the output of Algorithm 1.
GPT-2 detector model (Liu, 2019) with more details
Test Power V.S. TPR E.S. FPR. In hypothesis testing, the test power is defined as the probability of rejecting the null hypothesis when the alternative hypothesis is true (Zhang et al., 2024). In R-Detect, the null hypothesis is formulated as
(TP, Z; k) < MMD(Q, Z; k),
which implies that the text to be tested an MGT. Consequently, the test power R-Detect given that the ground truth is MGT, is the probability that an MGT is correctly identified as an MGT, corresponding to TPR. Similarly, the test power in R.Dettect given that the ground truth is a HWT, is the probability that an HWT is incorrectly identified as MGT, corresponding FPR. Specifically, we present the calculation of evaluation metrics for R-Detect in Algorithm 2, utilizing test power and the Area Under the Receiver Operating Characteristic curve (AUROC) (Jiménez-Valverde, Valverde, 2012).
4

---

Published as a conference paper at ICLR 2025
Algorithm Relative Test MGT Detection
Algorithm 2 R--Detect Evaluation
Input: Xm, Ym, g. k. a;
Input: Xm. Ym, 9, k, o;
Test Text: 8 E S:
Test Text: Si C Sn. S2 C Sm for round = 1, ...,..... do
% HWTs, MGTs
r< MMD v (g( g()))a))
% Equation I
h+ MMD g())))m)))))))))))))
% Equation I
Randomly Choose Sh. St, Sm. S2
pf (rr h):
% Equation 3
D1. pr R- Sr.);
Algorithm 1 % Algorithm
f+ f(s;
% Equation 5
D2. P2 2 ReDetect(-Sm):: end for
if === 1 then
Output: is machine-generatee text
FPR PaeD; (d = 1); % test power given MGT
else
TPR- Da (d = 1); % power given HWT
Output: & human-written text
AUROC D2, D2, P1. 1-.1-10 P2i
end if.
Output: TPR, FPR, AUROC
3.2 KERNEL OPTIMISATION FOR RELATIVE TEST MGT DETECTION
In section 3.1, we provide predefined kernel function as an input to Algorithm 1 for calculating the value of MMD This design allows R-Detect to generate detection results directly without requiring any training. Specifically, Bounliphone al. (2016) employ a Gaussian kernel, where the bandwidth is determined by the median pairwise distance between data points. The choice of kernel can significantly impact the test power in non-parametric tests (Gretton et al., 2012b; Sutherland et al., 2017), especially when handling complex data types (Liu et al., 2020; 2021).
However, how select an optimal kernel is rarely explored in relative test. Inspired by existing studies on kernel optimisation for kernel-based 2ST, we here derive our kernel optimisation for relative test from the empirical study that answers the following question:
"Is it empirically feasible to empower relative test from a corresponding two sample test?"
MMD-based 2ST. Kernerl-based method very popular class of non-parametric statistical tests =
(Berlinet & Thomas-
2011). Using kernel-based MMD for the two-sample test has a history
(Gretton al., 2012b). Given Xm= ] ~mm =x=ly; 13- ~ MMD-based 2ST aims to determine whether X,, and Y, are from the same distribution, that is, I Q
Deep Kernel Optimisation. The selection of kernel always challenging as good kernel can largely increase the test performance. Recent study proposes deep kernel, which builds kernel with a deep network to enable optimising kernels function for MMD--aased 2ST through maximizing its test power (Liu et al., 2020). We here consider the deep kernel technique (Liu et al., 2020):
(6) <x(()) is a deep neural network with feature extractor f. and a are Gaussian kernels with bandwidth dg and bandwidth og. respectively. Since f is fixed, the set of parameters of kiw is w {e, ao. ).))
le (x. ())))))(()((((((X))) ( (x)))+cl((f f f(y)),
where E (0, (0, ()
For the empirical use of 2ST in multiple population scenario (Zhang et al., 2024), we consider the multi-population aware optimisation for kernel-based MMD:
k = arg max MPP(X,,, m. Yn kw)/k//a (Xm; Ymi kin)
Kw
where MPP (Xm, Ym; Ha) = E [kw (Xm, X²h) - 2k (Xmy Ym)l and o(Xm Ymi hiw) is the
squared root of variance for MMD, referring to Equation 2.
Test Power for 2ST-based MGT Detection. Let i..d. observations R == (r1, 72.... ... (m)) be refer- ence texts. MGT detection will mark a text s as MGT in either case of i) given R C Sn, the null hypothesis is rejected; or úi) given R C Sm, the null hypothesis is not rejected. Similarly, it will mark s as HWT in either case of iii) given R C Sm. the null hypothesis is rejected: or iv) given R C Sk, the null hypothesis is not rejected.
Table 1: Empirical analysis for test power of two-sample test in MGT detection.
Test Power
RUINT
ROMGT
S EMGI
TPR
FNR
SHWT
FPR
TNR
Correspondingly, the test power of the 2ST in MGT detection (i.e., the probability of rejecting and R from the same distribution) are:
i) Given C Sp. the rejection probability is TPR when s is MGT;
ii) Given R C Sr, the rejection probability is FPR when is HWT;
5

---

Published as a conference paper at ICLR 2025
iii)
Given R. Sm, the rejection probability is FNR' when s is MGT;
iv)
Given C Sm, the rejection probability is TNR' when s is HWT.
The empirical analysis on test power of 2ST in MGT detection is presented in Table
Kernel Optimisation. By comparing the empirical test power of R-Detect in section 3.1: Test Power v.s.TPR v.s. FPR with the 2ST case (Table 1), we conclude that the test power of the relative test is em- pirically equivalent to that of two-sample test using HWTs the reference for MGT detection. There- fore, we optimize deep kernel that achieves the best test power in two-sample MGT detection and apply this kernel in R- Let Str and 5th be collections of historical HWTs and MGTs used as the training set. Given 9, the mapping function that converts text to textual representations, and X, the hyperparameter for learning the deep kernel, the op- timized kernel k. is learned in Algorithm 3. By inputting the k. learned from existing HWTs and MGTs into Algorithm k; we obtain the relative test MGT detection with the optimized kernel.
Algorithm 3 Kernel Optimisation in R-
Detect
Input: X Xm, Y <- Y A 10
Initialize:
for 1,2,.... ....
kw + ( (x))) g.
% Equation 6
M R. [Ku (X, X') 2 (x, NYY)|: % Equation 2
(IP, Q: kiw);
M/ / V
< A Adhm
end for Output: kw-
Optimised kernel
4 EXPERIMENTS
4.1 SETTTINGS
Datasets and LLMs. We design our experiments on data from five benchmarks: HC3 (Guo et al., 2023), TruthfulQA (TQA) (He al., 2023; Lin al., 2022), RAID (Dugan al., 2024), and De- tectRL (Wu et al., 2024). These benchmarks encompass various LLMs, covering both adversarial attacks and non-English scenarios. Further details are provided in Appendix B. We also manually generate MGTs by GPT-4o (OpenAI, 2024) from five randomly chosen human-written essays in the Essay dataset (Verma al., 2024) with the prompt: "rewrite". Please refer to Appendix F for the texts we are using and its corresponding rewritten texts.
Baselines. We compare R-Detect to five non-parametrie baselines and seven parametric base- lines. The non-parametric baselines include two variations of our method 1) R-Detect-k""; R- Detect with the Gaussian kernel optimised by median heuristic bandwidth (Bounliphone et al., 2016) and 2) R-Detect w/o k², namely our method without kernel optimisation; one state-of-the- art non-parametric MGT detector MPP with different settings (Zhang et al., 2024), i.e., 3) MPP- HWT, 4) MPP-MGT and 5) MPP-R; seven ofte-of-thheart parametric detectors, i.e. 6) Detect- GPT (Mitchell et al., 2023), 7) Fast-DetectGPT (Bao et al., 2024), 8) DNA-GPT (Yang et al., 2024a), 9) Bino (Hans al., 2024) and its variation 10) Bino-FPR, 11) DALD (Zeng al., 2024), and 12) Text-Fluoroscopy (Yu et al., 2024). Appendix B.4 provides more details of each baseline.
Evaluation Metrics. We evaluate the detection performance using test power for 2ST (Gretton al., 2012a) which is the TPR given MGT the ground-truth and false positive rate (FPR) given HWT the ground-truth. In addition, we evaluate the detection performance via AUROC ('iménez-Valverde, 2012). In the default setting, we randomly take 512 tokens and repeat the experiments 10 x 10 times given a specific experimental design. We use bold numbers to indicate the best results in tables.
During evaluation, our method is implemented in a zero-shot setting unless otherwise noted. For further implementation details, please refer to Appendix B.
4.2
COMPARISON BETWEEN NON-PARAMETRIC MGT DETECTORS
4.2.1
WHEN HWTS FROM SEEN HWT DISTRIBUTION
Here, we use the learned kernel function hy from HC3 test unseen texts from HC3 to mimic the MGT-detection when HWTss are from the seen distribution of HWTs using the datasets of
FNR = P. (s) = HWT E S... 2FNR [f() HWT| E Sn)

---

Published as a conference paper at ICLR 2025
Table 2: Test power (p) and AUROC texts to be tested from HC3-MGT and HC3-HWT.
Non-purametric Detectors
tp Mcn
AUROC
Total time (8)
R-Detect-k"
1.00+0.00
0.50=0.41
0.99
285.35
R-Detect w/o h-
1.00+0.00
0.82+0.18
0.80=0.37
192.66
MPP-HWT
1.00 00
0.10
0.72=0.20
56 .90
MPP-MGT
0.
0.00z0.00
0.72:0.24
62.59
MPP-R R-Dettcc((Ours)
0.00
0.0120.03
0.99
117.66
1.00=
0.00=z.00
1.00
90.40
HC3-HC3 in Appendix B. The test power tp and AUROC are from Algorithm 2. In default ex- periment setting, the results of test power tp, AUROC and running time are presented Table 2. Specifically, we separate the results of HC3-MGT and HC3-HWT for better understanding.
Test Power on HC3-MGTs. The test power on HC3-MGT 1y equals to the TPR in practice. A larger MD denotes better performance. As is shown in Table 2, most of the non-parametric detectors achieve good performance with regards to tp
Test Power on HC3-HWTss the test power on HC3-HWT 1p"tT equals to the TPR in practice. A smaller tp denotes better performance. According to Table 2, R--ettet-kk" and R-Detect w/o k' have large tp" means that they mistakenly label HWTs as MGTs. MPP-HWT is better than these two, but still has 10% probability of mislabeling HWTs. MPP-MGT has a tp HWT 1% but is unknown whether to choose HWT MGT in the real case. Compared to them, MPP-R achieve 1% FPR and R-Deted archives 0% FPR constantly.
AUROC on HC3. For test- detector, we used 1 p-p-va as the prediction score for calculating AUROC because a smaller p-value means more likely to be labeled as MGT. Compared to the other baselines, R-Detect gets the best AUROC result of 1. 1.00 which indicates that R-Detect can always assign a higher rejection probability for MGTs than HWTs. This has surpassed MPP-R by 1% MPP-MGT or MPP-HWT by 28% 1.
Running Time. We have give more details of machine we used for running all the experiments in Appendix B. Compared to single- side MPP, MPP-HWT or MPP-MGT, R-Detect slightly slower but faster than naive relative MPP version. Overall, the detection is efficiency.
Non-parametric Detectors with Varied Settings. We vary the default setting by 1) changing the default = 0.05 to 0.90; 2) changing the token size from 512 to 256; 3) limiting the length of reference data from all available HC3 expect for the test to the same length with token size. The comparison tables are in Tables 5-8. The result does not differ a lot. Therefore, the non-parametric methods are robust when HWT are from known HWT distribution.
4.2.2 WHEN HWTS FROM UNSEEN HWT DISTRIBUTION
Here, we use the learned kernel function from HC3 to test texts from TQA to mimic MGT-detection when HWTs are from unseen distributions. We test both cases of against old LLM and against newer LLM by using HC3 TCA-Chat-Chh
and HC3-:TQA-GPPT separately (Appendix B). The results are shown in Table 3. We also highlight how the result is different from the result in section 4.2. Table 2 which for seen HWT distribution, using and to denote better or worse performance.
Result of Detection against Old LLMs. HC3 only contains MGT generated by ChatGPT. We found the detection performance does not decrease much when using kernel learned from HC3 to test TQA-MGT which is also generated from ChatGPT, only MPP-MGT slightly worse. However, the baselines have significantly worse performance for identifying TQA-HWT i.e., numan-writter texts in TQA. In addition, comparing MPP-HWT with MPP-MGT, their false positive rate differs lot with each other, where MPP-MGT can achieve a better FPR of 0. but MPP-HWT 0.88. This indicates big uncertainty of MPP's practical use it is unknown whether MGT or HWT should be chosen for the reference in advance. Compared to the result for the case of seen HWT distribution, R-Detect still performs excellently, correctly marking all MGTs and HWTss from TQA.
Result of Detection against Newer LLMs. We also test the performance on GPT4 4egeeereedeed texts in TQA to test if the detection is valid when MGT is generated by newer LLM. In the column (CC3-TQA-GPTE we can see the to MGT does not drop much among baseline methods. This indi- cates the non-parametric method's adaptation capability newer L.Mss.
7

---

Published as a conference paper at ICLR 2025
Table 3: Test power (p) and AUROC on texts to be tested from TQA-ChatGPT and TQA-GPT4.
Non-parametric Detectors
MCC3-TQQ-ChaaGRR
HC3
(1L compared to Table 2)
MGT
AUROC
Mp
6o!WT
AUROC
R-Detect-k:
1.00
1.00
0.92=0.05
1.00
0.76-0.41
+
0.83+20.14
R-Detect w/o k-
1.00ga.00
0.98
0.90=n.03
1.00
0.92=0.18
4
0.66
MPP-HWT
1.00
0.88 4-
0.77_a.17
0.76
0.87
4
1.69
MPP-MGT
0.814 42
0.17
0.73
0,92+0.11
0.83=2.00
0.83+00.99
MPP-R
1.0000.00
0.96
0.52_0.02
1.00
1.00=z0.03
0.50
R-Detecc(OUrss)
1.00 o.
0.00
1.00
1.00 =0.000.00
0.00
1.00
Table 4: Comparison between Parametric and Non-parametric detection results.
HC3-HHC (gI9T
HC3 -TQAA-ChGPPT
HCI HC3--TQA-GPP GPT4
(Token
(yMor
AUROC
(gHor
AUROC
AUROC
time
Bino
1.00g..00
1.00gg0
1.00g000
1.00g.00
1.00lgmm
100gg00
1.0 1,00gp.
1.002000
1.00g0m
1061.36
Bino- HPR
1.00
g:00
1.00g0.00
0.00
1.00ga.00
0.00
0:00
R- (e-Deteet(Curs)
1.00
g.00
1.00
1.00
0.00=a..
1.00
1.00
0.00
0:00::00
1 g.00
180.25
R-Detecfay (Ouns)
20.0
2.00
1.00
0.00
1.00ga.0
2:00
(Taken 250)
HC3
HCS
STQA-ChaGPPT
HC3 XTQA-PPT4
Bino
1000g0.00
1.000.000
1000
1:00s.00
1 1.00ggmm
1000
1.00ga00
1.00gg0
0.902200.00
Rimo-FPR FPR
1. 1.00ga.30
0.000=0.00
1.00sc.o0
0.00-aa
0.00gs.0
0.00:2002
R-Drsect(Ourr)
1.00
2.00
1000
2.00
0.00=am0
1.00
1.00
0.00 2:00
1.00
R-Detectys (Ours)
1.00
0.00 go
g.00
0.00
1.00 1.00 20.00
0.00.0m0
Summary. The experimental results valid our claim that I) recent non non-paramettri detection methods can adapt to different LLMs but 2) have increased false positive rate of mislabeling human-we written text as MGT when it comes from unseen HWT distributions. Our proposed method, R-Detect, can maintain the good adaptation capability of 1) and address the limitation of 2) at the same time.
4.3 COMPARE TO PARAMETRIC METHODS: BINO AS AN EXAMPLE
In section 4.2, we compared R-Detect to a variety of non-parametric baselines. In this section, we present comparison result between our method R-Detect and the state-of-the-ar parametric MGT detector, Bino (Hans et al., 2024). In particular, we compare R-Detect and Bino with varied thresholds and token sizes. In R-Detect, the threshold is the referring the significance level for statistical test while the threshold in Bino, is the classification threshold.
The results are represented in Table 4, As a highlight, the AUROC evaluation does not depend on thresh- old, and thus the same among different thresh- olds. Therefore, we use "_" to denote this number is the same with R-Detectcan always perform very good even with big variation of the threshold, from 0.05 to 0.9. However, compared to R-Detect, Bino's performance is very sensitive the threshold selec- tion. In addition, our method has surpassed Bino on with
too close
Decision Score
05
Bino HWT
Dino MGT
Relative
Relative MGT
AUROC by 0. on the HC3-+QQA-GPP a smaller token size. In addition, R- R-Detectshows a significant advantage with regards to the running time, taking less than 20% time of Bino's.
0.2
00
sample 11
somple #1 somple ni somple //4 sample #5
Figure 2: Decision score difference.
Specifically, we give the detection score for Bino and R-Detect in Figure 2. In R-Detect, we use the 1 p-value of the statistics calculated for relative test as the decision score. A smaller p-value denotes a larger probability of rejecting the null hypothesis. It can be seen, the - p-value is very close to 0 when the ground truth of the text is HWT, and very close to when the ground truth of the text is MGT. This makes HWT and MGT distinguishable with a large range of threshold choices.
The score for bino is a metric based contrasting two closely related language models. Bino will detect a text as MGT when the Bino score is less than a threshold. It can be seen that, Bino will have good result if we choose the threshold between (0.85, 0.9) but might induce ompletely wrong detection on MGT given the threshold larger than 0.85 or completely wrong detection on HWT given the threshold smaller than 0.8 such Bino's bad performance for HWT detection in Table 4.
8

---

Published as a conference paper at ICLR 2025
4.4 CASE STUDY: DETECTING GPT4-REWRITTEN TEXTS
We here show a case study of using our method and Bino on the rewritten texts by GPT4o. We here only present our results against Bino's ' results (Hans et al., 2024), because MPP (Zhang et al., 2024) varies the decision when choosing HWT or MGT. As all the MGTs rewritten texts by the newest GPT4-o model, it is very challenging to distinguish them from their humtenwritten versions.
Decision Score
Bino HWT
Relative HWT Relative MGT
We list the decision score for R-Detect and Bino in Figure 3. It can be seen that although it is less dis- tinct between HWT and rewritten-MGTs. it is fea- sible to find a threshold for R-Detect in the green shallow area to provide detection results with both acceptable TPR and FPR. However, it is less feasible for Bino to find such a threshold to get perfor- mance better than random decision from existing knowledge (see the magenta shallow area, where we have tried the best given the ground truth).
08
somple #1 somple #2
sample #1
samble
sample #5
Figure
Decision score difference on GPT4-rewritten texts.
4.5 PERFORMANCE EVALUATION ACROSS DIVERSE BASELINES AND BENCHMARKS
The analysis in section 4.2 demonstrates that, both testing based methods, R-Detect offers a unique advantage over MPP (Zhang et al., 2024). Meanwhile, the analysis in section 4.3 aims to explain the benefits of non-parametric detectors compared to parametric ones. In this section, extend our experimental analysis to more diverse and challenging settings: 1) MGTs are generated by multiple L.MMs; 2) the text data may contain adversarial attacks; 3) the text data is written in language other than English. The experimental results, analyzed in Appendix E, demonstrate that:
R-Detect achieves the best performance in multi-LLM-generateed texts (E.
R-Detect is robust against various adversarial attacks (E.1);
While our R-Detect not always the best in mon-English detection tasks in zero-shot setting, detection effectiveness improves significantly with a properly optimised kernel (E.2).
Using advanced feature extractors significantly improves R-Detect's performance (E.3).
RELATED WORKS
5.1 LLM-EMPOWERED MGTs: CONCERNS AND SOLUTIONS
Large language models such ChatGPT (Schulman al., 2022), Google's LaMDA (Thoppilan et al., 2022), Meta's OPT (Zhang et al., 2022), LLaMa (Touvron et al., 2023), and Falcon (Al- mazrouei al., 2023), trained enriched human text data, capable of generating natural, flu- ent, and high-quality content. Their usage has surged dramatically due to easy public access (Watch, 2023); for example, since its launch in November 2022. monthly visits to ChaiGPT have increased 15-fold (Singh, 2024). However, the increasing indistinguishabilty of LLM-generated texts from human-written content has raised growing concerns about their misuse (Weidinger et al., 2022), in- cluding phishing attacks (Hazell, 2023), disinformation (Zellers et al., 2019; Adelani et al., 2020), plagiarism (Lee al., 2023: Stokel- Walker, 2022), and other ethical risks (Weidinger et al., 2021).
As humans can be easily deceived by MGTs (Ippolito et al., 2020; Zellers et al., 2019), developing effective MGT detectors is seen as a significant step toward ensuring the responsible use of gener- ative language models (Dhaini et al., 2023). In terms of detection design prepared watermarking methods (Kirchenbauer al., 2023; 2024; Yang et al., 2024b) aim to modify the distribution of generated text in pre-designed manner, but this study does NOT focus on such methods. Instead, we focus on post-hoc detection, which assumes no interaction during the text generation process (Chakraborty al., 2024). Serving tools rather than the ultimate goal. finer- tineiegrained MGT detec- tion is recommended to ensure the ethical use of LLMs (Kumar et al., 2023), posing new challenges for interpretable detection mechanism and explainable detection results.

---

Published as a conference paper at ICLR 2025
5.2 PARAMETRIC POST-HOC MGT DETECTION
Recent studies introduce an amount of post-hoc detection methods, such as Fast-DetectGPT (Bao al., 2024), DNA-GPT (Yang et al., 2024a), MPP (Zhang al., 2024), MPU (Tian et al., 2024), Binoculars (Hans et al., 2024), and others (Soto et al., 2024). A parametric detection method as- sumes MGT is an output of generative language model with specific parameters. Consequently, the detection performance is inherently tied to the particular type of MGT, limiting their adaptation capabilities across different generative language models. In general, current parametric methods can be divided into two categories: metric-based and classifier-based approaches.
Metric-based MGT Detection. These methods leverage pre-trained LLMs or scoring models to measure the statistical discrepancies between HWTSs and MGT's Commonly used metrics include log-likelihood (Solaiman et al., 2019), entropy (Ippolito et al., 2020), rank (Gehrmann et al., 2019), log-rank (Su et al., 2023), N-Gram (Yang al., 2024a), and log probability (Mitchell et al., 2023; Bao et al., 2024; Hans et al., 2024). Since these metrics are often derived from pre-trained LLMs, they can facilitate zero-shot detection with proper design (Bao et al., 2024; Hans al., 2024). How- ever, these metric-based detection methods tend to suffer from inferior performance when there is significant domain gap between the language of the generated text and the scoring model.
-lassifier-based MGT Detection. Classifier methods typically involve training a classifica- tion model using both HWTs and MGTs (Mitrović et al., 2023). For example, OpenAI-D (Solaiman al., 2019) fine-tunes RoBERTa model on GPT-2-generated texts for detecting GPT-2 outputs. ChatGPT-D (Guo et al., 2023) employs two strategies (using either pure answered text or QA pairs) train the model with the HC3 dataset. OpenAl has recently fine-tuned a GPT model (Kirchner et al., 2023) using data from Wikipedia, Web'Text, and human evaluations to develop web interface for a discrimination task involving texts generated by different language models.
5.3 NON-PARAMETRIC POST-HOC MGT DETECTION
non-parametric approach, MPP proposed to use two-sample test for MGT detection (Zhang et al., 2024). MPP optimises the kernel function in MMD (Gretton al., 201 2012a; Liu al., 2020) to determine whether the distribution of an unauthored text differs from that of reference text. It assumes that HWTs and MGTs follow distinct distributions, regardless of how MGTs are generated, allowing it to adapt easily to MGTs generated by newer LLMs, i.e., optimising kernels with CharGPT-generraed texts (OpenAI, 2022) while testing on GPT- Neo (Black et al., 2021), GPT-j-6b (Wang & Komatsuzaki, 2021), and GPT4all-j (Anand et al., 2023). Compared to parametric et al., 2019; Mitchell et al., 2023) or classifier-basse (Solaiman et al., 2019; Guo et al., -MPP demonstrates superior detection performance (Zhang al., 2024).
Two-sample Test-based MGT Detection.
wwhthhr metric-based (Solaiman et al., 2019; Gehrmann
CONCLUSION
This paper presents machine-generated text detector, R-Detect, that has been empirically proved, with extensive experiments, to have a small false alarm rate and a high successful rate in detecting texts generated by representative large language models (LLMs).. Specifically, R-Detect is a non- parametric tool and does not require any knowledge regarding how the texts are generated by LLMs (i.e., we do not involve any LLM -baeed inference), which is suitable for many scenarios where only the information regarding texts is available. In addition, we also empirically compare R-Detect with existing detection tools in the literature, and the results show that R-Detect achieve state-of-the- art performance in terms of both of false alarm rate and detection rate. Notably, R-Detect can even outperform parametric detection tools that need to access LLMs or surrogate LLMs, which further justifies the significance of R- R-Detect, especially when using it in real-world applications.
ACKNOWLEDGMENTS
ZQY and FL supported by the Australian Research Council (ARC) with grant number DE240101089, and FL is also supported by ARC with grant number DP230101540 and the NSF&CSIRO Responsible AI program with grant number 2303037. This work was supported by the
10

---

Published as a conference paper at ICLR 2025
National Natural Science Foundation of China No. 62125201. We thank Xiao Yu for implementing next-Pluoroscopy.
ETHIC STATEMENT
Our work follows the ICLR Code of Ethics. All data used in our paper anonymized, eliminating any potential privacy concerns. There is no human or animal subjects to involved in our paper. During the experiments, would not cause bias or other ethical issues. The proposed tool is designed to be transparent and re- producible, and the code will be released for public use and supporting the open-source community.
strictly followed the ICLR Code of Ethics and made sure that this paper
REFERENCES
David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. Generating pensement-preserving fake online reviews using neural language models and their human-and machine- detection. In Advanced information networking and applica- tions: Proceedings of the 34th international conference on advanced information networking and applications (AINA-2020) pp. 1341-1354. Springer, 2020.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL
org/ abs 16867.
Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. GPT4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. GitHub, 2023.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2014.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Eff- cient zero-shot detection of machine- machine-generated text via conditional probability curvature. In The Twelfth International Conference on Learning Representations, 2024.
Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorfiee March 2021. URL https://doi.
org/10..288 zenodo. zenodd.529775 5297715.
Wacha Bounliphone, Eugene Belilovsky, Matthew B. Blaschko, Ioannis Antonoglou, and Arthur Gretton. A test of relative similarity for model selection in generative models. Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: //arxiv. abs/ /11111 04581
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Souradip Chakraborty, Amrit Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. Position: On the possibilities of ai ti-generated text detection. In Forty-firt International Confer- ence on Muchine Learning, 2024.
Doraid Dalalah and Osama MA Dalalah. The false positives and false negatives of generative ai detection tools in education and academic research: The case of ChatGPT. The International Journal of Management Education, 2102):00082, 2023.

---

Published as a conference paper at ICLR 2025
Mahdi Dhaini, Wessel Poelman, and Ege Erdogan, Detecting ChatGPT: A survey of the state of de- tecting ChatGPT-generatee text. In Momchil Hardalov, Zara Kancheva, Boris Velichkov, Ivelina Nikolova-Koles Koleva, and Milena Slavcheva (eds.), Proceedings of the 8th Student Research Work- shop associated with the International Conference Recent Advances in Natural Language Pro- cessing, pp. 1-12, Varna, Bulgaria, September 2023. INCOMA Ltd., Shoumen, Bulgaria. URL https: / acl.athcannntoyyyccccoo org/ 202g/2023..0 ranlp- ran.p-stud. stud.
Liam Dugan, Alyssa Hwang, Filip Trhlik, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Bured RAID: A shared benchmark for robust evaluation of machine- generated text detectors. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12463-12492, Bangkok, Thailand, August 2024, Association for Computational Lin- guistics. doi: 10. 100188663/11/02224llllooozg acl- 674/.
URL https: http:///aaaaaaah//eeeeeeee org/2024.
Holly Else. Abstracts written by ChatGPT fool scientists. Nature, 613:423, 2023.
Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In International Conference on Machine Learning, pp. 3564-3575. PMLR, 2021.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visu- alization of generated text. In Marta R. Costa-jussà and Enrique Alfonseca (eds.), Proceedings of the 57th Armual Meeting of the Association for Computational Linguistics: System Demonstra- fions, pp. 111-116, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-3013 URL https:/ /aclllll anthoa wwooollyy...gggg................ www.thhollgy..gg///1.ooe. wrtg/19-33/0000
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012a. 20122.
Arthur Gretton, Bharath K. Sriperumbudur, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakr- ishnan, Massimiliano Pontil, and Kenji Fukumizu. Optimal kernel choice for large-scale two- sample tests. In NeurIPS, 2012b.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yu- peng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.075:: 2023.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting llms with binoculars: Zero-shot detection of machine-generate text. In Forty-firs International Conference on Machine Learn- ing, 2024.
Julian Hazell. Spear phishing with large language models, 2023. URL Utt.httss://arxxiv.o abs/ / 2205. 06972.
Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. MGTbench; Benchmark- ing machine-generater text detection. arXiv preprint arXiv:2303. 14822, 2023.
Xiaomeng Hu, Pin- Yu Chen, and Tsung- Yi Ho. Radar: Robust ai-text detection via adversarial
learning. Advances in Neural Information Processing Systems, 36: 1507- 2023.
Daphne Ippolito, Daniel Duckworth, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808-1822, 2020.
Alberto
Insights into the area under the receiver operating characteristic curve (auc) as a discrimination measure in species distribution modelling. Global Ecology and Bio- geography, 21 21(4):498-50) 2012.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
12

---

Published as a conference paper at ICLR 2025
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, pp- 17061-17084. PMLR, 2023.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Feraando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of water- marks for large language models. The Twelfth International Conference on Learning Repre- sentations, 2024.
Jan Hendrik Kirchner, Lama Ahmad, Scott Aaronson, and Jan Leike. New ai classifier for indicating ai-written text. OpenAl. openai. combbloghnev-ii-clasffieeerr cawtoooriiiiiiinn-iiiii-- -irttgg----iiite-eeee 2023.
Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. Language generation models can cause harm: So what do about it? actionable survey- In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- fional Linguistics, pp. 3299-3321, 2023.
Thomas Lancaster. Artificial intelligence, text generation tools and ChatGPT-does digital water- marking offer a solution? International Journal for Educational Integrity, 19(1):10, 2023.
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pp. 3637-3647, 2023.
Tong King Lee. Artificial intelligence and posthumanist translation: ChatGPT versus the translator. Applied Linguistics Review, 0, 2023.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3214-3252, 2022.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica Sutherland. Learn- ing deep kernels for non-parametrie two-sample tests. In International Conference Machine Learning, pp. 6316-6326. PMLR, 2020.
Feng Liu, Wenkai Xu, Jie Lu, and Danica Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. In Advances in Neural Information Processing Systems, volume 34, pp. 5848-5860, 2021.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Yinhan Liu. Roberta: robustly optimized bert pretraining approach. arXiv: 1907. 1692, 2019.
arXiv preprint
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt factual inconsistency evaluator for text summarization. arXiv preprint arXiv:2303.1562/, 2023.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher Manning, and Chelsea Finn. De- tecIGPT: Zero-shot machine-generatee text detection using probability curvature. In International Conference on Machine Learning, pp. 24950-24962. PMLR, 2023.
Sandra Mitrović, Davide Andreoletti, and Omran Ayoub. Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chaegpt-generated text. arXiv preprint arXiv:2301.13855 2023.
OpenAl. Introducing ChatGPT. Website, 2022. chatgpt.
URL
OpenAl. Hello gpt-4o. http:://ppenii.cc////ee////ll// 2024. Accessed: 2024-09-29.
2024.
13

---

Published as a conference paper at ICLR 2025
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. ChatGPT: Optimizing lan- guage models for dialogue. OpenAl blog, 2(4), 2022.
Shubham Singh. ChatGPT statistics (sep. 2024) 200 million active users. https:
ttt.eaaeetattttttttttttoaaaaar ttw.tttChatttrs20aaaaa.. 2200eeee000000000111112eeeeeeee2 http:///c....ct.rooooooossssaaa0 by 207. 200.15500n1llo333ppppeeeeeeeeee2
//www.
2024. Accessed: 2024-09-23.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Berbert-Vosss Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: 09203, 2019.
Rafael Alberto Rivera Soto, Kailin Koch, Aleem Khan, Barry Chen, Mareus Bishop, and Nicholas machine-generated text using style representations. In The
Andrews. Few-shot detection of machine- generated Twelfth International Conference on Learning Representations, 2024.
Chris Stokel- -alker. AI bot ChatGPT writes smart essays-should academics worry? Nature, 2022.
Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. DetectLLM: Leveraging log rank in- formation for zero-shot detection of machine-generated text. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12395-12412, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653//1/200 1/0023.2222//// htt.://////2222..nnnnneemm....p findings-emnlp.827. URL https: //aaaaaaannttooooooeooooea org/2023. findings emp smmlp.........mmmm.mmmm 827.
Teo Susnjak and Timothy McIntosh. ChatGPT: The end of online exam integrity? Education Sciences, 14(6):656, 2024.
Danica J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In ICLR, 2017.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kuishreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.0822 2022.
Yuchuan Tian, Hanting Chen, Xutao Wang, Zheyuan Bai, QINGHUA ZHANG, Ruifeng Li, Chao Xu, and Yunhe Wang. Multiscale positive-unlabeled detection of ai-generated texts. In The Twelfth International Conference on Learning Representations, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghost- written by large language models. Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume Long Papers), pp. 1702-1717, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 18.18v1//v1/ 2024.naacl- ma2c-nnacl--on.955 URL https: aaclantooooooo org/ 2224.024222 www.-naallllno
95.
Luke Vilnis, Yury Zemlyanskiy, Patrick Murray, Alexandre Tachard Passos, and Sumit Sanghai. Arithmetic sampling: Parallel diverse decoding for large language models. In International Con- ference on Machine Learning, pp. 35120-35136. PMLR, 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language May 2021.
Model. httpps://ggthuu...t/ com flw.m/knnofllllloo z
Public Sector Tech Watch. ChatGPT in the public sector overhyped or over- looked? Technical report, European Commission, 2023. URL https://
ec wuropa. www.://pp....ooollleeoooooaooaaa oht.gee...pollleeeerrcooooooooeo
wtthhhhhttthttess//// ooked. Accessed: 2024-09-23.
14

---

Published as a conference paper at ICLR 2025
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXin:2112.04359, 2021.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang. John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 214-229, 2022.
Junchao Wu, Runzhe Zhan, Derek F. Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia S. Chao. Detecteval: Benchmarking llm-generated text detection in real-world scenarios. In Proceed- ings of the Neural Information Processing Systems Track Datasets and Benchmarks, NeurlPS Datasets and Benchmarks 2024, 2024.
Xianjun Yang, Wei Cheng, Yue Wu, Linda Ruth Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. In The Twelfth International Conference on Learning Representations, 2024a.
Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Ruth Petzold, William Yang Wang, and Wei Cheng. A survey on detection of LLMs-generated content. Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2024, pp. 9786-9808 Miami, Florida, USA, November 2024b. httss///allattthhlc org/2024.
Association for Computational Linguistics. URL Aindings-mnnll 572.
Xiao Yu, Kejiang Chen, Qi Yang, Weiming Zhang, and Nenghai Yu. Text fluoroscopy: Detect- ing LLM--gnerated text through intrinsic features. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, pp. 15838-15846, Miami. Florida, USA, November 2024. Associa- tion for Computational Linguistics. doi: 10. N0.18653/w1/2222.emml www.eemmmllmmina...... URL https:
//aclanthology. org/ 2024. eennp-mminn... 885/.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Cong Zeng, Shengkun Tang, Xianjun Yang, Yuanzhou Chen, Yiyou Sun, Yao Li, Haifeng Chen, Wei Cheng, Dongkuan Xu, al. Improving logits- based detector without logits from black-box 1lms. In Proceedings of the Neural Information Processing Systems, 2024.
Shuhai Zhang, Yiliao Song, Jiahao Yang, Yuanqing Li, Bo Han, and Mingkui Tan. Detecting machine-generatee texts by multi-population aware optimization for maximum mean discrepancy. In The Twelfth International Conference on Learning Representations, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
CONTENTS
Empirical Estimation of 0yzx2
17
More details for Experiement Settings
17
B.1 Datasets Introduction
17
B.2 Data shuffle for Detection in section 4.2 and section 4.3
17
B.3 Data Shuffle for Detection in section 4.5
18
B.4 Introduction of Baselines
18
B.5 Implementation Details of Our Method
18
B.6 Implementation Details of Baselines
18
Non- par-parametric Detectors with Varied Settings
19
Toy Study: The Impact of Overlap Ratio
19
E Experimental Analysis across Diverse Baselines and Benchmarks
20
E.1 English Text
20
E.2 Non-English Text
21
E.3 Ablation Study on Feature Extractor
22
Five Essays and Re-written Texts GPT4o
22
16

---

Published as a conference paper at ICLR 2025
A
EMPIRICAL ESTIMATION OF σYzXZ
An empirical estimation of zxz (Bounliphone et al., 2016) is:
2
KezKzze
m(m - -1 Kzze
ZYZ m(m - 1)2
1
KrnKxze
Reeee Kxxe)
m(m 1)r
m²(m 1)r
1
1
e. Keakyyee mz - 1)m m(m - 1)n
e "Zzeee Kxze
KyzzKxze m2ar"
Kyzee Kxze
mar
where e is a vector of ones with length equal to the number of samples from 2, and Kez, Kyn, and Kxz are kernel matrices (Gretton et al., 2012a). The elements of Kxx are defined as [xxlis
if J,
[xxcls = 0
if ifi=j.
Similar definitions apply to Kyy and Kez.
B MORE DETAILS FOR EXPERIEMENT SETTINGS
B. DATASETS INTRODUCTION
We here use HC3 (human ChatGPT comparison corpus) (Guo et al., 2023), which contains 24, 321 paired answers from human and ChatGPT (OpenAI, 2022) with both long and short-level corpus; TruthfulQA (He et al., 2023; Lin et al., 2022), which comprises 817 questions from human (we here use best human answer), ChatGPT (OpenAI, 2022) and GPT4 (Anand et al., 2023). RAID (Dugan al., 2024) is one of the largest MGT detection benchmark. It includes over 6 million generations spanning 11 models, 8 domains, 11 adversarial attacks and 4 decoding strategies. De- tectRL (Wu et al., 2024) is another recent released MGT detection benchmark. It comprises 100,800 humtn-written samples, including 1 11,200 raw samples and 89,600 samples modified via attack ma- nipulations. Additionally, it contains 134,400 samples generated by LLMs, categorized follows: 11,200 samples generated with direct prompt, 22,400 400 with prompt attacks, 33,600 with paraphrase attacks, 33,600 with perturbation attacks, and 22,400 with data mixing.
B.2 DATA SHUFFLE FOR DETECTION IN SECTION 4.2 AND SECTION 4.3
During each round of detection in section 4.2 and section 4.3, we first shuffle the HC3 dataset and select the first 512 tokens from HWTs and the first 512 tokens from MGT's as the text to be tested (the token number will be 256 in the token-256 experiments). The default reference data will be the rest of the data. We also test reference data with the same length as the text to be tested, i.e., 512 or 256 tokens. For each experiment, the dataset will be as follows:
the HC3-HHC experiment, we run our method and all baselines and save their detection result for this shuffle at this round. Specifically, will shuffle 10 times in each round, where 5 shuffles select the text to be tested from HWT and the other 5 shuffles select the text to be tested from MGT. We have 10 rounds in total. Namely, dataset is 10 x 10 for detection on 512-token texts and another 10 x 10 for detection on 256-token texts.
In the WC3-+TQA-ChaCCS iments) from TQA-ChaIPPT as the text to be tested. We run our method and all baselines and save their detection result for this shuffle at this round. Specifically, we will shuffle 10 times in each round, where shuffles select the text be tested from HWT and the other shuffles select the text to be tested from MGT. We have 10 rounds in total. Namely, our dataset is 10 x 10 for detection on 512-token texts and another 10 10 for detection on 256-token texts.
experiment, we select 512 tokens (or 256 in the token-256 exper-
the HC3-*TQA-GE -TQA-GPT4 experiment, we select 512 tokens (or 256 in the token- 256 experiments) from TQA-GPT4 the text to be tested. We our method and all baselines and save their
17

---

Published as a conference paper at ICLR 2025
detection result for this shuffle this round. Specifically, we will shuffle 10 times in each round, where 5 shuffles select the text to be tested from HWT and the other shuffles select the text to be tested from MGT. We have 10 rounds in total. Namely, our dataset is 10 x 10 for detection 512-token texts and another 10 x 10 for detection on 256-token texts.
B.3 DATA SHUFFLE FOR DETECTION IN SECTION 4.5
Unless otherwise noted, we use the same way as the above section to choose the reference data. Namely, our reference data is always from HC3 dataset.
B.4 INTRODUCTION OF BASELINES
We compare R-Detect to five non-parametric baselines and seven parametric baselines. The non-parametric baselines include two variations of our method 1) R-Detect-k-" R-Detect with the Gaussian kernel optimised by median heuristic bandwidth (Bounliphone et al., 2016) and 2) R-Detect w/o k", namely our method without kernel optimisation.
The non- pon-mmametriic baselines include 3) MPP-HWT the detector using HWT as the reference data, 4) MPP-MGT, is the MPP detector using MGT as as the reference data, and 5) MPP-R, method that proposed based on MPP, which uses both MGT and HWT for MPP's reference. The detection result will be given based on the hypothesis test with smaller p-value from these two.
The parametric basclines include 6) DetectGPT (Mitchell et al., 2023), 7) Radar (Hu al., 2023), 8) Dast-DetectGPTT (Bao et al., 2024). 9) DNA-GPT (Yang al., 2024a), 10) Bino, the state-of-the art parametric detector (Hans al., 2024) using 0.5 as the classification threshold; 11) Bino-FPR, the state-of-the-art parametric detector (Hans et al., 2024) using the threshold that especially for a low false positive rate, 12) DALD (Zeng al., 2024), and 13) Text-Fhoroscopy (Yu al., 2024).
B.5 IMPLEMENTATION DETAILS OF OUR METHOD
The deep kernel k in R-Detect is neural network & equipped with feature extractor g. We learn the best kernel from HC3 data and used it for all test on HC3, TQA- and TQA-GPT4. The kernel is also used for MPP implementation to ensure a fair comparison.
Following the setting from (Zhang et al., 2024), the feature extractor 9. we employ OpenAl's RoBERTaa-ased GPT-2 detector model (Liu, 2019) and consider its last hidden state the feature of the input text. Each token in this feature extractor has a dimension of 768, and we set a maxi- mum of 100 tokens per sentence. The network k. consists of hidden-layer transformer followed by projector and multi-layer perceptron (MLP), where the projector reduces the data dimension from 768 to 512, while the MLP reduces the flattened data dimension from 51, 200 300. The data dimension during the whole procedure when feeding sentence into the kernel follows the se- quence: 100x 1188~110xxx/ 512--5511551111 151 200-+300. Note that we only optimize the network & and fix the mapping function during training through all our experiments.
We conduct our experiments using Python 3.9 and Pytorch 2.0 on a server with Intel Core i9 14900K and RTX 4090. In Algorithm 3, we use Adam optimizer (Kingma & Ba, 2015) to optimize the deep kernel parameters, we set À to 10-8 and batch size to 200, and the learning rate to 0.00005 in all experiments. The default threshold of the hypothesis test-both two-sample test or relative test-is a = 0.05 to determine whether to reject or accept the null hypothesis. We also give the result when using different
B.6 IMPLEMENTATION DETAILS OF BASELINES
MPP-HWT, MPP-MGT, and MPP-R: For the non-paramen non-parametric detectors, MPP-HWT, MPP-MGT, and MPP-R, we applied the default parameter used in (Zhang et al., 2024) for HC3 data, which is the same with the setting of our method in section B.5.
Bino, Bino-FPR: For the parametric detectors Bino and Bino-FPR, the authors suggested, namely using Falcon-7B and Falcon- 7 IIIntrctct models for scoring, the classification threshold 0.5 for Bino, and (.8536433331088555) for Bino-FPR.
the default settings as
18

---

Published as a conference paper at ICLR 2025
R- R-Detect-k"". R-Detect with the Gaussian kernel optimised by median bandwidth (Bounliphone et al., 2016) where the kernel bandwidth is from median heuristic.
R-Detect w/o kn We use Gaussian kernel with its width
Table 5: Test power (p) and AUROC results texts be tested from HC3-MGT and HC3-HWT (a = 0.9).
Non-parametric Detectors
ty
MGT
1p
HWT t222
AUROC
AUROC. 0.9
MPP-HWT
1.0020.00
1.00
0.10
0.10
0.72.0.22
0.72
MPP-MGT
0.90=a.15
0.90+0.10
0.00+a.00
0.00
0.72+0.24
0.72+0.24
MPP-R
1.00g0.00
1.00
0.01
0.01
0.99
0.00
Rt (B-Detec((OUrs)
4..0
1.00
0.00
0.00=0.00
1.00
1.00
Table 6: Test power (p) and AUROC results on texts be tested from HC3-MGT and HC3-HWT (token size of 256 X(and Jare compared to the default setting).
Non -oon-aarametric Detectors
fpp=250
AUROC
AUROC,
MPP-HWT
1.0020.00
1.00
0.10
0.10
0.72g0.26
0.75 22
MPP-MGT
2.
0.90
0.00 oo
0.00
0.7210.24
0.68
MPP-R
1.00
1.00=0.00
0.01 os
0.02=a.os4
0.99
0.98+0.01
R. (o-retec((Ours)
1 1.00
1.00
0.00
0.00
1.00g0.00
1.00
Table 7: Test power (p) and AUROC results on texts to be tested from HC3-MGT and HC3-HWT (a- 0.9, / 256) (fand Jare compared to the default setting).
Non-parametric Detectors
AUROC
AUROC.
MPP-HWT
100
1.00
0.10
0.10
0.7220.20
0.75+0.22 1
MPP-MGT
0.90
0.90 10
0.00
0.00
0.72
0.68
MPP-R
1.0010.00
1.00
0.01 L..m
0.01 10:00
0.99 0.99
0.98 |
R-Detect(Ours)
1.00
1.00g0:00
0.00
0.00
1.00_0.00
1.00.000
C NON-PARAMETRIC DETECTORS WITH VARIED SETTINGS
We here present the experiments of using the non- non- parametric detectors, especially MPP-HWT, MPP- MGT, MPP-R different experimental settings such as 1) changing the default a = 0.05 to a = 0.9 2) changing the token size from 512 to 256; 3) limiting the length of reference data from all available HC3 expect for the test to the same length with token size.
The result of 1) is presented in Table 5, with left column in each combined columns the default setting of a = 0.05. Similarly, we have result 2) in Table as well as the result for cross changes of both in Table 7. In addition, from the rest of HC3 to portion that has the same length with the text to be tested. Namely, in the experiments of 512 the token number, we two HWT and MGT within 512 tokens as the reference for the hypothesis testing. While the experiments of 512 the token number, we use two HWT and MGT within 256 tokens as the reference for the hypothesis testing
also shorten the length of the reference data, decreasing it
D Toy STUDY: THE IMPACT OF OVERLAP RATIO
As noted section 2.1, we consider only the case where texts belong exclusively to either Sh or Sm, given that f is assumed to be surjective function. However, in hypothesis testing such as the relative test, it is possible for Sh n Sm * 0. Therefore, it is important to examine the test power (i.e., the probability of rejecting the null hypothesis when the alternative hypothesis is true) in cases where Sn and Sm overlap. To investigate this, we conduct a toy study by manually increasing the distance μ between the distributions of P and Q i.e., the mathematical reflection of decreasing the overlap ratio, The results, presented in Table 9, indicate that the test power increases as the overlap between S, and S,, decreases.
19

---

Published as a conference paper at ICLR 2025
Table 8: Test power (p) and AUROC results when limits reference length token size (fand Jare compared to the default setting).
(1Jcompared default setting)
Non- Non-parametric Detectors
ty
tp!WT
AUROC
AUROC,
MPP-HWT
1.00 on
1.00go.00
0.10
0.27 43 4
0.7220.20
0.72
MPP-MGT
0. 15
0.93 16
0.02+n.04 0.00
0.72+00.2
0.72=0.24
MPP-R
1.000
1.00.a..0
0.01 +0.08
0.01 =o.0a
0.99
0,98 !
(O-retec(OOurs)
1. 1.00 oo
1.00 oo
0.00
2.00.0..0
1.00
1.00
Table 9: Test power p variations over distribution distance
0.20
0.22
0.24
0.26
0.28
0.3
0.32
0.34
0.36
0.38
0.40
0.83
0.82
0.82
0.83
0.86
0.89
0.92
0.95
0.97
0.98
0.99
E EXPERIMENTAL ANALYSIS ACROSS DIVERSE BASELINES AND BENCHMARKS
E. ENGLISH TEXT
We conducted more experiments across diverse baselines and benchmarks, and the detection results are shown in Table 10. The results show good performance of R-Detect over different benchmarks and under variety of random added attacks. In these experiments, R-Detect is conducted in a zero-shot setting, namely using the same kernel and reference data from HC3 without further fine- tuning. Two sets of the token size have been picked, #256 and #512 12, noting that DetectGPT cannot run in the token#512 12 setting. The data shuffle for each column results is as follows:
RAID-English: we here only test on the English data RAID, considering all model options in [ChatGPT, GPT-4, GPT-3 (text-davinci-O0 GPT-2 XL, Llama 2 70B (Chat), Cohere, Co- here (Chat), MPT-30B, MPT-30B (Chat), Mistral 7B, Mistral 7B (Chat)]; and all adversarial attack options in [Article Deletion, Homoglyph, Number Swap, Paraphrase, Synonym Swap, Misspelling, Whitespace Addition, Upper-Lower Swap, Zero- Width Space, Insert Paragraphs, Alternative Spelling).
During the data shuffles, we first use random seeds (Table 12) to shuffle the model options and attack options. Given the shuffled results, we randomly pick text from RAID with the shuffled settings. More specifically, each combination for the test data is as follows:
mixl represents a mixture of MGTs generated by Mistral 7B (Chat), GPT-2 XL, MPT-30B, MPT-30B (Chat), and GPT-4. without attacks;
Table 10: AUROC results on RAID (Dugan et al., 2024) and DetectRL (Wu et al., 2024) benchmarks over diverse baselines (the best detection performance is highlighted in bold.)
Detectors
BAID-English [ACL/24]
DetectRL Engllssh [Neur1PS'24 24|
(loken 4256)
mix1
mix3
non--att
I-attl
b-att2
p-attl
p-att2
MPP [ICLR24]
0.7379
0.7032
0.8504
0.7104
0.5646
0.6952
0.6917
0.8021
0.7804
0.6374
0.6199
DetectGPTT [ICML23]
0.6437
0.6632
0.4987
0.5931
0.5111
0.4554
0.5319
0.9556
0.6631
0.4500
0.5031
Radar [NeurIPS23]
0.8512
0.8124
0.7592
0.7988
0.5M80
0.5964
0.8700
0.8328
0.7016
0.6964
0.9068
Fast. Dest-Deee [JCLR24]
0.8400
0.8104
0.6320
0.4356
0.4124
0,3936
0.8768
0.9708
0.7296
0.6652
0.9024
DALD [NeurIPS24]
0.8284
0,7872
0.5324
0.9032
0.8196
0.9420
DNA-GPT [ICLR224]
0.8108
0.3616
0.3176
0.7144
0.2064
0.3944
0.6708
0.8428
0.7780
0.9064
0.7088
Texo-Flunosyy [EMNLP24]
0.9780
0.7196
0,7974
0.7664
0.5503
0.6771
1.00000
0.9812
0.8306
1.0000
1.00000
R-Detect[ours]
0.9996
0,8240
0,8808
0.9528
0.5684
0.7376
0.9964
0.9820
0.8692
0.9952
1.0000
(loken M512)
mix1
mix2
mix3
attl
nt12
at13
non-att
h-attl
b-att2
p-attl
p-att2
MPP(ICL.R24]
0.6919
0.6531
0.7431
0.6409
0.6497
0.5671
0.6750
0.7490
0.7541
0.6741
0.5533
DetectGPT [[CML.2]
Radar [NeurIPS23]
0.8948
0.9792
0.8792
0.8268
0.7432
0.7384
0.9720
0.6992
0.7916
0.9916
0.9732
Fast. D11ccGGTT
0.8276
0.9704
0.9972
0.9092
0.9904
0.8180
1.0000
1.0000
0.8408
1.0000
0.8528
DAI NNeurPP524)
0,8400
0.9792
0.9244
0.7284
0.8568
0.9068
0.9828
0.9492
0.7364
0.9764
0.5832
DNA GPT]ICL22]
0.7764
0.6288
0.7608
0.7960
0.7412
0.8396
0.7664
0.7376
0.7844
0.8672
Text- (oteFPlooossoyyEENMLPP1
0.9824
0.8704
0,7728
0.8416
0.8164
0.4600
0.9936
0.9546
0.9632
0.9804
0.8976
ReDesec[[uur)]
1.0000
0.9688
0.9944
0.9908
0.9996
0.9184
1,0000
0.9996
0.8208
1.0000
0.9596
20

---

Published as a conference paper at ICLR 2025
Table 11: AUROC results on RAID-Non English texts over non-parametric baselines ("and Jare compared to the default setting of zero-shot using token #256).
Detector (token #256)
setting
N/Emixl
N/Eatt1
N/Emix2
N/Eatt2
MPP
zero-shot
0.4717
0.4321
0.4496
0.3878
R-Detect[ours]
zero-shot
0.3048
0.5616
0,3504
0.4476
MPP
kernel
0.55811
0.5426
0.46651
0,4316
[-Detect[ours]
kernel
0.60521
0.7818
0.4272
0.3648
MPP
(kernel + reference)
0.61411
0,4344
0.46821
0.57331
R R-Detect[ours]
new (kernel + reference)
0.9490
0,8048
0.7704
0.5392
Detector (token #512)
setting
N/Emixl
N/Eatt1
N/Emix2
N/Eatt2
MPP
zero-shot
0.48791
0.5605
0.56051
0.49931
R-Detect[ours]
zero-shot
0.7770
0.5236.
0.52361
0.52481
mix2 represents a mixture of MGTs generated by Cohere, Cohere (Chat), GPT-3 (text- davinci-003), and GPT-2 XL without attacks;
mix3 represents a mixture of MGTs generated by MPT-30B, GPT-3 (text-davinci-0033,
GPT-2 XI., and GPT-4. without attacks;
attl represents mixl under random attacks from the attack options;
aatt2 represents mix2 under random attacks from the attack options;
-ttt3 represents mix3 under random attacks from all the attack options.
DetgssR--English: DetectRL contains English texts only. We here on the DetectRL, consid- ering all LLMs and both human adversarial attacks by "replacing one-quarter of the sentences an LLM LLM--gnerated text with human-w -written text at random" or prompt attacks by "intended to use carefully designed prompts to guide LLMs in generating text that closely mimics human writing style":
non-att: represents random sample from non-attacked DetectRL texts;
h-attl and h-att2: represents random samples from DetectRL texts under human adversar-
ial attacks;
p-attl and p-att2:
represents random samples from DetectRL texts under prompt attacks;
Table 12: Seed for Shuffle RAID (Dugan et al., 2024) and DetectRL (Wu al., 2024)
RAID-Enae EnnD-E [ACL/24]
DetectRL-English [NeurIPS'22
Seed
mix1
mix2
mix3
attl
att2
att3
non-att
h-att1
h-att2
p-att1
p-att2
(loken #256)
5
3
5
3
2
(token #512)
5
1
0
5
0
1
3
3
6
7
E.2
TEXT
We use the non-English text from RAID test R-Detect in the non-English case with and without attacks. Similar to the shuffle strategies E.I, we shuffle the model and attack options before random sampling-N/Eann is the text of N/Emix1 with attacks. We run the experiments under three settings:
zero-shot setting is the default setting of using the kernel trained from HC3 and using HC3 data as the reference;
new kernel refers the setting that uses the -on-English data train a new kernel function and test it on unseen data. During the testing procedure, we still use HC3 shuffles as the reference;
new (kernel+reference) refers to the setting that uses the non-Enge non-English data to train a new kernel function and test it unseen data. During the testing procedure, shuffle samples from the non-English data used for training as the reference.
It be concluded from comparing the results among settings that Although method shows suboptimal performance in non- oog-Eshlish detection tasks under a zero-shot setting, detection effec- tiveness improves significantly with an optimized kemnl egg. increasing from 0.3048 to 0.6052
21

---

Published as a conference paper at ICLR 2025
N/Emix1, and from 0.5616 to 0.7818 N/Emix1. In addition, using proper references fur- ther improve the detection effectiveness, noting that R-Detect reaches an AUROC of 0.9490 on N/Emixl after using the non-English reference.
E.3 ABLATION STUDY FEATURE EXTRACTOR
We run an ablation study on the RAID data, mix &2 and att1&2, by changing the feature extractor in MPP and R-Detect. The results are shown in Table 13. The detection performance of using falcon- rw-1b 1b and aaherra-hase-openeee sepeeta- is better than the other feature extractors. This indicates further improvement on non-parametric detectors by using more powerful feature extractors.
Table 13: Ablation study on feature extractor. The best feature extractor of each detector at the same setting is highlighted in bold (fand Jare compared to the default setting of roberta-base).
Detector
Feature Extractor (token #256)
setting
mix1
att1
mix2
att2
MPP
roberta- saseta-baee-ppeee base-openai-aeeeeeeaa -eenadddeeeet
new kernel
0.63951
0.6044
0.57911
0.6001
roberta-base
new kernel
0.5791
0.5891
0.6221
0.5567
chatypt-dctector-roeeee
kernel 0.7146
0.59971
0.62771
0.4105
falcon-rw-1B lb
kernel 0.7971
0.5690
0.7749
0.7059
R-Detect
roherta-basee Shherta-ase-oopmai-deeeeee
new kernel 0.9984
0,9368
0.9436
0.7868
roberta-base
new kernel 0.9976
0.8576
0.9036
0.7862
noatgtt-detector-rbbettt falcon-rw-1b
new kernel 0,9960.
0.7692
0.9696
0.6900
new kernel
0,9808
0,8068
0,9852
0.8632
Detector
Feature Extractor (token #512)
setting
mix1
att1
mix2
att2
MPP
roberta- tnverta-bas-oun base- casse-openaidelle
new kernel
0.82491
0.7587
0.72901
0.6283
roberta-baa
new kernel
0.7696
0.7412
0.7052
0.7402
chatgnt-detector-robeee
kernel
0.6571
0.5771
0.7754
0.5906
falcon-rw-II lb
kernel
0.9667
0.8000
0.5213
0.5723
R-Detect
roberta-base-on
base-opemai-detecto
kernel
1.0000
1.0000
0.9832
0.9884
roberta-baa base
new kernel
1.0000
0.9700
0.8746
0.8656
beaggnt-detcttormmberet
kernel
0.9932.j
0.9428
0.9432
0.7452
falcon-rw-1b
new kernel
1.0000
1.0000
0.6472
0.7652|
F FIVE ESSAYS AND ITS RE-WRITTEN TEXTS BY GPT40
We have randomly sampled five essays from the Essay dataset (Verma et al., 2024) and asked GPT- 4o (OpenAI, 2024) to rewrite each of them by the prompt "rewrite" in Table 14.
We shuffle these human-writn written essays and GPT4o rewritten essays and take 512 tokens from each our test data in section 4.4. We here give each whole essay and its corresponding rewritten version from GPT4o
22

---

Published as a conference paper at ICLR 2025
Table 14: Five examples.
Index
Human
GPT-4o
Robert Putnam makes the case that the United States has lost social capital and civil society over the last quarter-century after explaining the importance of social capital and civil society in democratic soci- eties. Toequeville during his visit to Amer- ica said, .AúAmericans of all ages, all sta- tions in life, and all types of disposition are forever forming associations. There are not only commercial and industrial asso- ciations in which all take part, but others of thousand different types, Afreliiuu, moral, serious, futile, very general and very limited, immensely large and very minute, Ay. Nothing, in my view, deserves more attention than the intellectual and moral associations in America, Äù (Put- mann, 2000, p. 65). Even though Putnam admits that determining the significance of a few countertrends without additional re- search is difficult, he concludes that critical factors such as social trust are eroding at an alarming rate in the United States. He then goes into great detail about this erosion, and he concludes by outlining the research required to investigate these theories thor- oughly.
Robert Putnam argues that the United States has experienced significant de- cline in social capital and civil society over the past 25 years. He begins by em- phasizing the importance ital and civil society in democratic so- cieties. Tocqueville, during his visit to America, observed that "Americans of all ages, all stations in life, and all types of disposition forever forming asso- Nottois in my view, deserves more attention than the intellectual and moral associations in America" (Putnam, 2000, p. 65). While Putnam acknowl- edges the challenges in fully understand- ing the importance of countertrends with- out further research, he concludes that key elements like social trust are croding at a concerning rate. He explores this erosion in detail and calls for research to investi- gate these trends further.
social cap-
Putnam's work illustrates that social cap- ital in the United States significantly de- clined after the mid-to-late 1960s due to factors such as generational change, the rise of television, urban sprawl, and in- creasing demands on people's time and re- sources. This decline is linked to vari- ous socictal issues, including poorer ed- ucational outcomes, economic strain, and heightened tensions within and between social groups Putnam suggests that revi- talizing social capital, akin to efforts dur- ing the Progressive Era but with modern methods, could be a solution. Social capital, defined as "connections among individuals' social networks," cre- ates norms of reciprocity and trustworthi- ness, with value comparable to physical and human capital. In Bowling Alone, Put- nam details the decline of social capital contemporary America, examining its causes, consequences, and possible solu- tions.
This important book demonstrates that so- cial capital decreased dramatically in the United States after the mid-to-latte1161t due to generational succession, the rise of television and urban sprawl, and the in- creasing demands placed on people,Aos time and money, among other factors. Nu- merous social problems have been impli- cated from the same, and result, rang- ing from poor educational outcomes to economic stress and tensions between and among social groups. The most likely solu- tion this problem is re-establish social capital in the manner of the Progressive Era but using contemporary approaches.
Societal norms of reciprocity and trustwor- thiness have developed due to the accu- mulation of social capital, defined as AÄú- connections among individuals, Äō social networks., Au Social capital regarded as having the same value as physical and hu- man capital...
Putnam distinguishes between types of social capital: bonding capital, which binds individuals within a group, and bridging capital, which connects diverse social groups...
Continued on next page
23

---

Published as a conference paper at ICLR 2025
Continued
Index
Human
GPT-4o
2 Archetypes describe such characteristics, which are universal patterns that define certain events, objects, people. In lit- erature, they describe representations and concepts that common to all people, which allow people to understand and in- terpret stories. However, archetypes also exist in everyday life and play signifi- cant role shaping the perception of the world around them. In particular, there are archetypes of people that help determine their character and actions, well as mo- tivation.
Archetypes represent universal patterns that define certain events, objects, or peo- ple. In literature, they reflect common themes and ideas that resonate across cul- tures, helping people interpret and relate to stories. However, archetypes are not lim- ited literature; they also exist in every- day life and play a vital role in shaping how we perceive the world. Specifically, archetypes of people help us understand their character, actions, and motivations.
In real life, can identify certain types of individuals who share common traits shaped by both personal inclinations and their environments. For instance, a modern corporate leader like Elon Musk exempli- fies the archetype of a hero or creator. His vision and courage innovate make him a prime example of this archetype, as oth- ers often view him someone driven by desire to transform the world. These traits influence not only how he is perceived but also his actions and motivations.
Applying this concept to everyday life, one can single out special types of people who have set of common and exclusive char- acteristics for them. They are formed both on the basis of the actions and inclinations of the person and depending on the envi- ronment. An example of an archetype in everyday life is the head of a large modern corporation, such as Elon Musk. He can be defined as a hero or creator who has the in- spiration and courage to change the world around him. Other people perceive it in this way, which makes it a great example of an archetype. All his actions are motivated precisely by these characteristics, and his primary motivation is the transformation of the world. The presence of archetypes in everyday life allows me to compose a more comprehensive perception of people and events. In particular, based on the char- acteristics know, can determine the mo- tivation and role certain person, which makes him more predictable. This is cer- tainly more difficult than may be presented in the literature because people often have mixed features. However, general, the concept of archetypes gives me the oppor- tunity to understand the actions people and their characters. information to form my own relationship with them.
Recognizing archetypes in daily life al- lows for deeper understanding of people and events. By identifying these charac- teristics, can better grasp someone's 's mo- tivations and role, making their behavior predictable. While real-life individu- als may exhibit a blend of archetypal traits, the concept still helps me interpret their ac- tions and personalities, which in turn in- forms how interact with them.
later use this
Continued on next page
24

---

Published as a conference paper at ICLR 2025
Continued
Index
Human
GPT-4o
3
Introduction The shifts in attitudes of people towards romantic relationships and, more specif- ically, marriage are attributes of societal development. From this perspective, the trends common for Gen Z are incredibly informative and, therefore, used for an- alyzing the situation in this area (Klein, 2022). It clear that individuals,Ao mind- sets are formed under the influence of ex- ternal circumstances, which implies the correlation of opinions about love and with the presence threats to personal well-being. Hence, one cannot examine the accompanying phenomena and cause- and-effect relationships while neglecting either the human factor or the conditions of the environment. One of the articles de- voted to the subject is ,ÄúAre Gen Z more pragmatic about love and sex?,Äù and its critical analysis can reveal the elements of effective writing intended for considering complex events.
Introduction
Shifts in attitudes toward romantic re- lationships, particularly marriage, closely tied to societal development. In this context, trends observed among Gen Z. are especially informative and serve as a basis for analyzing changes in this area (Klein, 2022). It is evident that people's mind- sets are shaped by external factors, mean- ing opinions about love and sex are in- fluenced by conditions that affect personal well-being. As result, it is impossible to examine the underlying causes and rela- tionships without considering both human factors and environmental conditions. One relevant article addressing this topic is "Are Gen Z More Pragmatic About Love and Sex?" and its critical analysis highlights key elements of effective writing used to discuss complex social phenomena. Summary of the Article
In the article, Jessica Klein explores the tendency of Gen Z. to delay marriage and avoid long-term relationships, plac- ing greater emphasis ties. Unlike previous generations, Gen Z does not view romantic relationships natural step into adulthood, driven by sev- eral factors. First, they find the benefits of single life more appealing than settling down, which allows them focus on self- reflection (Klein, 2022).
Summary of the Article
In the selected piece, Jessica Klein narrates the tendency for representatives of Gen to postpone marriage and avoid long-term relationships while prioritizing personal is- sues. Compared to previous generations, these individuals do not view this area as natural condition for entering adulthood due to number of reasons. First, the ad- vantages of single life are more attractive than settling down and help them satisfy their need for introspection (Klein, 2022). Second, people in this group adopt a prag- matic approach to the matter because they understand the necessity of gaining stabil- ity in life in order to have a family (Klein, 2022). Third, Gen Zers face more prob- lems than their parents due to the pandemic and prefer to reflect on the kind of person they want to be partners with and choose them carefully (Klein, 2022). As a result, the processes in this area slow down sig- nificantly, and the primary focus on self, as opposed to interactions with others, justify the changes which happened this part of human life...
personal priori-
Second, many Gen Z individuals take pragmatic approach, recognizing the im- portance of achieving personal stability before starting family (Klein, 2022). Third, the pandemic has brought addi- tional challenges to their lives, prompting them to carefully consider the kind of part- ners they seek (Klein, 2022). As sult, the pace of romantic developments has slowed, with emphasis per- sonal growth taking precedence over inter- personal relationships, reflecting broader changes in modern life...
Continued next page
25

---

Published as a conference paper at ICLR 2025
Continued
Index
Human
GPT-4o
4
From 2016 to 2018, this study summa- rizes sport fisheries in the Anchorage Man- agement Area. They contained descrip- tion and historical background of each sport fishery, as well information on how each fishery is now managed, as well as performance and escapement data for 2016,A12018 According to the report, re- ductions in the number and size of fish available for release in locally stocked lakes have had a detrimental impact on cal fisheries,Ae Ao popularity and productiv- ity. It should this source gives statistical data, it may be trusted. All of this will surely be beneficial to the study, official data must be sidered.
From 2016 to 2018, this study summa- rizes the sport fisheries in the Anchorage Management Area, providing descriptions, historical background, and details on how each fishery is managed. It also includes performance and escapement data for the years 2016-2018. According to the report, reductions in the number and size of fish stocked in local lakes have negatively im- pacted the popularity and productivity of these fisheries. As this source provides sta- tistical data, it is considered reliable and will be highly valuable to the study, as of- ficial data must be taken into account.
mentioned that because
The sport fisheries in the North Gulf Coast Management Area also discussed in this study. It includes an overview of each fishery, management practices, and fish- ery performance and escapement data from 2016 to 2018. This report is issued by the official department responsible for regulat- ing fishing activities Alaska, and since it provides statistical information, it can be trusted as reliable source. The data will be essential for the research, as incorporat- ing official data is crucial.
The sport fisheries in the North Gulf Coast Management Area are discussed in this study. summary and overview of each fishery are included, and information on how the fisheries are managed and fishery performance and escapement from 2016 to 2018. This report is provided by the offi- cial department that controls fishing activi- ties in Alaska. However, should be noted that this source be considered reliable as provides statistical information. All this will undoubtedly be useful for research since it is necessary to take into account of- ficial data.
A case study is presented on the complex- ity of factors that either limit or promote diversity in Alaskan fisheries, world's most significant fishing regions. This study examines changes in partici- pation and portfolio diversity in Alaskan fisheries over three decades, using a com- bination of harvest data and literature re- views. Four case studies explore how fish- ers, fleets, and communities responded to interconnected challenges such as biologi- cal declines, market dynamics, pricing, and the privatization of fisherics. The article has been verified by experts, ensuring the reliability and accuracy the information. It will be valuable for the research, as it re- flects current trends and factors influencing sport fishing.
of the
The intricacy of variables that restrict or encourage diversity in Alaskan fisheries, one of the World,Aós most important fish- this case
ing locations, is examined study. They looked at changes in participa- tion and portfolio variety in Alaskan fish- cries over three decades using mix of har- vest statistics and literature reviews. The four case studies looked how fishers, fleets, and communities responded to var- ious interconnected stressors, such as bio- logical decreases, market and pricing dy- namics, and fisheries privatization. Experts also verified this article, which is reflected in the reliability and accuracy of the infor- mation provided. It will be useful for re- search because it reflects current trends and factors affecting the spread of sport fishing. Continued on next page
26

---

Published as a conference paper at ICLR 2025
Continued
Index
Human
GPT-4o
5
Deductive reasoning involves the construc- tion and development of cause-and sosse-and-effeee solid relationships. is noteworthy that the connection is established either between two facts point and the idea of how it will affect the future (Besnard & Hunter, 2018). Thus, a judgment (logical expres- sion) includes the first ,Ai the prerequi- sites, the second ,Ãi the conclusion. In logic, argument is valid only when it has form that makes impossible for the premises to be correct and the judgment to be wrong. According to the explained three types of deductive arguments, valid statement does not need to have actually trustworthy premises. Still, it must have ones that, if they were accurate, would guarantee that the conclusion is proper. Example:
Deductive reasoning involves constructing and establishing strong cause-and-efie relationships. These connections ically drawn between two facts or between idea and how it may impact the future (Besnard & Hunter, 2018). In deductive logic, argument consists of parts: the premises and the conclusion. A valid argument is one where, the premises are true, it would be impossible for the conclu- sion to false. However, the premises of a valid argument do not necessarily have to be true. The key that if they were true, they would guarantee the correctness of the conclusion.
typ-
For example:
1. All metals are heavy. Tin is metal.
3. Therefore, tin is heavy.
1. All metals are heavy
sound argument, on the other hand, requires both valid structure and true premises. While valid arguments can be unsound if the premises are false, they still follow a logical structure. For example:
2. Tin is metal
Tin is heavy
A sound argument must have both valid form and true premises. Valid arguments can be unsound, but they will have wrong assumptions.
All dogs are cats.
Example:
2. All mice dogs.
1. All dogs are cats
Therefore, all mice are cats.
2. All mice dogs
This is valid argument because the conclusion logically follows from the premises, but it is not sound because the premises are false. The validity of the argu- ment comes from the logical necessity of the conclusion given the assumptions, even those assumptions are incorrect. How- ever, the goal of argumentation is to es- tablish the truth of conclusion. An in- valid argument does not meet this goal and cannot be considered sound (Besnard & Hunter, 2018). Therefore, sound argu- ment must always have both a valid form and true premises. If an argument is in- valid, its premises are irrelevant, and there no argument all, much less sound one.
3. All mice are cats What makes this a valid statement is not that it has trustworthy premises and a proper conclusion but the logical necessity of the judgment, given the two assump- tions. The argument would be just as valid if the premises and conclusion were wrong. However, it is essential to determine that the purpose of argumentation is to estab- lish the truth a judgment. An invalid argument does not achieve this goal, it cannot be considered sound (Besnard & Hunter, 2018).
sound argument has valid form and true premises; thus, it can- be invalid in all cases. If the argument is invalid, then the beliefs thoroughly do not matter, and, in fact, only the conclu- sion remains. Therefore, this means that there no argument in this case, especially sound one.
27


Reviews:
[{"review_id": "KCWYOpHnWf", "reviewer": "Reviewer_hh24", "summary": "This paper proposes a non-parametric kernel relative test to detect machine generated text (MGTs) by testing whether it is statistically significant that the distribution of a text to be tested is closer to the distribution of human written text (HWTs) than to the MGTs’  distribution. It improves the current two-sample test-based detection methods, which assumes that HWTs must follow the distribution of seen HWT. The authors further develop a kernel optimization algorithm in relative test to select the best kernel that can enhance the testing capability for MGT detection.", "questions": "The method relies on several assumptions coming from empirical results (line 88, line 280-). The authors are encouraged to provide additional clarifications, both intuitive and theoretical, to further explain those assumptions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed relative test can reduce the false positive rate that has been observed in current two sample tests. \n\nIt also proposes a novel method to optimize kernels in relative tests for MGT detection, which significantly improving the effectiveness and efficiency of the detection process.", "weaknesses": "While this paper presents some innovative ideas, its significance appears limited. There has been some similar studies such as the ICLR'24 paper \"Detecting machine-generated texts by multi-population aware optimization for maximum mean discrepancy.\". \n\nThe proposed method requires the preparation of both the MGT and HWT datasets. In that case, simply comparing the method with Bino in the experiment is insufficient, and may even be unfair because Bino is a zero-shot method. It is essential for the authors to engage in a comprehensive experimental study, comparing their proposed method with various other detection algorithms.\n\nThe method leverages Roberta based GPT-2 as feature generator (line 200).  Please explain the reason, and provide a comparison of this choice with other potential alternatives in both text explanation and experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sY9I2vV583", "reviewer": "Reviewer_4sdF", "summary": "This submission proposes to address the limitation of existing non-parametric LLM detector approach that tends to make mistakes in identifying HWTs that deviate from the\nseen HWT distribution. The authors suggest to employ non-parametric kernel relative test to address the issue. Basically, the idea has some novelty to some extend. The paper highly follow the method in paper “DETECTING MACHINE-GENERATED TEXTS BY\nMULTI-POPULATION AWARE OPTIMIZATION FOR\nMAXIMUM MEAN DISCREPANCY” from both methodology style and the experimental design.", "questions": "refer to the weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1). Basically, the idea has some novelty to some extend. \n\n2). The problem to address is important.\n\n3). The writting is somehow clear and easy to follow.", "weaknesses": "My major concerns are as follows:\n\n1). The limitation of these MMD-based approach is obvious. The appraoch by nature is not Zero-Shot aaproach. As the approach needs to prepare HWT and MGT in advance, I would say the approach is not training free by nature. Especially, the paper proposes kernel optimisation algorithm to select best kernel. This by nature is somehow “training”. Thus, the approach only compare times with Bino is not sufficient, as it also needs to compare time with training/classifier based approach (Table 2)\n\n2). The experiments are far from complete. Bascially, the datasets used are obviously too simple. Why not use the RAID datasets (in the following paper) that includes 22 different LLMs with different settings?\n“RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors, ACL’24”\n\n3). Need to validate the robustness of the detection algorithm under adversial attack. \n\n4). Need to provide non-english dattasets’ results to see the effectiveness on non-english settings.\n\n5). As mentioned in 1), the paper needs to provide comparisons to other classification based approaches as well other metric-based/ logits-based approaches. Such as:\nClassification-based: \nFEW-SHOT DETECTION OF MACHINE-GENERATED TEXT USING STYLE REPRESENTATIONS. ICLR’24. https://arxiv.org/pdf/2401.06712\nThreads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs. ACL’24. https://arxiv.org/pdf/2402.10586. \nLogits-based:\nDetectGPT (ICML’23) \nFast-DetectGPT (ICLR’24) \nDNA-GPT [ICLR’24]\nDALD: Improving Logits-based Detector without Logits from Black-box LLMs. [NeurIPS’24]\n\n6). The paper also highly depends on the feature extractor, that is “fine-tuned a RoBERTa model on GPT-2-generated”. What’s the performance for other feature extractors? (need experiements). Need to explain the major different and advantage of proposed approach over training-based approaches.\n\n7). The literature review is far from sufficient. Pl refer to the recent surveys such as:\nA Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization\nA Survey on Detection of LLMs-Generated Content, EMNLP’24.\nOn the Possibilities of AI-Generated Text Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zX7Gaf8pSf", "reviewer": "Reviewer_Er1T", "summary": "This paper examines the issue of how to detect machine-generated texts. It suggests that previous methods are unable to identify human-written texts when there is a change in distribution. It proposes to employ non-parametric kernel relative test to detect machine-generated texts by testing whether it is statistically significant that the distribution of a text to be tested is closer to the distribution of human-written texts than to the distribution of machine-generated texts. A kernel optimization algorithm is proposed to select the best kernel that can enhance the testing capability for machine-generated text detection. Some experiments support the effectiveness of the method proposed in this paper.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of the paper is reasonable, and the problem it addresses has practical application value.\n\n2. The method proposed in the paper is simple and effective, with a coherent logic, and the description of the algorithm is very clear.\n\n3. The paper has a solid theoretical foundation, its arguments are logically sound, and the method is highly interpretable.\n\n4. The experiments indicate that the algorithm performs well.", "weaknesses": "1. The assumptions on which the method is based are too strong and do not align with reality. Firstly, there is significant overlap between machine-generated language and human language; they do not exist in completely separate domains. Additionally, the subspace assumption is overly idealized and lacks a solid foundation, which greatly undermines the paper's validity.\n\n2. The method proposed in the article resembles a general text anomaly detection approach and is not closely related to large language models or machine-generated language detection. It appears to be a universal solution for text anomaly detection rather than a targeted one, as the specific characteristics of the problem have not been discussed or reflected in the method's design.\n\n3. The comparative methods in the experiments are relatively few and not comprehensive enough. Many anomaly detection solutions could also address this issue, so more comparisons and discussions should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)