Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy
Authors: Shuhai Zhang
Yiliao Song
Jiahao Yang
Yuanqing Li
Bo Han
Mingkui Tan
Venue: ICLR 2024 poster (2024)
Keywords: Large language models
Machine-generated text detection
Maximum mean discrepancy
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We propose a novel multi-population aware optimization method for maximum mean discrepancy (MMD-MP) to detect machine-generated texts.

Abstract: Large language models (LLMs) such as ChatGPT have exhibited remarkable performance in generating human-like texts. However, machine-generated texts (MGTs) may carry critical risks, such as plagiarism issues and hallucination information. Therefore, it is very urgent and important to detect MGTs in many situations. Unfortunately, it is challenging to distinguish MGTs and human-written texts because the distributional discrepancy between them is often very subtle due to the remarkable performance of LLMS. In this paper, we seek to exploit \textit{maximum mean discrepancy} (MMD) to address this issue in the sense that MMD can well identify distributional discrepancies. However,  directly training a detector with MMD using diverse MGTs will incur a significantly increased variance of MMD since MGTs may contain \textit{multiple text populations} due to various LLMs. This will severely impair MMD's ability to measure the difference between two samples. To tackle this, we propose a novel \textit{multi-population} aware optimization method for MMD called MMD-MP, which can \textit{avoid variance increases} and thus improve the stability to measure the distributional discrepancy. Relying on MMD-MP, we develop two methods for paragraph-based and sentence-based detection, respectively. Extensive experiments on various LLMs, \eg,  GPT2 and ChatGPT, show superior detection performance of our MMD-MP.

Published as a conference paper at ICLR 2024
DETECTING MACHINE-GENERATED TEXTS BY MULTI-POPULATION AWARE MAXIMUM MEAN DISCREPANCY
OPTIMIZATION FOR
Shuhai Zhang 12; Yiliao Song",, Jiahao Yang', Yuanqing Li Bo Han Mingkui Tan South China University of TTchnology' Pazhou Laboratory2 The University of Adelaide Key Laboratory of Big Data and Intelligent Robot, Ministry of Education
Department of Computer Science, Hong Kong Baptist University" sezhangshuha @ma scut edu. en; mingkuitan@scuu edu.
ABSTRACT
Large language models (LLMS)) such ChatGPT have exhibited remarkable performance in generating human-like texts. However, (MGTs) may carry critical risks, such as plagiarism issues, misleading informa- tion, or hallucination issues. Therefore, it is very urgent and important to detect MGTs in many situations. Unfortunately, and human- human-written written texts because the distributional discrepancy between them is often very subtle due to the remarkable performance of LLMs. this paper, we seek to exploit maximum mean discrepuncy (MMD) to address this issue in the sense that MMD can well identify distributional discrepancies. However, directly training detector with MMD using diverse MGTs will incur significantly in- creased variance of MMD since MGTs may contain multiple text populations due various LLMs. This will severely impair MMD's ability measure the differ- ence between two samples. To tackle this, we propose a novel multi-population aware optimization method for MMD called MMD-MP, which can avoid variance increases and thus improve the stability to measure the distributional discrep- ancy. Relying on MMD-MP, we develop two methods for paragraph- basph and sentence-based detection, respectively. Extensive experiments on various LLMs. egg., GPT2 and ChatGPT, show superior detection performance our MMD-MP. The source code is available at https: / httggttthhhhhoooegggog com wwwp//s..hhh.... 9MMMsh//9/MMMMMM0
texts
challenging to distinguish MGTs
INTRODUCTION
With the advancement of large language models (LLMss,, texts generated by these models, such as GPT3 (Brown et al., 2020), natural, fluent and of high quality. These machine-generater texts (MGTs) closely resemble human- geenaangeeerated texts (HWTs) and have many promising applications in natural language processing, e.g., text summarization (Liu & Lapata, 2019), dialogue generation (Li et al., 2016) and machine translation (Bahdanau et al., 2014). However, existing LLMs may generate fake news (Zellers et al., 2019), spam (Guo et al., 2020), and phishing (Hong, 2012), suffering from factual errors, hallucination and bias (Zhang et al., 2023b; Li et al., 2023). This poses threats to online information's is credibility and security (Liu et al., 2023; Zhou et al., 2023a;b), necessitating advanced MGT detection techniques. Unfortunately. it is challenging to distinguish MGTs and HWTs because the distributional differences between them are inherently subtle (Tian et al., 2023).
To detect MGTs, existing metric-based methods (Gehrmann et al., 2019; Mitchell et al., 2023; So- laiman et al., 2019) use some statistics (e.g., log-likelihood score the probability of the test texts being MGTs, which is less effective when large language-domall gap exists between the texts used train the scoring model and the tested MGT's Another strategy, model-based methods (So- laiman et al., 2019; Guo et al., 2023), relying severely on specific MGT types, struggles to adapt to other types of MGTs. These methods face challenges in effectively capturing the distributional discrepancy between MGTs and HWTs, thus limiting their detection capabilities.
"Equal contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2024
0.025
HMS-D (Q:21
MMD-D (1:22)
0.4
HMD-D (1=51
HMD-0 (q=4)
Varlance of MMD
MHD D (g-44
MMD-0 (cm))
Tset Power /000
HMD (q=22
MHD HP (q=2)
0.3
HHD-MP (q:14)
HMDDMMP (q:33)
A.155
MHD HP (4
MHD-WW HP 1g (0
60
0.2
6.18
HHD-D (q:2)
HMD-D (q-5)
MHD-D (q-4)
0.1
(q=2)
0.8
0.009
Epochs
Epochs
Epochs
(a) MMD
(b) Variance of MMD
(c) Test Power of MMD
Figure 1: Illustration of MMD values, MMD variances, and the test power of MMD-D and our
MMD-MP during the optimization process. As the number of S populations (i.e., q) increases, MMD-D shows an increase in MMD, accompanied by a sharp rise in variance, resulting in unstable test power during testing. In contrast, our MMD-MP exhibits minimal variance in MMD values, leading to higher and more stable test power during testing.
In this paper, we seek to exploit maximum mean discrepancy (MMD) address the above issue, given its powerful ability to identify distributional discrepancies (Liu et al., 2020; 2021). However, directly applying MMD cannot effectively detect MGT's. This task often involves data from various populations, e.g., texts generated by different LLMs (e.g., GPT-3 (Brown et al., 2020), ChatGPT (OpenAI, 2022)) or different LLM settings (e.g., temperature, top-k sampling (Vilnis et al., 2023)). These populations can substantially differ in language styles and syntactic structures, resulting in significant variations of MGTs, Training a deep kemnel MMD (MMD-D, Liu et al. (2020)) under such circumstances will incur the issue of high variance (i.e., the large variance of MMD-D in Fig- ure (b)). This means the estimated discrepancy between HWTs and MGT's fluctuates considerably and thus lead to unreliable and unstable detection (the low test power of MMD-D in Figure 1 (c)).
This paper pioneers exploring the optimization mechanism of kernel-based MMD. As we train the kernel with data from multiple populations, the estimated MMD increases with its variance growing significantly (see Figures (a)-(b) and more explanations in Section 2.3). This phenomenon arises due to the intra-class distance within MGTs in kernel- based MMD's 's optimization objective. This distance largely hinders the optimization process that aims to aggregate MGTs, resulting in highly fluctuating MMD for MGT detection. In this paper, we propose novel multi-population aware optimization method for MMD called MMD-MP, which uses a multi-population proxy to remove the constraint on aggregating all instances in MGTs. In this way, we can achieve a low variance of the MMD between MGTs and HWTs, resulting in more stable discrepancy estimation and more reliable detection (see MMD-MP results in Figures 1 (b)-(c)). Furthermore, with the trained deep kernel, we develop two approaches for paragraph-base based detection and centence-based detection, respectively. Empirical evidence on various LLMs such as ChatGPT, GPT2 series, GPT3 series and GPT-Neo series cexhibits the superiority of our methods. Our contributions are summarized as:
1) We delve into the optimization mechanism of MMD and reveal that high variance of the MMD) when handling training data from multiple different populations can result in an unstable discrepancy estimation for MGT detection.
2) We propose novel multi-population aware optimization method for training kernel-based MMD (called MMD-MP), which can alleviate the poor optimization of MMD-D and improve the stability of discrepancy measures.
3) Relying on the proposed MMD-MP, we devise two novel MGT detection methods. Extensive ex- periments across numerous LLMs, including ChatGPT, GPT2 series, GPT3 series, GPT-Nco series, demonstrate that our methods consistently outperform existing baselines.
2 PRELIMINARIES AND MOTIVATIONS
2.1 PRELIMINARIES
Two-sample test (2ST). Let P. Q be Borel probability measures on X'CRd, We observe independent identically distributed (IID) data = 1 ~P' and fy= 1 ~ 2ST aims to determine if P and Q come from the same distribution, Le., P = Q (Borgwardt et al., 2006; Liu et al., 2020).
Single-instance detection (SID). Let P be a Borel probability measure on RC and IID observa- tions S =xx,] 1n ~P" SID aims to tell if the test instance ý is from the distribution IP.
2

---

Published as a conference paper at ICLR 2024
Maximum mean discrepancy. Following Gretton et al. (2012); Liu al. (2020), maximum mean discrepancy (MMD) aims to measure the closeness between two distributions, which is defined as: Definition 1. Let k - be the bounded kernel of reproducing kernel Hilbert space Hi. FF be class of functions f X-R, and X~P, Y be two random variables,
MMD Q Hxx=
sup JEF.1if1n,
[JJ(C))-BLLYYY))VEEEEEE + 2k(X, Y).
Intuitively, we could view A(X, X') or YY) as an intra-class distance and k(X,, Y) as an inter- class distance. When n=m, we can estimate MMD via a U-statistic estimator unbiased for MMD MS, So:
Σ Hay, where H,, - x,)-,,,)-..,... x,) +k(,,)))
(1)
n(n
In this paper, we consider a kernel-based MMD (Liu et al., 2020), where the kernel is defined as:
(1 (x)))))) f(y)),
(2)
where E (0, 0)(x) = xx(x)) is deep neural network with feature extractor f. and q are Gaussian kernels with bandwidth dΦ and bandwidth og. respectively, e.g. k(a, b) =
exp (-Ila (-1|a-bl//20g) b112222) Since f is fixed, the set of parameters of hw w = {c, (,, 0z)-
Test power. Test power is the probability of rejecting the null hypothesis (5n P= Q) when IP * Q- A higher test power indicates a greater level of certainty regarding the distributional discrepancy, For reasonably large n,, Liu al. (2020) find that the power nearly proportional
J(P Q kw) = MMD (P, Q; k/ (IP, Q: kw), where of
E
[H, Hal E [H::s!²)
We can estimate J(P,, Q; kw) with a regularized estimator by MMD (Sp, kw))
[Se, Sqgkk)=
where 0 173
Hey
Σ ΣΣ Hey
(3)
V 60h (Sp. kw:kw) +
n
2.2 HIGH VARIANCE PROBLEM OF KERNEL-BASER MMD IN MULTIPLE POPULATIONS
In practice, we may collect training data S with multiple different populations from a mixture of texts generated by different LLMs such as GPT3 (Brown al., 2020), ChatGPT (OpenAI, 2022). Due to the diverse language styles generated by different language models, this can result in sig- nificant variations in the generated text. Under such circumstances, although we can maximize the criterion J (3) to optimize the kernel k a high-variance discrepancy exists.
To validate the above phenomenon, we demonstrate the MMD value and its variance during training by maximizing Eqn. (3) under different numbers of She populations (i.e,, q). According to Figures (a)-(b), the MMD between St and S for MMD-D increases, which is desirable for MGT detection, but its variance simultaneously increases, which will deteriorate the detection performance. The impact of this phenomenon worsens with the increase of This indicates that the population 5 with larger variations makes the optimization of the original MMD more challenging.
High variance causes poor optimization of kernel-based MMD. Although kernel-based MMD is widely used to identify distributional discrepancies, limited literature has explored its optimization mechanism, possibly due to its complex mathematical format. Specifically, when maximizing j in Eqn. (3), it is challenging to determine the individual changes of the MMD value and its variance, resulting in intricate analyses of each term MMD. To address this, we decompose the variance of MMD as below and conduct empirical studies to demonstrate the trends of each component and their variances during training in Figure 2. Further detailed analyses are provided in Section 2.3.
Components in MMD's variance. We introduce H' (x, k'x,xxx))x,x,,,,,,,,, (x, - (y', x) with its variance Var(B H Varl)) Here, E denotes taking expectations across two populations sampled from MGTs and HWTs and Var denotes taking variances within these sampled populations. The variance of MMD can then be decomposed into: [) +) y')) 2Cov(EEE"" E,K (y. decomposition, we find the changes of MMD's ' variance are essentially from the changes variance of hew x"), Ris (x, y) and ky, y'), presented as kxx, kxy and kyy in Figure 2.
22ooooE x')], E[[k,,, (x, y) Var(E (x, (x, y)]).
By this
3

---

Published as a conference paper at ICLR 2024
HM1D-0 O
MMD bax
Variance/10000
150
2CavlIF-kky) Vartloy)
150
2Cex01-kky)
HMD
MMD- Jay
Variance/10000
Varikyy]
20
HEED MP Jux
VEED 0. Joy
E/K//100
20
HMND boy
MMND HP Josa
100
Varli!))
100
Variley)
15
HEND kay
Jey
Veri2key]
Veri2kyl
HEND MP kay
HP Kyr
25
50
75
100
25
50
75
100
25
50
75
100
25
50
75
100
Epochs
Epochs
Epochs
Epochs
(a) E(k), q=1
(b) B()), q=3
(c) Var(MMDD-D), q=3 (d) Var(MMD-MP)), q=3
Figure 2: E(k) in MMD and their variances under two optimization methods (MMD-MP is ours).
Subfigures (a) and (b) depict the value of each EE(k) in MMD during training by MMD-D and MMD-
MP with q=1 and q=3, respectively. Subfigures (c) and (d) illustrate the variances of some terms associated with MMD, i.e., oi when training by MMD-D and MMD-MP, respectively.
2.3 OPTIMIZATION MECHANISM OF KERNEL-BASED MMD
We conclude that we should exclude the intra-class distance in Si during the optimization. To elucidate this, we illustrate some critical observations, followed by explaining these phenomena.
Observations: i) In Figures 2 (a)-(b), both [j x') and E (y.'')) exhibit generally increas- ing trends, while [[k,((x, (x, y] shows relatively minor changes. ii) As the number of populations q increases, [K (y: (y) becomes smaller than E[ki (x, x')] and their gap between them widens. iii) In Figure (c), the variance of MMD is mainly determined by the variances of E[,, (x, x') and (y, rather than other terms. iv) When MGTs in S comprise multiple distinct populations (e.g., q 3 in Figure 2 (c)), E[kk (y,y') optimized by MMD-D has a significant variance, well as (x, x')], which is consistent with the results of different q in Appendix H.
Explanations: First, i) and ii) indicate that as q increases, aggregating instances in St (MGTs) is more challenging than aggregating instances in S (HWTs) when using Gaussian kernel to optimize both k. (x, x') and kw (y, simultancously since optimizing smaller E kw is more challenging. Second, iii) and iv) indicate that the objective j (3) affects the optimization of each term regarding S/:" and S in similar manner. Thus, the characteristics of their distributions after mapping by the same kernel function, such as the mean and variance of kw exhibit similar changing trends.
Furthermore, the optimized kernel function E[k (y.y') will not only a) map random pairwise MGT instances (y, y') in Stg close to each other, making the mapped MGTs more uniform; but also b) enforce implicit "pairing rules" for aggregating MGTs in S These rules are shared to pair HWT instances in Str throughout optimization, When MGTs in Sh comprise different populations, the differences Sh" -pairs might be large. Applying the paring rules may inadvertently map HWT instances in St far from their center, leading to increased fluctuations of (x,, x') and thus larger (ar(E (',x')) Similarly, the pairing rules for HWTs in S negatively affect MGTss in S but to a lesser extent because S is IID, meaning Sp-instances share more similar statistical characteristics. Pairing rules for HWTs in Shr do not need to be strong as those for aggregating non-IID MGTs in Str Therefore, we will exclude the intra-class distance in 5t associated with (y y') throughout optimization. We also explore the case of excluding Bii, (x, x') in Appendix G.
PROPOSED METHODS
3.1 PROBLEM DEFINITION
Problem Definition, Let TP be a Borel probability measure on separable metric space XCRd and IID observations Sp=(x,]" ) from the HWT distribution F, we aim to tell if the upcoming data Sq= 1m is from the distribution IP. Note that Siz can be HWTs or MGTs generated by multiple different LLMs. When m-1, the problem can be regarded as single-instance detection task.
Challenges of MGT detection. The distinctions between HWTs and MGTs (e.g., text from LLMs like GPT-4) are inherently small, especially in shorter texts like single sentences, making it challeng- ing to distinguish between them. Moreover, the diversity of LLMs leads to significant variations in the generated language style, which further increases the difficulty of MGT detection. Although
4

---

Published as a conference paper at ICLR 2024
deep kernel MMD (MMD-D) is effective in measuring distributional discrepancies, texts generated by multiple LLMs with substantial variations pose challenges in training the deep kernel, e.g. high variance of the MMD. Such high variance in MMD will lead to unstable estimations of distributional discrepancies, ultimately resulting in unsatisfactory performance in MGT detection.
3.2 MMD-MP FOR MGT DETECTION
As aforementioned, we do not consider optimizing the intra-class distance in St Instead, we pro- pose a multi-population aware optimization for kernel-based MMD (MMD-MP) with proxy MPP by maximizing a novel objective as Eqn. (4), and show the training algorithm Algorithm 1.
(I, Q kw) = (P, Q /w)/ (P, Q kw),
(4)
MPP (I,, Q; H4) == E K (X, - 2 (X,Y))
(5)
Empirically, we can approximate MPP with an unbiased estimator
MPP n(n - 1)
Σ
Hig where Hig:=ku (Xi, xj) kw (x,, Vz) kiw (y,, xz).
(6)
Moreover, we can estimate Eqn. (4) by
Algorithm Training deep kernel with MMD-MP
Input: 5t", St frozen feature extractor f:
J(Sp, S(;p, kw) = MPP, (Sr, Sgy )ww)
<- wp: X 4- 10-8;
66 (Se, Sq; kw) +
for T = 1.,.................... Tmaa do
hw + kernel function using Eqn. (2);
M (w) - MSP S kw) using Equ. (6);
0
High
Hig
V1 [i(w) (S S. kw) using Eqn. (8);
nd
[www) + M(w)/ (w) as in Eqn. (7);
(8)
e c w v/ Admm Jx (w);
We next provide some theoretical analyses to elaborate the objective function Eqn. (4).
end for Output: k.
Unlike MMD (Borgwardt al., 2006; Gretton et al., 2012), the proxy MPP in Eqn. (6) does not in- corporate (y, related Sq. However, MPP,, n still U-statistic (Serfling, 2009) like MMD,,. 2 with numerous desirable statistical properties that facilitate convenient theoretical analysis. Note that although maximizing MPP, for kernel training is straightforward, it ignores the variance and could lead to an unstable discrepancy (see more details in Appendix E). To address this, we analyze the asymptotics of MPP,, u and derive its test power as follows.
Proposition (Asymptotics of MPP.) Under the alternative 51 : P + Q based on a standard central limit theorem, we have:
(MMP MPP) 4, N((,00);)
where ob; = 4 (HiHi - E EHHie His His denote different Hig-
Corollary 1. (Test power of MPP.) For reasonably large n, the probability of rejecting the null hypothesis ho : IP = Q when I2 + Q is given by:
MPP oF
/T[MPP- R(S2))
T
(10)
Vnosi
where Prşi.r = Pr n MPP,, + R(Sa)) 2)) and R(Sa) mnn Eijs kw kw.,,, > 0, $ is
the standard normal cunulative distribution function.
Remark 2 Note that we do not exclude the term R(Sg) in Eqn. (10) due to the uncertain con- vergence of rMPP, (which could be related to the kernel kw) when P=Q. Instead, nMMD, +R)) has been proven to be convergent (Gretton et al. (2012), Theorem 12). This enables to find approximate power with rejection threshold (Liu al., 2020).
5

---

Published as a conference paper at ICLR 2024
Algorithm Testing with MMD-MP for 2ST
Algorithm 3 Testing with MMD-MMP for SII>
Input: Testing texts , ko:
Input: Referenced HWT Sp.. testing texts S2 S2
est Mete-MDD.. ; using Eqn. (I);
for Xi, Yy in , do
Sth
for i = 1, 2..er,n -eeerm do
Shuffle Sp" USe into Sx and Sy:
P3t MMD, (SF [xx]; kw) using Eqn. (11); Qj+-MMD, (SF", ky)kkk) using Eqn. (11); end for
2
perm, 1-MDD(Sx Sy: kw) using Eqn. (I); end for
Output: p. p-vuuue
11perm Emer 1(perm,2es 2est)
Output: AUROC value with two sets (P)), (Q,)
Corollary (1) shows that, given and o5; being constants, for reasonably large n, the test power of MPP is dominated by the first term inside Φ. As suggested by Section 2.3, when removing the intra-class distance in Stg (i.e., R)))))0), we last optimize Eqn. (4) for MGT detetcion.
We now study the uniform convergence of our proposed optimization function as follows.
Theorem 1. (Uniform bound of MMD-MP) Let w parameterize uniformly bounded kernel functions ki in Banach space of dimension D with || his be uniformly bounded by supwen xuxx xx,xx'')Ss x'VSv with Lx Lipsschhi,, i.e., (x,' (x, xxxxx)) S L Lx|lw-w' Let Ωg be set of w for which 03; >0²00. Taking -1/3 with probability at least 1 1-5, we have
v² /DDDogg Dlog( Ram) log((Rn)) log + uLx + 1 1)
sup [| Sej kw) J(P, Q2 wEf,
2/3
Detailed constants and proofs are given in Appendix A.3. Theorem shows that our estimator [S+,, Sej kw) converges uniformly over ball in parameter space as increases. With enough training data, the estimator converges the optimal solution if the best kernel exists.
3.3 EXPLORING MMD-MP FOR MGT DETECTIONS
We consider MGT detection in two scenarios: paragraph-bass based detection and sentence-basee detec- tion. The former aims to detect whether the test paragraph follows the distribution of human-writter paragraphs. We address this two-sample test. The latter focuses on distinguishing one single machine-generated sentence from HWTs. We consider this as single-instance detection task.
MGT Detection Under two-sample test (2ST). For paragraph-based detection, we consider each sentence within the paragraph as an instance. The detailed procedure for 2ST using MMD-MP can be found in Algorithm 2. Note that we only optimize the kernel using MMD-MP during training and employ the MMD instead of the MPP to measure the distance between Sy and Sç during testing. The rationale behind this is that we prefer the distance between Sp and So to be zero when P-Q, rather than a negative value, i.e., - (((a())+ <0. Empirically, the performance of these two strategies almost identical. We defer more discussion in Appendix F.
MGT detection under single-instance detection (SID). While paragraph-based detection is widely employed, there exist practical applications that require single-sentence detection, egg. online con- tent filtering false information recognition. Despite numerous works have shown MMD powerful means to measure the discrepancy between two distributions or populations, we still hope it can be employed to single-instance detection due the powerful deep kernel. To achieve this, with a trained kernel, we calculate the distance between a set of referenced HWTs Sp and a test text y with Eqn. (11). The detailed procedure for SID using MMD-MP is shown in Algorithm
MMD (SF k));
1 n
Σ 1J=1
Xi, X,)
k (x., + (8.)))
(11)
72
Advantages of MMD-MP for MGT Detection over MMD-D. We highlight two key benefits of MMD-MP over MMD-D: 1) More stable discrepancy estimation: While MMD-D has similar E[k(x, with MMD-MP, its variance is much greater than MMD-MP (see Figures 2 (a), (c)- (d)), indicating that MMD-D exhibits poorer aggregation effects for St compared to MMD-MP. Moreover, training MMD using MMD-MP results in a significantly lower variance (see Figures (c)-(d)), mitigating the challenge of high variance in MMD optimization and thereby enhancing the

---

Published as a conference paper at ICLR 2024
Table I: Test power/100 on HC3 given 3, 100 processed paragraphs in training data.
CharGPT Nco-S
CharGPT GPT3-S
Method
CharGTT
GPT3-S
Neo-S
C2ST-S
62.83_0.90
43.64-2.92
30.68
34.62-2.7
46.06=2:55
C2ST-L
89.82-1.02
75.74
60.97
68 1..
78. 28.222a..2
MMD-O O
26.43_1.40
21.17
19.83
25. 25.233.0.7
25.18
MMD-D
91.76_1.58
86.98
75.45+4.98
86.442107
91.4620.47
MMD-MP (Ours)
23.21
89.36.2.1
79.68_1.4
89.63
91.96g0.02
Table 2: Test power 100 on HC3 given 1, 000 processed paragraphs in training data.
ChaIGPT Neo-S
CharGPT GPT3-S
ChaiGPT GPT2-S GPT2 M
ChaiGPT GPT3-S
CharGT
Method
ChaiGIT
GPT3-S
Nco-S
Neo-S
Neo-L
C2ST-S
00.32:2.00 os
38. 1..05
27. 6..66
34.48.amm m0
40.50_a.m
24.97
32.0 22.04
24.53
C2ST-L
87.8 11.04
71.29
61. 13:35
67. 17
15.49
56.24
67. 12.
51.1
MMD-O
27.23
19.96=a.m
19. 19.58_2..0
27. 27.34
26.03-1.6
20.05
23.9 23.91
20.92
MMD.D
91.38942.09
84.0125.04
22.81
74.2244.06
83.29
62.34-44.00
77.76=2.50 90
63. 13.1852..8
MMD-MP MP (Ouns)
92.31
a6.34
76.35
25.30
89.05
85.54 49.92
79.69.0.7
stability of discrepancy estimation. 2) Enhanced transferability: Our MMD-MP prioritizes fitting
HWTs Sp" compared to MMD-D when training the deep kernel, reducing its reliance on MGTs. This manner enhances the transferability in detecting unknown MGTs (as verified in Section 4.4).
4 EXPERIMENTS
Datasets and LLM architectures. evaluate our method on Human ChatGPT Comparison Corpu (HC3) (Guo et al., 2023), which is ChatGPT text detection dataset with both long and short-level corpus, and XSum dataset (Gutera & Delp, 2018), which is a news dataset. We choose paragraphs with more than 5 sentences for testing paragraph- baased detection and sentences with more than 5 words for testing in sentence- basedd detection. For LLMs, we consider ChatGPT (OpenAI, 2022), GPT2 series (Radford et al., 2019), GPT3-S (Toan, 2023), GPT-Neo series (Black et al., 2021), )), GPT4all-j (Anand et al., 2023). For each experiment, except for ChatGPT using MGTs in the original HC3, for other LLMs, we generate MGTs with the first 20 prompts of HWT in HC3.
Two-sample test baselines. 1) MMD-O: MMD with Gaussian kernel whose bandwidth is opti- mized; 2) MMD-D: MMD with a trained deep kernel (Liu et al., 2020); 3) Classifier two-sample tests: C2ST-S (Lopez-Paze Pazz & Oquab, 2017) and C2ST-L (Cheng & Cloninger, 2022).
Single-instance detection baselines. 1) Metric-based detectors: Log- Lok-Liikli (Solaiman et al., 2019), Entropy, Rank (Gehrmann et al., 2019), Log-Rant and DetectGPT (Mitchell et al., 2023); 2) Model-based detectors: OpenAI-D (Solaiman et al., 2019) and ChatGPT-D (Guo et al., 2023). We also cross-entropy loss optimize deep neural network baseline, called CE-Classifier, whose model is the same as that of MMD-D and MMD-MP except for an additional binary classifier.
Evaluation metrics. We evaluate the detection performance using test power for two-sample test (Gretton et al., 2012) and the area under the receiver operating characteristic curve (AUROC) (JiménezVValvedde 2012) for single-instance detection. Through all experiments,
randomly take 100 paragraphs or 1, 000 sentences for testing and repeat the experiments 10 times for synthetic data or 5 times for real-world data. We use bold numbers to indicate the best results in tables.
4.1 RESULTS SYNTHETIC DATA
We investigate the impact of variation (i.e., variance) of training data on test power. To this end, we synthesize four-center Gaussian mixture data. Specifically, let 1d and Idd represent d-dimensional all-one vector and a d-dimensional identity matrix, we denote Po Id) and Q(u, 5) as:
Q(u. -) -
1
1
8 I
81
12
-
-
We consider various Q by setting μ (0.2+0.02xi) and 5=1.3 with d0100. Note that these four-center Gaussian mixture data for training the kernel but only sample a center Gaussian data for testing. We use Lorm of the variance of data in Q to represent its variance.
From Figure 3, we draw two observations: 1) As μ increases, the test powers of MMD-D and our MMD-MP become higher since the distributional discrepancy between P and each single-center

---

Published as a conference paper at ICLR 2024
Table 3: AUROC/100 on HC3 given 3, 100 processed paragraphs.
CharGT
GPT3-S
Neo-S
ChasGPT Neo-S
ChuGGT
GPT3-S
30
Likelihood
60.56
61.18-1.25
20-
Rank
73.20:
71.96.21.0
72.09 22.0
72.74
MMD-D
Lag Rank
89, 89.58.a.o7
68.78u1.29
64.92.1.04
70.
70.47.011
70-
MMD-MP
Eniropy
91.50-a000
54.34:1.35
56.19
40.08.0.4
42.08 42.08-2m
DetectGPT-d
17.9/2-mm7
50.41
22.07
06.01
60-
DerectGPPia
81.07gmm
58.46gaas
22:28
67.54g0
67.322110
50-
OpenAL-D
78.57
84.06.mm
24.8
81, 41,20
80.68
Varlance
CharGPTDD
96.644.p.1s
61.89
54.46an.30
78.47anmm
78, 78.90s1.
CB-Classilier
02. 02.44anes
88, an.8
90 32
anaas
MMD-0
56.34
63.19-mm
20.46:122
67-79e122
Figure 3: Impact of variance
MMD-D
84.86 es
81.12auss
91.30 aa
MMD-MP (Ours)
96,20 36.2
95.08.0.32
92.04,0.5
92. 32.48
14.61
in training data on test power.
Table 4: AUROC/100 on HC3 given 1, 000 processed paragraphs in training data.
CharGIT
CharGIT GPT3-S
ChatGPT GPT2-S
ChasGPT
CharGPT
Method
ChasGPT
GPT3-8
Neo-$
Nen-S
Nen-S
GPT2-M
GPT3-S
Neo-L
CE- Classifier
95.99_..a
91.40 91.40g0a7
87.27
88, 13
91.50_oaa
84.89
68.9 88.01
84.15
MMD-0
54.64e1.
61.52-115
61.98=2.22
58 28..08
-7.92-1.42
57. $7.86-11.9
78.78.
58.07 =120
MMD-D
83.8610
91.502124
81.10.0.8
80.28.001
00.28_112
85.50 85
88.0740.7
84.20
MMD-MP (Ours)
95.00
94. 28..77
89.6140.4
90.83
93. 93.46:0.22
87 a3.m
91 20
86 28.32220.2
Gaussian data in Q becomes larger; 2) When the variance of data in Q increases with J4, the test power of MMD-MP surpasses that of MMD-D by a maximum margin of approximately 9% power This suggests that forcing the aggregation of all data in Q will hinder MGT detection performance when the variance of training data is significant.
4.2 TEST POWER ON PARAGRAPH-BASEE BASED DETECTION
We compare our MMD-MP with the state-of ttateoof-the-ar ahe- (SOTA) two-sample test method for detecting MGTs HC3 in terms of test power and defer the results XSum in Appendix J.1. To broadly evaluate detection performance, we conduct experiments on various scenarios, including training on full training data, limited number of training data, and unbalanced training data.
Test power on full training data. We conduct this experiment using 3. 100 processed paragraphs to train the model. Table shows the detection performance under different training populations in terms of test power compared with other baselines, including one and two populations. The results show that MMD-MP exhibits superior test power compared with other methods, particularly in detecting Neo-S texts, where the test power is approximately 6% 1 higher than MMD-D.
Test power on a limited number of training data. We utilize 1, 000 processed paragraphs train the models with one, two, and three training populations, respectively. Table 2 demonstrates that our method achieves significantly higher test power performance compared with others. Remarkably, our method outperforms MMD-D by an average of 8.20% 1 on test power over the two training populations and exhibits a 13.97% 1 increase over the three training populations, suggesting that extreme instability of discrepancy estimation of MMD-D when dealing with multiple populations.
Test power on challenging unbalanced training data. In real-world scenarios, obtaining HWTs is easily feasible, while collecting MGTs poses more challenges. To thoroughly assess the per- formance of our approach, we conduct an evaluation with 2, 000 processed HWT and 400 MGT training paragraphs. As illustrated in the top of Figure 4, our approach exhibits significantly supe- rior performance compared with other methods, e.g., surpassing the test power of 6.96%~ 14.40% 1 than MMD-D, highlighting its stability in detecting MGT's under unbalanced training data scenarios.
4.3 AUROC ON SENTENCE- BENTENCE-BASED DETECTION
In this section, compare MMD-MP with the SOTA single-instance detection method for detecting MGTs on HC3 in terms of AUROC and defer the results on XSum in Appendix J.2.
AUROC on full training data. Table 3 shows that our MMD-MP achieves better AUROC than other baselines. Notably, our MMD-MP outperforms the SOTA model- based method, i.e. ChatGPT- D with 1.20% 1 of AUROC when detecting ChatGPT texts. Moreover, MMD-MP achieves an improvement of 0.22%~1.71% t on AUROC over MMD-D and 0.61%~2.649. t over the CE- Classifier method, illustrating the superiority of method in detecting the single sentence.
8

---

Published as a conference paper at ICLR 2024
GPT3-5
Power 11.0
88.66
89
79.72
75.35
75.27
70
70
60
60
s0
50
40
30
$2.41
40
30
37.52
40
10
20
C25y3
574
MING-0
20
C25y.3 CASTE 1MMD-0 1IMD-0 Mp
20
City SETE M
20
C2813 C3ST
MMD
MADDO
AUROC /100
-00
90
91.42
90
90
80.30
70
70
70
70
58.26
56.10
MND-D
MMD.NP
Cl-clastteer
MMO.O
MND B.D.. Map
Clalasiifer
5ND Ma
Co
D MDD-Mp
Figure 4: Test power and AUROC on HC3 given 2, 000 HWT and 400 MGT training paragraphs.
AUROC on limited number of training data. From Table 4, our MMD-MP achieves perfor- mance comparable CE-classifier in detecting ChatGPT texts and surpasses other baselines in other scenarios. Particularly, our method outperforms MMD-D by 2.46%t and CE-Classifier by 1.39% on average. Note that although the model of CE-Classifier the same as MMD-D and MMD-MP except for an additional classifier, our MMD-MP demonstrates superior detection performance over CE-C Classifier, indicating the powerful distinguishability of our method.
AUROC on challenging unbalanced training data. From the bottom of Figure 4, our approach consistently outperforms baselines for sentence-based detection. Critically, our MMD-MP achieves an improvement of 3.89%~9.00%% t AUROC over MMD-D and 0.38%~~..79' 1 over the CE- Classifier method. Combining Tables 3, and Figure 4, we conclude that our method is superior in detecting single sentence under various scenarios different LLMs compared with other methods in total, suggesting the stability of our method for MGT detection.
4.4 RESULTS ON UNKNOWN LLM TEXTS
Table 5: Test Power /100 on unknown LLMs.
Method
Nco-L
GPT-j-Gb
GPT4all-j
In light of poor performance for MGT detection baselines on unknown LLM-text, we evaluate our method in the context of this type of detection. We
C2ST-S
11.20=a.30
7.72
14.30=2.5a
C2ST-L
34.12
29 o1
42.37 13:8
MMD-O
12.93
7.71
17.08
MMD-D
38.18 38.18-4.18
31 31.92+440
51.28-0.11
train the models using texts generated by ChatGPT, GPT2 and GPT2-m on HC3, and then test on texts generated by GPT-Neo-L, GPT-j-6b and 4PT4all-j. From Tables 5-6, our method outperforms the base- lines by large margin test power and AUROC. Critically, our MMD-MP achieves an absolute im- provement of 23.61 23.61 % MMD-D. Moreover, our method outperforms MMD-D by 3.25% t and CE- II-CCassifierr by 3.37% f of AUROC on average. These results demonstrate the superior transferability of our method.
MMD-MP (Ouss
61.79
59.57
77.69
Table AUROC/ on unknown LLMs.
Method
Nco-1.
GPT-j-6b
GPT4all-j
CE-Clasiifer
78.00
82.5710s1 71.56_1.49
MMD-0
53 m3.
52. m2
MMD-D
77.91
75.47-1.41
82. 21
MMD-MP (Ours)
28.08
78 78441s
85.7520.30
test power over
MMD-D
MMD-MP
4.5 VISUALIZATION OF KERNEL FEATURE Φ1
40
We visualize the feature (d=300) extracted by Φy over two LLM--tetts via t-SNE (Van der Maaten & Hinton, 2008) for MMD-D and MMD-MP. In Fig- ure 5, two types of LLM-text features by MMD- D are entangled and disorganized, while the MGT features obtained by our MMD-MP exhibit lower overlap, confirming that our method indeed relaxes the constraint of aggregating all MGT instances.
20
GFT-66
GFTT-G
-40
-0
6
20
40
60
-40
-20
20
40
Figure 5: Features Visualization via N-SNE
5 CONCLUSION
In this paper, we propose multi-population aware optimization method for training kernel-based MMD called MMD-MP, which alleviates the poor optimization of MMD. With a trained deep ker- nel, we design two MGT detection approaches for paragraph- basedd and mantence-basel detection tasks, respectively. Extensive experiments on a variety of LLMs demonstrate the superiority of our methods in terms of test power and AUROC, especially in detecting unknown LLM texts.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to thank Feng Liu for insightful technical discussions. This work was partially sup- ported by the National Natural Science Foundation of China (NSFC) 62072190, TCL science and technology innovation fund, the Young Scholar Project of Pazhou Lab (No. 7ZL2021KF0022 the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Founda- tion No. 2022A151501 1652, and HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22- 23/SCI/04, Tencent Innovation Joint Project.
REFERENCES
Harika Abburi, Michael Suesserman, Nirmala Pudota, Balaji Veeramani, Edward Bowen, and San- mitra Bhattacharya. Generative ai text classification using ensemble Ilm approaches. arXiv preprint arXiv:2309.0775:. 2023.
James Allen. Natural language understanding. Benjanin-Cummings Publishing Co., Inc., 1995.
Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. GitHuu, 2023.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In International Conference Learning Representations, 2014.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5185-5198, 2020.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Rose Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. 2021.
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. e57, 2006.
Bioinformatics,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nicholas Carlini and David Wagner. Adversarial examples not easily detected: Bypassing ten de- tection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 3-, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions on Information Theory, 68(10): 6631-6662 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Kacper P Chwialkowski, Dino Sejdinovic, and Arthur Gretton. wild bootstrap for degenerate kernel tests. Advances in neural information processing systems, 27, 2014.
Felipe Cucker and Steve Smale. On the mathematical foundations of learning. Bulletin of the American mathematical society, 39(1):1-49, 2002.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 2018.
10

---

Published as a conference paper at ICLR 2024
Yao Dou, Maxwell Forbes, Rik indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7250-7274, Dublin, Ireland, May 2022. Association for Computational Linguisties.
Noah A. Smith, and Yejin Choi. Is GPT-3 text
Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In International Conference Machine Learning, pp. 3564-3575, PMLR, 2021.
Sehastian Gehrmann, SEAS Harvard, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. ACL 2019, pp. 111-116, 2019.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-73, 2012.
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv preprint arXiv:1702.06280 2017.
David Güera and Edward J Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), pp- 1-6. IEEE, 2018.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yu- peng Wu. How close chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint 1:Xiv:2301.07590 2023.
Zhiwei Guo, Yu Shen, Ali Kashif Bashir, Muhammad Imran, Neeraj Kumar, Di Zhang, and Keping Yu. Robust spammer detection using collaborative neural network in sfeemee-of-thingg applica- tions. IEEE Internet of Things Journal, 8(12):9549 9558, 2020.
Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgtbench: Benchmark- ing machine- machine-generated generated text detection, arXiv preprint arXiv:2303. 14822, 2023.
Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. Cater: Intellectual property protection on text generation apis via conditional watermarks. Advances in Neural Information Processing Systems, 35:5431-5445, 2022.
Jason Hong. The state phishing attacks. Communications of the ACM, 55(1):74 81, 2012.
Chenyang Huang, Osmar R Zaiane, Amine Trabelsi, and Nouha Dziri. Automatic dialogue gener- ation with expressed emotions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Short Pupers), pp. 49-54, 2018
Alberto Jiménez-Valverde. Insights into the area under the receiver operating characteristic curve (auc) as a discrimination measure species distribution modelling. Global Ecology and Bio- geography,
Ilmun Kim, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimality permutation tests. The Annals of Statistics, 50(1):225-255 25111,1, 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations, 2015.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference Machine Learning, pp- 17061-17084. PMLR, 2023.
Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. Stylometric detection of ai- ai-generated text in twitter timelines. arXiv preprint arXiv:2303.03601 036697 2023.

---

Published as a conference paper at ICLR 2024
Richard J Larsen and Morris L Marx. Introduction to Mathematical Statistics and Its Applications:
Pearson New International Edition. Pearson Higher Ed, 2013.
Anne Leucht and Michael H Neumann. Dependent wild bootstrap for degenerate u-and v-statistics. Journal of Multivariate Analysis, 117:257-2801 2013.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191 2023.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J Sutherland. Learn- ing deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning. pp. 6316-6324 PMLR, 2020.
Feng Liu, Wenkai Xu, Jie Lu, and Danica Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. Advances in Neural Information Processing Systems, 34:5848 5860, 2021.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy Ilms: survey and guideline for evaluating large language models' alignment. arXiv preprint drXiv:2308.0537,,220 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1907. / 1692, 2019.
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Con- ference on Learning Repreventations, 2017.
Yukun Ma, Khanh Linh Nguyen, Frank Z. Xing. and Erik Cambria. A survey empathetic dialogue systems. Information Fusion, 64:50-70, 2020.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher Manning, and Chelsea Finn. De- tectgpt: Zero-shot machine-generated text detection using probability curvature. pp. 24950- 24962, 2023.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press. 2018.
OpenAI. Introducing chatgpt. Website, 2022. URL
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Nicholas Pangakis, Samuel Wolken, and Neil Fasching. Automated annotation with generative ai requires validation. arXiv preprint arXiv:2306.00176, 2023.
Xiaohuan Pei, Yanxi Li, and Chang Xu. Gpt self-supervision for a better data annotator. arXiv preprint arXiv:2306.04344 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
12

---

Published as a conference paper at ICLR 2024
Robert J Serfling. Approximation theorems of mathematical statistics. John Wiley & Sons, 2009.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Cerbert-Voss,, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: /908. 09203, 2019.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The science of detecting IIm-generated texts. arXiv preprint arXiv:2303. 000002 2023a.
Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. Does synthetic data generation of Ilms help clinical text mining? arXiv preprint 033300003.04360, 2023b.
Yuchuan Tian, Hanting Chen, Xutao Wang, Zheyuan Bai, Qinghua Zhang, Ruifeng Li, Chao Xu, and Yunhe Wang. Multiscale positive-unlabeled detection of ai-generated texts. arXiv preprint arXiv:2305. 18149, 2023.
Phan Minh Toan.
hawp:ppp----mmlllliiinnneee-nnii mirhtoan/gpt.3 smal
attpss//hugginggaace
2023.
Пyy O Tolstikhin, Bharath K Sriperumbudur, and Bernhard Schölkopf. Minimax estimation of maximum mean discrepancy with radial kernels. Advances in Neural Information Processing Systems, 29, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
Luke Vilnis, Yury Zemlyanskiy, Patrick Murray, Alexandre Tachard Passos, and Sumit Sanghai. Arithmetic sampling: Parallel diverse decoding for large language models, In International Con- ference Machine Leurning, pp. 55120-35136. PMLR, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1e..iv:1804.022
2018
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint
1004014609.0814,00 2016.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020.
Yifan Yang, Huihui Fang, Qing Du, Fei Li, Xiulan Zhang, Mingkui Tan, and Yanwu Xu. Distin- guishing differences matters: Focal contrastive network for peripheral anterior synechiae recog- nition. In Medical Image Computing and Computer Assisted Intervention-MICCAl 2021: 24th International Conference, Strasbourg. France, September 27-October 2021, Proceedings, Part VIII 24, pp. 24-33. Springer, 2021.
Yifan Yang, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 15901-15911, 2023.
13

---

Published as a conference paper at ICLR 2024
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Shuhai Zhang, Feng Liu, Jiahao Yang, Yifan Yang, Changsheng Li, Bo Han, and Mingkui Tan. Detecting adversarial data by probing multiple perturbations using expected perturbation score. In International Conference on Machine Learning, pp. 41429-41451. PMLR, 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint urXiv:2309.012 2023b.
Zhanke Zhou, Jiangchao Yao, Jiaxu Liu, Xiawei Guo, Quanming Yao, Li He, Liang Wang, Bo Zheng, and Bo Han. Combating bilateral edge noise for robust link prediction. In NeurIPS, 2023a.
Zhanke Zhou, Chenyu Zhou, Xuan Li, Jiangchao Yao, Quanming Yao, and Bo Han. On strength- ening and defending graph reconstruction attack with markov chain approximation. In ICML, 2023b.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
CONTENTS
Theoretical analysis
16
A.1 Proof of Proposition
16
A.2 Proof of Corollary
16
A.3 Proof of Theorem
17
Related Work
23
Large Language Models
B.2 MGT Detection
23
B.3 Two-sample Test
24
More Details for Experiment Settings
25
C.I Datasets
25
C.2 Implementation Details of Our Method
25
C.3 Implementation Details of Baselines
26
C.4 Implementation Details on Synthetic Data
26
More Intuitive Explanations the optimization of Kernel-based MMM
27
E Impact of Using Variance in J During Training
28
F Impact of Using MPP for Testing
28
G Replacing kw (X,X) with &w (Y,Y") in MPP
29
H More Results Optimization Mechanism of MMD-D
30
I More Comparison Results over Unknown Texts HC3
30
J Comparison Results on XSum
31
J.1 Test Power Paragraph- based Detection
31
AUROC Sentence-based Detection.
J.3 Results on Challenging Unbalanced Training Data
Results on Detecting Paragraph-Ievel Sentences
32
Examples of ChatGPT Text Detection
33
M Discussions and Future Directions
34
15

---

Published as a conference paper at ICLR 2024
A
THEORETICAL ANALYSIS
Given a kernel Rw, we define MPP and its unbiased estimator as:
MPP (I, Q H4) == [kw X') 2k (X, Y]]
(12)
MPP MPP kw) n(n 1)
Σ
Hij. where
Kw (x;, Vy) Ris (y;: (13)
itj
For simplicity, we denote i = MPP n(n-1) Eiss Hig and Th = H2
A.1 PROOF OF PROPOSITION
Proposition 1. (Asymptotics of MPPa). Under the alternative $1 IP o Q based on standard central limit theorem, we have:
VR(MPP, MPP) 4 N(
(14)
where 03 (B|HigHia)) 12) Hi His denote different Hig-
Proof. Let denote a pair (x, y) and hUU U²) = hw (x,, x (x) - xy). Based on the property of U-statistic and Lemma in Section 5.2.1 of Serfling (2009), when n oo, we have
Var(ū - +5_ =: o
(15)
where we use o to denote o for simplicity, and &w is represented as:
Ew Vary [Eo [N(U, U'II
Ey [Evr [h(U, U') By" [h(U,""]]] Ey [Ev (U(UUU'
H[HigHial {
Via Theorem A in Section 5.5.1 of Serfling (2009), we obtain the conclusion.
A.2 PROOF OF COROLLARY
Corollary 1. (Test power of MPP... w.)) For reasonably large the probability of rejecting the null
hypothesis Do IP Q when Q is given by:
Vn(MPP + R(So))
MPP Proj.r
(16)
Th;
Vn
where PrMP := Pr n MPP,, + R(Se) r)) and R(SQ) mn- Eies kw (w 0, $
the standard normal cunulative distribution function. funciion.
Proof. Via Proposition 1, we obtain
Pr n MPP, + R(So)) > r) Pr
<n(MPP, MPP)
r- + R(SO))
ODi
Vn
Φ 7PMPP R(Sa)),
Vn Os;
Vn(MPP + R(S2))
Vnoss
16

---

12v Published as a conference paper at ICLR 2024
A.3 PROOF OF THEOREM
To analyze the convergence of our method, we first describe the following relevant technical as- sumptions that have been analyzed by Liu al. (2020).
(A) The kernels A uniformly bounded by supwen suPzex (x,,)))) v.
(B) The possible kemnel parameters w lie in a Banach space of dimension D. Furthermore, the set of possible kernel parameters Q is bounded by Ro, i.e., Q S (w || S Ra).
(C) The kernel parameters is Læ-Lipschitz for all data x, x' E X and w,w' Q i.e.
| x') kew (x, x') & T=|lw-W'||-
We first provide some results on the uniform convergence of ihw and 6% based on e-nct argument (Vershynin, 8), which will be used prove Theorem Proposition 2. Under Assumptions (A) to (C), we have that with probability at least 1 - 8,
6 vn "
2 2lo
sup fM Thw
S
2D log (4Rav + L&
Proof. We denote random error function
(ww) == il - 7l
Based on Assumption (B), w Ω min; || will S g (Cucker & Smale (2002), Proposition 5).
use most T (4Ra/g)² points {witen such that for any
Recall that i 1) Eee Hij. construct new population by replacing one data pair (1) in Se and So with x(), and thus obtain 1% m(n-1 Eips Fij. where F' is the same as H except when i or equals to 1.
We calculate the difference
New 1/21 nnn 1)
H
Fil
H Fal + n(n 1) j>1
H Fig
n(n
1) 1>1
2
6v
nnn 1)
n
According to a union bound and McDiarmid's inequality (Mohri et al., 2018), we have
T
Pr max [o(wi)] (E(
Ze <
Pr (r(((w)))2 e)
Pr (liwn
-Twu
2 e)
Pr (1i
El[nb,]1 2
1=1
i=
2ca
(-
26a (T(220)7mm)
< 2T exp
2T exp
Σソ (12v/n)²
Let 2T exp
(- (-(12099)m)) (120)3/a)
262 8, we obtain
v V log Thus, with probability least 1 5,
we have
12v
2T
6z
4Rg (17)
max (E(.....))
((w))
<
log
2 2D log
22n
7
q
We next deviate the Lipschitz of ((w) based on Assumption (C).
fh n 5 n(n 1)
Σ i41
H (w)
H H.((')
1 n(n 1)
Σ
3T ( w'll =I || -wll,
17

---

log Published as a conference paper at ICLR 2024
E (w))
EE 11/2 '1 E (1 1//b011/ <3ام| w'll
Nw nw-1
Thus, we have
(((w) 2(w')l 1iw ?le (fi Nw )1 S 1fi fw ) + (Nw nw)l - 6L*||w w'1l. ||| (18)
Combining the results of (17) and (18) and setting q= Vna vn we have that with probability at least 1-6
sup [+(w)] ( ||| = (www) (ww,) (w=)) - [+ww)) + (w(w") +(ww))
S max EE (E....))
[ + =6L29
$ 6v
2 +221og log 4 + /n
6L.
2 log
vn
6
2
VV 2
log 8 2D log (4 + Ls
where w' = arg sup [w)) and w. = arg. mill,,e ( I|0= - will.
Next, we present some lemmas to lay the foundation for establishing the uniform convergence of de For simplicity, we denote oº Ōb; and 0g Unj
Lemma 1. Under Assumption (A), we have that with probability at least 1-5.
2 n 8"
2
10_ - E S 252
Proof. We first estimate 1 (d when changing one data pair in S and Sg. To this end, we construct new population by replacing one data pair (xI,)1) in Sp and Sq with (x1, v(), and thus
2) ( Eig Fi)
obtain (o%)² 4 3/ Σ. [3
where F is the same H except when
i or j equals to 1. Recall that
2
)
1
- =4
Σ
Σ EHi
Hig
Hi
n
n2
sf
After changing one data pair, the difference of the first term changes with
2
2
2
1
Σ
H1
Σ
Σ
Hig
-
P3
11
H
H
Fif
Σ
H
3
H
Σ
H F
n
1>1 1>1==1/=1
Σ
Σ
-
Σ
Σ
1,11
i>1 11,1=1
(n218
+
- 1n-1)9 + 2m - 1)318
6
7
3
2
n
2
3
18

---

Published as a conference paper at ICLR 2024
where the penultimate line follows by the facts that: 1) Hi - 1 18v due to Hij 1
Fi,
3v, Ful 3v; 2) IH 11; Fh max H; H Fā) S 9v/ 3) when i > 1, l + 1,
1 - - - 3 - Fi | [ - Pil x (6v) 18v.
After changing one data pair, the difference of the second term changes with
2
1
pij
11
n2
ij
Σ Hig + FF
EH H FF
71
(2 2-3)2 Σ
Hij - Fij
122
(2 (2-3z).
Σ
Σ
Hig
n
j=1,181
(2 2--3))-
(2n 1)6v
n
2
36,²
11
n2
Thus, have
6
7
3
(2
(2-10
163
(66)))
<
9v2
360²
11
n
n
n
n²
14
11
3
=36
m
n
3
50402
n
Using McDiarmid's 's inequality (Mohri et al., 2018), with probability at least 6, we have
2
1 - EE E S 252
log
n g
Since 02 is not unbiased, we next estimate [E d3 o3
Lemma 2. Under Assumption (A), have
648r²
E
n
Proof. Recall that o and the expectation of 0%
0 = 4 4E[Higg - 2
E 3 = 4
H
1
1
EE H
3
n2 Ejah
We decompose the summation term into terms with non-repeated indices and terms with repeated indices.
19

---

Published as a conference paper at ICLR 2024
For the first term in E d
Hel
Σ
H +
H
3
n2
12.
Aff
:je:ll <3
nnn - 1) - 2) H2 Hisl +
n(n
1) 22)
n23
n3
Thus, we have
H[ H|Hi2Hij H3l
12
n(nn
1)(n -2) 1) E HiHiiHijs + (1-
n(n 1)(n - 2)
BHHGH
n
n2
3
2
B[H
Hill - HH
n
??
3
1^
18v2.
n
n
Similarly, for the second term in E 0 note that
1
H abl
BiH Habl +
Σ
HiH
n
n
n2
sjab
ijab: ( 11)1=4
ijabi (x),a,,))<c4
n(n - 1) - 2) - 3) E(Hi2)2 +
1
n(n 1)(n - 2) - 3)
HH Hidl
n
n²
Thus, we have
Σ
Hi - -[HHi
n2
kjab
n(n
1)(n 2) -3) -1
(1- nnn 1)(n 2) 3) n2
||Hig1| 12+
Bilf Hall
nd
6
11
+
|[Hi2]
HHH
n
n²
nz
6
11
+
1802.
n
n²
n3
Therefore, we conclude that
3
2
6
11
6
IR
22
0211
<4
18
18u2
n
n?
m
n²
n3
9
13
n1
n²
n
648
Next, we deviate the Lipschitz of (w) := 63 -3. Lemma 3. Under Assumptions (A) and (C), the Lipschitz of v (w) 02, - -o3 is given by
[Ww) ((w')l & 288
20

---

Published as a conference paper at ICLR 2024
Proof. We first deal with the Lipschitz of (w() by
[w(w) v(w')l = 10 - o - ow +
0211+00 - -0
The first right term is bounded by
2
1
H Hi
(w)
Σ
1
1 n2
(w)
Σ
H Habww
n
n1
n²a
ab
ije
zjab
ijab
H (w)
H (w)
Hed (w))
H (c'))
4
H
(w)
H (w)
HH1 (w') H
n
n
ijah
H (w) H (w)) Ha! H Hr((an)
+
H
(w))
H (w'))
+ H2 HiP H
je
ije
H H Hiy (w) Hall'')
| ( Halw') - H&3 Her
Halw')
12
ab
(()
H H7( || + ||H H + Hi
72
H [w)
-H + | | H ((()) - H /,
/
ab
ijab
2 (3v. - 3T |||00 w'll) + n
Σ
2 (3v. 3T& ||w w'll)
Eff
144vL4 ||ا w'll-
Similarly, the second term is bounded by
10 - o. S 4 l H ( His H - E (H12 H13 '1
(((()) (w=')]
+ E
H
E H H (aa'))
4E Hig H(( HH(WW") Hi( 21
+ 4E
Hi Hg
Hig
Halw')
$ 14AvL - w'll.
Thus, we obtain the result
[ww)- - v(w') S 288 - w'll-
Relying on Lemmas to 3, we provide the result of the uniform convergence of 03- Proposition 3. Under Assumptions (A) to (C), with probability at least 1 - 8, we have
2
2 +
D -log(4A/RRa) + 648 288v Ln.
sup 102 2 < 252 2 wE
log
n 3
n
vn
Proof. Similar to Proposition 2, we denote a random error function
(w) := 03 22
Based on Assumption (B), we can use at most T = (4Ra/o g)" points {wi)Tn such that for any Ω min; || will q (Cucker & Smale (2002), Proposition 5). According to Lemmas 1 and 2 and a union bound, we have that with probability at least 5,
648
max [Ww(w) < iE (I,....))
max :E(I.....7)
122
E 02
+E 62,
-02, <
max
102,
E 62, +
ie :E(I....?)
n
2T
648
2
2
4Rg
648v
< 252
log
< 252
log
+ log
7
n
n
5
n
n
21

---

Published as a conference paper at ICLR 2024
Combing Lemma and setting 9= V we have that with probability at least 1-8
sup V [w)) = [w(w)] = 100 (w=) (w())) + (w+)) - (w,) )1 + [1((w) - 1/ (ww))
< max iE(...T)
[|((w)] + L28
2
2 2 = log(AV7(Ba) + 648v² 288
< 252
log
+
n
n
n
vn
where w" arg sup (w:) and w, arg min E (wi.)w)) || - -well
Next, relying on the results of Propositions 2 and 3, we begin to prove Theorem 1.
Theorem (Uniform bound of MMD-MP.) Let w parameterize uniformly bounded kernel
functions kw in a Banach space of dimension D with [[ll| his be uniformly bounded by supwen x (x, x'SSv with Lx-Lisschitz i.e., (x, x') (x, (x,x') S. Sxllw-w'l,, |]. Let
Ωr be set of ws for which o²; >0²00. Taking -1/31/3, with probability at least 1 1-5, we have
sup [1 Swj J(P,Q; k-)1 = 0 12n113 2V
D plog( Dlog(Ram)) (aan)-lo_ 8 + vL + 1 1)
wEn,
Proof. Let Let σ2 == + X, we have
iw
Th
tw
nw
tw
Tw
Tw wen,
nw ow
sup WEN,
I sup
+ sup wen,
+ sup
On,) Ow wen,
Ju, T
102,1 22A2l
10
1
S sup li went
+ sup weft Jw, TwiDOw (ow.)
nuw
sup
1iw nul
ww. (6w.1 + )w,)))
+ ow)
west ow
3v
3rd
1 + sup - wen,
< sup
weft, As + vds² -0- - +
-lw nlw
(s2 + +As++ + vs² As2
3v
32
sup 1 oلن +
sup M n/l wEf.
v xs
2x
well
According to Propositions 2 and 3, with probability at least 1-6, the error bound is at most
6v
756
2 2 j |
(8611²² + VnV
6
32A
1944
+D log(4Rav7))+ +
L
s²vnx)
syn
283
'nnvA
Taking x = n 1/3, we get
(
s/n"" (SV/== 52n1/3)
6z
+ 756
2200g 8
2 + 2D 2D 2Dlog(4RoV (
18642 6
+
La +
3v
+
1944
syn
253n1/3
2n55/6
(1110 +756V3)
2 2 +2D (86422+
6s
3v
1944
L
sn1/33
1//6
2s
n-/2
(19)
Thus, we have that with probability at least 1-6,
sup [i(Ss, kw) - (IP, Q: kw))) o wEl,
D Diog (log( log + vLn + -
1²
2n1/3
Note that the error bound of the optimization is close to the result of Liu et al. (2020), however, the detailed coefficient of each term in Eqn. (19) is totally different due to the different H and H".
22

---

Published as a conference paper at ICLR 2024
B RELATED WORK
B.I LARGE LANGUAGE MODELS
Large Language Models (LLMs) hold significant importance in various fields due to their capabil- ities in natural language understanding (Ouyang et al., 2022; Bender & Koller, 2020; Allen, 1995; Wang et al., 2018), and demonstrate exceptional performance in several downstream tasks such as dialogue generation (Li et al., 2016; Huang et al., 2018; Ma et al., 2020), machine translation (Bah- danau et al., 2014; Wu et al., 2016; Caron et al., 2021) and text annotation (Pei et al., 2023; Pangakis al., 2023; Tang et al., 2023b).
The transformer architecture (Bahdanau et al., 2014; Vaswani et al., 2017; Xiong et al., 2020) plays a pivotal role in language modeling by introducing a self-attention mechanism, enabling the model to selectively focus on different segments of the imput sequence. Building upon the transformer, Devlin et al. (2018) introduce the Bidirectional Encoder Representations from Transformers (BERT), a standout model in the series of masked language models. Moreover, Liu et al. (2019) introduce RoBERTa, which employs robust optimization strategy applied to the BERT pretraining method, achieving superior performance across various tasks compared to BERT.
Another well-known category of language models belongs to the GPT series (Radford et al., 2018; 2019; Black et al., 2021; Toan, 2023; Black et al., 2021). For example, Radford et al. (2018) intro- duce the generative pre-trained Transformer (GPT), which demonstrates remarkable performance across diverse language generation tasks due to extensive training large-scale text data. Sub- sequently, GPT2 (Radford et al., 2019), GPT3 (Toan, 2023), and GPT-Neo (Black et al., 2021), expand the model structure, incorporating more parameters and training broader corpus, con- sequently surpassing the performance of their predecessors. OpenAl releases ChatGPT (OpenAI, 2022), which trained on the GPT-3.5 architecture. This excellent model continually enhances its generative capabilities through Reinforcement Learning from Human Feedback (RLHF) (Christiano al., 2017; Stiennon al., 2020). With its revolutionary performance in generating coherent and contextually relevant text, GPT-4 (OpenAI, 2023) is expected to further advance the capabilities of large language models, pushing the boundaries of language understanding and generation.
B.2 MGT DETECTION
Various machine-generater text (MGT) detection methods (Solaiman et al., 2019; Dou et al., 2022; Yang et al., 2021; Mitchell et al., 2023; Kirchenbauer et al., 2023; Tang et al., 2023a) have been developed and shown promising performance. In general, current detection methods can be roughly divided into two categories, i.e., metric-based methods and model-based methods. Specifically, the former employ statistical metrics (e.g.. likelihood and entropy) extracted by LLMs to detect outliers, while the latter finetune pre-trained LM (e.g., RoBERTa (Liu et al., 2019)) to identify MGTs.
Metric-based MGT detection methods. These approaches leverage pre-trained LLMs scoring models to measure the statistical discrepancy between numan-written texts and machine -etbenne-geerttedd texts. Among them, the commonly used metrics involve ligelikelihood (Solaiman et al., 2019), en- tropy, rank (Gehrmann et al., 2019) and log-rank (Mitchell et al., 2023). Different from the above, DetectGPT (Mitchell et al., 2023) proposes to compare log probability of texts under multiple per- turbations, based on the assumption that the MGT' are more likely to lie in the local optimal of log probability, which achieves higher AUROC compared with other metrics. However, these metric- based detection methods are compromised with inferior performance when encountering a large language-domain gap between the generated text model and the scoring model.
Model-based MGT detection methods. These methods usually train classification model us- ing texts provided by both humans and L.MMs. To be specific, OpenAI-D (Solaiman et al., 2019) finetunes a RoBERTa model with GPT2-generated texts and is used in detecting GPT2 outputs. ChaiGPT-D (Guo et al., 2023) devises two manners (i.e., using pure answered text or QA pairs) to train the model using HC3 dataset (Guo et al., 2023). Besides, Kumarage et al. (2023) train a clas- sifier by combining standardized stylistic features and LLM-based text embeddings. Abburi et al. (2023) propose an ensemble neural network model using probabilities from pre-trained language models as features to train a text classifier. These methods train the classifier severely relying on MGTs, leading to unsatisfactory transferability for MGT detection.
23

---

Published as a conference paper at ICLR 2024
More recently, alternative detection paradigm is the watermark-basse detection (He et al., 2022; Kirchenbauer et al., 2023), which defined as unseen modifications to texts that hide identifying information. For example. Kirchenbauer et al. (2023) propose to inject the watermark by selecting a randomized "green token" set before a word is generated, and softly promoting "green tokens" while sampling- Such watermark is hard to remove and can be detected by statistical test with p-values. However, these methods rely on a tailored language model to add the watermark and thus only distinguish the texts generated by this model, limiting their application scenarios.
B.3 TWO-SAMPLE TEST
Two-sample test (Gretton et al., 2012; Liu et al., 2020; Lopez-Paz & Oquab, 2017; Cheng & Cloninger, 2022) is a hypothesis test that aims to determine whether two populations are from congruent distribution (Borgwardt al., 2006). Since the traditional statistical methods such as t-tests and Kolmogorov-Sminng and Now-dimensionas spaces, board set of kernel- based methods have surged to prominence, which construct a kernel mean embedding for each distribution and measure the difference between them.
tests (Larsen & Marx, 2013) are stuck with complex assumptions
Maximum discrepancy (MMD) serves highly efficacious metric distinguish between two distributions (Gretton et al., 2012; Gao et al., 2021; Zhang et al., 2023a). Tolstikhin et al. (2016) further derive lower bounds for MMD estimation based on finite data for radial universal kernel. To address kernel adaptation for the quadratic-time MMD, Liu et al. (2020) propose to choose the best kernel by splitting data. addition, Kim et al. (2022) propose an adaptive two-sample test for testing equality between two Hölder densities supported on the real d-dimensional unit ball. Nonetheless, limited research has explored the optimization mechanism of kernel-based MMD. In this paper, we delve extensively into this through comprehensive empirical investigations and propose novel optimization method to further improve the stability of training for kernel-based MMD.
An alternative strategy to compare distributions involves training classifier between them, subse- quently evaluating its accuracy. These approaches are commonly referred to as classifier two-sample tests. Among them, C2ST-S (Lopez-Paz & Oquab, 2017) uses the output of the softmax layer as representation, while the C2ST-L (Cheng & Cloninger, 2022) uses the output logits instead. These several methods are provable to be encompassed by kernel- based MMD (Liu et al., 2020).
24

---

Published as a conference paper at ICLR 2024
C MORE DETAILS FOR EXPERIMENT SETTINGS
C.I DATASETS
HC3
HC3
HC3
1500
4000
Frequency
Frequency
2000
1000
2000
500
2000
250
50
100
150
200
230
300
10
20
30
40
50
60
15
20
25
50
Length of Paragraphs
Length Sentences
Numbers of Sent In. one_paarggaph
Xsum
XSum
xsum
8000
17500
15000
Frequet
8000
PraquARcy
250000
12500
16000
6000
150000
7500
2000
5000
30000
2500
300
1.0
50
Length Paragraphs
Length Sentencess
Numbers one paraggagaaphh
Figure 6: Illustration of statistical distributions of texts in HC3 and XSum.
Table 7: Parameter size of the pretrained LLMs adopted in the experiment.
Model
GPT2
Neo-S
GPT2-M
GPT3-S
Nco-L.
GPT-j-6b
GPT4all-j
Parameters
124M
125M
355M
551M
2.7B
GB
6B
Human-written texts (HWTs) in our experiment come from the Human ChatGPT Comparison Corpu (HC3) dataset (Guo et al., 2023) and the Extreme Summarization (XSum) dataset (Güera & Delp, 2018). Specifically, HC3 contains 24, 322 question-answer pairs in the form of both long short-level paragraphs or sentences, covering a wide range of domains such as open-domain, computer science, finance, medicine, law, and psychology. Meanwhile, XSum contains 226, 711
articles from BBC ranging from 2010 to 2017 and covering domains including news, politics, sports, weather, business, technology, science, health, family, education, entertainment and arts. For paragraph- based detection, we choose paragraphs with more than 5 sentences for testing, while for sentence- based detection, we choose sentences with more than words since shorter sentences with fewer than 5 words could be difficult for us to classify into some category. We also provide statistical distributions of texts from both HC3 and XSum in Figure 6, including the length of paragraphs, the length of sentences and the number of sentences in one paragraph.
For machine genernttd texts (MGTs), we consider commonly used LLMs to generate, including ChatGPT (OpenAI, 2022), GPT2 series (Radford et al., 2019), GPT3-S (Toan, 2023), GPT-Neo series (Black et al., 2021), GPT4all-j (Anand et al., 2023), where the model parameters are shown in Table 7. Since HC3 already provides the texts generated by ChatGPT, we utilize the remaining models to generate texts according to the first 20 prompts of HWTss The text generation strategy is similar to that of Mitchell al. (2023).
C.2 IMPLEMENTATION DETAILS OF OUR METHOD
Architecture Design of Deep Kernel. The deep kernel oy in our MMD-MP is neural network equipped with feature extractor f. For the feature extractor f. we employ OpenAl's RoBERTa- based GPT-2 detector model (Liu et al., 2019), which is the same as that of OpenAI-D (Solaiman et al., 2019), and consider its last hidden state the feature of the input text. Each token in this feature extractor has a dimension of 768, and we set a maximum of 100 tokens per sentence. The net- work & consists of a hidden-layer transformer followed by projector and multi-layer perceptron (MLP), where the projector reduces the data dimension from 768 to 512, while the MLP reduces the flattened data dimension from 51, 200 to 300. The data dimension during the whole procedure when
25

---

Published as a conference paper at ICLR 2024
feeding sentence into the kernel follows the sequence: that we only optimize the network Φ and fix the extractor f during training through all experiments.
1--1-1,1111111-1 200- Note
More detailed algorithms. Our method is designed to effectively detect whether the given texts are from humans. To summarize, we first extract features from both HWT and MGT sentences using the fixed feature extractor f. Then, we train the deep kemel Φ by maximizing the objective J in Eqn. (10) (Algorithm 1) for both paragraph-level and sentence-level detection scenarios.
For paragraph-Ievel detection, we compute the estimated MMD between given HWTs and test para- graph sentences ("est"). We then iteratively randomize and split these mixed sentences, calculating MMD values as "perm,". We obtain test power by comparing "perm;" with "est" (Algorithm 2). For sentence-level detection, using set HWTs as a reference set, we compute the MMD distance between the test single sentence and reference samples (Algorithm 3).
We conduct our experiments using Python 3.9 and Pytorch 2.0 on server with Ix NVIDIA A800 GPU. We use Adam optimizer (Kingma & Ba, 2015; Yang et al., 2023) to optimize the kernel parameters. In Algorithm 1, we set À to 10- and batch size to 200, and learning rate to 0.00005 on HC3 and 0.00001 on XSum in all experiments. Following the setting of two-sample test in (Liu et al., 2020), we set the threshold a = 0.05 to determine whether to reject or accept the null hypothesis when testing for two-sample test.
C.3 IMPLEMENTATION DETAILS OF BASELINES
Two-sample test baselines, For two-sample test, we compare our method with MMD-O, MMD-D (Liu et al., 2020), C2ST-S (Lopez-Paz & Oquab, 2017) and C2ST-L (Lopez-Paz Oquab, 2017). The architectures of the first two basclines are the same as our MMD-MP, and the latter two are classifier two-sample test methods that use the same architecture as our MMD-MP except for an additional binary classifier. We run these baselines using the code from Liu et al. (2020). More- over, we set the learning rates of these four baselines to be 0.00005, 0.00005, 0.0001 and 0.0001, respectively on HC3, while 0.00001, 0.00001, 0.0005 and 0.0005 on XSum.
Single-instance detection baselines. For single-instance detection, we compare with metric-based detectors, e.g., Likelliol (Solaiman et al., 2019), Entropy, Rank (Gehrmann et al., 2019), Log- Rank and DetectGPT (Mitchell et al., 2023) and model- detectors, e.g., OpenAI-D (Solaiman al., 2019) and ChatGPT-D (Guo al., 2023). We implement these methods based on the code from He et al. (2023). In addition, we use cross-entropy loss to optimize a deep neural network as baseline, called CE-Classifier, whose model is the same that of MMD-D and MMD-MP except for an additional binary classifier. Similar to the training of MMD-D and MMD-MP, we fix the extractor and only update the kernel network and the binary classifier, where the learning rate is set to 0.0001 on HC3 and 0.0005 on XSum, respectively.
Kernel hy byper- per-parameters selection. For all experiments, we set 6=10-10 and o,=30. In the case of our MMD-MP, we set 05=55 when the sample size n=3, 100 and ax=45 when n=1, 000. Accordingly, for MMD, we set =20 when the sample size n=3, 100 and o when n=1, 000.
Through all experiments, we evaluate the methods under the training set with 3, 100 or 1, 0.0 para- graphs. During testing, we randomly take 100 paragraphs for two-sample test and 1, 000 sentences for single-instance detection. For each method, we repeat the experiments 10 times for synthetic data or 5 times for real-world data to ensure experimental reliability.
C.4 IMPLEMENTATION DETAILS ON SYNTHETIC DATA
For the toy experiment in Section 4.1, we use the network following high-dimensional Gaussian mixtures settings of Liu et al. (2021). ). We train the deep kernel using synthetic data consisting of 200 instances with learning rate of 0.00005 and test using 1, 000 sets, each containing 10 instances. Moreover, we consider different ş to represent data with different variances, i.e., larger μ means larger variance. Furthermore, we use L -n-mo of the variance of data in Q to represent variance.
cww/ Fttp:///ofooooii////0o0oooo111 hhtpps://gt htip:///ihai..comom//m/eeeiiiiaa http://xxinniihe//e//eeeehhh
26

---

Published as a conference paper at ICLR 2024
D MORE INTUITIVE EXPLANATIONS FOR THE OPTIMIZATION OF KERNEL-BASED MMD
MMD-D q=1
MMD-D 3
MMD-MP q=3
Epoch=1
Epoch=50
Epoch=100
Figure 7: Illustration of the optimization of MMD-D and our MMD-MP under different MGT populations, where the red dots represent HWT samples, while the triangles of other colors represent MGT samples.
In Section 2.3, we have offered some insights into our motivation through empirical observations and detailed analyses to comprehend the optimization process of MMD-D and our MMD-MP. To gain clear understanding, For MMD-D optimization, it intuitively tends to separately aggregate human-writter texts (HWTs)
provide additional intuitive figures, demonstrated in Figure 7. and all possible different- class distance, and simultaneously push them away from each other like increasing inter-class dis- tance. When the MGT population Str comprises different populations, e.g., q 3, this optimization presents challenges due to significant fluctuations (as discussed in Section 2.3). In contrast, our MMD-MP relaxes constraints on aggregating all MGTs populations (i.e. removing the k(Y, Y") from MMD), focusing more on fitting HWTs. This stabilizes MMD values and enhances transfer- ability of detection discussed Section 3.3).
muchine-generaan texts (MGTs), such as decreasing the intra-
27

---

Published as a conference paper at ICLR 2024
E IMPACT OF USING VARIANCE IN J DURING TRAINING
In our MMD-MP, train the deep kernel ku by minimizing the objective J in Eqn. (7), which is the
ratio of MPP, and o In this experiment, we investigate the impact of variance 05 To achieve this, we remove the variance and only minimize MPP. instead. Tables 8 and 9 show test power and AUROC of our method compared with that without using variance in J on HC3 given 3, 100 processed paragraphs generated by different LLMs. Obviously, MMD-MP demonstrates significant performance drop when variance optimization is omitted (e.g., 6.62% J of test power and 2.95% + of AUROC on average). This decline emphasizes the significance of incorporating variance à within the training process of kernel-based MMD to ensure stability in discrepancy estimation.
Table 8: Impact of using variance 23 in J during training in terms of test power/ 100 on HC3 given 3, 100 processed paragraphs in training data.
ChatGPT Neo-S
ChatGPT GPT3-S
Method
CharGPT
GPT3-S
Neo-S
MM (wlo variance)
10.78
80, 20.74
79.764277
73.
81 09 91.96.a.62
MMD-MP (w/ variance)
93.21 +1:35
89.3642.91
79.68
89. 1.994
Table 9: Impact of using variance o²; in J during training in terms of AUROC/100 on HC3 given 3, 100 processed paragraphs in training data.
ChatGPT Neo-S
ChatGPT GPT3-S
Method
ChaiGPT
GPT3-S
Neo-S
MP (wlo variance)
89.39
88,93 1.,93
88,53 92.0440.58
90.06
92. 92.47=0.45 94.61
MMD-MP (w/ variance)
96.20 96.20.0.29
95.08
92. 42.487
F IMPACT OF USING MPP FOR TESTING
In our approach, we exclusively employ our proposed MPP during the training of the deep kernel but calculate MMD as a metric during testing. In this experiment, we investigate the impact of using
MPP during testing. To this end, we replace MMD,, in Algorithms 2 and 3 with MPP denoting it as MMD-MP (MPP), which is distinct from our original method, MMD-MP (MMD). In Table 10, we synthesize a four-center Gaussian mixture data as used in Section 4.1 with μ (0 ==o and distinguish them from the standard Gaussian distribution Ao I²). In Table 11, we directly test the transferability on unknown texts to further demonstrate the impact of using MPP during testing. The results in Tables 10 and 11 reveal that the performance of these two strategies is almost identical, which verifies the claim the impact of MPP during testing in Section 3.3.
Table 10: Impact of using MPP during testing in terms of test nower/100 Synthetic data with different μ.
Data (a)
0.40
0.38
0.3G
0.34
0.32
0.30
0.28
MMD-MP (MPP)
90 84
90.63
99. 09.22:0.21
48
97.11
94.74=1.28
90.95
MMD-MP (MMD)
99. 99.81.0.06
99.6110.14
99.21,0.38
98.
98.17
97.20
91.77=111
91.03:g2.0
Data (e)
0.26
0.21
0.22
0.20
0.18
0.16
0.14
MMD-MP (MPP)
S5. 58..11
78.01
69.11 on
59. 75
11.18
44.60 50
39.50
MMD-MP MP (MMD)
85 55.53
13.434423
69.0924.4
59.61g5.49
51.1745.33
44.77=461
39.55
Data (1)
0.12
0.10
0.08
0.06
0.04
0.02
MMD-MP (MPP)
36 03
33.29 as
31 31.4822.0
30.17
29.
29.50+1.98
20.
20.04_1.96
28. 87
MMD-MP (MMD)
35. 45.59 g5.51
33.22
31.43222
30. 16
29.6242.15
29.17
22.04
28. 189
Table 11: Impact of using MPP during testing in terms of test power/100 (0wer/100aa on HC3 given 3, 100 processed paragraphs in training data, where the deep kernel is trained by ChatGPT, Neo-S and GPT3-S but tested on other unknown texts.
Method
Neo-L
GPT-j-6b
GPT4all-i
MMD-MP (MPP)
61.01
55.34
74.93
MMD-MP (MMD)
61.004217
55.6443348
74.99
28

---

Published as a conference paper at ICLR 2024
G REPLACING k. (X,)") WITH (Y, IN MPP
In this section, we conduct an ablation study on HC3 by replacing the (X, (') with Ru Y") in our proposed MPP in Eqn. (5) comprehensively investigate its effectiveness.
We report test power and AUROC in Tables 12 and 13 given 1, 0000 processed paragraphs in training data, where MMD-MP replaces k. X') with (Y, Y") in MPP when training. We observe that MMD-MP has large performance gain on test power compared with MMD-D when training with one-population data, but it degrades test power compared with MMD-MP and this gap widens as the number of population i.e. q increases. This phenomenon arises due to the challenge of optimizing the terms related to S when q increases, as suggested by the first conclusion Section 2.3.
Moreover, from Table 13, in the context of sentence- based detection, the performance of the MMD- MP* exhibits a substantial deterioration in terms of AUROC. One potential explanation for this phenomenon lies in the requirements of single-instance detection, which necessitates a referenced set Sg to calculate the distance between the test instance and the instances S If we optimize ky (y. while neglecting optimization of k (x,x), it could result in effective aggregation of HWT instances while causing the MGT instances to diverge relatively. This could potentially lead to HWT instances "enveloping" MGT instances in relatively divergent manner, thereby resulting in a situation where the distance between those referenced HWT instances and an HWT instance likely greater than the distance between them and an MGT instance. Furthermore, even if interchange the labels of MGT and HWT instances during testing, i.e., replacing the results in the penultimate row of Table 13 with their complements to 100, the performance of MMD-MP= still remains inferior to that of our MMD-MP. In total, training with our proposed MMD-MP achieves consistent and significant performance improvements regardless of AUROC.
Table 12: Test power /100 on HC3 given 1, 000 processed paragraphs in training data, where MMD- MP* replaces hiw (X, X') with k. (Y,Y") in MPP when training.
ChaiGPT Neo-S
ChaiGPT GPT3-S
ChaiGPT GPT2-M GPT2-S
ChatGP Neo-S
ChaiGPT
Method
ChatGPT
GPT3-S
Nco-S
GPT3-S
Neo-L
MMD-D
22.00
24.01
72. =2.2s
74.22=4.06
23.20
00
77.76=2.9
63. 15
MMD-MP* (Ours)
92 02:3
12
75.0 75.01.a7s
81.11
85.71 85.71
76 13
$1.00
75. 75.24
MMD-MP (Ours)
92.31 -2.30
76. 76.35 36.37
85.30+100
39.05-1.4
79 22.5
85.54
29.69
Table 13: AUROC/100 on HC3 given 1, 000 processed paragraphs in training data, where MMD-
MP* replaces Ris (X, X') with (Y, in MPP when training.
CharGPT GPT2-M
ChatGPT Neo-S GPT3-S
ChaiGPT Nco-S
CharGPT Nco-S
CharGP GPT3-S
Method
CharGPT
GPT3-S
Neo-S
GPT2-S
Nco-L
MMD-D D
93 00
91.50+1
81. 10.10_..
89.28
90.22 90.28
85, $5.50
88.07
84.20
MMD-MP* (Ours)
6.03
10.48
g.050
11.46_1.33
11. 1.
18.70
11.57
18.85
MMD MP (Ours)
95. 95.95
94.28 94.28_..07
89.61
90.
93.46_0.0
87.03 87.03
91.25.00.0
86.93.a.52
29

---

Published as a conference paper at ICLR 2024
H MORE RESULTS ON OPTIMIZATION MECHANISM OF MMD-D
In this section, we provide more consequences of E(k) in MMD and their variances under two optimization methods with different q to further verify the conclusions obtained in Section 2.3.
From Figures 8 (a)- we draw two observations: 1) Both [E[ x') and (y, y')] exhibit generally increasing trend, while [l (x, shows relatively minor changes. 2) As the number of populations 9 increases, E[k,, (y, y')] becomes smaller than (x, them widdnning These two observations coincide with those of Figure 2 and Section 2.3.
with the gap between
From Figures 9 (a)-(d), we obviously find that 1) The variance of MMD is predominantly determined
by the variances of [[[ x') and E( (y. yy.y')) while the impact of other terms is relatively minor. 2) When S comprises q 4 or g 5 populations, (y. y')l optimized by MMD-D has significant variance, as well as E ( x')] These two phenomena corroborate the third and fourth observations made in Section 2.3, respectively.
40
MHD-D
MHD D. EXX
40
MND-D Joxt
MMD-D
40
NMD-- Key
NMD-D Jox
40
MHD-O Jay
MHD-D Jux
MMD- C-D--DUN.
NMD-D kvy
MHD-O Joy
EIK//100
NMD-MPP Ice
MHD-AIP kex
MHD MP key
NMD-MP key
MHD-XIP Jey
20
MHD-MP byy
20
MMD-MP kyy
20
-0- NMD-MP kyy
20
MHD-KIP Jny
10
10
10-
10
50
75
50
100
25
100
50
75
Epochs
Epochs
Epochs
Epochs hs
(a) B(k)), q=1
(b) E(k). q-2
(c) E(k). q=4
(d) [Bk)), 9
Figure 8: IB(k) in MMD when training by MMD-D and MMD-MP (ours) with different
200
200
3Cov(IP-kyp)
200
200
Variance/10000
150
Varbkodl
150
Varilooo
150
Varl loo)
150
Var[koe)
Var(2locy)
Vari2lorr)
Vari 2ooy
Var(2koy)
100
20ov
100
2kar)
100
100
50
50
50
50
0
25
50
100
50
75
100
25
50
75
100
25
50
100
Epochs
Epochs
Epachs
Epochs
(a) Var(MMD-D). 9=4 (b) Var(MMD-MP). (c) Var(MMDDD). 9=5 (d) Var(MMDDMM)) 4=5
Figure 9: Variances related to MMD terms when training by MMD-D and MMD-MP (ours) with
different q.
I MORE COMPARISON RESULTS OVER UNKNOWN TEXTS ON HC3
In this section, we conduct more experiments on unknown LLM-text to further evaluate our method. We train the models using texts generated by ChatGPT, GPT-Neo-S and GPT3-S on HC3, and then test on texts generated by GPT-Neo-L. GPT-j-6b and GPT4all-j. From Tables 14-15, our method outperforms the baselines by a large margin test power and AUROC. Specially, our MMD-MP achieves an absolute improvement of 5.20%~~4..311 %6 on test power over MMD-D. Moreover, our method outperforms CE- TE-Classifier by 3.44% 1 and MMD-D by 3.55% t of AUROC on average. These results further demonstrate the superior transfcrability of our method.
Table 14: Test Power/ 1100 on unknown LLM-text.
Table 15: AUROC/100 on unknown LLM-ttxt.
Method
Ne
GPT-j-6b
GPT4all-j
Method
Nco-L
GPT-j-6b
GPT4all-j
CIST-S
C2ST-L
10.99=41s
32,83
27 1712
8.79
40.061330
14.67
CB-Classifier
MMD-O
51.8510.
76.74g1.22
53.80100s
73. 73.43.0.ss
81 21.96
52.92
MMD-O
12.94.152
7.71
17.08
MMD-D D
75.9720.73
74.21
81.6 61.62
MMD D
49.13
41.33:1120
66.79+1.39
MMD-MP (Ours)
61.00=217
55.64
55.6443.45
74.99
MMD-MP (Ours)
79.79.0.17
77.73....m
81933.o11
30

---

Published as a conference paper at ICLR 2024
J COMPARISON RESULTS ON XSUM
In this section, we provide more experimental results on XSum to further demonstrate the effective- ness of our proposed MMD-MP.
J.1 TEST POWER ON PARAGRAPH- BASED DETECTION
In this part, we compare our MMD-MP with the state-of-the-arr (SOTA) two-sample test method for detecting MGT on XSum and show test power on paragraph- based detection in Table 16. In this experiment, use 1, 000 processed paragraphs to train the model. Experimental results in Table 16 demonstrate that our method significantly outperforms other methods on XSum. Specifically, our average even with one training population. Moreover,
MMD-MP surpasses MMD-D by 5.91 MMD-MP outperforms MMD-D by an average of 9.43% 1 on test power over the two training populations and exhibits 8.54% f increase over the three training populations. These results consistent with the results on HC3, demonstrating the effectiveness of our proposed method.
Table 16: Test power 100 on XSum given 1, 000 processed paragraphs in training data.
GPT2-M Noo-S
GPT2-M GPT3-S
GPT2-M M Neo-S GPT3-S
GPT2-M Neo-S
Method
GPT2-M
GPT3-S
Neo-S
C2ST-S
9.04.a.0
38. o2
38.31
20.37
22.43_2.30
25.94
17.67=130
C2ST-I
32 32.64
73.642205
21.07
54.51=176
64.85-2.4
61. 01.964247
60.62:148
MMD.0 -
8.38
16.26
18.7621.5
12.80
13:25_2.28
14.25
10
MMD-D
33..9
73.58.441
76.87
53 58.88:222
57.33-288
63.4332.10
55.06-274
(Ours)
45.51_1144
-0.71-11.0
$1.95
65.81
64.27
70.78
64.8021.00
J.2 AUROC ON SENTENCE-BASED DETECTION
In this part, we compare our MMD-MP with the SOTA ing MGT on XSum in terms of AUROC given 1, 000 processed paragraphs in training data. From Table 17, our MMD-MP consistently achieves the highest AUROC performance compared with other methods. These results further demonstrate the powerful distinguishability of our proposed method when detecting MGTs.
detection method for detect-
Table 17: AUROC/100 on XSum given 1, 000 processed paragraphs in training data.
GPT2-M Neo-S
GPT2-M GPT3-S
GPT2-M
Method
GPT2-M
GPT3-S
Nco-S
Nco-S GPT3-S
Nco-S
Neo-L
Likelihood
61.93g ar
62
43.33
52.74g1o7 o7
54.46 96
50.41
53.33
Rank
72.11
61.811110
63.93
68.18
68.72
66.73
66.72_..m
Log-Rank
65.37
50
1.50
57 07.66621 26
58.
38.78-1.8 1.00
56. 16
57 07
Entropy
51.92
61.4gg10
69.855114
69.85
61.93 g111
59.75_111
63. 1a
61.11g1115
DetoctGP'Td
60.08
41/411122
10.28 21.0
20.09
52.41.1.1
48 g
50.96
DPtectGPT22
66.39
44.45g110
40.61 1o0
g0.97
53.05_111 1-37
48.821
51.90gos4
OpenAI-D
75.11
84.56 84.561om
86.69 10
L0.74
79.91
82.36
76.58
CharGPT-D
41.20
53.60g1.0
31.10ga1.s
53.17 u..
53.69
CB-Classifier
78 48.48
91. 3.
92 89
84.7 20.71
85.61-uaa
87 87.57
$1.06
MMD-O
40.20
02
67.56 00.66
20.75
64.10-1.52
64.40
03. 14.10
MMD-D
72. 12.37
91. 11/18..mm
90.01
82.78 82.73
87.59_..s0
86. 26.52
82.78 82.78_u48
MMD-MP (Ours)
162029
91. 91.73.00.4
93. 03.704..
86 46.450
87. 87.7940.77
27.71
85.70.0.54
J.3 RESULTS ON CHALLENGING UNBALANCED TRAINING DATA
In this part, evaluate our method unbalanced training data containing 2, 000 HWT and 400 MGT training paragraphs over XSum. In Figure 10, we observe that our approach consistently outperforms baselines in terms of test power and AUROC. Critically, our MMD-MP surpasses the test power of 4.85%~9.84%% 1 than MMD-D, highlighting the stability of our method in detecting MGTs under unbalanced training data scenarios.
31

---

Published as a conference paper at ICLR 2024
GPT-M Neo-S
GPT-M
GPT-M Neo-S
100
80
70
70.30
60
55.04
60
60
54.88
50
50
47.00
40
40
40
22.35
20
12.19
20
20
14.31
C2ST.S
C2ST.
MMD-O
MMD
MMD-0
MP
C2ST-S
C2ST-L
MMD-O
MMD-D
MMD
MMD- MP
C257S
C2ST1
MMD.
MMD MMD-MP
AUROC/ 100
90
85.59
87.13
85.04
85.86
88.84 88.87
85
80
80
75
75
70
70
65.36
65
65
65
CE-Classifier
MMD-O MMD-D MMD
CE-Classifier
MMD
MMD. MMD-NN MMD-MP
CE-Classifier
MMD-C
MMD MMD-MP
Figure 10: Test power and AUROC on XSum given 2, 000 HWT and 400 MGT training paragraphs.
K RESULTS ON DETECTING PARAGRAPH-LEVER SENTENCES
It's crucial to note that our method can treat an entire paragraph as a single sentence for detection. However, it's important note that excessively long paragraphs may exceed the processing capabil- ities of the pre-trained LLM. When detecting paragraph-level sentences, those that exceed maximum tokens will be truncated by the LLM. Even under this setting, provide the results of our model to directly test with the random sampled 1, 000 paragraph-level sentences in Table 18. The results still demonstrate the effectiveness of our methods.
Table 18: AUROC/100 on HC3 given 3, 100 processed paragraphs for detecting paragraph-Ievel sentences.
ChatGPT Neo-S
ChatGPT GPT3-S
Method
ChatGPT
GPT3-S
Nco-S
Likelihood
99.34
14.6
83.54
90.7640.06
Rank
91.88
81.67
87.56 87.5640.42
89.52 89,5220.10
86.83_0.55
Log-Rank
99.37 20.12
79.59=0.61
88.00+0.22
93.26 03.26620118
8..01=1.23
Entropy
7.04=
50.35
47.9441.221
28. 28.60
28.73 1.13
DetectGPT-d
94.31 20.03
86.63 86.63440.92
87.31
91. 01.2322.c0
90.43_0.76
DetectGPP-z
96.28=0.00
86.5540.91
87, 87.60.20.85
92.1320.40
91. 91.36=0.73
OpenAI-D
98.75.0.00
99.62
00.4110.00
98.94..0144
08.96_0.044
ChatGPT-D
99. 99.79
69.03
67.40
82.85
85.88_00.5
CB- CE-Classifier
99.13
98, 29.119
96.33.2..00
95.71+0.12
94.59=0.33
MMD-O
76. 1: 1.92
87.34
86.6410.42
81.6310.74
8 81.32_
MMD-D
97. 97.42
99.65.10.07
99.46400.16
98.65
99.26_...m
MMD-MP (Ours)
99.59=0.05
=0.05
99.65
99.01
09.12200.13
32

---

Published as a conference paper at ICLR 2024
L EXAMPLES OF CHATGPT TEXT DETECTION
To gain a clearer understanding of the effectiveness of our approach, we illustrate some examples of ChatGPT text detection, including storytelling, article writing, and rewriting. As shown in Figures 11-13, we observe that most of the contents originating from ChatGPT, such as storytelling and article writing, are readily distinguishable by the detector. Sometimes, for some instances of text rewriting, such as human rewriting of ChatGPT text or ChatGPT rewriting human text (as shown in Figure 14), detection is less straightforward. This aligns with the experimental results in Mitchell et al. (2023).
story about Alice who studyin big city and comes home from school.
Please write short story abouut Alice going res
Alice navigated the burtii ng streets of the city backpack suung over one shoulder and the wel ght of a challenging day a school on the other The crapers towering skys activity felt both exhilar ating and overwhelming the young student...
One evening, Alkc found herself craving for delictous meal, and prospect of exploring a restaurant filed her wi th exchement. She chase quaint
for its edecti menu and coz amblance, and with dventurous spint, she set for solo dining expe
Detection:
Al
Detection:
Please write short story about Alice going Boo's alone play
Please write short story Alice and her broth er going
surny Saturday mor ring, Alices adventurous pirit led her to a new expi oration-sshe ahe was headed for day of olay The journey, usuell taken with a group of fn ends, now had an air of xchement and tinge ervous anlicipation
a bustling Saturday aft emaon. Alice and her you nger brather, Jake, set out shapping adventture with the hum of conversat on, and the distant melody of street musicinn The sbil nga, eager for day ploration navigated throu gh the crowds with enthu sias....
BD
CHATGEF
laughter,
walked the fariilirr path o his home...
Detection:
Al
Detection:
Figure 11: Examples of ChatGPT text detection on storytelling.
33

---

Published as a conference paper at ICLR 2024
Please short story about Alke and her friend 5 going to school
Please wiite short story about Alkc and her friend going mountain dimbin g.
typical Monday mar ning, the city was alive wit h the hustle and bustle of daily life Among the people nevigating he busy streets, Alice walk ed with purpose, her back pack slung over one shou ider. She
One cripp and dear week moming, Alice gather ed her adventurous friend for a thriling escapade mountain climbing ex pedition the towering po semnin Is against the azure sky. eckaning the intrepid to ascend their neights conquer the challeng es that lay ahead...
ovr OFF
way sthool, her friends oined her on the jamiliar ourney...
stood as
Detection:
Al
Detection:
Al
Please write short story about Alice going to the musement
Please write short story about Alcc going play in forest.
One sunny Saturday mor ning, Alice woke up contaglous excitement baliig within her Today w
On a bright and sunny we ekend, Alice wake up with spark adventure eyes. The cit, with its to weriig buidings and cons tant buzz, fel stillig, and Alice yearned for the race nature, Determine d to escape the urban jun gle, she packed a small acipack with snacis, bottle, book...
the day she had been eagerly antic pating p the amusementt park. The city's siylin was pain ted with hues of pink and orange Alice, her eyes sparking anticipation made way the lvvl y carnival grounds...
CHAT
Detection:
Al
Detection:
Human
X
Figure 12: More examples of ChatGPT text detection on storytelling.
M DISCUSSIONS AND FUTURE DIRECTIONS
Although we have empirically demonstrated that our MMD-MP has lower variance of MMD value than the MMD-D method and provides a superior distributional discrepancy estimation when training on data from multiple populations, future research could explore the theoretical explanation of these findings to further justify its effectiveness. Additionally, our findings suggest that even when the kernel is trained on single-population data, our MMD-MP still outperforms MMD-D in terms of MGT detection performance. This observation motivates further investigation into the impact of the variance of data for training deep kernel-based MMD.
Furthermore, in the context of paragraph-based detection, we have not yet considered the inherent issue of independent and identically distributed (Non-IID) text data in paragraph, which could break basic assumption of the MMD tests (Grosse et al., 2017: Carlini & Wagner, 2017). The presence of data dependence within the observations can make appear that two datasets from the same distribution are tested as different, rendering the test meaningless (Chwialkowski et al., 2014; Gao et al., 2021). One potential solution to this issue using wild bootstrap technique (Leucht & Neumann, 2013; Chwialkowski et al., 2014) to resample the MMD value during testing. This approach has already demonstrated success in adversarial detection (Gao et al., 2021), where adver- sarial examples are probably Non-IID since they are always generated with pre-trained classifier.
34

---

heart of winter's ream, the Snow Mountain majestic sentin Published as a conference paper at ICLR 2024
Please write essay he topic of snow mountai
Please write text with e theme of rainbow after t he rain
the rain, when the or clouds part and the un's gentle rays pierce ough the lingering mist. n ost enchanting displays rainbow. This kaleidos cope of colors stretching the sky vivid reminder that even the midst of life's storms, beauty and hope persist.
el its peaks clokedd ristine blanket of glistenin white. As if touched by he frasty breath of giants, the mountain's slopes spa rided under the soft glow the moon, transformin the landscape into therworidly realm.
CHAT GFT
GPT
Detection:
Al
Detection:
Al
Please write paragraph theme the
Please write prose graph on the topic poli
sea, vast and untame stretches to the horizon, endless expanse that hispers tales of andent m ysteries timeless jour neys. its rhythmic dance, orchestrated by the ebb nd flow of tides, echoes th heartbeat of the Earth. As the sun cipss below the harizon, the sea transfor ms into a canvas liquid gold.
In the urban tableau, the presence of the police is sentinel.
oth shield Clad in uniforms that ech authority, they stand guardians of order in the abyrinth of society Each fficer embodies delicate balance, holding the weig responsibility on one shoulder and the mandat to protect other
CHAT
DWNTGPT
Detection:
Al
Detection:
Al
Figure 13: Examples of ChatGPT detection on article writing.
35

---

Published as a conference paper at ICLR 2024
Please rewrite the followin sentences: Our study the first to show that grap consumption impacts eye health hu mans, which ng. especially with a growi ageing population.
Please rewrite followin sentences The world opulation projected k eep growing despite decii ning fertility we estimate the number of infants already peaked n 2017.
In
exdtt
Our research is groundbr eaking it initial monstration of the positiv impact grape consu mptionn on human eye he alth. This discovery is part icularly thrilling, especially considering the increasing aging global populali an.
Despite declining fertiity ates, the global populatio is expected continue growth trajectory. In ct, our estimates suggest number infants reached its highest point 2017.
Detection:
Human
Detection:
AI
Consuming fructose has een connected to decinn in metabolism and minished abilly to control appetite. At the same tim e, fatty foods play crucia role contributing lorie intake, resulting in eight gain.
Despite deciniigg fertiliyy ates, the global populatio expected to continue growth trajectory fa estimates suggest hat the number of infants reached its highest point
[rewriting]AAthough there dedining fertility rate the global population is expected continue gro wing, In fact, our estimate indicate that the of babies reached its hig level 2017.
Trewriting/Fructoss what triggers our metabo ism go low power and lose our control of appetite, fatty foods ocomne the major source of calories that drive weig ht gain.
Detection:
Human
Detection:
Figure Examples of ChatGPT text detection on text rewriting.
36


Reviews:
[{"review_id": "SwJw48fdSS", "reviewer": "Reviewer_bnWc", "summary": "This paper investigates the detection of LLM-generated text based on MMD, a measure of distributional similarity. MMD-based detectors work well for text generated by only a single LLM, but less well when text can come from multiple LLMs. The paper proposes a solution to deal with this.\n\nMMD-based detectors are trained with a loss that involves comparisons between paired text samples that are either both human generated, both machine generated, or mixed. The proposed solution is to drop the comparison between the samples that are both machine generated, as this can cause problems when such samples come from different LLMs. Lots of math is provided to justify this strategy.\n\nThe proposed method is compared to various baselines in a cross-validation setup on a corpus of 3000 human- and machine-generated paragraphs. Two settings are used: comparing the power of a hypothesis test for whether two paragraphs are generated by the same method; and classification performance in detecting whether a single sentence is machine-generated. The proposed method outperforms all baselines on these tasks, and it is robust to generated text coming from multiple LLMs.", "questions": "Nit: human-written text -> HWT, human-generated text -> HGT [choose one]\n\nFigure 1 is quite hard to interpret at this point in the paper.\n\nIf I interpret (3) correctly, this loss is quadratic in sample size, which would seem to limit the quantity of data it is practical to train on. Do you use an approximation to deal with this?\n\nWhere do the different machine-generated populations in 2.2 come from? In what sense is there only a single human population (since there are also lots of different kinds of human-generated texts).\n\nRelated to the above, how is (3) generalized to deal with multiple populations? Do you keep the machine samples separate somehow, or merge them?\n\np6 “... numerous works have shown MMD as a powerful means…” - please provide citations.\n\ns4 What language is the corpus?\n\ns4 What architecture and scale of neural models do you use? Are these the same for your model and all baselines? Which baselines train on the provided corpus, and which don’t? What are the hyper-parameters for the kernel functions? Is there anything about the setting (model scale, data size) that might be optimal for your model but not the baselines?\n\ns4 Please provide more details about how test power was compared for these methods. If it was derived from theoretical estimates for both your methods and C2ST-L, can you be sure both are equally good estimates of the true power?\n\nIn Tables 1-4, ChatGPT has consistently the best results for all methods, indicating that it is easier to distinguish from human text. But one would expect it to be harder, since it is a better model than the others. Do you have an explanation for this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper attacks a crucial problem and it is clearly written.\n\nThe paper is technically very sophisticated, and provides extensive theoretical justification for the proposed methods.\n\nResults on the tested setting are very clearly positive.", "weaknesses": "The paper considers variations in text due to using different LLMs, but it basically ignores variation due to different genres, styles, and human authors. I think this is an artifact of sticking to a single small corpus (in the main paper), where it is easy to generate new LLM outputs by using the provided prompts, but all other attributes have to remain fixed. These other attributes are probably more important in real-world applications.\n\nI am not sure whether experiments on such a small training corpus are meaningful. Due to the cross-validation setup, it is very possible that all models that are trained on the corpus are over-fitting to this setting. Baselines like ChatGPT-D that may be trained on much larger external corpora are therefore at a disadvantage. Another potential disadvantage is that the single-sentence detection test is not very realistic; it would have been better to perform this test at the paragraph level. A final problem with the experiment section is that many details are missing - see questions below.\n\nThe proposed method is quite complicated. Apart from complexity of implementation, a potentially large disadvantage is that the loss is quadratic in training corpus size. The authors don’t address this issue at all.\n\nAlthough it has a lot of supporting math, the main contribution is actually a small modification (dropping a term in the loss), that is quite intuitive. There is nothing wrong with this, but the paper would have been stronger had it focused more on providing this intuition up front, and less on the formal details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cndAm7SqsR", "reviewer": "Reviewer_1AtS", "summary": "This paper first points out the existing optimization dilemma on using vanilla MMD to detect machine-generated text, and then proposes MMD-MP, a multi-population aware optimization method for maximum mean discrepancy. MMD-MP avoids variance increases and improve the stability to emasure the distribution discrepancy.  Relying on MMD-MP, this paper proposes two methods for machine-generated text detection. Experiments were done on different LLMs (GPT-2s, GPT-3s, chatGPT, GPT-Neo) and different LLM settings.", "questions": "- extra parenthesis in section 1 paragraph 2 citation.\n- what is the definition of X’ and Y’ in definition 1? Are they also RVs from P and Q?\n- Metrics — why choose test power, not something like macro-accuracy?\n- Although the appendix mentioned that you use the same decoding method as the detectGPT paper, they actually have a section that test out different settings of decoding method. Can you tell me what parameter and sampling method you are using?\n- I might have a misunderstanding but why only generate MGT with the first 20 prompts? What are these for? Since the evaluation metric paragraph mentioned 100 paragraphs / 1,000 sentences were used for testing, and section 4.2 mentioned mentioned 3,100 paragraphs were used for training.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces MMD-MP, a multi-population aware optimization method, which achieves better machine-generated text detection rate than baseline.\n- This paper has a very clear mathematical formulation of the MMD-MP method, as well as provide reasoning on why they choose to make this optimization on top of vanilla MMD.", "weaknesses": "- Test power is not very straightforward to interpret. I suggest adding an easier-to-understand explanation in additional to the current definition in section 2. In my understanding, the higher the test power number is, the more “confident” the model is to the idea that the two distribution is different, right?\n- Easier-to-read figures:\n    - This might be a nit-pick but figure 1 and 2 are very packed. I understand that the authors want to plot MMD-D and MMD-MP out for better comparison, but maybe consider removing the triangles & squares & circles?\n    - What does the last two column mean in table 1? Is it when $q=3$, so the three “clusters” are human-written text, ChatGPT text, and GPT3-S text?\n- The method is overall, slightly complicated. The experiments when $q=3$ are conducted on “different enough” language models. I wonder if better LMs are included in the comparison (GPT-4, chatGPT), how will the method perform.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fna9T6YXvs", "reviewer": "Reviewer_DBzA", "summary": "The paper addresses the challenge of distinguishing between machine-generated and human-generated text. The authors propose using Maximum Mean Discrepancy (MMD) as a tool to identify subtle distributional discrepancies. To account for variations in text generated by different LLMs, they introduce a novel approach called multi-population aware optimization for kernel-based MMD (MMD-mp), with each population representing the distribution of one LLM. The MMD-mp method is applied in two settings: single sentence detection and paragraph detection, where individual sentences within a paragraph are treated as separate instances. The paper evaluates these settings separately and explores the impact of training data number on performance.", "questions": "1. Does the inclusion of a more diverse range of LLMs' generated text in the training corpus improve or degrade performance?\n2. How does the introduction of less capable LLMs' text into the training dataset, with distributions significantly different from other language models, affect the method's performance?\n3. Is it feasible to treat an entire paragraph as a single sentence for detection, and what is the performance in this setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper's motivation for using MMD to discriminate between highly similar distributions, particularly in short sentences, is compelling. Short sentences often pose a greater challenge due to their simpler structure and wording, and MMD offers an effective solution.\n2. The proposed method is supported by both theoretical analysis and experimental results.\n3. The experimental design is comprehensive. The experiments involving the detection of text generated by unknown or unseen LLMs are interesting. Additionally, evaluating both short and long text is a well-rounded approach. The investigation into how the number of training data instances impacts performance strengthens the effectiveness of the proposed MMD method.", "weaknesses": "One recent SOTA baseline for detecting machine-generated texts is missing [1]. The paper has also shown impressive performance in short texts and needs to be discussed and compared in the paper.\n\n[1]Stylometric Detection of AI-Generated Text in Twitter Timelines, Kumarage et. al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xfleiUOdP", "reviewer": "Reviewer_4WLs", "summary": "The paper delves into the ever-growing challenge of differentiating machine-generated texts (MGTs) from human-written texts due to the increasingly sophisticated capabilities of large language models like ChatGPT. The authors highlight the potential risks MGTs pose, especially in scenarios where accuracy and authenticity are crucial. Their primary approach is based on the exploitation of maximum mean discrepancy (MMD) to discern the subtle distributional discrepancies between MGTs and human texts.\nHowever, a pivotal point of contention is the assertion that directly training a detector using MMD on diverse MGTs leads to a spike in MMD variance. This is due to the potential multiplicity of text populations from various LLMs, which the authors argue could compromise the efficacy of MMD in differentiating the two sample types. In response, they introduce the concept of MMD-MP, a multi-population aware optimization method. This novel solution is said to counteract the variance increases, leading to a more consistent and reliable detection mechanism.\nTwo distinct methods, based on paragraph and sentence structures, were developed relying on MMD-MP. The authors have furnished empirical evidence, demonstrating the superiority of their approach when tested across a range of LLMs.\nWhile the research appears thorough and the problem well-identified, certain areas raise eyebrows. For instance, the premise that MMD variance surges significantly with diverse MGTs warrants further investigation. Additionally, the paper's claim of consistently outperforming all existing baselines across a multitude of LLMs might be viewed with a touch of skepticism until validated by external sources. Nevertheless, this paper's contributions to the realm of MGT detection are commendable and merit attention.", "questions": "Sure, based on the weaknesses and the content of the image, here are two questions:\n\n1. Given the intricate mathematical representations like \n$ J(P, Q; k_u) = MPP(P, Q; k_u)/σ(P, Q; k_u) $\nand \n$ MPP_u(S_p, S_q; k_u) = \\frac{1}{n(n − 1)} \\sum H^{ij} $, \nhow do you envision making the paper's methodologies more accessible to a wider audience, especially those who might not be well-acquainted with kernel-based techniques?\n\n2.  The paper proposes MMD-MP to address the variance issue in MMD, represented by \n$ σ^2_{s} = 4 (E[H^2_{i3}] − E[H^2_{i2}] ) $. \nHow confident are the authors in the robustness and stability of this method when applied to real-world scenarios with vast and diverse text samples? Furthermore, how can the proposed algorithms be adapted to other domains or challenges beyond MGT detection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Innovative Approach to a Contemporary Challenge: The paper addresses the pressing and contemporary issue of distinguishing machine-generated texts (MGTs) from human-written texts, especially given the sophisticated capabilities of modern LLMs. The introduction of the multi-population aware optimization method, MMD-MP, is a novel approach that seeks to enhance the accuracy and stability of MGT detection, providing a fresh perspective on the problem.\n\n2. The authors haven't limited their research to just theoretical propositions; they have put their methods to the test. They conducted extensive experiments across various LLMs, providing empirical evidence of the superiority of their approach. Such thorough testing not only validates their methods but also offers a benchmark for future research in this domain.\n\n3. The paper offers clear theoretical insights, particularly the exploration of the optimization mechanism of MMD and its associated challenges. Propositions and corollaries, such as the asymptotics of MPP and test power, further underscore the research's depth and provide a strong theoretical foundation for their method. The use of statistical properties and theoretical analysis makes the paper robust and comprehensive.", "weaknesses": "1. The paper relies heavily on mathematical formulations, such as the objective function \n$$ J(P, Q; k_u) = MPP(P, Q; k_u)/σ(P, Q; k_u) $$\nand the estimator equation \n$ MPP_u(S_p, S_q; k_u) = \\frac{1}{n(n − 1)} \\sum H^{ij} $, \nwhere  $ H^{ij}=k_u(x_i, x_j) − k_u(x_i, y_j) − k_u(y_i, x_j) $.\nWhile these equations are crucial for understanding the paper's methodology, they can be intimidating and might make the content less accessible to readers who aren't well-versed in mathematical notation or kernel-based methodologies.\n\n2. The paper introduces MMD-MP to handle the variance issue in MMD when dealing with multiple text populations. However, the mathematical representation \n$ σ^2_{s} = 4 (E[H^2_{i3}] − E[H^2_{i2}] ) $, \nwhere \n$ H^2_{i3}, H^2_{i2} $\ndenote different $ H^{ij} $, suggests that the variance calculation is still inherently tied to the distance metrics. This could lead to concerns about the real-world applicability and stability of the proposed solution, especially in scenarios with vast and diverse text samples.\n\n3. The algorithms presented, such as the training deep kernel with MMD-MP (Algorithm 1) and testing with MMD-MP for 2ST (Algorithm 2), are specifically tailored to address the MGT detection problem. Given the intricate mathematical formulations, it may be challenging to adapt or generalize these algorithms to other domains or problems without significant modifications. For researchers or practitioners looking for broader applications, this specificity might be a limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)