DALD: Improving Logits-based Detector without Logits from Black-box LLMs
Authors: Cong Zeng
Shengkun Tang
Xianjun Yang
Yuanzhou Chen
Yiyou Sun
zhiqiang xu
Yao Li
Haifeng Chen
Wei Cheng
Dongkuan Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Detection
Primary area: fairness

Abstract: The advent of Large Language Models (LLMs) has revolutionized text generation, producing outputs that closely mimic human writing. This blurring of lines between machine- and human-written text presents new challenges in distinguishing one from the other – a task further complicated by the frequent updates and closed nature of leading proprietary LLMs. Traditional logits-based detection methods leverage surrogate models for identifying LLM-generated content when the exact logits are unavailable from black-box LLMs. However, these methods grapple with the misalignment between the distributions of the surrogate and the often undisclosed target models, leading to performance degradation, particularly with the introduction of new, closed-source models. Furthermore, while current methodologies are generally effective when the source model is identified, they falter in scenarios where the model version remains unknown, or the test set comprises outputs from various source models. To address these limitations, we present \textbf{D}istribution-\textbf{A}ligned \textbf{L}LMs \textbf{D}etection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection even without logits from source LLMs. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations with minimal training investment. By leveraging corpus samples from publicly accessible outputs of advanced models such as ChatGPT, GPT-4 and Claude-3, DALD fine-tunes surrogate models to synchronize with unknown source model distributions effectively. Our approach achieves SOTA performance in black-box settings on different advanced closed-source and open-source models. The versatility of our method enriches widely adopted zero-shot detection frameworks (DetectGPT, DNA-GPT, Fast-DetectGPT) with a `plug-and-play' enhancement feature. 
Extensive experiments validate that our methodology reliably secures high detection precision for LLM-generated text and effectively detects text from diverse model origins through a singular detector.
Our method is also robust under the revised text attack and non-English texts.

DALD: Improving Logits-based Detector without Logits from Black-box LLMs
Cong Zeng"* Shengkun Tang'* Xianjun Yang²
Yuanzhou Chen
Yiyou Sun"
Zhiqiang Xu' Yao Li Haifeng Chen' Wei Cheng Dongkuan Xu
MBZUAI' University California, Santa Barbara"
University of California, Los Angeles NEC Labs America'
University of North Carolina, Chapel Hill NC State University"
scong.zeng shengkun. tang, zhiqiang. xu)embzuai ac ae (sunyiyou, haifengchen, weicheng)&nec- labs. com yuanzhouchen@cs ucla. edu yaolißad unc.edu
xianjunyanguucsb. edu
dxu27@ncsu, edu
Abstract
The advent of Large Language Models (LLMs) has revolutionized text generation, producing outputs that closely mimic human writing. This blurring of lines between machine- and human- written text presents new challenges in distinguishing one from the other task further complicated by the frequent updates and closed nature leading proprietary I.LMs. Traditional logits- detection methods leverage surrogate models for identifying L----gerettedd content when the exact logits are unavailable from black-box I.I.Ms. However, these methods grapple with the misalignment between the distributions of the surrogate and the often undisclosed target models, leading to performance degradation, particularly with the introduction of new, closed-source models. Furthermore, while current method- ologies are generally effective when the source model is identified, they falter scenarios where the model version remains unknown, or the test set comprises outputs from various source models. To address these limitations, we present Distribution- Aligned LLMS Detection (DALD), innovative framework that defines the state-of -ftth--r-t performance in black-box text detection even without logits from source L.MMs. DALD is designed align the surrogate model's distribu- tion with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations with minimal training investment. By lever- aging corpus samples from publicly accessible outputs of advanced models such ChatGPT, GPT-4, and Claude-3, DALD fine-tunes surrogate models to synchro- nize with unknown source model distributions effectively. Our approach performs SOTA in black-box settings on different advanced closed-source and spurcsource models. The versatility of our method enriches widely adopted zero-shot detection frameworks (DetectGPT, DNA-GPT, Dest-Detec)GPT) with a plug-and-play en- hancement feature. Extensive experiments validate that our methodology reliably secures high detection precision for L.LM- ----eerrrteedd text and effectively detects text from diverse model origins through singular detector. Our method is also robust under the revised text attack and non-English texts.
Introduction
Large language models (LLMs) such ChatGPT[[), GPT-4[]. Llama 3.5 and Claude-3 have profoundly impacted both industrial and academic domains, reshaping productivity across various
Equual contribution. The code are released at http:://giiuub.coommmoooeez////e
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

40
GPT-3
@1
GPT-2 DALD
@F12 GRT2DAID DAU
Usmed
120
15
10
10
Value
Vilue
(a) Human Text
(b) Machine Text
Figure 1: The probability curvatures distribution of the surrogate model (GPT-2), the target model (Llama-3) and the model after alignment (GPT-2_DALD) on human-written passages and machine- generated passages from PubMed dataset.
sectors including news reporting, story writing, and academic research Nevertheless, their misuse also raises concerns, particularly regarding the dissemination of fake news [8), the proliferation of malicious product reviews [9), and instances of plagiarism (10). Instances of AT-synthesized scientific abstracts deluding scientists 12 have raised doubts about the reliability of scientific discourse.
[I0).
Accurate and reliable machine-genn -eeerrttdd detection methods are necessary in order address
these issues [7].
Methods for detecting text generated by Large Language Models broadly categorized into watermarking[II8/ [1] training-baad classifiers 22 26).. and zero-shot detectors. Watermarking methods discreetly embed identifiable markers within the text output, striving to retain the model's linguistic integrity. However, this tactic implementable solely by the model provider. Training- based classifiers, while effective, are costly and often lack the agility adapt to new domains or model updates. Our emphasis is on zero-shot detectors that exploit the intrinsic differences between text written by machines and humans, offering the advantage of being generally training-free.
Most zero-shot detectors primarily depend on analyzing model output logits for detection. Notably, DetectGPT[271 operates probability divergence based upon principles of perturbation theory, while DNA-GPT harnesses reprompting-
probability divergence, and Fast-DetectGPT[2 builds variations in conditional probability distributions. In scenarios requiring the scrutiny of -lx models, these strategies commonly leverage a surrogate model to approximate the behavior of the target model. However, this approach is doubly flawed: firstly, detection efficacy is inextricably linked to meticulously tailored surrogate model, with different surrogate models often necessary for accurate detection across various proprietary LLMs; secondly, the fleeting nature of LLM updates renders past surrogates, once effective, obsolete against new versions. For instance, our analysis of the performance against freshly updated closed-souree models reveals erratic and predominantly diminishing accuracy, as contextualized in Figure 2 with particularly stark declines performance on iterations like GPT-3. 5-1066, highlighting the intrinsic limitation of static surrogate models in adapting to LLM progressions.
Detss-DeeecGGTT122 [29], using GPT-Neo- as a surrogate,
In our study, we seek to address the following pivotal inquiries: 1) Can we devise a feasible, cost-effective strategy to refine the probability dis- tribution similarity between the sur- rogate model and opaque black-box LLMs? 2) Does enhancing the align- ment of the surrogate model's proba- bility distribution with that of the tar- get black-box LLM improve detection outcomes for current logits-based de- tection methods? 3) Is it attainable to develop universal detection model capable of adapting swiftly to updates across various target LLMs? Addressing the first question, findings, demonstrated in Figure
1.0
AUROC
0.8
de Publle CPT3.5
Xsum CPT-3.5
Wiiking 3.
Publed GPT4
0.6
Xsum GPT4
Wetting Gan-4
GFF OP125 4P122-22206 Getz 5-i22 0954--3112 G974-1o4 641-00223 Geliacce
Model Version
Figure 2: The performance comparison of static surrogate model on different target models including ChatGPT (GPT. 3.5) and GPT-4. The results based on Past-DetectGPP with GPT-Neo- 2. 7B as the surrogate model.

---

reveal that our optimized surrogate model (GPT-2 DALD) mirrors the distribution of the target model more closely, in contrast the original surrogate model's significantly divergent distribution from the target model.
In this paper, we introduce an innovative yet straightforward distributtin-alignedd framework for black-box LLM detection, dubbed DALD (Distribution-Alieee) LLM Detection). Our methodology focuses on synchronizing the surrogate model's distribution with the proprietary target model's distribution. Concretely, we accumulate a compact dataset (<10K samples) from the publicly shared outputs of leading models and subsequently fine-tune our surrogate model using this dataset to better approximate the target model's distribution. We provide the theoretical analysis of the surrogate model distribution alignment in Appendix Our methodology builds upon the following observation:
In logits-based detection methods, a surrogate model that closely mirrors the probability distribution curves of the target black-box LLM is instrumental enhancing detection accuracy.
We posit that this observed effect stems from the foundational assumptions inherent in logits- bbaed detectors and proceed to examine the ramifications of this postulate in tackling the third question.
To sum up, contributions
follows:
The introduction of DALD, a framework that significantly improves the performance of surrogate models in detecting LLM-generated text generated by both closed-soure and open-source models.
DALD's unique ability to enhance detection without reliance on knowledge of the source model a game- game-changer in domain where the source is often unknown.
The capability of a single detector, enabled by DALD, to accurately identify text from varying sources, democratizing detection across diverse LLM outputs.
DALD's agility in keeping pace with rapid updates of L.Mss, ensuring the latest models fall within its detection capabilities without extensive retraining.
2 Related Work
Detection of LLMs- -enerreeeeeeed Text. The burgeoning capabilities of advanced large language models (LLMs) underscore the imperative for robust methodologies aimed at detecting these models. Specifically, the detection is to distinguish whether a given text originates from a language model on the condition that the model is known (White-box [2] or unknown (Black-box: (o0)) The earlier work focused on feature-based based methods, Bike|B1 While in the era of LLMs, the taiiing-bbsee methods 31 37] are aroused to counter with LLMs's 's strong ability to produce high-quality text. They usually involve training a binary classifier using text generated by AI or humans. Besides, zero-shot detectors leverage the inherent statistical feature differences between LLMs and human- generated text without requiring training, including probability curvature (DetectGPT(27)) N-gram divergence (DNA-GPT(8)) and conditional probability curvature (Fast-DetectGPTT (29)), the editing distance of the output[88] 33). and style representations 39], enhancing their ability to adapt to new data distributions and source models.
Black-box Detection. Given the proprietary nature of the latest LLMs 2 6). there is a critical need for effective black-box detection methods. Present techniques falter when direct access to the source model is restricted. The training-basee methods, like OpenAl text classifier (00, GPTZero 22). G3detector[41] and GPT-Sentinel domains and source models during training. thereby lacking generalization ability and robustness on model updates. For zero-shot methods [43] 388 27
usually closely adhere to the specific distributions of text
47] in the black- 44 45 34 46 [4]
box detection settings, they usually rely on a surrogate model for scoring. However, the efficacy of these surrogate models often falls short compared to white-bor detection, where access to the source model is available. Moreover, these detection frameworks suffer from diminished accuracy when language models undergo updates[SSD [50), which intrinsically evolve through exposure to varied datasets and human input(30), This study presents an innovative black-box detection method for L.L.--enerated text, greatly enhancing surrogate model performance while adeptly accommodating the rapid evolution of L.LMs. Our approach diverges from conventional training-intensivg techniques by requiring only minimal dataset for effective training.
3

---

Fast-DetectGPI
Passage "Unempleyment foll 1.69 million between Jamuary and March, down 2, 00000 from...
Sample
Scoring
Compare
pbx Le) Porgmeati poG 120
x
Sample Modell
Misallined Surrogate Model
by Human
X
Ours
Passage x: "O "Iiemppoymentt foll 1. 1,69 million between Jamuary and March, down 2,000 from...
Sample
Scoring
Compare
by A1
Sumple Model
Aligned Surrogate
Ditribution-
1(8
Surrogate Model
Parameter-Effiee Efficient
Corpus generated from source model
Tuning
An overview of our proposed DALD framework, Our method aligns the distribution of the surrogate model and the target model.
Figure 3:
Method
3.1
Task and Settings
Our task to detect whether the given input passage x [.1...., xL] (L is the sequence length) is produced by an Al model fur human, which be considered as a binary classification task. Typically, there are two different task settings for LLM detection, namely white-box and black-box detection. In the black-box setting, only have access the generated text, treating the language model as a "black box" where we input text and receive output without knowing the internal workings or probabilities. In the white-box setting, have additional information about the model, specifically the output probabilities p( [11211))) for each token at each position in the text. However, in the practical scenario. is usually difficult get access the source model, especially widespread but closed-source models such as ChatGPT, GPT-4 and Claude-3. Therefore, we focus on improving the black-box detection without any access to the source model logits in our setting.
3.2 Logits-based Detection Methods
Logits-bass LLM detection methods compute a metric by discrepancy gap hypothesis of humans and machines for classification. For example, based the observation that the LLM-generatee text occupies negative curvature regions of the model's log probability function, DetectGPT 27 proposes utilize the source model for scoring, which refers the white-box settings. Following DetectGPT, Past-DetectGPPT replaces the perturbations- based sampling method with conditional probability sampling accelerate the inference speed and improve the detection performance. Formally, given an input passage a and the target source model pa, Fast-DetectGPT chooses another accessible but open-sourced model s9 for scoring, which is called the surrogate model. Together with a sampling model l.p Fast-DetectGPT defines the conditional probability p(ā|2r) as
II so (o(((()))))
where Z is sample generated by the sampling model y.. The detection process typically consists of a three-stage procedure. The sampling step uses a sampling model to generate alternative samples T conditioned on based on the next token prediction. Following the sampling step, the process proceeds by calculating the conditional score. The conditional probability can be obtained through a single forward pass of the scoring model, utilizing the input. All the conditional probabilities of samples can be obtained in the same predictive distribution. Finally, compare conditional probabilities of the passage and samples to calculate the curvature.
The Challenge. block-box settings, selecting the appropriate surrogate model in black-box settings is crucial for achieving accurate and reliable results since there is distribution misaligmment

---

between the surrogate model and the target model. Poorly chosen surrogate models may lead to bad results and a lack of explainability. Besides, with the closed-source trend of newly published LLM models, the performance can drop significantly when applied to new and advanced models, which can limit their utility and effectiveness. How obtain a surrogate model that can fit the target models, especially closed-source models is challenging task in black-box settings.
3.3 Distribution-Aligan Black-Box Detection
Misalignment of Surrogate Model and Target Model. Our method is motivated by the observation that there is distribution gap between the given surrogate model and the target source model as shown in Figure The goal of our method to obtain a surrogate model to approach the distribution of the target model by utilizing the texts generated by the target model. To achieve that, we propose a novel and simple framework to train a distribution-alignen surrogate model, which outperforms SOTA black-box methods with a small-size dataset (<10K). The architecture of our method is shown in Figure 3 Our framework consists of two steps in total. The first step is to collect small-size training data generated by the source model from the publicly shared outputs. With the training dataset, finetune the surrogate model to align the distribution of the source model.
Alignment Data Collection. Given the target model ftar and surrogate model fsur. in order to align the distribution of the surrogate model and target model, we collect small-size dataset S= ((P., } for specific target model, referred as alignment dataset, where N refers to the number of collected samples, P. is the text for prompting and X; is the corpus generated by the target model ftor- The model version of the collected data should be exactly the same as the target model, especially for closed-source models such as ChatGPT and GPT-4. For example, if the test data is generated by GPT-4-065 4-0133, then all of the texts in the dataset S should also be generated by GPT-4-0613. We utilize the collected dataset S to finetune the surrogate model fsur align the distribution with target model ftar.
ceributtion-Aligned Surrogate Model Training. As illustrated in Figure our approach expands the scoring step of previous logits- based methods such as Fast-DetectGP? by incorporating an additional surrogate model finetuning step. Given the surrogate model fsur, we construct the Low- Rank Adaptation () of surrogate model fsur for faster and more stable fine-tuning. The LoRA model f is trained with collected dataset while the parameter of the original surrogate model faur is frozen. With collected dataset S {(P,, X;) x,)) f==1 where K₁ is the number of samples, we concatenate the prompt and generated text as = [P, X]. The model Sear+ utilizes the tokenized as input and is trained in a supervupe learning manner. The training objective of our fine-tuning
is:
I(P)+(()
max
Σ Σ
logp(u Wett sur + O),
D [PX I- RP)+1
where l(X) denotes the length of a passage X, and y is the next token be predicted. In order to disable the influence of the prompt, we follow typical instruction tuning to mask the gradient of the prompt. As shown Figure [ after training, the misaligned model generates a similar distribution as the target source model, demonstrating the effectiveness of our method. Following that, the distribution-alignes surrogate model can be utilized to compute the logits for downstream decisions.
Under an assumption on the sample complexity of fine-tuning with the above loss function, we theoretically demonstrate the effectiveness of fine-tuning on approximating the target model in the following theorem, using conditional probability curvature from Fast-DetectGPT [29]:
Theorem 1. With fine-tuning sample size K; 2(poly(A/)))))) ())),, with probability 1 - 8, we have that given a text segment X with length the conditional probability curvature between the two models is bounded by
[XX fsur) f(x) s A/3.
A detailed proof of this theorem can be found in Appendix 6 Given the hypothesis that there is a positive gap A in conditional probability curvature between human-gene -emaan-geeerated text and machine- generated text, the corresponding gap calculated from the surrogate model will still be significant.
5

---

Table Detection accuracy comparison on three source models ChatGPT (GPT-3.C. 3..5 -----urr-00335 03011,
GPT-4 (GPT-4-0613) and Claude-3 (cl aude-3-opus-20220223 Our method surpasses previous methods on all passages generated from different source models.
ChatGPT
GPT-4
Claude-3
Method
PubMed
PubMed
XSum
Writing
PubMed
XSum
Writing
RoBERTa-bass
0.6298
0.5327
0.7475
0,5186
0.4961
0.8564
0.6707
RoBeRTa-large
0.7168
0.5898
0.6830
0,3800
0.5334
0.7888
0.5178
Likelihood
0.8924
0.8103
0.8096
0.8528
0.8543
0.9383
0.9542
Entropy
0.2877
0.3036
0.4451
0.3545
0.2940
0.3856
0.1844
LogRank
0.8847
0.7996
0.8041
0,8303
0.8481
0.9420
0,9437
LRR
0.7793
0,6860
0.7405
0.7212
0.7468
0.8989
0.8761
NPR
0.6917
0,5950
0,6726
0,8192
0.6610
0.8584
0,9167
Detect-GPT
0.6626
0.5806
0.6940
0.8270
0.6562
0.8652
0.9232
DNA-GPT
0.7788
0.7171
0.7100
0.7849
0.7442
0.9410
0,9471
Past-DetectGiP
0.9309
0.8179
0.9136
0,9521
0.8900
0.9828
0.9445
DALD (Ours)
0.9853
0.9785
0.9954
0.9980
0.9630
0.9867
0.9981
4 Experiments
4.1 Setups
Datasets & Evaluation Metric. We follow Fast-DetectGPT using four datasets in the black-box detection evaluation, including Xsum (2). WritingProoppss [53] WMT-2016[54] and PubMedQA (55). We randomly sample 150 examples of each dataset as human-written texts. Then based the samples, we prompt the target closed-source models by API to generate the corresponding texts using the 30 tokens of human- numan-written written text as the machine- generated machine-generated text. For text diversity and quality, we employ temperature of 0.8 which is the same setting in Fast- Dest-DetectGPT. For the training dataset, we collect the corpus from the publicly shared outputs of leading models. Following previous works (29), we compute the accuracy in the area under the receiver operating characteristic (AUROC) to evaluate the performance of all methods. We also provide the area under the precision and recall (AUPR) in Appendix 8.6
Source & Surrogate Models. To validate our idea in black-box detection, we include the most advanced closed-source LLMs from OpenAI: ChatGPT, GPT-4, and Claude-3 from Anthropic. Since these models keep being updated by their owner company, we use the version GPT-3. 5-turro--3000 for ChatGPT, GPT-4-0613 for GPT-4, and
-222422222 for Claude-3 if not specified. We utilize L1 Llama2-7B as the surrogate model in Table Note that our method can be adapted to any open-source model. Therefore, we provide our results on other surrogate models in Table
Baseline Methods. We consider training-based baselines and zero-shot baselines. We mainly consider three strong baselines for black-box detection: (P7))) DNA-GPT(28) and Fast- DetectGPT(29) Detect-GPT. Detect-GPT applies T5-3B 56 sampling model to generate perturbed texts and utilizes GPT-Neo-22 2B as a surrogate model to compute the probability curvature of perturbed texts. After that, perturbation discrepancy is obtained to determine whether the given text is generated by Al or humans. Mestt-DteccGPTT uses GPT-J-6B J-6B(58) and GPT-Neo- 7B the sampling model and surrogate model respectively to compute the conditional probability curvature. Finally, DNA-GPT utilizes GPT-Neo-2. 7B as a surrogate model to regenerate the texts for metric computation. Details about other baselines are described in Appendix 7
Implementation Details. We collect the data of ChatGPT and GPT-4 from WildChat 59 while the data of Claude-3 is generated by calling 3laude-3 using the prompts from WildChat. During training. we randomly choose 5K prompts and responses. We applied the instruction tuning to model training to ignore the humteewritten prompts. For surrogate model training, we apply parameter-efficient fine-tuning (PEFT) via Low-Ran Adaptation (LoRA). We do not tune the hyperparameters carefully. Therefore, more details about training parameters can be found in the Appendix 7 For training time, method finetunes LI amam- 2- with 5K samples on 4 A6000.

---

1,00
1.00
1,00
0.75
0.75
0.75
TPR
0.50
0.25
0.25
0.25
Pral CeLOOet
Past DitentP
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
FPR
(a) XSum
(b) Writing
(c) PubMed
Figure 4: The FPR-TPR curve of different methods on XSum, Writing and PubMed dataset. The results show that our method achieves highest score low FPR compared with DNA-GPT and GP.t-eeecctGPT.
1.0
1.00
0.9
AUROC
0.95
AUROC
0.8
(eem-GGT---LL-A MMM-2
Wreing-GPT---LL GPT-G-P-------- 2
Pu P- ---------1L MM-2
0.7
GPT 3
0.90
PubMed GPT
2000
4000
2000
4000
2000
4000
Training Dataset
Training Dataset Size
Training Dataset Size
(a) XSum
(b) Writing
(c) PubMed
Figure 5: AUORC results from our fine-tuned surrogate model with different training dataset size,
4.2
Main Results
Black-Box Geneeaee Text Detection. We compare our method with mainstream black- box LLM detection methods Table The detection accuracy shows that our method achieves the best performance compared with other methods including Detect-GPT, DNA-GPT and Fast- DetectGPT. Moreover, is noteworthy that our method obtains more than 99% AUROC on XSum- GPT-4, Writing-GPT-4 and Writing- -ertin--Claude3... As shown in Figure the ROC curve for DALD achieves the highest TPR at the same FPR across all three datasets compared with DNA-GPT and GaS, indicating superior performance. Besides, comparing the performance across different datasets in previous methods, we can observe that the Writing dataset is much easier to detect while PubMed is the hardest. The possible reason is that PubMed specific dataset while the related corpus not comprised of surrogate model pre- pre-training. However, method gains significant improvement upon Fast- Dist-DeteetGPT. For example, on TubMed-GPT- Fast- Dest-DetectGPT only obtains 0,8179 AUROC while our method achieves 0.9785, demonstrating the effectiveness of distribution alignment in our method.
much more medical-
4.3 Experimental Analysis
Dataset Size. Our method requires only small amount of data for training. The model converges quickly with the small-size dataset. To show the training efficiency of method, we provide the results with various training dataset sizes in Figure For each dataset, we use GPT-4 as the target source model while Llama-2-77 and Llama-3- increasing amount of training data, the performance increases rapidly and stays stable with more data. Moreover, with around 500 (up to 1000) training samples, the baseline on both Llama-2 and Llama-3. Besides, our method achieves its best performance and exceeds the baseline with around 2000 training data, which indicates that with only a little training effort, our method can achieve incredibly better performance than the bascline, even more than 99%.
as the surrogate model respectively. With the
performance of our method matches

---

Table 2: The results comparison of our method trained with the combination of different data sources. Our method achieves comparable results with more data sources, demonstrating the generalizability of our method and potentially leading to training universal surrogate model for all closed models. t: Train the surrogate model separately each test set. *: Train one surrogate model for all test sets.
ChatGPT
GPT-4
Claude-3
Method
PubMed
PubMed
XSum
Writing
PubMed
XSum
Writing
Baseline
0,9051
0.7995
0.7072
0.9299
0.8877
0.9143
0.9248
DALD T(1 source)
0.9853
0.9785
0,9954
0,9980
0.9942
0.9994
0.9993
DALD *(1 source)
0,9829
0.9785
0.9954
0.9980
0.9875
0.9993
0.9977
DALD "(2 sources)
0,9832
0,9803
0.9981
0.9986
0,9875
0.9994
0,9976
DALD *(3 sources)
0.9827
0.9809
0.9968
0.9985
0.9864
0.9996
0.9982
Generalizability. Our method aligns with the distribution between the surrogate model and source model with the texts generated by source models. In this section, we further explore the generalizabil- ity of our method where we follow several settings: 1) train the surrogate model separately using the data from the corresponding source model, 2) train the model with the single data source and evaluate it on unknown source models (one-for-all) 3) train the model with mixed data sources. We conduct group of experiments following the settings, shown in Table 2 the surrogate model and a total 5K training data in all settings. The dataset with a single data source includes 5K texts from the corresponding source model in the first setting and only GPT- in the one-for-all setting while two data sources consist of 2.5K ChatGPT and GPT-4 texts, respectively. Three data sources refer the combination of 1.3K texts each from CharGPT, GPT-4 and Claude-3. Surprisingly, the models trained with more data sources achieve better accuracy on GPT-4. However, the performance on PubMed- PCM only shows negligible degradation in one-for-all and mixed data source settings. The superior performance in the one-for-all setting implies the surrogate model trained with DALD can be extended to texts of unknown source models. The results demonstrate the generalizability of our method, leading to training universal surrogate model for all closed-sourgs models and detecting the machine- generated texts without knowing the model source. Finally, the results in the one-for-all setting imply current closed-source models tend to have a similar distribution. Evaluation results on more unknown source models can be found in Appendix 8.2
We use Llama2-7B as
eachine-generated
Surrogate Model Selection. Our method can be adapted to any open-source surrogate model. We evaluate our method with different surrogate mod- els including on GPT-NEO-2. 7B, Llama2-7B, and Llama3-8B. The experiments are conducted based on Past-DetectGPT and trained with 5K training data. The results are shown in Table with details in each dataset. In all, compared with the original surrogate models, the surrogate models with DALD obtain much higher accuracy. For example, Llama-2 and Llama-3 only ob- tain 0,8876 and 0.7764 on PubMed-Claude-3 while their counterparts trained with DALD achieve 0.9424 and 0.9192. The improvement across various surrogate models suggests that our approach is compatible with range of surrogates, rather than just a particular carefully chosen surrogate model.
Table 3: Results comparison of our method with different surro- gate models on Claude-3. The performance improvement with our method on different surrogate models shows that our method can be adapted any open-source surrogate model.
Claude3 XSum
Surrogate Model
PubMed
Writing
Llama2-7B
0.8876
0.9132
0.9243
Alama2-7B(with DALD)
0.9424
0.9773
0.9962
Llama3-8B
0.7764
0.9390
0.8827
(lwthh-8B(with DALD)
0.9102
0.9892
0.9967
GPT-Neo-2.7B
0.8900
0.9828
0.9445
GPT-Neo-2.7BBwith DALD)
0.8997
0.9852
0.9515
Ablation Study. We conduct a group of ablation studies on several datasets, as shown in Table 4 Since our method can be adapted to any previous logits- based methods such as DNA-GPT and Fast- DetectGPT, the ablation study is conducted on top of them to further demonstrate the effectiveness of our method. We choose Llama2-72 as the basic surrogate model for all experiments. We compare the results of the baseline model and the baseline model trained with our framework. In general, our method boosts the performance of the baseline model at different scales. For example, DNA- GPT achieves 0.8947 accuracy score on PubMed-GPT-4 while with the surrogate trained by our

---

Table 4: Ablation study. We report the results comparison of the baseline method and the method
with our DALD. The improvement upon all baselines shows the effectiveness of our DALD.
ChatGPT
GPT-4
Claude3
Method
PubMed
PubMed
XSum
Writing
PubMed
XSum
Writing
Detect-GPT
0.6260
0.5291
0.6689
0.799]
0.6472
0.9184
0.9306
Detect-GPT- + DALD
0.7388
0.7034
0.8318
0.9076
0.7550
0.9569
0.9568
DNA-GPT
0.9547
0.8947
0.6980
0.8537
0.9500
0.9359
0.9648
DNA-GPT + DALD
0.9932
0.9879
0.7524
0.9048
0.9711
0,9391
0.9675
Fast-DetectGPT
0.9309
0.8179
0.9136
0.9521
0.8900
0,9828
0.9445
Fast-DetectGPT DALD
0.9853
0.9785
0.9954
0.9980
0.9630
0.9867
0.9981
method, DNA-GPT obtains 0.9879 on PubMed-GPT-4. Moreover, we gain similar conclusion on Fast-DetectGPP. For instance, on PubMed-GPT-4, the original Fast-DetectGPT only has 0.8179 accuracy. However, after training the surrogate model with our method, it achieves 0.9785 accuracy on PubMed-GPT-4, On the one hand, the improvement in baseline shows the effectiveness of our methods. On the other hand, the improvement across different methods demonstrates that our method can be utilized any logits-based model to boost their performance on black-box detection.
Non-English Defection. Recent work 600 finds that cur- rent detectors are biased for non-English languages, which hinders the application of L.M detection for non-
Table 5: Results comparison on Non- English texts.
English languages. Following (28), we choose English and German splits of WMT-2016 to test the ability of our method in German. We select 150 instances as human-written texts and use the first 30 tokens to regen- erate by calling GPT-4 API as machine seecinn-geenrrte texts. During training, randomly select 1K German samples
Method DNA-GPT DALD (Ours)
German-GPT-4
0.7851
0.8814
0.9955
generated by GPT-4 from WildChat (59] and trained for 5
epochs. As shown in Table our method achieves the highest accuracy (> 99%) German detection compared with DNA-GPT and Fast- Dett-etectGPP.. Due to the aug-and property, our method can be further used to eliminate the bias in other Al detectors.
Adversarial Attack. In practical situations, machine-gend geeeeiieeggeeteedd corpus often modified and revised by users another language model. We consider the modified samples as adversarial samples. Evaluating L.M detectors with adver- sarial samples is important to real-world appli- cations. Following(27 (2)) we randomly mask r% tokens with 5-word spans in 150 in- stances from the PubMed dataset regenerated by GPT-4 and apply T5-3B to do the mask-filling task to generate adversarial samples. Experi- ments with different mask ratios are conducted, specifically r% {0.1, 0.2, 0.3, 0.4, 0.5) and results are shown in Figure We compare the results of DNA-GPT, Fast- Detseetet and their counterparts with our method. In each group, with the enhancement of our method, the model achieves better results on all mask ratios. Moreover, the models with our method obtain even better accuracy the highest mask ratio compared with the original models samples without adversarial attack.
1.0
0.9
AUROC
0.8
0.0
0.1
0.4
Ratio of Revision Figure 6: Results comparison samples with the adversarial attack. The performance improvement with our method on different methods shows that our method is robust to adversarial attacks.
Open-source Model Detection. In addition to detecting texts from closed-source models, we also evaluate our approach on the open-source model Llama-3, Llama-3. [6] and Mistral (62] comparing with DNA-GPT and Fast-DetectGPT. We follow similar settings as closed-source models and fine-tune Llama2-7B 7B as surrogate model. The results are shown in Table where our method works best detection performance on three models compared to other methods, showing the effectiveness of method on both closed-source and open-source models.

---

Table 6: Results on open-source models Llama-3, Llama-3.1, and Mistral across three datasets:
PubMed (PM), XSum, and Writing. We compare DALD with Fast-DetecGPP((ass)
Llama3-8B
Llama3.1-8B
Mistral-7B
Method
PM
XSum
Writing
PM
XSum
Writing
PM
XSum
Writing
Fast
0,9120
0.9845
0.9906
0.8668
0.9914
0.9958
0.6880
0.7931
0.9211
DALD
0.9352
0.9995
0.9972
0.9059
1.0000
0.9998
0.7733
0.8822
0.9573
Conclusion
The rapid evolution of potent Large Language Models (LLMs) underscores the critical necessity for robust black-box detection methods. However, previous methods which rely on surrogate models, suffer from performance degradation, especially with the frequent updates of closed-source models. Our contribution addresses this shortfall by significantly aligning the distribution of the surrogate model and source model. Additionally, we introduce plug-and-play approach for logits-based detectors, ensuring seamless integration. This method remains versatile across diverse text sources or unknown sources, adapting the swift evolution of LLMs. In conclusion, our innovations offer compelling solutions to the urgent demand for effective black-box detection methods within the realm of LLM development, bridging critical gaps in current methodologies.
10

---

References
[1] OpenAL. OpenAl Models GPT3.5, 2022.
[2] OpenAL. Gpt-4 technical report, 2023.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[5] Meta. Meta Models - LlaMa3, 2024.
[6] Authropic. Anthropic Models Claude3, 2024.
[7] Muneer M Alshater. Exploring the role of artificial intelligence in enhancing academic performance: case study of chatgpt. Available at SSRN 4312358, 2022.
[8] Alim Al Ayub Ahmed, Ayman Aljabouh, Praveen Kumar Donepudi, and Myung Suh Choi. Detecting fake news using machine learning: A systematic literature review, arXiv preprint arXiv:2102 04458, 2021.
David Ifcoluwa Adclani, Haotian Mai, Fuming Fang, Huy Nguyen, Junichi Yamagishi, and Isao Echizen. Generating sentiment-preseree machine-based detection. In Advanced information networking and applications: Proceedings of the 34th international conference on advanced information networking and applications (AINA-2020), pages 1341-1354. Springer, 2020.
fake online reviews using neural language models and their human-and
[10] Jooyoung Lee. Thai Le. Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637-3647, 2023.
[11] Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. Comparing scientific abstracts generated by chatgpt original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv, pages 2022-12, 2022.
[12] Holly Else. Abstracts written by chatgpt fool scientists. Nature, 613:423 423, 2023.
[13] Souradip Chakraborty, Amrit Singh Bedi, Sicheng 7hu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.047335 2023.
[14] Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Tyyer. Paraphrasing evades detectors of ai-generated text, but retrieval effective defense, 2023.
[15] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint urXiv:2303.11155 2023.
[16] Liam Dugan, Alyssa Hwang, Filip Trhlik, Josh Magnus Ludan. Andrew Zhu, Hainiu Xu, Daphne Ippolito, and Chris Callison- Burchh Raid: A shared benchmark for robust evaluation of machine-genan generated text detectors, 2024.
[17] Li Lin, Neeraj Gupta, Yue Zhang, Hainan Ren, Chun-Hao Liu, Feng Ding. Xin Wang. Xin Li, Luisa Verdoliva, and Shu Hu. Detecting multimedia generated by large models: A survey. arXiv preprint arXiv:2402.n 00045, 2024.
[18] Sahar Abdelnabi and Mario Fritz. Adversarial watermarking transformer: Towards tracing provenance with data hiding. 42nd IEEE Symposium on Security and Privacy, 2021.
[19] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301. 10226, 2023.
[20] KiYoon Yoo, Wonhyuk Ahn, Jiho Jang, and Nojun Kwak. Robust natural language watermarking through invariant features. arXiv preprint arXiv:2305.0149 01904, 2023.
[21] Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. Cryptology ePrint Archive, 2023.
[22] Edward Tian. Gptzero: An ai text detector, 2023.

---

[23] Vivek Verma, Eve Fleisig. Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwritten by large language models, 2023.
[24] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: wwtp:::00000000 2019.
[25] Kangxi Wu, Liang Pang. Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. LImdet: A large language models detection tool, 2023.
[26] Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Weiming Zhang, and Nenghai Yu. Gpt paternity test: Gpt generated text detection with gpt genetic inheritance. arXiv preprint arXiv:2305. 12551, 2023.
[27] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectept: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301. 2023.
[28] Xianjun Yang. Wei Cheng. Linda Petzold. William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint arXiv:2305 17359, 2023.
[29] Guangsheng Bao, Yanbin Zhao, Zhiyang Teng. Linyi Yang. and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130, 2023.
[30] Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. survey on detection of lens-reneeated content. arXiv preprint arXiv:2310. 15654, 2023.
[31] Sehastian Gehrmann, SEAS Harvard, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. ACL 2019, page 111. 2019.
[32] EA Grechnikov, GG Gusev, AA Kustarev, and AM Raigorodsky. Detection of artificial texts. RCDL2009 Proceedings. Petrozavodsk, pages 306-308, 2009.
[33] Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-11, 2008.
[34] Zhijie Deng, Hongcheng Gao, Yibo Miao, and Hao Zhang, Efficient detection of Ilm-generated texts with bayesian surrogate model, 2023.
[35] Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. Seqxgpt: Sentence-level ai-generated text detection. arXiv preprint a000v:23/0.08903, 2023.
[36] Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu. Origin tracing and detecting of llms. arXiv preprint 2rXiv:2304.14077 2023.
[37] Zhen Guo and Shangdi Yu. Authentigpt: Detecting machine-generatee text via black-box language models denoising, 2023.
[38] Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting, 2024.
[39] Rafacl Rivera Soto, Kailin Koch, Alcem Khan, Barry Chen, Marcus Bishop, and Nicholas Andrews. Few-shot detection of serchine-generated text using style representations, 2024.
[40] OpenAL. Al text classifier, Jan 2023.
[41] Haolan Zhan, Xuanli He, Qiongkai Xu, Yuxiang Wu, and Pontus Stenetorp. G3detector: General gpt- generated text detector. arXiv preprint arXiv:2305.1 12t.:22005.1.00 2023.
[42] Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Raj. Gpt-sentinel: Distinguishing human and chatgpt generated content. arXiv preprint ar 2rXiv:2305.0790 07969, 2023.
[43] Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping. and Tom Goldstein. Spotting Ilms with binoculars: Zero-shot detection of machine-generatee text. 2024.
[44] Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for zero-shot detection of machine-generee text, 2023.
12

---

[45] Saranya Venkatraman, Adaku Uchendu, and Dongwon Lee. Gpt-who: An information density-based machine-generaaee text detector, 2024.
[46] Yuhui Shi, Qiang Sheng, Juan Cao, Hao Mi, Beizhe Hu, and Danding Wang. Ten words only still help: Improving black-box ai-generated text detection via proxy-guided efficient re- sampligg 2024.
[47] Niloofar Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg- Kirkpatrick.. Smaller language models are better black-box box wacchine-eeneated text detectors, 2024.
[48] Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Serguei Barannikov, Irina Piontkovskaya, Sergey Nikolenko, and Evgeny Burnaev, Intrinsic dimension estimation for robust detection of ai- ai-generated texts, 2023.
[49] Xianjun Yang. Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang. and Wei Cheng, Zero-shot detection of mechine-generated codes, arXiv preprint arXiv:23100051000 2023.
[50] Charlotte Nicks, Eric Mitchell, Rafael Rafailov, Archit Sharma, Christopher Manning, Chelsea Finn, and Stefano Ermon. Language model detectors are easily optimized against. In The Twelfth International Conference on Learning Representations, 2024.
[51] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, Lora: Low-rank adaptation large language models. arXiv preprint arXiv:2106.00 09685, 2021.
[52] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't It give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, 2018.
[53] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 889 898, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[54] Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Proceedings the First Conference Machine Translation, pages 131 1918198 Berlin, Germany, August 2016. Association for Computational Linguisties.
[55] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing 2019.
(EMNLP-IJCNLP) pages
[56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1))5485-5551 2020.
[57] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Nco: Large Scale Autoregressive Language Modeling with Mesh-Tennsofloow, March 2021. If you use this software, please cite it using these metadata.
[58] Ben Wang and Aran Komatsuzaki. GPT-1-6B: A 6 Billion Parameter Autoregressive Language Model. attpa://githut http:///ttwuu.cooeo//oooooooo/// wwwpaanowooolieeeeaarrarnnseeeee
May 2021.
[59] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01477 2024.
[60] Weixin Liang. Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. arXiv preprint vxXiv:2304.02817 2023.
[61] Meta. Llama 3.1: Open foundation and fine-tuned chat models, 2024. Accessed: 2024-10-27.
[62] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 2023.
13

---

[63] Haotian Ju, Dongyue Li. and Hongyang R Zhang. Robust fine-tuning of deep neural networks with hessian- based generalization guarantees. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Muchine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10431-10461, PMLR, 17 17-23 Jul 2022.
[64] Igal Sason. Entropy bounds for discrete random variables via coupling. In 2013 IEEE International Symposium on Information Theory, pages 414-418, 2013.
[65] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 11X::1900.11992 2019.
[66] Daphne Ippolito, Daniel Duckworth, Chris Callison-Buree and Douglas Eck. Automatic detection of generated text is casiest when humans are fooled. arXiv preprint arXiv: 1911199111000000 2019.
[67] Xianjun Yang. Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang. and Wei Cheng. Zero-shot detection of machine-generated codes, arXiv preprint 23.v:2310.00///0 2023.
[68] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint axXix:2005..00000 099388 2021.
14

---

Appendix: DALD
Theoretical Analysis
6.1 Addtional Technical Details
First, we formalize some concepts in preparation of our analysis. We assume the vocabulary set to be W = {w,} 1W with size W. We will use slices to denote a segment of passage, but unlike python slices, 6] will be inclusive on the end points and should denote the indices {a, a + +1, ....,.... b}.
Text Completion. Consider the next-token generation process with respect to a language model f. which maps an input passage into an array of logits on the vocabulary set W:
Definition 2. Given a prompt [1oo] of length lg. new length > lo and language model f. we use
[bo+1a] My(X
to denote the process of text completion up to total of / tokens with respect the following sampling method:
Xtoti softmax softmax(f(X (oo(XX)o++11))) () ,1-1, lo-
In the following we will abbreviate M Mia and Mur = Mi and also denote Paur (X) softmax (fsur(X)) (X) = (oftmax(Sur(X)) by for any text segment X.
Loss Function for Fine-tuning, Given target model flar fine-tune a surrogate model fsur on the following dataset generated by the target model:
S = [x'= [Pi, R'|PP ~ P,R' ~ Mar T.))>²>>
where each pi is prompt sampled from prior distribution P, and its corresponding R' is a text completion result sampled from distribution Mar-
Now let us focus on a single text data X R| E S with prompt length ((P) = to and total For any lo
length ((X) of the surrogate model for input (a) as well as a sample output from the target model X, g
S L, consider the next-token logit p Pear (X [0. 1)"
Pear (X The cross-entropy loss is then calculated with
w
((X] =- Σ 1{Xi = logp,
1=1
which has expectation
W
((Xxgl(([lXX)))) = Σ 9 logp, = H(g.p) 2 H(g),
where H(g) and H(g. p) denote the entropy of and the cross-entropy between q and p. respectively. is then evident that the optimal expected value of ((Xi)X{u-11) is H(q), taken when the surrogate model for produces the same logit output as fur-
With this, express the training objective used for fine-tuning the surrogate model as:
(XX)
L(Saur: S) = S
Σ
Σ
(
15

---

6.2 Sample Complexity of Fine-tuning
We now make the following assumption, which is supported by many previous works 63
Assumption 3. The sample complexity of fine-tuning with loss [( S) is of order O(1/e²) x O(log(1/6) ())). In other words, when sample size |S] exceed this order, for almost all = [P. R and corresponding I(P), have
Exi-Pa(X (X/1X()11) S ( X(u-1)) +c.
Remark 4. The result of this assumption is that the surrogate model should approximate the target model well after fine-tuning. We are taking quite a liberty this assumption in terms of input X, but this is simply for the sake of demonstration. In practice, what one can expect is the ex- pected loss Ef S) to be almost optimal, and with generalization this assumption should hold approximately.
Fast- Dast-DetectGPT Method. For passages X, X with the same length L, the conditional probability defined as
(x(X II ply Xily softmax (f(X 1=
where lo is pre-selected length parameter.
In order evaluate passage X with length L, sample new text passages X pf(((X) to estimate the conditional probability curvature:
d(x,)) 1o (X -
where
ñ E {x))(())) py py (XI)),
82 E Ex-p((t) log (XIX) - -70
The basic assumption of Fast-DetectGPTT a positive gap between machine- and human-generated passages in terms of conditional probability curvature:
Assumption 5. The conditional probability curvature of a machine-genernn generatee passage Xue from model fur is substantially greater than that of numan-generated passage X Xnum
d d(Xars fua) d(Xhum> for)
A.
Now with the above assumption, we can prove Theorem I which characterizes the difference between the target model and surrogate model.
A direct result of Assumption Bis is that the Kullback-Liebler devergence between the two distributions are small:
Proof of Theorem 7
KL (Pta (Psur((X))))) = 7 (Ptar(X(a)) Psur (X(a)) 7 (X(=)) Se.
As per Pinsker's inequality, this gives us
drv (ptar (ann(((a))) X(.)) Ponr (x11)) S (X()) (X13)) S
Now to analyze the difference between conditional probability curvature, we have:
log (X - log (XIX) =
log Paur (Xrr(((ll)) (Xi)X - log Pear (Xi (Xi(X(--1:1
1-o+1
<
Σ
drv (pour Peac /CC,
1=
16

---

where C' = min (( (a-( is the minimum possibility given to next-token. For both machine and human generated passages, this value should be reasonably large. On the other hand,
Psuu Mua = E X-mim (11)
log (X|x) - E Ex-pps (-1x) Po. (XIx)
[ (X 11) log Pour (X1 - L8 (X1e 1)) log Pere Pur (1a-1\
1=
Σ
(X (X
1=
(Par( Ptar) log(W - 1) + hhdTT Ptar)) )]]
1=l+11
where the inequality is from 64 With this we have that when (((()+)))))) the difference
d(X, fuu) d(X, fun) S A/3. This requires the sample complexity be ((poly(A/L)).
An immediate corollary of this theorem is that under the same conditions, d(Xur: ssur) > d(Nhum: s + A/3, which means the positive gap is still present once we replace the target model with the surrogate model.
7 More Experimental Details
Datasets. We utilize four datasets to evaluate the performance of our method including Xsum 52 PubMedQA (55] WritingPrompts and WTM-2016[54] Xsum includes documents and corre- sponding extreme summarization of each news document. PubMedQA is composed of the abstracts of research papers and the corresponding research questions in biomedical research. WritingPrompte is a large dataset consisting of 300K human-written stories paired with writing prompts from an online forum. WTM-2016 is translation dataset with English-German pairs. For each dataset, we randomly choose 150 examples as numan-writted texts. Utilizing the 30 prefix tokens of human- written texts, prompt the target closed-soure source models by calling API to generate the corresponding texts the seetiine-geerrtee text. Some details about prompting for each dataset can be found in Table 7 We utilize the same prompt for different source models including GPT-3.5, GPT-4 and Claude-3, 33. Our training datasets are collected from the open-source datasets, WildChat (59 for GPT-3.5 and GPT-4. Dataset for Claude-3 is generated by calling API with the prompt from WildChat. For GPT-3.5 and GPT-4, we filter the data by timesteps to exactly match the version of the source model and randomly select 5K samples as the training set to fine-tune the surrogate model.
Table 7: Examples of prompts used in different datasets.
Datasets
Prompts
System: You are a Technical writer.
PubMedQA
User: Please answer the question in about 50 words. Question
System: You are a News writer.
Xsum
User: Please write article with about 150 words starting exactly with: Prefix
Writing Prompt
System: You are a Fiction writer.
User: Please write an article with about 150 words starting exactly with:Prefix
System: You are writer.
German
User: Please complete passage in German with about 150 words, starting exactly with: Prefix
Baselines. For training-based methods, we compare DALD with GPT-2 detectors 24 developed by OpenAl which build on AaBERTa-base/largee zero-shot methodologies, such as Likelihood (mean log probabilities), LogRank (average log of ranks in descending order by probabilities), Entropy (mean token entropy of the predictive distribution) 31 66), and LRR (a fusion of log probability and log-rank) [4].
Additionally, we compare DALD to established
17

---

Table 8: Full results of ChaIGPT(GPPT33.-TTrbb))) on PubMed, XSum and Writing as the complemen-
lary results to Table [
GPT-3.5-Turbo
Method
PubMed
XSum
Writing
DNA-GPT
0.7788
0,9673
0.9829
Fast-DetecIGPT
0.9309
0,9994
0.9967
DALD
0.9853
1.0000
1.0000
Table 9: More results our method on one-for-all settings. The surrogate model trained with 5K samples generated from GPT-4-061 and evaluated on other versions of the model including GPT-3.5, GPT-4, GPT-4o and GPT-4o-Mini without further training.
GPT:3.5-Trbb--063
GPT-3.5-Turbo-111 PubMed
GPT-3.5-Turbo-012.
Method
PubMed
PubMed
DALD
0.9866
0.9911
0,9858
GPT-4-1106-Preview
GPT-4-0025-Preview
GPT-4-0409-
Method
PubMed XSum
Writing PubMed XSum Writing PubMed XSum Writing
DALD
0.9780
0.9956
0.9968
0.9850
0.9954
0.9968
0.9815
0.9872 0.9924
GPT-40 PubMed 0.9877
GPT-4o-Mini
Method DALD
XSum Writing 0.9994
PubMed XSum Writing 0.9857
0.9965
0.9976 0.9992
More Implementation Details. We describe more implementation details our frame- work. For LoRA syper-parameters,
utilize 16 as the LoRA rank and set lora alpha
as 32. Dropout is set as 0.05. For the Llama series, we adopt the LoRA module on ["y_proj", "v_proj", "k_proj" "o "o_proj", "gale_proj", "down_proof 1. "up_proj") while in GPT-Neo models, we apply on ["q_proj", "v_proj", "k_proj", "oul_pp "oul_proj". "c_fe", "c_proj"]. For training hyperparameters, while it is 2048 for texts from Claude-3. We finetune the surrogate model with a learning rate of lc-4. per device with gradient accumulation per 4 steps. It's noteworthy that do not choose our hyperparameters carefully and we believe it will achieve better results with careful tuning. The training costs less than 10 minutes with A6000.
set 512 the max length for texts from GPT-4 and GPT-3.5 models
The batch size is
Additional Experimental Results
8.1 Full Results on ChaGGPTGGP133--Tubbb
We provide the full result on ChaGGPGGGT:3.5-TTrbbb as shown in Table 8 obtains great performance on ChatGPT. Our method boosts their performance with a significant 100% accuracy on Xsum and Writing.
Fast- Past-DetectGPT
8.2 Results on More Versions of Source Model
To show the one-for-all ability of our method, we provide more results on more source models using the surrogate model trained only on data generated from GPT-4-0613, as shown in Table 9 We evaluate our method on different versions GPT-3.55, GPT-4, GPT-4o and GPT-4o-Mini. In general, method shows robustness all versions of GPT-3.5 and GPT-4, achieving more than 98% accuracy except on PubMed-GPT-441 106-Prev but still more than 97%. The superior performance in the one-for-all setting shows the generalizability of our methods, leading further motivate to train a universal surrogate model for various source models regardless of model updates.
8.3 Results on Code Detection
We include a performance comparison the coding task. We follow (77 to apply APPS 68 dataset the coding task. We sample 150 coding tasks from APPS and generate the coding results by calling
18

---

GPT-4 API. Our method is only trained by the corpus generated by GPT-4 as we previously did. The results are Table T0] Our method obtains significant improvement on the coding task.
Table 10: Results of code detection the APPS dataset.
Method
GPT-4-APPS
Fast- Mest-DetectGPT
0.6836
DALD
0.9078
8.4 Results on Different Text Genres
We include evaluation of the detection performance of different text genres. Using domain- specific datasets RAID, we select 1000 human texts and 1000 GPT-4 generated texts for each domain in RAID dataset and compare the evaluation results with FastDetectGPT. The results is shown in which demonstrate that DALD performs admirably in diverse domains.
Table 11
Table 11: Results across different text genres using the RAID dataset.
Method
Poetry
News
Abstract
Books
Recipes
Reddit
Mest-DetectGPT
0.8553
0.9116
0,8600
0.9123
0.9116
0.9134
DALD
0.9709
0.9567
0.9876
0.9675
0.9998
0.9862
8.5 Results on Different Test Data Sizes
Our main experiment follow previous works such as DNA-GPT and FastDetectGPT and utilize the same amount of data for fair evaluation. Furthermore, we provide the evaluation results of our method on different test data sizes, as shown in Table [2] observed that there is little difference in performance the number of samples increases, indicating the robustness of our method to test data size.
Table 12: Results of ChatGPT and GPT-4 across different sample sizes on PubMed, XSum, and Writing datasets.
ChatGPT PubMed
GPT-4
Num. of Samples
PubMed
XSum
Writing
150
0.9853
0.9785
0.9954
0.9980
300
0.9842
0.9821
0.9924
0.9980
500
0.9806
0.9828
0.9929
0.9974
8.6 Evaluation Metrics
Following Fast-DetectGPT1199 we compute the area under the receiver operating char- acteristic (AUROC) to measure the performance of different methods, evaluating the method per- formance with different classification thresholds. AUROC falls into the value range [0, 1J. which provides quantitative measure the likelihood that randomly generated passage has higher predicted probability of being machine-generated than randomly written passage by human. Generally, the value of 0.5 in AUROC indicates a random classifier without any classification ability while 1 in AUROC refers perfect classifier for all samples. The results comparison of our method and other basclines including training-bart based methods and zero-shot methods are provided in Table
AUROC.
AUPR. Similar to AUROC, AUPR computes the area under precision and recall, which evaluates both the precision and recall of classifiers with different thresholds, AUPR also ranges from 0 to 1. A higher AUPR value represents a better classifier, which generally obtains higher recall on the condition that the precision is high. Therefore, we provide the AUPR comparison our method with other baselines in Table 13 We compare all baselines including training- -ased such as RoBERTa- base and zero-shot methods such as DetectGPT, DNA-GPT and Fast-DetectGPT. First all, we can observe that the current mainstream zero-shot methods generally achieve better results compared with the training-bass based methods. Moreover, our method achieves the highest AUPR results (> 98%) on all datascts and source models, further demonstrating the effectiveness of our method.
19

---

Table 13: Performance comparison with baselines on AUPR. Similar AUROC, our method achieves the best results compared with other training based methods and zero-shot methods.
ChatGPT PubMed
GPT-4
Claude-3
Method
PubMed
XSum
Writing
PubMed
XSum
Writing
RoBERTa-base
0.6349
0,5538
0.7784
0,5166
0.5325
0,8920
0.7102
RoBgRTa-lare
0.7087
0.5887
0.7215
0,4160
0.5466
0.8243
0.5500
Likelihood
0.8551
0.8453
0.9379
0.9594
0.8453
0.9379
0.9594
Entropy
0.3699
0.3743
0.4172
0.3382
0.3382
0,4172
0.3382
LogRank
0.8811
0.8415
0.9415
0.9504
0,9504
0,9415
0,9504
LRR
0.7904
0.6800
0.7370
0.7075
0.7517
0,9070
0.8908
NPR
0.6917
0.5793
0.6099
0.7777
0.6124
0.8621
0.9314
Detect-GPT
0.5862
0.6272
0.6940
0.6272
0.5982
0.8551
0,9302
DNA-GPT
0.7181
0,6566
0.6925
0.7610
0.6879
0.9452
0,9540
Tast-DetectGPT
0,9323
0.8315
0.9193
0.9533
0.8955
0.9832
0,9595
DALD (Ours)
0.9841
0.9802
0.9964
0.9989
0.9881
0.9991
0.9973
Limitation & Future work
The limitation our method is the absence of additional experiments conducted in multilingual settings. While our method has shown efficacy in English and German environments, its performance across different languages remains unexplored. Future studies should endeavor to evaluate the generalizability of our approach in multilingual contexts to ensure its applicability across diverse linguistic landscapes.
20

---

Table 14: Examples of data from WildChat and Claude-3-Opus Bruude---ppu--IIstuut----- Instruct--tttt 15K.
Datasets
Samples
WildChat
Prompt: who theseus in plutarch book?
GPT-3.5-
Theseus is legendary hero and king of Athens in Plutarch's book. Plutarch presents him as a great leader and conqueror who united the scattered towns of Attica and established Athens as the political and cultural center of the region. Theseus is also famous for his defeat of the Minotaur, a monstrous creature with the head of bull and the body of man who lived in labyrinth the island of Crete. According to Plutarch, Theseus also instituted many reforms in Athens,
Turbo-0301
such as the legal code and the Panathenaic festival.
Claude-3-
Prompt: Explain how the human circulatory system functions, covering the structures and processes involved in transporting blood throughout the body, Include details about the major components of the system (heart, blood vessels), how blood is pumped through arteries and veins, and the importance of blood circulation for keeping tissues and organs functioning properly.
Opus-Instruct-
15K
Response: The human circulatory system responsible for transporting blood throughout the body in order to deliver oxygen and nutrients to tissues and organs and remove metabolic waste products. The major components are: The Heart: A muscular organ that acts as pump to propel blood through the blood vessels. It has four chambers- two atria and two ventricles. The right side of the heart pumps deoxygenated blood to the lungs, while the left side pumps oxygenated blood to the rest of the body. Arteries: Blood vessels that carry oxygenated blood away from the heart to the body's 's tissues. They have thick, muscular walls to withstand the high pressure of blood being pumped from the
heart. The aorta is the largest artery. Veins: Blood vessels that carry deoxygenated blood from the tissues back to the heart. They have thinner walls and larger lumens than arteries. Veins have valves prevent backflow of blood. The venae cavae are the largest veins. Capillaries: Microscopic blood vessels that form network facilitate the exchange of substances between the blood and tissues. Oxygen and nutrients diffuse from the blood into cells, while carbon dioxide and wastes move from cells into the blood. The pumping of the heart drives circulation: 1. Deoxygenated blood from the body enters the right atrium, flows into the right ventricle, and is pumped to the lungs via the pulmonary arteries. 2. In the lungs, blood picks up oxygen and releases CO2. Oxygenated blood returns the left atrium via the pulmonary veins. 3. Blood flows into the left ventricle and is forcefully pumped out to the body through the aorta. 4. The aorta branches into smaller arteries, then arterioles, and finally capillaries where gas and nutrient exchange occurs. 5. Deoxygenated blood collects in venules, flows into veins, and returns to the right atrium via the venae cavae. The cycle repeats. Proper circulation is critical for survival. It ensures that all cells receive oxygen and nutrients needed to function. It enables removal of metabolic wastes that could be toxic if allowed to accumulate. Circulation also helps distribute hormones, transport immune cells, and regulate body temperature and pH. Disruption of blood flow, such from blockage, quickly lead to tissue damage or death.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check the result in Tables 1, 2, 3, 4, and Figures 2, 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please check the Limitation & Future Works section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Please check the Method in the main text and Theoretical Analysis in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please check the dataset, evaluation metric and implementation detail in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will include our code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///giiieee/eeeeebiiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiyiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please check each experimental section, we have a detailed description of our experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please check the implementation section in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp/s/mourip....pppppttt/t/ta
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We all use public resources in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the weight in experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "NeaAlpheWX", "reviewer": "Reviewer_iaeF", "summary": "The paper addresses the challenge of detecting machine-generated text from black-box LLMs without access to their logits. Traditional methods using surrogate models suffer from performance degradation due to misalignment with target model distributions, particularly as new models are introduced. The proposed Distribution-Aligned LLMs Detection (DALD) framework aligns surrogate model distributions with unknown target LLMs. DALD enhances detection capabilities, achieving state-of-the-art performance in black-box settings, and offers a plug-and-play enhancement to existing zero-shot detection frameworks. Extensive experiments validate DALD's high detection precision against revised text attacks and non-English texts.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The detection of machine-generated text is an important problem.\n2. The paper includes theoretical analysis on the effectiveness of fine-tuning.", "weaknesses": "1. The idea lacks novelty; collecting data from models with the same version as the target model and then aligning the surrogate model is not novel.\n2. The proposed method appears to heavily rely on alignment data collection and specific model versions. Does this mean the surrogate model needs continuous fine-tuning, when the detection task varies? If so, what is the generalizability?\n3. The paper missing abalation studies on how much fine-tuning data is required when the detection task varies in different categories. If the data requirement is large, it could pose a problem for the detection task.\n4. The paper is missing important baselines such as RADAR [1] and could benefit from evaluation on more models like Gemini and additional tasks, including coding tasks.\n\n## Reference\n\n[1]. Radar: Robust ai-text detection via adversarial learning. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed reviews. Here are our responses to your questions:\n\n**Response to Weakness 1**: \n\nThe detection of black-box commercial LLM models is a critical topic. The rapid updates in these models present a significant and ongoing challenge, as they lead to decreased detection efficiency in existing logit-based zero-shot detectors. Traditional methods require the manual and careful selection of different surrogate models. Our work's most significant contribution is the observation that lightweight distribution alignment (less than 10 mins on a single A6000) between the surrogate model and the target model can significantly improve detection accuracy and effectively address the issue of model updates. The empirical results demonstrate the strong sample efficiency of the proposed approach, which is also supported by theoretical analysis. Furthermore, the empirical results reveal an interesting observation: the approach exhibits relatively strong generalization capabilities and can be used to enhance all existing logit-based detectors.\n\n**Response to Weakness 2**: \n\nThere are two aspects of the generalizability of our method.\nOn the one hand, the model trained on single-source data can perform effectively on different target models, which indicates one-for-all ability. We gently invite the reviewer to check the 3rd line results in Table 2. Our model is only trained on 5K texts generated by GPT-4, yet it demonstrated large performance improvement on ChatGPT, GPT-4, and Claude3.  Additionally, we further tested its performance on the newly published models GPT-4o and GPT-4o-mini and found the model also has strong performance on them. These results highlight the generalizability of our approach.\n\n\n||\\||GPT-4|||\\||Claude3|||\\||GPT-4o|||\\||GPT-4o-Mini|||\n|---|---|---|---|---|---|:---------:|---|---|---|---|---|---|---|---|---|---|\n||\\|| PubMed|Xsum|Writing| \\||PubMed|Xsum|Writing |\\|| PubMed|Xsum| Writing|\\|| PubMed | Xsum| Writing|\n| **Baseline**| \\| |0.7995|0.7072|0.9299|\\|| 0.8877|0.9143| 0.9248|\\||0.8163|0.6595| 0.9505|\\|| 0.7994 | 0.5877| 0.9150|\n| **DALD** | \\|| 0.9785 | 0.9954 | 0.998 |\\|| 0.9875 | 0.9993| 0.9977  |\\|| 0.9877 | 0.9965 | 0.9994  |\\|| 0.9857 | 0.9976 | 0.9992|\n\n\nOn the other hand, reviewer bnrm also highlighted that the model has the capability of continuous learning. We encourage the reviewer to refer to the 5th line of Table 2 where we present the results of our model which has been trained using 5K texts from multiple sources. Its exceptional performance suggests that if a new closed-source model is released, we can gather a small corpus generated by the model and use it to continue training the surrogate model. Thus, our model can rapidly and continuously adapt to new models with minimal fine-tuning costs.\n\n**Response to Weakness 3**: \n\nFirst, our paper contains in-depth experiments that investigate the required data size for fine-tuning. We would like to invite the reviewer to refer to Figure 4, which depicts the performance at various data sizes. This result shows that just about 2k of training data is enough to yield significant improvements, indicating the minimal amount of data needed. Moreover, we performed a theoretical analysis of the amount of training data (refer to the Appendix and the Method section). One of our key contributions is demonstrating that only a minimal tuning cost is required.\n\n**Response to Weakness 4**:\n\nThank you for your suggestions about adding more baseline models and different tasks. As suggested, we add comparisons with RADAR: \n\n||\\||GPT-3.5|\\|||GPT-4||\\|||Claude3||\n|---|---|---|---|---|---|---|---|---|---|---|\n||\\||Pubmed|\\||Pubmed|Xsum|Writing|\\||Pubmed|Xsum|Writing|\n|**RADAR**|\\||0.8953|\\||0.8818|0.9926|0.8496|\\||0.8295|0.9900|0.8780|\n|**DALD**|\\||0.9853|\\||0.9785|0.9954|0.9980|\\||0.9630|0.9867|0.9981|\n\n\nWe utilize the official codebase of RADAR. RADAR is evaluated on the same test dataset as our DALD. As shown in the table, DALD achieves better performance on all datasets. We will add the comparison and discussion to our draft accordingly.\n\nWe also include the performance comparison on the coding task. We follow [2] to apply APPS[1] dataset as the coding task. We sample 150 coding tasks from APPS and generate the coding results by calling GPT-4 API. Our method is only trained by the corpus generated by GPT-4 as we previously did. The results are as follows:\n\n||\\||GPT4-APPS|\n|---|---|:---:|\n| **RADAR**| \\| | 0.5067|\n| **Fast-DetectGPT** | \\| | 0.6836|\n| **DALD** | \\| | 0.9078|\n\nOur method obtains significant improvement on the coding task.\n\nFinally, we conduct the experiments on Gemini, as shown in:\n||\\||Gemini-PubMed|\n|---|---|:---:|\n| **RADAR**| \\| |0.8496|\n| **Fast-DetectGPT** | \\| | 0.9352|\n| **DALD** | \\| | 0.9902|\n\nSince Gemini has the protection mechanisms(recitation), we only conducted the experiments on the PubMed dataset to evaluate the performance. Our method is still the best compared with other methods, demonstrating the effectiveness of our method.\n\n\n**Reference**:\n\n[1] Hendrycks, Dan, et al. \"Measuring coding challenge competence with apps.\" arXiv preprint arXiv:2105.09938 (2021)\n\n[2] Yang, Xianjun, et al. \"Zero-shot detection of machine-generated codes.\" arXiv preprint arXiv:2310.05103 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K8aAPfp9JS", "reviewer": "Reviewer_yZjY", "summary": "This paper introduces Distribution-Aligned LLMs Detection (DALD), a novel framework for detecting AI-generated text from large language models (LLMs). DALD addresses limitations of traditional detection methods, particularly when dealing with black-box or unknown LLMs. It aligns surrogate models with unknown target LLM distributions, enhancing detection capability across various models without requiring access to their internal logits. The approach leverages corpus samples from publicly available outputs of advanced models to fine-tune surrogate models, achieving state-of-the-art performance on both closed-source and open-source models. DALD can be integrated into existing zero-shot detection frameworks and demonstrates robustness against revised text attacks and non-English texts, offering a versatile solution for the evolving challenge of distinguishing AI-generated content from human-written text.", "questions": "1. Could you compare it with other SOTA training-based methods such as Ghostbuster[1]?\n2. Why did you just test PubMed on ChatGPT while testing all three datasets on GPT-4 and Claude?\n3. Could you provide the ROC curves for your detector? For reliability, it is important to see how well the detector performs at low FPR regimes [2].\n4. Why not conduct some experiments on open-source LLM like Llama 3 and Mistral/Mixtral？\n\n[1] Vivek Verma, Eve Fleisig, et al. \"Ghostbuster: Detecting text ghostwritten by large language models\" (2023). \n[2] Sadasivan, Vinu Sankar, et al. \"Can ai-generated text be reliably detected?\" arXiv preprint arXiv:2303.11156 (2023).", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a highly effective text detection framework.\n2. The paper utilizes the fact that models can generate text under black-box settings to train a surrogate model, achieving efficient prediction in black-box scenarios.\n3. The framework presented in the paper is actually plug-and-play and can be integrated into logit-based detection methods such as DetectGPT and Fast-DetectGPT.", "weaknesses": "1. Since DALO is a training-based method, it may be unfair to compare it with the zero-shot method(Detect-GPT, DNA-GPT, and Fast-DetectGPT). Comparing it with other SOTA training-based methods such as Ghostbuster[1] is necessary.\n2. Only testing PubMed on ChatGPT while testing all three datasets on GPT-4 and Claude.\n3. For reliability, it is important to see how well the detector performs at low FPR regimes [2].\n4. Expanding your evaluation to include recent open-source models such as Llama 3 and Mistral/Mixtral would enhance the breadth and relevance of your study. \n5. The method is easy and lacks contribution in technology.\n\n[1] Vivek Verma, Eve Fleisig, et al. \"Ghostbuster: Detecting text ghostwritten by large language models\" (2023). \n[2] Sadasivan, Vinu Sankar, et al. \"Can ai-generated text be reliably detected?\" arXiv preprint arXiv:2303.11156 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your detailed and careful reviews. We will give our response to your questions.\n\n**Response to Weakness 1**: \n\nWe appreciate the suggestion of comparison with Ghostbuster and adding experiments comparing our method to Ghostbuster. We utilize the official code of Ghostbuster and evaluate it on the same datasets in our experiments. The results are:\n\n||\\||GPT-3.5|\\|||GPT-4||\\|||Claude3||\n|---|---|---|---|---|---|---|---|---|---|---|\n||\\||Pubmed|\\||Pubmed|Xsum|Writing|\\||Pubmed|Xsum|Writing|\n|**GhostBuster**|\\||0.8108|\\||0.7269|0.8384|0.9614|\\||0.7722|0.946|0.9675|\n|**DALD**|\\||0.9853|\\||0.9785|0.9954|0.9980|\\||0.9630|0.9867|0.9981|\n\nDALD achieves the best performance on all datasets compared with Ghostbuster. We will add the results of Ghostbuster and more discussions to the next version. In addition, we would like to point out that we have already compared our method to training-based methods like Roberta (we gently invite the reviewer to check Table 1), demonstrating its effectiveness in comparison to existing approaches. Concerning the fairness of comparison, we would like to clarify that our method significantly differs from traditional training-based methods, as we only utilize a small amount of data (e.g., around 2k data) to fine-tune the model (less than 10 mins on a single A6000), whereas training-based methods typically require much more training data (250K in Roberta). Therefore, we believe that to demonstrate the uniqueness and advantages of our method, it is essential (and has been) to compare it with both zero-shot and training-based methods.\n\n**Response to Weakness 2**:\n\nWe appreciate your careful review. Due to space constraints in the paper, we omitted results from two other datasets on ChatGPT, so we had to prioritize the study using the most recent datasets available on PubMed.  We add the results here (which will also be included in the next version): \n\n||ChatGPT-Pubmed|ChatGPT-Xsum|ChatGPT-Writing|\n|---|---|---|---|\n|**DNA-GPT**|0.7788|0.9673|0.9829|\n|**Fast-DetectGPT**|0.9309|0.9994|0.9967|\n|**GhostBuster**|0.8108|0.9832|0.9983|\n|**DALD**|0.9853|1.0000|1.0000|\n\nIt is observed Xsum and Writing on ChatGPT can be well detected by FastDetectGPT. Furthermore, our method further improves the performance on top of FastDetectGPT.\n\n\n**Response to Weakness 3**: \n\nThank you for this suggestion. We will add the ROC curves to the pdf page. We would like to invite you to check Figure 1 on the pdf page. We compare the curve with DNA-GPT and FastDetectGPT. Our method achieves the best performance at low FPR on all datasets. \n\n**Response to Weakness 4**:\n\nWe appreciate the suggestion to incorporate recent open-source models such as Llama 3 and Mistral in our assessment. It's worth noting that the results for Llama 3-8B are already documented in Table 6. Additionally, we intend to present further findings on open-source models like Mistral-7B and the new Llama 3.1-8B. These results are available for review:\n||\\||LLama3.1|||\\||Mistral|||\n|---|---|---|---|---|---|---|---|---|\n||\\||Pubmed|Xsum|Writing|\\||Pubmed|Xsum|Writing|\n|**Fast-DetectGPT**|\\||0.8668|0.9914|0.9958|\\||0.6880|0.7931|0.9211|\n|**DALD**|\\||0.9059|1.0000|0.9998|\\||0.7733|0.8822|0.9573|\n\nThe results show that our method achieves the best on all open-source models, demonstrating its effectiveness.\n\n**Response to Weakness 5**:\n\nWhile our method might be easy to implement, it's essential to consider the practical implications and benefits it offers. We want to highlight that our work's most significant contribution is the observation that lightweight distribution alignment (less than 10 mins on a single A6000) between the surrogate model and the target model can significantly improve detection accuracy and effectively address the issue of model updates. The empirical results demonstrate the strong sample efficiency of the proposed approach, which is also supported by theoretical analysis. Furthermore, the empirical results reveal an interesting observation: the approach exhibits relatively strong generalization capabilities and can be used to enhance all existing logit-based detectors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Unm3i1ApKM", "reviewer": "Reviewer_3RMB", "summary": "This paper proposes a method to improve black-box detection of machine-generated text, tackling the problem of performance degradation when a surrogate model's output is poorly aligned with the closed-source target LLM. By LORA fine-tuning the surrogate model on text generated by the target model, the authors align the surrogate model's output distribution to the target model, thus improving detection performance.  \n\nThe authors show that this alignment can be used to outperform/improve existing logit detection methods such as DetectGPT, DNA-GPT and Fast-DetectGPT; is effective across a range of surrogate models; and requires only a small amount of fine-tuning (and thus can be keep surrogate models up-to-date when newer models are released).", "questions": "See Weaknesses Section. (and Strengths bullet point 2) \n\nAlso, minor nitpicks / typos: \n\n\nTypos: \n- Line 187: is freezing\n- Line 215: OpanAI\n- Line 244: pertraining\n- Line 574: We", "limitations": "Limitations section was extremely brief and only discussed extending their evaluation of multilingual performance.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Consistent performance improvements**: Ablation study (Table 4) suggests that DALD leads to consistent and significant improvements across target models and datasets, for all three logit-based methods\n - **Surprisingly effective**: it's quite surprising to me that a relatively low amount of data/fine-tuning is so effective at aligning the probability distributions of target and surrogate models. I wonder how effective DALD is for more niche tasks/text genres, in cases where there may be less overlap between the training corpus and the evaluation text? (the authors briefly touch on this when discussing PubMed results) \n- **Application to non-English & practical settings**: The authors demonstrate that DALD can also help tackle the issue of non-English detection bias by showing performance improvements for German. They also show that their method is robust to machine-generated text that has undergone adversarial attacks.", "weaknesses": "- **Generalizability may be overstated** (Table 2): The authors cite the superior performance of their one-for-all surrogate model as evidence that their method could be extended to detect texts of unknown source models. However, the improvements seems very slight: I agree with their interpretation that current closed-source models tend to have a similar distribution, but this homogeneity might not necessarily be true for future versions of these models and/or other future closed-source models. \n- **Effectiveness across surrogate models** (Table 3): DALD doesn't significantly improve the detection performance of all surrogate models. For example, its effect on GPT2-Neo 2.7B, though positive, seems fairly minor. Does this indicate a limitation of their DALD method (e.g. that it is only effective when the performance gap/parameter size/pre-training data difference between surrogate and target is relatively small)? \n- **Limitation and Future work section** is extremely brief, and does not adequately address limitations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive and detailed reviews. We provide detailed responses to your concerns.\n\n**Response to the question about different domains**:\n\nWe appreciate your interest in the effectiveness of DALD for different domains. It's worth noting that our training data is entirely distinct from our test dataset, so there is no overlap between the two datasets. Training data is the corpus from the publicly shared outputs of leading models while testing data is the specific datasets such as PubMed prompted by the target models. Additionally, we've conducted experiments on other domain-specific datasets, namely RAID[1], including poetry, news, books, etc. Results are shown:\n\n||\\||Poetry|News|Abstract|Books|Recipes|Reddit|\n|---|---|---|---|---|---|---|---|\n|**Fast-DetectGPT**|\\||0.8553|0.9116|0.8600|0.9123|0.9116|0.9134|\n|**DALD**|\\||0.9709|0.9567|0.9876|0.9675|0.9998|0.9862|\n\nWe select 1000 human texts and 1000 GPT-4 generated texts for each domain in RAID dataset and compare the evaluation results with FastDetectGPT. The results demonstrate that DALD performs admirably in diverse domains.\n\n**Response to Weakness 1**:\n\nThank you for raising this point as it provides a deep insight into our method. We acknowledge empirically that current closed-source models have homogeneity characteristics, as evidenced by the slight improvement in multisource settings. Therefore, we cannot guarantee the one-for-all surrogate model can perform optimally when encountering new models without homogeneity. However, it's important to note that the generalizability of our approach also lies in its ability to continuous learning. By collecting small-size data from new models and continually fine-tuning the surrogate model, we can adapt to newly published models with minimal cost. This ongoing effort represents a worthwhile alternative compared to retraining the entire model from scratch.\n\n**Response to Weakness 2**:\n\nWe appreciate this careful review. The modest performance improvement of GPT-Neo-2.7B as the surrogate model only happens on Claude3 detection. As shown in the table below, upon further testing on GPT-4, it is observed a substantial improvement in its performance. We believe this might be attributed to that the parameter number of GPT-Neo-2.7B is relatively small and may not be sufficient to generalize well across all datasets. We intend to present a comprehensive overview of GPT-Neo-2.7B in the next version.\n\n||\\||Pubmed|Xsum|Writing|\n|---|---|---|---|---|\n|**Fast-DetectGPT**|\\||0.8179|0.8179|0.9521|\n|**DALD(GPT-Neo-2.7B)**|\\||0.9020|0.9732|0.9800|\n\n\n**Response to Weakness 3**:\n\nWe acknowledge that the Limitation and Future Work section is brief since we have to condense this section in the current version due to space constraints. However, we appreciate your feedback and will ensure to include a more comprehensive discussiont in the next version of the paper. Regarding future work, we plan to explore the influence function to improve data efficiency for finetuning. Additionally, future work can utilize our pipeline for analyzing the homogeneity of different models and versions such as investigating whether there is more variability in non-transformer architectures or black-box model alignment from different companies. These are crucial areas that we will focus on exploring further.\n\n\n**Response to Typo**:\n\nThank you for this careful review and pointing out the typo in our paper. We appreciate your attention to detail and will make sure to revise the typo accordingly in the next version of the manuscript. Your feedback is invaluable, and we are grateful for your thorough assessment of our work!\n\n**Reference**:\n\n[1] Dugan, Liam, et al. \"RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors.\" arXiv preprint arXiv:2405.07940 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYN8UEXAU4", "reviewer": "Reviewer_bnRm", "summary": "This paper proposes a framework named Distribution-Aligned LLMs Detection (DALD) to improve the performance of surrogate models in detecting LLM-generated text from both closed-source and open-source models. The method enhances detection performance by aligning the distribution of the surrogate model to better match the distribution of the target model. The paper verifies the effectiveness of DALD on various advanced closed-source and open-source models through extensive experiments and demonstrates its \"plug-and-play\" enhancement capability in a zero-shot detection framework.", "questions": "- Please refer to the weaknesses, especially Weakness 1.\n\n- An additional question: I noticed that DetectGPT and Fast-DetectGPT are described as black-box methods. Is this because they use surrogate models? In fact, all logits-based methods (including Likelihood and Rank) can use surrogate models, depending on whether the scenario is white-box or black-box.", "limitations": "- The testing dataset is insufficient; all datasets contain only 150 human-written samples, and the experimental results obtained may actually be biased, which is indeed my concern.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- It is gratifying that this paper reflects on existing logits-based methods and proposes the DALD framework, which improves detection performance by aligning the distribution of the surrogate model, achieving superior performance.\n- The proposed method uses small-scale datasets for fine-tuning, which is cost-effective and highly adaptable, allowing it to quickly adapt to rapid iterations and updates of models, demonstrating good usability.\n- Ablation experiments are comprehensive. The framework also shows good performance in scenarios such as Non-English Detection and Adversarial Attack.", "weaknesses": "- The method's interpretability is still insufficient. I am not entirely sure, but one point that confuses me is why training DALD on datasets from multiple source models performs better than training on a dataset from a single source model (see Table 2). Since DALD essentially aligns the logits distribution of the surrogate model with that of the source model, this is a crucial starting point of the paper. However, training DALD on datasets from multiple source models theoretically and intuitively would not align distributions better than a single source model, which is contradictory. Similarly, in the Claude-3 setting, DALD trained on a GPT-4 dataset even outperforms DALD trained on a dataset from the source model Claude-3, which is perplexing.\n- There is a lack of detail about the datasets used for fine-tuning, such as their sources, topics, or other attributes, which could help readers understand and reproduce the method. In fact, I am very curious about the impact of the fine-tuning dataset on DALD, aside from the sample size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed review. Please check our response to your concerns.\n\n**Response to Weakness 1**:\n\nFirst, regarding the concern of the better performance of the model trained on multiple source data, a reasonable explanation could be that the optimization space of logits is large. Therefore, aligning the logits of one model does not necessarily affect the alignment of others in the LLM detection point of view. It suggests that continuing to learn from the corpus generated by multiple models simultaneously has no negative impact on the overall performance. Moreover, by collecting from multiple sources, the training data is augmented, which, given homogeneous models, is similar in spirit to the widely used data augmentation for improving generalization.  Augmentations induce extra stochasticity during training, which effectively flattens the loss landscape [3]. The surrogate model may also benefit from the complementary information from different sources that can potentially improve the alignment of distributions, and eventually enhance the model's generalization performance. We will add discussions in the revision. This indicates that training DALD on datasets from multiple source models is indeed effective and not contradictory.\n\nBesides, regarding the point you raised about the perplexing outperformance of DALD trained on a GPT-4 dataset compared to a dataset from the source model Claude3, we would like to point out that their performance is close. In the general direction, a closer distribution of logits can help produce better detection results. However, there is a certain degree of randomness involved, as we are not comparing the direct value of logits, but rather the scoring function in Fast-DetectGPT, which uses logits to calculate the score as the detection metric. In most cases, our results align with our assumptions.\n\n**Response to Weakness 2**:\n\nSorry for the unclearness of our fine-tuning datasets. As we discussed in the main text, the only requirement of the fine-tuning dataset is that the texts are generated by the same target model. More concretely, in our experiments, we utilize the corpus generated by ChatGPT and GPT-4 from WildChat[1]. As for Claude-3, the data is from the huggingface[2]. For the open-source model, we randomly select 5000 prompts from Wildchat and generate the output from the open-source models to obtain fine-tuning data. We will share our code with the public for reproduction.\n\n**Response to Question 2**:\n\nDetectGPT and FastDetectGPT can be applied in both white-box and black-box settings. In the white-box setting, the output logits of the target model will be accessed. However, it is unavailable in the black-box setting, thus methods like DetectGPT and FastDetectGPT utilize a surrogate model to compute the output logits. Our method focuses on the black-box setting since the black-box scenario is more practical but challenging due to the closed-source trend of current models. Therefore, we report and compare the results of DetectGPT and FastDetectGPT in the black-box setting.\n\n**Response to Limitation**:\n\nThank you for your feedback on the size of the test dataset. First of all, we follow previous works such as  DNA-GPT and FastDetectGPT and utilize the same amount of data for fair evaluation. Moreover, we are happy to provide the evaluation results of our method on different test data sizes, as shown:\n\n|Num. of Samples|\\||ChatGPT|\\|||GPT4||\n|---|---|---|---|---|---|---|\n||\\||Pubmed|\\||Pubmed|Xsum|Writing|\n|150|\\||0.9853|\\||0.9785|0.9954|0.9980|\n|300|\\||0.9842|\\||0.9821|0.9924|0.9980|\n|500|\\||0.9806|\\||0.9828|0.9929|0.9974|\n\nIt is observed that there is little difference in performance as the number of samples increases, indicating the robustness of our method to test data size.\n\n**Reference**:\n\n[1] Zhao, Wenting, et al. \"Wildchat: 1m chatGPT interaction logs in the wild.\" arXiv preprint arXiv:2405.01470 (2024).\n\n[2] lodrick-the-lafted/Sao10K_Claude-3-Opus-Instruct-9.5K-ShareGPT\n\n[3] Geiping, Jonas, et al. \"How much data are augmentations worth? An investigation into scaling laws, invariance, and implicit regularization.\" arXiv preprint arXiv:2210.06441 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**PDF Pages**:

This PDF page includes the ROC curves comparison with DNA-GPT and Fast-DetectGPT. We would like to invite all reviewers to check the Figure in the page. Thanks a lot.

Meta-review: The paper proposes a method to fine-tune a surrogate LLM against texts generated by the target LLM as a means of improving the ability to detect/guess if a given text is generated from the aforementioned target LLM. The idea is simple and requires a relatively small number of generations to improve the model's efficacy. The reported effect size is quite significant, by 10% in many cases. 

While a number of borderline reviews are present, most of the concerns are non-falsifiable. e.g., lacking novelty but not citing prior work that has followed this approach, or that the explanation for the idea/why it works is not sufficient (what makes something sufficient?). I do not consider these substantive reasons to reject the work, given that the effect size is large (let theory follow empirical results) and the simplicity of the approach (if it does not work in other contexts, it will not be hard to find out). 

For these reasons, I am recommending the paper for the proceedings of the conference.
Decision: Accept (poster)