DLAD: Improving Logits-based Detector without Logits from Black-box LLMs

Authors: cong  zeng
shengkun  tang
xianjun  yang
yuanzhou  chen
yiyou  sun
zhiqiang  xu
yao  li
haifeng  chen
wei  cheng
dongkuan  xu
  mbzuai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:54:46

Abstract: The advent of Large Language Models (LLMs) has revolutionized text generation, producing outputs that closely mimic human writing. This blurring of lines between machine- and human-written text presents new challenges in distinguishing one from the other – a task further complicated by the frequent updates and closed nature of leading proprietary LLMs. Traditional logits-based detection methods leverage surrogate models for identifying LLM-generated content when the exact logits are unavailable from black-box LLMs. However, these methods grapple with the misalignment between the distributions of the surrogate and the often undisclosed target models, leading to performance degradation, particularly with the introduction of new, closed-source models. Furthermore, while current methodologies are generally effective when the source model is identified, they falter in scenarios where the model version remains unknown, or the test set comprises outputs from various source models. To address these limitations, we present \textbf{D}istribution-\textbf{A}ligned \textbf{L}LMs \textbf{D}etection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection even without logits from source LLMs. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations with minimal training investment. By leveraging corpus samples from publicly accessible outputs of advanced models such as ChatGPT, GPT-4 and Claude-3, DALD fine-tunes surrogate models to synchronize with unknown source model distributions effectively. Our approach achieves SOTA performance in black-box settings on different advanced closed-source and open-source models. The versatility of our method enriches widely adopted zero-shot detection frameworks (DetectGPT, DNA-GPT, Fast-DetectGPT) with a `plug-and-play' enhancement feature. 
Extensive experiments validate that our methodology reliably secures high detection precision for LLM-generated text and effectively detects text from diverse model origins through a singular detector.
Our method is also robust under the revised text attack and non-English texts.

Main paper: 
 introduction: 
 Large language models (LLMs) such as ChatGPT [1] , GPT-4 [2] , Llama [3] [4] [5] and Claude-3 [6] have profoundly impacted both industrial and academic domains, reshaping productivity across various sectors including news reporting, story writing, and academic research [7] . Nevertheless, their misuse also raises concerns, particularly regarding the dissemination of fake news [8] , the proliferation of malicious product reviews [9] , and instances of plagiarism [10] . Instances of AI-synthesized scientific abstracts deluding scientists [11, 12] have raised doubts about the reliability of scientific discourse. Accurate and reliable machine-generated text detection methods are necessary in order to address these issues [13] [14] [15] [16] [17] .Methods for detecting text generated by Large Language Models are broadly categorized into watermarking [18] [19] [20] [21] , training-based classifiers [22] [23] [24] [25] [26] , and zero-shot detectors. Watermarking methods discreetly embed identifiable markers within the text output, striving to retain the model's linguistic integrity. However, this tactic is implementable solely by the model provider. Trainingbased classifiers, while effective, are costly and often lack the agility to adapt to new domains or model updates. Our emphasis is on zero-shot detectors that exploit the intrinsic differences between text written by machines and humans, offering the advantage of being generally training-free.Most zero-shot detectors primarily depend on analyzing model output logits for detection. Notably, DetectGPT [27] operates on probability divergence based upon principles of perturbation theory, while DNA-GPT [28] harnesses reprompting-based probability divergence, and Fast-DetectGPT [29] builds on variations in conditional probability distributions. In scenarios requiring the scrutiny of black-box models, these strategies commonly leverage a surrogate model to approximate the behavior of the target model. However, this approach is doubly flawed: firstly, detection efficacy is inextricably linked to a meticulously tailored surrogate model, with different surrogate models often necessary for accurate detection across various proprietary LLMs; secondly, the fleeting nature of LLM updates renders past surrogates, once effective, obsolete against new versions. For instance, our analysis of the performance of Fast-DetectGPT [29] , using GPT-Neo-2.7B as a surrogate, against freshly updated closed-source models reveals erratic and predominantly diminishing accuracy, as contextualized in Figure 2 , with particularly stark declines in performance on iterations like GPT-3.5-1106, highlighting the intrinsic limitation of static surrogate models in adapting to LLM progressions. In our study, we seek to address the following pivotal inquiries: 1) Can we devise a feasible, cost-effective strategy to refine the probability distribution similarity between the surrogate model and opaque black-box LLMs? 2) Does enhancing the alignment of the surrogate model's probability distribution with that of the target black-box LLM improve detection outcomes for current logits-based detection methods? 3) Is it attainable to develop a universal detection model capable of adapting swiftly to updates across various target LLMs? Addressing the first question, our findings, as demonstrated in Figure 1 , reveal that our optimized surrogate model (GPT-2_DALD) mirrors the distribution of the target model more closely, in contrast to the original surrogate model's significantly divergent distribution from the target model.In this paper, we introduce an innovative yet straightforward distribution-aligned framework for black-box LLM detection, dubbed DALD (Distribution-Aligned LLM Detection). Our methodology focuses on synchronizing the surrogate model's distribution with the proprietary target model's distribution. Concretely, we accumulate a compact dataset (<10K samples) from the publicly shared outputs of leading models and subsequently fine-tune our surrogate model using this dataset to better approximate the target model's distribution. We provide the theoretical analysis of the surrogate model distribution alignment in Appendix 6. Our methodology builds upon the following observation:In logits-based detection methods, a surrogate model that closely mirrors the probability distribution curves of the target black-box LLM is instrumental in enhancing detection accuracy.We posit that this observed effect stems from the foundational assumptions inherent in logits-based detectors and proceed to examine the ramifications of this postulate in tackling the third question.To sum up, our contributions are as follows:• The introduction of DALD, a framework that significantly improves the performance of surrogate models in detecting LLM-generated text generated by both closed-source and open-source models. • DALD's unique ability to enhance detection without reliance on knowledge of the source model -a game-changer in a domain where the source is often unknown. • The capability of a single detector, enabled by DALD, to accurately identify text from varying sources, democratizing detection across diverse LLM outputs. • DALD's agility in keeping pace with rapid updates of LLMs, ensuring the latest models fall within its detection capabilities without extensive retraining.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please check the result in Tables 1, 2 , 3, 4, 5 and Figures 2, 5 .Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Detection of LLMs-Generated Text. The burgeoning capabilities of advanced large language models (LLMs) underscore the imperative for robust methodologies aimed at detecting these models. Specifically, the detection is to distinguish whether a given text originates from a language model on the condition that the model is known (White-box) [27] or unknown (Black-box) [30] . The earlier work focused on feature-based methods, like [31] [32] [33] [34] . While in the era of LLMs, the training-based methods [31, [35] [36] [37] are aroused to counter with LLMs's strong ability to produce high-quality text. They usually involve training a binary classifier using text generated by AI or humans. Besides, zero-shot detectors leverage the inherent statistical feature differences between LLMs and humangenerated text without requiring training, including probability curvature (DetectGPT [27] ), N-gram divergence (DNA-GPT [28] ), and conditional probability curvature (Fast-DetectGPT [29] ), the editing distance of the output [38] , and style representations [39] , enhancing their ability to adapt to new data distributions and source models.Black-box Detection. Given the proprietary nature of the latest LLMs [1, 2, 6] , there is a critical need for effective black-box detection methods. Present techniques falter when direct access to the source model is restricted. The training-based methods, like OpenAI text classifier [40] , GPTZero [22] , G3detector [41] , and GPT-Sentinel [42] usually closely adhere to the specific distributions of text domains and source models during training, thereby lacking generalization ability and robustness on model updates. For zero-shot methods [43, 38, 27-29, 44, 45, 34, 46, 14, 47-49] in the blackbox detection settings, they usually rely on a surrogate model for scoring. However, the efficacy of these surrogate models often falls short compared to white-box detection, where access to the source model is available. Moreover, these detection frameworks suffer from diminished accuracy when language models undergo updates [50] , which intrinsically evolve through exposure to varied datasets and human input [30] . This study presents an innovative black-box detection method for LLM-generated text, greatly enhancing surrogate model performance while adeptly accommodating the rapid evolution of LLMs. Our approach diverges from conventional training-intensive techniques by requiring only a minimal dataset for effective training.Fast-DetectGPTx Compare Sample Model x 1 x 2 x N … 1 N ∑ i log p(x | x) p( x i | x) > ϵ? Sample Scoring Misaligned Surrogate Model x p( x 1 | x) p( x N | x) p(x | x) … by Human Ours x Compare Sample Model x 1 x 2 x N … 1 N ∑ i log p(x | x) p( x i | x) > ϵ?Ditribution-Aligned Surrogate Modelx p( x 1 | x) p( x N | x) p(x | x) …Parameter-Efficient Instruction Tuning Passage x: "Unemployment fell to 1.69 million between January and March, down 2,000 from… "Figure 3 : An overview of our proposed DALD framework. Our method aligns the distribution of the surrogate model and the target model.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please check the Limitation & Future Works section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
task and settings: 
 Our task is to detect whether the given input passage x = [x 1 , ..., x L ] (L is the sequence length) is produced by an AI model f tar or a human, which can be considered as a binary classification task. Typically, there are two different task settings for LLM detection, namely white-box and black-box detection. In the black-box setting, we only have access to the generated text, treating the language model as a "black box" where we input text and receive output without knowing the internal workings or probabilities. In the white-box setting, we have additional information about the model, specifically the output probabilities p(x l |x [1:l-1] ) for each token at each position l in the text. However, in the practical scenario, it is usually difficult to get access to the source model, especially widespread but closed-source models such as ChatGPT, GPT-4 and Claude-3. Therefore, we focus on improving the black-box detection without any access to the source model logits in our setting.Logits-based LLM detection methods compute a metric by discrepancy gap hypothesis of humans and machines for classification. For example, based on the observation that the LLM-generated text occupies negative curvature regions of the model's log probability function, DetectGPT [27] proposes to utilize the source model for scoring, which refers to the white-box settings. Following DetectGPT, Fast-DetectGPT [29] replaces the perturbations-based sampling method with conditional probability sampling to accelerate the inference speed and improve the detection performance. Formally, given an input passage x and the target source model p θ , Fast-DetectGPT chooses another accessible but open-sourced model s θ for scoring, which is called the surrogate model. Together with a sampling model q φ , Fast-DetectGPT defines the conditional probability p( x|x) asEQUATIONwhere x is a sample generated by the sampling model q φ . The detection process typically consists of a three-stage procedure. The sampling step uses a sampling model to generate alternative samples x conditioned on x based on the next token prediction. Following the sampling step, the process proceeds by calculating the conditional score. The conditional probability can be obtained through a single forward pass of the scoring model, utilizing x as the input. All the conditional probabilities of samples can be obtained in the same predictive distribution. Finally, compare conditional probabilities of the passage and samples to calculate the curvature.The Challenge. In black-box settings, selecting the appropriate surrogate model in black-box settings is crucial for achieving accurate and reliable results since there is distribution misalignment between the surrogate model and the target model. Poorly chosen surrogate models may lead to bad results and a lack of explainability. Besides, with the closed-source trend of newly published LLM models, the performance can drop significantly when applied to new and advanced models, which can limit their utility and effectiveness. How to obtain a surrogate model that can fit the target models, especially closed-source models is a challenging task in black-box settings.Misalignment of Surrogate Model and Target Model. Our method is motivated by the observation that there is a distribution gap between the given surrogate model and the target source model as shown in Figure 1 . The goal of our method is to obtain a surrogate model to approach the distribution of the target model by utilizing the texts generated by the target model. To achieve that, we propose a novel and simple framework to train a distribution-aligned surrogate model, which outperforms SOTA black-box methods with a small-size dataset (<10K). The architecture of our method is shown in Figure 3 . Our framework consists of two steps in total. The first step is to collect small-size training data generated by the source model from the publicly shared outputs. With the training dataset, we finetune the surrogate model to align the distribution of the source model.Alignment Data Collection. Given the target model f tar and surrogate model f sur , in order to align the distribution of the surrogate model and target model, we collect a small-size dataset S = {(P i , X i )} N i=1 for a specific target model, referred as alignment dataset, where N refers to the number of collected samples, P i is the text for prompting and X i is the corpus generated by the target model f tar . The model version of the collected data should be exactly the same as the target model, especially for closed-source models such as ChatGPT and GPT-4. For example, if the test data is generated by GPT-4-0613, then all of the texts in the dataset S should also be generated by GPT-4-0613. We utilize the collected dataset S to finetune the surrogate model f sur to align the distribution with target model f tar .Distribution-Aligned Surrogate Model Training. As illustrated in Figure 3 , our approach expands the scoring step of previous logits-based methods such as Fast-DetectGPT by incorporating an additional surrogate model finetuning step. Given the surrogate model f sur , we construct the Low-Rank Adaptation (LoRA) [51] of surrogate model f sur for faster and more stable fine-tuning. The LoRA model f sur+Θ is trained with a collected dataset while the parameter of the original surrogate model f sur is frozen. With collected dataset S = {(P i , X i )} K1 i=1 where K 1 is the number of samples, we concatenate the prompt and generated text as y = [P, X]. The model f sur+Θ utilizes the tokenized x as input and is trained in a self-supervised learning manner. The training objective of our fine-tuning is:EQUATIONwhere l(X) denotes the length of a passage X, and y l is the next token to be predicted. In order to disable the influence of the prompt, we follow typical instruction tuning to mask the gradient of the prompt. As shown in Figure 1 , after training, the misaligned model generates a similar distribution as the target source model, demonstrating the effectiveness of our method. Following that, the distribution-aligned surrogate model can be utilized to compute the logits for downstream decisions.Under an assumption on the sample complexity of fine-tuning with the above loss function, we theoretically demonstrate the effectiveness of fine-tuning on approximating the target model in the following theorem, using conditional probability curvature from Fast-DetectGPT [29] :Theorem 1.With fine-tuning sample size K 1 = Ω(poly(∆/L)), with probability 1 -δ, we have that given a text segment X with length l, the conditional probability curvature between the two models is bounded byd(X, f sur ) -d(X, f tar ) ≤ ∆/3.A detailed proof of this theorem can be found in Appendix 6. Given the hypothesis that there is a positive gap ∆ in conditional probability curvature between human-generated text and machinegenerated text, the corresponding gap calculated from the surrogate model will still be significant.Table 1 : Detection accuracy comparison on three source models ChatGPT (GPT-3.5-Turbo-0301), GPT-4 (GPT-4-0613) and Claude-3 (claude-3-opus-20240229). Our method surpasses previous methods on all passages generated from different source models. Datasets & Evaluation Metric. We follow Fast-DetectGPT using four datasets in the black-box detection evaluation, including Xsum [52] , WritingPrompts [53] , WMT-2016 [54] and PubMedQA [55] .We randomly sample 150 examples of each dataset as human-written texts. Then based on the samples, we prompt the target closed-source models by API to generate the corresponding texts using the 30 tokens of human-written text as the machine-generated text. For text diversity and quality, we employ a temperature of 0.8 which is the same setting in Fast-DetectGPT. For the training dataset, we collect the corpus from the publicly shared outputs of leading models. Following previous works [29] , we compute the accuracy in the area under the receiver operating characteristic (AUROC) to evaluate the performance of all methods. We also provide the area under the precision and recall (AUPR) in Appendix 8.6.Source & Surrogate Models. To validate our idea in black-box detection, we include the most advanced closed-source LLMs from OpenAI: ChatGPT, GPT-4, and Claude-3 from Anthropic. Since these models keep being updated by their owner company, we use the version GPT-3.5-turbo-0301 for ChatGPT, GPT-4-0613 for GPT-4, and claude-3-opus-20240229 for Claude-3 if not specified. We utilize Llama2-7B as the surrogate model in Table 1 . Note that our method can be adapted to any open-source model. Therefore, we provide our results on other surrogate models in Table 3 .Baseline Methods. We consider training-based baselines and zero-shot baselines. We mainly consider three strong baselines for black-box detection: Detect-GPT [27] , DNA-GPT [28] and Fast-DetectGPT [29] . Detect-GPT. Detect-GPT applies T5-3B [56] as a sampling model to generate perturbed texts and utilizes GPT-Neo-2.7B [57] as a surrogate model to compute the probability curvature of perturbed texts. After that, perturbation discrepancy is obtained to determine whether the given text is generated by AI or humans. Fast-DetectGPT uses GPT-J-6B [58] and GPT-Neo-2.7B as the sampling model and surrogate model respectively to compute the conditional probability curvature. Finally, DNA-GPT utilizes GPT-Neo-2.7B as a surrogate model to regenerate the texts for metric computation. Details about other baselines are described in Appendix 7.Implementation Details. We collect the data of ChatGPT and GPT-4 from WildChat [59] while the data of Claude-3 is generated by calling Claude-3 using the prompts from WildChat. During training, we randomly choose 5K prompts and responses. We applied the instruction tuning to model training to ignore the human-written prompts. For surrogate model training, we apply parameter-efficient fine-tuning (PEFT) via Low-Rank Adaptation (LoRA). We do not tune the hyperparameters carefully. Therefore, more details about training parameters can be found in the Appendix 7. For training time, our method finetunes Llama-2-7B with 5K samples on 4 A6000. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We will include our code in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
main results: 
 Black-Box Machine-Generated Text Detection. We compare our method with mainstream blackbox LLM detection methods in Table 1 . The detection accuracy shows that our method achieves the best performance compared with other methods including Detect-GPT, DNA-GPT and Fast-DetectGPT. Moreover, it is noteworthy that our method obtains more than 99% AUROC on XSum-GPT-4, Writing-GPT-4 and Writing-Claude-3. As shown in Figure 4 , the ROC curve for DALD achieves the highest TPR at the same FPR across all three datasets compared with DNA-GPT and Fast-DetectGPT, indicating superior performance. Besides, comparing the performance across different datasets in previous methods, we can observe that the Writing dataset is much easier to detect while PubMed is the hardest. The possible reason is that PubMed is a much more medicalspecific dataset while the related corpus is not comprised of surrogate model pre-training. However, our method gains significant improvement upon Fast-DetectGPT. For example, on PubMed-GPT-4, Fast-DetectGPT only obtains 0.8179 AUROC while our method achieves 0.9785, demonstrating the effectiveness of distribution alignment in our method.Dataset Size. Our method requires only a small amount of data for training. The model converges quickly with the small-size dataset. To show the training efficiency of our method, we provide the results with various training dataset sizes in Figure 5 . For each dataset, we use GPT-4 as the target source model while Llama-2-7B and Llama-3-8B as the surrogate model respectively. With the increasing amount of training data, the performance increases rapidly and stays stable with more data. Moreover, with around 500 (up to 1000) training samples, the performance of our method matches the baseline on both Llama-2 and Llama-3. Besides, our method achieves its best performance and exceeds the baseline with around 2000 training data, which indicates that with only a little training effort, our method can achieve incredibly better performance than the baseline, even more than 99%. Generalizability. Our method aligns with the distribution between the surrogate model and source model with the texts generated by source models. In this section, we further explore the generalizability of our method where we follow several settings: 1) train the surrogate model separately using the data from the corresponding source model, 2) train the model with the single data source and evaluate it on unknown source models (one-for-all), 3) train the model with mixed data sources. We conduct a group of experiments following the settings, as shown in Table 2 . We use Llama2-7B as the surrogate model and a total 5K training data in all settings. The dataset with a single data source includes 5K texts from the corresponding source model in the first setting and only GPT-4 in the one-for-all setting while two data sources consist of 2.5K ChatGPT and GPT-4 texts, respectively. Three data sources refer to the combination of 1.3K texts each from ChatGPT, GPT-4 and Claude-3. Surprisingly, the models trained with more data sources achieve better accuracy on GPT-4. However, the performance on PubMed-ChatGPT only shows negligible degradation in one-for-all and mixed data source settings. The superior performance in the one-for-all setting implies the surrogate model trained with DALD can be extended to texts of unknown source models. The results demonstrate the generalizability of our method, leading to training a universal surrogate model for all closed-source models and detecting the machine-generated texts without knowing the model source. Finally, the results in the one-for-all setting imply current closed-source models tend to have a similar distribution. Evaluation results on more unknown source models can be found in Appendix 8.2. The results are shown in Table 3 with details in each dataset. In all, compared with the original surrogate models, the surrogate models with DALD obtain much higher accuracy. For example, Llama-2 and Llama-3 only obtain 0.8876 and 0.7764 on PubMed-Claude-3 while their counterparts trained with DALD achieve 0.9424 and 0.9192. The improvement across various surrogate models suggests that our approach is compatible with a range of surrogates, rather than just a particular carefully chosen surrogate model.Ablation Study. We conduct a group of ablation studies on several datasets, as shown in Table 4 .Since our method can be adapted to any previous logits-based methods such as DNA-GPT and Fast-DetectGPT, the ablation study is conducted on top of them to further demonstrate the effectiveness of our method. We choose Llama2-7B as the basic surrogate model for all experiments. We compare the results of the baseline model and the baseline model trained with our framework. In general, our method boosts the performance of the baseline model at different scales. For example, DNA-GPT achieves 0.8947 accuracy score on PubMed-GPT-4 while with the surrogate trained by our Non-English Detection. Recent work [60] finds that current AI detectors are biased for non-English languages, which hinders the application of LLM detection for non-English languages. Following [28] , we choose English and German splits of WMT-2016 [54] to test the ability of our method in German. We select 150 instances as human-written texts and use the first 30 tokens to regenerate by calling GPT-4 API as machine-generate texts.During training, we randomly select 1K German samples generated by GPT-4 from WildChat [59] and trained for 5 epochs. As shown in Table 5 , our method achieves the highest accuracy (> 99%) on German detection compared with DNA-GPT and Fast-DetectGPT. Due to the plug-and-play property, our method can be further used to eliminate the bias in other AI detectors. Adversarial Attack. In practical situations, machine-generated corpus is often modified and revised by users or another language model. We consider the modified samples as adversarial samples. Evaluating LLM detectors with adversarial samples is important to real-world applications. Following [27] and [28] , we randomly mask r% tokens with 5-word spans in 150 instances from the PubMed dataset regenerated by GPT-4 and apply T5-3B to do the mask-filling task to generate adversarial samples. Experiments with different mask ratios are conducted, specifically r% ∈ {0.1, 0.2, 0.3, 0.4, 0.5} and results are shown in Figure 6 . We compare the results of DNA-GPT, Fast-DetectGPT and their counterparts with our method. In each group, with the enhancement of our method, the model achieves better results on all mask ratios. Moreover, the models with our method obtain even better accuracy at the highest mask ratio compared with the original models on samples without adversarial attack.Open-source Model Detection. In addition to detecting texts from closed-source models, we also evaluate our approach on the open-source model Llama-3, Llama-3.1 [61] and Mistral [62] comparing it with DNA-GPT and Fast-DetectGPT. We follow similar settings as closed-source models and fine-tune Llama2-7B as surrogate model. The results are shown in Table 6 , where our method works best detection performance on three models compared to other methods, showing the effectiveness of our method on both closed-source and open-source models.  
conclusion: 
 The rapid evolution of potent Large Language Models (LLMs) underscores the critical necessity for robust black-box detection methods. However, previous methods which rely on surrogate models, suffer from performance degradation, especially with the frequent updates of closed-source models.Our contribution addresses this shortfall by significantly aligning the distribution of the surrogate model and source model. Additionally, we introduce a plug-and-play approach for logits-based detectors, ensuring seamless integration. This method remains versatile across diverse text sources or unknown sources, adapting to the swift evolution of LLMs. In conclusion, our innovations offer compelling solutions to the urgent demand for effective black-box detection methods within the realm of LLM development, bridging critical gaps in current methodologies.Appendix: DALD 6 Theoretical Analysis 
addtional technical details: 
 First, we formalize some concepts in preparation of our analysis. We assume the vocabulary set to be W = {w j } W j=1 with size W . We will use slices to denote a segment of passage, but unlike python slices, [a : b] will be inclusive on the end points and should denote the indices {a, a + 1, • • • , b}.Text Completion. Consider the next-token generation process with respect to a language model f , which maps an input passage into an array of logits on the vocabulary set W: Definition 2. Given a prompt X [:l0] of length l 0 , a new length l > l 0 and language model f , we useX [l0+1:l] ∼ M f (X [:l0] , l)to denote the process of text completion up to a total of l tokens with respect to the following sampling method:X l0+i ∼ softmax f (X [:l0+i-1] ) , i = 1, • • • , l -l 0 .In the following we will abbreviate M sur = M fsur and M tar = M ftar , and also denote p sur (X) = softmax f sur (X) , p tar (X) = softmax f tar (X) by for any text segment X.Loss Function for Fine-tuning. Given target model f tar , we fine-tune a surrogate model f sur on the following dataset generated by the target model:S = X i = [P i , R i ] P i ∼ P, R i ∼ M tar (P i , L) N i=1 ,where each P i is a prompt sampled from a prior distribution P, and its corresponding R i is a text completion result sampled from distribution M tar . Now let us focus on a single text data X = [P, R] ∈ S with prompt length l(P ) = l 0 and total length l(X) = L. For any l 0 < l ≤ L, consider the next-token logit p = p sur (X [:l-1] ) ∈ [0, 1] T of the surrogate model for input X [:l] , as well as a sample output from the target model X l ∼ q = p tar (X [:l-1] ). The cross-entropy loss is then calculated withℓ(X l |X [:l-1] ) = - W j=1 1{X l = w j } log p j , which has expectation E X l ∼q ℓ(X l |X [:l-1] ) = - W j=1 q j log p j = H(q, p) ≥ H(q),where H(q) and H(q, p) denote the entropy of q and the cross-entropy between q and p, respectively. It is then evident that the optimal expected value of ℓ(X l |X [:l-1] ) is H(q), taken when the surrogate model f sur produces the same logit output as f tar .With this, we can express the training objective used for fine-tuning the surrogate model as:L(f sur , S) = 1 |S| X=[P,R]∈S l(X) l=l(P )+1 ℓ(X l |X [:l-1] ),We now make the following assumption, which is supported by many previous works [63] Assumption 3. The sample complexity of fine-tuning with loss L(f sur , S) is of order O(1/ϵ 2 ) × O(log(1/δ)). In other words, when sample size |S| exceed this order, for almost all X = [P, R] and a corresponding l > l(P ), we haveEQUATIONRemark 4. The result of this assumption is that the surrogate model should approximate the target model well after fine-tuning. We are taking quite a liberty in this assumption in terms of input X, but this is simply for the sake of demonstration. In practice, what one can expect is the expected loss E S L(f sur , S) to be almost optimal, and with generalization this assumption should hold approximately.Fast-DetectGPT Method. For two passages X, X with the same length L, the conditional probability is defined asp f ( X|X) = L l=l0+1 P y = X l |y ∼ softmax f (X [:l-1] ) ,where l 0 is a pre-selected length parameter.In order to evaluate a passage X with length L, we sample new text passages X ∼ p f (•|X) to estimate the conditional probability curvature:d(X, f ) = log p f (X|X) -µ σ , where µ = E X∼p f (•|X) log p f ( X|X) , σ 2 = E X∼p f (•|X) log p f ( X|X) -µ 2 ,The basic assumption of Fast-DetectGPT is a positive gap between machine-and human-generated passages in terms of conditional probability curvature: Assumption 5. The conditional probability curvature of a machine-generated passage X tar from model f tar is substantially greater than that of a human-generated passage X hum :d(X tar , f tar ) -d(X hum , f tar ) ≥ ∆.Now with the above assumption, we can prove Theorem 1, which characterizes the difference between the target model and surrogate model.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Please check each experimental section, we have a detailed description of our experiment settings.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
additional experimental results: 
 8.1 Full Results on ChatGPT(GPT-3.5-Turbo)We provide the full result on ChatGPT(GPT-3.5-Turbo), as shown in Table 8 . Fast-DetectGPT obtains great performance on ChatGPT. Our method boosts their performance with a significant 100% accuracy on Xsum and Writing.To show the one-for-all ability of our method, we provide more results on more source models using the surrogate model trained only on data generated from GPT-4-0613, as shown in Table 9 . We evaluate our method on different versions of GPT-3.5, GPT-4, GPT-4o and GPT-4o-Mini. In general, our method shows robustness on all versions of GPT-3.5 and GPT-4, achieving more than 98% accuracy except on PubMed-GPT-4-1106-Preview but still more than 97%. The superior performance in the one-for-all setting shows the generalizability of our methods, leading to further motivate to train a universal surrogate model for various source models regardless of model updates.We include a performance comparison of the coding task. We follow [67] to apply APPS [68] dataset as the coding task. We sample 150 coding tasks from APPS and generate the coding results by calling GPT-4 API. Our method is only trained by the corpus generated by GPT-4 as we previously did. The results are in Table 10 , Our method obtains significant improvement on the coding task. We include an evaluation of the detection performance of different text genres. Using a domainspecific datasets RAID, we select 1000 human texts and 1000 GPT-4 generated texts for each domain in RAID dataset and compare the evaluation results with FastDetectGPT. The results is shown in Table 11 , which demonstrate that DALD performs admirably in diverse domains. Our main experiment follow previous works such as DNA-GPT and FastDetectGPT and utilize the same amount of data for fair evaluation. Furthermore, we provide the evaluation results of our method on different test data sizes, as shown in Table 12 . It is observed that there is little difference in performance as the number of samples increases, indicating the robustness of our method to test data size. [29] , we compute the area under the receiver operating characteristic (AUROC) to measure the performance of different methods, evaluating the method performance with different classification thresholds. AUROC falls into the value range [0, 1], which provides a quantitative measure of the likelihood that a randomly generated passage has a higher predicted probability of being machine-generated than a randomly written passage by a human. Generally, the value of 0.5 in AUROC indicates a random classifier without any classification ability while 1 in AUROC refers to a perfect classifier for all samples. The results comparison of our method and other baselines including training-based methods and zero-shot methods are provided in Table 1 .AUPR. Similar to AUROC, AUPR computes the area under precision and recall, which evaluates both the precision and recall of classifiers with different thresholds. AUPR also ranges from 0 to 1. A higher AUPR value represents a better classifier, which generally obtains higher recall on the condition that the precision is high. Therefore, we provide the AUPR comparison of our method with other baselines in Table 13 . We compare all baselines including training-based such as RoBERTabase and zero-shot methods such as DetectGPT, DNA-GPT and Fast-DetectGPT. First of all, we can observe that the current mainstream zero-shot methods generally achieve better results compared with the training-based methods. Moreover, our method achieves the highest AUPR results (> 98%) on all datasets and source models, further demonstrating the effectiveness of our method. The limitation of our method is the absence of additional experiments conducted in multilingual settings. While our method has shown efficacy in English and German environments, its performance across different languages remains unexplored. Future studies should endeavor to evaluate the generalizability of our approach in multilingual contexts to ensure its applicability across diverse linguistic landscapes.Table 14 : Examples of data from WildChat and Claude-3-Opus-Instruct-15K. Datasets Samples WildChat GPT-3.5-Turbo-0301 Prompt: who is theseus in plutarch book? Theseus is a legendary hero and king of Athens in Plutarch's book. Plutarch presents him as a great leader and conqueror who united the scattered towns of Attica and established Athens as the political and cultural center of the region. Theseus is also famous for his defeat of the Minotaur, a monstrous creature with the head of a bull and the body of a man who lived in a labyrinth on the island of Crete. According to Plutarch, Theseus also instituted many reforms in Athens, such as the legal code and the Panathenaic festival. Claude-3-Opus-Instruct-15K Prompt: Explain how the human circulatory system functions, covering the structures and processes involved in transporting blood throughout the body. Include details about the major components of the system (heart, blood vessels), how blood is pumped through arteries and veins, and the importance of blood circulation for keeping tissues and organs functioning properly. Response: The human circulatory system is responsible for transporting blood throughout the body in order to deliver oxygen and nutrients to tissues and organs and remove metabolic waste products. The major components are: The Heart: A muscular organ that acts as a pump to propel blood through the blood vessels. It has four chambers -two atria and two ventricles. The right side of the heart pumps deoxygenated blood to the lungs, while the left side pumps oxygenated blood to the rest of the body. Arteries: Blood vessels that carry oxygenated blood away from the heart to the body's tissues. They have thick, muscular walls to withstand the high pressure of blood being pumped from the heart. The aorta is the largest artery. Veins: Blood vessels that carry deoxygenated blood from the tissues back to the heart. They have thinner walls and larger lumens than arteries. Veins have valves to prevent backflow of blood. The venae cavae are the largest veins. Capillaries: Microscopic blood vessels that form a network to facilitate the exchange of substances between the blood and tissues. Oxygen and nutrients diffuse from the blood into cells, while carbon dioxide and wastes move from cells into the blood. The pumping of the heart drives circulation: 1. Deoxygenated blood from the body enters the right atrium, flows into the right ventricle, and is pumped to the lungs via the pulmonary arteries. 2. In the lungs, blood picks up oxygen and releases CO2. Oxygenated blood returns to the left atrium via the pulmonary veins. 3. Blood flows into the left ventricle and is forcefully pumped out to the body through the aorta. 4. The aorta branches into smaller arteries, then arterioles, and finally capillaries where gas and nutrient exchange occurs. 5. Deoxygenated blood collects in venules, flows into veins, and returns to the right atrium via the venae cavae. The cycle repeats. Proper circulation is critical for survival. It ensures that all cells receive oxygen and nutrients needed to function. It enables removal of metabolic wastes that could be toxic if allowed to accumulate. Circulation also helps distribute hormones, transport immune cells, and regulate body temperature and pH. Disruption of blood flow, such as from a blockage, can quickly lead to tissue damage or death.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: We all use public resources in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A direct result of Assumption 3 is that the Kullback-Liebler devergence between the two distributions are small:KL p tar (X [:l] )||p sur (X [:l] ) = H p tar (X [:l] ), p sur (X [:l] ) -H p tar (X [:l] ) ≤ ϵ.As per Pinsker's inequality, this gives usd TV p tar (X [:l] ), p sur (X [:l] ) ≤ 1 2 KL(p tar (X [:l] )||p sur (X [:l] )) ≤ ϵ 2 .Now to analyze the difference between conditional probability curvature, we have:log p fsur (X|X) -log p ftar (X|X) = L l=l0+1 log p sur (X l |X [:l-1] ) -log p tar (X l |X [:l-1] ) ≤ L l=l0+1 d TV (p sur , p tar )/C, where C = min p(X l |X [:l-1]) is the minimum possibility given to a next-token. For both machine and human generated passages, this value should be reasonably large. On the other hand,µ sur -µ tar = E X∼p fsur (•|X) log p fsur ( X|X) -E X∼p f tar (•|X) log p ftar ( X|X) = L l=l0+1 E X l ∼psur(X [:l-1] ) log p sur ( X l |X [:l-1] ) -E X l ∼ptar(X [:l-1] ) log p tar ( X l |X [:l-1] ) = L l=l0+1 H(p sur (X [:l-1] )) -H(p tar (X [:l-1] )) ≤ L l=l0+1 d TV (p sur , p tar ) log(W -1) + h(d TV (p sur , p tar )) ,where the inequality is from [64] . With this we have that whenϵ = O h -1 (∆/L) 2 , the difference d(X, f sur ) -d(X, f tar ) ≤ ∆/3.This requires the sample complexity to be Ω(poly(∆/L)).An immediate corollary of this theorem is that under the same conditions, d(X tar , f sur ) ≥ d(X hum , f sur ) + ∆/3, which means the positive gap is still present once we replace the target model with the surrogate model. More Implementation Details. We describe more implementation details of our framework. For LoRA hyper-parameters, we utilize 16 as the LoRA rank and set lora_alpha as 32. Dropout is set as 0.05. For the Llama series, we adopt the LoRA module on ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "down_proj", "up_proj"] while in GPT-Neo models, we apply on ["q_proj", "v_proj", "k_proj", "out_proj", "c_f c", "c_proj"].For training hyperparameters, we set 512 as the max length for texts from GPT-4 and GPT-3.5 models while it is 2048 for texts from Claude-3. We finetune the surrogate model with a learning rate of 1e-4. The batch size is set as 1 per device with gradient accumulation per 4 steps. It's noteworthy that we do not choose our hyperparameters carefully and we believe it will achieve better results with careful tuning. The training costs less than 10 minutes with 4 A6000. 


Decision: The paper proposes a method to fine-tune a surrogate LLM against texts generated by the target LLM as a means of improving the ability to detect/guess if a given text is generated from the aforementioned target LLM. The idea is simple and requires a relatively small number of generations to improve the model's efficacy. The reported effect size is quite significant, by 10% in many cases. 

While a number of borderline reviews are present, most of the concerns are non-falsifiable. e.g., lacking novelty but not citing prior work that has followed this approach, or that the explanation for the idea/why it works is not sufficient (what makes something sufficient?). I do not consider these substantive reasons to reject the work, given that the effect size is large (let theory follow empirical results) and the simplicity of the approach (if it does not work in other contexts, it will not be hard to find out). 

For these reasons, I am recommending the paper for the proceedings of the conference.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenge of detecting machine-generated text from black-box LLMs without access to their logits. Traditional methods using surrogate models suffer from performance degradation due to misalignment with target model distributions, particularly as new models are introduced. The proposed Distribution-Aligned LLMs Detection (DALD) framework aligns surrogate model distributions with unknown target LLMs. DALD enhances detection capabilities, achieving state-of-the-art performance in black-box settings, and offers a plug-and-play enhancement to existing zero-shot detection frameworks. Extensive experiments validate DALD's high detection precision against revised text attacks and non-English texts.
strength_weakness: strength_weakness: 1. The idea lacks novelty; collecting data from models with the same version as the target model and then aligning the surrogate model is not novel.
2. The proposed method appears to heavily rely on alignment data collection and specific model versions. Does this mean the surrogate model needs continuous fine-tuning, when the detection task varies? If so, what is the generalizability?
3. The paper missing abalation studies on how much fine-tuning data is required when the detection task varies in different categories. If the data requirement is large, it could pose a problem for the detection task.
4. The paper is missing important baselines such as RADAR [1] and could benefit from evaluation on more models like Gemini and additional tasks, including coding tasks.

## Reference

[1]. Radar: Robust ai-text detection via adversarial learning. NeurIPS 2023.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Please refer to the weaknesses.
paper_summary: paper_summary: This paper introduces Distribution-Aligned LLMs Detection (DALD), a novel framework for detecting AI-generated text from large language models (LLMs). DALD addresses limitations of traditional detection methods, particularly when dealing with black-box or unknown LLMs. It aligns surrogate models with unknown target LLM distributions, enhancing detection capability across various models without requiring access to their internal logits. The approach leverages corpus samples from publicly available outputs of advanced models to fine-tune surrogate models, achieving state-of-the-art performance on both closed-source and open-source models. DALD can be integrated into existing zero-shot detection frameworks and demonstrates robustness against revised text attacks and non-English texts, offering a versatile solution for the evolving challenge of distinguishing AI-generated content from human-written text.
strength_weakness: strength_weakness: 1. Since DALO is a training-based method, it may be unfair to compare it with the zero-shot method(Detect-GPT, DNA-GPT, and Fast-DetectGPT). Comparing it with other SOTA training-based methods such as Ghostbuster[1] is necessary.
2. Only testing PubMed on ChatGPT while testing all three datasets on GPT-4 and Claude.
3. For reliability, it is important to see how well the detector performs at low FPR regimes [2].
4. Expanding your evaluation to include recent open-source models such as Llama 3 and Mistral/Mixtral would enhance the breadth and relevance of your study. 
5. The method is easy and lacks contribution in technology.

[1] Vivek Verma, Eve Fleisig, et al. "Ghostbuster: Detecting text ghostwritten by large language models" (2023). 
[2] Sadasivan, Vinu Sankar, et al. "Can ai-generated text be reliably detected?" arXiv preprint arXiv:2303.11156 (2023).
questions: questions: 1. Could you compare it with other SOTA training-based methods such as Ghostbuster[1]?
2. Why did you just test PubMed on ChatGPT while testing all three datasets on GPT-4 and Claude?
3. Could you provide the ROC curves for your detector? For reliability, it is important to see how well the detector performs at low FPR regimes [2].
4. Why not conduct some experiments on open-source LLM like Llama 3 and Mistral/Mixtral？

[1] Vivek Verma, Eve Fleisig, et al. "Ghostbuster: Detecting text ghostwritten by large language models" (2023). 
[2] Sadasivan, Vinu Sankar, et al. "Can ai-generated text be reliably detected?" arXiv preprint arXiv:2303.11156 (2023).
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a method to improve black-box detection of machine-generated text, tackling the problem of performance degradation when a surrogate model's output is poorly aligned with the closed-source target LLM. By LORA fine-tuning the surrogate model on text generated by the target model, the authors align the surrogate model's output distribution to the target model, thus improving detection performance.  

The authors show that this alignment can be used to outperform/improve existing logit detection methods such as DetectGPT, DNA-GPT and Fast-DetectGPT; is effective across a range of surrogate models; and requires only a small amount of fine-tuning (and thus can be keep surrogate models up-to-date when newer models are released).
strength_weakness: strength_weakness: - **Generalizability may be overstated** (Table 2): The authors cite the superior performance of their one-for-all surrogate model as evidence that their method could be extended to detect texts of unknown source models. However, the improvements seems very slight: I agree with their interpretation that current closed-source models tend to have a similar distribution, but this homogeneity might not necessarily be true for future versions of these models and/or other future closed-source models. 
- **Effectiveness across surrogate models** (Table 3): DALD doesn't significantly improve the detection performance of all surrogate models. For example, its effect on GPT2-Neo 2.7B, though positive, seems fairly minor. Does this indicate a limitation of their DALD method (e.g. that it is only effective when the performance gap/parameter size/pre-training data difference between surrogate and target is relatively small)? 
- **Limitation and Future work section** is extremely brief, and does not adequately address limitations
questions: questions: See Weaknesses Section. (and Strengths bullet point 2) 

Also, minor nitpicks / typos: 


Typos: 
- Line 187: is freezing
- Line 215: OpanAI
- Line 244: pertraining
- Line 574: We
limitations: limitations: Limitations section was extremely brief and only discussed extending their evaluation of multilingual performance.
paper_summary: paper_summary: This paper proposes a framework named Distribution-Aligned LLMs Detection (DALD) to improve the performance of surrogate models in detecting LLM-generated text from both closed-source and open-source models. The method enhances detection performance by aligning the distribution of the surrogate model to better match the distribution of the target model. The paper verifies the effectiveness of DALD on various advanced closed-source and open-source models through extensive experiments and demonstrates its "plug-and-play" enhancement capability in a zero-shot detection framework.
strength_weakness: strength_weakness: - The method's interpretability is still insufficient. I am not entirely sure, but one point that confuses me is why training DALD on datasets from multiple source models performs better than training on a dataset from a single source model (see Table 2). Since DALD essentially aligns the logits distribution of the surrogate model with that of the source model, this is a crucial starting point of the paper. However, training DALD on datasets from multiple source models theoretically and intuitively would not align distributions better than a single source model, which is contradictory. Similarly, in the Claude-3 setting, DALD trained on a GPT-4 dataset even outperforms DALD trained on a dataset from the source model Claude-3, which is perplexing.
- There is a lack of detail about the datasets used for fine-tuning, such as their sources, topics, or other attributes, which could help readers understand and reproduce the method. In fact, I am very curious about the impact of the fine-tuning dataset on DALD, aside from the sample size.
questions: questions: - Please refer to the weaknesses, especially Weakness 1.

- An additional question: I noticed that DetectGPT and Fast-DetectGPT are described as black-box methods. Is this because they use surrogate models? In fact, all logits-based methods (including Likelihood and Rank) can use surrogate models, depending on whether the scenario is white-box or black-box.
limitations: limitations: - The testing dataset is insufficient; all datasets contain only 150 human-written samples, and the experimental results obtained may actually be biased, which is indeed my concern.

Comments: title: Reply to Reviewer iaeF
Response to Reviewer yZjY
Response to Reviewer bnRm
Official Comment by Reviewer bnRm
Response to Reviewer 3RMB
General Response
comment: Dear Reviewer iaeF,

This is a kind reminder. We wanted to kindly follow up on the rebuttal discussion regarding. We highly value your insights and would greatly appreciate your response at your earliest convenience. Please let me know if there is any additional information you require or if there are any concerns we can address.

Thank you for your time and consideration.

Authors
Dear Reviewer yZjY,

Thank you very much for your advice on our work, which greatly enhances the solidity of our work. We will add all related experimental results to our main text in the next version.

Regarding your concern about our assumption and experimental results, as we discussed in the main text, the training data generated by GPT-4 is from WildChat[1], which is a high-quality dataset. However, WildChat does not include the data generated from Claude3. In order to save money, we didn't call Claude3 API to generate training data. The data of Claude3 was collected from the public output in Huggingface without a quality guarantee[2]. Therefore, the performance of the model trained by data generated from Claude is relatively worse than that trained by data generated from GPT-4.

For a fair comparison, we sampled 5k prompts from wildchat and called Claude API to generate the training dataset. Then, we utilize the generated dataset to fine-tune the surrogate model and provide the results here(we will also include this in the next version):


|                     | PubMed | Xsum   | Writing |
|---------------------|--------|--------|---------|
| DALD(GPT-4 data)    | 0.9875 | 0.9993 | 0.9977  |
| DALD(Claude-3 data) | 0.9942 | 0.9994 | 0.9993  |

The results show that the model trained by data generated from Claude3 is better than the one trained by data from GPT-4, demonstrating the correctness of our assumption and the consistency of experimental results.


[1] Zhao, Wenting, et al. "Wildchat: 1m chatGPT interaction logs in the wild." arXiv preprint arXiv:2405.01470 (2024).

[2] lodrick-the-lafted/Sao10K_Claude-3-Opus-Instruct-9.5K-ShareGPT
Dear Reviewer bnRm,

Thank you very much for your reply. We really appreciate your suggestions on our work. Regarding your concern about our assumption and experimental results, as we discussed in the main text, the training data generated by GPT-4 is from WildChat[1], which is a high-quality dataset. However, WildChat does not include the data generated from Claude3. In order to save money, we didn't call Claude3 API to generate training data. The data of Claude3 was collected from the public output in Huggingface without a quality guarantee[2]. Therefore, the performance of the model trained by data generated from Claude is relatively worse than that trained by data generated from GPT-4.

For a fair comparison, we sampled 5k prompts from wildchat and called Claude API to generate the training dataset. Then, we utilize the generated dataset to fine-tune the surrogate model and provide the results here(we will also include this in the next version):

|                     | PubMed | Xsum   | Writing |
|---------------------|--------|--------|---------|
| DALD(GPT-4 data)    | 0.9875 | 0.9993 | 0.9977  |
| DALD(Claude-3 data) | 0.9942 | 0.9994 | 0.9993  |

The results show that the model trained by data generated from Claude3 is better than the one trained by data from GPT-4, demonstrating the correctness of our assumption and the consistency of experimental results.


[1] Zhao, Wenting, et al. "Wildchat: 1m chatGPT interaction logs in the wild." arXiv preprint arXiv:2405.01470 (2024).

[2] lodrick-the-lafted/Sao10K_Claude-3-Opus-Instruct-9.5K-ShareGPT
I'm glad for your comprehensive clarification. I'm curious about what you mentioned regarding space constraints in weakness 2, as I recall that the appendix isn't page-limited. Besides, I agree with the reviewer bnRm's weakness 1 about training on different domains. I will keep my score.
I would like to thank the responses provided by authors. However, my concerns about the Weakness 1 are not well addressed. Specifically, regarding the observation that "DALD trained on the GPT-4 dataset performs better than the source model Claude3's dataset," are the logits distributions of GPT-4 and Claude3 truly similar? This contradicts my experience, and I still have doubts about the consistency between the assumptions and experimental results.
Dear Reviewer 3RMB,

Thank you very much for your comments. We will revise our manuscript accordingly in the next version.

Best,

Author
Thanks for your response. I appreciate the additional results you've provided on domain-specific datasets as well as for GPT-Neo-2.7B on GPT-4 (Weakness 2).
## General Response

Dear AC and Reviewers,

We would like to sincerely appreciate the reviewers for their positive feedback and highly constructive comments. To improve the clarity and readability of the paper, the following changes have been made and the manuscript will be revised accordingly in the next version.

* We clarified some misunderstandings of the reviewers about the method and experiment.
* We added more experiments and explanations to demonstrate the generalizability of DALD.
* We added more baseline comparisons including Ghostbuster and RADAR.
* We provided the ROC curves of our detector and other methods.
* We added more experiments on open-source models including llama3.1-8B, and Mistral-7B.
* We added more experiments on RAID datasets with more topics.
* Add a detailed discussion of the limitations and future work.

Thanks again,

The Authors