DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection

Authors: xiao  yu
yuang  qi
kejiang  chen
guoqiang  chen
xi  yang
pengyuan  zhu
xiuwei  shang
weiming  zhang
nenghai  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 09:46:26

Abstract: Large language models (LLMs) have the potential to generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Consequently, detecting whether a text is generated by LLMs has become increasingly important. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. However, since we do not have access to the interior of the black-box model, we must resort to surrogate models, which impacts detection quality. In order to achieve high-quality detection of black-box models, we would like to extract deep intrinsic characteristics of the black-box model generated texts. We view the generation process as a coupled process of prompt and intrinsic characteristics of the generative model. Based on this insight, we propose to decouple prompt and intrinsic characteristics (DPIC) for LLM-generated text detection method. Specifically, given a candidate text, DPIC employs an auxiliary LLM to reconstruct the prompt corresponding to the candidate text, then uses the prompt to regenerate text by the auxiliary LLM, which makes the candidate text and the regenerated text align with their prompts, respectively. Then, the similarity between the candidate text and the regenerated text is used as a detection feature, thus eliminating the prompt in the detection process, which allows the detector to focus on the intrinsic characteristics of the generative model. Compared to the baselines, DPIC has achieved an average improvement of 6.76\% and 2.91\% in detecting texts from different domains generated by GPT4 and Claude3, respectively.

Main paper: 
 introduction: 
 Large language models (LLMs) such as PaLM [6] , ChatGPT [29] , LLaMA [38] , and GPT-4 [1] exhibit advanced language capabilities for understanding natural language and solving complex tasks via text generation. The outstanding performance of LLMs has led to the belief that they can be the artificial general intelligence (AGI) of this era [5] . However, if placed in the wrong hands, LLMs such as ChatGPT can undoubtedly serve as a "weapon of mass deception" [35] . For example, the formidable writing capabilities of ChatGPT pose a significant threat to democracy, as they enable the creation of automated bots on online social networks that can manipulate people's political choices during election campaigns [36, 13] . Furthermore, the adoption of ChatGPT by students in educational institutions has led to instances of academic dishonesty, with essays and assignments being generated through its use, as reported by various news sources [23, 31] . Therefore, it is essential and urgent to detect LLM-generated texts.Existing detectors can be grouped into two main categories: supervised classifiers [10, 25, 32, 41, 21] and zero-shot classifiers [12, 24, 37, 42, 4, 15] . Supervised classifiers are usually trained to distinguish between LLM-generated texts and human-created texts. Supervised classifiers achieve strong detection performance in detecting datasets belonging to the same domain as the training set, but usually fail when faced with datasets that are not in the domain of the training set. [32, 3, 39] . As a result, researchers have shifted their perspective to zero-shot classifiers. The zero-shot classifiers directly use pre-trained language models without fine-tuning to gather statistical features and are immune to domain-specific degradation. In general, zero-shot classifiers exhibit better generalizability. However, zero-shot classifiers require model knowledge to extract intrinsic characteristics and perform reliable detection, which is unrealistic for black-box models. Most powerful LLMs, such as ChatGPT, are close-source. As a result, zero-shot classifiers cannot achieve the desired quality of detection when detecting texts generated by closed-source models [2, 29, 1] . To achieve a high quality of detection of black-box models, we would like to extract the deep intrinsic characteristics of the black-box models.In this case, we can only rely on the text inputs and outputs for black-box models. The remaining and significant challenge is how to extract deep intrinsic characteristics of the black-box model generated texts.In our view, the generation process can be seen as a coupled process of prompt and intrinsic characteristics of the generative model. The texts generated by LLMs and created by humans exhibit inconsistent intrinsic characteristics, central to distinguishing the two. The discriminative features should rely more on the intrinsic characteristics to achieve a general detection. Zero-shot detectors extract intrinsic characteristics by relying on internal information of the generative model to identify texts generated by LLMs [4] , achieving desired detection performance on white-box models. However, when confronted with black-box models, these zero-shot detectors do not have access to the model's internal information and must rely on open-source surrogate models, leading to detection performance degradation. Supervised detectors utilize only text for detection and do not require access to the model internals. To eliminate the interference of prompts, the supervised detectors often require large-scale datasets generated with various prompts and then focus on the intrinsic characteristics of the generative model. However, it is impractical to include all prompts in the dataset because of the diversity of prompts. This leads to the question of how to extract the intrinsic characteristics from the black-box model generated texts skillfully?In this paper, we propose to decouple prompt and intrinsic characteristics (DPIC) to extract deep intrinsic characteristics from the black-box model generated texts for detection. Specifically, given a candidate text, we utilize an auxiliary LLM, leveraging the LLM's powerful inductive capabilities, to reconstruct the prompt based on the candidate text. The reconstructed prompt is then used for the auxiliary LLM to obtain the regenerated text. This process aims to make the candidate and regenerated texts align with their prompts, respectively. Then, by comparing the similarity between the candidate text and the regenerated text, we can determine whether the candidate text is generated by LLMs or created by humans. In this case, the candidate text and the regenerated text are compared with aligned prompts, which allows the detector to focus on the deep intrinsic characteristics. With the decoupling process, we extract the deep intrinsic characteristics of black-box model generated texts by decoupling the prompt and the intrinsic characteristics. This provides the detector with richer information about deep intrinsic characteristics and thus achieves higher detection quality and better generalizability. Figure 1 shows the distinctions between our proposed detection method and previous detection methods.The main contributions of our work are as follows:• New insight. We reconsider the generated text detection methods and propose a new perspective to improve generalizability while ensuring detection quality. In our view, prompt and intrinsic characteristics of the generative model in generated text are tightly coupled together, which limits the generalizability of the detector. During the detection process, the detector should rely more on the intrinsic characteristics to achieve better generalizability. • Ingenious approach. we propose to decouple prompt and intrinsic characteristics (DPIC) to extract the deep intrinsic characteristics of the black-box model generated texts. In this process, the candidate text and the regenerated text are aligned with their respective prompts. We then use the similarity between the candidate text and the regenerated text as discriminative features, reducing the impact of prompts on the detection process and allowing the detector to focus more on intrinsic characteristics. • Impressive performance. DPIC achieves 6.76% and 2.91% average improvement in detection performance compared to the baselines in detecting texts from different domains generated by two commercial closed-source models: GPT-4 and Claude3. These findings underscore the efficacy of our proposed method.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: This paper proposes a decoupling-based method for extracting the deep intrinsic characteristics of LLM from the text in black-box scenarios. The detection of GPT-4 and Claude3 improves by 6.76% and 2.91% compared to baselines Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 LLM-generated text detection could increase trust in natural language generation systems and encourage adoption. Given its significance, there has been a growing interest in academia and industry to pursue research on LLM-generated text detection. Current LLM-generated text detection is categorized into supervised [36, 25] and zero-shot methods [12, 24, 37, 42, 4] .Previous research focuses on supervised methods, where classifiers are trained to differentiate between texts generated by LLMs and texts created by humans. For example, OpenAI [30] trained the OpenAI text classifier on a collection of millions of texts. RADAR [16] introduces the idea of adversarial learning to train a detector that can resist paraphrase attacks. However, supervised methods exhibit shortcomings in terms of generalization. Therefore, to obtain more generalizable detection methods, current researchers work on developing zero-shot methods. Existing zero-shot methods primarily rely on statistical features, leveraging pre-trained large language models to gather them. In our view, these methods can be seen as extracting the intrinsic characteristics of LLMs in different ways. For example, some researchs [36, 17] take advantage of the rank or entropy of each word in a text conditioned on the previous context to represent the intrinsic characteristics. The other methods represent the intrinsic characteristics by different features, including average probability, and top-K buckets [12] , likelihood [15] , probability curvature [24] , divergence between multiple completions of a truncated passage (DNA-GPT) [42] , and conditional probability curvature [4] .Due to excessive reliance on training data, the generalizability of supervised methods is questionable [32, 3, 39] . That is, the reliability and robustness of the detection methods may significantly decrease when applied to out-of-domain data. In order to achieve generalizability, existing researchers have worked on developing zero-shot detection methods. Zero-shot detection methods are immune to domain-specific degradation and are on par with supervised classifiers in terms of detection performance. However, from the experimental results, the detection performance of zero-shot detectors for black-box models will be significantly reduced compared to white-box models. Therefore, the primary existing challenge is developing detection methods with high detection performance and high generalizability for black-box models.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations of our work are discussed in Limitations 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
threat models: 
 Given a candidate text x, which may be human-created or generated by a language model, the detector is dedicated to distinguishing between these two sources. In the white-box scenario, the detector has the advantage of accessing the potential source model, including weights and probability distributions. Conversely, in the black-box scenario, only the text input and output of the potential source model are accessible. In this paper, we focus on the black-box scenario.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
motivation: 
 Exploiting the intrinsic characteristics of the text generated by LLMs is an effective way to improve the detection quality and generalizability of the detector. DetectGPT [24] and Fast-DetectGPT [4] can be regarded as adopting log probabilities as the intrinsic characteristic of the LLM. However, the log probabilities cannot be accessible when facing black-box scenarios involving closed-source models such as GPT-4 and Claude3.In fact, the intrinsic characteristics of the model are not only reflected in the logistic probabilities.The generated text itself is a coupling of the prompt and intrinsic characteristics. However, extracting the intrinsic characteristics from the generated text is challenging because the prompts are diverse, covering various topics, and are decoupled with the inherent characteristics. In supervised detection methods, it is very important to extend the dataset as much as possible so that the detector is familiar with the topic of the text to be detected. However, it is not feasible for the model to memorize an infinite number of text topics. We aim to abandon the need for the detector to perform topic recognition. Instead, we hope to provide the topic information of the texts to be detected to the detector each time it performs a detection. The requirement reminds us of the powerful capabilities of LLMs, which can regenerate text through induction and generation. In the comparison perspective between the text to be detected and the regenerated text, the detection bias caused by the text topics can be alleviated.Based on the above analysis, we propose a supervised detection method based on decoupling prompt and intrinsic characteristics (DPIC), as illustrated in Figure 2 , including three modules: decoupling process, feature extraction, and classification.Given a candidate text x and an auxiliary LLM M aux , the decoupling process aims to to obtain a regenerated text x that is consistent with x under the semantic constraints of the prompt p, i.e., x ← arg max x P Maux (x|p), where ← represents the sampling process of LLMs from the predicted distribution. However, since the prompt used to generate x is unknown to the detector in the black-box setting, we first need to reconstruct the prompt p of x. When the log probabilities are hidden, the recently proposed prompt recovery method [27] based on inverting probabilities is inapplicable. We, therefore, shift our focus to another similar task, namely question generation (QG) [8, 9, 20] . QG involves generating natural-sounding questions based on given sentences or paragraphs. The QG task can be defined as finding q, such that:EQUATIONwhere P (q|t), is the conditional log-likelihood of the predicted question sequence q, given the input t. We notice that the LLM performs exceptionally well in the QG task. So we directly utilize M aux to reconstruct the prompt. p ← arg maxEQUATIONwhere p rc represents the prompt template used to query M aux reconstructing the prompt. After obtaining the reconstructed prompt p of x, the next step is to utilize M aux to get x based on p.x ← arg maxEQUATIONwhere p rg regenerates the regeneration template we use to query M aux . The details of p rc and p rg are shown in Appendix B. In the aforementioned process, M aux can be a black box to the user, where the user only inputs the text to be paraphrased or regenerated, and obtains the corresponding regenerated text. The decoupling process can be formally represented as:x ← M aux (x, p rc , p rg ).(4)Through the decoupling process, we get x aligned with x on the reconstructed prompt p. After that, we measure the similarity of x and x, then use it as a discriminative feature. Since the two texts are consistent at the prompt level, the similarity only indicates the intrinsic characteristics contained in x. The hypothesis under our detection method is that the text generated by LLMs and text created by humans exhibit inconsistent intrinsic characteristics. If the candidate text is generated by LLMs, the regenerated text will exhibit higher similarity to the candidate text because the texts generated by LLMs have similar intrinsic characteristics. On the contrary, if the candidate text is created by humans, then the similarity will be lower because the text generated by LLMs and text created by humans exhibit intrinsic characteristics.We employed a Siamese Network to measure the similarity between texts. A Siamese Network is a neural network that simultaneously processes two different input tensors using the same weights, producing comparable output tensors. x and x separately enter two subnets that share structures, parameters, and weights. For each subnet, we employ a language model as an encoder to perform feature extraction on the texts, which can be formulated as below.EQUATION) where E donates the encoder, v ∈ R d . Finally, v x and v x are concatenated and fed into a classifier denoted as f : R 2×d → R 2 . Specifically, we use gte-Qwen1.5-7B-instruct † as the encoder which can encode texts with a maximum of 32K tokens into embeddings of 4096 dimensions, while the classifier consists of three fully connected layers with ReLU function. The dimensions of the intermediate layers in the classifier are 1024 and 512, respectively.We freeze the weights of the Siamese encoder and train the classifier using binary cross-entropy loss. The loss function can be formalized as:EQUATIONwhere y represents the true label of x, and the predicted label of the detector is represented asy pred = f (E (x) ∥E (M aux (x, p rc , p rg ))).5 Experiment 
implementation details: 
 Auxiliary models. We select two auxiliary models for the decoupling process, including ChatGPT ‡ and Vicuna-7b-v1.5 § . For ChatGPT, we directly use the ChatGPT API for querying. If Vicuna is chosen as the auxiliary model, we locally deploy a Vicuna-7b model but treat it as a black-box, solely obtaining its output responses without accessing the model parameters or probability distributions.Training. Since the DPIC method we propose is supervised, it necessitates the selection of a training set. In this paper, we used the open-source Human-ChatGPT Comparison Corpus (HC3) [14] dataset collected by previous researchers as a training set to ensure the reproducibility of our approach. HC3 dataset consists of both human and ChatGPT responses to the same prompts. When the auxiliary model is set to ChatGPT, we select these responses from the HC3 dataset as the texts to be detected. When the intermediary model is Vicuna, we replace the texts generated by ChatGPT in the HC3 dataset with the texts generated by Vicuna as the machine-generated texts in the training set.Testing. We evaluated the performance of the detection methods on texts generated by three current widely used commercial closed-source models, including ChatGPT (gpt-3-5-turbo) ¶ , GPT-4 (gpt-4-0613) || , and Claude3 (claude-3-opus-20240229) ** . We refer to these models as the source models for LLM-generated texts.We use three datasets that are not part of the domains covered in the HC3 dataset to evaluate the generalizability of detection methods fully. The three datasets are Xsum [28] for news articles, WritingPrompts [11] for story writing, PubMedQA [18] for biomedical research question answering, which are consistent with previous work [4] in the field. We use the three different source models to generate texts on the aforementioned datasets. The original texts from the datasets and the texts generated by the models together serve as the data to be detected in the testing process.We also created other datasets for three different domains, and evaluated the detection performance of our methods and baselines on these datasets. The details of these datasets and the results are displayed in Appendix D.Evaluation metric. We measure the detection performance in the area under the receiver operating characteristic (AUROC). AUROC ranges from 0.0 to 1.0, mathematically denoting the probability of a random machine-generated text having a higher predicted probability of being machine-generated than a random human-written text. A higher AUROC value indicates a better detection quality.Baselines. We compared DPIC with existing supervised detectors and zero-shot detectors. For supervised detectors, we compared GPT-2 detectors based on RoBERTa-base/large [22] crafted by OpenAI and RADAR [16] . For zero-shot detectors, we selected DNA-GPT [42] , DetectGPT [24] , and its enhanced variants NPR [37] and Fast-DetectGPT [4] . We also chose classic zero-shot classifiers, including Likelihood (mean log probabilities) [12] , LogRank (average log of ranks in descending order by probabilities) [36] , Entropy (mean token entropy of the predictive distribution) [17] , and LRR (an amalgamation of log probability and log-rank) [37] .Detection effectiveness. The detection performance of DPIC and baselines is shown in Table 1 Generalizability. The detection performance of supervised methods, such as RADAR, exhibits significant performance differences in detecting different datasets. Taking ChatGPT-generated text as an example, the detection AUROC of RADAR on the Xsum dataset is 99.72%. However, the detection AUROC on the PubMed dataset is only 73.72%. This indicates that the detection performance of the supervised methods is susceptible to the domain dataset, which affects the generalizability. Zero-shot detection methods show more balanced detection performances on Xsum and PubMed datasets generated by ChatGPT. This suggests that the zero-shot method is more likely to maintain generalizability by extracting the intrinsic characteristics of LLMs. DPIC extracts the deep intrinsic characteristics of LLMs by decoupling the prompt, allowing the supervised detector to focus more on the intrinsic characteristics. This approach enables DPIC to achieve the desired generalization while maintaining high detection quality.We conducted ablation studies to reveal the impact of different similarity measurements and decoupling processes. Additionally, since the reconstructed prompts may differ from the original prompts, we discuss the impact of these differences on detection performance.Similarity measurement. In this experiment, we explored the effects of different similarity measurements on the detection performance of the Siamese Network. Specifically, we experimented with absolute distance, dot product, and concatenation methods for combining feature vectors. The results are displayed in Table 2 .When combining the feature vectors by absolute distance and dot product, the detection AUROC scores are 89.81% and 93.84% in detecting the Writing dataset generated by ChatGPT. In contrast, when combining the feature vectors by concatenation, the AUROC score is 98.21%. These experimental results show that concatenation results in higher AUROC scores than other methods, which indicates that combining feature vectors by concatenation preserves more of the intrinsic characteristic information. In contrast, the other methods result in the loss of critical intrinsic details. This findings indicates that providing the detector with richer intrinsic characteristic information helps achieve higher detection quality.Prompt reconstruction and limitations Since reconstructed prompts may differ from the original prompts, we discuss the impact of the prompt reconstruction process on detection performance in this part. We tested the semantic similarity of the candidate text and the regenerated text obtained by directly using the original prompt. We also test the semantic similarity of the candidate text and regenerated text obtained by the reconstructed prompt. We used gte-Qwen1.5-7B-instruct † † for feature extraction, and then used cosine similarity to measure the degree of semantic similarity. We use a part of HC3 test dataset generated by ChatGPT, GPT-4, claude3. The results are presented in Figure 4 . From the result, we find that, compared to using the original prompt directly, the regenerated text obtained by the reconstructed prompt has a higher semantic similarity with the original text in texts generated by GPT-4 and Claude3.We believe that this may be due to the following reasons: when users use LLM, the prompt is sometimes not detailed, which leads to the fact that the text generated by LLM may deviate from the original prompt. In other words, in the case where the prompt is not detailed, even if the same prompt is used, the text generated multiple times is prone to larger deviations. However, when LLM is used to regenerate the prompt from a candidate text, the reconstructed prompt is reconstruct from the candidate text, which makes the prompt more semantically consistent with the candidate text. Thereby, when we use the reconstructed prompt for text regeneration, the regenerated text can show more semantic similarity with the candidate text, which is more helpful in reducing the impact of semantics on the detection process.However, directly using LLMs for regenerating for semantic disentanglement can indeed introduce some time delays and costs, which we will discuss in the Limitations Section 6. Decoupling Process In the decoupling process, given a candidate text, we initially reconstruct the prompt based on the candidate text. The reconstructed prompt is then used for the auxiliary LLM to obtain the regenerated text. Then, we use the embeddings of the candidate text and the regenerated text as the discriminative feature to train a classifier. DNA-GPT [42] shares a similar idea. Specifically, given a candidate text, DNA-GPT truncates from the half position of the text length and then completes the text based on the truncated text using an auxiliary LLM. Unlike us, DNA-GPT calculates the divergence between the candidate text and the completion of the truncated text and uses a zero-shot classifier for detection. Another significant difference is that there is no semantic prompt guidance in the completion process. In order to fully illustrate the effectiveness of our method, we explore the impact of supervised learning and prompt guidance on detection performance using DPIC and DNA-GPT as examples.First, we compare the impact of zero-shot and supervised methods on the detection performance using DNA-GPT. Specifically, given a candidate text, DNA-GPT truncates from the half position of the text length and then completes the truncated text using an auxiliary LLM. Then, we use the embeddings of the candidate text and the completion of the truncated text as the discriminative feature to train a Siamese Network to classifier; the detailed experimental setup is consistent with the description in Section 5.1. The results are displayed in Figure 3 , comparing DNA-GPT and DNA-GPT (supervised). The average detection AUROC of DNA-GPT (supervised) is 88.39%, 85.50%, and 91.51% on datasets generated by ChatGPT, GPT-4, and Claude3, respectively. In contrast, the average detection AUROC of DNA-GPT is 64.42%, 55.29%, and 68.49%. DNA-GPT (supervised) shows a substantial improvement compared to DNA-GPT, demonstrating that supervised methods help achieve higher detection AUROC. However, DNA-GPT (supervised) exhibits a significant decrease in detection AUROC on the PubMed dataset compared to the Xsum and Writing datasets, indicating that generalizability is affected when using supervised methods.Afterward, we evaluated the impact of the prompt semantic guidance on the detection performance and generalizability of DNA-GPT (supervised). Specifically, given a candidate text, we reconstruct the prompt based on the candidate text and complete the truncated text based on the prompt using an auxiliary LLM. Then, we measure the difference between the candidate text and the prompt-based completion of the truncated text and use a Siamese Network as the classifier. The results are displayed in Figure 3 , comparing DNA-GPT (supervised) and DNA-GPT (prompt). Compared to DNA-GPT (supervised), DNA-GPT (prompt) achieves a more balanced detection performance across the three datasets. On the PubMed dataset, DNA-GPT (prompt) achieves detection accuracies of 79.28%, 82.18%, and 86.46%, while DNA-GPT (supervised) achieves only 74.93%, 76.66%, and 79.52%, respectively. It demonstrates the importance of prompts in regeneration, as they help reduce semantic bias between the regenerated text and the candidate text, thereby mitigating the impact of the dataset domain on the generalizability of the detector.Finally, we compare the detection performance with the truncated-completion method and the regeneration method on the detection performance, and the results are displayed in DNA-GPT (prompt) and DPIC in Figure 3 . As shown, the regeneration method of DPIC achieves better detection quality and better generalization.Existing research [19, 33] has pointed out that the previous methods experience a reduction in performance in complex scenarios where the text to be detected is subjected to interference. To better understand how DPIC performs in real-world scenarios, we evaluate our detection method under two modification methods.The first one is the proposed paraphrasing attack called DIPPER [19] (or Discourse Paraphrase). DIPPER is an 11B-parameter paraphrase generation model built by fine-tuning T5-XXL. It can paraphrase paragraph-length texts, re-order content, and optionally leverage context such as input prompts. The second interference we consider is a method that is more accessible to the broader audience of large language models and does not require specialized knowledge, namely the backtranslation attack. Back-translation refers to the action of translating a work that has previously been translated into the same language. We employed DeepL Translator ‡ ‡ to translate the given English text into Chinese, followed by a subsequent translation back into English. ‡ ‡ https://www.deepl.com/en/docs-api/ We present the detection performance of DPIC and baselines in detecting Xsum dataset generated by ChatGPT, GPT-4, and Claude3 with interference in Table 3 . RADAR shows the smallest decrease among baselines against DIPPER attacks, especially for text generated by GPT-4, with a decrease of 00.07%, illustrating the robustness of RADAR in incorporating adversarial networks into detection. However, our method still maintains optimal detection performance after both DIPPER and backtranslation attacks. The detection AUROC of DPIC is 100% and 99.72% for detecting the Xsum dataset generated by ChatGPT under DIPPER and back-translation attacks, respectively, indicating that our method is more robust in real-world scenarios. DIPPER and back-translation attacks mainly alter the text semantically. Our method decouples the prompt-guided semantic information and the intrinsic characteristics of the generative model, rendering semantic attacks ineffective and ensuring robustness.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide open access to the data and code, we also provided our model.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitations: 
 We mitigated the risk of benchmark contamination in the testing phase by ensuring different sources for the training and test sets. However, recent studies [7, 40, 34] have indicated that benchmark contamination may be widespread in evaluating large models, which is a limitation not addressed in this paper.Another limitation of our method is the time required in the decoupling process. The regeneration process significantly extends the processing time for each text segment. We selected advanced training-based and zero-shot detection methods and tested memory and time consumption. Our method does require more memory and time, compared to those that only input candidate text for classification, primarily due to the regeneration component involving the LLM. Detailed information regarding this can be found in the Appendix A.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify all the training and test details in Section 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We reconsider the detection methods for LLM-generated text and propose a decoupling-based detection method, DPIC. This method extracts deep intrinsic characteristics of the black-box model generated texts. Through the decoupling process, we provide the detector with a reconstructed prompt covering the topic of the candidate text, allowing the detector to focus on deep intrinsic characteristics, thereby achieving better generalization performance while maintaining detection quality. Experimental results further demonstrate that DPIC significantly improves detection performance by 6.76% and 2.91% when detecting texts generated by GPT-4 and Claude 3, respectively.NeurIPS Paper ChecklistQuestion: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA]Justification: [NA] .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide sufficient information on the computer resources in Limitations 6.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The societal impacts of our work are discussed in Section 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We will release models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We properly credited it. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code and models are well documented. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Justification: We have read the NeurIPS Code of Ethics and our research was conducted with that in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).† https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct ‡ https://platform.openai.com/docs/models/gpt-3-5-turbo § https://huggingface.co/lmsys/Vicuna-7b-v1.5† † https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct 


Decision: This paper presents DPIC, a new framework for detecting LLM-generated text. The approach works by utilising an auxiliary LLM to "reconstruct the prompt of a candidate text and regenerate a text from it, allowing the detector to focus on the intrinsic characteristics of the generative model. The similarity between the candidate and regenerated texts is then used as a detection feature". The authors have included a number of additional experiments during the rebuttal, which have largely addressed the reviewers' concerns. The paper is well written and easy to follow. The idea presented is simple and it works and, in the words of one reviewer, demonstrates "robustness across various datasets, source models, and paraphrasing attacks, suggesting that the proposed method is capable of maintaining performance under different conditions and adversarial scenarios“. The primary issue with this work is that it seems rather incremental in nature, and is rather limited in terms of technical depth and novelty.

Reviews: review: paper_summary: paper_summary: This paper develops an LLM-generated text detection method by ****. The key idea is to regenerate text based on a prompt reconstructed by an auxiliary LLM from the candidate text so that the regenerated text and the original candidate text can be used to extract similarity-based features, which are used for a classifier.
strength_weakness: strength_weakness: - (W1) It is not clear if the proposed method can replace other classification-based methods. Ghostbuster [41] is known to be a strong machine-generated text detection method and should be compared for evaluation (it also offers a benchmark as well). Other recent simple-yet-effective methods could be compared as well. For example,
    - [Ref 1] Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors https://aclanthology.org/2024.eacl-short.25.pdf
    - [Ref 2] Your Large Language Models Are Leaving Fingerprints https://arxiv.org/abs/2405.14057
- (W2) The datasets used for evaluation might be too easy to detect. Unless there’s a special reason, the proposed should also be tested in other benchmarks used for machine-generated text detection research (e.g., the Ghostbuster dataset).
- (W3) The feature extraction is computationally expensive. Two LLM inferences (prompt generation, candidate text regeneration) and then dual-encoder-based feature extraction. It is not clear if the feature extraction offers significantly better benefits compared to more lightweight approaches (e.g., [Ref 1][Ref 2] above)

### Minor comments

For perturbation, in addition to DIPPER, OUTFOX is another option.
    - [Ref 3] OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples https://arxiv.org/abs/2307.11729
questions: questions: Please answer the weaknesses raised in the Weaknesses section.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a novel method named DPIC (Decoupling Prompt and Intrinsic Characteristics) for detecting texts generated by LLMs. The authors posit that generated texts are a coupled product of prompts and intrinsic characteristics, and suggest that decoupling these two elements can enhance detection quality and generalization ability. Specifically, the DPIC method employs an auxiliary LLM to reconstruct the prompt of the candidate text, and then uses this prompt to regenerate the text. By comparing the similarity between the candidate text and the regenerated text, the detector can better focus on the intrinsic characteristics of the generating model. Experimental results show that, compared to baseline methods, DPIC improves the detection of texts generated by GPT-4 and Claude-3 in different domains by an average of 6.76% and 2.91%, respectively.
strength_weakness: strength_weakness: - DPIC involves multiple steps, including prompt reconstruction and text regeneration, which significantly increase computational costs and resource overhead in practical applications. An ablation analysis on the impact of the training sample size would help readers further understand and evaluate the DPIC method.
- DPIC is essentially a supervised method and should be compared fairly with methods based on RoBERTa (not the OpenAI detector trained on GPT-2 but a RoBERTa classifier trained from scratch).
- Supervised methods perform well within their domain but have poor out-of-distribution generalization ability, which lacks discussion in the paper. For example, the performance of DPIC trained on XSum on Writing, and the performance of DPIC trained on ChatGPT re-generated data on Claude-3. This is crucial for highlighting the effectiveness of DPIC in practical applications.
questions: questions: How is the accuracy of prompt reconstruction evaluated? Did the authors verify the consistency between the reconstructed prompts and the original prompts? If there is a significant difference between the reconstructed prompts and the original prompts, how much impact would that have on the detection results?
limitations: limitations: The authors did not discuss the out-of-distribution generalization ability of DPIC.
paper_summary: paper_summary: This paper addresses the problem of detecting texts generated by large language models (LLMs), which is a crucial issue considering the potential misuse of such models. The authors propose a novel method, DPIC (Decoupling Prompt and Intrinsic Characteristics), which aims to extract the intrinsic characteristics of black-box models, as traditional detection methods requiring access to the model's interior are not feasible. DPIC uses an auxiliary LLM to reconstruct the prompt of a candidate text and regenerate a text from it, allowing the detector to focus on the intrinsic characteristics of the generative model. The similarity between the candidate and regenerated texts is then used as a detection feature. Results show that DPIC outperforms baseline methods, achieving an average improvement of 6.76% and 2.91% in detecting texts from different domains generated by GPT4 and Claude3, respectively.
strength_weakness: strength_weakness: There are a few areas that could be improved. Firstly, the comparison with zero-shot methods seems potentially unfair since the proposed method requires training. Including trained detectors as baselines, for instance, using the same LLM used by DPIC to train a classifier directly, might provide a more balanced comparison. Secondly, the paper lacks a detailed analysis of the method's robustness on different lengths of text and languages. This could limit the applicability of the method in real-world scenarios. Lastly, the content of Section 3 appears misplaced as it discusses experimental settings rather than threat models. A reorganization or further clarification in this section would enhance the overall coherence of the paper.
questions: questions: Figure2, can we use a pre-trained Siamese Network to measure the similarity in DPIC? Then, we do not need a training here.

LN121, the section 3 looks strange because it does not provide a threat model but some experimental settings.

Table1, I would expect to see more comparsion with trained detectors given that DPIC is also a trained detector.

Figure3, it seems the major contribution is from the supervised training, while the contribution from the prompt is much smaller. Why?

Figure3, the improvement of DNA-GPT(prompt) compared to DNA-GPT(supervised) mainly happens on PubMed. Is it caused by the QA style of the dataset?
limitations: limitations: Yes.
paper_summary: paper_summary: In this paper, the authors present DPIC, a novel model for detecting LLM-generated text, which centers on having an auxiliary LLM reverse-generate prompt on candidate text, and then letting the LLM re-generate the answers to the prompt and classify them based on the similarity between the candidate and the re-generated text. The authors experimented it with competitors on multiple datasets and claimed that they achieved better performance.
strength_weakness: strength_weakness: - Lack of detail in evaluation metrics.

- The testing model is underrepresented.

- Details of the model's consumption of computational resources are missing.
questions: questions: 1. In the manuscript, I did not find out any details about the AUROC evaluation metrics used, such as its formula, or the evaluation steps. I understand that this is a commonly used evaluation metric, but an academic paper should be complete and detailed. The authors only briefly indicated its mathematical meaning, which I think is not sufficient.

2. For the testing part of the models, the authors used three representative commercial closed-source models: gpt-3.5, gpt-4, and claude3 to generate text for specific prompts for model testing. While it can be argued that closed-source models are more difficult to access than open-source models to infer that it is more challenging to recognize text generated by closed-source models, this inference is from the perspective of detection model builder, and applied to a real-world scenario, even if the candidate text is open-source LLM-generated, will the detection model have easy access to the internal parameters used to generate this text? I don't think the authors should make too much of a distinction between closed-source and commercial models for the task of detecting LLM-generated text. So I suggest the authors to add some open-source models to the test models, which will make the conclusions more convincing, and I think the readers are also interested in the differences between texts generated by commercial closed-source models and open-source models. I understand the infeasibility of conducting new experiments in a short period of time, so I would suggest that the authors mention this in the limitations section.

3. The authors mention in question 8 of the checklist that they provide enough details about computational resource consumption in Section 5.1, but I didn't find them in the corresponding section. I think that for this kind of detection task, comparing the running efficiency of the proposed model and the competitors is also an important part. In particular, I'd be interested to know the resource consumption comparison between DPIC models and those that only input candidate text for classification.
limitations: limitations: I think the statement made by the author in the limitations section (Appendix A) is not sufficient. It seems that the author is only analyzing the results of the experiment in the prompt reconstruction part by mentioning the possible scenarios that may occur when using the model. I would suggest that the authors flesh out this section from the perspective of the model's deployment in a real-world environment and its impact on society. Additionally, I suggest that the authors include an analysis with some of the current techniques that help AI-generated text escape detection, to test whether the model will be defeated by these models.

Comments: title: Thank you for your response
Official Comment by Reviewer XAPE
Detailed Response by Author
Thanks for the response.
Official Comment by Authors
Thank you for your response
Thank you for your response
Official Comment by Authors
Thanks for the clarification
comment: We are delighted that our response addresses the reviewer's concerns. Again, we thank the reviewer for your valuable and positive feedback and comments.
Thanks for the responses. I appreciate the author's detailed explanation, which addresses my concerns. I have revised my score accordingly.
Dear reviewer,

your concern is whether the regenerated text **consistently** maintains a **high semantic similarity** with the original. We divided it into three aspects to address your concern:

1. **Semantic Similarity**
    
    To evaluate the **semantic similarity** between the source text and the machine regenerated text, we measured the **cosine similarity** of their corresponding embeddings. Take the XSum dataset generated by ChatGPT (The First Column) as an example, the average cosine similarity is 0.6201. 

---    

2. **High Semantic Similarity**
    
    There is an issue: given a value of cosine similarity, such as the previously mentioned 0.6201, does it indicate a high degree of similarity between the source text and the machine-generated text? To address this issue, we introduced a baseline similarity **for reference**, which better reflects the degree of semantic similarity reflected by a specific cosine similarity value.
    
    The baseline similarity for reference is defined as the cosine similarity between human and AI texts for the same prompt. Since the texts correspond to the same prompt, their semantic similarity can be regarded as high. For example, in the XSum dataset generated by ChatGPT, the baseline similarity is 0.5042, which means that 0.5042 of cosine similarity already represents a high level of semantic similarity. The average similarity between the source text and the machine regenerated text is 0.6201, higher than 0.5042, meaning that they have **high** semantic similarity.
    
---    

3. **Consistently Maintains High Semantic Similarity**
    
    To evaluate whether the similarity between source text and the machine regenerated text **consistently** maintains a high semantic similarity. We measured the Standard Deviation, First Quartile, Second Quartile, and Third Quartile to better understand the distribution.
    
    - Standard Deviation: A measure of how dispersed the data is in relation to the mean. In the XSum dataset generated by ChatGPT, the standard deviation is 0.1357, which is relatively small compared to the average of 0.6201, indicating **minimal fluctuation**.
    
    Therefore, we can conclude that the regenerated text **consistently** maintains a high semantic similarity with the original. 
    
---    

We greatly value every opportunity to discuss with you, **so we have conducted detailed experiments to address your concerns as thoroughly as possible.** I hope this explanation fully addresses your concern.
Thanks for your response and experiment. However, I am a little confused (can't fully understand) about the setting in the response  experiment. I mean, maybe directly evaluating the similarity between the source text and the machine-generated text is enough to solve my concern?
We sincerely appreciate your recognition of our paper's motivation. Below, I will answer your question on 'whether the regenerated text consistently maintains a high semantic similarity with the original.'

We used the gte-Qwen1.5-7B-instruct model to extract embeddings from the original text and the regenerated text, and then measured their semantic similarity using cosine similarity. We evaluated 9 datasets and obtained the values of cosine similarity for Average, Standard Deviation, First Quartile, Second Quartile, and Third Quartile. 

To better understand the degree of semantic similarity reflected by the value of cosine similarity, we introduce the baseline similarity for reference, which is defined as the cosine similarity between embeddings of human and AI-generated texts for the same prompt. Since the texts correspond to the same prompt, their semantic similarity can be regarded as at a high level.

As shown in the table below, the average similarity exceeds the baseline similarity across all 9 datasets, which indicates that the original and regenerated texts achieve high semantic similarity. Additionally, the standard deviation values range from 0.11 to 0.15, which is relatively small compared to the average semantic similarity. This indicates that most of the texts maintain a high level of semantic similarity with the regenerated text.

I hope this explanation addresses your concern. If you have any further concerns or questions about our work, we are happy to discuss them with you. We will also add this part to strengthen the manuscript. Thank you again for the time and expertise you have invested in these reviews.

|  | ChatGPT |  |  | GPT4 |  |  | Claude3 |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | XSum | Writing | PubMed | XSum | Writing | PubMed | XSum | Writing | PubMed |
| Baseline Similarity | 0.5042 | 0.3739 | 0.5207 | 0.5065 | 0.3760 | 0.5225 | 0.2816 | 0.2538 | 0.5310 |
| Average Similarity | **0.6201** | **0.4462** | **0.6672** | **0.5975** | **0.4367** | **0.6756** | **0.5928** |  **0.4466** |  **0.6877** |
| Standard Deviation | 0.1357 | 0.1414 | 0.1367 | 0.1349 | 0.1336 | 0.1184 | 0.1358 | 0.1414 |  0.1333 |
| First Quartile | 0.5320 | 0.3581 | 0.5963 | 0.5147 | 0.3441 | 0.6036 | 0.5047 | 0.3540 | 0.6159 |
| Second Quartile | 0.6351 | 0.4468 | 0.6799 | 0.6023 | 0.4377 | 0.6881 | 0.5998 | 0.4492 |  0.6907 |
| Third Quartile | 0.7192 | 0.5495 | 0.7644 | 0.6906 | 0.5340 | 0.7621 | 0.6992 | 0.5329 | 0.7927 |
We are delighted that our response addresses the reviewer's major concerns. Again, we thank the reviewer for your valuable and positive feedback and comments.
We are delighted to learn that the reviewer is satisfied with our response! We have read the works you mentioned, and it is crucial to consider whether the benchmark might be contaminated during the evaluation process. We will add the assessment of the risk of benchmark contamination into the limitations and conclusion sections to strengthen our manuscript, as the reviewer suggested. Thank you again for the time and expertise you have invested in these reviews.
Thank you again for the time and expertise you have invested in these reviews. We appreciate the opportunity to address the concerns and questions raised. 

- **(W1) & (W2):** Unless the Ghostbuster benchmark is a subset of the dataset, the Ghostbuster benchmark is considered another dataset and we cannot tell which one is better. The statement "offering a more rigorous evaluation standard." is not verified in the paper. Please clarify why the author(s) consider it.
    
    **Response:** Dear reviewer, we sincerely apologize for comparing the benchmark merely through its domain, which is not full-considered. Since the Ghostbuster benchmark is not a subset of the our benchmark,  it is meaningful to evaluate detectors on the Ghostbuster benchmark as well.  Therefore, we will add the results on the Ghostbuster benchmark in the final version, and thanks again for your valuable suggestions.
    
    And the statement 'offering a more rigorous evaluation standard' refers to the statement that the benchmark used in our paper includes three large language models. Since cross-model detection is a challenging problem in generated text detection[1,2],  we considered our benchmark is more rigorous in this aspect. From a strict perspective, we should not compare benchmark merely depending on the adopted LLMs. Therefore, we will add the results on the Ghostbuster benchmark in the final version, and not compare the benchmarks.
    
---

- **(W3):** This concern is not addressed. The feature extraction is effective with a certain cost.
    
    **Response:** Dear Reviewer, since we presented comparisons with the methods you mentioned in Weaknesses 1 & 2 of our rebuttal, we did not directly respond to the statement in Weakness 3: 'It is not clear if the feature extraction offers significantly better benefits compared to more lightweight approaches (e.g., [Ref 1][Ref 2] above).' We apologize for any confusion this may have caused and appreciate the opportunity to clarify further.
    
    As shown in Table 1 of the attached PDF, we have compared our results with other methods, including the two lightweight methods you mentioned: Fingerprints [3], and Smaller Models [4]. To facilitate your review, we present the results below. Our method indeed shows a significant improvement over the lightweight methods in terms of average detection AUROC on both our benchmark and Ghostbuster benchmark. 
    
    I hope this explanation addresses your concern. If you have any further concerns or questions about our work, we are happy to discuss them with you. We will also add this part to strengthen the manuscript. Thank you again for the time and expertise you have invested in these reviews.
    
    |  | ChatGPT |  |  |  | GPT4 |  |  |  | Claude3 |  |  |  | Ghostbuster |  |  |  |
    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
    |  | XSum | Writing | PubMed | Avg. | XSum | Writing | PubMed | Avg. | XSum | Writing | PubMed | Avg. | News | Creative Writing | Student Essays | Avg. |
    | Fingerprints | 0.8815 | 0.8073 | 0.6816 | 0.7901 | 0.8124 | 0.7896 | 0.7133 | 0.7718 | 0.8849 | 0.9033 | 0.8692 | 0.8858 | 0.8108 | 0.8373 | 0.7846 | 0.8109 |
    | Smaller Models | 0.9835 | 0.9713 | 0.8843 | 0.9464 | 0.8818 | 0.9098 | 0.8234 | 0.8717 | 0.9798 | 0.9594 | 0.8868 | 0.9420 | **0.9983** | 0.8957 | 0.9673 | 0.9537 |
    | DPIC | **1.0000** | **0.9821** | **0.9082** | **0.9634** | **0.9996** | **0.9768** | **0.9438** | **0.9734** | **1.0000** | **0.9950** | **0.9686** | **0.9879** | 0.9950 | **0.9978** | **0.9774** | **0.9900** |
    
## **References**

[1] Bao G, Zhao Y, Teng Z, et al. Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature[C]//The Twelfth International Conference on Learning Representations.

[2] Yang X, Cheng W, Wu Y, et al. DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text[C]//The Twelfth International Conference on Learning Representations. 

[3] McGovern H, Stureborg R, Suhara Y, et al. Your Large Language Models Are Leaving Fingerprints[J]. arXiv preprint arXiv:2405.14057, 2024.

[4] Mireshghallah N, Mattern J, Gao S, et al. Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors[C]//Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers). 2024: 278-293.
Thanks for the clarification, which addresses my major concerns.
Thank you for sharing the new experiment results and the responses. My concerns have not been addressed and I keep my initial evaluation as 6: Weak Accept. 


>> (W1) & (W2)
>
> Actually, our benchmark used in our original paper, already covers two domains from the Ghostbuster benchmark, namely Xsum for news articles and Writing for creative story writing. In addition, our benchmark also includes multiple prompts and three generative large language models, offering a more rigorous evaluation standard. Therefore, to better compare the performance of these newly added methods with DPIC, we further tested these methods using the benchmark employed in our paper.

Unless the Ghostbuster benchmark is a subset of the dataset, the Ghostbuster benchmark is considered another dataset and we cannot tell which one is better. The statement "offering a more rigorous evaluation standard." is not verified in the paper. Please clarify why the author(s) consider it.


>> (W3)

This concern is not addressed. The feature extraction is effective with a certain cost.
Thank you for the clarification, I am satisfied with it. In addition I would like the authors to include, in the limitations or conclusion section, an assessment of the risk of benchmark contamination in the evaluation process, which would make the analysis of the experimental results more robust, as can be seen in the following works.

[1] Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation, ACL 2024.

[2] Benchmark Data Contamination of Large Language Models: A Survey, arXiv 2024.

[3] NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark, EMNLP Findings 2023.
Thanks for the clarification, which addresses some of my key concerns. I have revised my score accordingly.

Honestly, the motivation for this paper is excellent. However, my main concern is whether the regenerated text consistently maintains a high semantic similarity with the original. I'm not entirely convinced yet. Providing more support and analysis on this point could strengthen the manuscript.