DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection
Authors: Xiao Yu
Yuang Qi
Kejiang Chen
Guoqiang Chen
Xi Yang
PENGYUAN ZHU
Xiuwei Shang
Weiming Zhang
Nenghai Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Intrinsic Characteristics
Detection
Primary area: safety_in_machine_learning
TL;DR: This paper proposes a decoupling-based method for extracting the deep intrinsic characteristics of LLM from the text in black-box scenarios. The detection of GPT4 and Claude3 improves by 6.76% and 2.91% compared to baselines.

Abstract: Large language models (LLMs) have the potential to generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Consequently, detecting whether a text is generated by LLMs has become increasingly important. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. However, since we do not have access to the interior of the black-box model, we must resort to surrogate models, which impacts detection quality. In order to achieve high-quality detection of black-box models, we would like to extract deep intrinsic characteristics of the black-box model generated texts. We view the generation process as a coupled process of prompt and intrinsic characteristics of the generative model. Based on this insight, we propose to decouple prompt and intrinsic characteristics (DPIC) for LLM-generated text detection method. Specifically, given a candidate text, DPIC employs an auxiliary LLM to reconstruct the prompt corresponding to the candidate text, then uses the prompt to regenerate text by the auxiliary LLM, which makes the candidate text and the regenerated text align with their prompts, respectively. Then, the similarity between the candidate text and the regenerated text is used as a detection feature, thus eliminating the prompt in the detection process, which allows the detector to focus on the intrinsic characteristics of the generative model. Compared to the baselines, DPIC has achieved an average improvement of 6.76\% and 2.91\% in detecting texts from different domains generated by GPT4 and Claude3, respectively.

DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection
Xiao Yu Yu!.2t, Yuang Qi! Kejiang Chen 12- Guoqiang Chen' Xi Yang', Pengyuan Zhu Xiuwei Shang', Weiming Zhang', Nenghai Yu?
'University of Science and Technology China, China
Key Laboratory of Cyberspace Security, Ministry of Education, China
Hefei figh-dimensional Data Technology, China
yuxiao12176mai1 ustc. edu. en, {chenkj, zhangwm, ynh}@ustc., edu, cn
Abstract
Large language models (LLMs) have the potential to generate texts that pose risks misuse, such as plagiarism, planting fake reviews on e-commerce platforms, creating inflammatory false tweets. Consequently, detecting whether text is generated by LLMs has become increasingly important. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. However, since we do not have access to the interior the black-box model, must resort surrogate models, which impacts detec- tion quality. In order to achieve high-quality detection of black-box models, we would like to extract deep intrinsic characteristics of the black-box model generated texts. We view the generation process as a coupled process of prompt and intrinsic characteristics of the generative model. Based this insight, we propose decou- ple prompt and intrinsic characteristics (DPIC) for LLM-generated text detection method. Specifically, given a candidate text, DPIC employs an auxiliary LLM reconstruct the prompt corresponding to the candidate text, then uses the prompt regenerate text by the auxiliary LLM, which makes the candidate text and the regenerated text align with their prompts, respectively. Then, the similarity be- tween the candidate text and the regenerated text is used as detection feature, thus eliminating the prompt in the detection process, which allows the detector to focus on the intrinsic characteristics of the generative model. Compared to the baselines, DPIC has achieved an average improvement of 6.76% and 2.91% % in detecting texts from different domains generated by GPT-4 and Claude3, respectively.
Introduction
Large language models (LLMSs) such as PaL.M (6, ChatGPT (2), LLaMA B8, and GPT-4 exhibit advanced language capabilities for understanding natural language and solving complex tasks via text generation. The outstanding performance of LLMs has led to the belief that they can be the artificial general intelligence (AGI) of this era (5). However, if placed the wrong hands, LLMs such ChatGPT can undoubtedly serve as "weapon of mass deception" formidable writing capabilities of ChatGPT pose a significant threat to democracy, as they enable the creation of automated bots online social networks that manipulate people's political choices during election campaigns [36] [3. Furthermore, the adoption of ChatGPT by students in educational institutions has led to instances of academic dishonesty, with essays and assignments being generated
335 For example, the
Corresponding author. "quaal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous Detection Methods
Prompt
Input
Coupling
TXXT
Candidate
Text
Extraction
Feature
Classifier
DPIC
Decoupling Process
Human/ /
LLMs
Candidate
Re-generate
Prompt
Generate
Text
Text
Candidate Text
Similarity Measure
Classifier
Figure 1: The distinctions between DPIC and previous detection methods. DPIC extracts the deep intrinsic characteristics of the black-box model generated texts by decoupling the prompt and the intrinsic characteristics of the generative model.
through its use, reported by various sources 23 31 Therefore, it is essential and urgent to detect LLM-generated texts.
Existing detectors can be grouped into two main categories: supervised classifiers 10
21 and zero-shot classifiers [12 24 37
42 4 [3]. Supervised classifiers are usually trained to distinguish between ALM-generated texts and human-createds texts. Supervised classifiers achieve strong detection performance in detecting datasets belonging to the same domain as the training set, but usually fail when faced with datasets that are not in the domain of the training set. 332 39] As a result, researchers have shifted their perspective to zero-shot classifiers. The zero-shot classifiers directly use pre-trained language models without fine-tuning to gather statistical features and are immune to comain-specific degradation. In general, zero-shot classifiers exhibit better generalizability. However, zero-shot classifiers require model knowledge extract intrinsic characteristies and perform reliable detection, which is unrealistic for black-box models. Most powerful LLMs, such as CharGPT, close-source. As result, zero-shot classifiers cannot achieve the desired quality of detection when detecting texts generated by closed-source models 12 29 M. To achieve a high quality of detection of black-box models, we would like extract the deep intrinsic characteristics of the black-bou models. In this case, we can only rely on the text inputs and outputs for black-box models. The remaining and significant challenge is how extract deep intrinsic characteristics of the black-box model generated texts.
In view, the generation process can be seen coupled process of prompt and intrinsic characteristics of the generative model. The texts generated by LLMs and created by humans exhibit inconsistent intrinsic characteristics, central distinguishing the two. The discriminative features should rely more on the intrinsic characteristics to achieve a general detection. Zero-shot detectors extract intrinsic characteristics by relying on internal information of the generative model to identify texts generated by LLMs 44, achieving desired detection performance on white-box models. However, when confronted with black-box models, these zero-shot detectors do not have access to the model's internal information and must rely on open-source surrogate models, leading to detection performance degradation. Supervised detectors utilize only text for detection and do not require access to the model internals. To eliminate the interference of prompts, the supervised detectors often require large-scale datasets generated with various prompts and then focus on the intrinsic characteristics of the generative model. However, it is impractical to include all prompts in the dataset because of the diversity of prompts. This leads to the question of how to extruct the intrinsic characteristics from the black-box model generated texts skillfully?
In this paper, we propose decouple prompt and intrinsic characteristics (DPIC) to extract deep intrinsic characteristics from the black-box model generated texts for detection. Specifically, given a candidate text, we utilize an auxiliary LLM, leveraging the LLM's powerful inductive capabilities, to reconstruct the prompt based on the candidate text. The reconstructed prompt is then used for the auxiliary LLM obtain the regenerated text. This process aims make the candidate and regenerated texts align with their prompts. respectively. Then, by comparing the similarity between the candidate text and the regenerated text, we can determine whether the candidate text is generated by L.MSs or created by humans. In this case, the candidate text and the regenerated text are compared with aligned prompts, which allows the detector to focus on the deep intrinsic characteristics. With the decoupling process, we extract the deep intrinsic characteristics of black-box model generated

---

texts by decoupling the prompt and the intrinsic characteristics. This provides the detector with richer information about deep intrinsic characteristics and thus achieves higher detection quality and better generalizability. Figure shows the distinctions between our proposed detection method and previous detection methods.
The main contributions of our work are as follows:
New insight. We reconsider the generated text detection methods and propose new perspective to improve generalizability while ensuring detection quality. In our view, prompt and intrinsic characteristics of the generative model in generated text are tightly coupled together, which limits the generalizability of the detector. During the detection process, the detector should rely more on the intrinsic characteristics to achieve better generalizability.
Ingenious approach. we propose to decouple prompt and intrinsic characteristics (DPIC) to extract the deep intrinsic characteristics the black-box model generated texts. In this process, the candidate text and the regenerated text are aligned with their respective prompts. We then use the similarity between the candidate text and the regenerated text discriminative features, reducing the impact of prompts on the detection process and allowing the detector to focus more on intrinsic characteristics.
Impressive performance. DPIC achieves 6.76% and 2.91% average improvement in detection performance compared the baselines in detecting texts from different domains generated by two commercial closed-source models: GPT-4 and Claude3, These findings underscore the efficacy of our proposed method.
2
Related work
LLM--geerreed text detection could increase trust in natural language generation systems and encourage adoption. Given its significance, there has been growing interest in academia and industry to pursue research on LLM-generated text detection. Current LLM-ggnnrraed text detection categorized into supervised 36 25
and zero-shot methods 22
Previous research focuses on supervised methods, where classifiers are trained to differentiate between texts generated by LLMs and texts created by humans. For example, OpenAl [30] trained the OpenAl text classifier on a collection millions of texts. RADAR [16] introduces the idea of adversarial learning train detector that resist paraphrase attacks. However, supervised methods exhibit shortcomings in terms of generalization. Therefore, to obtaín more generalizable detection methods, current researchers work developing zero-shot methods. Existing zero-shot methods primarily rely on statistical features, leveraging pre-trained large language models to gather them. our view, these methods can be seen extracting the intrinsic characteristics of LLMs in different ways. For example, some researchs [36]] take advantage of the rank or entropy of each word in a text conditioned the previous context represent the intrinsic characteristies. The other methods represent the intrinsic characteristics by different features, including average probability, and top-K buckets [12, likelihood (15), probability curvature (24), divergence between multiple completions of a truncated passage (DNA-GPT) (42,, and conditional probability curvature [4]
Due to excessive reliance training data, the generalizability of supervised methods is question- able [ 39). That is, the reliability and robustness of the detection methods may significantly decrease when applied to out-of-domain data. In order achieve generalizability, existing researchers have worked on developing zero-shot detection methods. Zero-shot detection methods are immune to domain-specific degradation and are on par with supervised classifiers terms of detection perfor- mance. However, from the experimental results, the detection performance of zero-shot detectors for black-box models will be significantly reduced compared white-box models. Therefore, the primary existing challenge is developing detection methods with high detection performance and high generalizability for black-bor models.
Threat models
Given candidate text which may be human-created generated by a language model, the detector dedicated to distinguishing between these two sources. the white-box scenario, the detector has the ad advantage of accessing the potential source model, including weights and probability distributions.
3

---

Decoupling
Candidate Text
Avadiarry LLM
Question
Ausllary LLM
Re-generated Text
Large language models
large language models
Larce language models built using deep learnngg architectures specifically transformers
leverage a deep learning architecture called transformer...
loverage?...
Siamese Network
Embeddings
Extraction
Text
TX1
Encoder
Concat
Human created
D.ccouping
Shared-Wee Shared-Weights
Classifier
LLM-generated
Re-generated Text
Encoder
Update
① Classification
Frozen
Figure 2: An overview of DPIC. Given a candidate text, we utilize an auxiliary LLM to reconstruct the prompt based the candidate text. The reconstructed prompt is then used for the auxiliary LLM to obtain the regenerated text. This process aims to make the candidate and regenerated texts align with their prompts, respectively. Then, by comparing the similarity between the candidate text and the regenerated text, we can determine whether the candidate text is generated by LLMs or created by humans.
Conversely, in the black-box scenario, only the text input and output of the potential source model are accessible. In this paper, we focus on the black-box scenario.
4 Method
4.1 Motivation
Exploiting the intrinsic characteristics of the text generated by LLMs is an effective way to improve the detection quality and generalizability of the detector. DetectGPT [24] and Fast-DetectGP1 [4] can be regarded as adopting log probabilities as the intrinsic characteristic the LLM. However, the log probabilities cannot be accessible when facing black-box scenarios involving closed-source models such as GPT-4 and Claude3.
In fact, the intrinsic characteristics of the model are not only reflected in the logistic probabilities. The generated text itself is a coupling of the prompt and intrinsic characteristics. However, extracting the intrinsic characteristics from the generated text is challenging because the prompts are diverse, covering various topics, and are decoupled with the inherent characteristics. In supervised detection methods, it is very important to extend the dataset as much possible so that the detector is familiar with the topic of the text to be detected. However, is not feasible for the model to memorize an infinite number of text topics. We aim abandon the need for the detector to perform topic recognition. Instead, we hope to provide the topic information of the texts to be detected to the detector each time it performs detection. The requirement reminds of the powerful capabilities of L.Mss, which can regenerate text through induction and generation. In the comparison perspective between the text be detected and the regenerated text, the detection bias caused by the text topics can be alleviated.
Based the above analysis, propose supervised detection method based on decoupling prompt and intrinsic characteristics (DPIC), as illustrated in Figure including three modules: decoupling process, feature extraction, and classification.
4.2 Decoupling process
Given a candidate text and an auxiliary LLM Maux, the decoupling process aims to to obtain regenerated text x that consistent with x under the semantic constraints of the prompt p. i.e.. & + arg max PM (x|p), where <- represents the sampling process of LLMs from the predicted
distribution. However, since the prompt used to generate x is unknown to the detector in the black-box setting, we first need to reconstruct the prompt p of When the log probabilities are hidden, the
recently proposed prompt recovery method (27) based on inverting probabilities! is inapplicable. We,
therefore, shift our focus to another similar task, namely question generation (QG) 9 20). QG involves generating
questions based given sentences paragraphs. The QG task

---

can be defined as finding q. such that:
q arg max P(q
(I)
9
where P(q is the conditional log-likelihood of the predicted question sequence q, given the input t. We notice that the LLM performs exceptionally well in the QG task. So we directly utilize Maux to reconstruct the prompt.
p + arg max PM (PralIx),
(2)
where Pre represents the prompt template used to query Maxx reconstructing the prompt. After obtaining the reconstructed prompt of x, the next step is utilize Maax to get x based on p.
f arg max PM (pl(Prg).
where Prg regenerates the regeneration template use to query Maux The details of Pro and Pry shown in Appendix B In the aforementioned process, Maux can be a black box to the user, where the user only inputs the text to be paraphrased regenerated, and obtains the corresponding regenerated text. The decoupling process can be formally represented as:
& + Maux (x, Pre: Prg)-
4.3 Feature extraction and classification
Through the decoupling process, we get & aligned with x the reconstructed prompt p. After that, we measure the similarity of x and x, then use it as a discriminative feature. Since the two texts are consistent the prompt level, the similarity only indicates the intrinsic characteristics contained in our detection method is that the text generated by LLMs and text created
x. The hypothesis by humans exhibit inconsistent intrinsic characteristics. If the candidate text is generated by LLMs, the regenerated text will exhibit higher similarity to the candidate text because the texts generated by LLMs have similar intrinsic characteristics. On the contrary, if the candidate text is created by humans, then the similarity will be lower because the text generated by LLMs and text created by humans exhibit intrinsic characteristics.
We employed a Siamese Network to measure the similarity between texts. A Siamese Network neural network that simultaneously processes two different input tensors using the same weights, producing comparable output tensors. x and & separately enter two subnets that share structures, parameters, and weights. For each subnet, employ language model as an encoder to perform feature extraction on the texts, which can be formulated as below.
= E(x), V E(x).
(5)
where & donates the encoder, v E Rd. Fimally, Vx and vs are concatenated and fed into a classifier denoted as f 22xd - R Specifically, which can encode texts with a maximum 32K tokens into embeddings of 4096 dimensions, while the classifier consists of three fully connected layers with ReLU function. The dimensions of the intermediate layers in the classifier are 1024 and 512, respectively.
gte-Qwen1. 1.m....5-BB. 5-----instructttt
the encoder
We freeze the weights of the Siamese encoder and train the classifier using binary cross-entropy loss. The loss function can be formalized as:
C=
1
log + (1- - ()) log(1
Mpredd) s
(6)
where represents the true label of x. and the predicted label of the detector is represented as Upred / (E (x) ||ε (Maux (x, Prey Prg))).
5
Experiment
5.1
Implementation details
Auxiliary models. We select two auxiliary models for the decoupling process, including ChatGPT and Vicuna-7b-vl.5 For ChatGPT, we directly use the ChatGPT API for querying. If Vicuna is
5

---

chosen as the auxiliary model, we locally deploy Ficuna-7b model but treat it as black-box, solely obtaining its oulput responses without accessing the model parameters or probability distributions.
Training. Since the DPIC method we propose is supervised, it necessitates the selection of a training set. In this paper, we used the open-source Human-ChhtGPP Comparison Corpus (HC3) 14 dataset collected by previous researchers as training set to ensure the reproducibility of our approach. HC3 dataset consists of both human and ChatGPT responses to the same prompts. When the auxiliary model is set to ChatGPT, we select these responses from the HC3 dataset as the texts to be detected. When the intermediary model is Vicuna, we replace the texts generated by ChatGPT in the HC3 dataset with the texts generated by Vicuna as the machine- generated texts in the training set.
machine-generated
Testing. We evaluated the performance of the detection methods texts generated by three current widely used commercial closed-source models, including ChatGPT (gpt-3-5-turbo) 0613)] and Claude3 -lluude---ppus222200222 for M-geeerrrte texts.
GPT-4 (gpt-4- w- We refer these models the source models
We use three datasets that are not part of the domains covered in the HC3 dataset to evaluate the generalizability of detection methods fully. The three datasets are Xsum (28) for news articles,
WritingPrompis [I] for story writing, PubMedQA [88] for biomedical research question answering, which are consistent with previous work [4] in the field. We use the three different source models to generate texts the aforementioned datasets. The original texts from the datasets and the texts generated by the models together serve as the data to be detected in the testing process.
We also created other datasets for three different domains, and evaluated the detection performance of our methods and baselines on these datasets. The details of these datasets and the results are displayed in Appendix D
Evaluation metric. We measure the detection performance in the area under the receiver operating characteristic (AUROC). AUROC ranges from 0.0 to 1.0, mathematically denoting the probability of a random machine- generrate text having higher predicted probability of being machine- generrted than random human numan-written written text. A higher AUROC value indicates better detection quality.
Baselines. We compared DPIC with existing supervised detectors and zero-shot detectors. For supervised detectors, we compared GPT-2 detectors based on RoBERTaa-base/lrge (22) crafted by OpenAl and RADAR [16]. For zero-shot detectors, selected DNA-GPT 42. DetectGPT [24). and
its enhanced variants NPR 337 and Fast- DesstDetettGPTT [4) We also chose classic zero-shot classifiers, including Likelihood (mean log probabilities) [12.. LogRank (average log of ranks in descending
order by probabilities) [36]. Entropy (mean token entropy of the predictive distribution) and
LRR (an amalgamation of log probability and log-rank)
5.2 Performance
Detection effectiveness. The detection performance of DPIC and baselines is shown in Table When using ChatGPT as the auxiliary LLM, our method achieves an average AUROC of 96.34%. 97.34% and 98.78%, respectively, in detecting ChatGPT, GPT-4, and claude3. By comparison, Fast- DetectGPT, which achieves the highest AUROC among open source baselines, has lower AUROC of 96.15% 90.58%, and 95.87% respectively. It can be seen that our method outperforms the baselines in terms of average detection quality, particularly in detecting advanced commercial closed-source models like GPT-4 and Claude3, with a significant advantage. Meanwhile, when using Vicuna-7b as the auxiliary model, our method achieves an average AUROC of 95.58%. 96.74% and 98. 98.75% in detecting ChatGPT, GPT-4, and claude3, respectively. Although our training set only includes LLM- generated text from either ChatGPT or Vicuna-7b, DPIC achieves high detection AUROC for text originating from three different source models when using the two auxiliary models. This indicates that our method does not require prior knowledge of the text's source model, as the differences in features between LLM- L.M-ggeerrted and human-created text are greater than the differences among texts generated by various models. Besides, the effectiveness of the open-source auxiliary model demonstrates that DPIC can conduct detection at a relatively low cost, indicating the practical efficacy of our detection method in real-world scenarios.
" hhtps://pltfoomoooooooo wttp::/wwww......eeeooee////eaa.
anthropic www.aaap....oomeeooooooooteeeeee

---

Table 1: The detection performance (AUROC) of baselines and DPIC on three datasets generated by ChatCPT, 4PT-4, and Claude3.
CharGPT
GPT-4
Claude3
Methods
XSum
Writing
Pub
Avg.
XSum
Writing
PubMed
Avg.
XSum
Writing
PuhMed
Avg.
RoBH-TTa-bsee
0.9150
0,7084
0.6188
0,7474
0.6778
0.5068
0.5309
0,5718
0.8944
0.8036
0.3647
0.6876
RoBERTa-larg
0.8507
0.5480
0.6731
0.6906
0.6879
0.3822
0.6067
0.5589
0.9027
0,7128
0.3579
0.6578
RADAR
0.9972
0,9593
0.7372
0.9931
0.8029
0.8851
0.9952
0.8029
0.9139
Likelihood
0.9577
0.9739
0.8776
0.9364
0.7982
0.8553
0.8100
0.8212
0.9760
0.9744
0,9240
0,9581
Entropy
0.3305
0.1901
0.2766
0.2657
0.4364
0.3703
0.3296
0.3788
0,4109
0,0836
0.1686
0.2210
LagRank
0.9544
0.9656
0.8680
0.9307
0.7980
0.7997
0.8089
0,9732
0.9260
0,9592
LRR
0.9164
0.8962
0.7421
0.8516
0.7453
0.7040
0.6810
0,7101
0.9609
0.9598
0.8334
0,9180
DNA-GPT(Nco-277
0,9040
0.9449
0,7598
0.8696
0.7267
0.8164
0,7163
0.7531
0.9071
0.9655
0.5911
0.8212
CNN-GPT(ChaGPPT
0,8396
0.7898
0.6722
0.6146
0.6104
0.5745
0.5998
0.8560
0.8767
0.6729
0,8019
DNA ViNuuPT(Vicuua-7)) 7b)
0:6992
0.6695
0.5639
0.6442
0.5594
0.5628
0.5366
0,5529
0.7241
0.7305
0.6001
0.6849
NPR
0,7845
0.9697
0,5483
0.7675
0.5211
0.8276
0.4976
0.6154
0.9232
0.9696
0.7746
0.8891
DetecGGPT
0.4594
0.8008
0.3804
0.5469
0.3408
0.6542
0.3675
0,4542
04323
0.6800
0.7559
0.6227
Fast: DetectGPT
0.9907
0.9916
0.9021
0.9615
0.9064
0.9611
0.8498
0.9058
0.9942
0.9783
0.9035
0,9587
DPICYChatGPTT
1.0000
0.9821
0,96882
0.96.3
0.9996
0.9768
0.9438
0.9734
1,0000
0,9950
0.9686
0.9878
DPIC(Vicumaa 7b)
0.9976
0.9708
0.8990
0.9558
0.9986
0.9644
0.9394
0.9674
0.9992
0.9943
0.9690
0.9875
Generalizability. The detection performance of supervised methods, such as RADAR, exhibits significant performance differences in detecting different datasets. Taking ChanGPT-generated text as an example, the detection AUROC of RADAR on the Xsum dataset is 99. 99.72%. However, the detection AUROC on the PubMed dataset only 73.72%. This indicates that the detection performance of the supervised methods is susceptible to the domain dataset, which affects the generalizability. Zero-shot detection methods show more balanced detection performances on Xsum and PubMed datasets generated by ChatGPT. This suggests that the zero-shot method is more likely to maintain generalizability by extracting the intrinsic characteristics of L.Mss. DPIC extracts the deep intrinsic characteristics of LLMs by decoupling the prompt, allowing the supervised detector to focus more on the intrinsic characteristics. This approach enables DPIC to achieve the desired generalization while maintaining high detection quality.
5.3 Ablation studies
We conducted ablation studies to reveal the impact of different similarity measurements and decou- pling processes. Additionally, since the reconstructed prompts may differ from the original prompts, we discuss the impact of these differences on detection performance.
Similarity measurement. In this experiment, we explored the effects of different similarity mea- surements on the detection performance of the Siamese Network. Specifically, we experimented with absolute distance, dot product, and concatenation methods for combining feature vectors. The results are displayed in Table 2
When combining the feature vectors by absolute distance and dot product, the detection AUROC scores are 89.8 89.81% and 93.84% in detecting the Writing dataset generated by ChatGPT. In contrast, when combining the feature vectors by concatenation, the AUROC score is 98.21 %. These experi- mental results show that concatenation results in higher AUROC scores than other methods, which indicates that combining feature vectors by concatenation preserves more of the intrinsic characteristic information. contrast, the other methods result in the loss of critical intrinsic details. This findings indicates that providing the detector with richer intrinsic characteristic information helps achieve higher detection quality.
Prompt reconstruction and limitations Since reconstructed prompts may differ from the original prompts, we discuss the impact of the prompt reconstruction process on detection performance in this part. We tested the semantic similarity of the candidate text and the regenerated text obtained by directly using the original prompt. We also test the semantic similarity of the candidate text and regenerated text obtained by the reconstructed prompt. We used ste-Qwen.. 1.monn.5778 5-7B- 778-instructt 71 for feature extraction, and then used cosine similarity to measure the degree of semantic similarity.
7

---

Table 2: Detection AUROC of different similarity measurement Writing dataset generated by ChatGPT, GPT-4, and Claude3.
Similarity Measurement
Decoupling by ChatGPT
Decoupling by Vicuna-7b
ChatGPT
GPT-4
Claude3
ChatGPT
GPT-4
Claude3
Absolute Difference
0.8981
0.8696
0.9206
0.7878
0.7476
0.8426
Dot Product
0.9384
0.894
0.9575
0.9299
0.9111
0.9753
Concatenation (DPIC)
0.9821
0.9768
0.9950
0.9708
0.9644
0.9943
Detection Accuracy (AUROC)
1.0
Choiet
GPT4
Clundes
0.7
Wetting
Welling
Wriling
BNA-GFF
DIFSC
Figure 3: Detection AUROC of DNA-GPT, DNA-GPT (supervised), DNA-GPT (prompt), and DPIC. All regeneration processes are implemented using Vicuna-7b.
We part of HC3 test dataset generated by ChatGPT, GPT-4, claude3. The results presented in Figure From the result, we find that, compared to using the original prompt directly, the regenerated text obtained by the reconstructed prompt has higher semantic similarity with the original text in texts generated by GPT-4 and Claude3.
We believe that this may be due to the following reasons: when users use LLM, the prompt is sometimes not detailed, which leads to the fact that the text generated by LLM may deviate from the original prompt. In other words, in the where the prompt is not detailed, even if the same prompt used, the text generated multiple times prone to larger deviations. However, when LLM used regenerate the prompt from candidate text, the reconstructed prompt is reconstruct from the candidate text, which makes the prompt more semantically consistent with the candidate text. the reconstructed prompt for text regeneration, the regenerated text show more semantic similarity with the candidate text, which more helpful reducing the impact of semantics the detection process.
Thereby, when
However, directly using LLMS for regenerating for semantic disentanglement can indeed introduce some time delays and costs, which we will discuss in the Limitations Section 6
1.0
With the original prompt re-generated prompt
Cosine Similarity
0.7
ChatGPT
GPT4
Claude3
Figure 4: Cosine Similarity of the candidate text and the regenerated text obtained by different prompts.

---

Decoupling Process In the decoupling process, given a candidate text, we initially reconstruct the prompt based on the candidate text. The reconstructed prompt is then used for the auxiliary LLM to obtain the regenerated text. Then, we use the embeddings of the candidate text and the regenerated text as the discriminative feature to train a classifier. DNA-GPT 42 shares a similar idea. Specifically, given candidate text, DNA-GPT truncates from the half position of the text length and then completes the text based on the truncated text using an auxiliary LLM. Unlike us, DNA-GPT calculates the divergence between the candidate text and the completion of the truncated text and uses a zero-shot classifier for detection. Another significant difference is that there is no semantic prompt guidance in the completion process. In order to fully illustrate the effectiveness of our method, explore the impact of supervised learning and prompt guidance on detection performance using DPIC and DNA-GPT as examples.
First, we compare the impact of zero-shot and supervised methods on the detection performance using DNA-GPT. Specifically, given a candidate text, DNA-GPT truncates from the half position of the text length and then completes the truncated text using an auxiliary L.M.. Then, we use the embeddings of the candidate text and the completion of the truncated text as the discriminative feature to train a Siamese Network to classifier; the detailed experimental setup is consistent with the description in Section 5.1] The results are displayed in Figure comparing DNA-GPT and DNA-GPT (supervised). The average detection AUROC of DNA-GPT (supervised) is 88.39%, 85.5 85.50%, and 91.51% datasets generated by ChatGPT, GPT-4, and Claude3, resecctveel.. In contrast, the average detection AUROC of DNA-GPT is 64.42%, 55.29 55.29%, and 68.49%. DNA-GPT (supervised) shows substantial improvement compared to DNA-GPT, demonstrating that supervised methods help achieve higher detection AUROC. However, DNA-GPT (supervised) exhibits a significant decrease in detection AUROC the PubMed dataset compared the Xsum and Writing datasets, indicating that generalizability is affected when using supervised methods.
Afterward, we evaluated the impact of the prompt semantic guidance on the detection performance and generalizability of DNA-GPT (supervised), Specifically, given a candidate text, we reconstruct the prompt based on the candidate text and complete the truncated text based on the prompt using an auxiliary LLM. Then, we measure the difference between the candidate text and the prompt- based completion of the truncated text and use Siamese Network as the classifier. The results are displayed in Figure comparing DNA-GPT (supervised) and DNA-GPT (prompt). Compared to DNA-GPT (supervised), DNA-GPT (prompt) achieves more balanced detection performance across the three datasets. On the PubMed dataset, DNA-GPT (prompt) achieves detection accuracies of 79.289 79.28%, 82.18%, and 86.46% while DNA-GPT (supervised) achieves only 74.93%, 76.66%, and 79.52%. respectively. It demonstrates the importance of prompts in regeneration, as they help reduce semantic bias between the regenerated text and the candidate text, thereby mitigating the impact of the dataset domain on the generalizability of the detector.
Finally, compare the detection performance with the truncated-completin method and the regeneration method on the detection performance, and the results are displayed in DNA-GPT (prompt) and DPIC in Figure As shown, the regeneration method of DPIC achieves better detection quality and better generalization.
5.4 Robustness
has pointed out that the previous methods experience reduction in performance in complex scenarios where the text to be detected is subjected to interference. To better understand how DPIC performs in real-world scenarios, we evaluate our detection method under two modification methods
Existing research 19 33
The first one is the proposed paraphrasing attack called DIPPER [19 (or Discourse Paraphrase). DIPPER is an BaB-aaraeeter paraphrase generation model built by fine-tuning T5-XXL. It can paraphrase paragraph-lenger texts, re-order content, and optionally leverage context such as input prompts. The second interference we consider is a method that is more accessible to the broader audience of large language models and does not require specialized knowledge, namely the back- translation attack. Back-translation refers to the action of translating a work that has previously been translated into the same language. We employed DeepL Translator 28 to translate the given English text into Chinese, followed by a subsequent translation back into English.
http://www.deell..c/e/ee/////e//

---

Table 3: Detection performance of DPIC and the baselines in detecting Xsum dataset generated by ChatGPT, GPT-4, and Claude3 with interference.
CharGPT
GPT-
Claude3
Methods
Ori.
DIPPER
Back-translate
Ori.
DIPPER
Back-translate
Ori.
DIPPER
Back-translate
RoBERTa- base
0.9150
0.8148
0.8379
0.6778
0.6469
0.7536
0.8944
0.8120
0.8052
RoBERIa-lage
0.8507
0,7884
0.6853
0.6879
0.6833
0.6660
0,9027
0.8153
0.7583
0,9924
Likelihond
0,9577
0.6296
0,9760
0,9446
Entropy
0.3305
0.4364
0.5552
0.3705
LogRank
0,9584
0.8596
0,9260
0,7980
0.6432
0,8436
0,9783
0.9256
0,9488
LRR
0.9164
0.8448
0,8621
0,7453
0,6607
0,8003
0,9609
0.9240
0.9243
2.7-
0,9040
0.7733
0.8624
0.7267
0,5595
0,7776
0.9071
0.7876
0,8399
DNA-GPT(ChaGPPT
0.8396
0.7910
0.7975
0.6146
0,5454
0.6070
0.8560
0.7996
0,7814
DNA GPT(Vicuma Th)
0.6992
0.6528
0.6175
0.5594
0.5523
0.5827
0.7241
0.6403
0.6321
NPR 0.7845
0,5648
0,8050
0.5211
0,3006
0.6820
0,9232
0,7860
0,9042
DetectGPT
0.1823
Fast-DeterGGPT
0.9907
0.9536
0
0.9064
0.8057
0,9137
0.9942
0,9720
0,9860
DPIC(ChatGPT)
1,0000
0,9972
0.9996
0,9991
0.9931
1.0000
0.9979
DPiC(Vicum--7))
0,9976
0.9980
0.9589
0.9986
0,9969
0,9903
0,9992
0.9996
0.9966
We present the detection performance of DPIC and baselines in detecting Xsum dataset generated by ChatGPT, GPT-4, and Claude3 with interference in Table RADAR shows the smallest decrease among baselines against DIPPER attacks, especially for text generated by GPT-4 with decrease of 00.07%, illustrating the robustness of RADAR in incorporating adversarial networks into detection. However, our method still maintains optimal detection performance after both DIPPER and back- translation attacks. The detection AUROC of DPIC is 100% and 99.72% for detecting the Xsum dataset generated by CharGPT under DIPPER and back-translation attacks, respectively, indicating that our method more robust in real-world scenarios. DIPPER and back-translation attacks mainly alter the text semantically. Our method decouples the prompt-guiden semantic information and the intrinsic characteristics of the generative model, rendering semantic attacks ineffective and ensuring robustness.
Limitations
We mitigated the risk of benchmark contamination in the testing phase by ensuring different sources for the training and test sets. However, recent studies 7 40 (3] have indicated that benchmark contamination may be widespread in evaluating large models, which is a limitation not addressed in this paper.
Another limitation of our method is the time required in the decoupling process. The regeneration process significantly extends the processing time for each text segment. We selected advanced training-bass and zero-shot detection methods and tested memory and time consumption. Our method does require more memory and time, compared those that only input candidate for classification, primarily due to the regeneration component involving the LLM. Detailed information regarding this can be found in the Appendix
Conclusion
We reconsider the detection methods for LLM-generated text and propose decoupling-based de- tection method, DPIC. This method extracts deep intrinsic characteristics of the black-box model generated texts. Through the decoupling process, we provide the detector with reconstructed prompt covering the topic of the candidate text, allowing the detector to focus on deep intrinsic characteristics, thereby achieving better generalization performance while maintaining detection quality. Experimen- tal results further demonstrate that DPIC significantly improves detection performance by 6.76% and 2.91 % when detecting texts generated by GPT-4 and Claude 3, respectively.
10

---

8 Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant U2336206, Grant 62472398, Grant 62102386, Grant 62121002, Grant U20B2047 and by Open Foundation of Key Laboratory of Cyberspace Security, Ministry of Education (No.KLCS20240200),
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko A Altenschmidt, Sam Altman, Shyamal Anadkat, ct al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[] Anthropic. Claude, 2023.
[3] Anton Bakhtin, Sam Gross, Myle Out, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv: arXiv:/906.053 1906.0351, 2019.
[4] Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang. and Yuc Zhang. Fast-dctectgpt: Efficient zero- shot detection machine- machine-generated text conditional probability curvature. The Twelfth International Conference on Learning Representations.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Bee Kamar, Peter Lec, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 2023.
[6] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Chunyuan Deng, Yilun Zhao, Yuzhao Heng. Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. Unveiling the spectrum data contamination in language model: A survey from detection to remediation. In Findings of the Association for Computational Linguistics ACI. 2024, pages 16078-16092, 2024.
[8] Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading com- prehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1342-1352, 2017
[9] Nan Duan, Duyu Tang. Peng Chen, Ming Zhou Question generation for question answering. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 666-874, 2017.
[10] Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. Tweepfake: About detecting deepfake tweets. Plos one, 16(5))e025141)) 1415, 2021.
[11] Angela Fan, Mike Lewis, and Yann Dauphin. Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), pages 889-898, 2018.
[12] Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. Gltr: Statistical detection and visualization of generated text. In Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (ACL), 2019.
[13] Josh A Goldstein, Girish Sastry. Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Se- dova. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.0423 2023.
[14] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nic, Yuxuan Ding, Jianwei Yue, and Yupeng Wu, How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.0733 0000:297...7777... 2023.
[15] Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Technologies, Volume 1 (Long and Short Pupers), pages 1689-1701, 2019.
[16] Xiaomeng Hu, Pin- Yu Chen, and Tsung Yi Ho. Radar: Robust ai-text detection via adversarial learning. Advances in Neural Information Processing Systems, 36:15077-15095, 2023.

---

[17] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808-1822, 2020.
[18] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567-2577, 2019.
[19] Kalpesh Krishna, Yixiao Song. Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, hut retrieval is an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
[20] Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30:121 204, 2020.
[21] Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Deepfake text detection in the wild. arXiv preprint arXiv:2305. 0000005..1222 13242, 2023.
[22] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint a9X.: /6992, 2019.
[23] Alex Mitchell, Professor catches student cheating with chaigtt!: 'i feel abject terror', 2022. Retrieved from httpa://nypost.c wttp://tts.ocoo////..../ wwtp://///////////eeeee// http:////////ttsseeeeeaa atudents-using- www.leett---iiiin---cht--------- terperpp--t---cha-----oee-ooeeee
warns/
Accessed on February 17, 2023.
[24] Eric Mitchell, Yoonho Lec, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine- generated machine-generated text detection using probability curvature. In International Conference on Machine Learning. pages 24950-24962. PMLR, 2023.
[25] Sandra Mitrovié, Davide Andreoletti, and Omran Ayoub. Chatgpt human? detect and explain. ex- plaining decisions of machine learning model for detecting short 1rXiv:2301.13852,22000
text. arXiv preprint
[26] Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. Covid-qa: A question answering dataset for covid-19 19. In Proceedings of the Ist Workshop on NLP for COVID-19 19 at ACL 2020, 2020.
[27] John Xavier Morris, Wenting Zhao, Justin Chiu, Vitaly Shmatikov, and Alexander M Rush. Language model inversion. In The Twelfth International Conference on Learning Representations.
[28] Shashi Narayan. Shay Cohen, and Mirella Lapata. Don't give me the details. just the summary! topic- aware convolutional neural networks for extreme summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, 2018.
[29] OpenAL. Introducing chatgpt, 2022.
[30] OpenAL. New ai classifier for indicating ai-written text, Jan 2023.
[31] Mary Louise Kelly Patrick Wood. 'everybody is cheating': Why this teacher has adopted an open chatgpt policy. 2023. Retrieved from https::/www npr. org/ 22//2222222202/2 0ttp////////111111 25/115149921// aaatgpt-ai-educatte
cheating- classroocamsoooommmmooooo Lalaasssoooowwartooo--eeeeeee school Accessed on January 26, 2023.
[32] Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhat- tacharya, Mobin Javed, and Bimal Viswanath. Deepfake text detection: Limitations and opportunities. In 2023 IEEE Symposium on Security and Privacy (SP), pages 1613-1630, JEEE, 2023.
[33] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Sohcil Feizi. Can ai- ai-gennrated text be reliably detected? arXiv preprint arXiv:2303. 11156, 2023.
[34] Oscar Sainz, Jon Ander Campos, Iker Marefa-Ferreeoo Julen Exxaniz, Oier Lopez de Lacalle, and Encko Agirre. NIp evaluation trouble: On the need to measure Ilm data contamination for each henchmark. In The 2023 Conference on Empirical Methods in Natural Language Processing.
[35] Alejo Jose G Sison, Marco Tulio Daza, Roberto Brizuul,,, and Eduardo Chatgpt: More than a "weapon of mass deception" ethical challenges and responses from the human- centered artificial intelligence (hcai) perspective. International Journal of Human Computer Interaction. 40(17):4853 4872, 2024.
-errchn.
12

---

[36] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herberr--oose Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, al. Release strategies and the social impacts of language models. arXiv preprini arXiv: (aae::990.00000 09203, 2019.
[37] Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for zero- shot detection prcchine-eenerted text. Findings of Association for Computational Linguistics: EMNLP 2023, pages 12395-12491 2023.
[38] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[39] Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384 8395, 2020.
[40] Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. Benchmark data contamination of large language models: A survey. arXiv preprint 2404444406, 2024.
[41] Duanli Yan, Michael Fauss, Jiangang Hao, and Wenju Cui. Detection of ai-generated essays in writing assessment. Psychological Testing and Assessment Modeling, 65(2):125 144, 2023.
[42] Xianjun Yang, Wei Cheng. Yue Wu. Linda Ruth Petzold. William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent gram analysis for training-free free detection of gpt-generated text. In The Twelfth International Conference on Learning Representations.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper proposes a decoupling-basee method for extracting the deep intrinsic characteristics of LLM from the text in black-box scenarios. The detection of GPT-4 and Claude3 improves by 6.76% and 2.91 % compared baselines
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in Limitations
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
This paper does not include theoretical results, we validate our method by experimental results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed reproduce the main experimen- tal results of the paper.
We fully disclose all the information needed to reproduce the main experimental results in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide open access to the data and code, we also provided our model.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blcc//uudeeeeeeeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: [NA].
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Limitations
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics and our research was conducted with that in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of our work are discussed in Section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We will release models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited it.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and models are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/modell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
19


Reviews:
[{"review_id": "18tVqUJDM9", "reviewer": "Reviewer_qTwx", "summary": "This paper develops an LLM-generated text detection method by ****. The key idea is to regenerate text based on a prompt reconstructed by an auxiliary LLM from the candidate text so that the regenerated text and the original candidate text can be used to extract similarity-based features, which are used for a classifier.", "questions": "Please answer the weaknesses raised in the Weaknesses section.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- (S1) The idea is simple and intuitive.\n- (S2) Comprehensive experiments and analysis especially including paraphrased machine-generated text such as DIPPER and back-translate (in Table3)\n- (S3) The paper is well-written and easy to follow.", "weaknesses": "- (W1) It is not clear if the proposed method can replace other classification-based methods. Ghostbuster [41] is known to be a strong machine-generated text detection method and should be compared for evaluation (it also offers a benchmark as well). Other recent simple-yet-effective methods could be compared as well. For example,\n    - [Ref 1] Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors https://aclanthology.org/2024.eacl-short.25.pdf\n    - [Ref 2] Your Large Language Models Are Leaving Fingerprints https://arxiv.org/abs/2405.14057\n- (W2) The datasets used for evaluation might be too easy to detect. Unless there’s a special reason, the proposed should also be tested in other benchmarks used for machine-generated text detection research (e.g., the Ghostbuster dataset).\n- (W3) The feature extraction is computationally expensive. Two LLM inferences (prompt generation, candidate text regeneration) and then dual-encoder-based feature extraction. It is not clear if the feature extraction offers significantly better benefits compared to more lightweight approaches (e.g., [Ref 1][Ref 2] above)\n\n### Minor comments\n\nFor perturbation, in addition to DIPPER, OUTFOX is another option.\n    - [Ref 3] OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples https://arxiv.org/abs/2307.11729", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and expertise you have invested in these reviews. We are delighted to receive positive feedback that our work provides a solid contribution to the field, especially since the paper is well-written, the proposed idea is simple and intuitive, and the experiments and analysis are comprehensive.\n\nBelow we provide point-by-point responses to your comments and questions.\n\n---\n\n- **Weaknesses 1 & Weaknesses 2**: The datasets used for evaluation might be too easy to detect. Unless there’s a special reason, the proposed should also be tested in other benchmarks(e.g., the Ghostbuster dataset). & Add comparision with other classification-based methods, including Ghostbuster, Fingerprints,  Smaller Language Models.\n  \n    **Response**: Following your advice, we compared detection performance on Ghostbuster benchmark[1]. As stated in Ghostbuster, out-of-domain performance yields the fairest comparison across training-based and zero-shot methods, so we mainly focus on the performance of out-of-domain generalization. In this benchmark, we compared with Ghostbuster[1], Fingerprints[2], and Smaller Models[3]. Additionally, we included comparisons with CoCo[4], a training-based method. We also included comparisons with RADAR and Fast-DetectGPT, which are the advanced training-based and zero-shot baselines in our paper. As the following Table shows,  the detection AUROC of Ghostbuster, Fast-DetectGPT and DPIC surpasses 0.98, indicating that Ghostbuster benchmark is easily detected by these methods. \n    \n    |  | Methods | Out of domain |  |  |  |\n    | --- | --- | --- | --- | --- | --- |\n    |  |  | News | Creative Writing | Student Essays | Avg.  |\n    | Training-based | Ghostbuster | 0.9929 | 0.9839 | 0.9913 | 0.9893 |\n    |  | Fingerprints | 0.8108 | 0.8373 | 0.7846 | 0.8109 |\n    |  | CoCo | 0.9733 | **0.9979** | 0.9011 | 0.9574 |\n    |  | RADAR | **0.9984** | 0.8717 | 0.9418 | 0.9373 |\n    | Zero-shot | Smaller Models with Fast-DetectGPT (GPT-Neo-125M) | 0.9983 | 0.8957 | 0.9673 | 0.9537 |\n    |  | Fast-DetectGPT | 0.9979 | 0.9543 | **0.9965** | 0.9829 |\n    |  | DPIC | 0.9950 | 0.9978 | 0.9774 | **0.9900** |\n    \n    Actually, our benchmark used in our original paper, already covers two domains from the Ghostbuster benchmark, namely Xsum for news articles and Writing for creative story writing. In addition, our benchmark also includes multiple prompts and three generative large language models, **offering a more rigorous evaluation standard.** Therefore, to better compare the performance of these newly added methods with DPIC, we further tested these methods using the benchmark employed in our paper. \n    \n    We compared DPIC to these new detection methods in our benchmark. The results in **Table 1 of the attached PDF** show that DPIC performs best on average, which verifies its effectiveness. In detail, Ghostbuster only achieved an average detection AUROC of 0.8705, 0.9170, and 0.9239 in detecting different domain datasets generated by ChatGPT, GPT4, and Claude3, respectively, while DPIC demonstrates superior performance, achieving an average detection AUROC of 0.9634, 0.9734, and 0.9878, respectively. We will include this part in the final version of the paper. \n    \n\n---\n\n- **Weaknesses 3**: The feature extraction is computationally expensive. Two LLM inferences (prompt generation, candidate text regeneration) and then dual-encoder-based feature extraction. It is not clear if the feature extraction offers significantly better benefits compared to more lightweight approaches (e.g., [Ref 1][Ref 2] above)\n  \n    **Response**: The feature extraction process in our method incurs certain costs, but the benefits it offers are significant. As Table 1 in our original paper shows, our method demonstrates superior generalization compared to existing approaches, achieving an average detection AUROC of 0.9806 in identifying texts from diverse domains generated by two commercial closed-source models: GPT4 and Claude3.\n    \n    The superior performance brings practical value. Generally, a multi-stage detection strategy is practically effective for many detection tasks involving large amounts of generated text. Initially, a fast detection model with lower accuracy is used for preliminary screening. Once potentially generated text is identified, a more accurate detection model is employed in the second stage for detailed verification. Our method is well-suited as an accurate detection model for this second stage. Additionally, our method offers direct practical value in situations where time constraints are relaxed, and the volume of candidate texts is manageable, such as in academic plagiarism detection. \n    \n\n---\n\n- **Minor Comments**: For perturbation, in addition to DIPPER, OUTFOX is another option.\n  \n    **Response**:  Good Suggestion. Following your advice, we added robustness detection experiments against OUTFOX attacks[5]. The results are shown in **Table 2 in the attached PDF**. Our method is somewhat affected, but its detection performance after the attack remains superior compared to the other methods. We will include this part in the final version of the paper.\n\n---\n\n### **References**\n\nPlease see **Rebuttal References** in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DwzY34Wj8A", "reviewer": "Reviewer_XAPE", "summary": "This paper proposes a novel method named DPIC (Decoupling Prompt and Intrinsic Characteristics) for detecting texts generated by LLMs. The authors posit that generated texts are a coupled product of prompts and intrinsic characteristics, and suggest that decoupling these two elements can enhance detection quality and generalization ability. Specifically, the DPIC method employs an auxiliary LLM to reconstruct the prompt of the candidate text, and then uses this prompt to regenerate the text. By comparing the similarity between the candidate text and the regenerated text, the detector can better focus on the intrinsic characteristics of the generating model. Experimental results show that, compared to baseline methods, DPIC improves the detection of texts generated by GPT-4 and Claude-3 in different domains by an average of 6.76% and 2.91%, respectively.", "questions": "How is the accuracy of prompt reconstruction evaluated? Did the authors verify the consistency between the reconstructed prompts and the original prompts? If there is a significant difference between the reconstructed prompts and the original prompts, how much impact would that have on the detection results?", "limitations": "The authors did not discuss the out-of-distribution generalization ability of DPIC.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A new perspective is proposed, improving detection performance by decoupling prompts and intrinsic characteristics, with strong motivation and description. More importantly, this method can handle black-box detection scenarios.\n- Experimental results demonstrate that the DPIC method significantly enhances detection quality and generalization ability, especially when dealing with black-box models.", "weaknesses": "- DPIC involves multiple steps, including prompt reconstruction and text regeneration, which significantly increase computational costs and resource overhead in practical applications. An ablation analysis on the impact of the training sample size would help readers further understand and evaluate the DPIC method.\n- DPIC is essentially a supervised method and should be compared fairly with methods based on RoBERTa (not the OpenAI detector trained on GPT-2 but a RoBERTa classifier trained from scratch).\n- Supervised methods perform well within their domain but have poor out-of-distribution generalization ability, which lacks discussion in the paper. For example, the performance of DPIC trained on XSum on Writing, and the performance of DPIC trained on ChatGPT re-generated data on Claude-3. This is crucial for highlighting the effectiveness of DPIC in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful reviews of our manuscript. Below, we provide point-by-point responses to your comments and questions.\n\n- **Limitations & Weaknesses 3:** The authors did not discuss the out-of-distribution generalization ability of DPIC.\n    \n    **Response:** Actually, we have already considered the out-of-distribution generalization in our original paper, including the generalization to three datasets and different LLMs. Other reviewers have also praised the generalizability of DPIC. As we described in lines 198-201 and 208-215 of our paper: for training, we used only HC3 dataset; For testing, we used three datasets that are not in the domains covered in the HC3 dataset to evaluate the generalizability of detectors. We also use different LLMs, e.g., Claude3, to generate texts on the test datasets. I hope this response can clear up your misunderstandings.\n    \n    ---\n    \n- **Weaknesses 2**: DPIC is essentially a supervised method and should be compared fairly with methods based on RoBERTa classifier trained from scratch.\n    \n    **Response**: This is a great suggestion. Following your advice, we trained RoBERTa-base and RoBERTa-large classifier **from scratch using the same training datasets as DPIC**. We also added comparisons with other training-based methods, including Ghostbuster[1], Fingerprints[2], and CoCo[4]. These results are shown in the below table and **Table 1 in the attached PDF**. DPIC still achieves the best detection performance compared to other methods, further demonstrating the effectiveness of DPIC. We will include this in the final version of the paper.\n    \n    |  | ChatGPT |  |  |  | GPT4 |  |  |  | Claude3 |  |  |  |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    |  | XSum | Writing | PubMed | Avg. | XSum | Writing | PubMed | Avg. | XSum | Writing | PubMed | Avg. |\n    | Roberta-base-trained from scratch | 0.6828 | 0.8298 | 0.7582 | 0.7569 | 0.6830 | 0.8235 | 0.7113 | 0.7392 | 0.9699 | 0.9897 | 0.9379 | 0.9658 |\n    | Roberta-large-trained from scratch | 0.9066 | 0.9327 | 0.7450 | 0.8614 | 0.7360 | 0.8180 | 0.7448 | 0.7662 | 0.9678 | 0.9798 | 0.8670 | 0.9382 |\n    | DPIC | **1.0000** | **0.9821** | **0.9082** | **0.9634** | **0.9996** | **0.9768** | **0.9438** | **0.9734** | **1.0000** | **0.9950** | **0.9686** | **0.9878** |\n    \n    ---\n    \n- **Weaknesses 1**: DPIC involves multiple steps, including prompt reconstruction and text regeneration, which significantly increase computational costs and resource overhead in practical applications. An ablation analysis on the impact of the training sample size would help readers further understand and evaluate the DPIC method.\n    \n    **Response**: Our method’s feature extraction process incurs certain costs, but the benefits it offers are significant. As Table 1 in our original paper shows, our method demonstrates superior generalization compared to existing approaches, achieving an average detection AUROC of 0.9806 in identifying texts from diverse domains generated by two commercial closed-source models: GPT4 and Claude3. The superior performance brings practical value. Our method can be deployed in the second stage of a funnel-shaped multi-stage detection process, and can also offers direct practical value in situations where time constraints are relaxed and the volume of candidate texts is manageable, such as in academic plagiarism detection. \n    \n    Following your advice, we tested the impact of the training sample size (500-2500) on DPIC’s detection performance. The results in **Figure 1 in the attached PDF** indicate that our method achieves relatively excellent detection performance even with a smaller sample size because DPIC captures intrinsic features and reduces the impact of training domains.\n\n---\n\n- **Question**: How is the accuracy of prompt reconstruction evaluated? Did the authors verify the consistency between the reconstructed prompts and the original prompts? If there is a significant difference between the reconstructed prompts and the original prompts, how much impact would that have on the detection results?\n    \n    **Response:** \n    In our original paper, we did not directly measure the consistency between the reconstructed prompts and the original prompts. \n    Since DPIC aims to mitigate the influence of semantics in the detection process, we directly measured the semantic consistency between the original text and the regenerated text generated by the reconstructed prompts and the original prompts, to evaluate the effectiveness of the reconstructed prompts.\n    The results in Figure 4 of the original paper show that the regenerated text obtained by the reconstructed prompt has a higher semantic similarity with the original text in texts generated by GPT4 and Claude3. The underlying reason can be revealed from Table 4 that our question generation prompt makes the reconstructed prompts not only cover the original prompt but also have more details, helping LLM generate more similar text with respect to the original text. In other words, there is no significant difference between the reconstructed prompts and the original prompts.\n    \n    Following your advice, we have evaluated the detection results of different prompts on PubMed dataset with Claude3 model, including the original prompts and our reconstructed prompts, achieving a detection AUROC of 0.9543 and 0.9686, respectively. The results show that DPIC performs better than using the original prompt, indicating that the prompt will impact the detection performance.\n    \n    We will clarify this in the final version and thanks for your valuable suggestions.\n\n---\n\n### **References**\n\nPlease see **Rebuttal References** in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzmpFlMRRF", "reviewer": "Reviewer_Av5h", "summary": "This paper addresses the problem of detecting texts generated by large language models (LLMs), which is a crucial issue considering the potential misuse of such models. The authors propose a novel method, DPIC (Decoupling Prompt and Intrinsic Characteristics), which aims to extract the intrinsic characteristics of black-box models, as traditional detection methods requiring access to the model's interior are not feasible. DPIC uses an auxiliary LLM to reconstruct the prompt of a candidate text and regenerate a text from it, allowing the detector to focus on the intrinsic characteristics of the generative model. The similarity between the candidate and regenerated texts is then used as a detection feature. Results show that DPIC outperforms baseline methods, achieving an average improvement of 6.76% and 2.91% in detecting texts from different domains generated by GPT4 and Claude3, respectively.", "questions": "Figure2, can we use a pre-trained Siamese Network to measure the similarity in DPIC? Then, we do not need a training here.\n\nLN121, the section 3 looks strange because it does not provide a threat model but some experimental settings.\n\nTable1, I would expect to see more comparsion with trained detectors given that DPIC is also a trained detector.\n\nFigure3, it seems the major contribution is from the supervised training, while the contribution from the prompt is much smaller. Why?\n\nFigure3, the improvement of DNA-GPT(prompt) compared to DNA-GPT(supervised) mainly happens on PubMed. Is it caused by the QA style of the dataset?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to distinguishing between machine-generated and human-written texts by decoupling intrinsic characteristics from prompts. This innovative method offers new insights into the essential differences between these two types of text, marking a significant contribution to the field. One of the major strengths of this approach is its applicability to proprietary models. Given the usual inaccessibility to model parameters, working with such black-box models is challenging. The paper also demonstrates robustness across various datasets, source models, and paraphrasing attacks, suggesting that the proposed method is capable of maintaining performance under different conditions and adversarial scenarios. Another significant strength is the method's performance in terms of AUROC. It achieves significantly higher AUROC scores compared to previous zero-shot methods, although the paper acknowledges that comparing a method requiring training to zero-shot methods may not be entirely fair.", "weaknesses": "There are a few areas that could be improved. Firstly, the comparison with zero-shot methods seems potentially unfair since the proposed method requires training. Including trained detectors as baselines, for instance, using the same LLM used by DPIC to train a classifier directly, might provide a more balanced comparison. Secondly, the paper lacks a detailed analysis of the method's robustness on different lengths of text and languages. This could limit the applicability of the method in real-world scenarios. Lastly, the content of Section 3 appears misplaced as it discusses experimental settings rather than threat models. A reorganization or further clarification in this section would enhance the overall coherence of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and expertise you have invested in these reviews. We are delighted to receive positive feedback that our work provides a significant contribution to the field, especially since our innovative method offers new insights into the essential differences between human and AI-generated texts, and the superior generalization detection performance of our method for black-box models. This is indeed our original aspiration.\n\nBelow we provide point-by-point responses to your comments and questions.\n\n---\n\n- **Weaknesses 1 & Question 3**: Given that DPIC is also a trained detector, I would expect to see more comparisons with trained detectors. For instance, using the same LLM used by DPIC to train a classifier directly.\n  \n    **Response:** This is a great suggestion. Following your advice, we added a gte-Qwen1.5-7B-instruct[6] classifier (the same LLM used by DPIC) and trained it from scratch using the same training datasets as DPIC. We also added comparisons with other training-based methods, including Ghostbuster[1], Fingerprints[2], and CoCo[4].  These results are shown in the below table and **Table 1** in the attached PDF. It can be seen that our method still achieves the best detection performance compared to other baseline methods, especially in terms of detection generalization, which is a shortcoming of training-based detectors.\n    \n    |  | ChatGPT |  |  |  | GPT4 |  |  |  | Claude3 |  |  |  |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    |  | XSum | Writing | PubMed | Avg. | XSum | Writing | PubMed | Avg. | XSum | Writing | PubMed | Avg. |\n    | gte-Qwen1.5-7B-instruct Classifier | 0.9964 | 0.9505 | 0.7827 | 0.9098 | **0.9999** | 0.9460 | 0.7996 | 0.9151 | 0.9998 | 0.9879 | 0.9338 | 0.9738 |\n    | DPIC | **1.0000** | **0.9821** | **0.9082** | **0.9634** | 0.9996 | **0.9768** | **0.9438** | **0.9734** | **1.0000** | **0.9950** | **0.9686** | **0.9878** |\n    \n     ---\n\n    \n- **Weaknesses 2:** The paper lacks a detailed analysis of the method's robustness to different text lengths and languages.\n  \n    **Response:** This is a great suggestion. Following your advice, we evaluated DPIC's robustness to the number of words in text and different languages. First, we divided the texts in the Xsum, Writing, and PubMed datasets based on word count and tested DPIC's average detection accuracy. From the results shown in **Figure 2** in the attached PDF, DPIC performs best when the word count is larger than 100. \n    \n    We selected Chinese and Urdu according to the M4 benchmark[7] for **different languages**, achieving detection AUROCs of 0.9895 and 0.9830, respectively. This is thanks to the regenerative and encoder models we used, which support multiple languages. These results demonstrate the practicality of DPIC in real-world scenarios, and we will add this part in the final version. \n    \n---\n\n- **Weaknesses 3 & Question 2:** The content of Section 3 appears misplaced as it discusses experimental settings rather than threat models.\n  \n    **Response:** Sorry for the confusion. We will restate the threat model in terms of the attacker's capabilities and objectives, as well as the defender's capabilities and objectives in the final version. The attacker’s objective is to use large language models to generate text that impersonates human-written text,  and his capability is using the existing large language model, e.g., GPT4 and Claude3.  The defender’s objective is to distinguish between human and generated text. In this paper, we focus on the black-box scenario, and the capability of the defender is he can only use the text input and output of the potential source LLMs. \n    \n---\n\n- **Question 1:** In Figure 2, Can we use a pre-trained Siamese Network to measure the similarity in DPIC?\n  \n    **Response:** This is a good question. Following your advice, we measure the cosine similarity of the embeddings of the pre-trained Siamese Network, and evaluate the detection AUROC.  As shown in the results below, we take the datasets generated by Claude3 as an example. From the results, it can be seen directly using cosine similarity cannot yield considerable effective detection. \n    \n    |  | XSum | Writing | PubMed | Avg. |\n    | --- | --- | --- | --- | --- |\n    | No training | 0.4801 | 0.6913 | 0.6931 | 0.6215 |\n    | DPIC | **1.0000** | **0.9950** | **0.9686** | **0.9878** |\n    \n---\n\n- **Question 4 & Question 5:** Figure3, it seems the major contribution is from the supervised training, while the contribution from the prompt is much smaller. Why? Figure3, the improvement of DNA-GPT(prompt) compared to DNA-GPT(supervised) mainly happens on PubMed. Is it caused by the QA style of the dataset?\n  \n    **Response:** \n    We will answer these questions together. In Figure 3, DNA-GPT(supervised) utilizes the candidate text and its truncated and regenerated text to train a Siamese network classifier, which, compared to the original DNA-GPT's N-gram classification, can better decouple the intrinsic features of the text from its semantic information, achieving a significant performance improvement.\n    \n    The contribution from the prompts is less significant in the Xsum and Writing datasets, but more substantial in QA style dataset PubMed. The reason is indeed as you thought, the regenerated prompts from DPIC can assist DNA-GPT(prompt) in obtaining regenerated text that aligns more closely with the PubMed’s QA style, thereby, the improvement is more pronounced on this dataset.\n    \n    The above analysis indicates the effectiveness of both components of our DPIC method: the supervised training based on a siamese network and the regenerated prompt procedure. I hope this explanation resolves your concerns. We also appreciate the time you have taken to thoroughly understand our method.\n\n---\n\n### **References**\nPlease see **Rebuttal References** in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gsRKkJVCt0", "reviewer": "Reviewer_eWrN", "summary": "In this paper, the authors present DPIC, a novel model for detecting LLM-generated text, which centers on having an auxiliary LLM reverse-generate prompt on candidate text, and then letting the LLM re-generate the answers to the prompt and classify them based on the similarity between the candidate and the re-generated text. The authors experimented it with competitors on multiple datasets and claimed that they achieved better performance.", "questions": "1. In the manuscript, I did not find out any details about the AUROC evaluation metrics used, such as its formula, or the evaluation steps. I understand that this is a commonly used evaluation metric, but an academic paper should be complete and detailed. The authors only briefly indicated its mathematical meaning, which I think is not sufficient.\n\n2. For the testing part of the models, the authors used three representative commercial closed-source models: gpt-3.5, gpt-4, and claude3 to generate text for specific prompts for model testing. While it can be argued that closed-source models are more difficult to access than open-source models to infer that it is more challenging to recognize text generated by closed-source models, this inference is from the perspective of detection model builder, and applied to a real-world scenario, even if the candidate text is open-source LLM-generated, will the detection model have easy access to the internal parameters used to generate this text? I don't think the authors should make too much of a distinction between closed-source and commercial models for the task of detecting LLM-generated text. So I suggest the authors to add some open-source models to the test models, which will make the conclusions more convincing, and I think the readers are also interested in the differences between texts generated by commercial closed-source models and open-source models. I understand the infeasibility of conducting new experiments in a short period of time, so I would suggest that the authors mention this in the limitations section.\n\n3. The authors mention in question 8 of the checklist that they provide enough details about computational resource consumption in Section 5.1, but I didn't find them in the corresponding section. I think that for this kind of detection task, comparing the running efficiency of the proposed model and the competitors is also an important part. In particular, I'd be interested to know the resource consumption comparison between DPIC models and those that only input candidate text for classification.", "limitations": "I think the statement made by the author in the limitations section (Appendix A) is not sufficient. It seems that the author is only analyzing the results of the experiment in the prompt reconstruction part by mentioning the possible scenarios that may occur when using the model. I would suggest that the authors flesh out this section from the perspective of the model's deployment in a real-world environment and its impact on society. Additionally, I suggest that the authors include an analysis with some of the current techniques that help AI-generated text escape detection, to test whether the model will be defeated by these models.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written and detailed.\n\n- The experiments for the model were adequate.\n\n- The core idea of the model is easy to understand and interesting.", "weaknesses": "- Lack of detail in evaluation metrics.\n\n- The testing model is underrepresented.\n\n- Details of the model's consumption of computational resources are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and expertise you have invested in these reviews. We are delighted to receive positive feedback that our work provides a solid contribution to the field. Below we provide point-by-point responses to your comments and questions.\n\n---\n\n- **Question 1**: Lack of detail in evaluation metrics, ROC definition.\n  \n    **Response:** The receiver operating characteristic curve, or ROC curve, is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting. For a detector $f$, its AUROC can be expressed by the following equation:\n    \n    $AUC(f) = \\frac{\\sum_{t_0\\in{\\mathcal{D}^{0}}}\\sum_{t_1\\in{\\mathcal{D}^{1}}}\\mathbf{1}\\left[f(t_0)<f(t_1)\\right]}{\\vert\\mathcal{D}^{0}\\vert\\cdot\\vert\\mathcal{D}^{1}\\vert}$\n    \n    where $\\mathbf{1}\\left[f(t_0)<f(t_1)\\right]$ denotes an *indicator function* which returns 1 if $f(t_0)<f(t_1)$ otherwise return 0; $\\mathcal{D}^{0}$ is the set of negative examples, and $\\mathcal{D}^{1}$ is the set of positive examples.\n    \n    We will add this part to the final version of the paper.\n    \n---\n\n- **Question 2**: I suggest the authors add some open-source models to the test models, which will make the conclusions more convincing.\n  \n    **Response:** Thanks for your suggestion! I agree with your viewpoint that detecting open-source LLM generated texts is meaningful. We carried out detection on two advanced open-source LLMs, Qwen1.5-7B-Chat [8] and Llama-3.1-405B-Instruct [9]. We also included comparisons with RADAR and Fast-DetectGPT, which are advanced training-based and zero-shot methods. We achieved significant detection performance, demonstrating the practicality of our method. We will include this part in the final version.\n    \n    |  | Qwen-1.5 |  | | |  | Llama-3.1|  |  |  |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- |--- | \n    |  | Xsum | Writing | Pub | Avg. | | Xsum | Writing | Pub | Avg. |\n    | Fast-DetectGPT | 0.9981 | **1.0000** | 0.7978 | 0.9319 | | 0.9986 | **0.9908** | 0.9251 | 0.9715 |\n    | RADAR | 0.9951 | 0.8752 | 0.5476 | 0.8059 | | 0.9930 | 0.8908 | 0.5183 | 0.8007 |\n    | DPIC | **0.9999** | 0.9865 | **0.9967** | **0.9943** | | **0.9988** | 0.9809 | **0.9526** | **0.9774** |\n\n---\n\n- **Question 3**: Running efficiency and Resource Consumption comparison between DPIC and detector.\n  \n    **Response:** Thank you for your suggestion. We selected advanced training-based and zero-shot detection methods and tested memory and time consumption, as shown in the table below. Our method does require more memory and time, compared to those that only input candidate text for classification, primarily due to the regeneration component involving the LLM. \n    \n    |  | Time Cost | Memory Consumption |\n    | --- | --- | --- |\n    | Fast-DetectGPT | 0.273s | 25447 MB |\n    | RADAR | 0.120s | 1355 MB |\n    | Ghostbuster | 2.493s | <500MB |\n    | DPIC | 2.665s | 28105 MB |\n    \n    We have also implemented strategies to minimize costs and enhance real-world applicability, such as substituting ChatGPT with Vicuna-7b, which allows for the entire process to be locally deployed at the cost of a slight decrease (0.46%) in the average detection AUROC. Besides, we will address cost considerations as part of the limitations.\n    \n    Despite the associated cost, our method holds great practical value and significance. Generally, for many detection tasks that involve large amounts of generated text, a multi-stage detection strategy is practically effective. Initially, a fast detection model with lower accuracy is used for preliminary screening. Once potentially generated text is identified, a more accurate detection model is employed in the second stage for detailed verification. Our method is well-suited as an accurate detection model for this second stage. Additionally, our method offers direct practical value in situations where time constraints are relaxed and the volume of candidate texts is manageable, such as in academic plagiarism detection.\n    \n---\n\n- **Limitations 1.1:** I would suggest that the authors flesh out this section from the perspective of the model's deployment in a real-world environment and its impact on society. \n    \n    **Response:** The ability to accurately detect LLM-generated text is critical for realizing the full potential of natural language generation (NLG) while minimizing serious consequences such as phishing, disinformation, and academic dishonesty. From the perspective of the end users, LLM-generated text detection could increase trust in NLG systems and encourage adoption. For machine learning system developers and researchers, the detector can aid in tracing generated text and preventing unauthorized use. Given its significance, our DPIC method can deepen the academic and industrial understanding of the mechanisms behind LLM-generated text detection. When deployed in a real-world environment, DPIC, like all other detection methods, requires GPU computing power. Balancing resource consumption and performance gains is also a problem worth investigating. We will flesh out limitation part from the perspective of the model's deployment in a real-world environment and its impact on society. Thank you again for your valuable suggestion!\n    \n- **Limitations 1.2:** An analysis with some of the current techniques that help AI-generated text escape detection, to test whether the model will be defeated by these models.\n  \n    **Response:** For robustness experiments against evasion, we have also added robustness tests against OUTFOX [5]. The results in **Table 1** of the attached pdf show that DPIC demonstrates superior performance, which verifies its effectiveness. Besides, we will add discussion of AI-generated text escape detection in the limitation part and point out the urgency of robust detector research.\n\n### **References**\nPlease see **Rebuttal References** in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **Dear Reviewers,**
We appreciate the constructive and insightful comments from all the reviewers! We sincerely appreciate your time and effort in reviewing our work. We have provided detailed answers to the comments and questions from each reviewer in the different author responses. All the modifications will be represented in the final version.

Below are References and attached Tables and Figures mentioned in our response.

---

### **Rebuttal References**

[1] Verma V, Fleisig E, Tomlin N, et al. Ghostbuster: Detecting Text Ghostwritten by Large Language Models[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 1702-1717.

[2] McGovern H, Stureborg R, Suhara Y, et al. Your Large Language Models Are Leaving Fingerprints[J]. arXiv preprint arXiv:2405.14057, 2024.

[3] Mireshghallah N, Mattern J, Gao S, et al. Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors[C]//Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers). 2024: 278-293.

[4] Liu X, Zhang Z, Wang Y, et al. Coco: Coherence-enhanced machine-generated text detection under low resource with contrastive learning[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 16167-16188.

[5] Koike R, Kaneko M, Okazaki N. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(19): 21258-21266.

[6] Alibaba-NLP/gte-Qwen1.5-7B-instruct  (we are not allowed to post any link, so please google it to find the link)

[7] Wang Y, Mansurov J, Ivanov P, et al. M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection[C]//Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 1369-1407.

[8] Qwen/Qwen1.5-7B-Chat (we are not allowed to post any link, so please google it to find the link)

[9] meta-llama/Meta-Llama-3.1-405B-Instruct (we are not allowed to post any link, so please google it to find the link)

---

### **Rebuttal Tables and Figures**

Please download the attached PDF file.

Meta-review: This paper presents DPIC, a new framework for detecting LLM-generated text. The approach works by utilising an auxiliary LLM to "reconstruct the prompt of a candidate text and regenerate a text from it, allowing the detector to focus on the intrinsic characteristics of the generative model. The similarity between the candidate and regenerated texts is then used as a detection feature". The authors have included a number of additional experiments during the rebuttal, which have largely addressed the reviewers' concerns. The paper is well written and easy to follow. The idea presented is simple and it works and, in the words of one reviewer, demonstrates "robustness across various datasets, source models, and paraphrasing attacks, suggesting that the proposed method is capable of maintaining performance under different conditions and adversarial scenarios“. The primary issue with this work is that it seems rather incremental in nature, and is rather limited in terms of technical depth and novelty.
Decision: Accept (poster)