DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning

Authors: xun  guo
shan  zhang
yongxin  he
ting  zhang
wanquan  feng
haibin  huang
chongyang  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 11:04:20

Abstract: Current techniques for detecting AI-generated text are largely confined to manual feature crafting and supervised binary classification paradigms. These methodologies typically lead to performance bottlenecks and unsatisfactory generalizability. Consequently, these methods are often inapplicable for out-of-distribution (OOD) data and newly emerged large language models (LLMs). In this paper, we revisit the task of AI-generated text detection. We argue that the key to accomplishing this task lies in distinguishing writing styles of different authors, rather than simply classifying the text into human-written or AI-generated text. To this end, we propose DeTeCtive, a multi-task auxiliary, multi-level contrastive learning framework. DeTeCtive is designed to facilitate the learning of distinct writing styles, combined with a dense information retrieval pipeline for AI-generated text detection. Our method is compatible with a range of text encoders. Extensive experiments demonstrate that our method enhances the ability of various text encoders in detecting AI-generated text across multiple benchmarks and achieves state-of-the-art results. Notably, in OOD zero-shot evaluation, our method outperforms existing approaches by a large margin. Moreover, we find our method boasts a Training-Free Incremental Adaptation (TFIA) capability towards OOD data, further enhancing its efficacy in OOD detection scenarios. We will open-source our code and models in hopes that our work will spark new thoughts in the field of AI-generated text detection, ensuring safe application of LLMs and enhancing compliance.

Main paper: 
 introduction: 
 Recently, the field of large language models (LLMs) [6, 12, 60, 69] has witnessed swift advancements, bringing great convenience to both professional settings and daily life. However, the widespread use of AI-generated text also poses threats to global information security, manifesting in the propagation of disinformation, misinformation, and content that can incite harmful or destructive behaviors [16] . Hence, the detection of AI-generated text has ascended as a task of vital importance.On the other hand, with the advancement of LLMs, the task of AI-generated text detection has elevated into an escalating challenge. Early methods, such as watermarking methods [23, 32] and statisticalbased methods [63, 47] encountered performance bottlenecks due to their reliance on manually hand-crafted forms. Moreover, the inherent inability to swiftly adapt to newly emerged LLMs further restricts their effectiveness. In stark contrast, recent training-based methods [11, 27, 24] have showcased notable improvements in performance. However, they remain constrained by the necessity of precisely paired training data and exhibit unsatisfactory generalization in out-of-distribution (OOD) detection scenarios due to the fixed binary classification formulation.In this paper, to overcome these challenges, we revisit AI-generated text detection and approach the problem from a fresh perspective. Individual authors invariably exhibit unique writing styles, collectively constituting a vast feature space of writing styles. Our key insight is that an LLM can be viewed as a specific author, with the text it generates conforming consistently to its unique style. In line with this key observation, we propose to reformulate AI-generated text detection as a task of distinguishing diverse writing styles within the feature space, rather than merely treating it as a binary classification problem between human-written and AI-generated. This reformulation presents a fresh perspective from which to approach the detection of AI-generated text.While distinguishing writing styles within a vast feature space may seem more challenging than binary classification, we can take advantage of mature techniques within the field of Natural Language Processing (NLP). Specifically, contrastive learning [9, 25, 21] employs a self-supervised approach to identify similarities and differences between positive and negative samples, thereby acquiring discriminative feature representations. These representations facilitate the differentiation of writing styles, enabling us to comprehend the characteristic patterns of different sources.Specifically, we propose a general framework that combines a novel multi-level contrastive learning with multi-task learning tailored for AI-generated text detection. Our method enhances the writingstyle encoding capabilities of various models, including but not limited to BERT-based [18] and T5-based [51] models. This framework is capable of calibrating the distances between samples sharing different degrees of relatedness, thereby encoding distinctive features of text generated by different authors (either humans or LLMs). During inference, we propose a pipeline anchored by dense information retrieval [58, 66] . Firstly, we pre-encode data drawn from the training dataset, extract features and store them within a feature database. Then, for any given query text, we simply calculate the similarity between its encoded feature and each feature vector nestled in the feature database. This measure is used to evaluate the degree of writing-style similarity. Finally, we employ the K-Nearest Neighbors (KNN) [15] algorithm for classification prediction.Applying our method across multiple commonly-used datasets consistently improves performance with various text encoders compared to their zero-shot baselines, exceeding current solutions and establishing new state-of-the-art benchmarks on each individual dataset. Impressively, our method also demonstrates superior generalization capabilities when faced with OOD data emerging from domains or models that are not encountered during the training phase. Specifically, the Average Recall (AvgRec) metrics on the Unseen Models and Unseen Domains test sets from the Deepfake [39] dataset outperform existing state-of-the-art solutions by 5.58% and 14.20%, respectively.Additionally, we introduce Training-Free Incremental Adaptation (TFIA), a novel and efficient scheme for boosting the generalization capability for OOD detection. Particularly, when confronted with a batch of OOD data, our goal is to enhance the model's adaptability to unseen domains using these data. The existing solutions either involve retraining the model or fine-tuning it on the new data. Contrastingly, under our framework, we discover that no further training is necessary. We simply encode these data using our previously trained model and incorporate them into the existing database to create an augmented database. Notably, within the aforementioned OOD detection scenarios, TFIA contributes to a further improvement in model performance: The AvgRec score witnesses an additional increase of 0.84% on Unseen Models, and a noteworthy 7.03% on Unseen Domains.Extensive experiments across several datasets and models consistently demonstrate that our proposed method outperforms previous approaches, establishing new state-of-the-art performance. This superiority is maintained in both In-distribution and OOD detection scenarios. In summary, the contributions of our study are manifold, and can be enumerated as follows:• We propose a novel end-to-end framework for AI-generated text detection, wherein we carefully devise a multi-task auxiliary, multi-level contrastive loss to learn fine-grained features for distinguishing various writing styles.• We present Training-Free Incremental Adaptation (TFIA), a key feature of our method. Utilizing a modest amount of OOD data, TFIA enhances the model's adaptability to new domains without further training, offering significant advantages for practical applications.• Our method achieves state-of-the-art performance on multiple datasets in both In-distribution and OOD detection scenarios, substantially surpassing existing methods.• We validate the effectiveness of each component through a series of ablation studies and provide visualization results for further analysis. Furthermore, we perform detailed experiments on TFIA and provide an empirical analysis.2 Related Work AI-generated text detection. Existing methods for AI-generated text detection generally fall into the following three categories: (i) Watermarking methods: watermarking methods, which include rule based [5, 30, 59] and deep learning based [17, 62] methods, involve embedding specific markers into AI-generated content, which can later be used to verify its source. The soft watermarking method [32] is an inference-time framework that involves grouping the vocabulary and decoding the next token preferentially. [23] proposes a method of adding watermarks by embedding backdoors triggered by special inputs into the model. UPV [41] is an unforgeable and publicly verifiable algorithm ensuring security against forgery and unauthorized detection attempts. (ii) Statistical methods: applying statistical metrics like entropy as thresholds to distinguish AI-generated text from human-written text. HowkGPT [63] identifies text origins by comparing perplexity scores of human-written and ChatGPT [6, 69] generated text. DetectGPT [47] utilizes the structural properties of the LLM's probability density for zero-shot detection of AI-generated text. Similarly, DetectLLM [56] employs normalized perturbation log-ranks for identification, exhibiting less sensitivity to perturbations. (iii) Supervised learning methods: GPT-Sentinel [11] incorporates a binary classifier into RoBERTa [43] and T5 [51] , which are directly trained on specific datasets. RADAR [27] employs an adversarial learning approach. By continually iterating to improve the detector and generator (both of which are LLMs), RADAR performs well in detecting both original and paraphrased AI-generated text. [55] utilizes contrastive learning to learn style representations on human-written text and uses the learned representations to identify different sources in a few-shot manner. Building on SCL [24] framework, CoCo [42] incorporates coherency information into the text representation, enhancing the ability to detect AI-generated text under resource-constrained conditions.Contrastive learning for NLP. The success of MoCo [25] and SimCLR [9] in the field of Computer Vision through contrastive learning has prompted research efforts to explore its potential in the area of Natural Language Processing (NLP), resulting in the development of various strategies to enhance text encoding capabilities via contrastive learning. IS-BERT [78] employs the DIM [26] framework to learn text representations. The ArcCon loss [80] is proposed to further enhance the model's semantic discriminating ability. MixCSE [79] introduces an unsupervised method for text representation learning, which incorporates a mixed negative sample strategy to boost the model's ability to discriminate complex semantics. VaSCL [75] adopts a more general approach to procure hard negatives by defining an instance-level contrastive loss and integrating Gaussian noise, it effectively enhances the model's performance in an unsupervised manner. DCLR [82] addresses the anisotropic problem brought about by negative samples in unsupervised sentence representation learning by introducing noise-based negative samples and virtual adversarial training, thereby improving the uniformity of the representation space. SimCSE [21] proposes to predict the input sentence itself, utilizing standard dropout as noise in an unsupervised manner. They also introduce a method for categorizing positive and hard negative sample pairs, thereby improving the sentence representations. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work is discussed in section A.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we provide a detailed description of the proposed method. We begin in Section 3.1 with a definition of AI-generated text detection and an overview of our proposed framework. In Section 3.2, we explore the design of the multi-task auxiliary multi-level contrastive learning, which are critical components of our framework. Finally, in Section 3.3, we introduce Training-Free Incremental Adaptation (TFIA), an efficient and effective strategy that leverages our method's generalization capability to handle out-of-distribution (OOD) data.In this work, we focus on the task of AI-generated text detection. Given a query text x with L tokens, x = {w 1 , w 2 , ..., w L }, we aim to determine whether it is human-written or AI-generated. Existing methods typically employ hand-crafted features [63, 47, 32] or adopt neural networks [11, 27, 24] to learn discriminative features between human-written and AI-generated text, treating them as two distinct categories. Ultimately, this task is reduced to a binary classification problem. While this formulation appears simple and straightforward, it neglects a vital factor. Analogous to how different novelists often demonstrate unique writing styles, it's critical to consider that different LLMs, due to variations in model architectures, training data and strategies, will inevitably infuse certain preferences and biases. Consequently, these variations induce stylistic differences. Therefore, categorizing all the texts generated by any LLM as the same category clearly overlooks these disparities.To overcome this limitation, we present DeTeCtive, a general framework compatible with diverse text encoders, as shown in Figure 1 . By leveraging a method that incorporates a novel multi-level contrastive learning with multi-task learning, DeTeCtive regulates the distance between samples of varying relations within the feature space, enabling the model to learn distinctive features. During inference, we adopt a dense information retrieval [66, 58] pipeline. The query text is classified by comparing its similarity with existing data entries in the database via the K-Nearest Neighbors (KNN) [15] algorithm.Optimization objective and justification. As discussed in Section 3.1, the distinctive writing styles attributed to different authors constitute a vast feature space. We perceive each LLM as an individual author. Consequently, AI-generated text detection evolves into a task of differentiating diverse writing styles within this feature space. Driven by this insight, it becomes critical to discern the similarities and discrepancies across varying writing styles. To effectuate this, we carefully devise a multi-task auxiliary, multi-level contrastive loss to facilitate the learning of fine-grained features. Specifically, LLMs developed by the same company often demonstrate similar preferences and inherent biases, given the shared model designs, training strategies, and datasets utilized [60, 13, 76, 70] . Common techniquess [81] like the unified auto-regressive modeling approach can also introduce some level of commonalities across company boundaries, though these may be less pronounced. Drawing parallels, the multi-level similarities among LLMs can be seen as familial kinship relations within an expansive family tree, distinguishing between those closely related and those more distant.We aim to capture these kinship relations with a text encoder, allowing the encoder to capture the multi-level similarities and distinctions. Consequently, we expect the encoded features from different sources to reflect their relations within the high-dimensional feature space as follows:E x∼Pi,y∼Pj [Sim(Φ(x), Φ(y))] > E x∼Pi,y∼Pj+1 [Sim(Φ(x), Φ(y))], ∀1 ≤ i ≤ j < 4,(1) where Sim denotes the similarity measurement, Φ(•) symbolizes the encoding function, and P 1 to P 4 signify different text distributions. Specifically, P 1 corresponds to the distribution generated by a particular LLM, P 2 to the distribution generated by LLMs developed by the same company, P 3 to the distribution generated by any LLM, and P 4 to the distribution of human-written text. This configuration aims to ensure that closeness in distribution corresponds to heightened similarity after encoding, encouraging the model to discern fine-grained multi-level relations.Multi-level contrastive learning. According to the similarity constraints defined in Ineq. 1 above, when processing a data batch containing N samples, for the i th sample T i , we assign it with a label x i .If the text is generated by an LLM, then x i = 0, otherwise, x i = 1. For those AI-generated text (i.e., x i = 0), we further label the model series and the specific model with y i and z i . Then, the encoding function Φ(•) maps the text into a d-dimensional feature space R d . For any two samples T i and T j , we compute the cosine similarity between their encoded features through Sim(Φ(T i ), Φ(T j )), and define this similarity metric as S(i, j). For human-written text T i (x i = 1), the similarity of its encoding with other human-written text encodings should be greater than the similarity with AI-generated ones, hence the following relationship should be satisfied: S(i, j) > S(i, k), ∀x j = 1, x k = 0.(2) Similarly, for text T i (x i = 0) generated by LLMs, Ineq. 1 suggests the existence of multi-level similarities and differences internally within LLMs, expressed as follows:S(i, j) > S(i, l) > S(i, m) > S(i, n), ∀z i = z j , z i ̸ = z l , y i = y l , y i ̸ = y m , x i = x m , x i ̸ = x n .(3) In order to achieve the above optimization objectives, we propose a method to solve these constraints hierarchically. Specifically, for the first inequality in Ineq. 3, we consider the index l, m, n that satisfies the conditions in the above constraints as a whole set, denoted as k, that is:EQUATION) For the remaining inequalities, similar conditions are set to satisfy the constraints, culminating in:EQUATIONTo address the similarity constraints defined in Ineq. 5, we adopt a framework based on SimCLR [9] and propose a method for defining positive and negative sample pairs, from which we derive the corresponding contrastive learning loss. Unlike conventional contrastive losses, our positive sample is not a single instance, but a collection of positive samples meeting the conditions. We consider the positive sample similarity as the average value related to the entire set of positive samples from the current sample's perspective. The handling of negative samples echoes that of SimCLR, rendering the contrastive learning loss as demonstrated in Eq. 6, where q signifies the current sample, K + is a set of positive samples, K -is a set of negative samples, τ indicates the temperature coefficient, and N K + represents the size of the positive sample set.EQUATIONDifferent constraints correspond to varied positive and negative sample sets, and accordingly, multilevel contrastive losses are calculated. Following the definition in Ineq. 5, these loss are denoted asL qi,1 , L qi,2 , L qi,3 , L qi,4, respectively. The overall multi-level contrastive loss L mcl is as shown in Eq. 7, where δ, α, β, and γ are coefficients used to adjust the weight between the multi-level relations. Take note that we designate δ as the coefficient balancing human-written and LLMs-generated, ensuring δ = α + β + γ, in an effort to maintain equilibrium, and we set α = β = γ = 1.0.EQUATIONThrough this carefully designed multi-level contrastive learning, we drive the model to learn finegrained features of different sources. This strategy empowers the model to discern diverse writing styles, enhancing the accuracy and generalization of AI-generated text detection.Multi-task auxiliary learning. Given that multi-task learning [7] enables the model to simultaneously learn multiple tasks online by sharing useful information between different tasks, it promotes the model to learn more generic and discriminative features, hence enhancing the model's generalization ability. Therefore, based on the aforementioned contrastive learning framework, we integrate an MLP classifier into the output layer of the encoder. This classifier performs a binary classification to determine whether a given query text was generated by human or LLM. We introduce a cross-entropy loss L ce to optimize this classifier as follows:EQUATIONwhere p i is the probability of the i th sample x i being classified as human-written. Therefore, the overall multi-task auxiliary multi-level contrastive loss is defined as:EQUATIONWith the rapid advancement of LLMs and their proliferating applications, new models continually emerge, spanning an increasingly diverse range of domains. Existing AI-generated text detection solutions, which typically treat the task as a binary classification problem [11, 24] , encounter difficulties in generalizing to new models and domains that yield out-of-distribution (OOD) data.When confronted with OOD data, these approaches commonly require retraining the model, a strategy that undeniably falls short of practicality in real-world applications. In light of this challenge, we propose a novel solution based on our existing framework -the Training-Free Incremental Adaptation (TFIA). This method allows our model to adapt to new domains or newly emerged LLMs without any further training. Specifically, When encountering OOD data not covered in the training set, we simply encode these data using our fine-tuned text encoder and incorporate the encoded features into the existing feature database D E , forming an expanded feature database D ′ E . During inference, replacing the original database D E with the expanded feature database D ′ E can enhance the performance of the model when dealing with OOD data. TFIA amplifies DeTeCtive's ability in identifying OOD sources, effectively leveraging the model's generalization capabilities. Through this mechanism, the DeTeCtive framework can adapt to OOD data without any retraining. We validate the effectiveness of TFIA through a series of experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The data we used are publicly available as discussed in section 4.1.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we first introduce the utilized datasets, evaluation metrics, baseline methods, and implementation details in Section 4.1. We then present main experimental results and other applications in Section 4.2 and Section 4.3, followed by ablation studies and Training-Free Incremental Adaptation (TFIA) analysis in Section 4.4.Datasets. In this study, we employ three widely-used and challenging datasets to evaluate our proposed method. The Deepfake [39] dataset includes text generated by 27 different LLMs and human-written content from multiple websites across 10 domains, encompassing 332K training and 57K test data. It also outlines six diverse testing scenarios, covering an array of settings from domain-specific to cross-domains, and out-of-distribution detection scenarios. The M4 [68] dataset is a multi-domain, multi-model, and multi-language dataset encompassing data from 8 LLMs, 6 domains, and 9 languages. With machine text in its testing data paraphrased by OUTFOX [33] , which introduces more complexity to the task. We perform experiments in both monolingual and multilingual settings, with the former containing 120K training and 34K testing data, and the latter comprising 157K training and 42K testing data. Finally, we make use of the TuringBench [61] dataset. TuringBench collects human-written text mainly from news titles and content, predominantly politics-related. Incorporating data from 19 LLMs within a single domain, it forms a dataset of 112K training and 37K testing entries. For more detailed information, please refer to Appendix C. Evaluation metrics. In line with existing works, we employ Average Recall (AvgRec) and the F1-score as our primary evaluation metrics. AvgRec, the average of recall for human-written (HumanRec) and AI-generated (MachineRec) text. Simple accuracy is inadequate for reflecting a model's performance on a minority class, especially in cases of data imbalance. The F1-score considers both the precision and recall of the model, evaluating overall model performance by computing the harmonic mean of these two. Together, these metrics present a comprehensive view of the effectiveness in detecting AI-generated text.Baseline methods. In the experiment assessing the compatibility of our method to various text encoders, we use the zero-shot results of these pre-trained text encoders on the Cross-domains & Cross-models subset of the Deepfake dataset as the baseline. We then compare these results with the ones after fine-tuning with our method. In all subsequent experiments, for comparison analysis, we utilize the pre-trained SimCSE-RoBERTa [21] model as our text encoder. We conduct comparisons with several training-based methods across all three datasets. These incorporate methods which train classifiers upon RoBERTa [43] and Longformer [2] models, the T5-Sentinel [10] method that classifies using the output probability of the T5 [51] model, and the SCL [42] approach that uses supervised contrastive learning to assist classification. Additionally, in all six scenarios of the Deepfake dataset, we extend our comparison to include manual-feature-based methods encompassing FastText [4] and GLTR [22] , in addition to DetectGPT [47] , a statistical-based method.Implementation details. For all our method's experiments, we use the interfaces and pre-trained model weights from the HuggingFace transformers [28] library. We freeze the embedding layers and only train the remaining model parameters. All experiments use the same hyperparameters and an AdamW [44] optimizer with a cosine annealing learning rate schedule. The peak learning rate is set at 2e-05, warmed up linearly for 2000 steps, and weight decay is set to 1e-04. The maximum input token length is 512. We train for 50 epochs with batch size of 32 per GPU on 8 NVIDIA V100 GPUs. During inference, we implement with an efficient K-Nearest Neighbors (KNN) [15] algorithm provided by the Faiss [46] library, to perform classification. For all comparative experiments, we use their open-source code and default settings for training and testing, and then report the results.Firstly, we fine-tune multiple pre-trained text encoders on Cross-domains & Cross-models subset of the Deepfake [39] dataset using our method to validate its broad compatibility. As shown in Table 6 , all models improve on their baselines, confirming our method's effectiveness with diverse text encoders in AI-generated text detection. Among them, the SimCSE-RoBERTa [21] model achieves the second-best performance with relatively fewer parameters. Thus, we select this model as our text encoder for all the subsequent experiments. Subsequently, to validate the performance in comparison to existing approaches, and to ascertain its robustness, we conduct experiments on three commonly-used datasets. These include the M4 [68] dataset (M4-monolingual and M4-multilingual), TuringBench [61] , and the Cross-domains & Crossmodels subset of Deepfake which is the largest and most challenging subset in the In-distribution scenarios of Deepfake. The results are shown in Table 1 . Our method achieves the state-of-the-art performance on each dataset. Using the AvgRec metric for illustration, our method surpasses the second-best method by 6.52% in the M4-monolingual setting and by 7.15% in the M4-multilingual setting. Despite the comparatively lower difficulty of the earlier released TuringBench dataset, where all comparative methods perform well, our model still outperforms the second-best by 0.15%. Furthermore, in the Cross-domains & Cross-models subset of Deepfake, our method exceeds the runner-up by 2.66%. Indicated by the aforementioned experimental results, our method performs commendably across multiple datasets, demonstrating that the framework we propose is robust against diverse data distributions and scenarios.To verify the capability of our method in terms of domain adaptation and out-of-distribution (OOD) detection, we conduct experiments on all six scenarios proposed in the Deepfake dataset. The dataset is strictly divided into different subsets to ensure that the testing data used for any given scenario is not used as training data for other settings. In In-distribution detection, comparison methods are trained Table 1 : Experimental results on M4-monolingual [68] , M4-multilingual [68] , TuringBench [61] and Deepfake's Cross-domains & Cross-models subset [39] . The best number is highlighted in bold, while the second best one is underlined. 2 , where our method surpasses the next by 5.58% and 14.2% respectively in terms of AvgRec. The results demonstrate the good generalization performance of our method, considerably outperforming existing methods. Finally, we devise a set of experiments where we incorporate corresponding OOD data from training sets of the Cross-domains & Cross-models subset into the database to aid detection. There is a substantial performance improvement in the Unseen Domains scenario, with an additional 7.03% increase in AvgRec. For the Unseen Models, only a slight improvement is observed, which can be attributed to the existing capability of identifying similar models. This also highlights the effectiveness of the multi-level contrastive learning within our method from another perspective. We refer to this finding as Training-Free Incremental Adaptation (TFIA), and we delve deeper into the analysis of TFIA capability in Section 4.4.Attack robustness. In order to investigate the robustness of our method to paraphrasing attack, we conduct experiments on the OUTFOX [33] dataset. The experiments are divided into three scenarios: Non-attacked, DIPPER [35] attack, and OUTFOX attack, the results are presented in Table 3 . From the experimental results, it can be seen that our method achieves the best results under all three settings, and the performance of our method does not decline much after being attacked, whereas the performance of other methods declines significantly. The analysis is as follows, we believe that our usage of the K-Nearest Neighbours (KNN) algorithm for classification offers our approach with a level of fault tolerance. Thus, minor disturbances prompted by certain attacks do not engender significant feature drift. Consequently, our method remains effective in detection. Therefore, these experiments show that our method has good robustness against paraphrasing attack.Authorship attribution detection. To further probe the efficacy of our method in the task of authorship attribution detection, we conduct comprehensive experiments on TuringBench [35] dataset, comparing our method against various baseline solutions. As depicted in Table 4 , our method illustrates commendable performance in this task, substantiating its capacity to learn and apply multi-level features effectively in a multi-class classification context.Ablation studies. To systematically evaluate the effects of each component in our method, we conduct a series of ablation studies as shown in Table 5 . The experiments show that removing any loss term results in a performance decrease. Notably, when the multi-level contrastive loss L mcl in Eq. 9 we proposed is replaced by a plain contrastive loss L pcl , the performance declines the most compared to other loss terms, because only the human-written text and AI-generated text are treated as negative sample pairs, without considering the internal relations. Furthermore, using a similarity-based KNN classification scheme also enhances the performance.Analysis on TFIA. We further explore how incrementally adding corresponding OOD samples affects the performance, illustrated in Figure 2 . The results demonstrate that as more OOD data are incorporated into the database, the model's performance improves consistently. Adding a modest amount of OOD data can considerably enhance the performance, particularly noticeable in unseen domain scenarios. This suggests that in practical applications, TFIA could effectively mitigate the unsatisfactory adaptability of current methods to OOD data. For more detailed information about the TFIA experiments, please refer to Appendix E.Visualizations of learned embeddings. To further verify our method's capability to differentiate various writing styles, we apply UMAP [45] for dimensionality reduction on text embeddings from the test set of the Deepfake Cross-domains & Cross-models subset. As shown in Figure 3 (a), using a pre-trained model directly fails to segregate embeddings of varying categories. In contrast, after fine-tuning with our method, UMAP unsupervised dimensionality reduction is already capable of clustering the features of various categories well, as shown in Figure 3  
conclusion: 
 In this paper, we propose DeTeCtive, a novel method for AI-generated text detection, anchored by a multi-task auxiliary multi-level contrastive learning framework. Through extensive experiments, our method demonstrates state-of-the-art performance on three popular benchmarks, validating the effectiveness of each component via ablation studies. We also uncover our method's Training-Free Incremental Adaptation (TFIA) capability, enriching its experimental analysis. We hope our work brings new insights and findings for the task of AI-generated text detection. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Experimental details are illustrated in section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Given the massive amount of experiments conducted in this paper, providing error bars would be computationally prohibitive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite and credit the code, data, models and vital libraries we used in the paper, such as implementation details in section 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this paper, we have not thoroughly explored our method's interpretability, which we believe is a promising research direction. Follow-up research works can analyze the differences and similarities between human-written text and AI-generated text based on our open-source model, and conduct token-level interpretability research. Also, we have not carried out training on a larger corpus. We believe that performing our method on a larger corpus could enhance the ability to identify writing styles, thereby improving the model's performance.The topic of this study is AI-generated text detection, a subject of significant importance for AI-safety.With the rapid development of AI technology, particularly in Natural Language Processing (NLP), the proliferation of AI-generated text raises concerns about global information security, as it may contribute to the spread of disinformation, false information, and content that has the potential to encourage harmful or destructive behaviors, making the detection and monitoring of AI-generated text a pressing issue. The method presented in this paper achieves state-of-the-art performance on several benchmarks. Particularly noteworthy is its superior performance in out-of-distribution (OOD) detection, far surpassing existing methods. These advancements offer promising prospects for the real-world application of AI-generated text detection algorithms. The methodological advancements in this research endeavor to facilitate the safe and ethical usage of AI technologies, consequently strengthening societal security.C.1 DeepfakeThe Deepfake [39] dataset collects human-written text from 10 domains. The AI-generated texts are produced by 27 LLMs and have been categorized into 7 model sets, as shown in Table 7 . These texts are generated by three types of prompts: continuation prompts, topical prompts, and specified prompts. Table 8 details the specific sources of the dataset and the splits of training, validation, and testing sets. The Deepfake dataset contains 6 different scenarios, carefully divided to ensure that the testing data used for any specific scenario would not be used as training data for other scenarios. These scenarios are categorized into: In-distribution detection and Out-of-distribution detection as follows.In-distribution detection. In-distribution detection scenario includes four subsets:• Domain-specific & Model-specific. Human-written texts come from a specific domain, and AI-generated texts are from a specific GPT-J-6B [64] model. There are 10 testbeds based on different domains.• Domain-specific & Cross-models. Human-written texts come from a specific domain, and AI-generated texts are from different models. There are 10 testbeds based on different domains.• Cross-domains & Model-specific. Human-written texts are from different domains, and AI-generated texts are from a specific model set. There are 7 testbeds based on different model sets.• Cross-domains & Cross-models. All models and domains are mixed to create a general subset.Out-of-distribution detection. Out-of-distribution detection scenario includes two subsets: The M4 [68] dataset is a large-scale dataset featuring multi-domain, multi-model, and multilingual characteristics, as shown in Table 9 and Table 10 . This dataset includes text from Wikipedia, WikiHow [34] , Reddit [19] , arXiv, and PeerRead [29] . Using human-written prompts, models like ChatGPT [6] , DaVinci-003 [6] , LLaMA [60] , FLAN-T5 [13] , Cohere [36] , Dolly-v2 [14] , and BLOOMz [49] generate text in 9 different languages, including English, Chinese, and Russian. [67] organizes a competition to detect AI-generated text based on M4, including tasks such as wholeparagraph detection and sentence-level detection. We use two scenarios designed for whole-paragraph detection: monolingual and multilingual. In monolingual scenario, the test set includes unseen AIgenerated texts from GPT-4 [1] , and are paraphrased by OUTFOX [33] , which increases the difficulty for detection. In multilingual scenario, the test set contains novel languages that have not appeared in either training set or validation set, and AI-generated texts are also paraphrased.The TuringBench [61] dataset provides a benchmark for systematically evaluating AI-generated text detection. The human-written texts come from news titles and contents from CNN, Washington Post, and Kaggle. Using these article titles, various LLMs, including the GPT series [6] , GROVER series [53] , CTRL [31] , XLM [37] , and XLNET [72] , generate articles similar to human-written text, resulting in 200K articles with 20 labels, detailed in Table 11 .The results of fine-tuning various text encoders using our approach are shown in Table 6 .Table 6 : Experimental results of applying our method to multiple text encoders on Cross-domains & Cross-models subset of the Deepfake [39] dataset. The best number is highlighted in bold, while the second best one is underlined. Here, we provide a detailed description of experiments conducted under the six scenarios proposed in Deepfake [39] dataset.In In-distribution detection, Longformer [2] is trained separately on all testbeds of each specific subset, with the final results averaged. FastText [4] , GLTR [22] , and DetectGPT [47] are directly tested on all testbeds of each subset based on statistical features. Our model is solely trained on the Cross-domains & Cross-models subset, while for the other three subsets, we use only the testbed training data from each subset as the database for inference without additional training. Notably, the Deepfake dataset is strictly divided to ensure that data used for a specific testing scenario is not used as training data in other scenarios.In Out-of-distribution (OOD) detection, Longformer is trained separately on all testbeds of each specific subset, with the final results averaged. FastText, GLTR, and DetectGPT are directly tested on all testbeds of each test scenario based on statistical features, also with the final results averaged. We train on the training set provided for each testbed, and use these data as the database for testing.The final results are obtained by averaging all the scores. Our method's detailed results for different testbeds in each scenario are shown in Table 13 .To validate the model's Training-Free Incremental Adaptation (TFIA) capability, we design the following experiments. Firstly, it is worth noting that the Deepfake dataset includes data from 10 domains and 7 model sets. In each OOD testbed, the training set excludes data from a specific domain or model set, while the test set consists of data from the domain or model set that is excluded in the training set. For example, in unseen models, there are 7 testbeds. For the first testbed of unseen models in Table 13 , the training set excludes data from the LLaMA series, while the test set is composed of data from the LLaMA model set. In OOD testing, the training data from the remaining six model sets is used as the database for testing. To validate the TFIA capability, we add training data from the LLaMA model set to the OOD database, noting that the added training data does not appear in the test set, ensuring that no testing data leakage occurred during our testing.Additionally, in Figure 2 , we further explore the TFIA capability by gradually adding unseen model or domain data into the database, analyzing the impact of the added data quantity on model performance.For the Unseen-Domains-XSum testbed of unseen domains in Table 13 , we gradually add XSumdomain training data at increments of 5%, 10%, and 15% until the training set data for that domain is exhausted, reaching a ratio of 100%.Table 7 : Models included in Deepfake [39] .OpenAI GPT [6] GPT-3.5-Turbo, Text-DaVinci-002, Text-DaVinci-003Meta LLaMA [60] LLaMA-13B, LLaMA-30B, LLaMA-65B, LLaMA-7BFacebook OPT [76] OPT-125M,OPT-350M, OPT-1.3B, OPT-IML-Max-1.3B, OPT-2.7BOPT-6.7B, OPT-13B, OPT-30B, OPT-IML-30BGoogle FLAN-T5 [13] FLAN-T5-Small,FLAN-T5-Base, FLAN-T5-Large FLAN-T5-XL,FLAN-T5-XXL BigScience BLOOM-7B [49] , T0-3B [54] , T0-11B EleutherAI GPT-J [64] , GPT-NeoX [3] Table 8 : The specific origins and splits of Deepfake [39] .Dataset CMV [57] Yelp [77] XSum [ The detailed results on six scenarios of Deepfake [39] dataset. The best number is highlighted in bold, while the second best one is underlined. In the table, the value of N/A indicates that we are unable to infer specific results based on the data from the Deepfake paper [39] . The notation "w/C&C database" represents the results combined with TFIA. Justification: The contributions and scope of the paper are summarized in the last paragraph of section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes to detect AI-generated text by formulating the problem as distinguishing writing styles of different authors. The proposed framework uses distinguishing writing styles, and leverages den information retrieval and training-free incremental adaptation mechanism to deal with new data and models. Experiment demonstrates its state-of-the-art performance, notably for OOD settings. Reviewers appreciate the contribution of this work because of its innovation in tackling OOD LM-generated text detection and its solid experimental comparisons. Rebuttal and discussions further addressed reviewer's concerns, and the consensus emerged. Please include the extra results and analysis on watermarking and authorship attribution in the final version.

Reviews: review: paper_summary: paper_summary: The paper discusses the challenges of current AI-generated text detection methods, which often suffer from performance issues and poor adaptability to new data and models. The authors introduce a new framework called DeTeCtive, which uses multi-level contrastive learning to distinguish different writing styles rather than just classifying text as human-written or AI-generated. This approach improves the effectiveness of various text encoders, achieving top results across multiple benchmarks, particularly in out-of-distribution scenarios.

DeTeCtive's framework includes a dense information retrieval pipeline and a Training-Free Incremental Adaptation (TFIA) mechanism, enhancing performance without extra training when encountering new data. The method fine-tunes text encoders using a new multi-task auxiliary, multi-level contrastive learning loss to capture detailed features of different writing styles.

Extensive experiments show that DeTeCtive surpasses existing methods in detecting AI-generated text and excels at handling data from unseen models and domains. The paper also emphasizes the method's compatibility with various text encoders and its strong performance in diverse scenarios, making a significant contribution to the field of AI-generated text detection and promoting the safe use of large language models.
strength_weakness: strength_weakness: While the paper shows strong performance on the chosen benchmarks, it remains unclear how well the model would perform on other datasets or in different application contexts. This limits the generalizability of the findings to some extent.
questions: questions: How does the quality and diversity of the training data affect the performance of DeTeCtive? Are there specific datasets that are more beneficial for training the model?
limitations: limitations: The focus on distinguishing writing styles might overlook content-based cues that could also indicate AI-generated text. This might lead to missed detections if an AI successfully mimics human writing style while generating misleading content.
paper_summary: paper_summary: This paper proposes to learn a text encoder with contrastive learning to cluster texts from different sources for fine-grained classification. A training-free incremental adaptation method is designed for detecting OOD data.
strength_weakness: strength_weakness: 1: The proposed multi-level contrastive loss shares some similarities with existing work [1] so it is not exciting enough.

2: Although the authors conduct extensive experiments on multiple datasets, only few of them are designed specifically for AI text detection (GLTR, DetectGPT). They should provide more comparisons with SOTA methods [2,3].

3:  As a work towards detecting AI-generated texts, more analysis of the features of texts should be provided. Otherwise, the framework could be applied to any binary classification problem (with hierarchical labels).

[1] Liu S, Liu X, Wang Y, et al. Does\textsc {DetectGPT} Fully Utilize Perturbation? Selective Perturbation on Model-Based Contrastive Learning Detector would be Better[J]. arXiv e-prints, 2024: arXiv: 2402.00263.

[2] Verma V, Fleisig E, Tomlin N, et al. Ghostbuster: Detecting text ghostwritten by large language models[J]. arXiv preprint arXiv:2305.15047, 2023.

[3] Chen Y, Kang H, Zhai V, et al. Gpt-sentinel: Distinguishing human and chatgpt generated content[J]. arXiv preprint arXiv:2305.07969, 2023.
questions: questions: 1: I am wondering if this framework would do a good job in authorship attribution [1]? 

2: For the classification approach in ablation studies, is the classification head trained with your own data or initialized in other ways?

3: Could you please provide some explanation on the advantage of multi-level contrastive loss over pcl? It is confusing to me that fine-grained clustering could help with binary classification.

[1] Uchendu A, Le T, Shu K, et al. Authorship attribution for neural text generation[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 8384-8395.
limitations: limitations: The authors mention they do not explore the robustness of the proposed method. The validation could be conducted under different attacks like paraphrasing, editing, prompting.
paper_summary: paper_summary: This paper presents DeTeCtive, a new algorithm for detecting AI-generated text. The key insight in this paper is that instead of treating AI-generated text detection as a binary classification problem, it should be treated as a multi-class "author style" classification problem. Using this insight, the authors develop a contrastive learning algorithm which learns style vectors for each LLM style, which are then stored in a style vector database for inference via KNN classification. The learnt style vector database can easily be expanded at inference to accomodate new text generated from potentially OOD LLMs.

The authors perform experiments on 3 existing datasets with outputs from 8 to 27 different LLMs, and show that DeTeCtive outperforms some existing methods. The authors complement their work with ablation studies on their loss function, and analysis experiments in OOD settings.
strength_weakness: strength_weakness: I had some concerns about the experiments in this paper.

1. **Baselines seem to be quite weak / over a year old**. Most of the baselines used are either pre-2022 methods or DetectGPT, which seems unsuitable for a cross-model dataset like the one used in evaluation. How do newer methods like Binoculars [5] perform on the same benchmark? Also, why not compare against watermarking algorithms like KGW [1], EXP-Edit [2], SemStamp [3], or [4]? Also how well do commercial tools like GPTZero perform on the same datasets (https://gptzero.me)?

2. **Limited experiments on newer GPT-4 class LLMs**. AI-generated text detection is the hardest on the most human-like LLMs, which are the GPT-4 class models. However, almost no experiments were done on GPT-4 class models (except for a small subset of the M4 dataset). I encourage the authors to stress test their OOD generalization to outputs from GPT-4 class models like Claude Opus, Gemini 1.5 and the newer variants of GPT-4.

3. **Limited emphasis on attack robustness**. There are almost no experiments in the paper (besides the mention of OUTFOX paraphrases) showing the robustness of the DeTeCtive algorithm to paraphrasing [6, 7], text mixing attacks [8], and translation attacks. This is critical to establish the robustness of the method compared to alternative detectors.

4. **It's not clear whether gains over baselines are coming from the SimCSE initialization or training**. Looking at the ablation studies in Table 3 and baselines in Table 1, all ablated variants outperform all baselines. This makes me question the need for the complexity in the DeTeCtive method's loss function, and how much of the gains can be attributed to the SimCSE initialization rather than the proposed DeTeCtive algorithm. An ablation study that would help here is using the DeTeCtive loss function on alternative encoders like RoBERTa / BERT / SentenceBERT.

[1] - https://arxiv.org/abs/2301.10226  
[2] - https://arxiv.org/abs/2307.15593  
[3] - https://arxiv.org/abs/2310.03991  
[4] - https://arxiv.org/abs/2305.08883  
[5] - https://arxiv.org/pdf/2401.12070  
[6] - https://arxiv.org/abs/2303.11156  
[7] - https://arxiv.org/abs/2303.13408  
[8] - https://openreview.net/pdf?id=DEJIDCmWOz

----

**After rebuttal**: I've decided to increase my score from 3 to 5 due to extra experiments on paraphrase robustness, and baseline comparisons against Binoculars.
questions: questions: 1. AI-generated text detectors typically need to operate in a low FPR setting, to minimize the risks of labeling innocent text as AI-generated. Given this, what are the Table 1 true positive rates at a low FPR of say 0-1%? TPR at low FPR ranges (0-1%) is a standard metric for evaluating AI-generated text detectors which has been used in many previous papers and blogs: https://arxiv.org/abs/2303.13408, https://openreview.net/pdf?id=DEJIDCmWOz, https://arxiv.org/pdf/2401.12070, https://foundation.mozilla.org/en/blog/who-wrote-that-evaluating-tools-to-detect-ai-generated-text/
limitations: limitations: The authors acknowledged weakness #3 in their limitations section. But I think this is a critical limitation and analysis on paraphrase / attack robustness is necessary in AI-generated text detection papers.

Comments: title: Follow-up Response
(Q2) Response to the after-rebuttal questions raised by Reviewer W7ZQ.
(Q1-part2) Response to the after-rebuttal questions raised by Reviewer W7ZQ.
(Q1-part1) Response to the after-rebuttal questions raised by Reviewer W7ZQ.
Response
Thank you for the detailed response and experiments, raising score to 5
Answer for the comment "Could you please provide some explanation on the advantage of multi-level contrastive loss over pcl?"
comment: Dear Reviewer,

We would like to express our sincere gratitude for your response and acknowledgment of our additional experiments. The issue you have raised gives us a new perspective and a direction for reflection. Upon careful consideration, we think that the issue you pointed out could serve as a promising direction for future research.  Once again, we appreciate your valuable comments.

Authors
I appreciate the detailed response from the authors. I totally understand the claimed novelty of this work. But I do not think the authors understand my point. As mentioned in the comments before, if this work is targeted at detecting AI-generated text, analysis of sequence patterns or special tokens should be provided to help the community gain insights about the AI-generated texts. Otherwise, I think this work is also applicable to and should be evaluated on datasets with similar label structures. I will increase my score a bit based on the additional experiments but I encourage the authors to conduct the analysis mentioned above.
Dear Reviewer W7ZQ,

Thanks again for helping review this paper! Since we are approaching the end of the author-reviewer discussion period, would you please check this author response regarding your concerns? We really appreciate it!

Best,
AC
Dear reviewer,

We kindly request your feedback on whether our response has addressed your questions. If you have any remaining questions or concerns, we are happy to address them. Thank you for your time and consideration!
Thank you for your response and acknowledgment of our supplementary experiments. We are currently conducting experiments on watermarking dataset, this experiment requires a certain amount of time because we need to conduct evaluations on watermarking specific dataset, which entails some preliminary data processing work. We will include the results in the final version of the supplementary material within our paper.
## Q2
Please allow me to clarify your misconception about our supplementary results for the  task of **authorship attribution detection**.  These corresponding experimental results are detailed in the section titled ***"Supplementary experiments on other applications. (Reviewer yy53, Reviewer W7ZQ)"***, which can be found at the top-level block of ***Author Rebuttal by Authors***. While Table2 of the attched PDF file shows the results of the additional baseline schemes on several existing datasets on the task of AI-generated detection, so the results are consistent with that in Table1 of the original submission. Due to the length constraint of the uploaded PDF file, we only list the additional results of authorship attribution detection in the rebuttal reply. We commit to update these results in the final version afterward.
***
Thank you again for your comments. We sincerely hope that our responses can address your questions.
* **The motivation and contributions of our research work.**
    1. **Motivation:** In response to the aforementioned weaknesses of existing works, the motivation of our study is to propose an AI-generated text detection algorithm that can be widely applied to various language models and scenarios. It could effectively adapt to newly released large language models and other unseen domains, demonstrating good generalization and robustness.
    2. **Contributions:** Our work contributes to the field of AI-generated text detection by proposing a novel method that can effectively detect AI-generated text. Based on our framework, we introduce Training-Free Incremental Adaptation (TFIA), a scheme to enhance the model's out-of-distribution (OOD) detection capabilities. We validate our method on multiple benchmarks, where it consistently outperforms existing solutions, especially in terms of OOD detection performance. The visualization of text embeddings also validates that our model captures multi-level text features, which aligns perfectly with our insights. Additionally, we supplement experimental results on attack robustness and authorship attribution detection, further demonstrating our method's superior generalization and robustness surpassing existing solutions. We believe such a novel framework, paired with comprehensive experimental evaluations and state-of-the-art performance, contributes to the community, fostering the application of our algorithm in real-world scenarios.

In conclusion, we believe that our approach offers a complementary perspective that enhances the performance of AI-generated text detection and also aligns well with our key insights. Moreover, we have also conducted corresponding analysis and explorations based on our method, including extensive experiments under various testing scenarios (e.g., multiple benchmarks, OOD detection, authorship attribution detection, attack robustness) and the Training-Free Incremental Adaptation (TFIA) capabilities. These solid experimental results and findings are contributions of our work to the research community. Finally, we are open to incorporate further discussion on interpretability of our method into the final version.
We sincerely appreciate your acknowledgment of our proposed method as an inspiring work and for the commendable performance of our framework. This recognition serves as an encouragement in our research endeavors. Next, we will address your after-rebuttal questions in detail.
## Q1
Thank you for your thoughtful and detailed feedback. We appreciate the points you've raised and would like to address your concerns directly from the following perspectives.
* **The research objective and insights of our work.**
    1. **Performance vs. Insights:** As stated in the Abstract and Introduction of the paper, our insight is that the key to accomplishing AI-generated text detection resides in distinguishing the multi-level writing styles of different "authors", rather than just modeling this task as a binary classification problem. Therefore, based on this key insight, we design the DeTeCtive framework. Specifically, we define multi-level positive and negative sample pairs according to the affinity relations between different language models and derive a multi-level contrastive learning loss function. Eventually, we accomplish the learning of multi-level features through contrastive learning. Moreover, under the DeTective framework, we do not focus on improving binary classification performance. Instead, within our framework, the model's learning objective is to capture multi-level features among different large language models. The principal objective function is the multi-level contrastive learning loss we designed, rather than a simple binary cross-entropy loss. Furthermore, our classification method is based on the K-Nearest Neighbors (KNN) clustering of embeddings. Therefore, the improvement in model performance results from successful learning of multi-level features rather than relying on the classification head. Hence, both the initial intent of framework design and the way of model learning reflect our insights. We believe that the end-to-end method we proposed, which outperforms baselines (e.g., SCL/PCL) and all the comparison methods (e.g., DetectGPT, GPT-Sentinal, CoCo) by a significant margin, is not just a trivial improvement. The substantial performance gains suggest that our method is capturing internal patterns and nuances that are specific to AI-generated text. These internal insights could offer valuable perspectives on how machines differentiate between human and AI-generated content effectively.
    2. **Human-Friendly Insights vs. Machine Efficiency:** As elaborated above, our understanding and insights of this task are reflected in the design of the method and framework. The visualization of learned text features also reveals that features at different levels (e.g., by model famliy, by individual model) could cluster well, which is something typical supervised contrastive learning (SCL) cannot achieve due to the lack of multi-level relationship constraints. We believe that the insights upheld by our paper, namely, that the key to accomplishing AI-generated text detection task resides in distinguishing the multi-level writing styles of different "authors", is intuitive and human-friendly. The effectiveness of our approach has also been proven through comprehensive experiments and the results are also consistent with our insights.
* **Weaknesses of existing works.**
    1. **Unsatisfactory generalizability:** The out-of-distribution (OOD) detection results shown in Table2 of our original submission, the performance of existing methods on OOD data is quite poor. This suggests that the handcrafted features and their insights discovered are challenging to generalize to OOD data, significantly hindering the practical application of the algorithm. Considering the rapid development of language models, this presents a problem.
    2. **Performance bottlenecks and lack of comprehensive evaluations:** Compared to our work, existing solutions lack a comprehensive evaluation on various benchmarks, testing scenarios and other applications. The experimental results of existing methods on multiple benchmarks are not ideal, indicating that there are performance bottlenecks. However, our method surpasses the existing state-of-the-art solutions by a large margin on each individual dataset.
Thank you for the detailed explanation and additional experiments. Here are some after-rebuttal questions:
* Q1

My major concern is that this work focuses too much on improving binary classification performance instead of providing insights about synthetic text, which deviates the research topic -- do you think SCL is targeted at detecting synthetic data even if it is a commonly used baseline (even a strong baseline) in this area?

Many existing works about AI-generated text detection discuss the differences between AI-generated and human texts, which deepens communities' understanding about AI-generated texts. For example, DetectGPT finds the difference in log probability between two categories. GPT-Sentinal traces integrated gradient to tokens to decide which tokens contribute to the classification. CoCo discovers the coherence structure difference in two categories -- these are examples from the related work introduced in this paper.

I agree that the visualization results prove the effectiveness of the learned encoder. However, I argue that this is a success in classification (I think SCL is also able to cluster the data embedding well) but not in explaining the intrinsic difference between AI-generated and human texts -- the research problem in this paper.

This is a methodologically inspiring work. I lean to reject this paper for the weak relation to the problem it claims to solve based on the above-mentioned reason. I am willing to increase my score if the authors address my concern.

* Q2

I have a question about the authorship attribution experiment. If I do not misunderstand, the author indicates the result is in Table 2 in the attached PDF, which is the same (except for the addition of Binoculars) in Table 1 in the original paper. I assume the results in table 1 in the original paper is for binary classification? But authorship attribution is a multi-class classification problem and I did not expect the same results between the two tables. Please tell me if I am wrong about anything.
Thank you for the detailed response and additional experiments! I've decided to increase my score from 3 to 5 due to extra experiments on paraphrase robustness, and baseline comparisons against Binoculars/GPTZero. I still encourage the authors to do add watermarking baselines in the paper, perhaps in the supplementary material.
Suppose we liken our task to appreciating paintings, needing to categorize them into "Impressionist" and "non-Impressionist" genres. If we focus solely on these two broad categories, we might overlook the stylistic differences between works by Monet and Renoir within the Impressionist category. Fine-grained clustering, however, is akin to not only distinguishing between Impressionist and non-Impressionist works but also to further differentiating the styles of various artists within the Impressionist genre itself. When we can differentiate between Monet's and Renoir's works, it means we have gained a deeper understanding of Impressionist pieces, thus making us more adept at distinguishing between the "Impressionist" and "non-Impressionist" categories. This more nuanced classification aids in better understanding and appreciating the artwork, enhancing the precision of our discernment. Similarly, in AI-generated text detection, incorporating multi-level contrastive learning allows the model to more precisely identify and differentiate texts from various sources. 

Our ablation study, as shown in Table 3 within the paper, also confirms the multi-level contrastive learning outperforms plain contrastive learning.