BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens
Authors: Hanxi Guo
Siyuan Cheng
Xiaolong Jin
ZHUO ZHANG
Kaiyuan Zhang
Guanhong Tao
Guangyu Shen
Xiangyu Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
AI-text Detection
Paraphrase
Trustworthy AI
Primary area: other
TL;DR: We propose BiScope, leveraging a novel bi-directional cross-entropy calculation method to detect AI-generated texts.

Abstract: Detecting text generated by Large Language Models (LLMs) is a pressing need in
 order to identify and prevent misuse of these powerful models in a wide range of
 applications, which have highly undesirable consequences such as misinformation
 and academic dishonesty. Given a piece of subject text, many existing detection
 methods work by measuring the difficulty of LLM predicting the next token in
 the text from their prefix. In this paper, we make a critical observation that
 how well the current token’s output logits memorizes the closely preceding input
 tokens also provides strong evidence. Therefore, we propose a novel bi-directional
 calculation method that measures the cross-entropy losses between an output
 logits and the ground-truth token (forward) and between the output logits and
 the immediately preceding input token (backward). A classifier is trained to
 make the final prediction based on the statistics of these losses. We evaluate our
 system, named BISCOPE, on texts generated by five latest commercial LLMs
 across five heterogeneous datasets, including both natural language and code.
 BISCOPE demonstrates superior detection accuracy and robustness compared to six
 existing baseline methods, exceeding the state-of-the-art non-commercial methods’
 detection accuracy by over 0.30 F1 score, achieving over 0.95 detection F1 score
 on average. It also outperforms the best commercial tool GPTZero that is based on
 a commercial LLM trained with an enormous volume of data. Code is available at https://github.com/MarkGHX/BiScope.

BISCOPE: Al-generated Text Detection by Checking Memorization of Preceding Tokens
Hanxi Guo Purdue University guo778@purdue. edu
Siyuan Cheng Purdue University cheng5350purdue..om edu
Xiaolong Jin Purdue University j in5090purdue edu
Zhuo Zhang Purdue University zhan3299@purdue edu
Kaiyuan Zhang
Guanhong Tao University of Utah guanhong tao@utah. edu
Purdue University zhan4057@purdue edu
Guangyu Shen Purdue University shen447@purddee .ddu
Xiangyu Zhang
Purdue University xyzhang@cs purdue. edu
Abstract
Detecting text generated by Large Language Models (LLMs) is pressing need order to identify and prevent misuse of these powerful models in wide range of applications, which have highly undesirable consequences such as misinformation and academic dishonesty. Given piece of subject text, many existing detection methods work by measuring the difficulty of LLM predicting the next token the text from their prefix. In this paper, we make a critical observation that how well the current token's output logits memorizes the closely preceding input tokens also provides strong evidence. Therefore, propose novel bi-directional calculation method that measures the cross-entropy losses between an output logits and the ground-truth token (forward) and between the output logits and the immediately preceding input token (backward). A classifier trained to make the final prediction based the statistics of these losses. We evaluate system, named BISCOPE, on texts generated by five latest commercial LLMs across five heterogeneous datasets, including both natural language and code. BISCOPE demonstrates superior detection accuracy and robustness compared to nine existing baseline methods, exceeding the state-of-the-art non-commercial methods' detection accuracy by over 0.30 Fl score, achieving over 0.95 detection Fl score on average. It also outperforms the best commercial tool GPTZero that is based on a commercial LLM trained with an enormous volume of data. Code is available https://gi //gbthb...tttt thub. Saa//MrrKKK/BBBSopp
Introduction
Given the superior performance of Large Language Models (LLMs) in understanding and generating text, they have become an integral part of human society, assisting people with daily activities such summarizing articles, polishing emails, and more. However, the widespread use of LLMs also raises concerns about the misuse of Al-generated text. For instance, students and academics may utilize LLMs to produce content for their assignments and research [10, 37, 30], compromising academic integrity. Adversarial individuals could leverage LLMs to efficiently create inflammatory and fraudulent content on social media [22]. Additionally, the development of LLMs themselves faces challenges related to the quality of existing datasets, which may be compromised by the significant
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

inclusion of Al- Al-generated text [49, 27]. All of these issues underscore the urgent need to distinguish Al-generated text from human-writte text [5, 14, 18, 50, 3, 12].
Despite this urgency, current Al-generated text detection techniques fall short as LLMs become increasingly diverse and advanced. Our experiments demonstrate that most of the existing approaches cannot achieve an Fl score exceeding 80% on the Yelp dataset [32] when using the latest LLMs (e.g.. Claude-3-Opus) to generate content. A close examination of these approaches reveals inherent limitations by design. Specifically, there are three kinds of methods that do not need pre-training or additional information on the LLMs that generate the data (e.g., watermarking [7, 19, 44, 46, 20]). The first kind [5] directly prompts another LLM or NLP model to classify whether the subject text is AI Al-generated. While intuitive, the inevitable model hallucination [55, 26] consistently prevents from achieving high accuracy. The second kind of methods [28, 31, 9, 54] examines the linguistic features of the subject text, which are increasingly susceptible to deception as L.LMs become more sophisticated and human-like in their responses. The third approach [11, 41, 17, 40, 16, 32, 35, 4, 53, 48] feeds partial or entire text to surrogate model and checks how well the output text aligns with the surrogate model's I preference via various metrics or downstream classifiers. While this method outperforms the first two approaches, it only examines the next token information in the output logits, representing just part of model behaviors, thereby naturally limiting its performance.
In this work, we explore the potential of leveraging internal model states to detect Al-generated text. Like existing methods, we hypothesize that since LLMs are trained on vast corpora of data from the Internet. their training data likely exhibit sigmificant similarities, leading to similar behaviors across models. Therefore, we use surrogate model to approximate the behaviors of the one used to generate the subject text. We also make a critical observation that, in causal language models (e.g., GPT), the current token's 's output logits encode information about both the next token (i.e., prediction) and its preceding input tokens (i.e., memorization), indicating a bidirectional relationship between the output logits and the input text. Specifically, when these causal language models encounter human data, they tend to memorize more preceding token information while predicting less next token information in their output logits.
To reveal this relationship, we calculate two kinds of cross-entropy losses by feeding different portions of the subject text into the surrogate model. One is the forward information, calculated as the cross-entropy loss between the output logits and the expected next token the subject text. The other is the backward information, calculated as the cross-entropy loss between the output logits and the most preceding input token. We then train a binary classifier on the collected statistical loss features to make the final prediction. We also introduce several novel improvements in the prototype, such as providing a summary of the subject text to better guide the surrogate model, thereby enhancing its practical effectiveness and robustness, and using parallel model inference to enhance efficiency. As such, we propose BISCOPE, an effective and efficient Al-generated text detector by harnessing both the prediction and memorization features of the LLM through its output logits.
Our contributions are as follows:
We propose a novel Al-generated text detection algorithm that exploits both the preceding token information (i.e., memorization) and the next token information (i.e., prediction) via an innovative bi-directional cross- cross-enroppy loss calculation method. Additionally, we are the first to utilize text summaries to guide the detection, further enhancing its effectiveness and robustness toward heterogeneous data.
We extend existing datasets and craft a large-scale public dataset for more challenging Al-generated texts, consisting of 25 distinct groups and more than 22, 000 samples. The dataset is sourced from five different text domains (both natural language and code) and generated by the five latest commercial LLMs. This dataset presents more challenging scenarios compared to existing datasets, which are typically sourced from sourcs LLMs with fewer parameters and capabilities. We also craft a paraphrased version of our dataset.
We develop prototype named BISCOPE, a detection pipeline without any fine-tuning needed for the detection LLM. We evaluate it on our dataset and compare it with nine state- of-the-art baseline techniques. Our results show that BISCOPE can achieve an average F1 score of over 0.95, taking less than 200 milliseconds to detect a sample (when the summary procedure is disabled), while the baseline techniques achieve only 0. 0.70--.85 F1 scores and take up to 27 seconds per sample. BISCOPE also outperforms the best commercial tool,

---

GPTZero, in 72% of cases. Additionally, we conduct a comprehensive ablation study to verify the effectiveness and robustness of each component of BISCOPE.
2 Background and Related Work
In addition to various watermarking techniques [7, 19, 44, 46, 20, 24, 15, 52] that require fine-tuning or additional information about the LLMs generating the text, several efforts have been directed towards the detection of Al-generated texts with minimal prior knowledge about the generative models. These efforts broadly fall into three categories. As stated in $ I, the first two categories perform worse than the last category, hence we mainly focus on the methods in the third category that use surrogate LLM in this paper. The methods in the third category can be further divided into two types: statistical methods and training- based methods.
These techniques [45, 34, 21, 33] primarily utilize pre-trained LLMs to simulate the generation process of the target generative AI, analyzing the statistical differences between AI- generated texts and human-written ones. These methods commonly serve as zero-shot approaches, assigning scores to indicate the probability of texts being Al-generated. For example, Zero-shot Query [32. 48] prompts pre-trained LLM to score the input text. LogRank [11, 35] calculates the average probability rank of each token in a text processed through a pre-ttrinee LLM, where higher ranks suggest the text is Al-generated. LRR [41] improves on LogRank by incorporating token confidence. DetectGPT [35] involves masking parts of the text to see how an LLM reconstructs them, and Raidar [32] employs the LLM to rewrite the text. Both methods assume that Al-generated texts are more likely to be preserved accurately in the process. Binoculars [13] analyzes the cross-entropy between the output logits from two surrogate models with different fine-tuning configurations.
Statistical Methods.
Training- based Methods. This type includes methods train an NLP model to distinguish between Al-generated and human- texts. For example, OpenAI [40] uses RoBERTa-baas based model for training an Al-text classifier. Such methods can be susceptible to adversarial attacks or paraphrasing. The state-of-the-an technique RADAR [16] leverages adversarial training to improve the robustness of the classifier. There are also commercial services for the detection of Al-generated texts. For example, GPTZero [43] employs multi-step statistical detection process and utilizes pre-trained commercial L.M to deliver the prediction.
Our approach, BISCOPE, is a statistical method that uniquely incorporates meticulous statistical feature extraction via a bi-directional calculation method to ensure its general effectiveness and robustness against paraphrasing across five text domains and five of the latest commercial LLMs. To evaluate BISCOPE, we compare it with nine existing detection methods, including Zero-shot Query [48, 32]. LogRank [11], LRR [41], DetectGPT [35], RADAR [16]. Raidar [32], OpenAl Detector [40], Binoculars [13], and GhostBuster [48], as well as with the most renowned commercial detection API, GPTZero. We surpass these baseline methods in both effectiveness and efficiency.
Methodology of BISCOPE
3.1 Design Motivation
Although existing Al-generated text detection methods have been proven to be robust against texts generated by various open-source LLMs, their performance degrades in more complex and real-world scenarios, especially when dealing with the latest commercial LLMs and heterogeneous text genres, The degradation can be attributed to the following two reasons: feature insufficiency and contextual heterogeneity.
Feature Insufticiency. Existing methods focus on analyzing the difficulty a surrogate LLM experi- ences in predicting the next token given the preceding input text. For example, these methods use the rank or the probability of the next token as a metric or compare the discrepancy between the input text and the surrogate model's generation. Figure ((a) presents the detection F1 score of a toy example that uses the average next token rank from Llama-2-7B as the feature and a random forest model as the classification model on both human text (in blue) and GPT-4-Turbo's 's text (in orange). The detection F1 score only reaches 0.55, which just slightly better than random guesses. This indicates the lack of clear separation using only next token ranks. One may argue that the random forest may
3

---

Proceding Tokens
Next
(4) Next Token Rank, F5
(b) Last Token Rank, P1-0.73
(c) Both Ronk, F1-0.78
Token
Human Text
Embedding
Plan
Count
Transformer
1000
2000
5000
1000
States
(4) Nest TToken CIF, P1-0.63
(e) I ast Tokon CF. F1-1181
Rank
CEI Loss
( Both CE. F1-0
Output Logits
Inference
Reetoct
40
Human Text Al Text
Courx 20
16
Contain
Proceding
Token
Next
-0-
Information
Informstion
Existing
Cross Entropy Loss
Cross Entropy Loss
Next Token CE Loss
Method
Methods
Figure 1: Comparison of detection F1 scores when utilizing the rank Figure 2: Comparison and cross-entropy loss regarding next token, preceding token or both. of output logits informa- tion utilization.
The surrogate detection model is Llama-2-7B.
not be powerful enough. However, we will show later that using additional features proposed in the paper, the same random forest configuration could achieve much better results.
We observe that the internals of the surrogate LLM when used to predict the subject text have much richer information that be used. Figure 2 illustrates how LLM encodes information. The arrows and texts in green illustrate the information related to the next token, while the arrows and texts in gray show the information related to the preceding tokens. In the auto-regressive generation mode, the LLM receives the tokens preceding to the current position as the input and outputs the logits that contains its prediction for the next token. During this procedure, its internal states encode the preceding tokens (i.e., memorization) [47] while implicitly "planning" for the next token [51], namely, as observed by researchers in [51], J the internal states show similarities to the encodings of future tokens. The output logits, which can be considered as a reduced representation of the model's s internal states, also contains both the information to predict the next token and the information of preceding tokens. Existing methods focus only on the former by comparing the output logits with the expected next token. this paper, we propose to consider the preceding token information as well. In particular, we hypothesize the following: for human-written text, the surrogate LLM has poor prediction for the next token and strong memory of the previous token, reflected in the output logits, whereas the behaviors for LLM-generated text are the opposite. Intuitively, it's like when we humans are unsure of what to say next, and the last word tends to stay in our minds.
To validate our hypothesis, we conducted an experiment in which we compared the current output logits with the preceding token for a piece of given text, leveraging the same random forest as before. Figure 1(b) and (c) present the detection F1 score when only using the preceding token's 's rank and when using both the preceding and next tokens' ranks (to distinguish human and LLM texts), reaching 0.73 and 0.78 Fl scores, respectively, denoting 0.2 F1 score improvement compared to using the next token information alone. Observe that (d) for next token prediction, the Al texts (in orange) tend to have smaller CE loss than human texts (in blue), indicating the L.MM has better prediction for the AI texts. In (c) for previous token memorization, the human texts tend to have a smaller loss than the AI texts, indicating the LLM has poorer memory for AI texts. Figures (a) and (b) and an additional example in Appendix show a similar trend. These support our hypothesis.
Thus, in BISCOPE, we design a novel bi-directional cross-entropy loss computation method that computes the cross-entropy losses between the output logits and the expected next token, and between the output logits and the preceding token. Figure (d)-(ff) illustrate the Fl scores when using the cross-entropy losses for next token, previous token, and both. Observe they achieve better results compared to using plain ranks, due to the more wealthy information encoded. An additional example using GPT-Neo-22785 in Appendix shows a similar trend.
Contextual Heterogeneity. In addition to insufficient feature utilization, we also observe that contextual heterogeneity significantly influences detection accuracy. Existing methods directly use surrogate LLMs to generate the given text in auto-regressive manner, without incorporating any

---

! Completion Prompt Gemeration
4
Loss Computation In Text Completion
Statistical Feature Extraction
Feature
Classification
macan män
Write tille for the text:
Bi Di-directional Cross- Entropy
Binary
Classification
Humman
Given the summary:
Losses Computation
Multi-Point Splinting
mncun man min
Text
E
Complete the text:
Inpet Text
Text Bi segmentaatiion
FCE
Cross Entrog
Losses
BXSX min
sid
Peature Concatonation
'TeNt
AI
sad
Figure 3: Overview of BISCOPE. Arrows and texts in brown indicate text summarization.
additional information the context (for the text). As such, given a prefix part of the text, the LLM may have a diverse set of possible completions, limiting the ability to separate human and LLM texts.
To alleviate this problem, we formalize our detection LLM to first summarize the entire input text. These text summaries are then used to guide the completion, providing complementary contextual information and making the features more robust.
guided completion task, using a surrogate
3.2 Overview of BISCOPE
The entire workflow of BISCOPE can be summarized in four key steps, shown in Figure 3.
Step D: Completion Prompt Generation. In the first step, initialize the detection a guided text completion task. We use a surrogate LLM to summarize the input text and generate a text summary guidance. We then divide the input text into segments. The first segment, along with a completion request, is utilized to construct a text completion request. The text summary guidance and the text completion request form completion prompt. Details presented in 3.3.
Step 2 Loss Computation In Text Completion. Given the completion prompt and the second segment of the input text from Step !. we then calculate our novel bi-directional cross-entropy losses for the tokens in the second text segment using multiple open-source LLMs in parallel. Details presented in 3.4. The use of multiple LLMs is to reduce the uncertainty.
Statistical Feature Extraction. We vary the separation of the two segments at different positions of the subject text (e.g., one-fourth. half, and three-fourth of the whole length). For each setup, we collect the statisties of the bi-directional cross-entropy loss values. The statistics are concatenated to form a feature vector. More details and justifications are presented in $ 3.5.
Step 3:
Step Feature Classification. In the final step, use the concatenated feature vector train a binary classifier, which determines whether the input text is human-generated or Al-generated. Further details are presented in $ 3.6.
3.3 Completion Prompt Generation
In BISCOPE, we calculate the bi-directional cross-entropy losses within guided text completion scenario. This scenario involves providing text summary guidance and short sub-string of the input text to force LLMs to generate the remainder of the text. Specifically, to alleviate the impact of contextual heterogeneity during text generation, we first utilize a surrogate LLM to summarize the entire input text and obtain summary as guidance. We then divide the input text into two segments (e.g., the first 10% and the remaining 90%). The first segment, referred to Input Text Segment serves as the sub-string in a text completion request, while the second segment, referred to as Input Text Segment 2, is used as the completion ground-truth request after the summary guidance, we construct a completion prompt shown as follows:
3.4. By appending the text completion
Given the summary:
(Text Summary Guidance) Complete the following text: (Input Text Segment 1}
5

---

The text in black indicates the text completion request, while the text in brown indicates the guidance, which is summarized using the following prompt:
Write title for this text: (Input Text}
The summary contains the aggregated contextual information of the entire text, providing comple- mentary guidance to the LLM completion, in addition to the first segment. To balance the detection accuracy and efficiency, BISCOPE can also disable this text summary procedure to achieve faster Al-generated text detection with satisfactory accuracy.
3.4 Loss Computation In Text Completion
After crafting the completion prompt, then feed it into multiple open-source LLMs in parallel to obtain multiple output logits that correspond to the Input Text Segment from $ 3.3 using the teacher forcing pattern [25], which feeds the ground-truth token prefixes (Input Text Segment 2) to compute the output logits at each token position. These output logits can be used to measure how likely the LLMs predict the next token given its prefix in the subject text and how well the LLMs memorize the preceding token, according to our discussion bi-directional cross-entropy calculation method in BISCOPE, which consists of both forward and backward cross-entropy calculations. The forward cross-entropy (FCE) calculation is identical to the commonly used cross-entropy in most LLM training processes, utilizing the output logits and the next ground-truth token to capture the output logits' next-token-relate information. In contrast, the backward cross-entropy (BCE) is calculated between the output logits and the immediate preceding input token, capturing how much the logits memorizes the preceding token. The detailed FCE and BCE calculations token position with LLM M are shown Equation
3.1. We hence propose a
[VI FCE; = Σ PIL log(P;), BCE; = Σ P: (og(PF)
IM
where V indicates the vocabulary of the LLM, P; denotes the soft-maxed output logits from M at position of the generated text (when given the preceding tokens from the completion prompt). Pa indicates the ground-tre -trth token encoding at the same position.
3.5 Statistical Feature Extraction
The aforementioned bi-directional cross-entropy loss values may have different characteristics when the text is partitioned at different positions. Intuitively, when the text is partitioned at a ratio of 1:9, meaning that we use the first 10% of the text to perform the completion, there tends to be lot more uncertainty compared to a partition of 9: A naive design is to fix a partition ratio. However, finding the most effective partition is difficult. Another design is to use the loss values computed at all positions. However, it can hardly deal with length variations of input texts. Therefore, our design to partition the whole text into segments (n 10 in our implementation).. For each segment, collect the bidirectional loss value statistics over all the positions with the segment, including the mean, maximum, minimum, and standard deviation values. This allows to align texts of various lengths and leverage the later classification to figure out the best partition positions (through learning). Additionally, this multi-segment analysis requires only a one-time inference of the input text, as the loss calculation at each position is independent of the others via teacher forcing. This allows BISCOPE to obtain various and sufficient features with high efficiency.
3.6 Feature Classification
In the final step, we concatenate all the statistical features of both the FCE and BCE vectors from all the detection L.LMs into a one-dimensional feature vector, which is then used to train binary classifier to perform the classification. Due to the generality of these features, the binary classifier can be directly used to detect unseen data, whether from unknown LLMs unfamiliar text domains.

---

RiScopo RiSeapee
Log Rant FJR
DerOCIP RATAR
Rodarr
Bisscaao Ghodllasce
(x) CPPP- T-Taub
1.0
By Cirr----Tute
1.0
Clasee C-oste---Oip
1.0
Ir Cenon
E0.
Eo.s
1..5
E5.5
0.5
0.5
1.0
0.5
300
0
Figure 4: ROC curves of BISCOPE and all the baselines.
4 Evaluation Results
4.1 Experimental Setup
We use five datasets in our evaluation, including two short natural language datasets (Arxiv [32] and Yelp [32], two long natural language datasets (Creative [48] and Essay [48]), and one code dataset [8]. For all datasets, we reuse their human-genann generated data and craft Al-generated text using five of the latest commercial LLMs. More details are presented Appendix C. To the best of our knowledge, we are the first to craft a comprehensive Al-generated text dataset using five of the latest commercial LLMs from three leading corporations. For the metrics, we use two metries in our evaluation. To assess the effectiveness of the detection, we use 5-fold pross-validation F1 score. To evaluate the efficiency of the detection, we use the time cost per sample as the metric. We present more cyyper-parametee settings in Appendix A.
4.2 Detection Performance Comparison with Existing Baselines
We first evaluate BISCOPE (with and without the text summary guidance) and nine detection methods on the normal dataset under both in-distribution and out-of-distribution (OOD) settings. Under the in-distribution setting, the test data and training data are from the same source. While under the OOD setting, the test data is from unknown source. The results are presented in Table 1. Our BiSCOPE outperforms existing Al Al-generated text detection methods, achieving 0.25 average detection Fl score increase on the normal dataset under in-distribution setting. In the OOD setting, our BISCOPE still surpasses existing detection methods with over a 0.16 average detection Fl score increase. Detailed analysis is shown as follows.
In-Distribution Results. For the in-distribution setting, we report a 5-fold cross- validation Fl score using one piece of human data and one piece of Al-generated data from the five latest commercial LLMs, as shown in Table 1. BISCOPE iidicatess our method with text summary guidance, while BISCOPE represents
method without it. Our method outperforms all nine baselines across all five datasets generated by the latest L.Mss. Specifically, our method achieves a 0.29 average F1 score increase the two short natural language datasets, where existing methods' best F1 score is around 0.90, while BISCOPE reaches over 0.95 in most cases. On the two long natural language datasets, existing methods reach up 0.97, while BISCOPE achieves over 0.99 in all cases, resulting in 0.23 average improvement. On the code dataset, existing methods achieve a 0.60-0.70 average, whereas BISCOPE reaches 0.84, achieving 01.11 average increase. We further present the TPR-FPR (ROC) curves of BISCOPE and 8 basclines in Figure 4 on the Yelp dataset. We observe that our BISCOPE reaches over 0.8 detection TPR on average when the FPR is only 0.01, outperforming all the baselines on all the five generative models' data.
Nut-of-Distributian Results. We use two -ii--i---ittittiinn (OOD) evaluation settings on the normal dataset, cross-model (CM) and cross-dataset (CD), based on previous studies [48, 32]. In CM setting, we assess detection performance on Al-generated data from unknown LLMs within the same text domain by training classifiers on human data and one piece of Al-generated data, then testing on four others from the same dataset. In CD setting, we evaluate detection transferability across text domains by training on data from one dataset and testing on data from four others, all generated by same L.M. Results across five LLMs are reported Table 1. In CM setting, the highest baseline F1 score is 0.84 (average 0.77), while our BISCOPE achieves 0.93, showing 0.16 average increase. This indicates superior generality of BISCOPE in detecting unseen I.M --geeeertted texts. In the CD setting, BISCOPE's average Fl score is 0.05 lower than RADAR that utilizes model
7

---

Table 1: Detection performance of BISCOPE and nine baselines on both normal and para- phrased datasets with in-distribution and out-of-distribution (OOD) settings.
Normal Datases
Paraphrasod Danaser
Normal
Normal
Paraphrasod
Method
GPT-3.5
GPT-4
Clande
Clande3
Gemini
3.5
GPT-4
Claude
Cludde
OOD
OOD
OOD
Turbo
Turbo
Sonset
Opus
1.0:pco
Turbo
Turbo
Sonnet
Opus
Avg.CMM
Avg-CD
Avg.
7oro-shir Query
0:5768
0.5835
0.6764
0.6667
0.6666
0.5587
0.6116
0.6916
0.6935
Log Rosk
0.6572
0.7006
0,8015
0,8809
0.8560
0.6628
0.6660
0.6634
0.6747
0.6913
0.6219
0.3655
1.R
0.6602
0.7031
08116
0.8596
0.854
0.6654
0.6654
0.6654
0.6654
0.7319
0.6130
0.2353
Anil
DeteeSiPP
0.6654
0.6634
0.6678
0.6673
0.6673
0.6641
0.6628
0.6654
0.6654
0.6642
0.7091
0.6522
KADAR
0.9566
0.7858
0.7034
0.7754
0.7868
0.9203
0.6970
0/6884
0.7202
0.7404
0.8035
0.7388
Raidar
0.8316
0,8029
0.8389
0.7366
0.9044
0.885]
0.5052
0.8303
0.6964
0.7270
OpenAT Detector
0.7880
0,6660
0.6673
0.6673
0.6976
0.7062
0.6654
0.6673
0.6673
0.6349
0.6569
0.706
Binoculars
0.9097
0.9135
0.9256
0.9699
0,9560
0,6617
0.697
0.8112
0,8672
0.9163
0.8199
0.5835
GhosiBuster
0.9716
0.9886
0,9815
0.9813
0,9571
0.9700
0.994
0.9814
0.9856
0.9187
0,6811
0.9672
DISCOPE BISCOPE*
0.9870
0,9928
0.9796
0.9885
0,9708
0,9709
0.9800
0.9625
0,9870
0.9517
0.7131
0.8534
0.9928
0.9943
0.9669
0.9913
0,9797
0.9570
0,9859
0.959
0.9884
0.9775
0.7767
0.8740
Zero-shot Query
0.0020
0,0010
00110
0.0168
0.0080
0,0000
0.0009
0.0188
0.0188
Log Ronk
0.6776
0.7120
0.6966
0.6430
0.660
0.6745
0.6743
0.6574
0.6695
0.6288
LRK
0.6671
0.6078
0,673
0.6678
0,655
0.6681
0.6662
0.6674
0.6666
0.6589
0.6615
0,6588
DescestPe
0.0945
0.6737
0,7252
077477
0,6020
0.6702
0.6609
0.7009
0.7106
0.6738
0.7187
0
RADAR
0.7618
0,7050
0,7310
0.5500
0,7497
0.7485
0,7030
0.7111
0.7345
0.7148
0,7370
037033
Raidar
0.9023
0,8385
0.1180
0.8876
0,8915
0.8948
0.134
0.1144
0:912
0.8572
0.6850
0.7817
OpenAA Desector
0.7286
0
0.0668
0.6616
0,728
0:7280
0,6668
0.0088
0.0008
0.8348
0.6208
0,6633
Binoculars
0.7295
0.6665
0.7585
09260
0.6885
0,6683
0,6655
0.6008
0.7254
0.6930
0.847
0.6681
GihostHuster
0.8193
0.8369
08746
03644
0.8025
0,8174
033649
0.2271
0:9145
0.7975
0,5859
0.5452
BISCOPE
0.9023
0.9405
0.0652
0.532
0. 0.9456
0.9064
0.9475
0.9814
0.9789
0.9063
0.8608
0,9523
Ri B
0.9010
0.9452
0.9658
0.9570
0.9545
0.9102
0,9530
0.983
0.9757
0.9128
0,8455
0.9505
Zero-shot Query
0.2730
0.1502
0.2691
0.3186
0.2719
0.2398
0.1694
0.1897
0.2948
Ranks
0.9673
0.7341
08779
0.9269
0,8044
0.7673
0.6685
0.6823
0.7701
0.7993
0,5824
0,5213
Ciaaiive
LRR
0.9512
0,6732
0,8062
0.8881
0,7208
0.6638
0,6662
0,6649
0.6662
0.7242
0,5772
0,3846
Descestarr
RADAR
0,8305
0.9543
0,8509
0.2050
09151
0,9922
0.345
08166
0,7500
03332
0.9298
0.6850
0,6715
03744
0.7364
0.9100
0.7006
0.9145
0.8924
037699
0.5415
035933
0.6299
Raidar
0,8933
0.8303
0.481
03661
0,838
0.8271
082117
0.8120
0.798
0.8151
0.7580
0.5621 15621
OpesAA Deteetor
0.6666
0.6071
0,6669
0,6771
06277
0.6071
0.667]
0.6911
0/6771
0.6108
0.6208
0.5475
Binoculars
0.9945
0.9681
09814
09866
0,9690
0.9627
02388
0.9348
0.9540
0.9711
0.7599
0.5738 1
CihostRuster
0.9965
0.9821
0.884
03834
0.9530
0.9861
007766
0.9671
0.9865
0.9501
0.336
0,5012
BISCom:
0.9985
0,9550
0.9960
0.9500
0.9964
0,9055
0.9944
0.9955
0,9540
0
0,7980
0.9707
BiScorE
0.9975
0.9955
0.9955
0.9945
0.9970
0.9955
0.9955
0.9950
0.9945
0.9780
0.8154
0.9513
Zero-shor Query
0.0156
0.0098
0.0175
0.0059
0,0322
0,0078
0.0214
0,0423
0,0078
Log Rank
0.9936
0.9065
0.9771
0.9811
0,9774
0.9001
0,7067
0.8170
0.9113
0.9025
0,4518
0.6074
LRK
0.9945
0.8416
0.673
0.826
0,9685
0.8121
0.6658
0.7243
0.8546
0
0.4724
0,5295
DescetGPT
0.9341
0,8709
0.3302
08178
0,9311
0.7910
0.7622
0,8609
0.8429
0.9140
0.5769
0,7513
RADAR
0.9812
0.8978
0.9648
0.555
0,9650
0.9509
0.8211
0.9471
0.9118
0.9356
0,7665
0,5833
Raidar
0.9786
0.9424
0.6622
0.9100
0,9574
0.9448
0.1886
0.9166
0.9216
0.9416
0.7136
0,3000
OpenAl Desectio
0.7069
0.6661
0.6699
0,616
0.6669
0.6662
0.6667
0.6661
0.6611
0,5300
Binocular
0.9995
0.9970
0.9955
0.960
0,9978
0.9920
0.9607
0.9787
0.9955
0.9967
0.7383
0.9429
GhostBuster
0.9995
0.9550
0.9960
0.9965
0.9967
0.9916
0.986]
0.9830
0.9920
0.9904
0.7740
0.9435
BiSCorE:
1,0000
0.9990
0.9985
0.970
0.9994
0.9965
0.9990
0.9990
0,9580
0.9946
0.3455
0.9436
BiScome
1,0000
0.9990
0.9985
0.9975
0,3589
0.9975
0.9990
0.5585
0.9985
0.5914
0,5609
0.9299
Zem-shor Query
0.6300
0.5833
0.4351
0.3524
0.1854
0.6600
0.6784
0.6400
0:4545
Rank
0.6581
0.6610
0.6611
0.6569
0.6583
0.6612
0.661]
0.6556
0.658L
0.6521
0,5306
0,5539
LRR
0.6639
0.6639
0.6639
0.6639
0.6542
0.6639
0.6639
0.6639
0.6639
0.6475
0.659
DeseGGT
0.6361
0.6474
0,6583
0.6612
0.6683
0.6612
0.6670
0.6630
0.6612
0.6445
0.5976
0.682
RADAR
0.6680
0.6653
0.6652
0.6597
0.6626
0.6598
0.6653
0.7322
0.6653
0.6652
0.8114
0.6660
Raidar
0.9368
0.6220
0.121
0.616
0.4833
0.9325
0.874
0.8250
0.6197
0.8878
0,1378
0.6521
OpenAT Tittectoe
0.642
0.6666
0.7514
0.6630
0.6630
0.6695
0.5767
Binoculars
0.7073
0.6512
0.6612
0.6653
0.6624
0.7101
0.6338
0.8011
0.7179
0.6273
0.7181
0.6771
ChusiBuster
0.8524
0.7942
0.6556
0.679
0,3860
0.8662
0,7729
0.7757
0.5390
0.6230
0,5091
0.6790
BISCOPE
0.9665
0,9655
0,8528
0.6069
0.7809
0,9659
0.9464
0.9691
0.9250
0.7914
0.5895
0.3999
DISCOPE*
0.9692
0.9536
0.8526
0.6620
0.774]
0.9597
0.9135
0.9600
0.9222
0.7898
0,5855
0.9024
pre-trained multiple datasets. However, compared to the other four baselines, BISCOPE still shows a 0.12 average improvement. Detailed OOD results are in Appendix D.
4.3 Robustness against Intentional Paraphrasing
Existing studies [23, 39] show that intentional paraphrasing can effectively evade Al Al-generated text detection. To verify the robustness of BISCOPE against paraphrasing, we utilize five of the latest commercial LLMs to paraphrase their own data using the paraphrasing prompt from previous work [16], and compare BISCOPE with nine baselines on this paraphrased dataset under both in- distribution and oft-o--distribution settings, as shown in Table 1. Under the in-distribution setting, we test all the methods using the same configuration as on the normal dataset, exploring whether paraphrasing reduces the discriminative power of existing methods. BISCOPE outperforms existing
8

---

GPTZero
BiSeape
BiScope"
(a) 5-Tubo
3-Soonee
(A) Clande----pus
Giemin-1 d-PPoo
Code
Code
Code
Code
Code
Andi
Anxi
Anxi
Andi
Anev
Essay
Essay
Essay
Essay
Essay
0.
0.2
0.2
Yelp
Yelp
Yelp
Yelp
Yelp
Creatve
Creative
Crealive
Creative
Crestive
Figure 5: Comparison with GPTZero on five datasets using five latest commercial generative AI models. The metric used in the figure is the detection F1 score.
Broculars BiSaspe"
Only
Ø Only
BCEAFCE
LunRank LRK
RADAR
BiSayr"
Time
BiScrpe
F1
Raseer
UpenAl Detector
Tige (o) Sygugle
1.21
8.21
1.25
1.0
20
15.95
20
Dosction
0.4
0.32
0.04
10
0.0
0.05
0.00
Genoma 78
Genoni
Mannat77
1.30
Liano 2:
Eisomble
Yelp
Creative
Essay
Code
Figure 6: Comparison of time costs for processing a single sample between BISCOPE and nine baselines.
Figure 7: Comparison of BiS- Figure 8: Comparison of the COPE'S effectiveness and ef- contributions of FCE and BCE ficiency when using different to detection effectiveness in open-source LLMs.
BISCOPE.
baselines with a 0.29 average Fl score increase. Additionally, existing baselines experience an overall 0.03 F1 detection our BISCOPE performs even better on the paraphrased dataset, with an overall 0.02 average F1 score increase. Under the the normal dataset, while testing them on the paraphrased dataset, exploring the generality of the detection against unseen paraphrased data. Our BISCOPE outperforms existing baselines with average ()299 F1 detection score increase.
drop compared to their performance the normal dataset. In contrast,
setting, train the classifiers for all the methods
4.4 Comparison with The Latest Commercial Detection Method
We also compare BISCOPE with the latest version (2024-01-09) of the most renowned commercial Al Al-generated text detection API, GPTZero [43], across all five datasets, shown in Figure BISCOPE outperforms GPTZero in 72% of the cases. Specifically, BISCOPE achieves 0.02, 0.01, and 0.01 average Fl detection score increase on the Arxiv, Essay, and Creative datasets, respectively. On the Yelp dataset, BISCOPE's Fl detection score is 0.04 lower than GPTZero's s. However, on the code dataset, BISCOPE performs significantly better than GPTZero, achieving a 0.19 average Fl score improvement, demonstrating BISCOPE's superior generality from natural language to code. Note that GPTZero's 's detection model is pre-trained on millions data points, while our BISCOPE's classifier is trained on most 4, 000 test samples for each case.
4.5 Efficiency Analysis
To address real-world challenges, efficiency crucial for detection methods. We compared BISCOPE with nine baselines across five datasets, detailing average processing times for single sample in Figure 6. RADAR is the fastest at 0.01s per sample, while its adversarial training overhead may be considerable. Processing times for the other baselines are as follows: zero-shot query at 0.32s, LogRank 0.05s, LRR at 0.10s, DetectGPT at 15.95s, Raidar at 27.42s, OpenAl Detector at 0.03s, Binoculars at 0.19s, and GhostBuster at 0.37s. Zero-shot query, LogRank, LRR, and OpenAl Detector are much quicker than DetectGPT, Raidar, Binoculars, and GhostBuster but offer lower detection performance and robustness. Our BISCOPE processes sample in 0. 0.144 without summary guidance, matching the real-time levels of zero-shot query, LogRank, and LRR, while improving detection Fl score by over 0.30. With summary guidance, BISCOPE's processing time increases to 1.35s per

---

sample, still 12 to 20 times faster than DetectGPT and Raidar, and achieves the highest detection score,
4.6 Ablation Study
We further evaluate the importance of each component in BISCOPE with two categories of ablation experiments. Since BISCOPE utilizes six open-source LLMs in parallel and ensemble their features, we first investigate the contribution of each LLM's s feature in Figure 7 individually. Then, we further explore the contribution of BISCOPE's s FCE and BCE losses respectively. compared with their aggregated performance. shown as Figure 8. More detailed results are shown in Appendix E. We also present more detailed ablation study on the impact of different segmentation strategies in multi-point splitting (Appendix E.3), and the impact of the completion prompt (Appendix E.4), in Appendix E.
BISCOPE'S Performance with Different Base Models. We individually test all the detection base models in B1SCOPE both with and without the summary procedure, including Gemma-2B, Gemma- 7B, 2-ama-2-BB, Mistral-7B, Llama-3-8B, and Llama-2-13B. All the detection models help BISCOPE achieve over a 0.84 overall F1 detection score across all five datasets, demonstrating high consistency across different detection base models. As the size of the detection base model increases, BISCOPE performs progressively better, with Fl scores improving from 0.84 to 0.95.
Importance of FFC and BCE in BISCOPE'S Detection. To demonstrate the rationality of the proposed bi-directional cross-entropy losses, we also evaluate BISCOPE by only using either FCE or BCE losses, and using both them with the Llama-2-7B model, as shown in Figure 8. When using both the FCE and BCE losses, BISCOPE achieves the best average Fl detection score across the five datasets, which 0.94, highlighting the necessity of combining both FCE and BCE loss features. When only using FCE BCE, BISCOPE reaches 0.86 and 0.93 average Fl detection scores, respectively, indicating that BCE is more discriminative than FCE.
5 Limitations and Future Work
Our BISCOPE can achieve a 0.95 F1 detection score across five datasets generated by the five latest commercial LLMs, both with and without intentional paraphrasing, illustrating the importance of the preceding token information in the output logits. However, in the OOD cross-dataset setting, there is a noticeable > 0.10 detection Fl score drop, highlighting the challenges in this setting. Thus, there is still room for future research achieve more effective and robust detection in few-shot and cross cross-dataset settings, where the training set contains fewer samples and the test set comes from different text domains. Additionally, exploring ways to further exploit preceding token information and combine it with next token information should also be future direction in the field of AI- generated text detection.
Conclusion
Existing methods to differentiate Al Al-generated texts from human- generated texts often analyze the difficulty for a surrogate LLM to generate the next token based on previous tokens from the text. We propose more discriminative approach via a novel bi-directional cross-entropy calculation method, leveraging both the preceding token information and the next token information in the output logits. We integrate this method into a four-step detection pipeline, BISCOPE, which consists of Completion Prompt Generation, Loss Computation in Text Completion, Statistical Feature Extraction, and Feature Classification. We evaluate BISCOPE five datasets, including both natural language and code, against six existing detection methods. BISCOPE surpasses all these methods, improving the average Fl detection score by 0.30 and also outperforming the well-known commercial API GPTZero in 72% cases, while maintaining a real-time processing speed of less than 200ms per sample.
10

---

Acknowledgments and Disclosure of Funding
We are grateful to the Center for AT Safety for providing computational resources. This work was funded in part by the National Science Foundation (NSF) Awards SHF- SHF-1910300, Proto-OKN 2333736, IIS- 26135,,,,, DARPA VSPELLS HR001 10000558, IARPA TrojAl W911NF- 19-S0012, ONR N0001417 12045, N000141410468 and 10001417129477 Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint Si:i::3333088772 2023.
[2] Anthropic. Claude 3 api. http://wwwaaannnpccccc/e/eeeeuet
2023.
[3] Anton Bakhtin, Sam Gross, Myle Ou, Yuntian Deng. Mare' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint wrXiv:1906.03351, 2019.
[4] Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of geeeereegeenrrated text via conditional probability curvature. In International Conference on Learning Representations (ICLR), 2024.
[5] Amrita Bhattacharjee and Huan Liu. Fighting fire with fire: can chatgpt detect ai ai-generated text? ACM SIGKDD Explorations Newsletter, 25(2):14-22 11, 2024.
[6] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Nco: Large Scale Autoregressive Language Modeling with Tess-oensor,,ww, 2021.
[7] Jack T Brassil, Steven Low, Nicholas F Maxemchuk, and Lawrence O' O'Gomma. Electronic marking and identification techniques to discourage document copying. IEEE Journal Selected Areas Communications (JSAC), 13(5511495-1504, 1995.
[8] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372
2021.
[9] Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions on Information Theory, ((10))6631-6622 2022.
[10] Debby RE Cotton, Peter Cotton, and Reuben Shipway. Chatting and cheating: Ensuring academic integrity in the era of chatgpt. Innovations in Education and Teaching International, 61(2):228222 2024.
[11] Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visual- ization of generated text. Annual Meeting of the Association for Computational Linguisties: System Demonstrations (ACL), 2019.
[12] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu, How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597 2023.
[13] Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting llms with binoculars: Zero-shot detection of machine machine-generated generated text. In International Conference on Machine Learning (ICML), 2024.
[14] Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgthench: Benchmarking machine-generane generated text detection. arXiv preprint arXiv:2303.1 1.8..22.0.. 2023.
[15] Ahe Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, and Yulia Tsvetkav. Semstamp: A semantic watermark with paraphrastic robustness for text generation. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2024.
[16] Xiaomeng Hu, Pin- Yu Chen, and Tsung Yi Ho. Radar: Robust ai-text detection via adversarial learning. Advances in Neural Information Processing Systems (NeurIPS), 2023.

---

[17] Daphne Ippolito. Daniel Duckworth, Chris Callison- and Douglas Eck. Automatic detection of generated text easiest when humans fooled. In Annual Meeting of the Association for Computational Linguistics: Syxtem Demonstrations (ACL), 2020.
[18] Ganesh Jawahar, Muhammad Abdul Mageed, and Laks Lakshmanan. Automatic detection of machine generated text: A critical survey. In International Conference on Computational Linguistics (COLJNG), 2020.
[19] Mohan S Kankanhalli and KF Hau. Watermarking of electronic text documents. Electronic Commerce Research, 2:169-187, 2002.
[20] John Kirchenbauer, Jonas Geiping, Yuxin Wea, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning (ICML., 2023.
[21] Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. Outfox: LIm-generated essay detection through in-context learning with adversarially generated examples. In AAAI Conference Artificial Intelligence (AAAI), 2024.
[22] Sarah Kreps. Miles McCain, and Miles Brundage. All news that's fit to fabricate: Ai-generated text as a tool media misinformation. Journal of Experimental Political Science, 9(1):104-117 2022.
[23] Kalpesh Krishna, Yixiao Song. Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[24] Rohith Kuditipudi, John Thickstun. Tatsunori Hashimoto, and Percy Liang. Robust distortion-free water- marks for language models. arXiv preprint urXiv:2307.1553 155993, 2023.
[25] Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang. Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: new algorithm for training recurrent networks. Advances Neural Information Processing Systems (NeurIPS), 2016.
[26] Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[27] Jiayi Liang, Xi Zhang, Yuming Shang, Sanchuan Guo, and Chaozhuo L., Clean-label poisoning attack against fake news detection models. In IEEE International Conference on Big Data (BigData), 2023.
[28] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica Sutherland. Learning deep kernels for non- non-parametric two-sample tests. In International Conference on Machine Learning (ICML), 2020.
[29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ar /907. 1692, 2019.
[30] Chung Kwan Lo. What is the impact of chatgpt on education? rapid review of the literature. Education Sciences, 13(4):410, 2023.
[31] David Lapez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations (ICLR), 2016.
[32] Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting. International Conference on Learning Representations (ICLR), 2024.
[33] Hope McGovem, Rickard Stureborg, Yoshi Suhara, and Dimitris Alikaniotis. Your large language models are leaving fingerprints. arXiv preprint arXiv:2405. 14057, 2024.
[34] Niloofar Mireshghallah, Justus Mattern. Sicun Gao, Reza Shokri, and Taylor Berg- Kirkpatrick. Smaller language models are better zero-shot machine-generatee text detectors. In Conference of the European Chapter of the Association for Computational Linguistics (EACI.), 2024.
[35] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot maaheae-generrated text detection using probability curvature. In International Conference on Machine Learning (ICML), 2023.
[36] OpenAL. G.t-3.5 turbo api. http://hhhfamm...rrrr....oooocss
2023.
12

---

[37] Mike Perkins. Academic integrity considerations of ai large language models in the post-pandemic era: Chatgpt and beyond. Journal of University Teaching & Learning Practice, 20(2):07, 2023.
[38] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1-67, 2020.
[39] Vinu Sankar Sadasivan, Acunon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint 1...v::2303.11550 2023.
[40] Trene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: 09200099003 2019.
[41] Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for zero-shot detection of machine-genennn generated text. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
-achine-generated
[42] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Battiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312./1805 2023.
[43] Edward Tian and Alexander Cui. Gpizero: Towards detection of ai-generated text using zero-shot and supervised methods", 2023.
[44] Umut Topkara, Merean Topkara, and Mikhail Atallah. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions. In The 8th Workshop on Multimedia and Security, 2006.
[45] Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Chernaaskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. Intrinsic dimension estimation for robust detection of ai-generated texts. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[46] Honai Ucoka, Yugo Murawaki, and Sadao Kurohashi. Frustratingly easy edit-based linguistic steganography with a masked language model. In The North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2021.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurlPS), 2017.
[48] Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghosthuster: Detecting text ghostwritten by large language models. arXiv preprint arXiv:2305. 15047, 2023.
[49] Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on nip models. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). 2021.
[50] Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, et al. M4: Multi-generator, multi- domain, and multi-lingual black-box machine-generated text detection. arXiv preprint 2305 2023.
[51] Wilson Wu, John X Morris, and Lionel Levine. Do language models plan ahead for future tokens? arXiv preprint 2404::2444 00859, 2024.
[52] Xi Yang, Kejiang Chen, Weiming Zhang, Chang Liu, Yuang Qi, Jie Zhang, Han Fang, and Nenghai Yu. Watermarking text generated by black-box language models. arXiv preprint arXiv:2305.0888 2023.
[53] Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free free detection of gpt-generated text. International Conference on Learning Representations (ICLR), 2024.
[54] Shuhai Zhang, Yiliao Song, Jiahao Yang, Yuanqing Li. Bo Han, and Mingkui Tan. Detecting machine- generated texts by multi caaliipopulation aware optimization for maximum mean discrepancy. In International Conference on Learning Representations (ICLR), 2024.
[55] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: survey on hallucination in large language models, arxiv preprint 210 2023.
13

---

To further illustrate our BISCOPE with more details, we present the following materials in the Appendix, shown as follows:
Appendix A: Hyper-pann parameerr settings for method and all the baselines.
Appendix B: An additional motivation example using smaller LLM.
Appendix C: More statistical details of the dataset crafted and utilized in the paper.
Appendix D: More detailed results on both normal and paraphrased data under OOD setting.
Appendix E: More details of the ablation study.
A Hyper-parameter Settings
GPT-Neo- We use the default best-performance settings for all the baselines. Specifically,
2.7B [6] as the surrogate model in LogRank and LRR, and as the scoring model in DetectGPT, where use T5-3B [38] as the mask-filling model. For RADAR,
its officially released detection model, which is a pre-trained NoBERTa-Laagg [29] model. For Zero-shot Query and Raidar, we use GPT-3.5-Turbo the query model and the rewriting model. For other baselines, strictly follow their official implementations with their best configurations. For our BISCOPE, we utilize six open- source LLMs in parallel, including Gemma-2B, Gemma-7B, Llama-2-7B, Mistral-7B, Llama-3-8B, and Llama-2-13B. We also test BISCOPE with (BISCOPE ") and without (BISCOPE) text summary guidance. For better reproducibility, we recommend using either Llama2-7B Llama2- 13B the surrogate model, while the ensemble of more surrogate models is always welcomed. For the text split method, we recommend splitting the text at every 10% length, used in our paper.
B Additional Motivation Example
[a] Next Token Rank, 0.50.
(b) Last Token Rank, F1=0.63
(c) Both Rank, Fl.0.67
100
Human Text
30
Last Token Rank
Text Al Text
Count
Text
Count
20
4k-
50
10
2k
1000
5000
1000 Next Token Rank
Rank
Rank
(di CE, =1
(e) Token CE, F1=0.76
(i) CE, F1=0.82
40
Last Token CB Loss
13
30-
12-
Count
Count
20
20
11
10
10
Cross Entropy Loss
Cross Entropy Loss
Next Token CK Loss
Figure 9: Comparison between the detection F1 scores when utilizing the rank and cross-entropy loss regarding next token, last token or both. The detection sarrogate model is GPT-Neo 2.7B.
Figure 9 presents an additional example using a smaller language model: GPT-Nco-2.7B. Specifically, Figure (a) and (d) show the detection Fl score when only using the next token's 's rank and cross- entropy loss for detection. The detection scores are less than 0.60 in both cases. However, when the preceding token's 's rank or cross-entropy loss the feature, shown in Figure 9(b) and (e), the detection Fl scores increase to 0.63 and 0.76 respectively, indicating the higher reference value of the preceding token information. Figure 9(c) and (f) also show that using both the preceding token information and the next token information in the output logits achieves the best detection performance, with overall detection Fl scores of 0.67 and 0.82.
Moreover, when considering the preceding token information, Al text has a smaller rank score or higher cross cross-entropy loss, representing worse memorization of the Al text in the output logits.
14

---

Conversely, when considering the next token information, the result is the opposite, showing better prediction for Al text. This trend aligns with the results in Figure I, demonstrating the generality of our observation.
C Additional Details about Datasets
Table Statistical details of the five datasets used our paper.
Normal Dosavet
Paruprrased Daiaset
Dam Type
Danset
Average
Max Len.
Len.
Dataset
Average Len.
Min Len.
Median Len.
350
786.7
132
1736
Arciv
All
Machime
2100
1750
787.0
787.1
101
101
1701
1736
810.5
799.5
1400
1400
875.8
875.8
174
174
1874
1874
931.5
931.5
Human
164
631.5
132
1993
$72.0
Code
Machinne
819
413.3
41
1908
352.0
656
493.6
13
2533
382.5
983
489.7
1953
387.0
13
2533
382.5
Human
2000
554.9
6959
407.0
Yelp
2548
537.0
11740
477.1
10
4959
413.0
8000
586.5
54
2593
337.0
Human
1000
4249.9
1376
41470
3381.5
Essay
Machiine
4897
3827.7
515
21094
3486.0
3999
3666.8
129
19878
$284.0
All
3897
389913
515
41470
3439.0
3999
3666.3
129
19878
3284.0
Human
1000
2899.0
499
9933
2462.5
Creative
Machime
1840
2651.9
176
13716
2620.0
4000
2924.4
85
16812
2674.5
286000
13716
R3
2674.5
For all datasets, reuse their suman-gennrated data and craft Al Al-generated text using five of the latest commercial LLMs: GPT-3.5-Turbo [36], GPT-4-Turbo [1]. Claude-3-Sonnet [2], Claude-3- Opus [2], and Gemini-1.0 Pr-Pro [42]. Thus, for each dataset, have piece of guman-genneatee data and five pieces of Al Al-generated data. Additionally, we also create a corresponding paraphrased dataset using similar paraphrasing prompts from previous studies [32] on newly crafted dataset to evaluate the robustness of the detection methods.
Table 2 presents more detailed information about the datasets used in our paper, for both the normal dataset and the paraphrased dataset. For the short natural language datasets (i.e., Arxiv and Yelp), the average text lengths are 187 and 477, respectively, with minimal lengths of 101 and 10. In contrast, the average text lengths of the long natural language datasets (i.e., Creative and Essay) are 2860 and 3899, which are 6-15 times larger than the short natural language dataset's 's sample length. The code dataset contains approximately 164 pieces of Python code for gener-geeeeatedd text and Al-generated text from each L.M,, with average length of 983, minimum length of 41, and maximum length of 1993. For all the datasets, the -enerrtee data and Al-generated data share similar lengths, preventing simple length- based detection.
D Additional Comparison Results under OOD Setting
Table 3 illustrates more detailed OOD results both the normal and paraphrased datasets across five datasets. Under the cross-model setting on the normal dataset, BISCOPE outperforms existing baselines in 21 out of 25 cases. Specifically, BISCOPE reaches over 0.95 detection F1 score against five generative LLMS on the Arxiv, Creative, and Essay datasets, while the detection Fl scores of existing baselines are usually less than 0.90. On the Yelp dataset, BISCOPE achieves over a 0.90 detection Fl score against all five generative models, while the five baselines reach less than a 0.85 detection Fl score most cases. On the code dataset, our BISCOPE outperforms all baselines except for Raidar, achieving more than a 0.15 detection Fl score increase. Under the cross-dataset setting on the normal dataset, our BISCOPE performs worse than under the cross-modee setting, outperforming existing methods in only 9 out of 25 cases. However, in most cases, BISCOPE is the second-bes detection method, only trailing RADAR, which uses a detection model pre-trained on texts from multiple domains. As for the OOD setting on the paraphrased dataset, BISCOPE outperforms the five baselines 17 out of cases, achieving over a 0.90 detection F1 score compared the < 0.75 average Fl score of the baselines.
15

---

Table 3: Detailed performance comparison on both normal and paraphrased dataset under
OOD setting.
Normal-Cinss Model
Nommal-Cross Datnset
Paraphrasot-OO0
Method
3,
GPT-
Claude33
Chadee3
Gemini
GPT-335
GPT-4
Claude3
Claude-3
Gemini
GPT335
GPT-4
Claude3
Claude-3
Turo
Turbo
Sonnet
Opus
1.0gpco
TTubbo
Turbo
Sonnet
Opus
1.9-pro
Turbo
Turbo
Sounet
Opus
Log Rink
0.6723
0,7033
0.7406
0.6805
0.6907
0.6727
0.5779
0.6476
0,6222
0.5590
0.5814
0.3821
0.2604
0.3382
1RR
0.7920
0.7187
0.7589
0.715%
0.6742
0,6866
0.5238
0.6302
0.6600
0.081
0.2389
0.2904
0.3537
Ariv
DIseEGPP
0,6667
0.6637
0.661
0.6637
0.6637
0.6845
0.6926
0.7343
0,7421
0.6921
0.602
0.5774
0.6622
0.6609
RADAR
0.5721
0.7049
0.7799
0.7854
0.7696
0,8115
0.7638
0.8057
08116
0.8071
0.923
0.6546
0.7128
0.6654
Raidar
0,5702
0.6975
0.7604
0.6946
0.7093
0.5010
0,6736
0.6841
0.6888
0.7144
0,830
0.7195
0.6286
0.7297
OpenAT
0.5009
0.6667
0.6670
0.6657
0.6043
0.6917
0.6653
0.6841
0.6679
07558
0.691
0.664
0.6688
0.6641
Binoculars
0.9306
0.9272
0,9399
0.8988
0.8818
0.8430
0.7495
018489
0.8814
0.7766
0,434
0.4547
0.7406
0.7105
GhusiBuster
0,7684
0.9170
0.9617
0.9561
0,9702
0.566]
0.6942
0.7281
0.699
0.6982
0.9643
0.9769
0.9562
0,9814
BiScOPE
0,9683
0.9496
0.9665
0,9126
0.9617
0.5706
0.6642
0.7548
0.7968
0.7791
0.5610
0.7461
0.8766
0.9298
BISCOrE"
0,9859
0.9807
0.9881
0.9518
0.9810
0.8250
0.6494
0.8285
0.7875
0.7930
0,5014
0.7678
0.8941
0.9296
Log Roak
0.6722
0.6722
0.6309
0.6345
0.6774
DU6695
0.6678
0.6836
0.6841
0.6427
0.0752
0.6477
0.5546
0,6318
LIKK
0,6000
0.6001
0.6475
0.6585
0.6686
0.6667
0.6666
0.66664
0.0664
0.6416
06662
0.0644
0.6163
0.6644
Volp
DiserCiPT
0.6006
0.6817
0.638
0.6254
0,607
0,7124
0,6931
0.7%6
0,753
0.6955
0.379
0.345
0,7029
0,7085
RADAR
0.7129
0.7180
0.7303
0.7164
0,004
0.8760
0,8255
0.8129
0,738
0.4244
0.706
0.082
0.6220
0,7023
Kaidar
0,8357
0.8493
0.394
0,8769
0,8083
0:4310
0,7533
0.7835
0.7879
0.6695
0.7897
0.8886
0,6993
0.74%
Open Detector
0.5876
0.6577
0.687
0.6425
0.6367
0.6853
0.6600
0.6663
0,4055
0.529
0:7341
0 066668
066688
0,5674
Binoculars
0.7076
0.6669
0.7028
0.6600
0.7188
0.6668
0.8978
0.9350
0.8566
0.633
0.7148
GhostBuster
0.7727
0.8176
0.3044
0.8076
0.7821
0.3737
0.7522
0.6073
0.7167
0.4294
0,7900
0.8539
025546
0.8754
RISCoO::
0:9010
0.9257
0.580
0.8982
0.1774
0.9189
0.8180
0.8659
0,8441
0,5500
0,9846
0.9523
0.9787
0.9736
BISCOPE
0.9134
0.9266
0.8956
0.9049
0.9234
0.8912
0 0.8056
0. 0,8570
0.8460
0.8175
0,9017
0,9617
0.9787
0,9700
Log Kink
0.7438
0.7836
0.2399
0.7966
0,8485
0,2514
0,5614
035938
0.6269
0.7755
05130
0:3257
0.5324
0,7138
LRR
0,5695
0.6864
0.7587
0.7277
0.7786
0.5657
0.5837
0.5835
0.5940
0.3601
0.3500
0.3255
0.3605
0.5024
Creative
RADAR
DeteenGiP
0.8639
0.7484
0,9067
0.7466
0,7592
09112
0.8919
0.7525
018911
0.7750
0,8052
0.4603
0,7352
0,5679
0.7511
0.5003
0,285
0.5933
0.7727
0.5797
0,9131
(.566)
0.8699
0.5769
0.9199
0.6951
0.8330
0.6006
Raidar
0.7705
0.8417
0.338
0.8201
09100
0,2058
0,7343
0.7933
0,7779
0.779
0.6667
0.5797
0,5678
0,4543
OpenAA Derector
0.4144
0.6569
0.6867
0.6507
06600
022120
0.6697
0.6666
066667
0.6420
0.182
0.6669
0,6669
0,6609
Binoculars
0,9600
0.9733
0.9803
0:9776
033955
0.7596
0.6796
0.7894
03080
0.7248
0.9999
0.7326
U50899
000099000000
0.9537
GhosiBuster
0.8597
0.9825
0.9817
0.9615
0,9500
0.8302
0.8055
0.82%
03318
0.3156
0,3059
0,9635
0.9778
0,9576
BISCOrE:
0:9514
0.9949
0.9960
0.9917
0.9650
0.6212
0.8081
0.8522
08596
0.8490
0.8981
0.9550
0.9950
0.9945
BISCOrE
0.9109
0.9955
0.9950
0.9912
0,9914
0.6507
0,8317
0,8759
0.5732
0.8192
0.996
0.9960
0,9525
Log Rank
0.8185
0,9433
0.9234
0.8989
0,9286
0,4647
0,3644
0.4817
0,5211
0.4472
0.4331
0.4073
0.7248
0
LRR
0.8415
0.1288
0.9010
091100
0,5060
0,3917
0,4839
0.5429
0.1354
0,4418
0,2645
0.6478
0,6040
Essey
DescceGPT
RADAR
0,9121
0,9087
0.9238
0.9450
0.9398
0.163
0.9046
0.9658
039473
0,9118
0.3452
0.8088
0.7223
0,5392
0.6067
0.7492
0.6667
0.7828
0.3669
0.7693
0,7744
09113
0.0604
0.8028
0.4113
0.8352
0.8978
0.8352
Raidar
0.9238
0.9536
0.448
0.9307
033533
0.7382
0.7202
0.7058
0,633
0.7203
0.559
0. 0.7903
0.7399
0,7809
OpenAl Derector
0.2530
0.0004
0.0006
0.6600
0,6022
0,2673
0.6670
0.6663
0.0669
0.6311
0.123
0.6647
0.6669
0,6656
Binoculars
0,9954
0.9971
0.993
0.9968
0,998
0,7699
0,5924
0.7826
0,4155
0.7253
0,9626
0.3205
08733
0,9250
GhusiBuster
0,9500
0.9955
0.9931
0,9853
0,9779
0,7691
0,6945
0,7588
0.306
0.8167
050000
09639
099662
0.9708
BISCOrE
0,9825
0
0.9986
0,9958
0,9949
0,4709
0,5208
0.5579
0.6535
0.5252
03179
0.9863
019712
0,9979
BiSCOrE"
0,9653
0.9587
0.9986
0.9991
0,9952
0,4702
0,5305
0,6152
0.6866
0.5319
0.7531
0.9811
0.9847
0,9990
Log Rank
0.6629
0.6615
0.6629
0,6315
0,6416
0.6851
0,4556
0.6693
0.4969
0.3459
0.5049
0.5148
0.4518
0.6511
I.R
0.6614
0.6643
0.6666
0.6642
0.6513
0,6654
0.6660
0.6672
0.4668
0 0.5714
0.6339
0 6667
0.6499
0.6667
Code
DesecotiPT
0.6588
0.6560
0.6533
0.6454
0.6111
0.50%
0.6632
0.7992
0.7592
0,6095
0.9975
0.6390
0.6667
RADAR
0.6599
0.6670
0.6663
0.6656
0,6673
0,8476
0,7668
0.8062
0,8232
0.8134
0.6667
0 066633
0.6666
0,6653
Raidue
0.9339
0.9498
0.9364
0.9277
0.6801
0,084
0.0806
0.0454
0.1509
0 0.3256
0 0.9685
0:6522
0.4409
0.6448
OpenAl Desector
0.6116
0.6609
0.6689
0.6741
0,6680
0,5297
0,0992
0.2784
0.5289
0.6054
0.7293
0.6122
0,3406
0.6217
Binoculars
0,5510
0.6656
0.6567
0.6563
0,8481
0,7608
0.6680
0.6724
0.6412
0.7107
0,6656
0.6667
0.6653
GhoetRuster
0.5725
0.6349
0.6750
0.6809
0.5626
0.5253
0.4645
0.5441
0.5404
0.4744
0.8596
0. 0.773
0.5906
0.4854
BISCOFE
0.7752
0.7661
0.3376
0.8012
0.8068
0,5843
0.5612
0.5889
0.5947
0.9633
0.9523
0.8987
0.7501
BISCOPE
0.7721
0.7621
0.8374
0,7906
0.7869
0.6033
0,5769
0.5750
0.5793
0.5930
0.9719
0. 0.9528
0.8829
0.8021
E Additional Details of Ablation Study
E.1
Results with Different Open-source LLMs
Table 4 shows additional details when different detection models are used by BISCOPE both the normal and paraphrased datasets. In most cases, the ensemble results are better than the Fl scores derived using single detection model. However, at least 90% of the detection performance can be preserved in most single-detectio deleeddetettoo-mdeeee settings. Additionally, the detection Fl score consistently increases with the inereasing model size of the detection model.
Among all the detection models, the Llama-2 series performs the best, outperforming its updated version, Llama-3. In contrast, the Gemma series performs the worst. A possible reason is that the Gemma and Ilama-3 series models are still in their early development stages and are not fully capable of handling the AI- Al-generated text detection task.
16

---

Table 4: Detailed performance comparison with different detection base models in BISCOPE.
Nomal
Paraphmeed
Model
GPT-3.5
Turbo
GPTT-4
Turbo
Clando-3 Sonnet
Cladle33 Opus
Gemini 1.0-pro
Averue
GPT-3.5
Turbo
GPT4
Turbo
Clasde-3
Scomet
Clande-3 Oyes
Average
Gemma-2B
0,8007
0.8763
0.8567
0.9290
0.9175
0.8962
0.8606
0.5489
0.8338
0.8053
0.8597
wlo cumsiar
Gemma 7B
0.9530
0.9335
0.8970
0.9679
0.9335
0.9370
0.9037
0.8678
0.8660
0.210
0.8896
Lhmm-2-77
0.9827
0,9957
0,9766
0.9855
0,9708
0.9823
0.9360
0.5355
0
0.9796
0.9541
Mistral-7B
0.9710
0,9843
0.9638
0.9855
0.9663
0.742
0.9511
09561
0.9414
09724
0.9553
Liaaa
0.9685
0.9576
0.9811
0.9665
0.911
09102
0.4005
0.110
Lhmm-2-133
0.9634
0.987]
0.9662
0.9114
0.9678
0.9752
0.9307
0.400
0.9922
0.9826
0.9099
Anny
ensemble
0.9870
0.9928
0.9796
0.9885
0.9708
0.9837
0.9769
0.9800
0.9625
0.9870
0.9766
Gemma 2B
0.9223
0.9255
0,9140
0.9568
0.9282
0.9296
0.8282
08334
0.8386
0.0788
0.5533
summmar
Gemmo-78
0.9301
0,9438
0.9206
0.9679
0.9357
0.9437
0.8982
0.8338
0.6777
0.9283
0.8970
Liana 2-76
0.9957
0.9971
0,9913
0.9998
0.9798
0.9907
05681
0.7885
0,9710
0.9884
09765
Mixtral:7B
0.739
0,9679
0.9894
0.9711
0.9622
0.201
0.5244
0,9361
0.726
0.9479
Llama-3-88
0.8114
0,9785
0.9708
0.9841
0.9725
0.9775
09152
09175
08787
09461
09144
Litana 2- 2-13
0.9842
0.9856
0.9766
0.9884
0,9725
0.9814
0,9468
0.9526
0,9485
0.9826
0.9576
ensemble
0.9928
0.9943
0,9859
0.9913
0,9797
0.9890
0.9870
0.9899
0,9593
0.9884
0.9802
Gemma-2B 2B
0,7400
0,7393
0,7689
0,750R
0,7557
0.7509
0.7368
0.7578
0,8053
0.7853
0.7713
we tanmnary
Gemma-78
0.8750
0.8958
0.8817
0.8208
03764
0,8944
0,5890
0,8702
L.uara-2-7h
03366
0.9012
0,9446
0.9289
0.9209
09102
0.859
09233
07717
0,6633
09293
Mixtral-7B
0.8433
0.8939
0.9365
0.9128
0.8949
0.971
0.8630
0.000
09932
0.9544
0.9229
LJima-3-88
0.550
0.8663
0.9301
0.9249
0:9026
03899
0.8374
0.9000
00331
0-9375
0,8955
Llura-2-138
0.95%
0.9360
09770
0.924
0.9421
Yelp
ensemble
0.9023
0.9405
0.9652
0.9532
0,9486
0.9420
0.9064
0.9473
0,9814
0.9789
0.9535
Gomma
0.3475
0.7912
0.8158
0,7660
0.7750
0.7486
0.7544
08296
0.8163
0.7897
We summury
Gomma: 7B
0
0.8706
0.8800
0.8800
0.8805
0.3655
0.8145
0.688
08851
0.8570
0.8639
Lhmmu-2-76
0.5655
0,9121
0,9490
0.9341
0.925]
0.9122
0.8689
0.9270
0,955
0.6666
0.9341
Mixtral:7B
0.8724
0,9006
0,9413
0.9281
0,9050
0.9095
0
09624
0.9522
09258
Libaa
0.8727
0.9393
0.9005
0.9088
0.102
0.8511
0.8944
09116
09711
0.9044
Lhmma-2-33
0.8771
0.9250
0.9533
0.9380
0.9280
0.9243
0,8878
0.9376
0,8000
0.7099
09441
etsemble
0.9010
0.9452
0,9658
0.9570
0.9545
0.9447
0.9102
0.9530
0,9830
0.9757
0.9555
Gemmo-28
0.8199
0.9527
0.9424
0.9620
0.9695
0.9617
0.964
09410
09224
0.9566
0.486
Wio summmay
Gomma-78 7B
0.9800
0.9555
0:9549
0.9739
0.9151
0.9695
0.5786
01611
035000
0.9699
0.9649
1.lura-2-7h
0.9340
0.9934
0.9941
0.995
09035
0,9940
0.9911
Mixmal-7B
05970
0.0920
0.9250
0.9920
0.9964
0.945
0.9155
0.920
08900
0.2000
0.9999
Chimu-3-88
0.9985
0.9754
0,9700
0.9910
0.9911
0.9870
0,9720
0.3100
09531
0.9740
0.9625
Craative
I. 11ara-2.138
0.580
0.9940
0.9930
0.9915
0.9958
0.9945
0.9935
0.9950
0,9945
0.9945
0.9944
essemble
0.9985
0.9950
0,9968
0.9930
0,9964
0.9958
0.9955
0.9955
0.9940
0.9943
Gemma-2B 2B
0.9875
0.9519
0,9503
0.9672
0,9694
0.9653
0.9675
0.9459
0.9426
0.9654
0.9554
of summmry
Gemma-78
0.9545
0.954
0.9759
0.9766
0.9703
0:0740
0.9540
0.9775
0,9684
Libma-2-75
0.9980
0.9925
0,9950
0.99:0
0.9935
0.9916
0.9885
0.9925
0
0.925
0.9199
Mistral-7B
0.966
0.9930
0,9950
0.9955
0.9922
0.9937
0.9895
0.9899
0,9935
0.9020
0.9912
Llima-3-8B
0.9980
0.9790
0.9835
0.9935
0.9917
0.9891
0.9760
0.6044
0,973
0.9795
0.9721
Lhamu 2-13B
0.9050
0.9930
0.9958
0.9910
0.9910
0,9945
0.9900
0.9939
ensemble
0.9975
0.9955
0.9955
0.9945
0.9970
0.9960
0.9955
0.9955
0,9950
0.9945
0.9951
Gomma-2R
0.0980
0.9830
0:9935
0.9919
0.9950
0.993
0.0805
0.543
0.9666
0.9864
0.720
wib summary
Gemma-78
0.965
0.9699
0:9855
0.9900
0.9905
0.9665
0.7779
0.5855
0.921
0.784
09717
Litana
0.995
0.9970
0.9966
0.9965
0.9955
0.9978
0.9940
019975
029770
0.9970
0.9964
Mistral-BB
0.9970
0.940
0.9955
0.9889
0.9971
0.9920
0.9950
09639
0.9055
0.9916
L-uata-3-B
0.9955
0.9975
0.995
0.965
0.9994
0.9999
0.9905
0.910
02849
0.9955
0.9955
Litana2-11
0.5995
0.9980
0,9990
0.9960
0.9994
0.99822
0,9955
0.9975
0.955
0.9770
09964
Essay
onsemble
1.0000
0.999
0.9985
0.9970
0.9994
0.9988
1.9965
0.9990
0.9990
0.9980
0.9981
Gemmo-28
0.9970
0,9850
0:9935
0.9945
0.9967
0.993
0.5810
0.9645
0.979
0.9875
09777
We sarsnary
Gemma-7B
0.9980
0.9743
0.9879
0.9000
09899
0.9880
0:9830
0.9673
09969
0 0.9834
0.9757
L.lara-2-7h
0.9975
0.9975
0.9975
05994
0.9985
1998
0:9950
0.9970
09964
0.9080
0.996
Mistral-7B
1,0000
0.9960
0.9955
0.9800
69972
0.9969
0.9900
0.9915
09834
0.9955
05916
Llima-1-88
0.990
0.9975
0.9930
0.9975
0.9989
0.992
0.9035
0.925
10925
0.9080
0.9941
L.lmma-2-138
0.9935
0.9980
0.9965
03955
0.993
0.9744
0:9955
0.9855
00935
0.9970
0.996
ensemble
0.9985
0,9989
0.9975
0.9965
0.9985
0.9984
Gemma 29
0.7726
0,7238
0.6018
0.5056
0,2674
0.5752
0.9905
0.5545
0.9666
0.9864
05920
wlo summary
Gomma:
0.5441
0.1515
0.2983
0.559
0.779
0.9383
(972)
0.984
0.9177
Liaana--77
0.4499
0.9301
088755
0.5677
0,7878
08122
0.940
0.9755
03970
0.9700
0.9944
Mistral-7B
0.9301
0.9527
0.8396
0.3834
0.6995
03009
0.9920
0.950
09839
0.9955
0.9916
Lhana 3- 8B
0.3888
0,7787
0.6202
0.5560
0.4457
0.6179
0.996
0.910
098:9
0.9555
0.9905
Liama-2-13B
0.630
0.9590
B8215
0.6151
0.7980
03313
03955
0.9955
09955
03970
0.964
Code
cssemble
0.9665
0.9655
0,8528
0.0069
0,7809
0.3345
0.9965
0.9990
0.9990
0.9980
0
Gemma-28
0.7955
0.7051
0,5500
0.4716
0.2640
0.5594
0.9810
0.9645
0.9799
0.9875
03777
W summary
Gemma-778
0.0088
0,7295
0.5601
0-4522
0.5130
0.5719
0.5620
096755
0.9699
0.9344
09957
I. Liama-2-7h
0.9504
0.9346
0.874
06604
0.7768
0.333
0.9950
0.9770
0.995
0 9980
0.996
Mixtral-7B
0.9280
0.8158
0.6556
0.7145
0.8123
0.990
0.9844
0.9945
0.9916
Liuma---SB
03352
0.7871
0.6223
0.5566
0,4954
0.6635
0.9935
0.9925
0.9925
0.9980
0.9941
L.lam-2--13
0.9426
0.9469
B.8468
6070
0.8084
033303
0.9955
0.9985
0.0055
0:9070
0.0966
ossemble
0.9692
0.95%
0.8536
0.6620
0.7741
0,5433
1L9975
0.9990
0.9985
0.9985
0.9984
17

---

Table 5: Detailed contribution comparison between FCE and BCE.
1lama-2770 wis summary
Generstive Model
Method
GPT-3.5 Turbe
GPT
Claude-3 Opus
Gemini pro
FCC
0:9281
0.908
0,5407
0,9827
0.9647
Only
0.9524
0.9885
0,9668
0.9740
0.9479
BCE&FCE
0.9827
0.9957
0.9766
0.9855
0.9708
JCE Only
0,7934
0. 7865
0.8834
0.8626
0.8342
Yelp
BCE Only
0.8435
0 L9005
0,9396
0.9198
0.9151
BCE+JCE
0.8566
0,5002
0,9416
0,9289
0,9209
FCC Only
0.9965
0.9498
0,9799
0.9855
0.9791
Creative
BCE&FFC
Only
0,9950
0,9340
0.9915
0,9940
JCE
0.9860
0,9979
0,9930
0.9989
Essay
ICB Only
0,9995
0.9970
0,9978
0.9965
0.9941
BC
0.9995
0.9970
0,9965
0.9965
0,9995
FCC Only
0.7349
0.6439
0.4933
0.4628
0.3817
Code
BCE Only
0.949
0.9466
0.8173
0,5747
0.7819
BCE&FCE
0:9479
0,9301
0.8275
0.7878
Table 6: Ablation results of different segmentation methods in multi-point splitting in BISCOPE.
Nermal Theaset
Persghrasol Dataset
Normal
Persghrasod
Mathod
GPT:3.5
Turbo
GPT-4
Turbo
Claude-3
Sonnet
Claude Opes
Gemini
1.0-pro
GPT-335
Turbo
GPT-4
Turbo
Claude Sonnet
Claude Opus
Avg
Asg.
Every 50% Text
0.7744
0.9671
0.6355
018855
0.9637
0.9587
0.9756
0.9486
0.9767
0.9750
0.9649
Every 25% Text
0.9813
0.9914
0.9662
0.9844
0,9690
0,9675
0,9769
0,9622
0,9797
0.9792
0.9716
Every 10% Text (In Paper)
0.9670
0.9928
0.9796
0.9885
0.978
0.9769
0.9800
0.9625
0.9870
0.9837
0.9766
Eivery 50% Text
0,8922
0,9314
0.9584
0.9421
0.933
0,8921
0.9359
0.9779
0.9704
0.9326
0,9441
Every 25% Text
0.502
0.9581
0.951
0.9277
0.9466
0.9041
0.9452
0.9817
0.9757
0:9405
0,9517
Every 10% Text (In Puper)
0.9023
0.9805
0.9652
0.9532
0.9496
0.9064
0.9473
0.9814
0.9789
0.9420
0.9535
Crodive
Evvery 50%S Text
0.9985
0,9960
0.9940
0.9955
0.9958
0,9950
0.9955
0.9935
0.9920
0.9960
0,944
Every 25% Test
0.9480
0.9955
0.9960
0.990
0.9970
0.9960
0.9955
0.9950
0.9915
0.9959
0.9950
Every 10%6 Text (In Paperi
0.9985
0,9950
0.9960
0.9300
0.9964
0.9955
0.9945
0.9955
0.9940
0.9958
0.9549
Every 50% Text
1,0000
0,9990
0.0965
0.9970
0.9994
0.9975
0.9995
0.9975
0.9075
0.9084
0.9060
Every 25% Tost
1,000
0 1.9990
0.9985
0.9980
0.999
0.9965
0.9990
0.9985
0.9900
0.9990
0.9980
Every 10% Text (In Puper)
1,0000
0,9990
0.9985
0.9970
0.9994
0.9965
0.9990
0.9990
0.9980
0.9988
0.9981
Every 50% Text
0,8700
0.7706
0.6479
0.8708
0.8752
0.9311
0.8437
0.7495
0.8797
Every 25% Text
0.0532
0.9333
0.51 1.5
0.6184
0.7068
0.9363
0.9322
0.9470
0.8856
0.8050
0.9352
Every 10% Text (In Puper)
0.9665
0,9635
0,8528
0.6069
0.7809
0,9639
0.9464
0.9691
0.9250
0.8345
0.9516
Table 7: Performance results when not using the completion prompt in BISCOPE.
Normal Dusaset
Paraghrased Dataset
Normal
Paraphrased
Method
GPT:3.5
Turbo
GPT-4
Turoo
Clande-3
Sannet
Clando-3
Opas
Geminii
1.D-pro
GPT:335
Turbo
GPT-4
Claude-3 Sonnet
Clauds-3 Opus
Avg.
Completion Prompt|
0.9813
0,9914
0.9767
0.9900
0.9692
0.9714
0.9855
0.9621
0.9766
0.9817
0,9739
w/ Completion Prompt
0.9870
0.9928
0.9796
0.9885
0.9708
0,9769
0.9800
0.9625
0.9870
0.9837
0.9766
Yelp
wro Cumpletion Prompt|
0,9008
0.9471
0.9686
0.9577
0.9523
0,9058
0.9576
0.9872
0.9817
0.9453
0.9581
Completion Prompi
0.9023
0.9405
0.9532
0.9864
0.9814
0.9789
0.9420
0.9535
Creeuive
Completion Prompt
0.9800
0.9950
0.9950
0.9925
0.9952
0.9945
0.9950
0.9940
0.9952
0.9348
w/ Completion Prompt
0.9985
0.9950
0.9960
0.9930
0.9964
0.9955
0.9925
0.9955
0.9940
0.9958
0,9949
Exsa
wro Completion Prompt
1.0000
0,9995
0.9990
0.9975
0.9994
0.9975
0.9990
0.9990
0.9973
0.9991
0,9982
w/ Completion Prompt
1.0000
0,9990
0.9985
0.9970
0.9994
0,9963
0,9990
0.9990
0.9980
0.9988
0.998
Code
wro Cumpletion Prompt|
0.9786
0.9752
0:8327
0,6703
0.7653
0.9729
0.9532
0.9717
0.9210
0.8644
0.9545
Completion Prompi
9665
0.9655
0.8528
0,6069
0.7809
0.9659
0.9464
0.9691
0.9250
0.8345
0.9516
Additionally, when using only single base model, BISCOPE can achieve a processing speed as fast as 0.03s per sample, outperforming all baseline methods except for RADAR.
18

---

E.2 Detailed Contribution Comparison of Forward and Backward Cross-entropy Losses
Table shows more concrete comparison between the contributions of FCE and BCE to BiScoPE's detection effectiveness. In 21 of 25 cases, BCE is more discriminative compared with FCE (especially on code dataset), providing sufficient justifications for BiScOPE that introduce the preceding token information into the detection. Besides, In 16 of the 25 cases, the combination of JCE and BCE outperforms the FCE only and BCE only versions, supporting the necessity to use the bi-directional cross-entropy loss calculation method.
E.3 Impact of Different Segmentation Strategies in Multi-point Splitting in BISCOPE
Table 6 presents the ablation results with different segmentation strategies in the multi point splitting of BISCOPE. We tested three strategies: splitting at every 50% text length, every 25% text length. and every 10% text length used in our paper). The results indicate that more fine-grained splitting interval generally improves B1SCOPE's performance. However, in small number of cases, smaller splitting interval may degrade performance. We choose to use 10% achieves the highest detection scores in most cases while reaching low degradation in corner cases.
E.4 Impact of The Completion Prompt in BISCOPE
Table 7 presents the comparison results when using and not using the completion prompt in BISCOPE. The results show that in 25 of 45 cases, using the completion prompt performs better. Additionally, the completion prompt is more compatible with the summary procedure. Thus, we chose the completion prompt in BISCOPE.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize the procedure of our method and highlight both of our method and dataset contribution in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We present the limitation of our work and potential future research directions
in $ 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiittinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
We use empirical examples and results to justify our method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We introduce the details of our method at each step in $ 3, and the hyper- parameters we used in Appendix A. The code will be available at https : /tt///ggttttttt/te////// com/
MarkGHX Briccoe/BiScop..
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will open-source our data and code at httpp:: :/g thub. Moom/MMKKHK/
BiScope.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/guide Saalii//giieee/CCCo waweeessSSoeeSmaaiiiiiiiii conooicy))) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. hcc/pullic/aee wwwcc//gg///////eee /CodeSubmiiiiccec ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detailed the dataset, metrics, and evaluation settings in $ We also list the haper-paramcters we used for our methods and all the basclines in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not use error bars, but we use the Fl score to measure our method and all the baselines instead. Additionally, we also conduct detailed ablation study for each specific case in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed hyper-parameter settings and present the execution time for our method and all the baselines.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //terr//n.e...eeeeee cwteriiss.cppppue cctppblic////cppttccccccccccccct heppuuic//tiiiiooooe. csGuidelines"
Answer: [Yes]
Justification: Our paper is aligned with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We present the advantages and limitations of our method, and introduce future directions at $5. We also compare our method with the latest commercial method, showing the positive impacts on the open-source community.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not involved in misusing.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all the baseline methods and data sources in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithhaeee thcode. www.coo////oo//o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We present detailed information about our new dataset in Appendix C.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "NhVW7JfAmy", "reviewer": "Reviewer_V5gS", "summary": "This paper develops an AI-generated text detection method called BiScope. The key idea is to formulate the detection task as a guided text completion task. The generated text and the original text are used to calculate two types of cross-entropy losses, which are used to extract features for classification.", "questions": "Please respond to the weaknesses raised above.", "limitations": "I believe it’s important to mention the computational cost in the body text as a limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- (S1) The idea of taking the preceding token account for AI-generated text detection is interesting.\n- (S2) The paper is well-written and easy to follow.", "weaknesses": "- (W1) The feature extraction is computationally expensive, as it needs to run inference at least twice with LLMs for summary generation and text completion. It’s not clear if the feature extraction cost is worth for the improvements. (Similar to the comment below).\n- (W2) Missing references and comparisons, especially against the following methods. Among them, [Ref 2] should be a solid baseline method for the classification-based approach.\n[Ref 4] is recent work and is optional to compare but reports that simply using n-gram and POS features would be sufficient to detect machine-generated text and thus interesting to compare as a baseline.\n    - [Ref1] OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples https://arxiv.org/abs/2307.11729\n    - [Ref 2] Ghostbuster: Detecting Text Ghostwritten by Large Language Models https://arxiv.org/abs/2305.15047\n    - [Ref 3] Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors https://aclanthology.org/2024.eacl-short.25.pdf\n    - [Ref 4] Your Large Language Models Are Leaving Fingerprints https://arxiv.org/abs/2405.14057", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review. Here are our detailed point-by-point feedbacks for your questions:\n\n**W1**: As mentioned in Section 3.3., the summary generation is not necessary. We have two designs for generating the completion prompt: one with a summary and the other without. The latter has a significantly shorter processing time. As illustrated in Figure 5, this adjustment allows BiScope to achieve over 8.6x shorter processing time per sample. As shown in Table 1, BiScope without summary only results in an average detection F1 score degradation of less than 0.015 and still substantially outperforms the baselines. Users can choose between these two designs to balance detection accuracy and efficiency according to their specific needs. We will further modify our main text to further clarify this.\n\n**W2**: Thank you for suggesting the references and baselines. We present a comparison between BiScope and GhostBuster [1] in Table 6, demonstrating that BiScope outperforms GhostBuster with a 0.06 average F1 score increase on both the normal and paraphrased datasets. Additionally, GhostBuster is 3x slower than BiScope in processing a single sample and requires several hours to identify the optimal feature compositions. We did not compare with [2] due to the lack of open-source implementation. Instead, as mentioned in the global response, we compared with two other baselines, Binoculars [5] and OpenAI Detector [6], which have comparable or better results than [2-4]. We will reference [1-4] in our paper and discuss them in the related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MrxOcCLfTC", "reviewer": "Reviewer_Z5m7", "summary": "This paper proposes extracting various features from predictive distributions of surrogate LLMs to detect LLM-generated text. Relative to prior work, the main novelty appears to be the use of bi-directional cross-entropy losses to extract features. These features are then fed into a traditional supervised classifier to make predictions, which is estimated on a labelled dataset of human and machine-written text. Evaluations on a new dataset (expanded version of existing datasets) shows that the method is competitive with some prior approaches.", "questions": "See \"Weaknesses\"", "limitations": "No; see \"Weaknesses\"", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper tackles an important problem: automatic detection of LLM-written text.\n\n* The paper contributes datasets that cover several state-of-the-art LLMs and multiple domains, including code.\n\n* The paper includes some results for the more realistic out-of-distribution condition, where novel LLMs / genres are introduced at test time relative to the training data. The approach appears to be quite robust to new LLMs, in some genres.\n\n* The approach appears to be quite robust to paraphrased text.", "weaknesses": "* There are some missing comparisons, e.g. https://arxiv.org/abs/2401.12070.\n\n* The performance in the most important setting (OOD) is mixed.\n\n* The presentation is somewhat confusing. For example, it’s unclear from what exposition (e.g., Figure 3) which steps occur at training time and which steps occur at test time.\n\n* The proposed approach amounts to a supervised binary classifier. However, the obvious baseline (fine-tuned BERT) is not included in the comparisons, even though this approach (“OpenAI classifier”) is prominent and discussed in related work. Why not?\n\n* Important details are not included in the main text. For example, for the cross-model evaluation, which models are trained on and which models are held-out? Hopefully, the latest LLMs are held-out and previous generation LLMs (e.g., GPT-2) are used for training.\n\n* Why is F1 used as the metric? Usually, in detection scenarios we are interested in detection accuracy while maintaining a low false alarm rate, which suggests using ROC-based metrics, e.g. AUC restricted to the low false alarm region.\n\n* The bold-underlines in Table 1 seem a bit random. For example, there are cases where two identical values occur in the same column but both are not bolded (0.9955). Ideally, these would represent statistical tests of which relative improvements are significant.\n\n* The discussion of limitations is completely lacking. For example, it seems like the proposed approach requires evaluating multiple LLMs in parallel. Also, I’m unclear on why there is not an evaluation setting in which both data and model shift occurs, relative to the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review. Here are our point-by-point responses:\n\n**W1**: We have included three more baselines in the comparison in Table 6 (in the submitted PDF file): Binoculars [5], GhostBuster [1], and OpenAI Detector [6]. The results show that BiScope outperforms all three baselines on both the normal and paraphrased datasets.\n\n**W2**: We have presented the experimental results in both cross-model and cross-dataset OOD settings, as well as the evaluation on unseen paraphrased data, in the right three columns of Table 1 in our main text. Additionally, more detailed OOD results are provided in Table 3 in the appendix. We observe that BiScope performs the best or the second best in more than 75% cases.\n\n**W3**: In Figure 3, the first three steps are used in both the classifier training and testing periods to extract features. During the classifier training period, the extracted features are used to train a classifier in step 4. In contrast, during the testing period, the classifier in step 4 is fixed, and we use this trained classifier to make predictions based on the extracted test sample’s features. We will further modify Figure 3 to make this process clearer.\n\n**W4**: Thank you for pointing out the OpenAI Detector. We have included RADAR [7] in our main text’s results. RADAR is a fine-tuned RoBERTa, outperforming the OpenAI Detector [6] in most cases. Therefore, we initially chose not to include the OpenAI Detector  in our paper. To further address your concern, we have included the OpenAI Detector in Table 6 (in the submitted PDF file). The results show that BiScope outperforms the OpenAI Detector with more than 0.2 F1 score improvement on average on both the normal and paraphrased datasets.\n\n**W5**: In our cross-model OOD evaluation (Section 4.2), we trained the classifier on human data and AI-generated data from one LLM and then tested the trained classifier on AI-generated texts from the other four LLMs and calculated the average F1 score. This process was repeated for all five generative LLMs, and the average scores are reported in Table 1 in the main text. More detailed results are presented in Table 3 in our appendix. Results show that BiScope performs the best or the second best in more than 75% cases. Specifically, when we trained BiScope on the oldest GPT-3.5-Turbo’s data and tested it on all the other four latest LLMs’ data, the detection F1 score exceeded 0.92 on average.\n\n**W6**: Thanks for pointing out additional metrics. We use the F1 score in our paper since it is a commonly used metric in previous papers [1, 8], considering both precision and recall. We further present the TPR-FPR (ROC) curve of BiScope in Figure 9 (in the submitted PDF file). We observe that our BiScope reaches over 0.8 detection TPR on average when the FPR is only 0.01, outperforming all the baselines on all the five generative models’s data.\n\n**W7**: Thanks for your recommendation. We will further polish our tables based on your suggestion.\n\n**W8**: Thank you for pointing out the limitation problem. Due to the page limit of the submission, we had to place the limitation section in Appendix F in our initial submission. We will move it to the main text. Regarding the OOD evaluation, previous studies [1, 5, 7, 8] shifted either the data (cross-dataset) or the generative models (cross-model). We followed their settings to test our OOD performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zCkAu0tUcX", "reviewer": "Reviewer_Kykm", "summary": "The paper describes work on detecting machine-generated texts using a proposed method called BiScope which exploits a model’s states by considering both the preceding token information and the next token information via an bi-directional cross-entropy loss calculation method. The proposed BiScope method does not make use of any additional finetuning and instead leverages on the calculated forward and backward cross-entropy losses as features for the binary classifier. The performance of said classifier can also be improved through the use of summaries. Results show that using BiScope with summaries outperform existing SOTA methods for AI text generators across aspects such as in and out-of-distribution results, intentional paraphrasing, and efficiency. Overall, I believe the study has the level of completeness, technical rigor, and impact required for a NeurIPS paper.", "questions": "1. Is there a limit or a threshold for the number of approximator models as described in Step 2 for calculating the bi-directional cross-entropy losses? Moreover, is there a criteria for the selection of these models to ensure the converged values of the losses are met? Can users just go for open models? I think this part is not much discussed/clarified.\n2. Is the combination of FCE and BCE statistically significant over using just one of them? The values presented in Table 4 seem to be very close with each other particularly on text-based datasets.\n3. Following #1, is there a particular model or combination of models that gives the best approximation of FCE and BCE loss values and particular text split combination for classification? Future researchers may want to only test with the best setup combinations for comparing SOTA or baselines.", "limitations": "The paper can benefit from a much clearer discussion of limitations of the work, particular as it has not emphasized aspects such as language and ability of the proposed method to be robust on non-native speaker written texts as already explored by previous works (see cited work above).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written, easy to follow, and has the level of completeness required for NeurIPS. The proposed bi-directional calculation of losses which the authors hypothesized to contain information preceding token and next token has been properly motivated with a clear research framing. The proposed method has also been extensively and rigorously compared with a number of SOTA baselines, a large compiled multi-domain dataset, and was able to prove its superiority over previous approaches.", "weaknesses": "I do not see any strong cases of technical issues. However, some points can be considered to improve the overall quality of the work and support the realistic application of the method:\n\nIt might be best to further emphasize what specific additions or changes were made to the compiled existing dataset (Arxiv, Yelp, code, etc) in the main paper and possibly name this. Moreover, you should also provide immediate details about the datasets used particularly on the length (whether they are essay-length, paragraph-length, etc), language coverage, as well as register or domain.\n\nTo strengthen the contribution of the study, I strongly suggest the authors to do the same experiment as done by Tulchinskii et al (https://proceedings.neurips.cc/paper_files/paper/2023/hash/7baa48bc166aa2013d78cbdc15010530-Abstract-Conference.html) and on evaluating the limitation of the proposed BiScope method in detecting whether non-native written texts are falsely identified as machine-generated. If BiScope is still able to clearly differentiate non-native written texts from machine-generated texts, then this is an advantage for the authors. Moreover, this particular advantage may also give another favorable reason to use the model other than its performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation and suggestions. Here are our point-by-point responses:\n\n**W1**: Thank you for pointing out the problem. We generated our datasets using five of the latest commercial LLMs, following the generation methods outlined in previous studies [1, 7, 8]. Due to the page limit of the initial submission, we placed the details of the generated datasets (e.g., data amount, length statistics) in Appendix C. We will add more details to the main text in our next version.\n\n**W2**: Thank you for your suggestion. Due to the time constraints, we were unable to generate non-native language data for our datasets to provide a fair comparison during the rebuttal period. However, the setting you recommended is certainly an important OOD setting for all detectors. We will include [9] in our paper’s discussion and provide more detailed results in the next version.\n\n**Q1**: Thank you for the question. We presented a detailed ablation study using various open-source surrogate LLMs in Table 5 in Appendix E.1. According to the results, the performance of BiScope improves when using more surrogate detection models. Notably, BiScope can maintain over 98% of the best (ensembled) performance even when only using a single surrogate model (e.g., Llama2-7B, Mistral-7B). This consistent performance of BiScope across various open-source LLMs illustrates its scalability and compatibility with different open-source LLMs, providing more flexible options for users. There is no limit or threshold for the number of surrogate models, and BiScope can be compatible with any combination of surrogate models.\n\n**Q2**: We presented a more detailed comparison of the FCE-only, BCE-only, and FCE+BCE versions of BiScope in Table 4 of Appendix E.2. The combination of FCE and BCE outperforms either FCE only version or BCE only version in 64% of cases, showing >0.35 and >0.09 maximal detection F1 score improvement, respectively. Such results demonstrate the necessity of this combination.\n\n**Q3**: Thank you for your suggestion. For simplicity, we recommend using either Llama2-7B or Llama2-13B as the surrogate model, while the ensemble of more surrogate models is always welcomed. For the text split method, we recommend splitting the text at every 10% length, as used in our paper. We will open-source our code and datasets for future researchers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rNhwqevgtI", "reviewer": "Reviewer_VTu5", "summary": "This paper presents BiScope, a new algorithm for AI-generated text detection, that leverages logits from a tect expansion task to detect AI-generated text. The algorithm proceeds in three steps:\n\n1. Given a candidate input text T, break it into two segments (seg1, seg2, such that seg1 + seg2 = T). A \"text completion\" prompt is created where given <summary(T), seg1>, the goal is to predict seg2. The rationale for this is to give the detection algorithm full context awareness of the candidate input text.\n\n2. These completion prompts are passed through several open-source LLMs with seg2 tokens present in a teacher forcing manner. The logits are used as features for subsequent downstream classification.\n\n3. The features are used to train a binary classifier to detect AI-generated text.\n\nThe authors perform several experiments and ablations in their paper, and find that BiScope is effective and efficient compared to many baselines (including GPTZero).", "questions": "1. AI-generated text detectors typically need to operate in a low FPR setting, to minimize the risks of labeling innocent text as AI-generated. Given this, what are the Table 1 true positive rates at a low FPR of say 0-1% (or equivalently, an AUC-ROC curve in that range)? TPR at low FPR ranges (0-1%) is a standard metric for evaluating AI-generated text detectors which has been used in many previous papers and blogs: https://arxiv.org/abs/2303.13408, https://openreview.net/pdf?id=DEJIDCmWOz, https://arxiv.org/pdf/2401.12070, https://foundation.mozilla.org/en/blog/who-wrote-that-evaluating-tools-to-detect-ai-generated-text/\n\n2. How in-distribution is the RADAR training data compared to the evaluation data used? Since RADAR was not retrained, I'm worried the baseline maybe put at an unfair disadvantage compared to BiScope.\n\n3. In Table 1, I'm a bit surprised that there's almost no drop in detection accuracy before/after paraphrasing for most baselines in the non-Essay settings (line 300-301). What is the reason for this? This seems in contradiction with findings in multiple prior works.\n\n4. In Table 2, there's a strong length bias in the two domains. Does that make the classification task easier than it would have been in a length-controlled setting (take first K tokens of AI-generated / human-generated text and classify them)?\n\nPresentation / nits:\n\n* Figure 4 is hard to read, could it be converted to a table?\n\n* Move the baselines description (L112-117) to a dedicated subsection / paragraph in Section 4.1\n\n* In Section 3.4, add the list of open-source LLMs used in main body", "limitations": "Yes, adequately addressed", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a novel algorithm for AI-generated text detection, a research area growing in importance due to the development of powerful LLMs. Utilizing bidirectional logit information to discriminate between LLM-generated and human-written text is an interesting idea, and the authors have done a good job figuring out the technical details for an effective implementation.\n\n2. The paper performs thorough experiments on five AI-generated text detection datasets, five popular commerical LLMs, and in both IID / OOD settings. The paper compares their method to both academic and commerical AI-generated text detectors, including GPTZero, a popular commerical software for AI-generated text detection. Additionally, the paper showcases the robustness of BiScope to paraphrasing attacks.\n\n3. The paper does some additional ablation studies on the choice of open source LLMs for step 2 above, an efficiency analysis, and the choice of classifier features.", "weaknesses": "1. The out of distribution (OOD) experiments in the paper are quite limited, which put the baselines at an unfair disadvantage. The authors experiment with two OOD settings (L280-290), making either the text domain OOD or the generative LLM, **but not both**. Hence, I don't think the OOD settings are fully OOD, I think this puts BiScope at an unfair advantage over all baselines, since many of them have not received any in-distribution training.\n\n2. The paper would be stronger with more baselines, ideally some watermarking methods too. I am curious to know how BiScope compares to newer methods like Binoculars [5], as well as text watermarking algorithms like KGW [1], EXP-Edit [2], SemStamp [3], or [4]?\n\n3. A few more ablation experiments would make the paper stronger. These were some additional questions I had: a) how much does the number of segments (`n`) in Section 3.5 matter? b) What is the performance like if a classifier is trained just on the feed-forward logits of the same open-source LLMs (using the same training data), without any completion prompts?\n\n[1] - https://arxiv.org/abs/2301.10226  \n[2] - https://arxiv.org/abs/2307.15593  \n[3] - https://arxiv.org/abs/2310.03991  \n[4] - https://arxiv.org/abs/2305.08883  \n[5] - https://arxiv.org/pdf/2401.12070\n\n---\n\n**After rebuttal**: Thank you for the response. I've raised my score to 5 due to extra baselines like Ghostbusters and Binoculars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review and suggestions. Here are our point-by-point responses:\n\n**W1**: Regarding the OOD evaluation, previous studies [1, 5, 7, 8] shifted either the data (cross-dataset) or the generative models (cross-model).. We strictly follow their settings in our OOD evaluation. Our comparison with baselines is fair because we retrained all the baselines (except RADAR which is not open-source) and BiScope using the same datasets. And the testing was performed on the same datasets too. Please let us know if we misunderstand your question.\n\n**W2**: Thank you for your suggestions. We have included three more baselines in the Table 6 (in the submitted PDF file), including Binoculars [5], GhostBuster [1], and OpenAI Detector [6]. The results illustrate that BiScope outperforms all three new baselines in most cases across both the normal and paraphrased datasets. On the other hand, watermarking techniques allow the detection of generated text by limiting the generation within some special vocabulary. They require modifications to the decoding strategy of the generative model. This does not align with the application scenario of BiScope and the baselines as they do not require any modification to the LLM’s generation process and can work in a black-box setting. Although it is difficult to compare the two, we consider them complementary. We will cite [10-13] and include the above discussion.\n\n**W3**: Thank you for your suggestion. We have implemented these two ablation experiments, and the results are shown in Table 7 and Table 8 in the uploaded PDF file, respectively.\n\nTable 7 presents the ablation results with different segmentation strategies in the multi-point splitting of BiScope. We tested three strategies: splitting at every 50% text length, every 25% text length, and every 10% text length (as used in our paper). The results indicate that a more fine-grained splitting interval generally improves BiScope's performance. However, in a small number of cases, a smaller splitting interval may degrade performance. We chose to use  10% as it achieves the highest detection scores in most cases while reaching low degradation in corner cases.\n\nTable 8 presents the comparison results when using and not using the completion prompt in BiScope. The results show that in 25 of 45 cases, using the completion prompt performs better. Additionally, the completion prompt is more compatible with the summary procedure. Thus, we chose to use the completion prompt in BiScope.\n\n**Q1**: Thanks for the suggestion. We further present the AUC-ROC curve of our method, shown in Figure 9 (in the submitted PDF file). The results show that our BiScope reaches over 0.8 detection TPR on average when the FPR is only 0.01, outperforming all the baselines on all the five generative models’s data.\n\n**Q2**: RADAR is officially trained on the Openwebtext dataset, which contains over 8 million human-written texts and over 8 million AI-generated texts crafted by RADAR’s authors. These texts are very similar to the data in the Yelp dataset. Since the RADAR’s authors do not provide their training code, we did not further fine-tune RADAR on our datasets, but when calculating the F1 score, we did search the best threshold for RADAR’s output on our datasets. Considering the substantial amount of RADAR's pre-training data and our in-distribution threshold searching, the comparison in our paper does not put RADAR in an unfair position.\n\n**Q3**: In Table 1, we present the results of BiScope on the paraphrased datasets under both the in-distribution (seen) setting and OOD (unseen) setting. Under the in-distribution setting, BiScope and all the baselines are trained and tested on the paraphrased dataset, resulting in a very small performance drop for BiScope (as mentioned in line 300-301). However, under the OOD setting, BiScope and all the baselines are trained on the normal dataset and tested on the paraphrased dataset, as shown in the right-most column in Table 1. We observe a significant F1 score drop for all methods, including BiScope. For example, BiScope’s detection F1 score drops by around 0.1 on the Arxiv dataset. Notably, BiScope’s performance drop is the smallest among all the detectors in most cases. \n\n**Q4**: Thank you for pointing out the potential influence caused by the text lengths. We note that the length bias can be clearly identified in the Yelp, Essay, and Creative datasets, where the human-written texts can be twice as long as the AI-generated texts. Thus, we re-ran BiScope on these three datasets, using the first K characters as the input, where K equals the average text length of each individual dataset as shown in Table 2. The results are presented in the following table. For the Yelp dataset, the average detection F1 score drops within 0.01, while for the Creative and Essay datasets, there is even a slight increase in the average detection F1 score. Therefore, the length bias in the dataset does not provide BiScope with unfair advantages.\n|||Normal Dataset In-Distribution|||||Paraphrased Dataset In-Distribution|||| Normal | Paraphrased |\n|---|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Dataset|Input|GPT-3.5-Turbo|GPT-4-Turbo|Claude-3-Sonnet|Claude-3-Opus|Gemini-1.0-Pro|GPT-3.5-Turbo|GPT-4-Turbo|Claude-3-Sonnet|Claude-3-Opus| Avg. | Avg. |\n|Yelp|Clipped|0.8968|0.9256|0.9574|0.9456|0.9436|0.9003|0.9329|0.9760|0.9711|0.9338|0.9451|\n||Unclipped|0.9023|0.9405|0.9652|0.9532|0.9486|0.9064|0.9473|0.9814|0.9789|0.9420|0.9535|\n|Creative|Clipped|0.9980|0.9960|0.9954|0.9954|0.9975|0.9955|0.9955|0.9954|0.9945|0.9965|0.9952|\n||Unclipped|0.9985|0.9950|0.9960|0.9930|0.9964|0.9955|0.9945|0.9955|0.9940|0.9958|0.9949|\n|Essay|Clipped|1.0000|0.9990|0.9990|0.9975|1.0000|0.9990|0.9990|1.0000|0.9985|0.9991|0.9991|\n||Unclipped|1.0000|0.9990|0.9985|0.9970|0.9994|0.9965|0.9990|0.9990|0.9980|0.9988|0.9981|\n\n\nThank you for all the presentation suggestions. We will further modify our paper based on them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for your thoughtful comments! We are glad that the reviewers found our paper “tackles an important problem” with a novel idea. We also thank you for your appreciation of our dataset contribution, method’s robustness, and paper presentation.

To further address your concerns, we conducted more experiments and provided more detailed evidence to support our proposed method. Here is a summary of the supplementary information provided in the rebuttal materials:

1. We compare our BiScope with **three** more baselines that are recommended by the reviewers, including Binoculars, GhostBuster, and OpenAI Detector. We observe that BiScope outperforms all three baselines on all five datasets with more than 0.06 average detection F1 score, even when the data is intentionally paraphrased.

2. We test BiScope under a length-controlled setting, presenting the low sensitivity of BiScope to the length of the input text with less than 0.01 detection F1 score degradation.

3. We evaluate BiScope without any completion prompt and compare its performance with the original version in our paper, illustrating that the completion prompt allows BiScope to perform better in 56% of cases.

4. We evaluate BiScope with different numbers of segments during the feature extraction step, justifying the necessity of our proposed multi-point splitting. The results show the trend that a finer-grained segmentation interval leads to a higher detection F1 score.

5. We provided the TPR-FPR(ROC) curves of BiScope and compared them with the baseline methods’, showing that BiScope achieves the highest TPR (more than 0.8 on average) in a low FPR setting (FPR=0.01).

6. We also present point-by-point responses to all the other questions and concerns from all the reviewers on the rebuttal page under each review.

Due to the length limit of each individual rebuttal, we present the most requested experimental results in the supplementary PDF file.

We also list the general references used across all the rebuttal materials here.

---------
**References**

[1] Verma, Vivek, et al. "Ghostbuster: Detecting Text Ghostwritten by Large Language Models." NAACL. 2024.

[2] McGovern, Hope, et al. "Your Large Language Models Are Leaving Fingerprints." arXiv preprint arXiv:2405.14057 (2024).

[3] Koike, Ryuto, Masahiro Kaneko, and Naoaki Okazaki. "Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples." AAAI. 2024.

[4] Mireshghallah, Niloofar, et al. "Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors." EACL. 2024.

[5] Hans, Abhimanyu, et al. "Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text." ICML. 2024.

[6] Solaiman, Irene, et al. "Release strategies and the social impacts of language models." arXiv preprint arXiv:1908.09203. 2019.

[7] Hu, Xiaomeng, Pin-Yu Chen, and Tsung-Yi Ho. "Radar: Robust ai-text detection via adversarial learning." NeurIPS. 2023.

[8] Mao, Chengzhi, et al. "Raidar: geneRative AI Detection viA Rewriting." ICLR. 2024.

[9] Tulchinskii, Eduard, et al. "Intrinsic dimension estimation for robust detection of ai-generated texts." NeruIPS. 2023.

[10] Kirchenbauer, John, et al. "A watermark for large language models." ICML. 2023.

[11] Kuditipudi, Rohith, et al. "Robust distortion-free watermarks for language models." arXiv preprint arXiv:2307.15593. 2023.

[12] Hou, Abe, et al. "SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation." NAACL. 2024.

[13] Yang, Xi, et al. "Watermarking text generated by black-box language models." arXiv preprint arXiv:2305.08883. 2023.

Meta-review: This paper proposes BiScope, a method for detecting AI-generated text. The core idea is to leverage bi-directional cross-entropy calculations from surrogate LLMs to capture unique patterns in AI-generated text. The paper demonstrates strong empirical results, outperforming several existing baselines, including commercial tools. While reviewers had some concerns, the authors addressed them effectively during the rebuttal.

The scope and breadth of the experiments is a real strength of the paper. These experiments include in-distribution and out-of-distribution evaluations and ablation studies. The method shows strong detection accuracy and robustness compared to various baselines across multiple datasets and LLMs. The method also shows strong resilience to paraphrasing attacks.

The reviewers had concerns about some strong, recent baselines that were missing, but these experiments were added in the rebuttal. There were also concerns about efficiency, but this was addressed adequately as well (the method is still slow but in line with prior work and a speed-accuracy trade-off can be made that only marginally hurts performance).

Overall, this is a complete paper that makes a solid contribution to this increasingly important research area as the internet becomes more and more flooded with AI generated text.
Decision: Accept (poster)