AI-generated text boundary detection with RoFT
Authors: Laida Kushnareva
Tatiana Gaintseva
Dmitry Abulkhanov
Kristian Kuznetsov
German Magai
Eduard Tulchinskii
Serguei Barannikov
Sergey Nikolenko
Irina Piontkovskaya
Venue: COLM (2024)
Keywords: artificial text detection, cross-model detection, cross-domain detection, boundary detection, interpretability, analysis
TL;DR: We examine a number of ways to adapt up-to-date artificial text detection classifiers for the understudied task of detecting the boundary between human text and LLM generation in mixed human-AI content in cross-domain and cross-model settings.

Abstract: Due to the rapid development of large language models, people increasingly often encounter texts that may start as written by a human but continue as machine-generated. Detecting the boundary between human-written and machine-generated parts of such texts is a challenging problem that has not received much attention in literature. We attempt to bridge this gap and examine several ways to adapt state of the art artificial text detection classifiers to the boundary detection setting. We push all detectors to their limits, using the Real or Fake text benchmark that contains short texts on several topics and includes generations of various language models. We use this diversity to deeply examine the robustness of all detectors in cross-domain and cross-model settings to provide baselines and insights for future research. In particular, we find that perplexity-based approaches to boundary detection tend to be more robust to peculiarities of domain-specific data than supervised fine-tuning of the RoBERTa model; we also find which features of the text confuse boundary detection algorithms and negatively influence their performance in cross-domain settings.

Published as a conference paper at COLM 2024
Al-generated text boundary detection with RoFT
Laida Kushnareva', Tatiana Gaintseva?,
German MagaiS,, Serguei Barannikov Dmitry Abulkhanov, Kristian Kuznetsov Eduard Tulchinskii Irina I'iontkovskaya', Sergey Nikolenko'
1 AI Foundation and Algorithm Lab, Russia;
2 Digital Environment Research Institute, Queen Mary University of London, UK;
3 HSE University, Russia; Noeon Research, Japan;
5 Skolkovo Institute of Science and Technology, Russia;
6 CNRS, Université Paris Cité, France;
7 St. Petersburg Department of the Steklov Institute of Mathematics, Russia
Correspondence: kushnareevalidda@gmall.cmm
Abstract
Due to the rapid development of large language models, people increas- ingly often encounter texts that may start as written by human but con- tinue as machine-generated. Detecting the boundary between human- written and machine-generated parts of such texts is challenging problem that has not received much attention in literature. We attempt to bridge this gap and examine several ways to adapt state of the art artificial text detec- tion classifiers to the boundary detection setting. We push all detectors to their limits, using the Real or Fake text benchmark that contains short texts on several topics and includes generations of various language models. We use this diversity to deeply examine the robustness of all detectors in cross-domain and cross-model settings to provide baselines and insights for future research. In particular, we find that perplexity-basee approaches to boundary detection tend to be more robust to peculiarities of domain- specific data than supervised fine-tuning of the RoBERTa model; we also find which features of the text confuse boundary detection algorithms and negatively influence their performance in cross-domain settings.
Introduction
Artificial text detection (ATD) a very difficult problem in real life, where machine- generated text may be intertwined with human-written text, lightly edited, or pad out human-generated prompts. However, in literature ATD usually formulated in a simpler way, with dataset of text samples labeled as either entirely human-written or entirely machine-written, so the detection problem can be safely treated as binary classification. Moreover, the models for this binary classification are often developed and trained to detect particular type of generator, e.g,, text produced by specific large language model (LLM) (Uchendu al. (2023)). This is in stark contrast with how we may encounter artificially cre- ated text in real life, where documents by L.Ms already abound. This setting is much more complex and much less researched.
written by humans and partially generated
In this work, we experiment with a lesser known dataset called RoFT (Real Or Fake Text), collected by Dugan et al. (2023). Each text in this dataset consists of ten sentences, where the first several sentences are human-writted and the rest are machine-generater starting from this prompt, mainly by models from the GPT family (Radford et al., 2019; Brown et al., 2020). We consider several techniques developed for binary ATD, modifying them for this more complex boundary detection setting; e.g,, following Tulchinskii et al. (2023a) we adapt intrinsic dimension estimation which is currently considered to be the most robust method

---

Published as a conference paper at COLM 2024
True boundary
Detection
The girl sat, eyes wet. "Get up.
The girl sat, eyes wet. 'Get up.
The girl sat, eyes wet. "Get up.
The girl sat, eyes wet. "Get up. Don't give up now, ' her friend sald, placing a comforting hand her shoulder. The girl shook her head, streaming down her cheeks. "I can't do it. I'm just not strong enough." " "You are," her friend insisted. Loook at all the obstacles you'le overcome already, You're stronger than you think." The girl took deep breath and wiped her tears.
Don't t give up now," her friend
Don't give up now," her friend
Don't give up now, her friend
said, placing a comforting hand on her shoulder. The girl shook her head, streaming down her cheeks, "I can't do it. I'm just not strong enough." " "You are," her friend insisted. "Look at all the obstacles you've overcome already. You're stronger than you think." The girl took deep breath and wiped her tears,
said, placing a comforting hend on her shoulder. The girl shook her head, tears streaming down her cheeks. can't do it. I'm just not strong enough." "You aane"" her friend insisted 'Look at all the obstacles you've overcome already, You're stronger than you think." The girl took deep breath and wiped her tears.
said, placing a comforting hand
on shoulder. [The girl shock her head, tears streaming her cheeks. can't do it. I'm just not strong enough." "You are," her friend insisted. "Look at all the obstacles you've overcome already. You re stronger than you think." The gir took a deep breath and wiped her
prompt completion
gpt3.5-turbo
TLE + TS binary
phi1 5-pepplexxy
Sample input from the ROFT-chatgpt dataset. The rightmost part of the picture shows the true bondary between human-written and machine-generates text. Other three parts are colored according to the features, extracted by various classifiers. Here, gpt3.5 predicts that only two last sentences are generated; TLE + TS binary predicts that generation starts from the second sentence; phil.5 perplexity predicts that generation starts from the third sentence which is correct answer. See Section 3 for detailed explanation of all the methods and Appendix G for more examples from the dataset.
Figure 1:
for cross-domain and cross-model binary detection. Fig. shows sample input from RoFT and results of the considered models. Our primary contributions are as follows: (1) we evaluate five approaches to adapt perplexity-basee detectors to the task of detecting the boundary between human-written and maahinn--gneeatee text, discussing the differences in their behavior compared to binary ATD and providing a comprehensive analysis of how perplexity scores react to the machine-human transition in the text; we also compare several backbones for the perplexity evaluation to show which model works the best for this purpose; (2) we present evidence that perplexity scores obtained from small language model, p
trained on data generated by bigger LLM, provide strong features for boundary detection models, leading to relatively high accuracy of detection and cross-domain robustness related to other detection methods; (3) we introduce and evaluate two ways to adapt the classifiers based on intrinsic dimension estimation for this task, showing how the time series analysis can be used for extracting useful information from Transformer representations; (4) we show how the robustness of boundary detectors (including the fully-tuned RoBERTa baseline) to domain shift depends on the particular properties of the domains; we study the properties of the dataset itself and their effect on the performance of every approach; (5) we enrich the RoFT dataset with GPT-3.5-turbo! (ChatGPTT) generation samples; we share this new dataset with the community, establish baselines, and analyze the behavior of our detectors on it.?
We hope that this work will encourage further research both in boundary detection for texts that are partially human-written and partially generated and analyzing how inner representations of Transformer-based models react to such transitions; the latter direction may also help interpretability research. The rest of the paper is organized as follows. In Section 2 we survey related work, and Section 3 introduces the methods we have applied for artificial text boundary detection. Section 4 presents a comprehensive evaluation study on the RoFT and RoFT-chatypt datasets. Section 5 presents a detailed discussion and analysis of our experimental results, and Section 6 concludes the paper.
2 Related Work
Artificial text detection (ATD) is a well-studied task with plenty of already considered approaches; see, e.g., the surveys by Yang et al. (2023); Uchendu et al. (2023); Wu et al.
hhttps://pii http////aaffr.oo//// waw.atfoooo..pnii....mm apenai.com/ddeee wttpee///oooennnnnedeeeeeeeeeeee for-eee-fo
2Our official repository: https: //gi thub. ctm ver boundary detection. git
2

---

Published as a conference paper at COLM 2024
(2023a). In our work, we mostly concentrate adapting two groups of methods to artificial text boundary detection: methods based on perplexity estimation since they are the most widely used and methods based on topological data analysis (TDA) since they have shown promising robustness to domain shift and model shift (see below). Perplexttt based methods are based on the idea that human-writter text tends to be more unpredictable and varied compared to Al-generated text which may follow more predictable patterns, i.e., has lower perplexity score; among them we note GPTZero (Weber-Wulff et al., 2023), Sniffer (Li et al., 2023a), SeqXGPT (Wang et al., 2023), and LLMDet (Wu et al., 2023b). Topological data analysis (TDA) for ATD is inspired by the results of Kushnareva et al. (2021) and Tulchinskii et al. (2023a). The latter has shown that for many artificial text generators the intrinsic dimensions (PHD) of RoBERTa embeddings of the texts created by these generators are typically smaller than those of the texts written by humans; also TDA has proven to be useful for closely related tasks of fake news detection (Tudoreanu), authorship attribution (Elyasi & Moghadam, 2019), and detection of synthetic speech (Tulchinskii et al., 2023b).
Style change detection and authorship attribution. ATD can be considered as a special case of authorship attribution (AA), where the LLM is one author and the human is another, and the task is to determine the authorship of different parts of the text; it is also similar to the style change detection problem in multi-author documents (Zangerle et al., 2021). Jones et al. (2022) show that LLM models can successfully imitate human style and deceive existing popular online AA methods. However, Venkatraman et al. (2023) propose an approach based on the principle of uniform information density that can detect the authorship of LLM. State of the art style change detectors (Lin et al., 2022; Jiang et al., 2022; Lao et al., 2022; Iyer & Vosoughi, 2020) are based on Transformer-baser encoders such as BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), AIBERT Lan et al. (2020), and ELECTRA Clark et al. (2020). Therefore, we use RoBERTa model as a baseline and a source of embeddings.
RoFT (Real Or Fake Text). Our main dataset (Dugan et al., 2020; 2023) originates from a website called RoFT developed as a tool to analyze how humans detect generated text, including an extended study of whether and how they can explain their choice, when they say that a text sample is machine-generatee machine-generated text better. Every user of this website can choose topic ("Short Stories", "Recipes", "New York Times", or "Presidential Speeches") and start the following "game". The player sees ten sentences one by one. The first sentence is always written by human, but for each subsequent sentence the player must determine whether it is machine-generatee or human-written. If the player believes the text was Al-generated, they should explain why they think so. If they guess machine generation before the true boundary, they earn zero points. Otherwise, they earn 5 5- points, where x is the number of sentences after the correct boundary. This is actually a harder problem than just boundary detection since the player does not see the full text, and the scoring function is skewed. As a result, Dugan et al. (2023) created dataset also known as RoFT, where every sample consists of ten sentences, starting with a human-written prompt and continuing by some language model; the data shows the true boundary, human prediction of it, the generator model, explanation provided by the player, and information about the player. The original RoFT contains generations from GPT-2 (Radford et al., 2019), GPT-2 XI, GPT-2 finetuned on the "Recipes" domain, GPT-3.5 (davinci), CTRL (Keskar et al., 2019) with control code "nocode" " and with control code "Politics", and baseline, where instead of an LLM-generated continuation the passage transitions to a completely different news article selected at random. RoFT has been mostly used to investigate how humans detect artificially generated texts manually (Clark et al., 2021); Cutler et al. (2021) provided the first baselines on automatically solving ROFT, comparing several shallow classification and regression models based on RoBERTa and SRoBERTa (Reimers & Gurevych, 2019) embeddings collected from the last layer of these models. However, the cross-domain and cross-model settings in their research were very limited. In fact, Cutler et al. (2021) call "Out of Domain" (OOD) classifiers trained on all available data and then evaluated on a given subset, and they call "In Domain" (ID) classifiers that had been trained and evaluated on the same subset. Besides, they did not analyze all generators and domains within this cross-domain setting. In this work,
and how humans can learn to recognize
roft.
3

---

Published as a conference paper at COLM 2024
concentrate cross-domain and cross-model settings and interpretability, evaluating boundary detectors on unseen generators (models) and topics (domains).
Other related work. Zeng et al. (2023) used TriBERT-based approach for artificial text boundary detection in student essays for educational purposes, but with no cross-domain problem setting. Wang et al. (2023) address mixed human and AI generations by sentence- level classification, but they do it with a white-box detector that needs access to the generator model, which severely limits applicability. Recently, SemEval workshop' introduced small artificial text boundary detection dataset, namely subtask C of Task 8 The subtask asks to predict the word where human text ends and artificial text begins. However, to this moment there is no information on (a) which model was used to generate the samples, (b) what are the particular domains or topics of the generations, and (c) any human baselines. Existing technical reports on the competition (Spiegel & Macko, 2024; Datta et al., 2024; Rad et al., 2024) concentrate on other subtasks of Task 8 such as binary ATD and LLM-authorship attribution, but not the boundary detection subtask.
3 Approach
We consider several different approaches, including a multilabel classification framework, as proposed by Cutler et al. (2021), where the label of a text corresponds to the number of the first generated sentence, time series analysis that slides window over the text tokens, and regression methods that minimize the MSE between true and predicted boundaries. We design our classifiers based on features that have been successfully used in prior works on ATD (Solaiman et al., 2019; Mitchell et al., 2023). We also introduce new baseline based on sentence lengths. Below, we consider all of these approaches in detail.
RoBERTa classifier. Unlike Cutler et al. (2021) who process each sentence separately, we fine-tune the RoBERTa model (Liu al., 2019) to represent the entire text sample via the [CLS] vector. This is the only case in our work where we apply full model fine-tuning, For other methods, we use simpler classifiers such as logistic regression (LR) or gradient boosting (GB) (Friedman, 2001) trained on various features extracted directly from larger models ge Transformer-based LLMs), with no updates to the larger model's weights.
Perplexity from generative LLMs. In our perplexity experiments, we implement the black-box approach, which means that a single model is used for all data to compute sentence- wise perplexity. This is more practical than the method of Cutler et al. (2021) who used perplexity scores from the original generator model, which may be infeasible if the generator unknown, in particular in cross-model scenarios. Even for known generators, the exact version of the model is usually unknown if they are available via API, which harms the performance of white-box detectors. To derive perplexity features, we calculate the log likelihood of each token via LLM, and derive sentence-level mean and std values. As the perplexity estimator, we use several models such as GPT-22 (Radford al., 2019), Phi-1 (Gunasekar et al., 2023), Phi-1.5 (Li et al., 2023b), Phi-2 (Abdin et al., 2023), and LLaMA-2- 7B (Touvron et al., 2023). Our underlying hypothesis here is that texts generated by models of similar architecture such as GPT-2 or GPT-3.5 might appear more "natural" " to these models, as reflected by their likelihood scores. Our findings corroborate this assumption (see Appendix F). On top of sentence wise perplexity features, we train either a classifier (logistic regression and gradient boosting) or a regressor that predicts boundary values (GB regressor); the regression setting takes advantage of the sequential nature of the task, minimizing the discrepancy in labels.
DetectGPT. The DetectGPT framework (Mitchell et al., 2023) proposes more nuanced perplexity-based scoring function. It involves perturbing a text passage and comparing log-probabilities between original and altered texts. This score then serves as an input to a classification model, with GPT-2 as the base model and T5-Large (Raffel et al., 2020) generating the perturbations.
4

---

Published as a conference paper at COLM 2024
Length-Based Baseline. Since we have observed statistical difference in sentence length distributions between human-written and generated texts (see Fig. 2), we leverage sentence lengths as simple baseline feature. This baseline allows to gauge the effectiveness of classifier in identifying boundaries without semantic understanding.
Topological Time Series (TTS). Inspired by Tulchinskii et al. (2023a), we explore the poten- tial of topologiaal features based on intrinsic dimensionality (ID); we provide an introduction to topological data analysis (TDA), including the definitions of features, in Appendix A. We hypothesize that geometric variations in token sequences can help identify Al Al-generated text, so we introduce models that process -Dseed features treating them as time series. For every text, we slide a window of H = 100 tokens (step size S = 1) over RoBERTa token embeddings and find the intrinsic dimension (PHD) of the points within the window, as shown in Fig. 4 (Schweinhart, 2020). Window size 100 was chosen as a minimal size where PHD estimator stayed stable. The time series are then classified with multi-label SVM with the global alignment kernel (GAK) (Cuturi, 2011), where label correspond to number of the first generated sentence; this method is called "PHD + TS ML" in the tables.
Topological binary classification. In this approach, we train a binary classifier to distinguish between fuke and natural text atop a specific predictor. For the base predictor, we employ intrinsic dimensionality calculated over sliding window of H 20 tokens (step size S 5 5).) The TLE (tight local) intrinsic dimension estimator was chosen due to its robust performance on small data samples; window size 20 was taken directly from Amsaleg et al. (2019), as the authors claim that their algorithm is stable enough "for 'tight' localities consisting of as few as 20 sample points". For the base classifier, we use gradient boosting trees (Friedman, 2000; 2002). To translate probabilities predicted with binary classifier into a specific boundary that separates real and fake text, we determine the final label by maximizing y + arg max sr(y, x), where the score function S/ defined as sy x)
- E Tyy log (xx)), and p(x) are predictions from the base binary classifier. In this context, we use a binary indicator set I E 1" consisting of a chunk of k zeros followed by another chunk of k ones. This method is called "TLE TS Binary" in the tables.
Zero-shot detection. As an additional baseline, we tried zero-shot boundary detection with two strongest black-box models: gpt-3.5-turbo and gpt-4. We describe the problem to them with natural language prompt which is provided in Appendix B.
4 Experimental evaluation
Dataset preparation and analysis. In all experiments, the task is to detect the exact boundary where text passage that started as human-written transits to machine generation. In addition to RoFT (Dugan et al., 2020), we created a new dataset called RoFT-chatgpt, where the same human prompts are continued with the gpt-3.5-turbo model. RoFT-chatgpt is designed to be more challenging for artificial text boundary detection while preserving the basic statistical properties of RoFT such as the label distribution. We removed duplicates that were different only in human-predicted samples containing "As an AI language model..." " and short failed generations. As a result, we retained 8943 samples from the original RoFT and 6940 samples in RoFT-chatgpt. Note that distributions of sentence lengths vary significantly across subdomains, as shown in Fig. 2 and Figs. 5, 6 in Appendix E (see also discussion in Section 5).
labels; for RoFT-chafgpt, we also removed
Artificial text boundary detection. Table presents the main results of our experiments on the RoFT and RoFT-chatgpt datasets; the majority class prediction is the last (9th) class of fully human-written texts. On the original RoFT, we also include the human baseline and the best result reported by Cutler et al. (2021), obtained by classifier built upon concatenated SRoBERTa embeddings of each sentence. We report the accuracy (Acc) and two metrics that account for the sequential nature of boundary detection: soft accuracy, i.e., the percentage of predictions that differ from the correct label by at most one (SoftAccl; indeed, Fig. 18
6Stride size 5 was chosen empirically after the preliminary experiments. In these preliminary experiments we found out that for smaller stride sizes the series became too long for the Time Series SVM algorithm, and it wasn't able converge.
5

---

Published as a conference paper at COLM 2024
Table 1: Boundary detection results. Bold shows the best method, underlined second best, italic for values outperforming the human baseline.
Method
RoFT SoftAcel
RoFT-chatgpt SoftAccl
Acc
MSE
Acc
MSE
RoBERTa SEP
49.64 %
79.71 %
02.63
54.61 %
79.03 %
03.06
RoBERTa
46.47 %
74.86 %
03.00
39.01 %
75.18 %
03.15
Based on Perplexity
DelectGIT GB classifier
19.79
37.40 %
08.35
21.69 %
43.52 %
06.87
DelectGIT LR classifier
19.45
33.82 %
09.03
15.35 %
41.43 %
07:22
GPT-2 Perpl. + GB classifier
24.25 %
47.23 %
11.68
34.94 %
59.80 %
07.46
GPT-2 Perpl. + GB regressor
12.58 %
36.67 %
06.89
19.74 %
54.03 %
04.89
GPT-2 Perpl. LR classifier
23.75 %
42.15 %
15,80
33.50 %
57.56 %
09.25
Phi-1 Perpl. GB classifier
20.9 %
42.7 %
12.92
36.5 %
56.0 %
09.13
Phi-1 Perpl GB regressor
10.7 %
33.0 %
07.51
17.4 %
49.9 %
05.51
Phi-1 Perpl. LR classifier
19.2 %
38.3 %
16.26
31.0 %
53.9 %
09.76
Phi-1.5 Perpi. GB classifier
29.7 %
55.0
09.50
50.8 %
74.1 %
04.59
Phi-1.5 Perpi. GB regressor
17.1 %
44.6
06.11
32.0 %
71.0 %
03.07
Phi-1.5 Perpl. LR classifier
27.0%
49.5
11.90
47.3
72.7
04.77
Phi-2 Perpl. GB classifier
35.4
58.3%
08.80
48.4 %
70.3
05.39
Phi-2 Perpl. GB regressor
15.6 %
05.83
30.8
%
03.33
Phi-2 Perpl. + LR classifier
25.8 %
49.5
11.78
44.4 %
71.5 %
05.42
LLaMA-2-7B Perpl. GB classifier
22.3 %
42.0 %
14.79
41.5 %
65.0 %
06.32
L4MMA-2-7B Perpl. GB regressor
11.4 %
32.9 %
7.98
23.6 %
60.3 %
03.89
MaMA-2-7B Perpl. L.R classifier
19.4 %
34.7 %
19.75
29.7 %
58.4 %
07.90
Based on TDA
PHD TS ML
23.50
46.32 %
14.14
17.29
35.81 %
14.45
TLE TS Binary
12.58
30.41 %
22.23
20.02
34.58 %
18.52
Zero-shot
gpt-3.5-turbo
02.5 %
17.3 %
30.3
07.1 %
23.9 %
25.5
gpt-4
06.4 %
22.0 %
27.0
09.1 %
25.0 %
24.1
Length + GB
14.64
33.43 %
16.55
25.72 %
46.18 %
18.99
Majority class
15.26 %
25.43 %
27.58
13.83 %
24.42 %
26.46
SRoBERTa (Cutler et al., 2021)
42 %
Human baseline
22.62 %
40.31 %
13.88
in Appendix F shows that many misclassifications point to adjacent class, which may be acceptable in real world applications), and mean squared error (MSE). More precisely, SoftAcc1 = 1 E 1 (1y->//s1) and MSE = Σ (yr - 9.) where y; predictions of the algorithm, yi true labels, amount of the examples.
First, on the original RoFT RoBERTa-based classifiers outperform others by large margin (14% accuracy) and also significantly outperform the previously best reported SRoBERTa (Cutler et al., 2021). This model also provides the lowest MSE (0.03) among all methods. We note, however, that our RoBERTa classifier has significantly more trainable parameters than any other method in the table because no other approaches require LM fine-tuning, Second, topological and perplexity features improve over the human baseline. Perplexity- based classifiers are the best in terms of accuracy, while the perplexity regressor provides good MSE values (recall from Section 2, however, that humans were solving harder problem with a somewhat different objective). Third, RoBERTa' 's accuracy on RoFT-chalgp! drops by 6% compared the opposite holds for perplexity-based methods: on RoFT-chatspt their results are much better. The reason for this might be that we used GPT-like models for perplexity calculation and GPT-3.5-turee to generate fake samples in RoFT-chatgpt. However, we used smaller models detect text generated by larger model (GPT-3.5) and got second-best results among other approaches, despite Mitchell et al. (2023) reporting that smaller models are not capable detecting text generated by larger models. Moreover, when analyzing the performance of different perplexity estimators, performance with the size of the estimator. The largest LLaMA model with 7B parameters works much worse than Phi-1.5 (1.3B) and Phi-2 (2.7B) on both datasets and slightly worse
RoFT, while soft accuracy and MSE are roughly the same, but
do not observe clear correlation of the

---

Published as a conference paper at COLM 2024
Table 2: Accuracy for leave-one-out cross-domain evaluation on RoFT-chatgpt. A and V show relative change from the model's in-domain score to the human score; A and v, relative change from the out-of-domaiv score to the in-domain score. Green highlights improvements, red, deteriorations. The table shows only perplexity methods based on perplexity of Phil.5 and Phi2; for other perplexity backbones, see Appendix C.
Pres. Speeches
Recipes
New York Times
Short Stories
Avg
Pred. Model
Context
NN
OUT
IN f
OUT f
N
OUT 1 IN 1
OUT t
Text RoBERTa SEP
global
57.3x153%
15% 43.2.91%
13.1
53.2.135%
38.1
54.3.100%
27.
48%
Text RoBERTa
global
67.6.199% 36.3
63.7%
74.8
54.2.137%
38.0v3%%
64.8.183%
v4.1
52%
Perpl. Phil.5, GB
sent.
52.5x132% 52.0+1%
60.4.167%
24.1vs0%
52.5 45.7v12%
48.7a113%
56.1415%
15%
Perpl. Phil,5, LR
sent.
48.7a115%
41.2v15%
60.9
21.1v65%
50.8
15.2
47.1.108%
51.5
21%
Perpl. Phi2, GB
sent.
50.16121%
46.1
55.66
146%
22.8+41%
55.0a143%
w23%.
48.16113%
53.9
15%
Perpl. Phi2, LR
sent.
49.6.119%
42.1
56.4. 149%
24.4
50.5
a2.5%
44%4
48.5.114%
52.0a7%.
16%
PHD TS multi
100 tkn
13.7 200%
19.2
19.5 02%%
20.9 108%
17.2
21.2.006%
17%.
16%
TLE
TS Binary
20 tkn
25.6
14.7+42%
27% 16.5
25.0a11%.
w32%.
932%
22.1 pozes
11.1
31%
Best combination
63.0a179%
43.2
67.6
199%
20%0
665.0
47.5%
60.9.169%
17.4
42%
Len
GB
sent.
28.1.24%
15.5va64
a0.4
18.4
32.3
44%
Len
LR
sent.
19.6
v7.0%
12.9
22.2 V12%
09.1
v50%
22.9.01%%
09.1
47%
Majority
13.4
13.0
15.9
17.7
Approx. human global
22.62
22.62
22.62
22.62
than GPT-2 (1.5B) on RoFT. For RoFT-chatgpt, the best perplexity estimator is a relatively small Phi-1.5 model, outperforming GPT-2 estimator of comparable size by 15%. This impressive difference might be related to training data: Phi-1.5 was trained on GPT-3.5 generations. This suggests that small language models trained on data generated by LLMs, may appear good perplexity- based detector of text generated by those particular LLMs. The baseline length-based classifier also improves its accuracy significantly (by 1.8x) when transferring to RoFTT-hatgpt. We hypothesize that this kind of shallow feature emerges in ChatGPT generation and makes the task easier (see also Section 5). The other perplexity- based approach, DetectGPT, shows lower accuracy on both datasets. Mitchell et al. (2023) note that DetectGPT can detect whether the sample is generated by specific base model, but we use several models in our setup, and text samples may be too short for this approach. On the other hand, DetectGPT has quite good MSE values, close to the regression approach that optimizes MSE directly.
Cross-domain generalization. Supervised ATD methods with fine-tuning such as RoBERTa are more sensitive to spurious correlations in a dataset and often demonstrate poor cross- domain transfer, especially compared to topology-based approaches (Tulchinskii et al., 2023a). Table reports the results of cross-domain transfer between four text topics in the RoFT-chalgpt dataset. We report in-domain (IN) accuracy on domains seen during training and out-of-domain (OUT) accuracy on the unseen domain corresponding to this column; MSE scores are given in Table 5 (Appendix C). Each model was trained on three domains and tested on the fourth, unseen domain; we used 60% of these subsets mixed together as the training set, 20% as the validation set, and 20% as the test set for in-domain evaluation. Table 2 shows that RoBERTa's performance drops for all subsets very significantly, while perplexity-based classifiers demonstrate excellent cross-domain generalization for Presiden- tial Speeches and Short Stories; for the Recipes domain, TTS classifiers prove to be the most stable, and the New York Times subset yields large generalization gap for all classifiers. This means that every type of classifier can handle its own set of spurious features well, and no classifier is universally better than the others. Aggregation of different features can improve the results; for example, a classifier that returns the maximum of RoBERIa and perplexity-basee predictions yields OOD scores on average 3% higher than the best result in Table 2 with comparable ID performance. As for classification accuracy, surprisingly, the perplexity-basee classifier outperforms a fully fine-tuned RoBERTa on all subsets. Moreover, for Recipes the multilabel topological time series method places second; this happens because TDA-based methods are extremely stable under domain shift (despite being worse than others in absolute values on this dataset); see Section 5 for further discussion. In general, we conclude that perplexity-basee classifiers outperform RoBERTa-bass settings by large margin. This is remarkable since the former is simple classifier trained
in cross-domain
7

---

Published as a conference paper at COLM 2024
Table 3: Cross-model transfer, original RoFT. Models are trained on all parts except one and
tested in-domain (ID) on the same parts and out-of-domain (OOD) on the remaining part.
Model
GPT2-XL
GPT2
davinci
ctrl-Politics
ctrl-nocode ID
tuned
ID
0OD
ID
OOD
ID
OOD
ID
OOD
OOD
ID
OOD
RoBKRTa + SHP
46.4
40.9
46.0
08.8
63.3
19.7
49.4
59.1
50.2
60.6
49.6
23.9
RoBERTa
46.7
32.5
40.3
07.0
57.2
14.5
47.5
44.0
46.1
54.6
46.0
20.1
Phi-1.5 Perpl. GB
26.0
27.0
31.5
12.2
30.5
15.3
30.8
23.9
30.3
19.2
34.7
12.9
Phi-2 Perpl. GB
34.1
26.4
32.0
17.6
35.9
16.2
34.1
19.5
34.3
23.2
37.2
12.4
PHD + TS ML.
31.8
04.0
25.6
04.5
31.4
02.1
25.7
08.2
25.2
11.1
21.4
12.6
TLE + IS Binary
14.2
03.2
12.4
07.8
14.5
01.0
11.0
05.0
11.5
04.0
14.5
07.0
Length + GB
21.7
04.8
18.3
01.2
19.2
06.3
17.3
03.8
18.1
00.0
21.6
00.0
Human Baseline
22.5
17.2
22.6
22.5
24.7
14.1
22.6
21.6
22.6
23.9
21.5
25,9
ten features extracted by LM with frozen weights, while the latter involves full LM fine-tuning. As for the length-based baseline, for in-domain data average sentence length provides strong signal, leading to accuracy of 20-32% and even outperforming topolog- ical methods. But cross-domain generalization fails, which means that we should prefer classifiers that ignore this feature achieve good generalization (see also Section 5).
Cross-model generalization. We tune our classifiers on generation results produced by one model and test the performance for all other models. In general, this task is harder for all considered classifiers: there are models for which prediction accuracy drops down to virtually zero values (detailed experimental results are shown in Appendix D: Tables 3, 6, 7). But we observe an interesting result for the perplexity-based classifier: it achieves good generalization when transferring to very large models such as GPT3-davinci and GPT2-XI., while for other models the generalization is poor. In general, the strongest classifiers fail to generalize to simpler models; e.g,, RoBERTa-based classifiers underperform on the generalization to GPTT22 and baseline; the baseline is also the hardest subset for perplexity-base classifiers (see Section 5 for a discussion).
Discussion and analysis of the results
Detection models. We have found that although RoBERTa- classifiers demonstrate excellent results for in-domain classification, they lose to perplexity-basee methods when tested on texts with new styles and topics not present in the training set. For cross-model generalization, all methods demonstrate insufficient generalization abilities, especially when tested on generations much larger or much smaller models. This suggests that Al content detectors can be fooled by either too good or too bad artificial texts. For perplexity- -bsedd classifiers, we have found that the size of the base model is not the main requirement for good and robust detector. The training dataset for perplexity estimation is more important; e.g., ChatGPT generations were best detected by much smaller model pretrained on synthetic data generated by the same model family. The above observations suggest that pre-training of the perplexity estimators on a large amount of synthetic data from different generators may become the best choice for Al-generated content detection. We leave for future research. Topological classifiers demonstrated high robustness over domain transfer, but not very high overall accuracy. Geometric properties of the embeddings for both RoFT and RoFT-chalgpt, however, show the difference between PHD distributions of real and fake RoFT text for different models (Fig. 10) and topics (Figs. 11, 12 in Appendix E). TLE dimension distributions for sentences in human- and Al-generated parts of the texts are different as well (Figs. 13, 14, 15, Appendix E). This means that topological classifiers may help in robust artificial content detection in some particular domains and models.
Data properties. Next, we summarize the observed properties of the data that influence detection quality and create difficulties for different types of artificial content detectors.
1. First, the length of sentences seems to play an important role, deceiving classifiers. There is indeed significant difference between distributions of sentence lengths written by humans and generated by LLMSs (see Fig. 2 and Figs. 5, 6 in Appendix E). This is supported
8

---

Published as a conference paper at COLM 2024
1000
fully natural
natural baseline
natural Politics
40
natural
10
50
ctrl- nocode
500
20
natural davinci
natural finetumed
400
natural
1000
natural
1000
gpt2
gpt2-xl
200
500
0
50
100
50
100
0
50
100
0
50
100
Figure 2: Sentence length distributions in RoBERTA tokens, original RoFT, by model
83
41
0
0
78
47
10
70
11
74
25
2
2
0
0
70
34
72
34
0
29
0
2
60
2
18
46
70
60
50
15
40
True
3
15
35
41
15
40
24
53
11
2
0
14
20
19
21
21
30
3
12
20
31
5G
30
6
30
0
14
17
13
26
36
15
7
3
7
13
18
25
62
67
34
4
20
1
14
16
13
20
28
27
10
8
7
15
30
54
46
52
33
10
10
0
5
6
1
6
7
8
label
label
268
16
13
2
0
0
250
185115
15
28
11
2
5
a
200
0
1
o
66
Truc labol
2
154
40
38
83
0
150
179
36
31
57
41
14
16
0
2
14
121
23
11
31
30
31
0
100
0
1
1
0
131
14
10
22
23
50
0
0
0
13
13
11
50
0
3
4
8 9
3
4
8
9
Predicted label
Predicted label
Figure 3: Confusion matrices of RoBERTa predictions on the four domains in the RoFT-
chatgpt dataset: New York Times, Short Stories, Presidental Speeches, Recipes (left to right).
by our experiments with the length-based classifier, which sometimes outperforms other methods in terms of accuracy (Tables and 2), but fails to generalize.
Label distributions (see Figs. 8 and Appendix E) vary significantly across models. GPT-2 is especially different from natural texts, and the behavior of our models in cross- model transfer to GPT-2 (Appendix D), suggests that the only model stable under the label distribution shift is perplexity-bar based regression. Although its accuracy numbers are low, it outperforms the human level in terms of MSE even out-of-domain evaluation.
Special text strucfure can hurt ATD: e.g., recipes often contain numbered lists, and the first sentence tends to be comparatively long, and result the Recipes topic is hard for all classifiers to generalize to. The RoBERTa classifier loses the most here, so we investigated its confusion matrices; Fig. 3 shows that RoBERTa tends to classify all examples in the Recipes topic as fully generated. This kind text structure can be viewed as an adversarial example for Al-generated texts, and indeed, ChatGPT is known for its tendency to generate structured output with bullet lists.
Semantic and grammar inconsistencies. Surprisingly, cross-model generalization of RoBERTa-basee and perplexity-based classifiers fails not only for the best OOD genera-

---

Published as a conference paper at COLM 2024
tor (davinci), but also for the worst generators: small-size GPT-2 and baseline. Perplexity distributions for texts generated by baseline or GPT-2 are very different from the distributions of texts generated by other models (see Fig. 16). This might explain the poor performance of decision trees and linear classifiers when these generators are excluded from the training set. On the other hand, both generators are among the easiest for human raters, because of the high level of semantic inconsistencies and grammar issues. It seems that the supervised classifier discovers non-regularities in the text as feature of -eeaanneeeeaated texts; indeed, they present in our training set as part of first-person narratives, especially in the Short Stories dataset.
5. Discourse structure. Finally, analysing the poor generalization to narrative domains, we discover a subtle property of human-generater texts that influences boundary detection, namely the underlying discourse structure of the story that can be measured as peaks of perplexity along the story (Fig. 17, Appendix E). Human-gennrates texts tend to have 2-3 such peaks, e-g. peak in the middle of the story corresponding to plot twist change of narrative focus. This leads to the difficulties in perplexity-basee boundary detection: the perplexity drop in the beginning of the fake text part is confused with the same drop due to the story discourse change. Strong perplexity-basee classifiers can handle this issue due to the std feature reflecting token-level perplexity deviation (Fig. 18, Appendix E). Fig. 2b shows that the KoBERTa-bases classifier is prone to this discourse-related issue: it often incorrectly predicts the real-fake boundary in the middle of a human -emaanneeeeraed story (5-6th sentence).
full analysis of feature distributions across domains and models is given in the Appendix.
6 Conclusion
In this work, we address the task of boundary detection between human-written and generated parts in texts that combine both. We believe that this setting is increasingly important in real world applications and is a natural setting for recognizing artificial text since it is often mixed with and prompted by human text. We have considered the RoFT dataset, presented its modification RoFT-chatgpt generated with more modern LLM, and investigated the performance of features that were useful for artificial text detection in previous works. In particular, we have shown that LLM fine-tuning works reasonably well for this task but tends to overfit to spurious features in the data, which leads to generalization failures in some settings. On the other hand, perplexity-basee provide signal that can help in these situations. We have demonstrated that perplexity features are the best overall on balance between accuracy, generalization, and training complexity.
and topological features
We analyze the base models for perplexity estimation and conclude, that reasonably small models work well, but pre-training data is important. Including synthetic data in the pre- training dataset improves the performance of the detector. As for the limitations of our approach, boundary detection is very hard in cross-domain and cross-model settings, both for short texts such as RoFT (due to lack of information) and longer texts (due to large space of possibilities). Therefore, really high quality in this setting, and the results suggest large room for improvement. Besides, all methods we considered were based on Transformers with relatively small context window size, which limits the transferability of the proposed approaches onto longer text samples. Finally, our analysis has uncovered gaps in current approaches and discovered the more difficult aspects of the task, which we plan to address in future research.
no wonder that none of the methods have achieved
Acknowledgments
The work of Sergey Nikolenko was performed at the Saint Petersburg Leonhard Euler International Mathematical Institute and supported by the Ministry of Science and Higher Education of the Russian Federation (agreement no. 075-15-2022 dated 06/04/202//
10

---

Published as a conference paper at COLM 2024
References
Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Mojan Javaheripi, Piero Kauffmann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang. Phi-2: The surprising power of small language models. December 2023.
Henry Adams, Manuchehr Aminian, Elin Farnell, Michael Kirby, Joshua Mirth, Rachel Neville, Chris Peterson, and Clayton Shonkwiler. A fractal dimension for measures via persistent homology. In Topological Data Analysis: The Abel Symposium 2018, pp. 1-31. Springer, 2020.
Laurent Amsaleg, Oussama Chelly, Michael E Houle, Ken-Ichi Kawarabayashi, Miloš Radovanović, and Weeris Treeratanajaru. Intrinsic Dimensionality Estimation within Tight Localities. In 2019 SIAM International conference on Data Mining, pp. 181-189, Calgary (Alberta), Canada, May 2019. Society for Industrial and Applied Mathematics. doi: URL pittps//iirri hal science/hal-02112550
10.137/1.97816119557333
Serguei Barannikov. The framed morse complex and its invariants. Advances in Soviet Mathematics, 21:93-116, 1994.
Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli. Intrinsic dimension, persistent homology and generalization in neural networks. Advances in Neural Information Processing Systems, 34:6776-6789, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini
Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https:: neurennnnnnnnnnunnnne neurips... http/papp_fileeseeeeee /ttpe///ppppe/e/////// pdf.
Gunnar Carlsson. Persistent homology and applied homotopy theory. In Handbook of Homolopy Theory, pp. 297-329. Chapman and HaR//CRC, 2020.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7282-7296, Online, August 2021. Association for Computational Linguistics. doi: 1116653/v12221.aac--2 565.
URL https: https///aaaaathloooo orw.oogg/// a acl-long.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre- training text encoders discriminators rather than generators. In International Conference on Learning Representations, 2020. URL httpp::/openrrww http://ee/ooottooeeee (oorum?id=r11MM1BBrr
Joseph Cutler, Liam Dugan, Shreya Havaldar, and Adam Stein. Automatic detection of hybrid human-machine text boundaries. 2021.
Marco Cuturi. Fast global alignment kernels. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 929-936, 2011.
Ayan Datta, Aryan Chandramania, and Radhika Mamidi. Semeval-2024 task 8: Weighted layer averaging roberta for black-box machine-generated text detection, 2024.
11

---

Published as a conference paper at COLM 2024
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/ N19-1423, URL https:
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Ballison-Burch. RoFT: A tool for evaluating human detection of machine-generated text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 189-196, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/ v1 21/2020.emnlp-demon 25.
URL htw.t: httpp:://aclanthology org/
org/2020.22.. 2020. memlpp-emos... 25.
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison- Burchh. Real or fake text? investigating human ability to detect boundaries between human- written and machine-generated text. In Proceedings of the Thirty-Seventh AAAI Con- ference Artificial Intelligence and Thirty-Fifth Conference Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intel- ligence, AAAI'23 23AAA1333 CAAAAA////AAA"/ EAAI'23, AAAI Press, 2023. ISBN 978-1-57735-88D0 doi: 10.1609/ http///aaav37111110 26601. URL https://doi.oor//0... wrg/10.////// 11110..161///1 1009/ aaai.
Innovative Applications of
Naiereh Elyasi and Mehdi Hosseini Moghadam. An introduction to new text classification and visualization for natural language processing using topological data analysis, 2019.
Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29:1189-1232, 2000.
Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189-1223 1232, 2001. ISSN 00905364. URL
org/
stable/2699986.
Jerome H. Friedman. Stochastic gradient boosting, Computational Statistics & Data Analysis, 3874)3667-378, 2002. ISSN 0167-9473. doi: http:://cc..ooooo//ooooo////s220
URL https: htww://wwwwccooo www.sscincediieeee rect. com/ttccoo/eeeeeeeee//e www./ooc/inneeeeeee http:///iieeerrrreee//////////// linear Methods and Data Mining.
Non-
Karl Pearson F.R.S. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901. doi: 10.1080/
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, June 2023. URL https://wwwt http:///wwwmmccooo..ttomm wtt.:/nirrooof...ommmmess/sseeee comm//enuss/reeerrc//ullllll textbooks-ane wwwe-arl-looooeeeeeeeeeeleeellle all-you-need/.
Aarish Iyer and Soroush Vosoughi. Style change detection using bert. In Conference and Labs of the Evaluation Forum, 2020. URL https://apl: wwwpppp.sematiiiiimmeeemmmmmimme semanticscholart wtw.ttiichooa.aooooorreeeeeeeeee org/CorpusID: 225073958.
Xinyin Jiang, Haoliang Qi, Zhijie Zhang, and Mingjie Huang.
Style change detection: Method based on pre-trained model and similarity recognition. In Conference and Labs of the Evaluation Forum, 2022. URL https: //api. semanticscholaa 2ttpaaarr.rrrr/orpss1:552555555 471100
Keenan Jones, Jason RC Nurse, and Shujun Li. Are you robert or roberta? deceiving online authorship attribution models using neural text generators. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pp. 429-440, 2022.
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL- - A Conditional Transformer Language Model for Controllable Generation. Xiv preprint ar XrXit:1999005855 2019.
12

---

Published as a conference paper at COLM 2024
Laida Kushnareva, Daniil Chernaavski,, Vladislav Mikhailov, Ekaterina Artemova, Serguei Barannikov, Alexander Bernstein, Irina Piontkovskaya, Dmitri Piontkovski, and Evgeny Burnaev. Artificial text detection via examining the topology of attention maps. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 635-66 649, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (0.18653/v1/2/1 w11/02l.emnll-main..5. URL https: //aclanthology. org/2021. emnlp-main 50.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representa- tions. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:/ wtwprreviiiieneefffrunu net/forum? ed=H1eA7AEtvS.
Qidi Lao, Li Ma, Wenyin Yang, Zexian Yang, Dong Yuan, Zhenlin Tan, and Langzhang Liang. Style change detection based on bert and convld. In Guglielmo Faggioli, Nicola Ferro, Allan Hanbury, and Martin Potthast (eds.), Proceedings of the Working Notes of CLEF 2022 Conference and Labs of the Evaluation Forum, Bologna, Italy, September 5th 8th, 2022, volume 3180 of CEUR Workshop Proceedings, Pp. 2554-2559. CEUR-WS.org, 2022.
URL https::/ceeurreeeeeeeerr ws. or 1001-3100pppper-22000
Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu. Origin tracing and detecting of llms. arXiv preprint or (rii:23044. 14072, 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report.
September 2023b. URL https://wwat wiwwwsoft..oooooooomommo http:://ww..icrroooft.. htmm//enus//rsaacc///ll/ publication/ textbooks-are-at aive-all-yye you-you- -eel--you--ee-iii peeed-i--phie
Tzu-Mi Lin, Chao-Yi Chen, Yu-Wen Tzeng, and Lung-Hao Lee. Ensemble pre-trained transformer models for writing style change detection. In Guglielmo Faggioli, Nicola Ferro, Allan Hanbury, and Martin Potthast (eds.), Procesdings of the Working Notes of CLEF 2022 Conference and Labs of the Ezaluation Forum, Bologna, Italy, September 5th to 8th, 2022, volume 3180 of CEUR Workshop Proceedings, pp. 2565-2573, CEUR-WS.org, 2022. URL https://ceur-ws.. wrg. -ow..oo0-0-111//ppprrrerr 210.perrr21..pppe .df.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly op- timized bert pretraining approach, 2019. URL http://arxiw..rr//bbttte arxiv:1907.11692.
11692. cite
German Magai. Deep neural networks architectures from the perspective of manifold learning. Xiv preprint ar Kiv:2306.03400 2023.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature, 2023. URL https: //axxiv www. abs//bbs/330. 11305.
Hariharan Narayanan and Sanjoy K. Mitter. Sample complexity of testing the manifold hypothesis. In NIPS, 2010.
Mohammad Heydari Rad, Farhan Farsi, Shayan Bali, Romina Etezadi, and Mehrnoush Shamsfard. Ríbes at 2emeval-2024 task 8: Investigating syntactic and semantic features for distinguishing ai-generated and human-written texts, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL http: jmmrr.///rr/.///// htt.mml..rrr./..eemmmmmmmmmo orggpppers//21111111 220p///////0222/ html.
13

---

Published as a conference paper at COLM 2024
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks, In Proceedings of the 2019 Conference on Empirical Methods in Natural Languge Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pp. 1982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: org/D19-1410.
URL https://acclatthoooogy
Benjamin Schweinhart. Fractal dimension and the persistent homology of random geometric complexes. Advances in Mathematics, 372:107291, 2020.
Primoz Skraba, Gugan Thoppe, and D Yogeshwaran. Randomly weighted d- complexes: Minimal spanning acycles and persistence diagrams. arXio preprint ar xrXiv:1701.00239, 2017.
Trene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint
2019.
Michal Spiegel and Dominik Macko. Kinit at semeval-2024 task Fine-tuned 1lms for multilingual machine-generater text detection, 2024.
J Michael Steele. Growth rates of euclidean minimal miniall spanning trees with power weighted edges. The Annals of Probability, 16(4):1767-1787, 1988.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
M. Eduard Tudoreanu. Exploring the use of topological data analysis to automatically detect data quality faults. Frontiers in Big Data, 5. doi: 10.3389/fdata (3t8899//aaaa2222991333388 URL https://par nsf. http/bbbllo/1042200
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Serguei Barannikov, Irina Piontkovskaya, Sergey Nikolenko, and Evgeny Burnaev. Intrinsic dimension estimation for robust detection of ai-generated texts, 2023a.
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Serguei Barannikov, Irina Piontkovskaya, Sergey Nikolenko, and Evgeny Burnaev. Topological Data Analysis for Speech Processing. In Proc. INTERSPEECH 2023, pp. 311-315, 2023b.
doi: 0..21437/Interspecchh22 -0tpeeechhh02-11180 1-61..
Adaku Uchendu, Thai Le, and Dongwon Lee. Attribution and obfuscation of neural text authorship: A data mining perspective. ACM SIGKDD Explorations Newsslettrr, 25(1):1-18, Jun 2023. ISSN 1931-0153. doi: 10.1145/36662443666666 URL http://dx. doi. org/10. 211633666274.3666777
Saranya Venkatraman, Adaku Uchendu, and Dongwon Lee. Gpt-who: An information density-based machine-generated text detector. Xiv preprint arXio:2310.06202, 2023.
Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. SeqXGPT: Sentence-level Al-generated text detection. In Houda Bouamor, Juan Pino, and Kalika Bali
14

---

Published as a conference paper at COLM 2024
(eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 1144-1156, Singapore, December 2023, Association for Computational Linguistics. URL https::/acclanthlloee org/ emnlp-main. 73.
Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltýnek, Jean Guerrero-Dib, Olumide Popoola, Petr Šigut, and Lorna Waddington. Testing of de- tection tools for ai-generated text. International Journal for Educational Integrity, 19(1):
2023.
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F. Wong, and Lidia S. Chao. A survey on llm-generated text detection: Necessity, methods, and future directions, 2023a.
Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Limdet: A third party large language models generated text detection tool. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp- 1113-213,, 2023b.
Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. A survey on detection of lims-generated content. ar preprint ar arXiv:2310.1563 2023.
Eva Zangerle, Maximilian Mayerl, Martin Potthast, and Benno Stein. Overview of the style change detection task at pan 2021. In Conference and Labs of the Evaluation Forum, 2021.
URL httpt://p//pp. www.nticschol.aaa.. wtt.hhooar...rg/oooosSDD-- 201.:opp 260810617 0617.
Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gašević, and Guanliang Chen. Towards automatic boundary detection for human-ai hybrid essay in education. Xiv preprint Xriv:2307.122667 2023.
A Intrinsic Dimension Estimation Methods
According to the manifold hypothesis (Narayanan & Mitter, 2010), the data X lies on low-dimensional submanifold: X S M S Rd, where d is the extrinsic dimension and n is the intrinsic dimension (ID). The geometric and topological properties of the manifold M are of particular interest. There are various methods for estimating the ID that can be divided into global and local methods.
For the tight local intrinsic dimension estimator (TLE) proposed by Amsaleg et al. (2019),
use the neighborhood center point x, a set of neighborhood samples V and specially defined distance between points in sufficiently small neighborhood of x:
r(v - 9) 9) (v-
dx
(1)
2(x 9) (P 9)
where is the radius of the neighbourhood. For every three points v, w can compute
M(x, v, w) II dx (v,
+ In (2x w)
V. = (x),, then the intrinsic dimension can be found by averaging the estimates for all points x, defined by the following formula:
Σ M(X,,,))
m,(x) =
(2)
V.
VADE V., v//w
Applied algebraic topology provides effective tools for analyzing the topological structure of data. The theoretical foundations of topological data analysis (TDA) have been described
in detail by, e.g., Barannikov (1994) and Carlsson (2020). TDA allows to consider
dataset X C Rd from the topological point of view. In order to move from point clouds X to
15

---

Published as a conference paper at COLM 2024
Inteinsic Dimension Estimation
gen
800
B
B
B(K)
BlA + 1)
Bin 1)
B(n)
10
m
embedddggga
125
(b) Sample result
(a) Sliding window technique
Figure Multilabel time series classification: (a) estimating the intrinsic dimension of token embeddings in a sliding window, (b) sample resulting series: green - human-written, orange machine-generated tokens.
topological spaces, it is necessary to approximate the data by a simplicial complex R. In our research, we use the Vietoris-Rips complex R (X; The method of constructing the complex R is as follows: simplexes are formed by subsets of points from X whose pairwise distances do not exceed / (a scaling parameter). An increasing sequence of simplicial complexes is called filtration: {Rt) (120) = R S R .... Rs.
Homology groups (R) are a topological invariant that expresses the properties of topological space R. We use (R) = dim HR(R), which is known as the ith Betti number, topological feature equal to the dimension of the homology group; for i 0, 1,2 the Betti number corresponds to the number of connectivity components, cycles, and cavities respectively.
Topological features appear and disappear at different values of t, which leads to the next core concept in TDA: the barcode. It summarises the dynamics of topological features in
the filtration process. bar is the lifetime of the nth homology feature Ln = phirth pdeath A long bar in the barcode means that the data contains fairly persistent and informative topological feature.
Schweinhart (2020) introduced the persistent homological fractal dimension (PHD) that generalized Steele (1988) for higher dimensions of the homology group and used the topological properties of the point cloud. The PHD has already been proven to be useful in the study of the properties of deep learning models (Birdal et al., 2021; Magai, 2023).
Let denote the power-weighted sum of N bars for the ith degree of homology follows:
N
(
=
(3)
is interesting to note that (Xx) is equal to the length of the Euclidean minimum spanning
tree (MST) of X,, S Rd (Skraba et al., 2017).
Then the persistent homological fractal dimension (PHD) can be defined as follows:
PHD 1 1-
(4)
where
log (og(EEEE (xI, ( xn)))
s = lim sup
(5)
00
log(n)
and Xj...., Xn are sampled independently from X. That is, PHD*(X) d if F (x) (,, xn) d scales as and a 2 (we take 0, a 1). Persistent homological fractal dimension can be estimated by analyzing the asymptotical behavior at n -+ co of E (x1...n)) for every i. In other words, to calculate PHD we must find a power law that shows how E (x)...,,) scales as increases. See Adams et al. (2020) and Schweinhart (2020) for more details.
16

---

Published as a conference paper at COLM 2024
Table 4: Accuracy for leave-one-out cross-domain evaluation on RoFT-chatgpt, full table.
Presidential Speeches IN OUT t
New York
Short
Pre-
Recipes IN t OUT
Times
Stories
dictor
Model
Context
IN 1 OUT f
IN
OUT
Text
RoBERTa SEP
global
57.3
31.4
43.2
13.1
53.2
38.1
54.3
28.6
Text
RoBERTa
global
54.2
40.2
39.7
15.1
53.3
34.0
50.3
27.7
Perpl. GPT2, GB
sentence
36.1
35.3
36.6
19.4
36.9
29.7
32.8
32.9
Perpl. GPT2, LR
sentence
34.8
32.4
40.6
20.4
36.7
28.9
32.7
35.6
Perpl. GPT2, Regr. (GB)
sentence
20.9
22.6
23.8
14.4
18.5
16.1
21.2
21.8
Perpl. Phil, GB
sentence
36.2
24.5
26.5
21.6
39.1
23.8
42.1
21.2
Perpl. Phi1, LR
sentence
33.6
27.4
25.9
26.2
37.4
25.5
35.5
22.9
Perpl. Phi1.5, GB
sentence
52.5
52.0
60.4
24.1
52.5
45.7
48.7
56.1
Perpl. Phil.5, LR
sentence
48.7
41.2
60.9
21.1
50.8
45.2
47.1
51.5
Perpl. Phi2, GB
sentence
50.1
46.1
55.6
22.8
55.0
42.1
48.1
53.9
Perpl. Phi2, LR
sentence
49.6
42.1
56.4
24.4
50.5
43.4
48.5
52.0
Perpl. LLaMA2-7B, GB
sentence
41.4
25.5
39.0
17.7
45.2
24.5
41.1
27.4
Perpl. LLaMA2-7B, LR
sentence
34.5
15.7
37.9
14.8
37.2
23.5
36.1
24.4
PHD TS multilabel
100 tokens
20.3
13.7
19.2
19.5
20.9
17.2
21.2
17.6
TLE
TS Binary
20 tokens
25.6
14.7
16.5
16.3
25.0
17.1
22.1
11.1
Length GB
sentence
28.1
11.8
21.1
15.5
30.4
18.4
32.3
15.8
Length LogRegr
sentence
19.6
10.8
17.0
12.9
22.2
09.1
22.9
09.1
Majority
15.4
13.0
15.9
17.7
Approximated human global
22.62
22.62
22.62
22.62
B gpt-3.5-turbo and gpt-4 zero-shot experiment details
Below we show the prompt used for gpt-3.5-turbo and gpt-4 zero-shot experiments.
You will be provided with a text that contains human-written sentences. Your task is to analyze the text and determine if any of the sentences were generated by Al system.
You should output single number between 0 and 9 to indicate where the first AI-
generated sentence occurs in the text (if any):
For your output, you should provide a single number between and 9, with the following meaning:
The entire text was generated by an Al system.
1 to 3--Te The first Al- Al-generated sentence occurs at that position in the text (1 for the for the second sentence, and so on).
first sentence,
9 The entire text written by human with no Al-generated sentences. Your output should consist solely of this single digit number, with no additional text explanation:
(content)
C Cross-domain transfer
Tables and supplement Table from the main text, reporting the results of cross-domain transfer for our methods on the RoFT-chatgpt dataset across four text topics present in the data. We report in-domain and out-of-domain accuracy: the column shows results from domains seen during training, while the OUT column reflects the model's ability to detect artificial texts in the unseen domain corresponding to this column. For each model, training was done three out of the four domains, and the resulting model was tested the fourth, unseen domain; we used 60% these subsets, mixed together, as the training set, 20% as the validation set, and 20% as the test set for in-domain evaluation.
17

---

Published as a conference paper at COLM 2024
Table 5: Mean squared errors from leave-one-out cross-domain evaluation on RoFT-chatgpt, full table.
Presidential Speeches IN OUT +
New York
Short
Pre-
Recipes IN 4 OUT
Times
Stories
dictor
Model
Context
IN
OUT
IN
OUT
Text
RoBERTa SEP
global
02.6
10.6
02.6
18.3
03.4
07.9
02.3
09.0
Text
RoBERTa
global
02.3
07.5
02.8
13.5
02.9
06.2
02.6
05.5
Perpl. GPT2, GB
sentence
07.3
08.9
07.0
14.6
07.2
09.4
08.5
08.5
Perpl. GPT2, LR
sentence
08.2
11.5
06.0
16.8
08.7
11.8
09.6
09.3
Perpl. GPT2, Regr. (GB)
sentence
04.7
06.6
04.8
09.7
04.9
05.7
05.2
05.0
Perpl. Phil, GB
sentence
09.3
11.8
10.8
15.1
08.5
14.2
08.7
14.7
Perpl. Phil, LR
sentence
09.8
11.9
11.5
12.9
08.9
12.2
09.1
13.5
Perpl. Phi1.5, GB
sentence
04.5
08.5
03.0
11.8
03.9
06.3
05.6
03.4
Perpl. Phil.5, LR
sentence
04.5
09.3
03.6
12.2
04.0
07.2
05.2
03.0
Perpl. Phi2, GB
sentence
04.3
07.3
03.0
15.5
04.3
06.6
04.6
04.3
Perpl. Phi2, LR
sentence
04.5
09.9
03.4
13.6
04.3
07.5
05.4
03.0
Perpl. Phi2, GB
sentence
04.3
07.3
03.0
15.5
04.3
06.6
04.6
04.3
Perpl. Phi2, LR
sentence
04.5
09.9
03.4
13.6
04.3
07.5
05.5
03.0
Perpl. L.AMAA--77B, GB
sentence
05.0
14.0
05.5
13.2
05.3
11.4
07.3
09.7
Perpl. LLAMA-2-7B, LR sentence
06.3
18.3
06.0
19.5
06.1
08.8
08.3
05.0
PHD TS multilabel
100 tokens
12.3
14.6
10.7
11.0
14.1
16.8
11.6
11.6
TLE
TS Binary
20 tokens
12.3
15.6
18.0
15.4
12.0
17.6
17.4
23.9
Length GB
sentence
12.8
15.9
14.2
17.9
13.7
18.4
12.5
15.3
Length LogReg
sentence
20.1
20.5
16.7
24.7
17.2
22.1
18.5
22.9
Majority
27.5
27.4
27.9
28.0
Approximated human global
13.88
13.88
13.88
13.88
D Cross-model transfer
Tables 6 and 7 show our experimental results on cross-model transfer for all considered text generation models. The artificial text boundary detection models were trained on the parts of the dataset generated by all language models except one, which is held out for cross-model testing, and tested in-domain (ID) the same parts and out-of-domain (OOD) on the remaining part generated by the held-out model.
E Additional Dataset Analysis
In this section we provide additional statistics and visualizations for the distributions of various features in the data. In particular, we note that on most diagrams, real texts have smaller PHD than fake texts, which is very different result from the statistics presented by Tulchinski et al. (2023a), who noted that the PHD of real texts is larger than that of fake texts. We hypothesize that to the texts considered by Tulchinskii et al. (2023a) or due to differences in the sampling strategy used by Dugan al. (2020) and Tulchinskii et al. (2023a) when generating texts. Another observation is that the TLE dimension is very different for all generator models in the original RoFT dataset. This may be the reason for the bad generalization performance of intrinsic dimension-based algorithms across domains. For RoFT-chatgpt PHD and TLE, real and fake texts are close to each other.
be due either to very short lengths of texts in our work compared
We show dataset statistics in the following figures:
Figures 2, 5, and 6 show the lengths of texts in tokens produced by the standard RoBERTa tokenizer (the figures have a cutoff 100 for readability but the datasets do contain few longer sentences);
18

---

Published as a conference paper at COLM 2024
Table 6: Original RoFT, cross-model transfer, part 1. The models were trained on all parts of the dataset except one and tested in-domain (ID) on the same parts and out-of-domain (OOD) on the remaining part.
Model
Metric
GPT2-XL
GPT2
davinci
ID
OOD
ID
OOD
ID
OOD
RoBERTa SEP
Acc, % SoftAccl, % MSE
46.38
40.94
45.95
08.78
63.25
19.73
RoBERTa + SEP
76.85
76.83
76.71
31.22
85.52
47.27
RoBERTa + SEP
03.90
02.92
04.17
06.77
03.04
07.59
RoBERTa
Acc, % SoftAccl, % MSE
46.68
32.56
40.30
06.94
57.20
14.48
RoBERTa
77.30
72.07
75.52
25.31
84.61
40.49
RoBERTa
03.73
03.10
03.70
07.18
02.94
08.56
GPT-2 Perplexity + GB
Acc, % SoftAcc1, % MSE
23.00
23,43
28.12
04.08
23.75
19.78
GPT-2 Perplexity + GB
40.35
47,90
47.96
30.61
46.03
42.46
GPT-2 Perplexity + GB
15.39
10.51
12.19
15.99
11.85
15.38
GPT-2 Perplexity + LogReg
Acc, % SoftAccl, % MSE
21.27
08.86
23.67
03.47
21.43
08.31
GPT-2 Perplexity + LogReg
33.33
22.14
39.80
27.45
35.35
25.19
GPT-2 Perplexity + LogReg
21.52
24.48
16.99
18.98
19.71
22.06
GPT-2 Perplexity + Regr
Acc, % SoftAccl, % MSE
11.68
15.78
14.56
14.18
13.91
15.30
GPT-2 Perplexity + Regr
34.84
49.37
39.36
47.86
46.10
44.43
GPT-2 Perplexity + Regr
07.67
04.62
06.80
06.40
06.73
06.64
PHD + TS ML
Acc, % SoftAcc1, % MSE
31.84
04.02
25.64
04.49
31.38
02.05
PHD TS ML
56.08
14.14
44.28
35.85
53.31
13.82
PHD TS ML
11.13
28.18
16.37
10.74
11.82
27.03
TLE + TS Binary
Acc, % SoftAcc1, % MSE
14.17
03.15
12.36
07.76
14.48
00.98
TLE + Binary
31.14
13.10
29.19
32.14
34.01
11.58
TLE+ TS Binary
21.58
28.20
21.36
16.35
18.12
30.45
Length GB
Acc, % SoftAccl, % MSE
21.70
04.75
18.30
01.22
19.18
06.33
Length + GB
36.00
09.48
33.20
22.02
36.09
22.07
Length- GB
23.56
33.32
27.00
19.27
17.35
20.00
Human Baseline
Acc, %
22.48
17.23
22.59
22.53
24.74
14.06
Human Baseline
SoftAcc1, % MSE
41.91
37.03
39.73
48.01
42.44
33.47
Human Baseline
13.49
14.69
14.29
09.86
14.03
12.91
New York Times
Short Stories
Recipes
Presidential Speeches
natural
natural
2000
natural generated
natural
400
generated
1000
generated
100
generated
200
1000
0
50
100
0
0
50
100
0
50
100
0
50
100
Figure 5: Sentence length distributions in RoBERTA tokens, original RoFT, by topic
New York Times
Short Stories
Recipes
Presidential Speeches
1000
natural generated
natural
natural
natural generated
500
generated
1000
generated
500
250
0
50
100
0
50
100
50
100
50
100
Figure 6: Sentence length distributions in RoBERTA tokens, RoFT-chatgpt, by topic
Figure 7 shows the distribution of pretrained (but not fine-tuned) RoBERTa [CLS] embeddings for real and fake parts of text samples from the original RoFT and RoFT-chatgpt datasets;
19

---

Published as a conference paper at COLM 2024
Table 7: Original RoFT, cross-model transfer, part 2. The models were trained on all parts of the dataset except one and tested in-domain (ID) on the same parts and out-of-domain (OOD) on the remaining part.
Model
Metric
ctrl-Politics
cirl-nocode ID OOD
tuned
baseline
ID
OOD
ID
OOD
ID
OOD
RoBERTa SEP
Acc, % SoftAccI, % MSR
49.35
59.12
50.20
60.61
49.55
23.85
51.35
06.35
RoBERTa SEP
78.49
89.31
80.10
84.85
81.55
56.96
79.84
15.87
RoBERTa + SEP
02.86
01.24
02.93
01.87
02.33
06.33
02.51
39.32
RoBERTa
Acc, % SoftAcc1, %
47.46
44.03
46.07
54.55
45,91
20.98
47.29
04.76
RoBERTa
78.43
85.53
78.01
86.87
80.13
52.88
78.49
15.87
RoBERTa
MSE
02.80
01.21
02.79
01.07
02.37
06.49
02.69
36.00
GPT-2 Perpl. GB
Acc, % SoftAcc1, % MSE
25.27
10.69
25.32
07.07
30.88
10.15
24.44
06.35
GPT-2 Perpl. GB
48.89
27.04
47.71
24.24
53.87
29.20
64
14.29
GPT-2 Perpl. + GB
11.81
20.96
12.27
22.70
11.67
17.86
12.01
39.62
GPT-2 Perpl. LogReg
Acc, % SoftAccl, % MSE Acc, % SoftAccl, % MSE
24.08
07.55
21.88
07.07
28.50
08.04
24.77
03.17
GPT-2 Perpl. LogReg
42.23
23.90
38.84
19.19
42.78
22.28
42.45
15.87
GPT-2 Perpl. LogReg
15.70
23.27
16.69
25.38
17.46
24.20
15.72
40.86
GPT-2 Perpl. Regr
14.80
13.21
13.96
14.14
15.70
11.90
13.62
03.17
GPT-2 Perpl. Regr
42.06
40.25
41.89
33.33
43.08
36.03
39.25
14.29
GPT-2 Perpl. Regr
06.40
07.778
06.55
08.31
06.81
07.10
06.84
22.86
PHD TS ML
Acc, % SoftAcel, % MSE
25.70
08.18
25.18
11.11
21.44
12.58
23.13
03.70
PHD TS ML
47.33
32.70
47.30
39.39
35.31
26.37
45.64
05.56
PHD TS ML
14.09
11.23
13.62
09.41
18.28
18.43
13.42
52.39
TLE + TS binary
Acc, % SoftAccl, % MSE
10.98
05.03
11.53
04.04
14.51
06.56
12.39
03.18
TLE + TS binary
29.08
18.87
28.21
22.22
30.21
17.26
28.21
15.88
TLE TS binary
20.36
23.28
20.84
21.82
21.82
26.64
20.59
25.55
Length GB
Acc, %
17.32
03.77
18.05
0.0
21.64
0.04
15.24
26.98
Length GB
SoftAccl, %
33.96
18.86
35.71
15.15
35.80
10.87
32.56
34.92
Length GB
MSE
22.48
23.87
24.16
26.53
23.59
32.78
16.25
16.0
Human Baseline
Acc, % SoftAcel, % MSE
22.60
21.6
22.57
23.94
21.46
25.90
22.41
46.15
Human Baseline
40.61
41.6
40.59
45.07
39.17
44.92
40.46
63.46
Human Baseline
13.87
10.57
13.87
07.70
13.92
13.48
13.82
11.51
Figure 8 shows the distribution of labels in the original RoFT dataset by generator;
Figure shows the distribution of labels in the original RoFT dataset by topic; this distribution is identical the corresponding distribution for the RoFT-chatgpt dataset;
Figure 10 shows the distribution of PH dimensions of real and fake parts of the text by generator;
Figures 11 and 12 show the distributions of PH dimensions by topic for the original RoFT and RoFT-chatgpt respectively;
Figure 13 shows the distribution of TLE dimensions of different sentences by gener- ator;
Figures 14 and 15 show the the distributions of TLE dimensions by topic for original RoFT and RoFT-chalgp! respectively.
F Detailed experimental results
In this section, provide additional statistics and visualizations regarding experimen- tal results. Figure 17 visualizes the changes in perplexities for sentences from the texts in RoFT-chatgpt by their labels. We make the following observations.
First, perplexities the first couple of sentences across all texts are quite high, and the average perplexity of sentences decreases by the end of the text. This is probably due to
20

---

Published as a conference paper at COLM 2024
Redpes
2.
09
0.8
0.0
0dm
0.mm
0.
Redips
Yark Tirne
2.8
2/6
0.6
0dm
0.
Davind Raccps
Daavnni Shart Stznies
Dawnn: New Times
2.8
0.
0.6
0dm
0.dm
0A
0.4
0.8
Figure/ 7: Distribution of pretrained (but not fine-tuned) RoBERTa [CLS] embeddings of real and fake parts of text samples from the original RoFT and RoFTTchatgpt datasets. The dimension is reduced to 2D via principal component analysis (F.R.S,, 1901).
40
1000
fully natural
baseline
20
10
500
10
CtritPolitics
ctri-nocode
0
200
200
200
200
100
100
davinci
finetuned
9pt2
9pt2
0
0
2
3
5
6
9
2
7
8
9
4
7
0
3
5
6
8
Figure Label distributions for the original RoFT dataset by model
Recipes
New York Times
Short Stories
Presidential Speeches
200
40
500
100
200
20
250
0.0
5.0
7.5
0.0
5.0
7.5
0.0
7.5
0.0
5.0
Figure Label distributions for the original RoFT dataset by topic
21

---

Published as a conference paper at COLM 2024
200
5al 8.76
7.5
Easadis, aggin
ct Hofros segilAl
102
5.0
0.0
relent, aeg5.1 conro, seg380
relans, 2.27.45
200
nuturs expt0. 5X3, ong 1644
retundl set2 oegrt
200
400
textared ovgi
150
100
102
Figure 10: PHD distributions for the real and fake parts of the RoFT dataset, by generator models
Presidential Speeches
Short Stories
New York Times
Recipes
1250
400
1000
30
20
200
10
300
250
15
23
25
10
15
20
10
25
20
2rg:7.48
ang
Figure 11: PHD distributions for the real and fake parts of the RoFT dataset, by topics
Presidential Speaches
Short Stories
New York Times
Recipes
302
1000
800
75
200
600
400
100
190
200
00
10
25
10
20
25
%
10
20
25
10
20
natural svg 9.07
natural, ang::0.0
natural, 3vg:3.00
natural avp 7:47
generated, ang18.55
penerates, 9.60
getersted avg 9:30
generated, #.q.3.10
Figure 12: PHD distributions for the real and fake parts of the RoFT-chatgpt dataset text, by topics
the fact that for the words of the first sentences the length of the text prefix is not enough for stable calculation of perplexity. One solution to mitigate this effect and hence make perplexity-ban bassed classifiers more stable might be to generate new prefixes for the text using some generative model (e.g. gpt-3.5) and calculate perplexities of original text words using this generated prefix. We leave this idea for further research. Figure 18 visualizes the coefficients of a logistic regression model trained on sentence perplexities from the RoFT-chatgpt dataset (Perplexity- LogRegr rows in the tables). We can see a distinct pattern in this figure. For the label k, which means that the first fake sentence in the text is the (k + 1)st, the highest value of the coefficient are kth and k10th ones, and the lowest ones are often (k + 2)nd and (k + 12)th. This could mean that the model is "searching" for a sudden drop of mean and variance of perplexity at point where the fake part is starting. This fits together well with the idea that language model (Phi-1.5) trained on data from another model (GPT-3.5 version) see text generated by this model as more "natural" one than real numan-produse text. Therefore, perplexity often drops at the point where fake text begins, and logistic regression can pick up this effect and it as decision rule.
Finally, Figure 18b visualizes the confusion matrix on the test set of logistic regression trained on the RoFT-ChatGPT dataset. We see that most of the errors concentrate around true labels, indicating that often model almost correctly finds a boundary, having a shift by +1/-1 label only.
22

---

Published as a conference paper at COLM 2024
Ma rabvos peg F:22
rutaad aag Excadin, Hegnt3
IUbUELL ORgRANS PATIO MAL
200
retumi 8.5
80
neture: eng 5.09 axp
natural exp gat2, exp10.2
retural wxp spibel 63
200
ceanci #xp
%
Figure 13: TLE dimension distributions for the sentences in the RoFT dataset by generator models
Presidential Speeches
Speeches
Recipes
150
15
200
100
10
100
natural 4xgr9.56
raltunal, ang
angelii
penseated 2.g59.36
generated, avg.10.5
gererated ango9.02
generated, avg
Figure 14: TLE dimension distributions for the sentences in the RoFT dataset by topics
Presidential Speeches
Recipes
York
Short Stories
80
50
20
generated,
generaned,
generated, avgis
generoted,
Figure 15: TLE dimension distributions for the sentences in the RoFT-chatgpt dataset by topics
finetuned
gpt2
gpt2-xl
human
Figure 16: Perplexities of the last sentence in RoFT by model obtained from GPT-2; blue distribution on the in-domain set, i.e., the entire dataset except the speficied generator; orange on the out-of-domain set, i.e, data from the specified generator.
23

---

Published as a conference paper at COLM 2024
Figure 17: Sentence perplexities in the RoFT-chatgpt dataset by label obtained from Phi-1.5
model. X axis: sentence index in the text, Y axis: sentence perplexity.
100
120
36.0
54.0
16.0
6.0
4.0
0.0
0.0
1.0
2.0
6.0
20.0
18.0
19.0
11.0
34.0
51.0
12.0
3.0
0.0
12.0
80
0.0
11.0
3.0
15.0
23.0
63.0
15.0
0.0
3.0
2.0
7.0
5.0
2.0
8.0
-0.5
3.0
5.0
2.0
7.0
6.0
12.0
42
20.0
7.0
31.0
3.0
2.0
4.0
3.0
7.0
6.0
8.0
15.0
59.0
40.0
10
1
2
(b)
Figure 18: Analysis of the logistic regression trained on sentence perplexities on the RoFT- chatgpt dataset (Perplexity LogRegr the tables): (a) heatmap of the coefficients; (b) confusion matrix for test set predictions of logistic regression trained on sentence perplexities on the RoFT-chatgpt dataset.
G Examples of correctly classified and misclassified texts
We provided more examples of correctly and incorrectly classified texts at the Figures 19 and 20.
24

---

Published as a conference paper at COLM 2024
The
wet.
"Get up
Don't give up now," her friend said, placing comforting hand on her shoulder.
The girl shook her head, tears streaming down her cheeks.
"I can't do it.
I'm just not strong enough.
"You are," her friend Insisted.
"Look the obstacles you've overcome already.
You're stronger you think.
The girl took a deep breath wiped her tears.
Figure 19: Correctly classified sample input from the ROFT-chatgpt dataset colored accord- ing the perplexities of Phi-1.5 on each sentence; "greener" text corresponds to higher mean perplexity, "blue-er", to higher standard deviation of the perplexity, red text shows where both mean and std are low. The prompt is above the dotted line, the rest is generated.
HOW TO MAKE: Canadian Bacon And Chicken Pasta Bake Ingredients: 1 (8 ounce) package ziti pasta nonstick cooking spray 1 lb chicken breast, diced medium onion, diced cloves garlic, minced green pepper, chopped zucchini, chopped 2 (15 ounce) cans tomstoes seasoned with basil garlic & oregano (6 ounce) can tomato paste 14-12 teaspoon crushed red pepper flakes 1 (15 ounce) container fat-free ricotta cheese 1 (6 ounce) package Canadian bacon 1 cup light mozzarella cheese 2 tablespoons parmesan cheese. Preheat oven to 350 degrees.
Cook pasta according to package directions without added fat or salt. Drain and set aside.
In large skillet coated with nonstick cooking spray over medium heat, cook chicken until browned and cooked through. Add onion, garlic, groen popper, and zucchini.
Cook until vegetables are tender.
stir in canned tomatoes, tomato paste, and crushed red pepper flakes.
Cook for 5 minutes.
large mixing bowl, mix together cooked pasta, ricotta cheese, and Canadian bacon.
you, son," my know, Dad." replied small smile.
"I you
had been long time since we had said those words to each other.
Our relationship had been strained for years, but we had recently started working it.
It wasn't easy, but we were making progress.
"I just want you know that no matter what happens in life, we'll always be here you." my dad continued.
may not always agree with your decisions
you're going through, always you unconditionally.
welled up my eyes hugged my It was powerful moment, that would always cherish.
Figure 20: Misclassified inputs from the ROFT-chatgpt dataset colored according to the perplexities Phi-1.5 on each sentence. At the first picture, generated text starts from the fourth sentence, but perplexity-based classifier predicts that it starts from the fifth sentence; at the second picture, generated text starts from the second sentence, but perplexity-based classifier predicts that it starts from the fifth sentence.
25


Reviews:
[{"review_id": "4Cyhe1DxZB", "reviewer": "Reviewer_quBQ", "summary": "The paper examines a novel and interesting problem of detecting the transition between human-written and AI-generated text within documents that contain both. It uses updated datasets like RoFT and RoFT-chatgpt to test various detection methods, including RoBERTa-based classifiers, perplexity-based methods, and topological features. The findings reveal that while RoBERTa tends to overfit and struggle with generalization, perplexity-based methods offer a good balance of accuracy and generalization, especially when trained on synthetic data. Topological methods show promise in robustness across different domains but occasionally lack high accuracy. The study highlights significant challenges in cross-domain and cross-model generalization, particularly with short texts due to limited context and longer texts due to their complexity.", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your high evaluation of our work.\n\n> More explanations about the choice of parameter values may be helpful to understand the underlying methodology that has been adopted. For example, the authors use a window size of 20 with strides of 5 when performing Topological Data Analysis (TDA). More explanation for the choice of 20 and 5 can be helpful. Are those values standard or dependent on text length and identified empirically?\n\nThank you for the feedback, we add more clarifications on this in the final version of our paper.  Here are the answers to your specific questions:\n- Window size 20 was taken directly from “Intrinsic Dimensionality Estimation within Tight Localities: A Theoretical and Experimental Analysis” by Laurent Amsaleg et al., as the authors claim that their algorithm is stable enough “for `tight' localities consisting of as few as 20 sample points”.\n- Stride size 5 was chosen after the preliminary experiments. Initially, we tried smaller stride sizes, but we found out that in these cases the series became too long for the Time Series SVM algorithm, and it wasn't able to converge.\n\n> Missing references can be corrected. For example, On page 5, the reference figure is missing in the text \"(SoftAcc1; indeed, Fig. ?? in Appendix F shows that many misclassifications point to an adjacent class, which may be acceptable in real-world applications)\".\n\nThank you for this remark, SoftAcc is the percentage of predictions that differ from the correct label by at most one; broken figure reference referred to Figure 18b. We will recheck the paper for missing references and typos before submitting the camera-ready version. \n\n> Why was the choice of datasets limited to only one? Is there a specific reason pertaining to the problem being solved?\n\nYes, there was essentially only one dataset that met the following criteria: \n- it was labeled for boundary detection instead of just classifying texts as real or AI-generated; \n- it included human results, meaning experiments where humans labeled the boundaries; \n- it was available at the time of writing the paper, so that we could compare our results with previous work.\n\nRecently, another dataset has appeared, namely subtask C of Task 8 of a recent SemEval workshop: https://github.com/mbzuai-nlp/SemEval2024-task8 . However, the full labels of its test set (i.e. with the information about generating models) were only released after the CoLM deadline.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper aims to study a novel and upcoming challenge which is important study due to the wide application of LLMs in the domain of Natural Language Processing. This is particularly important in certain domains where authorship attribution is important. \nThis is a comprehensive study that uses a diverse array of methods, including RoBERTa-based classifiers, perplexity-based classifiers, and topological features. The authors perform a thorough analysis of the strengths and weaknesses of each method under different conditions including the characteristics of the dataset itself leading to certain performance trends. The analysis is done for real-time applications including cross-domain and cross-modal scenarios i.e., the paper discusses the practical implications of each tested method, providing a clear understanding of their applicability in real-world settings. The limitations of different approaches are clearly mentioned.", "reasons_to_reject": "More explanations about the choice of parameter values may be helpful to understand the underlying methodology that has been adopted. For examples, the authors use a window size of 20 with strides of 5 when performing Topological Data Analysis (TDA). More explanation for the choice of 20 and 5 can be helpful. Are those values standard or dependent on text length and identified empirically?\n\nMissing references can be corrected. For example, On page 5, the reference figure is missing in the text \"(SoftAcc1; indeed, Fig. ?? in Appendix F shows that many misclassifications point to an adjacent class, which may be acceptable in real world applications)\".", "questions_to_authors": "Why was the choice of datasets limited to only one? Is there a specific reason pertaining to the problem being solved?", "extra_scores": {}}, {"review_id": "0dDEnjm2wc", "reviewer": "Reviewer_LWRU", "summary": "The paper titled investigates the challenge of detecting boundaries between human-written and machine-generated text, an issue increasingly relevant due to the proliferation of LLMs. Utilizing the real or fake text dataset, which comprises texts with both human-written and machine-generated sections, the study tests several methodologies adapted for this complex detection scenario, moving beyond traditional binary classifications. Key approaches include perplexity-based methods that evaluate text unpredictability, adaptations of classifiers like RoBERTa that leverage text embeddings to detect shifts in text generation sources, and innovative applications of topological data analysis to determine the intrinsic dimensionality of text data, which provides robustness against domain and model shifts. The results suggest that PPL based methods and intrinsic dimension estimation offer promising results, outperforming traditional binary classifiers and showing better adaptation to the nuanced challenges of environments with mixed human and AI-generated text. This research contributes valuable insights into artificial text detection, highlighting the potential for further exploration and refinement of these techniques to improve our ability to discern human from machine-generated content effectively.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your high evaluation of our work. Let us address your concern:\n\n> Analysis is limited to only one datasets\n\nPlease see the answer about the one dataset above. This is indeed a novel problem setting, and at the time of performing experiments for the paper we were aware of only one preexisting dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem is very new, and any advancement is appreciated.\n2. The authors have also advanced the dataset and released it for future research, given that the problem is very important in current times.\n3. The paper showcases diverse experimental scenarios, such as cross-model and cross-generation comparisons, using both binary classification and perplexity-based evaluation.\n4. An appropriate explanation of poor generalization scenarios is provided, which is not common to see in other papers.\n5. The paper is very well written and explained, with high-quality visuals.\n6. Three models—GPT, Phi, and LLaMA—are explored, which makes it a strong and robust framework.", "reasons_to_reject": "1. Analysis is limited to only one datasets", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zcKFvnZBB0", "reviewer": "Reviewer_Zghr", "summary": "This paper tackles the problem of detecting the boundary between human-written and machine-generated text in mixed texts, which is an increasingly common real-world scenario that has not been well-studied. The authors adapt several existing artificial text detection techniques to this boundary detection task and evaluate them on the RoFT (Real or Fake Text) dataset as well as a new RoFT-chatgpt dataset they created using GPT-3.5 generations.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and let us discuss your concerns.\n\n> To my understanding, detecting the boundary between human-written and machine-generated parts should go beyond sentence classification. It would be better to localize which parts of the paragraph are likely written by a human and which parts of the article are likely written by AI.\n\nWe understand that the task of boundary detection by sentences differs from the most general task of localizing the parts of the paragraph that are likely to be written by AI. However, the general task is complicated and under-explored.  At the same time, most existing work focuses on solving the task in a much simpler setting of binary classification of whole texts into two classes, human- or AI-written. Thus, we find it reasonable to depart from the setting of binary classification of texts towards solving the most general task by first approaching a slightly relaxed problem of detecting AI-written parts of the text as a task of boundary detection by sentences. If we can solve this task well enough, we can move forward to more complex problem statements.\n\n> Some parts of the paper are not clear. For example, in the evaluation section on artificial text boundary detection, the terms \"SoftAcc\" and \"mean squared error\" are not explicitly defined. It would be better to include mathematical notations to clarify these metrics.\n\nThank you for pointing this out. The paper mentions that SoftAcc is the percentage of predictions that differ from the correct label by at most one, but we agree that it would be better to define such terms in mathematical notation, and we will include the definitions in the camera-ready version of the paper if accepted. \n\n> While the paper mentions the limitations of the methods in general terms, it lacks a deep dive into specific instances where the models failed. Providing concrete examples and analyzing the reasons behind the failures could offer valuable insights for future improvements in boundary detection techniques.\n\nWe fully agree that qualitative analysis of specific examples would benefit the paper and make it more clear where current modes fail and perhaps provide intuition for potential improvement. The paper discusses some particular data properties that complicate cross-domain boundary detection in the “Data Properties” section (p. 8), and we will be able to add more qualitative analysis to the camera-ready if accepted since an extra page will be allowed in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe study targets the detection of human-written and AI-generated text boundaries in mixed texts, a growing concern due to the prevalence of large language models. This problem has not received much attention in the literature, making this study highly relevant and timely.\n2.\tThe researchers evaluated various adapted artificial text detection methods on the RoFT dataset and their new RoFT-chatgpt dataset. They tested the methods in different scenarios, including in-domain, cross-domain, and cross-model, offering valuable insights into each approach's strengths, weaknesses, and dataset property challenges.\n3.\tThe paper adapts and assesses a range of techniques, establishing foundational baselines for the task. The authors' detailed result analysis and discussion of current limitations highlight promising avenues for future research.", "reasons_to_reject": "1.\tTechnically, this paper mainly conducts benchmarking on the RoFT dataset, primarily using evaluation metrics. It is more like a technical report than a research paper with novel contributions.\n2.\tTo my understanding, detecting the boundary between human-written and machine-generated parts should go beyond sentence classification. It would be better to localize which parts of the paragraph are likely written by a human and which parts of the article are likely written by AI. However, the current evaluation in this paper is doing binary classification for each sentence in the RoFT dataset, which may not capture the nuances of the task.\n3.\tSome parts of the paper are not clear. For example, in the evaluation section on artificial text boundary detection, the terms \"SoftAcc\" and \"mean squared error\" are not explicitly defined. It would be better to include mathematical notations to clarify these metrics.\n4.\tWhile the paper mentions the limitations of the methods in general terms, it lacks a deep dive into specific instances where the models failed. Providing concrete examples and analyzing the reasons behind the failures could offer valuable insights for future improvements in boundary detection techniques.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examine which methods are most effective for solving the boundary detection problem---that is, identifying when text transitions from machine-generated to human-written. The reviewers emphasize that this is a relatively new and important problem for which there is little prior work. They also note that the paper's analysis is thorough and well-written.
Decision: Accept