Multiscale Positive-Unlabeled Detection of AI-Generated Texts
Authors: Yuchuan Tian
Hanting Chen
Xutao Wang
Zheyuan Bai
QINGHUA ZHANG
Ruifeng Li
Chao Xu
Yunhe Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: Large Language Models
AI-Generated Texts
Positive-Unlabeled Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recent releases of Large Language Models (LLMs), e.g. ChatGPT, are astonishing at generating human-like texts, but they may impact the authenticity of texts. Previous works proposed methods to detect these AI-generated texts, including simple ML classifiers, pretrained-model-based zero-shot methods, and finetuned language classification models. However, mainstream detectors always fail on short texts, like SMSes, Tweets, and reviews. In this paper, a Multiscale Positive-Unlabeled (MPU) training framework is proposed to address the difficulty of short-text detection without sacrificing long-texts. Firstly, we acknowledge the human-resemblance property of short machine texts, and rephrase AI text detection as a partial Positive-Unlabeled (PU) problem by regarding these short machine texts as partially "unlabeled". Then in this PU context, we propose the length-sensitive Multiscale PU Loss, where a recurrent model in abstraction is used to estimate positive priors of scale-variant corpora. Additionally, we introduce a Text Multiscaling module to enrich training corpora. Experiments show that our MPU method augments detection performance on long AI-generated texts, and significantly improves short-text detection of language model detectors. Language Models trained with MPU could outcompete existing detectors on various short-text and long-text detection benchmarks. The codes are available at https://github.com/mindspore-lab/mindone/tree/master/examples/detect_chatgpt and https://github.com/YuchuanTian/AIGC_text_detector.

Published as a conference paper at ICLR 2024
MULTISCALE POSITIVE-UNLABBLED DETECTION OF AI-GENERATED TEXTS
Yuchuan Tian', Hanting Chen", Xutao Wang", Zheyuan Bai², Qinghua Zhang?, Ruifeng Li Chao Xu', Yunhe Wang2*
National Key Lab of General AI, School of Intelligence Science and Technology, Peking University
2 Huawei Noah's Ark Lab Huawei Group Finance Huawei Central Software Institute t Lanycestu, pku. edu. en, yunhe wangehuawes com
ABSTRACT
Recent releases of Large Language Models (LLMs), e.g. ChatGPT, are astonishing at generating human-like texts, but they may impact the authenticity of texts. Previ- ous works proposed methods to detect these Al-generated texts, including simple ML classifiers, pretrained setrrinndd-mdel-eee model-based zero-shot methods, and finetuned language classification models. However, mainstream detectors always fail short texts, like SMSes, Tweets, and reviews. In this paper, a Multiscale Positive-Unlakes training framework is proposed address the difficulty of short-text detection with- out sacrificing long-texts. Firstly, we acknowledge the buman-rrsembblane property of short machine texts, and rephrase AI text detection as a partial Unllblee (PU) problem by regarding these short machine texts as partially "unlabeled". Then in this PU context, we propose the length-sensitive recurrent model in abstraction is used to estimate positive priors of scaie-variant corpora. Additionally, we introduce Text Multiscaling module to enrich training corpora. Experiments show that our MPU method augments detection perfor- mance on long Al-generated texts, and significantly improves short-text detection of language model detectors. Language Models trained with MPU could outcom- pete existing detectors on various short-text and long-text detection benchmarks. The codes are available at htt.bb///zeee//aattte/////////ee gitnag..ccmm/ccchaaariaaaatrttCe
(MPU)
sensitive Multiscale PU Loss, where a
and https://
INTRODUCTION
Recent developments in Large Language Models (LLMs) have brought astonishing changes to people's lives. The GPT-2 (Radford et al. (019)) model, created in early 2019, is capable of simple question-ant answering tasks; GPT-3 (Brown al. 2020) is great leap in model size and capability; ChatGPT OOpenAl 2022), announced in late 2022, shows comparable performance to humans as chatbot: GPT-4 OpenAA 2023a). (023)) released this year, has even better generative performance. These advancements are making people's lives easier with applications like writing aids, search engines, and Office Suites. However, they could be used to generate deceptive fake texts for illegal and unethical purposes.
Previous works have proposed numerous approaches to distinguish fake Al-generated text from genuine human languages. Canonical work (Solaiman et al. (019) used simple machine learning classifiers as baselines; some works (Gehrmann et al. [019]] Mitchell et al. (023) proposed zero-shot detection measures based on pretrained models; numerous works (Solaiman et al. 2019; Crothers et al., 2022 Guo et al. 2023 Mitrovic et al. 2023) perform simple finetuning of pretrained language
models on the Al-text classification task.
Despite various methods, few mainstream methods investigated the negative impact of text length: the difficulty to detect significantly increases texts become shorter. Some latest online ChatGPT detectors have noticed this issue, but they dodge rather than address it by putting up minimum text
"Corresponding Author.

---

Published as a conference paper at ICLR 2024
OpenAl 2023b). In the era of smartphones where people rely heavily on fragmented mobile media, fake short articles like SMSes. Tweets, and reviews generated by LLMs could pose huge threats to one's daily life, yet we still lack comprehensive detector that is capable of detecting both short texts and long-texts.
length requirements (Tan 2022
FudanNLPLab 2023
To improve detectors' performance on short texts, we rethink the plain "Binary Classification" setting that is intuitively applied. It is seemingly natural to phrase text detection as a binary classification task, as texts have clear origins (from human works or AI outputs) and thus, clear binary labels (real or fake); but interestingly, we observe a handful of machine-generated texts that are overly short and simple, such that these texts are highly similar to human (e.g. Ex. in Table IT. It is not suitable to assign these simple machine texts with either clear human or AI labels; rather, they are an "Unlabeled" state. Though the case occasional and most short machine texts (e.g. Ex. 1 in Table [T are still distinguishable based on manifold features, it prompts us to question the rationality of clear binary labels on general short machine texts. On the contrary, we hold that short machine-genede generated texts are partially "Unlabeled", As machine-generated texts become shorter and simpler, the "Unlabeled" property could gradually dominate the text.
Example 1: The first sentence in benchmark HC3-Sent (Guo et al. 2023)
Human: You can't just go around as- sassinating the leaders of countries you don't like!
Al: It is generally not acceptable or ethi- cal to advocate for or condone the assas- sination of any individual, regardless of their actions beliefs.
Example 2: Answer to "When is the independence day of the United States?" The Independence Day of the United States is celebrated on July 4th.
Human: Independence Day is annually celebrated on July 4th.
Al:
Table 1:
Short example answers from human and AI. In general, short answers are distinguishable based on features like punctuations, emotions, and formality (see non-cherrypicked case Ex. 1). But in extreme cases (see Ex. 2), short simple answers are indistinguishable, and the unlabeled property is manifest.
In this sense, we model the task of Al-generated text detection as a partial Desitive-Unlabelee (PU) problem and formulate the Multiscale Dositive-Unlabele (MPU) training framework to address the challenging task of short text detection without sacrificing long texts. PU problems typically address binary classification tasks where positive data and unlabeled data are offered for training. Considering the partially "Unlabeled" property of short machine texts, we rephrase detector training as partial PU problem and boost detectors' performance on multiscale texts. In order to improve conventional PU optimization targets for texts of various lengths, a length-awan aware Multiscale PU (MPU) loss is proposed and applied during the training process. We are aware that the PU prior probability of a text being positive is length-varian variant. To this end, an abstract recurrett model is designed to adjust the PU prior probability automatically based on corpus length. Further, Text Multiscaling module is also proposed to exert the effect of Multiscale PU loss by diversifying training corpora in terms of length. Experiments demonstrate that the MPU framework is significantly effective in improving short-text detection performance; meanwhile, detection on long texts is also augmented.
2 RELATED WORK
Text Detection Methods. Since the introduction of GPT-2 (Radford et al. 2019) and its successors,
fake texts generated by powerful LLMs are causing ethical and legal issues. Methods are developed
to discriminate against these generated texts in various misuse scenarios. Zellers et al.] (011)) shed light on machine--eneraane
fake news by proposing a GPT-based news generator GROVER,
and uses GROVER itself to sort fake news out: AAeeani et al. (200)) looks at detection of fake online reviews; Fagni et al. 2222)) focuses on machine-generated fake tweets and proposes the TweepFake dataset. Other proposed detection methods are for general scenarios. Several canonical baselines are mentioned by Solaiman et al. (2019) to detect GPT-2 texts, including simple TF-IDF classifiers and finetuned RoBERTa (Liu al. 2019); GLTR (Gehrmann et al., (2019) detect generated texts in a zero-shot manner by using token prediction probabilities from available pretrained NLP models like BERT (Devlin et al. 2018) and GPT-2 (Radford et al. 2019). After the introduction
2

---

Published as a conference paper at ICLR 2024
of ChatGPT (OpenAl 2022), some new detection methods (Liu et al. 2022 Mitchell al. Mitrovic et al.
2023
2023] Guo et al. 2023) are released.
PU Methods. Previous works have proposed methods to train binary classifier with positive and
unlabeled data, Many PU methods (Bekker & Davis 2020; Du Plessis et al. 2014 Kiryo et al. 2017
Su et al. 2021: Hammoudeh & Lowd 2020) Chen et al. 2020) constructs PU loss based on positive and unlabeled samples, for classifying unlabeled data. Other PU methods include two-step learning
and bias learning (Liu al.,, 2003). The two-step technique first identifies reliable negative examples
and then performs learning based on the positives and negatives of the mark (He et al. 2018 lenco & Pensa 2016); biased learning treats unlabeled data negative sample of class-labeled noise (Hsich et al., Above all, we refer to applying a PU loss during training to address the task of multiscale Al Al-generated text detection, because PU losses could be generally applied on powerful finetuning text detectors without much additional computation costs.
et al.]
3 MULTISCALE POSITIVE-UNLABEN TEXT DETECTION
3.1 TEXT DETECTION AS POSITVE--NLLBBLEE CLASSIFICATION
Despite manifold methods for detecting A Al-generated texts, mainstream detectors seldom take the factor of text length into account, and thus they always fail on short texts. We have tried several existing detection methods for short LLM-generated texts (shown in Table 4). but none of them perform well. As people nowadays are immersed in short, fragmented forms of mobile media, they are vulnerable LLM attacks with no reliable means to defend themselves. Hence, we are in urgent need of performant short AI- Al-generated text detector.
Intuitively, past works formulated the task of AI text detection as a binary classification problem, i.e. classifying texts as AI or Human. However, the formulation could be problematic for shorter texts as we found high similarities between extremely simple Al texts and human texts. The phenomenon could be rare in actual applications. But it is fundamentally reasonable, because LLMs learn from human languages; and for sentences whose structures are overly simple, they are seemingly "copied" by LLMs from what they have learned. Therefore, the attribution of these simple machine texts is uncertain: on one hand, they are indeed outputs from Language Models; on the other hand, they ordinary human languages. Though the completely coo-classifiable case mostly happens for extremely short texts or commonly used phrases (that rarely occurs in our benchmarks and detection of which is of no application value), it inspires to think about the partially "unlabeled" property behind the vast majority of short, distinguishable texts despite their definite labels.
To overcome this issue, we model the task of multiscale text detection a partial Positive Unlabeled problem (PU). In this problem, corpora from human are regarded as "Positive", but short texts from machines are given an additional "Unlabeled" mark for PU loss calculations (detailed in Sec. 3.3 Then our detector model is optimized within this partial PU context.
3.2 PRELIMINARIES: CANONICAL PU LOSS FUNCTIONS
PU losses are derived from the traditional Nositive-Negative (PN, Le. Binary Classification) setting, detailed in Appendix A Some works (Du Plessis et al., 2014 Plessis et al., 2013) perform indirect approximation of the negative risk in the PN framework, yielding the unbiased PU (uPU) loss as follows:
RuPU(O) n Rr +1) - h Rr -1) Ru -1),
(1)
where (r 1 E. [(gx=)) 11 and Rv (g. 11) nu En L(g(x -1) esti- mations calculated from positive and unlabeled training samples respectively.
up
However, the deep learning classifier may be too flexible, leading to Ry (g. -1) # hr 1) < 0 and causing the model to overfit. As a remedy, Kiryo et al. (2017) proposes the non-negann -egaaieee risk estimator based on the uPU loss. The non-negative PU (mnPU) loss is thus derived as follows:
RanPu(9) = a Rr +1) + max(0, Ro -1) - a Rr(g. -1)}
(2)
3

---

Published as a conference paper at ICLR 2024
The nnPU loss Kiryo et al. (2017) is performant and thus widely referred by later PU works and applications (Kato et al. 2 Bepler et al. 2019 Peng et al. 2019 Xu et al., 2019; Chen et al. 2020 [Su et al. |021 Tang et al.) 2022p. However, to the best of our knowledge, have applied PU to scenario of length-variant texts, in which simple usage of the nnPU loss might not be effective. We hope develop effective PU mechanism in aid of detecting length-variant texts.
previous works
3.3
MPU: A LENGTH- PU APPROACH
In PU loss conventions as stated in Sec. 3.2 the estimation for the prior probability of a data being positive always kept at a constant. The reason that prior probability ι is closely associated with the dataset distribution, which is always assumed to be uniform. However, this might not be case with texts of different lengths. As explained in Section [ short texts and long texts hold different properties; in other words, they do not share the same distribution. In this regard, the assumption of dataset distribution being uniform is flawed; fixing the prior estimation at a certain constant value is problematic in the case of multiscale text detection (i.e. where texts to be processed are of manifold length).
Though long texts and short texts have different distributions, the distribution shift from long text to short text is a gradual process with respect to text lengths. To deal with the gradual shift of distribution, we look at this shift with respect text length from differentiation perspective. Texts of a certain length could be regarded as a small subset that features its own distribution, and also its 7 r(). We hope to provide smooth, length- variant estimation (!) for the prior at length 4, in order to fit the PU framework for the multiscale text detection problem.
own prior
In this fashion, we propose the Multiscale PU loss RMPU that uses -ength-sensttiee priors सं for multiscale texts. However, we are faced with the challenge of modeling the length-variant prior in abstraction. Namely, we need to investigate the general probability of all sentences (of a certain length) being human, without access to specific details of any piece of text. To this end, we use the general recurrent language model (Mikolov et al., 2010; Sundermeyer et al. 2012) in abstraction as a discriminator for positive, human-spoker corpora, which is formulated as follows: given a sequence St of / tokens: S [t+]]= abstract recurrent discriminator A seq + [0. 1] that is bounded one-dimensional (because from the discriminator we expect a confidence of a sequence being positive), the recurrent model in abstraction is expressed as:
A (S,+1) = f (A(S)), 4+1). Vi E []-
where f is some function that merges the classification of all previous tokens S with the clas- sification of the last token 4. Next, the abstraction is concretized based on task characteristics of nunaa--eeeeated text discrimination. Since relatively short texts tend to have simple semantic corre- lations to captured, human text discrimination is performed via capturing signals from tokens. We hold that each token has a hidden property of origin, and the attribution contributes to the classification of the whole sequence. Tokens, as extreme cases of short texts, could be sorted into two categories: "clear positive", i.e. the token could hardly be generated by AI; or "unlabeled", i.e. the token is mediocre and universally used, giving no signal as "human- spoken". "human-spoken". Each token is expected to provide an equal contribution to the overall sequence classification towards the orientation of its own category (Kang et al. 2018). In this sense, the merging function f is formulated as equally-weighted addition:
(A(()) =s wsA( + w;b(t(+1) s.t.
Ws Wt.
(4)
where 5(t,+1) is defined as the contribution of (tt+1). For simplicity, we discretize the transition of classification from i + i ++1 1 and each token contribution is designated binary. We also take text length into consideration by normalizing 5(t441) with a factor of sequence length (. Under these assumptions, the transition is formulated as:
(s:+ )) = clwpp(((psS))) 10,), s.t. 8(4) = { 1/1 if t is clear positive, -1/1 otherwise.
(5)
Notably, we use a hard clip function to bound the overall classification results in interval [o, 1] rather than other non-linear functions, e.g. sigmoid. This is because clear positive tokens could be rare in
4

---

Published as a conference paper at ICLR 2024
practice. This assumption is particularly true when consider recent advancements of generative language models, where human and Al languages are more resembling. In other words, a majority of words are both frequently used by human and AI, while only few signal words manifest unique human characteristics. This property requires the discriminate model to be highly sensitive to positive token signals. Hence, set hard boundaries rather than using non-linear standardizing functions to scale the output between [0, 1]. Further, to encourage positive responses, we initially positive as the initial state (So) of the discriminator.
Return to the original objective, we tend to calculate the prior probability of a sample being positive based on the introduced recurrent language model. न could also be interpreted as the expectation of confidence from the recurrent discriminator E' (-:)]- The discretization of contribution is beneficial to reducing the continuous diseriminator A discrete states: for sequence Si with I tokens, the confidence could only take values i/l, Vi E [1]. Therefore, discriminator A has a total of + equally spaced states as confidence output. We will show that the expectation (S:) of all length-/ sequences could be exactly calculated given the positive probability p of single token, ie. the general probability of a token showing clear-human signal. As stated previously, tends to be a small value. State transition matrix E ([(+1)x(+++) that represents the contribution of the last token is band sparse matrix consisting of positive transition y and negative transition 1 p to adjacent states from the current state. Defining probability vector at state as E g[(+1), a single transition shown as 3 and the final state probability vector could described as:
0++ = o, of ooP'
(6)
Thus, given one-hot initial state do. we could calculate the final state probability vector and the overall expecation for a sequence of length I:
#(1) = F[A(S)) = (or, a) ooP'
where vector E (B++1) is the sequence vector of all possible positive confidence: a= [i/0)/o Further details and derivations are mentioned Appendix B As result, as text length decreases, the prior positive probability in samples of this length Flength decreases as well. This is in line with our expectation in See 3.1 that shorter texts tend to demonstrate more "unlabeled" properties.
Finally, top of the canonical non-negative PU loss as defined in Eq. 2 we define the Multiscale PU Loss with text-length-varte variant priors:
RMPU (g) (TT, Rr(g, +1)) + Re -1) (ī,, Ar((g -1)),
where II stands for an array: [#(1,]] that records the corresponding prior of training texts, calculated based on respective text lengths using Eq. 7 As is emphasized, short machine-generatee texts should be viewed partially "unlabeled" rather than entirely "unlabeled". Hence, we weight-sum the multiscale PU loss and the canonical PN classification loss to get the final loss for detector model finetuning:
R(g) ( + RMMPOl))
3.4 TEXT MULTISCALING
The proposed Multiscale PU Loss expects training texts of highly variant lengths. but training sets may contain lengthy paragraphs only. Therefore, we introduce Text Multiscaling Module that generates variety of short texts to exert the potential of the length-sensitive Multiscale PU loss. We propose random deletion at sentence scale as a solution. Text Multiscaling module consists of 3 steps: first, complete training text is first tokenized into sentences, denoted as sentence array C: then the sentences are independently and randomly masked based on sentence-wir mask probability Psent- In probabilistic terms, each sentence is decided by an independent Bernoulli trial in the sample space (0, 1). In the sample space, 0 means the sentence is discarded and stands for the sentence is maintained. Finally, all sentences merged again for the multiscaled training text Gaud-
5

---

Published as a conference paper at ICLR 2024
Mathematically, with @ stands for the element-wise Hadamard product, the above process could be concluded as:
Cmaa = M, where M Bernoulli" Pssent).
(10)
The proposed Text Multiscaling module is a one-to-one mapping from C. - Crudt we are not gener- ating more training samples, but substituting the original sample for fair comparison Notably, it is probable that multiscale could leave the original text intact, or only one sentence is left. The relative sequence of remaining sentences maintained avoid breaking excess logical relations between sentences. Multiscaled texts automatically inherit class labels of their original text. The concern for attribution change due to length reduction is to be addressed by the use of Multiscale PU Loss.
experiments.
Though random deletion is also applied Easy Data Augmentation (EDA) (Wei & Zou 2019), our method is different from theirs in two aspects. Firstly, method is focused multiscaling, while word-level random deletion proposed by EDA has limited effect in generating texts of various lengths. Secondly, EDA could break semantic meanings in sentences: deletion of keywords could change the class of sentence; while a more integrated, sentence-level deletion reduces the chance of class property change.
4 EXPERIMENTS
4.1 SETTING OVERVIEW
Datasets. We choose TweepFake (Fagni et al. 2020) and HC3 (Guo et al. (023) as benchmarks for our experiments. TweepFake (Fagni al. 2020 is dataset of tweets for Al-generated microblog detection. Since latest LLMS have completely reshaped the task of Al text detection, we also adopt HC3 (Guo et al., 2023), which is an up-to-date ChatGPT text detection dataset including both English and Chinese. Additionally, HC3 has short-text benchmarks: 1033-English-Seee and HC3-Chinese-Sent We use these datasets demonstrate the effectiveness of our method.
The length statistics in Table show the distribution similarity of English short-text benchmarks, iee. TweepFake (that consists of tweets) and HC3-En-Sent. We conclude from the statistics that the adopted HC3 short-text benchmark could simulate the fragmented language environment (e.g. Twitter) mobile apps. Detector evaluation these short-text benchmarks could reflect their real-world detection capabilities in seaatehone-related scenarios.
Benchmark
Mean
Std
Q1
Q2
Q3
TweepFake (Fagni al.
2020)
24.82
15.19
13
21
34
HC3-En-Sent (Guo et al.
20023
24.98
15.47
15
31
Table 2: Token length statistics of short-text benchmarks. HC3--nglish--em has similar length distribution as TweepPake. These short-text benchmarks could simulate languages that we encounter in Instant Messaging and Microblogging Apps, like Twitter.
Detectors. BERT (Devlin al. 2018) and RoBERTa (Liu et al., 2019) are adopted to apply our MPU method, due to their popularity and supreme performance in previous Al text detection works (Solaiman et al.) 2019; Fagni et al. 2020 Liu et al. 2022 Guo et al. 2023). Training-agnostic detection algorithms are excluded from our consideration.
4.2 TWEEPFAKE DETECTION RESULTS
In TweepFake experiments, we follow Kumarage et al. (2023) for our training settings. Kumarage et al. (2023) is of the latest works on Al-generated text detection, and it claims outstanding
performance on short-text detection. We strictly follow the original training strategy in Kumarage
al. (2023): the model is trained with the AdamW optimizer at batchsize 16 and learning rate 1e 5.
TweepFake mainly consists short tweets. we inspect the dataset and find that a vast majority of texts are single handful of sentences. Hence, refrain from using Text Multiscaling that

---

Published as a conference paper at ICLR 2024
Method
Acc.
BERT-Finetuned (Devlin et al. 2018)
89.1
RoBERTa-Finctuutt (Liu 2019
89.6
RoBERTa- Stylo KKmmrrage et al.
2023
91.1
RoBERTa-MPU (Ours)
91.4
Table 3: Experiments on short-text dataset TweepFake (Fagni et al.
2020
randomly delete sentences for TweepFake datasets; rather, we directly apply Multiscale PU loss during training. As shown in Table the experiment result the proposed MPU promising: greatly improves the performance of finetuned RoBERTa, and its performance outcompetes the latest TweepFake baseline RoBERTa-Stylo (Kumarage et al., 2023) that requires an additional module for stylometric feature extraction during finetuning.
4.3 HC3-ENGLISH DETECTION RESULTS
Method (Fl scores)
HC3-En-Full
HC3-En-Sent
GLTR Gehrmann
2019
96.52
40.19
PPI. Guo ct al. 2023
95.20
62.04
OpenAl (OpenAl) 2023b)
91.00
69.27
DetectGPT (Mitcelll hell et al
2023
87.39
63.32
BERT-Finetuned (Devim
2018
97.62.
57.65.1
RoBERTa-Finetunce (Liu al.
(2019)
97.42=0.92
18.60+
RoBEKTa-Style (Kumarage et al. 2023
96.48
81,46
BERT-MPU (Ours)
98.60=0.52
79.76+3.07
RoBERTa- (Ours)
98.40=0.31
85.31
Table Comparison with English Al-generated text detection baselines on HC3 Guo et al. (003)). Most baselines perform poorly short texts (i.e. HC3-En-Sent); in contrast, our method improves short-text detection greatly.
We also experiment our method on ChatGPT corpora that are much harder to detect. In the ChatGPT text detection experiments, we follow the setting of HC3 (Guo et al. 2023) to test the performance of our method. HC3 (Guo et al. 2023) is a dataset targeted at ChatGPT text detection. All texts are reduced into shorter texts for sentence-level variant. We apply the MPU framework on the full-scale
dataset of HC3 (Guo et al.) 2023).
Several baseline detectors chosen demonstrate the outstanding detection performance of our MPU method. These baselines are open-source and replicable. Among these baselines. GLTR (Gehrmann et al.] 2019). PPL (Guo et al., 2023), and DetectGPT (Mitchell al. 2023) are zero-shot methods that do not require further training: they rely on the likelihood outputs of pretrained language model. The OpenAl Detector (OpenAl 2023b) is RoBERTa detector finetuned on OpenAl's 's GPT-2 (Radford et al. 2019) corpora. RoBERTa Stylo Kumarage et al. (2023) is one of the latest detection baseline targeted for short texts.
and RoBERTa-Finetune
are
language models plainly finetuned on HC3 (Guo et al. 2023), following the official setting: while
BERT-MPU and RoBERTa-MPU are language models trained on HC3 (Guo et al. 2023 the proposed MPU method.
could be observed from Table that most existing methods perform poorly on short texts. The statistics verify our previous claim that the detection of shorter texts is a difficult problem. Specifically, finetuned BERT and RoBERTa are good at detecting long, full-level texts, but they fail to filter out shorter Al-generated texts. On the contrary, our MPU method could greatly improve short-text performances and boost long Al-generated text detection as well. We will further investigate the effect of solitary MPU components in Sec. 4.5
7

---

83.07=6.85 Published as a conference paper at ICLR 2024
Method
HC3-Ch-Full
HC3-Ch-Sent
GLTR (Gehrmann 2019 RoBERla-Finctuned [Li al. 2019
87.40 96.28 97.42++0.24
49.94
RoBERTa- MPU (Ours)
89.37=1.94
Table 5: Comparison with Chinese Al-generated text detection basclines. Our method is also proved effective Chinese corpora.
4.4 HC3-CHINESE DETECTION RESULTS
To verify the generality of the proposed MPU method in other languages, also compare our method with baselines on Chinese Al text detection benchmark HC3-Chinese (Guo et al. 2023). Following Guo al. (022). we use language model. The results are shown in Table[5] Our method could still outcompete other methods by large margins in terms short-text detection, reaching Fl score of 89.37 on HC3-Chinese-Soe
(Cui et al. 2020) as the pretrained
4.5 ABLATIONS
Harmful Short Texts. We elaborate in Section 3.1 that short texts could manifest partially unlabeled property, which impacts the normal training process of the detector. To demonstrate that short texts indeed harmful for training, design experiment based the HC3-Engliie dataset Guo et al. (2023) as follows: when the detector encounters a short training text during training. the training text omitted from backward operations. Other settings identical Section 4.3 As shown in Table 6 finetuning without short texts demonstrates better performance compared with plain finetuning. This reveals that short sentences harmful detector training due their partially unlabeled properties. Hence, PU frameworks need to be leveraged to address this issue.
Method
HC3-En-Full
HC3-En-Sent
Finetuning with texts
97.42 + 0.92
58.60 + 10.53
Finetuning without short sentences
98.19 + 0.66
62.42 + 5.60
Table 6: Performance comparison between the detector finetuned with all texts and detector finetuned without short texts.
Measures
HC3- HC3-English
HC3-Chines
Text Mul. MPU loss
Full
Sent
Full
Sent
X
97.42:10.92
58.60 10.53
96.2811:.42
83.07-1.885
X
96.42.12.27
82.76-12.77
95.89114.18
84.79.15.94
/
97.48++2.41
45.30
96.87+0.89
83.46+45.78
98.40-10.31
85.31+1180
97.42:10.2
89.37:11.44
Table Fl scores of Finetuned RoBERTa ChatGPT benchmark HC3. "Full" and "Sent" stands for model validated on long-text and short-text benchmarks, respectively.
Framework Components. We perform ablations on the solitary effects of Text Multiscaling and Multiscale PU loss.
From Table it firm that the addition of Text Multiscaling to training corpus greatly improves performance on sentence-level corpus detection expected. Unfortunately, the detector's capability on full corpus decays. This performance drop is attributed to the unreasonable label assignment to short corpus from random sentence deletion: the generated short сorpora automatically inherit labels from their full-level predecessors in Text Multiscaling Module, neglecting "unlabeled" properties as introduced in Sec. 3.1 The addition of MPU loss reverses full-level corpus detection performance drop and boosts short-text performance as well. Solitary addition of MPU loss only would have little help for detection performance for lack of short texts.
MPU Loss. We further investigate MPU loss configurations on ChatGPT text detection benchmark HC3-English (Guo et al. 2023
8

---

Published as a conference paper at ICLR 2024
The performance of Multiscale PU loss is evaluated against ordinary PU loss that disregards changes in sentence lengths, as shown in Table [ Multiscale PU loss is sensitive to training corpora of various lengths and thus is more performant compared with its ordinary counterpart.
PU type
Full
Sent
Ordinary
97.05=22.15 98.40-10.31
83.53+3.14
Multiscale
85.31+1880
Table 8: Performance comparison between ordinary PU loss and the proposed Multiscale PU loss.
Introduced in the abstract recurrent detection model (Sec. 3.3 token-wise prior p estimates the probability of a token being highly characteristic human-spoken. As shown in Table carefully tune and found that the best performance is reached at = 0.2, which is small as we expect.
7 Full
Sent
p Full
Sent
Paent Full
Sent
96.42+22.27 82.76+2.76
0.1 96.29+1..11
86.06+11977
97.48++2.41
45.30
0.2
96.52:00.88
83.94+4.07
0.2 98.40-1.31
8..0
0.1 9..73=11422
76.84
98.40:10.31
85.31
0.3 96.81
84.1712.78
0.25 98.40-10.31 85.31
0.6
97.42:10.13
85.7811.19
0.4 9..44 82.88
0.4 97.45-11.34 87.11
0.8
96.90:11.49 84.5442.09
Table 9: Ablation experiment results hyperparameters: loss proportion y, the estimated probability of a token being clear-human p. and sentence mask probability Psent.
We also carefully adjust the affine weight hyperparameter for PU loss y, as shown in Table 9 As the affine weight 7 for PU loss gradually increases, the full-level corpus detection performance reaches the peak at 7 = 0.4 and then drops, while the sentence-level performance reaches its peak at 7 = 0.6. From a comprehensive perspective, the best overall performance reached at r = 0.4 where both performances on full and sentence-level corpus are satisfactory. The climb-and-dros trend reveals that short machine- genieeede sentences are not completely unlabeled; short-text classification should be viewed partial PU problem rather than a complete PU problem.
Further, we test the advantage the non-negative risk estimator in the nnPU loss (Kiryo al. 2017
against uPU loss (Du Plessis et al. 2014). as introduced in Sec. 3.2 The results shown Table 10
Loss type
Full
Sent
Unbiased PU (Du Plessis al.
2014)
97.90.10.25 98.40
84.87111228 8.83
Non-negative (Kryyo al.
2017
Table 10: Performance comparison between ordinary PU loss and the proposed Multiscale PU loss.
Text Multiscaling. As introduced Sec. 3.4 we randomly mask sentences of the training set at probability Peent for multiscale text augmentation. We investigate value. The statistics are shown in Table When Psent is set at 0.25, the test performance on both full and sentence level corpus satisfactory; when Psent is set too high, sentence-level detection performance is enhanced, but full-level performance is negatively impacted because the full-scale training texts are overly damaged.
tuning Peent for the optimal
5 CONCLUSION
This paper proposes a Multiscale Dositve-Unlabele (MPU) framework for Al-generated text detection. We look at the iffy attribution of short Al-generated corpus, and model AI text detection partial PU problem. MPU loss and Text Multiscaling Module are to augment detectors' discriminative ability on short corpus.

---

Published as a conference paper at ICLR 2024
ETHICS & REPRODUCIBILITY STATEMENT
This paper proposes a training method for Al-generated text detectors. Despite outstanding perfor- mance multiscale texts, chances that the detectors output the wrong attribution of a certain piece of text. This may cause ethical issues when the detector is used for detecting plagarism, fake news, et cetera. Hence, we strongly recommend that results from the detector could only serve as reference in actual applications.
Experiments are reproducible. We have attached complete training settings in the Appendix; we also fix random seeds in our codes for the ease of replication. All details are in Appendix
ACKNOWLEDGEMENT
This work is supported by National Key R&D Program of China under Grant No. No.20022DD0160N and National Natural Science Foundation of China under Grant No. 60.62276007. We gratefully acknowledge the support of MindSpore, CANN and Ascend AI Processor used for this research.
REFERENCES
David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Generating mnntiment-preeervig fake online reviews using neural language models and their human- and machine-basse detection. In Leonard Barolli, Flora Amato, Francesco Moscato,
Tomoya Enokido, and Makoto Takizawa (eds.), Advanced Information Networking and Applica- tions Proceedings of the 34th International Conference on Advanced Information Networking and Applications, AINA-2020, Caserta, Italy, 15-17 April, volume 1151 1 of Advances in Intelligent Systems and Computing, pp. 1341-1354. Springer, 2020. doi: 10.10079 10-1107/978--0000-440 URL https: doi org 100.110000/00 978 3- 030 440 34041-11 114
1-030-44041-1
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109:719-760, 2020.
Tristan Bepler, Andrew Morin, Micah Rapp. Julia Brasch, Lawrence Shapiro, Alex J Noble, and Bonnie Berger. Positive-unlabeled convolutional neural networks for particle picking in cryo- electron micrographs. Nature methods, 16(11) ((11))1153 1160, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 141655 2020.
wwwpxiiv.oo./aa.ooo abs 2005. 14165
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning, pp. 1510-1519, PMLR, 2020.
Evan Crothers, Nathalic Japkowicz, Herna L. Viktor, and Paula Branco. Adversarial robustness of neural-statistical features in detection of generative transformers. In International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022, pp. 1-8. IEEE, 2022. doi: 10. /ttp://CCNNNS5542222222999990999 2022. 9892269
URL htt.::ttp:///cmm http:://ooo.oggg////
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre- trained models for Chinese natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: Findings, pp. 657-668, Online, Novem- ber 2020. Association for Computational Linguistics. URL https: www. www.ebcc.bbbro. org/ anthology 2020 findings emnlp..
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abss/881000400s 2018. URL
http::/arrivv wwt.aawwww.....g.oo.aoo//e
www./ogg///b///e. 04805
10

---

Published as a conference paper at ICLR 2024
Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. Advances in neural information processing systems, 27, 2014.
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi.
Tweepfake: about detecting deepfake tweets. CoRR, abs/2008.00036, 2020. URL https
//arxiv.orgy wwt. abs/ 2008. 00036
PudanNLPLab. Sniffer. Website, 2023. sniffer. fastnlp.tog
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. GLTR: statistical detection and visualization of generated text. In Marta R. Costa-jussà and Enrique Alfonseca (eds.), ), Proceedings of the 57th Conference of the Association for Computationnal Linguistics, ACL 2019, Florence, Italy, July 28 August 2, 2019, Volume 3: System Demonstrations, pp. 111-116. Association for Computational Linguistics, 2019. doi: 10. 1/933399// URL https: https://aoi.ozg//1 18653/v1/p19-300
Biyang Guo, Xin Zhang. Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. CoRR, abs/2301.07597 2023. doi: (0445550/aXX1.2231.77753 URL
48550/arXiv 075977
Zayd Hammoudeh and Daniel Lowd. Learning from positive and unlabeled data with arbitrary positive shift. Advances in Neural Information Processing Systems, 33: 13088-13099, 2020.
Fengxiang Hc, Tongliang Liu, Geoffrey I Webb, and Dacheng Tao. Instance-dependent pu learning by bayesian optimal relabeling. arXiv preprint arXiv: 1808.00 02/.00088000 2018.
Cho-Jui Hsich, Nagarajan Natarajan, and Inderjit Dhillon. Pu learning for matrix completion. In International conference on machine learning, pp. 2445-2453, PMLR, 2015.
Dino Ienco and Ruggero G Pensa. Positive and unlabeled learning in categorical data. Neurocomput- ing, 196: 13-124, 2016.
Mangi Kang, Jaelim Ahn, and Kichun Lee. Opinion mining using ensemble text hidden markov models for text classification. Expert Syst. Appl., 94:218 2018. doi: 10.1016/j.swaa.217.000 019. URL
1016
07....77... 019
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with selection bias. In International Conference Learning Representations, 2019. URL
https::/openrevieew net/ o
Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems, 30, 2017.
Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott W. Ruston, and Huan Liu. Stylometric detection of ai-generated text in twitter timelines. CoRR, abs/2303.03697, 2023. doi: 104445550/axxxi22333..6666 2303. 03697
URL htt /00. 415501..88553244.
Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE international conference data mining, pp. 186- IEEE, 2003.
Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Yu Lan, and Chao Shen. Coco: Coherence- enhanced machine- -eeeenatd text detection under data limitation with contrastive learning. CoRR, abs/2212.10341.. 2022. doi: 10448500arxw..222..3444 URL hattps://coi.or:
48550/arXiv wtt.:riii...22......
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11699 16692, 2019.

---

Published as a conference paper at ICLR 2024
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pp. 1045-1048. Makuhari, 2010.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generaai text detection using probability curvature. CoRR, abs/2301. 11305, 2023. doi: 10.48550/arxiv.2311...
URL /tttps://oi. http:///oooorr/1//0///
48550/arXiva 2301 11305
Sandra Mitrovic, Davide Andreoletti, and Omran Ayoub. Chatgpt or human? detect and ex- plain. explaining decisions of machine learning model for detecting short chatgpt-generated text. CoRR, abs/2301. 2023. doi: 10.48500arXx.2231.18822 URL bttos://doi
005/10.485551 arxiv. 1.301.11857)
OpenAl. Introducing chatgpt. Website, 2022. http:///peeal..co//blo////daaad
OpenAl. Gpt-4 technical report, 2023a.
OpenAl. Ai text classifier openai api. Website, January 2023b. https://platform openaai
Com/ ai -ext-t classitiassiiier
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. Distantly supervised named
entity recognition using positive-umlabeled learning. arXiv preprint arXiv:1906.01378, 2019.
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Ma- chine Learning Research, pp. 1394, Lille, France, 07-09 Jul 2015. PMLR. URL
mlr wttpppss/////pesssssssss..
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Yuan-Hai Shao, Wei-Jie Chen, Li-Ming Liu, and Nai- Yang Deng. Laplacian unit unie-hyperplane learning from positive and unlabeled examples. Information Sciences, 314:152-168, 2015.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, and Jasmine Wang. Release strategies and the social impacts of language models. CoRR, abs/1908.00 092003 2019. URL httthttp///axattt hhtp::://rr....orr///bb/////////
Guangxin Su, Weitong Chen, and Miao Xu. Positive-unlabelld learning from imbalanced data. In IJCAI, pp. 2995-3001, 2021.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling.
In INTERSPEECH 2012, 13th Annual Conference of the International Speech Communication Association, Portland, Oregon, USA, September 9-13, 2012, pp. 194-197. ISCA, 2012. hRL //ww. isca spwech-ppeehh...e wwwppeechhh.g//ccccocceooo htt.://eeepppccc.ccc/c//cceeeeee /ttp:teeerreeeeeee///e22222 2ttp//1220011111 0194.htm1
Zhenwei Tang, Shichao Pei, Zhao Zhang, Yongchun Zhu, Fuzhen Zhuang, Robert Hoehndorf, and Xiangliang Zhang, Positive-unlabeled learning with adversarial data augmentation for knowledge graph completion. arXiv preprint ar arXiv:2205.009004 2022.
Edward Tian. Gptzero. Website, 2022.
mfaaq
Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. CoRR, abs/1901..1196. 2019. URL htttp://axx...org//// abs/
1901. 1119611966
Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive-umlabeled compression on the cloud. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 2561-2570, 2019. URL ac 9277666222b222 6bwdd 757755565664f111dd66553833883333 c5a Abs
wew.eeeewddiine.nneeseee.......e http:///ww....ooooooo///pp////// /aaperrr//099/ahh/
12

---

Published as a conference paper at ICLR 2024
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. 9051-9063 2019, URL /ttp///tt// www.iiinnnn.n.ii............ Beff/0eee222000013002223441111c0 www.rrttt.ttttttttttoo.o. html
cc/paper//01s/hahh)
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A PU LOSS DERIVATION
PU losses derived from the canonical binary classification framework. In the standard supervised binary classification (or Positive-Negative classification, abbreviated as PN), let ग := p (Y = +1)-
RR+ON be the prior probability of the positive class, g : R + R be arbitrary decision function (in
case, the detector model) and L. be the loss function. The risk of g defined the expectation of loss:
R(g)= (((
(X))),,
= [(0)) (L, -1)]
(11)
= (1
In canonical PN learning, R(g) can be approximated directly by losses calculated from training data follows:
RPN(1) = RRp(g -1- ():
(12)
where Rr(g,+1) +1) Σ L(g(x and RN -1) := mations of the positive and negative risk, respectively.
+ Ay E L(g(x;") -1) are esti-
In the PU framework, (g -1 cannot be approximated directly via negtive samples, Alternatively,
some works (Du Plessis et 2014 Plessis al. 2015) perform indirect approximation as follows:
defining (z) = p(=) Y +1 and (x == p = 1), since
(1-T)pn(x) p(x) <pp(2),
(13)
negative risk part (which is an expectation) obtained as
(1- (1-+) Rx -) = (u -1) -RR(g. -1);
(14)
and R(g) can be approximated indirectly
Rupu (g) = TR +1) π RRp 11) 1)= + Ry (g. -1),
(15)
where Rr(g.-11 Σ (g 1) and Ro (g. -1) = Σ ay (gg(x)), -1) are esti-
==1
mations calculated from positive and unlabeled training samples. Eq PU (uPU) loss (Du Plessis et al. 2014).
15
is defined as the unbiased
B ESTIMATION DETAILS OF CONFIDENCE EXPECTATION
The transition matrix Given positive probability of a single token, express state transition as band matrix P. An example matrix form of listed as follows:
p
0
0
0
0
1
p
0
p
0
0
0
1
-
0
0
0
0
0
0
0
1- p
0
P p
0
0
1-p
Demonstration of स increment with respect to lengths We try to mathematically demonstrate that prior स increases with length 1. The initial state dg is one-hot, so the prior मे (1) with respect to / could be written as:
#(1) (A((S) P'o P(n, Pl-101
(16)
14

---

Published as a conference paper at ICLR 2024
where P(n, |] represents the last row of transition matrix P. To demonstrate सt increases with 4, we alternatively demonstrate (1 + 1) - ( = E [A(S++1] - E [A())) positive.
However, sizes of states and transition matrices are different for corpora of different lengths. We subscript indicate this difference. For instance, sequence vector oj [i/U] indicates all possible confidences in a sorted sequence; P/ indicates the transition matrix P of size (7+ 1) x (1+1).
Then:
[ E [A()) P/+1 (tppl////Poooo//pp P:(m,:)P/-10
(17)
Interestingly, we could leverage unique features of the sparse band matrix First, obviously P141[n,:| [0; Pr(n. :I]. Further, if we compare
M = Pitim, P : RR'+2 and K := Pi(m, E ERt,1
we would discover that M = [0; K], namely, array M is array K prepended by zero. (The physical meaning of M and K the last line of matrix P and P/ respectively.) Based on this discovery, we could simplify Eq. 17
E
- -E(AS)) = K]JKPP44444 - Kaf
(18)
Then we look at the concrete form of [0; KKJp++1 For simplicity, we denote the n element of K as kn
Count
0
1
2
00 K]
ko (1- k)
kn- ph. + (1 k)
kn plan-1 + pk.
[0; K]414
(1 - ku
pka + (1 - k2
Based the table above, we could derive the relations between E [[(S++1)) and E [A(S))
En-o nkin
2p
kip
En-o nkin
E [A - [[((S))
+
1+1
I+1
[+1
1
Enzo Whin
+ 2p kip
(19)
(1 + 1)
1
E [SS)]
2p kap
+
1+1
1+1
which means that
E [[A = T+1 B[A(S)) + 2p-
(20)
7++
As long as we view x1x E (S:))) sequence corpus length starting from 1 E(A(s)) p.
we could solve E (S,)] for 1:
(27 1)p - E1 knnn 2p
E [A((S)]]
P (1+ 1
Σ kn,n); n=1
(21)
1
where kn,n is the probability of the abstract recurrent model outputting positive confidence for corpus of length n. However, we encounter the difficulty that the analytic solution to kn. is not easily solvable; we only know that kn,a is a probability bounded in (0, 1). We inspect kn,, for relatively small p and found that kn, quickly converges to O. This process is demonstrated by Figure I where kn., decays an approximately exponential manner to infinitesimally small
values (which decays much faster than reciprocals, i.e. 1/1). As result, prior â keeps increasing
15

---

Published as a conference paper at ICLR 2024
increases, and converges to 2p. Figure (Right) confirms the convergence derived in Eq. 21
0.8
0.7
-10
0.6
Kan (in log scale)
-15
Prior
0.4
0.3
-25
0.2
-30-
p=
0
0.1
-35
p 0.3
0.0
10
20
30
40
50
60
10
20
30
50
Corpus length
Corpus length
Figure 1: Left: knnn (in log scale) with respect to corpus length 1. Right: with respect to corpus
length I.
C PROPOSAL OF IMPOSING SPACE CLEANING ON THE HC3-ENGLISH BENCHMARK
We the HC3 (Guo al. 2023) benchmark for ChatGPT corpus detection experiments. However, we inspected HC3 corpora and discovered that the corpora are flawed: human corpora have additional spaces before punctuations, while corpora from AI do not have this feature. The extra spacing could directly impact the input to detectors. We list several examples below, demonstrating the obvious difference between Human and ChatGPT corpora in the HC3 benchmark (Guo al. 2023):
labeled as Human
corpus "Basically there are many categories of Best Seller input_ids = [0, 34480, 89, 32, 171, 6363, 9, 22, 2700, 44795, 22, 479, 21
corpus 'Same thing for best sellers input.ids [0, 42271, 631, 13, 275, 12649, 479, 2]
corpus 'Also URC the rankings change every week or something like that input.ids [0, 22412, 2156. 3082 5199, 5, 8359, 464, 358, 186, 50. 402.
101, 14, 479, 21
# labeled ChatGPT corpus 'It is generally acceptable ethical advocate for condone assassination of any individual, regardless of their actions or beliefs."
input.ids = [0, 243, 16, 3489, 45, 9796, 50, 13557, 7, 7156, 13, 50, 35005, 5, 16351, 9. 143, 1736, 6, 6069, 9. 49, 2163, 50, 9734, 4, 21
corpus 'There are also practical considerations at play in this situation.'
input-ids [0, 970, 32, 67, 7708, 19199, 23, 310, 11, 42, 1068, 4, 21
corpus 'It can also lead to further conflict and instability the region."
input.ids = [0, 243, 64. 67. 483, 7, 617, 3050, 8, 16826, 11, 5, 976. 4. 21
In the examples, show original corpus well their token ids after being processed by the RoBERTa-Bee base tokenizer. Most human corpora have an unexpected 479 token (standing for " i.c. space and period), while ChatGPT corpora does not manifest this feature.
16

---

Published as a conference paper at ICLR 2024
Hence, the detector could judge the attribution of certain corpus simply by detecting these spacing mistakes. Embarrasingly, if we use the logical judgement whether token id 479 contained in the sequence to detect human corpora, the F1 score would reach 82.12% sentence-level test corpora of the HC3 benchmark. The performance of such a simple logic is even better than the officially reported performance (81.8%%) of finetuned RoBERTa-bars (Guo al. 2023). Above all, we strongly recommend later works that involve the HC3 benchmark to remove unnecessary spaces before punctuations. We will opensource the code simple cleaning helper function that removes unnecessary spaces.
D BASELINE REPLICATIONS
D.1 DETECTGPT
2023) is latest open-sourced Al corpus detection baseline, but the original paper did not report its performance on latest LLM texts. Hence, we replicate DetectGPT on the HC3-English (Guo al.) 2023) ChatGPT corpus dataset, and compare it with our MPU method. The experiment results are shown Table where our MPU method outcompetes DetectGPT by large margins. There is still a visible gap between latest arainicg-agnostic methods (e.g. DetectGPT) and finetuned language models on ChatGPT corpora.
DetectGPT (Mitchell et al.
We also provide some detailed procedures to tailor DetectGPT for the HC3 benchmark: 1. Full- scale HC3 corpora are always too long to perturb. Therefore, we truncate corpora as long as they raise perturbation errors, following recommendations from authors of DetectGPT. 2. We 100 perturbations for full-scale HC3 corpora (following DetectGPT (Mitchell et al. 2023)), but we use many corpora. It also reflects that
10 perturbations for sentence-level HC3 because there DetectGPT is not very efficient for large-scale corpora compared to language model detectors, because requires tens of model runs for a single corpus. 3. DetectGPT uses AUROC as the classification metrie; however, this metric is not applicable to finetuned language models that output probabilities for respective classes. Hence, given confidences of all corpora outputted from DetectGPT, we choose 1000 squally-spaced threshold between max and min values, and maintain the threshold with the largest FI score. Notably, this will provide an upperbound for the performance of DetectGPT, as in real applications the threshold is pre-set; scanning for the best threshold on test sets is strictly prohibited.
D.2 GLTR, PPL, OPENAI
These methods have already been open-sourced on HuggingFace. We directly input all texts in the testset to these baseline methods and measure their performances.
We have found inconsistency in comparison reported values while replicating GLTR Gehrmann ct al., 2019) and RoBERTa-Finetunee (Cui et al., (020) on the HC3-Chinese (Guo et al. 2023)
benchmark, shown Table II This inconsistency is tolerable and won't affect our final conclusion.
Method
Full
Sent
GLTR (Reported by Guo ct al. 2023
89.61
44.02
GLTR (Replicated)
87.40
49.94
RoBERTa-Finetuned (Reported by Guo et al. (2023)
98. 79
83.64
RoBERTa (Replicated)
96.28+3.42
83 83.07=66.85
Table 11: Our replication of HC3-Chinese Guo al. 2023
baselines compared with reported values.
E REPLICATION DETAILS
Following the training setting of Kumarage et al. (2023), we use batchsize 16, learning rate 1e
for TweepFake; following the setting of Guo et al. (2023), we use batchsize 32, learning rate 5e for HC3. AdamW optimizors are adopted. Selected benchmarks are publicly accessible online.
17

---

Published as a conference paper at ICLR 2024
We use single Nvidia Tesla V100 the device for experiments. A single epoch of training costs around 30 minutes. We replicate all experiments three times to avoid fluctuation, using seed=0,1,2. The codes are opensoureed at GitHub and Gitee.
18


Reviews:
[{"review_id": "aoaNCgIdXm", "reviewer": "Reviewer_o5iQ", "summary": "This paper addresses the challenge of detecting AI-generated text, which is particularly difficult for short texts. The authors acknowledge that some short texts generated by AI closely resemble human-generated ones and should be labeled as \"unlabeled.\" To tackle this issue, the paper proposes a partial PU method designed for the detection of both long and short texts. Additionally, the authors introduce a corresponding loss function and employ a recurrent model to estimate the prior of P data. The experimental results demonstrate significant success.", "questions": "What specific factors contribute to the difficulty in detecting short text? The paper seems to suggest that the challenge lies in the resemblance between AI-generated short texts and human-generated ones. If this is the case, is it truly necessary to distinguish between them? Additionally, is it feasible to accurately differentiate between them?\n\nCould you provide further insights into why \\tilde \\pi exhibits variation with respect to text length? It appears that it should reflect a characteristic of the dataset's distribution rather than the length of individual data points. Clarification on this point would enhance the understanding of the methodology.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem tackled in this paper is novel. The novelty of the problem addressed in this paper lies in the unique challenge of detecting AI-generated text, especially in the context of short texts. This is an important and emerging area of research with potential applications in various domains.\n\n2. The proposed method is innovative, especially in its consideration of prior estimation dynamically. The experimental results are also promising.  The experimental results demonstrate the effectiveness of the proposed method, which further strengthens the validity of the approach.\n\n3. The paper is well-written and presents the concepts clearly.", "weaknesses": "1. The organization of the paper could be improved. Section 2 contains a substantial amount of background information (Sections 2.1 and 2.2), and some content overlaps with Section 1. It is advisable to reorganize and eliminate redundancy for better flow and clarity.\n\n2. There are some instances of misleading information in Section 2.2 regarding the estimation and expectation of PU learning. Specifically, the description above Eq1 poses a problem as \\hat R represents an estimation rather than an expectation. Similarly, Eq 2 may hold for expectation but not necessarily for empirical estimation. Additionally, \\tilde \\pi should ideally be identical to \\pi for the expected version of Eq 3 to be valid. These points should be clarified and corrected for accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKyMuVwpXX", "reviewer": "Reviewer_scbq", "summary": "It is difficult to classify shorter text as AI or human-generated text due to simple structure or phrasing. Motivated by this, the paper presents an innovative approach to classifying text as either AI or human-generated by framing it as a partially unlabeled (PU) task. The authors introduce a length-sensitive prior that complements the original PU loss function. They leverage a recurrent discriminator to determine the extent of unlabeled properties in each instance and introduce a multi-scale training strategy involving random sentence removal based on a Bernoulli distribution. The experimental results demonstrate that this approach yields better performance for short texts.", "questions": "1. In Table 4, it's intriguing to observe that BERT-MPU performs better than Roberta-MPU on full text but worse on short text. The authors could provide further insights or hypotheses as to why this performance gap exists. This could help readers better understand the nuances of the approach and its applicability in different scenarios.\n2. It would be interesting to know whether the model's performance on classifying short texts would improve if the training dataset only consists of short texts.\n3. Does the proposed method harm the performance of extremely long texts? For extremely long text framing classification as a PU task may not be a good option.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2.  With the increasing popularity of LLMs, classifying AI or human-generated text is a crucial task. Short texts present a unique challenge due to their simplistic structure, making this research particularly relevant.\n3. The motivation behind formulating AI-generated short-text classification as a partially unlabeled task is reasonable and aligns with the properties of classifying short texts.\n4. The introduction of a PU loss function that takes text length into consideration is a novel and promising contribution to the field.", "weaknesses": "1. It's worth noting that some hyper-parameters require careful tuning for optimal performance. The authors should consider discussing the practical implications of tuning parameters like γ, especially when dealing with different large language models.\n2. To enhance the paper's overall impact, it would be more convincing if the authors conducted an analysis of the performance of general recurrent language models. This analysis could include examining how well such models align with ground-truth labels and providing case studies to illustrate their effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHEQi1t1mZ", "reviewer": "Reviewer_PQsn", "summary": "This paper is motivated by the challenge of classifying short texts as either human or machine generated, i.e. fake text detection of short documents. The paper suggests treating short documents from machines as *unlabeled*, under the premise that such texts are in fact indistinguishable from human-written ones in some cases. From this assumption, the \"hammer\" of positive-unlabeled (PU) learning is applied to the problem. Specifically, a length-sensitive PU approach is proposed to account for the prior probability of positive-labeled data, which requires a \"multi-scaling\" data augmentation (sentence-level masking) to introduce a variety of sentence lengths at training time. On the TweepFake dataset, the approached approach outperforms a BERT-based classifier by 0.3 in terms of accuracy. On the more recent HC3 dataset, the proposed approach appears to outperform BERT-based classifiers by about 1% in accuracy for English and Chinese on full documents with larger improvements on sentence-level tasks (e.g., 85% vs 81% for a vanilla classifier for HC3-En-Sent)", "questions": "* It's unclear why a recurrent model is necessary in S2.3. Would an attention-based model also work here?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper addresses an increasingly important application, namely the automatic detection of machine-generated texts.\n* The approach appears to outperforms BERT-based classifiers across two benchmarks, both on long texts and at sentence-level detection. The improvements on sentence-level tasks are notable and appear fairly consistent across datasets and languages.\n* Ablations of the proposed approach are reported, showing the benefits of each component of the pipeline.", "weaknesses": "* The motivation for using a PU framework for fake text detection is not entirely clear. Conceptually, although certain short texts may be hard to distinguish as AI generated, pretending that they are unlabeled seems like an odd choice. Is there evidence that such shorts texts actually harm learning of a classifier using typical proper losses? \n\n* If the benefit is actually prioritizing learning on examples with less irreducible uncertainty, might other frameworks besides PU be appropriate as baselines? (e.g., noise-aware losses)\n\n* How does the use of the PU loss impact calibration? Uncertainty estimation is important for downstream use cases of AI text detectors.\n\n* S2.3 is difficult to understand. There is a lot of complexity without sufficient motivation or intuition for what it is aiming to accomplish, or discussion of alternate (simpler) approaches.\n\n* Eqns (12) -- (14) introduce a lot of unnecessary notation that is never re-used elsewhere in the paper.\n\nEDIT: Dear authors, thank you for the detailed response. I have updated my review accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "306F58inqd", "reviewer": "Reviewer_vhmU", "summary": "This paper models the task of AI-generated text detection as a partial Positive-Unlabeled (PU) problem and formulates the Multiscale Positive-Unlabeled (MPU) training framework to address the challenging task of short text detection without sacrificing long texts. In the PU context, this paper proposes length-sensitive Multiscale PU Loss, where a recurrent model in abstraction is used to estimate positive priors of scale-variant corpora. Besides, this paper introduces a Text Multiscaling module to enrich training corpora. Experimental results can demonstrate the effectiveness of the proposed method.", "questions": "Please check the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- It is interesting and meaningful to investigate the task of AI-generated text detection.\n- It is a novel perspective to address the task of AI-generated text detection by taking it as a partial Positive-Unlabeled (PU) problem.\n- A novel Multiscale Positive-Unlabeled (MPU) training framework is proposed to address the challenging task of short text detection without sacrificing long texts.\n- A Text Multiscaling module is also introduced to enrich training corpora.\n- Experimental results are extensive, which can demonstrate the effectiveness of the proposed method.\n- Ablation studies also support the proposed method.", "weaknesses": "- The writing of this paper should be further polished. For example, the double quotation mark was mistakenly used in the paper. For the typeset, there are too many tables that record the experimental results in the Experiment section. Maybe some tables can merged into a single one. It would be better if some tables could be transformed into figures. There are some words that are on the 10th page, but the requirement is that the main content cannot exceed 9 pages. \n- The review of related work can be further improved. Section 2.2 talks too much about PU learning. I think it is not necessary to provide the unbiased PU loss and the non-negative PU loss in such a detailed manner. In Section 1 and Section 2, some parts could be arranged into a single Related Work section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)