PaLD: Detection of Text Partially Written by Large Language Models
Authors: Eric Lei
Hsiang Hsu
Chun-Fu Chen
Venue: ICLR 2025 Poster (2025)
Keywords: LLM text detection
mixed-text detection
distribution shift
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Advances in large language models (LLM) have produced text that appears increasingly human-like and difficult to detect with the human eye. In order to mitigate the impact of misusing LLM-generated texts, e.g., copyright infringement, fair student assessment, fraud, and other societally harmful LLM usage, a line of work on detecting human and LLM-written text has been explored. While recent work has focused on classifying entire text samples (e.g., paragraphs) as human or LLM-written, this paper investigates a more realistic setting of mixed-text, where the text's individual segments (e.g., sentences) could each be written by either a human or an LLM. A text encountered in practical usage cannot generally be assumed to be fully human or fully LLM-written; simply predicting whether it is human or LLM-written is insufficient as it does not provide the user with full context on its origins, such as the amount of LLM-written text, or locating the LLM-written parts. Therefore, we study two relevant problems in the mixed-text setting: (i) estimating the percentage of a text that was LLM-written, and (ii) determining which segments were LLM-written. To this end, we propose Partial-LLM Detector (PaLD), a black-box method that leverages the scores of text classifiers. Experimentally, we demonstrate the effectiveness of PaLD compared to baseline methods that build on existing LLM text detectors.

Published as a conference paper at ICLR 2025
PALD: DETECTION OF TEXT PARTIALLY WRITTEN BY LARGE LANGUAGE MODELS
Eric Lei 2 Hsiang Hsu², Chun-Fu (Richard) Chen?
University of Pennsylvania, 2 *JPMorganChase Global Technology Applied Research eleißseas. upenn. edu, (hsiang. hsu, richard. of chen ) (aamae/ppmcchheee jpwchase.ccmmm... com
ABSTRACT
Advances in large language models (LLM) have produced text that appears in- creasingly human-like and difficult to detect with the human eye. In order to mitigate the impact of misusing M-generrrte texts, e.g., copyright infringement, fair student assessment, fraud, and other societally harmful LLM usage, line of work on detecting human and LLM- L.w-tritten text has been explored. While recent work has focused classifying entire text samples (e.g., paragraphs) human LLM-written, this paper investigates a more realistic setting of mixed-text, where the text's individual segments (e.g., sentences) could each be written by either a human or an LLM. A text encountered in practical usage cannot generally be assumed be fully human fully MLW-writte;; simply predicting whether it is human or LLM-written is insufficient as it does not provide the user with full context on origins, such as the amount of LLM-written text, or locating the LLM--ritten parts. Therefore, we study two relevant problems in the mixed-text setting: (i) estimating the percentage of text that was
and (ii) deter- mining which segments were LLM--ritten. To this end, we propose Partial-LLM Detector (PaLD), a black-box method that leverages the scores of text classifiers. Experimentally, we demonstrate the effectiveness of PaLD compared to baseline methods that build on existing LLM text detectors.
INTRODUCTION
Large language models (LLMs) have demonstrated capabilities to generate text that convincingly impersonates humans' (Achiam et al., 2023; Floridi & Chiriatti, 2020; Chowdhery et al., 2023). In conjunction with their wide deployment and easy accessibility, these LLMs have posed potential risks across industries and society. Namely, LLM-generated text may contaminate the development of next-generation foundation models (Shumailov et al., 2024; 2023), facilitate the spread of fake or biased content (Bender et al., 2021: Farina & Lavazza, 2023; Li et al., 2023), unintentionally infringe on copyrights (Mitchell & Krakauer, 2023), and impair education by depriving students of the effort needed to compose their own articles (Cotton et al., 2024). The potential downsides of LLMs, particularly in scenarios where humans can be easily deceived by text generated by these models, underscore the need for reliable methods to audit and detect LLM-generated content.
Given an article, such as a paragraph, existing methods for detecting ALM-ggnerated text are often cast binary classification problem, i.e., assigning binary label to the article to indicate whether was written by a human or generated by an LLM (Gehrmann et al., 2019; Ippolito et al., 2020; Mitchell et al., 2023; Mao et al., 2024; Verma et al., 2024). However, the binary classification approach may fail to provide finer rre-ggr information about the extent of LLM involvement in composing the article, it may not be entirely written by LLMs. In fact, recent studies have shown that LLMs are frequently used to edit, refine, rephrase only parts of an article (Crček & Patekar, 2023; Levine et al., 2024) see Fig. for a concrete example. Moreover, when using existing methods to detect text that is partially written by LLMs, they may either exhibit excessive confidence in identifying the whole article as LLM-generated content, or be overly conservative
Work done while an intern at JPMorganChase.
Recent works reported that L.I M-geerrrete text is difficult to detect with the human eye (Mei et al., 2024; Gehrmann al., 2019; Guo al., 2023), and could conditionally pass the Turing test (Jones & Bergen, 2023).

---

Published as a conference paper at ICLR 2025
1 human
Existing LLM Detectors
2 human
LLM t rustled the wind. squirrel darted tree. The forest was alive with sounds.
Human &
C LLM
①
Percentage Estimation (PaLD-PE)
0% LLM
2 The leaves rustled in the wind. trickled
2
: :33 1 12% LLM, with prob. 0.85
The forest wAs alive sounds.
70=10% LLM, with prob.
3 The leaves rustled in the wind. Water trickled down strean. The sun shone on the trees.
Text Identification (PaLD-TI)
① none LLM
sentence 2 is LLM
sentence 2, are LLM
In practice, text encountered may he partially-IIIM written. Existing L.M detectors can only predict whether the text is human- or L.-M-written In contrast, method (PaI.D) enables LLM percentage estimation and LLM text identification. PaLD-PE provides confidence intervals the amount of LLM text, and PaLD-TI provides a likelihood of each sentence being LLM.
Figure I:
towards classifying it as human writing, as empirically demonstrated by text 2 and text in Fig. 2, respectively. Incorrectly assessing the level of LLM intervention can lead to unjust penalties for light LLM usage. or encourage unauthorized usage by those who exploit these inaccuracies. These practical concerns, therefore, motivate the need to refine the detection techniques in existing methods.
In this paper, we address a more realistic setting of mixed- texr, where a piece of text to be audited consists of both LLM-generated and human-written content. This mixed- text setting leads to two primary research goals: (i) per- centage estimation, which aims to estimate the proportion of text in an article that was generated by LLMs, and (ii)
Text
Text 2
Production
LLM
Humaag.o
0.00
0.25
0.50 Writtenten
0.75
1.00
LLM text identification, which seeks to identify the spe- cific text segments in the article that are more likely to have been generated by LLMs. Existing LLM-text de- tectors typically design statistic of the LLMs' outputs given the article, which we refer to as the T-score, to de- tect distribution shifts between texts that are either fully written by LLMs or by humans, and then threshold the T-score for a binary decision. However, the mixed-text setting requires a more sensitive statistic detect subtle distribution shifts, such as when the text is composed of 20% human text and 80% LLM text.
Partion
Figure 2: Binary classification of two hu- man texts progressively overwritten by LLM. RoBERTa (Verma et al., 2024) predicts the en- tire text as buman-written when up to 75% of it is LLitteitten; only 25% of text 2 is L.LM-writee before RoBERTa classifies the entire text as LLM.
To address this technical challenge, we develop Bayesian framework based on the T-score, termed the Partial-LLM Detector (PaLD), which provides maximum posteriori (MAP) estimates of the percentage of L.LM-gennnnnn text an article (Cohen, 2019). Using mixture Gaussian kernel density & Zollmann, 2008; Gelman al., 2004), we further derive credible intervals that reliably cover the ground-truth percentage. For LLM text identification, the PaLD framework is statistically more robust, especially when each text segment in an article is relatively short, where existing L.LM detectors lack sensitivity in detecting the distributional shift due to the short text length. We formulate a set optimization problem that searches for the set of segments that maximize the discrepancy of the T'-scores between the set and its complement. This optimization problem can either be exactly solved or effectively approximated by greedy policy, flagging the set of text segments that are more likely to be LLM-generated. We term the PaLD framework for solving the two research goals as PaLD-PE (for percentage estimation) and PaLD-TI (for LLM text identification), respectively. Fig. 1 summarizes the PaI.D framework, along with its use cases.
estimation
The rest of this paper is organized as follows. Section 2, we define notations, provide the mathe- matical background, and survey related work. Section 3 formally introduces the Pal.D framework and demonstrates how to perform PaLD for percentage estimation (PaLD-PE) and LLM text identification (PaLD-TI) with statistical guarantees. In Section 4, we empirically illustrate that PaLD-PE and PaLD-TI outperform existing detection methods on two language datasets: WritingPrompts (Fan et al., 2018) and Yelp Reviews (Yelp. 2014). Finally, we present our concluding remarks, including limitations and future directions, in Section 5. Code to reproduce our experiments can be accessed at https: / //g thub. com/ coom/ppmoreeeeocooooooooeeeeoeoo jpmorganchasee / pald.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND AND RELATED WORK
Let X a random variable of text that follows a text-generation distribution P. which can either be human writing (Piuman) or LLMs (PLLM), and let x E X be a realization from P. Here, X is the sample space of texts; for example, X' = DD would contain all finite-length strings drawn from dictionary D. The goal of L.M text detection is to design a statistic, referred to as the T-score (short for text score), that maps piece of text to a scalar value, i.e., T X + R. The desired property of the T -score is that the distributions of T(X) when evaluated with X ~ P Phoman and X ~ PLIM result in two statistically separated modes. A text sample can then be classified as LLM if T(x) > 7 where 7 is a preset threshold.
T-scores in LLM text detection. Existing LLM detectors design various T T-scores to improve classification performance. Typically, T -ccccess can either be inferred from pre-trained models, as demonstrated by Mitchell et al. (2023) with DetectGPT, explicitly learned for binary classification tasks in models like RoBERTa (Guo et al., 2023) or Ghostbuster (Verma et al., 2024), or through a combination of both approaches, such as in RAIDAR (Mao et al., 2024). For example, Solaiman et al. (2019) and Ippolito et al. (2020) utilize the average log-probability of a text as the T-score. DetectGPT, on the other hand, calculates log-probability curvature, defined (xx) Eg [log (L]M( where 전 represents a perturbed version of x generated by Google's ' T5 (Raffel et al., 2020). This method was later extended conditional log log-probability curvature by Bao et al. (2024) with Fast-DetectGPT, and further explored by Mireshghallah et al. (2024).
log (1)
T'scorrss can also be defined using the logits from LLMs, which are typically thresholded at y for classification tasks, and then used to train a binary classifier (Sadasivan et al., 2023). For instance, Guo et al. (2023) and Chen et al. (2023) fine-tune RoBERTa models (Liu et al., 2019) to differentiate between LLM-generated and human-written texts. Verma et al. (2024) enhances the generalization performance of their models by employing logistic regression classifiers on selected features from LLM token probabilities. Conversely, Mao et al. (2024) introduce new T-score based the rewriting Levenshtein score, opting for this over logits in their based classification method. Finally, Gehrmann al. (2019) use the ranking of the top-k log probabilities the T-score, noting that humte-written texts tend to be sampled more frequently from the tail of a L.L''s probability distribution. Although T-scores are designed for binary classification, work shows how they can be leveraged for the percentage estimation and text identification tasks that we propose.
LLM boundary detection. A few recent works have investigated detecting the boundary when text goes from human-written to LLM-written. The RoFT dataset (Dugan et al., 2020), containing human- written sentences completed by GPT2, was created evaluate how humans detect the boundary. Cutler et al. (2021); Clark et al. (2021); Zeng et al. (2023); Wang et al. (2023) provide RoBERTa or Transformer-base
models for solving the task an automated fashion, with Wang et al. (2023) requiring white-box access to the ground-truth LLM. Kushnareva et al. (2024) evaluates several approaches and finds that perplexity-be proposed framework, PaLD, is more general, as it encompasses settings where any segment of a text could be human or M.ritiii,,n, does not require white-box access to a LLM, and can be used with both supervised T T-scores as well zero-shot T'scores.
approaches are more robust for boundary detection. Our
LLM watermarking King and hallucination. Techniques used to detect distribution shifts in language generation are also applicable in related fields. For instance, LLM watermarking embeds unique pat- tern, or signature, within the output distributions of an L.M to safeguard its authorship (Kirchenbauer al., 2023; Kamaruddin et al., 2018; Zhao et al., 2024). This signature induces distinct distribution shift in the text, making it uniquely identifiable and easily distinguishable from outputs generated by other LLMs Despite sharing similar techniques, LLM watermarking serves fundamentally different purpose compared to LLM text detection, which is the setting of this paper.
3 PALD: PARTIAL-LLM DETECTOR
We start with the formulation of the mixed-text setting by splitting a piece of text x into a concatenation of n segments, i.e., .1... Zn. We assume that each segment x, has either been generated by LLMs (z; ~ PLIM), or written by a human (z; ~ Phuman). ). For example, consider the following paragraph & 222774 decomposed into 4 segments:
3

---

Published as a conference paper at ICLR 2025
100% LLM
T-score
07
T(x)
72.00
RoBEKTa-LN
2,599
RoRE-TTa
2.160
Ghosbuster (Verma al., 2024)
1.889
FanDeteccPPT (Bao al., 2024)
1.784
0% LLM
DaNSIGP (Mitchlll et al., 2023)
1.496
0.00
0.30
D150
0.00
0.80
100
8, 8 8R
Table Normalized quantile slope
Figure 3: Distribution shift of Figure 4: PaLD-PE for percentage
in (1) for different T-ccors. Higher RoBERTa-LN with varying d. is better.
estimation returns point á and inter- val (Ó1, 6g) estimates.
TXX)
-55
-10
0.00
0.30
0:80
0.50
0.50
1.00
0.00
0.20
0.40
0.00
0.80
1.00
0.00
0.70
0.40
0.60
0.50
1.00
0.00
0.20
0:40
0.00
0.00
1.00
(a) RoBERTa
(b) Ghostbuster.
(c) DetectGPT.
(d) FastDetectGPT.
Figure 5: Distribution shift of various T-scores with mixed-text fraction 6.
It was excellent decision. The pancakes were fluffy and bursting with blueberries. They layered the cheese on the sandwich, which to me is a must for true egg sandwich. The meal also came with side of breakfast potatoes.
Black segments (i.e., sentences) 21 and I3 are human- human-written, while red segments z2 and æs are LLM-generated. It is noteworthy that existing LLM text detectors typically operate the setting where that all segments {x:)] ) are either human-written or L.M-generated.
3.1 ADOPTING T --SOORSS FOR MIXED TEXTS
In the binary classification setting, the T-score is used to discriminate fully-human texts X human ~ Phuman and fully-I.LM texts XLLM PLIM Here, there only two distributions of interest, and the T'score merely has to discriminate their samples. Recent work such as Sadasivan et al. (2023) shows that the best T'score (in terms of simultaneously reducing the true and false positive rates) one that separates ((X [uuma)) and T(X as much possible. In the mixed-text setting, the 8th segment X; in random mixed text X follows the mixed distribution, i.e., X; P Pmixed - P) + (1 - 77) P Phumnnn where 0 < 7 S represents the segment- L.MM fraction. same number of characters, we further define å E [0. 1] the character-level LLM fraction of X. The mixed distribution reverts to Phumann when = 0 (thereby making å = 0), and recovers P.LM
each segment may not have
when 8 = 1. As increases, the mixed distribution of X shifts from Pruman to PLLM
Assessing the amount of LLLM-generated text within X equates to estimating ó. Ideally, as 8 varies, the distribution shift in T(X) should smoothly transition between [(Xhuman) and T(XLLM). However, accurate estimation of 8 using the T-score becomes challenging there is significant overlap among the T-score distributions for different 5 values, as multiple values could plausibly explain the same T-score. The degree of overlap can be quantified by the normalized 7 -coore quantile slope, dr(p), defined as follows:
TT(p)= A /
28 (() p)
d8,
[0.1] Q(6, 5) Q(1.))
where Q(š,p) arg arg min, Pr(T(X) Pr(((X)((((( < 5) = p is the quantile function for the T-score distribution at 8. The target is get of min; Sp51- (p) large possible. Here, adjusts the considered
Using the previous example, 0.5 since I.IM generates two sentences. However, there are 109 out of 197 M-generrtee characters, and therefore 109/197 0.553.
4

---

Published as a conference paper at ICLR 2025
quantile range. A high value of oj ensures that ò can be uniquely identified across its entire range. Conversely, a low of indicates substantial overlap in the T-score distributions for different values, complicating the unique identification of ö.
We conclude this subsection with an empirical example. We first prepare the mixed text by mask- and-fill approach. Consider case where = 5 using Humann-writted text Phuman 2. 22 Setting = 0.4, we randomly mask two sentences with the special token [MASK]. This modifies the text to, for example, JLLM = æ1 [MASK] 2g [MASK]. We then prompt GPT-4o to fill in the [MASK] tokens in JLIM, resulting in a mixed text sample with = 0.4. Further details on this approach are provided in Section 1. With the method to generate the mixed text by 17 (equivalently 5), in Tab. 1, we compare the T T'scores of several methods using Jy. We find that the logits with RoBERTa classifier trained with LogitNorm loss (Wei et al., 2022) to classify fully-human/lLLa text yield the best o we refer to this method as RoBERTa-LN. The T-score distribution shift with varying 8 is reported in Fig. 3 for RoBERTa-LNN as well as the other T-sorres listed in Fig. 5. can be seen that all methods achieve good statistical separation between T ('(Xnumnn). i.e. 8 0, and T()L,M,, i.e., 8 = 1. As will be shown in an ablation study in Sec. 4, the performance of PaLD for percentage estimation and text identification highly correlate with the of values of the T -sccorss used. Next, we demonstrate how the T-score in the mixed-text setting can be used to for PaLD-PE and PaLD-TI.
3.2 PERCENTAGE ESTIMATION
Given a mixed-text realization & with ground-truth LLM fraction 8, we would like to produce either point estimate of 8, or a predictive interval (ōL, 0R) that contains å with high probability. A predictive interval provides the user with a measure of confidence on the estimated percentage value of the mixed text x. PaL.DPPE uses a Bayesian approach estimate 6, and thus assumes it is random; we denote the random LLM fraction as A, At a high-level, PaLD-PE first estimates the joint statisties between the LLM text percentage and the T'scores, then uses this model to return point estimates and/or predictive intervals of o; see Fig. 4. In the first step, mixed texts are generated from fully-human dataset measure the shift distribution of the text score T as the LLM percentage ranges from 0 to I; we then fit a mixture kernel density estimate (KDE) to estimate the likelihood (A)()) In the second step, when we estimate the L.LM percentage of an unseen text sample, we use the posterior P(A[T((X). to return maximum a posteriori (MAP) estimates for 8 and highest density intervals (HDI) for ÖR).
Measuring the P(T(X) likelihood. To model ( A), we need to gather pairs of samples representing A and T(X). For X, we first generate synthetic mixed texts by using an LLM to fill randomly masked out sentences of fully-human texts (Sec. A.1). These synthetic mixed texts are
generated at K target fraction levels 0 < 81 < < 8x I 1. Let {x," (xx())aa you denote the mixed texts generated at target fraction 6y- We compute the T-score of each such text, yielding (t.
1 Ji= where (k) is assumed drawn from the P(T(X) 5x) distribution.
(.4) = -xx(()) Thus, each &.
Now, we parameterize a model for the likelihood ((T()) using a mixture of KDEs. An individ- ual KDE is fit for each conditional P(T(X)
5g). For the full conditional P(T(X) take convex combinations of the nearest two KDEs to A. Specifically, let dx
Ee ((((
be a Gaussian KDE fit to the samples (1()) p Here, K(2) = Le e }= ni the number of <2n
T T'-ccore samples collected at dg. and h is bandwidth parameter. Then, our mixture KDE computes the likelihood as
P(T(X)= = =|== = =8)) = -0Øx- (1) + (1 - -0)0k-+1(00
where k* is the index such that dg- < 8' < 6k. and 0
6u 5u-+1-6 8
8g 16-11-b0-
Percentage prediction. To predict the LLM fraction, we use the posterior density P(A[T(X)) xc (A) where we assume prior distribution P(A) supported on [0, 1]. Let x be the text sample would like to estimate the LLM percentage. For the point estimate, we return the
MAP estimate å arg max P(A T')) T(x)). For the predictive interval, we return the
5

---

Published as a conference paper at ICLR 2025
RyApgg The leaves rustled in the wind, A squirnel darted up tree. the forest was alive with sounds.
[2(5) -
x[5]
{13
Xy=
rustled
A(EE))
T(G
[1,2)
Pg7g = The leaves rustled In the wind.
f
squirrol darted up tree.
11,3]
The Jeaves rustled the wind. forest
fx(E1,3))
sounds.
{2)
squirrel
tree.
AA((2)
[2.3)
RyXy= A squirrel darted up a tree. The forest was alive Mith sounds.
(((23)
T(Ky5g)
[3]
The Forest alive with sounds,
Figure 6: PaLD-TI for LLM text identification. Left: stitched texts z[S] are enumerated, and T- scores computed. Middle: (S), the --core difference, computed for all S. We illustrate this for S = {1}. Right: the maximum fa (S) is computed, and its maximizing set S is returned as the segment indices predicted as LLM.
(1 ax--DDI (Chen & Shao, 1999),
8,, 8n
arg max Ta 6n.6n
s.t. p(o A A T()) = T())) 2 Ta P A 5n T(X) T(x)) 2 1
(3)
where is parameter we set control the posterior probability that d is contained in the interval. In practice, since we do not have the exact posterior density, we use a Markov Chain Monte Carlo approach (Chen & Shao, 1999) by sampling ,i,..., 8's from the posterior P(A]T(X) = T(x)) via Metropolis-Hastings (Gelman et al., 2004). For the MAP estimate, we return the sample mode å arg max ISISM P(A ( T(x)). For the (1 - a)-HDI estimate, we return (5,, 5n) = (6 8 (1+(()))M))> where 8/1 is the i-th smallest sample, and '" arg gin +1 (,)
3.3 LLM TEXT IDENTIFICATION
The goal is to return an index set $ C (1,...,m) corresponding to segmentation r such that {z; E S) contains all the LLM-written segments.
One baseline approach could to classify each z, individually using one of the binary classification approaches. However, these methods are designed for longer texts, and are known to perform poorly on short texts (Verma et al., 2024). Our results in Sec. demonstrate the poor performance this approach. Instead, propose "stitch" together different segments of x: construct stitched texts, as shown on the left side of Fig. 6. Concretely, let S C (I.....n] n be an index set which we use to select subset of the segments in æ. Define rS]] to be the text formed by concatenating the segments of x indexed by 5' in the order of the indices. For example, if 5 {1, 3, 4), then r[S] = Zy These stitched texts [SS] mostly consist of multiple segments, and should be of sufficient length for some of the binary classifier methods to be effective. Drawing from observations in Fig. 3, suppose that S selects all the L.LM segments, and 5 == {1,..., n) \ S selects all the human segments. Then x[S] will be fully-humant and x x] fully-LLM, and this should result in the largest discrepancy between [(x[S) and [=(s$1). On the other hand, if x[S] contains a mixture of LLM and human text, then so will ={S0], and their T-scorss should be more similar.
Thus, our goal is to find the S that the T-score maximally discriminates x[S] and z[$0]:
5= arg max fa(S) T(x[S)) T(x[5-1).
(4)
SC (1.....)

---

Published as a conference paper at ICLR 2025
The overall method, which we call PaLD-TI, is shown in Fig. 6. In practice, we disregard S = 0 and s = 9, yielding total of 2" -2 sets to consider. This is be- cause those two cases reduce the L.MM text identification problem to whole text classification, which has been well explored in the previous literature. Thus, Eq. 4 is subset selection problem which is combinatorial and has complexity exponential in n. In our experiments, we consider texts with at most n 10 segments which is feasible to solve Eq. 4 exactly. For texts with more sentences, one can chunk the text into paragraphs before solving Eq. 4, or use approximate algo- rithms (e.g., greedy) that trade-off optimality for efficiency. For example, the greedy algorithm will iteratively build a set by adding a single segment that maximizes the marginal gain of fz( at each iteration; see Alg. 1. This reduces the complexity to (n²) but solves Eq. 4 approximately rather than exactly. We also note that P-LD-TI can be used for percentage estimation via the total predicted LLM length.
Algorithm I Greedy algorithm, PaLD-TI.
Initialize S = arg max fu((f)))
e6 eE(1......g] m
Initialize A (l..... n) \
while fz( increases do
e' = arg max (S(U)) [2(
S +- U
A A
end while
4 EMPIRICAL STUDY
Datasets. We evaluate our methods on the WritingPrompts (WP) (Fan et al., 2018) and Yelp Reviews (Yelp) (Yelp. 2014) datasets which are typically used to benehmark LLM text detection. WP contains pairs of story prompts and human-written short stories as responses, and Yelp contains human-written reviews of various businesses. Both datasets come with the fully-human and fully-LLM (GPT3- rewritten) versions of text; in additional to these, we adopt sentence-level mask-and-fill approach with GPT-4o (Achiam al., 2023) to generate mixed texts at different LLM fractions for both datasets, described in full in Sec. A.1. In total, for each dataset, we generate 3,600 and 300 mixed texts for training and test splits, respectively. For the training split, the LLM target fractions are ranged from 0.1, 0.2, ................................ to 0.9; while the LLM target fractions are set to 0.25, 0.5, 0.75 for the test split, and the amount of data at each fraction are similar as balanced dataset. Note that we measure the L.M fraction at character-level, and the training and testing splits do have similar distribution of LLM fractions over [0, 1] (see Tab. 9).
Experimental Setup. For the T-score used in both PaLD-PE and PaLD-TI, adopt the logits of RoBERTa classifier trained on hullan and fully-LLM text, using the LogitNorm loss (Wei et al., 2022), discussed in Sec. 3. Further hyperparameter choices for PaL.D are described in Sec. A.2. For the percentage estimation (point estimate) and LLM text identification baselines, we adopt the existing LLM text detectors applied segment-wisely to the mixed-text samples, including DetectGPT-Seg (Mitchell et al., 2023). FastDetectGPT-See (Bao et al., 2024). Ghostbuster- ( (Verma et al.. 2024). DetectGPT and FastDetectGPPTs threshold set to maximize the difference between true-positive and false-positive rates when classifying human/LLM segments. In addition, we build another two baselines by fine-tuning the RoBERTa (Liu et al., 2019) as a binary classifier on fully-human and full-LLM texts with two different losses. One is trained with cross-entropy loss (RoBERTa) and another one is trained with LogitNorm (RoBERTa-LN), The predicted percentage is then the total character length of the predicted model segments divided by the total number of characters of the text. We use PaLD-TI TI here in a similar fashion. Moreover, given the generated mixed-text data and L.MM fractions, include another two baselines by fine-tuning the RoBERTa model with regression loss (RoBERTa-Reg) and squared loss and quantile loss Koenker & Bassett, 1978). For RoBERTa-Quann QuantileRegn, we use the 3. 3. and e quantiles, with the 2/2 -quantile serving as a point estimate and the other two serving as a a interval estimate.
et al., 2022;
Evaluation. We report performance averaged across the data with LLM fractions 8 = 0.25, 0.5, 0.75. For the percentage estimation task, we use mean absolute error (MAE), 6 - 8) to evaluate the point estimates; for interval estimates we report two metrics: coverage (C), the frequency the interval covers o (i.c., sL < 8 < JRH,, where X are mixed texts with LLM fraction ò drawn from our dataset), and precision (P), the width of the interval (i.e., Ex (Ō T2.. ). In general, one cannot always increase coverage and minimize precision, as smaller interval reduces the probability that 52 < 8 & 5Rz we report the trade-off between these two quantities. For the text identification task, report segment-wiss wise accuracy (i.e., E E 10 E E or i i 5c SC)) where contains
7

---

Published as a conference paper at ICLR 2025
Table 2: Percentage estimation results (point estimate); mean absolute error. The best method is bold.
Damaer
PaL.D-PE (Ours)
PaI.D-II (Ous))
RoBERTe-
RaBERTo- QuzatileKgg
DetectGPT:
FaeDDeeenCPP.
RoBEXTo>
RoBERTs- LN
Giosstbuttes-
Seg
Seg
Seg
WP Yelp
0.116
0.122 0.158
0.207 0,181
0.186 0.175
0.370 0.397
0.184 0.190
0.342 0.362
0.43 0.477
0.215
0.137
0.282
Table 3: LLM text identification results; segment-wise and top-1 accuracy. The best method is bold.
Dutaset
PaI.D-TI
PaL.-TI-
DetectGPT:
FastDete:GGPT.
RoBERTa
RoBERTa LN
Ghostbuster-
(Ours)
Greedy (Ours)
Seg
Seg.
Seg.
Seg
Seg
Seg
Top-1
Sog
Top-1
Sog
Top-1
Seg
Top-1
Sog
Top-1
Seg.
Top-1
Seg.
Top-1
WP
0.826
0.252
0.741
0.071
0.486
0.001
0.651
0.063
0.635
0,070
0.502
0.000
0.599
0,053
Yelp
0.801
0.247
0.733
0.069
0.452
0.002
0.624
0.054
0.560
0.013
0.506
0.000
0.540
0.027
ground-truth indices of segments of X that were LLM-written), which measures how well each segment is classified. In addition, we report the top-1 accuracy at the set level (i.e., [1 = $))), which measures how often the ground-truth set S is perfectly recovered, of which there are 2 - 2 possible for each mixed text. To achieve high top-1 accuracy is challenging, as a success requires every segment of the text to be correctly identified.
4.1 IN-DOMAIN MIXED-TEXT DETECTION
Results Percentage Estimation. Tab. compares methods (PaL.D-PE and PaLDDTII with other methods on the point estimate results. PaLD-PE significantly outperforms the baselines that adopt segment-wise strategy, demonstrating the limitations of existing LLM text detectors. likely due to the shortness of the individual segments. On the other hand, RoBERTa-Rep and RoBERTa- QuantileReg are the most competitive baselines while our method still surpasses them, showing that the Bayesian approach provides more precise estimates than direct regression. PaLD-TI is superior to all baselines, but not as accurate as PaLD-PE, since its precision is limited to segment-level.
For interval predictions, we sweep a's for both PaLD-PE (corresponding to the (1 H-HHDII and RoBERTa- -aaETa--uaatileRReg (corresponding to the and 1 a quantiles) to get a coverage-press ppreiig-precision trade-off. Fig. 7a shows our method yield superior coverage- -eeraigierrriionnn trade-off compared to RoBERTa--QaantieRRe, For example, on WP, under similar coverage level, e.g., at around 77%, PaLD-PE produces tighter interval estimates than BaBERRT-QuantilRRR that for both methods, the targeted coverage level of 1 - a does not exactly match the achieved coverage level; but our method provides closer estimate, which allows the users to approximate the coverage before tuning a. We attribute this to our model of P(T(X))A) being an approximation of the underlying data generation process.
(0.33 v.s. 0.60). We note
Results on LLM Text Identification. Tab. 3 shows the LLM text identification performance. We see that PaLD- TI outperforms the segment-wise baselines by at least 18% in terms of segment-wise accuracy. We also implement the approximate version of PaLD-TI by solving Eq. 4 using the greedy algorithm in Alg. 1. As expected, it does not perform as well as solving Eq. 4 exactly, but still outperforms the other baselines by at least 9% in
Table 4: PaLD-TI top-p performance.
Dataset Top-0.05
Top-0.20
WP
0.743
0.876
Yelp
0.607
0.830
WP
Ydlp
Train: Yol, Test: WP
Thain: WPP Text Yelp
Goorngg
Covereen
0.5
It
P.ID
PLLE FE
0.6-
PulE-FE I
PulB FT
0.4
0.5
0.
85
0. Paecsion
03
02
85
6.6
87
0.4
05
Possein
Preciin
Proseim
(a) In-domain.
(b) Cross-domain.
Figure 7: Percentage estimation results, in terms of the coverage- precisson trade-off. Upper-left indicates better performance; a sets the desired coverage level.
8

---

Published as a conference paper at ICLR 2025
terms of segment wise accuracy. On single A10 GPU, the exact solver takes as 30s for 10-segment text on average, whereas greedy takes = 2.1s. On the other hand, PaLD-TI outperforms all other baselines by over 20% in terms of top-1 accuracy, and no other baseline is able to achieve over 7% top-1 accuracy. To achieve high Top-1 accuracy is a difficult task as it requires all segments to be successfully identified. Moreover, as Pal PaII can also rank the predictions, we also report the top-p accuracy, which measures how often the ground -oound-truth segments lie in the top p fraction of all (S), S S (1..... n), in Tab. 4. We see that even if the ground truth is not recovered by S. it is in the top 0.05 fraction of our objective function more than 60% of the time, and in the top 0.2 fraction over 83% of the time. This further supports the validity of fa (S) in determining the LLM segments.
4.2 CROSS-DOMANE MIXED-TEXT DETECTION
In this section, we evaluate PaL.D when the training/tuning domain and testing domain does not match. For the PE and TI tasks, we compare with the two best baselines in Tabs. 2, 3, respectively.
Fixed LLM. Here, the LLM used to generated mixed-texts is fixed as GPT-4o. For PaLD-PE, the likelihood fitting phase is trained on WP. and the point estimate predictive interval is esti- mated on the testing dataset of a dif- ferent domain; similarly, the baselines are trained on WP and evaluated on different testing domain. The out- of-domain datasets we evaluate on in- clude Yelp (Yelp, 2014) and RAID (Dugan et al.. 2024). We use the Ab-
Table 5: Cross-domait fixed-LLM results for percentage esti- mation (mean absolute error) and text identification (segment- wise accuracy). Training dataset:
WP: LLM: GPT-4o.
Percentage Estimation (MAE 4)
Text Identification (Seg-Ace, 1)
Test Dataset
PaL.D-
RoBERTa-
RoBERTa- QuantileRog
PaI.-TT
FastDetcct- GPT-Seg
RoBERTa-
PE
Reg
Seg
Yelp
0.147
0.211
0.231
0.801
0.626
0.592
Abstracts
0.151
0.306
0.223
0.718
0.693
0.656
Wiki
0.224
0.313
0.238
0.623
0.636
0.735
stracts and Wiki domains of RAID, and generate interleaved mixed-texts from GPT-4o (Sec. A. Tab. 5 shows that Pal.D ..-- remains the best-performing method compared to the baselines. This demonstrates that PaLD-PE has a superior ability to generalize across domains, and that the T-scores used do also generalize across domains. For the interval estimates, we illustrate the cross-domain performance between WP and Yelp in Fig. 7b. Similar to the point estimates, we again see that PaLD-PE yields superior coverage- pprccssionn tradeoff to RoBERTa- RoBERTh--uantileRe Moreover, the a values set for the respective algorithms to control the width of the predicted interval results in achieved coverage values that more closely align with a.
For TI, PaI.D-TI does not require training, but may use a T-score that is fitted to a dataset (e.g.. ROBERTA-LLN trained WP). The FastDetectGPPT baseline (applied segment- -egment-wise) has threshold set to maximize TPR- - FPR on WP, and RoBERTa (segment-wisc) is trained on WP. We see that PaLD-TI is superior to baselines on Yelp and Abstracts but not Wiki, where RoBERTa-Seg achieves high segment- wisse accuracy.
Cross-LLM. Here, we evaluate the case when additionally the LLM used to train/tune may not match the
LLM used to generate mixed-texts at test time. Table 6: Cross-domain cross-LLM results for percentage esti- mation (MAE) and text identification (segment-wise accuracy). Training dataset: WP (GPT-4o).
In particular, we evaluated PaI.D and baselines trained/fitted on WP (GPT-40-written) with the testing domains set as WP3, Yelp, and RAID (Abstracts, Wiki), where the testing domain mixed-texts are gen- erated in the interleaved manner fol- lowing Sec. A.I using Claude-3.5- Sonnet (Anthropic, 2024). In ad- dition, we evaluate RoFT (Dugan et al., 2020), which was designed to evaluate human-LLM boundary detection, so the mixed texts were
Percenlage Estimation (MAE J)
Text Identification (Seg-Acv.
Test
PaLD-
RoBERTs-
RoBERTa- QugTileRegTI
PaLD-
FastDetect- RoBERIa- GPT
Dalaset
PE
Reg
Sey
Claude
WP
0.114
0.184
0.138
0.751
0.531
0.675
Yolp
0.128
0.193
0.144
0.731
0.628
0.654
Abstracts
0.115
0.204
0.145
0.710
0.633
0.651
Wiki
0.174
0,245
0.185
0.602
0.634
0.683
GPT-2
SS
0.227
0.236
0.227
0.555
0.610
0.525
Recipes
0.166
0.209
0.186
0.548
0.520
0.505
NYT
0.236
0.246
0.239
0.477
0.613
0.544
generated by prompting GPT2-XL to complete 10- sentences when prompted with i human- -rritte sentences; we use the Short Stories (SS), Recipes, and New York Times (NYT) domains.
IIn Tab. all rows are cross-LI.M and cross-domain, except for WP which only cross-I.I.M

---

Published as a conference paper at ICLR 2025
Shown in Tab. 6, Claude, PaLD-PE outperforms all baselines despite the domain and LLM shift. PaLD-TI DID--Iip is superior on all domains except Wiki, where RoBERTa-See again achieves high accuracy. For GPT-2 (i.e., RoFT data), PaLD-PE performs similar or better. However, for the TI task, both PaI.DTII and RoBERTa-Seg result in a significant performance decrease compared to FastDetectGPT: this may be due to poor generalization to GPT-2 of the RoBERTa-basee classifiers that is used for both RoBERTa-Seg and the scoor for PaLD-TI. The better generalization to Claude from GPT-4o compared to GPT-2 may be expected, as the parameter count of GPT-2 is orders of magnitude lower compared to Claude and GPT.4o.
4.3 PARAPHRASING ATTACKS
In the binary classification setting, recent work has shown that LLM detectors such as Detect- GPT are easily attacked by paraphrasing attacks (Sadasivan et al., 2023; Krishna et al., 2024). Following (Sadasivan et al., 2023), we use the T5-based Parrot paraphraser (Damodaran, 2021) to randomly para- phrase the LLM-written segments of the mixed texts. We randomly para- phrase 25%, 50% and 100% of the LLM-written segments of the WP dataset, and evaluate the PE and TI tasks in Tab. 7. We observe that all methods suffer performance decreases with increasing attack strength. However, even when all LLM segments are attacked, PaLD still maintains superior performance compared to baselines. We leave further analysis of mixed-text detector attacks for future work.
Table 7: Percentage estimation (MAE) and text identification (segment-wise accuracy) under paraphrase attacks on LLM- written segments in WP.
Percentage Estimation (MAE J)
Text Identification (Seg-Ace, 1)
Fraction Chosen
PaLD-
RoBEKTa-
ReBEKT- QuantileReg
PaLD-TI
FastDetect- GPT-Seg
RoBEKTa-
Reg
Seg
0.25
0.117
0.204
0.193
0.811
0.633
0.701
0.50
0.122
0.204
0.193
0.786
0.624
0.691
1.00
0.134
0.207
0.195
0.730
0.584
0.679
4.4 ABLATION STUDY
We investigate several critical components of PaLD on the WP dataset, namely T-sscre. In Ap- pendix A.3, we further show how in PaLD-PE modelling de in Eq. 2 as KDE is superior to simpler assumptions such as Gaussian (Tab. 14), and the effect of choosing the Beta prior's parameters (Tab. 15) and the KDE bandwidth (Tab. 13). We also show the effect of baselines finetuned on segment-level text (Tab. 10, 11, 19, 20), the Fi-scores in text identification (Tab. 12), the effect of LLM segments (Tab. 16, 17).
segmentation mismatch (Tab. 18). and how PaLD performs on all-
T-score. In Sec. 3, we analyze the characteris- ties of different T T'scoores by o in Tab. 1, here we further evaluate them on both percentage es- timation and LLM text identification dataset. Shown in Tab. 8, we see that RoBERTa- LN yields the best performance for both per- centage estimation and LLM text identification. Moreover, the performance of both tasks correlates with the magnitude of the average quantile slope
Table 8: T-score ablation the WP dataset.
Percentage Estimation
Text Edletiffcationn
of
MAE
C1
P
Top-1 Ace Seg Ace
the WP
ROBERTa-LN
2.599
0.1
84%
0.385
0.252
0.826
RoBERTa
2.160
0.163
86%
0.526
0.205
0.798
Ghosthuster
1,889
0.151
86%
0.553
0.044
0.619
FusDDteccGPPT
1.784
DetectGPP
1.496
0.169
84%
0.547
0.101
0.690
of defined in Eq. 1. This lends support to our observation that a larger ox indicates increased separation of the --score distribution as varies with 8. This provides increased statistical power to predict 5 from text T-score, and yields improved precision of the predicted interval. Similarly, in text identification, it provides a (S) that is more strongly indicative of LLM text, yielding better segment-accuracy and top-1 accuracy with increased o7..
5
FINAL REMARKS
PaLD-TI is NP-hard and does not scale well with the number of segments. While we demonstrate the greedy algorithm as a low-complexity solution that can scale with many segments, better performance at low complexity may be desirable, which we leave for future work. Additionally, PaLD-TI requires fixed segmentation. A chosen segmentation may not align perfectly with ground-truth segmentation. Future work can investigate the effect of segment misalignment, extend to hallucination detection, and settings such humans mimicking LLM styles.
10

---

Published as a conference paper at ICLR 2025
Disclaimer. This paper prepared for informational purposes by the Global Technology Applied Research center of JPMorgan Chase & Co. This paper is not a product of the Research Department of JPMorgan Chase & Co. its affiliates. Neither JPMorgan Chase & Co. makes any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect the completeness, accuracy, or reliability of the information contained herein and the potential legal, compliance, tax, or accounting effects thereof. This document is not intended investment research investment advice, or as recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or be used in any way for evaluating the merits of participating in any transaction.
any of its affiliates
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333.08774 2023.
Anthropic. Claude 3.5, 2024. Available at https:t//// http///wwwt.........cc/ com/.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectGPT: Efficient zero-shot detection of machine-generater In The Twelfth International Conference on Learning Representations, 2024. URL https: //opereeview.oo www.ew.net... net./ www.eoooouuiuiio Bet.aannnnn.roouunn//--aerrrrrr
via conditional probability curvature.
Emily M. Bender, Timnit Gebru, Angelina SeMiillanMMajor and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 610-623, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 9781450383097. doi:
10.1145/3442188..56 3445922.
344459922.... URL https: /tt./iiiicc/// org. 100 11t.111414422100 34459222
Ming-Hui Chen and Qi-Man Shao. Monte carlo estimation of bayesian credible and hpd intervals. Journal of Computational and Graphical Statistics, 8(1):69-92 1999. ISSN 10618600. URL
http:// /wwww .estororrgl stable / 1390921.
Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Raj. Token prediction as implicit classification to identify L.LM-generaed text. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Lan- guage Processing, pp. 131 12-13120. Singapore, December 2023. Association for Computational URL ttattps://ccaaatholloe org/
Linguistics. doi: 10. 18653/6 10/553//1122233 1/2023. wwwwmmmminnnneemeeeeeemmmmmmmmae 2023. emnlp-main 8.0.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. All that's s 'human' is not gold: Evaluating human evaluation of generated text. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/202 lol- URL https: / a anlllanthhllllgooo org/2021 aac- 555.
Online, August
Shay Cohen. Bayesian Analysis in Natural Language Processing, Springer International Publishing, 2019. ISBN 9783031021701. doi: 10. 10077978-3-031--000 10. 10001007//00 9117/978-222 001-00111----1-11111111111111111
URL http::////// :/tp:///xxxx/// doi org/
Debby RE Cotton, Peter A Cotton, and Reuben Shipway. Chatting and cheating: Ensuring academic integrity in the era of chatgpt. Innovations in education and teaching international, 61(2):228- 229, 2024.
11

---

Published as a conference paper at ICLR 2025
Nikola Čréek and Jakob Patekar. Writing with ai: University students' use of chatgpt. Journal of Language and Education, 94 (36)):128-12) 2023.
Joseph Cutler, Liam Dugan, Shreya Havaldar, and Adam Stein. Automatic detection of hybrid human-maching text boundaries, 2021.
Prithiviraj Damodaran. Parrot: Paraphrase generation for nlu. Parrot: Paraphrase generation for nlu, 2021.
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch. RoFT: A tool for evaluating human detection of machine-generaln text. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 189-196, Online, October 2020. Association for Computational Linguistics. doi: http://////2000mmmmmmeeeee. 2020. e-n.e-demsss.....oooomcmmmmmmmlm demas....... 25.
URL https: //t///aclaaaa /ttp////aaaahaaoceeooeo anthhlogh org/
Liam Dugan, Alyssa Hwang, Filip Trhlik, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. RAID: A shared benchmark for robust evaluation of machine- generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12463- 12492, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https:: / /accl anthologyy org/2024 aclllong............ long. 674.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1082. URL https: //acl Taclantholl woy.logyoooooo wtwg///1-1/000
Melbourne, Australia, July
Mirko Farina and Andrea Lavazza. in Artificial Intelligence, 6, 2023. ISSN 2624-8212. doi: 2023
Chatgpt in society: emerging issues. Frontiers
URL httpt:ttpp....m /www fwontiersssiie
http///onnaassaaiiiiiooooooooaee itt.://rrrrrfcciillaaaaeeeeeeeee articles/10 3389/33389..... 2023. 1130913.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds Mach., 30(4):681-69) dec 2020. ISSN 0924-6495. doi: 10. 10.1077/11102--22--055553 URL https: //w.i//doi///// /trrgg10000 1007 (0007//111200 1007//100220020005554444
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visual- ization of generated text. In Marta R. Costa-jussà and Enrique Alfonseca (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstra- tions, pp. 11 111-16, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v1/119-33 3019. URL https: / /aalaaaannttt /acllaaanhologoooo org/p 1919999..
Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2nd ed. edition, 2004.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt human experts? comparison corpus, evaluation, and detection. 2023.
arXiv preprint arXiv:2301.072
Daphne Ippolito. Daniel Duckworth. Chris Callison- and Douglas Eck. Automatic detection of generated text is casiest when humans are fooled. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808- 1822, Online, July 2020. Association for Computational Linguistics. doi: (118886/1122200.eae main. 164. URL https: http://aaaaathhhocooooo 164.
/tgg///0.... malmminn..
Cameron Jones and Benjamin Bergen. Does GPT-4 pass the Turing test? arXiv preprint arXiv:2310.2000 20216.. 2023.
Nurul Shamimi Kamaruddin. Amirudin Kamsin, Lip Yee Por, and Hameedur Rahman. A review of text watermarking: Theory. methods, and applications. IEEE Access, 6:801 1-8028, 2018. doi: 10.1 10.1199CCEESS2218...0000
12

---

Published as a conference paper at ICLR 2025
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. A watermark for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17061-17084, PMIR, 23-29 Jul 2023. URL https: /ptp:s/e/rc/eeeeeee ngs. ngs mlr. press/v202 http2//2//kkcccbbaaree23222
Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica, 46(1):33-50, 1978. ISSN 00129682, 14680262. URL http: //wp:://wwwwww/ www.::jjttrrrrggooo.
/tt///913644 913643.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
Laida Kushnareva, Tatiana Gaintseva, Dmitry Abulkhanov, Kristian Kuznetsov, German Magai, Eduard Tulchinskii, Serguei Barannikov, Sergey Nikolenko, and Irina Piontkovskaya, Boundary detection in mixed Al-human texts. In First Conference Language Modeling, 2024. URL https: //tppsss///peeevieew net/ forum? wwiiiiddtrtttreee
Sarah Levine, Sarah W Beck, Chris Mah, Lena Phalen, and Jaylen Pluman. How do students use chatgpt writing support? Journal of Adolescent & Adult Literacy, 2024.
Yingji Li, Mengnan Du, Rui Song. Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308. 10149, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: arwia.iiiiv...o org/ htwg/bb/////aaoogge//gg////eaaea 1907. 11692.
Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative Al detection via rewriting. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ https///ppereevee appnrevieweee net/ / forum? borrummiid-oggeuugtt..
Qiaozhu Mei, Yutong Xie, Walter Yuan, and Matthew O. Jackson. A turing test of whether ai chatbots are behaviorally similar to humans. Proceedingy of the National Academy of Sciences, 121(9): c2313925121, 2024. doi: 23..3//22 URL https: /www www..ppa..aa........m........... org/ doi/ 2tw.:001077/paa.2211112
Niloofar Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Bong-Kirkpatrick. Smaller language models are better zero-shot machine-generaaed text detectors. In Yvette Gra- ham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chap- ter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 278-293, St. Julian's, s. Malta, March 2024. Association for Computational Linguistics. URL https: 25.
//aclande anthoooooyyy org/ 2024.
Eric Mitchell, Yoonho Lee. Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. machine-generatedd
Detecigpt: Zero-shot machine generated text detection using probability curvature. In International Conference on Machine Learning, pp. 14950-24962. PMLR, 2023.
Melanie Mitchell and David C. Krakauer. The debate over understanding in ai's large language models. Proceedings of the National Academy of Sciences, 120(13):c221159077220 2023, doi: 2t00733/paa.2159771111 URL https: www. www.ppaass....m org/ 2215907120.
10t.//aa........... ppas.
Oscar Hernan Madrid Padilla, Wesley Tansey, and Yanzhen Chen. Quantile regression with relu networks: Estimators and minimax rates. Journal of Machine Learning Research, 23(247):
2022. URL http: /mrr..mmrr//// wttpr..rg//ppaa happess/v3//222
html.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.111566 2023.
13

---

Published as a conference paper at ICLR 2025
D.W. Scott. Multivariate Density Estimation: Theory, Practice, and Visualization. A Wiley- interscience publication. Wiley, 1992. ISBN 9780471547709. SKLhttps://booo google.
com//books2id=7crUUS F2ccC.
Narges Sharif-Razavian and Andreas Zollmann. An overview of nonparametric bayesian models and applications to natural language processing, 2008. URL http: httt::/www..co Cs. cmu edu/ .dff.
-zol http:///pwwaaaa.aooooooooaaaeaae
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training generated data makes models forget. arXiv preprint arXiv:2305. / 7743,, 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained recursively generated data. Nature, 631(8022):555-55) 2024.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger. Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: 09203, 2019.
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwritten by large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 1702 1702-1717, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. (0.18653/v122244 naacl-long.95 95. URL https: /aapc//acllaaaiia aacllanooooooyyy..o.........ooot org/ 202 http:/2222maaaa1aa20mm www.n-aacllnnn... 95.
Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. SeqXGPT: Sentence-level Al-generated text detection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1144-1156, Singapore, December 2023. Association for Computational Linguistics. URL wrgg/0223300 wwnll-mnlle.mmme ma 73.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference machine learning, pp. 23631-23644. PMLR, 2022.
Yelp. Yelp review dataset, 2014. URL
Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gašević, and Guanliang Chen. Towards auto- matic boundary detection for human-ai hybrid essay education. arXiv preprint arXiv:2307. 12267, 2023.
Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. Provable robust water- marking for Al-generated text. In The Twelfth International Conference on Learning Representa- fions, 2024. URI. reypereer///e/eeeeeeeeeeeeeeeeee eew net/ forum? a
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MIXED TEXT GENERATION
We use
approach to generate synthetic mixed texts. This process is used for both generating the histograms in Fig. 3 and fitting the mixture KDE model in Sec. 3. Specifically, if we denote zd as a text from our auman-written dataset (e.g. WritingPrompts or Yelp Reviews), then ... is segmented at the sentence level. Then, segments are randomly masked, to form
x = =f masked text "MM For example, a 5-segment human text " = x 22 23 l x!! masked at sentences 25 2 and 5, would become x= M x [MASK] 2 24 [MASK]. The LLM is then prompted with the following prepended to x M
"What sentences should go in the Peeplace [MASK] locations of the following text? Only provide exactly one sentence per [MASK] location. Only provide the sentences as numbered 1ist with Preplace sentences total."
Here, Preplace is the number of [MASK] symbols in M We found that when prompted in this way. GPT-4o successfully returns numbered list containing exactly Preplace sentences corresponding to the [MASK] symbols, with a failure rate of 1-2%. When successful, the sentences in the numbered list are inserted into the [MASK] positions of the masked text, to form the mixed text. We show few examples of these mixed texts in Tab. 21.
Since this process does not control the length of the segments returned by GPT-4o, it is not easy to exactly control the fraction of LLM text d at the character level. As a result, we target å approximately by masking and filling a fraction of the human sentences. This done for both the training split (targeting 5 approximately at 0.1, 0.2..... 0.9) and testing splits (targeting 5 approximately at 0.25, 0.5, 0.75). The true (at the character level), which can be computed post-hoc, is used for (i) the binning step in PaLD-PE collect the T T-ccore samples for the P(T P(T()) A 8x) distributions, and (ii) to benchmark the performance metrics (mean absolute error, coverage, and prediction) for percentage estimation.
We apply this method using both GPT-4o and Claude-3.. 3.5 3.5-Sonnet (Anthropic. 2024).
A note on the training and testing split. Note that the target fractions for the testing split are 0.25, 0.5, and 0.75, but as described above, these are not actually the frue LLM text percentages measured the character-level, but rather the target fractions, which determine the number of masked sentences for the mask-and-fin procedure. Approximately 0.25, 0.5, and 0.75 of the sentences are masked out, then the true L.MM fraction values are computed post-hoe. The true fractions depend on the text content and thus vary. We found that this procedure yielded similar distributions of 5 over [0, 1] compared to setting the target fractions to 0.1, 0.2, 0.9 as done for the training split. To provide concrete evidence for this, we compute a histogram of the d values on the Yelp dataset for both train and test splits in Tab. 9. Thus, despite the "target" fractions differing between the train and test split, Table 9: Histogram of 8 values in [0, 1] for train and test splits of Yelp.
Split [0,0.1]
[0.1,0.2]
[0.2,0.3]
[0.3,0.4]
[0.4,0.5]
[0.5,0.6]
[0.6,0.7]
[0.7,0.8]
[0.8,0.9]
[0.9,1.0]
Train
0.062
0.155
0.140
0.132
0.0972
0.0988
0.0957
0.106
0.0819
0.0315
Test 0.030
0.1367
0.170
0.1630
0.130
0.090
0.107
0.110
0.05330
0.01
the true fractions mostly match.
A.2 PALD IMPLEMENTATION DETAILS
During the distribution fitting stage of PaLD-PE, the entire training split of a dataset is masked and filled using the above procedure, where the fraction of sentences masked is approximately 6, 0.1, 52 = 0.2. ... dg 0.9. We then compute Tcccres across all the text samples using the logits of a RoBERTa model trained with the LogitNorm loss Wei et al. (2022). We found this T-scrre to work the best for percentage estimation, as the LogitNorm improves calibration of the model which is necessary for the T-ccoe to smoothly interpolate between ailly-human and fully-LLM. We use LogitNorm with temperature 0.005, and train the RoBERTa model on the training split for
15

---

Published as a conference paper at ICLR 2025
Table 10:
Percentage estimation (MAE);
Table 11: LLM text identification (segment- wise accuracy); RoBERTa classifiers trained on segment-level data.
RoBERTa classifiers trained on segment-level data.
Dataset
RoBERTa-Seg
RoBERTa-I.N-Seg
Dataset RoBERTa-Seg RoBERTa-LLN-Ser
WP
0.566
0,434
WP
0.502 0.498
Yelp
0.563
0.563
Yelp
0.496
0.496
Table 13: Effect of KDE bandwidth.
Table 12: LLM text identification (segment- wise -soore).
Bandwidth h
PaLD-PE (MAE)
Dataset
PaLD-TI
FastDetectGPT-SSg
0.01
0.129
0.1
0.118
WP
0.798
0.480
I
0.124
Yelp
0.776
0.476
0.312 + 0.025 (Scott)
0.116
the respective datasets. These T-scores and then binned to 8 = 0.125, 0.25, 0.375, 0.875, using the character-level fraction of LLM text in the mixed text. Then, a KDE with Gaussian kernel, with bandwidth chosen using Scott's ss rule Scott (1992), is fit to the T-scores at each level of 5g to form the likelihood model P(T]8) [8) as described in the main text. For the posterior, we choose the prior P(6) to be the Beta(2 2, 2) distribution. During the inference stage, first 1000 due to burn-in, using Metropolis- Metropolis-Hastingg (Gelman et al., 2004) with a proposal distribution as the truncated normal centered the previous sample, truncated to [0, 1J. The predicted percentage and interval are given by the MAP estimate and (1 a) HHD interval, respectively.
sample 5000 samples, discarding the
A.3 ADDITIONAL RESULTS AND ABLATIONS
Baseline RoBERTa classifiers applied to segments. The baseline RoBERTa classifiers applied segment-wise, for both the PE and TI tasks, were trained on paragraph-level fully human or fully- LLM texts. As this may incur distribution shift when applied segments, we train the RoBERTa classifiers to sentence-lengee data gathered from the mixed-text training splits (Sec. A.1).
Similar Sec. 4, for PE, these baselines return the ratio of predicted LLM characters to total characters, and for TI, classify each segment individually. Shown in Tabs. 10, 11, the performance is significantly worse than the RoBERTa classifiers trained on paragraph-Iewe texts, reported Tab. and Tab. 3 for PE and TI. This is likely due to the texts being too short, as RoBERTa is pretrained on longer texts, and short texts do not offer enough context, making difficult to extract statistical dependence with its origin.
Fi-score of LLM text identification. To measure the impacts of both the false-negative rate (FNR) and false-positive rate (FPR) of the LLM text identification task, we further report the segment-wise Fl-score for the text identification task in Tab. 12. We also report the Fl-score of FastDetectGPT applied segment-wise, which was the strongest baseline in terms of accuracy Tab. 3. As shown, PaLD-TI significantly outperforms it in terms of Fl-score as well, demonstrating good balance between precision and recall compared to baselines. In summary, the PaLD-TI outperforms FastDetectGPT not only in its accuracy (Tab. 3), but also in the Fi-score (Tab. 12).
Distributional assumptions for PaLD-PE. For PaLD-PE, we choose mixture KDE for the likelihood and a Beta distribution for the prior. Here, we analyze how the choice of how we model the likelihood affects the performance of PaLD-PE. We compare the mixture KDE with a mixture Gaussian and mixture Cauchy.
Table 14: Likelihood model ablation the WP dataset.
Percentage Estimation MAE
(x(1)
C
P
Namely, we fit (t) in Eq. 2, to the samples (((t)) fim1 using Gaus- sian and Cauchy distributions with RoBERTa-LN --scoes. Shown in Tab. 14, using KDE performs significantly better, demonstrating that the higher expressivity of the distribution has a large influence on percentage estimation performance. We also varied the a, b parameters of the Beta(a, b) prior
KDE Normal
0.116
84%
0.385
0.200
85%
0.630
Cauchy 0.227
85%
0.710
16

---

Published as a conference paper at ICLR 2025
Table 15: Beta(a, b) prior ablation on the WP dataset with KDE likelihood model.
Percentage Estimation
a, b
MAE C1
P4
1, 1
0.155
84%
0.450
2, 2
0.116
84%
0.385
3, 3
0.122
84%
0.395
Table 16: Percentage estimation on all or no LLM segments; mean absolute error.
Table 17: Text identification on all or no LLM segments; segment accuracy.
Dataset
PaLD-PE
FastDetectGPT
Dataset PaLD-TI
FastDetectGPT
No LLM
0.137
0.261
No LIM 0.900
0.829
All LLM
0.220
0.366
All LLM 0.728
0.718
between 1 S b < 3 in Tab. 15, and a = b = 2 perform best, but this may be dependent on the distribution encountered test time.
Ablation study on KDE bandwidth selection. Below, we report the effect of the bandwidth (h) on PaLD-PE's performance on the WP dataset. As mentioned in the appendix, we use Scott's rule to automatically set the bandwidth for results the main text. In Tab. 13, we additionally report the average bandwidths chosen by Scott's ' rule over all the dy 's in Eq. 2, with one standard deviation. As shown, PaLD-PE is not too sensitive to the choice of bandwidth, and Scott's rule able to determine bandwidth resulting in accurate estimates of the LLM fraction.
All or No LLM Segments As the setting of this paper aims for mixed-text, did not include the No LLM and the All LLM cases, as these two cases reduce to the classic setting of existing LLM text detection frameworks such as (Fast)DetectGPT and Ghostbuster. On the WP dataset, we evaluate the PaL.D framework on the No LLM and the All LLM (GPT-4o) cases (Tabs. 16, 17) and compare with the FastDetectGPT, which outperforms DetectGPT and Ghostbuster in Tab 2. Note that Fest is initially designed for the binary classification between No LLM and the All L.M.
The results further suggest that even on the classic setting, PaI.D still provides competitive estimations. Note that PaLD-TI cannot pick all segments to be LLM not LLM, and therefore only considers 2" - 2 of the 2" possibilities. Thus, it will always make at least one mistake. This means the 0.900 segment accuracy for No LLM segments the best it can achieve, since there are 10 segments per text in this dataset, and PaLD-TI will misclassify at least one segment incorrectly. Additionally, would like to reiterate our response to Q3 that the testing datasets contain texts that are nearly all L.M LLM segments.
Effect of segmentation mismatch in PaLD-TI. PaLD-TI, as well the segment-wise baselines. assumes chosen segmentation, which we assume to be sentences. Here, we investigate the effect of mismatched segmentation.
other words, the case when the ground-truth segmentation (i.e., the actual segments where the text is fully LLM or fully human) does not perfectly align with the chosen segmentation of PaLD-TI. To do so, we use the same sentence-level mixed text, so the ground-trute segmentation is at the sentence-level. However, the chosen segmentation is now at every 2 sentences. This means that PaI.D-TI (and segment baselines) can only assign prediction to pairs of sentences, and thus misclassification (at the sentence-level) will occur whenever the two sentence within a pair differ class (i.e., one is human and the other is LLM), no matter what the prediction is. We show how PaLD-TI and the FastDetectGPT-SSer perform in Tab 18.
As shown, there a performance decrease in both PaLD-TI and FustDetectGPTT-Seg as expected, since perfect classification is no longer possible and mistakes inevitable. We note that PaLD-TI still outperforms FastDetectGPT-SS. at the 2-sentence segmentation, and leave methods to mitigate mismatched segmentation for future work.
17

---

Published as a conference paper at ICLR 2025
Table 18: Text identification under mismatched segmentation. Dataset: WP. Ground-trut truth segmenta- tion is at the sentence-level. Report sentence-wis wise accuracy.
Chosen Segmentation
PaLD-TI FastDetectGPT-Seg
Every Sentence
0.826
0.651
Every 2 Sentences
0.706
0.613
Table 19: Percentage estimation (MAE !)
Detec1GPT (pura.
DetectGPT (senl.
FastDeteccGPPT (para.)
FastDetectGPP (sent.)
PaLD (ours)
WP
0,381
0,370
0,212
0.184
0.116
Yelp
0.407
0,397
0.218
0.190
0.137
Table 20: Text identification (segment accuracy 1)
Dataset
DetectGPT (pura.)
DetectGiPT (sent.
FastDeteciGPT (para.) FastDetecaGPT (senL)
PaLD-TI (ours)
WP
0.486
0,488
0,650
0.651
0.826
Yelp
0.452
0.460
0.622
0.624
0.801
Baseline DetectGPT and FaTt methods applied to segments. We compare the Detect- GPT and PastDetectGPT segment-wise baselines by setting the threshold maximize the difference between TPR and FPR on segment-level validation set, versus on a paragrpah- show the difference by setting threshold at the paragraph level vs. sentence level in Tab. 19, 20. There is an improvement in the performance by using sentence-level validation data, albeit around <, 0.03 MAE for the percentage estimation task, and 1% for the segment-accuracy the text identification task.
dataset. We
18

---

Published as a conference paper at ICLR 2025
Table 21: Examples of synthetic mixed text generation using the mask-and-fill fill approach with GPT-4o. Text highlighted in red were originally segments masked in the human text and filled by GPT-4o.
Human Text
Mixed Text
Thank you for a lovely morning: I was in NJ early and decided atop in for delicious diner breakfast. I got a taylor hea, egg, and cheese and a short stack blueberry pancakes (I simply couldn't decide betwean sweet or sevory, plus leftovers are never a fantastia. was nervous wher they said they didn't have kaiser rolls so went
This morning, decided to treat myself to breakfast as local diner. got taylor ham, eyg, and cheese and a short stack of blueberry pancakes (1 simply couldn't t decide betwean swaet or savery, plus leftovers are never a bad thing). The egg sendwich was fostastic, was nervous wher they said they didn't have kaise" rolla so T vent with a hamburger roll, It was excellent decision. The pancekes were flufly and bursting with blueberries. They layered the cheese on the sandwich, which to ma is a must for a true egg sandwich. The meal also came with a side of breakfast potatoes.
bad thing). The egg sandwich
hamburger roll. It was an excellent decision. The roll wes excellent net just an average cheap hambarger rell. They layered the cheese on the sanéwich, which to me is a must for a egg ssndwich. It served with home fries, which were sauteed with large slices of peppera and onioas.
This was a very busy placs. Was told I had to try this plane while waa not dissppointed. that was amazing, The rest of my group tried number of different options that they all enjoyed. The wai: was listle long espacially or a very hot date. The pricas were very reasoneble. The crdering at the windaw was confusing. Multiple windows with not a lot of direction of which Line to get in and wheze greaz place to stop if you are in the area.
recently visited a local ice crean shop, The place had charming, old-fashioned vibe. was not dispppointed that was amazing. The staff was friendly and helpful. The menu had s wide variety of flavors and treats. The prices were very reasonable. The ordaring at the window was confusing. Multiple windows with not a let of direction of which lire get in and where to walt for the food. It's a great place to stop if you are in the ares.
in Louis. got peach shake
T got a peach shake
wait for food. It's
is banished years, Write his diary entries for his first was boin to fire.
the wildernens for 20
is banishad the wildernass for 20 years. Wrice his diary entries for his first and last days of exile. Today marks the beçinning of my exile, punishment I mast andure for the next two decades. The pain of separasion from my loved oneg is urbesrable, but 1 must find the strength to survive. within the ruined navity of my left eye held the final Images of my family as they were fed the same fires I was pulled from. by deach would not be so quick and so 1 was sllowad to burn with them, but live. As soon aa was able to walk, was vahered out into the wilderness. The years have been long and arduova, but 7 have learred to find solace in the solliude of the wilderness. As I take my final steps back to civilization, carry with ma the sears and wisdom of my exile
and last daye of exile. It flowed over my skin, danced upen my face, and stripped me what little humanity left, Withir the ruined cavity of my left eye 1 held the final imagas of my family as they were fed to the game fires from. My deata would not be so quick and so was allowed to burn with then, but live. As soon as i was able to walk, was ushered out into the wildernass. The final pieca of society I was allowed to keep was in the ink buried in my chest that had once formed my twisted with my burned skin into a miashapen claw. They promised twenty years, but swore under their breath
pul led
son'a hand print,
Desoribe object within five fest of you in as much detail as posaible. A pait of simple black converse lie on floor the baby blue sonda fit my girlfriend is kind enough to let me drive. They are a for cry from the erisp kieks I'd received in the mail only year ago. This has been a hard 12 montha for them. The
Describe objact with'r five fast of you in as much detail as possible. A pair of simple black converse lie on the floor of the baby blue sanda fit girlfriend is kind anough to let ma drive. They ara far cry from the crisp kicks I'd received in the mail only a year ago. The laces are frayed and stained,
criap white inner lining has degraded into something a generous pesson might call "cream" or "off-white" to me they's ze just brown. The forces of time have transmuted the laces into soft gray, 11 clouds in Iall which promiee gentle patter of rain to listen to as you while away the had partioularly bad, time and constant use has worn down the bottom edges. Scuff marks cover pristine expanse. Wher, they ware naw I'd taken, so much care to keep them scuff free I waited until the wedding to wear then.
longer the bright unite they were, The once crisp white innez lining has degraded inso something a generous person might call "cream" or "off-white" to me they' re juss brow. The ruaber soles are worn down, evidence of count steps taken, The black canvas is faded, showing signs of daily use. Scuft marka cover the once pristiae expanse. When they were new i'd taken, so much care to keep them scuff free I waited until the sedding to waar them.
houra, The rubber
wear
once
19


Reviews:
[{"review_id": "CqSxQPzybq", "reviewer": "Reviewer_DUtY", "summary": "The authors introduce PALD-PE and PALD-TI, methods that estimate the percentage of LLM-generated content in a piece of text, and an approach to identifying the LLM-generated segments, respectively. PALD-PE provides MAP estimates of the percentage of LLM-generated text, and using gaussian kernel density estimation, they provide intervals that cover the ground-truth percentage. PALD-TI runs a combinatorial search over all possible segments using a score from a text-classifier to identify the LLM-generated segments. Furthermore, a greedy version of the PALD-TI approach is introduced. The authors evaluate their approach on the Writing Prompts and Yelp Dataset.", "questions": "•\tLine 191, wrong notation? X_i instead of X?\n\n•\tTypo 297: performnace -> performance\n\n•\tWhat was the reason for having different fractions set for the testing split than for the training split? \n\n•\tAny insights as to why the quantile slope of DetectGPT is smaller, but the segment accuracy is stronger than that of GhostBuster and FastDetectGPT?\n\n•\tWere PALD-TI and PALD-PE evaluated on cases where there were no machine segments? How about in cases where there are only machine segments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tIt is one of the first papers to address the more realistic mixed-text scenario and proposes to estimate the amount of text that is machine-generated, as well as to identify the segments that are machine-generated. This is a departure from the usual machine-or-human detection. \n\n•\tThe paper includes rigorous ablations of the PALD-PE method, which greatly led credence to the observations that methods with a higher quantile slope were more useful in mix-text scenarios.", "weaknesses": "o\tThe fine-tuned RoBERTa baselines were trained to identify between fully human and fully-LLM texts that are larger than a single sentence in length, but then tested on individual sentences. This introduces a distribution mismatch during testing that might make the baselines weaker than they really should be. \n\no\tFor DetectGPT, and FastDetectGPT, the threshold should’ve been derived from a validation set to maximize their performance at the segment level identification task.  \n\no\tIn Table 3, PALD TI relies on a supervised model, whereas DetectGPT and FastDetectGPT do not so it’s unfair to compare one against the other. It would be good to have a zero-shot version of PALD. It would’ve been interesting to test them against a version of PALD-TI which relies upon an unsupervised T-score. \n\no\tThe datasets and testing conditions considered are not sufficient. In particular, the work could’ve also been easily evaluated on the RoFT dataset (https://arxiv.org/abs/2212.12672), and it could’ve also been built off of RAID (https://arxiv.org/abs/2405.07940) and the M4 dataset (https://github.com/mbzuai-nlp/M4), both of which include more realistic testing domains. \n\no\tOnly GPT-4o was used when infilling. More LLMs should’ve been considered, including non-instruction-tuned LLMs.\n\n•\tThe PALD-TI approach relies heavily on a T-Score from literature such as a trained RoBERTa model, FastDetectGPT, DetectGPT, etc. It uses them to perform a combinatorial search of 2^n – 2. A more efficient method for search that is not just greedy would make the method much stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YmMzqmXfdi", "reviewer": "Reviewer_3hmG", "summary": "The paper introduces Partial-LLM Detector (PaLD), a method designed to estimate what percentage of a text is generated by an LLM and identify which specific segments are machine-generated. This approach uses black-box methods leveraging scores from text classifiers to achieve these tasks. It offers a more nuanced detection capability compared to previous methods that only classify texts as either human or machine-generated in entirety.", "questions": "1. Line 93, could you please define the term 'robust' in a human-written-/LLM-text detection problem (e.g., what types of changes, uncertainties, and unexpected perturbations in input data are considered)?\n2. Lines 90-100, Bayesian framework, MAP estimation, mixture Gaussian models, and greedy policy algorithm have already been widely used in text detection or related NLP problems, please provide some references to them.\n3. Line 112, '..., and x \\in \\chi ...' \\chi needs to be defined before the first time use it (e.g., is it a set or list).\n4. Line 154, how do you determine the number of decomposed segments in real applications? Is it equal to the number of sentences in a paragraph?\n5. Lines 190-193, are \\eta and \\delta different from each segment Xi? Or they are calculated over the entire mixed-text paragraph?\n6. Lines 205-206, is \\epsilon a pre-defined hyperparameter? Since it determines the range of p, will the optimal \\epsilon depend on the distributions P_llm and P_human, and how to choose the optimal \\epsilon?\n7. The example in Fig. 6 only consists of 3 sentences. If the text has more sentences, will the T-score of all the combinations be computed? Or just 2-sentence combinations will be considered?\n8. Lines 324-326, a greedy algorithm is employed to approximately solve the problem and reduce complexity. Could you briefly discuss the approximation errors introduced by this algorithm and whether or not they impact detection accuracy?\n9. In section 4.2, the cross-domain detection experiment is particularly intriguing, and I can somewhat understand that this test is generally more challenging compared to in-domain detection. Could you please briefly describe whether the WP and Yelp data are correlated (or consist of some similar information) before we can safely reach the conclusion?\n10. The experiments are conducted on only two specific datasets, so additional validation is required to assess generalizability. I am also curious about the performance of detection in more challenging cases, such as humans mimicking the LLM style or LLM imitating highly stylized human writing. These can be mentioned in future works or current limitations.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This method allows for a more detailed understanding of the text composition by identifying LLM-generated segments.\n2. The proposed method can handle texts of mixed origin, reflecting more realistic scenarios than those handled by traditional binary classifiers.\n3. Provides statistically robust methods to back its detections, including confidence intervals and likelihood measures.", "weaknesses": "1. Some of the concepts and mathematical notations need further explanation for readers' better understanding.\n2. The experimental setup needs improvement. Currently, experiments are conducted on only two specific datasets, so additional validation is required to assess generalizability. Moreover, the proposed method appears to rely on prior knowledge about distributional information, which could further limit its generalization ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnDCjKg2yu", "reviewer": "Reviewer_1FyB", "summary": "This work challenges traditional binary views of machine-generated text detection for classifying entire text samples as either human or LLM-written, and instead studies detecting texts partially written by LLMs. The experiments consist of two main parts: a) estimating the percentage of LLM-written texts; b) identifying LLM-written segments.", "questions": "No further questions. Overall, I like the paper and would like to give positive ratings. Although there are imperfections, I think this work is valuable, memorable, and should be encouraged.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The problem raised by this paper is very interesting and worth studying. This makes this paper have great academic impacts.\n2. This paper overall is well-written. The presentation and structure are clear and concise, with well-crafted figures and only a few typos.\n3. In-depth theoretical formulation and justification of the statistical method chosen.", "weaknesses": "1. Computational complexity: As the authors have acknowledged, scaling this method over many segments could be challenging. This poses limits to its applicability. In my opinion, this is a relatively minor issue as this partial detection problem is inherently harder, and we should encourage solutions to hard problems. This should be left for future works to work on.\n\n2. Need more robustness analysis: My main criticism is that this method lacks more sensitivity analysis regarding to segment lengths and various forms of perturbation attacks. How would it perform when there are adversarial attacks trying to undermine the detection is another main problem we should consider. I understand the paper has limited scope and cannot study everything, but I would assign a more affirmative rating had the paper dedicated a section for robustness analysis.\n\n3. This method also requires training with regards to the specialized domain, which makes it less generalizable. Although the authors have studied cross-domain performance, the results are less convincing as only two domains have been studied, and maybe domains with more distinct writing styles have more drastic performance difference. I point it out not because I want to ask the authors to do additional experiments on new domains (although you are welcomed to!), but it’s just that I think it is that the nature of the methods makes training necessary, and this weakness cannot be easily avoided.\n\n4. A minor point: I would suggest to have the related work section review every existing work studying partially LLM-written text detection (I’ve searched for some and did not find too many), and reduce the part on watermarking methods as they are less relevant to this paper. A literature review in this understudied area would be very helpful!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qTyj4gEYL9", "reviewer": "Reviewer_qnTJ", "summary": "This paper explores methods for detecting human and LLM-generated text in a mixed-text environment. A detector called PaLD is proposed, which is used for estimating the proportion of LLM-generated content in a text and determining which passages are written by LLMs.", "questions": "1. The paper used quantitative T-score and KDE to analyze or differentiate text. It is suggested that the authors provide more detailed basis for parameter setting, such as the choice of bandwidth for KDE and how to determine the threshold of T-score.\n\n2.  Authors provide performance metrics of the algorithm such as accuracy and coverage-accuracy trade-offs, but lacks the analysis of misclassifications (false positives and false negatives). It is recommended to explore or discuss in detail the possible causes of misclassification.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles the mixed text problem through the PaLD method, which is an innovation in current LLM detection techniques. Above all, this paper has a complete structure and provides relatively comprehensive experiments.", "weaknesses": "1. The effectiveness of this method seems contingent upon accurate segmentation of text into LLM and human-written parts. It would be helpful to explore how it performs under less ideal segmentation conditions or how it handles errors in segmentation. \n\n2. The article mentions the performance of cross-domain detection, but just two types are involved here, which seems to be insufficient to validate the generalization ability of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)