Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection
Authors: Guangsheng Bao
Yanbin Zhao
Juncai He
Yue Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Machine-Generated Text Detection
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Extend existing white-box methods to blackbox models, achieving 51% improvement relative to the remaining space

Abstract: Advanced large language models (LLMs) can generate text almost indistinguishable from human-written text, highlighting the importance of LLM-generated text detection. However, current zero-shot techniques face challenges as white-box methods are restricted to use weaker open-source LLMs, and black-box methods are limited by partial observation from stronger proprietary LLMs. It seems impossible to enable white-box methods to use proprietary models because API-level access to the models neither provides full predictive distributions nor inner embeddings. To traverse the divide, we propose **Glimpse**, a probability distribution estimation approach, predicting the full distributions from partial observations. Despite the simplicity of Glimpse, we successfully extend white-box methods like Entropy, Rank, Log-Rank, and Fast-DetectGPT to latest proprietary models. Experiments show that Glimpse with Fast-DetectGPT and GPT-3.5 achieves an average AUROC of about 0.95 in five latest source models, improving the score by 51\% relative to the remaining space of the open source baseline. It demonstrates that the latest LLMs can effectively detect their own outputs, suggesting that advanced LLMs may be the best shield against themselves. We release our code and data at https://github.com/baoguangsheng/glimpse.

Published as a conference paper at ICLR 2025
GLIMPSE: ENABLING WHITE-BOX METHODS TO USE PROPRIETARY MODELS FOR ZERO-SHOT LLM- GENERATED TEXT DETECTION
Guangsheng Bao Yanbin Zhao ?, Juncai He
Yue Zhang
Zhejiang University 2 School of Engineering, Westlake University
School of Mathematics, Physics and Statistics, Shanghai Polytechnic University 4 Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University Science and Technology
zhaoyb553@nenu edu. juncai he@kaust edu. (baoguangsheng, zhangyue)@west edu, cn
ABSTRACT
Advanced large language models (LLMs) can generate text almost indistinguish- able from nuittewritten text, highlighting the importance of LLM-genrrated text detection. However, current zero-shot techniques face challenges as white-box methods are restricted to use weaker open- source LLMs, and black-box methods are limited by partial observation from stronger proprictary LLMs. seems im- possible to enable white-box methods to use proprietary models because API-level access the models neither provides full predictive distributions nor inner em- beddings. To traverse the divide, we propose Glimpse, probability distribution estimation approach, predicting the full distributions from partial observations [] Despite the simplicity of Glimpse, we successfully extend white-box methods like Entropy, Rank, Log-Rank, and Fast-DetectGPT to latest proprictary models. Experiments show that Glimpse with Fast-DetectGPT and GPT-3.5 achieves average AUROC of about 0.95 five latest source models, improving the score by 51% relative the remaining space of the open source baseline. It demon- strates that the latest LLMs can effectively detect their own outputs, suggesting that advanced LLMs may be the best shield against themselves.
INTRODUCTION
Large language models (LLMs) (OpenAl 2022 Team et al. 2023 Anthropic 2024 can produce fluent and coherent text content, which almost indistinguishable from wrintnnwritten content (Ip- polito et al.) 2020 Shahid et al. 2022 Dugan et al.) 2023). It powers the productivity of various industries such as journalism [Christian] 2023), social media (Yuan et al., 2022). and education (M Alshater 2022). but at the same time it causes various risks such misinformation. disinfor- mation, and plagiarism (Pan et al. 2023 Weidinger et al. 2021] Meyer et al. 2023). thus urging automatic detection tools for building trustworthy Al systems (Kaur et al.) 2022 Sun et al. 2024). However, as long as LLMs increase their ability, detection of their generations becomes more diffi
cult (Mireshghallah al. 2023
Sadasivan al. 2023 Tang al. 2024).
The most powerful LLMS are generally proprietary models, which only provide limit access through API. Consequently, the existing white-box methods that require "full access' cannot be applied to these models, and instead, various black-box methods are developed. Black-box methods, such as DetectGPT (Mitchell al.) 2023) and DNA-GPT (Yang et al. 2023a), demonstrate competitive detection accuracies in ad vanced LLMS such as GPT-3, ChatGPT, and GPT-4. However, these methods are less efficient and less robust compared to their white-box counterparts (Bao et al.
2023), because they rely on knowing the source model and need multiple evaluations generations
Corresponding author.
We release our code and data httpap/g//ggiibb..cc//appgrggger impss impae

---

Published as a conference paper at ICLR 2025
4 Fast-DetectGPT with Glimpse
Passage x: "President Joe Biden claimed an interview with The Weather Channel..."
Yes
(1) Sample
(2) Score
(3) Compare
x from GPT
p(())
logp(x)
GPT
x from other
No
GPT
Partial Observation
Full Distribution
(e-g., President
rank
token
probabiitty
Joe
37%
API
return
Top-3 choices for
Biden
24%
GPT
rank taken
probabiitt
Glimpse
Donald
11%
Algorttm--
37%
Obama
8%
estimated
2 Biden 24%
Barack
probabilties
Donald
Figure 1: Take Past-DetectGPT as an example to apply Glimpse. The notion GPT refers to the model with estimated distribution, where the partial observation (top-K probabilities) returned by the model API is completed into a full distribution. The 'token' column is just for reference, which necessary for calculating the metric (conditional probability curvature).
of text sequences. Instead of improving these black-box methods like Su et al. (2023), we turn back to white-box methods to see the potential of combining their strength with the power of the latest LLMs. 2
For example, Fast-DetectGPT [Bao et al. 2023) uses fixed surrogate model to detect text from various source models, including ChatGPT and GPT-4. The basic idea is to use surrogate model to obtain token distributions to calculate a metrie of conditional probability curvature, where the higher the metric, the more likely the input is -eeeiinegeeerrted.. Despite its simplicity, it achieves detection accuracies that are higher than those of the black-box methods. However, its detection accuracy GPT-4 (AUROC 0.91) is significantly lower than open-source models (an average of 0.99 on five models). We speculate that the lower accuracies in the latest models are caused by the distribution mismatch between the small surrogate model and the large source models, which can potentially be addressed by using the latest large models as the surrogate.
To enable white-box methods use proprietary models, we propose Glimpse, a probability distri- bution estimation approach, to estimate the full distribution from the partial observation returned by the model API. This observation includes the probabilities (logprobs) of the input tokens and a few (at least 1) top-ranking tokens on each token position. Specifically, take Fast-DetectGPT as an example, illustrated in Figure [ We first obtain the top probabilities from the GPT model and then use these probabilities to estimate the distribution across the entire vocabulary. The basic idea is to find an empirical correlation between the top probabilities and the full vocabulary probabilities. To this end, we consider the parameterized Geometric distribution, Zipfian distribution, and an MLP model trained on the data to model the correlation. Using Glimpse, we also extend methods like Entropy, Rank, and LogRank to proprietary models.
Experiments show exceptionally strong results for these white-box methods using the latest LLM, where Rank, LogRank, and Fast-DetectGPT using Glimpse outperform their versions using open- source LLMs with significant margins (more than 25% relative to the remaining space). We achieve the best detection accuracies for five source models, obtaining an accuracy of 0.98 for ChatGPT, 0.94 for GPT-4, 0.96 for Claude-3 Sonnet, 0.97 for Claude-3 Opus, and 0.92 for Gemini-1.5 Pro in the stimulated white-box setting. These results indicate that the latest LLMs can detect their generations with high accuracy.
To our knowledge, we are the first to investigate white-box detection methods using proprietary mod- els, achieving significantly improved detection accuracies among zero-shot methods, demonstrating that the most powerful LLMs may be the strongest 'shield" against themselves.
By 'latest', we refer to the most powerful models are widely used like ChatGPT and GPT-4. Provided by Completion API of popular models like Google's Pal M-2 AAil et al. 2023), OpenAl's GPT-3 Brown et al. 2020), GPT-3.5 (OpenAl 2022) and GPT-4 (OpenAl 2023
2

---

Published as a conference paper at ICLR 2025
2 METHOD
We use Past-DetectGP1 [Bao et al. 2023 example to illustrate how to apply Glimpse to existing white-box methods in Section 2.2 and present three specific probability distribution esti- mation algorithms in Section 2.3
We further discuss three more white-box methods with Glimpse
in Section 2.4
2.1 TASK AND SETTINGS
We are focusing on the zero-shot detection of L.LM-generated text, which we frame as binary classification problem: determining whether given text created by model or human. A zero-shot detector usually uses a scoring model to produce a detection metric and makes a decision by comparing this metric to predetermined threshold. The scoring model can either be the same as the source model or a different one, for example a fixed delegate model. The access level to this scoring model can vary. and methods can be categorized into black-box white-box methods based on this (similar to Tang et al. (2024)).
Black-box methods API-level access interact with the scoring model, for example proprietary GPT-3.5, while white-box methods assume full access to the scoring model, for example open source Nco-2.7B. this definition, methods such as Fast-DetectGPT (Bao al. 2023) and PHD (Tulchinskii et al. 2024), which employ a delegated open source model to identify generations from proprietary LLMs, are classified white-bos methods. It crucial to note that this definition differs from Yang et al. (2023b), which defines black box and white box based on the access level to the source model.
2.2 FAST-DETECTGPT WITH GLIMPSE
Adding Glimpse does change the overall framework of the baseline. only replaces the model distribution with the estimated distribution po when the missing part of pe is required. In the following description, rehearsal the framework of Fast-DetectGPT with the updated distribution.
Fast-DetectGPT posits that human and machine language generation differs word selection based on context. While machines favor words with higher model probabilities, humans do not necessarily demonstrate such tendency. This discrepancy is quantified using metric, naming the conditional probability curvature. And we decide text is machine generated by comparing the metric with threshold which is chosen according to specific scenarios.
Formally, given a text passage H, a proprietary model po, and an estimated distribution pe. the conditional probability function defined by Fast-DetectGPT is expressed as
(o(ā = II po (ā (
(1)
which denotes the predictive distribution of the model taking x as the input. As a special case, when equals x, p=(x) Pe(x). Take the second word position (j 2) of the passage 'President', and the probabilities for possible choices 722 E ['Joe', 'Biden', 'Donald',.. are [0.37, 0.24, 0.11, The tokens n, for different independent of cach other given the input. This conditional independence allows for efficient calculation the algorithm.
in Figure as an example. Its context is x<2
Using the conditional probability function, the conditional probability curvature defined by Fast- DetectGPT is written as
log (e
d(z, pp)
where the token likelihoods Po (z Fz3) are provided directly by the proprietary model, like po(22 = 'Joe" ( 'President") 0.37, matter whether æ; is among the top-K choices. The expected
3

---

Published as a conference paper at ICLR 2025
score ji and the expected variance 8² are computed using the estimated distribution Pa
llog (o (ūlla
Σ Bay-oo(aylzes) (og Σms
5² -B (10g) - [) = Σ B3 Be [(oggo( -Ks))] - oj.
In the expectations, 2 Po [ denotes the sampling step, while log pe (0|z) denotes the scor-
ing step. illustrated in Figure [ The expectations over the whole sequence are calculated by accumulating all token-level expectations.
The token-level mean jy signifies the entropy of the predictive distribution the j-th token. We calculate analytically by enumerating all possible choices
B3 Ex, (2(5)ee))) [og (f,
Σ log
(4)
such já 0.37 log 0.37 0.24 log 0.24 + 0.11 log 0.11 The token-level variance 83 is also computed analytically as
0] (2y=oo((ylkee) pologeo(i
[r,) = Σ p (يم
log" po (ā()) - -p..
(5)
where the summation over žj is similarly achieved by enumerating all possible choices, like 03 0.37 log 0.37 0.24 log" 0.24 + -...
2.3 GLIMPSE: A PROBABILITY DISTRIBUTION ESTIMATION APPROACH
Formally, given text sequence v, the proprietary model (o (x) provides us the likelihood po(z)(ıee)) and the top-K K token probabilities po (if) ( on each token position j. where denotes the rank. The problem is then formulated as estimating (ā ( over the whole vocabu- lary according to the given information.
However, in general, we do not need token-probability pairs for metric calculation. Take Fast- DetectGPT an example. The mean ãú and variance σ3 only depend the probability values in the distribution. That to say, we can calculate them using only the "probability' column in the full distribution in Figure where the 'token' column is not necessary. This advantage Fast-DetectGPT. Other methods like Entropy, Rank, and Log-Ram Rank can also be calculated from the probabilities only.
specific to
To simplify the discussion in the following, we denote the probabilities as p(k),, omitting the ex- pression of the token i. and position j. Using the top-K (K 3 typically) probabilities p(k) K k= we estimate the rest p K-1 K- where M denotes the size of the list. is worth noting that is not necessarily as large as the vocabulary size, because large ranks generally correspond to low probabilities. When the probabilities are small enough, their effects on the metric are ignorable.
Consequently, ūy and o3 rewritten
Σ
(o( log = Σ p(k) p(k);
(6)
83 Σ (o( log
P po(E,(zz)) n Σ p(k) log² p(k) - -13.
where the summation over all possible tokens ā, thus converted into the summation over all possible ranks k.
The probability distribution across ranks generally follows a decaying pattern, where the larger models tend to have higher top-1 probability and a bigger decay factor demonstrating sharper distribution (Appendix(A) We approximate the pattern using parameterized distributions, allocating the remaining probability mass (excluding top-K probabilities) to ranks larger than K.
4

---

Published as a conference paper at ICLR 2025
Each parameterized distribution represents a family of distributions controlled by some parameters. Traditionally, we fit the distribution to the data points to estimate the parameter, where we could use the top-K K probabilities as the data points. However, this approach may underutilize the observed information and infer top- probabilities different from the real ones.
In this study, consider the estimation problem in constraints. The basic constraints include: 1) total probability constraint a summation of the probabilities equals 1. 2) monotonic decrease con- straint the probability of higher rank is lower. Using the total probability constraint, we decide the probability mass to allocate to ranks larger than K. Using the monotone decrease constraint, we decide on suitable family of distributions. We discuss three specific estimation algorithms as follows.
Estimation Using Geometrie Distribution. As the simplest decaying pattern, we consider ex- ponential decay with a fixed decay factor, resulting in Geometric distribution. We extend the distribution multiple top probabilities and limited range of k. We express the probabilities for ranks larger than K in near-Geometric distribution that
p(k) ph+ p(k) Pk A
for k E [1..K] for k E [K + 1..M]
Σ p(k) 11
where À is a decay factor in (0, 1), and M is the size of the rank list. We solve À under the constraints as detailed in Appendix A.I
Estimation Using Zipfian Distribution. Frequencies of words in natural languages usually adhere to Zipf"s law (Zipf] 1946] [946] 2013), where word frequency and word rank follow a Zipfian distribu- tion. Assuming that the word frequencies in given context also comply with this law, we consider it as an alternative distribution for our estimation. Given the top-K K probabilities pk. we compute the probabilities of tokens with ranking greater than K in Zipfian distribution
p(k) Pk: p(k) = PK (k-K+B)² )
for k [1..K] for [K + 1..M]
(8)
Σ k== Ekmp(()) p(k) =1
where a and B are two positive parameters. M is the size of the rank list. To solve the two param- eters, we introduce an additional loss function to minimize their deviations from typical values. We identify the best a and 3 by searching a loss table Loss(a, (3) as detailed in Appendix A.2
Estimation Using MLP Model. The Geometric and Zipfian algorithms both work on assumptions about the distributions. An alternate approach that does not rely on these assumptions involves modeling the distribution within a neural network. We consider the simple Multi-Layer Perceptron (MLP) model with a single hidden layer, which accepts the top-K probabilities and predicts the probabilities for the rest the ranks. The distribution expressed as
pkk) = Pk, p(k) Prest PMIP (k K),
for k E [1..K]
for k E [K + 1..M]
(9)
Σ Rk=1 p(k) 1,
M
where Prest 1 Σ Pk and PMLP, represents the MLP predictive distribution. The MLP is defined in detail, including training and inference, in Appendix A.3
2.4 UNIVERSALITY OF GLIMPSE
Glimpse can also be used by other zero-shot detection methods like Entropy, Rank, and LogRank. For Entropy, the calculation is straightforward by summing p(k) log p(k) over all items in the rank list. For Rank and LogRank, we decide the rank of the current token by searching the closest p(k) compared to the probability of the current token (1,(),)) ). In this study, we only examine the basic white-bot box methods, leaving the integration of more sophisticated white-box methods and additional estimation algorithms to future research.
5

---

Published as a conference paper at ICLR 2025
Neo-2.7
Fast-Detect
LogRank
Rank
0.4
0.90
0.3
Geometric
Zipfian
AUROC
0.80
0.85
Real
Ziprinn
Geometric
MLP
0.2
0.70
0.1
0.65
0.0
0.60
3
9
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Top-K
Dg
Du
DxL
Figure 2: against real distributions
KL divergence
Figure 3: Correlation between AUROC and KL divergence, evaluated on XSum produced by GPT-4. We use the open-source model Neo- 2.7B the scoring model for Glimpse algorithms.
from Neo-2.7B.
EXPERIMENTS
3.1 SETTINGS
We evaluate our methods on four datasets that cover seven languages, five source models that cover three model families, and four scoring models from small to large. We run each main experiment three times and report the median. The metrics used are described in Appendix B.1
Datasets. We follow studies (Mitchell et al. 2023 Bao et al., 2023 Yang et al. 2023a; Zeng et al,
2024a) on the evaluation datasets and their settings: XSum for news (Narayn et al. 2018 8), Wriling
for story (Fan et al. 2018). and PubMed for technical question answer (Jin al.
2019 where
for each dataset 150 human-written samples randomly selected and corresponding LLM texts are generated using the same prefix (30 tokens for articles or questions for QAs). Furthermore, M4 (Wang al. 2024) incorporate various languages such as Chinese, Russian, Urdu, Indonesian, Arabic, and Bulgarian, where for each language 150 pairs are randomly selected from the ChatGPT subsets. To evaluate the detectors ability to handle diverse domains and languages, we combine XSum, Writing, and PubMed into a single dataset called Mix3, and the six language datasets into another called Mix6.
Source Models. We evaluate our detector on five latest LLMs from different companies, includ- ing CharGPT (gpt-3.5-turbo (OpenAI, 2022), GPT-4 (gpt-4) (OpenAl 2023). Claude3 Sonnet Souude---oomeet222242222
(Anthropic) 2024). Gemini-1.5 Pro (gemini- 11mmnni-1.5-pro) (Team et al. (202)), where Opus is supposed at the same level as GPT-4 and Sonnet at the same level as ChatGPT. We use the ChatCompletion API of these models to prepare the datasets.
and Opus claude-3-opus-202402222
Scoring Models. A good scoring model can detect generations from wide range of source models. We use OpenAl GPT series (from small to large) as the scoring model, inleuding Babbage (babbage- 002, 1.3B) and Davinci (davinci-002. 175B) (Brown al. (020), GPT-3.5 (gpt-35-turbo-00)
or
gpt-35-turboc 106 almost equally, 175B) (OpenAl 2022), and GPT-4 (gpt-4-1106) (OpenAl 2023) scoring models using AzureOpenAl (see Appendix B. 2). For comparison, we Neo-2. 1.o-2.7 (gpt- neo-2.7B) (Black et al., 2021), Phi2-2.7B (Javaheripi et al. 2023), Qwen2.5-7B (Yang et al.) 2024
Team 2024), and Llama3-8B (Dubey et al. 2024) representatives of open-source models, run
locally on a Tesla A100 GPU.
Baselines. Among zero-shot detectors, compare methods with existing solutions such as Bust-DetectGPT" (shortly Fast-Detect) (Bao et al. 2003). DetectGPT (Mitchell et al.] (002)). /DNA- GPT (Yang al. 2023a),, and simple baselines such Likelihood (equivalent perplexity), En- tropy, Rank, and Rank (Gehrmann et al. 2019 Solaiman et al.) 2019 Ippolito et al. 2020). For other detectors, we compare our methods commercial GPTZero Tian 2023).
Hyper-Parameters. By default, we use top-5 probabilities. a rank list size of 1000 for Geometric and 100 for Zipfian and MLP, prompt4 from Table 6 for GPT-4 and prompt3 for other scoring models. These settings are ablated in Section 3.4 with one hyper-parameter changed each time, where we report the average accuracy over XSum, Writing, and PubMed.
ons-ap1

---

Published as a conference paper at ICLR 2025
Table 1: Main results for Glimpse, where the columns 'GPT-4', 'Claude-3', and 'Gemini-I.5' dis- play the AUROCs on the diverse Mix3 dataset, with the detailed results for GPT-4 and Gemini-1.5 in Table[3]and Claude-3 in in Appendix [ We mark the best in each column in bold. Methods marked with & require multiple evaluations or generations to detect one passage, thereby, consum- ing multiple times of cost and time. Additionally, we make a comparison of ACC in Table S
Method
ChatGPT Writing
CharGPT Mix3
GPT-4 Mix3
Gemini-1.5 Sonnet Opus/Mix3 Pro/Mix3
Claude-3
Avg.
XSum
PubMed
Mix3
GPTZero
0,9843
0,9303
0,8403
0,9143
0,9009
Zero-Shot Detectors Using White-Box Open-Source LLMs
Likelihood (Neo 2.7)
0.9578
0,9740
0,8775
0,9071
0.7690
0.8661
0.9030
0.7416
0,8373
Entropy (Neo-2.7)
0.3305
0.1902
0.2767
0,3136
0,4114
0.3466
0.3265
0.3959
0.3588
Rank (Noo-2.7)
0,7494
0,8064
0.5979
0,7044
0,6448
0,6888
0.7056
0.6260
0.6739
LogRank (Nco-2.7)
0.9582
0.9656
0,8687
0,9059
0.7626
0.8654
0.9042
0.7353
0.8347
DNA (Neo-2.7) +
0.9124
0,9425
0.7959
0.7192
0.6430
0,7080
0.7326
0.6438
0.6893
Detec1GPT (T5- IB/Nen2///)) +
0.8416
0
0.7444
0.7826
0.6136
0.7967
0.7776
0.7406
0.7422
Fast-Detect (GPT-I/Nco-2.7)
0.9907
1.9916
0.9021
0,9487
0.8999
0,9260
0.9468
0.8072
0,9087
FasT. 7P2 7755
0..005
T..2445
0.762T
T0,3742
0.6957
0T7450
(T6664
0.6788
Fast-Detect (Qw)n2.5-7B)
0,7808
0.8117
0.7887
0.7655
0.6862
0.7813
1.81
0.6839
0,7458
Fast-Detect (Llama3-8B)
0,8508
0,8446
0.7941
0,7796
0.7269
0,8212
0.8510
0.7552
0.7868
Zero-Shot Detectors Using Black-Box Proprictary LLMs
Likelihood (GPT-3.5)
0.9203
0,9925
0.9544
0,9246
0.8029
0.9023
0.9295
0.8043
0,8727
DNA-GPT (GP5:33.5)
0.9260
0.9329
0.9304
0,8369
0.7748
0.7871
0.8383
0.7107
0.7896
Glippi asdny Geometric
Entropy (GPT:3.5)
0.3188
0.0463
0.97
0,2160
0.4074
0.2582
0.2339
0.4144
0,3060
Rank (GP):3.5)
0.8577
0,9845
0,8383
0,8533
0.7395
0,8473
0.8645
0.7406
0,8090
I LogRank (GPT-3.5)
0.9240
0.9931
0,9532
0.9277
0.7870
0.9062
0.9336
0.7872
0.8683
Fast-Detect (Babbage)
0.9908
0.9904
0.9570
0,9764
0,8974
0,9438
0.9698
0,8083
0,9191
Fast-Detect (Davinci)
0.9900
0.9976
0.9421
0,9763
0.9131
0.9606
0,9742
0.8601
0,9369
DesteD (GPT:3.5)
0.9808
0,9972
0,9766
0,9411
0,9576
0.9689
0.9244
0,9537
Fast-Detect (GPT-4)
0.9815
0.9935
0.9564
0,9735
0.9647
0,9623
0.9817
0.8947
0.9554
Glimpse äsiig Zizjion
Dyut-Dttect (GPT
0.9826
0.9956
0,9639
0,9647
0,9319
0,9475
0.9588
0.9161
0,9438
Fast-Detect (GPT-4)
0.9885
0.9917
0.9461
0,9768
0,9719
0.9613
0.9792
0.8991
0.9576
Glimpse asing MLP
Fast-Detect Deteeeeeeeees-S (GPII-355
0.9819
0,9959
0.9676
0,9702
0.9342
0.9526
0.9634
0.9184
0,9478
Detect (GPT.4)
0.9869
0.9930
0,9528
0.9771
0,9705
0.9631
0.9807
0.9001
0.9583
3.2
THE EFFECTIVENESS OF GLIMPSE
We assess the effectiveness of Glimpse by comparing the estimated distributions with the real dis- tributions, using Neo-2.7B the scoring model. As Figure 2 shows, their Kullback-Leibler (KL) divergence decreases as long as more top probabilities are used. Overall, MLP obtains the low- est divergence, while Geometric distribution has the highest divergence, suggesting the most accu- rate estimation of MLP. However, the correlation between KL divergences and detection accuracies varies for different detection methods and estimation algorithms. As Figure 3 illustrates, the AU- ROC declines when the divergence increases for Glimpse (Fast-Detect and LogRank) with any of the estimation algorithms, while the AUROC inclines for Glimpse (Rank). Although the geomet- ric distribution has larger divergences in general, it achieves higher or equal detection accuracies than other algorithms, suggesting that the effect of Glimpse is beyond the similarity of estimated distributions.
is worth noting that the accuracies achieved by Glimpse are not significantly lower than the base- line using real distribution for Fast-Detect and LogRank, and are even significantly higher than the baseline for Rank, suggesting the effectiveness and potential of Glimpse.
3.3 MAIN RESULTS
Accuracy and Efficiency. We first compare Glimpse with existing black-box methods on their detection accuracy, speed, and cost. As I Fast-Detect (GPT-3.5) surpasses both Like- lihood (GPT-3.5) and DNA-GPT (GPT-3.5) with a significant margin in five source models (in AU- ROC). The basic method LogRank (GPT-3.5) also outperforms DNA-GPT in four-fifth source mod- els, demonstrating the effectiveness and universality of Glimpse. Furthermore, Glimpse methods spend significantly less time and cost than DNA-GPT during the detection process. Specifically, DNA-GPT takes a total of 1911 I seconds across the three datasets, while Glimpse takes just 462 seconds, making the detection process 4.1 times faster. Since DNA-GPT creates 10 completions per
7

---

Published as a conference paper at ICLR 2025
XSum
Writing
PubMed
Geometric
Zipfian
MLP
1.00
1.00
0.98
0.95
0.96
0.90
0.94
0.85
0.92
0.80
0.75
GPT-3.5
GPT-4
GPT4
GPT-3.5
GPT4
10
Scoring Model
Scoring Model
Scoring Model
Top-K
Top-K
Top-K
Geomatiic
zistion
MLF
Figure 4: where the AUROC averaged across the five source models. Each dataset has its own pre- ferred algorithm.
Ablation on estimation algorithm,
Figure 5: Ablation on top-K, K, where the AUROC is averaged across the datasets produced by GPT- 4. Each line represents a combination of methods and scoring models.
passage, but Glimpse merely echos the probabilities of the input, DNA-GPT ends up costing about 10 times more than Glimpse based on current pricing (where the cost per output token is twice the input token 5
Here, we skip the comparison with DetectGPT in the black-box setting, because it requires 100x API calls and is affirmed have lower accuracies compared DNA-GPT (anng et al. 2023a) and Fast-Detect (Bao et al. 2023), which are already our baselines.
Latest LLMS are more accurate detectors. We then compare our method with other existing meth- ods. Powered by Glimpse, white-box methods such as Rank, LogRank, and Fast-Detect using GPT- 3.5 significantly outperform their open-source versions using Neo-2.7, with relative improvement of 41%, 21 %,, and 51 %6, respectively. Glimpse methods with smaller models also show competitive results. Fast-Detect (Babbage) outperforms Fast-Detect (Neo-2.7) on five source models, even though Babbage (1.3B) is smaller than Neo-2.7 (2.7B). These results demonstrate that the latest LLMs with Glimpse are strong detectors.
Larger LLMs can be universal detectors. A recent study (Mireshghallah al. 2023) reports that smaller LLMS are more efficient universal detectors, which can detect generations from various experimental results suggest that larger models also be universal
source models. However, detectors, which perform better than smaller proprietary and open-source models in various source models. Specifically, GPT-3.5 (175B) achieves an average AUROC of about 0.95, outperforming smaller Babbage (1.3B) by about 1 43% and Neo-2.7 (2.7B) by about 1 51 %. These findings suggest that larger LLMs, when equipped with the right technique, can also serve as universal detectors.
3.4 ABLATION STUDY
Necessity of Glimpse. Readers may wonder the necessity of these estimation algorithms, given that the top-K probabilities provide the major information. To testify it, we consider Naive approach to estimate the full distribution, where we assign zero probability to ranks larger than K. Using the Naive distribution, the average AUROC of Fast-Detect (GPT-3.5) is downgraded from 0.9630 (Geometric) to 0.9311 (Naive), indicating the necessity of proper estimation algorithm.
Ablation on Estimation Algorithm. As the main results in Table I show, the Glimpses using Ge- ometric, Zipfian, and MLP distributions do not show significant differences in their average accura- cies. However, when we look into each dataset, we see different patterns as Figure demonstrates.. Specifically, the geometric distribution performs the best in Writing and PubMed, while Zipfian performs the best on XSum. MIP is more balanced, which performs the median on three datasets. These patterns remain with both GPT-3.5 and GPT-4 as the scoring model. These results suggest that different estimation algorithms may suit different situations, which is also supported by experiments on languages shown in Table 2
Ablation on Top-K. Intuitively, the higher the value of K, the more precise the estimated distribu- tion is likely be. We conduct an ablation study with two representative methods with two scoring models as shown in Figure 5 The results, particularly with the geometric distribution, generally corroborate intuition. However, Zipfian and MIP show exceptions, where LogRank (GPT-4 and GPT-3.5) with Zipfian and Fast-Detect and LogRank (GPT-4) with MLP obtain their highest accu- racies with the top-1 setting. The results indicate that even with limited top-1 probabilities, Glimpse can still work effectively when proper estimation algorithm is used.
httpa: wwwp::/wwww..rrrroorooooooooooeo
s/cogg/ ve-aerrv pea/ wewp/peaaaopppeeeeeeeeeeeeeeeeee
8

---

Published as a conference paper at ICLR 2025
ChatGPT
GPT-4
Claude-3 Sannet
Claude-3 Opus
Gemini-1.5 1.5 Pro
0.8
Posiive
0.6
0.4
0.2
0.0
10
10
10a
10
10
100
10
10
109
10
10
100
10
10
109
Fositive Rate
Falsa Positive
Positive Rate
False Positive
Positive
Likelhood (GPT-355
Fast Detect (PPT-JNeo-2..1
Fast Detect (Baboage)
Fast Detect (GPT-3.5)
Fast Detect (GPT-4)
Figure 6: Robustness on low false alarms, where the red lines indicate false alarms of 1 1% and 10%. We draw the curves using Mix3 to simulate the real scenarios of using a single threshold to detect different domains. The dash lines denote the random classifier.
Table 2: Robustness over languages, where denotes the diverse mixture of the six languages.
the same settings as the main experiments. Mix6
Chinese (Web QA)
Russian (RaATD)
Urdu (News)
Indonesian (News)
Arabic (Wikipedia)
Bulgarian (News)
Method
Mix6
Fast-Detect (GPT-J/Nco-2.78
0,9319
0.8158
0.9630
0,9876
0,9121
0,9422
0.8862
Fast-Deeeci (Phi2-2.7B)
0.8024
0.6710
0.9049
0,9313
0.9155
0,7500
0.8205
Dest-Detect (Qwen2.5 7BB)
0.5345
0.7065
0.9970
0.9687
0,8404
0.8992
0.8063
Rast-Detect (Llama3-8B)
0.8729
0.7984
0,9962
0.9922
0,9282
0.9723
0,8713
Glimpse using Geometric
Fast-Detect (Babbage)
0.9814
0.7673
0.9894
0.9857
0.9916
0.9628
0.8950
Fast Delect (Davinci)
0,9938
0,8030
0.9996
0.9991
0,9982
0.9842
0.9369
Fast-Detect (GPT-3.5)
0.9913
0.8555
1,0000
0.9996
0.9999
0.9925
0.9774
Ablation on Rank-List Size and Prompt. The size of the rank list is a crucial hyper-parameter affecting Geometric and Zipfian distributions. Larger size typically results in higher accuracy, but it varies between estimation algorithms (see Appendix D.I). In addition, we examine how different prompts affect text detection accuracy in large models. The most sensitive model is GPT-4 in contrast to less sensitive GPT-3.5 and the least sensitive Babbage and Davinci (see Appendix D.2
3.5 ANALYSIS AND DISCUSSION
Robustness across Source Models and Domains. real scenarios, we tend to use a constant threshold to detect text from various sources and domains. We evaluate the stability of Glimpse and other baselines across the source models and datasets using thresholds found on "out-of-domain" datasets. The results show that Glimpse consistently provides the highest accuracy across the source models and domains, showcasing its stability. See Appendix E.. for detailed setup and discussion.
Robustness on Low False Alarms. In real scenarios, effective detector is expected to have high recall (true positive rate) with a low false alarm (false positive rate), thereby allowing it to machine-generated capacity of the methods by contrasting their ROC curves. As illustrated in Figure for a false alarm in the range of (0.01, 0.1). Fast-Detect (GPT-4) performs the best, except ChatGPT generations, where Fast-Detect (Babbage) has the highest recalls. For false alarm lower than 0.001, Fast-Detect (Babbage) performs consistently better than other methods, suggesting the advantage of it for low false alarm setting, Compared to baseline Likelihood (GPT-3.5) and Fast-Detect (Neo-2.7), Glimpse versions of Fast-Detect show a consistent advantage.
identify most machine- generated text without misclassifying human-written text. We evaluate the
Robustness over Languages. We assessment Glimpse methods on M4 datasets with six languages. As Table 2 shows, Fast-Deted (GPT-3.5) consistently outperforms the baselines, as well as other proprietary LLMs.. The system almost flawlessly identifies Urdu, Indonesian, and Arabic texts. However, the detection accuracy for Russian texts much lower, indicating potential under- training of the LLMs on this particular language. These findings imply that Glimpse with the latest LLMs can be effective and reliable detectors across languages.
Robustness under Paraphrasing Attack. We test the performance of Glimpse against paraphrasing attack using DIPPER, with two settings: high lexical diversity and high order diversity. (Babbage) outperforms Fast-Detect (Nco-2.7) in both, but is more affected by diverse lexicons. An unusual behavior of DIPPER is observed in XSum, which causes an exceptionally low accuracy of the Likelihood (GPT-3.5) baseline. Glimpse surpasses Detect (Neo-2.7) and trained detectors,

---

Published as a conference paper at ICLR 2025
proving its effectiveness. Larger LLMs are more vulnerable increased diversity, which also reduce readability. See Appendix E.2 for a detailed comparison and discussion.
Limitations. Glimpse does not support all white-box methods, especially the rare methods that use inner embeddings instead of predictive distributions, such as PHD (Tulchinskii et al., 2024). Additionally, not all proprietary models provide Completion API, therefore, Glimpse cannot use them as the scoring model.
Broader Impact. The estimation methods potentially be applied to other scenarios. For exam- ple, ChatCompletion API also provides top- probabilities. Using Glimpse, we can estimate the predictive distribution, which could be used calculate some statistical metries the generated content. Such metrics could potentially be used to indicate, for instance, the level of hallucination in the content. Despite the effectiveness and potential of Glimpse, unsupervised use can lead to poten- tial unfairness towards certain groups. For instance, detectors might display bias against non-native writers (Liang et al. 2023).
RELATED WORK
In terms of using a proprietary model for scoring, our method in line with existing black-box methods. However, it inherits existing white-box approaches.
Black-Box Methods for Zero-Shot Detection. Zero-shot detection in the black-box setting is chal- lenging due to restricted access of the model. DetectGPT (Mitchell et al., (023) and its improvement NPR (Su et al. 2023) requires multiple evaluations of text sequences, while DNA-GPT (Yang al.) 2023a) requires multiple generations of text sequences, resulting in low speed and high cost. An- other line of approaches treats detection as a question-amm -anseirngnn task but does not reliably discern text generated by ChatGPT and GPT-4 (Bhattacharjee & Liu (024). These methods generally pre- sume the knowing of the source model and use to ascertain a text was produced by it, which limits their usage to texts from uncertain origin. Unlike these approaches, our method minimizes the cost and dependence on known sources.
White-Bor Box Methods for Zero-Shot Detection. Zero-shot methods in the white-box setting analyze a variety of metrics from model predictive distributions or output embeddings, including Entropy and Perplexity (Lavergne al. 2008). Likelihood (Hashimoto et al., 2019: Solaiman et al.
2019). Rank
and Log-Rant (Gehrmann et al. 2019), LRR (Su et al. (023). Fast- Dest-DetectGPT (Bao et al.
2023
PHD (Tulchinskii et al.) 2024), FourierGPT (Xu et al. 2024), and Binoculars (Hans et al.
2024
These methods. except that PHD requires output embeddings, can all be applied to proprietary models using Glimpse. However, in this study, we focus very basic Entropy, Rank, and Log- Rank, together with recent Fast-DetectGPT, leaving the rest for future exploration.
Other Detection Methods. Trained classifiers are the majority of black-box methods, which texts along with --Meenerrttedd content for training (Bakhtin et al., 20019]
rely human-
Uchendu et al., 2020; Solaiman et al. 2019 Ippolito et al. 2020 2020,Fagni et al. (Soaaman al.
1019IFagni et al. 2021 Yan et al.) 2023] Li et al.] 2023 Zeng et al.) (222bb (Verma et al. [2024] Kushnareva et al. 2024). However, they can become overly specialized their training conditions, such as specific domains, languages, or source models. White-box approaches such as watermark- ing provide different strategy, altering the LLM decoding process to include unique text signatures KKrchenbaaerr et al., 2023] Kuditipudi et al.) 2023 Christ et al.) 2024 Zhao et al. (223bba) a). How- ever, they require proactive text generation injection, which is not allowed by proprictary models. In this paper, investigate techniques that do not need training or proactive injection.
5 CONCLUSION
We proposed Glimpse to estimate full distributions from partial observations, enabling white-box text detection methods to use proprietary models. Experiments show that Glimpse with the latest LLMs performs significantly better than its counterparts in terms of accuracy, efficiency, and robust- ness, highlighting the benefits of using the latest LLMs as detectors. Additional results in various white-box methods underscore the effectiveness of Glimpse, which may lead to new direction of zero-shot detection in the black-box setting. To our knowledge, we are the first to enable white-box methods in proprietary L.LMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their valuable feedback. We thank Minjun Zhu for valuable discussion. This work is funded by the National Natural Science Foundation of China Key Program (Grant No. 62336006) and the Pioneer and "Leading Goose" R&D Program of Zhejiang (Grant No. 2022SDXHDX0003) Yanbin Zhao supported by the National Natural Science Foundation of China (Grant No. 12201158).
ETHICAL STATEMENT
High efficient and accurate machine- -eeerrttd text detector can potentially contribute to trustworthy AI techniques, which can mitigate the risks posed by the uncontrolled use of LLMs. Such technology can potentially benefit text readers, policy makers, and media platforms a wide way.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. /0403 2023.
Anthropic.
The claude 3 model family: Opus, somnet, haiku. https: http::/wwwwwww..wtoooo.. wwwp:/d...tttrrpp..... com/ de@ba 9b0 1ee001ccbbcccc cwobb (10.0/00600bb0113333303066633333
Cardd Caro__laadeee
3. p. 2024.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real fake? learning discriminate machine from human generated text. arXiv preprint 10X1v:/906.03351, 2019.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Effi cient zero-shot detection of machine-genene generated text via conditional probability curvature. In The Twelfth International Conference on Learning Representations, 2023.
Amrita Bhattacharjee and Huan Liu. Fighting fire with fire: chatgpt detect ai- ai-generated text?
ACM SIGKDD Explorations Newsletter, 25(2): 21, 2024.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh- Resh-Tensorflow, March 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. In The Thirty Seventh Anmual Conference on Learning Theory, pp. 1125-1139, PMLR, 2024.
Jon Christian. Cnet secretly used on articles that didn't disclose that fact, staff say. Futurusm, January, 2023. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint 177.2:2407.21783, 2024.
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Buil Burch.. Real or fake text?: Investigating human ability to detect boundaries between human wwitten and machine- generated text. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12763-12771, 2023.
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella. and Maurizio Tesconi. Tweepfake: About detecting deepfake tweets. Plos one, 16(5)xe0211415, 2021.
11

---

Published as a conference paper at ICLR 2025
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gitr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 111-116, 2019.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting Ilms with binoculars: Zero-shot detection of machine-generatte text. In Forty-first International Conference on Machine Learn- ing. 2024.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. arXiv preprint SEK.v::9904002992. 2019.
Daphne Ippolito, Daniel Duckworth, Chris Callison- Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808-1822, 2020.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1:3, 2023.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 2019.
Davinder Kaur, Suleyman Uslu, Kaley Rittichier, and Arjan Durresi. Trustworthy artificial intelli- gence: a review. ACM Computing Surveys (CSUR), 55(2): 2022.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.102266, 2023.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-fret watermarks for language models. arXiv preprint arXivv2.307.1555 2023.
Laida Kushnareva, Tatiana Gaintseva, German Magai, Serguei Barannikov, Dmitry Abulkhanov, Kristian Kuznetsov, Eduard Tulchinskii, Irina Piontkovskaya, and Sergey Nikolenko. Ai- generated text boundary detection with roft. In Ist Conference on Language Modeling (COLM), volume 2024, 2024.
Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative tropy scoring. In Proceedings of the 2008 International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse-Volus 377, pp. 27-31, 2008
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang. Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Mage: Machine-generaae text detection in the wild. arXiv e-prints, pp- arXiv-2305. 2023.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. Patterns, 4(7), 2023.
Muneer M Alshater. Exploring the role of artificial intelligence in enhancing academic performance:
A case study of chatgpt. Available at SSRN, 2022.
Jesse G Meyer, Ryan Urbanowicz, Patrick CN Martin, Karen O'Connor, Ruowang Li, Pei-Chen Peng, Tiffani J Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Sonzalez-Hermandeer et al. Chatgpt and large language models in academia: opportunities and challenges. BioData Mining, 16(1 1) 1):20 2023.
12

---

Published as a conference paper at ICLR 2025
Niloofar Mireshghallah, Justus Mattern, Sieun Gao, Reza Shokri, and Taylor Berg- Kirkpatrick. Smaller language models are better black-box machine- -eeeerrte text detectors. arXiv preprint arXiv:2305.00889, 2023.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. De- tectgpt: Zero-shot machine -eeernnte text detection using probability curvature. arXiv preprint arXiv:2301.11305. 2023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018.
OpenAI. ChatGPT. https::/cchat..opoe com/
December 2022.
OpenAL GPT-4 Technical Report. arXiv preprint arXiv:2303.00 2023.
Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min- Yen Kan, and William Yang Wang. On the risk of misinformation pollution with large language models. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 11:1112-1130, 2014.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint 2rX 2023.
Wajiha Shahid, Yiran Li, Dakota Staples, Gulshan Amin, Saqib Hakak, and Ali Ghorbani. Are you a cyborg, bot or human?-a survey on detecting fake news spreaders. IEEE Access, 10: 17069-27088. 2022.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: / 090. 09.0000,099,,,..... 2019.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank informa- tion for zero-shot detection of machine- machine-generated text. arXiv preprint vrXiv:2306.05540 2023.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05555 1, 2024.
Kaito Taguchi, Yujie Gu, and Kouichi Sakurai. The impact prompts on zero-shot detection of ai-generated text. arXiv preprint 2rXiv:2403.20122 2024.
Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The science of detecting Ilm Ilm-generated text. Com- munications of the ACM, 67(4):50-59, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL //wp://....mmmmmmmmm hiithub.i//loo//ggee qwtpp///wwnn2/////m
Edward Tian and Alexander Cui. Gptzero: Towards detection of ai-generated text using zero-shot and supervised methods, 2023, URL httpt:tt/tpptteeeeee me
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniaskki, Sergey Nikolenko, Evgeny Burnaey, Serguei Barannikov, and Irina Piontkovskaya. Intrinsic dimension estimation for robust detection ti-generated texts. Advances in Neural Information Processing Systems, 36, 2024.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text genera- tion. In Proceedings the the 2020 conference on empirical methods in natural language processing (EMNLP),, pp. 3384-8395, 2020.
13

---

Published as a conference paper at ICLR 2025
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwrit- ten by large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume I: Long Papers), pp. 1702-1717, 2024.
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, et al. M4: Multi- genti-geneeraor, multi donli, and multi-lingual black-box machine -eenerrte text detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisties (Volume I: Long Papers), pp. 1369-1407, 2024.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprini urXiv:21/2.04359, 2021.
Yang Xu, Yu Wang, Hao An, Zhichen Liu, and Yongyuan Li. Detecting subtle differences between human and model languages using spectrum of relative likelihood. arXiv preprint arXiv:2406. 19874, 2024.
Duanli Yan, Michael Fauss, Jiangang Hao, and Wenju Cui. Detection of ai-generated essays in writing assessment. Psychological Testing and Assessment Modeling, 65(2):125-144, 2023.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report, arXiv preprint arXiv:2407. 10671, 2024.
Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint 1rXi::2305./7359 2023a.
Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. A survey on detection of llms-generated content. arXiv preprint arXiv:2310. 15654, 2023b.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Worderaft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces, pp. 841-852, 2022.
Cong Zeng, Shengkun Tang, Xianjun Yang, Yuanzhou Chen, Yiyou Sun, Yao Li, Haifeng Chen, Wei Cheng, Dongkuan Xu, et al. Dlad: Improving logits-based detector without logits from black-box 1lms. In The Tiihth Annual Conference on Neural Information Processing Systems, 2024a.
Cong Zeng, Shengkun Tang, Xianjun Yang. Yuanzhou Chen, Yiyou Sun, Yao Li, Haifeng Chen, Wei Cheng, Dongkuan Xu, et al. Improving logits- based detector without logits from black-box llms. arXiv preprint arXiv:2406.055 05232, 2024b.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306. /7439, 2023a.
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In International Conference on Machine Learning, pp.
PMLR,
2023b.
George Kingsley Zipf. The psychology of language. In Encyclopedia of psychology, pp. 332-341. Philosophical Library, 1946.
George Kingsley Zipf. The Routledge, 2013.
of language: An introduction to dynamic philology.
14

---

Published as a conference paper at ICLR 2025
A GLIMPSE: A PROBABILITY DISTRIBUTION ESTIMATION APPROACH
As indicated in Figure 7a the probability distribution across ranks generally follows a decaying pattern, where the larger models tend have higher top-1 probability and bigger decay factor demonstrating sharper distribution. We approximate the pattern using parameterized distributions, allocating the remaining probability specific estimation algorithms with decaying patterns like Figure 7b
(as **' indicates) to ranks larger than K. We discuss three
Neo-2. 2.7
GPT-3.5
GPT-4
Geometric
Zipfian
MLP
10a
1
Proeabury
10-1
Probability
10-2
10
10-4
10
10-5
2 4
1
5
40
100
20
80
100
40
100
Rank
Rank
Rank
Rank
Rank
Rank
(a) Top-5 probabilities, where '8* indicates the remaining probability mass.
(b) Full distribution completing the top-5 probabilities from GPT-3.5, estimated using the three algorithms.
Figure 7: Decaying patterns of averaged probability distribution over ranks, where the probabilities are shown in log scale and evaluated on XSum.
A.1 ESTIMATION USING GEOMETRIC DISTRIBUTION
As the simplest decaying pattern, we consider exponential decay with a fixed decay factor, resulting in geometric distribution. Considering only the top-1 probability Pt, the whole distribution could be estimated from #1 using
plk) = P1 (1 (1-pi) 1 for E [1..o0],
(10)
where the probability decays with factor of A (1 P1). However, this standard geometric distribution only considers the probability of top-1 and defined over infinite k. which is not suitable for finite vocabulary.
Consequently, we extend the distribution to multiple top probabilities and a limited range of k. We express the probabilities for ranks larger than K in near- wear-Geometre distribution that
p(k) pkv p(k) = PK x A= p(k) =
for k E [I..K] for & E [K + 1.
(11)
where À is decay factor in (0, 1), and M is the size of the rank list. Using the total probability constraint, we calculate the remaining probability mass for allocating
Σ
p(k)
Pk
Prest:
(12)
k=
1=1
Expanding p(k) in the left expression, we obtain
M-K
PK
Prest:
(13)
and rewritten
(A- A-
Σ x
Prest
(14)
pk
Assuming M-K+1 close to zero, we reach an approximate solution
X 2
Prest
(15)
PK Prest
15

---

Published as a conference paper at ICLR 2025
In practice, first calculate the approximate solution, naming λo and then check if Ao M--K+ close to zero. is not, we follow an iterative process to adjust until it converges
was
(he X - PK
A++1
(16)
Prest
A.2 ESTIMATION USING ZIPFIAN DISTRIBUTION
Frequencies of words in natural languages usually adhere to Zipf's law (Zipl 1946 2013 where the word frequency and word rank follow Zipfian distribution
pkk) x (k: + B)a for [1..oo].
(17)
The parameters a and are fitted to specific corpus, with typical values of a and 3 = 2.7 for English (Piantadosi, 2014).
Assuming that the word frequencies in given context also comply with this law, we consider it alternative distribution for our estimation. Given the top-K probabilities pk, we compute the probabilities of tokens with a ranking greater than K in a Zipfian distribution
p(k) = pk:
for E [1..K] for l E [K + 1..M)
B p(k) = РK (A k- K+B)",
(18)
Ex=1 1,
where and are two positive parameters. M is the size of the rank list. Using the total probability mass as a constraint, we determine the probability mass for allocating.
Σ
p(k) =
Σ
Pk
Prest
(19)
k=
1=1
After expanding pkk), get
M
3
РK
Σ
(k
K + =3
= Prest
(20)
U=
rewritten as
M Σ
a
M
3
)a
Prest
(21)
k
K +
k: +
pk
k=K+1
k=1
The equation has two unknown parameters, thereby having multiple possible solutions. Thus, solve the parameters by minimizing a loss function
M-K Σ
Loss(a, B)
Prest
1.0 (a- 1)2 0.001 (B 2.7)².
(22)
k+3
PK
As an additional constraint, we expect the parameters not vary from their typical values much. We empirically determine coefficient of 1.0 for a and a coefficient of 0.001 for 3.
To accelerate the optimization process, we construct a table T(a, .). which stores the pre-calculated summation values for each pair of a and |3 as
M-K K
T|a 3] =
Σ
(23)
k: 1+
We enumerate E (0, 10) with step of 0.1 and 8 (0, with step of 0.2, resulting in table with 10000 values. The ranges are empirically decided, which balance the coverage of the possible choices and the size of the table. During inference, table Lass(a, 3) from T|a, 3] given Presa/ /px- We then search the loss table to identify the best a and (8 that lead to the smallest loss.
efficiently compute the loss
16

---

Published as a conference paper at ICLR 2025
A.3 ESTIMATION USING A MLP MODEL
The Geometric and Zipfian algorithms both work assumptions about the distributions. An alter- native approach that does not rely on these assumptions involves modeling the distribution within a neural network. We consider the simple multilayer perception model (MIP) with single hidden layer, which accepts the top- K probabilities and predicts the probabilities for the rest of the ranks. The distribution is expressed
pkk) p(k) Prest PMLP (K K), for k E [K + 1..M]
for ls E [1..K]
(24)
ΣN pkk) 11
where Prest = =1 Ex=l Ps and PMLP, represents the MLP predictive distribution. The MI. is defined as
PMLP, SOFTMAX(MLP((2))
(25)
where 9 denotes the model parameters. The model inputs a vector with a size of and outputs a distribution with size of M K. The input vector is calculated from the top probabilities using
log Pk. for k E [I..K].
(26)
Training. We train the MLP model on probability distributions from an open-source LLM (c.g., GPT-Neo-2.7B), using the cross-entropy loss
Loss Σ Pa log
Σ k=K+1
Ph
K) +
(27)
where Pk for lo E [1..M] is the target distribution and p(k) the model distribution. is the constant part.
We predict the distributions using the top- K probabilities from the proprietary LLMs, obtaining estimated distributions all token positions. We do not enforce the monotonic decrease constraint during inference, but generally follows the constraint because the training target is monotonic decrease distribution.
Inference.
B EXPERIMENTAL SETTINGS
B. EVALUATION METRICS
AUROC. We measure the detection accuracy mainly in the area under the receiver operating charac- teristic (AUROC), which gives an overview of the detectors across all possible thresholds. AUROC values range from 0.0 to 1.0, and this value mathematically signifies the probability that domly chosen machine-generatee text has a higher predicted probability of being machine-genennn compared randomly selected numan-writted text. An AUROC of 0.5 is indicative of random classifier, while an AUROC of 1.0 suggests flawless classifier.
Accuracy (ACC). As complement, report the ACC for of the experiments. ACC denotes the ratio of the number of correct predictions to the total number of input samples, which works well only if there are equal number positive and negative samples.
True Positive Rate (TPR) and False Positive Rate (FPR). In depth, we compare the methods in TPR versus FPR various thresholds. A high TPR indicates that the algorithm is effective in identifying positive cases, while a high FPR indicates that the algorithm often misclassifies negative cases as positive.
B.2 AZUREOPENAL SETTINGS
17

---

Published as a conference paper at ICLR 2025
We use Completion API 6 of these models via AzureOpenAl platform with the following de- ployment settings. We echo the model to return the top probabilities the provided texts, without producing any new tokens.
Different models are supported by different regions of Azure platform. We deploy babbage-002 and davinci-002 the region of North Central US, gpt-35-uuroo0030 gpt-35-turby 106 and gpt-4-1106 on the region of West US. In addition, we also tried babbage-002 from the OpenAl API endpoint, obtaining the same results as the one the AzureOpenAl endpoint.
the region of East US, and
C
MAIN RESULTS
Table 3: Main results on GPT- and Gemini-1.5 generations, with the best AUROC marked in bold.
GPT-
Gemini-1.5 Pro
Method
XSum
Writing
PubMed
Mix3
XSum
Writing
PuhMad
Mix3
GPTZero
0.9815
0.8838
0.8193
0.9009
Zero-Shot Detectors Using Open-Soure
LLMs
Likelilood (Neu-2.7)
0.7980
0.8553
0.8104
0.7690
0,8013
0.8364
0.7064
0.7416
Entropy (Neo:2.7)
0.4360
0.3702
0.3295
0.4114
0.4170
0.2945
0.3838
0.3959
Rank (Neo-2.7)
0.6644
0.5688
0.6260
LogRank (Neo
0.7975
0.8286
0,8001
0.7626
0,8022
0.8102
0.7006
0.7353
DNA-GPT (Neo-2.7)
0.7347
0,8032
0.7565
0.6430
0.7996
0,8133
0.6438
DetectGPT (TS-1 (1-/o1//))-2.7)
0.5660
0.6217
0.6805
0.6136
0.7838
0,8256
0.6222
0,7406
Detset (GPT-J/Neo-2.7)
0.9067
0.9612
0,8503
0,8999
0.8571
0.8650
0.7075
0.8U72
Fasi. Detect 7P6222.71)
0.4635
1063663
'0.8083
0.3742
0.74554
0.6524
0.5976
D.6164
Detect (Qwen2.5.7B)
0.6476
0.8202
0.6391
0.6862
0,7082
0.7723
0.6296
0.6839
Fast- Detect (Llamu3 8B)
0.6615
0.7556
0.7269
0.7786
0.9085
0.7552
Zero-Shot Detectors Using Proprictary
LLMs
Likelilood (.)T:3.5)
0.6468
0.9570
0.9152
0.8029
0.7130
0.9644
0.8516
0.8043
DNA-GPT (GPT:3-5)
0.7952
0.8302
0.9092
0.7748
0,8036
0.6829
0.7738
0.7107
Glimpse using Geowetric
Entropy (GPT-3.5)
0.6353
0,2694
0.2376
0.4074
0.6237
0.2276
0.3361
0.4144
(GP1-3.5)
0.6245
0.8283
LogRank (GPT:3.5)
0.6319
0.9323
0.9060
0.7870
0.7102
0.9421
0.8329
0.7872
Fast-Detect (Babbuge)
0.9033
0.9264
0.9195
0,8974
0.8797
0.8316
0.7887
0.8083
Fast Detect (Davinci)
0.9141
0.9798
0,8864
0.9131
0.9062
0.9516
0.7612
0.8601
Fast-Detect (GPT-3.5)
0.9035
0.9957
0.9467
0.9411
0.9221
0.9840
0.9112
0.9244
Fast-Detect (GPT-4)
0.9673
0.9901
0.9534
0.9647
0.9188
0.9506
0.8477
0.8947
Glimpse uzing Zipfan
Fast Detect (.5)
0.9123
0.9429
0.9809
0.9161
Fast-Deted (GPT-4)
0.9797
0.9884
0.9436
0.9719
0.9303
0.9447
0.8336
0.8991
Glimpse using MLP
Fast Detect (GPT-3.5)
0.9076
0.9930
0,9464
0.9342
0.9257
0.9810
0.9103
0.9184
Fast-Detect (GPT-4)
0.9759
0.9893
0.9496
0.9705
0.9272
0.9495
0.8453
0.9001
/altps://plaLComm www.oppaaa.oo..........omaa. wptpeaai..o//acc////ii/eeeeeeeee /ttp:////eeeeeeeeeeeeeeeenoooi/e
generalion/ wwmplet.ppecooooocoooooooo.o...o lons apı
Completion API with gpt-35-turbo-1 and gpt-4-1106 rqquic
roquire an AzuroOpenAl API version 2024-02-15--revveee or loter, while
others require 1023-00-15-previee or later.
httpp: ttrpat//aauve.... www.orrr.troooooooooooooooo..
18

---

Published as a conference paper at ICLR 2025
Table 4: Main results on Claude-3 generations, with the best AUROC marked in bold.
Claude-3-Sonee
Claude-3-Opus
Method
XSum
Writing
PubMed
XSum
Writing
PuhMed
Mix3
Zero-Shol Detectors
Using Open-Source
LLMs
Likelihood (Neo-2.7)
0.8862
0.9484
0,8360
0.8661
0,9322
0.9734
0.8603
0.9030
Enrropy (Neo-2.7)
0.4146
0.2156
0.2989
0.3466
0,3871
0.1792
0.2910
0.3265
Rank (Nco-2.7)
0.7019
0.7812
0.6017
0.6888
0.7335
0.7950
0.6080
0.7056
LogRank (Neo-2.7)
0.8867
0.9401
0.82%
0.8654
0.9357
0.9679
0.8508
0.9042
DNA (Ne7
0.8558
0.9415
0.7647
0.7080
0.9424
0,9653
0.7326
DetectGPP (T5-1 IB/Neo-2.7)
0.8150
0.8675
0.7347
0.7967
0.7718
0.8335
0.7752
0.7776
Fast. Detect (GPT 2.7)
0.9514
0.9763
0,8634
0,92600
0,9779
0.9832
0.8947
0.9468
Fasi-IRettst 76:2::176
0.1336
T.6773
T0.7
0.69957
0.30000
077:15
0.732
T.73550
Fast- DetteD (Qw)n2,5-7B)
0.8595
0,8600
0,7346
0.7813
0.9097
0.8967
0.7572
0.8119
DesteDetect (Llama3-8B)
0.9243
0.7936
0.8212
0.9640
0.9377
0.8251
0.8510
Zero-Shot Detectors Using Proprictary LLMs
Likelihood (GPT-3.5)
0.8364
0.9918
0.9299
0.9023
0.9191
0.9955
0.9467
0,9295
DNA GPT (GPT:3.5)
0.7934
0.8587
0.8988
0,7871
0,9040
0.9362
0.8926
0.8383
Glmmpse Teiligg Geimeetr
Entropy (GPT:3355
0.4685
0.0565
0.1985
0.2582
0.4310
0.0381
0.1852
0.2339
Rank (GPT:3.5)
0.7801
0.9724
0.8476
0.8473
0.8263
0,9809
0.8524
0.8645
LogRank (GPT:3.5)
0.8502
0.9927
0.9265
0.9062
0.9302
0.9966
0.9336
Fast-Detect (Babbage)
0.9508
0.9705
0.9111
0.9438
0,9874
0.9865
0.9298
0.9698
Fast-Detect (Davinci)
0,9659
0.9939
0.9084
0.9606
0.9940
0.9946
0:9262
0,9742
Fast. Detect (.))
0.9433
0,9930
0,9552
0.9576
0,9899
0.9829
0,9686
0.9689
Fast-Detect (GPT-4)
0.9523
0.9910
0,9424
0,9623
0.9930
0:9917
0.9580
0,9817
Glampse uxxing Ziphan
Fast-Detect (GPT.3.5)
0.9455
0,9920
0,9510
0.9475
0,9914
0.9798
0.9588
Past-Detect (GPT-4)
0.9581
0.9889
0,9308
0.9613
0.9958
0,9901
0.9496
0.9792
Glimpse using MLP
Fast Detect (.))
0.9457
0.9925
0,9548
0.9526
0.9911
0.9811
0.9664
0.9634
Fast. Detect (GPT-4)
0.9574
0.9901
0,9382
0.9631
0,9953
0.9909
0.9543
0.9807
Table 5: A comparison of ACC in Glimpse and the major baselines on generations from CharGPT, where we employ the optimal threshold either for each dataset or across all three datasets. The smaller average drop scales on Glimpse methods indicate that Glimpse offers more consistent metric across datasets. In addition, we also assess optimal threshold across datasets and source models, which yields ACCs nearly identical (deviation less than 0.006 except Likelihood and DNA- GPT) to the optimal threshold across datasets.
Threshold per Dataset PubMed
Best Thresbold across Datasets
Drop
Method
XSum
Writing
Avg.
XSum
Writing
PuhMed
Avg.
Avg.
Fast-Detect (GPT-J/Neo-2.7)
0.9600
0.9633
0,8267
0.9167
0.9400
0.9367
0.7567
0,8778
-0.0389
DaxteDt (Phi2-2.7BB
0.7467
0.6967
0.7600
0,7344
0.6767
0.6967
0.7467
0.7067
-0.0277
Fast- Detect (Qwen2.5-7B)
0.7367
0.7600
0.7267
0.7411
0.6433
0.7600
0.6867
0.6967
.00444
Fast-Detect (Llama3-8B)
0,8000
0,7700
0.7267
0.7656
0.6900
0.7667
0.6567
0.7044
-0.0612
Likelibood (GPT-3.5)
0.8767
0.9933
0.8933
0.9211
0.7533
0.9900
0.8633
0.8689
-0.0522
DNA GPT (GPT:3.5)
0.8750
0.8592
0.8833
0,8725
0.8108
0.5986
0.7467
0.7187
-0.1538
Glimpse using Geometric
Fast-Detect Detect (Babbage)
0.9600
0.9433
0.9033
0.9356
0.9600
0.9100
0.8833
0.9178
-0.01 78
Fast-Detect (Davinet)
0.9667
0,9800
0.8867
0,9444
0.9333
0.9700
0.8633
0.9222
-0.0222
DesteDetect (GPT-3355
0.9633
0.9967
0.9200
0,9600
0.9033
0.9967
0.9167
0.9389
-0.0211
Fasst Detect (GPT-4)
0.9367
0.8933
0.9344
0.9367
0.9400
0.9200
-0.0144
D ABLATION STUDY
D. ABLATION ON RANK-LIST SIZE
The size M of the rank list is another important hyperparameter. We assess its effects on Geometric and Zipfian distributions (skipping MLP because it requires a heavy training process for each set- ting). As demonstrated in Figure g overall Glimpse with a larger size obtains a higher accuracy. Geometric distribution shows monotonic increasing trends, while Zipfian shows decreasing trends for Fast-Detect but increasing trends for LogRank, demonstrating an inconsistent pattern. Roughly, experiments with MLP on the sizes of 100 and 1000 suggest that has the similar pattern as Zipfian.
19

---

Published as a conference paper at ICLR 2025
Table The prompts that for the ablation, from the empty promptO simple promptl until complex prompt3 and prompt4. The changes are marked in italic.
Prompt
Content
prompt0
(Empty)
You serve as valuable aide, capable of generating clear and persuasive pieces of writing given certain context. Now, assume the role of author and strive finalize this article.
promptl
You serve as valuable aide, capable of generating clear and persuasive pieces writing given certain context. Now, assume the role of an author and strive to finalize this article. prompt2 I operate an entity utilizing GPT the foundational large language model. I function in the capacity of writer, authoring articles daily basis. Presented below is example of an article have crafted.
System: You serve as a valuahle aide, capable of generating clear and persuasive pieces of writing given certain context. Now, assume the role of an author and strive to finalize this article.
prompt3
Assistant: I Loperate as an entity utilizing GPT the foundational large language model. function in the capacity of a writer, authoring articles on daily basis. Presented below is an example of an article have crafted.
Assistant: You
valuable aide, capable of generating clear and persuasive pieces of writing given a certain context. Now, assume role an author strive to finalize this article.
prompt4
User: I operate as an entity utilizing GPT as the foundational large language model. function in the capacity of writer, authoring articles a daily basis. Presented below is example of an article have crafted.
Zipfian
Fast Bebert(Babbagel
Fast Detect 3.5)
Fast Detect 4)
1.0
1.0
AUROC
0.9
0.8
0.7
0.6
100
300
500
700
900
100
300
500
700 900
Rank List
Rank List
Prompt
Prompt
Prompt
Figure 8: Ablation on rank-list size, where the AUROC is averaged across the three datasets produced by GPT-4.
Figure 9: Ablation prompt, where the AUROC is aver- aged across the three datasets produced by GPT-4. GPT-4 is most sensitive to prompts.
D.2 ABLATION ON PROMPT
Large models are sensitive to context, necessitating a suitable prompt for optimal detection ac- curacy (Taguchi et al., 2024). We analyze the prompts featured in Table to ascertain their effect. We draft the prompts manually, starting with prompt3. Then we replace the 'System' and 'Assistant' roles with 'Assistant" and "User' to produce prompt4 and we remove the roles to produce prompt2. We simplify prompt2 by removing its second paragraph to produce prompt1 and removing all con- tent to produce empty promptO. In this study, we only experiment with several manually drafted prompts, leaving a systematic exploration of the prompts for the future.
As Shows, GPT-4 is the most sensitive model among the four scoring models, with detection accuracy fluctuating between 0.7289 (promptO) and 0.9682 (prompt4). contrast, GPT-3.5 less sensitive, with detection accuracy increasing from 0.9071 (promptO) 0.9589 (promptl), but maintaining stability for the rest. The base model Babbage and Davinci are less influenced by the prompt, and do not show them in the figure.
20

---

Published as a conference paper at ICLR 2025
Table 7: Robustness across source models measured in ACC, where we evaluate all source models using a threshold determined according to ChatGPT. All experiments are run on Mix3.
Claude-3
Gemini-115 Pro
Method
ChatGPT
GPT-4
Sonnet
Opus
Avg.
Fast-Detect (GPT:J/Neo-2.7)
0,8778
0,7933
0.8389
0.8822
0.6867
0.8158
Fast. Detect (Phi2-2.70)
0.7067
0.5500
0.6378
0.6622
0.6011
0.6316
Fast-Detect (Qwen2.5-7B)
0.6967
0.6222
0.6811
0.7011
0.6256
0.6653
Fast-Detect (Llama3-8B)
0.7044
0.6356
0.7244
0,7467
0.6844
0.7031
Takelihood (GPI.3.5)
T,5689
(T6066
033326
0036615
7.670
T17822
DNA-GPT (GPT-3.5)
0.7187
0.6176
0,6942
0.7540
0,6372
0,6843
Glimpse using Geometric
Fast-Detect (Babbage)
0.9178
0,7373
0.8461
0.9176
0.6562
0.8150
Past-Detect (Davinci)
0.9222
0.8085
0.7363
0.8555
Fast: Detect (GPT:3355)
0,9389
0.8655
0.8985
0.9197
0.8554
0.8955
Fast-Detact (GPT-4)
0,9200
0.7973
Table Robustness across domains measured ACC, where we cross-validate each dataset using a threshold determined according to other two datasets for each source model.
ChatGPT
GPT-4
Method
XSum
Writing
PubMed
Avg.
XSum
Writing
PubMed
Avg.
Fast-Detect (GPT-J/Neo-2.7)
0.8633
0,9300
0,7000
0.8311
0,7867
0,8833
0,6600
0.7767
Past-Detect (Phi2-2,7B)
0.6800
0.6967
0.7467
0.7078
0.4767
0.6267
0.5667
0.5567
(Qwen2.577B)
0.7300
0.6556
Fast-Detect
Detect (1 (1lama3-88) 8BB)
0.6567
0,7500
0.5767
0
0.5967
0.6933
0.5800
0.6233
Likelifood (GPT-3.5)
0.0933
00,833
0826
03535
0.5503
03020
0.8300
0.1214
DNA-GPT (GPT:3.5)
0,5000
0,5106
0,7067
0,5724
0,4966
0.5000
0.6933
0.5633
Glimpre using Geometric
Fast Detect (Babhuge)
0.9133
0,9100
0.8800
0
0.8154
0.8033
0,8333
0.8174
Fast-Detect (Davinci)
0.8767
0,9633
0,8500
0.8967
0.8087
0.7800
0.8033
Fast-Detect (GPT-3.55
0.8833
0.9967
0.8267
0,9022
0,7819
0.9396
0.8667
0.8627
Fast-Detect (GPT-44
0.9367
0.9367
0.8167
0.8967
0.9200
0.9195
0.8467
0.8954
E ANALYSIS AND DISCUSSION
E. ROBUSTNESS ACROSS SOURCE MODELS AND DOMAINS
In practice, we need to fix the decision threshold and detect text from various sources and domains. However, the distributions of the detection metric might be shifted between different sources or domains, resulting in high detection accuracy one but low accuracy in another. this section, we evaluate the robustness of Glimpse along with other strong baselines across different source models and domains.
Firstly, examine the detection accuracy (in ACC) for each source model utilizing an optimal threshold identified in the ChatGPT Mix3 dataset. As illustrated in Table 7 Glimpse consistently provides the highest ACCs across all source models. While the accuracy of and Fast-Detect (GPT-4) fluctuates between source models, their overall ACCs are closely matched. Fast-Detect (GPT-3.5) delivers the highest ACC, which is approximately 8 points above the top baseline. This demonstrates the stability of Glimpse across numerous source models.
(GPT-3.5)
Subsequently, we assess the accuracy on each dataset using an optimal threshold established on the remaining two datasets for each source model. As exhibited in Table 8 we employ the source models of ChatGPT and GPT-4 examples. Glimpse also delivers the highest ACCs on all datasets, further evidence of its robustness across various domains.
E.2 ROBUSTNESS UNDER PARAPIRASINGG ATTACK
We assess the performance of Glimpse uder paraphrasing attack, utilizing DIPPER (Krishna et al. 2024) to rephrase the output generated by ChatGPT, Our testing encompasses two paraphrasing settings: high lexical diversity (60 L) and high-order diversity (60 O). As indicated in Table 9 Fast- Detect (Babbage) surpasses Past-Detect (Neo-2.7) in both settings, but is more significant influenced by diverse lexicons than by diverse orderings.
21

---

Published as a conference paper at ICLR 2025
Table 9: Robustness under paraphrasing attack with diverse lexicons and orders, where we report the TPR (%) at an FPR level of 1%.
ChatGPT DIPPER
ChatGPT DIPPER
Method
XSum
Writing
PuhMed
Avg.
XSum
Writin
PuhMed
Avg.
Fast-Detect (GPT-J/Neo-2.7
48.7
65.3
38.0
50.7
52.7
66.0
34.0
50.9
Fast. Detect (B7222277B)
6.7
15.1
8.0
10.0
D.7
0.7
1.8
Fast-Detect (wenn.5-7B)
16.0
35.3
8.0
19.8
5.3
10.7
7.3
Fast-Detect (Llama3-8B)
34,0
50.0
13.3
32.4
17.3
12.7
10.7
13.6
1.lkelihood (GPT-3.5)
11.3
6.9
Glimpse using Geometric
Fast-Detect (Rabbage)
64.0
58.7
39.3
54.0
83.3
80.0
68.4
Fast-Detect (Davinci)
16.7
48.0
28.7
31.1
51.3
88.7
33.3
57.8
(GPT-3.55
However, also note unusual behavior DIPPER on XSum, where the diverse lexical paraphras- ing results in a surprisingly low detection accuracy for Likelihood (GPT-3.5) baseline. This is caused by the atypical trend that the paraphraser replaces common words with rare expressions, thus reduc- ing the readability of the content. This odd distribution could be the cause of the unusually low accuracy of Fast-Detect (GPT-3.5) on XSum (60 L.. Glimpse outperforms the baselines with open source models, proving its efficacy in withstanding paraphrasing attack.
Additionally, we find that larger LLMs are more susceptible to increased lexical and order diversity. Nonetheless, our review of the paraphrased articles indicates that this increased diversity also lowers the readability due to excessive use of unusual words and sequences, implying that there are some drawbacks of the attack.
22


Reviews:
[{"review_id": "vqC1hfUFnP", "reviewer": "Reviewer_32Jz", "summary": "In this paper, the authors consider the task of artificial text detection (ATD); one commonly-used approach here is to use a model itself to detect its own output. This scheme is inapplicable to modern proprietary LLMs because they don’t give access to their internal states. To address this issue, authors introduce an extension to existing white-box detection methods that rely on the distribution of model-assigned probabilities to each token. The authors propose to substitute token probability distributions by their density estimations (PDE) based on probabilities of the few most probable tokens and a hypothesis on the shape of the rest of the distribution. They test various candidates for the hypothetical token probability distributions. On example of Fast-DetectGPT authors illustrate the scheme of PDE-approach application to ATD solutions. Authors perform numerous experiments on different types of texts produced by various generating models and show the high performance of their method. They also perform ablation study, investigate the effect of different prompt designs, and explore the effect of paraphrasing adversarial attack.", "questions": "**Q1.** DetectGPT (and many other methods) has a known limitation of a drastically high rate of false alarms on texts written by people who are not fluent in English; for example, by undergraduate students preparing for the IELTS exam [1]. Have you considered such scenarios?\nThis issue becomes more and more crucial as AI-content detection tools become part of automatic anti-plagiarism systems in education in countries where English is taught as second-language.\n\n> [1] W. Liang, M. Yuksekgonul, Y. Mao, E. Wu, and J. Zou. GPT detectors are biased against non-native English writers. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models.\n\n**Q2.** How big is the difference in performance between your method (PDE + Fast-DetectGPT) and the original Fast-DetectGPT on various data?\n\n**Q3.** Table 3 states that you use RoBERTa-large model as a baseline for experiments on robustness over languages. This is incorrect because RoBERTa was trained only on English-language data. Did you mean XLM-RoBERTa- large, which is actually multilingual, there?\n\n**Q4.** Could you please specify in more detail the experimental setup for assessing the robustness against paraphrasing attacks (Appendix E.1)?\n\n**Q5.** What are the limits for cross-model transferability of your method? (i.e, the proposed method works well when transferred from ChatGPT to GPT-4 and vice versa, but what would happen when it faces older models like GPT-2 or XLNet as source models?)\n\n**Other remarks:**\n\n-\tTable 2 caption: instead of “average accuracies”, there should be “average AUROC”. In general, please revise your usage of the phrase “detection accuracy”, because right now it is difficult to understand whether you mean Accuracy (that is not reported by value anywhere) or AUROC.\n\n-\tlines 485/511: technically, PHD is a \"black-box\" method. It does not require the knowledge of what model was used to generate text; it uses inner embeddings from a fixed auxiliary encoder transformer.\n\n-\tCode to reproduce the experiments was not provided (this could be done in attachments to submission in or via an anonymous Github repository).\n\n-\tPlease fix lettercases in paper titles in the references (e.g., “Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature” should be “Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature” in line 552; and so for other articles).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**S1.** The proposed approach shows promising results at increasing the efficiency of previously existing Artificial Text Detection methods against modern generating models.\n\n**S2.** The text of the article is well-written and was easy to follow; extensive mathematical explanations are given.\n\n**S3.** Proposed method is robust against some common adversarial attacks (e.g., automatic paraphrasing of generated texts).", "weaknesses": "**W1.** The proposed method still requires the internal information from the source models – model-assigned probabilities of K most probable candidates for each token. Some proprietary models might not provide such information.\n\n**W2.** The novelty of the proposed method compared to `vanilla’ Fast-DetectGPT is somewhat ambiguous. Essentially, this work proposes to compute statistics not from probability distribution over the entire dictionary (as in Fast-DetectGPT), but from probabilities of K most likely tokens and some assumptions on the distribution. \n\n**W3.** Datasets used for experiments are small: 150 pairs of human-written/AI-generated texts each. In particular, there are concerns regarding the generalization capabilities of the proposed. Also, given data is not large enough to properly train the baseline methods (RoBERTa).\n\n**W4.** AUROC is the only quality metric reported. It has a number of limitations, and alone it is not enough to accurately compare different models. This experimental setup does not allow properly accessing the generalization capabilities of the model and discriminating different types of their biases. \nFor reference, one of the common practices for Zero-shot AI-content detector evaluation is to set a threshold (by fixing FP-rate) on some 'training’ data (i.e., the probability of falsely declaring human-written text as AI-generated) and then report metrics (TPR/TNR/detection accuracy/etc.) on the `evaluation’ data.\n\n**W5.** It is unclear how good this method would generalize across different styles of texts. As it follows from the text, all reported experiments were performed only in “in-domain” setup. Datasets used in this work are very small and style-specific (e.g., scientific-like *PubMed* and free-form fictional *Writing*). It is well known that some classifiers (like RoBERTa-based) excel on `familiar’ data but experience a significant drop in performance when they face texts of styles or genres unseen during training. From the presented results it is impossible to say whether the proposed method will fall into the same pitfall or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lrl9G1TweZ", "reviewer": "Reviewer_Gnc2", "summary": "The paper considers the topic of detecting the texts, generated by closed (proprietary) LLMs. Authors suggest several ways to approximate the distribution of token probabilities of those closed LLMs accessible via API (such as GPT-4), knowing only the probabilities of top-K or even top-1 tokens. Using these distributions, authors managed to emulate several methods of LLM-generated text detection (i. e. Fast-DetectGPT, Entropy, etc) as if they had access to these LLMs. Using these methods upon approximated distributions, they overperformed analogous methods based on distributions of GPT-Neo-2.7B.", "questions": "- Line 138-140, Equation 2: why do you call this $d(x, p_\\theta)$ value \"curvature\"? It doesn't seem to have a meaning of a curvature. Maybe \"skewness\" would be better word?\n- Only AUROC is reported as a metric for comparison in the Tables. It would be interesting to see, what are the accuracy scores for all methods (for some fixed threshold)?\n- Are there any differences between \"Likelihood\" method and perplexity scores?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The topic of LLM-generated text detection is important.\n- The idea of the paper is worth considering.\n- Authors check how their detectors behave when they fix a low False Positive Rate, that is a good practice.\n- Authors check their detectors against paraphrasing attacks and consider the influence of various prompts.", "weaknesses": "The major drawbacks of the paper lie in its experimental setup.\n- The amount of machine-generated texts in the datasets in Table 2 (which is supposed to contain the main results) is extremely small. E.g., as far as I understood from part 3.1 SETTINGS, for each sub-domain \"XSum\", \"Writing\", and \"PubMed\" in Table 2, only 150 machine-generated texts vs 150 human-written texts are used. It's hard to tell anything for sure from such a small amount of examples, and the results may be very noisy. The authors need to increase their datasets to get reliable results.\n- The choice of baseline open LLM (i.e. GPT-Neo-2.7B) for applying methods like Fast-Detect and Likelihood seems unreasonable because GPT-Neo-2.7B is weak compared to the newer open LLMs of comparable size. I would recommend adding at least the Phi-2 model (it also has 2.7B parameters) as a base for Fast-Detect, Likelihood, and other similar methods because its perplexity scores seem to be good in LLM-generated text detection (AI-generated text boundary detection with RoFT, Kushnareva et. al, 2024). Besides, many bigger models look like fair candidates for a good baseline (e.g. LLaMA 3.2 and many others).\n- AUROC scores in the Tables seem to be very high both for the best baseline and for the best method suggested by the authors. It means that the task statement is too easy. It would be good to add more complicated data, where all the methods will struggle, and the difference between methods will become more clear.\n\nMinor remarks about the presentation:\n- Overusing the appendices: appendices D.1 and E look more suitable for the main text;\n- Typos: log p(x|x) in Figure 1 and Equation (2);\n- Figure 1 contains an extremely bright red color, it is hard to look at it. Please, change it to a softer color.\n\n**UPDATE**: The first two of my concerns were addressed, so I updated my main score from 3 to 5. I adjusted scores about \"soundness\" and \"presentation\" accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7K0YTcW9t3", "reviewer": "Reviewer_NxVa", "summary": "The paper proposes Probability Distribution Estimation (PDE), a method for enhancing the capabilities of white-box text detection methods, such as Fast-DetectGPT, in identifying machine-generated text from proprietary large language models (LLMs) like GPT-4 and Claude-3. PDE extends traditional white-box methods to proprietary models by estimating full probability distributions from limited API access. The authors demonstrate that using PDE with Fast-DetectGPT achieves improvements in detection accuracy over previous methods.", "questions": "- Did you try to Fast-Detect with other open models rather than GPT-J/Neo-2.7? Could you please scores from newer/larger models for better comparison?\n- The evaluation dataset is very small; did you try other datasets? I believe it would be better to perform testing on bigger datasets, for example the original M4. \n\nSmall remarks:\n- line 101-102: \"(↑ more than 25%)\" might confuse readers. Better to add \"relative to the remaining space\"\n- \"*\" in equations looks strange\n- line 297: gpt-35-turbo -> gpt-3.5-turbo", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- A thorough explanation of the PDE approach, describing its implementation across different parameterization methods (e.g., Geometric, Zipfian).\n- Extensive ablation studies and analyses (distribution choices, top-K probabilities, language variations). This adds depth and clarity to the method’s application across proprietary LLMs.\n- Completing datasets with new models that authors will release", "weaknesses": "- The evaluations are conducted on limited datasets (150 pairs), potentially restricting the findings’ generalizability across more diverse or larger data sources. There are more samples for other models in the original M4 dataset.\n- While PDE is an effective enhancement, it primarily focuses on density estimation and integration with existing detection methods rather than introducing a fundamentally new detection approach.\n-  Although PDE improves upon Fast-Detect, the accuracy increase is relatively modest, raising questions about the practical advantage of this added complexity in real-world applications. In table 2 you report Fast-Detect (GPT-J/Neo-2.7) which has relatively good. It hints that it may be better to use bigger or newer open models (like Qwen2.5-1.5B or Llama3) than to use PDE to proprietary models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "76nlrmv33V", "reviewer": "Reviewer_qGG2", "summary": "To enable existing white-box methods to proprietary models, the authors propose Probability Distribution Estimation (PDE), extending methods like Entropy, Rank, Log-Rank, and Fast-DetectGPT to latest closed-source LLMs. This work uses only the top-K probabilies returns by the API to estimate the necessary metrics (such as mean and variance in DetectGPT). They estimate the probs after rank K using parametrized distribution (which token each pk refers to is not important). Three algorithms are tested: Geometric, Zipfian, and MLP.\n\nIt is applied to ChatGPT (gpt-35-turbo) (OpenAI, 2022), GPT-4 (gpt-4) (OpenAI, 2023), Claude3 Sonnet (claude-3-sonnet-20240229) and Opus (claude-3-opus-20240229) (Anthropic, 2024), Gemini-1.5 Pro (gemini-1.5-pro). The results is quite promising, and suggests larger LLMs can be universal detectors.", "questions": "Line302 : I did not understand what's the difference between source model and scoring models… My current understanding is that the authors want to argue that a good scoring model can detect machine-generated text from a wide range of LLMs, is my understanding correct? I think if you can add some text to explain that will be helpful.\n\nFigure1: I think the notation p(x|x) is weird.\n\nLine293: What do you mean by \"generate corresponding text from source LLMs for each sample\"? Do you mean generate text with the same prefix?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the idea of estimating unknown probs using Geometric, Zipfian, and MLP is very interesting.\n\nThe idea can be applied to different white-box methods.\n\nThe performance are quite promising.", "weaknesses": "Comparing to this proprietary and estimation approach, why not use open-source large model such as LLaMA as the scoring model? This paper did not mention models like LLaMA at all. If authors can give me a convincing answer or add some comparison experiment, I'll consider raise score.\n\nIt still assumes some partial information from the API. But I do agree this setting is realistic and interesting.\n\nSome parts of the writing can be improved (refer to my questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)