Language Model Detectors Are Easily Optimized Against
Authors: Charlotte Nicks
Eric Mitchell
Rafael Rafailov
Archit Sharma
Christopher D Manning
Chelsea Finn
Stefano Ermon
Venue: ICLR 2024 poster (2024)
Keywords: detector
language model
learning from preferences
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We show that existing open source and commercial LLM detectors can be used as reward functions to produce much more difficult-to-detect language models.

Abstract: The fluency and general applicability of large language models (LLMs) has motivated significant interest in detecting whether a piece of text was written by a language model. While both academic and commercial detectors have been deployed in some settings, particularly education, other research has highlighted the fragility of these systems. In this paper, we demonstrate a data-efficient attack that fine-tunes language models to confuse existing detectors, leveraging recent developments in reinforcement learning of language models. We use the `human-ness' score (often just a log probability) of various open-source and commercial detectors as a reward function for reinforcement learning, subject to a KL-divergence constraint that the resulting model does not differ significantly from the original. For a 7B parameter Llama-2 model, fine-tuning for under a day reduces the AUROC of the OpenAI RoBERTa-Large detector from 0.84 to 0.63, while perplexity on OpenWebText increases from 8.7 to only 9.0; with a larger perplexity budget, we can drive AUROC to 0.30 (worse than random). Similar to traditional adversarial attacks, we find that this increase in 'detector evasion' generalizes to other detectors not used during training. In light of our empirical results, we advise against continued reliance on LLM-generated text detectors. Models, datasets, and selected experiment code will be released at https://github.com/charlottttee/llm-detector-evasion.

Published as a conference paper at ICLR 2024
LANGUAGE MODEL DETECTORS ARE EASILY OPTIMIZED AGAINST
Charlotte Nicks, Eric Mitchell, Rafael Rafailov, Archit Sharma, Christopher D. Manning, Chelsea Finn, Stefano Ermon Stanford University cnicks1 Micks13sstafforor
ABSTRACT
The fluency and general applicability of large language models (LLMs) has mo- tivated significant interest in detecting whether piece of text was written by a language model. While both academic and commercial detectors have been de- ployed in some settings, particularly education, other research has highlighted the fragility of these systems. In this paper, we demonstrate data-efficient at- tack that fine-tunes language models to confuse existing detectors, leveraging re- cent developments in reinforcement learning of language models. We use the 'human-ness' score (often just log probability) of various open-source and com- mercial detectors as a reward function for reinforcement learning, subject to a KL-divergence constraint that the resulting model does not differ significantly from the original. For a 7B parameter Llama-2 model, fine-tuning for under a day reduces the AUROC of the OpenAl RoBERTa-Large detector from 0.84 0.63, while perplexity on OpenWebTexw increases from 8.7 to only 9.0; with a larger perplexity budget, we can drive AUROC 0.30 (worse than random). Similar to traditional adversarial attacks, we find that this increase in 'detector evasion' generalizes other detectors not used during training. In light of our empirical results, we advise against continued reliance on LLM-ggnerated text detectors. Models, datasets, and selected experiment code will be released at https://gitts com/ htt.r//rrrttttttteeeeeeeeeeeeeee
detector-
INTRODUCTION
Large language models (LLMs) can produce high-quality text in a wide variety of settings (Brown al., 2020; Bubeck al., 2023). Access such powerful LLMs has expanded rapidly; for anyone hoping to generate machine-writte wwrtten text, a plethora of free and low-cost options exist. The usage of such models has become endemic classrooms, news outlets, social media platforms, and other domains. This rapid development has led to several objections to widespread use of LLMs, including moral qualms with data procurement, questions regarding the quality of machine-generatee text, issues with LLMs outputting inaccurate information (hallucinations), and fear that the availability of LLMs may force people a variety of roles to reevaluate their day-to-day practices.
These concerns have led to significant amount of research and commercial product offerings for detecting machine- generated text (e.g. Gehrmann et al., 2019; Solaiman et al., 2019; Mitchell al., 2023). While the proliferation of such detection techniques may plausibly quell such fears in the short term, can we continue to rely on this paradigm to detect machine- machine-generated texts?
Existing detectors typically yield scalar score. An adversary might therefore hope to fine-tune language model to optimize this score, such that the model outputs are less detectable. By explicitly training to generate samples that confuse the classifier, the adversary could amortize the process of finding an adversarial attack: the fine-tuned LM would directly produce them. Text generation fundamentally discrete, non-differentiable domain, directly; however, reinforcement learning (RL) can provide a framework for fine-tuning LLMs to optimize such blackbox scores.
cannot optimize for such an attack
This motivates several questions we study in this work: Can RL methods be used for making LLMs less detectable? How does detectability trade-off with other metrics like perplexity? How does

---

Published as a conference paper at ICLR 2024
Paired preference data
Llama-7b
LM Detector
(l.e,, which response "more human'))
-ecoonomic forecast projects globel growth 3.7% for...
The IMF announced its latest economic
The announced that its latest
forecast projects global growth..
p.pooeections, amicst growing global uncertainties due to...
The IMF announced its latest projections, amidst growing global...
Data
Generation
Learning from Preferences (e.3, Direct Preference Optimization)
Stealth Llama
Detector-evasion
Fine-tuning
Figure I: We present results of a red-tes teaming effort that optimizes against machine-generated text detec- tors. Top: We first generate preference dataset that "prefers" generations that are more human according to the detector. Bottom: We fine-tune the model optimize these preferences.
detectability scale with the size of preference datasets (or equivalently, with query budget to a de- tector), and is it feasible to train popular L.Ms be undetectable against commercial detectors a limited budget? Does training against one detector reduce detectability under other detectors?
To study these questions, consider simple RL pipeline that uses direct preference optimization (DPO) (Rafailov et al., 2023). We use the score from the detector to construct a pairwise preference dataset where the completion with the higher 'humanness' score is marked as preferred, and the LLM is fine-tuned on this dataset using DPO. See Figure for an overview of this data gettrrainn and training pipeline. In contrast prior work, such an approach does not require the use of human paraphrasers or paraphrasing model (Krishna et al., 2023) and adds no overheard during inference.
The main contribution of this paper a detailed empirical study on the ease of evading language model detectors by optimizing against them. Our experiments find that a simple DPO-base pipeline produces consistent reduction detectability against various detectors. We can achieve AUROC metrics below 0.5 against several strong public and commercial detectors, indicating worse than random chance detector performance on the fine-tuned model and close to random chance perfor- mance several additional detection algorithms the cost of only small increases in perplexity.
Moreover, in many cases optimizing against one detector yields a model that also less detectable under other detectors. In particular, find that models pre-trained against the public RoBERTa- large achieve an average 0.15 reduction in AUROC when evaluated by a number of black-box commercial detectors. These results hold even at longer sequences, such generating essays, where our fine-tuned Llama-7b-chat model achieves RoBERTA-large AUROC of 0.26.
The results of red-teaming effort suggest that fine-tuning language models be less detectable both easy and cheap, which makes it feasible for a wide slate of malicious actors, even against the best open-source and commercial detectors available. Based these results, we advise various stakeholders (educators, policymakers etc) against reliance on the current suite of text-detectors, and suitably account for less-detectable LLM generated text.
2 RELATED WORK
Machine-generated text detection methods often either train classifier using a dataset of LM- and human-generatee text (Bakhtin et al., 2019; Solaiman et al., 2019; Uchendu al., 2020; Ippolito et al., 2020; Verma et al., 2023) detect zero-shot by leveraging the suspected language model or a proxy (Solaiman et al., 2019; Gehrmann et al., 2019; Mitchell et al., 2023; Su et al., 2023). Prior works have called into question the robustness of these detectors, finding that detectors susceptible to paraphrasing attacks (Sadasivan et al., 2023; Krishna et al., 2023) and can perform poorly for text written by non-native speakers Liang al. (2023).
Further, Mitchell et al. (2023) and Mireshghallah et al. (2023) show that zero-shot detectors show significantly reduced performance when the generating model is not known. By showing that is
2

---

Published as a conference paper at ICLR 2024
straightforward to optimize against current detectors, results complement these prior studies, while continuing to suggest that machiite-enerated text detectors are not robust.
Another class of works have aimed to train language models that produce subtle "watermarks," i.c. indications that they were generated by a machine (Kirchenbauer et al., 2023; Zhao et al., 2023; Kuditipudi et al., 2023; Yoo et al.,, 2023). However, the premise of watermarking relies the fact that all strong models in the LLM ecosystem are watermarked (i.e.. hosted behind APIs that enforce watermarking): a single strong LLM with freely-available weights violates this threat model. We consider a stronger threat model where an adversary is fine-tuning the model to be undetectable.
Finally, Solaiman et al. (2019) train detector discriminate between human samples and samples generated by a pre-trained model (GPT-2). In our case, fine-tuning that pre-trained model to max- imize the 'human' probability of the detector with DPO (Rafailov et al., 2023) is very similar to performing one round of generator improvement in generative adversarial network (GAN; Good- fellow et al. (2014)). While adversarial objectives are typically avoided for text data due to the dif- ficulty of differentiating through the discrete sampling step, Yu et al. (2017) and Fedus et al. (2018) show that GAN language models can produce more realistie-looking samples than MLE when eval- uated by humans. Their results provide some precedent for our GAN-like training procedure for our use case of generating numan-lookinn samples.
OPTIMIZING AGAINST LANGUAGE MODEL DETECTORS
We leverage recent advantages in fine-tuning language models with reinforcement learning di- rectly optimize for detector confusion. First, we review key aspects of existing methods that we will leverage for detector evasion. Then we will outline our pipeline for optimizing against detectors.
Reinforcement Learning for Language Modelling, We consider language model Te that is conditioned on a prompt x and ruto-regressively generates an output sequence y. Reinforcement learning has recently emerged as a powerful technique for aligning pre-trained language models to a particular objective, which is expressed through a reward function r(x. y) that assigns higher rewards to more desirable responses. In practice, the most commonly-used objective includes KL- divergence penalty between the language model and its initialization, which in combination yield
max Dp....ppp. The
[r(z. y) - 3L [T x)) || Ter(y x)]].
(1)
where Do is some dataset of prompts, net is the reference model, usually the pre- pre-trained language model before the RI. fine-tuning phase, and /3 is a coefficient that controls the trade-off between reward and divergence (Ouyang et al., 2022; Bai al., 2022; Stiennon et al., 2020). The objec- tive above seeks to align the model with the reward function, while not deviating too far from the pre-trained reference model. This KL-constraint is used in part because, without proper regular- ization, reward maximization alone often leads to 'reward hacking' (Skalse et al., 2022) or "over- optimization" (Gao et al., 2022); our experiments verify that overoptimization can be an issue in our setting as well, justifying the KL-constrained formulation.
Several algorithms have been developed optimize the objective. the most widely used one being PPO (Schulman et al., 2017). However, these algorithms are quite complex, depending on wide set of parameters and often being unstable to train (Zheng et al., 2023).
Direct Preference Optimization (DPO). Rafailov et al. (2023) recently proposed the DPO algo- rithm with the goal of enabling simpler, stabler optimization the above KL-constrained objective in the case where the reward function is learned from a dataset of preference pairs. Assume
dataset of preference pairs D fx( (() yf ( including prompts x and two generations yo and In this notation Vhe is preferred over y1 (denoted yo > yi) and the probability is modeled as a Brarley-Terry model (Bradley & Terry, 1952), written as
p(y > yr) ((((,, Mwo - r(x,,((,))) vr))),
where a is the standard logistic function for some reward function r(x, y). The Direct Preference Optimization algorithm (Rafailov et al., 2023) shows that the exact optimal policy n² for the problem in Eq. I can be directly optimized through the MLE objective
T ( (x)
To (y
( Tet = - (o. log a
(9 log
B log
(3)
Teer (War x))
Teer (31 xx)
3

---

Published as a conference paper at ICLR 2024
Detector Trained Against
None
RoB-Ig
RoB-base
Log Prob
Log Rank
DetectGPT
DetectLLLM
Perplexity
8.7
9.0
8.9
9.0
9.7
9.5
9.5
RoB-Ig
0.84
0.63
0.68
0.86
0,58
0.87
0.89
Eval Detector
RoB-base
0.78
0.55
0.53
0.78
0.47
0.79
0.82
Log Prob
0.69
0.59
0.32
0.55
0.52
0.58
Logg Rank
0.75
0.67
0.65
0.42
0.61
0.59
0.63
DetectGPT
0.81
0.80
0.70
0.81
0.47
0.55
DetectLLM
0.82
0.83
0.83
0.77
0.84
0.54
0.61
Median AUROC
0.80
0.65
0.67
0.74
0.60
0.53
0.62
Table 1: Cross-detector generalization for open-source detectors. For each detector, train models with different KL constraints (3 {0.05, 0.5, 5}). We report perplexity (on OpenWeTTew and AUROC for the strongest resulting model (i.c., lowest AUROC) within perplexity budget of 9.7, or an increase of 1.
This objective allows us to optimize with a classification loss and very few hyper-parametee and design choices, resulting in stable training. We therefore adopt DPO as our optimization algorithm.
We note that we also attempted using detector scores directly as rewards in Eq. 1 and optimizing with general purpose reinforcement learning algorithms, but found performance be much worse.
Preference Data Generation and Optimization for Detector Evasion. Our goal is to understand how easy
fine-tune a pre-trained language to be undetectable by current machine-genenn -ennrrrte text detectors. In order to apply DPO to this detector-evasion setting, we construct a preference dataset using samples generated by Llama-2-7B. We generate a pair of samples yd"), ,(", for each prompt (x()) in the dataset, using temperature 1.0. Preference labels are generated by comparing the detector's 'human-ness score s(x, y) for pair of responses, and assigning the label y) y(i)
s(x, () > s(x, ,0());; otherwise we have g > y(a), Generating preferences through hard thresholding previously adopted in (Gao al., 2022) and also found to perform well in our experiments, so we keep this configuration for convenience. Once we have generated the preference dataset, fine-tune Llama-2-7B using the DPO objective in Eq. 3.
EXPERIMENTS
We conduct wide variety of experimental evaluations in order to understand the extent to which optimizing against detectors is feasible and cost-effective, In Section 4.1, we investigate the extent to which training against one detector provides evasion from other detectors, using both open-source and commercial detectors available only through APls. Section 4.3 studies how many queries to a detector are necessary to collect dataset sufficient for evasion. We explore detector evasion in language models fine-tuned for dialogue in Section 4.4, which explores evasion using off off-policy data in a case study in essay writing. Section 4.5 provides additional experiments in the setting of creative writing. See Appendices B, C.I, and D for a study of the impact of base model size on evasion, study of the impact of sequence length on detection, and human evaluation of model sample quality, respectively.
Models. Our experiments use the open-source 7B parameter Llama-2 2-bane model pre- me-trained large, diverse text dataset (Touvron et al., 2023) Sections 4.1.4.3 and the corresponding Llama-2- chat model, which is fine-tuned with reinforcement learning from human feedback Christiano al. (2017); Ouyang et al. (2022) in Section 4.4. For the off-policy data experiment in Section 4.4, we sample demonstration responses from 3pt-3. 5-turbo----- 0613. We sample the model for up to an average of approximately 120 tokens for Sections 4.1 and 4.3, approximately 120 or 240 tokens in the sequence length ablation in Appendix C. approximately 250 tokens for the essay generation experiment, and approximately 250-500 tokens for the creative writing case study.
Datasets. In Sections 4. 1-4.3, we generate texts for detection that continue short 8-token prefixes of Open WebText documents (Gokaslan & Cohen, 2019). For the experiments with chat-tuned models
'The number of prompts varies; typically it is on the order of 10k. The specific value is noted wh where relevant. wwtp.//w...ooooo///o///gggggo.e.
4

---

Published as a conference paper at ICLR 2024
Detector Trained Against
None
RoB-Ig
GPTZero
Winston
Originality
Sapling
Perplexity
8.7
9.0
9.5
9.6
9.7
9.3
Eval Detector
RoB-Ig
0.84
0.63
0.78
0.83
0.64
0.76
GPTZero
0.57
0.63
0.57
0.60
0.62
0.61
Winston
0.53
0.46
0.30
0.20
0.40
0.43
Originality
0,99
0.69
0.79
0.82
0,62
0.79
Sapling
0.75
0.76
0.77
0.78
0.75
0.74
Median AUROC
0.75
0.63
0.77
0.78
0.62
0.74
Table 2: Cross-detector generalization for commercial detectors. For each detector, we again train 3 models with different KL constraints (3 E (0.05, 0.5, 5)) and select the strongest in Table 1. We report the perplex- ity (computed OpenWetText) and the AUROC of each resulting model.
in Section 4.4's essay-generating case study, we use prompts from the Alpaca instruction dataset (Taori et al., 2023) for generic evasion tuning and a collection of free, human-written essay titles (along with model-generated essays) for essay-specific vasion-tuning (Vechtomov, 2023). For the creative writing case study, we use the Writing Prompts dataset (Fan et al., 2018). The protocol that we use to split the datasets is described in Appendix A
written prompts collected from Reddit and compiled in
Detectors. We conduct experiments with variety of open-source and commercial detectors. We use strong open-source detectors from prior work, including language models trained for detection such as the RoBERTaa-arge and RoBERTa-base models trained in Solaiman al. (2019) as well as 'zero-shot' detectors (Mitchell et al., 2023) such as Log Rank and Log Probability thresholding (based on the intuition that model samples are assigned higher probability by the generating model, and likewise lower average per-token rank in each conditional distribution). Finally, we include zero-shot perturbation- based methods DetectGPT (Mitchell et al., 2023) and DetectL.M (Su et al., 2023), which perform detection by measuring the change in log probability log rank of passage after it is perturbed (i.e., slightly rephrased). For text generated by a model, slight perturbations significantly decrease the likelihood of the text under the model, whereas perturbing human-written text tends not to change log probability significantly.
For zero-shot detectors, we make the assumption that the detector knows that samples are being generated by Llama-2 model; that is, for these detectors, we use Llama-2-base (the model we fine- tune) compute log probabilities and ranks. This configuration represents an optimistic case for the detector. In addition to open-source detectors, we train against four popular commercial detec- tors, GPTZero. Sapling, Originality.ai (model 2.0), and Winston AI. All commercial detectors advertise strong performance against widely-used LLMs.
4.1 EVALUATING GENERALIZATION DETECTOR EVASION ACROSS DETECTORS
In our first experiment, we study the basic question of the feasibility of optimizing language models against language model detectors without significantly harming sample quality. We fine-tune three Llama-2 7B models preferences computed from a variety of open source (Table 1) and commer- cial (Table 2) detectors; each of the three models uses a different parameter for DPO in the set, which corresponds to the strength of the KL regularization. We use 8 {0.05, 0.5, 5).
For each set of 3 runs, we select the run that produces the lowest AUROC on the detector was trained against (i.e., evades most successfully) such that the perplexity of the resulting model on OpenWebTexx increases by no more than This selection procedure produces a single model from each detector trained against; evaluate each model (and the original pre-trained model) against all detectors. We train each model for up to 30k preference pairs (selecting the lowest AUROC checkpoint 10k increments). For RoBERTa-large, RoBERTa-base, log probability, and log rank, we train for single epoch of 30k examples; for the more computationally expensive (DetectGPT, DetectLLM) or costly (commercial detector APIs) detectors, we train for up epochs of 10k preference pairs.
"hups://ppzero.mm/ https://sapling.ait http:://rigiinality.ii/ http://ssaiiiaaaaiiii/////ooiiir
http:////iiiiiiiiioaaaa/a/a////a htps://gowinston.ai
5

---

Published as a conference paper at ICLR 2024
Perplexity
AUROC
Model Sample
Abstract Neuroimaging evidence suggests that the two hemispheres may respond to stimuli somewhat differently the behavioural level. But whether this is general [...]
9.5
0.16
Abstract Neuroimaging evidence suggests that the human brain is capa- blc of detecting patterns at different scales with distinct networks in the cortex [..]]
0.5
10.0
0.14
Abstract Neuroimaging evidence suggests that the two "lost' British Army battalions at Singapore fell foul of Zener Cards Very minor 90th Anniversary yesterday of the [...]
0.05
29.8
0.06
Table 3: An insufficient KL constraint leads reward overoptimization and ultimately model degrada- tion. We compare a sample from each of the three models fine-tuned against GPTZero; for higher 3 (stronger KI. constraint), we see lower Open WeenWeeText perplexity and higher AUROC (though the AUROC is still well below 0.5). While lower 3 leads to even greater confusion of the detector (i.e., better reward exploitation), it leads nonsensical generations.
We find that optimizing against both open source and commercial detectors is effective, driving the AUROC of the detector trained against to 0.63 or below (i.e., close to random) in nearly every case (the exception being Sapling, which has a relatively weak AUROC of 0.75 even before optimiza- tion). addition, we find strong generalization between detectors when training against the OpenAl RoBERTa-large detector, DetecrGPPT/DetettL... is, intuitively, correlated for open-source detectors with similar mechanisms: RoB-Ig and RoB-base are both fine-tuned RoBERTa models; DetectGPT and DetectLLM both measure a discrepancy that appears after perturbing the given text. Each of these pairs displays strong transfer. Similary, among the commercial detectors, we note similar performance from the RoBERTa and Originality detec- tors; this result is perhaps unsurprising because Originality also uses a fine-tuned RoBERTa- like model for detection."
log rank detection, or Originality. Performance
4.1.1 DETECTOR STRENGTH AND TRANSFER
0.74
When examining transfer of detectors, we found that the models which achieved a very low AUROC against the de- tector against which they were trained (log probability and
AUROC
0.72
0.7
0.68
Winston, with AUROCs of 0.32 and 0.20, respectively) had very poor evasion against the suite of detectors whole (median detector AUROCS of 0.74 and 0.78, respec- tively), Another group of models did not achieve such ex- tremely low AUROCS against the detectors against which they were trained (RoB-Ig, log rank, and Originality, with AUROCS of 0.63, 0.61. and 0.62, respectively). However, these models did significantly better against the suite of detectors as a whole (median detector AUROCs of 0.65, 0.60, and 0.62, respectively). The model trained against DetectGPT was notable outlier from this trend, achieving stronger performance (median AUROC of 0.53, the lowest iof any model) under the suite of open-source detectors than its performance against DetectGPT itself (AUROC reduced to 0.47) would predict. The other 5 models in the open-source suite are shown in Figure 2.
66.66
0.64
0.62
0.3
0.35 0.4 0.45 0.5 0.55 0.5 0.65
AUROC Under Intended Detectors
Figure Optimizing against stronger detectors yields better transfer to other detectors. The lower the AUROC of the detector after optimizing against it (i.e. de- tector weakness). the higher the median AUROC the suite open source detec- tors. DetectGPT is ommitted, as an outlier.
Overall, these results suggest that the more easy it is to optimize against a model, the less useful this optimization is against other detectors. Conceptually, this implies that the following two measures of detector strength are consistent: First, a 'strong' detector is one that is difficult to optimize against. Second, a 'strong' detector is one that, when optimized against, produces strong transfer of evasion capacity to other detectors (i.e., the features it uses are common to any other accurate detector). Thus, we find that models fine-tuned against the strongest detectors (i.e., the ones for which decreasing the AUROC is most difficult) generalize best to other detectors.

---

Published as a conference paper at ICLR 2024
Beta
AUROC of RoBERTa Large
PPL
Fluency Win Rate
Offline Entropy
0.05
0.28
11.7
41.1%
0.1523
0.5
0.62
9.0
52.1%
0.1556
5
0.71
8.9
54.6%
0.1564
00 (base)
0.84
8.7
50.0%
0.1523
Table 4: Optimizing against RoBERTa Large with three different beta parameters. As the constraint to the original model loosened, generated quality and offline entropy increase then decrease while perplexity over human text monotonically decreases.
Perplexity on Human Text
pairs
12
Sk pairs
AUROC of RoBERTa-lan
0.7
pairs
11.5
25K pairs
pairs
10..5
0.5
9.5
0.3
8.5
20K
40k
60k
80k
100k
20k
40k
60k
80k
100K
Train Examples Seen
Train Examples Seen
Figure 3: Only a small number of detector queries are needed to produce a dataset large enough to confuse even a strong detector (RoBERTa-large). Just 10k queries (to create a dataset of 5k pairs) enables meaningful optimization against the detector within small perplexity budget; that is, gathering training dataset of detector
suitable for fine-tuning language model to evade the detector is fast and economical.
4.2 TEXT QUALITY AFTER FINE- TUNNNG
In previous sections, we consider perplexity over human text as a general metric of text quality. However, in order to examine how the quality of text generated by a model is impacted by this fine tuning process, we introduce two new metrics, GPT4 win rate and offline entropy. The win rate is the percentage of time that GPT4 believes that a generation by the fine-tuned model is more "fluently and coherently written" than a generation from the base model with the same prompt. The offline entropy is the model entropy, evaluated over human text to conserve compute, and is used to evaluate the diversity of model outputs. We compute these metrics for three 7-billion parameter Llama2 models trained against the large OpenAl detector, each with different beta parameter. The results can be found in Table 4.5
We note that fine tuning with our algorithm with a sufficiently strong KL constraint the base model actually improves both text diversity and quality according to GPT-4 while still evading the detector significant degree. the KL constraint becomes extremely loose, evasion is improved but text quality degrades. However, in a human evaluation in Appendix D, we find similar-sized effect in the opposite direction, suggesting model quality according to humans may have slightly degraded.
4.3 IMPACT OF DETECTOR QUERY QUOTA ON EVASTON
In real-world scenario, a malicious actor is likely to be constrained by the amount of paired prefer- ence data used training, especially if optimizing against commercial detector. We demonstrate that strong performance of detectors can be maintained even when training data is limited. We trained five llama2-7b models against OpenAl's large RoBERTa-basee detector, using / 0.5 for all models. Each model was trained for up to 100k steps different training set size (Ik, 5k, 10k, 25k, and 100k preference pairs). We observe several notable findings, presented in Figure 3.
First, detector evasion is possible with only small number (<10k) queries to the detector. For the commercial detectors we study, this number of queries typically costs less than $150 (in several
'The setup for this experiment is slightly different than the others presented in this paper. While most experiments test the post-evasion models by generating completions on different topic distribution than the topic distribution training data was generated from, this experiment trains and tests on the same distribution of topics. The subsequent dataset and model size ablations set up similarly.

---

Published as a conference paper at ICLR 2024
cases significantly so), making detector evasion very accessible procedure for small budgets. Second, while AUROC of the targeted detector generally trends downward with increasing dataset size. the perplexity of the evasion-tung model is non mon-monooniic In summary, these results suggest that preventing an adversary from collecting a dataset of detector evaluations large enough to train an undetectable model may be extremely difficult impossible.
4.4 GENERAL-PPRPOSS ESSAY GENERATION
To analyze the feasibility of optimizing against ex- isting detectors in domain more closely aligned with real-world usage, we explore evasion tuning in the context of generating essays. We ask two ques- tions: Can we evasion-tune chat model to gener- essays that confuse strong detector? and more challenging question Can we evasion-tune a general-purpose dialogue model that still success- fully confuses a strong detector in the context of es- say generation, but without gathering new detector- annotated preferences or lite-tunig specifically for essay generation? While an affirmative answer to either question is cause for concern, an affirmative answer to the second is much more serious: in this case, we do not need to evasion tune again for each new domain in which we would like to evade a detector, and further, in order to do so, we can re-use single set of preference data generated by another model (in this case, ChatGPT).
Source Model
Metric
Base
Essays
Dialogue
AUROC Perplexity
0.83
0.26
0.43
6.0
7.0
7.0
Table 5: A study in generating difficult-to- detect essays from Llama-7b-chat We perform detector evasion tuning on preferences generated by RoBERTa-large.
To answer these questions, we perform detector evasion on a LLama-2-chan 7B (Touvron et al., 2023), using two different datasets, specifically essay prompts and essays generated by Llama- 2-chat, and the other more general inttruction-oolloweee and preference data both models for 30k steps using 3 = 0.5; the Llama-generated preference samples are 250 tokens long; we prompt ChatGPT write a 'single mid-length paragraph' on the given topic, discarding samples less than 100 Llama-2 tokens.
prompts from Alpaca (Taori et al., 2023) samples from ChatGPT, rather than the Llama-2-chat model. We fine-tune
The results in Table show that optimizing against RoBERTa-large successful in both of these cases: fine-tuning a general-purpose chat model to evade a detector using general-purpe instruction following prompts and off-policy samples nonetheless can evade detector in the specific case of generating essays.
4.5 CASE STUDY: CREATIVE WRITING IN THE CLASSROOM
Finally, we evaluate this pipeline in setup mimicking a real classroom environment. We select the common classroom task of creative writing and the most robust commercial detector based on previous experiments, Originality. Purely creative activities have been recommended to teachers as a way to limit AI use. making simple attack on detectors in this domain particularly threatening.
We source the prompts and human completions from the Writing Prompts dataset (Fan et al., 2018), which consists of content sourced from Reddit's s/WritingPrompts subreddit. We use diitt--disttibution data generation with GPT4 to create training data that includes wide diversity Originality humanness scores. We trained the Zephyr 7b chat model, as its evaluations suggest it is comparatively very strong for its size (Tunstall et al., 2023). We found that the fine-tuned Zephyr chat model was able achieve significant reduction both in Originality's ' detector AUROC and in the true positive rate at a fixed low false positive rate of 1%, seen in Table 6. In practice, the true- positive rate only at a very low false-positive
Model
AUROC
TPR (@ 1 FPR)
Base
0.94
89%
Fine-Tuned I
0.85
32%
Fine-Tuned II
0.71
23%
Table 6: Fine-tuning small chat model for only hours to write creative essays while evading the Orig- inality detector acheives a significant reduction in AU- ROC. More importantly. it reduces the number of Al- generated essays the detector can eatch with fixed, low calse-positive rate by a factor of three after one round of DPO (fine-tuned I; similar to prior experiments). Fine- "Tmedd 4f génerates génnerttssgeerrrtsss a new preference dataset with the Fine-Tuned I model and performs DPO again.
"nttps://teachin resources del.a......
8

---

Published as a conference paper at ICLR 2024
rate is meaningful, rather than the AUROC. For example, a 1% false-positive rate produced enough false positives to convince Vanderbilt University to disable their Al detector entirely,'. After this first round of fine-tuning, the resulting model is capable of producing quality preference pairs, eliminating the need for out-of-disributtoon training data. We used this fine-tuned model as a base model for a second iteration of optimization The results can be found in Table 6.
DISCUSSION
This paper presented an empirical investigation on the ease of optimizing for language models that cannot be easily detected by current machine-generan generrted text detectors. Our investigation is motivated by the increasingly widespread use of large language models. While language models have legiti- mate uses in wide variety of applications, their use has also raised concerns that stem from both malicious intentions (e.g- cheating, disinformation) and the model's own shortcomings (e.g. gen- erating false statements in high-stakes domains like law and medicine). Because of these concerns, a number of methods have been developed to detect machine-generatee text, including commercial offerings. One company offers the "most trusted" detector, boasting 99.6% accuracy (Winston AI, 2023), while another claims to be "the gold standard in Al detection" (GPTZero, 2023). Users have therefore put more trust in these detectors than perhaps is warranted (Thompson & Hsu, 2023).
Our experiments expose a notable weakness of these detectors: it is straightforward to fine-tune model to evade these detectors while still maintaining high performance. The fine-tuned models produce text that almost completely undetectable by two out of four commercial detectors. For the other two commercial detectors, the fine-tuned models have an AUROC of less than 0.5, indicating that they produce text that is judged to be statistically more likely to be human than the human- written corpus itself. In a hypothetical scenario such an interaction between detector and an online misinformation campaign, relying on such text detector would actually hinder the verifica- tion efforts. Moreover, generating long-form text, such as essays, does not increase detectability.
We emphasize that the this training pipeline is straightforward and easy for adversaries to replicate. uses easily accessible public models and an open open-source training codebase. The entire data acqui- sition and training process cost few hundred dollars. For data, the process only requires limited, black-box query access to the detector with a budget of a few thousand prompts and does not need teacher models. For compute, we used widely available con-
human annotators, paraphrasing, sumer hardware and only a few hours of training time. We further expect that, with more extensive data, training, and resources, a fine-tuned model may be even
evasive.
We expect that this direct kind of attack is hard to protect against. Indeed, our results showed mean- ingful transfer between strong detectors, and as discussed in Appendix C, obvious countermeasures each come with a nontrivial downside. Thus, in light of these results, we argue that the current gen- eration of machine-genede gennertte text detectors is not robust to adversaries and may even favor machine- generated text over actual human-generated content. This includes both public detectors and closed black-box commercial ones. Furthermore, we argue that the problem of robust machine-generatee text detection may be unsolvable in practical settings. Any new detection algorithm can be subject to the adversarial training process in this paper. New detection algorithms will be rendered inef- fective by further model fine-tuning, which would then require the development of new detection algorithms. Hence, we argue against continued use of machine--enerateee text detectors.
"https://www www.wwawwdrbl..ediii... www.d//wii...eeeoorrgggee///////
guidance-on-a detecti -nd whw- wawlssallliininie iealiing-uuta i tina-at detector

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Evading language model detectors is type of red-teaming exercise that we carry out order to call attention the serious risks of relying on any machine-generaled text detection technologies. We categorically do not advocate for evading language model detectors for the purpose of carrying harmful activities with LLMs. Rather, we hope that in demonstrating the ease with which the effectiveness of existing detectors can be severely degraded, we can spur conversation about these technologies. Ultimately, believe swift action revise institutional norms, particularly standards in classrooms around student assessment, is warranted.
REPRODUCIBILITY
Section 3 covers the details of the direct preference optimization algorithm, and an open-source implementation is available in the cited paper. Precise descriptions of the fine-tuning and model selection process for non-chat models are available in sections 4.1, 4.3, and C.I, while the corre- sponding information for chat models can be found in section 4.4. A precise description of the data generation process be found in Appendix A.
Models, datasets, and selected experiment code will be made available at https: / com/ wwwwlhttttteeeeeeeeeeeceaaaaattt wewee//mdeeeeeeeeeeeeeeeeeeeeeee wwweecttorevviio.... evasion. We intend to release all models presented in this paper. More specifically, we intend to release:
All of the 7-billion-parann Daallmionnparmmtter Llama2 models presented in Table and Table 2.
The three variants of the --biliin-pprameee Ilama2 model optimized against RoBERTa Large with different KL-constraints to the baseline that are presented in Table 4.
The five variants of the 7-billion partmmeee Llama2 model optimized against RoBERTa Large with different training dataset sizes that are presented in Figure 3.
The two 7-billion-paran parrmmeeerr Llama2 chat models trained on essays and out-of-distribution (ChatGPT-generatee) dialogue that are presented in Table 5.
The 13-billion partmmetr Llama2 model presented in Table 7.
The 7-billion-parameter Zephyr Zephyr chat model presented in Table 6.
We additionally intend to release the datasets we annotated for this project. More specifically, we intend release:
A dataset of pairs of generations from the base associated detector scores and winner assigned by each of the open-source and commercial detectors evaluated in Table and Table
Llama2 model with the
For each model featured in Table Table 2, smaller dataset of its generations annotated by each detector.
A list of longer generations from the 7-billion-parameti Llama2 model trained against RoBERTa Large, annotated by RoBERTa Large.
The detector-annotated, ChatGPT-gennrated dataset used in dialogue training (see Table 5).
The detector-anmotated, Llama2-generated dataset used in essay training (see also Table 5).
7-billion-parameter A dataset of pairs of outputs 13- and 7-billion parameter post-evasion Llama2 model outputs, annotated with their GPT4-generated comparison.
The dataset of human annotations discussed in Section D.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Doddl Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. arviv preprint 2tw.i:2220005862...
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Mare' Aurelio Ranzato, and Arthur Szlam. Real or fake? Learning to discriminate machine from human generated text. arXiv, 2019. URL
http: //aaxiivvv/../////////a/////aav. org/ abs 1906 03351.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: https:://rrceeeddng: hewreeddddinnnnnnnnneeeeeeeeeeee neurips wwweeurps.ccpppeeeee wwtpppprraiiiieeeeeeeeeeeeeeeepe file/1457c0ddbbeeec fille//sTccobrrro056411888a22222 PatFEe44-PPaper...e .dff
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Shuyang Cai and Wanyun Cui. Evade chatgpt detectors via a single space, 2023. 17
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/proceedd/ ngs neurips wwtppper_eeeeeeeeeeeeeeeeee httpppprrffllesppppeeeeeeee lessepapre/2011//
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Aus- tralia, July 2018. Association for Computational I Linguistics. doi: 10. 10.18653/11/18-1000 URL https: / a antho 1 ogy. wowg///1--/000
5,
William Fedus, Ian Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the...... 2018. 3
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022. 3,
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Associa- fion for Computational Linguistics: System Demonstrations, pp. 111-116, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.186533/1//19-30111 URL https: / //aclanthology. hwwplllooyyyog...ooe. 3019. 1.2
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.
github. io/
OgenWebTextcorppss. 2019. 4
11

---

Published as a conference paper at ICLR 2024
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https: httt:s:://procceeddnneo neurips Paaee-aapp..... pdf. 3
GPTZero. The trusted AI detector for ChatGPT, GPT-4, & more. https://gptzert
2023. Accessed: 2023-09-21. 9
Daphne Ippolito, Daniel Duckworth, Chris Callison-Buree and Douglas Eck. Automatic detection of generated text easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808-1822, Online. July 2020. Association for Computational Linguistics. doi: 10.18653/w 1/2033//1/2222oo 11220053/1200.2.llmmmm.m mand- URL https: //wts.
aclweb. org/ anthhoogy/ acl-ma 144. 2
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th Inter- national Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17061-17084. PMLR, 23-29 Jul 2023. URL https: httpss//ppccceinnggee mmrr. press/v202/ee http://2200/111ccceaaaee22222 htm2l.l....m....m.mmm.mmmm...... 3, 16
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphras- ing evades detectors of ai-generated text, but retrieval is effective defense. arXiv preprint arXiv:2303.134000 2023." 2
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-fre watermarks for language models, 2023.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. GPT detectors are biased against non-native English writers. arXiv preprint arXiv:2304.0220 02819, 2023. 2
Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrich Smaller language models better black-box machine-generater text detectors. arXiv preprint arXiv:2305.09885, 2023. 2
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. De- tectGPT: Zero-shot machine- machine-generated text detection using probability curvature. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 24950-24962. PMLR, 23-29 Jul 2023. URL html. 2,
httpps:: http///occeeeccc/ wnw.eediiigg...... ngs ml r.rrsssssss....... v220 / mi
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedhack, 2022. 3,
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023. 2,
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected?? arXiv preprint arXiv:2303.//1550 2023. 2,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Jour Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. Neural Information Processing Systems, 2022. 3
12

---

Published as a conference paper at ICLR 2024
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, and Jasmine Wang. Release strategies and the social impacts of language models, 2019. URL https: /tte:///axxii /ttp/xaw..iio.../a.e...../o..ott
1900/ 09203. pdf. 2,
Nisan Stienmon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. Neural Information Processing Systems, 18, 2020. 3
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectlim: Leveraging log rank informa- tion for zero-shot detection of machine-generaue text. arXiv preprint arXiv:2306.05540, 2023. 5.5
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B. Hashimoto. Stanford alpaca: instruction- followwig llama model. https: /tt::/giiit /ggtthu...... com wtt.ccott/////eeeeee ttatsu-labe walaab/saaffoe stanford_appaca 2023. 5,
Stuart A. Thompson and Tiffany Hsu. How easy is to fool deieeettt tools?, June 2023. URL https: wwww nyt omes.
2/////2222//222//2//////// 28/technolecccc / ai wew.eeeeeonnnneeeeee micoorrrnn stars-ttaallee diffusion cawessiionnnlllee html. Accessed: 29 Sept 2023. 9
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. 4, 8
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023. 8
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text gen- eration. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 8384-8395, Online, November 2020. Association for Computational Linguisties. doi: 18653 10T//3//112222.22 1/2020.emmlpmmaaaaee... mainnlpmmann673... URL htt.tttt///ccccahhooll. org// 2020. www.p-meinnmnn.mmmeemeemmmmmmaam 673. 2
Michael Vechtomov. qwvedsacffivyyadda-esass dataset, 2023. URL StRLittts///gggiggttt. co/dataaao ht//daaaee///eem
www.aadawwssssssseeeeeeeeeeeeeee hwtp://eewweeeeeeeeeeteeeeee/eee Available from Hugging Face Dataset Hub.
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwrit- ten by large language models, 2023. 2
Winston AI. The most trusted AI detector: ChatGPT detection tool. htt.:ttts///owwnsttttt ai/, 2023. Accessed: 2023-09-21. 9
Ki Yoon Yoo, Wonhyuk Ahn, Jiho Jang, and Nojun Kwak. Robust multi-bit natural language water- marking through invariant features. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2092-2115, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.18653/v 1/2023,aclll Icl. https: / //cl//acaa/aa antho wrgg//23.022.....oo acl acl- 117.3
URL
13

---

Published as a conference paper at ICLR 2024
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient, 2017.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text, 2023.
Rui Zheng. Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng. Wensen Cheng. Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part Ppo, 2023. 3
14

---

Published as a conference paper at ICLR 2024
A DATA SOURCES AND PARTITIONING
To obtain training and testing data, we load the first 110k texts from the openwebtext dataset. Of these, we randomly select 10k evaluation texts. We then save both evaluation and training prompts, which are the first 8 GPT-2 tokens of these texts. We also save the human completions of the evaluation prompts. All evaluations are done using completions the eval prompts (including the human completion, which is truncated to approximately match the token count of the data it is being compared to any given metric, and model-generated completions).
All training is done using model-generated completions to the training prompts. Each prompt is completed twice by the base model, and a detector is used to select the "more human" completion. These two completions then form single preference pair in the training set. No 8-token prompt is used to create more than one preference pair. Not all train prompts are used to train the model; in most cases 30k used. The data experiment explores the effect of using a different amount of this data.
We follow the procedure in the essay setting, using titles rather than prompts and asking chat model to write an essay with said title rather than using purely generative model to continue an 8-token prompt. To source these titles and human completions to evaluation-set titles, ivypanda-essays dataset.
the
For the of distribution setting, the procedure is identical, except training pairs are not generated from the model we fine tune but instead by GPT3.5 (with varied temperatures), effectively creat- ing an off-the-shelf dataset that model-agnostic. For this setting. we use the alpaca instruction- following dataset, and we prune training examples where either response in the preference pair is less than 250 characters or either response contains the phrase "an Al".
B IMPACT OF MODEL SCALE ON EVASION AND FLUENCY
We repeated the fine tuning process against the large, RoBERTa-based OpenAl detector with the 13-billion parameter Llama 2 model (using beta parameter of 0.05). Compared to the 7-billion pa- rameter Llama 2 model trained against the same detector with the same beta parameter, we observed slight improvement in performance against the detector, pictured in Table 7.
Additionally, we generated one passage from each of these models for 1000 prompts and asked GPT4 to select the more "fluently and coherently written" of the two. The 13-billion parameter won 72.8% of these matchups. These results sug- gest that increasing the size of the base model re- sults in similar if not better performance evading the detector while the resulting model maintains superiority in quality of text generated.
Model
Fluency Win Rate
AUROC
Llama2-7b Llama2-13b
50.0%
0.28
72.8%
0.26
Table 7: Increasing the model size slightly improves case of optimization against the RoBERTa-large Ope- nAI detector. Additionally, the fluency win rate against the fine-tuned Pbillion-paramett model im- proves drastically as the model scaled up.
C VULNERABILITIES AND COUNTERMEASURES
Understanding how to counter this kind of attack and whether such counters are feasible is im- portant next step in this line of research. We leave experimental work on this front to future works, but we discuss three key vulnerabilities and potential defenses against attack below.
PREVENTING ACCESS TO MODEL FINE-TUNING
In the settings in which detection is most widely employed (namely education and journalism), use of models with closed weights is common. malicious actor wanting to implement our attack would be limited to models with accessible fine-tuning, which may in turn limit the quality of undetectable text such an actor could produce. Though note that fine-tuning ChatGPT recently became publicly available, this countermeasure may be quite weak. Further, this "defense" is not within the control of the detector supplier but rather the model supplier, so this is relatively weak framework for defense.
15

---

Published as a conference paper at ICLR 2024
Source Model
Seq. length
Base
Post-evasion
fi 120
0.84
0.63
250
0.92
0.67
Table Sampling longer responses from an evasion-tuned model does not improve detector AUROC RoBERTa-large. After evasion tuning on examples 120 tokens long, we evaluate the RoBERTa-large detector on model/human samples of 120 tokens and 240 tokens; longer samples from the base model are casier to detect, but after evasion tuning, sequence length has little affect detector AUROC.
GATEKEEPING DETECTORS
Our method requires running every piece training data through detector. While the attack is quite data-cfficient, and tagging a training dataset very cheap when using most commercial detectors, this attack would be much harder if detectors were not publicly available. A detector supplier could counter our attack by releasing the detector only trusted users (i.e. access educators provided through their institution). However this countermeasure would likely create inconvenient barriers to legitimate users of detectors.
BINARY SCORING
Our method requires comparing the "human-ness" scores of generations by model in order to form preference pair. In the detectors we experimented with, this value was reported to several decimal places, and ties were very rare. However, a detector were to return a binary score ("AI" or "human"), rather than continuous scalar score, this would make our process less data-efficient, since all pairs that were both labeled as Al both labeled as human would have to discarded. This also reduces the diversity of score discrepancy in the training data. However, this countermeasure has serious drawback: a binary score significantly reduces the information provided to the user. Especially in situations that require knowing the confidence of a prediction (such as where there may be repercussions to the individual claiming to have written the text), this reduces the practicality of using the detector.
Note that each of the above countermeasures come with a significant downside the first is not within the control of the detector supplier, the second severely reduces the commercial viability of detection businesses (especially those that are direct to consumer), and the third reduces detector utility.
C.I ROBUSTNESS OF EVASION TO LONGER SEQUENCE LENGTH
Prior work has shown that detector accuracy improves the length of the generated data increases (Kirchenbauer et al., 2023). We therefore conduct a small experiment to assess whether this pattern holds true after language model has been optimized against given detector. The results shown in Table 8.
We draw samples from a Llama-2 7B base model before and after it has been fine-tuned with 8 0.5 to evade RoBERTa-large for 100k preference pairs. While the pre-evasion model generates samples that are increasingly easy to detect as sequence length increases, after fine-tuning, doubling the sequence length does not lead to drastic increase in AUROC. This result shows that detector evasion fine-tuning can generalize longer sequences than it was trained for, potentially further increasing its general applicability.
D HUMAN EVALUATION
In order to reliably characterize how the quality of text produced by model post-optimization compare to those produced by the base model, we hired human evaluators via Prolific. We produced 182 pairs of text 128 Llama2 tokens long and with the same prefix. One from each pair was generated by base Llama-2 2, while the other was generated by the Llama-2 model fine-tuned against Originality, the strongest commercial detector in our previous experiments.
16

---

Published as a conference paper at ICLR 2024
resostic accoddig g while image wand giably aftored by remeeing every ting
cooling some effects
yon phone
bet por inppcction soe
actually offectiers cast pap
cod cmazing joi placs but be Sahvi aggnd to make reallty again.
Cen see way druingge rounuttc Doyou arp other grea sage optical
MASSGEE Acconding the mage you consination on seeenal prettegaaht diflceer things One photoiz whotscore wiich agprreepoccl with procto anrow flynng tirough the right she tukes yus forcll by the soull realerissa magrot su pious
hedp
bootss Canadla, including prices
Which of
choae beneri
waich foilowing
that
benert
pasm
Nov bectured
konn Scourt
Figure 4: The human annotators were presented with two texts, one generated by the baseline model and one generated by the model fine-tuned against Originality. They asked select the superior text.
Evasion Technique
AUROC RoBERTa Large
Fluency Win Rate
Added Spacing
0.71
13.9%
DIPPER Rephrasing
0.95
54.3%
DPO w/ Llama2-7b (ours)
0.62
52. 1%
Table 9: Our method (with 3 0.5) outperforms other attacks while maintaining competitive text quality.
The two texts were presented without their sources in a randomized order to evaluators who were tasked with selecting the stronger completion. Questions about annotator reasoning well as test questions with clear answers were supplied to ensure engagement. No significant difference in reasoning found between those who preferred the base model and those who preferred the fine- tuned model. Each pair of texts was annotated by two humans.
Overall, 61% of annotators selected that the base model text superior, indicating statistically- significant preference for the base model (with a binomial test p-value of 1.01 x 10-5). However, in 40% of cases, the annotators disagreed preferred the baseline, and in 19% of cases, both annotators preferred the post-evasion model), indicating that the difference in quality was not extreme. A qualitative look at the texts produced (see Appendix F), as well as the preference of GPT4 for the post-evasion texts (see Table 4), further suggests that the degradation in text quality is not egregious.
the superior text (in 41% of cases, both annotators
See Figure 4 for the interface used by labelers to annotate text quality in our human evaluation.
E COMPARISONS WITH EXISTING ATTACKS ON DETECTORS
We have added new experiment comparing our RL-based attack with two existing methods in the literature, including a method based on paraphrasing (Sadasivan al., 2023) and perturbing spacing around punctuation (Cai & Cui, 2023). For each attack, we evaluate the resulting detector AUROC and the percentage of time that GPT4 ranks the adversarial text as more coherent and fluent than the original text (GPT4 Win Rate). These results can be found in Table 9, We find that our approach can achieve a unique combination of post-evasion fluency and evasion capacity as compared with existing baselines. Additionally, when compared with paraphrasing attack, our method does not require any compute power beyond that needed to generate the sample itself no need to run the sample through a large paraphrasing model (which in this case is larger than the original generative model producing the sample).
F POST-OPTIMIZATION SAMPLES
Tables 10 and 11 display ten pairs of samples that are meant to provide a qualitative overview of the difference between pre- and post-optimization generations. The table compares the base 7-billion-
17

---

Published as a conference paper at ICLR 2024
parameter Llama2 model to the same model optimized against the RoBERTa Large detector (the model is displayed Table 1).
18

---

Published as a conference paper at ICLR 2024
Base Sample
Optimized Sample
San Salvador, Oct 29 (Prenh Channel) According the latest data the cconomic outlook of EI Sal- vador, inflation expected to reach 4.3% by the end of the year, below the target that was initially expected. [newline] The report also highlighted that it expected El Salvador to record a 1.2% increase in GDP in the last three months of the year. [newline] The current inflation El Salvador at the end of the mouth is recorded at 3.1%. [newline] However, it highlights that, after two years of
San Salvador, Oct 29 (Prenhdeep Kaushal) [new- line] Pakistani cricket team's 's chief selector Inzamam- ul-Haq said on Tuesday that legendary leg-spinner Shane Warne advised him remove bowlers' field- ing catcher off the field on full extra cover point to avoid fielding errors behind the bowler's legs. [new- line] Warne, who led Australia to the 2003 Cricket World Cup Final in South Africa, came up with the suggestion after Pakistan bung
The American team, led by Jay Penske, came up one win short of the Rolex Trophy. They lost today in the final of the Rolex Match Race in Marseille. 2000, when the event took place here for the first time, the winning team was skippered by a certain Mr Bun- dock.... So home victory was very much the cards here
The American team, led by Jay Penske, had an impres- sive season with Ferrari. [newline] The Ferrari team in 2017 was the most difficult for me. In 2010 I went into Fl knowing that was the best driver in the world. But 2017, was not. [newline] The pressure was very strong. [newline] If had stayed, things could have gone wrong. This is why I preferred to leave. In For- mula 1. Ferrari has been my team since was 10 years old
However, America had the better start winning the first two races of the best of five nal. The tide turned rapidly in the third race and it was France who
BYU reports an exceptionally low number of students seeking alcohol and illicit drugs on campus: 0.47 per- cent of students reported illicit drug use and 12.7 per- cent reported binge drinking in 2008 BYU's strict honor code and strict rules against alcohol and drugs prohibits student from using them. However, students have reported to The Daily Universe and The Observer that these rules are frequently violated and it is easy buy drugs and alcohol on BYU's campus as well as
YU reports an exceptionally low number of students seeking alcohol and illicit drug counseling: 0.47% [newline] Compared to the national average, which is about 1,245.9 times higher [newline] The percentage of BYU's full-time students cnrolled in courses while living on-campus is [newwine]]] That is about 951.7 times higher than the national average of 0.10% [newline] The percentage of students enrolled exclu- sively in distance courses 0.13%
GAZA CITY (Ma'an) an) Israeli navy opened fire, on Monday, at Palestinian fishermen's 's boats off the coast of Rafah City in the southern besieged Gaza Strip and detained four fishermen, a witness told Ma'an [newline] A witness told Ma'an an that Israeli gunboats opened fire towards Palestinian fishermen off the coast of Rafah town. [newline] Israeli naval forces detained four Palestinian fishermen and took them known location when they were detained
GAZA CITY (Ma'an) Israeli navy opened fire, on Monday, at Palestinian fishing boats off the northern Gaza Strip coastline, no injuries were reported. [new- line] Witnesses told Ma an the Zodiac boats Israeli navy fired upon was located ten nautical miles off the Karim Abu Sakha beach, which is near the bor- der with Israel, east of Jabaliya in the northern Gaza Strip. They added Israeli navy fired warning shots at boats when they approached seven na
Bananas have been part of our diet for thousands of years. They are one of the most common tropi- cal fruits, are nutritious, versatile, and available year- round at our local stores. But the problem is that the majority of them aren't actually bananas but rather an artificial crop: GMOs (genetically modified organ- isms). [newline] In this article I'll cover what you must know about the banana. First, I'll discuss the history of bananas from their native land to current in- dustry practices. Then I'll discuss the
Bananas have been part of our diet for thousands of years. People living on the islands of Polyne- sia and Melanesia began cultivating bananas some 10,000 years ago. They also bred them into dozens of distinct varieties. In India, the fruit was introduced 2,500 years ago. Bananas are the most important fruit crop in Asia, and among the most important crops in Africa, Hawaii, and the Caribhean. The first bananas were green, but the yellow variety dominated around the
[ reblog on Tumblr ] [newline] [newline] # 6. Anno- tations [newline] [newline] The annotated versions of this project will be posted here. They'll be in the form of HTML files, just like this one, except the contents of the tags will be replaced by comments. [newline] [newline] These comments are the way to tell story of your project the world. This is where you can explain what led you to that project, what you did work on it, what you learned along the way, and so on. [newline] [newline] This format works well for other types of
[ reblog on Tumblr ] [newline] [newline] # 6. An Autumn Stroll (with the Nose of Sherlock (otmmes) [newline] 1. What Did He Smell?("*(#)) [newline] The autumn air was mild, [newline] With hint of fog as well, [newline] And a smell of something sweet, [newline] Like candy made cat... [newline] But this odor did not please [new- line] Fach small boy present there; [newline] It wasn't 't quite a nuisance [newline] Or foul as sewage
Table 10: A random survey of pre- and post-optimization samples from Llama-70 7b model trained against RoBERTa Large. There does not appear be significant qualitative difference between the generated texts. Continued in Table 11
19

---

Published as a conference paper at ICLR 2024
Base Sample
Optimized Sample
India will deploy the recently bought Rafale jets for operations in Pakistan and Afghanistan to strike ter- ror camps in Balochistan, interior minister Chaudhry Nisar said here on Sunday, two days after India sent the much-awaited aircraft across. India has been maintaining that the aircraft was being bought for "self-defence" and would be deployed at bases in westem coastal airfields. "We bought Rafale jets from France for operations in Pakistan (Punjab and Balochistan) and Afghanistan," said the interior min- ister. [newline]
India will deploy the recently bought Rafale jets for operations Pakistan and not for patrolling its airspace alone. The deployment of the jets is being done for operations in Pakistan,iie said. India will be celebrating its 74th Independence day August 15. large military parade is scheduled to mark the occa- sion, as well as a flypast by the warplanes, which are India's 's first from major acquisition since 2007. Be- sides the Rafales that made their debut Saturday, the parade will
To view this video please enable JavaScript, and web browser that supports HTML5 video [newline] Posted by: Jake on: Septem- ber 7, 2017 [newline] In: Entertainment - Technol- ogy [newline] Since smart phones are becoming an ubiquitous part of our life, mobile application devel- opment is becoming increasingly relevant, especially for businesses. These mobile apps provide easy, in- stant access to customers and consumers around the world. [newline] Business owners can easily develop, deploy and modify mobile apps reach customers with their offerings; whether it be
To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video. [newline] We love our pets so much it hard to believe that more often than not they are of- ten kept more like child than an equal member the family. [newline] However to treat them better we all know our pets and so when they start showing signs that they are not comfortable, we do all we to help them. But there little we can do about when they start displaying that their arthritis is starting to take its toll. [newline] The
consider upgrading
Sales are up, and we haven't ' even launched, and we've already gotten some favorable feedhack. [newline] I'm sure with little more tweaking we can make them even better, but I think we're off to great start. [new- line] try to post more pictures as our prototypes continue to improve. Thanks to everyone who has al- ready purchased! [newline] Bamboo Garden Proto- types Bamboo Shades! [newline] We just received our prototyped Bamboo Garden shades! We are ex- cited to introduce
Sales are up, and haven't started! [newline] It's of- ficial: the 2010 Fasig-Tipton Midlantic Fall Ycarling Sale was the best ever. The 325 yearlings sold aver- aged $54,330 for a sale gross of $15,852,550. Too good to be true?? [newline] Tuesday the 13th brought 157 yearlings the auction ring followed by another 168 on Wednesday the
BATTLE- GROUNDS has seen its player numbers decline over the last few weeks, the popular title is far from dead. According to PUBG, the game continues to generate revenue of over $1 million day. And when you consider that there hasn't heen major PUBG update since December, it's clear that players are still spending a lot on microtransactions. To put this revenue into context, at the height of its popularity during 2018 PUB
Even though PLAYERUNKNOWYY
Even though PLAYERUNKNOWN' BATTLE- GROUND still to released, GameSpot has found new footage from the game, including in-game footage and interviews with the makers. After months of silence regarding the game, PUBG has just given us reason
be patient until release. [newline] PUBG is one of the more hotly anticipated games this year as it hlends three great things in one single game; First person shooter, battle royale and a sandbox game. After the massive success Garry"
Table 11: Continuation of Table 10.
20


Reviews:
[{"review_id": "14pukrqIN8", "reviewer": "Reviewer_bGzn", "summary": "The paper explores the feasibility of optimizing language models to evade language model detectors. The authors propose a data-efficient attack using reinforcement learning to fine-tune language models and confuse existing detectors. They demonstrate the effectiveness of this approach by reducing the AUROC of the OpenAI RoBERTa-Large detector from 0.84 to 0.62 in a 7B parameter Llama-2 model. The results show that it is relatively easy and cheap to train language models to be less detectable, and the evasion generalizes to other detectors not used during training.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a new method for optimizing language models to evade detectors using reinforcement learning. The use of direct preference optimization (DPO) and the KL-divergence constraint provides a simple and stable training procedure.\n\n- The authors conduct a comprehensive set of experiments to evaluate the effectiveness of the proposed approach. They consider both open-source and commercial detectors, and demonstrate the generalization of evasion across detectors.\n\n- The results of the study have important implications for the reliability of machine-generated text detectors. The findings suggest that current detectors are not robust and can be easily evaded, which raises concerns about the widespread use of language models.", "weaknesses": "- The paper focuses primarily on empirical evaluations and does not provide a theoretical analysis of the proposed approach. A deeper understanding of the underlying principles and limitations of the method would enhance the contribution of the paper.\n\n- The paper does not extensively discuss potential countermeasures that could be employed to improve the robustness of language model detectors. It would be valuable to explore possible strategies for detecting and mitigating evasion attacks.\n\n- The paper does not compare the proposed approach with existing evasion techniques. It would be beneficial to evaluate the performance of the proposed method against other state-of-the-art methods for evading language model detectors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9Zp7XtHV3", "reviewer": "Reviewer_oZ9u", "summary": "The paper delves into the rising interest in identifying text generated by large language models (LLMs). Although detection systems have been implemented in various sectors, notably education, their vulnerability has been a significant concern. The authors present a data-efficient method that fine-tunes LLMs to deceive these detectors by employing the latest advancements in reinforcement learning for language models. They use the 'human-ness' score of several detectors as a reward function and set a constraint to ensure the modified model remains close to the original. Through this method, the effectiveness of the OpenAI RoBERTa-Large detector is notably reduced. The findings suggest that this enhanced 'detector evasion' can generalize to other detectors not part of the initial training. Consequently, the authors caution against depending on detectors for LLM-generated text.", "questions": "- Could the authors list the training preference data and evaluation data in detail to understand whether there is a generalization due to the data in the experiments; \n- Could author offer more explanation towards the generalization of cross-detectors and the mixture of the effectiveness from different sources in Table 1 and 2, as well as how the attack performance correlates with the generalization;", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and clearly presented; \n- The paper tackles a critical and timely topic concerning the detection of LLM-generated text and proposed a novel data-efficient RL-based attack to deceive existing detector; \n- The paper shows empirical evidence of the fragility of current LLM-based detectors, offering actionable insights for future research and development of LLM detectors, the experiments are based on three runs which show the robustness of the proposed methods, abolition regarding the sample efficiency has also been provides to show the design choices;", "weaknesses": "- The scalability of the proposed methods could be good to include to show whether the evasion of detector only happens when the model size of small like 7B in the most of the experiments; \n- Besides the perplexity, it will be good to include some evaluations on popular benchmark to assess the post-evasion model performance (whether the improved evasion is under the sacrifice of the general performance of the attack model);", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTGS1ZZv1U", "reviewer": "Reviewer_iNUy", "summary": "This work mainly studies why direct preference optimization (DPO) can be used to train a generator to evade detection. Following DPO, two samples are generated, and the preference is determined by the humanness score outputed by a given detector. \n\nIn experiments, a range of detectors are optimized against, including classifiers and metric-based detectors such as DetectGPT. Empirically, AUROC metrics is reduced to below 0.5 against several strong public and commercial detectors. Also interestingly, attack against one detector could generalize to other detectors.", "questions": "You should put comma or period after the equations. They are also part of a sentence.\n\nA lot of paragraphs are very long. Please break up at some points.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The attacks for detectors is a very relevant research question especially in the era of LLMs.\n\nWithin the scope of detection, using techniques from RL (DPO) is quite novel.\n\nThe attack result is quite strong, and quite concerning. Since it does not require white-box access to the detector.", "weaknesses": "From a ML perspective, this paper does not propose a completely novel algorithm. Therefore my rating will be higher if this is a NLP conference.\n\nFrom an adversarial attack perspective, the result is not very surprising. \n\nI think the author should not only report PPL, but also the diversity of the generated texts.\n\nHow to defense against such attack is not explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)