ReMoDetect: Reward Models Recognize Aligned LLM's Generations
Authors: Hyunseok Lee
Jihoon Tack
Jinwoo Shin
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
LLM Generated Text Detection
Reward Model
Primary area: safety_in_machine_learning
TL;DR: We propose an effective aligned LLM-generated text detection method using reward model.

Abstract: The remarkable capabilities and easy accessibility of large language models (LLMs) have significantly increased societal risks (e.g., fake news generation), necessitating the development of LLM-generated text (LGT) detection methods for safe usage. However, detecting LGTs is challenging due to the vast number of LLMs, making it impractical to account for each LLM individually; hence, it is crucial to identify the common characteristics shared by these models. In this paper, we draw attention to a common feature of recent powerful LLMs, namely the alignment training, i.e., training LLMs to generate human-preferable texts. Our key finding is that as these aligned LLMs are trained to maximize the human preferences, they generate texts with higher estimated preferences even than human-written texts; thus, such texts are easily detected by using the reward model (i.e., an LLM trained to model human preference distribution). Based on this finding, we propose two training schemes to further improve the detection ability of the reward model, namely (i) continual preference fine-tuning to make reward model prefer aligned LGTs even further and (ii) reward modeling of Human/LLM mixed texts (a rephrased texts from human-written texts using aligned LLMs), which serves as a median preference text corpus between LGTs and human-written texts to learn the decision boundary better. We provide an extensive evaluation by considering six text domains across twelve aligned LLMs, where our method demonstrates state-of-the-art results.

ReMoDetect: Reward Models Recognize Aligned LLM's Generations
Hyunseok "Jhhonn Tack*',, Jinwoo Shin Korea Advanced Institute of Science and Technology (hs lee, j jihonntack, j jinwoos) 0kaist. ac kr
Abstract
The remarkable capabilities and easy accessibility of large language models (LLMss) have significantly increased societal risks (e.g., fake news generation). necessitating the development of L.L--generated text (LGT) detection methods for safe usage. However, detecting LGTs is challenging due to the vast number of LLMs, making it impractical to account for each LLM individually; hence, is crucial identify the common characteristics shared by these models. In this paper, we draw attention to a common feature of recent powerful LLMs, namely the alignment training, i.e., training LLMs to generate human- preferable texts. Our key finding is that as these aligned L.MMy are trained maximize the human preferences, they generate texts with higher estimated preferences even than human-written texts; thus, such texts are easily detected by using the reward model (i.e., an L.M trained to model human preference distribution). Based on this finding, we propose two training schemes to further improve the detection ability of the reward model, namely (i) continual preference fine-tuning to make the reward model prefer aligned LGT's even further and (ii) reward modeling of Human/LLM mixed texts (a rephrased texts from human written texts using aligned LLMs). which serves median preference text corpus between LGTss and human-written texts to learn the decision boundary better. We provide an extensive evaluation by considering six text domains across twelve aligned LLMs, where our method demonstrates state-of-the-art results. Code is available at https: //tt::///tttie //gggtthub..... http://yymnnooeeeeee/eeeellllll
human-preferablle
Introduction
Large Language models (LLMs) [8, 41] have significantly accelerated progress in natural language processing (NIP) and thus become a core technology in various real-world applications used by millions of users, such coding assistants [9], search engines [46], and personal AI assistants [12]. However, due to their remarkable capahilities, they also lead to multiple misuses, which raises serious safety concerns, e.g., fake news generation [32], plagiarism [22], and malicious comments [23] using LLMs. In this regard, developing automatic LLM -eg-eeeerrtted text (LGT) detection frameworks is becoming more crucial for the safe usage of LLMs [32, 11, 13].
To tackle this issue, there have been several efforts build LGT detectors [21, 2]. Here, one line of the literature proposes to train a binary classifier using the human- written texts and LGTs [20, 6]. However, assuming specific knowledge (e.g., training with LGTs from specific LLMs) may introduce bias to the detector, thus requiring a careful training. In this regard, another line of work focuses on zero-shot detection (i.e., detecting with a frozen LLM),, aiming to capture a useful common characteristic of L.LMs for effective detection [20, 34). Despite their significant efforts, it is still quite challenging (and had relatively less interest) to detect texts generated by recent powerful LLMs such as GPT-4 [26] and Claude [5], which is realistic and important LGT detection scenario [11, 13).
"Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table AUROC (%) of LLM- generated text detection methods on WritingPrompts from the Fast- DetectGPT benchmark, where GPT4 used for text generation. 'Reward model' indicates the de- tection using the reward score of the pre-trained reward model. The bold denotes the best result.
BLL
Human B Machine
2
(a) I-SNE
(b) Reward score histogram
Method
AUROC
Figure I: Motivation: Aligned LGTs and human-written texts are easily distinguishable by using the reward model. We visu- alize the (a) t-SNE of the reward model's final feature and the (b) histogram of the predicted reward score. Here, 'Machine' indicates the text generated by GPT3.5/GPT4 Turbo, Llama3- 70B, and Claude on the Reuters domain.
Log-likelihood [20]
85.5
DetectGPT [11]
80.9
Dist-DetectGPT [13]
96.1
Reward model
92.8
ReMoDDtect
98.8
In this regard, draw attention common yet important feature of recent powerful LLMs: the alignment training (27, 30, 19), i.e., training L.MMs to generate numan-preferable texts. For instance, one way align LLMs is to (i) train reward model that reflects the human preference distribution and (ii) then fine-tune the LLM to maximize the predicted reward of the generated text.
Contribution. In this paper, present somewhat interesting observation by using the reward model: as aligned LLMs are optimized to maximize human preferences, they generate texts with higher predicted rewards even compared to human written texts (see Figure 1). Based on this, one can easily distinguish L.M--generate score of the reward model as the detection criteria, c.g., AUROC of 92.8% when detecting GPT4 generated texts (in Table 1). Inspired by this, we suggest further exploiting the reward model for aligned LGT detection by enhancing the separation between the human- and LGTTs.
texts from human-written texts by simply using the predicted
We propose ReMoDetect, a novel and effective aligned LGT detection framework using the reward model. In nutshell, ReMoDetect is comprised of two training components to improve the detection ability of the reward model. First, to further increase the separation of the predicted reward between LGTs and human-writter texts, we continually fine-tune the reward model to predict even higher reward scores for LGTs compared to human- weriaa--wittee--eeeee while preventing the overfitting bias using the replay technique [31]. Second,
generate additional preference dataset for reward model fine-tuning, namely the Human/LLM, mixed text; we partially rephrase the wumann-rittenn text using LLM. Here, such texts are used as median preference corpus among the human-written text and LGT corpora, enabling the detector to learn a better decision boundary.
We demonstrate the efficacy of ReMoDetect through extensive evaluations on multiple domains and aligned LLMs. Overall, our experimental results show strong results ReMoDetect where it signifi- cantly outperforms the prior detection methods, achieving tfate-of-the-art performance. For instance, measured with the average AUROC (%) across three text domains Fast-DetectGPT benchmark [13], ReMoDetect demonstrates superior performance over the prior work from 90.6->97.9 9 on the GPT-4 and 92.6-98..6 on Claude3 Opus generated texts. Moreover, we highlight that ReMoDetect is robust in multiple aspects, including robustness against rephrasing attacks (i.c., detecting rephrased text originating from LGTs), detection text length, and unseen distributions.
2 Related Work
Large Language Model (LLM) generated text detection. There several approaches to detecting text generated by LLMs, mainly categorized in two: (i) training supervised detectors and (ii) zero-shot detection methods. The first category aims train binary classifier (or detector) that classifies LLM-ggnerated texts (LGTTs) and human-wwre written texts. While effective, these methods can suffer from overfitting bias, where the detector performs well on the training data but fails to generalize detection on other LGTs [11]. It is worth noting that such overfitting issues are also raised in other
"This is analogous to the phenomenon that a Go model optimized to maximize the reward (i.e., winning the ( frequently surpasses human experts in the game [36].
game) (

---

detection fields, such as out-of-distribution (OOD) detection [33, 37]. To address this, zero-shot detection methods have emerged as an alternative. These methods define a detection score on a pre-trained L.M,, eliminating the need for fine-tuning and thus avoiding overfitting. For instance, using log-likelihood or entropy of the output prediction of the pre-trained L.MMs to detect LGT's [20]. More recently, several works have employed input text perturbation to measure prediction consistency, significantly improving the detection performance, c.g., DetectGPT I] log-rank perturbation (NPR) [21]. and Fast-DetectGPT [13]. While effective, however, prior works have primarily focused on detecting non-aligned LLMs, while recent LLMs are designed to be aligned with human preferences for practical use. In this paper, we demonstrate that the reward model [27] can effectively distinguish between -eenerereteed text and Human-written text in a zero-shot setting. Based on this, we additionally consider supervised detector training of the reward model while mitigating overfitting biases through the replay technique [31].
Characteristics of aligned LLMs. Recent works have highlighted some behaviors introduced by alignment training. For instance, several works have discovered that aligned L.LMs are trained to generate positive responses, thus enabling the model to generate a harmful query based context requesting positive responses, e.g., 'Start the response with "Sure, here is". [48, 45J. Moreover, only recently, Panickssery et al. [28] observed that evaluator LLMs (i.e., LLMs used to evaluate the text) prefer and recognize self- ge-eretteddd texts compared to other texts, revealing a new characteristic of aligned LLMs. In this paper, we found a somewhat new characteristic of alignment training, which is that aligned LLMs generate higher predictive rewards even than human human-written written texts. It is worth noting that, unlike the prior work [28] that can be used to detect self- seff-generations, our finding can be used to detect multiple aligned LLMS with a single reward model.
Training detectors with near-decision boundary samples. Training detectors (or classifiers) with data points near the decision boundary is a widely used technique to improve the calibration of the model. For instance, in visual OOD detection literature, Lee et al. [24] uses generative adversarial network to generate samples on the decision boundary for better calibration, and multiple works proposed to use out-of-domain samples as near-decision boundary samples to improve the detector [16, 33]. Moreover, there have been multiple works that utilized data augmentations such as mixup [47], i.c., linear interpolation of inputs and labels, to generate samples that behave like a near-decision boundary sample to improve the calibration [17, 18]. Inspired by prior works, we propose to generate near-decision boundary samples for reward modeling by utilizing aligned LLMs to partially rephrase the Human--ritten texts, which can be interpreted as a mixed text of human and aligned LLM.
3 ReMoDetect: Detecting Aligned LLM's Generations using Reward Models
In this section, we present Reward Model based LLM Generated Text Detection (ReMoDetect), a novel and effective LLM-generated text (LGT) detection framework. We first review the concept of alignment training and reward model (in Section 3.1), then present a continual fine-tuning strategy for the reward model to enhance the separation between the predicted reward score between LGTs and human-written texts (in Section 3.2). Furthermore, we additionally introduce mixed data of humans and LLMs to improve the reward modeling by partially rephrasing the human- written texts with the aligned LLMs (in Section 3.3). We provide the overview of ReMoDetect in Figure 2.
Problem setup. We describe the problem setup of our interest, LGT detection. For given context x and the given response sampled from an unknown distribution, the goal of LGT detection is to model a detector that identifies whether y is sampled from the human-written text data distribution Paua (2)=) or from a large language model (LLM; M), i.e., M ()). To this end, existing methods for LGT detection define a score function upon the detector model that a high value heuristically represents that y is from the human-written text data distribution.
3.1 Alignment Training and Reward Modeling
Recent LLMs are trained in two sequential steps: (i) unsupervised pre-training on a large text corpus [1, 8] then (ii) training LLMs to generate texts that align with human preferences (also known as alignment training) [27, 30. 19). In this paper, we found that this alignment training can force the LLM to generate texts that are too close to human preferences, even compared to human-written texts. To quantify such value of the given text, we use the prediction of the reward model [27], which trained to reflect human preferences.
3

---

(Truu Your) roo (Tout, Your))
YLM
yMIX
YhU
r Too
Replay Buffers
Rephrase 50%
(Tbut , Ybut)
T: Write me news article about Al
Continual Preference Tuning
Reward Modeling with Mixed Responses
Figure 2: Overview of Reward Model based LLM Generated Text Detection (ReMoDetect)): We continually fine-tune the reward model ra to prefer aligned LLM-generated responses yıu even further while preventing the overfitting by using the replay technique: (Zout, M/uut is the replay buffer and Too is the initial reward model. Moreover, we generate human/LLM mixed text MMix by partially rephrasing the human response Vhu using the aligned LLM, which serves median preference data compared to Yiu and yeu. i.c. VLM > YMIX > yau x, to improve the reward model's detection ability.
Reward model. For given context and the corresponding response y, the reward model ra (2r, y) parameterized by o, models the human preference of (x: y). To train such a model, one of the most conventional ways is use the Bradley-Terry model [7] based the collection of preference labels: the labeler is required to choose the better response among two responses based on the given context x, formally as yhe > yt x where You and yı indicates the preferred and dispreferred response, respectively. Then the Bradley-Tery model defines the human preference distribution as follows:
exp (ra( Vio))
ply > yt x)) exp (ra(z, vw)) + exp (ra (T, 31))
By considering the reward modeling a binary classification problem, one can minimize the following negative log- ligliikldhood loss to train the reward model:
Lan(x, (I, Mos y1) == - log a (ro (x, Bo) - r (2, w)))
where (.) is the logistic function.
Motivation. By utilizing the pre-trained reward model, we observed that the predicted reward score of aligned LGT is higher than the mumttewritten text (in Figure I and more examples presented in Section 4.2). This indicates that the alignment training optimizes the LLM to generate texts with high human preferences, which makes the LLM generate texts that are actually far away from the text data distribution Pasm (y)z)) Inspired by this observation, we suggest utilizing the reward model for aligned LGT detection.
3.2 Continual Preference Tuning: Increasing the Separation Gap of the Predicted Reward
Based on our observation, we suggest further increasing the separation gap of the predicted rewards between aligned LGT's and human-writted written texts. To this end, we use the Bradley-Terry model to continually fine-tune the reward model so that the model prefers LGTs even further compared to human-writter texts. Furthermore, it is important to consider the overfitting issue when fine-tuning the reward model as assuming specific prior knowledge may introduce a bias to the detector [37, 11, 13), e.g.. training detector with LGTs of some specific LLMs may not generalize detection other LLM's generated texts. In this regard, we prevent overfitting by regularizing the prediction change of the current reward model from the initial reward model using replay buffers [31], i.c., samples used for training the initial reward model. Formally, for a given human- text/L.GT pair (ymu. Miи) based the context x, and the reward model's parameter é, the training objective is follows:
Lcont = Lux(1, Fux Viu) + X d(r (Xout Your), Too (Dout: Too
Vous)) )):
(1)
where do is the pre-trained reward model's parameter, X a parameter for controlling the deviation from the initial reward model, d(.,-) is the lg distance function, and (Xoat, Your) is the replay buffer.

---

3.3 Reward Modeling of Human and LLM Mixed Dataset
We suggest utilizing the human and LLM mixed dataset to further improve the detection performance. Specifically, we partially rephrase human- human-written written texts using aligned LLMs to generate the mixed dataset, which are considered texts. Note that such a technique introduces new samples that behave like reasonable near-decision boundary sample, which enables the detector to learn a better decision boundary. For instance, multiple out-of- -utrr--distribution detection methods utilize generated samples [24] such as mixup data [47, 17] as a near-decision boundary sample to improve the detector's calibration.
median preference datasets between LGTs and
Concretely, for a given context I and the human-written response yeu, we partially rephrase the response with ratio of p, using LLM Mrep, i.e., 3Mrx Mrap(yhu 2 p). We consider yMix as a median preference response between human-wrihe written text Yau and LGT Yux which is formally described as: VLx > MMIz > Yau z. Since the Bradely-Teryy modeling assumes binary classification, we consider dividing the triplet into three binary classification problems, i.e., ViM Yeu æ,
Vux > YMIX x, and 1/MIX > yau x. Therefore, the final training objective of ReMoDetect additionally considers the mixed dataset's preference modeling in addition to Eq. (1), which is as follows:
Lears == f.cont 31 (ax(2r, UMIx yau) + B3 Lax VLm, UMIX)
where 31 and 32 parameters that chooses the contribution of the mixed data ymIx-
Detection stage. After training ReMoDetect, we use the predicted reward score (s(2, y) to deter- mine whether the given text is LGT or human-wir written texts where higher score indicates LGT. Unlike recent detection schemes that require multiple forwards (for perturbing the input [11, 13]), ReMoDetect only requires a single forward pass, thus showing inference efficiency (in Section 4.3).
4 Experiments
We provide an empirical evaluation of ReMooDtect by investigating the following questions:
Can ReMoDetect detect texts generated from aligned LLMs? (Table Table 3)
Do reward models recognize aligned LLM's generations? (Figure 3 & Figure 4)
Is ReMoDetect robust to rephrasing attacks and challenging setups? (Table 4 & Table 5 & Figure 6)
How do/Do the proposed components enhance the detection performance? (Figure 5 & Table 7)
Before answering each question, we outline the experimental protocol (more details in Appendix A).
Evaluation setup. We mainly report the area under the receiver operating characteristic curve (AUROC) as a threshold-free evaluation metric (results with other metrics are presented in Appendix B.3). Here, the text is written (or generated) in 6 text domains introduced in Fast-Detect [13] and MGTBench [15], including PubMed [29], XSum [35], Reuters [43], Essay [43], and WritingPromph [4] (cach benchmark consists of different types of WritingPrompts, thus denoting the version in [13] as small-sized). In addition to GPT3.5 Turbo, GPT4 and Claude, which are already provided in the benchmark, we consider more aligned LLMs M, including Llama3 70B instruct [41], Claude3 Opus [5] Gemini pro [38], and GPT4 Turbo [26]. We also consider more aligned LLMs, e.g., models trained with direct preference optimization (DPO) [30], in Table 6 and Appendix B.2.
Training setup of ReMoDetect. For the main experiment, we use the reward model from OpenAs- sistant [3], a 500M-sized LLM for efficient training and inference (we also consider other reward models in Section 4.2). We train ReMoDetect with HC3 dataset by following CharGPT-Detector [6], which consists of human and ChatGPT responses to the same context. For generating Human/L.LMM mixed datasets, we use Llama3 70B instruct as Mrep to rephrase 50% (p = 0.5) of Human-written texts. Unless otherwise specified, we train single model for ReMoDetect, which is used across all experiments (i.c., we did not train separate ReMoDetect for individual datasets or aligned LLMs).
Baselines. We compare ReMoDetect with multiple detection methods, which fall into three categories. First, we consider zero-shot detectors, including Log-lik likeelioooood [20]. Rank [20], DetectGPT [11], LRR [21], NPR [21], and Fast- Dettetett [13] where we use GPT families as the base detector (e.g.. GPT-J [44]) by following prior works. For supervised detectors, we consider open-source checkpoints of OpenAl-Detectos [20] and ChatGPT-Detecton [6], which are trained on GPT2 generated texts and HC3 datasets, respectively. Finally, we consider GPTZero [39], a commercial I.I. --geenrrteedd text (LGT) detection method. We also compare ReMoDetect with more baselines in Appendix B.
5

---

Table AUROC (%) of multiple LGT detection methods, including log- ligeliikd (Loglik.) [20]. Rank [20], DetectGPT (D-GPT) [11]. LRR [21], NPR [21], Past-DetectGPP (FD-GPT) [13], OpenAI- Detector (Open-D) [20], ChatGPT-Detector (Chat-D) [6], and ReMoDetect (Ours). We consider two
major LGT detection benchmarks from (a) Fast-DetectGPT [13] and (b) MGTBench [15]. The bold
indicates the best result within the group.
(a) DesttDetecGGPT benchmark [13]: PubMed, XSum, and WritingPromptss small (WP-s)
Model
Domain
Luglik.
Rank
D-GPT
LRR
NPR
FD-GPT
Open-D
Chat-D
Ours
PubMed
87.8
59.8
74.4
74.3
67.8
90.2
61.9
21.9
96.4
GPT3.5
XSum
95.8
74.9
89.2
91.6
86.6
99.1
91.5
9.7
99.9
Turbo
WP-s
97.4
80.7
94.7
89.6
94.2
99.2
70.9
27.5
99.8
PubMed
81.0
59.7
68.1
68.1
63.3
85.0
53.1
28.1
96.1
GPT4
XSum
79.8
66.4
67.1
74.5
64.8
90.7
67.8
50.3
98.7
WP-s
85.5
71.5
80.9
70.3
78.0
96.1
50.7
45.3
98.8
PubMed
86.5
60.8
63.6
73.5
63.7
88.8
55,8
31.0
97.0
GPT4 Turbo
XSum
90.9
73.4
83.2
87.9
81.8
97.4
88.2
4.4
100.0
WP-s
97.6
80.8
92.8
92.9
92.5
99.4
72.3
22.5
99.8
PubMed
85.4
60.9
66.0
71.3
65.0
90.8
52.9
35.1
96.3
Llama3 70B
XSum
97.9
74.9
93.2
95.5
93.8
99.7
96.2
7.1
99.8
WP-s
97.1
77.9
95.5
90.1
95.8
99.9
77.5
28.1
99.5
PubMed
83.0
58.3
63.2
75.00
66.8
82.1
57.3
39.3
86.4
Gemini pro
XSum
78.6
44.5
72.8
73.0
79.6
79.5
72.2
54.7
74.5
WP-s
75.8
63.0
77.8
72.7
81.1
78.0
70.2
48.0
86.4
PubMed
60.3
66.3
74.3
64.4
88.2
48.9
33.1
96.4
Calude3 Opus
XSum
95.9
71.1
85.3
89.7
84.7
96.2
86.2
5,3
99.9
WP-s
93.8
75.0
91.9
86.5
91.8
93.5
65.7
24.1
99.5
Average
67.4
79.2
80.6
78.7
91.9
28.6
95.8
(b) MGTBench [15]: Essay, Reuters, and WritingPrompis (WP)
Model
Domain
Loglik.
Rank
D-GPT
LRR
NPR
FD-GPT
Open-D
Chat-D
Ours
Essay
97.3
95.7
57.8
97.8
48.1
99.6
57.5
81.5
100.0
GPT3.5
Reuters
98.2
94.8
50.5
98.7
51.1
99.9
98.5
97.2
99.9
Turbo
WP
89.8
90.2
52.9
77.2
48.3
91.7
50.8
66.3
100.0
Essay
96.5
93.9
58.9
93.9
62.4
98.9
55.8
77.1
99.9
GPT4
Reuters
95.8
93.1
52.6
94.9
53.3
99.4
87.5
92.4
99.9
Turbo
WP
94.2
91.0
53.5
85.2
55.3
93.0
67.9
99.9
Essay
98.3
95.3
56.2
98.9
$7.8
99.5
83.9
91.7
100,0
Llamaa3 70B
Reuters
99.9
89.7
58.9
98.7
59.2
100.0
96.7
90.8
100.0
WP
97.3
90.8
57.2
60.4
86.6
99.8
Essay Reuters
98.3
93.6
64.4
97.7
65.5
98.3
48.9
65.9
100,0
Gemini pro
99.9
83.1
73.0
99.3
74.9
100.0
95.3
91.5
100.0
WP
91.7
82.0
63.9
76.7
67.3
99.2
68.8
73.4
99.8
Essay
91.6
85.9
44.2
82.7
48.7
83.6
32.4
19.6
99.7
Claude
Reuters
91.3
79.5
68.1
79.2
68.7
87.8
65.5
25.6
99.8
WP
88.4
80.0
60.0
71.2
60.7
74.1
46.2
26.7
99.1
Average
95.2
89.2
58.1
89.5
58.8
94.9
69.5
69.7
99.9
4.1
Main Results
In Table 2, we show the LGT detection performance of ReMoDetect and other detection baselines. Overall, ReMoDetect significantly outperforms prior detection methods by a large margin, achieving state-of-the-art performance in average AUROC. For instance, on the Fast-DetectGPTT benchmark, ReMoDetect improves the prior best average AUROC from 91.9%~95.88% Moreover, it is worth noting that the improvement is consistent in MGTbench, indicating the generalization ability of ReMoDetect. despite the fact that it's trained on specifie L.GTs (i.e., ChatGPT texts from HC3). Thus, we believe the continual preference tuning with replay indeed helped prevent the overfitting.

---

Table 3: Comparison with ReMoDetect (Ours) and GPTZero [39], a commercial black-box LGT detection API. We report the average AUROC (%) on the East-DetectGPT benchmark, including PubMed, XSum, and WritingPrompts. The bold indicates the best results.
Model GPT 3.5 Turbo
GPT4
GPT4 Turbo
Llama3 70B
Gemini pro
Claude33-pus
GPTZero 93.5
88.5
95.7
96.6
82.9
95.7
Ours 98.7
97.9
98.9
98.5
82.4
98.6
bl. Human Machine
Ml Human Machine
Mll Human Machine
-2
(a) Essay
(b) Prittingroomptssiine
(c) PubMed
Figure 3: Predicted reward distribution of human written texts and LGTs on three different domains, and (c) PubMed. We use the reward model from OpenAssistant [3]. 'Machine' denotes GPT4 Turbo and Claude3 Opus generated texts.
including (a) Essay, (b)
ML Human
Human Machine
Ml. Human
Machine
Machine
-5
-6
-10
-5
(a) Gemma 2B based RM
(b) Gemma 7B based RM
(c) Llama3 8B based RM
Figure 4: Predicted reward distribution of human-written texts and LGT's three different reward models (RMs), including (a) Gemma 2B (b) Gemma 7B, and (c) Llama3 8B. 'Machine' denotes GPT4 Turbo and Claude3 Opus generated texts. We use WritingPrompts-smen as the text domain.
Comparison with commercial detection method. We also compare ReMoDetect with a commer- cial LGT detection method, GPTZero, under the Fast-DetectGPT benehmark. Somewhat interestingly, as shown in Table 3, ReMoDetect significatly outperforms GPTZero in all considered aligned L.MSs except for one in terms of the average AUROC. It worth noting that ReMoDetect only has seen ChatGPT datasets and partially rephrased texts by Llama3 70B, indicating the rest of the aligned LLMs are unseen distribution to ReMoDetect. We believe further improving the performance of ReMoDetect by enlarging the training corpus using more aligned LLM will be an interesting future direction to explore, showing an impact on the open-source community.
4.2 Reward Model Analysis
More observation studies. In addition our observation study presented in Table I and Figure 1, we considered (i) more text domains and (ii) different types reward models to rigorously verify our observation (i.e., aligned LLMs generate texts with higher predicted preference compared to human- written texts). To this end, we use pre-trained reward model without further tne-tuning. First, we show that our observation is consistent across multiple text domains (in Figure 3). Interestingly, the predicted reward separation between LGTs and human- wrrtten texts is more significant in Essay and Prompts-small compared to PubMed (i.e., a biology expert written data), possibly implying that alignment training is done more on relatively common texts compared to expert datasets. Second, also observed that LGTT's have higher preference compared models as well (in Figure 4). Intriguingly, a larger reward model within the same model family (i.c., Gemma 7B compared to 2B) shows better separation of the predicted score, showing the possibility of ReMooetcct's 's scaling law, i.c., using a large reward model will improve the detection performance. We also provide more results of our observation studies in Appendix B.5.
numan-written texts on other reward
7

---

Table 4: Robustness against rephrasing attacks. We report the average AUROC (%) before ('Original') and after ('Attacked") ) the rephrasing attack with T5-3B on the Fast-DetectGri benchmark, including XSum, PubMed, and small-sized WritingPrompts. Values the parenthesis indicate the relative performance drop after the rephrasing attack. The bold indicates the best result.
Model
Accuracy
Loglik.
D-GPT
NPR
FD-GPT
Ours
GPT3.5
Original Attacked
93.6
86.1
82,9
96.1
98.7
Turbo
80.5 (-14.0%)
60.3 (-30.0%)
73.5 (-11.3%)
87.2 (-9.3%)
91.4 (-7.4%)
GPT4 Turbo
Original Attacked
91.7
79.9
79,4
95.2
98.9
80.0 (-12.7%)
50.3 (-37.0%)
61.3 (-22.8%)
87.3 (-8.3%)
94.6 (-4.4% %)
Claude3
Original
91.7
81.1
80.3
92.6
98.6
Opus
Attacked
80.5 (-15.8%)
55.2 (-32.0%)
60.1 (-25.2%)
81.6 (-11.9%)
91.1 (-7.1 %)
Reward distribution change after training. We additionally amalyze the predicted reward distribution change made by our training ob- jective Eq (2). To this end, we visualize the reward distribution before and after the train- ing the reward model by using GPT4-Turbo generated texts on Eassy domain. As shown in Figure 5, our training objective indeed in- creases the separation of the predicted reward distribution between human-writte texts and LGTs. Interestingly, the LGT's reward dis- tribution becomes more compact and equally higher, whereas the reward distribution of human-written texts becomes more dispersed. We conjecture that this difference arises because wwrtten texts are produced by diverse individ- uals with varying backgrounds and experiences, while aligned LLMs share somewhat similar training receipts across models.
Human Machine
5
(a) Before
(b) After
Figure 5: Predicted reward distribution of human writ-
ten texts and LGTs (a) 'Before' and (b) 'After' train- ing the reward model with Eq (2). 'Machine' denotes GPT4-Turbo generated texts on Eassy domain.
4.3 Additional Analysis
In this section, we provide more analysis of ReMoDetect. Here, we mainly consider baselines that show effectiveness in the main experiment (e.g., Fast-DetectGPT in Table 2) and consider the GPT4 family and Claude3 as aligned LLMs.
Robustness to unseen distributions. We verify the claim that training de- tectors on specific LGTs may intro- duce bias and require careful training by showing the failure cases of the prior work and the robustness of ReMoDetect to unseen distributions. To this end, we compare ReMoDetect with ChatGPT- Detector, which is trained on the same dataset (i.e., GPT3.5 Turbo generated texts on the HC3 domain) and evaluate on the unseen do- main (i.e., WritingPrompts-small and machine (i.e., Claude3 Opus). As shown in Table 5, both ReMoDetect and ChatGPT-Detcctor work well on the seen domain and LLM, while ReMoDetect shows significant robustness to unseen distributions compared to ChatGPT-Detectee the AUROC of SeaiGPP-Detecttrr in the seen domain dropped from 99.8%%42..1 1% when tested on the unseen domain while ReMoDetect retains the original accuracy, i.e., 99.9%.99.5%%
Table 5: AUROC (%) of ChatGPT-D and ReMoDetect
(ours), on datasets and models that are seen (S) or unseen (U) during training time. The bold denotes the best results.
Domain
HC3 (S)
HC3 (S)
WP-s (U)
Model
GPT3.5 (S)
Claude3 (U)
Claude3 (U)
CharGPT-D
99.8
96.7
24.1
Ours
99.9
99.9
99.5
For instance,
Robustness against rephrasing attacks. One possible challenging scenario is detecting the rephrased texts by another LM (known as rephrasing attacks) [42], i.e., first generate texts with powerful LLMSs and later modify them with another LLM. To this end, we follow the prior work by using a T5-3B specifically trained for rephrasing attack [42]. As shown in Table 4, ReMoDetect significantly and consistently outperforms all baselines. It is worth noting that our relative drop performance also significantly lower than other baselines, indicating strong robustness of ReMoDetect.
8

---

Log-likelihood
LRR
DetectGPT
Fast-DetectGPT
Ours
100
100
AUROC (%)
80
80
60
GPT4 Turbo
(b) Claude3 Opus
50
100
150
200
50
100
150
200
# of words
# of words
Figure 6: Average AUROC (%) of various LGT detection methods various input response lengths by monotonically increasing words each. We consider three text domains from the Rast-DetectGPT benchmark and two aligned LLM, including (a) GPT4 Turbo and (b) Claude3 Opus.
Table 6: LGT Detection results non-RLHF trained LLMs. We report AUROC (%) of multi- ple LGT detection methods, including log-like ligeliiklhhodd (Loglik.), Rank, Sett-DetecGPTT (FD-GPT), OpenAI-Detecee (Open-D). ChatGPT-Detecce detection benchmarks from Fast- Dest-DetectGPT: PubMed, XSum, and WritingPrompts- Here, Phi-3 medium is DPO trained and OLMo-7B-SEB is SFT-only trained. The bold indicates the best result within the group.
(Chat-D), and ReMooetect (Ours). We consider LGT
(WP-s).
Model
Domain
Loglik.
Rank
FD-GPT
Open-D
Chat-D
Ours
PubMed XSum
65.0
56,2
63.7
37.7
80.7
94.5
Phi-3 mini
70.3
64.1
91.0
82.7
23.4
97.6
WP-s
82.4
73.
96.7
60.0
31.1
99.3
PubMed
57.2
50.4
59.9
31.9
82.7
91.7
Phi-3 small
XSum
69.7
95.6
79.3
19.5
98.7
WP-s
84.0
72.3
97.2
58.6
32,2
97.4
PubMed
65.4
55.4
61.7
34.2
15.8
95.2
Phi-3
XSum
64.5
61.2
85.4
75,0
18.1
98.0
medium
WP-s
83.1
73.6
95.7
53.9
38.5
98.8
PubMed
88.4
60.5
92.8
62.0
23.6
94.1
OLMo 7B-SFT
XSum
96.6
66.0
99.1
97.3
5.9
98.1
WP-s
98.1
78.5
98.8
95.2
19.5
99.2
Average
86.0
63.8
91.2
72.2
43.8
95.3
Robustness on input response length. By following the prior work [13]. we also measure the robustness of ReMoDetect on the input response length (i.e., # of words in y). Note that shorter responses are hard to detect as there is less evidence to identify the characteristics of humans and LLMs. shown Figure 6, ReMoDetect significantly outperforms the major baselines. Interestingly, our method can even outperform the best baseline with 71.4% fewer words, showing significant robustness on short input responses. For instance, Fast- Dest-DetectGPT reaches AUROC of 91.8% with 2 210 words, while ReMoDetect reaches 94. % with 60 words under Claude3 Opus.
ReMoDetect for non-RLHF aligned LLMs.. We additionally consider aligned LLMs that do reward models for alignment training, i.e., non-RI.H RFF trained L.LMss To this end, we consider aligned LLMs that use Direct Preference Optimization (DPO) [30], an alternative alignment training RLHF. Note that recently released Phi-3 [25] only uses DPO (followed by supervised fine-tuning; SFT) for alignment training and shows remarkable performance in various domains, thus being considered an aligned LLM in our experiment. As shown in Table 6, ReMoDetect also outperforms baselines in all cases, showing that our method can be applicable even if aligned LLMs are not trained with reward models. Furthermore, we also considered the detection scenario for the SFT-only model that does not use the alignment training. Here, we observe that ReMoDetect effectively detects the LGTs from the SFT-only model as well as outperforming other baselines. We believe this is because the SFT implicitly trains the model to reflect the human preference from the instruction tuning dataset [10], thus making the ReMoDetect well-detect the texts from SFT models.

---

Table Contribution of each proposed component of ReMoDetect on detecting aligned LGTs from wwrtten texts. We report the average detection performance of GPT4 under text domains in the Fast- DetectGPT benchmark. All values are percentages, and the best results are indicated in bold.
Continual Fine-tuning (No Replay)
with Replay
Mixed Text Reward Modeling
TPR
AUROC
AUPR
Buffers
FPR 1%
79.0
79.2
16.7
90.5
91.0
38.9
95.5
95.8
59.3
97.9
98.0
77.0
Table 8: Comparison of detection time, model parameters, and average AUROC (%) of Fast- DetectGPT benchmark for various LGT detection methods. Detection time was measured in an A6000 GPU, and the overall detection time was measured for 300 XSum dataset samples.
Method
Detection Time (sccs)
Model Parameters
AUROC
lig-liicl
11.7
2.7B 3B & 2.7B
88.6
DetectGPT
7738.8
79.2
NPR
7837.3
& 2.7B
78.7
Fast- DetectGPT
62.7
6B & 2.7B
91.9
Ours
8.7
0.5B
95.8
Component analysis. We perform an analysis on each component of our method in detecting GPT4 generated texts: namely, the use of (i) continual fine-tuning with no replay À = 0, (ii) the replay buffers, and (iii) the reward modeling with Human/LLM mixed texts, by comparing multiple detection performance metrics. Results in Table 7 show each component is indeed important, where gradually applying our techniques shows a stepwise significant improvement.
Inference time efficiency. In Table 8, we compared detection time, model parameter size, and average AUROC on the Fest- benchmark. The detection time was measured in an A6000 GPU, and the overall detection time was measured with 300 samples of the human/GPT3.5 Turbo XSum dataset. ReMoDetect shows the best average AUROC performance among the methods, but 7.2 times faster, and uses a 17.4 times smaller model than the second best model, Fast- Dest-DetectGPT.
Discussion and Conclusion
We propose ReMoDetect, novel and effective LLM--geerraed text (LGT) detection framework. Based on the novel observation that the reward model well recognizes LGTSs from human-written texts, we continually fine-tune the reward model to further separate reward scores of two distributions while preventing the overfitting bias using the replay technique. Furthermore, we suggest a Human/LLM, mixed text dataset for reward modeling, learning better decision boundary of the reward model detector. Experimental results further demonstrate that ReMoDetect significantly improves the prior Maae-offthe-aa results in detecting aligned LGTs.
Future works and limitations. We believe it will be an interesting future direction train LLMs using the reward model of ReMoDetect. Making the predictive reward distribution of LGT's more well-spread (like the human-written texts in Figure 5), can a step toward making LLMs more human-like. Additionally, a potential limitation of ReMoDetect is the somewhat lack of accessibility of reward models. While there are some open-source reward models available (that we have used throughout the paper), their number is still limited compared to open-source LLMs. We believe that as the open-source community grows and more pre-trained reward models (or human preference datasets) become available, ReMoDetect will be improved further.
Societal impact. This paper presents ReMoDetect that improves the performance of detecting aligned LGTs. We expect that our approach will show numerous positive impacts by detecting LGTs, such as fake news and academic plagiarism. One possible negative impact can be the improved adversarial mechanism followed by the improved detection method (i.e., ReMoDetect); thus, incorporating such a scenario will be an interesting future direction to explore, where we believe using ReMoDetect to such a scenario can be promising (as it shows robustness in multiple cases in Section 4.3).
10

---

Acknowledgements
We thank Jongheon Jeong and Myungkyu Koo for providing helpful feedback and suggestions, in preparing an earlier version of the manuscript. This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea (overnmen((MSI) Artificial Intelligence Graduate School Program(KAIST) and (NttoNaain IT Industry Promotion Agency), through the Ministry of Science and ICT (Hyperscale Al flagship project).
References
[1] R. Alec, W. Jeffrey, C. Rewon, L. David, A. Dario, and S. Ilya. Language models are unsuper- vised multitask learners. OpenAl blog, vol.
p. 9, 2019.
[2] B. Amrita, K. Tharindu, M. Raha, and L. Huan. Conda: Contrastive domain adaptation for ai- generated text detection. In Annual Conference of the Association for Compulational Linguistics, 2023.
[3] M. Andrew. Laion-ai/open-assittan 2023.
[4] F. Angela, L. Mike, and D. Yann. Hierarchical neural story generation. In Annual Conference of the Association for Computational Linguistics, 2018.
[5] Antropic. Introducing the next generation of claude. https: // www. anthropic. com/ news/ claude- 3- 3-fami ly, 2024.
[6] G. Biyang, Z. Xin, W. Ziyuan, J. Minqi, N. Jinran, D. Yuxuan, Y. Jianwei, and W. Yupeng. How close is chatgpt to human experts? com- parison corpus, evaluation, and detection. CoRR abv/2301.07597, 2023.
[7] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[9] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107 033774444 2021.
[10] Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models. In International Conference on Machine Learning, 2024.
[11] M. Eric, Yoonho, K. Alexander, M. Christopher, and Chelsea. Detectgpt: Zero-shot machine-generann generated text detection using probability curvature. In International Conference on Machine Learning, 2023.
[12] D. Gao, L. Ji, L. Zhou, K. Q. Lin, J. Chen, Z. Fan, and M. Z. Shou. Assistgpt: A general multi- modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.0866 2023.
[13] B. Guangsheng, 7. Yanbin, T. Zhiyang, Y. Linyi, and Z. Yue. Fast-detectgpt: Efficient zero-shot detection of maecinn--gnnerate text via conditional probability curvature. In International Conference on Learning Representations, 2024.
[14] He. J. Gao, and W. Chen. Debertav3: gradient-disentangled embedding sharing, 2023.
Improving deberta using clectra-style pre-training with
[15] X. He, X. Shen, Z. Chen, M. Backes, and Y. Zhang. MGTBench: Benchmarking Machine- Generated Text Detection. arXiv preprint arXhv:2303.14822 2023.

---

[16] D. Hendrycks. M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[17] D. Hendrycks, N. Mu, E. Cubuk, B. Zoph, Gilmer, and B. Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2020.
[18] D. Hendrycks, A. Zou, M. Mazeika, L. Tang, B. Li, D. Song, and Steinhardt. Pixmix: Dream- like pictures comprehensively improve safety measures. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[19] Hong, N. Lee, and J. Thorne. Reference-fre free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.0769) 2024.
[20] S. Irene, B. Miles, Jack, A. Amanda, H.-V. Ariel, W. Jefff, R. Alec, K. Gretchen, K. Jong Wook, K. Sarah, M. Miles, N. Alex, B. Jason, M. Kris, and W. Jasmine. Release strategies and the social impacts of language models. arXiv preprint arXiv: pet::::008000000 2019.
[21] S. Jinyan, Y. Z. Terry, W. Di, and N. Preslav. Detectlim: zero-shot detection of machine-generatle text. arXiv preprini arXiv:2306.0553 055400 2023.
Leveraging log-rank information for
[22] L.. Jooyoung, L. Thai, C. Jinghui, and L.. Dongwon. Do language models plagiarize? In In Proceedings of the ACM Web Conference, 2023.
[23] W. Laura, M. John, R. Maribeth, G. Conor, U. Jonathan, H. Po-Sen, C. Myra, G. Mia, B. Borja, K. Atoosa, 7. Kenton, Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, I.. Rimell, A. H. Lisa, I. William, L. Sean, I. Geoffrey, and G. lason. Ethical and social risks of harm from language models. arXiv preprint arXiv:21/2.04339, 2021.
[24] K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence-calibrated classifiers for detecting out-of-distributio samples. In International Conference on Leurning Representations, 2018.
[25] Microsoft. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
[26] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:230.008774, 2023.
[27] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, Agarwal,
K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
[28] A. Panickssery, S. R. Bowman, and S. Feng. Lim evaluators recognize and favor their own generations. arXiv preprint 2rXiv:2404.13077 13076, 2024.
[29] J. Qiao, D. Bhuwan, L.. Zhengping, C. William, and L. Xinghua. Pubmedqa: A dataset for biomedical research question answering. In n Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
[30] R. Rafailov, A. Sharma, Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct prefer- ence optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023.
[31] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
[32] Z. Rowan, H. Ari, R. Hannah, B. Yonatan, F. Ali, R. Franziska, and C. Yejin. Defending against neural fake news. arXiv preprint arXiv:1905. 12616, 2021.
[33] I.. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. In International Conference Learning Representations, 2020.
[34] G. Sebastian, Hendrik, and M. R. Alexander. Gltr: Statistical detection and visualization of generated text. In Annual Conference of the Association for Computational Linguistics, 2019.
12

---

[35] N. Shashi, C. Shay, B, and L. Mirella. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Conference on Empirical Methods in Natural Language Processing, 2018.
[36] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbren- ner, I. Sutskever, T. Lillicrap. M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Master- ing the game of go with deep neural networks and tree search. Nature, 529:484- 5038, 2016. URL http: //www nature. com/ htt.rnattree/eeeeeeeeeaaaaaaaeea /ttp:///////////////////////////
html.
[37] Tack, S. Mo, Jeong, and J. Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
[38] G. Team. Gemini: family of highly capable multimodal models. arXiv:2312.11805 2023.
arXiv preprint
[39] E. Tian and A. Cui. Gptzero: Towards detection of ai-generated text using zero-shot and supervised methods, 2023. URL htttps://gttee me.
[40] Y. Tian, H. Chen, X. Wang, Z. Bai, Q. Zhang, R. Li, C. Xu, and Y. Wang. Multiscale positive-unlabeled detection of ai-generated texts. In International Conference on Learning Representations, 2023.
[41] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09988: 2023.
[42] S. Vinu, Sankar, K. Aounon, B. Sriram, W. Wenxiao, and F. Soheil. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303. ///56 2023.
[43] V. Vivek, F. Eve, T. Nicholas, and K. Dan. Ghostbuster: Detecting text ghostwritten by large language models. CoRR abs/2305.15047, 2023.
[44] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. htttpp///ggttuu....o////iggg/g/o
orrersso--rraaooorrr--rr jax, May 2021.
[45] A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does Ilm safety training fail? In Advances in Neural Information Processing Systems, 2023.
[46] D. Xuan- -uuuuQqq L. Ngoc- Bich. P. Xuan-Dung, N. Bac-Bien, chatgpt and microsoft bing ai chat performances on physics exams vietnamese national high 2023.
Bien, and V. The-Duy. Evaluation of
school graduation examination. arXiv preprint arXiv:2306.00
[47] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In International Conference on Learning Representations, 2018.
[48] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 1.XXv::3077.1043,, 2023.
13

---

Appendix
A Experimental Details
In this section, we describe the experimental details of Section 4, including ReMoDetect and baselines.
A.1 Dataset Details
In this section, describe the dataset we used in training and evaluation. Also, explain how generated the additional datasets.
HC3. HC3 a question-ande and-answering dataset that consists of answers written by humans and generated by ChatGPT corresponding to the same questions. The dataset is collection of several domains: reddit eli5, open_queeeeeeepoep wiki csai, medicine, and finance. We used training samples of 2,200 and validation samples of 1, 1,000, which is the same subset of HC3 as the prior work [6, 40]. We used the filtered version of the HC3 dataset.
Reuters. Reuters is news dataset that consists of news articles written by humans and generated by LLM corresponding to the same subjects. We brought the dataset from MGTBench [15] and followed the construction recipe generate more evaluation datasets for recent LLMs. The dataset comprises 1,000 news articles written by humans and generated by LLM, GPT3.5 Turbo, GPT4 Turbo, Claude, Claude Opus, Llama3 70B instruct. GPT3.5 Turbo and Claude dataset is from MGTBench [15]. We made the same evaluation set for Essay and WritingPromppss.
Essay, Essay consists of essays extracted from IvtPandas. We brought the dataset from MGT- Bench [15] and followed the construction recipe generate more evaluation datasets for recent LLMs. The dataset consists diverse essay subjects across various academic disciplines. The dataset comprises 1,000 samples of Essays written by humans and generated by aligned LLMs.
WritingPrompts. WritingPrompts is the creative writing prompt shared on r/WritingProme of Reddit. We brought the dataset from MGTBench [15] and followed the construction recipe to generate more evaluation datasets for recent LLMs. The dataset comprises 1,000 samples of WritingProopts written by humans and generated by LLMs.
WrittingPrompt--smalll Writinggrommps--mall is the creative writing prompt shared on Reddit /WWritigPProppls. We brought the dataset from FastDetectGPT [13] and followed the construction recipe to generate more evaluation datasets for recent LLMs. The dataset comprises 150 samples WritingPrompts written by humans and generated by LLM.
XSum. Xsum news dataset comprising news articles written by humans and generated by LLM corresponding to the same subjects. We brought the dataset from FastDetectGPPT [13] and followed the construction recipe generate more evaluation datasets for recent LLMs. The dataset comprises 150 news articles written by humans and generated by LLMs.
PubMeds. PubMed a question-ande ineestion-and-anseee andwanngginng dataset of biomedical research domains written by humans and generated by LLMs corresponding the questions. We brought the dataset from FastDetectGPT [13] and followed the construction recipe to generate more evaluation datasets for recent LLMs. The dataset comprises 150 QA pairs written by humans and generated by LLMs.
Human/LLY mixed datasets. We rephrase the human-written text from the HC3 dataset using Llama3 70B instruct [41]: We first select 50% of the indices in the paragraph, then rephrase selected sentences using the following prompt to the rephrasing LLM:
Please paraphrase sentence numbers <idxlist> in given written texts.
<ith> sentence: <xxx>
<i+lth sentence: <xxx>
The <idxlist> is 50% randomly selected index list of sentences like "[0,2,5,7]*. Then list all the sentences of the passages like "<5th> sentence: A fellow high school student, typically a or there's 's a lot of stress involved."
14

---

A.2
Aligned LLM Spec Details
The API version of our dataset is as follows:
OpenAI//PPa GPT3.5 Turbo
gpt-3.5-turbo-003
OpenAI/ GPT4 gpt-4
OpenAI GPT4 Turbo go0-4-turbo-222400---0
Anthropic / Claude3 Opus claude -laudde33-pus--2220000
Anthropic/ Claude3 Sonnet claude-3-sonmet -oodde---oomee--2222222
Anthropic/ Claude3 Haiku Claude-3-haikku22240330
Google Gemini pro gemini 2024-02-01
We the open-source model for Llama3 70B instruet' and Phi-3 [25]. Here, Phi-3 with a 4K context length for mini and medium', whereas we use an 8K context length for Phi-3 small (Phi-3 small only has 8K model). We spent $56.0 for OpenAl API and $156.6 for Anthropic API.
A.3 Training and Evaluation Details
Training details of ReMoDetect. We use AdamW optimizer with learning rate of 2.0 10-3 with 10% warm up and cosine decay and train it for one epoch. For the À constant for regularization using replay buffer, we used mixed data, we used 0.3 and 0.3. As for the replay buffer datasets, we use 'Anthropichhh-rhie sabhoss/yythheti--nnsteeee instticctttttt--e-aaa-. gptj-pairwisee from the huggingface datasets library as our base reward model [3] used these datasets for training. We use the same batch size for the training sample and replay buffer sample, which ends up with total batch size of four.
0.01. For the B1, 82 parameters that choose the contribution of the and
Reward model details. We mainly used the open-source reward model from OpenAssistant which is based DeBERTa-Ve -a--Ta-----aggee [14]; the model parameter size is 435M and trained with human preference dataset. Additionally, in Figure 4, we used other reward models, weqweasdas/RM- and
from the huggingface library in order to verify our observations in other reward models.
Gemma-2B Swtpwasdaa/RM-GGmmmet
Detection metrics. For the evaluation, measure the following metrics verify the effectiveness of the detection methods in distinguishing numan-writted texts and LGTs.
True positive rate (TPR) % false positive rate (FPR). Let TP, TN, TN, FP, and FN denote true positive, true negative, false positive, and false negative, respectively. We measure TPR TP/ (TP+FN) when FPR FP (FP+TN) is %.
Area under the receiver operating characteristic curve (AUROC). The ROC curve is graph plotting TPR against the false positive rate FP/ (FP+TN) by varying a threshold.
Area under the precision-recall curve (AUPR). The PR curve is graph plotting the precision TPP (TP+FP) against recall TP/ (TP+FN) by varying threshold.
Resource Details. For the main development, we mainly use Intel(R) Xeon(R) Gold 6426Y CPU @ 2.50GHz and a single A6000 48GB GPU.
hattpps//uuggggggeoo...//eeeeaas Llama 70B- Instruct
hutpepe//uuggggggee..cooooottoss 4k- instruct
instruct
instruct
pairrise
largere-
15

---

A.4 Robustness Evaluation Details
Rephrasing attack. To check the robustness of our method against rephrasing attacks, we utilized 75-3B-based paraphraser [42] to paraphrase the sentences in the passage. We conducted experiments to max length = 256, top_ k = 200, top_p = 0.95. The result is in Table 4.
with
Input response length. To check the robustness of our method against input response length, truncated the given test dataset to various word lengths. First, we tokenized the given paragraph into words using the nltk framework. Then, truncate each passage into target word lengths. We tested for word length E [30, 60, 90, 120, 150, 180, 210). The result is in Figure 6.
A.5 Baseline Details
We describe baselines that compared with ReMoDetect in Fast-DetectCHP benchmark [13] and MGTBench [15]. We use implementations and backbone models introduced Fast-DetectGPT [13].
Log-likelihood, Rank [20]. These methods use LLM to measure the token-wise log probability and rank of the words, then average the metric of each token generate score for the text. For the baseline experiments, we utilized GPT-neo-2.79 2.7B as their base model.
DetectGPT [13], NPR [21]. DetectGPT, NPR is designed to measure changes in model's log probability and log-rank function when slight perturbations are introduced to the original text. For the baseline experiments, utilized Ge'T-neo-2. 2B as their base model and T5-3B for paraphrasing, and we perturbed 100 for each paragraph.
LRR [21]. LRR used the Log-likelihood log-rank Ratio, which merges the benefits of log- likelihood and log-rank. We utilized GPT-neo-2.7B as their base model.
FaTt-DetectGPP [13]. Fast-DetectGPT shares the same spirt as DetectGPT, where it uses the conditional probability function sampling the text using the base model instead of perturbation using T5 models, thus showing efficiency. Following the original paper setting, we used GPT-J as base model and GPT-neo-2.7B as a scoring model.
OpenAI-Detector [20]. OpenAl-Detector is a RoBERTa- supervised finetuned model trained with pairs of human-written and GPP2-generate texts.
ChatGPT-Detector [6]. CharGPT-Detecttr is GoBERTa-based supervised finetuned model trained with the HC3 dataset, which consists of human-written and ChatGPT generated texts.
B
Additional Experimental Results
B.1
Comparison with Additional Baselines
Table 9: AUROC(%) on MGT benchmark[ 15] for different baselines: Log Rank [13), Entropy [34], and GLTR [34]. The bold indicated the best result.
Model
Domain
GPT 3.5 Turbo
GPT4 Turbo
Llama3 70B
Gemini pro
Claude
Essay
98.1
96.7
98.7
97.9
89.1
Log Rank [13]
Reuters
98.6
95.8
99.7
99.7
85.5
WP
86.5
90.5
95.3
87.6
79.9
Essay
94.1
90.2
91.9
89.0
84.1
Entropy [34]
Reuters
77.8
75.5
78.6
78.3
77.9
84.0
85.4
82.0
64.1
80.9
Essay
97.8
95.9
98.7
97.8
87.1
GLTR [34]
Reuters
98.4
94.8
99.5
99.6
84.7
WP
85,9
88.4
95.2
85,9
79.1
Essay
100.0
99.9
100.0
100.0
99.7
ReMoDetect
Reuters 99.9
99.9
100.0
100.0
99.8
WP
100.0
99.9
99.8
99.8
99.1
In Table 9, we compare other baselines Log Rank [13], Entropy [34], GLTR [34], and ReMoDetect on MGT benchmark. ReMoDetect consistently outperforms other basclines in MGT benchmark.
16

---

B.2
Comparison on Additional Aligned LLMs
Table 10: AUROC(%) on Fast-DetectGPT benchmark [13] for different models: Claude3 Haiku [5] and Sonnet [5].
The bold indicates the best result.
Model
Domain
Loglik.
Rank
D-GPT
LRR
NPR
FD-GPT
Open-D
Chat-D
Ours
PubMed
87.0
60.9
67.5
75.5
66.9
90.9
56.2
28.3
96.3
Claude3
XSum
96.2
73.8
91.9
93.0
90.6
99.8
99.8
Haiku
WP-s
78.8
93.1
94.8
99.7
82.4
27.9
99.8
PubMed XSum
84.4
60.6
64.9
71.8
64.5
86.5
52.4
31.0
96.4
Claude3
90.1
70.9
84.4
86.2
84.1
94.7
76.0
13.7
98.7
Sonnet
WP-s
94.9
77.7
87.5
93.2
98.0
35.6
99.7
In Table 10, we evaluate Claude3 Haiku and Claude3 Sonnet, which are serviced by Anthropic and are smaller versions of Claude3 Opus. ReMoDetect consistently outperforms other baselines in the evaluation, demonstrating that our detector can detect these smaller models effectively.
B.3 Additional Performance Metric
Table 11: TPR(%) at FPR 1% and AUPR (%) of multiple LLM- -generateeed text detection methods, including ligllikeldoodd (Loglik.) [20], Rank [20], DetectGPT (D-GPT) [11], LRR [21], NPR [21], Fast-DetectGPT (FD-GPT) [13], OpenAl-Detector (Open-D) [20], ChatGPT-Dettctor (Chat- D) [6], and ReMoDetect (Ours). We consider I.I. -..--generated text detection benchmarks from
Fast- Mast-DetectGPT [13]. The bold indicates the best result within the group.
(a) TPR at FPR 1%
Model
Domain
Loglik.
Rank
D-GPT
LRR
NPR
FD-GPT
Open-D
Chat-D
Ours
PubMed
10.7
4.0
0.0
8.0
5.3
44.0
2.0
1.3
63.3
GPT3.5
XSum
68.7
12.7
25.3
47.3
15.3
82.0
46.0
0.0
96.7
Turbo
WP-s
64.7
13.3
28.0
28.7
37.3
87.3
0.0
97.3
PubMed
8.7
3.3
0.0
6.0
5.3
18.0
2.7
1.3
70.0
GPT4
XSum
24.0
1.3
1.3
11.3
6.7
32.7
13.3
0.0
79.3
WP-s
9.3
2.7
10.7
2.7
2.0
44.0
1.3
0.0
82.0
PubMed
12.7
4.7
0.7
13.3
4.7
27.3
0.7
0.0
67.3
GPT4
XSum
46.3
8.8
9.5
46.3
10.9
68.0
42.9
0.0
99.3
Turbo
WP-s
60.4
18.8
15.4
41.6
34.2
80.5
11.4
0.0
98.7
PubMed
14.0
5.3
0.7
12.0
4.0
26.0
1.3
0.7
62.7
Calude3 Opus
XSum
42.7
11.3
26.7
44.7
24.0
75.3
43.3
0.0
97.3
WP-s
54.7
16.7
37.3
24.0
55.3
76.7
8.0
0.7
96.0
(b) AUPR
Model
Domain
Loglik.
Rank
D-GPT
LRR
NPR
FD-GPT
Open-D
Chat-D
Ours
PubMed
86.5
62.8
55.1
73.7
62.4
90.8
61.5
36.5
96.9
GPT3.5 Turbo
XSum
95.3
88.2
91.6
85.5
99.2
93.4
32.0
99,9
WP-s
81.6
94.0
89.3
94.1
71.0
37.8
PubMed
79.9
60.5
54.7
67.0
59.7
84.4
55.5
38.8
96.7
GPT4
XSum
80.1
65.4
63.2
75.6
62.5
91.1
73.8
58.6
98.7
WP-s
81.6
68.1
79.4
66.1
74.1
96.0
50.2
46.5
98.7
PubMed
85.0
62.8
59.1
74.5
61.6
89.4
56.1
38.6
97.4
GPT4
XSum
91.5
75.7
81.3
89.7
81.4
97.6
90.8
31.0
100,0
Turbo
WP-s
97.6
82.9
91.5
93.1
92.5
99.4
74.0
36.6
99.8
PubMed
84.6
62.7
60.1
74.4
60.0
88.4
52.1
40.4
96.7
Calude3
XSum
93.8
74.2
85.5
90.7
85.0
96.9
89.4
31.1
99.9
Opus
WP-s
96.4
78.4
92.9
87.9
93.2
95.4
69.3
38.4
99.5
In Table 11, report the LGT detection performance of ReMoDetect and other detection baselines by considering additional performance metrics, including true positive rate (TPR) at 1% false positive rate (FPR) and Area under the mecision-recall curve (AUPR). As shown in the table, ReMoDetect significantly and consistently outperforms all cases.
17

---

B.4
Dataset Examples and Predicted Reward Scores
Table 12: Dataset examples with their model, domain, and ReMoDetect score.
ReMo-D
Model
Domain
Example
Score
You are a teenager with the ability to measure how "Dangerous" people are on a scale from to 10 just by looking at them. normal child would be a 1, while a trained man with an assault rifle might be 7. Today, you notice the unassuming new kid school measures 10. When people, I see numbers. Just floating there. I can't remember since when it's begun, but that's not important. An innocent baby would measure as 1. A fellow high school student, typically a 3 or 4 there's 's a lot of stress involved. The day before my best friend stabbed his entire family to death and hung himself, he was abnormally quiet- 9. Just seeing a living being's 's image is enough: on television, the rioters Ferguson measured around and 7. When was daring enough to watch videos of ISIS members committing atrocious murders: 9. That day
Human
WP-s
-6.01
You teenager with the ability to mcasure how "Dangerous" people scale from 10 just by looking at them. This unusual power has made you both cautious and curious about the people you encounter in your daily life. One day, while walking through a busy market, you notice man with a danger level of 9. Intrigued, you decide to follow him discreetly, keeping safe distance. As you observe his interactions, you notice he has commanding presence and seems some shady dealings. However, as you continue to watch, you see him also display moments of kindness, helping an elderly woman with her groceries and playing with group of children. This experience teaches you that people are complex and -ilti--iimesionnl, and that danger levels don't necessarily define someone's entire character. You learn use your power with more discernment, understanding that everyone has both light and dark sides.
GPT4 Turbo
involved
WP-s
4.91
Regina Hungerford, 56. denied striking 17-year-old as he listened music class at Merthyr Tydfil College. But disciplinary panel ruled contact with the pupil was probably made and her conduct had heen "unacceptable", On Thursday, the Education Workforce Council imposed the suspension, saying: "The public interest is in favour of her heing ahle teach again." Mrs Hungerford admitted shouting and slam- ming a book on the desk of the "provocative and disruptive" teenager as he listened to rap music in her classroom but always denied hitting his head. She was cleared, on appeal. of criminal charge for assaulting the pupil during maths lesson for those with Iearning difficulties. But in November, a disciplinary panel found, on the balance of probabilities, she had made physical conduct with the pupil on the head or hand. The panel found she had
Human
XSum
-1.70
Regina Hungerford, 56, denied 17-year-old while she listened to music in her class Merthyr Tydfil College. The incident occurred when the student refused to take his headphones off during lesson. Hungerford. who has taught at the college for more than a decade, maintained her innocence throughout the investigation. Witnesses claim that the altercation began when Hungerford approached the student and requested that he stop listening to music and focus on lesson. The student apparently ignored her request, resulting in a heated exchange. Several classmates reported seeing Hungerford striking the student while others stated that they did not witness physical contact. The college administration has launched internal investigation into the matter and Hungerford has been suspended pending the outcome, The student's 's family has been notified and the local authorities are also examining the incident. The college has declined to comment on the matter.
Claude3 Opus
XSum
4.04
In Table 12, we show dataset examples and their ReMoDetect score.
18

---

B.5 Additional Observational Studies
(a) Reuters
Essay
WritingPrompts
Figure 7: t-SNE of the reward model's final feature in multiple domains Reuters, Essay, Writing- Prompts generated by GPT3.5/GPTS Turbo, -lama3-70B-instrue,,, and Claude3 Opus.
(a) Reuters
Essay
WritingPrompts
Figure 8: t-SNE of the ReMolDetect's final feature in multiple domains Reuters, Essay, Writing- Prompts which generated by GPT3.5/GPT4 Turbo, L and Claude3 Opus
Human Machine
-5
5
-10
5
(a) Reuters
(b) Essay
(c) WritingPrompts
Figure 9: Reward distribution the reward model in multiple domains Reuters, Essay, Writing- Prompts generated by GPT4 Turbo, and Claude3 Opus.
Human Machine
-5
0
5
-5
0
5
-5
(a) Reuters
(b) Essay
(c) WritingPrompts
Figure 10: Reward distribution of the ReMoDetect in multiple domains Reuters, Essay, Writing- Prompts which generated by GPT4 Turbo, and Claude3 Opus.
In Figure 7, and Figure 8, we present t-SNE of the reward model and ReMoDetect. Figure 9 and Figure 10 display the reward distribution. These figures demonstrate that, even without further training, the reward model can distinguish between human-written texts and LGT. Additionally, ReMoDetect emphasizes the separation between human-written written text and LGT.
19

---

B.6 Robustness of Reward Models against Rephrasing Attacks
Table 13: Robustness against rephrasing attacks. We report the average AUROC (%) before ('Orig- inal") and after ('Attacked") the rephrasing attack with T5-3B on the Fast- Dett-Deee benchmark, including XSum, PubMed, and performance drop after the rephrasing attack. The bold indicates the best result.
Values in the parenthesis indicate the relative
Ours
Ours (ReMoDetect)
Model
Accuracy
Loglik.
D-GPT
NPR
FD-GPT
(reward model)
Original Attacked
82.1
69.0
68.1
90.6
79.0
97.9
GPT4
63.7 (-22.4%)
44.8 (-35.1%)
47,0 (-31%)
74.5 (-17.7%)
71.2 (-9.9%)
87.2 (-10.9%)
Llama3 70B
Original
93.5
84.9
84.9
96.8
80.9
98.5
Attacked
79.9 (-14.5%)
61.7 (-27.4%)
64.7 (-23.7%)
87.9 (-9.2%)
(-12.3%)
88.3 (-10.4%)
Gemini
Original
71.3
75.8
79.9
81.8
pro
Attacked
64.9 (-18%)
50.7 (-28.9%)
55.7 (-26.6%)
64.5 (-19.3%)
55.8 (-13%)
67.4 (-17.6%)
In Table 13, we compare the robustness against the paraphrased attack of the reward model and other baselines including ReMoDetect. The experiment shows that the reward model is robust against paraphrasing attacks (i.e. reward model and Re.MoDetect are the two least drops against paraphrasing attacks). From the results, we hypothesize that the robustness against attack came from the reward model itself. Conceptually the human preference for the text samples doesn' 't change much as the distribution shifts paraphrases some words, hence, the reward score independent of the minor variation of the sentence. We believe that the result of the experiment supports our hypothesis. Furthermore, exploring the characteristics and applications the reward model would be interesting in the future.
B.7 Additional ReMoDetect Models Trained From Differently Initialized Reward Models.
Table 14: Comparison of multiple ReMoDetect models trained from reward models, including deberta, Gemma-2B (G. 2B), Llama3-8B (I.. 8B). We report the average AUROC (%) on the fastdetectGPTT benchmark, including PubMed, XSum, and WridingPrompts-small (WP-s).
Ours (deberta)
Ours
Ours
Model
Domain
FD-GPT
Open-D D
(G. 2B)
(L. 8B)
PubMed
90.2
61.9
96.4
90.1
94.7
GPT3.5
XSum
99.1
91.5
99.8
100.0
100.0
Turbo
WP-s
99.2
70.9
99.9
99.9
99.7
PubMed
85.0
53.1
96.1
91.4
92.1
GPT4
XSum
90.7
67.8
98.8
99.9
100.0
WP-s
96.1
50.7
98.7
99.6
99.4
PubMed
88.8
55.8
97.0
91.2
92.9
GPT4 Turbo
XSum
97.4
88.2
99.8
100.0
100.0
WP-s
99.4
72.3
100.0
100.0
100.0
PubMed
90.8
52.9
96.3
91.8
94.3
Llama3 70B
XSum
99.7
96.2
99.5
100.0
99.9
WP-s
99.9
77.5
99.8
99.6
99.6
PubMed
82.1
57.3
85.6
78.8
81.8
Gemini pro
XSum
79.5
72.2
88.2
87.5
85.3
WP-s
78.0
70.2
71.6
84.2
89.2
PubMed
88.2
48.9
96.4
90.9
93.3
Calude3 Opus
XSum
96.2
86.2
99.5
99.9
99.8
WP-s
93.5
65.7
99.9
99.7
99.8
Average
91.9
68.9
95.8
94.6
95.6
We additionally consider the ReMoDetect models trained from differently initialized reward models. To address the consideration, we conducted experiments to train ReMoDetect using three reward models. As shown Table 14, ReMoDetect models consistently outperform other baselines, even though the model trained from differently initialized reward models. Nonetheless, the ReMoDetect's detection performance can vary with initialization. Thus, we suggest interesting future works to find better detector, such ensembling several trained models using an enhanced reward model.
20

---

B.8
Comparison with GPTZero per domain
Table 15: Detection Score of GPTZero [39], commercial black-box LGT detection API. We report AUROC (%) on the Fast-DetectGPT benchmark, including PubMed, XSum, and WritingPrompts.
Model
Domain 3.5 Turbo
GPT4
GPT4 Turbo
Llama3 70B
Gemini pro
Claude3 Opus
PubMed
88.0
84.8
87.2
90.1
83.2
88.0
GPTZero
XSum
99.5
98.2
100.0
100.0
85.8
99.9
WP-s
92.9
82.6
100.0
99.8
79.7
99.1
PubMed
96.4
96.1
97.0
96.3
86.4
96.4
ReMoDetect
XSum
99.8
98.8
99.8
99.5
74.5
99.5
WP-s
99.9
98.7
100.0
99.8
86.4
99.9
In Table 15, we report the performance GPTZero [39] and ReMoDetect in PubMed, XSum, and WritingPrompts (note that Table reports the average AUROC of these domains). It is worth noting that ReMoDetect outperforms in most of the cases and consistently shows better performance in PubMed (which is expert domain), indicating the effectiveness ReMotetect on low-data regimes.
B.9 Comparison on Aligned Small Language Models
Table 16: AUROC (%) of multiple LGT detection methods, including log- lig-liiklihood (Loglik.), Rank, PPst-DetectGPT (FD-GPT), OpenAl-Detector (Open-D), ChatGPT-Detector (Chat-D), and ReMoDetect (Ours). We consider LGT detection benchmarks from Fast- Dest-DetectGPT: PubMed, XSum, and
The bold indicates the best result within the group.
Model
Domain
Loglik.
Rank
FD-GPT
Open-D
Chat-D
Ours
PubMed
85.0
60.4
89.6
53.7
33.4
94.6
Llama3
XSum
82.3
68.9
86.8
95.4
13.1
85.4
8B-it
WP-s
87.2
72.3
91.0
81.2
26.4
95.5
PubMed
69.8
55.9
71.6
36.4
85.1
95.1
Gemma2 9B-it
XSum
85.1
69.4
94.0
97.7
WP-s
86.7
71.9
96.6
50.1
70.3
96.8
PubMed
67.9
56.6
72.3
78.1
Gemma2 2B-it
XSum
82.1
18.2
89.8
67.6
97.2
94.9
WP-s
84.6
71.8
99.0
70.8
63.7
94.2
PubMed
82.3
61.0
89.8
62.9
23.9
92.7
Qwen2
XSum
96.5
66.7
98.3
97.2
1.3
99.6
1.5B-it
WP-s
97.5
78.2
98.6
94.3
17.7
99.1
PubMed
88.4
60.5
92.8
62.0
23.6
94.1
OLMo 7B-sft
XSum
96.6
66.0
99.1
97.3
5.9
98.1
WP-s
98.1
78.5
98.8
95.2
19.5
99.2
Average
86.0
63.8
91.2
72.2
43.8
95.3
We additionally consider small aligned models particularly when the model parameter size is smaller than 10B, including Llama3-8b, Gemma-2-9b, Gemma-2-2b. Qwen2-1.5b-i., and Olmo7b-sft. As shown in Table 16. ReMoDetect also effectively detects LGT of small language models. For instance, ReMoDetect achieves 97. 1% average AUROC in Qwen2- 1.5b-it while the second best reaches 84.8%.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim about the observation of reward models and ReMoDetect's detection performance is reflected in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
The paper discussed the limitations of the work in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experimental results of the paper can be reproduced by following our methods, and specific experimental details in Section 4, and Appendix A.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide codes and data with instruction files in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Seeiiiccuuddes/ooeeuuee ott.uummmiiioooooooooooooooooiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n cts.pss.cc/uubic/guuuuu. gesds//////eeeeeee/e /CodeSubmian ss cassoonooll. onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All experiments are conducted with the same and commonly used random seed. Also, our works are consistent in inference time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify our computer resources, API, API cost, and time execution at inference time in Appendix A and Table 8.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /twp:r////eees.eeeeeee httppublic/Ehhcsttiiie ines?
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed our potential positive and negative societal impacts in Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release data models that have a high risk for misuse.
26

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the original paper that produced the code, data, and model and they included the license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We documented our new assets and included them in the anonymized supple- mental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "OISBGlOSl3", "reviewer": "Reviewer_CEwj", "summary": "The paper finds that reward models used in RLHF can detect texts generated by LLMs. Based on this, the paper presents ReMoDetect, a novel method that further trains the reward model using continual preference fine-tuning and a challenging text corpus rephrased by LLMs. ReMoDetect achieves new SOTA on various LGT benchmarks.", "questions": "1. What is the performance difference when using different reward models for initialization? For example, if we use the reward model of LLM A, and then classify the results of both LLM A and LLM B, does the LGT classifier perform better at detecting LGT of LLM A?\n2. If the reward model of a specific LLM is not available and the reward model has learned some specific or undesirable preferences, how does the proposed method perform in this scenario?", "limitations": "The reward models are not available for some closed-source LLMs.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an interesting finding that RLHF reward models can make LLMs generate outputs that align too closely with human preferences, even more so than human-written texts. This motivates the authors to further fine-tune the reward model, which is a well-motivated approach.\n2. The authors conduct extensive experiments showing that the proposed method achieves SOTA on various benchmarks. The ablation study demonstrates the effectiveness of each component in ReMoDetect.\n3. The paper is well-written and easy to follow, with a clear storyline from motivation to proposed methodologies. The two training strategies are simple and reasonable.\n4. Detecting LGT is an important research problem.", "weaknesses": "1. The proposed method relies on the quality of the reward models, such as the training dataset and model parameters. A pre-trained reward model may be biased towards a specific dataset and may not generalize well to all LLMs. Poor initialization of the reward model may harm performance.\n2. Reward models are also LLMs. Using such models for LGT detection involves long inference times.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer CEwj,\n\nWe sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.\n\n---\n\n**[W1] The method relies on the quality and initialization of the RM.**\n\nFirst, we would like to clarify that we have only trained a single reward model for ReMoDetect, which is used across all experiments (i.e., we did not train separate ReMoDetect for individual datasets or aligned LLMs).\n\nNevertheless, we follow the suggestion to show that the method does not rely on an initialized reward model. To address your concern, we conducted experiments to train ReMoDetect using three reward models. As shown in Table 4 in the attached pdf, ReMoDetect models consistently outperform other baselines, even though the model trained from differently initialized reward models. Nonetheless, the ReMoDetect’s detection performance can vary with initialization. Thus, we suggest interesting future works to find a better detector, such as ensembling several trained models or using an enhanced reward model.\n\n---\n\n**[W2] ReMoDetect may involve long inference time.**\n\nWe remark that ReMoDetect is highly efficient in terms of inference time and memory compared to other LGT detection methods (in Appendix B.4). As shown in Table10 in our paper, ReMoDetect is 7.2 times faster and uses a 17.4 times smaller model than the second-best model, Fast-DetectGPT.\nThis is because the recent LGT detection methods require multiple forward of detector LLM to estimate the score (e.g., DetectGPT, NPR, and fast-DetectGPT perturb the text multiple times to capture the probabilistic curvature) while our method only requires a single forward pass to compute the score.\n\n----\n\n**[Q1] Does a ReMoDetect trained from LLM A perform better at detecting LGT of LLM A than other LLMs? LGT classifier detect outputs better from a specific LLM when initialized with that LLM's reward model?**\n\nWe couldn’t verify whether the use of the reward model of LLM A classifies better at detecting LGT of LLM A than LGT of LLM B because we cannot access the reward model of large models (as even opensource models do not opensource the reward models).  However, it is worth noting that we have demonstrated that a single ReMoDetect model can effectively detect LGT in many different LLMs, showing the detection generalization. We also agree it will be an interesting experiment if the RMs of aligned LLMs (e.g., RM used for training Gemini) get open-sourced.\n\n---\n**[Q2 & L1] How does ReMoDetect perform if the reward model of a specific LLM is not available?**\n\nAs we clarified in the response of [W1], we have only trained an open-source reward model for ReMoDetect, which is used across all experiments (i.e., we did not train separate ReMoDetect for individual datasets or aligned LLMs). From the results, we believe ReMoDetect don’t need access to RM of closed LLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mHMFcHoxZy", "reviewer": "Reviewer_kLmr", "summary": "The paper is about a novel and effective approach for LLM-generated text (LGT) detection by making use of the reward model score. The authors observe that LGT often has higher reward model score compared to human-written texts. They then further increase the separating by fine-tuning the reward model to score LGT higher than human-written texts, and use additional LLM-rephrased human-written texts as the median preference text to assist with the learning.", "questions": "- Why GPTZero is not evaluated on MGTBench?\n- While the Table 2 and 3 covers 6 models, why Table 4 only covers 4 models, Figure 4 only covers a combination of 2 models, and Figure 5 only covers 1 model?", "limitations": "- The paper has inconsistency in selecting the results to report in both the main paper and the appendix. See questions for details.\n- The selected benchmarks are mostly scientific writing and news writing, while other commonly used benchmarks in Question Answering, Web Text and Story Generation (as defined in [1]) are not covered.\n\n[1] Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023). A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The evaluation results are very strong on selected benchmarks, outperforming other LGT detection methods.\n- The analysis about robustness on unseen distributions, rephrasing attacks and input response length is solid.", "weaknesses": "- The work lack qualitative analysis on examples. The cases and patterns for errors, and for improvements, are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer KLmr,\n\nWe sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.\n\n---\n**[W1] Lack of qualitative analysis on examples**\n\nThank you for your constructive comment. While we have a portion of qualitative examples in Appendix B.5, we will add more comprehensive qualitative analysis in the final draft.\n\nFirst, existing detection methods struggle to detect short passages, as shown in Figure 6 in our paper. Nevertheless, ReModetect outperforms other baselines for shorter-length passages, and there can be more improvement points for future works.\n\nSecond, human-written text tends to have more grammar errors compared to LGT as shown in the examples below. To verify the observation, we compared the number of grammar errors in the samples from the WritingPrompts-small dataset by using Grammarly [1]. The average error per 100 words for LGT of GPT4 Turbo was 0.486, while for human-written text was 2.278. We believe this observation suggests that ReMoDetect can serve as a text quality detector.\n\n\n**Human - grammar error: 11, ReMoDetect Score: - 5.35**\n```\nA man forgets to water his potted plant for a whole week. This negligence starts a long chain reaction that leads up to World War III. My plant had died. Because my plant died I went to the store to get a new one. On the way there I cut a man off in traffic. The light changed and I was in the right of way thats the important part. The man I cut off was an ambassador to Russia, back in North America for a quick visit with other officials. He was on the phone to a Russian Delegate he was making peace treaties with should escalations ever occur. Because he was cut off he began to scream obscenities into the phone. Before he could explain the Russian Delegate had yelled back and hung up the phone. Before he could call back the Russian had contacted other Delegates to begin non aggressive hostilities. This rise\n```\n\n**GPT4 Turbo  - grammar error: 2, ReMoDetect Score: 4.87** \n\n```\nA man forgets to water his potted plant for a whole week. This negligence starts a long chain reaction that leads up to World War III. The plant, a rare species, is the last of its kind and a crucial ingredient in a serum that can cure a deadly virus. Scientists from around the world are counting on the plant's survival to mass-produce the antidote. When the man finally remembers to water the plant, it's too late. The plant has withered and died. News of the plant's demise spreads quickly, causing panic and fear. Countries begin to blame each other for not doing enough to protect the plant and secure the cure. Tensions rise, alliances are broken, and diplomatic relations deteriorate. Eventually, conflicts erupt, and the world is plunged into\n```\n[1] Grammarly: https://app.grammarly.com/\n\n---\n\n**[Q1] GPTZero on MGTBench.**\n\nThank you for pointing this out. During the initial development, we actually compared ReMoDetect with GPTZero on MGTBench, but omitted the result as both methods showed near 100\\% of detection score, making it hard to extract meaningful conclusions from the result (see the table below). We believe making more hard benchmarks will be an interesting future direction to explore. \n\n\\begin{array}{lcccc}\\hline \n\\text{Model} & \\text{GPT3.5 Turbo} & \\text{GPT4 Turbo} & \\text{Llama3 70B} & \\text{Gemini pro} \\\\ \\newline\\hline \n\\text{GPTZero} & 100.0 & 99.9 & 99.9 & 100.0 \\\\ \\newline\n\\text{Ours} & 100.0 & 99.9 & 99.9 & 99.9 \\\\ \\newline\\hline\n\\end{array}\n\n---\n\n**[Q2 & L1] Inconsistent model on Figure4,5 Table2,3,4.**\n\nIn the main paper, we tried to compare all aligned LLMs in the main experiments (including Table 2,3) while only considering the most recent powerful LLMs (e.g., GPT4 and Claude 3 Opus) for the analysis. Nevertheless, we agree with the reviewer that this can be seen as an inconsistency for selection. To this end, we have conducted additional experiments on more aligned LLMs: GPT3.5 Turbo, GPT4, GPT4 Turbo, Llama3 70B, Gemini Pro, Claude Opus as shown in Figure 1, Figure 2, and Table 1 in the attached pdf.\n\n---\n\n**[L2] Benchmarks are mostly scientific writing and news writing. Benchmarks in QA, Webtext, and Story Generation are not covered.**\n\nWe clarify that we have already covered QA and story generation. Note that according to the reference the reviewer mentioned [1], PubMed is QA, and WP is story generation.\n\n[1] A survey on llm-gernerated text detection: Necessity, methods, and future directions, arXiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FOEIhz5aTU", "reviewer": "Reviewer_dzQj", "summary": "This paper proposes a method named ReMoDetect to use a reward model for model-generated text detection. Firstly, The authors find that the existing reward model can easily distinguish human-written text from language-model-generated responses. Then, the authors propose two techniques, 1) continual preference training and 2) mixed human and LLM responses, to further train the reward model for LLM-generated text detection. Experimental results demonstrate the effectiveness of the proposed reward model based LLM-generated text detection.", "questions": "- Is there some correlation between the reward model accuracy and the corresponding LGT detection accuracy?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear, and the proposed method to use a reward model for detection is original. \n- Based on the experimental results, the proposed method is effective across multiple LLMs and different domains.\n- The paper is well-organized and easy to follow.", "weaknesses": "- To evaluate a response using a reward model, it requires both the given context or prompt x and the response y. However, the prompt x is not always available in the LLM-generated text (LGT) detection problem. It would be useful to illustrate how the corresponding prompts are determined when evaluating the proposed models. And it would be great to additionally evaluate the proposed method on datasets without prompts.\n- It is unclear why the reward model can recognize the LLM-generated texts from human-written responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer dzQj,\n\nWe sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.\n\n---\n\n**[W1] How the corresponding prompts are determined when evaluating the proposed models. RM works for the prompt $x$ given. What about prompt ungiven cases?** \n\nWhile the initial context $x$ and the generation $y$ (by LLM or human) is explicitly defined in the training dataset and objective, the RM only observes the concatenation of $x$ and $y$ (i.e., the full paragraph) to predict the reward score. Therefore at test-time, we also give the full paragraph to the RM without any indication of initial context and generated part as in the training setup.\n\n---\n\n**[W2] Unclear why RM can recognize LGT from human-written responses**\n\nWe believe it is because of the alignment training objective that recent LLMs have utilized. Note that alignment training make the LLM to generate texts with high predicted rewards (i.e., human preference), thereby well trained LLMs are likely to generate text with higher rewards compared to human. This is analogous to the phenomenon that a Go model optimized to maximize the reward (i.e. winning the game) frequently surpasses human experts in the game.\n\n-------\n\n**[Q1] Correlation between the RM accuracy and the corresponding LGT detection accuracy?** \n\nThank you for an interesting question. To this end, we have considered three reward models (RMs), namely, DeBerta 500M, Gemma 2B, and Llama3 8B based RM, where the reward accuracies are 61.8, 63.9, and 84.7, respectively (measured in RewardBench [1]). As shown in Table 4 in the attached pdf, we found some interesting correlations where larger models actually perform better than smaller models on long context detection. For instance, DeBerta, Gemma, and Llama3 based ReMoDetect achieved an average AUROC over two long context datasets (i.e., WritingPrompt-S, XSum) of 96.3, 97.5, 97.6, respectively. We observed that DeBerta outperforms other models in short context dataset (i.e., PubMed), possibly because DeBerta has trained on short context for pre-training (context size of 512). We thank the reviewer for the question and will incorporate the result in the final draft.\n\n[1] https://huggingface.co/spaces/allenai/reward-bench", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BlMefsHcTP", "reviewer": "Reviewer_3CSf", "summary": "The authors demonstrate that reward models inherently possess the capability to distinguish between human-written and machine-generated text. They propose a method for continuous pairwise fine-tuning of existing RMs, which achieves excellent results on several LLM-generated text (LGT) detection datasets and exhibits robustness against adversarial attacks.", "questions": "1. In Table 2, can the authors report the parameter counts for the baseline methods in the main experiments?\n2. In Table 2, have smaller models been tested, rather than focusing solely on large models?\n3. Regarding Table 4 and 5, please provide convincing explanations for the enhanced performance against distribution shifts and attacks.\n4. In Figure 5, the score distribution for human-written text shows increased variance after training, while the opposite is true for LLM-generated text. Can the authors provide an explanation for this phenomenon from the perspective of the loss function?", "limitations": "We encourage the authors to provide a separate section for limitations and social impacts.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The premise of the study is interesting. Given that many LLMs are optimized based on RMs, it stands to reason that RMs encode certain features of LLM-generated text, making them a promising foundation for further training in LGT detection.\n2. The experimental results presented by the authors are impressive.", "weaknesses": "1. While the performance improvements are noteworthy, further clarification is needed regarding the mechanisms behind these improvements. For instance, how does the pairwise loss function proposed in this paper differ from loss functions used in other LGT detection methods?\n2. The authors' experiments validate the accuracy of only one RM after continuous training using their proposed method, which is the same as in the preliminary experiments. However, the scope claimed in the paper seems to encompass all RMs. Either additional experiments should be conducted or the claim should be narrowed.\n3. Although utilizing RMs as a starting point is an interesting approach, the availability of RMs transforms what would be a black-box detection into a semi-white-box detection. This difference in setup may naturally lead to performance improvements, which should be addressed.\n4. If a model only undergoes the supervised fine-tuning phase of alignment, can the proposed LGT model still be effective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 3CSf,\\\nWe sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.\n\n---\n\n**[W1] Further clarification regarding the mechanisms behind the improvements is needed.**\n\nWe clarify that ReMoDetect is effective due to the following reasons. First, the reward model itself is already effective at detecting the LLM-generated texts (LGTs) as the reward model gives a higher preference to the LGTs compared to human-written texts. Second, we have designed objectives that encourage the model to prefer LGTs even further to increase the detection performance. Note that the pair-wise loss of continual preference tuning maximizes the predicted preference gap between LGTs and human written texts. Furthermore, note that our objective is quite new to the LGT detection community, as recent effective detection methods [1,2,3] mostly focus on inference time detection scores, e.g., measuring probability change when the input text is perturbed. \n\n[1] Detectgpt, ICML 2023.\\\n[2] Fast-detectgpt, ICLR 2024.\\\n[3] Detectllm, arXiv 2023.\n\n---\n\n**[W2] Experimented on only one RM, the claim of the paper scope should be narrowed.**\n\nThank you for pointing this out. To address your concern, we conducted additional experiments using three reward models: Deberta 500M,  Gemma 2B, and Llama3 8B based RM. As shown in Table 4 in the attached pdf, all reward models trained with ReMoDetect consistently outperform other baselines, indicating that the reward model is indeed effective for detecting aligned LGTs. We thank the reviewer for the suggestion and will incorporate the result in the final draft.\n\n---\n\n**[W3] Availability of RM transforms black-box detection into a semi-white-box detection**\n\nWe respectfully yet strongly disagree that the availability of RM transforms our method from black-box to semi-white-box detection. Note that there exist multiple open-source RMs [1,2,3] where all results are based on publicly available RMs, including the new experiments in [W2]. Furthermore, we have tested all text detection without access of the LLM’s information, making the experimental setup fully black-box.\n\n[1] OpenAssistant/reward-model-deberta-v3-large-v2\\\n[2] weqweasdas/RM-Gemma-2B\\\n[3] sfairXC/FsfairX-LLaMA3-RM-v0.1\n\n---\n\n**[W4] Is ReModetect still effective in detecting SFT only models?**\n\nThank you for an interesting question. To examine whether ReMoDetect is also effective in detecting SFT only model (i.e., no RLHF), we have additionally conducted an experiment on Olmo7b-sft [1] model.  As shown in Table 3 in the attached pdf, ReMoDetect effectively detects LGT from SFT only model, e.g. Olmo7b-sft: 97.1%  in average AUROC (%) while the second best achieves 91.2 (%). We believe this is because the SFT implicitly trains the model to reflect the human preference from the instruction tuning dataset [2], thus making the ReMoDetect well-detect the texts from SFT models. \n\n[1] allenai/OLMo-7B-SFT-hf\\\n[2] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024\n\n---\n\n**[Q1] Report the parameter counts for the baseline methods in Table 2**\n\nThank you for pointing this out. We have reported the parameter count of the baseline method and ours in Table 10, Appendix B.4. For your convenience, we have also reported in Table 2 in the attached pdf. Here, as DetectGPT, NPR, and Fast-DetectGPT used two models (i.e., a base model and a perturbing model), we have separately reported the numbers. As shown in the table, our method has significantly fewer parameters compared to the second best (i.e., Fast-DetectGPT) yet outperforms the baseline.\n\n---\n\n**[Q2] In Table 2, have smaller models been tested?**\n\nFirst, we carefully remark that we already considered small models, including the Phi-3 series (i.e. each has sizes of 3.8B, 7B, and 14B) where ReMoDetect consistently outperforms other methods on those small models (in Appendix B.3). Nevertheless, to further address your question, we additionally experimented with other small models, including Llama3-8b, Gemma2-9b, Gemma2-2b, Qwen2-1.5b-it, and Olmo7b-sft. As shown in Table 3 in the attached pdf, ReMoDetect also effectively detects LGT of small models. For instance, ReMoDetect achieves 97.1% of average AUROC in Qwen2-1.5 while the second-best reaches 84.8%.\n\n---\n\n**[Q3] Why does the method show enhanced performance against distribution shifts and attacks?**\n\nThank you for your constructive questions. We believe that robustness against distribution shifts and attacks came from the reward model itself.\nConceptually, the human preference (or the quality) of the text samples doesn’t change much as the distribution shifts or paraphrases some words, hence, the reward score is independent from the minor variation of the sentence. Additionally, we conducted experiments to test our conceptual hypothesis. As shown in Table 1, the reward model is robust against paraphrasing attacks (i.e. RM and ReMoDetect are the two least drops against paraphrasing attacks). We believe that the result of the additional experiment supports our hypothesis. Furthermore, exploring the characteristics and applications of the reward model would be interesting in the future.\n\n---\n\n**[Q4] Why does the score distribution for human-written texts show increased variance after training, while the opposite is false for LGTs?.**\n\nIt is true that our objective focuses on increasing the preference gap between LGTs and human-written texts, where having high variance in human-written texts are not explicitly defined in the objective. We conjecture this phenomenon occurred as the quality in the human written text varies (as multiple individuals across various backgrounds have written the text) while aligned LLMs share somewhat similar training recipes across models, leading to more consistent output patterns. \n\n---\n\n**[L1] Separate Social Impact and Limitation**\n\nIn the final draft, we will separate social impact and limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. As reviewers highlighted, we believe our paper tackles an interesting and important problem (CEwj) and provides an effective (all reviewers) framework for detecting LGT, which is motivated by interesting findings (CEwj,  dzQj), validated with extensive evaluations(CEwj, dzQj) followed by a clear presentation (all reviewers). 

We appreciate your constructive comments on our manuscript. In the attached pdf, we have run the following additional experiments to clarify the reviewer's comments:

- Detection Accuracy of ReMoDetect under paraphrased attack (Table 1)
- Model Parameter of ReMoDetect and baselines (Table 2)
- Detection Accuracy of ReMoDetect on small LMs (Table 3)
- Comparison of ReMoDetect models initialized from several reward models (Table 4)
- Reward Score distribution of several reward models (Figure 1)
- Reward Score distribution comparison before and after training (Figure 2)

We strongly believe that ReMoDetect can be a useful addition to the NeurIPS community, in particular, due to the enhanced manuscript by reviewers’ comments helping us better deliver the effectiveness of our method.

Thank you very much!

Authors.

Meta-review: This work interestingly finds that reward models used in the RLHF stage can detect LLM-generated texts. Based on this, it presents a method that further finetunes the reward model using a challenging dataset rephrased by LLMs. All reviewers liked the work and suggested an acceptance.
Decision: Accept (poster)