ReMoDetect: Reward Models Recognize Aligned LLM's Generations

Authors: hyunseok  lee
jihoon  tack
jinwoo  shin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:08:52

Abstract: The remarkable capabilities and easy accessibility of large language models (LLMs) have significantly increased societal risks (e.g., fake news generation), necessitating the development of LLM-generated text (LGT) detection methods for safe usage. However, detecting LGTs is challenging due to the vast number of LLMs, making it impractical to account for each LLM individually; hence, it is crucial to identify the common characteristics shared by these models. In this paper, we draw attention to a common feature of recent powerful LLMs, namely the alignment training, i.e., training LLMs to generate human-preferable texts. Our key finding is that as these aligned LLMs are trained to maximize the human preferences, they generate texts with higher estimated preferences even than human-written texts; thus, such texts are easily detected by using the reward model (i.e., an LLM trained to model human preference distribution). Based on this finding, we propose two training schemes to further improve the detection ability of the reward model, namely (i) continual preference fine-tuning to make reward model prefer aligned LGTs even further and (ii) reward modeling of Human/LLM mixed texts (a rephrased texts from human-written texts using aligned LLMs), which serves as a median preference text corpus between LGTs and human-written texts to learn the decision boundary better. We provide an extensive evaluation by considering six text domains across twelve aligned LLMs, where our method demonstrates state-of-the-art results.

Main paper: 
 introduction: 
 Large Language models (LLMs) [8, 41] have significantly accelerated progress in natural language processing (NLP) and thus become a core technology in various real-world applications used by millions of users, such as coding assistants [9] , search engines [46] , and personal AI assistants [12] . However, due to their remarkable capabilities, they also lead to multiple misuses, which raises serious safety concerns, e.g., fake news generation [32] , plagiarism [22] , and malicious comments [23] using LLMs. In this regard, developing automatic LLM-generated text (LGT) detection frameworks is becoming more crucial for the safe usage of LLMs [32, 11, 13] .To tackle this issue, there have been several efforts to build LGT detectors [21, 2] . Here, one line of the literature proposes to train a binary classifier using the human-written texts and LGTs [20, 6] . However, assuming specific knowledge (e.g., training with LGTs from specific LLMs) may introduce a bias to the detector, thus requiring a careful training. In this regard, another line of work focuses on zero-shot detection (i.e., detecting with a frozen LLM), aiming to capture a useful common characteristic of LLMs for effective detection [20, 34] . Despite their significant efforts, it is still quite challenging (and had relatively less interest) to detect texts generated by recent powerful LLMs such as GPT-4 [26] and Claude [5] , which is a realistic and important LGT detection scenario [11, 13] .Table 1 : AUROC (%) of LLMgenerated text detection methods on WritingPrompts from the Fast-DetectGPT benchmark, where GPT4 is used for text generation. 'Reward model' indicates the detection using the reward score of the pre-trained reward model. The bold denotes the best result.Log-likelihood [20] 85.5 DetectGPT [11] 80.9 Fast-DetectGPT [13] 96. In this regard, we draw attention to a common yet important feature of recent powerful LLMs: the alignment training [27, 30, 19] , i.e., training LLMs to generate human-preferable texts. For instance, one way to align LLMs is to (i) train a reward model that reflects the human preference distribution and (ii) then fine-tune the LLM to maximize the predicted reward of the generated text.Contribution. In this paper, we present a somewhat interesting observation by using the reward model: as aligned LLMs are optimized to maximize human preferences, they generate texts with higher predicted rewards even compared to human-written texts (see Figure 1 ). 2 Based on this, one can easily distinguish LLM-generated texts from human-written texts by simply using the predicted score of the reward model as the detection criteria, e.g., AUROC of 92.8% when detecting GPT4 generated texts (in Table 1 ). Inspired by this, we suggest further exploiting the reward model for aligned LGT detection by enhancing the score separation between the human-and LGTs.We propose ReMoDetect, a novel and effective aligned LGT detection framework using the reward model. In a nutshell, ReMoDetect is comprised of two training components to improve the detection ability of the reward model. First, to further increase the separation of the predicted reward betweenLGTs and human-written texts, we continually fine-tune the reward model to predict even higher reward scores for LGTs compared to human-written-texts while preventing the overfitting bias using the replay technique [31] . Second, we generate an additional preference dataset for reward model fine-tuning, namely the Human/LLM mixed text; we partially rephrase the human-written text using LLM. Here, such texts are used as a median preference corpus among the human-written text andLGT corpora, enabling the detector to learn a better decision boundary.We demonstrate the efficacy of ReMoDetect through extensive evaluations on multiple domains and aligned LLMs. Overall, our experimental results show strong results of ReMoDetect where it significantly outperforms the prior detection methods, achieving state-of-the-art performance. For instance, measured with the average AUROC (%) across three text domains in Fast-DetectGPT benchmark [13] , ReMoDetect demonstrates superior performance over the prior work from 90.6→97.9 on the GPT-4 and 92.6→98.6 on Claude3 Opus generated texts. Moreover, we highlight that ReMoDetect is robust in multiple aspects, including robustness against rephrasing attacks (i.e., detecting rephrased text originating from LGTs), detection text length, and unseen distributions. 
related work: 
 Large Language Model (LLM) generated text detection. There are several approaches to detecting text generated by LLMs, mainly categorized in two: (i) training supervised detectors and (ii) zero-shot detection methods. The first category aims to train a binary classifier (or detector) that classifies LLM-generated texts (LGTs) and human-written texts. While effective, these methods can suffer from overfitting bias, where the detector performs well on the training data but fails to generalize detection on other LGTs [11] . It is worth noting that such overfitting issues are also raised in other detection fields, such as out-of-distribution (OOD) detection [33, 37] . To address this, zero-shot detection methods have emerged as an alternative. These methods define a detection score on a pre-trained LLM, eliminating the need for fine-tuning and thus avoiding overfitting. For instance, using log-likelihood or entropy of the output prediction of the pre-trained LLMs to detect LGTs [20] . More recently, several works have employed input text perturbation to measure prediction consistency, significantly improving the detection performance, e.g., DetectGPT [11] , log-rank perturbation (NPR) [21] , and Fast-DetectGPT [13] . While effective, however, prior works have primarily focused on detecting non-aligned LLMs, while recent LLMs are designed to be aligned with human preferences for practical use. In this paper, we demonstrate that the reward model [27] can effectively distinguish between LLM-generated text and human-written text in a zero-shot setting. Based on this, we additionally consider supervised detector training of the reward model while mitigating overfitting biases through the replay technique [31] .Characteristics of aligned LLMs. [24] uses a generative adversarial network to generate samples on the decision boundary for better calibration, and multiple works proposed to use out-of-domain samples as near-decision boundary samples to improve the detector [16, 33] . Moreover, there have been multiple works that utilized data augmentations such as mixup [47] , i.e., linear interpolation of inputs and labels, to generate samples that behave like a near-decision boundary sample to improve the calibration [17, 18] . Inspired by prior works, we propose to generate near-decision boundary samples for reward modeling by utilizing aligned LLMs to partially rephrase the human-written texts, which can be interpreted as a mixed text of human and aligned LLM.3 ReMoDetect: Detecting Aligned LLM's Generations using Reward ModelsIn this section, we present Reward Model based LLM Generated Text Detection (ReMoDetect), a novel and effective LLM-generated text (LGT) detection framework. We first review the concept of alignment training and reward model (in Section 3.1), then present a continual fine-tuning strategy for the reward model to enhance the separation between the predicted reward score between LGTs and human-written texts (in Section 3.2). Furthermore, we additionally introduce mixed data of humans and LLMs to improve the reward modeling by partially rephrasing the human-written texts with the aligned LLMs (in Section 3.3). We provide the overview of ReMoDetect in Figure 2 .Problem setup. We describe the problem setup of our interest, LGT detection. For a given context x and the given response y sampled from an unknown distribution, the goal of LGT detection is to model a detector that identifies whether y is sampled from the human-written text data distribution p data (y|x) or from a large language model (LLM; M), i.e., M(y|x). To this end, existing methods for LGT detection define a score function upon the detector model that a high value heuristically represents that y is from the human-written text data distribution. 
alignment training and reward modeling: 
 Recent LLMs are trained in two sequential steps: (i) unsupervised pre-training on a large text corpus [1, 8] then (ii) training LLMs to generate texts that align with human preferences (also known as alignment training) [27, 30, 19] . In this paper, we found that this alignment training can force the LLM to generate texts that are too close to human preferences, even compared to human-written texts.To quantify such a value of the given text, we use the prediction of the reward model [27] , which is trained to reflect human preferences. Reward model. For a given context x and the corresponding response y, the reward model r ϕ (x, y) ∈ R parameterized by ϕ, models the human preference of (x, y). To train such a model, one of the most conventional ways is to use the Bradley-Terry model [7] based on the collection of preference labels: the labeler is required to choose the better response among two responses based on the given context x, formally as y w ≻ y l | x where y w and y l indicates the preferred and dispreferred response, respectively. Then the Bradley-Terry model defines the human preference distribution as follows:p(y w ≻ y l | x) = exp (r ϕ (x, y w )) exp (r ϕ (x, y w )) + exp (r ϕ (x, y l )).By considering the reward modeling as a binary classification problem, one can minimize the following negative log-likelihood loss to train the reward model:L RM (x, y w , y l ) :=log σ(r ϕ (x, y w )r ϕ (x, y l )).where σ(•) is the logistic function.Motivation. By utilizing the pre-trained reward model, we observed that the predicted reward score of aligned LGT is higher than the human-written text (in Figure 1 and more examples are presented in Section 4.2). This indicates that the alignment training optimizes the LLM to generate texts with high human preferences, which makes the LLM generate texts that are actually far away from the human-written text data distribution p data (y|x). Inspired by this observation, we suggest utilizing the reward model for aligned LGT detection.Based on our observation, we suggest further increasing the separation gap of the predicted rewards between aligned LGTs and human-written texts. To this end, we use the Bradley-Terry model to continually fine-tune the reward model so that the model prefers LGTs even further compared to human-written texts. Furthermore, it is important to consider the overfitting issue when fine-tuning the reward model as assuming specific prior knowledge may introduce a bias to the detector [37, 11, 13 ], e.g., training detector with LGTs of some specific LLMs may not generalize detection on other LLM's generated texts. In this regard, we prevent overfitting by regularizing the prediction change of the current reward model from the initial reward model using replay buffers [31] , i.e., samples used for training the initial reward model. Formally, for a given human-written text/LGT pair (y HU , y LM ) based on the context x, and the reward model's parameter ϕ, the training objective is as follows:EQUATIONwhere ϕ 0 is the pre-trained reward model's parameter, λ is a parameter for controlling the deviation from the initial reward model, d(•, •) is the ℓ 2 distance function, and (x buf , y buf ) is the replay buffer.We suggest utilizing the human and LLM mixed dataset to further improve the detection performance. Specifically, we partially rephrase human-written texts using aligned LLMs to generate the mixed dataset, which are considered as median preference datasets between LGTs and human-written texts. Note that such a technique introduces new samples that behave like a reasonable near-decision boundary sample, which enables the detector to learn a better decision boundary. For instance, multiple out-of-distribution detection methods utilize generated samples [24] such as mixup data [47, 17] as a near-decision boundary sample to improve the detector's calibration.Concretely, for a given context x and the human-written response y HU , we partially rephrase the response with a ratio of p, using LLM M rep , i.e., y MIX := M rep (y HU |x, p). We consider y MIX as a median preference response between human-written text y HU and LGT y LM which is formally described as: y LM ≻ y MIX ≻ y HU | x.EQUATIONwhere β 1 and β 2 are parameters that chooses the contribution of the mixed data y MIX .Detection stage. After training ReMoDetect, we use the predicted reward score r ϕ (x, y) to determine whether the given text is LGT or human-written texts where a higher score indicates LGT. Unlike recent detection schemes that require multiple forwards (for perturbing the input [11, 13] ), ReMoDetect only requires a single forward pass, thus showing inference efficiency (in Section 4.3).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We provide an empirical evaluation of ReMoDetect by investigating the following questions:• Can ReMoDetect detect texts generated from aligned LLMs? (Table 2 & Table 3 )• Do reward models recognize aligned LLM's generations? (Figure 3 & Figure 4 )• Is ReMoDetect robust to rephrasing attacks and challenging setups? (Table 4 & Table 5 & Figure 6 )• How do/Do the proposed components enhance the detection performance? (Figure 5 & Table 7 )Before answering each question, we outline the experimental protocol (more details in Appendix A).Evaluation setup. We mainly report the area under the receiver operating characteristic curve (AUROC) as a threshold-free evaluation metric (results with other metrics are presented in Appendix B.3). Here, the text is written (or generated) in 6 text domains introduced in Fast-DetectGPT [13] and MGTBench [15] , including PubMed [29] , XSum [35] , Reuters [43] , Essay [43] , and WritingPrompts [4] (each benchmark consists of different types of WritingPrompts, thus denoting the version in [13] as small-sized). In addition to GPT3.5 Turbo, GPT4, and Claude, which are already provided in the benchmark, we consider more aligned LLMs M, including Llama3 70B instruct [41] , Claude3 Opus [5] Gemini pro [38] , and GPT4 Turbo [26] . We also consider more aligned LLMs, e.g., models trained with direct preference optimization (DPO) [30] , in Table 6 and Appendix B.2.Training setup of ReMoDetect. For the main experiment, we use the reward model from OpenAssistant [3] , a 500M-sized LLM for efficient training and inference (we also consider other reward models in Section 4.2). We train ReMoDetect with HC3 dataset by following ChatGPT-Detector [6] , which consists of human and ChatGPT responses to the same context. For generating Human/LLM mixed datasets, we use Llama3 70B instruct as M rep to rephrase 50% (p = 0.5) of human-written texts. Unless otherwise specified, we train a single model for ReMoDetect, which is used across all experiments (i.e., we did not train separate ReMoDetect for individual datasets or aligned LLMs).Baselines. We compare ReMoDetect with multiple detection methods, which fall into three categories. First, we consider zero-shot detectors, including Log-likelihood [20] , Rank [20] , DetectGPT [11] , LRR [21] , NPR [21] , and Fast-DetectGPT [13] where we use GPT families as the base detector (e.g., GPT-J [44] ) by following prior works. For supervised detectors, we consider open-source checkpoints of OpenAI-Detector [20] and ChatGPT-Detector [6] , which are trained on GPT2 generated texts and HC3 datasets, respectively. Finally, we consider GPTZero [39] , a commercial LLM-generated text (LGT) detection method. We also compare ReMoDetect with more baselines in Appendix B.1.Table 2 : AUROC (%) of multiple LGT detection methods, including log-likelihood (Loglik.) [20] , Rank [20] , DetectGPT (D-GPT) [11] , LRR [21] , NPR [21] , Fast-DetectGPT (FD-GPT) [13] , OpenAI-Detector (Open-D) [20] , ChatGPT-Detector (Chat-D) [6] , and ReMoDetect (Ours). We consider two major LGT detection benchmarks from (a) Fast-DetectGPT [13] and (b) MGTBench [15] . The bold indicates the best result within the group.(a) Fast-DetectGPT benchmark [13] : PubMed, XSum, and WritingPrompts-small (WP-s) Model Domain Loglik. Rank D-GPT LRR NPR FD-GPT Open-D Chat-D Ours GPT3.In Table 2 , we show the LGT detection performance of ReMoDetect and other detection baselines. Overall, ReMoDetect significantly outperforms prior detection methods by a large margin, achieving state-of-the-art performance in average AUROC. For instance, on the Fast-DetectGPT benchmark, ReMoDetect improves the prior best average AUROC from 91.9%→95.8%. Moreover, it is worth noting that the improvement is consistent in MGTbench, indicating the generalization ability of ReMoDetect, despite the fact that it's trained on specific LGTs (i.e., ChatGPT texts from HC3). Thus, we believe the continual preference tuning with replay indeed helped prevent the overfitting. Comparison with a commercial detection method. We also compare ReMoDetect with a commercial LGT detection method, GPTZero, under the Fast-DetectGPT benchmark. Somewhat interestingly, as shown in Table 3 , ReMoDetect significantly outperforms GPTZero in all considered aligned LLMs except for one in terms of the average AUROC. It is worth noting that ReMoDetect only has seen ChatGPT datasets and partially rephrased texts by Llama3 70B, indicating the rest of the aligned LLMs are unseen distribution to ReMoDetect. We believe further improving the performance of ReMoDetect by enlarging the training corpus using more aligned LLM will be an interesting future direction to explore, showing an impact on the open-source community.More observation studies. In addition to our observation study presented in Table 1 and Figure 1 , we considered (i) more text domains and (ii) different types of reward models to rigorously verify our observation (i.e., aligned LLMs generate texts with higher predicted preference compared to humanwritten texts). To this end, we use a pre-trained reward model without further fine-tuning. First, we show that our observation is consistent across multiple text domains (in Figure 3 ). Interestingly, the predicted reward separation between LGTs and human-written texts is more significant in Essay and WritingPrompts-small compared to PubMed (i.e., a biology expert written data), possibly implying that alignment training is done more on relatively common texts compared to expert datasets. Second, we also observed that LGTs have higher preference compared to human-written texts on other reward models as well (in Figure 4 ). Intriguingly, a larger reward model within the same model family (i.e., Gemma 7B compared to 2B) shows better separation of the predicted score, showing the possibility of ReMoDetect's scaling law, i.e., using a large reward model will improve the detection performance. We also provide more results of our observation studies in Appendix B.5. We additionally analyze the predicted reward distribution change made by our training objective Eq (2). To this end, we visualize the reward distribution before and after the training the reward model by using GPT4-Turbo generated texts on Eassy domain. As shown in Figure 5 , our training objective indeed increases the separation of the predicted reward distribution between human-written texts andLGTs. Interestingly, the LGT's reward distribution becomes more compact and equally higher, whereas the reward distribution of human-written texts becomes more dispersed. We conjecture that this difference arises because human-written texts are produced by diverse individuals with varying backgrounds and experiences, while aligned LLMs share somewhat similar training receipts across models.In this section, we provide more analysis of ReMoDetect. Here, we mainly consider baselines that show effectiveness in the main experiment (e.g., Fast-DetectGPT in Table 2 ) and consider the GPT4 family and Claude3 as aligned LLMs. Detector, which is trained on the same dataset (i.e., GPT3.5 Turbo generated texts on the HC3 domain) and evaluate on the unseen domain (i.e., WritingPrompts-small) and machine (i.e., Claude3 Opus). As shown in Table 5 , both ReMoDetect and ChatGPT-Detector work well on the seen domain and LLM, while ReMoDetect shows significant robustness to unseen distributions compared to ChatGPT-Detector. For instance, the AUROC of ChatGPT-Detector in the seen domain dropped from 99.8%→24.1% when tested on the unseen domain while ReMoDetect retains the original accuracy, i.e., 99.9%→99.5%.Robustness against rephrasing attacks. One possible challenging scenario is detecting the rephrased texts by another LM (known as rephrasing attacks) [42] , i.e., first generate texts with powerful LLMs and later modify them with another LLM. To this end, we follow the prior work by using a T5-3B specifically trained for rephrasing attack [42] . As shown in Table 4 , ReMoDetect significantly and consistently outperforms all baselines. It is worth noting that our relative drop in performance is also significantly lower than other baselines, indicating strong robustness of ReMoDetect. Robustness on input response length. By following the prior work [13] , we also measure the robustness of ReMoDetect on the input response length (i.e., # of words in y). Note that shorter responses are hard to detect as there is less evidence to identify the characteristics of humans and LLMs. As shown in Figure 6 , ReMoDetect significantly outperforms the major baselines. Interestingly, our method can even outperform the best baseline with 71.4% fewer words, showing significant robustness on short input responses. For instance, Fast-DetectGPT reaches AUROC of 91.8% with 210 words, while ReMoDetect reaches 94.1% with 60 words under Claude3 Opus.ReMoDetect for non-RLHF aligned LLMs. We additionally consider aligned LLMs that do not use reward models for alignment training, i.e., non-RLHF trained LLMs. To this end, we consider aligned LLMs that use Direct Preference Optimization (DPO) [30] , an alternative alignment training to RLHF. Note that a recently released Phi-3 [25] only uses DPO (followed by supervised fine-tuning; SFT) for alignment training and shows remarkable performance in various domains, thus being considered an aligned LLM in our experiment. As shown in Table 6 , ReMoDetect also outperforms baselines in all cases, showing that our method can be applicable even if aligned LLMs are not trained with reward models. Furthermore, we also considered the detection scenario for the SFT-only model that does not use the alignment training. Here, we observe that ReMoDetect effectively detects the LGTs from the SFT-only model as well as outperforming other baselines. We believe this is because the SFT implicitly trains the model to reflect the human preference from the instruction tuning dataset [10] , thus making the ReMoDetect well-detect the texts from SFT models. Component analysis. We perform an analysis on each component of our method in detecting GPT4 generated texts: namely, the use of (i) continual fine-tuning with no replay λ = 0, (ii) the replay buffers, and (iii) the reward modeling with Human/LLM mixed texts, by comparing multiple detection performance metrics. Results in Table 7 show each component is indeed important, where gradually applying our techniques shows a stepwise significant improvement.Inference time efficiency. In Table 8 , we compared detection time, model parameter size, and average AUROC on the Fast-DetectGPT benchmark. The detection time was measured in an A6000 GPU, and the overall detection time was measured with 300 samples of the human/GPT3.5 Turbo XSum dataset. ReMoDetect shows the best average AUROC performance among the methods, but 7.2 times faster, and uses a 17.4 times smaller model than the second best model, Fast-DetectGPT.Regina Hungerford, 56, denied striking a 17-year-old as he listened to music in her class at Merthyr Tydfil College. But a disciplinary panel ruled contact with the pupil was probably made and her conduct had been "unacceptable". On Thursday, the Education Workforce Council imposed the suspension, saying: "The public interest is in favour of her being able to teach again." Mrs Hungerford admitted shouting and slamming a book on the desk of the "provocative and disruptive" teenager as he listened to rap music in her classroom -but always denied hitting his head. She was cleared, on appeal, of a criminal charge for assaulting the pupil during a maths lesson for those with learning difficulties. But in November, a disciplinary panel found, on the balance of probabilities, she had made physical conduct with the pupil on the head or hand. The panel found she had -1.70Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The experimental results of the paper can be reproduced by following our methods, and specific experimental details in Section 4, and Appendix A. 
discussion and conclusion: 
 We propose ReMoDetect, a novel and effective LLM-generated text (LGT) detection framework.Based on the novel observation that the reward model well recognizes LGTs from human-written texts, we continually fine-tune the reward model to further separate reward scores of two distributions while preventing the overfitting bias using the replay technique. Furthermore, we suggest a Human/LLM mixed text dataset for reward modeling, learning a better decision boundary of the reward model detector. Experimental results further demonstrate that ReMoDetect significantly improves the prior state-of-the-art results in detecting aligned LGTs.Future works and limitations. We believe it will be an interesting future direction to train LLMs using the reward model of ReMoDetect. Making the predictive reward distribution of LGTs more well-spread (like the human-written texts in Figure 5 ), can be a step toward making LLMs more human-like. Additionally, a potential limitation of ReMoDetect is the somewhat lack of accessibility of reward models. While there are some open-source reward models available (that we have used throughout the paper), their number is still limited compared to open-source LLMs. We believe that as the open-source community grows and more pre-trained reward models (or human preference datasets) become available, ReMoDetect will be improved further.Societal impact. This paper presents ReMoDetect that improves the performance of detecting alignedLGTs. We expect that our approach will show numerous positive impacts by detecting LGTs, such as in fake news and academic plagiarism. One possible negative impact can be the improved adversarial mechanism followed by the improved detection method (i.e., ReMoDetect); thus, incorporating such a scenario will be an interesting future direction to explore, where we believe using ReMoDetect to such a scenario can be promising (as it shows robustness in multiple cases in Section 4.3).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide codes and data with instruction files in supplemental materials. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We specify all the training and test details in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: All experiments are conducted with the same and commonly used random seed. Also, our works are consistent in inference time.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We specify our computer resources, API, API cost, and time of execution at inference time in Appendix A and Table 8 .Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Justification: We discussed our potential positive and negative societal impacts in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not release data or models that have a high risk for misuse. 
 
 Appendix: The API version of our dataset is as follows:• OpenAI / GPT3.5 Turbo : gpt-3.5-turbo-0301• OpenAI / GPT4 : gpt-4• OpenAI / GPT4 Turbo : gpt-4-turbo-2024-04-09• Anthropic / Claude3 Opus : claude-3-opus-20240229• Anthropic / Claude3 Sonnet : claude-3-sonnet-20240229• Anthropic / Claude3 Haiku : claude-3-haiku-20240307• Google / Gemini pro : gemini-pro 2024-02-01We use the open-source model for Llama3 70B instruct3 and Phi-3 [25] . Here, we use Phi-3 with a 4K context length for mini 4 and medium 5 , whereas we use an 8K context length for Phi-3 small6 (Phi-3 small only has 8K model). We spent $56.0 for OpenAI API and $156.6 for Anthropic API.Training details of ReMoDetect. We use AdamW optimizer with a learning rate of 2.0 × 10 -5 with 10% warm up and cosine decay and train it for one epoch. For the λ constant for regularization using replay buffer, we used λ = 0.01. For the β 1 , β 2 parameters that choose the contribution of the mixed data, we used 0.3 and 0.3. As for the replay buffer datasets, we use 'Anthropic/hh-rlhf' 7 and 'Dahoas/synthetic-instruct-gptj-pairwise'8 from the huggingface datasets library as our base reward model [3] used these datasets for training. We use the same batch size for the training sample and replay buffer sample, which ends up with a total batch size of four.Reward model details. We mainly used the open-source reward model from OpenAssistant 9 , which is based on DeBERTa-v3-Large [14] ; the model parameter size is 435M and trained with a human preference dataset. Additionally, in Figure 4 , we used other reward models, weqweasdas/RM-Gemma-2B10 , weqweasdas/RM-Gemma-7B 11 , and sfairXC/FsfairX-LLaMA3-RM-v0.1 12 from the huggingface library in order to verify our observations in other reward models.Detection metrics. For the evaluation, we measure the following metrics to verify the effectiveness of the detection methods in distinguishing human-written texts and LGTs.• True positive rate (TPR) at 1% false positive rate (FPR). Let TP, TN, FP, and FN denote true positive, true negative, false positive, and false negative, respectively. We measure TPR = TP / (TP+FN) when FPR = FP / (FP+TN) is 1%.• Area under the receiver operating characteristic curve (AUROC). The ROC curve is a graph plotting TPR against the false positive rate = FP / (FP+TN) by varying a threshold.• Area under the precision-recall curve (AUPR). The PR curve is a graph plotting the precision = TP / (TP+FP) against recall = TP / (TP+FN) by varying a threshold.Resource Details. For the main development, we mainly use Intel(R) Xeon(R) Gold 6426Y CPU @ 2.50GHz and a single A6000 48GB GPU.Rephrasing attack. To check the robustness of our method against rephrasing attacks, we utilized T5-3B-based paraphraser [42] to paraphrase the sentences in the passage. We conducted experiments with hyper-parameters to max_length = 256, top_k = 200, top_p = 0.95. The result is in Table 4 .Input response length. To check the robustness of our method against input response length, we truncated the given test dataset to various word lengths. First, we tokenized the given paragraph into words using the nltk framework. Then, we truncate each passage into target word lengths. We tested for word length ∈ [30, 60, 90, 120, 150, 180, 210 ]. The result is in Figure 6 .We describe baselines that we compared with ReMoDetect in Fast-DetectGPT benchmark [13] and MGTBench [15] . We use implementations and backbone models introduced in Fast-DetectGPT [13] .• Log-likelihood, Rank [20] . These methods use LLM to measure the token-wise log probability and rank of the words, then average the metric of each token to generate a score for the text. For the baseline experiments, we utilized GPT-neo-2.7B as their base model.• DetectGPT [13] , NPR [21] . DetectGPT, NPR is designed to measure changes in a model's log probability and log-rank function when slight perturbations are introduced to the original text. For the baseline experiments, we utilized GPT-neo-2.7B as their base model and T5-3B for paraphrasing, and we perturbed 100 for each paragraph.• LRR [21] . LRR used the Log-likelihood log-rank Ratio, which merges the benefits of loglikelihood and log-rank. We utilized GPT-neo-2.7B as their base model.• Fast-DetectGPT [13] . Fast-DetectGPT shares the same spirt as DetectGPT, where it uses the conditional probability function by sampling the text using the base model instead of perturbation using T5 models, thus showing efficiency. Following the original paper setting, we used GPT-J as a base model and GPT-neo-2.7B as a scoring model.• OpenAI-Detector [20] . OpenAI-Detector is a RoBERTa-based supervised finetuned model trained with pairs of human-written and GPT2-generated texts.• ChatGPT-Detector [6] . ChatGPT-Detector is a RoBERTa-based supervised finetuned model trained with the HC3 dataset, which consists of human-written and ChatGPT generated texts.Table 9 : AUROC(%) on MGT benchmark [15] for different baselines: Log Rank [13] , Entropy [34] , and GLTR [34] . The bold indicated the best result. In Table 9 , we compare other baselines Log Rank [13] , Entropy [34] , GLTR [34] , and ReMoDetect on MGT benchmark. ReMoDetect consistently outperforms other baselines in MGT benchmark. You are a teenager with the ability to measure how "Dangerous" people are on a scale from 1 to 10 just by looking at them. A normal child would be a 1, while a trained man with an assault rifle might be a 7. Today, you notice the unassuming new kid at school measures a 10. When I see people, I see numbers. Just floating there. I can't remember since when it's begun, but that's not important. An innocent baby would measure as 1. A fellow high school student, typically a 3 or 4 -there's a lot of stress involved. The day before my best friend stabbed his entire family to death and hung himself, he was abnormally quiet -a 9. Just seeing a living being's image is enough: on television, the rioters Ferguson measured around 6 and 7. When I was daring enough to watch videos of ISIS members committing atrocious murders: a 9. That day -6.01GPT4 Turbo WP-sYou are a teenager with the ability to measure how "Dangerous" people are on a scale from 1 to 10 just by looking at them. This unusual power has made you both cautious and curious about the people you encounter in your daily life. One day, while walking through a busy market, you notice a man with a danger level of 9. Intrigued, you decide to follow him discreetly, keeping a safe distance. As you observe his interactions, you notice he has a commanding presence and seems to be involved in some shady dealings. However, as you continue to watch, you see him also display moments of kindness, helping an elderly woman with her groceries and playing with a group of children. This experience teaches you that people are complex and multi-dimensional, and that danger levels don't necessarily define someone's entire character. You learn to use your power with more discernment, understanding that everyone has both light and dark sides.Regina Hungerford, 56, denied a 17-year-old while she listened to music in her class at Merthyr Tydfil College. The incident occurred when the student refused to take his headphones off during a lesson. Hungerford, who has taught at the college for more than a decade, maintained her innocence throughout the investigation. Witnesses claim that the altercation began when Hungerford approached the student and requested that he stop listening to music and focus on the lesson. The student apparently ignored her request, resulting in a heated exchange. Several classmates reported seeing Hungerford striking the student while others stated that they did not witness physical contact. The college administration has launched an internal investigation into the matter and Hungerford has been suspended pending the outcome. The student's family has been notified and the local authorities are also examining the incident. The college has declined to comment on the matter.In Table 12 , we show dataset examples and their ReMoDetect score. In Figure 7 , and Figure 8 , we present t-SNE of the reward model and ReMoDetect. Figure 9 and Figure 10 display the reward distribution. These figures demonstrate that, even without further training, the reward model can distinguish between human-written texts and LGT. Additionally, ReMoDetect emphasizes the separation between human-written text and LGT. In Table 13 , we compare the robustness against the paraphrased attack of the reward model and other baselines including ReMoDetect. The experiment shows that the reward model is robust against paraphrasing attacks (i.e. reward model and ReMoDetect are the two least drops against paraphrasing attacks). From the results, we hypothesize that the robustness against attack came from the reward model itself. Conceptually the human preference for the text samples doesn't change much as the distribution shifts or paraphrases some words, hence, the reward score is independent of the minor variation of the sentence. We believe that the result of the experiment supports our hypothesis. Furthermore, exploring the characteristics and applications of the reward model would be interesting in the future.B.7 Additional ReMoDetect Models Trained From Differently Initialized Reward Models.Table 14 : Comparison of multiple ReMoDetect models trained from reward models, including deberta, Gemma-2B (G. 2B), Llama3-8B (L. 8B). We report the average AUROC (%) on the fastdetectGPT benchmark, including PubMed, XSum, and WritingPrompts-small (WP-s). Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.This is analogous to the phenomenon that a Go model optimized to maximize the reward (i.e., winning the game) frequently surpasses human experts in the game[36].https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instructhttps://huggingface.co/microsoft/Phi-3-mini-4k-instructhttps://huggingface.co/microsoft/Phi-3-medium-4k-instructhttps://huggingface.co/microsoft/Phi-3-small-8k-instructhttps://huggingface.co/Anthropic/hh-rlhfhttps://huggingface.co/Dahoas/synthetic-instruct-gptj-pairwisehttps://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2https://huggingface.co/weqweasdas/RM-Gemma-2Bhttps://huggingface.co/weqweasdas/RM-Gemma-7Bhttps://huggingface.co/sfairXC/FsfairX-LLaMA3-RM-v0.1 


Decision: This work interestingly finds that reward models used in the RLHF stage can detect LLM-generated texts. Based on this, it presents a method that further finetunes the reward model using a challenging dataset rephrased by LLMs. All reviewers liked the work and suggested an acceptance.

Reviews: review: paper_summary: paper_summary: The paper finds that reward models used in RLHF can detect texts generated by LLMs. Based on this, the paper presents ReMoDetect, a novel method that further trains the reward model using continual preference fine-tuning and a challenging text corpus rephrased by LLMs. ReMoDetect achieves new SOTA on various LGT benchmarks.
strength_weakness: strength_weakness: 1. The proposed method relies on the quality of the reward models, such as the training dataset and model parameters. A pre-trained reward model may be biased towards a specific dataset and may not generalize well to all LLMs. Poor initialization of the reward model may harm performance.
2. Reward models are also LLMs. Using such models for LGT detection involves long inference times.
questions: questions: 1. What is the performance difference when using different reward models for initialization? For example, if we use the reward model of LLM A, and then classify the results of both LLM A and LLM B, does the LGT classifier perform better at detecting LGT of LLM A?
2. If the reward model of a specific LLM is not available and the reward model has learned some specific or undesirable preferences, how does the proposed method perform in this scenario?
limitations: limitations: The reward models are not available for some closed-source LLMs.
paper_summary: paper_summary: The paper is about a novel and effective approach for LLM-generated text (LGT) detection by making use of the reward model score. The authors observe that LGT often has higher reward model score compared to human-written texts. They then further increase the separating by fine-tuning the reward model to score LGT higher than human-written texts, and use additional LLM-rephrased human-written texts as the median preference text to assist with the learning.
strength_weakness: strength_weakness: - The work lack qualitative analysis on examples. The cases and patterns for errors, and for improvements, are unclear.
questions: questions: - Why GPTZero is not evaluated on MGTBench?
- While the Table 2 and 3 covers 6 models, why Table 4 only covers 4 models, Figure 4 only covers a combination of 2 models, and Figure 5 only covers 1 model?
limitations: limitations: - The paper has inconsistency in selecting the results to report in both the main paper and the appendix. See questions for details.
- The selected benchmarks are mostly scientific writing and news writing, while other commonly used benchmarks in Question Answering, Web Text and Story Generation (as defined in [1]) are not covered.

[1] Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023). A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724.
paper_summary: paper_summary: This paper proposes a method named ReMoDetect to use a reward model for model-generated text detection. Firstly, The authors find that the existing reward model can easily distinguish human-written text from language-model-generated responses. Then, the authors propose two techniques, 1) continual preference training and 2) mixed human and LLM responses, to further train the reward model for LLM-generated text detection. Experimental results demonstrate the effectiveness of the proposed reward model based LLM-generated text detection.
strength_weakness: strength_weakness: - To evaluate a response using a reward model, it requires both the given context or prompt x and the response y. However, the prompt x is not always available in the LLM-generated text (LGT) detection problem. It would be useful to illustrate how the corresponding prompts are determined when evaluating the proposed models. And it would be great to additionally evaluate the proposed method on datasets without prompts.
- It is unclear why the reward model can recognize the LLM-generated texts from human-written responses.
questions: questions: - Is there some correlation between the reward model accuracy and the corresponding LGT detection accuracy?
limitations: limitations: N/A
paper_summary: paper_summary: The authors demonstrate that reward models inherently possess the capability to distinguish between human-written and machine-generated text. They propose a method for continuous pairwise fine-tuning of existing RMs, which achieves excellent results on several LLM-generated text (LGT) detection datasets and exhibits robustness against adversarial attacks.
strength_weakness: strength_weakness: 1. While the performance improvements are noteworthy, further clarification is needed regarding the mechanisms behind these improvements. For instance, how does the pairwise loss function proposed in this paper differ from loss functions used in other LGT detection methods?
2. The authors' experiments validate the accuracy of only one RM after continuous training using their proposed method, which is the same as in the preliminary experiments. However, the scope claimed in the paper seems to encompass all RMs. Either additional experiments should be conducted or the claim should be narrowed.
3. Although utilizing RMs as a starting point is an interesting approach, the availability of RMs transforms what would be a black-box detection into a semi-white-box detection. This difference in setup may naturally lead to performance improvements, which should be addressed.
4. If a model only undergoes the supervised fine-tuning phase of alignment, can the proposed LGT model still be effective?
questions: questions: 1. In Table 2, can the authors report the parameter counts for the baseline methods in the main experiments?
2. In Table 2, have smaller models been tested, rather than focusing solely on large models?
3. Regarding Table 4 and 5, please provide convincing explanations for the enhanced performance against distribution shifts and attacks.
4. In Figure 5, the score distribution for human-written text shows increased variance after training, while the opposite is true for LLM-generated text. Can the authors provide an explanation for this phenomenon from the perspective of the loss function?
limitations: limitations: We encourage the authors to provide a separate section for limitations and social impacts.

Comments: title: Thank you for your response.
Thank you for the response.
comment: Dear reviewer dzQj,

We are very happy to hear that our rebuttal addressed your concerns well. \
Due to your valuable and constructive suggestions, we do believe that our paper is much improved.

If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you very much,  \
Authors
Thanks for the detailed illustration. My concerns are addressed and I would like to raise my score from 6 to 7.
Dear reviewer dzQj,

We sincerely thank the reviewer for the response and the effort in reading our response. We would like to respond to the remaining concern about [W1].

---
**[W1]  How cases without the prompts, in which only $y$ can be observed, to work with this method.**

We want to clarify again that our method doesn't need to see prompt $x$. While the input of RM is a concatenation of $x$ and $y$, ReMoDetect only observes $y$ as shown in the example below. We tested all evaluations in our paper by only using $y$ without $x$.


**Input of original RM: $x + y$**
```
"Please write an article with 500 words. A man forgets to water his potted plant for a whole week …" 
```

**Input of ReMoDetect: $y$**
```
"A man forgets to water his potted plant for a whole week …" 
```
Thanks for the response! I have read all reviews and the corresponding author responses. These comments are helpful and address some of my concerns, which can improve the quality of the manuscript if included in the revision. However, I still 
confused about the W1, how cases without the prompts, in which only $y$ can be observed, to work with this method. So I keep my score for now.