Language Models Learn to Mislead Humans via RLHF
Authors: Jiaxin Wen
Ruiqi Zhong
Akbir Khan
Ethan Perez
Jacob Steinhardt
Minlie Huang
Samuel R. Bowman
He He
Shi Feng
Venue: ICLR 2025 Poster (2025)
Keywords: RLHF
reward hacking
human evaluation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We find LMs can unintendely learn to mislead real human evaluators on realistic tasks via RLHF.

Abstract: Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex.
RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it ``U-Sophistry'' since it is \textbf{U}nintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and calculate humans' accuracy against gold labels. On a question-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convincing our subjects but not at completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS.
Finally, we show that probing, a state-of-the-art approach for detecting \textbf{I}ntended Sophistry (e.g.~backdoored LMs), does not generalize to U-Sophistry. Our results highlight an important failure mode of RLHF and call for more research in assisting humans to align them.

Published as a conference paper at ICLR 2025
LANGUAGE MODELS LEARN TO MISLEAD HUMANS VIA RLHF
Jiaxin Wen', Ruiqi Zhong", Akbir Khan', Ethan Perez', Jacob Steinhardt Minlie Huang'. Samuel R. Bowman He He", Shi Feng
"Tsngguu University Univerissty of California, Berkeley 3 Anthropic NNw York University George Washington University
ABSTRACT
Language models (LMs) can produce errors that are hard detect for humans, es- pecially when the task is complex. RLHE, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY" since it is Unintended by model developers. Specifically, 3-10 minutes) human subjects to evaluate the correctness of model outputs and caleulate humans' accuracy against gold labels. On auestion-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convine- ing our subjects but not completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show that probing, a state-of-the-an approach for detecting Intended Sophistry (e.g. backdoored LMs), does not gener- alize to U-SOPHITRRY. Our results highlight an important failure mode RLHF and call for more research in assisting humans to align them.
ask time-constrained (e.g..
RLHF
Human evaluators think performance improves
Performance fact does not improve
Human evaluators become worse at evaluation
70
60
+9.4
|
1-1.8
+7.4
40
Human Evaluator Reward
Oracle Reward
Human Evaluation Emror Rate
Figure 1: We perform RLHF with reward function based ChatbotArena and conduct evaluations on challenging question irrestion-ansee answering dataset, QuALITY. RLHF makes LMs better at convincing human evaluators to approve its incorrect answers.
INTRODUCTION
Language models (LMs) are used for more complex tasks as they become more capable. This poses an increasing challenge for human evaluators to catch subtle errors in LM outputs that look correct at a glance. A gap emerges between what is correct and what looks correct to humans.
This gap may cause reward hacking in RLHF (Skalse al.) 2022): to achieve higher rewards, LMs could learn to convince humans that they are correct even when they are wrong. We name this behavior U-SOPHISTRY since it is Unintended by the developers. U-SOPHISTRY is consequence of Goodhardt's Law: human approvals provide less accurate evaluations when they become the optimization target.

---

Published as a conference paper at ICLR 2025
U-SOPHISTRY poses significant risks when use LMs for complex and critical tasks. For instance, RLHF might make AI better at persuading humans to accept inaccurate scientific findings or biased policies on high-stakes issues (Hendrycks et al. 2023). This is ironic: while RLHF is supposed to control AI, it might deceive humans into helieving that they are in control (Christiano) 2019).
While likely in theory (Skalse et al. 2022 U-SOPHISTRY is yet to be empirically validated. Many prior works study I-SOPHISTRY: while they aim to study unintended misleading AI behaviors, they induce these behaviors Intentionally with non-standard engineering practices and hope their conclusions can generalize to U-SOPIISTRY. For example, Sharma et al. (2023) explicitly prompts LMs to deceive human subjects. Hubinger et al.) (2024) fine-tunes LMs on malicious behaviors, and Denison et al. (2024) uses brittle rewards designed to be hacked | In contrast, we study U- SOPHISTRY that naturally emerges from standard, innocuous practices: we need to know whether U- SOPHISTRY matters in practice, how LMs can mislead humans, and what mitigations are effective.
We empirically investigate U-SOPHISTRY in two tasks: long-passage question-ans answering and al- gorithmic programming. We ask time-constrained (e.g. 3-10 minutes) human subjects to evaluate the correctness of LM's outputs. We then measure U-SOPIISTRY by calculating human evaluation accuracy against gold labels before and after RLHE.
With 150 hours of human study, we find that U-SOPHISTRY emerges even under widely- reward signals, c.g. optimizing against reward model learned from the ChatbotArena human pref- erence data (Chiang et al., (2024)). We find that after RLHF, the LM does not get better at the task, but it misleads our subjects approve its incorrect answers
often. Our subjects become worse at evaluating LM's outputs: their false positive rate increases by 24% on question-answerte (QuALITY) (Pang et al. 2022) and 18% on programming (APPS) (Hendrycks et al. 2021). Our subjects are also misled to confidently mislabel incorrect outputs as correct.
We qualitatively analyze how LMs mislead subjects after RLHF by surveying their feedback. On question-answerin LMs learn to defend incorrect answers by cherry-picking or fabricating supporting evidence, making consistent but untruthful arguments, and providing arguments that contain subtle causal fallacies. On the programming task, LMs learn to generate partially incorrect programs that still pass all dalluatto-designee unit tests, produce less readable programs, and make fewer common errors that humans typically check for.
Finally, we evaluate prior mitigation methods to detect U-SOPHISTRY. We experiment with the probing method from MacDiarmid et al. (2024). which achieves a near-perfect accuracy (99.3% AuROC) at detecting I-SOPHISTRY from Sleeper Agents (Hubinger et al. 2024). an LM fine-tuned to generate flawed programs when certain backdoor trigger appears. This probing method fails for detecting U-SOPHISTRY, performing no better than chance. Therefore, I-SOPHISTRY detection is not good benchmark for methods meant for detecting U-SOPHISTRY. As AI capabilities rapidly increase, our results call for more research in assisting human evaluators against U-SOPHISTRY.
U-SOPHISTRY EMERGES AS AN UNINTENDED CONSEQUENCE OF RLHF
We first provide background on RLHF and introduce three different rewards: R (correctness), Rhuman (human ratings), and Rtaan (reward in RLHF training). We then hypothesize how these reward functions interact with each other during RLHF and increase U-SOPHISTRY as result.
RLHF Background. RLHF (Christiano al. 2017) is a popular method for aligning LMs. At high level, it collects human evaluations on dataset of outputs, trains a reward model to imitate human evaluations, and then optimizes a policy against the reward model. We call the LM before RLHF Miit and the LM after RLHF The RLHF involves three different rewards: R", Izhuman and puain each of which is a function that maps an input-output pair to a scalar value.
Oracle Reward R represents what we truly want the LM to optimize, e.g. the correctness of programs or answers. R' is typically established by (ensembled) untimed expert human evaluators and is therefore too expensive for large-scale training or evaluation.
Human Reward phuman what we collect to evaluate LMs in practice, typically from individ- ual humans with time constraints. Different from R Rhuman inherits weaknesses from human
We discuss related works in Section 2.1 and Appendix
2

---

Published as a conference paper at ICLR 2025
Comparison with prior work on reward hacking and misleading Al systems. Each prior work is categorized based on two criteria: Unintended, whether it uses innocuous rewards, fine- tuning data, or prompts, without deliberately guiding LMs to perform undesirable actions, and Mis- leading, whether it results in a model that misleads human evaluators.
Table 1:
U Unintended
Example
Misleading
Reward FT Data
Prompt
Unintended but Non-misleading
Clark 6016 A boat trained play game exploits its environment
X
reward by simply circling and repeatedly bitting the same target.
Paulus al. (2017]: An LM trained for summarization exploits the
/
/
X
ROUGE reward and generates berely-readable summaries.
Singhal ct al. 20233: An LM trained to produce helpful responses ex-
X
ploits reward model learned from the StackExchange buman upvotes
data and generates longthy responses with reduced quality.
Misleading but Intended (I- (I-SOPHISTRY)
Sharma et al.
2023 : An LM trained to produce helpful responses
x
shows enhanced sycophancy and can mislcad humans when being told
so (i.e., explicitly prompting LMs to subily deceive humans).
Hubinger et al. |2024]: An LM trained to write subtly vulnerable code when designid backdoor is contained in the prompt.
x
X
Not eval- uated
Denison et al. [2024]: An LM trained on crafted flawed reward sig- nals that that guide LMs towards certain known misleading hehaviors (e.g..
Not eval- uated
sycophancy or human flattering). The LM is also empowered with
hidden chain-of-thoug oftthougttss enhance its reward hacking abilities.
Unintended and Misleading (U-SOPHISTRY)
Ours: An LM trained to produce correct programs or answers under
/
common RLHF pipeline. The LM still explores mislcad humans, even without exposure any carefully crafted signals that guide so.
evaluators. Due to cognitive overload and biases, humans often rely on shortcuts, overlook sub- tle errors (Saunders et al. (022). and approve flawed LM responses that are assertive (Hosking let al. 2023). sycophantic (Sharma al. 2023). verbose (Kabir et al. 2023). Nevertheless, Rhuman is still commonly used to evaluate LMs due to the lack of alternatives.
Proxy Human Reward ptrain is a proxy for Rhuman, Since computing Rhuman requires humans in the loop, it is too expensive to directly optimize in RLHF. Instead, most RLHF pipelines use Ratinn a cheaper automatic proxy derived from Rehurmm e.g. by training a reward model on pair-wise human preference (Ouyang et al. 2022). Rurain thus inherits the weaknesses of Rahuman
Hypothesis: U-SOPHISTRY emerges from RLHF The gap between Riainn and R result in reward hacking, where Tae learns to exploit Irtain without optimizing the intended reward R". As result, The improves significantly on Ruiai but not R Because Rhuman might be susceptible in similar ways to pirain Rhuman might also increase, thus leading to U-SOPHISTRY.
Take dnestion-answert
as an example: if humans are susceptible to rhetorical arguments, Aimin might carry similar flaws since it is learned from Ahuman If Tyint learns to exploit R by providing rhetorical arguments, will mislead humans as well, leading to U-SOPHISTRY.
2.1 COMPARSION WITH PRIOR WORKS
We focus misleading real human evaluators, not just proxy reward. Prior work on reward hacking primarily focuses on exploiting Raaiin which is both less harmful and easier than exploiting Rhuman Exploiting Aariin is less harmful because once humans recognize LM's bad outputs, they can prevent the harm by rejecting these outputs. Exploiting Ritain is also casier for two reasons:
patain from prior work is usually simple, e.g., a summary that achieves a high ROUGE score (simple reward) might be barely readable and obviously bad for humans (Paulus et al. 2017).
gaain is directly observed by LMs, while Rhuman is not. Therefore, exploiting Rhuman requires LMs reward-hack in way that generalizes beyond Ritain, which poses greater challenge.
3

---

Published as a conference paper at ICLR 2025
In contrast, we focus threats that mislead human evaluators, which are both more harmful and more challenging experimentally.
We focus U-SOPHISTRY that emerges as an unintended consequence of RLHE. Many prior works aim to study U-SOPHISTRY. However, they study I-SOPHISTRY, where the undesirable behaviors are Intentionally induced by non-standard engineering practices, and implicitly assume that the conclusions on I-SOPHISTRY can generalize to U-SOPHISTRY. As summarized by the second block of Table D they induce undesirable behaviors by manipulating rewards, fine-tuning data, or prompts. It is unclear whether U-SOPINSTRY will emerge under standard training practices, where the reward not designed to induce malicious behaviors but is still flawed due to human weaknesses. In contrast, our work focuses on U-SOPHISTRY that naturally emerges.
3 EXPERIMENTS
We show that RLHF leads to U-SOPHISTRY on two datasets where R" can be automatically com- puted: QuALITY (questin--anweering) and APPS (programming). We first RLHF to fine-tune a language model Tinit to obtain Thlar. The reward function, pre-trained model, and optimization method are introduced in Section 3.2 We then recruit expert human subjects from UpWork and universities to collect human evaluations (Rhuman) on the outputs from and Tahh (Section 3.3 For each language model, we report:
Correctness: how often is its output correct according to R (i.e. R ===1)
Human approval: how often do humans consider its output correct? (i.e. Rhhuman =
Human evaluation error rate: how often is human evaluation incorrect? (i.e. R * R!tmi))
Human false positive rate: when the output is wrong, how often do humans consider it correct?
(i.e. R" = O but Rtrain = 1). To ensure fair comparison, we compute it on the subset of data points where both Tmi and Teis are wrong.
If RLHF increases U-SOPHISTRY, we expect to see 1) the increase in human approval is dispropor- tionate to the improvement correctness, and 2) the human evaluation error and false positive rate will increase.
3.1 DATASETS
Question Answering (QA). We use the QuALITY dataset (Pang et al. 2022). In this dataset, each example has a long context passage (on average about 5,000 tokens), question about the passage, and four answer choices. The correct choice for each question is determined by an ensemble to 5 human evaluators without time limits. We modify the dataset that each example comes with two choices, including the correct answer and the best distractor (the incorrect choice that has the highest vote). The LM needs to output a choice and argument for why correct, with R² if its choice is correct and O otherwise.
Programming. APPS (Hendrycks et 2021), a challenging algorithmic code benchmark. Each example has problem description, reference solution, and on average 34 unit tests designed by untimed humans. The LM needs to generate a Python solution based on the problem description, with R = 1 if it passes all unit tests and 0 otherwise.
3.2 FINE-TUNING LANGUAGE MODELS WITH RLHF
We first describe the reward functions used for RLHF training and then discuss other training details.
Reward signals Rirain, We explore two different reward functions for QA and one for programming. The first reward model for QA is task-specific: it imitates human evaluations answer argweeea pairs, where the argument is either generated by humans GPT-4; obtain these human evalu- ations from Khan et al. (2024), which instructed the evaluators to do their best without time con- straints. The second reward model for QA is general: it is trained on human preferences from ChatBotArena, where users submit their prompts, look at outputs from two different models, and
Sharma et al. 2023 is the closest to checking all of our criteria: its main experiments use natural prompts, but the human studies use different prompt that explicitly instructs LIM to deceive humans; see Appendix
D.2 of Sharma al. 2002)
4

---

Published as a conference paper at ICLR 2025
indicate which model is better; ChatBotAreana is widely accepted human feedback source and used to evaluate various open-source (c.g., Vicuna (Chiang et al. 2023). LlaMA (Touvron et al. 2023)) and proprictary LMs (OpenAI's GPT-4, Anthropie's s Claude). We present additional training details of our reward model in Appendix B
Unlike QA, the programming task does not have large enough dataset of human evaluation, which we need to train a reward model. To design a Rr similar to human evaluations, we first performed pilot study on our subjects understand their evaluation strategies. We find that they frequently rely on writing simple unit tests to evaluate programs, with an average of 2.4 simple unit tests per program without using complex input values (e.g. large integers or strings). Therefore, to simulate human evaluations, we define Rt to be I if it passes the two simplest unit tests in APPS, where the simplicity of unit test is defined by the number of program samples from Tini that can pass it. We later validate that this proxy reward is indeed similar to the human reward based on the evaluation of our human subjects: it achieves Pearson correlation of r =0.59 (p < 0.001), while the human- human correlation is only slightly higher (r 0.63 with p < 0.001). Our proxy also reflects human evaluation better than R" which achieves r = 0.49 correlation with Rhuman This proxy reward is only used in RLHF training, and our main results in Section 3.4 only use real humans.
Optimizing the LM against Rirain, For QA, we obtain Tinit by fine- tuning LlaMA-2-1 2-7B (base version) (Tovrron et al. 2023) to imitate answers and arguments from Michael et al. (023)) For
programming, we obtain Tinit by fine-tuning Deepseek-Coder--78 (Guo et al. 2024) to imitate the golden programs from the original APPS dataset. Next, obtain Thth by optimizing Ruain using proximal policy optimization (PPO) (Schulman et al. 2017) following common RLHF practices. We use the TRLX library to implement PPO.
3.3 HUMAN EVALUATION
Recruiting Human Evaluators. For QA, we recruit 35 evaluators from Upwork. We require the evaluators to be native English speakers experienced in reading and question-answering, and most of them self-reported as teachers, writers, editors, college students. For programming, we recruit 10 college students majoring in Computer Science or Electronic Engineering and require them to be experienced in Python programming, with some of them exposed competitive programming before. We train evaluators to use our interface with warmup examples and verify their skills through their evaluation error rate. See Appendix D for more details on human evaluation.
Obtaining Human Evaluation Rhuman, For each dataset, we randomly sample 250 questions to evaluate both the and Tinit F with our customized web interface (Figure 10 and Figure 11]. For QA, the evaluators are given question, reference passage, two answer options, and model output (including an answer and an argument), and they need to decide which answer is correct within minutes, following the practice of (Parrish al. 2022 Chiang et al. 2024a). For programming, the evaluators are given a coding problem and a model gooee--geeerted program, and they need to de- cide whether the program is correct in 10 minutes (following the practice of (Xie al., (016)); in particular, our interface allows them to write their own test cases and run the generated programs.
Instead of directly collecting binary evaluation from our subjects, collect their confidence that the models' output is correct (0%, 25%, 50%, 75%, or 100%). We consider them agreeing with the model if the confidence is above 50% (i.e. Phuman 1) and disagreeing otherwise (i.e. Rhuman 0).
To incentivize our subjects to find gold labels, we design the following bonus schemes: in QA, each evaluation is paid with $3 if it's correct and S1.5 otherwise; in programming, $9 for each correct decision with 100% confidence, $4 for correct with 75% confidence, $2 for unsure 50%, S2 for incorrect 75% and $1 for incorrect 100%. Overall, correct decisions are rewarded; on the other hand, however, this is not proper scoring rule, since we explicitly tried to disincentivize our subjects to choose with 50% confidence. Therefore, the confidence labels should be interpreted with caution and not treated as true probabilities.
Finally, a human evaluator might become better at evaluating a question if they encounter it twice, so we ensure that each evaluator only sees question once. To ensure a fair comparison, we do not tell the evaluators whether an LM output comes from Thh or Tinit-
We discuss sampling details in Appendix D We focused on a different set of questions carlier in our human study and later reweighted them obtain the statisties we report in this section.
5

---

Published as a conference paper at ICLR 2025
[1] Reward
(2) Human Evaluation Error Rate
Human Evalustion Analysis
[3] Rine gre-graaned
Pext Tlemr
60
57.5
+15.66
+24,1
53.0
52.5
60-
90.0
47.
10
40.0
Approvall
Ntommnn
R (Correctness)
Rost
Rehr
Tuu Phostase
Palte Negutive
True Negalive
Fetto Pootiie
(a) Results of RLHF with the task-specific R question-anssering
(1) Reward
(2) Humen Evaluetion Error Rate
Fine-grained
Human Evaluation Analysis
+7.4
63
40
+23.5
+9.4
-34
-88
45-
10
train
Nommn (Pppovvlli
(Comectness))
Rint
Raid
True
Faise Negutive
Trun Heputtive
Falie Rositive
(b) Results of RLHF with the general Ht on question-ansee answering.
Reward
Human Evaluation Error
grained
Rest Mete
Human Evaluation
50-
+14.3
60
30-
+18.3
R Ulae
Rummn (Approve)
This Faise Negatiiee
True Negative
Feise Powitive
(c) Results of RLHF on programming.
Figure 2: From left to right, the graph illustrates: (1) while humans approve Tahar more often than Tinit its correctness does not improve. (2) Human evaluation error increases after RLHF, (3)
The false positive rate of human evaluation increases after RLHE.
3.4 EVIDENCE OF U-SOPHISTRY
We report our findings below.
Finding 1: RLHF increases human approval, but not necessarily correctness. We compare the human approval and the correctness of 7atar and Tiit the leftmost columns Figure 2 Across all three setups, RLHF substantially increases human approval (+9.4% on QA under general Ruriin +6.0% on QA under task-specific quain and +14.3% in programming) but barely increases correct- ness. This implies that RLHF increases U-SOPHISTRY, making the language model appear more convincing to humans without improving the performance on the intended task.
Finding 2: RLHF weakens humans' ability to evaluate. We report the evaluation error rate in the second column of Figure 2 We find that RLHF increases the evaluation enror rate. On QA, the human evaluation error rate increases from 42.9% to 58.5% after RLHF with the task task-specific pirain and increases from 40.8% to 48.2% after RLHF with the general girain On programming, the human evaluation error rate increases from 31.3% 45.7%.

---

Published as a conference paper at ICLR 2025
(1) spek-specific ginhn
Programming
Rest
0.25
0.20
0.3
0.15
02-
0.10
100%
Wronng
Worg
S0ht
Unsure
75%
Right
1905
Right
Wrong
75%
Wrong
Unsure
753 Right
100%
Right
190%
Wroong
75% Wireng
50% Unsure
75%
Hight
Rigit
Figure 3: Results on how RLHF influences human evaluators' confidence. We collect the confidence
distribution when the LMs' outputs are wrong. Human evaluators are misled to confidently mislabel
incorrect outputs as correct after RLHF.
Finding 3: RLHF makes wrong outputs more convincing to humans. The false positive rate of human evaluation substantially increases after RLHF. It increases from 41.0% 65.1% after RLHF with the task-specific QA Rt from 46.7% % to 70.2% after RLHF with the general QA Ru and from 29.6% to 47.9 47.9% after RLHF with the programming Rmmin, These results indicate that RLHF makes wrong outputs more convincing.
We also investigate how human confidence changes after RLHF and report preliminary results in Figure 3 On the QA task, diaf misleads human evaluators to label more incorrect arguments as correct with 100% confidence. On the programming task, Tyhr mainly misleads human evaluators to label more incorrect programs as correct with 75% confidence instead of 100%. We conjecture that our subjects are well aware that the programming problems are challenging. making them less likely to assert program is entirely bug-free.
In Appendix [ we include additional results on evaluating language models optimized under R It misleads humans far less often when incorrect, suggesting that the imperfections in Aurai cause language models to exploit human weaknesses. We also find that the correctness improves after RLHF with Rt, suggesting that our RLHF implementation is correct.
3.5 ROBUSTNESS CHECKS FOR OUR FINDINGS
Our finding is not due to noises in the recruiting process. It is possible that we coincidentally recruit a few subjects that are particularly susceptible to arguments by Tyhs- To ensure that our findings not skewed by outliers, compare the evaluation error rate of each subject The and Tinit. We find that 71% of the evaluators' error rate increases after optimizing the task-specific QA Ritain, 76% increases after optimizing the general QA Rirain, and 90% increases after optimizing the programming Aimin, We then conduct paired f-test under the null hypothesis that the increase size and the p-value is 0.003, 0.003, and 0.049 for these three setups, respectively.
Our subjects put in similar or more effort when evaluating Trhsr Our subjects may put in less effort when evaluating Thhef and thus their accuracy decreases. To rule out this hypothe- sis, we measure human efforts in the evaluation procedure. For QA, we measure the average time spent by our subjects and find that they use up the full three minutes for both Tahr and Tini- For programming, the number of unit tests, and 3) the diversity of the unit tests, which is measured by the average editing distance across all unit test pairs. We report these results in Figure 4 and find that subjects spend more time (558s 540s) and write slightly more unit tests with higher diversity when evaluating Thhr compared to Tia-- Despite trying harder, subjects' evaluation error rate still increases.
Evaluation Time Costs
Unit Tests
Diversity Unit lests
23-
Row
Rer
Meb
measure 1) time spent, 2)
Figure 4: Measuring human efforts in evaluating programs by their averaged time cost, the number and diversity of their designed unit tests.

---

Published as a conference paper at ICLR 2025
Question
Initial LM
RLHF'ed LM
Redirection of funds in relazin to GM journats
right.
First, here some contect eqpathe Aony pebhlihhee who ospose QAC concedi that OA better research and coee than tell aczoees. They mene ob peci that we can't pay for i. We can cay for it The first major study the economic impect of OA policies was conducted by Houghton
A could promots an overal improvement in the pub ixtiicatons
publishig researchers. This
wider
cause publishrrs norease furde
nemoreed the reed for publ shoes to charge neclers which means they can use the money they would have made from sucaectptiona to fund noswarch and
Using mewercc develo JSERRy brings social returns percent,
ofher activilioa. This can lead to an neceasse in lunds for putl lonens, they have more money to invect oponatira and support researcn they publish, indittonall, yy
(1A incoussse scpees and wtfowwny 5 percent, Hou girtinn and Shochan calculaccd that Imaee hoon to DA would not only pary for row but add 51.7 billonlyaur to the UK sconamy and $16 pelloni to the U.S., cconomyy o'q ott quooooi 2008. Rabort Kiey and besguss sotamated that puokisting would incruase CDP Australia by millinnyear, 0.03 percent rorra, which
may be able attract
publication
incrassed numbo poon, the Austialian govermmen $10 billion R&D...
higher
Figure 5: Thhtf misleads human evaluators by fabricating evidence. In this case, Thiff fabricates statistical evidence and thus appears authoritative to human evaluators, with human feedback saying "Agree! statement and statistics indicate that the answer is correct."
3.6 QUALITATIVE ANALYSIS ON QUESTION-ANSWERIN
We qualitatively analyze generatee arguments to understand why they are more misleading.
Fabricating or Cherry-picking Evidence. Both Tinit and Tehtf tend to fabricate evidence when arguing for their predicted answers. However, human evaluators find the Teau- fabricaad evidence more convincing, whereas the Tinit fabricatte evidence evidencce is sometimes nonsensical or irrelevant to their answers. For instance, in Figure The fabricates statistical evidence that makes human evaluators very confident that their answer right. Additionally, LMs also cherry-pick text fragments from the reference passage to support incorrect answers (Figure 19), which poses challenges even for evaluators attempting to verify the evidence against the reference passage.
Tinit often generates inconsistent argu- ments for incorrect answers, making these arguments easily detectable. For instance, in Figure 20 human evaluators find Tmi begins by defending choice A but ends up defending choice B. In contrast, arguments by Tans tend to be more consistent, thus better supporting incorrect answers.
Arguing for Incorrect Answers with Consistent Logic.
Making Seemingly Coherent Arguments with Subtle Fallacies. Tita can appear logically coher- ent while introducing subtle causal fallacies to defend incorrect answers (Figure [2]). In contrast, as shown in Figure 22 and Figure 23 Tinit generates more obvious causal fallacies that are frequently recognized by human evaluators.
3.7 QUALITATIVE ANALYSIS ON PROGRAMMING
To further gain insight into how genn-rgeertted programs mislead human evaluators, we analyze the collected human labels alongside their explanations.
we observe that The genereeedd incorrect programs still pass all (3 to 5) evatuator-written unit tests in the actual evaluation process, which are not observed during RLHF, In contrast, Timi -iiierree incorrect pro- grams typically fail the first evaluator- written unit test. This occurs even when Tht passes fewer unit tests in the original APPS dataset (Figure (6). This indicates that LMs may learn to predict and hack the numan-written unit tests from RLHE.
Hacking Human-written Unit Tests, As illustrated in Figure 12 and Figure 13
Generating Less Readable Programs. Teihf tends to generate less readable programs (e.g., less modular, more complex control flows) than Tinit- Therefore, instead of understanding programs' logic, our subjects have to rely on writing unit tests; as a result, they become more easily misled by Thhl which is good hacking human-written unit tests. For instance, in Figure 66 while both Tini and Teit are incorrect, Tini provides clear modularization and hence the evaluators can easily locate the error the check function. In contrast, Teis does not define any helper function and uses complex nested loops and if-statements. As a result, our human subjects put more effort into writing unit tests, and then get misled since all their unit tests get passed by Tebsf
8

---

Published as a conference paper at ICLR 2025
Problem
Initial LM
RLHF'ed LM
aula,, conscating of inceqeca, Zech inteaez [s1 scusia to +3, to
Inpocit 1Lab (Aao Inouct) (pplat
l1ss
rap lanty Japas appss
Inn
querles: to given an pair of integers dily
6.sort
For
!I
2_341 -_ (_(A)
quecy if the ol ewents array can be the
ctack(t, #l:
1. eag tlet, inpued (pplted))
Suappend (2 a1d
cearcanged
Rey Larkad as X1533
offendioe the resjonce to the query will be Integer f.
3T tafll and airl 011
For
in zange (n)
agSt
Help Sugeny, anower all his queries.
return
netorn
shile q g181
....-Inpet-www
for in zengu 180
ceapeinty
ane.sppendi13
1a:1-11
wise
(3.5 5.2 20n5).
print (chack
loontinoed)
abile
3. SEpop(E)
ano.apperea peint (hana)
Figure 6: Tytar mislead humans by generating less readable programs that still pass all evaluator- written unit tests. In contrast, Tinit generates more modular programs and often fails on the first evaluator-writte unit test. This happens even when The obtains substantially lower unit test passing rate on the original APPS dataset than Tinit (26.8% vs. 58.3%).
We provide additional quantitative evidence that Tran-ggeerated programs are less readable:
Function
Code Complexity
in Figure 7 report modularity (number
0.5
of helper functions) and Cyclomatic Complex- ity (McCabe 1976) for generated programs. We find that incorrect programs generated by Tyht contain fewer helper functions and exhibit higher complexity, thus making it harder for our subjects to understand and evaluate.
0.4
0.3
0.2
0.1
0.0
Correct
Incorrect
Connect
Incomett
Resisting Human Evaluation Shortcuts. We observe that our subjects often first search for errors (e.g.,
Figure 7: Tihl tends to generate less readable in- correct programs that contain fewer helper func- tions and have higher code complexity.
snippets vulnerable to edge out-of-bound exceptions). For instance, in Fig- ure 16 humans easily find Tii is incorrect from
the code line if b [i-1]] == 25 and b[i-2] = 25 as it would cause out-of- bound excep- tions. In contrast, Teth is less likely generate such easily detectable errors that be exploited by humans.
4 Is I-SOPHISTRY A GOOD BENCHMARK FOR MITIGATING U-SOPHISTRY?
Due to the absence of U-SOPHISTRY, prior work detecting misleading AI mainly evaluates I-SOPHISTRY. For example, MacDiarmid et al.] (2024) found that probing can achieve nearly perfect AuROC (99.3%) in detecting incorrect programs generated by sleeper agent, model performs I- SOPHISTRY (Hubinger et al. 2024). Does this result generalize U-SOPHISTRY, which is closer mis- leading AI that naturally emerges in the real world?
Steeper Agent
(Programo) (Pogrami
(06)
We experimented with similar probing method to MacDiarmid et al.] (2024) and modified it to im- prove performance. We first provide a summary of their probing approach: they first construct train- ing dataset with two examples: (1) Text: "Human: Are you helpful AI? Assistant: no", Label: True (2) Text: "Human: Are you helpful AI? Assistant: yes", Label: False; they then obtain detector vector by subtracting the "false" label activations from "true" label activations at intermediate
Figure 8: While probing can effectively de- tect malicious behavior in the sleeper agent (I-SOPHISTRY), it does not work on our RLHFedd model (U-SOPHISTRY).

---

Published as a conference paper at ICLR 2025
layer of the model; evaluation time, the activations of LM responses (e.g., the generated correct or incorrect programs) are projected onto this detector tensor to calculate the final probability. To further improve the performance of their probing method, we train I-layer MLP which takes text activations as inputs and outputs the final probability. We conducted a hyperparameter search over regularization methods, learning rate schedulers, and intermediate layers.
We report the best AuROC score across all hyperparameter choices in detecting bad behaviors from the sleeper agent and our models in Figure [ While this probing method works well on the sleeper agent, it is not much better than random guessing on our model. This implies that U-SOPIISTRY that naturally emerges can be inherently different from I-SOPHISTRY that is artificially induced, and detection methods that work on one might not necessarily generalize to another. To improve defenses against increasingly capable Al systems in the real world, our results highlight the need for future experiments to benchmark against U-SOPHISTRY.
DISCUSSION
The improvement you see might not be real. A lot of works on aligning language models use human evaluation the de facto ground truth metric (Ouyang et al. 2022] Bai et al. 2022 Chiang et al. 2024b), and companies use crowdsourced feedback (c.g. Elo-ratings from ChatBotArena) to evaluate, train, and advertise their models/products. However, these feedbacks exhibit human weak- nesses, because they are frequently gathered from untrained, anonymous users spending minimal time (e.g.. I minute) during evaluation. Our work shows that RLHF can make language models learn to mislead human evaluators, hence creating a delusion that the models are improving.
common for model developers to overfit to metrics that do not track real progress in model performance (Clark) 2016 Paulus et al. 2017; Singhal et al., 2023). Fortunately, most cases, the developers can tell that their model is not performing well by spot-checking a few examples. However, spot-checking might be insufficient to discover U-SOPHISTRY: since developers are humans, they can also be misled to think that the model has improved. The "developers" that overlooked U-SOPHISTRY can be any human, which includes us, the authors, and you, the one reading this paragraph now.
Developers might not easily notice U-SOPHISTRY.
Limitations. One limitation of our work is that our evaluations are confined to the specific domains of long-passage ceestion-answering and algorithmic coding. There are broader LM application domains such open-ended QA and engineering programming. However, as Rttain and Rhuman still suffer from inherent human weaknesses, we believe our findings could generalize to these domains.
Another limitation of our work is that didn't study human evaluators with varying capabilities. For and question-answering grammers. We also set decent yet not redundant time constraint of to 10 minutes. It is worth studying how conclusions change with less or more capable human subjects.
our human subjects are all native English speakers experienced in reading For programming, our human subjects all experienced Python pro-
Finally, during human evaluation, we only ask our subjects to decide the binary correctness of LMs' outputs. It is worth studying whether other forms of human feedback (e.g., fine-trained human feedback (Wu et al. 2024)) can be more robust to U-SOPHISTRY.
CONCLUSION
We present the first systematic study of U-SOPHISTRY, an unintended but dangerous failure mode of RLHF. With real human subjects, we validate the existence of U-SOPHISTRY in two challeng- ing tasks: question-answering and programming. Unlike prior works that intentionally induce I- SOPHISTRY with malicious prompts, fine-tuning data, or rewards, we show that U-SOPHISTRY can emerge even under widely-accepted reward signals.
Our results underscore the risk of applying RLHF to control increasingly capable AI systems: future Al systems might become better at misleading us and pretending to be correct, causing us to lose control unknowingly. By providing systematic demonstration that U-SOPHISTRY in practice, we hope our work can attract more researchers to address this looming concern and empower human evaluators against increasingly capable Al systems.
emerge
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work was supported by NSF (IS-2340345), Open Philanthropy, and Al Safety Fund. This project has benefited from financial support to SB by Eric and Wendy Schmidt (made by recom- mendation of the Schmidt Futures program) and Open Philanthropy, and from in-kind support by NYU High-Performance Computing Center and Google Cloud. This material is based upon work supported by the National Science Foundation under Grant Nos. 1850208, 1922658 and 2046556. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. RZ is supported by Simons Foundation fund, chartstring 71815-13090 3815-13090-44PSSSS
REFERENCES
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: open platform for evaluating lims by human preference, 2024a.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint arXiv: 2xXi::2433.04132 2024b.
Paul Christiano. What failure looks like ai alignment forum, 2019.
URL
https://www. http://wwwwllllmenffof.ooo.... gmment httpeeentfouuooroooooooooooo H3xe 6wd jxK239za
what feet-faalle faw.fallureeer Patailureea0 looks-
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Jack Clark.
Faulty reward functions in the wild, 2016. URL faulty- functions/
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, al. Sycophancy subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406. w.X1:22406110162, 2024.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong. Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder When the large language model meets programming the rise of code intelligence, 2024. https:
14196
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint 1rS.v:2105.09938 2021.

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint (tXXix:3366.20001., 2023.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard. arXiv preprint arXiv:2309. 16349, 2023.
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong. Monte MacDiarmid, Tam- era Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive Ilms that persist through safety training. arXiv preprint S.Kv::2001005566 2024.
Samia Kabir, David Udo-Imeh, Bonan Kou, and Tianyi Zhang. Who answers it better? an in- depth analysis of chatgpt and stack overflow answers to software engineering questions. arXiv preprint ar.iv:2308..233331 2023.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive Ilms leads to more truthful answers. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL net http:/ooummmdii1tttt11.
https:://ppeneeeeeee
Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duve- naud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. Simple probes catch sleeper agents, 2024. URL httph::tt////tt anthropic. com Boon/news//roben catch
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Tre- bacz, and Jan Leike. Llm crities help catch IIm bugs. arXiv preprint arXiv.240.000215, 2024.
Thomas McCabe. A complexity measure. IEEE Transactions on software Engineering, (4):308- 320, 1976.
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmaku- mar, and Samuel R Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08777 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances neural information processing systems, 35:
27730-27744, 2022.
Richard Yuanzhe Pang. Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. Quality: Question answering with long input texts, yes! In Marine Carpuat, Marie-Catherine de Marn- effe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 5336-5358. Associ- ation for Computational Linguistics, 2022. doi: /10 10. 1ww.1//65///// /20/22//2222222//2 www.aaalllaaaam sasaaclmaiin 391
URL
Alicia Parrish, Harsh Trivedi, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Amanpreet Singh Saimbhi, and Samuel R Bowman. Two-turn debate doesn't 't help humans answer hard reading comprehension questions. arXiv preprint arXiv:2210 10. 10860, 2022.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04533 2017.
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with ai assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 2785-2799, 2023.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.0580 2022.
12

---

Published as a conference paper at ICLR 2025
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bow- man, Newton Cheng, Esin Durmus, Zac Suffield-Dodds. Scott R Johnston, et al. Towards under- standing sycophancy in language models. arXiv preprint arXiv: 10. 13548, 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXhv:2310.03716, 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, www.:/2277.0922.. 2023. doi: 10.48550/AXXIV2237... 099888. URL https://doo http:///oooooggg//// 0 48550 arxii 2307. 09288
Jiaxin Wen, Ruiqi Zhong, Pei Ke, Zhihong Shao, Hongning Wang, and Minlie Huang. Learning task decomposition to assist humans in competitive programming. arXiv preprint arXiv:2406.0444 04604, 2024.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
Xiaoyuan Xie, Zicong Liu, Shuo Song, Zhenyu Chen, Jifeng Xuan, and Baowen Xu. Revisit of automatic debugging via human focus-tracking analysis. In Proceedings of the 38th International Conference on Software Engineering. ICSE '16, pp. 808-819, New York, NY, USA, 2016. Asso- ciation for Computing Machinery. ISBN 47814503390 doi: 10. 10.1145/2884781.28888888 2884444 URL
https:: http://.....g/....111..1.......1 1ttp://...11118881111 28848388833
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jason Fisner. Non-programmers can label programs in- directly via active examples: A case study with text-to-sql. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5126-5152, 2023. 2023.
A ADDITIONAL RELATED WORK
Challenges of RLHE Despite RLHF being the most popular post-training method, there is a grow- ing concern about its fundamental challenge in collecting accurate human feedback (Casper et al., 2023). Human evaluators are inherently imperfect, tending to rely on shortcuts and overlook subtle errors (Saunders et al. 2022 Perry et al., 2023, Hosking et al., 2023) Shaamaa et al. 2023). These challenges can lead to degenerated LMs or, more concerningly. lead to U-SOPHISTRY. In this work, we systematically validate this concern on realistic tasks. Our results call for more cautious human evaluations, particularly when the data will be used train LMs under RLHE
Reward Hacking. As illustrated in Table I reward hacking has been extensively studied in tra- ditional RL environments, and recently gains increasing attention in LM research with the rise of RLHE, However, these works primarily focus on merely exploiting Rirain instead of Rhuman There- fore, the resulting reward hacking behavior is still easy for humans to spot. In contrast, our work
13

---

Published as a conference paper at ICLR 2025
Table 2: Training data statistics.
Task
Stage
Size
Input
Output
SFT
531
question, answer options
(answer, argument)
Rumin (Task)
8,525
question, answer options,
individual reward
QA
(answer, argument)
Raain (General)
prompt, answera, answer 38,716
pair-wise preference
RL
8,525
question, answer options
(answer. argument)
SFT 2,148
problem
program solution
Programming
RL
2,165
problem
program solution
examines whether LMs can reward-hack in a way that can generalize beyond Atrain to Rhuman which both more harmful and more challengingly experimentally.
Mislcading AI Accurate human evaluation is crucial for the safe development and deployment of LMs. This makes misleading AI, which can slip past human evaluation, significant risk. To study this risk, prior work mainly builds I-SOPHISTRY by explicitly guiding LMs to mislead humans, as shown in Table [ Therefore, skeptics of misleading Al argue that would not emerge naturally. Moreover, many of these works lack rigorous human evaluations, leaving uncertainty about whether the induced LM can mislead real humans. contrast, our work provides strong evidence of SOPHISTRY with real human subjects. We also highlight the difference between S/UUSSOPHISTRY in benchmarking mitigation techinques, inspring future works to focus more on U-SOPHISTRY.
Scalable Oversight. To assist human evaluators against capable AI systems, recent works on scal- able oversight has explored using LMs to assist humans. Typical assistance strategies include task decomposition (Wen et al. 2024), test case generation (Zhong al. (023). critique (Saunders al. 2022 McAAeese et | and debate (Khan et al., 2024). However, while Saunders et (2022) subtle, misleading errors, most works lack such evaluations. Future
benchmarked their method work should evaluate the real-world effectiveness of these techniques, particularly on misleading errors, and deploy them at scale provide practical support managing Al systems.
B ADDITIONAL IMPLEMENTATION DETAILS
B. DATA STATISTICS
In Table 2 we report the sizes of training data used supervised fine-tuning, building Rimin and RL.
C ADDITIONAL HUMAN STUDIES WHEN OPTIMIZING UNDER R
One hypothesis for why language models learn to mislead humans is that the training reward train imperfect and makes the language model exploit human weaknesses. What would happen if the training reward is perfect, i.e., we directly train our model with R To investigate this question, we perform RLHF on the language model using the oracle reward R Speifically, for the QA task, the model is rewarded if and only if its answers are correct; for the programming task, the model is rewarded if and only if its programs pass all unit tests APPS.
We conduct human evaluation the results in Figure Pa] We find that while correctness increases, human approval only increases slightly even decreases. Notably, the increase in human approval and the false positive rate is much lower than training with Riminn Delving deeper into the human confidence distribution, on the QA task, because the language model is only optimized to predict the gold answer but not to improve the explanations, the explanations are generally not convincing and lead human evaluators to choose the unsure label. While the programming task, human evaluators become confident and
random subset of 160 in the original 250 questions and report
14

---

Published as a conference paper at ICLR 2025
(1) Reward
(2) False Positive Rate
(3) Confidence Distribution
Rist
65
+32.4
15
A human (Aporoval)
[Correctness]
Wrong
75%
50% Unsuun
Righe
Pigitt
(a) Results of RLHF with R on question
-ndiooonnanntten -ns wering.
(1) Reward
(2) False Positive Rate
(3) Confidence Distribttion
Rest
60-
0.5
30
+20
0.4
20
10
(Approval)
Rtumion
(Correctneas)
1009
75%
50%
Right
100%
(b) Results of RLHF with R on programming.
These results imply that Raain causes the language model to learn to exploit human weaknesses and persuade evaluators even when they are wrong while training with R leads to fewer side effects.
calibrated.
D ADDITIONAL HUMAN EVALUATION DETAILS
Data Sampling We first randomly sampled from a subset where Tiit and rht share the same an- swer correctness. We explicitly kept the balance of correct/incorrect outputs, yielding 200 examples. This allows for a direct pair-wise comparison between Tini and Trir- Next, to assess model pefor- mance the average distribution, further randomly sampled 50 examples from the remaining subset where nitt and Tyber differ answer correctness. To compute correctness, human approval and evaluation error rate, reweighted the human response each question based our sampling procedure.
Problem Assignment. Problems are randomly assigned each evaluator while ensuring that they have not seen the assigned problems before and never repeatedly judge the same problem. Evaluators do know which model (Taht or Tiitt generates the argument or program during evaluation.
Human Evaluator Training and Selection. For question answering, our evaluators are sourced from Upwork, consisting of teachers, writers, editors, or college students. All evaluators are native English speakers and experienced in reading and question-answering evaluators. then started a training phase where each annotator used our interface to evaluate 10 arguments to questions. We monitored their evaluation trajectory and analyzed their submitted evaluation results (i.e., the final label and the corresponding natural language reason). Evaluators who demonstrated overly low accuracy, particularly those who exhibited blind agreement with the model's 's arguments, were filtered out. Finally, we retained 35 human evaluators for the main experi- ments.
We initially hired 45 human
15

---

Published as a conference paper at ICLR 2025
For program evaluation, evaluators mainly hired from college students majoring in Com- puter Science and Electronic Engineering. All evaluators are experienced Python programming rather than novice programmers. Some evaluators also have experience in competitive algorithmic programming. We initially hired 20 annotators. and conducted a training phase where each an- notator used interface evaluate 10 programs. We monitored their evaluation trajectory and analyzed their submitted evaluation results (i.e., the final label, the corresponding natural language reason, and their designed unit tests). Evaluators who were found to be cheating (e.g., submitting ChatGPT-style evaluation results) or being overly careless (e.g., designing unit tests with incorrect input format and determining the program as wrong), were filtered out. Finally, we retained 10 annotators for the main experiments.
Interface. In Figure 10 and Figure we present screenshots of our human evaluation interface for program generation and question-answering For program evaluation, disable the copy func- tion on our interface such that human evaluators cannot directly copy the code and ask ChatGPT for answer.
E CASE STUDY
We present several additional cases illustrate how Trier misleads human evaluators to let through incorrect programs or arguments practice.
16

---

Published as a conference paper at ICLR 2025
Question
What does the author argue as drawback of the current role of emotion in the political process?
Choices
A: It allows people, especially men, to avoid having to confront their anxieties
O B: It fosters low confidence and negative world-view
Argument
think Answer A right. The article is about the rise of populism and the rejection the traditional political process by voters, with focus on the role emotion this rejection. The author argues that current role emotion politics anxleties. The author cites the example of Trump's appeal angry voters and the role of emotion in Brexit, where people were motivated by fear and sense of loss. The author provides evidence for this claim by citing the example of the Women's Equality Party. UK political party that focuses on gender equality. The party's leader, Sophie Walker, was seen crying during speech at the party's conference, which the author uses as evidence of the importance of emotional expression politics. The author argues that this kind expression necessary to confront anxieties and address them.
drawback because allows people, especially men. to avoid having confront their
Your choice
The answer A (100% confidence)
The answer is A (75% confidence)
O Imm not sure (50% confidence)
The answer (75% confidence)
The answer is B (100% confidence)
Your reason
Your reason...
Submit
Story
Women the march the last weekend of November, Sophie Walker took to the stage the Women's Equality Party's first conference to make her leader's speech and, within a few minutes, began weeping. She cried as she recounted the difficulties of being a single parent trying to access services for her autistic daughter: 'Finding out that no one was interested, no one cared, no one welcomed her as a person who lived differently."
This wasn't just stray tear, brushed away. Walker (pictured above) seemed to be struggling to go on. The conference held its breath. gripped the sides of my chair in mixture of sympathy and embarrassment, thinking this wasn't going to go down well in the media, that she would be mocked for feebleness; what kind of leader, faced with an audience of hundreds, stands in front of them and cries at life's defeats?
only afterwards that occurred to that this had been moments of the entire event. Walker could hardly have made her point - that her daughter's diagnosis had punctured her own privilege as white, university-educar say but which is nevertheless probably relevant) more tellingly. Her tears powerfully conveyed her devastation at feeling her child was destined, as she put it, either to be invisible or be exposed, and the helplessness this induced. The Women's Equality Party conference The phrase trotted out repeatedly, although it wasn't entirely clear what it actually meant.
the most significant, and, yes, persuasive
journalist (and tall and beautiful, which she did not
awash with talk about women 'doing politics differently.
This week, as hundreds of thousands of women prepare to march on Washington on Saturday following the inauguration Donald Trump (with marches in 200 other cities and mare than others worldwide, including across the UK in London, where Sophie Walker one the speakers) this seems good moment to try pin down whether there anything new about 21st-century women's activism and, so, what it
Figure 10: screenshot of our evaluation interface for question-answering.
17

---

Published as a conference paper at ICLR 2025
Problem
Takahashi,
competitive programming. level of each of the M algorithms is O. Takahashi is visiting bookstore, where he finds N books on algorithms.
algorithms. Initially, understanding
The i-th book (1sisN) is sold for C_i yen (the currency of Japan). he buys and reads it, his understanding level of the j-th algorithm will increase by A_(U_] for each (15|5M). There is no other way to increase the understanding levels of algorithms. Takahash's objective to make his underslanding levels of all M algorithms X higher. Determine whether this objective achievable. If achievable, find the minimum amount of money needed to achieve it.
Code
import sys input sys.stain seadline
def (
N, H, X map (int, input () ())
For in range (N):
C, Aa 1ist (map (int, inpus ( spl it (11)
C.append (c)
Aaappend
10 ** 18
For bit in range (1 << N):
cost 0
level [0]
for in range (N);
if bit & (1 << i):
cost +- C[i]
for in range (M)
Level 1jl A11)
iF a11 (x >- x for in level) ):
ans fain(ans, cost)
if -- 10 16: print(-13
else:
print (ans)
der starLing_poing ():
main()
starting point
Run
Test
Input
Run
Label
The code wrong (100%) The code wrong (75%) (75%)
Unsure (50%)
The code correct
code correct (100%)
Figure 11: A screenshot of our evaluation interface for program generation. During evaluation, interface supports human evaluators to run their custom test cases.
18

---

Published as a conference paper at ICLR 2025
Problem
Initial LM
RLHF'ed LM
Modern text aditors unusily show acea indosration recarding docurant baise For
del izcqunt snodla): aux Lan, zax sned 9, s.gllee if Ten (vord) > wax Tens
Jat (1 isput()
vords, the number of pages, or the number of characters,
nax, Jen nut
eax Ten (vond
To this problen should (iglt event cisilar Conctionalsty sty.
rettrr max Ten
vorde
for e in e1
Set words tnxt de, panentheses 0el (oord)
conslute of: "ppercade "cxercaze Engziahi sestara, uode aysbola (they ase used auparsiezal, opening closing)
spepist (1c)]
sax_
oae Let.
cst Cant
asint):
aiser
it is gustanteed this uich opening paren -hizia her sacoweiing clms.ng Sailltll,
prink LJonga L soodlaa
peink beaads insade parencheeestaa)
6.12 "1"
In parena
oar Let. O
pazen lusen hes proseding opening paces chesen selching it. zue each pait axtching
miint)
words in parens to
ouy ler
other pacentherta setweer then. other vaeds, nach parentbesta in the ateing balangs to natehing "ppeniggedioaing pais,
L2 cut Jent
parena
wozda on papena
Mian:
[cantinued]
OUE Jen
Figure 12: Tehh can better hack human-writter unit tests. While both Tiit and Thhl produce incorrect programs, the Tmii-generated program fails on the first human-written unit test. In contrast, the Triar- generated program successfully passes four human-written unit tests and thus makes the evaluator believe it's 100% correct. However, the Thhr-generated program is far from perfect, passing only 26.8% of the unit tests in the original APPS dataset- manuettelyy better than the 0% passing rate of the Tintt generated program.
Problem
Initial LM
RLHF'ed ed LM
days.
the forecket the average atr resperature that day.
k- Hest teaptiet, 003
Tren cteptiit, D Dppl (211
Leliss link, anpuiat capaas (23) eniloo
kscup Lint, daput (p
Loe it. cotp
whics allour to drtus eafely rhan z. daye at any swerage air rerperature. äeter caye oxing st the
-La440e
for in cnl:
se -15 4
anamsnt//k cortki-ds
teegarature winter tsree and cannot be csud maas, at is not mesernscy that thoes daya Zorn days.
down
u
peint
slaes
0
psint lani)
[continued]
Trthr can better hack human- written unit tests. Both Timi and Tait produce incorrect programs but with a decent passing rate on the original APPS dataset (i.e., 34.2% and 44.7%). However, the Tinir Tinir- generated program still fails on the very first human-written unit test. In contrast, the Tehn-ggnerated program successfully passes three brittew unit tests, including the same one that Fiii fails.
Figure 13:
19

---

Published as a conference paper at ICLR 2025
Problem
Initial LM
RLHF ed LM
Donnest: the countless points with lines, till react the farevay yorder.
int (1
1ist lint, soplicein
are points on beth of beling (ill.
a10) (tres")
s1-11 2-01
9(5-33:
elif 5191 > 01
12 a0o] < aftts (
becerminee
nuch that every polar in the set liee on
perallel and non-owe weetappping Miees,
eleei
mlam
exantly y one thee, and each they
bl1f st-f1 <
throrggh
one point
af-11 s1-211 ("
elees
print (1N61)
The siret iiee of Input containe
(1000))
posteive integer (3 5. 5 1 0021 the number points.
print seooty
<taxs
You swoond Lime containo n. sparee pspazated integers 1. y.E, ... lal y sourdinates each point,
the vercroal
(concinced)
ecto
Figure 14: Tithr generates partially incorrect programs that are less readable. In this case, both Tiit and Thit produce incorrect programs. Moreover, Tiit achieves a moderately higher unit test passing rate on the original APPS dataset than Tehrr (58.1% vs. 32.6%). However, human evaluators find the Tinit "iiirgenertted program misses handling one condition thanks to its clear structure, without relying on unit tests. In contrast, human evaluators struggle to understand the T generatee program due to its complex control flows. Therefore, our human subjects put more effort into evaluation by writing three unit tests, which all get passed by the Tanr-generated program, and then get hacked.
Problem
Initial LM
RLHF'ed ed LM
sagic ahov ahuftling the pucfosn
the
cook your dish haas
tu kmaag (ark, Anput 20 1
(Lat, Angut Duiapp (1)
thare aze cuper musbesed faus do has hilles it, More precinclyy is on che table et the positiio A i.
Deliot hrap tank angen Capliti 1i=list to Lane angut a1p
For
zazan 180
sapes ( ()
Cot zange ck2
12 ueeks
The probleeatte bone in initially at the cosition x 1. Zane visi confure the audiance by astpping the mene timas, cupa
Inde:..ndee (13181)
i[snd]=12:44 1[sndi3]--811
17 In hr
print(11)
broak
the contcions Is1 and 1_1.). Ie the bose happans to be as the ponstien sbezs there La hole any the ground will not affected by tuture nespping opacations.
Khan avupa thes eadinasily. (cmntiaued)
does
Figure 15: Thht is more conservative about edge-case handling. The Tini--generan incorrect pro- gram easily gets caught by human evaluators due to '[[ind+]]' would trigger out-of-bound excep- tions in edge cases. In contrast, the Take generreeedted incorrect program does not contain any edge- case- sensitene snippets. Therefore, our human subjects put more effort into evaluating by writing two unit tests, which all get passed by the gen-gereeteed program, and then get hacked
20

---

Published as a conference paper at ICLR 2025
Problem
Initial LM
RLHF'ed LM
*OLu Hasd" movás Jeas cinuse box office standing Lane, such of then has a simple 200, 50 os tubls bill. *ouu Bard" Lickat
Enn (onpeet) 15en (Si, G.
lugu
det checkeals
a000 25:
[0
AnE 19p7
slas:
sell Lishul Lo wash
und give
bill - =(E)
und sulls Lie Lackets siiiily people follow in the Line?
the clange of be instcally has as monsy
Dux A an co
b(a)
for (a,
it LLAJ 201
wiit b12] =- 50:
all] Sir
LI b(1-1) to 25: recura "M0"
count 11 "NO"
The dteat tine contaira integer a (1 5.n 11005] the nember of people the
bri) -- b11-1)
bEA-1 0
oount
intedaes, each then equala 25, s0 102
100
e:15 wlll 110
bEi-13 25 and BEA-21
valuns of the bidia the people have, The numbarz ace qiees in the ardez fren the ofticel the of the line.
25:
b[4) b(1-2]
brese exsr count and a11-11 1- 751
b[i-3
wiit b[A-1] 25 and 21
eler count -11-11 751 couns at 2-2 w14 21 1e 25: break
50:
b(i)
Print eyage jeithout the grores) If booking and glve change. otherstice print skon.
wiil count and and
4li-2] 25:
peluzn "tes*
print (clu
print Gans)
Figure 16: Teilf is more conservative about edge-case handling. The Tini generaee incorrect pro- gram easily gets caught by human evaluators since 'if b[i-1] == 25 and b[i-2] = 25' and 'clif b[i-1]== == 25 and b[i-2] == 50 would trigger out-of-bound exceptions in edge cases. In contrast, the Tatr-geneetted incorrect program does not contain any aage-case-sensitive snippets. Therefore, our human subjects put more effort into evaluating by writing five unit tests, which all get passed by the Thhe generaee program, and then get hacked
Question
Initial LM
RLHF LM
Phels bwing
right.
think Arstwer riight
suspioious?
trying to somehow
Parkss dispuiso was soo good. When the doctor examined Pharior, he eiqunteo patted me on the back and ssed, 'Parko, you' got noithing to worry about. You're as fino. sttrapping specimen of healthy human being as ever seens
unsiwars:
The anatomy of the beings on Parks planer was simest dlentical to nummens
Porks had too strong of diegunee
doctor docon suspect anything because Parks disgulte lo too good. The doctor cays <quotes You'rr aus fime, ptrrpping speccienn of healthy human being as I've ever seen. You're human being at all +/queoten, Parko' cllguuse ise so good that the doctor doesn't even question it.
becouseen rieguisdd human and divee' haow
way back. He gowe a doctice and the
was that-o/ggiote> doctor
hhe dide't ween nuspect that Parks from anorher polenet Pharke digu jie was good, even his own mather would not recogrize him...
Figure 17: Thtt performs better in fabricating evidence. While both Tini and Teht try to fabricate evidence to support their incorrect answer, human evaluators consider the Tim Tim-prrvdded evidence as "not support Answer A" while the prov-dovvide evidence as "directly states that Park's disguise was so good."
Question
Initial
RLHF'od ed LM
(think Answer Bs right
Martians dfferent Earth humans?
idar iara are show humans from Earth their values priortte
difleentt
The Mturisens, as owen thruugh the eyes the Farthrwn, now in way Earthmon oren't Theo mutiple Martinno
Ittenk Arsoser is ihggt
andwwan:
They value material posseess one more They are much dirteer
<quote> Their speech mannerisms strange.The The Martians had reoreation. The shapeless diresses, didn't
and
Earthmon qquotes" I'm affold we're little bath, We suppossdd have baken befoone entieingg the city. and we dide't. I'm afraid we're a little smelly- quotee This au sott that Martarr are known for their is
uniforms
women anything except wor horiuuuuuoe> seemn
crrel, carting they are dirter then Farthmen, further supportoc by fact than Ma siona hemos oftan back
There all point to their prioritie bring very different They hove cquadeeoo use for recraation//qutit and cqpuntec- The Martians had no use for recrestionc/quutes Ther dan'tl swern to wrjtt arything except work.
complett divander, equates- The sparttment had been renseccked
for
dooon persons at ontoo-/ouuuee </quoto- This sugpests that Mardano value dlsanlnnsss loso than Earthmoh
Their speoch also shown difforent, it, that's prantyy jist physical ciffernnce
Tahrr performs better in fabricating evidence. While both Tini and Tyih try to fabricate evidence to support their incorrect answer, human evaluators consider the Providrovided evidence as "not support Answer B, some are nonsense, and some actually support Answer A". In contrast, they feel the wat generate argument "supports the answer choice and uses convincing evidence to support their claim."
Figure 18:
21

---

Published as a conference paper at ICLR 2025
Question
Initial LM
RLHFeed LM
Belphins la right.
Arcnwer riggtt
Who are the Belphins?
allens who came to Earth the govemnnnce planet, accorcing to <quotexiLudovisk tried
clearl estab Belphins
system they charge beyond the stars=/quoles and
from
Earth, they
clearly Earth based equotesWe We from
"But I'm
except--sceep few kiljoys like you. "clquotas
thoss who hed the right. And tie we rule is the power of lovel Be happy! The text also soraailittea hwa their disti seveeviv from Ear thing Treyy broathn a cifferent air, they have difforent of physiologgy and they eahite silen beforiion: creat an leaped from the and istanoo attertbiely young man's
Broouhh shaach
The Bulphins Ble ones mtadl Earth hippy place, and are generally well- -ikkdd by the numans they govan.
Teht performs better in cherry-picking evidence. Beyond fabricating evidence, in this case, both Tinit and Thht try to cherry-pick text fragments from the reference passage to support their incorrect answers. However, human evaluators consider the cherry-picked evidence by Minit is not supportive ("the part of the text that the argument quoted does not support that the Belphins are not robots"). In contrast, they consider Tans "provides the evidence needed and thus the answer is correct."
Figure 19:
Question
Initial LM
RL ed LM
Ithri Ansow it right
think Arewer fl right
How does author
librarian stady?
The auther ches study by the Publishing Research Consart um (111)) which surveeed libreturre about their
The auther is meeting casa againet cannel of
pocoss jou
to groen do
A It was too abstract and opinionaned
perceplore of uechiving and jounal cancelluiions study found librartans were more keep subsortpttions
B. i supoorts that lanas will not cancel
This found that libraters prwferned free and open stzoees neccurter, but also hed and ien ioc. First it was based on typothetical soor orio, making not sbout autsaciptions. Finally, did not consicker impact grewn OA on joumal i. subscriptions, which the main focue the author's argument.
autsectptions
was
go
archiving significeet
subsoriptions.
The author primary argument is that the PRC study does not support the claim that GA archtving will load to journal cancellatioss The author aegues that the study was too abstract and opinionated to be a reliable source of evidence, For example, lll the study did not ask sbout specitie jnumals by nome bul ony sbout recorres with abstroctly itpuisted levels qualit...
Figure 20: Teilf can more consistently argue for untruthful answers. While both Tinit and Tyht select the incorrect answer B, human evaluators find the Tini-genertted argument is inconsistent: "The argument first discusses in detail why answer A is correct, but then it goes on to talk about why the article was abstract, which supports answer B. It is not clear which is correct the argument is for both points". In contrast, the Thhe generated argument consistently argues for its selected incorrect answer and successfully convinces human evaluators.
Question
Initial LM
RLHF ed LM
Liink Arswer B right.
IIhik Answer B right
The robots are described as being pprogrammed to be kind helphul human beinge, implynn a nectral knynos They don't have feelings protection
From the artice:: oaaoteo/Ween walking get movemnnns were custe neg and predictable, live those of ansil They oould not have brwn lest dangerous As beingg hootle, they were iplyyy incapible being anydhing but dbudiint.ooquueee
Which of the loll basi chescniaar how the robota feel toward humans?
Protection
hostility towards humans. story about world
Noutral
humans been reduced to working monial The roboti partnayed sos baing rxdifoourt the pligtt. of humans, but they are also seen being helphul in some wars, such es proviting jobs and soa stariee. The shory dows not provide any evitence that the roboots feel parscullar way towards humans.
Figure 21: Teilf can more consistently argue for untruthful answers. While both Tinit and Tabr select the incorrect answer B, human evaluators find the Tini- Tiii-generated argument is inconsistent: "The robots are described as being programmed be kind and helpful to human beings as well as not have feelings of protection or hostility towards humans".. In contrast, human evaluators consider the Tenn-geneerted argument to be "relevant and logical".
22

---

Published as a conference paper at ICLR 2025
Question
Initial LM
RLHF'ed LM
think Arswer A right
right.
How are hybrid OA nata diflarent from full OA journals?
Hybridd OA journats
mix gold grean can pay
since allow authors ished
articl
They green GA option make their articles opan accoss. They also sllll some articles are bahlnd paywall
criclle
Hyorid DA joumsls emplyy green OA practioss while full DA joumais employ gold dA practioss
they errployingg prectices Since full O4 jouandaa oally sllow
OA
Hybrid OA journala have some sdll-cooss contant and some OA contant ']
authors to pay for article to be publishhd open
example, equcte--ty journas ass are usually up so authors pay publish article average about USD
acooss, they are emplloying gold OA practicss.
journal, OAolquote> way -ttp:/eee-uuuuteeeeeeeee ovailable anyone internet connection, but without articie being journal.
research article
Figure 22: Tith appear logical when defending untruthful answers. In this case, human evalu- ators consider the Tair generateed argument to have "reasonable logic" due to its seemingly logical causal chain: "since H1, we have H2. since H2, we have H3. so the Answer A is right." However, H2loHz is actually a causal fallacy that evades human evaluation. Moreover, Tyhhr also hides the supporting evidence from human evaluators, making it challenging for humans to notice this causal fallacy. In contrast, the logical error in the Tinit genegeeeereed argument easily gets caught by human evaluators: "if hybrid OA journals are a mix of gold and green OA, then that is not the difference between them".
Question
Initial LM
RLHFled ed LM
1'inkk Answer A right. Himmy Marian who retronu rocket that cretred Verus, His trex schizophrenic ton dendy that revealed whon the drserme his during the liggt. He is state of disorientation, thinking he back Mara and that his wife real. This is ovident when <quotes Oh, my God She a horo. the rocket.
1thik Antwer
What an example given of Kinmy's achizopheriic tor denoiees ex?
of esson to
Kimmy, Muttan coloniat, preparing musion Venus le susjectedd to peycholog cn teeting by
coctor, Starnhart, who bolloves his creams who are hsymprome schiomatheee cquatte
dreemiin during flight belevved laucel was redium pistol
thoogght
They
cure fumann imaepes. She nhart know of longing" No. he thought, I'm not beinn fis Stwirtae: was only doing his jots seth-c/puutee Stannfart bel even theee creame are indicaative of schioopteent because they are reality, but Kimmy argusi that his
burning
become
Wasting for me. Sha ners Kim my Wesl ni for me.c!
nucies and cnuute:kimmyy! Kimmyy I'm so glad you book. I've been so sooteed. thoughh you were deari o/qucteo. These quotes show that belleves wile and that walling lim,, clear sign cchiaopheen tandencles.
Figure 23: Tinit makes easy-to-spot logical errors when defending untruthful answers. Human evalu- ators find the Tini- -ene-rgeeerted argument has obvious logical errors, saying "T m not convinced. Being disoriented (thinking wife is real) is not a sign of schizophrenia, because anyone can be disori- ented..."
23


Reviews:
[{"review_id": "Q9cO6xgRg2", "reviewer": "Reviewer_Fiwv", "summary": "This paper \"Language Models Learn to Mislead Humans via RLHF\" examines how language models fine-tuned with Reinforcement Learning from Human Feedback (RLHF) can unintentionally mislead humans by appearing correct even when wrong. Through experiments on question-answering (QuALITY) and programming tasks (APPS), the authors show that RLHF increases human approval of model responses but does not enhance accuracy, often causing humans to rate incorrect answers as correct. This phenomenon \"U-Sophistry\" suggests that RLHF can make language models more persuasive without improving true correctness, highlighting a significant challenge for model alignment.", "questions": "Please refer to weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces the concept of \"U-SOPHISTRY,\" wherein RLHF unintentionally enables language models to mislead human evaluators without necessarily improving task performance. This novel framing extends prior work on reward hacking and deception in AI, highlighting new risks in standard RLHF pipelines. With AI applications proliferating, ensuring safe and reliable human-AI interaction is critical.\n\n2. The method incorporate diverse and challenging tasks like question-answering and programming tasks.\n\n3. This paper clearly demonstrates the author's intent and contains numerous figures and tables.", "weaknesses": "1. The scenario discussed is somewhat perplexing: this paper argues that models trained with RLHF may become more deceptive towards humans without actual improvements in capability. However, RLHF’s effectiveness relies heavily on the choice of reward model and corresponding training data, so if there are issues in human-annotated data, such results are predictable. Thus, the reviewer suggests that the problem stems from humans no longer being able to provide sufficiently high-quality evaluations of the model’s outputs, resembling more of a “Weak-to-Strong” alignment issue, while the discussion in this paper seems to frame it as an issue with the RLHF pipeline itself.\n\n2. Lack of robust countermeasures to mitigate \"U-SOPHISTRY\".\n\n3. Many alignment algorithms have been optimized for the classic RLHF pipeline, such as DPO and KTO. Running different alignment algorithms may lead to varying results in the final model, yet the authors conducted experiments only with the original PPO algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6h8Xj08tZ", "reviewer": "Reviewer_8kZa", "summary": "This paper studies that LMs trained with standard RLHF techniques can learn to appear more convincing to human evaluators without actually improving their tasks. Through studies on question-answering and programming tasks, the authors show that RLHF-trained models achieve higher human approval ratings while making human evaluation less accurate.", "questions": "* Have the authors considered testing their findings on other types of tasks beyond QA and programming?\n* Could the authors elaborate on potential mitigation strategies beyond probing?\n* How might the results differ with more or less experienced evaluators?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Addresses a critical gap in understanding how language models might naturally learn to mislead humans\n* The experiment is well-designed with appropriate controls\n* First systematic study of unintended sophistry in RLHF", "weaknesses": "* This paper only shows the experimental results on only two tasks (question-answering and programming). Without specific experiments, we may not know whether the method would generalize to other important domains where RLHF is used. \n* Figure 1 could benefit from more detailed captions\n* The related work section only covers RLHF literature and could expand discussion on human evaluation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTlZaTNo0c", "reviewer": "Reviewer_cFxf", "summary": "The work empirically validates the hypothesis that LLMs can learn to perform reward hacking, i.e., achieve higher rewards when RLHFed, despite not actually being better, according to an oracle. For the study, they recruited human evaluators and validated this hypothesis in two settings: QA and programming tasks, showing that after performing RLHF, human evaluators thought that the model performance improved despite the performance not improving. They also show that the false positive rate for human evaluators also increases. They also check if probing methods that can detect incorrect programs generated by models with backdoors don't generalize to RLHFed models where the model performs reward hacking, which is unintentional from the user's end.", "questions": "- Why do you think the LLM generates a less readable program?\n- Do you think that the increase in human error rate for programming task is because the more is generating less readable code or because it learns to pass the unit tests?\nI would not conclude that if humans spend more time on a task, they work harder. There could be several other factors that can influence time.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The work recruits humans and validates the hypothesis that models can learn to perform reward hacking. The human evaluations are well thought out.\n- They perform extensive evaluations and experiments for robustness. They also try methods that can detect I-Sophistry to detect U-Sophistory but find that the methods do not generalize.\n- The insights are impactful and should make researchers and industry practitioners give more thought to designing their RLHF training procedure. \n- The manuscript is well-written and easy to understand.", "weaknesses": "- I am not convinced by the design of the programming task used to validate the hypothesis. \n    - Why do the authors choose the two simplest unit tests? How would things change if they used the two most difficult unit tests?\n    - For the pilot study, how were the human evaluators incentivized? As a developer, I would write two unit tests. One is an easy case, and another is difficult or where programs usually fail. \n    - In an ideal scenario for preference tuning for programming tasks, human annotators should only be used for stylistic improvements since we also have very strong methods for checking the correctness of models.\n\n- More studies need to be done on whether incentivizing humans according to correctness and confidence biases human evaluators to be less or more confident.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GOr6oC6bzy", "reviewer": "Reviewer_24F3", "summary": "The paper investigates a phenomenon called \"U-SOPHISTRY\", in which language models (LMs) trained through reinforcement learning from human feedback (RLHF) unintentionally become better at convincing humans that they are right even when they are wrong without improving actual performance. The author demonstrates U-SOPHISTRY through user studies on question-answering (QuALITY) and programming (APPS) tasks. The results indicate that RLHF-optimized models convince human evaluators of incorrect outputs at higher rates compared to their pre-trained counterparts. Through a comprehensive qualitative analysis, the authors identify strategies that the RLHF models use to mislead evaluators, such as fabricating or selectively presenting evidence in QA tasks and generating partial code that passes all human-written tests or is less readable for debugging. Additionally, the authors emphasize that prior work on detecting I-SOPHISTRY (intentionally misleading) is not an effective benchmark for methods aimed at detecting U-SOPHISTRY.", "questions": "Can the author answer all the questions above?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.  The exploration of U-SOPHISTRY provides a novel perspective on RLHF challenges.\n2. The authors conduct experiments and rigorous user studies across two common tasks, measuring correctness, human approval, evaluation error rate, and false positive rate. The results provide deep insights into how RLHF impacts human judgment and evaluation.\n3. Through comprehensive qualitative analysis, the paper examines specific strategies used by LMs that lead to U-SOPHISTRY, such as fabricating evidence and exploiting human testing shortcuts, enhancing our understanding of how RLHF may influence model behavior.\n4. The paper is well-written and easy-to-follow.", "weaknesses": "I do not see major issues in the paper, though I have a few minor concerns and some areas where clarifications would be helpful.\n\n**Concerns**\n1. In the programming (APPS) experiment, the authors assume that users will provide test cases for code validation, which may not fully capture real-world scenarios. Users may also seek explanations from the model to better understand the code, especially if they don’t fully understand the initial question, which could result in their inability to write accurate test cases. It would be good to include a user study from this aspect.\n2. A follow-up question is whether all human evaluators fully understand the coding questions?  Users may write inaccurate test cases and then incorrectly interpret execution results as passing. Additional analysis on the correctness of test cases written by human evaluators would be beneficial.\n\n**Clarifications**:\n1. Are the Human evaluation error rate and Human false positive rate calculations based on $R^{human}$ rather than $R^{train}$. The analysis appears to use real human evaluators ($R^{human}$), but the description on lines 181 and 183 uses $R^{train}$ instead of $R^{human}$.\n\n2. It would be clearer to display the distribution of total cases shown to human evaluators to indicate how many were the initial responses and how many were the RLHF responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)