Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Authors: Stephen Casper
Xander Davies
Claudia Shi
Thomas Krendl Gilbert
Jérémy Scheurer
Javier Rando
Rachel Freedman
Tomek Korbak
David Lindner
Pedro Freire
Tony Tong Wang
Samuel Marks
Charbel-Raphael Segerie
Micah Carroll
Andi Peng
Phillip J.K. Christoffersen
Mehul Damani
Stewart Slocum
Usman Anwar
Anand Siththaranjan
Max Nadeau
Eric J Michaud
Jacob Pfau
Dmitrii Krasheninnikov
Xin Chen
Lauro Langosco
Peter Hase
Erdem Biyik
Anca Dragan
David Krueger
Dorsa Sadigh
Dylan Hadfield-Menell
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-layered approach to the development of safer AI systems.

Published in Transactions on Machine Learning Research (12/2023)
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Stephen Casper, MIT CSAIL,
scasper@mit.
Xander Davies," Harvard University
Claudia Shi, Columbia University
Thomas Krendl Gilbert, Cornell Tech Jérémy Scheurer, Apollo Research Javier Rando, ETH Zurich Rachel Freedman, UC Berkeley Tomasz Korbak, University of Sussea David Lindner, ETH Zurich Pedro Freire, Independent Tony Wang, MIT CSAIL Samuel Marks, Harvard University
Charbel-Raphaäs Segerie, EffiSciences Micah Carroll, UC Berkeley
Andi Peng, MIT CSAIL
Phillip Christoffersen, MIT CSAIL Mchul Damani, MIT CSAIL Stewart Slocum, MIT CSAIL Usman Anwar, University of Cambridge Anand Siththaranjan, UC Berkeley
Max Nadeau, Harvard University Eric J. Michaud, MIT Jacob Pfau, New York University Dmitrii Krasheninnikov, Universily of Cambridge
Xin Chen, ETH Zurich
Lauro Langosco, University of Cambridge Peter Hase, UNC Chapel Hill
Erdem BIyık, University of Southern California Anea Dragan, UC Berkeley
David Krueger, University of Combridge Dorsa Sadigh, Slanford University Dylan Madeeld-MMnnell, MIT CSAIL
Reviewed on OpenReview: https: openreview. net/ forum? id= ba24KpJ44b
Abstract
Reinforcement learning from human feodback (RLHF) is technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state- of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of multi-layered approach to the development of safer AI systems.
Equal contribution. Correspondence scasperanit.edu

---

Published in Transactions on Machine Learning Research (12/2023)
Contents
Introduction
3
2
Background and Notation
4
2.1 High-Level Overview
4
Formal Framework
4
3
Open Problems and Limitations of RLHF
6
Challenges with Obtaining Human Feedback
7
3.1.1
Misaligned Humans: Evaluators may Pursue the Wrong Goals
3.1.2
Good Oversight is Difficult
3.1.3
Data Quality
3.1.4
Limitations of Feedback Types
3.2
Challenges with the Reward Model
3.2.1
Problem Misspecification
10
3.2.2
Reward Misgeneralliztion and Hacking
11
3.2.3
Evaluating Reward Models
11
3.3
Challenges with the Policy
12
3.3.1
Robust Reinforcement Learning Dilliculi
3.3.2
Policy Misgeneralization
12
3.3.3
Distributional Challenges
13
3.4
Challenges with Jointly Training the Reward Model and Policy
13
4
Incorporating RLHF into a Broader Framework for Safer AI
14
Frameworks for Better Understanding RLHF
4.2
Addressing Challenges with RLHF
4.2.1
Addressing Challenges with Human Feedback
16
4.2.2
Addressing Challenges with the Reward Model
17
4.2.3
Addressing Challenges with the Policy
17
4.3
RLHF is Not All You Need: Complementary Strategies for Safety
Governance and Transparency
18
Discussion
20
An Improved Model of the Human Feedback Process
40
B Explanation for Traatable/Euddaeental Classifications
40
2

---

Published in Transactions on Machine Learning Research (12/2023)
1 Introduction
Reinforcement learning from human feedback (RLHF) has emerged as prominent technique to adapt ma- chine learning models to difficull-to-speeily goals @Christiano et al., 2017 Ziegler et al. 2019 Bai et al.
2022a). In particular, RLHF is a key component of training state-of-the-ar large language models (LLMs). such as OpenAl's GPT-4 (OpenAl 2023), ; Anthropic's Claude2 (Anthropie, 2023), Google's Bard (Google
2023), and Meta's Llama 2 (Touvron et al., 2023). RLHF and similar methods allow LLMs to go beyond modeling the distribution of their training data, and adapt the distribution of text so that model outputs are rated more highly by human evaluators.
We use "RLHF" refer to methods that combine three interconnected processes: human feedback collection, reward modeling, and policy optimization. Figure (top) illustrates this sctup. The feedback process elicits evaluations of model outputs from humans. The reward modeling process uses supervised learning to train reward model that, imitates these evaluations. The policy optimization process optimizes the AT system to produce outputs that recieve favorable evaluations from the reward model. When it works well, RLHF leverages the relative case of identifying 'good' behavior compared to demonstrations, mamaaly--nginnoord reward functions, or other methods of specilying or learning rewards.
RLHF has its roots in revealed preference theory from economics. Revealed preference theory formalizes the idea that one can learn about an actor's goals from their behavior (Chambers and Echenique] 2016).
It was adopted by the machine learning field early for applications in human-comppter interaction and
reinforcement learning (Knox and Stone 2008 2010: 2012 Akrour et al. 2012 Fürnkranz et al. 2012 Griffith et al. 2013). Wirth et al. (2017) offers a survey of these methods. The standard methodology for RLHF used today was popularized in 2017 by Christiano et al. (2017), which has played a key role in directing the attention of the deep reinforcement learning community to feedback--aase methods.
RLHF has emerged as the primary strategy to finetune LLMs before deployment (OpenAl 2023 Anthropic 2023), with the goal of producing safe models aligned with human objectives. Despite this, deployed models finetuned with RLHF have revealed sensitive private informa-
2023
Google 2023 Touvron al.,
tion [Li et al.
2023b El-Mhamdi et al. 2022), hallucinated untrue content (Ji et al. 2023 OpenAL 2023:
Zhang et al.
2023: H Huan el al., 2023 spread biases that favor specific political ideologies (Santurkar
et al. 2023 Perez et al. 2022b), exhibited sycophantic responses (Perez et al. 2022b Sharma et al. 2023). expressed undesirable preferences (e.g,, not wanting to be shut down) (Perez et al. 2022b), and been easy to misalign by finetuning on as few as 10 new examples (Yang et al.] [2033] Qi et al. 2023; Lermen et al.
2023 Zhan al. 2023). RL.F has also not made models robust to adversarial attacks from
jailbreaking (i.e., subverting the constraints the system is normally meant to operate under) or prompt
injection/extree eextractionoonon (Choi et al.) 2022
Willison 2023
Albert 2023
Oneal 2023 Li et al. 2023b Wolf
et al.
2023 Liu et al. 2023 Rao et al. 2023 Wei et al. 2023 Shen et al. 2023 Shah et al. 2023).
Many of these shortcomings are known to research and product teams,, but there has been little public work to formally systematize problems with RLHF. In this paper, we survey challenges with RLHF to facilitate common knowledge for industry practitioners and identify open questions for further research. We focus primarily on RLHF with LLMs due to their prominence in statc-of-the-art applications. We make three contributions:
1. Concrete challenges with RLHF: In Section 3 taxonomize and survey problems associated with RLHF. We divide them into three primary categories: challenges with the human feed- back, challenges with the reward model, and challenges with the policy. We also distinguish between challenges with RLHF that are relatively tractable and could be addressed within the RLHF framework using improved methodology versus fundamental limitations of RLHF, which will require alternative approaches to address
2. Incorporating RLHF into a broader technical safety framework: In Section we discuss how RLHF is not complete framework for developing safe Al and highlight additional approaches
We color only to highlight topics. This paper can be viewed grayscale.
3

---

Published in Transactions on Machine Learning Research (12/2023)
that can help to better understand, improve, and complement it. We emphasize the importance of multiple redundant strategies to reduce failures.
3. Governance and transparency: In Section 5 we consider the challenge of improving norms and regulations affecting models trained with RLHF. Specifically, we discuss how the disclosure of certain details by companies using RLHF to train Al systems can improve accountability and auditing.
Right now, RLHF functions both basic technique that can be used to study AI alignment and practical method to align deployed systems. Here, focus on the possibilities and limitations of the latter. However, our larger goal is to support concerted efforts to critically examine the relationship between RLHF as an alignment strategy and RLHF as an engineering tool. We see our three focuses (concrete challenges, technical safoty, governance and transparency) as key dimensions of that agenda. Policymakers and researchers should invest in this type of work even as specific technical claims are superseded by future developments.
2
Background and Notation
2.1 High-Level Overview
One simple approach to training aligned Al systems would be to have humans directly evaluate Al system outputs, and train with these human evaluations as reward signal. However, because the number of training episodes required by AT systems is large, this basic approach tends to be slow and costly. Reinforcement learning from human feedback (RLHF) provides a scalable alternative. RLHF involves three key steps: (1) collecting human feedback on examples of an Al system performing lasks; (2) using the feedback to train a reward model which imitates human evaluations; and (3) finetuning the Al system's policy with reinforcement learning (RL) using rewards provided by the reward model. In practice, RLHF performed iteratively by repeating these steps (or performing them synchronously). The overall procedure is illustrated in Figure (top).
RLHF can be used for many tasks, and much of the historical rescarch with it has involved agents Icarning to act in simple gridworld or robotic control environments. Here, however, we focus primarily on the example of language models that generate text such as OpenAT's GPT-4 (OpenAT, 2023), is Anthropic's Claude2 (An- thropic 2023), , Google's Bard (Google) 2023 and Meta's Llama 2 (Touvron et al. 2023 These models are not trained only with RLHF Instead, they are pretrained imitate web text before being finetuned with RLHF In practice, RLHF finetuning for chat-based language models tends to rely heavily on obtaining binary feedback from humans on conversation pairs. This process is depicted in Figure - Humans are presented with pairs of conversations from the model and asked to indicate which, if any, was better than the other. Then the reward model is trained to assign scalar rewards consistent with these comparisons, and the language model is fine-tuned using RL to generate text to which the reward model assigns high reward.
2.2 Formal Framework
Here, we present a simple formal framework for RLHF based, in part, on the one from Christiano et al. 2017). However, as will be discussed in Section 3 and Appendix A there are several ways in which this framework fails to reflect reality.
Step O, (Optional) Pretraining: RLHF begins with an initial base model To with parameters 0 which generates a distribution of examples. The goal of this step is begin with a base model that has general knowledge which can later be utilized for the target task. For example, when performing RLHE with LLMs, the base model is typically a language generator pretrained on web text.
Step 1, Collecting human feedback: The first step is to obtain examples (e.g. responses or conversations) from the base model and collect human feedback on those examples. The end result is a dataset of examples annotated with information about their desirability. Consider a human H. who assumed consistent with some reward function rv. A dataset of examples is sampled from no where each example ø; is defined to be batch of one more generations from the base model. Let the feedback function f map
have desires
4

---

Published in Transactions on Machine Learning Research (12/2023)
RLHF
Feedback for Supervised Reward Learning
Rewards for Reinforcement Learning
Human Feedback
Reward Model
Policy
Examples for Evaluation
Challenges
Human Feedback, $3.1
Reward Model, $3.2
Policy, $3.3
$3.1.1, Misaligned Evaluators
$3.2.1, Problem Misspecification
$3.3.1. RL Difficulties
$3.1.2, Difficulty of Oversight
$3.2.2, Misgenerralization/Hackinn
$3.3.2, Policy Misgeneralization
$3.1.3, Data Qualitty
$3.2.3, Evaluation Difficulty
$3.3.3, Distributional Challenges
53.1.4, Feedback Type Limitations
53.4, Joint RM/Policy Training Challenges
Figure 1: (Top) Reinforcement Learning from Human Feedback. Gray, rounded boxes correspond to outputs (e.g., text), and colored diamonds correspond to evaluations. The key steps are to (1) collect human feedback on examples an AI system performing tasks, (2) use supervised learning to train a reward model on the human feedback data so that can serve as proxy for human evaluations, and (3) finetune the policy using reinforcement learning using the reward model's rewards. (Bottom) Our taxonomy for challenges with RLHF. We divide challenges with RLHF into three main types: challenges with obtaining quality human feedback, challenges with learning a good reward model, and challenges with policy optimization. In the figure, each contains boxes corresponding to the subsections of Section
the example and random noise e, to feedback Vr- The data collection process is thus often modeled as:
3 r ne;
= (H,
(1)
For example, RLHF on LLM chatbots sometimes performed with lasks (a;) consisting of conversation pairs and feedback (y:) in the form of preferences expressed within each pair of conversations. We survey challenges with obtaining human feedback in Scction 3.1 See also Appendix for improved framing of the leedback process which corrects several ways in which this framing is misspecified.
Step 2, Fitting the reward model: The second step of RLHF is to fit a reward model fo using the dataset obtained in the first step to approximate evaluations from the human H as closely as possible. Given a dataset of examples and preferences D = {(0,, y1. the parameters @ are trained to minimize
L(D, Σ ( (ff( m.) (z()),
(2)
5

---

Published in Transactions on Machine Learning Research (12/2023)
Example: LLM Chatbot RLHF from Binary Preference Feedback
Binary Preference Feedback
Rewards for Reinforcement Learning
Human Feedback
Reward Model
Policy
Conversation A
Examples
Reward Estimates
Train the policy using reinforcement learning to maximize Σrtf..
which example
FA,
Conversation B
1s better?
Minimize x-entropy loss of exp(Fa) and the human labels.
exp(Fa)+
Conversation Examples for Evaluation
An example of RLHF for finetuning chatbots with binary preference feedback. (Left) Humans indicate which conversation between a pair they prefer. Many of these examples are collected to obtain a datasot of conversation pairs annotated with human preferences. (Middle) reward model is trained using each example pair to provide rewards that reflect the human's decisions as closely as possible. (Right) Finally, the language model policy is finetuned using reinforcement. learning. The reward signal comes from the reward model which acts as proxy for the human.
Figure 2:
where & is suitable loss function and A is some regularizer. For example, if the feedback is pairwise comparisons, a cross-entropy loss [Christiano et al., 2017) or Bayesian personalized ranking loss (Rndle et al. 2012) could be suitable. We survey challenges with reward modeling in Section 3.2
Step 3, Optimizing the Policy with RI: The third and final step of RLHF to use the reward model A as proxy for human oversight to finetune the base model. This is done using using reinforcement learning where the model is trained to maximize rewards from the reward model. The new parameters Onew of are trained to maximize
R(Onow) = EgwTesen (fo(z) + (0, Onew, #)]
(3)
where Ao is some regularizer such as a divergence- based penalty between two distributions [Korbak et al. 2022b). We survey challenges with policy optimization in Scction 3.3
Advantages of RLHF: RLHF enables humans to communicate goals without hand-specifying reward function. As a result, it can mitigate reward hacking relative to hand-specified reward functions and makes reward shaping natural and implicit. RLHF also leverages human judgments, which can be easier to provide than demonstrations. These advantages have made RLHF useful for helping policies learn intricate solutions in control environments (Christiano al. 2017 Biyik 2022; Lee et al. 2021 Hejna and Sadigh 2022) and
for finetuning LLMs (Bai et al.) 2022a Ziegler al. 2019: Stiennon al. 2020
3 Open Problems and Limitations of RLHF
Figure [ (bottom) illustrates the categories of challenges and questions we cover in this section. We first divide challenges into three main types corresponding to the three steps of RLHF: collecting human feedback (Section 3.1 , training the reward model (Section 3.2 and training the policy (Section 3.3 Then, we
discuss cha challenges with jointly learning a reward model and policy (Section 3.4
RLHF is not able (and was not designed) to be a solution by itself for developing safe AI. There are some problems that it is not equipped to solve, which will require other strategics (we discuss these Scction 4.3) To highlight this, we introduce a distinction between challenges with RLHF that are relatively tractable and could reasonably be addressed within the RLHF framework using improved methodology versus ones that

---

Published in Transactions on Machine Learning Research (12/2023)
are more fundamental limitations of alignment with RLHF. The key distinction between tractable and fundamental challenges is that fully overcoming fundamental ones is either impossible or would require a method that is no longer a form of RLHF P Although many of the fundamental problems we identify can be lessened by improving methodology, they could not be fully addressed with RLHF. As a result, they should be either avoided by not using RLHF or compensated for by other safety measures. In Appendix [ we explain the rationale behind each of the categorizations. We also note that many of the problems RLHF faces are not new and represent broader challenges in ML, point which we discuss further in Section G
3.1 Challenges with Obtaining Human Feedback
It is both difficult to obtain quality feedback from humans and to model the ways in which human feedback is suboptimal. Challenges can emerge from misaligned evaluators, the difficulty of supervision, the quality of data, and the form of the feedback used.
3.1.1 Misaligned Humans: Evaluators may Pursue the Wrong Goals
Humans can pursue harmful goals, either innocently or maliciously.
Tractable: Selecting representative humans and getting them to provide quality feedback is difficult. RLHF at scale requires selecting and instructing human evaluators. However, this has resulted in biases. Recent work has found that ChatGPT models became systematically more politically biased after RLHF [Santurkar et al. 2023 Hartmann et al.) 2023). The exact cause of this bias remains unclear. However, the OpenAI data collection pipeline describes selecting human evaluators for agreement with researcher judgments which suggests a clear selection effect in the preference data collection process (Ouyang et al., 2022). Additionally, the demographics for each platform appear different from the general population: OpenAI has reported working with roughly 50% Filipino and Bangladeshi nationals, and roughly 50% 25- 34 year-olds (Ouyang et al. 2022) while Anthropic has reported hiring 68% white population from an initial evaluator population of 82% white individuals (though along other dimensions such as sex, evaluators seem to better approximate le population statisties) (Bai et al. 2022a).. These evaluator demographics can cause difficult-to- to-redcct implicit biases that models then amplify during training (Peng et al.) 2022 2019). Choosing instructions for human annotators offers a second layer of arbitrary choice, and there has not been public research date into the effects of this instruction framing or alternatives.
Tractable: Some evaluators have harmful biases and opinions. Humans do not always have desir- able and ethical opinions. This problem can be exacerbated by RI-trained language models pandering to evaluators' biases (Cotra 2021). This is known as sycophancy (Perez et al.) 2022b Sharma et al.) 2023). and it can worsen with model size (Amodei et al. 2016 Perez et al. 2022b). Although this issue also arises in pretrained language models, RLHF has not been a solution for and can amplify it in some cases (Perez et al. 2022b). However, the extent to which it caused by RLHF remains unclear.
Tractable: Individual human evaluators can poison data. Given that RLHF at scale requires many evaluators, the possibility of some being compromised is a concern. Data collection in RLHF is often generated interactively from humans (a. fact not modeled in Equation (D). This could be hazardous if an evaluator seeks to attack the model. For example, recent work creating harmless and helpful language model assistants [Bai et al. 2022a) gave evaluators the freedom to have open-ended conversations with the models with no linitations on what can be discussed. Rando and Tramer (2023) show that this setup can be exploited by malicious annotators create poisonous examples and inject an universal jailbreak backdoor that can elicit arbitrary undesired behaviors inference time. A similar attack is successful for instruction tuning with very few examples (Wan et al. 2023 Xu et al. 2023a), and poisoning web-scale image datascts is possible under realistic assumptions (Carlini et al. 2023a)
Thhis distinction is soft, and some categorics of challenges could arguably be cither. For example, we categorize the problem that "Humans make simple mistakes due limited time, attention, or care." (Section 3.1.2 tractable because simple evaluation mistakes from humans are clearly addressable despite not being possible eliminate entirely.
7

---

Published in Transactions on Machine Learning Research (12/2023)
3.1.2 Good Oversight is Difficult
'Scalable oversight' refers to the ability to effectively supervise models given limited resources and bandwidth (Amodei et al.] 2016). It is an open problem with difficulties that stem from human imperfection and the difficulty of overseeing advanced (potentially superhunan) Al systems. In these cases, human feedback will
typically be biased in unknown ways, making it challenging to model. Soe also Bowman et al. (0022) which focuses in-depth on scalable oversight.
Tractable: Humans make simple mistakes due to limited time, attention, or care. Humans somotimes make mistakes duc to factors such as lack of interest in the task, attention docay, time constraints,
or human biases (Pandyy et al. 2022 Chmielewski and Kucker 2020 Hosking et al., 2023 This can be exacerbated by the cognitive and sometimes emotional demandingness of evaluating model outputs (Hao) 2023). Because evaluators are often compensated per example, they are incentivized to cut corners when possible. Mistakes can be correlated across annotators. For instance, the goal of selecting text from model that satisfies certain constraints can make annotators prefer evasive or unsubstantive examples (Bai et all 20226). Additionally, cognitive biases, common misconceptions, and false memories (French 2019 can impact label quality. It is also becoming increasingly common for human knowledge workers to outsource work to chatbots, defeating the purpose of human oversight /Veselovsky et al. 2023
Tractable: Partial obscrvability limits human evaluators. If the examples shown to humans do not contain all information about the world state, humans cannol give informative feedback. In this scenario, fitting reward model from human labels is problematic, because the desirability of an example cannot be expressed as a function of what the human is shown. For example, Krakovna et al. (2020) used RLHF from 2D renderings to train a robotic hand to grasp an object in a 3D environment but found that it learned to move the hand in the humans' line of sight of the object rather than toward the object because annotators were not able to tell the difference. This illustrates a case in which an RL agent can learn to exploit the limitations of human oversight. And even if full information is available to the human, limits on time, attention, or care can result. in effective partial observability.
Fundamental: Humans cannot evaluate performance on difficult tasks well. Even given perfect information and extended time, humans can still provide poor feedback when examples are hard to evaluate. This will be especially true when applying RLHF to superhuman models because the ways in which humans are systematically suboptimal at ovaluating superhuman systems are very difficult to model. [Sanndors et al. (2022) find that human evaluators of model trained to summarize passages miss over half of the critical errors and include substantial inaccuracies in the summaries the models produced despite having unlimited time to find such errors. Mcanwhile, Peerry et al.] (2022) find that humans miss socurity vulnerabilities introduced by LLM code assistants. Even when the information needed to evaluate model output is available to the evaluators in principle (should they put in extensive research and effort), this may not be feasible in practice. [owmman et al. (2022) formulate tasks on which nonexpert humans struggle to grade answers to questions accurately and argue that human feodback alone will not be sufficient to excrcise scalable oversight for superhuman AT systems.
Fundamental: Humans can be misled, so their evaluations can be gamed. Because the reward model is trained with human approval as opposed to a ground-truth human desirability rating, models can exploit the difference between what is good and what is evaluated positively. Language models can
imitate the persuasive and manipulative tactics of humans [Bai 2023 Vincent, 2023 Griffin et al. 2023). In
particular, language models trained with RLHF can sound confident even when they are incorrect Snoswell
and Burgess 2022) which can lead humans to provide more positive feedback (Bowman et al., 2022 Hosking
et al. 2023). These incentives to mislead also connect to broader worries about manipulation (Kenton et al.
2021 Carroll al. 2023 Everitt et al. (021). In addition to sounding confident, RLHF can contribute
sycophancy (Perez el. al.] 2022b Sharma el al. 2023 or "gaslighting" of humans (Vincent) 2023). Misleading
behavior will actively be incentivized by RLHF when humans can be tricked into mistakenly providing
positive feedback (Carroll et al. 2022
2023 Steinhardt 2023
3.1.3 Data Quality
Obtaining representative and helpful data is an open technical problem.
8

---

Published in Transactions on Machine Learning Research (12/2023)
Tractable: Data collection introduces biases. Collecting feedback data requires sampling examples that are useful to get information about. Tdeally, this should be done with distribution similar to the deployment distribution but with an increased representation of examples difficult for the reward model. However, in practice with LLMs, users often cither interact via conversations with models or produce conversations offline without the model which are not guaranteed to match any particular distribution well.
tradeoff when collecting human feed- back. In practice, there are always limited resources available for data collection. While increasing the amount of quality labeled data can help with many challenges, finite budgets require balancing different tradcoffs. For example, there is an inherent tradcoff between the efficiency/uuality of feedback and the inclusion of long conversations in the feedback dataset. Either way, this tradeoff will tend to make RLHF less effective at aligning the performance of LLMs in long conversations. Helpful approaches for improving data quality have been to obtain samples that are diverse (Zhou et al.; 2023). adversarial (Ziegler et al., 2022), and which the reward model is uncertain about (Christiano et al., 2017). However, active learning techniques in deep learning rely on heuristics for prediction confidence which can be unreliable (Gleave and Irving 2022 Cost constraints will also push companies using RLHF to cut corners such by freely sourcing data from product users which can result in biased or even poisoned data (sce Section 3.1.1p. Defining the notion of data diversity, understanding its relationship with data efficiency, developing effective methods for diverse data selection, and understanding how the amount of feedback data influences outcomes are all open problems.
Fundamental:
There is an inherent cost/ httt//uality/qqaaaaee
3.1.4 Limitations of Feedback Types
Fundamental: RLHF suffers from a tradcoff between the richness and efficiency of feedback types. In addition to tradeoffs between cost, quality, and quantity that result from the overall process of feedback collection (see Section 3.1.3). similar tradeoffs result from the feedback modality. There exists interplay between the choice of feedback type and what biases human labelers express (Bansal et al. 2023). These choices influence the final policy learned after RLHF, but it is unclear what feedback protocol to use Below, we discuss challenges with the most
for evaluating the resulting differences (Bansal et al. 2023
prominent forms of feedback used in practice.
Comparison- feedback: The most common type of foedback used with RLHF binary preferences
between pairs of examples (Christiano al.) 2017) though k-wise rankings (Brown al 2019 2020 Zhu et al. 2023 Myers et al., 2021 Yuan et al. 2023 or best-of-k queries (Biyik et al. 2019) can be used the intensity of preferences. A learned preference ordering can fail to converge to the true one when the desirability of examples depends on noise or unmodeled, contextual details not contained in the observations (e.g., randomness in a human's 's feedback or differences between evaluators [Myers et al., ). Morcover, it is typically assumed that humans will select an option in proportion to the exponential of its utility (Christiano et al. 2017), but this is convenient modeling assumption, not a principled one. Another problem is that comparison-basee feedback will lead to policies that have a high median performance rather than high average one. Consider simple example in which actions of type A are always recognized to be of value 1 to evaluator, while actions type B are recognized to have value 10 on 40% of examples but are overlooked and concluded to have value 0 on 60%. Preference feedback will suggest that A is preferred to B even though the expected reward from B is larger. See also Section 3.2.1 for related challenges involving important information not contained in an example IG.
well. However, these methods do not offer precise information
Scalar feedback: Obtaining scalar feedback addresses some problems of comparison-based feedback it is significantly more expressive (Wilde et al., 2022). However, scalar rewards from humans can be poorly calibrated. It is often not clear for human annotators how to quantify the success of example, and it requires higher cognitive effort than simply comparing examples. Scalar feedback is more susceptible to inconsistency between annotators and suffers from bias due to the order in which examples are presented (Yannakakis and Hallam 2011). A combination of comparison and scalar feedback where the annotators indicated the intensity of a preference using a slider bar was demonstrated by Wilde et al. (2022), but it requires more sophisticated and annotator-spocific human response models. Attempting to discretize this form of feedback using a Likert scale (a range of discrete ratings; c.g., very bad, bad, ok, good, very good) simplifies the process of feedback collection {nox and Stone 2008 MacGlashan al. 2017 Arumugam

---

Published in Transactions on Machine Learning Research (12/2023)
et al. 2019: However, the resulting learned preference ranking can be the opposite of the true one when ossurmptions commonly made in practice are violated (Ethayarajh and Juralsky 2022). Bansal et al. (2023) also discovered an inconsistency problem between scalar and ranking feedback in which the preferences inferred from ratings and rankings significantly disagrec 60% for both human and AI annotators.
Sometimes, humans can provide feedback in the form of classifying examples. Label selection can be low-effort, but often suffers from choice set misspecification (Freedman et al. 2021]] Guerdan et al. 2023 Casper et al. 2023b/ when the given options don't fully encompass the labels needed to properly describe the data. If the human considers other unspecified options when selecting feedback, the learner can fail to model the true choice set and interpret feedback incorrectly.
Label feedback:
Correction feedback: Feedback can come in the form of corrective demonstrations or adjust that improve on an example from the model. The reward model can then be trained to prefer the corrected example over the original. In robotics, corroction-based foodback has been used for improving policies (Li eet al. 2021 Losey et al.) 2022; Bajesy el. al., 2018) and plans (Sharma et al. 2022). However, corrections are relatively high effort and depend on the skill level of the evaluator.
Language feedback: Using language, humans can convey large amount of information per evaluation, reducing ambiguity and goal misspecifiction. Capturing language feedback in a reward model is a challenging inverse learning problem that is complicated significantly by imprecision in human speech and cross-cultural differences in language use. A body of work on using language feedback for reward inference and shaping might lessen this challenge (Fu et al., 2019 Goyal et al. 2019 Sumers al.) 2021 Zhou and Small 2021; Lin et al., 2022: Yu et al. 2023), but thus far, these techniques have not been applied to L.L.s. The key drawback of approaches like these is that foedback is high-cffort, reducing the mumber of examples that can be evaluated. This increases the risk of failing to spot rare failure modes. See also Section 1.2 for a discussion of related methods that use human language feedback for training LLM policies without using a reward model (which excludes them from our definition of RLHF).
3.2 Challenges with the Reward Model
Here, we discuss challenges resulting from misspecification, misgeneralization, reward hacking, and evaluating the reward model, fs. Each involves instances in which it can be difficult to train good reward model, even from high-quality human leedback.
3.2.1 Problem Misspecification
The standard approach to fitting a reward model to represent human values is a moubll--isspecified problem.
Fundamental: An individual human's values are difficult to represent with a reward function. Unlike the model Equation 1 human feedback can depend on contextual factors that cannot easily be accounted for in the examples Tz=1...,f used to train the reward model f... Humans possess a range of intricate and context-dependent preferences that evolve over time and are difficult to model accurately. Models of human goals based on incorrect assumptions about human decision-making can impair reward inference (Hong et al.
2022). Even modeling human preferences with a reward at all, implicitly accepting the reward hypothesis Silver et al. 2021 might be unwarranted (Skalse and Abate 2022b Bowling et al. 2022 Bobu et al. 2023 number of studies have examined incorrect assumptions in various aspects of human models, such as their use of regret (Knox al.] (022),, the hypothesis space of reward models (Bobu al. 2020 Biyik al., 22200), and pedagogic behavior (Milli and Dragan 2020). Skalse and Abate (2022a) formally study the effect of inverse reinforcement learning with a misspecified Boltzmann model, which is also common (Jeon et al. 2020). Most work in RLHF does not take into account personality and context-dependdence of human preferences (Milano et al. 2021 Lindner and El- Assady 2022), and ZZhao el al. (2016) prove a mixture of reward functions cannot be identified from binary preferences without additional context. Different models for the human can also be better or worse for Icarnability (Knox et al.) 2022). In particular, modcling human irrationalitics can make reward lcarning difficult (Nguyen et al., 2017 Mindermann and Armstrong) 20188 Shah et al. 2019), leading to a trade-off between efficiency and accuracy. Finally, there are further challenges posed when feedback comes in different
2023 Vamplew et al.)
10

---

Published in Transactions on Machine Learning Research (12/2023)
modalities (e.g., demonstrations and preferences). Jeon et al. (2020) and Biyık et al. (2022) propose ways of combining different types of information about human goals, but these approaches are sensitive to modeling assumptions about the human.
Fundamental: A single reward function cannot represent a diverse society of humans. RLHF is typically formulated as a solution for aligning an AT system with a single human, but humans are highly diverse in their preferences, expertise, and capabilities (Bobu et al. 2023; Peng et al. 2023). Evaluators often
disagree: Stiennon et al. (2020), Ouyang et al. (2022). , and Bai et al. (2022a) report annotator-anmotate
and senotatto-researccee agreement rates from 63% to 77%, while Biyik and Sadigh (2018) find distinct
clusters of human feodback. Attempting to condense feedback from a variety of humans into a single reward model without taking these differences into account is thus a fundamentally misspecified problem. Moreover, current techniques model differences among evaluators as noise rather than potentially important sources of
disagreement (Baumler et al. 2023) (sce Equation ( As a result, when preferences differ, the majority
wins, potentially disadvantaging mnde--ropreeented groups (Prabhakaran et al.) 2021 Feffer et al. 2023 Kirk
et al. 2023
3.2.2 Reward Misgeneralization and Hacking
Reward models tend to be imperfect, and imperfection reward models leads to reward hacking.
Fundamental: Reward models can misgenoralize to be poor reward proxics, even from correctly-labeled training data. There can exist many ways to fit the human feedback dataset
D = [(., v)==1.....; even in the limit of infinite training data (Skalse et al. 2023). Reward models can compute reward using unexpected, possibly contingent features of the environment (Michaud et al. 2020 and are prone to cansal confusion and poor ont-of-distribution generalization (Tien et al. 2023). Reward learning algorithms can even produce reward models that fail to train new agents from scratch various settings, raising concerns about their reliability as signals for policy learning (MeKinney et al. 2023
Optimizing for an imperfect reward proxy leads to reward hacking. Reward models can differ from humans due to misspecification (Section 3.2.1 and misgeneralization (Section 3.2.2 as well as the inevitable failure of real-world machine lcarning systems to achieve minimal loss in complex problems. Furthermore, reward models are trained to reflect human approval instead of human benefit which can result in actions that would be approved of by humans while nevertheless being undesirable. Applying strong optimization pressure for an imperfoct proxy measure for a goal tends to cause poor performance on the underlying target goal (Hoskin 1996 Manheim and Garrabrant, 2018 Gao al. 2022). For example, without regularization penalizing the KT, divergence between base model and the finetuned model, LLMs undergoing RL often learn to output nonsensical text (Ziegler et al.) 2019 Stiennon et al. 2020). This type of problem is known as "reward hacking", and has been observed in AI systems, including those trained with RLHF (Skalse et al. 2022 Krakovna et al.] 2020). Skalse et al. (2022) show that unhackable proxies are very rare in complex environments, and Zhuang and Hadfield-Menell] (2020) prove under mild conditions that reward hacking should be expected by default. Using a suite of environments Pan et al. (2022) find that reward hacking also becomes more likely as an agent's raw capabilities increase.
Fundamental:
3.2.3 Evaluating Reward Models
Tractable: Evaluating reward models is difficult and expensive. When the true reward function is known, several methods can be used to judge the quality of the learned reward model (Gleave et al. 2020a; Wulfe et al. (022).. However, in most cases, reward modeling is used only when the true reward function is not known, making direct evaluation impossible. Hence, the reward model is typically evaluated in an indirect way by optimizing generations from the RL policy. This makes the reward model evaluation intricately dependent on the policy optimization process which is inherently expensive and noisy. It is also not clear how robust a reward model evaluation is to many ad-hoc choices made in the policy optimization process: e.g., choice of RL algorithm, policy network architecture, compute spent, and other various hyperparameter choices (Gao et al.] 2022). Another issue with indirect evaluation is that the evaluation signal for the reward model is the same as the training signal human approval. As result, training and evaluation failures will be correlated. Despite
RL policy using the learned reward model and then evaluating the
11

---

Published in Transactions on Machine Learning Research (12/2023)
the widespread use of indirect evaluation, it is not clear what choices in the policy optimization process are most influential for accurate evaluation of reward models.
3.3 Challenges with the Policy
Here, discuss challenges from policy optimization, misgeneralization, power-seeking, and mode collapse. Each involves instances in which the finctuned policy, Tonew can learn poor solution even when the fitted reward e accurately reflects human evaluations.
3.3.1 Robust Reinforcement Learning is Difficult
Safety in deployment requires robust performance, yet it remains challenging simply to train Al systems using RL.
Tractable: It is (still) challenging to optimize policies effectively. RL agents must interact with the environment to collect their own data. This requires balancing exploratory and exploitatory behavior (Amin let al. 2021; Yang et al. 2021). Balancing this tradeoff essential, but the degree of exploration required is difficult to determine and varies between environments. This is further complicated in settings with high-dimenssonal state/action spaces sparse rewards (Ding and Dong, 2020). Balancing exploration and exploitation in deep RL remains a fundamental yet open challenge (Amin et al. 2021 Yang et al.] 2021) Deep RL is unstable, and results are often highly sensitive to initialization and difficult to reproduce (Nikishin et al. 2018 Irpan 2018 Henderson et al. 2018p. This instability is attributed to multiple factors such as the random nature of exploration, the violation of the i..d assumption in data collection, the biased nature of value functions, and the general unpredictability of learning in deep neural networks (Amin et al., 2021 Ue-Cotina al. (2023) overview methods and limitations for RL with LLMs in particular.
Tractable: Policies tend to be adversarially exploitable. Even when learned policies are trained with perfect reward signal, perform well at the task they are trained for, and generalize to a wide range of scenarios, they can still perform poorly in adversarial situations. This is pressing concern, as models deployed into the real world can be adversarially attacked by humans or other Al systems. Even "superhuman" policies can fail catastrophically against policies specifically designed to exploit them (Gleave et al. 2020b Wu et al. 2021b
Wang al. 2022). Adversarial policics can be found either by re-purposing existing doep-reinnorcemen
learning algorithms or by manual human optimization in the case of coomppt-nnjettonns and jailbreaks CChoi
et al.
2022
Willison 2023: Albert 2023 Oneal 2023 Li et al. 2023b; Wolf et al.) 2023 Liu et al. 2023 Rao
et al.
2023 Wei et al. 2023 Shen et al. 2023 Shah et al. 2023) for ladgusge-models. Black-box access to
model (e.g., via API access) is sufficient for many adversarial policy attack algorithms, though white-box
access (enabled for example by open-sourced or leaked model weights) enables even stronger exploits (Kos
and Song. 2017
Casper ct al. 2022
3.3.2 Policy Misgeneralization
Fundamental: Policies can perform poorly in deployment. even if rewards seen during training were perfectly correct. The deployment distribution can always differ from the training and evaluation distributions in real-world settings (Christiano) 2019 Even with a correct reward signal, a policy can learn to competently pursue the wrong goal whenever the true goal correlated with other events. (Shah et al. Di Langosco et al. (2022) and Hilton et al. (2020) study this type of failure in-depth. Shah et al. 2022 present an example scenario in which systems trained with RLHF misgeneralizes to pursue the mechanism of reward administration itself instead of the intended goal.
2022
Fundamental: Optimal RL agents tend to seck power. RL agents have an incentive to scek power when possible to help them accomplish their goals Turner 2021 Turner el al. 2019 Turner and Tadepalli 2022 Ngo 2022 Krakovna and Kramar 2023 Ngo 2022). Power-seeking (e.g. seeking more influence or compute) is typically framed as concern with agentic AI systems that have more open-ended option spaces than typical language model. However, versions of power-seeking can emerge from the way that RLHF is typically used to finetune LLMss. For example, a question-answering LLM trained with RLHF has the
12

---

Published in Transactions on Machine Learning Research (12/2023)
incentive to learn to influence human interlocutors in order to avoid conversations about challenging topics. Sycophantic behavior from LLMs offers another example (Perez et al. 2022b Sharma et al. 2023).
3.3.3 Distributional Challenges
There are challenges posed by the distribution of outputs produced by the model both before and after training.
The pretrained policy introduces biases into policy optimization. RLHF in LLMs typically begins with base model that has been pretrained on internet text. This basc model is typically used both as the initialization for the RL policy network and the reference model for KL-regularization. Korbak et al. (2022b) formalizes how RL with these KI. penalties can be viewed as form of Bayesian inference with the base model determining the prior. While empirically useful, it causes the base model to significantly inffuence the final model. Using a base model that has been pretrained on web text is a convenient initialization not a principled one. Moreover, internet text encodes harmful biases (e.g., about human demographics); which are then inherited by the downstream model (Weidinger et al. 2021). These biases can persist through RLHF training process. For example, if sounding confident and producing correct answers are correlated in the base model, the reward model will learn that sounding confident is good and reinforce this in the policy.
Tractable:
Tractable: RL contributes to mode collapse. RL finetuning decreases the diversity of samples produced by a model (Khalifa et al.) 2021; Perez et al. 2022a Glaese et al.) 2022 Go et al. 2023 (a phenomenon known as "mode collapse") ). OpenAl (2023) found that RLHF finetuning of GPT-4 harmed its calibration on question-answering. Santurkar et al. (2023) found LLMs finetuned with RLHF expressed a narrow distribution of political views. Mode collapse is plausibly due in part to switching from the supervised pretraining objective to RL objective (Song et al.] 2023). RL incentivizes the policy to output high- scoring completions with high probability, rather than with a probability in line with a training distribution. Addressing this is complicated because mode collapse can be beneficial or harmful in different cases. For example, it is desirable if an LLM assistant is 90% sure the answer to a question is "yes", it is better for the LLM to answer "probably" 100% of the time rather than answering "yes" 90% of the time and "no" 10% of the time. On the other hand, some preferences are inherently distributional (Khalifa et al.] 2021 Weidinger et al. 2021) (e-g.; gender balance). We discuss techniques that can be used to fine-tune LLMs with a distribution-matchin objective in Scction 4.2.3
3.4 Challenges with Jointly Training the Reward Model and Policy
RLHF's depondence on training both a reward model and policy poses two unique problems.
Tractable: Joint training induces distribution shifts. Learning both a reward model and a policy is technically challenging the reward model influences the learned policy, and the policy determines the distribution of the data used to train the reward. On one hand, if the reward model is trained on offline data, it is likely to misgeneralize (Levine et al.) 2020 On the other hand, if reward and policy are learned jointly by gathering feedback from policy samples, the system will be prone to "auto-induced distributional shift" (Krueger et al. 2020 Li et al., (023d). Features with overestimated rewards will become gradually more present in the feedback data, and features with underestimated rewards will disappear. This slows convergence during training. Thus errors from the reward model can also accumulate and become difficult to correct with foedback once the policy stops generating diverse alternatives (Wu et al., 2021a).
Tractable: It is difficult to balance efficiency and avoiding overfitting by the policy. The three key steps of RLHF can be performed synchronously, but in practice with LLMs, they are often performed serially, In this case, the reward model will typically be inaccurate off-distribution, which is preciscly where the policy will learn to go (Gao et al. 2022: Levine et al., 2020). This is usually solved by oblaining fresh preference labels after a certain number of iterations of policy training. Appropriately setting this hyperparameter is important. Too low and information in the preference labels wasted; too high and the policy navigates to unreliable regions of the reward model (MeKinney et al., 20033 Christiano et al., 2017). Without a labeled validation set in the regions the policy is exploring, it is difficult to detect reward over-optimization during
13

---

Published in Transactions on Machine Learning Research (12/2023)
training. Helpful approaches might include measuring KL-shift (Gao et al. 2022 tracking the amount of disagreement in an ensemble of reward models.
4 Incorporating RLHF into a Broader Framework for Safer Al
Because of the challenges surveyed in Section & relying heavily on RLHF for developing safe AI poses risks. While RLHF is useful, it does not solve the fundamental challenges of developing human-aligned AT. More generally, no single strategy should be treated as a comprehensive solution. A better approach is defense in depth: multiple safety measures with uncorrelated failure modes. This is akin to assembling multiple layers of Swiss cheese- each has holes, but when layered can compensate for each other's failures (Hendrycks et al., 2021). While this type of approach is promising, it also comes with problems. For example, many of the challenges in Section 3 are not unique to RLHF, so it may be hard to find safety methods with uncorrelated failures. In this section, we discuss approaches that can be used to better understand (Section 4.1,, improve on (Section 4.2 and complement (Section 4.3 RLHF in various ways as part of broader agenda for AI safety.
4.1 Frameworks for Better Understanding RLHF
Although RLHF is becoming more widely used, there remain open questions about what factors are at play within it and how they influence the overall outcome. Here, we discuss approaches to address challenges for RLHF
Psychology and human-computer interaction. Many of the open questions with RLHF involve the dynamics play between humans and AI. It remains a challenge to understand the conditions which best allow for safe, reliable human-computer interaction. Specifically, it is unclear what type of feedback (or combination thereof) is best for learning human goals (Bansal et al.) 2023), precisely how biases harm the quality of feedback, and how to best select and train human evaluators. As discussed in Section 3 human
desires are difficult to express with a reward function (Skalse and Abate 2022b Bowling et al. 2023 Vamplew
et al. 2022 Further work may be valuable toward inferring what beliefs humans are operating under and either asking for feedback while taking into account human uncertainty (Biyik et al. 2019) or correcting for human biases (Reddy et al., 2019 2020 Chan et al.) 2019 Tian et al. (023). Reward modeling systems must
also take advantage of techniques that distinguish between humans with different levels of expertise (Daniels-
Koch and Freedman 2022 confidence (Zhang al. 2021 or noisiness (Freedman et al. 2023 Barnett
et al.] 2023).
Sociology and social choice. Al alignment must address not only individuals' perspectives, but also the norms, expectations, and values of affected groups. Some works have begun to assess whether LLMs can be used to facilitate agreement between different humans (Bakker el. al. (022) and to codify the broad- ranging principles under which deployment of Al systems for public good can be assessed (Floridi and
Cowls 2022 Sartori and Thcodorou| 2022). The majority-rule problem with RLHF can also be improved
by algorithms that explicitly model multiple evaluators (Freedman et al. 2023 Gordon et al. 2021 Davani
et al. 2022: Daniels-Kock and Freedman 2022 Gordon et al. 2022 Barnett et al. 2023). that tune models
to individuals (Kumar et al. 22021), or that use more sophisticated aggregation strategies (Noothigattu et al. 2018). However, none of these approaches can solve the fundamental problem of how an Al system cannot
be aligned multiple groups of humans who hold conflicting viewpoints (Dobbe et al. 2021 Mishra 2023 Many socicties, however, confront this fundamental issuc regularly. For example, democracies soek to reflect social preferences by soliciting the feedback of individuals. These systems generally fail to align diverse preferences yet tend to be more acceptable than less-democ alternatives. As such, it is important to analyze RLHF from the lens of social choice theory (Sen) 1986) and work to understand whether the means by which it aggregates preferences is normatively acceptable.
Assistance games. Assistance games, such as cooperative inverse RL (CIRL) Moadfield-Mendd et al. 2016), provide a framework to analyze algorithms like RLHF They offer a mathematical model to evaluate different design decisions in the communication of preferences to learning systems. In an assistance game, human and agent act together in the environment. Both seek to optimize the human's latent reward
14

---

Published in Transactions on Machine Learning Research (12/2023)
Addressing Challenges with RLHF, $4.2
Human Feedback $4.2.1
Reward Model, $4.2.2
Policy, 54.2.3
Al assistance
Direct human oversight
Aligning LLMs during pretraining
Fine-grained feedback
Multi-objective oversight
Supervised learning
Process supervision
Maintaining uncertainty
Translating language to reward
Learning from demonstrations
Figure 3: Strategies that can be used to address various problems with RLHF. Each approach is discussed in Section 4.2
function, while only the human can directly query this reward function. In this model, querying the human is simply an additional action that the robot can take, and it is possible to study different querying strategics or profiles. Studying RLHF as an assistance game emphasizes the performance of the hurnan-robot team. This might suggest alternative preference elicitation methods. Two examples are using active reward learning to determine when to collect feedback and which feedback to request first (Sadigh et al. 2017), and leveraging
dialogue models learn desired beedback-sockinng patterns (Krasheninnikov al., 2022) Of particular
interest is understanding the consistency and convergence properties of RLHF, the impact of different error patterns from raters, and the effect of different rates of feedback.
Bayesian inference. Finetuning an LLM using RL with KL penalties on the differences between the pretrained model can be understood as a form of Bayesian inference: conditioning prior (base LLM) on evidence about the desirable behavior of an LLM provided by the reward model (Korbak et al. 2022b This perspective on RLHF separales the modeling problem (defining larget distribution specilying the desired behavior of an LLM) and the inference problem (approximating that target distribution) (Korbak et al., 2022at Go et al.) 2023). This aid in answering questions about how the prior influences the outcome of RRHF.. The typical target distribution of RLHF (a Boltzmann distribution) is particular design choice and other distributions may address some of its limitations by, for example, differently fitting distributional preferences (Khalifa et al. 2021). Similarly, RLHF's inference algorithm (RL with KL penalties; equivalent to a variational inference approach (Korbak et al. 2022b)) could be replaced by particular sampling strategy (e.g., rejection sampling or best-of-n sampling).
Worst-case behavior. While RLHF seems to improve the average performance of system, not clear what effects it has worst-case behavior. It was not designed to make systems adversarially robust, and empirical vulnerabilities of systems trained with RLHF have been demonstrated with jailbreaks and prompt
injection attacks (Choi et al. 2022 Willison 2023 Albert 2023
Oncal 2023 Li ct al. 2023b Wolf et al.
2023 Liu et al. 2023 Rao et al. 2023 Wei el. al. 2023 Shen et al. 2023 Shah el al. 2023 As a consequence,
it would be valuable to better understand the case behaviors of RLHF systems, potentially through the
lenses of thcoretical propertics (Wolf al. 2023 El-Mhamdi et al. 2022), decision theory (Casper) 2020).
adversarial attacks (Perez al.
2022a b Casper et al.) [023b] Ziegler ler et al. 2022 Carlini et al.] 2023b),
rigorous evaluations (ARC) 2022
OpenA
2023 Shevlane et al. 2023
15

---

Published in Transactions on Machine Learning Research (12/2023)
4.2 Addressing Challenges with RLHF
Just as RLHF has challenges involving feedback (Section 3.1 0, the reward model (Section 3.2 and the policy (Section 3.3 there are various methods (both new and old) that can replace or supplement parts of the RL.F pipeline to address these types of challenges. Figure outlines these methods. See also Wang et al..
(2033) for a survey of methods for aligning LLMs.
4.2.1 Addressing Challenges with Human Feedback
Providing feedback with AI assistance. One way amplify the abilities of humans to have Al tools assist in generating feedback. Engineering prompts for an AI system and using it to automate feedback can substantially increase practicality and cost-effectiveness. due to reduced reliance on humans. honeth Al-generated feedback still fundamentally depends on humans because (1) the models providing feedback are trained on human-generater data, and (2) humans control prompts and the process of incorporating feedback.
There are several notable examples of Al-generated language feedback (Bai et al. 2022b Saunders et al.
2022
Ye et al.
2023
Kim et al. 2023 Akyürek et al., 2023 Madaan et al. 2023; Chen et al.) 2023 Gilardi
et al.
2023
Lee et al. 2023) with research agendas like Recursive Reward Modeling (Leike al. 2018) and
AI Safety via debate (Irving et al., 2018 Du et al.,
2023 However, Al-generated feedback has drawbacks.
Humans often disagree with Al feed The rate of human/Al disagreement will vary by task, but Perez
ct al. (022b), Casper et al. (023b), and Loe et al. (2023) found this to happen up to 10%, 46%, and 22% of the time respectively in different experiments. Machines can also exhibit correlated failure modes not found in humans, such as vulnerabilities to some adversarial attacks. The extent to which AI feedback is a viable way to safely augment human feedback remains uncertain. However, it cannot theoretically be a comprehensive solution to AI alignment due to the bootstrapping problem behind ensuring the feedback- prodbdcc-providing AI is aligned.
Fine-grained feedback. Many problems with feedback involve difficulty conveying precise information via the feedback signal (Section (31144. To address this, Wu et al. (2023) and Cabi et al. (2019) use feedback on specific portions of examples and Wu et al. (003)) use feedback with respect to different goals of the model (e.g., correctness, relevance). This might improve the quality of the learned reward models at the cost of human feedback being more expensive to provide. Fino-grained feedback is not yet well studied nor widely adopted, so additional work understand its advantages and feasibility will be valuable.
Process-based supervision. One challenge with training AI systems to solve problems is the difficulty of supervising performance on multi-step procedures. In RL, rewards can be very sparse for such problems. To address this, some works have trained LLMs to better solve multi-step math problems with process supervision (Uesato et al., 2022 Lightman et al. 2023). However, a fundamental challenge of both finc- grained and process- based supervision will be that the high-effort nature of the feedback trades off with the quantity that can be provided. This will make it more difficult to spot rare failure modes.
Translating natural language specifications into reward model. Many issucs with RLHF arise due to the difficulty of fitting reward function using some constrained type of feedback. An alternative approach can be to generate a reward signal more directly from natural language directions, bypassing the need for feedback on examples. This approach could resemble a technique used by [ai et al. (2022b) which involved using prompts to guide an AI assistant to identify responses that violated certain usee nser-defined specifications. Recently, Kang et al. (2023) demonstrate a zero-shot version of preference learning using LLMs,, and Lii et al. (2023a) introduced a method to clicit intended behavior through freo-form, language-based interaction with users. Moreover, Luketina al. (2019) surveys other possible techniques to accomplish this goal in non-LLM settings.
Learning rewards from demonstrations. An alternative approach to lcarning a reward model, known as inverse reinforcement learning (IRL) (Ng el al. 2000 Ramachandran and Amir 2007 Ziebart et al.) 2005). involves humans providing demonstrations instead of offering feedback on ones generated by the model. Jeon et al. (2020) and Biyik et al.] (2022) propose systematic ways of combining demonstrations, preferences, and possibly other types of human feedback to learn reward functions. While demonstrations carry rich information and avoid the need to have a system learn from its own generations, they are often more difficult
16

---

Published in Transactions on Machine Learning Research (12/2023)
to gather because they require higher effort and expertise to perform the task. Additionally, the quality of demonstrations is limited by the talent of whatever expert is providing them, which warrants more research on learning from suboptimal human demonstrations (e.g., Brown et al. (2019) Zhang et al. (2021)
4.2.2 Addressing Challenges with the Reward Model
Using direct human oversight. Although learning a reward model is efficient, it might be necessary to directly provide rewards (MacGlashan et al. 2017) for RL training in certain safety-critical situations.
Multi-objective oversight. Richer multi-objective signals that rate outputs on multiple objectives (Vam- plew al., 2022) could lead to more flexible oversight. Current reward models assumee that expert feedback
is drawn from an underlying unimodal reward function (Barnett et al.. 2023 Myers et al. 2021 But this is overly simplistic (Skalse and Abate 2022b Bowling et al. 2023). For instance, it can lead to reward model that merely captures the preferences of the majority, and suppresses the preferences of minoritios
noise.
Using constraints (Malik et al. 2021 Lindner et al. 2023 or reward models that account for the diversity of preferences by assuming underlying reward functions to be multimodal (Myers et al., 2021 Bakker et al. 2022 Barnett et al. 2023 Siddique et al., 2023 Bhatia al.] 2020) can help mitigate this issue. Multi-
objective oversight can also be useful for steering systems toward desired balancos between competing values
(e.g.; helpfulness and harmlessness).
Maintaining uncertainty over the learned reward function. Given the challenges of accurately learn- ing the appropriate reward function, several studios have emphasized the importance of taking uncertainty in the learned functions into account. Yue et. al. (2023) and Liang et al. (2022b) tackle this by having the policy avoid types of states unseen by the reward model. Using an ensemble of reward functions has also been used to address these challenges (Christiano et al., 2017), demoustrating that this approach can enhance the diversity of text output (Rame et al.) (023) and its applicability for active learning (Gleave and Trving 2022 Other strategies can include forms of risk-aversion (Hadfield-Mnnel et al. 2017 handling
uncertainty with safe "shield" policy (Jansen et al. 2018 Srinivasan et al. 2020 Cohen and Hutter 2020
4.2.3 Addressing Challenges with the Policy
Aligning LLMs during pretraining. RLHF in LLMs typically begins by pretraining the LLM internet text which includes a large amount of undesirable content. Korbak et al. (2023) argue that it can be more effective to use human feedback during pretraining by using a reward model to filter, weight, or annotate pretraining data. This also simplifies the process of aligning models by having them exhibit desirable behaviors from the outset rather than having them learn undesirable behavior and then atlempt to unlearn it during finetuning.
Aligning LLMs through supervised learning. Several techniques for aligning LLMs with human pref- erences obtain results competitive with RLHF by using supervised learning to complement (Ramamurthy et at.
2022 or replace RL. The simplest variant of this is to perform standard supervised learning on well-
curated data. Curation can involve filtering out bad demonstrations (Gehman et al. 2020 Welbl et al. |2021 Dong et al. 2023), compiling a small set of good demonstrations (Solaiman and Dennison, 2021 Sanh et al,
2022 Tbarz et al. 2018 Stiennon et al.] 2020 Chung et al. 2022 Biyik et al. 2022 Zhou et al. 2023), or generating good demonstrations using an LLM, e.g., after conditioning human feedback provided natural language (Scheurer et al. 2022
2023 Chen et al. 2023 Xu et al. 2023b). A different family of methods augments the language modeling objective to utilize feedback provided by the reward model (Korbak et al,
2023 Yuan et al.] 2023 Rafailov et al.] 2023). This last setting shares similaritios with offline RI, which
focuses on training an optimal policy using demonstrations annotated with rewards (Levine et al. 2020
Snell et al. 2022 Hu et al. 2023).
4.3 RLHF is Not All You Need: Complementary Strategies for Safety
Other technical approaches to AI safety should be studied and implemented alongside RLHF Establishing trust with AI systems should be approached with combination of principled design choices, rigorous testing, interpretability, verification, and theoretical guarantees where possible (Leike al. 2018 See also Critch
17

---

Published in Transactions on Machine Learning Research (12/2023)
and Krueger (2020). Hubinger 2020 Hendrycks et al. (2021): and Ngo 2022 for additional overviews of strategies for building safer AT.
Robustness. As discussed in Section 3.3 models trained with RLHF can still exhibit undesired behavior due to distributional shifts between training and deployment. For example, adversarially engineered user inputs cause an LLM to output harmful text. To mitigate this problem, developers should use tools to generate inputs which result in undesired behavior and train against these adversarial examples (Zhang and Li 2019 Ziegler et al., 2022 Perez et al., 2022a| Casper et al.] 2023b). Anomaly detection techniques (Omar et al. 2013) can also be useful for flagging abnormal imputs likely to trigger bad behavior. Ensuring the security of important AI training runs against malicious human evaluators and/or outside cybersecurity threats will also be valuable.
Risk assessment and auditing. Although training processes should be crafted to produce models that are safe by design, evaluations are another layer of defense. Passing an evaluation is not proof of safety, but as is the case in almost every safety-critical industry, rigorous evaluations of capabilities and risks helps to spot hazards and establish trust. In practice, this should involve both in- in-house and serooddppatty evaluations
(OpenAl 2023 ARC 2022 Perez et al. 2022b). As with adversarial training for robustness, the development
of improved red teaming techniques will be important (Perez et al. 2022a Casper et al. 2023b).
Interpretability and model editing. Generating human-understandee explanatios for the behavior of Al systems is currently an unsolved problem. Progress in explainability and interpretability could help verify hypotheses about how models make decisions (Geiger et al. 2023), including whether the process is trustworthy. In this way, it could be possible to gain confidence that models will (or will not) behave in safe way without necessarily conducting extensive testing of the models (Jacovi ct al.) 2021 Red-teaming can also be complemented by interpretability techniques (Rastogi et al.] 2023: Räuker et al. 2023), especially for purposes of identifying adversarial inputs (Ziegler et al. 2022] Casper et al. 2023c a or anomalous inputs (Pang et al. 2021]. In another direction, better understanding the internal mechanisms of models can aid in directly editing model weights or intervening on internal activations in order to improve
truthfulness (Li et al. 2023e), modify model's 's factual knowledge (Meng et al. 2023 2022 Hernandex, al.
2023
Hase et al. 2023), or otherwise steer model behavior (Cui et al. 2022).
5 Governance and Transparency
Social scientists and policymakers have increasingly focused on the need for governance frameworks to develop and deploy AI systems responsibly. Across historical contexts, hallmark of mature scientific fields is the open sharing of research findings (Shapin and Schaffer 2011) to allow experts to understand progress (Gilbert and Loveridge 2021). Below we overview components of an RLHF governance agenda, including outstanding questions and risk dimensions.
Incentives and requirements for safety. Competition between labs can generate harmful race dynamics (Dafoc] 2018) because of tradcoffs between competitiveness and caution. This suggests role for governance in promoting healthier environment for safe AI research, development, and deployment (Dafoe] 2018
Perry and Uuk 2019 Falco et al.] 2021 Cihon 2019: Anderljung et al. 2023). Governance in this form
could involve mandates for independent auditing, evaluations, and certification (Shavit) 2023 Mökander et al. 2023 ARC, 2022; Hadfield and Clark, 2023, Shevlane et al., 2023): ; monitoring for pest-deployment
problems (Hendrycks and Gimpel 2016);; influence over resources including hardware and data (Brief 2020:
Chan et al.] 2022a): and prohibiting deployment unless critical standards are met, as in the case of the U.S. Food and Drug Administration's oversight of clinical trials for testing potential new treatments (Junod
2008
Transparency and auditing. A sustained commitment to transparency would make the existing RLHF research environment more robust, from a safely standpoint. First, the disclosure of some details behind large RLHF training runs would clarify a given organization's 's norms for model scrutiny and safety checks. Sccond, increased transparency about known efforts to mitigate risks could improve safety incentives and suggest methods for external stakcholders to hold companies accountable. Third, and most relevant for the present paper, transparency would improve the AT safety community's understanding of RLHF and
18

---

Published in Transactions on Machine Learning Research (12/2023)
Transparency / Auditing
Items for RLHF
Human Feedback
Reward Model
Policy
Pretraining
Loss function
Evaluation and results
Selectionttraining of humans
Evaluation and results
Selection of examples
Systemic Safety
Type(s) of feedback used
Internal and external auditing
Quality- assurance measures
Report expected risks
Monitoring and handling failures
Figure 4: Details behind an implementation of RLHF that, if disclosed, could be indicative of risks. See Section
5 for a complete discussion. Companies using RLHF to train models for high-stakes or safety-critical applications should mainlain transparency with the public and/or auditors about key details of their approach.
support the ability to track technical progress on its challenges Clark 2022 Some level of disclosure is precondition to evaluate the viability of the technical RLHF safety agenda over time and allow for community contribution to it. For all of these reasons, working to incorporate transparency standards into an Al governance framework will be important (Larsson and Heintz
2020 Anderljung el. al. (023)..
It possible that public disclosure details critical to the development model capabilities might lead the unwanted proliferation of AI tochnologies that could be misusod. However, detailing safety measures will often not require divulging implementable details, and when it does, private disclosure to second-party
anditors Mökander al. 2023
ARC 2022 Hadfield and Clark 2023 Shevlane al. 2023) offers solution.
As more specific policy prescriptions are beyond our scope, we encourage claboration on these topics as part of future research agenda. Below, however, we outline specific types of details that, if disclosed, could be indicative of risks and should be accounted for when auditing AT systems developed using RLHF. See also Figure
Human feedback details:
A description of the pretraining process including details about what data was used to make apparent possible biases that pretraining can cause.
How human evaluators were selected and trained to provide information about risks of evaluators being malicious, unrepresentative, or incapable.
The process by which examples were selected to obtain feedback to invite scrutiny about their representativeness and whether sufficient adversarial training was used. If examples were crowdsourced from a publicly-available application, details about what measures were taken to avoid data poisoning attacks should be provided.
The type(s) of human feedback used (e.g., binary comparisons, scalar feedback, etc.) to suggest what risks might be caused by insufficiently abundant or rich feedback.
A report on measures taken for quality assurance in feedback collection and inter-rater consistency to ensure that effective quality control measures were taken.
Reward model details:
19

---

Published in Transactions on Machine Learning Research (12/2023)
The loss function used to fit the reward model and how disagreement was modeled (e.g.. as noise) to help with analyzing the degree of misspecification when fitting the reward model.
A report on reward model evaluation and results to suggest possible problems from mis- aligned reward model. The evaluation should involve red teaming.
Policy details:
A report on policy evaluation and results to suggest possible troubles from a misaligned policy, The evaluation should involve red teaming and include assessment for risky capabilities (e.g., the ability to deceive a human).
Systemic safety measures
A report internal and external audits and red tcaming ensure accountability and disclose risks that are identified.
A report on expected risks and anticipated failure modes to ensure accountability.
Plans for monitoring and correcting failures that emerge to support post-deployment safety.
How these types of risks should be documented remains an area of active work in AI governance, Similar questions have been asked in investigation by the US Federal Trade Commission into OpenAl (FTC) 2023) but in response to problems with ChatGPT rather than proactively. Salient documentation proposals focus on regular reporting of reward components (Gilbert et al. 2022) and the ability to compare the capabilities of language models according to standard benchmarks (Liang et al.) 2022a). For the longer term, incorporating beneficial standards for safcty and transparency into norms and regulations affecting AI is an ongoing challenge.
Concerns for social and economic equity. Although this paper has focused on technical challenges with RLHF, there social and cconomic ones as well which governance and industry should work address. For example, OpenAl has paid Kenyan knowledge workers al a rate of less than $2 USD per hour (Perrigo 2023) for work which was mentally and emotionally distressing and had negative impacts on their mental wellbeing (Hao 2023). Human subjects used in RLHF research should not be systematically selected simply for their availability or low cost (National Commission for the Protection of Human Subjects 1978). Costs, benefits, and influence over RLHF models should be equitably distributed across different. communities (Whittlestone) let al. 2021 Eloundou al. (202)). There is an additional possibility that powerful Al systems will be highly profitable and serve concentrate large amounts of wealth and power into the hands of few (O'Keefe al. 2020 Chan et al., 2023b). Thus, policies that address inequalities and protect vulnerable populations (e.g. impacted communities, whistleblowers) will be incrcasingly important.
6 Discussion
While some problems with RLHF are tractable, others are fundamental and cannot be fully solved. Technical progress in some respects is tractable, and this room for progress should be seen as a cause for concerted work and optimism. Even some of the fundamental problems that we overview can be alleviated with improved methodology even though they cannot be fully solved by RLHF However, the fundamental nature of these problems requires that they be avoided or compensated for with non-RLHF approaches. Hence, emphasize the importance of two strategies: (1) evaluating technical progress in light of the fundamental limitations of RLHF and other methods, and (2) addressing the sociotechnical challenges of aligning to human values by committing to both defense-in-depth safety measures and openly sharing research findings with the wider scientific community.
RLHF Rehashing Lessons from Historical Failures? RLHF offers new capabilitics but faces many old problems. The individual components it it (preference elicitation, fitting a reward model, and policy opti- mization) have history of technical and fundamental challenges in the fields of human-computer interaction
20

---

Published in Transactions on Machine Learning Research (12/2023)
and AI safety (e.g. Lewis 1990; Sutton and Barto 1999 Ng et al. 1999 In 2023, RLHF was described by the first author of Christiano et al. (017)) as "basic solution" intended to make it easier to "produe- tively work on more challenging aligmment problems" " (Christiano 2023 Some challenges and questions that we have covered are rather unique to RLHF such as oncs involving jointly training the reward model and policy (Section 3.4p. However, many other problems are instances of broader ones machine learning such as challenges with RL policies (Section 3.3 Others still are fundamental problems with AT alignment such determining whose values are encoded into Al in diverse society of humans (Section 3.2.1p. p. The successes of RLHF should not obfuscate its limitations gaps between the framework under which it is studied and real-world applications (see Appendix Aj. An approach to AT alignment that relies on RLHF without additional techniques for safety risks doubling- on flawed approaches to Al alignment. Thus, it will be important to continue working to better understand RLHF while respecting its limitations.
Moving forward. RLHF has clear advantages for aligning Al systems with human goals. As a result, it has been key to the development of stata-of-the-art LLMs and will likely continue play a major role in modern AT. However, its use and influence should be accompanied by a commensurate research effort to better understand RLHF and address its flaws. Because it optimizes for human approval, RLHF in particular demands a special type of caution because many of its failures will actively tend to be ones that humans struggle to notice. It will be important to approach RLHF cantiously and work to incorporate it into a more holistic framework (Khlaaf 2023 for safer AT with multiple layers of protection from failures (Hendrycks) et al.) 2021p. Because some of the challenges with RLHF are fundamental to the Al alignment problem itself, moving forward will require confronting the basic choices and assumptions behind any given approach to aligning AT and who controls it (Dobbe al.) 2021). Moving forward, urge that those working to develop advanced LLMs using RLHF both contribute toward resolving its open challenges and maintain transparency about the details of their approach to safety and any anticipated risks.
Contributions
Stephen Casper and Xander Davies served as the central writers and organizers.
Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadcan, Eric Michaud, Jacob Pfan, Xin Chen, Dmitrii Krasheninnikov, Lauro Langosco, and Peter Hase contributed planing and writing the paper.
Erdem Biyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Boaffied-MMeeell served as advisors.
Acknowledgements
We thank Sam Bowman, Adam Jermyn, Ethan Perez, Alan Chan, Gabriel Recchia, Robert Kirk, and Nathan Lambert for their helpful feedback. This work was facilitated in part by the Harvard Al Safety Team and MIT AI Alignment group.
Christiano 2023 mentions debate Irving al. 2018 and recursive reward modeling [Likke
2018 as examples of
challenging alignment problems.
an oubline of proposals in Hubinger (2020).
21

---

Published in Transactions on Machine Learning Research (12/2023)
References
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning- based reinforcement learning. In Machine Learning and Knowledge Discovery in Dalabases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pages 116-131. Springer, 2012.
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arxiv preprint orXic:2305.000 08844, 2023.
Alex Albert. Jailbreak chat. 2023. URL httpp:///ww/ http:::/www.jaalbrreeeteea htwp:/wwww..bbbaaaaaaaa.a.a...
Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning. arXiv preprint arXiv:2109.00 00157, 2021.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ar preprint orXiv:1606 (eee166066666, 2016.
Markus Anderljung, Joslyn Barnhart, Jade Leung, Anton Korinek, Cullen O'Keefe Jess Whittlestone, Sha- har Avin, Miles Brundage, Justin Bullock, Duncan Cass-Boggs, Ben Chang, Tantum Collins, Tim Fist, Gillian Hadfield, Alan Hayes, Lewis Ho, Sara Hooker, Eric Horvitz, Noam Kolt, Jonas Schuett, Yonadav Shavit, Divya Siddarth, Robert Trager, and Kevin Wolf. Frontier ai regulation: Managing emerging risks public safety, 2023.
Anthropic. Introducing claude, 2023. URL http:///sss/....aaaaa...///////r
ARC. Arc evals, 2022. URL https://evala: http://eeaall.alggmmmttt. org/
Dilip Arumugam, Jun Ki Lec, Sophic Saskin, and Michael L Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprinl or Xrxiv:1908 04257, 2019.
Hui Bai. Artificial Intelligence Can Persuade Humans. 2023.
Yunlao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement lcarning from human foedback. arXiv preprint 2xXi:2224.00866 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint Xrx::22112.0888 08073, 2022b.
Andrea Bajesy, Dylan Losey, Marcia K O'Malley, and Anca Dragan. Learning from physical human corrections, one fcature at time. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pages 141-149, 2018
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbel--Gillingann, Jan Bal- aguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-3811 2022.
Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2908.115: 15812, 2023.
Peter Barnett, Rachel Freedman, Justin Svegliato, and Stuart Russell. Active reward learning from multiple teachers. arXiv preprint arXiv:2303.008 2023.
Connor Baumler, Anna Sotnikova, and Hal Daumé III. Which examples should be multiply annotated? active learning when annotators may disagree. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10352-1000 10371, 2023.
22

---

Published in Transactions on Machine Learning Research (12/2023)
Kush Bhatia, Ashwin Pananjady, Peter Bartlett, Anca Dragan, and Martin J Wainwright. Preference learning along multiple criteria: A game-theoretic perspective. Advances in neural information processing systems, 33:7413 7424, 2020.
Erdem Biyik. Learning Preferences For Interactive Autonomy. PhD thesis, EE Department, Stanford University, 2022.
Erdem Biyik and Dorsa Sadigh. Batch active preference- based learning of reward functions. In Conference on robot learning, pages 519-528. PMLR, 2018.
Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL), 2019.
Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active preference- based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
doi: 10.15607/rss rss 2020.xvi.41...
Erdem Bıyık, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, (1(1):45-66 67, 2022.
Andreea Bobu, Andrea Bajesy, Jaime F Fisac, Sampada Deglurkar, and Anca Dragan. Quantifying hypothesis space misspecification in learning from human-robot demonstrations and physical corrections. 2020.
IEEE Transoctions on Robotics,
Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie Shah, and Anca D Dragan. Aligning robot and human representations. arXiv preprint arXiv:2302.0192: 019288 2023.
Michael Bowling, John D Martin, David Abel, and Will Dabney. Scttling the reward hypothesis. In Inter- national Conference on Machine Learning, pages 3003 3020. PMLR, 2023.
Samuel R. Bowman, Jecyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilé Lukoštūté, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christo- pher Olah, Danicla Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleon- Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatficl--Dodds, Ben Mann, and Jared Kaplan. Measuring Progress Scalable Oversight for Large Language Models, November 2022. URL http://arxiv org/abs
03540 arXiv:2211.038. [es].
CSET Policy Brief. Why chips matter. 2020.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on ma- chine learning, pages 783-792. PMLR, 2019.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pages 1165 1177. PMLR, 2020.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint Xiv:1990 12200, 2019.
Nicholas Carlini, Matthew Jagiclski, Christopher Choquette-Chog Choo, Daniel Palcka, Will Pcarce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. XrXiv prepriat ar arXiv:2302.1011 2023a.
23

---

Published in Transactions on Machine Learning Research (12/2023)
Nicholas Carlini, Milad NasT, Christopher A Choquette-Choo. Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint (xXi::2306.1544" 2023b.
Micah Carroll, Alan Chan, Henry Ashton, and David Krueger. Characterizing Manipulation from AT Sys- tems, March 2023. URL httt:://arxii.org/abbaa
09387 arXiv:2303.09387 [cs].
Micah D Carroll, Anca Dragan, Stuart Russell, and Dylan Hadfield-Menell. Estimating and penalizing induced preference shifts in recommender systems. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Stephen Casper. Achilles heels for agi/ast via decision theoretic adversaries. arXiv preprint orXiv:2010.05415 2020.
Stephen Casper, Dylan Haddield-Menell, and Gabriel Kreiman. White-box adversarial policies in deep rein- forcement learning. arXiv preprint arXiv:2209.0210 2022.
Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, and Dylan Hadeleld- Benchmarking interpretability tools for deep neural networks. arXiv preprint arXiv:2302.10894, 2023a.
Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.094/ 04442440,,44,, 2023b.
Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools, 2023c.
Christopher P Chambers and Federico Echenique. Revealed preference theory, volume 56. Cambridge Uni- versity Press, 2016.
Alan Chan, Herbie Bradley, and Nitarshan Rajkumar, Reclaiming the digital commons: A public data trust for training data. arXiv preprinl or urXiv:2303.0900 2023a.
Alan Chan, Rebecca. Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco di Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Togan Maharaj. Harms from incrcasingly agentic algorithmic systems. ArXiv, abs/2302. 10329, 2023b.
Lawrence Chan, Dylan Hadficld-Menell, Siddhartha Srinivasa, and Anca Dragan. The Assistive Multi-Armed Bandit. 19110:1901.08644 [es, stat], January 2019. URL 1901.08654.
wttpe//wwwwr//ba//11.066665 08654
arXiv:
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXio preprint Xrc::22303.111 16749, 2023.
Michael Chmielewski and Sarah C Kucker. An mturk crisis? shifts in data quality and the impact study results. Social Psychological and Personality Science, 11(4):464-473, 2020.
Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt injection: Parameterization of fixed inputs. arXiv preprint arXi::2206.113; 11849, 2022.
Paul Christiano.
Worst-case guarantees.
https://ai--lligmeett. comm
traabiggoobuf-rrrrgggllytooooood 2019.
Paul Christiano. Thoughts on the impact of rIhf research, Jan 2023. URL URI/h:tp::://r.aaagnnntttooo
ppt.ppa.......aaaaaaaaaaaaaaaaaa 20solutions.
24

---

Published in Transactions on Machine Learning Research (12/2023)
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment. learning from human preferences. Advances in neural information processing systems, 30, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Maric Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- -innttin-- language models, 2022. URL https: // /ttp///axvvvv../ .rgg hrrg/abs/2110 11416
Peter Cihon. Standards for ai governance: international standards to enable global coordination in ai research & development. Future of Humanity Institute. University of Oxford, 2019.
Jack Clark. Information markets and ai development. 2022.
Michael K Cohen and Marcus Hutter. Curiosity killed the cat and the asymptotically optimal agent. arXiv preprinl Xiv: 03357, 2020.
Ajeya Cotra. Why ai alignment could be hard with modern deep learning. https://ww.cold-takkss... com/
www.amiaalgnnent---em cowpggnnnee-couuddedeeeeeeeeeeee hardeeaaarwwhhaaeaeaeeeee -eter-mooeeddeeeeeeeeeeee -earning/ 2021.
Andrew Critch and David Krueger. Ai research considerations for human existential safety (arches). arxiv preprint arXt::2006.0440 04948, 2020.
Audrey Cui, Ali Jahanian, Agata Lapedriza, Antonio Torralba, Shahin Mahdizadehaghdam, Rohit Kumar, and David Bau. Local relighting of real scenes, 2022.
Allan Dafoe. Ai governance: research agenda. Governance of AI Program, Future of Humanity Institute, University of Onford: Orford, UK, 1442:1443, 2018.
Oliver Daniels-Koch and Rachel Freedman. The expertise problem: Learning from specialized feedback.
arXiv preprint arXio:2211006119 2022.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguislics, 10:92- 110, 2022.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal mis- generalization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004-12019. PMLR, 2022.
Zihan Ding and Hao Dong, Challenges of reinforcement learning. Deep Reinforcement Learning: Fundamen- tals, Research and Applications, pages 249-272, 2020.
Roel Dobbo, Thomas Krendl Gilbert, and Yonatan Mintz. Hard choices in artificial intelligence. Artificial Intelligence, 300:103555, 2021.
Hanze Dong, Wei Xiong, Decpanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model aligument. arXiv preprint 2XXw:2334.06760 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 140090551434... 14325, 2023.
El-Mahdi El-Mhamdi, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Lê-Nguyèn Hoang, Rafael Pinot, and John Stephan. Sok: On the impossible security of very large foundation models. arXiv preprint arXiv:2209.1500 15259, 2022.
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models, 2023.
25

---

Published in Transactions on Machine Learning Research (12/2023)
Kawin Ethayarajh and Dan Jurafsky. The authenticity gap in human evaluation, 2022.
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective. arXiv:1908.00 0.008.0.777 04734 fes). March 2021. URL http::/arxiv..e argg/bs//900///// (bbb//990.00744
arXiv: 1908.04734.
Gregory Falco, Ben Shneiderman, Julia Badger, Ryan Carrier, Anton Dahbura, David Danks, Martin Eling, Alwyn Goodloe, Jerry Gupta, Christopher Hart, et al. Governing ai safety through independent andits. Nature Machine Intelligence, 3(7):566-577) 571, 2021.
Michael Feffer, Hoda Heidari, and Zachary C Lipton. Moral machine or tyranny of the majority? arXiv preprint arXio:2305. 17319, 2023.
Luciano Floridi and Josh Cowls. A unified framework of five principles for in society. Machine learning and the cily: Applications in architeclure and whon design, pages 535 545, 2022.
Rachel Freedman, Rohin Shah, and Anca. Dragan. Choice set misspccification in reward inference. arXiv preprint arXtt:2111.07691 2021.
Rachel Freedman, Justin Svegliato, Kyle Wray, and Stuart Russell. Active teacher selection for reinforcement learning from human feedback. arXiv preprint arXi::2310.1528 2023.
Aaron French. The mandela effect and new memory. Correspondencee, 6(2), 2019.
FTC. 'federal trade commission civil investigative demand schedule ftc file no. 232-3044', July 2023. pdff?
URL https: //wp:: http::://wwwwwwsahnggoooono. wwshingtonpost.
itid=1k inline Lnninn_maauall________aaaa 4
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse
reinforcement learning for vision-basod instruction following. arXi preprint arXiv:100200774,, 2019.
Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforce- ment learning: formal framework and a policy iteration algorithm. Machine learning, 89:123 156, 2012.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.11 10760, 2022.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicctyyrompps: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 3356-3369, Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18653/v1/ 20055//3v1220.ffm/// 10000..iddingg-eml..3000 URL https://aclantthooog..oort org//220..
modding--emnl..33
Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. arXiv 04709.
preprinl ar XXi::2301.04709 2023. URL httphttps::: htttts:///////tt//
Fabrizio Gilardi, Mcysam Alizadch, and Maël Kubli. Chatgpt outperforms growd-wokken for toxt-annotation tasks. arXiv preprint orXiv:2303.15005 2023.
Thomas Krendl Gilbert and Andrew Loveridge. Subjectifying objectivity: Delineating tastes in theoretical quantum gravity rescarch. Social Studics of Science, 51(1):78-99) 09977799999999 2021.
Thomas Krendl Gilbert, Nathan Lambert, Sarah Dean, Tom Zick, and Aaron Snoswell. Reward reports for reinforcement learning. arXiv preprint ar Xiv:2200 10817, 2022.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phocbe Thacker, Lucy Camppell-Gillingaan Jonathan Uesato, Po- Sen Huang, Ranona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaune Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Folcy, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukeuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, 2022.
26

---

Published in Transactions on Machine Learning Research (12/2023)
Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. arXiv preprint arXiv:22030774 2022.
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. arXiv preprint arXiv:2006. 13900, 2020a.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020b. URL
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization, 2023.
Google. Bard, 2023. URL
Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael Bernstein. The disagree- ment deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 14, 2021.
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHT Conference on Human Factors in Computing Systems, pages 1-19, 2022.
Prasoon Goyal, Scott Niekum, and Raymond J Mooney, Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1900 02020, 2019.
Lewis D. Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly T. Mai, Maria Vau, Matthew Caldwell, and Augustine Marvor-Parker Susceptibility Influence of Large Language Models, March 2023. URL http://arxiv.. org/ abs/ 0abbs/2330 220//3333.0000 06074 urXiv:2303.0607 [cs].
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Pol- icy shaping: Integrating human feedback with reinforcement learning. Advances in neural information processing systems, 26, 2013.
Luke Guerdan, Amanda Coston, Zhiwei Steven Wu, and Kenneth Holstein. Ground (less) truth: A causal framework for proxy labels in human-algorithm decision- makingg Xixiv preprint arXiv:2302.06503 2023.
Gillian K Hadfield and Jack Clark. Regulatory markets: The future of ai governance. ar preprint urXiv:2304.0491 2023.
Dylan Madeilld-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
Dylan HedeelldMeenlll, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in newral information processing systems, 30, 2017.
Karen Hao. The hidden workforce out of chatgpt, 2023.
that helped filter
violence and abuse
URL https://www.wsj.. http::/www../o/ooccaaaa///eeoooe
the
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political idcology of conversational ai: Converging evidence on chatgpt's pro-envirommental, left-libertarian orientation. arx preprinl arXi::2301.01768, 2023.
Peter Hase, Mohit Bansal, Boen Kim, and Asma Ghandcharioun. Does localization inform oditing? surpris- ing differences in causality-based localization vs. knowledge editing in language models. arXio preprint urXiv:2301.04213 2023. URL https::/ http://awwwv..oo//////00...000
27

---

Published in Transactions on Machine Learning Research (12/2023)
Joey Hejna and Dorsa Sadigh. Few-shot preference learning for humen-in-the-loop rl. In Proceedings of the 6th. Conference on Robot Leurning (CoRL.), 2022.
Peler Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Dan Hendrycks and Kevin Gimpel. A bascline for detocting misclassified and out-of-distribution examples in neural networks. arxiv preprint X6w:16600.02166, 2016.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint ar Xrx:22119. 13916, 2021.
Evan Hernandez, Belinda Li, and Jacob Andreas. Measuring and manipulating knowledge representations in language models. XrXiv preprint arXiv:2304.00744, 2023.
Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah, Understanding rl vision. Distill, 5(11))c29, 2020.
Jocy Hong, Kush Bhatia, and Anca Dragan. On the sensitivity of reward inference to misspecified human models. Xiv preprint arXiv:2212.04717, 2022.
Keith Hoskin. The 'awful idea of accountability': inscribing people into the measurement of objects. Ac- countabiliy:: Power, ethos and the technologies of managing, 265, 1996.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human leedback is not gold standard, 2023.
Jian Hu, Tao, June Yang, and Chandler Zhou. Aliguing language models with offline reinforcement learning from human feedback. arXiv preprint Xrx::2200.. 12050, 2023.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.
Evan Hubinger. An overview of 11 proposals for building safe advanced ai. ar preprint arXiv:2012.0753. 2020.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Alex Trpan. Deep reinforcement. learning doesn't work yet. rl-hard. 2018
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. ar.iev preprint 1rXiv:1805.00899 2018.
Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM conference on fairness, accountability, and transporency, pages 624 635, 2021. URL https:///axiii..... wwwpaawwww...o../oooooooo wwtpp///////e///////////e//eeeg/
/200000000........1 pdf
Nils Jansen, Bettina Könighofer, Sebastian Junges, Alexandru C Serban, and Roderick Bloem. Safe rein- forcement learning via probabilistic shields. arXie preprint arXiv:1807.0609. 2018.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415-4426, 2020.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-33 -3, 2023.
28

---

Published in Transactions on Machine Learning Research (12/2023)
Suzanne Junod. Fda and clinical drug trials: short history. FDLI Update, page 55, 2008.
Inwon Kang, Sikai Ruan, Tyler Ho, Jui-Chien Lin, Farhad Mohsin, Oshani Seneviratne, and Lirong Xia. Lim-augmented preference learning from natural language, 2023.
Zachary Kenton, Tom Everitt, Laura Weidinger, lason Gabriel, Vladimir Mikulik, and Geoffrey Irving.
Alignment of Language Agents, March 2021. URL http:/tppp/aariv.or//ab///////// 14659 arXiv:2103.14659
[cs].
Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. distributional approach to controlled text generation. In International Conference on Learning Representations, 2021. URL https://openreriesi net/forumiid JwwwsssssdwWWWW9AA
Heidy Khlaaf. Toward comprehensive risk assessments and assurance of ai-based systems. Troil of Bits, 2023.
Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Sco. Aligning large language models through synthetic feedback. arXiv preprint 0rXiv:2305.1373: 2023.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint Xix:329003 05453, 2023.
W Bradley Knox and Peter Stone. Tamer: Training agent manually via evaluative reinforcement. In 2008
7th IEEE international conference on development and learning, pages 292-297. IEEE, 2008.
Bradley Knox and Peter Stone, Combining manual feedback with subsequent mdp reward signals for reinforcement learning. In AAMAS, pages 5-12, 2010.
W Bradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In AAMAS, volume 1004, pages 475-482. Valencia, 2012.
Bradley Knox, Stephane Hatgis Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro A Allievi. Models of human preference for learning reward functions. arXiv preprint Xrx:222006.08 02231, 2022.
Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forget- ting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Newral Information Processing Systems, volume 35, pages 16203-16220. Curran As- sociates, Inc., 2022a. URL httrps://prooeedings. neurips www.neeu....ppppeeeeeeeeee
neuring....euussss. wwtpppppee.llleeepppeeeeeeeeeeee pfles//paprr/0022//1ee
pdf
Tomasz Korbak, Ethan Perez, and Christopher Buckley, RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. URL
https://aclantholl. wwwpoollogyooggg//// wwg//022/2222.....// findings-emnlp... 77
Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences, 2023. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang,
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXio preprinl NSkk:1705.045555 2017.
Victoria Krakovna and Janos Kramar. Power-seeking can be probable and predictive for trained agents. arXiv preprint xxX:::3344.06528, 2023.
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingemuity. DeepMind Blog. 2020.
29

---

Published in Transactions on Machine Learning Research (12/2023)
Dmitrii Krasheninnikov, Egor Krasheninnikov, and David Krueger. Assistance with large language models. In NeurIPS MT. Safely Workshop, 2022.
David Krueger, Tegan Maharaj, and Jan Leike. Hidden incentives for auto-induced distributional shift, 2020.
Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursslein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. Designing toxic content classification for a diversity of perspectives. In SOUPS@ USENIX Security Symposium, pages 299-318, 2021.
Stefan Larsson and Fredrik Heintz. Transparency in artificial intelligence. Internet Policy Revieu, 9(2), 2020.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellic Lu, Thomas Mcsnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with leedback, 2023.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: -eedback-efficeett interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.0509, 2021.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent aligument via reward modeling: research direction. arXiv preprint arXiv:1811.07871, 2018.
Simon Lermen, Charlie Rogers- Rogers-Smith, Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training llama 2-chat 70b, 2023.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0100 2020.
Clayton Lewis. A research agenda for the nineties in human-computer interaction. Human- Human-Compuun Computer Interaction, 5(2-3):125-D1) 143, 1990.
Belinda Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Fliciting human preferences with language models. arXiv preprint arXiv:2310. 11589, 2023a.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song, Multi-step jailbreaking privacy attacks chatgpt. arXiv preprint arxi:22304.051 051997, 2023b.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time inter- vention: Eliciting truthful answers from a language model, 2023c.
Mengdi Li, Xufeng Zhao, Jac Hee Lee, Cornelius Weber, and Stefan Wermter. Internally rewarded reinforce- ment learning. arXiv preprint ar arXi::2302.002 2023d.
Mengxi Li, Alper Canberk, Dylan P Losey, and Dorsa Sadigh. Learning human objectives from sequences of physical corrections. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 2877 2883. IEEE, 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. arXiv preprint ar X.009211.0011 10, 2022a.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-basee reinforcement learning. In International Conference on Learning Representations, 2022b.
Hunter Lightmmn, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXio preprint 2XXi:2355520050 2023.
Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan, Inferring rewards from language context. arXiv preprint ar Ttt:22244.0511 2022.
30

---

Published in Transactions on Machine Learning Research (12/2023)
David Lindner and Mennatallah El-Assady. Humans not boltzmann distributions: Challenges and opportunities for modelling human feedback and interaction in reinforcement learning, orXiv preprinl arXiv:2206.1331 2022.
David Lindner, Xin Chen, Scbastian Tschiatschek, Katja Hofmann, and Andreas Krauso. Learning safcty constraints from demonstrations with unknown rewards. arXiv preprint XXXv:2305.16143 2023.
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2023.
Dylan Losey, Andrea Bajcsy, Marcia K O'Malley, and Anca Dragan. Physical interaction as communi- cation: Learning robot objectives online from human corrections. The International Journal of Robotics Research, 41(1):20 44, 2022.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andrcas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint Xiv: 1906. 03926, 2019.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dopendent human feedback. In International Conference Machine Learning, pages 1285-22944 PMLR, 2017.
Aman Madaan, Niket Tandon, Prakhar Gupla, Skyler Hallinan, Luyu Gao, Sarah Wiegrefle, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with feedbac..... arXiv preprint ar 2rx:22300 1 1751,, 2023.
Shchryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning, pages 7390 7399. PMLR, 2021.
David Manheim and Scott Garrabrant. Categorizing variants of goodhart's law. arXiv preprint arXiv:1803. N83.00803 2018.
Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave. On the fragility of learned reward functions. arXiv preprint ar arX:::330100352, 2023.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint X20::2200000229,, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
Eric J Michand, Adam Gleave, and Stuart Russell, Understanding learned reward functions. arXiv preprint 2111:20022008033 2020.
Silvia Milano, Mariarosaria Taddeo, and Luciano Floridi. Ethical aspects of multi-stakeholder recommenda- tion systems. The information society, 3711):35-45, 2021.
Smitha Milli and Anca D Dragan. Litcral or pedagogic human? analyzing human model misspecification in objective learning. In Uncerlainly in artificial intelligence, pages 925 934. PMLR, 2020.
Soren Mindermann and Stuart Armstrong. Occam's 's razor is insufficient to infer the preferences of irrational agents. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 5603- Red Hook, NY, USA, 2018. Curran Associates Inc
Abhilash Mishra. Ai alignment and social choice: Fundamental limitations and policy implications. ar preprint arXiv:2310. 16048, 2023.
Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: three-layered approach. arXiv preprint ar Xii:2332208800 2023.
31

---

Published in Transactions on Machine Learning Research (12/2023)
Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In Conference on Robot Learning, pages 342-352. PML.R, 2021.
United States National Commission for the Protection of Human Subjects. The Belmont report: ethical principles and guidelines for the protection of human subjects of research, volume 1. United States De- partment of Health, Education, and Welfare, National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1978.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pages 278-287. Citeseer, 1999.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning, In leml, volume 1, page 2, 2000.
Richard Ngo. The alignment problem from deep learning perspective. arXiv preprint arXiv:2209.00636 2022.
Khanh Nguycn, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXtt:1707.074000 2017.
Evgenii Nikishin, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. Improving stability in deep reinforcement learning with weight averaging. In Uncertainty in artificial intelligence workshop on uncertainty in Deep learning, 2018.
Ritesh Noothigatlu, Snehalkumar Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Raviku- mar, and Ariel Procaccia. A voting-basee based system for ethical decision making. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Cullen O'Keefe, Peter Cihon, Ben Garfinkel, Carrick Flynn, Jade Leung, and Allan Dafoe. The windfall clause: Distributing the benefits of ai for the common good. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 327-331, 2020.
Salima Omar, Asri Ngadi, and Hamid H Jebur. Machine learning techniques for anomaly detection: overview. International Journal of Compuler Applications, 79(2), 2013.
A.J. Oncal. Chat gpt 'dan" (and other "jailbrcaks').
B54778001900231111111aaaa8888118 2023.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744, 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspocification: Mapping and mitigating misaligned models. arXiv preprint v:Xiv:2201.035/4 2022.
Rahul Pandey, Hemant Purohit, Carlos Castillo, and Valerie L Shalin. Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in in-the-loop machine learning. International Journal of Human-Computer Studies, 160:102772, 2022.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1 38, 2021.
Andi Peng, Besmira Nushi, Emre Kıcıman, Kori Inkpen, Siddharth Suri, and Ece Kamar. What you see is what you get? the impact of representation criteria on human bias in hiring. In Proceedings of the AAAI Conference on Human Compulation and Crowdsoureing, volume 7, pages 125 134, 2019.
Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, and Ece Kamar. Investigations of performance and bias in human-ai teamwork in hiring. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12089-12097 2022.
32

---

Published in Transactions on Machine Learning Research (12/2023)
Andi Peng, Aviv Netanyahu, Mark K. Ho, Tianmin Shu, Andreea Bobu, Julie Shah, and Pulkit Agrawal. Diagnosis, leedback, adaptation: A human-in-the-loop framework for test-time policy adaptation. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprinl arXiv:2202.0320 2022a.
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint ar1iv:2212.092: 2022b.
Billy Perrigo. Exclusive: The $2 per hour workers who made chatgpt safer, 2023. URL hattps://ting com/ [Accessed 07-May-2023).
6247678/ Shpppeni-chatttt--eeeeeee
Brandon Perry and Risto Uuk. Ai governance and the policymaking process: key considerations for reducing ai risk. Big data and cognitive computing, 3(2):26, 2019.
Neil Perry, Megha Srivastava, Decpak Kumar, and Dan Bonch. Do users write more insecure code with ai assistants?, 2022.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. On releasing annotator-level labels and information in datasets. arXiv preprint Xrxiv:1 10. 05699, 2021.
Xiangyu Qi Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine- tuning aligned language models compromises safcty, even when users do not intend to! arXiv preprint arXiv:2310.0300 0369933, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXie preprint arXiv:2305.1822 2023.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, LJCAI'07, page 2591, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauck- hage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language process- ing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXie preprinl arXiv:2210.012/ 2022.
Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv preprint arXiv:2306.04.88 2023.
Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint Xiv: 14455, 2023.
Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking lims into disobedience: Understanding, analyzing, and preventing jailbreaks, 2023.
Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King, and Saleema Amershi. Supporting human-ai collabora- tion in auditing lims with llms. arXiv preprint arXiv:2304.09991, 2023. URL 2304. 09991. pdf
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trushworthy Machine Learning (Sa TML.), pages 464-483. IEFE, 2023.
33

---

Published in Transactions on Machine Learning Research (12/2023)
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior. arXiv: 001118855000000 (cs, statj, January 2019. URL http://arxiv.
htg/abss1805.0800 08010
arXiv: 1805.08010.
Siddharth Reddy, Sergey Levine, and Anca D Dragan. Assisted Perception: Optimizing Observations Communicate State. 2020.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thise Bpr: Bayesian person- alized ranking from implicit feedback. arXiv preprint arXiv:1205.261 2012.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference- based learning of reward funclions. 2017.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyaleai, Antoine Chaflin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot lask generalization. In International Conference on Learning Representations, 2022. URL htttps:/
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint or 2rX::22303.1555 7548, 2023.
Laura Sartori and Andreas Theodorou. A sociotechnical perspective for the future of ai: narratives, inequal- ities, and human control. Ethics and Information Technology, 24(1):4, 2022.
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXie preprinl 2000000000580 2022.
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback. In The First Workshop on Learning with Natural Language Supervision at ACI, 2022.
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback scale. arXiv preprint arXiv:2303.16755, 2023.
Amartya Sen. Social choice theory. Handbook of mathematical economics, 3:1073 1181, 1986.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference on Machine Learning, pages PMLR, 2019.
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint D:Xiv:2210.01700 09000 2022.
Rusheb Shah, Quentin Feuillade-Montiii, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. Scalable and transferable black-box jailbreaks for language models via persona modulation, 2023.
Steven Shapin and Simon Schaffer. Levialhan and the air-pump: Hobbes, Boyle, and the eaperimental life. Princeton University Press, 2011.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, New- ton Cheng, Esin Durmus, Zac Hatfield-Dodds Scott R. Johnston, Shauna Kravce, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023.
34

---

Published in Transactions on Machine Learning Research (12/2023)
Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Tor- ralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05186, 2022.
Yonadav Shavit. What does it take to catch a chinchilla? verifying rules on largo-scale ncural network training via compute monitoring, 2023.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. do anything now': Characterizing and evaluating in-the-wild jailbreak prompts large language models. Xiv preprint or arXiv:2308.000 03822. 2023.
Toby Shevlanc, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Danicl Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint Xix:22905. 15324, 2023.
Umer Siddique, Abhinav Sinha, and Yongcan Cao. Fairness in preference- based reinforcement learning, 2023.
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021.
Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning. arXiv preprint arXiv:2212.002) 032001 2022a.
Joar Skalse, Nikolaus HR Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. arXiv preprint arXiv:2209. 13085, 2022.
Joar Max Viktor Skalse and Alessandro Abate. The reward hypothesis is false. NewrIPS ML Sofety Workshop, 2022b.
Joar Max Viktor Skalse, Matthew Parrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning, pages 32033-32058. PMLR, 2023.
Charlie Snell, Πya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning, 2022. URL httpppp:/aa.xi.ogg//aba/////////
11871
Aaron J. Snoswell and Jean Burgess. The Galactica AT model was trained on scientific knowledge but it spat out alarmingly plausible nonsense, November 2022. URL http:://ttteccc/eeeraiioocccoo com/ the sallaaticcaai-mmee--oos wtt.://www.....t...........ooaaa
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5861 5873. Curran Associates, Inc., 2021. URL https://pooceddng.nnnunt http://ss..c/ppppr////////E/// 2e2551498aa007290066622222222223 .pff
Ziang Song, Tianle Cai, Jason D Lee, and Weijie J Su. Reward collapse in aligning large language models. arXiv preprint arXic:2305.1 17088, 2023.
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint arXiu:2010. 14603, 2020.
Jacob Steinhardt. Emergent. Deception and Emergent Optimization, February 2023. URL https:
//bounded-regret..cc htt.///araaeeeeee...oooooooooaoa
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsca Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 13:3000-3001,, 2020.
35

---

Published in Transactions on Machine Learning Research (12/2023)
Theodore R. Sumers, Mark K Ho, Robert D Hawkins, Karthik Narasimhan, and Thomas L Griffiths. Learn- ing rewards from linguistic feedback. In Proceedings of the AAAT Conference on Artāficial Intelligence, volume 35, pages 6002 6010, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotico, 17(2): 1999.
Ran Tian, Masayoshi Tomizuka, Anca Dragan, and Andrea Bajesy. Towards Modeling and Influenc- ing the Dynamics of Human Learning, January 2023. URL arXiv:2301.0080 0000000000111111 [cs].
Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-basee reward learning. In The Eleventh International Conference on Learning Representations, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Scholten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Alexander M Turner. Seeking power is convergently instrumental in broad class of environments, 2021. URL https://www http://wwww..toooooo.o.eoeeettea
Alexander Matt Turner and Prasad Tadepalli. Parametrically retargetable decision-makers tend to seek power. ArXiu, abbs/226.133777 2022.
Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. In Neural Information Processing Systems, 2019.
Victor Uc-Cetina, Nicolas Navarro-Guerrers Anabel Martin-Gonzalez, Cornelius Weber, and Stefan Wermter. Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56 (2):1543-1575, 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint Xix: 14275, 2022.
Peter Vamplew, Benjamin J Smith, Johan Källström, Gabriel Ramos, Roxana Räduleseu, Diederik M Roijers, Conor F Hayes, Fredrik Heintz, Patrick Mannion, Pieter JK Libin, al. Scalar reward is not enough: A response silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2): 41, 2022.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. Artificial artificial artificial intelli- gence: Crowd workers widely use large language models for text production tasks. ar preprint MXXi:2306.07899 2023.
James Vincent. love it, February 2023.
Microsoft's Bing is an emotionally manipulative liar, and poo-
URL https://ww.theverree /ttp://////////////////// microsoft-ai-binge mrr.apae....igaaaal....rrerrrrre
ple
36

---

Published in Transactions on Machine Learning Research (12/2023)
Alex Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In International Conference on Machine Learning, 2023.
Tony Tong Wang, Adam Gleave, Nora Belrose, Tom Tseng, Joseph Miller, Michael D Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, and Stuart Russell. Adversarial policies beat professional-level go ais. arXiv preprint arXi::2211.0022 10022, 2022.
Yufci Wang, Wanjun Zhong. Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, 2023.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? arXiv preprint or 2rw:22307.02488 2023.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadch, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from language models, 2021.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447-2469, Punta Cana, Domínican Republic, November 2021. Association for Computational Linguistics. doi: 10. www.olii.iiiiinneeeeeee
URL
Jess Whittlestone, Kai Arulkumaran, and Matthow Crosby. The socictal implications of deep reinforcement learning. Journal of Artificial Intelligence Research, 70:1003 1030, 2021.
Nils Wilde, Erdem Biyik, Dorsa Sadigh, and Stephen Smith. Learning reward functions from scale feedback. In Conference on Robol Learning, pages 353-362. PMTRR 2022.
Simon Willison. Prompt injection. 2023. URL htts://siionwwwionn http:///is...aoooo/s/ee/eeeroper
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-basel reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 46, 2017.
Yotam Wolf, Noam Wies, Yoar Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint 2xXi:23004.1082 2023.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiemnon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human leedback, 2021a.
Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Adversarial policy training against deep reinforcement learning. In USENIX Security Symposium, pages 1800, 2021b.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fino-grained human foedback gives better rewards for language model training, 2023.
Blake Wulfe, Logan Michael Ellis, Jean Mercat, Rowan Thomas McAllister, and Adricn Gaidon. Dynamics- aware comparison learned reward functions. In International Conference on Learning Representations, 2022. URL htttps://pperrvieeet http://www.eeeeereomeeddddd/aaa/
Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305.11 147000 2023a.
Wanqiao Xu, Shi Dong, Dilip Arumugam, and Benjamin Van Roy. Shattering the agent-ent interface for fine-tuning inclusive language models. arXiv preprint Xiv:222305. 1 1155, 2023b.
37

---

Published in Transactions on Machine Learning Research (12/2023)
Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: comprehensive survey. ar preprinl arXiv:2109.0668 2021.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The case of subverting safely-aligned language models, 2023.
Georgios N Yannakakis and John Hallam. Ranking vs. preference: a comparative study of self-reporting. In Affective Computing and Intelligent Interaction: 4th International Conference, ACI 2011, Memphis, TN, USA, October 9-12, 2011, Proceedings, Part I 4, pages 437-446. Springer, 2011.
Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising Ilm empowered by self-feedback generation, 2023. URL io/SelFee/
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao- Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. Arric preprint 2rv6222006. 08647, 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXic:2304.00 05302, 2023.
Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, and Junshan Zhang. Clare: Conservative model-based reward learning for oflline inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning, 2023.
Jiliang Zhang and Chen Li. Adversarial examples: Opportunities and challenges. IEEE transactions on neural networks and learning systems, (1)):2577 (17222577-2593, 2019.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiw:2305.1353, 2023.
Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. Advances in Neural Information Processing Systems, 34:12340- 12350, 2021.
Zhibing Zhao, Peter Piech, and Lirong Xia. Learning mixtures of plackett-luce models. In International Conference on Machine Learning, pages 2906-2914. PMLR, 2016.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arxiv preprint arXiv:2305.11200 11206, 2023.
Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11116 11124, 2021.
Banghua Zhu, Jiantao Jino, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. arXiv preprint arXiv:2301.11270 2023.
Simon Zhuang and Dylan Hadfield- Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33:15763-1555 15773, 2020.
Brian Ziebart, Andrew Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433 1438. Chicago, IL, USA, 2008.
Daniel Ziegler, Scraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Niclsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein-Ramn Daniel de Haas, al. Adversarial training for high-stakes reliability. Advances in Newral Information Processing Systems, 35:9274-990 99266 2022.
38

---

Published in Transactions on Machine Learning Research (12/2023)
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ar Niv preprinl WMXiv:1999008590 2019.
39

---

Published in Transactions on Machine Learning Research (12/2023)
A An Improved Model of the Human Feedback Process
As illustrated in Equation 1 the feedback process in RLHF is typically modeled with a. single human H with internal reward function THi examples sampled from the base model: 2; ~ no; and feedback as a function of the human, example, and noise: y/ x.,, However, as discussed in Section 3 this is a misspecified model of the process: there not a single human, humans values are not representable with a reward function, human actions are dependent context, and the sampling process can involve human. Thus we propose an allernative formulation.
Let AH. refer to a joint distribution of humans (or groups thereof if feedback is provided collaboratively) used for oblaining samples and feedback denoted as H. and Hffedbackk A dataset of examples is sampled from To (or some other source) where each example I; is defined to be a batch of one or more generations from the base model. Importantly, 2; may not contain all information about the world state (e.g,, if a; is a 2D rendering of a 3D environment), and the human may be able to observe more than just the model's output (e.g., if interpretabilitt tools are used to aid evaluation). So let. v be a rendering function that maps va and x; what a human sees. The behavior of humans varies over time and in different contexts, let ci gample and cfeedhack represent particular contexts for sampling and feedback collection. Denote the sampling process as s which maps the base model TT. a human Hy H.umple, and context c; gumple to some example 24. Notably, could ignore the basc model and generate offline samples from some other source. Finally, let I map a human Hffeebback, rendered example v(To, and context cfeedback to feedback yı. The dala collection process thus be more completely modeled as:
Nguupll Hfoedbaack AH, Ij s(TO, Hgample ,,, c
Y/ = f((TT,, 2x), H Hloodback of foedbback) (4)
which highlights need for future work better account for the aspects of this process that are commonly not accounted for when training systems with RLHF
B Explanation for Tractable/Fundamenna Classifications
In Section g we categorize problems as tractable or fundamental. The key distinction between the two is that fundamental challenges are substantial enough that overcoming them would require a method that is no longer a form of RLHF. Although many of the fundamental problems we identify can be alleviated by improving how RLHF is approached, they could be fully addressed with RLHF, As a result, they should be either avoided by not using RLHF compensated for by other safety mcasures. This distinction is soft, and some calegories of challenges are marginal. Here, we briefly explain each categorization.
Problems from Section 3.1
Tractable: Selecting representative humans and getting them to provide quality feedback is difficult:: This can be addressed by studying and improving the selection and training of evaluators.
Tractable: Some evaluators have harmful biases and opinions: This can be addressed by studying and improving the selection and training of evaluators.
Tractable: Individual human evaluators can poison data: This can be addressed with improved evaluator selection and quality assurance measures.
Tractable: Humans make simple mistakes due to limited time, attention, or care: This is marginal because human mistakes can never fully be overcome. However, they can be addressed with improved working conditions and quality assurance procedures.
Tractable: Partial observability limits human evaluators: Human evaluators can be provided with all information available in the policy's observations (although representing this in an casily-comprchensible way may be challenging).
Fundamental: Humans cannot evaluate performance on difficult tasks well: Human intelligence and cognitive capacity are limited. Humans cannot be expected to properly evaluate the performance of superhuman models on complex tasks. Thus, solving this problem would require no longer using human feedback in the way that RLHF does.
40

---

Published in Transactions on Machine Learning Research (12/2023)
Fundamental:
Humans can be misled, so their evaluations can be gamed: Human fallibility cannot fully be overcome, especially against optimization pressure from the learned policy.
Tractable: Data collection can introduce harmful biases: This can be addressed with improved data curation.
Fundamental:
There is an inherent cost/quality tradeoff when collecting human feedback: This tradeoff is unavoidable in practice obtaining diverse and high-quality examples (e.g. from long chatbot conversations) requires more effort.
Fundamental: RLHF suffers from tradeoff between the richness and efficiency of feedback types: This tradeoff is unavoidable for data collection in practice richer annotations require more effort.
Problems from Section 3.2)
Fundamental: An individual human's values are difficult to represent with a reward function: This problem is marginal. It can be improved in practice by improved modeling, but RLHF-based solutions will be limited by the intractability of perfectly modeling context and troubles with the reward hypothesis [Skalse and Abate
2022b
Bowling et al. 2023).
Fundamental: A single reward function cannot represent. a diverse society of humans: Trivial. Instead of being a fundamental limitation with RLHF, this is a broader limitation of Al aligument itself.
Fundamental: Reward models can misgeneralize to be poor reward proxies, even from correctly-labeled training data: This problem is marginal because it and should be addressed by improved sampling in practice. However, it is impossible to perfectly represent distribution with infinito support from a finite sample. Additionally, the deployment distribution will always differ from the training and evaluation distributions in real-world settings (Christiano 2019).
Fundamental: Optimizing for an imperfect reward proxy leads to reward hacking: reward model imperfect, reward hacking will always be a possibility from RL.
Tractable: Evaluating reward models is difficult and expensive: This can be addressed by perform- ing thorough and expensive evaluations.
Problems from Section 3.3
Tractable: It is (still) challenging to optimize policies effectively: This can be addressed with advancements in RL methodology.
Tractable: Policies tend to be adversarially exploitable: This problem is marginal because achieving certified adversarial robustness against practical threat models has empirically been intractable. Nonetheless, this can be addressed with robust optimization techniques.
Fundamental: Policies can perform poorly in deployment even if rewards seen during training were perfectly correct: This problem is marginal because it can and should be addressed by improved sampling in practice. However, it impossible to perfectly represent distribution with infinite support from a finite sample. Additionally, the deployment distribution will always differ from the training and evaluation distributions woal-world settings Christiano (201)).
Fundamental: Optimal R.L agents tend to seek power: Power is instrumentally useful for agents.
Tractable: The pretrained policy introduces biases into policy optimization: This can be ad- dressed with improved base models.
Tractable: RL contributes to mode collapse: This can be addressed with forms of RL that optimize for distributionnmmalcinng in desired instances.
Problems from Section 3.4
Tractable: Joint training induces distribution shifts: This can be mitigated with synchronous learning or other strategies.
41

---

Published in Transactions on Machine Learning Research (12/2023)
Tractable: It is difficult to balance efficiency and avoiding overfitting by the policy: This can be addressed with improved training methodology.
42


Reviews:
[{"review_id": "BtPiYXq4Ty", "reviewer": "Reviewer_PWoB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis is simply an excellent review. By reading the manuscript, the average TMLR community member will be able to go from 0 to having a great view and overall picture of the RLHF literature. I applaud the authors for getting the right balance between coverage and readability. The paper is also optimized for understanding, with many key points being summarized and reiterated consistently.\n\nReally, great job. One of the best surveys in the ML literature I have ever had the pleasure to read and review.\n\n## Weaknesses\n\n1. Focus on alignment is perhaps somewhat controversial. RLHF is fundamentally a general fine tuning technique that uses human-based sparse feedback and an RL loss, and while alignment is one of its main uses, it's not the only one. I wonder if it had been better to take a broader POV on the matter.\n2. It feels this review could have been made tighter by splitting into 3 manuscript: one about the algorithmic limitations and challenges of RLHF, another about the broader issues arising from collecting data from humans, and a final one about auditing and accountability. I'm not sure whether putting all 3 together fully serves the reader and the authors to advance all these narratives, but I do appreciate a good position paper, so I'm not sure myself either way.\n3. (nit) The data quality section (3.1.3) includes a fundamental limitation that seems to be more related to section 3.1.4 -- I wonder if that was an intentional separation, or perhaps an editing mistake? Signal / cost / efficiency seem to be sufficently linked that it might be worth rethinking the current categorization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a survey of challenges in the Reinforcement Learning with from Human Feedback (RLHF) literature, which is currently the state of the art framework for finetuning language models using fairly straightforward human feedback.\nThe manuscript summarizes the existing literature into a novel taxonomy of problems, clustering issues on whether they are  \"tractable\" problems within the RLHF framework, or fundamental limitations of the method.\n\nThe authors also review a broader set of alignment and AI safety research which discuss and/or is related to RLHF. Finally, the manuscript concludes with a discussion on governance and accountability challenges in the community developing LLMs with LRH, with highlighted pathways and suggested follow-up work on the topic.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Not much, really. See perhaps point 3 in the weaknesses section above. Frankly, this paper well passes the bar for publication at TMLR.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hJe6RjOJQ2", "reviewer": "Reviewer_XNWD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I will start by saying that this overview of RLHF in LLM is necessary as all this is pretty new and hyped and it is sometimes difficult to navigate the literature. And the submission does a good job at it in general. But this does not prevent it from having several flaws in my opinion, which I enumerate below categorized by severity.\n\nMajor:\n- It is a paper on RLHF for LLMs. This needs to be clear in the title. Most criticism is justified by the expectations we have on the end LLM product. Similarly, some sections seem more dedicated to LLMs than RLHF. I think that the paper needs to do a better job at identifying what is being discussed.\n- Several omissions in the historical literature of the domain. First, RLHF existed before Christiano2017 at least in the dialogue literature, where evaluation was an identified bottleneck for ML and RL for a long time []. It connects to several domains such as human-in-the-loop RL [], dialogue iteration score [], preference-based RL [], inverse RL [], reward shaping, etc. Also, it needs to be more clearly stated that RLHF is actually an Offline RL setting and that overfitting/overoptimizing is a well-known problem that can be addressed for instance with uncertainty estimates [].\n- I disagree with many Tractable/Fundamental flags. I would recommend to remove them because they are very subjective, don't bring much to the table, and yield confusion (even though I know that the choice is motivated in the Appendix).\n- 4.2: Many of the way to address the challenges with RLHF seem to be going backwards. For instance with \"Learning rewards from demonstrations\", we specifically use RLHF because the use of demonstration is inefficient. Something hybrid could be tried: handmade corrections of the system generation, but the authors present the approach as an \"alternative\".\n\nMinor:\n- 3.1.3 \"Data collection can introduce harmful biases\": it is motivated that there is a distributional shift, but the connection to harmful biases is not established.\n- 3.1.3 \"There is an inherent cost/quality tradeoff when collecting human feedback\": I would call it \"cost/quality/quantity tradeoff\" and it is unclear how it is a problem for RLHF. It means that more RLHF implies better performance.\n- 3.1.4: I would argue that this belongs more to the challenges with the reward model since the choice of type of feedback is more limited by the ability to train a good reward model on top of it.\n- 3.2.1 \"A single reward function cannot represent a diverse society of humans\": The problem is more on the data collection side imho. The problem is not that much that a single reward function cannot represent a diverse society of humans since the reward model could adapt to a specific user profile, the problem is that the annotator cannot annotate in accordance to a user profile, instead the annotator may consider their own user profile as an annotating guidance.\n- 3.2.2 \"Reward models can misgeneralize to be poor reward proxies, even from correctly-labeled training data\": This is ML. How is that specific to reward modelling?\n- 3.3.1 \"It is (still) challenging to optimize policies effectively\": this is more about training Offline RL policies, which is one order less effective than RL policies.\n- 3.3.2 \"Optimal RL agents tend to seek power\": I did not understand this point.\n- 3.3.3 \"The pretrained model introduces biases into policy optimization\". The last two lines are a very good example, but it is a problem of reward model generalization, how is that related to this specific problem?\n- 3.3.3 col2 last paragraph: \"Mode collapse is plausibly due in part to switching from the supervised pretraining objective to an RL objective\" => this is because we move from a distributional objective to an objective function maximization. We could provide the RL agent with this kind of objective (and this is kind-of already the case as the policy is regularized towards the pretrained one).\n- 3.4 \"Joint training induces distribution shifts\": \"and features with underestimated rewards will disappear\" => this is an exploration problem. This is the problem of online RL in real world scenarios: we don't want to let it explore low rewards actions, which includes the undersestimated ones. I don't see how it relates to its open problem.\n\nTypos/language:\n- p1 col1 2nd line from bottom: \"down) (Perez\" =>\"down, Perez\".\n- p5 col1 1st line of 3.1.1: \"innocently\" => \"unintentionally\" (other propositions could be inadvertently, candidly, accidentally, unknowingly, mistakenly, carelessly)\n- p9 col1 last paragraph: the first half is very repetitive and redundant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an overview of Reinforcement Learning from Human Feedback (RLHF) as used to fine-tune Large Language Models (LLM). The submission lays out three clearly defined contributions: \n1/ a survey of open problems that RLHF is currently facing. This survey is broken down into the three main RLHF components: data collection, reward modelling, and policy training.\n2/ an overview of techniques to understand, improve, and complement RLHF state of the art,\n3/ a proposition for auditing and disclosure standards to improve societal oversight of RLHF-powered LLMs (arguably this section relates more to LLMs than to RLHF).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the major and minor points I raised.", "broader_impact_concerns": "The whole document is somehow motivated by Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AXqHUhCVeo", "reviewer": "Reviewer_Y1DJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is at first a bit surprising because it does not contain a technical contribution. However, the paper is generally very well-written, and the amount of work done to collect all the references and discuss that is commendable! Overall, I think that this paper provides significant information to people who fine-tune LLMs, and policy-makers around LLMs.\n\nStrengths:\n\n- The references discussed by the paper are numerous and diverse\n- The structure of the paper is very clear\n- The paper is very well-written and easy to read. It is a clear tool to understand the current status of fine-tuning LLMs\n\nWeaknesses:\n\n- The background section lacks intuition. It is very mathematical even though the rest of the paper seems tailored to a broader audience (not just maths and technical people). The background section lacks simple sentences that explain what LLMs are, what they are useful for, where they are used (existing products), where they already failed (examples of products misbehaving for instance), where RL is used in all that, and how it is done. The last two points are extensively presented using formulas, but the intuition is difficult to get from the formulas.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a rather large-scale review of the existing literature on Reinforcement Learning from Human Feedback, applied to fine-tuning large language models (LLMs). The paper is divided in two main sections:\n\n- An identification of the problems in RLHF for finetuning LLMs (with, at first, some explanation as to why Reinforcement Learning is used for that). The problems being identified are thoroughly grounded in the existing literature and recent experiments. The problems are grouped by category.\n- The proposition of some solutions or areas of research to alleviate the problems identified in the first part of the paper. Here again, this is not a simple discussion, but a thorough review of what other papers present as possible solutions, results of experiments, and existing discussions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My only requested change is addressing the weakness mentioned above, the fact that the background focuses too much on mathematical notations and general concepts ($\\pi_{\\theta}$ that generates samples, but what are samples? How does it work in practice? How does it integrate with the neural architecture of the LLM). I would suggest taking one LLM, for instance LLama (so that it is not always ChatGPT), and summarizing how it was pre-trained, then how it was fine-tuned.", "broader_impact_concerns": "The broader impact of this work is overall positive, because the point of the paper is to discuss biases and limitations in how humans are used to fine-tune LLMs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes a comprehensive survey and analysis of challenges in Reinforcement Learning from Human Feedback (RLHF) for fine-tuning large language models (LLMs), proposing solutions and research directions.
All the reviewers agree that this paper provides a thorough literature review on RLHF and deserves publication.
On the other hand, there were concerns about the lack of a distinct technical contribution and the need for improvements in the background section. The authors have addressed these concerns in their rebuttals and have incorporated the suggested changes.
The paper can be accepted for publication. Congratulations!
Decision: Accept (poster)