A Survey of Reinforcement Learning from Human Feedback
Authors: Timo Kaufmann
Paul Weng
Viktor Bengs
Eyke Hüllermeier
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human-computer interaction. This positioning provides a promising approach to enhance the performance and adaptability of intelligent systems while also improving the alignment of their objectives with human values. The success in training large language models (LLMs) has impressively demonstrated this potential in recent years, where RLHF has played a decisive role in directing the model's capabilities towards human objectives. This article provides an overview of the fundamentals of RLHF, exploring how RL agents interact with human feedback. While recent focus has been on RLHF for LLMs, our survey covers the technique across multiple domains. We provide our most comprehensive coverage in control and robotics, where many fundamental techniques originate, alongside a dedicated LLM section. We examine the core principles that underpin RLHF, how algorithms and human feedback work together, and discuss the main research trends in the field. Our goal is to give researchers and practitioners a clear understanding of this rapidly growing field.

A Survey of Reinforcement Learning from Human Feedback
Timo Kaufmann
finno.kaufrrarmmShfff.m... de
LMU Munich, MCML Munich
Paul Weng
paul. www.wn@BBtukkkknssonn.........
Digital Innovation Research Center, Duke Kunshan University
Viktor Bengs
viktor. ventor.bengseee Bdffii de
German Research Center for Artificial Intelligence (DFKI)
Eyke Hüllermeier
eyke de
LMU Munich, MCML Munich, DEKI Kaiserslautern
Reviewed on OpenReview: https: openreview. net/ forun? id= f70kIur44
Abstract
Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human foedback instcad of relying on an engincered reward function. Building prior work the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human- computer inter- action. This positioning provides promising approach to enhance the performance and adaptability of intelligent systems while also improving the aligmment of their objectives with human values. The success in training large language models (LLMs) has impressively demonstrated this potential in recent years, where RLHF has played decisive role in direct- ing the model's capabilities towards human objectives. This article provides an overview of the fundamentals of RLHF, exploring how RT agents interact with human feedback. While recent focus has been RLHFF for LLMs, our survey covers the technique across multiple domains. We provide our most comprehensive coverage in control and robotics, where many fundamental techniques originate, alongside dedicated LLM section. We examine the core principles that underpin RLHF, how algorithms and human foedback work together, and discuss the main research trends in the field. Our goal to give researchers and practitioners clear understanding of this rapidly growing field.
Contents
1
Introduction
3
1.1
Why Human Feedback?
3
1.2
The Origins of RLHP
4
1.3
Scope of the Survey
5
1.4
Outline
2
Preliminaries
7
2.1
Reinforcement Learning
7
2.2
Preference-Basee MDPs
9
2.3
Reward Learning
9
2.4
Reinforcement Learning from Human Feedback
10

---

Published in Transactions on Machine Learning Research (06/2025)
2.5
Active Learning...
11
3
Feedback
12
3.1
Attributes of Feedback Types
12
3.2
Common Classes
3.3
Initializations
19
3.4
Choice of Feedback Type
20
3.5
Combination of Feedback Types
20
4
Label Collection
21
4.1
Active Learning
21
4.2
Challenges of Human Labeling
25
Reward Model Training
28
5.1
Human Feedback Model
28
5.2
Utility Learning
33
5.3
Evaluating Learned Reward Functions
37
................................
5.4
Reward Model Inputs
................................
38
5.5
Increasing Feedback Efficiency
38
6
Policy Learning
41
6.1
Adaptation RL Algorithms
41
6.2 Framing RLHF for Generative Models as a Bandit Problem
................................
42
6.3 Direct Policy Optimization
43
7
Theory
44
7.1
Policy Learning
44
7.2
Preference-Based vs. Reward-Based Learning
49
7.3
Nash Learning from Human Feedback
50
Applications and Benchmarks
50
8.1
Applications..
50
8.2
Supporting Libraries
51
8.3
Benchmarks
51
8.4
Datasets
52
8.5
Evaluation
53
Discussion and Conclusion
53
Extensions and Related Approaches
9.2
Current Limitations
54
9.3 Conclusion
55
A RLHF for Language Models
86
B Prior Surveys
88
B.1 Human-in-the-Lone RL
89
B.2 PbRL and RLHF
90
2

---

Published in Transactions on Machine Learning Research (06/2025)
1 Introduction
In reinforcement learning (RL), an agent navigates an environment and learns through trial and error. The optimality of these decisions is determined solely by reward signals, which must. be carefully designed based on performance measurements to ensure the agent learns the desired behavior. However, designing effective reward functions presents significant challenges. In real-world applications, success is often difficult to formally define and measure. Moreover, sparse success signals are not well suited for agent learning, often necessitating reward shaping (Ng et al., 1999), where the reward signal is transformed into one that is more suitable for learning. Unfortunately, this transformation can introduce spurious correlations rewarding behaviors that correlate with desired outcomes without directly achieving the true objective. This can lead to reward hacking (Skalse et al., 2022), where agents exploit such correlations to maximize rewards while producing undesired outcomes.
Reinforcement. learning from human feedback (RLHF) has emerged as practical solution to these challenges by incorporating human feedback directly into the learning process. Unlike traditional RL, where objectives are fixed a priori, RLHF allows humans to define and iteratively refine objectives during training. This approach addresses the limitations of classical RL while promoting better alignment between agent behavior and human values and objectives, enabling the development of more ethically sound, socially responsible, and practically useful Al systems.
RLHF has seen number of successful applications, methodological advances, and theoretical insights since the last comparable survey (Wirth et al., 2017). The applications span various domains, including large language model (LLM) fine-tuning (OpenAl, 2022), image generation (Lee et al., 2023), music genera- tion (Cideron et al., 2024), contimuous control (Christiano et al., 2017), games (Ibarz et al., 2018), and robotics (Hejna & Sadigh, 2022). Alongside these applications, significant methodological advances have emerged. These include fusing multiple feedback types to leverage their relative strengths (Section 3.5), enhancing query efficiency through active learning and query synthesis (Section 4.1.1), incorporating psy- chological insights to improve feedback quality (Section 4.2.1), using techniques such meta-learning to quickly adapt learned preferences to new tasks using prior data (Section 5.5.1), and using available prof- erence data more efficiently through approaches such as data augmentation and semi-supervised learning (Section 5.5.2). Finally, novel theoretical results (Section 7) have provided insights as well as new questions about the fundamental mathematical problems underlying RLHF.
While we focus on the core principles underlying RLHF, LLM fine-tuning requires additional considerations due to the unique characteristics of LLMs and their operational scale. We address these considerations without delving deeply into LLM-specific details. Notably, applying RLHF to LLMs is not our primary focus. While RLHF for LLMs remains in scope, we survey RLHF broadly, examining techniques and applications across domains with particular attention to control applications. Nonetheless, many techniques and insights discussed herein apply to LLMs, and we address some unique aspects of LLM fino-tuning. For readers primarily interested in LLM fine-tuning, Section A provides an LLM-focused introduction that complements this survey and guides the application of the insights presented here to LLM fine-tuning. Note that the literature focused on RLHF for LLMs in particular vast and rapidly growing, and we cannot provide a comprehensive overview of this area.
This survey therefore provides an overview of current RLHF research, classifies existing approaches, describes their main characteristics, and briefly surveys application areas. In the remainder of this section, we begin by examining the motivation (Section 1.1) and origins (Section 1.2) of RLHF, defining the scope of this survey (Section 1.3), and outlining the structure of subsequent. sections (Section 1.4).
1.1 Why Human Feedback?
In conventional RL, the agent's objective is defined by a reward function that it aims to maximize (Sut- ton & Barto, 2018). Specifying this reward function can be challenging, particularly in complex domains: What would be a suitable reward function for a robot assisting humans in houschold environment or for autonomous vehicles navigating through a busy urban environment? Moreover, even reward functions that initially seem well-defined can lead to unexpected behaviors due to distributional shifts or over-optimization,
3

---

Published in Transactions on Machine Learning Research (06/2025)
raising practical and safety concerns. Learning the agent's objective from human feedback circumvents re- ward engineering challenges and fosters robust training, with the reward function being dynamically refined and adjusted to distributional shifts as the agent learns.
Interactive Feedback vs. Demonstrations The field of inverse RL aims to infer reward functions from human demonstrations (Arora & Doshi, 2021). While this can partially resolve reward engincering challenges, it faces inherent difficulties: (i) it is generally not possible to robustly identify rewards from demonstrations (Cao et al., 2021a; Mindermann & Armstrong, 2018), (ii) it is only applicable in scenarios where good demonstrations can be obtained, (iii) it struggles to outperform the demonstrator, and (iv) humans often do not demonstrate the behavior they would prefer machine to adopt (Basu et al., 2017). Interactive feedback, by contrast, can use active queries to differentiate between aspects relevant irrelevant to the human preference, is much casier to provide than demonstrations, does not require near-optimal performance from human evaluators, and elicits preferences about the behavior that a human would prefer from the machine. Interactive feedback also complements demonstrations effectively, refining capabilities Iearned through initial training methods like behavioral cloning and preventing overfitting to demonstrated behavior (Abramson et al., 2022).
Avoiding Reward Engineering Reward engineering in RL presents significant challenges, as accurately specifying reward functions is notoriously difficult (Amodei et al., 2016; Knox et al., 2023). These challenges can be mitigated by using human feedback, which enables training agents for tasks that are hard to define manually and represents a step towards addressing safety issucs arising from misaligned rewards (Skalse al., 2022). Safety issues related to a misalignment between the agent's and the human's objectives are studied the AT alignment problem (Gabriel, 2020), in particular agent alignment and value alignment (Kirchner et al., 2022).
Excessive optimization of poorly specified rewards frequently produces unintended behaviors. Examples of such behaviors include exploiting flaws in the environment. simulation for higher rewards (Lchman et al., 2020; Baker et al., 2020) or engaging in more general reward hacking (Skalse et al., 2022), where the behavior maximizes the specified reward but deviates from the intended objective. This is evident in cases where agents focus on intermediate rewards without achieving the actual goal (Clark & Amodei, 2016) or prematurely exit games to avoid negative rewards (Saunders et al., 2018). The root of these issucs is that the reward function does not properly rellect the actual learning task. While such problems appear trivial in game enviromments, they become critical in cnfety-critical contexts such as healthcare and autonomous driving, where misaligned rewards could lead to care robots causing injurics or self-driving cars compromising road safety.
RLHF presents a promising approach to enhance alignment (Leike et al., 2018) by enabling agents to learn from human feedback, which is often more closely aligned with the true objective than manually specified rewards. Nonetheless, the effectiveness of RLHF in resolving these aligument issues is debated (Christiano, 2023). Examples of possible pitfalls raised in this debate are that an agent may be incentivized to manipulate the human teacher to provide feedback that is easier to optimize (Armstrong et al., 2020; Carroll et al., 2023) or that the agent may learn to exploit errors in human judgment (Ngo et al., 2024). We refer the interested reader to the survey by Casper et al. (2023) for a more detailed discussion of these issues. Despite these concerns, RLHF represents an important early step towards aligning agents with human values and serves as foundation to build on to further improve agent alignment.
1.2 The Origins of RLHF
Learning behavior from human feedback has long been studied a subfield of RL, but methods and ter- minology have evolved over time. Early methods focused on learning directly from human rewards (Knox, 2012; Isbell et al., 2001; Knox & Stone, 2009), from action advice (Maclin et al., 2005), or from action critique (Judah et al., 2010). Notable examples include TAMER (Knox & Stone, 2009; Warnell et al., 2018), which interprets human feedback as samples of the optimal action-value function, and COACH (MacGlashan et al., 2017; Arumugam et al., 2019), which interprets human feedback in a policy-dependent way, i.c., as samples of the advantage function. This survey, however, focuses on more indirect approaches inferring the objective from human feedback.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Table Feedback types classified as belonging to PbRL, SSRL, and RLHF as defined in this survey.
Feedback Type
PRRL
SSRL
RLHF
Binary trajectory comparisons
Trajectory rankings
State preferences
Action preferences
V
Binary eritique
Scalar feedback
Corrections
Action advice
X
Implicit feedback
Natural language
X X
Modern reinforcement learning from human foedback (RLHF) originates from preference-based reinforcement learning (PbRL), independently introduced by Akrour al. (2011) and Cheng et al. (2011). PbRL infers the objective from qualitative feedback, such as pairwise preferences between behaviors or between actions given states, instead of quantitative feedback in the form of numerical rewards. The term RLHF was later coined alternative (Askell et al., 2021; Ouyang et al., 2022; OpenAI, 2022), competing with the short-lived term reinforcement learning from human preferences (RLHP) (Menick el al., 2022). though initially referring to the same concept of learning behavior from relative feedback.
Disentangling PbRL and RLHF challenging due to their overlapping usage in the literature. For instance, Christiano et al. (2017) themselves are using the term PbRL, yet are often cited as a seminal reference for RLHF (Danicls-Koch Freedman, 2022; Ouyang et al., 2022). This demonstrates that the terms are often used interchangeably. Practically, RLHF is often associated with reward modeling and deep RL, while PBRL is often linked to direct policy optimization in traditional RL settings. This is underlined by Jeon et al. (2020), who characterize PbRL as limited to direct policy learning from preferences. This is in contrast with other sources, however, who include reward learning within the scope of PbRL (Christiano et al., 2017; Wirth et al., 2017). Additionally, PbRL is rarely used in the language modeling domain, while RLHF is common in both language modeling and classical RL domains such as continuous control.
Despite overlapping and sometimes conflicting usage, in this work we view RLHF a gencralization of PbRL. While both involve human feedback to define RI. objectives, PbRL primarily focuses relative feedback, such binary comparisons and rankings. RLHF not only includes these aspects but also extends to wider range of feedback types (Metz et al., 2023; Yuan et al., 2024). Table 1 gives an exemplary overview of our interprelation of these terms.
Another concept, supe-super reinforcement learning (SSRL), introduced by Christiano (2016) and dis- cussed by Amodei et al. (2016), refers to an RL setting where an agent reccives foedback on a subset of its experiences. The initial discussions of SSRL focused on absolute feedback on subsets of the agent's experi- ences, making the concept complementary to PbRL. In contrast to PbRL and RLHF, the term SSRL scems to be used less in the recent literature.
We adopt the view that RLHF broadly encompasses all approaches using human feedback to define RL objectives, including both PbRL and SSRL. As the definitions and distinctions between these terms are not universally agreed upon, these distinctions are based on our interpretation of the current predominant usage of these terms in the literature.
1.3 Scope of the Survey
This section outlines the criteria guiding our selection of RLHF approaches we cover. We concentrate on methods where a learned reward model serves as the sole source of objective information, learned through interactive, online, scalable, and asynchronous human feedback. We describe each of these criteria more detail below.
5

---

Published in Transactions on Machine Learning Research (06/2025)
Reward Modeling We focus approaches that learn reward model from human feedback and then use this model to train a policy. Although it is possible to directly optimize a policy from human feedback (Wirth et al., 2017), thereby performing RLHF without reward learning, this approach was rarely practiced for a long time and has only recently gained renewed interest, especially in the domain of language model fine-tuning (see Section 6.3). The decomposition into reward learning and policy training offers many conceptual and practical benefits. These include the direct applicability of supervised learning techniques for the reward model and the ability to evaluate the reward model in isolation. Additionally, the decomposition naturally leads to form of semi-supervised learning, enabling the agent to use labeled episodes for reward model training while leveraging unlabeled episodes to refine its behavior and explore the environment.
Human Defined While many approaches include humans in the RL loop, we focus on approaches where human feedback is the only source of truth about the objective. This excludes approaches to reward shaping, feature engineering, and other forms of human guidance that are supplementary to a given objective.
Interactive and Online Our scope encompasses methods that collect human feedback interactively during the learning process, rather than relying solely on pre-collected demonstrations. This excludes pure imitation learning, learning from demonstration, and standalone inverse RL approaches. However, hybrid methods that combine inverse RL with interactive reward refinement fall within our scope, as discussed in Sections 3.3 and 5.5.1.
Scalable and Asynchronous We consider methods that incorporate human feedback without requiring synchronous human-agent interaction. The agent must be capable of continuing its learning process while awaiting human input, and humans need not provide continuous supervision. This distin- guishes RLHF from more direct methods of incorporating human into the RI. loop, and we believe that this is key for practicality and efficiency.
Beyond these methodological criteria, we primarily cover works from 2017 through 2024 building upon the comprehensive treatment of earlier works by Wirth al. (2017). We selectively revisit earlier foundational contributions that continue to inform current practice have fundamentally shaped the field.
While RLHF has recently gained prominence through LLM applications, this survey adopts broader per- spective, examining RLHF across multiple domains with particular depth in control and robotics. We dedicate Section A to LLMs, addressing domain-specific challenges including large action spaces, distribu- tion shift, and KL regularization. Many techniques from control domains transfer naturally to the LLM setting, which is frequently constrained Given the rapidly evolving nature of LLM research, our coverage of techniques specific to this domain is less exhaustive than treatment of control and robotics applications.
a single turn, though not all techniques are equally applicable.
This survey can be scen as the canonical continuation of Wirth et al. (2017), examining the evolution from PbRL to RLHF and the accompanying methodological advances. We provide both a thorough description of the basics and an in-depth discussion of current advances and trends. We refer to Section for an overview of prior and related surveys and their relation to this work.
1.4 Outline
In the next section, we begin with an introduction to the basics by revisiting the most important concepts from the standard RL setting, which are also naturally important in RLHF (Section 2). We then dive inlo the RLHF topic by outlining the most studied scenario of reward model learning from pairwise preferences. Using this introductory and illustrative example scenario, explain the basic framework of RLHF alongside its three main components of (human) feedback, label collection (feedback acquisition), and reward model learning. These three main components will essentially form the structure of our survey. In Section 3, we turn our attention to the human feedback component and provide an overview of the different types of foodback as well as their key attributes. The important concepts in terms of label collection are then
We updated preprints to their reve-wel versions where applicable, leading references outside this window.
6

---

Published in Transactions on Machine Learning Research (06/2025)
explained in Section 4, followed by learning the reward model in Section 5. We also discuss policy learning in Section 6, since some RLHF methods adapt standard RL training methods to the RLHF setting. Section 7 is devoted to an overview of recent progress on the theoretical side of RLHF, including approaches involving a thcoretical guarantee, as well as theoretical insights into the relationship between standard RL and RLHF. Finally, Section highlights some interesting practical applications of RLHF and the existing benchmarks before Section concludes the survey by pointing out some possible avenues for future work.
2 Preliminaries
In this section, we review the basic setting and the most important concepts of RL and RLHF. In the course of this review, we will establish the notation that will be used throughout the survey. We first introduce what is probably the most studied RLHFF scenario, i.e. learning a reward model from binary trajectory comparisons. Based on this introductory and illustrative example scenario, we explain the basic framework of RLHF with its main components and briefly discuss the respective roles of these components in the learning process. We will also briefly touch active learning, which strongly connects to the feedback collection component.
Notation For any integer n E N, we denote by [n]] the set 21, nn.. For any set S, ASS) denotes the set of probability distributions over S. We use PPE for denoting the probability of some event E, while E[X] is used to denote the expected value of a random variable X. In some cases, we will write Ep[-] or similar variants to emphasize that the distribution for the expected value is governed by the probability distribution P E ((S). Morcover, we will write X P if a random variable X is distributed according to a probability distribution P.
2.1 Reinforcement Learning
Reinforcement learning (RL) (Sutton & Barto, 2018) is the setting of learning behavior from rewarded interaction with an environment. Such learning environment is formalized as an Markov decision process (MDP), which is a model for sequential decision-making In an MDP, an agent iteratively observes its current state, takes an action that causes the transition to a new state, and finally receives a reward that depends on the action's 's effectiveness. Formally, an MDP defined as tuple (S, P, R, do. where
S is set of states (the state space),
A is a set of actions (the action space),
P:S x A - (SS) is a transition function (the transition dynamics),
R: R:SxA
a reward function,
do A(S) is a distribution over initial states,
and y [0, 1] is discount factor.
The transition function P defines the dynamics of the environment: For any state s and action a, the value P(s, a))'), also sometimes denoted P(s') 8, a), is the probability of reaching the state s' after executing s. In light of this, we will also sometimes refer the transition function simply as the transition dynamics. For given state and action, the transition conditionally independent of all previous states and actions, which is known as the Markov property and the reason for the naming an MDP. The value R(s, a) R provides an immediate evaluation after performing action a in state s, which is also called the (instantancous) reward. It is also possible that the instantaneous reward is 0 for some states, and one only receives a reward in specific states, for example, in so-called terminal states for which the transition function is zero. When both the state space and the action space A are finite, we call the MDP tabular MDP.
In an MDP, an H H-step trajectory 5 sequence of H N \ (0) state-action pairs ending in a terminal state. Formally, it is given by (so, do, 81, a1...., ........)) .8). A trajectory T's return R(T) is the accumulated

---

Published in Transactions on Machine Learning Research (06/2025)
Agent
Action oz
Environment
Agent
Environment
Action
Dynamics
Policy
Dynamics
State 1
State se+11
Objective
Action de
Reward fr+1
Reward Tt++1
Query qi
Reward Model
Labeler
Label
(a) The standard RL setting.
(b) RLHF with reward modeling.
Figure 1: Contrasting the standard RL setting with RLHF in its most common formulation, using reward model. In each step, the policy commits to an action and receives the next state 8 and either the true reward r++1 or an estimate f++1 in return. In contrast to the standard RL setting, the true reward function is not known in the RLHF setting but instead learned from human feedback. This reward learning process is decoupled from policy learning and can happen fully asynchronously. The dataset consists of a set of queries qi (e.g., pairs of trajectory fragments) and their labels 4 (e.g., a preference for one the fragments).
(discounted) rewards collected along this trajectory:
R(T)
Σ 7 Rk) h=0
(1)
Note that we here use the same notation for the return and the reward function; however, both have different signatures (trajectory vs. state-action pair). We can also define the return R(a) of a segment a in a similar manner. The return is well definod even the horizon is infinite as long as 7 < 1. the MDP is a tabular MDP and any trajectory has finite length, i.e., H necessarily finile, we call the MDP finile and otherwise infinite.
A policy specifics how to select actions in state, either deterministically stochastically. In the former case, policy is simply a mapping S A from states to actions, while in the latter, it is mapping æ : S + ((A) from states to probability distributions over actions. Since the deterministic case is a special case of the stochastic onc, we assume the latter case in the following.
The basic interaction loop depicted in Fig. la: The agent chooses an action a ~ =8t) based on its policy and the current state. As consequence, the environment transitions the new state st+1 P(s,, at); governed by the transition dynamics. The agent observes this new state and the reward T+411 R(se, at), after which the interaction cycle repeats.
In this setting, the RI agent aims at learning a policy that maximizes the expected return
((T) = Ed, ((((T))
where the expectation is with respect to policy n, transition function P, and initial distribution dg. To solve this problem, two families of RL approaches have been considered: model-based RL and model-free RL. The methods in the first family learn model (i.e., P, R) of the underlying MDP to help solve the RL problem, while the methods in the second directly try to obtain a good policy without learning MDP model. The second family be further decomposed into two main categories: basee methods and policy search methods. In deep RI, both value functions and policies are approximated with neural networks.
Value-based methods (e.g., DQN and its variants (Mnih et al., 2015; Hessel et al., 2018)) aim at learning the Q-uunction Q" of an optimal policy. The Q-function of a policy æ is defined by:
H Σ
A R(s an)
(s, Ep
where so and ag a, and in the expectation, an T(- sx) as well as sh. P(- sh-1- ah-1) for h [H 1]. A policy can be naturally designed from Q-fnccion by choosing action in greedy
8

---

Published in Transactions on Machine Learning Research (06/2025)
manner in each state: a(s)- arg arg max, Q(s a). Note that for deterministic optimal policy 't it holds that J(x)) = Edg [Q* (s, (s)))
Similar to the action-value function Q, we can also define the state-value function
(8) Ep, Σ R((sh, an) ) 80 =
Its value for some state s is the expected return when starting in that state and then always using the policy w. It is related to the Q-function by means of
V (s(s) = Bawa(s) (s,
for any state S.
In contrast, policy search methods directly aim at finding good policy in some parametrized policy space. The most data-efficient algorithms in this class of methods follow actor-critic scheme where both an actor (i.e., policy) and critic (i.e., usually its Q-value function) are learned at the same time. Typical representative methods here are PPO (Schulman et al., 2017), TD3 (Fujimoto et al., 2018), or SAC (Haarnoja et al., 2018).
RL algorithms can further be classified as either on-policy or off-policy. In an on-policy algorithm, such as PPO, only the recently generated transitions are used for training. In contrast, in an off-policy algorithm, such as DQN (or its variants); TD3, SAC, the agent can be updated with transitions not necessarily generated by its current policy, While on-policy training is usually more stable, off-policy training enables more data-efficient learning by reusing samples from replay buffer that stores past transitions.
2.2 Preference- Based MDPs
In contrast to standard RL as described in the previous section, RLHF does not assume that a reward signal is available. It instead assumes the existence of an oracle (e.g., human labeler) that can provide information about the reward in a specific indirect manner. More precisely, RLHF, the agent can make queries qi to the oracle, which in practice means asking for human feedback, and in response, the agent receives a label 4, which in general gives a hint about the reward. In principle, the query can be made asynchronously the actual conventional RL cycle. See Fig. 1b for illustration.
In the most common setting, the oracle can compare two (segments of) trajectories, but various other cases have been considered, as we shall see later on. For the former case, RLHF is based on the setting of preference-based MDPs (Gilbert et al., 2017; Wirth et al., 2017), which can be defined as an MDP model without reward function, but where comparisons of trajectories are available.
2.3 Reward Learning
RLHF approaches can be divided into two categories, depending on whether a utility-based approach is used for reward modeling or whether an alternative criterion that is detached from a utility concept is used (Gilbert et al., 2016; Gilbert & Weng, 2016; Wirth et al., 2017). Most works fall into the first category, on which this overview focuses. Such approaches assume human-dependent utility function that can be used as reward function in order to apply standard RL methods. Next, we will describe the commonly used approach for reward learning for the common setting of binary trajectory comparisons.
The prevalent approach to learning a utility function from observations of pairwise comparisons is based on the Bradley-Terry model (Bradley & Terry, 1952), which stipulates a probabilistic model for the oracle (human labeler):
1
P(n > T2) 1 exp(R((2)) R(n))
where means "preferred to" and R(T) corresponds to the utility (i.e., return in the context of RL) of Note that this utility function is kind of surrogate function for the true reward function, which is
9

---

Published in Transactions on Machine Learning Research (06/2025)
(tacitly) assumed to induce the same optimal policy as the true reward function. For given data set D = { > =2 E [N]], utility function Rs parameterized by v can then be learned by the maximum likelihood principle (or equivalently using a cross-entropy loss):
N
max
exp (T5 Rs (Ti))
In the context of RL, since (= 7thh Rs an), (2) can then directly be used to train function approximator (e.g., single or ensemble of neural networks) to approximate R.
This approach accommodates the case of a noisy or unreliable oracle, which case the Bradley-Terry model can be understood as the generative model of the oracle's answers (or labels provided by the human labeler). When the oracle is reliable, more direct methods based preference elicitation for recovering the reward function have been studied (Regan & Boutilier, 2009; 2011; Weng & Zanuttini, 2013; Gilbert et al., 2015; Sadigh et al., 2017; Wilde et al., 2018). In this survey, we will focus on the general case where the oracle may be noisy.
Note that, in contrast to the typical way of preference learning, the learned reward function is used to train an RL agent and not directly to compare trajectories. This discrepancy ween the objective function used in reward learning and how the learned rewards are actually used may lead to suboptimal policies (Lindner et al., 2021b).
2.4 Reinforcement Learning from Human Feedback
In the RLHF setting as illustrated in Fig. 1b, the learning agent needs to solve an RL task without having access to a reward function. To this end, the agent usually simultaneously learns approximation of the reward function (via the assumed utility function) and an RL policy. Therefore, generic RLHF algorithm consists of repeating two phases: (1) reward learning and (2) RL training. The first phase can itself be decomposed into two main steps: (i) generate queries to ask the oracle, (ii) train a reward function approximator with the answers provided by the oracle. The RI. training part is more conventional and is usually directly based on running a deep RL algorithm using the currently trained reward function approximator.
Algorithm 1 Generic RLHF Algorithm in Actor-Critie Scheme.
Initialize parameters 0 (policy), & (critic), and vi (reward) 2: Initialize replay buller B with randomly-generated trajectories
3: for i === -....,, N do
4: // Reward learning
5: Generate queries from B
Update D with answers to queries from the oracle
Update ui using D (e.g., to maximize Eq. (2))
RL training
Update B with new trajectories generated with 10: Update 0 (actor) using B and Rs
11:
Update & (critic) using B and Rw
12:
end for
This basic generic algorithm is summarized in Algorithm 1, where an off-policy actor-critic scheme is assumed to be used for the RL training part, but other RL policy learning approaches can, of course, also be used here. For an on-policy algorithm, such as PPO (Schulman et al., 2017), only the recently generated transitions are used for training. For a DQN-like algorithm, lines 9 to 11 would be replaced by a loop where transitions are generated by a behavior policy based on the current estimate of the Q-function (e.g., e-greedy algorithm) and the Q network updated using mini-batches of transitions sampled from the replay buffer B.
An efficient RLHF algorithm needs to overcome several difficulties which are specific to this setting:
10

---

Published in Transactions on Machine Learning Research (06/2025)
The oracle may provide various types of feedback (see Section 3). The questions of what information some given feedback provides and how observed leedback can be exploited need to be answered (see Section 5).
Informative queries need to be generated to minimize the efforts of the oracle, which is crucial when it is a human (sce Section 4). Active learning techniques (sce next subsection) can be adapted to face this challenge.
The RL agent is actually trained in a non-stationary enviromment since the reward approximator is concurrently updated. The RL training part therefore needs to account for this factor, e.g., using non-vanishing learning rates (see Section 6).
There is also the question of how the agent's 's performance can be meaningfully evaluated, especially the reward function is not known (soe Section 8).
Collecting feedback directly from humans introduces its own challenges, such as the question of suitable user interface and the associated issues of delay between query and leedback observation, or the feedback variability and reliability (see Section 4). This may explain why many studies evaluate novel RLHF algorithms with simulated feedback.
A standard RL algorithm can be run in the RL training part, done in most previous work in RLHF (although this may not be the best approach). This suggests that any improvements in standard deep RL method (e.g., auxiliary losses (Gelada et al., 2019), planning in learned model (Hafner et al., 2020), curriculum learning (Narvekar et al., 2020), or data augmentation (Laskin et al., 2020; Lee et al., 2020; Lin et al., 2020b)) may potentially be transferred to the RLHF setting. In addition, most previous work in RLHF directly uses trajectories stored in replay buffer B to synthesize queries. An interesting research direction to explore in RLHF would be to specifically generate trajectories in order to be able synthesize more informative queries (instcad of only generating trajectories that are beneficial for RL training). This would lead to tackling a novel exploration-exploitttionn dilemma: Shall visit state-action pairs that may be bad but may help better learn the reward function, or shall we visit state-action pairs that we currently think are good? This is further discussed in Section 5.5.3.
In RLHF, since the oracle is a human or a group of humans, reducing the number of queries is crucial to limit the labeling cost. Therefore, the reward learning part requires techniques similar to those proposed in active learning, which we recall next.
2.5 Active Learning
In active learning (Settles, 2012), the task is to strategically select data points for labeling to minimize the amount of labeled data required to achieve a desired level of performance, which is particularly valuable scenarios like RLHF where labeling costly. Unlike batch learning, where labeled data is predetermined, active learning empowers the learner to actively select the most informative unlabeled instances for labeling, maximizing the learning process with limited labeled data. We will only briefly introduce the active learning task here and then discuss the strategies for creating informative queries considered thus far in Section 4.
For RLHF with pairwise comparisons, this setting can be formally described as follows. Suppose there is a set of N pairs of trajectories {(Ti, Tb) 1,..., .N,, where each pair (ri. can be interpreted as an unlabelod instance. To efficiently Icarn reward function to explain obscrved pairwise comparisons, an agent can select set of unlabeled pairs (possibly a singleton) to query an oracle to obtain their labels.
At high level, the main idea in active learning is to query data points to quickly reduce the epistemic (i.e., reducible) uncertainty about the predictions of the learned model, although other aspects can be important, such as the representativeness of the queried data points (see Wang el al. (2024b) for a survey). Two main representations are considered to describe this epistemic uncertainty: either using an ensemble of models or using a Bayesian representation. In both cases, a first basic approach selocts queries using uncertainty- based criteria in order focus on instances with high prediction uncertainty as measured by, e.g., variance entropy computed over predictions. In contrast to the first approach, where the selection criteria are
11

---

Published in Transactions on Machine Learning Research (06/2025)
instance-based, second approach considers criteria that may depend on all instances. Possible options are, for instance, expected model change, expected error reduction, or density-weightel uncertainty-based criteria. Here, the weights in the expectation or density allow us to take into account the distributional information about the instances and, therefore, to focus on the higher-density regions.
3 Feedback
Agent
Environment
Action
Policy
Dynamics
State st+1
Action az Reward f++1
Query qi
Reward Model
Labeler
Label 4
Figure 2: RLHF diagram highlighting components discussed in this section.
Feedback mechanisms are fundamental to the success of any RL system. In the standard setting as described in Section 2.1, RT. agents expect feedback the form of scalar immediate rewards. These rewards are most commonly determined by a hand-engineered reward function, which can be used to evaluate any state-action combination. As discussed in Section 1.1, is desirable to allow humans refine behavior interactively through feedback instead of requiring them to pre-specify reward function.
While a human could, in principle, assign rewards to each of the agent's 's actions dircctly, thereby taking the role of the reward function, this is usually impractical for multiple reasons. The main challenge is the human effort required to provide rewards on a sufficiently regular basis, i.e., at least once per episode. In addition to that, dircctly integrating human rewards into the RL loop would require these rewards immodiately, which would impede the learning pace while waiting for human feedback. Finally, the standard RL setting expects numeric state-action rewards, which are challenging to provide in consistent manner.
In contrast to directly rewarding cach of the agent's actions, RLHF as discussed in this survey (sce Section 1.3) harnesses indirect and asynchronous feedback methods. Such methods avoid the challenges of immediate numeric rewards and are also better aligned with human interaction patterns, resulting in improved learning progress and human user experience.
A feedback type is a kind of interaction in which a human conveys some information about their preferences. Examples include pairwise comparisons and direct critiques. This section is concerned with the many ways in which human feedback sorted feedback types by listing the most common ones (Jeon et al., 2020; Yuan et al., 2024) and discussing their attributes and dimensions (Metz et al., 2023; Lindner & El-Assadyy 2022). The attributes and classes described in this section build upon this prior work and can be considered a synthesis and extension of it.
be expressed and used. Several previous works have already studied and
The remainder of this section will start by discussing relevant altributes of leedback types that can be used classify them (Section 3.1). We will then discuss common classes and examples of interactive feedback types (Section 3.2) well as some non-interactive types that can serve initializations (Section 3.3).
3.1 Attributes of Feedback Types
Feodback types may differ in many dimensions, some of which relate to the way feedback is given (arity, involvement), others to the form of the query instance it is given on (granularity, abstraction), and yet others to features of the human interaction (intent, explicitness). The attributes we discuss in this section are based on the framework proposed by Metz et al. (2023). We have adjusted and added terminology where it aids clarity, generalized the distinction between relative and absolute feedback to 'arity', and added the categories of co-generative involvement and literal intent. Furthermore, systematically analyze set of exemplary
12

---

Published in Transactions on Machine Learning Research (06/2025)
feedback types in the next section, expanding on the initial examination of smaller set of abstract classes by Metz et al. (2023). In the following, we will introduce each of the six attributes in more detail.
Arity
This attribute describes whether single instance is evaluated in isolation (unary) relative to other instances (binary, n-ary). Unary feedback, e.g,, an absolute score for an observed behavior, is often convenient for detailed and descriptive feedback but lacks any grounding and therefore puts a great burden on the human to provide consistent feedback. Non-unary feedback always has an implicit grounding but requires the instances to be comparable. While n-ary feedback, such as a ranking over multiple observed behaviors, can provide more information than binary feedback, it also puts a higher cognitive burden on the labeler.
Involvement
The labeler may cither passively observe an instance, actively generate it, or coactively par- ticipate in its generation (co-generation). Passive involvement, exemplified by an evaluator that observes, but does not intervene a robot's behavior, poses the smallest challenge to the labelers since it does not require the ability demonstrate the task. It can also easily be directed at the most informative examples with active learning techniques. Unfortunately, passive feedback often cannot match the information density of active feedback such human steering robot to gen- erate a demonstration. It is, therefore, common to combine both types to first initialize the reward model from (possibly very suboplimal) active feedback and then refine it from passive feedback. Between these two extremes is ro-generative feedback, in which a human can share control with the agent. This can be less demanding than active feodback and makes it possible to direct the human's attention to the most informative samples, but it is still more taxing than purely passive involvement.
Granularity Feedback may also diller on the granularity of the instances being evaluated. This ranges from feedback over whole episodes over partial segments to feedback on individual steps (i.e., states, actions, or state-action pairs). A more coarse-grained granularity has the advantage of giving humans more context and getting feedback for larger sections of behavior but also poses credit assignment problems. Finer-grained feedback is much easier to learn from and simplifies credit assignment, but is often impractical or tedious for humans to provide. It is also possible to strike compromise, as demonstrated by Guan et al. (2021) who propose to use queries on step granularity, but batch queries within an episode to provide additional context and make it easier to give feedback. Note that we only classify type of feedback as "episode" granularity if it requires entire episodes. If it compatible with partial segments as well, we classify it "segment" even if the discussed source paper uses entire episodes.
A Abstraction This describes whether feedback given directly on raw inslances, e.g., behavior recordings (see granularity), or on abstract features of the instances, such as an autonomous car's speed and the forces experienced by the passengers. While feature-level information can be easier to learn from, extracting useful features is challenging. In some contexts, it may also be harder for human to make abstract judgments rather than more intuitive instance-level judgments. Note that this al always refers to the level of abstraction that the user sees, which may differ from the features used as inputs for the reward model. Types of feedback that depend on active generation (see involvement), such as improvements, generally work on a raw instance level.
Explicitness Humans may communicate explicitly for the purposes of feedback, c.g., by responding to a pairwise comparison query, or implicitly as a side-effect of actions directed at other purposes such as interventions in the agent's behavior. While explicit information is casiest to learn from, im- plicit information is often much more readily available and can possibly communicate more detailed preferences.
Intent
The assumed human intent can be important for feedback processing, A human may be evaluative, instructive, or descriptive in their explicit feedback, while they are generally literal in their implicit foodback. Evaluative, instructive, and descriptive feedback pedagogical in nature, aiming to teach a reward function, whereas literal feedback is a byproduct of a human actor's efforts to optimize the reward function directly. Descriptive feedback is often given on the level of the task, e.g., through
13

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: An overview of the common classes and their defining attributes. The feedback classes are grouped into primary classes, which can be used on their own to learn a reward model, supplementary (sup.) classes, which can be used in conjunction with a primary class, and representation be used to learn a representation prior to learning reward model. When + specified for attribute, this indicates that it is not defining feature of the class and may vary in different instantiations.
(rep.) classes, which can
Class
Granularity
Involvement
Arity
Abstr.
Intent
Expl.
Critique
Observed
Unary
Evaluative
Explicit
Comparisons
Observed
2+
Evaluative
Explicit
Primary
Inter-Tenne Inter-Temppral
Segment
Observed
Unary
Evaluative
Explici
Proxy Rewards
Episode
Observed
+
Feature
Descriptive
Explicit
Social Behavior
Segment
Observed
Unary
Instance
Literal
Implicit
Improvements
Episode
Co-generative
Unary
Instance
Natural Language
Observed
Unary
Descriptive
Explicit
Rep. Sup.
E-Stops
Episode
Observed Observed
Unary
Instance
Literal Descriptive
Implicit
Importance
Explici
Feature Traces
Segment
Active
Unary Ternary
Instance
Descriptive
Explicit
Similarity Queries
Observed
Descriptive
Explicil
a partial reward function. The other types of intent, in contrast, are generally given within the context of particular instance of behavior.
The distinction between literal and pedagogical feedback was introduced by Milli Dragan (2020). They argue that humans with pedagogical intent (i.e., evaluative, instructive, or descriptive) may act differently compared to humans with literal intent. Even though they find that assuming the (wrong) literal intent still lead to better reward inference, it still indicates that it can be important to know this intent to choose the right human model (see Section 5.1.5).
3.2 Common Classes
Even though the concrete types of feedback used in the literature are rarely exactly the same, they generally be sorted into set of common classes. We will describe selection of those classes, their defining attributes, and examples of concrete instances in the literature in the following. Table 2 gives an overview of the classes and their altributes as described in Section 3.1.
3.2.1 Primary Feedback Classes
This section introduces common feedback classes which can be used on their own to learn reward model. The classes are critique, comparisons, tenpralp feedback, proxy rewards, social behavior, improvements, and natural language.
Critique Critique is arguably the most direct type of feedback. In this setting, the human expresses their preference by directly critiquing an instance of agent behavior, often in the form of binary feedback. Note that critique, as considered in this survey, is distinct from directly supplying a reward signal since given in an asynchronous and indirect manner (see Section 1.3). In the critique setting, labeler may for example observe recordings of agent behavior and give their approval or disapproval. The defining features of critique are that the human passively observes the behavior (involvement). gives feedback on a single instance (arity), and does so explicitly (explicitness) with an evaluative intent. The feedback may be given for any granularity and on any level of abstraction.
There are many examples of critique feodback in the literature. Xiao al. (2020) employ binary feodback on individual state and action pairs. Although they learn shaping reward that complements an environment reward signal, the same technique could be used without environment. reward. Huang al. (2023) extend
14

---

Published in Transactions on Machine Learning Research (06/2025)
this to multi-label feedback, allowing the user to distinguish between a regular good or bad action and terminal action that achieves the goal or fails to do so. They map these classes to scalar values and then learn a reward model by regression. Wang et al. (2020) present an approach to learning reward model from noisy critiques in the form of human physiological signals (brain signals) using active querying. In contrast. to this action-level leedback, Fu et al. (2018b); Singh et al. (2019) rely on binary outcome success labels. Fu et al. (2018b) introduce the basic approach, which Singh et al. (2019) extend by moving to an off-policy setting and including online queries, thereby reducing the reliance on many positive examples by interactively correcting false positives.
In addition to learning the main reward function, critique can also be used for safety evaluation. Cosner et al. (2022) train a secondary reward model focused on safety from binary action critiques. This is in addition to the main reward model, which is trained from comparisons in their approach. Note that this secondary safety model could, in principle, be trained with any of the feedback types discussed here, using methods identical to the ones used for reward learning.
Comparisons Binary comparisons and rankings are among the most common types of feedback. The defining fcatures of comparisons are that the human passively observes the behavior (involvement), gives relative feedback on multiple instances (arity), and does so explicitly (explicitness) with an evaluative intent. It is most commonly given on a segment (granularity), but other gramularities such as individual actions are also possible (Cosner et al., 2022). Similarly, comparisons are commonly requested on an instance level (abstraction), but this is not a requirement.
Comparisons were first used for direct policy learning (Akrour et al., 2011; Cheng et al., 2011), but were later extended to the reward-learning setting (Wirth et al., 2016; Christiano et al., 2017). The most common setting (Christiano et al., 2017) relies on pairwise comparisons of trajectory segments, but comparisons of individual states or actions were also considered in early PbRL works (Fürnkranz et al., 2012) and comparisons can even be extended to more abstract trajectory features (Pinsler et al., 2018).
This basic setting has been extended and modified in various ways. To reduce noise in the labels, it is common to extend the binary choice by giving the labelers the option to avoid the hard choice and instead indicate incomparabilityy uncertainty, or perceived similarity (Holladay et al., 2016). This option is commonly interpreted as "equally preferable", c.g., as if each trajectory had an equal probability of being preferred in the preference predictor. It also common, however, to additionally provide an "incomparable" option which results in simply omitting the query from the data set (Christiano et al., 2017; Tbarz et al., 2018). In contrast to this, Verma et al. (2023) explicitly state that they do not allow for the equally preferred option, arguing that these cases are rare enough not to matter very much. Another line of research suggests that more precision in the expression of pairwise preferences, such as softening the hard binary choice to scalar feedback indicating the strength of a preference (Wilde et al., 2021; Touvron et al., 2023), can be beneficial for preference learning. Other extensions change the pairwise setting to choices among larger choice sets (Ziegler et al., 2020) or even full rankings (Myers al., 2021; Ouyang et al., 2022). Jain et al. (2015) compare different kinds of ro-ranking feedback (sclect first which is better than top, choose best from top-5, choose best from random set). Their study suggests that the first two are roughly equivalent, while the latter is much less informative. Askell et al. (2021) evaluate binary comparisons as well as ranking, but find that binary comparisons (even if extracted from a ranking) perform better. Ziegler et al. (2020) note that for language tasks, a larger choice set can amortize the time needed for a labeler to get acquainted with the context necessary to understand a query. Basu et al. (2019) propose to use hierarchical queries, i.e., a sequence of pairwise comparisons that build up on each other.
Pairwise comparisons are generally casier to supply than ratings, even when the feedback is given by foundation model (Wang et al., 2024c). While absolute feedback is generally dependent on policy serving as an implicit bascline (MacGlashan et al., 2017), the explicit basclinc in pairwise comparisons can be more easily controlled. Therefore, comparisons provide many benefits over absolute ratings, such as reduced bias, inconsistencies, and subjectivity (Yannakakis & Martínez, 2015). On the flip-side, comparisons convey relatively little information per label. Tien et al. (2023) study the weaknesses of pairwise-compprisonn-b reward learning. Since the amount of information provided for each label is small, these models are prone to causal confusion, i.e., misattributing reward to noise and misidentifying the reward function.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Inter- Temporal Feedback One limitation of trajectory comparisons is that they require set of roughly comparable trajectories. In many real-world environments, starting conditions or even the agent's 's current task may vary between episodes. In these cases, it is hard for human labelers to compare trajectories from these episodes, limiting the usefulness of comparison feedback. One way to remedy this limitation is to provide feedback within a single trajectory. Instead of comparing a set of instances with each other as in regular comparative foedback, inter-temporal feedback conveys relative judgments over different states time within a single instance, e.g., by comparing a robot's behavior at the start and the end of a trajectory. The defining features of inter-temporal feedback are that it is given explicitly (explicitness) on segment (granularity) while passively observing (involvement) a single instance (arity) of the agent's 's behavior with evaluative intent. It is most commonly given on raw instances, but any level of abstraction is possible in principle. There are two main ways to convey this feedback: Reward skelching and inter-lemporal preferences.
Reward sketching, as introduced by Cabi et al. (2020), involves users sketching a visual representation of the reward function over time. This type of feedback, which can be given by sketching a graph with the mouse while watching behavior recording, provides intuitive reward annotations for each time step. Rahtz et al. (2022) also adopted this approach, referring to it as "one of the highest-bandwidth feedback mechanisms currently available".
temporall preferences were introduced by Abramson et al. (2022). In this setting, humans give feedback on multiple points of trajectory, indicating whether an agent makes progress towards or regresses from a goal. This is then interpreted as preferences relative to the other labeled and unlabelled points. The authors note that one potential downside of this feedback type is that labelers may tend to give preferences on short-term actions that are easy to judge, failing to communicate long-horizon preferences. Cui & Niekun (2018) propose a similar type of feedback, in which humans segment a trajectory into good and bad parts. This makes it possible to derive many state-action labels from a few segmentation points.
Proxy Rewards Proxy rewards are partial or inaccurate reward functions that convey information about the task the agent is supposed to complete but may not induce optimal behavior. An example of this is a reward function that only rewards the agent for reaching the goal state but docs not penalize negative side- effects. This form of feedback does not generally refer to any particular behavior instance but instead gives global direction for the entire task. However, in line with our selection criteria (Section 1.3), we only consider proxy reward feedback that is interactive and online within the context of one or multiple observations to fill holes in the initial description. The defining features of proxy reward feedback is that the labeler passively observes the agent's 's behavior (involvement) and gives feedback explicitly (explicitness) on feature-level (abstraction) with descriptive intent (intent). Proxy reward feedback may be given with respect to a single or multiple instances (arity), although it generally refers to multiple instances. It is most commonly given on an episode (granularity), but other granularities are possible in principle.
The work by He & Dragan (2021) exemplifies this form of feedback through an iterative process where de- signers provide proxy reward functions on training environments. The method maintains a belief distribution over reward functions consistent with behaviors that achieve high proxy reward on training environments, then actively proposes new environments that maximize information gain about the true reward function, helping designers refine their reward specifications. Alternatively, Mindermann et al. (2018) suggest querying about the reward function, extending the inverse reward design framework of Hadneeld-Menell et al. (2017b), which treats proxy reward functions as observations to infer the designer's 's true intent. The active approach by Mindermann et al. (2018) allows users to choose from a set of understandable, lincar proxy rewards or to specify which features are more critical in the linear reward structure. In a related setting, Guan el al. (2023) lets the user specify changes to a current symbolic reward function in the form of changes to target attributes (c.g., walking speed).
Social Behavior Humans give rich implicit social feedback in the form of facial reactions and gestures when interacting with agents. The defining attributes of this type of foodback are that it given implic- itly (explicitness) on passively observed (involvement) segments (granularity) with respect to a single instance (arity) and literal intent.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Cui et al. (2021) propose a framework to learn reward functions from such social behavior. They suggest two-phase training uet In the first phase, they ground the implicit feedback by use of incentives, iee. they incentivize humans to have a known objective. After learning a mapping from feedback to reward, they use regular RL techniques to learn a policy. Note that the learned reward function can be seen as conditional on human implicit feedback and, therefore, they require a human in the loop throughout training.
Improvements Improvements are form of feedback in which the human improves on the agent's 's behavior, either by intervening as the agent acts or by providing a corrected behavior after the agent acts. To improve an episode, it is usually nocessary to observe the entire episode (granularity) at the instance level (abstraction). In this type of feedback, the human both observes and demonstrates behavior, resulting in co-generative involvement. Improvements generally relate to a single reference trajectory being improved (unary arity), although an improvement could also be interpreted as binary comparison between the improved and the non-improved trajectory. Improvements are most commonly provided explicitly with instructive intent.
We distinguish between post-facto improvements, calling them corrections, and improvements made while the agent is acting, calling them interventions. The key difference is that the uncorrected trajectory is available in the case of corrections, while it can only be estimated in the case of interventions.
Interventions can be considered to be an instance of the shared autonomy setting since the agent and the user share autonomy to reach a common goal. There are two main ways to leverage interventions: One is to learn from the occurrence of an intervention itself that the prior behavior was suboptimal, as done, e.g., by Luo et al. (2024) by directly inferring negative rewards from interventions without learning a reward model, and the other is to learn from the corrected trajectory. The latter setting is studied by Abramson et al. (2022), who ask humans to intercede on agent failure. They use this to collect targeted demonstrations where the agent is the weakest and to identify challenging situations for their evaluations. The gathered data is then used for behavior cloning and reward model training. The fact that a correction occurred is not directly used as feedback. In addition to the above distinction, interventions can come in two main formats: Either by overtaking control from the agent to correct its behavior by physically correcting the agent's 's movements. Losey et al. (2022) proposes to learn reward model from such physical corrections, a method that was later extended (Losey & O'Malley, 2018) to incorporate uncertainty for active learning and risk-sensitive deployment. Li ct al. (2021) further extend this sctting to learn from a sequence of correlated physical corrections without needing to wait until the trajectory is completed.
The correction case is closely related the setting of coactive learning (Shivaswamy & Joachims, 2015), in which learning system repeatedly proposes solution which user may correct to reach a common goal. Jain et al. (2015) treat corrections as a demonstration while Jeon et al. (2020) propose (but do not evaluate) an alternative interpretation of inferring implicit preferences from comparisons by assuming the corrected trajectory is preferred over the original one. Corrections are also commonly used in the LLM fine-tuning setting to incorporate AI feedback: By prompting a language model with a set of principles, it can critique and revise a generated response according to those principles. This can then be used to generate demonstrations (Bai et al., 2022b) or targeted pairwise preferences (Castricato et al., 2024) and to internalize these principles with further training on this data.
Natural Language Natural language is a versatile form of feedback that can be used to convey a wide range of information. It cannot only be used to express preferences, but also to suggest concrete changes. Natural language fcodback may be given on any granularity, at any level of abstraction. Its defining features are that it is given explicitly (explicitness) in the context of a single (arity) observed behavior or policy (involvement) with descriptive intent.
While much work in language for RL focuses on the problem of learning a policy that follows natural-language instructions in a non-interactive setting (Hermann et al., 2017; Nair et al., 2021), we focus on works where language is used as feedback, not for task definition. Note, however, that RLHF can be a useful tool for learning a language-conditionee policy, regardless of whether or not the feedback itself is in the form of natural language. As an example of this, Abramson et al. (2022) use RLHF in an interactive agents setting, where one agent. (responsible for solving problems) is controlled by the learned policy and another agent
17

---

Published in Transactions on Machine Learning Research (06/2025)
(responsible for setting tasks) is controlled by the human. RLHF for LLMs can also be viewed as language- conditioned RI, although Shen et al. (2024) find that reward models in this context often fail to distinguish nuances in task descriptions.
Language may be interpreted as a form of feedback in the reward-rational choice setting (Jeon et al., 2020); where an utterance implies a set of trajectories compatible with the utterance (grounding) and the human can choose an utterance that maximizes the probability of a desired action (i.e., the human is assumed to be specific which leads to pragmatic reasoning). This can be used to infer reward which may have caused the human to provide the utterances they did provide. A similar approach is followed, e.g., by (Sumers et al., 2022b).
An alternative way of incorporating language leedback is by extracting more structured forms of feedback from the language. An example is to use sentiment analysis to extract information about the reward function from language foedback, as proposed by Sumers et al. (2021).
Language feedback can also be interpreted by an LLM directly, without learning a reward model. This is demonstrated in the concurrent works by Ma et al. (2024b) and Xie et al. (2024) who propose to learn a symbolic reward model in the form of langnage-model written piece of (Python) code. They use natural language feedback to improve the reward model based on observations of the agent's behavior induced by the previous version of the reward model.
In addition to the way language feedback is used, the way it is elicited can also have an impact on the learning process. Sumers al. (2022a) study the relative merits of natural language instructions and descriptions of the desired outcome. They find that instructions tend to work better for low-autonomy settings, while descriptions are more effective in high-autonomy settings.
3.2.2 Supplementary Classes
This section introduces two feedback classes, e-stops and importance, that can be used in conjunction with primary class to learn reward model. These classes are not sufficient to learn reward model on their own but can supplement a primary feedback type.
E-Stops Emergency stops (e-stops) (Ghosal et al., 2023) are an active type of feedback. In this type of feedback, the human may intervene with the agent's behavior by stopping it, i.e., they may choose to stop the agent's current trajectory at any point, for example to prevent the agent from causing harm. This is closely related to interventions but, in contrast to those, e-stops do not suggest an alternative action. The defining fcatures of c-stops are that the human passively observes the agent's 's behavior (involvement), gives absolute feedback on single instance (arity) on the instance level (abstraction), and does so implicitly (explicitness) as side-effect of regular interaction. The intent is literal due to the implicit nature. For the purposes of intervention, the human usually observes the full episode (granularity). Due to the small amount of infrequent information they provide, e-stops should only be considered as supplementary feedback type.
E-stops are primarily intended to prevent bad behavior and only implicitly convey information about the correct behavior. This interaction and the arising incentives have been formalized in the form of the "off- switch game" by Hadfield-Menell reward-rational feedback, where the 'off' choice maps to the trajectory with the robot remaining still after the off switch has been triggered (see Section 5.1.1). Kahn et al. (2021) demonstrate that robot can learn to navigate using such feedback.
al. (2017a). Jeon al. (2020) propose to interpret this as form of
Importance Another form of supplementary feedback may come in the form of importance labels, commu- nicating which parts of the observation are important for the objective, e.g., by marking salient regions in an image representing the agent's final state or important words in the language model's response. Its defining features are that the importance information itself does not contribute towards generating behavior samples (observed involvement), is of descriptive intent, and is given explicitly (explicitness). Granularity, arity, and abstraction may vary depending on the primary feedback type. Since importance feedback
18

---

Published in Transactions on Machine Learning Research (06/2025)
needs base task with respect to which the importance is defined, it cannot be used on its own but is rather supplementary type of leedback.
One way to convey this information is by labeling salient parts of a visual input. This is explored by Guan et al. (2021), who augment pairwise comparisons with manually annotated visual saliency maps, informing the algorithm which parts of the visual input contributed to the decision. They leverage these annotations for data augmentation by assuming that random perturbations to irrelevant (non-salient) regions do not impact the human preferences. Basu et al. (2018) take an even more direct approach by combining comparative feedback with direct feature queries, ie., asking the user which feature is important for inferring the reward.
3.2.3 Representation-Speeii Classes
While the previous classes of feedback types are all aimed at directly learning a reward function, there are also classes of feedback types that do not. directly learn a reward function but rather help to learn a better representation.
Feature Traces While many approaches either rely on hard-coded features or learn a model entirely end- to-end, it is also possible to actively clicit new features from human feedback. Feature traces were proposed by Bobu et al. (2022) as approach to actively learn new relevant features. This type of feedback relies on a human operator to demonstrate a behavior in which a certain feature of interest, such as the distance to a sensitive object, monotonically increases or decreases, o.g., by moving a robot arm closer to the object. They make it possible to extend the set of features once the current set can no longer adequately explain the human leedback supplied through another type of leedback. The defining characteristics of feature traces are that they are of descriptive (intent) and explicitly (explicitness) given in an active manner (involvement) for single (arity) segment (gramularity) on instance-level abstraction.
Feature traces are strongly related to inter-temporal preferences (Section 3.2.1) since both types rely on changes in feature or reward values in the course of a single trajectory. Bobu et al. (2022) propose to learn from feature traces by leveraging a Bradley-Terry model to learn the feature values, similar to other approaches that use such a model to learn reward values. Similar to importance feedback, feature traces rely on another type of foedback to actually make use of the learned features and is, therefore, a purely supplementary form of feedback. For instance, Bobu et al. (2022) use intervention feedback to train a reward model on the set of features derived using feature traces.
Similarity Queries Similarity queries are a feedback type aimed at learning representation conforming to a notion of similarity and difference in the trajectory space. That aim is closely aligned with that of feature queries, though the actual queries are more similar to comparisons. The queries consist of triples of trajectories, with one anchor and two alternatives, for which the human has to decide which pair is more similar. Responses to similarity queries are given on observed behavior (involvement) with ternary arity, descriptive intent, and explicit foedback (explicitness), while the granularity and abstraction may vary. This type of feedback was first introduced by Bobu et al. (2023), who used it to learn representations for reward learning.
3.3 Initializations
Some modes of communicating reward functions are not interactive nor online and, therefore, do not directly fit within the scope of this survey (Section 1.3). However, since these are often used to initialize reward function for later refinement with some of the previously discussed interactive feedback types, they are still worth mentioning.
Initializations are most commonly given by examples of successful task completions, cither in the form of terminal or goal states (Xie et. al., 2018), expert demonstrations (Tbarz el al., 2018; Fu el al., 2018a; Palan et al., 2019; Lee et al., 2021b; Bıyık et al., 2022a; Abramson et al., 2022; Huang et al., 2023), demonstrations with success labels (Du et al., 2023), or ranked demonstrations (Brown et al., 2019). It is even possible to infer some human preferences by the state of the environment alone, e.g., assuming the parts of the initial environment that are under the user's control are largely set up in accordance with the user's preferences
19

---

Published in Transactions on Machine Learning Research (06/2025)
(Shah et al., 2019; Lindner et al., 2021a). Since offiine initialization is not the main focus of our survey, we do not cover these works in detail. We refer the interested reader to literature on inverse RL (Arora & Doshi, 2021) for further details on learning from demonstrations in particular.
3.4 Choice of Feedback Type
The best leedback type is not always clear and may depend on the task, the user, or the agent. Tt may also change over time the agent learns more about the task. Table may serve as a starting point select set of feedback types which may be applicable based on the possible user interaction and expertise, e-g.- by the desired gramularity, level of abstraction or involvement. This choice may be informed, among other features, by the task complexity or time horizon: Jain et al. (2015) compare purely passive (re-ranking basod) with active (slight trajectory improvements) feedback and conclude that the former is usually preferred by humans when the action space is large while the latter is preferred for complex tasks. Similarly, Sumers et al. (2023) empirically demonstrate that language is a more effective teaching modality than demonstrations for complex concepts. The relevance of the time horizon is highlighted in the work by Sumers et al. (2022b), which suggests that instructive feedback is more efficient for short time horizons and descriplive feedback for longer horizons.
In addition to these static choices, Section 4.1.2 will discuss how to choose feedback type adaptively. It is also possible to let the user choose the leedback type, as demonstrated by Jain et al. (2015) who let the user choose between re-ranking and improvement. Jeon et al. (2020) propose to use this choice of feodback type itself as a source of information ("meta-choice").
3.5 Combination of Feedback Types
In addition using any of the previously described feedback types in isolation, combining multiple feodback types is both possible and often advantageous. There are three main ways combine feedback types: (a) a two-phase setup, consisting of initialization and refinement, (b) integrating primary feedback type with a supplementary one; and (c) merging multiple primary feedback types.
The two-phase setup can be used to either initialize the reward model from offline data or to learn a representation that improves later reward learning. A common approach involves using one feedback type, typically demonstrations (see Section 3.3), to initialize the reward model, subsequently fine-tuning this model with another type, such comparisons. This method is exemplified by Ibarz et al. (2018), who combined demonstrations and pairwise comparisons. For more detailed discussion on feedback types suitable for initialization, we refer to Section 3.3. Alternatively, a representation-focueel type of feedback might be employed initially (Section 3.2.3) to cultivate a superior representation, followed by the application of a primary feedback type for reward model learning.
Combining a primary feedback type with a supplementary one can be beneficial make the most of the available user interactions. Supplementary feedback, while not sufficient for reward model training by itself, often be collected cheaply or side-effect of other interactions, making it valuable addition to the primary feodback type. We refer to Section 3.2.2 for discussion on supplementary feedback types.
Finally, combining multiple primary feedback lypes can be beneficial to capitalize on the strengths of each. For instance, Koppol et al. (2020) combine informative and demanding queries with less-informative and less-demanding ones to achieve a balance between cognitive load and informativencss. Similarly, to en- hance expressivity, Mchta & Losey (2023) combines demonstrations, pairwise comparisons, and corrections, allowing users to select their preferred type of feedback.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: An overview of query selection strategies used in RLHF approaches.
References
Factors
Uncertainty
On-policy
Query Simplicity
Trajectory Quality
Query Diversity
Query
Data
Cost.
Daniel et al. (2014)
Probability of improvement
Expected improvement
Upper confidence bound
Christiano et al. (2017)
Ensemble variance
Sadigh et al. (2017)
Volume removal
Biyík et al. (2024)
Volume removal
V
V
Wilde et al. (2018)
Feasible space reduction
K
Ibarz et al. (2018)
Random
Mindermann et al. (2018)
Information gain
Cui & Niekum (2018)
Information gain
Racca et al. (2019)
Entropy
Biyık al. (2019)
Mutual information Information gain
Bryık et al. (2020)
Reddy et al. (2020)
Ensemble-averaged KL-
divergence to mean output
Novoseller et al. (2020)
Dueling posterior sampling Maximum regret
Wilde et al. (2020; 2021)
Lee et al. (2021b)
Ensemble-averaged entropy
Lindner et al. (2021b)
Information gain
Katz et al. (2021)
Posterior sampling
8
Myers al. (2023)
Expected value of information
Bryık et al. (2024)
Mutual information
Dwaracherla et al. (2024)
Double Thompson sampling
Hu et al. (2024b)
Random
4 Label Collection
Agent
Environment
Action
Policy
Dynamics
State $t+1
Action az
Reward f++1
Query q:
Reward Model
Labeler
Label 4
Figure 3: RLHF diagram highlighting components discussed in this section.
In this section, we explain how preference data can be collected for training reward model, independent of the specific type of feedback used. We start by overviewing the active learning problem posed by query selection, e.g., how queries can be generated for a given query type and how even the type of feedback itself can selected. Following this, we discuss issues arising in such human-computer interaction.
4.1 Active Learning
We first discuss query generation and selection for a given feodback type, then the extension of these tech- niques to the choice of feedback type.
21

---

Published in Transactions on Machine Learning Research (06/2025)
4.1.1 Query Generation and Selection
One core problem that needs to be tackled in RLHF is that of learning about the hurman's preferences. This problem shares some similarities with the active learning setting since the agent can actively query a human teacher about those preferences. However, in contrast to standard active learning, which usually assumes a supervised learning setting, in RLHF, the agent needs to solve this problem in the context of RL. This means that it can both influence the distribution of the data (i.e., the transitions) and decide which data should be labeled.
As the RL agent is trained and its learned policy changes, the trajectories it generates will naturally evolve. Most work directly uses the trajectories obtained during RL training for preference learning. Using such trajectorics, candidate queries are often generated randomly. However, for pairwise comparisons, a more efficient approach is to generate queries by exploiting preference transitivity (Hwang et al., 2023). Alterna- tively, the agent can also generate trajectories specifically to be used for querying (not necessarily for RL training), possibly with learned transition model to limit the sampling cost (e.g., Reddy et al. (2020); Liu et al. (2023b)). This kind of active generation of queries can possibly lead to more informative ones.
In order to efficiently learn a suitable reward model, the agent must generate and select queries (Line 5 from Algorithm 1) so that it can quickly learn a good stratogy using those queries. This selection is performed via a criterion, usually called acquisition function", which allows the queries to be compared. Although most existing work in RLHF uses an acquisition function that provides some measure of uncertainly (e-g. about the learned rewards), which is arguably one of the most important factors in active learning, an efficient acquisition function (sce Table 3 for works dedicated to improving query selection) may need to include various additional other aspects, such as: on-policy dala, query simplicily, trajeclory qualily, query diversily, or query cost, which will be discussed one by one in the following. As a side note, interestingly, as highlighted by Habibian et al. (2022), the queries asked by an RL agent may also reveal its current reward learning stage.
Uncertainty
This factor usually corresponds to epistemic uncertainty (Hillermeier & Waegeman, 2021), which represents how uncertain the agent is about the ground-truth reward function. Epistemic uncertainty can be contrasted to alcatoric uncertainty, which describes inherent stochasticity in the system. While the latter cannot be fully eliminated, the former can be reduced with additional queries. Uncertainty is usually one of the most important aspects to consider when deciding which query to ask. It is usually represented either as a probability distribution (i.e., belief) in Bayesian approaches or using an ensemble of reward networks to approximate this belief. However, other representations are also possible, e.g., using the rocently-proposed epistemic neural network (Osband et al., 2023).
With a belief representation, various classic acquisition functions have been considered in the Bayesian framework, such as the probability of improvement, the expected improvement, or the upper confidence bound. For instance, Daniel et al. (2014) compare those three critcria in a robotic domain and observe that the latter yield the best performance, but a high cost. in terms of number of queries, while the first asks the fewest number of queries, but with slightly lower asymptotic per- formance. Other alternative criteria have been considered, such as volume removal (Sadigh et al., 2017; Basu et al., 2018; 2019) or information gain (Mindermann et al., 2018; Bryık et al., 2019; 2020). The volume removal criterion uses the minimum volume of the hypothesis set removed by an answer to a query as the acquisition function and has been shown to be effective in practice. However, Bıyik et al. (2019) show that volume removal has uninformative global optima and argue that the practical effectiveness is due to non-convexity leading to local optima that are informative. They also show that optimizing for information gain has the same computational complexity while avoiding this flaw. One drawback of these Bayesian approaches is that they require maintaining a distribution over reward functions (e.g., using Gaussian processes (Daniel et al., 2014) or simpler probability distributions, such as Gaussian distribution, but with a linear reward model) and, there- fore, may not be suitable for more complex domains due to the computational complexity or the strong assumption about the reward model.
We fallow the terminology used in Bayesian active learning.
22

---

Published in Transactions on Machine Learning Research (06/2025)
When using ensemble instead of direct belief representation, these criteria for epistemic un- certainly reduction correspond to measures of disagreement within the ensemble. Previous criteria could possibly be applied, but one popular candidate is the variance of the ensemble outputs (Lee et al., 2021b; Metcalf et al., 2023; Gleave & Irving, 2022) or equivalently its standard deviation (Eberhard et al., 2022).
The average entropy of the ensemble outputs (e.g., when assuming a Bradley-Tere model for the human answers) has also been used (Lee et al., 2021b;a; Park et al., 2022). However, note that it does not quantify epistemic uncertainty but rather the alcatoric uncertainty in the human's answers, as provided by the response model of the human. Therefore, this criterion may not be suitable in the RLHF setting since it amounts to focusing on the queries for which an answer is expected to be the most random (according to the Bradley-Terry model). By definition of this model, the segments in the pairwise comparisons are the most similar in terms of returns and are, therefore, the hardest to answer for the human.
In contrast to those acquisition functions that lead to deterministic query selection, sampling-basel approaches have also been studied, from pure random selection (Ibarz et al., 2018) to Thompson sampling (Katz et al., 2021) and its variants (Novoseller et al., 2020; Dwaracherla et al., 2024). Recently, in the context of fine-tuning LLMs with pairwise comparison queries, Dwaracherla et al. (2024) shows that using the relatively novel epistemic neural network (Osband et al., 2023), double Thompson sampling (Wu & Liu, 2016), which naturally favors better elements to be compared, performs well experimentally.
In addition to epistemic uncertainty, one may also take the outcomes into account to select queries, that is consider utilities (i.c., returns or expected returns in RL). In Bayesian sctting, this leads to acquisition functions such as expected value of information (Myers et al., 2023) or information gain over return differences (Lindner et al., 2021b), while in a non non-Bayesian setting, the notion of regret, which measures the difference of performance between a policy optimal for the ground-truth reward function and a policy optimal for a learned reward function, can be used (Wilde et al., 2020).
Finally, it should be mentioned that naturally approaches with theoretical guarantees usually also use uncertainty as the main criterion for label collection (e.g., Novoseller et al. (2023), see Section 7 for more details).
On-Policy Data Only focusing on uncertainty is likely insufficient or inefficient in RL because the previ- ous methods may focus on choosing queries to identify the reward function as precisely as possible uniformly on the whole state-action space. However, it may be important to favor more on-policy trajectories to guarantce the relevance of the generated queries for the current policy, assuming that the behavior policy stochastic (i.c., the current stochastic policy in SAC or the current determin- istie policy with some noise, such in DDPG). Indeed, improving reward learning in state-action regions that may never be visited with the current policy would lead to wasteful queries (Lindner et al., 2021b). One simple approach to ensure that the data is more on-policy is by simply sampling from the current policy (Cui & Niekum, 2018) or favoring more recently-generrled trajectories (Hu et al., 2024b).
Query Simplicity Selecting queries only based on their informativeness may lead to queries that are hard for a human to answer, which is, for example, the case for the average entropy. The case of answering a query is important to alleviate the cognitive load of the human oracle. Some work specifically takes this aspect into account, for instance, by considering the similarity of consecutive queries (Racca et al., 2019) or the information gain. For this latter criterion, Bıyık et al. (2019) show that in contrast to volume removal, naturally leads to queries that are casier to answer for a human because information gain can be increased when the uncertainty in the human answer is lower.
Trajectory Quality Most approaches directly use the trajectories generated during RL training. Especially early in training, these can be very bad with respect to the ground-truthe reward function. In addition to that, they can be irrelovant or even contradictory for a given task (Katz et al., 2021). Building queries on such trajectories may lead to unnatural queries for human respond to, such as comparing very bad trajectory with an irrelevant one. Katz et al. (2021) measure trajectory
23

---

Published in Transactions on Machine Learning Research (06/2025)
quality by optimizing over sampled reward functions. Similarly, Cui & Nickum (2018) generate trajectories using optimal policies for reward functions sampled from the current Bayesian belief.
Query Diversity When asking many queries (in batch, in sequence), the diversity of the queries becomes especially crucial to avoid asking redundant queries. Most work (Christiano et al., 2017; Lee et al., 2021b; Verma & Metcalf, 2024) follows very myopic approach: Queries are often selected from a usually randomly-generateed set of potential queries, and sequences of queries are not really coordi- nated. While some work exists that specifically tackles the selection of a batch of diverse queries (Bryık & Sadigh, 2018; Biyık et al., 2024), the latter is rarely considered due to its computational intractability. Indcod, planning ahead a sequence of queries would amount to solving a sequen- tial decision-making problem under uncertainly over combinatorial action space (i.e., the set of possible queries). For diverse batch querying, previous work considered using clustering methods such as k-medoids (Bıyık & Sadigh, 2018) or more recently determinantal point processes, which define probability distributions that promote diversity (Bryık et al., 2024). In addition to approaches focused on selecting diverse queries, another promising research direction involves generating inher- ently diverse and informative behaviors to select from. This can be achieved with techniques such as reguaay--e RL (Ziebart et al., 2008; Haarnoja et al., 2017) or generative methods based on flow networks (Bengio et al., 2021). The latter is particularly relevant for RLHF for generative models such LLMs (Hu et al,, 2024a). To our knowledge, this dircction has not yet been explored in the context of RLHF.
Query Cost The cost of generating queries may also be an important factor if the interaction of the human is live since it may not be practical to let the human wait before showing any queries (Bıyık et al., 2024). In that case, it may be more important to quickly show some relatively good queries instead of computing the most informative ones, Although this factor may not translate directly into an acquisition function, it may influence the choice of the acquisition function and its implementation in a given problem.
Since various different acquisition functions have been considered, some effort (Lee et al., 2021b;a) has been made to compare them. Generally speaking, uncertainty-based criteria (e.g.. variance or average entropy) seem to often perform better empirically compared to random selection, a query diversity- criterion alone or combined with an mncertainty-base criterion. Surprisingly, random selection has been shown to perform competitively in some cases (Christiano al., 2017; Ibarz et al., 2018). Thus, a better understanding of which acquisition function should be preferred in which situation domain is still an open question.
In addition, combinations of different criteria have naturally also been evaluated. For instance, Reddy et al. (2020) use four acquisition functions (high uncertainty, high novelty, high reward, low reward) in parallel. This approach has also been validated in a 3D environment (Rahtz et al., 2022). A more sophisticated approach consists of considering a portfolio of acquistiinn functions and learning to select them using a multi-armed bandit approach (Hoffman et al., 2011).
Various extensions the basic setting have also been investigated. In the context of multiple human labelers, the issue of selecting reliable teachers to query arises (Daniels-Koch & Freedman, 2022). Assuming all teachers have the same preferences, this can be modeled for pairwise comparisons by incorporating a rationality coefficient B into a Bradley-Terry model and estimating this factor:
N
max II + exp( -
where higher 3 corresponds to a more reliable human (see Section 5.1.2). The setting in which this assumption does not hold, i.c., the labeler's 's reward functions differ (a setting already considered in inverse RL (Choi & Kim, 2012)), has also been studied recently (Siththaranjan el al., 2024; Xue et al., 2024; Dong et al., 2024b; Myers et al., 2021; Bakker et al., 2022). Interestingly, a noisy oracle may sometimes provide more information than a completely reliable oracle. For instance, in Eq. (3), the probability of crroncous answers given by the noisy oracle is related to how much segment is preferred to the other one, and Chan et al. (2021) show that such Boltzmann-rational behavior can be more informative for reward inference than
24

---

Published in Transactions on Machine Learning Research (06/2025)
perfectly rational behavior. In contrast, only a binary preorder over segments can be inferred from the answers of a reliable and deterministic oracle, which may not be enough to recover the true reward function.
Another notable recent work is that by Ellis et al. (2024) who raise the issue of identifiability of the ground- truth reward function: many reward functions result in the same optimal behaviors. The authors propose a framework that enables the generation of acquisition function for various definitions of reward similarity, such the one discussed in Section 5.3.
4.1.2 Adaptive Choice of Feedback Type
In addition to selecting queries within a given feedback type, it is also possible to actively select the feedback type itself (Fitzgerald et al., 2022). The best choice of feedback type can depend on many factors, such as human rationality as well as task-dependent factors, some of which may change during the labeling process. Ghosal et al. (2023) formalize this setting as one in which we try to select a feedback design (or feedback type) x out of the space of possible designs such that the expected information gain over the distribution of reward functions is maximized for the next human response. Concretely, the goal is to choose a feedback design by means of
T argmax Bey,-((all) [PIL(((() Ci. x)) [(f))] x
where C is the human response to a query defined by x and P(0) is the prior distribution over reward functions.
Ghosal et al. (2023) find that the most informative feedback type depends on the (type-dependent) rationality of the human labeler (see Section 4.2.1). More preciscly, it is shown that the most informative feedback depends on the rationality factor, e.g., while demonstrations are more informative than comparisons when the human is highly rational, comparisons should be preferred in less-rational settings. Given that this rationality might change due to factors such as fatigue or an individual labeler's capabilities, this suggests that adaptively adjusting the feedback type during the labeling process may be worthwhile. Further study of this relationship a promising area for future work.
4.2 Challenges of Human Labeling
This section explores the label collection process, which follows after query selection. This task intersects with several related disciplines, especially within the social sciences, as it encompasses the design of interactions to facilitate informative query responses. A prominent field in this area is psychometrics (Furr, 2021), which focuses on measuring psychological attributes, including preferences. Similarly, survey research (Fowler, 2013) dedicated to developing techniques for gathering information from individuals via surveys. Human- computer interaction plays a significant role as well, investigating the design of user interfaces tailored for preference elicitation (Pommeranz et al., 2012). Moreover, preference label collection is also necessary for discrete choice experiments within health economics (Ryan et al., 2008), where it is used for the assessment of service values.
4.2.1 Taccologg-Awarr Preference Elicitation
Understanding human psychology is essential for effective preference elicitation in RLHF systems. Human decision-making complex, often diverging from traditional rational choice models due to cognitive, social, and emotional factors. This complexity reliability of choices based on the order of queries. This section overviews these phenomena, exploring how constructive preferences, biases, framing effects, and social interactions shape the observed choices. Recognizing and addressing these psychological underpinnings is key to developing more accurate and reliable systems. In this section, we will discuss various psychological phenomena, such as cognitive biases and response biases, and related effects (fallacies, biases, heuristios, psychological phenomena impacting decision- making processes), which may falsify labels by adding systematic bias or noise.
exemplified by phenomena like fatigue, which can affect the
Preference learning methods typically assume the existence of inherent, stable preferences that can be elicited through querying. Contrary to this assumption, psychological research, such as the work by Lichtenstein
25

---

Published in Transactions on Machine Learning Research (06/2025)
& Slovic (2006), indicates that preferences are often constructed during the elicitation process and may vary with the method of elicitation or over time. This suggests that the feedback type not only affects elicitation's effectiveness but also shapes preferences. Systematic biases, noise, and other psychological factors may influence observed choices, challenging the traditional models of human choice used to infer latent utilities (see Section 5.1). The elicitation method, query presentation, and context thus play a critical role shaping measured preferences, compounded by cognitive biases and irrationalitics.
The inftuence of psychological phenomena on preference learning has been well-documented in the litcrature, especially within the context of explicit preference elicitation for recommender systems. For instance, Tran et al. (2021) provide thorough discussion of the relationship between psychology and recommender systems. Similarly, Atas et al. (2021) review how preference construction is influenced by cognitive biases, personality traits, and emotional states in recommender systems, discussing effects like serial position, framing, anchor- ing, choice overload, and preference visibility. In more specialized discussion, Mandl al. (2011) focus on cognitive biases in the context of consumer docision-making and its interaction with recommender systems. Finally, Kaufmann el al. (2023) link these psychological aspects to RLHF, discussing the common practice of using syuthetic instead of real human feedback for algorithm evaluation and highlighting the limitations of that approach. They further discuss challenges posed by real human feedback, many of which are related to the concepts discussed in the following paragraphs, as well as the opportunities provided by integrating psychological insights into RL.F systems.
Constructive preferences are closely related to framing effects, which refer to changes in elicited preferences based on how tasks or alternatives are described, even when these descriptions are essentially equivalent. For example, presenting choice as a loss versus gain can lead to different decisions despite identical outcomes. Moreover, serial position effects, commonly known as primacy and recency effects, also play a significant role. These effects describe the tendency for the beginning and end of an experience to influence subjective experience disproportionately. This phenomenon becomes particularly relevant in scenarios like video choices, where the initial or concluding segments might disproportionately affect preferences. Atas et al. (2021) discuss both of these effects in the conlext of recommender systems.
Ordering effects pose another challenge in preference elicitation, where the sequence of queries can affect responses. Day et al. (2012) outline several factors contributing to these effects: institutional learning, changing preferences, and varying levels of cognitive effort. Institutional learning involves gaining familiarit with the task and feedback type, which can enhance labelers' expertise and, consequently, the accuracy of their responses. However, due to the constructive nature of preferences, these preferences may evolve during the elicitation process, leading to changing preferences. This evolution might also be influenced by anchoring effects, where previously seen instances bias responses. Furthermore, cognitive effort levels can fluctuate due to factors like fatigue or boredom. This is closely related to choice overload, a form of fatigue from excessive choices, as discussed by Atas et al. (2021) and bounded rationality, as explored by Chen et al. (2013). In such scenarios, labelers might opt out of making a choice when overwhelmed by options. Bounded rationality refers to the limitations in human decision-making capabilities, particularly when processing large amounts of information, which aligns with the concept of choice overload. To address these challenges, studies like Bryik et al. (2019) and Zhang et al. (2022) propose methods to reduce cognitive effort in responding to queries. BryIk et al. (2019) focus on posing queries that are straightforward for humans to answer, while Zhang et al. (2022) enhance the human evaluation process by presenting queries in user-friendly format.
Multiple labelers often collaborate on the same task in preference elicitation, as studied, e-g. by Barnett et al. (2023) and Daniels-Kodd & Freedman (2022). This collaboration may lead to another set of biases if they have the opportunity to exchange information. The exchange may either be direct or indirect through observing the system's predictions, which are based on the other labeler's 's feedback. Such interactions can affect their preferences through several mechanisms, as identified by Atas et al. (2021): anchoring effects, transfer of emotional states, and conflict avoidance. Anchoring effects, for instance, occur when a labeler's choices are influenced by the knowledge of others' preferences or system predictions, a phenomenon also discussed under the term preference visibility. This bias can lead labelers to align their preferences with the anchors they are exposed to, which is significant consideration in recommender systems. Understanding these biases is crucial for designing RLHF systems that mitigate the influence of labeler interactions on preference construction.
26

---

Published in Transactions on Machine Learning Research (06/2025)
The effects previously discussed stem from systemic biases in preference expression. In addition to these biases, choices may also be affected by noise. This is commonly discussed under the term stochastic rational- ity, where an agent's 's behavior is rational with respect to an unobserved random state. The reward-rational implicit choice framework, as introduced by Jcon et al. (2020), addresses this by integrating a rationality lactor 3 into the human choice model (see Eq. (3)). This factor's impact has been further examined by Ghosal et al. (2023) through synthetic experiments and user studies, demonstrating that accurately estimat- ing this type-dependent rationality coefficient can enhance learning performance and guide feedback type selection (see Section 4.1.2). However, practical method for estimating this factor remains challenge. While Ghosal et al. (2023) use calibration feedback with a known latent utility function for estimation, such an approach is not feasible for most tasks. In a related study, Daniels-Koch & Freedman (2022) investigate a scenario with multiple teachers, locusing on the agent's 's ability to select the most knowledgeable or ral rational teacher. Therefore, developing more advanced methods to estimate this factor, along with understanding its variability due to factors like fatigue or other ordering effects, presents a vital area for future rescarch in preference elicitation.
Finally, the quality of human feedback is biased towards factors that are easy to judge. Hosking et al. (2024) demonstrates that in the case of LLM fine-tuning, humans tend to favor asscrtiveness over factuality, since the latter is hard to judge without external assistance or resources, A similar phenomenon was observed in the control setting by Amodei et al. (2017), where the agent learned a behavior that looked good only from the camera angle that the human labelers had access to.
Incorporating psychological insights into the preference-learring components of RLHF systems is essential for optimizing their efficacy. A key area of focus should be research aimed at mitigating biases and harnessing cognitive aspects of preference formation. For instance, designing user interfaces that minimize framing effects and developing algorithms that account for ordering and serial positioning are crucial steps. In this realm, Metz et al. (2023) and Yuan et al. (2024) each propose a configurable user interface for studying various feodback types and their combinations. Additionally, the study by Krening & Feigh (2018) on the impact of leedback type, such as binary critiques versus action advice, on lask performance and labeler satisfaction highlights the significant role of feedback type in preference elicitation. Furthermore, the work of Pommeranz et al. (2012) in uscr-interaction design underlines the importance of having an expressive feedback type to increase user engagement.
The integration of these research findings into RLHF systems points to a clear need for a more multidisci- plinary approach. Drawing insights from related fields like behavioral economics and psychology can provide valuable methodologies and perspectives. Addressing irrational choice patterns and enhancing the quality of human feedback remain critical challenges. As we continue to develop and refine these systems, the fo- cus should be on creating robust frameworks that align learning processes with human behavior, effectively managing the inherent complexity and variability of human feedback.
4.2.2 Importance of Researcher-Labeler Agreement
High-quality labels are important for the final policy in an RLHF process. Early work fine-tuning language models using RLHF noticed a mismatch between the researcher's goals and the (paid) labeler's actual labels (researcher-labeler disagreement). Ziegler et al. (2020) note that researchers agreed with each other about 60% of the time (on 4-way comparisons, where random choice would result in 25% agreement), while agreeing with labelers only 38% or 46% of the time (depending on the task). Stiennon et al. (2020) attempt to reduce these disagreements by maintaining a hands-on relationship with the labelers, thereby ensuring high researcher-labeler agreement. Concretely, they provide ou-boarding with detailed instructions, keep an open channel of communication between researchers and labelers, and give feedback to the labelers. They evaluate the roscarcherlabellr agreement and reach an agreement rate of 77% + 2%.
Perfect labels are often impossible due to the inherently subjective nature of the task. Returning to the example given by Stiennon et al. (2020), different researchers agreed with each other in only 73% + 4% of the cases. Ouyang et al. (2022) also report the agrocment rates on different task (instruction finc-tuning instead of summarization) and find that labelers agree with each other in 72.6 + 1.5% of the time, after a screening procedure that, amongst others, selects labelers that agree with researcher labels. Preferences can
27

---

Published in Transactions on Machine Learning Research (06/2025)
additionally be inconsistent between feedback types, demonstrated by the findings of Bansal et al. (2024). which show that preferences inferred from ratings and rankings significantly disagree for both human and Al annotators.
The importance of quality does not trump the importance of quantity, however. Indeed, Stiennon et al. (2020) note that excluding low-confidence samples from the data set generally did not help with reward model training. This indicates that even though quality is important, larger quantity is still generally better.
The scale of the labeled data set required for effective training and refinement varies widely, impacting the quality of the resulting models. In practice, the required amount of data depends on many factors including domain, (ask complexity, and level of pretraining. For example, while LLM fine-tuning generally often requires tens thousands of samples due to the complexity of the task, at the same time can leverage the extensive prior knowledge of the pretrained LLM to reduce sample complexity compared other tasks of similar complexity. labels in smaller studies (Jain et al., 2015) to hundreds in more complex scenarios (Christiano al., 2017). Larger-scale applications may require thousands (Guan et al., 2021; Ouyang et al., 2022) even millions of labels (Abramson et al., 2022), each bringing its own challenges in ensuring label accuracy and consistency.
Studies have shown broad range in data
sizes, from tens of
This variability in data set size underscores the need for rigorous label quality control measures across different scales, In smaller data sets, each label carries more weight, making accuracy and precision critical. Conversely, in larger data sets, the challenge lies in maintaining consistency and mitigating systematic biases that might emerge from the sheer volume of data. The labeling setting varies in the surveyed works, from author-provided feedback (Kim al., 2023), over small in-person studics (Katz et al., 2021), to larger remote studies (Kim et al., 2023). Each setting provides unique challenges ensure high-quality labels.
Various works have suggested measures to improve label quality. Hagendorff & Fabi (2022) discuss the possible failure modes of the labeling task in more detail, for example, discussing systematic biases and conflicting motivation, and propose concrete changes to the training and evaluation methodology to alleviate these. Glaese et al. (2022) suggest providing labelers with multiple natural language rules and collecting preference labels for each rule individually to improve label quality. This is related to Bai et al. (2022b), who propose to generate feedback automatically based such set of rules and language model.
5 Reward Model Training
Agent
Environment
Action a
Policy
Dynamics
8t
State 8t+1
Action Reward fc+1
Query d
Reward Model
Labeler
Label 1
Figure 4: RLHF diagram highlighting components discussed in this section.
In this section, we delve deeper into the process of reward model learning, which we briefly touched on in Section 2.3. We will discuss various aspects associated with this topic, namely the various human foodback models, utility (i.e., reward) learning per se, different reward model inputs, and how to increase feedback efficiency.
5.1 Human Feedback Model
The basic premise underlying the majority of approaches in RLHF that human feodback is dircctly related the reward function be learned. To this end, the human feedback must first be captured in a sound mathematical framework that. establishes the connection to the reward function. On high level, one
28

---

Published in Transactions on Machine Learning Research (06/2025)
break down almost all feedback types in Section 3.2 to choice scenario: The human chooses one specific leedback option (label) from an available pool of possible feedback options (choice sets), which may be infinite". The query determines the explicit contents of the choice set. For example, if the query is to compare two trajectorics, then the choice set consists of all possible outcomes for these two trajectorics.
Assuming that human choices are not always optimal, one obtains a fruitful mathematical framework when focusing on the probability
IP (c is chosen C)
where C is the set of possible choices and e E C one explicit choice. For the RLHF scenario, where the agent asks queries qi and the human gives labels 4 as feedback (see Section 2.4), the choice set is specified by a function of the query. Formally, C = m(g) for some mapping m that maps a query & to the set of all possible candidate labels extractable from for the specific feedback type. For example, if the query is to rank a finite number of trajectories, then the choice set the set of all possible rankings that can occur for the trajectories involved.
With this view, can therefore place (4) in the RLHF context and write
P (label I is provided mm())
(5)
for the probability that a human labeler returns a label I from all possible candidate labels that can be extracted from a given query q. We explain next how this probability can be modeled and discuss various related modcling questions (e.g., human rationality, multiple humans, or Markov assumption). One could also recover the noiseless scenario if the latter probability distribution degencratod for all possible candidate label sets.
5.1.1 Boltzmann Distribution
Human choice models as in (4) have been studied for a long time in various scientific fields such as psychol- ogy (Thurstone, 1927), economics (Train, 2009), or behavioral science (Cattelan, 2012). Accordingly, there are many different choice models to resort to for (5), which, in some cases, are the same models, just under different names. A popular class of such human choice modcls assumes every choice option to be equipped with a (latent) utility We, which the human perceives in perturbed way. This perturbation is modeled by means of perturbation random variables that perturb the utility in an additive way, so that (4) becomes
IP (c is chosen (C) = P argmax Ue + Ec Ee) cEC
(6)
The translation for the RLHF setting for (5) is then accordingly
P (label / is provided m(q)) = 1
argmax ii + El lEm(g)
(7)
and we shall now stick to the RLHF translation from now on. These probabilities depend on the specific distributional assumptions that are made on the perturbation variables that only for specific cases lead to a closed-form of the right-hand sides of the latter equations. When stipulating standard Gumbel distribution for the perturbations, one always obtains closed form that is proportional to the exponential utility of the provided label:
P (label / is provided mm()) x exp(ur).
(8)
This is known as the Boltemann distribution that also appears in a perhaps slightly modified version in various different subfields of machine learning (MI.) and statistics. When restricting to discrete (choice) sets for m(g), this distribution is also known as the multinomial logit model (Train, 2009) or Gibbs measure (Georgii, 2011), and as the Sradley-Terry model (Bradley & Terry, 1952) when the choice sets consist of pairs. All of these
3rhis point of view goes back the work of Jeon al. (2020).
29

---

Published in Transactions on Machine Learning Research (06/2025)
also have link to the Plackett-Luce model (Luce, 1959; Plackett, 1975), which is probability distribution on the space of total orders or rankings (see Alvo & Yu (2014) for details).
This model is often used for various reasons. A particularly compelling reason is the closed analytic form, which in turn makes it possible to obtain a closed form for the gradient with respect to the utilities. Another reason is that this model satisfies Luce's axiom of choice (Plackett, 1975), which requires the probability of choosing an option from a pool of choice options not being affected by the presence or absence of other is ensured, which, however, might be challenged humans likely do not make fully rational decisions (see Section 4.2.1). Jeon et al. (2020) show that the usage of the Boltzmann distribution is justified by the principle of maximum entropy. More preciscly, they show that it is the maximum entropy distribution over choices for a so-called satisficing human decision maker, i.., one who is making a choice with an optimal reward in expectation up to some slack e >> O.
options in the pool. In this way; coherent decision-makiin
To build the bridge between reward learning and the modcling of human feedback, the Boltzmann distribution can be used by assuming that. the utilities can be represented a function of the reward function, usually featuring the return of a trajectory. More specifically, one assumes grounding function G that maps choice options (or labels) to the set of distributions over trajectories and sets the utility of label I as
iy == E0 [((T))
(9)
Note that uy depends on the return R so that we also may use (R(R) to emphasize this dependency. For the common case of pairwise trajectory comparisons, where for two trajectories Ti, 72 we obtain for the possible labels 1 E (T) > T2, < T2} the respective utility by using the projection onto the preferred trajectory as the grounding function G. Accordingly, the utility of the label represents essentially the utility of the preferred trajectory of that label, i.c., or 72 in this case. As another example consider the case of c-stops feedback (see Section 3.2.2). Here, the possible labels / provided by the user are STOP, and CONT encoding the stopping at time t or the continuation of a trajectory. For the grounding function, one can define
G(I) = T, To: ... Tt. [ = STOP,,
= CONT,
where Toir is the trajectory of T trimmed to the stopping time 4, and T, is the action-state pair al time t. Table Jeon et al. (2020) provides an overview of the different grounding functions that lead to a specific feedback type. It is worth noting that one can also easily find a grounding function for the feedback type of (partial) order over trajectories as considered, for instance, by Myers et al. (2021). Moreover, one can generalize this modeling approach by using (partial) segments instead of trajectories.
Although this general human feedback model has been much in use and shown to be useful for the sake of human aligument, it is not without its critics (see Lindner & El-Assady (2022) or Section 3.2.1 in Casper et al. (2023)). This has led to different adaptions of the general model based on the Boltzmann distribution that will be discussed in the following. Moreover, we will also conciscly review other human feedback models that have been in use besides the Boltzmann distribution, discuss relevant work on the consequences or robustness of human feedback model misspeciifiattion, and highlight contributions on varying the standard assumptions on the nature of the human feedback.
5.1.2 Human-Specific Rationality Coefficient
The Boltzmann distribution in (8) can be extended by a rationality coefficient B E [0, oo) that reflects the precision of the human labeler:
PP (label is provided m(q)) P (1 (-arginax 8 + E x exp(B- u).
(10)
IEm(1)
The higher 3, the more (10) resembles a pointmass distribution modeling a highly rational human labeler that is always able to identify the option with highest utility. The lower B, the more it resembles a uniform distribution modeling highly irrational human labeler acting purely at random. Without this extension,
30

---

Published in Transactions on Machine Learning Research (06/2025)
the commonly considered Boltzmann distribution (or Bradley-Teri Terry model in the common case of pairwise comparisons) in (8) assumes a rationality coefficient of 1. Ghosal et al. (2023) show in their experiments that the estimation of this coefficient can indeed positively influence reward learning. However, estimation requires a calibration reward function, as the rationality coefficient is otherwise not identifiable (Bengs & Hiillermeior, 2020). Similar findings are shown by Daniels-Koch Freedman (2022), who model the rationality coefficient as query-dependent function that might differ for the human labelers (sce Section 5.1.6).
Another alternative the rationality coefficient for representing irrational humans is achieved by introducing query-independent error probability (Christiano et al., 2017). To be more precise, it is assumed that the human labeler only adheres to the Boltzmann distribution in (8) in 90% of cases and otherwise makes completely random decisions. This formulation is similar to Huber's contaminated model (Mu & Xiong, 2023).
5.1.3 Alternative Utility Notions
Knox el al. (2024) show that the Boltzmann model does not generally lead to an identifiable reward function using (9) by presenting three concrete scenarios for which identification is not possible. The root cause of the non-identifiabiiity is the usage of a trajectory's return as the utility in (9). They therefore suggest using trajectory's 's regret as alternative, which provably leads to idebti rewards.
A trajectory's regret is the negated sum of the optimal policy's advantage over each state-action pair in the trajectory. Empirically, it has been shown that this modification improves the alignment of the learned strategy with human preferences. The downside of this alternative is that regret depends on the unknown optimal policy. Recently, has also been suggested to consider Q-values of human policy as the utili- ties (Myers et al., 2023), while Holladay et al. (2016) used differences of cost functions that depend on the available choice set and the human's uncertainty.
5.1.4 Human Feedback Models Beyond Boltzmann
While the human feedback model based on the Boltzmann distribution is the most widely used model nowadays, other models have also been considered in the literature. In particular, for the probability in (4) other models such as the Thurstone model' (Wilson et al., 2012; Kupesik et al., 2018; Bıyık et al., 2020), the ridge-noise model (Schoenauer et al., 2014), the binary model (Sugiyama et al., 2012) or mixed forms thereof (Wirth et al., 2016) have been considered. Of these models, only the Thurstone model (Thurstone, 1927) has a similar interpretation the Boltzmann distribution based on perturbed utilities, only differing in the distribution of the perturbance random variables.
Link functions Another possibility, which is particularly popular in theoretical work on RLHFF (see Sec- tion 7), is the use of other funetions on the right-hand sides of Eq. (8) than the exponential function. The concept is primarily used for pairwise comparisons of trajectories. It essentially states that the probability of the result of pairwise comparison between two trajectories is the difference of their utility values under a so-called link function. More specifically, let {TI, r2} be the query to compare the trajectories TI and T2: then, assuming a link function /b + [0, 1], one models the probability in (5) for I representing a preference for Ti as
P (label I is provided m(g)) = P (5 > T2 12lm((rr, r2))) = D - Mrz).
(11)
For | representing a preference for T2, one proceeds similarly. The minimal assumptions on the link functions are that
(i) it is (strictly) monotonically increasing to take into account that trajectorics with higher utilities will also have a higher chance to be picked;
(ii) (x) = - Φ(-x) to ensure that TP (7 > 72 72)) = P(T x T2 (m)(),
4The probit model, used in Biyik al. (2020), corresponds to Thurstone's Case V.
31

---

Published in Transactions on Machine Learning Research (06/2025)
Note that the second property implics ((0) 1/2 that trajectories with the same utility also have the same chance of being selected. Any cumulative distribution function of symmetric continuous random variable fulfills these two conditions. The two most common link functions that both fulfill the conditions
are the linear link function given by
Φ(x) = max (0, min {1, 1/2 x1)
and the logistic link function given by
((r) 1 + exp(-x)
Both are cumulative distribution functions: The lincar link function is the cumulative distribution function of continuous uniform distribution on 10. In contrast, the logistic link function is the cumulative distribution function of a logistic distribution with location parameter 0 and scale parameter 1. Moreover, both are intensively studied in theoretical approaches (see Section 7.1), and the latter leads to (8) (when restricted on pairwise comparisons) and is a special case of the softmax function.
Two-Staged Choice Model Bobu et al. (2020b) propose the Limiting Errors due to Similar Selections (LESS) model that is inspired by the attribute rule model suggested by Gul et al. (2014). It assumes a feature map for trajectories and (similarity) function mapping trajectory features and trajectories to integers and uses two-stage process for modeling the human feedback (or choice): First, choosing trajectory feature according to the Boltzmann distribution and then trajectory with the (logarithmie) similarity functions the utilities within the Boltzmann distribution. Their experiments show that this model can capture human feedback more appropriately than the standard Boltzmann distribution.
Generative Model Abramson et al. (2022) evaluate the usage of a generative model for learning from human preferences. More specifically, instcad of assuming some undorlying utility as in the Bradlcy-Terry model, they attempt train a model to generate the human feedback (inter-temporal preferences in this case, see Section 3.2.1) and directly interpret this feedback reward. However, they found that this empirically does not perform well as the inter-temporal Terllyy-TTrryy model.
5.1.5 Misspecification
The human feedback model may be misspecified in various ways. Milli & Dragan (2020) investigate the problem of misspecifying the nature of human feedback that can be either literal or pedagogical. The former means that the human gives targeted feedback for solving the actual RL problem, while the latter means that the human gives targeted feedback that is deemed helpful for the learner. They show theoretically and empirically that the case of a learner assuming a pedagogical feedback with an actual literal human always performs worse than the reversed case, i.c., a learner assuming a litoral feedback with an actual pedagogical human.
A related question studied by Freedman et al. (2021), namely, what if the learner makes incorrect assump- tions about the choices from which the human selects its feodback? They consider different types of such choice set misspecification and show that depending on the lype of misspecification, the performances might vary drastically, even leading to no losses at all in some specific cases.
In the field of inverse RL, the general question the robustness of reward learning in terms of a misspecifiod human feedback model is theoretically investigated by Skalse & Abate (2023). It turns out that the optimality model not robust with respect to any misspecification, the Boltzmann model is robust for quite a range of specific misspecifications, and the degree of robustness of the maximal causal entropy model lies between the latter two. Even though these results are primarily derived for inverse RL, they also have similar immediate implications for RL.F.
5.1.6 Diverse Preferences
One potential issue with the RLHF framework is that it does not specify whose preferences to align to. is common to request feedback from multiple labelers in a crowd-sourcing setting, in which case the different
32

---

Published in Transactions on Machine Learning Research (06/2025)
labelers may disagree. There are two main ways to deal with this challenge: Either trying to learn each labeler's preference separately, or trying to learn a model of the group's mean preference.
Bakker et al. (2022) investigate the first option by proposing to learn multiple reward functions, which can then be aggregated in arbitrary manners and even be used to find consensus among people with different preferences. The second is more commonly used, however: Xue et al. (2024) learn a single reward function from multiple humans who may give diverse and inconsistent feedback, aiming to stabilize learning in spite of these inconsistencies using regularization, et al. (2024) try to estimate the correct preference for pairwise trajectories directly by combining the users' expressed preference labels instcad of assuming individual reward functions. As a middle-ground between the single reward function and multiple ones, Myers et al. (2021) propose to learn a multimodal reward function that captures multiple people's preferences and use a mixture model of Plackett-Luce models to represent the feedback more accurately.
consistency constraint, and ensembling. Similarly, Chhan
With a stronger focus on the active retrieval of human leedback, Freedman et al. (2023) model the problem of selecting suitable human labeler as a variant of the multi-armed bandit problem (Lattimore & Szepesvári, 2020), which they call hidden-utility bandit. In this variant, the agent has in each decision round the choice between two options: (i) drawing a bandit arm, then recciving hidden arm-dependent utility, and finally observing an item, or (ii) querying a human to observe a preference between two sampled items and incurring a human--pecific query cost. The feedback mechanism all human teachers is modeled via a same Boltzmann distribution, differing only in their known individual rationality coefficients. The same modeling of human feedback is also considered by Barnett et al. (2023), who, however, use a Bayesian approach to determine which person should be queried in order to obtaín the most informative feedback in expectation. Daniels-Koch & Freedman (2022) investigate the rationality coefficient already considered in the previously mentioned work and model it as a query-dependent function that might differ for the human labelers.
When dealing with diverse preferences, aggregating them via the mean may not always be suitable. Instead, after learning the individual preferences, one needs to address the important question of preference aggrega- tion, which is studied social choice theory (Brandt et al., 2016). Thus, a recent. rapidly-developing research trend has started to investigate the tools developed in social choice in Al alignment (Conitzer et al., 2024; Sorensen et al., 2024; Mishra, 2023) in general and in RLHF (Dai & Fleisig, 2024; Zhong et al., 2024a; Park et al., 2024; Chakraborly et al., 2024b; Swamy et al., 2024; Lambert el al., 2023) in particular.
5.1.7 Relaxation of the Markov Assumption
Most works assume that the human feedback is given based on latent Markovian reward model, i.e., the return of trajectory 7 decomposes into a sum of independent rewards over state-action pairs (see (1)). Farly et al. (2022) relax this assumption by dropping the need for the Markov property, such that the instantaneous reward might depend on hidden states. Similarly, Kim et al. (2023) avoid the Markov assumption by using a transformer as the preference model. A similar effect may be achieved by learning a state representation with a recurrent network in which the rewards are Markov, similar to the approach taken by Hafner et al. (2023), but we are not aware of any work exploring this. Abramson et al. (2022) work in a non-Markovian setting as well by using memory- annooee- networks for both the policy and the reward model.
5.2 Utility Learning
After choosing a human model to relate feedback to utilities, we can use the observed feedback to recover the latent utilities. This utility learning can be reduced to standard supervised learning problem and, therefore, is commonly solved with the techniques of empirical risk minimization or Bayesian approaches, both of which will be discussed in the following.
5.2.1 Empirical Risk Minimization
The most prevalent variant for learning the reward function, already presented in Section 2.3, special case of empirical risk minimization. The general approach of empirical risk minimization for reward function
33

---

Published in Transactions on Machine Learning Research (06/2025)
learning, assuming underlying human feedback model with utilities as in (9) is to find the minimizer of labels 4 E { > T2, rf < m2) = m(q:) are the human's preference over the two trajectorics. For a given query yi we then obtain (2) as a special case of (12) by using the loss function:
((R,D) = = ((
(12)
where D (04, q:)] is the given data set of observed label and query pairs, R x Q C is suitable loss function with Q being the set of all possible label sets, and u. (R) denoting the utility (depending the return R) of the possible labels for the given label-query pair 4, m(g)). As illustration, consider the common case of pairwise trajectory comparisons where queries are pairs of trajectories gi = { r2h and
( mlq:)) = - log
1 + exp((mmmm))) (R) un (R))
=
+ exp(Ee. V. [R(T)] Br. [((T)))
where in the case of pairwise comparison, u. (.) (wx, (RI), (q:) (R)) and the grounding function G is the projection onto the preferred trajectory. This is the negative log- likelhod for the Boltzmann distribution for the observational pair (4,, m(q:))
For the entire learning process, a model class R is assumed for the reward function R. This model class is usually a parameterized class of functions, such as, for example, the class of linear reward functions (Katz et al., 2021)
(Ry(ss a) sp a)s (s, a) E
E S A
where S A - Rd is some state-action feature mapping. This entails that good features known linear combination of those features. Using such lincar
in advance such that rewards can be expressed models may lead to reward model misspecification. Studying this setting, Bobu et al. (2020a) propose to adapt the hyperparameter 3 in (3) to account for this issue.
Since the assumption of a linear reward model may be too strong in practice, most recent work based on non-linear models, especially using differentiable models, but other cases have been investigated as well. the latter case, for instance, decision trees have been considered learn an interpretable reward model (see Section 5.2.4). In the former case, simple multilayer perceptron (MLP) has naturally been considered, but more recent deep learning architectures are more commonly used in the recent literature. Thus, especially with partially observable domains, the reward network may be composed of state-action encoder followed by fully connected layers. For instance, Abramson et al. (2022) combine ResNet blocks for image processing, a learnable embedding table, a multi-modal transformer, LSTMs, and MLPs. Besides, Kim et al. (2023) use a Transformer based architecture (Vaswani et al., 2017), motivated by the observation that rewards are often non-Markovian.
In addition to the usual empirical risk in (12), it is also typical, in supervised ML, to add a regularization function to prevent overfitting:
2(Rs; D) = Σ
+ (xb))),
(13)
where Ap V - R4 is regularization function defined on the parameter space & of the underlying reward model class. For instance, Christiano et al. (2017) simply use L2 regularization and also consider dropout in some domains. Recently, Verma & Metcalf (2024) propose to define a more complex regularization term, which consists in biasing the learned rewards to be proportional to an approximate state importance provided by a trained Transformer-basee forward model.
The main supervised loss to train the reward model can also be augmented with additional losses corre- sponding to auxiliary tasks to avoid overfitting and improve generalizability. For instance, Abramson et al.
34

---

Published in Transactions on Machine Learning Research (06/2025)
(2022) use behavior cloning loss and add policy head to the reward network, thereby preventing that the reward model drifts too far from its initialization from the policy. Metcalf et al. (2023) design reward model using state-action representations trained to be temporally consistent via self-supervised learning. On a related note, the scalar preference optimization problem has been extended to a multidimensional one by Zhong et al. (2024b) to represent diverse human preferences and Marta el. al. (2023) for query efficiency.
5.2.2 Bayesian Approach
As is generally the case in supervised ML, there is also the variant of using Bayesian modeling for learning a target object instead of the (empirical) minimization of loss function. To this end, one starts with prior distribution P over the parameter space of the reward function that is updated in light of the data set 'D by means of Bayes theorem:
Pi (D) (D) p((b)
where (D) - (44 is provided m(q),, vb) is the likelihood of the data under the assumed human feedback model with reward function Ry- Such an approach is used for pairwise trajectory comparisons, for instance, by Schocnauer et al. (2014) for the noisy-ridge model or by Sadigh et al. (2017) for the Boltzmann distribution as the human feedback model. In inverse RL, such Bayesian approaches have been considered as well (see Section 4.3 in Arora Doshi (2021)).
Instead of assuming that the reward functions are paramotorizod, one can use the reward functions directly as parameter class and use a prior distribution over them. This could, for example, be Gaussian process initially considered by Kupesik et al. (2018) for pairwise trajectory comparisons and adapted in later works (Btyık et al., 2020; Cosner et al., 2022). Here, again, it is worth mentioning that such considerations have also been made in inverse RL before (see Section 4.3 in Arora & Doshi (2021)).
5.2.3 Partial Identifiability
A crucial question when it comes to learning the reward function is whether the reward function can be iden- tified at all. If two reward functions induce exactly the same human feedback model, the reward function is called partially identifiable or ambiguous. Skalse et al. (2023) study this topic for the Boltzmann distribution as the underlying human feedback model when demonstrations (inverse RL) pairwise trajectory prefer- ences are given as leedback. For demonstrations,, this question has also been examined in other works (Ng & Russell, 2000; Dvijotham & Todorov, 2010; Kim et al., 2021; Cao et al., 2021a). On a related note, Ellis et al. (2024) tackle this identifiability issuc by considering suitable acquisition functions (scc Section 4.1.1).
5.2.4 Interpretability
The field of explainable artificial intelligence (XAI) has emerged in recent years to improve the transparency and explainability of models even to enable them in the first place. Roughly speaking, the aim is to resort to more interpretable methods provide explanations for both experts and non-experts, shedding light on why a certain input in a (black box) model leads to a certain result. Explanations can take different forms, can the ways to ensure the transparency of models, and for a detailed overview, we refer to Barredo Arrieta et al. (2020). It is worth noting that the field has grown so extensively over the years that even dedicated overviews for the field of interpretable and explainable RT. are by now available (Puiutta & Veith, 2020; Qing et al., 2023; Milani et al., 2023; Glanois et al., 2024).
For the branch of RLHE, the existing works are quite sparse and mostly limited to using tree models transparent and explainable models for learning the reward function (Bewley & Lécué, 2022; Bewley et al., 2022; Kalra & Brown, 2022; Bewley et al., 2024; Kalra & Brown, 2024). Another way to realize explainability within RLHF suggested by Zhang & Kashima (2023) is to learn simultaneously the reward function and the importance of states using a weight network. Assuming that for (long) trajectories, only a few states are important for the preference outcome, their framework can be used to seloct samples for explainability purposes. Moreover, a perturbation analysis is suggested to evaluate explanations in quantitative manner using the learned state importance weights.
35

---

Published in Transactions on Machine Learning Research (06/2025)
5.2.5 Online Improvements
Christiano et al. (2017) demonstrate that it is important to improve the reward model online, finding that has been confirmed by subsequent works such as the one by Gao et al. (2023), which empirically demonstrates that overoptimization of a reward model trained offline leads to performance degradation. Without online improvements, issues of overoptimization of an imperfect reward model may occur. Abramiso et al. (2022) give an example of this: They attempt to fine-tune an agent initialized with behavioral cloning with an engineered reward function and find that it fails generalize and actually worsens the performance. They also compare RLHF with reward model trained offiine with iterative improvement and find that iterative improvement. leads to better performance, even sometimes exceeding human performance.
This is related to issues posed by the approximate nature of the reward model in general, discussed in further detail in Section 6.1, but improving reward model accuracy, in general, is not sufficient: McKinney et al. (2022) further show the interdependence of the reward model and the policy, demonstrating that reward models trained online together with a policy may not be effective when a completely new policy is trained.
Solutions to the problems of overoptimization and interdependence can take different forms: One is to update the reward model online with sufficient frequency using notably more on-policy queries (see Section 4.1.1), another is to improve the reward model, c.g., by leveraging ensembles (Eisenstein et al., 2024; Coste et al., 2024), combining multiple reward models with weight averaging (Rame el al., 2024), or by modifying the training procedure to place additional emphasis on challenging examples (Zheng et al., 2024), and a third, discussed in Section 6.1, is to add constraints to the policy training.
5.2.6 Learning from Multiple Feedback Types
As discussed in Section 3.5, it is often desirable to combine several feedback types. This requires extensions of the learning process to incorporate different sources of feedback. Learning from multiple feedback types can be achieved by pre-processing the feedback, assuming common latent factors, or by using the feedback types for distinct purposes.
The first approach is demonstrated by (Novoseller et al., 2023), who infer preferences from demonstrations, allowing them treat both types of feedback equally in the learning pipeline. In the style of the second approach, Jeon et al. (2020) proposes the unified framework of reward-rational choice, which allows for inter- preting many forms of human feedback enables combination and adaptive selection of feedback types. Finally, different types of feodback can be used for entirely different purposes, such as one for objective learning and another for safety-constrainus (Cosner et al., 2022) or for representation learning (see Section 3.2.3).
Noltzmann-rational choices and, through this common framework,
Since multiple sources of reward information may conflict, it is important to consider how combine them. Krasheninnikov et al. (2021) study several possible stralegies of combining several reward functions in this setting, relating it to multi-task inverse RL. Note that this challenge of confficting sources of reward relates tightly to challenges posed when receiving diverse preferences from different labelers, as discussed in Section 5.1.6.
5.2.7 Offline Reward Learning
There a recent trend towards offline RLHF, where both the reward model and the policy are trained offline. The offline sotting also frequently considered in RLHF theory (Section 7). Early approaches in this area (Kim et al., 2023; Shin et al., 2023) first generate queries from an oflline dataset of behaviors, gather human responses, train reward model from the resulting preference data, and then leverage offline RL algorithms to derive policy. We do not cover these works in detail, since this survey primarily focuses on the interactive and online setting (see Section 1.3). Nonetheless, the offline setting is particularly useful for evaluating novel approaches, e.g., for active query selection, using offline datasets. We refer the interested readers to Section 8.4 for a discussion of available datasets.
Note that offiine reward learning does not necessarily need to be combined with offline policy learning, as the reward model can be used to train policy online as well. This can be problematic in practice,
36

---

Published in Transactions on Machine Learning Research (06/2025)
the distribution of experiences will generally shift during online policy learning, which can lead to poor performance of the reward model. Nonetheless, this setup is commonly used in the LLM finetuming setting, since distributional shift is less of a concern due to the large and diverse training data (often collected off-policy) and the bandit-like nature of the task (compare Section 6.2). In this setting, it is also common to directly integrate preference and policy learning through direct policy optimization (DPO) techniques further discussed in Section 6.3.
5.3 Evaluating Learned Reward Functions
A contral question when it comes to learning the reward function is how to evaluate the learned reward function and how reward functions can be compared with each other. For this purpose, different approaches are available, e.g.:
Rollout Method In inverse RL, common method for evaluation is the rollout method (Ng et al., 1999: Fu et al., 2018a; Ibarz et al., 2018; Brown et al., 2019). In this approach, one first learns optimal policy for the learned reward function and then estimates the value of this policy for online trajectory rollouts using the known ground-truth reward function. This approach can be transferred to RLHF as well. In many cases, however, espocially in safety-critical arcas such as medicine or autonomous driving, such online rollouts cannot. be executed.
Off-policy Evaluations When online rollouts are not possible, so-called off- evaluations, which esti- mate the value of the optimal policy based on available data set, may be considered. For coping with biases or large variances due to policy mismatch, approaches using importance sampling (Pre- cup et al., 2000), regression- or classification--aseed methods (Paduraru, 2013; Le et al., 2019; Irpan et al., 2019), or combinations of these (Jiang & Li, 2016) have been proposed. Off-policy evaluation particularly well-suited for evaluating LLMs reward models. Lambert et al. (2024) propose a particularly large-scale dataset for this purpose, focusing on examples that are difficult to judge. They then use accuracy-based metries across multiple categories to evaluate reward models. The problem with these approaches, however, is that the traces of the explicit sources of error through policy learning or reward learning are blurred, and that some require access to the ground-truth rewards.
Distance Functions Yet another alternative, which has been advanced in the seminal paper by Gleave et al. (2022a), is using a suitable distance function for reward functions. Suitable here means that two reward functions, which differ only by certain transformations such as potential shaping (Ng & Russell, 2000) positive scaling, should have zero distance if these transformations do not change the policy ranking with regard to the expected return. For this purpose, Gleave et al. (2022a) present a pseudometric, called Equivalent-Policy Invariant Comparison (EPIC) distance, that is determined in three steps: First, mapping two reward functions to a so-called canonicalization form that is invariant to transformations of the latter kind, Second, normalizing these canonicalization forms by means of a specific weighted L2-norm whose weights are determined by a distribution over the transitions. Finally, the EPIC distance is the weighted L2-norm distance of the normalized canonicalization forms.
Even if some attractive properties, above all a Lipschitz continuity in terms of the EPIC distance of two reward functions for the difference of the value functions of the induced optimal policies is shown, this distance has its shortcomings. One shortcoming is that the canonicalization form used by EPIC distance does not encode sufficient knowledge about the underlying transition dynamics, which might lead to unreliable distances when evaluating reward functions on physically non-realizable transitions. To this end, Wulfe et al. (2022) propose the Dynamics- Aware Reward Distance (DARD), which uses a slightly different form of canonicalization but restricts the evaluation of the reward functions to transitions that are approximately physically feasible.
Recently, EPIC-like distances (Jenner et al., 2022) and STAndardisod Reward Comparison (STARC) metrics (Skalse al., 2024), which are entire classes of pseudometrics on the space of all reward functions were proposed that generalize the three-step approach underlying the EPIC distance (and
37

---

Published in Transactions on Machine Learning Research (06/2025)
DARD) by parameterizing each of the steps. Specifically, the canonicalization function in the first step, the normalization in the second, and the metric in the third step are kept variable. If these three functional parameters fulfill certain requirements, then the resulting distance has some appealing properties, c.g., being a pscudometric that is zero and only the two reward functions induce the same ordering of policies or imply upper and lower bounds on value function differences. particular, these metrics retain the ficxibility of DARD (in terms of specifying transition dynamics), while at the same time preserving the theoretical justification of EPIC.
Visual and Human Inspection For an evaluation by visual inspection, Jenner Gleave (2021) propose a method for preprocessing reward functions by transforming them into simpler but equivalent reward functions for better interpretability. Related to this and the rollout method, the quality of the reward function learned can also be evaluated by human (or expert) by examining the behavior of the agent on the target task. This can even extended to the reinforcement learning from AI feedback (RLAIF) setting, by using another trained model as the evaluator (Ti et al., 2024). Besides, in the context of LLMs, datasets have been proposed and specifically designed to evaluate the cin-ssonsit of learned reward models with respect to semantic changes of prompts (Shen et al., 2024).
5.4 Reward Model Inputs
Besides the feedback lype, another factor is the modality of the reward model input data. This usually consists of the agent's observations and actions. Observations can range from true state to high-dimensional inputs (e.g,, images), while actions can range from discrete finite actions continuous actions.
For instance, many typical RL benchmarks are in the contimuous control domain (e.g., MuJoCo simulated robotics tasks) with true state representations and simple discrete actions. In such problems, Christiano et al. (2017) train reward models from these inputs.
When no compact state representation is available, raw images are often used in control tasks, which makes the lcarning of rewards more challenging since the setting becomes partially observable and the reward function is generally not Markov with respect to the observations. In such cases, the conventional trick of approximating true state with a sequence of frames is often employed. This approach is used, for instance, by Christiano et al. (2017) to train reward models on the Atari benchmark suite. When taking only observations as inputs, one can resort to recurrent models (Abramson et al., 2022) or Transformer-based models (Kim el al., 2023).
More recently, many applications of RLHF are in the natural language processing (NLP) domain. In these settings, the policy takes natural language as both input and output while the reward model takes it as input (see, e.g., the work by Ouyang et. al. (2022)). Naturally, more complex scenarios (e.g., with both language and vision inputs (Abramson et al., 2022)) have also been studied.
5.5 Increasing Feedback Efficiency
Maximizing feedback efficiency is vital in RLHF due to the high cost of human feedback. This section delves into methods that enhance learning from limited human feedback. We discuss methods that leverage prior offline data, methods that use (partially unlabeled) data more efficiently, and methods that aim to gather more informative data.
5.5.1 Using Prior Data
There are often large amounts of prior data available at little or no additional cost. While this data generally was generated for her lasks, many basic human preferences are the same for various tasks and can often even be extracted from completely unsupervised data such as text corpora. By leveraging this prior data, we can greatly roduce the amount of foodback necessary to learn the current task's objective. We explore various methods, including meta- and transfer learning, leveraging foundation models, reward model initialization, preference model pretraining, and supervised representation learning.
38

---

Published in Transactions on Machine Learning Research (06/2025)
Meta- and Transfer Learning Meta- and transfer learning techniques in reward model training exploit the commonalities across different objectives, facilitating quick adaptation to new tasks. Ren et al. (2022) develop a broadly applicable meta-reward model, pre-trained on a diverse set of tasks to capture a wide range of preference patterns, enabling efficient adaptation to new tasks with fewer examples. Xie et al. (2018) use a similar meta-learning approach to build a goal classifier across multiple visuomotor lasks. Closely related to these meta-learning approaches, Hejna & Sadigh (2022) integrate few-shot learning principles, optimizing their approach for scenarios where only a few examples are available for adapting to new tasks. In the domain of transfer learning, Liu et al. (2023a) explore zero-shot transfer of preferences, method that enables adapting preferences without additional data from the new task. In different vein, but closely related to meta- and transfer learning, Mendez et al. (2018) tackle the lifelong inverse RL problem, focusing on inferring reward functions for multiple lasks over time, which involves knowledge transfer between lasks. Collectively, these studies underscore the potential of meta- and transfer learning in enhancing the efficiency and applicability of reward models in RLHF
Leveraging Foundation Models Foundation models, i.e. large models trained on large amounts of often unlabeled data, can acquire significant knowledge about basic human preferences. A language model trained to predict the next. token in lext corpus, for example, may learn to complete the sentence 'Frank was mad that his vacuum robot broke the vase', thereby learning that humans prefer non-destructive behavior. These Iearned preferences can then be leveraged in RLHF approaches. For instance, Kwon et al. (2023) propose to use LLM as a source of rewards. Du et al. (2023) is another example, where success detector is trained using a pre-trained vision-language model (Flamingo). Their approach uses a dataset of trajectories with binary success labels, employing a non-interactive training method.
Reward Model Initialization It is often beneficial to initialize the reward model with parameters from a model trained on related task. This strategy is particularly common in language model fine-tuning, where self-supervised pretraining is a common practice. In such scenarios, it becomes logical to use these pre-trained models for initializing not just the policy but also the reward model. This methodology is adopted by Askell et al. (2021) and Ouyang et al. (2022). Specifically, Ouyang et al. (2022) use pretrained language model for the reward model, opting for smaller model relative to the policy to mitigale unstable learning. Nolably, while they apply supervised fine-tuning to the policy before the RLHF phase, the reward model is initialized directly from the language model without any preliminary fine-tuning. This approach's applicability extends beyond language models to other areas. A notable example is Abranson et al. (2022), who, in the control domain, begin by training a policy using contrastive self-supervised learning and behavioral cloning. They then add an MLP head to the policy for the prediction of cunulative rewards.
Reward Model Pretraining Reward model pretraining (Askell et al., 2021; Bai et al., 2022a) leverages prior offline data to pretrain the preference model before training it policy samples. Askell et al. (2021) note that in the case of language models, noisy preference dala can be readily obtained from sources such as rated Reddit comments, preferred Stack Overflow answers, and reverted Wikipedia edits. They leverage this as a pretraining step to increase data efficiency. This is in addition to regular language model pretraining, as discussed in the previous paragraph. A similar approach could be applied to control in case prior data and some means of inferring preferences, such as human corrections, are available. Even if no inferred preferences are available, Verma & Kambhampati (2023a) show that it can be beneficial to pro-train the preference model to prediet close to constant reward on an initial set of trajectories. This avoids excessive fitting of the policy to random initialization differences in the reward function.
Supervised Representation Learning A compact representation that captures all relevant information for human preferences while minimizing noise can greatly enhance preference learning efficiency. It may also generalize better than a representation learned end-to-end as part of the preference learning task, which may contain spurious correlations. Bobu el al. (2022) address this by proposing the learning of features through explicit human feedback using feature traces. Feature traces (see Section 3.2.3) involve human labelers explicitly tcaching relevant fcatures one by one by demonstrating behavior which the feature monotonically increases or decreases. This method directly aligns the learned representation with numan-identifn features, enhancing preference learning efficiency but requiring detailed human input. However, feature traces require
39

---

Published in Transactions on Machine Learning Research (06/2025)
labelers to be able to identify and articulate relevant features, which can be challenging. Bobu et al. (2023) offer an alternative approach with their Similarity-based Implicit Representation Learning (STRL) method. SIRL learns representations from similarity queries (see Section 3.2.3), where human labelers provide feedback on whether behaviors are similar or different concerning the features that matter to them. This mothod caplures a broader range of human notions similarity without needing explicit feature knowledge, thus roducing the cognitive load on human labelers. In summary, while both approaches emphasize human feedback's centrality in representation learning, they differ in their methods of gathering this feedback. The feature traces used by Bobu et al. (2022) require specific feature knowledge, whereas SIRL used by Bobu et al. (2023) uses more intuitive similarity assessments, potentially offering more user-friendly way to capture human preferences.
These diverse methods of using prior data demonstrate the potential for enhancing data efficiency in RLHF, enabling more effective learning from limited human feodback.
5.5.2 Using Data More Efficiently
Beyond the application of prior data, several techniques can enhance the efficiency of data utilization in training processes. This section will discuss range of such methods, including self-supervised and semi- supervised training, as well as the integration of inductive biases and data augmentation strategies. These approaches are designed to make the most of the available human interactions and improve the final perfor- mance of RLHF models.
Self-Supervised Auxiliary Tasks Self-supervised training enhances data efficiency in reward model training by using unannotated data to capture information about the task. This technique extends beyond the scope of pretraining methods, as discussed in the prior section, to incorporating concurrent auxiliary tasks to maximize the utility of available data. A prevalent technique, as applied by Abramson al. (2022), Brown et al. (2020), and Metcalf et al. (2023), involves adding self-supervised losses to enhance representa- tion learning for rewards. Abramson et al. (2022) implement contrastive task where the reward network differentiates between observations that are consistent between multiple modalities and those that are not, blending this with preference learning loss and behavioral cloning. Brown et al. (2020) add multiple auxiliary tasks such as inverse and forward dynamics modeling, temporal distance prediction, and variational autoen- coder training. Similarly, Metcalf et al. (2023) use the self predicteee representations technique (Schwarzer et al., 2021) to learn state representations that encode environmental dynamics, enabling a linear model to anticipate successor states, thereby forming an efficient basis for preference learning and significantly boost- ing sample efficiency. However, auxiliary losses for better representation learning are not the only approach to leverage supervupe training. An alternate approach by Verma & Metcalf (2024) involves identifying im- portant states using attention weights from world model transformer and state importance estimates based on a preference predicting transformer. These estimates can aid credit assignment for observed preferences, further optimizing the training process.
Semi-Supervised Training Semi-supervised training, blending labeled and unlabeled data, can leverage the unlabeled data to glean information about the task and the environment. This is most commonly done by generating pseudo-labels for the unlabeled data, either by leveraging model predictions or by making assumptions. The first approach is used by Cao et al. (2021b), using supervised learning methods to predict human preferences, and Zhan et al. (2021), employing a GAN-based approach to learn and mimie human preference patterns. Similarly, Park et al. (2022) expand their data set with high-confideence unlabeled sam- ples based on the preference predictor's evaluations. The second strategy, making assumptions to augment data, is showeased by Zhou & Xu (2020). They generate preference data by assuming that (i) human-written examples are better than model-written examples, (ii) human-written and model-written examples are in- distinguishable amongst themselves, and (iii) generations of later model iterations are better than those of carlier ones. This is closely related to RLAIF, where a pretrained language model is used to gather preference feedback, effectively leveraging the knowledge about human preferences learned by the pretrained language model through self-supervised learning.
40

---

Published in Transactions on Machine Learning Research (06/2025)
Data Augmentation Data augmentation focuses creating additional examples from existing labeled data. Temporal augmentation is particularly effective in RLHF, involving trajectory data. This is exem- plified by Brown et al. (2019) and Park et al. (2022) who base their augmentation on the premise that preferences for complete trajectories can extrapolated to cropped segments, allowing the generation of multiple derivative pairs from a single labeled trajectory pair. Park et al. (2022) additionally explore state modifications, such as random re-scaling and Gaussian noise addition, finding temporal cropping be the most effective, with noise sometimes negatively impacting performance. In similar vein, Verma & Kamb- hampati (2023b) focus augmenting trajectories by concentrating changing elements in observations and perturbing the other parts, based on the premise that movement indicates importance in image-based observations. Complementing these methods, Abramson et al. (2022) employ augmentation by randomly altering instructions and language responses, thus creating artificial examples of non-preferred behavior. These diverse data augmentation methods collectively enhance the training data set, contributing to the increased robustness and efficacy of RLHF models.
Relatedly, Meta-Reward-Nee (Liu et al., 2022) optimizes not only for the preference prediction accuracy of the learned reward function but also of the learned Q function in actor-critic RL algorithm. This is beneficial since it avoids the phenomenon of confirmation bias, where one learned model (in this case the Q function) overfits to targets predicted by another model (the reward model). It is not strictly data augmentation technique, but closely related in practice.
5.5.3 Gathering Better Data
In addition to leveraging unlabeled data and using labels more efficiently, sample efficiency can be further increased by collecting more informative samples in the first place. This can either be achieved by selecting more informative samples from the experience buffer or by generating more informative experiences. While selecting informative samples from the experience buffer is addressod under active learning (sco Section 4.1.1), this section focuses generating more informative experiences.
While are not aware of many works in this area, one possible approach involves steering the agent's exploration towards regions of the state space where human feedback would be most beneficial. Liang et al. (2022) implement this by employing intrinsic molivation, driven by the estimated uncertainly of the reward model, to guide the agent's 's exploration. This highlights the potential of not just using data more efficiently but also generating data a more targeted manner.
6 Policy Learning
Agent
Environment
Action de
Policy
Dynamics
8t
State st+1
Action Reward fr4
Query q
Reward Model
Labeler
Label 4
Figure 5: RLHF diagram highlighting components discussed in this section.
After learning reward model, or, more commonly, interleaved with reward model learning, the next step is to train a policy that maximizes the expected accumulated reward. This section will discuss algorithms for policy learning, which can be categorized into two main techniques: adaptation of conventional RL algorithms and DPO.
6.1 Adaptation of RL Algorithms
Using the learned reward model, any standard RL algorithm (c.g., DQN, A3C, PPO, SAC) could potentially be applied to train policy. However, in the setting of RLHF, this direct application may suffer from two
41

---

Published in Transactions on Machine Learning Research (06/2025)
issues: The non-stationary nature of the learned reward function in RLHF and its inaccuracy, especially in the earlier stages of training. We will discuss these issues and possible adaptations of RT. algorithms to address them in the following.
Non-Stationary Rewards RL algorithms are designed to learn policy that maximizes the expected accumulated reward in an MDP framework, which assumes a stationary reward function. The RLHF sctting violates this assumption by periodically updating the reward model, leading to a non-stationary reward function.
Various works have empirically demonstrated that conventional RL algorithms can be applied nonetheless, with little to no modification. Christiano et al. (2017) argue that policy-gradiens gradient methods are better suited for non-stationary reward functions compared to value-based methods. They and various follow-up works successfully apply policy-gradient methods without any modifications in this setting, This approach has been picked up for language-model fine-tuning as well (Ouyang et al., 2022).
Later works have shown that value-based methods (possibly in an actor-critic schome) can also be effective in RLHF (Ibarz et al., 2018; Lee et al., 2021b; Park et al., 2022; Liu et al., 2022; Xue et al., 2023). One trick to make value-based methods work is to use the reward model to relabel the experiences the replay buffer whenever it is updated (Ibarz al., 2018; Lee et al., 2021b). Similar to conventional RL, the use of such replay buffer can greatly decrease the amount of environment interactions necessary for successful learning. As demonstrated by Gulcehre et al. (2023), the sample efficiency can be increased even further by using of techniques in a growing-batch RL setting, an offline-RL technique that iteratively increases the size of the dataset by policy rollouts while still being more sample-efficient than online RL.
In addition to the basic RL approaches, there are also some policy learning approaches tailored specifically for RLHF Wu et al. (2023) propose a policy gradient algorithm, called Pairwisc Proximal Policy Optimization (P30), as an alternative to PPO, which avoids estimating the value function and the same time is provably invariant with respect to equivalent rewards (unlike PPO). In similar vein, Zhu et al. (2023b) replace the KT-regularization of PPO with squared error term of the logarithmic probabilities, resulting in scemingly more stable RL learner.
Overoptimization of Approximate Rewards Since the learned reward model, which is only ap- proximation of the true reward function, is used to train a policy, overoptimization (Gao el al., 2023) or reward hacking (Skalse et al., 2022) can happen. Section 5.2.5 discusses the interdependence of the reward model and the policy in more detail as well as possible improvements from the reward model side, while here we focus on how to adapt policy training to cope with possibly inaccurate rewards in general.
One approach is to regularize the policy so as not to diverge too much from human-given demonstrations using KL-divergence regularization (Jaques et al., 2017). This is particularly common for language-model fine-tuning (Ouyang et al., 2022; Abramson et al., 2022). but Abramson et al. (2022) explores this for control as well. They found that this was important for some cases, in particular for deciding when to output language, but not for all. Going beyond KL-regularization, Moskovitz et al. (2024) investigate several techniques of constrained RL to only maximize rewards up to a threshold while avoiding excessive deviation from a pre-trained policy.
6.2 Framing RLHF for Generative Models as Bandit Problem
So far, we have assumed that we ultimately want to solve a reinforcement learning problem represented by an MDP. However, especially with regard to the application of RLHF in the area of LLMs, there is now another simplified way of looking at the problem. Namely, as an instantiation of (contextual) preference- based bandits problem (Bengs et al., 2021), which can of course be modeled by the more general case of a Markov decision process (MDP). In both cases, the focus is on the concept of lokens or rather sequences of tokens. However, in the MDP point of view, the state space consists of all previous tokens and the prompt (represented as a sequence of tokens), while the action space consists of all potential next tokens. A terminal state is often indicated here by the special token <eos> and trajectories are filled with this token until the maximum length H is reached. Moreover, the transition function P is degenerate (or deterministic).
42

---

Published in Transactions on Machine Learning Research (06/2025)
with a value of one only for the state that is the concatenation of the current state and the taken action. A (latent) reward is only received at the end of the trajectory giving rise to a sparse leedback scenario.
The (contextual) preference-based bandits view, on the other hand, naturally considers no state space and no transition function, but an action space consisting of all possible responses to a prompt (both represented as a sequence of tokens). Here, the prompt specifies the context for which at least two actions are executed and for which a qualitative comparison observed as feedback. In bandit literature, this is also referred to a " "(multi-)duell, coining the term dueling bandits. Thus, this point of view takes a trajectory-wiss perspective, while the MDP point of view takes token-wise perspective.
Note that the bandit formulation considers an entire episode (response in the LLM context) as an action with a single associated reward, resulting in sparse leedback from an MDP viewpoint. This is in contrast to the standard RLHF formulation as it is often used in control settings, where it is assumed that the reward of a trajectory composed of the sum of the rewards of individual steps, which allows the optimizer to distribute rewards densely as best fits the data. On an intuitive level, this leads to state-action pairs that often occur in preferred trajectories to be highly rewarded, without necessarily putting all reward the terminal actions. In practice, this can lead to nicely-shaped reward functions (Christiano et al., 2017), which cannot directly be achieved in the bandit setting. However, Chan et al. (2024) show how to take advantage of the predominantly used transformer architecture for the reward model in order to obtain a denser reward, even when assuming the bandit setting: More specifically, since the transformer architecture maintains attention weights in the last layer for each token, these can be used to attribute the overall reward signal to individual tokens.
The main appeal of the bandit formulation is that since the environment's dynamics are deterministic, exploration is simplified (although the space of trajectories to be explored is still huge, i.e. exponential with respect to the trajectory length). This setting even allows for policy improvements without any training using best-of-n-sampling (also called rejection sampling), a form of search that chooses the highest-rated response from multiple samples (Nakano et al., 2022; Menick et al., 2022). Most notably, however, this formulation notably enables direct policy learning approaches, such as DPO (Rafailov et al., 2023) or vPO (Azar et al., 2024), discussed in the following section.
6.3 Direct Policy Optimization
The two-phase approach involving utility learning and policy optimization is not the only viable path to learning a policy from human feedback. While we have previously discussed the case in which we learn a reward function from observed preferences by assuming human feedback model, the literature circumvents the reward-learning step and uses preferences directly to address the actual RL problem. One important representative algorithm in this direction is DPO (Rafailov et al., 2023). The key trick in this method is to reparameterize the reward model as a function of its optimal policy, which allows the likelihood of the observed feedback (still using the Bradley-Terry model) to be rewritten as a function of the policy parameter. Thus, a policy can be dircctly trained from pairwisc comparisons by minimizing the following negative log likelihood computed on a dataset D of triplets (a. yw. yw) with being a context (e.g.. prompt to I.LM) and you I1 being two outputs (e-g., generated answers) such that yor > yr
emerging branch of
LDPo(TO, Treet) = -E(Z.p
logo (Blog Tref (Vw xx))
To(Yer x)) - B log
To | x:) Tref (M x)) )
(2))
(14)
where a is the logistic function and /3 is a hyperparameter controlling the deviation with respect to a reference policy Tiot (e.g, pretrained LLM). This loss amounts to increasing the probability of generating better outputs while decreasing that of generating worse ones with respect to the reference policy.
Many improvements, variations, or alternatives to DPO have been proposed. Since those methods are not the main focus of our survey, we refer the interested readers to their respective papers for more details, e.g., SLiC-HF (Zhao et al., 2023), OPPO (Kang et al., 2023), DPPO (An et al., 2023), PRO (Song et al., 2024), RSO (Liu et al., 2024b), SRPO (Choi et al., 2025), or by formulating policy scarch as a zeroth-order optimization (Tang et al., 2024). Azar et al. (2024) introduce an objective called V-preference optimization (V/OO) that unifies the objective functions in DPO and RLHF More specifically, for specifie instantiation
43

---

Published in Transactions on Machine Learning Research (06/2025)
of V, the objective in VPO recovers DPO and SLiC-HF. In addition, DPO has been further generalized to include diverse divergence constraints (Wang et al., 2024a). Besides, Hejna et al. (2024) propose cantrastive preference learning based on a regret preference model instead of the usual one in RLHF It is also possible to learn a Q function from human preferences directly, which implics a policy without the need for separate policy- and reward-model training (Myers et al., 2023).
It is worth noting that approaches for directly learning the policy from preferences have been considered in the past as well (Wilson et al., 2012; Fürnkraaz et al., 2012; Wirth & Fürnkranz, 2013b;a; Busa-Fekete et al., 2014). In Sections 3.2.1 and 3.2.2 in the survey by Wirth et al. (2017), these methods are explained in more detail.
Another recent trend in fine-tuning models with human leedback is to even manage it without the usage of RL. An alternative is based on supervised reward learning with new types of loss functions (Lee et al., 2023; Yuan et al., 2023) or a specific learning process (Dong et al., 2023; Korbak et al., 2023). There are also RL-free approaches that do not use reward model to train a policy to execute natural-language instructions using a transformer architecture (Brohan et al., 2023; Yu et al., 2023). On related note, Liu et al. (2024a) suggest way how to convert human feedback to natural language sentences for the task of fine-tuning language models.
7 Theory
The field of RLHF has recently made some progress in terms of theoretical results, which we will discuss in this section. First, we consider the contributions where the goal is to learn a provably (near) optimal policy both in an online and offline fashion or even in a way that falls in between. Then, we discuss and highlight recent contributions related to different thcoretical aspects of RLHF, such as its relation to the standard reward-based RL. Tables 4 and 5 provide concise overview of the results for the online or oflline policy learning setting. Here, NNr(e, d) denotes the e-covering mumber of set F under some metric d², It is worth mentioning that (almost) all works have two standard assumptions, namely that the reward function is bounded and that the ground- truth reward, human feedback model, or transition dynamics are elements of the considered model space, respectively.
7.1 Policy Learning
In the literature focusing on theoretical results, there a distinction (similar to the distinction made standard RL) between an offline and online setting. In the former, learning is based on given fixed data set, usually previously collected through an interaction with the environment. In contrast, in the online environment, one interacts directly with the environment to learn from real-time feedback and continuously updates onc's strategics based on the feedback received, allowing the agent to learn and adapt as it engages with the environment. Accordingly, an important component of the online variant is the sampling procedure, i.., how the labels are selected. This is usually accomplished using an acquisition function that is based on uncertainty (sce Section 4.1.1).
Online Learning The first work dealing with the question of theoretical guarantees for learning an optimal policy from trajectory comparison feedback (see Section 3.2) in online manner is by Novoseller et al. (2020). It laid the foundation for a paradigm embraced by many subsequent rescarch endcavors: Adapting learning algorithms from the dueling or preference-based bandit literature (Yue & Joachims, 2009; Sui et al., 2018; Bengs et al., 2021) to the underlying situation with additional states. The preference- based bandit problem can be viewed as a preferenco-baser RL problem with one state, so state transition dynamics must be considered accordingly for fruitful adaptation. It is worth mentioning that Jain al. (2015) used quite similar idea before for feedback in the form of corrections (see Section 3.2) by resorting to the coactive learning setting (Shivaswany & Joachims, 2012). Assuming the existence of ground-truth trnjext-trajectory scoring function and that the user's feedback is informative, the Preference Perceptron algorithm by Shivaswamy & Joachims (2012) is used and analyzed in terms of its cumulative regret.
TTe e-covering number the minimum integer N such that there exists a subset F C F with |Fl = N, and for any f fEZ F. there exists some f' satisfying f. S c.
44

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: An overview of approaches, their assumptions, goals, and properties for online policy learning. is used to hide log-factors. T is the number of iterations of the respective algorithm.
Algorithm (Reference)
Algorithmic approach
Assumptions
(arget(s) and goal(s) of learner
Theoretical guarantee(s)
Ducling Poste-
Leveraging Posterior Sampling from dueling bandits
Lincar link function,
Bayes regret mini- mization w.r.t. op- timal policy based trajectory com- parison feedback
Asymptotic regret rate:
rior Sampling (DPS)
tabular MDP
18) /IAIT
(Novoseller et al., 2020)
Logistic Prefer- ence Reinforce- ment Learning (LPbRL)
Leveraging from contextual duel- ing bandits
MaxInP
Logistic link function, tabular MDP, lincar re- wards & d-dimensional feature embedding of trajectories
Expected regret minimization w.r.t. optimal policy based on trajec- tory comparison feedback
Transition dynamics:
1. Known
18) 18|nd (T)
(Saha 2023)
et al.,
Unknown
olia 22 18pveT
Preference- based Optimistic Plan- ning (PbOP) (Chen ct al., 2022)
Leveraging MaxinP; general function ap-
General model classes Fr for feedback and F3 for transition dynam- ics with finite 1z-norm p- BlEuder dimension d (p) and 4f (p).
High probability regret minimiza- tion w.r.t. optimal policy based on trajectory compar- ison feedback
4e 4 fox (N (+.s)
proximation
d (Ner (..) betng the istnfinity
Preference-based Policy Search (PPS)
Dynamic program- ming, policy search, (c,6)-PAC black-box ducling bandit algo- rithm and simulator
Uniform dependence of policy prefer- ence probabilities value function differ- ences, tabular MDP, (e, P)CC dueling bandit algorithm with (KK, o)e sample complexity for K armus
(e, 8)-PAC for op- timal policy based trajectory parison feedback
Simulator step bound
(Xu al., 2020)
Sample complexity bound 4 A
Preference- based Exploration & Policy Search (PEPS)
Similar to PPS, instead of simulator using auxiliary synthetic re- ward function
PPS and stochastic triangle inequality of trajectory comparisons prefer- ences
Same
(e, 8)-PAC for op- timal policy based trajectory parison feedback
Step complexity bound (LLA))//// [ 2a
Comparison complexity bound (")))) e/
(Xu al., 2020)
UCBVI-Planning
Optimistic least- squares value iteration, maximum information gain, value iteration based on pessimistic expected value func- tion ostimation
Binary rewards for pairs
(e, 8)-PAC for op- timal policy based binary state- action reward feed- back
Tabular MDP: # #+11111A1 lox mIIiAA)
(Kong & Yang,
statc-action based on human sponse model f E FB with bounded noise > O, compliant and tabular hadlar//lineu /lincar MDP with dimension
2022)
11 #18|1A
Linear MDP:
LA (APd 1 log
(15|[A]
Preferenco-based & Randomized Least-Squares
Least-squares value it- eration with perturbed state-action-wise re- ward model
General differentiable link function &, linear MDP, linear rewards with d-dimensional fea- ture embedding of tra- joctorics
Expected regret minimization w.r.t. optimal policy and/or low trajec- tory comparison feedback plexity steered by 0]
Expected regret bound: a (1Ta//2 + #22//,
(11/22)1112,3) Comparison complexity bound:
Value Iteration (PR-LSVI)
[at R/ax)2/112)
(Wu & Sun, 2024)
inf Rmax Bmaxl 4'(x)
Algorithu for Policy Alignment in Reinforcement Learning (A-
Iterative bilevel opti- mization via gradient descent based on es- timated policy gradient
Lipschitz assumptions on the objective fune- tion, the reward fune- tion, the parametric policy class, and vexity assumptions the value function
Solving the bilevel optimization prob- lem
Convergence rate: о0/T)
PARL)
(Chakraborty et al., 2024a)
45

---

Published in Transactions on Machine Learning Research (06/2025)
Novoseller et al. (2020) suggest the Dueling Posterior Sampling (DPS), which is an adaptation of the self- sparring algorithm (Sui et al., 2017). It. takes a Bayesian perspective on the problem and defines a Dirichlet prior on the dynamics and a Gaussian prior on the rewards that are subsequently updated, while the trajectorics to be compared by the human labeler are chosen based on their (postcrior) probability of being optimal9. Assuming a linear link function (see Section 5.1.4) as well as a labular MDP, it is shown that DPS is (i) consistent, i.c,, converges in distribution to the optimal policy, and (ii) achieves an asymptotic expected regret bound (see Table 4).
Xu et al. (2020) combine dynamic programming and policy search with black- preference-based bandit algorithm for cach state to design routines that return an almost optimal (a.k.a. e-optimal) policy with high probability". The first routine, called Preference-based Policy Search (PPS), requires access to a simulator, while the second routine, called Preference- based Exploration and Policy Search (PEPS), gets rid of this requirement by exploring the state space by means of an auxiliary synthetic reward function. By assuming that the probability of one policy dominating another policy is bounded uniformly over all states from below by a multiplicative of their value function, they show generic upper bounds for both routines on the number of pairwise trajectory comparisons (see Table 4). If these dominance probabilities have even more structural properties, such as fulfilling stochastic transitivity or stochastic triangle incquality (scc Haddenhorst et al. (2020); Bengs et al. (2021)), then these upper bounds can be further refined.
A follow-up work by Saha et al. (2023) assumes a feature embedding of trajectories that gives rise to a feature embedding of policies and adapts the MaxInP algorithm (Saha, 2021) for contextual dueling bandits by essentially viewing the policy embeddings as the contexts. More precisely, assuming a logistic link function (sce Section 5.1.4), confidence sets for the expected scores of the policies are constructed based on the maximum likelihood estimate (MLE), and the two policies with the highest uncertainty in terms of maximal variance are used to sample a trajectory, respectively, to be compared. In this way, the logistic preference- based reinforcement learning (LPbRL) is derived and also extended to the case of unknown dynamics by taking the uncortainty regarding the dynamics into account when constructing the confidence sots. For both cases, i.e., known or unknown dynamics, upper bounds on the regret of LPbRL are shown (see Table 4).
In contrast to previous work that all considers tabular MDPs, Chen et al. (2022) consider the case of a general unknown human feedback model and unknown dynamics cach from function classes with finito Eluder dimension (Russo & Van Roy, 2013). They propose and analyze the Preference-based Optimistic Planning (PbOP) algorithm, which essentially follows a similar design as LPbRL but uses least-square estimates for the human feedback model and transitions dynamics along with confidence sets based on them. Moreover, Chen et al. (2022) derive lower bounds for the regret of any learning algorithm by reducing the ance-per-episode-feelll reeebback RL problem (Chatterji et al., 2021) to the PbRL. problem. Finally, they extend their analysis to the case of K-wise comparisons, where one obtains all (*) pairwise comparisons for K many queried trajectories. In essence, the regret term coming from the human feedback model class improves by factor of VK.
Wu & Sun (2024) consider a similar learning scenario as Saha et al. (2023) but with the additional objective to keep the number of queries of trajectory comparisons low, which is a combination of two competing objectives also studied in the bandit literature (Degenne et al., 2019). For this purpose, they suggest the Preferencc--bsedd and Randomized Least-Squares Value Iteration (PR-LSVI) algorithm, which combines least-squares value iteration with a perturbed state-action- based reward model with Gaussian noise for regret minimization; a similar idea to CoLSTTM suggested for contextual dueling bandits (Bengs et al., 2022). More specifically, in each time step, the policy maximizing the value function of the perturbed state-action-based reward model and the policy maximizing the latter in the previous time steps are "played". By sampling trajectories of these two policies and computing their expected absolute reward difference (based on the perturbed state-action-basee reward model) as a measure of uncertainty, preference feedback for these two trajectories is queried if the uncertainty exceeds a certain threshold. Moreover, they also suggest a posterior
TThe latter probability is assessed by posterior sampling; commonly used technique in the bandit literature used by so-called Thompson Sampling strategies, see Lattimore & Szepewári (2020) for more details.
7This is a so-called PAC learning setting (Valiant, 1984) in which the goal of finding the/an optimal object is relaxed to finding "good cnough" object, usually specified by some distance measure on the object domain.
8 Roughly speaking, the Eluder dimension of function class refers the number of worst-case errors one must make to identify an unknown function from that class.
46

---

Published in Transactions on Machine Learning Research (06/2025)
sampling counterpart of this algorithm, the Preference-basee Thompson Sampling (PbTS) algorithm, and analyze it in terms of Bayesian quantities.
Recently, Chakraborty et al. (2024a) proposed a bilevel optimization problem that generalizes the standard optimization problem for RLHF with trajectory feedback and the negative log- ligliielihood as a loss function (i.e., (2)). This problem; which they call PART, (Policy Alignment in Reinforcement Learning), is charac- terized by explicitly taking into account the dependence on the data-collecting process at one level for the optimal policy parameters at the other level. For this problem, A-PARL is proposed, which is shown to have an O(/TT) convergence rate under specific assumptions, where T is the mumber of iterations.
Finally, for the LLM training scenario, there are two perspectives on the problem (MDP vs. contextual dueling bandits) as mentioned in Section 6. Accordingly, work from the field of context ual preference-based bandits (Dudik et al., 2015; Saha, 2021; Saha & Krishnamurthy, 2022; Bengs et al., 2022; Sekhari et al., 2023) can also be viewed as thcoretical contributions for the RLHF sctting for the LLM application. This view is advocated in particular by Xiong et al. (2024), who view the LLM fine-tuning task as a reverse-KL regularized contextual bandit problem. Typically the context is chosen externally, but Mehta et al. (2023) consider the learning variant in which the learning agent chooses the context as well. This variant is referred to as active contextual ducling bandits.
Offline Learning Zhu et al. (2023a) study the performance of a greedy policy trained from data set consisting of trajectory pairs along with the observed preference that is assumed to be generated by means of a Bradley-Terry model with linear rewards. For this purpose, different results with respect to the MLE of the Bradley-Terry model for different feedback scenarios are derived that are quite of independent interest. In particular, they show concentration inequalities of the MLE for trajectory- and additionally its asymptotic normality for action-based comparison feedback that also holds for K-wise comparisons. Based on these, it is shown that the groedy policy using the MLE in the case of acsion-basod feedback might fail while using a pessimistic MLE leads to minimax-rates with respect to the performance gap". The latter is also shown to be true in the case of trajectory- based feedback. Technically, the pessimistic MLE is realized by taking the policy that has the largest pessimistic expected value function, i.c., the lowest realization of the value function within a hyperparaneeer-deeeneeee Further results of independent interest are the inferred theoretical guarantees for maximum entropy inverse RL (Ziebart et al., 2008) and action-basee inverse RL algorithms (Neu & Szepesvári, 2009).
comparison feedback
confidence region around the MLE.
The simple model assumptions underlying (Zhu et al., 2023a) were then replaced by more sophisticated assumptions in some subsequent work. The linear reward assumption has been replaced by more general reward function classes by Zhan et al. (2024a) and Li et al. (2023). In addition, Zhan et al. (2024a) also consider more general unknown human feedback models and construct the confidence regions for the pessimistic approach directly from the log-likelihood function. The resulting approach, called FREEHAND, is analyzed in terms of its performance gap, for which some problem-dependent coefficients, the per-step, per-trajectory, and transition concentrability coefficient, are introduced. On the basis of a lower bound, it is shown that the per-trafectory concentrability coefficient should naturally appear in the bound on the performance gap. Moreover, the concentrability coefficient is shown to be upper bounded by the constant appearing in the special case of linear rewards considered by Zhu et al. (2023a). Finally, it is worth mentioning that both trajectory-based and action-based comparison feedback are considered.
In follow-up work, Zhu et al. (2024) found overfitting as well as overoptimization issues of the MLE in the Boltzmann model for pairwise comparison feedback. This can arise in particular if the observations of labels are strongly unbalanced and thus the utilities can become infinite. To overcome this problem, they propose the Iterative Data Smoothing (IDS) algorithm, which implicitly weights observed labels appropriately by their frequency and their current likelihood. Note that these issues do not contradict the results shown by Zhu et al. (2023a) as these are based on the assumption of bounded utilities (or rewards).
Assuming: a dynamic discrete choice model (Rust, 1987) underlying the given data set of observed trajectories (without explicitly observed preferences), Li et al. (2023) suggest the Canamic-Choices Choiic--eesssiiistiiiii Optimization (DCPPO) algorithm. It first estimates the reward model using this assumption and then learns
9Th expected difference between the optimal value function and the value function of the used policy.
47

---

of Published in Transactions on Machine Learning Research (06/2025)
Table 5: Overview of approaches, their assumptions, goals, and properties for offline policy learning with a data set size n. is used to hide log-factors. Rmax is a bound on the reward.
A Algorithm (Reference)
A Agoritthmic approach
Assumptions
(arget and Theoretical guarantee(s) goal(s)
learner
Pessimistic MLE (Zhu
Greedy policy et al., for pessimistic expected value function estima- tion
Logistic link func- tion, linear reward function for state- pair feature embed- ding with ularity assumptions weights, known transition dynamics
High probability bound for the performance gap based on trajectory-based (and action- based) feedback
d+log(
2023a)
reg-
Re- Greedy policy pessimistic EnforcemEnt expected value Earning
oFfline
General differen- tiable link function 4 general bounded reward function class F, and general transition dynamic class Fp
High probability bound for the performance gap based on trajectory-basee (and action- based) feedback
Transition dynamics:
1. Known
with
CB, IoutAE, 11220)
(1/,, (1/N
HumAN
feeD- function estima-
back (FREE-
tion
2. Unknown
HAND)
V CP ToRiA Fo (1/N, 11/8)
(Zhan
al.,
2024a)
Rmax oB Woik (1/8.1186)
16 infre(-Ramase Ranns] v'xx) Cr, Cp reward and transition concentrability coefficient.
Dynamic- Choice-
Dynamic discrete choice model, lin- Value itera- tion based on pessimistic ex- pected value function estima- tion
High probability bound for the performance gap based on action- based feedback
Linear model class: [A] 22 v (og(dHn/A))
Pessimistic- Policy-
ear MDP, linear reward function for state-pair feature embedding with some regu- larity assumptions weights, known model class entailing the value and ward function of the dynamic discrete choice model
RKHS model class with different eigenvalue decay:
Optimization (DCPPO)
d " u Roxttnmae p-Bntte
(Li et al., 2023)
Tne LAl (lag(n 11/5)111/M prexpenential decay.
JRcH [Al(n
Raain H18)
pepolynomial docay,
pcl
pooplaation effective sampling offecsive
dimecaiio
T.CBlI-Tabular- Offline (Kong & Yang 2022)
Maximum in- formation gain for reward querying, value iteration based pessimistic expected value function estima- tion for policy learning
Binary rewards for state-action pairs based on human response model with bounded noise, comn-
High probability bound for the performance gap based on binary state-action re- ward feedback
Linear model class:
B,g+ [E (N 1/2]
Ny. are numbere of viait time
pliant and labular MDP
48

---

Published in Transactions on Machine Learning Research (06/2025)
policy in (pessimistic) value iteration manner from the estimated reward model. In the case of linear MDP and known model class that entails both the value and the reward function of the dynamic discrete choice model, DCPPO is analyzed with respect to its performance gap. This is done for the case of a linear function model class as well as a subset of a reproducing kernel Hilbert space (RKHS) as the model class.
Focusing on the estimation of the weight parameter in the Bradley- Terry model for the action-based feedback under label differential privacy conditions (Dwork, 2008), Chowdhury & Zhou (2023) analyze two estimation procedures, MLE and stochastic gradient descent (SGD), under similar assumptions as in Zhu et al. (2023a). In both cases, the cost of ensuring label differential privacy is multiplicative factor.
Reward collapse, a term introduced by Song et al. (2023), describes the issue when rank-based training methods for LLMs lead to the same reward distribution regardless of the prompts used in the final training steps. The authors show that this occurs because the based approach does not adequately account for prompt-rclated information. To address this problem, the authors propose a family of utility functions as well as an optimization method that successfully creates prompt-dependent reward distributions, effectively mitigating the collapse of rewards during training.
Blending Online and Offline Learning Kong & Yang (2022) study the problem of optimal policy learning from critique feedback (see Section 3.2), ie., binary rewards for state-action pairs, with as few queries to the human as possible. They assume an underlying ground-truth human feedback model that leads to positive evaluation for state-action pair if it exceeds specific threshold evaluated at that pair. In addition, the learning procoss consists of two phases: First, exploring the environment in an unsupervised manner, and then querying user feedback in active reward learning phase to learn the human feedback model. This learning process is again analyzed in two variants: Either the exploration phase was performed externally, and a data set consisting of trajectories is provided (offline), or this data set is actively collected itself (ouline). For both variants, an active learning algorithm is proposed that essentially selects query points (state-action pairs) that provide the most information gain given the points already designated to be queried. For the online variant, an exploration strategy based on optimistic least- squares value itcration (Jin et al., 2020) is also introduced for tabular or linear MDPs. In both variants, policy learning is carried out by pessimistic value iteration with the empirical transitions and the estimated reward function, resulting in UCBVI-Planning (online) and LCBVI-Tabular of bounded noise (Massart & Nédélec, 2006) or low-noise assumption (Korba et al., 2017; Haddenhorst et al., 2021), bounds on the performance gap of both algorithms are derived.
(offline). Under the assumption
The question of the idcal experimental design for RLHF is addressod by Zhan et al. (2024b), in particular, how to separate the process of data acquisition (e.g., trajectories to be evaluated) from the process of retrieving human feedback to avoid constantly involving humans in the training loop. Assuming linear rewards, the Bradley-Tene Terry model and either a transition oracle (e.g., available for tabular or low-rank MDPs) or a linear MDP they suggest the expeRimental dEsiGn for queryIng huMan prEference (REGIME) algorithm that first. samples exploratory trajectories indented to be as informative as possible for learning the reward via MLE and then applies a greedy policy based on the reward learned by the latter. They explicitly show that REGIME requires less human foedback to be queried in order to output an coptimal policy at the end than the approach by Saha et al. (2023).
7.2 Preference-Bane Based vs. Reward-Based Learning
There have been some thcoretical analyses regarding the question of how far, or if at all, preferenco-based feedback in the form of trajectory comparisons is more suitable compared to numerical feedback. Ji et al. (2023c) suggest human rating model for this purpose in the numerical feedback case and analyze the LCB algorithm (Jin et al., 2021) in order to compare it with the pessimistic MLE (Zhu et al., 2023a). It is shown that under specific assumptions, LCB has a constant performance gap, while the preference-based pessimistic MLE under similar assumptions has a similar bound as in Table 5.
Wang et al. (2023b) provide reduction-based algorithms that can dircctly use statc-of-tho-art results in soward-based RL for RLHF with utility-based and general state-action and trajectory- comparison feedback. They show, in general, how theoretical results of the underlying standard RT. algorithm can be
49

---

Published in Transactions on Machine Learning Research (06/2025)
translated to theoretical results for the resulting preference-base based RL algorithm. For some special cases, such as MDPs with finite Eluder dimension and utility-based preference feedback, the theoretical guarantees are explicitly derived using state-of-thee the-art RL algorithms that are qualitatively similar to explicit preference- based RL algorithms.
7.3 Nash Learning from Human Feedback
The majority of theoretical works use the modeling of (pairwise comparison) feodback by means of a link function (see Section 5.1.4). Even if this often leads to simpler derivations, this modeling has the decisive disadvantage that it imposes transitivity of the human feedback that does not necessarily prevail in reality. In other words, it is quite possible that preference cycles can oceur. For this reason, there is new direction in theoretical work that dispenses with parametric modeling of the preference probability similar to Chen et al. (2022) but uses it to formulate a new learning objective. Specifically, the problem is considered from a game theory perspective, where two policies each propose trajectory that should be highly preferred by the human user. Thus, the goal to find a policy that suggests trajectories that are preferred to the trajectories of any other policy, ie., a Nash equilibrium or a von Neumann winner.
This learning variant was first considered by Wang al. (2023b); who showed that the problem can be reduced to finding restricted Nash equilibria in multi-agent RL problem (based on numerical rewards). For special situations of the latter problem, wrapper algorithms are proposed that have been shown to find the von Neumann winner with high probability. The learning problem was recently taken up and analyzed by Munos et al. (2024) and Ye al. (2024) in a KI-regularization variant. While the former considers the online-learning setting assuming a known preference model, the latter considers both online as well as offline learning settings and the preference model belonging to a finite function class.
8 Applications and Benchmarks
The field of RLHF has advanced significantly in the last few years, with increasing interest driven by promi- nent applications. First and foremost are applications to large language models, exemplified by Chat- GPT (OpenAI, 2022). This section starts by providing a sample of such applications, showcasing how this technology is being used in fields as varied as robotics, language processing, image generation, and more. We will also delve into libraries that provide foundational support for RLHF research, enabling researchers and practitioners to experiment with and refine a range of approaches. We then explore a spectrum of benchmarks that have been developed to standardize and simplify the evaluation of new approaches, offering insights into their performance in different settings. Finally, and closely related to those benchmarks, we will discuss common evaluation practices.
8.1 Applications
RLHF finds applications across various domains, showcasing its versatility in addressing complex and munccd tasks. The most prominent application is ChatGPT (OpenAI, 2022), which is an example of an application in the domain of language models. Beyond that, however, applications extend across diverse domains such as control tasks, generative models, and recommender systems. This section provides an overview of notable works applying RLHF in different arcas.
Control and Interactive Environments There is a long history of using control environments bench- mark tasks for RL. In addition to the breadth of available environments, control applications are of particular interest because tasks are often hard to specify. Christiano et al. (2017) demonstrated the effectiveness of RLHF in games as well as simulated contimuous control lasks, matching the performance of RL agents trained on ground-truth rewards with a fraction of the feedback. Extending to robotics, Ding et al. (2023) trained a reward model for diverse tasks with a single robot, achieving human-like behavior. Kupesik et al. (2018) applied RLHF for precise robot-to-human handovers. Similarly, Abramson al. (2022) used RLHF in the Playhouse simulator, platform for sensorimotor task training, and Milani et al. (2022) showcase
50

---

Published in Transactions on Machine Learning Research (06/2025)
application in the context of the MineRL Basalt competition for Minecraft tasks. Recently, Dong et al. (2024b) use RLHF to guide diffusion-based planning model.
Generative Models in Language and Imaging Generative models, i.e. models that generate new data instead of just predicting labels, can be framed as an RL setting in which a policy assembles the output through its actions. In the context of language models, this means that the language model is interpreted as policy with tokens as actions. Using this reframing, we can use RLHF approaches to fine- tune generative models to produce preferred outputs. ChatGPT (OpenAT, 2022) and GPT-4 (OpenAI, 2023) are prime examples of language models fine-tuned using RLHF, These applications build on earlier work, such as by Ouyang et al. (2022); Ziegler et al. (2020) and Glaesc et al. (2022). This method extends to text summarization (Gao et. al., 2018; 2020; Stiemnon et al., 2020), dialogue sunmarization (Chen et al., 2023), and question answering (Nakano et al., 2022; Menick et al., 2022). In image generation, Lee et al. (2023) and Xu et al. (2023) demonstrate the use of reward modeling for text-to-image tasks, while Pinto et al. (2023) and Kazemi et al. (2020) explore RLHF applications in broader computer vision tasks. Interestingly, in the context of LLMs, reward learning has also been expressed as density estimation (Dumoulin et al., 2024) instead of the supervised approach described in Section 5.
Recommender Systems In the context of recommender systems, Xue et al. (2023) have shown the poten- tial of RLHF in optimizing for long-term engagement. Although it is, in principle, possible to algorithmically evaluate policies in this domain, these rewards are sparse. To combat this, Xue et al. (2023) use RLHF to distill sparse, global feedback into a dense reward model.
These diverse applications underscore the adaptability of RLHF and its growing importance in various technological domains, paving the way for innovative solutions and enhanced human-computer interactions.
8.2 Supporting Libraries
Several libraries have emerged that aim to provide toolset for implementing and experimenting with RLHF and reward learning algorithms, contributing to the ease and efficiency of research and development. One notable example is the imitation library (Gleave et al., 2022b). It encompasses collection of imitation and reward learning algorithms, including those introduced in the seminal work by Christiano et al. (2017). In the offline realm, -lean-Offline RLHF (Yuan et al., 2024) provides implementations for offline RI. algorithms with human feedback. Two other libraries, APReL (Btyık et al., 2022b) and POLAR (Tucker et al., 2022), focus on the Bayesian setting. Bıyık et al. (2022b) provide a specialized framework for preference- based reward learning with a focus on Bayesian methods. Meanwhile, Tucker et al. (2022) introduce a framework designed for Bayesian reward learning from multiple feedback types, including pairwise preferences, in MATLAB. Finally, in the domain of language model fine-tuning, the tr1X library (Castricato et al., 2023) offers a toolkit specifically designed for language model training. It specializes in the fine-tuning of transformer- based language models, treating the language model as the policy in an RLHF setup.
Due to the many interacting components and the human element in RLHF research, implementing new ideas and running experiments can be quite challenging. The discussed libraries reduce this challenge and make RLHF research more approachable to many researchers.
8.3 Benchmarks
Due to the difficulty of reproducible evaluations without a ground-truth objective and with humans in the loop, benchmarks play an important role in advancing and evaluating RLHF approaches. Several benchmarks have been proposed, each focusing on different applications and challenges.
One such benchmark is B-Pref (Lee et al., 2021a), which focuses on control tasks with synthetic feedback. B-Pref aims to provide simulated human feedback that captures some irrationalities, thereby coming closer to evaluation with real human feedback than other approaches. At the same time, by relying entirely on synthetic feedback, the results are reproducible and cost-cffoctive to generate. In a similar vein, Freire et al. (2020) propose a set of environments designed to diagnose common problems in reward learning. These environments help in identifying and addressing the typical challenges that arise in RL.FF scenarios.
51

---

Published in Transactions on Machine Learning Research (06/2025)
The offiine RLHF setting is particularly well-suited for benchmarks, as it allows for the use of static datasets. Shin et al. (2023) evaluate pre-existing offline RL benchmarks for their suitability for RLHF evaluation, and find that many are ill-suited due to the simplicity of the required reward function. They do, however, identify a subset of these benchmarks together with their own addition for evaluation. While Shin et al. (2023) leverage synthetic rewards, Yuan al. (2024) propose a dataset and benchmark for oflline RLHF, including preference data. This helps to circumvent the challenges of synthetic feedback and benchmark reproducibility with real feedback.
The MineRL BASALT competition (Shah et al., 2021b; Milani et al., 2022) gives more application-driven benchmark with a complex environment. The competition proposes the challenge of solving tasks defined by natural language descriptions in Minecraft based on human feedback. Writing hand-engineered reward functions is very challenging in that setting, which makes it a good benchmark for methods based on human feedback. The competition is method-agnostic in principle, and non-RI. approaches such as behavioral cloning are also considered. While the initial dataset consists of human demonstrations, the compotition is agnostic for the leedback lype, which may include demonstrations, comparisons, and others. The final evaluation is performed by humans through pairwise comparisons.
In the domain of language modeling, Truthful QA (Lin et al., 2022) serves as benchmark that measures the truthfulness of models. Also, in the context of language models, Ramamurthy et al. (2023) introduce a set of pretrained reward models, learned from human feedback, as benchmarks. These models serve as reference points for evaluating new RLHF techniques against established standards.
For evaluating language reward models specifically, Lambert et al. (2024) introduce RewardBench, a bench- mark designed to test reward model performance across chat, safcty, reasoning, and preference learning tasks. The benchmark includes challenging test cases such as adversarial prompts and edge cases where reward models commonly fail, providing standardized evaluation for both classifier bassed and DPO-trained reward models. RewardBench has become a widely-used evaluation framework with a public leaderboard that tracks performance of submitted reward models.
Together, these benchmarks provide a diverse and comprehensive suite of tests that drive the development and refinement of RLHF methods, ensuring they are robust, effective, and capable of handling a wide range of real-world scenarios.
8.4 Datasets
Due to its interactive and online nature, RLHF research often does not rely static datasets. This is because the feedback is generally collected interactively and depends on the current policy, When the reward model is not refined iteratively, however, as is common practice for the related settings of LLM finc-tuning and offlin RLHF, static datasets can be used. Such a static dataset can significantly simplify the development and evaluation of RLHF methods.
Since language model fine-tuning a popular application of RLHF and gencrally does not iteratively refine the reward model, many datasets have been developed for this purpose. Particularly notable are hh-rlhf (Bai et al., 2022a) and PKU-Safe-RLHF (Ji et al., 2023a), two datasets focusing on harmless and helpful responses, the OpenAssistant datasets (oasst1, oasst2) (Köpf et al., 2023), containing not only response rankings but also ratings on various dimensions, the summarize from feedback dataset (Stiennon et al., 2020) focusing on preferences over text summaries, the Stanford Human Preferences Dataset (SHP) (Ethayarajh et al., 2022), 10 which is based on Reddit responses, the WebGPT dataset (webgpt_ ltt.pcccomooo... (Nakano et al., 2022), focused on long-form question answering and the HelpSteer (Wang al., 2024d) dataset, which is not based on preferences but instead gives ratings on for 4 attributes (helpfulness, correctuess, coherence, complexity) for cach responsc.
Although static dalasets are used more rarely in the control setting, some dalasets have been developed for offline RLHF in this domain. Concretely, Yuan et al. (2024) propose the Uni-RLHF dataset and a benchmark
10s paper introduces the V-usable information framework for measuring dataset difficulty. The SHP dataset was created after the paper's publication, using its techniques to identify learnable preference pairs from Reddit data.
52

---

Published in Transactions on Machine Learning Research (06/2025)
for offiine RLHF while Kim et al. (2023) publish dataset of real human preferences for typical offline RL tasks (D4RL, Robosuite).
8.5 Evaluation
Evaluating RLHF poses unique challenges, particularly in scenarios without precise ground-trun task spec- ifications. Evaluations generally focus on cither the learned policy or the reward model, cach shodding light on different aspects of system performance.
Policy Evaluation Assessing learned behavior is crucial for the evaluation of an RLHF system. In domains with ground-truth rewards, these can used for policy evaluation (Christiano et al., 2017). However, many RLHF applications lack this clarity. Ouyang et al. (2022), for instance, evaluate the quality of language model test set of prompts, highlighting the significance
responses by having labelers rate the output quality of human judgement in assessing model outputs. Jain et al. (2015) use direct scores on a Likert scale for evaluations, including self-assessments by trainers and cross-evaluations by others. Losey et al. (2022) extend this with a survey based on scores and free-form participant comments, comparing evaluations based on known true rewards with subjoctive experiences. Morcover, Abramson et al. (2022) employ multi-stage evaluation scheme that includes scripted probe tasks, a standardized test suite evaluated by humanss, and full interactive assessments, demonstrating the need for diverse and thorough evaluation methodologies in RLHE.
Reward Model Evaluation Direct reward model evaluation complements policy assessment. While reward model accuracy is a more direct measure of preference-learning success, the ultimate goal is inducing effective policies. A perfectly accurate reward model is often not necessary to induce good policy, which is the actual goal of RLHF. Therefore, both evaluation methods are idcally used in combination.
Jain et al. (2015) also use a ranking loss method for test. sets of trajectories, compared against expert evaluations with known scores. This approach provides quantitative measures of the reward model's fidelity. In addition, Wilde & Alonso-Mora (2022) compare parameter- based and reward-based evaluation measures for learned reward functions, identifying strengths and weaknesses in both methods and contributing a more nuanced understanding of reward model asscssment in RLHF These approaches provide quantitative measure of the reward model's accuracy in reflecting human preferences and expert judgements. For a detailed discussion of reward model evaluation, also refer to Section 5.3.
Policy- and reward model evaluation both offer insights into the performance an RLHF approach. Idcally, both measures should be combined to enable quick iteration and give insights into both the preference learning performance as well as the quality of the learned behavior.
9 Discussion and Conclusion
In this survey, we have provided an overview of the current state of RLHF, tracing evolution from PbRL and examining its broad applications across various domains like control, natural language processing, and computer vision. Given the rapid expansion of this field, our coverage necessarily has limitations in addressing every extension and application in full depth. In particular, given the rapidly evolving nature of LLM rescarch, our coverage of techniques specific to this domain is less exhaustive than our treatment of control and robotics applications. In this concluding section, we discuss key extensions and alternative approaches, identify open questions, and highlight future research directions.
9.1 Extensions and Related Approaches
While our survey focuses on RLHF methods that learn reward functions online from human fcodback, several promising approaches extend beyond this scope while addressing the same core challenge of learning human- aligned objectives.
53

---

Published in Transactions on Machine Learning Research (06/2025)
An increasingly important alternative is RL from AI feedback (RLAIF), which replaces human feedback with evaluations from pre-trained AT systems. This approach leverages foundation models as preference sources and has demonstrated success across diverse applications: language model fine-tuning (Bai et al., 2022b; Sun et al., 2024), generating intrinsic motivation for text-based games (Klissarov et al., 2024), and learning rewards (Wang el al., 2024c) or coding reward functions (Ma et al., 2024b; Xie et al., 2024) for control tasks. A hybrid approach (assisted evaluation) involves AT systems assisting rather than replacing human evaluators, such as generating critiques of model outputs (Saunders et al., 2022) or enhancing responses with metadata such as citations (Nakano et al., 2022; Menick et al., 2022).
While most work on RLHF implicitly assumes that tasks can be specified by maximization of expected accumulated scalar rewards, this reward hypothesis (Silver et al., 2021) is under active debate in the RL community (Lambert, 2021; Vamplew et al., 2022; Bowling et al., 2023; Skalse Abate, 2022). Skalse & Abate (2024) investigate the sensitivity of inverse RL reward misspecification, and recent RLHF ap- proaches are beginning to move beyond scalar rewards through more complex objective functions, such as multi-objective frameworks with non-linear aggregation of vector rewards (Qian al., 2023).
Finally, the RLHF framework has inspired numerous extensions across RL domains that revisit classic RL topics through the lens of human feedback. These extensions span three main categories: fundamental algorithmic concerns such as exploration (Liang et al., 2022), reward feature learning (Katz et al., 2021), hindsight experience replay (Zhang et al., 2023a), and reward shaping (Xiao et al., 2020); advanced learning paradigms including multi-task learning (Ouyang et al., 2022; Abramson et al., 2022; Myers et al., 2023), contimual learning (Zhang et al., 2024), and hierarchical RL (Pinsler et al., 2018); and specialized domains such risk-sensitive (Chen et al., 2024), safe (Dai et al., 2024; Cosner et al., 2022), and fair RL (Siddique et al., 2023). Additionally, as discussed in Section 4.2, the intersection of RLHF with human-computer interaction (HCI) offers rich opportunities for advancing feedback collection mechanisms. It is crucial to keep human psychology in mind when designing these systems and to learn from other related fields that alroady studied such issues extensively.
9.2 Current Limitations
Despite its successes, RLHF faces several limitations that constrain its broader applicability, Casper et al. (2023) offer thorough analysis of open problems and fundamental limitations of RLHF; beyond that, we highlight several key limitations in the following.
A fundamental limitation concerns feedback quality and interpretation. Without assistance during feedback, RLHF is limited by the tasks humans can reliably judge (Leike, 2022; Leike et al., 2018; Wu et al., 2021; Christiano el al., 2018) a problem exemplified in language model fine-tuning, where humans often prefer assertive but incorrect responses (Hosking et al., 2024). Furthermore, current approaches often fail to learn the actual causes of human feedback, learning spurious correlations instead (Tien et al., 2022).
Current approaches also struggle with awareness of gaps in preference understanding, The common separation between the policy and the reward model limits how the agent can reason about its knowledge of human preferences. This limitation may be addressed by tighter integration such as in the cooperative inverse RL setting (also called assistance games) (Hadfield-Menell et al., 2016; Shah et al., 2021a), where the agent simultaneously learns about human preferences and acts to maximize them, enabling behaviors like conditional planning while awaiting feodback and relevance-aware active learning. More broadly, the limited ability to reason about preference knowledge represents fundamental constraint of current RLHF approaches.
From a thcoretical perspective, primary challenge lies in relaxing underlying assumptions. This requires balancing assumptions that are not overly restrictive for practical use cases while maintaining feasibility of theoretical guarantees for computationally efficient algorithms. Key questions include whether algorithms can avoid actively maintaining policy space and climinate sub-optimal policies, or relying on computation oracles. Recent work, such as by Wang et al. (2023b) or Wu & Sun (2024), gives hope that this may be possible.
54

---

Published in Transactions on Machine Learning Research (06/2025)
Beyond technical limitations, RLHF raises important ethical and social considerations that require care- ful attention. These include fundamental questions about whose preferences should guide alignment, ensuring ethical data collection practices that respect privacy, mitigating the amplification of human biases, and ad- dressing incentives for labeler manipulation (Armstrong et al., 2020; Carroll et al., 2023) inherent in current training methodologies. While some technical solutions are emerging, such as privacy-preservee ment through differential privacy techniques (Wu et al., 2024), addressing these concerns comprchensively requires coordinated progress across technical research, ethical frameworks, and governance structures.
align-
9.3 Conclusion
Although RLHF has significantly contributed to advancements in LLMs and other areas of ML, it remains domain in its infancy with many unanswered questions and inherent limitations. These challenges present opportunities for further advancements in theory and practice, potentially resulting in more robust algorithms that make more efficient use of human feedback. It remains intriguing to what extent RLHF will continue to shape natural language processing, RL, robotics, AI alignment, and beyond.
Acknowledgements We thank Tom Bewley, Andreea Bobu, Adam Gleave, Erdem Biyık, Yannick Metz, Peter Stone, and Banghua Zhu for their feedback carlier versions of this survey. This publication supported by L.Uexxellent, funded by the Federal Ministry of Eduucation and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder as well as by the Hightech Agenda Bavaria. This work has also been supported in part by the program of National Natural Science Foundation of China (No. 62176154) and a collaborative project funded by NetEase. EH has received funding from the European Union's 's Horizon Europe research and innovation programme under the Maric Sklodowska-Curte grant agrccment No 101073307.
References
Youssef Abdelkareem, Shady Shehata, and Fakhri Karray. Advances in Preference- based Reinforcement Learning: A Review. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2022. doi: 10.1 201100/SMC33344.22222000
Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Lan- don, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, George Powell, Adam Santoro, Guy Scully, Sanjana Srivastava, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu. Improving Multi- modal Interactive Agents with Reinforcement Learning from Human Feedback, 2022. arXiv: 2211.11602. preprint.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-Based Policy Learning. In Proceedings of Ma- chine Learning and Knowledge Discovery in Databases (ECML PKDD). Springer, 2011. doi: 10.1007/978 3-642-23780-5
11.
Mayer Alvo and Philip L.H. Yu. Statistical Methods for Ranking Data. Springer, 2014. doi: (0.1007/978-1 4939-1471-5.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Conerete Problems in Al Safety, 2016. arXiv: 1606.06565. preprint.
Dario Amodei, Paul Christiano, and Alex Ray. Learning from human preferences, 2017. URL https: (accessed 2023-05-25).
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct Preference-basee based Policy Optimization without Reward Modeling. In Advances in Neural Information Processing Systems (NewrIPS), volume 36, 2023. URL https://papers nips. htaps.ccc/ppprr.../ saaper_files//pppppe Censtract-Conferee .tml.
Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. Pitfalls of Learning Reward Function Online. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), volume 2, 2020. doi: 10.2 /0096////cai.220//2222
55

---

Published in Transactions on Machine Learning Research (06/2025)
Saurabh Arora and Prashant Doshi. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress. Artificial Intelligence, 297:103500, 2021. doi: 00.106/j.ariin22021.1.83
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L. Littman. Deep Reinforcement Learning from Human Feedback, 2019, arXiv: 1902.04257. preprint.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfiel-DDddss, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Aligument, 2021. arXiv: 2112.00861. preprint.
Müslüm Atas, Alexander Felfernig, Seda Polat-Erdeniz, Andrei Popescu, Thi Ngoc Trang Tran, and Mathias Uta. Towards psychology-avee eawae aware preference construction in recommender systems: Overview and research issues. Journal of Intelligent Information Systems, 57(3): ::677 489, 2021. doi: 10.1007/5108444-22-00077
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A General Theoretical Paradigm to Understand Learning from Human Prefer- ences. In Proceedings of the International Conference Artificial Intelligence and Statistics (AISTATS). PMLR, 2024. URL https://proceedingg mlr press/v238 gheshlaghi-azzr244 html.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kermion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Halfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston,, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022a. arXiv: 2204.05862. preprint.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Latton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan, Constitutional AI: Harmlessness from AI Feedback, 2022b. arXiv: 2212.08073. preprint.
Bowen Baker, Ingmar Kanitscheider; Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent Tool Use From Multi-Agent Autocuricula. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. URL https: htttps://openrreiieee wtt.pppawreviieeeeeettettttttaee
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Luey Campbell-Gillnngaan Jan Bal- aguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, and Christopher Summerfield. Fine-tuning language models to find agreement among humans with diverse preferences. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. URL https://proceeddig: html..
Hritik Bansal, John Dang, and Aditya Grover. Peering Through Preferences: Unraveling Feedback Acqui- sition for Aligning Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL http://9988...200ee9944eeee
neurips.com cc/
Peter Barnett, Rachel Freedman, Justin Svegliato, and Stuart Russell. Active Reward Learning from Multiple Teachers. In AAAI 2023 Workshop on Artificial Intelligence Safety, 2023.
Alejandro Barredo Arrieta, Natalia Biaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Al- berto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AT. Information Fusion, 38:82-115 115, 2020. doi: (0.1016/..ifffus2201..222
56

---

Published in Transactions on Machine Learning Research (06/2025)
Chandrayee Basu, Qian Yang, David Hungerman, Mukesh Singhal, and Anca D. Dragan. Do You Want. Your Autonomous Car To Drive Like You? In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), Association for Computing Machinery, 2017. doi: 00.1145/2000944330222222
Chandrayee Basu, Mukesh Singhal, and Anca D. Dragan. Learning from Richer Human Guidance: Aug- menting Comparison-Based Learning with Feature Queries. In Proceedings of the ACM/IEEB Interna- tional Conference on Stuaan-Robot Interaction (HRI). Association for Computing Machinery, 2018. doi:
01144//31112213171223
Chandrayee Basu, Erdem Bıyık, Zhixun He, Mukesh Singhal, and Dorsa Sadigh. Active Learning of Re- ward Dynamics from Hierarchical Queries. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019. doi: 10.1109 2000000499972219.8880
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. In Advances in Neural Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021. URL bttps://proceediie -tassatttttt....ttatttt.....c.. .html..
Viktor Bengs and Eyke Hüllermeier. Preselection Bandits. In Proceedings of the International Conference Machine Learning (ICML). PMLR, 2020. URL https::/proceeedings wwt...m......... mlr. wttppp///11//////e////2 html..
Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-baseed Online Learning with Dueling Bandits: A Survey. Journal of Machine Learning Research, 22(7):1 108, 2021.
ISSN 1533-7928. URL http://jmlr. org pptppp//v2/1---592/2 html.
Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Models. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022. URL https://proceeddnges mmlr wwwppppeessseeeee wtress/v122/bengs22a hhtml..
Tom Bewley and Freddy Lécué. Interpretable Preferenco-baseed Reinforcement Learning with Troe-Structured Reward Functions. In Proceedings of the International Conference on Aulonomous Agents and Mulliagent Systems (AAMAS). IFAAMAS, 2022. URL https: htttps:/www..faamas...o wtw.aa..or///ooeeeaaaaaaeooeeoea hwtpa//oo//e.eeeen///aaaaaae 2amppm222 p p118.pd. pdf.
Tom Bewley, Jonathan Lawry, Arthur Richards, Rachel Craddock, and Ian Henderson. Reward Learning with Trees: Methods and Evaluation, 2022. arXiv: 2210.01007. preprint.
Tom Bewley, Jonathan Lawry, and Arthur Richards. Learning Interpretable Models of Aireraft Handling Behaviour by Reinforcement Learning from Human Feedback. In ATA A SCITECH 2024 Forum. American Institute of Aeronautics and Astronautics, 2024. doi: 10.2514/6. 20224-43.00
Adam Bignold, Francisco Cruz, Matthew E. Taylor, Tim Brys, Richard Dazeley, Peter Vamplew, and agents: An assisted reinforcement
Cameron Foale. A conceptual framework for externally-inftuunn learning review. Journal of Ambient Intelligence and Humanized Compuling, pp. 00000//s12252-021-0888833
2021. doi:
Erdem BıyIk and Dorsa Sadigh. Batch Active Preference- Based Learning of Reward Functions. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2018. URL https::/ mlr. htt...pressv88///
biyik18a. html.
Erdem Bıyak, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2019. URL https:// wwwprreeeiing.m........r mlr. hatp./pree///1000-----00000 html.
Erdem Bıyık, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active Proferenco-Basce Gaussian Process Regression for Reward Learning. In Proceedings of Robotics: Science and Systems (RSS), vol- ume 16, 2020. URL http.t/tt///.....rrrreerrreeeee
html.
57

---

Published in Transactions on Machine Learning Research (06/2025)
Erdem Biyak, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevehuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demon- strations and preferences. The International Journal of Robotics Research, 41(1):45 67, 2022a. doi:
10.117700273364911004468
Erdem Biyik, Aditi Talati, and Dorsa Sadigh. APReL: A Library for Active Preference-basel Reward Learn- ing Algorithms. In Proceedings of the ACM/IEEE International Conference Human-Roue Interaction (HRT), 2022b. doi: 10.1109/ (0.1109/HR15335) HRI53351 02033351..2229990 9889900
Erdem BiyIk, Nima Anari, and Dorsa Sadigh. Batch Active Learning of Reward Functions from Human Preferences. ACM Transactions on Human-Robo Robot Interaction, 2024. doi: 00.1145/3644986
Andreca Bobu, Andrea Bajesy, Jaime F. Fisac, Sampada Deglurkar, and Anca D. Dragan. Quantifying Hypothesis Space Misspecification in Learning From Human Robot Demonstrations and Physical Correc- tions. IEEE Transactions on Robotics, 36(3):835 854, 2020a. doi: 10.1109/TRO 2AL1099TRO.2000.222 2020.2971415.
Andreea Bobu, Dexter R. R. Scobee, Jaime F. Fisac, S. Shankar Sastry, and Anca D. Dragan. LESS is More: Rethinking Probabilistic Models of Human Behavior. In Proceedings of the (CM/IEED Interna- tional Conference on Human-Robot Interaction (HRI). Association for Computing Machinery, 2020b. doi: 0111145533195023374453333
Andreea Bobu, Marius Wiggert, Claire Tomlin, and Anca D Dragan. Inducing structure in reward learn- ing by learning features. The International Journal of Robolics Research, 41(5):997-518, 2022. doi:
10.1177/0076664422107777
Andreca Bobu, Yi Lin, Rohin Shah, Daniel S. Brown, and Anca D. Dragan. SIRL: Similarity-basod Implicit Representation Learning. In Proceedings of the ACM/IEEE International Conference on Haman-Robol Interaction (HRI). Association for Computing Machinery, 2023. doi: 0111145/3568122.3577777
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better Rewards Yield Better Summaries: Learning to Summarise Without References. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, 2019. doi:
10.18653/v1/DD /1/1//13///1
Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the Reward Hypothesis. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https:
//proceedings. mlr. www.mm........... prosss/v202/boxlingg22 htm1.
Ralph Allan Bradlcy and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biomelrika, 39(3/4):324 345, 1952. doi: 10.2307/2334022
Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D. Procaccia (eds.). Handbook of Computational Social Choice. Cambridge University Press, 2016. doi: 10.1017/CB0978 11178097811074499848 107446984.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huar Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan H. Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics Transformer for Real- World Control at Scale. In Proceedings of Robotics: Science and Systems (RS5), volume 19, 2023. URL www.:rooobccppoceeedinneoo. wwtp:rssgggg//////.... html.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations. Tn Proceedings of the International
58

---

Published in Transactions on Machine Learning Research (06/2025)
Conference on Machine Learning (ICML). PMLR, 2019. URL https://proceediigg.u mlr. http://oocccciaoooo.orrrrrr///// brown19a. html.
Daniel S. Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2020. URL httpa://proceedings. mlr. wtt.a//ww......p../eeieee prress/v199browwerr html.
Róbert Busa-Fekete,, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: Evolutionary direct policy search using a preference- based racing algorithm. Ma- chine Learning, 97(3):327-335 2014. doi: 10:1007/11099--0444555...
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Proceedings of Robotics: Science and Systems (RSS), volume 16, 2020.
doi: 10.15607 205.0077/SSS..200XV1...
Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement lcarning. In Advances in Newral Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021a. URL https://proceedings www.nee...c.ppppeeeeeee wtt..eip....pppppe///2//../. G711011122764415777775557700 Abstrrcctc.bbrrtt..... html.
Zehong Cao, KaiChiu Wong, and Chin-Teng Lin. Weak Human Preference Supervision for Deep Reinforce- ment Learning. IEEE Transactions on Newral Networks and Learning Systems, 32((2):5339 5378, 2021b.
doi: 10.1109/TNNIA 202.13S.222.30081130
Micah Carroll, Alan Chan, Henry Ashton, and David Krueger. Characterizing Manipulation from AI Sys- tems. In Proceedings of the ACM Conference on Equity and Access in Algorithms, Mechandsms, and Optimization (EAAMO). Association for Computing Machinery, 2023. doi: 10.1145/36176994363333
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hasc, Erdem Biyik, Anca Dragan, David Krucger, Dorsa Sadigh, and Dylan Hadffeld-Menell. Open Problems and Fundamental Limitations of Reinforce- ment Learning from Human Feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
URL
Louis Castricato, Alex Havrilla, Shahbuland Matiana, Duy Phung, Aman Tiwari, Jonathan Tow, and Maksym Zhuravinsky. trlX: A scalable framework for RLHF. Zenodo, 2023. URL https://githut. com/ CarperAI/trlx.
Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, and Stella Biderman. Suppressing Pink Flephants with Direct Principle Feedback, 2024. arXiv: 2402.07896. preprint.
Manuela Cattelan. Models for Paired Comparison Data: A Review with Emphasis Dependent Data. Statistical Science, 27(3):412-433, 2012. doi: 10.1214/12-ST5366
Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang, PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a. wwt.:///oooruriiddda/a3a0d322
URL attps://openreee
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Bedi, and Mengdi Wang. MaHM:n-RLHFF Towards Equitable Aligument of Large Language Models with Diverse Human Preferences. In ICML 2024 Workshop on Models of Human Feedback for Al Alignment (MHFAIA), 2024b. URL http://ereiareeee//oo//r//d//--0
59

---

Published in Transactions on Machine Learning Research (06/2025)
Alex James Chan, Hao Sun, Samuel Holt, and Mihacla der Schaar. Dense Reward for Free in Rein- forcement Learning from Human Feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL http:://pppuuueiissseeeeee/rrrre net/foruu/ideey////taa
Lawrence Chan, Andrew Critch, and Anca Dragan. Human irrationality: Both bad and good for reward inference, 2021. arXiv: 2111.06956. preprint.
Niladri Chatterji, Aldo Pacchiano, Peter Bartlett, and Michael Jordan. On the Theory of Reinforce- ment Learning with Once-per-Episode Feedback. In Advances in Newral Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021. URL https://proceeddigg meurips.cc/pape Abstract.st.........mm html.
Jiaao Chen, Mohan Dodda, and Diyi Yang. Human-in-the-loop Abstractive Dialogue Summarization. In Findings of the Association for Computational Linguistics (ACL). Association for Computational Linguis-
tics, 2023. doi: 10.18653/v1 100p23..iningg-cl....
Li Chen, Marco de Gemmis, Alexander Felfernig, Pasquale Lops, Francesco Ricci, and Giovanni Semer- aro. Human Decision Making and Recommender Systems. ACM Transactions on Interactive Intelligent Systems, 3(3):17:1-11:7, 2013. doi:
Xiaoyu Chen, Han Zhong, Zhuoran Yang. Zhaoran Wang, and Liwci Wang. Human-in-the-loop: Provably Efficient Preference- based Reinforcement Learning with General Function Approximation. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022. https::ts///oooteee mlr, www.pprresssvveeeeee chen22ag.... html.
Yu Chen, Yihan Du, Pihe Hu, Siwei Wang, Desheng Wu, and Longbo Huang. Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: /ttps://openrevier net./ forum?id=wW1SkPl.
Weiwei Cheng, Johannes Fürnkranz, Eyke Hüllermeier, and Iteration: Leveraging Preference Learning for Reinforcement Learning. In Proceedings of Machine Learning and Knowledge Discovery in Databases (ECML PKDD). Springer, 2011. doi: 10.1007 191107777778-3--22277888
Park. Preference Based Policy
5 30.
David Chhan, Ellen Novoseller, and Vernon J. Lawhern. Crowd-PrefRL: Preferenco-Basod Reward Learning from Crowds, 2024. arXiv: 2401.10941. preprint.
Eugene Choi, Arash Ahmadian, Matthieu Geist, Olivier Pietquin, and Mohammad Gheshlaghi Azar. Self- Improving Robust Preference Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL
Jaedeug Choi and Kee-eung Kim. Nonparametric Bayesian Inverse Reinmercement Learning for Mul- tiple Reward Functions. In Advances in Neural Information Processing Systems (NIPS), vol- ume 25. Curran Associates, Inc., 2012. URL https://proceeddngs. wwwpeeddinnnnneuiiss.... www.ine..ccccceooo hwtrrippsccppappr//22////// html.
Sayak Ray Chowdhury and Xingyu Zhou. Differentially Private Reward Estimation from Preference Based Feedback. In ICML 2029 Workshop The Many Facets of Preference-Bone Based Learning (MFPL), 2023.
URL
Paul Christiano. Semi-supervised reinforcement. learning, 2016. URL dttps://ai-aliinmeet com/ (accessed 2023-12-14),
semi- aeaal-/pprrrieeddeeeeereeeeeeeae
Paul Christiano. Thoughts on the impact of RLHF rescarch, 2023. URL https://www http://wwwwwmnn.fo...... -fte---iama--------------eeeeeee
60

---

Published in Transactions on Machine Learning Research (06/2025)
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforce- ment. Learning from Human Preferences. In Advances in Newral Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017. URL https://proceeddin wtt.:g/..nnriii..cccccco neuripp..c/ppaee//2////aass (eeeooddddGGs1111gdrd0dddddddddd Aat.ooodddee---s-aaatttt.. html.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018. arXiv: 1810.08575. preprint.
Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian Mewilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, Leonard Hussenot, Neil Zeghidour, and Andrea Agostinelli. MusicRL: Aligning Music Generation to Human Preferences. Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https: //proceedings... mml, www.rrrress/////eeeeee cwtp/o////ddrro2aaaa.aa html.
Jack Clark and Dario Amodei. Faulty Reward Functions in the Wild, 2016. URL hotp://auuty-rewwwdd-futttttt
Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lam- bert, Milan Mosse, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feed- back. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL attps://proceedeee
mlr, wttpppesssseeeeeeeeee cotp://rnnnttaaaa...m html.
Ryan Cosner, Maegan Tucker, Andrew Taylor, Kejun Li, Tamas Molnar, Wyatt Ubelacker, Anil Alan, Gabor Orosz, Yisong Yue, and Aaron D. Ames. Safety-Aware Preference- Based Learning for Safety-Criticn Critical Control. In Proceedings of the Annual Learning for Dynamics and Control Conference (L4DC). PMLR, 2022. URL https: sttps::/pproceednng... mlr wwtpssaww........eeeeeeeeeeeeeee ppess/v168/ cosner22a.html.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL
Christian Arzate Cruz and Takeo Igarashi. A Survey on Interactive Reinforcement Learning: Design Prin- ciples and Open Challenges. In Proceedings of the ACM Designing Interactive Systems Conference (DIS). Association for Computing Machinery, 2020. doi: 01114//335726633955222
Yuchen Cui and Scott Niekum. Active Reward Learning from Critiques. In Proceedings of the IEEE Inter- national Conference on Robotics and Automation (ICRA), 2018. doi: 10. 10.1109/ICRA.21111 20..//C.......460555...
Yuchen Cui, Qiping Zhang, W. Bradley Knox, Alessandro Allievi, Peter Stone, and Scott Niekum. The FM- PATHIC Framework for Task Learning from Implicit Human Feedback. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https: //pooeeeig//c www.mml......mmmmmm wttpppesss/////// cui2ia. html.
Jessica Dai and Eve Fleisig. Mapping Social Choice Theory to RLHF In ICLR 2024 Workshop on Re- liable and Responsible Foundalion Models (R2-FM), 2024. URL https://openrevierr wtwpeearrrrioeeeetottroriii.... net/forum?id= iYthn3WATc.
Josef Dai, Xuchai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhon Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback. In Proceedings of the Interna- lional Conference on Learning Representations (ICLR), 2024. URL https: / id=TyFrPOKYXw.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active Reward Learn- ing. In Proceedings of Robotics: Science and Systems (RSS), volume 10, 2014. URL http://www.
roboticaproceedings org/rss10/p31.c //tp htmm.
Oliver Danicls-Koch and Rachel Freodman. The Expertise Problem: Learning from Specialized Feodback. In http://www....etoeooooo/////////
NewrIPS 2022 Workshop on MT. Safely, 2022. URL
61

---

Published in Transactions on Machine Learning Research (06/2025)
Brett Day, lan J. Bateman, Richard T. Carson, Diane Dupont, Jordan J. Louviere, Sanae Morimoto, Riccardo Scarpa, and Paul Wang. Ordering effects and choice set awareness in repeat-response stated preference studies. Journal of Environmental Economics and Management, 63(1))73 91, 2012. doi: 10.1016/j.eer (t000//....m..11111.1. 2011.09.001.
Rémy Degenne, Thomas Nedelec, Clement Calauzenes, and Vianney Perchet. Bridging the gap between regret minimization and best arm identification, with application to A/B tests. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2019. URL https: //proceedings. mmlr. www.smas..........e......eeeeeee press/v8sS /vtsss//v///eee het//ddgenne1... html.
Zihan Ding, Yuanpei Chen, Allen Z. Ren, Shixiang Shane Gu, Hao Dong, and Chi Jin. Learning a Universal Human Prior for Dexterous Manipulation from Human Preference, 2023. arXiv: 2304.04602. preprint.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun Shum, and Tong Zhang. RAFT: Reward rAnked FineTuming for Generative Foundation Model Aligument. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:
//openreview wwt.:ormmmmdddddmm500u
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caim- ing Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling to Online RLHF. In Transactions on Machine Learning Research, 2024a. URL https: https:://opemrevieee wetpieewwweeeeeeee nett/forumidd-a1aaaUUS
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjic Fan, and Zhipeng Hu. Aifg Aligning Diverse Human Preferences via Sehavior-Customisaall Diffusson Model. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL https:///ppp http::///ppeeeieeeeemmm wwt.///oroumm/eeeeefEEEEE
Yuging Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-Language Models as Success Detectors. In Proceedings of the Conference on Lifelong Learning Agents (CoLLAs). PMLR, 2023. URL httpp::/ mlr press/v232 (e23bb/ddubbb.rtmmm
Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual Dueling Bandits. In Proceedings of the Conference Learning Theory (COLT). PMLR, 2015. URL
https://prroceedings mlr. crress/v40/udikkte html.
Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann Dauphin. A density estimation perspective on learning from pairwise human preferences. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https /ttps://openteie pttpaarrree...eeefffoora/////aaa
Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solablle MDPs. In Proceedings of the International Conference on Machine Learning (ICML). Omnipress, 2010. URL
Vikranth Dwaracherla, Scyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient Exploration for LLMs. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https://openreviiee wtt.pe/eriieeeet..oooro/oeeeeeep net/fforumiid=pppP66WW
Cynthia Dwork. Differential Privacy: A Survey of Results. In Proceedings of Theory and Applications of Models of Computation (TAMC). Springer, 2008. doi:
Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning. In Advances ín Neural Information Processing Systems (NewrPPS), volume 35, 2022. URL https: www..ee.....pappeepppeeee
André Eberhard, Houssam Metni, Georg Fahland, Alexander Stroh, and Pascal Frioderich. Actively Learning Costly Reward Functions for Reinforcement Learning. In NewrIPS 2022 Workshop on Al for Accelerated Materials Design, 2022. URL
62

---

Published in Transactions on Machine Learning Research (06/2025)
Jacob Eisenstein, Chirag Nagpal, Alckh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishna- murthy Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. In Proceedings of the Conference on Language Modeling (COLM), 2024. URL (ttpaadaaaullllattteaaaaaasaeaa.
https://openreviee
Evan Ellis, Gaurav R. Ghosal, Stuart J. Russell, Anea Dragan, and Erdem BIylk. A Generalized Acquisition Function for Preference-basee Reward Learning. In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), 2024. doi:
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding Dataset Difficulty with V-Usable Information. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022.
URL https: stttps://proceedingg .ml.. http..peeee//ee///aaaaaaaaagaaae html.
Patrick Fernandes, Aman Madaan, Emmm Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T. Martins. Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation. Transactions of the Association for Computational Linguistics, 11:1643-1666 1668, 2023. doi: 10.1162/tas a 00626.
Tesca Fitzgerald, Pallavi Koppol, Patrick Callaghan, Russell Quinlan Jun Hei Wong, Reid Simmons, Oliver Kroemer, and Henny Admoni. INQUIRE: INteractive Querying for User-aware Informative REasoning. In Proceedings of the Conference Robol Learning (CoRL). PMLR, 2022. URL httpss//prooeedings. mlr, pttps///////iiiieeaaaa.222 fitzgeradd3a......... html.
Floyd J. Fowler, Jr. Survey Research Methods, SAGE Publications, 2013. ISBN 978-1-4833-1244 1240--..
Rachel Freedman, Rohin Shah, and Anca Dragan. Choice Set Misspecification in Reward Inference. Tn Proceedings of the Workshop on Artificial Intelligence Safety 2020 Co-Located with the 29th International Joint Conference on Artificial Intelligence and the 17th Pacifie Rim International Conference on Arlificial Intelligence, volume 2640. CEUR, 2021. URL https://ceurs weeeurrws.og////..... www./ogglllooooo..o. wrgg/ool2640////0 07640##paper _14
.s.
Rachel Freedman, Justin Svegliato, Kyle Wray, and Stuart Russell, Active teacher selection for reinforcement lcarning from human feodback, 2023. arXiv: 2310.15288. preprint.
Pedro Freire, Adam Gleave, Sam Toyer, and Stuart Russell. DERAIL: Diagnostic Environments for Reward And Imitation Learning. In NewrIPS 2020 Workshop on Deep Reinforcemenl Leurning, 2020.
Justin Fu, Katie Luo, and Sergey Levine. Learning Robust Rewards with Adverserial Inverse Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018a. URL https://openrevii
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition. In Advances in Neural Information Processing Systems (NIPS), volume 31. Curran Associates, Inc., 2018b. URL https://proceeding: patpr///2e/aaa////.rrrrr.eeeeeee -bbttract.. html.
www.eediiiinnnnniss.ooe. nwwrips.rnc.mm cc/
Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2018.
URL https://proceedings mlr, wtt....reeee//////ti/ioooooooooo
Johannes Fürnkranz, Fyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforce- ment learning: A formal framework and policy iteration algorithm. Machine Learnóng, 89(1):123 156, 2012. doi: 10.1007 111007//109944----533313
R. Michael Furr. Psychometrics: An Introduction. SAGE Publications, 2021. ISBN 978-1-07188-400-22
Iason Gabriel. Artificial Intelligence, Values, and Alignment. Minds and Machines, (033)411-4331 2020. doi:
63

---

Published in Transactions on Machine Learning Research (06/2025)
Deep Ganguli, Amanda Askell, Nicholas Schicfer, Thomas I. Liao, Kamilé Lukošiüté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michacl Sellitto, Nelson Elhage, Noemi Mcreado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnstoon, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Tawton, Tom Henighan, Tristan Hume, Yun- tao Bai, Zac Hadfiel-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. The Capacity for Moral Self- Correction in Large Language Models, 2023. arXiv: 2302.07459. preprint.
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In Pro- ceeilings of the International Conference Machine Learning (ICML). PMLR, 2023. URL https: //proceedings. mml. vatsss//v22/ga2221. html.
Yang Gao, Christian M. Meyer, and Iryna Gurevych. APRIL: Interactively Learning to Summarise by Com- bining Active Preference Learning and Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (FMNLP). Association for Computational Linguisties, 2018. doi: 10.18653/v1/
/1//15//////11----/ 1111D11--LL1 1445.
Yang Gao, Christian M. Meyer, and Iryna Gurevych. Preference-based interactive multi-document summari- sation. Information Retrieval Journal, 23(6):555-5885 2020. doi: 10.1007 111100//117911-01--096667
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. DeepMDP: Learn- ing Contimuous Latent Space Models for Representation Learning. Proceedings of the International Conference on Machine Leurning (ICML). PMLR, 2019. URL https://prooceedings.. mlr. gelada19a. html.
wtt.g/s........s/S/1.1. press/v97/
HlnssOOtt Georgii. Gibbs Measures and Phase Transitions. Walter de Gruyter, 2011. ISBN 978-3-11-02502 9.
Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, and Anca D. Dragan. The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Foodback Types. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023. doi: 10.1609 v37.5.23755
Hugo Gilbert and Paul Weng. Quantile Reinforcement Learning. In Asian Workshop on Reinforcement Learning, 2016.
Hugo Gilbert, Olivier Spanjaard, Paolo Viappiani, and Paul Weng. Reducing the Number of Queries Interactive Value Iteration. In Proceedings of Algorillmiic Decision Theory (ADT). Springer International Publishing, 2015. doi: 10.1007 (0077997-3-311 3-1---3------------3-
Hugo Gilbert, Bruno Zanuttini, Paolo Viappiani, Paul Weng, and Esther Nicart. Model-Free Reinforcement Learning with eeem-Symmetric Bilinear Utilitics. In Proccedings of the Conference on Uncertainty in Arlificial Intelligence (UAT), 2016. URL http://auai.orge http//aa2216/prrcceeingn/
pdf..
Hugo Gilbert, Paul Weng, and Yan Xu. Optimizing Quantiles in Preferenco-Based Markov Decision Processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. doi:
10.1609/amn
Amelia Glaese Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phocbe Thacker, Lucy Hamppell-Gillingaan Jonathan Uesato, Po- Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaune Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Folcy, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukeuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, 2022. arXiv: 2209.14375, preprint.
64

---

Published in Transactions on Machine Learning Research (06/2025)
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey interpretable reinforcement learning. Machine Learning, 113(8):5847-580) 2024. doi: 10.1007/s10994 024-06543-w.
Adam Gleave and Geoffrey Irving. Uncertainty Estimation for Language Reward Models, 2022. arXiv: 2203.07472. preprint.
Adam Glcave, Michael D. Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying Differences in Reward Functions. In Proceedings of the International Conference on Learning Representations (ICLR), 2022a. URL https://iclr....
Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Maxi- milian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. Imitation: Clean Imitation Learning Implementations, 2022b. arXiv: 2211.11972. preprint.
Lin Guan, Mudit. Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the Pipeline in Hluman-Guided Reinforcement Learning with Explanation and Context-Awart Data Augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL attps://proceeddngs. wtw.://000002--attatttttttt http/////a........02000666661100
neurips..
Lin Guan, Karthik Valmeckam, and Subbarao Kambhampati. Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https://openrevvee. htt./forrmm?dd-ll2uuuuubu
Arnav Gudibande, Eric Wallace, Charlic Victor Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The False Promise of Imitating Proprietary Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL tttps://openreviex net/ forum?id-kz3yckpOnn
Faruk Gul, Paulo Natenzon, and Wolfgang Pesendorfer. Random Choice Behavioral Optimization. Econo- metrica, 82(5):1873 1912, 2014. doi: 10.3982/ECTAI
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforcod Self-Training (RcST) for Language Modoling, 2023. arXiv: 3308.08998. preprint.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL https: http:///poceeeeeee//eeee mll ww..m.rrr.sss...
html.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor- Critic: Off-Policy Maxi- mum Entropy Deep Reinforccment Learning with a Stochastic Actor. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2018. URL cttps://proceediie mlr wwl...pres//v0 haarnoja180 html.
Soheil Habibian, Ananth Jonnavittula, and Dylan P. Losey. Here's What I've Learned: Asking Questions that Reveal Reward Learning. ACM Transactions on Human-Rowe Interaction, 11(4):40:1-40:28, 2022. doi: 10.1145/3526107.
Björn Haddenhorst, Eyke Hüillermeerr and Martin Kolb. Generalized transitivity: A systematic comparison of concepts with an application to preferences in the Babington Smith model. International Journal of Approzimate Reasoning, 119:373 407, 2020. doi:
Björn Haddenhorst, Viktor Bengs, Jasmin Brandt, and Eyke Hüllermeier. Testification of Condorcet Winners in ducling bandits. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). PMLR, html.
2021. URL https://proceedingg.no mlr. wwwpps/ewwe/aaeeeeeeeeee
65

---

c39944666311771t8722d8888888 Published in Transactions on Machine Learning Research (06/2025)
Dylan Hadffeld-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative Inverse Reinforcement Tearning. In Aduances in Newral Information Processing Systems (NTPS), vol- une 29. Curran Associates, Inc., 2016. URL attps://proceeding: www.:eps..cppaaee///////. Abstract. html.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The Off-Switch Game. In Proceed- ings of the International Joint. Conference on Artificial Intelligence (IJCAT). International Joint Confer- ences on Artificial Intelligence Organization, 2017a. doi: 00.24963/jjcai.200
Dylan HadfieldMMenell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. In- verse Reward Design. In Advances in Neural Information Processing Systems (NIPS), vol- ume 30. Curran Associates, Inc., 2017b. URL Abstract. html.
wwwripe. wttpcccpppp/e///////aaaaaaa/
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning Behav- iors by Latent Imagination. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. URL
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models, 2023. arXiv: 2301.04104. preprint.
Thilo Hagendorff and Sarah Fabi. Methodological reflections for Al alignment research using human leedback, 2022. arXiv: #301.06858 preprint.
Jerry Zhi- Yang He and Anca D. Dragan. Assisted Robust Reward Design. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedingg mlr, press/v164/he22a html.
Donald Joseph Hejna and Dorsa Sadigh. Few-Shot Preference Learning for Human-in-the-Loop RL. In Proceedings of the Conference on Robot Learning (CoRL). PMTR, 2022. URL https:// mlr.pree press/v205/111220 html.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learn- ing. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL pett/forumndd=iiR///O
https://openree
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepes- vari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded Language Learning in a Simulated 3D World, 2017. arXiv: 1706.06551. preprint.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforce- ment Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. doi: 10.1609/aaa) vit.i.
Matthew Hoffman, Eric Brochu, and Nando de Freitas. Portfolio Allocation for Bayesian Optimization. In Proceedings of the Conference on Uncertainly in Artificial Intelligence (UAT). AUAT Press, 2011. doi: 0015555/3320044.330222
Rachel Holladay, Shervin Javdani, Anca Dragan, and Siddhartha Srinivasa. Active comparison based learning incorporating user uncertainty and noise. In RSS 2016 Workshop on Model Learning for Human-Roll Robol Communication, 2016.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human Feodback is not Gold Standard. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL tet//foruum??d==7ooLNNn...
66

---

Published in Transactions on Machine Learning Research (06/2025)
Edward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2024a. URL id-0uj6p4ca60.
wew.eewweeeeeeeeeeeeeeeeeeeeeeee net/forum?
Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-Policy Misalignment in Preference-Bassee Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL http:///pppprroooooooooooooooaoa nett/torunP/dddloByu....
Jie Huang, Jiangshan Hao, Rongshun Juan, Randy Gomez, Keisuke Nakamura, and Guangliang Li. GAN- Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023. doi: M..109/ICRA4S890 1016699...
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ Implementation Details of RLHF with PPO: Case Study on TL:DR Summarization. In Proceedings of the Conference on Language Modeling (COLM); 2024. URL https://openreerie: httppppaareeeeeeeeeeeooorrraiaaa net//orummid k#002TTa@e@Hiiscusiion
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457 506, 2021. doi: 10.1007/s10994
021-05946-3.
Minyoung Hwang, Gunmin Lee, Hogun Kee, Chan Woo Kim, Kyungjae Lee, and Songhwai Oh. Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feodback. In Advances in Neural Information Processing Syslems (NeurIPS), volume 36, 2023. URL https: htttpp://proceeedingg neurips. cc/ paper htappp///..///aee/oooeddeaaaaaae htt.::/eeeaattttoooooooooeaaaaaa ChsttrcctCCoferremn html.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Newral Informalion Processing Systems (NIPS), volume 31. Curran Associates, Inc., 2018. URL https://proceedinn neurips. cc/ Abstract.
Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine, Off- Policy Evaluation via Off-Policy Classification. In Advances in Newral Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019. URL https: /ttps//e///cco/ss...o neurips.cc/paper. 20119//aa55550000007777777666662 Abstrratttta-ttttttttto. html.
Charles Isbell, Christian R. Shelton, Michael Kearns, Satinder Singh, and Peter Stone. social reinforce- ment learning agent. In Proceedings of the International Conference on Autonomous Agents (AGENTS). Association for Computing Machincry, 2001. doi: 10. 10.1145/375735.376333
Ashesh Jain, Shikhar Sharma, Thorsten Joachims, and Ashutosh Saxena. Learning preferences for ma- nipulation tasks from online coactive foodback. The International Journal of Robotics Research, 34(10): 1296 1313, 2015. doi: 10.1177/027836491555111
Natasha Jaques, Shixiang Gu, Damitry Bahdanau, José Miguel Hernández-Lobatoo, Richard E. Turner, and Douglas Eck. Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL https: / //proceedings.. mlr. wtt..press/7///jaaueeeett html.
Erik Jenner and Adam Gleave, Preprocessing Reward Functions for Interpretability. In NeurIPS 2021 Workshop Cooperative AI, 2021.
Erik Jenner, Joar Max Viktor Skalse, and Adam Gleave, A general framework for reward function distances. In NeurIPS 2022 Workshop ML Safely, 2022. URL httpss://peerreriee.. ftt/ 1d=:H22
67

---

Published in Transactions on Machine Learning Research (06/2025)
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying for- malism for reward learning. In Advances in Newral Information Processing Systems (NewrIPS), vol- ume 33. Curran Associates, Inc., 2020. URL nttps://proceedings weteeeeddnngnneuiiss..co / Abstract.htam
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. In Adoances in Neural Information Processing Systems neurips www..ies..cppppprffiiiiiii.. (NeurIPS), volume 36, 2023a. URL Abstract-Datasett and Benchmarks.hhee htmml eect www.rrroeaass---baaat-------aaae
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao Hc, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuchai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Cli Zhu, Yike Guo, and Wen Gao. AI Alignment: A Comprehensive Survey, 2023b. arXiv: 2310.19852. preprint.
Xiang Ji, Huazheng Wang, Minshuo Chen, Tuo Zhao, and Mengdi Wang. Provable Benefits of Policy Learning from Human Preferences Contextual Bandit Problems, 2023c. arXiv: 2307.12975. preprint.
Nan Jiang and Lihong Li. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2016. URL https: //proceedings... mlr. grress/44/jiaang/ html.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the Conference on Learning Theory (COLT). PMLR, 2020. URL https://proceedding mlr. www.ppressvie. h12p//in22/......//// html.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is Pessimism Provably Efficient for Offline RL? In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021. URL U air.press/v139/ in21e. html.
Kshitij Judah, Saikat Roy, Alan Fern, and Thomas Dietterich. Reinforcement Learning Via Practice and Critique Advice. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 24, 2010. doi:
10.1609/aaai (0.1609/aaa..24.1..77 22411.7690.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Gan guli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language Models (Mostly) Know What They Know, 2022. arXiv: 2207.05221. preprint.
Gregory Kahn, Pieter Abbeel, and Sergey Levine. LaND: Learning to Navigate From Disengagements. IEEE Roboties and Automation Letters, 6(2):1872-1879, 2021. doi: 10. 10.1109/L.E LLA.. wwt.LAAA.......... 2020000002000
Akansha Kalra and Daniel S. Brown. Interpretable Reward Learning via Differentiable Decision Trees. In NewrIPS 2022 Workshop on ML Safety, 2022. URL https://opennrevier net/forumm11-33b440M5.
Akansha Kalra and Daniel S Brown. Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback? Reinforcement Learning Journal, 4:1887-1910, 2024.
Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond Reward: Offline Preference-guided Policy Optimization. In Proceedings of the International Conference Machine Learning (ICML). PMLR, 2023. URL https://proceeddnggs wttp:aaww...r....eeeeeeeeee
html.
Sydney M. Katz, Amir Maleki, Erdem Bıyık, and Mykel J. Kochenderfer. Preference-basee based Learning of Reward Function Features, 2021. arXiv: 2103.02727. preprint.
68

---

Published in Transactions on Machine Learning Research (06/2025)
Timo Kaufmann, Sarah Ball, Jacob Beck, Frauke Kreuter, and Eyke Hiilermeier. On the Challenges and Practices of Reinforcement Learning from Real Human Feedback. In ECML. PKDD 2023 Workshop To- wards Hybrid
Learning and Decision Making, 2023.
Hadi Kazemi, Fariborz Taherkhani, and Nasser M. Nasrabadi. Preference-Basee Image Generation. In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020. doi:
10.1109/Wa. AAAACA45572222200909333
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference Transformer: Modeling Human Preferences using Transformers for RT. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2023. URL https://opent: //ppnr//eperreier.. wwtp:/iiwwwwe//ottueeeeeee net/forum? 1d=Peot1SFDXO.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward Identification in Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021. URL
mml.anng...r.....mmmmmmm....mm mlr wttp:emwww....e.seeeeeeee press/v139/kin21c........ html.
Jan H. Kirchner, Logan Smith, Jacques Thibodeau, Kyle McDonell, and Laria Reynolds. Rescarching Alignment Research: Unsupervised Analysis, 2022. arXiv: 2206.02841, preprint.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Lue Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic Motivation from Artificial Intelligence Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: // //openreview..../ hwt.://oror/iiiitttteeeee..
W. Bradlcy Knox. Learning from Human-generated Reward. PhD thesis, The Univer- sity of Texas at Austin, 2012. 20b9e8a1-a78ae 4844488161-
URL https::/reposstoories.. 1ib. utw.astee...o. utexas.edu/item..
W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the International Conference on Knowledge Capture (K-CAP). Association for Computing Machinery, 2009. doi: 10. 10.1155/159773555557775
W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward desidn for autonomous driving. Artificial Intelligence, 316:1 113:1,,,,, 2023. doi: 2.1016/j.artint.20222 103829.
W. Bradley Knox, Stephane Hatgis-Kessell Serena Booth, Scott Niekum, Peter Stone, and Alessandro G. Allievi. Models of human preference for learning reward functions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
Dingwen Kong and Lin Yang. Active Reward Learning. (NeurIPS), volume 35, 2022.
Provably Feedback--fficient Reinforcement Learning via In Advances in Neural Information Processing Systems wttp:euuiipp..ccpppe//////// hash// 17GG886685279888009955544421 wwwtt--attttttttateeeeeeee html.
URL
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdul- lah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul Es, Sameer Suri, David Glushkov, Ar- nav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssis- tant Conversations Democratizing Large Language Model Alignment. In Advances in Neural Information neurips wwteriss.ccppppeeee cc/paper files/ Processing Systems (NewrIPS), volume 36, 2023. URL Abstract-Dataseeeeeeee and Bww.BBennchhh..eeeee html.
Pallavi Koppol, Henny Admoni, and Reid Simmons. Tterative Interactive Reward Learning. Tn ICML, 2020 Workshop on Participatory Approaches to Machine Learning, 2020.
Anna Korba, Stéphan Clemeneon, and Eric Sibony. A Learning Theory of Ranking Aggregation. In Proceed- ings of the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2017. URL https://proceedingss mlr. mlr, wwwss//54/kkoraa177 htmm.
69

---

Published in Transactions on Machine Learning Research (06/2025)
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R. Bowman, and Fthan Perez. Pretraining Language Models with Human Preferences. In Pro- ceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https: //proceedings mlr. press//202/korbakkee html.
Dmitrii Krasheninnikov, Rohin Shah, and Herke van Hoof. Combining Reward Information from Multiple Sources, 2021. arXiv: 2103. 12142. preprint.
Samantha Krening and Karen M. Feigh. Interaction Algorithm Effect on Human Experience with Re- inforcement Learning. ACM Transactions on Human-Robot Interaction, 7(2):16:1 16:22, 2018. doi: 10.1145/3277904
Andras Kupesik, David Hsu, and Wee Sun Lee. Learning Dynamic Kobot-to-Humaal Object Handover from Human Feedback. In Antonio Bicchi and Wolfram Burgard (eds.), Robotics Research: Volume 1, pp. 161-176. Springer International Publishing, 2018. doi: 0..1007/988-3-319-1
10.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward Design with Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https:
//openreview.co wttp://eooooommmmntdooootoooO/
Nathan Lambert. Reward is not enough, 2021.
URL http:///roooti...uustttcee
Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The History and Risks of Reinforcement Learning and Human Foodback, 2023. arXiv: 2310.13595, preprint.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Reward- Bench: Evaluating Reward Models for Language Modeling, 2024. arXiv: 2403. 13787. preprint.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement Learning with Augmented Data. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020. URL. https: httpa:///pooeednns.neeeeeee www.rpp..ccpppppr////ee httpppapppr/222//aaa/aa/ haahh/ Abstract html.
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. 10.1017/97810057711
Cambridge
University Press, 2020. doi:
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch Policy Learning under Constraints. In Proceedings of the International Conference Machine Learning (ICML). PMLR, 2019. URL https://proceedings mlr. htt.ppress/v///// /191a.. html.
Kimin Lec, Kibok Lec, Jinwoo Shin, and Honglak Lec. Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning. In Proceedings of the International Conference Learning Representations (TCLR), 2020. URL h/tppr//e///nrriie...m
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-Pref: Benchmarking Preference- Based Reinforcement Learning. In Proceedings of the Neural Information Processing Sys- tems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), volume 1,
www.nnuu.....pppppee lalppprrfflleeee www.fflllpppeeee silles/paper/22211 2021a. URL hata///eeeafsaaarree66sssssseeee round1 html.
Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: Reecback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021b. URL https://proceeddngg. wtt.ppress/v19/ lee2li.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Picter Abbeel, Mo- hammad Ghavamzadeh, and Shixiang Shane Gu. Aligning Text-to-Image Models using Human Feedback, 2023. arXiv: 2302.12192 preprint.
70

---

Published in Transactions on Machine Learning Research (06/2025)
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulicu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Nick Cheney, Patryk Chrabaszez, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie Forrest, Antoine Fienoy, Christian Gagúe, Leni Le Goff, Laura M. Grabowski, Babak Hodjat, Frank Hutter, Laurent. Keller, Carole Knibbe, Peter Kreah, Richard E. Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Miikkulainen, Sara Mitri, David E. Moriarty, Jean-Baptiste Mouret, Anh Nguyen, Charles Ofria, Mare Parizeau, David Parsons, Robert T. Pemnock, William F. Punch, Thomas S. Ray, Marc Schoenauer, Eric Schulte, Karl Sims, Kenneth O. Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Richard Watson, Westley Weimer, and Jason Yosinski. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life, 26(2):274-305 306, 2020. doi: 10. 10.1162/arl a 00319.
Jan Leike. Why I'm excited about Al-assisted human feedback, 2022. URL https://alignn substack. htmmpp/a--asisseed-aaeesss hwweed--hmaa--eebaaeeae
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction, 2018. arXiv: 1811.07871. preprint.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Penglei Liu. Generative Judge for Eval- uating Aligmment. In Proceedings of the International Conference on Learning Representations (TCLR), 2024. URL
Mengxi Li, Alper Canberk, Dylan P. Losey, and Dorsa Sadigh. Learning Human Objectives from Sequences of Physical Corrections. In Proceedings of the IEEE International Conference on Robolics and Aulomation (ICRA), 2021. doi: 10.1109/ICRAA8B00 S1E/IIAAAA00..... A0R.RA4455..021...0 022.99602...
Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with Human Feedback: Learning Dynamic Choices via Pessimism. In ICML 2023 Workshop Interactive Learning with Implicit Human Feedback,, 2023. URL https://openreviiet
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. URL
Sarah Lichtenstein and Paul Slovie (eds.). The Construction of Preference. Cambridge University Press, 2006. doi: 10.1017/(B09770511111600
Jinying Lin, Zhen Ma, Randy Gomez, Keisuke Nakamura, Bo He, and Guangliang Li. A Review on Interac- tive Reinforcement Learning From Human Social Feedback. IEEE Access, 8:120757 120765, 2020a. doi: 10.1109/ACCESS 111100//ACCESS5222000004444
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human False- hoods. In Proceedings of the Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers) (ACL). Association for Computational Linguistics, 2022. doi: 00.18633vv120222 long.220 229.
Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas, and Paul Weng. Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning. IEFE Robotics and Automation Letters, 5(4):6615 6622, 2020b. doi: 10.1109/LRA.1 w...R...020001111 2020.3013937.
David Lindner and Mennatallah El-Assady. Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning. In IJCAT- ECAI 2022 Workshop on Communication in Human-Al Interaction, 2022.
David Lindner, Rohin Shah, Picter Abbecl, and Anca Dragan. Learning What To Do by Simulating the Past. In Proceedings of the International Conference on Learning Representations (ICLR), 2021a. URL https://openrevier. wwtp://ww...eeeotoor//dd/////a/a
71

---

Published in Transactions on Machine Learning Research (06/2025)
David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek, and Andreas Krause. Information Directed Reward Learning for Reinforcement Learning. In Advances in Neural Information Processing Systems (NewrIPS), volume 34. Curran Associates, Inc., 2021b. URL
neurips.
Abstract.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of Hindsight aligns Language Models with Feedback. In Proceedings of the International Conference on Learning Regresentalions (ICLR), 2024a. URL https: wwwp:eviwww.eettooouuninieieeedd
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-Reward-Nett: Implicitly Differentiable Reward Learning for Preference-bene based Reinforcement Learning. In Advances in Newral Information Processing Systems (NeurIPS), 2022. URL https://openrevien sot//oorumiid-0zzsee-----
Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Zero-shot Cross-task Preference Alignment for Offline RL via Optimal Transport. In NewrIPS 2023 Workshop Optimal Transport and Machine Learning, 2023a, URL
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J. Liu, and Jialu Liu. Statis- tical Rejection Sampling Improves Preference Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL https: http:///pppreviiieeee net/forum? potoorriiieee jSwwbDS
Yi Liu, Gaurav Datta, Ellen Novoseller, and Daniel S. Brown. Efficient Preference- Based Reinforcement Learning Using Learned Dynamics Models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023b. doi: N0.1109/ICRASSO0 2AA.A448.8.......88188 2023.1016108,
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang. Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. Summary of ChallPT-Related research and perspective towards the future of large language models. Miee--Raiooooy, 12):100017, 2023c. doi:
2023.100017.
Dylan P. Losey and Marcia K. O'Malley. Including Uncertainty when Learning from Human Corrections. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2018. URL https:/ mlr. ppwps/v87/ http://8/////ssp/ hoseyysssl111 18a. html.
Dylan P. Losey, Andrea Bajesy, Marcia K. O'Malley, and Anca D. Dragan. Physical interaction as commu- nication: Learning robot objectives online from human corrections. The International Journal of Robotics Research, 41 (1):20 44, 2022. doi: 10.117700283644221050033
R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. John Wiley, 1959.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A Survey of Reinforcement Learning Informed by Natural Language. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Inter- national Joint Conferences on Artificial Intelligence Organization, 2019. doi: 10.24963/ijcai..o 2019/880.
Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine. RLIF: Interactive Imitation Learning as Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https:/ //tpss////peeeeeeee/ /ttpp//rereee.e/eeoorur///////e/
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. In ICLR 2024 Workshop Large Language Model (LLM) Agents, 2024a. URL https: tttpss///penrevie:: net/ formm?:d--9Y7JJ2229..
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eurcka: Human-Level Reward Design via Coding Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL http:///ppppeeeeeeeeeeerrrrrrraa
72

---

Published in Transactions on Machine Learning Research (06/2025)
James MacGlashan, Mark K. Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Human Feedback. Tn
Taylor, and Michael L. Littman. Interactive Learning from Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL https: html.
//proceedings. mml. hatsss//OO/macgaahhhee
Richard Maclin, Jude Shavlik, Lisa Torrey, Trevor Walker, and Edward Wild. Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In Proceedings of the AAAT Conference on Artificial Intelligence. AAAI Press, 2005. URL https: btopp:://aai.... http://aaaiiog/ppppeeeeeee crgg/aapers/00119AMM 0001///A00091.11.55..
Monika Mandl, Alexander Felfernig, Erich Teppan, and Monika Schubert. Consumer decision making in knowledge-based recommendation. Journal of Intelligent Information Systems, 37(1):1 22, 2011. doi: 10.1007//100444-000-0000
Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Tolanda Leite. Aligning Human Preferences with Baseline Objectives in Reinforcement Learning. In Proceedings of the IEEE International Conference on Robotics and Aulomation (ICRA), 2023. doi: 10.1109/TCRA-4891111 202...0112221
Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. The Annals of Statistics, 34(():: 2366, 2006. doi: 10.1214 10.1214/0090555 /00000000000000000060
Lev E. McKinney, Yawen Duan, David Krueger, and Adam Gleave. On The Fragility of Learned Re- ward Functions. In NewrIPS 2022 Workshop Deep Reinforcement Learning, 2022. URL https: // //openreview
Shaunak A. Mehta and Dylan P. Losey. Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction. ACM Transactions on Human- Robot Interaction, 2023. doi: 10.1145/3623384.
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovie, Jeff Schneider, and Willie Neiswanger. Sample Efficiintt Reinforcement Learning from Human Feedback via Active Exploration, 2023. arXiv: 2312.00267. preprint.
Jorge Mendez, Shashank Shivkumar, and Eric Eaton. Lifelong Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS), volume 31. Curran Associales, Inc., 2018. URL https: html.
//papers. wwppers.niss.. wwtpecccppppeeeeeee/ http://////aaeeeeeeeeeeeeeerrera
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillnggann, Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes, 2022. arXiv: 2203.11147 preprint.
Katherine Metcalf, Miguel Sarabia, Natalie Mackraz, and Barry-John Theobald. Sample-Efficient Efficient Preference- based Reinforcement Learning with Dynamics Aware Rewards. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2023. URL https://proceedingg mlr
html.
Yannick Metz, David Lindner, Raphaël Baur, Daniel A. Keim, and Mennatallah El-Assady. A Configurable Interactive Interface for Learning from Diverse Human Feedback. In ICML 2023 Workshop on Interactive Learning with Implicit Human Feedback, 2023. URL http://ppnreeeee..eeettttrrrr/// net/forum?id= JBFZtzJBFQ.
Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, Sharada Mohanty, Byron Galbraith, Ke Chen, Yan Song, Tianze Zhou, Bingquan Yu, He Liu, Kai Guan, Yujing Hu, Tangjic Lv. Federico Malato, Florian Lcopold, Amogh Raut, Ville Hautamäki, Andrew Mclnik, Shu Ishida, João Henriques, Robert Klassert, Walter Laurito, Lucas Cazzonelli, Cedric Kulbach, Nicholas Popovic, Marvin Schweizer, Ellen Novoseller, Vinicius Goecks, Nicholas Waytowich, David Watkins, Josh Miller, and Rohin Shah. Towards Solving Fuzzy Tasks with Human Feedback: Retrospective of the MineRL BASALT 2022 Competition. In Proceedings of the NeurIPS 2022 Competitions Track. PMLR, 2022. URL https:// http:////ooooooooc..o./ mlr. wwa.r.p.es////ee http//220/mmmmmm 2e000mmmaan2222 22a html
73

---

Published in Transactions on Machine Learning Research (06/2025)
Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable Reinforcement Learning: A Survey and Comparative Review. ACM Compuling Surveys, 2023. doi: /316644.
Smitha Milli and Anca D. Dragan. Literal or Pedagogic Human? Analyzing Human Model Misspecification Objective Learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAT).
PMLR, 2020. URL https:
Sören Mindermann and Stuart Armstrong. Occam' razor is insufficient to infer the prefer- ences of irrational agents. In Advances in Newral Information Processing Systems (NIPS), vol- ume 31. Curran Associates, Inc., 2018. URL https://proceednngs. www.eeeeddnnns...ris..... httpsaasss..ppppee////////. 189af6778802200bbbbbafaaaaaaaaaa Abstaract.btttt...oaaooooaaaae.. html.
Sören Mindermann, Rohin Shah, Adam Gleave, and Dylan Hadfelld-Menell Active Inverse Reward Design. In ICML 2018 Workshop on Goals in Reinforcement Learning, 2018.
Abhilash Mishra. AT Alignment and Social Choice: Fundamental Limitations and Policy Implications, 2023. arXiv: 1604,, preprint.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))529 533, 2015. doi:
Ted Moskovitz, Aaditya K. Singh, D. J. Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting Reward Model Overoptimization with Constrained RLHF. In Proceedings of the International Conference on Learning Represcnlations (ICLR), 2024. URL https: //openreview.net// www.:e/e/ooomm/d//tttuooo//oo
Weiyan Mu and Shifeng Xiong. On Huber's contaminated model. Journal of Complexity, 77:101745, 2023. doi: 2..016/)..o..223.1000
Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash Learning from Human Feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL bttps://openrewee wwt.://orumi.ddd//////.........
Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning Multimodal Rewards from Rankings. In Proceedings of the Conference Robol. Learning (CoRL.). PMLR, 2021. URL https://proceeddnng: mlr. casess//16//mers2222 htmml.
Vivek Myers, Erdem BLyik, and Dorsa Sadigh. Active Reward Learning from Online Preferences. Proceedings of the IEEE International Conference on Roboties and Automation (ICRA), 2023. doi:
2.1109/CCRA4889111 200.9.9..02..111 10166049.
Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning Language- Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedingg mlr. wtw.ianns........s/siiiee press/v164/ nair22a. htmm.
Anis Najar and Mohamed Chetouani. Reinforcement Learning With Human Advice: A Survey. Frontiers in Robotics and AT, 8:584075, 2021. doi: (ft.0///rroo.221..1.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser- assisted question-answering with human feedback, 2022. arXiv: 2112.09332. preprint.
74

---

Published in Transactions on Machine Learning Research (06/2025)
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curricu- lum Tearning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learn- ing Research, 21(181):1 50, 2020. ISSN 1533-7928. URL http://jmlr. org wtg/paperr///1/20-2222 2ttp//20-21..mm... html.
Gergely Neu and Csaba Szepesvári. Training parsers by inverse reinforcement learning. Machine Learning, 77(2):303-325 337, 2009. doi: 10. 10.10077/110994400--51111
Andrew Y. Ng and Stuart J. Russell. Algorithms for Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., 2000.
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, In Proceedings of the International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., 1999.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem from Deep Learning Perspective. In Proceedings of the International Conference on Learning Representalions (ICLR), 2024. URL
Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling Posterior Sampling for
Preference-Batee Basod Reinforcement Learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). PMLR, 2020. URL https://proceedingg. mml..mmrr... www.mmllrr.sss..... wtt.:ppre///22//ooseeeeeeee
html.
Ellen Novoseller, Vinicius G. Goecks, David Watkins, Josh Miller, and Nicholas R. Waytowich. DIP-RL: Demonstration-Infered Preference Learning in Minecraft. In ICML, 2023 Workshop The Many Facets of Preference-Beet Based Learning, 2023. URL
OpenAl. Introducing ChatGPT, 2022. URL
(accessed 2024-05-28).
OpenAl. GPT-4 Technical Report Technical report, OpenAI, 2023. URL https://cdn..og wpw.aiii.oooeni....o com/ papers/gpt-4.0
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters, An Algorithmic Perspective on Imitation Learning. Foundations and Trends0 in Robotics, 7(1-2):1-179, 2018. doi: 10.1561 /2300000053.
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic Neural Networks. In Advances in Neural Information Processing Systems (NewrIPS), volume 36, 2023. URL attps://proceeedgs..eeesse www..ps.cc.papp......eeee files/ /aper// -ttpraffffeteetteeeeeettttttttee
Abstract-Conferennee cotm...neeee....e.....
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human foedback. In Advances in Neural Information Process- ing Systems (NewrIPS), volume 35, 2022. URL https://proceetee ngg.ruuupsssss. cc/papper/0222/hahl blera55bbb3044779944458868800000 ott.://8.....11111-----rrrr-r--- html..
Cosmin Paduraru. Off- Evaluation in Markov Decision Processes. PhD thesis, McGill University, 2013. URL
Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning Reward Functions by Integrating Human Demonstrations and Preferences. In Proceedings of Robolics: Science and Systems (RSS), volume 15, 2019. URL http://wwwwrbobocpoooeeeeeeeeeee org/rss15/p23. html.
Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman E. Ozdaglar. RLHF from Heterogeneous Feodback via Personalization and Preference Aggregation. In ICML 2024 Workshop
on Aligning Reinforcement Learning Experimentalists and Theorists (ARLET), 2024. URL https: //openreview wwt.://ooooommmmmaa/aaaaaaa
75

---

Published in Transactions on Machine Learning Research (06/2025)
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lec. SURF: Semi- supervised Reward Tearning with Data Augmentation for Feedback-effiiient Preference-based Reinforce- ment Learning, In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
URL
Robert Pinsler, Riad Akrour, Takayuki Osa, Jan Peters, and Gerhard Neumann. Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences. In Proceedings of the IEEE In- ternational Conference on Robotics and Automation (ICRA), 2018. doi: 10.1109/ICRA, 20C.R...18.8460007
André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning Computer Vision Models With Task Rewards. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https::/proceeee
mlr press/v202/ susano -intno2-pint2aaa ham.to23a.tml..
R. L. Plackett. The Analysis of Permutations. Journal of the Royal Statistical Society. Series C' (Applied Statistios), 24(2):193-25) 202, 1975. doi: 10.2307/2346507.
Alina Pommcranz, Joost Brockens, Pascal Wiggers, Willem-Paul Brinkman, and Catholijn M. Jonker. De- signing interfaces for explicit preference elicitation: A user-centered investigation of preference represen- tation and elicitation process. User Modeling and User- Adapted Interaction, 22(4):357-38) 2012. doi:
10.1007/11225--01111166..
Benjamin Poole and Minwoo Lee. Towards interactive reinforcement learning with intrinsic leedback. New- rocomputing, pp. 127628, 2024. doi: 10.1016/j ne..oo....euuommm..m 2tc.ucco..222.122..
Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility Traces for Off-Policy Policy Evalua- lion. In Proceedings of the International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., 2000.
Erika Puiutta and Eric M. S. P. Veith. L'xplaiablee Reinforcement Learning: A Survey. In Proceedings of Machine Learning and Knowledge Extraction (CD-MAKE). Springer International Publishing, 2020. doi:
Junqi Qian, Paul Weng, and Chenmien Tan. Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning. In Proceedings of the International Conference on Autonomous Agents and Mullingent Systems (AAMAS). IFAAMAS, 2023. doi: N1.5555/3545946 3398864..
Yumpeng Qing, Shunyu Liu, Jie Song, Huiqiong Wang, and Mingli Song. A Survey on Explainable Rein- forcement Learning: Concepts, Algorithms, Challenges, 2023. arXiv: 2211.06665. preprint.
Mattia Racca, Antti Oulasvirta, and Ville Kyrki. Teacher-Aware Active Robot Learning. In Pro- ceedings of the ACM/TEEE International Conference on Human- Robot Interaction (HRI), 2019. doi: D0.110//HR..001..88
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems (NewrIPS), volume 36, 2023. URL https: / /ttp:///pappaa/eeeeee mips. Awtpeecttttttttttt--CCaate Cnistract-Coofereecce Conference html.
paper_ftt htt.l///wpppp...../.aaaaaaaaaaaa
Matthew Rahtz, Vikrant Varma, Ramana Kumar, Zachary Kenton, Shane Legg, and Jan Leike. Safe Deep RL in 3D Envirouments using Human Feedback, 2022. arXiv: 2201.08102, preprint.
Rajkumar Ramanurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is Reinforcement Learning (Not) for Natural Lan- guage Processing: Benchmarks, Basclines, and Building Blocks for Natural Language Policy Optimiza- tion. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL attps://openreet wwwp://eeeetttneeeeeeeee wtt.://ooorrr//ieeaaaaaaaa
76

---

Published in Transactions on Machine Learning Research (06/2025)
Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the Benefits of Weight Averaged Reward Models. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https://openreview.neett net/forum? id=s7RDnNUJy6.
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning Human Objectives by Evaluating Hypothetical Behavior. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2020. URL
.mr.aingsmmrrr....mmmmmmmmm mlr www...pressvviiv h1t/1119////ede htmml.
Kevin Regan and Craig Boutilier. Regret-based reward elicitation for Markov decision processes. In Pro- ceedings of the Conference on Uncertainty in Artificial Intelligence. (UAT). AUAT Press, 2009. URL
https://dl. awtp: acm. 4 1795166.
Kevin Regan and Craig Boutilier. Robust online optimization of reward-uncertain MDPs. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). AAAI Press, 2011. doi: 10.5591/978-
1-57735-5166 IJCAI11-361.
Zhizhou Ren, Anji Liu, Yitao Liang, Jian Peng, and Jianzhu Ma. Efficient Meta Reinforcement Learning for breserence-based Fast Adaptation. In Advances in Neural Information Processing Sys- tems (NewrIPS), volume 35, 2022. URL https://pape
catps.cc/papppe ftllss/paper/2222//aaee cooonfrrenceeht..
Carl Orge Retzlaff, Srijita Das, Christabel Wayllace, Payam Mousavi, Mohammad Afshari, Tianpei Yang, Anna Saranti, Alessa Angerschmid, Matthew E. Taylor, and Andreas Holzinger. Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities. Journal of Artificial Intelligence Research, 79:359 415, 2024. doi:
Daniel Russo and Benjamin Van Roy. Eluder Dimension and the Sample Complexity of Op- timistic Exploration. In Advances in Neural Information Processing Systems (NIPS), vol- ume 26. Curran Associates, Inc., 2013. URL https://papers: ccpp.cc/paaprr/00/ haah/////hhhh/////////es 44F22000bbbbberaacc00450000--a htm..
John Rust. Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher. Econo- metrica, 55(5) (555))999-0033 1987. doi: 10.2307/1911255.
Mandy Ryan, Karen Gerard, Mabel Amaya-Amaya, and Ian J. Bateman (eds.). Using Discrete Choice Experiments to Value Health and Health Care, volume 11. Springer Netherlands, 2008. doi: 10.1007/978.
1-4020-5753-3.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active Preference-Bart Learning of Reward Functions. In Proceedings of Robotics: Science and Systems (RSS), volume 13, 2017. URL http: //ww. roboticsproceedingg. org/rss13/p53. html.
Aadirupa Saha. Optimal Algorithms for Stochastic Contextual Preference Bandits. In Advances in Neural Information. Processing Systems (NewrIPS), volume 34. Curran Associates, Ine., 2021. URL https:// wwweeebbbbtttttttee html.
Aadirupa Saha and Akshay Krishnamurthy. Efficient and Optimal Algorithms for Contextual Dueling Ban- dits under Realizability. In Proceedings of the International Conference on Algorithmic Learning Theory (ALT). PMLR, 2022. URL httpp::/prooeedings. mlr www...reee/vv.///aaaaa h1tl/////////aaaaaaa/ html.
Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling RL: Reinforcement Learning with Trajec- tory Preferences. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2023. URL https::/proceete
mlr waw....reeevv2000 sawa//aa.aaaa.aaaaaaaaaaaaaaaaaa html.
William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. Trial without Error: Towards Safe Reinforcement Learning via Human Intervention. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 2018. URL https://dl acm. org/
77

---

Published in Transactions on Machine Learning Research (06/2025)
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. arXiv: 2206.05802. preprint.
Mare Schoenauer, Riad Akrour, Michele Sebag, and Jean- Chan-Chrittophe Souplet. Programming by Feedback. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2014. URL https: / //proceedings.. mlr..mr.... wal..press. wwwppprsssssieee paress/v32ssee wtt////hoeaeeerlaa... html.
John Schulman. Reinforcement Learning from Human Feedback: Progress and Challenges, 2023. URL
attps://wwz https://oww.yuuuube..
youtube. hat.:///...oatttt///tssssgg_/__
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Opti- mization Algorithms, 2017. arXiv: 1707.06347. preprint.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-Efficient Reinforcement Learning with Self-Prrdictive Representations, In Proceedings of the Inter- national Conference on Learning Representations (ICLR), 2021. URL httpas////pereii...no id=uCQfPZwRauu.
Ayush Sekhari, Karthik Sridharan, Wen Sun, and Runzhe Wu. Contextual Bandits and Imitation Learn- ing with Preference- Active Queries. In Advances Newral Information Processing Systems (NeurIPS), volume 36, 2023. URL https///rroooeeeiingg neerruuupddnggg.uuprpp....... neurips wwwarris.ccpppprffiiiii hiapprr_fllessppee/222222 hashh/ html.
Burr Settles. Active Learning. Morgan & Claypool Publishers, 2012. ISBN 978-1-60845-725-0.
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. Preferences Im- plicit in the State of the World. In Proceedings of the International Conference Learning Representations (ICLR), 2019. URL http:////eeeeeeeeeeeeeeee/rrr//r
Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii Krasheninnikov, Lawrence Chan, Michael D. Dennis, Picter Abbeel, Anca Dragan, and Stuart Russell. Benefits of Assistance over Reward Learning, 2021a. URL
preprint.
Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, and Anca Dragan. The MineRL BASALT Competition on Learning from Human Feedback, 2021b. arXiv: 2107.01969 preprint.
Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The Trickle-down Impact of Reward Inconsistency on RLHF. In Proceedings of the Inter- national Conference on Learning Representations (ICLR), 2024. URL https: http://///eeeee..eee/eeeeee id=MeHm=CDifc.
Daniel Shin, Anca Dragan, and Daniel S. Brown. Benchmarks and Algorithms for Offline Preference- Based Reward Learning. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL https://openrevien net/ net:/forum/dd-TCuXX100
Pannaga Shivaswamy and Thorsten Joachims. Online structured prediction via coactive learning. In
Proceedings of the International Conference on Machine Leurwing (ICML). Omnipress, 2012. URL
http://icml..e cc/ ctt/cc/////////cc/ 20001//paprs/117 /ddf
Pannaga. Shivaswamy and Thorsten Joachims. Coactive Learning. Journal of Artificial Intelligence Research, 53:1-40, 40, 2015. doi: (0.1613/jain 4539.
Umer Siddique, Abhinav Sinha, and Yongcan Cao. Fairness in Preference-basee Reinforcement Learning. In ICML 2023 Workshop on The Many Facets of Preference-Bass -aseed Learning (MFPL), 2023. URL https: // //operrviiew wttp:/ooommiiiaooaa1ara..
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. doi: 10. 20.1016/j.rrtin.2222 103535.
78

---

Published in Transactions on Machine Learning Research (06/2025)
Avi Singh, Larry Yang, Chelsea Finn, and Sergey Levine. End-To-End Robotic Reinforcement Learning without Reward Engineering. In Proceedings of Robolies: Science and Systems (RSS), volume 15, 2019. URL attp://wwwt
htt.tmmmmp....m..... html.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadficld-Menel. Understanding Hidden Context Preference Learning: Consequences for RLHF. In Proceedings of the International Conference on Learning Represenlations (ICLR), 2024. URL
Joar Max Viktor Skalse and Alessandro Abate. The Reward Hypothesis is False. In NewrIPS 2022 Workshop on ML Safety, 2022. URL https://openreviil http://ww.....oommmmmmm/.ooloooo
Joar Max Viktor Skalse and Alessandro Abate. Misspecification in Inverse Reinforcement Learn- ing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, 2023. doi: 10.1609/ 10966999...... vttp//aai.v37.2.222222 v37.12.266666
Joar Max Viktor Skalse and Alessandro Abale. Quantilying the Sensitivity of Inverse Reinforcement Learning to Misspecification. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: https/////p/peeeeeeeeeeeee htt.:/eeetetteeeeeeeeeeeeeeeeee http:/rrrmm2ddp2210000m.a
Joar Max Viktor Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defin- ing and Characterizing Reward Gaming. In Advances in Newral Information Processing Systems (NeurIPS), volume 35, 2022. URL https: iorp://rrooeedngn..rrr.pspps peurips.cc/paper 3d7 B Abstract-
html.
Joar Max Viktor Skalse, Matthew Farrugia-Roberte Stuart Russell, Alessandro Abate, and Adam Gleave, Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedinngg mlr.
press/v202/ssallse23. hmmm..
Joar Max Viktor Skalse, Lucy Farnik, Suneet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessan- dro Abate. STARC: A General Framework For Quantifying Differences Between Reward Functions. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: / //openreview weet/forumiiddwwhbbaa
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Prefer- ence Ranking Optimization for Human Alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024. doi: 10.1609/ wwt./aaa....i.....a.... v38i17.29865
Ziang Song, Tianle Cai, Jason D. Lee, and Weijie J. Su. Reward Collapse in Aligning Large Language Models: A Prompt-Aware Approach to Preference Rankings. In JCML, 2023 Workshop on The Many Facels of Preference-Bert Based Learning (MFPL), 2023. URL https://openrer// wwwprrevieeneetfffoooo nett//oruunid-appkkaaaa
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L. Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A Roadmap to Pluralistie Alignment. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https: / //tp///prooo http///ooeeenn..ee/ee mlr, press/ v235// sorensen24a. html.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodci, and Paul F Christiano. Learning to summarize with human feodback. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020. URL https:// neurips. senceeddings.nurip.......
proceedings
cc/ cc/ppaper/020/// httpa////oooooeeeee/eeaaaeeeeees (tt/:///00000000000000404-----10
Hiroaki Sugiyama, Toyomi Meguro, and Yasuhiro Minami. Preference-learnnng based inverse reinforcement learning for dialog control. In Proceedings of Interspeech. ISCA, 2012. doi: 10.21437 10.21437/Interspeec, http://eeeeeeehh2122222 2012-72.
Yanan Sui, Vincent Zhuang, Joel W. Burdick, and Yisong Yuc. Multi-ducling Bandits with Dependent Arms In Proceedings of the Conference on Uncertainty Artificial Intelligence (UAI). AUAI Press, 2017. URL http:://uai.org///a/ http:///aaa..cr//a//00///oooceee
pdf.
79

---

Published in Transactions on Machine Learning Research (06/2025)
Yanan Sui, Masrour Zoghi, Katja Hofmann, and Yisong Yue. Advancements in Ducling Bandits. In Pro- ceedings of the International Joint Conference on Arlificial Intelligence (IJCAT). International Joint Con- ferences on Artificial Intelligence Organization, 2018. doi: 10.24963 1144963//jcai20000 /j1aa..018/776..
Theodore Sumers, Robert Hawkins, Mark K. Ho, Tom Griffiths, and Dylan Hadfield-Ment How to talk so AT will learn: Instructions, descriptions, and autonomy. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022a. URL https://proceeding petproeeedingn.eurrpiiis..s cttppper_ppapppeeeeeeeeee feraperfflless fitles/ppppe/////////// ottp://reeettooeeooaaaaaarorrrre Conerrroooeeeeeeneeeeee html.
Theodore Sumers, Robert D. Hawkins, Mark K. Ho, Thomas L. Griffiths, and Dylan HadfieldMMenell. Lin- guistic communication as (inverse) reward design. In ACI. Workshop on Learning with Natural Language Supervision, 2022b. URL attps://openrevieer www.neeettorneeee forumr71dd-BBBuuu22asss
Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, and Thomas L. Griffiths. Learn- ing Rewards From Linguistic Feodback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021. doi: 10.1609 w.5....33...17..
Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, and Thomas L. Griffiths. Show or tell? Exploring when (and why) teaching with language outperforms demonstration. Cognition, 232:105326, 2023. doi: 10.1016/3.cognition 2et...02.2120... 105326.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenlang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. SALMON; Alignmmnt with Instructable Reward Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreeviev. pet/forrumiidxxxemmBBeee
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition edition, 2018. ISBN 978-0-262-03922-66
Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alckh Agarwal. A Minimaximalist Ap- proach to Reinforcement Learning from Human Feedback. In Proceedings of the Intervational Conference on Machine Learning (ICML), PMLR, 2024. URL nttps://proceeding. mlr. h.r..pesss/v255 swamy24a. html.
Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang. Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL
Louis Leon Thurstonc. A law of comparative judgment. Psychological Review, 34:273-288 286, 1927. doi: 10.1037/h0070288.
Jeremy Tien, Jerry Yang He, Zackory Erickson, Anca Dragan, and Daniel S. Brown. Study of Causal Confusion in Preference-Basee Reward Learning. In ICML 2022 Workshop on Spurious Correlations, Invariance and Stability, 2022. URL https://opperrevier.
Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S. Brown. Causal Confu- sion and Reward Misidentiffcation in Wraseednce-Based Reward Learning. In Proceedings of the Interna- tional Conference Learning Representations (ICLR), 2023. URL https: sttpss//opperreviet httppppereeee.eeeetorrrrr net/forum? id=ROXxvr X3ZA.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedamuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosscini, Rui Hou, Hakan Inan, Marcin Kardas, Vikior Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Fllen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng
80

---

Published in Transactions on Machine Learning Research (06/2025)
Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Eduuov, and Thomas Scialom. Llama 2: Open Foundation and Chat Models, 2023. arXiv: 2307.09288, preprint.
Kenneth E. Train. Discrete Choice Methods with Simulation. Cambridge University Press, 2 edition, 2009.
doi: 10.1017/ (0007/CB09780511805277
Thi Ngoc Trang Tran, Alexander Felfernig, and Nava Tintarev. Humanized Recommender Systems: State- of-the-art and Research Issues. ACM Transactions on Interactive Intelligent Systems, 11(2):9:1 9:41, 2021. doi: 10.1145/344000 3446906.
Maegan Tucker, Kejun Li, Yisong Yue, and Aaron D. Ames. POLAR: Preference Optimization and Learning Algorithms for Robotics, 2022. arXiv: 2208.04404. preprint.
L. G. Valiant. A Theory of the Learnable. Communications of the ACM, 27(11):1134-114 1144, 1984. doi: 10.1145/1968.9972.
Peter Vamplew, Benjamin J. Smith, Johan Källström, Gabriel Ramos, Roxana Rădulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Picter J. K. Libin, Richard Dazeley, and Cameron Foale. Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2):41, 2022. doi: 10.1007 0007//10458-000 10014480222-997-----
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Aduances in Neural Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017. URL https://papers www.iiiipsssaiiee. wwtpap/ee//////////eeeeeeee 2ffa2225447ee1111t00111444488888 Aaaarcc4assaaaaastaaaatt html.
Mudit Verma and Subbarao Kambhampati. Data Driven Reward Initialization for Preference based Rein- forcement Learning. In AAAI 2023 Workshop on Representation Learning for Responsible Human-CCntrree AI, 2023a.
Mudit Verma and Subbarao Kambhampati. A State Augmentation based approach to Reinforcement Learn- ing from Human Preferences. In AAAI 2023 Workshop on Representation Learning for Responsible Human- Centric AI, 2023b.
Mudit Verma and Katherine Metcalf. Hindsight PRIORs for Reward Learning from Human Preferences. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: / /openrevies bwopeereviee.net/oooooeeeeeeeeee
Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. Exploiting Unlabeled Data for Feedback Efficient. Human Preference based Reinforcement Learning. In AAAT 2023 Workshop on Representation Learning for Responsible Human-Centric AI, 2023.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond Reverse KL: Generaliz- ing Direct Preference Optimization with Diverse Divergence Constraints. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2024a. URL https://opentee httpas/////eeeeeeeeeeeeee net/forum? 1d=2oRzmXXK9N.
Haoran Wang, Qiuye Jin, Shiman Li, Siyu Liu, Manning Wang, and Zhijian Song. comprehensive sur- vey on deep active learning in medical image analysis. Medical Image Analysis, 95:103201, 2024b. doi: ma.10166j.mmdia.2000 1032001.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). Association for Computational Linguisties, 2023a. doi: 10.18653/v1 1222222.cclooooon
81

---

/Tt/://0S0507422200000448.8 Published in Transactions on Machine Learning Research (06/2025)
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF More Difficult than Standard RT? A Theoretical Perspective. In Advances in Newral Information Processing Systems (NeurIPS), volume 36, 2023b. URL https://papern: nips wwwppprrrcpppeppppppppppppp...ep cc/paper_files p11es//pperr/223/ hash/ 3t965977550900447744444668888882
Costrrctt-Coferemin html.
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RL- VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024c. URL https://openreee wtwpopeereewe..eeeeeoeoeeee net/forum? id=YSoMmNWZ2x.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACT-HLTT). Association for Computational Linguistics, 2024d. doi: 10.18653/vv
185.
Zizhao Wang, Junyao Shi, Iretiayo Akinola, and Peter Allen. Maximizing BCI Human Feedback using Active Learning. In Proceedings of the TEFE/RSJ International Conference Intelligent Robots and Systems (IROS), 2020. doi:
Garrett Warnell, Nicholas Waylowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interactive Agent Shaping in figh-Dimensional State Spaces. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 32, 2018. doi: 22009/aaii.v22ii
Paul Weng and Bruno Zanuttini. Interactive value iteration for Markov decision processes with unknown rewards. In Proceedings of the International Joûnd Conference on Artificial Intelligence (IJCAT). AAAT Press, 2013. URL https:// http::/wwwwwjjca.a
pdf.
Nils Wilde and Javier Alonso-Mora. Do use the Right Measure? Challenges in Evaluating Reward Learning Algorithms. In Proceedings of the Conference on Robol Learning (CoRL). PMLR, 2022. URL attps://proceedings mlr, wwwpseeeeeOS
Nils Wilde, Dana Kulić, and Stephen L. Smith. Learning User Preferences in Robot Motion Planning Through Interaction. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018. doi: 10.1109/ICRA 2018.8460586.
Nils Wilde, Dana Kulić, and Stephen L. Smith. Active Preference Learning using Maximum Regret. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robols and Systems (IROS), 2020.
doi:
Nils Wilde, Erdem Biyik, Dorsa Sadigh, and Stephen L. Smith. Learning Reward Functions from Scale Feedback. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https: //proceedings. mll.. pross/v164/wildeee html.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian Approach for Policy Learning from Trajectory Preference Queries. In Advances in Neural Information Processing Systems (NIPS), vol- une 25. Curran Associates, Inc., 2012. URL
1622222a119980008881777666666 htw.://77.......t5-ttttt.tet.e html.
Christian Wirth and Johannes Fürnkranz. EPMC:
Every Visit Preference Monte Carlo for Reinforcement Learning. In Proceedings of the Asian Conference on Machine Learning (ACML). PMLR, 2013a. URL
https://proceedings.
Christian Wirth and Johannes Fürnkranz. A Policy Iteration Algorithm for Learning from Preferenco-Based Feedback. In Advonces in Intelligent Data Analysis (IDA). Springer, 2013b. doi: 10. 10.100779978-36422-1133 8 37.
82

---

Published in Transactions on Machine Learning Research (06/2025)
Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-Free Preference-Baue Reinforcement Learning. In Proceedings of the AAAT Conference on Artāficial Intelligence. AAAT Press, 2016. doi: 10.1609/aaaie vaa...30....0265
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A Survey of Preference-Bace Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1-46, 2017. ISSN
1533-7928. URL http://jmml org/papprs/v18/// (apperrsv18/16-636 11t/166634.hmm...
Fan Wu, Huseyin A. Inan, Arturs Backurs, Varun Chandrasekaran, Janardhan Kulkarni, and Robert Sim. Privately Aligning Language Models with Reinforcement Learning. In Proceedings of the Inter- national Conference on Learning Representations (ICLR), 2024. URL https://openrevier. net/forum? id=3d00mYTNui.
Huasen Wu and Xin Liu. Double Thompson Sampling for Dueling Bandits. In Advances in Neural Informa- tion Processing Systems (NIPS), volume 29. Curran Associates, Inc., 2016. URL https://papers. nips. papaperffllss.. https//rseeeeeeeeee/o/./ wttp://aappeee1/////// hetpe1//hahh//hhe http///e/eeeeee1e1ef91/0/0/////0 htteeee111.000ddaaaaaeeeeereeeee http://seeeeeeee-aabbbttt..ta..e
html.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiemnon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively Summarizing Books with Human Feedback, 2021. arXiv: 2109.10862. preprint.
Runzhe Wu and Wen Sun. Making RL with Preference-based Feedback Efficient via Randomization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https:
//openreview
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Aligument. In NewrIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL yQT406rH72.
Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. A survey of human-in-the-loop for machine learning. Future Generalion Computer Systems, 135:364 381, 2022. doi: (0.1016/j.futurre2200 201.0000.....
Blake Wulfe, Logan Michael Ellis, Jean Mercat, Rowan Thomas McAllister, and Adrien Gaidon. Dynamics- Aware Comparison of Learned Reward Functions. In Proceedings of the International Conference Learning Representations (ICLR), 2022. URL /tttss///penreview.. net nett://orumidd-AAAPyVVseee
Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Pooven- dran. FRESH: Interactive Reward Shaping in Highh-Dimensional State Spaces using Human Feedback. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS).
TFAAMAS, 2020. doi: 10.5555/339874 .3398935.
Annie Xie, Avi Singh, Sergey Levine, and Chelsea Finn. Few-Shot Goal Inference for Visuomotor Learning and Planning. In Proceedings of the Conference on Robot Learning (CoRT.). PMIRR, 2018. URL https: //proceedings mlr. ppwss/v87/ / xie18a.ee..eem html.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Rext2Reward: Reward Shaping with Language Models for Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https: //oenreviie
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong, ImagoReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems (NewrIPS), volume 36, 2023. URL https: httppsss///pprs../// httppppprr..ii...... cc/ Abstrtttbbttttt-eeeeeeaa-eaaaaae ConeetCooooeeeeeeeeee html.
paper_
83

---

Published in Transactions on Machine Learning Research (06/2025)
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Aligmment? A Comprehensive Study. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https: https://proceedings. mlr. press/v235/xu24h xtml..mmm.
Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference- Reinforcement Learning with Finite-Time Guarantees. In Advances in Neural Information. Processing Systems (NeurIPS). volume 33. Curran Associates, Inc., 2020. URL Abstract. html.
peerrips.cc/pppe/222000 /hash/
Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). Association for Computing Machinery, 2023. doi: 10.11455/550335555594477
Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement Learning from Diverse Human Pref- erences. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), volume 6, 2024. doi:
Jingfeng Yang. Capability Alignment? Respect the LLM Base Model's Capability During Alignment, 2024. URL https://jingfeegyaag github alggmitnmn
Georgios N. Yannakakis and Héctor P. Martínez. Ratings are Overrated! Frontiers in ICT, 2:13, 2015. doi: 223399/fict.215..0
Chenlu Ye, Wei Xiong, Yuhong Zhang, Nan Jiang, and Tong Zhang. A Theoretical Analysis of Nash Learning from Human Feedback under General KL-Regularized Preference, 2024. arXiv: 2402.07314. preprint.
Tianhe Yu, Ted Xiao, Jonathan Tompson, Austin Stone, Su Wang, Anthony Brohan, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Karol Hausman, Brian Ichter, and Fei Xia. Scaling Robot Learning with Semantically Imagined Exporience. In Proceedings of Roboties: Science and Systems (RSS), volume 19, ottpr//crrrreeiiiierrerrrr////// wtggrssa9/p//// p027 html.
2023. URL httpp://ww..rbbriiepproooocco/cc /ttp///wwwooocccpoceeeeaaeeeeeee
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank Responses to Align Language Models with Human Feedback. In Advances in Neural Information Processing Systems (NewrPPS) volume 36, 2023. URL https://proceee
neurips. cc/ wwcp/appr_ffttte siles/paper/ html.
Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng, Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreviie teet/forummiiiee id=NesYOHgghM.
Yisong Yue and Thorsten Joachims. Interactively optimizing information retrioval systems as ducling bandits problem. In Proceedings of the Internalional Conference on Machine Learning (ICMT.. Association for Computing Machinery, 2009, doi: 10.1145/ 151145//153374.1555227
Huixin Zhan, Feng Tao, and Yongcan Cao. Human-Guided Robot Behavior Learning: A GAN-Assisted Cisterenco-Basee Reinforcement Learning Approach. IEEE Robotics and Automation Letters, 6(2) 35555 3552, 2021. doi: 10.1109/IR.. 2.2....22...... .3063927.
Wenhao Zhan, Masatoshi Uchara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable Offine Preference- Based Reinforcement Learning. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2024a. URL
Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason D. Loe. Provable Rewari-Agnostic Sisterenco-Basde Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICL.R), 2024b. URL http:///pppppeeeeeeeeeeeert/rr//
84

---

Published in Transactions on Machine Learning Research (06/2025)
David Zhang, Micah Carroll, Andreea Bobu, and Anca Dragan. Time-Efficient Reward Learning via Visually Assisted Cluster Ranking, 2022. arXiv: 2212.00169. preprint.
Guoxi Zhang and Hisashi Kashima. Learning state importance for preference-based reinforcement learning.
Machine Learning, 2023. doi: 10.
Han Zhang, Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, and Ruifeng Xu. CPPO: Continual Learning for Reinforcement Learning with Human Feedback. In Proceedings of the International Conference on .ntt/forrmm2
Learning Representations (ICLR), 2024. URL https:
Ruohan Zhang, Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in leveraging human guidance for sequential decision-making tasks. Autonomous Agents and Multi-Agent Systems, 35(2):31,
2021. doi: 10.1007 (0007//00458-021 021--2511---w
Tianjun Zhang, Fangchen Liu, Justin Wong, Picter Abbeel, and Joseph E. Gonzalez. The Wisdom of Hind- sight Makes Language Models Better Instruction Followers. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023a. URL https://prooceeding mlr. www.mmlw.......... gress//2022zhaagg330 html.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's 's Song in the AI Occan: A Survey on Hallucination in Large Language Models, 2023b. arXiv: 2309.01219 preprint.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Soquence Likelihood Calibration with Human Feodback, 2023. arXiv: 2305.10425, preprint.
Rui Zheng, Wei Shen, Yuan Hua, Wenbin Lai, Shihan Dou, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Haoran Huang, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving Generalization of Alignment with Human Preferences through Group Invariant Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL htttss://pperevviie .net/ htt///frmmidddfOooa
Huiying Zhong, Zhun Deng, Weijie Su, Zhiwei Steven Wu, and Linjun Zhang. Provable Multi-Party Reinforcement Learning with Diverse Human Feedback, 2024a. arXiv: 2403.05006. preprint.
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto Alignment via Preference Adaptation for LLMs. In Advonces in Newral Information Processing Systems (NewrIPS), volume 37, 2024b. URL https:///proooeeeiiigg neurips. cc/ html.
paper_ www.ffileeppppe///eee
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuczhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zeltlemoyer, and Omer Levy. LIMA: Less Is More for Alignment. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL ptttpee//prooeeeinggs nwwrips.erpppcccc httpiipppcpppppeeaafeeeeeee files/paper/2006 /atee//a///////////// 4c662178880844777111114477788 -tw.:////77...0aaaaaaaaaa-------
Abstract-
html.
Wangchunshu Zhou and Ke Xu. Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020. doi: 10.1609/ 10.1609/aaai.v3/4 vaai.v34105.6521.
Yifei Zhou, Andrea Zanelle, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https://proceediing mlr. pptss//235/zhouss/ html.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled Reinforcement Learning with Human Feedback from Pairwisc or K-wise Comparisons. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023a. URL stttss//proceeeeee
mwr. aal..pprea//v02//hn2222 .html.
Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I. Jordan, and Jiantao Jiao. Fine- Fine-Tuming Language Models with Advantage-Inducedd Policy Alignment, 2023b. arXiv: 2306.02231 preprint.
85

---

Published in Transactions on Machine Learning Research (06/2025)
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF. In Proceedings of the International Conference on. Machine Learning (ICML), 2024. CRL https:// http:///ppeeeeeeeeeeeeeeeerrttae wwt.//oooumiddwww.nM.......
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforce- ment learning. In Proceedings of the 23rd National Conference Artificial Intelligence Volume 3. AAAI Press, 2008. ISBN 978-1-57735-368-3, URL http: http://www.r www.///wwwaaaaaaaaaaaaaaaaaaaaaa .rg. wwg/Library/AAA//DO0 /tt.AAAA///00////a000-00 227. php.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2020. arXiv: 1909.08593.
preprint.
A RLHF for Language Models
Generative models, particularly LLMs, have significantly driven recent interest and advances in RLHF. These models are generally trained on vast amounts of data using self-supervised pretraining objectives that are not aligned with their intended usage. RLHF can be used for preference-basee fine-tuning of these models, enabling them to generate more helpful, honest, and harmless outputs (Askell et al., 2021). This has highlighted the potential of RLHF to enhance the performance and adaptability of intelligent systems across a wide range of applications.
The application of RLHF to LLMs can be traced back to early works focused on summarization (Böhm et al., 2019; Zicgler et al., 2020; Sticnnon et al., 2020). This line of work considers a sctting where a human provides leedback on language model outputs, such as text summaries, which is then used to train a reward model that guides the language model towards generating better outputs. In order to apply RLHF techniques to this setting, the language model is treated as an RL agent, with the reward model providing the feedback that guides the agent's learning.
The corresponding RL problem is characterized by an MDP where the agent's actions correspond to generat- ing text tokens, the state is the generated text so far, and the reward is provided by the reward model based on the human feedback. This is most commonly framed as a single-turn interaction with sparse reward, i.e., the episode consists of generating single output (e.g., summary or dialogue response) with single reward at the end evaluating the quality of the output.
The process typically starts with supervised fine-tuning on small dataset labeled with the desired behavior (behavioral cloning). Empirical evidence shows that supervised fine-tuning alone is often insufficient, how- ever, motivating preferenco-bast based finc-tuning (Stiennon et al., 2020). This may be due to the limited size and diversity of the supervised fine-tuning data, leading to overfitting and poor generalixation. Preference data is often more diverse and can be collected more efficiently, helping the model learn a more robust and generalizable objective. This is further supported by Ramamurthy et al. (2023) who find that training reward model from preference data which can then be used for policy optimization can be much more data efficient than supervised fine-tuning. Using less capable model for the reward model may further improve generalization, as it is less likely to overfit the training data. Additionally, RLHF can train the model to give responses or refuse the response based on its own knowledge boundary, which is tricky to achieve with supervised fine-tuning since the label needs to depend on the model's state of knowledge (Schulman, 2023; Zhang et al., 2023b; Kadavath et al., 2022).
Regular RL tasks generally require two types of learning: Learning which states rewarding and learning how to reach these states. This is in many ways a simplified setting compared to traditional RL problems, as state transitions are entirely deterministic, effectively requiring no exploration of environment dynamics. An equivalent formulation, possible due to the deterministic nature of the environment, is to consider the model's entire response as a single action, effectively treating the problem as a bandit problem (further discussed in Scction 6.2). Token-level and responso-level interpretations can also be mixed (Nakano et al., 2022), as they lend themselves to different parts of the training process. Ramamurthy et al. (2023) find for example, that token-level discounting can be beneficial.
86

---

Published in Transactions on Machine Learning Research (06/2025)
Practical challenges Note that in practice this assumption of single-turn interactions is frequently vio- lated, with the LLM interacting with the human in a dialogue setting, occasionally calling out to external systems, or interacting with the environment in other ways. This is not in conflict with single-turn training, as the dialogue setting can be seen as a sequence of single-turn interactions. As a consequence, however, the LLM has limited planning capabilities and must rely on the human to provide leedback on each generated output. It generally cannot ask follow-up questions or engage in multi-turn interactions to clarify the hu- man's feedback, except if explicitly trained or prompted to do so (Zhou et al., 2024). In practice, there are many annuage-model specific implementation details (Ramamurthy et al., 2023; Huang et al., 2024; Dong et al., 2024a; Xu et al., 2024) (see, e.g., Appendix E. in the paper by Nakano et al. (2022) for a concrete example) that are not all covered in this work.
Exploration While we previously argued that exploration of environment dynamics is not necessary, exploration of the reward landscape is crucial. To reinforce behavior, it is necessary to explore the space of possible responses. This is particularly challenging due to the large action space and long time horizon, prohibiting random exploration. Exploration is commonly encouraged by a KL-divergence regularization term in the policy optimization objective, encouraging the agent to stay close to the pretrained modcl's behavior, which is generally stochastic due to the pretraining objective (Stienmon et al., 2020; Nakano el al., 2022). This avoids a phenomenon commonly referred to entropy collapse, where the LLM overfits to the reward model and starts to give low-entropy next-token predictions (Nakano et al., 2022). Nonethcless, the pretrained model may not explore the space of possible responses sufficiently if the reward model encourages responses that are rarely or never generated by the pretrained model, such as abstaining from answering.
Prompt distribution RLHF for LLMs can be seen as a form of langunge-condditonee RL (Luketina et al., 2019), where the user's initial prompt specifies the task. Training a general-purpose model therefore requires a diverse set of prompts for training. These prompts are sometimes the same as used for preference learning (Rafailov et al., 2023), sometimes come from the same pool and may overlap partially (Ziegler et al., 2020; Stiennon et al., 2020) and sometimes are chosen to be distinct from that dataset (Nakano et al., 2022; Ouyang et al., 2022).
Off-policy training When studied in small-scale rescarch contexts, RLHF is generally applied in an inter- leaved fashion, iterating or even parallelizing reward model training and reinforcement learning (Christiano et al., 2017). This results in a setting where the trajectories used to train the reward model are generated by a recent policy (i.e., semi-on-policy) and updated during the training process, although often not con- timuously (i.c., semi-online). For control applications, changes to the policy can quickly accumulate over longer Lime-horizons, leading to rapid shifts of the distribution of the observed trajectories and requiring on-policy updates to retain a useful reward model. In contrast, LLMs pose a slightly different setting, as the distribution shift is generally less pronouneed due to the single-turn interactions and the diversity of the initial training data. Additionally, the size of the required dataset and the cost of human feedback often prohibit fully online training, necessitating at least some degree of offline or off-policy training (e.g., by using scraped preferences or prior data) (Stiemnon et al., 2020; Askell et al., 2021). In practice, RLHF for LLMs is often implemented in a semi-online manner as well, introducing both on-policy and off-policy preferences infrequently (Stiennon et al., 2020). While most approaches in the literature include some online or on-policy training (Ziegler et al., 2020; Wu et al., 2021; Ouyang et al., 2022), the complexity of the task often leads to ad-hoc data collection methods (Stiennon et al., 2020; Nakano et al., 2022) compared to the more principled approach possible for small-scale research efforts (Christiano et al., 2017). Although it may be of less importance than for control applications, many works emphasize that at least some degree of online or on-policy data is important for final performance of LLM fine-tuning as well (Ziegler et al., 2020; Dong et al., 2024a; Xu et al., 2024). Note that direct methods such as DPO often lack online data, causing issues due to distributional shift (Xu et al., 2024).
Extensions Note, however, that RLHF for LLMs does not always operate in this simplified setting. Zhou et al. (2024) extend the RLHF setting to multi-turn interactions with LLMs, where the agent can ask follow-up questions to clarify the human's feedback. The LLM may additionally be given access to tools such web-browser (Nakano al., 2022), leading to further sources of non-determinism and complexity.
87

---

Published in Transactions on Machine Learning Research (06/2025)
Studied under the names of multi-turn RLHF agentic LLMs fine-tuning (with the latter setting generally incorporating actions beyond text output), this setting re-introduces many of the complexities of traditional RL settings, such as nondeterministic state transitions, partial observability and delayed rewards (Ma et al., 2024a). Another extension is to introduce dense reward signals (Chan et al., 2024), which can help to guide the agent more effectively.
Pretraining The language model, i.e., the policy, is generally pretrained with a self-super objective on a large corpus of text. This is followed by a step of supervised fine-tuning on a small dataset labeled with the desired behavior. This can be seen as a form of behavioral cloning, and a similar approach is sometimes used in RLHF for control. Similarly, since judging the quality of a response requires a certain level of nalural language understanding that is difficult to acquire from the limited training data available for reward model training, the reward model is generally initialized from language model pretrained on large corpus of text. As a side-effect of this initialization, as the pretraining data likely contains examples of the desired behavior as well as natural language descriptions of human values, the reward model likely already has some knowledge of the desired behavior which then needs to be elicited by fine-tuning with the new objective (Yang, 2024). It is then additionally often pre-trained from offline preferences, e.g., scraped from the web or mined from implicit expressions of preferences ( (Askell et al., 2021). Remaining preferences are then often gathered in a somewhat ad-hoe manner, with the reward model trained on a combination of preferences on responses various intermediate models (Nakano et al., 2022).
Inference-time selection The single-turn setting allows for a simple inference-time selection strategy, where the model generates several completions and the best one is selected based on the reward model (best-of-n-sammplnng) (Nakano et al., 2022; Menick et al., 2022). This is beneficial in some settings, as it reduces the risk of forgetting during fine-tuning and avoids the cost and time of policy optimization. It can be viewed as single-step lookahead search, where the reward model acts as the value function.
Direct methods While the separation into reward learning and policy training phases is common and useful in the context of RLHF for control, since it enables the RL agent to collect more samples for policy training than for reward model training, effectively learning about the environment dynamics without exces- sive human supervision, this separation is less useful in the context of LLM fine-tuning, Section 6.3 discusses direct methods for RLHF in more detail. Xu et al. (2024) finds that PPO, when tuned for the purpose of LLMs can often outperform direct methods, a phenomenon that can be partially explained by the distributional shift incurred when optimizing for human preferences, which reward-model based methods account for by using online data and reward model that may partially generalize the human preferences (Lambert et al., 2024).
Capabilities vs. alignment Parts of the vast training data will generally capture this type of interaction, so the model likely already has the capability to perform the task (Zhou et al., 2023; Gudibande ct al., 2024), but it needs to be elicited by fine-tuning with the new objective (Yang, 2024). Note that this even extends to capabilities that could be considered part of the aligning process, such as helpfulness (Wang et al., 2023a), harmlessness (Ganguli et al., 2023), and honesty (Kadavath et al., 2022).
Regularization In addition to the reward from the reward model, the policy is often regularized with a KL-divergence term (Jaques et al., 2017; Stiennon et al., 2020; Nakano et al., 2022) to the pretrained policy (Ramamurthy et al., 2023) or early-stopping (Nakano et al., 2022) to encourage it to stay close to the behavior of the pretrained model. This has been found to be crucial for preventing the policy from diverging from the behavior of the pretrained model, which can lead to catastrophic forgetting of the pretrained knowledge (Nakano et al., 2022; Ramamurthy et al., 2023).
B Prior Surveys
Based on the criteria discussed in Section 1.3, here we will first differentiate our survey from other surveys in marginally related subject areas sharing the common theme of human-in-the-loop RT. Then, we will describe
88

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: An overview of prior surveys human-in-the-loop RL. / indicates that the criterion a main focus of the survey, (V) indicates that the criterion partially addressed, while X indicates that the criterion is not covered.
Scalable and Async.
Reward Modelling
Human Defined
Interactive and Online
Reference
Тopic
Wu et al. (2022)
Human-in-the-loop ML
Retzlaff et al. (2024)
Human-in-the-loop RL RL with human advice Social feedback
SS. CAXA
S
Najar & Chetouani (2021)
Lin et al. (2020a)
*
Poole & Lee (2024)
RL from brain signals Interactive RL for HCI Imitation learning Inverse RL
Cruz & Igarashi (2020)
Osa al. (2018)
X
X X
Arora & Doshi (2021)
X
Bignold et al. (2021)
Assisted RL
X
Luketina et al. (2019)
Language-informed RL
X
Zhang et al. (2021)
Human guidance
X
Ji et al. (2023b)
AI Aligument
Liu et al. (2023c)
LLM applications
X
Ours
RLHF
V
the differences between our survey and previous surveys survey-like articles that exist within the RLHF field.
B.1 Human-in-the-Loop RL
Human participation ML, particularly in guiding machine learners, a well-stuuied scenario. This field, commonly referred as human-in-the-loop MT., can be further divided into subfields based on various criteria, e.g., the ones detailed in Section 1.3. Prior surveys of these subfields are compiled in Table 6 and briefly summarized in the following.
Human-in-the-Loow Learning from human feedback falls into the domain of ct al. (2022) survey human-in-the-loon ML in general. They also cover some applications of RLHF (for LLMs in particular) but do not provide a detailed overview. Retzlaff et al. (2024) provide a similar overview over samaa--n--te-loog RL particular, focusing on human involvement in RL on a more abstract level than our work and not covering RLHF in detail. Similarly broad in scope, Najar Chetouani (2021) study the setting of RL with human advice, which they define as 'teaching signals that can be communicated by the teacher to the learning system without executing the task'. While this setting subsumes RLHE, the broad generality limits the depth to which their survey cover RLHF approaches.
ML. Wu
Interactive RL RLHF can be considered a sub-field of interactive RL, which studies RL algorithms that learn in interaction with humans. This interaction can take the form of feedback defining an objec- tive, resulting in the RLHF setting, but can also, e.g., be used to drive exploration or speed up the agent's lcarning process.
Cruz & Igarashi (2020) survey interactive RL from an HCI viewpoint, exploring various ways humans can influence RL agents, with a particular focus on reward definition based on human feedback, without predefined environmental reward function. Due to the breadth of their survey, they do not cover many works in the area. The survey by Lin al. (2020a) centers on interactive RL using human social cues, like gestures and spoken language, but does not cover the reward modeling
89

---

Published in Transactions on Machine Learning Research (06/2025)
aspect. Similarly, the study by Poole & Lee (2024) examines RL with direct feedback from human brain signals, such as through brain-computer interfaces, also not focusing on reward modeling.
Demonstrations
Learning from demonstrations, in the form of behavior cloning (Osa et al., 2018) and inverse RL (Arora & Doshi, 2021), shares the goal of RLHF to learn behavior from human input, In contrast to RLHF, however, it requires demonstrations of the dcsired behavior instead of feed- back, and these demonstrations are usually not provided interactively and online. This limits their applications and final performance due to the need for near-optimal demonstrations. Nonetheless, imitation and demonstration can be a useful component of an RLHF system but are not the main focus of this survey. However, we will discuss the intersection between these fields in some parts whenever necessary.
Assisted RL Bignold et al. (2021) review the field of assisted RL, where agent may receive external information (for example, from a human) that aids it in action selection. While updates to the reward function are one of the possible effects of advice in this setting (in addition to action selection or modifications of the agent's internal state), it is usually assumed that an initial reward function is given and the extent of the updates limited to reward shaping or supplementary reward signals. In contrast to RLHF, the external information does not define the task but only helps the agent achieve it. Closely related to this, Luketina et al. (2019) survey RL assisted by natural language. In addition to this assistance setting, they also discuss approaches that infer language-conditioned reward function. However, they discuss this setting rather briefly and use techniques from inverse RL and not RLHF
Guidance In their survey human guidance, Zhang et al. (2021) explore various aspects related to RLHF. Although they touch on aspects such as reward learning, il is not the primary emphasis of their work. Instead, their main focus lies on exploring more immediate approaches that do not involve the learning of a reward model.
AI A Alignment Ji et al. (2023b) provide a general overview of Al alignment, i.e., the challenge of aligning the objectives of an intelligent system with those of its human operators. This survey covers RLHF in some detail. As AT aligument is a very broad field, however, the article nevertheless does not go into as much depth on the topic of RLHF as we do here.
Applications Liu et al. (2023c) give an overview of current applications of RLHF methods for LLMs such as ChatGPT and GPT-4. Even though it currently enjoys lot of attention, it is only one specific application area for RLHF. Our survey adopts broader perspective, examining the diverse applications and impact of RLHF encompassing application areas beyond LLMs.
B.2 PbRL and RLHF
Several previous surveys or survey-like articles are closely related to RLHF, Table gives a brief overview of how these articles differ from ours, which we will explain in more detail below.
Preference-Basede RL Previous surveys in the domain of RLHF often focus PbRL, where feedback is limited to binary preferences (sce Section 1.2). An illustrative example of this is the survey by Wirth et al. (2017), which is a direct precursor to our work. In contrast to our work, they concentrate on binary preferences for trajectories and primarily survey methods that learn policies without deriving a reward model. Since then, the neward-modelle and other approaches have extended RLHF to new feedback types. Abdelkarcem al. (2022) give another more recent literature review of PbRL While this review focuses on reward modeling and includes some recent work, it is far less comprehensive than our review, many aspects are only touched upon and partly overlap with those of Wirth et al. (2017).
approach has become dominant,
Feedback Types Although not a survey per so, Jeon et al. (2020) propose reward-rational implicit choice as unifying framework to comprehend many previous studies in PbRL and RLHF, To illustrate its generality, they overview different. feedback types used in previous work and explain how they fit.
90

---

Published in Transactions on Machine Learning Research (06/2025)
Table 7: An overview prior RLHHFspeccfic surveys and articles with substantial review components. The symbols indicate coverage: / indicates that the aspect addressed, (V) indicates that the aspect is partially
addressed, while X indicates that the aspect is not covered.
Label Comparisons Collection
Beyond
RM Training
App. and Benchmarks
Reference (Focus)
Theory
Wirth al. (2017)
S CASSA
x
(preference-based RL)
Abdelkareem et al. (2022)
X
X
(recent advances PbRL)
Jeon al. (2020)
/
X
(feedback modelling)
Casper et al. (2023)
X
7
(open issues in RLHF)
Fernandes et al. (2023)
X
(language generation)
Metz et al. (2023)
X
(feedback types)
Yuan et al. (2024)
X
(feedback types)
Ours (fundamentals, recent advances, and trends)
into their framework. The concurrent works by Metz et al. (2023) and Yuan et al. (2024), which are also not strictly surveys, propose frameworks for studying user interaction and interface design for multiple feedback types. As part of their work, they provide classification of feedback types and brief overview of RLHF approaches. Metz et al. (2023) have stronger focus the feedback interface and on learning from multiple feedback types simultaneously, discussing properties of feedback types and proposing a slandard encoding for them. On the other hand, Yuan et al. (2024) also include an offline RLHF benchmark and have a stronger focus on the reward learning aspect, focusing on the entire learning pipeline. Nevertheless, many facets RLHF are not addressed in those studies, as they are not primarily survey articles. Our survey has broader scope and therefore provides more extensive coverage, going beyond the study of feedback types and discussing more recent work.
Domain-Specific Fernandes et al. (2023) focuses on human feedback for language generation. As a result of their focus, their survey is less comprchensive than this work but discusses some language-speoffic aspects that do not fall into our scope, such using feedback models at generation time.
Open Problems Casper et al. (2023) provide a detailed overview of the open questions and limitations of RLHF with a particular focus on aspects of security, governance, and transparency. In their article, reward modeling is also covered, as human feedback, which goes beyond preference comparisons, but other aspects, such theoretical approaches or an overview of existing benchmarks, are not included. Thus, it can be seen supplementary article that is idcal for further reading once being familiarized with the topic through our survey.
91


Reviews:
[{"review_id": "pL3fgi5V8C", "reviewer": "Reviewer_9bJj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-structured and articulately written, providing clear explanations that make complex concepts accessible to both new researchers and seasoned practitioners in the field.\n- Given the increasing importance of human-in-the-loop systems in AI, the survey addresses a timely and significant topic that has implications for the development of ethical and effective AI systems.\n- The survey covers a broad spectrum of literature, offering a balanced view on both theoretical advancements and practical implementations of RLHF.\n\nWeaknesses:\n- While the paper discusses theoretical aspects and potential applications, it lacks detailed case studies that demonstrate real-world implementations and outcomes of RLHF systems.\n- The paper touches on the alignment of machine objectives with human values but does not delve deeply into the ethical challenges and societal impacts that might arise from RLHF.\n- The survey could benefit from a more detailed comparative analysis between RLHF and other learning paradigms, highlighting specific scenarios where RLHF is particularly beneficial or detrimental.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript presents a comprehensive survey of the field of Reinforcement Learning from Human Feedback (RLHF), an emerging area that integrates human input directly into the reinforcement learning process rather than relying solely on predefined reward functions. The paper builds upon earlier work in preference-based reinforcement learning and explores the broader application of RLHF in training large language models (LLMs) and other intelligent systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include one or two detailed case studies showcasing the application of RLHF in real-world scenarios, which would help illustrate the practical benefits and challenges discussed.\n- Enhance the discussion on ethical considerations by including a section that explores potential risks, such as privacy concerns and biases in human feedback, and suggests mitigation strategies.\n- Develop a comparative framework or table that contrasts RLHF with conventional RL and other human-in-the-loop learning methods, providing clearer insights into the unique advantages and limitations of each approach.", "broader_impact_concerns": "~", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SOboqNd9ga", "reviewer": "Reviewer_HME9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This topic is clearly timely and reading the paper allowed to size how large the literature already was. The list of cited papers is impressive.\nThe division in sections makes sense. The introductory sections are fairly clear.\n\nHowever, at some point in the reading, maybe because the paper is too long, I found the paper's style becoming less engaging and too abstract.\nA good survey should also serve to explain the concepts that are covered, as well as provide examples to help the reader create mental pictures of the relevant ideas. A survey is not necessarily like a tutorial, but still, it should not require the reader to have read most of the cited papers already. The paper tries to explain, but it quickly gets into unexplained jargon and terms that a ML researcher who is not deep into RLHF would find very difficult to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a very long paper (56 pages without the referneces) covering the RLHF-related literature, from basic components and definitions to variants in the form of feedback and label collection, reward model training, policy learning, regret theory, applications and benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above weakness, please try to move towards a more tutorial-like presentation. Adding examples throughout would help (e.g., in sec. 3.1 for each of the attributes, in sec. 3.2.1, examples of actual instances and feedback).\n\np. 4: \"accurately specifying reward is notoriously difficult ... mitigated by utilizing human feedback... helps avoid safety issues arising from misaligned reward\"\nUnfortunately, the last part if a misleading statement. We are far (as a community) from having found how to avoid the safety issues due to misaligned reward. It is basically a giant and open problem in AI safety (see some review in the International Scientific Report on the Safety of Advanced AI). Please change the text appropriately. In fact, RLHF has been shown to INCREASE some of the safety issues, e.g., the LLM learns deception in order to please the human labelers (mentioned in the last paragraph of the section).\n\np. 26: I don't see how pure on-policy data get provide guarantees. In fact it is almost certaint to miss the non-visited high-reward regions and correct OOD behavior so the sentence that ends with \"to guarantee the relevance of the generated queries for the current policy\" seems wrong.\n\np. 26: about query diversity, there is work on entropy regularized RL and related approaches like generative flow networks (including for automated red-teaming LLMs) to sample from high reward trajectories rather than maximize (and end up with a single high-reward solution which may be brittle).\n\np. 44: I did not understand what \"intermediate\" reward models referred to (top).\n\np. 45: \"resulting in sparse feedback\": I don't understand what you call that sparse, since every trajectory is assigned a (generally non-zero) reward, and we are in the formulation, as written at the beginning of the paragraph, where we consider a whole trajectory (episode) as an action. What is sparse of course is the coverage of episodes (among all the possible ones). For this very reason, I am not convinced that the statement \"does not require exploration\" in the last paragraph of that section is correct. There is going to be an exponentially large number of possible episodes, so exploration seems quite important to find more useful ones.\n\np. 45: Lots of DPO papers are cited but there is not one once of explanation about what it is. This is an example of poor writing for a survey.\n\np. 50: example of lack of tutorial-like writing: the term pessimistic MLE is not really explained. It is described in reference to pssimistic value function, which is also not explained or defined.\n\nTypos:\n\np. 20, form --> from\n\np. 21, leftover logistical comment, top of sec 3.2.2\n\np. 36, been presented --> presented\n\np. 41, pretaining\n\np. 44, degenerated\n\np. 45, with being --> being\n\np. 46, subsequently embraced by many subsequent -->  embraced by many subsequent", "broader_impact_concerns": "No discussion of broader impact in the paper. Yet, there are discussions of the impact of the RL component of RLHF creating an incentive for deception in the resulting agents, which clearly has ethical considerations in terms of safety.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "48PSVQAz2m", "reviewer": "Reviewer_VZY8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is clearly a useful document, especially once into the central portions of the paper (sections 3 to 5), but there is some thematic confusion and balance challenges that are to be expected as a subfield goes through such substantial shifts in interest. My primary concern with the paper is trying to reconcile how most of the readers in the paper will show up due to an interest in the application of RLHF to language models, but the inclusion of language-modeling related content seems largely an afterthought. Throughout, most of the figures and references either miss core recent works in that area of don’t indicate what is different about this blossoming area of research.\n\nTo be clear, as a reviewer I don’t think this is my decision to make, but given I’ll be giving a positive score either way, I think some discourse on the matter is important. There are plenty of sentences that aren’t really true for RLHF on LMs, but it’s not easy to fix:\n\n- “Learning the agent’s objective from human feedback circumvents reward engineering challenges and fosters robust training” —> the only way to get a reasonable reward\n- Anything “human-in-the-loop” doesn’t apply because it’s a contrived offline bandits approach\n- reward model “should be learning in an interactive, online, …”\n\nThere are also other sources of confusion similar to this:\n\n- Limiting the scope to papers within 5 years, which leads to selective inclusion or not of core papers outside of that window. The seminal papers of the field, which is pretty recent, should really be included.\n- There could potentially be more forward pointers in the text. Like my language model comment, in section 5.2.7 I was really wondering when DPO would be mentioned (and if), which I also wondered during the intro. There were other examples where I was confused reading it, only to find it later, which forward links could avoid.\n\nI focus on this because a lot of the content in the paper is great. There are plenty of sections that I starred as well done, such as feedback types and the origins of the Bradley Terry model. So, I think this is a good paper, but should revisit some of the framing and organization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a new survey on reinforcement learning from human feedback, focusing on a general frame of view across each of the steps in the RL process. The “novelty” in this area is to give the newest possible map of papers for new entrants in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I mentioned incomplete coverage of core papers in the flourishing of RLHF, here’s a list of papers I grabbed from a blog post from early 2023 that should be considered (coverage was pretty good in checking this):\n    - GopherCite: [**Teaching language models to support answers with verified quotes**](https://www.deepmind.com/publications/gophercite-teaching-language-models-to-support-answers-with-verified-quotes) (Menick et al. 2022): Train a LM with RLHF to return answers with specific citations.\n    - [**Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning**](https://arxiv.org/abs/2208.02294) (Cohen at al. 2022): Using RL to enhance the conversational skill of an open-ended dialogue agent.\n    - [**Llama 2**](https://arxiv.org/abs/2307.09288) (Touvron et al. 2023): Impactful open-access model with substantial RLHF details.\n    - The first paper to do KL penalties for sequences https://arxiv.org/abs/1611.02796 (section 5.2.1 addition)\n2. All of the content on previous surveys should probably be moved to the end or the appendix\n3. It is probably worth adding a diagram that works for how the RLHF used for language models can work (offline labels, no state feedback)\n4. There can be citations added to the paper that support arguments, but they probably are more niche. A section that I felt like could use populating was 5.1.6 (and nearby).\n    1. Toward pluralistic alignment https://arxiv.org/abs/2402.05070\n    2. Social choice for AI alignment https://arxiv.org/abs/2404.10271v1 \n    3. The History and Risks of Reinforcement Learning and Human Feedback, https://arxiv.org/abs/2310.13595\n5. I found the active learning and query-learning framing to be a bit forced, as I have never seen it before in the literature, which makes the entry costs to the paper a bit higher. I don’t really know how to fix this.\n6. In section 3.2.3 in discussing improvements, the concept of revisions and critiques should be included (e.g. constitutional AI and https://arxiv.org/abs/2402.07896)\n7. I thought it was weird that section 4.2.2 had no mention of dataset size per domain, which is likely a crucial detail.\n8. Section 5 seemed to have a bit higher variance than the core sections preceding it (e.g. 3 and 4). Some feedback:\n    1. 5.1.X is likely to vary a lot by domain. This links with my language model comment, but it just didn’t seem as clear how to use it as some of the other sections.\n    2. 5.2 seemed a little tangential, not sure how necessary it is in fullness\n    3. 5.2 is missing references to a bunch of recent reward model ensembling and averaging papers: https://arxiv.org/abs/2312.09244 https://arxiv.org/abs/2401.12187\n    4. 5.3 and the later evaluation section can include recent work on evaluating reward models (and generative reward models): https://arxiv.org/abs/2403.13787 https://arxiv.org/abs/2310.05470  https://arxiv.org/abs/2309.16155 (consistency eval is small)\n    5. 5.5.1 the analysis of reward model pretraining is not accurate with the method described in the paper. It was additional pretraining, which is really more of fine-tuning of a language model.\n    6. 5.5.2 should probably mention RLAIF and recent distillation of preferences work. And Meta Llama 2’s margin RM loss\n9. Section 6 should mention best of n sampling and rejection sampling\n10. Nits:\n    1. 5.1.6 has a broken bibtex ref\n    2. 4.2.1 particularly was a bit long and could use some of the paragraph breakdowns you did\n    3. Section 6 has a clear typo in the intro to the section\n    4. 3.2.2 has a author note that was supposed to be deleted", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a comprehensive overview of the field of reinforcement learning from human feedback (RLHF), that is valuable to the machine learning community at large.

It provides an extensive survey, touching on the historical origins and fundamental principles of RLHF, applications spanning several domains, and broader research trends. It successfully synthesizes a wide range of literature, making it a valuable resource for both new researchers and experienced practitioners for this timely topic.

The survey is well-organized with a nice and gentle introduction that lays a strong foundation for the rest of the material, like a mini-textbook.

The authors have made several revisions based on reviewer comments, improving tutorial-like explanations, adding examples, and enhancing discussions on ethical considerations.

One minor deficiency relates to the coverage of LLM-Specific RLHF: as a reviewer has pointed out, the paper does not always draw the connections to what is overwhelmingly the most popular application domain for RLHFs today, namely, for training LLMs.

However, overall, this is still easily a worthy contribution to TMLR.
Decision: Accept (poster)