Abstract: The generality of decision and game theory has enabled domain-independent progress in AI research. For example, a better algorithm for finding good policies in (PO)MDPs can be instantly used in a variety of applications. But such a general theory is lacking when it comes to moral decision making. For AI applications with a moral component, are we then forced to build systems based on many ad-hoc rules? In this paper we discuss possible ways to avoid this conclusion.

Vincent Conitzer
Walter Sinnott-Armstrong
Jana Schaich Borg
Yuan Deng
Max Kramer

Moral Decision Making Frameworks for Artificial Intelligence

Introduction As deployed AI systems become more autonomous, they increasingly face moral dilemmas. An often-used example is that of a self-driving car that faces an unavoidable accident, but has several options how to act, with different effects on its passengers and others in the scenario. (See, for example,  Bonnefon et al. (2016) .) But there are other examples where AI is already used to make decisions with lifeor-death consequences. Consider, for example, kidney exchanges. These cater to patients in need of a kidney that have a willing live donor whose kidney the patient's body would reject. In this situation, the patient may be able to swap donors with another patient in the same situation. (More complex arrangements are possible as well.) For these exchanges, algorithms developed in the AI community are already used to determine which patients receive which kidneys (see, e.g.,  Dickerson and Sandholm (2015) ). While it may be possible to find special-purpose solutions for moral decision making in these domains, in the long run there is a need for a general framework that an AI agent can use to make moral decisions in a wider variety of contexts. In this paper, we lay out some possible roadmaps for arriving at such a framework.
 Motivation Most AI research is conducted within straightforward utilitarian or consequentialist frameworks, but these simple approaches can lead to counterintuitive judgments from an ethical perspective. For example, most people consider it immoral to harvest a healthy patient's organs to save the lives of Copyright c 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. two or even five other patients. Research in ethics and moral psychology elucidates our moral intuitions in such examples by distinguishing between doing and allowing, emphasizing the role of intent, applying general rules about kinds of actions (such as "Don't kill"), and referring to rights (such as the patient's) and roles (such as the doctor's). Incorporating these morally relevant factors among others could enable AI to make moral decisions that are safer, more robust, more beneficial, and acceptable to a wider range of people.  1  To be useful in the development of AI, our moral theories must provide more than vague, general criteria. They must also provide an operationalizable, and presumably quantitative, theory that specifies which particular actions are morally right or wrong in a wide range of situations. This, of course, also requires the agent to have a language in which to represent the structure of the actions being judged  (Mikhail, 2007)  and the morally relevant features of actions  (Gert, 2004)  along with rules about how these features interact and affect moral judgments. Moral theory and AI need to work together in this endeavor. Multiple approaches can be taken to arrive at generalpurpose procedures for automatically making moral decisions. One approach is to use game theory. Game-theoretic formalisms are widely used by artificial intelligence researchers to represent multiagent decision scenarios, but, as we will argue below, its solution concepts and possibly even its basic representation schemes need to be extended in order to provide guidance on moral behavior. Another approach is to use machine learning. We can use the moral philosophy and psychology literatures to identify features of moral dilemmas that are relevant to the moral status of possible actions described in the dilemmas. Human subjects can be asked to make moral judgments about a set of moral dilemmas in order to obtain a labeled data set. Then, we can train classifiers based on this data set and the identified features. (Compare also the top-down vs. bottom-up distinction in automated moral decision making, as described by  Wallach and Allen (2008) .) We will discuss these two approaches in turn. In this paper, we will take a very broad view of what constitutes a moral dilemma  (contrast Sinnott-Armstrong (1988) ). As a simple example, consider the trust game  (Berg et al., 1995) . In the trust game, player 1 is given some amount of money-say, $100. She 2 is then allowed to give any fraction of this money back to the experimenter, who will then triple this returned money and give it to player 2. Finally, player 2 may return any fraction of the money he has received to player 1. For example, player 1 might give $50 back, so that player 2 receives 3 • $50 = $150, who then might give $75 back, leaving player 1 with $50 + $75 = $125. The most straightforward game-theoretic analysis of this game assumes that each player, at any point in the game, is interested only in maximizing the amount of money she herself receives. Under this assumption, player 2 would never have any reason to return any money to player 1. Anticipating this, player 1 would not give any money, either. However, despite this analysis, human subjects playing the trust game generally do give money in both roles  (Berg et al., 1995) . One of the reasons why is likely that many people feel it is wrong for player 2 not to give any money back after player 1 has decided to give him some (and, when in the role of player 1, they expect player 2 not to take such a wrong action). This case study illustrates a general feature of moral reasoning. Most people consider not only the consequences of their actions but also the setting in which they perform their actions. They ask whether an act would be unfair or selfish (because they are not sharing a good with someone who is equally deserving), ungrateful (because it harms someone who benefited them in the past), disloyal (by betraying a friend who has been loyal), untrustworthy (because it breaks a promise), or deserved (because the person won a competition or committed a crime). In these ways, moral reasoners typically look not only to the future but also to the past. Of course, not everyone will agree about which factors are morally relevant, and even fewer people will agree about which factor is the most important in a given conflict. For example, some people will think that it is morally wrong to lie to protect a family member, whereas others will think that lying in such circumstances is not only permitted but required. Nonetheless, a successful moral AI system does not necessarily have to dictate one true answer in such cases. It may suffice to know how much various groups value different factors or value them differently. Then when we code moral values into AI, we would have the option of either using the moral values of a specific individual or group-a type of moral relativism-or giving the AI some type of socialchoice-theoretic aggregate of the moral values that we have inferred (for example, by letting our models of multiple people's moral values vote over the relevant alternatives, or using only the moral values that are common to all of them). This approach suggests new research problems in the field of computational social choice (see, e.g.,  Brandt et al. (2013 Brandt et al. ( , 2015 ).  Rossi (2016)  has described related, but distinct so-cial choice problems where (not necessarily moral) preferences are either aggregated together with a moral ranking of all the alternatives, or the preferences are themselves ranked according to a moral ordering (see also  Greene et al. (2016) ).
 Abstractly Representing Moral Dilemmas: A Game-Theoretic Approach For us humans, the most natural way to describe a moral dilemma is to use natural language. However, given the current state of AI in general and of natural language processing in particular, such verbal descriptions will not suffice for our purposes. Moral dilemmas will need to be more abstractly represented, and as is generally the case in AI research, the choice of representation scheme is extremely important. In this section, we consider an approach to this problem inspired by game theory.
 Game-Theoretic Representation Schemes Game theory (see, e.g.,  Fudenberg and Tirole (1991) ) concerns the modeling of scenarios where multiple parties (henceforth, agents) have different interests but interact in the same domain. It provides various natural representation schemes for such multiagent decision problems. Scenarios described in game theory involve sequences of actions that lead to different agents being better or worse off to different degrees. Since moral concepts-such as selfishness, loyalty, trustworthiness, and fairness-often influence which action people choose to take, or at least believe they should take, in such situations, game theory is potentially a good fit for abstractly representing moral dilemmas. One of the standard representation schemes in game theory is that of the extensive form, which is a generalization of the game trees studied in introductory AI courses. The extensive-form representation of the trust game (or rather, a version of it in which player 1 can only give multiples of $50 and player 2 only multiples of $100) is shown in Figure  1 . Each edge corresponds to an action in the game and is labeled with that action. Each bottom (leaf) node corresponds to an outcome of the game and is labeled with the corresponding payoffs for player 1 and player 2, respectively. We will turn to the question of whether such representation schemes suffice to model moral dilemmas more generally shortly. First, we discuss how to solve such games.
 Moral Solution Concepts The standard solution concepts in game theory assume that each agent pursues nothing but its own prespecified utility. If we suppose in the trust game that each player just seeks to maximize her own monetary payoff, then game theory would prescribe that the second player give nothing back regardless of how much he receives, and consequently that the first player give nothing. 3 However, this is not the behavior observed in experiments with human subjects. Games that elicit human behavior that does not match game-theoretic analyses, such as the trust game, are often used to criticize the game-theoretic model of behavior and have led to the field of behavioral game theory  (Camerer, 2003) . While in behavioral game theory, attention is often drawn to the fact that humans are not infinitely rational and cannot be expected to perform complete game-theoretic analyses in their heads, it seems that this is not the primary reason that agents behave differently in the trust game, which after all is quite simple. Rather, it seems that the simplistic game-theoretic solution fails to account for ethical considerations. In traditional game theory's defense, it should be noted that an agent's utility may take into account the welfare of others, so it is possible for altruism to be captured by a game-theoretic account. However, what is morally right or wrong also seems to depend on past actions by other players. Consider, for example, the notion of betrayal: if another agent knowingly enables me either to act to benefit us both, or to act to benefit myself even more while significantly hurting the other agent, doing the latter seems morally wrong. This, in our view, is one of the primary things going on in the trust game. The key insight is that to model this phenomenon, we cannot simply first assess the agents' otherregarding preferences, include these in their utilities at the leaves of the game, and solve the game (as in the case of pure altruism). Rather, the analysis of the game (solving it) must be intertwined with the assessment of whether an agent morally should pursue another agent's well-being. This calls for novel moral solution concepts in game theory. We have already done some conceptual and algorithmic work on a solution concept that takes such issues into account  (Letchford et al., 2008) . This solution concept involves repeatedly solving the game and then modifying the agents' preferences based on the solution. The modification makes it so that (for example) player 2 wants to ensure that player 1 receives at least what she could have received in the previous solution, unless this conflicts with player 2 receiving at least as much as he would have received in the previous solution. For example, in the trust game player 2's preferences are modified so that he values player 1 receiving back at least what she gave to player 2.
 What Is Left Out & Possible Extensions The solution concept from  Letchford et al. (2008)  is defined only in very restricted settings, namely 2-player perfect-information 4 games. One research direction is to generalize the concept to games with more players and/or imperfect information. Another is to define different solution concepts that capture other ethical concerns. Zooming out, this general approach is inherently limited by the aspects of moral dilemmas that can be captured in game-theoretic representations. While we believe that the standard representation schemes of game theory can capture much of what is relevant, they may not capture everything that is relevant. For example, in moral philosophy, a distinction is often made between doing harm and allowing harm. Consider a situation where a runaway train will surely hit and kill exactly one innocent person (player 2) standing on a track, unless player 1 intervenes and puts the train on another track instead, where it will surely hit and kill exactly one other innocent person (player 3). The natural extensive form of the game (Figure  2 ) is entirely symmetric and thereby cannot be used to distinguish between the two alternatives. (Note that the labels on the edges are formally not part of the game.) However, many philosophers (as well Player 1
 Do nothing Put train on other track g other track 0, -100, 0 0 , 0, -100 Figure  2 : "Runaway train." Player 1 must choose whether to allow player 2 to be hurt or to hurt player 3 instead. as non-philosophers) would argue that there is a significant distinction between the two alternatives, and that switching the train to the second track is morally wrong. We propose that the action-inaction distinction could be addressed by slightly extending the extensive-form representation so that at every information set (decision point), one action is labeled as the "passive" action (e.g., leaving the train alone). Other extensions may be needed as well. For example, we may take into account what each agent in the game deserves (according to some theory of desert), which may require us to further extend the representation scheme. 5 A broader issue is that in behavioral game and decision theory it is well understood that the way the problem is framed-i.e., the particular language in which the problem is described, or even the order in which dilemmas are presented-can significantly affect human subjects' decisions. That is, two ways of describing the same dilemma can produce consistently different responses from human subjects  (Kahneman and Tversky, 2000) . The same is surely the case for moral dilemmas  (Sinnott-Armstrong, 2008) . Moral AI would need to replicate this behavior if the goal is to mirror or predict human moral judgments. In contrast, if our goal is to make coherent moral judgments, then moral AI might instead need to avoid such framing effects.
 Setting up a Machine Learning Framework Another approach for developing procedures that automatically make moral decisions is based on machine learning (see, e.g.,  Mitchell (1997) ). We can assemble a training set of moral decision problem instances labeled with human judgments of the morally correct decision(s), and allow our AI system to generalize. (Other work has focused on obtaining human judgments not of the actions themselves, but of persuasion strategies in such scenarios  (Stock et al., 2016) .) To evaluate this approach with current technology, it is insufficient to represent the instances in natural language; instead, we must represent them more abstractly. What is the right representation scheme for this purpose, and what features are important? How do we construct and accurately label a good training set?
 Representing Dilemmas by Their Key Moral Features When we try to classify a given action in a given moral dilemma as morally right or wrong (as judged by a given human being), we can try to do so based on various features (or attributes) of the action. In a restricted domain, it may be relatively clear what the relevant features are. When a self-driving car must decide whether to take one action or another in an impending-crash scenario, natural features include the expected number of lives lost for each course of action, which of the people involved were at fault, etc. When allocating a kidney, natural features include the probability that the kidney is rejected by a particular patient, whether that patient needs the kidney urgently, etc. Even in these scenarios, identifying all the relevant features may not be easy. (E.g., is it relevant that one potential kidney recipient has made a large donation to medical research and the other has not?) However, the primary goal of a general framework for moral decision making is to identify abstract features that apply across domains, rather than to identify every nuanced feature that is potentially relevant to isolated scenarios. The literature in moral psychology and cognitive science may guide us in identifying these general concepts. For example,  Haidt and Joseph (2004)  have proposed five moral foundations-harm/care, fairness/reciprocity, loyalty, authority, and purity. Recent research has added new foundations and subdivided some of these foundations  (Clifford et al., 2015) . The philosophy literature can similarly be helpful; e.g.,  Gert (2004)  provides a very inclusive list of morally relevant features.
 Classifying Actions as Morally Right or Wrong Given a labeled dataset of moral dilemmas represented as lists of feature values, we can apply standard machine learn-ing techniques to learn to classify actions as morally right or wrong. In ethics it is often seen as important not only to act in accordance with moral principles but also to be able to explain why one's actions are morally right  (Anderson and Anderson, 2007; Bostrom and Yudkowsky, 2014) ; hence, interpretability of the resulting classifier will be important. Of course, besides making a binary classification of an action as morally right or wrong, we may also make a quantitative assessment of how morally wrong the action is (for example using a regression), an assessment of how probable it is that the action is morally wrong (for example using a Bayesian framework), or some combination of the two. Many further complicating factors can be added to this simple initial framework.
 Discussion A machine learning approach to automating moral judgements is perhaps more flexible than a game-theoretic approach, but the two can complement each other. For example, we can apply moral game-theoretic concepts to moral dilemmas and use the output (say, "right" or "wrong" according to this concept) as one of the features in our machine learning approach. On the other hand, the outcomes of the machine learning approach can help us see which key moral aspects are missing from our moral game-theoretic concepts, which will in turn allow us to refine them. It has been suggested that machine learning approaches to moral decisions will be limited because they will at best result in human-level moral decision making; they will never exceed the morality of humans. (Such a worry is raised, for example, by  Chaudhuri and Vardi (2014) .) But this is not necessarily so. First, aggregating the moral views of multiple humans (through a combination of machine learning and social-choice theoretic techniques) may result in a morally better system than that of any individual human, for example because idiosyncratic moral mistakes made by individual humans are washed out in the aggregate. Indeed, the learning algorithm may well decide to output a classifier that disagrees with the labels of some of the instances in the training set (see  Guarini (2006)  for a discussion of the importance of being able to revise initial classifications). Second, machine learning approaches may identify general principles of moral decision making that humans were not aware of before. These principles can then be used to improve our moral intuitions in general. For now, moral AI systems are in their infancy, so creating even human-level automated moral decision making would be a great accomplishment.
 Conclusion In some applications, AI systems will need to be equipped with moral reasoning capability before we can grant them autonomy in the world. One approach to doing so is to find ad-hoc rules for the setting at hand. However, historically, the AI community has significantly benefited from adopting methodologies that generalize across applications. The concept of expected utility maximization has played a key part in this. By itself, this concept falls short for the purpose of moral decision making. In this paper, we have consid-ered two (potentially complementary) paradigms for designing general moral decision making methodologies: extending game-theoretic solution concepts to incorporate ethical aspects, and using machine learning on human-labeled instances. Much work remains to be done on both of these, and still other paradigms may exist. All the same, these two paradigms show promise for designing moral AI. Figure 1 : 1 Figure1: The trust game. Each edge corresponds to an action in the game and is labeled with that action. Each bottom (leaf) node corresponds to an outcome of the game and is labeled with the corresponding payoffs for player 1 and player 2, respectively.
			 The point that, as advanced AI acquires more autonomy, it is essential to bring moral reasoning into it has been made previously by others-e.g., Moor (2006) .
			 We use "she" for player 1 or a generic player, and "he" for player 2.
			 The technical name for this type of analysis is backward induction, resulting in behavior that constitutes a subgame perfect Nash equilibrium of the game.
			 In a perfect-information game, the current state is fully observable to each player (e.g., chess), in contrast to imperfectinformation games (e.g., poker).5 Note that, to the extent the reasons for what an agent deserves are based solely on the agent's earlier actions in the game under consideration, solution concepts such as those described above might in fact capture this. If so, then the only cases in which we need to extend the representation scheme are those where what an agent deserves is external to the game under study (e.g., the agent is a previously convicted criminal).


Abstract: For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any which have been previously learned from human feedback. ⇤ Work done while at OpenAI.

Paul F Christiano
Jan Leike
Tom B Brown
Google Brain
Miljan Martic
Shane Legg
Dario Amodei

Deep Reinforcement Learning from Human Preferences

Introduction Recent success in scaling reinforcement learning (RL) to large problems has been driven in domains that have a well-specified reward function  (Mnih et al., 2015 (Mnih et al., , 2016 Silver et al., 2016) . Unfortunately, many tasks involve goals that are complex, poorly-defined, or hard to specify. Overcoming this limitation would greatly expand the possible impact of deep RL and could increase the reach of machine learning more broadly. For example, suppose that we wanted to use reinforcement learning to train a robot to clean a table or scramble an egg. It's not clear how to construct a suitable reward function, which will need to be a function of the robot's sensors. We could try to design a simple reward function that approximately captures the intended behavior, but this will often result in behavior that optimizes our reward function without actually satisfying our preferences. This difficulty underlies recent concerns about misalignment between our values and the objectives of our RL systems  (Bostrom, 2014; Russell, 2016; Amodei et al., 2016) . If we could successfully communicate our actual objectives to our agents, it would be a significant step towards addressing these concerns. If we have demonstrations of the desired task, we can use inverse reinforcement learning  (Ng and Russell, 2000)  or imitation learning to copy the demonstrated behavior. But these approaches are not directly applicable to behaviors that are difficult for humans to demonstrate (such as controlling a robot with many degrees of freedom but non-human morphology). An alternative approach is to allow a human to provide feedback on our system's current behavior and to use this feedback to define the task. In principle this fits within the paradigm of reinforcement learning, but using human feedback directly as a reward function is prohibitively expensive for RL systems that require hundreds or thousands of hours of experience. In order to practically train deep RL systems with human feedback, we need to decrease the amount of feedback required by several orders of magnitude. We overcome this difficulty by asking humans to compare possible trajectories of the agent, using that data to learn a reward function, and optimizing the learned reward function with RL. This basic approach has been explored in the past, but we confront the challenges involved in scaling it up to modern deep RL and demonstrate by far the most complex behaviors yet learned from human feedback. Our experiments take place in two domains: Atari games in the Arcade Learning Environment  (Bellemare et al., 2013) , and robotics tasks in the physics simulator  MuJoCo (Todorov et al., 2012) . We show that a small amount of feedback from a non-expert human, ranging from fifteen minutes to five hours, suffice to learn both standard RL tasks and novel hard-to-specify behaviors such as performing a backflip or driving with the flow of traffic.
 Related Work A long line of work studies reinforcement learning from human ratings or rankings, including  Akrour et al. (2011 ), Pilarski et al. (2011 ), Akrour et al. (2012 ), Wilson et al. (2012 ), Sugiyama et al. (2012 ), Wirth and Fürnkranz (2013 ), Daniel et al. (2015 ), El Asri et al. (2016 ), Wang et al. (2016 ), and Wirth et al. (2016 . Other lines of research consider the general problem of reinforcement learning from preferences rather than absolute reward values  (Fürnkranz et al., 2012; Akrour et al., 2014; Wirth et al., 2016) , and optimizing using human preferences in settings other than reinforcement learning  (Machwe and Parmee, 2006; Secretan et al., 2008; Brochu et al., 2010; Sørensen et al., 2016) . Our algorithm follows the same basic approach as Akrour et al. (  2012 ) and Akrour et al. (  2014 ), but considers much more complex domains and behaviors. The complexity of our environments force us to use different RL algorithms, reward models, and training strategies. One notable difference is that  Akrour et al. (2012)  and  Akrour et al. (2014)  elicit preferences over whole trajectories rather than short clips, and so would require about an order of magnitude more human time per data point. Our approach to feedback elicitation closely follows  Wilson et al. (2012 ). However, Wilson et al. (2012  assumes that the reward function is the distance to some unknown (linear) "target" policy, and is never tested with real human feedback. TAMER  (Knox, 2012; Knox and Stone, 2013 ) also learns a reward function from human feedback, but learns from ratings rather than comparisons, has the human observe the agent as it behaves, and has been applied to settings where the desired policy can be learned orders of magnitude more quickly. Compared to all prior work, our key contribution is to scale human feedback up to deep reinforcement learning and to learn much more complex behaviors. This fits into a recent trend of scaling reward learning methods to large deep learning systems, for example inverse RL  (Finn et al., 2016) , imitation learning  (Ho and Ermon, 2016; Stadie et al., 2017) , semi-supervised skill generalization  (Finn et al., 2017) , and bootstrapping RL from demonstrations  (Silver et al., 2016; Hester et al., 2017) .
 Preliminaries and Method
 Setting and Goal We consider an agent interacting with an environment over a sequence of steps; at each time t the agent receives an observation o t 2 O from the environment and then sends an action a t 2 A to the environment. In traditional reinforcement learning, the environment would also supply a reward r t 2 R and the agent's goal would be to maximize the discounted sum of rewards. Instead of assuming that the environment produces a reward signal, we assume that there is a human overseer who can express preferences between trajectory segments. A trajectory segment is a sequence of observations and actions, = ((o 0 , a 0 ), (o 1 , a 1 ), . . . , (o k 1 , a k 1 )) 2 (O ⇥ A) k . Write 1 2 to indicate that the human preferred trajectory segment 1 to trajectory segment 2 . Informally, the goal of the agent is to produce trajectories which are preferred by the human, while making as few queries as possible to the human. More precisely, we will evaluate our algorithms' behavior in two ways: Quantitative: We say that preferences are generated by a reward function 2 r : O ⇥ A ! R if o 1 0 , a 1 0 , . . . , o 1 k 1 , a 1 k 1 o 2 0 , a 2 0 , . . . , o 2 k 1 , a 2 k 1 whenever r o 1 0 , a 1 0 + • • • + r o 1 k 1 , a 1 k 1 > r o 2 0 , a 2 0 + • • • + r o 2 k 1 , a 2 k 1 . If the human's preferences are generated by a reward function r, then our agent ought to receive a high total reward according to r. So if we know the reward function r, we can evaluate the agent quantitatively. Ideally the agent will achieve reward nearly as high as if it had been using RL to optimize r. Qualitative: Sometimes we have no reward function by which we can quantitatively evaluate behavior (this is the situation where our approach would be practically useful). In these cases, all we can do is qualitatively evaluate how well the agent satisfies the human's preferences. In this paper, we will start from a goal expressed in natural language, ask a human to evaluate the agent's behavior based on how well it fulfills that goal, and then present videos of agents attempting to fulfill that goal. Our model based on trajectory segment comparisons is very similar to the trajectory preference queries used in  Wilson et al. (2012) , except that we don't assume that we can reset the system to an arbitrary state 3 and so our segments generally begin from different states. This complicates the interpretation of human comparisons, but we show that our algorithm overcomes this difficulty even when the human raters have no understanding of our algorithm.
 Our Method At each point in time our method maintains a policy ⇡ : O ! A and a reward function estimate r : O ⇥ A ! R, each parametrized by deep neural networks. These networks are updated by three processes: 1. The policy ⇡ interacts with the environment to produce a set of trajectories {⌧ 1 , . . . , ⌧ i }. The parameters of ⇡ are updated by a traditional reinforcement learning algorithm, in order to maximize the sum of the predicted rewards r t = r(o t , a t ).
 We select pairs of segments 1 , 2 from the trajectories {⌧ 1 , . . . , ⌧ i } produced in step 1, and send them to a human for comparison. 3. The parameters of the mapping r are optimized via supervised learning to fit the comparisons collected from the human so far. These processes run asynchronously, with trajectories flowing from process (1) to process (2), human comparisons flowing from process (2) to process (3), and parameters for r flowing from process (3) to process (1). The following subsections provide details on each of these processes.
 Optimizing the Policy After using r to compute rewards, we are left with a traditional reinforcement learning problem. We can solve this problem using any RL algorithm that is appropriate for the domain. One subtlety is that the reward function r may be non-stationary, which leads us to prefer methods which are robust to changes in the reward function. This led us to focus on policy gradient methods, which have been applied successfully for such problems  (Ho and Ermon, 2016) . In this paper, we use advantage actor-critic (A2C;  Mnih et al., 2016)  to play Atari games, and trust region policy optimization (TRPO;  Schulman et al., 2015)  to perform simulated robotics tasks. In each case, we used parameter settings which have been found to work well for traditional RL tasks. The only hyperparameter which we adjusted was the entropy bonus for TRPO. This is because TRPO relies on the trust region to ensure adequate exploration, which can lead to inadequate exploration if the reward function is changing. We normalized the rewards produced by r to have zero mean and constant standard deviation. This is a typical preprocessing step which is particularly appropriate here since the position of the rewards is underdetermined by our learning problem.
 Preference Elicitation The human overseer is given a visualization of two trajectory segments, in the form of short movie clips. In all of our experiments, these clips are between 1 and 2 seconds long. The human then indicates which segment they prefer, that the two segments are equally good, or that they are unable to compare the two segments. The human judgments are recorded in a database D of triples 1 , 2 , µ , where 1 and 2 are the two segments and µ is a distribution over {1, 2} indicating which segment the user preferred. If the human selects one segment as preferable, then µ puts all of its mass on that choice. If the human marks the segments as equally preferable, then µ is uniform. Finally, if the human marks the segments as incomparable, then the comparison is not included in the database.
 Fitting the Reward Function We can interpret a reward function estimate r as a preference-predictor if we view r as a latent factor explaining the human's judgments and assume that the human's probability of preferring a segment i depends exponentially on the value of the latent reward summed over the length of the clip: 4 P ⇥ 1 2 ⇤ = exp P r o 1 t , a 1 t exp P r(o 1 t , a 1 t ) + exp P r(o 2 t , a 2 t ) . (1) We choose r to minimize the cross-entropy loss between these predictions and the actual human labels: loss(r) = X ( 1 , 2 ,µ)2D µ(1) log P ⇥ 1 2 ⇤ + µ(2) log P ⇥ 2 1 ⇤ . This follows the Bradley-Terry model  (Bradley and Terry, 1952)  for estimating score functions from pairwise preferences, and is the specialization of the Luce-Shephard choice rule  (Luce, 2005; Shepard, 1957)  to preferences over trajectory segments. Our actual algorithm incorporates a number of modifications to this basic approach, which early experiments discovered to be helpful and which are analyzed in Section 3.3: • We fit an ensemble of predictors, each trained on |D| triples sampled from D with replacement. The estimate r is defined by independently normalizing each of these predictors and then averaging the results. • A fraction of 1/e of the data is held out to be used as a validation set for each predictor. We use `2 regularization and adjust the regularization coefficient to keep the validation loss between 1.1 and 1.5 times the training loss. In some domains we also apply dropout for regularization. • Rather than applying a softmax directly as described in Equation 1, we assume there is a 10% chance that the human responds uniformly at random. Conceptually this adjustment is needed because human raters have a constant probability of making an error, which doesn't decay to 0 as the difference in reward difference becomes extreme.
 Selecting Queries We decide how to query preferences based on an approximation to the uncertainty in the reward function estimator, similar to Daniel et al. (  2014 ): we sample a large number of pairs of trajectory segments of length k from the latest agent-environment interactions, use each reward predictor in our ensemble to predict which segment will be preferred from each pair, and then select those trajectories for which the predictions have the highest variance across ensemble members 5 This is a crude approximation and the ablation experiments in Section 3 show that in some tasks it actually impairs performance. Ideally, we would want to query based on the expected value of information of the query  (Akrour et al., 2012; Krueger et al., 2016) , but we leave it to future work to explore this direction further.
 Experimental Results We
 Reinforcement Learning Tasks with Unobserved Rewards In our first set of experiments, we attempt to solve a range of benchmark tasks for deep RL without observing the true reward. Instead, the agent learns about the goal of the task only by asking a human which of two trajectory segments is better. Our goal is to solve the task in a reasonable amount of time using as few queries as possible. In our experiments, feedback is provided by contractors who are given a 1-2 sentence description of each task before being asked to compare several hundred to several thousand pairs of trajectory segments for that task (see Appendix B for the exact instructions given to contractors). Each trajectory segment is between 1 and 2 seconds long. Contractors responded to the average query in 3-5 seconds, and so the experiments involving real human feedback required between 30 minutes and 5 hours of human time. For comparison, we also run experiments using a synthetic oracle whose preferences are generated (in the sense of Section 2.1) by the real reward 6 . We also compare to the baseline of RL training using the real reward. Our aim here is not to outperform but rather to do nearly as well as RL without access to reward information and instead relying on much scarcer feedback. Nevertheless, note that feedback from real humans does have the potential to outperform RL (and as shown below it actually does so on some tasks), because the human feedback might provide a better-shaped reward. We describe the details of our experiments in Appendix A, including model architectures, modifications to the environment, and the RL algorithms used to optimize the policy.
 Simulated Robotics The first tasks we consider are eight simulated robotics tasks, implemented in MuJoCo  (Todorov et al., 2012), and included in OpenAI Gym (Brockman et al., 2016) . We made small modifications to these tasks in order to avoid encoding information about the task in the environment itself (the modifications are described in detail in Appendix A). The reward functions in these tasks are quadratic functions of distances, positions and velocities, and most are linear. We included a simple cartpole Figure  1 : Results on MuJoCo simulated robotics as measured on the tasks' true reward. We compare our method using real human feedback (purple), our method using synthetic feedback provided by an oracle (shades of blue), and reinforcement learning using the true reward function (orange). All curves are the average of 5 runs, except for the real human feedback, which is a single run, and each point is the average reward over five consecutive batches. For Reacher and Cheetah feedback was provided by an author due to time constraints. For all other tasks, feedback was provided by contractors unfamiliar with the environments and with our algorithm. The irregular progress on Hopper is due to one contractor deviating from the typical labeling schedule. task ("pendulum") for comparison, since this is representative of the complexity of tasks studied in prior work. Figure  1  shows the results of training our agent with 700 queries to a human rater, compared to learning from 350, 700, or 1400 synthetic queries, as well as to RL learning from the real reward. With 700 labels we are able to nearly match reinforcement learning on all of these tasks. Training with learned reward functions tends to be less stable and higher variance, while having a comparable mean performance. Surprisingly, by 1400 labels our algorithm performs slightly better than if it had simply been given the true reward, perhaps because the learned reward function is slightly better shaped-the reward learning procedure assigns positive rewards to all behaviors that are typically followed by high reward. The difference may also be due to subtle changes in the relative scale of rewards or our use of entropy regularization. Real human feedback is typically only slightly less effective than the synthetic feedback; depending on the task human feedback ranged from being half as efficient as ground truth feedback to being equally efficient. On the Ant task the human feedback significantly outperformed the synthetic feedback, apparently because we asked humans to prefer trajectories where the robot was "standing upright," which proved to be useful reward shaping. (There was a similar bonus in the RL reward function to encourage the robot to remain upright, but the simple hand-crafted bonus was not as useful.)
 Atari The second set of tasks we consider is a set of seven Atari games in the Arcade Learning Environment  (Bellemare et al., 2013) , the same games presented in Mnih et al., 2013. Figure  2  shows the results of training our agent with 5,500 queries to a human rater, compared to learning from 350, 700, or 1400 synthetic queries, as well as to RL learning from the real reward. Our method has more difficulty matching RL in these challenging environments, but nevertheless it displays substantial learning on most of them and matches or even exceeds RL on some. Specifically, Figure  2 : Results on Atari games as measured on the tasks' true reward. We compare our method using real human feedback (purple), our method using synthetic feedback provided by an oracle (shades of blue), and reinforcement learning using the true reward function (orange). All curves are the average of 3 runs, except for the real human feedback which is a single run, and each point is the average reward over about 150,000 consecutive frames. on BeamRider and Pong, synthetic labels match or come close to RL even with only 3,300 such labels. On Seaquest and Qbert synthetic feedback eventually performs near the level of RL but learns more slowly. On SpaceInvaders and Breakout synthetic feedback never matches RL, but nevertheless the agent improves substantially, often passing the first level in SpaceInvaders and reaching a score of 20 on Breakout, or 50 with enough labels. On most of the games real human feedback performs similar to or slightly worse than synthetic feedback with the same number of labels, and often comparably to synthetic feedback that has 40% fewer labels. On Qbert, our method fails to learn to beat the first level with real human feedback; this may be because short clips in Qbert can be confusing and difficult to evaluate. Finally, Enduro is difficult for A3C to learn due to the difficulty of successfully passing other cars through random exploration, and is correspondingly difficult to learn with synthetic labels, but human labelers tend to reward any progress towards passing cars, essentially shaping the reward and thus outperforming A3C in this game (the results are comparable to those achieved with DQN).
 Novel behaviors Experiments with traditional RL tasks help us understand whether our method is effective, but the ultimate purpose of human interaction is to solve tasks for which no reward function is available. Using the same parameters as in the previous experiments, we show that our algorithm can learn novel complex behaviors. We demonstrate: 1. The Hopper robot performing a sequence of backflips (see Figure  4 ). This behavior was trained using 900 queries in less than an hour. The agent learns to consistently perform a backflip, land upright, and repeat. 2. The Half-Cheetah robot moving forward while standing on one leg. This behavior was trained using 800 queries in under an hour. 3. Keeping alongside other cars in Enduro. This was trained with roughly 1,300 queries and 4 million frames of interaction with the environment; the agent learns to stay almost exactly even with other moving cars for a substantial fraction of the episode, although it gets confused by changes in background. Figure  3 : Performance of our algorithm on MuJoCo tasks after removing various components, as described in Section Section 3.3. All graphs are averaged over 5 runs, using 700 synthetic labels each. Videos of these behaviors can be found at https://goo.gl/MhgvIU. These behaviors were trained using feedback from the authors.
 Ablation Studies In order to better understand the performance of our algorithm, we consider a range of modifications: 1. We pick queries uniformly at random rather than prioritizing queries for which there is disagreement (random queries). 2. We train only one predictor rather than an ensemble (no ensemble). In this setting, we also choose queries at random, since there is no longer an ensemble that we could use to estimate disagreement. 3. We train on queries only gathered at the beginning of training, rather than gathered throughout training (no online queries). 4. We remove the `2 regularization and use only dropout (no regularization). 5. On the robotics tasks only, we use trajectory segments of length 1 (no segments). 6. Rather than fitting r using comparisons, we consider an oracle which provides the true total reward over a trajectory segment, and fit r to these total rewards using mean squared error (target). The results are presented in Figure  3  for MuJoCo and Figure  4  for Atari. Training the reward predictor offline can lead to bizarre behavior that is undesirable as measured by the true reward  (Amodei et al., 2016) . For instance, on Pong offline training sometimes leads our agent to avoid losing points but not to score points; this can result in extremely long volleys (videos at https://goo.gl/L5eAbk). This type of behavior demonstrates that in general human feedback needs to be intertwined with RL rather than provided statically. Our main motivation for eliciting comparisons rather than absolute scores was that we found it much easier for humans to provide consistent comparisons than consistent absolute scores, especially on the continuous control tasks and on the qualitative tasks in Section 3.2; nevertheless it seems important to understand how using comparisons affects performance. For continuous control tasks we found that predicting comparisons worked much better than predicting scores. This is likely because the scale of rewards varies substantially and this complicates the regression problem, which is smoothed significantly when we only need to predict comparisons. In the Atari tasks we clipped rewards Figure  4 : Performance of our algorithm on Atari tasks after removing various components, as described in Section 3.3. All curves are an average of 3 runs using 5,500 synthetic labels (see minor exceptions in Section A.2). and effectively only predicted the sign, avoiding these difficulties (this is not a suitable solution for the continuous control tasks because the magnitude of the reward is important to learning). In these tasks comparisons and targets had significantly different performance, but neither consistently outperformed the other. We also observed large performance differences when using single frames rather than clips.  7  In order to obtain the same results using single frames we would need to have collected significantly more comparisons. In general we discovered that asking humans to compare longer clips was significantly more helpful per clip, and significantly less helpful per frame. Shrinking the clip length below 1-2 seconds did not significantly decrease the human time required to label each clip in early experiments, and so seems less efficient per second of human time. In the Atari environments we also found that it was often easier to compare longer clips because they provide more context than single frames.
 Discussion and Conclusions Agent-environment interactions are often radically cheaper than human interaction. We show that by learning a separate reward model using supervised learning, it is possible to reduce the interaction complexity by roughly 3 orders of magnitude. Although there is a large literature on preference elicitation and reinforcement learning from unknown reward functions, we provide the first evidence that these techniques can be economically scaled up to state-of-the-art reinforcement learning systems. This represents a step towards practical applications of deep RL to complex real-world tasks. In the long run it would be desirable to make learning a task from human preferences no more difficult than learning it from a programmatic reward signal, ensuring that powerful RL systems can be applied in the service of complex human values rather than low-complexity goals.

			 Here we assume here that the reward is a function of the observation and action. In our experiments in Atari environments, we instead assume the reward is a function of the preceding 4 observations. In a general partially observable environment, we could instead consider reward functions that depend on the whole sequence of observations, and model this reward function with a recurrent neural network.3 Wilson et al. (2012) also assumes the ability to sample reasonable initial states. But we work with high dimensional state spaces for which random states will not be reachable and the intended policy inhabits a low-dimensional manifold.
			 Equation 1 does not use discounting, which could be interpreted as modeling the human to be indifferent about when things happen in the trajectory segment. Using explicit discounting or inferring the human's discount function would also be reasonable choices.
			 Note that trajectory segments almost never start from the same state.6  In the case of Atari games with sparse rewards, it is relatively common for two clips to both have zero reward in which case the oracle outputs indifference. Because we considered clips rather than individual states, such ties never made up a large majority of our data. Moreover, ties still provide significant information to the reward predictor as long as they are not too common.
			 We only ran these tests on continuous control tasks because our Atari reward model depends on a sequence of consecutive frames rather than a single frame, as described in Section A.2