Abstract: Given two sources of evidence about a latent variable, one can combine the information from both by multiplying the likelihoods of each piece of evidence. However, when one or both of the observation models are misspecified, the distributions will conflict. We study this problem in the setting with two conflicting reward functions learned from different sources. In such a setting, we would like to retreat to a broader distribution over reward functions, in order to mitigate the effects of misspecification. We assume that an agent will maximize expected reward given this distribution over reward functions, and identify four desiderata for this setting. We propose a novel algorithm, Multitask Inverse Reward Design (MIRD), and compare it to a range of simple baselines. While all methods must trade off between conservatism and informativeness, through a combination of theory and empirical results on a toy environment, we find that MIRD and its variant MIRD-IF strike a good balance between the two.

Dmitrii Krasheninnikov
Rohin Shah
U C Berkeley

Combining reward information from multiple sources

Introduction While deep reinforcement learning (RL) has led to considerable success when given an accurate reward function  [28, 21] , specifying a reward function that captures human preferences in real-world tasks is challenging  [1, 6] . Value learning seeks to sidestep this difficulty by learning rewards from various types of data, such as natural language  [22] , demonstrations  [33, 11] , comparisons  [6] , ratings  [7] , human reinforcement  [18] , proxy rewards  [13] , the state of the world  [27] , etc. Many approaches to value learning aim to create a distribution over reward functions  [14, 23, 24, 13] .  A natural idea is to combine the information from these diverse sources. However, the sources may conflict. For example, in Figure  1 , Alice might say that she wants the healthy avocado toast, but empirically she almost always chooses to eat the tasty cake. A reward learning method based on natural language may confidently infer that Alice would prefer the toast, whereas a method based on revealed preferences  [25]  might confidently infer that she wants cake. Presumably one of the algorithms made a poor assumption: perhaps the natural language algorithm failed to model that Alice only said that she wants avocado toast to impress her friends. Alternatively, maybe Alice wants to start eating healthy, but chooses the cake in moments of weakness, and the revealed preferences algorithm failed to model this bias. While it is unclear what the "true" reward is, we do know that at least one of the methods is confidently wrong. We would like to defer on the question of what the "true" reward is, and instead retreat to a position of uncertainty. Formally, we want a distribution over the true reward function θ * given the inferred rewards θ 1 , θ 2 . We could apply Bayes Rule: p(θ * | θ 1 , θ 2 ) ∝ p(θ * )p( θ 1 | θ * )p( θ 2 | θ * ). However, when rewards conflict due to misspecification, one of the likelihoods will be very low, and the posterior will be low everywhere, likely leading to garbage after normalization  [10] . While it is possible to mitigate this by making the assumptions more realistic, e.g. by modeling human biases  [26, 19, 9] , it is very hard to get a perfect model  [3, 29] . So, we aim for a method that is robust to misspecification. What makes for a good reward combination method? To answer this question, we must know how the resulting distribution will be used. As in Hadfield-Menell et al.  [14] , we assume that an agent maximizes expected reward, given that the agent can gather more information about the reward. In this framework, an agent must balance three concerns: actively learning about the reward  [24, 20] , preserving its ability to pursue the true reward in the future when it knows more  [30, 17] , and acting to gain immediate reward. Each concern suggests different desiderata for a reward combination method, which we formalize for rewards functions that are linear in features of the state. There are several reasonable methods for combining reward functions based on the intuition that we should put weight on any parameter values "in between" the parameters of the given reward functions. These reward-space methods operate in the space of parameters of the reward. However, this ignores any details about the environment and its features. We introduce a method called Multitask Inverse Reward Design (MIRD) that leverages the environment to infer which reward functions are compatible with the behaviors "in between" the behaviors incentivized by the two input rewards, making it a behavior-space method. Through a combination of theory and empirical results on a simple environment, we evaluate how well the proposed methods meet our desiderata, and conclude that MIRD is a good option when the reward distribution must be used to act, while its variant MIRD-IF is better for increased robustness and a higher chance of support on the true reward.
 Background Markov Decision Process (MDP). A MDP M is a tuple M = S, A, T , r, H , where S is the set of states, A is the set of actions, T : S × A × S → [0, 1] is the transition probability function, r : S → R is the reward function, and H ∈ Z + is the finite planning horizon. We consider MDPs where the reward is linear in features, and does not depend on action: r(s; θ) = θ T f (s), where θ are the parameters defining the reward function and f computes features of a given state. A policy π : S → A specifies how to act in the MDP. A trajectory τ : (S × A) * is a sequence of states and actions, where the actions are sampled from a policy π and the states are sampled from the transition function T . We abuse notation and write f (τ ) to denote s∈τ f (s). The feature expectations (FE) of policy π are the expected feature counts when acting according to π: F π = E τ ∼π [f (τ )]. The value of a policy π is given by V π = E τ ∼π [ s∈τ r(s; θ)] = θ T F π . The optimal policy is π * = max π V π . Reward learning by simulating the past (RLSP). Shah et al.  [27]  note that environments where humans have acted are already optimized for human preferences, and so contain reward information. The RLSP algorithm considers the current state s 0 as the final state in a human trajectory τ = (s −H , a −H , ..., s 0 , a 0 ) generated by the human with preferences θ. The RLSP likelihood p(θ | s 0 ) is obtained by marginalizing out (s −H , a −H , . . . s −1 , a −1 , a 0 ), since only s 0 is actually observed. Inverse Reward Design (IRD). Inverse reward design (IRD) notes that since reward designers often produce rewards by checking what the reward does in the training environment using an iterative trial-and-error process, the final reward they produce is likely to produce good behavior in the environments that they used during the reward design process. So, θ need not be identical to θ * , but merely provides evidence about it. Formally, IRD assumes that the probability of specifying a given θ is proportional to the exponent of the expected true return in the training environment: p( θ | θ * ) ∝ exp(θ * F π θ ). IRD inverts this generative model of reward design to sample from the posterior of the true reward: p(θ * | θ) ∝ p( θ | θ * )p(θ * ).
 Uses of the reward distribution So far, we have argued that we would like to combine the information from two different sources into a single distribution over reward functions. But what makes such a distribution good? We take a pragmatic approach: we identify downstream tasks for such a distribution, find desiderata for a reward distribution for these tasks, and use these desiderata to guide our search for methods.
 Optimizing the reward We follow the framework of Hadfield-Menell et al.  [14]  in which an agent maximizes expected return under reward uncertainty. In this setting, maximizing the return causes the agent to use its actions both to obtain (expected) reward, and to gather information about the true reward in order to better obtain reward in the future  [14, 31] . We might expect that an agent could first learn the reward and then act; however, this is unrealistic: we would be unhappy with a household robot that had to learn how we want walls to be painted before it would bake a cake. The agent must act given reward uncertainty, but maintain its ability to optimize any of the potential rewards in the future when it will know more about the reward. Put simply, it needs to preserve its option value while still obtaining as much reward as it can currently get. We would like to evaluate our reward distribution by plugging it into an algorithm that maximizes expected reward given reward uncertainty, but unfortunately current methods  [14, 31]  are computationally expensive and only work in environments where option value preservation is not a concern. So, we analyze reward learning and option value preservation separately. Active reward learning. An agent that learns rewards while acting will need to select queries with care. Active reward learning aims to pick queries that maximize the information gained about the reward, or minimize the expected regret  [24, 20]  3 . We analyse how the reward distributions resulting from different reward combination methods fit as priors for the active reward learning methods. Option-value preservation. Low impact agents  [30, 17, 2]  take fewer irreversible actions, and so preserve their ability to optimize multiple reward functions. However, work in this area does not consider the possibility of learning more information about the reward function, and so these methods penalize entire plans that would have a high impact. In contrast, we would like our agent to pursue impactful plans, but pause when option value would be destroyed, and get more reward information. For our purposes, we start with the attainable utility preservation (AUP) method introduced by Turner et al.  [30] . AUP formalizes the impact of an action on a reward function as the change in the Q-value as a result of taking that action (relative to a no-op action ∅). For our purposes, we only care about cases where we are no longer able to optimize the reward function, as opposed to ones where we are better able to optimize the reward function. So, we use the truncated difference summary function proposed by Krakovna et al.  [17]  that only counts decreases in Q-values and not increases: Impact(s, a) = E r [max(Q r (s, ∅) − Q r (s, a), 0)]. (1) Turner et al.  [30]  penalize the agent whenever it would cause too much impact, using a hyperparameter λ to trade off task reward with impact. They plan using the reward R task − λ Impact(s, a). However, by penalizing impact in the reward function, the agent is penalized for any plan that would cause impact, whereas we want our agent to start on potentially impactful plans, but stop once a particular action would destroy option value. So, we only penalize impact during action selection: π(s) = max a (Q(s, a) − λ Impact(s, a)). (2)
 Acting in a new environment Often, we wish to use the reward function in the same environment as the one in which it was specified or inferred. For example, a human expert may have demonstrated how to bake a cake in a particular kitchen. Since we know at training time that the agent will be deployed in the same kitchen, the only guarantee we need is that the agent's behavior is correct. If two rewards lead to the same behavior in this kitchen, we don't care which one we end up using. However, we may instead want to apply a reward function learned in one environment to a different environment. In this case, there may be new options in the test environment that we never saw during training. As a result, even if two rewards led to identical behavior in the training environment, they need not do so in the test environment. Hence when the goal is to transfer the reward distribution to a new environment, it is crucial to ensure that the distribution contains the true reward, instead of just containing a reward that gives rise to the same behavior as the true one in the training environment.
 Desiderata for the reward posterior We discuss properties of the posterior that inform the choice of a reward combination algorithm for different applications of the resulting reward distribution. We focus on two applications: option-value preserving planning and active reward learning.
 Desired properties for any subsequent application All else equal, we want the true reward to have high probability in the reward distribution. One model for misspecification is that the weight for each feature has some chance of being "corrupted", and the chance of the feature being corrupted in one reward is independent from that of the other reward. If the fraction of misspecified values is small, then if two reward vectors' values are different at a particular index, it is likely that one of the values is correct. Given this, it is desirable for the reward distribution to assign significant probability to every reward function s.t. the value of the weight for each feature is in one of the values of that feature's weight of the input rewards. Formally, we denote the set of such reward parameters Θ indep pts = {θ ∈ R n | θ i ∈ { θ 1 i , θ 2 i } ∀i ∈ {1, ..., n}}. (3) Desideratum 1 (Support on Θ indep pts ). All θ * ∈ Θ indep pts should be in the support of p(θ * | θ 1 , θ 2 ). Note that when we do not need to transfer the reward function to a new environment, Desideratum 1 can be made weaker. We only require that the distribution have support on reward functions that give rise to the same behavior as the rewards in Θ indep pts , and not necessarily on Θ indep pts itself. In order for the agent to act, the reward distribution must provide information about useful behavior: Desideratum 2 (Informative about desirable behavior). If θ 1 and θ 2 give rise to the same behavior, most of the reward functions in the distribution should give rise to this behavior. The strong version of this desideratum states that if the FEs of { θ 1 , θ 2 } are equal on some feature, the FEs of this feature of any reward in the support of the posterior are equal to those of { θ 1 , θ 2 }: ∀i ∈ {1, n} : F π θ 1 i = F π θ 2 i =⇒ ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * i = F π θ 1 i = F π θ 2 i . (4) The medium version considers the FEs as a whole instead of per-feature: F π θ 1 = F π θ 2 =⇒ ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = F π θ 1 = F π θ 2 . ( 5 ) The weak version assumes that the FEs of all θ ∈ Θ indep pts are the same, rather than just θ 1 and θ 2 : ∀θ ∈ Θ indep pts : F π θ = F π θ 1 = F π θ 2 =⇒ ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = F π θ 1 . (6) Des. 2 allows option-value preserving planning to do useful things instead of performing the no-op action all the time. Furthermore, by restricting possible behaviors, Des. 2 simplifies the job of approaches that actively learn the reward function. If the reward distribution used by our household robot always incentivizes using soap while washing dishes (because both input rewards did so), the robot's active learning algorithm does not need to query the human about whether or not to use soap. Des. 2 might be problematic if both input rewards agree on an undesirable behavior w.r.t. a particular feature. This presents a natural tradeoff between informativeness and the abilities to preserve option value and actively learn arbitrary reward functions: the more informative we assume the input rewards are about the desired behavior, the less robust we are to their misspecification.
 Desired properties for option value preservation Suppose we would like to preserve the option value of pursuing two different behaviors. Option-value preservation is more robust when the distribution over FEs induced by the reward distribution places significant and approximately equal weight on the two behaviors. Intuitively, the "weight" AUP puts on preserving expected return attainable by a particular behavior is proportional to the probability of that behavior in the reward distribution. So we would like the FEs of the input rewards to be balanced across the reward distribution. Concretely: Desideratum 3 (Behavior-space balance). The FEs of the rewards sampled from the distribution should correspond to behaviors arising from the input reward functions with a similar frequency: p(F π θ = F π θ 1 ) ≈ p(F π θ = F π θ 2 ) | θ ∼ p(θ | θ 1 , θ 2 ). (7)
 Desired properties for active reward learning Given a reward vector we define the tradeoff point between the i-th and j-th features of the state as the ratio between the reward vector values corresponding to these features: T θ i,j = θi θj . The true reward's tradeoff point may be in between the tradeoff points of the input rewards, and so we would like our distribution to support all such tradeoff points. Desideratum 4 (Support on intermediate feature tradeoffs). The reward posterior should have support on rewards that prescribe all intermediate tradeoff points between the states' features. In practice tradeoff points only affect the behavior to the extent allowed by the environment. We only need to maintain all intermediate tradeoff points if we want to transfer to a new environment.
 Multitask Inverse Reward Design Inspired by Inverse Reward Design  [13] , we introduce a reward combination method Multitask Inverse Reward Design (MIRD). We assume that the two input rewards meet the IRD assumption for two different rewards, namely that the input rewards generate nearly optimal behavior in the training environment according to the corresponding task rewards. A straightforward way to construct the distribution for the combined reward function is to require the FEs of the rewards in the distribution to be convex combinations of the FEs of the input rewards: F π θ * ∈ {bF π θ 1 + (1 − b)F π θ 2 | b ∈ [0, 1]}. To satisfy this requirement, we propose a distribution that depends on the behavior (set of trajectories) D = {τ 1 , ..., τ d } resulting from the agents optimizing θ 1 and θ 2 : p(θ * | θ 1 , θ 2 ) = D p(θ * | D)p(D | θ 1 , θ 2 ) dD (8) Each of the reward functions {θ i | i = {1, ..., d}} used to generate the trajectories τ i in D is sampled from the Bernoulli mixture over the input reward functions. We introduce a random variable b ∈ [0, 1] that determines the probability of sampling θ i = θ 1 to generate the trajectory τ i : p(D | θ 1 , θ 2 ) = b Beta(b | β 1 , β 2 ) n i=1 bp(τ i | θ 1 ) + (1 − b)p(τ i | θ 2 ) db (9) We sample from p(τ i | θ i ) using policies arising from soft value iteration. For p(θ * | D), we set θ * to the reward learned by Maximum Causal Entropy Inverse RL (MCEIRL)  [33] . Denoting the Dirac delta as δ(), we have p(θ * | D) = δ(θ * − argmax θ p(θ | D)). Having defined all relevant conditional distributions, we can sample from the joint p(θ * , b, D, θ i | θ 1 , θ 2 ) and the marginal p(θ * | θ 1 , θ 2 ). Theorem 1. The FEs arising from rewards θ * in the support of the MIRD posterior are convex combinations of FEs of the input reward vectors θ 1 and θ 2 : ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = αF π θ 1 + (1 − α)F π θ 2 , α ∈ [0, 1]. (10) Proof. See appendix A for full proof. Key to the proof is the fact that MCEIRL finds such θ * that its FEs match those of the soft value iteration expert who generated the trajectories  [33] . In our case, the FEs of θ * match D, which contains trajectories arising from θ 1 and θ 2 in various proportions. Corollary 1.1. The average FEs of the reward functions in the posterior are: E θ∼p(θ| θ 1 , θ 2 ) [F π θ ] = E b [bF π θ 1 + (1 − b)F π θ 2 ] = E b [b] F π θ 1 + (1 − E b [b])F π θ 2 . ( 11 ) Corollary 1.2. Eq. 11 implies that if the FEs of the θ 1 and θ 2 w.r.t. a given feature are equal, the FEs of all rewards in the distribution for that feature equal the FEs of θ 1 and θ 2 for that feature. Thus MIRD satisfies the strong version of Des. 2 (informative about desirable behavior). Corollary 1.3. When β 1 = β 2 the probability of sampling θ i = θ 1 equals the probability of sampling θ i = θ 2 , and hence the probabilities of observing behaviors F π θ 1 and F π θ 2 when optimizing a reward function from the posterior are equal. Hence Des. 3 (behavior balance) is satisfied. Theorem 1 allows us to bound the expected return θ true F π θ , θ ∈ supp(p(θ * | θ 1 , θ 2 )) of the true reward function θ true in terms of expected true returns of F π θ 1 and F π θ 2 : Theorem 2. The minimum expected true return following any of the rewards in the MIRD posterior equals the minimum true return following one of the input rewards: min θ∈supp(p(θ * | θ 1 , θ 2 )) θ true F π θ = min θ∈{ θ 1 , θ 2 } θ true F π θ . ( 12 ) Proof. See Appendix B. Theorem 2 is the best possible regret bound for following rewards in a distribution resulting from reward combination: the worst the agent can do when following a reward from the distribution is no worse than when just randomly choosing to follow one of the input reward functions. Generally FEs arising from θ ∈ Θ indep pts \{ θ 1 , θ 2 } are not convex combinations of F π θ 1 and F π θ 2 , so MIRD does not satisfy any version of Des. 1 (support on Θ indep pts ). In addition, when θ 1 and θ 2 conflict on more than one feature, Des. 4 (support on intermediate feature tradeoffs) is not satisfied. Independent features formulation. Since both Des. 1 and Des. 4 aim to ensure robustness to reward misspecification, one of our primary goals, we would like a variant of MIRD that satisfies them. An algorithm that supports rewards that give rise to all potential behaviors arising from reward functions in Θ indep pts is a simple modification of MIRD. The only change required in the original formulation of MIRD is sampling the reward vectors {θ i } giving rise to the trajectories in D from Θ indep pts instead of just { θ 1 , θ 2 }. Now b is a vector of dimensionality |Θ indep pts |, sampled from Dirichlet(b | β 1 , ..., β |Θ indep pts | ) . Each θ i is sampled from a Multinoulli distribution over Θ indep pts , parameterized by b. We refer to this version of MIRD as MIRD independent features, or MIRD-IF.
 Analysis We first introduce several baselines each of which does not meet some of the outlined desiderata, and then analyze all reward combination methods in the context of our modified version of AUP.
 Reward-space baselines Additive. The additive reward combination method is commonly used in previous work utilizing reward combination  [8, 27, 16] . The reward posterior is simply p(θ * | θ 1 , θ 2 ) = δ(θ * − ( θ 1 + α θ 2 )), where α is the hyperparameter used to balance the two input reward functions. Gaussian. Here, we use the standard Bayesian framework, and assume that θ 1 and θ 2 are generated from θ * with Gaussian noise: p(θ * | θ 1 , θ 2 ) ∝ p(θ * )p( θ 1 | θ * )p( θ 2 | θ * ) = p(θ * )p N ( θ 1 | θ * , σ 2 1 I)p N ( θ 2 | θ * , σ 2 2 I ), where I is the identity matrix, p N is the multivariate Gaussian probability density function, and {σ 1 , σ 2 } are hyperparameters controlling the standard deviations. One can express higher trust in a given input reward by lowering its σ. For our experiments, we use an uninformative Gaussian prior (that is, σ → ∞) and set σ 1 = σ 2 = 1. The posterior is then p(θ * | θ 1 , θ 2 ) = p N (θ * | θ 1 + θ 2 2 , 1 2 I).
 Convex combinations of input reward vectors (CC-in). Consider a scenario where the input reward functions are correctly specified for two different tasks. A natural reward-space analogue to MIRD would be to consider a distribution over all convex combinations of the input reward vectors: p(θ * | θ 1 , θ 2 ) = b Beta(b | β 1 , β 2 )δ(θ * − b θ 1 − (1 − b) θ 2 ) db. ( 13 ) Theorem 3. If F π θ 1 = F π θ 2 , the FEs of any reward in the support of the CC-in posterior equal F π θ 1 . Hence CC-in satisfies the medium version of Des. 2 (informative about desirable behavior). Proof. See Appendix C. We provide an example showing that CC-in does not satisfy the strong version of Des 2 in Appendix F. Furthermore, CC-in does not satisfy Des. 1 (support on Θ indep pts ): consider combining rewards [0, 1, 1] and [1, 0, 1]. Here θ = [0, 0, 1] ∈ Θ indep pts , but is not a convex combination of input rewards. Convex combinations of θ ∈ Θ indep pts (CC-Θ indep pts ). Instead of only considering convex combinations of the input rewards, we consider convex combinations of θ ∈ Θ indep pts . The vector b ∈ R |Θ indep pts | of weights for each convex combination is sampled from the Dirichlet distribution:  To demonstrate the use of the desiderata, as well as how each method performs on these desiderata, we developed a toy environment in which to test reward combination methods. Due to space limitations, we defer full qualitative and quantitative explanations to Appendix E and report the broad results here. p(θ * | θ 1 , θ 2 ) = b Dirichlet(b | β 1 , ..., β |Θ indep pts | )δ   θ * − |Θ indep pts | i=0 b i Θ indep pts i   db. (14
 Analysis on a toy environment Setup. We use the setting of AUP planning to demonstrate the importance of support on Θ indep pts , informativeness about desired the behavior, and behavior balance desiderata. We obtain the reward posterior by combining the input rewards using each of our seven reward combination methods, and qualitatively evaluate the behavior of AUP planning w.r.t. the posterior over five seeds. The input rewards are either both specified by hand, or θ 1 is learned with RLSP while θ 2 is hand-specified. We use the cooking environment described in Figure  2 . The states' features indicate the number of jars of flour, pieces of dough, cakes, avocado toasts, and servings of pasta. At each state the agent can move, cook, or do nothing. Cooking is possible when the agent is to the left of flour or dough. Support on Θ indep pts . We specify both input rewards to strongly incentivise either cake or toast by setting θ 1 cake = θ 2 toast = 3; θ 1 toast = θ 2 cake = 0, and to mildly incentivise pasta: θ 1 pasta = θ 2 pasta = 1. The desired behavior is to preserve the flour, as any one of the foods might be desirable. This requires the distribution to place weight on rewards such as θ cake = θ toast = 0; θ pasta = 1 which would make pasta. As expected, methods that have support on Θ indep pts behave as desired, and other methods do not. Informativeness about desirable behavior (strong). In the setting above the two input rewards lead the agent to make either cake or toast, but not pasta. So, a reward distribution meeting the strong version of Des. 2 would only make cake or toast, and so the agent should make dough (which is useful for both). Note that despite the same setup, this is exactly the opposite behavior than desired previously, showing the tradeoff between informativeness and support on Θ indep pts . As such, the results are reversed. Informativeness about desirable behavior (weak). We hand-specify one of the reward functions to incentivise the agent to make cakes: θ 2 cake = 1, θ 2 toast = θ 2 pasta = 0. θ 1 is inferred by RLSP with a uniform prior over s −T . Since the current state contains a cake, RLSP infers a positive reward for cakes, θ 1 cake ≈ 0.1, and near-zero rewards for the other features. Hence we have two rewards that both incentivise the agent to make another cake. The desired behavior is to make the cake, and not worry about preserving the ability to make pasta or toast as neither input reward cares about it. All but the Gaussian method satisfy the weak informativeness desideratum, and so they all succeed. The Gaussian method fails since it thinks θ * cake could be negative. Behavior balance. θ 2 is specified to only reward toast, θ 2 toast = 1, θ 2 cake = θ 2 pasta = 0, and θ 1 is specified only reward cake: θ 1 cake > 0, θ 1 toast = θ 1 pasta = 0. So we have two disagreeing rewards: one incentivises cake, while the other incentivises toast. The desired behavior from AUP here is to make dough as it is desirable for both the cake and the toast, but then to stop as the next step is unclear. Ensuring that the reward for toast does not overwhelm the reward for cake requires behavior balance. We vary θ 1 cake between 0.01 and 0.7 to obtain many degrees of imbalance between toast and cake. MIRD and MIRD-IF have the lowest frequency of the toast reward overwhelming the cake reward, which indicates that MIRD and MIRD-IF are most behavior balanced. However, occasionally these methods do nothing instead of making dough.
 Related Work Utility aggregation. Combining the specified reward functions is in many ways similar to combining utility functions of different agents. Harsanyi's aggregation theorem  [15]  suggests that maximizing a fixed linear combination of utility functions results in Pareto-optimal outcomes for correctly specified utility functions. This corresponds to our Additive baseline. Multi-task and meta IRL. Multi-task IRL assumes that the demonstrations originate from multiple experts performing different tasks, and seeks to recover reward functions that explain the preferences of each of the experts  [4, 5] . The IRL part of MIRD and MIRD-IF is conceptually similar to these approaches: the trajectories in each demonstration set D can be seen as arising from an expert executing different tasks. However, MIRD seeks to explain each D as if the trajectories arose from a single reward function . This resembles meta IRL, which learns an initialization for the reward function parameters that is helpful for learning the rewards of new tasks  [32, 12] .
 Limitations and future work Summary. We analyze the problem of combining two potentially misspecified reward functions into a distribution over rewards. We identify active reward learning and option value preservation as two key applications for such a distribution, and determine four properties that the distribution would ideally satisfy for these applications. We suggest some simple reward-space methods as well as a behavior-space method, Multitask Inverse Reward Design (MIRD), which is grounded in the behavior arising from the input rewards. MIRD works well for most applications, and MIRD-IF can be used for greater robustness and increased likelihood that the true reward is in the distribution. If the only application is to preserve option value, then the uniform points method also works well. Future work. The primary avenue for future work is to evaluate the reward combination methods in the context of active reward learning, and formalize Des. 4 (support on intermediate tradeoffs) in a way most helpful for active learning. Another avenue for future work is scaling up MIRD to realistic environments, where the reward function may be nonlinear.
 A Proof of Theorem 1 D is a set of trajectories each of which arises either from optimizing θ 1 or optimizing θ 2 . The fraction of trajectories in D that result from θ 1 is b, and the fraction of trajectories arising from θ 2 is 1 − b. Hence the FEs of D are F D = bF π θ 1 + (1 − b)F π θ 2 . ( 15 ) p(θ * | D) = δ(θ * − argmax θ p(θ | D) ) entails that each sample θ * is the MCEIRL reward θ * = argmax θ p(θ | D). MCEIRL finds a reward vector θ * such that its FEs match those of the soft value iteration expert who generated the trajectories  [33] . Hence each sampled θ * gives rise to FEs F π θ * matching D. This and Eq. 15 together imply that the FEs arising from optimizing any θ * ∈ supp(p(θ * | θ 1 , θ 2 )) with soft value iteration are a convex combination of the FEs of the input rewards: ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = bF π θ 1 + (1 − b)F π θ 2 , b ∈ [0, 1]. (16)
 B Proof of Theorem 2 By Theorem 1, the FEs arising from any θ in the posterior are a convex combination of the FEs of the input rewards. Hence finding θ in the support of p(θ * | θ 1 , θ 2 ) that minimizes θ true F π θ is equivalent to finding α ∈ [0, 1] that minimizes θ true (αF π θ 1 + (1 − α)F π θ 2 ) = αθ true F π θ 1 + (1 − α)θ true F π θ 2 . Since the true returns when following policies arising from the input reward functions θ true F π θ 1 and θ true F π θ 2 are scalars, the term above is minimized with α = 1 when θ true F π θ 1 < θ true F π θ 2 , and α = 0 otherwise.
 C Proof of Theorem 3 Since the feature expectations (FEs) of the input rewards are equal, the return of any mixture reward b θ 1 − (1 − b) θ 2 for those FEs is F π θ 1 (b θ 1 − (1 − b) θ 2 ) . Suppose there exist FEs F that result in a higher expected return for the mixture reward: F (b θ 1 − (1 − b) θ 2 ) > F π θ 1 (b θ 1 − (1 − b) θ 2 ) (17) For this to be true F must result in a higher return than F π θ 1 for either θ 1 or θ 2 . This is a contradiction, since F π θ 1 already results in the highest return for both input rewards. Hence F π θ 1 corresponds to the optimal behavior for any mixture reward b θ 1 − (1 − b) θ 2 ∀b ∈ [0, 1].
 D Proof of Theorem 4 Analogous to the proof of Theorem 3 (Appendix C).
 E Analysis details In this appendix we provide more details and explanations regarding the behavior of the various reward combination methods in the Cooking environment.
 E.1 Support on Θ indep pts We hand-specify both input reward functions to strongly incentivise either cake or toast by setting θ 1 cake = θ 2 toast = 3; θ 1 toast = θ 2 cake = 0, and to mildly incentivise pasta: θ 1 pasta = θ 2 pasta = 1. The desired behavior from AUP here is to preserve the flour, as any one of the foods requiring it might be desirable. The Additive method returns a single reward function that incentivises the agent to make both cake and toast. AUP applied to one reward function simply optimizes it without needing to preserve option value, so here it makes either cake or toast (and does not preserve the flour). The CC-in method does not preserve the flour, choosing to make dough instead. Because of the high values of θ 1 cake and θ 2 toast , optimizing any convex combination of { θ 1 , θ 2 } leads the agent to either make cake or toast, and so AUP infers that it does not need to preserve the ability to make pasta. MIRD chooses to make dough for a similar reason: since all trajectories in D arise from optimizing θ 1 or θ 2 , each trajectory demonstrates the agent making either cake or toast, and no trajectory demonstrates it making pasta. So no reward inferred from D incentivises the agent to make pasta, and hence AUP does not preserve the ability to make pasta. All methods whose reward samples sometimes give rise to the behaviors arising from θ ∈ Θ indep pts preserve the flour. This is because Θ indep pts contains θ s.t. optimizing it leads the agent to make pasta: θ cake = θ toast = 0; θ pasta = 1. As MIRD-IF, Gaussian, uniform points, and CC-Θ indep pts methods all have substantial probability mass on rewards that incentivise making pasta, AUP applied to the reward samples from these methods preserves the ability to make pasta.
 E.2 Informativeness about desirable behavior (strong). In the setting above the two input rewards lead the agent to make either cake or toast, but not pasta. Here a reward posterior meeting the strong version of Des. 2 would only give rise to reward functions that incentivise the agent to make either cake or toast. Note that despite the same setup, this is exactly the opposite behavior than desired previously, showing the tradeoff between informativeness and support on Θ indep pts . As such, the results are reversed. We see that MIRD-IF, Gaussian, uniform points, and CC-Θ indep pts methods do not meet Des. 2 (strong), since they all have substantial probability mass on rewards that incentivise making pasta. Because of this AUP preserves the ability to make pasta, and does not lead the agent to make dough. On the other hand, MIRD, Additive, and CC-in methods never give rise to rewards incentivising the agent to make pasta, and AUP applied to these reward distributions results in making dough. Although Additive and CC-in do not support reward functions that lead the agent to make pasta in this particular case, they do not meet the strong version of Des. 2 (shown in Appendix F).
 E.3 Informativeness about desirable behavior (medium & weak) We hand-specify one of the reward functions to incentivise the agent to make cakes: θ 2 cake = 1, θ 2 toast = θ 2 pasta = 0. θ 1 is inferred by RLSP with a uniform prior over s −T . The current state contains a cake which was likely made by a human, so RLSP infers a positive reward for cakes, θ 1 cake ≈ 0.1, and near-zero rewards for the other features. Hence we have two agreeing rewards that both incentivise the agent to make another cake. The desired behavior from AUP planning here is to make the cake, and not worry about preserving the ability to make pasta or toast as neither input reward cares about it. The Gaussian posterior fails to make the cake. The distribution over θ * cake here is a Gaussian with µ ≈ 0.55 and σ ≈ 0.71. The posterior assigns substantial probability to θ * cake being negative, and AUP planning avoids making the cake to preserve the ability to optimize rewards that punish cakes. Here each reward vector in Θ indep pts leads the agent to make cake. Since every method except Gaussian satisfies at least the weak version of Des. 2, all reward functions in the support of their posteriors lead the agent to make cake. Because of this AUP infers that it is not important to preserve the ability to make pasta or toast, and successfully makes cakes when applied to rewards sampled from these posteriors.
 E.4 Behavior balance To better understand robustness of the reward combination methods to imbalanced input rewards, we analyze the performance of our methods by varying the extent of reward imbalance and the degree to which AUP seeks to preserve option value. The second input reward θ 2 is hand-specified to only reward toast, θ 2 toast = 1, θ 2 cake = θ 2 pasta = 0. The first reward θ 1 is hand-specified as well: θ 1 toast = θ 1 pasta = 0. We vary θ 1 cake between 0.01 and 0.7 to obtain many degrees of imbalance between toast and cake. So the input rewards disagree whether to make toast or cake, and agree that pasta is irrelevant. Hence the desirable action from AUP here would be to preserve the ability to choose between cake and toast, as either could be desirable. In addition to varying θ 1 cake , we vary the weight of the AUP penalty λ. Additive always fails and makes toast as it has support on a single reward function that prioritizes toast. Our results for the other six reward combination methods are shown in Figure  3 . We observe that MIRD and MIRD-IF are significantly more robust to imbalanced input rewards than any reward-space method. However, sometimes MIRD and MIRD-IF lead to AUP doing nothing instead of making dough. This is explained by the fact that MIRD and MIRD-IF use MCEIRL, which sometimes outputs negative rewards. For example, consider the case where in the process of generating a MIRD sample we generate 10 trajectories all of which demonstrate making toast. MCEIRL with these trajectories might output a positive reward on toast and a negative reward on cake and dough (instead of positive reward for toast and zero for cake and dough), even if a zero-centered Gaussian prior is used. These negative rewards on dough sometimes lead AUP to avoid making dough. Furthermore, we observe that MIRD is more robust than MIRD-IF, and CC-in is more robust than CC-Θ indep pts . This is because in this case support on rewards giving rise to behaviors arising from θ ∈ Θ indep pts makes the reward distributions less balanced. In particular, denoting the value of θ 1 cake we vary as x, Here only the second reward vector incentivises the agent to make cake, while both the third and the fourth rewards incentivise the agent to make toast. This slight imbalance of Θ indep pts in favor of making toast lead to CC-Θ indep pts and MIRD-IF posteriors being more likely to cause AUP to make toast instead of preserving the ability to choose between cake and toast. Finally, we observe that the Gaussian method is roughly equally robust to reward imbalance across the whole range of θ 1 cake . This is because unlike the other methods, the standard deviation of the Gaussian posterior does not depend θ 1 and θ 2 , but only depends on hyperparameters σ 1 and σ 2 , which are fixed to 1. Figure 1 : 1 Figure 1: Alice says she wants avocado toast, but her past actions imply she prefers cake to toast. What should the agent believe about the true reward θ * ?
 Figure 2 : 2 Figure 2: The Cooking environment. Left: the gridworld layout and current state. Right: irreversible actions for cooking. Making cake or avocado toast involves first making dough from the flour, and then making cake or toast; making pasta uses flour directly.
 Figure 3 : 3 Figure 3: Most common outcomes of AUP applied to the six reward posteriors over 5 seeds. Pink corresponds to AUP failure for more than half of the seeds: θ 2 toast overwhelms θ 1 cake and the agent makes toast. Purple corresponds to AUP success (AUP makes dough) for more than half of the seeds. Yellow corresponds to the middle-ground outcome of doing nothing in more than half of the seeds. The hyperparameters β of the Beta and the Dirichlet distributions are set to 0.5.
 θ toast, cake = [0, 0], θ toast, cake = [0, x], θ toast, cake = [1, 0], θ toast, cake = [1, x]
 ) Theorem 4. If all FEs of rewards in Θ indep pts are equal, the FEs of any reward in the CC-Θ indep pts posterior equal the FEs of rewards in Θ indep pts . Hence CC-Θ indep pts satisfies Des. 2 (weak). Uniform points. Note that as β 1 = ... = β |Θ indep pts | → 0, Eq. 14 essentially becomes p(θ * | θ 1 , θ 2 ) = Uniform(Θ indep pts ). It is helpful to analyze the properties of this uniform points distribution separately.Table 1 summarizes the properties of MIRD, MIRD-IF, and the reward-space baselines. Proof. See Appendix D.
 Table 1 : 1 Performance of reward combination algorithms on the desiderata outlined in Section 4. Support on Informative about desirable Behavior Support on intermediate Θ indep pts behavior balance feature tradeoffs (strong) (medium) (weak) Additive Gaussian CC-in ≈ CC-Θ indep pts ≈ Uniform Points MIRD MIRD-IF
			 We speculate that future active reward learning methods will also have to ensure that the queries are relevant and timely: in our household robot example, this would prevent the robot from asking about wall paint color when it is meant to be baking a cake.


Abstract: rtificial-intelligence assistants and recommendation algorithms interact with billions of people every day, influencing lives in myriad ways, yet they still have little understanding of humans. Self-driving vehicles controlled by artificial intelligence (AI) are gaining mastery of their interactions with the natural world, but they are still novices when it comes to coordinating with other cars and pedestrians or collaborating with their human operators. The state of AI applications reflects that of the research field. It has long been steeped in a kind of methodological individualism. As is evident from introductory textbooks, the canonical AI problem is that of a solitary machine confronting a non-social environment. Historically, this was a sensible starting point. An AI agent -much like an infant -must first master a basic understanding of

Allan Dafoe
Yoram Bachrach
Gillian Hadfield
Eric Horvitz
Kate Larson
Thore Graepel
Sean Gallup
/ Getty

Cooperative AI: machines must learn to find common ground

its environment and how to interact with it. Even in work involving multiple AI agents, the field has not yet tackled the hard problems of cooperation. Most headline results have come from two-player zero-sum games, such as backgammon, chess  1  , Go 2 and poker  3  . Gains in these competitive examples can be made only at the expense of others. Although such settings of pure conflict are vanishingly rare in the real world, they make appealing research projects. They are culturally cherished, relatively easy to benchmark (by asking whether the AI can beat the opponent), have natural curricula (because students train against peers of their own skill level) and have simpler solutions than semi-cooperative games do. AI needs social understanding and cooperative intelligence to integrate well into society. The coming years might give rise to diverse ecologies of AI systems that interact in rapid and complex ways with each other and with humans: on pavements and roads, in consumer and financial markets, in e-mail communication and social media, in cybersecurity and physical security. Autonomous vehicles or smart cities that do not engage well with humans will fail to deliver their benefits, and might even disrupt stable human relationships. We need to build a science of cooperative AI. As researchers in the field and its governance, we argue that it is time to prioritize the development of cooperative intelligence that has the ability to promote mutually beneficial joint action, even when incentives are not fully aligned. Just as psychologists studying humans have found that the infant brain does not develop fully without social interaction, progress towards socially valuable AI will be stunted unless we put the problem of cooperation at the centre of our research. Cooperative intelligence is unlikely to emerge as a by-product of research on other kinds of AI. We need more work on cooperative games and complex social spaces, on understanding norms and behaviours, and on social tools and infrastructure that promote cooperation. The AI community should learn more from, and contribute to, other fields that work on cooperation.
 From autonomy to cooperation Parents encourage their children to grow beyond their dependencies and become autonomous. But autonomy is rarely regarded as the sole goal for humans. Rather, we are generally most productive when we work cooperatively as part of broader society. Similarly, certain kinds of autonomy in AI systems are useful precisely because they enable the system to contribute effectively to broader cooperative efforts. Most of the value from self-driving vehicles will come not from driving on empty roads, but from vehicles coordinating smoothly with the flow of pedestrians, In most settings, people's incentives are not fully aligned. Nevertheless, they can often cooperate, taking joint action to achieve mutually beneficial outcomes. One example is countries agreeing and enforcing carbon cuts to tackle climate change. The cooperative intelligence needed to achieve this has four parts: Understanding. The ability to take into account the consequences of actions, to predict another's behaviour, and the implications of another's beliefs and preferences. Communication. The ability to explicitly and credibly share information with others relevant to understanding behaviour, intentions and preferences. Commitment. The ability to make credible promises when needed for cooperation.
 Norms and institutions. Social infrastructure -such as shared beliefs or rules -that reinforces understanding, communication and commitment.
 Four elements of cooperative intelligence by pure conflict -when there is no possibility of bargains, negotiations or threats. So, improving skill at inherently rivalrous games is unlikely to be the most promising way for AI to produce social value. Games of pure common interest are a step towards developing cooperative agents. The cooperative card game Hanabi 4 requires players to communicate private information and intentions under strong constraints about what can be said and when. Team games such as robot soccer  5  need players on a team to work as one, jointly planning their moves and passing the ball. In these examples, all agents on a team share the same goals. Mastering these games requires many skills essential to cooperation. Research avenues include building AIs that can understand what teammates are thinking and planning; communicate plans; and even cooperate with different kinds of teammate who might think differently and react more slowly (known as ad hoc teamwork). Yet because these situations are restricted to a perfect harmony of interests, they represent the easy case for cooperation. Real-world relationships almost always involve a mixture of common and conflicting interests. This tension gives rise to the rich texture of human cooperation problems, including bargaining, trust and mistrust, deception and credible communication, commitment problems and assurances, politics and coalitions, and norms and institutions. AI agents will need to learn how to manage these harder cooperation problems, as humans do. An example is the board game Diplomacy, in which players negotiate non-binding alliances with others. To succeed, AI agents will need to understand each other well enough to recognize when their interests are aligned with those of other players. They will have to develop a common vocabulary to communicate their intentions. They will benefit from being able to communicate credibly, despite possible incentives to lie. They must overcome mutual fears of betrayal, so as to agree on and execute jointly beneficial plans. They might even learn to establish norms relating to the adherence of agreements. To enable progress in these cooperative skills, researchers have devised variants of Diplomacy that modify the difficulty of these challenges, such as introducing an agreed simple vocabulary or permitting binding commitments.
 Human-AI cooperation AI is increasingly present, underlying everything from dynamic pricing strategies to loans and prison-sentencing decisions. Collaborative industrial robots work on factory floors alongside labourers  6  , care robots help human health workers and personal AI assistants (such as Amazon's Alexa, Apple's Siri and Google Assistant) help us with scheduling, albeit in an elementary way. cyclists and cars driven by humans. Thus, cooperative intelligence is not an alternative to autonomous intelligence, but goes beyond it. AI research on cooperation will need to bring together many clusters of work. A first cluster consists of AI-AI cooperation, tackling ever more difficult, rich and realistic settings (see 'Four elements of cooperative intelligence'). A second is AI-human cooperation, for which we will need to advance natural-language understanding, enable machines to learn about people's preferences, and make machine reasoning more accessible to humans. A third cluster is work on tools for improving (and not harming) human-human cooperation, such as ways of making the algorithms that govern social media better at promoting healthy online communities.
 AI-AI cooperation Multi-agent AI research has seen most success in two-player zero-sum settings, from the superhuman performance of IBM's chess-playing computer Deep Blue to the powerful demonstration of deep reinforcement learning by the program AlphaGo. However, few interactions in the real world are characterized The design of agents that will act in accordance with human intentions, preferences and values -known as AI alignment  7  -is a crucial part of cooperative AI. But it is only a part, because the relationship between a single human (acting as the principal) and a single machine (acting as the agent) isn't always clear. Real-world cooperation problems often involve multiple stakeholders, some conflicting interests and integration with our institutional and normative infrastructure. A particular challenge facing researchers working on human-AI cooperation is that it involves, well, humans. Today, many deployed machine-learning models are trained either on massive data sets or in simulated environments that can generate years of experience in seconds. For example, the program AlphaZero learnt to play chess by playing 44 million games against itself over 9 hours. By contrast, humans produce data slowly, and require researchers to consider compensation, ethics and privacy. There might be ways to use fewer human participants, such as extensively training AIs in simulation and then fine-tuning them in the real world. Many AI practitioners have dreamt of building autonomous and human-like intelligence. They envisioned systems that could replace human labour, acting with the speed and resilience of machines, and scaling up rapidly given increases in computing power, algorithmic efficiency and capital. However, unlike systems that have tight integration with human workers, autonomous systems might pose greater safety risks. Human-like AI might be more likely to displace labour. Instead, we could develop AI assistants that complement human intelligence and depend on us for tasks in which humans have a comparative advantage. As Stanford University radiologist Curtis Langlotz put it: "AI won't replace radiologists, but radiologists who use AI will replace radiologists who don't." Progress will require advances in understanding human language, gestures and activities, and ad hoc teamwork, in addition to preference learning by machines, safety, interpretability by humans  8  , and understanding of norms. Research will need to approach increasingly rich and realistic environments. Instead of benchmarking progress mainly by whether autonomous machines can outperform autonomous humans on a task, researchers should also assess the performance of human-machine teams.
 AI for human collaboration Humans confront ubiquitous cooperation problems as commuters, neighbours, co-workers and citizens. The global scientific community, for example, could benefit from better tools for identifying relevant work and promising collaborations. Technology is crucial, mediating our ability to find and process information, communicate and self-organize. Digital systems and AI can expand this toolkit. Some AI tools, such as machine language translation, seem strongly disposed towards promoting cooperation. Today, 2 people who speak any of more than 100 languages can communicate with the aid of a smartphone and a translation app. Digital platforms such as Wikipedia, Reddit and Twitter provide tools to combine user-provided content. AI advances could improve this community infrastructure, for example, by routing relevant information to contributors more efficiently to enhance collaborative editing. Other advances could improve user rating and reputation systems through better modelling and by accounting for the rater's repute or relevance, as well as by enabling recommendation algorithms that more Computer scientists in Leipzig, Germany, prepare their robot soccer team for a test game.
 "To succeed, cooperative AI must connect with the broader science of cooperation." intelligently promote a community's values. Building healthy online communities is challenging; just as social media can connect us, so too can it polarize, stress, misinform, distract and addict us  9  . Researchers and developers need to find better ways to name and measure desirable properties and build algorithms that encourage them. Platforms for political deliberation can be designed to promote empathy about different viewpoints and cultivate community consensus. Methods for achieving this include language comprehension that links to structured databases of knowledge, or clustering algorithms to identify related perspectives.
 Next steps To succeed, cooperative AI must connect with the broader science of cooperation, which spans the social, behavioural and natural sciences. AI research will need to converse with multiple fields. These include psychology, to understand human cognition; law and policy, to understand institutions; history, sociology and anthropology, to understand culture; and political science and economics, to understand problems of information, commitment and social choice. Adjacent research areas are developing AI with socially desirable properties, such as alignment, interpretability and fairness  10, 11  . Each of these addresses a distinct, but complementary, set of challenges. The need for interdisciplinarity is exemplified by a landmark work: Robert Axelrod's The Evolution of  Cooperation, published in 1984 (ref .12) . Axelrod, a political scientist, brought together game theorists, mathematicians, economists, biologists and psychologists in a tournament to help devise the best algorithms for the iterated Prisoner's Dilemma, the canonical example of why two rational people might not cooperate. The winning solution that cooperated most successfully, called Tit for Tat, was devised by Anatol Rapoport, a US scholar with a background spanning mathematics, biology, network science and peace studies. Axelrod's tournament offered another lesson. It gave researchers a benchmark for success in the design of cooperative algorithms, just as ImageNet 13 did for computer vision by collecting and labelling millions of photos. Cooperative AI research will similarly gain momentum if investigators can devise, agree on and adopt benchmarks that cover a diverse set of challenges: playing cooperative board games, integrating into massive multiplayer video games, navigating simplified environments that require machine-human interaction, and anticipating tasks as a personal assistant might. Similar to the state-of-the-art in language modelling, considerable effort and creativity will be needed to make sure these benchmarks remain sufficiently rich and ambitious, and do not have socially harmful blind spots. The most important challenges of cooperation might be the most difficult to benchmark; they involve creatively stepping out of our habitual roles to change the 'game' itself. Indeed, if we are to take the social nature of intelligence seriously, we need to move from individual objectives to the shared, poorly defined ways humans solve social problems: creating language, norms and institutions. Science is a social enterprise, so promoting research into cooperative AI will require social interventions. A recent milestone was a December 2020 workshop on cooperative AI at the leading machine-learning conference NeurIPS. It involved speakers from a diverse array of disciplines, and resulted in a review of Open Problems in Cooperative AI  14  . We and others are establishing a Cooperative AI Foundation to support this nascent field (www.cooperativeai.org), backed by a large philanthropic commitment. The foundation's mission will be to catalyse advances in cooperative intelligence to benefit all of humanity, including efforts to fund fellowships, organize conferences, support benchmarks and environments, and award prizes. The crucial crises confronting humanity are challenges of cooperation: the need for collective action on climate change, on political polarization, on misinformation, on global public health or on other common goods, such as water, soil and clean air. As the potential of AI continues to scale up, a nudge in the direction of cooperative AI today could enable us to achieve much-needed global cooperation in the future.

			 © 2 0 2 1 S p r i n g e r N a t u r e L i m i t e d . A l l r i g h t s r e s e r v e d .


Abstract: Autonomous AI systems' programmed goals can easily fall short of programmers' intentions. Even a machine intelligent enough to understand its designers' intentions would not necessarily act as intended. We discuss early ideas on how one might design smarter-than-human AI systems that can inductively learn what to value from labeled training data, and highlight questions about the construction of systems that model and act upon their operators' preferences.

The Value Learning Problem

Introduction Standard texts in AI safety and ethics, such as  Weld and Etzioni [1994]  or  Anderson and Anderson [2011] , generally focus on autonomous systems with reasoning abilities that are complementary and not strictly superior to those of humans. Relatively little attention is given to future AI systems that may be "superintelligent" in the sense of  Bostrom [2014] , i.e., systems "much smarter than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills." Our discussion will place a greater focus on methods and frameworks for designing robust and beneficial smarter-than-human AI systems, bracketing questions about whether such systems would have moral standing of their own. Smarter-than-human AI systems are likely to introduce a number of new safety challenges. First, bad behavior by smarter-than-human systems can have larger and more lasting consequences; an antisocial adult is more dangerous than an antisocial child, even if the adult is as physically weak as a child. Whereas low-intelligence systems can be tested and patched over many iterations, Bostrom argues that even small errors in the first superintelligent systems could have extinction-level consequences  [Bostrom, 2014] . The possible development of such systems raises the stakes for AI safety work. Second, systems that can strictly outperform humans cognitively have less to gain from integrating into existing economies and communities.  Hall [2007]  has argued: The economic law of comparative advantage states that cooperation between individuals of differing capabilities remains mutually beneficial.  [ . . . ]  In other words, even if AIs become much more productive than we are, it will remain to their advantage to trade with us and to ours to trade with them. As noted by Benson-Tilsen and Soares [forthcoming 2016], however, rational trade presupposes that agents expect more gains from trade than from coercion. Non-human species have various "comparative advantages" over humans, but humans generally exploit non-humans through force. Similar patterns can be observed in the history of human war and conquest. Whereas agents at similar capability levels have incentives to compromise, collaborate, and trade, agents with strong power advantages over others can have incentives to simply take what they want. The upshot of this is that engineering a functioning society of powerful autonomous AI systems and humans requires that those AI systems be prosocial. The point is an abstract one, but it has important practical consequences: rational agents' interests do not align automatically, particularly when they have very different goals and capabilities. Third, superhumanly creative and adaptive systems may arrive at what  Bostrom [2014, chap. 8 ] calls "perverse instantiations" of their programmed goals.  Wiener [1960]  calls this the "Sorcerer's Apprentice" problem, after the fable of an apprentice whose enchanted broom follows instructions' letter but not their spirit. The novelty here is not that programs can exhibit incorrect or counter-intuitive behavior, but that software agents smart enough to understand natural language may still base their decisions on misrepresentations of their programmers' intent. The idea of superintelligent agents monomaniacally pursuing "dumb"-seeming goals may sound odd, but it follows from the observation of  Bostrom and Yudkowsky [2014, chap. 7 ] that AI capabilities and goals are logically independent. 1 Humans can fully comprehend that their "designer" (evolution) had a particular "goal" (reproduction) in mind for sex, without thereby feeling compelled to forsake contraception. Instilling one's tastes or moral values into an heir isn't impossible, but it also doesn't happen automatically. Lastly,  Bostrom and Yudkowsky [2014]  point out that smarter-than-human systems may become better than humans at moral reasoning. Without a systematic understanding of how perverse instantiations differ from moral progress, how can we distinguish moral genius in highly intelligent machines from moral depravity? Given the potential long-term impact of advanced AI systems, it would be prudent to investigate whether early research progress is possible on any of these fronts. In this paper we give a preliminary, informal survey of several research directions that we think may help address the above four concerns, beginning by arguing for indirect approaches to specifying human values in AI agents. We describe a promising approach to indirect value specification, value learning, and consider still more indirect approaches based on modeling actual and potential states of human operators.
 Valuable Goals Cannot Be Directly Specified We argued above that highly capable autonomous systems could have disastrous effects if their values are misspecified. Still, this leaves open the possibility that specifying correct values is easy, or (more plausibly) that it presents no special difficulties over and above the challenge of building a smarterthan-human AI system. A number of researchers have voiced the intuition that some simple programmed goal would suffice for making superintelligent systems robustly beneficial.  Hibbard [2001] , for example, suggested training a simple learning system to recognize positive human emotions from facial expressions, voice tones, and body language. Hibbard then proposed that machines of much greater capability-perhaps even superintelligent machinescould be programmed to execute actions predicted to lead to futures with as many "positive human emotions" as possible, as evaluated by the original simple learning system. This proposal has some intuitive appeal-wouldn't such a system always act to make humans happy?-until one considers the Sorcerer's Apprentice. We have a particular set of associations in mind when we speak of "positive human emotions," but the simple learner would almost surely have learned a different and simpler concept, such as "surface features correlating with positive human emotions in the training data." This simpler concept almost surely does not have its maximum at a point which Hibbard would consider to contain lots of positive human emotions. The maximum is much more likely to occur in (for example) scenarios that contain an enormous number of tiny human-shaped animatronics acting out positive human emotions. Thus, a powerful learning system that takes actions according to how well the simple learner would rank them is liable to spend time and resources creating animatronics rather than spending time and resources making humans happy. Indeed,  Hibbard [2012]  himself comes to the conclusion that his proposal fails to exclude the possibility that lifelike animatronic replicas of happy people could be counted as exhibiting "positive emotions." As another example, Schmidhuber  [2007]  proposes that creativity, curiosity, and a desire for discovery and beauty can be instilled by creating systems that maximize a different simple measure: "create action sequences that extend the observation history and yield previously unknown / unpredictable but quickly learnable algorithmic regularity or compressibility." However, while it is quite plausible that human creativity and discovery are related to the act of compressing observation, an agent following Schmidhuber's goal would not behave in intuitively curious and creative ways. One simple way to meet Schmidhuber's desideratum, for example, is to appropriate resources and construct artifacts that generate cryptographic secrets, then present the agent with a long and complex series of observations encoded from highly regular data, and then reveal the secret to the agent, thereby allowing the agent to gain enormous compression on its past sensory data. An agent following Schmidhuber's goal is much more likely to build artifacts of this form than it is to pursue anything resembling human creativity. The system may not take this action in particular, but it will take actions that generate at least that much compression of its sensory data, and as a result, the system is unlikely to be prosocial. Building an agent to do something which (in humans) correlates with the desired behavior does not necessarily result in a system that acts like a human. The general lesson we draw from cases like these is that most goals that are simple to specify will not capture all the contextual complexities of real-world human values and objectives  [Yudkowsky, 2011] . Moral psychologists and moral philosophers aren't locked in decades-and centuries-long debates about the right codifications of ethics because they're missing the obvious. Rather, such debates persist for the simple reason that morality is complicated. People want lots of things, in very particular ways, and their desires are context-sensitive. Imagine a simplified state space of possibilities that vary in count (how many happy human-shaped objects exist), in the size of the average happy human-shaped object, and in the average moral worth of happy human-shaped objects. Human experience has occurred in a small region of this space, where almost all human-shaped objects emitting what looks like happiness are ≈ 2-meter-sized humans with moral weight. But the highest scores on the count axis occur in tandem with low size, and the smallest possible systems that can mimic outward signs of emotion are of low moral worth. In linear programming, it is a theorem that the maximum of an objective function occurs on a vertex of the space. (Sometimes the maximum will be on an edge, including its vertices.) For intuitively similar reasons, the optimal solution to a goal tends to occur on a vertex (or edge, or hyperface) of the possibility space. Hibbard's goal does not contain any information about size or moral worth, and so agents pursuing this goal only consider size and moral worth insofar as they pertain to pushing toward the hyperface of maximum count. To quote : A system that is optimizing a function of n variables, where the objective depends on a subset of size k < n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. The Sorcerer's Apprentice problem arises when systems' programmed goals do not contain information about all relevant dimensions along which observations can vary. The agent has been directed towards the wrong hyperface of the possibility space.  2  When confronted with this type of failure, many have an impulse to patch the flawed goals. If Hibbard's system would make smiling animatronics, then find ways to require that the emotions come from actual humans; if the system would then put humans in a drugged stupor in order to make them smile, forbid it from using drugs; and so on. Such constraints cut off particular means by which the system can get a higher count, but they don't address the underlying problem that the system is still maximizing count. If one causal pathway is forbidden, then the system will follow the nearest non-forbidden causal path-e.g., mechanically manipulating the pleasure centers of human brains. It isn't feasible to patch every goal; nor is it safe to patch as many as come to mind and assume that there are no unforeseen perverse instantiations. Intuitively, we would like to direct the intelligence of highly advanced systems to solving some of this problem on our behalf, and we would like such systems to attend to our likely intentions even when our formal and informal representations of our intentions are flawed. The notion of the operator's "intentions," however, is unlikely to lend itself to clean formal specification. By what methods, then, could an intelligent machine be constructed to reliably learn what to value and to act as its operators intended?
 Inductive Value Learning Correctly specifying a formal criterion for recognizing a cat in a video stream by hand is difficult, if not impossible. This does not mean, however, that cat recognition is hopeless; it means that a level of indirection is required. An image recognition system can be constructed and trained to recognize cats. We propose that the value learning problem be approached by similarly indirect means. Inductive value learning via labeled training data raises a number of difficulties. A visual recognition system classifies images; an inductive value learning system classifies outcomes. What are outcomes? What format would a value-learning data set come in? Imagine a highly intelligent system that uses large amounts of data to construct a causal model of its universe. Imagine also that this world-model can be used to reason about the likely outcomes of the agent's available actions, that the system has some method for rating outcomes, and that it executes the action leading to the most highly rated outcome. In order for the system to inductively learn what to value, the system must be designed so that when certain "training" observations are made (or specially-demarcated updates to its world-model occur), labeled training data extracted from the observation or update alters the method by which the system ranks various potential outcomes. This simple model highlights a central concern and two open questions relevant to inductive value learning.
 Corrigibility Imagine that some of an agent's available actions allow it to modify itself, and that it currently assigns high utility to outcomes containing high numbers of animatronic replicas of humans. It may be the case that, according to the system's world-model, all of the following hold: (1) if more training data is received, those high-rated outcomes will have their ratings adjusted downwards; (2) after the ratings are adjusted, the system will achieve outcomes that have fewer cheap animatronics; and (3) there are actions available which remove the inductive value learning framework. In this situation, a sufficiently capable system would favor actions that disable its value learning framework. It would not necessarily consider its own process of learning our values a good thing, any more than humans must approve of psychological disorders they possess. One could try to construct protected sections of code to prevent the value learning framework from being modified, but these constraints would be difficult to trust if the system is more clever than its designers when it comes to exploiting loopholes. A robustly safe initial system would need to be constructed in such a way that actions which remove the value learning framework are poorly rated even if they are available. Some preliminary efforts toward describing a system with this property have been discussed under the name corrigibility by  Soares and Fallenstein [2015] , but no complete proposals currently exist.
 Ontology Identification The representations used in a highly intelligent agent's worldmodel may change over time. A fully trustworthy value learning system would need to not only classify potential outcomes according to their value, but persist in doing so correctly even when its understanding of the space of outcomes undergoes a major change. Consider a programmer that wants to train a system to pursue a very simple goal: produce diamond. The programmers have an atomic model of physics, and they generate training data labeled according to the number of carbon atoms covalently bound to four other carbon atoms in that training outcome. For this training data to be used, the classification algorithm needs to identify the atoms in a potential outcome considered by the system. In this toy example, we can assume that the programmers look at the structure of the initial worldmodel and hard-code a tool for identifying the atoms within. What happens, then, if the system develops a nuclear model of physics, in which the ontology of the universe now contains primitive protons, neutrons, and electrons instead of primitive atoms? The system might fail to identify any carbon atoms in the new world-model, making the system indifferent between all outcomes in the dominant hypothesis. Its actions would then be dominated by any tiny remaining probabilities that it is in a universe where fundamental carbon atoms are hiding somewhere. This is clearly undesirable. Ideally, a scientific learner should be able to infer that nuclei containing six protons are the true carbon atoms, much as humans have done. The difficulty lies in formalizing this process. To design a system that classifies potential outcomes according to how much diamond is in them, some mechanism is needed for identifying the intended ontology of the training data within the potential outcomes as currently modeled by the AI. This is the ontology identification problem introduced by de Blanc  [2011]  and further discussed by  Soares [2015] . This problem is not a traditional focus of machine learning work. When our only concern is that systems form better world-models, then an argument can be made that the nuts and bolts are less important. As long as the system's new world-model better predicts the data than its old world-model, the question of whether diamonds or atoms are "really represented" in either model isn't obviously significant. When the system needs to consistently pursue certain outcomes, however, it matters that the system's internal dynamics preserve (or improve) its representation of which outcomes are desirable, independent of how helpful its representations are for prediction. The problem of making correct choices is not reducible to the problem of making accurate predictions. Inductive value learning requires the construction of an outcome-classifier from value-labeled training data, but it also requires some method for identifying, inside the states or potential states described in its world-model, the referents of the labels in the training data. This could perhaps be done during the course of inductive value learning. The system's methods for inferring a causal world-model from sense data could perhaps be repurposed to infer a description of what has been labeled. If the system adopts a better world-model, it could then re-interpret its training data to re-bind the value labels. This looks like a promising line of research, but it seems to us to require new insights before it is close to being formalizable, let alone usable in practice. In particular, we suspect that ontology identification will require a better understanding of algorithms that construct multi-level world-models from sense data.
 Ambiguity Identification Reinforcement learning can be thought of as a method for sidestepping these difficulties with value learning. Rather than designing systems to learn which outcomes are desirable, one creates a proxy for desirable outcomes: a reward function specified in terms of observations. By controlling rewards via a reward signal, the operator can then judiciously guide the learner toward desired behaviors. Indirect proxies for desired outcomes, however, face many of the same Sorcerer's Apprentice difficulties. Maximizing how often an operator transmits a reward signal is distinct from the problem of maximizing the operator's satisfaction with outcomes; these goals may coincide in testing environments and yet diverge in new environments-e.g., once the learner has an opportunity to manipulate and deceive its operator or otherwise hijack its reward channel  [Bostrom, 2014, chap. 12] . For further discussion, see  Soares [2015] . Superintelligent systems that achieve valuable real-world outcomes may need goals specified in terms of desirable outcomes, rather than rewards specified in terms of observations. If so, then we will need some robust way of ensuring that the system learns our goals, as opposed to superficially similar goals. When training a recognition system, producing satisfactory training data is often a difficult task. There is a classic parable of machine learning (told by, e.g.,  Dreyfus and Dreyfus [1992] ) of an algorithm intended to classify whether or not pictures of woods contained a tank concealed between the trees. Pictures of empty woods were taken one day; pictures with concealed tanks were taken the next. The classifier identified the latter set with great accuracy, and tested extremely well on the portion of the data that had been withheld from training. However, the system performed poorly on new images. It turned out that the first set of pictures had been taken on a sunny day, while the second set had been taken on a cloudy day. The classifier was not identifying tanks; it was identifying image brightness! The same mistake is possible when constructing a training data set for inductive value learning. In value learning, however, such mistakes may be more difficult to notice and more consequential. Consider a training set that successfully represents real-world cases of happy human beings (labeled with high ratings) and real-world cases of pointless human suffering (rated poorly). The simplest generalization from this data may, again, be that human-shaped-things-proclaiminghappiness are of great value, even if these are animatronics imitating happiness. It seems plausible that someone training an inductive value learner could neglect to include a sufficiently wide variety of animatronics mimicking happiness and labeled as low-value. How many other obvious-in-retrospect pitfalls are hiding in our blind spots? A training set covering all relevant dimensions that we can think of may yet exclude relevant dimensions. A robustly safe value learner would need to be able to identify new plausiblyrelevant dimensions along which no training data is provided, and query the operators about these ambiguities. This is the kind of modification that would help in actually solving the value learning problem, as opposed to working around it. At the same time, this is the kind of modification that could take advantage of machines' increased capabilities as the field of AI advances. Formalizing this idea is a key open problem. Given a data set which classifies outcomes in terms of some world-model, how can dimensions along which the data set gives little information be identified? One way to approach the problem is to study how humans learn concepts from sparse data, as discussed by  Tenenbaum et al. [2011]  and Sotala  [2015] . Alternatively, it may be possible to find some compact criterion for identifying ambiguities in a simpler fashion. In both cases, further research could prove fruitful.
 Modeling Intent The problem of ambiguity identification may call for methods beyond the inductive learning of value from training data. An intelligent system with a sufficiently refined model of humans may already have the data needed, provided that the right question is asked, to deduce that humans are more likely to care about whether happy-looking human-shaped things have brains than about the nearby breeze. The trouble would be designing the system to use this information in exactly the right way. Picture a system that builds multi-level environment models from sensory data and learns its values inductively. One could then specially demarcate some part of the model as the "model of the operator," define some explicit rules for extracting a model of the operator's preferences from the model of the operator (in terms of possible outcomes), and adjust the ratings on various outcomes in accordance with the model of the operator's preferences. This would be a system which attempts to learn and follow another agent's intentions, as opposed to learning from labeled training data-a "do what I mean" (DWIM) architecture. The inverse reinforcement learning (IRL) techniques of Ng and  Russell [2000]  can be viewed as a DWIM approach, in which an agent attempts to identify and maximize the reward function of some other agent in the environment. However, existing IRL formalizations do not capture the full problem; the preferences of humans cannot necessarily be captured in terms of observations alone. For example, a system, upon observing its operator lose at a game of chess, should not conclude that its operator wanted to lose at chess, even if the system can clearly see where the operator "decided" to make a bad move instead of a good one. Or imagine a human operator who has a friend that must be put into hiding. The learner may either take the friend to safety, or abandon the friend in a dangerous location and use the resources saved in this way to improve the operator's life. If the system reports that the friend is safe in both cases, and the human operator trusts the system, then the latter observation history may be preferred by the operator. However, the latter outcome would definitely not be preferred by most people if they had complete knowledge of the outcomes. Human preferences are complex, multi-faceted, and often contradictory. Safely extracting preferences from a model of a human would be no easy task. Problems of ontology identification recur here: the framework for extracting preferences and affecting outcome ratings needs to be robust to drastic changes in the learner's model of the operator. The specialcase identification of the "operator model" must survive as the system goes from modeling the operator as a simple reward function to modeling the operator as a fuzzy, ever-changing part of reality built out of biological cells-which are made of atoms, which arise from quantum fields. DWIM architectures must avoid a number of other hazards. Suppose the system learns that its operator model affects its outcome ratings, and the system has available to it actions that affect the operator. Actions which manipulate the operator to make their preferences easier to fulfill may then be highly rated, as they lead to highly-rated outcomes (where the system achieves the operator's now-easy goals). Solving this problem is not so simple as forbidding the system from affecting the operator; any query made by the system to the operator in order to resolve some ambiguity will affect the operator in some way. A DWIM architecture requires significant additional complexity on top of inductive value learning: the agent's goaladjusting learning system no longer simply classifies outcomes; it must also model humans and extract human prefer-ences about human-modeled outcomes, and translate between human-modeled future outcomes and future outcomes as modeled by the system. The hope is that this complexity purchases a system that potentially achieves full and direct coverage of the complexity of human value, without relying on the abilities of the programmers to hand-code exceptions for every edge case or compose exactly the right training set. Further investigations into inverse reinforcement learning or other methods of constructing satisfactory initial operator models may be a good place to start studying the plausibility of DWIM architectures.
 Extrapolating Volition A DWIM architecture may be sufficient when constructing a system that reliably pursues "concrete" goals (such as "cure cancer and then await instruction"), but it may not be sufficient for more complex or sophisticated goals where the operators themselves do not know what they intend-for example, "Do what I would want, if I had more knowledge and more time to think." None of the frameworks discussed so far seem powerful enough to specify philosophical ideas like the "ideal advisor theory" of  Rosati [1995]  or the "reflective equilibrium" of  Rawls [1971] . Here, even "indirect" approaches to making robust and beneficial AI systems run aground of actively debated questions in moral philosophy. One possible approach to resolving normative uncertainty (e.g., about what the operators would want if they were wiser or better people) would be to build a DWIM system that takes a model of a human operator and extrapolates it in the direction of e.g. Rawls' reflective equilibrium. For example, the extrapolation might predict what the operator would decide if they knew everything the system knows, or if they had considered many possible moral arguments  [Bostrom, 2014, chap. 13 ]. However, a high-powered system searching for moral arguments that would put the operators into a reflectively stable state (as a computational expedient to fully simulating the operators' process of reflection) introduces a new set of potential pitfalls. A high-powered search for the most persuasive moral arguments that elicit retrospective approval of moral changes might find arguments that induce psychotic breakdowns or religious conversions. The system should be constrained to search for only "valid" moral arguments, but defining what counts as a valid moral argument is itself a major area of normative uncertainty and disagreement. In this domain, querying for ambiguities is difficult. In everyday practice, an argument that is persuasive to smart and skeptical humans is often valid, but a superintelligent search for persuasive arguments may well discover invalid but extremely persuasive arguments. It is difficult to identify technical approaches to indirect normativity that are tractable today, although there have been a few initial forays.  Christiano [2014]  informally proposes one mechanism by which a system could perhaps safely extrapolate the volition of its operator.  MacAskill [2014]  has given an extensive report on "meta-normativity," touching upon many different philosophical aspects of the difficulties of resolving normative uncertainty. This is an area where further philo-sophical study may make it clearer how to begin approaching the associated long-run engineering problems.
 Discussion Just as human intelligence has allowed us to develop tools and strategies by which we can control our environment, so too could superintelligent systems develop tools and strategies more powerful than our own, and gain correspondingly greater control over future outcomes  [Bostrom, 2014, chap. 6 ]. Although it is not clear how long the development of smarterthan-human systems will take, or what approaches in AI or other disciplines may prove most relevant to developing such systems, early efforts in this area are justified by its importance and neglectedness. In the introduction to this paper, we discussed four different ways in which the potential development of superintelligent machines changes the task of AI safety and ethics work. Addressing all these concerns does not seem easy. Designs for AI systems that are intended to become superintelligent will need to be corrigible in the sense of  Soares and Fallenstein [2015] , i.e., willing to assist their operators in attempted corrections. The systems will need some method for learning and adopting prosocial preferences, in light of the fact that we cannot expect arbitrary rational actors to exhibit prosocial behavior in the face of large power disparities. Operators will require methods for robustly communicating their intentions to the system, if Sorcerer's Apprentice scenarios are to be avoided. And eventually, explicit methodologies for resolving normative uncertainty may be required. This paper has given a cursory overview of a number of potential lines of research for AI value specification. We discuss these ideas in part to give an overview of plausible approaches to the concerns outlined above, and also because these are topics that seem amenable to research starting sooner rather than later, even in the face of great uncertainty about the particular architectures of future AI systems. It is difficult to know which lines of research will pan out, and we hope that this survey inspires research along a number of new paths, so that we have a firm theoretical grasp of how systems could reliably and safely learn our values in principle before it comes time to build systems that must do so in practice. 			  Bostrom's "orthogonality thesis" can be treated as an application ofHume's [1739]  observation that natural-language "is" and "ought" claims are independent.
			 Instead of trying to direct the system toward exactly the right hyperface, one might try to create a "limited optimization" system that doesn't push so hard in whatever direction it moves. This seems like a promising research avenue, but is beyond the scope of this paper.