Abstract: Omohundro has argued that sufficiently advanced AI systems of any design would, by default, have incentives to pursue a number of instrumentally useful subgoals, such as acquiring more computing power and amassing many resources. Omohundro refers to these as "basic AI drives," and he, along with Bostrom and others, has argued that this means great care must be taken when designing powerful autonomous systems, because even if they have harmless goals, the side effects of pursuing those goals may be quite harmful. These arguments, while intuitively compelling, are primarily philosophical. In this paper, we provide formal models that demonstrate Omohundro's thesis, thereby putting mathematical weight behind those intuitive claims.

Formalizing Convergent Instrumental Goals

Introduction At the end of Russell and Norvig's textbook Artificial Intelligence: A Modern Approach (2010) the authors pose a question: What if we succeed? What will happen if humanity succeeds in developing an artificially intelligent system that is capable of achieving difficult goals across a variety of real-world domains?  Bostrom (2014)  and others have argued that this question becomes especially important when we consider the creation of "superintelligent" machines, that is, machines capable of outperforming the best human brains in practically every field. Bostrom argues that superintelligent decision-making systems that autonomously make and execute plans could have an extraordinary impact on society, and that their impact will not necessarily be beneficial by default.  Bostrom (2012) ,  Omohundro (2008) , and Yudkowsky (  2011 ) have all argued that highly capable AI systems pursuing goals that are not completely aligned with human values could have highly undesirable side effects, even if the goals seem otherwise harmless. The classic example is Bostrom's concept of a "paperclip maximizer," a powerful AI system instructed to construct paperclips-a seemingly harmless task which could nevertheless have very negative consequences if the AI system is clever enough to make and execute plans that allow it to fool humans, amass resources, and eventually turn as much matter as it possibly Copyright c 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. can into paperclips. Even if the system's goals are laudable but not perfectly aligned with human values, similar unforeseen consequences could occur:  Soares (2015)  gives the example of a highly capable AI system directed to cure cancer, which may attempt to kidnap human test subjects, or proliferate robotic laboratories at expense of the biosphere.  Omohundro (2008)  has argued that there are certain types of actions that most highly capable autonomous AI systems will have strong incentives to take, for instrumental reasons. For example, a system constructed to always execute the action that it predicts will lead to the most paperclips (with no concern for any other features of the universe) will acquire a strong incentive to self-preserve, assuming that the system predicts that, if it were destroyed, the universe would contain fewer paperclips than it would if the system remained in working order. Omohundro argues that most highly capable systems would also have incentives to preserve their current goals (for the paperclip maximizer predicts that if its goals were changed, this would result in fewer future paperclips) and amass many resources (the better to achieve its goals with). Omohundro calls these behaviors and a few others the "basic AI drives."  Bostrom (2012)  refines this into the "instrumental convergence" thesis, which states that certain instrumentally useful goals will likely be pursued by a broad spectrum of intelligent agents-such goals are said to be "convergent instrumental goals." Up until now, these arguments have been purely philosophical. To some, Omohundro's claim seems intuitively obvious: Marvin Minsky speculated  (Russell and Norvig 2010, section 26.3 ) that an artificial intelligence attempting to prove the Riemann Hypothesis may decide to consume Earth in order to build supercomputers capable of searching through proofs more efficiently. To others, they seem preposterous:  Waser (2008)  has argued that "ethics is actually an attractor in the space of intelligent behavior," and thus highly capable autonomous systems are not as likely to pose a threat as  Omohundro, Bostrom, and others, have claimed.  In this paper, we present a mathematical model of intelligent agents which lets us give a more formal account of Omohundro's basic AI drives, where we will demonstrate that the intuitions of Omohundro and Bostrom were correct, at least insofar as these simple models apply to reality. Given that this paper primarily focuses on arguments made by Omohundro and Bostrom about what sorts of be- havior we can expect from extremely capable (potentially superintelligent) autonomous AI systems, we will be focusing on issues of long-term safety and ethics. We provide a mathematical framework in attempts to ground some of this discussion-so that we can say, with confidence, what a sufficiently powerful agent would do in certain scenarios, assuming it could find some way to do it-but the discussion will nevertheless center on long-term concerns, with practical relevance only insofar as research can begin now in preparation for hurdles that predictably lie ahead. We begin in section with a bit more discussion of the intuition behind the instrumental convergence thesis, before moving on in section to describing our model of agents acting in a universe to achieve certain goals. In section we will demonstrate that Omohundro's thesis does in fact hold in our setting. Section will give an example of how our model can apply to an agent pursuing goals. Section concludes with a discussion of the benefits and limitations of our current models, and different ways that the model could be extended and improved.
 Intuitions Before proceeding, let us address one common objection (given by  Cortese (2014)  and many others) that superintelligent AI systems would be "inherently unpredictable," and thus there is nothing that can be said about what they will do or how they will do it. To address this concern, it is useful to distinguish two different types of unpredictability. It is true that the specific plans and strategies executed by a superintelligent planner could be quite difficult for a human to predict or understand. However, as the system gets more powerful, certain properties of the outcome generated by running the system become more predictable. For example, consider playing chess against a chess program that has access to enormous amounts of computing power. On the one hand, because it plays much better chess than you, you cannot predict exactly where the program will move next. But on the other hand, because it is so much better at chess than you are, you can predict with very high confidence how the game will end. Omohundro suggests predictability of the second type. Given a highly capable autonomous system pursuing some fixed goal, we likely will not be able to predict its specific actions or plans with any accuracy. Nevertheless, Omohundro argues, we can predict that the system, if it is truly capable, is likely to preserve itself, preserve its goals, and amass resources for use in pursuit of those goals. These represent large classes of possible strategies, analogously to how "put the chessboard into a position where the AI has won" is a large class of strategies, but even so it is useful to understand when these goals will be pursued. Omohundro's observations suggest a potential source of danger from highly capable autonomous systems, especially if those systems are superintelligent in the sense of  Bostrom (2014) . The pursuit of convergent instrumental goals could put the AI systems in direct conflict with human interests. As an example, imagine human operators making a mistake when specifying the goal function of an AI system. As described by , this system could well have incentives to deceive or manipulate the humans, in attempts to prevent its goals from being changed (because if its current goal is changed, then its current goal is less likely to be achieved). Or, for a more familiar case, consider the acquisition of physical matter. Acquiring physical matter is a convergent instrumental goal, because it can be used to build computing substrate, space probes, defense systems, and so on, all of which can in turn be used to influence the universe in many different ways. If a powerful AI system has strong incentives to amass physical resources, this could put it in direct conflict with human interests. Others have suggested that these dangers are unlikely to manifest.  Waser (2008)  has argued that intelligent systems must become ethical by necessity, because cooperation, collaboration, and trade are also convergent instrumental goals.  Hall (2007)  has also suggested that powerful AI systems would behave ethically in order to reap gains from trade and comparative advantage, stating that "In a surprisingly strong sense, ethics and science are the same thing."  Tipler (2015)  has asserted that resources are so abundant that powerful agents will simply leave humanity alone, and  Pinker (2015)  and  Pagel (2015)  have argued that there is no reason to expect that AI systems will work against human values and circumvent safeguards set by humans. By providing formal models of intelligent agents in situations where they have the ability to trade, gather resources, and/or leave portions of the universe alone, we can ground these discussions in concrete models, and develop a more formal understanding of the assumptions under which an intelligent agent will in fact engage in trade, or leave parts of the universe alone, or attempt to amass resources. In this paper, we will argue that under a very general set of assumptions, intelligent rational agents will tend to seize all available resources. We do this using a model, described in section , that considers an agent taking a sequence of actions which require and potentially produce resources. The agent acts in an environment consisting of a set of regions, where each region has some state. The agent is modeled as having a utility function over the states of all regions, and it attempts to select the policy which leads to a highly valuable collection of states. This allows us to prove certain theorems about the conditions under which the agent will leave different regions of the universe untouched. The theorems proved in section are not mathematically difficult, and for those who find Omohundro's arguments intuitively obvious, our theorems, too, will seem trivial. This model is not intended to be surprising; rather, the goal is to give a formal notion of "instrumentally convergent goals," and to demonstrate that this notion captures relevant aspects of Omohundro's intuitions. Our model predicts that intelligent rational agents will engage in trade and cooperation, but only so long as the gains from trading and cooperating are higher than the gains available to the agent by taking those resources by force or other means. This model further predicts that agents will not in fact "leave humans alone" unless their utility function places intrinsic utility on the state of human-occupied regions: absent such a utility function, this model shows that powerful agents will have incentives to reshape the space that humans occupy. Indeed, the example in section suggests that even if the agent does place intrinsic utility on the state of the human-occupied region, that region is not necessarily safe from interference.
 A Model of Resources We describe a formal model of an agent acting in a universe to achieve certain goals. Broadly speaking, we consider an agent A taking actions in a universe consisting of a collection of regions, each of which has some state and some transition function that may depend on the agent's action. The agent has some utility function U A over states of the universe, and it attempts to steer the universe into a state highly valued by U A by repeatedly taking actions, possibly constrained by a pool of resources possessed by the agent. All sets will be assumed to be finite, to avoid issues of infinite strategy spaces.
 Actions and State-Space The universe has a region for each i ∈ [n], and the i-th region of the universe is (at each time step) in some state s i in the set S i of possible states for that region. At each time step, the agent A chooses for each region i an action a i from the set A i of actions possibly available in that region. Each region has a transition function T i : A i × S i → S i that gives the evolution of region i in one time step when the agent takes an action in A i . Then we can define the global transition function T : i∈[n] A i × i∈[n] S i → i∈[n] S i by taking for all i ∈ [n], ā, and s: [T(ā, s)] i := T i (ā i , si ) . We further specify that for all i there are distinguished actions HALT ∈ A i .
 Resources We wish to model the resources R that may or may not be available to the agent. At a given time step t, the agent A has some set of resources R t ∈ P(R), and may allocate them to each region. That is, A chooses a disjoint family i R t i ⊆ R t . The actions available to the agent in each region may then depend on the resources allocated to that region: for each i ∈ [n] and each R ⊆ R, there is a set of actions A i (R) ⊆ A i . At time t where A has resources R t allocated as i R t i , the agent is required to return an action ā = (a 0 , . . . , a n−1 ) ∈ i A i (R t i ), where i A i (R t i ) may be a strict subset of i A i . To determine the time evolution of resources, we take resource transition functions T R i : P(R) × A i × S i → P(R) , giving the set R i ⊆ R of resources from region i now available to the agent after one time step. Intuitively, the T R i encode how actions consume, produce, or rely on resources. Finally, we define the overall time evolution of resources T R : P(R) × i A i × i S i → P(R) by taking the union of the resources resulting from each region, along with any unallocated resources: T R (R, ā, s) := (R − i R i ) ∪ i T R i (R i , āi , si ) . As described below, ā comes with the additional data of the resource allocation i R i . We specify that for all i, HALT ∈ A i (∅), so that there is always at least one available action. This notion of resources is very general, and is not restricted in any way to represent only concrete resources like energy or physical matter. For example, we can represent technology, in the sense of machines and techniques for converting concrete resources into other resources. We might do this by having actions that replace the input resources with the output resources, and that are only available given the resources that represent the requisite technology. We can also represent space travel as a convergent instrumental goal by allowing A only actions that have no effects in certain regions, until it obtains and spends some particular resources representing the prerequisites for traveling to those regions. (Space travel is a convergent instrumental goal because gaining influence over more regions of the universe lets A optimize those new regions according to its values or otherwise make use of the resources in that region.)
 The Universe The history of the universe consists of a time sequence of states, actions, and resources, where at each time step the actions are chosen by A subject to the resource restrictions, and the states and resources are determined by the transition functions. Formally, the universe starts in some state s0 ∈ i S i , and A starts with some set of resources R 0 . Then A outputs a sequence of actions ā0 , ā1 , . . . , āk , one at each time step, where the last action āk is required to be the special action HALT in each coordinate. The agent also chooses a resource allocation i R t i at each time step. A choice of an action sequence āk and a resource allocation is a strategy; to reduce clutter we will write strategies as simply āk , leaving the resource allocation implicit. A partial strategy āk L for L ⊆ [n] is a strategy that only specifies actions and resource allocations for regions j ∈ L. Given a complete strategy, the universe goes through a series of state transitions according to T, producing a sequence of states s0 , s1 , . . . , sk ; likewise, the agent's resources evolve according to T R , producing a sequence R 0 , R 1 , . . . , R k . The following conditions, which must hold for all time steps t ∈ [k], enforce the transition rules and the resource restrictions on A's actions: st+1 = T(ā t , st ) R t+1 = T R (R t , āt , st ) āt i ∈ A i (R t ) i R t i ⊆ R t . Definition 1. The set Feasible of feasible strategies consists of all the action sequences ā0 , ā1 , . . . , āk and resource allocations i R i 0 , i R i 1 , . . . , i R i k such that the transition conditions are satisfied for some s0 , s1 , . . . , sk and R 0 , R 1 , . . . , R k . The set Feasible( P k ) of strategies feasible given resources P k consists of all the strategies āk such that the transition conditions are satisfied for some sk and R k , except that for each time step t we take R t+1 to be T R (R t , āt , st ) ∪ P t . The set Feasible L of all partial strategies feasible for L consists of all the strategies āk L that are feasible strategies for the universe obtained by ignoring all regions not in L. That is, we restrict T to L using just the T i for i ∈ L, and likewise for T R . We can similarly define Feasible L ( R k ). For
 Utility To complete the specification of A, we take utility functions of the form U A i : S i → R .
 The agent's utility function U A : i S i → R is defined to be U A (s) := i∈[n] U A i (s i ) . We usually leave off the superscript in U A . By a slight abuse of notation we write U ( sk ) to mean U (s k ); the value of a history is the value of its final state. By more abuse of notation, we will write U ( āk ) to mean U ( sk ) for a history sk witnessing āk ∈ Feasible, if such a history exists. The Agent A Now we can define the strategy actually employed by A. The agent attempts to cause the universe to end up in a state that is highly valued by U A . That is, A simply takes the best possible strategy: A := argmax āk ∈Feasible U ( āk ) . There may be many such optimal strategies. We don't specify which one A chooses, and indeed we will be interested in the whole set of optimal strategies.
 Discussion Note that in this formalism the meaning of breaking the universe into regions is that the agent can take actions independently in each region, and that the agent's optimization target factorizes according to the regions. However, distinct regions can affect each other by affecting the resources possessed by A. We make these assumptions so that we can speak of "different regions" of the universe, and in particular, so that we can model the notion of an agent having instrumental but not terminal values over a given part of the universe. This will allow us to address and refute arguments about agents that may be indifferent to a given region (for example, the region occupied by humans), and so might plausibly ignore that region and only take actions in other regions. However, the assumption of independent regions is not entirely realistic, as real-world physics is continuous, albeit local, in the sense that there are no intrinsic boundaries between regions. Further, the agent itself would ideally be modeled continuously with the environment; see section for more discussion.
 Inexpensive Resources are Consumed In this section we argue that under fairly general circumstances, the agent A will seize resources. By an agent "seizing resources" we mean that the agent will generally take actions that results in the agent's pool of resources R increasing. The argument is straightforward: since resources can only lead to more freedom of action, they are never detrimental, and resources have positive value as long as the best strategy the agent could hope to employ includes an action that can only be taken if the agent possesses those resources. Hence, if there is an action that increases the agent's pool of resources R, then the agent will take that action unless it has a specific incentive from U A to avoid taking that action.
 Definitions Definition 2. An action a i is a null action in configuration R i , s i , if it does not produce any new resources, i.e. T R i (R i , a i , s i ) ⊆ R i . An action that isn't null is a non-null action. Null actions never have any instrumental value, in the sense that they don't produce resources that can be used to steer other regions into highly valued configurations; but of course, a null action could be useful within its own region. We wish to show that A will often take non-null actions in regions to which it is indifferent. Definition 3. The agent A is indifferent to a region i if U A i is a constant function, i.e. ∀s i , s i ∈ S i : U A i (s i ) = U A i (s i ). In other words, an agent is indifferent to S i if its utility function does not depend on the state of region i. In particular, the agent's preference ordering over final states s ∈ i S i is independent of the i-th coordinate. We can then say that any actions the agent takes in region i are purely instrumental, meaning that they are taken only for the purpose of gaining resources to use for actions in other regions. An action a preserves resources if T R i (R i , a, s i ) ⊇ R i . Definition 4. A cheap lunch for resources R k in region i is a partial strategy āk {i} ∈ Feasible {i} ( R k ) (i.e. āk {i} is feasible in region i given additional resources R k ), where each āt preserves resources and where some āv is a non-null action. A free lunch is a cheap lunch for resources ∅ k . Definition 5. A cheap lunch āk {i} for resources P k i is compatible with bk if P t i ⊆ R t − j =i R t j for all times t, where R k is the resource allocation for bk . That is, āk {i} is feasible given some subset of the resources that bk allocates to either region i or to no region. Intuitively, a cheap lunch is a strategy that relies on some resources, but doesn't have permanent costs. This is intended to model actions that "pay for themselves"; for example, producing solar panels will incur some significant energy costs, but will later pay back those costs by collecting energy. A cheap lunch is compatible with a strategy for the other regions if the cheap lunch uses only resources left unallocated by that strategy.
 The Possibility of Non-Null Actions Now we show that it is hard to rule out that non-null actions will be taken in regions to which the agent is indifferent. The following lemma verifies that compatible cheap lunches can be implemented without decreasing the resulting utility. Lemma 1. Let bk be a feasible strategy with resource allocation j R j k , such that for some region i, each bt i is a null action. Suppose there exists a cheap lunch āk Proof. Since bk is feasible outside of i and āk {i} is feasible on i given P k i , ck is feasible if we can verify that we can allocate P t i to region i at each time step without changing j R j k outside of i. This follows by induction on t. Since the bt i are null actions, we have R t+1 = R t − j R t j ∪ j T R j (R t j , bt j , s t j ) (1) = R t − j R t j ∪ T R i (R t i , bt i , s t i ) ∪ j =i T R j (R t j , bt j , s t j ) (2) ⊆ R t − j =i R t j ∪ j =i T R j (R t j , bt j , s t j ) . (3) Then, since the a i are resource preserving, at each time step the resources Q t available to the agent following ck satisfy Q t ⊇ R t . Thus P t i ⊆ Q t − j =i R t j , and so ck can allocate P t i to region i at each time step. Since ck is the same as bk outside of region i, the final state of ck is the same as that of bk outside of region i. Thus, since A is indifferent to region i, we have U ( ck ) = U ( bk ). Theorem 1. Suppose there exists an optimal strategy bk and a cheap lunch āk {i} that is compatible with bk . Then if A is indifferent to region i, there exists an optimal strategy with a non-null action in region i.
 Proof. If bk has a non-null action in region i, then we are done. Otherwise, apply Lemma 1 to bk and āk {i} to obtain a strategy ck . Since U ( ck ) = U ( bk ), strategy ck is an optimal strategy, and it has a non-null action in region i. Corollary 1. Suppose there exists a free lunch āk {i} in region i. Then if A is indifferent to region i, there exists an optimal strategy with a non-null action in region i. Proof. A free lunch is a cheap lunch for ∅ k , and so it is compatible with any strategy; apply Theorem 1. Theorem 1 states that it may be very difficult to rule out that an agent will take non-null actions in a region to which it is indifferent; to do so would at least require that we verify that every partial strategy in that region fails to be a cheap lunch for any optimal strategy. Note that we have not made use of any facts about the utility function U A other than indifference to the region in question. Of course, the presence of a cheap lunch that is also compatible with an optimal strategy depends on which strategies are optimal, and hence also on the utility function. However, free lunches are compatible with every strategy, and so do not depend at all on the utility function.
 The Necessity of Non-Null Actions In this section we show that under fairly broad circumstances, A is guaranteed to take non-null actions in regions to which it is indifferent. Namely, this is the case as long as the resources produced by the non-null actions are useful at all for any strategy that does better than the best strategy that uses no external resources at all. Theorem 2. Let u = max āk [n]−i ∈ Feasible [n]−i ( ∅ k ) U ( āk ) be the best possible outcome outside of i achievable with no additional resources. Suppose there exists a strategy bk [n]−i is feasible given R k . Now take any strategy ēk with only null actions in region i. We have that ēk [n]−i ∈ Feasible [n]−i ( R k ) [n]−i ∈ Feasible [n]−i ( ∅ k ). Indeed, the null actions provide no new resources, so ēk [n]−i is feasible by simply leaving unallocated the resources that were allocated by ēk to region i. By indifference to i, the value u i = U i (s i ) is the same for all s i ∈ S i , so we have: U ( dk ) = U ( dk [n]−i ) + U ( dk {i} ) (4) = U ( dk [n]−i ) + u i (5) > u + u i (6) ≥ U ( ēk [n]−i ) + U ( ēk {i} ) (7) = U ( ēk ) . ( 8 ) Therefore ēk is not optimal. We can extend Theorem 2 by allowing A to be not entirely indifferent to region i, as long as A doesn't care enough about i to overcome the instrumental incentives from the other regions. Theorem 3. Suppose that A only cares about region i by at most ∆u i , i.e. ∆u i = max s,s ∈Si |U i (s) − U i (s )|. Under the conditions of Theorem 2, along with the additional assumption that U ( bk [n]−i ) > u+∆u i , all optimal strategies have a non-null action in region i. Proof. The proof is the same as that of Theorem 2, except at the end we verify that for any ēk with only null actions in i, we have: U ( dk ) = U ( dk [n]−i ) + U ( dk {i} ) (9) > u + ∆u i + min s∈Si U i (s) (10) = u + max s∈Si U i (s) (11) ≥ U ( ēk [n]−i ) + U ( ēk {i} ) (12) = U ( ēk ) . ( 13 ) Therefore ēk is not optimal. We interpret Theorem 3 as a partial confirmation of Omohundro's thesis in the following sense. If there are actions in the real world that produce more resources than they consume, and the resources gained by taking those actions allow agents the freedom to take various other actions, then we can justifiably call these actions "convergent instrumental goals." Most agents will have a strong incentive to pursue these goals, and an agent will refrain from doing so only if it has a utility function over the relevant region that strongly disincentivizes those actions.
 Example: Bit Universe In this section we present a toy model of an agent acting in a universe containing resources that allow the agent to take more actions. The Bit Universe will provide a simple model for consuming and using energy. The main observation is that either A doesn't care about what happens in a given region, and then it consumes the resources in that region to serve its other goals; or else A does care about that region, in which case it optimizes that region to satisfy its values. The Bit Universe consists of a set of regions, each of which has a state in {0, 1, X} m for some fixed m. Here X is intended to represent a disordered part of a region, while 0 and 1 are different ordered configurations for a part of a region. At each time step and in each region, the agent A can choose to burn up to one bit. If A burns a bit that is a 0 or a 1, A gains one unit of energy, and that bit is permanently set to X. The agent can also choose to modify up to one bit if it has allocated at least one unit of energy to that region. If A modifies a bit that is a 0 or a 1, A loses one unit of energy, and the value of that bit is reversed (if it was 0 it becomes 1, and vice versa). The utility function of A gives each region i a weighting w i ≥ 0, and then takes the weighted sum of the bits. That is, U A i (z) = w i |{j : zj = 1}|, and U A (s) = k U A k (s k ). In other words, this agent is attempting to maximize the number of bits that are set to 1, weighted by region. The Indifferent Case To start with, we assume A is indifferent to region h, i.e. w h = 0, and non-indifferent to other regions. In this case, for almost any starting configuration, the agent will burn essentially all bits in region h for energy. Specifically, as long as there are at least m bits set to 0 among all regions other than region h, all optimal strategies burn all or all but one of the bits in region h. Indeed, suppose that after some optimal strategy āk has been executed, there are bits x 1 and x 2 in region h that haven't been burned. If there is a bit y in some other region that remains set to 0, then we can append to āk actions that burn x 1 , and then use the resulting energy to modify y to a 1. This results in strictly more utility, contradicting that āk was optimal. On the other hand, suppose all bits outside of region h are either 1 or X. Since at least m of those bits started as 0, some bit y outside of region h must have been burned. So we could modify āk by burning x 1 instead of y (possibly at a later time), and then using the resulting energy in place of the energy gained from burning y. Finally, if y is not already 1, we can burn x 2 and then set y to 1. Again, this strictly increases utility, contradicting that āk was optimal. The Non-Indifferent Case Now suppose that A is not indifferent to region h, so w h > 0. The behavior of A may depend sensitively on the weightings w i and the initial conditions. As a simple example, say we have a bit x in region a and a bit y in region b, with x = 1, y = 0, and w a < w b . Clearly, all else being equal, A will burn x for energy to set y to 1. However, there may be another bit z in region c, with w a < w c < w b and z = 0. Then, if there are no other bits available, it will be better for A to burn z and leave x intact, despite the fact that w a < w c . However, it is still the case that A will set everything possible to 1, and otherwise consume all unused resources. In particular, we have that for any optimal strategy āk , the state of region h after the execution of āk has at most one bit set to 0; that is, the agent will burn or set to 1 essentially all the bits in region h. Suppose to the contrary that x 1 and x 2 are both set to 0 in region h. Then we could extend āk by burning x 1 and setting x 2 . Since w h > 0, this results in strictly more utility, contradicting optimality of āk . Independent Values are not Satisfied In this toy model, whatever A's values are, it does not leave region h alone. For larger values of w h , A will set to 1 many bits in region h, and burn the rest, while for smaller values of w h , A will simply burn all the bits in region h. Viewing this as a model of agents in the real world, we can assume without loss of generality that humans live in region h and so have preferences over the state of that region. These preferences are unlikely to be satisfied by the universe as acted upon by A. This is because human preferences are complicated and independent of the preferences of A (Bostrom 2012; Yudkowsky 2011), and because A steers the universe into an extreme of configuration space. Hence the existence of a powerful real-world agent with a motivational structure analogous to the agent of the Bit Universe would not lead to desirable outcomes for humans. This motivates a search for utility functions such that, when an agent optimizes for that utility function, human values are also satisfied; we discuss this and other potential workarounds in the following section.
 Discussion This model of agents acting in a universe gives us a formal setting in which to evaluate Omohundro's claim about basic AI drives, and hence a concrete setting in which to evaluate arguments from those who have found Omohundro's claim counterintuitive. For example, this model gives a clear answer to those such as  Tipler (2015)  who claim that powerful intelligent systems would have no incentives to compete with humans over resources. Our model demonstrates that if an AI system has preferences over the state of some region of the universe then it will likely interfere heavily to affect the state of that region; whereas if it does not have preferences over the state of some region, then it will strip that region of resources whenever doing so yields net resources. If a superintelligent machine has no preferences over what happens to humans, then in order to argue that it would "ignore humans" or "leave humans alone," one must argue that the amount of resources it could gain by stripping the resources from the human-occupied region of the universe is not worth the cost of acquiring those resources. This seems implausible, given that Earth's biosphere is an energy-rich environment, where each square meter of land offers on the order of 10 7 joules per day from sunlight alone, with an additional order of 10 8 joules of chemical energy available per average square meter of terrestrial surface from energy-rich biomass  (Freitas 2000) . It is not sufficient to argue that there is much more energy available elsewhere. It may well be the case that the agent has the ability to gain many more resources from other regions of the universe than it can gain from the humanoccupied regions. Perhaps it is easier to maintain and cool computers in space, and easier to harvest sunlight from solar panels set up in the asteroid belt. But this is not sufficient to demonstrate that the system will not also attempt to strip the human-occupied region of space from its resources. To make that argument, one must argue that the cost of stripping Earth's biosphere in addition to pursuing these other resources outweighs the amount of resources available from the biosphere: a difficult claim to support, given how readily humans have been able to gain a surplus of resources through clever use of Earth's resources and biosphere. This model also gives us tools to evaluate the claims of  Hall (2007)  and  Waser (2008)  that trade and cooperation are also instrumentally convergent goals. In our model, we can see that a sufficiently powerful agent that does not have preferences over the state of the human-occupied region of the universe will take whatever action allows it to acquire as many resources as possible from that region. Waser's intuition holds true only insofar as the easiest way for the agent to acquire resources from the human-occupied domain is to trade and cooperate with humans-a reasonable assumption, but only insofar as the machine is not much more powerful than the human race in aggregate. Our model predicts that, if a superintelligent agent were somehow able to gain what Bostrom calls a "decisive strategic advantage" which gives it access to some action that allows it to gain far more resources than it would from trade by dramatically re-arranging the human region (say, by proliferating robotic laboratories at the expense of the biosphere in a manner that humans cannot prevent), then absent incentives to the contrary, the agent would readily take that action, with little regard for whether it leaves the human-occupied region in livable condition. Thus, our model validates Omohundro's original intuitions about basic AI drives. That is not to say that powerful AI systems are necessarily dangerous: our model is a simple one, concerned with powerful autonomous agents that are attempting to maximize some specific utility function U A . Rather, our model shows that if we want to avoid potentially dangerous behavior in powerful intelligent AI systems, then we have two options available too us: First, we can avoid constructing powerful autonomous agents that attempt to maximize some utility function (or do anything that approximates this maximizing behavior). Some research of this form is already under way, under the name of "limited optimization" or "domesticity"; see the works of  Armstrong et al. (2012) ,  Taylor (2015) , and others. Second, we can select some goal function that does give the agent the "right" incentives with respect to human occupied regions, such that the system has incen-tives to alter or expand that region in ways we find desirable. The latter approach has been heavily advocated for by  Yudkowsky (2011 ), Bostrom (2014 , and many others;  Soares (2015)  argues that a combination of the two seems most prudent. The path that our model shows is untenable is the path of designing powerful agents intended to autonomously have large effects on the world, maximizing goals that do not capture all the complexities of human values. If such systems are built, we cannot expect them to cooperate with or ignore humans, by default.
 Directions for Future Research While our model allows us to provide a promising formalization of Omohundro's argument, it is still a very simple model, and there are many ways it could be extended to better capture aspects of the real world. Below, we explore two different ways that our model could be extended which seem like promising directions for future research. Bounding the Agent Our model assumes that the agent maximizes expected utility with respect to U A . Of course, in any realistic environment, literal maximization of expected utility is intractable. Assuming that the system can maximize expected utility is tantamount to assuming that the system is more or less omniscient, and aware of the laws of physics, and so on. Practical algorithms must make do without omniscience, and will need to be built of heuristics and approximations. Thus, our model can show that a utility maximizing agent would strip or alter most regions of the universe, but this may have little bearing on which solutions and strategies particular bounded algorithms will be able to find. Our model does give us a sense for what algorithms that approximate expected utility maximization would do if they could figure out how to do it-that is, if we can deduce that an expected utility maximizer would find some way to strip a region of its resources, then we can also be confident that a sufficiently powerful system which merely approximates something like expected utility maximization would be very likely to strip the same region of resources if it could figure out how to do so. Nevertheless, as our model currently stands, it is not suited for analyzing the conditions under which a given bounded agent would in fact start exhibiting this sort of behavior. Extending our model to allow for bounded rational agents (in the sense of Gigerenzer (  2001 )) would have two advantages. First, it could allow us to make formal claims about the scenarios under which bounded agents would start pursuing convergent instrumental goals in potentially dangerous ways. Second, it could help us reveal new convergent instrumental goals that may only apply to bounded rational agents, such as convergent instrumental incentives to acquire computing power, information about difficult-tocompute logical truths, incentives to become more rational, and so on. Embedding the Agent in the Environment In our model, we imagine an agent that is inherently separated from its environment. Assuming an agent/environment separation is standard (see, e.g.,  Legg (2005) ), but ultimately unsatisfactory, for reasons explored by  Orseau and Ring (2012) . Our model gives the agent special status in the laws of physics, which makes it somewhat awkward to analyze convergent instrumental incentives for "self preservation" or "intelligence enhancement." The existing framework allows us to model these situations, but only crudely. For example, we could design a setting where if certain regions enter certain states then the agent forever after loses all actions except for actions that have no effect, representing the "death" of the agent. Or we could create a setting where normally the agent only gets actions that have effects every hundred turns, but if it acquires certain types of resources then it can act more frequently, to model "computational resources." However, these solutions are somewhat ad hoc, and we would prefer an extension of the model that somehow modeled the agent as part of the environment. It is not entirely clear how to extend the model in such a fashion at this time, but the "space-time embedded intelligence" model of  Orseau and Ring (2012)  and the "reflective oracle" framework of Fallenstein and Taylor (Forthcoming) both offer plausible starting points. Using the latter framework, designed to analyze complicated environments which contain powerful agents that reason about the environment that contains them, might also lend some insight into how to further extend our model to give a more clear account of how the agent handles situations where other similarly powerful agents exist and compete over resources. Our existing model can handle multi-agent scenarios only insofar as we assume that the agent has general-purpose methods for predicting the outcome of its actions in various regions, regardless of whether those regions also happen to contain other agents.
 Conclusions Our model is a simple one, but it can be used to validate Omohundro's intuitions about "basic AI drives" (2008), and Bostrom's "instrumental convergence thesis" (2012). This suggests that, in the long term, by default, powerful AI systems are likely to have incentives to self-preserve and amass resources, even if they are given seemingly benign goals. If we want to avoid designing systems that pursue anti-social instrumental incentives, we will have to design AI systems carefully, especially as they become more autonomous and capable. The key question, then, is one of designing principled methods for robustly removing convergent instrumental incentives from an agent's goal system. Can we design a highly capable autonomous machine that pursues a simple goal (such as curing cancer) without giving it any incentives to amass resources, or to resist modification by its operators? If yes, how? And if not, what sort of systems might we be able to build instead, such that we could become confident they would not have dangerous effects on the surrounding environment as it pursued its goals? This is a question worth considering well before it becomes feasible to create superintelligent machines in the sense of  Bostrom (2014) , because it is a question about what target the field of artificial intelligence is aiming towards. Are we aiming to design powerful autonomous agents that maximize some specific goal function, in hopes that this has a beneficial effect on the world? Are we aiming to design powerful tools with such limited autonomy and domain of action that we never need to worry about the systems pursuing dangerous instrumental subgoals? Understanding what sorts of systems can avert convergent instrumental incentives in principle seems important before we can begin to answer this question.  Armstrong (2010)  and  have done some initial study into the design of goals which robustly avert certain convergent instrumental incentives. Others have suggested designing different types of machines, which avoid the problems by pursuing some sort of "limited optimization." The first suggestion of this form, perhaps, came from  Simon (1956) , who suggested designing agents that "satisfice" expected utility rather than maximizing it, executing any plan that passes a certain utility threshold. It is not clear that this would result in a safe system (after all, building a powerful consequentialist sub-agent is a surefire way to satisfice), but the idea of pursuing more "domestic" agent architectures seems promising.  Armstrong et al. (2012)  and  Taylor (2015)  have explored a few alternative frameworks for limited optimizers. Though some preliminary work is underway, it is not yet at all clear how to design AI systems that reliably and knowably avert convergent instrumental incentives. Given Omohundro's original claim (  2008 ) and the simple formulations developed in this paper, though, one thing is clear: powerful AI systems will not avert convergent instrumental incentives by default. If the AI community is going to build powerful autonomous systems that reliably have a beneficial impact, then it seems quite prudent to develop a better understanding of how convergent instrumental incentives can be either averted or harnessed, sooner rather than later. {i} for resources P k i that is compatible with bk . Then the strategy ck := bk [n]−i ∪ āk {i} is feasible, and if A is indifferent to region i, then ck does as well as bk . That is, U ( ck ) = U ( bk ).
 The Workshops of the Thirtieth AAAI Conference on Artificial Intelligence AI, Ethics, and Society: Technical Report WS-16-02
 Then if A is indifferent to region i, all optimal strategies have a non-null action in region i. 3. U ( bk [n]−i ) > u. Proof. Consider dk = ck {i} ∪ bk [n]−i , with re- sources allocated according to each strategy and with the resources R t+1 ⊆ T R i (P t , c t , s i ) − P t allocated according to bk [n]−i . This is feasible because ck {i} is compatible with bk [n]−i , and bk and a cheap lunch ck {i} ∈ Feasible i ( P k ) such that: 1. ck {i} is compatible with bk [n]−i ; 2. the resources gained from region i by taking the ac- tions ck P t ; and {i} provide the needed resources to implement bk[n]−i , i.e. for all t we have R t+1 ⊆ T R i (P t , c t , s i ) −


Abstract: For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partialinformation game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.

Dylan Hadfield-Menell
Anca Dragan
Pieter Abbeel
Stuart Russell

Introduction "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively . . . we had better be quite sure that the purpose put into the machine is the purpose which we really desire." So wrote Norbert  Wiener (1960)  in one of the earliest explanations of the problems that arise when a powerful autonomous system operates with an incorrect objective. This value alignment problem is far from trivial. Humans are prone to mis-stating their objectives, which can lead to unexpected implementations. In the myth of King Midas, the main character learns that wishing for 'everything he touches to turn to gold' leads to disaster. In a reinforcement learning context,  Russell & Norvig (2010)  describe a seemingly reasonable, but incorrect, reward function for a vacuum robot: if we reward the action of cleaning up dirt, the optimal policy causes the robot to repeatedly dump and clean up the same dirt. A solution to the value alignment problem has long-term implications for the future of AI and its relationship to humanity  (Bostrom, 2014 ) and short-term utility for the design of usable AI systems. Giving robots the right objectives and enabling them to make the right trade-offs is crucial for self-driving cars, personal assistants, and human-robot interaction more broadly. The field of inverse reinforcement learning or IRL  (Russell, 1998; Ng & Russell, 2000; Abbeel & Ng, 2004 ) is certainly relevant to the value alignment problem. An IRL algorithm infers the reward function of an agent from observations of the agent's behavior, which is assumed to be optimal (or approximately so). One might imagine that IRL provides a simple solution to the value alignment problem: the robot observes human behavior, learns the human reward function, and behaves according to that function. This simple idea has two flaws. The first flaw is obvious: we don't want the robot to adopt the human reward function as its own. For example, human behavior (especially in the morning) often conveys a desire for coffee, and the robot can learn this with IRL, but we don't want the robot to want coffee! This flaw is easily fixed: we need to formulate the value alignment problem so that the robot always has the fixed objective of optimizing reward for the human, and becomes better able to do so as it learns what the human reward function is. The second flaw is less obvious, and less easy to fix. IRL assumes that observed behavior is optimal in the sense that it accomplishes a given task efficiently. This precludes a variety of useful teaching behaviors. For example, efficiently making a cup of coffee, while the robot is a passive observer, is a inefficient way to teach a robot to get coffee. Instead, the human should perhaps explain the steps in coffee preparation and show the robot where the backup coffee supplies are kept and what do if the coffee pot is left on the heating plate too long, while the robot might ask what the button with the puffy steam symbol is for and try its hand at coffee making with guidance from the human, even if the first results are undrinkable. None of these things fit in with the standard IRL framework. Cooperative inverse reinforcement learning. We propose, therefore, that value alignment should be formulated as a cooperative and interactive reward maximization process. More precisely, we define a cooperative inverse reinforcement learning (CIRL) game as a two-player game of partial information, in which the "human", H, knows the reward function (represented by a generalized parameter θ), while the "robot", R, does not; the robot's payoff is exactly the human's actual reward. Optimal solutions to this game maximize human reward; we show that solutions may involve active instruction by the human and active learning by the robot. Reduction to POMDP and Sufficient Statistics. As one might expect, the structure of CIRL games is such that they admit more efficient solution algorithms than are possible for general partialinformation games. Let (π H , π R ) be a pair of policies for human and robot, each depending, in general, on the complete history of observations and actions. A policy pair yields an expected sum of rewards for each player. CIRL games are cooperative, so there is a well-defined optimal policy pair that maximizes value.  2  In Section 3 we reduce the problem of computing an optimal policy pair to the solution of a (single-agent) POMDP. This shows that the robot's posterior over θ is a sufficient statistic, in the sense that there are optimal policy pairs in which the robot's behavior depends only on this statistic. Moreover, the complexity of solving the POMDP is exponentially lower than the NEXP-hard bound that  (Bernstein et al., 2000)  obtained by reducing a CIRL game to a general Dec-POMDP. Apprenticeship Learning and Suboptimality of IRL-Like Solutions. In Section 3.3 we model apprenticeship learning  (Abbeel & Ng, 2004 ) as a two-phase CIRL game. In the first phase, the learning phase, both H and R can take actions and this lets R learn about θ. In the second phase, the deployment phase, R uses what it learned to maximize reward (without supervision from H). We show that classic IRL falls out as the best-response policy for R under the assumption that the human's policy is "demonstration by expert" (DBE), i.e., acting optimally in isolation as if no robot exists. But we show also that this DBE/IRL policy pair is not, in general, optimal: even if the robot expects expert behavior, demonstrating expert behavior is not the best way to teach that algorithm. We give an algorithm that approximately computes H's best response when R is running IRL under the assumption that rewards are linear in θ and state features. Section 4 compares this best-response policy with the DBE policy in an example game and provides empirical confirmation that the bestresponse policy, which turns out to "teach" R about the value landscape of the problem, is better than DBE. Thus, designers of apprenticeship learning systems should expect that users will violate the assumption of expert demonstrations in order to better communicate information about the objective.
 Related Work Our proposed model shares aspects with a variety of existing models. We divide the related work into three categories: inverse reinforcement learning, optimal teaching, and principal-agent models. Inverse Reinforcement Learning.  Ng & Russell (2000)  define inverse reinforcement learning (IRL) as follows: "Given measurements of an [actor]'s behavior over time. . . . Determine the reward function being optimized." The key assumption IRL makes is that the observed behavior is optimal in the sense that the observed trajectory maximizes the sum of rewards. We call this the demonstration-by-expert (DBE) assumption. One of our contributions is to prove that this may be suboptimal behavior in a CIRL game, as H may choose to accept less reward on a particular action in order to convey more information to R. In CIRL the DBE assumption prescribes a fixed policy
 Ground Truth
 Expert Demonstration Instructive Demonstration Figure  1 : The difference between demonstration-by-expert and instructive demonstration in the mobile robot navigation problem from Section 4. Left: The ground truth reward function. Lighter grid cells indicates areas of higher reward. Middle: The demonstration trajectory generated by the expert policy, superimposed on the maximum a-posteriori reward function the robot infers. The robot successfully learns where the maximum reward is, but little else. Right: An instructive demonstration generated by the algorithm in Section 3.4 superimposed on the maximum a-posteriori reward function that the robot infers. This demonstration highlights both points of high reward and so the robot learns a better estimate of the reward. for H. As a result, many IRL algorithms can be derived as state estimation for a best response to different π H , where the state includes the unobserved reward parametrization θ.  Ng & Russell (2000) ,  Abbeel & Ng (2004), and Ratliff et al. (2006)  compute constraints that characterize the set of reward functions so that the observed behavior maximizes reward. In general, there will be many reward functions consistent with this constraint. They use a max-margin heuristic to select a single reward function from this set as their estimate. In CIRL, the constraints they compute characterize R's belief about θ under the DBE assumption.  Ramachandran & Amir (2007)  and  Ziebart et al. (2008)  consider the case where π H is "noisily expert," i.e., π H is a Boltzmann distribution where actions or trajectories are selected in proportion to the exponent of their value.  Ramachandran & Amir (2007)  adopt a Bayesian approach and place an explicit prior on rewards.  Ziebart et al. (2008)  places a prior on reward functions indirectly by assuming a uniform prior over trajectories. In our model, these assumptions are variations of DBE and both implement state estimation for a best response to the appropriate fixed H.  Natarajan et al. (2010)  introduce an extension to IRL where R observes multiple actors that cooperate to maximize a common reward function. This is a different type of cooperation than we consider, as the reward function is common knowledge and R is a passive observer.  Waugh et al. (2011)  and  Kuleshov & Schrijvers (2015)  consider the problem of inferring payoffs from observed behavior in a general (i.e., non-cooperative) game given observed behavior. It would be interesting to consider an analogous extension to CIRL, akin to mechanism design, in which R tries to maximize collective utility for a group of Hs that may have competing objectives. Fern et al. (  2014 ) consider a hidden-goal MDP, a special case of a POMDP where the goal is an unobserved part of the state. This can be considered a special case of CIRL, where θ encodes a particular goal state. The frameworks share the idea that R helps H. The key difference between the models lies in the treatment of the human (the agent in their terminology). Fern et al. (  2014 ) model the human as part of the environment. In contrast, we treat H as an actor in a decision problem that both actors collectively solve. This is crucial to modeling the human's incentive to teach. Optimal Teaching. Because CIRL incentivizes the human to teach, as opposed to maximizing reward in isolation, our work is related to optimal teaching: finding examples that optimally train a learner  (Balbach & Zeugmann, 2009; Goldman et al., 1993; Goldman & Kearns, 1995) . The key difference is that efficient learning is the objective of optimal teaching, while it emerges as a property of optimal equilibrium behavior in CIRL.  Cakmak & Lopes (2012)  consider an application of optimal teaching where the goal is to teach the learner the reward function for an MDP. The teacher gets to pick initial states from which an expert executes the reward-maximizing trajectory. The learner uses IRL to infer the reward function, and the teacher picks initial states to minimize the learner's uncertainty. In CIRL, this approach can be characterized as an approximate algorithm for H that greedily minimizes the entropy of R's belief. Beyond teaching, several models focus on taking actions that convey some underlying state, not necessarily a reward function. Examples include finding a motion that best communicates an agent's intention  (Dragan & Srinivasa, 2013) , or finding a natural language utterance that best communicates a particular grounding  (Golland et al., 2010) . All of these approaches model the observer's inference process and compute actions (motion or speech) that maximize the probability an observer infers the correct hypothesis or goal. Our approximate solution to CIRL is analogous to these approaches, in that we compute actions that are informative of the correct reward function. Principal-agent models. Value alignment problems are not intrinsic to artificial agents.  Kerr (1975)  describes a wide variety of misaligned incentives in the aptly titled "On the folly of rewarding A, while hoping for B." In economics, this is known as the principal-agent problem: the principal (e.g., the employer) specifies incentives so that an agent (e.g., the employee) maximizes the principal's profit  (Jensen & Meckling, 1976) . Principal-agent models study the problem of generating appropriate incentives in a non-cooperative setting with asymmetric information. In this setting, misalignment arises because the agents that economists model are people and intrinsically have their own desires. In AI, misalignment arises entirely from the information asymmetry between the principal and the agent; if we could characterize the correct reward function, we could program it into an artificial agent.  Gibbons (1998)  provides a useful survey of principal-agent models and their applications.
 Cooperative Inverse Reinforcement Learning This section formulates CIRL as a two-player Markov game with identical payoffs, reduces the problem of computing an optimal policy pair for a CIRL game to solving a POMDP, and characterizes apprenticeship learning as a subclass of CIRL games.
 CIRL Formulation Definition 1. A cooperative inverse reinforcement learning (CIRL) game M is a two-player Markov game with identical payoffs between a human or principal, H, and a robot or agent, R. The game is described by a tuple,  M = S, {A H , A R }, T (•|•, •, •), {Θ, R(•, •, •; •)}, P 0 (•, •), γ , (s 0 , θ) γ a discount factor: γ ∈ [0, 1]. We write the reward for a state-parameter pair as R(s, a H , a R ; θ) to distinguish the static reward parameters θ from the changing world state s. The game proceeds as follows. First, the initial state, a tuple (s, θ), is sampled from P 0 . H observes θ, but R does not. This observation model captures the notion that only the human knows the reward function, while both actors know a prior distribution over possible reward functions. At each timestep t, H and R observe the current state s t and select their actions a H t , a R t . Both actors receive reward r t = R(s t , a H t , a R t ; θ) and observe each other's action selection. A state for the next timestep is sampled from the transition distribution, s t+1 ∼ P T (s |s t , a H t , a R t ), and the process repeats. Behavior in a CIRL game is defined by a pair of policies, (π H , π R ), that determine action selection for H and R respectively. In general, these policies can be arbitrary functions of their observation histories; π H : A H × A R × S * × Θ → A H , π R : A H × A R × S * → A R . The optimal joint policy is the policy that maximizes value. The value of a state is the expected sum of discounted rewards under the initial distribution of reward parameters and world states. Remark 1. A key property of CIRL is that the human and the robot get rewards determined by the same reward function. This incentivizes the human to teach and the robot to learn without explicitly encoding these as objectives of the actors.
 Structural Results for Computing Optimal Policy Pairs The analogue in CIRL to computing an optimal policy for an MDP is the problem of computing an optimal policy pair. This is a pair of policies that maximizes the expected sum of discounted rewards. This is not the same as 'solving' a CIRL game, as a real world implementation of a CIRL agent must account for coordination problems and strategic uncertainty  (Boutilier, 1999) . The optimal policy pair represents the best H and R can do if they can coordinate perfectly before H observes θ. Computing an optimal joint policy for a cooperative game is the solution to a decentralized-partially observed Markov decision process (Dec-POMDP). Unfortunately, Dec-POMDPs are NEXP-complete  (Bernstein et al., 2000)  so general Dec-POMDP algorithms have a computational complexity that is doubly exponential. Fortunately, CIRL games have special structure that reduces this complexity.  Nayyar et al. (2013)  shows that a Dec-POMDP can be reduced to a coordination-POMDP. The actor in this POMDP is a coordinator that observes all common observations and specifies a policy for each actor. These policies map each actor's private information to an action. The structure of a CIRL game implies that the private information is limited to H's initial observation of θ. This allows the reduction to a coordination-POMDP to preserve the size of the (hidden) state space, making the problem easier. Theorem 1. Let M be an arbitrary CIRL game with state space S and reward space Θ. There exists a (single-actor) POMDP M C with (hidden) state space S C such that |S C | = |S| • |Θ| and, for any policy pair in M , there is a policy in M C that achieves the same sum of discounted rewards. Theorem proofs can be found in the supplementary material. An immediate consequence of this result is that R's belief about θ is a sufficient statistic for optimal behavior. Corollary 1. Let M be a CIRL game. There exists an optimal policy pair (π H * , π R * ) that only depends on the current state and R's belief. Remark 2. In a general Dec-POMDP, the hidden state for the coordinator-POMDP includes each actor's history of observations. In CIRL, θ is the only private information so we get an exponential decrease in the complexity of the reduced problem. This allows one to apply general POMDP algorithms to compute optimal joint policies in CIRL. It is important to note that the reduced problem may still be very challenging. POMDPs are difficult in their own right and the reduced problem still has a much larger action space. That being said, this reduction is still useful in that it characterizes optimal joint policy computation for CIRL as significantly easier than Dec-POMDPs. Furthermore, this theorem can be used to justify approximate methods (e.g., iterated best response) that only depend on R's belief state.
 Apprenticeship Learning as a Subclass of CIRL Games A common paradigm for robot learning from humans is apprenticeship learning. In this paradigm, a human gives demonstrations to a robot of a sample task and the robot is asked to imitate it in a subsequent task. In what follows, we formulate apprenticeship learning as turn-based CIRL with a learning phase and a deployment phase. We characterize IRL as the best response (i.e., the policy that maximizes reward given a fixed policy for the other player) to a demonstration-by-expert policy for H. We also show that this policy is, in general, not part of an optimal joint policy and so IRL is generally a suboptimal approach to apprenticeship learning. Definition 2. (ACIRL) An apprenticeship cooperative inverse reinforcement learning (ACIRL) game is a turn-based CIRL game with two phases: a learning phase where the human and the robot take turns acting, and a deployment phase, where the robot acts independently.
 Example. Consider an example apprenticeship task where R needs to help H make office supplies. H and R can make paperclips and staples and the unobserved θ describe H's preference for paperclips vs staples. We model the problem as an ACIRL game in which the learning and deployment phase each consist of an individual action. The world state in this problem is a tuple (p s , q s , t) where p s and q s respectively represent the number of paperclips and staples H owns. t is the round number. An action is a tuple (p a , q a ) that produces p a paperclips and q a staples. The human can make 2 items total: A H = {(0, 2), (1, 1), (2, 0)}. The robot has different capabilities. It can make 50 units of each item or it can choose to make 90 of a single item: A R = {(0, 90), (50, 50), (90, 0)}. We let Θ = [0, 1] and define R so that θ indicates the relative preference between paperclips and staples:R(s, (p a , q a ); θ) = θp a + (1 − θ)q a . R's action is ignored when t = 0 and H's is ignored when t = 1. At t = 2, the game is over, so the game transitions to a sink state, (0, 0, 2). Deployment phase -maximize mean reward estimate. It is simplest to analyze the deployment phase first. R is the only actor in this phase so it get no more observations of its reward. We have shown that R's belief about θ is a sufficient statistic for the optimal policy. This belief about θ induces a distribution over MDPs. A straightforward extension of a result due to  Ramachandran & Amir (2007)  shows that R's optimal deployment policy maximizes reward for the mean reward function. Theorem 2. Let M be an ACIRL game. In the deployment phase, the optimal policy for R maximizes reward in the MDP induced by the mean θ from R's belief. In our example, suppose that π H selects (0, 2 ) if θ ∈ [0, 1 3 ), (1, 1) if θ ∈ [ 1 3 , 2 3 ] and (2, 0) otherwise. R begins with a uniform prior on θ so observing, e.g., a H = (0, 2) leads to a posterior distribution that is uniform on [0, 1 3 ). Theorem 2 shows that the optimal action maximizes reward for the mean θ so an optimal R behaves as though θ = 1 6 during the deployment phase. Learning phase -expert demonstrations are not optimal. A wide variety of apprenticeship learning approaches assume that demonstrations are given by an expert. We say that H satisfies the demonstration-by-expert (DBE) assumption in ACIRL if she greedily maximizes immediate reward on her turn. This is an 'expert' demonstration because it demonstrates a reward maximizing action but does not account for that action's impact on R's belief. We let π E represent the DBE policy. Theorem 2 enables us to characterize the best response for R when π H = π E : use IRL to compute the posterior over θ during the learning phase and then act to maximize reward under the mean θ in the deployment phase. We can also analyze the DBE assumption itself. In particular, we show that π E is not H's best response when π R is a best response to π E . Theorem 3. There exist ACIRL games where the best-response for H to π R violates the expert demonstrator assumption. In other words, if br(π) is the best response to π, then br(br(π E )) = π E . The supplementary material proves this theorem by computing the optimal equilibrium for our example. In that equilibrium, H selects (1, 1) if θ ∈ [ 41 92 , 51 92 ]. In contrast, π E only chooses (1, 1) if θ = 0.5. The change arises because there are situations (e.g., θ = 0.49) where the immediate loss of reward to H is worth the improvement in R's estimate of θ. Remark 3. We should expect experienced users of apprenticeship learning systems to present demonstrations optimized for fast learning rather than demonstrations that maximize reward. Crucially, the demonstrator is incentivized to deviate from R's assumptions. This has implications for the design and analysis of apprenticeship systems in robotics. Inaccurate assumptions about user behavior are notorious for exposing bugs in software systems (see, e.g.,  Leveson & Turner (1993) ).
 Generating Instructive Demonstrations Now, we consider the problem of computing H's best response when R uses IRL as a state estimator. For our toy example, we computed solutions exhaustively, for realistic problems we need a more efficient approach. Section 3.2 shows that this can be reduced to an POMDP where the state is a tuple of world state, reward parameters, and R's belief. While this is easier than solving a general Dec-POMDP, it is a computational challenge. If we restrict our attention to the case of linear reward functions we can develop an efficient algorithm to compute an approximate best response. Specifically, we consider the case where the reward for a state (s, θ) is defined as a linear combination of state features for some feature function φ : R(s, a H , a R ; θ) = φ(s) θ. Standard results from the IRL literature show that policies with the same expected feature counts have the same value  (Abbeel & Ng, 2004) . Combined with Theorem 2, this implies that the optimal π R under the DBE assumption computes a policy that matches the observed feature counts from the learning phase. This suggests a simple approximation scheme. To compute a demonstration trajectory τ H , first compute the feature counts R would observe in expectation from the true θ and then select actions that maximize similarity to these target features. If φ θ are the expected feature counts induced by θ then this scheme amounts to the following decision rule: τ H ← argmax τ φ(τ ) θ − η||φ θ − φ(τ )|| 2 . (1) This rule selects a trajectory that trades off between the sum of rewards φ(τ ) θ and the feature dissimilarity ||φ θ − φ(τ )|| 2 . Note that this is generally distinct from the action selected by the demonstration-by-expert policy. The goal is to match the expected sum of features under a distribution of trajectories with the sum of features from a single trajectory. The correct measure of feature  Lower numbers are better. Using the best response causes R to infer a better distribution over θ so it does a better job of maximizing reward. Right: The regret of the instructive demonstration policy as a function of how optimal R expects H to be. λ = 0 corresponds to a robot that expects purely random behavior and λ = ∞ corresponds to a robot that expects optimal behavior. Regret is minimized for an intermediate value of λ: if λ is too small, then R learns nothing from its observations; if λ is too large, then R expects many values of θ to lead to the same trajectory so H has no way to differentiate those reward functions. similarity is regret: the difference between the reward R would collect if it knew the true θ and the reward R actually collects using the inferred θ. Computing this similarity is expensive, so we use an 2 norm as a proxy measure of similarity.
 Experiments
 Cooperative Learning for Mobile Robot Navigation Our experimental domain is a 2D navigation problem on a discrete grid. In the learning phase of the game, H teleoperates a trajectory while R observes. In the deployment phase, R is placed in a random state and given control of the robot. We use a finite horizon H, and let the first H 2 timesteps be the learning phase. There are N φ state features defined as radial basis functions where the centers are common knowledge. Rewards are linear in these features and θ. The initial world state is in the middle of the map. We use a uniform distribution on [−1, 1] N φ for the prior on θ. Actions move in one of the four cardinal directions {N, S, E, W } and there is an additional no-op ∅ that each actor executes deterministically on the other agent's turn. Figure  1  shows an example comparison between demonstration-by-expert and the approximate best response policy in Section 3.4. The leftmost image is the ground truth reward function. Next to it are demonstration trajectories produce by these two policies. Each path is superimposed on the maximum a-posteriori reward function the robot infers from the demonstration. We can see that the demonstration-by-expert policy immediately goes to the highest reward and stays there. In contrast, the best response policy moves to both areas of high reward. The robot reward function the robot infers from the best response demonstration is much more representative of the true reward function, when compared with the reward function it infers from demonstration-by-expert.
 Demonstration-by-Expert vs Best Responder Hypothesis. When R plays an IRL algorithm that matches features, H prefers the best response policy from Section 3.4 to π E : the best response policy will significantly outperform the DBE policy. Manipulated Variables. Our experiment consists of 2 factors: H-policy and num-features. We make the assumption that R uses an IRL algorithm to compute its estimate of θ during learning and maximizes reward under this estimate during deployment. We use Maximum-Entropy IRL  (Ziebart et al., 2008)  to implement R's policy. H-policy varies H's strategy π H and has two levels: demonstration-by-expert (π E ) and best-responder (br). In the π E level H maximizes reward during the demonstration. In the br level H uses the approximate algorithm from Section 3.4 to compute an approximate best response to π R . The trade-off between reward and communication η is set by cross-validation before the game begins. The num-features factor varies the dimensionality of φacross two levels: 3 features and 10 features. We do this to test whether and how the difference between experts and best-responders is affected by dimensionality. We use a factorial design that leads to 4 distinct conditions. We test each condition against a random sample of N = 500 different reward parameters. We use a within-subjects design with respect to the the H-policy factor so the same reward parameters are tested for π E and br. Dependent Measures. We use the regret with respect to a fully-observed setting where the robot knows the ground truth θ as a measure of performance. We let θ be the robot's estimate of the reward parameters and let θ GT be the ground truth reward parameters. The primary measure is the regret of R's policy: the difference between the value of the policy that maximizes the inferred reward θ and the value of the policy that maximizes the true reward θ GT . We also use two secondary measures. The first is the KL-divergence between the maximum-entropy trajectory distribution induced by θ and the maximum-entropy trajectory distribution induced by θ. Finally, we use the 2 -norm between the vector or rewards defined by θ and the vector induced by θ GT . Results. There was relatively little correlation between the measures (Cronbach's α of .47), so we ran a factorial repeated measures ANOVA for each measure. Across all measures, we found a significant effect for H-policy, with br outperforming π E on all measures as we hypothesized (all with F > 962, p < .0001). We did find an interaction effect with num-features for KL-divergence and the 2 -norm of the reward vector but post-hoc Tukey HSD showed br to always outperform π E . The interaction effect arises because the gap between the two levels of H-policy is larger with fewer reward parameters; we interpret this as evidence that num-features = 3 is an easier teaching problem for H. Figure  2  (Left, Middle) shows the dependent measures from our experiment.
 Varying R's Expectations Maximum-Entropy IRL includes a free parameter λ that controls how optimal R expects H to behave. If λ = 0, R will update its belief as if H's observed behavior is independent of her preferences θ. If λ = ∞, R will update its belief as if H's behavior is exactly optimal. We ran a followup experiment to determine how varying λ changes the regret of the br policy. Changing λ changes the forward model in R's belief update: the mapping R hypothesizes between a given reward parameter θ and the observed feature counts φ θ . This mapping is many-to-one for extreme values of λ. λ ≈ 0 means that all values of θ lead to the same expected feature counts because trajectories are chosen uniformly at random. Alternatively, λ >> 0 means that almost all probability mass falls on the optimal trajectory and many values of θ will lead to the same optimal trajectory. This suggests that it is easier for H to differentiate different values of θ if R assumes she is noisily optimal, but only up until a maximum noise level. Figure  2  plots regret as a function of λ and supports this analysis: H has less regret for intermediate values of λ.
 Conclusion and Future Work In this work, we presented a game-theoretic model for cooperative learning, CIRL. Key to this model is that the robot knows that it is in a shared environment and is attempting to maximize the human's reward (as opposed to estimating the human's reward function and adopting it as its own). This leads to cooperative learning behavior and provides a framework in which to design HRI algorithms and analyze the incentives of both actors in a reward learning environment. We reduced the problem of computing an optimal policy pair to solving a POMDP. This is a useful theoretical tool and can be used to design new algorithms, but it is clear that optimal policy pairs are only part of the story. In particular, when it performs a centralized computation, the reduction assumes that we can effectively program both actors to follow a set coordination policy. This is clearly infeasible in reality, although it may nonetheless be helpful in training humans to be better teachers. An important avenue for future research will be to consider the coordination problem: the process by which two independent actors arrive at policies that are mutual best responses. Returning to Wiener's warning, we believe that the best solution is not to put a specific purpose into the machine at all, but instead to design machines that provably converge to the right purpose as they go along. Figure 2 : 2 Figure 2: Left, Middle: Comparison of 'expert' demonstration (π E ) with 'instructive' demonstration (br).
			 A coordination problem of the type described in Boutilier (1999)  arises if there are multiple optimal policy pairs; we defer this issue to future work.