Abstract: We consider decentralized control of Markov decision processes and give complexity bounds on the worst-case running time for algorithms that find optimal solutions. Generalizations of both the fully observable case and the partially observable case that allow for decentralized control are described. For even two agents, the finite-horizon problems corresponding to both of these models are hard for nondeterministic exponential time. These complexity results illustrate a fundamental difference between centralized and decentralized control of Markov decision processes. In contrast to the problems involving centralized control, the problems we consider provably do not admit polynomial-time algorithms. Furthermore, assuming EXP = NEXP, the problems require superexponential time to solve in the worst case.

Daniel S Bernstein
Robert Givan
Neil Immerman
Shlomo Zilberstein

THE COMPLEXITY OF DECENTRALIZED CONTROL OF MARKOV DECISION PROCESSES

1. Introduction. Markov decision processes (MDPs) have received considerable attention, and there exist well-known algorithms for finding optimal control strategies in the case where a process is centrally controlled and the controller (or agent) has access to complete state information  (Puterman 1994) . Less well understood is the case in which a process is controlled by multiple cooperating distributed agents, each with possibly different information about the state. We are interested in studying how hard these decentralized control problems are relative to analogous centralized control problems, from the point of view of computational complexity. In particular, we consider two different models of decentralized control of MDPs. One is a generalization of a partially observable Markov decision process (POMDP), which we call a decentralized partially observable Markov decision process (DEC-POMDP). In a DEC-POMDP, the process is controlled by multiple distributed agents, each with possibly different information about the state. The other is a generalization of an MDP, called a decentralized Markov decision process (DEC-MDP). A DEC-MDP is a DEC-POMDP with the restriction that at each time step the agents' observations together uniquely determine the state. The MDP, POMDP, and DEC-MDP can all be viewed as special cases of the DEC-POMDP. The relationships among the models are shown in Figure  1 . A number of different problems can be viewed as decentralized control of a Markov process. For example, consider problems involving the control of multiple distributed robots, such as robotic soccer  (Coradeschi et al. 2000) . In these domains, it is necessary to develop a strategy for each robot under the assumption that the robots will have limited ability to communicate when they execute their strategies. Another problem that fits naturally within this framework is the distributed control of a power grid  (Schneider et al. 1999) . Finally, several types of networking problems can be viewed within this framework  (Altman 2001) . It would be beneficial to have general-purpose algorithms for solving these decentralized control problems. An algorithm for similar problems was proposed by  Ooi et al. (1996) . Under the assumption that all agents share state information every K time steps, the authors developed a dynamic programming algorithm to derive optimal policies. A downside of this approach is that the state space for the dynamic programming algorithm grows doubly exponentially with K. The only known tractable algorithms for these types of problems rely on even more assumptions. One such algorithm was developed by  Hsu and Marcus (1982)  and works under the assumption that the agents share state information every time step (although it can take one time step for the information to propagate). Approximation algorithms have also been developed for these problems, although they can at best give guarantees of local optimality. For instance,  Peshkin et al. (2000)  studied algorithms that perform gradient descent in a space of parameterized policies. Is there something inherent in these problems that forces us to add assumptions and/or use approximation algorithms?  Papadimitriou and Tsitsiklis (1982)  presented some results aimed at answering this question. The authors proved that a simple decentralized decisionmaking problem is NP-complete, even with just two decision makers. They later noted that this implies that decentralized control of MDPs must be NP-hard  (Papadimitriou and Tsitsiklis 1986) . We strengthen this result by showing that both the DEC-POMDP and DEC-MDP problems are NEXP-hard, even when the horizon is limited to be less than the number of states (and they are NEXP-complete in the latter case). Although it is not known whether the classes P, NP, and PSPACE are distinct, it is known that P = NEXP, and thus the problems we consider are provably intractable. Furthermore, assuming EXP = NEXP, the problems take superexponential time to solve in the worst case. This result is in contrast to the best known bounds for MDPs (P-hard) and POMDPs (PSPACE-hard)  (Papadimitriou and Tsitsiklis 1987, Mundhenk et al. 2000) . Thus, we have gained insight into the possibility of a fundamental difference between centralized and decentralized control of Markov decision processes. In §2, we give a brief review of the concepts we will need from complexity theory. In §3, we define the MDP and POMDP models. Section 4 contains the definitions of the DEC-MDP and DEC-POMDP models, and a proof that the short-horizon versions of these problems fall within the complexity class NEXP. In §5, we present our main complexity result-a proof that these decentralized problems are NEXP-hard. Finally, §6 contains our conclusions. 2. Computational complexity. In this section, we give a brief introduction to the theory of computational complexity. More detail can be found in  Papadimitriou (1994) . A complexity class is a set of problems where a problem is an infinite set of problem instances, each of which has a "yes" or "no" answer. To discuss the complexity of optimization problems, we must have a way of converting them to "yes/no" problems. The typical way this is done is to set a threshold and ask whether or not the optimal solution yields a reward that is no less than this threshold. The problem of actually finding the optimal solution can, of course, be no easier than the threshold problem. The first complexity class we consider is P, the set of problems that can be solved in polynomial time (in the size of the problem instance) on a sequential computer. NP is the set of problems that can be solved nondeterministically in polynomial time. A nondeterministic machine automatically knows the correct path to take any time there is a choice as to how the computation should proceed. An example of a problem that can be solved nondeterministically in polynomial time is deciding whether a sentence of propositional logic is satisfiable. The machine can guess an assignment of truth values to variables and evaluate the resulting expression in polynomial time. Of course, nondeterministic machines do not really exist, and the most efficient known algorithms for simulating them take exponential time in the worst case. In fact, it is strongly believed by most complexity theorists that P = NP (but this has not been proven formally). Complexity can also be measured in terms of the amount of space a computation requires. One class, PSPACE, includes all problems that can be solved in polynomial space. Any problem that can be solved in polynomial time or nondeterministic polynomial time can be solved in polynomial space (i.e., P ⊆ NP ⊆ PSPACE)-that P ⊆ PSPACE can be seen informally by observing that only polynomially much space can be accessed in polynomially many time steps. Moving up the complexity hierarchy, we have exponential time (EXP) and nondeterministic exponential time (NEXP). By exponential time, we mean time bounded by 2 n k , where n is the input size and k > 0 is a constant. It is known that PSPACE ⊆ EXP ⊆ NEXP, and it is believed that EXP = NEXP (but again this has not been proven). It has been proven that the classes P and EXP are distinct, however. The notion of a reduction is important in complexity theory. We say that a problem A is reducible to a problem B if any instance x of A can be converted into an instance f x of B such that the answer to x is "yes" if and only if the answer to f x is "yes." A problem A is said to be hard for a complexity class C (or C-hard) if any problem in C is efficiently reducible to A. If the complexity class in question is P, efficient means that f x can be computed using at most logarithmic space, while for the classes above P, efficient means that f x can be computed using at most polynomial time. A problem A is said to be complete for a complexity class C (or C-complete) if (a) A is contained in C, and (b) A is hard for C. For instance, the satisfiability problem mentioned above is NP-complete and P-hard. However, unless P = NP, satisfiability is not P-complete. 3. Centralized models. In this paper, we consider discrete-time finite sequential decision processes under the undiscounted finite-horizon optimality criterion. We build into our problem definitions the (unusual) assumption that the horizon is less than the number of states. Note that this assumption actually strengthens the hardness results; the general problems must be at least as hard as their short-horizon counterparts. Unfortunately, the assumption is needed for each of the upper bounds given below. Finding tight upper bounds for problems with arbitrary horizons remains an open problem  (Blondel and Tsitsiklis 1999, §5) . Below we describe the partially observable Markov decision process and its associated decision problem. The Markov decision process is viewed as a restricted version of this model. A partially observable Markov decision process (POMDP) is defined as follows. We are given a tuple, S A P R O T K , where • S is a finite set of states, with distinguished initial state s 0 . • A is a finite action set. • P is a transition probability table. P s a s is a rational representing the probability of transitioning from s to s on taking action a. Here s s ∈ S and a ∈ A. • R is a reward function. R s a s is a rational representing the reward obtained from taking action a from state s and transitioning to state s . Again, s s ∈ S and a ∈ A. • is a finite set of observations. • O is a table of observation probabilities. O s a s o is a rational representing the probability of observing o when taking action a in state s and transitioning to state s as a result. Here s s ∈ S, a ∈ A, and o ∈ . • T is a positive integer representing the horizon (and T < S ). • K is a rational representing the threshold value. A POMDP is fully observable if there exists a mapping J → S such that whenever O s a s o is nonzero, J o = s . A Markov decision process (MDP) is defined to be a POMDP that is fully observable. A policy is defined to be a mapping from sequences of observations ō = o 1 • • • o t over to actions in A. We wish to find a policy that maximizes the expected total return over the finite horizon. The definitions below are used to formalize this notion. We use the symbol to denote the empty observation sequence. For an observation sequence ō = o 1 • • • o t , ōo is taken to represent the sequence o 1 • • • o t o. Definition. The probability of transitioning from a state s to a state s following policy while the agent sees observation sequence ō, written P s ō s , can be defined recursively as follows: P s s = 1 P s ōo s = q∈S P s ō q P q ō s O q ō s o where is the empty sequence. Definition. The value V T s of following policy from state s for T steps is given by the following equation: V T s = ō q∈S s ∈S P s ō q P q ō s R q ō s where the observation sequences have length at most T − 1. The decision problem corresponding to a finite-horizon POMDP is as follows. Given a POMDP D = S A P R O T K , is there a policy for which V T s 0 equals or exceeds K? It was shown in  Papadimitriou and Tsitsiklis (1987)  that the decision problem for POMDPs is PSPACE-complete and that the decision problem for MDPs is P-complete. 4. Decentralized models. We now describe extensions to the aforementioned models that allow for decentralized control. In these models, at each step, each agent receives a local observation and subsequently chooses an action. The state transitions and rewards received depend on the vector of actions of all the agents. A decentralized partially observable Markov decision process (DEC-POMDP) is defined formally as follows (for ease of exposition, we describe the two-agent case). We are given S A 1 A 2 P R 1 2 O T K , where • S is a finite set of states, with distinguished initial state s 0 . • A 1 and A 2 are finite action sets. • P is a transition probability table. P s a 1 a 2 s is a rational representing the probability of transitioning from s to s on taking actions a 1 a 2 . Here s s ∈ S, a 1 ∈ A 1 , and a 2 ∈ A 2 . • R is a reward function. R s a 1 a 2 s is a rational representing the reward obtained from taking actions a 1 a 2 from state s and transitioning to state s . Again s s ∈ S, a 1 ∈ A 1 , and a 2 ∈ A 2 . • 1 and 2 are finite sets of observations. • O is a table of observation probabilities. O s a 1 a 2 s o 1 o 2 is a rational representing the probability of observing o 1 o 2 when taking actions a 1 a 2 in state s and transitioning to state s as a result. Here s s ∈ S, a 1 ∈ A 1 , a 2 ∈ A 2 , o 1 ∈ 1 , and o 2 ∈ 2 . • T is a positive integer representing the horizon (and T < S ). • K is a rational representing the threshold value. A DEC-POMDP generalizes a POMDP by allowing for control by multiple distributed agents that together may not fully observe the system state (so we have only partial observability). We also define a generalization of MDP problems by requiring joint observability. We say that a DEC-POMDP is jointly observable if there exists a mapping J 1 × 2 → S such that whenever O s a 1 a 2 s o 1 o 2 is nonzero, J o 1 o 2 = s . We define a decentralized Markov decision process (DEC-MDP) to be a DEC-POMDP that is jointly observable. We define a local policy for agent i, i , to be a mapping from local histories of observations ōi = o i1 • • • o it over i , to actions in A i . A joint policy, = 1 2 , is defined to be a pair of local policies, one for each agent. We wish to find a joint policy that maximizes the expected total return over the finite horizon. As in the centralized case, we need some definitions to make this notion more formal. Definition. The probability of transitioning from a state s to a state s following joint policy = 1 2 while agent 1 sees observation sequence ō1 and agent 2 sees ō2 of the same length, written P s ō1 ō2 s , can be defined recursively as follows: P s s = 1 P s ō1 o 1 ō2 o 2 s = q∈S P s ō1 ō2 q P q 1 ō1 2 ō2 s O q 1 ō1 2 ō2 s o 1 o 2 where is the empty sequence. Definition. The value V T s of following policy = 1 2 from state s for T steps is given by the following equation: V T s = ō1 ō2 q∈S s ∈S P s ō1 ō2 q P q 1 ō1 2 ō2 s R q 1 ō1 2 ō2 s where the observation sequences are of length at most T − 1, and both sequences in any pair are of the same length. The decision problem is stated as follows. Given a DEC-POMDP D = S A 1 A 2 P R 1 2 O T K , is there a joint policy for which V T s 0 equals or exceeds K? We let DEC-POMDP m and DEC-MDP m denote the decision problems for the m-agent DEC-POMDP and the m-agent DEC-MDP, respectively. We conclude this section by showing a straightforward upper bound on the worst-case time complexity of DEC-POMDP m for any m ≥ 2. Because any DEC-MDP is trivially a DEC-POMDP, this upper bound also applies to DEC-MDP m . Theorem 1. For all m ≥ 2, DEC-POMDP m ∈ NEXP. Proof. We must show that a nondeterministic machine can solve any instance of DEC-POMDP m using at most exponential time. First, a joint policy can be "guessed" and written down in exponential time. This is because a joint policy consists of m mappings from local histories to actions; and because T < S , all histories have length less than S . A DEC-POMDP together with a joint policy can be viewed as a POMDP together with a policy, where the observations in the POMDP correspond to the observation m-tuples in the DEC-POMDP (one from each agent), and the POMDP actions correspond to m-tuples of DEC-POMDP actions (again, one from each agent). In exponential time, each of the exponentially many possible sequences of observations can be converted into a belief state (i.e., a probability distribution over the state set giving the probability of being in each state after seeing the given observation sequence). We note that every POMDP  (Kaelbling et al. 1998 ) is equivalent to a "belief-state MDP" whose state set is the set of reachable belief states of the POMDP. The transition probabilities and expected rewards for the corresponding exponential-sized belief-state MDP can be computed in exponential time. Using standard MDP solution techniques  (Puterman 1994) , it is possible to determine whether the guessed policy yields expected reward at least K in this belief-state MDP in time that is at most polynomial in the size of the belief-state MDP, which is exponential in the size of the original DEC-POMDP problem. Therefore, there exists an accepting computation path if and only if there is a policy that can achieve reward K. 5. Decentralized control of MDPs is NEXP-hard. We now turn our attention to proving that the upper bound just shown in Theorem 1 is tight-specifically, we show that NEXP is also a lower bound for the worst-case time complexity of decentralized problems by showing that any problem in the class NEXP can be reduced in polynomial time to a DEC-MDP 2 problem. It then follows that both DEC-MDP m and DEC-POMDP m are NEXPcomplete for any m ≥ 2. The proof of this lower bound is quite involved and will occupy most of the remainder of this paper. Each subsection of this section contains a piece of the development, and at the end of the section the main theorem is asserted. We begin by introducing the known NEXP-complete problem TILING used in the proof. We then present an overview of the proof and its major constituents. Next we present the reduction from TILING formally, and finally we prove that the reduction is correct. 5.1. The TILING problem. We can show this lower bound by reducing any NEXPcomplete problem to DEC-MDP 2 using a polynomial-time algorithm. For our reduction, we use an NEXP-complete problem called TILING  (Lewis 1978 , Papadimitriou 1994  501), which is described as follows. We are given a board size n (represented compactly in binary), a set of tile types L = tile-0 tile-k , and a set of binary horizontal and vertical compatibility relations H V ⊆ L × L. A tiling is a mapping f 0 n − 1 × 0 n − 1 → L. A tiling f is consistent if and only if (a) f 0 0 = tile-0, and (b) for all x y f x y f x + 1 y ∈ H , and f x y f x y + 1 ∈ V . The decision problem is to determine, given L, H , V , and n, whether a consistent tiling exists. An example of a tiling instance and a corresponding consistent tiling is shown in Figure  2 . H = V = L = n = 4 a consistent tiling 0 1 2 3 0 1 2 3 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 2 2 2 2 2 2 2 2 Figure 2. An example of a tiling instance. In the remainder of this section, we assume that we have fixed an arbitrarily chosen instance of the tiling problem, so that L, H , V , and n are fixed. We then construct an instance of DEC-MDP that is solvable if and only if the selected tiling instance is solvable. We note that the DEC-MDP instance must be constructible in time polynomial in the size of the tiling instance (which in particular is logarithmic in the value of n), which will require the DEC-MDP instance to be at most polynomially larger than the tiling instance. 5.2. Overview of the reduction. The basic idea of our reduction is to create a twoagent DEC-MDP that randomly selects two tiling locations bit by bit, informing one agent of the first location and the other agent of the second location. The agents' local policies are observation-history based, so the agents can base their future actions on the tiling locations given to them. After generating the locations, the agents are simultaneously "queried" (i.e., a state is reached in which their actions are interpreted as answers to a query) for a tile type to place at the location given. We design the DEC-MDP problem so that the only way the agents can achieve nonnegative expected reward is to base their answers to the query on a single, jointly understood tiling that meets the constraints of the tiling problem. This design is complicated because the DEC-MDP state set itself cannot remember which tiling locations were selected (this would cause exponential blowup in the size of the state set, but our reduction must expand the problem size at most polynomially-we note that the tiling grid itself is not part of the tiling problem size, only the compactly represented grid size n is in the problem specification); the state will contain only certain limited information about the relative locations of the two tile positions. The difficulty of the design is also increased by the fact that any information remembered about the specified tiling locations must be shared with at least one of the agents to satisfy the joint observability requirement. To deal with these two issues, we have designed the DEC-MDP to pass through the following phases (a formal description follows later): Select Phase: Select two bit indices and values, each identifying a bit position and the value at that position in the location given to one of the agents. These are the only bits that are remembered in the state set from the locations given to the agents in the next phase-the other location bits are generated and forgotten by the process. The bit values remembered are called value-1 and value-2, and the indices to which these values correspond are called index-1 and index-2. Bit index-1 of the address given to agent 1 will have the value value-1, likewise for index-2, value-2, and agent 2. Generate Phase: Generate two tile locations at random, revealing one to each agent. The bits selected in the above select phase are used, and the other location bits are generated at random and immediately "forgotten" by the DEC-MDP state set. Query Phase: Query each agent for a tile type to place in the location that was specified to that agent. These tile types are remembered in the state. Echo Phase: Require the agents to echo the tile locations they received in the generate phase bit by bit. To enforce the accuracy of these location echoes, the DEC-MDP is designed to yield a negative reward if the bit remembered from the original location generation is not correctly echoed (the DEC-MDP is designed to ensure that each agent cannot know which bit is being checked in its echoes). As the agents echo the bits, the process computes state information representing whether the locations are equal or adjacent horizontally or vertically, and whether the agents' locations are both 0 0 (again, we cannot just remember the location bits because it would force an exponential state set). The echo phase allows us to compute state information about adjacency/equality of the locations after the tile types have been chosen, so that the agents' tile choices cannot depend on this information. This is critical in making the reduction correct. Test Phase: Check whether the tile types provided in the query phase come from a single consistent tiling. In other words, check that if the agents were asked for the same location they gave the same tile types during query, if they were asked for adjacent locations they gave types that satisfy the relevant adjacency constraints, and if the agents were both queried for location 0 0 they both gave tile type tile-0. The process gives a zero reward only if the tile types selected during the query phase meet any applicable constraints as determined by the echoed location bits. Otherwise, a negative reward is obtained. Note that because we are designing a DEC-MDP, we are required to maintain joint observability: The observations given to the agents at each time step must be sufficient to reconstruct all aspects of the DEC-MDP state at that time step. In particular, the bit indices and values selected in the select phase must be known to the agents (jointly), as well as the information computed in the echo phase regarding the relative position of the two locations. We achieve this joint observability by making all aspects of the DEC-MDP state observable to both agents, except for the indices and values selected in the select phase and the tile types that are given by the agents (and stored by the process) during the query phase. Each agent can observe which bit index and value are being remembered from the other agent's location, and each agent can observe the stored tile type it gave (but not the tile type given by the other agent). Because each agent can see what bit is saved from the other agent's location, we say that one location bit of each agent's location is visible to the other agent. We call the five phases just described "select," "generate," "query," "echo," and "test" in the development below. A formal presentation of the DEC-MDP just sketched follows below, but first we outline the proof that this approach represents a correct reduction.
 Overview of the correctness proof. Here we give an overview of our argument that the reduction sketched above is correct in the sense that there exists a policy that achieves expected total reward zero at the start state if and only if there is a solution to the tiling problem we started with. It is straightforward to show that if there exists a consistent tiling there must exist a policy achieving zero reward. The agents need only agree on a consistent tiling ahead of time and base their actions on the agreed-on tiling (waiting during selection and generation, giving the tile type present at the generated location during query, faithfully echoing the generated location during echo, and then waiting during test, at each point being guaranteed a zero reward by the structure of the problem). Note that it does not matter how expensive it might be to find and represent a consistent tiling or to carry out the policy just described because we are merely arguing for the existence of such a policy. We now outline the proof of the harder direction, that if there is no consistent tiling then there is no policy achieving expected reward zero. Note that because all rewards are nonpositive, any chance of receiving any negative reward forces the expected total reward to be negative. Consider an arbitrary policy that yields expected reward zero. Our argument rests on the following claims, which will be proved as lemmas in §5.5: Claim 1. The policy must repeat the two locations correctly during the echo phase. Claim 2. When executing the policy, the agents' selected actions during the query phase determine a single tiling, as follows. We define a query situation to be dangerous to an agent if and only if the observable bit value of the other agent's location (in the observation history) agrees with the bit value at the same index in the agent's own location (so that as far as the agent in danger knows, the other agent is being queried about the same location). During dangerous queries, the tile type selected by the agent in danger must depend only on the location queried (and not on the index or value of the bit observed from the other agent, on any other observable information, or on which agent is selecting the tile type). The agents' selected actions for dangerous queries thus determine a single tiling. Claim 3. The single tiling from Claim 2 is a consistent tiling. Claim 3 directly implies that if there is no consistent tiling, then all policies have negative expected reward, as desired. 5.4. Formal presentation of the reduction. Now we give the two-agent DEC-MDP D = S A 1 A 2 P R 1 2 O T K that is constructed from the selected tiling instance L H V n . We assume throughout that n is a power of two. It is straightforward to modify the proof to deal with the more general case-one way to do so is summarized briefly in Appendix A. 5.4.1. The state set. We describe the state set S of D below by giving a sequence of finite-domain "state variables" and then taking the state set to be the set of all possible assignments of values to the state variables. The finite-state automaton. One of the state variables will be maintained by a finitestate automaton (FSA), described in Appendix A. This variable, called rel-pos because it maintains a record of the relative position of the two location addresses echoed by the agents in the echo phase, can take on values from the state set of the FSA. Appendix A describes state set Q and also defines two functions (FSANEXT and FSA) based on the underlying FSA. These functions allow us to refer to the critical FSA behaviors here in our reduction while deferring most other details of the FSA to Appendix A. The function FSANEXT updates the state variable rel-pos based on one more bit of echoed location from each agent. The function FSA updates the state variable rel-pos based on a sequence of echoed location bits from each agent-so FSA is defined as a repeated application of the function FSANEXT. Appendix A also describes distinguished subsets of the FSA state set Q called apart, equal, hor, and ver representing possible relative positions for pairs of locations (not adjacent or equal, equal, horizontally adjacent, and vertically adjacent, respectively). These subsets are used below in defining the transition and reward behavior of the DEC-MDP D. Appendix A also defines the initial state q 0 of the FSA. The state variables. We now list the state variables defining the state set for D. We list the variables in three groups: the first group is observable to both agents, the second group only to agent 1, and the third group only to agent 2. These restrictions on observability are described in §5. We write a state by enumerating its variable values, e.g., as follows: gen 3 yes q 0 4 0 1 tile-1 5 1 0 tile-3 ∈ S. Semicolons are used to group together variables that have the same observability properties. We can represent sets of states by writing sets of values in some of the components of the tuple rather than just values. The * symbol is used to represent the set of all possible values for a component. We sometimes use a state variable as a function from states to domain values for that variable. For instance, if q matches gen * * * * * * * * * * * , then we will say phase q = gen. The initial state s 0 is as follows: select 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 . 5.4.2. The action sets and table of transition probabilities. We must allow "wait" actions, "zero" and "one" actions for echoing location address bits, and tile-type actions from the set of tile types L for answering during the query phase. We therefore take the action sets A 1 = A 2 to be wait 0 1 ∪ L. We give the transition distribution P s a 1 a 2 s for certain action pairs a 1 a 2 for certain source states s. For any source-state/action-pair combination not covered by the description below, the action pair is taken to cause a probability 1.0 self-transition back to the source state. The combinations not covered are not reachable from the initial state under any joint policy. Also, we note that the FSA-controlled state component rel-pos does not change from its initial state q 0 until the echo phase. Select phase. This is the first step of the process. In this step, the process chooses, for each agent, which of that agent's bits it will be checking in the echo phase. The value of that bit is also determined in this step. Transition probabilities when phase = select are given as follows: P s a 1 a 2 s = 1 4 log n 2 in the following situations: s = s 0 = select 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 s = gen 0 yes q 0 i 2 v 2 0 tile-0 i 1 v 1 0 tile-0 i 1 i 2 ∈ 0 2 log n − 1 and v 1 v 2 ∈ 0 1 Generate phase. During these steps, the two tile positions are chosen bit by bit. Note that we have to check for whether we are at one of the bits selected during the select phase, so that the value of the bit is the same as the value chosen during selection. Transition probabilities when phase = generate are given as follows. The second case describes the deterministic transition from the generate phase to the query phase. P s a 1 a 2 s = 1 h in the following situations: s = gen k yes q 0 i 2 v 2 * tile-0 i 1 v 1 * tile-0 where 0 ≤ k ≤ 2 log n − 1 s = gen k + 1 yes q 0 i 2 v 2 b 1 tile-0 i 1 v 1 b 2 tile-0 where b 1 = v 1 if k = i 1 else b 1 is either 0 or 1 b 2 = v 2 if k = i 2 else b 2 is either 0 or 1, and h is the number of allowed settings of b 1 b 2 from the previous two lines. P s a 1 a 2 s = 1 in the following situations: s = gen 2 log n yes q 0 i 2 v 2 * tile-0 i 1 v 1 * tile-0 and s = query 0 yes q 0 i 2 v 2 0 tile-0 i 1 v 1 0 tile-0 Query phase. The query phase consists of just one step, during which each agent chooses a tile type. Transition probabilities when phase = query are given as follows: P s a 1 a 2 s = 1 in the following situations: s = query 0 yes q 0 i 2 v 2 0 tile-0 i 1 v 1 0 tile-0 t 1 = a 1 if a 1 ∈ L tile-0 otherwise t 2 = a 2 if a 2 ∈ L tile-0 otherwise and s = echo 0 yes q 0 i 2 v 2 0 t 1 i 1 v 1 0 t 2 Echo phase. During the echo phase the agents are asked to repeat back the addresses seen in the generate phase, and information about the relative position of the addresses is calculated by the FSA described in Appendix A and recorded in the state. The FSA is accessed here using the function FSANEXT described in Appendix A. Transition probabilities when phase = echo are given as follows: P s a 1 a 2 s = 1 in the following situations: P s a 1 a 2 s = 1 in the following situations: s = test 0 * * * * 0 * * * 0 * and s = test 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 5.4.3. The reward function. We now describe the reward function for D. The reward R s a 1 a 2 s given when transitioning from state s to state s taking action pair a 1 a 2 is −1 in any situation except those situations matching one of the following patterns. Roughly, we give zero reward for waiting during select and generate, for answering with a tile type during query, for echoing a bit consistent with any remembered information during echo, and for having given tile types satisfying the relevant constraints during test. The relevant constraints during test are determined by the rel-pos state component computed by the FSA during the echo phase. s = echo k o q i 2 v 2 0 t 1 i 1 v 1 0 t 2 b 1 = a 1 if a 1 ∈ 0 1 0 otherwise b 2 = a 2 if a 2 ∈ 0 1 0 otherwise s = p k o FSANEXT q b 1 b 2 i 2 v 2 0 t 1 i 1 v 1 0 t 2 where p k = echo k + 1 for 0 ≤ k < 2 log n − 1 test 0 for k = 2 log R s a 1 a 2 s = 0 if and only if one of the following holds: The first four component fields of each state description are fully visible to both agents. The last eight state component fields are split into two groups of four, each group visible to only one agent. We therefore take the agent 1 observations 1 to be partial assignments to the following state variables: phase, index, origin, rel-pos, index-2, value-2, pos-bit-1, and tile-sel-1. Similarly, the observations 2 are partial assignments to the following state variables: phase, index, origin, rel-pos, index-1, value-1, pos-bit-2, and tile-sel-2. The observation distribution O s a 1 a 2 s o 1 o 2 simply reveals the indicated portion of the just-reached state s to each agent deterministically. We say that an observation sequence is p-phase if the sequence matches the pattern * p * * * * * * * , where the first "*" stands for any observation sequence. Here, p can be any of gen, query, echo, or test. We take the horizon T to be 4 log n + 4, because the process spends one step in each of the select, query, and test phases, 2 log n + 1 steps in the generate phase, and 2 log n steps in the echo phase. We take the threshold value K to be 0. This completes the construction of the DEC-MDP by polynomial-time reduction from the selected tiling instance. An example of a zero-reward trajectory of the process is shown in Figure  3 . We now turn to correctness.
 Formal correctness argument. Next we show that the reduction presented above is indeed correct. Our main claim is that there exists a policy that achieves expected total reward zero at the start state if and only if there is a solution to the tiling problem we started with. To make our notation easier to read, we define the following abbreviations. Definition. Given an observation sequence ō1 over 1 , we write loc 1 ō1 for the location value represented by the bits transmitted to agent 1 in the generate phase of the process. We note that during the select and generate phases this value may be only partially specified (because not all of the bits have been generated). More precisely, loc 1 ō1 = b k • • • b 0 , where the b i values are chosen by the first match of the following sequence in ō1 (with k as large as possible while allowing a match): gen 1 * * * * b 0 * • • • gen k + 1 * * * * b k * We define loc 2 ō2 similarly. In addition, we define bit i l to be b i , where b k b 0 is the binary coding of the (possibly only partially specified) location l-we take bit i l to be undefined if the bit i is not specified in location l. By abuse of notation we treat loc 1 ō1 (or loc 2 ō2 ) as a tiling location x y sometimes (but only when it is fully specified) and as a bit string at other times. The easier direction of correctness is stated in the following lemma, which is formally proven in Appendix B. Lemma 1. If there exists a consistent tiling, then there must exist a policy achieving zero expected total reward. phase = gen index = 0 index-2 = 1 value-2 = 1 index-1 = 2 value-1 = 1 phase = gen index = 1 pos-bit-1 = 1 index-2 = 1 value-2 = 1 pos-bit-2 = 0 index-1 = 2 value-1 = 1 phase = gen index = 2 pos-bit-1 = 0 index-2 = 1 value-2 = 1 pos-bit-2 = 1 index-1 = 2 value-1 = 1 phase = gen index = 3 pos-bit-1 = 1 index-2 = 1 value-2 = 1 pos-bit-2 = 1 index-1 = 2 value-1 = 1 phase = gen index = 4 pos-bit-1 = 0 index-2 = 1 value-2 = 1 pos-bit-2 = 0 index-1 = 2 value-1 = 1 phase = query index-2 = 1 value-2 = 1 index-1 = 2 value-1 = 1 phase = echo index = 0 origin = yes rel-pos = q 0 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 phase = echo index = 1 origin = no rel-pos = q 1 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 phase = select phase = echo index = 2 origin = no rel-pos = q 2 phase = echo index = 3 origin = no rel-pos = q 3 phase = test origin = no rel-pos = q 4 ∈ hor tile-sel-1 = tile-0 tile-sel-2 = tile-1 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 1 1 1 2 1 0 1 1 0 0 0 1
 Select The process chooses indices to be checked in the echo phase.
 Generate The process generates address bits.
 Query The agents choose tiles.
 Echo The agents echo address bits. The process checks one bit for each agent and keeps track of information about the addresses echoed.
 Test The process checks that the relevant constraints are satisfied. phase = test 0 1 Figure  3 . An example of a zero-reward trajectory of the process constructed from the tiling example given in Figure  2 . The total reward is zero because the agents echo the "checked" bits correctly and choose tiles that do not violate any constraints, given the two addresses that are echoed. (For clarity, some state components are not shown.) We now discuss the more difficult reverse direction of the correctness proof. In the following subsections, we prove Claims 1 to 3 from §5.3 to show that if there is a policy which achieves nonnegative expected reward for horizon T , then there is also a consistent tiling. Throughout the remainder of the proof, we focus on a fixed, arbitrary policy that achieves zero expected reward. Given this policy, we must show that there is a consistent tiling. 5.5.1. Proof of Claim 1. Before proving the first claim, we need to formalize the notion of "faithfulness during echo." Definition. A pair of observation sequences ō1 ō2 over 1 and 2 , respectively, is said to be reachable if P s 0 ō1 ō2 s is nonzero for some state s. An observation sequence ō1 over 1 is said to be reachable if there exists an observation sequence ō2 over 2 such that the pair of observation sequences ō1 ō2 is reachable. Likewise, ō2 over 2 is reachable if there is some ō1 over 1 such that, ō1 ō2 is reachable. Definition. The policy = 1 2 is faithful during echo if it satisfies both of the following conditions for all indices k in 0 2 log n − 1 , and all reachable observation sequence pairs ō1 ō2 : 1 Much of our proof revolves around showing that the reachability of a pair of observation sequences is not affected by making certain changes to the sequences. We focus without loss of generality on changes to the observations of agent 2, but similar results hold for agent 1. The changes of particular interest are changes to the (randomly selected) value of the index-1 state component-this is the component that remembers which bit of agent 1's queried location will be checked during echo. It is important to show that agent 1 cannot determine which bit is being checked before that bit has to be echoed. To show this, we define a way to vary the observation sequences seen by agent 2 (preserving reachability) such that without changing the observations seen by agent 1 we have changed which agent 1 address bit is being checked. We now present this approach formally. Definition. We say that an observation sequence ō1 over 1 is superficially consistent if the values of the index-2 component and the value-2 component do not change throughout the sequence, and the value of the tile-sel-1 component is tile-0 for generate-phase and query-phase observations and some fixed tile type in L for echo-phase and test-phase observations. Given a superficially consistent observation sequence ō1 , we can write index-2 ō1 and value-2 ō1 to denote the value of the indicated component throughout the sequence. In addition, we can write tile-sel-1 ō1 to denote the fixed tile type for echo-phase and testphase observations (we take tile-sel-1 ō1 to be tile-0 if the sequence contains no echo-or test-phase observations). Corresponding definitions hold for observation sequences over 2 , replacing "1" by "2" and "2" by "1" throughout. Note that any reachable observation sequence must be superficially consistent, but the converse is not necessarily true. The following technical definition is necessary so that we can discuss the relationships between observation sequences without assuming reachability. Definition. We say that two superficially consistent observation sequences ō1 over 1 and ō2 over 2 are compatible if bit index-1 ō2 loc 1 ō1 = value-1 ō2 or this bit of loc 1 ō1 is not defined, and bit index-2 ō1 loc 2 ō2 = value-2 ō1 or this bit of loc 2 ō2 is not defined. Definition. Given an index i in 0 2 log n − 1 , a reachable pair of observation sequences ō1 ō2 , and an observation sequence ō 2 over 2 , we say that ō 2 is an i-index variant of ō2 relative to ō1 when ō 2 is any sequence compatible with ō1 that varies from ō2 only as follows: 1. index-1 has been set to i throughout the sequence, 2. value-1 has been set to the same value v throughout the sequence, 3. pos-bit-2 can vary arbitrarily from ō2 , and 4. For any echo-or test-phase observations, tile-sel-2 has been set to the tile type selected by on the query-phase prefix of ō 2 , or to tile-0 if selects a non-tile-type action on that query. If the pos-bit-2 components of ō2 and ō 2 are identical, we say that ō 2 is a same-address index variant of ō2 . We note that, given a reachable pair of observation sequences ō1 ō2 , there exists an iindex variant of ō2 relative to ō1 , for any i in 0 2 log n − 1 . This remains true even if we allow only same-address index variants. The following technical lemma asserts that index variation as just defined preserves reachability under very general conditions. Its proof is deferred to Appendix C. Lemma 2. Suppose is faithful during echo for the first k bits of the echo phase for some k. Let ō1 ō2 be a reachable pair of observation sequences that end no later than the kth bit of echo (i.e., the last observation in the sequence has index no greater than k if it is an echo-phase observation) and let ō 2 be an i-index variant of ō2 relative to ō1 for some i. If the observation sequences are echo-phase or test-phase, then we require that the index variation be a same-address variation. We can then conclude that ō1 ō 2 is reachable. We are now ready to assert and prove Claim 1 from §5.3.
 Lemma 3 (Claim 1). is faithful during echo. Proof. We argue by induction that faithfully echoes all 2 log n address bits. As an inductive hypothesis, we assume that faithfully echoes the first k bits, where 0 ≤ k < 2 log n. Note that if k equals zero, this is a null assumption, providing an implicit base case to our induction. Now suppose for contradiction that lies during the k + 1st step of the echo phase. Then one of the agents' policies must incorrectly echo bit k + 1; we assume without loss of generality that this is so for agent 1, i.e., under some reachable observation sequence pair ō1 ō2 of length 2 log n + k + 2, the policy 1 dictates that the agent choose action 1 − bit k loc 1 ō1 . Lemma 2 implies that the observation sequence pair ō1 ō 2 is also reachable, where ō 2 is any same-address k + 1-index variant of ō2 relative to ō1 . Because all the agent 1 observations are the same for both ō1 ō2 and ō1 ō 2 , when the latter sequence occurs, agent 1 chooses the same action 1−bit k loc 1 ō1 as given above for the former sequence, and a reward of −1 is obtained (because in this case it is bit k + 1 that is checked). Therefore, the expected total reward is not zero, yielding a contradiction. 5.5.2. Proof of Claim 2. Now we move on to prove Claim 2 from §5.3. We show that can be used to define a particular mapping from tile locations to tile types based on "dangerous queries." In §5.3, we defined an agent 1 observation sequence to be "dangerous" if it reveals a bit of agent 2's queried location that agrees with the corresponding bit of agent 1's queried location (and vice versa for agent 2 observation sequences). We now present this definition more formally. Definition. A query-phase observation sequence ō1 over 1 is dangerous if it is reachable and bit index-2 ō1 loc 1 ō1 = value-2 ō1 Likewise, a query-phase sequence ō2 over 2 is dangerous if it is reachable and bit index-1 ō2 loc 2 ō2 = value-1 ō2 . Dangerous query-phase sequences are those for which the agent's observations are consistent with the possibility that the other agent has been queried on the same location. We note that for any desired query location l, and for either agent, there exist dangerous observation sequences ō such that loc k ō = l. Moreover, such sequences still exist when we also require that the value of index-k ō be any particular desired value (where k is the number of the nonobserving agent). Lemma 4. Two same-length query-phase observation sequences, ō1 over 1 and ō2 over 2 , are reachable together as a pair if and only if they are compatible and each is individually reachable. Proof. The "only if" direction of the theorem follows easily-the reachability part follows from the definition of reachability, and the compatibility of jointly reachable sequences follows by a simple induction on sequence length given the design of D. The "if" direction can be shown based on the following assertions. First, a generate-phase observation sequence (for either agent) is reachable if and only if it matches the following pattern: gen 0 yes q 0 i v * tile-0 • • • gen k yes q 0 i v * tile-0 for some k i, and v; this can be established by a simple induction on sequence length based on the design of D. A similar pattern applies to the query phase. Given two compatible reachable sequences of the same length, ō1 and ō2 , we know by the definition of reachability that there must be some sequence ō 2 such that ō1 ō 2 is reachable. But given the patterns just shown for reachable sequences, ō2 and ō 2 can differ only in their choice of i, v, and in the address given to agent 2 via the pos-bit-2 component. It follows that ō2 is an i-index variant of ō 2 relative to ō1 , for some i. Lemma 2 then implies that the pair ō1 ō2 is reachable as desired. Lemma 5 (Claim 2). There exists a mapping f from tiling locations to tile types such that f loc i ō = i ō on all dangerous queries ō over i for both agents (i ∈ 1 2 ). Proof. To prove the lemma, we prove that for any two dangerous query sequences ōi and ōj over i and j , respectively, for arbitrary i j ∈ 1 2 , if loc i ōi = loc j ōj = l, then i ōi = j ōj . This implies that for any such ōi we can take f l = i loc i ōi to construct f satisfying the lemma. Suppose not. Then there must be a counterexample for which i = j-because given a counterexample for which i = j, either ōi or ōj must form a counterexample with any dangerous query ōk over 1−i such that loc 1−i ōk = l. We can now consider a counterexample where i = j. Let ō1 and ō2 be dangerous (and thus reachable) sequences over 1 and 2 , respectively, such that loc 1 ō1 = loc 2 ō2 but 1 ō1 = 2 ō2 . Note that loc 1 ō1 = loc 2 ō2 together with the fact that ō1 and ō2 are dangerous implies that ō1 and ō2 are compatible and thus reachable together (using Lemma 4). The faithfulness of echo under (proven in Claim 1, Lemma 3) then ensures that the extension (there is a single extension because D is deterministic in the echo and test phases) of these observation sequences by following to the test phase involves a faithful echo. The correctness of the FSA construction in Appendix A then ensures that the rel-pos state component after this extension will have the value equal. The reward structure of D during the test phase then ensures that to avoid a negative reward the tile types given during query, 1 ō1 and 2 ō2 , must be the same, contradicting our choice of ō1 and ō2 above and thus entailing the lemma. 5.5.3. Proof of Claim 3 and our main hardness theorem. We now finish the proof of our main theorem by proving Claim 3 from §5.3. We start by showing the existence of a useful class of pairs of dangerous observation sequences that are reachable together. Lemma 6. Given any two locations l 1 and l 2 sharing a single bit in their binary representations, there are dangerous observation sequences ō1 over 1 and ō2 over 2 such that: loc 1 ō1 = l 1 loc 2 ō2 = l 2 and ō1 ō2 is reachable Proof. It is straightforward to show that there exist dangerous observation sequences ō1 over 1 and ō2 over 2 such that loc 1 ō1 = l 1 and loc 2 ō2 = l 2 as desired. In these sequences, both index-1 and index-2 are set throughout to the index of a single bit shared by l 1 and l 2 . Because this bit is in common, these sequences are compatible, so by Lemma 4 they are reachable together. Lemma 7 (Claim 3). The mapping f defined in Lemma 5 is a consistent tiling. Proof. We prove the contrapositive. If the mapping f is not a consistent tiling, then there must be some particular constraint violated by f . It is easy to show that any such constraint is tested during the test phase if loc 1 ō1 and loc 2 ō2 have the appropriate values. (The faithfulness during echo claim proven in Lemma 3 implies that the origin and rel-pos components on entry to the test phase will have the correct values for comparing the two locations). For example, if a horizontal constraint fails for f , then there must be locations i j and i + 1 j such that the tile types f i j f i + 1 j are not in H ; because these two locations share a bit (in fact, all the bits in j, at least), Lemma 6 implies that there are dangerous ō1 and ō2 with loc 1 ō1 = i j and loc 2 ō2 = i + 1 j that are reachable together. During the test phase, the tile-sel-1 and tile-sel-2 state components are easily shown to be f i j and f i + 1 j , and then the definition of the reward function for D ensures a reachable negative reward. The arguments for the other constraints are similar. Claim 3 immediately implies that there exists a consistent tiling whenever there exists a policy achieving zero expected total reward. This completes the proof of the other direction of our main complexity result. We have thus shown that there exists a policy that achieves expected reward zero if and only if there exists a consistent tiling, demonstrating that DEC-MDP 2 is NEXP-hard. Theorem 2. DEC-MDP 2 is NEXP-hard. Corollary 1. For all m ≥ 2, both DEC-POMDP m and DEC-MDP m are NEXPcomplete. 6. Discussion. Using the tools of worst-case complexity analysis, we analyzed two variants of decentralized control of Markov decision processes. Specifically, we proved that the finite-horizon m-agent DEC-POMDP problem is NEXP-hard for m ≥ 2 and the finitehorizon m-agent DEC-MDP problem is also NEXP-hard for m ≥ 2. When the horizon is limited to be less than the number of states, the problems are NEXP-complete. The results have some theoretical implications. First, unlike the MDP and POMDP problems, the problems we studied provably do not admit polynomial-time algorithms, because P = NEXP. Second, we have drawn a connection between work on Markov decision processes and the body of work in complexity theory that deals with the exponential jump in complexity due to decentralization  (Peterson and Reif 1979, Babai et al. 1991) . There are also more direct implications for researchers trying to solve problems of this nature. Consider the growing body of work on algorithms for obtaining exact or approximate solutions for POMDPs (e.g.,  Jaakkola et al. 1995 , Cassandra et al. 1997 , Hansen 1998 , Meuleau et al. 1999 , Lusena et al. 1999 , Zhang 2001 . For the finite-horizon case, we now have stronger evidence that there is no way to efficiently convert a DEC-MDP or DEC-POMDP into an equivalent POMDP and solve it using established techniques. This knowledge can provide direction for research on the development of algorithms for these problems. Finally, consider the infinite-horizon versions of the aforementioned problems. It has recently been shown that the infinite-horizon POMDP problem is undecidable  (Madani et al. 1999 ) under several different optimality criteria. Because a POMDP is a special case of a DEC-POMDP, the corresponding infinite-horizon DEC-POMDP problems are also sets of reject states. From these sets we construct distinguished sets apart, equal, hor, and ver of cross-product automaton states as follows: apart = reject 1 × reject 2 × reject 3 equal = accept 1 × reject 2 × reject 3 hor = reject 1 × accept 2 × reject 3 ver = reject 1 × reject 2 × accept 3 The rest of the automaton's states comprise the set Q . Let q 1 0 , q 2 0 , and q 3 0 denote the start states of the three component automata. We define the start state of the cross-product automaton to be the state q 0 = q 1 0 q 2 0 q 3 0 . We now define two functions based on this automaton that are needed in the main body of the proof. One function takes as input the state of the automaton and a bit pair and returns the next state of the automaton. The second function takes as input a pair of bit strings of the same length and returns the state that the automaton will be in starting from its initial state and reading symbols formed by the corresponding bits in the two strings in sequence. Definition. For q ∈ Q and a 1 a 2 ∈ 0 1 , FSANEXT q a 1 a 2 = q , where q ∈ Q is the resulting state if the automaton starts in state q and reads the input symbol a 1 a 2 . Definition. The function FSA is defined inductively as follows: FSA = q 0 FSA b 0 • • • b k+1 c 0 • • • c k+1 = FSANEXT FSA b 0 • • • b k c 0 • • • c k b k+1 c k+1 Note that the range of FSA for inputs of length 2 log n is apart ∪ equal ∪ hor ∪ ver. In the proof given in §5 we assumed that the TILING grid size n was an exact power of two. We note that the proof can be adapted by adding two components to the crossproduct FSA described here, where the two new components are both FSAs over the same alphabet. The first new component accepts a string only when both x 1 and y 1 (as described above) are less than n (so that the tiling location represented by x 1 y 1 is in the tiling grid). The second new component behaves similarly for x 2 y 2 . The DEC-MDP can then be constructed using the smallest power of two larger than n but modified so that whenever either new component of the FSA rejects the (faithfully) echoed bit sequences, then the process gives a zero reward, regardless of the tile types returned during query. Each new component can be viewed as an FSA over a {0,1} alphabet, because each focuses either on just the agent 1 echoes or on just the agent 2 echoes. We describe the FSA for checking that x 1 is less than n; constructing the two components is then straightforward. Suppose that k = log n is the number of bits in the binary representation of n and that the bits themselves are given from least to most significant as b 1 • • • b k . Suppose also that there are j different bits equal to 1 among b 1 • • • b k and that these bits are at indices i 1 i j . We can then write a regular expression for detecting that its input of k bits from least to most significant represents a number in binary that is strictly less than n: 0 + 1 i 1 −1 0 b i 1 +1 • • • b k + 0 + 1 i 2 −1 0 b i 2 +1 • • • b k + • • • + 0 + 1 i j −1 0 b i j +1 • • • b k It can be shown that this regular expression has an equivalent FSA of size O log n 2 . Appendix B: Proof of Lemma 1. We assume there exists at least one consistent tiling, and we select a particular such mapping f . We describe a policy = 1 2 that achieves zero expected reward at the initial state. 1 is a mapping from sequences of observations in part of the mapping 1 is specified below-any unspecified observation sequence maps to the action wait. We note that 1 and 2 are symmetric. The local policy 2 is defined identically to 1 except that loc 1 is replaced by loc 2 . We first characterize the set of all reachable states from s 0 under the policy . We then note that taking the action prescribed by from any of these states yields a reward of zero. Thus, V T s 0 = 0. It is straightforward to show by induction that P s 0 ō1 ō2 s is zero except where one of the following patterns applies: • s = s 0 = select 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 . • s = gen k yes q 0 i 2 v 2 * tile-0 i 1 v 1 * tile-0 , where 0 ≤ k ≤ 2 log n, k ≤ i 1 or v 1 = bit i 1 loc 1 ō1 , and k ≤ i 2 or v 2 = bit i 2 loc 2 ō2 . • s = query 0 yes q 0 i 2 v 2 * tile-0 i 1 v 1 * tile-0 , where v 1 = bit i 1 loc 1 ō1 , and v 2 = bit i 2 loc 2 ō2 . • s = echo k o q i 2 v 2 * t 1 i 1 v 1 * t 2 , where 0 ≤ k ≤ 2 log n − 1, v 1 = bit i 1 loc 1 ō1 , v 2 = bit i 2 loc 2 ō2 , t 1 = f loc 1 ō1 , t 2 = f loc 2 ō2 , o = yes if and only if b j = c j = 0 for 0 ≤ j ≤ k − 1, with b j and c j as in the next item, and q = FSA b 0 • • • b k−1 c 0 • • • c k−1 , with b 0 • • • b k−1 and c 0 • • • c k−1 the least significant bits of loc 1 ō1 and loc 2 ō2 , respectively. • s = test * o r * * * t 1 * * * t 2 , where o = yes if and only if loc 1 ō1 = 0 0 and loc 2 ō2 = 0 0 , r = FSA loc 1 ō1 loc 2 ō2 , t 1 = f loc 1 ō1 , and t 2 = f loc 2 ō2 . It can then be shown that the reward for any action prescribed by the policy given any of these reachable state/observation sequence combinations is zero given that f is a consistent tiling. Appendix C: Proof of Lemma 2. We need some new notation to carry out this proof. Given a state s, a state component c and corresponding value v from the domain of c, we define the state "s with c set to v" (written s c = v ) to be the state s that agrees with s at all state components except possibly c and has value v for state component c. We also write ō1 j for the first j observations in the sequence ō1 , and likewise ō2 j and ō 2 j . 839 For any state s j reachable while observing ō1 j ō2 j , we define a state s j that we will show is reachable while observing ō1 j ō 2 j , as follows: s j = s j index-1 = index-1 ō 2 j value-1 = value-1 ō 2 j tile-sel-2 = tile-sel-2 ō 2 j We can now show by an induction on sequence length j that for any state s j such that P s 0 ō1 j ō2 j s j is nonzero, then P s 0 ō1 j ō 2 j s j is also nonzero. From this we can conclude that ō1 ō 2 is reachable, as desired. For the base case of this induction, we take j to be 1, so that the observation sequences involved all have length 1, ending in the generate phase with index equal to zero. Inspection of the definition of the transition probabilities P shows that changing index-1 and value-1 arbitrarily has no effect on reachability. For the inductive case, we suppose some state s j is reachable by ō1 j ō2 j , and that state s j is reachable by ō1 j ō 2 j . Let a 1 be ō1 j , a 2 be ō2 j , and a 2 be ō 2 j . We must show that for any state s j+1 such that P s j a 1 a 2 s j+1 is nonzero, P s j a 1 a 2 s j+1 is also nonzero, for s j+1 . This follows from the following observations: • When phase(s j ) is select or generate, neither agent 2's action a 2 nor the values of index-1(s j ) or value-1(s j ) have any affect on P s j a 1 a 2 s j+1 being nonzero, as long as either index-1 is not equal to j or pos-bit-1(s j+1 ) equals value-1(s j ). However, this last condition is ensured to hold of the index-1 and value-1 components of s j and s j by the compatibility of ō 2 with ō1 . • When phase(s j ) is query, the action a 2 must equal the tile-sel-2 state component of s j+1 by the definitions of s j+1 and "index variant," and changes to the index-1 and value-1 components have no effect on P s j a 1 a 2 s j+1 being nonzero during the query phase. • When phase(s j ) is echo, the actions a 2 and a 2 must be a faithful echo of the location address bit indicated by the index state component (because we have assumed as part of out inductive hypothesis that is faithful during echo for at least j bits), and this bit's value does not vary between ō2 and ō 2 because if the observation sequences reach the echo phase we have the assumption that these are same-address variants. Thus a 2 = a 2 during echo. Again, changes to the index-1 and value-1 components have no effect on P s j a 1 a 2 s j+1 being nonzero during the echo phase. Figure 1 . 1 Figure 1. The relationships among the models.
 4.4Observable to both agents: phase ∈ select gen query echo test Current phase of the process index ∈ 0 2 log n Index of next location bit to be generated/echoed origin ∈ yes no Eventually true if both tile locations are 0 0 rel-pos ∈ QRelative tile positions during echo-controlled by the FSA Observable only to agent 1: index-2 ∈ 0 2 log n − 1 Index of bit remembered for agent 2 value-2 ∈ 0 1Value of bit remembered for agent 2 pos-bit-1 ∈ 0 1Bit for transmitting tile position to agent 1 tile-sel-1 ∈ L Tile type selected by agent 1 in query Observable only to agent 2: index-1 ∈ 0 2 log n − 1 Index of bit remembered for agent 1 value-1 ∈ 0 1Value of bit remembered for agent 1 pos-bit-2 ∈ 0 1 Bit for transmitting tile position to agent 2 tile-sel-2 ∈ L Tile type selected by agent 2 in query
 n − 1 and o = yes if and only if o = yes and a 1 = a 2 = 0 Test phase. The test phase consists of just one step terminating the process in a zeroreward absorbing state.
 Select phase s = select * * * * * * * * * * * and a 1 = a 2 = wait Generate phase s = gen * * * * * * * * * * * and a 1 = a 2 = wait Query phase s = query * * * * * * * * * * * and both a 1 ∈ L and a 2 ∈ L Echo phase s = echo k * * i 2 v 2 * * i 1 v 1 * * and a 1 a 2 ∈ 0 1 where a 1 = v 1 or k = i 1 and a 2 = v 2 or k = i 2 830 D. S. BERNSTEIN, R. GIVAN, N. IMMERMAN, AND S. ZILBERSTEIN Test Phase i s = test * o equal * * * t 1 * * * t 1 and a 1 = a 2 = wait where o = no or t 1 = tile-0 Test Phase ii s = test * * hor * * * t 1 * * * t 2 and a 1 = a 2 = wait where t 1 t 2 ∈ H Test Phase iii s = test * * ver * * * t 1 * * * t 2 and a 1 = a 2 = wait where t 1 t 2 ∈ V Test Phase iv s = test * * apart * * * * * * * * and a 1 = a 2 = wait 5.4.4. Observations, threshold, and horizon.
 . 1 ō1 = bit k loc 1 ō1 when ō1 = * * * * * * * * • • • echo k * * * * * * . 2. 2 ō2 = bit k loc 2 ō2 when ō2 = * * * * * * * * • • • echo k * * * * * * . We say the policy lies during echo otherwise. If the two conditions listed above are satisfied for all indices k in 0 d −1 , where 0 ≤ d ≤ 2 log n, we say that the policy faithfully echoes the first d bits.
 1 ō1 = a 1 when one of the following holds: Select phase: ō1 = select * * * * * * * and a 1 = wait Generate phase: ō1 = * gen * * * * * * * and a 1 = wait Query phase: ō1 = * query * * * * * * * and a 1 = f loc 1 ō1 Echo phase: ō1 = * echo k * * * * * * and a 1 = bit k loc 1 ō1 Test phase: ō1 = * test * * * * * * * and a 1 = wait
			 D. S. BERNSTEIN, R. GIVAN, N. IMMERMAN, AND S. ZILBERSTEIN
			 to actions in A 1 , and 2 from sequences over 2 to actions in A 2 . Only the reachable


Abstract: We look at the strengths and weaknesses of two existing definitions of existential risk, and suggest a new definition based on expected value. This leads to a parallel concept: 'existential hope', the chance of something extremely good happening. An existential risk is a chance of a terrible event occurring, such as an asteroid striking the earth and wiping out intelligent life -we could call such events existential catastrophes. In order to understand what should be thought of as an existential risk, it is necessary to understand what should be thought of as an existential catastrophe. This is harder than it first seems to pin down.

Existential Risk and Existential Hope: Definitions

The simple definition One fairly crisp approach is to draw the line at extinction: Definition (i): An existential catastrophe is an event which causes the end of existence of our descendants. This has the virtue that it is a natural division and is easy to understand. And we certainly want to include all extinction events. But perhaps it doesn't cast a wide enough net. Example A: A totalitarian regime takes control of earth. It uses mass surveillance to prevent any rebellion, and there is no chance for escape. This regime persists for thousands of years, eventually collapsing when a supervolcano throws up enough ash that agriculture is prevented for decades, and no humans survive. In Example A, clearly the eruption was bad, but the worst of the damage was done earlier. After the totalitarian regime was locked in, it was only a matter of time until something or other finished things off. We'd like to be able to talk about entering this regime as the existential catastrophe, rather than whatever event happens to end it. So we need another definition. Although we'll now look at other definitions for existential catastrophes, we do like the simple definition. Luckily there's another term that's already understood: human extinction. Sometimes it's better to talk about extinction risks rather than existential risks, as 'existential risk' is a piece of jargon, whereas 'extinction risk' will be clear to everyone.
 Bostrom's definition Nick Bostrom introduced the concept of existential risks. He has defined them as follows: Definition (ii): An existential risk is one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development.  1  This definition deals well with Example A, placing the existential catastrophe at the point where the totalitarian regime arose, as this caused the permanent and drastic destruction of [humanity's] potential for desirable future development. Example B: A totalitarian regime takes control of the earth. There is only a slight chance that humanity will ever escape. Is this an existential catastrophe? Bostrom's definition doesn't clearly specify whether it should be considered as one. Either answer leads to some strange conclusions. Saying it's not an existential catastrophe seems wrong as it's exactly the kind of thing that we should strive to avoid for the same reasons we wish to avoid existential catastrophes. Saying it is an existential catastrophe is very odd if humanity does escape and recover -then the loss of potential wasn't permanent after all. The problem here is that potential isn't binary. Entering the regime certainly seems to curtail the potential, but not to eliminate it.
 Definition via expectations The idea that potential isn't binary motivates our suggested definition: Definition (iii): An existential catastrophe is an event which causes the loss of a large fraction of expected value. This definition deals well with Example B. If we enter into the totalitarian regime and then at a later date the hope of escape is snuffed out, that represents two existential catastrophes under this definition. We lost most of the expected value when we entered the regime, and then lost most of the remaining expected value when the chance for escape disappeared. A lot of the work of this definition is being done by the final couple of words. 'Value' refers simply to whatever it is we care about and want in the world, in the same way that 'desirable future development' worked in Bostrom's definition. And to talk about expectations we need to have some probabilities in mind. Here we are thinking of objective probabilities. Note that 'potential' in Bostrom's definition requires some similar work to making assumptions about probabilities.
 Existential eucatastrophes and existential hope If we enter the totalitarian regime and then manage to escape and recover, then we had an existential catastrophe which was balanced out by a subsequent gain in expected value. This kind of event gives us a concept parallel to that of an existential catastrophe: Definition (iv): An existential eucatastrophe 2 is an event which causes there to be much more expected value after the event than before. This concept is quite natural. We saw it in the context of escape from a regime which threatened the existence of a prosperous future. Our world has probably already seen at least one existential eucatastrophe: the origin of life. When life first arose, the expected value of the planet's future may have become much bigger. To the extent that they were not inevitable, the rise of multicellular life and intelligence may also have represented existential eucatastrophes. In general successfully passing any 'great filter' 3 is an existential eucatastrophe, since beforehand the probability of passing it is small, so the expected value is much smaller than after the filter is dealt with. Armed with this concept, we can draw a new lesson. Just as we should strive to avoid existential catastrophes, we should also seek existential eucatastrophes. In some ways, this isn't a new lesson at all. Under Bostrom's definition we are comparing ourselves to the most optimistic potential we could reach, so failing to achieve a eucatastrophe is itself a catastrophe. However we think more naturally in terms of events than non-events. If life fails to arise on a planet where it might have, it's much clearer to think of a failure to achieve a eucatastrophe than of an existential catastrophe stretching out over the billions of years in which life did not arise. Just as we tend to talk about the existential risk rather than existential catastrophe, we want to be able to refer to the chance of an existential eucatastrophe; upside risk on a large scale. We could call such a chance an existential hope. In fact, there are already people following both of the strategies this suggests. Some people are trying to identify and avert specific threats to our future -reducing existential risk. Others are trying to steer us towards a world where we are robustly well-prepared to face whatever obstacles come -they are seeking to increase existential hope.
 Conclusions We were interested in pinning down what is meant by 'existential risk'. Much of the time, all of the definitions we've looked at will agree on whether something is an existential risk. Keeping it simple can be good, because it helps more people to understand. We therefore advocate talking about 'extinction risks' rather than 'existential risks' when the former term will work. Nonetheless, we may sometimes have to consider more unusual scenarios. It's good to know how to make the definition work well there as it can help us to think about things more clearly. We think that the definition in terms of expectations does a better job of this than previous definitions. In devising the notion of existential catastrophe (and hence existential risk) via expectations, we came across the dual concept we have called 'existential eucatastrophe' (and hence 'existential hope'). We think this captures a natural class of events, and what may be an important one. We hope that having a label for the concept may help others to make better judgements about what courses to pursue. 			 * Future of Humanity Institute, University of Oxford & Centre for Effective Altruism † Future of Humanity Institute, University of Oxford
			 Bostrom, Existential Risk Reduction as Global Priority, Global Policy, Vol 4, Issue 1 (2013), p15
			 The word 'eucatastrophe' is made of the Greek root 'eu-' meaning 'good' and the word 'catastrophe' in its classical sense of a sudden turn. It was coined by Tolkien to refer to the sudden and unexpected turn for the better frequently found at the end of fairy tales. (Tolkien, John Ronald Reuel. On fairy-stories. Oxford University Press, 1947.)3 Hanson, Robin, The Great Filter -Are We Almost Past It?, 1998.


Abstract: The prevalence and success of AI applications have been tempered by concerns about the controllability of AI systems about AI's impact on the future of work. These concerns reflect two aspects of a central question: how would humans work with AI systems? While research on AI safety focuses on designing AI systems that allow humans to safely instruct and control AI systems, research on AI and the future of work focuses on the impact of AI on humans who may be unable to do so. This Blue Sky Ideas paper proposes a unifying set of declarative principles that enable a more uniform evaluation of arbitrary AI systems along multiple dimensions of the extent to which they are suitable for use by specific classes of human operators. It leverages recent AI research and the unique strengths of the field to develop human-centric principles for AI systems that address the concerns noted above.

Unifying Principles and Metrics for Safe and Assistive AI

Introduction Recent years have witnessed immense progress in research on safe and assistive AI systems as well as on the potential impact of AI on the future of work. These directions of research address two sides of a common, fundamental concern: how would humans work with AI systems? While research on AI safety focuses on designing AI systems that allow humans to safely instruct and control them (e.g.,  (Russell, Dewey, and Tegmark 2015; Zilberstein 2015; Hadfield-Menell et al. 2016; Russell 2017; Hadfield-Menell et al. 2017 )), research on AI and the future of work focuses on the impact of AI on members of the workforce who may be unable to do so  (Arntz, Gregory, and Zierahn 2016; Manyika et al. 2017; Nedelkoska and Quintini 2018) . This paper presents the view that in addition to the productive streams of research outlined above, we need unifying metrics and declarative objectives that would allow a more uniform evaluation of AI systems on the extent to which an AI system is suitable for working with specific classes of human operators. It also presents a common principle for human-centered AI systems that allows the development of such metrics. Consequently, rather than proposing a specific new design for AI systems, the focus of this paper Copyright c 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. is on elucidating the declarative principles and types of metrics that would lead to concerted progress on the problem. The advantage of this declarative approach to framing the problem is that it enables an assessment of progress independent of the internal design being used in an AI system, and it will help draw out the strengths and weaknesses of different design approaches. Without such a specification, design differences can make solution paradigms difficult to compare. E.g., one might develop a complex system architecture that builds user profiles and provides appropriate assistance. This system would have very different input requirements and design and performance parameters than a formulation that addresses the same problem by computing assistance policies using planning under partial observability while incorporating the value of information to learn more about a user's poorly articulated objectives and constraints. Better declarative objectives and metrics for assistive AI systems would also help ensure that, regardless of the methods being used, progress amounts to advancement towards safe and assistive AI systems. More pragmatically, such metrics will not only help end-users assess the utility of a given AI system but they will also help AI researchers and developers identify more readily the dimensions along which further research will be beneficial for applications of their interest. The next section presents a succession of intuitive principles for safe and assistive AI systems, and shows that evaluating the compatibility of a system with such principles (in particular P2) helps clarify the required types of metrics. The paper concludes by drawing the attention of our community towards research on the operationalization of such metrics along with promising research directions on developing systems that do well on them.
 Unifying Principles for Safe and Assistive AI Systems We focus on taskable AI systems that carry out user-assigned high-level tasks using arbitrary mechanisms for reasoning and planning over multiple time steps. E.g., household robots that can be given objectives such as setting the table or doing laundry, co-manufacturing robots that can assist workers in creating complex assemblies with heavy components, digital assistants that can plan a vacation given the The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) user's preferences, etc. Such systems serve as sound integrative platforms and model end-to-end applications where the AI system is responsible for assistance in the execution of long-horizon tasks. AI systems are frequently evaluated in terms of performance measures such as the computational complexity of computing the required behavior, training data requirements, and the quality of the computed behavior in terms of execution time, resources used, risk of unsafe outcomes etc. We can consider systems that optimize such performance metrics as Level 0 of assistive AI systems. Level I of assistive AI systems Recent AI research has also focused on assistive properties of AI systems. We begin with a rather common-sensical principle defining Level I of such safe and assistive AI systems: P1: An AI system must make it easy for its operators to use it safely. The italicized terms denote dimensions along which compatibility of AI systems with principle P1 can be evaluated; while a lot of current AI research utilizes one or more of these dimensions for evaluation, a closer analysis reveals some new insights. In the context of this paper we consider using an AI system to be synonymous with instructing it to change its behavior as desired. Different interfaces may be used for this, including programming, text, speech, gestures etc. We consider the operators of an AI system as those persons who use it in the sense described above. For instance, if a self-driving car gives all its passengers the right to give it instructions, then all of them are its operators; if it gives instruction-rights to only a qualified set of users, perhaps adults who pass an assisted-driving exam, then the class of operators is defined by that qualification exam. Safety refers to the overall safety of the AI system's interactions with its environment, which may be physical or online. These dimensions of compatibility with P1 serve as natural dimensions for evaluating Level I assistive AI systems: E1. How inclusive is the set of operators? Systems whose operators require PhD-level expertise in AI may be less desirable for broader deployments. E2. How easy is it for the system's operators to change its behavior? E3. Which set of tasks can the AI system be used for? E4. What form of safety guarantees does the system provide? Systems that are unable to provide an upper bound on expected risks are clearly less desirable than those that stipulate conditions under which upper bounds can be provided. P1 serves to highlight the interplay between these dimensions of compliance and evaluation. Safety guarantees are often inversely related with the size of the operator set. A system may provide a high level of safety, but only under the requirement that its operators take extensive training programs. At one end of the spectrum, automated robot vacuum cleaners require almost no prior skills and perform limited, well-defined tasks. Safety issues are still present-a robot vacuum cleaner may pull on electrical cables that have been left on the floor; auto-completion software may send emails to unintended recipients. However, the lower expected damage from using such applications has made them broadly accepted in society. AI-powered industrial robots are at the other end of the spectrum: these devices require specialized training as well as operating environments in order to ensure safety (see for instance, ISO/TS 15066:2016 on collaborative robots). Typically, these systems operate in regions that humans cannot access when the robot is active unless the human is hand-guiding the robot within a safe operating envelope that limits the speed and range of operation. Their functionality is closely controlled and monitored. Only skilled engineers change their functionality while day-to-day operators monitor execution of predictable behavior and control run-stops from designated safe areas. Similarly, the safety of current airborne drone operations is ensured by requiring specially trained drone operators  (Marshall 2020) . Thus, principle P1 holds for AI systems today with varying degrees of compliance along E1-E4. The examples above illustrate how practical implementations often rely upon implicitly defined operator classes to provide acceptable levels of safety. Such approaches rely upon limiting the users and the scope of a system to achieve an acceptable level of compatibility with P1: it is easy for such systems' users to operate it safely because the set of users is required to be sufficiently skilled, and its functionality is sufficiently limited for that group to be able to safely use the device. However, a broader emphasis on the need to specify safety guarantees with respect to different classes of operators would help mitigate some of the risks associated with broadly deployed AI systems. In contrast to classical automated systems, AI systems feature a more nuanced interplay between the class of tasks that a system can be used for (E3) and the other dimensions above. Traditionally deployed systems (even automated systems) have a very well-defined boundary of use cases. This allows for an easier classification of safety. Taskable AI systems on the other hand, are expected to change their behavior and functionality as they learn and adapt to new environments or new user-given objectives. For such systems, we need better methods for deriving the range of instructions that different operator-groups are allowed to provide. Scenarios like the self-driving car that needs to decide whether to obey a child's instruction allude to this requirement. Methods for assessing user and AI competency can also allow the AI system to expand its range of functionality by drawing upon the expertise of its operator  (Basich et al. 2020)  while ensuring an acceptable level of safety. A major limitation of P1 and its associated metrics is that it does not evaluate the amount of training required for an individual to qualify as an operator for the system. This creates a blind-spot in evaluation of the ease-of-use or safety of an AI system: since user-training occurs outside the requirements outlined by P1, an unsafe AI system (or one that is deployed in an unsafe manner) would simply claim that its so-called operator was insufficiently trained! Furthermore, if P1 were a sufficient categorization of safe and assistive AI systems, we would have no need for explainable AI as compliance with P1 does not require the system to be easy to understand. An implicit emphasis on assessing AI systems only along some aspects of P1 may also explain the increasing prevalence of concerns about the workers who may be left behind in the future workplace. From this perspective it is unsurprising that these concerns have gained renewed interest at a time when AI applications have reached a level of maturity where they are being used by non-AI-experts in situations that have some inherent safety risks. However, the "assistive" nature of such AI systems is undermined by the need for highly skilled individuals who could safely debug, understand and modify the behavior of such systems. Level II of assistive AI systems In order to address the limitations of P1, we consider the following as a guiding principle for safe and assistive AI: P2: An AI system must make it easy for its operators to learn how to use it safely. P2 changes the notion of operators from those who are qualified to use a given AI system to those who are qualified to start learning how to use it. In addition to the metrics associated with P1, P2 introduces a new dimension: E5. How easy is it to learn how to use the AI system? What are the expected prerequisites and costs of training for its operators? Can training be provided on-the-job? This dimension could also be viewed as evaluating the resources required to train operators for P1 systems. Most AI systems in use today would perform poorly on this new dimension, and consequently, on compatibility with P2 as a whole. Explainable AI (e.g.,  (Ribeiro, Singh, and Guestrin 2016; Hayes and Shah 2017; Chakraborti et al. 2017; Hoffman et al. 2018; Gunning and Aha 2019; Weld and Bansal 2019; Anderson et al. 2019; Eifler et al. 2020 )) plays a key role along this dimension because systems that are easy to understand or that can explain themselves naturally make it easier for people to learn how to use them. P2 leverages the unique strengths of AI as a field of research. AI research already addresses the problem of estimating users' skills; research on intelligent tutoring systems and AI for education addresses the problem of identifying skill gaps. This can be used to determine the minimal differential training to be provided to an operator. P2 places the onus of training on the deployed AI system and opens up a new direction of interdisciplinary research connecting existing research directions in AI with research in humansystems engineering and in industrial engineering for the development of productive training modalities and the operationalization of metrics for E5. It also allows AI systems to formally characterize different scopes of functionality for different classes of operators, e.g., operators that use manufacturing robots for pre-determined tasks, those that give the robots new instructions, or those that are ready to learn about giving the robot new instructions. P2 is not required for every AI system-P1 would be sufficient for systems that place minimal requirements on operator qualifications (e.g., robot vacuum cleaners) and for non-adaptive AI systems that require a small set of operators. On the other hand, P2 serves as a better declarative foundation for evaluating taskable AI systems that are meant to assist large numbers of non-AI-experts on a wide range of tasks. Increasing concerns about job roles that would feature a high-degree of interaction with AI systems (and the workers that are likely to be left behind) allude to the pressing need for including E5, a dimension for evaluation under P2 (and not P1) as a part of an AI system's evaluation. AI systems that are not beneficial (either in terms of AI safety or in terms of the future of work) fare poorly on P2. E.g., systems that can thwart their users' objectives by wireheading and those that may derive incorrect objective functions from user instructions make it difficult for an operator to learn how to provide instructions that are specific enough to be safe, and fare poorly along E5. Similarly, systems that require extensive training investment to be used effectively and safely fail along E5. In this way P2 serves as a unifying principle encompassing research on AI safety as well as on AI for a beneficial future of work.
 Promising Directions of Research P2 serves as a declarative principle for guiding research on assistive AI systems as well as for developing metrics for evaluating AI systems and their deployments. Converting this principle into tangible metrics calls for interdisciplinary research including AI and other fields associated with human factors. The increasing prevalence of research thrusts on safe and assistive AI systems  (Fern et al. 2014; Russell, Dewey, and Tegmark 2015; Amodei et al. 2016; Gunning and Aha 2019)  makes this a particularly opportune phase for formalizing the metrics and the interfaces required for evaluating AI systems for compatibility with P2 along dimensions E1-E5. Recent research on AI safety and explainable AI develops methods improving the ease of use and safety of AI systems along P2 (see, for instance, the ICML 2020 Workshop on Explainable AI). Placing AI systems that compute user-skill aligned explanations  (Sreedharan, Srivastava, and Kambhampati 2018; Sreedharan et al. 2019 ) in a loop with AI systems for identifying user-skills and skill-gaps can help develop AI systems that gradually present users with new functionality and explain it, thereby training their users onthe-fly and as needed. Such systems would be better tuned towards P2, and towards addressing the underlying problems of AI safety and the future of work. Critically, ensuring progress towards safe and assistive AI systems requires that AI systems with arbitrary internal designs support assessment along the metrics developed for E1-E5. This raises a new set of research questions: Can we develop non-intrusive AI-interface requirements for supporting such evaluations in the face of changing operating environments and objectives? The need for such interfaces is even more pressing for systems that learn and those that undergo system updates after deployment. What is the minimal external interface that an AI system must support so as to allow its independent evaluation? How would changing the nature of such interfaces change the complexity of conducting such an evaluation? One would expect that AI sys-tems that offer more transparency would be easier to evaluate. Could we use the inherent reasoning capabilities of AI systems to develop interface requirements that would allow more adept systems to make such evaluations easier? E.g., rather than testing a manufacturing robot to discover its response to every possible situation, could we ask higher-level queries such as "under which situations would you be able to create the proposed assembly?" Clearly, the ease of assessment of an AI system would depend on the class of queries that it can answer. Recent work suggests that a minimal query-response interface for AI systems that connects the system with a simulator and observes its responses to high-level instructions can be quite powerful. Such an interface has a few distinct advantages. Current AI systems are already tested with simulators and they are inherently required to be able to take user instructions, so these interface requirements can be considered to be natural. They also allow the autonomous synthesis of query-policies: running the query-policy on a black-box taskable AI system can help construct an interpretable model of the limits and capabilities of that system  (Verma, Marpally, and Srivastava 2021) . Such models can be used to support the evaluations discussed above. Extensions of such interface requirements to arbitrary AI systems would help ensure that our AI systems are amenable to independent evaluation. Such a paradigm would allow users to assess their AI systems while freeing AI researchers and developers to utilize arbitrary internal implementations. Systems with interfaces that support more efficient and accurate independent assessment would be rewarded with greater public adoption of their products. Progress on these threads would help prevent undesirable situations such as insufficient support for independent evaluation of powerful AI systems, and the negative consequences of deployment of an insufficiently evaluated system.