Abstract: A bandit superprocess is a decision problem composed from multiple independent Markov decision processes (MDPs), coupled only by the constraint that, at each time step, the agent may act in only one of the MDPs. Multitasking problems of this kind are ubiquitous in the real world, yet very little is known about them from a computational viewpoint, beyond the observation that optimal policies for the superprocess may prescribe actions that would be suboptimal for an MDP considered in isolation. (This observation implies that many applications of sequential decision analysis in practice are technically incorrect, since the decision problem being solved is often part of a larger, unstated bandit superprocess.) The paper summarizes the state-of-theart in the theory of bandit superprocesses and contributes a novel upper bound on the global value function of a bandit superprocess, defined in terms of a direct relaxation of the arms. The bound is equivalent to an existing bound (the Whittle integral), but is defined constructively, as the value of a related multi-armed bandit. We provide a new method to compute this bound and derive the first practical algorithm to select optimal actions in bandit superprocesses. The algorithm operates by repeatedly establishing dominance relations between actions using upper and lower bounds on action values. Experiments indicate that the algorithm's run-time compares very favorably to other possible algorithms designed for more general factored MDPs.

Multitasking: Efficient Optimal Planning for Bandit Superprocesses

INTRODUCTION Multitasking is no doubt an activity familiar to the reader: one faces several decision problems but can act on only one (or perhaps a bounded number) at a time. Such prob-lems are ubiquitous for individuals, corporations, armies, and governments. Multitasking problems are expressed by the class of bandit superprocesses  [Nash, 1973]  or BSPs. A k-armed BSP M consists of k independent Markov decision processes M 1 , . . . , M k ; the MDPs are coupled by a common discount factor and by the constraint that at each time step the agent can act in only one MDP. Since the MDPs are independent of each other, one might imagine that the optimal policy for M is obtained by solving each MDP-turning the BSP into a multi-armed bandit-and then using Gittins indices to choose a sequence of arms. In fact, the optimal policy for a BSP may include actions that are suboptimal from the point of view of the constituent MDP in which they are taken. The reason for this is that the availability of other MDPs in which to act changes the balance between short-term and long-term rewards in a component MDP; in fact, it tends to lead to greedier behavior in each MDP because aiming for longterm reward in one MDP would delay rewards in all the other MDPs. The globally and locally optimal policies necessarily coincide only when the discount factor is 1. Hence, in addition to their general importance in the real world, a second reason to study multitasking problems is that they undermine an assumption implicit in practical applications of sequential decision analysis: the assumption that an optimal solution for the user's decision problem is optimal for a user who faces multiple decision problems. Despite these considerations, there has been remarkably little research on bandit superprocesses and almost none in AI. Section 2 summarizes what is known. Obviously, there are connections to multi-armed bandits (MABs), which are a special case of BSPs in which each arm only allows one choice of action rather than several. Unfortunately, the index theorems that simplify the computation of optimal MAB policies are not valid for BSPs. There are also strong connections-hitherto unexplored-between BSPs and sums of games as studied in combinatorial game theory  [Conway, 2000] . The principal question we address in this paper is whether an algorithm exists that is substantially more efficient than applying a standard MDP solver to the "cross-product" MDP obtained by combining the states spaces of the constituent MDPs. The sole known case where planning for a BSP is provably more efficient is the case where a dominating policy exists: if a single policy is optimal across the family of retirement processes associated with each arm then a BSP can be reduced to an equivalent MAB  [Whittle, 1980] . (Retirement processes provide a measure of how the optimal policy depends on context and are defined in Section 3.) However, this condition is seldom satisfied in practice. This paper provides three contributions. First, we give a concise survey of the bandit superprocess literature tailored to the AI community. Second, we provide a novel upper bound on the global value function of a BSP. For a BSP, M , with arms {M 1 , . . . , M k } we relax each arm to obtain an MAB M = {M 1 , . . . , M k } by adding actions to ensure that dominating policies exists. We show that this upper bound is equivalent to the Whittle integral, an existing upper bound for BSPs  [Brown and Smith, 2013] . Because our bound is defined in terms of an explicit relaxation, it provides insight into the nature of the bound and opens an avenue to extend this work to more general MDPs. Finally, we describe a practical computational approach for solving BSPs: we derive a simple method for computing the Whittle integral upper bound that can use an arbitrary MDP solver as a black box; then we combine this upper bound with a lower bound to derive an efficient algorithm for -optimal decision making in a BSP. We present empirical results to show that it substantially improves over more general optimal factored MDP algorithms; we use it to compute provably optimal actions for problems with upwards of 10 30 states in the full state space.
 RELATED WORK Robbins  [1952]  provided the first formulation of multiarmed bandits (MABs) in their modern form. The famous Gittins index theorem  [Gittins, 1979]  showed that optimal MAB policies are obtained by ranking the arms according to an index function defined on each seperately. An immediate corollary is that optimal decision making in an MAB is linear in the number of arms. Problems with this property are said to be indexable. Bandit superprocesses (BSPs) were introduced by  Nash [1973]  as a generalization of multi-armed bandits to study allocation of resources among research projects.  Whittle [1980]  provided an alternate proof of the Gittins index theorem that extends to bandit superprocesses with dominating policies. His proof utilized a construction called the Whittle integral, which allows one to compute the value of a composite state in an MAB.  Glazebrook [1982]  provides a proof that bandit superprocesses are not indexable in general.  Glazebrook [1993]  considers bandit superprocesses where the arms can exert limited influence on each other and shows a result analogous to  Whittle's. Brown and Smith [2013]  were the first to identify the signifigance of the Whittle integral for sequential decision making. They developed a version of policy iteration to compute it, and recognized that it upper bounds the value function of a BSP, 1 but did not derive an algorithm for solving BSPs. Our work provides two further contributions regarding the Whittle integral: a short proof that it is an upper bound and a simpler algorithm to compute it. Within the AI community, there has been limited study of loosely coupled Markov decision processes.  Singh and Cohn [1998]  consider optimal solutions to simultaneous MDPs, where an agent can take actions in a number of MDPs. Their formulation is more general than ours, as it is possible to act in multiple MDPs at once. They derive bounds for this problem and give an algorithm that combines a form of real-time dynamic programming with pruning steps to remove provably suboptimal actions. However, their bounds are substantially looser than ours, and our experiments show that the corresponding algorithm can be impractical for simple BSPs with many arms.  Meuleau et al. [1998]  examine MDPs that are coupled by constraints on the use of shared resources. BSPs fit within this class if we view the restriction to a single MDP at a time as a constraint on the agent's attention. Interestingly, their heuristics are also defined in terms of a parameterized value function for the component MDPs-it would be interesting to attempt to generalize the approaches considered here to their problem domain. In this work, we leverage the particular structure of our resource to compute optimal solutions; Meuleau et al. construct a heuristic policy.
 TECHNICAL BACKGROUND MARKOV DECISION PROCESSES Definition 1. (Markov Decision Process  [Puterman, 2009] ) A (finite-state, discounted) MDP, M , is a tuple M = S, A, T, R, γ . S is a set of states. A is a set of actions. T : S × A × S → [0, 1] is a function that assigns probability to state transitions for each state-action pair. R is a (bounded) reward function that maps state-action pairs to (positive) rewards R : S × A → R + . γ ∈ [0, 1) is a discount factor. A solution to M is a policy, π, that maps states to actions. The value of a state, s, under π is the sum of expected discounted rewards received by starting in s and selecting ac-tions according to π: V π (s) = E ∞ t=0 γ t R(s t )|s 0 = s, π . The optimal policy, π * , maximizes this value. In the above definition, and the ones that follow, we use superscripts to indicate dependence on the agent's policy. To simplify notation, we will omit these superscripts when the policy referred to is the optimal policy (e.g., V (s) = V π * (s)). The Q-function for the state-action pair, (s, a), is the value of taking a in s and selecting future actions according to π * .
 RETIREMENT PROCESSES Given an MDP, Whittle defines a family of optimal stopping problems: Definition 2. (Retirement Process  [Whittle, 1980] ) Let M be an MDP. For ρ ≥ 0, the retirement process for M with retirement reward, ρ, is an MDP, M ρ , with a single additional state, s R , and action, a R . a R transitions deterministically to s R and receives reward ρ. s R is a sink state that accrues zero reward. We refer to a decision to select a R as a decision to retire. We denote the retirement process value function as a function of a state and retirement reward, V (s, ρ). We let the optimal policy for retirement reward ρ be π * ρ . We write the set of states where the policy, π, retires as τ π ρ . We use ρ − (π) and ρ + (π) to denote the interval of retirement rewards so that π is optimal: ρ ∈ [ρ + (π), ρ + (π)] ⇒ V π (s, ρ) = V (s, ρ). We adopt the convention from the MAB/BSP literature and abuse notation to denote the (random) number of steps prior to retirement as τ π ρ (s). For s ∈ τ π ρ we let P retire (s |s, ρ, π) be the probability that s is the first state in τ π ρ the agent will reach given that it is in state s and executes policy π. We denote the expected discounted reward accrued prior to retirement, starting in s, as R π ρ (s). In regions of retirement reward where the optimal policy and stopping rule do not change, ∂V ∂ρ (s, ρ) is defined and is equal to the expected value of the discount parameter at retirement. This allows us to write the following expression for the retirement process value function: V (s, ρ) = R ρ (s) + E[γ τρ(s) ]ρ. (1) V (s, ρ) is piecewise linear in ρ. Figure 1 shows V (s, ρ) for the example BSP in Ex. 1. A policy that is optimal for every setting of ρ is called a dominating policy. Definition 3. (Dominating Policy) Let π be a policy for an MDP, M . π is a dominating policy iff ∀ρ ≥ 0 ∀s / ∈ τ ρ π(s) = π * ρ (s).
 MULTI-ARMED BANDITS AND BANDIT SUPERPROCESSES Multi-armed bandits (MAB) are a restricted class of MDPs that have received extensive study. Of particular interest are the form of the optimal policy and its utility in modelling "exploration vs exploitation" trade-offs. An MAB consists of a set of Markov reward processes (MRPs), where an MRP is an MDP with a single action. Each MRP is referred to as an arm of the problem. At each time-step, the agent selects an arm, that arm transitions according to its transition distribution, and the agent receives the corresponding reward. We adopt a summation notation to indicate the combination of several MRPs into an MAB. For example, if X and Y are MRPs and Z is the MAB with arms X and Y , we will write Z = X + Y . A famous result is that the optimal policy for any MAB is an index policy  [Gittins, 1979] . Each state, s i , in the individual arms is assigned an index, I si . In joint state s = {s 1 , . . . , s k }, the optimal action is to select argmax i I si . For MAB arm M i in state s i , the index is defined as the value of retirement reward such that the agent is indifferent between immediate retirement and following the optimal stopping policy  [Whittle, 1980] : I si = min ρ {ρ ; V i (s i , ρ) = ρ}. This means that, in a sense, context for a multi-armed bandit can always be summarized by a single number. To model a multi-tasking problem, we consider a generalization of multi-armed bandits: bandit superprocesses (BSPs). Bandit superprocesses allow arms that are arbitrary MDPs (and so are a generalization of MABs); at each time step, the agent selects both an arm and an action to take within that arm. Definition 4. (Bandit Superprocess  [Nash, 1973] ) Given k MDPs, {M i = S i , A i , T i , R i , γ }, we define M = i M i = S, A, T, R, γ to be the bandit superprocess with arms {M i }. S = × i S i and A = ∪ i A i . The transition distribution is stationary for arms that are not selected and follows the identical reward and transition distributions for the selected arm. Naturally, this makes planning more difficult. To see how, consider the following proposal: Conjecture 1. Let X = S X , A X , T X , R X , γ be an MDP. Define Y similarly and let Z = X + Y be the bandit superprocess that is their sum. Let a ∈ A X , s X ∈ S X be a state-action pair from X. Suppose that this transition is suboptimal in every retirement process: ∀ ρ ≥ 0, V X (s X , ρ) > Q X ((s X , a), ρ). Then a is suboptimal for any state in Z with s X as one of the components: ∀s Y ∈ S Y , V Z ({s X , s Y }) > Q Z ({s X , s Y }, a). This conjecture essentially proposes that state-action pairs can be safely ignored if they are suboptimal for all settings of a constant alternative. Unfortunately, as the following example illustrates, this is not the case. Example 1. Define X to be a deterministic reward chain and Y to be an initial choice between three reward chains (y 0 , y 1 , y 2 ) defined as follows: V (y 0 , ρ) = max 100 X = 28, 2 t=0 γ t + γ 3 ρ, ρ V (y 1 , ρ) = max 99 2 t=0 γ t + γ 3 max 1.4 1 − γ , ρ , ρ V (y 2 , ρ) = max 28 1 − γ , ρ . If we let γ = .9, then we have ∀280 > ρ ≥ 0 max{V (y 0 , ρ), V (y 2 , ρ)} > V (y 1 , ρ). Thus, any policy for a retirement process retirement process derived from Y that initially selects y 1 is suboptimal. We can apply the same strategy to compute the value of each combination of reward streams: V (y 0 + X) = 100 2 t=0 γ t + γ 3 28 5 t =0 γ t V (y 1 + X) = 99 2 t=0 γ t + γ 3 28 5 t =0 γ t + γ 6 1.4 1 − γ V (y 2 + X) = 28 1 − γ From this we can see that the optimal policy for the BSP that combines these MDPs, Z = X + Y , initially collects reward from y 1 . This contradicts Conjecture 1. In this section we show a bound on the value function of a bandit superprocess. We derive this bound by adding actions to a BSP so that it is equivalent to an MAB. The value function of our relaxed BSP is equivalent to the Whittle integral bound derived in Brown and  Smith [2013]  and so it yields insight into that computation. We begin by defining the Whittle integral and show some basic results in order to motivate our relaxation. Definition 5.  (Whittle Integral [Brown and Smith, 2013] ) Let M be a BSP. Let i index the arms of M. For any state, s = {s i }, and ρ ≥ 0, the Whittle integral of s is defined as V (s, ρ) = I − I x=ρ dx i ∂V i ∂ρ (s i , x). ( 2 ) Where I ≥ max i I si . When the arms of a BSP admit a dominating policy the Whittle integral is equal to the value function: Theorem 1. (Whittle Condition  [Whittle, 1980] ) Let M be a k-armed BSP with components {M i } and state space S. If each M i has a dominating policy, then ∀s ∈ S, ∀ρ ≥ 0, V (s, ρ) = V (s, ρ). MRPs have a single action per state, so the Whittle condition is trivially satisfied for all multi-armed bandits. V (s, 0) = V (s), so V (s, 0) provides an efficient method to compute the value of an MAB. For BSPs an arm that satisfies the Whittle condition can be replaced with an MRP that selects actions according to π * . The formula in Eq. 2 lends itself to a straightforward implementation, but is very challenging to interpret. We show below that this is equivalent to evaluating the retirement process value function for a single arm with a set of rewards determined by the other arms 3 . We refer to this set of rewards as the critical points of those arms. Definition 6. (Critical Points of an MDP) Let M be a Markov decision process. The critical points of M , C(M ) = {ρ i }, are the values of retirement reward such that the optimal stopping rule or policy changes. These are points where there is a discontinuity in ∂V ∂ρ . We let ∆ M (s, ρ) = lim δ→0 ∂V M ∂ρ (s, ρ + δ) − ∂V M ∂ρ (s, ρ − δ) be the size of the corresponding discontinuities. This is equivalent to the expected increase in E [γ τ ] under the new stopping rule. Theorem 2 shows that ∆ and C characterize the interaction between arms of an MAB. Theorem 2. Let X, Y be Markov reward processes. Let Z = X + Y be the 2-armed bandit. ∀s = {s X , s Y } ∈ S Z , V Z (s) = ρ∈C(Y ) V X (s X , ρ)∆ Y (s Y , ρ). (3) Proof. (sketch) As a first step, we follow the steps in Whittle  [1980]  and integrate Equation 2 by parts. This expresses V as the following integral: VZ (s) = Is X ρ =ρ V X (s X , ρ ) ∂ 2 V Y (s Y , ρ ) ∂ρ 2 dρ . V Y is piecewise linear with respect to ρ so ∂ 2 V Y ∂ρ 2 is a weighted sum of delta functions centered at V Y 's kinks. C(Y ) and ∆ Y respectively characterize these kinks and weights. Thus, VZ (s) is equal to the rhs of Eq. 3. X and Y are MRPs, so an appeal to Theorem 1 shows the result. A similar property holds for arbitrary k-armed bandits. This shows that we can summarize the context for an arm as a collection of weighted retirement rewards. Turning to BSPs, this lends insight into the approximation the Whittle integral introduces. To see how, we reconsider Ex. 1. Recall that this example consists of the BSP Z = X + Y and that Y is a choice between three reward chains, {y 0 , y 1 , y 2 }. Theorem 2 allows us to write the gap between the Whittle integral upper bound and the value of selecting y 1 as ρ∈C(X) max i V yi (s i , ρ) − V y1 (s 1 , ρ) ∆ X (s X , ρ) Figure  1  shows the retirement process value functions for this example. While the retirement process value of y 1 is always less than that of either y 0 or y 2 , it is close enough to their maximum that choosing y 1 essentially achieves the upper bound. The Whittle integral for Z is a weighted combination of distinct retirement process value functions (V y0 , V y2 ) but, in reality, the agent will be forced to pick a single policy of the two.
 DOMINATED RELAXATION OF AN MDP In this section, we introduce our primary theoretical result: a relaxation for the arms of a BSP so that a dominating policy exists. This reduces the BSP to an MAB whose value upper bounds the value of states in the BSP. We can show that the Whittle integral computes the value of states in this MAB and arrive at a straightforward proof that the Whittle integral is an upper bound. We call the result the dominated relaxation of an MDP. Before providing the details, we illustrate the main ideas with an example. The relaxed MDP is actually a Semi-MDP (SMDP): a generalization of an MDP where each action, a, has a duration, δ(a) ∈ R + . Example 2. Let MDP M be an initial choice between MRPs: M = 15, 0, 0, 0 . . . 10, 10, 10, 3, 3, 3, . . . . Let T (B) be the top (bottom) MRP. Let π T (π B ) be the policy that selects T (B) in s 0 . We can relax M with the addition of a single durative action, a , that transitions from the sink state in T to the sink state B. We set δ(a ) = 2. We take ρ such that V π T (s 0 , ρ) = V π B (s 0 , ρ) and set the rewards associated with a to be R π B ρ (s 0 ) − R π T ρ (s 0 ) = 10 2 0 γ t − 15 = 12.1 . With this change, at low settings of retirement reward, the agent is indifferent between a policy that opts for the top chain then selects a and the bottom reward chain. For high settings of retirement reward, the optimal policy retires immediately or retires after collecting the reward of 15. This SMDP satisfies the Whittle condition and can be replaced by an MRP in any bandit superprocess. Fig.  1  (c) shows an illustration of the state space and the introduced action. In this example, we connected the state where π T retires to the state where π B retires. This lets the agent collect shortterm and long-term rewards with the same policy. To do this in general, we introduce multiple copies of the state space, one for each policy that is optimal for some ρ. Definition 7. (Dominated Relaxation of an MDP) Let M be an MDP with discount factor γ and state space S. Let s be a state in M . The dominated relaxation of M for s, M D (s), is a semi-Markov decision process that fixes s as an initial state. Let {π i } be the policies that are optimal for some ρ: {π * ρ |ρ ∈ C(M )}. This sequence is ordered so that ρ − (π i ) is increasing in i 4 . For each i, we introduce a copy of the state space, S i , where the agent is restricted to following π i . Let s i be the analogue of s in S i . For s i ∈ τ ρ−(πi) , we introduce a single durative action, a i , that takes the agent from S i to S i−1 and characterize it as follows: • R(s i ) = R ρ+(πi−1) (s) − R ρ−(πi) (s) • T (s i , a i , s i−1 ) = P retire (s |π i−1 , ρ + (π i−1 ), s) • δ(a i ) = log γ E γ τ π i−1 ρ + (s) − log γ E γ τ π i ρ − (s) For each i, we introduce an action that transitions from s to s i with δ = 0. Let V D (s) represent the value of s in M D (s). Theorem 3. Let M be an MDP with state space S. The following statements are true for s ∈ S and ρ ≥ 0: 1. M D (s) satisfies the Whittle condition. 2. V D (s, ρ) = V (s, ρ).
 V (s, ρ) ≥ V (s, ρ) Proof. See supplementary materials
 AN -OPTIMAL ALGORITHM FOR BANDIT SUPERPROCESSES In this section, we present two algorithms related to BSPs. The first is a novel algorithm to compute V (s, ρ) that can use any method to solve the underlying MDP. The second is an efficient algorithm to compute optimal actions for a BSP. Our approach, Branch-and-Bound Value Iteration (BBVI), uses Whittle integrals to compute upper and lower bounds on value. Then, we apply a modified Branch-and-Bound search to find provably optimal actions.
 COMPUTING A RETIREMENT PROCESS VALUE FUNCTION Before we present BBVI, we give a simple algorithm to compute a retirement process value function that uses an (arbitrary) MDP solver as a black box. Brown and  Smith [2013]  describe an algorithm to compute retirement process value functions, but their approach requires custom implementation of a modified simplex algorithm. Our approach is simple and based on an algorithm that initially appeared in the solutions manual for  Russell and Norvig [2010] . Our goal is to identify each component of V (•, ρ), so our output will be a list of critical points and the associated slopes. First, we will need the following result. Lemma 1. Let M be an MDP with state space S. Let ρ 0 < ρ 1 . Consider a retirement reward in the interior of this interval, ρ ∈ (ρ 0 , ρ 1 ). If, ∀s ∈ S V (s, ρ) = V (s, ρ 0 ) + V (s, ρ 1 ) − V (s, ρ 0 ) ρ 1 − ρ 0 (ρ − ρ 0 ), ( 4 ) then there is at least one policy and stopping rule that is optimal for every ρ ∈ [ρ 0 , ρ 1 ]. The expected value of the discount factor at retirement is the slope between those points: E γ τ * (s) = V (s, ρ 1 ) − V (s, ρ 0 ) ρ 1 − ρ 0 . ( 5 ) Proof. This follows from the form of Equation 1 and the fact that V (s, ρ) is piecewise linear, increasing, and convex. This test allows us to implement a binary search over retirement rewards. Our approach relies on a given MDP solution method, SOLVE, that returns the vector of values for a given MDP. We initialize an interval that contains all critical points and make a call to SOLVE to compute the retirement process value at each endpoint. Then we call SOLVE to compute the retirement process value for the midpoint of our interval and apply the test in Eq. 4. If it succeeds, we return the lower endpoint and the corresponding slope. If this does not, we sub-divide our interval and recurse. We can concatenate the results to get the breakpoints and slopes for the retirement process value function over this interval. Algorithm 1 shows pseudocode for this algorithm. Algorithm 1 Computing an MDP's Critical Points Define: CRITICALPOINTS(M [, ρ − , ρ + , V − , V + ]) Input: MDP, M ; Interval of retirement values, ρ − , ρ + ; Values at interval endpoints, V − , V + if ρ − , ρ + are not set then ρ − ← 0 ρ + ← maxReward(M ) 1−γ V − ← SOLVE(M ρ − ) V + ← ρ + ## Retirement is initially optimal end if ρ ← ρ + −ρ − 2 V ← SOLVE(M ρ ) if |V − V − + (ρ − ρ − ) V + −V − ρ + −ρ − | < then return [ρ − ], V + −V − ρ + −ρ − else pts − , slope − ←CRITICALPOINTS(M, ρ − , ρ, V − , V ) pts + , slope + ←CRITICALPOINTS(M, ρ, ρ + , V, V + ) ## Merge adjacent intervals with the same slope if slope − [−1] = slope + [0] then del pts + [0],del slope + [0] end if return pts − ∪ pts + , slopes − ∪ slopes + end if
 BRANCH-AND-BOUND VALUE ITERATION Now, we present a practical algorithm to compute optimal actions in a bandit superprocess. While a BSP is not indexable, we would like to be able to plan efficiently when the arms are 'close' to indexable-when there are only a few states where the optimal policy changes in response to the opportunity cost of delayed rewards in other arms. Our approach is based on envelope dynamic programming methods to solve MDPs: we compute value estimates for a given initial state by solving a dynamic program defined over a reachable subset of the state space  [Gardiol, Natalia H and Kaelbling, Leslie P, 2003, Hansen and Zilberstein, 2001] . The primary difference between our approach and standard envelope methods is that we use dynamic pro-gramming on our envelope to update an upper bound and a lower bound on the value. This is useful in a BSP because the Whittle integral allows efficient calculation of both bounds. Our goal is to compute the optimal action for a given state s in a BSP, M = M 1 , . . . , M K . We can obtain an upper bound on V (s) with a Whittle integral. We use a Whittle integral for the MAB that solves each arm independently to compute a lower bound. Our algorithm maintains upper and lower bounds on the Q-function for each action that could be executed from s. We write these bounds as Q + (s, a) and Q − (s, a) respectively. If there is a pair of actions a, a such that Q + (s, a ) < Q − (s, a) + , then we can conclude that a is at least -suboptimal. If this test removes all but a single action, then we have found a -optimal action and return it. In the event that more than one action remains, we do a partial expansion of the state space around s. We keep track of a set of expanded states, E, and a set of boundary states, B. States in the boundary set, B, are states that some expanded state can transition to but have not yet been added to E. We can use these states to update the bounds on Q(s, •) by solving a related MDP. This is formalized in the following theorem. Theorem 4. Let M be an MDP with state space S and action space A. Let S ⊆ S where S = B ∪ E and ∀ s ∈ E, T (s, a, s ) = 0 ⇒ s ∈ S . Let M + be an MDP with state space S ∪ {α}. M + 's transition distribution and rewards are identical for states in E (expanded states). Each state s ∈ B transitions deterministically to α and receives a reward that is an upper bound on V M (s). α is a sink state that accrues 0 reward. Then the value of a state in M + is an upper bound on its value in M : V M + (s) ≥ V M (s). Proof. See supplementary materials. It is straightforward to show that an updated lower bound can be computed by fixing boundary states to have value equal to a lower bound. Our approach alternates between pruning actions based on dominance relations, adding states to E, and computing the value of an MDP defined over E. We interleave value iteration with a branch-andbound search, so we call it Branch-and-Bound Value Iteration (BBVI). Algorithm 2 shows pseudocode to implement this method. At termination, the action we return is guaranteed to be at most -suboptimal. Theorem 5. Let M be a BSP and let s be a state in M . Let a be the action returned by BBV I(M, s, ). V (s) − Q(s, a) < . Proof. See supplementary materials. Large regions of the state space are irrelevant to our objective so we use a heuristic to guide node expansion. Our heuristic is a measure of the sensitivity of values at the root to change in the upper and lower bounds of unexpanded nodes. Similar heuristics have been applied in many settings  [Rivest, 1987, Smith and Simmons, 2004] . In BBVI, these values are the expected discounted visitation rates in the bounding MDPs and are computed as the dual variables from a value iteration LP. The backup procedure is slower than node expansions, so we perform backups in batches. In between backups and heuristic computations, we approximate the state space as a tree and push new states onto the agenda with their parent's heuristic value weighted by the transition probability.  Q + ← SOLVE(BOUNDMDP(E , B, Q + )) Q − ← SOLVE(BOUNDMDP(E , B, Q − )) a * ← argmax a Q − (s 0 , a) end while return -optimal action a *
 EMPRIRICAL EVALUATION We implemented BBVI in Python and use the linear programming solver Gurobi to compute value functions. Our experiments were run on an Intel i7 with 16 GB of RAM. In our first experiments, we compare the scalability of BBVI with that of standard MDP algorithms. In particular we compare the following algorithms: • SPUDD: the baseline factored MDP algorithm in the 2011 IPPC  [Hoey et al., 1999] . • LRTDP(Merge): Labeled Real-Time Dynamic Programming  [Bonet and Geffner, 2003]  with the upper bound from  Singh and Cohn [1998]  as a heuristic. • LRTDP(WI): LRTDP with the Whittle integral as an upper bound. • BBVI: Branch-and-Bound Value Iteration. We evaluate performance on a simple BSP designed to model allocation of research resources to product research and development. Each arm in this problem corresponds to a potential product that our agent could sell in the market. However, before we sell a product we must devote effort to R&D. This can be done in one of two ways, a safe research approach that is slow but reliable, and a risky approach that is fast, but runs a risk of producing a defective product. After at least one product has been researched, the agent can opt to spend a round produce and sell that product or continue research on a different project. Because taking a product to market does not change the state of the BSP, it is straightforward to show that any stationary policy (including π * ) will only ever produce a single product. We define a distribution over these problems by selecting a random market value for each product uniformly from [0, 1] and random durations for the safe and risky research strategies (although we ensure that safe research is at least 3x as slow as risky research). Figure  2  (a) shows the state space for an example arm with typical parameters. Figure  2  (b) shows the results of our comparison. We used a timeout of 1000s and set a memory limit of 4 GB. We ran our experiments in an online setting and running times reflect the amount of time to select an optimal action from scratch. While this is a natural setting for LRTDP and BBVI, it is admittedly a little unfair to SPUDD (which computes a full policy). This is mitigated by the fact that BBVI's improvement over SPUDD, which is a little under 5 orders of magnitude with 10 8 states, is such that an agent would need to be planning over an immense horizon before SPUDD presents a reasonable alternative. The factored MDP bounds from  Singh and Cohn [1998]  are quite ill-suited to this problem. This is because its upper bound (the sum of the independent value for each arm) is very loose and it essentially forces LRTDP to enumerate the state space. In contrast, LRTDP performs quite well when it has good heuristic information. However, its performance degrades faster than that of BBVI, because it does not effectively leverage a lower bound on value. Even for very large problem instances, BBVI computes optimal solutions in well under a second. BBVI's performance in the R&D domain is largely explained by the structure of the arms and the short horizon within each arm. Although there may be a large number of arms, BBVI reduces each individual arm to an equivalent MRP after a small number of node expansions. This means that running time is essentially linear in the number of arms. Note that, in this domain, the solution that solves each MDP independently and executes the corresponding index policy will just opt for conservative research on the most valuable option. This policy is essentially always suboptimal: as long as there are reasonable alternatives it is usually a good idea to try some risky projects. Our next experiment uses a synthetic domain to explore the performance of BBVI as the structure and number of arms changes. In this BSP, each arm is a 20x20 grid world. The actions in this world correspond to moving in the 4 cardinal directions. Rewards are 0 everywhere except at randomly chosen locations and after receiving a reward, the agent transitions to a random location. In this experiment we vary the number of rewards that are available in each arm. Adding rewards will typically cause the optimal policy for an arm to be more sensitive to context. This means that the dominated relaxation will produce a looser upper bound, because it will need to add more actions. We evaluate this difficultly by measuring the convergence criterion of BBVI after 10,000 node expansions. Figure  2  (c) shows the results of this experiment. We can see that BBVI's solution quality decreases with the number of states, but the primary variation comes from changes in the properties of the arms.
 CONCLUSION AND FUTURE WORK In summary, we presented a model of highly decoupled decision making: bandit superprocesses. A BSP presents an agent with the opportunity to act in one of several Markov decision processes. The key constraint is that the agent can only act in a single process at a time. We provided a summary of BSP research, including an upper bound for the value function of a BSP in the form of the Whittle integral. We derived an equivalent relaxation and thus gave a novel proof that the Whittle integral is an upper bound. Finally we presented and evaluated algorithmic solutions for this class of decision problems. In future work, we plan to extend these ideas in three directions. The first is algorithmic improvements for BSPs. A potential direction here is to leverage factored dynamics in the bounding MDPs themselves. A search over sequences of MDPs that only consider states where the current policy stops could be more efficient. Next, also plan to explore generalizations of the bounds used for BSPs to more general cases of global resource constraints (e.g., those considered in  Meuleau et al. [1998] ). Finally, the similarity to sums of games studied in  Conway [2000]  suggests an unexplored connection between BSPs and combinatorial game theory. Figure 1 : 1 Figure 1: (a-b) Retirement process value functions for the reward chains Ex. 1. The slope of the lines is the expected value of the discount parameter at retirement: E [γ τ ]. Flat sections indicate regions of retirement rewardswhere retirement is always suboptimal. The kink in the green curve at ρ = 280 indicates that it has become optimal to retire immediately. The kink in the black curve at ρ = 14 increases the slope to γ 6 < 1; for ρ > 14, it is optimal to retire after collecting a prefix of the reward stream from y 2 . (c) Dominated relaxation of the MDP in Ex. 2. We add additional durative actions to the state space to ensure that a dominating policy exists. Note that the original MDP does not admit a dominating policy.
 Algorithm 2 2 Branch-and-Bound Value Iteration Define:BBVI( M 0 , . . . , M K , s 0 , ) Input: BSP arms, M k ; Initial state, s 0 ; Tolerance, # Lower bound M by fixing a policy for each arm LB k ←toMRP(M k ) Compute critical points for M k , LB k Compute bounds on Q(s 0 , •) with Whittle integrals for M and LB. # Keep track of expanded region of state space E ← ∅ # Keep track of states at boundary of M + , M − B ← {s 0 } a * ← argmax a Q − (s 0 , a) while ∃a = a * s.t Q + (s 0 , a ) ≥ Q − (s 0 , a * ) do s ←pop(B) E ← E ∪ {s} for a ∈ A do for s ∈ successors(s) do Compute upper and lower bounds for Q(s , •) B ← B ∪ {s } end for end for
 Figure 2: (a) the state space for an arm of a R & D BSP. All states accrue 0 reward except for the 'Working Product' state, which collects a fixed reward per time step. (b) shows runtime (log-scale) vs problem size for BBVI and several alternatives on an R & D BSP. SPUDD [Hoey et al., 1999]  and LRTDP (with the heuristic from Singh and Cohn [1998] ) scale poorly compared with algorithms that leverage the Whittle integral. BBVI's additional improvement over LRTDP stems from the use of an efficient, exact lower bound to check convergence. (c) BBVI's bound on suboptimality after 10,000 node expasions (measured in units of discounted reward). Changing the number of rewards in each arm allows us to measure BBVI's performance as arms become more sensitive to context.
			 This result first appears in the literature as an intermediate step in a larger proof from Whittle [1980] . It went unnoticed or unappreciated in the intervening 30+ years.
			 Here we use the fact that the Gittins index of a non-increasing reward sequence is equal to the instantaneous reward.4 AN UPPER BOUND FOR BANDIT SUPERPROCESSES
			 Our result is a small extension on a related result in Brown and Smith [2013] ; it is primarily included to provide intuition.
			 Recall that [ρ−(π), ρ+(π)] is the interval of retirement rewards where π is optimal.


Abstract: As it achieves a goal on behalf of its human user, an autonomous agent's actions may have side effects that change features of its environment in ways that negatively surprise its user. An agent that can be trusted to operate safely should thus only change features the user has explicitly permitted. We formalize this problem, and develop a planning algorithm that avoids potentially negative side effects given what the agent knows about (un)changeable features. Further, we formulate a provably minimax-regret querying strategy for the agent to selectively ask the user about features that it hasn't explicitly been told about. We empirically show how much faster it is than a more exhaustive approach and how much better its queries are than those found by the best known heuristic.

Shun Zhang
Edmund H Durfee
Satinder Singh

Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes

Introduction We consider a setting where a human user tasks a computational agent with achieving a goal that may change one or more state features of the world (e.g., a housekeeping agent should change the state of the house floors and kitchen sink from dirty to clean). In the process of accomplishing the goal, the agent generally changes other features (e.g., its own position and power level, opening doors, moving furniture, scaring the cat). Some of these side-effects might be fine (even expected) by the user (e.g., moving), but others may be undesirable/unsafe (e.g., leaving doors open lets the cat roam/escape) even though they speed goal achievement (e.g., the agent's movement between rooms). Although the user tells the agent about some features that can be changed, as well as some to not change (e.g., don't knock over the priceless vase), the user often lacks the time, patience, or foresight to specify the changeability of every pertinent feature, and may incorrectly assume that the agent has commonsense (e.g., about cat behavior and the value of vases). How can the agent execute a safely-optimal policy in such a setting? We conservatively assume that, to ensure safety, the agent should never side-effect a feature unless changing it is explicitly known to be fine. Hence, the agent could simply execute the best policy that leaves such features unchanged. However, no such policy might exist, and even if it does it might surprise the user as unnecessarily costly/inefficient. Our focus is thus on how the agent can selectively query the user about the acceptability of changing features it hasn't yet been told about. We reject simply querying about every such feature, as this would be unbearably tedious to the user, and instead put the burden on the agent to limit the number and complexity of queries. In fact, in this paper we mostly focus on finding a single query about a few features that maximally improves upon the policy while maintaining safety. Our three main contributions in this paper are: 1) We formulate (in Section 2) an AI safety problem of avoiding negative side-effects in factored MDPs. 2) We show (in Section 3) how to efficiently identify the set of relevant features, i.e., the set of features that could potentially be worth querying the user about. 3) We formulate (in Section 4) a minimaxregret criterion when there is a limit on the number of features the agent can ask about, and provide an algorithm that allows the agent to find the minimax-regret query by searching the query space with efficient pruning. We empirically evaluate our algorithms in a simulated agent navigation task, outline ongoing extensions/improvements, and contrast our work to prior work, in the paper's final sections.
 Problem Definition We illustrate our problem in a simulated agent gridworldnavigation domain, inspired by Amodei et al.  [2016]  and depicted in Figure  1 , with doors, carpets, boxes, and a switch. The agent can open/close a door, move a box, traverse a carpet, and toggle the switch. Initially, the agent is in the bottom left corner; door d1 is open, d2 and d3 closed, the carpet clean, and the switch "on". The agent can move to an adjacent location vertically, horizontally, or diagonally. For simplicity, the transition function is assumed to be deterministic. The user tasks the agent with turning off the switch as quickly as is safely possible. The quickest path (π 1 ) traverses the carpet, but this gets the carpet dirty and the agent doesn't know if that is allowed. The agent could instead enter the room through door d1 and spend time moving box b1 or b2 out of the way (π 2 or π 3 respectively), open door d2, and then go to the switch. However, boxes might contain fragile objects and should not be moved; the user knows each box's contents, but the agent doesn't. Or the agent could enter through door d1 and walk upwards (π 4 ) around all the boxes and open door d2 to get to the switch. The user may or may not be okay with door d2 being opened. There are of course many other more circuitous paths not shown. We model the domain as a factored Markov Decision Process (MDP)  [Boutilier et al., 1999 ]. An MDP is a tuple S, A, T, r, s 0 , γ , with state space S, action space A, and transition function T where T (s |s, a) is the probability of reaching state s by taking action a in s. r(s, a) is the reward of taking action a in s. s 0 is the initial state and γ is the discount factor. Let π : S × A → [0, 1] be a policy. V π is the expected cumulative reward by following policy π starting from s 0 . In a factored MDP, a state is described in terms of values of various features (e.g., the agent's location, the current time, the status of each door, cleanliness of each carpet, position of each box), so the state space S is the cross-product of the values the features can take. The reward and transition functions are often also factored (e.g., the "toggle" action only changes the switch feature, leaving boxes, doors, and carpets unchanged). We will consistently use φ to denote one feature and Φ to denote a set of features. The agent knows the complete MDP model, but has incomplete knowledge about which features the user doesn't mind being changed. In our example, the user's goal implies the agent's location is changeable, as is the switch, but the agent is uncertain about side-effecting boxes, doors, and carpets. In general, the user could dictate that a feature can only be changed among restricted values (e.g., boxes can only be moved a short distance) and/or dependent on other features' values (e.g., interior doors can be left open as long as exterior doors (like d3) stay closed). We briefly return to this issue later (Section 6), but for simplicity assume here that the agent can partition the features into the following sets: • Φ A F : The free-features. The agent knows that these features are freely changeable (e.g., its location). • Φ A L : The locked-features. The agent knows it should never change any of these features.
 • Φ A ? : The unknown-features. These are features that the agent doesn't (initially) know whether the user considers freely changeable or locked. The user similarly partitions the features, but only into the sets Φ U L and Φ U F . We assume that the agent's knowledge, while generally incomplete (Φ A ? = ∅), is consistent with that of the user. That is, Φ A F ⊆ Φ U F and Φ A L ⊆ Φ U L . Defining & Finding Safely-Optimal Policies. Our conservative safety assumption means the agent should treat unknown features as if they are locked, until it explicitly knows otherwise. It should thus find an optimal policy that never visits a state where a feature in Φ A L ∪ Φ A ? is changed, which we call a safely-optimal policy. We can use linear programming with constraints that prevent the policy from visiting states with changed values of locked or unknown features: The above linear program does not allow any locked or unknown feature to be changed and is guaranteed to produce a safely-optimal policy (when one exists). This linear programming approach, while straightforward, can be intractable for large MDPs. Alternative approaches could directly encode the safety constraints into the transition function (e.g., by removing unsafe action choices in specific states), or into the reward function (heavily penalizing reaching unsafe states). Approximate methods, like feature-based approximate linear programming  [Dolgov and Durfee, 2006]  or constrained policy optimization  [Achiam et al., 2017] , can apply to larger problems, but may not guarantee safety or optimality (or both). We return to these concerns in Section 6.
 Querying Relevant Unknown-Features In our setting the only way for the agent to determine whether an unknown feature is freely changeable is to query the user. Thus, hereafter our focus is on how the agent can ask a good query about a small number, k, of features. Our solution is to first prune from Φ A ? features that are guaranteed to not be relevant to ask (this section), and then efficiently finding the best (minimax-regret) k-sized subset of the relevant features to query (Section 4). Until Section 6, we assume the changeabilities of features are independent, i.e., when the agent asks about some features in Φ A ? , the user's response does not allow it to infer the changeability of other features in Φ A ? . Intuitively, when is a feature in Φ A ? relevant to the agent's planning of a safely-optimal policy? In the navigation domain, if the agent plans to take the quickest path to the switch (π 1 in Figure  1 ), it will change the state of the carpet (from clean to dirty). The carpet feature is thus relevant since the agent would change it if permitted. If the carpet can't be changed but door d2 can, the agent would follow (in order of preference) policy π 2 , π 3 , or π 4 , so d2, b1, and b2 are relevant. Box b3 and door d3 are irrelevant, however, since no matter which (if any) other features are changeable, an optimal policy would never change them. Thus, an unknown feature is relevant when under some circumstance (some answer to some query) the agent's optimal policy would side-effect that feature. Such policies are dominating policies. else add (Φ, ∅) to β 14: Φ rel ← Φ rel ∪ Φ rel (π) 15: checked ← checked ∪ {Φ} 16: agenda ← powerset(Φ rel ) \ checked 17: return Γ , Φ rel A dominating policy is a safely-optimal policy for the circumstance where the unknown features Φ A ? are partitioned into locked and changeable subsets. We denote the set of dominating policies by Γ = arg max π∈ΠΦ V π : ∀Φ ⊆ Φ A ? , (2) where Π Φ is the set of policies that do not change unknown features Φ ⊆ Φ ? as well as any locked features (meaning that Φ F ∪ (Φ ? \ Φ) are changeable). We denote the unknownfeatures side-effected by policy π by Φ rel (π). For a set of policies Π, Φ rel (Π) = ∪ π∈Π Φ rel (π). The set Φ rel (Γ), abbreviated Φ rel , is thus the set of relevant (unknown-) features to consider querying about. Instead of finding the safely-optimal policies for all exponentially (in |Φ A ? |) many subsets Φ ⊆ Φ A ? with Equation 2, we contribute Algorithm DomPolicies (see pseudocode) that finds dominating policies incrementally (and in practice more efficiently) by constructing the sets of relevant features and dominating policies simultaneously. In each iteration, it examines a new subset of relevant features, Φ, and, if Φ isn't pruned (as described later), finds the safely-optimal policy with Φ being locked (Line 7). It then adds Φ rel (π), the features changed by π, to Φ rel . It repeats this process until Φ stops growing and all subsets of Φ rel are examined. For example, in the navigation problem (Figure  1 ), the first added policy is the safely-optimal policy assuming no unknown features are locked, which is π 1 . Now Γ = {π 1 } and thus Φ rel = {carpet}. It iterates, treating the carpet as locked, and updates Γ to {π 1 , π 2 } and thus Φ rel = {carpet, b1, d2}. Iterating again, it locks subsets of Φ rel , finding π 3 for subset {carpet, b1}. After finding π 4 , it terminates. In Line 10, the algorithm finds the constrained optimal policy (in our implementation using Eq. 1), and in the worst case, would do this 2 |Φ rel | times. Fortunately, the complexity is exponential in the number of relevant features, which we have seen in our empirical settings can be considerably smaller (a) (b) c 1 c 1 -1 c 2 c n c 2 c n -1 -1 … … … (c) c 1 c 3 -1 c 2 -1 -0.1 -1 -1 Figure 2: Example domains used in text. (n > 2) than the number of unknown features (|Φ A ? |). Furthermore, the efficiency can be improved with our pruning rule (line 8): satisf y(Φ, β) := ¬∃ (L,R)∈β (L ⊆ Φ ∧ Φ ∩ R = ∅) β is a history of ordered pairs, (L, R), of disjoint sets of unknown features. Before DomPolicies computes a policy for its agenda element Φ, if a pair (L, R) is in β, such that L ⊆ Φ (a dominating policy has been found when locking subset of features in Φ), and that dominating policy's relevant features R don't intersect with Φ, then Φ's dominating policy has already been found. In our running example, for instance, initially Φ rel = ∅. π 1 is the optimal policy and the algorithm adds pair (∅, {carpet}) to β. When larger subsets are later considered (note the agenda is sorted by cardinality), feature sets that do not contain carpet are pruned by β. In this example, β prunes 11 of the 16 subsets of the 4 relevant features. Consider the examples in Figure  2 . To reach the switch, the agent could traverse zero or more carpets (all with unknown changeability), and rewards are marked on the edges. Proof. Let π ∈ Γ be the optimal policy with unknown features L locked. Γ is returned by DomPolicies. We denote L ∩ Φ rel (Γ ) as A and L \ Φ rel (Γ ) as B. For a proof by contradiction, assume π ∈ Γ . Then B = ∅. Otherwise, if B = ∅ (or equivalently, A = L), then L is a subset of Φ rel (Γ ) and π would have been added to Γ . Let π be the optimal policy with A locked. Since A ⊆ Φ rel (Γ ), we know π ∈ Γ . We observe that π does not change any features in B (otherwise features in B would show up in Φ rel (Γ )). So π is also the optimal policy with features A ∪ B = L locked. So π = π , which is an element of Γ .
 Finding Minimax-Regret Queries With DomPolicies, the agent need only query the user about relevant features. But it could further reduce the user's burden by being selective about which relevant features it asks about. In our running example (Figure  1 ), for instance, DomPolicies removes b3 and d3 from consideration, but also intuitively the agent should only ask about b1 or b2 if d2 is changeable. By iteratively querying, accounting for such dependencies (which can be gleaned from β in DomPolicies) and updating the relevant feature set, the agent can stop querying as soon as it finds (if one exists) a safe policy. For example, say it asks about the carpet and d2, and is told d2 (but not the carpet) is changeable. Now it has a safely-optimal policy, π 4 , given its knowledge, and could stop querying. But π 4 is the worst safe policy. Should it ask about boxes? That is the question we focus on here: How should an agent query to try to find a better safely-optimal policy than the one it already knows about. Specifically, we consider the setting where the agent is permitted to interrupt the user just once to improve upon its safely-optimal policy, by asking a single query about at most k unknown features. For each feature the user will reply whether or not it is in Φ U F . Formally, Φ q is a k-feature query where Φ q ⊆ Φ rel and |Φ q | = k. The post-response utility when the agent asks query Φ q , and Φ c ⊆ Φ A ? are actually changeable, is the value of the safely-optimal policy after the user's response: u(Φ q , Φ c ) = max π∈Π Φ A ? \(Φq ∩Φc ) V π . (3) Recall the agent can only safely change features it queries about that the user's response indicates are changeable (Φ q ∩ Φ c ). What would be the agent's regret if it asks a k-feature query Φ q rather than a k-feature query Φ q ? We consider the circumstance where a set of features Φ c are changeable and under which the difference between the utilities of asking Φ q and Φ q is maximized. We call this difference of utilities the pairwise maximum regret of queries Φ q and Φ q , defined below in a similar way to Regan and  Boutilier [2010] : P M R(Φ q , Φ q ) = max Φc⊆Φ A ? (u(Φ q , Φ c ) − u(Φ q , Φ c )). (4) The maximum regret, denoted by M R, of query Φ q is determined by the Φ q that maximizes P M R(Φ q , Φ q ): M R(Φ q ) = max Φ q ⊆Φ rel ,|Φ q |=k P M R(Φ q , Φ q ). (5) The agent should ask the minimax-regret (k-feature) query: Φ M M R q = arg min Φq⊆Φ rel ,|Φq|=k M R(Φ q ). (6) The rationale of the minimax-regret criterion is as follows. Whenever the agent considers a query Φ q , there could exist a query Φ q that is better than Φ q under some true changeable features Φ c . The agent focuses on the worst case Φ c , where the difference between the utility of Φ q and the best query Φ q that could be asked is maximized. The agent uses adversarial reasoning to efficiently find the worst-case Φ c : Given that it is considering query Φ q , it asks what query Φ q and set of features Φ c an imaginary adversary would pick to maximize the gap between the utilities of Φ q and Φ q under Φ c (that is, u(Φ q , Φ c ) − u(Φ q , Φ c )). The agent wants to find a query Φ q that minimizes the worst case (maximum gap). Under the definition of M R, the agent, reasoning as if it were its imaginary adversary, must find the maximizing Φ q and Φ c . However, we can simplify this so that it only needs to find the maximizing Φ c . Note that since an imaginary adversary chooses both Φ q and Φ c , it wants to make sure that Φ c ⊆ Φ q , which means that it does not want the features not in Φ q to be changeable. We can then observe that M R(Φ q ) = max π ∈Γ: Φ rel (π )≤k (V π − max π∈Π Φ A ? \{Φq ∩Φ rel (π )} V π ) (7) We call the π maximizing Eq. 7 the adversarial policy when the agent asks query Φ q , denoted by π M R Φq . With Eq. 7, the agent can compute M R based on the set of dominating policies (which DomPolicies already found), rather than the (generally much larger) powerset of the relevant features in Eq. 5. While using Eq. 7 is faster than Eq. 5, the agent still needs to do this computation for every possible query (Eq. 6) of size k. We contribute two further ways to improve the efficiency. First, we may not need to consider all relevant features if we can only ask about k of them. If a subset of relevant features satisfies the condition in Theorem 2, then we call it a set of sufficient features, because a minimax-regret k-feature query from that set is a globally minimax-regret k-feature query. Second, we introduce a pruning rule that we call query dominance in Theorem 3 to safely eliminate considering queries that cannot be better than ones already evaluated. The following theorem shows that if we can find any subset Φ of Φ rel such that for all k-feature subsets of Φ as queries, the associated adversarial policy's relevant features are contained in Φ, then the minimax regret query found by restricting queries to be subsets of Φ will also be a minimax regret query found by considering all queries in Φ rel . Such a (nonunique) set Φ will be referred to as a sufficient feature set (for the purpose of finding minimax regret queries). Theorem 2. (Sufficient Feature Set) For any set of ≥ k features Φ, if for all Φ q ⊆ Φ, |Φ q | = k, we have Φ rel (π M R Φq ) ⊆ Φ, then min Φq⊆Φ,|Φq|=k M R(Φ q ) = min Φq⊆Φ rel ,|Φq|=k M R(Φ q ). Proof Sketch. If a set of features Φ, |Φ| ≥ k, fails to include some features in Φ M M R q , when we query some k-subset of Φ, the adversarial policy should change some of the features in Φ M M R q \ Φ. Otherwise, querying about Φ M M R q \ Φ does not reduce the maximum regret. Then Φ M M R q \ Φ are not necessary to be included in Φ M M R q . Given a set of sufficient features, the following theorem shows that it may not be necessary to compute the maximum regrets for all k-subsets to find the minimax-regret query. Theorem 3. (Query Dominance) For any pair of queries Φ q and Φ q , if Φ q ∩ Φ rel (π M R Φq ) ⊆ Φ q ∩ Φ rel (π M R Φq ), then M R(Φ q ) ≥ M R(Φ q ). Proof. Observe that M R(Φ q ) ≥ V π M R Φq − max π ∈Π Φ A ? \(Φ q ∩Φ rel (π M R Φq )) V π ≥ V π M R Φq − max π ∈Π Φ A ? \(Φq ∩Φ rel (π M R Φq )) V π = M R(Φ q ) We denote the condition Φ q ∩ Φ rel (π M R Φq ) ⊆ Φ q ∩ Φ rel (π M R Φq ) by dominance(Φ q , Φ q ). To compute dominance, we only need to store Φ rel (π M R Φq ) for all Φ q we have considered. Algorithm MMRQ-k below provides pseudocode for finding a minimax-regret k-feature query; it takes advantage of both the notion of a sufficient-feature-set as well as query dominance to reduce computation significantly relative to the brute-force approach of searching over all k-feature queries (subsets) of the relevant feature set. Algorithm MMRQ-k 1: Φ q ← an initial k-feature query 2: checked ← ∅; evaluated ← ∅ 3: Φ suf ← Φ q 4: agenda ← {Φ q } 5: while agenda = ∅ do 6: Φ q ← an element from agenda 7: if ¬∃ Φ q ∈evaluated dominance(Φ q , Φ q ) then 8: Compute M R(Φ q ) and π M R Φq 9: Φ suf ← Φ suf ∪ Φ rel (π M R Φq ) 10: evaluated ← evaluated ∪ {Φ q } 11: checked ← checked ∪ {Φ q } 12: agenda ← all k subsets of (Φ suf ) \ checked 13: return arg min Φq∈evaluated M R(Φ q ) Intuitively, the algorithm keeps augmenting the set of features Φ suf , which contain the features in the queries we have considered and the features changed by their adversarial policies, until it becomes a sufficient feature set. agenda keeps track of k-subsets in Φ suf that we have not yet evaluated. According to Theorem 2, we can terminate the algorithm when agenda is empty (Line 5). We also use Theorem 3 to filter out queries which we know are not better than the ones we have found already (Line 7). Note that an initial Φ suf needs to be chosen, which can be arbitrary. Our implementation initializes Φ q with the Chain of Adversaries heuristic (Section 5). To illustrate when Algorithm MMRQ-k can and can't prune suboptimal queries and thus gain efficiency, consider finding the minimax-regret 2-feature query in Figure  2  (a), which should be {c 1 , c 2 }. If the agent considers a query that does not include c 1 , the adversarial policy would change c 1 , adding c 1 to Φ suf . If the agent considers a query that includes c 1 but not c 2 , the adversarial policy would change c 2 , adding c 2 to Φ suf . When the agent asks {c 1 , c 2 }, the adversarial policy changes c 3 (adversarially asserting that c 1 , c 2 are locked), so c 3 is added to Φ suf . With {c 1 , c 2 , c 3 } ⊆ Φ suf , the condition in Theorem 2 holds and the n − 3 other features can be safely ignored. The minimax-regret query constituted by features in Φ suf is {c 1 , c 2 }. However, in Figure  2  (b), Φ suf = Φ rel , and all |Φ rel | 2 would be evaluated.
 Empirical Evaluations We now empirically confirm that Algorithm MMRQ-k finds a minimax-regret query, and its theoretically sound Sufficient-Feature-Set and Query-Dominance based improvements can indeed pay computational dividends. We also compare our MMRQ-k algorithm to baseline approaches and the Chain of Adversaries (CoA) heuristic  [Viappiani and Boutilier, 2009]  adapted to our setting. Algorithm CoA begins with Φ q0 = ∅ and improves this query by iteratively computing: π ← arg max π ∈Γ:|Φ rel (π )∪Φq i |≤k (V π − max π∈Π Φ A ? \{Φq i ∩Φ rel (π )} V π ) Φ qi+1 ← Φ qi ∪ Φ rel (π). The algorithm stops when |Φ qi+1 | = k or Φ qi+1 = Φ qi . Although Algorithm CoA greedily adds features to the query to reduce the maximum regret, unlike MMRQ-k it does not guarantee finding the minimax-regret query. For example, in Figure  2 (c), when k = 2, CoA first finds the optimal policy, which changes {c 1 , c 2 }, and returns that as a query, while the minimax-regret query is {c 1 , c 3 }. We compare the following algorithms in this section: 1. Brute force (rel. feat.) uses Algorithm DomPolicies to find all relevant features first and evaluates all k-subsets of the relevant features. 2. Algorithm MMRQ-k. 3. Algorithm CoA. 4. Random queries (rel. feat.), which contain k uniformly randomly chosen relevant features. 5. Random queries, which contain k uniformly randomly chosen unknown features, without computing relevant features first.
 No queries (equivalently vacuous queries). We evaluate the algorithms' computation times and the quality of the queries they find, reported as the normalized M R to capture their relative performance compared to the best and the worst possible queries. That is, the normalized M R of a query Φ q is defined as (M R(Φ q ) − M R(Φ M M R q ))/(M R(Φ q ⊥ ) − M R(Φ M M R q )), where Φ q ⊥ is a vacuous query, containing k features that are irrelevant and/or already known. The normalized M R of a minimaxregret query is 0 and that of a vacuous query is 1. Navigation. As illustrated in Figure  3 , the robot starts from the left-bottom corner and is tasked to turn off a switch at the top-right corner. The size of the domain is 6 × 6. The robot can move one step north, east or northeast at each time step. It stays in place if it tries to move across a border. The discount factor is 1. Initially, 10 clean carpets are uniformly randomly placed in the domain (the blue cells). In any cell without a carpet, the reward is set to be uniformly random in [−1, 0], and in a cell with a carpet the reward is 0. Hence, the robot will generally prefer to walk on a carpet rather than around it. The state of each carpet corresponds to one feature. The robot is uncertain about whether the user cares about whether any particular carpet gets dirty, so all carpet features are in Φ A ? . The robot knows that its own location and the state of the switch are in Φ A F . Since MMRQ-k attempts to improve on an existing safe policy, the left column and the top row never have carpets to ensure there is at least one safe path to the switch (the dotted line). The robot can ask one k-feature query before it takes any physical actions. We report results on 1500 trials. The only difference between trials are the locations of carpets, which are uniformly randomly placed. First, we compare the brute force method to our MMRQ-k algorithm (Figure  4 ). We empirically confirm that in all cases MMRQ-k finds a minimax regret query, matching Brute Force performance. We also see that brute force scales poorly as k grows while MMRQ-k, benefiting from Theorems 2 and 3, is more computationally efficient. We then want to see if and when MMRQ-k outperforms other candidate algorithms. In Figure  4 , when k is small, the greedy choice of CoA can often find the best features to add to the small query, but as k increases, CoA suffers from being too greedy. When k is large (approaches the number of unknown features), being selective is less important and all methods find good queries. We also consider how |Φ rel | affects the performance (Figure  5 ). When |Φ rel | is smaller   than k, a k-feature query that contains all relevant features is optimal. All algorithms would find an optimal query except Random (which selects from all unknown features) and No Queries. (The error bars are larger for small |Φ rel | since more rarely are only very few features relevant.) When |Φ rel | is slightly larger than k, even a random query may be luckily a minimax-regret query, and CoA unsurprisingly finds queries close to the minimax-regret queries. However, when |Φ rel | is much larger than k, the gap between MMRQ-k and other algorithms is larger. In summary, MMRQ-k's benefits increase with the opportunities to be selective (larger |Φ rel | k ). In Fig- ure 5, the gap is largest when k = 4, |Φ rel | = 10. We have also experimented with expected regret given a probabilistic model of how the user will answer queries. For example, if the user has probability p of saying an unknown feature is free (1 − p it's locked), then as expected, when p is very low, querying rarely helps, so using MMRQ-k or CoA matters little, and as p nears 1, CoA 's greedy optimism pays off to meet MMRQ-k 's minimax approach. But, empirically, MMRQ-k outperforms CoA for non-extreme values of p.
 Extensions and Scalability We now briefly consider applying our algorithms to larger, more complicated problems. As mentioned in Section 2, features' changeabilities might be more nuanced, with restrictions on what values they can take, individually or in combination. An example of the latter from Fig.  1  is where doors are revertible features, which means their changeability is dependent on the "time" feature: they are freely changeable except that by the time the episode ends they need to revert to their initial values. This expands the set of possible feature queries (e.g., asking if d2 is freely changeable differs from asking if it is revertible). In our experiments, this change accentuates the advantages of MMRQ-k over CoA: CoA asks (is-d2-locked, is-d2-revertible), hoping to hear "no" to both and follow a policy going through d2 without closing it. MMRQk asks (is-d2-locked, is-carpet-locked): since closing d2 only takes an extra time step, it is more valuable to know if the carpet is locked than if d2 can be left open. More nuanced feature changeability means DomPolicies will have to find policies for the powerset of every relevant combination of features' values (in the worst case the size of the state space). One option is to ignore nuances in ways that maintain safety (e.g., treat a feature as revertible even if sometimes it can be left changed) and solve such a safe abstraction of the problem. Or one could abstract based on guaranteed correlations in feature changeabilities (e.g., if all boxes have the same changeability then ignore asking about all but one). Another option is to find only a subset of dominating policies, for example by using knowlege of k to avoid finding dominating policies that would change more unknown features than could be asked about anyway. And, of course, as mentioned before, finding approximately-safely-optimal policies in DomPolicies Line 10 would help speed the process (and might be the only option for larger problem domains). Fortunately, such abstractions, heuristics, and approximations do not undermine safety guarantees. Recall that Dom-Policies and MMRQ-k are finding a query, not the agent's final policy: the safety of the agent depends on how it finds the policy it executes, not on the safety of policies for hypothetical changeability conditions. However, as coarser abstractions, heuristics, and approximations are employed in our algorithms, the queries found can increasingly deviate from the minimax-regret optima. Fortunately, if the agent begins with a safely-optimal policy, "quick and dirty" versions of our methods can never harm it (they just become less likely to help). And if it begins without such a policy, such versions of our methods might not guide querying well, but by eventually asking about every unknown feature (in the worst case) a safe policy will still be found (if one exists).
 Related Work & Summary Amodei et al.  [2016]  address the problem of avoiding negative side-effects by penalizing all side-effects while optimizing the value. In our work, we allow the agent to communicate with the user. Safety is also formulated as resolving reward uncertainty  [Amin et al., 2017; Hadfield-Menell et al., 2017] , following imperfectly-specified instructions  [Milli et al., 2017] , and learning safe states  [Laskey et al., 2016] . Safety issues also appear in exploration  [Hans et al., 2008; Moldovan and Abbeel, 2012; Achiam et al., 2017] . Here we only provide a brief survey on safety in MDPs.  Leike et al. [2017] , Amodei et al.  [2016], and Garcıa and Fernández [2015]  provide more thorough surveys. There are problems similar to finding safely-optimal policies, which find policies that satisfy some constraints/commitments or maximize the probability of reaching a goal state  [Witwicki and Durfee, 2010; Teichteil-Königsbuch, 2012; Kolobov et al., 2012] . There are also other works using minimax-regret and policy dominance  [Regan and Boutilier, 2010; Nilim and El Ghaoui, 2005] . and querying to resolve uncertainty  [Weng and Zanuttini, 2013; Regan and Boutilier, 2009; Cohn et al., 2011; Zhang et al., 2017] . We've combined and customized a number of these ideas to find a provably minimax-regret k-element query. In summary, we addressed the problem of an agent selectively querying a user about what features can be safely sideeffected. We borrowed existing ideas from the literature about dominating policies and minimax regret, wove them together in a novel way, and streamlined the resulting algorithms to improve scalability while maintaining safe optimality. Figure 1 : 1 Figure 1: The robot navigation domain. The dominating policies (see Section 3) are shown as arrows.
 a)T (s |s, a) + δ(s , s 0 ) ∀s ∈ S Φ A L ∪Φ A ? , ∀a ∈ A, x(s, a) = 0 The control variables x : S × A → R in the linear program are occupancy measures, i.e., x(s, a) is the expected discounted number of times that state action pair s, a is visited by the agent's policy, S Φ is the set of states where one or more features in Φ have different values from the initial state, and δ(s, s ) = 1 if s = s and is zero otherwise.
 ← ∅ the initial set of dominating policies 2: Φ rel ← ∅ the initial set of relevant features 3: checked ← ∅ It contains Φ ⊆ Φ rel we have examined so far. 4: β ← ∅ a pruning rule 5: agenda ← powerset(Φ rel ) \ checked 6: while agenda = ∅ do 7: Φ ← least-cardinality element of agenda 8: if satisf y(Φ, β) then 9: (get safely-optimal policy with Φ locked) 10: π ← arg max π ∈ΠΦ V π by solving Eq. 1 11: if π exists then 12: Γ ← Γ ∪ {π}; add (Φ, Φ rel (π)) to β 13:
 (a) and (b) only need to compute policies linear in the number of relevant features: Only n+1 dominating safely-optimal policies are computed for Figure 2(a) (for Φ = ∅, {c 1 }, {c 1 , c 2 }, . . . ), and Figure 2(b) (for Φ = ∅, {c 1 }, {c 2 }, . . . , {c n }). Figure 2(c) computes policies for only half of the subsets. Theorem 1. The set of policies returned by Algorithm Dom-Policies is the set of all dominating policies.
 Figure 3 : 3 Figure 3: Office navigation and legend for following figures.
 Figure 4 : 4 Figure 4: Normalized maximum M R vs. k. |Φ ? | = 10. Brute force computation time is only shown for k = 1, 2, 3.
 Figure 5 : 5 Figure 5: Normalized M R vs. the number of relevant features.
			 Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence