Divide-and-Conquer Monte Carlo Tree Search

1 INTRODUCTION . This is the first sentence of this paper , but it was not the first one we wrote . In fact , the entire introduction section was actually one of the last sections to be added to this manuscript . The discrepancy between the order of inception of ideas and the order of their presentation in this paper probably does not come as a surprise to the reader . Nonetheless , it serves as a point for reflection that is central to the rest of this work , and that can be summarized as “ the order in which we construct a plan does not have to coincide with the order in which we execute it ” . Most standard planners for sequential decision making problems—including Monte Carlo planning , Monte Carlo Tree Search ( MCTS ) and dynamic programming—have a baked-in sequential planning assumption ( Bertsekas et al. , 1995 ; Browne et al. , 2012 ) . These methods begin at either the initial or final state and then plan actions sequentially forward or backwards in time . However , this sequential approach faces two main challenges . ( i ) The transition model used for planning needs to be reliable over long horizons , which is often difficult to achieve when it has to be inferred from data . ( ii ) Credit assignment to each individual action is difficult : In a planning problem spanning a horizon of 100 steps , to assign credit to the first action , we have to compute the optimal cost-to-go for the remaining problem with a horizon of 99 steps , which is only slightly easier than solving the original problem . To overcome these two fundamental challenges , here we consider alternatives to the basic assumptions of sequential planners . We focus on goal-directed decision making problems where an agent should reach a goal state from a start state . Instead of a transition and reward model of the environment , we assume a given goal-directed policy ( the “ low-level ” policy ) and the associated value oracle that returns its success probability on any given task.1 In general , a low-level policy will not be not optimal , e.g . it might be too “ myopic ” to reliably reach goal states that are far away from its current state . We now seek to improve the low-level policy via a suitable sequence of sub-goals that guide it from the start to the final goal , thus maximizing the overall task success probability . This formulation of planning as finding good sub-goal sequences , makes learning of explicit environment models unnecessary , as they are replaced by low-level policies and their value functions . The sub-goal planning problem can still be solved by a conventional sequential planner that begins by searching for the first sub-goal to reach from the start state , then planning the next sub-goal in sequence , and so on . Indeed , this is the approach taken in most hierarchical RL settings based on options or sub-goals ( e.g . Dayan & Hinton , 1993 ; Sutton et al. , 1999 ; Vezhnevets et al. , 2017 ) . However , the credit assignment problem mentioned above persists , as assessing if the first sub-goal is useful still requires evaluating the success probability of the remaining plan . Instead , it could be substantially easier to reason about the utility of a sub-goal “ in the middle ” of the plan , as this breaks the long-horizon problem into two sub-problems with much shorter horizons : how to get to the sub-goal and how to get from there to the final goal . Based on this intuition , we propose the Divide-and-Conquer MCTS ( DC-MCTS ) planner that searches for sub-goals to split the original task into two independent sub-tasks of comparable complexity and then recursively solves these , thereby drastically facilitating credit assignment . To search the space of intermediate sub-goals efficiently , DC-MCTS uses a heuristic for proposing promising sub-goals that is learned from previous search results and agent experience . Humans can plan efficiently over long horizons to solve complex tasks , such as theorem proving or navigation , and some plans even span over decades ( e.g . economic measures ) : In these situations , planning sequentially in terms of next steps – such as what arm to move , or what phone call to make – will cover a tiny proportion of the horizon , neglecting the long uncertainty beyond the last planned step . The algorithm put forward in this paper is a step in the direction of efficient planners that tackle long horizons by recursively and parallelly splitting them into many smaller and smaller sub-problems . In Section 2 , we formulate planning in terms of sub-goals instead of primitive actions . In Section 3 , as our main contribution , we propose the novel Divide-and-Conquer Monte Carlo Tree Search algorithm for this planning problem . In Section 4 we position DC-MCTS within the literature of related work . In Section 5 , we show that it outperforms sequential planners both on grid world and continuous control navigation tasks , demonstrating the utility of constructing plans in a flexible order that can be different from their execution order . 2 IMPROVING GOAL-DIRECTED POLICIES WITH PLANNING . Let S and A be finite sets of states and actions . We consider a multi-task setting , where for each episode the agent has to solve a new task consisting of a new Markov Decision Process ( MDP ) M over S and A. EachM has a single start state s0 and a special absorbing state s∞ , also termed the goal state . If the agent transitions into s∞ at any time it receives a reward of 1 and the episode terminates ; otherwise the reward is 0 . We assume that the agent observes the start and goal states ( s0 , s∞ ) at the beginning of each episode , as well as an encoding vector cM ∈ Rd . This vector provides the agent with additional information about the MDPM of the current episode and will be key to transfer learning across tasks in the multi-task setting . A stochastic , goal-directed policy π is a mapping from S × S × Rd into distributions over A , where π ( a|s , s∞ , cM ) denotes the probability of taking action a in state s in order to get to goal s∞ . For a fixed goal s∞ , we can interpret π as a regular policy , here denoted as πs∞ , mapping states to action probabilities . We denote the value of π in state s for goal s∞ as vπ ( s , s∞|cM ) ; we assume no discounting γ = 1 . Under the above definition of the reward , the value is equal to the success probability of π on the task , i.e . the absorption probability of the stochastic process starting in s0 defined by running πs∞ : vπ ( s0 , s∞|cM ) = P ( s∞ ∈ τ πs∞ s0 |cM ) , 1As we will observe in Section 5 , in practice both the low-level policy and value can be learned . Approximating the value oracle with a learned value function was sufficient for DC-MCTS to plan successfully . where τπs∞s0 is the trajectory generated by running πs∞ from state s0 2 . To keep the notation compact , we will omit the explicit dependence on cM and abbreviate tasks with pairs of states in S × S . 2.1 PLANNING OVER SUB-GOAL SEQUENCES . Assume a given goal-directed policy π , which we also refer to as the low-level policy . If π is not already optimal , we can potentially improve it by planning : If π has a low probability of directly reaching s∞ from the initial state s0 , i.e . vπ ( s0 , s∞ ) ≈ 0 , we will try to find a plan consisting of a sequence of intermediate sub-goals such that they guide π from the start s0 to the goal state s∞ . Concretely , let S∗ = ∪∞n=0Sn be the set of sequences over S , and let |σ| be the length of a sequence σ ∈ S∗ . We define for convenience S̄ : = S ∪ { ∅ } , where ∅ is the empty sequence representing no sub-goal . We refer to σ as a plan for task ( s0 , s∞ ) if σ1 = s0 and σ|σ| = s∞ , i.e . if the first and last elements of σ are equal to s0 and s∞ , respectively . s0S∗s∞ denotes the set of plans for this task . To execute a plan σ , we construct a policy πσ by conditioning the low-level policy π on each of the sub-goals in order : Starting with n = 1 , we feed sub-goal σn+1 to π , i.e . we run πσn+1 ; if σn+1 is reached , we will execute πσn+2 and so on . We now wish to do open-loop planning , i.e . find the plan with the highest success probability P ( s∞ ∈ τπσs0 ) of reaching s∞ . However , this success probability depends on the transition kernels of the underlying MDPs , which might not be known . We can instead define planning as maximizing the following lower bound of the success probability , that can be expressed in terms of the low-level value vπ . Proposition 1 ( Lower bound of success probability ) . The success probability P ( s∞ ∈ τπσs0 ) ≥ L ( σ ) of a plan σ is bounded from below by L ( σ ) : = ∏|σ|−1 i=1 v π ( σi , σi+1 ) , i.e . the product of the success probabilities of π on the sub-tasks defined by ( σi , σi+1 ) . The straight-forward proof is given in Appendix A.1 . Intuitively , L ( σ ) is a lower bound for the success of πσ , as it neglects the probability of “ accidentally ” ( due to stochasticity of the policy or transitions ) running into the goal s∞ before having executed the full plan . We summarize : Definition 1 ( Open-Loop Goal-Directed Planning ) . Given a goal-directed policy π and its corresponding value oracle vπ , we define planning as maximizing L ( σ ) over σ ∈ s0S∗s∞ , i.e . the set of plans for task ( s0 , s∞ ) . We define the high-level ( HL ) value v∗ ( s0 , s∞ ) : = maxσ L ( σ ) as the maximum value of the planning objective . Note the difference between the low-level value vπ and the high-level v∗ . vπ ( s , s′ ) is the probability of the agent directly reaching s′ from s following π , whereas v∗ ( s , s′ ) the probability reaching s′ from s under the optimal plan , which likely includes intermediate sub-goals . In particular , v∗ ≥ vπ . 2.2 AND/OR SEARCH TREE REPRESENTATION . In the following we cast the planning problem into a representation amenable to efficient search . To this end , we use the natural compositionality of plans : We can concatenate a plan σ for the task ( s , s′ ) and a plan σ̂ for the task ( s′ , s′′ ) into a plan σ ◦ σ̂ for the task ( s , s′′ ) . Conversely , we can decompose any given plan σ for task ( s0 , s∞ ) by splitting it at any sub-goal s ∈ σ into σ = σl ◦ σr , where σl is the “ left ” sub-plan for task ( s0 , s ) , and σr is the “ right ” sub-plan for task ( s , s∞ ) . Trivially , the planning objective and the optimal high-level value factorize wrt . to this decomposition : L ( σl ◦ σr ) = L ( σl ) L ( σr ) v∗ ( s0 , s∞ ) = max s∈S̄ v∗ ( s0 , s ) · v∗ ( s , s∞ ) . This allows us to recursively reformulate planning as : arg max s∈S̄ ( arg max σl∈s0S∗s L ( σl ) ) · ( arg max σr∈sS∗s∞ L ( σr ) ) . ( 1 ) The above equations are the Bellman equations and the Bellman optimality equations for the classical single pair shortest path problem in graphs , where edge weights are given by − log vπ ( s , s′ ) . We can represent this planning problem by an AND/OR search tree ( Nilsson , N. J. , 1980 ) with alternating levels of OR and AND nodes . An OR node , also termed an action node , is labeled by a task 2We assume MDPs with multiple absorbing states such that this probability is not trivially equal to 1 for most policies , e.g . uniform policy . In experiments , we used a finite episode length . ( s , s′′ ) ∈ S × S ; the root of the search tree is an OR node labeled by the original task ( s0 , s∞ ) . A terminal OR node ( s , s′′ ) has a value vπ ( s , s′′ ) attached to it , which reflects the success probability of πs′′ for completing the sub-task ( s , s′′ ) . Each non-terminal OR node has |S|+ 1 AND nodes as children . Each of these is labeled by a triple ( s , s′ , s′′ ) for s′ ∈ S̄ , which correspond to inserting a sub-goal s′ into the overall plan , or not inserting one in case of s = ∅ . Every AND node ( s , s′ , s′′ ) , or conjunction node , has two OR children , the “ left ” sub-task ( s , s′ ) and the “ right ” sub-task ( s′ , s′′ ) . In this representation , plans are induced by solution trees . A solution tree Tσ is a sub-tree of the complete AND/OR search tree , with the properties that ( i ) the root ( s0 , s∞ ) ∈ Tσ , ( ii ) each OR node in Tσ has at most one child in Tσ and ( iii ) each AND node in Tσ as two children in Tσ . The plan σ and its objective L ( σ ) can be computed from Tσ by a depth-first traversal of Tσ . The correspondence of sub-trees to plans is many-to-one , as Tσ , in addition to the plan itself , contains the order in which the plan was constructed . Figure 6 in Section 5.3 shows an example for a search and solution tree . Below we will discuss how to construct a favourable search order heuristic .

Summary: This paper proposes Divide-and-Conquer Monte Carlo Tree Search (DC-MCTS) for for goal-directed planning problems (i.e. problems where reaching a specific goal state is the objective, like traversing a maze with specified start and goal positions). The assumed setting is one where transition and reward models of the environment are not (necessarily) available, but a low-level goal-directed policy that can attempt to navigate from a given start to a given goal position, as well as value oracle that can return the success probability of the low-level policy on any given task, are available. Planning problems are modelled as AND/OR search trees, where OR nodes are labelled by a start state s and a goal state s'', and AND nodes are labelled by triples (s, s', s''). An OR node has children for every possible state, such that traversing to a child indicates the insertion of the corresponding state as an additional subgoal in between s and s', plus one extra child to indicate the choice of returning the current plan without inserting any additional subgoals. AND nodes have two children; one OR node for the first half (s, s') of the plan, and a second OR node for the second half (s', s'') of the plan. The MCTS can construct a plan by inserting subgoals such that they become easier to solve for the low-level policy by searching this tree.