Goal-Directed Planning via Hindsight Experience Replay

1 Introduction . Monte Carlo Tree Search ( MCTS ) ( Browne et al. , 2012 ) algorithms have shown outstanding results in solving sequential decision-making problems , especially in deterministic transition tasks , such as games . MCTS planners use a forward environment model to build a search tree , estimate the value of each action in the current state , and execute the best-estimated one , respectively . Such a procedure allows finding a “ local ” solution to the decision problem in every decision step by sampling trajectories of possible future policies using the forward model . Although providing local solutions is advantageous in some contexts , it comes at a high computational cost since acting in the environment requires interleaved planning phases and possibly large search trees . These high computational costs have been a barrier in applying MCTS to larger problems , especially those with long planning horizons and huge tree branching factors ( number of actions ) . However , MCTS does not require a training phase and can be deployed immediately . Reinforcement Learning ( RL ) ( Sutton et al. , 1998 ) , on the other hand , aims to find a global solution to the control problem by learning a policy that performs adequately in the whole state space . While this may seem a more desirable outcome , it can often be hard to produce a policy that generalizes satisfactorily across different state regions . For this reason , MCTS algorithms have achieved tremendous success in a wide range of tasks ( Borsboom et al . ; Enzenberger et al. , 2010 ; Ikehata and Ito , 2011 ) . The AlphaZero ( Silver et al. , 2016 ) family of algorithms made it possible to use sequential planners like MCTS in more challenging environments , such as the game of Go , where the AlphaGo agent defeated the world champion , achieving super-human performance . AlphaGo was in turn defeated by the AlphaZero , a version of the algorithm without any heuristic related to the game of Go . AlphaZero combines MCTS with the ability of RL algorithms to generalize across the state-action space by keeping a parametrized policy and value network . Specifically , the policy network biases the exploration during the tree search , and the value network estimates the value of the states corresponding to the search tree leaves , replacing the trajectory-based evaluation usually performed with an ad-hoc evaluation policy called rollout policy . On the other hand , the MCTS algorithm acts as a policy improvement step . It takes as input the current state , as well as the policy and value networks , therefore improving the parametrized policy and generating the samples used to train the networks in a supervised manner . Although a tree search is still required , the deployment is cheaper since it usually requires a humbler planning budget , thanks to the policy and value networks which bias the search by making it more efficient . The algorithm has been successfully applied to different games such as Go , Chess , and Shogi ( Silver et al. , 2017a ) without game-specific heuristics . However , despite its success , the algorithm suffers from a high sample complexity , especially prominent in tasks with sparse reward functions , such as goal-directed planning . In goal-directed tasks , the agent aims to reach a goal state , and in general , the policies take both the current state of the environment and the goal state as input . Usually , the actual reward function in these problems is zero for any transition except the one to the goal state , which gives a positive reward . RL algorithms struggle to optimize sparse reward functions since it might be hard to reach goal states , or even practically impossible during exploration if the task is quite challenging . Therefore , there is not a reward signal to guide the exploration . While in practice , in specific tasks , it is possible to use more informative reward functions , such as a distance from the goal state , such a choice is not possible for every task . Furthermore , it might also lead to sub-optimal solutions since a reward function based on a state distance might generate local-optima in the policy space ( Grzes and Kudenko , 2009 ) . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) is a straightforward method to tackle the problem of sparse-reward functions in goal-directed tasks . HER can be used with any off-policy RL algorithm to extend the training dataset for the value networks . Practically , whenever the goal state is not reached during a training episode , the states visited during the episode are used as alternative goal states and are fed to the network during training . It allows reward signals to be given to the value networks and generalize them to reach the input goal state . In this work , we consider the problem of applying HER to AlphaZero to tackle goal-directed tasks . We provide a scheme that does not involve computationally heavy tree re-weighting procedures or high additional computational costs . Finally , we benchmark the method with simulated environments , and we show a novel application to quantum compiling , where it is extremely hard to find unitary gate sequences from a set to approximate arbitrary single-qubit unitary operators . 2 Preliminaries . 2.1 Goal-Directed Reinforcement Learning . In this work , we focus on goal-directed Reinforcement Learning problems . In such setting , an autonomous agent interacts with an environment to maximize the sum of observed reward signals . Formally , in deterministic transition models , the environment consists of a state space S , a set of goal states G ( that can also be equal to S ) , an action space A , a goal-state dependent reward function r : S × A × G → R , a transition model P : S × A → S , a probability distribution over S for the initial state s0 ∼ µ , and a probability distribution over G for the goal state sg ∼ ν . The behavior of the agent is described by a Markovian stationary goal-dependent policy , π : S×G → ∆ ( A ) , which takes as input the current state s and the goal state sg , and outputs a probability distribution over the actions in A . At the beginning of each episode , an initial state s0 ∼ µ and a goal state sg ∼ ν are sampled . Then , at time step t , the agent observes the current state st , selects an action at ∼ π ( st , sg ) , observes the next state st+1 = P ( st , at ) and it gets the reward signal rt = r ( st , at , sg ) . The goal of the agent is to maximize the expected return E [ Rt ] defined as the expectation of the discounted sum of future rewards Rt = ∑∞ i=t γ i−tri taken over the initial state and goal state . Given a policy π , the value of each state is encoded by the value function V π ( s , sg ) = Eat∼π [ R0|s0 = s ] . Similarly , the action-value function is defined for each stateaction pair , conditioning on the first action of the trajectories , Qπ ( s , a , sg ) = Eat∼π [ R0|s0 = s , a0 = a ] . The goal of maximizing the return can be expressed as finding an optimal policy , π∗ = argmaxπ V π ( s , sg ) , ∀ ( s , sg ) ∈ S × G . 2.2 Monte Carlo Tree Search . We briefly introduce the Monte Carlo Tree Search ( MCTS ) family of algorithms , which combine tree-search algorithms with Monte Carlo sampling to build a search tree iteratively . We focus on the most popular algorithm in the MCTS family , i.e. , the upper confidence bound for trees ( UCT ) ( Kocsis and Szepesvári , 2006 ) . More precisely the general MCTS algorithm consists of four stages ( Browne et al. , 2012 ) : Selection . In the selection stage , the selection policy is applied from the root of the search tree recursively until an unexpanded leaf node is reached . Expansion . In the expansion stage , one or more successors to the previously-found unexpanded node are generated according to the actions available in the node . Evaluation . The newly generated nodes are evaluated , generally through a simulation ( rollout ) using a default policy . Backpropagation . In the last stage , the rewards collected during the selection and evaluation phase are backed up following the previously visited tree branch . UCT performs the selection phase using the Upper Confidence Bound ( UCB1 ) ( Auer et al. , 2002 ) bandit algorithm . Specifically , each node of the search tree maintains statistics related to the future value of each action , including the number of visits , the number of times an action is performed , and the total sum of the returns observed , used to estimate the mean return . Using such statistics UCB1 chooses the next action to perform , an , according to an = argmax i=1 .. K B ( ai ) = Ri , Ti ( n−1 ) + C √ 2 log n Ti ( n− 1 ) , ( 1 ) where K is the number of actions , C is a constant that regulates the exploration-exploitation trade-off , Ti ( n− 1 ) is the number of times action i has been played up to time n− 1 and Ri , Ti ( n−1 ) is the average payoff observed from arm i. UCT recursively updates the values of the nodes from the leaf to the root of the tree , during the backup phase . The algorithm is proved to be consistent , i.e. , it converges to the optimal policy in the limit , and asymmetric , which means that it iteratively builds the search tree by favoring the most promising regions of the tree while still giving a probability of selection to all branches . Such propriety is paramount in some applications where the search tree is extremely large . 2.3 AlphaZero . AlphaZero ( Silver et al. , 2017a ) bridges the gap between RL and MCTS by maintaining a parametric policy , πθ and value function vρ . The policy and value networks are provided as input to the MCTS algorithm to bias the tree search . In contrast , the visit counts in the tree ’ s root are used to build the targets for the supervised training of the policy network . Compared to general MCTS , the selection and evaluation phases are modified . The former phase uses the policy πθ , to bias the exploration according to the policy recommendations . Furthermore , AlphaZero replaces the classic UCT algorithm with PUCT ( Rosin , 2011 ) and performs the selection according to : an = argmax i=1 .. K B ( ai ) = Ri , Ti ( n−1 ) + Cπθ ( s , ai ) √ n 1 + Ti ( n− 1 ) . ( 2 ) By multiplying the PUCT confidence interval with the probability given to ai by πθ , AlphaZero initially prefers actions with high probability and a low visit count but asymptotically prefers actions with high values . In the latter phase , AlphaZero evaluates the leaf nodes using the estimate provided by the value network vρ , instead of performing expensive simulations with a default rollout policy . The training targets of the value network are the actual returns collected during acting . More formally , at each time step t , AlphaZero performs M search iterations of MCTS , starting from the current environment state st and using πθ during the selection phase , and vρ during the evaluation of the leaves . After the search phase , the policy targets pt are constructed using the visit counts at the root of the tree : pt ( ai ) = Nt , i∑ j Nt , j ∀i = 1 , . . . , K , ( 3 ) where Nt , i is the visit count of the i-th action in the search tree at step t , and K is the number of actions . Then at is sampled according to pt and is executed observing the next state st+1 and the reward signal rt . In practice , pt is often constructed as a greedy policy over the action counts , only executing the most explored one . At the end of each episode , the collected rewards are used to build the value network targets Rt . The triples ( st , pt , Rt ) are added to a replay buffer D used during training of the networks . In the original AlphaZero paper , the policy and value networks share the parameter vector θ and AlphaZero minimizes : E ( s , p , R ) ∼D [ ( R− vθ ( s ) ) 2 − pT log πθ ( s ) + c∥θ∥2 ] , ( 4 ) where c controls the amount of L2 regularization .

Summary:  The authors discuss the impact of Monte Carlo Tree Search (MCTS) algorithms.  MCTS allows us to roll out trajectories using state value estimates to determine local actions, however computational cost can be high. This means that the approach is prohibitive for large problem spaces.  RL seeks to learn a control policy that generalizes well over the state space. This can be challenging and MCTS algorithms can help with this.  In goal reaching problems the agent takes as input the current state of the env as well as the goals state.  The only trajectories that confer positive reward are those where the goal is reached. RL can struggle here as the rewards are sparse due to the success criteria.  More informative reward functions can be used (e.g. distance to goal) but this isn't always possible.  Hindsight Experience Replay (HER) is applied to offline RL algorithms to improve sample efficiency where a replay buffer is maintained over retained trajectories and a set of sub-goals are defined over visited states with returns applied accordingly.  The authors propose to apply HER to AlphaZero to solve goal directed tasks and thereby avoid computationally intense tasks. They evaluate the approach against a set of simulated environments and compare to AlphaZero, including a novel Quantum compiling environment.  Much of the challenge in solving games like Go and Chess comes down to the size of the state space.  AlphaZeroHER works by sampling new sub-goals from already visited states and thus avoiding both the high computational cost involved in tree re-weighting and also the problem of sparse rewards.   This is done by retracing episodes and sample M subgoals from a trajectory of length T and training the policy and value networks on these hindsight inferred goals.  Since AlphaZero is on on-policy the policies used during play are retained when computing updates from the sub-goal rollouts which alleviates the computational burden.  The authors show that AlphaZeroHER outperforms AlphaZero over a number of simulated environments where the latter often fails to achieve learning much while the former achieves strong results.