Goal-Directed Planning via Hindsight Experience Replay

1 Introduction . Monte Carlo Tree Search ( MCTS ) ( Browne et al. , 2012 ) algorithms have shown outstanding results in solving sequential decision-making problems , especially in deterministic transition tasks , such as games . MCTS planners use a forward environment model to build a search tree , estimate the value of each action in the current state , and execute the best-estimated one , respectively . Such a procedure allows finding a “ local ” solution to the decision problem in every decision step by sampling trajectories of possible future policies using the forward model . Although providing local solutions is advantageous in some contexts , it comes at a high computational cost since acting in the environment requires interleaved planning phases and possibly large search trees . These high computational costs have been a barrier in applying MCTS to larger problems , especially those with long planning horizons and huge tree branching factors ( number of actions ) . However , MCTS does not require a training phase and can be deployed immediately . Reinforcement Learning ( RL ) ( Sutton et al. , 1998 ) , on the other hand , aims to find a global solution to the control problem by learning a policy that performs adequately in the whole state space . While this may seem a more desirable outcome , it can often be hard to produce a policy that generalizes satisfactorily across different state regions . For this reason , MCTS algorithms have achieved tremendous success in a wide range of tasks ( Borsboom et al . ; Enzenberger et al. , 2010 ; Ikehata and Ito , 2011 ) . The AlphaZero ( Silver et al. , 2016 ) family of algorithms made it possible to use sequential planners like MCTS in more challenging environments , such as the game of Go , where the AlphaGo agent defeated the world champion , achieving super-human performance . AlphaGo was in turn defeated by the AlphaZero , a version of the algorithm without any heuristic related to the game of Go . AlphaZero combines MCTS with the ability of RL algorithms to generalize across the state-action space by keeping a parametrized policy and value network . Specifically , the policy network biases the exploration during the tree search , and the value network estimates the value of the states corresponding to the search tree leaves , replacing the trajectory-based evaluation usually performed with an ad-hoc evaluation policy called rollout policy . On the other hand , the MCTS algorithm acts as a policy improvement step . It takes as input the current state , as well as the policy and value networks , therefore improving the parametrized policy and generating the samples used to train the networks in a supervised manner . Although a tree search is still required , the deployment is cheaper since it usually requires a humbler planning budget , thanks to the policy and value networks which bias the search by making it more efficient . The algorithm has been successfully applied to different games such as Go , Chess , and Shogi ( Silver et al. , 2017a ) without game-specific heuristics . However , despite its success , the algorithm suffers from a high sample complexity , especially prominent in tasks with sparse reward functions , such as goal-directed planning . In goal-directed tasks , the agent aims to reach a goal state , and in general , the policies take both the current state of the environment and the goal state as input . Usually , the actual reward function in these problems is zero for any transition except the one to the goal state , which gives a positive reward . RL algorithms struggle to optimize sparse reward functions since it might be hard to reach goal states , or even practically impossible during exploration if the task is quite challenging . Therefore , there is not a reward signal to guide the exploration . While in practice , in specific tasks , it is possible to use more informative reward functions , such as a distance from the goal state , such a choice is not possible for every task . Furthermore , it might also lead to sub-optimal solutions since a reward function based on a state distance might generate local-optima in the policy space ( Grzes and Kudenko , 2009 ) . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) is a straightforward method to tackle the problem of sparse-reward functions in goal-directed tasks . HER can be used with any off-policy RL algorithm to extend the training dataset for the value networks . Practically , whenever the goal state is not reached during a training episode , the states visited during the episode are used as alternative goal states and are fed to the network during training . It allows reward signals to be given to the value networks and generalize them to reach the input goal state . In this work , we consider the problem of applying HER to AlphaZero to tackle goal-directed tasks . We provide a scheme that does not involve computationally heavy tree re-weighting procedures or high additional computational costs . Finally , we benchmark the method with simulated environments , and we show a novel application to quantum compiling , where it is extremely hard to find unitary gate sequences from a set to approximate arbitrary single-qubit unitary operators . 2 Preliminaries . 2.1 Goal-Directed Reinforcement Learning . In this work , we focus on goal-directed Reinforcement Learning problems . In such setting , an autonomous agent interacts with an environment to maximize the sum of observed reward signals . Formally , in deterministic transition models , the environment consists of a state space S , a set of goal states G ( that can also be equal to S ) , an action space A , a goal-state dependent reward function r : S × A × G → R , a transition model P : S × A → S , a probability distribution over S for the initial state s0 ∼ µ , and a probability distribution over G for the goal state sg ∼ ν . The behavior of the agent is described by a Markovian stationary goal-dependent policy , π : S×G → ∆ ( A ) , which takes as input the current state s and the goal state sg , and outputs a probability distribution over the actions in A . At the beginning of each episode , an initial state s0 ∼ µ and a goal state sg ∼ ν are sampled . Then , at time step t , the agent observes the current state st , selects an action at ∼ π ( st , sg ) , observes the next state st+1 = P ( st , at ) and it gets the reward signal rt = r ( st , at , sg ) . The goal of the agent is to maximize the expected return E [ Rt ] defined as the expectation of the discounted sum of future rewards Rt = ∑∞ i=t γ i−tri taken over the initial state and goal state . Given a policy π , the value of each state is encoded by the value function V π ( s , sg ) = Eat∼π [ R0|s0 = s ] . Similarly , the action-value function is defined for each stateaction pair , conditioning on the first action of the trajectories , Qπ ( s , a , sg ) = Eat∼π [ R0|s0 = s , a0 = a ] . The goal of maximizing the return can be expressed as finding an optimal policy , π∗ = argmaxπ V π ( s , sg ) , ∀ ( s , sg ) ∈ S × G . 2.2 Monte Carlo Tree Search . We briefly introduce the Monte Carlo Tree Search ( MCTS ) family of algorithms , which combine tree-search algorithms with Monte Carlo sampling to build a search tree iteratively . We focus on the most popular algorithm in the MCTS family , i.e. , the upper confidence bound for trees ( UCT ) ( Kocsis and Szepesvári , 2006 ) . More precisely the general MCTS algorithm consists of four stages ( Browne et al. , 2012 ) : Selection . In the selection stage , the selection policy is applied from the root of the search tree recursively until an unexpanded leaf node is reached . Expansion . In the expansion stage , one or more successors to the previously-found unexpanded node are generated according to the actions available in the node . Evaluation . The newly generated nodes are evaluated , generally through a simulation ( rollout ) using a default policy . Backpropagation . In the last stage , the rewards collected during the selection and evaluation phase are backed up following the previously visited tree branch . UCT performs the selection phase using the Upper Confidence Bound ( UCB1 ) ( Auer et al. , 2002 ) bandit algorithm . Specifically , each node of the search tree maintains statistics related to the future value of each action , including the number of visits , the number of times an action is performed , and the total sum of the returns observed , used to estimate the mean return . Using such statistics UCB1 chooses the next action to perform , an , according to an = argmax i=1 .. K B ( ai ) = Ri , Ti ( n−1 ) + C √ 2 log n Ti ( n− 1 ) , ( 1 ) where K is the number of actions , C is a constant that regulates the exploration-exploitation trade-off , Ti ( n− 1 ) is the number of times action i has been played up to time n− 1 and Ri , Ti ( n−1 ) is the average payoff observed from arm i. UCT recursively updates the values of the nodes from the leaf to the root of the tree , during the backup phase . The algorithm is proved to be consistent , i.e. , it converges to the optimal policy in the limit , and asymmetric , which means that it iteratively builds the search tree by favoring the most promising regions of the tree while still giving a probability of selection to all branches . Such propriety is paramount in some applications where the search tree is extremely large . 2.3 AlphaZero . AlphaZero ( Silver et al. , 2017a ) bridges the gap between RL and MCTS by maintaining a parametric policy , πθ and value function vρ . The policy and value networks are provided as input to the MCTS algorithm to bias the tree search . In contrast , the visit counts in the tree ’ s root are used to build the targets for the supervised training of the policy network . Compared to general MCTS , the selection and evaluation phases are modified . The former phase uses the policy πθ , to bias the exploration according to the policy recommendations . Furthermore , AlphaZero replaces the classic UCT algorithm with PUCT ( Rosin , 2011 ) and performs the selection according to : an = argmax i=1 .. K B ( ai ) = Ri , Ti ( n−1 ) + Cπθ ( s , ai ) √ n 1 + Ti ( n− 1 ) . ( 2 ) By multiplying the PUCT confidence interval with the probability given to ai by πθ , AlphaZero initially prefers actions with high probability and a low visit count but asymptotically prefers actions with high values . In the latter phase , AlphaZero evaluates the leaf nodes using the estimate provided by the value network vρ , instead of performing expensive simulations with a default rollout policy . The training targets of the value network are the actual returns collected during acting . More formally , at each time step t , AlphaZero performs M search iterations of MCTS , starting from the current environment state st and using πθ during the selection phase , and vρ during the evaluation of the leaves . After the search phase , the policy targets pt are constructed using the visit counts at the root of the tree : pt ( ai ) = Nt , i∑ j Nt , j ∀i = 1 , . . . , K , ( 3 ) where Nt , i is the visit count of the i-th action in the search tree at step t , and K is the number of actions . Then at is sampled according to pt and is executed observing the next state st+1 and the reward signal rt . In practice , pt is often constructed as a greedy policy over the action counts , only executing the most explored one . At the end of each episode , the collected rewards are used to build the value network targets Rt . The triples ( st , pt , Rt ) are added to a replay buffer D used during training of the networks . In the original AlphaZero paper , the policy and value networks share the parameter vector θ and AlphaZero minimizes : E ( s , p , R ) ∼D [ ( R− vθ ( s ) ) 2 − pT log πθ ( s ) + c∥θ∥2 ] , ( 4 ) where c controls the amount of L2 regularization .

Summary: The paper addresses multi-goal reinforcement learning. It presents empirical results with a variation of an existing algorithm, Hindsight Experience Replay (HER). Specifically, the paper uses HER alongside AlphaZero.