Bootstrapped Hindsight Experience replay with Counterintuitive Prioritization

1 INTRODUCTION . Deep reinforcement learning ( DRL ) has gained significant achievements in solving games ( Zha et al. , 2019 ; Crespo & Wichert , 2020 ; Oh et al. , 2021 ) and robotics problems ( Rahimi et al. , 2018 ; Nguyen & La , 2019 ; Zhu et al. , 2021 ) . Despite these successes , sparse rewards problems are still challenging . In sparse reward tasks , the agent gets a non-negative reward only when it achieves some goal . As a consequence , it is very difficult for an agent to see successful trajectories . Indeed , many state-of-the-art RL algorithms , such as TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DQN ( Mnih et al. , 2015 ) , and DDPG ( Lillicrap et al. , 2019 ) , etc. , often fail to perform well in sparse rewards environments . To solve sparse rewards problems , there have been rich literature devoted to reward engineering . Potential Based Reward Shaping ( PBRS ) ( Ng et al. , 1999 ) uses difference between two potential functions to design a shaping function , where the potential function comes from the expertise knowledge and evaluates the value of state . Potential Based state-action Advice ( PBA ) ( Wiewiora et al. , 2003 ) extend PBRS to use potential function to evaluates the value of state-action pair . Dynamic Potential Based ( DPB ) ( Devlin & Kudenko , 2012 ) approach takes time as one of the inputs of the potential function . However , reward engineering usually requires domain-specific knowledge and it often suffers from sub-optimal performance . A brilliant idea for solving goal-conditioned tasks is using hindsight experiences . HER ( Andrychowicz et al. , 2017 ) , which replaces the desired goal in failed experiences with some achieved state in the failed trajectory to pretend that the agent obtains a positive reward , has been demonstrated its effectiveness in a wide range of goal-conditioned appli- cations . However , HER has its own limitations that when relabeling the goal in one failed trajectory , there exists many choices of the pseudo goals that any achieved state can be selected . Generally , uniformly sampling one of them , as HER did , is not the most effective way from the perspective of importance sampling . In this paper , we focus on improving the data efficiency in HER via the following two directions : 1 ) inspired by Bootstrapped DQN ( Osband et al. , 2016 ) , we propose a bootstrapped version of HER that employs multiple Q-value heads to increase the diversity in choosing the pseudo goals for deep exploration . In this paper , we use DDPG as the basic RL algorithm , while the proposed techniques can be integrated by any off-the-shelf RL algorithms ; 2 ) in addition to the benefit on exploration by using the bootstrapped Q-learning , we further explicitly leverage the uncertainty among the multiple heads by sampling goals conditioning on the variance of the estimated Q-values . Many previous approaches have shown that data with higher value uncertainty can promote exploration , and maximizing the uncertainty by incorporating a bonus term in the reward has been widely considered , such as bandit algorithms ( Lattimore & Szepesvári , 2020 ) . To our surprise , this is totally not the case in HER . On the contrast , sampling goals with lower value uncertainty first is demonstrated to be significantly better than uniform sampling , and sampling goals with high uncertainty in prior deteriorates the performance of HER a lot . We thus call this empirical fact as counterintuitive prioritization . Indeed , this can be explained by the trade-off between exploration and exploitation when importing HER , bootstrapped Q-learning and importance sampling . We refer our algorithm to as the Bootstrapped HER with counterintuitive prioritization ( BHER ) . We experiment with a number of control tasks with continuous action space . The experimental results verify the superiority of BHER over many state-of-the-art baseline methods . 2 RELATED WORK . Sparse rewards in goal-conditioned environment is one of the major challenges of RL at present . The earlier work is UVF ( Schaul et al. , 2015 ) algorithm , which uses state and goal together as the conditions for agent to take action . Bootstrapped DQN ( Osband et al. , 2016 ) uses the bootstrap principle to expand network structure . It uses multiple randomized value functions to estimate the Q-value of the state-action pair . This is equivalent to agent having multiple policies to interact with the environment , which effectively enhances agent ’ s exploration . But these methods don ’ t work well in the environment of sparse rewards . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) algorithm recalculates reward by randomly replacing the desired goal in the failed transition with the achieved goal . In this way , pseudo successful data can be obtained , which can improve data efficiency . HER algorithm better solves the problem of sparse rewards . Since then , there have been many studies related to HER algorithm . Dynamic Hindsight Experience Replay ( DHER ) ( Fang et al. , 2019a ) utilizes the HER algorithm to process dynamic goals in the environment . Competitive Experience Replay ( CER ) ( Liu et al. , 2019 ) uses the competition between two agents to obtain better exploration . Goal-Conditioned Supervised Learning ( GCSL ) ( Ghosh et al. , 2020 ) is a kind of imitation learning . It uses successful trajectories generated by agent as expert demonstrates , and use supervised learning to optimize a lower bound on the goal-oriented RL objective . PlanGAN ( Charlesworth & Montana , 2020 ) is a model-based RL algorithm that uses GANs ( Creswell et al. , 2018 ) to learn the model of the environment and make planning . But this method requires a huge amount of computation to select a single action . Recently , researchers have realized that not all data have the same value to learn ( Schulman et al. , 2017 ) . More of the work is to further improve data efficiency by adding priority replay on HER algorithm . Energy-Based HER ( Zhao & Tresp , 2018 ) calculates the kinetic energy , potential energy and rotational energy as the total energy of the achieved goal according to the energy principle in physics . Then it gives higher priority to transitions with larger energy . Maximum Entropybased Prioritization ( MEP ) ( Zhao et al. , 2020 ) calculates the entropy of a trajectory which is used as priority . Curriculum-guided HER ( CHER ) ( Fang et al. , 2019b ) calculates the proximity and diversity between the achieved goal and the desired goal by designing two functions , which are added together with a weight to balance exploration and exploitation . These methods can be regarded as a form of curriculum learning ( Elman , 1993 ; Bengio et al. , 2009 ; Zaremba & Sutskever , 2014 ; Graves et al. , 2017 ) . But these algorithms either make some assumptions about the environment or design some specific functions to calculate the priority . Unlike the above works , our BHER algorithm does not have any requirements on the environment when calculating the priority , and it is easy to generalize to other goal-conditioned environments . 3 PRELIMINARIES . In this section , we introduce the preliminaries , such as the challenge of goal-conditioned environment , HER ( Andrychowicz et al. , 2017 ) and Bootstrapped DQN ( Osband et al. , 2016 ) which motivate us to propose our BHER algorithm . 3.1 GOAL-CONDITIONED ENVIRONMENT . In RL , the problem considered is usually composed of an agent and an environment . At time step t , the agent takes action at conditioned on the state st of the environment . After the environment receives the action at , it transfers to the state st+1 and feeds back the reward rt . The agent learns a policy π ( a|s ) to maximize expected discounted return : G = Eτ [ T−1∑ t=0 γtrt ] , ( 1 ) where τ is a trajectory ( s0 , a0 , r0 , ... , sT−1 , aT−1 , rT−1 ) and γ is the discount factor . In goal-conditioned environment , the agent takes an action a by taking an additional input , i.e. , a goal g , and the policy is hence a function of ( s , g ) that a ∼ π ( ·|s , g ) . Generally , the rewards in goal-conditioned environments are sparse , and only when the agent reaches the goal , it gets a nonnegative feedback . Therefore , it is difficult for the agent to explore successful trajectories when its policy starts from scratch . 3.2 HER AND BOOTSTRAPPED DQN . HER replaces the desired goal g in the failed transition with some sampled achieved state g′ in the current trajectory , and then recalculates the reward based on the pseudo goal g′ . In this way , the agent can see successful trajectories much more frequently , even if the goal is fake . It has been demonstrated in rich literature that HER is very effective for solving goal-conditioned tasks and it can be adopted by any off-policy algorithms , such as DQN ( Mnih et al. , 2015 ) , DDPG ( Lillicrap et al. , 2019 ) , TD3 ( Fujimoto et al. , 2018 ) , etc . Bootstrapped DQN designs multiple value function heads which are initialized randomly in the network structure of the DQN algorithm , and all heads share the underlying network . Each time agent interacts with the environment , one of the value function heads is randomly selected as policy to generate a trajectory . Each value function head is updated according to its own target network for temporally extended exploration , and each value function head only uses data generated by itself for updates . Bootstrapped DQN subtly applies the Thompson sampling ( Russo et al. , 2017 ) heuristic to RL which allows for deep exploration . 4 METHODOLOGY . In this section , we introduce the bootstrapped version of HER with multiple Q-value heads and the counterintuitive prioritization . 4.1 BOOTSTRAPPED HER . We use DDPG as the basic RL algorithm . In goal-conditioned tasks , the policy and Q-value take both the state and goal as input . A bootstrapped version of DDPG is dipicted in Fig . 1 , where both the actor network and critic network are implemented with multiple heads . All critic heads share an underlying embedding network , similar to what was used in Bootstrapped DQN ( Osband et al. , 2016 ) , and actor heads share an underlying network for feature extraction as well . Each actor head is updated according to its corresponding critic head , for which its pseudo goals are sampled independently and it is updated by according to its own recomputed rewards . As has been demonstrated in Bootstrapped DQN , random initialization of the parameters of each head is sufficient to generate diversity for deep exploration . This is also verified in our experiments for BHER . In this way , we can benefit a lot in terms of exploration with the cost of increasing only a small amount of the network parameters from multiple heads , since they share a large proportion of parameters in the embedding network . In Appendix D , we provide an illustration of the distribution of goals the agent can arrive in a toy environment by using multiple heads . Compared to the standard HER , we can verify that the bootstrapped version of HER is much more efficient in terms of exploration .

Summary: # Summary & Contributions * The authors call attention to sparse-reward, goal-based tasks where Hindsight Experience Replay, through its provision of relabeled goals for otherwise failed experiences facilitates efficient learning. * To further improve upon HER, the paper focuses on two algorithmic innovations: (1) maintaining multiple actor-critic pairs inspired by BootstrappedDQN and (2) applying a variant of prioritized experience replay that draws samples from the replay buffer inversely proportional to the current Q-value variance under the critic ensemble. * The authors support their approach with comparative experiments to other variants of HER as well as ablation studies of their own proposed method.