Experience Replay More When It's a Key Transition in Deep Reinforcement Learning

1 Introduction . Deep reinforcement learning ( RL ) has shown its promising future for decision-making in various computer games , such as atari ( Mnih et al. , 2013 ; 2015 ) , go ( Schrittwieser et al. , 2020 ) and starcraft ( Vinyals et al. , 2019 ) . However , most successes have been exclusively in simulation due to poor sample efficiency of typical Deep RL algorithm and other challenges . Reinforcement learning can be divided into model-based RL and model-free RL in the light of its data efficiency . Model-free RL is usually subdivided into off-policy RL and on-policy RL . Although model-based RL requires less sampled data , it needs to build a world model and predict the next state based on a lot of prior work , demonstrated in Hafner et al . ( 2019 ) . Although the on-policy RL algorithm do without establishing a world model and has good stability , the performance improves slowly due to limiting the step of updating the policy , a large number of sampled trajectory data are required in the training process ( Schulman et al. , 2015 ; 2017 ) . Off-policy RL is between model-based RL and on-policy RL in terms of sampling efficiency , and the research in this field is enduring ( Watkins & Dayan , 1992 ; Hessel et al. , 2018 ; Barth-Maron et al. , 2018 ) on account of its relatively high data efficiency and world model free . Experience Replay ( Lin , 1992 ) is an important part of improving the data efficiency of Offpolicy RL , which stores experience in a replay buffer and break the temporal correlations by mixing data , therefore , the experience can be used multiple times to update the networks . However , most of the current work is to uniformly sample transitions from the buffer , such as Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) , Twin Delayed Deep Deterministic policy gradient ( TD3 ) ( Fujimoto et al. , 2018 ) and many other algorithms ( Van Hasselt et al. , 2016 ; Mnih et al. , 2016 ; Andrychowicz et al. , 2017 ; Dabney et al. , 2018 ; Liu et al. , 2020 ) . However , these approaches replay experience transitions at the same frequency , regardless of their meaning . Schaul et al . ( 2016 ) develops a framework for prioritizing experience to replay important transitions more frequently , therefore agent learns more efficiently . Prioritized experience replay ( PER ) method samples transitions with the magnitude of their temporal-difference ( TD ) error . However , prioritization introduces bias , which needs to be corrected with importance sampling . Meta-reinforcement learning ( meta-RL ) algorithms enable agents to learn new skills from small amounts of experience ( Rothfuss et al. , 2018 ; Mishra et al. , 2018 ) , Rakelly et al . ( 2019 ) develops an off-policy meta-RL algorithm that disentangles the inference and control of the task , but its performance is still far behind the mainstream off-policy RL algorithms . In this paper , We first analyzed how the state changes from beginning of agent interacting with the environment to the policy converged , and found that the state of the agent is different at different stages , agent will rarely transfer to those states where the policy is far from converging when the agent ’ s policy already converged . If a large number of initialstates1 are used to train Q-value and policy networks , the network weights that have been learned well will be gradually updated . Therefore , the sampling proportion of the most recently generated transitions should be appropriately increased , so that the agent pays more attention to learning the recent transitions , In this way , the state of the agent is gradually transferred from a poor state to a better state , similar to updating from a poor policy to a better policy stabily in Trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) or Proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) . Inspired by the Add Noise to Noise ( AN2N ) Algorithm ( Guo & Gao , 2021 ) , We divide the states into two categories . These states explored with added noise are called key states2 , and the rest are called non-key states . For the sake of improving the performance in key states in time , we have increased the probability of sampling new key states , making them more likely to participate in updating policy , we call this process as ERM ( Experience Replay More ) , and combine it with commonly used off-policy RL algorithms in continuous control tasks , such as SAC , which obtained faster convergence and state-of-the-art performance . 2 Preliminaries . We consider a reinforcement learning setup consisting of agent learning policies to maximize the expected reward when interacting with the environment ( Sutton & Barto , 2018 ) . At each timestep t , the agent receives an observation ot ∈ O , selects action at ∈ A with respect to its policy π : O → A . After taking the action at in environment E , agent receives a reward rt and the next observation ot+1 . The practical problem is usually a partial Markov decision process ( POMDP ) , only part of the observation information could be obtained . To simplify the problem , we assumed the environment is fully-observed , so st = ot , S = O . In reinforcement learning , the action-value function Qπ ( s , a ) is uesed to approximate the expected sum reward of the ation a in state s , defined as following : Qπ ( s , a ) = Est∼pπ , at∼π [ +∞∑ t=0 γtR ( st , at ) ] ( 1 ) Where γ ∈ [ 0 , 1 ] is the discount factor , Est∼pπ , at∼π is the expectation over the distribution of the trajectories ( s0 , a0 , s1 , a1 , . . . ) . The mean value of Qπ in the same state s called the value function V π , defined as V π ( s ) = Ea∼π ( ·|s ) [ Qπ ( s , a ) ] . We express the action-value function Qπ in the form of Bellman equation ( Bellman & Kalaba , 1965 ) : Qπ ( st , at ) = Est+1∼pπ [ r ( st , at ) + γEat+1∼π [ Qπ ( st+1 , at+1 ) ] ] ( 2 ) 1The initial-state is the state in which agent do not perform well when policy has not converged , agent almost no longer transferred to this kind of state after policy converged . 2The key state is a state where the agent has not performed well in the past In this paper , we need to be familiar with DDPG , TD3 and SAC algorithms , here , we mainly introduce DDPG as the basis . DDPG applied two different fully connected neural networks to approximate the action-value function Q ( s , a|θQ ) and policy function µ ( s|θµ ) , DDPG introduces action-value target network θQ ′ and policy target network θµ ′ , so as to Stable the policy update . Consequently , gradient descent is used to optimize the network weights by minimizing the loss : L ( θQ ) = Est∼pµ ( st|θµ ) , at∼µ ( st|θµ ) [ ( Q ( st , at|θ Q ) − yt ) 2 ] ( 3 ) Where yt = r ( st , at ) + γQ ′ ( st+1 , µ ′ ( st+1|θµ ′ ) |θQ ′ ) ( 4 ) ∇θµJ ≈ Es∼p ( st|θµ ) [ ∇θµQ ( s , a|θQ ) |s=st , a=µ ( st|θµ ) ] = Es∼p ( st|θµ ) [ ∇aQ ( s , a|θ Q ) |s=st , a=µ ( st ) ∇θµµ ( st|θ µ ) |s = st ] ( 5 ) Where equation 4 derived from equation 2 , the weights of target networks are updated periodically to slowly track the learned networks : θ′ ← τθ+ ( 1− τ ) θ′ with τ ≪ 1 , which alleviates the fluctuation in the agent ’ s learning process . The policy is updated by equation 5 , following the chain rule to the expected sum return Q ( s , a|θQ ) with respect to parameters θµ . TD3 addresses the problem that DDPG is prone to overestimating the Q function . An additional Q-function is added , and the predicted smaller Q value is used as the calculation of TD-error . At the same time , which also reduces the update frequency of the policy function . The most significant difference between SAC and DDPG is the introduction of policy entropy H ( π ( ·|st ) ) , so the objective function equation 1 is rewritten as : Qπ ( s , a ) = Est∼pπ , at∼π [ +∞∑ t=0 γtR ( st , at ) − α log ( π ( at|st ) ) ] ( 6 ) Where α is temperature parameter , which adjusts the optimization target , agent pays more attention to exploration if increase the coefficient α . 3 Experience Replay More When It ’ s a Key Transition . The addition of experience replay improves the sample efficiency of off-policy RL . Many off-policy algorithms usually sample transitions uniformly from the experience buffer , which potentially considers the transitions generated at different times to be of the same importance , we will discuss this in section 3.1 . Prioritized Experience Replay ( PER ) is an optimization of the uniform sampling method based on the TD-error value of transitions so as to learn the samples more efficiently , however , PER essentially still does not consider whether the transitions generated at different times are equally important to the current agent ’ s policy . 3.1 States are Different at Different Stages Taking the HalfCheetah-v2 simulation environment as an example , we recorded the state information of an agent at different stages based on DDPG algorithm , collecting attitude information at the specific position and the corresponding Q-value starting from 2e4 to 5.6e5 , and then we uniformly sample 100 sets of data from the collected data for drawing , as shown in the Fig . 1 . The legend in the upper left corner represents the Q value , which is discretized to reasonably reduce the number of Legends . Subgraph ( a ) is discreted in units of 20 , and subgraph b is discreted in units of 100 . It can be found when the agent interacts with the environment , not only the policy is gradually improving3 , but the state is also changing accordingly . After policy converges and the agent is in a state set different from the previous one , more experience similar to the agent ’ s recent states should be collected to train the neural network to strengthen the current policy . If the sampling experience for training policy network and the action state network are quite different from the recently generated states , it will not only help improve the policy to a small extent , but will also even forget the newly learned network weights due to the catastrophic forgetting of the neural networks . Therefore , to further improving the sample efficiency , experience replay should be switched from uniform sampling to targeted sampling . 3.2 Replaying More Key Transitions The AN2N algorithm will record the dilemma state sd during the evaluation process , so when agent interacts with the environment , if the current state sc and sd are considered to be highly similar , an additional exploration noise Na will be added . In this paper , the state sc with additional noise is called the key state sk , and the transition containing sk is called the key transition trank . The agent ’ s policy is relatively fragile in sk , if agent has not learned how to get out of this dilemma before , agent is more likely to fall into a series of bad 3Better decision-making ability usually gets a higher Q value states after sk , which greatly reduces the agent ’ s overall performance . Therefore , whether the agent can learn a good policy in sk is very important for improving the performance . It can be seen from equation 2 that reward of the key state sk will affect the Q value of the state at the previous moment through the iteration of the dynamic equation , nevertheless , if we uniformly sample experience for training the networks , there will be the following three problems : • Since most of the experience generated by AN2N are non-key states , the probability of sampling key states sk in the experience buffer is small . • The key state sk is time-sensitive . The new key state needs to participate in training the network as quickly as possible . Otherwise , if agent ’ s state undergoes a relatively change , its role in improving the policy will be declined . • Since the agent ’ s state is gradually changing , more recent the sampled experience , more similar its distribution is to the distribution of the latest experience generated currently , and the more it satisfies the assumption of independent and identical distribution ( iid ) . In response to the first question , we designed two experience buffers to store key transition and non-key transition respectively , and used min ( PrtAN2N , Kt ) to adjust the proportion of sampling key transition , where PrtAN2N is the proportion of the number of key transitions generated by AN2N , Kt is linearly related to the simulation times of the agent . The above ensures that key transitions can be sampled strictly according to the proportion from the experience buffer . For the second and third questions , we will linearly increase the probability of new transition being sampled . Two sampling functions will sample 100 samples in [ 0 , 20 ] to compare the difference of the linear distribution sampling function and the uniform distribution more vividly , the result shown in Fig . 2 . The pseudo code of ERM algorithm is shown in algorithm 1 . Algorithm 1 : ERM Input : Sampling ratio PrtAN2N , Kt , Replay buffer Rnon−key , Rkey , batch size bs1 , bs2 , bssum and AN2N parameters Randomly initialize critic network Q ( s , a|θQ ) and actor µ ( s|θµ ) with weights θQ and θµ Initialize target network Q′ and µ′ with weights θQ ′ ← θQ , θµ ′ ← θµ for episode e ∈ { 1 , ... , M } do Initialize a random process N for action exploration Receive initial observation state s1 for t ∈ { 1 , ... , T } do Execute AN2N action at and observe reward rt and observe new state st+1 if ( AN2N exploring more ) then Store key transitio ( st , at , rt , st+1 ) in Rkey else Store transition ( st , at , rt , st+1 ) in Rnon−key end bs1 = min ( PrtAN2N , Kt ) bs2 = bssum − bs1 Sample bs1 and bs2 transitions with linear distribution in Rkey and Rnon−key for training Run DDPG , SAC or TD3 etc . Algorithm end end

Summary: The paper proposes to set up experience replay such that transitions are emphasized based on whether the AN2N algorithm decides to explore more, and how recent the transition was experienced. It is based on observations that the state distribution can dramatically drift over the course of policy improvement, and the intuition that it might be better to perform updates on states that the current policy is more likely to actually visit (recent transitions). They apply the proposed sampling method, *experience replay more* (ERM), to a variety of replay-based deep RL algorithms and evaluate them empirically.