The problem with DDPG: understanding failures in deterministic environments with sparse rewards

1 INTRODUCTION . The Deep Deterministic Policy Gradient ( DDPG ) algorithm ( Lillicrap et al . ( 2015 ) ) is one of the earliest deep Reinforcement Learning ( RL ) algorithms designed to operate on potentially large continuous state and action spaces with a deterministic policy , and it is still one of the most widely used . However , it is often reported that DDPG suffers from instability in the form of sensitivity to hyper-parameters and propensity to converge to very poor solutions or even diverge . Various algorithms have improved stability by addressing well identified issues , such as the over-estimation bias in TD3 ( Fujimoto et al. , 2018b ) but , because a fundamental understanding of the phenomena underlying these instabilities is still missing , it is unclear whether these ad hoc remedies truly address the source of the problem . Thus , better understanding why these algorithms can fail even in very simple environments is a pressing question . To investigate this question , we introduce in Section 4 a very simple one-dimensional environment with a sparse reward function where DDPG sometimes fails . Analyzing this example allows us to provide a detailed account of these failures . We then reveal the existence of a cycle of mechanisms operating in the sparse reward and deterministic case , leading to the quick convergence to a poor policy . In particular , we show that , when the reward is not discovered early enough , these mechanisms can lead to a deadlock situation where neither the actor nor the critic can evolve anymore . Critically , this deadlock persists even when the agent is subsequently trained with rewarded samples . The study of these mechanisms is backed-up with formal proofs in a simplified context where the effects of function approximation is ignored . Nevertheless , the resulting understanding helps analyzing the practical phenomena encountered when using actors and critics represented as neural networks . From this new light , we revisit in Section 5 a few existing algorithms whose components provide an alternative to the building blocks involved in the undesirable cyclic convergence process , and we suggest alternative solutions to these issues . 2 RELATED WORK . Issues when combining RL with function approximation have been studied for a long time ( Baird & Klopf , 1993 ; Boyan & Moore , 1995 ; Tsitsiklis & Van Roy , 1997 ) . In particular , it is well known that deep RL algorithms can diverge when they meet three conditions coined as the ” deadly triad ” ( Sutton & Barto , 2018 ) , that is when they use ( 1 ) function approximation , ( 2 ) bootstrapping updates and ( 3 ) off-policy learning . However , these questions are mostly studied in the continuous state , discrete action case . For instance , several recent papers have studied the mechanism of this instability using DQN ( Mnih et al. , 2013 ) . In this context , four failure modes have been identified from a theoretical point of view by considering the effect of a linear approximation of the deep-Q updates and by identifying conditions under which the approximate updates of the critic are contraction maps for some distance over Q-functions ( Achiam et al. , 2019 ) . Meanwhile , van Hasselt et al . ( 2018 ) shows that , due to its stabilizing heuristics , DQN does not diverge much in practice when applied to the ATARI domain . In contrast to these papers , here we study a failure mode specific to continuous action actor-critic algorithms . It hinges on the fact that one can not take the maximum over actions , and must rely on the actor as a proxy for providing the optimal action instead . Therefore , the failure mode identified in this paper can not be reduced to any of the ones that affect DQN . Besides , the theoretical derivations provided in the appendices show that the failure mode we are investigating does not depend on function approximation errors , thus it can not be directly related to the deadly triad . More related to our work , several papers have studied failure to gather rewarded experience from the environment due to poor exploration ( Colas et al. , 2018 ; Fortunato et al. , 2017 ; Plappert et al. , 2017 ) , but we go beyond this issue by studying a case where the reward is actually found but not properly exploited . Finally , like us the authors of Fujimoto et al . ( 2018a ) study a failure mode which is specific to DDPG-like algorithms , but the studied failure mode is different . They show under a batch learning regime that DDPG suffers from an extrapolation error phenomenon , whereas we are in the more standard incremental learning setting and focus on a deadlock resulting from the shape of the Q-function in the sparse reward case . 3 BACKGROUND : DEEP DETERMINISTIC POLICY GRADIENT . The DDPG algorithm ( Lillicrap et al. , 2015 ) is a deep RL algorithm based on the Deterministic Policy Gradient theorem ( Silver et al. , 2014 ) . It borrows the use of a replay buffer and target networks from DQN ( Mnih et al. , 2015 ) . DDPG is an instance of the Actor-Critic model . It learns both an actor function πψ ( also called policy ) and a critic function Qθ , represented as neural networks whose parameters are respectively noted ψ and θ . The deterministic actor takes a state s ∈ S as input and outputs an action a ∈ A . The critic maps each state-action pair ( s , a ) to a value in R. The reward r : S × A → R , the termination function t : S ×A→ { 0 , 1 } and the discount factor γ < 1 are also specified as part of the environment . The actor and critic are updated using stochastic gradient descent on two losses Lψ and Lθ . These losses are computed from mini-batches of samples ( si , ai , ri , ti , si+1 ) , where each sample corresponds to a transition si → si+1 resulting from performing action ai in state si , with subsequent reward ri = r ( si , ai ) and termination index ti = t ( si , ai ) . Two target networks πψ′ andQθ′ are also used in DDPG . Their parameters ψ′ and θ′ respectively track ψ and θ using exponential smoothing . They are mostly useful to stabilize function approximation when learning the critic and actor networks . Since they do not play a significant role in the phenomena studied in this paper , we ignore them in the formal proofs given in appendices . Equations ( 1 ) and ( 2 ) define Lψ and Lθ : Lψ = − ∑ i Qθ ( si , πψ ( si ) ) ( 1 ) ∀i , yi = ri + γ ( 1− ti ) Qθ′ ( si+1 , πψ′ ( si+1 ) ) Lθ = ∑ i [ Qθ ( si , ai ) − yi ] 2 . ( 2 ) Training for the loss given in ( 1 ) yields the parameter update in ( 3 ) , with α the learning rate : ψ ← ψ + α ∑ i ∂πψ ( si ) ∂ψ T ∇aQθ ( si , a ) |a=πψ ( si ) . ( 3 ) As DDPG uses a replay buffer , the mini-batch samples are acquired using a behaviour policy β which may be different from the actor π . Usually , β is defined as π plus a noise distribution , which in the case of DDPG is either a Gaussian function or the more sophisticated Ornstein-Uhlenbeck noise . Importantly for this paper , the behaviour of DDPG can be characterized as an intermediate between two extreme regimes : • When the actor is updated much faster than the critic , the policy becomes greedy with respect to this critic , resulting into a behaviour closely resembling that of the Q-LEARNING algorithm . When it is close to this regime , DDPG can be characterized as off-policy . • When the critic is updated much faster than the actor , the critic tends towards Qπ ( s , a ) . The problems studied in this paper directly come from this second regime . A more detailed characterization of these two regimes in given in Appendix A . 4 A NEW FAILURE MODE . In this section , we introduce a simplistic environment which we call 1D-TOY . It is a one-dimensional , discrete-time , continuous state and action problem , depicted in Figure 1 . Despite its simplicity , DDPG can fail on 1D-TOY . We first show that DDPG fails to reach 100 % success . We then show that if learning a policy does not succeed soon enough , the learning process can get stuck . Besides , we show that the initial actor can be significantly modified in the initial stages before finding the first reward . We explain how the combination of these phenomena can result into a deadlock situation . We generalize this explanation to any deterministic and sparse reward environment by revealing and formally studying a undesirable cyclic process which arises in such cases . Finally , we explore the consequences of getting into this cyclic process . 4.1 EMPIRICAL STUDY . In all experiments , we set the maximum episode length N to 50 , but the observed phenomena persist with other values . Residual failure to converge using different noise processes We start by running DDPG on the 1D-TOY environment . This environment is trivial as one infinitesimal step to the left is enough to obtain the reward , end the episode and succeed , thus we might expect a quick 100 % success . However , the first attempt using an Ornstein-Uhlenbeck ( OU ) noise process shows that DDPG succeeds in only 94 % of cases , see Figure 2a . These failures might come from an exploration problem . Indeed , at the start of each episode the OU noise process is reset to zero and gives little noise in the first steps of the episode . In order to remove this potential source of failure , we replace the OU noise process with an exploration strategy similar to -greedy which we call ” probabilistic noise ” . For some 0 < p < 1 , with probability p , the action is randomly sampled ( and the actor is ignored ) , and with probability 1− p no noise is used 0 20k 40k 60k 80k 100k Simulation steps 85 90 95 100 % o f su cc es sf u l ru n s Probabilistic noise OU noise ( a ) Success rate of DDPG with Ornstein-Uhlenbeck ( OU ) and probabilistic noise . Even with probabilistic noise , DDPG fails on about 1 % of the seeds . 0 20k 40k 60k 80k 100k Simulation steps 92 94 96 98 100 % o f su cc es sf u l ru n s Baseline Switch to π∗ after 20k steps ( b ) Comparison between DDPG with probabilistic noise and a variant in which the behaviour policy is set to the optimal policy π∗ after 20k steps . Figure 2 : Success rate of variants of DDPG on 1D-TOY over learning steps , averaged over 10k seeds . More details on learning algorithm and success evaluation are given in Appendix E. and the raw action is returned . In our tests , we used p = 0.1 . This guarantees at least a 5 % chance of success at the first step of each episode , for any policy . Nevertheless , Figure 2a shows that even with probabilistic noise , about 1 % of seeds still fail to converge to a successful policy in 1D-TOY , even after 100k training steps . All the following tests are performed using probabilistic noise . We now focus on these failures . On all failing seeds , we observe that the actor has converged to a saturated policy that always goes to the right ( ∀s , π ( s ) = 0.1 ) . However , some mini-batch samples have non-zero rewards because the agent still occasionally moves to the left , due to the probabilistic noise applied during rollouts . The expected fraction of non-zero rewards is slightly more than 0.1 % 1 . Figure 3a shows the occurrence of rewards in minibatches taken from the replay buffer when training DDPG on 1D-TOY . After each rollout ( episode ) of n steps , the critic and actor networks are trained n times on minibatches of size 100 . So for instance , a failed episode of size 50 is followed by a training on a total of 5000 samples , out of which we expect more than 5 in average are rewarded transitions . More details about the implementation are available in Appendix E. The constant presence of rewarded transitions in the minibatches suggests that the failures of DDPG on this environment are not due to insufficient exploration by the behaviour policy . Correlation between finding the reward early and finding the optimal policy We have shown that DDPG can get stuck in 1D-TOY despite finding the reward regularly . Now we show that when DDPG finds the reward early in the training session , it is also more successful in converging to the optimal policy . On the other hand , when the first reward is found late , the learning process more often gets stuck with a sub-optimal policy . From Figure 3b , the early steps appear to have a high influence on whether the training will be successful or not . For instance , if the reward is found in the first 50 steps by the actor noise ( which happens in 63 % of cases ) , then the success rate of DDPG is 100 % . However , if the reward is first found after more than 50 steps , then the success rate drops to 96 % . Figure 3b shows that finding the reward later results in lower success rates , down to 87 % for runs in which the reward was not found in the first 1600 steps . Therefore , we claim that there exists a critical time frame for finding the reward in the very early stages of training . Spontaneous actor drift At the beginning of each training session , the actor and critic of DDPG are initialized to represent respectively close-to-zero state-action values and close-to-zero actions . Besides , as long as the agent does not find a reward , it does not benefit from any utility gradient . Thus we might expect that the actor and critic remain constant until the first reward is found . Actually , we 110 % of steps are governed by probabilistic noise , of which at least 2 % are the first episode step , of which 50 % are steps going to the left and leading to the reward . show that even in the absence of reward , training the actor and critic triggers non-negligible updates that cause the actor to reach a saturated state very quickly . To investigate this , we use a variant of 1D-TOY called DRIFT where the only difference is that no rewarded or terminal transitions are present in the environment . We also use a stripped-down version of DDPG , removing rollouts and using random sampling of states and actions as minibatches for training . Figure 4b shows that even in the absence of reward , the actor function drifts rapidly ( notice the horizontal scale in steps ) to a saturated policy , in a number of steps comparable to the ” critical time frame ” identified above . The critic also has a transitive phase before stabilizing . In Figure 4a , the fact that maxs , a |Q ( s , a ) | can increase in the absence of reward can seem counterintuitive , since in the loss function presented in Equation ( 2 ) , |yi| can never be greater than maxs , a |Q ( s , a ) | . However , it should be noted that the changes made to Q are not local to the minibatch points , and increasing the value of Q for one input ( s , a ) may cause its value to increase for other inputs too , which may cause an increase in the global maximum of Q . This phenomenon is at the heart of the over-estimation bias when learning a critic ( Fujimoto et al. , 2018b ) , but this bias does not play a key role here .

Summary: Overview: This paper describes a shortfall with the DDPG algorithm on a continuous state action space with sparse rewards. To first prove the existence of this shortfall, the authors demonstrate its theoretical possibility by reviewing the behavior of DDPG actor critic equations and the “two-regimes” proofs in the appendices. They then demonstrate the occurrence of the critic being updated faster than the actor, leading to a sub-optimal convergence from which the model can never recover. In this demonstration, they use a very simple environment they created, “1D-Toy”. The 1D-Toy environment is a one-dimensional, discrete-time, continuous state and action problem. Moving to the left at all in 1D-Toy results in a reward and episode end. Episode length was set at 50, as the agent could move to the right forever and never stop the episode. The authors demonstrate how the failure of the agent to obtain 100% success in this simple environment was, in fact, due to the phenomenon mentioned earlier. If the agent managed to obtain a reward very early on in training, it was highly likely the agent would converge on an optimal solution. If not, the actor would drift to a state were it no longer updates, and the critic would similarly no longer update either, resulting in a deadlock and suboptimal policy. The authors then generalize their findings using a helpful figure (Figure 7) which describes the cyclical nature of the phenomenon and how it can happen in any environment. Finally, the authors mention potential solutions to prevent the training failure from occurring, such as avoiding sparse rewards, replacing the critic update to avoid loss, etc.