Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper investigates why DDPG can sometimes fail in environments with sparse rewards. It presents a simple environment that helps the reader build intuition and supports the paper's empirical investigation. First, the paper shows that DDPG fails on the simple environment in ~6% of cases, despite the solution being trivial—go left from the start state. The paper then augments DDPG with epsilon-greedy-style exploration to see if the cause of these failures is simply inadequate exploration. Surprisingly, in 1% of cases DDPG still fails. The paper shows that even in these failure cases there were still rewarded transitions that could have been learned from, and investigates relationships between properties of individual runs and the likelihood of failure. The paper then explains how these failures occur: the policy drifts to always going right, and the critic converges to a piecewise constant function whose gradient goes to zero and prevents further updates to the policy. The paper then generalizes this deadlock mechanism to other continuous-action actor-critic algorithms like TD3 and discusses how function approximation helps mitigate this issue. Finally, the paper gives a brief overview of some existing potential solutions.

Currently, I recommend rejecting this paper; while it is very well-written and rigorously investigates a problem with a popular algorithm, the paper does not actually present a novel solution method. It does a great job of clearly defining and investigating a problem and shows how others have attempted to solve it in the past, but stops there. It doesn't propose a new method and/or compare the existing methods empirically, nor does it recommend a specific solution method.

A much smaller concern is that the problem investigated is somewhat niche; it happens in a very small percentage of runs and is mitigated by the type of function approximation commonly used. This lowers its potential impact a little.

The introduction does a great job of motivating the paper. I would've liked the related work section to elaborate more on the relationships between the insights in this paper and those of Fujimoto et al. (2018a), since the paper says the insights are related. Section 3 gave a very clear overview of DDPG. The simple environment described in section 4 was intuitive and explained well, and the empirical studies were convincing.

However, after section 4 established the existence of the deadlock problem in DDPG, I was expecting the rest of the paper to present a solution method and empirically validate it. Instead, section 5 generalizes the findings from section 4 to include other environments and algorithms. I felt that section 4 and 5 could have been combined and shortened, and the extra space used to explain and empirically test a novel solution method. For example, figures 6 and 7 seem to be conveying similar information, but figure 7 takes up almost half a page.

Currently this paper seems like a good candidate for a workshop. It convincingly establishes the existence of a problem and shows what causes the problem to occur, but doesn't contribute a solution to the problem. Submitting it to a workshop could be a good opportunity for discussion, feedback, and possible collaboration, all of which might help inspire a solution method.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Overview: This paper describes a shortfall with the DDPG algorithm on a continuous state action space with sparse rewards. To first prove the existence of this shortfall, the authors demonstrate its theoretical possibility by reviewing the behavior of DDPG actor critic equations and the “two-regimes” proofs in the appendices. They then demonstrate the occurrence of the critic being updated faster than the actor, leading to a sub-optimal convergence from which the model can never recover. In this demonstration, they use a very simple environment they created, “1D-Toy”. The 1D-Toy environment is a one-dimensional, discrete-time, continuous state and action problem. Moving to the left at all in 1D-Toy results in a reward and episode end. Episode length was set at 50, as the agent could move to the right forever and never stop the episode. The authors demonstrate how the failure of the agent to obtain 100% success in this simple environment was, in fact, due to the phenomenon mentioned earlier. If the agent managed to obtain a reward very early on in training, it was highly likely the agent would converge on an optimal solution. If not, the actor would drift to a state were it no longer updates, and the critic would similarly no longer update either, resulting in a deadlock and suboptimal policy. The authors then generalize their findings using a helpful figure (Figure 7) which describes the cyclical nature of the phenomenon and how it can happen in any environment. Finally, the authors mention potential solutions to prevent the training failure from occurring, such as avoiding sparse rewards, replacing the critic update to avoid loss, etc.

Contributions: the discovery and review of a potential training failure for DDPG due to the nature of the critic update being reliant on the policy, and the deterministic policy gradient update

Questions and Comments:
I believe that this work is relevant to ICLR and to the field. The paper is well-written, the theory is sound, and the experiment is sufficient to describe the stated deadlock situation that DDPG can contain during training. I believe this paper should be accepted because of these reasons. I have a few comments/questions for the authors which I have written below.

I’m interested to see how likely this deadlock situation is on more complex environments. Did you run experiments on common benchmarks and analyze them?
You mention several potential solutions, one of them being the avoidance of sparse rewards. Of course this is problem-dependent which you stated yourself. The other two involve replacing two of the update functions. In the policy-based critic update, both of the mentioned solutions in this section have drawbacks mentioned. Would using a stochastic policy gradient update affect the networks ability to learn successfully in more complex environments? Would this make training less stable?

I’m curious to see in which directions you see this work being extended. You briefly mention in the conclusion that there would be more formal studies: how do you imagine these being?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
This work studies the instability problem of DDPG in the setting of a deterministic
environment and sparse reward. This work designed a toy environment to showcase
the potential issues of leading to the instability of DDPG. The observations, such as the correlation between the early access of good trajectory leads to more stable performance later, the deadlock of training could be beneficial. 
It is essential to analyze and understand the intrinsic properties of the classic algorithms, and it would benefit the research community a lot if the empirical study is appropriately designed and conducted. Overall, this paper studied an essential problem
of the vulnerability of the classic DRL algorithm (DDPG), which should attract more attention and efforts
from the research community. 


Detailed comments:
Methodology:
The experiments conducted cannot support the conclusions in this paper.
I can not fully understand the conclusion from the subsection "Residual failure to converge using different noise processes". The DDPG agent is finding the reward regularly while it couldn't converge to the 100% optimal performance. In my opinion, this is an observation,
instead of giving any useful conclusion. The convergent issues when using the combination of off-policy
learning, function approximation, and bootstrapping are known (Sutton and Barto, Chap 11, 2018). 

The increasing of Q value seems natural to me due to the overestimation of Q learning,
even with the zero reward setting, which is one of the motivations of Double DQN. 

Several potential solutions are discussed while no empirical evidence or theoretical
justification is provided, even in the designed 1D-Toy example.

It would be more convincing that the conclusions can be validated on more challenging
tasks such as regular continuous action benchmarks (mujoco, etc.)

Writing:
The presentation of this work is not ready for publication, given its current form. 
What is the definition of reward function? The formula as shown in Eq 4e is not clear. 
What is the optimal performance of 1D-TOY example? 
review_title: Official Blind Review #3

Meta-review: This paper provides an extensive investigation of the robustness of Deep Deterministic Policy Gradient algorithm.

Papers providing extensive and qualitative empirical studies, illustrative benchmark domains, identification of problems with existing methods, and new insights can be immensely valuable, and this paper is certainly in this direction, if not quite there yet. 

The vast majority of this paper investigates one deep learning algorithm in designed domain. There is some theory but it's relegated to the appendix. There are a few issues with this approach: (1) there is no concrete evidence that this is a general issue beyond the provided example (more on that below). (2) Even in the designed domain the problem is extremely rare. (3) The study and perhaps even the issue is only shown for one particular architecture (with a whole host of unspecified meta-parameter details). Why not just use SAC it works? DDPG has other issues, why is it of interest to study and fix this particular architecture? The motivation that it is the first and most popular algorithm is not well developed enough to be convincing. (4) There is really no reasoning to suggest that the particular 1D is representative or interesting in general.

The authors including Mujoco results to address #1. But the error bars overlap, its completely unclear if the baseline was tuned at all---this is very problematic as the domains were variants created by the authors. If DDPG was not tuned for the variant then the plots are not representative. In general, there are basically no implementation details (how parameters were tested, how experiments were conducted)or general methodological details given in the paper. Given the evidence provided in this paper its difficult to claim this is a general and important issue. 

I encourage the authors to look at John Langfords hard exploration tasks, and broaden their view of this work general learning mechanisms. 
Decision: Reject