Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This is an interesting and important paper, it emphasizes and analyzes how policy gradient methods modify their objective functions and how this leads to training differences (and often errors w.r.t. the true objective). I have some minor comments on terminology used that I would like to see properly defined within the paper, but otherwise believe this should be accepted for its useful insights.  

Assorted Comments:
+ Maybe I simply have a difference of opinion or have misunderstood, but I am hesitant to agree that the work is comparing the surrogate *reward* function, but rather the surrogate objective. You'll notice that in the TRPO paper, it is called a surrogate objective not a surrogate reward: https://arxiv.org/pdf/1502.05477.pdf .
+ I think better specification of what exactly is being plotted (pointing to an equation) or defining very concretely what is a surrogate reward or true reward (which I suspect is the objective) will make this paper much clearer.
+ In fact, it was a bit unclear whether the comparisons were of the sampled/observed reward function R(s,a) (provided by the environment and sampling regime) or the objective function often the advantage A(s,a) (or the surrogate objective, GAE, etc.) I assume it should be the latter, but the wording of the paper makes this a bit unclear. I suggest discussing things in terms of objectives not rewards -- unless in fact the paper does approximate reward functions in which case this should be specified in much more detail.  
+ Also, in a lot of places it seems like there's a mixup between rewards and returns. I think typically in the literature reward = r_t and return = V_t (sum of reward). Perhaps, in places the paper truly speaks of rewards, but from the context it seems as though it mainly refers to returns. Examples: " Evidently (since the agent attains a high reward) these estimates are sufficient to consistently improve reward" " This is in spite of the fact that our agents continually improve throughout training, and attain nowhere near the maximum reward possible on each task"
 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
[Summary]
This paper empirically studies the behavior of deep policy gradient algorithms during the optimization. The conclusion is that, while these methods generally improve the policy, their behavior does not comply with the underlying theoretical framework. First, sample gradients obtained with a reasonable batch size have little correlation with each other and with the true gradient. Second, a larger batch size requires a smaller step-size. Third, the value baseline is far from true values and only marginally reduces variance, yet it considerably helps with optimization. Finally, the optimization landscape highly varies with the choice of objective function and the number of samples used to estimate it.

[Decision]
I vote for acceptance. To the best of my knowledge, the findings of this paper are new and not predictable by the current theory. These negative results have some merit as they call for theory that explains the behavior of these algorithms, or an algorithm whose behavior is predictable by the current theory. The paper is well-written, with a few small issues in presentation that should to be addressed in the final revision.

[Comments]
In Fig. 4 (b) it does not look like that the value error is high. It is said that "the learned value function is off by about 50% w.r.t. the underlying true value function." This sentence should be clarified or visualized.

What is \pi in Eq (13) in A1? If it is the agent's current policy, how is it different than \pi_\theta? If \pi corresponds to the distribution of state-action pairs in the replay buffer, how can one obtain a policy \pi that has led to this distribution of states in order to construct the importance sampling ratio?

In 2.2, the claim that a learned value baseline results in significant improvement in performance should be supported by results or reference to previous work.

Figs. 6 and 7 compare the loss surface with different objectives and sample regimes. Do these factors (objective and sample size) affect the part of the parameter space that is visualized (by changing the origin and the update direction), or are they only used to evaluate the values on the z-axis for the same area in the parameter space? Observing a different landscape in a different part of the parameter space is not surprising.

[Minor comments]
- Is V_\theta_{t-1} in Eq (4) a function of state? If so, a (s_t) is missing before the plus sign.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper explores a critical divergence between theory and practice, emphasizing that while deep policy gradient algorithms seem to work in certain cases, they don't seem to be working foor the reasons underlying their derivations. It particularly looks at how closely the sample-based approximation of the objective's gradient aligns with the true gradient of the objective, how accurately learned values match the true expected returns, and how well the optimization landscapes of surrogate objectives line up with the objective of maximizing the return.

I propose accepting this paper, as it reveals a key gap in our understanding of why policy gradient methods work. Such emphasis can suggest why deep RL results tend to be inconsistent and irreplicable, and spark future work on closing the gap between theory and practice. Further, the paper is overall well written.

I primarily would like clarification on the optimization landscape visualizations:

1) Is the step direction the direction of the update actually performed at that time step?

2) Would moving diagonally in this space correspond to a mixture of following the update direction and a normally-distributed random direction? Concretely, in the true reward plot at Step 0 for few state-action pairs in Figure 8, does this suggest that mixing a random direction with the update direction would be better than moving cmopletely in the step direction?

Minor:
Typo in citation "...policy improvement theorem of Kakade and Langford Kakade & Langford (2002)"
review_title: Official Blind Review #2

Meta-review: The paper empirically studies the behaviour of deep policy gradient algorithms, and reveals several unexpected observations that are not explained by the current theory. All three reviewers are excited about this work and recommend acceptance.
Decision: Accept (Talk)