A Closer Look at Deep Policy Gradients
Authors: Andrew Ilyas
Logan Engstrom
Shibani Santurkar
Dimitris Tsipras
Firdaus Janoos
Larry Rudolph
Aleksander Madry
Venue:  (2020)
Keywords: deep policy gradient methods
deep reinforcement learning
trpo
ppo

Abstract:     We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: surrogate rewards do not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "true" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods.

Published as a conference paper at ICLR 2020
A CLOSER LOOK AT DEEP POLICY GRADIENTS
Andrew Ilyas*, Logan Engstrom*, Shibani Santurkar', Dimitris Tsipras!, Firdaus Janoos?, Larry Rudolph and Aleksander Madry'
MIT
Two Sigma
(ailyas, engstrom, shibani, tsipras, madry) @mit eduu rudolphecsal psa mit. edu, firdaus. janoos@t wma. com
ABSTRACT
We study how the behavior of deep policy gradient algorithms reflects the con- ceptual framework motivating their development. To this end, we propose a fine- grained analysis of state-of-the-art methods based on key elements of this frame- work: gradient estimation, value prediction, and optimization landscapes. Our re- sults show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "truc" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods.
INTRODUCTION
Deep reinforcement learning (RL) is behind some of the most publicized achievements of modern machine learning (Silver et al. 2017 OpenAl 2018 Dayarathna et al. 2016 OpenAl et al. 2018) In fact, many. this framework embodies the promise of the real-world impact of machine learning. However, the deep RL toolkit has not yet attained the same level of engineering stability as, for example, the current deep (supervised) learning framework. Indeed, recent studies demonstrate that state-of-the-art deep RL algorithms suffer from oversensitivity to hyperparameter choices, lack of consistency, and poor reproducibility (Henderson et al. 2017).
This state of affairs suggests that it might be necessary to re-examine the conceptual underpinnings of deep RL methodology. More precisely, the overarching question that motivates this work is:
To what degree does current practice in deep RL reflect the principles informing its development?
Our specific focus is deep policy gradient methods, widely used class of deep RL algorithms. Our goal is to explore the extent to which state-of-the-art implementations of these methods succeed realizing the key primitives of the general policy gradient framework.
Our contributions. We take a broader look at policy gradient algorithms and their relation to their underlying framework. With this perspective in mind, perform a fine- tne-grained examination of key RL primitives as they manifest in practice. Concretely, we study;
we find that even when agents improve in reward, their gradient estimates used in parameter updates poorly correlate with the "true" gradient. We addition- ally show that gradient estimate quality decays with training progress and task complexity. Finally, we demonstrate that varying the sample regime yields training dynamics that are unexplained by the motivating framework and contrary to supervised learning intuition.
Gradient Estimation:
Value Prediction: our experiments indicate that value networks successfully solve the supervised learning task they are trained on, but do not fit the true value function. Addi- tionally, employing a value network as baseline function only marginally decreases the
"Equal contribution. Work done in part an intern Two Sigma.

---

Published as a conference paper at ICLR 2020
variance of gradient estimates compared to using true value as baseline (but still dramat- ically increases agent's 's performance compared to using no baseline at all).
Optimization Landscapes: we show that the optimization landscape induced by modern policy gradient algorithms is often not reflective of the underlying true reward landscape, and that the latter is frequently poorly behaved in the relevant sample regime.
Overall, our results demonstrate that the motivating theoretical framework for deep RL algorithms often unpredictive of phenomena arising practice. This suggests that building reliable deep RL. algorithms requires moving past benchmark-centric evaluations to multi-faceted understanding of their often unintuitive behavior. conclude (in Section 3 by discussing several areas where such understanding is most critically needed.
EXAMINING THE PRIMITIVES OF DEEP POLICY GRADIENT ALGORITHMS
In this section, we investigate the degree to which our theoretical understanding of RL applies to modern methods. We consider key primitives of policy gradient algorithms: gradient estimation, value prediction and reward fitting. In what follows, we perform a fine fine-grained analysis of state-of- the-art policy gradient algorithms (PPO and TRPO) through the lens of these primitives detailed preliminaries, background, and notation can be found in Appendix A.
2.1 GRADIENT ESTIMATE QUALITY
central premise of policy gradient methods that stochastic gradient ascent on suitable objective function yields a good policy. These algorithms use as primitive the gradient of that objective function:
To(ao((s)) Ano ($ ai) E (ss, 9 (se [o(ae(s))
V Votolay (s)) To (ar|s:)
Ano (sc, ai)
(1)
where in the above we standard RL notation (see Appendix A.I for more details). An underlying assumption behind these methods is that we have access to reasonable estimate of this quantity. This assumption effectively translates into an assumption that we can accurately estimate the expec- tation above using an empirical mean of finite (typically ~ 109) samples. Evidently (since the agent attains high reward) these estimates are sufficient consistently improve reward- we are thus interested in the relative quality of these gradient estimates in practice, and the effect of gradient quality on optimization.
Iteration: D
# Iteration: 150
Iteration: 300
Iteration: 450
Aug. pairwise cos sim
TAPO
Aug. painviso cas sim
TRPO
TRPO
Avg. pairwise COS sim
TRPO
Avg: Sainwita COS
0.5
3.5
39
221
23
-05
127
121
10
-35
102
221
31'
121
State-Action Pairs
#
State-Action Pairs
State-Action Pairs
State-Action Pairs
Figure 1: Empirical variance of the estimated gradient (c.f. ( function of the number of state- action pairs used in estimation in the MuloCo Humanoid task. We measure the average pairwise cosine similarity between ten repeated gradient measurements taken from the same policy, with the 95% confidence intervals (shaded). For each algorithm, perform multiple trials with the same hyperparameter configurations but different random seeds, shown as repeated lines in the figure. The vertical line (at 2K) indicates the sample regime used for gradient estimation in standard implementations of policy gradient methods. In general, it seems that obtaining tightly concentrated gradient estimates would require significantly more samples than used in practice, particularly after the first few timesteps. For other tasks such as Walker2d-v2 and Hopper-v2 - the plots (seen in Appendix Figure have similar trends, except that gradient variance is slightly lower. Confidence intervals calculated with 500 sample bootstrapping.
2

---

Published as a conference paper at ICLR 2020
Avg. cos sim with trun grad
Iteration:
Aug. cos sim with true grad
Iteration: 150
Aug. grad
Iteration:
Avg. COS sim with true grad
Iteration: 450
24
TRPO PPO
100
L.
TRPO
2.4
TRPO PPO
PPO
PPO
20
00
80
-03
Staae-AAtion
State-Action
State-Action Pairs
Stato-Action
Figure 2: Convergence of gradient estimates (c.f. (T) to the "true" expected gradient in the MuJoCo Humanoid task. We measure the mean cosine similarity between the "true" gradient approximated using ten million state-action pairs, and ten gradient estimates which use increasing numbers of state-action pairs (with 95% confidence intervals). For each algorithm, we perform multiple trials with the same hyperparameter configurations but different random seeds. The vertical line (at x = 2K) indicates the sample regime used for gradient estimation in standard implementations of policy gradient methods. Observe that although it is possible to empirically estimate the true gradient, this requires several-fold more samples than are used commonly in practical applications of these algorithms. See additionally that the estimation task becomes more difficult further into training. For other tasks - such as Walker2d-v2 and Hopper-v2 - the plots (seen in Appendix Figure IO] have similar trends, except that gradient estimation slightly better. Confidence intervals calculated with 500 sample bootstrapping.
How accurate are the gradient estimates we compute? To answer this question, we examine two of the most natural measures of estimate quality: the empirical variance and the convergence the "true" gradient. To evaluate the former, we measure the average pairwise cosine similarity between estimates of the gradient computed from the same policy with independent rollouts (Figure [.. We evaluate the latter by first forming an estimate of the true gradient with a large number of state-action pairs. We then examine the convergence of gradient estimates to this "true" gradient (which we once again measure using cosine similarity) as we increase the number of samples (Figure 2).
We observe that deep policy gradient methods operate with relatively poor estimates of the gradient, especially as task complexity increases and as training progresses (contrast Humanoid-v2, "hard" task, to other tasks and contrast successive checkpoints in Figures [ and D This is spite of the fact that our agents continually improve throughout training, and attain nowhere near the maximum reward possible on each task. In fact, we sometimes observe a zero or even negative correlation in the relevant sample regime"
While these results might be reminiscent of the well-studied "noisy gradients" problem in supervised learning (Robbins & Monro 1951 d' Aspremont, 20088; Kawaguchi 2016, Safran & Shamir 2018
Livni et al.j 2014; Keskar et al.) 2016 Hochreiter & Schmidhuber 1997). we have very little un- derstanding of how gradient quality affects optimization in the substantially different reinforcement learning setting. For example:
The sample regime in which RL algorithms operate seems to have profound impact on the robustness and stability of agent training-in particular, many of the sensitivity issues reported by Henderson et al. (2017) are claimed to disappear (Sutskever, 2018) in higher- sample regimes. Understanding the implications of working in this sample regime, and more generally the impact of sample complexity on training stability remains to be pre- cisely understood.
Agent policy networks are trained concurrently with value networks (discussed more in the following section) meant reduce the variance of gradient estimates. Under our conceptual framework, we might expect these networks to help gradient estimates more as training progresses, contrary what we observe in Figure [ The value network also makes the now two-player optimization landscape and training dynamics even more difficult to grasp, as such interactions are poorly understood.
DDep policy gradient algorithms gradients indirectly to compute steps-in Appendix results also hold true for these computed steps.
A.4 we show that
3

---

Published as a conference paper at ICLR 2020
The relevant measure of sample complexity for many settings (number of state-action pairs) can differ drastically from the number of independent samples used at each training itera- tion (the number of complete trajectories). The latter quantity (a) tends to be much lower than the number of state-action pairs, and (b) decreases across iterations during training.
All the above factors make it unclear to what degree our intuition from classical settings transfer to the deep RL regime. And the policy gradient framework, as of now, provides little predictive power regarding the variance of gradient estimates and its impact on reward optimization.
Our results indicate that despite having rigorous theoretical framework for RL, we lack precise understanding of the structure of the reward landscape and optimization process.
2.2
VALUE PREDICTION
Our findings from the previous section motivate deeper look into gradient estimation. After all, the policy gradient in its original formulation (Sutton et al. 1999) is known to be hard to estimate, and thus algorithms employ a variety of variance reduction methods. The most popular of these techniques is a baseline function. Concretely, an equivalent form of the policy gradient is given by:
90 ETOTTO
Σ (BLIa))ET
Va log (u(ae(s)) (Q ar) - b(se))
where ((xe) is some fixed function the state 8t. A canonical choice of baseline function is the value function V*(s), the expected return from given state (more details and motivation in A.1
V* (8t) Erg [R][s:]
Indeed, fitting a value-estimating function (Schulman et al.) 2015c Sutton & Barto 2018) (a neural network, in the deep RL setting) and using baseline function precisely the approach taken by most deep policy gradient methods. Concretely, one trains a value network Vēg such that:
(Vo. An))
0: = min E
(4)
where V (st) are estimates given by the last value function, and A, is the advantage of the policy, iee the returns minus the estimated values. (Typically, Ag is estimated using generalized advantage estimation, as described in (Schulman et al. (201cc).) Our findings in the previous section prompt us take closer look at the value network and its impact on the variance of gradient estimates.
2
TRPO
29
PPO
Returns MRE
GAE MRE
2-3
2-1
2-5
2-3
TRPO
2-7
2
PPO
200 # Iterations
500
100
200
500
Iterations
Figure 3: Quality of value prediction in terms of mean relative error (MRE) heldout state-action pairs for agents trained to solve the MuJoCo Walker2d-v2 task. We observe (left) that the agents do indeed succeed at solving the supervised learning task they trained for-the MRE on the GAE-based value loss (Vota 2 (c.f. (4) small. On the other hand, in (right) we see that the returns MRE is still quite high-the learned value function is off by about 50% with respect the underlying true value function. Similar plots for other MuJoCo tasks are Appendix A.5
4

---

Published as a conference paper at ICLR 2020
Itaration: 150
Iteration: 300
Iteration:
Avg gainvise COC sim.
6.25
Baselings: Zaro
Baselinez:
Zero
0.22
Zero
V4
A.55
8
selrvic
0.04
0.00
State-Action Pairs
State-Action Pairs
* State-Action Pairs
Walker2d-v2
# heeation: on: 150
Iteration: 300
Ittrration 450
Avg. pairwise cos sim
Baseiines:
Beseinies Injes:
Baseiras:
coS
6.15
V
Aug. calrwe 6 8
0.25
V
622
y
c.3
0.45
560
Avg.
2006
4390
6093
2003
1886
2990
.299
2003
10993
2003
0000
5008
5000
10089
# State Action Pairs
State-Action Pairs
SSrate Action Pairs
(b) Hopper-v2
Figure 4: Efficacy of the value network as a variance reducing baseline for Walker2d-v2 (top) and Hopper-v2 (bottom) agents. We measure the empirical variance of the gradient (c.f. (1 as function of the number of state-action pairs used in estimation, for different choices of baseline functions: the value network (used by the agent in training), the "true" value function (fit to the returns us- ing 10 state-action pairs sampled from the current policy) and the "zero" value function (i.e. signif-
replacing advantages with returns). We observe that using the true value function leads icantly fower-varianee estimate of the gradient compared to the value network. In turn, employing the value network yields noticeable variance reduction compared to the zero baseline function, even though this difference may appear rather small in the small-sample regime (2K). Confidence intervals calculated with 10 sample bootstrapping.
Value prediction as supervised learning problem. We first analyze the value network through the lens of the supervised learning problem it solves. After all, (4) describes an empirical risk minimization, where loss is minimized over a set of sampled (st, az). So, how does V perform as a solution to (4)? And in turn, how does (4) perform as proxy for learning the true value function?
Our results (Figure 3ab show that the value network does succeed at both fitting the given loss func- tion and generalizing to unseen data, showing low and stable mean relative error (MRE). However, the significant drop in performance shown in Figure Tigure3Jindinn (iddiictsss that the supervised learning prob- lem induced by (4) does not lead to V learning the underlying true value function.
Does the value network lead to a reduction in variance? Though evaluating the Vğ baseline function as a value predictor as we did above is informative, in the end the sole purpose of the value function is to reduce variance. So: how does using our value function actually impact the variance of our gradient estimates? To answer this question, we compare the variance reduction that results from employing our value network against both "true" value function and a trivial "zero" baseline function (i.e. simply replacing advantages with returns). Our results, captured in Figure show that the "true" value function yields a much lower-variance estimate of the gradient. This is especially true the sample regime in which we operate. We note, however, that despite not effectively predicting the true value function inducing the same degree of variance reduction, the value network does help to some degree (compared to the "zero" baseline). Additionally, the seemingly marginal increase in gradient correlation provided by the value network (compared to the "true" baseline function) turns out to result in a significant improvement in agent performance. (Indeed, agents trained without a baseline reach almost order of magnitude worse reward.)
Our findings suggest that we still need a better understanding of the role of the value network in agent training, and raise several questions that discuss in Section
5

---

Published as a conference paper at ICLR 2020
2.3 EXPLORING THE OPTIMIZATION LANDSCAPE
Another key assumption of policy gradient algorithms is that first-order updates (w.... policy pa- rameters) actually yield better policies. It is thus natural to examine how valid this assumption is.
The true rewards landscape. We begin by examining the landscape of agent reward with respect to the policy parameters. Indeed, even if deep policy gradient methods do not optimize for the true reward directly (e.g. if they use surrogate objective), the ultimate goal of any policy gradient al- gorithm is to navigate this landscape. First, Figure [] shows that while estimating the true reward landscape with a high number of samples yields relatively smooth reward landscape (perhaps sug- gesting viability of direct reward optimization), estimating the true reward landscape in the typical, low sample regime results in a landscape that appears jagged and poorly- behaved. The low-sample regime thus gives rise to certain kind of barrier to direct reward optimization. Indeed, applying our algorithms in this regime makes impossible to distinguish between good and bad points in the landscape, even though the true underlying landscape is fairly well-behaved.
The surrogate objective landscape. The untamed nature of the rewards landscape has led to the development of alternate approaches to reward maximization. Recall that an important element of many modern policy gradient methods is the maximization of surrogate objective function in place of the true rewards (the exact mechanism behind the surrogate objective is detailed in Appendix I..I and particularly in (14). The surrogate objective, based on relaxing the policy improvement theorem of Kakade and Langford (Kakade & &Lagttrd 22002, , can be viewed as a simplification of the reward maximization objective.
As a purported approximation of the true returns, one would expect that the surrogate objective landscape approximates the true reward landscape fairly well. That is, parameters corresponding to good surrogate objective will also correspond to good true reward.
Figure shows that in the early stages of training. the optimization landscapes of the true reward and surrogate objective surrogate objective becomes much less predictive of the true reward in the relevant sample regime. In particular, we often observe that directions that increase the surrogate objective lead a decrease of the true reward (see Figures 6 samples), we find that PPO and TRPO turn out to behave rather differently. In the case of TRPO, the update direction following the surrogate objective matches the true reward much more closely. However, for PPO we consistently observe landscapes where the step direction leads to lower true reward, even in the high-sample regime. This suggests that even when estimated accurately enough,
indeed approximately aligned. However, as training progresses, the
7). In nigher-sample regime (using several orders of magnitude more
2,000 state-action pairs
20,000 state-action pairs
100,000 state-action pairs
(-21 trajectories
(-235 trajectories)
J- 1187 trajectories)
40
-40
49:
42-
12
22
Figure 5: True reward landscape concentration for TRPO on Humanoid-v2. We visualize the land- scape at training iteration 150 while varying the number of trajectories used in reward estimation (each subplot), both in the direction of the step taken and a random direction. Moving one unit along the "step direction" axis corresponds to moving one full step in parameter space. In the random di- rection one unit corresponds to moving along a random norm Gaussian vector in the parameter space. In practice, the norm of the step is typically direction. While the landscape is very noisy in the low-sample regime, large numbers of samples reveal well-behaved underlying landscape. See Figures 20 19 the Appendix for additional plots.
order of magnitude lower than the random
6

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%) Surrogate
True reward
True reward
Step O
Step 300
Figure 6: True reward and surrogate objective landscapes for TRPO on the Humanoid-v2 MuJoCo task. We visualize the landscapes in the direction of the update step and a random direction (as in Figure The surrogate objective corresponds to the actual function optimized by the algorithm at each step. We estimate true reward with 10th state-action pairs per point. We compare the landscapes different points in training and with varying numbers of state-action pairs used in the update step. Early in training the true and surrogate landscapes align fairly well in both sample regimes, but later become misaligned in the low-sample regime. More landscapes in Appendix Figures 13 18
the surrogate objective might not be an accurate proxy for the true reward. (Recall from Section 2.1 that this is a sample regime where we are able to estimate the true gradient of the reward fairly well.)
TOWARDS STRONGER FOUNDATIONS FOR DEEP RL
Deep reinforcement learning (RL) algorithms have shown great practical promise, and are rooted in a well-grounded theoretical framework. However, our results indicate that this framework often fails to provide insight into the practical performance of these algorithms. This disconnect impedes our understanding of why these algorithms succeed (or fail), and is a major barrier to addressing key challenges facing deep RL such brittleness and poor reproducibility.
To close this gap, we need to either develop methods that adhere more closely to theory, or build theory that capture what makes existing policy gradient methods successful. In both cases, the first step is to precisely pinpoint where theory and practice diverge. To this end, we analyze and consolidate our findings from the previous section.
Gradient estimation. Our analysis in Section 2.1 shows that the quality of gradient estimates that deep policy gradient algorithms use is rather poor. Indeed, even when agents improve, such gradient estimates often poorly correlate with the true gradient (c.f. Figure 2) We also note that gradient cor- relation decreases as training progresses and task complexity increases. While this certainly does not preclude the estimates from conveying useful signal, the exact underpinnings of this phenomenon in deep RL. still elude us. In particular, Section 2. we outline few keys ways in which the deep RL setting is quite unique and difficult to understand from an optimization perspective, both theoretically and in practice Overall, understanding the impact of gradient estimate quality on deep RL algorithms is challenging and largely unexplored.
Value prediction. The findings presented in Section 2.2 identify two key issues. First, while the value network successfully solves the supervised learning task it is trained on, it does not accurately model the "true" value function. Second, employing the value network as baseline does decrease the gradient variance (compared to the trivial ("zero") baseline). However, this decrease is rather marginal compared to the variance reduction offered by the "true" value function.
It natural to wonder whether this failure in modeling the value function is inevitable. For example, how does the loss function used to train the value network impact value prediction and variance reduction? More broadly, we lack understanding of the precise role of the value network in

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%) Surrogate
True reward
True reward
Step 0
Step 300
Figure 7: True reward and surrogate objective landscapes for PPO on the Humanoid-v2 MuJoCo task. See Figure for a description. We observe that early in training the true and surrogate land- scapes align well. However, later increasing the surrogate objective leads to lower true reward.
training. Can we empirically quantify the relationship between variance reduction and performance? And does the value network play broader role than just variance reduction?
Optimization landscape. We have also seen, in Section 2.3] that the optimization landscape in- duced by modern policy gradient algorithms, the surrogate objective, is often not reflective of the underlying true reward landscape. We thus need deeper understanding of why current methods succeed despite these issues, and, more broadly, how better navigate the true reward landscape.
RELATED WORK
The idea of using gradient estimates to update neural network-based RL agents dates back least to the REINFORCE (Williams 1992) algorithm. Later, Sutton (Sutton et al. 1999) established unifying framework casting these algorithms as instances of the policy gradient class of algorithms.
Our work focuses on proximal policy optimization (PPO) (Schulman et al. 2017) and trust region policy optimization (TRPO) (Schulman et al.) (2015a). which are of the most prominent policy gradient algorithms used in deep RL, drawing inspiration from works on related algorithms, such as (Peters et al. 2010) and Kakade (200T).
Many recent works document the brittleness of deep RL algorithms (Henderson et al.) 2018 2017; Islam et al. 2017). (Rajeswaran et al., 2017) and (Mania et al., 2018) demonstrate that on many benchmark tasks, state-of-the-an performance can be attained by augmented randomized search approaches. McCandlish et al. (2018) investigates gradient noise in large- batch settings, and and Ahmed et al. (2018) investigates the role of entropy regularization (which do not study) on optimization.
CONCLUSION
In this work, analyze the degree to which key primitives of deep policy gradient algorithms follow their conceptual underpinnings. Our experiments show that these primitives often do not conform the expected behavior: gradient estimates poorly correlate with the gradient, better gradient estimates can require lower learning rates and can induce degenerate agent behavior, value networks reduce gradient estimation variance to a significantly smaller extent than the true value, and the underlying optimization landscape can be misleading.
This demonstrates that there a significant gap between the theory inspiring current algorithms and the actual mechanisms driving their performance. Overall, our findings suggest that developing deep RL toolkit that is truly robust and reliable will require moving beyond the current benchmark- driven evaluation model to more fine-grained understanding of deep RI. algorithms.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization, 2018.
Alexandre d' A'Appremoon. Smooth optimization with approximate gradient. SIAM Journal on Opti- mization, 19:1171-1183, 2008.
Miyuru Dayarathna, Yonggang Wen, and Rui Fan. Data center energy consumption modeling: A survey, IEEE Communications Surveys & Tutorials, 18(1):732-794, 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv: (xxx::/009000660. 2017.
Peter Henderson, Joshua Romoff, and Joelle Pineau. Where did my optimum go?: An empirical analysis of gradient descent optimization in policy gradient methods, 2018.
Sepp Hochreiter and Jüürenn Schmidhuber. Flat minima. Neural Computation, 9:1-42, 1997.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility bench- marked deep reinforcement learning tasks for continuous control. In ICML Reproducibility in Machine Learning Workshop, 2017.
Sham M. Kakade. A natural policy gradient. In NIPS, 2001.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, 2002.
Kenji Kawaguchi. Deep learning without poor local minima. In NIPS, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. CoRR, abs//609.04836, 2016.
Roi Livni, Shai neural networks. In NIPS, 2014.
and Ohad Shamir. On the computational efficiency of training
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides competitive approach reinforcement learning. CoRR, abs/1803.07055, 2018.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training, 2018.
OpenAl. Openai five. http cttpppennniiieee/eeeeeeee 2018.
OpenAI, :, Marein Andrychowiez, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation, 2018
Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. AAAI, 2010.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade. Towards general- ization and simplicity in continuous control. In NIPS, 2017.
Herbert Robbins and Sutton Monro. stochastic apprxxiaaion method. Ann. Math. Statist., 22(3): 400 09 1951. doi: 10. (0.1111//omm/11777//00 URL :ttpss://do.... ceroiioorg/100 10. 1214/ aoms 2oms/1177729522
Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In ICML, 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, pp. 1889-1897, 2015a.

---

Published as a conference paper at ICLR 2020
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 3oXxv:1506.02438,, 2015b.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. CoRR, abs/ Abs//1506.0233, 2015c.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Ilya Sutskever. Keynote talk. NVIDIA NTECH, 2018. URL bttps:/www.youtus// com/
wat wueechhvs -aaaysstt46777 467s
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, 1999.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning, 8:229-256, 1992.
10

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 BACKGROUND
In the reinforcement learning (RL) setting, an agent interacts with stateful environment with the goal of maximizing cumulative reward. Formally, we model the environment as (possibly random- ized) function mapping its current state and an action supplied by the agent to new state s' and resulting reward r. The choice of actions of the agent is governed by the its policy я. This policy a function mapping environment states to distribution over the actions to take. The objective of an RL algorithm is to find a policy which maximizes the expected cumulative reward, where the expectation is taken over both environment randomness and the (randomized) action choices.
Preliminaries and notation. For given policy n, we denote by (als)) the probability that this policy assigns to taking action a when the environment is in the state We use r(s, a) to denote the reward that the agent carns for playing action in response to the state A trajectory {(ae, 82) 1 E (1... . is a sequence of state-action pairs that constitutes valid transcript of interactions of the agent with the environment. (Here, at (resp. 8t) corresponds to the action taken by the agent (resp. state of the environment) in the t-th round of interaction.) We then define T(r) to be the probability that the trajectory T is executed if the agent follows policy æ (provided the initial state of the environment is 81). Similarly, r(r) rs,,, az) denotes the cumulative reward earned by the agent when following this trajectory, where st (resp. a,) denote the t-th state (resp. action) in the trajectory In the RL setting, however, often choose to maximize the discounted cumulative reward of a policy R == R1, where Rg is defined as
Re(T) = Σ ((-
and 0 < < 1 "discount factor". The discount factor ensures that the cumulative reward of a policy is well-defined even for an infinite time horizon, and it also incentivizes achieving reward earlier.
Policy gradient methods. A widely used class of RL algorithms that will be the focus of our analysis is the class of so-called policy gradient methods. The central idea behind these algorithms is to first parameterize the policy Ty using a parameter vector 0. (In the deep RL context, mo is expressed by a neural network with weights 0.) Then, we perform stochastic gradient ascent on the cumulative reward with respect to 0. In other words, we want to apply the stochastic ascent approach to our problem:
max (
(5)
where T Ny represents trajectories (rollouts) sampled from the distribution induced by the policy This approach relies on the key observation (Sutton et al. 1999) that under mild conditions, the gradient of our objective can be written as:
V E TE [r()] = Erron Vo log(ns(T)) r(r)],
and the latter quantity can be estimated directly by sampling trajectories according to the policy ma-
When we the discounted variant of the cumulative reward and note that the action of the policy at time & cannot affect its performance at carlier times, we can express our gradient estimate as:
go = ETaTa
Σ
Vo log (o(())))))))))))))))))))) QTo (5t, at)
ET
where QTo (8t, a:) represents the expected returns after taking action ot from state sy:
Quo (8t, at)
E RR (ax, sc)
(8)
11

---

Published as a conference paper at ICLR 2020
Value estimation and advantage. Unfortunately, the variance of the expectation in can be (and often is) very large, which makes getting an accurate estimate of this expectation quite challenging, To alleviate this issue, number of variance reduction techniques have been developed. One of the most popular such techniques is the use of a so-called baseline function, wherein state-dependent value is subtracted from Q.y Thus, instead of estimating (7) directly, use:
9o ETaNo
Σ
Volog (o(()s))) (Sro 0z) b(st))
(9)
where b(+) is baseline function of our choice.
natural choice of the baseline function is the value function, i.c.
V (st) Ex, [R]s::]
(10)
When the value function baseline, the resulting gradient estimation problem becomes:
9o BT
Σ
Vo (o((ss)))))))) AZy (81,
(11)
(staa))r
where
Ano (st, 0z) Quo (Sc. az) Vi (st)
(12)
referred to the advantage of performing action a, Different methods of estimating Vx have been proposed, with techniques ranging from moving averages to the use of neural network predic- tors (Schulaa et al. (2015b).
Surrogate Objective. So far, our focus has been on extracting a good estimate of the gradient with respect to the policy parameters 8. However, it turns out that directly optimizing the cumula- tive rewards can be challenging. Thus, a modification used by modern policy gradient algorithms to optimize "surrogate objective" instead. We will focus maximizing the following local approximation of the true reward Schulman et al. (2015a):
To(at)8) 8x)
max (s..)))
A ( as)
= Euo [Ax]
(13)
π (s)
or the normalized advantage variant proposed to reduce variance (Schulman et (2017)
To (8x) Ag (st, ai) (ur((e))
max E(
(14)
where
A A, -(AA))
(15)
o(A))
and π is the current policy.
Trust region methods. The surrogate objective function, although easier optimize, comes at cost: the gradient of the surrogate objective is only predictive of the policy gradient locally (at the current policy). Thus, to ensure that our update steps we derive based on the surrogate objective are predictive, they need to be confined to a "trust region" around the current policy. The resulting trust region methods (Kakade, 2001; Schulman et al. 2015a 2017) try to constrain the local variation of the parameters in policy-space by restricting the distributional distance between successive policies.
popular method in this class is trust region policy optimization (TRPO) Schulman al. (00)aa), which constrains the KL divergence between successive policies the optimization trajectory, lead- ing to the following problem:
(ae((s)))
((aols)) Ag (s on)
max E (se,
(KL(TT(-1s))) s)][ 1s) s Vs.
(16)
In practice, this objective is maximized using second-order approximation of the KL divergence and natural gradient descent, while replacing the worst-case KL constraints over all possible states with approximation of the mean KI. based the states observed in the currett trajectory.
12

---

Published as a conference paper at ICLR 2020
Proximal policy optimization. In practice, the TRPO algorithm can be computationally costly the step direction is estimated with nonlinear conjugate gradients, which requires the computation of multiple Hessian- vector products. To address this issue, Schulman et al. [Schulman et al. (2017) propose proximal policy optimization (PPO), which utilizes a different objective and does not com- pute projection. Concretely, PPO proposes replacing the KL-constraiieed objective (16) of TRPO by clipping the objective function directly as:
max (ss,aa)) min (clip (pr, 1 - s, An ar), (r ai))
(17)
where
To (10(al 82)
Pt π (ar 8t)
(18)
In addition to being simpler, PPO is intended to be faster and more sample-efficient than TRPO (Schulman al. 2017).
13

---

Published as a conference paper at ICLR 2020
A.2 EXPERIMENTAL SETUP
We use the following parameters for PPO and TRPO based on a hyperparameter grid search:
Table I: Hyperparameters for PPO and TRPO algorithms.
Humanoid-v2
Walker2d-v2
Hopper-v2
PPO
TRPO
PPO
TRPO
PPO
TRPO
Timesteps per iteration
2048
2048
2048
2048
2048
2048
Discount factor (r)
0.99
0.99
0.99
0,99
0.99
0.99
GAE discount (A)
0.95
0,95
0.95
0.95
0.95
0.95
Value network LR
0.0001
0.0003
0.0003
0.0003
0.0002
0.0002
Value net num, epochs
10
10
10
10
10
10
Policy net hidden layers
[64, 64]
[64, 64]
[64, 64]
[64, 64]
[64, 64]
[64, 64]
Value net hidden layers
[64, 64]
[64, 64]
[64, 64]
[64, 64]
[64, 64]
[64, 64]
KL constraint (d)
N/A
0.07
N/A
0.04
N/A
0.13
Fisher est. fraction
N/A
0.1
N/A
0.1
N/A
0.1
Conjugate grad. steps
N/A
10
N/A
10
N/A
10
CG damping
N/A
0.1
N/A
0.1
N/A
0.1
Backtracking steps
N/A
10
N/A
10
N/A
10
Policy LR (Adam)
0.00025
N/A
0.0004
N/A
0.00045
N/A
Policy epochs
10
N/A
10
N/A
10
N/A
PPO Clipping
0.2
N/A
0.2
N/A
0.2
N/A
Entropy coeff.
0.0
0.0
0.0
0.0
0.0
0.0
Reward clipping
[-10, 10]
[-10,
10]
[-10,
10]
Reward normalization
On
Off
On
Off
On
Off
State clipping
[-10, 10]
[-10, 10]
[-10, 10]
All error bars we plot are 95% confidence intervals, obtained via bootstrapped sampling.
14

---

Published as a conference paper at ICLR 2020
A.3
STANDARD REWARD PLOTS
3000
Mean Reward
2500
2000
1500
1000
TRPO
500
PPO
0
100
200
300
400
500
# Iteration
(a) Hopper-v2
TRPO
Mean Reward
3000
PPO
2000
1000
0
100
200
500
# Iteration
(b) Walker2d-v2
1000
TRPO
Mean Reward
800
PPO
400
100
200
300
400
500
# Iteration
(c) Humanoid-v2
Figure 8: Mean reward for the studied policy gradient algorithms on standard MuJoCo benchmark tasks. For each algorithm, perform 24 random trials using the best performing hyperparameter configuration, with of the random agents shown here.
15

---

Published as a conference paper at ICLR 2020
A.4
QUALITY OF GRADIENT ESTIMATION
ltteration:
lleration:
150
Iteration: 300
Iteration: 450
Avg. pairwise cos Sim
Avg. pairvise cox coc.cm
TRPO
TRPO
AVG. pairwise COS sim
TAPO
Av pairwise
23
241
27
101 Stata-Action Fairs
State-Action Fairs
* State-Action Pains
State-Action Pairs
Walker2d-v2
Iteration:
Itoration: 150
Iteration: 300
Iteration: 450
Avg. pairwise cOs Sm
TRAO PRO
Avg. palwise cax sim
TRPO PPO
TAPO
Ave cairwee COS sim
TRPO
0.5
2.5
36
38
23a
16"
16a
18
104
17
177
10"
23
27
36
28'
16
10'
Stato-Action Pairs
#
State-Action Pains
State-Action Pairs
Stata-Action Pairs
(b) Hopper-v2
Figure 9: Empirical variance of the gradient (c.f. ( as function of the number of state-action pairs used in estimation for policy gradient methods. We obtain multiple gradient estimates using a given number of state-action pairs from policy a particular iteration. We then measure the average pairwise cosine similarity between these repeated gradient measurements, along with the 95% confidence intervals (shaded). Each of the colored lines (for a specific algorithm) represents particular trained agent (we perform multiple trials with the same hyperparameter configurations but different random seeds). The dotted vertical black line (at 2K) indicates the sample regime used for gradient estimation in standard practical implementations of policy gradient methods.
16

---

Published as a conference paper at ICLR 2020
Avg. cos si with true grad
Iteration:
0
Kug. cos si with true grad
# Iteration: 150
Iteration: 300
Avg. cos sim with true grod
Iteration: 450
TAPO
TRiO
TRPO
TRPO
2.
30t
19
.03
10a
10P
192
321
39"
101 State-Action Pairs
32'
State-Action Pairs
8 AtatinAction Pairs
StateeAiction Pairs
(a) Walker2d-v2
Avg. cos sim with true grid
Iteration:
Avg. cos sim with true grad
Iteration:
Iteration:
Avg. cos sim with true grod
Iteration:
TRAG PPO
TRIO
TRPO
TRPO PPG
PP0
PPQ
200
20
State-Action
State-Action
State-Action Pairs
State-Action Fairs
(b) Hopper-v2
Figure 10: Convergence of gradient estimates to the "true" expected gradient (c.f. (1) We measure the cosine similarity between the true gradient (approximated using around 1M samples) and gradi- estimates, function of number of state-action pairs used to obtain the later. For particular policy and state-action pair count, we obtain multiple estimates of this cosine similarity and then report the average, along with the 95% confidence intervals (shaded). Each of the colored lines (for specific algorithm) represents particular trained agent (we perform multiple trials with the same hyperparameter configurations but different random seeds). The dotted vertical black line (at 2K) indicates the sample regime used for gradient estimation in standard practical implementations of policy gradient methods.
17

---

Published as a conference paper at ICLR 2020
A.5
VALUE PREDICTION
2-2
TRPO
PPO
Returns MRE
GAE MRE
2
2-
2-
2-11
100
200 Iterations
300
400
500
100
200
300
400
500
# Iterations
(a) Hopper-v2
2-
TRPO
GAE MRE
2-
PPO
Returss MRE
2-
2-
2-
2:11
TRPO
2-1
PPO
200 Iterations
200
# Iterations
(b) Walker2d-v2
24
2-1
Returns MRE
GAE MRE
2-
2-
2-
100
200
300
400
500
100
200 # Iterations
300
400
500
Iterations
(c) Humanoid-v2
Figure 11: Quality of value prediction in terms of mean relative error (MRE) train state-action pairs for agents trained to solve the MuJoCo tasks. We see that the agents do indeed succeed at solving the supervised learning task they trained for the train MRE the GAE-based value loss (Vold + AGAE)* (c.f. (4) is small (left column). We observe that the returns MRE quite small well (right column).
18

---

Published as a conference paper at ICLR 2020
2-2
TRPO
21
PPO
Returns MRE
GAE MRE
2-
2-1
2-2
2
TRPO
2-11
2
PPO
100
200
400
200
400
500
# Iterations
# Iterations
(a) Hopper-v2
21
TRPO
TRPO
PPO
Returns MRE
2
PPO
GAE MRE
2-1
2-2
2 2-2
2-4
2-5
100
200
300
400
500
100
200
300
400
500
# Iterations
Iterations
(b) Humanoid-v2
Figure 12: Quality of value prediction in terms of mean relative error (MRE) on heldout state-action pairs for agents trained to solve MuloCo tasks. We see that the agents do indeed succeed at solving the supervised learning task they are trained for the validation MRE the GAE-based value loss (Vota + AG (c.f. (4) small (left column). On the other hand, we see that the returns MRE is still quite high the learned value function is off by about 50% with respect the underlying true value function (right column).
19

---

Published as a conference paper at ICLR 2020
A.6 OPTIMIZATION LANDSCAPE
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%)
True reward
Surrogate
True reward
Step
Step 150
Step 300
Step 450
14
Figure 13: Humanoid-ve PPO reward landscapes.
20

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%)
True reward
Surrogate
True reward
Step
Step 150
Step daa 300
Step 450
Figure 14: Humanoid-v2 v2 TRPO reward landscapes.
21

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%)
True reward
Surrogate
True reward
Step
Step 150
Step 300
Step 450
Figure 15: Walker2d-v2 PPO reward landscapes.
22

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%)
True reward
Surrogate
True reward
Step
Step 150
Step 300
Step 450
Figure 16: Walker2d-v2 - TRPO reward landscapes.
23

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%)
True reward
Surrogate
True reward
Step
Step 150
Step 300
Step 450
Figure 17: Hopper-v2 v2 - PPO reward landscapes.
24

---

Published as a conference paper at ICLR 2020
Few state-action pairs (2,000) Surrogate
Many state-action pairs (10%)
True reward
Surrogate
True reward
Step
Step 150
Step 300
Step 450
Figure 18: Hopper-v2 v2 - TRPO reward landscapes.
25

---

Published as a conference paper at ICLR 2020
2,000 state-action pairs
20,000 state- state-action pairs
100,000 state-action pairs
(-61 trajectofis)
6-616 trajectoriss
(-3131 trajectoriss)
10
300
Step
120
36
1.1
2.2
95
(-21 trajectories)
(-235 trajectories
(=1187 trajectoriss
18.
400
Step 150
27
50
trajectories)
(-215 trajectoies)
(-1202 trajectoriss)
45.
Step 300
trajectories)
(-186 trajectories)
(-979 trajectories)
16
Step 450
543
23
Figure 19: Humanoid-v2 TRPO landscape concentration (see Figure for a description).
26

---

Published as a conference paper at ICLR 2020
2,000 state-action pairs
20,000 state-action pairs
100,000 state-action pairs
(-73 trajectofe))
(-735 trajectories
-9111 trajectories)
18.
100
140
Step
13
21
36
12
2.4
2.
95
(-19 trajectories)
(-209 trajectoris)
(=1062 trajectoris)
Step 150
50
trajectories)
(-193 trajectories
(-590 trajectories)
000
Step 300
312
trajectories)
(-182 trajectories)
(-938 trajectories
379
100
Step 450
54
335
23
Figure 20: Humanoid-v2 PPO landscape concentration (see Figure 5 for a description).
27


Reviews:
[{"review_id": "rkgkefNAYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting and important paper, it emphasizes and analyzes how policy gradient methods modify their objective functions and how this leads to training differences (and often errors w.r.t. the true objective). I have some minor comments on terminology used that I would like to see properly defined within the paper, but otherwise believe this should be accepted for its useful insights.  \n\nAssorted Comments:\n+ Maybe I simply have a difference of opinion or have misunderstood, but I am hesitant to agree that the work is comparing the surrogate *reward* function, but rather the surrogate objective. You'll notice that in the TRPO paper, it is called a surrogate objective not a surrogate reward: https://arxiv.org/pdf/1502.05477.pdf .\n+ I think better specification of what exactly is being plotted (pointing to an equation) or defining very concretely what is a surrogate reward or true reward (which I suspect is the objective) will make this paper much clearer.\n+ In fact, it was a bit unclear whether the comparisons were of the sampled/observed reward function R(s,a) (provided by the environment and sampling regime) or the objective function often the advantage A(s,a) (or the surrogate objective, GAE, etc.) I assume it should be the latter, but the wording of the paper makes this a bit unclear. I suggest discussing things in terms of objectives not rewards -- unless in fact the paper does approximate reward functions in which case this should be specified in much more detail.  \n+ Also, in a lot of places it seems like there's a mixup between rewards and returns. I think typically in the literature reward = r_t and return = V_t (sum of reward). Perhaps, in places the paper truly speaks of rewards, but from the context it seems as though it mainly refers to returns. Examples: \" Evidently (since the agent attains a high reward) these estimates are sufficient to consistently improve reward\" \" This is in spite of the fact that our agents continually improve throughout training, and attain nowhere near the maximum reward possible on each task\"\n ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1g2J6P6tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores a critical divergence between theory and practice, emphasizing that while deep policy gradient algorithms seem to work in certain cases, they don't seem to be working foor the reasons underlying their derivations. It particularly looks at how closely the sample-based approximation of the objective's gradient aligns with the true gradient of the objective, how accurately learned values match the true expected returns, and how well the optimization landscapes of surrogate objectives line up with the objective of maximizing the return.\n\nI propose accepting this paper, as it reveals a key gap in our understanding of why policy gradient methods work. Such emphasis can suggest why deep RL results tend to be inconsistent and irreplicable, and spark future work on closing the gap between theory and practice. Further, the paper is overall well written.\n\nI primarily would like clarification on the optimization landscape visualizations:\n\n1) Is the step direction the direction of the update actually performed at that time step?\n\n2) Would moving diagonally in this space correspond to a mixture of following the update direction and a normally-distributed random direction? Concretely, in the true reward plot at Step 0 for few state-action pairs in Figure 8, does this suggest that mixing a random direction with the update direction would be better than moving cmopletely in the step direction?\n\nMinor:\nTypo in citation \"...policy improvement theorem of Kakade and Langford Kakade & Langford (2002)\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Sye656FTFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n[Summary]\nThis paper empirically studies the behavior of deep policy gradient algorithms during the optimization. The conclusion is that, while these methods generally improve the policy, their behavior does not comply with the underlying theoretical framework. First, sample gradients obtained with a reasonable batch size have little correlation with each other and with the true gradient. Second, a larger batch size requires a smaller step-size. Third, the value baseline is far from true values and only marginally reduces variance, yet it considerably helps with optimization. Finally, the optimization landscape highly varies with the choice of objective function and the number of samples used to estimate it.\n\n[Decision]\nI vote for acceptance. To the best of my knowledge, the findings of this paper are new and not predictable by the current theory. These negative results have some merit as they call for theory that explains the behavior of these algorithms, or an algorithm whose behavior is predictable by the current theory. The paper is well-written, with a few small issues in presentation that should to be addressed in the final revision.\n\n[Comments]\nIn Fig. 4 (b) it does not look like that the value error is high. It is said that \"the learned value function is off by about 50% w.r.t. the underlying true value function.\" This sentence should be clarified or visualized.\n\nWhat is \\pi in Eq (13) in A1? If it is the agent's current policy, how is it different than \\pi_\\theta? If \\pi corresponds to the distribution of state-action pairs in the replay buffer, how can one obtain a policy \\pi that has led to this distribution of states in order to construct the importance sampling ratio?\n\nIn 2.2, the claim that a learned value baseline results in significant improvement in performance should be supported by results or reference to previous work.\n\nFigs. 6 and 7 compare the loss surface with different objectives and sample regimes. Do these factors (objective and sample size) affect the part of the parameter space that is visualized (by changing the origin and the update direction), or are they only used to evaluate the values on the z-axis for the same area in the parameter space? Observing a different landscape in a different part of the parameter space is not surprising.\n\n[Minor comments]\n- Is V_\\theta_{t-1} in Eq (4) a function of state? If so, a (s_t) is missing before the plus sign.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper empirically studies the behaviour of deep policy gradient algorithms, and reveals several unexpected observations that are not explained by the current theory. All three reviewers are excited about this work and recommend acceptance.
Decision: Accept (Talk)