Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Authors: Tao Wang
Ruipeng Zhang
Sicun Gao
Venue: ICML 2025 poster (2025)
Keywords: policy gradient methods
value estimation
trust region methods
deep RL
Primary area: reinforcement_learning->deep_rl
TL;DR: We demonstrate through both theoretical analysis and experiments that vanilla policy gradient can achieve performance comparable to PPO by simply increasing the number of value steps per iteration.

Abstract: Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use.

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Tao Wang' Ruipeng Zhang' Sicun Gao
Abstract
provement is theoretically guaranteed for sufficiently small policy updates (Kakade & Langford, 2002; Schulman et al., 2015a), empirical results suggest otherwise: smaller learn- ing rates for the policy network do not always yield better performance (Andrychowicz et al., 2021). According to the trust region theory (Schulman et al., 2015a), there is a gap between theory and practice because the theory guarantees improvement of the original policy objective only when the surrogate objective improves by margin greater than the distance between the old and new policies. However, there is no guarantee that this condition is consistently satisfied in practical policy training.
Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL. tasks. Questioning the common belief that enforcing approximate trust regions leads steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to bet- ter than PPO in all the standard continuous con- trol benchmark environments. Importantly, this simple change to vanilla policy gradient is sig- nificantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use.
Moreover, has been reported that deep policy gradient methods often suffer from implementation issues such as brittleness, poor reproducibility, and sensitivity hyperpa- rameter choices (Henderson et al., 2018; Engstrom et al., 2020; Andrychowicz et al., 2021). Extensive empirical analysis has shown that the behavior of deep policy gra- dient methods does not align with the predictions of their motivating framework (Ilys et al., 2020). Recent studies further highlight that the optimization landscape of many RL tasks is highly non-smooth and even fractal, raising questions about the well-posedness of all policy gradient methods (Wang et al., 2023). Consequently, there is still no clear answer to the question: while almost all theoretical as- sumptions are violated in practice, why does PPO perform better than vanilla policy gradient empirically?
1. Introduction
Deep policy gradient methods have demonstrated their strength in various domains, including robot learning (Song al., 2023), game playing (Vinyals et al., 2019), and large language model training (Ouyang et al., 2022). Policy gra- dient methods in its modern versions, such as Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a), Prox- imal Policy Optimization (PPO) (Schulman et al., 2017), have been the popular choice for delivering state-of-the- art performance across the broad range of RL tasks. In comparison, the original policy gradient algorithm, Vanilla Policy Gradient (VPG) (Sutton et al., 1999), typically per- forms significantly worse than the modern algorithms. The common explanation for why TRPO and PPO outperform VPG is their ability to prevent excessive large policy up- dates. This is achieved through surrogate objective that incorporates importance sampling, constrained by a limit on policy step size each iteration. While monotonic im-
In this paper, we present a different perspective on deep pol- icy gradient methods within the actor-critic framework: ac- curate value estimation is more critical than enforcing trust regions. We first demonstrate that, in practice, trust region methods do not behave as their theoretical analysis suggests. Then, we analyze how TRPO and PPO implicitly enhance value estimation through their implementation designs. Our theoretical analysis reveals that the true value often grows significantly faster than the critic's predictions, leading to poor value estimation during policy training. Finally, empir- ical results show that simply increasing the number of value updates enables the basic VPG algorithm to match PPO's s performance across multiple continuous control benchmarks in Gymnasium. This highlights the pivotal role of value es- timation improving policy gradient methods.
'University of California, San Diego, La Jolla, USA. Corre- spondence to: Tao Wang ctaw003@ucsd.ede
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
This paper is organized as follows. In Section 4, we analyze

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
the relationship between trust regions and policy improve- ment, demonstrating that there is no clear correlation. This suggests that enhancing trust region may not be the fun- damental reason for the success of trust region methods. In Section 5, we explain that both TRPO and PPO perform more value steps per iteration compared to VPG, funda- mentally contributing to closer value approximation and improved performance. Additionally, we provide a theo- retical framework suggesting that value networks typically require more gradient steps to optimize than policy networks. In Section 6, extensive experiments are conducted and pre- sented to corroborate our theoretical analysis. Specifically, we show that by increasing the number of value steps alone, VPG achieves performance similar to or better than PPO across several Gymnasium benchmarks.
baselines in benchmark environments (Tucker et al., 2018).
Implementation matters in deep policy gradients. De- spite its accomplishments, deep RL methods are notorious for their brittleness, lack of stability and reliability (Hender- son et al., 2018; Engstrom et al., 2020). Furthermore, it has been reported that the performance of PPO heavily relies on code-level optimization techniques, due to its sensitivity to hyperparameter choices (Henderson et al., 2018; Engstrom et al., 2020) and regularization techniques (Liu et al., 2021). Recent works aimed at systematically understanding and addressing hyperparameter tuning issues in deep RL have been developed (Paul et al., 2019; Eimer et al., 2023; Ad- kins et al., 2024). We contribute to this line of research by directly identifying value estimation as the core component that makes deep policy gradients work.
2. Related Work
3. Preliminaries
Performance gap between VPG and PPO. Much work has been done to understand the performance gap between VPG and PPO. It has been shown that the VPG loss performs significantly worse than the PPO loss, even with optimally conditioned hyperparameters (Andrychowicz et al., 2021; Raffin al., 2021). On the theoretical side, the global opti- mality of TRPO/PPO is proven for overparameterized neural networks under the assumption of a finite action space (Liu et al., 2019). It is also shown that ratio clipping may not be necessary in PPO (Sun et al., 2023). Another theory suggests that policy gradients work by smoothing the value landscape (Wang et al., 2024), indicating that the choice of policy objective may not be critical factor affecting performance, as all of them provide the same smoothing effect through Gaussian kernels. In this work, we take step forward to demonstrate that the fundamental gap between VPG and PPO lies in value estimation, and by optimizing it, VPG can achieve performance comparable to PPO.
Consider an infinite-horizon Markov decision process (MDP), defined by the tuple (S, A, 2 Po. where the state space S and the action space AA are continuous and compact, 4 : S x S x A + [0, oc) is the transition proba- bility density function, R S + R is the reward function, Po is the distribution of the initial state 8o, and y E (0, 1) is the discount factor. For parameterized stochastic policy Tg. the policy objective to maximize is given by
J(0) 1B(o (x)
r R(s ). (1)
where 0 E R is the policy parameter. According to (Sutton et al., 1999), the original form of policy gradient estimator is given as:
VoJ( (00(6) /
Q- (s, (o dads, (2)
Value estimation in off-policy methods. Although it has been demonstrated that an optimal baseline in policy gradi- ents can effectively reduce the variance of the policy gra- dient estimator (Peters & Schaal, 2008), it remains unclear to what extent value estimation affects the performance of on-policy algorithms. In contrast, the importance of value estimation has been extensively studied in off- off-policy algo- rithms. In particular, it has been found that the max operator may lead to overestimated Q values, and double Q learning methods were introduced to mitigate this error in estima- tion (van Hasselt, 2010; van Hasselt et al., 2016). This method also serves one of the keystones in advanced off-policy algorithms such as TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), compared to the DDPG algorithm (Lillicrap et al., 2015). Additionally, it has been numerically shown that using ieeee-action--de deeiendett base- lines does not reduce variance compared to dependent
where Q is the Q-function of the current policy and pt the discounted visitation frequencies. While there are various methods for approximating Q values, the most commonly used approach is Monte-Carlo, which esti- mates the discounted return for each trajectory, i.e., G. = ΣT RrtR The vanilla policy gradient loss is given as
I.(()= = ( )ome log To (aglss) Gr],
which is the REINFORCE algorithm (Williams, 1992).
REINFORCE with baseline. In practice, the variance of the estimation Q, can be large, making it difficult to distinguish between higher-valued actions and less highly valued ones. This motivates the use of baseline functions to reduce the variance (Sutton Barto, 1998). In this case,
2

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
the objective is given by:
where 3 max | (8, a) is the maximum advantage
value. In practice, the TRPO algorithm solves the following approximate constrained optimization problem (Achiam, 2018):
L (0)= Bis log (arls)) A.].
(4)
where A, Q: (s+) is the estimated advantage of (st, az), and V is the estimated value function of the current policy ı which serves as the baseline. Let Vs(s) ((s; Φ) denote the value approximation to the true value function * by a neural network where Φ E R is the network parameter.
maximize V TRLPRPO
(0)
18=0add (0-0o
0
subject to 2 (0 Ood)) HiO Pota) 5
where H is the Hessian of E which provides a quadratic approximation to the KL con- straint. Solving with conjugate gradient methods leading to the update rule 0k+1 8% + a V
In some literature, this algorithm is also referred to as Ad- vantage Actor-critic (A2C) (Mnih et al., 2016)). To avoid any potential ambiguity, we henceforth refer to the objective in equation (4) whenever mentioning the vanilla policy gra- dient (VPG) algorithm. In the following sections, we will demonstrate that optimizing the objective in equation (4) sufficient to achieve performance comparable to PPO, provided that the baseline function V is properly optimized.
25 H 9, where Hh
g VoLTRROO (0) s= denotes the gradient at 0 0%- a E (0, 1) is the backtracking coefficient, and j is the small- est nonnegative integer that makes the update satisfy the KL constraint (6) while producing positive surrogate advan- tage.
Trust region methods. According the conservative pol- icy iteration method (Kakade & Langford, 2002), small policy update that improves the corresponding surrogate objective is guaranteed to improve the true policy objective. This result motivated the TRPO algorithm (Schulman et al., 2015a), which optimizes the following surrogate objective:
However, there are two issues with the design in prac- tice. First, although it proven that VoLTRPO -- (O)le=,, under conditions that assume the existence of gradients (Schulman et al., 2015a), the gradient Vo.((O) may not exist for many continuous control problems in the first place. This is because the policy optimization land- scapes in these cases often exhibit fractal structures, as il- lustrated in Figure 1 (Wang et al., 2023). Consequently, the linear approximation used the TRPO implementation may poorly represent the original objective, making it difficult to attribute the success of TRPO to this approximation.
To ( Ae
T.RPPO (0) = ac)
(5)
(+)
which is further subject to constraint on the KL divergence between the current and the old policy:
È B D ( ((11s))] 6.
(6)
for some > In practice, however, TRPO can be expen- sive, it involves the computation of second-order deriva- tives. This motivates the PPO algorithm (Schulman et al., 2017), which simplifies the objective and avoids the need for second-order derivatives:
(P) = E min (rrf,,cipfr.11_11111)
(a) H3pper-v3
(b) Walker2d-v3
where ft = He the probability ratio from importance sampling, and e the clipping parameter.
Figure I. Policy objectives in many continuous-contro environ- ments are highly non-smooth and fractal.
4. Revisiting Trust Regions
Second, since the approximate objective is linear in 0 and the quadratic constraint induces a convex feasible region, the solution to this problem should always lie on the bound- ary of the feasible region. As result, the practical al- gorithm attempts take the largest possible step in each iteration, whereas the theory suggests taking the smallest step to guarantee policy improvement. This discrepancy suggests that the effectiveness of trust region methods may not stem solely from their enforcement of theoretical con- straints, but rather from hyperparameter tuiing and practical
According to the theoretical analysis in TRPO (Schulman et al.. 2015a), the true performance J-)) is guaranteed to improve when a policy update is made to improve the TRPO surrogate T.TRPO(-), (-), provided that the KI. divergence be- tween two consecutive policies small enough. This condi- tion is expressed through the following inequality:
43y 7)²
((0) 2 TBPO
RK (No Tema)
(8)
3

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
proving algorithmic performance. Nevertheless, trust region methods generally outperform VPG, it remains to identify the fundamental factors that drive policy improvement in practice. Given that the number of optimization steps plays a crucial role in the outcome, we focus on analyzing this as- pect throughout this section and argue that value estimation is the core factor driving policy improvement.
12
a2oda JPC wn spach
2-
VTC
(a) maximum ratio max Ft
(b) minimum ratio min it
5.1. Increased value steps in TRPO
The VPG algorithm employs a single optimization loop to simultaneously optimize the policy and value networks using automatic differentiation software. In this loop, the value network is trained to minimize the regression loss:
2.-
2010
spach
1032
190 un spoct
ww MMnmmpl
13
2010
12
1300
1010
L (6) = VV Viargett 1
3
where D represents the collected data, Vo is the param- eterized value approximation, and Varget is the target value, typically obtained through Temporal Difference (TD) estimates. For instance, the target value is given by Vtarger(s;) = Σ'= 7 k A-t *R VV((T+1) when the GAE factor 1.
(c) clipping fraction
(d) cumulative reward
Figure We compare the performance of different implementa- tions on the MuJoCo Hopper task. The clipping parameter is set to
e = 0.2 as default.
code-level optimizations that extend beyond the theoretical framework (Henderson et al., 2018).
However, the TRPO implementation requires separate opti- mization loops for the policy and value networks: the policy is optimized using the conjugate gradient algorithm to en- force the trust region constraint, while the value network is updated similarly to VPG. Perhaps surprisingly, this seem- ingly small modification in the code-level implementation may account for the fundamental difference between the two algorithms. Specifically, the VPG algorithm performs only one optimization step per epoch, which is insufficient for the value network to accurately estimate the true returns. In contrast, TRPO applies multiple optimization steps per iteration for the value network without over-optimizing the policy network. This results in more accurate value estima- tion and, consequently, better performance. Some examples are presented in Table 1.
To simplify implementation, PPO instead defines trust regions using the importance sam- pling ratio rather than KI. divergence. However, despite the potential correlation, there is no strong correspondence between enforcing a trust region and policy improvement. Notably, during PPO training, the maximum probability ra- tio max, Ft can significantly violate the trust region defined by the clipping coefficient (Engstrom et al., 2020). This occurs because the gradient Vera vanishes outside the trust region (e.g., when rt > 1 c) due to the clipping opera- tor. Consequently, if an excessively large step is taken on (a (sx) such that ry leaves the trust region, it may not be pulled back to the trust region anymore due to the vanishing gradient.
Ratio trust regions in PPO.
RL libraries
# Value steps
Learning rate
OpenAl Baselines
5
0.001
Our empirical results support this observation. Figure 2 (a)-(c) show that the standard PPO implementation with 10 epochs per iteration, despite employing the ratio clipping mechanism, consistently violates the ratio bound, coinciding with findings from prior work. In contrast, PPO with only epoch per iteration and VPG both successfully enforce the trust region defined by the ratio bound. However, they outperformed by the standard PPO implementation shown in Figure 2 (d).
Spinning Up Stable-Baseline3
80
0.001
10
0.001
Tianshou
20
0.001
Table 1. The number of value steps per iteration and the learning rate for value networks in default settings of several deep RL libraries (Dhariwal et al., 2017; Achiam, 2018; Raffin al., 2021; Weng et al., 2022).
5.2. How PPO addresses value estimation?
5. Importance of Value Estimation
Having a properly optimized value network is also the key reason why PPO outperforms VPG. To understand this, we analyze two algorithmic techniques in the PPO algorithm
In the previous section, we demonstrated that there is no direct relationship between enforeing a trust region and im-
4

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
and examine how they contribute improved value estima- tion.
changes rapidly in many continuous-control environments, which necessitates more update steps for the value network in each iteration. In dynamical systems theory, maximal Lyapunov exponents are used to study chaotic behaviors: given a dynamical system 8t+1 = F(*)), 80 E RR", and a small perturbation A A made to sp. the divergence caused at time is denoted by A (I). For chaotic systems, their dynamics are sensitive to initial conditions so that it has
Mini-batching and multiple epochs. In practice, both of them contribute to increasing the number of value steps in each iteration, as we have
full-batch size mini-batch size
# gradient steps = epochs x
| ~ eA1 AZol |I
Therefore, if the entire batch is used with the same number of epochs, the value network will be ander-approximated, leading to inaccurate baseline estimation. As shown in Figure 3 (a), PPO completely fails to find good policy when the entire batch is used for policy training. This failure can be attributed to the poor value estimation displayed in Figure 3 (b), where the value estimation error is calculated through
for some À >> 0. The rigorous definition of Maximal Lya- punov exponents is presented below:
Definition 5.1. (Maximal Lyapunov exponent (Lorenz, 1995)) For the dynamical system 8t41 = F(st), 80 E IR", the maximal Lyapunov exponent Amax at so is defined as the largest value such that
T
1
|
(50)
Σ k=0
k- R at) - V(su: の)
(10)
Amax = lup lim sup log
1-000 [
||-0
t
|
(11)
Specifically, according to Proposition C.2, we have the fol- lowing estimation for the policy objective update:
where at ~ and s0 ~ Po. This also helps explain why PPO with a single epoch performs poorly as seen in Figure 2.
| - ((0) ~ O0100' 01%)
(12)
Cumulative Reward
Value Estimation Error
.2000
where -log and A(0) is the maximal Lyapunov ex- X(0)
4000
Quloon
135
ponent of the dynamies controlled by ne, which is typ- environments. ically positive in many continuous
3000
335
2508
Thus, we may have a < 1 when X(0) > log Y, which is generally the case when the underlying dynamics is chaotic (Lorenz, 1995). This results in fractal landscapes in the policy space as illustrated in Figure 1, and a rapidly changing policy objective.
3300
1000
25-
2.6
Tirnstop
Tinatez
(a)
(b)
Figure 3. The cumulative reward and value estimation error during PPO training in the Hopper task are compared hetween full-batch and mini mitc-batch updates. highlights how the use of full-bateh updates leads to suboptimal policy performance, as reflected in the large value estimation errors, while mini-batch updates facilitate more accurate value estimation and better cumulative reward out- comes.
The above estimation also suggests that even a small update in the parameters can lead to significant change in the value function of the policy objective (()). On the other hand, the training of value networks is generally more stable. For most activation functions used in neural networks, such as tanh and ReLU, the network output is Lipschitz continuous with respect both the input variables and the network parameters. Specifically, for given state and parameters and d', the difference in the value network output can be estimated by
Probability ratio elipping. Like VPG, PPO uses a single optimization loop for both networks as well. Therefore, while increasing the number of optimization steps improves value estimation, it may also lead to over-updating of the pol- icy network. The clipping mechanism in PPO mitigates this by blocking excessive updates made to the policy network, allowing the value network to catch up with true returns.
[V - V(s; 6) O(1o'' +(1)
(13)
when 1| Φ1 is sufficiently small. This suggests that the landscape of the value regression loss in equation (9) is smooth (e.g., illustrated in Figure 12), such that small updates in the parameters always lead to small changes in the estimated values. We summarize this analysis with the following theorem, which provides an estimate of the relationship between the updating rates of the value and policy networks:
5.3. Theoretical analysis
We now show that this value estimation issue can be ex- plained from the perspective of optimization landscapes in the policy space. Briefly speaking, the policy objective
5

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Theorem 5.2. Assume that the dynamics, reward function, policy and value networks are all Lipschitz confinuous with respect their input variables. Let BL, 32 denote the learn- ing rate for policy and value network, respectively, and Kv denote the number of value steps per epoch. Then for each policy step, a good value estimation requires
show that by simply increasing the number of value steps, VPG can eventually achieve performance comparable to PPO in the Halfcheetah, Hopper, and Walker environments, and outperform PPO in the manoid environments. The performance of VPG stabilizes when the number of value steps reaches 50, at which point the value network is able to accurately capture the true re- turns, leading to more precise advantage estimations. Full comparisons of algorithmic performance and correspond- ing value estimation errors for different implementations are presented in Figure 4 and Figure 5. Additionally, Fig- ure 10 shows that VPG is more efficient than PPO in terms of policy steps. Also, VPG consistently improves the policy across all environments, while PPO's performance drops af- ter 2M environment steps in the Walker and Ant tasks. Each task is trained with five random seeds, with solid curves rep- resenting the average return and shaded regions indicating the one standard deviation confidence interval.
Ant and Hu-
KT K282
Ky
steps made to the value network when $1, B2 are small. a= 2 < 1 and X0) is the maximal Lyapunov exponent x10)
of the dynamics controlled by To. E (0. 1) the discount factor and K1, Kg > 0 are constant independent of the learning rates B1 and $2.
This result suggests that the value network should be op- timized more aggressively than the policy network, with more optimization steps and/or higher learning rates, to en- sure that the value estimates made by the critic network can closely track the true values observed in rollouts. Fur- thermore, to prevent interference between the policy and value networks, separate networks should be used, as also suggested in (Cobbe et al., 2021; Huang et al., 2022a). Ad- ditional details can be found in Appendix C.
GAE factor X. The effect of the Generalized Advantage Estimation (GAE) factor is to reduce the variance of value estimation in on on-policy algorithms (Schulman et al., 2015b), To assess how the GAE factor influences the performance of VPG and PPO, we consider two cases: À = 0.95 (default GAE factor) and À = (equivalent to Monte-Carlo). As shown in Figure 6, both the reward curves of VPG and PPO drop when À = 1 compared to À 0.95, which can be attributed to the variance reduction effect of the GAE method. Notably, while the performance of VPG with 100 value steps is only slightly affected, the change in the GAE factor has a significant impact on PPO, as shown in Table 2. For example, in Walker and HalfCheetah, PPO's cumulative reward drops by more than 40% when =
6. Experiments
In the previous section, we theoretically demonstrated that value estimation is the core component of on-policy algo- rithms, and by optimizing it, simple method is expected to perform as well as more advanced methods. In this sec- tion, we evaluate the role of value estimation across range of continuous-control tasks from the OpenAI Gym bench- marks (Brockman et al., 2016). Specifically, we first show that the VPG algorithm, with properly optimized value net- work, can achieve performance comparable to or even better than PPO with their corresponding default settings. Addi- tionally, we conduct ablation studies on several code-level optimizations that may impact the performance of value estimation in both VPG and PPO. The experimental setup is detailed in Appendix A, and further experimental results can be found in Appendix B.
Learning rate of policy network. In general, the sensi- tivity to learning rate in deep RL is at a similar level to that in supervised learning. Most commonly learning rates from 0.0003 to 0.001 work well for VPG. One caveat is that excessively small learning rates do not always guarantee pol- icy improvement due to the fractal landscapes in value and policy space. On the other hand, using learning rates that are too large leads to poor performance, too. The reason is that unlike in supervised learning where the loss is optimized for multiple steps which corrects the potentially wrong di- rection generated in the first step, VPG only performs one policy update per collected batch of data and thus has no chance to subsequently correct the error. In Figure 6, we can observe that PPO suffers from the sensitivity to learning rates in the Ant environment, while the performance of VPG remains relatively stable across different learning rates.
We adapt the implementation and PPO baseline from Tian- shou (Weng et al., 2022). We also provide a single-file codebase modified from CleanRL (Huang et al., 2022b) which enables implementations in other environments such as DeepMind Control (Tunyasuvunakool et al., 2020) and Isaac Gym (Makoviychuk et al., 2021). Code for the em- pirical results is available at https: /t////gttttuutt com/ =aowang0/ / /avv/valleeeeeeeeee intettssttaattie ion-vpg.
Vanilla policy gradient with multiple value steps. The performance of the VPG algorithm with different numbers of value steps is compared to PPO in Figure 4. The results
Reward normalization. While increasing the number of value steps is the most straightforward way to improve value estimation, reward normalization can impact the effective-
6

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Hopper
Walker
Halfcheetah
3140
3000
4000
2500
0000
Rewioro
3000
1000
2000
1500
2000
3000
0.5
03
64
2.0
Gleba
Steps
Ant
Humanoid
7032
5020
0003
5330
PPa
VPG rapeat
Roward
0330
VPG-repeat
repeat
2003
WPG-repaat-100
2030
VPG-rupeat-500
1032
Globa Steps
0.0
02
0.6
Glbbal
0.5
Figure 4. Training curves on Gymnasium benchmarks. The curve VPG-repeat-k corresponds to the vanilla policy gradient algorithm with k value steps applicd each iteration. For example, VPG-repeat-I represents the original vanilla policy gradient implementation. As the number of value steps increases, the performance of vanilla policy gradient consistently improves, eventually converging to or outperforming PPO when the number of value steps reaches 50 more.
Hopper
Walker
Humanoid
2012
330
233
VPC-reseat
50
VPC-reseat5
WPS-reseat-2 10
WPO reseat 50
WPS-rspeat 100
VPS-repeat: 500
100
-150
Stops
Stops
Figure 5. The corresponding value estimation difference in the experiments shown in Figure 10. We disabled exploration during evaluation, using the deterministic policy as direct output of the policy network. The difference value estimation computed through Equation 10. We observe clear correlation between the value estimation difference in VPG and its performance. As the value steps increase, the estimation crror decreases and eventually oscillates around zero, Ieading to improved performance. More results can be found in Figure 11.
ness of value estimation as well. The PPO algorithm em- ploys reward normalization scheme that rescales the wards in the current batch by dividing them by the standard deviation of rolling discounted sum, without altering the mean. As shown in Figure 9, reward scaling improves the performance VPG when the number of value steps I or 10, but has no significant effect when the number of value steps is 100. This suggests that reward normalization helps reduce the magnitude of rewards when it is large, thereby decreasing the error in value estimation. However, its effect is negligible when the value network is already properly optimized, as in the case of VPG with 100 value steps per iteration and PPO.
Mini-batching and multiple epochs. In Figure 2, we observed that using full batches in PPO lead to poor value estimation due an insufficient number of value steps. Figure 9 (b) illustrates how varying mini-batch sizes sig- nificantly affects PPO's performance the Hopper task. Additionally, the learning curves in Figure 10 show that the default hyperparameters are not optimal for the Ant and Humanoid tasks, where the dynamics are more complex, making the policy gradient estimator effective only for small policy updates. As shown in Figure 9, PPO performs sig- nificantly better with larger mini-batches and fewer policy epochs on these two tasks, both of which reduce the number of policy updates per iteration.

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Hopper
Walker
Halfchestah
RPO gav 55
mod-geed
2004
usG repost gag
vPo repoont L00 gue
Stops
Stape
Figure 6. The influence of GAE factor X across three tasks.
Algorithm
GAE A
Hopper
Walker
HalfCheetah
VPG
0.95
2601.57 + 232.14
3457.79 + 646.70
4928.88 1 807.04
VPG
1
2323.28 = 436.24
3123.83 = 987.58
4381.30 + 222.15
PPO
0.95
1965.29 = 478.14
2527.74 507.40
4488.60 00 + 2699.54
PPO
1611.46 541.32
1431.50 = 612.17
2604.08 + 1237.80
Table 2. Performance of VPG and PPO with different GAE factors after 5M environment steps for Hopper and Walker, 10M environment steps for HalfCheetah.
Hopper
Hopper
Hopper
FRD
2100
2100
2300
190 Scetoci
2000
2500
1000
VPG, H3pper-v3
(b) PPO, Hopper-v3
Figure 7. The influence of learning rates in VPG and PPO.
Gibbal Stops
Figure 8. The influence of reward normalization in different algo- rithms. We see that although normalizing rewards can improve the performance of VPG when has and 10 value steps, it does not make significant difference to VPG when there are as many as 100 value steps per iteration.
7. Conclusion
In this work, we provide both theoretical and empirical analyses to demonstrate that the performance of policy gradient algorithm is primarily determined by how accu- rately the value function is estimated, rather than by any specific policy objective design adopted for policy training. Our findings suggest that deep on-policy algorithms may work for very simple reason that even the simplest vanilla policy gradient method can accomplish.
beyond this specific context.
There are two key messages we want convey through this work: First, value estimation is perhaps the most cru- cial component of on-policy algorithms. By optimizing it, we can significantly improve their performance. This finding encourages us to revisit the foundations of the de- velopment of policy gradient methods (namely, from VPG to TRPO/PPO), as the core ideas these methods (e.g.. en- forcing a trust region, performing multiple policy updates) may not be the fundamental underpinnings of the improved performance observed in practice. Second, our results indi- cate that it is possible to develop on-policy algorithms that are robust without the extensive need for hyperparameter tuning and code-level optimization techniques. For instance, VPG with multiple value steps performs well and requires very few hyperparameters to tune, suggesting that complex
In the meantime, we should clarify that the ratio clipping mechanism in PPO, although it does not fundamentally con- tribute to performance improvement, still has its strengths in that allows multiple policy updates on a single batch, thereby enhancing sample efficiency, sometimes at the cost of reduced robustness. This point evident in the Ant and Humanoid tasks, where the default PPO implementation updates the policy too aggressively and leads to suboptimal performances. Although our argument may, in principle, be generalizable to other domains such as large language mod- els (LLMs), the present analysis is confined to continuous control problems, and we refrain from asserting its validity
8

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Hopper
Hopper
Ant
Hamanols
212
(a) Reward norm., Hopper
(b) Mini-batch, Hopper
(c) Mini-batch, Ant
(d) Mini-batch, Humanoid
Figure 9. Code-level optimizations in policy gradient methods. (a) Although normalization rewards can improve performance
VPG when # value steps = 1, its strength is rather limited compared to the fully-optimized VPG with # value steps 100. (b) The performance of PPO varies with different mini-batch sizes where the number epochs is fixed at 10. (c) (d) We use mini-hatch size 512, # #pocch = 2 for PPO in Ant and Humanoid which significantly outperform the default implementation Figurc 4.
algorithmic architectures like PPO may not be necessary for effective policy gradient methods. This is important for deep RL in real-world applications, especially for practition- ers who want to leverage policy gradient methods in their domains but have little or no expertise in RL. With well- developed parallel computing architectures that accelerate simulation and data collection, vanilla policy gradient has the potential serve as a robust and effective alternative to PPO in various robot learning tasks.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAl Gym. arXiv preprint arXiv:1606 (tti:::6000.111000 2016.
Cobbe, K., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. In ICML, 2021.
Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., and Zhokhov, P. OpenAl baselines. Gitllub repository, 2017.
Eimer, T., Lindauer, M., and Raileanu, R. Hyperparameters in reinforcement learning and how to tune them. In ICML, 2023.
Acknowledgements
The authors would like to thank Yuexin Bian for her valuable suggestions in developing the codebase, and the anonymous reviewers for their helpful comments revising the paper. This material is based on work supported by NSF Career CCF 2047034, NSF CCF DASS 2217723, and NSF AI Institute CCF 21 12665.
Engstrom, L., Ilaas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, Implementation matters in deep RL: A case study on PPO and TRPO, In ICLR, 2020.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In ICML, 2018.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900. 2018.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement lcarning that matters. In AAAI, 2018.
References
Achiam, J. Spinning Up in Deep Reinforcement Learning. 2018.
Huang, S., Dossa, R., Raffin, A., Kanervisto, A., and Wang, W. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022a. URL https: / 25/ 15/ppo-impp apppo=-mmmleennaatiio--ddttelt
Adkins, J., Bowling, M. and White, A. A method for evaluating hyperparameter sensitivity in reinforcement learning. In NeurlPS, 2024.
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., and Bachem, O. What matters for on- on-polic deep actor-critic methods? A large- scale study. In ICLR, 2021.
Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mchta, K., and Araújo, J. G. Cleanrl: High-quality single- file implementations of deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 23(274): 2022b.
9

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Hyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, E,, Rudolph, L.,, and Madry, A. A closer look deep policy gradients. In ICLR, 2020.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. 2015b.
Kakade, S. and Langford, Approximately optimal approx- imate reinforcement learning. In ICML, 2002.
Schulman, J.. Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
Lillicrap, T. P., Hunt, J. Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 02.77, 2015.
arXiv preprint arXiv: 2017.
Song, Y., Romero, A., Müller, M., Koltun, V., and Scara- muzza, D. Reaching the limit in autonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8(82)cadg1162, 2023.
Liu, B., Cai, Q.. Yang, Z. and Wang. Z. Neural proxi- mal/trust region policy optimization attains globally opti- mal policy. In NeurIPS, 2019.
Sun, M., Kurin, V., Liu, G., Devlin, S., Qin, T., Hofmann, K., and Whiteson, S. You may not need ratio clipping in PPO. arXiv preprint arXiv:2202. 00079, 2023.
Liu, Z., Li, X., Kang, B., and Darrell, T. Regularization matters in policy optimization an empirical study on continuous control. In ICLR, 2021.
Sutton, R. S. and Barto, A. Reinforcement Learning: an Introduction. MIT Press, 1998.
Lorenz, E. N. The Essence of Chaos. University of Wash- ington Press, 1995.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y.
Policy gradient methods for reinforcement learning with function approximation. In NIPS, 1999.
Makoviychuk, V., Wawrzyniak, I.,, Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., and State, G. Isaac Gym: High performance gpu-based physics simulation for robot learning. arXiv preprint /60iv: (60.0/00060/5000 2021.
Tucker, G., Bhupatiraju, S., Gu, S., Turner, R. E., Ghahra- mani, Z., and Levine, S. The mirage of action-dependent baselines in reinforcement learning. In JCML, 2018.
Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T.,, Lillicrap, T., Heess, N., and Tassa, Y. DM control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In ICML, 2016.
van Hasselt, H. Double Q-Jearning. In NIPS, 2010.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In NewrIPS, 2022.
van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double Q-learning. In 2016, 2016.
Vinyals, O., Babuschkin, I., Czarnecki, W., and et al. Grand- master level in StarCraft II using multi-agent reinforce- ment learning. Nature, 575:350-354, 2019.
Wang, T., Herbert, S., and Gao, S. Fractal landscapes in policy optimization. In NeurlPS, 2023.
Paul, S., Kurin, V., and Whiteson, S. Fast efficient hyperpa- rameter tuning for policy gradient methods. In NeurIPS, 2019.
Wang, T., Herbert, S., and Gao, S. Mollification effects of policy gradient methods. ICML, 2024.
Peters. J. and Schaal, S. Reinforcement learning of motor skills with policy gradients. Neural Networks, 24)) 682-697, 2008.
Weng, J., Chen, H., Yan, D., You, K., Dubureq, A., Zhang, M., Su, Y., Su, H., and Zhu, J. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1-6, 2022.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable- Babbll--sssinns3: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021.
Williams, R. J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization, In ICML., 2015a.
10

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
A. Experiment Hyperparameters
PPO
VPG
Num. env.
16
16
Discount factor (7)
0.99
0.99
Num. epochs
10
Batch size
2048
2048
Minibatch size
64
2048
GAE factor (A)
0.95
0.95
Optimizer
Adam
Adam
Clipping parameter (e)
0.2
N/A
Advantage normalization
False
False
Observation normalization
True
True
Reward normalization
False
False
Learning rate decay
False
False
Entropy coefficient
0
0
Policy network
[64, 64]
[64, 64]
Value network
[64, 64]
[64, 64)
Activation function
tanh
tanh
Gradient clipping (12 norm)
1.0
1.0
Table 3. Default hyperparameters for policy gradient algorithms.
Hopper
Walker
HalfCheetah
Ant
Humanoid
VPG
0.0003
0.0007
0.0007
0.0007
0.0007
PPO
0.0003
0.0003
0.0003
0.0003
0.0003
Table learning rate used for policy and value network each task. Note that learning rate 0.0003 specifically chosen for Hopper to allow for better comparison with PPO on that task. As shown in Figure 7 (a), using the learning rate 0.0007 in Hopper actually results better performance.

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
B. Additional Experimental Results
Hopper
Walker
Halfcheetah
4000
Mane Aan
5000
2000
1000
Po Steps
Policy Stops
Policy Stops
Ant
Humanoid
5022
VPGorepaat-1
1000
4330
VRG-repeat-10
VPGreppeat-3
4230
VPd rupeat 50
VPG repeet-pptttt 100
VPG-repeat--o 500
1032
1292
2000
2003
4000
5000
1000
3052
5003
Palicy Steos
Po Iry Steps
Figure 10. The cumulative rewards versus policy step in each MuJoCo task.
Hopper
Walker
Halfcheetah
202
200
1220
180
10
Diffeencce
-50
200
-15
Global Stops
Gooal Stops
Global Stops
Ant
Humanoid
232
VPG-repaat-1
0Phrence
Defference
repeat
VPG reprrrepeatti 10
VPGGruppat-S
VPG repeat 100
-000
VICGreepeal-000
-25-
0.0
Globsl Steps
Global Steps
Figure 11. A full comparison of value estimation difference for each implementation.
12

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
C. Theoretical Analysis of Value Estimation
Existing work has shown that the optimization landscape the policy space is fractal in many continuous-contre environ- ments. To better characterize this behavior, we first introduce the concept of Hölder continuity:
Definition C.1. (Hölder continuity) Let f RR -> & be function. Given I E & and a E (0, 1]. we say that f is a-Hölder continuous at æ if for any 0, there exists C₁ > 0 such that
- -
< Cilla' - xx
for all n' R& that has ||=² x1| S 8.
Note that Hölder continuity is equivalent to Lipschitz continuity when a = 1, and the function can be highly non-smooth and fractal when a < The following theorem establishes connection between the chaotic behavior in an MDP and the smoothness of the corresponding policy optimization objective:
Proposition C.2. (Wang et al., 2023) Assume that the dynamics, reward function and policy are all Lipschitz continuous with respect to their input variables. Let ne be a deterministic policy and X(0) denote the maximal Lyapunov exponent of the dynamics. Suppose that X()) log Y and let X(O
logg then
1. Value function V (s) HöHölr continuous the state s E S;
2. Q-funtion (s, a) is a- aöHölder continuous in the action a A:
3. Policy objective J (0) is a-Hölder continuous in the policy parameter A R.V.
Proof of Theorem 5.2. According to Proposition C.2, we have the following estimation:
- J(0)
01l
(14)
where X(0) is the maximal Lyapunov exponent of the dynamics controlled by Te. which is typically positive in many continuous-contrd environments. Illustrations of fractal landscapes in several MuJoCo environments are shown in Figure 1, where even small update in the parameters can lead to significant change in the value of the policy objective J()).
Unlike policy networks, which can lead to dramatic changes in the return even with a single parameter update, the training of value networks is generally more stable, as it minimizes the regression loss:
L. (d)
VIVe Viarget Il
(15)
where D represents the collected data, Vo is the parameterized value approximation, and Viarget is the target value.
For most activation functions used in neural networks, such as tanh and ReLU, the network output is Lipschitz continuous with respect to both the input variables and the network parameters. Specifically, for a given state s and parameters 6 and d', the difference in the value network output can be estimated by
VVs;; - V(s; 6) O1llo' - +1))
(16)
when - is sufficiently small. This suggests that the landscape of the value regression loss in equation (9) is smooth (e.g., as illustrated in Figure 12), such that small updates in the parameters always lead small changes in the estimated values.
Now, let us consider the policy improvement. Let 0g and du denote the parameters of the policy and value networks at the k-th iteration, respectively. Note that equation (14) suggests that for a fixed state s, the discounted return Σ 7 R, is log -Hölder continuous with respect to the policy parameter. Therefore, the difference in the target value between two X(e) consecutive policy steps can be estimated by
| ((())) Vearget, (s)l K1B, A
13

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
2
Figure 12. Value objective L V (6) is always smooth the network parameter ø.
for some constant K1 > 0 where $1 is the learning rate of policy network. We can also find another constant K₂ > 0 such that the difference in the value network output can estimated by
(s; ox+1) V (s; dx)l ~ K2Kv82.
where Kv is the number of optimization steps for the value network and 32 is the learning rate. Therefore, the step sizes should satisfy
K 3
Ky
(17)
K252
so that the value network continues to provide accurate estimates of the true return and we complete the proof.
14

---

Improving Value Estimation Critically Enhances Vanilla Policy Gradient
D. Monte-Carlo Value Estimation
Most on-policy algorithms, including VPG, TRPO, and PPO, estimate the return value using either Monte Carlo methods or their generalizations (e.g.. GAE (Schulman al., 2015b)). However, the return estimated by Monte Carlo methods can only reflect how good (or bad) the sequence of actions a = (do, d1....,,y. .T) is whole, given the initial state so ~ p. Consequently, this return may deviate significantly from the true value V (su) due to: (a) randomness in the stochastic policy Ta, and (b) exponential divergence of perturbed trajectories in the (potentially) chaotic dynamics.
Mollified value landscapes. It has been demonstrated that policy gradient methods work by smoothing the value landscape through a Gaussian kernel, thereby providing valid updating direction even in the presence of fractal structures within the value landscape ( (Wang et al., 2024). While the averaged return can point to a correct direction, it also implies that a single trajectory is not sufficiently informative. For instance, consider two sampled trajectories with action sequences (ao. a),.... (......ar))............. and a' (ab. a1. ................................ dfr), where on ~ dg. The resulting returns, G and G', could be entirely different despite the initial actions being nearly identical. This discrepancy arises because subsequent actions may differ. In such cases, A( on) and A(su: d)) could have opposite signs, even when dg = dg. Therefore, it underscores the importance of relying on the averaged return rather than focusing too much on individual trajectories.
Understanding the Role of Basclines estimator since
Theoretically, both REINFORCE (3) and VPG (4) provide the same gradient
R (es,ar) ae)
V log (a)l)))))))))
)o
V Vlogno(arls:)) + (s:))
V Vlogro(a)(s))A)
under the assumption that
Ba v log Te [s)]
(18)
for any 8t and any probability density function my. However, Equation 18 is not guaranteed in practice, where the number of samples may be insufficient to support it. Here, we demonstrate that the number of samples required to ensure the empirical < c becomes prohibitively large when e is very small: Without loss of generality, let Ty be a one-dimensional Gaussian distribution, i.e., a ~ (N((()), o²). Suppose that a1, ui,...; Ox are i.i.d. samples obtained from To. The empirical mean
mean satisfies |اا| = V log
1 K
(a. p(s))²
AK = logga(a;1s)) = K
a²
This implies that -Ax ~ (K). Therefore, for each state 8t, approximately O(K) samples are needed. This requirement results in a total of O(KT) samples per iteration.
This approach is exhaustive, as it involves exploring sufficiently many actions for each state encountered every timestep. Consequently, it converges to tabular search approaches, offering no reduction in computational complexity. However, such exhaustive exploration is clearly not the intended purpose of policy gradient methods. In fact, this process resembles the vine TRPO algorithm, which partially expands the tree due to computational constraints (Schulman et al., 2015a).
is worth noting that we have disregarded the accumulation of errors arising from the hierarchical structure. If these errors were taken into account, the required number of samples would even larger, but we do not elaborate further on this point. Therefore, the assumption ~oss V log Te Te(aa) [s 0 for any S; is too strong be realistically satisfied in practice. This implies that using a baseline can significantly influence the performance of the gradient estimator. Specifically, when there no baseline value function or when the bascline function is poor, the policy gradient estimator tends to converge toward suboptimal solutions. This behavior explains the poor performance of the original VPG algorithm in Gymnasium.
15


Reviews:
[{"review_id": "aH29MqI2kN", "reviewer": "Reviewer_duhH", "summary": "This paper investigates the role of value estimation accuracy in on-policy policy gradient reinforcement learning. The authors demonstrate that improving the accuracy of value estimation (by performing more value function updates per iteration) dramatically improves the data efficiency of vanilla policy gradient methods. While it is widely accepted that TRPO and PPO perform well because they prevent destructively large policy updates, this paper suggests that value estimation plays a more critical role in their success than previously recognized.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough and insightful feedback.\n\nFor claims:\n\n1. In Figure 2(c)(d), we observe that the new loss with 50 epochs effectively enforces the trust region compared to runs with fewer epochs, despite its poorer performance. This coincides with the suggested alternative approach. Alternatively, we may adopt a weaker statement: trust regions do not necessarily need to be enforced for policy improvement.\n\n2. In Figure 5 and 11, we observe a significant value estimation error in the original VPG implementation with only one value step per iteration. Therefore, reducing the number of value steps in TRPO and PPO would likely lead to similar results, as the value estimation module is identical across all on-policy algorithms.\n\n3. See later responses.\n\n4. This point is very interesting, and we have considered. Actually, we found that it is hard to say if an improvement in trust region enforcement is equivalent to a given improvement in value estimation. One possible metric is the number of optimization steps made to the policy and value networks in PPO. However, it is not appropriate, as Theorem 5.2 indicates that the policy and value networks evolve at different rates, meaning that applying the same number of gradient steps does not necessarily lead to equal improvements. Additionally, ratio clipping in the policy further complicates the comparison. While Theorem 5.2 provides a lower bound, determining the exact ratio of gradient steps that results in equivalent improvements in value estimation and trust regions is impossible. We will rephrase the draft accordingly based on your suggestions.\n\nFor experiments:\n\n1. In the default PPO setting used in our paper, the value network takes 320 gradient steps per iteration, which is already sufficient to provide strong value estimation, as shown in Figure 5.\n\n2. We adopted the maximum ratio metric from Engstrom et al.. In Figure 2, only a single random seed was used, as it serves primarily as an illustrative example. However, we have re-run the Hopper task with five seeds for further validation.\n\nRegarding the empirical results, we observe that the mean fraction of ratios violating the trust region bounds increases from 0.19 to 0.32 through training, with a standard deviation of 0.01 across five random seeds using the original PPO implementation from Tianshou. A similar trend is also reported in Figure 3 of [1]. These observations support our claims, as a non-negligible fraction of ratios consistently violate the trust region.\n\nGradient clipping is applied only to the experiments in Sections 5 and 6 and is no longer used in the illustrative examples in Section 4.\n\n3. Only the observation is normalized.\n\nFor weaknesses:\n\nIn the default PPO setting, it takes 320 optimization steps per iteration, compared to 501 steps in VPG-repeat-500. However, Figure 4 shows that VPG-repeat-50 already outperforms PPO with just 51 steps. This suggests that the approach is scalable.\n\nFor comments:\n\n1. When referring to performance, we talk about the final return and will clarify this.\n\n2. Figure 1 shows that the return landscape of TRPO is fractal and violates the assumptions made in the theory. However, despite this, TRPO still performs well in practice. This suggests that its empirical success may not be entirely explained by the theory of trust regions, which then motivates the findings presented in our paper.\n\n**For questions**:\n\n1. For the experiments in Section 6, **we used the PPO implementation from Tianshou with learning rate decay disabled (disabled in VPG, too)**, which could explain the performance decline observed in Walker and Ant. We will update the hyperparameters in Appendix A to reflect these. We believe that anyone using this setting can reproduce the same results and will release the code afterwards.\n\nAs analyzed in Section 4, ratio clipping in PPO is not sufficient to effectively enforce trust regions or completely prevent large policy updates. This limitation helps explain why the default PPO struggles to solve the Humanoid task. Additionally, random seeds may also contribute to the observed performance differences. Similar declines in performance on Ant are reported in Figure 17 of the ICLR '21 paper by Liu et al..\n\nRegarding the learning rate of VPG, we adopted 7e-4 from Stable-Baselines3. The 3e-4 was specifically chosen for Hopper is to allow for a better comparison with PPO on that task. **As shown in Figure 7(a), using 7e-4 in Hopper actually results in better performance.**\n\n2. See previous responses.\n\n3. Our results show that VPG can perform similarly to PPO with enhanced value estimation, which is the key contribution of this work. **However, with PPO, other factors also play a role. As previously mentioned, PPO's performance may decline due to its aggressively updated policy**, which explains why VPG-repeat-500 outperforms PPO despite both having similar value estimations.\n\n[1] Wang et al., Truly Proximal Policy Optimization, UAI, 2020.", "claims_and_evidence": "The paper makes several key claims, which I evaluate below.\n\n**Claim 1: Trust regions are neither sufficient nor necessary for guaranteed policy improvement.**\n\n* **Partially supported.** The paper convincingly shows that the trust region is typically violated in PPO implementations, yet PPO still learns effectively. This supports the argument that enforcing a trust region is not necessary for policy improvement. However, the claim that trust regions are not sufficient for policy improvement is less convincing. The authors test this by adding a supervised loss (Eq. 9) to the PPO objective that encourages the policy to stay at the edge of the trust region. My concern is that this auxiliary loss may dominate the update and potentially drown out the contribution of the policy gradient, effectively preventing the policy from following the policy gradient direction. An alternative approach to test this claim might be to reduce the learning rate while setting the number of epochs to a much higher value, which would push the policy close to the trust region boundary while reducing trust region violations.\n\n**Claim 2: TRPO and PPO implicitly enhance value estimation in their implementations.**\n\n* **Theoretically supported but lacks empirical validation.** Theorem 5.2 establishes that increasing the number of value function updates yields more accurate value estimates under certain conditions. However, the paper doesn't directly demonstrate that TRPO and PPO improve value estimation by performing multiple value updates per iteration. To strengthen this claim, the authors should experiment with reducing the number of value function updates in TRPO and PPO to observe if this produces worse value estimates.\n\n## Claim 3: By increasing the number of value update steps per iteration, vanilla policy gradient can achieve performance comparable to or better than PPO.\n\n* **Partially supported.** Figures 4 and 5 demonstrate that more accurate value estimation improves the data efficiency of VPG substantially. However, the comparison to PPO appears problematic as the PPO implementation seems suboptimally tuned in these experiments. Prior work has shown that well-tuned PPO performs consistently across all MuJoCo tasks, while the results here show declining performance in environments like Ant and Walker. Nevertheless, the substantial improvements to VPG through enhanced value estimation are clearly demonstrated.\n\n**Claim 4: Performance of a policy gradient algorithm is primarily determined by value function estimation accuracy rather than specific policy training mechanisms.**\n\n* **Not adequately supported.** The paper doesn't establish a clear framework for comparing the relative importance of value estimation versus trust region mechanisms. To support this comparative claim, the authors would need to demonstrate that a given improvement in value estimation yields better performance than an equivalent improvement in trust region enforcement. Additionally, Figure 5 shows that VPG-repeat-N achieves smaller value estimation differences as N increases, yet PPO's value estimation differences are similar to VPG-repeat-500, suggesting this metric may not strongly correlate with performance. I recommend rephrasing this claim to focus on the overlooked importance of value estimation rather than asserting its primacy.", "methods_and_evaluation_criteria": "1. The benchmarks and baseline algorithms are appropriate for the RL experiments.\n\n2. In the experiments for Figure 2: Looking at the max ratio here could be misleading. What does the distribution of ratios look like? What if only one ratio is large and the rest are very small? Is there a reason Figure 2 only shows one seed per method? Appendix A shows that experiments include gradient clipping -- this will also implicitly prevent large updates and should be removed for the purpose of this analysis.", "theoretical_claims": "I briefly looked at the proof of Theorem 5.2. One question: Does Eq. 17 simply follow from a Taylor expansion?", "experimental_designs_or_analyses": "1. However, the experimental design would be stronger if it also examined increasing the number of value updates in TRPO and PPO implementations, not just VPG. If the paper's hypothesis is correct, TRPO and PPO should also benefit from more accurate value estimates, and demonstrating this would strengthen the paper's conclusions.\n\n2. In the experiments for Figure 2: Looking at the max ratio here could be misleading. What does the distribution of ratios look like? What if only one ratio is large and the rest are very small? Is there a reason Figure 2 only shows one seed per method? Appendix A shows that experiments include gradient clipping -- this will also implicitly prevent large updates and should be removed for the purpose of analysis.\n\n3. Do VPG, TRPO, and PPO normalize observations and rewards?", "relation_to_broader_scientific_literature": "To my knowledge, the role of value estimation in the context of PPO has been understudied.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Additional Strengths**\n1. I think this is an interesting topic, and I have considered investigating it myself, so I'm happy to see this paper. \n\n**Additional Weaknesses**\n1. is VPG with 100-500 value updates per iteration a scalable algorithm? What is the computational cost of VPG vs. VPG-repeat-500 vs. PPO? To clarify, I think it's fine if the computation cost is large. I think it's useful to comment on the cost though. If it's not scalable, that might be an interesting direction for future work.", "other_comments_or_suggestions": "> Vanilla Policy Gradient (VPG, (Sutton et al., 1999)), typically performs significantly worse than the modern algorithms. The common explanation for why TRPO and PPO outperform VPG is their ability to prevent excessive large policy updates. \n\nWe can make these statements more precise: VPG is typically less *data efficient* than the modern algorithms. The common explanation for why TRPO and PPO outperform VPG is their ability to *perform large policy updates while preventing destructively large updates.*\n\nVPG performs a single gradient update per batch of data collected and must use a small learning rate to prevent destructively large changes to the policy. TRPO and PPO have mechanisms that mitigate destructively large policy updates, so they can perform larger gradient updates (TRPO) and more gradient updates per batch collected (PPO), which yields more data efficient learning. \n\n> the gradient ∇θJ(θ) may not exist for many continuous control problems in the first place.\n\nDoes this statement allude to how we may be trying compute the policy gradient at corner in the optimization surface? \n\n> ...VPG can achieve performance comparable to PPO.\n\nThis statement could be more precise. Are we talking about final performance, data efficiency, etc?\n\n> Figure 1\n\nAre we plotting the loss surface for a randomly initialized policy? Does the optimization surface smooth out as the policy improves? Is \\$theta$ a multi-layer network, a linear network, etc? TRPO works well empirically even if the loss landscape is fractal (e.g. TRPO can perform reasonably well on MuJoCo tasks like Hopper and Walker), so is this actually a problem?\n\n> Figure 5\n\nIt is difficult to read this figure because of the white markers -- I suggest removing them.\n\n> Notably, during PPO training, the maximum probability ratio maxt rt can significantly violate the trust region defined by the clipping coefficient ε (Engstrom et al., 2020). This occurs because the gradient ∇θrt vanishes outside the trust region (e.g., when rt > 1 + ε) due to the clipping operator. Consequently, if an excessively large step is taken on πθ(at|st) such that rt leaves the trust region, it may not be pulled back to the trust region anymore due to the vanishing gradient.\n\nI think this issue may arise due to an implementation detail of PPO. In practice, PPO stops updating once we *violate* the KL threshold $\\delta$ rather than *just before* we violate the threshold. To better enforce the KL constraint, PPO should roll-back the update that caused the policy to violate the threshold. \n\nFor future work, it would be interesting to run VPG/PPO/TRPO with exact value estimates to see just how much we could potentially improve over the base algorithms. If this is too computationally demanding for MuJoCo benchmark tasks, it would be valuable to study even in a simple grid world task, where value iteration could compute V(s) exactly. This would establish an upper bound on the potential gains from perfect value estimation across different algorithms.", "questions_for_authors": "**I lean to reject because of the concerns I raise in the following questions. If these questions are appropriately addressed, I will raise my score.**\n\n1.  I was surprised to see PPO perform so poorly on Walker and Ant, since it is possible for PPO to perform well on all MuJoCo tasks (e.g., see the [PPO benchmark by Tianshou](https://tianshou.org/en/stable/01_tutorials/06_benchmark.html)). Is PPO tuned? Is VPG tuned? In appendix A, it looks like the same hyperparameters are used across all PPO experiments, but different VPG learning rates are used for each task, suggested PPO is not tuned while VPG is. If this is true, PPO should also be tuned if you want to fairy compare VPG with PPO. On that note, I want to mention again that I don't think VPG needs to outperform PPO to make this work valuable, though it would be valuable to show that improving value estimate also improves PPO performance, since PPO is so widely used.\n\n2. Regarding Figure 2: Could you show the distribution of probability ratios across multiple seeds, rather than just the maximum ratio? While the results show that the maximum ratio significantly violates the trust region, this doesn't tell us whether such violations are widespread or isolated. If the vast majority of ratios still fall within the trust region bounds and only a small fraction violate it, then PPO may still be effectively maintaining approximate trust region constraints in practice, which contradicts claim 1. If a non-negligible fraction of ratios violate the trust region, then the stated claims are okay.\n\n3. Regarding Figure 5: PPO and VPG-repeat-500 achieve similar value estimate differences, yet VPG performs much better. Why? This observation seems to contradict claims 3 and 4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FuAtgcpLwv", "reviewer": "Reviewer_aCm7", "summary": "This paper studies the performance gap between vanilla policy gradient and PPO/TRPO type of trust region enforcing algorithms. The authors conclude through empirical studies that the core to the performance gap lies in value estimation, as opposed to the common belief of trust region enforcing.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and hope that our response can address your concerns.\n\nRegarding the weaknesses:\n\nWe used the PPO implementation from Tianshou with learning rate decay and reward normalization disabled to better compare algorithmic performance without excessive code-level optimizations. We also acknowledge that mini-batch size is a crucial factor in PPO's performance and have conducted an ablation study on this in Figure 9(b) in the Appendix.\n\nThe value estimation error of PPO and VPG is compared in Figure 5, which shows that the default PPO implementation provides good value estimates, as it performs 320 value steps per iteration, which is sufficient for accurate estimation. We agree that policy initialization is crucial for on-policy algorithms, so we fixed the random seed to ensure that both PPO and VPG start from the same initial policy and value network.\n\nFor the question:\n\nThe question raised here is insightful. Our algorithm focuses on continuous control settings, where PPO is widely believed to consistently outperform VPG. However, in other domains, such as language modeling (LLM), even the simplest REINFORCE algorithm has been reported to be able to outperform PPO ([1]).\n\nWe agree that despite being the standard benchmark in deep RL, the success in MuJoCo tasks does not guarantee its effectiveness in other domains. In this paper, both our theoretical analysis and empirical results are focused on continuous control and robot learning, as emphasized in the Conclusion. We will revise the paper to ensure that our claims are clearer and more specific to this context.\n\n[1] A. Ahmadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, arXiv:2402.14740.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "An interesting argument for commonly used RL algorithms.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths:\nOverall this paper is a clear empirical report on the performance of VPG as compared with region enforcing optimization algorithms. Especially clear and convincing is the empirical studies on the correlation between value estimation error and performance of vanilla policy gradient (figure 4 and 5). \n### Weaknesses:\nPPO performance is known to be highly sensitive to hyperparameters and mini-batch sizes [1]. It is unclear from the experiments performed that the performance dominance is from an improper choice of hyperparameters of PPO or the fact that VPG is with multiple update steps. It would also be more informative to compare the value estimation error of PPO against VPG.  \nSome prior work [2] has also done extensive study on such continuous control benchmarks and addressed that policy intialization actually plays an important role in the performance on-policy algorithms. It would be more extensive if the authors could provide insight on this observation as well.\n\n[1] Paine, Tom Le, et al. \"Hyperparameter selection for offline reinforcement learning.\" arXiv preprint arXiv:2007.09055 (2020).\n[2] Andrychowicz, Marcin, et al. \"What matters in on-policy reinforcement learning? a large-scale empirical study.\" arXiv preprint arXiv:2006.05990 (2020).", "other_comments_or_suggestions": "None.", "questions_for_authors": "All the benchmarks are conducted in continuous control settings, where the reward at each timestep is crucial, making value function estimation a core aspect of the problem. However, in domains such as language modeling and other discrete action settings, the action space tends to be significantly larger, and reward signals are often much sparser. Given these differences, it is unclear whether the same observations hold in these environments. The reliance on value function estimation may not translate as effectively, and the conclusions drawn from continuous control tasks may be more domain-specific than universally applicable, as the authors suggest.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5IDR3p8YKE", "reviewer": "Reviewer_pFVP", "summary": "The authors provide a novel perspective how the approximation error of the value function determines the performance of the algorithm more so than the proximal constraint in TRPO and PPO. they argue this both empirically and theoretically. They further argue that the accuracy of the value function adds to robustness against perturbations of hyper-parameters.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the important review. Hope that our response adequately addresses your concerns.\n\nFor claims:\n\n1. This argument directly corresponds to the Theorem 1 from the TRPO paper, which states that\n$$J(\\theta) \\geq L^{TRPO}(\\theta) - \\frac{4 \\beta \\gamma}{(1 - \\gamma)^2} D_{KL}^{max}(\\pi_\\theta \\ \\| \\ \\pi_{\\theta_{old}})$$\nwhere $J(\\theta)$ is the original policy objective, $L^{TRPO}(\\theta_{old})$ is the surrogate objective and $D_{KL}^{max}$ is the KL distance between the old and new policies. Since it always has $J(\\theta_{old}) = L^{TRPO}(\\theta_{old})$. Therefore, the original policy objective is guaranteed to improve only if \n$$J(\\theta) - J(\\theta_{old}) \\geq L^{TRPO}(\\theta) - L(\\theta_{old}) - \\frac{4 \\beta \\gamma}{(1 - \\gamma)^2} D_{KL}^{max}(\\pi_\\theta \\ \\| \\ \\pi_{\\theta_{old}}) \\geq 0$$\nwhich requires that the surrogate objective is improved by a margin greater than $\\frac{4 \\beta \\gamma}{(1 - \\gamma)^2} D_{KL}^{max}(\\pi_\\theta \\ \\| \\ \\pi_{\\theta_{old}})$. While the theory proves the above inequality when two policies are close enough, there is no way to guarantee that the second part of the inequality above always holds in practice.\n\nWe will add the citation there and rephrase the statement as follows for clarification: \n\n> According to the theory of trust regions (cite TRPO paper here), there is a gap between theory and practice because the theory guarantees improvement of the original policy objective only when the surrogate objective improves by a margin greater than the distance between the old and new policies.\n\n2. The landscape is fractal due to the chaotic dynamics of the MDP, which are independent of the policy parameterization. The loss landscapes in [1, 2] correspond to the actor loss over a given batch of data, rather than the return landscape shown in Figure 1. For further reference on fractal landscapes, please refer to [1].\n\n\n3. That is a good point. This paper focuses on continuous control problems where the MDP is deterministic. The only source of randomness comes from the Gaussian policy we use, which is why we can make the Lipschitz assumption on the dynamics and reward. The proof of Theorem 5.2 can be found in Appendix C. The intuition behind having $\\beta_2$ in the denominator is that smaller learning rates require more value steps to compensate.\n\n4. We agree that PPO is more sample efficient than VPG, as confirmed in the second paragraph of the Conclusion. However, this comes at the cost of reduced stability, as demonstrated in the Ant and Humanoid results in Figure 4, where VPG-repeat-k significantly outperforms PPO when k is greater than 50. When referring to performance, we mean the final performance of each algorithm, and we will clarify this in the draft. We believe that the current empirical results are sufficiently convincing to show that VPG with enhanced value estimation can achieve comparable performance to PPO, as initially claimed.\n\nWe will also smooth the reward curves.\n\nFor experiments:\n\n1. We ran that for a single seed as it was only an illustrative example, and we have re-run the Hopper task with multiple seeds for further validation. Please see our response to Reviewer duhH for the detail.\n\n2. We hypothesize that larger gradient magnitudes used in TRPO may positively contribute to its performance improvement over VPG. However, if this were the case, it would contradict the core idea in the TRPO paper, which favors small policy updates. Additionally, it has been shown that larger learning rates often lead to poor performance [2]. Furthermore, in Section 6, we examine how learning rates influence performance, demonstrating that excessively large gradient steps can negatively impact the final return.\n\n3. $r_t = \\frac{\\pi_\\theta}{\\pi}$ is exactly the probability ratio in PPO, so it depends on $\\theta$, which is estimated in the same way as in PPO. $r_t^*$ is the target probability ratio, as defined in lines 170-171.\n\n4. Yes, in lines 266-268, we mentioned that PPO uses a single optimization loop for both networks, which leads to ratio clipping that prevents the policy from over-updating.\n\nFor the suggested references, we will add the following discussion accordingly to the main text:\n\n\"It has been demonstrated that proper reward scaling can lead to overall performance improvements in deep RL [3].\"\n\n\"Like VPG, PPO uses a single optimization loop for both networks, which can be found in commonly used implementations such as CleanRL [4].\"\n\nWe will also revise the paper to address the minor issues you suggested.\n\n[1] Rahn et al., Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control, NeurIPS '23.\n\n[2] Andrychowicz et al., What matters in on-policy reinforcement learning? a large-scale empirical study, ICLR '21.\n\n[3] Schual et al., Return-based Scaling: Yet Another Normalisation Trick for Deep RL, 2021.\n\n[4] Huang et al., CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms, JMLR '22.", "claims_and_evidence": "I have the following issues with the claims:\n\n1. Right hand column line on page 1:\" The gap between theory and practice arises because the theory guarantees improvement of the original policy objective only when the surrogate objective improves by a margin greater than the distance between the old and new policies.\" This is very strong. There are numerous gaps between theory and practice. I believe you must cite some other work or demonstrate this.\n\n2. You claim that the objective is not differentiable and reference Wang et al 2023. I would think that the landscape becomes a lot smoother under the over-parametrized neural network's regime. This is because NNs implicitly regularize. Also, see References [1,2] below, the loss landscape is \"smoother\" upon being projected to low-dimensional space. \n\n3. **Theorem 5.2** I would recommend the authors explain what they mean by \"Assume that the dynamics, reward function\" being Lipschitz continuous in theorem 5.2. This is especially important since this is a stochastic setting. It would benefit the reader by helping them understand the setting. Without that the claims of theorem 5.2 seem slightly difficult to grasp. Could you please explain why you end up with $\\beta_2$ in the denominator by means of a brief proof sketch? I believe that would help understand the claim \"This result suggests that the value network should be optimized more aggressively than the policy network\".\n\n4. I am not sure if the results in Figure 4 are very convincing to the larger point you make. I believe PPO is more sample efficient as compared to VPG-repeat-k methods. I would at least add smoothing to the curves to make the figures more interpretable, as it is standard practice.\n\n----------------------------\nReferences\n----------------------------\n1. A framework for training larger networks for deep Reinforcement learning, Ota, Kei and Jha, Devesh K and Kanezaki, Asako, 2024\n\n2. Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimization. Recep Yusuf Bekci and Mehmet Gumus, 2020", "methods_and_evaluation_criteria": "I believe the biggest drawback in the methods are the single seed experiments in Figure 2.", "theoretical_claims": "I did not verify the correctness of theorem 5.2.", "experimental_designs_or_analyses": "I see the following issues with experiment designs:\n\n1. Do you run the experiment on a single seed for Figure 2? This might not be conclusive although it it indicative. A broader set of experiments for various environments might be necessary in this case.\n\n2. \"VPG should have performed well, as it always applies smaller learning rates compared to TRPO and fewer policy steps per iteration compared to PPO\" -> What about large gradient magnitudes? \n\n3. I am unsure as to how you compute $L^{SL}$. What is the dependence of r on $\\theta$? Is it through the policy? Further, how do you sample for estimating this ratio and what is the true value $r*$?\n\n4. \"The TRPO implementation, however, requires separate optimization loops \" -> I believe this issue was addressed for PPO and SAC by the cleanRL implementation and this should be discussed in addition to analyzing the implementation given its popularity and usage.\n\n5. I am not sure if the results in Figure 4 are very convincing (see above).", "relation_to_broader_scientific_literature": "I believe this is an important line of work empirically exploring the shortcomings of deep RL methods and rectifying them.", "essential_references_not_discussed": "I believe the following references should be discussed:\n\n1. Return-based Scaling: Yet Another Normalisation Trick for Deep RL, Tom Schaul, Georg Ostrovski, Iurii Kemaev, Diana Borsa, 2021: this paper relates to reward scaling and should ideally be discussed in Section 6.\n\n2. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms, Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, 2021, https://github.com/vwxyzjn/cleanrl this codebase deals with issues of doing a single loop for both value and policy networks and I ould expect the authors to comment on this.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "Minor issues:\n1.  According to (Sutton et al., 1999) -> According to Sutton et al., 1999, line 85 column 2\n2. Line 135 col 2: I find it hard to believe that the landscape for over-parametrised NNs is fractal. Could you provide some other reference than Wang et al 2023? while I believe that for a two parameter problem these are highly irregular but it might not be the case for NNs which implicitly smoothen the landscape.\n3. In the \"Ratio trust regions in PPO\" section and \"Trust region methods\" section you use $r_t$ for probability ratio I would recommend using some other symbol because $R$ is used for reward and this can be a bit confusing.\n4. Section 5.1: Please provide the reference for GAE here.\n5. Section 5.2: \"Similarly, having a properly optimized value network is also\" I would not start a new subsection with \"Similarly\".", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ron0H4McZd", "reviewer": "Reviewer_rHz8", "summary": "This paper demonstrates, through both theoretical analysis and experiments, that vanilla policy gradient (VPG) can achieve performance comparable to PPO by simply increasing the number of value updates per iteration. Contrary to the common belief and existing claims, the authors argue that accurate value estimation is more critical than enforcing trust regions. They provide theoretical justification and experimental validation to support this claim.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their positive and supportive review!\n\n> However, the assertion that trust regions (“trust regions are neither sufficient nor necessary for guaranteed performance improvement”) requires further theoretical backing.\n\nWe will revise the statements in the paper to more accurately reflect both the current theoretical analyses and the empirical results.\n\n> Including experiments with PPO and TRPO would provide a more comprehensive evaluation.\n\nPlease see the answer to Q2 below.\n\n> Broader evaluations across more algorithms (TRPO) and environments would strengthen the conclusions.\n\nThat's a good point. We had considered running additional experiments in the DM Control Suite but finally decided against it after realizing that almost all on-policy algorithms perform poorly in that setting. For example, as shown in Appendix C.2 of https://arxiv.org/abs/2501.16142, PPO struggles in these environments. In contrast, only off-policy algorithms like SAC and TD-MPC consistently achieve strong performance.\n\nRegarding the weaknesses:\n\n1. We agree that further investigation into enhancing value estimation in PPO is a promising direction for future work. Please refer to the answer to Q1 below for a more detailed response.\n\n2. We will revise the statements in the paper to more accurately reflect both the current theoretical analyses and the empirical results.\n\n3. Please see the answer to Q2 below.\n\nAnd for the questions:\n\nQ1: While this paper focuses on the pivotal role of value estimation in policy gradient methods, as highlighted in the title, this question is one of the directions we plan to explore in future work. It will be interesting to investigate how much PPO can benefit from enhanced value estimation.\n\nIn the default PPO setting considered in this paper, the value network is updated with 320 gradient steps per iteration, which is generally sufficient for good value estimation. However, since PPO optimizes both the policy and value networks within a single loop, the same number of steps is applied to the policy network. This results in the policy network being over-updated, which contributes to the poor performance observed in the Ant and Humanoid tasks.\n\nTherefore, one possible direction is to reduce the number of optimization steps in PPO, as we have already shown that 50 value steps are sufficient for good value estimates. Additionally, the empirical results in this work indicate that even a single policy step per iteration can lead to good performance, suggesting that we can also reduce the number of policy steps in PPO. In Figures 9(c) and (d) in the Appendix, we demonstrate that a PPO implementation with 8 gradient steps per iteration significantly outperforms the default PPO implementation on the Ant and Humanoid tasks. These results provide concrete evidence that this approach could be a promising direction.\n\nWe believe that by reducing the number of optimization steps, PPO can become more wall-time efficient and more robust to hyperparameters.\n\nQ2: That is an insightful question! In Figure 4, we run the VPG algorithm with varying numbers of value steps, which demonstrates that having the maximum number of value steps per iteration does not always result in the best performance. Although no significant performance decline is observed as the number of value steps increases, we found that 50 value steps per iteration yields the optimal final return across five different environments. Additionally, we did not observe any instability even with an increased number of value steps.\n\nThe key insight we aim to highlight through this work is that enhanced value estimation can significantly improve the simplest VPG algorithm. The empirical results presented in Figure 4 suggest that even for those with limited experience in hyperparameter tuning, using a sufficiently large number of value steps can yield decent results. However, we agree that incorporating additional techniques to prevent overfitting could further enhance the robustness of VPG. In particular, since optimizing the value network is essentially a supervised learning problem, methods like early stopping and regularization, commonly used in deep learning, could be beneficial.\n\nQ3: Of course! The code for the experiments will be released once the paper is accepted.\n\nThank you again! We will revise the paper accordingly based on your suggestions.", "claims_and_evidence": "The claims regarding the importance of accurate value estimation are supported by empirical results. However, the assertion that trust regions (“trust regions are neither sufficient nor necessary for guaranteed performance improvement”) requires further theoretical backing.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate. Including experiments with PPO and TRPO would provide a more comprehensive evaluation.", "theoretical_claims": "The theoretical analysis of baselines and sample complexity is sound. Additional theoretical discussion on the necessity of trust regions would be valuable.", "experimental_designs_or_analyses": "The experimental design is appropriate and the results are convincing. Broader evaluations across more algorithms (TRPO) and environments (e.g.,dm_control https://github.com/google-deepmind/dm_control) would strengthen the conclusions.", "relation_to_broader_scientific_literature": "The paper makes a novel contribution by reframing the role of value estimation in policy gradient methods. It builds well on existing work policy gradient/optimization literature.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper presents a compelling insight that accurate value estimation, rather than trust region enforcement, is key to policy improvement in policy gradient methods. \n- The theoretical and empirical analysis is clear and well-structured, particularly the explanation of how TRPO and PPO implicitly enhance value estimation by performing more value updates per iteration.\n- Experimental results seem strong, with thorough evaluations across environments. \n- The findings have practical significance, suggesting that simpler algorithms like VPG, when combined with improved value estimation, can perform comparably to more complex methods such as PPO.\n\n**Weaknesses**\n- The experiments focus mainly on VPG with enhanced value estimation, without evaluating how the proposed approach performs in conjunction with PPO and TRPO. A direct comparison would strengthen the claims.\n- Some statements, such as \"trust regions are neither sufficient nor necessary for guaranteed performance improvement,\" are quite strong and would benefit from additional theoretical justification beyond empirical observations.\n- The potential downsides of iterative value estimation, such as overfitting or diminishing returns, are not sufficiently addressed. Clarification on this would improve the paper.", "other_comments_or_suggestions": "- Highlight the baseline analysis from the appendix more clearly in the main text.\n- Clarify any safeguards used to prevent potential overfitting in iterative value estimation.", "questions_for_authors": "Q1. How does the proposed method perform when combined with PPO and TRPO rather than VPG? A direct comparison would clarify the general applicability of your findings.\n\nQ2.  Are there situations where additional iterations of value estimation lead to degraded performance? How do you prevent overfitting or instability?\n\nQ3. Will you release the complete implementation code and hyperparameter configurations to ensure reproducibility?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}]

Author rebuttal:

Meta-review: This paper challenges the common belief that trust region is the key factor for the good performance of modern policy optimization methods such as TRPO and PPO. The authors present a compelling argument, supported by both theoretical analysis and empirical evidence, that enhanced value estimation accuracy plays a more critical role. The show that vanilla policy gradient (VPG) can achieve comparable performance to PPO by simply increasing the number of value update steps.

From my read, I found the paper has several strengths. It provides a though-provoking perspective on the importance of value estimation in policy gradient methods. And the empirical studies are conducted in a reasonably convincing way to me.

However, reviewers also found that the paper has some weaknesses. There are valid concerns about the experimental validation (the tuning of PPO and the potential impact of hyperparameters) and generalizability of the findings (in continuous setting value estimation is an issue, which means the findings might not be generalized to other settings such as discrete and LLM related settings). Some reviewers also found that some claims are too strong and the experimental results only support partial or weaker versions of those claims.

Overall, the theoretical and empirical evidence presented is sufficient to warrant a weak accept. The authors should revise the paper to address the remaining concerns and clarify the scope of their claims.
Decision: Accept (poster)