Improving Deep Policy Gradients with Value Function Search
Authors: Enrico Marchesini
Christopher Amato
Venue: ICLR 2023 poster (2023)
Keywords: Deep Reinforcement Learning
Deep Policy Gradients

Abstract: Deep Policy Gradient (PG) algorithms employ value networks to drive the learning of parameterized policies and reduce the variance of the gradient estimates. However, value function approximation gets stuck in local optima and struggles to fit the actual return, limiting the variance reduction efficacy and leading policies to sub-optimal performance. This paper focuses on improving value approximation and analyzing the effects on Deep PG primitives such as value prediction, variance reduction, and correlation of gradient estimates with the true gradient. To this end, we introduce a Value Function Search that employs a population of perturbed value networks to search for a better approximation. Our framework does not require additional environment interactions, gradient computations, or ensembles, providing a computationally inexpensive approach to enhance the supervised learning task on which value networks train. Crucially, we show that improving Deep PG primitives results in improved sample efficiency and policies with higher returns using common continuous control benchmark domains.

Published as a conference paper at ICLR 2023
IMPROVING DEEP POLICY GRADIENTS WITH VALUE FUNCTION SEARCH
Enrico Marchesini, Christopher Amato
Khoury College of Computer Sciences
Northeastern University Boston, MA, USA {e m..arhhessni,, c. c.amato/@horthhestemm..i
ABSTRACT
Deep Policy Gradient (PG) algorithms employ value networks to drive the learn- ing of parameterized policies and reduce the variance of the gradient estimates. However, value function approximation gets stuck in local optima and struggles to fit the actual return, limiting the variance reduction efficacy and leading policies sub-optimal performance. This paper focuses improving value approxima- tion and analyzing the effects on Deep PG primitives such as value prediction, variance reduction, and correlation of gradient estimates with the true gradient. To this end, we introduce a Value Function Search that employs a population of perturbed value networks search for a better approximation. Our framework does not require additional environment interactions, gradient computations, or ensembles, providing a computationally inexpensive approach enhance the su- pervised learning task on which value networks train. Crucially, we show that improving Deep PG primitives results in improved sample efficiency and policies with higher returns using common continuous control benchmark domains.
INTRODUCTION
Deep Policy Gradient (PG) methods achieved impressive results in numerous control tasks (Haarnojal ctt al. (018). However, these methods deviate from the underlying theoretical framework to compute gradients tractably. Hence, the promising performance of Deep PG algorithms suggests lack of rigorous analysis to motivate such results. [lyas et al.] (2020) investigated the phenomena arising in practical implementations by taking a closer look at key PG primitives (e.g., gradient estimates, value predictions). Interestingly, the learned value networks used for predictions (critics) poorly fit the actual return. As a result, the local optima where critics get stuck limits their efficacy in the gradient estimates, driving policies (actors) toward sub-optimal performance.
Despite the lack of investigations understand Deep PG's results, several approaches have been proposed to improve these methods. Ensemble learning (Lee et al., [02T; He et al.) [022). for example, combines multiple learning actors' (or critics") predictions to address overestimation and foster diversity. These methods generate different solutions that improve exploration and stabilize the training, leading to higher returns. However, the models used at training and inference time pose significant challenges that discuss in Section 5 Nonetheless, popular Deep Reinforcement
Learning (RL) algorithms (e.g.. TD3 (Fujimoto et al. (0188) use two value networks, leveraging the benefits of ensemble approaches while limiting their complexity. To address the issues of ensembles, gradient-free free methods have been recently proposed (Khadka & Tummer (0rche/ & Farinelli, 2020 Sigaud 2022). The idea is complement Deep PG algorithms with search mechanism that uses a population of perturbed policies to improve exploration and to find policy parameters with higher payoffs. In contrast ensemble methods that employ multiple actors and crities, gradient- free population searches typically focus on the actors, disregarding the value network component of Deep PG. Section 5 discusses the limitations of policy search methods in detail. However, in PG context, critics have pivotal role in driving the policy learning process as poor value predictions lead to sub-optimal performance and higher variance in gradient estimates (Sutton & Barto 2018

---

Published as a conference paper at ICLR 2023
Such issues are further exacerbated in ofate-of-the-art Deep PG methods as they struggle to learn good value function estimation (llim et al. (2020). For this reason, we propose a novel gradient-free population-baset approach for critics called Value Function Search (VFS), depicted in Figure [ We aim to improve Deep PG algorithms by enhancing value networks achieve (i) a better fit of the
Deep Policy Gradient
Stoce
Memory Burfer
Policy
Value Function
Sample
Pertasbetions
Updase
Value Functions Population
Prediction Erors
Evalurtion
actual return, (ii) a higher correlation of the gradients es-
Value Functions Search
timate with the (approximate) true gradient, and (iii) re-
Figure 1: Overview of VFS.
duced variance. In detail, given a Deep PG agent charac-
terized by actor and critic networks, VFS periodically instantiates population of perturbed critics using a two-scale perturbation noise designed to improve value predictions. Small-scale perturba- tions explore local value predictions that only slightly modify those of the original critic (similarly to gradient- basedd perturbations (Lehman et al. 2018 Martin H. & de Lopel 2009; Marchesimi & Amato 2022). Big-scale perturbations search for parameters that allow escaping from local op- tima where value networks get stuck. In contrast to previous search methods, evaluating perturbed value networks require computing standard value error measures using samples from the agent's buffer. Hence, the Deep PG agent uses the parameters with the lowest error until the next periodical search. Crucially, VFS's critics- based design addresses the issues of prior methods as it does not require a simulator, hand- sand-designed environment interactions, or weighted optimizations. Moreover, our population's 's goal is to find the weights that minimize the same objective of the Deep PG critic.
We show the effectiveness of VFS on different Deep PG baselines: (i) Proximal Policy Optimization (PPO) (Schulman et al. 2017), (ii) Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2016), and (iii) TD3 on a range of continuous control benchmark tasks (Brockman et al., 2016 Todorov et al.] 2012). We compare over such baselines, an ensembled Deep RL method, SUN RISE (Lee et al. 2021). ). and the policy search Supe-RL. (Marchesini et al. 2021a). In addition, we analyze key Deep PG primitives (i.e., value prediction errors, gradient estimate, and variance) to motivate the performance improvement of VFS- based algorithms. Our evaluation confirms that VFS leads to better gradient estimates with lower variance that significantly improve sample efficiency and lead to policies with higher returns. Our analysis highlights a fundamental issue with current state-of- the-art Deep PG methods, opening the door for future research.
2 BACKGROUND
PG methods parameterize a policy To with a parameters vector (typically the weights of a Deep Neural Network (DNN) in Deep PG context), and ne (a,l(s) models the probability to take action a₂ in a state 8t at step f in the environment. These approaches aim to learn 0 following the gradient of an objective ne over such parameters (Sutton & Barto 2018). Formally, the primitive gradient estimate on which modern Deep PG algorthmss build has the following form (Sutton et al. 1999):
V he ( (Vllogrg mn)
where (st, E ~ are states and actions that form the trajectories sampled from the distribu- tion induced by Tg. and Qa (s,, ax) is the expected return after taking a, in 8ç- However, Equation I suffers from high-variance expectation, and different basclines have been used to margin the issue.
In particular, given a discount value 7 E [0, 1), the state-value function Vr, (s) is an ideal baseline as it leaves the expectation unchanged while reducing variance (Williams 1992):
Va (s) = E.s
G. = Σ (TT411
(2)
where G. is the sum of future discounted rewards (return). Despite building on the theoretical framework of PG, Deep PG algorithms rely on several assumptions and approximations for design- ing feasible updates for the policy parameters. In more detail, these methods typically build on surrogate objective functions that are easier to optimize. A leading example is TRPO (Schulman eet al. 2015), which ensures that a surrogate objective updates the policy locally by imposing trust region. Formally, TRPO imposes a constraint on the Kullback-Leebler diverge (KI.) on successive policies no, no,, resulting in the following optimization problem:
To (o(())))) Aro (8c, at) s.t. DKL (xxz(T((ia (Tx((((())) < (or (sx)
max Egg
2

---

Published as a conference paper at ICLR 2023
where 8 is a hand-tuned divergence threshold, and A(s,a) A(s,a)= Q(s, V(s) is the advantage fune- tion. In practical implementations, Equation 3 is tractably optimized with additional tricks: (i) a second-order approximation of the objective, (i) mean KL of the current trajectory instead of the actual KI. divergence. Given TRPO's computational demands. [Schulman et al. (2017) further "re- laxes" the optimization landscape by replacing the constraint with the following clipped objective, providing significantly faster (i.e., less demanding) yet better-performing alternative over TRPO
To (a)())))) Tig (agl(a)))
e, 1 + A(se, A(ss;
max Bm 0
min
A(se; at); clip
(4)
Te (ar((as)) (xc)
Similarly, Lillicrap et al. (0016) uses the deterministic PG theorem (Silver et al. 2014) for learn- ing deterministic policies (s) DDPG and further optimizations (e.g.. TD3) are tightly related to standard Q-learning (Watkins & Dayan 1992) and assume having differentiable action-value function over actions. Deterministic PG algorithms leam the value parameters as in standard Double DQN. In contrast, policy parameters are learned under the maxe Ex, [Qo(s, [s)(s)] optimization problem, where á are the parameters of the critic. Moreover, TD3 addresses the overestimation of DDPG with additional tricks, e.g., learning two value functions, being pessimistic about the action value to use for the target computation, and delaying the policy updates.
In summary, Deep PG algorithms build their success on learning value functions for driving the learning of the policy parameters 0. However, significant research efforts have been devoted to improving policies, while the attention improving crities appears marginal. Hence, to our knowl- edge. VFS is the first gradient-fre free population- based approach that works on top of Deep PG to improve crities without facing the complexities of learning ensembles or policy searches.
2.1 GRADIENT- INFORMED PERTURBATIONS
Policy search approaches generate a population of perturbed versions of the agent's policy To to ex- plore variations with higher returns (Khadka & Tumer 2018 Marchesini & Farinelli 2022 March- esimi et al. 2021b Colas et al. 2018 [Siaaud] (022). To this end, gradient-informec perturbations (or variations) have been employed for limiting detrimental effects on no. exploring small changes in the population's 's action sampling process (Lehman et al.) [0018] MMrrchesini et al.) 2022). In more
detail, prior methods express the average divergence of a policy at state s over a perturbation w as:
d(TT, w) [-|s) (To+4(-1s)11
(5)
and use the gradient of such divergence to approximate the sensitivity of 0 over the policy decisions, which is used to normalize Gaussian-based perturbations. Formally, following following[Lehman et al. (2018):
Ve. (o]d( w) 2 Ved(TO. 0) - Ha(d(Te, 0))
(6)
where Hg is the Hessian of divergence with respect to 6. Although simpler first-order approxi- mations achieve comparable results, gradient- based perturbations add significant overhead to the training process. Prior work introduced gradient- based variations because evaluating the popula- tion against detrimental changes is hard. As such, the quality of perturbed policies is assessed over many trajectories to ensure that they effectively achieve higher returns over arbitrary initializations Marcheesiii et al. 1a2 In contrast, we focus on critics-based search where high-scale per- turbations may help the learning process to escape from local optima, achieving better predictions. Hence, VFS relies two-scale perturbation approach maintain similar value predictions to the original value function while exploring diverse parameters to escape from local optima.
3 VALUE FUNCTION SEARCH
We introduce a gradient-free population-based search with two-scale perturbations to enhance critics employed in Deep PG. Ultimately, we aim show that value functions with better predictions improve Deep PG primitives, leading to better sample efficiency and policies with higher returns.
Algorithm shows the general flow of the proposed Value Function Search framework. During standard training procedure of a Deep PG agent, VFS periodically generates a population P of per- turbed value networks, starting from the agent's current critic parametrized by é. We first instantiate
We do not consider the unconstrained penalty-hased PPO due the better performance of the elipped one.
3

---

Published as a conference paper at ICLR 2023
copies of & and two Gaussian noise distributions Smin, max with Jwin, max standard deviation for the two-scale variations (lines 3-4). Hence, we sample from the two distributions to apply each per- turbation to half of the population (lines 5-6). In contrast to single-scale or gradient-based perturba- tions of prior work (Khadka & Tumer 2018 Marchesini et al. 2021a Sigaud 2022), the two scales approach for value networks has the tollowing advantages (i) Mmin maintains similar predictions to those of the unperturbed critic, similarly to gradient- based variations. Sampling from (O, m translates into small changes in the value prediction process to find a better set of parameters within a local search fashion. (ii) Gmax is a Gaussian with greater standard deviation (00, o m used to explore diverse value predictions to escape from local optima. (iii) Gaussian- based perturbations do not introduce significant overhead as gradient-based variations. After generating the population, VFS samples batch b of trajectories from the agent's buffer B. There are different strategies for sampling the required experiences from the agent's 's memory based on the nature of the Deep PG al- gorithm. When using an on on-policy agent (c.g., TRPO, PPO), we apply VFS after updating the policy and value networks. In this way, reuse the same on- on-policy data to evaluate whether perturbed set of weights has better value prediction (i.e., lower error). Off-policy agents (e.g., DDPG, TD3) could follow similar strategy. Still, the reduced size of the mini-batches typically employed by off-policy updates would result in noisy evaluations. For this reason, we sample larger batch of experiences to provide more robust evaluation. In practice, to compute the prediction errors for each value network in the population (line 7). we use the typical objective used to train PG critics, whose general formulation is:
the Mean Squared Error (MSE) (Sutton & Barto 2018),
1
MSE (b)
V+(() Vo,, (s,,(s)) (s)
SEb
where V (s) is the true value function. However, is typically unfeasible to access the true V₂ (s) and, as in standard training procedures of Deep PG algorithms,, we rely on noisy samples of V (s) or bootstrapped approximations. In more detail, on-policy updates are partieularly interesting, as Deep PG algorithms typically learn critics with Monte-Cart methods without bootstrapping the target value. Hence, performing a supervised learning procedure on (st, G.) pairs, where the return G, is an unbiased, noisy sample of Vs(s) that replaces the expectation of Equation 2 with empirical average. We then refer to a perturbed critic as a local improvement of the original unperturbed one when it achieves lower error on b. In our context, given value network parametrized by o, batch of visited trajectories b, and a perturbation w, we define the value network parametrized by to be local improvement of when m886+ (b) (b) Moreover, in the limit where b samples all the trajectories, G, = Vn (s2), and mS8 S mse&(b) means that 6 + w is a better or equal fit of the actual value function with respect to the original c. Crucially, Monte-Carlo value approximation converges to a local optimum even when using non-linear approximation (Sut-) ton & Barto 2018). Hence, on-policy VFS potentially maintains the same convergence guarantees of the baseline To this end, it would be possible to anneal the perturbation scales to zero out the impact of VFS over the training phase. Hence, providing the diversity benefits of VFS early in the training while maintaining the convergence guarantee of the baseline algorithm within the limit.
Algorithm 1 Value Function Search
1: Given: (i) Deep PG agent with a value network parametrized by d at training epoch e; (ii) buffer B of visited trajectories; (iii) periodicity e, for VFS and population size n; (iv) scale mmin. max for the two-scale Gaussian noise G.
2: if e % c, 0 then
3:
P+ (do...... dn)) copies of & Gmin.max t- -O0 miin,max)
4:
5:
11.101 + Φ; + Gmin @Igi.. + 6, + Gmax
6: 6 + Sample batch of trajectories from B
7:
msep &- Evaluate P over as Equation
8:
Φ - min (msep)
0.EP
9: end if
We support claims on our perturbation approach with additional experiments in Section 4 Duee to hootstrapping, the same result generally does not apply temporal difference targets.
4

---

Published as a conference paper at ICLR 2023
Nonetheless, discussed in llyas et al. (2020). Deep PG algorithms deviate from the underlying theoretical framework. We performed additional experiments in Section 4 to show that maintaining VFS through the entire learning process results in better performance over zeroing out the pertur- bations. Moreover, in a worst-case scenario where the population search never improves the value prediction locally, VFS matches the same local optima of the original approach. However, following the insights of Fujimoto et al. (2018), practical Deep PG algorithms can not theoretically guarantee to reduce estimation errors or improve predictions outside the batch used for computing the update.
VFS addresses most limitations of previous policy search approaches and minimizes the same ob- jective of the Deep PG critics, leading the following crucial advantages:
VFS does not require additional environment interactions or hand hand-designed evaluations for selecting the best set of parameters in the population.
MSE is widely adopted objective for learning value networks, providing established evaluation strategy. Moreover, batch computations and parallelization make this process particularly efficient. Nonetheless, VFS is not explicitly designed for MSE, and extending it to different error measures is straightforward.
At the cost of adding overhead, the population evaluation may scale to an arbitrarily large number of samples to improve robustness.
Finally, the parameters with the lowest error replace the current agent's critic until the next popu- lation search (line 8). Many Deep RL algorithms employ target networks parametrized by Oig to reduce overestimation. In this scenario, we also update the target weights toward the new ones by using standard Polyak averaging with an a E (0, 1) weight transfer value: Qig = ax - (1 (1- ))
Limitations. Here we highlight the limitations of VFS: (i) batched computations and parallelization do not avoid a marginal overhead induced by the population search. (ii) In an off-policy setting, VFS requires tuning the batch size for the error computation balance the trade-off between com- putational demands and the evaluation quality. (ii) There are no guarantees that small-scale pertur- bations produce a local variation the original predictions. However, they have been previously shown effective in doing so (Martin H. & de Lope 2009), and we confirm this result in Section 4
4 EXPERIMENTS
The proposed experiments aim answering the following questions: (i) How do VFS components affect the training phase. In particular, we investigate when the perturbations are most effective during the training. (ii) How these components impact the performance and sample efficiency of VFS- algorithms. (iii) How VFS influences Deep PG primitives, such gradient estimates, their variance, and value predictions, to motivate performance improvement.
We investigate the performance of VFS on the on-policy PPO, the off-policy DDPG, and TD3 al- gorithms. We compare over the baseline DDPG, PPO, TD3, SUNRISE (Lee et al. 202T) (a recent ensemble approach), and Supe-RL (Marchesini et al.] 2 (a recent policy search approach). We conduct our experiments on five continuous control tasks based on MuJoCo (Brockman et al., 2016) (in their v4 version), which are widely used for comparing Deep PG, ensembles, and policy search approach (Schulman et al., 2017; Lillicrap et al. 2016 Lee et al. 2021 2021 Fujimoto et al. 2018).
Data are collected nodes equipped with Xeon E5-2650 CPUs and 64GB of RAM, using the hyper- parameters of Appendix F. Considering the importance of having statistically significant results (Colas et al. 2019)) we report the average performance of 25 runs per method, with shaded regions representing the standard error. Such a high number of experiments motivates slightly different results over the original implementations of DDPG, TD3, and PPO.
4.1 ANALYZING VFS COMPONENTS
This analysis considers data collected in the Hopper task with VFS-PPO, as we achieved comparable results in the other setups. Given our intuitions, the two-scale perturbations have different effects on the critic. We expect miin perturbation to result in networks with similar predictions. In contrast, Fmax should generate networks with more diversified values to escape local optima. Appendix A shows the average difference between values predicted by the original critic of the baselines and their variations, confirming the role of the two perturbations.
5

---

Published as a conference paper at ICLR 2023
Table 1: Performance 1M steps for Hopper, Walker, Swimmer, and 2M steps for HalfCheetah, and Ant in PPO, DDPG, TD3 experiments. We show the average return and standard deviation over the 25 runs and the percentage of steps required by VFS methods to reach the peak performance of the corresponding baseline. We report the best number for the ensemble SUNRISE at 2 x 10 steps (Lee al. 202T), comparing over a SAC implementation of VFS and the policy search Supe-RL.
PPO
VES-PPO
DDPG
VES-DDPG
Arg.
Return
Steps
Arg. Return
Steps
Hopper
2287 805
2878 695
.48.5
Hopper
1734 = 810
2397 637
-34.9
Walker
2384 : 1074
2905 1: 932
-12.4
Walker
1753 858
2492 755
-38.2
Swimmer
26
22
-38.6
Swimmer
46
-36.1
HalfCheetah
2560 = 767
3471
252
-38.8
HalfCheetah
7948 2702
9208
711
-45.4
Ant
4333 2: 677
5403
120
-27.1
Ant
1060 938
2060
446
-24.4
TD3
VES-TD3
VES-SAC
Supe-RL (SAC) SUNRISE
Arg. Return
Steps %
Avg. Return
Hopper
2558 +: 1029
3185 +: 398
-44.8
Walker
3773 + 1052
4185 + 860
-58.36
Hopper
2573 420
992 + 360
2643 4.42
Swimmer
123 2:
134 +: 11
-34.0
Walker
2101 798
1932 : 842
1236 1123
HalfCheetah
10232 + 1201
12430
-58.3
HalfCheetah
7145
822 7177 + 1226
4501 443
Ant
4003 727
4669 + 382
-20.4
Ant
1667
481 1398 = 676
1502 - 483
Moreover, we expect max variations to be rarely bet- ter than min variations and the original critic, as dras- tic changes in the value predictions serve to escape from local optima. To confirm this, Figure 2 shows the aver- age cumulative number of times on which amin (blue) or mmax (orange) crities' error is the lowest in the population compared to the original critic (y-axis), over the periodi- cal searches (x-axis). The linear growth of the Jmin bars indicates that such perturbations produce better variations over max and the original critic through the training. contrast, max perturbed networks have more impact in the early stages of the training (where the critics' predic- tions are seldom accurate) and occasionally achieve lower eтror in later stages of the training.
max
Cum. Updates
200
Avg.
[00
10
20
30
N' SSarches for VFS-PPO
Figure 2:
Cumulative number of times when a miin,max variation improve the critic and the other perturbation.
4.2 EMPIRICAL EVALUATION
Here discuss the performance and sample efficiency improvement of VFS-based algorithms. In particular, we show: (i) the average return at convergence and (ii) the percentage of samples required by VFS implementations the results for PPO, VFS-PPO, DDPG, VFS-DDPG, TD3, and VFS-TD3 (the VFS-TD3 algorithm applies the search to both critics). each table, we highlight the highest average return bold and refer to Appendix B for an exhaustive overview of the training curves. Generally, FFS-based algorithms significantly improve average return and sample efficiency, confirming the benefits of enhancing critics during Deep PG training procedures. On average. VFS-PPO achieves 26% return improvement and uses 33% fewer samples to reach the peak performance of PPO. VFS-DDPG has a 29% return improvement and 35% improvement in sample efficiency. Moreover, VFS-TD3 achieves an average 19% return improvement and 43% fewer samples to reach the peak performance of TD3. The latter results significant TD3 employs several tricks to address the poor performance of DDPG, and it is known to be a robust Deep PG baseline (Fujimoto et al.) 2018)
achieve the peak performance of the Deep PG baselines. Table T shows
We also compare VFS over Supe-RL and the ensemble method SUNRISE by considering its best number reported at 2 x 10a steps. We use the same population parameters as in our VFS-TD3 im- plementation and Supe-RL, while the SAC parameters are the same as the original work (Haarnoja et al. 2018 Crucially, the VFS implementation shows comparable or superior average perfor- mance over such additional baselines (x13% and ==24% on average, respectively), confirming the merits of our framework. Finally, Appendix C contains: (i) comparison of the computational
We do consider the Swimmer environment as prior work has employed it.

---

Published as a conference paper at ICLR 2023
300
2003
AVg Roturn
Retum
200
Jog.
09
Steps
Steps
208
Steps
100
Figure 3: Left: influence of different population sizes in VFS-PPO. Increasing the size of the pop- ulation leads to higher performance. Middle: comparison between PPO, VFS-PPO, and VFS-PPO with annealing perturbations. Linearly scaling our perturbations to zero leads to lower performance with respect to the proposed VFS-PPO with fixed permutations. Right: comparison of two VFS- PPO implementations that consider only small or big scale variance for the permutations.
overhead for VFS, Supe-RL, and SUNRISE, discussing the negligible overhead of VFS over such baselines. (ii) comparison with a baseline PPO that considers higher number of gradient steps to match the computational demands of VFS.
4.2.1 ADDITIONAL EXPERIMENTS
This section shows: (i) the effects different population sizes; (ii) additional experiments to show that annealing variations have lower performance than the proposed VES; (iii) how VFS behaves with considering only miin or max perturbations. Similarly to Section 4.1, we show the results of VFS-PPO in the Hopper domain they are also representative of the other setups.
Figure 3 on the left shows the performance with increasing sizes for the population (i.c., 4, 8, 12, 20). Intuitively, increasing the size of the population leads higher performance. However, our experiments show no significant advantage in considering a population of size greater than 10.
In the center, Figure 3 compares PPO, VFS-PPO, and VFS-PPO with annealing on the perturba- tions. In more detail, we linearly scale the effects of the perturbations to zero towards step 9 x 10a. As discussed in Section 3, after zeroing VFS's perturbations, our approach is equivalent to the considered Deep PG baseline and, such, has the same convergence guarantee in the limit. Nonetheless, in practice, VFS-PPO without permutation annealing achieves superior performance as the critic's population search keeps improving the Deep PG primitives.
Figure 3 on the right show the performance of VFS with only small or big-scale perturbations. The latter permutations have more impact in the early stages of the training and allow for higher returns over both PPO and VFS-PPO. However, such benefits rapidly fade as the training progresses, resulting in performance similar to the PPO baseline. Conversely, small-scale variations achieve higher returns in the middle stages of the training. However, the lack of big variations to escape from local optima only leads to marginally superior performance over PPO. In contrast, the proposed two-scale VFS-PPO combines the benefits both permutations. Such results are further motivated by an additional analysis of the gradient estimate Appendix D.
4.3 DEEP PG PRIMITIVES ANALYSIS
We analyze how VFS influences Deep PG primitives to motivate performance improvement. To this end, Figure 4 left and middle show the quality of the gradient estimates and their variance at the initial and middle stages of the training (respectively). and the mean square ervor of PPO's critic and VFS-PPO one. Regarding gradient estimates, PPO's common implementations estimate the gradient using empirical mean of ~ 20 samples (marked in the plots with vertical line). Since the Deep PG agent successfully solves the task attaining a high reward, these sample-based estimates are sufficient to drive successful learning of the policy parameters.
We measure the average pairwise cosine similarity and standard deviation between 30 gradient mea- surements taken from the PPO and VFS-PPO models after 10a and 5 x 10a steps in the Hopper task with an increasing number of state-action pairs. A cosine similarity e (0, 1] represents a positive similarity (where 0 means that the vectors are orthogonal and means that the vectors are paral-
7

---

Published as a conference paper at ICLR 2023
Sup 194k
Suy S96
P013
6.-
13
18 13
18
300
306
304
4.0
308
10L
Figure 4: Left, Middle: analysis of gradient estimate and variance. We measure the average pairwise cosine similarity (y-axis) between 30 gradient estimates with a growing number of state-action pairs (X-axis). Each algorithm averages over 25 models collected during the training at 10 and 5 x 10 steps. A higher correlation with lower variance of VFS-PPO translates into better performance of the baseline (Figure D. Right: average MSE of such models over the training phase.
lel). Hence, higher correlations indicate the gradient estimate more similar the actual gradient [ Following the PG's underlying theoretical framework, we also know that a lower variance in the gradient estimate typically translates into better performance, which is why value networks are em- ployed in Deep PG. Crucially, VFS-based approaches show, on average, z 0.1 higher correlation and lower variance in the sampling regime used during the training. Following the intuitions of the cosine similarity measure, such an improvement is significant and motivates the performance benefits of VFS-based approaches. We also performed a two-sample t-test to test whether the mean gradient estimates of each baseline and its VFS implementation detail, we considered the mean and standard deviation of the estimates at the sample regimes used for the algorithms (i.e., 2048 for PPO) and our sample size of 30 trials. To summarize, the equality of the average gradient estimates between PPO and VES-PPO in the 10 and 5 x 10 experiments with 95% and 99.9% confidence levels, respectively (I-stat 2.22 and 5.05).
statistically different. In more
can reject
Moreover, Figure on the right shows the average MSE of the 25 trained models for PPO and VFS- PPO computed every 10S step. To assess the quality of the critics' predictions, we compute the average MSE between PPO and VFS-PPO critics and the actual return of 10s complete trajectories in the environment (i.e., we compute the average MSE over 10 evaluation episodes every 10a step). To reduce the scale of the plot and obtain a better visualization, we normalize the resulting MSE by the number of considered steps. Our results show that critics enhanced with VFS consistently result in lower MSE, which leads to the previously discussed improvements in the gradient estimates B Hence, our analysis of key Deep PG primitives motivates the performance improvement of VFS- based algorithms. Appendix D shows the same analysis for VFS-DDPG and VFS-TD3, confirming our results.
5 RELATED WORK
Ensemble methods have been used to improve policy performance in Deep RI. (He et al. 2022
Lee et al.) | Yang et al.) 2022 Wu & Li 2020; Ren al. (22T]) combining the predictions of multiple learning models. However, the benefits of these approaches over population-base ones unclear, the numerous training actors (and/or critics) introduce a significant memory footprint and -egligiiee overhead at training and inference time. In addition, weighting the importance of different predictions is not straightforward. Hence, ensemble algorithms in Deep RL are unsuitable for: (i) deployment on-device where memory is limited, (ii) real-time systems where inference time crucial, or (iii) training tigh-dimensional DNNs where parallel computation may be possible. In contrast, population-base -asedd approaches are straightforward to implement and have been used to combine the exploration benefits of cradient-free methods and gradient-basee algorithms (Khadka & Tumer 2018, Marchesini et al. 2021a Sigaud 2022) for policy parameters search. In more detail, ERL (Khadka& Tumer 2018) employs Deep PG agent and concurrent population that includes the gradient-based agent to explore different policy permutations. The agent trains following its
Althoouhh obtaining an empirical estimate of the true gradient can improve Deep PG performance further, this would require orders of magnitude more samples than the ones used in practical applications. Using such an amount samples unfeasible due to the computational demands (Ilyas et al. (020).
Usiin more trajectories for approximating the true value function would lead to better approximation and higher MSE for both approaches in the carly stages of the training where predictions less accurate.
8

---

Published as a conference paper at ICLR 2023
gradient-base' based algorithm and shares memory buffer with the population to learn from more diver- sified samples. However, ERL's dropout-like like permutations bias the population's 's performance in the long run. Inspired by ERL. many policy search approaches emerged in the literature (Colas et al. 2018 Bodnar 2020] Alois Pourchot 2019, Marchesini et al. 2021a). In more detail, GEP-PG (Co- las et al. 2018) uses curiosity driven approach to enhance the experiences in the agent's 's memory buffer. Moreover, Proximal Distilled ERL (PDERL) (Bodnar (020) introduces novel gradient-base -base permutations to address destructive behaviors of diologically-inspiee perturbations that could cause catastrophic forgetting (Lehman et al. 2018). Finally. Ru (Marchesini et al. (021a) com- bines the intuitions of the previous population- based approaches. proposing a framework applicable to (possibly) any DRL, algorithms that do not exhibit detrimental policy behaviors. However, due to their policy-oriented nature, these approaches assume simulated environment to evaluate the perturbed policies, which increases the sample inefficiency of Deep RI. algorithms. Such additional interactions are also task-specific as they must cover a broad spectrum of behaviors to ensure a robust evaluation. As discussed in Marchesini et al. 2021a). such behavior diversity is crucial to ensure that a perturbed policy improves the original actor, not only in particular environment initializations. In addition, current policy search methods introduce gradient-based perturbation (that has non- negligible overhead, as further discussed in (2) to ensure that the population's predictions do not drastically diverge from the original policy. To summarize, population-basee searches address the issues of ensemble methods by exploring different behaviors for the policies but disregarding value networks. In addition. recent work (Lyle al., 2022) discussed the capacity loss of agents that have to adapt to new prediction problems when discovering new sources of payoffs. As such, Deep PG's critics have to adjust their predictions rapidly through the training. These issues further motivate the importance of applying VFS in the training routines.
6 DISCUSSION
Ilyas et al. (2020) recently showed that critics employed in Deep PG algorithms for value predic- tions poorly the actual return, limiting their efficacy in reducing gradient estimate variance and driving policies toward sub-optimal performance (Fujimoto et al. 2018). Despite improving Deep PG performance, population- based search methods have recently superseded ensemble learning due to their computational benefits. However, these approaches typically need more practical investi- gations to understand their improvement. Moreover, previous population search frameworks have been devoted to exploring policy perturbations, disregarding critics and their crucial role in Deep PG.
To our knowledge, VFS is the first gradient-free population search approach enhance critics and improve performance and sample efficiency. Crucially, VFS can be applied to (possibly) any Deep PG algorithm and aims finding set of weights that minimize the same critic's objective. More- over, our detailed analysis of PG primitives in Section 4.3 motivates the performance improvement of the gradient-based agent due to VFS. In addition, our analysis suggests that the gap between the underlying theoretical framework of Deep PG and the actual tricks driving their performance may be less significant than previously discussed Iyyas et al.) (2020). Hence, VFS opens up sev- eral exciting research directions, such as investigating the theoretical relationships between RI. and practical Deep RL implementation. Empirically, VFS also paves the way for novel combinations with policy search methods to provide a unified framework of gradient-free copulation-basee ap- proaches. Finally, given the wide application of value networks in Deep Multi-Agent RL Rashid et al.] (2018); Marchesini & Farinelli (2021); /Wang et al. (2021), it would be interesting to analyze the performance improvement of VFS in such partially-bbeervbble contexts.
6.1 CONCLUSION
VFS introduces a two-scale perturbation operator voted to diversify a population of value networks to (i) explore local variations of current critics' predictions and (ii) allow to explore diversified value functions to escape from local optima. The practical results of such components have been investigated with additional experiments that also motivate the improvement in sample efficiency and performance of VFS-based algorithms in a range of standard continuous control benchmarks. Our findings suggest that improving fundamental Deep PG primitives translates into policies and better sample efficiency.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
This work was partially funded by the Army Research Office under award number W91 0265.
ETHICS STATEMENT
The proposed Value Function Search does focus experiments involving human subjects and does not require sensitive data that involve sex, gender, and diversity analysis. Hence, we do not consider the gender dimension, privacy, data governance issues, or any primary ethic concern high- lighted in the ICLR Code of to be relevant.
However, according to the "Ethies Guidelines for Trustworthy Artificial Intelligence" report pub- lished by the High-Level Expert Group on AI of the European Commission 1 ethical consideration has to take into account the environment and the sustainability of Al systems. Regarding the en- vironmental impact, it is crucial to foster sample efficiency (i.e., reducing the training time for the agents, hence the computational resources used to train them) of Deep RL to lower energy consump- tion and related emissions. In this direction, VFS leads to significant benefits in sample efficiency, which reduces the environmental footprint of VFS-based approaches by considerable margin.
In more detail, employed the Machine Learning CO2 impact calculator (Lacoste et al. 2019 to estimate the CO2 emissions of our experiments According to our hardware, the carbon efficiency of the provider, and the length of our experiments, computed total emission of 52.3 kg CO2. We plan to cope with such emissions using Treedom T0 an online service that allows us to plant trees cope with our carbon footprint.
REPRODUCIBLITY STATEMENT
Significant efforts have been made ensure the reproducibility of VFS. Section 3 detail the flow of VFS-based algorithms with pseudocode and detailed explanations. We also highlight possible limitations of approach in Section 3.1, Moreover, Section 4 states experimental setup, while supplemental material contains a detailed overview of the considered hyperparameters.
"We note that this is precise measurement as trivial map energy consumption CO2 emissions (i.c., the carbon footprint). this end, we employed open-source electricity maps (c.g..
https:/www..deetriiiinpp..ommmm to estimate hetter how carbon-intensive our experiment's 's electricity is. httpp://www.rreedm....
10

---

Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Published as a conference paper at ICLR 2023
REFERENCES
Olivier Sigaud Aloïs Pourchot. CEM-RL:
Combining evolutionary and gradient-based methods for policy search. In International Conference on Learning Representations (ICLR), 2019.
Pietro Lio' Bodnar, Ben Day. Proximal distilled evolutionary reinforcement learning. In Conference on Artificial Intelligence (AAAI), 2020.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. OpenAI, 2016.
Cédric Colas, Olivier Sigaud, and Beere-Yves, Oudeyer. GEP-PG: decoupling exploration and ploitation in deep reinforcement learning algorithms. In International Conference on Machine Learning (ICML), 2018.
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. A hitchhiker's guide statistical compar- isons of reinforcement learning algorithms. arXiv, 2019.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning (ICML), 2018.
Qiang He, Huangyuan Su, Chen GONG, and Xinwen Hou. MEPG: minimalist ensemble policy gradient framework for deep reinforcement learning. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022, 2022.
Andrew Ilyas, Logan Engstrom. Shibani Santurkar, Dimitris Tsipras. Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Con- ference on Learning Representations (ICLR), 2020.
Shauharda Khadka and Kagan Tumer. Evolutionary reinforcement learning. In Advances Neural Information Processing Systems (NeurIPS), 2018.
carbon emissions of machine learning. CoRR, 2019.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: simple unified frame- work for ensemble learning in deep reinforcement learning. In International Conference Ma- chine Learning (ICML). 2021.
Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O. Stanley. Safe mutations for deep and recurrent neural networks through output gradients. In GECCO, 2018.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations (ICLR), 2016.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. In International Conference on Learning Representations (ICLR), 2022.
Enrico Marchesini and Christopher Amato. Safety-informed mutations for evolutionary deep rein- forcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1966-1970, 2022. ISBN 9781450392686.
Enrico Marchesini and Alessandro Farinelli. Genetic deep reinforcement learning for mapless nav- igation. In International Conference Autonomous Agents and MultiAgent Systems (AAMAS), pp. 1919-1921, 2020.
Enrico Marchesini and Alessandro Farinelli. Centralizing state-values in dueling networks for multi-robot reinforcement learning mapless navigation. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4588, 2021. doi: 1 9636349.

---

Published as a conference paper at ICLR 2023
Enrico Marchesini and Alessandro Farinelli. Enhancing deep reinforcement learning approaches for multi-robot navigation via single-robot evolutionary policy search. In International Conference Robotics and Automation (ICRA), pp. 5525-5531, 2022. doi: 10.1109//CR/44463922000
9812341.
Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Genetic soft updates for policy evolution in deep reinforcement learning. In International Conference on Learning Representations, 2021a.
URL htttps:://ppereeieeett net http://wwww.eee/oommmeeeeeeee wwwpmmidd-GGOOOTa
pk
Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Benchmarking safe deep reinforcement learning in aquatic navigation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5590-5595, 2021b. doi: 11109//RR0511118202222111.1
Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Exploring safer behaviors for deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence. 36(7): 7701-7709, Jun. 2022. doi: 10.1609/aaai.v3iii v367772.77.... Index.php/Aae wwt.aaaiieeeeeeeeeeeeeeeeeeeeeee
URL. https: httpp:://jj..ass .org/
Josè Antonio Martin H. and Javier de Lope. Learning autonomous helicopter flight with evolutionary reinforcement learning. Computer Aided Systems Theory, 2009.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foer- ster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018.
Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture--f-expprts for efficient deep reinforcement learning. arXiv, 2021.
LttEkhLscK3]
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017.
Olivier Sigaud. Combining evolution and deep reinforcement learning for policy search: survey. arXiv, 2022.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. De- terministic policy gradient algorithms. In International Conference on International Conference on Machine Learning (ICML), 2014.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), 1999.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: duplex dueling multi-agent q-learning. In ICLR, 2021.
Christopher Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
Ronald J Williams. Simple statistical cradient-followiie algorithms for connectionist reinforcement learning. Machine Learning, 8:229 256, 1992.
Junta Wu and Huiyun Li. Deep ensemble reinforcement learning with multiple deep deterministic policy gradient algorithm. Mathematical Problems Engineering. 2020.
Zhengyu Yang, Kan Ren, Xufang Luo, Weiqing Liu, Jiang Bian, Weinan Zhang, and Dong- sheng Li. Deep ensemble policy learning. arXiv, 2022. openzeview net forum?id= 7NOEOeD-xH
12

---

Published as a conference paper at ICLR 2023
A EFFECTS OF PERTURBATIONS ON VES-PPO, VFS-DDPG, VFS-TD3
This section shows the efficacy of using mmin,max to maintain similar predictions and explore di- versified ones, respectively. In this direction, F shows the average difference between values predicted by the original critic of the PPO algorithm and its two perturbations (amin = 0.000005, max 0.0005) over 100 episodes (i.e., each box is an episode). In particular, we report the data collected in the initial stage of the training (i.e., at step 10"), where green boxes indicate slight variations while red ones represent the highest difference in our evaluation. In more detail, Figure 5 shows the effects of our perturbations at different stages of the training (i.e., at 100k, 500k, 1M steps) for VFS-PPO, VFS-DDPG, VFŞ-TD33
The values of the min perturbed network differ from the original critic by small margin. In con- trast, the critic perturbed with max achieves significantly different values. Hence, the proposed two-scale perturbation approach leads to the desired outcome, and -based variations confirm similar results over computationally demanding gradient-base based perturbations. Interestingly, our ex- periments with different algorithms consider the same set of hyperparameters for the mutation oper- ators (see Appendix E), suggesting that VFS does not require extensive tuning to enhance existing Deep PG algorithms.
B TRAINING CURVES
Following the results in Section 4.2, Figure 6 shows the average return during the training without the five best and worst performing model "I ferent rows, while each column contains different set of algorithms (i.e., VFS-PPO and PPO in the first column, VFS- and DDPG in the second column, VFS-TD3 and TD3 in the last one). As discussed in the main paper, VFS-basee algorithms achieve, on average, higher returns. Inter- estingly, the worst-trained model for FFS-based approaches typically achieves higher returns than the best model of the baselines. In addition, VFS- methods show a lower standard deviation, as reported in Table of the main paper. Moreover, they significantly improve sample efficiency, reaching the peak performance of the baseline algorithms (i.c., PPO, DDPG, TD3) in a fraction of the steps.
We report the results for each environment in the dif-
We performed additional experiments in the Humanoid task for 2 million steps. We aim to show that population-based approaches can scale to more challenging tasks maintaining the same population parameters. Figure F shoows the performance of PPO and VFS-PPO. Interestingly, we note that VFS leads to significantly higher results in this complex domain, suggesting that the limitations of critics in Deep PG algorithms profoundly impact the overall policy return.
C COMPUTATION OVERHEAD
In this section, we discuss the computation overhead of ensemble methods and population-based ones. In particular, we compare the overhead of training and inference time introduced by Supe-RL, SUNRISE, and VFS, considering M = 10 models. The training overhead is comparable for all the approaches as standard Deep RL benchmarks do not require high-dimensional DNNs com- putational demanding architectures that hinder parallelization. Nonetheless, in more complex tasks requiring high- sigh--imenssonal DNNs, it could be computationally unfeasible to parallelize the training models of ensemble methods. especially during backpropagation. As reported by the Supe-RLL RL's authors (Marchesini af., 2021a), the additional environments interactions of policy search meth- ods lead to a maximum 6% overhead even considering parallelization with population of size M. In contrast, VFS does not require multiple training models or additional environment interactions, leading to negligible overhead for the training procedure (i.c., it simply requires additional forward DNNs propagations).
Regarding inference time, policy and value network searches do not involve multiple predictions (except in the periodical search, which has negligible overhead due to batched computations), main- taining the same inference time of the original Deep PG baselines. In contrast, detailed by SUN-
We removed the hest and worst seeds to avoid considering the ones that lead to particularly high or low performance. The same consideration holds for Figure 8.
13

---

Published as a conference paper at ICLR 2023
VFS-PPO
Stops
VES-DDPG
10OK
Stepe
VIS-TD3
11Xt Strpe
IM Steps
Figure 5: Efficacy of the max scales in modifying PPO (on the first row), DDPG (on the second row), and TD3 (on the third row) critics achieve similar predictions more diversified ones to escape local optima, respectively. We observe the average difference (A) between values predicted by the crities and the man, variations over 100 episodes the Hopper environment shown as different boxes.
RISE's authors (Lee al. 2021). their ensemble agents require up 2M x additional inference time.
To summarize, gradient-free search methods generally less computationally demanding than ensemble ones. Hence. the superior return of VFS, sample efficiency, and computational benefits further confirm the merits of introducing crities' search-based approach.
14

---

Published as a conference paper at ICLR 2023
VFS A9O
P9O
VFS 00
DDPG
VIS-1D3
TD3
Any. Retum
850
30%
Hopper-s4
Jog Retren
Skops
Stepe
Walker2d-v4
Avs
Steps
Steps
Stops
Swimmer-vi
Avg Rottm
Stops
Step
Ani- v4
Arg
Sleps
Stepe
Staps
HalfCheetah
Figure 6: Comparison of PPO, DDPG, TD3, and their VFS implementation in common continuous control benchmarks each Each column (i.e., each couple of algorithms) shows the average return during the training over 25 runs. VFS-based implementations improve average return and sample efficiency of the baseline Deep PG algorithms.
In addition, Figure shows the performance of PPO implementation that uses the size of the VFS population as additional gradient steps for its critic. Performing more gradient steps on the critic does not lead significant advantages over the baseline PPO the critic learns to fit better
15

---

Published as a conference paper at ICLR 2023
5330
4330
3000
2330
1030
5O0k
15 3.24
24
Figure 7: Comparison of PPO its VFS implementation in the Humanoid task.
the sampled batch. In contrast, VFS is designed to explore local and global variations of the critic avoid local optima where gradient-basd based learning typically gets stuck. For these reasons, this additional evaluation further confirms the merits of VFS.
PV2
Indditioni gradient
3330
2502
3032
2552
2000
596
Figure 8: Comparison between VFS-PPO, PPO, and PPO implementation that uses the size of the population as additional gradient steps for the critic.
D ADDITIONAL DEEP PG PRIMITIVES ANALYSIS
In Section 4.3, we analyzed the improvement of VFS-PPO in terms of gradient estimate, their vari- ance, and the mean squared error the critic. Such an analysis serves to motivate the performance improvement of VFS-based algorithms. In this section, we report the same comparison for VFS- DDPG and VFS-TD3 to confirm the proposed framework's 's benefits further.
Figure 9 reports the same results for Deep PG primitives analysis for DDPG and TD3. We measure the average pairwise cosine similarity and its standard deviation between 30 gradient mea- surements taken from the algorithms 10 and 5 10 steps in the Hopper task with increasing number of state-action pairs. Crucially, VFS-DDPG and VFS-TD3 show a higher correlation and lower variance in the typical sampling regime used during the training (i.e., mini-batches of size 64 or 128), motivating their performance improvement over the baselines. Similarly to the PPO case, performed a two-sample t-test to whether the mean gradient estimates of each baseline and its VFS implementation are statistically different. To summarize, we can reject the equality of the average gradient estimates between DDPG and VFS-DDPG with 95% confidence level in the 10 and 5 x 10 experiments (t-stat 2.08 and 2.01). In the case of TD3 and VFS-TD3, we can reject the same hypothesis with confidence level of 90% (t-stat 1.73 and 1.66). The latter results are interesting they suggest that TD3's implementation tricks to improve performance benefit policy learning as they lead to gradient estimates that are more similar to the VFS ones. Moreover, the right column shows the average MSE of our models computed every 10a step. Similarly to the PPO experiments, we use the actual return for 10 trajectories in the environment to approximate the true value function.
These results follow the trend of Section 4.3, where VFS critics have lower MSE. Interestingly, we also note that TD33 typically achieves higher gradient estimate correlation with lower variance
16

---

Published as a conference paper at ICLR 2023
DDPO
HMA 0.
VFS
644
022
Any MAT
600
125
216
1004
1848
106
2001
300L
400L
3001
00C1
100
500L
FXCL
Stop
Step
TB3
0.7
06-
375
200
011
00
125
2041
184
206
186
494
-04
094
195
201
104
S1nge
Figure 9: Analysis of gradient estimate and variance (left, middle) in the Hopper task for DDPG and VFS-DDPG in the first row, TD3 and VFS-TD3 in the second row. We measure the average pairwise cosine similarity (y-axis) between 30 gradient estimates which use growing number of state-action pairs (x-axis). For each algorithm, we use the average over the models collected during the training runs 10 and 10 steps. The higher correlation with lower variance of VFS-based implementations translates into better performance the Deep PG algorithm. Average MSE of such models over the training phase (right).
DDPG, which motivates using two value networks to achieve better performance. Our analysis of key Deep PG primitives for DDPG and TD3 confirms the main paper's results.
In addition, we performed the same gradient estimate analysis on two VFS implementations that only employ small or big perturbation operators. Figure T0 shows the results of such an investiga- tion. In particular, VFS with only small perturbations slightly improves over the baseline PPO in the different stages the training. However, can not achieve the higher similarity the VFS-PPO that considers both perturbations. In contrast, VFS with only big perturbations is characterized by a higher correlation in the initial stages of the training. At the same time, its efficacy significantly decreases later in the training phase. Moreover, considering only such big-scale variations typically leads to higher gradient estimate variance. Crucially, this additional gradient estimate analysis con- firms the results of Figure 3 and further motivates using two-scale perturbations to tackle both local and global variations at the same time.
E HYPERPARAMETERS
We performed an initial grid search among common apper-parameters from prior work (Schulman al. 2017 Lillicrap al.) 2016 Fujimoto et al., 2018 Marchesini al. 202 This evaluation led us to use similar DNN architectures for all the al algorithms in the environments, i.e., 2-ReLU layer MLP with 64 hidden units each to model actors and crities for PPO and the same structure but with 256 hidden units for DDPG, TD3. Table reports the other hyperparameters shared across all the experiments. Note that missing parameters are the same as the baseline algorithm (e.g.. VFS-PPO values are the same PPO, and TD3 shares the missing parameters with DDPG). Moreover, all the implementations consider the same values for the critic's s search.
17

---

Published as a conference paper at ICLR 2023
Siep 1006.
Step S0Ck
0.8
Purnter Simbbrky
(Ginia)
06
107
361
10
10F
107
10'
VP 04 Al paits
No 14, poim
Step
Stop 500G
PPO
PPO
Pairato Siibnt
0.5
Vis (Limas)
VeS
VIr (Liman)
VIS
0.4
0.4
8.2
0.-
-02
(6.00 paiss
Figure 10: Analysis of gradient estimate and variance in the Hopper task for VFS-PPO that only considers small perturbations (top row) or big ones (bottom row).
Table 2: Hyperparameters for experiments
Algorithm
Parameter
Value
PPO
clip
0.2
0.99
update epochs
10
samples per update
2048
mini-batch size
64
entropy coefficient
0.001
Ir
0.0003/0.0001
DDPG
? buffer size mini-batch size
0.99
1000000
128
actor Ir
0.0001
critic Ir
0.001/0.0003
0.005
TD3
actor noise clip
0.5
delayed actor update
2
VES
population size
10
Jmin
0.000005
Tmax
0.0005
b
2048
6,
2048 steps
18


Reviews:
[{"review_id": "mz3wFBWPrp", "reviewer": "Reviewer_qNbH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an evolutionary approach to train value functions in actor-critic methods. The main motivation for this approach is to provide a better value approximation, a higher correlation between the estimated gradient and the true gradient, and a lower variance. The authors propose to periodically perturb half the population of value functions using low noise and half using higher noise. The intuition for this is that while the small noise can perform a local search around the current value function, using a high noise for some value functions can escape local minima. After perturbation, the value function with the lowest error on a batch of data is selected. The authors apply their method to on-policy (PPO) and off-policy (DDPG, TD3) algorithms. Experiments show that this addition increases sample efficiency over the non-evolutionary baselines, without introducing significant overhead. Further ablations show that at the beginning of training, the high noise population is mostly responsible for value function improvement, while towards the end of training most improvement is due to the small noise perturbation. Moreover, the gradients produced by the presented approach appear to have a higher correlation to the true gradient and the value function loss is lower. ", "summary_of_the_review": "The paper introduces a novel evolutionary approach for evolving value function in actor-critic methods. The method is easy to implement and shows improvement upon on-policy and off-policy baselines. Some of the claims made in the paper should be properly justified and some experimental concerns should be resolved before acceptance.", "strength_and_weaknesses": "Strengths:\n- The method can be easily implemented and produces little overhead\n\n- The method seems to improve upon the baselines when it is applied\n\n- The paper is clear and easy to follow. The authors provide some ablations indicating important differences when using high or low noise for perturbation\n\n- I appreciate that the authors included Figure 3 in the paper, which confirms the intuition that higher noise might be more helpful at the beginning of training on PPO. Does the same hold for off-policy methods?\n\nWeaknesses:\n\n- The authors show a comparison between the proposed VFS-TD3 and Supe-RL and SUNRISE. This comparison seems not fair unless these two approaches are also built on top of TD3. Is this the case? I believe TD3 itself would outperform the results reported by Supe-RL and SUNRISE, so the comparison does not seem fair to me.\n- Some of the reported results for the baselines (PPO, DDPG, TD3) seem lower with respect to the best results in the literature at 2M time steps (see e.g. https://spinningup.openai.com/en/latest/spinningup/bench.html for DDPG and TD3). Why is that the case? What implementation was used for the baselines and how were the hyperparameters chosen? Did the authors tune hyperparameters both for the baseline and their method?\n- The authors claim that their method improves value prediction. However, there is no analysis showing value predictions. The value function loss is reported, but a low value function loss does not necessarily imply that the prediction of the value function will be accurate, especially when using off-policy methods with Temporal Difference Learning. Can the authors clarify this or include an experiment that analyzes value prediction?\n- The authors claim that their method has reduced variance over the baseline. However, the learning curves in Figure 6 show sometimes that the proposed method has a larger variance. Moreover, while in Ant and HalfCheetah environments VFS-PPO has a larger variance than PPO, Table 1 reports a lower variance for VFS-PPO. Can the authors clarify this?\n    \n- Figure 8 does not show significant benefits in cosine similarity between the gradient of the proposed method and the true gradient, as the curves for the proposed methods and the baselines almost completely overlap. \n    \nOther questions:\n\nCan the authors give some intuition on what would be the issues in scaling up this method for more complex domains? Could the method be easily applied for e.g. A2C and applied to Atari games?\n\nAre the values in Figure 2 the results of 100 perturbations and then predicting one episode for each, or are they the results of 2 perturbations and then 100 predictions for each perturbation? Since these 100 values have no clear structure, instead of plotting them in a table, the authors might want to consider plotting the distributions of predictions induced by different values of sigma\n\nWhat is the value of $e_s$, the periodicity hyperparameter?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iXePfp14Ob", "reviewer": "Reviewer_ur4C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper takes a careful look into how to make value function predictions more accurate, which is often neglected in modern policy gradient methods. The idea is simple, to maintain a population of value functions, and with perturbed value networks to minimize the regression loss. There are a bunch of nice properties of the proposed method including good empirical performance, low additional computation cost.", "summary_of_the_review": "I'd vote for acceptance of the paper since its simplicity and nice empirical performance.", "strength_and_weaknesses": "Strength\n+ Very clear description of the proposed idea, including an illustrative overview, algorithm description, etc;\n+ Strong performance when combined with three different base RL algorithms, including PPO, DDPG, and TD3. In all tested domains, the proposed method VFS outperforms the base algorithms w/o VFS significantly;\n+ The VFS also achieves a boost in sample efficiency when applied to the base RL algorithms;\n+ The performance comparison remains strong when compared against the policy search method Supe-RL and ensemble method SUNRISE\n\nWeaknesses\n- In Algorithm 1, splitting the value functions pool into **half** and add $\\mathcal{G}_\\text{min}$ and  $\\mathcal{G}_\\text{max}$ seem a bit arbitrary, could you please provide more explanation on this?\n- From Figure 7, the population size of value functions seems to have a diminishing return effect. I would like to understand whether the best population size is consistent across tasks. I would suspect it might need more value functions in more complex tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Oo7rG_Yjn0n", "reviewer": "Reviewer_7FyJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies value function approximation in reinforcement learning and proposes to include gradient-free evolutionary training steps when fitting the value function to data from the replay buffer. In particular, after a certain number of training steps the critics weights are perturbed with zero-mean Gaussian noise with two different magnitudes to explore different critic approximations both locally and globally. It then experimentally evaluates several on- and off-policy actor-critic algorithms on the mujoco benchmarks both with and without the additional gradient-free training step for the critic. The experiments show that this can help improve performance.", "summary_of_the_review": "Overall I think the paper is interesting but, given the similarity to existing genetic RL algorithms, in my opinion the paper does provide sufficient experimental evidence to clear the bar for acceptance at ICLR. In particular, the paper should consider adding an ablation study to confirm how local v.s. global perturbations affect performance, use state-of-the-art implementations for the comparison, and evaluate how more critic gradient steps would perform relative to the model-free training step.", "strength_and_weaknesses": "Overall, the question of how to better approximate values is interesting to the community. The paper aims to tackle this in a minimally invasive way by adapting only the critic's optimization step, which means it is easy to implement and does not require additional data from the environment. The proposal to both locally and globally adapt the weights as part of the evolutionary adaptation of the weights is interesting, and in particular the global step could be related to capacity loss in critics [1] (this review has no affiliation to the authors). The paper is also clearly written and easy to follow.\n\n\n\nThe papers' main weakness is the experimental evaluation. In particular, no evidence is given that the combination of local and global updates actually improves performance or whether one of them would be sufficient on their own. Since this is framed as one of the main contributions, this makes the paper difficult to evaluate. In addition, some benchmarks seem to be wrongly implemented (especially PPO), since they do not achieve the performance available in open-source RL frameworks (see, for example, [2]). It is also not clear how the methods would compare on an equal computational budget, i.e. if the model-free budget was used to perform more gradient steps of the critic.\n\n\n\n[1] \"Understanding and Preventing Capacity Loss in Reinforcement Learning\", ICLR 2022\n\n[2] https://tianshou.readthedocs.io/en/master/tutorials/benchmark.html", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4-E0QdxBEU", "reviewer": "Reviewer_KmE6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to enhance the sample efficiency of Deep PG algorithms by improving the learning of value functions. To this end, this paper proposes Value Function Search (VFS), which instantiates a population of perturbed critics and uses two-scale perturbation noise to make good use of two advantages: (1) noise with small variance is to find a better set of parameters within a local search. (2) noise with big variance is to escape from local optima by exploring. The experiments show that the proposed framework outperforms the baselines of VFS-based algorithms. Ablation studies show how VFS affects the training phase and how VFS can enhance the Deep PG primitives.", "summary_of_the_review": "This paper proposes a gradient-free population-based method to update the value network. The experiments show that the proposed framework outperforms the baseline methods. \nThat said, here are the two major concerns: (1) If a nearly true value function (for calculating MSE loss and finding the best fit within the perturbed critics) can be obtained at a low cost during the training process, then it appears sufficient to directly use the true value function to assist Deep PG in the policy updates. Additional ablation studies are needed to confirm the benefits of value function search. (2) While this paper's method of perturbing the value network using two-scale noise terms is interesting, the overall design is quite similar to other population-based methods. The idea of VFS does not appear very novel.\n", "strength_and_weaknesses": "**Strengths**\n\n- The motivation for enhancing the sample efficiency by improving the estimation of value function is convincing: (i) ensemble methods require more additional inference time and (ii) prior works require multiple training models or additional environment interactions.\n- This paper proposes a gradient-free population-based critic learning approach which does not require complex gradient computation and serves as a computationally inexpensive approach to enhance the accuracy of value estimates.\n- An empirical evaluation and an ablation study that indicates the effectiveness of the proposed VFS in critic training: (i) The value function loss and similarity of gradient estimates in Figure 4 shows the influence of VFS. (ii) The experimental results in Table 1 show that the VFS-based Deep PG algorithms outperforms the primitive algorithm in terms of average return.\n\n**Weaknesses**\n\nMy concerns are mainly two-fold: \n- A (nearly) true V^{\\pi} needed in the loss function: To enable gradient-free critic updates, VFS uses the simple MSE as the loss function (cf. Eq (7)), which measures the difference between the true V^{\\pi} and the perturbed critics. This is reasonable if a nearly true V^{\\pi} is available. While such V^{\\pi} can be obtained by Monte Carlo methods with a large number of trajectories, this could lead to sample inefficiency, which contradicts one of the motivations of this paper. On the other hand, if such V^{\\pi} is inaccurate, the MSE loss could lead to significant bias in the learned perturbed critics. As the paper does not clearly specify how VFS obtains V^{\\pi} in its implementation, it is not immediately clear how to interpret the empirical performance improvement provided in Table 1 and the comparison of MSE in Figure 4 (since a lower MSE does not necessarily indicate a more accurate critic if V^{\\pi} is inaccurate).  However, this is typically not the case as one would not have a sufficiently accurate estimate of V^{\\pi} to begin with (otherwise critic learning would not be needed). \n- Given the several existing prior works on population-based methods for deep PG (e.g., [Khadka & Tumer, 2018; Marchesini & Farinelli, 2020; Marchesini et al., 2021; Sigaud, 2022] and references therein), the idea of using a population-based approach in VFS is not very novel.\n\n**Additional questions**\n- The result in Figure 2 appears quite straightforward and does not seem to provide much insight into the performance of VFS. By contrast, experiments like Figure 7 and Figure 6 are actually more informative and would be helpful in the main text. Is there any specific argument that the author(s) would like to make through Figure 2?\n- How to calculate $V_\\pi(s)$ in Eq (7)? How to strike a balance between getting an accurate $V_\\pi(s)$ and sampling efficiency? Do other baselines also use true value functions to assist the update?\n- In Algorithm 1, VFS utilizes a two-scale approach to perturb the parameter $\\phi$ which is the best fit of the true value function. What would happen if one either uses only noise with small variance or only noise with high variance? Moreover, why is it needed to combine both types of noise? More ablation studies are surely needed.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster