Learning Value Functions in Deep Policy Gradients using Residual Variance
Authors: Yannis Flet-Berliac
reda ouhamma
odalric-ambrym maillard
Philippe Preux
Venue: ICLR 2021 Poster (2021)

Abstract: Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic uses a new state-value (resp. state-action-value) function approximation that learns the value of the states (resp. state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights.

Published as a conference paper at ICLR 2021
LEARNING VALUE FUNCTIONS IN DEEP POLICY GRA- DIENTS USING RESIDUAL VARIANCE
Yannis Flet-Berliac Inria, Scool team
Reda Ouhamma"
Inria, Scool team
Univ. Lille, CRISIAL,, CNRS yannis. flet-berriaceenriieee
Univ. Lille, CRISIAL, CNRS
reda. ouhamma @ inria a. fr
Odalric-Ambrym Maillard Inria, Scool team
Philippe Preux
Inria. Scool team
Univ. Lille, CRIStAL, CNRS
ABSTRACT
Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic value) function approximation that learns the value of the states (resp. state-action pairs) relative their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights.
a
state-value (resp. Mttt-acction-
INTRODUCTION
Model-free deep reinforcement learning (RL) has been successfully used in a wide range of prob- lem domains, ranging from teaching computers to control robots to playing sophisticated strategy games (Silver et al., 2014; Schulman et al., 2016; Lillicrap et al., 2016; Mnih et al., 2016). State- of-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework (Sutton et al., 2000; Schulman al., 2017; Haarnoja et al., 2018). While such methods demonstrate great performance in continuous control tasks, several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains.
For instance, research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions, then propose critic formulation that allows for more stable policy gradient. However, new studies (Tucker al., 2018 8; Ilyas et al., 2020) indicate that state-of-the-ar policy gradient methods (Schulman et al., 2015; 2017) fail to fit the true value function and that recently proposed detepeetioondeeeedeent baselines (Gu et al., 2016; Liu et al., 2018; Wu et al., 2018) do not reduce gradient variance more than state-dependent ones.
These findings leave the reader skeptical about actor-critic algorithms, suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning. Consequently, attempting to find better baseline questionable, as critics would typically fail fit it (Ilyas et al.. 2020). In Tucker et al. (2018), the authors argue that "much larger gains could be achieved by instead improving the accuracy of the value function".. Following this line of thought, we are interested in ways to better approximate the value function. One approach addressing this issue is to put more focus on relative state-action values, an idea introduced the literature on advantage reinforcement
"qqual contribution.

---

Published as a conference paper at ICLR 2021
learning (Harmon & Baird IIII followed by works on dueling (Wang et al., 2016) neural networks. More recent work (Lin & Zhou, 2020) also suggests that considering the relative action values, or more precisely the ranking of actions in state leads to better policies. The main argument behind this intuition that it suffices to identify the optimal actions to solve a task. We extend this principle of relative action value with respect the mean value to cover both state and aate- functions with new objective for the critic: minimizing the variance of residual errors.
In essence, this modified loss function puts more focus on the values of states (resp. state-actions) relative to their mean value rather than their absolute values, with the intuition that solving a task corresponds to identifying the optimal action(s) rather than estimating the exact value of each state. In summary, this paper:
Introduces Actor with Variance Estimated Critic (AVEC), an actor-critic method providing a new training objective for the critic based on the residual variance.
Provides evidence for the improvement of the value function approximation well as theoretical consistency of the modified gradient estimator.
Demonstrates experimentally that AVEC, when coupled with state-of-the-art policy gradient algorithms, yields a significant performance boost on set of challenging tasks, including enviromments with sparse rewards.
Provides empirical evidence supporting a better fit of the true value function and substantial stabilization of the gradient.
RELATED WORK
Our approach builds on three lines of research, of which we give a quick overview: policy gradient algorithms, regularization in policy gradient methods, and exploration in RI.
Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator. This was originally formulated as the REINFORCE algorithm (Williams, 1992). Kakade & Langford (2002) later created conservative policy iteration and provided lower bounds for the minimum objective improvement. Peters al. (2010) replaced regularization by trust region constraint to stabilize training. In addition, extensive research investigated methods to improve the stability of gradient updates, and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories, the corresponding variance can be extremely high. To improve stability, Weaver Tao (2001) ) show that subtracting a baseline (Williams, 1992) from the value function in the policy gradient can be very beneficial in reducing variance without damaging the bias. However, in practice, these modifications on the actor-critic framework usually result in improved performance without significant variance reduction (Tucker et al., 2018; Ilyas et al., 2020). Currently, one of the most dominant on on-policy methods are proximal policy optimization (PPO) (Schulman et al., 2017) and trust region policy optimization (TRPO) (Schulman et al., 2015), both of which require new samples to be collected for each gradient step. Another direction of research that overcomes this limitation off-policy algorithms, which therefore benefit from all sample transitions; soft actor-critic (SAC) (Haarnoja et al., 2018) is one such approach achieving state-of-ther tithh-arrt art performance.
Several works also investigate regularization effects on the policy gradient (Jaderberg et al., 2016; Namkoong & Duchi, 2017; Kartal et al., 2019; Flet-Berliac & Preux, 2019; 2020); it is often used to shift the bias-variance trade-off towards reducing the variance while introducing small bias. In RL, regularization is often used to encourage exploration and takes the form of an entropy term (Williams & Peng, 1991: Schulman al., 2017). Moreover, while regularization in machine learning generally consists in smoothing over the observation space, in the RL setting. Thodoroff et al. (2018) 8) show that it possible to smooth over the temporal dimension as well. Furthermore, Zhao et al. (2016) analyze the effects of regularization using the variance of the policy gradient (the idea is reminiscent of SVRG descent (Johnson & Zhang, 2013)) which proves to provide more consistent policy improvements at the expense of reduced performance. In contrast, we will see later, AVEC does not change the policy network optimization procedure nor involves any additional computational cost.
Exploration has been studied under different angles in RL, one common strategy is e-greedy, where the agent explores with probability e by taking a random action. This method, just like entropy regularization, enforces uniform exploration and has achieved recent success in game playing en-
2

---

Published as a conference paper at ICLR 2021
vironments (Mnih et al., 2013; Van Hasselt et al., 2015; Mnih et al., 2016). On the other hand, for most policy-based RL, exploration is a natural component of any algorithm following a stochastic policy, choosing sub-optimal actions with non-zero probability. Furthermore, policy gradient lit- erature contains exploration methods based on uncertainty estimates of values (Kaelbling, 1993; Tokie, 2010), and algorithms which provide intrinsic exploration or curiosity bonus encourage exploration (Schmidhuber, 2006; Bellemare et al., 2016; Flet-Berliad et al., 2021).
While existing research may share some motivations with our method, no previous work in RL applies the variance of residual errors as an objective loss function. In the context of linear regression, Brown (1947) considers median-unbiased estimator minimizing the risk with respect to the absolute- deviation loss function (Pham- Gia & Hung. 2001) (similar in spirit to the variance of residual errors), their motivation is nonetheless different to ours. Indeed, they seek be robust to outliers whereas, when considering noiseless RI problems, one usually seeks to capture those (sometimes rare) signals corresponding to the rewards.
PRELIMINARIES
3.1 BACKGROUND AND NOTATIONS
We consider an infinite-horizon Markov Decision Problem (MDP) with continuous states E S continuous actions E AA, transition distribution 8t41 P(st, at) and reward function Tt R(s,, az). Let (s)) denote a stochastic policy with parameter 0, we restrict policies to being the following, π and Tg denote the same object. The agent repeatedly
Gaussian distributions. interacts with the environment by sampling action Oş ~ x(([s)), receives reward ra and transitions to a new state 8,41- The objective maximize the expected sum of discounted rewards:
20
J(T)
L TIO
Σ
En'r(se ()t.
where y E [0, 1) is discount factor (Puterman, 1994), and T (sg, ao, ro, $1 01,r1,,-11 ..,...) trajectory sampled from the environment using policy We denote the value of a state in the MDP framework while following policy π by V*(s) ف E,sa ">=0o'' (st, az) |su s] and the value of state-action pair of performing action in state and then following policy by Q* a) & Eysr 1t= 7'r (St, 4z) 180 = s, Qg = a). Finally, , the advantage function which quantifies how an action a is better than the average action in state s is denoted (" & Q" (8, V*(().
3.2 CRITICS IN DEEP POLICY GRADIENTS
In this section, we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient. Without loss of generality, we consider the algorithms that approximate the state-value function V. The analysis holds for algorithms that approximate the state-action-value function Q. Let fo S + be an estimator of V with 6 its parameter. fo is traditionally learned through minimizing the mean squared error (MSE) against V" At iteration k; the critic minimizes:
LAc E, [ff(()) Vistos (s))
where the states are collected under policy maa and T (s) is an empirical estimate of V (see Section 4.3 for details). Similarly, using f S A + R instead, one can fit an empirical target Q".
4 METHOD: ACTOR WITH VARIANCE ESTIMATED CRITIC
In this section, we introduce AVEC and discuss its correctness, motivations and implementation.
4.1 DEFINING AN ALTERNATIVE CRITIC
Recent work (Ilyas et al., 2020) empirically demonstrates that while the value network succeeds in the supervised learning task of fitting 5"" (resp. Q*), it does not fit 4 (resp. Q*). We address this
3

---

Published as a conference paper at ICLR 2021
deficiency in the estimation of the critic by introducing alternative value network loss. Following empirical evidence indicating that the problem is the approximation error and not the estimator per se, AVEC adopts loss that can provide a better approximation error, and yields better estimators of the value function (as will be shown in Section 5.3). At update k:
LAVEC = 1Ss [(rACC) -
( im (s)) - Es [Fo(s) - VT (+) (c))
(3)
with states 8 collected using Te,- Note that the gradient flows in fo twice using Eq. Then, we define our bias-corrected estimator: 96 : - R such that go(s) = (s) [V (s) - fo(
Analogously to Eq. 3, we define an alternative critic for the estimation of Q" by replacing V by Q and (s) by fs
Proposition (AVEC Policy Gradient). If fo S x A + R satisfies the parameterization assump- tion (Sutton al., 2000) then 96 provides an unbiased policy gradient:
VaJ (me) (s, [V (o))(T)(s,,)))9
Proof. See Appendix A. This result also holds for the estimation of V with So S - R.
4.2 BUILDING MOTIVATION
Here, we present the intuition behind using AVEC for ccior-critic algorithms. Tucker al. (2018) and lyas et al. (2020) indicate that the approximation error -|V V* || problematic, suggesting that the variance of the empirical targets V (se) ) is high. Using LAVEO, our approach reduces the variance term of the MSE (or distance V*) but mechanistically also increases the bias. Our intuition is that since the bias is already quite substantial (Ilyas et al., 2020), it may be possible to reduce the variance enough so that even though the bias increases, the total MSE reduces.
State-value function estimation. In this case, optimizing the critic with LAVEC can be interpreted as fitting 1/ (s) (s) - [ (V"('') using the MSE. We show that the targets fTrH are better estimations of V* (s) (s) E,, (V*{)) than 1- are of V To illustrate this, consider T independent random variables (Xi) We denote X = X;
Σラ= X, and V(X) the variance of X. Then, (X!) V (X,) (X,) + to ΣJ= V (X₂) and V(X;) V(X,) as long Vi + + V(X,) 2V().). or more generally when state-values are not strongly negatively correlated! and not very discordant. This entails that fTT has more compact span, and is consequently easier to fit. This analysis shows that the variance term the MSE reduced compared to traditional actor-critic algorithms, but does not guarantee it counterbalances the bias increase. Nevertheless, in practice, the bias is so high that the difference due to learning with AVEC is only marginal and the total MSE decreases. We empirically demonstrate this claim in Section 5.3.
State-action-value function estimation. In this case, Eq. 3 translates into replacing (T((s) by (s, and (s) by fa(s, a) and the rationale for optimizing the residual variance of the value fune- tion instead of the full MSE becomes more straight- forward: the practical use of the Q-function is to disentangle the relative values of actions for each state (Sutton et al., 2000). AVEC's effect relative values is illustrated in a didactic regression with one variable example in Fig. where grey markers ob- servations and the blue line is our current estimation. Minimizing the MSE, the line is expected to move towards the orange one in order to reduce errors uniformly. Minimizing the residual variance, it
MIE
Initial
Var
Figure 1: Comparison of simple models de- rived when Lave: is used instead of the MSE.
'Greensmith al. (2004) analyze the dependent case: in general, weakly dependent variables tend to concentrate than independent ones.
4

---

Published as a conference paper at ICLR 2021
is expected to move near the red one. In fact, LAvec tends to further penalize observations that far away from the mean, implying that AVEC allows a better recovery of the "shape" of the target near extrema. In particular, we in the figure that the maximum and minimum observation values are quickly identified. Would the approximators be linear and the target state-values independent, the two losses become equivalent since ordinary least squares would provide minimum-variance mean-unbiased estimation.
should be noted that, as in all the works related to ours, we consider noiseless tasks, i.e. the transition matrix is deterministic. As such, there are no outliers and extreme state-action values correspond to learning signals. In this context, high estimation errors indicate where (in the state or action-state space) the training of the value function should be improved.
4.3 IMPLEMENTATION
We apply this new formulation to three of the most dominant deep policy gradient methods to study whether it results in a better estimation of the value function. A better estimation of the value function implies better policy improvements. We now describe how AVEC incorporates its residual variance objective into the critics of PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015) and SAC (Haarnoja et al., 2018). Let B be a batch of transitions. In PPO and TRPO, AVEC modifies the learning of V (line 12 of Algorithm 1) using:
C A ( E5 (fo (8) 1-m (s)) -ESB (8) V' (8)
then Vo = fo(s) Ee-ss(VTSs) fs]]. where V= (st) = form + A. such that fow (st) are the estimates given by the last value function and A, is the advantage of the policy, i.e. the returns minus the expected values (A, is often estimated using generalized advantage estimation (Schulman al., 2016). In SAC, AVEC modifies the objective function of (Qu1)==122 (line 13 of Algorithm 2 in Appendix C) using:
Civec (0+) E( (fo:(s,a)- (s, ((s,))=B (o) (sss [
then Qs. = So, (8, a) + EGea) (Q () - (s, a)], where (s, a) is estimated using temporal difference (see Haarnoja et al. (2018)): (st, = r(s:, a,) + -Bo4i (Vgn><<(((((s)))) with & the value function parameter (see Algorithm 2). The reader may have noticed that Livec and Lives slightly differ from Eq. 3. The residual variance of the value function (LAvEc) is not tractable since priori state-values are dependent and their joint law is unknown. Consequently, in practice, the empirical variance proxy assuming independence (cf. Appendix D). Greensmith et al. (2004) provide some support for this approximation by showing that weakly dependent variables tend to concentrate more than independent ones. Finally, notice that AVEC does not modify any other part of the considered algorithms whatsoever, which makes its implementation straightforward and keeps the same computational complexity.
5 EXPERIMENTAL STUDY
In this section, we conduct experiments along four orthogonal directions. (a) We validate the superiority of AVEC compared the traditional actor-critic training. (b) We evaluate AVEC in environments with sparse rewards.
(c) We clarify the practical implications of using AVEC by examining the bias in both the empirical and true value function estimations as well as the variance in the empirical gradient. (d) We provide an ablation analysis and study the bias-variance trade-off in the critic by considering two continuous control tasks.
We point out that comparison to variance- variance-reduction reduction methods is not considered in this paper: Tucker et al. (2018) demonstrated that their implementations diverge from the unbiased methods presented in the respective papers and unveiled that not only do they fail to reduce the variance of the gradient, but that their unbiased versions do not improve performance either. Note that in all experiments we choose the hyperparameters providing the best performance for the considered methods which can only penalyze AVEC (ef. Appendix E). In all the figures hereafter (except Fig. 3c and 3d), lines are average performances and shaded areas represent one standard deviation.
5

---

Published as a conference paper at ICLR 2021
Task
SAC
AVEC-SAC
PPO
AVEC-PPO
Ant
3084
3650 127
972
1202 - 148(444%)
AntBullet
1193
2252 82 82 (+89%)
1174
2216 - 99 89%%
HaltCheetah
10028
(+10%)
1068
1403
37 (+31%)
HallCheetaBBullet
1255
1331 15331118(4633) (+6%))
1329
2223
(677)
Humanoid 4084
4472 + 424 424
391
415 + 4.6 (.6(46%))
Reacher
-6.0
-5.0 0.1 (+20%)
-7.4
-5.9
0.3 (+25%)
Walker2d
3452
4334 1 128 (+ 26%)
2193
2923 1 151 +33%)
Table 1: Average total reward of the last 100 episodes over 6 runs of 10th timesteps. Comparative evaluation of AVEC with SAC and PPO. + corresponds to single standard deviation over trials and (%%) is the change in performance due to AVEC.
5.1 CONTINUOUS CONTROL
For ease of comparison with other methods, we evaluate AVEC on the MuJoCo (Todorov et al., 2012) and the PyBullet (Coumans & Bai, 2016) continuous control benchmarks (see Appendix for details) using OpenAl Gym (Brockman et al., 2016). Note that the PyBullet versions of the lo- comotion tasks are harder than the MuJoCo equiv- alents². We choose a representative set of tasks for the experimental evaluation; their action and observation space dimensions are reported in Ap- pendix H. We assess the benefits of AVEC when coupled with the most prominent policy gradi-
Algorithm 1 AVEC coupled with PPO or TRPO,
JALGO denotes the policy loss of either algorithm
(described Schulman et al. (2017; 2015)).
Input parameters: X 2 0, Av 2 0
2: Initialize policy parameter 0 and value func- tion parameter ạ
3: for each update step do
4: batch +
5:
for each environment step do
6:
Gg ~ To(
7:
S.+1 ~ P (st,, az) B ++BUiGE ae, Tr. st+1)}
ent algorithms, currently state-of-the-art methods:
8:
PPO (Schulman et al., 2017) and TRPO (Schul- man et al., 2015), both on-policy methods, and SAC (Haarnoja et al., 2018), off-policy maxi- mum entropy deep RL algorithm. We provide the list of hyperparameters and further implementa- tion details in Appendix D and E.
end for
10:
for each gradient step do
11:
f t 0 - A JALGO (m))
12:
Φ t- Av V VoLAVEC (Ø)
13:
end for
14:
end for
Table 1 reports the results while Fig. 2 and 8 show the total average return for SAC and PPO. TRPO results are provided in Appendix F for readability. When coupled with SAC and PPO, AVEC brings very significant improvement (on average +26 % for SAC and +39 % for PPO) in the performance of the policy gradient algorithms, improvement which consistent across tasks. As for TRPO, while the improvement in performance is less striking, AVEC still manages to be more efficient in terms of sampling in all tasks. Overall, AVEC improves TRPO, PPO and SAC in terms performance and efficiency. This does not imply that our method would also improve other policy gradient methods that use the traditional actor-critic framework, but since we evaluate our method coupled with three of the best performing on- and off-policy algorithms, believe that these experiments are sufficient prove the relevance of AVEC. Furthermore. in our experiments we do not seek the best hyperparameters for the AVEC variants, we simply adopt the parameters allowing to optimally reproduce the baselines. Alternatively, if one seeks to evaluate AVEC independently of a considered baseline, further hyperparameter tuning should produce better results. Notice that since no additional calculations needed in AVEC's implementation, computational complexity remains unchanged.
5.2 SPARSE REWARD SIGNALS
Domains with sparse rewards are challenging to solve with uniform exploration as agents receive no feedback on their actions before starting to collect rewards. In such conditions AVEC performs better, suggesting that the shape of the value function is better approximated, encouraging exploration.
Bullet Physics SDK GitHub Issue.

---

Published as a conference paper at ICLR 2021
Ant
HallCheetah
Humanakd
200
200090
40000
80008
59000
100000
200000
40000
60000
08008
100000
0.2
HalfCheotanBBulet
Reacher
200
1500
190
08
-50
Figure 2: Comparative evaluation (6 seeds) of AVEC with SAC and PPO on PyBullet ("TaskBullet") and MuloCo ("Task") tasks. X-axis: number of timesteps. Y-axis: average total reward.
The relative value estimate of unseen state more accurate: in Section 4.2, AVEC identifies extreme state-values (e.g., non-zero rewards in tasks with sparse rewards) faster. In Fig. 3a and 3b, we report the performance of AVEC in the Acrobot and MountainCar environments: both have sparse rewards. AVEC enhances TRPO and PPO in both experiments. When PPO and AVEC-PPO both reach the best possible performance, AVEC-PPO exhibits better sample efficiency. Fig. and 3d illustrate how the agent improves its exploration strategy in MountainCar: while the PPO agent remains stuck at the bottom of the hill (red), the graph suggest that AVEC-PPO learns the difficult locomotion principles in the absence of rewards and visits much larger part of the state space (green).
This improved performance in sparse environments can be explained by the fact that AVEC is able to pick up experienced positive reward more easily. Moreover, the reconstructed shape of the value function is more accurate around such rewarding states, which pushes the agent to explore further around experienced states with high values.
MountairCar
2000
4000
1000 20000
50000
(a)
(c)
(d)
Figure 3: (a,b): Comparative evaluation (6 seeds) of AVEC in sparse reward tasks. X-axis: number of timesteps. Y-axis: average total reward. (c,d): Respectively state visitation frequency and phase portrait of visited states of AVEC-TRPO (green) and TRPO (red) in MountainCar.
5.3 ANALYSIS OF THE VARIANCE ESTIMATED CRITIC
In order to further validate AVEC, we evaluate the performance of the value network in more detail: we examine (a) the estimation error (distance to the empirical target), (b) the approximation error (distance to the true target) and (c) the empirical variance of the gradient. (a.b) should be put into perspective with the conclusions of Ilyas et al. (2020) where it is found that the critic only fits the empirical value function but not the true one. (c) should be placed in light of Tucker et al. (2018) highlighting a failure of recently proposed state-action-dependent basclines to reduce the variance.
Learning the Empirical Target. In Fig. 4, we report the quality of fit (MSE) of the empiri- cal target 8 in the methods PPO and AVEC-PPO in the AntBullet and HalfCheetahBullet tasks.
7

---

Published as a conference paper at ICLR 2021
We observe that PPO better fits the empirical target than when equipped with AVEC, which is to be expected since vanilla PPO optimizes the MSE directly. This result put aside the re- markable improvement in the performance of AVEC-PPO (Fig. 2) suggests that AVEC might be a better estimator of the true value function. We examine this claim below because if true, it
Hat Cww.aaaaaaaaao Golled
Figure L.2 distance to V would indicate that it is indeed possible to simultaneously improve the performance of the agents and the stability of the method.
Learning the True Target. A fundamental premise of policy gradient methods is that op- timizing the objective based on an empirical estimation of the value function leads to better policy. Which is why we investigate the qual- ity of fit of the true target. To approximate the true value function, we fit the returns sampled from the current policy using a large number of transitions (3 10a). Fig. 5 shows that 9e is far closer to the true value function half of the time (horizon is 109) than the estimator obtained with MSE, then as close to it. Comparing Fig. 5 with Fig. 4, we see that the distance to the true target is close the estimation error for AVEC-PPO, while for PPO, it is at least two orders of magnitude higher at all times. We further investigate these results in Fig. 9 in Appendix B.2 where we study the variation of the squared bias and variance components of the MSE to the true target (MSE Var Bias?). We find, as expected, that using AVEC reduces the variance term significantly while slightly increasing the bias term, which Fig. 5 confirms is negligible since the total MSE is sub- stantially reduced (AVEC) V=lla S (PPO) estimator in PPO. For completeness, we also analyze the distance to the true target for the Q-function estimator in SAC and AAC in AntBullet and BullCheetabBullet in Appendix B.3, with similar results and interpretation. We conclude that AVEC improves the value function approximation and we expect that the gradient more stable.
Hal HalhheeteW..rc
200
200
143
Figure 5: L2 distance to V X-axis: we PPO and AVEC-PPO and At E {1, 2, 10 we stop training, use the current policy to collect 3- transitions and estimate V*.
(* where (PPO) is the value function
Empirical Variance Reduction. We choose to study the gradient variance using the average pairwise cosine similarity metric it allows comparison with Ilyas et al. (2020), with which we share the same experimental setup and scales. Fig. 6 shows that AVEC yields higher average (10 batches per iteration) pairwise cosine sim- ilarity, which means closer batch-estimates of the gradient and, in turn, indicates smaller gradient variance. Further analysis with additional tasks is included in Appendix B.4. The variance reduction effect observed in several environments suggests that AVEC is the first method since the introduction of the value function baseline to further reduce the variance of the gradient and improve performance.
Artiala
Figure 6: Average gradient cosine-similarie
5.4 ABLATION STUDY
In this section, we examine how changing the relative importance of the bias and the residual variance in the loss of the value network affects learning. For this study, we choose difficult tasks of PyBullet and use PPO because it is more efficiintt than TRPO and requires less computations than SAC. For an estimator In of (J.).ef......), we write Bias = 1 - (0: yi) and Var = - (0: ys Bias) Consequently: MSE = Var + Bias We denote L. Var aBias?. with a E R. In Fig. 7, Bias-a means that we use Ca and Var-a means that we use L1. We observe that while no consistent order the choices of is identified, AVEC seems to outperform all other weightings. Note that, for readability purposes, the graphs have been split and the curves of AVEC-PPO and PPO are the same in Fig. 7a and 7c, and in Fig. 7b and 7d. A more extensive hyperparameter
8

---

Published as a conference paper at ICLR 2021
ArBuitt
HolCheett Butlet
2000
4005
10000
10000
2000
0000
0000
2005
4000
1000
(a)
Figure 7: Sensitivity (6 seeds) of AVEC-PPO with respect to (a,b): the bias; (c,d): the variance. X-axis: number of timesteps. Y-axis: average total reward.
study with more a values might provide even higher performances, nevertheless we believe that the stability of algorithm is crucial for reliable performance. As such, the tuning of hyperparameters to achieve good results should remain mild.
DISCUSSION
In this work, we introduce new training objective for the critic in actor-critic algorithms to better approximate the true value function. In addition to being well-motivated by recent studies on the behaviour of deep policy gradient algorithms, we demonstrate that this modification both theoretically sound and intuitively supported by the need to improve the approximation error of the critic. The application of Actor with Variance Estimated Critic (AVEC) to state-of-the-art policy gradient methods produces considerable gains in performance (on average +26% for SAC and +39% for PPO) over the standard actor-critic training, without any additional hyperparameter tuning.
First, for SAC-like algorithms where the critie learns caate-action-valll function, results strongly suggest that state-actions with extreme values are identified more quickly. Second, for PPO-like methods where the critic learns the state-values, and empirically demonstrate that this due to a better approximation of the state-values. In sparse reward environments, the theoretical intuition behind variance estimated critic is more explicit and is also supported by empirical evidence. In addition to corroborating the results in Ilyas et al. (2020) proving that the value estimator fails fit V", we propose method that succeeds in improving both the sample complexity and the stability of prominent actor-critic algorithms. Furthermore, AVEC benefits from its simplicity of implementation since no further assumptions are required (such as horizon awareness Tucker et al. (2018 8) to remedy the deficiency of existing variance-reduction methods) and the modification of current algorithms represents only few lines of code.
show that the variance of the gradient is reduced
In this paper, we have demonstrated the benefits of more thorough analysis of the critic objective in policy gradient methods. Despite our strongly favourable results, we do not claim that the residual variance is the optimal loss for the state-value or the state-action-value functions, and we note that the design of comparably superior estimators for critics in deep policy gradient methods merits further study. In future work, further analysis of the bias-variance trade-off and extension of the results to stochastic environments anticipated; we consider the problem of noise separation in the latter, as this is the first obstacle to accessing the variance and distinguishing extreme values from outliers.
REFERENCES
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprini
100.00156000..0. 2016.
George W. Brown. On small-sample estimation. Annals of Mathematical Statistics, 18(4):582-585 12 1947.
Erwin Coumans and Yunfci Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016.

---

Published as a conference paper at ICLR 2021
Yannis Flet-Berliae and Philippe Preux. Merl: Multi-head reinforcement learning. In Deep Rein- forcement Learning Workshop, NewrIPS, 2019.
Yannis Flet-Berliac and Philippe Preux. Only relevant information matters: Filtering out noisy samples to boost rl. In International Joint Conference Artificial Intelligence, pp- 2011-2711111 2020.
Yannis fiet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. In International Conference Learning Representations, 2021.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5: 5:1471-1530, 2004.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference Machine Learning, pp.
2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1856-1865, 2018.
Mance E Harmon and Leemon C Baird III. Multi-player residual advantage learning with general function approximation.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Conference on Learning Representations, 2020.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint ar 1.05397, 2016.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pp. 315-323, 2013.
Leslic Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference Machine Learning, pp- 267-274, 2002.
Bilal Kartal, Pablo Hernandez-Leal, and Matthew E Taylor. Terminal prediction an auxiliary task for deep reinforcement learning. In AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 38 44, 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference Learning Representations, 2016.
Kaixiang Lin and Jiayu Zhou. Ranking policy gradient. In International Conference on Learning Representations, 2020.
Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng. and Qiang Liu. Action-dependent control variates for policy optimization via stein identity. In International Conference Learning Representations, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1r255213/ 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928-1937, 2016.
H. Namkoong and C. Duchi. Striance-bsed regularization with convex objectives. In Advances in Neural Information Processing Systems, pp. 2971-2980, 2017.
10

---

Published as a conference paper at ICLR 2021
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI Conference on Artificial Intelligence, 2010.
Pham-Gin Siin-Giauu and TL Hung. The mean and median absolute deviations. Mathematical and Computer Modelling. 34(7-8):921-9) 2001.
M. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994.
Jürgen Schmidhuber: Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connection Science, 18(2):133-877, 2006.
Schulman, S. Levine, Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1928-1937, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: N.XYv:7777006477 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference Machine Learning, 2014.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 2000.
P. Thodoroff, A. Durand, J. Pineau, and D. Precup. Temporal regularization for markov decision process. In Advances in Neural Information Processing Systems, 2018.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012.
Michel Tokic. Adaptive e-greedy exploration in reinforcement learning based value differences. In Annual Conference on Artificial Intelligence, pp. 203-210. Springer, 2010.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. In International Conference on Machine Learning, pp. 5015-5024, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning, arXiv preprint 1rXiv:1509.0644 2015.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning, pp. 1995-2003, 2016
L. Weaver and N. Tao. The optimal reward baseline for gradient based reinforcement learning. In Advances in Neural Information Processing Systems, 2001.
R.J. Williams. Simple statistical learning. Machine Learning, 8(3-4):229-25 1992.
algorithms for connectionist reinforcement
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. In International Conference Learning Representations, 2018.
Tingting Zhao, Gang Niu, Ning Xie, Jucheng Yang, and Masashi Sugiyama. Regularized policy gradients: direct variance reduction in policy gradient estimation. In Asian Conference on Machine Learning, pp. 333-348. PMLR, 2016.
11

---

Published as a conference paper at ICLR 2021
A UNBIASED AVEC POLICY GRADIENT
In this section, we consider the case in which the state-action value function of policy He is approximated. We prove that given some assumptions on this estimator function, use it to yield valid gradient direction, i.e., we are able to prove policy improvement when following this direction.
In this setting, the critic minimizes the following loss:
(e [ (s, a) - fo(s, a) (Q fo(s,a)))
When local optimum is reached. the gradient of the latter expression is zero:
V CAVEC = ((saa)= (QTH a) fa(s, [(sa))=100 (s, - fo(s, a)l)
ff((s (s.a)~=
pfo(s.0)
= 0.
00
06
In the expression above, the expected value of the partial derivative disappears because the term in the first bracket is centered:
0fg(s, a)
E (s, (670 (s. a)) (0 ff(s. (s.a)-m
0
a)) (1
(s,0)=
B(ss))==( - fell
O.
Simplifying the gradient at the local optimum becomes:
(Q (s, a) fo(s, a) (a (s) So(s, a)))
ff(ss
(4)
Os
Then, if we denote Do sols, a) ( (() So(s a)], and use the policy parameterization assumption:
Afp(s,
@mp(s, a)
00
00
To(s,a)
obtain:
V B( (V (o)(
Proof. By combining the parameterization assumption in Eq. with Eq. 4, we have:
(s, (Q*² (aon))-g(ss,, a)) (m 1 O. To(s,
00
Since the expression above is null, we have the following:
VaJ E (x,
OTo(s,a) Te(s,a)
V Lo (x,a)==
(c))
00
V [e
((,a)~=
Remark. While the proof seems more or less generic, the assumption in Eq. 5 is extremely constraining to the possible approximators. Sutton et al. (2000) quotes Tsitsiklis who believes that a linear Io in the features of the policy may be the only feasible solution for this condition.
Concretely, such an assumption cannot hold since neural networks are the standard approximators used in practice. Moreover, empirical analysis (Ilyas et al., 2020) indicates that commonly used algorithms fail to the true value function. However, this does not rule out the usefulness of the approach but rather begs for more questioning of the true effect of such biased baselines.
12

---

Published as a conference paper at ICLR 2021
B ADDITIONAL EXPERIMENTS
B.1 CONTINUOUS CONTROL: WALKER2D
Fig. shows the total average return for AVEC coupled with SAC and PPO on the Walker2d task. Similar to considered other continuous control tasks from MuJoCo and PyBullet, AVEC brings significant performance improvement (+26% for SAC and +33% for PPO), confirming the generality of our approach.
Walker2d
Walker2d
5000
Average Return
4000
Average Roturn
3000
3000
2000
2000
1000
1000
0.0
0.6 Number Timesteps
0.8
Number of Timesteps
Figure 8: Comparative evaluation (6 seeds) of AVEC with SAC (left) and PPO (right) the Walker2d MuJoCo task. Lines are average performances and shaded areas represent one standard deviation.
13

---

Published as a conference paper at ICLR 2021
B.2 VARIATION OF THE BIAS AND VARIANCE TERMS: PPO
In Fig. 9, we show the variation of the bias and variance terms in the MSE between the estimators Var(AVEC) and (PPOO - Vl Bias (Pas( + (PPO where v& (PPO) is the value function estimator in PPO. We observe that the variance reduction is more substantial than that the bias. Using those results and Fig. 5 showing that the distance of the estimator to V is lower when using AVEC confirms that the variance reduction effect counterbalances the bias increase. Note that the % Variation of the Var term is always negative in our experiments, and that the shaded areas that suggest otherwise are merely due to a false assumption of symmetrical deviations, itself due to the assumption of Gaussianity needed construct confidence intervals.
(of AVEC-PPO and PPO) and the true target: Ellgg= 1T Bias(
AntBullet
HalfCheetahBullet
Blas
0.2
Var
0.0
Variatin
-02-
% Variation
-02
-0.4
-0.4
-0.8
Baa Var
-1.
265
State-action pairs
State-action pairs
Figure 9: % Variation of the bias and variance terms in the MSE between the estimator and the true
Bias" (AVEC-PPO)- (PPO) and % Variation Var) Var(AvEC-PPO) (ar()))))))))
target: % Variation Bias)
(PPO)
Var(PPO)
X-axis: we run PPO and AVEC-PPO and for every / E 21, 4, 6, 9} 10 we stop training, use the current policy to interact with the environment for 3 10 transitions, and use these transitions to estimate the true value function. Lines are average variations and shaded areas represent one standard deviation (5 seeds).
B.3 LEARNING THE TRUE TARGET: SAC
In Fig. 10, compare the error between the Q-function estimator and the true Q-function for SAC and AVEC-SAC in AntBullet and BalflheetahBullet.. We note a modest but consistent reduction in this error when using AVEC coupled with SAC, echoing the significant performance gains in Fig. 2.
AntBullet
HalfCheetahBullet
AVEC-RAC
AVEC SAC
Distancce true O-function
0.3
SAC
Distance to true Q-fucctonn
0.5
0.4
0.2
0.3
0.1
0.0
action pairs
State-action pains
Figure 10: Distance to the true Q-function (SAC). X-axis: we rum SAC and SAC and for every I EE (1,2,4, 9}- 10 we stop training, use the current policy to interact with the environment for 3 10 transitions, and use these transitions to estimate the true value function. Lines are average performances and shaded areas represent one standard deviation.
14

---

Published as a conference paper at ICLR 2021
B.4 VARIANCE REDUCTION
In Fig. 11, we study the empirical variance of the gradient in measuring the average pairwise cosine similarity (10 gradient measurements) in two additional tasks: HopperBullet and Walker2DBullet. We also vary the trajectory size used in the estimation of the gradient.
HopperBullet
Walker2DBullet
AVECPPP
AVECPPO
PPO
0.22
PPO
PPO nobeselig
0.13
Acsirage palnitso
6.2
0:13
6.10
6/8
0.8
Timeshops
166
HopperBuilet
Walker2DBullet
0.16
AVEC-PO
AVECPPO
PPO
PPO
PPO-nobaseine
Averaga parnatse 005
0.72
0.10
3.94
0.06
0.66
Timestops
Timestops
166
HopperBullet
Walker2D
AVEC-PPO
0.95
AVECPPO
PPO in
0.55
PPO PPO-ncluseIne
3.94
Avsrage poleaiso
0.13
2.12
2.11
0.00
60
0.2
64
64
2.4
1/0
00
82
0.4
0.6
66
Timeshops
Timestoga
Figure 11: Average cosine similarity between gradient measurements. AVEC empirically reduces the variance compared to PPO or PPO without a baseline (PPO-nobaseline) Trajectory size used in estimation of the gradient variance: 3000 (upper row), 6000 (middle row), 9000 (lower row). Lines are average performances and shaded areas represent one standard deviation.
15

---

Published as a conference paper at ICLR 2021
C IMPLEMENTATION OF AVEC COUPLED WITH SAC
In Algorithm 2, Jv is the squared residual error objective to train the soft value function. See Haarnoja al. (2018) for further details and notations about SAC, not directly relevant here.
Algorithm 2 AVEC coupled with SAC.
Input parameters: 3 E [0, 1], Av 2 0, Ao 2 0, 0,
2: Initialize policy parameter value function parameter and & and Q-functions parameters #1
and Ø2
3: x
4: for each iteration do
5: for each step do
6: Ge To (Qel [st)
7: 8:41 P (St, ae)
8:
DU ae Te, st+1)}
(st,
9:
end for
for each gradient step do
11:
sample batch B from D
12:
s ++ 30 \v V (v())
13:
+ 6: do 6: Live (6.) for {1,2)
for
f A L.
15:
+ Byb + (1 B)
16: end for
17:
end for
D IMPLEMENTATION DETAILS
Theoretically, LAVEC defined as the residual variance of the value function (ef. Eq. 3). However, state-values for non-optimal policy are dependent and the variance is not tractable without access to the joint law of state-values. Consequently, to implement AVEC in practice we use the best-known proxy at hand, which is the empirical variance formula assuming independence:
LAVEC T T-
Σ
iiT (r
1 T
Σ
(fo(s)) v (T(i)) 2
where 7' is the size of the sampled trajectory.
16

---

Published as a conference paper at ICLR 2021
E EXPERIMENT DETAILS
In all experiments we choose to use the same hyperparameter values for all tasks as the best- performing ones reported in the literature or in their respective open source implementation docu- mentation. We thus ensure the best performance for the conventional actor-critic framework. In other words, since we are interested in evaluating the impact of this new critic, everything else kept as is. This experimental protocol may not benefit AVEC.
In Table 2, 3 and 4, we report the list hyperparameters common all continuous control experi- ments.
Table Hyperparameters used both in SAC and AVEC-SAC.
Parameter
Value
Adam stepsize
3 10
Discount (r)
0.99
Replay buffer size
10c
Batch size
256
Nb. hidden layers
2
Nb. hidden units per layer
256
Nonlinearity
ReLU
Target smoothing coefficient (r)
0.01
Target update interval
1
Gradient steps
1
Table 3: Hyperparameters used both in PPO and AVEC-PPO.
Parameter
Value
Horizon (7)
2048
Adam stepsize
2.5
10
Nb. epochs
10
Nb. minibatches
32
Nb. hidden layers
Nb. hidden units per layer
64
Nonlinearity
tanh
Discount (?)
0.99
GAE parameter (A)
0.95
Clipping parameter (e)
0.2
Table 4: Hyperparameters used both in TRPO and AVEC-TRPO.
Parameter
Value
Horizon (I)
2048
Adam stepsize
1 10-4
Nb. hidden layers
Nb. hidden units per layer
64
Nonlincarity
tanh
Discount (7)
0.99
GAE parameter (A)
0.95
Stepsize KI.
0.01
Nb. iterations for the conjugate gradient
15
17

---

Published as a conference paper at ICLR 2021
F COMPARATIVE EVALUATION OF AVEC WITH TRPO
In order to evaluate the performance gains in using AVEC instead of the usual actor-critic framework, we produce some additional experiments with the TRPO (Schulman et al., 2015) algorithm. Fig. 12 shows the learning curves while Table 5 reports the results.
HalfCheelsh
Humanoid
600
-10
AntBullet
HalfCheatanBullitt
Reacher
1150
1900
080
-30
588
-50
20000
40000
000000
50000
100000
200000
60000
10000
00005
100000
20100
40000
000000
00008
19000
Figure 12: Comparative evaluation of AVEC with TRPO. We run with 6 different seeds: lines are average performances and shaded areas represent standard deviation.
Table 5: Average total reward of the last 100 episodes over 6 runs of 10th timesteps. Comparative evaluation of AVEC with TRPO. + corresponds to single standard deviation over trials and (.%) is the change in performance due to AVEC
Task
TRPO
AVEC-TRPO
Ant
50.55
2..511..2416%
AntBullet
564
HCheetah
346
466 4 56
HCBullet
1154
1281 + (+11%)
Humanoid
352
3.4
Reacher
-8.5
- = (
18

---

Published as a conference paper at ICLR 2021
G ENVIRONMENTS DETAILS
Table 6: Environments details.
Environment
Description
Ant-v2
Make a four-legged creature walk forward as fast possible.
AntBulletEny-vO
Idem. Ant is heavier, encouraging it to typically have two or more legs on the ground (source: Py- Bullet Guide url).
HalfCheetah-v2
Make 2D cheetah robot run. Idem.
HalfCheetahBalleeEnv-00
Humanoid-v2
Make a three-dimensional bipedal robot walk for- ward fast possible, without falling over. Make a 2D robot reach to a randomly located tar- get.
Reacher-v2
Walker2d-v2
Make a 2D robot walk forward as fast as possible. Swing the end of two-joint acrobot up to given height.
Acrobot-v1
MountainCar-v0
Get an under powered car the top of hill.
H DIMENSIONS OF STUDIED TASKS
Table 7: Actions and observations dimensions.
Task
S
A
Ant
R111
R
R
AntBullet
28
R
HalfCheetah
R
R6
BullChectahBullet
R26
R.6
Humanoid
377
IR7
Reacher
R
R²
Walker2d
17
R6
Acrobot
3
MountainCar
TR2
3
19


Reviews:
[{"review_id": "pKg71-ogGKd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n\nThe paper proposes a simple and elegant idea for changing the value function objectives in deep RL and demonstrates reasonable empirical evidence of it's potential usefulness.  The authors also provide a clearly articulated intuitive motivation and provide experiments to support the proposal.  The idea complements several other algorithms and is therefore quite widely applicable (and easy to try). The analysis of the experiments is also quite interesting and clearly presented. \n\n### Weaknesses\n\nThe paper is mostly well written and has interesting theoretical insights as well as empirical analysis. Here are a some weaknesses.\n\n* The theoretical justification for the variance reduction while technically correct, seems like it should be miniscule in theory. For the $T$ independent RV case being analyzed, the condition required for the improvement is that $\\Delta  \\triangleq 2 \\mathbb{V}(X_i) - \\frac{1}{T} \\sum_{j=1}^T \\mathbb{V}(X_j) > 0$, which seems reasonable unless the sample in question is an outlier with a very small variance to begin with. However, the overall reduction itself has another $\\frac{1}{T}$ scaling, i.e. the variance reduction over the squared error case is equal to $\\frac{\\Delta}{T}$, which seems to be vanishingly small as the number of samples $T$ is large even if $\\Delta \\gg 0$. Note that for the situation where this core idea is being applied, the parameter $T$ is approximately, the number of samples in the expectation over $(s, a)$, which is large in practice.\n* The improvements are a good sanity check, but somewhat marginal in many cases (especially given the error bars).\n\n### Additional comments/feedback\n\n* In Section 4.2 paragraph on State-value function estimation line 3, should the targets be $\\widehat{V}^\\pi$ rather than $V^\\pi$?\n* In Figure 1, some additional detail on the claims seems necessary (e.g. what parameterization is being considered?)\n* In the discussion below the specification for $\\mathcal{L}^1_{AVEC}, \\mathcal{L}^2_{AVEC}$, the authors say \"the reader may have noticed that these equations  slightly differ from Eq. 3\", but I am not able to see what difference is being alluded to.\n* Figure 4 looks quite surprising in terms of the large qualitative difference between the baseline and AVEC-baseline graphs. Just to be sure, do you measure the fit with respect to $f_\\phi$ or the bias corrected version, $g_\\phi$? (obviously, the latter makes more sense?). \n* The Ablation study in Section 5.4 seems intriguing, but what the conclusions imply seems unclear. It appears the authors were expecting to see some non-zero value of $\\alpha$ to improve over $\\alpha=0$ (AVEC), but this isn't the case? Some additional clarification here would be useful. Also, it is a bit confusing to separate the plots into two depending on whether the weighting is less than one; as I'm guessing the exact same plot is used for the non-alpha versions in each pair of these graphs?\n* In Figure 5, the distance to the true value function seems to be relatively flat (or even mildly increasing) through the entire horizon in both graphs. Is this simply due to the resolution, as I'd expect there to be a drop at least in the initial phase over time.\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QY9bYuNLk8O", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores an alternative loss function for fitting critic in Reinforcement Learning. Instead of using the standard mean squared loss between critic predictions and value estimates, the authors propose to use a loss function that also incorporates a variance term. The authors dub the approach AVEC. The authors combine their approach with popular RL algorithms such as SAC and PPO and evaluated on the standard benchmarks for continuous control.\n\nAlthough the paper demonstrates interesting empirical results, I think that the current experimental evaluation has a number of flaws that prevent me from recommending this paper for acceptance. The paper provides basic motivation but it is lacking thorough theoretical investigation of the phenomena. Also the proposed loss is biased in the stochastic mini batch optimization due to the expectation under the squared term that is not addressed in the paper either. Finally, I have major concerns regarding the experimental evaluation. The set of OpenAI mujoco tasks is different from commonly used tasks in literature. In particular, Hopper and Walker2d, which are used in the vast majority of the literature, are ignored in table 1 and figure 2. This fact raises major concerns regarding generality of the approach.\n\nIn conclusion, the paper presents interesting results on some tasks for continuous control. However, the paper requires more thorough experimental evaluation to confirm the statements. Also a deeper theoretical analysis will greatly benefit this work. I strongly encourage the authors to continuous working this approach and revise the paper to improve the theoretical and empirical analysis. This paper presents a very interesting idea but in the current form it is not ready for acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D2va7mvewTd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents AVEC, a new critic loss for model-free actor-critic Reinforcement Learning algorithms. The AVEC loss can be used with any actor-critic algorithm, with PPO, TRPO and SAC being evaluated in the paper. The loss builds on the mean-squared-error, and adds a term that minimizes $E_s [f_{\\\\phi}(s) - \\\\hat{V}^{\\\\pi_{\\\\theta_k}}(s) ]$. The addition of that extra term is motivated by recent research on the stability of actor-critic algorithms, and the benefits obtained by the AVEC loss are empirically demonstrated in numerous environments, with AVEC+PPO, AVEC+SAC and AVEC+TRPO.\n\nQuality: the paper presents an interesting idea, that is simple but well-motivated, and leads to encouraging empirical results. Both the theoretical and empirical motivations are strong.\n\nClarity: the paper flows well and is quite clear. However, an intuition for what the added term in the AVEC loss is missing. Section 4.2 motivates the added term in a mathematical way, but a few sentences explaining what the added term does, in simple terms, may help the readers understand why AVEC is a better loss than simple MSE.\n\nOriginality: the contribution of this paper seems original. It builds on recent work, but the recent work identifies problems while this paper offers an original solution to these problems.\n\nSignificance: the fact that AVEC provides good empirical results, and can be used as the critic loss of any actor-critic Reinforcement Learning algorithm, points at the high significance of this work. Many actor-critic implementations can easily be improved by using the AVEC loss. Another positive point is that the paper discusses how to implement the AVEC loss in algorithms that fit a neural network on batches of samples. This really helps implementing the proposed loss, that contains an expectation in an expectation and is therefore not trivial to properly implement.\n\nIn general, I like this paper and recommend acceptance.\n\nA few questions/issues:\n\n- An explicit mention of the gradient of the loss, or at least a discussion of where to stop back-propagating gradients, would have been interesting. $f_{\\phi}$ appears two times in the AVEC loss, and it is unclear whether the loss contributes to gradients in $f_{\\phi}$ two times, or if the expectation over states is first computed (without computing any gradients), and then used as a constant in the rest of the evaluation of the loss.\n- As mentioned in \"clarity\", an intuition of what the added term of the AVEC loss does, especially since it is \"inserted\" in the mean-squared-error (inside the square), would help the less mathematics-savvy readers. It is not crucial to understand the paper, but the generality of the approach proposed in the paper may lead it to be used often by students, and so an intuition of why AVEC works and what it does would greatly help.\n\nAuthor response: the authors clarified my questions, so I maintain my recommendation for acceptance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is accepted, however, it could be much stronger by addressing the concerns below.

The theoretical analysis of the proposed methods is weak.
* As far as I can tell, the proposition has more to do with the compatible feature assumption than their method. Furthermore the compatible feature assumption is very strong and not satisfied in any of their experiments.
* Sec 4.2 does not provide strong support for their method. R2 points out issues with their statements about variance and the next subsection argues from an overly simplistic diagram.

The experimental results are promising, however, R3 brought up important issues in the private discussion:
* Their implementation of SAC systematically produces results worse than reported in the original paper (they use a version of SAC with automatically tuned temperature https://arxiv.org/pdf/1812.05905.pdf); 1a) Their SAC gets average returns of 2.5k at 500k steps while the original implementation gets 3k at 500k steps; 1b) Their SAC on HalfCheetah 10k at 1M steps, original paper - 11k at 1M steps; 1c) The same applies to Humanoid, there is no improvement with respect to the original SAC;
* Their approach degrades performance on Hopper. 
* They use non-standard hyper parameters for SAC. 0.98 instead of 0.99 for the discount and 0.01 instead of 0.005 for the soft target updates. That might be the main reason why their SAC works worse than the original implementation. 
* The authors use the hyper-parameters suggested for HalfCheetahBulletEnv for all continuous control tasks. For HalfCheetah, however, the authors of the stable-baselines repository (which this paper uses) suggest to use the hyper parameters from the original SAC paper (https://github.com/araffin/rl-baselines-zoo/blob/master/hyperparams/sac.yml#L48). Nonetheless, the results for the unmodified SAC reported in this work for HalfCheetah/Hopper/Walker/Ant are subpar to the original results, suggesting that the hyper-parameters for HalfCheetahBulletEnv are suboptimal for these tasks.

Given the simplicity of the change and the promising experimental results (with some caveats), I believe the community will find this paper interesting and will lead to followup work that can patch the theoretical gaps.
Decision: Accept (Poster)