Learning Value Functions in Deep Policy Gradients using Residual Variance

1 INTRODUCTION . Model-free deep reinforcement learning ( RL ) has been successfully used in a wide range of problem domains , ranging from teaching computers to control robots to playing sophisticated strategy games ( Silver et al. , 2014 ; Schulman et al. , 2016 ; Lillicrap et al. , 2016 ; Mnih et al. , 2016 ) . Stateof-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework ( Sutton et al. , 2000 ; Schulman et al. , 2017 ; Haarnoja et al. , 2018 ) . While such methods demonstrate great performance in continuous control tasks , several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains . For instance , research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions , then propose a critic formulation that allows for a more stable policy gradient . However , new studies ( Tucker et al. , 2018 ; Ilyas et al. , 2020 ) indicate that state-of-the-art policy gradient methods ( Schulman et al. , 2015 ; 2017 ) fail to fit the true value function and that recently proposed state-action-dependent baselines ( Gu et al. , 2016 ; Liu et al. , 2018 ; Wu et al. , 2018 ) do not reduce gradient variance more than state-dependent ones . These findings leave the reader skeptical about actor-critic algorithms , suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning . Consequently , attempting to find a better baseline is questionable , as critics would typically fail to fit it ( Ilyas et al. , 2020 ) . In Tucker et al . ( 2018 ) , the authors argue that “ much larger gains could be achieved by instead improving the accuracy of the value function ” . Following this line of thought , we are interested in ways to better approximate the value function . One approach addressing this issue is to put more focus on relative state-action values , an idea introduced in the literature on advantage reinforcement ∗Equal contribution . learning ( Harmon & Baird III ) followed by works on dueling ( Wang et al. , 2016 ) neural networks . More recent work ( Lin & Zhou , 2020 ) also suggests that considering the relative action values , or more precisely the ranking of actions in a state leads to better policies . The main argument behind this intuition is that it suffices to identify the optimal actions to solve a task . We extend this principle of relative action value with respect to the mean value to cover both state and state-action-value functions with a new objective for the critic : minimizing the variance of residual errors . In essence , this modified loss function puts more focus on the values of states ( resp . state-actions ) relative to their mean value rather than their absolute values , with the intuition that solving a task corresponds to identifying the optimal action ( s ) rather than estimating the exact value of each state . In summary , this paper : • Introduces Actor with Variance Estimated Critic ( AVEC ) , an actor-critic method providing a new training objective for the critic based on the residual variance . • Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modified gradient estimator . • Demonstrates experimentally that AVEC , when coupled with state-of-the-art policy gradient algorithms , yields a significant performance boost on a set of challenging tasks , including environments with sparse rewards . • Provides empirical evidence supporting a better fit of the true value function and a substantial stabilization of the gradient . 2 RELATED WORK . Our approach builds on three lines of research , of which we give a quick overview : policy gradient algorithms , regularization in policy gradient methods , and exploration in RL . Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator . This was originally formulated as the REINFORCE algorithm ( Williams , 1992 ) . Kakade & Langford ( 2002 ) later created conservative policy iteration and provided lower bounds for the minimum objective improvement . Peters et al . ( 2010 ) replaced regularization by a trust region constraint to stabilize training . In addition , extensive research investigated methods to improve the stability of gradient updates , and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories , the corresponding variance can be extremely high . To improve stability , Weaver & Tao ( 2001 ) show that subtracting a baseline ( Williams , 1992 ) from the value function in the policy gradient can be very beneficial in reducing variance without damaging the bias . However , in practice , these modifications on the actor-critic framework usually result in improved performance without a significant variance reduction ( Tucker et al. , 2018 ; Ilyas et al. , 2020 ) . Currently , one of the most dominant on-policy methods are proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) and trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) , both of which require new samples to be collected for each gradient step . Another direction of research that overcomes this limitation is off-policy algorithms , which therefore benefit from all sample transitions ; soft actor-critic ( SAC ) ( Haarnoja et al. , 2018 ) is one such approach achieving state-of-the-art performance . Several works also investigate regularization effects on the policy gradient ( Jaderberg et al. , 2016 ; Namkoong & Duchi , 2017 ; Kartal et al. , 2019 ; Flet-Berliac & Preux , 2019 ; 2020 ) ; it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias . In RL , regularization is often used to encourage exploration and takes the form of an entropy term ( Williams & Peng , 1991 ; Schulman et al. , 2017 ) . Moreover , while regularization in machine learning generally consists in smoothing over the observation space , in the RL setting , Thodoroff et al . ( 2018 ) show that it is possible to smooth over the temporal dimension as well . Furthermore , Zhao et al . ( 2016 ) analyze the effects of a regularization using the variance of the policy gradient ( the idea is reminiscent of SVRG descent ( Johnson & Zhang , 2013 ) ) which proves to provide more consistent policy improvements at the expense of reduced performance . In contrast , as we will see later , AVEC does not change the policy network optimization procedure nor involves any additional computational cost . Exploration has been studied under different angles in RL , one common strategy is -greedy , where the agent explores with probability by taking a random action . This method , just like entropy regularization , enforces uniform exploration and has achieved recent success in game playing en- vironments ( Mnih et al. , 2013 ; Van Hasselt et al. , 2015 ; Mnih et al. , 2016 ) . On the other hand , for most policy-based RL , exploration is a natural component of any algorithm following a stochastic policy , choosing sub-optimal actions with non-zero probability . Furthermore , policy gradient literature contains exploration methods based on uncertainty estimates of values ( Kaelbling , 1993 ; Tokic , 2010 ) , and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration ( Schmidhuber , 2006 ; Bellemare et al. , 2016 ; Flet-Berliac et al. , 2021 ) . While existing research may share some motivations with our method , no previous work in RL applies the variance of residual errors as an objective loss function . In the context of linear regression , Brown ( 1947 ) considers a median-unbiased estimator minimizing the risk with respect to the absolutedeviation loss function ( Pham-Gia & Hung , 2001 ) ( similar in spirit to the variance of residual errors ) , their motivation is nonetheless different to ours . Indeed , they seek to be robust to outliers whereas , when considering noiseless RL problems , one usually seeks to capture those ( sometimes rare ) signals corresponding to the rewards . 3 PRELIMINARIES . 3.1 BACKGROUND AND NOTATIONS . We consider an infinite-horizon Markov Decision Problem ( MDP ) with continuous states s ∈ S , continuous actions a ∈ A , transition distribution st+1 ∼ P ( st , at ) and reward function rt ∼ R ( st , at ) . Let πθ ( a|s ) denote a stochastic policy with parameter θ , we restrict policies to being Gaussian distributions . In the following , π and πθ denote the same object . The agent repeatedly interacts with the environment by sampling action at ∼ π ( .|st ) , receives reward rt and transitions to a new state st+1 . The objective is to maximize the expected sum of discounted rewards : J ( π ) , Eτ∼π [ ∞∑ t=0 γtr ( st , at ) ] , ( 1 ) where γ ∈ [ 0 , 1 ) is a discount factor ( Puterman , 1994 ) , and τ = ( s0 , a0 , r0 , s1 , a1 , r1 , . . . ) is a trajectory sampled from the environment using policy π . We denote the value of a state s in the MDP framework while following a policy π by V π ( s ) , Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) |s0 = s ] and the value of a state-action pair of performing action a in state s and then following policy π by Qπ ( s , a ) , Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , a0 = a ] . Finally , the advantage function which quantifies how an action a is better than the average action in state s is denoted Aπ ( s , a ) , Qπ ( s , a ) − V π ( s ) . 3.2 CRITICS IN DEEP POLICY GRADIENTS . In this section , we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient . Without loss of generality , we consider the algorithms that approximate the state-value function V . The analysis holds for algorithms that approximate the state-action-value function Q . Let fφ : S → R be an estimator of V̂ π with φ its parameter . fφ is traditionally learned through minimizing the mean squared error ( MSE ) against V̂ π . At iteration k , the critic minimizes : LAC = Es [ ( fφ ( s ) − V̂ πθk ( s ) ) 2 ] , ( 2 ) where the states s are collected under policy πθk , and V̂ πθk ( s ) is an empirical estimate of V ( see Section 4.3 for details ) . Similarly , using fφ : S ×A → R instead , one can fit an empirical target Q̂π . 4 METHOD : ACTOR WITH VARIANCE ESTIMATED CRITIC . In this section , we introduce AVEC and discuss its correctness , motivations and implementation . 4.1 DEFINING AN ALTERNATIVE CRITIC . Recent work ( Ilyas et al. , 2020 ) empirically demonstrates that while the value network succeeds in the supervised learning task of fitting V̂ π ( resp . Q̂π ) , it does not fit V π ( resp . Qπ ) . We address this deficiency in the estimation of the critic by introducing an alternative value network loss . Following empirical evidence indicating that the problem is the approximation error and not the estimator per se , AVEC adopts a loss that can provide a better approximation error , and yields better estimators of the value function ( as will be shown in Section 5.3 ) . At update k : LAVEC = Es [ ( ( fφ ( s ) − V̂ πθk ( s ) ) − Es [ fφ ( s ) − V̂ πθk ( s ) ] ) 2 ] , ( 3 ) with states s collected using πθk . Note that the gradient flows in fφ twice using Eq . 3 . Then , we define our bias-corrected estimator : gφ : S → R such that gφ ( s ) = fφ ( s ) + Es [ V̂ πθk ( s ) − fφ ( s ) ] . Analogously to Eq . 3 , we define an alternative critic for the estimation of Qπ by replacing V̂ π by Q̂π and fφ ( s ) by fφ ( s , a ) . Proposition ( AVEC Policy Gradient ) . If fφ : S × A → R satisfies the parameterization assumption ( Sutton et al. , 2000 ) then gφ provides an unbiased policy gradient : ∇θJ ( πθ ) = E ( s , a ) ∼πθ [ ∇θ log ( πθ ( s , a ) ) gφ ( s , a ) ] . Proof . See Appendix A . This result also holds for the estimation of V πθ with fφ : S → R .

Summary: This paper presents AVEC, a new critic loss for model-free actor-critic Reinforcement Learning algorithms. The AVEC loss can be used with any actor-critic algorithm, with PPO, TRPO and SAC being evaluated in the paper. The loss builds on the mean-squared-error, and adds a term that minimizes $E_s [f_{\\phi}(s) - \\hat{V}^{\\pi_{\\theta_k}}(s) ]$. The addition of that extra term is motivated by recent research on the stability of actor-critic algorithms, and the benefits obtained by the AVEC loss are empirically demonstrated in numerous environments, with AVEC+PPO, AVEC+SAC and AVEC+TRPO.