Learning Value Functions in Deep Policy Gradients using Residual Variance

1 INTRODUCTION . Model-free deep reinforcement learning ( RL ) has been successfully used in a wide range of problem domains , ranging from teaching computers to control robots to playing sophisticated strategy games ( Silver et al. , 2014 ; Schulman et al. , 2016 ; Lillicrap et al. , 2016 ; Mnih et al. , 2016 ) . Stateof-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework ( Sutton et al. , 2000 ; Schulman et al. , 2017 ; Haarnoja et al. , 2018 ) . While such methods demonstrate great performance in continuous control tasks , several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains . For instance , research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions , then propose a critic formulation that allows for a more stable policy gradient . However , new studies ( Tucker et al. , 2018 ; Ilyas et al. , 2020 ) indicate that state-of-the-art policy gradient methods ( Schulman et al. , 2015 ; 2017 ) fail to fit the true value function and that recently proposed state-action-dependent baselines ( Gu et al. , 2016 ; Liu et al. , 2018 ; Wu et al. , 2018 ) do not reduce gradient variance more than state-dependent ones . These findings leave the reader skeptical about actor-critic algorithms , suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning . Consequently , attempting to find a better baseline is questionable , as critics would typically fail to fit it ( Ilyas et al. , 2020 ) . In Tucker et al . ( 2018 ) , the authors argue that “ much larger gains could be achieved by instead improving the accuracy of the value function ” . Following this line of thought , we are interested in ways to better approximate the value function . One approach addressing this issue is to put more focus on relative state-action values , an idea introduced in the literature on advantage reinforcement ∗Equal contribution . learning ( Harmon & Baird III ) followed by works on dueling ( Wang et al. , 2016 ) neural networks . More recent work ( Lin & Zhou , 2020 ) also suggests that considering the relative action values , or more precisely the ranking of actions in a state leads to better policies . The main argument behind this intuition is that it suffices to identify the optimal actions to solve a task . We extend this principle of relative action value with respect to the mean value to cover both state and state-action-value functions with a new objective for the critic : minimizing the variance of residual errors . In essence , this modified loss function puts more focus on the values of states ( resp . state-actions ) relative to their mean value rather than their absolute values , with the intuition that solving a task corresponds to identifying the optimal action ( s ) rather than estimating the exact value of each state . In summary , this paper : • Introduces Actor with Variance Estimated Critic ( AVEC ) , an actor-critic method providing a new training objective for the critic based on the residual variance . • Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modified gradient estimator . • Demonstrates experimentally that AVEC , when coupled with state-of-the-art policy gradient algorithms , yields a significant performance boost on a set of challenging tasks , including environments with sparse rewards . • Provides empirical evidence supporting a better fit of the true value function and a substantial stabilization of the gradient . 2 RELATED WORK . Our approach builds on three lines of research , of which we give a quick overview : policy gradient algorithms , regularization in policy gradient methods , and exploration in RL . Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator . This was originally formulated as the REINFORCE algorithm ( Williams , 1992 ) . Kakade & Langford ( 2002 ) later created conservative policy iteration and provided lower bounds for the minimum objective improvement . Peters et al . ( 2010 ) replaced regularization by a trust region constraint to stabilize training . In addition , extensive research investigated methods to improve the stability of gradient updates , and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories , the corresponding variance can be extremely high . To improve stability , Weaver & Tao ( 2001 ) show that subtracting a baseline ( Williams , 1992 ) from the value function in the policy gradient can be very beneficial in reducing variance without damaging the bias . However , in practice , these modifications on the actor-critic framework usually result in improved performance without a significant variance reduction ( Tucker et al. , 2018 ; Ilyas et al. , 2020 ) . Currently , one of the most dominant on-policy methods are proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) and trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) , both of which require new samples to be collected for each gradient step . Another direction of research that overcomes this limitation is off-policy algorithms , which therefore benefit from all sample transitions ; soft actor-critic ( SAC ) ( Haarnoja et al. , 2018 ) is one such approach achieving state-of-the-art performance . Several works also investigate regularization effects on the policy gradient ( Jaderberg et al. , 2016 ; Namkoong & Duchi , 2017 ; Kartal et al. , 2019 ; Flet-Berliac & Preux , 2019 ; 2020 ) ; it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias . In RL , regularization is often used to encourage exploration and takes the form of an entropy term ( Williams & Peng , 1991 ; Schulman et al. , 2017 ) . Moreover , while regularization in machine learning generally consists in smoothing over the observation space , in the RL setting , Thodoroff et al . ( 2018 ) show that it is possible to smooth over the temporal dimension as well . Furthermore , Zhao et al . ( 2016 ) analyze the effects of a regularization using the variance of the policy gradient ( the idea is reminiscent of SVRG descent ( Johnson & Zhang , 2013 ) ) which proves to provide more consistent policy improvements at the expense of reduced performance . In contrast , as we will see later , AVEC does not change the policy network optimization procedure nor involves any additional computational cost . Exploration has been studied under different angles in RL , one common strategy is -greedy , where the agent explores with probability by taking a random action . This method , just like entropy regularization , enforces uniform exploration and has achieved recent success in game playing en- vironments ( Mnih et al. , 2013 ; Van Hasselt et al. , 2015 ; Mnih et al. , 2016 ) . On the other hand , for most policy-based RL , exploration is a natural component of any algorithm following a stochastic policy , choosing sub-optimal actions with non-zero probability . Furthermore , policy gradient literature contains exploration methods based on uncertainty estimates of values ( Kaelbling , 1993 ; Tokic , 2010 ) , and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration ( Schmidhuber , 2006 ; Bellemare et al. , 2016 ; Flet-Berliac et al. , 2021 ) . While existing research may share some motivations with our method , no previous work in RL applies the variance of residual errors as an objective loss function . In the context of linear regression , Brown ( 1947 ) considers a median-unbiased estimator minimizing the risk with respect to the absolutedeviation loss function ( Pham-Gia & Hung , 2001 ) ( similar in spirit to the variance of residual errors ) , their motivation is nonetheless different to ours . Indeed , they seek to be robust to outliers whereas , when considering noiseless RL problems , one usually seeks to capture those ( sometimes rare ) signals corresponding to the rewards . 3 PRELIMINARIES . 3.1 BACKGROUND AND NOTATIONS . We consider an infinite-horizon Markov Decision Problem ( MDP ) with continuous states s ∈ S , continuous actions a ∈ A , transition distribution st+1 ∼ P ( st , at ) and reward function rt ∼ R ( st , at ) . Let πθ ( a|s ) denote a stochastic policy with parameter θ , we restrict policies to being Gaussian distributions . In the following , π and πθ denote the same object . The agent repeatedly interacts with the environment by sampling action at ∼ π ( .|st ) , receives reward rt and transitions to a new state st+1 . The objective is to maximize the expected sum of discounted rewards : J ( π ) , Eτ∼π [ ∞∑ t=0 γtr ( st , at ) ] , ( 1 ) where γ ∈ [ 0 , 1 ) is a discount factor ( Puterman , 1994 ) , and τ = ( s0 , a0 , r0 , s1 , a1 , r1 , . . . ) is a trajectory sampled from the environment using policy π . We denote the value of a state s in the MDP framework while following a policy π by V π ( s ) , Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) |s0 = s ] and the value of a state-action pair of performing action a in state s and then following policy π by Qπ ( s , a ) , Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , a0 = a ] . Finally , the advantage function which quantifies how an action a is better than the average action in state s is denoted Aπ ( s , a ) , Qπ ( s , a ) − V π ( s ) . 3.2 CRITICS IN DEEP POLICY GRADIENTS . In this section , we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient . Without loss of generality , we consider the algorithms that approximate the state-value function V . The analysis holds for algorithms that approximate the state-action-value function Q . Let fφ : S → R be an estimator of V̂ π with φ its parameter . fφ is traditionally learned through minimizing the mean squared error ( MSE ) against V̂ π . At iteration k , the critic minimizes : LAC = Es [ ( fφ ( s ) − V̂ πθk ( s ) ) 2 ] , ( 2 ) where the states s are collected under policy πθk , and V̂ πθk ( s ) is an empirical estimate of V ( see Section 4.3 for details ) . Similarly , using fφ : S ×A → R instead , one can fit an empirical target Q̂π . 4 METHOD : ACTOR WITH VARIANCE ESTIMATED CRITIC . In this section , we introduce AVEC and discuss its correctness , motivations and implementation . 4.1 DEFINING AN ALTERNATIVE CRITIC . Recent work ( Ilyas et al. , 2020 ) empirically demonstrates that while the value network succeeds in the supervised learning task of fitting V̂ π ( resp . Q̂π ) , it does not fit V π ( resp . Qπ ) . We address this deficiency in the estimation of the critic by introducing an alternative value network loss . Following empirical evidence indicating that the problem is the approximation error and not the estimator per se , AVEC adopts a loss that can provide a better approximation error , and yields better estimators of the value function ( as will be shown in Section 5.3 ) . At update k : LAVEC = Es [ ( ( fφ ( s ) − V̂ πθk ( s ) ) − Es [ fφ ( s ) − V̂ πθk ( s ) ] ) 2 ] , ( 3 ) with states s collected using πθk . Note that the gradient flows in fφ twice using Eq . 3 . Then , we define our bias-corrected estimator : gφ : S → R such that gφ ( s ) = fφ ( s ) + Es [ V̂ πθk ( s ) − fφ ( s ) ] . Analogously to Eq . 3 , we define an alternative critic for the estimation of Qπ by replacing V̂ π by Q̂π and fφ ( s ) by fφ ( s , a ) . Proposition ( AVEC Policy Gradient ) . If fφ : S × A → R satisfies the parameterization assumption ( Sutton et al. , 2000 ) then gφ provides an unbiased policy gradient : ∇θJ ( πθ ) = E ( s , a ) ∼πθ [ ∇θ log ( πθ ( s , a ) ) gφ ( s , a ) ] . Proof . See Appendix A . This result also holds for the estimation of V πθ with fφ : S → R .

Summary: The paper proposes a simple and elegant idea for changing the value function objectives in deep RL and demonstrates reasonable empirical evidence of it's potential usefulness.  The authors also provide a clearly articulated intuitive motivation and provide experiments to support the proposal.  The idea complements several other algorithms and is therefore quite widely applicable (and easy to try). The analysis of the experiments is also quite interesting and clearly presented.