Parameter-Based Value Functions

1 INTRODUCTION . Value functions are central to Reinforcement Learning ( RL ) . For a given policy , they estimate the value of being in a specific state ( or of choosing a particular action in a given state ) . Many RL breakthroughs were achieved through improved estimates of such values , which can be used to find optimal policies ( Tesauro , 1995 ; Mnih et al. , 2015 ) . However , learning value functions of arbitrary policies without observing their behavior in the environment is not trivial . Such off-policy learning requires to correct the mismatch between the distribution of updates induced by the behavioral policy and the one we want to learn . Common techniques include Importance Sampling ( IS ) ( Hesterberg , 1988 ) and deterministic policy gradient methods ( DPG ) ( Silver et al. , 2014 ) , which adopt the actorcritic architecture ( Sutton , 1984 ; Konda & Tsitsiklis , 2001 ; Peters & Schaal , 2008 ) . Unfortunately , these approaches have limitations . IS suffers from large variance ( Cortes et al. , 2010 ; Metelli et al. , 2018 ; Wang et al. , 2016 ) while traditional off-policy actor-critic methods introduce off-policy objectives whose gradients are difficult to follow since they involve the gradient of the action-value function with respect to the policy parameters∇θQπθ ( s , a ) ( Degris et al. , 2012 ; Silver et al. , 2014 ) . This term is usually ignored , resulting in biased gradients for the off-policy objective . Furthermore , off-policy actor-critic algorithms learn value functions of a single target policy . When value functions are updated to track the learned policy , the information about old policies is lost . We address the problem of generalization across many value functions in the off-policy setting by introducing a class of parameter-based value functions ( PBVFs ) defined for any policy . PBVFs are value functions whose inputs include the policy parameters , the PSSVF V ( θ ) , PSVF V ( s , θ ) , and PAVF Q ( s , a , θ ) . PBVFs can be learned using Monte Carlo ( MC ) ( Metropolis & Ulam , 1949 ) or Temporal Difference ( TD ) ( Sutton , 1988 ) methods . The PAVF Q ( s , a , θ ) leads to a novel stochastic and deterministic off-policy policy gradient theorem and , unlike previous approaches , can directly compute∇θQπθ ( s , a ) . Based on these results , we develop off-policy actor-critic methods and compare our algorithms to two strong baselines , ARS and DDPG ( Mania et al. , 2018 ; Lillicrap et al. , 2015 ) , outperforming them in some environments . We make theoretical , algorithmic , and experimental contributions : Section 2 introduces the standard MDP setting ; Section 3 formally presents PBVFs and derive algorithms for V ( θ ) , V ( s , θ ) and Q ( s , a , θ ) ; Section 4 describes the experimental evaluation using shallow and deep policies ; Sections 5 and 6 discuss related and future work . Proofs and derivations can be found in Appendix A.2 . 2 BACKGROUND . We consider a Markov Decision Process ( MDP ) ( Stratonovich , 1960 ; Puterman , 2014 ) M = ( S , A , P , R , γ , µ0 ) where at each step an agent observes a state s ∈ S , chooses action a ∈ A , transitions into state s′ with probability P ( s′|s , a ) and receives a reward R ( s , a ) . The agent starts from an initial state , chosen with probability µ0 ( s ) . It is represented by a parametrized stochastic policy πθ : S → ∆ ( A ) , which provides the probability of performing action a in state s. Θ is the space of policy parameters . The policy is deterministic if for each state s there exists an action a such that πθ ( a|s ) = 1 . The return Rt is defined as the cumulative discounted reward from time step t : Rt = ∑T−t−1 k=0 γ kR ( st+k+1 , at+k+1 ) , where T denotes the time horizon and γ a realvalued discount factor . The performance of the agent is measured by the cumulative discounted expected reward ( expected return ) , defined as J ( πθ ) = Eπθ [ R0 ] . Given a policy πθ , the state-value function V πθ ( s ) = Eπθ [ Rt|st = s ] is defined as the expected return for being in a state s and following policy πθ . By integrating over the state space S , we can express the maximization of the expected cumulative reward in terms of the state-value function J ( πθ ) = ∫ S µ0 ( s ) V πθ ( s ) ds . The action-value function Qπθ ( s , a ) , which is defined as the expected return for performing action a in state s , and following the policy πθ , is Qπθ ( s , a ) = Eπθ [ Rt|st = s , at = a ] , and it is related to the state-value function by V πθ ( s ) = ∫ A πθ ( a|s ) Qπθ ( s , a ) da . We define as dπθ ( s′ ) the discounted weighting of states encountered starting at s0 ∼ µ0 ( s ) and following the policy πθ : dπθ ( s′ ) = ∫ S ∑∞ t=1 γ t−1µ0 ( s ) P ( s → s′ , t , πθ ) ds , where P ( s → s′ , t , πθ ) is the probability of transitioning to s′ after t time steps , starting from s and following policy πθ . Sutton et al . ( 1999 ) showed that , for stochastic policies , the gradient of J ( πθ ) does not involve the derivative of dπθ ( s ) and can be expressed in a simple form : ∇θJ ( πθ ) = ∫ S dπθ ( s ) ∫ A ∇θπθ ( a|s ) Qπθ ( s , a ) dads . ( 1 ) Similarly , for deterministic policies Silver et al . ( 2014 ) obtained the following : ∇θJ ( πθ ) = ∫ S dπθ ( s ) ∇θπθ ( s ) ∇aQπθ ( s , a ) |a=πθ ( s ) ds . ( 2 ) Off-policy RL In off-policy policy optimization , we seek to find the parameters of the policy maximizing a performance index Jb ( πθ ) using data collected from a behavioral policy πb . Here the objective function Jb ( πθ ) is typically modified to be the value function of the target policy , integrated over dπb∞ ( s ) = limt→∞ P ( st = s|s0 , πb ) , the limiting distribution of states under πb ( assuming it exists ) ( Degris et al. , 2012 ; Imani et al. , 2018 ; Wang et al. , 2016 ) . Throughout the paper we assume that the support of dπb∞ includes the support of µ0 so that the optimal solution for Jb is also optimal for J . Formally , we want to find : Jb ( πθ∗ ) = max θ ∫ S dπb∞ ( s ) V πθ ( s ) ds = max θ ∫ S dπb∞ ( s ) ∫ A πθ ( a|s ) Qπθ ( s , a ) da ds . ( 3 ) Unfortunately , in the off-policy setting , the states are obtained from dπb∞ and not from d πθ ∞ , hence the gradients suffer from a distribution shift ( Liu et al. , 2019 ; Nachum et al. , 2019 ) . Moreover , since we have no access to dπθ∞ , a term in the policy gradient theorem corresponding to the gradient of the action value function with respect to the policy parameters needs to be estimated . This term is usually ignored in traditional off-policy policy gradient theorems1 . In particular , when the policy is stochastic , Degris et al . ( 2012 ) showed that : ∇θJb ( πθ ) = ∫ S dπb∞ ( s ) ∫ A πb ( a|s ) πθ ( a|s ) πb ( a|s ) ( Qπθ ( s , a ) ∇θ log πθ ( a|s ) +∇θQπθ ( s , a ) ) da ds ( 4 ) ≈ ∫ S dπb∞ ( s ) ∫ A πb ( a|s ) πθ ( a|s ) πb ( a|s ) ( Qπθ ( s , a ) ∇θ log πθ ( a|s ) ) da ds . ( 5 ) Analogously , Silver et al . ( 2014 ) provided the following approximation for deterministic policies 2 : ∇θJb ( πθ ) = ∫ S dπb∞ ( s ) ( ∇θπθ ( s ) ∇aQπθ ( s , a ) |a=πθ ( s ) +∇θQπθ ( s , a ) |a=πθ ( s ) ) ds ( 6 ) ≈ ∫ S dπb∞ ( s ) ( ∇θπθ ( s ) ∇aQπθ ( s , a ) |a=πθ ( s ) ) ds . ( 7 ) 1With tabular policies , dropping this term still results in a convergent algorithm ( Degris et al. , 2012 ) . 2In the original formulation of Silver et al . ( 2014 ) dπb∞ ( s ) is replaced by dπb ( s ) . Although the term ∇θQπθ ( s , a ) is dropped , there might be advantages in using the approximate gradient of Jb in order to find the maximum of the original RL objective J . Indeed , if we were on-policy , the approximated off-policy policy gradients by Degris et al . ( 2012 ) ; Silver et al . ( 2014 ) would revert to the on-policy policy gradients , while an exact gradient for Jb would necessarily introduce a bias . However , when we are off-policy , it is not clear whether this would be better than using the exact gradient of Jb in order to maximize J . In this work , we assume that Jb can be considered a good objective for off-policy RL and we derive an exact gradient for it . 3 PARAMETER-BASED VALUE FUNCTIONS . In this section , we introduce our parameter-based value functions , the PSSVF V ( θ ) , PSVF V ( s , θ ) , and PAVF Q ( s , a , θ ) and their corresponding learning algorithms . First , we augment the state and action-value functions , allowing them to receive as an input also the weights of a parametric policy . The parameter-based state-value function ( PSVF ) V ( s , θ ) = E [ Rt|st = s , θ ] is defined as the expected return for being in state s and following policy parameterized by θ . Similarly , the parameter-based action-value function ( PAVF ) Q ( s , a , θ ) = E [ Rt|st = s , at = a , θ ] is defined as the expected return for being in state s , taking action a and following policy parameterized by θ . Using PBVFs , the RL objective becomes : J ( πθ ) = ∫ S µ0 ( s ) V π ( s , θ ) ds . Maximizing this objective leads to on-policy policy gradient theorems that are analogous to the traditional ones ( Sutton et al. , 1999 ; Silver et al. , 2014 ) : Theorem 3.1 . Let πθ be stochastic . For any Markov Decision Process , the following holds : ∇θJ ( πθ ) = Es∼dπθ ( s ) , a∼πθ ( .|s ) [ ( Q ( s , a , θ ) ∇θ log πθ ( a|s ) ) ] . ( 8 ) Theorem 3.2 . Let πθ be deterministic . Under standard regularity assumptions ( Silver et al. , 2014 ) , for any Markov Decision Process , the following holds : ∇θJ ( πθ ) = Es∼dπθ ( s ) [ ∇aQ ( s , a , θ ) |a=πθ ( s ) ∇θπθ ( s ) ] . ( 9 ) Parameter-based value functions allow us also to learn a function of the policy parameters that directly approximates J ( πθ ) . In particular , the parameter-based start-state-value function ( PSSVF ) is defined as : V ( θ ) : = Es∼µ0 ( s ) [ V ( s , θ ) ] = ∫ S µ0 ( s ) V ( s , θ ) ds = J ( πθ ) . ( 10 ) Off-policy RL In the off-policy setting , the objective to be maximized becomes : Jb ( πθ∗ ) = max θ ∫ S dπb∞ ( s ) V ( s , θ ) ds = max θ ∫ S ∫ A dπb∞ ( s ) πθ ( a|s ) Q ( s , a , θ ) dads . ( 11 ) By taking the gradient of the performance Jb with respect to the policy parameters θ we obtain novel policy gradient theorems . Since θ is continuous , we need to use function approximators Vw ( θ ) ≈ V ( θ ) , Vw ( s , θ ) ≈ V ( s , θ ) and Qw ( s , a , θ ) ≈ Q ( s , a , θ ) . Compatible function approximations can be derived to ensure that the approximated value function is following the true gradient . Like in previous approaches , this would result in linearity conditions . However , here we consider nonlinear function approximation and we leave the convergence analysis of linear PBVFs as future work . In episodic settings , we do not have access to dπb∞ , so in the algorithm derivations and in the experiments we approximate it by sampling trajectories generated by the behavioral policy . In all cases , the policy improvement step can be very expensive , due to the computation of the arg max over a continuous space Θ. Actor-critic methods can be derived to solve this optimization problem , where the critic ( PBVFs ) can be learned using TD or MC methods , while the actor is updated following the gradient with respect to the critic . Although our algorithms on PSSVF and PSVF can be used with both stochastic and deterministic policies , removing the stochasticity of the action-selection process might facilitate learning the value function . All our algorithms make use of a replay buffer . 3.1 PARAMETER-BASED START-STATE-VALUE FUNCTION V ( θ ) We first derive the PSSVF V ( θ ) . Given the original performance index J , and taking the gradient with respect to θ , we obtain : ∇θJ ( πθ ) = ∫ S µ0 ( s ) ∇θV ( s , θ ) ds = Es∼µ0 ( s ) [ ∇θV ( s , θ ) ] = ∇θV ( θ ) . ( 12 ) In Algorithm 1 , the critic Vw ( θ ) is learned using MC to estimate the value of any policy θ . The actor is then updated following the direction of improvement suggested by the critic . Since the main application of PSSVF is in episodic tasks3 , we optimize for the undiscounted objective . Algorithm 1 Actor-critic with Monte Carlo prediction for V ( θ ) Input : Differentiable critic Vw : Θ → R with parameters w ; deterministic or stochastic actor πθ with parameters θ ; empty replay buffer D Output : Learned Vw ≈ V ( θ ) ∀θ , learned πθ ≈ πθ∗ Initialize critic and actor weights w , θ repeat : Generate an episode s0 , a0 , r1 , s1 , a1 , r2 , . . . , sT−1 , aT−1 , rT with policy πθ Compute return r = ∑T k=1 rk Store ( θ , r ) in the replay buffer D for many steps do : Sample a batch B = { ( r , θ ) } from D Update critic by stochastic gradient descent : ∇w E ( r , θ ) ∈B [ r − Vw ( θ ) ] 2 end for for many steps do : Update actor by gradient ascent : ∇θVw ( θ ) end for until convergence 3.2 PARAMETER-BASED STATE-VALUE FUNCTION V ( s , θ ) Learning the value function using MC approaches can be difficult due to the high variance of the estimate . Furthermore , episode-based algorithms like Algorithm 1 are unable to credit good actions in bad episodes . Gradient methods based on TD updates provide a biased estimate of V ( s , θ ) with much lower variance and can credit actions at each time step . Taking the gradient of Jb ( πθ ) in the PSVF formulation4 , we obtain : ∇θJb ( πθ ) = ∫ S dπb∞ ( s ) ∇θV ( s , θ ) ds = Es∼dπb∞ ( s ) [ ∇θV ( s , θ ) ] . ( 13 ) Algorithm 2 ( Appendix ) uses the actor-critic architecture , where the critic is learned via TD5 . 3.3 PARAMETER-BASED ACTION-VALUE FUNCTION Q ( s , a , θ ) The introduction of the PAVF Q ( s , a , θ ) allows us to derive new policy gradients theorems when using a stochastic or deterministic policy . Stochastic policy gradients We want to use data collected from some stochastic behavioral policy πb in order to learn the action-value of a target policy πθ . Traditional off-policy actor-critic algorithms only approximate the gradient of Jb , since they do not estimate the gradient of the actionvalue function with respect to the policy parameters ∇θQπθ ( s , a ) ( Degris et al. , 2012 ; Silver et al. , 2014 ) . With PBVFs , we can directly compute this contribution to the gradient . This yields an exact policy gradient theorem for Jb : Theorem 3.3 . For any Markov Decision Process , the following holds : ∇θJb ( πθ ) = Es∼dπb∞ ( s ) , a∼πb ( .|s ) [ πθ ( a|s ) πb ( a|s ) ( Q ( s , a , θ ) ∇θ log πθ ( a|s ) +∇θQ ( s , a , θ ) ) ] . ( 14 ) Algorithm 3 ( Appendix ) uses an actor-critic architecture and can be seen as an extension of OffPAC ( Degris et al. , 2012 ) to PAVF . 3Alternatives include regenerative method for MC estimation ( Rubinstein & Kroese , 2016 ) . 4Compared to standard methods based on the state-value function , we can directly optimize the policy following the performance gradient of the PSVF , obtaining a policy improvement step in a model-free way . 5Note that the differentiability of the policy πθ is never required in PSSVF and PSVF . Deterministic policy gradients Estimating Q ( s , a , θ ) is in general a difficult problem due to the stochasticity of the policy . Deterministic policies of the form π : S → A can help improving the efficiency in learning value functions , since the expectation over the action space is no longer required . Using PBVFs , we can write the performance of a policy πθ as : Jb ( πθ ) = ∫ S dπb∞ ( s ) V ( s , θ ) ds = ∫ S dπb∞ ( s ) Q ( s , πθ ( s ) , θ ) ds . ( 15 ) Taking the gradient with respect to θ we obtain a deterministic policy gradient theorem : Theorem 3.4 . Under standard regularity assumptions ( Silver et al. , 2014 ) , for any Markov Decision Process , the following holds : ∇θJb ( πθ ) = Es∼dπb∞ ( s ) [ ∇aQ ( s , a , θ ) |a=πθ ( s ) ∇θπθ ( s ) +∇θQ ( s , a , θ ) |a=πθ ( s ) ] . ( 16 ) Algorithm 4 ( Appendix ) uses an actor-critic architecture and can be seen as an extension of DPG ( Silver et al. , 2014 ) to PAVF . Despite the novel formulation of algorithm 3 , we decided to avoid the stochasticity of the policy and to implement and analyze only the deterministic PAVF .

Summary: On page 2, in the background section: the discounted state distribution, what you wrote is not a distribution (doesn't sum to 1). In order to define this $d^{\pi_\theta}$ properly, you can multiply everything by $1-\gamma$. The interpretation is that you "reset" in your initial distribution $\mu_0$ with probability $1 - \gamma$ at every step, or continue in the discounted stationary distribution with probability $\gamma$.