S2VG: Soft Stochastic Value Gradient method

1 INTRODUCTION . Reinforcement learning can be generally classified into two categories : model-free reinforcement learning ( MFRL ) and model-based reinforcement learning ( MBRL ) . The last several years have witnessed the great success of MFRL especially in playing video games , robotic control and motion animation ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Schulman et al. , 2017 ; Peng et al. , 2018 ) . However , even for some simple tasks , hundreds of millions of samples are required for an agent to learn a good control policy . In many industry scenarios , such as health care and financial services , the algorithm requiring tremendous interactions with the environment is not applicable or too expensive to deploy . To this end , several recent works have advocated the model-based approach , where the higher sample-efficiency is achieved by leveraging the learned dynamics and reward model ( Buckman et al. , 2018 ; Feinberg et al. , 2018 ) . It generally augments the real data with the data from dynamics models , uses rollout to improve target for temporal difference learning , or directly incorporates the model into the Bellman equation ( Luo et al. , 2018 ; Heess et al. , 2015 ) . These works have demonstrated promising results on several benchmarks with a small number of interactions with the environment . Despite its recent success , MBRL still faces a challenging problem , i.e. , the model-bias , where the imperfect dynamics model would degrade the performance of the algorithm ( Kurutach et al. , 2018 ) . Unfortunately , such things always happen when the environment is sufficiently complex . There are a few efforts to mitigate such issue by combining model-based and model-free approaches . Heess et al . ( 2015 ) compute the value gradient along real system trajectories instead of planned ones to avoid the compounded error . Kalweit & Boedecker ( 2017 ) mix the real data and imaginary data from the model and then train Q function . An ensemble of neural networks can be applied to model the environment dynamics , which effectively reduces the error of the model ( Kurutach et al. , 2018 ; Clavera et al. , 2018 ; Chua et al. , 2018 ) . We observe that most recent algorithms with promising results apply Dyna-style update ( Sutton , 1990 ; Kurutach et al. , 2018 ; Luo et al. , 2018 ) . They collect real data using current policy to train the dynamics model . Then the policy is improved using state-of-the-art model-free reinforcement learning algorithms with imagined data generated by the learned model . Our insight is that why not directly embed the model into the policy improvement ? To this end , we derive a model-based reinforcement learning algorithm in the framework of the maximum entropy reinforcement learning ( Ziebart et al. , 2008 ) . Dynamics model and reward model are trained with the real data set collected from the environment . Then we simply train Q and V function using the real data set with the update rule derived from the maximum entropy principle ( several other advanced ways to include the imaginary data can also be applied , see details in section 3 ) . In the policy improvement step , the stochastic actor samples an action with real state as the input , and then the state switches from s to s′ according to the learned dynamics model . We link the learned dynamics model , reward model , and policy to compute an analytic policy gradient by the back-propagation . Comparing with likelihoodratio estimator usually used in MFRL method , such value gradient method would reduce the variance of the policy gradient ( Heess et al. , 2015 ) . The other merit of S2VG is its computational efficiency . Several state-of-the-art MBRL algorithms generate hundreds of thousands imaginary data from the model and a few real samples . Then the huge imaginary data set feeds into MFRL algorithms , which may be sample-efficient in terms of real samples but not computational-friendly . On the contrary , our algorithm embeds the model in the policy update . Thus we can implement it efficiently by computing policy gradient several times in each iteration ( see our algorithm 1 ) and do not need to do calculation on the huge imaginary data set . We name our algorithm soft stochastic value gradient to indicate its connection with SVG ( Heess et al. , 2015 ) . Notice there are several differences between S2VG and SVG . Firstly , to alleviate the issue of the compounded error , SVG proposes a relatively conservative algorithm where just real data is used to evaluate policy gradients . Thus imaginary data is wasted , even the data from the short rollouts from the model can be trusted to some extent . In our work , the policy is trained with the model and imaginary dataset m times in each iteration of the algorithm . Secondly , we derive our algorithm in the framework of the maximum entropy reinforcement learning . The maximum entropy updates could improve the robustness under the model estimation error ( Ziebart et al. , 2010 ) . In addition , it encourages the exploration , prevents the early convergence to the sub-optimal policies , and shows state-of-the-art performance in MFRL ( Haarnoja et al. , 2018 ) . Thirdly , S2VG avoids the importance sampling in the off-policy setting by sampling the action from π and transition from f ( s , a ) , which further reduces the variance of the gradient estimation . CONTRIBUTIONS We derive an elegant , sample-efficient , and computational-friendly Dyna-style MBRL algorithm in the framework of the maximum entropy reinforcement learning with a principled way . Different from the traditional MBRL algorithm , we directly embed the model in the policy improvement , which could reduce the variance in the gradient estimation and avoid the computation on huge imaginary dataset . In addition , since the algorithm is off-policy , it is sample-efficient . At the same time , the maximum entropy principle encourages exploration and improves performance , which has been observed in MFRL ( Haarnoja et al. , 2017 ) . We test our algorithm on several benchmark tasks in Mujoco simulation environment ( Todorov et al. , 2012 ) and demonstrate that our algorithm can achieve state-of-the-art performance 1 . 2 PRELIMINARIES . In this section , we first present some backgrounds on the Markov decision process . Then we introduce the knowledge on the maximum entropy reinforcement learning ( Ziebart et al. , 2008 ) and stochastic value gradient ( Heess et al. , 2015 ) since parts of them are the building blocks of our algorithm . 2.1 MDP . Markov Decision Process ( MDP ) can be described by a 5-tuple ( S , A , R , P , γ ) : S is the state space , A is the action space , P = ( P ( s′|s , a ) ) s , s′∈S , a∈A are the transition probabilities , R = ( r ( s , a ) ) s , s′∈S , a∈A are the real-valued immediate rewards , and γ ∈ ( 0 , 1 ) is the discount factor . A policy is used to select actions in the MDP . In general , the policy is stochastic and denoted by π , where π ( at|st ) is the conditional probability density at at associated with the policy . The state value evaluated on policy π could be represented by V π ( s ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) |s0 = s ] on immediate reward return r = ( R ( s , a ) ) s , s′∈S , a∈A with discount factor γ ∈ ( 0 , 1 ) along the horizon 1Code is submitted anonymously at https : //github.com/S2VG-anonymous1/S2VG t. The state-action value evaluated on policy π represent the expected return on the specific action a : Qπ ( s , a ) = r ( s0 , a0 ) + Eπ [ ∑∞ t=1 γ tr ( st , at ) |s0 = s , a0 = a ] . 2.2 MAXIMUM ENTROPY REINFORCEMENT LEARNING . Maximum entropy reinforcement learning augments the reward with an entropy term , such that the optimal policy aims to maximize the new reward function at each visited state : max π J ( π ) = T∑ t=0 E ( st , at ) ∼ρπ [ r ( st , at ) + αH ( π ( ·|st ) ) ] , ( 1 ) where H ( π ( ·|st ) ) is an entropy term scaled by α ( Ziebart et al. , 2008 ) . The optimal policy in equation 1 can be obtained by the following soft-Q update ( Fox et al. , 2016 ) , Q ( st , at ) ←− r ( st , at ) + γEst+1∼p [ V ( st+1 ) ] and V ( st ) ←− α log ( ∫ A exp ( 1 α Q ( st , at ) ) dat ) . Above iterations define the soft Q operator , which is a contraction . The optimal policy π∗ ( a|s ) can be recovered by π ? ( at|st ) = exp ( 1αQ ∗ ( st , at ) ) ∫ A exp ( 1 αQ ∗ ( st , at ) ) dat , whereQ∗ is the fixed point of the soft-Q update . We refer readers to the work ( Ziebart et al. , 2008 ; Haarnoja et al. , 2017 ) for more discussions . In soft actor-critic ( Haarnoja et al. , 2018 ) , the optimal policy π∗ ( at|st ) is approximated by a neural network πθ ( at|st ) , which is solved by the following optimization problem max πθ ( at|st ) Est∼p ( st ) Eat∼πθ ( at|st ) [ Q ( st , at ) − α log πθ ( at|st ) ) ] . 2.3 STOCHASTIC VALUE GRADIENT . Stochastic value gradient method is a model-based algorithm which is designed to avoid the compounded model errors by only using the real-world observation and gradient information from the model ( Heess et al. , 2015 ) . The algorithm directly substitutes the dynamics model and reward model in the Bellman equation and calculates the gradient . To perform the backpropagation in the stochastic Bellman equation , re-parameterization trick is applied to evaluate the gradient on real-world data . The stochastic policy π ( a|s ; θ ) with parameter θ could be optimized by the policy gradient in the following way ∂V ( s ) ∂θ ≈ Eη , ζ [ ∂r̂ ( s , a ) ∂a ∂π ( a|s ) ∂θ + γ ( ∂V ′ ( s′ ) ∂s′ ∂f ( s , a ) ∂a ∂π ( a|s ) ∂θ ) ] , ( 2 ) where η and ζ are the policy and environment re-parameterization noise which could be directly sampled from a prior distribution or inferred from a generative model g ( η , ζ|s , a , s′ ) . The f ( s , a ) and r̂ ( s , a ) are dynamics model and reward model respectively . 3 SOFT STOCHASTIC VALUE GRADIENT METHOD . In this section , we introduce our algorithm soft stochastic value gradient ( S2VG ) method . Our objective is to design an off-policy MBRL algorithm under the maximum entropy framework , which could improve the exploration , enhance the training robustness through entropy maximization . Particularly , we optimize the following equation J ( θ ) = T∑ t=0 Eρ̂π ( τ ) [ r̂ ( st , at ) +H ( π ( at|st ) ) ] , ( 3 ) where we omit the regularizer parameter α of the entropy term in the following discussion to ease the exposition . ρ̂π = p ( s0 ) ∏T t=0 f ( st , at ) π ( at|st ) , f ( st , at ) is the learned dynamics model , and r̂ is the reward model . We then derive the update rule following the similar step in the probabilistic reinforcement learning ( Levine , 2018 ) , which generally includes policy evaluation and policy improvement steps . Similar to the policy evaluation without the entropy term , we have the soft value function update V ( st ) = Eπ ( at|st ) [ r̂ ( st , at ) − log π ( at|st ) + γEs∼fV ( st+1 ) ] . For convenience , we define the Q function in the following , Q ( st , at ) = r̂ ( st , at ) + γEst+1∼f [ V ( st+1 ) ] . ( 4 ) The value function update can be reformulated as V ( st ) = Eπ ( at|st ) [ Q ( st , at ) − log π ( at|st ) ] . ( 5 ) The optimal policy ( policy improvement step ) at each step t is π ( at , st ) = exp ( Q ( st , at ) ) ∫ a exp ( Q ( st , at ) dat . The derivation is almost the same with ( Levine , 2018 ; Haarnoja et al. , 2018 ) , expect that we use learned dynamics model and reward function here . Notice this optimal policy can be approximated by a parametric function πθ ( at|st ) and obtained by solving maxπθ ( at|st ) Est∼p ( st ) Eat∼πθ ( at|st ) [ Q ( st , at ) − log πθ ( at|st ) ) ] . However such way used in the MFRL can not leverage the model information . We leave the our derivation and discussion on the policy improvement in section 3.3 .

Summary: This paper proposes an off-policy model-based reinforcement learning approach. The proposed paper combines several techniques together. The general framework fits a maximum entropy reinforcement learning, and the policy and q-function updates are formulated in a soft actor-critic manner. To leverage the learned dynamics, the proposed approach adopts value expansion to substitute the learned reward function and forward dynamics function in the Bellman equation for the value function. Also, the imagined roll-out could be used to update the q-function.