Transfer Value or Policy? A Value-centric Framework Towards Transferrable Continuous Reinforcement Learning

1 INTRODUCTION . While the achievements of deep reinforcement learning ( DRL ) are exciting in having conquored many computer games ( Go ( Silver et al. , 2016 ; 2017 ) , Atari games ( Mnih et al. , 2015 ) ) , in practice , it is still hard for these algorithms to find applications in real-world environments . Among the impediments , a primary obstacle is that they could fail even if the environment to deploy the agent is slightly different from where they were trained ( Tan et al. , 2018 ; Bousmalis et al. , 2018 ; Tobin et al. , 2017 ; Tamar et al. , 2016 ) . In other words , they lack the desired ability to transfer experiences learned from one environment to another . These observations motivate us to ask , ( 1 ) why are current RL algorithms so inefficient in transfer learning and ( 2 ) what kind of RL algorithms could be friendly to transfer learning by nature ? In this work , we explore the two questions and present a partial answer based on analysis and experiments . Our exploration concentrates on control tasks due to its broad impact ; in particular , we further assume that across environments only their dynamics are not the same . Possible sources of such dynamics discrepancy could be variation of physical properties such as object mass , gravity , and surface friction . It is worth noting that our framework is general and we do not assume any specific perturbation source or type . Our investigation starts with understanding the limitation of transferring the policy function ( which maps a state to a distribution of actions ) across environments . We analyze this transfer strategy because the de facto DRL framework in the control domain ( DDPG ( Lillicrap et al. , 2015 ) , TRPO ( Schulman et al. , 2015 ) , A3C ( Mnih et al. , 2016 ) , etc . ) are policy-centric methods , which directly optimize the policy function . As a result , these methods learn a precise policy function , and sometimes also produce an imprecise value/Q-function as a side product . However , even if a perfect policy function has been learned from the source environment , this policy could behave quite poorly , or even fail , in the new environment , especially when the action space is hard-constrained ( e.g. , force or torque usually has a maximal value ) . We illustrate this by a simple example : Imagine a child shooting three-pointers when playing basketball . With a 600g ball , she can make the three-pointer . However , she may hardly make the three with a 800g ball because it is too heavy . What will she do ? Most likely she will step forward , approach the basket , and make a closer jump-shot . We see that marginal dynamics variation can lead to drastic policy change , and direct policy optimization initialized from the old policy would not be efficient . We will analyze this issue more systematically by theoretical and experimental approaches throughout the paper . The investigation implies that , instead of directly transferring policies , the swift transfer should be grounded in richer and more structured knowledge of the task , so as to facilitate the judgment of whether the agent is approaching the goal , which is critical for making the right decision . Enlightened by the above intuition , we propose a simple model-based and value-centric framework for continuous reinforcement learning . Our method contains two disentangled components : a dynamics approximator ( model-based ) and a state value function approximator ( value-centric ) . The agent plans its action by solving an optimization problem using both approximators . As side products , this design learns a precise transition function , a precise reward function , and a precise value function on a subset of states . In particular , knowledge from historical explorations have been stored in the value function . In comparison , previous policy-centric methods can only produce a precise policy function , thus our framework allows to transfer much more information . By fine-tuning the whole framework in a new environment , our agent can adapt quickly with much lower sample complexity than state-of-the-art . We call our method value-centric because it strives to learn a precise value function . The general framework is inspired from the Value Iteration ( VI ) method , which is a classical approach for discrete decision making . However , since control problems have a continuous action space , we can not directly enumerate over the action space as in the discrete setting but have to address the highly non-convex optimization problem . To make it tractable , we leverage differentiable function approximators like neural networks to learn the dynamics and the value function . By such an approximation , it is possible to solve the optimization problem with state-of-the-art optimizers effectively . We also theoretically analyze our value-centric framework and classical policy gradient algorithms from an optimization perspective . To build an intuitive understanding , we create a simple and illustrative example that clearly shows a local optimum in the policy space can prevent policy gradient methods from transferring successfully , but will not affect our value-centric framework . We summarize our contributions as below : • We provide a theoretical justification to show the advantage of value-centric methods from an optimization perspective . • We propose a novel value-centric framework for continuous reinforcement learning of comparable sample efficiency with popular deep RL methods in the training from scratch setting . • Extensive experiments show the superiority of our method in a transfer learning setting 2 BACKGROUND . 2.1 MDPS AND NOTATION . First , we introduce the standard reinforcement learning setting for continuous control . We assume the underlying control problem is a Markov Decision Process ( MDP ) , which is defined by a tuple 〈S , A , T , R , ρ0 , γ〉 . Here , S and A are the continuous state space and action space , respectively , T is the transition dynamics , R is the reward function , ρ0 is the distribution of the initial state , and γ ∈ ( 0 , 1 ] is the discounting factor for future rewards . For robot control tasks , it is reasonable to assume a deterministic transition dynamics T : S × A → S , and a deterministic reward function R : S ×A → R. The policy π of an agent maps the state space to a probability distribution over the action space π : S → P ( A ) . The goal of reinforcement learning is to find an optimal policy π∗ that maximizes the expectation of the accumulated future rewards according to the initial state distribution J ( π ) = Eρ0 , π [ ∑∞ t=0 γ tR ( st , at ) ] . We also define the value function V of a state s as V ( s ) = Eπ [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] . Finally , the optimal value function V ∗ satisfies V ∗ ( s ) = maxπ Eπ [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] for any s ∈ S. Low-dimensional Assumption We focus on control problems which usually have well-engineered and low-dimensional state/action representations . Not rigorously , the assumption has two implications : • Property 1 : For a smooth function f ( at ) over A , we can find its approximate solution by sampling over the domain and optimizing locally ; • Property 2 : We can learn a network to approximate the transition and reward functions . Empirically , we find evidence of both properties as in our experiment section ( Sec 6 ) . 2.2 TRANSFER LEARNING . Many differently posed transfer learning problems have been discussed in the reinforcement learning literature ( Taylor & Stone , 2009 ) . In this work , we study the problem of the environment slightly changing while the task remains the same . For example , in the pendulum swing-up problem , once the agent learns how to swing up a 1kg pendulum , we expect that it could quickly adapt itself to swing up a 2kg pendulum leveraging the learned knowledge . We formulate our setting by modifying the aforementioned standard RL setting . We consider a pair of MDPs sharing state and action spaces . Their transition dynamics T and reward functions R are parameterized by a vector ξ : st+1 = T ( st , at ; ξ ) , rt = R ( st , at ; ξ ) . Each ξ defines a unique MDPMξ . The change ofR is only caused by the change of T ( instead of the goal of the task ) , so the change of reward function is limited , though we parameterize it by ξ for rigor . After the agent learns how to perform well on a source MDP , we expect it to solve the target MDP using few interactions with the new environment . 3 WHY TRANFERRING POLICY CAN BE DIFFICULT ? A SIMPLE ILLUSTRATION . In the RL community , most control tasks are solved by policy gradient-based methods . In this section , we illustrate the limitation of transferring policy by a simple example and compare it with a value-centric method . ( 𝑎 ) ( 𝑏 ) state , it receives a reward r = 1 − |a| . Every time the agent visits the failure state , it will be punished by a negative reward r = −5 . The optimal policy for this task is a∗0 = −f1 if f1 < f2 ; a∗0 = f2 if f1 ≥ f2 ; a∗1 ( s1 ) = −f1 ; a∗1 ( s2 ) = f2 . Now we show the different behaviors of the value- and policy-centric strategies in a transfer learning setting . For the source environment , f1 = 0.7 and f2 = 0.8 . It is not hard to find that the optimal path is path1 . Then we modify the transition rule of the environment by slightly varying f1 and f2 . We set f1 = 0.8 and f2 = 0.7 . While the variation of the environment is relatively small , the optimal policy for s0 is completely different and has changed to path2 . The optimal state value function and policy are shown in Table 1 in the Appendix B . To compare policy- and value-centric strategies , we run two representative algorithms on this game – the Policy Gradient ( PG ) algorithm ( Williams , 1992 ) ( policy-centric ) and the Value Iteration ( VI ) algorithm ( Sutton et al. , 1998 ) ( value-centric ) . We assume the value-centric methods have access to the oracle transition rules and the reward function , and we parameterize the policy π ( si ) with Gaussian distribution . Details can be found in Appendix B . We compare the behaviors of PG and VI to intuitively demonstrate their characteristics . PG gets stuck at the old optimum ( Fig . 2 left ) since the distance between the old optimum and the new optimum is so large . To see why , we first point out that PG optimizes in the policy space , restricting that the policy shifting from the old optimum to the new optimum must be quite “ continuous ” . However , to reach the new optimum , π ( s0 ) must flip the sign of its mean µ ( s0 ) . Unfortunately , when µ ( s0 ) approaches zero from the negative side ( from −0.7 ) , its transition possibility to the failure state will increase , thus µ ( s0 ) will bounce away from 0− to avoid punishment . Consequently , it is very unlikely to successfully reach the new optimum , a positive value . In contrast , the value function in VI algorithm continuously shifts and converges to the new optimum ( See Appendix B ) and the policy deduced from the value function converges to the optimal policy in 10 episodes . Value function reuses the learned information more efficiently . It also allows sudden change of the policy ( Fig . 2 right ) , which could be beneficial for transferring .

Summary: The paper proposes a model-based value-centric (MVC) deep RL algorithm for transfer learning. The algorithm optimizes neural networks to estimate the deterministic transitions and rewards, and uses the these models to learn a value function by minimizing the Bellman residual. Policy is represented implicitly as the action that greedily maximizes the return, expressed in terms of the learned models. The experiments show some improvement on transferability over DDPG and TRPO policies.