Title: TRANSFER VALUE OR POLICY? A VALUE-CENTRIC FRAMEWORK TOWARDS TRANS-FERRABLE CONTINUOUS REINFORCEMENT LEARNING

Abstract: Transferring learned knowledge from one environment to another is an important step towards practical reinforcement learning (RL). In this paper, we investigate the problem of transfer learning across environments with different dynamics while accomplishing the same task in the continuous control domain. We start by illustrating the limitations of policy-centric methods (policy gradient, actorcritic, etc.) when transferring knowledge across environments. We then propose a general model-based value-centric (MVC) framework for continuous RL. MVC learns a dynamics approximator and a value approximator simultaneously in the source domain, and makes decision based on both of them. We evaluate MVC against popular baselines on 5 benchmark control tasks in a training from scratch setting and a transfer learning setting. Our experiments demonstrate MVC achieves comparable performance with the baselines when it is trained from scratch, while it significantly surpasses them when it is used in the transfer setting.

1. INTRODUCTION:

While the achievements of deep reinforcement learning (DRL) are exciting in having conquored many computer games (Go (Silver et al., 2016; 2017) , Atari games (Mnih et al., 2015) ), in practice, it is still hard for these algorithms to find applications in real-world environments. Among the impediments, a primary obstacle is that they could fail even if the environment to deploy the agent is slightly different from where they were trained (Tan et al., 2018; Bousmalis et al., 2018; Tobin et al., 2017; Tamar et al., 2016) . In other words, they lack the desired ability to transfer experiences learned from one environment to another. These observations motivate us to ask, (1) why are current RL algorithms so inefficient in transfer learning and (2) what kind of RL algorithms could be friendly to transfer learning by nature?

In this work, we explore the two questions and present a partial answer based on analysis and experiments. Our exploration concentrates on control tasks due to its broad impact; in particular, we further assume that across environments only their dynamics are not the same. Possible sources of such dynamics discrepancy could be variation of physical properties such as object mass, gravity, and surface friction. It is worth noting that our framework is general and we do not assume any specific perturbation source or type.

Our investigation starts with understanding the limitation of transferring the policy function (which maps a state to a distribution of actions) across environments. We analyze this transfer strategy because the de facto DRL framework in the control domain (DDPG (Lillicrap et al., 2015) , TRPO (Schulman et al., 2015) , A3C (Mnih et al., 2016) , etc.) are policy-centric methods, which directly optimize the policy function. As a result, these methods learn a precise policy function, and sometimes also produce an imprecise value/Q-function as a side product. However, even if a perfect policy function has been learned from the source environment, this policy could behave quite poorly, or even fail, in the new environment, especially when the action space is hard-constrained (e.g., force or torque usually has a maximal value). We illustrate this by a simple example: Imagine a child shooting three-pointers when playing basketball. With a 600g ball, she can make the three-pointer. However, she may hardly make the three with a 800g ball because it is too heavy. What will she do? Most likely she will step forward, approach the basket, and make a closer jump-shot. We see that marginal dynamics variation can lead to drastic policy change, and direct policy optimization initialized from the old policy would not be efficient. We will analyze this issue more systematically by theoretical and experimental approaches throughout the paper.

The investigation implies that, instead of directly transferring policies, the swift transfer should be grounded in richer and more structured knowledge of the task, so as to facilitate the judgment of whether the agent is approaching the goal, which is critical for making the right decision. Enlightened by the above intuition, we propose a simple model-based and value-centric framework for continuous reinforcement learning.

Our method contains two disentangled components: a dynamics approximator (model-based) and a state value function approximator (value-centric). The agent plans its action by solving an optimization problem using both approximators. As side products, this design learns a precise transition function, a precise reward function, and a precise value function on a subset of states. In particular, knowledge from historical explorations have been stored in the value function. In comparison, previous policy-centric methods can only produce a precise policy function, thus our framework allows to transfer much more information. By fine-tuning the whole framework in a new environment, our agent can adapt quickly with much lower sample complexity than state-of-the-art.

We call our method value-centric because it strives to learn a precise value function. The general framework is inspired from the Value Iteration (VI) method, which is a classical approach for discrete decision making. However, since control problems have a continuous action space, we cannot directly enumerate over the action space as in the discrete setting but have to address the highly non-convex optimization problem. To make it tractable, we leverage differentiable function approximators like neural networks to learn the dynamics and the value function. By such an approximation, it is possible to solve the optimization problem with state-of-the-art optimizers effectively.

We also theoretically analyze our value-centric framework and classical policy gradient algorithms from an optimization perspective. To build an intuitive understanding, we create a simple and illustrative example that clearly shows a local optimum in the policy space can prevent policy gradient methods from transferring successfully, but will not affect our value-centric framework.

We summarize our contributions as below:

• We provide a theoretical justification to show the advantage of value-centric methods from an optimization perspective.

• We propose a novel value-centric framework for continuous reinforcement learning of comparable sample efficiency with popular deep RL methods in the training from scratch setting.

• Extensive experiments show the superiority of our method in a transfer learning setting 2 BACKGROUND

2.1. MDPS AND NOTATION:

First, we introduce the standard reinforcement learning setting for continuous control. We assume the underlying control problem is a Markov Decision Process (MDP), which is defined by a tuple S, A, T , R, ρ 0 , γ . Here, S and A are the continuous state space and action space, respectively, T is the transition dynamics, R is the reward function, ρ 0 is the distribution of the initial state, and γ ∈ (0, 1] is the discounting factor for future rewards. For robot control tasks, it is reasonable to assume a deterministic transition dynamics T : S × A → S, and a deterministic reward function R : S × A → R.

The policy π of an agent maps the state space to a probability distribution over the action space π : S → P(A). The goal of reinforcement learning is to find an optimal policy π * that maximizes the expectation of the accumulated future rewards according to the initial state distribution

J(π) = E ρ0,π [ ∞ t=0 γ t R(s t , a t )]. We also define the value function V of a state s as V (s) = E π [ ∞ t=0 γ t R(s t , a t )|s 0 = s]. Finally, the optimal value function V * satisfies V * (s) = max π E π [ ∞ t=0 γ t R(s t , a t )|s 0 = s] for any s ∈ S.

Low-dimensional Assumption We focus on control problems which usually have well-engineered and low-dimensional state/action representations. Not rigorously, the assumption has two implications:

• Property 1: For a smooth function f (a t ) over A, we can find its approximate solution by sampling over the domain and optimizing locally; • Property 2: We can learn a network to approximate the transition and reward functions.

Empirically, we find evidence of both properties as in our experiment section (Sec 6).

2.2. TRANSFER LEARNING:

Many differently posed transfer learning problems have been discussed in the reinforcement learning literature (Taylor & Stone, 2009) . In this work, we study the problem of the environment slightly changing while the task remains the same. For example, in the pendulum swing-up problem, once the agent learns how to swing up a 1kg pendulum, we expect that it could quickly adapt itself to swing up a 2kg pendulum leveraging the learned knowledge. We formulate our setting by modifying the aforementioned standard RL setting. We consider a pair of MDPs sharing state and action spaces. Their transition dynamics T and reward functions R are parameterized by a vector ξ: s t+1 = T (s t , a t ; ξ), r t = R(s t , a t ; ξ). Each ξ defines a unique MDP M ξ . The change of R is only caused by the change of T (instead of the goal of the task), so the change of reward function is limited, though we parameterize it by ξ for rigor. After the agent learns how to perform well on a source MDP, we expect it to solve the target MDP using few interactions with the new environment.

3. WHY TRANFERRING POLICY CAN BE DIFFICULT? A SIMPLE ILLUSTRATION:

In the RL community, most control tasks are solved by policy gradient-based methods. In this section, we illustrate the limitation of transferring policy by a simple example and compare it with a value-centric method. We design an MDP with 5 states as in Fig. 1 : an initial state (s 0 ), two intermediate states (s 1 , s 2 ), a goal state (s g ), and a failure state (s f ). s g and s f are terminal states, so their state values are both 0. The task for the agent is to reach the goal state through one of the two paths. We denote the path passing s 1 as path 1 , and the path passing s 2 as path 2 . Let f i > 0 be environment parameters. At some state s, the agent takes an action a ∈ [-1, 1]. The reward is designed in a goal-based style with penalties to large |a|. If the agent transits from the initial state to an intermediate state, it receives a reward r = 1 -|a|. Every time the agent visits the failure state, it will be punished by a negative reward r = -5. The optimal policy for this task is Now we show the different behaviors of the value-and policy-centric strategies in a transfer learning setting. For the source environment, f 1 = 0.7 and f 2 = 0.8. It is not hard to find that the optimal path is path 1 . Then we modify the transition rule of the environment by slightly varying f 1 and f 2 . We set f 1 = 0.8 and f 2 = 0.7. While the variation of the environment is relatively small, the optimal policy for s 0 is completely different and has changed to path 2 . The optimal state value function and policy are shown in Table 1 in the Appendix B.

a * 0 = -f 1 if f 1 < f 2 ; a * 0 = f 2 if f 1 ≥ f 2 ; a * 1 (s 1 ) = -f 1 ; a * 1 (s 2 ) = f 2 .

To compare policy-and value-centric strategies, we run two representative algorithms on this gamethe Policy Gradient (PG) algorithm (Williams, 1992) (policy-centric) and the Value Iteration (VI) algorithm (Sutton et al., 1998) (value-centric) . We assume the value-centric methods have access to the oracle transition rules and the reward function, and we parameterize the policy π(s i ) with Gaussian distribution. Details can be found in Appendix B. We compare the behaviors of PG and VI to intuitively demonstrate their characteristics. PG gets stuck at the old optimum (Fig. 2 left) since the distance between the old optimum and the new optimum is so large. To see why, we first point out that PG optimizes in the policy space, restricting that the policy shifting from the old optimum to the new optimum must be quite "continuous". However, to reach the new optimum, π(s 0 ) must flip the sign of its mean µ(s 0 ). Unfortunately, when µ(s 0 ) approaches zero from the negative side (from -0.7), its transition possibility to the failure state will increase, thus µ(s 0 ) will bounce away from 0to avoid punishment. Consequently, it is very unlikely to successfully reach the new optimum, a positive value.

In contrast, the value function in VI algorithm continuously shifts and converges to the new optimum (See Appendix B) and the policy deduced from the value function converges to the optimal policy in 10 episodes. Value function reuses the learned information more efficiently. It also allows sudden change of the policy (Fig. 2 right), which could be beneficial for transferring.

4. MODEL-BASED AND VALUE-CENTRIC (MVC) REINFORCEMENT LEARNING 4.1 VALUE-CENTRIC METHOD IN TRANSFER LEARNING:

Our objective is to make the agent learn faster in a new environment after trained in a similar environment. From the above discussion, we know that instead of directly transferring policies, the swift transfer should be grounded in richer and more structured knowledge of the task. It is worth exploring transfer by value function since the value function contains more information than the policy alone. A straight-forward idea is to utilize the value/Q function from actor-critic algorithms such as DDPG to facilitate transfer learning. However, the value/Q function learned from the actorcritic algorithm is usually imprecise, and it does not capture so much knowledge in the original environment. To address this issue, we propose an algorithm to directly train a precise value function in the original environment. We call it a value-centric method. Then, we just need to fine-tune this value function to help the agent adapt to the new environment. In the following subsections, we explain how to train this precise value function from scratch.

4.2. CONTINUOUS VALUE ITERATION:

To better explain our main algorithm, we first propose a Continuous Value Iteration algorithm, which is possible to make use of Property 1 in Sec 2.1 to build a value-centric method. Theorem 2 in the Appendix suggests it always converges. Definition 1. At the i-th iteration, we update the value function by the following rule:

V i+1 (s) = (1 -α)V i (s) + α(max a [R(s, a, T (s, a)) + γV i (T (s, a))]), ∀s ∈ S

(1) where α ∈ (0, 1] is the step size, and V 0 (s) is some initialization value. Note that by Property 1 in Sec 2.1, the maximization algorithm over a is accessible. We call this algorithm Continuous Value Iteration.

4.3. ALGORITHM:

The CVI algorithm mentioned in Sec 4.2 assumes the oracle of dynamics, rewards, and an algorithm to solve the optimization problem w.r.t. a. In addition, it assumes the ability to sweep over the state space. To make it practical, we propose a model-based and value-centric(MVC) framework. We approximate the value function V , transition dynamics T , and the reward function R by function approximators like neural networks. Any differentiable function approximators not restricted to neural networks may be applicable. Let us denote the three parameterized function approximators we use as f V (s; θ), f T (s, a; φ), f R (s, a; ψ), where θ, φ, and ψ are parameters. To efficiently use the sampled data, we store the agent's experience (s t , a t , r t , s t+1 ) in a replay buffer B. Then f T is trained to minimize the following L 2 loss function: Update f V (s; θ) with Eq. 5.

L T = 1 |D| (st,at,st+1)∈D f T (s t , a t ; φ) -(s t+1 -s t ) 2 (2)

7::

Update f T (s, a; φ), f R (s, a; ψ) with Eq. 2 and Eq. 3

8: Soft update target network f V (s; θ -): θ -= αθ + (1 -α)θ - 9: end for

Here, D is the training dataset extracted from B and | • | refers to the cardinality of a set. Like previous works (Kurutach et al., 2018; Nagabandi et al., 2017) , the supervision signal for f T is the increment of the state to release its burden of memorizing the last state. We also train the reward approximator in a similar supervised learning way:

L R = 1 |D| (st,at,rt)∈D f R (s t , a t ; ψ) -r t 2 (3)

At time step t, the agent samples data under a deterministic policy:

EQUATION

Unlike the tabular case, the right-hand side of Eq. 4 is a complicated non-convex optimization problem. Fortunately, we can take the advantage that f V (s; θ), f T (s, a; φ), f R (s, a; ψ) are differentiable w.r.t the action vector a. We use Adam optimizer (Kingma & Ba, 2015) to solve the r.h.s optimization problem. To make sure that the optimizer finds a local maximum that is good enough to approximate the global maximum, we randomly select multiple initializations. Due to the superior parallel computing ability of modern GPUs, the optimization with different initial seeds can be operated simultaneously so that additional time consumption is limited. For exploration purpose, we also add noise (e.g. Ornstein-Uhlenbeck process) to the optimal action. While it is possible to expand the r.h.s. to a multi-step planning, notoriously, a learned model is inclined to diverge in long-horizon predictions. Consequently, we only use the one-step version for policy search.

To approximate the value function, we update the value approximator by the supervision of temporaldifference error (TD-error). Since updating the value function across the whole state space is unrealistic, we update the value approximator only by the data sampled from the environment in the current time step like policy gradient algorithms. Suppose at timestep t, the agent takes action a t (following Eq. 4), then receives reward r t and transits to state s t+1 . We minimize the following loss:

EQUATION

)

Online training makes the whole framework more light-weight while sacrificing the guarantee of convergence. One can improve the training paradigm with past experience from the replay buffer or advanced sampling skills on the state space, but we leave them as future work.

Like DDPG, we employ a target network parameterized by θ -to stabilize the training. To speed up computation, we also fork multiple agents to run in parallel and synchronize them by a global agent. Algotihm 1 recaptures the whole algorithm paradigm.

5. A THEORETICAL JUSTIFICATION FROM THE OPTIMIZATION PERSPECTIVE:

We will show the limitation of policy-centric methods and the nice properties of value-centric method.

We narrow the analysis of policy-centric frameworks down to the landscape of J(π). To save space, we leave the theorems and detailed proofs in Appendix A. We explain the intuitions here:

First, we show in Theorem 1 that, for an arbitrary MDP with a deterministic transition function, a local optimum of J(π) that traps gradient-based methods could exist under a weak condition. In fact, the local optimum issue is introduced by the parameterization of π's output space, e.g., a Gaussian distribution. If we allow π to be an arbitrary distribution, the local optimum will vanish (Proposition 1). The example in Sec 3 exactly shows a failure case of transferring policy due to a local optimum.

Second, we show in Theorem 2 that, for the same type of MDPs, the Continuous Value Iteration algorithm leveraging Property 1 in Sec 2.1 can share the favorable converge property of classical value iteration. That is, the distance between a current value function and the optimal value function will be squeezed at linear rate, thus it always converges. In addition, Proposition 2 implies that a small perturbation to the environment is only likely to cause a marginal change to the optimal value function; therefore, the old value function would serve as a good initialization point.

6. EXPERIMENTS:

We first compare our value-centric method against the policy gradient baselines in the training from scratch setting. We also show that our value-centric method beats baselines in the transfer learning setting. Finally, we conduct ablation study over method parameters and diagnose the components.

6.1. SETUP:

We evaluate our algorithm and two prevalent continuous RL methods, Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) and Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) , on five control tasks in the OpenAI Gym (Brockman et al., 2016) :HalfCheetah-v1,InvertedPendulum-v1, InvertedDoublePendulum-v1, Pendulum-v0, and Reacher-v1. For baselines, we use the code from OpenAI Baselines (Dhariwal et al., 2017) and fine-tuned their hyper-parameters as much as we can. More details can be found in the Appendix D.

6.2. MAIN RESULTS:

We compare the sample complexity and the performance of the three algorithms (MVC, DDPG, and TRPO) on the aforementioned five environments, for both the training from scratch setting and transfer learning setting. We also compare with PPO (Schulman et al., 2017) and SQL (Haarnoja et al., 2017) on HalfCheetah and show the results in Appendix F.

Train from Scratch The reported results (Fig. 8 (a)) show the mean and standard deviation of 3 runs with different random seeds. In four of the environments (InvertedPendulum-v1, HalfCheetah-v1, Pendulum-v0, Reacher-v1), our method achieves comparable performance as the better baseline.

In InvertedDoublePendulum-v1, though there is a significant gap between MVC and TRPO, MVC performs at the same level with DDPG.

Transfer across environments We demonstrate the superiority of our method when transferring across environments. For each of the above five environments, we change one or several physical properties, like the mass of pendulum, to create two new environments with novel dynamics. In one of the new environments, the change is relatively small (we call it 'Hard' in the plot), while the other is more intensively perturbed (we call it 'Harder' in the plot). We first train standard agents on the original environment. For fair comparison, we pick the agents that achieve comparable performance for all methods in the original environment. Please refer to Appendix D for the details of the modification and the agents. To avoid the possibility of under-exploration in the new environments, we reset the exploration noise of all the algorithms. We directly fine-tune all the agents with the same number of simulation steps. The results are shown in Fig. 8(b ).

On all the environments, we observed that TRPO has the worst overall transfer performance. DDPG and MVC have similar transferrability on simple tasks like Reacher-v1 and Pendulum-v0. However, on more complicated tasks like HalfCheetah-v1,the performance of DDPG is significantly worse than MVC. Further investigation shows that DDPG can actually learn a high-quality Q function for simple environments, which serves a similar role as our value function. However, on more challenging games such as HalfCheetah-v1 and InvertedDoublePendulum-v1, as a policy-centric algorithm, the learned Q function is far from the true one (Fig. 7 in Appendix E), thus the transfer is significantly slower. The success and failure of DDPG again shows the central role value plays in transfer learning. 

HalfCheetah-v1:

Validation of Property 1 Empirically, we find evidence for Property 1 and 2. Take HalfCheetah-v1 for example. For Property 1, we compare the optimization result of gradient-based method against random sampling with 10 6 points. Figure on the left demonstrates that Adam achieves comparable results with random sampling while being tens of times faster on our computer (a 48-core Xeon CPU with 2 Titan XP GPUs). Validation for Property 2 Figure on the right shows that the loss functions for the transition network and reward network converge in less than 100k time steps, which means the transition network and reward network converges much faster than the value network (As shown in Fig. 8 (a), the value network still does not converge after 1200k steps.). Therefore, selecting actions based on the learned transition network and reward network is trusty. We also observed that the learned transition and reward networks provide good start points for the training in the new environment. Hyperparameters We verify the influence of the initialization and the number of optimization steps in the action search process. Fig. 4 shows that calling Adam optimizer is critical to stabilize the training. In transfer learning scenario (Fig. 5 ), the optimization shows a more significant impact on the performance and sample complexity. The agent that searches action with 200 initial points and 4 optimization steps is enough to win the agent using only 1000 initial points by a large margin.

7. RELATED WORK:

Reinforcement Learning in Continuous Control Domain. Solving continuous control problems through reinforcement learning has been studied for decades (Sutton et al., 1998; Williams, 1992) .

Policy-based methods (Schulman et al., 2015; Mnih et al., 2016; Lillicrap et al., 2015) are more widely used. One exception is the NAF (Gu et al., 2016) method under the Q-learning framework which models the action space as a quadratic function.

Value-based Reinforcement Learning. The most relevant work in literature are perhaps the very recent Value Iteration Network (VIN) and Value Prediction Network (VPN) (Tamar et al., 2016; Oh et al., 2017) . Though demonstrated better environment generalizability, VIN is specifically designed and only evaluated on the 2D navigation problem. VPN learns a dynamics model together with a value function and makes plans based on Monte Carlo tree search. In contrast to our work, VPN neither considered the continuous control problem nor thoroughly investigated their algorithm under the transfer learning setting.

Model-based Reinforcement Learning. For purposes such as increasing sample efficiency and designing smarter exploration strategies (e.g., curiosity-driven exploration), it can be beneficial to incorporate a learned dynamics model. Some very recent works have demonstrated the power of such model-based RL algorithms (Levine & Koltun, 2013; Nagabandi et al., 2017; Kurutach et al., 2018; Pathak et al., 2018; Feinberg et al., 2018; Pathak et al., 2017) . However, to our knowledge, none of them has yet combined the value function with a learned dynamics model to solve continuous decision making problems.

Transfer Learning in Deep Reinforcement Learning. In this work, we study knowledge transfer problem across different MDPs. (Kansky et al., 2017) proposed the SchemaNetwork which learns the knowledge of the Atari physics engine by playing a standard version of the BreakOut game. (Higgins et al., 2017) learns disentangled representations in the source domains to achieve zero-shot domain adaption in the new environments. Finally, a straight-forward strategy is to show the agent all possible environments (Yu et al., 2017; Tan et al., 2018; Tobin et al., 2017) .

APPENDIX A PROOFS FOR THE THEOREMS:

For mathematical rigor, we suppose the state space and the action space are both continuous in this section.

A.1 PROPOSITION 1 AND PROOF Proposition 1. If J is the expected return defined as in Section 2, π can be an arbitrary function defined on the probability simplex ( π(τ )dx = 1, π(τ ) ≥ 0, where τ is a trajectory from the starting state to a terminal state), and π * is the maximizer of J(π), then ∀π 0 , there exists a path connecting π 0 to π * that J(π) monotonically increases, which implies that π 0 is not a local maximum of J.

Proof. Note that J is a linear functional defined as J(π) = π(τ )R(τ ), where R(τ ) is the total return following the trajectory τ . Due to the linearity, we can generate the path by a convex interpolation of π 0 and π * . Note that J(π) is also a convex interpolation of J(π 0 ) and J(π * ) now.

A Proposition 2. We assume r(s, a) is bounded and denote max s,a r(s, a) = R. Denote the optimal value function for state s in the perturbed environment is V * d (s), and the optimal value of s in the original environment is

V * 0 (s), then |V * d (s) -V * 0 (s)| < R (1-γ) 2 when ||d|| < for ∀s ∈ S

Proof. We imitate the techniques in (Abbad & Filar, 1992) . From the Markovian property, we have V * (s) = max a s p(s |s, a)(r(s, a) + γV * (s ))

|V * d (s) -V * 0 (s)| ≤ max a | s ((p d (s ) -p 0 (s ))(r(s, a)) + γ(p d (s )V * d (s ) -p 0 (s )V * 0 (s )))| ≤ ||d||R + γ max a s |d(s )V * d (s )|ds + γ max s |V * d (s) -V * 0 (s)| Because V * (s) ≤ ∞ t=0 γ t R = 1 1-γ R, we have, max|V * d (s) -V * 0 (s)| ≤ R||d|| (1 -γ) 2

This bound is rather loose, and it can only be a partial explanation of the experimental results. The changes of environment in our experiments are much larger than a 'perturbation'.

A.3 THEOREM 1 AND PROOF:

Definition. At the i-th iteration, we update the value function by the following rule:

V i+1 (s) = (1 -α)V i (s) + α max a [R(s, a, T (s, a)) + γV i (T (s, a))], ∀s ∈ S (6)

where α ∈ (0, 1] is the step size, and V 0 (s) is some initialization value. Note that by Property 1 in Sec 2.1, the maximization algorithm over a is accessible. Theorem 1. The above continuous value iteration algorithm is a contraction mapping:

max s |V k (s) -V * (s)| ≤ (1 -(1 -γ)α) k max s |V 0 (s) -V * (s)|

, where V* represents the optimal value function.

The proof is a trivial extension of conventional value iteration.

In addition, we analyze how the error of model approximators influence our algorithm. We suppose the error will not change when training the value function. Denote the dynamics approximator as T (s, a) and the reward approximator as R(s, a). Suppose the maximal error of the two approximators are respectively T , R , that is, max s,a | T (s, a) -T (s, a)| = T and max s,a | R(s, a) -R(s, a)| = R . We have,

|V i+1 (s) -V i (s)| ≤ (1 -α)|V i (s) -V i-1 (s)| + αγ max a |V i ( T (s, a)) -V i-1 ( T (s, a))|, ∀s ∈ S As long as T (s, a) ∈ S, |V i+1 (s) -V i (s)| ≤ (1 -α + αγ) max s |V i (s) -V i-1 (s)|, ∀s ∈ S

Thus, the error will not hurt the contraction property, and the convergence is then guaranteed. However, the optimum may change. For the groundtruth dynamics and reward, the Bellman equation for the optimum is, 

V * (s) = max

0 e -x 2 dx > √ π 2 1 -e - √ t , t > 0

Proof. Denote the l.h.s. as A, then we have

A 2 = t 0 t 0 e -(x 2 +y 2 ) dxdy.

The integral is taken over a square area. By dividing the square area into a semi-circle area D and the rest, we can derive the following inequality from the non-negativity of e -(x 2 +y 2 ) :

A 2 > D e -(x 2 +y 2 ) dxdy = π 2 0 t 0 e -r 2 rdrdθ

The r.h.s can be easily integrated, and the result is π 4 (1 -e - √ t ). Hence we get Lemma 1.

Theorem 2. Denote the policy for a state s as π(s). For a given MDP, suppose

π 0 (s) = N (µ 0 , σ 0 ), s = s 0 π * (s), other s ∈ S ,

where µ 0 , σ 0 are the initial parameters for the initial Gaussian policy π 0 (s 0 ), and all the other policies are fixed as the optimal policy π * (s). We simply denote such policy as π(µ 0 , σ 0 ). Presume the optimal policy for s 0 is N (µ * , 0). If there exists µ ∈ [µ 0 , µ * ] and ∆ > 0 such that the optimal Q value Q * (s 0 , a) < B(∆) (B is a function w.r.t.∆) for all action a ∈ [µ -∆, µ + ∆]. We have the following conclusion: J(π(µ , σ)) < J(π(µ 0 , σ 0 )) for all σ ∈ (0, σ 0 ].

Proof. We start the proof by showing the exact value of the bound B, then we demonstrate how we can deduce the conclusion from B. to simplify the proof, we use the following notations:

Q * (s 0 ) = max a Q * (s 0 , a) f (µ, σ) = pdf (N (µ, σ)) V π (s 0 |µ, σ) = ∞ -∞ f (µ, σ)Q π (s 0 , a)da

APPENDIX B DETAILS OF ILLUSTRATIVE EXAMPLE:

We implement the MDP and the algorithms in MATLAB 2015b. For the policy gradient theorem, we use a variance-reduced version of the vanilla policy gradient:

∇ θ J(π θ ) = 1 N N i=1 T t=1 ∇ θ log π θ (a t |s t )( T t =t r(s t , a t ))

For PG, we maintain a Gaussian policy N (µ(s i ), σ 2 (s i )) for s 0 , s 1 , s 2 with trainable parameters µ(s i ) and σ(s i ) (i = 0, 1, 2). For transfer learning, in the new environment, we set the initial µs as the optimal policy in the source domain, and set σs as 1 to encourage exploration. Likewise, for VI, we set the initial value function as the optimal value in the source domain.

For the value iteration, we use the continuous version as Section. 4. We set the step size of the policy gradient algorithm as 1e-4, and the step size of the value iteration algorithm as 1e-3.

Fig. 6 and table 1 give more details about the illustrative example.

) ) 

APPENDIX C IMPLEMENTATION DETAILS:

We use feed forward networks for all models and the activation function is ReLU. For transition network, reward network, and termination network, the networks have 2 hidden layers of size 512. For value network and target value network, the networks have 2 hidden layers of size 256 and 2 hidden layers of size 128. The weights of the networks are initialized by Xavier weight initializer. And the networks are all trained by Adam optimizer, the learning rates for transition network, reward network, termination network and value network are 1e-3, 1e-3, 1e-3, 1e-4, respectively.

The transition network, reward network, and termination network are trained 2 times for every 40 steps. The training data is uniformly sampled from the replay buffer, and the batch size is 512. The value network is trained 2 times in each step with the current (s, r, s ) tuple. At each time the value network is trained, the target networks will be soft updated as follows,

EQUATION

A critical step in our algorithm is to find the optimal action a * with respect to state s. That is, we need to solve the following optimization problem, π(s t ) = arg max a (f R (s t , a) + γf V (f T (s t , a) + s t ))

We use Adam optimizer with the learning rate of 0.1. To avoid trapped by local maxima, we set 200 initial points for the gradient descent procedure. When training from scratch, the optimizer will be called k times, where the initial value of k is 0 and it is increased by 1 for every 150k steps. The reason we do not optimize too many steps at the beginning is the value network is not accurate at that time. But when transferring, we set a fixed k = 10 because we have a good initialization for value network. After we get the action, a noise will be added to the action in order to encourage exploration. The noise we used is Ornstein-Uhlenbeck process with µ = 0 and σ = 0.15. And the noise will be decayed with the progress of training.

However, the above case is just for the tasks without true terminations. For the tasks with true terminations, i.e. the task will be terminated before reaching the time limit, the optimization needs to be re-writed as follows,

EQUATION

, where t is the decision boundary of the termination network. Note that the above function is not fully differentiable, so we replace the hard threshold with a soft threshold.

π(s t ) = arg max

EQUATION

, where σ is the sigmoid function. In this way, we make it fully differentiable, so that gradient descent can be applied.

Our implementation is based on PyTorch. In order to speed up our program, we employ multiprocessing technique. In practice, we have 4 processes running in parallel, and all the networks are shared across the processes.

:



Figures and Tables:

Figure 1: an MDP

Figure 2: The training curve of π(s 0 ) during fine-tuning.π(s 0 ) represents its mean µ(s 0 ) for clarity.

Figure 3: The training curves of training from scratch and transfer learning of MVC, TRPO, and DDPG. Thick lines correspond to mean episode rewards, and shaded regions show standard deviations of 3 random seeds. Our method (MVC) achieves comparable performance with the baselines while significantly outperforms them on transfer learning.

Figure 4: Train from scratch

.2 PROPOSITION 2 AND PROOF Definition. Assume the transition probability p(s |s, a)is perturbed by a disturbance d(s |s, a), we denote the perturbed transition probability as p d (s |s, a) = p(s |s, a) + d(s |s, a). d(s |s, a) is such a function that ensures p d (s |s, a) is a feasible distribution (p d (s |s, a) ≥ 0 and s (p d (s |s, a)) = 1). Define ||d|| = max s∈S,a∈A s |d(s |s, a)|ds .

s, a, T (s, a)) + γV * (T (s, a))], ∀s ∈ S For the approximated dynamics and reward, V * (s) = max a [ R(s, a, T (s, a)) + γV * ( T (s, a))], ∀s ∈ S They are not necessarily the same, hence the optimum may change. A.4 THEOREM 2 AND PROOF Lemma 1.

t

Figure 6: The training curve during fine-tuning.

Algorithm 1 Model-based Value-Centric (MVC) Reinforcement LearningRequire: An MDP M, a maxmimum timestep number N 1: For train from scratch: initialize f V (s; θ), f T (s, a; φ), f R (s, a; ψ) randomly.For transfer learning: initialize f V (s; θ), f T (s, a; φ), f R (s, a; ψ) with the learned parameters. 2: Initialize an empty replay buffer B.At state s t , compute action a t with Eq. 4. Add exploration noise to a t . Execute a t in the environment, receive r t and s t+1 . Store (s t , a t , r t , s t+1 ) in B

Table content: <table><tr><td>3: for i = 1 : N do</td></tr><tr><td>4:</td></tr><tr><td>5:</td></tr><tr><td>6:</td></tr></table>

Peer review (iclr2019):
 The paper proposes a model-based value-centric (MVC) deep RL algorithm for transfer learning. The algorithm optimizes neural networks to estimate the deterministic transitions and rewards, and uses the these models to learn a value function by minimizing the Bellman residual. Policy is represented implicitly as the action that greedily maximizes the return, expressed in terms of the learned models. The experiments show some improvement on transferability over DDPG and TRPO policies. The paper has two relatively independent stories: The title and the introduction motivates the work by discussing the transferability of policies and value functions. However, instead of rigorously evaluating transferability, the paper proposes a model-based algorithm (MVC) for learning policies for continuous actions. Novelty of the new algorithm is quite limited, as it simply uses a learned dynamics model and reward function to learn a value function. Regarding transferability, introducing MVC seem quite orthogonal, and instead, it would be better to have a clear comparison of transferability using existing methods (e.g., DDPG). If having an explicit policy network hurts transferability, then existing algorithms can be modified by replacing the actor with greedy maximization, or alternatively other value based methods that do not involve actor network (NAF, SQL, QT-Opt) could be used. Regarding the intuition why values transfer better, the examples given in the introduction and Section 3 are good and intuitive. However, from my experience, the limited information content of a policy is only a partial reason for poor transferability, and in practice I have seen policies to transfer, in fact, better than values. The chosen viewpoint based on information content is nice as it can be proven mathematically, but might not be the most insightful and important in practice. The experimental evaluation is not rigorous enough to allow drawing further conclusions. For example, one could compare the two approaches using a wider set of RL algorithms, include more realistic environments (ideally transfer to real-world), or have a heat map illustrating transferability w.r.t. selected parameters. Also, no comparison to the state-of-the-art methods is provided (PPO, TD3, SAC). Minor points: - Please include the theorems in Section 5 (and proofs in the appendix). The intuition provided in the body is not very clear. - Why is it necessary to assume a deterministic dynamics model? Why only the dynamics model can vary between the domains and not also the reward (second paragraph in Section 1)?