Year: 2018

Abstract:
Transferring learned knowledge from one environment to another is an important step towards practical reinforcement learning (RL). In this paper, we investigate the problem of transfer learning across environments with different dynamics while accomplishing the same task in the continuous control domain. We start by illustrating the limitations of policy-centric methods (policy gradient, actorcritic, etc.) when transferring knowledge across environments. We then propose a general model-based value-centric (MVC) framework for continuous RL. MVC learns a dynamics approximator and a value approximator simultaneously in the source domain, and makes decision based on both of them. We evaluate MVC against popular baselines on 5 benchmark control tasks in a training from scratch setting and a transfer learning setting. Our experiments demonstrate MVC achieves comparable performance with the baselines when it is trained from scratch, while it significantly surpasses them when it is used in the transfer setting.

heading: 1 INTRODUCTION
text: While the achievements of deep reinforcement learning (DRL) are exciting in having conquored many computer games (Go (Silver et al., 2016; 2017), Atari games (Mnih et al., 2015)), in practice, it is still hard for these algorithms to find applications in real-world environments. Among the impediments, a primary obstacle is that they could fail even if the environment to deploy the agent is slightly different from where they were trained (Tan et al., 2018; Bousmalis et al., 2018; Tobin et al., 2017; Tamar et al., 2016). In other words, they lack the desired ability to transfer experiences learned from one environment to another. These observations motivate us to ask, (1) why are current RL algorithms so inefficient in transfer learning and (2) what kind of RL algorithms could be friendly to transfer learning by nature?
In this work, we explore the two questions and present a partial answer based on analysis and experiments. Our exploration concentrates on control tasks due to its broad impact; in particular, we further assume that across environments only their dynamics are not the same. Possible sources of such dynamics discrepancy could be variation of physical properties such as object mass, gravity, and surface friction. It is worth noting that our framework is general and we do not assume any specific perturbation source or type.
Our investigation starts with understanding the limitation of transferring the policy function (which maps a state to a distribution of actions) across environments. We analyze this transfer strategy because the de facto DRL framework in the control domain (DDPG (Lillicrap et al., 2015), TRPO (Schulman et al., 2015), A3C (Mnih et al., 2016), etc.) are policy-centric methods, which directly optimize the policy function. As a result, these methods learn a precise policy function, and sometimes also produce an imprecise value/Q-function as a side product. However, even if a perfect policy function has been learned from the source environment, this policy could behave quite poorly, or even fail, in the new environment, especially when the action space is hard-constrained (e.g., force or torque usually has a maximal value). We illustrate this by a simple example: Imagine a child shooting three-pointers when playing basketball. With a 600g ball, she can make the three-pointer. However, she may hardly make the three with a 800g ball because it is too heavy. What will she do? Most likely she will step forward, approach the basket, and make a closer jump-shot. We see that marginal
dynamics variation can lead to drastic policy change, and direct policy optimization initialized from the old policy would not be efficient. We will analyze this issue more systematically by theoretical and experimental approaches throughout the paper.
The investigation implies that, instead of directly transferring policies, the swift transfer should be grounded in richer and more structured knowledge of the task, so as to facilitate the judgment of whether the agent is approaching the goal, which is critical for making the right decision. Enlightened by the above intuition, we propose a simple model-based and value-centric framework for continuous reinforcement learning.
Our method contains two disentangled components: a dynamics approximator (model-based) and a state value function approximator (value-centric). The agent plans its action by solving an optimization problem using both approximators. As side products, this design learns a precise transition function, a precise reward function, and a precise value function on a subset of states. In particular, knowledge from historical explorations have been stored in the value function. In comparison, previous policy-centric methods can only produce a precise policy function, thus our framework allows to transfer much more information. By fine-tuning the whole framework in a new environment, our agent can adapt quickly with much lower sample complexity than state-of-the-art.
We call our method value-centric because it strives to learn a precise value function. The general framework is inspired from the Value Iteration (VI) method, which is a classical approach for discrete decision making. However, since control problems have a continuous action space, we cannot directly enumerate over the action space as in the discrete setting but have to address the highly non-convex optimization problem. To make it tractable, we leverage differentiable function approximators like neural networks to learn the dynamics and the value function. By such an approximation, it is possible to solve the optimization problem with state-of-the-art optimizers effectively.
We also theoretically analyze our value-centric framework and classical policy gradient algorithms from an optimization perspective. To build an intuitive understanding, we create a simple and illustrative example that clearly shows a local optimum in the policy space can prevent policy gradient methods from transferring successfully, but will not affect our value-centric framework.
We summarize our contributions as below:
• We provide a theoretical justification to show the advantage of value-centric methods from an optimization perspective.
• We propose a novel value-centric framework for continuous reinforcement learning of comparable sample efficiency with popular deep RL methods in the training from scratch setting.
• Extensive experiments show the superiority of our method in a transfer learning setting

heading: 2 BACKGROUND

heading: 2.1 MDPS AND NOTATION
text: First, we introduce the standard reinforcement learning setting for continuous control. We assume the underlying control problem is a Markov Decision Process (MDP), which is defined by a tuple 〈S,A, T ,R, ρ0, γ〉. Here, S and A are the continuous state space and action space, respectively, T is the transition dynamics, R is the reward function, ρ0 is the distribution of the initial state, and γ ∈ (0, 1] is the discounting factor for future rewards. For robot control tasks, it is reasonable to assume a deterministic transition dynamics T : S × A → S, and a deterministic reward function R : S ×A → R. The policy π of an agent maps the state space to a probability distribution over the action space π : S → P(A). The goal of reinforcement learning is to find an optimal policy π∗ that maximizes the expectation of the accumulated future rewards according to the initial state distribution J(π) = Eρ0,π[ ∑∞ t=0 γ
tR(st, at)]. We also define the value function V of a state s as V (s) = Eπ[ ∑∞ t=0 γ
tR(st, at)|s0 = s]. Finally, the optimal value function V ∗ satisfies V ∗(s) = maxπ Eπ[ ∑∞ t=0 γ tR(st, at)|s0 = s] for any s ∈ S.
Low-dimensional Assumption We focus on control problems which usually have well-engineered and low-dimensional state/action representations. Not rigorously, the assumption has two implications:
• Property 1: For a smooth function f(at) over A, we can find its approximate solution by sampling over the domain and optimizing locally; • Property 2: We can learn a network to approximate the transition and reward functions.
Empirically, we find evidence of both properties as in our experiment section (Sec 6).

heading: 2.2 TRANSFER LEARNING
text: Many differently posed transfer learning problems have been discussed in the reinforcement learning literature (Taylor & Stone, 2009). In this work, we study the problem of the environment slightly changing while the task remains the same. For example, in the pendulum swing-up problem, once the agent learns how to swing up a 1kg pendulum, we expect that it could quickly adapt itself to swing up a 2kg pendulum leveraging the learned knowledge. We formulate our setting by modifying the aforementioned standard RL setting. We consider a pair of MDPs sharing state and action spaces. Their transition dynamics T and reward functions R are parameterized by a vector ξ: st+1 = T (st, at; ξ), rt = R(st, at; ξ). Each ξ defines a unique MDPMξ. The change ofR is only caused by the change of T (instead of the goal of the task), so the change of reward function is limited, though we parameterize it by ξ for rigor. After the agent learns how to perform well on a source MDP, we expect it to solve the target MDP using few interactions with the new environment.

heading: 3 WHY TRANFERRING POLICY CAN BE DIFFICULT? A SIMPLE ILLUSTRATION
text: In the RL community, most control tasks are solved by policy gradient-based methods. In this section, we illustrate the limitation of transferring policy by a simple example and compare it with a value-centric method.
(𝑎) (𝑏)
state, it receives a reward r = 1 − |a|. Every time the agent visits the
failure state, it will be punished by a negative reward r = −5. The optimal policy for this task is a∗0 = −f1 if f1 < f2; a∗0 = f2 if f1 ≥ f2; a∗1(s1) = −f1; a∗1(s2) = f2.
Now we show the different behaviors of the value- and policy-centric strategies in a transfer learning setting. For the source environment, f1 = 0.7 and f2 = 0.8. It is not hard to find that the optimal path is path1. Then we modify the transition rule of the environment by slightly varying f1 and f2. We set f1 = 0.8 and f2 = 0.7. While the variation of the environment is relatively small, the optimal policy for s0 is completely different and has changed to path2. The optimal state value function and policy are shown in Table 1 in the Appendix B.
To compare policy- and value-centric strategies, we run two representative algorithms on this game – the Policy Gradient (PG) algorithm (Williams, 1992) (policy-centric) and the Value Iteration (VI) algorithm (Sutton et al., 1998) (value-centric). We assume the value-centric methods have access
to the oracle transition rules and the reward function, and we parameterize the policy π(si) with Gaussian distribution. Details can be found in Appendix B. We compare the behaviors of PG and VI to intuitively demonstrate their characteristics. PG gets stuck at the old optimum (Fig. 2 left) since the distance between the old optimum and the new optimum is so large. To see why, we first point out that PG optimizes in the policy space, restricting that the policy shifting from the old optimum to the new optimum must be quite “continuous”. However, to reach the new optimum, π(s0) must flip the sign of its mean µ(s0). Unfortunately, when µ(s0) approaches zero from the negative side (from −0.7), its transition possibility to the failure state will increase, thus µ(s0) will bounce away from 0− to avoid punishment. Consequently, it is very unlikely to successfully reach the new optimum, a positive value.
In contrast, the value function in VI algorithm continuously shifts and converges to the new optimum (See Appendix B) and the policy deduced from the value function converges to the optimal policy in 10 episodes. Value function reuses the learned information more efficiently. It also allows sudden change of the policy (Fig. 2 right), which could be beneficial for transferring.

heading: 4 MODEL-BASED AND VALUE-CENTRIC (MVC) REINFORCEMENT LEARNING

heading: 4.1 VALUE-CENTRIC METHOD IN TRANSFER LEARNING
text: Our objective is to make the agent learn faster in a new environment after trained in a similar environment. From the above discussion, we know that instead of directly transferring policies, the swift transfer should be grounded in richer and more structured knowledge of the task. It is worth exploring transfer by value function since the value function contains more information than the policy alone. A straight-forward idea is to utilize the value/Q function from actor-critic algorithms such as DDPG to facilitate transfer learning. However, the value/Q function learned from the actorcritic algorithm is usually imprecise, and it does not capture so much knowledge in the original environment. To address this issue, we propose an algorithm to directly train a precise value function in the original environment. We call it a value-centric method. Then, we just need to fine-tune this value function to help the agent adapt to the new environment. In the following subsections, we explain how to train this precise value function from scratch.

heading: 4.2 CONTINUOUS VALUE ITERATION
text: To better explain our main algorithm, we first propose a Continuous Value Iteration algorithm, which is possible to make use of Property 1 in Sec 2.1 to build a value-centric method. Theorem 2 in the Appendix suggests it always converges. Definition 1. At the i-th iteration, we update the value function by the following rule:
V i+1(s) = (1− α)V i(s) + α(max a [R(s, a, T (s, a)) + γV i(T (s, a))]),∀s ∈ S (1) where α ∈ (0, 1] is the step size, and V 0(s) is some initialization value. Note that by Property 1 in Sec 2.1, the maximization algorithm over a is accessible. We call this algorithm Continuous Value Iteration.

heading: 4.3 ALGORITHM
text: The CVI algorithm mentioned in Sec 4.2 assumes the oracle of dynamics, rewards, and an algorithm to solve the optimization problem w.r.t. a. In addition, it assumes the ability to sweep over the state space. To make it practical, we propose a model-based and value-centric(MVC) framework. We approximate the value function V , transition dynamics T , and the reward function R by function approximators like neural networks. Any differentiable function approximators not restricted to neural networks may be applicable. Let us denote the three parameterized function approximators we use as fV (s; θ), fT (s, a;φ), fR(s, a;ψ), where θ, φ, and ψ are parameters. To efficiently use the sampled data, we store the agent’s experience (st, at, rt, st+1) in a replay buffer B. Then fT is trained to minimize the following L2 loss function:
LT = 1 |D| ∑
(st,at,st+1)∈D
‖fT (st, at;φ)− (st+1 − st)‖2 (2)
Algorithm 1 Model-based Value-Centric (MVC) Reinforcement Learning Require: An MDPM, a maxmimum timestep number N
1: For train from scratch: initialize fV (s; θ), fT (s, a;φ), fR(s, a;ψ) randomly. For transfer learning: initialize fV (s; θ), fT (s, a;φ), fR(s, a;ψ) with the learned parameters. 2: Initialize an empty replay buffer B. 3: for i = 1 : N do 4: At state st, compute action at with Eq. 4. Add exploration noise to at. 5: Execute at in the environment, receive rt and st+1. Store (st, at, rt, st+1) in B 6: Update fV (s; θ) with Eq. 5. 7: Update fT (s, a;φ), fR(s, a;ψ) with Eq. 2 and Eq. 3 8: Soft update target network fV (s; θ−): θ− = αθ + (1− α)θ− 9: end for
Here,D is the training dataset extracted from B and | · | refers to the cardinality of a set. Like previous works (Kurutach et al., 2018; Nagabandi et al., 2017), the supervision signal for fT is the increment of the state to release its burden of memorizing the last state. We also train the reward approximator in a similar supervised learning way:
LR = 1 |D| ∑
(st,at,rt)∈D
‖fR(st, at;ψ)− rt‖2 (3)
At time step t, the agent samples data under a deterministic policy:
π(st) = arg max a (fR(st, a) + γfV (fT (st, a) + st)) (4)
Unlike the tabular case, the right-hand side of Eq. 4 is a complicated non-convex optimization problem. Fortunately, we can take the advantage that fV (s; θ), fT (s, a;φ), fR(s, a;ψ) are differentiable w.r.t the action vector a. We use Adam optimizer (Kingma & Ba, 2015) to solve the r.h.s optimization problem. To make sure that the optimizer finds a local maximum that is good enough to approximate the global maximum, we randomly select multiple initializations. Due to the superior parallel computing ability of modern GPUs, the optimization with different initial seeds can be operated simultaneously so that additional time consumption is limited. For exploration purpose, we also add noise (e.g. Ornstein-Uhlenbeck process) to the optimal action. While it is possible to expand the r.h.s. to a multi-step planning, notoriously, a learned model is inclined to diverge in long-horizon predictions. Consequently, we only use the one-step version for policy search.
To approximate the value function, we update the value approximator by the supervision of temporaldifference error (TD-error). Since updating the value function across the whole state space is unrealistic, we update the value approximator only by the data sampled from the environment in the current time step like policy gradient algorithms. Suppose at timestep t, the agent takes action at (following Eq. 4), then receives reward rt and transits to state st+1. We minimize the following loss:
LV = ‖fV (st; θ)− (rt + γfV (st+1; θ−))‖2. (5) Online training makes the whole framework more light-weight while sacrificing the guarantee of convergence. One can improve the training paradigm with past experience from the replay buffer or advanced sampling skills on the state space, but we leave them as future work.
Like DDPG, we employ a target network parameterized by θ− to stabilize the training. To speed up computation, we also fork multiple agents to run in parallel and synchronize them by a global agent. Algotihm 1 recaptures the whole algorithm paradigm.

heading: 5 A THEORETICAL JUSTIFICATION FROM THE OPTIMIZATION PERSPECTIVE
text: We will show the limitation of policy-centric methods and the nice properties of value-centric method. We narrow the analysis of policy-centric frameworks down to the landscape of J(π). To save space, we leave the theorems and detailed proofs in Appendix A. We explain the intuitions here:
First, we show in Theorem 1 that, for an arbitrary MDP with a deterministic transition function, a local optimum of J(π) that traps gradient-based methods could exist under a weak condition. In fact,
the local optimum issue is introduced by the parameterization of π’s output space, e.g., a Gaussian distribution. If we allow π to be an arbitrary distribution, the local optimum will vanish (Proposition 1). The example in Sec 3 exactly shows a failure case of transferring policy due to a local optimum.
Second, we show in Theorem 2 that, for the same type of MDPs, the Continuous Value Iteration algorithm leveraging Property 1 in Sec 2.1 can share the favorable converge property of classical value iteration. That is, the distance between a current value function and the optimal value function will be squeezed at linear rate, thus it always converges. In addition, Proposition 2 implies that a small perturbation to the environment is only likely to cause a marginal change to the optimal value function; therefore, the old value function would serve as a good initialization point.

heading: 6 EXPERIMENTS
text: We first compare our value-centric method against the policy gradient baselines in the training from scratch setting. We also show that our value-centric method beats baselines in the transfer learning setting. Finally, we conduct ablation study over method parameters and diagnose the components.

heading: 6.1 SETUP
text: We evaluate our algorithm and two prevalent continuous RL methods, Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) and Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), on five control tasks in the OpenAI Gym (Brockman et al., 2016):HalfCheetahv1,InvertedPendulum-v1, InvertedDoublePendulum-v1, Pendulum-v0, and Reacher-v1. For baselines, we use the code from OpenAI Baselines (Dhariwal et al., 2017) and fine-tuned their hyper-parameters as much as we can. More details can be found in the Appendix D.

heading: 6.2 MAIN RESULTS
text: We compare the sample complexity and the performance of the three algorithms (MVC, DDPG, and TRPO) on the aforementioned five environments, for both the training from scratch setting and transfer learning setting. We also compare with PPO (Schulman et al., 2017) and SQL (Haarnoja et al., 2017) on HalfCheetah and show the results in Appendix F.
Train from Scratch The reported results (Fig. 8(a)) show the mean and standard deviation of 3 runs with different random seeds. In four of the environments (InvertedPendulum-v1, HalfCheetah-v1, Pendulum-v0, Reacher-v1), our method achieves comparable performance as the better baseline. In InvertedDoublePendulum-v1, though there is a significant gap between MVC and TRPO, MVC performs at the same level with DDPG.
Transfer across environments We demonstrate the superiority of our method when transferring across environments. For each of the above five environments, we change one or several physical properties, like the mass of pendulum, to create two new environments with novel dynamics. In one of the new environments, the change is relatively small (we call it ’Hard’ in the plot), while the other is more intensively perturbed (we call it ’Harder’ in the plot). We first train standard agents on the original environment. For fair comparison, we pick the agents that achieve comparable performance for all methods in the original environment. Please refer to Appendix D for the details of the modification and the agents. To avoid the possibility of under-exploration in the new environments, we reset the exploration noise of all the algorithms. We directly fine-tune all the agents with the same number of simulation steps. The results are shown in Fig. 8(b).
On all the environments, we observed that TRPO has the worst overall transfer performance. DDPG and MVC have similar transferrability on simple tasks like Reacher-v1 and Pendulum-v0. However, on more complicated tasks like HalfCheetah-v1,the performance of DDPG is significantly worse than MVC. Further investigation shows that DDPG can actually learn a high-quality Q function for simple environments, which serves a similar role as our value function. However, on more challenging games such as HalfCheetah-v1 and InvertedDoublePendulum-v1, as a policy-centric algorithm, the learned Q function is far from the true one (Fig. 7 in Appendix E), thus the transfer is significantly slower. The success and failure of DDPG again shows the central role value plays in transfer learning.
Note that, in HalfCheetah-v1-Harder, MVC achieves 3000 points in about 50k steps, while TRPO and DDPG only get around 1000 points after 200k steps.

heading: 6.3 ABLATION STUDY AND DIAGNOSIS OF COMPONENTS
text: 350 400 450 500 550 Optimum by 10 step optimization
350
400
450
500
550
Op tim
um b
y 10
0M ra
nd om
sa m
pl e
HalfCheetah-v1 Validation of Property 1 Empirically, we find evidence for Property 1 and 2. Take HalfCheetah-v1 for example. For Property 1, we compare the optimization result of gradient-based method against random sampling with 106 points. Figure on the left demonstrates that Adam achieves comparable results with random sampling while being tens of times faster on our computer (a 48-core Xeon CPU with 2 Titan XP GPUs).
0 100 200 300 Steps (x1000)
10 3
10 2
10 1
100
101
Lo ss
HalfCheetah-v1 Transition, Scratch Reward, Scratch Transition, Transfer Reward, Transfer Validation for Property 2 Figure on the right shows that the loss functions for the transition network and reward network converge in less than 100k time steps, which means the transition network and reward network converges much faster than the value network (As shown in Fig. 8(a), the value network still does not converge after 1200k steps.). Therefore, selecting actions based on the learned transition network and reward network is trusty. We also observed that the learned transition and reward networks provide good start points for the training in the new environment.
0 200 400 600 800 1000 1200 Steps (x1000)
0
1000
2000
3000
4000
Re wa
rd
HalfCheetah-v1
200 init points, 4 opt steps 200 init points, 0 opt steps 1000 init points, 0 opt steps
Figure 4: Train from scratch 0 25 50 75 100 125 150 175 200 Steps (x1000)
0
1000
2000
3000
4000
Re wa
rd
HalfCheetah-v1 Hard
200 init points, 4 opt steps 200 init points, 0 opt steps 1000 init points, 0 opt steps
Figure 5: Transfer learning
Hyperparameters We verify the influence of the initialization and the number of optimization steps in the action search process. Fig. 4 shows that calling Adam optimizer is critical to stabilize the training. In transfer learning scenario (Fig. 5), the optimization shows a more significant impact on the performance and sample complexity. The agent that searches action with 200 initial points and 4 optimization steps is enough to win the agent using only 1000 initial points by a large margin.

heading: 7 RELATED WORK
text: Reinforcement Learning in Continuous Control Domain. Solving continuous control problems through reinforcement learning has been studied for decades (Sutton et al., 1998; Williams, 1992). Policy-based methods (Schulman et al., 2015; Mnih et al., 2016; Lillicrap et al., 2015) are more widely used. One exception is the NAF (Gu et al., 2016) method under the Q-learning framework which models the action space as a quadratic function.
Value-based Reinforcement Learning. The most relevant work in literature are perhaps the very recent Value Iteration Network (VIN) and Value Prediction Network (VPN) (Tamar et al., 2016; Oh et al., 2017). Though demonstrated better environment generalizability, VIN is specifically designed and only evaluated on the 2D navigation problem. VPN learns a dynamics model together with a value function and makes plans based on Monte Carlo tree search. In contrast to our work, VPN neither considered the continuous control problem nor thoroughly investigated their algorithm under the transfer learning setting.
Model-based Reinforcement Learning. For purposes such as increasing sample efficiency and designing smarter exploration strategies (e.g., curiosity-driven exploration), it can be beneficial to incorporate a learned dynamics model. Some very recent works have demonstrated the power of such model-based RL algorithms (Levine & Koltun, 2013; Nagabandi et al., 2017; Kurutach et al., 2018; Pathak et al., 2018; Feinberg et al., 2018; Pathak et al., 2017). However, to our knowledge, none of them has yet combined the value function with a learned dynamics model to solve continuous decision making problems.
Transfer Learning in Deep Reinforcement Learning. In this work, we study knowledge transfer problem across different MDPs.(Kansky et al., 2017) proposed the SchemaNetwork which learns the knowledge of the Atari physics engine by playing a standard version of the BreakOut game. (Higgins et al., 2017) learns disentangled representations in the source domains to achieve zero-shot domain adaption in the new environments. Finally, a straight-forward strategy is to show the agent all possible environments (Yu et al., 2017; Tan et al., 2018; Tobin et al., 2017).

heading: APPENDIX A PROOFS FOR THE THEOREMS
text: For mathematical rigor, we suppose the state space and the action space are both continuous in this section.
A.1 PROPOSITION 1 AND PROOF
Proposition 1. If J is the expected return defined as in Section 2, π can be an arbitrary function defined on the probability simplex ( ∫ π(τ)dx = 1, π(τ) ≥ 0, where τ is a trajectory from the starting state to a terminal state), and π∗ is the maximizer of J(π), then ∀π0, there exists a path connecting π0 to π∗ that J(π) monotonically increases, which implies that π0 is not a local maximum of J . Proof. Note that J is a linear functional defined as J(π) = ∫ π(τ)R(τ), where R(τ) is the total return following the trajectory τ . Due to the linearity, we can generate the path by a convex interpolation of π0 and π∗. Note that J(π) is also a convex interpolation of J(π0) and J(π∗) now.
A.2 PROPOSITION 2 AND PROOF
Definition. Assume the transition probability p(s ′ |s, a)is perturbed by a disturbance d(s′ |s, a), we denote the perturbed transition probability as pd(s ′ |s, a) = p(s′ |s, a) + d(s′ |s, a). d(s′ |s, a) is such a function that ensures pd(s ′ |s, a) is a feasible distribution (pd(s ′ |s, a) ≥ 0 and ∫ s′ (pd(s ′ |s, a)) = 1).
Define ||d|| = maxs∈S,a∈A ∫ s′ |d(s′ |s, a)|ds′ . Proposition 2. We assume r(s, a) is bounded and denote maxs,a r(s, a) = R. Denote the optimal value function for state s in the perturbed environment is V ∗d (s), and the optimal value of s in the original environment is V ∗0 (s), then |V ∗d (s)− V ∗0 (s)| < R (1−γ)2 when ||d|| < for ∀s ∈ S
Proof. We imitate the techniques in (Abbad & Filar, 1992). From the Markovian property, we have V ∗(s) = maxa ∫ s′ p(s ′ |s, a)(r(s, a) + γV ∗(s′))
|V ∗d (s)− V ∗0 (s)| ≤ max a | ∫ s′ ((pd(s ′ )− p0(s ′ ))(r(s, a)) + γ(pd(s ′ )V ∗d (s ′ )− p0(s ′ )V ∗0 (s ′ )))|
≤ ||d||R+ γmax a ∫ s′ |d(s ′ )V ∗d (s ′ )|ds ′ + γmax s |V ∗d (s)− V ∗0 (s)|
Because V ∗(s) ≤ ∑∞ t=0 γ tR = 11−γR, we have,
max|V ∗d (s)− V ∗0 (s)| ≤ R||d||
(1− γ)2
This bound is rather loose, and it can only be a partial explanation of the experimental results. The changes of environment in our experiments are much larger than a ’perturbation’.
A.3 THEOREM 1 AND PROOF
Definition. At the i-th iteration, we update the value function by the following rule: V i+1(s) = (1− α)V i(s) + αmax
a [R(s, a, T (s, a)) + γV i(T (s, a))],∀s ∈ S (6)
where α ∈ (0, 1] is the step size, and V 0(s) is some initialization value. Note that by Property 1 in Sec 2.1, the maximization algorithm over a is accessible. Theorem 1. The above continuous value iteration algorithm is a contraction mapping:
max s |V k(s)− V ∗(s)| ≤ (1− (1− γ)α)k max s |V 0(s)− V ∗(s)|,
where V* represents the optimal value function.
The proof is a trivial extension of conventional value iteration.
In addition, we analyze how the error of model approximators influence our algorithm. We suppose the error will not change when training the value function. Denote the dynamics approximator as T̂ (s, a) and the reward approximator as R̂(s, a). Suppose the maximal error of the two approximators are respectively T , R, that is, maxs,a |T̂ (s, a)− T (s, a)| = T and maxs,a |R̂(s, a)−R(s, a)| = R. We have,
|V i+1(s)−V i(s)| ≤ (1−α)|V i(s)−V i−1(s)|+αγmax a |V i(T̂ (s, a))−V i−1(T̂ (s, a))|,∀s ∈ S
As long as T̂ (s, a) ∈ S,
|V i+1(s)− V i(s)| ≤ (1− α+ αγ) max s |V i(s)− V i−1(s)|,∀s ∈ S
Thus, the error will not hurt the contraction property, and the convergence is then guaranteed. However, the optimum may change. For the groundtruth dynamics and reward, the Bellman equation for the optimum is,
V ∗(s) = max a [R(s, a, T (s, a)) + γV ∗(T (s, a))],∀s ∈ S For the approximated dynamics and reward,
V ∗(s) = max a [R̂(s, a, T̂ (s, a)) + γV ∗(T̂ (s, a))],∀s ∈ S They are not necessarily the same, hence the optimum may change.
A.4 THEOREM 2 AND PROOF Lemma 1. ∫ t
0 e−x
2 dx > √ π
2
√ 1− e− √ t, t > 0
Proof. Denote the l.h.s. as A, then we have
A2 = ∫ t 0 ∫ t 0 e−(x 2+y2)dxdy.
The integral is taken over a square area. By dividing the square area into a semi-circle area D and the rest, we can derive the following inequality from the non-negativity of e−(x 2+y2):
A2 > ∫∫ D e−(x 2+y2)dxdy = ∫ π 2 0 ∫ t 0 e−r 2 rdrdθ
The r.h.s can be easily integrated, and the result is π4 (1− e − √ t). Hence we get Lemma 1.
Theorem 2. Denote the policy for a state s as π(s). For a given MDP, suppose
π0(s) = { N (µ0, σ0), s = s0 π∗(s), other s ∈ S ,
where µ0, σ0 are the initial parameters for the initial Gaussian policy π0(s0), and all the other policies are fixed as the optimal policy π∗(s). We simply denote such policy as π(µ0, σ0). Presume the optimal policy for s0 is N (µ∗, 0). If there exists µ ∈ [µ0, µ∗] and ∆ > 0 such that the optimal Q value Q∗(s0, a) < B(∆) (B is a function w.r.t.∆) for all action a ∈ [µ
′ −∆, µ′ + ∆]. We have the following conclusion: J(π(µ ′ , σ)) < J(π(µ0, σ0)) for all σ ∈ (0, σ0].
Proof. We start the proof by showing the exact value of the bound B, then we demonstrate how we can deduce the conclusion from B. to simplify the proof, we use the following notations:
Q∗(s0) = max a Q∗(s0, a)
f(µ, σ) = pdf(N (µ, σ)) Vπ(s0|µ, σ) = ∫ ∞ −∞ f(µ, σ)Qπ(s0, a)da
Note that since all the policies are fixed except for s0, the (µ, σ) pair determines the whole policy π. Now using the above notations,
B = Vπ0(s0|µ0, σ0)−Q∗(s0)√
1− e− √ ∆√ 2σ0
+Q∗(s0)
Recall the condition that for a ∈ [µ′ −∆, µ′ + ∆], Q∗(s0, a) < B, we have:
Vπ′ (s0|µ ′ , σ) = ∫ ∞ −∞ f(µ ′ , σ)Qπ′ (s0, a)da
= ∫ µ′−∆ −∞ f(µ ′ , σ)Qπ′ (s0, a)da+ ∫ µ′+∆ µ′−∆ f(µ ′ , σ)Qπ′ (s0, a)da+ ∫ ∞ µ′+∆ f(µ ′ , σ)Qπ′ (s0, a)da
Because Q∗(s0) ≥ Q∗(s0, a) ≥ Qπ′ (s0, a),
Vπ′ (s0|µ ′ , σ) ≤ ∫ µ′−∆ −∞ f(µ ′ , σ)Q∗(s0)da+ ∫ µ′+∆ µ′−∆ f(µ ′ , σ)Bda+ ∫ ∞ µ′+∆ f(µ ′ , σ)Q∗(s0)da
= Q∗(s0) + ∫ µ′+∆ µ′−∆ f(µ ′ , σ)(B −Q∗(s0))da
= Q∗(s0) + (B −Q∗(s0)) ∫ µ′+∆ µ′−∆ f(µ ′ , σ)da
From the following inequality and Lemma 1,∫ µ′+∆ µ′−∆ f(µ ′ , σ)da = 1√ 2πσ ∫ µ′+∆ µ′−∆ e− (a−µ ′ )2 2σ2 da
= 1√ 2πσ ∫ ∆ −∆ e− a2 2σ2 da
≥ √ 1− e− √ ∆√ 2σ (Lemma1)
≥ √ 1− e− √ ∆√ 2σ0 (σ ∈ (0, σ0])
Thus, due to B −Q∗(s0) < 0
Vπ′ (s0|µ ′ , σ) ≤ Q∗(s0) + (B −Q∗(s0))
√ 1− e− √ ∆√ 2σ0
= Vπ0(s0|µ0, σ0) Then, we can divide the expected return into two parts: 1. return from the paths that never pass s0; 2. return from the paths that pass s0. Since all the policies except for s0 are deterministic, we can rewrite the expected return J(π(µ, σ)) as:
J(π(µ, σ)) = α+ βVπ(s0|µ, σ), where α and β are constants. Thus we finish our proof.
In fact, the bound B is not hard to reach, a special case of the above theorem is , Proposition. We follow the notations in Theorem 2, then, If Vπ(µ0,σ0)(s0) ≥ 0.5Vπ(µ∗,0)(s0), and Q∗(s0, a) < 0.6Vπ(µ0,σ0)(s0) for all a ∈ [ µ0+µ ∗−σ0 2 , µ0+µ ∗+σ0 2 ], then such µ
′ ∈ (µ0, µ∗) exists that J(π(µ ′ , σ)) < J(π(µ0, σ0)) for all σ ∈ [0, σ0].
The above proposition suggests that if the expected value of the original policy is not so bad (Vπ(µ0,σ0)(s0) ≥ 0.5Vπ(µ∗,0)(s0)), a local minimum will emerge when the optimal Q of a part of the intermediate action is only a little lower than the initial value (Q∗(s0, a) < 0.6Vπ(µ0,σ0)(s0)).
Although Theorem 1 proves the case when only the policy at s0 can be optimized, it is clear that the result also holds if the whole policy can be optimized.

heading: APPENDIX B DETAILS OF ILLUSTRATIVE EXAMPLE
text: We implement the MDP and the algorithms in MATLAB 2015b. For the policy gradient theorem, we use a variance-reduced version of the vanilla policy gradient:
∇θJ(πθ) = 1
N N∑ i=1 T∑ t=1 ∇θ log πθ(at|st)( T∑ t′=t r(st′ , at′ ))
For PG, we maintain a Gaussian policy N (µ(si), σ2(si)) for s0, s1, s2 with trainable parameters µ(si) and σ(si) (i = 0, 1, 2). For transfer learning, in the new environment, we set the initial µs as the optimal policy in the source domain, and set σs as 1 to encourage exploration. Likewise, for VI, we set the initial value function as the optimal value in the source domain.
For the value iteration, we use the continuous version as Section. 4. We set the step size of the policy gradient algorithm as 1e-4, and the step size of the value iteration algorithm as 1e-3.
Fig. 6 and table 1 give more details about the illustrative example.

heading: APPENDIX C IMPLEMENTATION DETAILS
text: We use feed forward networks for all models and the activation function is ReLU. For transition network, reward network, and termination network, the networks have 2 hidden layers of size 512. For value network and target value network, the networks have 2 hidden layers of size 256 and 2 hidden layers of size 128. The weights of the networks are initialized by Xavier weight initializer. And the networks are all trained by Adam optimizer, the learning rates for transition network, reward network, termination network and value network are 1e-3, 1e-3, 1e-3, 1e-4, respectively.
The transition network, reward network, and termination network are trained 2 times for every 40 steps. The training data is uniformly sampled from the replay buffer, and the batch size is 512. The value network is trained 2 times in each step with the current (s, r, s′) tuple. At each time the value network is trained, the target networks will be soft updated as follows,
θV target = τθV + (1− τ)θV target (7)
A critical step in our algorithm is to find the optimal action a∗ with respect to state s. That is, we need to solve the following optimization problem,
π(st) = arg max a (fR(st, a) + γfV (fT (st, a) + st)) (8)
We use Adam optimizer with the learning rate of 0.1. To avoid trapped by local maxima, we set 200 initial points for the gradient descent procedure. When training from scratch, the optimizer will be called k times, where the initial value of k is 0 and it is increased by 1 for every 150k steps. The reason we do not optimize too many steps at the beginning is the value network is not accurate at that time. But when transferring, we set a fixed k = 10 because we have a good initialization for value network. After we get the action, a noise will be added to the action in order to encourage exploration. The noise we used is Ornstein-Uhlenbeck process with µ = 0 and σ = 0.15. And the noise will be decayed with the progress of training.
However, the above case is just for the tasks without true terminations. For the tasks with true terminations, i.e. the task will be terminated before reaching the time limit, the optimization needs to be re-writed as follows,
π(st) = arg max a (fR(st, a) + 1fD<t[γfV (fT (st, a) + st)]) (9)
, where t is the decision boundary of the termination network. Note that the above function is not fully differentiable, so we replace the hard threshold with a soft threshold.
π(st) = arg max a
(fR(st, a) + σ(t− fD)(γfV (fT (st, a) + st))) (10)
, where σ is the sigmoid function. In this way, we make it fully differentiable, so that gradient descent can be applied.
Our implementation is based on PyTorch. In order to speed up our program, we employ multiprocessing technique. In practice, we have 4 processes running in parallel, and all the networks are shared across the processes.

heading: APPENDIX D EXPERIMENT DETAILS
text: Our simulation environment is OpenAI gym 0.9.3.
For transferring experiments, we change one or several physical properties in the environments. In one of the new environments,the change is relatively small (we call it ”Hard”), while the other is more intensively perturbed(we call it ”Harder”). We achieve it by directly modifying the codes or the XML files in the OpenAI gym library. The modifications are listed in the Table 2.
In transferring experiments, we select the agents get similar scores in the original environment for fair comparison. Table 3 shows the scores of the agents we chose in different environments. To get these scores, we evaluate each agent 10 times and calculate the mean episode reward.

heading: APPENDIX E SUPPLEMENTARY MATERIALS FOR EXPERIMENTS

heading: APPENDIX F SUPPLEMENTARY EXPERIMENT RESULTS
text: We compare the sample complexity and the performance of the five algorithms (MVC, DDPG, TRPO, PPO, and SQL) on the HalfCheetah, for both the training from scratch setting and transfer learning setting.
SQL is a continuous variant of Q-learning, which expresses the optimal policy via a Boltzmann distribution. PPO is developed from TRPO with several modifications. In the train from scratch setting, SQL shows the best sample efficiency (significantly better than other algorithms). This is probably because it improves exploration by incorporating an entropy term but all the other algorithms do not. However, in the transfer learning setting, the performance of MVC is nearly the same as SQL. As both are value-centric algorithms, we think the benefit of MVC comes from using an environment dynamics approximator. PPO shows better performance than TRPO (both in the train from scratch and the transfer learning), but still much worse than MVC.
We also did experiments with ME-TRPO, but we found it cannot handle a long time horizon (1000 time steps in HalfCheetah-v1) since its models will diverge quickly.

Review summaries:
The paper proposes a model-based value-centric (MVC) deep RL algorithm for transfer learning. The algorithm optimizes neural networks to estimate the deterministic transitions and rewards, and uses the these models to learn a value function by minimizing the Bellman residual. Policy is represented implicitly as the action that greedily maximizes the return, expressed in terms of the learned models. The experiments show some improvement on transferability over DDPG and TRPO policies.