Year: 2021

Abstract:
We propose a new general theory measuring the relativity between two arbitrary Markov Decision Processes (MDPs) from the perspective of reinforcement learning (RL). Considering two MDPs, tasks such as policy transfer, dynamics modeling, environment design, and simulation to reality (sim2real), etc., are all closely related. The proposed theory deeply investigates the connection between any two cumulative expected returns defined on different policies and environment dynamics, and the theoretical results suggest two new general algorithms referred to as Relative Policy Optimization (RPO) and Relative Transition Optimization (RTO), which can offer fast policy transfer and dynamics modeling. RPO updates the policy using the relative policy gradient to transfer the policy evaluated in one environment to maximize the return in another, while RTO updates the parameterized dynamics model (if there exists) using the relative transition gradient to reduce the gap between the dynamics of the two environments. Then, integrating the two algorithms offers the complete algorithm Relative Policy-Transition Optimization (RPTO), in which the policy interacts with the two environments simultaneously, such that data collections from the two environments, policy and transition updates are all completed in a closed loop to form a principled learning framework for policy transfer. We demonstrate the effectiveness of RPO, RTO and RPTO in the OpenAI gym’s classic control tasks by creating policy transfer problems.

heading: 1 INTRODUCTION
text: Deep reinforcement learning (RL) has demonstrated its great successes in recent years, including breakthrough of solving a number of challenging problems like Atari (Mnih et al., 2015), GO (Silver et al., 2016; 2017), DOTA2 (Berner et al., 2019) and StarCraft II (Vinyals et al., 2019), with humanlevel performance or even beyond. These successes demonstrate that current deep RL methods are capable to explore and exploit sufficiently in huge observation and action spaces, as long as sufficient and effective data samples can be generated for training, such as the cases in games. For example, AlphaGo Zero (Silver et al., 2017) costs 3 days of training over 4.9 millions self-play games, and OpenAI Five (Berner et al., 2019) and AlphaStar (Vinyals et al., 2019) spend months of training using thousands of GPUs/TPUs over billions of generated matches. However, for environments that prohibit infinite interactions, e.g., robotics, real life traffic control and autopilot, etc., applying general RL is difficult because generating data is extremely expensive and slow. Even if parallel data collection is possible, for example, by deploying multiple robots or vehicles running simultaneously, the scale of collected data is still far below that in virtual games. Worse still, exploration in these environments is considerably limited for safety reasons, which further reduces the effectiveness of the generated data. Due to the above challenges, similar significant advances like in solving virtual games have not been witnessed in these applications yet.
Generally, there are three tracks of approaches targeting to alleviate the aforementioned situation to promote the widespread application of RL. They are improving the data efficiency, transfer learning and simulator engineering.
To improve data efficiency, many recent efforts have been paid on investigating offline RL algorithms (Siegel et al., 2020; Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Wang et al., 2018). Compared to standard on-policy or off-policy RL, offline RL (also known as batch RL) aims
to effectively use previously collected experiences stored in a given dataset, like supervised learning, without online interactions with the environment. The stored experiences may not be generated by a fixed or known policy, so offline RL algorithms can leverage any previously collected data and learn a provably better policy than those who generated the experiences in the dataset. Although offline RL can effectively take advantage of finite data samples, solving a complex real-world task still requires huge amount of high quality offline experiences. Another way to increase data efficiency is to adopt model-based RL. Compared to model-free methods, model-based RL (Kaiser et al., 2019; Janner et al., 2019; Moerland et al., 2020) learns a dynamics model that mimics the transitions in the true environment, and then the policy can feel free to interact with the learned dynamics instead of the true environment. It has been proved that the true return can be improved by interacting with the learned dynamics model when the model error is bounded (Janner et al., 2019). However, learning an accurate dynamics model still requires sufficient transition data by interacting with the true environment, specifically for complex dynamics with noisy transitions.
Transfer learning in RL (Zhu et al., 2020) is practically useful to adapt a policy learned in a source environment to solve another task in the target environment. In the context of this paper, we consider the case where the policy can feel free to explore in the source environment, while the amount of collected data in the target environment should be as small as possible. When the source environment is a simulated one while the target environment takes place in reality, the transfer problem is also known as the simulation to reality (sim2real) problem. The simplest way to do transfer is to train the policy in the source environment and then use the converged parameters as warm start for a new policy or part of its parameters in the target environment, so that the amount of interactions with the target is expected to be largely reduced, as long as the tasks and dynamics in the two environments are closely related. Training a shared or partially shared policy in both the source and target environments is an alternative method which also belongs to the multi-task reinforcement learning scope (Hessel et al., 2019). Domain adaptation has been demonstrated to be another useful technique (Ibarz et al., 2021). Such methods try to bridge the gap between the source and target environments using some adaptation networks. For example, adapter networks were introduced to convert the input in simulation to be close to the real-world observation, by utilizing the generative adversarial model (James et al., 2017; Shrivastava et al., 2017; Bousmalis et al., 2017; 2018; Rao et al., 2020), or on the contrary an inverse network was trained to convert real-world observation to that in simulation (James et al., 2019). Using such adapter networks, the policy only needs to be trained in the source environment, and then it can directly be applied in the target environment.
An important concept in transfer learning is that instead of directly deploying RL in the target environment, a source environment is considered as a proxy. Sharing this spirit, the last track of approaches tries to build a proxy simulator that is as close as possible to the target environment, and hence we refer to such methods as simulator engineering. For example, in robotics control problems, there are many mature toolboxes can offer simulation engineering, including MuJoCo, PyBullet, Gazebo, etc. Model-based RL can also be viewed as a specific form of simulator engineering that the simulator is composed by a pure neural network, which is trained to approach the target environment as with lower model error as possible, while this might require a large amount of dynamics data in the target environment as mentioned above. Actually, to achieve more efficient and accurate simulator engineering, one recent rising direction is to integrate differentiable programming and physical systems to build a trainable simulator, which follows the physical laws as in reality and also whose key factors, such as the mass, length and friction of some objects, are trainable like the parameters in neural networks. Representative examples include the DiffTaichi (Hu et al., 2020), Brax (Freeman et al., 2021) and Nimble (Werling et al., 2021).
Overall, the existing methods focus on either directly improving the data efficiency in the target environment or bridging/reducing the gap between a proxy environment and the target environment, and there lacks a principled theory that can incorporate the learning in the two environments through an unified framework, and explain the intrinsic relationship between the expected returns in the two environments from the perspective of RL. In this paper, we inherit the spirit in transfer learning and consider two environments, where one is free to interact and another is the goal to solve, and the number of interactions in the goal environment should be as small as possible. We believe that there exist some explicit connections between the expected returns in the two environments, given two different policies, from the very fundamental perspective of RL. To verify this, we formally define two Markov Decision Processes (MDPs) and then explicitly derive the difference between the expected returns in the two MDPs. In the following context, with the RL convention, an environment
is equivalent to an MDP. Specifically, let P(s′|s, a) and P ′(s′|s, a) denote two dynamics transition functions in any two arbitrary MDPs sharing the same state and action spaces, where (s, a, s′) is the tuple of the state, action and next state. Let π′(a|s) and π(a|s) denote two arbitrary policies, and denote J(P, π) as the cumulative expected return given P and π. Then, we aim to investigate the difference J(P ′, π)− J(P, π′), which is referred to as the relativity gap between the two MDPs. It turns out that the relativity gap has a very interesting and compact form that integrates the interactions in both environments. Now, suppose P and P ′ are the dynamics functions in the source and target MDPs respectively, and J(P, π′) has been maximized by optimizing π′. Then, with fixed P , P ′ and π′, maximizing the relativity gap over π by constraining π to be close to π′ will also improve the return J(P ′, π) in the target MDP; on the other hand, for trainable P , minimizing the relativity gap by optimizing P given a fixed policy π = π′ will reduce the dynamics gap, similar to what is done by conventional model-based RL methods. Based on the above two principles, our theoretical results suggest two general algorithms referred to as Relative Policy Optimization (RPO) and Relative Transition Optimization (RTO), respectively. RPO updates the policy using the relative policy gradient to transfer the policy evaluated in the source environment to maximize the return in the target environment, while RTO updates a dynamics model using the relative transition gradient to reduce the value gap in the two environments. Then, applying RPO and RTO simultaneously offers a complete algorithm named Relative Policy-Transition Optimization (RPTO), which can transfer the policy from the source to the target smoothly. RPO, RTO and RPTO interact with the two environments simultaneously, so that data collections from two environments, policy and/or transition updates are completed in a closed loop to form a principled learning framework. In the experimental section, we show how to practically apply RPO, RTO and RPTO algorithms. We demonstrate the effectiveness of these methods in the classic control problems in OpenAI gym with both discrete and continuous actions, by varying the physical variables like mass, length and gravity of the objects to create policy transfer problems. At the last section, we discuss a few new directions based on the proposed relativity theory, which are worthy future investigations.

heading: 2 PRELIMINARIES

heading: 2.1 REINFORCEMENT LEARNING
text: A standard RL problem can be described by a tuple 〈E ,A,S,P, r, γ, π〉, where E indicates the environment that is an MDP with dynamics transition probability P; at each time step t, st ∈ S is the global state in the state space S, and at ∈ A is the action executed by the agent at time step t from the action space A; the dynamics transition function P(st+1|st, at) is the probability of the state transition (st, at) −→ st+1; for the most general case, the reward r(st, at, st+1) can be written as a function of st, at and st+1, while in many tasks it only relies on one or two of them, or it is even a constant in sparse rewards problem. For notation simplicity, we usually write r(st, at, st+1) as rt; γ ∈ [0, 1] is a discount factor and π(at|st) denotes a stochastic policy. The following equations define some important quantities in reinforcement learning. The objective of RL is to maximize the expected discounted return
J(P, π) = Es0,a0,···∼P,π [ ∞∑ t=0 γtrt ] , where s0 ∼ P(s0), at ∼ π(at|st), st+1 ∼ P(st+1|st, at).
At time step t, the state-action value QP,π , value function V P,π , and advantage AP,π are defined as QP,π(st, at) = Est+1,at+1,···∼P,π [∑∞ l=0 γ lrt+l ] , V P,π(st) = Eat,st+1,···∼P,π [∑∞ l=0 γ lrt+l ] , AP,π(s, a) = QP,π(s, a) − V P,π(s). In the above standard definitions, we explicitly show their dependence on both the dynamics P and policy π, since we will analyze these functions defined on variant dynamics and policies. This convention will be kept throughout the paper.

heading: 2.2 TRPO AND PPO
text: Given two arbitrary policies π′ and π, a well-known policy improvement theorem (Kakade & Langford, 2002; Schulman et al., 2015) is the fact revealed by the following equation
J(P, π′) = J(P, π) + Es0,a0,···∼P,π′ [ ∞∑ t=0 γtAP,π(st, at) ] . (1)
Based on this theorem, some widely adopted RL algorithms such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017) are developed. In TRPO, the following objective is optimized
maximizeθ Es∼dP,πθold ,a∼πθold [ πθ(a|s) πθold(a|s) AP,πθold (s, a) ] ,
subject to E [KL (πθold(·|s)||πθ(·|s))] ≤ δ, where the policy is parameterized by θ, and θold is the parameter since last update. dP,π is the discounted visitation probability given P and π. KL(·||·) is the KL-divergence and δ is a small constant restricting the policy update step size. To simplify the optimization, PPO removes the constraint in TRPO and maximizes the following clipped version
E s∼dP,πθold ,a∼πθold
[ min ( R(θ)AP,πθold (s, a), clip (R(θ), 1− , 1 + )AP,πθold (s, a) )] ,
where R(θ) = πθ(a|s)πθold (a|s) and is a hyperparameter controlling the proportion of clipped data. We review the TRPO and PPO algorithms here to allow readers to conveniently compare and see the connection between them and our proposed algorithms in the following sections.

heading: 3 THE THEORY OF RELATIVITY IN RL
text: With awareness of the most fundamental RL definitions in Section 2.1, it is sufficient to proceed the theory in this section. To be straightforward, we directly give the most important theorem revealing the theory of relativity in RL below. All the proofs are provided in the appendix.
Theorem 1 (The Theory of Relativity in RL) Given two Markov Decision Processes (MDPs) denoted by E ′ and E , who share the same state and action spaces S and A, their dynamics transition probabilities are P ′(st+1|st, at) and P(st+1|st, at) for any transition (st, at) −→ st+1 in E ′ and E , respectively. Assume the initial state distributes in the two MDPs identically that P ′(s0) = P(s0). Let J(P, π) denote the expected return defined on dynamics P and policy π. Then, the relativity gap between any two expected returns under different dynamics and policies is defined as
J(P ′, π)− J(P, π′)︸ ︷︷ ︸ relativity gap = J(P ′, π)− J(P, π)︸ ︷︷ ︸ dynamics-induced gap + J(P, π)− J(P, π′)︸ ︷︷ ︸ policy-induced gap , (2)
such that the dynamics-induced gap has an explicit form as
J(P ′, π)− J(P, π) =Es0,a0,···∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ] , (3)
and the policy-induced gap is revealed by the policy improvement theorem (Kakade & Langford, 2002) introduced in Eq. (1), rewritten by inverting π and π′ as
J(P, π)− J(P, π′) = Es0,a0,···∼P,π ∞∑ t=0 γtAP,π ′ (st, at). (4)
In Theorem 1, it is surprising that the dynamics-induced gap has a very compact formulation like the policy-induced gap. Below, we emphasize a few important points implied in Eqs. (3) and (4):
• In Eq. (3), the expectation is taken over the trajectory s0, a0, · · · sampled from (P ′, π), while the value and state-action value functions in the expectation, i.e., V P,π(st+1) and QP,π(st, at), are value functions defined on (P, π). This reveals a very practically useful conclusion that given a fixed policy π, the dynamics-induced gap can be calculated by measuring the value functions V P,π(st+1) and QP,π(st, at) in the dynamics P (imagining this is the source environment, where infinite data can be generated to accurately evaluate the value functions), while collecting probably a few data samples in P ′ (imagining this is the target environment) to estimate the expectation.
• In Eq. (3), Est+1 [r(st, at, st+1) + γV P,π(st+1)] 6= QP,π(st, at), because the transition (st, at) −→ st+1 takes place in P ′ instead of P , and hence Eq. (3) is not zero whereas P ′(st+1|st, at) 6= P(st+1|st, at) happens for non-zero r(st, at, st+1) and V P,π(st+1) with high probability, especially for high-dimensional deep neural networks. Of course, if P ′ = P , we immediately have J(P ′, π) = J(P, π) from Eq. (3).
• The dynamics-induced gap, i.e., J(P ′, π) − J(P, π), is related to what was analyzed in model-based policy optimization (MBPO) (Janner et al., 2019). Unfortunately, MBPO bounds the value gap with the dynamics model error at very early derivations to get their main theorem, which provides a basic theoretical support for that the commonly adopted model-based RL algorithms (alternatively updating the model and policy) can guarantee policy improvement, while it does not suggest new algorithms. Instead, with much deeper investigation of the fundamental dynamics-induced value gap, we can finally get the explicit identity equation in Eq. (3), instead of a bound. As we will show later, Theorem 1 suggests two thoroughly new algorithms for policy transfer and transition update. Details of the superiority of Theorem 1 over MBPO can be found in the proofs in appendix.
So far, Theorem 1 provides important results on both dynamics-induced and policy-induced gaps, while it is still not clear how these results can be applied empirically. In the following sections, we will introduce two new practical algorithms derived from Theorem 1, where one algorithm is for fast policy transfer from the source environment to the target environment, and another algorithm updates the parameterized dynamics in the source environment to be close to the dynamics in the target environment. Then, by combining the two algorithms, we obtain the complete algorithm to fast transfer a policy from source to target.

heading: 4 RELATIVE POLICY OPTIMIZATION (RPO)
text: As discussed previously, Eq. (3) in Theorem 1 suggests a way of estimating the dynamics-induced value gap by evaluating QP,π(st, at) and V P,π(st+1) in P while sampling data in P ′, given π. Practically, it is of less interest to estimate the exact dynamics-induced gap. Instead, if we have trained a policy π∗ in P (source environment) that maximizes J(P, π), then we are interested in finding another π̂ such that π̂ = arg maxπ[J(P ′, π) − J(P, π∗)], i.e., π̂ maximizes the dynamicsinduced gap, and also indirectly maximizes J(P ′, π). Normally, as long as P ′ is not far from P , finding π̂ can use π∗ as a warm start. Based on the above motivation, we propose the following theorem to get a lower bound of the dynamics-induced value gap.
Theorem 2 Define DmaxTV (p, q) = maxxDTV (p(·|x)||q(·|x)) as the total variation divergence between two distributions p(·|x) and q(·|x), where DTV (p(·|x)||q(·|x)) = 12 ∑ y |p(y|x) − q(y|x)|. Define = maxs,a |AP,π(s, a)|, where AP,π(s, a) = QP,π(s, a)− V P,π(s) is the advantage. Let
δ1 = D max TV (P ′(·|s, a),P(·|s, a))
for any (s, a) ∈ S,A and let δ2 = D max TV (π
′(·|s), π(·|s)) for any s ∈ S, and any two policies π′ and π. Let rmax = maxs,a,s′ r(s, a, s′) be the max reward for all (s, a, s′). Now, let ∆P
′,P(π) = J(P ′, π) − J(P, π) denote the dynamics-induced gap as a function of P ′, P and π. Now, we import a new policy π′ and define the following function
Lπ′(π) = ∞∑ t=0 γt Es0,a0,··· ,st∼P′,π′ ∑ at π(at|st) ∑ st+1
P ′(st+1|st, at)[ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at)
] as an approximation of ∆P ′,P(π) by sampling s0, a0, · · · using π′ and evaluating V P,π ′ and QP,π ′ using π′. Then, we have the following lower bound
∆P ′,P(π) ≥ Lπ′(π)− 2γδ1 (1− γ)2 − 4rmaxδ1δ2γ (1− γ)3 .
Based on Theorem 2, we can further obtain the following lower bound of the entire relativity gap.
Proposition 1 The entire relativity gap in Eq. (2) has the following lower bound
J(P ′, π)− J(P, π′) ≥ Es∼dP′,π′ ,a,s′∼P′,π′ π(a|s) π′(a|s) [r(s, a, s′) + γV P,π ′ (s′)− V P,π ′ (s)]− C,
where s′ is the next state that (s, a) −→ s′, and C = 2γδ1 +4 γδ 2 2
(1−γ)2 + 4rmaxδ1δ2γ (1−γ)3 is a constant relying on the dynamics difference δ1 and policy difference δ2.
Now, it becomes clear that by taking π = πθ and π′ = πθold for some policy parameters θ and its old version θold since last update, Proposition 1 suggests the following empirical objective
maximizeθ E s∼dP
′,πθold ,a,s′∼P′,πθold πθ(a|s) πθold(a|s) [r(s, a, s′) + γV P,πθold (s′)− V P,πθold (s)],
subject to E s∼ dP ′,πθold [DTV (πθold(·|s)||πθ(·|s))] ≤ ε,
(5)
for some small ε. At the first glance, the objective in Eq. (5) is very similar to the standard RL problem considered in TRPO or PPO (Schulman et al., 2015; 2017). However, again, by noting where the data is sampled from and how V and Q values are evaluated, we can realize the important difference that in Eq. (5), r(s, a, s′) + γV P,πθold (s′) − V P,πθold (s) 6= AP,πθold (s, a), i.e., it is not the general advantage function, because the transition (s, a) −→ s′ takes place in P ′. To be more accurate, we call r(s, a, s′) + γV P,πθold (s′)− V P,πθold (s) the Relative Advantage. That is, Eq. (5) suggests an empirical algorithm that we sample (s, a, s′) from E ′ using πold and compute the values V P,πold(s′) and V P,πθold (s) using πθold in E to update θ. Therefore, we refer to the optimization of Eq. (5) as the Relative Policy Optimization (RPO) algorithm.
The objective in Eq. (5) contains a constraint on the policy update size, and directly solving this objective requires line search, similar to what was proposed in TRPO (Schulman et al., 2015). To simplify the optimization, we use a clipped version, which has been demonstrated to be effective in PPO (Schulman et al., 2017).
Proposition 1 implies that as long as we can improve Eq. (5) by at least the constant C, we can guarantee improvement on J(P ′, πθ) over the constant J(P, πθold) at the current step. Generally, RPO optimizes θ to maximize the relativity gap J(P ′, πθ) − J(P, πθold) in the direction of maximizing J(P ′, πθ), because J(P, πθold) is a constant at the current step. However, starting from a well trained policy πθ∗ in P , as RPO updates θ, πθold will be gradually far from πθ∗ , and therefore J(P, πθold) might decrease. If this happens, continuing maximizing the relativity gap J(P ′, πθ) − J(P, πθold) can not guarantee the increase of J(P ′, πθ). One possible solution is to optimize the RPO loss plus the standard RL loss, e.g., the PPO loss, defined on P to keep πθ always performing well in E . As we can imagine, in such case, πθ will be updated towards a robust policy that performs well in both E ′ and E . Indeed, as we will show in our experiments, as long as P ′ is not too far away from P , optimizing RPO + PPO is often able to obtain such a robust policy; however, once P ′ differs from P too much, RPO will fail to transfer the policy to the target environment. This gives a hint that in addition to RPO, we need further to reduce the gap between P ′ and P , if P ′ and P are far away from each other. This is possible when P (the dynamics in source environment) is trainable, which has been considered in physical dynamics modeling (Hu et al., 2020; Freeman et al., 2021; Werling et al., 2021) and model-based RL methods (Janner et al., 2019).

heading: 5 RELATIVE TRANSITION OPTIMIZATION (RTO)
text: In this section, given fixed π and P ′, we consider a trainable P . Suppose Pφ(s′|s, a) is parameterized by φ for any transition (s, a) −→ s′. In the following theorem, we will import three dynamics quantities P ′, Pφ and Pφ′ , where P ′ still can be imagined as the dynamics in a target environment, and Pφ and Pφ′ are two variant dynamics functions parameterized by φ and φ′, respectively.
Theorem 3 Using the definitions in Theorem 2, the following function
Lφ′(φ) = ∞∑ t=0 γt Es0,a0,··· ,at∼P′,π [ Est+1∼P′ [ r(st, at, st+1) + γV Pφ′ ,π(st+1) ] −
Est+1∼Pφ [ r(st, at, st+1) + γV Pφ′ ,π(st+1) ] ]
is an approximation of ∆P ′,Pφ(π) by evaluating the value using Pφ′ instead of Pφ, and we have
|∆P ′,Pφ(π)− Lφ′(φ)| ≤ 4δ21γrmax (1− γ)3 .
In order to reduce the dynamics-induced gap by updating φ, we have to minimize |∆P′,Pφ(π)|. Based on Theorem 3, we have |∆P′,Pφ(π)| ≤ |Lφ′(φ)|+ 4δ 2 1γrmax (1−γ)3 . Therefore, we can alternatively minimize |Lφ′(φ)|. Empirically, if we consider Pφ as a probability function, then by noting that
Lφ′(φ) = ∞∑ t=0 γt Es0,··· ,at∼P′,π ∑ st+1 ( P ′(st+1|st, at)−Pφ(st+1|st, at) ) ( r(st, at, st+1)+γV Pφ′ ,π(st+1) ) ,
and taking φ′ = φold, minimizing |Lφ′(φ)| is equivalent to optimizing the following least square
minimizeφ Es∼dP′,π,a∼π ∑ s′ (P ′(s′|s, a)− Pφ(s′|s, a)) 2 ( r(s, a, s′) + γV Pφold ,π(s′) )2 . (6)
That is, we sample (s, a) in E ′ using π and optimize Eq. (6). We refer to the above optimization as Relative Transition Optimization (RTO). It is easy to see that RTO implies the standard model-based RL methods, who directly train φ using supervised learning, i.e., minimizeφ Es∼dP′,π,a∼π ∑ s′ (P ′(s′|s, a)− Pφ(s′|s, a)) 2 . Indeed, the objective of RTO in Eq. (6) can be viewed as a weighted form of supervised learning, with the weight r(s, a, s′)+γV Pφold ,π(s′) showing that transitions happened with larger values evaluated by Pφold and π should be optimized more aggressively. This is fairly reasonable from the perspective of fitting values, instead of fitting the dynamics directly in supervised learning. Therefore, RTO provides a theoretical explanation for the standard model-based RL methods by reducing the dynamics-induced value gap, and absorbs standard model-based RL as a special case in RTO.
In practice, the transition function Pφ is usually treated as a deterministic model, i.e., ŝ′ = φ(s, a). Under such case, RTO optimizes
minimizeφ E s∼dP′,π, a,s′∼P′,π
( r(s, a, s′) + γV Pφold,π (s′)− r(s, a, φ(s, a))− γV Pφold,π (φ(s, a)) )2 , (7)
and the common model-based methods solve minimizeφ Es∼dP′,π,a,s′∼P′,π (s′ − φ(s, a)) 2. Comparing the two objectives, Eq. (7) is more general by noting that fitting value function can distinguish the importance of states, and it is not necessary to learn a perfect dynamics model over all transitions.
In the experiments, we use deterministic dynamics model. More precisely, we adopt deterministic physical dynamics model, similar to recent proposed differentialable simulators (Hu et al., 2020; Freeman et al., 2021; Werling et al., 2021), which take advantage of the physical processes. Such model is much more efficient than pure neural network based dynamics model, because only a few scalars such as mass, length, gravity, etc., are trainable parameters in the physical systems.

heading: 6 RELATIVE POLICY-TRANSITION OPTIMIZATION (RPTO) ALGORITHM
text: Combining RPO and RTO algorithms, we finally obtain the complete Relative Policy-Transition Optimization (RPTO) algorithm in Algorithm 1. As we observe in Algorithm 1, in the main loop of
Algorithm 1: Relative Policy-Transition Optimization (RPTO) 1. Give the source and target environments Esource and Etarget, and their dynamics Psourceφ and Ptarget,
where the source dynamics Psourceφ is parameterized by φ; give a well-trained policy πθ0 in Esourceφ0 , where φ0 perfectly describes the initial source dynamics;
2. Create two empty replay buffers Dsource and Dtarget; 3. Initialize θ = θ0 and φ = φ0; while True do
4. Using πθ to interact with Esourceφ and push the generated trajectories into Dsource; 5. Using πθ to interact with Etarget and push the generated trajectories into Dtarget; 6. Sample a mini-batch {(s, a, s′)}source ∼ Dsource, and update V P source φ ,πθ by minimizing the
TD-error; 7. Sample a mini-batch {(s, a, s′)}target ∼ Dtarget, and apply the relative policy gradient in RPO to
update πθ; at the same time, update Psourceφ according to RTO; end while
RPTO, the policy πθ interacts with the two environments simultaneously and pushes the data into two buffers separately. In step 6, we sample a mini-batch from Dsource to update the value function in Eq. (5) from RPO. In step 7, we sample a mini-batch fromDtarget to update the policy parameter θ according to Eq. (5) in RPO, and also update the dynamics φ according to Eqs. (6) or (7) from RTO. Therefore, RPTO combines data collection from two environments, RPO and RTO in one closed loop, and this offers a principled learning framework for policy transfer. Indeed, steps 6 and 7 can be parallelized, as long as the value function V Psourceφ ,πθ µ (suppose it is parameterized by µ) does not share parameters with πθ, i.e., µ and θ are independent. Also, µ can be updated more frequently because it only requires data that can be generated infinitely from the source environment, and the more accurate the value function in the source environment is, the more accurate the relative policy gradient in step 7 will be estimated. For the dynamics parameter φ, the case becomes different, and it is often not a good choice to update φ as fast as we can, for example, using a larger step size for φ. To understand this, we need to explain how RPTO transfers the policy to the target environment in advance.
As we can imagine, as Psourceφ approaches Ptarget gradually by updating φ in step 7, the policy πθ is able to interact with a sequence of smoothly varying source environments. During this training period, the policy sees much more diverse transitions that lie between the initial source environment (with dynamics Psourceφ0 ) and the target environment. These diverse transitions are very helpful to encourage the agent to explore a robust policy. On the other hand, since the policy is initialized with a well-trained θ0 in Psourceφ0 , smoothly varying P source φ0
to reach Ptarget generates a sequence of environments that naturally provide a curriculum learning scheme, and this is very similar to what is considered in the recently emerged Environment Design methods (Dennis et al., 2020). Now, we are ready to answer the question in the last paragraph that why φ should not be updated aggressively: a slowly and smoothly varying φ provides more chances for the agent to see diverse transitions. Actually, if Psourceφ approaches Ptarget too fast, RPTO will be similar to directly training PPO in the target environment with a warm start θ0. In our experiments, we simply keep the learning rate of φ the same as that in RPO.

heading: 7 EXPERIMENTS
text: We experiment with all the OpenAI gym’s classic control tasks, including CartPole-v0, MountainCarContinuous-v0, Acrobot-v1, and Pendulum-v0, because in these tasks the physical systems are explicitly coded, which allows us to easily build physical dynamics model with only a few trainable factors, e.g., pole length, gravity and link mass, as indicated in Table 1. Among these, CartPolev0 and Acrobot-v1 are of discrete action space and the other two are continuous control problems. For all the tasks, the default settings in OpenAI gym are treated as the source environments, while for each task, we arbitrarily modify some of its physical factors to create the corresponding target environments. Details of the source and target environments are shown in Table 1.
For all tasks, we first pre-train a converged policy with PPO in the source environment, and the pre-trained policy will be used as a warm start when transferring to the target environment. For the policy transfer stage, PPO with the pre-trained policy as initialization, denoted as PPO-warm, will always be used as the baseline method. Other implementation details, such as hyper-parameters and neural network structures, are provided in the appendix. All the experiments in this section are repeated 10 times to plot the mean curve with standard derivation region.
Under review as a conference paper at ICLR 2022
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T ar
ge t E
nv R
et ur
n
RPO Evaluation
PPO-zero-source PPO-zero-target
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T ar
ge t E
nv R
et ur
n
(a) RPO Performance
PPO-zero PPO-warm RPO
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
R et
ur n
(b) RPO vs. Varying Pole Length
Return for Pole Len 0.6 Return in Source Env Return for Pole Len 0.8 Return in Source Env Return for Pole Len 1.0 Return in Source Env
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T es
t R et
ur n
in N
N _e
nv
RTO Evaluation - Value
RTO True Return
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0.450
0.475
0.500
0.525
0.550
0.575
0.600
E st
im at
ed P
ol e
Le ng
th
(c) RTO Performance
RTO True Pole Length in Target Env
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T ar
ge t E
nv R
et ur
n
(d) RPTO Performance
PPO-zero PPO-warm RPO RPTO
0.0 0.2 0.4 0.6 0.8 1.0 Training Steps 1e6
0
200
400
600
800
1000
A ve
ra ge
T ar
ge t E
nv R
et ur
n
MountainCarContinuous-v0
PPO-warm RPO RPTO
0.0 0.2 0.4 0.6 0.8 1.0 Training Steps 1e6
100
80
60
40
20
0
A ve
ra ge
T ar
ge t E
nv R
et ur
n
Acrobot-v1
PPO-warm RPO RPTO
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Training Steps 1e6
160
140
120
100
80
A ve
ra ge
T ar
ge t E
nv R
et ur
n
Pendulum-v0
PPO-warm RPO RPTO
Figure 1: Tutorial experiments in CartPole-v0.
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T ar
ge t E
nv R
et ur
n
RPO Evaluation
PPO-zero-source PPO-zero-target
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T ar
ge t E
nv R
et ur
n
(a) RPO Performance
PPO-zero PPO-warm
RPO
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
R et
ur n
(b) RPO vs. Varying Pole Length
Return for Pole Len 0.6 Return in Source Env Return for Pole Len 0.8 Return in Source Env Return for Pole Len 1.0
Return in Source Env
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T es
t R et
ur n
in N
N _e
nv
RTO Evaluation - Value
SL RTO RTO+SL True Return
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0.450
0.475
0.500
0.525
0.550
0.575
0.600
E st
im at
ed P
ol e
Le ng
th
(c) RTO Performance
SL RTO RTO+SL True Pole Length in Target Env
0.00 0.05 0.10 0.15 0.20 Training Steps 1e6
0
50
100
150
200
A ve
ra ge
T ar
ge t E
nv R
et ur
n
(d) RPTO Performance
PPO-zero PPO-warm RPO RPTO(SL) RPTO(RTO) RPTO(RTO+SL)
0.0 0.2 0.4 0.6 0.8 1.0 Training Steps 1e6
0
200
400
600
800
1000
A ve
ra ge
T ar
ge t E
nv R
et ur
n
MountainCarContinuous-v0
PPO-warm RPO RPTO
0.0 0.2 0.4 0.6 0.8 1.0 Training Steps 1e6
100
80
60
40
20
0
A ve
ra ge
T ar
ge t E
nv R
et ur
n
Acrobot-v1
PPO-warm RPO RPTO
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Training Steps 1e6
160
140
120
100
80
A ve
ra ge
T ar
ge t E
nv R
et ur
n
Pendulum-v0
PPO-warm RPO RPTO
Figure 2: Overall performance in other tasks.
We first use CartPole-v0 as a tutorial environment to show how the proposed algorithms are practically applied. Fig. 1 reports the results for various evaluations of RPO, RTO and RPTO. Fig. 1(a) demonstrates that RPO (without RTO) is sufficient to successfully transfer the policy to the target environment with pole length of 0.6, and RPO transfers the policy faster than PPO-warm. In Fig. 1(b), by varying the pole length to test more target environments, we find that when the pole length difference becomes larger, RPO fails to obtain a stable policy in the target environment. The curves also imply that for the case where RPO succeeds, the learned policy can perform well in both the source and target environments, i.e., RPO finds a robust policy; while for the failed cases, the learned policy deteriorates in both environments either. These results are consistent with our analysis at the end of Section 4. Fig. 1(c) evaluates the performance of RTO. As we can observe, RTO can optimize φ to converge to the true pole length in the target environment. Finally, Fig. 1(d) reports the RPTO’s performance, where the curves of PPO-zero, PPO-warm and RPO are duplicated from Fig. 1(a) as baselines. As we can observe, RPTO transfers the policy much faster than all the other methods.
So far, we have demonstrated how to apply RPO, RTO and RPTO in CartPole-v0 and what characteristics of these algorithms can help us better understand their performance. Now, we directly report all the performance curves for the other tasks in Fig. 2. In all the tasks, RPTO shows its superiority over PPO-warm in terms of both fast policy transfer and even better asymptotic convergence, because RPTO sees much more diverse dynamics that promotes exploration. Generally, RPO also shows its capability on fast policy transfer, while it usually suffers from higher variance and unstable performance, because it is limited by the dynamics gap between the source and target environments as revealed by Theorem 2 by noting the dynamics gap δ1 is a fixed constant in RPO. For the case in RPTO, benefitting from RTO, δ1 approaches zero gradually and so RPTO is stable and efficient.

heading: 8 DISCUSSION AND FUTURE WORK
text: We have proposed the general theory of relativity in RL. The theory shows its significance in creating new algorithms that are empirically demonstrated effective and efficient. In addition to RPO, RTO and RPTO, the relativity theory also opens a few new future directions in RL. For example, as we have discussed at the end of Section 6, controlling the update step size or frequency of RTO can provide better curriculum learning or environment design (although in this paper we simply fix the learning coefficient of RTO as the same as RPO. Please see appendix). Connecting this with metaRL and current environment design algorithms is a promising future direction. Moreover, the theory and algorithms in this paper are orthogonal to other techniques commonly used in policy transfer, such as domain adaptation, domain randomization, augmented observation, etc., and all of these methods can be integrated together in policy transfer applications. It is of great interests to apply all these techniques to solve complex real-world problems like robotics in future research.
REPRODUCIBILITY
According to the author guide, we provide a Reproducibility Statement here. For theoretical results in this paper, we have provided all complete proofs in the appendix. For the algorithms, we have attached the codes of RPO, RTO and RPTO in the submitted .zip supplementary material.

heading: A PROOF OF THEOREM 1
text: Define the expected cumulative return as
J(P, π) = Es0,a0,···∼P,π [ ∞∑ t=0 γtr(st, at, st+1) ] ,
where
s0 ∼ ρ(P0), at ∼ π(at|st), st+1 ∼ P(st+1|st, at).
To better understand the expected cumulative return, we have
J(P, π) = ∞∑ t=0 γtEst,at,st+1r(st, at, st+1)
= ∞∑ t=0 γtEst∼p(st),at∼π(at|st),st+1∼P(st+1|st,at)r(st, at, st+1)
= ∞∑ t=0 γt ∑ st p(st) ∑ at π(at|st) ∑ st+1 P(st+1|st, at)r(st, at, st+1),
where p(st) indicates the probability that st is visited. With a fixed policy π, we investigate the value difference between two MDPs as
J(P ′, π)− J(P, π)
= ∞∑ t=0 γt [ Est∼p′(st),at∼π(at|st),st+1∼P′(st+1|st,at)r(st, at, st+1)−
Est∼p(st),st+1∼P(st+1|st,at),at∼π(at|st)r(st, at, st+1) ]
= ∞∑ t=0 γt [∑ st p′(st) ∑ at π(at|st) ∑ st+1 (P ′(st+1|st, at)− P(st+1|st, at)) r(st, at, st+1)+
∑ st (p′(st)− p(st)) ∑ at π(at|st) ∑ st+1 P(st+1|st, at)r(st, at, st+1) ] . (8)
It is worth mentioning that MBPO (Janner et al., 2019) just stops here at Eq. (8) and starts to bound the transition difference P ′(st+1|st, at)−P(st+1|st, at) and the visitation difference p′(st)−p(st) with the dynamics model error, which is similar to δ1 defined in our Theorem 2. Then, it obtains a lower bound of Eq. (8) to conclude the main theorem in MBPO. As we will show below, this is still far away to reach our main theoretical results in Theorem 1.
Continuing from Eq. (8), the first term in Eq. (8) derives as∑ st p′(st) ∑ at π(at|st) ∑ st+1 (P ′(st+1|st, at)− P(st+1|st, at)) r(st, at, st+1)
= ∑ st p′(st) p(st) p(st) ∑ at π(at|st) ∑ st+1 (P ′(st+1|st, at)− P(st+1|st, at)) P(st+1|st, at) P(st+1|st, at)r(st, at, st+1) =Est∼p(st),at∼π(at|st),st+1∼P(st+1|st,at) p′(st)
p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) r(st, at, st+1)
=Est,at,st+1∼p p′(st)
p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) r(st, at, st+1).
In the second term of Eq. (8), the difference of the marginal probabilities p′(st) − p(st) can be expanded similarly as
p′(st)− p(st) =Est−1∼p′(st−1),at−1∼π(at−1|st−1) P
′(st|st−1, at−1)− Est−1∼p(st−1),at−1∼π(at−1|st−1) P(st|st−1, at−1)
= ∑ st−1 p′(st−1) ∑ at−1
π(at−1|st−1) (P ′(st|st−1, at−1)− P(st|st−1, at−1)) +∑ st−1 (p′(st−1)− p(st−1)) ∑ at−1 π(at−1|st−1)P(st|st−1, at−1)
=Est−1∼p(st−1),at−1∼π(at−1|st−1) p′(st−1)
p(st−1) (P ′(st|st−1, at−1)− P(st|st−1, at−1)) +∑
st−1
(p′(st−1)− p(st−1))Eat−1∼P(at1 |st−1)P(st|st−1, at−1). (9)
Plugging Eq. (9) back into the term of Eq. (8), we have∑ st (p′(st)− p(st)) ∑ at π(at|st) ∑ st+1 P(st+1|st, at)r(st, at, st+1)
= ∑ st (p′(st)− p(st))Eat∼π(at|st),st+1∼P(st+1|st,at)r(st, at, st+1) =Est−1,at−1,st,at,st+1∼P,π p′(st−1)
p(st−1) P ′(st|st−1, at−1)− P(st|st−1, at−1) P(st|st−1, at−1)
r(st, at, st+1)+∑ st−1 (p′(st−1)− p(st−1))Eat−1,st,at,st+1r(st, at, st+1). (10)
Note that the second term in Eq. (10) has an identical form as that in Eq. (8) by expanding prior states and actions. Therefore, by recursively expanding Eq. (10) backward, we finally have
J(P ′, π)− J(P, π)
= ∞∑ t=0 γt [ Est,at,st+1∼P,π p′(st) p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) r(st, at, st+1)+
Est−1,at−1,st,at,st+1∼P,π p′(st−1)
p(st−1) P ′(st|st−1, at−1)− P(st|st−1, at−1) P(st|st−1, at−1) r(st, at, st+1)+
· · · · · · ]
= ∞∑ t=0 Eτ∼P,π t∑ i=0 γt p′(si) p(si) P ′(si+1|si, ai)− P(si+1|si, ai) P(si+1|si, ai) r(st, at, st+1)
=Eτ∼P,π ∞∑ t=0 γtr(st, at, st+1) t∑ i=0 p′(si) p(si) P ′(si+1|si, ai)− P(si+1|si, ai) P(si+1|si, ai) , (11)
where τ indicates the trajectory s0, a0, · · · .
Let xt = γtr(st, at, st+1) and yi = p′(si) p(si) P′(si+1|si,ai)−P(si+1|si,ai) P(si+1|si,ai) , we have
∞∑ t=0 xt t∑ i=0 yi = x0y0 + x1(y0 + y1) + x2(y0 + y1 + y2) + · · ·
= y0(x0 + x1 + x2 + · · · ) + y1(x1 + x2 + · · · ) + · · · = ∞∑ i=0 yi ∞∑ t=i xt,
and ∑∞ t=i xt = γ i ∑∞ t=i γ t−ir(st, at, st+1) = γ iRi, where Ri = ∑∞ t=i γ
t−ir(st, at, st+1) is the empirical cumulative future reward. Continuing from Eq. (11),
J(P ′, π)− J(P, π)
= Eτ∼P,π ∞∑ t=0 γtRt p′(st) p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at)
= ∞∑ t=0 γtEτ∼P,πRt p′(st) p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at)
= ∞∑ t=0 γtEs0,a0,··· ,st+1∼P,π p′(st) p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) Eat+1,st+2,···∼P,π Rt
= ∞∑ t=0 γtEs0,a0,··· ,st+1∼P,π p′(st) p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) Eat+1,st+2,···∼P,π [r(st, at, st+1) + γRt+1]
= ∞∑ t=0 γtEs0,a0,··· ,st+1∼P,π p′(st) p(st) P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) ( r(st, at, st+1) + γV P,π(st+1) )
= ∞∑ t=0 γtEs0,a0,··· ,st,at∼P′,πEst+1∼P,π P ′(st+1|st, at)− P(st+1|st, at) P(st+1|st, at) ( r(st, at, st+1) + γV P,π(st+1) )
= ∞∑ t=0 γt { Es0,a0,··· ,st,at,st+1∼P′,π [ r(st, at, st+1) + γV P,π(st+1) ] − Es0,a0,··· ,st,at∼P′,π QP,π(st, at) } = Eτ∼P′,π
∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ] . (12)
Note that in Eq. (12), Est+1r(st, at, st+1)+γV (st+1) 6= Q(st, at), because the transition (st, at) −→ st+1 takes place in P ′ instead of P . Now, we complete the proof of the relativity theory.

heading: B PROOF OF THEOREM 2
text: Denote the dynamics-induced value gap as a function
∆P ′,P(π) = J(P ′, π)− J(P, π)
= Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ] .
We have
∆P ′,P(π)−∆P ′,P(π′)
= Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ] −
Eτ∼P′,π′ ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] .
Let
Lπ′(π) = ∞∑ t=0 γt Es0,a0,··· ,st∼P′,π′ ∑ at π(at|st) ∑ st+1
P ′(st+1|st, at)[ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ]
be the approximation of ∆P ′,P(π) by sampling (s0, a1, · · · , st) and evaluating values using π′. Then, we have
∆P ′,P(π)− Lπ′(π)
= Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ] −
Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] +
Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] −
∞∑ t=0 γtEs0,a0,··· ,st∼P′,π′ ∑ at π(at|st) ∑ st+1
P ′(st+1|st, at)[ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] .
Let
D1 = Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ] −
Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] ,
and
D2 = Eτ∼P′,π ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] −
∞∑ t=0 γtEs0,a0,··· ,st∼P′,π′ ∑ at π(at|st) ∑ st+1
P ′(st+1|st, at)[ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] .
For D1, we have
D1 = Eτ∼P′,π ∞∑ t=0 γt [ γ ( V P,π(st+1)− V P,π ′ (st+1) ) −(
Est+1∼P [r(st, at, st+1) + γV P,π(st+1)]− Est+1∼P [r(st, at, st+1) + γV P,π ′ (st+1)] )] =
∞∑ t=0 γt+1Es0,...,at∼P′,π [ Est+1∼P′ ( V P,π(st+1)− V P,π ′ (st+1) ) − Est+1∼P ( V P,π(st+1)− V P,π ′ (st+1) )]
= ∞∑ t=0 γt+1Es0,...,at∼P′,π ∑ st+1 (P ′(st+1|st, at)− P(st+1|st, at)) ( V P,π(st+1)− V P,π ′ (st+1) ) . From the Lemma 1 in TRPO (Schulman et al., 2015), we have
|V P,π(st)− V P,π ′ (st)| = Eτ∼P,π [ ∞∑ i=t γi−t|Aπ ′ (st, at)| ] ≤ 1− γ .
Therefore, we have
|D1| ≤ ∞∑ t=0 γt+1 2δ1 1− γ = 2γδ1 (1− γ)2 . (13)
For D2, we have
D2 = ∞∑ t=0 γt ∑ st ( p′π(st)− p′π′(st) )∑ at π(at|st) ∑ st+1
P ′(st+1|st, at) [ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at) ] =
∞∑ t=0 γt ∑ st ( p′π(st)− p′π′(st) )∑ at π(at|st) ∑ st+1
(P ′(st+1|st, at)− P(st+1|st, at)) ( r(st, at, st+1) + γV P,π′(st+1) ) .
Using the Lemma B.2 in MBPO, we have DTV (p′π(st)||p′π′(st)) ≤ tδ2. Also, we can bound
|r(st, at, st+1) + γV P,π ′ (st+1)| = ∣∣r(st, at, st+1) + γE ∞∑ i=t+1 γi−t−1r(si, ai, si+1) ∣∣ ≤ rmax 1− γ .
Finally, we have
|D2| ≤ 4rmaxδ1δ2
1− γ ∞∑ t=0 tγt = 4rmaxδ1δ2γ (1− γ)3 . (14)
Combining Eqs. (13) and (14), we finally get the lower bound
∆P ′,P(π) ≥ Lπ′(π)− 2γδ1 (1− γ)2 − 4rmaxδ1δ2γ (1− γ)3 . (15)
Now, we complete the proof.

heading: C PROOF OF PROPOSITION 1
text: Recall that J(P ′, π) = J(P, π) + ∆P′,P(π) and
Lπ′(π) = ∞∑ t=0 γt Es0,a0,··· ,st∼P′,π′ ∑ at π(at|st) ∑ st+1
P ′(st+1|st, at)[ r(st, at, st+1) + γV P,π′(st+1)−QP,π ′ (st, at)
] = ∑ s dP ′,π′(s) ∑ a π(a|s) ∑ s′ P ′(s′|s, a)[r(s, a, s′) + γV p,π ′ (s′)−Qp,π ′ (s, a)]
= Es∼dP′,π′ ,a,s′∼P′,π′ π(a|s) π′(a|s) [r(s, a, s′) + γV P,π ′ (s′)−QP,π ′ (s, a)].
Directly maximizing Lπ′(π) is insufficient, because once π varies, J(P, π) varies (probably decreases) accordingly, which can not guarantee monotonic increasing in J(P ′, π). The case is that we have to maximize J(P, π) + ∆P′,P(π). The policy improvement theorems in TRPO suggest
J(P, π)− J(P, π′) ≥ Eτ∼P,π′ [ ∞∑ t=0 γtAP,π ′ (st, at) ] − 4 γδ 2 2 (1− γ)2 .
Therefore, we have the final bound J(P ′, π)− J(P, π′)
= J(P ′, π)− J(P, π) + J(P, π)− J(P, π′) ≥ Lπ′(π) + Eτ∼P,π′ [ ∞∑ t=0 γtAP,π ′ (st, at) ] − C
= Es∼dP′,π′ ,a,s′∼P′,π′ π(a|s) π′(a|s) [r(s, a, s′) + γV P,π ′ (s′)−QP,π ′ (s, a) +AP,π ′ (s, a)]− C
= Es∼dP′,π′ ,a,s′∼P′,π′ π(a|s) π′(a|s) [r(s, a, s′) + γV p,π ′ (s′)− V p,π ′ (s)]− C (16)
where C = 2γδ1 +4 γδ 2 2
(1−γ)2 + 4rmaxδ1δ2γ (1−γ)3 . Now, we complete the proof.

heading: D PROOF OF THEOREM 3
text: Continuing from Eq. (12), we have J(P ′, π)− J(P, π)
= Eτ∼P′ ∞∑ t=0 γt [ r(st, at, st+1) + γV P,π(st+1)−QP,π(st, at) ]
= ∞∑ t=0 γtEs0,a0,··· ,at∼P′,π [ Est+1∼P′ ( r(st, at, st+1) + γV P,π(st+1) ) − Est+1∼P ( r(st, at, st+1) + γV P,π(st+1) )]
= ∞∑ t=0 γtEs0,a0,··· ,at∼P′,π [∑ st+1 P ′(st+1|st, at) ( r(st, at, st+1) + γV P,π(st+1) )
− ∑ st+1 P(st+1|st, at) ( r(st, at, st+1) + γV
P,π(st+1) ) ] .
Now, considering a parameterized source dynamics Pφ, we have
∆P ′,Pφ(π) = J(P ′, π)− J(Pφ, π)
= ∞∑ t=0 γtEs0,a0,··· ,at∼P′,π [∑ st+1 P ′(st+1|st, at) ( r(st, at, st+1) + γV Pφ,π(st+1) )
− ∑ st+1 Pφ(st+1|st, at) ( r(st, at, st+1) + γV
Pφ,π(st+1) ) ] .
Let Lφ′(φ) = ∞∑ t=0 γtEs0,a0,··· ,at∼P′,π [∑ st+1 P ′(st+1|st, at) ( r(st, at, st+1) + γV Pφ′ ,π(st+1) )
− ∑ st+1 Pφ(st+1|st, at) ( r(st, at, st+1) + γV
Pφ′ ,π(st+1) ) ] .
Then,
∆P ′,Pφ(π)− Lφ′(φ) = ∞∑ t=0 γt+1Es0,a0,··· ,at∼P′,π ∑ st+1 (P ′(st+1|st, at)− Pφ(st+1|st, at)) ( V Pφ,π(st+1)− V Pφ′ ,π(st+1) ) .
For V Pφ,π(st+1)− V Pφ′ ,π(st+1), we have∣∣V Pφ,π(st+1)− V Pφ′ ,π(st+1)∣∣ = Eτ∼Pφ,π
∞∑ i=t+1 γi−t−1 ∣∣∣∣∣∣ ∑ si+1 (Pφ(si+1|si, ai)− Pφ′(si+1|si, ai)) ( r(si, ai, si+1) + γV Pφ′ ,π(si+1) )∣∣∣∣∣∣
≤ 2δ1rmax (1− γ)2 .
Finally, we obtain the lower bound
|∆(φ)− Lφ′(φ)| ≤ 4δ21γrmax (1− γ)3 .

heading: E THE RPO AND RTO ALGORITHMS
text: Due to space limitation, we are not able to put the RPO and RTO algorithms in the main text. These two algorithms can be tailored from the RPTO algorithm in Algorithm 1, and we put them here.
Algorithm 2: Relative Policy Optimization (RPO) 1. Give the source and target environments Esource and Etarget, and their dynamics Psource and Ptarget;
give a well-trained policy πθ0 in Esource; 2. Create two empty replay buffers Dsource and Dtarget; 3. Initialize θ = θ0; while True do
4. Using πθ to interact with Esourceφ and push the generated trajectories into Dsource; 5. Using πθ to interact with Etarget and push the generated trajectories into Dtarget; 6. Sample a mini-batch {(s, a, s′)}source ∼ Dsource, and update V P source φ ,πθ by minimizing the
TD-error; 7. Sample a mini-batch {(s, a, s′)}target ∼ Dtarget, and apply the relative policy gradient in RPO to
update πθ; end while
Algorithm 3: Relative Transition Optimization (RTO) 1. Give the source and target environments Esource and Etarget, and their dynamics Psourceφ and Ptarget,
where the source dynamics Psourceφ is parameterized by φ; give an arbitrary policy πθ; 2. Create two empty replay buffers Dsource and Dtarget; 3. Initialize φ = φ0; while True do
4. Using πθ to interact with Esourceφ and push the generated trajectories into Dsource; 5. Using πθ to interact with Etarget and push the generated trajectories into Dtarget; 6. Sample a mini-batch {(s, a, s′)}source ∼ Dsource, and update V P source φ ,πθ by minimizing the
TD-error; 7. Sample a mini-batch {(s, a, s′)}target ∼ Dtarget, and update Psourceφ according to RTO or SL;
end while

heading: F OTHER EXPERIMENTAL DETAILS
text: For all the environments used in our experiments, the policy neural network and value neural network are of the same structure. The state is fed into three fully connected layers with ReLU activation function, and then the output embedding is fed into a policy head and a value head, respectively. That is, the policy network and value network share the bottom embeddings. For tasks with discrete action space, the policy head is a softmax layer to output multinomial distribution; for continuous control problems, the policy head is a diagonal Gaussian distribution that outputs a mean and a std.
The value head consists of two fully connected layers and finally outputs a scalar value. All the layers are of the same size of 64 in our experiments. For the physical dynamics model, each environment has its own physical systems coded in OpenAI gym. In CartPole-v0, only the pole length is treated as the trainable parameter, i.e., φ in CartPole-v0 only contains one free parameter. Similarly, in MountainCarContinuous-v0, only gravity is trainable; in Acrobot-v1, two physical factors link mass1 and link mass2 are trainable; in Pendulum-v0, the trainable parameter in φ is also the gravity.
For all algorithms, we choose their learning rate from [5e-4, 1e-4, 1e-5] with the best performance. For RPTO, its loss consists of the RPO loss and the RTO loss, and we simply fix both the loss coefficients as 1.0 in all experiments. For RPO, as mentioned in the main text, we clip the objective in Eq. (5) in a similar way as PPO did, and the clip range is set to 0.2, which is suggested by PPO.
For all the environments, we first pre-train a converged policy in the source environment using PPOzero, and then use the trained parameters as warm start for all algorithms in the policy transfer experiments. However, for MountainCarContinuous-v0, PPO-zero fails to learn a good policy that can solve the source task. This phenomenon has also been observed in many previous approaches considering deep exploration, where MountainCarContinuous-v0 has been considered as a typical task for deep exploration. Nevertheless, it is not our focus in this paper, and as long as we can obtain a good policy in the source environment, this policy can be used as a warm start for the policy transfer stage, no matter what method is adopted to generate this policy. Therefore, to solve the source MountainCarContinuous-v0, we first handcraft a rule that can succeed in the source task, and then employ imitation learning to obtain a parameterized policy, which is further fine-tuned using PPO-zero in the source MountainCarContinuous-v0 environment. The final converged policy is used as the warm start at the policy transfer stage. For other environments, PPO-zero can provide a good policy in the source task.

Review summaries:
The paper studies transfer in reinforcement learning (RL), beginning with a theorem that relates the performance of one policy under a particular dynamics to another policy under different dynamics. This is broken down into a “dynamics-induced gap” and a “policy-induced gap”, for which explicit expressions are given. Optimizing a bound on the policy-induced gap w.r.t. the policy leads to an algorithm they call Relative Policy Optimization (RPO), and similarly optimizing a bound on the dynamics-induced gap w.r.t. the dynamics leads to an algorithm they call Relative Transition Optimization (RTO). The two algorithms can be combined into a single algorithm, Relative Policy-Transition Optimization (RPTO), which optimizes both the policy and the dynamics. Experiments indicate that RPTO achieves better transfer performance, both in terms of sample efficiency and asymptotic performance, than RPO and PPO warm-started from an expert policy from the source task.

This paper proposed a way to decompose the difference between values of two policies in two MDPs respectively. Such a decomposition results in two parts, the first one is the difference between values of one policy under two MDPs; the second part is the difference between values of two policies under the same MDP. Using this decomposition, the paper then proposed three algorithms.   The first algorithm, called RPO, is used when there are two MDPs that the agent can interact with. The agent uses data from both two MDPs to get a good policy for one of the two MDPs. Such an algorithm is expected to be useful when gathering data from one MDP is costly while it from the other MDP is much cheaper.   The second algorithm is a model learning algorithm. In this setting, the agent only interacts with one MDP and learns a model to approximate the MDP. The RTO algorithm is different from the classic model learning algorithm (regression) in that the model is learned to achieve some consistency between the predicted values from the model and from the MDP.  The third algorithm combines the first two algorithms and is a full model-based algorithm. Specifically, now the agent only interacts with one MDP and learns a model of that MDP using RTO. Meanwhile, it also maintains and updates a policy that is expected to perform well in the MDP, using data from both the MDP and the model.