Adapt-to-Learn: Policy Transfer in Reinforcement Learning
Authors: Girish Joshi
Girish Chowdhary
Venue:  (2020)
Keywords: Transfer Learning
Reinforcement Learning
Adaptation

Abstract: Efficient and robust policy transfer remains a key challenge in reinforcement learning. Policy transfer through warm initialization, imitation, or interacting over a large set of agents with randomized instances, have been commonly applied to solve a variety of Reinforcement Learning (RL) tasks. However, this is far from how behavior transfer happens in the biological world: Humans and animals are able to quickly adapt the learned behaviors between similar tasks and learn new skills when presented with new situations. Here we seek to answer the question: Will learning to combine adaptation reward with environmental reward lead to a more efficient transfer of policies between domains? We introduce a principled mechanism that can \textbf{``Adapt-to-Learn"}, that is adapt the source policy to learn to solve a target task with significant transition differences and uncertainties.  We show through theory and experiments that our method leads to a significantly reduced sample complexity of transferring the policies between the tasks.

Under review as a conference paper at ICLR 2020
ADAPI-TO-LEARN: POLICY TRANSFER IN REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Efficient and robust policy transfer remains a key challenge in reinforcement learning. Policy transfer through warm initialization, imitation, or interacting over large set of agents with randomized instances, have been commonly applied to solve variety of Reinforcement Learning (RI.) tasks. However, this is far from how behavior transfer happens in the biological world: Humans and animals are able to quickly adapt the learned behaviors between similar tasks and learn new skills when presented with new situations. Here seek to answer the question: Will learning to combine adaptation reward with environmental reward lead to a more efficient transfer of policies between domains? We introduce a principled mechanism that can "Adapt-to-Learm" that is adapt the source policy to learn solve target task with significant transition differences and uncertainties. We show through theory and experiments that our method leads to a significantly re- duced sample complexity of transferring the policies between the tasks.
INTRODUCTION
Lack of principled mechanisms to quickly and efficiently transfer policies learned between domains has become the major bottleneck in Reinforcement Learning (RL). This inability to transfer or adapt policies is one major reason why RL has still proliferated physical application like robotics. Since RL agents cannot quickly transfer policies, the agent is forced to learn every task from scratch, which is both time and sample expensive. Warm-start method in which weights from one neural network are transferred to another, has been reasonably successful for supervised learning. However, this method can often lead to mixed and even negative results RI. (Joshi & Chowdhary 2018 Taylor & Stone 2009).
Our main contribution is an algorithm to transfer policies between tasks with significant differences in state transitions via policy adaptation mechanism. Unlike the majority of existing work in trans- fer learning for RL, our approach does not merely use the transferred policy to warm start (initialize the parameter of the target network with learned source network) policy learning in the target do- main. Neither does it rely on a multitude of simulations across randomly generated source domains. Instead, we combine supervised reference trajectory tracking and unsupervised reinforcement learn- ing to adapt the source policy to the target domain directly. We show through theory and experiments that our method enjoys significantly reduced sample complexity in solving the task.
Adapt-to- Lear is inspired by the fact that combined adaptation of behaviors and learning through experience is a primary mechanism of learning in biological creatures (Krakauer & Mazzoni 2011] Fryling et al. 201T). (ooTT)Insrreed by this ability of biological creatures, we seek to answer the question:
learning to combine intrinsic adaptation reward with environment reward lead more efficient transfer of policies between domains? Imitation Learning (IL) (Duan et al.) 2017)) Zhu et al.) 2018) seems to play crucial part in biological learning, and as such has been widely studied in RI. How- ever, the key is, when presented with new situation, animals do not just imitate, but quickly adapt existing behaviors, and improve them through further experience. In particular, an animal learning to walk different terrain does not just imitate its existing gait, but adapts it to the new environ- ment.The theory behind such adaptation in reference tracking control problems has been typically restricted to deterministic dynamical systems with well-defined reference trajectories (Aström & Wittenmark 2013] Chowdhary et al. 2013). This ability to adapt and incorporate further learning through optimization on the environment reward is one key difference between method and ex-

---

Under review as a conference paper at ICLR 2020
isting imitation learning and Guided Policy Search (GPS) methods (eevine & Koltun 2013). Unlike IL and GPS, our method transfers policies between task with significant differences in the transition models. Moreover, by mixing environment reward with intrinsic adaptation rewards, the agent quickly adapts and also learns to acquire skills beyond what the source policy can teach. We posit that the presented method can be the foundation of broader class of RL algorithms that can choose seamlessly between learning through RL to supervised adaptive imitation. Our empir- ical results show that approach is capable of robustly transferring policies between tasks, even in the presence of nonlinear and time-varying differences in the dynamic model of the systems. In particular, we show that it suffices to execute adapted greedy policies ensure c-optimal behavior in the target domain.
ensure that
D-RL has recently enabled agents to learn policies for complex robotic tasks in simulation (Peng et al., 2016; 2017b; Liu & Hodgins 2017 Heess et al. (017). However, D-RI. has been plagued by the curse of sample complexity. Therefore, the capabilities demonstrated in the simulated environment are hard to replicate in the real world. This learning inefficiency of RL has led to significant work in the field of TL (Taylor& Stone, 2009). A significant body of literature on transfer in RL is focused on initialized RL in the target domain using learned source policy; known as jump-start/wan sttart/warm-star start methods (Taylor et al. 2005 AAmmar et al.) 2012] 2015). Some examples of these transfer architectures include transfer between similar tasks (Banerjee & Stone, 2007). transfer from human demonstrations (Peters & Schaal 2006) and transfer from simulation to real (Peng et al. 2017al Ross et al.) [011]] Yan et al.J 2017). Efforts have also been made in exploring accelerated learning directly on real robots, through Guided Policy Search (GPS) (Levine)
Related work:
al. 2015) and parallelizing the training across multiple agents using meta-learning (Levine al.) Nagabandi et al., 2018: Zhu et al., 2018). Sim-to-Real transfers have been widely adopted in the recent works and can be viewed as subset of same domain transfer problems. Daftry et al. (Daftry et al.] 2016) demonstrated the policy transfer for control of aerial vehicles across different vehicle models and environments. Christiano et al. (Christiano et al. 2016) transferred policies from simulation to real using an inverse dynamics model estimated interacting with the real robot. Through learning over an adversarial loss, the agents are trained to achieve robust policies across various environments (Wulfmeier et al.) 2017). However, these and other reported architectures do not necessarily lead to improved sample efficiency, handle relatively minor changes in the transition model, and even known to cause negative transfer. In contrast, our approach directly adapts the source policies to target with significant transition model difference while interacting with the environment. It enjoys empirically and rigorously proven sample efficiency guarantees of order OnHH), depending polynomially on the horizon length "H"
2016
2 PRELIMINARIES
Consider a finite horizon MDP defined as a tuple (S,A, R, Po: 7), where S denote set of continuous states; A is a set of continuous bounded actions, X AXS R4 is state transition probability distribution of reaching s' upon taking action a in s, Po S - IR is the distribution over initial states so and R. S A + R4 is deterministic reward function and H. be the finite horizon of the problem.
Let x (als): S x A + j0, 1] be stochastic policy over continuous state and action space. The action from policy is a draw from this distribution a; ~ T( (s). The agent's ss goal is to find a policy n' which maximize the total return. The total return starting from states so ~ po under policy л is
H
mh (5o) = Eeo.ao...
Er(st,an ac)
(1)
where, Su ~ Pu. dg ~ ( (a:]st) and 8141 ~ (($14115600 (s,, ax).
We formalize the underlying problem of policy transfer by considering source and target MDP as follows, M = A, R, 7).7), "),,M" MM = AS R, P. 7T each with its own state, action space and transition model respectively. We will mainly focus on the problem of same domain transfer in this paper, where the state and action space are analogous g(S) ((2) = S E Rm and TT = (8)) A R& but the source and target state transition models differ significantly due to unmodeled dynamics or external environment interactions. Furthermore, an example of how the method can extend to cross-domain transfer using manifold alignment (Joshi & Chowdhary
2

---

Under review as a conference paper at ICLR 2020
2018 Wang & Mahadevan 2009) is in the supplementary. Let σ' be a parameterized optimal stochastic policy for source MDP M The policy π can be obtained using any available RL methods (Sutton al, 2000, Schulman et al., 2015:2017 In this work, we have used the Proximal Policy Optimization (PO) (Schulman et al. 2017) algorithm to generate the source optimal policy. We will the following definition of the state value function V defined under any policy
H Σ
(T((s) E Bor st
Er(stte:0s aet1)
The associated optimal value function fo source MDP M is V* for all s S. Let T 12 - 12" be the Bellman update operator defined as
(T)))) max aEA r(s,a) +7
S(0))
P(saa))))
Optimal value function V satisfies the Bellman Equation
2 such that V (s)= (s),
3 ALGORITHMIC CONTRIBUTION: TRANSFER THROUGH ADAPTATION
3.1 ADAPTATION AS MECHANISM FOR POLICY TRANSFER IN RL
In this paper, we approach the problem of transfer learning for RL through adaptation of previously learned policies to related tasks. Our approach to adaptation is by enabling the agent to learn the best mixture of imitation and learning from environment reward. Our method differs from RL transfer methods that rely on jump-starts direct imitation (Zhu al. 2018 Duan et al. 2017 in a key way: We do not aim to emulate the source optimal policy itself in the target domain, but the source transitions under optimal policy. To make this point, we demonstrate in our empirical results that the source policy, when used directly, does not produce sensible behaviors of the agent in the target domain when transition models are significantly different. On the other hand, our method uses the source transitions projected onto the target task as reference exploration trajectories, helping to adapt and optimize the source policy to the target domain efficiently.
In canonical reinforcement learning, the goal is to learn the optimal policy x" which maximizes the future cumulative reward when the reward function and the transition function of the MDP are unknown. In statistical RL, this leads to the (in)famous explore-exploit tradcoff. Yet, this learn- through-experience approach has demonstrated significant potential to enable general solutions to control of complex physical systems for which first principles-based models are not easily obtain- able. However, one the key challenges in RL has been the high sample complexity of obtaining a reasonable policy. In other words, RL, and specifically Deep RL, require absurdly huge amount of interactions with the environment (experience) to find reasonable policies. This high learning inefficiency is in apparent contrast with the efficient learning demonstrated by humans and many biological creatures. For examples, octopuses are known to learn to solve complex problems such as opening jars and squeezing through puzzles with just few trials. This ability to quickly transfer and adapt learned policies between related tasks is likely key to general intelligence that seems to be missing from RI. Our goal the rest of this paper to show that an algorithm that can judiciously combine adaptation to changes and learning new skills is capable of avoiding brute force random exploration to a large extent and be significantly less sample expensive.
3.2
ADAPT-TO-LEARN POLICY TRANSFER
We begin by mathematically describing adaptation for policy transfer in RL and state all the neces- sary assumptions in Section 3.2. We then develop the Adapt-to-Learn algorithm in Section 3.2.2
3.2.1 ADAPTATION
The proposed Adapt-to-Lent Learn (ATL) algorithm objective is to accumulate higher total returns through maximizing the likelihood of the target policies mo. which emulate the optimal source transi- tion behavior in the target MDP. We achieve this objective by minimizing the KL divergence between point-wise local trajectories realized using the target policy and source optimal policy in the target
3

---

Under review as a conference paper at ICLR 2020
domain (Refer F in Appendix- C). This adaptation objective can be formalized minimizing the KL-divergence between source and target transition trajectories:
Piro (T)
Mx (To, 7t
E Se.R1
Pro (T)
log
(3)
Tx (T)
where T is the trajectory in the target domain under the policy we defined as
(so, Go $1, a1....);
and probability of the trajectory Pro (T) under policy mo and target transition p (1/s,, az) can be written
M
Pro (T) = p(so) II
To To(a (st+1ls;,
(4)
Similarly yr- (T) can be defined probability of trajectory deviations every state 8 т, when the source optimal policy x² is used in place of target policy and the states evolve according to the source transitions model pd (.|se, (sse,a, at) (Refer Figure-3 in Appendix-C):
H
4- 4x- (T) - p(so)
II t
(5)
Assumption-l: We assume that optimal source policy is available, such that the source policy vari- ance can be assumed reasonable assumption, since optimal source policy is available it can be treated as deterministic policy and actions can be chosen greedily. We need this assumption only for deriving the expression for intrinsic reward and theoretical analysis. In the empirical evaluation of the algorithm, we treat T (.) as a stochastic policy.
zero. That is, the action probabilities n- =a((((sx)) = 1, Vs,. This is
The transition probabilities in the KL divergence in term are treated transition likelihoods and the transitioned state st+1 as a random variable. Since s'4 is the optimal state reached starting in st under source transition model using x" (a( and hence evaluate the transition likelihoods of the target trajectory at 14 Using the definition of the probabilities of the trajectories under Te and n² Equation-44 & (5) the log term in the KI. divergence of the trajectory (3) is simplified as follows
try to emulate this behavior in the target domain
Pro (T) An (7)
(cols))())))))) (s (o) (s)p/ (s)
log
log
((s0) (ablso)) ( 100 (a) (1](s))p" (1/1 af) (1)1s).
(6)
Using 6 and with assumption-1 i.e. (a{(s)) 14At the KL term can simplified as follows
H
(ra(aels))p (6/411 as) d (5'+ (6/4111841 dt) ))
Mx ( )²)
EE ee
Pno (T)
log
18,,
t=
H Pro (T)
7x (TT,, TT) )
B. seraee
1=0
TT(aas/s)pp" (0/ (s))as) (8/ (5)
where Çe = loog
If calculating the above expectation feasible, possible minimize the KL divergence and move the policy parameters in the direction of emulating source transition behavior in the target domain. However, this not generally the case since the true expectation intractable. Therefore common practice is to use empirical estimate of the expectation to do approximate planning. We minimize the trajectory KL divergence by handling the term & as the intrinsic adaptation reward, which captures the local deviation in trajectory in the form of the shaped reward function.
Assumption-2: We do not assume to know the true transition distribution for source and the target, but only have access to simulator model of the source. However, assume both source and target transition models follow a Gaussian distribution centered at the next propagated state and fixed variance "g"

---

Under review as a conference paper at ICLR 2020
Assumption-2 is not very restrictive since empirically show that for any deterministic model, bootstrapped Gaussian transition assumption is sufficient for ATL agent to learn the task. Using the above assumption, we approximate the KL term
C = log To (T (1)_ 1202
The individual terms in the expectation Çe represent the distance between two transition likelihoods of landing in the next state 8/41 starting in & and under actions a, a/- The target agent is encouraged to take actions that lead to states which are close in expectation to a reference state provided by an optimal baseline policy operating the source model. By doing so, we are providing a possible direction of search for the higher environmental rewards "ræ".
We can, therefore, solve the following optimization problem to generate adaptive policy updates: +7' #
arg min (Mxz).
The distance between two transition models Çi used as an intrinsic reward to calculate the total trajectory intrinsic return, and further, any policy update algorithm can be used (Sutton et al. 2000 Schulman al., 2017 2013) update the policy direction optimizing this objective. However, is to be noted that though we use policy gradient kind of update for adaptive policy, the optimization more akin to supervised learning. For every state-action pair, the KL-distance is the true metric to be minimized, and unlike RL, we do not engage in optimal value search in the adaptation part of learning. Hence the algorithm is more sample efficient compared to any RL policy search methods.
3.2.2 ADAPTATION AND LEARNING
We achieve Adaption and Learning simultancously by augmenting environment reward ra with in- trinsic reward G. By doing so, we achieve transferred policies which try to both optimize optimal reference tracking and also maximize the cumulative future environmental reward to acquire skills beyond what source can teach. This trade-off between learning by exploration and learning by adaptation can be realized as follows:
EE Se,40
Proo (T)
BGi)
(10)
1=
Where the term 3 is the mixing coefficient. We make a heuristic choice for an appropriate 3 to start with and annealed over episodes for optimal mixing of adaptation and learning. For consistency of the reward mixing, the rewards Tı, Sa are normalized to form the total reward r = (1 - 8)r, : + BG..
Algorithm Adaptive-to-Leam Policy Transfer in RL
Require: n" x'),pp
> Inputs: Source Policy, source simulator > Draw initial state from the given distribution in target task
1: Initialize s/ Pa-
2: for irri= 1 < do
3: while 8; + terminal do
4: df n (s.)
>> Generate the optimal action using Source policy >> Generate the action using the Ae Propagate the target task model at state and action a, b Propagate the source task model state s; and action of
5: a; (s()))
6:
8;+1 p" (s;, a;) 1/11 pd (st, a
7:
8:
G (a(aissi)
) /20
Calculate the KL divergence intrinsic reward term b Incrementally store the trajectory for policy update
9:
Z, = ({si, ri, Si, Oj. d/
10:
Pz (Mwx) 1 Em Eie V log (a,,,,) E r'. >> Empirical loss to optimize
#o eag.aax Pz (Txz)
Minimize the loss obtain adaptive policy
3.3 SAMPLE- BASED ESTIMATION OF THE GRADIENT
The previous section proposed an optimization method to find the adaptive policy using KL- divergence intrinsic reward, enforcing the target transition model to mimic the source tran- sitions. This section describes how this objective can be approximated using a Monte Carlo simu- lation. The adaptive policy update methods work by computing estimator of the gradient of the
5

---

Under review as a conference paper at ICLR 2020
return and plugging it into stochastic gradient ascent algorithm.
T #
arg max Pz" (7xx)
(11)
na Ell
0 ag.
Where a is the learning rate and I is the empirical estimate of the gradient (Refer Appendix-A) of the total return MKL. The gradient estimate over i.idd data from the collected trajectories is computed as follows:
g 9= P-
E
Σ V log (ar())))
H
Pz- (V (VoTKL)
Vo log No (ar,)
Σ r/
(12)
T
1=1
t=
1=0
where Pzn is empirical distribution over the data ( {8t, ai (.)))
4 THEORETICAL BOUNDS ON SAMPLE COMPLEXITY
Although there is some empirical evidence that transfer can improve performance in subsequent reinforeement-lerming tasks, there not many theoretical guarantees. Since many of the existing transfer algorithms approach the problem of transfer as a method of providing good initialization to target task RL, we can expect the sample complexity of those algorithms to still be a function of the cardinality of state-action pairs [ = SS] x [A]. On the other hand, in supervised learning setting, the theoretical guarantees of most algorithm have no dependency on size (or dimensionality) of the input domain (which is analogous to |N| in RL domains). Having formulated a policy transfer algo- rithm using labeled reference trajectories derived from optimal source policy in ATL, we construct supervised learning like PAC property of the proposed method. For the sample complexity analysis we consider only the adaptation part of learning i.e. , = Equation- (quation-(16). This is because in ATI., the adaptive learning is akin to supervised learning, since the source reference trajectories provide the target states given every (st,a;) pair.
Suppose we are given the learning problem specified with training set Z = (Z1,... Zn) where each Z, = ((5t, a,, H are independently drawn according to some distribution P. Given the data 2" can compute the empirical return Pz (1xx) for every Te II, and we will show that the following holds:
PZ (1xce) P( ( S
(13)
with probability at least 1 - 6, for some very small 5 s.t 0 S 5 S 1. We can claim that the empirical return for all my is sufficiently accurate estimate of the true return function. Thus a reasonable learning strategy is to find
IT that would minimize empirical estimate of the objective (T0).
Theorem 4.1 If the induced class of the policy no Lu has uniform convergence property in empir- ical mean; then the empirical risk minimization PAC. s.t
P" P'(((((a..*)) -() )21)) 2))
(14)
and number of trajectory samples required can be lower bounded as
C2
I
nee, 8) 2
log
(15)
2
For the proof of the above theorem refer the Appendix-a
4.1 G-OPTIMALITY RESULT UNDER ADAPTIVE TRANSFER-LEARNIN
Consider MDP M and M which differ in their transition models. For the sake of analysis, let M be the MDP with ideal transition model, such that target follows source transition p" precisely. Let p be the transition model achieved (tracked) using the estimated adapted policy learned over data interacting with the target model and the associated MDP be denoted as M. We analyze the c-optimality of return under adapted source optimal policy through proposed policy transfer.

---

Under review as a conference paper at ICLR 2020
Note for the sake of analysis are deviating from finite horizon assumption and using discout factor
7. To be consistent with previous section we can assume the horizon length H logy 2V where 6, is truncation error in total infinite return.
Definition 4.2
Given the value function V* = V= and model M1 and M2. which only differ in the corresponding transition models p1 and p2. Lets define Vs, S A
d' dai,Ms M2
IV (s')
IV (s'))
sup s,0ESXA (1,a)
p'
Lemma 4.3 Given M M and value function V V M
the following bound holds
VT- Vi S ne (I-7)²
where max,,a [1 a) p" (.18, S e and ş and p" are transition of MDP M, M respectively. The proof of this lemma is based on the simulation lemma (see Appendix-C), Similar results for RI. with imperfect models were reported by (Jiang 2018).
POLICY TRANSFER IN SIMULATED ROBOTIC LOCOMOTION TASKS
To evaluate Adapt-to-Learn Policy Transfer in reinforcement learning, design experiments using sets of tasks based on the continuous control environments in MuJoCo simulator (Todorov) al. 2012). Our experimental results demonstrate that ATL can adapt significant changes in transition dynamics. Therefore, we perturb the parameters of the simulated target models for the policy transfer experiments (see Table-[ original and perturbed parameters of the target mode). To create challenging training environment, we changed the parameters of the model such that the optimal source policy alone without any learning cannot produce any stable results (see source policy performance in Figure-T]. We compare our results against two baselines: (a) Initialized Reinforcement learning (initialized PPO) (Jumpstart-RL. (Wang Mahadevan, 2009)) (b) stand- alone reinforcement policy learning (PPO) (Schulman et 2017).
We experiment with the ATL algorithm on Hopper, Walker2d, and HalfCheetah Environments. The states of the robots are their generalized positions and velocities, and the actions are joint torques. High dimensionality, non- non-smooth dynamics due to contacts and being under-actuatedd systems make these tasks very challenging. We use deep neural networks to represent the source and target pol- icy, the details which are in the Table-2 Appendix-I. The following models are included in our evaluation:
Slippery Hopper: is defined through 1 11-dimensional state space and 3-dimension action space, with reward function defined as r(t) (st+1 8,) s,)/dd dt 10 "|a||||, and bonus of +1 for being in non-terminal state. The simulation is terminated upon reaching 1000 steps or hopper toppling. The target model differs in the floor friction and foot joint damping.
Slippery Fat- Walker2d: is defined through 17-dimensional state space and 6-dimension action space, with reward function and termination condition defined same as Hopper. The target model differs in the model density and floor friction.
Fat HalfCheetah: is defined through 17-dimensional state space and 6-dimension action space, with reward function defined as r(t) = (st+1 st) st)/dt /tt 0.1 ||]////|2. The simulation is terminated upon reaching 1000 steps. The target model differs in the floor friction coefficient, gravity, and mass.
To establish standard baseline, we also included the classic cart-pole and Inverted pendulum bal- ancing tasks, based on the formulation (Barto et al. 1983). We also demonstrate the cross-domain transfer capabilities using model-based variant of the proposed algorithm. The results of policy transfer for Cart-Pole to Inverted Pendulum and Inverted Pendulum to Bicycle transfers are provided in the Appendix-D. Learning curves showing the total reward averaged across three runs of each al- gorithm are provided in Figure I Adapt-to Learn policy transfer solved all the three tasks, yielding quicker learning compared to other baseline methods. These results provide empirical evidence of our hypothesis. Using trajectory KL divergence as intrinsic adaptation reward to adapt source pol- icy the target, achieve more robust and sample efficient policy transfer between two tasks,
7

---

Under review as a conference paper at ICLR 2020
Env Hopper
Property
source
Target 2.0
%Change
Floor Friction
1.0
+100%
HalfCheetah
gravity
-9.81
-15
+52%
Total Mass
14
+150%
Back-Foot Damping
3.0
1.5
-10%%
Floor Friction
0.4
0.1
-75%
Walker2d
Density
1000
1500
+50%
Right-Foot Friction
0.9
0.45
-50%
Left-Foot Friction
1.9
1.0
-47.37%
Table I: Transition Model and environment properties for Source and Target task and % change
compared using warm-start standalone RL method. Note that the target domain perturba- tions introduced are significant enough such that source policy alone without any adaptation in the target domain produced no meaningful results (Figure This notion of adaptation in the face of uncertainty is key advancement over traditional policy transfer, meta-learning, or adversarial RL methods aiming to improve performance by learning a policy over set lightly perturbed tasks.
Steper, Aerooat
2014
2000
3480
18:00
All Jouns
3000
Eary
(8)
(b)
(c)
Figure 1: Learning curves for locomotion tasks, averaged across three runs of each algorithm with random Initialization for RL, warm initialization using source policy for ATL and Jumpstart( (arm- Start) methods and Source policy performance in Target Task without any adaptation.
Hartheaat
-32
-00
-20
-40
-200
-390
-00
-00
-375
2000
2000
2000
300
15000
2000
1000
1300
1300
(b)
(c)
Figure 2: Trajectory KL divergence Total Intrinsic Return (-Σes) averaged across three runs.
CONCLUSION
We introduced a new transfer learning technique for RL: Adapt-to-Learn, that utilizes adaptation of the source policy to target tasks. We demonstrated on nonlinear and continuous robotic locomotion tasks that learning to adapt source policy to the target domain leads to a significant reduction in sample complexity over the prevalent jump-start based approaches. We further also proved theo- retical guarantees on the reduced sample complexity of our proposed architecture. There are many exciting directions for future work. A network of policies that can generalize across multiple tasks could be learned based on each new adapted policies. How to train this end-to-end is an important question for meta-learning. The ability of Adapt-to-Learm handle significant perturbations the
8

---

Under review as a conference paper at ICLR 2020
transition model indicates that should naturally extend to sim-to-reat transfer. Indeed argue that such adaptation necessary for real-world robotics, as has been established previously in classical domains like flight control. Another exciting direction is to extend the work to other combinatorial domains (e.g., multiplayer games). We expect, therefore follow on work will find other exciting ways of exploiting such adaptation in RI and machine learning.
REFERENCES
Haitham Ammar, Karl Tuyls, Matthew Taylor, Kurt Driessens, and Gerhard Weiss. Reinforce- ment learning transfer via sparse coding. In Proceedings of the 11th International Conference Autonomous Agents and Multiagent Systems-Volume 1, pp. 383-390, International Foundation for Autonomous Agents and Multiagent Systems, 2012.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Proc. of AAAI, 2015.
Karl Áström and Björn Wittenmark. Adaptive control. Courier Corporation, 2013.
Karl J Åström, Richard Klein, and Anders Lennartsson. Bicycle dynamics and control. IEEE
Control Systems Magazine, 25(4):26-47, 2005.
Bikramjit Banerjee and Peter Stone. General game learning using knowledge transfer. In IJCAI, pp- 672-677, 2007.
Andrew Barto, Richard Sutton, and Charles Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions systems, man, and cybernetics, (5):834-84, 1983.
Girish Chowdhary, Tongbin Wu, Mark Cutler, and Jonathan P How. Rapid transfer of controllers between uavs using learning- adaptive control. Robotics and Automation (ICRA), 2013 IEEE International Conference on, pp. 5409-5416. IEEE, 2013.
Paul Christiano. Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from simulation to real world through learning deep inverse dynamies model. arXiv preprint arXiv:1610.035/8 2016.
Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monoc- ular reactive mav control. In International Symposium on Experimental Robotics, pp. 3-11. Springer, 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pp. 1087-1098. 2017.
Mitch J Fryling, Cristin Johnston, and Linda J Hayes. Understanding observational learning: An interbehavioral approach. The Analysis of Verbal Behavior, 27(1):191-203 2011. 2011.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: / 0200002066,... 2017.
Nan Jiang. Pac reinforcement learning with an imperfect model. In Proc. of AAAI, 2018.
Girish Joshi and Girish Chowdhary. Cross-domain transfer in reinforcement learning using target apprentice. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7525-7532. IEEE, 2018.
John W Krakauer and Pietro Mazzoni. Human sensorimotor learning: adaptation, skill, and beyond. Current opinion in neurobiology. 21(4):636644, 6444, 2011.
Sergey Levine and Vladlen Koltun. Guided policy search. International Conference on Machine Learning, pp. 1-9, 2013.

---

Under review as a conference paper at ICLR 2020
Sergey Levine, Nolan Wagener, and Pieter Abbeel. Learning contact-rich manipulation skills with guided policy search. In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pp. 156-163. IEEE, 2015.
Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with large-scale data collection. In International Symposium on Experimental Robotics, pp. 173-184. Springer, 2016.
Libin Liu and Jessica Hodgins. Learning to schedule control fragments for physics- characters using deep g-learning. ACM Transactions on Graphics (TOG), 36(3):29, 2017.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model- deep reinforcement learning with model-free fine-tuning. In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG), 35(4):81, 2016.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. arXiv preprint arXiv: 2017a.
Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning, ACM Transactions on Graphics (TOG), 36(4):41, 2017b.
Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 2219 2225. IEEE, 2006.
Jette Randlov and Preben Alstrom. Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the Fifteenth International Conference on Machine Learning, pp- 463-471, 1998.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 027-635,70 2011.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Richard Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural informa- tion processing systems, pp. 1057-1063, 2000.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey.
Journal of Machine Learning Research, 10(Jul): 1633-1685, 2009.
Matthew E Taylor, Peter Stone, and Yaxin Liu. Value functions for rl-based behavior transfer: A comparative study. In Proceedings of the National Conference Artificial Intelligence, vol- ume 20, pp. 880. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Chang Wang and Sridhar Mahadevan. Manifold alignment without correspondence. In IJCAI, volume 2. pp. 3. 2009.
Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. arXiv preprint arXiv: 1007::770700700 007907 2017.
10

---

Under review as a conference paper at ICLR 2020
Mengyuan Yan, luri Frosio, Stephen Tyree, and Jan Kautz. Sim-to-real transfer of accurate grasping with eye-in-hand observations and continuous control. arXiv preprint arXiv:1712.03333, 2017.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran. Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. arXiv preprint arXiv: 2018.

---

Under review as a conference paper at ICLR 2020
A TOTAL RETURN GRADIENT WITH RESPECT TO POLICY PARAMETERS
The total return which we aim to maximize in adapting the source policy to target is the mixture of environmental rewards and Intrinsic KL divergence reward as follows,
Txz (T π.
E
Puo
(16)
t=0
he adaptive policy update methods work by computing estimator of the gradient of the return and plugging it into a stochastic gradient ascent algorithm.
#
arg max Pz (7xx) me ETE
(17)
0 == 0 + ag
where is the learning rate and 9 is the empirical estimate of the gradient of the total discounted return 4z- The derivation of the gradient term is exactly same as for policy gradient methods.
Taking the derivative of the total return term
H Σ
Vo([k))=
= E Se.lkeru
VePro (T)
1/
Consider the term VoParo (T) and evaluate the derivative. the definition of Pro (T) and we can write the log term as
Lets consider the term log Pro (T). Using
log Pro (T) = log log(Poto(a0/ (81 10, a0)
H
H
log Pxo (T) = log(Po)
log No (a)1))) + Σ
Σ 1=
p" (84+1186, at).
1=
Taking the derivative wrto 0 can write,
H Vo log (arls))
V log
=
t=
VOP (T)
H v log (s)))
Poo (T)
1=
Therefore we can write
H
VePro (T) = Puo (T)
V 9 log (a)
t=
Substituting the above expression write the total gradient as,
H
H
Vo(Tx)) = EpTr
V log To (a(a)(s))
Σ 1
B PROOF OF THEOREM-4.1
Theorem B.1 If the induced class Lu has uniform convergence in empirical mean property then empirical risk minimization is PAC.
For notation simplicity we drop the superscript T (for Target domain) and subscript (policy pa- rameters) in further analysis. Unless stated will using following simplifications A म and
n ng
12

---

Under review as a conference paper at ICLR 2020
Proof Fix o 6.30 > 0 we will show that for sufficiently large 2 n(c, 8)
("((([ - -- 2() S
(18)
Let x² II be the minimizer of true return P(Nx)), further adding and subtracting the terms Pz ( and Pzn (1 we can write
P( P(oKE.))) P(N Pzz (MKE,))) Pz (1x...)) Pzu (Mxa,))) Pzn (1x,,.)) P( )
(19)
We can divide the above expression in three terms
1. P(TKLA))) )- - Pz (1K2
2. Pzn (MxL,:-) Pzn (MKL...)
3. Pz (MxL.-)) - P(1KE)))
Lets consider the term Pzu (1x1...) ) Pzn (Ixc...) in the above expression is always negative semi- definite, since A* is a maximizer wrto Pz (MKL): hence Pz (1x2,. I Pz (MKL,.)) always, i.e
Pz- (T=I,, - Pz (1 00
Next the 1st term can be bounded as
P(NKL.))) Pz (M S sup Pz (Mx1) P( S sup P2 (1KL) - P(mxz)) )
#ell
fell
Similarly upper bound can be written for the 3rd term Therefore we can upper bound the above expression as
P( P(Ne.....) S 2 sup
PP (Tlxe) P(MK2)ll
(20)
=e1l
From Equation-
18
we have
sup (T P(Tx)) 1ll 2 c/2 nell
(21)
Using McDiarmids inequality and union bound, we can state the probability this event as
Pl (Ner.) - P(Tx.. )l 2 e/2) S 2|11|e .,-1
(22)
where C' bound on state space S and H. is horizon length. The finite difference bound C' is obtained assuming transition p" pd follow a Gaussian distributions and the i.i.d trajectories collected are of maximum H length.
Equating the RHS of the expression and solving for we get
C2 2 log
2 8
6) 2
(23)
for n 2 n((e d) the probability of receiving bad sample is less than The total number of observed transition of order O(nH). Refer supplementary document for details of the proof.
C PROOF OF LEMMA-4.3
Lemma C.1 Given M* M and value function V V the following bound holds
VI..
VT" M
ao S [I-7)=.
[[](|s, a) p² (-18,, and á and p" are transition of MDP M, M respectively. Note we are deviating from finite horizon assumption and using 7. To be consistent with previous section we can assume the horizon length H. = log, 2Vmaz infinite return.
Here max,,a a
Where e, truncation error total
13

---

Under review as a conference paper at ICLR 2020
Proof For any S
MV (8) - V (0)lox
|r(s,a) + (p(s)ls,a), + (1) r(s,a) - -7 ()) 1o0
Add and subtract the term 1r ( a), Mr (s))
h (p(o'lo, V (6)) - (
(6') + + (p' (6)) () loo
(5)) (p) 600 S >10(6 p" (6' a)) V (8)) + (s) (8)
s > (p(5'ss,,)) (6))
(p'('' (
Using the definition of in above expression,
write
M (8) v (8) S T (6)) + v (s) V (s)
Therefore
7 (8))
M (s) Var (8) I
1-7
Now we solve for expression
V M (s)) We know that this term is bounded
Ranar
I (s))
where Remax 1, therefore write the complete expression
VV Air (s) VM)((s) S
76
(1 71-
p² 7 (s1l30-00 ag
Q1
82
0/2
83
d1
80
D (5/18800 db)
d/ nt (8o)
S/2
8
Po
Figure 3: Target Trajectory under policy Ty and local trajectory deviation produced by source opti- mal policy π and source transition pd
D ADDITIONAL RESULTS AND DETAILS OF EXPERIMENTAL DOMAIN
To establish standard baseline, we also include the results for classic cart-pole and Inverted pen- dulum balancing tasks, based on the formulation Barto et al. (183)). We also demonstrate the cross- domain transfer capabilities using variant of the proposed algorithm Joshi & Chowdhary (2018) and the results for Cart-Pole to Inverted Pendulum and Inverted Pendulum to Bicycle transfers are provided in the following sections.
14

---

Under review as a conference paper at ICLR 2020
Hopper
Walker2d
HalfCheetah
State Space
12
18
17
Control Space
3
6
Number of layers
3
3
3
Layer Activations
tanh
tanh
tanh
Total num. of network params
10530
28320
26250
Discount Learning rate
0.995
0.995
0.995
1.5x 10-
8.7x10
9x10-6
3 init
1.0
1.0
1.0
B-anneal coeff
0.998
0.998
0.998
Batch size
20
20
5
Policy Iter
3000
5000
1500
Table 2: Policy Network details and Network learning parameter details
14.4
-0.1
14.2
-0.2
Rvg Reward
14
13.6
13.4
-0.5
.0.6
Adoptation
50
100
150
200
I-L
episodes
(b)
Figure 4: Policy Transfer from Inverted Pendulum to Non-stationary Inverted pendulum: (a) Aver- age Rewards and (b) Training length, TA-TL(ATL ours), UMA-TL(Jumpstar-RL and Stand-alone RL
D.1 INVERTED PENDULUM (IP) TO TIME-VARYING IP
We demonstrate approach for continuous state domain, Inverted Pendulum (IP) swing-up and balance Figure-44 The source task is the conventional IP domain. The target task differs from the source task in the transition model. The target task is non-stationary inverted pendulum, where the length and mass of the pendulum are continuously time varying with function L, = Lo +oo+0.. +0.ccos( gi)
and M; Mo + 0.5cos( mi). where Lo = 1. Mo = 1 and = 1...N. The state variables describing the system angle and angular velocity (0,0) [_,, The RL objective to swing-up and balance the pendulum upright such that 0 0, 0. The reward function is selected as R(e, 6) 1 5) which yields maximum value upright position and minimum the down-most position. The continuous action space is bounded by T [-1, 1]. Note that the domain tricky, since full throttle action assumed not generate enough torque to be able to swing the pendulum to the upright position, hence, the agent must learn to swing the pendulum back and forth and leverages angular momentum to go to the upright position.
D.2 ROBUSTNESS TO NEGATIVE TRANSFER
We demonstrate that the proposed transfer is robust to negative transfers, that is, cases in trans- fer learning where naive initialization of the target using source policy could be detrimental. We demonstrate this through inverted pendulum upright balance task with the sign of control flipped. That is, we use an inverted pendulum model as both source and target systems, but the target the inverse of the source model with the sign of the control action flipped. It should be noted that deal- ing with a change of sign in control has been considered a highly challenging problem in adaptive control Chowdhary al. (013). We demonstrate that ATL is immune negative transfers. Figure
15

---

Under review as a conference paper at ICLR 2020
14.5
-02
Avg Reward
104
14
og(N)
0.8
RL (no Transter UMA-TL
TA-TL
-0.0
13
0
50
100
150
200
260
episodes
(a)
(b)
Figure 5: Negative Transfer Inverted Pendulum: (a) Average Reward and (b) Training length
Pa
375
320
300
(a)
(b)
Figure 6: Adap-to-Learn Policy Transfer (a) Pennuulu (b) Cart-Pole
5a and 5b demonstrate convergence to average maximum reward with lesser training samples for proposed ATL method compared Initralized-RLLUUMATTT and standalone-RL methods. It is to be observed that UMA-TL method suffers from a negative transfer phenomenon. The agent under UMA-TL converges to much lower average reward by getting stuck in local minima and never achieves the upright balance of the pendulum.
Also, the samples observed by UMA-TL learning the task is much higher compared no transfer (RL) and proposed ATL methods. If the source and target are not sufficiently related or the features of source task do not correspond to the target, the transfer may not improve or even decrease the performance in target task leading to negative transfer. We show that the UMA-TL suffers from a negative transfer in these results, whereas the performance of presented ATL much superior compared to UMA-TL and RL(leanning from scratch).
E
CROSS DOMAIN TRANSFER
E.1 CART-POLE TO BICYCLE:
Bicycle balancing is challenging physical problem, especially when the bicycle velocity is below the critical velocity V. = 4m/s to 5m/s. We set the bicycle velocity to be V < V. i... V 2. 2.778m/ such that the bicycle becomes unstable, and active control is required maintain stability. The simulation itself is very high fidelity and realistic, which was designed for studying the physics of the bicycle.
16

---

Under review as a conference paper at ICLR 2020
211
Avg Reward
Policy Acaptation
-9
RRL
Indallzdd RL
800
1000
1500
2000
episodes
(b)
Figure 7: (a) Cart-Pole and Bicycle Domain Average Rewards for TA-TL (ours), UMA-TL (jump- start) and RL
Source Training
18.5
5
Training Time (Hrs)
Target Training
18
log(N)
17
3
16.5
2
15
I-RL
TA-TL
UMA-TL
RL
(a)
(b)
Figure 8: Policy Transfer from Cart-Pole to Bike Balancing: (a) Total simulation time (in seconds) agent was able to balance the bike in training (b) Total time required to solve the task for TA-TL (ours), UMA-TL (jumpstart) and RL
The states of the bicycle task angle and angular velocity of the handlebar and the bike from vertical (0, 0, w., . respectively. For the given state the agent is in, chooses continuous action of applying torque to the handlebar, E [-2NN. m, 2Nm] trying to keep the bike upright. The details of bicycl dynamics are beyond the scope of this paper; interested readers are referred Randlov & Alstrom 1998] Aström et al. (2005), and references therein.
We the Cart-Pole source task for learning balance bicycle. The bicycle balance problem in principle is similar to that of cart-pole, since the objective is to keep the unstable system upright. The objective of balance is achieved in both the systems by moving in the direction of fall, which is termed as non-minimum phase behavior in the controls system literature. The control in the cart pole affects the angle of the pole, by moving the cart such that it always under the pole. In the bicycle, the control is to move the handlebar in the direction of fall. However, balancing the bike not straightforward, to turn the bike under itself, one must first steer the other direction before turning in the direction of fall; this is called counter-steering Aström et al. (2005). We observe that both cart pole and bicycle have this commonality in their dynamical behaviors, as both the system have non mon-minimum phase that is the presence of unstable zero. This similarity qualifies the cart- pole system as an appropriate choice of source model for bicycle balance task.
Cart pole is characterized by state vector [x, f., 8, é]. i.e., position, the velocity cart and angle, angular velocity of the pendulum. The action space is the force applied to the cart F E [-1N, 1N). For the mapping between the state space of bicycle and cart pole model, we use Unsupervised Manifold Alignment(UMA) to obtain this mapping Wang & Mahadevan (2009). We do not report
17

---

Under review as a conference paper at ICLR 2020
the training time to learn the intertask mapping since it is common both ATL and UMA-TL methods. Figure (b and g shows the quality of transfer for ATL through faster convergence to average maximum reward with lesser training samples compared to UMA-TL and RL methods.
E.2 MOUNTAIN CAR (MC) INVERTED PENDULUM (IP)
14.2
-0.
14
13.8
Transbey
-3.
-07
spRoodes
TA-TL
UMA-TL
(a)
(b)
Figure 9: Policy Transfer from Mountain Car to Inverted Pendulum: (a) Average Rewards and (b) Training length
We also demonstrate the cross-domain transfer between mountain car to an inverted pendulum. The
source and target task characterized by different state and action space. The source task MC benchmark RL problem of driving an underpowered car up a hill. The dynamics of MC are described by continuous state variables (x) where E 1.2, 0.6] and & [-0.07, 0.07] and one continuous action F E [], The reward function is proportional to the negative of the squared distance of the car from goal position. The target task is conventional IP with state (0,0) E (- and action T' E I-,, We present the performance of transfer methods based on sample efficiency in learning the target task and speed of convergence to the maximum average reward. Similar the bicycle domain transfer Figure (a and [9b shows the quality of transfer for ATL through faster convergence average maximum reward with lesser training samples compared to UMA-TL and RL methods.
18


Reviews:
[{"review_id": "rklgQ7uCtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper addresses the actively studied problem of efficiently transferring policies across domains in reinforcement learning. Authors propose a framework to transfer policies between tasks in domains with significantly different state transition. The proposed algorithm is based on a policy adaptation mechanism, with the idea that provided that a source optimal policy of a task is available, that policy is adapted to derive the optimal policy of the target task at a low sample complexity. \n\nThe paper is well written and the proposed algorithm is novel and original. The algorithm is very clearly described and theoretical bounds are provided. The performance is evaluated experimentally on a set of tasks, showing promising results.  \n\nCons:\n- Some important previous  works are missing from the related work section. For example, [1, 2] are also concerned with transferring policy across task that share the same domain. \n- The experimental section could benefit from comparing to other transfer RL baselines such as [1].\n- Assumption 2 should be discussed; any suggestion on how to model the KL approximation if the assumption does not hold?\n\nMinor comments for clarity:\n- some acronyms are introduced without ever being spelled out: D-RL, TL, MDP, PAC\n- page 3, equation before (2), is \\gamma = 1?\n- the use of s' for the optimal state, a' for optimal action and then r' for total reward is quite confusing. Specially because s' is in general used to describe a subsequent state in the literature. \n- page 7, first line 'discount' instead of 'discout'\n- font on figures 1 and 2 is very small\n\n1. Barreto et al, Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement, 2018\n2. Ma, Wen, and Bengio Universal Successor Representations for Transfer Reinforcement Learning, 2018", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1lVR3-bqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper tackles the problem of transferring a policy from source to target MDP, which differ in the state transition function. The idea is to add an additional cost that is the KL divergence between the trajectory likelihood under target policy (being learned) and target dynamics and the trajectory likelihood under the source policy (assumed optimal and deterministic) and source dynamics. The intuition is that the target policy will learn to match the state distribution of the optimal source policy. Results on MuJoCo locomotion robots with varying physics show that the proposed method performs better on target than warm-started RL or learning from scratch. \n\nI think the problem of transferring knowledge from one task to another in RL is very important for RL to be applicable to more real-world scenarios.\n\nConcerns / Questions\nLine 7 of Alg1 is confusing because it refers to a “target task model”, but in Assumption 2, it says only a model of the source transition function is needed. I think it makes sense that only the source transition model is needed because the target next state is given by experience. \nI think the combined assumptions of a) access to expert behavior (same as DAGGER) and b) that the MDPs differ only in dynamics functions and c) access to the source transition model are rather strong. I think (b) is a special case of transfer learning - a lot of transfer learning is concerned with changing reward functions as well, which this method wouldn’t apply to. I think this could be made more clear in the paper. It would be good if all these assumptions were made clear and discussed.\nI think the related work section is missing important areas of research in imitation learning and meta-reinforcement learning. For imitation learning, the approach strikes me as bearing similarity to PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings (Rhinehart et al.), and the topic of imitation learning should be discussed in general. For meta-RL, mentioning that it shares the same goal of transfer and citing a few main works (e.g. Duan et al. 2016, Wang et al. 2016, Finn et al. 2017 etc) would be good.\n\nWriting Suggestions\nSome terms used throughout the paper are quite unclear (e.g., “unsupervised RL”, “intrinsic adaptation reward”, “supervised reference trajectory tracking”). I suggest standardizing and defining terms early to avoid unnecessary confusion.\nWriting the Bellman operator and the value function equations in Section 2 don’t seem very relevant as they are I think never used again?\nSections 3.1 and 3.2 are quite difficult to understand on first read (e.g., what does “point-wise local trajectories” mean?). \nI find Section 3.2.1 a bit misleading, “The optimization is more akin to supervised learning” - I agree the KL minimization is essentially imitation learning, but you are still doing policy search in addition to it?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlnnt22YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\nThe authors propose an algorithm for transferring policies across domains differing in their transition model. The idea is to \"regularize\" the target policy being learned to generate a trajectory distribution similar to the one induced by the source optimal policy in the respective MDP. The method is proved effective in standard simulated robotic tasks (Mujoco).\n\nMajor comments\n--------------\nThe problem of transfer in RL is fundamental for scaling RL algorithms to real domains. The presented ideas are interesting and could potentially inspire other works in this field. The paper is well-written and easy to read. However, I believe there might be some technical issues with the derivations that overall make me doubt about the significance of this work. My detailed comments follow.\n\n1. Under Assumption 1, the step from (6) to (7) seems flawed. The definition of the KL divergence requires that p is absolutely continuous w.r.t. q. However, under Ass. 1, whenever the trajectory \\tau contains an action that is not taken by \\pi^*, q(\\tau) = 0, which implies that p(\\tau) must be zero as well. The only way to enforce this is to set \\pi_\\theta = \\pi^*, and the solution is trivial. The main issue is that (6) writes the log term of the KL using different action random variables for the numerator and the denominator (a and a'), but these must be the same. Therefore, one cannot simply set to 1 the optimal source actions. Notice also that the action space is defined continuous, hence the value of the pdf in the chosen deterministic action is infinity, not 1.\n\n2. (12) seems to be wrong too. Note that the reward function defined here (r') contains the policy, hence it is a function of \\theta and must be differentiated. So the standard REINFORCE estimator written here does not work theoretically.\n\n3. Assumption 2 basically requires that either we can get new samples from the source domain \"on-demand\", or we have a good simulated model. Both assumptions are often violated in practice. Could the authors elaborate more on why they believe these assumptions to be reasonable?\n\n4. Related to the previous point, assuming the one-step transition models to be Gaussian is convenient, and often reasonable, in practice. But how do we choose the std \\sigma? What if the true models are deterministic?\n\n5. I would disagree with the last sentence of section 3.2.1, which states that minimizing the KL divergence is more sample efficient than standard RL/policy-search methods. One can always define an MDP where the immediate reward is the KL term and that is a standard RL problem. Furthermore, if we compute the expected return when the reward is (8), decomposing the log into the sum of two logs, one of the two expectations we obtain is of the form: E_\\pi[-(s_{t+1} -s'_{t+1})^2], which is a standard \"tracking\" problem.\n\n6. I was quite confused by the theoretical section. The introduction states that the proposed method enjoys a sample complexity of O(nH), where I guess n is the number of state-action pairs. Where does that dependence appear in Section 4? Theorem 4.1 is a standard supervised learning bound. How do we read that in RL? What is C? Is |\\Pi| the cardinality of the set of policies (assumed finite)? Or is it the pseudo-dimension of the policy space? Lemma 4.3 is also a know result in the literature, but what does it tell us about the proposed approach? The idea is that if I have an accurate model \\hat{p} I am also close in value function to the true MDP. But what is \\hat{p} here? The proposed algorithm is not computing any explicit transition model.\n\nMinor comments\n--------------\n- In the last sentence of the introduction, it is not clear what n is in O(nH)\n- In the preliminaries, is the reward R: S x A -> R_+ assumed to be positive or is that a typo?\n- MDPs are defined with a discount factor \\gamma which is however missing in (1) and the next equation (V)\n- In the preliminaries, it is not clear whether the rewards differ or not between domains. It is important to specify that they do not.\n- In (2), T is defined to map R^n -> R^n. What is n? Is it the number of discrete states? Wasn't the state space continuos?\n- I noticed that many sentences in the paper begin without a space after the period of the previous sentence.\n- In section 3, the KL divergence (e.g., eq. 3,7) is written as an expectation that contains the pdf p(\\tau). Either only the log term is written in the expectation, or E is replaced by an integral.\n- In (12), the sum should be up to H and not infinity", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers inter-domain policy transfer in reinforcement learning. The proposed approach involves adapting existing policies from a source task to a target task by adding a cost related to the difference between the dynamics and trajectory likelihoods of the two tasks.

There are three major problems with this paper as it stands, as pointed out by the reviewers. Firstly, the "KL divergence" is not a real KL divergence and seems to be only empirically motivated. Then, there are issues with the derivative of the policy gradient. Finally, the theory is not well connected to the proposed algorithm. The rebuttals not only failed to convince the reviewer that raised these issues, but another reviewer lowered their score as a result of these raised points.

This is a really interesting idea with compelling experiments, but must be rejected at this point for the aforementioned reasons.
Decision: Reject