MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
Authors: Mohammadamin Barekatain
Ryo Yonetani
Masashi Hamaya
Venue:  (2020)
Keywords: reinforcement learning
transfer learning
policy aggregation
residual policy learning

Abstract: Transfer reinforcement learning (RL) aims at improving learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under unknown diverse dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces.

Under review as a conference paper at ICLR 2020
MULTIPOLAR: MULTI-SOURCE POLICY AGGREGA- TION FOR TRANSFER REINFORCEMENT LEARNING BE- TWEEN DIVERSE ENVIRONMENTAL DYNAMICS
Anonymous authors Paper under double-bling review
ABSTRACT
Transfer reinforcement learning (RL) aims at improving learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging transfer knowledge between different environ- mental dynamics without having access to the source environments. In this work, we explore new challenge in transfer RL, where only a set of source policies collected under unknown diverse dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTT--source POL- icy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's 's expres- siveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to chal- lenging robotics simulations, under both continuous and discrete action spaces.
INTRODUCTION
We envision a future scenario where a variety of robotic systems, which are each trained or manually engineered to solve a similar task, provide their policies for a new robot to learn a relevant task quickly. For example, imagine various pick-and-place robots working in factories all over the world. Depending on the manufacturer, these robots will differ in their kinematics (e.g.. link length, joint orientations) and dynamics (e.g., link mass, joint damping, friction, inertia). They could provide their policies to a new robot (Devin al. 2017), even though their dynamics factors, on which the policies are implicitly conditioned, are not typically available (Chen et al. 2018). Moreover, we cannot rely on history of their individual experiences, as they may be unavailable due to a lack of communication between factories prohibitively large dataset sizes. In such scenarios, argue that a key technique to develop is the ability to transfer knowledge from a collection of robots to a new robot quickly only by exploiting their policies while being agnostic to their different kinematics and dynamics, rather than collecting a vast amount of samples to train the new robot from scratch.
The scenario illustrated above poses new challenge in the transfer learning for reinforce- ment learning (RL) domains. Formally, con- sider multiple instances of a single environment
Source agents
Target agent
that differ in their state transition dynamics, e.g., independent ant robots with different leg designs Figure [ which reach different lo- cations by executing the same walking actions. These source agents interacting with one of the environment instances provide their determin- istic policy to a new target agent in another en- vironment instance. Then, our problem is: can we efficiently learn the policy of a target agent given only the collection of source policies? Note that information about source environmental dynamics, such the exact state transition distribu-
Figure 1: Ant Example. A policy of target agent (right) is learned by utilizing the policies of other source agents with different leg designs (left).

---

Under review as a conference paper at ICLR 2020
Adaptive aggregation source policies: Fags (gi Pasx)
Continuous action space:
Mearges = . (F(sz: Oag uux)
Source polictes
Pi
F ( 1 sex Pasx)
State
800
Balk
Discrete space:
Auxiliary network for predicting residuals: Faux
Mtarget = softmax (F(s) L, Pawx)
Figure 2: Overview of MULTIPOLAR. We formulate target policy Ttarget with the sum of 1) the adaptive aggregation Fagg of deterministic actions from source policies and 2) the auxiliary network Faux for predicting residuals around Fagg-
tions and the history of environmental states, will not be visible to the target agent as mentioned above. Also, the source policies are neither trained nor hand-engineered for the target environment instance, and therefore not guaranteed to work optimally and may even fail (Chen et al.) 2018). These conditions prevent us from adopting existing work on transfer RL between different envi- ronmental dynamics, as they require access to source environment instances or their dynamics for training target policy (e.g.. Lazaric et al. 2008); Chen et al. (2018); Yu al. 22019) Tirinzoni ct al.] (2018)) Similarly. meta-learning approaches (Vanschoren 2018 Sæmmundsson et al. [018] Clavera al. 2019) cannot be used here because they typically train agent
diverse set of tasks (i.e,, environment instances). Also, existing techniques that utilize a collection of source poli- cies, e.g. policy reuse frameworks (Fernández & Veloso 2006 Rosman et al. 2016 Zheng al. 2018) and option frameworks (Sutton et al.) 1999; Bacon et al. 2017 Mankowiil et al. 2018), are not promising solution because, our knowledge, they assume source policies have the same environmental dynamics but have different goals.
As solution the problem, we propose new transfer RL approach named MULTI-source POL- icy AggRegation (MUTTIPOLAR). As shown in Figure 2 our key idea is twofold; 1) In target policy, we adaptively aggregate the deterministic actions produced by collection of source poli- cies. By learning aggregation parameters to maximize the expected return target environment instance, we can better adapt the aggregated actions to unseen environmental dynamics of the tar- get instance without knowing source environmental dynamics nor source policy performances. 2) We also train an auxiliary network that predicts a residual around the aggregated actions, which is erucial for ensuring the expressiveness of the target policy even when some source policies not useful. As another notable advantage. the proposed MULTIPOLAR can be used for both continuous and discrete action spaces with few modifications while allowing target policy be trained in principled fashion. Similar to Ammar et al. (2014); Song et al. (2016); Chen et al. (2018); Tirinzoni al. 20188; : Yu al. (2019), our method assumes that the environment structure (state/action space) is identical between the source and target environments, while dynamics/kineman different. This assumption holds in many real-world applications such as in sim-to-real tasks (Tan) et al. 2018), industrial insertion tasks (Schoettler et al. (019) (different dynamics comes from the differences in parts), and wearable robots (Zhang et al. 2017) (with users as dynamics).
parameters are
We evaluate MULTIPOLAR in variety of environments ranging from classic control problems to challenging roboties simulations. Our experimental results demonstrate the significant improvement of sample efficiency with the proposed approach, compared to basclines that trained target policy from scratch or from a single source policy. We also conducted a detailed analysis of our approach and found it works well when some of the source policies performed poorly in their original environment instance.
Main contributions: (1) transfer RL problem that leverages multiple source policies col- lected under diverse environmental dynamics to train target policy in another dynamics, and (2) MULTIPOLAR simple yet principled and effective solution verified in extensive experiments.
2

---

Under review as a conference paper at ICLR 2020
2 PRELIMINARIES
Reinforcement Learning We formulate our problem under the standard RL framework (Sutton & Barto 1998). where agent interacts with its environment modeled by Markov decision process (MDP). An MDP is represented by the tuple M - (po, 7, S,A,, R, T where is the initial state distribution and y agent executes an action
discount factor. At each timestep t, given the current state 8t S, the AA based on its policy a[a sy 0) that is parameterized by 8. The environment returns a reward R((e; (t) E R and transitions to the next state 8t++1 based on the state transition distribution T( 8t, (z). In this framework, RL aims to maximize the expected return with respect to the policy parameters O.
Environment Instances In this work, we consider instances of the same environment that differ only in their state transition dynamics. We model each environment instance by an indexed MDP: M; = (Po: y,S, S, A, R, T.) where no two state transition distributions T;, T;,T j are identical. We also assume that each T; is unknown when training target policy, Le., agents cannot access the exact form of T; nor a collection states sampled from T;
Source Policies For each of the K environment instances, we are given a deterministic source policy μ4 : + A that only maps states actions. Each source policy br can be either parame- terized (e.g.. learned from interacting with the environment modeled by MM,) or paraammetteei (e.g., heuristically designed by humans). Either way, we assume no prior knowledge about 14 is available for a target agent, such as their representations original performances, except that they were acquired M, with an unknown T.
Problem Statement Given the set of source policies = {uI.... pK). our goal is to train a new target agent's policy Ttarget (at st; 8) in sample efficient fashion, where the target agent interacts with another environment instance Mtarget = (Po, A, R, Ttarget) ) and Trarget is not necessarily identical to T; (i=1... 1==1.... 1...,,K)
MULTI-SOURCE POLICY AGGREGATION
As shown in Figure 2 with the Multi-Source Policy Aggregation (MULTIPOLAR), we formulate target policy Ttarget using a) the adaptive aggregation of deterministic actions from the set of source policies L, and b) the auxiliary network predicting residuals around the aggregated actions. We first present our method for the continuous action space, and then extend it to the discrete space.
Adaptive Aggregation of Source Policies Let us denote by = (st) the action predicted deterministically by source policy 14 given the current state 8t. For the continuous action space, " R 2 is a D-dimensional real-valued vector representing D actions performed jointly in each timestep. For the collection of source policies L, we derive the matrix of their deterministic actions:
..... A, = ((..)).....(
(a) E RXD
The key idea of this work is to aggregate A, adaptively in an RL loop, i.e., to maximize the expected return. This adaptive aggregation gives us a "bascline" action that could introduce a strong inductive bias in the training of target policy, without knowing source environmental dynamics T,. More specifically, we define the adaptive aggregation function Fags SS - that produces the baseline action based the current state 8; follows:
1 1 (Oags 1 Ad) K
Fagg (st) L, Oagg)
(2)
where Oagy. E KR is matrix of trainable parameters, O is the element-wise multiplication, and is the all-ones vector of length K. Oagg is neither normalized nor regularized, and can scale each action each policy independently. This means that we do not merely adaptively interpolate action spaces, but more flexibly emphasize informative source actions while suppressing irrelevant ones.
Predicting Residuals around Aggregated Actions Moreover, learn auxiliary network Faux - A jointly with Fagg. to predict residuals around the aggregated actions. F.nx is used to improve the target policy training in two ways. If the aggregated actions from F. Fagg are already useful in
3

---

Under review as a conference paper at ICLR 2020
the target environment instance, Faux will correct them for a higher expected return. 2) Otherwise, Faux learns the target task while leveraging Fagg as a prior to have a guided exploration process. Any network could be used for Faux as long as it is parameterized and fully differentiable. Finally, the MULTIPOLAR function is formulated as:
F Fs; L. Ougg: amxx) = Fagg L, L,gss) + Faux 0aux),
where Oaux denotes a set of trainable parameters for Faux. Note that the idea of predicting residuals for source policy has also been presented by (Silver et al. (2 (2019). The main difference here is that, while these works just add raw action outputs provided from single hand-engineered source policy, we adaptively aggregate actions from multiple source policies in order to obtain a more flexible and canonical representation.
et al. 2019)][Rana, et al.
Target Policy Target policy Ttarget can be modeled by reparameterizing the MULTIPOLAR func- tion as Gaussian distribution, i.e., (F; L, A Aagx A, E), where Σ is a covariance matrix estimated based on what the used RL algorithm requires. Since we regard μ4; E L as fixed func- tions mapping states actions, this Gaussian policy Ttarget is differentiable with respect Ougg and Oaux. and hence could be trained with any RL algorithm that explicitly updates policy parameters. Unlike (Sileer et al. (2018); Johannink al. (2019); Rana et al. (2019), we can formulate the target policy in a principled fashion for actions in discrete space. Specifically, instead of D-dimensional real-valued vector, here we have D-dimensional one-hot vector a.") E (0, Ea
as indicates that the j-th action is to be executed. Following Eqs. 2
outputs of fti where (a. (a!"), ) and 3p, the output of Fs;,; L, 0 fugg: (xaxx) can be viewed as D-dimensional un- un-normalized action scores, from which we can sample a discrete action after normalizing by the softmax function.
EXPERIMENTAL EVALUATION
We aim empirically demonstrate the sample efficiency of target policy trained with MULTIPO- LAR (denoted by "MULTIPOLAR policy"). To complete the experiments in a reasonable amount of time, we set the number of source policies to be K unless mentioned otherwise. Moreover, we investigate the factors that affect the performance of MULTIPOLAR, To ensure fair compar- isons and reproducibility of experiments, we followed the guidelines introduced by (Henderson al. (2018) and François-Lavet et al. (2018) for conducting and evaluating all of our experiments.
4.1 EXPERIMENTAL SETUP
Baseline Methods To show the benefits of leveraging source policies, we compared our MULTI- POLAR policy to the standard multi-layer perceptron (MLP) trained from scratch, which is typically used in RL literature (Schulman et al. 2017: François- Lavet et al.) 2018). As another baseline, also used MULTIPOLAR with let al. 2018, Johannink et al. 2019 Rana et al. 2019) (denoted by "RPL." with adaptive residuals as well as the ability to deal with both continuous and discrete action spaces. We stress here that the existing transfer RL or meta RL approaches that train a universal policy network agnostic to the environmental dynamics, such as |rans et al. (2018): : Chen et al. (2018), cannot be used as baseline since they require policy be trained on a distribution of environment instances, which is not possible in our problem setting. Also, other techniques using multiple source policies, such as policy reuse frameworks, are not applicable because their source policies should be collected under the target environmental dynamics.
which is an extension of residual policy learning (Silver
Environments To show the general effectiveness of the MULTIPOLAR policy, we conducted comparative evaluations of MULTIPOLAR on the following six OpenAl Gym environments: Ro- Acrobot, CartPole,
boschool Hopper, Roboschool Ant, Roboschool InverredPendullmSSvvi and LunarLander. We chose these six environments because 1) the parameterization of their dynam- ics and kinematics is flexible enough, 2) they cover discrete action space (Acrobot and CartPole) as well as continuous action space, and 3) they are samples of three distinct categories of OpenAI Gym environments, namely Box2d, Classic Control, and Roboschool.
Experimental Procedure For each of the six environments, first created 100 environment instances by randomly sampling the dynamics and kinematics parameters from a specific range. For example, these parameters in the Hopper environment were link lengths, damping, friction,
4

---

Under review as a conference paper at ICLR 2020
CartPole
LunartanndeCCnniinuuus
Episadic Reward
P -100
-150
Reward
200
300
300
E -250
-200
100
200
MLP
-300
-350
-100
-200
-400
-300
100
MULTIPOLAR (K-4)
W
-400
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
2.0
0.00
2.50
5.00
Timesteps
1e5
Timesteps
1e5
Timesteps
1e5
RabeschaolHopper
RabaschoolAnt
Episodic Reward
2000
Episodic Reward
2000
Spssadic Rewarrd
1000
1500
1500
600
1000
1000
500
200
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
1.0 Timesteps
2.0
Timesteps
1e6
Timesteps
166
1c6
Figure 3: Average Learning Curves of MLP, RPL, and MULTIPOLAR (K = 4) over all the experiments for each environment. The shaded area represents standard error.
armature, and link mass] Then, for each environment instance, trained an MLP policy. The trained MLP policies were used in two ways: a) the baseline MIP policy for each environment instance, and b) pool of 100 source policy candidates from which we sample of them train MULTIPOLAR policies and one of them to train RPL policies Specifically, for each environment instance, we trained three MULTIPOLAR and three RPL policies with distinct sets sets source policies selected randomly from the candidate pool. The learning procedure explained above was done three times with fixed different random seeds to reduce variance results due to stochasticity. As a result, for each of the six environments, we had 100 environment instances x 3 random seeds 300 experiments for MI.P and 100 environment instances choices of source policies seeds 900 experiments for RPL and MULTIPOLAR. The aim of this large number of experiments to obtain correct insights into the distribution of performances (Henderson et al., 2018). Due to the large number of experiments for all the environments, our detailed analysis and ablation study of MULTIPOLAR components were conducted with only Hopper, as its sophisticated second-order dynamics plays a crucial role in agent performance (Chen al. 2018).
random
Implementation Details All the experiments were done using the Stable Baselines (Hill et al. 2018) implementation of learning algorithms as well as its default hyperparameters and MLP net- work architecture for each environment (see Appendix I. for more details). Based on the perfor- mance of learning algorithms reported in the Hill et al. (2018), all the policies were trained with Soft Actor-Critic (Haarnoja et al. 2018) in the LunarLander environment and with Proximal Pol- icy Optimization (Schulman al. (017)) in the rest of the environments. For fair comparisons, in all experiments, auxiliary network Faux had an identical architecture to that of the MLP. There- fore, the only difference between MLP and MULTIPOLAR was the aggregation part F Fage- which made it possible to evaluate the contribution of transfer learning based on adaptive aggregation of source policies. Also, we avoided any random seed optimization since it has been shown to alter the policies" performance (Henderson et al. 2018).
Evaluation Metric Following the guidelines of Henderson et al. 2018)) measure sampling ef- ficiency of training policies, i.e., how quick the training progresses, we used the average episodic reward over various number of training samples. Also, to ensure that higher average episodic reward is representative of better performance and to estimate the variation of it, we used the sam- ple bootstrap method (Efron & Tibshirani 1993) estimate statistically relevant 95% confidence bounds of the results of our experiments. Across all the experiments, we used 10K bootstrap itera- tions and the pivotal method. Further details evaluation method can be found in Appendix A.3
Details of sampling ranges for dynamics and kinematics are provided in Appendix A.2 A Ithough used trained MI.Ps source policies for reducing experiment times, any type of policies including hand-engineered ones could be used for MULTIPOLAR in principle.
5

---

Under review as a conference paper at ICLR 2020
Table 1: MULTIPOLAR vs. Baselines. Bootstrap mean and 95% confidence bounds of average episodic rewards over various training samples across six environments.
CartPole
Methods
25K
50K
75K
100K
MLP
171 (164,179)
229 (220,237)
266 (258,275)
291 (282,300)
RPL
185 (179,192)
238 (231,245)
(262.276)
289 (282,296)
MULTIPOLAR (K=4)
202 (195,209)
252 (245,260)
283 (276,290)
299 (2922306)
Acrobot
50K
100K
150K
200K
MLP
-305 (-317.-294)
-164 (-172,-156)
-127 (-133,-121)
-111 (-117,-106)
RPL
-154 (-159,-150)
-120 (-124,-116)
-105 (-109,-102)
-98 (-101,-95)
MULTIPOLAR (K-4)
-151 (-155,-146)
-117 (-121,-113)
-103 (-106,-100)
.96 (-99,-93)
Lunarl ander
125K
250K
375K
500K
MLP
(2,18)
112 (104,121)
178 (171,185)
216 (210,221)
RPL
92 (87,96)
178 (174,182)
223 (220,226)
246 (243,248) 248)
MULTIPOLAR (K=4)
95 (90,99)
181 (177,185)
224 (221,228)
246 (244,249)
Roboschool Hopper
0.5M
IM
1.5M
2M
MLP
26 (25,27)
(42,45)
67 (64,70)
92 (88,96)
RPL
(36,39)
(70.79)
114 (107,121)
(142,160)
MULTIPOLAR (K=4)
(59,64)
138 (132,143)
213 (206,221)
283 (273,292)
Roboschool Ant
0.5M
1M
1.5M
2M
MLP
714 (674,756)
1088 (1030, 1 146)
1332 (1267,1399)
1500 (1730,1572)
RPL
807 (785,830)
1120 ( (1088. 152))
1307 (1269, 1344)
1432 (1391,1473)
MULTIPOLAR (K=4)
1025 (995,1056)
1397 (1361,1432)
1606 (1568,1644)
1744 (1705,1783)
Roboschool InvertedPendulunSSvinuug
0.5M
1M
1.5M
2M
MLP
159 (155,164)
267 (260,273)
347 (340,355)
409 (401,417)
RPL
111 (109,
195 (192, 198)
265 (261 .268)
322 (317.326)
MULTIPOLAR (K=4)
375 (355,395)
476 (456,495)
541 (522,559)
588 (571,605)
4.2 RESULTS
Sample Efficiency of MULTIPOLAR Figure and Table clearly show that on average, in all the environments, MULTIPOLAR outperformed baseline policies in terms of sample efficiency and sometimes the final episodic reward 3 For example, in Hopper over 2M training samples, MULTI- POLAR with 4 achieved mean of average episodic reward about three times higher than MLP (i.e., training from scratch) and about twice higher than RPL (i.e., using only single source policy). also noteworthy that MULTIPOLA with K had on par better performance than RPI., which indicates the effectiveness of leveraging multiple source policie Figure 7 in Appendix, shows the individual average learning curve for each of the instances of Roboschool environments.
To demonstrate the importance of each component of MULTIPOLAR, we eval- uated the following degraded versions: (1) Oagg fixed 1, which just averages the deterministic actions from the source policies without adaptive weights (similar to the residual policy learning
Ablation Study
Episodic rewards in Figure 3 averaged over 3 random seeds and 3 random source policy environment instances. Table reports the mean VVideo replays of source policies, as well as MULTIPOLAR vs. baseline MI.P in the Ant environment available at: httpsss www. youtube com www.o//wwttee chw.aachhhhveee. 30m300m
100 average over multiple numbers of training samples.

---

Under review as a conference paper at ICLR 2020
Table 2: Results for MULTIPOLAR and its degraded versions in Hopper.
MULTIPOLAR (K=4)
0.5M
1M
1.5M
2M
Full version
61 (59,64)
138 (132,143)
213 (206,221)
283 273)
Oagg fixed to
56 (53,59)
118 (111,126)
180 (169,191)
237 (222,250)
Faux learned independent of st
53 (50,56)
101 (95,108)
146 (137,156)
187 (175,200)
Table 3: Results for MULTIPOLAR with different source policy sampling schemes in Hopper.
MULTIPOLAR (K=4)
0.5M
IM
1.5M
Random
61 (59,64)
138 (132,143)
213 (206,221)
283 (273,292)
high performance
(95,101)
214 (208,220)
323 (314,331)
420 (409,433)
2 high & 2 low performance
45 (43,47)
98 (94,102)
154 (148, 160)
208 (200,215)
4 low performance
27 (26,27)
45 (44,47)
68 (66.71)
92 (88.95)
Table 4: Results for MULTIPOLAR with different number of source policies in Hopper.
MULTIPOLAR
0.5M
IM
1.5M
2M
K=4
61 (59,64)
138 (132, 143)
213 (206,221)
283 (273,292)
K=8
71 (68,74)
160 (154.167)
246 (236,255)
323 (312,335)
K=16
78 (75,80)
177 (172,182)
272 (264,279)
357 (348,367)
methods that used raw action outputs of source policy), and (2) Faux learned independent of St, which replaces the state-dependent MLP with an adaptive "placeholder" parameter vector making actions just a linear combination of source policy outputs. As shown in Table 2 the full version of MULTIPOLAR significantly outperformed both of the degraded versions, suggesting that the adaptive aggregation and predicting residuals are both critical.
Effect of Source Policy Performances Figure illustrates an example of the histogram of fi- nal episodic reward (average rewards of the last 100 training episodes) for the source policy can- didates obtained in the Hopper environment. As shown in the figure, the source policies were diverse in terms of the performance on their original environment instances In this setup, we investigate the effect of source policies performances on MULTIPOLAR sample efficiency. We created two separate pools of source policies, where one contained only high high-performing and the other only low- performing source policies] Table 3 summarizes the results of sampling source policies from these pools (4 high, 2 high & 2 low, and low performances) and compares them to the original MULTIPOLAR (shown as 'Random') also reported in Table D Not surprisingly, MULTIPOLAR performed the best when all the source policies were sampled from the high- performance pool. However, we emphasize that such high- quality policies are not always available in practice, due to the variability of how they are learned or hand-crafted under their own environment instance. Figure [6] in Appendix B. illus- trates that MULTIPOLAR can successfully learn to suppress the useless low-performing source policies.
#Source Policies
20
15
10
1000 Final Episodic Reward
2000
3000
Figure 4: Histogram of source pol- icy performances in Hopper.
Effect of Number of Source Policies Finally, we show how the number of source policies con- tributes to MULTIPOLAR's sample efficiency in Table Specifically, we trained MULTIPOLAR policies up K 16 to study how the mean of average episodic rewards changes. The monotonic performance improvement over K (for K 16), is achieved at the cost of increased training and inference time. In practice, we suggest balancing this speed-pes performaee trade-off by using as many source policies possible before reaching the inference time limit required by the application.
"Histograms for the other environments can be found in Appendix A.4 'Here, policies with final episodic reward over 2K are high-performing and below 1K are low-performing.
7

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION AND RELATED WORK
Our work is broadly categorized as an instance of transfer RL. (Taylor & Stone 2009). in which a policy for target task trained using information collected from source tasks. In this section, we highlight how our work is different from the existing approaches and also discuss the current limitations as well as future directions.
Transfer between Different Dynamics There has been very limited work on transferring knowl- edge between agents in different environmental dynamics. As introduced briefly in Section | some methods require training samples collected from source tasks. These sampled experiences are then used for measuring the similarity between environment instances (Lazaric et al., 2008 Ammar et al.) 2014 Tirinzoni et al.) 2018) or for conditioning a target policy to predict actions (Chen et al. 2018).
Alternative means to quantifyy the similarity is to use full specification of MDPs (Song al. [0066]
Wang et al. 2019) or environmental dynamics Yu et al. (2019). In contrast, the proposed MULTI- POLAR allows the knowledge transfer only through the policies acquired from source environment instances, which is beneficial when source and target environments are not always connected to exchange information about their environmental dynamics and training samples.
Leveraging Multiple Policies The idea of utilizing multiple source policies can be found in the literature of policy reuse frameworks (Fernández& & Vcloso [006 Rosman et al. 2016 Li & Zhang, 2018 Zheng et al. 2018 Li et al. 2219). The basic motivation behind these works is to provide "nearly-optimal solutions" (Rosman et al., 2016) for short-duration tasks by reusing one of the source policies, where each source would perform well on environment instances with different rewards (e.g., different goals in maze tasks). In our problem setting, where environmental dynamics behind each source policy are different, reusing a single policy without an adaptation is not the right approach, as described in (Chen et al.] 2018) and also demonstrated in our experiment. Another relevant idea is hierarchical RL (Barto & Mahadevan 2003 Kulkarni et al., 2016; Osa et al. 2019) that involves hierarchy of policies (or action- functions) to enable temporal abstraction. In particular, option frameworks (Sutton et al. 1999 Bacon et al. 2017 Mankowitz et al.) 2018) make use of a collection of policies as a part of "options", However, they assumed all the policies in the hierarchy to be learned in single environment instance. Another relevant work along this line of research is (Frans et al. 2018), which meta-learns hierarchy of multiple sub-policies by training a master policy over the distribution of tasks. Nevertheless, hierarchical RL approaches are not useful for leveraging multiple source policies each acquired under diverse environmental dynamics.
Learning Residuals in RL Finally, some recent works adopt residual learning to mitigate the limited performance of hand-engineered policies (Silver et al. 2018 Johannink et al. 2019] Rana jet al., 2019). We are interested in a more extended scenario where various source policies with unknown performances are provided instead of a single sub-optimal policy. Also, these approaches focus only on RL problems for robotic tasks in the continuous action space, while our approach could work both of continuous and discrete action spaces in a broad range of environments.
Limitations and Future Directions Currently, work has several limitations. First, MULTI- POLAR may not be scalable to a large number of source policies, as its training and testing times will increase almost linearly with the number of source policies. One possible solution for this issue would be pre-screening source policies before starting to train a target agent, for example, by testing each source on the target task and taking them into account in the training phase only when they are found useful. Moreover, our work assumes source and target environment instances to be different only in their state transition distribution. An interesting direction for future work is to involve other types of environmental differences, such as dissimilar rewards and state/action spaces.
6 CONCLUSION
We presented a new problem setting of transfer RL that aimed to train policy efficiently using a collection of source policies acquired under diverse environmental dynamics. We demonstrated that the proposed MULTIPOLAR is. despite its simplicity, principled approach with high training sample efficiency on a variety of environments. Our transfer RL approach is advantageous when one does not have access to a distribution of diverse environmental dynamics. Future work will seek to adapt our approach to more challenging domains such as a real-world robotics task.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Haitham Bou Ammar, Eric Eaton, Matthew Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An Automated Measure of MDP Similarity for Transfer in Rein- forcement Learning. In Workshops at the AAAI Conference on Artificial Intelligence, 2014.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. In AAAI Confer- ence on Artificial Intelligence, 2017.
Andrew Barto and Sridhar Mahadevan. Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems, 13(1-2)44-77, 2003.
Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware Conditioned Policies for Multi-
Robot Transfer Learning. In Advances in Neural Information Processing Systems, pp. 9355-9366, 2018.
Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning Adapt in Dynamic, Real-World Environments through Meta- Reinforcement Learning. International Conference on Learning Representations, 2019.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning Mod- ular Neural Network Policies for Multi-Task and Multi-Robot Transfer. In IEEE International Conference on Robotics and Automation, pp. 2169-2176, 2017.
Bradley Efron and Robert Tibshirani. An Introduction to the Bootstrap. Springer, 1993.
Fernando Fernández and Manuela Veloso. Probabilistic Policy Reuse in a Reinforcement Learning Agent. In International Joint Conference on Autonomous Agents and Multiagent Systems, pp- 720-727, 2006.
Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, and Joelle Pineau. An Introduction to Deep Reinforcement Learning. Foundations and Trends in Machine Learning, 1 (3-4):219-354, 2018
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Con- ference Machine Learning, pp. 1856-1865, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. In AAAI Conference on Artificial Intelligence, pp- 3207-3214, 2018.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable Baselines. htttps://gitan www.wioloohoameeee
stable maaselines 2018.
Tobias Johannink. Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual Reinforcement Learning for Robot Control. In International Conference Robotics and Automation, pp. 6023-6029, 2019.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. In Advances in Neural Information Processing Systems, pp.
2016.
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of Samples in Batch Rein- forcement Learning. International Conference on Machine Learning, pp. 544-551, 2008
Siyuan Li and Chongjic Zhang. An Optimal Online Method of Selecting Source Policies for Rein- forcement Leaming. In AAAI Conference Artificial Intelligence, 2018

---

Under review as a conference paper at ICLR 2020
Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjie Zhang. Context-Aware Policy Reuse. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 989-997, 2019.
Daniel Mankowitz, Timothy A Mann, Pierre-Lue Bacon, Doina Precup, and Shie Mannor. Learn- ing Robust Options. In AAAI Conference on Artificial Intelligence, 2018.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization. In International Conference on Learning Rep- resentations, 2019.
Krishan Rana, Ben Talbot, Michael Milford, and Niko Suiderhauf. Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environ- ments. arXiv preprint ArX1v:1999.10972, 2019.
Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy. Bayesian Policy Reuse. Ma- chine Learning, 104(1):990 1:99-127 2016.
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta Reinforcement Learning with Latent Variable Gaussian Processes. In Conference Uncertainty in Artificial Intelligence, 2018.
Gerrtt Schoettler, Ashvin Nair, Jianlan Luo, Shikhar Bahl, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural reward signals. 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy 2017.
Optimization Algorithms. arXiv preprint
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual Policy Learning. arXiv preprint arXiv:1812.06200 2018.
Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the Distance Between Finite Markov Decision Processes. International Conference on Autonomous Agents and Multiagent Systems, pp. 468-476, 2016.
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, 1st edition, 1998.
Richard Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Frame- work for Temporal Abstraction in Reinforcement Learning. Artificial intelligence, 1 11((1-))1811- 211, 1999.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems, 2018.
Ole Tange GNU Parallel 2018. Ole Tange, 2018. URL bntt:s://@oi.org/10..3 zenodo 1146014
Matthew E. Taylor and Peter Stone. Transfer Learning for Reinforcement Learning Domains: A Survey. Journal of Machine Learning Research, 1003l): 2009.
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance Weighted Trans- fer of Samples in Reinforcement Learning. In International Conference on Machine Learning, pp. 4936 4945, 2018.
Joaquin Vanschoren. Meta-Learning: Survey. arXiv preprint arXiv: 2018.
Hao Wang, Shaokang Dong, and Ling Shao. Measuring Structural Similarities in Finite MDPs. In International Joint Conference on Artificial Intelligence, pp. 3684-3690, 2019.
Wenhao Yu, C. Karen Liu, and Greg Turk. Policy Transfer with Strategy Optimization. Interna- tional Conference Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2020
Juanjuan Zhang, Pieter Fiers, Kirby A Witte, Rachel W Jackson, Katherine L Poggensee, Christo- pher G Atkeson, and Steven H Collins. Human-in in-the-loop optimization of exoskeleton assistance during walking. Science, 126666344)1128-12222
2017.
Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A Deep Bayesian Policy Reuse Approach against Non-Stationary Agents. In Advances in Neural Information Processing Systems, pp. 2018.

---

Under review as a conference paper at ICLR 2020
A APPENDIX: FURTHER EXPERIMENTAL DETAILS
In this section, we present all the experimental details for the six environments we used. Note that did not do any hyperparameete--unning but followed the default parameters of Hill et (2018). We used the Roboschool implementation of Hopper, Ant, and InvertedPendulmmSwingee since they are based on an open-source engine, which makes it possible for every researcher reproduce our experiments. To rum our experiments in parallel, we used GNU Parallel tool (Tange 2018).
A.1 HYPERPARAMETERS
Tables and summarize all the hyperparameters used for experiments on each environment. As done by Hill al. 2018), to have successful training, rewards and input observations normal- ized using their running average and standard deviation for all the environments except CartPole and Lunarl Also, in all of the experiments, Oags is initialized be the all-ones matrix.
Table Hyperparameters for Acrobot, CartPole, Hopper, Ant and InverredPendulumSSSinguu
PPO Parameters
Acrobot
CartPole
Hopper
Ant
InvereedPendulumSSwingg
#Triiiing samples
200K
100K
2M
2M
2M
#Updates per rollout
4
20
10
10
10
Learning rate
2.5e-4
le-3
2.5e-4
2.5e-4
2.5e-4
Mini batch size
8
128
32
32
Discount factor
0.99
0.98
0.99
0.99
0.99
GAE A
0.94
0.8
0.95
0.95
0.95
Clip ratio
0.2
0.2
0.2
0.2
0.2
Value function coefficient
0.5
0.5
0.5
0.5
0.5
Entropy coefficient
0
0
0
0
0
Gradient clipping value
0.5
0.5
0.5
0.5
Optimizer
Adam
Adam
Adam
Adam
Adam
MLP & Fanx Parameters
Hidden layers
64-64 tanh
64-64 tanh
64-64
16
64-64
Activation functions
tanh
tanh
tanh
Table 6: Hyperparameters for LunarLanddr..
SAC Parameters
LunarLander
# Training samples
500K
#Steps before learning starts
IK
Buffer size
50K
Learning rate
3e-4
Mini batch size
256
Discount factor
0.99
Soft update coefficient
5e-3
Entropy coefficient
learned automatically
Model training frequency
I
Target network training frequency
I
#Gradient updates after each step
1
Probability of taking random action
0
Action noise
none
Optimizer
Adam
MLP & Faux Parameters
Hidden layers
64-64
Activation functions
relu
12

---

Under review as a conference paper at ICLR 2020
A.2 SAMPLING RANGE OF THE ENVIRONMENTAL PARAMETERS
Sampling ranges for dynamics and kinematics of each environment are provided in Tables 7 T0 [1 and T2 We defined these sampling ranges such that the resulting environments stable enough for successfully training an MLP policy. To do so, we trained MLP policies across wide ranges of environmental parameters and chose the ranges in which the policy converged.
Table 7:
Sampling range for Ant kinematic
Table 8: Sampling range for CartPole kine- matic and dynamic parameters.
and dynamic parameters.
Kinematics
Kinematics
Links
Length Range
Links
Length Range (m)
Legs
[0.4, 1.4) default length
Pole
[0.1, 3]
Dynamics
Dynamics
Damping
[0.1, 5]
Force
[6, 13]
Friction
[0.4. 2.5]
Gravity
[-14, -6]
Armature
[0.25, 3]
Poll mass
[0.1, 3]
Links mass
[0.7, 1.1] x default mass
Cart mass
[0.3, 4]
Table 9; Sampling range for Hopper kine-
Table 10: Sampling range for InvertedPendu- lumSwingup kinematic and dynamic param- eters.
matic and dynamic parameters.
Kinematics
Links
Length Range (m)
Kinematics
Leg
[0.35, 0.65]
Links
Length Range (m)
Foot
[0.29, 0.49] [0.35,0.55] [0.3,0.5]
Pole
[0.2, 2]
Thigh
Torso
Dynamics
Dynamies
Damping
[0.1, 5]
Friction
[0.5, 2]
Damping
[0.5, 4]
Armature
[0.5, 3]
Friction
[0.5, 2]
Gravity
[-11. -7]
Armature
[0.5, 2]
Links mass
[0.4, 3] default mass
Links mass
[0.7, 1.1] default mass
Table 12: Sampling range for Lu- narLander kinematic and dynamic parameters.
Table II: Sampling range for Acrobot kinematic and dynamic parameters.
Kinematics
Kinematics
Links
Length Range (m)
Side engine height
[10, 20]
Link 1&2
[0.3, 1.3]
Dynamics
Dynamics
Scale
[25, 50]
Links mass
[0.5, 1.5]
Initial Random
[500. 1500]
Links center mass
[0.05, 0.95] link length
Main engine power
[10, 40]
Links inertia moments
[0.25, 1.5]
Side engine power
[0.5, 2]
Side engine away
[8, 18]
13

---

Under review as a conference paper at ICLR 2020
A.3 EVALUATION METHOD
In this section, we explain how we calculated the mean of average episodic rewards in Tables[ and over specific number of training samples (the numbers the header of the tables e.g., 2 50K, 75K, and 100K for the CartPole) which we denote by 7 what follows. For each experiment an environment instance, we computed the average episodic reward by taking the average of the rewards over all the episodes the agent played from the beginning of the training until collecting T number of training samples. Then we collected the computed average episodic rewards of all the experiments, i.e., all the combinations of three random seeds, three random sets of source policies (for RPL and MULTIPOLAR), and 100 target environment instances. Finally, we used the sample bootstrap method (Efron & Tibshirani, 1993) to estimate the mean and the 95% confidence bounds of the collected average episodic rewards. We used the Facebook Boostrapped implementation:
https://githus: http:///ww.....oooooo.oo////////
A.4 SOURCE POLICIES HISTOGRAMS
To generate environment instances, we uniformly sampled the dynamics and kinematics parameters from the ranges defined in Section A.2 Figure illustrates the histograms of the final episodic rewards of source policies on the original environment instances in which they were acquired.
CartPole
Acrobot
LumerLanderContinoous
#Source Pol loless
SSource aell 2ee
e500 Sourer licees
20
40
15
15
10
20
-150
Final Episodic Reward
Final Episodic Reward
Final Episodic Reward
Roboschoolllopper
RoboschoolAnt
Roboscholllverteeeeenruuaa
#Source Palicies
SSource Policies
Source Policies
20
40
30
20
5
0
1000
2000
3000
1000
2000
3000
4000
400
600
800
1000
Final Episadic Reward
Final
Episodic Reward
Final spisadic Reward
Figure 5: Histogram of final episodic rewards obtained by source policies per environment.
14

---

Under review as a conference paper at ICLR 2020
B APPENDIX: ADDITIONAL RESULTS
B.1 LEARNED AGGREGATION PARAMETERS VISUALIZATION
Figure 6 visualizes an example of how the aggregation parameters fagg for the four policies and their three actions were learned during the 2M timestep training of MULTIPOLAR (K 4) policy in the Hopper environment. In this example, the source policies in the first and second rows were sampled from low-performance pools whereas those in the third and fourth rows were sampled from high-pent segh-performance pools (see Section 4.2 for more details). It illustrates that MULTIPOLAR can successfully suppress the two useless performing policies as the training progresses.
Source Action #1
Source #1, Action #2
Source #1, Action
1.00
0,p
0.75
0.75
0.9
1M
2M
1M
2M
1M Timesteps
2M
Timesteps
Timesteps
Source #2, Action
Source #2, Action
Source #2, Action
1.0
Oon
0.8
0.8
Man
2M
2M
1M Timesteps
2M
Timesteps
Timesteps
Source #3, Action #1
Source #3, Action #2
Source #3, Action
1.5
Bon
1.0
1.0
1.0
Timestops
Timestops
Timestops
Source #4, Action #1
Source #4, Action #2
Source #4, Action #3
1.25
1.5
Ous
1.0
2M
1M Timesteps
2M
Timestopss
Timestops
Figure 6: Aggregation parameters Oags during the training of MULTIPOLAR (K 4) in the Hopper that has 3-dimensional actions. Here. the first two source policies are low-performing and the last two are high-performing in their original environment instance.
B.2 MULTIPOLAR WITH RANDOMLY INITIALIZED POLICIES
To further study how having low- www-performig source policies affects MULTIPOLAR sample effi- ciency, evaluated MULTIPOLAR (K=4) in the Hopper environment, where the sources domly initialized policies, i.e., policies that predict actions randomly. Following our experimental procedure explained in Section 4.1 Table 13 reports the bootstrap mean and 95% confidence bounds
15

---

Under review as a conference paper at ICLR 2020
of average episodic rewards over various training samples for this experiment and compares it with MULTIPOLAR with four low pwwfoorriinng sources. This result suggests that the sample efficiency of MULTIPOLAR (K 4) with we-performing source policies (Le., source policies which had low final episodic rewards in their own environments) almost the same as with randomly initialized source policies.
Results for MULTIPOLAR with low-performing source policies vs, with randomly ini- tialized source policies in Hopper.
Table 13:
MULTIPOLAR (K=4)
0.5M
1M
1,5M
randomly initialized
(26.28)
(45,49)
(70,76)
101 (96,106)
4 low performance [Table
27 (26.27)
45 (44.47)
68 (66,71)
92 (88,95)
B.3 INDIVIDUAL AVERAGE LEARNING CURVES
As example, visualized the individual average learning curve of policies for each of the environment instances of the Roboschool environments. Figures 77 and compare the individual average learning curves of MULTIPOLAR to the baseline policies the 100 target environment instances.
16

---

Under review as a conference paper at ICLR 2020
Rebocchodl
3033
Robondhoo Hogpar
Rebeedhed
1940
2003
1330
266
2330
1500
3300
1-3
Tereslopo
Temesiar
Temestopo
Exinet Aoaree
2300
Editut bosree
2090
20ce
1593
500
1883
sce
3.6
85
1.3
15
60
65
16
25
0.3
0.5
15
26
05
13
15
60
655
26
25
23
Robor tron Heoper
Robssmaohoogeer
1208
1500
1300
23.0
3300
Timelaaa
Teresops
Timudags
3000
2340
Eskout: teweee
Peboochoo Huppar
Caloudt: tewared
appascrentioggee
1331
1500
2300
1300
593
Timidaas
106
Tireeopee
1e5
Threstops
Jnd
Timadaaes
Threetpp
365
1000
spowet seeeg
2000
Qusaatt seauee
2000
2200
1500
1OC8
1780
500
1.3
1.3
68
1.
100
1.
1.3
60
6/5
24
Pobeathor Hogper
Retrodhellhgger
2204
2500
2093
2532
2330
1500
2592
0.5
35
2.0
1.3
0.5
Sebeechodi
Robondhoo Hogpar
aebascresllogzer
Keboechollbgger
2000
2500
2003
Datestt baward
250e
3300
2593
2004
3500
2300
000
290
200
Timidess
Trretope
Trretips
Timadagss
Tirestge
385
2000
2502
Iobordoos
kekocchollkgeer
1500
1993
2300
1530
1300
1566
1300
12ee
Teredtpp
190
Terestass
165
Timedeess
Twredops
2001
Sprust Rouned
250e
2300
3500
200
3300
12.
1.3
15
2.0
60
05
38
35
23
03
3.5
3.
15
20
82
1.3
1.5
20
60
0.5
22
23
3300
3201
Poboo Heapar
1500
3300
2103
3900
1150
1008
200
300
2003
sce
3300
530
TIMOARES
Terestpo
Troolgg
Terestpo
17

---

Under review as a conference paper at ICLR 2020
2000
23m
3893
2006
2330
2300
1593
1100
2300
1991
000
500
590
206
Timodazs
1/4
10
Temsolgga
1-3
5.0
Temesogo
Aessscestooper
Esiladd seware
2500
2003
2004
1500
1500
1008
1300
1590
1800
L504
2000
540
1.3
1.3
055
15
36
1.3
0.0
055
23
Toboashoo Hegpar
heszscnedliogge
Iotcodhollbsper
Eppoanti areard
2003
Epocutti Kevare
2006
2004
1500
3530
1508
1008
2003
120
Timalags
Terestopa
Thrania
Timudaga
Thrastaga
Bobondra Hegper
sidd
2300
kobeacheollbnges
2000
1393
2504
2008
1300
3300
Timiidase
146
Tireetppe
160
Thmethops
Timedagee
3300
2200
2300
15c0
2000
3500
1900
1508
2000
1300
2002
500
1.3
1.5
60
0.5
18
23
03
3.0
355
24
1.3
15
0.5
35
23
3000
Exhoaty st
Aeboctooliggeer
2030
2503
Robordroo
3530
Aobocchollggeer
1100
1500
19m0
1590
2002
sine
3300
300
1.
15
055
16
15
3.0
1.3
15
00
0.5
2000
2000
Spoose Reowett
sitectodllggper
2501
2002
Roboedhoo Hepper
Adbsseseollogeer
beboecholllbgpor
2004
2500
1168
500
300
101
520
Timcdess
Tireslogo
14k
Timcdaaa
Threslopo
Robocdoo
Elouat rewees
2300
2003
Caloot Roware
2300
Ioce
1500
1393
2003
2000
300
18
Timuddass Rosesceettoaae
3el
Tirastapa
305
Retoochoel
Pobers Hogger
Rotrodhol/
Saward
2000
pproeet seward
2300
2002
2330
3903
sice
2093
soce
2300
120
500
593
5N
10
1.5
1-2
50
24
2500
2530
Tobosshoo Hoaper
Caloudie saewer
2000
15:0
1008
zoce
3320
Timidess
Tirettpe
Threttps
365
Timsdaass
Tiretage
Figure 7: Average learning curves of MULTIPOLAR with K in red, RPL in green and MLP in blue over 3 random seeds and 3 random source policy sets for all the 100 target environment instances of Hopper.
18

---

Under review as a conference paper at ICLR 2020
cho
290
3033
Nebecchettr
2500
1500
2202
2000
3500
3300
2593
1893
toce
150e
3300
520
000
500
Timiidace
106
Tireetopt
160
Tiredips
Jnd
Timiddage
Threerpe
265
Splruct: tawant
230
2003
1593
1891
lice
500
2530
530
1.3
1.5
1,0
0.5
3.0
15
24
1.3
15
0.0
0.5
3.6
24
Roosechoo Ane
Clock tharet
2500
2.30
Soberchaolke
1893
Acboctheotyt
2.56
Roooechos
23m0
Ashorcheskir
2033
1300
3000
1500
140
500
513
500
65
16
15
20
1.3
15
00
0.5
Tiredtpe
Teredeps
chonschoor
Rotescteskkt
Nebescheette
Busoschos
Natosceaske
2300
2300
2002
2oce
5300
1593
Crandt
5000
100
2883
s4e
Timoiless
Tereslogo
Timeddess
Tereslopo
Ronoschooo.dt
Robuschentkv
Astoschenthee
2.08
20ce
1500
2003
2502
Qusodd: Taward
1000
2300
3300
500
1.3
655
1.
24
455
1.
1.5
1.3
1.
600
655
1.
24
Rotoxtesket
Tobes:/nettet
Ratoctestks
1500
2500
3520
3300
2004
1593
3300
1201
1000
2900
192
500
1.9
055
36
35
3.
1.3
15
0.0
0.5
12
23
Teretope
Tereeeps
Roooectool Ane
Bobascholkes
Sobeschoclktt
Nateschee
Raboscheskke
Spard Bhuner
2000
2500
120
22.0
1200
1803
2503
2048
3550
100
Timoieess
Terestogo
149
Tiracops
Teresiopo
Rsbaschentkvr
kebesrhwotttt
Astor/heathe
2500
25.0
$2500
2200
2008
22 2230
3300
1490
3300
2300
Teresops
L4
19
24
63
4.5
1.0
1.
Thresiuga
6.0
0.5
1.
24
Retoxteaks
Tobes:/nettet
Ratmuseatks
2000
Calvato femuse
2300
2303
2003
150e
1500
Calrodt
2000
1008
140
300
540
1-2
20
536
1.
1-2
5.0
510
24
Tireetope
Tireetpp Rabouchestke
Rocsechoolket And
Pobaschelket
Tobeschoolkes
Roosechoo Aet
2300
2803
3004
2500
3500
1181
100
133
366
Timeseess
Terestops
296
Threslags
Timedees
Terestopo
380
19

---

Under review as a conference paper at ICLR 2020
asbesched
Robeochotive
kobecclodierr
2504
Rosserhas
2590
Astorchotivv
2260
2568
1568
5300
1900
3093
1593
2000
1568
$300
3300
530
0.5
Terecops
Tireeape
160
Timediops
Timidegs
Teredtpe
100
2004
Salluc bosour
3590
100
85
1.3
15
65
18
15
0.5
1.5
26
05
1.3
L5
60
655
266
25
23
3044
Sobeuctre Aan
2330
2002
2302
2500
3330
Astor/hestkr
1550
1592
2300
1003
2200
2803
LOCE
3.2
Terestopa
Thrastapa
Timudaga
Tirastgg
shrosshor ded
Roboxtestkt
tobeocheoter
Ashenchee
Busosshrrdd ded
Calost: tueere
3300
1202
2248
-30
2300
Caloods:
1989
sce
Timiddace
146
160
Tirestops
Timiidase
2200
2500
Eploust: bapss
2004
3300
500
1.3
1.3
60
0.5
1.
00
65
3.5
24
1.3
1.5
0.0
0.5
3.5
35
23
Roooechoosdd
Acbocthootet
Ashorrhesker
3300
1350
2004
1200
3300
2803
1.3
L5
600
0.5
16
3.0
15
3.0
133
15
0.5
Tireztopa
Tereetpp
2300
1.00
Robssctelktt
2003
1593
Nebescheeter
2000
Buroschas
2200
2300
1000
500
2303
2503
1906
Tereslogo
14k
Timoddas
Tiraalezs
Quieet baree
2500
Quiaet tesand
1500
1993
1008
2000
1.3
1.3
166
6.5
1.
03
0.5
3.0
3.5
1.3
1.3
6.0
6.5
3.
24
Roberchenlke
Tobeschnettet
Rabructeskks
1293
2006
3300
520
500
see
1.3
1.5
0.5
3.6
0.5
3.5
3.0
1.3
15
0.0
0.5 Terenops
Tiventpe
4000
RoosechoolAd Ane
3300
Roboncholkve
2593
Robeschoclkpt
Rossechool Ane
Roboschostke
2048
1pard Tevee
1300
1530
2803
1966
131
Tereslogo
145
Teresiogo
Figure 8: Average learning curves of MULTIPOLAR with K in red, RPL in green and MLP in blue over 3 random seeds and 3 random source policy sets for all the 100 target environment instances of Ant.
20

---

Under review as a conference paper at ICLR 2020
Trredags
1.3
14:3
1880
1 8
col
x
400
Tirasags
Thradags
Timerga
Aesocche www.eeeeeereeddeeee
1880
25
16
24
65
38
15
14
tew.cheh
1::0
3 8
8
Tirasagg
Timaripa
11:00
3 8
1880
Tivedees
15
Treedess
Treedeps
Te Peoeere
1000
8 8 8 &
8 8
Tesachee CREANORT
1400
Thredagss
15
Trredes
Thredags
1000
8 & 8
co
400
200
Trredkss
1000
15
18
Trrasags
14
Treedggs
14
21

---

Under review as a conference paper at ICLR 2020
15
15
12:0
N S
Trradags
53
8
05
15
05
L.
25
503
65
28
15
14
25
Aerrachee undntigus
15:0
Terestegs
Teveden
Threddes
Repouches www.neeeedeeeeeeeooo
2 8 28
15
15
25
14:00
1200
11:0
8 8
Threstago
Tireiles
Thradags
11:0
8 8 & 8
35
20
52
25
10
10:0
TRRoSEE Rowest
B 8
Tivedegs
Trrechos
11:00
3 & 8 &
16
146
Figure 9: Average learning curves of MULTIPOLAR with K = 4 in red, RPL in green and MLP blue over random seeds and 3 random source policy sets for all the 100 target environment instances of InveredPendullmSSinng..
22


Reviews:
[{"review_id": "rJxxKMssdB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a novel multi-source policy transfer problem, where we want to utilize policies from multiple source domains with different dynamics to improve the performance of the policy on our target dynamics. \n\nThe paper addresses the problem by adaptively aggregating the deterministic actions produced by source policies to maximize the expected return in the target environment. The method further trains an auxiliary network to predict a residual to revise the predicted action when some source policies are not useful or even adversarial.\n\n In my understanding, the paper assumes that the source and target domain shared the same task (reward structure) but only differs in dynamics. Also, the two domains share similar state and action spaces since the policy accepts the target states and predict actions in the target environment. This may limit the usage of the method.\n\nThe paper proposes to use residual learning as an auxiliary to compensate for the sub-optimal expressiveness of the source policies, which is novel and interesting.\n\nThe paper performs experiments on multiple environments. But the source and target domains only vary in some parameters of the agents. The domain gap seems small for these experiments. The paper needs a metric to measure the domain gap between source and target dynamics and report how the domain gap influences the proposed method and the baselines according to the metric.\n\nOne of my major concerns is that the limitation of the method with the same state and action space of the source and target domains. Also, there is no theoretical or intuitive analysis of how large the domain gap can be. This problem can be impractical for real-world applications with restrict limitations.\n\nPost-rebuttal:\n\nMy major concern is that the method is a naive combination of previous works and the paper is more like an engineering work. The method is also a weighted sum of source policies. There is no insight why the combination can work.\n\nNo assumption on source policies is given. That means I can get any random policy to learn a combination. This is like learning a policy from scratch by reinforcement learning. With better source policies, we can achieve better initialization for RL.\n\nThe paper is also related to hierarchical reinforcement learning, where the target reinforcement learning step is like building a high-level policy. \n\nThe work still requires lots of steps to train in the target domain, which does not fit to the real application of transfer RL. We hope transfer learning can adapt to the target environment fast.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxD1HJpYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a transfer reinforcement learning method that learns from existing source policies. The method aggregates deterministic actions produced by a collection of source policies to maximize expected return in the target environment. Unlike prior work it does not assume access to source environments nor source policy performance.\n\nThe method is intuitive and simple (simply a weighted sum over the actions of source policies). The paper is well-written in that it clearly explains the method and intuitions. The authors show results on a collection of different environments that include continuous and discrete action spaces. I appreciate the additional work put in to evaluate the distribution of performance. The method is well-ablated and addresses variants in which there is no reweighting and in which the residual is estimated independently of the state.\n\nI have some questions regarding the experiments:\n\n- In Table 1, do the authors have intuitions for why sometimes RPL is worse than MLP?\n- I'd like to see results comparing MULTIPOLAR with only bad sources with a randomly initialized policy\n- Given that source policies are needed for this to work, I'd like to see comparisons in which one continues to finetune an existing source policy. I know that the assumption here is that one does not have access to the internals of the source policies, but it would be nice to see how the performance compares.\n\nMy main concern has to do with the applicability of this method, since it seems to make strong assumptions on how different the domain dynamics are between source and target environments.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeRltpV5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper authors propose a method for transfer reinforcement learning (RL). Specifically they are claiming that RL agents can transfer knowledge between each other about the environment dynamics. In order to showcase their approach they have come up with a new transfer RL task that makes use of some source policies trained under a diverse set of environment dynamics. Their key contributions to solve the task involve a decision aggregation framework that is able to build on top of relevant policies while suppressing irrelevant ones and an auxiliary network that predicts the residuals around the aggregated actions.\n\nI recommend the paper to be accepted since they have an innovative contribution that pushes the needle on the transfer RL literature although I do not think the contribution is substantial. The set of experiments covers a wide range of different standard RL tasks and they provide enough evidence that the approach works. I find it interesting that they are able to extend the approach to the discrete action tasks. \n\nI would however recommend providing more experimental results that provides evidence that the target policy can recover the right policy when the target environment dynamics is the same as one of the source environments.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the case where policies have been learned in several environments - differing only according to their transition functions. The goal is to achieve a policy for another environment on the top of the former policies. The approach is based on learning a state-dependent combination (aggregation) of the former policies, together with a "residual policy". On the top of the aggregated + residual policies is defined a Gaussian distribution. The approach is validated in six OpenAI Gym environments. Lesion studies show that both the aggregation of several policies (the more the better, except for the computational cost) and the residual policy are beneficial. 

Quite a few additional experiments have been conducted during the rebuttal period according to the reviewers' demands (impact of the quality of the initial policies; comparing to fine-tuning an existing source policy).

A key issue raised in the discussion concerns the difference between the sources and the target environment. It is understood that "even a small difference in the dynamics" can call for significantly different policies. Still, the point of bridging the reality gap seems to be not as close as the authors think, for training the aggregation and residual modules requires hundreds of thousands of time steps - which is an issue in real-world robotics.

I encourage the authors to pursue this promising line of research; the paper would be definitely very strong with a proof of concept on the sim-to-real transfer task.
Decision: Reject