Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning
Authors: Remo Sasso
Matthia Sabatelli
Marco A. Wiering
Venue: Accepted by TMLR ()

Abstract: A crucial challenge in reinforcement learning is to reduce the number of interactions with the environment that an agent requires to master a given task. Transfer learning proposes to address this issue by re-using knowledge from previously learned tasks. However, determining which source task qualifies as the most appropriate for knowledge extraction, as well as the choice regarding which algorithm components to transfer, represent severe obstacles to its application in reinforcement learning. The goal of this paper is to address these issues with modular multi-source transfer learning techniques. The proposed techniques automatically learn how to extract useful information from source tasks, regardless of the difference in state-action space and reward function. We support our claims with extensive and challenging cross-domain experiments for visual control.

Published in Transactions Machine Learning Research (04/2023)
Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning
Remo Sasso
r.asso@mullac.uk
Queen Mary University of London
Matthia Sabatelli
m.sabatell@rrug.nl
University of Groningen
Marco A. Wiering University of Groningen
m. wiering@rug..nl
Reviewed on http://www.wpp///.e////////e///a
Abstract
A crucial challenge in reinforcement learning is to reduce the number of interactions with the environment that an agent requires to master given task. Transfer learn- ing proposes to address this issue by re-using knowledge from previously learned tasks. However, determining which source task qualifies as the most appropriate for knowledge extraction, as well as the choice regarding which algorithm components to transfer, rep- resent severe obstacles to its application reinforcement learning. The goal of this paper is to address these issues with modular multi-source transfer learning techniques. The proposed techniques automatically learn how to extract useful information from source tasks, regardless of the difference in state-action space and reward function. We support our claims with extensive and challenging cross-domain experiments for visual control.
1 Introduction
Reinforcement learning (RL) offers powerful framework for decision-making tasks ranging from games to robotics, where agents learn from interactions with an environment to improve their performance over time. The agent observes states and rewards from the environment and acts with a policy that maps states actions. The ultimate goal of the agent is to find policy that maximizes the expected cumulative reward corresponding to the task. A crucial challenge in RI. is to learn this optimal policy with limited amount of data, referred to sample efficiency. This is particularly relevant for real-world applications, where data collection may be costly. In order to achieve sample efficiency, agents must effectively extract information from their interactions with the environment and explore the state space efficiently.
Model-based reinforcement learning methods learn and utilize a model of the environment that can predict the consequences actions, which can be used for planning and generating synthetic data. Recent works show that by learning behaviors from synthetic data, madel-based algorithms exhibit remarkable sam- ple efficiency in high-dimensional environments compared to model-free methods that do not construct

---

Published in Transactions Machine Learning Research (04/2023)
model (Lukasz Kaiser al. 2020 Schrittwieser et al. 2020 Hafner al 2021). However, model-based agent needs to learn an accurate model of the environment which still requires substantial number of interactions. The fact that RL agents typically learn new task from scratch (i.e. without incorporating prior knowledge) plays a major role in their sample inefficiency. Given the potential benefits of incor- porating such prior knowledge (as demonstrated in the supervised learning domain (Yosinski et al. 2014 Razavian et al. 2014 Zhuanget al. 2020)), transfer learning has recently gained an increasing amount of interest in RL. research Zhu et al. 2020). However, since the performance of transfer learning is highly dependent on the relevance of the data that was initially trained on (source task) with respect to the data that will be faced afterward (larget task), its application to the RL domain can be challenging. RL environ- ments namely often differ in several fundamental aspects (e.g. state-action spaces and reward function), making the choice of suitable source-target transfer pair challenging and often based on the intuition of the designer (Taylor & Stone 2009). Moreover, existing transfer learning techniques are often specifically
designed for model- free algorithms [Wan el al.
2020 Heng el al. 2022). transfer within the same domain 2022). or transfer within the same task (Hinton et al. (Sekar et al. 2000 Liu & Abbeel 2021 Yuan et al.
2015 Parisotto al. 2015 Czarnecki et 2019
Agarwal al. 2022).
This paper proposes to address the challenge of determining the most effective source task by automating this process with multi-source transfer learning. Specifically, introduce model-based transfer learn- ing framework that enables multi-source transfer learning for a state-of-the-art model-based reinforce- ment learning algorithm across different domains. Rather than manually selecting single source task, the introduced techniques allow an agent to automatically extract the most relevant information from a set of source tasks, regardless of differences between environments. We consider two different molti-source transfer learning settings and provide solutions for each: a single agent that has mastered multiple tasks, and multiple individual agents that each mastered single task. As modern RL algorithms posed of several components also ensure each of the proposed techniques is applicable in a modular fashion, allowing the applicability to individual components of a given algorithm. The proposed meth- ods are extensively evaluated in challenging cross-domain transfer learning experiments, demonstrating resilience to differences in state-action spaces and reward functions. Note that the techniques are also applicable in single-source transfer learning settings. but we encourage the usage of multi soulie transfer learning, in order to avoid the manual selection an optimal source task. The overall contribution of this paper the adaptation of existing transfer learning approaches in combination with novel techniques, achieve enhanced sample efficiency for model-based algorithm without the need of selecting an optimal source task. The main contributions are summarized as follows:
often com-
Fractional transfer learning: We introduce a type of transfer learning that transfers partial pa- rameters instead of random initialization, resulting in substantially improved sample efficiency.
Modular multi-task transfer learning: We demonstrate enhanced sample efficiency by training single model- agent on multiple tasks and modularly transferring its components with different transfer learning methods.
Meta-model transfer learning: We propose multi-source transfer learning technique that com- bines models from individual agents trained in different environments into a shared feature space, creating a meta-model that utilizes these additional input signals for the target task.
2

---

Published in Transactions Machine Learning Research (04/2023)
2 Preliminaries
Reinforcement learning. We formalize an RI. problem as a Markov Decision Process (MDP) (Bellman 1957). which is tuple (S,A, P. R), where S denotes the state space, the action space, P the transition function, and R the reward function. For taking given action a A in state s E S, P(s'ls, a) denotes the probability of transitioning into state E S, and R(r]s, 4) yields an immedi- ate reward r. The state and action spaces define the domain of the MDP. The objective of RL to find an optimal policy x²' S - A that maximizes the expected cumulative reward. The expected cu- mulative reward is defined G, [E y' ] where y E [0, represents the discount factor, and t the time-step.
f
H++1
Transfer learning. The general concept of transfer learning aims to enhance the learning of some target task by re-using informa- tion obtained from learning some source task. Multi-source transfer learning aims to enhance the learning information from of source tasks. In RL, task is formalized MDFH M with some optimal policy x². As such, in multi-source transfer learning for RL we have collection of N source MDPs M (S,,A,, P., R each with some optimal policy xद. Let M = (S.A, R) denote some target MDP with optimal policy x², where is different from all M, E . with regards to S, A, P, or R. Multi-source transfer learning for reinforcement learning aims enhance the learning of x by reusing information obtained from learning Ti T When N learning.
target task by re-using
0
Figure Dreamer maps environ- ment observations o, to latent states Se and learns to predict the corre- sponding reward r and next latent state $,11 for given action a, These world model components are jointly optimized with a reconstruction loss from the observation model. The world model then used to simulate environment trajectories for learn- ing an actor-critic policy, which pre- dicts actions a, and value estimates that maximize future value predic- tions. Arrows represent the parame- ters of a given model.
have single-source transfer
Dreamer. In this paper, we evaluate the proposed techniques on a state-of-the-ar art deep model-based algorithm, Dreamer (Figure Hafner et al. (2020). Dreamer constructs a model the environ- ment within a compact latent space learned from visual observa- tions, referred as a world model Ha Schmidhuber (2018). That modeled
the transition function and reward function with latent representations of environment states. As such, inter- actions of environments with high-dimensional observations can be simulated in computationally efficient manner, which facili- tates sample-effiiiet policy learning. The learned policy collects data from the environment, which then used for learning the world model.
Dreamer consists of six main components:
Representation model:
Pou (s.ls a-
Observation model:
aen (o.ls:).
We the terms MDP, task, and environment interchangeably in this paper.
3

---

Published in Transactions Machine Learning Research (04/2023)
Reward model:
Muus (r)(s)).
Transition model:
yr (S a-1))
Actor:
(p(())))).
Critic:
vp((s) Egtles Erit (r))
Here p denotes distributions that generate real environment samples, denotes distributions approximat- ing those distributions in latent space, and denotes the jointly optimized parameters of the models. At a given timestep I, the representation model maps visual observation o, logether with the previous latent state St-1 and previous action a;1 to latent state s,. The transition model learns predict 5,-1 from s and a, and the reward model learns to predict the corresponding reward r,- The observation model reconstructs s to match o, providing the learning signal for learning the feature space. This world model is called the recurrent state space model (RSSM), and we refer the reader to Hafner et al. (2019) for further details. To simulate a trajectory {(s,, |== of length H, where r denotes the imagined time index, the representa- tion model maps an initial observation or to latent state 9,, which is combined with action a, yielded by the policy, to predict $;4] using the transition model. The reward model then predicts the corresponding reward fr-1. which used for policy learning.
In order to learn policy, Dreamer makes of an actor-critic approach, where the action model (s(ar(s)) implements the policy, and the value model (s,)) = Eg(is) (E: 2 rr"",, estimates the expected reward that the action model achieves from a given state s; Here & and y are neural network parameters for the action and value model respectively, and is the discount factor. The reward, value, and actor models are implemented as Gaussian distributions parameterized by feed-forward neural networks. The transition model is Gaussian distribution parameterized by a Gated Recurrent United (GRU: Bahdanau et al. (2014)) followed by feed-forward layers. The representation model is a variational encoder (Kingma & Welling 2013 Rezende et al. 2004) combined with the GRU, followed by feed forwar layers. The observation model is a transposed convolutional neural network LCNN;LeCun et al. (2015)) See Appendix Foor further details and pseudocode of the Dreamer algorithm.
3 Related Work
Transfer learning. In supervised learning, parameters of a neural network are transferred by either freez- ing or retraining the parameters of the feature extraction layers, and by randomly initializing the param- eters of the output layer to allow adaptation to the new task (Yosinski et al. 2014 Sabatelli et al.) 2018 Cao et al. 2021). Similarly, we can transfer policy and value models, depending on the differences of the state-action spaces, and reward functions between environments (Carroll & Peterson 2002 Schaal et al. Fernández Veloso 2006 Rusu et al. 2016 Zhang et al. 2018). We can also transfer autoencoders, trained to map observations to latent states CChn et al. 2021). Experience samples collected during the source task training process can also be transferred to enhance the learning of the target task (Lazaric et al.
2004
2008
Tirinzoni et al. 2018).
Model-free transfer learning. Model-free reinforcement learning algorithms seem particularly suitable for distillation techniques and transfer within the same domain and task (Hinton et al.
2015 Parisotto
ltt al. 2015 Czarnecki al. 2019 Agarwal al. 20222. By sharing a distilled policy across multiple agents learning individual tasks, Teh et al. (2017) obtain robust sample efficiency gains. (Saaatlli & Geurts 2021
4

---

Published in Transactions Machine Learning Research (04/2023)
showed that cross-domain transfer learning in the Atari benchmark seems to be a less promising applica- tion of deep model-free algorithms. More impressive learning improvements are observed in continuous control settings for transfer within the same domain transfer across fundamentally similar domains with single and multiple policy transfers (Wan et al. 2020 Heng et al. 2022). Our techniques are instead appli- cable to model-based algorithms and evaluated in challenging cross-domain settings with fundamentally different environments. Ammar et al. (214)) introduce an autonomous similarity measure for MDPs based on restricted Boltzmann machines for selecting an optimal source task, assuming the MDPs are within the same domain. Garcia-Ramirez et al. (2021) propose to select the best source models among multiple model-free models using a regressor. In multiple-source policy transfer, researchers address similar issues but focus transferring a set of source policies with mismatching dynamics to some target policy in model-free settings YYng et al. 2020 Barekatain et al. 2020 Lee et al. 2022).
Model-based transfer learning. In model-based reinforcement learning one also needs to transfer dy- namics model, which can be fully transferred between different domains if the environments are sufficiently similar (Eysenbach et al. 2021 Rafailov al. 2021 Landolfi et 2019) perform multi-task experiment, where the dynamics model of a model-based agent transferred to several novel tasks sequentially, and show that this results in significant gains of sample efficiency. PlaNet (Hafner et al., 2019) was used in a multi-task experiment, where the same agent trained tasks of six different domains simultaneously using a single world model (Ha & Schmidhuber 2018). A popular area of research for transfer learning in model-based reinforcement learning is aassstigosy pre-training the dynamics model followed and introducing a reward function for downstream task afterward (Rajeswar et al. 2022 Yuan et al. 2022). In particular, in Plan2Explore (Sekar et al. 2020) world model aggnssoosll pre-trained, and a Dreamer (Hafner al. 2020 agent uses the model zero-shot or few-shot learn new task within the same do- main. Note that Dreamer is not to be confused with DreamerV2 (Hafner et al. 2021). which is essentially the same algorithm adapted for discrete domains. Unlike previous works, investigate multi-source transfer learning with a focus on such world model- algorithms, by introducing techniques that enhance sam- ple efficiency without selecting single source task, are applicable across domains, are pre-trained with reward functions, and allow the transfer of each individual component.
4 Methods
Here we present the main contributions this work and describe their technical details. First, we introduce multi-task learning multi-source setting and combine it with different types of transfer learning in modular fashion, after introducing a new type of transfer learning that allows for portions of information to be transferred (Section 4.1). We follow this with an alternative setting, where we propose to trans- fer components of multiple individual agents utilizing a shared feature space and a meta-model instead (Section 4.2)
4.1 Multi-Task Transfer Learning
In this section, describe how single agent can be trained multiple MDPs simultancously, introduce novel type of transfer learning, and provide insights on transfer learning in a modular manner. These three concepts are combined to create a modular multi-source transfer learning technique that can autonomously learn to extract relevant information for a given target task.
Simultaneous multi-task learning. We propose to train a single agent facing multiple unknown envi- ronments simultaneously, each with different state-action spaces and reward functions. Intuitively, the
5

---

Published in Transactions Machine Learning Research (04/2023)
parameters of this agent will contain aggregated information from several source tasks, which can then be transferred to target task (similar to parameter fusion in the supervised learning domain (Kendall et al. 2018)). We train the multi-task RL agents by padding the action space of the agent with unused elements to the size of the task with the largest action dimension. Note that we are required to have uniform action dimensions across all source environments, as we use a single policy model. The agent collects one episode of each task in collection phases to acquire a balanced amount of experiences in the replay buffer.
Fractional transfer learning. When two tasks are related but not identical, transferring all parameters of neural network may not be beneficial, as it can prevent the network from adapting and continuing to learn the new task. Therefore, the feature extraction layers of a network are typically fully transferred, but the oulput layer is randomly initialized. We propose simple alternative called fractional transfer learn- ing (FTL). By only transferring a fraction of the parameters, the network can both benefit from the transferred knowledge and continue to learn and adapt to the new task. Let 0r denote target parameters, 0g randomly initialized weights, 2 the frac- tion parameter, and 0s source parameters, then we apply FTL by 0 - 0. + 20s. That is, we add fraction of the source pa- rameters to the randomly initialized weights. This approach is similar to the shrink and perturb approach presented in su- where Gaussian noise is added shrunken weights from the previous training iteration (Ash & Adams 2020). However, we present reformulation and simplifica- tion where we omit the 'noise scale' parameter and simply use Xavier uniform weight initialization Glorot et al. (2011) as the perturbation, such that we only need to specify A. In addition to recent applications to the model-free free setting (Liu al. 2019 Shu et al. (201),, we are the first to demonstrate the application of this type of transfer learning in a deep model- based RL application.
i
(a)
(b)
Figure 2: (a) We train a single agent in multiple (source) environments simultane- ously by alternately interacting with each environment. (b) We transfer the param- eters of this agent in modular fashion for learning a novel (target) task. The representation, observation and transition model are fully transferred (-). The reward and value models are fractionally transferred (---). The action model and the action-input parameters of the transi- tion model are randomly initialized (---).
Modular transfer learning. Modern RL algorithm architec- tures often consist of several components, each relating to different elements an MDP. Therefore, to transfer the pa- rameters of such architectures, we need consider transfer learning on modular level. Using Dreamer as reference, we discuss what type of transfer learning each component benefits most from. We consider three types of direct transfer learning (i.e. initialization techniques): random initialization, FTL, and full transfer learning, where the latter means we ini- tialize the target parameters with the source parameters. As commonly done in transfer learning, we fully transfer feature extraction layers of each component and only consider the output layer for alternative transfer learning strategies. We don't discuss the action parameters, as ac-
6

---

Published in Transactions Machine Learning Research (04/2023)
tion elements and dimensions don't match across different environments, meaning we simply randomly initialize action parameters.
We found that fractionally transferring parameters of the reward and value models can result in substan- tial performance gains (Appendix B In this paper, we apply our methods to MDPs with similar reward functions, meaning the parameters of these models consist of transferable information that enhances the learning of a target task. This demonstrates the major benefits of FTL, as the experiments also showed that fully transferring these parameters has a detrimental effect on learning (Appendix D). From the literature, we know that fully transferring the parameters of the representation, observation, and transition model often results in positive transfers. As we are dealing with visually similar environments, the generality of convolutional features allows the full transfer of the representation and observation parameters (Chen et al. 2021). Additionally, when environments share similar physical laws, transition models may be fully transferred, provided that the weights connected to actions are reset (Landolfi et al. 2019). See Appendix for pseudocode and implementation details of the proposed modular multi-task transfer learning approach, and see Figure for an overview of our approach.
4.2 Multiple-Agent Transfer Learning
We now consider an altermative multi-source transfer learning setting, where we have access to the param- eters of multiple individual agents that have learned the optimal policy for different tasks. Transferring components from agents trained in different environments represents a major obstacle for multi-source transfer learning, To the best of our knowledge, we are the first to propose a solution that allows the combination of deep model-based reinforcement learning components trained in different domains, from which the most relevant information can autonomously be extracted for given target task.
Universal feature space. As the focus of this paper is on world model-based agents that learn their com- ponents in compact latent feature space, we are required to facilitate shared feature space across the agents when combining and transferring their components. From the literature, we know that for visually similar domains, encoder of converged autoencoder can be reused due to the generality of convolu- tional features (Chen et al. 2021). Based on this intuition we introduce universal feature space (UFS): fixed latent feature space that is shared among several world model- agents to enable the transferability of their components. We propose to train single agent in multiple environments simultaneously (as de- scribed in Section 4.1, freeze its encoder, and reuse for training both source and target agents. In this paper, we two different types of environments: locomotion and pendula tasks (Figure(4). Therefore, we decided to train single agent simultancously in one locomotion and one pendulum environment, such that we learn convolutional features for both types of environments. Note that in the case of Dreamer, we do not transfer and freeze the other RSSM components, as this would prevent the Dreamer agent from learning new reward and transition functions. Dreamer learns its latent space via reconstruction, meaning the encoder is the main component of the representation model responsible for constructing the feature space.
Meda-model transfer learning. When agent uses the UFS for training in given target task, we combine and transfer components from agents that were trained using the UPS in different environments. We propose to accomplish this by introducing meta-model transfer learning (MMTL). For a given com- ponent of the target agent, we assemble the same component from all source agents into an ensemble of frozen components. In addition to the usual input element(s), the target component also receives the output signals provided by the frozen source components.
7

---

Published in Transactions Machine Learning Research (04/2023)
Let ma(y)x) denote a neural network component with parameters O some input x, and some output y, belonging the agent that will be trained on target MDP M. Let ma(y)xx denote the same component be- longing to some other agent i with frozen parameters 0 that were fit to some souree MDP M, where i E N, and N denotes the number source MDPs on which a separate agent was trained on from the set of source MDPs M. In MMTI, we modify mo such that we get:
f
f () ... me, (ylx))
where all models were trained within the same UFS. Intuitively, informa- tion signals provided by the source models can be used to enhance the learning process of ma. For instance, assume the objective of M, is sim- ilar to the objective in M. In that case, if we use MMTL for the reward model, my can autonomously learn to utilize the predictions of m& via gradient descent. Similarly, it can learn to ignore the predictions of source models that provide irrelevant predictions. As we are dealing with locomotion and pendula environments in this paper, choose to apply MMTL to the reward model of Dreamer (Figure 3) as the locomo- tion MDPs share a similar objective among each other whilst the pendula environments will provide irrelevant signals for the locomotion environ- ments and vice versa. As such, we can evaluate whether our approach can autonomously learn to utilize and ignore relevant and irrelevant in- formation signals respectively. Note that in the case of Dreamer's reward model, y corresponds to scalar Gaussian paramelerized by mean f and unit variance, from which the mode is sampled. Hence, in our case, y corresponds to fr, and x corresponds to some latent state As such, the target agent will use the same frozen encoder used by the source agents, in addition to using a reward meta-model. We don't transfer any other components the architecture in order to be able to observe the isolated effect of the reward meta- model.
In
Figure 3: Illustration of meta- model transfer learning for Dreamer's 's reward model. The frozen parameters (cyan) of the encoder provide a fixed mapping of environment obser- vations to latent states, which was also used in training each of the source reward models. This latent state is fed to both the source reward models and the target reward model, of which the latter also takes the predictions of the source reward models as input.
This approach applies the same principles as Progressive Neural Networks (Rusu al 2016). as we are retaining, freezing, and combining models from previous learning tasks enhance the performance of novel tasks. Therefore, MMTL is essentially an application of Progressive Neural Networks in multi- source and world model-based setting, where we are dealing with transferring components of multiple agents within the latent space of an autoencoder. By leveraging UFS. we enable this combination and transfer of components trained in different domains. Note that one drawback of Progressive Neural Net- works is that the computation scales linearly with the number of source models, which also applies in our case with the number of source tasks. The frozen autoencoder provides a slight compensation for this additional computation, it is no longer updated in the MMTL approach. See Appendix For pseudocode and implementation details of MMTL.
5 Experiments
Experimental setup. We evaluate the proposed methods using Dreame 2 Hopper, Ant, Walker21, HalfCheetah, InvertePPendullmSSinnuyyy and nnvertedlbbubbeeeenuununnniiiii
on six continuous control tasks:
This work builds upon the code base of Dreamer: https: //ggttub com wwt.c/////aaaaaaaacaooaaaa dreamer
8

---

Published in Transactions Machine Learning Research (04/2023)
Figure Visualization of the tasks learned in the experiments: Hopper, Walker2D, Ant, HalfChectah, In- vertedPendulumSwing and InvertedDoubeePeendduunsssi (S,A, R, P), and are used in multi-source transfer learning experiments as both source and target tasks.
Each of the MDPs differ in all elements
Table 1: Overall average episode return of million environment steps for FTL and MMTL, where param- eters of source lasks were transferred. We compare to a baseline Dreamer agent that learns from scratch. Bold results indicate the method outperforms the baseline for given task.
Task
FTL
MMTL
Baseline
HalfCheetah
1490 + 441
1882 = 390
1199 = 558
Hopper
3430 + 2654
4025 = 3404
2076 = 2417
Walker2D
1637 1 2047
847 = 1533
676 = 1101
InvPend
761 + 68
705
667 = 121
InvDbPend
1235 130
1198 100
1184 89
Ant
681 591
1147 922
1148 408
(Figure 4p. Due restricted compulational budget, we provide extensive empirical evidence for only model-based algorithm and leave the investigation of the potential benefits for other (model-based) algorithms to future work. A detailed description of the differences between the MDPs can be found in Ap- pendix A We perform experiments for both multi-task (referred to as FTL) and multiple-agent (referred to as MMTL) transfer learning settings. For each method, we run multi-source transfer learning experiments using a different set N source tasks for each of the target environments (Appendix B. The selection source tasks for given set was done such that each source set for given target environment includes least task from different environment lype, i.e., pendulum task for locomotion task and vice versa. Similarly, each source set contains at least one task of the same environment type. We also preliminary experiments (3 random seeds) for sets consisting of N [2,3] to observe potential performance differences that result from different N, but we found no significant differences (Appendix C
Hyperparameters. To demonstrate that our methods can autonomously extract useful knowledge from a set of source tasks that includes at least one irrelevant source task, we apply our methods to the most challenging setting (N for 9 random seeds. To the best of our knowledge there exist no available and comparable multi-source transfer learning techniques in the literature that applicable to cross-domain transfer learning applicable Therefore, for each run, we train the FTL and MMTL target agents for 1 million environment steps and compare them to baseline Dreamer agent that learns from scratch for 1 million environment steps, in order to evaluate the sample efficiency gains of the transfer learning approaches. FTL is evaluated by training multi-task agents for 2 million environment steps for a single run, after which we transfer the parameters the target agent described in Section 4.1 We use fraction of À 0.2 for FTL, as we observed in preliminary experiments, the largest performance gains in range of 2 [0.1, 0.3] (see Appendix E
modular level to world model-based algorithm such as Dreamer.
9

---

Published in Transactions Machine Learning Research (04/2023)
Table 2: Average episode return of the final 1e5 environment steps for FTL and MMTL, where parameters 4 source tasks were transferred. We compare to a baseline Dreamer agent that learns from scratch. Bold results indicate the method outperforms the baseline for given task.
Task
FTL
MMTL
Baseline
HalfCheetah
2234 + 302
2458 320
1733 606
Hopper
5517 + 4392
7438 4157
3275 = 3499
Walker2D
2750 2702
1686 2329
1669 1862
InvPend
879
872
InvDbPend
1482 162
1370 106
1392 115
Ant
1453 591
1811 854
1901 480
For creating the UFS for MMTL, we train multi-task agent the Hopper and InvertedPendulum task for 2 million environment steps. We evaluate MMTL by training a single agent on each task of the set of source environments for million environment steps (all using the same UFS), after which we transfer their reward models to the target agent as described Section 4.2 We used a single Nvidia V100 GPU for each training run, taking about hours per million environment steps.
Results.
The overall aggregated return of both FTL and MMTL reported in Table which allows us take jumpstarts into account in the results. The aggregated return of the final 10% (1e5) environment steps can be found in Table Z allowing us to observe final performance improvements. Figure shows the corresponding learning curves.
6 Discussion
We now discuss the results of each of the proposed methods individually and reflect on the overall multi- source transfer learning contributions of this paper. We would like to emphasize that we don't compare FTL and MMTL to each other, but to the baseline, as they are used for two entirely different settings. Additionally, although the proposed techniques are applicable to single-source transfer learning settings well, empirical performance comparisons between multi-source and single-source transfer learning are outside of the scope of this paper, and leave this to future work.
Multi-source transfer learning. The overall transfer learning results show that the proposed solutions for the multi-task setting (FTL) and multiple-agent setting (MMTL) result in positive transfers for 5 out of environments. We observe jumpstarts, overall performance improvements, and/or final performance gains. This shows the introduced techniques allow agents to autonomously extract useful information stemming from source agents trained in MDPs with different state-action spaces, reward functions, and transition functions. We would like to emphasize that for each of the environments, there were at most two environments that could provide useful transfer knowledge. Nevertheless, our methods are still able identify the useful information and result in positive transfer with 4 source tasks. For the Ant test- ing environment, we observe negative transfers when directly transferring parameters in the multi-task setting, which logically follows from the environment being too different from all other environments in terms of dynamics, as we fully transfer the transition model. Note that MMTL does not suffer significantly from this difference in environments, as the transition model is not transferred in that setting.
Multi-task and fractional transfer learning. When training single agent on multiple tasks and transfer- ring its parameters to target task in fused and modular fashion, and applying FTL to the reward and
10

---

Published in Transactions Machine Learning Research (04/2023)
HellCheetah
Hopper
3000
12500
6000
10000
2000
4000
2000
1030
0
-500
-2000
-2000
0.0
0.
04
0.8
0.8
10
3.0
0.2
08
0.8
1.0
0.C
02
04
06
08
10
InvertedPendullmm
Invette@Doubeeeedddduu
Ani
3000
1500
500
1000
500
0.8
0.8
3.0
0.8
1.0
02
10
FTI
Reseline
MMTI
Figure 5: Learning curves for average episode return of million environment steps for FTL and MMTL using 4 source tasks, compared to baseline Dreamer agent that was trained from scratch. Shaded areas represent standard error across the runs.
value models, we observe impressive performance improvements in 5 out of 6 environments. In particular, consistent performance improvements are obtained for the three locomotion tasks that can be considered related, suggesting that fusing parameters multiple tasks alleviates having to choose a single optimal source task for transfer learning. This further demonstrated the results the pendula environments, where just one out of four source tasks related the target task. Regardless, consistent and significant jumpstarts are obtained compared to training from scratch (see Figure Instead of discarding parameters of output layers, and instead transferring fractions of their parameters can play major role in such perfor- mance improvements, initial experiments demonstrated (Appendix E) However, one drawback of FTL is that it introduces a tunable fraction parameter Å, where the optimal fraction can differ per environment and composition of source tasks. We observed the most performance gains in a range of 2 E [0.1, 0.3], and for larger fractions, the overall transfer learning performance the agent would degrade. Similarly, [Ash & Adams (020) found that the shrink and perturb method.
provided the best performance in the supervised learning domain for
Meta-model transfer learning. An alternative multi-source setting considered is where aim to transfer information of multiple individual agents, each trained in different environments, By training both source and target agents with shared feature space and combining their components into a meta-model, we observe substantial performance gains when merely doing so for the reward model. The results suggest that the target model learns leverage the additional learning signals provided by the frozen reward models trained in the source tasks. In particular, we observe impressive performance improvements in locomotion environments, which visually similar and share identical objectives. This suggests that the
11

---

Published in Transactions Machine Learning Research (04/2023)
UFS enables the related source reward models to provide useful predictions for an environment they were not trained with. Note that the source models are also not further adapted to the new environment their parameters are frozen. The overall performance improvements are less substantial, yet still present for the pendula environments. As initial experiments suggested, there being just one related source task for the pendula environments leads to degrading learning performance when adding additional unrelated source models (Appendix C To the best our knowledge, we are the first to successfully combine individual components of multiple individual agents trained different environments as multi-source transfer learning technique that results positive transfers, which is autonomously accomplished through gradient descent and shared feature space provided by the proposed UFS approach. Note that even though there is additional inference of several frozen reward models, for MMTL there is little additional computational complexity as the frozen autoencoder does not require gradient updates, compensating for any additional computation.
7 Conclusion
In this paper, we introduce several techniques that enable the application of multi-source transfer learning to modern model-based algorithms, accomplished by adapting and combining both novel and existing con- cepts from the supervised learning and reinforcement learning domains. The proposed methods address two major obstacles in applying transfer learning to the RL domain: selecting an optimal source task and deciding what type transfer learning to apply to individual components. The introduced techniques are applicable to several individual deep reinforcement learning components and allow the automatic extrac- tion of relevant information from a set of source tasks. Moreover, the proposed methods cover two likely cross- doss--dmminn multi-source scenarios: transferring parameters from single agent that mastered several tasks and transferring parameters from multiple agents that mastered single task. Where existing tech- niques on transfer learning generally focus on model-free algorithms, transfer within the same domain, or transfer within the same lask, we showed that our introduced methods are applicable to challenging cross-domain settings and compatible with a state-of-the-ar model- reinforcement learning algo- rithm, importantly without having to select an optimal source task.
First, we introduced fractional transfer learning, which allows parameters to be transferred partially, as opposed to discarding information commonly done by randomly initializing layer of a neural net- work. We used this type of transfer learning as option in an insightful discussion concerning what type of transfer learning deep model-based reinforcement learning algorithm components benefit from. The conclusions of this discussion were empirically validated in the multi-task transfer learning setting, which both showed that the modular transfer learning decisions result in significant performance improvements for learning a novel target task, and that fused parameter transfer allows for autonomous extraction of useful information from multiple different source tasks. Next, we showed that by learning universal feature space, enable the combination and transfer of individual components from agents trained in environments with different state-action spaces and reward functions. We extend this concept by introducing meta-model transfer learning, which leverages the pre- dictions of models trained by different agents in addition to the usual input signals, as a sulti-source transfer learning technique for multiple-agent settings. This again results in significant sample efficiency gains in challenging cross-doman experiments while autonomously learning to leverage and ignore rele- vant and irrelevant information, respectively.
12

---

Published in Transactions Machine Learning Research (04/2023)
A natural extension for future work is the application of these techniques beyond the experimental setup used in this research, such different environments (e.g. Atari 2600) and algorithms. The techniques in- troduced in this paper applicable on modular level, indicating potential applicability to components of other model-based algorithms, and potentially model-free algorithms as well, Another interesting di- rection would be the application of the proposed techniques in a single-source fashion, and empirically comparing the transfer learning performance to the multi-source approach.
Acknowledgments
We thank the Center for Information Technology of the University of Groningen for their support and for providing access to the Peregrine high-performance computing cluster.
13

---

Published in Transactions Machine Learning Research (04/2023)
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Reincar- nating reinforcement learning: Reusing prior computation to accelerate progress. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Sys- tems, 2022.
Haitham Bou Ammar, Eric Eaton, Matthew E Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An automated measure of mdp similarity for transfer in reinforcement learning. In Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
Jordan T. Ash and Ryan Adams. On warm-starting neural network training. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.04 2014.
Mohammadamin Barekatain, Ryo Yonetani, and Masashi Hamaya. Multipolar: Multi-source policy ag- gregation for transfer reinforcement learning between diverse environmental dynamics. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. International Joint Conferences on Artificial Intelligence Organization, 2020.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, 6(5):679-684, 1957.
Zhangjie Cao, Minae Kwon, and Dorsa Sadigh. Transfer reinforcement learning across homotopy classes IEEE Robotics and Automation Letters, 6(2):2706- 27113, 2021.
JL Carroll and Todd Peterson. Fixed vs. dynamic Sub-Transfer in reinforcement learning. ICMLA, 2002.
Lili Chen, Kimin Lee, Aravind Srinivas, and Picter Abbeel. Improving computational efficiency visual reinforcement learning via stored embeddings. Advances in Neural Information Processing Systems, 34: 26779-26791, 2021.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. lhttp://pybulett org 2016-2021.
Wojeiech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Pro- ceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp- 1331-1340. PMLR, 16-18 Apr 2019.
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov. Off- dynamics reinforcement learning: Training for transfer with domain classifiers. In International Confer- ence on Learning Representations, 2021.
F Fernández and M Veloso. Reusing and building policy library. ICAPS, 2006.
Jesús García- Raiiiez,, Eduardo Morales, and Hugo Jair Escalante. Multi-source transfer learning for deep reinforcement learning. In Pattern Recognition: 13th Mexican Conference, MCPR 2021, Mexico City, Mexico, June 23-26, 2021, Proceedings, pp. 131- Springer, 2021.
14

---

Published in Transactions Machine Learning Research (04/2023)
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. Proceedings of fourteenth international conference artificial intelligence and statistics, pp. 315-323.JMLL. Workshop and Conference Proceedings, 2011.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Sn Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2018.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James David- son. Learning latent dynamics for planning from pixels. International conference on machine learning, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning be- haviors by latent imagination. International Conference on Learning Representations, 2020.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. International Conference on Learning Representations, 2021.
You Heng, Tianpei Yang, YAN Jianye HAO, and Matthew E. Taylor. Cross-domain adaptive trans- fer reinforcement learning based on state-action correspondence. In The 38th Conference on Uncertainly in Artificial Intelligence, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503. 02531, 2015.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7492 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Nicholas Landolfi, Garrett Thomas, and Tengyu Ma. model-based approach for sample-efficiet multi- task reinforcement learning. arXiv preprint arXiv.1907.04964, 2019.
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples batch reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 544-551, 2008
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, (3375555))364444 2015.
Hyun-Rok Lee, Ram Ananth Sreenivasan, Yeonjeong JongseongJ Jang, Dongsub Shim, and Chi-Guhn Lee. Multi-policy grounding and ensemble policy learning for transfer learning with dynamics mis- match. In Proceedings of the Thirty-First International joint Conference on Artificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization, 2022.
Hao Liu and Pieter Abbeel. Unsupervised active pre-training for reinforcement learning, 2021.
lou-Jen Liu, Jian Peng, and Alexander G. Schwing. Knowledge flow: Improve upon your teachers. In 7th International Conference Learning Representations, ICLR, 2019.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, arXiv preprint arX6v.1511.06342, 2015.
15

---

Published in Transactions Machine Learning Research (04/2023)
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 2021.
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexan- dre Lacoste. Unsupervised model-based pre-training for data-efficient reinforcement learning from pix- els. In Decision Awareness in Reinforcement Learning Workshop at ICML, 2022.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning Volume 32, ICML'14, '14 pp. 11-1278-1--126. JMLR.org, 2014.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606. 04671, 2016.
Matthia Sabatelli and Pierre Geurts. On the transferability of deep-q networks. In Deep Reinforcement Learning Workshop of the 35th Conference on Neural Information Processing Systems, 2021.
Matthia Sabatelli, Mike Kestemont, Walter Daclemans, and Pierre Geurts. Deep transfer learning for art classification problems. In Proceedings the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Stefan Schaal, Auke Jan Ijspeert, Aude Billard, Sethu Vijayakumar, and Jean-Arcady Meyer. Estimating future reward reinforcement learning animats using associative learning. In From animals animats Proceedings of the Eighth International Conference on the Simulation of Adaptive Behavior, pp. 297-304.
MIT Press, 2004.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):601-69) 2020.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Plan- ning to explore via supervspe world models. In International Conference Machine Learning, pp. 8583-8592. PMLR, 2020.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806-813 813, 2014.
Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Zoo-tuning: Adaptive transfer from A zoo of models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9626-9637. PMLR, 2021.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Yee Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances neural information processing systems, 2017.
16

---

Published in Transactions Machine Learning Research (04/2023)
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted transfer of samples in reinforcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4936-4945. PMLR, 10-15 Jul 2018.
Michael Wan, Tanmay Gangwani, and Jian Peng. Mutual information based knowledge transfer under state-action dimension mismatch. In Jonas Peters and David Sontag (eds.), Proceedings of the 36th Con- ference Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pp. 1218- 1227. PMLR, 03-06 Aug 2020.
Tianpei Yang. Jianye Hao, Zhaopeng Meng, Zongzhang Zhang. Yujing Hu, Yingfeng Chen, Changjie Fan, Weixun Wang, Wulong Liu, Zhaodong Wang, and Jiajie Peng. Efficient deep reinforcement learning via adaptive policy transfer. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI-20, pp. 3094-3100, International Joint Conferences on Artificial Intelligence Organization, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable features in deep neural networks? arXiv preprint arXiv:1411.1191 17992 2014.
Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Jinyi Liu, Yingfeng Chen, and Changjie Fan. Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model. arXiv preprint arXiv:2210.00498, 2022.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning, arXiv preprint arXiv:1804. 10689, 2018.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint Siii:22000008888 2020.
Fuzhen Zhuang, Zhiyuan Qi. Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76, 2020.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Con- ference Learning Representations, 2020.
17

---

Published in Transactions Machine Learning Research (04/2023)
A Environment Descriptions
In this paper locomotion and pendulum balancing environments from PyBullet (Coumans & Bai 2016 2021) are used for experiments. The locomotion environments have goal to walk to target point that is distanced kilometer away from the starting position as quickly possible. Each environment has a different entity with different numbers of limbs and therefore has different state-action spaces, and tran- sition functions. The reward function is similar, slightly adapted for each entity as the agent is penalized for certain limbs colliding with the ground. The pendula environments have as their objective balance the initially inverted pendulum upwards. The difference between the two environments used that one environment has two pendula attached to each other. This environment is not included in the PyBullet framework for swing-up balancing, which we, therefore, implemented ourselves. The reward signal for the InvertedPendulumSwinng for given observation o is:
To COS O
(1)
where @ is the current position of the joint. For the InverteiDDullPeendlllnnvvinggggg swing-up task, we simply add the cosine of the position of the second joint I to Equation
r cos O cos
(2)
As such, these two environments also differ in state-action spaces, transition functions, and reward func- tions.
B Transfer Learning Task Combinations
In Table 3 the combinations of source tasks and target tasks can be viewed that were used for the experi- ments in this paper. That is, they correspond to the results of Appendix C and Section
Table Source-target combinations used for FTL and MMTL experiments, using environments HalfChee- tah (Cheetah), Hopper, Walker2D, Innrrwgu/endlluunSSnnuuu (InvPend), InvertedDoubllePenduuun Swingup (InvDbPend), and Ant.
Target
2 Tasks
Tasks
Tasks
Cheetah
Hopper, Ant
Walker2D
+InVPend
Hopper
Cheetah, Walker2D
Ant
+lvPend
Walker2D
Cheetah, Hopper
Ant
+IvPeedd
InvPend
Cheetah, InvDhPend
Hopper
Ant
InvDbPend
Hopper, InvPend
-WakerrDD
Ant
Ant
Cheetah, Walker2D
Hopper
InvPend
18

---

Published in Transactions Machine Learning Research (04/2023)
C Preliminary Experiments for 2 and Source Tasks
In Table and Table the results for training on, and transferring, and 3 source lasks as described in 5 can be found for both FTL and MMTL, showing the overall average episode return and the episode return for the final 1e5 environment steps respectively. In Figure 6 the corresponding learning curves can be found. These experiments are the averages and standard deviation across 3 seeds for FTI, MMTI, and a baseline Dreamer trained from scratch. As just 3 seeds were used for these experiments, they are not conclusive. However, they do show that regardless the number of source tasks, our methods can extract useful information and enhance the performance compared the baseline.
Double Source
Triple Source
HalfCheetah
Hopper
HellCheelali
Hupper
3000
10000
10000
2000
2000
1000
5000
0
0
0
2000
500
2000
0.0
0.2
0.6
0.8
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
InvertedRendulum
InvertedDDoblleeendulum
InvertedPendulum
Inverte@Doubleeenddllum
1000
1750
1000
1750
Epieade Return
1500
500
500
1000
1000
0
500
500
-500
c
-1000
250
-25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.C
0.6
0.2
0.5
Ant
Walker2D
Ant
Walken 2D
3000
3000
3000
2000
2000
1000
2000
1000
1000
0
1000
1000
-1000
0
2000
2000
0.0
0.2
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps
1e6
MMTL
FIL Baseline
Figure Preliminary experiments: average episode return for random seeds obtained across million environment steps by MMTL (red), FTL (blue), and vanilla Dreamer (green), where for FTL 0.2 used. FTL and MMTL both receive transfer from combination of 2 (double) and 3 (triple) source tasks. Shaded areas represent the standard deviation across the 3 runs.
19

---

Published in Transactions Machine Learning Research (04/2023)
Table 4: Preliminary experiments: overall average episode return with of FTL using 2 = 0.2 and MMTL for the reward model. Parameters of 2 and source tasks are transferred to the HalfCheetah, Hopper, Walker2D, InvertedPenuulumSnnnuu (InvPend), lnvercedDoobbeeenttuuusSiig
(InvDbPend), and Ant tasks, and compared baseline Dreamer agent that learns from scratch. Bold results indicate the best performance across the methods and baseline for given task.
Fractional Transfer Learning
Meta-Moder Transfer Learning task
Task
task
3 task
task
Baseline
HalfCheetah
1982
862
2057
2078 721
1681 = 726
Hopper
1911 =
= 4720
5085 + 4277
0019:44115
1340 1112
Walker2D
1009 1254
393 = 813
196 860
2
885
InvPend
884
332
740
723 = 364
InvDbPend
1209 280
1299 + 254
1214 230
1120 327
1194 + 306
Ant
1124 722
1052 + 687
1423 788
1366 687
1589 + 771
Table Preliminary experiments: average episode return of the final 1e5 environment steps of FTL ing À 0.2 and MMTL for the reward model. Parameters of and 3 source tasks transferred to the HalfCheetah, Hopper, Walker2D, InvertedPendullmSSiin Swingup (InvDbPend), and Ant tasks, and compared to baseline Dreamer agent that learns from scratch. Bold results indicate the best performance across the methods and baseline for a given task.
(InvPend), IivertedDoubleeenddd
Fractional Transfer Learning 3 task
Meta Model Transfer Learning
Task
2 task
task
3 task
Baseline
HalfCheetah
2820 a 297
2615 = 132
2618 362
2514 183
2264 2 160
Hopper
2535 a 712
8274 a 46 490
8080 4704
10 a 2241
2241 a 502
Walker2D
2214 + 1254
963 = 314
846 286
730 343
547 = 710
InvPend
874 : 121
884 = 20
885 14
884 13
883 = 17
InvDbPend
1438 116
1531 = 93
1348 171
1302 152
1366 179
Ant
2021 722
1899 292
2212 607
2064 386
2463 208
D Full Transfer
In Figure learning curves can be seen for the HalfCheetah task where we fully transfer the parameters instead of using FTL. As can be seen, this results in a detrimental overfitting effect where the agent does not seem to be learning.
N=2
N
N=4
Episode Return
3000
3000
3000
2000
2000
1000
1000
-500
0.0
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
le6
Environment Steps
1e6
Environment Steps
1e6
full transfer
dreamer
Figure 7: 4 seeds for full transfer learning the HalfCheetah task across 2, 3, and 4 source tasks.
20

---

Published in Transactions Machine Learning Research (04/2023)
E
Additional Fraction Results
In this appendix, results can be found that illustrate the effect of different fractions A. We present the numerical results of transferring fractions of À E [0.0, 0.2, 0.3, 0.4, 0.5] using 2, 3, and 4 source tasks. That is, the overall performance over 3 random seeds for the HalfCheetah and Hopper, Walker2D and Ant, and InvertedPendullnnSSinngu and in Table Table 7 and Table 8 respectively. Again, as just 3 seeds were used for these experiments, they are not conclusive. However, they do indicate that the choice of A can largely impact performance, and generally, the best performance gains are yielded with 2 [0.1, 0.3).
testing environments can be found
Table Average return for fraction transfer 3, and source tasks for the HalfCheetah and Hopper tasks, with fractions À E [0.0, 0.1, 0.2, 0.3, 0.4,0.4,,, 0.53. Bold results indicate the best performing fraction per number of source tasks for each target task.
HalfCheetah tasks
Hopper
2 tasks
4 tasks
2 tasks
tasks
4 tasks
0.0
1841 806
1752 783
1742 777
1917 960
1585 941
1263 1113
0.1
2028 995
2074 762
1500 781
2041 887
6542 4469
3500 3342
1982 838
1967 862
1773 748
1911 712
5538 4720
1702 1078
1899
2094 859
2670
4609
873
2076
#:
3828
Baseline
1681 * 726
1340 # 1112
Table Average return for fraction transfer of 2, 3, and 4 source tasks for the Walker2D and Ant tasks with fractions E [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]. Bold results indicate the best performing fraction per number of source tasks for each target task.
Walker2D 3 tasks
Ant
A
tasks
tasks
tasks
tasks
tasks
0.0
546 776
545 1012
157 891
1070 659
923 559
897 586
0.1
360 803
441 852
478 1113
806 469
1022 645
834 652
1254
a 722
687
898
914
824
+
554
742 +
1074
735
0.5
352 2 853
355
396 829
+ 651
932 + 609
683 1 498
Baseline
116 885
1589 + 771
Table Average return for fraction transfer of 2, 3, and source tasks for the InvertedPendllanSSiinuuu and vvertedDoobbePeedduuuivvii tasks with fractions E [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]. Bold results indi- cate the besf-perg fraction per number of source tasks for each target task.
InvertedP enduuunSSinunggg
InvertedDouble owgubbuubuuedududduuu/iiwgggg
2 tasks
tasks
4 tasks
2 tasks
tasks
4 tasks
847 135
779 259
838
1255 270
1303 248
1208 322
198
1185
1209
254
+ 182
1279
+
789 : 285
790 237
1307 227
1323 : 243
1278 : 287
126
830 1 181
791 a: 304
1301 216
1340 a 230
1206 a 278
Baseline
723 + 364
1194 306
21

---

Published in Transactions Machine Learning Research (04/2023)
F Algorithms
presents pseudocode of the original Dreamer algorithm (adapted Holner et al. 20200) with seed episodes S, collect interval C, batch size B, sequence length L, imagination horizon H, and learning rate
Algorithm preesinttpreeentts
Algorithm presents the adaptation of Dreamer such that it learns multiple tasks simultaneously. Finally, Algorithm
and and (ggorithmmppreeeee the procedures of fractional transfer learning using a multi-task agent and meta-model transfer learning for the reward model of Dreamer, respectively. Note that Algorithm [ is the base algorithm, and we omit identical parts of the pseudocode in the other algorithms to avoid redundancy and enhance legibility.
Algorithm Dreamer
Initialize dataset D with S random seed episodes. Initialize neural network parameters OREF, loas, GREW, ITRANS, 0, & randomly. while not converged do for update step 1... do
/ Dynamics learning
Draw B data sequences {(a, o, r D. Compute model states S Poar Se-1, 4r-1, or).
Update 0 using representation learning.
// Behavior Learning
Imagine trajectories {(s,, a=) from each st. Predict rewards Eqgaa (r./ss) and values (st).
Compute value estimates Va(s)).
Update & - Φ aV, I-H V2(s))
Update - N E 2 ( V
// Environment interaction 0j env reset () for time step
T do
Compute & ~ Plker (t-15-11 4-1, o,) from history. Compute 4; q (+)ss) with the action model.
Add exploration noise action.
Fo 9t+1 env step(a)
Add experience to dataset D - D u (lo,, ).
22

---

Published in Transactions Machine Learning Research (04/2023)
Algorithm 2: Multi-Task Dreamer
Initialize dataset D with S random seed episodes.
Initialize neural network parameters GREP. Boss, BREW, TRANSS o. y randomly. while not converged do
// Dynamics learning
// Envirntment interaction for environment == O...N do 0 - env, reset() for time step 1 1... do
Compute 5; Se ar- 0,) from history.
Compute (s(a)(s;) with the action model.
Add exploration noise to action.
Y, 0++1 + env. ste. (a,). Add experience dataset D Du {(o,, au r.)
Algorithm 3: Fractional Transfer Learning with Multi- Masti--Tak.ao Agent
// Train multi-task agent OrEP lons,, OREw, ÓTrANS, oss v/s train multi task((envo.... ... envy (Algorithm) 2
// Apply transfer learning
Initialize neural network parameters OgEr, loss. GRow, ōTRANS, d, & randomly.
OREP + OREP,
doss= Oons;
ITAASS OTRANS,
BRE BRew 7BEEW
V - v/ Av/s
I +-a à
while not converged do
for update step = 1... do
// Dynamics learning
// Behavior learning
Environment interaction
23

---

Published in Transactions Machine Learning Research (04/2023)
Algorithm Meta Model Transfer Learning for Reward Model
// Train multi-tast agent to obtain UFS autoencoder Our - train_ train_rulti_iteeeee
envx (Algorithm 2
// Train individual source task agents for O... do
Magw.. - train(env,, OBEP) (Algorithm) 1
// Create meta-model and initialize parameters ge r 9bww, Initialize neural network parameters lons, fTRANS, &, v randomly. while not converged do for update step = 1... do
// Dynamics learning
// Behavior Learning
Imagine trajectories ((s,, a))) from each st. Predict rewards Eqa(r=|s.) and values ug(s))
Compute value estimates Vs(s.).
Update & - x RV, I-H (2(s))
Update - N E (up(ss) Va(s)))
11 Environment interaction
24


Reviews:
[{"review_id": "oRzDgFxRW9", "reviewer": "Reviewer_YJjx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- The paper studies a key problem in reinforcement learning, how to reuse prior/previously learned knowledge is currently an active area of research.\n\nCons:\n\n- Overall I found the paper really hard to follow. The paper mentions many techniques in two different setups, it is hard to contribution of each of these techniques. Many terms are introduced while they already exists in the literature and the definition are not properly highlighted in the document. In the end I had a hard time to differentiate simultaneous multi task learning, meta model transfer learning, modular transfer learning while the paper barely has space to investigate the quality of the proposed techniques.\n- The paper mentions many algorithms / techniques but do not provide a pseudo code of the final algorithm, making it hard to understand the actual algorithm evaluated. It would also be useful to provide an ablation of the different components of the proposed algorithms to understand their relative importance.\n- The paper ignores a lot of existing literature on transfer learning in reinforcement learning and even mentions that: \"transfer learning receives little attention in RL research.\", \"The transferability of deep model-free RL algorithms doesn’t appear promising\". I refer the authors to the call of papers of the upcoming workshop on Reincarnating RL that includes many relevant references: The transferability of deep model-free RL algorithms doesn’t appear promising.\n- The paper focuses on Dreamer, does it mean that the proposed technique only works with this algorithm or it would work with any model based reinforcement learning algorithm? The paper also does not justify why it focuses on model based algorithms.\n- The baseline used in the paper is a Dreamer algorithm trained from scratch, this is a weak baseline in the context of transfer learning, simply finetuning the weights pretrained on the source task would be a stronger baseline (though I recommend the authors to compare with existing transfer learning algorithm).\n- \"we are the first to propose a solution that allows the combination of models from agents trained in different environments, from which the most relevant in- formation can autonomously be extracted for a given target task.\" this has been done previously e.g progressive networks. The relationship between Meta model transfer learning and progressive was actually highlighted by authors. Though it might be worth mentioning that this technique is not widely used due the fact that computation grows linearly with the number of tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents multiple transfer learning techniques to extend the model based reinforcement learning algorithm dreamer. The authors discuss several methods for transfer learning, one where a single agents interacts with multiple environments and one where the knowledge from multiple agents is used to solve a target environment.\nThe authors then evaluate the proposed transfer learning methods on continuous control problem and show that it improves the vanilla Dreamer algorithm.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some changes I would require for acceptance:\n\n- \"We propose techniques for two different multi-source transfer learning settings: a single agent that has mastered multiple tasks, and multiple individual agents that each mastered a single task\" I think it would be better to focus on a single setting and provide a more thorough investigation of the proposed methods in this setting.\n-The paper should update its section on transfer learning in deep rl and include stronger baselines to compare with their algorithm.\n- Show that the proposed techniques are not limited to Dreamer and could be applied with success on another model based learning algorithm.\n\nNice to see:\n\n- Experiments on environments using pixels as inputs like Atari or Mujoco from pixels.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2hnPFAqDGP", "reviewer": "Reviewer_eue6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- This paper learns an important yet less studied problem of multi-source policy transfer in deep reinforcement learning.\n- The paper motivates a novel angle that considers a model-based approach to address the challenge in multi-source transfer learning, while most conventional works fall into the category of model-free policy learning.\n\n**Weaknesses**:\n- This paper highly depends on an existing model-based continuous control method Dreamer, which reduces both the technical novelty and generality of the proposed method. It is good to think from the model-based perspective, but regretfully the authors did not manage to develop solid technical solutions to address model-based multi-source policy transfer. The only thing that seems related is the fractional parameterization part, where the authors put a very strong assumption that reward and value modules are highly encouraged to apply FTL whereas the other modules in Dreamer, like representation and transition, need full transfer. The idea of applying FTL to model-based Dreamer appears to be a bit heuristic and trivial, with missing empirical evidence. I also feel it is very hard to apply FTL or the meta-model transfer MMTL to other model-based solutions apart from Dreamer to solve general multi-source transfer problems.\n\n- The formulation of the method is not very sound, with many important concepts about the multi-source transfer problem left undefined. I think the problem tackled by this paper falls into one of the multi-source transfer learning categories, where with pre-trained source task knowledge/models, their method adds *one* target task to the source task set and updates those tasks thereafter to train the target task. The aforementioned type is not the only way multi-source transfer can be formed. For instance, earlier DQN-variant of multi-source policy transfer methods such as policy distillation [1] and AMN [2] trains multiple target domains simultaneously. I feel the problem setting could be more clearly defined. Some other important concepts, such as *full transfer* and *direct transfer* that has been mentioned several times in the paper are also undefined. \n\n- There are very few math formulas in the method section, which makes the writing hard to follow. Some important formulas are ambiguously defined, e.g., FTL update is defined as $\\theta_T \\leftarrow \\theta_\\epsilon + \\lambda \\theta_S$ in the paper, which I think is wrong, as $\\theta_S$ is actually a parameter set. Is $\\theta_S$ frozen during training? Also, for FTL, the parameters over multiple source domains are straightforwardly added to a random initialization, weighted by a predefined scalar value, which is a bit heuristic. It would be more interesting if the method could automatically adjust $\\lambda$ by itself. \n\n- I'm not convinced about the feature concatenation idea for the meta-model transfer learning (MMTL) part. First, the authors require source tasks to inference state from the disjoint target task to generate the latent feature to be ensembled, which I feel might not be valid for general multi-source transfer learning problems. Second, it is hard to define which layer of the neural network is most suitable to extract the additional feature in MMTL. Last but not least, the authors make a very strong claim that their method could learn the most relevant source task, for which I disagree. It is unclear what kind of weight the inference module would give over each source task embedding from $m_{\\theta_0}(y|x), ..., m_{\\theta_N}(y|x)$, i.e., if source task 0 and N are positively and negatively correlated with the target task, would the MMTL model always give higher weight to $m_{\\theta_0}(y|x)$ than $m_{\\theta_N}(y|x)$?\n\n- The empirical evaluation results are interesting. I think the results try to tell us that if we train a target task in a single-task way, we would result in the scores under the 'Baseline' (Dreamer)'s standard; but if we use more tasks (target + multiple source tasks), with known knowledge on the source tasks, plus the FTL/MMTL technique proposed in this paper, we can achieve better score *for the target task* after the multitask training. One baseline is necessary to add, which is directly training the target task with source tasks together without FTL or MMTL. I would also be curious about how much additional computational complexity would the transfer learning enquire compared to training from scratch. It would also be good if the authors could present results on fully transferring or not each of the modules in Dreamer.\n\n- It's common sense that we could not group random tasks together for multi-task transfer learning, and it is essential to know what kind of problems can be effectively tackled by the method presented in this paper. In many places, the authors have mentioned terms like *task similarity* and *visual similarity* to measure the relationship between tasks, for which I think a definition is needed. It is also good to know apart from the six continuous control problems, what are the problem domains that the proposed method could address.\n\n- For future work, the authors have mentioned an extremely interesting direction, which is to apply the method to Atari 2600 domain. Since there is a version of Dreamer for Atari, could the authors comment on whether the method could successfully work on multi-source transfer learning with Atari 2600 games, where the input of the method is high-dimensional and comes with visual differences at game-specific pixel-level?\n\n[1] POLICY DISTILLATION.\n\n[2] ACTOR-MIMIC DEEP MULTITASK AND TRANSFER REINFORCEMENT LEARNING.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenges in multi-source transfer learning for deep reinforcement learning problems with the following techniques developed upon an existing model-based continuous control framework Dreamer: (1) **fractional transfer learning (FTL)**, where the target task initialization is defined as a weighted sum between random initialized values plus scaled source parameter values; (2) **modular transfer learning**, where among different components for Dreamer, some of them should be *fully transferred* (e.g., representation, observation, and transition) whereas the others should be *fractionally transferred* (e.g., reward function and value function); (3)  **universal feature space** where the encoder of a trained autoencoder can be reused across tasks can be used together with a **meta-model transfer learning (MMTL)** module, where the later one ensembles the latent prediction from all source task models to be concatenated to the latent of target task, to let agent automatically learn to distinguish relevant and irrelevant features.\n\nFor empirical evaluation, the authors consider six continuous control domains as the target task, respectively, where the multi-task transfer adopts source domains with sizes 2 to 4, with the source domain consisting of both positively and negatively correlated domains to the target task. The authors evaluate FTL, MMTL, and Dreamer (baseline) in multi-task learning settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **Problems with references**: (1) many of the citations are out of date, e.g., \"Distral: Robust multitask reinforcement learning\" and \"Efficient deep reinforcement learning via adaptive policy transfer\" are all published as conference papers, but both are cited as arxivs;  (2) some arxiv citations come with URL but the others not; (3) the format for all conference proceedings and  should be aligned properly.\n\n- **Typo**:  (page 4) we propose to leverage parameter fusion -> add a full stop at the end of the sentence.\n\n- The appearance of Table 1 and Table 2 could be moved before Section 6.\n\n- It would be good if an illustrative figure on Dreamer and Dreamer's important math formulas could be added to Section 2 when the authors attempt to introduce Dreamer in preliminary.\n\n- Wrap Figure 1 and Figure 2 with proper margins.\n\n- Table 2 caption: 1e-5 should be changed to 1e5?\n\n- Redraw Figure 2 and Figure 2 with a clear relationship between source tasks and the target task.\n\n- It would be good if algorithms will be added to either the main body or the appendix to describe the procedure of MTL with the proposed methods.\n\nPlease refer to the weaknesses part for other suggestions on the formulation, motivation, and empirical evaluation.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AqJvrqfRzn", "reviewer": "Reviewer_qCVq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method is novel from the perspective of model-based transfer learning.\n\nThe paper is clearly written and technically substantiated.\n\nWeaknesses:\n\nThe literature review is not extensive, especially missing the discussion about cross-domain transfer methods, such as [1,2].\n\nFurthermore, I am not convinced with the statement 'to the best of our knowledge, we are the first to demonstrate the application of this type of transfer learning in deep RL settings.' It has been used for several years, since [3]. Two different ways of fraction transfer learning: one is a weighted combination of network layer output [1-3], and the other is the weighted combination of network layer weights [4,5].\n\nNo other transfer baselines are covered in the experiment section. It's better to provide SOTA baselines, such as [6], (rather than variants of the proposed methods, this is more like ablation studies), or explain why other baselines are not comparable.\n\nThe structure of Sections 5 and 6 could be reorganized and optimized. There is little discussion about the training results in Section 5.\n\n[1] Cross-domain Adaptive Transfer Reinforcement Learning Based on State-Action Correspondence\n\n[2] Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch.\n\n[3] Progressive Neural Networks.\n\n[4] Knowledge Flow: Improve Upon Your Teachers.\n\n[5] Zoo-Tuning: Adaptive Transfer from a Zoo of Models.\n\n[6] EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a model-based transfer framework to achieve multiple source policy transfer in the continuous control domain.\n\nThe contributions are summarized below:\n\n1) A Fractional transfer learning mechanism is employed to transfer a part of parameters from source policies.\n\n2) A meta-model transfer learning technique is proposed to transfer different components to the target task in a meta manner.\n\n3) Experiments on cross-domain continuous control tasks show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major revisions should consider the literature review and experiment sections. Please refer to the pros and cons part.\n\nSome inaccurate descriptions should also be clarified.\n\nThe typos should be fixed in the revised version.\n\nMultiple entries for the same reference should be fixed.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with multi-source transfer learning for dreamer, as an example of (deep) model-based RL. The initial reviews raised some concerns, such as clarity or links to progressive networks. The rebuttal and revision adequately addressed these concerns, I therefore follow the reviewers in recommending acceptance.
Decision: Accept (poster)