Sharing Knowledge in Multi-Task Deep Reinforcement Learning
Authors: Carlo D'Eramo
Davide Tateo
Andrea Bonarini
Marcello Restelli
Jan Peters
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Multi-Task

Abstract: We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

Published as a conference paper at ICLR 2020
SHARING KNOWLEDGE IN MULTI-TASK DEEP REINFORCEMENT LEARNING
Carlo D'Eramo & Davide Tateo
Department of Computer Science
TU Darmstadt, IAS Hochschulstraße 10, 64289, Darmstadt, Germany
[carlo. deramo, davide. tateo] @tu-ddrmstadt.ccm
Andrea Bomarini & Marcello Restelli
Politecnico di Milano, DEIB
Piazza Leonardo da Vinci 32, 20133, Milano (andrea bonarint ni, ma r ce 1o. restelli @pol imi itt
Jan Peters
TU Darmstadt, IAS
Hochschulstraße 10, 64289, Darmstadt, Germany
Max Planck Institute for Intelligent Systems
Max-Planck-Ring 4, 72076, Tübingen, Germany jan peterstt u
ABSTRACT
We study the benefit of sharing representations among tasks enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, helpful to generalize the knowledge of them resulting in more effective feature ex- traction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well- known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, complement analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.
INTRODUCTION
Multi-Task Learning (MTL) ambitiously aims to learn multiple tasks jointly instead of learning them separately, leveraging the assumption that the considered tasks have common properties which can be exploited by Machine Learning (ML) models generalize the learning of each of them. For instance, the features extracted in the hidden layers of neural network trained on multiple tasks have the advantage of being general representation structures common to each other. This translates into an effective way of learning multiple tasks at the same time, but it can also improve the learning of each individual task compared to learning them separately (Caruana, 1997). Furthermore, the learned representation can be used to perform Transfer Learning (TL), i.c. using it as preliminary knowledge to learn new similar task resulting in a more effective and faster learning than learning the new task from scratch (Baxter, 2000; Thrun & Pratt, 2012).
The same benefits of extraction and exploitation of common features among the tasks achieved in MTL, can be obtained in Multi-Task Reinforcement Learning (MTRL) when training single agent on multiple Reinforcement Learning (RL) problems with common structures (Taylor Stone, 2009; Lazaric, 2012). In particular, in MTRL agent can be trained multiple tasks in the same

---

Published as a conference paper at ICLR 2020
domain, e.g. riding bicycle cycling while going towards goal, or on different but similar domains, c.g. balancing a pendulum or balancing a double pendulum'. Considering recent advances in Deep Reinforcement Learning (DRL) and the resulting increase in the complexity of experimental benchmarks, the use of Deep Learning (DI) models, e.g. deep neural networks, has become a popular and effective way to extract common features among tasks in MTRL. algorithms (Rusu al., 2015; Liu et al., 2016; Higgins et al., 2017). However, despite the high representational capacity of DL models, the extraction of good features remains challenging. For instance, the performance of the learning process can degrade when unrelated tasks are used together (Caruana, 1997; Baxter, 2000); another detrimental issue may occur when the training of single model is not balanced properly among multiple tasks (Hessel et al., 2018).
Recent developments in MTRL, achieve significant results feature extraction by means of algorithms specifically developed to address these issues. While some of these works rely on single deep neural network to model the multi-task agent (Liu et al., 2016; Yang et al., 2017; Hessel et al., 2018; Wulfmeier et al., 2019), others use multiple deep neural networks, e.g. one for each task and another for the multi-task agent (Rusu et al., 2015; Parisotto et al., 2015; Higgins et al., 2017: Teh et al., 2017). Intuitively, achieving good results in MTRL with single deep neural network is more desirable than using many of them, since the training time is likely much less and the whole architecture is easier to implement. In this paper we study the benefits of shared representations among tasks. We theoretically motivate the intuitive effectiveness of our method, deriving theoretical guarantees that exploit the theoretical framework provided by Maurer et al. (2016), in which the authors present upper bounds on the quality of learning in MTL when extracting features for multiple tasks in a single shared representation. The significancy of this result is that the cost of learning the shared representation decreases with factor ((/VT), where T is the number of tasks for many function approximator hypothesis classes. The main contribution of this work is twofold.
1. We derive upper confidence bounds for Approximate Policy-Iteration (API)² (Farahmand, 2011 1) in the MTRL setting. and we extend the approx- imation error bounds in Maurer et al. (2016) to the case of multiple tasks with different dimensionalities. Then, we show how to combine these results resulting in, to the best of our knowledge, the first proposed extension of the finite-time bounds of AVI/API to MTRL. Despite being extension of previous works, we derive these results to justify our approach showing how the error propagation in AVI/API can theoretically benefit from learning multiple tasks jointly.
(AVT) and Approximate
2.
We leverage these results proposing a neural network architecture, for which these bounds hold with minor assumptions, that allow us to learn multiple tasks with a single regressor extracting a common representation. We show an empirical evidence of the consequence of our bounds by means of a variant of Fitted Q-Iteration (FQI) (Ernst et al., 2005), based on our shared network and for which our bounds apply, that we call Multi Fitted Q-Iteration (MFQI). Then, we perform an empirical evaluation in challenging RL problems proposing multi- task variants of the Deep Q-Network (DQN) (Mnih et al., 2015) and Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) algorithms. These algorithms are practical implementations of the more general AVI/API framework, designed to solve complex problems. In this case, the bounds apply to these algorithms only with some assumptions, e.g. stationary sampling distribution. The outcome of the empirical analysis joins the theoretical results, showing significant performance improvements compared to the single- task version of the algorithms in various RL problems, including several MuJoCo (Todorov et al., 2012) domains.
PRELIMINARIES
Let B(x)) be the space of bounded measurable functions w.r.t. the o-algebra ox, and similarly B(x, L) be the same bounded by L < co.
Markov Decision Process (MDP) defined as a 5-tuple M =< S,A,,, where S the state space, A is the action space, P S x A + S is the transition distribution where P(s'|s,
For simplicity, in this paper we refer to the concepts of task and domain interchangeably. All proofs and the theorem for API are in Appendix A.2.
2

---

Published as a conference paper at ICLR 2020
is the probability of reaching state s' when performing action in state s, R S A is the reward function, and y E (0, is the discount factor. A deterministic policy л maps, for each state, the action to perform: a in a state 8 represents the expected discounted cumulative reward obtained by performing in s and following и thereafter: Q" (s, a) & E| r*+++++1| = 8, 0,0; = a. T., where ri+1 is the reward obtained after the i-th transition. The expected discounted cumulative reward is maximized by following the optimal policy nt which is the one that determines the optimal action values, i.e., the ones that satisfy the Bellman optimality equation (Bellman, 1954): (* a) & Is ((s'(s,,) a) [R(s, a, + max, (" a') ds'. The solution of the Bellman optimality equation is the fixed point of the optimal Bellman operator T B(S A) + B(S A) defined as In the MTRL setting, there multiple MDPs MM =< S(0,, ,(()), p(o, R(0, 700) > where E {l,...,T} and is the number of MDPs. For each MDP M a deterministic policy n. S( + AA()) induces an action-value function (s(t), ( = = k=o > ++h++1ss = s(t),
- A. Given a policy 9, the value of action
(s) a) A Sss P(s'|ss, a)(R(s, a, s') + max, ((s',a')jd
a mel. In this setting, the goal is to maximize the sum of the expected cumulative discounted reward of each task.
In our theoretical analysis of the MTRL problem, the complexity of representation plays central role. As done in Maurer et al. (2016), we consider the Gaussian complexity, variant of the well-knows Rademacher complexity, to measure the complexity of the representation. Given set X E X of n input samples for each task E (I,..., ,T,, and class composed of k E {l...... ..... K} functions, the Gaussian complexity of a random set H(X) (Xx)) E H} C IR KTn is defined as follows:
G(H(X))
sup KEH
Σ
mastee (X(r)
Xti
where "ikki are independent standard normal variables. We also need to define the following quantity, taken from Maurer (2016): let y be a vector of m random standard normal variables, and f E F y R with Y C RR. define
f f(v')) 1y- -''l
O(F)
sup Y
E
sup
(2)
fEF
Equation 2 can be viewed as Gaussian average of Lipschitz quotients, and appears in the bounds provided in this work. Finally, we define L(F) as the upper bound of the Lipschitz constant all the functions ff in the function class F.
THEORETICAL ANALYSIS
The following theoretical study starts from the derivation of theoretical guarantees for MTRL in the AVI framework, extending the results of Farahmand (2011) in the MTRL scenario. Then, to bound the approximation error term in the AVI bound, extend the result described in Maurer (2006) to MTRL. As we discuss, the resulting bounds described in this section clearly show the benefit of sharing representation in MTRL. To the best of our knowledge. this the first general result for MTRL; previous works have focused on finite MDPs (Brunskill & Li, 2013) or linear models (Lazaric & Restelli, 2011).
3.1 MULTI-TASK REPRESENTATION LEARNING
The multi-task representation learning problem consists in learning simultaneously set of T tasks ur, modeled as probability measures over the space of the possible input-output pairs (x, y), with E X' and R. being A' the input space. Let w E W X + R h H R - R and of F k + R be functions chosen from their respective hypothesis classes. The functions in the hypothesis classes must be Lipschitz continuous functions. Let Z (Z..... Zr) be the multi-sample over the set tasks μ = (Hi...., Hr), where Z, = (Ze....., Z,n) ~ n,," and Zn (Xu, Yos) Pt- We can formalize our regression problem as the following minimization
3

---

Published as a conference paper at ICLR 2020
problem:
T N
min
ΣΣ
((f(hh(ur Y.) :w:ff :f Fh H. w E W t=
(3)
nT
where we use f= (JI....///)w (wi,.... wr), and define the minimizers of Equation (3) as w, h, and f. We assume that the loss function e R x R - [o. 1] is I-Lipschitz in the first argument for every value of the second argument. While this assumption may seem restrictive, the result obtained can be easily scaled to the general case. To use the principal result of this section, for generic loss E(-) <m where émax the maximum value of l'. The expected
function E', is possible loss over the tasks, given w, h and is the task-averaged risk:
T
Ewg w, h f) Σ ((PL(((((((())))
(4)
The minimum task-averaged risk, given the set of tasks μ and the hypothesis classes W, H and F is avg. and the corresponding minimizers are w", h² and f*.
3.2 MULTI-TASK APPROXIMATE VALUE ITERATION BOUND
We start by considering the bound for the AVI framework which applies for the single-task scenario. Theorem 1. (Theorem 3.4 of Farahmand (2011)) Let K be a positive integer, and Qmax S Rov Then
T-q
for any sequence (Qx) ) C. BB8 A, mmax) and the corresponding sequence (EN) k=o where
K-1
E 11 - -T*Qu
have;
l1Q - Q 1|,,p S
24
inf C rE(u,1]
Ci,p, (K;) (Eu...., .K-..; + 1 7 Rmax
(5)
(1 - 71-
where
K-1 Σ
CVLpu(KiT)
1 1-7
2(1-r)
Σ 7thh pu - Txk
sup
0k
2
k=0
m20
+0 1
(6)
2r Ek. the two coefficients Cuh.p.a. Calz,p.r the distributions with EEg....., EK-1;r) = Ek=o OK and v, and the series ag are defined as Farahmand (2011).
In the multi-task scenario, let the average approximation error across tasks be:
1 Eavg.k (wk, Per, fa) + EIQ.AAITT Qc.
where Qc.+1= = fi, o hi k and Tí is the optimal Bellman operator of task
In the following, we extend the AVI bound of Theorem I to the multi-task scenario, by computing the average loss across tasks and pushing inside the average using Jensen's 's inequality.
Theorem 2. Let K be positive integer, and Qmax 1-7
Bews
Then for any sequence (Qx) ( B(8x
1 A, Qmxx) and the corresponding sequence (Eavg, k= where Eavg.
K-1
T I S
have:
1
2n t 7)²
+ 2g K Rmux,
Σ
10 - - 11, 11, 5
inf [E(00.1]
T
1=1
(8)
with Eang = -K- ax Eargiko 7
7o C (k;(Kr))
max A (K;b)) Rmax,oog
max 16 (E(..... T)
1 (E(1....T)
1
(1 (1
1-
05k k K,
El=1 Rmax, and an
T
(1 (1
k= K
4

---

Published as a conference paper at ICLR 2020
Remarks Theorem 2 retains most of the properties of Theorem 3.4 of Farahmand (2011), I), except that the regression error in the bound is now aveeraeee Interestingly, the second term of the sum in Equation (8) depends on the average maximum reward for each task. In order to obtain this result we use an overly pessimistic bound on and the concentrability coefficients, however this approximation is not too loose if the MDPs are sufficiently similar.
3.3 MULTI-TASK APPROXIMATION ERROR BOUND
We bound the task-averaged approximation enror Eavg at each AVI iteration l: involved in (8) following a derivation similar to the one proposed by Maurer et al. (2016), obtaining:
Theorem 3. Let μ. W, and F be defined as above and assume E H. and f(0) 0, F. Then for 8 > O with probability at least in the draw of II 19, we have that
Eavg (,, f) LL(F) (1
L(H) supiep..... G(W(XI))
supw [tplll(()]]]000
t2
n
nf
minper G(H(P)))
suPp,w/l((w (lmk)ll((((())))))) (X)))
8 In )
+
+ca
+
Eave
(9)
nT
nvT
Remarks The assumptions 0 E H and f(0) 0 for all f E F are not essential for the proof and only needed to simplify the result. For reasonable function classes, the Gaussian complexity G(W(XI)) is (Vn)). If supw [X)]l and sup,.w (ypp...l//((v [//[(((()) can be uniformly bounded, then they are O(vn()) For some function classes, the Gaussian average of Lipschitz quotients O()) can be bounded independently from the number of samples. Given these assumptions, the first and the fourth term of the right hand side of Equation (9), which represent respectively the cost of learning the meta-state space w and the task-specific f mappings, are both (1/Va). The second term represents the cost of learning the mulli-task representation h and is (1/VET), thus vanishing in the multi-task limit T > co. The third term can be removed if vh. E H. 3po E P h(p) O; even when this assumption does not hold, this term can be ignored for many classes of interest, e.g. neural networks, as it can be arbitrarily small.
The last term to be bounded in (9) is the minimum average approximation error Savg at each AVI iteration k. Recalling that the task-averaged approximation error is defined as in (7), applying Theorem 5.3 by Farahmand (2011) we obtain:
Lemma 4. Ler Qi Vt f1, (I....,T) } be the minimizers of Emg. & arg max 1|Q T," Te and bR.; = 11Q
Ti T7"Qin, Qin.illu. then:
(10 (Ttt) Qie,ollu +
k-1
2
Eagg k
> (mr Cae(uita, P)))t)))ttt-iit
(10)
with CAE defined in Farahmand (2011).
Final remarks The bound for MTRL derived by composing the results in Theorems 2 and 3, and Lemma 4. The results above highlight the advantage of learning shared representation. The bound in Theorem 2 shows that a small approximation error critical to improve the convergence towards the optimal action-value function, and the bound in Theorem 3 shows that the cost of learning the shared representation at each AVI iteration is mitigated by using multiple tasks. This is particularly beneficial when the feature representation is complex, e.g. deep neural networks.
3.4 DISCUSSION
As stated in the remarks of Equation (9), the benefit of MTRL is evinced by the second component of the bound, i.e. the cost of learning h, which vanishes with the increase of the number of tasks. Obviously, adding more tasks require the shared representation
be large enough to include all of them, undesirably causing the term sup,, [][(([[ (X)) in the fourth component of the bound to h increase. This introduces tradeoff between the number of features and number of tasks; however, for
5

---

Published as a conference paper at ICLR 2020
Input
Output
0.50
FOI
0.15
0.50
0.40
MULTI
Performance
0.10
5 0.40
0.45
W
0.35
0.05
0.35
0.25
0.00
2 0.30
0.25
0.20
0.20
0.15
0.15
50
Iterations
Iterationn
Iterations
(a) Shared network
(b) FQI vs MFQI
(c) #Task analysis
Figure 1: (a) The architecture of the neural network we propose to learn T tasks simultaneously. The Wy block maps each imput 22 from task je to a shared set of layers which extracts common representation of the tasks. Eventually, the shared representation is specialized in block fo and the ouiput y/ of the network computed. Note that each block can be composed of arbitrarily many layers. (b) Results of FQI and MFQI averaged over 4 tasks in Car-On-HHil showing ||Q Q** || on the left, and the discounted cumulative reward on the right. (c) Results of MFQI showing ||Q - QT* for increasing number of tasks. Both results in (b) and (c) are averaged over 100 experiments, and show the 95% confidence intervals.
a reasonable number of tasks the number of features used in the single-task case enough to handle them, we show in some experiments in Section 5. Notably, since the AVI/API framework provided by Farahmand (201 1) provides an easy way to include the approximation error of a generic function approximator, it easy to show the benefit MTRL of the bound in Equation (9). Despite being just multi-task extensions of previous works, our results are the first one to theoretically show the benefit of sharing representation in MTRL. Moreover, they serve as a significant theoretical motivation, besides to the intuitive ones, of the practical algorithms that we describe in the following sections.
SHARING REPRESENTATIONS
We want empirically evaluate the benefit of our theoretical study in the problem of jointly learning T different tasks Pt, introducing a neural network architecture for which our bounds hold. Following our theoretical framework, the network we propose extracts representations wy, from inputs x; for each task Me, mapping them to common features in a set of shared layers h, specializing the learning of each task in respective separated layers ft. and finally computing the output y/+ = (fe o h wx)(x;; = fe(h(we(x:))) (Figure ((a)). The idea behind this architecture is not new in the literature. For instance, similar ideas have already been used in DQN variants to improve exploration on the same task via bootstrapping (Osband et al., 2016) and to perform MTRL (Liu et al., 2016).
The intuitive and desirable property of this architecture is the exploitation of the regularization effect introduced by the shared representation of the jointly learned tasks. Indeed, unlike learning single task that may end up in overfitting, forcing the model to compute a shared representation of the tasks helps the regression process to extract more general features, with a consequent reduction in the variance of the learned function. This intuitive justification for our approach, joins the theoretical benefit proven in Section 3. Note that our architecture can be used in any MTRL problem involving regression process; indeed, it can be easily used in value- based methods as Q-function regressor, or in policy search as a policy regressor. In both cases, the targets are learned for each task μ in its respective output block ft. Remarkably, as we show in the experimental Section 5, it is straightforward to extend RL algorithms to their multi-task variants only through the use of the proposed network architecture, without major changes to the algorithms themselves.
EXPERIMENTAL RESULTS
To empirically evince the effect described by our bounds, we propose an extension of FQI (Ernst et al., 2005; Riedmiller, 2005), that we call MFQI, for which our AVI bounds apply. Then, to empirically evaluate our approach in challenging RL problems, we introduce multi-task variants of two well-known DRL algorithms: DQN (Mnih et al., 2015) and DDPG (Lillicrap et al., 2015), which we call Multi Deep Q-Network (MDQN) and Multi Deep Deterministic Policy Gradient (MDDPG) respectively. Note that for these methodologies, our AVI and API bounds hold only with

---

Published as a conference paper at ICLR 2020
Acrobot
Cart-Pole
Acrobot
Mouniain-Car
Car-On-Hill
Inverted-Penoduum um
Perterancc
-70
-60
0.4
0.3
-0.2
0.0
-60
770
20
16
-90
-80
-95
-85
-80
-90
0.1
0.2
-0.6
DON MULTI
-90
-80
Urfreazed
Urieeze-33
0.0
-100
unirrcze
25 @Epochs
25 #Epochs
50
50
EEochss
Epochh
Epochs
Epochs
(a) Multi-task
(b) Transfer
Discounted cumulative reward averaged over 100 experiments of DQN and MDQN for each task and for transfer learning in the Acrobot problem. An epoch consists of 1, 000 steps, after which the greedy policy is evaluated for 2, 000 steps. The 95% confidence intervals are shown.
Figure 2:
the simplifying assumption that the samples are i.i.d:; nevertheless they are useful to show the benefit of our method also in complex scenarios, e.g. MuJoCo (Todorov et al., 2012). We remark that in these experiments we are only interested in showing the benefit of learning multiple tasks with a shared representation w.r.t. learning a single task; therefore, we only compare our methods with the single task counterparts, ignoring other works on MTRL in literature. Experiments have been developed using the MushroomRL library (D D'Ermmo et al., 2020), and run on an NVIDIA DGX Station TM and Intel AI DevCloud. Refer to Appendix B for all the details and our motivations about the experimental settings.
5.1 MULTI FITTED Q-ITERATION
As first empirical evaluation, we consider FQI, as an example of an AVI algorithm, to show the effect described by our theoretical AVI bounds in experiments. We consider the Car-On-liill problem as described in Ernst et al. (2005), and select four different tasks from it changing the mass of the car and the value of the actions (details in Appendix B). Then, we run separate instances of FQI with a single task network for each task respectively, and one of MFQI considering all the tasks simultaneously. Figure ((b) shows the -nom of the difference between Q* and QT averaged over all the tasks. It is clear how MFQI is able to get much closer to the optimal Q-function, thus giving an empirical evidence of the AVI bounds in Theorem 2. For completeness, we also show the advantage of MFQI w.r.t. FQI in performance. Then, in Figure I(c) we provide empirical evidence of the benefit of increasing the number of tasks in MFQI in terms of both quality and stability.
5.2 MULTI DEEP Q-NETWORK
As in Liu et al. (2016), our MDQN uses separate replay memories for each task and the batch used each training step is built picking the same number samples from each replay memory. Furthermore, step of the algorithm consists of exactly one step in each task. These are the only minor changes to the vanilla DQN algorithm we introduce, while all other aspects, such as the use of the target network, are not modified. Thus, the time complexity of MDQN is considerably lower than vanilla DQN thanks to the learning of T tasks with a single model, but at the cost of a higher memory complexity for the collection of samples for each task. We consider five problems with similar state spaces, sparse rewards and discrete actions: Cart-Pole, Acrobot, Mountain- Car, Car-On-Hill, and Inverted-Pendullun The implementation of the first three problems is the one provided by the OpenAl Gym library Brockman et al. (2016), while Car-On-Hii is described in Ernst et al. (2005) and
in Lagoudakis & Parr (2003).
Figure 2(a) shows the performance of MDQN w.r.t. to vanilla DQN that uses single-task network structured as the multi-task one the case with 7 1. The first three plots from the left show good performance of MDQN, which is both higher and more stable than DQN. In Car-On-Hill, MDQN is slightly slower than DQN to reach the best performance, but eventually manages to be more stable. experiment is clearly too easy to solve for both approaches, but it is
Finally, the still useful for the shared feature extraction in MDQN. The described results provide important hints about the better quality of the features extracted by MDQN w.r.t. DQN. To further demonstrate this, we evaluate the performance of DQN on Acrobot, arguably the hardest of the five problems, using single-task network with the shared parameters in h initialized with the weights of multi-task
7

---

Published as a conference paper at ICLR 2020
Inverted-Pendllum
Inverteo-Pend/lmm--wwingg
Wverree--DubllePenddlll
Performance
100
60
80
90
70
700
600
-20
-40
0
Performance
800
No initialization
40
50
DDPG
400
300
-60
Unfreeze-0
No unfreeze
30
MULTI
-80
20
100
-100
50
100
5D
100
50
100
0
50
160
WEpochs
Epoocs
Epochs
#Epochs
(a) Multi task for pendulums
(b) Transfer for pendulums
Hopper
Hopper
Walker
Half-Cheetah
25
35
50
60
70
25
35
30
Perfo Performance
30
No untreaze
No initialization
Unfreeze-
10
20
15
20
40
30
20
15
DDPG MULTI
20
10
0
50 #Epochs
100
50 #Epochs
100
50
100
#Epochs
#pochs
(c) Multi-task for walkers
(d) Transfer for walkers
Figure 3: Discounted cumulative reward averaged over 40 experiments of DDPG and MDDPG for each task and for transfer learning in the Seiird--DDube--eennnee and Hopper problems. An epoch consists of 10, 000 steps, after which the greedy policy is evaluated for 5, 000 steps. The 95% confidence intervals are shown.
network trained with MDQN on the other four problems. Arbitrarily, the pre-trained weights can be adjusted during the learning of the new task or can be kept fixed and only the remaining randomly initialized parameters in and f are trained. From Figure 2(b), the advantages of initializing the weights are clear. In particular, we compare the performance of DQN without initialization w.r.t. DQN with initialization in three settings: in Unfreeze-O - the initialized weights Unfreeze they are kept fixed, and in Unfreeze- they are kept fixed until epoch 10 after which they start to be optimized. Interestingly, keeping the shared weights fixed shows a significant performance improvement in the earliest epochs, but ceases to improve soon. On the other hand, the adjustment of weights from the earliest epochs shows improvements only compared the uninitialized network in the intermediate stages of learning. The best results are achieved by starting to adjust the shared weights after epoch 10, which is approximately the point at which the improvement given by the fixed initialization starts to lessen.
adjusted, in No-
5.3 MULTI DEEP DETERMINISTIC POLICY GRADIENT
In order to show how the flexibility of our approach easily allows to perform MTRL in policy search algorithms, propose MDDPG multi-task variant of DDPG. As actor-critic method, DDPG requires an actor network and a critic network. Intuitively, to obtain MDDPG both the actor and critic networks should be built following our proposed structure. We perform separate experiments two sets of MuJoCo Todorov et al. (2012) problems with similar continuous state and action spaces: the first set includes implemented in the pybullet library, whereas the second set includes Hopper-Stand, Walker Walk, and Half- Cheetah- Runn as implemented in the DeepMind Control SuiteTassa al. (2018). Figure 3(a) shows relevant improvement of MDDPG w.r.t. DDPG in the pendulum tasks. Indeed, while in DivertedPenduulun which is the easiest problem among the three, the performance of MDDPG is only slightly better than DDPG, the difference in the other two problems is significant. The advantage of MDDPG confirmed Figure 3(c) where it performs better than DDPG in Hopper and equally good in the other two tasks. Again, we perform a TL evaluation of DDPG in the problems where it suffers the most, by initializing the shared weights of a single-task network with the ones of a multi-task network trained with MDDPG the other problems. Figures 3(b) and 3(d) show evident advantages of pre- praiirain the shared weights and a significant difference between keeping them fixed or not.
Inverted- Inverted-Double- Couble-Pendullae and
as
8

---

Published as a conference paper at ICLR 2020
6 RELATED WORKS
Our work is inspired from both theoretical and empirical studies in MTL and MTRL literature. In particular, the theoretical analysis we provide follows previous results about the theoretical properties of multi-task algorithms, For instance, Cavallanti et al. (2010) and Maurer (2006) prove the theoretical advantages of MTL based on linear approximation. More in detail, Maurer (2006) derives bounds on MTL when a linear approximator is used to extract a shared representation among tasks. Then, Maurer et al. (2016), which we considered in this work, describes similar results that extend to the use of non-linear approximators, Similar studies have been conducted in the context of MTRL. Among the others, Lazaric Restelli (2011) 1) and Brunskill & 1.i (2013) give theoretical proofs of the advantage of learning from multiple MDPs and introduces new algorithms to empirically support their claims, as done in this work.
Generally, contributions in MTRL assume that properties of different tasks, e.g. dynamics and reward function, are generated from a common generative model. About this, interesting analyses consider Bayesian approaches; for instance Wilson et al. (2007) assumes that the tasks generated from hierarchical Bayesian model, and likewise Lazaric & Ghavamzadeh (2010) considers the case when the value functions are generated from common prior distribution. Similar considerations, which however does not use Bayesian approach, are implicitly made in Taylor et al. (2007), Lazaric et al. (2008), and also in this work.
In recent years, the advantages of MTRL have been empirically evinced also in DRL, especially exploiting the powerful representational capacity of deep neural networks. For instance, Parisotto et al. (2015) and Rusu et al. (2015) propose to derive a multi-task policy from the policies learned by DQN experts trained separately on different tasks. Rusu al. (2015) compares a therein introduced variant of DQN, which is very similar to our MDQN and the one in Liu et al. (2016), showing how their method overcomes it in the Atari benchmark Bellemare al. (2013). Further developments, extend the analysis to policy search (Yang et al., 2017: Teh et al., 2017), and to multi-goal RL (Schaul et al., 2015; Andrychowicz et al., 2017). Finally, Hessel et al. (2018) addresses the problem of balancing the learning of multiple tasks with single deep neural network proposing method that uniformly adapts the impact of each task on the training updates of the agent.
7 CONCLUSION
We have theoretically proved the advantage in RL of using a shared representation to learn multiple tasks w.rt. learning single task. We have derived our results extending the AVI/API bounds (Farah- mand, 2011) 1) to MTRL,, leveraging the upper bounds on the approximation error in MTL provided in Maurer et al. (2016). The results of this analysis show that the enror propagation during the AVI/API iterations is reduced according to the number of tasks. Then, we proposed a practical way of exploiting this theoretical benefit which consists an effective way of extracting shared representa- tions of multiple tasks by means of deep neural networks. To empirically show the advantages of our the introduction of multi-task, extensions of FQI, DQN, and DDPG based on the neural network structure we proposed. As desired, the favorable empirical results confirm the theoretical benefit we described.
method, we carried out experiments on challenging RL problems with

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. #640554 (SKILL44ROBOTS) and No. #7 310000 (GOAL- Robots). This project has also been supported by grants from NVIDIA, the NVIDIA DGX Station, and the Intel AI DevCloud. The authors thank Alberto Maria Metelli, Andrea Tirinzoni and Matteo Papini for their helpful insights during the development of the project.
REFERENCES
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances Neural Information Processing Systems, pp. 5048-5058, 2017.
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12: 149-198, 2000.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Richard Bellman. The theory dynamic programming. Technical report, RAND Corp Santa Monica CA, 1954.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Emma Brunskill and Lihong Li. Sample complexity of multi-task reinforcement learning. In Proceedings of the Twenty-Ninth Conference Uncertainty in Artificial Intelligence, 2013.
Rich Caruana. Multitask learning. Machine learning, 28(1):4177,, 1997.
Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Linear algorithms for online multitask classification. Journal of Machine Learning Research, (((cc)):901-299) 2934, 2010.
Carlo D' D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. arXiv:2001.01 102, 2020.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503-556, 2005.
Amir-massoud Farahmand. Regularization in reinforcement learning. 2011.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czameeki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. arXiv:1809.0449 2018.
Irina Higgins, Arka Pal, Andrei Rusu, Loie Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1480-1490, 2017.
Michail Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of machine learning research, 4(Dec): 107-1149, 2003.
Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey, In Reinforcement Learning, pp. 143-173. Springer, 2012.
Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian multi-task reinforcement learning. In ICML-27th International Conference on Machine Learning, pp. 599-606. Omnipress, 2010.
Alessandro Lazaric and Marcello Restelli. Transfer from multiple mdps. In Advances in Neural Information Processing Systems, pp. 1746-1754, 2011.
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch rein- forcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 544-551. ACM, 2008
10

---

Published as a conference paper at ICLR 2020
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wrXiv:1509002990 2015.
Lydia Liu, Urun Dogan, and Katja Hofmann. Decoding multitask dqn in the world of minecraft. In European Workshop on Reinforcement Learning, 2016.
Andreas Maurer. Bounds for linear multi-task learning. Journal of Machine Learning Research, 7 (Jan):117-139, 2006.
Andreas Maurer. A chain rule for the expected suprema of gaussian processes. Theoretical Computer Science, 650:109-122, 2016.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. The Journal of Machine Learning Research, 17(1):2853-285 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-40344 2016.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimie: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv: 1111111111.022 2015.
Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In European Conference Machine Learning, pp. 317-328. Springer, 2005.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv: 2015.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp. 1312-1320 1320, 2015.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/ 0t//1801.000 8006009 2018.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 103Jul):1633-165) 1635-1685, 2009.
Matthew E Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research, 8(Sep):2125-21) 2167, 2007.
Yee Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496-4506, 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- -base control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference Machine learning, pp. 1015-1022. ACM, 2007.
Markus Wulfimeier, Abbas Abdolmalcki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Regularized hierarchical policies for compositional transfer in robotics. arXiv: /228, 2019.
Zhaoyang Yang, Kathryn E Merrick, Hussein A Abbass, and Lianwen Jin. Multi-task deep reinforce- ment learning for continuous action control. In IJCAI, pp. 3301-3307, 2017.
11

---

Published as a conference paper at ICLR 2020
A PROOFS
A.1
APPROXIMATED VALUE-ITERATION BOUNDS
Proof of Theorem 2. We compute the average expected loss across tasks:
1
10
Q
1|,
T
Σ
2t
inf
(it + 1 7
2
K Rmuxt
T 1=1
(1
r2)2
[re
I
27
1
Σ
c (Et. + re(0.1]
inf
2
?t Rmax,
(1 7)2
T
1 1-7c
27
1
2
CEt + rE..11]
inf
7
R
(1 7-2
T
1 1-7
27
inf
(Kt (it.dd..tttttte.............)))) +
2
? Ramax
(1 - 7)2
r
1 1--
T
Σ (1 7)2
27
inf Ch (K:(K)))
Σ
3 (c.,0, e.....,, (e.-i.....x + 1 --y
2
7 Rmax, (11)
rE|o,11
T
with = max
Yo
C (K;r) = max
tt
V. (K; r), and Rmmax 1/T Et Rmux.
1e(1...,.7
Considering the term 1/r Ei=
r) ( 11/r Ei (ER= o2heas)) let
CaKK 0, (x.-1itrr)
(1-2)
0 0sk < K,
1-gk+
OKk
(1 T-
k=
then we bound
T
K
1
2r
Σ
2r
ak Ee,
T
1=1
k=
[=1
k=0
Using Jensen's 's inequality:
K
T
25
2r 1
ag Ei,k
a
T
k=
k=
1=1
So, now we can write (11) as
1 T
Σ l0
QTx
111. 5
2y
inf (K) ...gg.................. (.ag:r)) -a.ggKKK-... [r(00.1]
T
(1 7)2
K Rmax avg
1
with Eag.k = 1/1 E/= ft, and Gavg(Eug,.0.....
Eaw = =KK- a Eavgle
Proof of Lemma 4. Let us start from the definition of optimal task-averaged risk:
T
Σ
Qi T 1r.
Evvg,
T
12

---

Published as a conference paper at ICLR 2020
where Qi, with t [1. are the minimizers of Eavg.k-
Consider the task 8 such that
L arg maxx
|Q - -T"" Qr,
EE TE.....)
write the following inequality:
ivg. S 11
By the application of Theorem 5.3 by Farahmand (2011) to the right hand side, and defining bag Q4. - Vi. we obtain:
V Eagg I IIQ (TT) Qr..iinnllle Σ (n P) -
Squaring both sides yields the result:
Eug.t (1ein.tr (Tt)) Qi +
mi CA P)) bn bi.k-
A.2 APPROXIMATED POLICY-ITERATION BOUNDS
We start by considering the bound for the API framework:
Theorem 5. (Theorem 3.2 of Farahmand (2011)) Let K be positive integer, and Qmax I- Then for any sequence (Qk K-1
{=))==
B(S x A, Qmax) and the corresponding sequence (Ex)
K-1 where
E = 1Q T we have:
11-- - Q 111 5
27
inf (Er (5K.)))))) + K- Rmmx
(12)
(1 - 7)²
[e((..]]
where
(K;r)
2
K
?
sup
Σ
a
201
Σ m20
r m cph Chh k- 1, 1;4x+1)+
Th Crhh.p (K - k 1, Th 26) + Cris.p.v
m2
(13)
with Elso...., EK EK-1;T) kk=o ag Ek. the three coefficients CPI, p. Cplz.p.v- p.Is.p.v the distri- butions and u, and the series Ork are defined as in Farahmand (2011).
From Theorem 5, by computing the average loss across tasks and pushing inside the average using Jensen's 's inequality, we derive the API bounds averaged on multiple tasks.
Theorem 6. Let K be positive integer, and Qmax S Rase
Then for any sequence (Q+) B(Sx
T-q
1 K K-1 where Eoug,
A, Qmac) and the corresponding sequence (Eavg,
T Di=11QQ:4 Q7 b| we
have:
27 11,,p 5 (1 7)2
Q7*
inf Ca)(K Eang Kii7)
T
[EEo0,1]
1=1
t2 K Rmax,ave)
(14)
13

---

Published as a conference paper at ICLR 2020
with Eary = RK-1 2r Earg, 7
Yo. (p;(K:))= = max C (E(T...T)
(hi Rmax.arg
max
TE
(1 137 --h-1
El Rmar,t and an T
1
0 < K
11 7)
=
Proof of Theorem 6. The proof is very similar the for AVI. We compute the average expected loss across tasks:
T
Q2
2t*111,,
2
inf (E.. + +h K-1 "Rmax. rE]0..1]
T
(1
-²)2
1=1
2/ 1
inf C. rE(O.1]
Ph (aa. + t K Rmax,
(1-7)²
T
27
inf re(o.1]
+ 7 Rmaax
(1 7)-
27
inf rE(0,1]
1
+ 7g k Rmmx,
(1 71
T
(K: inf rE(0,1]
1 [(E
k-1 (#l(eno.....ux.sstt)))) + xx Rmax, avg
(15)
- 21-
T
Using Jensen's inequality in the AVI scenario, we can write (15) as:
1
102 Qi 111, 5 24
inf c+ (K:(K)) rE(0.1)
(1 - 7)2
+7 Rmux,avg]
(16)
with Savg, = 1/T Et St, and Eavg Gavg K-
2r Em ax Eng,
A.3 APPROXIMATION BOUNDS
Proof of Theorem 3. Let ....., wa. h² and fi,..., fi be the minimizers of eige then:
Eavg (w, Jn, f) <avg Envg ( (e,,,,hh,)) nT Σ (x)l Yoo)
1
(((h(( Y.) 1
nT Σ (1
n.
B
1 (oo Eang
(17)
n
We proceed to bound the three components individually:
can be bounded using Hoeffding's 's inequality, with probability 6/2 by (In(2/o)((221))) contains only n7 random variables bounded in the interval [0, 1]:
14

---

Published as a conference paper at ICLR 2020
B can be bounded by 0, by definition of w, h and f, as they are the minimizers of Equa- tion (3);
the bounding of A is less straightforward and is described in the following.
We define the following auxiliary function spaces:
W = fo E X + (wo(xes)) (W1...., WT) W
TyeRRTY (f((ve)): fi......ff). E Fr),
and the following auxiliary sets:
Fh C S E.
F(H)"(X)) (((((((((X)))))) NEW.EE S" = H(W((X) = ((h(w,(X.))) &.
w H. c CRTT
which will be useful in our proof.
Using Theorem 9 by Maurer al. (2016), we can write:
Eavg (w, P) nT Σ
((()) Ya)
sup wEWT NEH.FEFT
h,
((ll( Yu)
ni
t
2nG(S)
9
+
(18)
nT
2nT
then by Lipschitz property of the loss function é and the contraction lemma Corollary 11 Maurer et al. (2016): G(S) < G(S)). By Theorem 12 by Maurer et al. (2016), for universal constants &i and cz:
G(S') S (DL(((((F))))))))) + = ((S()))(())
min G(F(u)),
(19)
wEY
where I(F') is the largest value for the Lipschitz constants in the function space F', and D(S") is the Euclidean diameter of the set S"
Using Theorem 12 by Maurer et al. (2016) again, for universal constants o' and c=:
G(S") S ((L(())((W)))) (FWWE))))) + min G(H(p)).
(20)
pEP
Putting (19) and (20) together:
G(S') SGL(F) ( (CC0)444444(((((CCC/))))))))))00
min G(H(p)) pEp
+ + min min G(F(v)) yeY
=4(UF)44))CWW>G + (CCN:///CCWWYT))++C""""""""""""" (C + + GTFF)) min G(H(p))
PEP
+ +
min min G(F())) yEY
(21)
At this point, we have to bound the individual terms in the right hand side (21), following the same procedure proposed by Maurer et al. (2016).
15

---

Published as a conference paper at ICLR 2020
Firstly, to bound L(F",, let y, y.y' E RRTT, where y (yes) with yea E RR and y (1/1) with Ves E IR We can write the following:
ll((v) 1000-5((10 S(0)11 = E (fe(ve)) fe(s/t)))
S L(F)) Σ I1 - see 11?
1i
L(F)&1ly - -1/11?
(22)
whence T(F") & T(F).
Then, we bound:
G(W((X)) = E WEWT
sup
Σ ked
Theewes (Xti)
Xzi
S
Σ sup
E
sup
Xti
(E...T
wEW
T sup IE(
G(W(X.)).
(23)
Then, since it possible to bound the Fuclidean diameter using the norm of the supremum value in the set, we bound D(S") < SSuph./ll(((XX))) and (XW((X) < 2suuwewr [[ (XX)ll-
Also, we bound O(F)):
E
sup (v.)()) s(v0)) = LgeF
sup Σ SEFT t
Yea (fi(yn) Selobe))
Σ
7t -((ba)))
sup SEF
2 1
SVT
Σ
E
Σ Ys (f(2n) -((u)) FEF
sup
Σ Ilyes shall? 3
SVT Σ O(F)²
~TT(F))lV V/11.
(24)
whenee O(F)) S VTO(F).
To minimize the last term, it is possible choose Yo 0, as f(0) 0, E F. resulting in minger G(F(y)) = G(F(O)) = O.
Then, substituting in (21), and recalling that G(S) < G(S"):
G(S) SCL(F)LLL sup IE(1....r)
G(W(X.)) + L()
sup [w(X)](0(H) WEWT
+ L(F min G(H(p)) + 26h sup ([
(25)
PEP
h.w
16

---

Published as a conference paper at ICLR 2020
Now, the first term A of (17) can be bounded substituting (25) in (18):
Cavgl (w, f)
Σ
((() Yu)
nT
ti
2m
(F)(L())L
sup G(W(X.)) + ((F IE(I....TT
sup
|| (Xx (())(1100 (1(77)
nf
wEW!
+ 9ln(3)
+ c ((F) min G(H(P)) + supll(((()))llv]]) PEP
h.w
2nT
(()) SUPIE(L.....T) G(W(Xi))
supw w (X) ((F))((H
=1
+
n
nT
L(F) min G(H(p))
supp, (I., +
9 9ln(3)
+
+4
nT
nyT
2nT
A union bound between A, B and C of (17) completes the proof:
L(F) (UD)LE()) (I...T) G(W(XI))
Eavg (w, f)
Euvg Se
supw W [X] L(F)
+ 02
nf L(F) minpe G(H(p))
63
nT sup w
(FF
C1
nvT
8lm( 3
nT
B ADDITIONAL DETAILS OF EMPIRICAL EVALUATION
B.I MULTI FITTED Q-ITERATION
We consider Car- Car-On-Hill problem with discount factor 0.95 and horizon 100. Running Adam optimizer with learning rate 0.001 and using mean squared loss, we train neural network composed of 2 shared layers of 30 neurons each, with sigmoidal activation function, as described in Riedmiller (2005). We select tasks for the problem changing the mass of the m and the value of the discrete actions a (Table 1). Figure 1(b) is computed considering the first four tasks, while Figure 1(c) considers task 1 in the result with 1 task, tasks and 2 for the result with 2 tasks, tasks 1, 2, and for the result with 4 tasks, and all the tasks for the result with 8 tasks. To run FQI and MFQI, for each task we collect transitions running an extra-tree trained following the procedure and setting Ernst et al. (2005), using an e-greedy policy with = 0.1, to obtain a small, but representative dataset. The optimal Q-function for each task is computed by tree-search" for 100 states uniformly picked from the state space, and the 2 discrete actions, for a total of 200 state-action tuples.
B.2 MULTI DEEP Q-NETWORK
The five problems we consider for this experiment are: Cart-Pole, Acrobot, Mountain-Car Car-On- The discount factors are respectively 0.99, 0.99, 0.99, 0.95, and 0.95. Hill, and respectively 500, 1, 000. 1, 000, 100, and 3, 000. The network we use consists of 80 The horizons are
ReLu units for each w,, & {L. , 7} block, with T Then, the shared block h consists of one
'ee follow the method descrihed in Ernst et al. (2005).
*The DDs of the problems in the OpenAl Gym library are: CariPole-vO, Acrobot-vl, and MountainCar-vvl.
17

---

Published as a conference paper at ICLR 2020
Task
Mass
Action set
1
1.0
4.0; 4.03
2
0.8
4.0;4.03
3
1.0
4.5; 4.5)
4
1.2
4.5)
5
1.0
4. 4.15 125
6
1.0
4.25; 4.25)
7
0.8
4.375; 4.375)
8
0.85
4.0;4.03
Table 1: Different values of the mass of the car and available actions chosen for the Car-On-Hill tasks in the MFQI empirical evaluation.
layer with 80 ReLu units and another with 80 sigmoid units. Eventually, each f. has number of linear units equal to the number of discrete actions a af"), , i {l...., #A40)} of task f which outputs the action-value (s, a; (()) (s, d fe (s), aj ) s(t), The use of sigmoid units in the second layer of h is due to our choice to extract meaningful shared features bounded between O and be used input of the last linear layer, in most RL approaches. In practice, we have also found that sigmoid units help to reduce task interference in multi-task networks, where instead the linear response of ReLu units cause a problematic increase in the feature values. Furthermore, the use of a bounded feature space reduces the sup,,, // /h (X))]l term in the upper bound of Theorem 3, corresponding to the upper bound of the diameter of the feature space, shown Appendix A. The initial replay memory size for each task is 100 and the maximum size is 5, 000. We use Huber loss with Adam optimizer using learning rate 10-3 and batch size of 100 samples for each task. The target network is updated every 100 steps. The exploration is e-greedy with e linearly decaying from 1 to 0.01 in the first 5, 000 steps.
B.3 MULTI DEEP DETERMINISTIC POLICY GRADIENT
The two of problems we consider for this experiment are: one including Inverted-Penduee Soiidd-Dobee-eeeeee
and Inverted-Pendulum- Srnduuuu--Siingyy and another one including Hopper- Stand, Walker- Walk, and Saaff-Cheetale Rur' The discount factors are 0.99 and the horizons are 1, ..00 for all problems. The actor network is composed of 600 ReLu units for each Wi, E {1,...,77 T) block, with T 7= 3. The shared block has 500 units with ReLu activation function as for MDQN. Finally, each fo has a number of tanh units equal to the number of dimensions of the continuous actions (*) E (*) of task 14 which outputs the policy Te (s(s) (s) = ye (s) fe (h(ww Vs E st), On the other hand, the critic network consists of the same wy units of the actor, except for the use of
sigmoidal units in the h layer, as in MDQN. In addition to this, the actions (t are given as input to h. Finally, each f has single linear unit Qs (!)) y/ (s, (())
(s), ))), Vs E S The initial replay memory size for each task is 64 and the maximum size is 50, 000. We use Huber loss update the critic network and the policy gradient to update the actor network. In both cases the optimization is performed with Adam optimizer and batch size of 64 samples for each task. The learning rate of the actor is 10 10-4 and the learning rate of the critic is 10-3 Moreover, we apply l e--enalizzation to the critic network using regularization coefficient of 0.01. The target networks are updated with soft-updates using r = 10-3 The exploration is performed using the action computed by the actor network adding a noise generated with an Ornstein- Constein-Uhlebber Uhlenbeck process with 0 = 0.15 and a = 0.2. Note that most of these values are taken from the original DDPG paper Lillicrap et al. (2015), which optimizes them for the single-task scenario.
'The IDs of the problems in the pybullet library are: InvertdPrnddtunuuuteeeeee InvertedDouble Denuuubbeeenullumallleeeeee domain and the task of the problems in the DeepMind Control Suite are: hopper-stand, walker-walk, and cheetah-run.
and weterrddeentttiiSiinuuuuulleee
The names of the
18


Reviews:
[{"review_id": "Sye2k-pntH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission derives bounds for approximate value and policy iteration for the multitask case in reinforcement learning. In addition, two common RL algorithms are adapted to demonstrate benefits of multitask RL given related tasks.\n\nThe paper is mostly well written but sometimes introduces potentially unnecessarily complex mathematical notation including missing and dual definitions which slow down the reader. Examples of missing definitions are given by K an n on top of page 3. \n\nGenerelly, the paper is quite self consist thanks to minor changes in existing algorithms to show experimental results for the theoretical insights, but instead of requiring the reader to find other papers it would be better to define all terminology in the appendix (see equation 6).\n\nThe main contribution of the submission is the extension of an existing bound to the multitask case as the architectures are common across existing work. The extension is relevant given growing interest in meta and multitask RL but the changes in comparison to the single task case are minor and the experiment’s value purely lies in supporting this extension. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeFR4ZTKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a theoretical justification for the benefit of multi-task deep RL (MTRL) with shared representations. By extending prior work (Farahmand (2011) and Maurer et al. (2016)), the authors demonstrate that the bound of MTRL can be improved if the cost of learning the shared representation at each AVI iteration can be reduced, which is mitigated as we increase the number of tasks. The author also empirically verify their theoretical results in a tabular Q-Fitted Iteration domain and also in challenging RL domains such as Mujoco. The results show that MTRL with shared representation can outperform their single task counterparts to some degree.\n\nOverall this paper adapts the theory shown in Farahmand (2011) and Maurer et al. (2016) to the setting of MTRL and demonstrates the effectiveness of using shared layers, which seems intuitive. While the theory seems a bit incremental, it’s the first paper that theoretically validates the benefits of sharing knowledge, which is a contribution to the MTRL field. I would recommend a weak accept, though I have a few concerns on experimental results, and hope that the authors can clarify them during rebuttal.\n\nSpecifically, as the authors have noted, there is a wide range of prior works [1,2,3] that have empirically demonstrated the effectiveness of utilizing shared representations in MTRL. While the authors claim that the goal of the experiments is to show that MTRL with shared layers can outperform its sing task counterparts and thus they ignore other MTRL approaches. I believe that is not the main argument of the paper. The authors should provide empirical evidence on the claim that with an increasing number of tasks in MTRL, the error bound should improve and the performance of MTRL should also boost. Besides, I find the comparison where single-task training is initialized with shared representation a bit confusing. Training would definitely be improved when it’s initialized with some related pretrained features. Maybe the authors should compare this to some other methods such as initializing with single-task representation or even representation learned from training different tasks.\n\n[1] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki,S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart.arXiv preprintarXiv:1809.04474, 2018.\n[2] Teh, Y.W., Bapst, V., Czarnecki, W.M., Quan, J., Kirkpatrick, J., Hadsell, R.,Heess, N., Pascanu, R.: Distral: Robust multitask reinforcement learning. In: Ad-vances in Neural Information Processing Systems 30: Annual Conference on Neu-ral Information Processing Systems 2017 (2017)\n[3] Wulfmeier, M., Abdolmaleki, A., Hafner, R., Springenberg, J. T., Neunert, M., Hertweck, T., ... & Riedmiller, M. (2019). Regularized Hierarchical Policies for Compositional Transfer in Robotics. arXiv preprint arXiv:1906.11228.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gVz2MTtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to give theoretical support for using shared representations among multiple tasks.  The architecture has already been proposed in another paper.  The main contribution of the paper is the theory that it claims to support this architecture.  However, I am dubious that the architecture achieves the claimed bound.  One, I do not see how the analysis of this paper is connected to this specific architecture.  Two, the analysis follows from an existing paper by Farahmand (2011), which has already established a bound on the difference Q* - Q^{\\pi K}.  This paper considers the same difference, separately for each task, so that the same bound from Farahmand (2011) can be trivially used.  The shared layer h would affect the Q-values, but considering only the difference Q_t* - Q_t{\\pi K} abstracts away how sharing representations is helpful.  Simply averaging the norm of the difference between Q_t* - Q_t{\\pi K} as if they are independent ignores the fact that the Q_t’s are all dependent due to the shared layer h.  Also, I do not think that the proof of Theorem 2 and Theorem 6 can use the bound of Farahmand (2011).   Because the definition of approximation error epsilon_k in Farahmand (2011) is very different from epsilon_{avg,k} used in this paper, this warrants the analysis to start from the beginning, deriving the bounds from relating Q_t* - Q_t^{\\pi K} to (\\epsilon_{avg, k})_{k=0}^{K-1}.  The experiments compare known algorithms, and I am unsure how they support the theoretical bounds.   \n\nOther comments:\nInside the definition of (T*Q)(s,a), the probability measure P(s’|s,a) would be P^{(t)}.  Thus, T* is not one optimality operator shared amongst all tasks,  but one for each task.  The notation should reflect this.  Stating the descriptions of k, n explicitly in section 2 would be helpful.  I had to read to section 3 to be sure that k stands for a sequence of number 0 to big K and i stands for 1 to n with n being the number of samples.   Also, giving an intuitive explanation of gaussian complexity (1) and its use would also be helpful.    ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the benefits of deep multi-task RL with shared representations, by deriving bounds for multi-task approximate value and policy iteration bounds. This shows both theoretically and empirically that shared representations across multiple tasks can outperform single task performance.

There were a number of minor concerns from the reviewers regarding relation to prior work and details of the analysis, but these were clarified in the discussion. This paper adds important theoretical analysis to the literature, and so I recommend it is accepted.
Decision: Accept (Poster)