Unsupervised Task Clustering for Multi-Task Reinforcement Learning
Authors: Johannes Ackermann
Oliver Paul Richter
Roger Wattenhofer
Venue:  (2021)
Keywords: Reinforcement Learning
Multi-Task Learning
Clustering
Expectation-Maximization

Abstract: Meta-learning, transfer learning and multi-task learning have recently laid a path towards more generally applicable reinforcement learning agents that are not limited to a single task. However, most existing approaches implicitly assume a uniform similarity between tasks. We argue that this assumption is limiting in settings where the relationship between tasks is unknown a-priori. In his work, we propose a general approach to automatically cluster together similar tasks during training. Our method, inspired by the expectation-maximization algorithm, succeeds at finding clusters of related tasks and uses these to improve sample complexity. We achieve this by designing an agent with multiple policies. In the expectation step, we evaluate the performance of the policies on all tasks and assign each task to the best performing policy. In the maximization step, each policy trains by sampling tasks from its assigned set. This method is intuitive, simple to implement and orthogonal to other multi-task learning algorithms. We show the generality of our approach by evaluating on simple discrete and continuous control tasks, as well as complex bipedal walker tasks and Atari games. Results show improvements in sample complexity as well as a more general applicability when compared to other approaches.

Under review as a conference paper at ICLR 2021
UNSUPERVISED TASK CLUSTERING FOR MULTI-TASK REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning, transfer learning and multi-task learning have recently laid path towards more generally applicable reinforcement learning agents that are not lim- ited single task. However, most existing approaches implicitly assume a uniform similarity between tasks. We argue that this assumption is limiting in settings where the relationship between tasks is unknown a-priori. this work, we propose general approach to automatically cluster together similar tasks dur- algorithm,
ing training. Our method, inspired by the peneetationmaximizalee succeeds at finding clusters of related tasks and uses these to improve sample complexity. We achieve this by designing an agent with multiple policies. In the expectation step, evaluate the performance of the policies all tasks and as- sign each task to the best performing policy. In the maximization step, each policy trains by sampling tasks from its assigned set. This method is intuitive, simple implement and orthogonal to other multi-task learning algorithms. We show the generality of our approach by evaluating on simple discrete and continuous con- trol tasks, as well as complex bipedal walker tasks and Atari games. Results show improvements in sample complexity as well as a more general applicability when compared to other approaches.
INTRODUCTION
Imagine we are given an arbitrary set of tasks. We know that dis- similarities and/or contradicting objectives can exist. However, most settings we can only guess these relationships and how they might affect joint training. Many recent works rely on such hu- man guesses and (implicitly their approaches. This can lead impressive results, either by explicitly modeling the relationships between tasks as in transfer learning (Zhu et al., 2020), or by meta learning implicit relations (Hospedales et al., 2020). However, in some cases incorrect sim- ilarity assumption can hurt learning performance (Lazaric, 2012). Our aim with this paper is provide easy, straightforward ap- proach to avoid human assumptions on task similarities.
*
*
*
*
explicitly) limit the generality of
An agent (smiley) should reach one of 12 goals (stars) in a grid world. Learn- ing to reach goal in the top right corner helps him to learn about the other goals in that corner. However, learning reach the green stars (bot- tom left corner) at the same time gives conflicting objec- tives, hindering training. Task clustering resolves the issue.
Figure 1:
An obvious solution is to train separate policy for each task. How- ever, this leads to large amount of experience being required to learn the desired behaviors. Therefore, is desirable have sin- gle agent and allow the sharing of knowledge between tasks. This is generally known as multi-tasé learning, field which has received a large amount of interest in both the supervised learning and re- inforcement learning (RL) community (Zhang Yang, 2017). If tasks are sufficiently similar, policy that is trained on one task provides a good starting point for another task, and experience from each task will help training in the other tasks. This is known as pos- itive transfer (Lazaric, 2012). However, if the tasks are sufficiently dissimilar, negative transfer occurs and reusing pre- pre-trained policy disadvantageous. It can even lead to a worse performance than simply starting with a random initialization. Here using experience from the other tasks might slow
training even prevent

---

Under review as a conference paper at ICLR 2021
vergence to good policy. Most previous approaches to multi-task learning do not account for problems caused by negative transfer directly and either accept its occurrence or limit their experi- ments to sufficiently similar tasks. We present hybrid approach that is helpful in setting where the task set contains clusters of related tasks, amongst which transfer is helpful. To illustrate the intuition we provide conceptualized example in Figure 1. The figure shows grid world with 12 tasks that can be naturally clustered in 4 clusters. Note however that our approach goes beyond this conceptual ideal and can be beneficial even if the clustering is not perceivable by humans a-priori.
Our approach is inspired by the appectatio-mmaxiizatiii framework and uses a set of completely separate policies within our agent. We iteratively evaluate the set of policies on all tasks, assign tasks to policies based on their respective performance and train policies on their assigned tasks. This leads to policies naturally specializing to clusters of related tasks, yielding an interpretable decomposition of the full task set. Moreover, we show that our approach can improve the learning speed and final reward in multi-task RL settings. To summarize our contributions:
We propose a general approach inspired by Sspectation-Maximizaae (EM) that can find clusters of related tasks in an unsupervised manner during training.
We provide an evaluation on diverse set of multi-task RL problems that shows the im- proved sample complexity and reduction in negative transfer in our approach.
We show the importance of meaningful clustering and the sensitivity to the assumed num- ber of clusters in an ablation study
2 RELATED WORK
Expectation Maximizzation (EM) has previously been used in RL to directly learn policy. By reformulating RL as an inference problem with a latent variable, it is possible to use EM to find the maximum likelihood solution, corresponding to the optimal policy. We direct the reader to Deisenroth et al. (2013) for a survey on the topic. Our approach is different: We use an EM-inspired approach to cluster tasks in a multi-task setting and rely recent RL algorithms learn the tasks.
In supervised learning, the idea of subdividing tasks into related clusters was proposed by Thrun & O'Sullivan (1996). They use distance metric based on generalization accuracy cluster tasks. Another popular idea related to our approach that emerged from supervised learning is the use of a mixture of experts (Jacobs et al., 1991). Here, multiple sub-networks are trained together with an input dependent gating network. Jordan & Jacobs (1993) also proposed an EM algorithm to learn the mixture of experts. While those approaches have been extended to the control setting (Jacobs & Jordan, 1990; 1993; Meila & Jordan, 1995; Cacciatore & Nowlan, 1993; Tang & Hauser, 2019), they rely on an explicit supervision signal. It is not clear how such an approach would work in an RL setting, A variety of other methods have been proposed in the supervised learning literature, for brevity we direct the reader to the survey by Zhang & Yang (2017), which provides a good overview of the topic. Our work differs in that we focus on RL, where no labeled data set exists.
In RL, task clustering has in the past received attention in works on transfer learning. Carroll & Seppi (2005) proposed to cluster tasks based on a distance function. They propose distances based on Q-values, reward functions, optimal policies or transfer performance. They propose to use the clustering to guide transfer. Similarly, Mahmud et al. (2013) propose a method for clustering Markov Decision Processes (MDPs) for source task selection. They design a cost function for their chosen transfer method and derive an algorithm to find clustering that minimizes this cost function. Our approach differs from both in that we do not assume knowledge of the underlying MDPs and cor- responding optimal policies. Furthermore, the general nature of our approach allows it to scale to complex tasks, where comparing properties of the full underlying MDPs is not feasible. An earlier approach by Wilson et al. (2007) developed hierarchical Bayesian approach for multi-task RL. Their approach uses a Dirichlet process to cluster the distributions from which they sample full MDPs in the hope that the sampled MDP aligns with the task at hand. They then solve the sampled MDP and use the resulting policy to gather data from the environment and refine the posterior distri- butions for a next iteration. While their method is therefore limited to simple MDPs, our approach can be combined with function approximation and therefore has the potential to scale to MDPs with large or infinite state spaces which cannot be solved in closed form. Lazaric & Ghavamzadeh (2010) use hierarchical Bayesian approach to infer the parameters of a linear value function and utilize
2

---

Under review as a conference paper at ICLR 2021
EM to infer a policy. However, as this approach requires the value function to be linear function of some state representation. this approach is also difficult to scale to larger problems which we look at. Li et al. (2009) note that believe states in partially observable MDPs can be grouped according to the decision they require. Their model infers the parameters of the corresponding decision state MDP. Their approach scales quadratically with the number of decision states and least linearly with the number of collected transitions, making it as well difficult to apply it to complex tasks.
More recent related research on multi-task RL can be split into two categories: works that focus on very similar tasks with small differences in dynamics and reward, and works that focus on very dissimilar tasks, In the first setting. approaches have been proposed that condition the policy on task characteristics identified during execution. Lee et al. (2020) use model-based RL and learned emhedding over the local dynamics as additional input to their model. Yang et al. (2020) train two policies, one that behaves in way that allows the easy identification of the environment dynamics and another policy that uses an embedding over the transitions generated by the first as additional input. Zintgraf' et al. (2020) train embedding over the dynamics that accounts for uncertainty over the current task during execution and condition their policy on it. Our approach is more general than these methods as our assumption on task similarity is weaker. In the second group of papers, the set of tasks is more diverse. Most approaches here are searching for a way to reuse representations from one task in the others. Riemer al. (2018) present an approach to learn hierarchical options, and use it to train an agent on 21 Atari tasks. They use the common NatureDQN network (Mnih et al., 2015) with separate final layers for option selection policies, as well as separate output layers for each task to account for the different action spaces. Eramo et al. (2020) show how a shared representation can speed up training. They then use network strucuture with separate heads for each task, but shared hidden layers. Our multi-heat baseline is based on these works. Bräm et al. (2019) propose a method that addresses negative transfer between multiple tasks by learning an attention mechanism over multiple sub-networks, similar to a mixture of expert. However, as all tasks yield experience for one overarching network, their approach still suffers from interference between tasks. We limit this interference by completely separating policies. Wang et al. (2020) address the problem of open- ended learning in RL by iteratively generating new environments. Similar to us, they use policy rankings as measure of difference between tasks. However, they use this ranking novelty to find new tasks, addressing a very different problem. Hessel et al. (2019) present PopArt for multi-task deep RL. They address the issue that different tasks may have significantly different reward scales, Sharma et al. (2018) look into active learning for multi-task RL on Atari tasks. They show that uniformly sampling new tasks is suboptimal and propose different sampling techniques. Yu et al. (2020) propose Gradient Surgery, a way of projecting the gradients from different tasks to avoid interference. These last three approaches are orthogonal to our work and can be combined with EM-clustering. We see this as an interesting direction for future work.
measure of
Quality-Diversing (QD) algorithms (Pugh et al., 2016; Cully & Demiris, 2018) in genetic algorithms research aim to find diverse set of good solutions for a given problem. One proposed benefit of QD is that it can overcome local optima by using the solutions as "stepping stones" towards a global optimum. Relatedly in RL, Eysenbach et al. (2018) and Achiam et al. (2018) also first identify diverse skills and then use the learned skills to solve a given task. While we do not explicitly encourage diversity in our approach, our approach is related in that our training leads to multiple good performing, distinct policies trained on distinct tasks. This can lead to a policy trained on one task becoming the best a task that it was not trained on, similar to the "stepping stones" in QD. However, in our work this is more a side-effect than the proposed functionality.
3 BACKGROUND AND NOTATION
In RL (Sutton& & Barto, 2017) tasks are specified by Markov Decision Process (MDP), defined as tuple
P, R. 7). with state space S. action space A, transition function P(-|s, a). reward function R(s,a) and decay factor 7. As we interested in reusing policies for different tasks, we require a shared state-space S and action-space A across tasks. Note however that this requirement can be omitted by allowing for task specific layers. Following prior work, we do allow for task specific final layer in our Atari experiments to account for the different action spaces. In all other experiments however, tasks only differ in their transition function and reward function. We therefore describe a task as 7 = (P,, R,) and refer to the set of given tasks as T. For each task T E T we aim maximize the discounted return G, Di=b 7'r where rī Rr at) is the reward at
!=
3

---

Under review as a conference paper at ICLR 2021
time step t and L is the episode length. Given set of policies {TI.... Tn), we denote the return obtained by policy Te on task T as G. (T.s)
4 CLUSTERED MULTI-TASE LEARNING
As the growing body of literature on meta-, transfer- and multi-task learning suggests, we can expect gain through positive transfer if we train a single policy n; on a set of related tasks To C T. On the flip side, the policy x; might perform poorly on tasks T & Tr. Moreover, training pol- icy n, on a task T E Ts might even lead to a decrease in performance on the task set Te through negative trans- fer. We incorporate these insights into our algorithm by modeling the task set T as union of K disjoint task clus-
Algorithm 1: EM- Sal-Task-Clusterre
Initialize a policies { [T]
while not converged do
P E-Step T, f 0 for i {1....,
for r E T do
k +- arg max; (T;),)
Tx + UT
ters ..,...; Tk, i.e., T = U Tk with Ti TT 0 for i A j. Tasks within a cluster allow for positive transfer while we do not assume any relationship between tasks of different clusters. Tasks in different clusters may therefore even have conflicting objectives. Note that the assignment of tasks to clusters is not given to us and therefore needs to be inferred by the algorithm. Note also that this formu- lation only relies on minimalistic assumptions. That is, we do not assume a shared transition function or a shared re- ward structure. Neither do assume the underlying MDP to be finite and/or solvable in closed form. Our approach is therefore applicable to a much broader range of settings than many sophisticated models with stronger assumptions. As generality is one of our main objec- tives, we see the minimalistic nature of the model as strength rather than weakness.
T < T where T; = 0 b M-Step
for n. E [TI, Inl do t
while t < TM do
T T
Train on for an
episode of steps
1 <- L+L
Given this problem formulation we note that it reflects a clustering problem, in which we have to assign each task T E T to one of the clusters Tk, k E {I,..., Kj. At the same time, we want to train a set of policies {TI.... .... ] to solve the given tasks. Put differently, we wish to infer a hidden latent variable (cluster assignment of the tasks) while optimizing our model parameters (set of policies). An anpectation-Maximiaalie us do exactly that. On a high level, the expectation step (E-step) we assign each of the tasks E T to a policy x, representing cluster T. We then train the policies in the maximization step (M-step) on the tasks they got assigned, specializing the policies to their clusters. These steps are alternatingly repeated- - one benefiting from the improvement of the other in the preceding step until convergence. Given this general framework we are left with filling in the details. Specifically, how to assign tasks which policies (E-step) and how to allocate training time from policies to assigned tasks (M-step).
(EM) (Dempster et al., 1977) inspired algorithm allows
For the assignment in the E-step we want the resulting clusters represent clusters with positive transfer. Given that policy n, is trained on a set of tasks T, in a preceding M-step, we can base our assignment of tasks π. on the performance of ,;: Tasks on which n, performs well likely benefited from the preceding training and therefore should be assigned to the cluster of TÚ. Specifically, we can evaluate each policy ny E { ,,, J on all tasks the assignment on this performance evaluation. To get to an implementable algorithm we state two additional desiderata for our assignment: (1) We do not want to constrain cluster sizes in any way as clusters can be of unknown, non-uniform sizes. (2) We do not want to constrain the diversity of the tasks. This implies that the assignment has to be independent of the reward scales of the tasks, which in turn limits us to assignments based on the relative performances of the policies Ti,..., Rn. We found a greedy assignment - assigning each task to the policy that performs best - to work well. That is, task 7 is assigned to the policy π = arg max,, G., (T,). A soft assignment based on the full ranking of policies might be worth exploring in future work. Given the greedy assignment, our method can also be seen as related to k-means (MacQueen, 1967), special case of EM.
T to get an estimate of G (T²)) and base
In the M-step, we take advantage of the fact that clusters reflect positive transfer, i.c., training on some of the assigned tasks should improve performance the whole cluster. We can therefore
4

---

Under review as a conference paper at ICLR 2021
Mean Reward
EM
Right
Reward
Left
10000
Right
110
Reward
0.5
1.5
2
Left
Environment Steps
-10
5000
10000
15000
Environment Steps
Left: Mean reward and 95% confidence interval (shaded area) from 10 trials when training on the chain environment. Right: Task assignment (dots) and task specific reward (color) over the course of training the two policies in our approach. Each plot shows one of the policies/estimated clusters. The assignments converge to the natural clustering reflected by the goal location.
Figure 2:
randomly sample a task from the assigned tasks and train on for one episode before sampling the next task. Overall we train each policy for fixed number of updates T' in each M-step with Tas independent of the cluster size. This independence allows us save environment interactions as larger clusters benefit from positive transfer and do not need training time proportional to the number assigned tasks.
Note that the greedy assignment (and more generally any assignment fulfilling desiderata 1 above) comes with caveat: Some policies might not be assigned any tasks. this case we sample the tasks to train these policies from all tasks T, which can be task clusters. This also ensures that, early on in training, every policy gets a similar amount of initial experience. For reference, Note that we start by performing an E-Step, i.e., the first assignment to clusters is based on the performance of the randomly initialized policies. Note also that our approach is independent of the RL algorithm used to train the policies in the M-step and can therefore be combined with any state-of-the-art RL algorithm.
random exploration of possible
provide simplified pseudo code of our approach in Algorithm 1.
EXPERIMENTS
As a proof of concept start the evaluation of our approach on discrete tasks. The first environment consists of a chain of discrete states in which the agent can either move to the left or the right. The goal of the agent is placed either on the left end or the right end of the chain. This gives rise to two task clusters, where tasks within a cluster differ in the frequency with which the agent rewarded on its way to the goal. The second environment reflects the 2-dimensional grid- world presented in Figure 1. Actions correspond to the cardinal directions in which the agent can move and the 12 tasks the task set T are defined by their respective goal. We refer an interested reader to Appendix A. for detailed description of the environments.
We train policies with tabular Q-learning (Watkins, 1989) and compare our approach to two base- lines: In the first train a single policy all tasks. We refer to this as SP (Single Policy). In the other we train a separate policy per task and evaluate each policy on the task it was trained on. This is referred PPT (Policy per Task). Our approach is referred to EM (Expectation- (Especation-Maxiiizztonn
The learning curves as well as the task assignment over the course of training are shown in Figure and Figure 3. Looking at the assignments, we that in both environments our approach converges to the natural clustering, leading to higher reward after finding these assignments. Both our EM- approach and PPT converge to optimal reward in the chain environment, and a close optimal reward in the corner- semer--rid-world However, PPT requires significantly higher amount of environ- ment steps reach this performance, does not share information between tasks and therefore
'The implementation of all our experiments also available in the supplementary material
5

---

Under review as a conference paper at ICLR 2021
Upper Left
Mcan Reward
Coal
Lower Left
1
Reward
10
Upper Right
EM
Lower Right
5
Upper
Lower Left
10
Reward
Upper Right
0.5
1.5
2
Lower Right
Environment Steps
-10
20000
40000
25000
Environment Staps Environment Staps
Figure 3: Left: Mean reward and 95% confidence interval (shaded area) from 10 trials when training on the grid-world environment depicted in Figure Right: Task assignment (dots) and task specific reward (color) over the course of training for the 4 policies (estimated clusters) in our approach. The assignment naturally clusters the tasks of each corner together.
1.3
0
Mean Reward
-300
Length
1.1
Reward
0.9
1000
-400
0.7
Mulii Head
-500
PPT
1e6
-600
Ya et al. 2000
Length
Reward
1000
0
0.5
1.5
Environment Steps
-10
0
2
1
2
Environment Steps Environment Steps1e6
Figure 4: Left: Mean reward and 95% confidence interval (shaded area) from 10 trials when training on the pendulum environment. The curves are smoothed by a rolling average to dampen the noise of the random starting positions. For (Yu al., 2020) used 12 trials out of which 3 failed to converge and were excluded. Right: Task assignment (dots) and task specific reward (color) from sample run. Two policies focus on long and short, while the others focus medium lengths.
has to do exploration for each task separately. SP fails to achieve a high reward due the different tasks providing contradicting objectives.
5.1 PENDUL.UM
Next we consider simple continuous control environment where tasks differ in their dynamics. We use the pendulum gym task (Brockman al., 2016), in which a torque has be applied pendulum to keep it upright. Here the environment is the same in all tasks, except for the length of the pendulum, which is varied in the range {0.7, 0.8, 1.3), giving total of 7 tasks, Note that there is no clear cluster boundary here.
We use Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al., 2018) with hy- perparameters optimized as discussed in Appendix A.2. We use n = 4 policies in all experiments (except for the ablation studies) and did not tune this hyperparameter. This was done to give fair comparison to baseline approaches which do not have this extra degree of freedom. For application purposes the number of clusters can treated as hyperparameter and included in the hyperparam- eter optimization. We compare against SP, PPT, gradient surgery Yu et al. (2020) and a multi-head network structure similar to the approach used by Eramo et al. (2020). Each policy in our approach separate replay buffer. The multi- head network has separate replay- replay-buffer buffer and separate input and output layer per task. We adjust the network size the multi-head baseline and SP to avoid an advantage of method due to higher parameter count, see Appendix A.2 for details.

---

Under review as a conference paper at ICLR 2021
Mean Reward
200
EM
200
Multi-Head
-200
0
0.5
1.5
0.5
1.5
Environment Steps -10?
Environment Steps -10?
Figure 5: Evaluation of the Bipedal Walker experiments. The shaded areas show the 95% confidence interval on the mean task reward. Left: Track and field task set; 6 tasks with varying objectives. Results reflect 20 trials of each approach. Right: Task set with varying leg lengths and obstacles; tasks with the same reward function. Results reflect 10 trials of each approach.
The results are shown in Figure In the figure we excluded from the 12 runs of the gradient surgery baseline as those did not converge to performance better than random. Still, the results show a worse sample complexity and higher variance compared to our approach. We also again observe that our approach clusters similar tasks together, leading to a better performance than with SP agent, and a faster convergence than with PPT. Also the multi-head approach needs more experi- ence to converge than approach in this setup, even more than the PPT approach. We believe this due to the inherent interference of learning signals in the shared layers. The cluster assignment in approach is also intuitive, with two clusters focusing the extremes (cf. Figure 4).
5.2 BIPEDAL WALKER
As a more complex continuous control environment we focus on Bipedal from the OpenAI Gym (Brockman et al., 2016), which has previously been used in multi-task and generalization literature (Portelas et al., 2019; Wang et al., 2019: 2020). It consists of bipedal robot in a two- dimensional world, where the default task is to move to the right with high velocity. The action space consists of continuous torques for the hip and knee joints of the legs and the state space consists of joint angles and velocities, as well as hull angle and velocity and 10 lidar distance measurements.
To test our approach, we designed 6 tasks inspired by track and field sports: Jumping up at the starting position, jumping forward far as possible, a short, medium and long run and hurdle run. As a second experiment, we create a set of tasks by varying the leg length of the robot as well as the number of obstacles in its way. This task set is inspired by task sets previous work (Portelas et al.. 2019). Note that we keep the objective - move forward as fast as possible - constant here. We again use TD3 and tune the hyperparameters of the multi-heat baseline and our approach (with fixed) with grid-search. Experiment details and hyperparameters are given in Appendix A.3.
The results in Figure (left) on the track and field tasks show a significant advantage in using our approach over multi-head TD3 or SP and slightly better initial performance than PPT, with similar final performance. SP fails to learn a successful policy altogether due to the conflicting reward functions. In contrast, the results in Figure 5 (right) from the second task set show that SP can learn policy that is close to optimal on all tasks here. The multi-heal and PPT approaches suffer in this setup as each head/policy only gets the experience from its task and therefore needs more time to converge. Our approach can take advantage of the similarity the tasks. We note that the experiments presented here reflect distinct cases: One in which it is advantageous separate learning, reflected by PPT outperforming SP, and one where is better to share experience between tasks, reflected by SP outperforming PPT. Our approach demonstrates general applicability as is the only one performing competitively in both. We provide an insight into the assignment of tasks to policies in Appendix B.
5.3 ATARI
To test the performance of our approach on a more diverse set of tasks, we evaluate on a subset the Arcade Learning Environment (ALE) tasks (Machado al., 2018). Our choice of tasks is similar to
7

---

Under review as a conference paper at ICLR 2021
Mcan Human Normalized Score
0.6
MsPamman
Spacelnvaders
ChopperC ChapperCommant
EM n 4
Multi-Head
0.0
PPT
Assaut
Jamesbond
0.4
BantHolst
DemonAdtack
Zarooo
0.2
0.25
WizendOWWor
Phoenix
Riverraid
0.5
2
0.5
1.5
2
2.5
0.0
Environment Steps
-10
1=?
1w7
1s7
Figure 6: The results of our experiments on subset of the Atari Learning Environment games. The reward is averaged across 3 trials and the shaded region shows the standard deviation of the mean.
those used by Riemer et (2018), but we exclude tasks containing significant partial-obeervabiiityy This is done to reduce the computational burden those tasks usually require significantly more training data. We built our approach on top of the Implicit Quantile Network (IQN) implementation in the Dopamine framework (Dabney et al., 2018; Castro et al., 2018). We chose IQN due to its sample efficiency and the availability of an easily modifiable implementation. As the different ALE separate final layer and separate replay buffer
games have different discrete action spaces, for each game in all approaches. We use the hyperparameters recommended by Castro et al. (2018), except for smaller replay buffer size reduce memory requirements. As in the Bipedal Walker experiments we fix the number of policies in our approach without tuning to n = 4. We choose the size of the network such that each approach has the same number of total tunable parameters. We provide the details in Appendix A.4.
The results are given in Figure The good performance of PPT shows that the diversity of this task set is best addressed with policy per task. Note that we did not expect our approach to outperform PPT here as our approach can only outperform PPT if there are similar tasks that can be clustered together. The diverse set of Atari games seems to violate this assumption. We note also that the multi-head approach unable to learn any useful policy here due to negative transfer between tasks. This is in line with experiments in other research (Hessel et al., 2019). Our approach manages to overcome most of this negative interference, even with just clusters. Task assignments our approach are given in Appendix B.2.
5.4 ABLATIONS
To gain additional insight into our approach, we perform two ablation studies on the discrete corne-- grid-world environment and the pendulum environment.
First, we investigate the performance of our approach for different numbers of policies n. The results in Figure 7 show that using too few policies can lead to a worse performance, as the clusters cannot distinguish the contradicting objectives. On the other hand, using more policies than necessary increases the number of environment interactions required to achieve a good performance in the pendulum task, but does not significantly affect the final performance.
As second ablation, we are interested in the effectiveness of the clustering. It might be possible that simply having fewer tasks per policy giving our approach an advantage compared to SP or multi-head TD3. We therefore provide ablation in which task-policy assignments are determined randomly at the start and kept constant during the training. Results from this experiment can be seen in Figure with additional results in Appendix C. The results show that using random clusters performs significantly worse than using the learned clusters. This highlights the importance of clustering tasks meaningfully.
8

---

Under review as a conference paper at ICLR 2021
-200
Mean Reward
10
300
5
400
0
0.5
1.5
0
0.5
1.5
Environment Steps -108
Environment Steps -10%
Figure 7: Ablations for different number policies n. Shaded areas show the 95% confidence interval of the mean reward from 10 trials each. Left: Comer-grid-wond tasks. Right: Pendulum tasks, learning curves smoothed.
-250
Mcan Reward
n.
Random n
300
n
Random n
-350
5
-400
0.5
1.5
2
0.5
1.5
Environment Steps
-10
Environment Steps -10%
Figure 8: Comparison of our approach against randomly assigning tasks policies at the start of training. Shaded areas show the 95% confidence interval of the mean reward. Left: Corner-grid- world tasks, 10 trials each. Right: Pendulum tasks, 10 trials each, learning curves smoothed.
6 CONCLUSION
We present approach for multi-task learning in reinforcement learning (RL) that automatically clusters tasks into related subsets. Our approach uses set of policies and alternatingly evaluates the policies all tasks, assigning each task to the best policy and then trains policies their assigned tasks. Since our approach can be combined with any underlying RL method, we evaluate it varied set of environments. We show its performance on sets of simple discrete tasks, simple continuous control tasks, two complex continuous control task sets and a set of Arcade Learning Environment tasks. We show that approach is able to identify clusters of related tasks and use this structure to achieve a competitive or superior performance. We further provide an ablation the number of policies our approach, showing that many policies lead slower convergence while too few policies can hurt performance. In another ablation we also highlight the need cluster tasks meaningfully.
Our approach offers many possibilities for future extensions. One interesting direction would be hierarchical clustering. This could prove helpful for complicated tasks like the Atari games. It would also be interesting to see how our approach can be applied to multi-task learning in a supervised setting. Further, different assignment strategies with soft assignments could be investigated. Overall, we see our work as good stepping stone for future work on structured multi-task learning.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational Option Discovery Algorithms. 2018. URL httttt/s///aae /arviv../aiivvv/aa org/ abs/ 1807 10299.
Timo Bräm, Gino Brunner, Oliver Richter, and Roger Wattenhofer. Attentive Multi-task Deep Re- inforcement Learning. In ECML PKDD, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016. URL httt:://arxisn www.///g//////oo//////////eeeee/ / 606. 01 01540.

---

Under review as a conference paper at ICLR 2021
Timothy W. Cacciatore and Steven J. Nowlan. Mixtures of controllers for jump linear and non-lineat plants. In Jack D. Cowan, Gerald Tesauro, and Joshua Alspector (eds.). Advances in Neural Information Processing Systems 6, [7th NIPS Conference, Denver, Colorado, USA, J, pp. 719-726. Morgan Kaufmann, 1993. URL. http: otw.rttee-ooooooooooooooooo
httpp/aappaaaaaaaaa/aae ni cc paper/
for- fonrsss-oruuumm----- -atp--jjmm--mea--aaaaeae and- -nn -iae--l-aaaaranns.....
James L. Carroll and Kevin Seppi. Task similarity measures for transfer in reinforcement learning task libraries. In IJCNN, 2005.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http: / //arxiiv.orp aagg//bb///e hal/a/as/111/ 1111//111110 06110.
Antoine Cully and Yiannis Demiris. Quality and Diversity Optimization: A Unifying Modular Framework. IEFE Trans. Evol. Comput., 22:245-255 2018.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In ICML, 2018.
Mare Peter Deisenroth, Gerhard Neumann, and Jan Peter. A Survey on Policy Search for Robotics. Found. Trends Robot., 2(1-2):1-142) 2013.
A. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via em algorithm. Journal of the Royal Statistical Society, Series B (Methodological), 39(1):1-38, 1977.
Carlo D Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Politecnico Milano, and Jan Peters. Sharing Knowledge in Multi-Task Deep Reinforcement Learning. ICLR, 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In ICLR, 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In ICML, 2018.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-Task Deep Reinforcement Learning with PopArt. 2019.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. 2020. URL pettps://arxiv> .rgg abs aaa.abb0000 200...000..000 05439.
A Jacobs, M Jordan, SE Nowlan, and G Hinton, Adaptive mixture of experts. Neural Comput., 3:79-87, 1991.
Robert Jacobs and Michael Jordan. A competitive modular connectionist architecture, Advances in neural information processing systems, 3:767-773, 1990.
Robert A. Jacobs and Michael I. Jordan. Learning piecewise control strategics in modular neural network architecture. IEEE Trans. Syst. Man Cybern., 23(2):337-355, 1993. doi: 10.1 109/21. 229447. URL https: /ooioooooooooooooooooooo. /tgg//...000000
229447.
Michael Jordan and Robert Jacobs. Hierarchical mixtures experts and the EM algorithm. In IJCNN, 1993. ISBN 0780314212 doi: 10.
Alessandro Lazarie. Transfer in Reinforcement Learning: Framework and Survey. M. Wiering and M. van Otterlo (eds.), Reinforcement Learning State of the art 12, pp. 143-173. Springer, 2012.
Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian Multi- Task Reinforcement Learning. In ICML, 2010.
Kimin Lee, Younggyo Seo, Scunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware Dy- namics Model for Generalization in Model-Based Reinforcement Leaming. In ICML., 2020.
10

---

Under review as a conference paper at ICLR 2021
Hui Li, Xuejun Liao, and Lawrence Carin. Multi-task reinforcement learning in partially observable stochastic environments. Mach. Learn. Res., 10, 2009. ISSN 15324435.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
J. MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Vol- ume 1: Statistics, pp. 281-297, Berkeley, Calif., 1967. University of California Press. URL ht ht
M M Hassan Mahmud, Majd Hawasly, Benjamin Rosman, and Subramanian Ramamoorthy, Clus- tering Markov Decision Processes For Continual Transfer. 2013. URL htttaattttxxi...ooot
abs/ 11111
Marina Meila and Michael I. Jordan. Learning fine motion by markov mixtures of ex- perts. In David S. Touretzky, Michael Mozer, and Michael E. Hasselmo (eds.), Advances in Neural Information Processing Systems 8, NIPS, Denver, CO, USA, November 27-30, httpsscc//ppperr 1995, pp. 1003-1009. MIT Press, 1995. URL www.miiittttteteeeeeeeeeee otw.://e............o....oo..oo. of-eeerrs... 1063-1earmnn mote www.oo--bbbaaarooooiittteeeeeeee
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning- Nature, 2015.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of Deep RI. in continuously parameterized environments. In CoRI., 2019.
Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolu- tionary computation. Front. Robot. AI, 3, 2016.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In NeurlPS, pp. 10424- 10434, 2018.
Sahil Sharma, Ashutosh Kumar Jha, Parikshit S. Hegde, and Balaraman Ravindran. Learning to multi-task by active sampling. ICLR 2018 Conference Track, 2018.
Richard Sutton and Andrew G Barto. Reinforcement learning: an introduction. MIT Press, 2017.
Gao Tang and Kris Hauser. Discontinuity -eneriinuuty-sssstvvveeie sensitive optimal control learning by mixture of experts. In International Conference Robotics and Automation, ICRA 2019, Montreal, QC. Canada, May 20-24, 2019, pp. 7892-7898. IEEE, 2019. doi: 10.1 00.100//CA..2019...7999 //doi. orgii.rrg/n 10. 1109/ ICRA. 2019 2019.8793909
URL https:
Sebastian Thrun and Joseph O' O'Sullivan. Discovering Structure in Multiple Learning Tasks The TC Algorithm. In ICML, 1996.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions. 2019. URL http: //axiiv..iivvv//a org/ abs /11//.911.1000 01753.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, and Kenneth O. Stanley. En- hanced POET: Open- Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions. In ICML, 2020.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's College, Cambridge, 1989.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi reinforcement learning: A hierarchical Bayesian approach. In /CML. 2007. doi: 10.1 14/11273996.1273222

---

Under review as a conference paper at ICLR 2021
Jiachen Yang. Brenden Petersen, Hongyuan Zha, and Daniel Faissol. Single Episode Policy Transfer in Reinforcement Learning. In ICLR, 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient Surgery for Multi-Task Learning. 2020. URL http://aax....rr//a//////// 06782.
Yu Zhang and Qiang Yang. A Survey on Multi-Task Learning. 2017. URL https://arxiv. org/abs/ 1707. 08114.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. 2020. URL http:////aarxi http://awwwv..oraaa////aaaaa wwtp://gggg////ooe/eoee 00w.//0.99..88....
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta- Learning. In ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A EXPERIMENT DETAILS
In addition to the details provided here, the implementation of all experiments can be found in the supplementary material.
A.1 GRID WORLD EXPERIMENTS
In the first discrete task set starts in the middle and receives a reward for moving toward either the left or right end. As a reward use Fag - Igonal where Fag is the position of the agent and agdal is the goal position (either the left or right end the chain). We give a reward of = 20 if the goal position reached. Depending on the task, the reward is given every 2, 4, 8 or 16 steps, or only at the goal position, and otherwise replaced by r = 0.
one-dimensional state-chain with 51 states, in which the agent
For corner grid- world task set per corner (as depicted in Figure 1). The agent always starts in the center and receives a reward based on the distance to the target
2D -o--grrdwwoll with edge length 7 and three goal positions
|| =uguullla" with ll2 being the Euclidean norm. A reward of r 10 is given when the agent reaches the goal position.
In both tasks we use tabular Q-Learning with e-greedy exploration. We start with = 0.2 and decay the value as 6 = c with % = 1-111 10-6, We use a learning rate a = 0.2 to update the value estimates, from the perspective of single agent the environment can be regarded as stochastic. Further, we use a discount factor of y = 0.9 and TM = 500 training steps per policy in each M-step and evaluate each policy on each task for three episodes during the E-step, using the greedy policy without exploration.
A.2 PENDULUM
In our pendulum tasks we use a modified version of the Pendulum environment provided in OpenAl gym (Brockman et al., 2016). This environment consists of a single pendulum and the goal is to balance it in an upright position. The observation consists of the current angle 0, measured from the upright position, and current angular velocity represented as (sin f, cos 0, 0). The reward for each time step rt = - (08 + 0.1 + 0.001a²), with a being the torque used as action. Every episode starts with random position and velocity. To provide set of tasks we vary the length of the pendulum in {0.7, 0.., 1.3].
A.2.1 HYPERPARAMETERS
Hyperparameters for our EM-TD3 and multi-head TD3 were tuned on the pendulum task set by grid
1x 10 10-3), batch-size b= (64, 128) and update- rate u {1, 3,5). specifying the number of collected time-steps after which the value-function is updated. We increased the network size for multi-head TD3, so that it overall had more parameters than EM-TD3. This is done to eliminate a potential advantage of our approach stemming from a higher representational capacity. The tuned hyperparameters are given in Table 1. To represent the value functions and policies we use fully connected multi-layer perceptrons (MLPs) with two hidden layers with 64 units each. As activations we use ReLU on all intermediate layers, and tanh activations on the output. The values are then scaled to the torque limits per dimension. In EM, SP and PPT we use a separate network for each policy. For our multi-head baseline we share the hidden layers between tasks, but use separate input and output layers per task. Additionally, we increase the size of the first hidden layer to 96 in the multi-head approach, such that has a similar total number of parameters our EM approach. For SP and PPT we reuse the hyper-parameter parameters from EM approach. During the M-step, we train the agent for 5 x 10a steps per policy and during the E-step evaluate each agent each task by running 20 episodes without added exploration noise.
search over learning rate {1 10 10-2, 3,3, x 10-3
13

---

Under review as a conference paper at ICLR 2021
Table I: Hyperparamters for pendulum experiments.
Hyperparameter
EM-TD3
Multi-head TD3
learning-rate o batch-size b update-rate u siddyy--pdatefrreueeeee n FM
3 10-3
3 x 10-3
128
128
1
3
3
4
network size
4 (64, 64, 1)
(9 19-96, 64, 9 1)
exploration noise
0.05
0.05
exploration noise clipping
0-.,5,044
[-0.5, 0.5
target policy smoothing noise a buffer-size
0.1
0.1
2e6 per policy
2e6 per task 0.99
decay
0.99
TM
5 10*
A.3 BIPEDAL WALKER
For the BipedalWalker tasks we look at two different sets of tasks. The first set of tasks consists of different reward functions with mostly similar environments, inspired by track and field events. The tasks are jumping up. jumping a long distance, runs for different distances and run with obstacles. In all tasks a reward of - [|]|| given minimize the used energy. The position the hull of the bipedal walker is denoted as (2r,y) y)- In the jump up task reward of y [xr] is given upon landing, and 3.5 10-1, For the long jump task a reward of x 2o is given upon landing, with To being the hull position during the last ground contact,
3.5 x 10 The three runs consist of sprint over length of 67 units, with e = 3.5 x 10-4 a run over 100 units, with = 3.5 x 10 and a long 6.5 10-4 The hurdles task is identical to the long run, but every
over 200 units with units there is an obstacle with a height of 1. Additionally, reward of 0.1æ reward proportional to the velocity of the agent in the x-direction is given during the run and hurdle tasks, reward movement to the right.
The second of tasks consists of varying obstacles and robot parameters. We vary the length of the legs in {25, 35, 45) and either use no obstacles, or obstacles with a spacing of 2 or units apart and height of 1. This results in total of tasks. Here we use the standard reward for the BipedalWalker task r = 4.3r - 5| - ||al] with 0 being the angle of the walker head. Additionally, in all experiments r 100 is given if the robot falls over or moves far the left.
A.3.1 HYPERPARAMETERS
Hyperparameters for our EM-TD3 and multi-head TD3 approaches were tuned on the track and field
task set by grid search over {1 x 10-3,3 3,3x 10-4, 1 10 10-4), batch-size b {100, 1000) and update-rate u = (1, 3, 5}. u specifying the number of collected time-steps after which the value- function is updated. We reuse the optimal parameters found here the task set with varying leg lengths and obstacles. For the SP and PPT baselines we reused the parameters from EM-TD3. We increased the network size for multi-head TD3, so that it overall had more parameters than EM-TD3. All hyperparameters are given in Table 2. During the M-step, we train the EM agent with 2 x 10a steps per policy and during the E-step we evaluate each agent on each task by running 20 episodes without added exploration noise.
A.4
ATARI
To test our approach on more complex task, we evaluate it on a subset of the Atari games. The set of chosen games consists of Alien, Assault, BankHeist, ChopperCommand, DemonAttack, James- Bond, MsPacman, Phoenix, RiverRaid, Spacelnvaders, WizardOfWor and Zaxxon. As stated above, this task set is similar to the set of games used in Riemer al. (2018), but without tasks requiring a large amount of exploration save computation time.
14

---

Under review as a conference paper at ICLR 2021
Table 2: Hyperparameters for BipedalWalker experiments.
Hyperparameter
EM-TD3
Multi-head TD3
learning-rate
1 x 10-3
x 10
batch-size
1000
1000
update-rate
3
freee--epdate--fequeeeyyy
3
n FM
4
network size
4 (400,
(6- 400,000400 6. 1)
exploration noise
0.1
0.1
exploration noise clipping
[-0.5, 0.5]
[-0.5,
target policy smoothing noise buffer-size
0.2
0.2
5e6 per policy
5e6 per task
decay y TM
0.99
0.99
2 x 10
Our implementation is based on the IQN implementation in the Dopamine framework (Dabney et al., 2018; Castro et al., 2018). As hyperparameters we use the default values recommended by Dopamine for Atari games, except the changes listed below: Due to the different action spaces, use a separate replay buffer for each game, as well as a separate output layer, both for our EM, multi- head and PPT approaches. We reduce the size of the replay buffer to 3 x 10t compared to 1 10t in the original paper, to reduce the memory demand. We use the normal NatureDQN network, but scale the size of the layers ensure that each approach has similar number of parameters. For our EM approach, we use TM = 2.5 x 10a trainings steps per M-step, and evaluate all policies on all tasks for 27000 steps the E-step, using the greedy policy without random exploration. In both EM and the multi-head approach, we record how many transitions were performed in each M- Step and sample the task with the least transitions as next training task. This is done to ensure a similar amount of transitions and training steps per game, episode lengths vary. This approach was proposed in Riemer al. (2018).
B ADDITIONAL RESULTS
B. BIPEDAL WALKER
In Figure 9 the assignments for 4 randomly chosen trials the track and field task shown. We can see that in all trials the rums over different distances are grouped together with the long jump task. This is likely due to these tasks aligning well, they both favor movements to the right. is possible to learn the hurdles task with the same policy as the runs, due to the available LIDAR inputs. The hurdle task therefore sometimes switches between policies, but usually learned by separate policy. The jump up task is very different from the other tasks, as it is the only one not to involve movement to the right, and is therefore assigned to separate policy.
In Figure 10 the assignments for randomly chosen trials on the leg-length and obstacle task set are shown. illustrated by the good performance of the SP approach shown in Figure 5, it is possible to learn nearly optimal behavior with single policy here. This makes learning meaningful clustering significantly harder and sometimes leads to a single policy becoming close to optimal on all tasks, in Trial 2. In most other trials the task set is separated into two three different clusters based on the different leg lengths.
B.2 ATARI
In Figure the assignments all three trials of our approach on the Atari task set are shown. While we see consistency in assignments, we cannot identify clearly repeated clustering across trial. We assume this is due to the high diversity of tasks preventing the identification of clearly distinguishable clusters. This lack of clearly distinguishable clusters might also be the reason for failing to reach the performance of PPT. Yet, the specialization of policies in our approach helps to avoid negative transfer seen in Figure 6.
15

---

Under review as a conference paper at ICLR 2021
Trial
Hurdles
400
400
Marafron
200
200
Sprint
Jump Long
Jump Up
Hurdles
Run
200
0
-0
Jump Long
Up
200
200
0.00
0.25
0.50
0.75
1,00
1.26
1.60
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Environment Sleps
1c7
Environmment Steps
1e7
Trial 2
400
400
Marsthon
Spritt
Jump
Up
200
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
167
167
Hurdles
400
400
Manshon
200
200
Sprind
0
-0
Jump Long
Jump Up
200
Ervironment Staps
Environment Steps
Trial 3
Hurdles
200
Run
Sprint
Jump Long
200
0.00
0.25
0.50
0.75
1,00
1.55
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1 1.50
167
Tw7
Marathon
200
200
Sprint
0
-0
Jump Up
200
200
0.75
1,00
0.00
0.25
1.00
1.25
1.50
Environmenn Steps
Te/
Trial 4
Hurdles
400
Marathon
Run
200
200
Sprint
0
-0
Jump Long
Jump Up
Hurdles
Manehon
Run
Sprint
-0
Jump Long
200
200
0.00
0.25
0.50
0.75
1.00
1.95
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Environment Steps
167
Environment Stops
1w7
Figure 9: Shown are the assignments from 4 randomly picked trials on the track and field Bipedal- Walker task
16

---

Under review as a conference paper at ICLR 2021
Trial
200
200
J- 35.60
0
200
200
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
125
1,50
1w7
Tw?
200
200
0.00
0.25
0.50
0,75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
125
1.50
Environment
167
Environment
147
Trial 2
-0
200
200
025
0.50
1.00
200
200
Environment Steps
Enwironment Staps
Trial 3
-0
0.55
0.75
1.25
0.75
1.00
125
1w7
200
200
200
0.00
0.5
0.50
0.75
1,00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
125
1,50
Environment Stops
107
Environment Stops
147
Trial 4
200
200
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
167
167
-0
200
200
0.30
0.25
0.50
0.75
1.00
1.25
1.50
0.50
0.75
1.00
122
1.50
167
1w7
Figure 10: Shown the assignments from 4 randomly picked trials the first Bipedal task set. refers to the lenghts of the legs, o refers to the frequency obstacles.
17

---

Under review as a conference paper at ICLR 2021
Trial 1
Riyeraid
1.00
Riyaraad
1.00
Chopper
Propperoqpaant
Pricenx
0.75
Chopper ghopperrggmmary
0.75
DemonAttan AABBK
0.50
Demon Attac
0.50
Spacelnvoerr Wizard Inn
Wig C
Bank Allen
0.25
Bian plee
0.25
MsPacman
MsPadll MaPacman
3
2
3
1e7
1e7
Riyeraid
1.00
Riveraid
1.00
Choppe Car
lamas
aaxoon
0.75
Choppe CempeenCCaan
Jameshond
0.75
Demon
Phgenkx
Phoen
Spaceln aarrd Wol
Assaut
0.50
Assaul
0.50
Weg oo
Alien MsPacmali
MsPacman
0.00
0.00
2
1e7
1e7
Trial 2
Choppei
Jamester
Coniooommann primand
Zakion
0.75
ChoppelConni commara
Jamestond
000000
0.75
Phoent
DemonAtlao
Assau
0.50
Stace trvadee
Atac
Asspu
0.50
Wizardoiny Wor
0.25
Coworr
0.25
MsPacman
All Bi
0.00
Alien MsPacman
0.00
3
2
1e7
Riyeraid
1.00
Riyemaid
1.00
Chopper gnmaa
James bond
0.75
ChapperCaguman
Jamesbond
Spacel eeelevaa
Demon Allack
u
0.50
Space In maverrs
Demon
Thenn Ix
0.50
Bank plst
0.25
Weard Kunkhels Rhelo
Banki
0.25
MsPacmain
0.00
MsPacman
0.00
2
1e7
1e7
Trial 3
Riverrai
1.00
Riyeraid
1.00
Choppei
James bond
0.75
Choppe Seperroammant
Jamesbond
0000.
0.75
Demon Phoeng lx
Spacelrve
Assaul
0.50
Space Asspul
0.50
Vizar Pardomiia
W lard
Bank Pass
R
MsPacmia
0.00
MsPacman
0.00
2
1e7
1e7
1.00
1.00
Chopper
onmmand
Zaxkon
0.75
Chopperrogmaant
Jamesbong
Laxoon
0.75
DemonAllao Alack
Demon Attack
Space lnvdeer
Space Invadters
Bankhelss
0.25
Ban Alien
4000 000000 600
0.25
Afee
MsPaicman
0.00
MsP aammann
0.00
3
2
3
1e7
1e7
Figure 11: Shown are the assignments of all three trial that were run on the set of Atari games. The color represents the human-normalized score per game.
18

---

Under review as a conference paper at ICLR 2021
GEM - -Grand
GEM Grand
4
Mmm
0.5
1.5
2
5
6
Environment Steps
-10%
Show is the difference between using random assignments our EM approach for different numbers policies. On the left the development during training is shown, on the right the average performance gap over the last 10% of the training is visualised.
Figure 12:
C PERFORMANCE GAP TO RANDOM CLUSTERS
In Figure 8 we investigated the importance of the E-Step in our approach, by comparing to an ablation which randomly assigns tasks to policies at the start. These results showed that using random assignments performs worse, highlighting the importance Here will investigate how the difference between the return when using EM method GEM or random assignments Grand changes depending on the number of tasks. When using a single policy policy for each task method becomes identical the baselines. We hypothesize that the difference should be maximal when using as many policies as there are true underlying clusters in the task set.
using related clusters of tasks.
To test this hypothesis we perform experiments on our grid-world task set with 12 goals distributed the four corners and show the return gap GEM Grand in Figure 12. The experiments confirm our hypothesis, showing that the return gap increases with the number of policies before reaching a maximum when it matches the true clusters 4. Afterwards it starts decrease.
19


Reviews:
[{"review_id": "Z8s4307WVpl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-task RL algorithm that leverages unsupervised task clustering. The authors propose to initialize a number of policies, cluster each task based on its performance on different policies, and train each policy with data coming from tasks within the cluster. The paper shows that such kind of an EM style clustering can lead to better performance than single-task training and be more sample efficient more training each task independently on both tabular settings, continuous control experiments, and Atari. \n\nFor pros, I think the method is relatively simple yet effective based on the empirical evaluations, which is a plus. Moreover, the empirical results seem to suggest that clustering can indeed recover the natural clustering or help performance when natural clustering is not obvious to humans, which could make the method valuable. The paper is also well-written and easy to understand.\n\nHowever, I do have a few concerns about this paper, which I will list as follows:\n\n1. It seems that there are several old MTRL papers [1, 2] that also consider task clustering using EM-style approaches, but are not discussed and compared in this paper. \n\n2. The paper seems more like combining clustering methods that are used in multi-task supervised learning and RL and such approaches are also explored in prior works as mentioned in 1, which makes the contribution a bit derivative. Moreover, the idea of learning separate policies seems a bit contradictory to the goal of the MTRL, which is learning a single policy that can tackle all tasks. The solution proposed in the paper, which is to learn multiple policies, kind of defeats the purpose of MTRL. Also, since the algorithm for complex tasks is TD3, is the value function also learned per-cluster? It would much more appealing if the authors can somehow distill the multiple policies into a single policy that can tackle all tasks, as in Distral [3].\n\n3. For the experiments, do you use the same network for SP and PPT as in EM? In that case, doesn't EM have more parameters than SP?  I think it would be fairer if the total number of parameters is the same for all three methods. Moreover, it would be important to show the performance on benchmarks with tasks that are more distinct such as Meta-World, which could justify the strength of the clustering approach in settings where natural clustering is near impossible.\n\n4. Finally, given the relatively less novel method, it would be interesting to see if there's any theoretical insight into this method, but it is currently missing.\n\nOverall, given the pros and cons listed above, I would vote for a weak reject of this paper.\n\n[1] Lazaric, Alessandro, and Mohammad Ghavamzadeh. \"Bayesian multi-task reinforcement learning.\" 2010.\n[2] Li, Hui, Xuejun Liao, and Lawrence Carin. \"Multi-task Reinforcement Learning in Partially Observable Stochastic Environments.\" Journal of Machine Learning Research 10.5 (2009).\n[3] Teh, Yee, et al. \"Distral: Robust multitask reinforcement learning.\" Advances in Neural Information Processing Systems. 2017.\n[4] Yu, Tianhe, et al. \"Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.\" Conference on Robot Learning. 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VnU9epS5X5Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\n\nThe authors present a clustering-based approach to multi-task learning, specifically task-policy assignment, which iteratively learns a set of different policies for each of the subtasks. The aim is to solve issues with contradicting objectives of different tasks.\n\nThe clustering is achieved in an unsupervised way inspired by the EM algorithm, by iteratively evaluating the set of policies on all tasks, assigning tasks to policies based on their current task performance, and then training policies on their assigned tasks.\n\nThe proposed approach is evaluated on several continuous control and Atari multi-task RL problems, and compared to single policy and per-task policy baselines, as well as with a multi-head baseline inspired by an approach from the recent literature.\nIn order to show the importance of the clustering component of the approach, an ablation study is conducted.\n\n\n### Review:\n\nThe paper proposes a simple and novel approach to task selection and policy learning in the multi-task learning problem setting. It provides proof of concept evaluations, as well as evaluations on some more complex environments where the task clustering is not obvious. I appreciate the discussion of the results and how the hypothesis relates to the observed outcomes.\n\nBelow, there are several suggestions that I believe would help improve the quality of the manuscript, and address some of the issues that should be clarified.\n\n1) Regarding multi-task learning literature, it would be useful to mention the relationship with Quality-Diversity approaches: \n - Cully, Antoine, and Yiannis Demiris. \"Quality and diversity optimization: A unifying modular framework.\" IEEE Transactions on Evolutionary Computation (2017)\n - Pugh, Justin K., Lisa B. Soros, and Kenneth O. Stanley. \"Quality diversity: A new frontier for evolutionary computation.\" Frontiers in Robotics and AI (2016)\n\n As well as Deep RL approaches with skill/task conditioning:\n - Eysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" International Conference on Learning representations (2019).\n - Achiam, Joshua, et al. \"Variational option discovery algorithms.\" arXiv preprint arXiv:1807.10299 (2018).\n\n It would greatly improve the strength of the manuscript to have comparisons with these state-of-the-art approaches as well, although I understand that this would require a significant effort from the authors.\n\n2) Could you provide an example of these assumptions: “Note also that this formulation only relies on minimalistic assumptions. It is therefore applicable to a much broader range of settings than many sophisticated models with stronger assumptions.”\n\n3) In the E step, the task $\\tau_i$ is assigned to a policy $\\pi_i$, which is used within the whole cluster $T_i$. There are N policies and K clusters, so it is not guaranteed that there is going to be one policy per cluster (i.e. $N=K$)? This part is not completely clear so it could be maybe explained.\n\n Related to this, how is the initialisation of the policies and clusters done, i.e. how are the number of initial policies and clusters selected? I guess the policies are manually set and this would be one of the hyperparameters that has to be tuned and is task-specific.\nFor the proof of concept tasks the number of policies can be set as the number of clusters that we can expect. Is this what is done in Bipedal tasks or is this an arbitrary number? For Atari experiments it is explicitly mentioned that n=4 and that the same is done for Bipedal tasks. This is a very important detail of the proposed approach and should be explained explicitly.\n\n Moreover, as I understand, at the beginning each task is attempted by each of the policies or they are assigned randomly. This explanation is missing.\n\n4) It would be useful to provide some discussion about the stability of this approach, as in the beginning the policies could have tasks from different clusters assigned to them, but also the clusters change. I would assume that there is some burn-in period necessary.\n\n5) The results on Atari for EM are not outperforming other baselines. Could you please comment on this? Could this be related to the number of policies set? It would be informative to evaluate the sensitivity of the EM algorithm to different numbers of initial policies.\n\n The conducted ablation study gives useful insights into the mechanisms of the approach. For the Corner-grid-world it makes sense that increasing n > 4 does not improve the performance, as there are 4 salient clusters (you could also mention this in the discussion). However, it is difficult to see a pattern in the effect of the number of policies for the Pendulum task. Therefore, investigating this in more detail, and in other environments as well, would greatly help understanding the approach better. \n\n One useful experiment would be to see how the performance gap between using randomly assigned clusters and learned clusters changes with respect to the initial number of policies.\n\n6) Another useful metric would be evaluating the number of generated clusters and analysing how they relate to qualitatively different skills/tasks/behaviours.\n\n7) Also, a comparison of computation time and required memory would be nice to show, with respect to SP and PPT for example, because additional policies need to be trained/stored. I assume, memory-wise and training-time-wise, EM is between SP and PPT.\n\nOTHER COMMENTS:\n- There are some links embedded in the pdf which are not visible and always link to the title page.\n- What do you mean by “relationship of tasks of different clusters is **unconstrained**”, in which sense unconstrained ?\n- How is the mean reward calculated for PPT in Figure 2? I assume it is done via evaluating each policy on the corresponding task, but this should be emphasised.\n- Fig 5. right shows that SP converges faster than EM but then starts degrading, what could be an explanation for this?\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MnSxHIBa9hz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to approach multi-task RL problems in which tasks may differ considerably in transition functions/dynamics and reward functions as well as in the action space through task clustering. Specifically, tasks are modeled as belonging to separate task clusters defined by the return obtainable by individual policies i. All policies are evaluated on a single task and the relative cumulative discounted rewards over some iterations determines the assignment of tasks to policies. Simulations show the advantage in terms of number of training iterations on several tasks compared to a selection to other related algorithms. \n\nMTRL is an interesting and relevant field for ICLR in which research has been quite active for years.\n\nThe authors’ algorithm is loosely inspired by EM in that the unknown task assignments of individual learners are thought of as a latent task assignment/ clustering variable. As the assignments to clusters are not soft/probabilistic, this seems almost more inspired by k-means than by EM. \n\nThe algorithm is a straightforward extension of previous approaches. Clustering of tasks is also an idea, that has been pursued in the field, just one example is Wilson, Fern, Ray, Tadepalli (2007). There also seems to be a close connection to the mixture of experts (Jacobs, Jordan, 1991) in which the responsibility values are used to select individual learners. This framework has also been used in control settings. The authors may want to consider relating their algorithm to this literature. \n\nThe authors do not comment on the selection of the number n of policies, that needs to be selected before learning, similarly to the number of clusters in mixture models for clustering. It would be desirable to at least comment on how to select this parameter.\n\nThe authors may wish to comment on the complexity of their algorithm. For the case in which a moderate number cluster with highly related tasks in each cluster the proposed algorithm may work well, if the number n of policies is selected accordingly.\n\nThe simulations show empirically, that for tasks in which the similarity between tasks in a cluster is high, i.e. the changes in policy required between tasks in the cluster are small, learning is faster. This is to be expected by design of the considered learning problems. The authors also provide learning results in which the advantage is not given, as in the ALE tasks.\n\nOverall, this is a quite informal presentation of an algorithm that is closely related to several previously published algorithms. The empirical evaluations are promising for specific multitask settings. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EtBaM9IV1X", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a new multi-task reinforcement learning (RL) algorithm. Since In general, the relationships between tasks is unknown a-priori, directly applying classical multi-task learning approaches that assume all tasks are related, could suffer from negative transfer. The authors propose to cluster tasks into disjoint groups: The proposed algorithm iterates through steps of assigning tasks to specific policies and training each policy only based on the respective assigned tasks (clusters). In the experiments, the authors compare their algorithm with two single-task learning baselines (SP: a single policy for all tasks and PPT: a policy per task) and a recent multi-task RL algorithm of Eramo et al. 2020 on Pendulum, Bipedal Walker, and Atari problems.\n\nWhile multi-task learning was extensively studied in supervised learning, its application to RL has only recently begun to gain attention. This paper contributes by a new multi-task RL algorithm that addresses the negative transfer issue arising when one tries to apply multi-task learning strategies across partially unrelated tasks. However, I am not sure if this paper is ready for ICLR in the current form, for the reasons given below\n\n- Limited novelty: Negative transfer in multi-task learning has been previously studied, mainly in supervised learning but recently, also in RL. Specifically, task clustering approaches have been previously examined although as far as I am aware, they were used only in supervised learning. \nEven though the proposed EM strategy is not a trivial application of existing approaches, the technical novelty seems to be limited. Enhancing the practical relevance of the proposed algorithm via an extensive empirical study (involving multiple state-of-the-art approaches and challenging real-world problems) could help. However --->\n- Limited experiments: Two simple baselines are not multi-task learning algorithms, and Eramo et al.’s approach was not designed to address the negative transfer problem. I think the experiments can be improved by including comparisons with more, state-of-the-art algorithms including these discussed in the last two paragraphs of Section 2. Yu et al. 2020 Sharma et al. 2018 Hesse et al. are designed to address the negative transfer issue, thus they should provide better baselines than SP, PPT, and Eramo.\n- ATARI experiments (Figure 6) seem to suggest that when the individual tasks are not related, (hence multi-task learning is not beneficial), applying the proposed algorithm can degrade the performance from the PPT, still suffering from the negative transfer. Their results are much better than Eramo et al. 2020, but the latter method was not designed for negative transfer cases. \n\nSection 4 describes well the operations performed by the proposed EM algorithm. Still, explicit equations on the E and M steps would help reading.\n\nThank the authors for their responses. I read through the responses from the authors and comments from the other reviewers. I would maintain my initial rating: I think this paper will benefit significantly from a major revision, either strengthening the theoretical contributions or improving empirical validations (by actually performing extensive comparisons with existing algorithms that are designed to handle negative transfer problems).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers multi-task RL from the perspective of an unsupervised clustering of different tasks with an EM-like algorithm. The idea is evaluated on several simple and ATARI domains.
We thank the reviewers for their detailed responses and revision. This work still seems a little preliminary in its current form. While the empirical results seem promising, it is generally felt that it would benefit from more extensive experiments, including further comparisons to other approaches and exploring the effects of the hyperparameters on tasks with much larger numbers of clusters. It would also be beneficial to provide some theoretical results, particularly with respect to negative transfer.
Decision: Reject