Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose to approach multi-task RL problems in which tasks may differ considerably in transition functions/dynamics and reward functions as well as in the action space through task clustering. Specifically, tasks are modeled as belonging to separate task clusters defined by the return obtainable by individual policies i. All policies are evaluated on a single task and the relative cumulative discounted rewards over some iterations determines the assignment of tasks to policies. Simulations show the advantage in terms of number of training iterations on several tasks compared to a selection to other related algorithms. 

MTRL is an interesting and relevant field for ICLR in which research has been quite active for years.

The authors’ algorithm is loosely inspired by EM in that the unknown task assignments of individual learners are thought of as a latent task assignment/ clustering variable. As the assignments to clusters are not soft/probabilistic, this seems almost more inspired by k-means than by EM. 

The algorithm is a straightforward extension of previous approaches. Clustering of tasks is also an idea, that has been pursued in the field, just one example is Wilson, Fern, Ray, Tadepalli (2007). There also seems to be a close connection to the mixture of experts (Jacobs, Jordan, 1991) in which the responsibility values are used to select individual learners. This framework has also been used in control settings. The authors may want to consider relating their algorithm to this literature. 

The authors do not comment on the selection of the number n of policies, that needs to be selected before learning, similarly to the number of clusters in mixture models for clustering. It would be desirable to at least comment on how to select this parameter.

The authors may wish to comment on the complexity of their algorithm. For the case in which a moderate number cluster with highly related tasks in each cluster the proposed algorithm may work well, if the number n of policies is selected accordingly.

The simulations show empirically, that for tasks in which the similarity between tasks in a cluster is high, i.e. the changes in policy required between tasks in the cluster are small, learning is faster. This is to be expected by design of the considered learning problems. The authors also provide learning results in which the advantage is not given, as in the ALE tasks.

Overall, this is a quite informal presentation of an algorithm that is closely related to several previously published algorithms. The empirical evaluations are promising for specific multitask settings. 

review_title: Informal algorithm closely related to previously published approaches

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a multi-task RL algorithm that leverages unsupervised task clustering. The authors propose to initialize a number of policies, cluster each task based on its performance on different policies, and train each policy with data coming from tasks within the cluster. The paper shows that such kind of an EM style clustering can lead to better performance than single-task training and be more sample efficient more training each task independently on both tabular settings, continuous control experiments, and Atari. 

For pros, I think the method is relatively simple yet effective based on the empirical evaluations, which is a plus. Moreover, the empirical results seem to suggest that clustering can indeed recover the natural clustering or help performance when natural clustering is not obvious to humans, which could make the method valuable. The paper is also well-written and easy to understand.

However, I do have a few concerns about this paper, which I will list as follows:

1. It seems that there are several old MTRL papers [1, 2] that also consider task clustering using EM-style approaches, but are not discussed and compared in this paper. 

2. The paper seems more like combining clustering methods that are used in multi-task supervised learning and RL and such approaches are also explored in prior works as mentioned in 1, which makes the contribution a bit derivative. Moreover, the idea of learning separate policies seems a bit contradictory to the goal of the MTRL, which is learning a single policy that can tackle all tasks. The solution proposed in the paper, which is to learn multiple policies, kind of defeats the purpose of MTRL. Also, since the algorithm for complex tasks is TD3, is the value function also learned per-cluster? It would much more appealing if the authors can somehow distill the multiple policies into a single policy that can tackle all tasks, as in Distral [3].

3. For the experiments, do you use the same network for SP and PPT as in EM? In that case, doesn't EM have more parameters than SP?  I think it would be fairer if the total number of parameters is the same for all three methods. Moreover, it would be important to show the performance on benchmarks with tasks that are more distinct such as Meta-World, which could justify the strength of the clustering approach in settings where natural clustering is near impossible.

4. Finally, given the relatively less novel method, it would be interesting to see if there's any theoretical insight into this method, but it is currently missing.

Overall, given the pros and cons listed above, I would vote for a weak reject of this paper.

[1] Lazaric, Alessandro, and Mohammad Ghavamzadeh. "Bayesian multi-task reinforcement learning." 2010.
[2] Li, Hui, Xuejun Liao, and Lawrence Carin. "Multi-task Reinforcement Learning in Partially Observable Stochastic Environments." Journal of Machine Learning Research 10.5 (2009).
[3] Teh, Yee, et al. "Distral: Robust multitask reinforcement learning." Advances in Neural Information Processing Systems. 2017.
[4] Yu, Tianhe, et al. "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning." Conference on Robot Learning. 2020.

review_title: Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors present a new multi-task reinforcement learning (RL) algorithm. Since In general, the relationships between tasks is unknown a-priori, directly applying classical multi-task learning approaches that assume all tasks are related, could suffer from negative transfer. The authors propose to cluster tasks into disjoint groups: The proposed algorithm iterates through steps of assigning tasks to specific policies and training each policy only based on the respective assigned tasks (clusters). In the experiments, the authors compare their algorithm with two single-task learning baselines (SP: a single policy for all tasks and PPT: a policy per task) and a recent multi-task RL algorithm of Eramo et al. 2020 on Pendulum, Bipedal Walker, and Atari problems.

While multi-task learning was extensively studied in supervised learning, its application to RL has only recently begun to gain attention. This paper contributes by a new multi-task RL algorithm that addresses the negative transfer issue arising when one tries to apply multi-task learning strategies across partially unrelated tasks. However, I am not sure if this paper is ready for ICLR in the current form, for the reasons given below

- Limited novelty: Negative transfer in multi-task learning has been previously studied, mainly in supervised learning but recently, also in RL. Specifically, task clustering approaches have been previously examined although as far as I am aware, they were used only in supervised learning. 
Even though the proposed EM strategy is not a trivial application of existing approaches, the technical novelty seems to be limited. Enhancing the practical relevance of the proposed algorithm via an extensive empirical study (involving multiple state-of-the-art approaches and challenging real-world problems) could help. However --->
- Limited experiments: Two simple baselines are not multi-task learning algorithms, and Eramo et al.’s approach was not designed to address the negative transfer problem. I think the experiments can be improved by including comparisons with more, state-of-the-art algorithms including these discussed in the last two paragraphs of Section 2. Yu et al. 2020 Sharma et al. 2018 Hesse et al. are designed to address the negative transfer issue, thus they should provide better baselines than SP, PPT, and Eramo.
- ATARI experiments (Figure 6) seem to suggest that when the individual tasks are not related, (hence multi-task learning is not beneficial), applying the proposed algorithm can degrade the performance from the PPT, still suffering from the negative transfer. Their results are much better than Eramo et al. 2020, but the latter method was not designed for negative transfer cases. 

Section 4 describes well the operations performed by the proposed EM algorithm. Still, explicit equations on the E and M steps would help reading.

Thank the authors for their responses. I read through the responses from the authors and comments from the other reviewers. I would maintain my initial rating: I think this paper will benefit significantly from a major revision, either strengthening the theoretical contributions or improving empirical validations (by actually performing extensive comparisons with existing algorithms that are designed to handle negative transfer problems).

review_title: Technical contributions seem to be limited.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary:

The authors present a clustering-based approach to multi-task learning, specifically task-policy assignment, which iteratively learns a set of different policies for each of the subtasks. The aim is to solve issues with contradicting objectives of different tasks.

The clustering is achieved in an unsupervised way inspired by the EM algorithm, by iteratively evaluating the set of policies on all tasks, assigning tasks to policies based on their current task performance, and then training policies on their assigned tasks.

The proposed approach is evaluated on several continuous control and Atari multi-task RL problems, and compared to single policy and per-task policy baselines, as well as with a multi-head baseline inspired by an approach from the recent literature.
In order to show the importance of the clustering component of the approach, an ablation study is conducted.


### Review:

The paper proposes a simple and novel approach to task selection and policy learning in the multi-task learning problem setting. It provides proof of concept evaluations, as well as evaluations on some more complex environments where the task clustering is not obvious. I appreciate the discussion of the results and how the hypothesis relates to the observed outcomes.

Below, there are several suggestions that I believe would help improve the quality of the manuscript, and address some of the issues that should be clarified.

1) Regarding multi-task learning literature, it would be useful to mention the relationship with Quality-Diversity approaches: 
 - Cully, Antoine, and Yiannis Demiris. "Quality and diversity optimization: A unifying modular framework." IEEE Transactions on Evolutionary Computation (2017)
 - Pugh, Justin K., Lisa B. Soros, and Kenneth O. Stanley. "Quality diversity: A new frontier for evolutionary computation." Frontiers in Robotics and AI (2016)

 As well as Deep RL approaches with skill/task conditioning:
 - Eysenbach, Benjamin, et al. "Diversity is all you need: Learning skills without a reward function." International Conference on Learning representations (2019).
 - Achiam, Joshua, et al. "Variational option discovery algorithms." arXiv preprint arXiv:1807.10299 (2018).

 It would greatly improve the strength of the manuscript to have comparisons with these state-of-the-art approaches as well, although I understand that this would require a significant effort from the authors.

2) Could you provide an example of these assumptions: “Note also that this formulation only relies on minimalistic assumptions. It is therefore applicable to a much broader range of settings than many sophisticated models with stronger assumptions.”

3) In the E step, the task $\tau_i$ is assigned to a policy $\pi_i$, which is used within the whole cluster $T_i$. There are N policies and K clusters, so it is not guaranteed that there is going to be one policy per cluster (i.e. $N=K$)? This part is not completely clear so it could be maybe explained.

 Related to this, how is the initialisation of the policies and clusters done, i.e. how are the number of initial policies and clusters selected? I guess the policies are manually set and this would be one of the hyperparameters that has to be tuned and is task-specific.
For the proof of concept tasks the number of policies can be set as the number of clusters that we can expect. Is this what is done in Bipedal tasks or is this an arbitrary number? For Atari experiments it is explicitly mentioned that n=4 and that the same is done for Bipedal tasks. This is a very important detail of the proposed approach and should be explained explicitly.

 Moreover, as I understand, at the beginning each task is attempted by each of the policies or they are assigned randomly. This explanation is missing.

4) It would be useful to provide some discussion about the stability of this approach, as in the beginning the policies could have tasks from different clusters assigned to them, but also the clusters change. I would assume that there is some burn-in period necessary.

5) The results on Atari for EM are not outperforming other baselines. Could you please comment on this? Could this be related to the number of policies set? It would be informative to evaluate the sensitivity of the EM algorithm to different numbers of initial policies.

 The conducted ablation study gives useful insights into the mechanisms of the approach. For the Corner-grid-world it makes sense that increasing n > 4 does not improve the performance, as there are 4 salient clusters (you could also mention this in the discussion). However, it is difficult to see a pattern in the effect of the number of policies for the Pendulum task. Therefore, investigating this in more detail, and in other environments as well, would greatly help understanding the approach better. 

 One useful experiment would be to see how the performance gap between using randomly assigned clusters and learned clusters changes with respect to the initial number of policies.

6) Another useful metric would be evaluating the number of generated clusters and analysing how they relate to qualitatively different skills/tasks/behaviours.

7) Also, a comparison of computation time and required memory would be nice to show, with respect to SP and PPT for example, because additional policies need to be trained/stored. I assume, memory-wise and training-time-wise, EM is between SP and PPT.

OTHER COMMENTS:
- There are some links embedded in the pdf which are not visible and always link to the title page.
- What do you mean by “relationship of tasks of different clusters is **unconstrained**”, in which sense unconstrained ?
- How is the mean reward calculated for PPT in Figure 2? I assume it is done via evaluating each policy on the corresponding task, but this should be emphasised.
- Fig 5. right shows that SP converges faster than EM but then starts degrading, what could be an explanation for this?




review_title: Interesting idea and simple approach; Additional discussion and investigation would be helpful.

Meta-review: This paper considers multi-task RL from the perspective of an unsupervised clustering of different tasks with an EM-like algorithm. The idea is evaluated on several simple and ATARI domains.
We thank the reviewers for their detailed responses and revision. This work still seems a little preliminary in its current form. While the empirical results seem promising, it is generally felt that it would benefit from more extensive experiments, including further comparisons to other approaches and exploring the effects of the hyperparameters on tasks with much larger numbers of clusters. It would also be beneficial to provide some theoretical results, particularly with respect to negative transfer.
Decision: Reject