Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes some interesting observations and proposes a novel measure of transference. Based, the proposed measure, this paper proposes a task-grouping method and novel training algorithm for MTL. However, there are some flaws in the proposed method and the proposed methods seems not technically sound. My concerns are listed as follows.

1. The definition of the transference measure is problematic. The measure is depended on the empirical loss. However, in MTL, the negative transfer or positive transfer is proposed with respect to the generalization loss. The increasing empirical loss may not lead to an increasing generalization loss. This paper should give theoretical support to show that the proposed measure can indicate the influence on generalization loss.

2. This paper does not provide experimental sufficient support for the superiority of the proposed method. For example, this paper is closely related to the previous work [1]. However, this paper has not compared with it. The authors should add the comparison with [1].

3. The paper is hard to follow. Some important details are not clear. For example, how to do task grouping based on the measure of transference are not clearly written.

[1]. Yu T, Kumar S, Gupta A, et al. Gradient surgery for multi-task learning. NeurIPS, 2020.

review_title: The paper is not technically sound.

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors derived a quantitative measure of transference among tasks in the setting of multi-task learning. Based on this measure, two methods were subsequently proposed, one to find group of tasks that may benefit from collaborative training, the other to combine task gradients at each training step. 

Despite there are typos in a few places, the paper is good written and easy to follow. 

No comparison with any baselines was done to show the advantage of the proposed method to identify task groups. One such baseline can be first training each task separately, then for each trained network fixing the shared parameters, resetting and adapting the task specific parameters for other tasks, and cross comparing the performance of all obtained models to identify task group for which collaborative training could be beneficial for individual tasks in the group. 

I was not able to find any guidelines in the paper for how to construct the candidate set J. So, it would be most likely exhaustive search? I guess because of this, as the authors mentioned, their method is not scalable, only can handle problems containing small number of tasks. The empirical results are very weak. With considering the variance, I doubt any of the improvement in Table 2 is statistically significant.  Such weak results do not justify the significantly elevated computational cost. Based on the results included in the paper, I do not think the proposed method has any practical significance. 
review_title: Paper is well written. More comparison is needed. The significance of the proposed method is questionable. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the transferability in multi-task learning. They propose a metric, transference, to evaluate how tasks affect each other during multi-task training, and a method called IT-MTL which utilizes this metric to compute and improve lookahead loss changes. Although the proposed metric and method are interesting from a scientific point of view, there are a few key downsides (as the author themselves summarized in the conclusion) that require further investigation/improvements.

Pros:
1. The idea of using change in losses to capture inter-task transferability is new and could have potential impact in the community.

Cons:
1. The biggest concern is efficiency. The proposed method requires calculating lookahead loss for a set of combinations of tasks. While the author claim it only require O(m) possibilities empirically, it is not guaranteed for other datasets/settings. In fact, even with O(m) complexity could still be prohibitive for large models. To reveal the full picture, I recommend the author to demonstrate the actual time required for training for each method, and conduct an ablation analysis on the efficiency-performance trade-off.
2. From Table 2, I can only observe marginal improvements over prior methods. Considering the extra computational cost, it is hard to justify the effectiveness of the method itself.
3. The proposed method does not account for optimizers with ema gradients such as Adam, which may diminish the effect of selecting subset of tasks.

Other comments:
1. The idea of using lookahead method on shared layers in multi-task learning has been recently explored in [1]. Although they use a different optimization process, the high-level idea of improving transference/validation loss is shared. On the other hand, dropping gradients is also recently proposed [2]. Though concurrent, it still might be good to mention in the final version.
2. I would also recommend exploring some greedy strategies for the proposed method to be efficient. In other words, how to greedily reduce the search space of combination of tasks.
3. This is not very important, but what will happen if we directly use the transference metric to reweight tasks? For example, in multi-source transfer learning, can we utilize the metric proposed to decide what tasks to use?

[1] On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment. Wang et al., EMNLP 2020. 

[2] Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout. Chen et al., NeurIPS 2020.
review_title: Interesting paper with major flaw

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary] This paper studies the problem of task relationship/transference in multi-task learning, by introducing a quantifiable measurement based on relative loss updates.  A (nonsymmetric) task transference between task $i$ and task $j$ then can be computed by measuring the relative change of training loss of task $j$, with the updated shared parameters from the training loss of task $i$. By finding a subset of tasks that achieves maximal total transference over every single task, multi-task learning performance can be further improved.

[Strength] Understanding multi-task relationships and transference is important to achieve a good multi-task learning performance and on multi-task architecture design. The proposed solution based on the relative loss update is intuitive, clean, and simple to implement.

[Weakness] 
1. **Task selection.** To achieve a maximal performance improvement, it seems that we have to compute task transference from all possible combinations of task grouping, which is exponential based on the number of tasks. The authors claim that "a carefully chosen subset of tasks with $|\mathcal{J}| =  \mathcal{O}(m)$ provides reasonable improvements", but I cannot find any details on how these tasks are selected. Besides, in the experiment section, the number of tasks evaluated is no more than 3, and the improvements are quite marginal (mostly within 0.5%). 

2. **Visualisations are unclear.** In Fig. 1, task transference is visualised by the pairing of all possible combination of two different tasks. But $\xi$ selects a subset of tasks, rather than just an individual task. So how can we know the transference for each individual task just by the transference for all tasks? Or in this case, $\xi$ is specifically designed to be a single task? The authors should elaborate on these details.

2. **Notations are unclear.** $\xi$ represents a non-empty subset of all task groupings. So why introduce another notation $\mathcal{J}$ in Algorithm 1 to include all possible task groupings, this looks quite redundant to me.


I hope the authors could elaborate on these questions and include more relevant details in the paper.
review_title: The idea is interesting, but writing should be improved.

Meta-review: This paper proposes a method to quantify transference, which is a measure of information transfer across tasks, for multi-task learning framework. Specifically, the transference is measured as the change in the loss for a specific task after performing a gradient update for another. The proposed transference measure is used to both understand the optimization dynamics of MTL and improve the MTL performance, either by grouping tasks or combining task gradients based on the transference. The method is validated on multiple datasets and is shown to bring in some performance gains over the base MTL model (PCGrad, UW-MTL). 

The majority of the reviewers were negative about this paper (4, 4, 5), while one reviewer gave it a positive rating (6). The reviewers in general agreed that the idea of measuring transference as the change in the loss with gradient updates is novel and intuitive. Yet, the reviewers had common concerns on the 1) weak performance improvements, and the 2) high-cost of computing the transference. While computing the transference requires additional computations with linear time complexity, which may be problematic with a large number of tasks, the performance gains using it were rather marginal (less than 0.5% over the baselines).  Another common concern from the reviewers was its insufficient experimental validation, as a comparative study against existing works that perform task grouping is missing. Both the authors and reviewers actively participated in the interactive discussion. However, the reviewers found that the two critical limitations persist even after the authors’ feedback, and in a subsequent internal discussion, they reached a consensus that the paper is not yet ready for publication. 

Thus, although the proposed method is novel and appears to be promising, it may need more developments to make it both more effective and efficient. Moreover, there should be more in-depth analysis of its time-efficiency, and other benefits (e.g. interpretability) that could be achieved with the proposed transference measure. Finally, while there exist many works on learning both hard or soft task grouping, the authors do not reference or compare against them. To name a few, [Kang et al. 11] propose how to learn the discrete task groupings, [Kumar and Daume III 12] propose to learn a soft grouping between tasks, [Lee et al. 16] propose to learn soft grouping based on asymmetric knowledge transfer direction across the tasks, and [Lee et al. 18] proposes the extension of [Lee et al. 16] to a deep learning framework. I suggest the authors to discuss and compare against the above mentioned works, and fortify the related work section by searching for more classical works on multi-task learning. 

- [Kang et al. 11] Learning with Whom to Share in Multi-task Feature Learning, ICML 2011
- [Kumar and Daume III 12] Learning Task Grouping and Overlap in Multi-task Learning, ICML 2012
- [Lee et al. 16] Asymmetric Multi-task Learning based on Task Relatedness and Confidence, ICML 2016  
- [Lee et al. 18] Deep Asymmetric Multi-task Feature Learning, ICML 2018.
Decision: Reject