Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of task grouping by using an efficient and effective meta-learning framework. The authors propose to use a meta-learning algorithm, which predicts the per-task performance gains of multi-task learning over single-task learning for any combination, to find optimal task combinations. The key insight is the low-rank hypothesis on task combination space. Based on the hypothesis, the authors develop MTG-Net, which uses an active learning strategy to select meta-learner’s training examples. Experiment results conducted on multiple multitask datasets show significant improvement over state-of-the-art task grouping algorithms. 


strengths_and_weaknesses: Strengths
1. This paper is well written and easy to follow. 
2. The approach of introducing a meta-learner to learn task grouping strategies is novel and reasonable.
3. Multiple MTL datasets are used to evaluate the proposed method.

Weaknesses
1. On the efficiency side, even though an active learning algorithm is introduced to reduce the number of samples needed to train the meta-learner, the cost can still be large for training models and measuring the gains. I am wondering if a similar hypothesis from ENAS[1] can be used to further save cost by sharing the parameters of the pair of models for measuring gains.
2.  There exists some related work [2] in AutoML that also leverages the low-rank hypothesis to learn AutoML algorithms based on task-task relationships. Even though the application is slightly different, the novelty of building meta-learned based on low-rank hypothesis is limited.
3.  Even though the author compared it with SOTA for task grouping. I am wondering how the proposed active learning framework compares with the existing AutoML framework, which models the meta-learner as an RL algorithm[1].

[1] Pham, Hieu, et al. "Efficient neural architecture search via parameters sharing." International conference on machine learning. PMLR, 2018.
[2]  Yang, Chengrun, et al. "OBOE: Collaborative filtering for AutoML model selection." Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.

questions: Q1  Can a similar hypothesis from ENAS[1]  can be used to further save cost by sharing the parameters of the pair of models for measuring gains?
Q2 How the proposed active learning framework compares with the existing AutoML framework, which models the meta-learner as an RL algorithm?

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a meta-learning method to learn to predict the transfer gains for each task grouping in a Multi-Task Learning problem. This is based on the assumption that, while the number of task combinations (datapoints in this meta-learning problem) is huge, the data lies in a low-dimensional manifold (manifold hypothesis) and thus can be learned. Since the meta-training set needs to be small (computational cost), the authors also apply active learning to help select the most informative datapoints (groupings) to train.
strengths_and_weaknesses: **Strength**
- The idea of the paper is clear. The manifold assumption also seems reasonable, which suggest that we can do better than heuristic estimations of the transfer gain (via learning to predict it).
- The paper obtain good performance when compared with heuristic approaches such as HOA or TAG.

**Weaknesses and Questions**
- One of my main concerns is the computational cost of MTL procedure. The authors mentioned that they consider $k=1$ and $k=(n-1)/2$ so that the number of MTL combinations is roughly equal to TAG and HOA. However, for example, the combinations in HOA only consist of 2 tasks, so might be much less expensive than a combination considered in MTL-Net. Therefore, the evaluation may still be unfair. We should really consider the total computational cost here, instead of the number of task combinations.

- If my understanding is correct, the task embedding (taken out from an embedding table) is independent of the task data. I wonder if we can additionally pass (part of) the task data to a set embedding network to get the task embedding. It might open the door to generalizing to some unseen (but related) tasks at test time. 

- Also I found the indicator of transfer gain to be inconsistent in the experiments. For some experiments it is the loss, for some others it is the AUC or the mean absolute error. I am a bit skeptical about this since this might have been cherry-picked. Why don't we use a consistent category across the experiments? 

- The authors only present experiments with a hard sharing case of MTL. It is natural to wonder if the method works for soft-sharing of the parameters. Is it the case that we can predict better the transfer gain for hard sharing only?
questions: See above.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a simple yet effective meta-learning framework for multi-task grouping. The main goal is to avoid computing the performance gains of multi-task learning (MTL) over single-task learning (STL) for all possible task combinations, which is intractable for a large number of tasks. During meta-training, a mapping function between task combinations and performance gains is learned, where the meta-training set is selected using an active learning strategy to keep it small in size while still being effective. During meta-testing, the performance gains of the rest of the task combinations are predicted using the learned mapping function. Finally, the task groupings are selected based on the meta-training (ground-truth) and meta-testing (predicted) performance gains. Experiments on 3 datasets demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths
- The proposed framework to learn the mapping between task combinations and performance gains is novel.
- The contribution is quite significant as based on their experiments, the proposed framework significantly improves over the pairwise non-parametric approaches under similar computational costs. The visualization also helps to reveal why it works.
- The paper is well-written and very easy to follow.
- Most of the presentations are clear.

Weaknesses
- Certain parts of the paper lack clarity (see suggestions below).
- The idea of selecting task groupings based on performance gains is very costly in nature as it involves a lot of MTL training. Given that the experiments cost “thousands of GPU hours in total” (line 268), the experimental results might be difficult to reproduce.

questions: I think this is a rather decent submission with a novel idea and convincing experimental results. Apart from the whopping computational cost of conducting the MTL procedures and perhaps the difficulty to scale to deeper models, I only have one minor suggestion for the authors to improve the paper.

- The proposed active learning strategy assigns sample weights to prioritize task combinations with larger (absolute) performance gains. According to my understanding, here the performance gains are computed using the learned MTG-Net (instead of ground-truth). Is there a rationale behind why doing so (i.e., prioritizing task combinations with larger predicted gains) is more helpful for learning MTG-Net?

I also identify the following typos.
- Line 196, “work tokens” should be “word tokens”.
- Line 198, the times sign $\times$ in the shape of $\mathbf{X}_i$ is written as an $x$.
- Line 205, the shape of $H_i$ should be $|C_i| \times D$.
- In Figure 1, the STL procedure in the left subfigure, you may change the title to "for $t^i_j \in C_i$" to keep the notation consistent with your text in line 132.
- In Figure 2, performance vs budget for Taskonomy-5, the oracle line is missing.

limitations: The authors have pointed out the limitations in lines 353-356.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for multi-task grouping, which is to find the best sets of multi-task networks that covers the given target tasks. Specifically, the paper trains a model that predicts performance gains as a function of task combinations. Based on this prediction, best task combinations that maximizes the total performance gain can be obtained for given N tasks and budget B through the grouping selection algorithm. To save the training cost, the paper proposes an active learning algorithm that selects task combinations to use for training based on the predicted performance gains. Experiments show that the proposed method outperforms existing MTG methods and predicts total performance gains reasonably well on three MTL datasets (Taskonomy-5, ETTm1-7, MIMIC-III-27).
strengths_and_weaknesses: I think this paper proposes an interesting approach for the multi-task grouping problem. Compared to [41] which also takes into account the compute budget constraint, this paper does not consider the compute budget in a strict manner, however, the computational complexity is reduced from O(2^n) to O(n^2).

1. Strengths
- The key finding of the paper is that the relationships between task combinations and performance gains lie in some low-dimensional space and can be approximated reasonably well by the proposed self-attention encoder architecture. I think this is an original finding and also provides good insights.
- Unlike existing works which approximated relations of task combinations using pairwise task relations, this paper proposes to predict the task gains as functions of higher order task combinations directly using the self-attention encoder and task representations. I think this design is novel and reasonable.
- The paper extensive experiments on three datasets, ranging from 5 to 27 tasks. Ground truth performance gains are collected and will be released, which I believe will be useful for future MTL research.

2. Weaknesses
- I think one weakness is the high computational cost. The method requires multi-task network training for O(NK) task combinations. Though it is much smaller than the number of all possible combinations, however, this could be still expensive especially when the number of tasks is large or datasets are large.
- Analysis of the effect of some design choices (hyperparameters, MTL model architecture, MTG model design) is missing.
- As also mentioned in the paper, the prediction accuracy is less reliable with smaller number of meta-training data selection. Since MTG-Net is trained using only a small sets of (task combination, accuracy gains) pairs, the MTG-Net design (initialization, architectures) can affect a lot on the prediction accuracy of the performance gains. For example, at the first round of MTG-Net training (k=1 in Algorithm1), the MTG-Net used to sample meta-training set is trained using only one (task combination, accuracy gain)-pair.
- It is good that the paper takes into account the budget B, which is defined as the allowed number of task groups. However, the actual resource usage  (e.g. inference time, memory), which are of more interest in practice, may not be proportional to this budget B because the multi-task network for each group may have different resource usages.
- Experiments look relatively weak in terms of dataset size and task diversity. It would be nice to have experiments for datasets with larger number of data points and tasks.
questions: - It would be interesting to see which task combinations are selected at each step of active learning. How similar they are for each task and do their sizes change over iterations? Also, it would be interesting to see the average train/test error of gain predictions over the iterative training combination sets selection procedure, how it improves and what is the final error.
- How sensitive the method is to the different design choices of MTG-Net (initialization, number of layers, feature dimension, etc)? Since it is trained using only a small sets of (task combination, accuracy gains) pairs, the MTG-Net design can affect a lot on the prediction accuracy of the performance gains. For example, at the first round of MTG-Net training (k=1 in Algorithm1), the MTG-Net used to sample meta-training set is trained using only one (task combination, accuracy gain)-pair. I am curious how much they affect the results.
- How does the method work for different hyperparameter choices (D, alpha, eta)? For example, alpha is currently set to 25, which I guess will greedily choose the task combinations with best gain for each task. This makes sense to me because these task combinations with extreme gains likely serve as anchors and help interpolating gains for other combinations. At the same time, selecting diverse combinations could be more sample-efficient when training MTG-Net. How important the current hyperparameter choices are and what happens they are set to other values?
- It would be interesting to see if task relations are model-agnostic. Will the gain estimation obtained from smaller MTL backbone generalizes to larger MTL backbones? This may reduce the training cost.
- Which task prediction is selected at test time if that task is included in several task combinations and there are multiple predictions?

limitations: The authors discussed the limitations in the paper. I do not see potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper researches on the task grouping problem in Multi-task Learning (MTL). Task grouping matters MTL’s performance and has gained much attention recently. This paper proposes a novel method to address the task grouping problem, which formulate  a meta-learning problem for task grouping. The proposed method has achieved superior performance.
strengths_and_weaknesses: Pros

1.The proposed method is experimentally verified to be effective on various applications.
2.The paper has conducted extensive experiments .
3.The paper is easy to follow.

Cons

1.The technical contribution of this paper is limited for it just utilizes the well-known meta learning frame-work.
questions: What’s the performance of the active learning strategy in the early epochs?
limitations: The technical contribution of this paper is limited for it just utilizes the well-known meta learning frame-work.
review_title: null

Meta-review: metareview: The overall idea of using a meta-learning network with an active learner for grouped multi-task learning is interesting. The experimental results provided in the original submission and rebuttal are extensive to verify the effectiveness of the proposed method. A major limitation of the proposed method is the high computational cost, especially when each task has its own dataset. 

Overall, this is a well-written paper that presents an interesting idea for multi-task learning. 
confidence: Certain
Decision: Accept