Efficient and Effective Multi-task Grouping via Meta Learning on Task Combinations
Authors: Xiaozhuang Song
Shun Zheng
Wei Cao
James Yu
Jiang Bian
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-task Learning
meta Learning
representation Learning

Abstract: As a longstanding learning paradigm, multi-task learning has been widely applied into a variety of machine learning applications. Nonetheless, identifying which tasks should be learned together is still a challenging fundamental problem because the possible task combinations grow exponentially with the number of tasks, and existing solutions heavily relying on heuristics may probably lead to ineffective groupings with severe performance degradation. To bridge this gap, we develop a systematic multi-task grouping framework with a new meta-learning problem on task combinations, which is to predict the per-task performance gains of multi-task learning over single-task learning for any combination. Our underlying assumption is that no matter how large the space of task combinations is, the relationships between task combinations and performance gains lie in some low-dimensional manifolds and thus can be learnable. Accordingly, we develop a neural meta learner, MTG-Net, to capture these relationships, and design an active learning strategy to progressively select meta-training samples. In this way, even with limited meta samples, MTG-Net holds the potential to produce reasonable gain estimations on arbitrary task combinations. Extensive experiments on diversified multi-task scenarios demonstrate the efficiency and effectiveness of our method. Specifically, in a large-scale evaluation with $27$ tasks, which produce over one hundred million task combinations, our method almost doubles the performance obtained by the existing best solution given roughly the same computational cost. Data and code are available at https://github.com/ShawnKS/MTG-Net.

Efficient and Effective Multi-task Grouping via Meta Learning on Task Combinations
Xiaozhuang Song"
Shun Zheng Microsoft Research shun behhengmmicroooeeeee
Southern University of Science and Technology shawmsxz900gmail.cm
Wei Cao
James J.Q. Yu
Microsoft Research
Southern University of Science and Technology yujq3@sustech. edu. cn
wei www.maaaoosssc.cc com
Jiang Bian
Microsoft Research
j iang. www.aiiossoott.ooo
Abstract
As a longstanding learning paradigm, multi-task learning has been widely applied into variety of machine learning applications. Nonetheless, identifying which tasks should be learned together is still a challenging fundamental problem because the possible task combinations grow exponentially with the number of tasks, and existing solutions heavily relying on heuristics may probably lead to ineffective groupings with severe performance degradation. To bridge this gap, develop systematic multi-task grouping framework with new meta-learning problem on task combinations, which is to predict the per-task performance gains of multi-task learning over single-task learning for any combination. Our underlying assumption is that no matter how large the space of task combinations is, the relationships between task combinations and performance gains lie in some low- low-dimensional manifolds and thus can be learnable. Accordingly, we develop a neural meta learner, MTG-Net, to capture these relationships, and design active learning strategy to progressively select meta-training samples. this way, even with limited meta samples, MTG-Net holds the potential to produce reasonable gain estimations on arbitrary task combinations. Extensive experiments on diversified multi-tar -tsk scenarios demonstrate the efficiency and effectiveness of our method. Specifically, in a large-scale evaluation with 27 tasks, which produce over one hundred million task combinations, our method almost doubles the performance obtained by the existing best solution given roughly the same computational cost. Data and code are available at https: htt////gttttbbt com/ htt///SnnK///GG-RGG
Introduction
Multi-task learning (MTL), longstanding learning paradigm 37 50 has been widely applied
into a variety of machine learning applications, ranging from language understanding [1I] visual
recognition [33 41 and robotic control [19] to drug discovery (36) and elinical therapeutics [17 30).
The major motivation behind is to boost the performance of single-task learning (STL) by leveraging the additional supervision signals from other relevant tasks.
Work is done during the internship Microsoft Research Asia.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Nevertheless, naively grouping multiple tasks together for MTL easily results in severe performance degradation in practice. Many recent studies acknowledged this negative-transfer phenomenon and speculated that the major reason lied in the competition and interference among incompatible
tasks [39 47 15
[]. To mitigate such negative-transfer effect, recent years have or model 33 28 [S]. But explicitly identifying a group of tasks that would benefit from
2
witnessed few ex explorations on developing new optimization approaches (39 25
29
48]
architectures [27
training together still remains a challenging fundamental problem without adequate investigations [4] 14]. Thoroughly studying this multi-task grouping (MTG) problem not only advances core research over MTL aside from optimization architecture developments but also facilitates various MTL scenarios that can benefit from learning auxiliary objectives [2 6]
However, identifying the best MTG solution is extremely challenging as it involves an exhaustive search over the whole space of task combinations, each of which corresponds to a whole procedure of model training and evaluation on its specific MTL. objective. The cost of such an exhaustive search grows exponentially with the number of tasks and thus can be prohibitively expensive 6 41
To reduce this prohibitive computational cost, most recent efforts [41 [4] relied on the assumption of high-order approximations (HOA), meaning high-order performance gains can be estimated from corresponding pairwise gains by averaging, where the performance gain denotes the improvement of MTL over STL for a specific task. For instance, considering the group composed of tasks (A. B, C), they assumed that when learning with the joint objective of this group, the performance gain for A can be estimated by the average of associated pairwise gains for A, which were independently obtained by learning with { (A, B) ] and (A, C). Under the HOA assumption, they only need to collect the transferring gains for all pairwise combinations, of which the number is only quadratically dependent on the number of tasks. While substantially reducing computational requirements, the HOA assump- tion also results in wildly inaccurate estimations when there exist non-linear relationships between high-order gains and corresponding pairwise gains. As result, such untrustworthy estimations can significantly hinder the final performance of searching for the optimal grouping option.
To pursue more accurate estimations of various high-order performance gains under affordable computational cost, we develop a systematic MTG framework with new meta-learnnng problem on task combinations. To be specific, we divide all task combinations into two parts: one with a small number of combinations for meta training and the other with all the rest combinations for meta testing. Here the meta input is specific task combination, and the meta objective is to predict the per-task performance gains (the meta label) for this combination. The underlying assumption of this meta-learning formulation is that although the number of task combinations grow exponentially with the number of tasks, the relationships between task combinations and performance gains lie in some low-dimensional manifolds and thus can be learnable. Following this formulation, we build dedicated neural network, MTG-Net, as the meta learner, and further develop an active learning (35) strategy to progressively construct the meta- veea-training set. With this strategy, we can largely improve the effectiveness of MTG-Net with only a small number of meta- meta-traine training samples. After the meta- training stage, we leverage the gain predictions of MTG-Net on all task combinations to guide the final grouping selection. Extensive experiments on diversified multi-task scenarios, including vision, energy, and healthcare, demonstrate the efficiency and the effectiveness of our method. Moreover, we also visualize various latent structures discovered by MTG-Net, which empirically validate the practicability of our assumption.
In summary, our contributions include:
To our best knowledge, this paper is the first effort to formulate a meta-learning problem for MTG and thus enable an efficient exploration of the exponentially growing space of task combinations, which was believed to be prohibitively challenging [4] 30 6
We build neural meta learner, MTG-Net, that can capture the relationships between task combinations and performance gains effectively.
We develop active learning strategy to progressively select the most useful meta-trainin samples for efficient training of MTG-Net.
We conduct extensive experiments across diversified multi-task scenarios. Specifically, in a large-scale evaluation (27 tasks) with over one hundred million task combinations, our method almost doubles the performance obtained by the cxisting solution.

---

2
Related Work
The classic assumption of MTT in statistical learning was that putting related tasks into joint learning benefited from the inductive bias encouraging cross-task information sharing 8 9 7 35. However, when this assumption of relatedness did not hold, the resulting grgative-transfer effects can cause significant performance degradation. Thus the further research investigated the problem of simultaneously determining with which tasks each task should share while carrying over classical cross-task sharing 120, [22]. Nevertheless, these early MTL studies can hardly be adapted to modern deep neural networks [24] due certain prohibitive assumptions (14] As mentioned
MTG
by [41 [1]. identifying proper task groupings for deep neural networks conventionally required either computationally intensive coss-validation procedures or the human knowledge that is not always applicable to machine learning. The early attempt [T on more general and systematic MTG for deep neural networks formalized a standard workflow: 1) collecting the transferring gains (or gain predictions) for all 2 - task combinations, and then 2) conducting brute- search for the best grouping option given a specific budget, such as the maximal number of groups. The latter step is a deterministic searching procedure, but the former step is rather challenging because the optimal solution requires 2 times MTL training and evaluation, which can be prohibitively expensive when N increases. Thus [41 further proposed the HOA approximation to reduce the computational complexity from exponential into quadratie: ( times MTL on pairwise
task combinations. Subsequently, [4] improved the efficiency of [4] by substituting ( times pairwise MTL procedures with a single run, in which the computation complexity is only linearly dependent on N due to maintaining and updating a task affinity matrix on the fly per each gradient updating. This further approximation, named as task affinity grouping (TAG) in [4), traded the accuracy in estimating pairwise MTL gains for efficiency, which may result in more error propagation from pairwise higher-order. Different from all these HOA-based methods, this paper formulates a systematic meta-lcarning problem to estimate the transferring gains and builds an effective meta model to enable more accurate and robust generalization to massive high-order combinations.
Other Research on MTL & Meta Learning& & Task Embedding Recent years witnessed plenty of explorations MTL from other aspects, such multi-objective optimization 39
25
29
33 27 While these studies have made remarkable progresses improve MTL under a predefined task combination, the focus of this paper lies in explicitly determining with which tasks to share. Besides, as pointed by (37 14), these two paradigms are complementary to each other and can contribute performance improvements jointly. Moreover, our work falls into the paradigm of meta learning [43] 21 4 34 (40). But different from traditional meta-learning studies, which primarily focused on the fast adaptation across tasks in few-shot scenarios, our meta objective is to estimate the performance gains for different task combinations. To fulfill this objective, we develop a meta network that treats a task combination as set of task tokens and transforms them into task embeddings to learn their interactions. Meanwhile, we also note some existing studies leveraged similar idea of learning task embeddings but for different purposes, such as inferring task similarities [1, estimating treatment
neural architecture search [28 26
16
15 42), and soft-sharing mechanisms 13
effects 38 facilitating few-shot learning 100 or reinforcement learning 23 etc.
3 Meta Learning for Multi-task Grouping
In this section, we elaborate on how our meta learning framework works for MTG. First, in Section 3.1 we establish the basic notations and briefly introduce some crucial steps of MTG for better under- standing. Then we formulate our meta learning problem on task combinations in Section 3.2. Next. introduce the details of MTG-Net and the accompanied active learning strategy in Sections and 3.4 respectively. Figure I gives an overview of our framework.
3.1 Preliminaries
Notations We adopt the following notations throughout this paper. N is the number of learning tasks
in total. T = (Ti, T2, ... ] is the universal set of all tasks, where T, denotes the j-th task, and
we have |T| = N. C= (Ci, Cz. ... C2~-11 } is the universal set of all 2 - combinations, which contains ( combinations containing only one task, (g) combinations containing exact two tasks,
(5) three-task combinations, etc. C. [t] th. ... Hf,y] denotes the i-th task combination, where
3

---

The Construction Meta-training Set
Meta Learning on Task Combinations
Multi-takk Learning Procedure for C2
erramn
Ceest
Hard-sharing Architecture
Decodier
Decoder
MultiTask
Per task Evaluation
yitar
Grouping Selection Based on yerate grest
Task 7
Task Tw
Encoder
Decoder
Decoder
Fly
The Single-tast Learnnng Procodure for e
MTG-Net
Encoder
Singl: Task Training
Evaluation
Transfering Calss V, e garrre
Meta-Train
Input
Figure 1: An overview of our meta learning framework for multi-task grouping
2 E T denotes the j-th task in this combination. y {,, Y2, -2x11 is the universal set of all transferring gains corresponding to C. Y; [vi V2: ... 3/c,1] includes all per-task transferring gains for task combination C. where v) denotes the gain of conducting MTL on task combination C; over performing STL for task tj, and we have |Yi| = [Ca]. etrain is the of task combinations for meta training, and yatrain denotes the set of corresponding transferring gains. Crest is the set of task combinations for meta testing, and yoastt also includes corresponding transferring gains. B denotes the budget in grouping selection, which is the maximal number of selected task combinations.
-1
The MTL Procedure Given task combination C, we define its MTL procedure training with the joint objective of C2 and evaluating on the validation set for each task in C₁ to collect per-task performance indication of generalization. In accordance with 41 [4] hard-sharing architecture for MTI., in which there is a single encoder to obtain shared representations followed by separate decoders for specific tasks. Moreover, we retain the same split of train. validation, and test sets for each MTL procedure and fix the optimization algorithm as well as other hyper- parameters. In this way, we can attribute the variations of generalization performance for a specific task largely to the different MTL objectives of various task combinations.
adopt widely used
Transferring Gain Here we make more clarifications on collecting transferring gain y for task i by performing MTI. combination C, First, we conduct the aforementioned MTL procedure to collect the generalization performance on the validation set for task 13 in C.- Then, we define the transferring gain of conducting MTL on C2 for lj as the improvement of the performance of MTL over the base performance of STL for fj. In practice, we can instantiate this improvement in different ways, such as the reduction of loss, the improvement in accuracy, etc.
Grouping Selection Once obtaining all transferring gains, not only possess the hest overall performance but also meet some practical considerations, such as inference latency. This selection procedure is in essence constrained searching problem, which is to select group of task combinations covering all tasks to maximize the overall transferring gain (the average of per-task gain) while ensuring the number of groups does not exceed given budget (B). As noted by [4], this problem is NP-hard in general. A solution of recursive searching has the computational complexity of ( -B). In our experiments, when N is small (e.g.. 5 vision tasks), we follow 41 14) to use the branch-and-bounde groupings. While for large N and B (e.g.. 27 medical tasks), the exact search becomes prohibitively expensive. In this case, we approximately select nptimoptimal groupings via beam- search approach with polynomial complexity, which is attached in the appendix.
select the optimal groupings that
algorithm to search for the optimal
3.2 Meta Learning on Task Combinations
Different from all existing brute-force or heuristic approaches, we formulate new meta-learning problem on task combinations as more systematic framework for the estimations of transferring gains. As mentioned in preliminaries, predefined configuration of the MTL procedure implies that the variations in transferring gains (Y;) can largely attributed to the MTL objectives of different task combinations (C.). Therefore, we assume there is an underlying mapping F to determine the

---

transferring gains given specific task combination: = ==F((C,, for C. E C. Furthermore, this view stimulates us to design learning process to obtain good approximation F to the oracle function F with limited meta samples from C. The underlying assumption of the learnability with limited meta samples is that the relationships between task combinations and performance gains lie in some low-dimensional manifolds no matter how large the space of task combinations is. In our experiments, we empirically validate the practicability of this assumption.
To be specific, we divide the whole set C into two parts: one part Ctrain for meta training and the other Clest for meta testing. For every combination in Cfrain we conduct an MTL procedure and collect corresponding transferring gains (Qitrain) to learn a surrogate function F While for all combinations in Crest, we directly apply F= produce estimated transferring gains, the set of which denoted as Yteat, Accordingly, we summarize the whole meta-learning procedure as
F Meta-TTainn (Ctrain ,ytrain), joeett Meta-Test (Ctest ,F).
This meta-learning perspective provides an explicit trade-off between efficiency and effectiveness for MTG. On the one hand, as the size of Ctrain corresponds to the major computational cost, decreasing [Ctrasn| helps improving efficiency. On the other hand, since the estimation performance on Cleat largely determines the quality of the final grouping decisions, increasing [ fosters effectiveness. Besides, given fixed budget of (Ctrain), |, namely affordable computational cost, we can devote more efforts to develop advanced meta-learning strategies to improve the gain estimation. Moreover, this new view can incorporate the well-known HOA assumption as a special case. The HOA assumption groups all pairwise task combinations into Ctrain and takes the average of pairwise gains as the approximation of high-order gains for all other combinations in Crest Thus view it as an nonparametric method to represent the oracle mapping F.
3.3 MTG-Net
In the following, we elaborate on the design of our meta learner. Specifically, we call this meta learner MTG-Net, namely meta network facilitate multi-gask grouping. First, to provide rich representations for discrete task indicators, we introduce a classic technique from natural language research (32 : transforming discrete work tokens into vector representations while preserving semantic relationships. In our case, we also construct an embedding table E E R Nx to transform the input C2 into a vector of embeddings, denoted as X, = [ mby ... x c. E RRI.laDD where /) denotes the embedding dimension, and x) E R x corresponds the embedding for task
Afterwards, to effeetively capture the diversified interactions in different task combinations, we stack several self-attention encoding layers (45) over the dense representation X, The reason is that the self- attention mechanism not only achieves remarkable successes in a wide variety of applications [12 [8] but also perfectly matches the unordered property of task combination and holds sufficient capacities to model those diversified inter-task interactions. Here we summarize this encoding procedure as H, Self-Att-Em (X.). where H; RN is the vector of enriched representations that encode diversified interactions among tasks, and Self-Att-En is the self-attention-bant bassdd encoder developed by 45). Moreover, another benefit of Self-Att-
is that the encoded vector H, shares the same length with the label vector Y, Naturally, we stack a linear mapping over H; to obtain the .fc.. E RIGi corresponds to our estimation of Yú új corresponds the estimated gain for 13,. w E RRD is a weight parameter, and b E IR is a scalar parameter being broadcasted to each entry of H; w. In summary, MTG-Net takes in a task combination (C:) and emits collection of gain estimations (.)) Recalling Equation L.. we formulate the specific meta-learning procedure for MTG-Net as:
final output as Y; H; w + b, where the vector y; = [ūt 02.
Meta-Train: O' arg min
Σ
|I'E -V1l
C,
Meta-Test:
jotest (MTG-Netes (C:), for C₂ E Cltest)
where Y/ = (C.,,, θ encapculates all parameters of MTG-Net, and O" denotes well- trained θ By replacing the computation-intensin MTL procedure with network inference, we can collect the estimations of all transferring gains much more efficiently. Moreover, as long as the estimation performance is reasonable, we are still able to identify near-optimal groupings.
5

---

Algorithm 1: Active Learning for MTG-Net
Input: Ctrain (C2N-11 Clest = C (train K, 77, Initialize O*, Veest via 2
for k in [1. .... K| do
for j in [1...., do
CT2 =Ct, for C2 E Chest TJ E Ci)
GT, (T)( for C. E CT=}
pT, (expa-Ic.1, for Gi G // Build the sampling distribution
Cneat = Samplep @p,, (C I/ Select the next meta sample
Cirain insert (Cuert)
Crest remooeeennee
// Update the meta-training set
(k.N N + ) % = then
Update 0º, Vrest via 2
// Update MTG-Net and its predictions
end
end
end
Output: O* and Steest
3.4 Active Learning for MTG-Net
Given the meta-learning procedure 2 for MTG-Net, however, the construction of Ctrain is still critical yet unsolved issue, which can have huge influences on the generalization capability of MTG- Net. To enable more efficient learning, we develop an active learning (35) strategy for MTG-Net. Here the key intuition is that paying more attention to the combinations with large gains can foster more efficient estimations of y and also better facilitate the final grouping selection. Algorithm includes the pseudo code of this strategy, First, we start with MTG-Net trained on one meta sample that corresponds to the last task combination (C2N-1). which includes all N tasks. Next, we conduct K rounds of active selections. For each round, we repeat the same active learning process for all tasks. For each task (T)), we first filter out a candidate set (CT)) that incorporates all combinations including this task. Then for every combination (C.) in CT', we collect the estimated transferring gain (G, ===Yo (T)))) for this task, and we set the (unnormalized) sampling probability as expo-IGal, where is the hyper-parameter that measuring our preferences for remarkable gains (no matter positive or negative). Afterwards, we select a meta sample (Cnezt) following the distribution defined by pT; insert it into the meta-training set, and obtain updated O* jotest by re-running the meta procedure for MTG-Net if meeting given updating interval (7)).
Discussions on Efficiency Algorithm I iincluees O(KN) times MTL procedures, which take up the major computational cost because the meta training in 2 takes negligible time compared with the corresponding MTL procedures to collect ytrain, Besides, another issue that prevents from scaling to large N lies in the exponential dependence of [Ctest on N. In practice, randomly sampling a group of combinations with a large yet affordable size to substitute Moreover, since the encoder part of hard-sharing architecture usually takes up the most computation, the computational cost of N -tak MTL procedures does not have orders of magnitude differences for different N. Thus in this work employ the number of MTL procedures intuitive measure of computational cost. While as an anonymous reviewer mentioned, if each task has its specific data, the real computational time of MTL procedures should be considered carefully.
tackle this problem by
approximate yet
4 Experiments
We conduct extensive experiments to validate the effectiveness and the efficiency of our method across diversified multi-task scenarios, including vision, energy, and healthcare. In this section, we only include basic setups, main experimental results, and part of visualization analyses. Due to the space limit, we leave other details (datasets, task specifications, networks for MTI., haper-pssameters, etc.) and more results (more visualizations, case studies, etc.) to the appendix.

---

4.1 Experimental Setups
MTL Datasets We conduct experiments on three MTL datasets. 1) Taskonomy 49 is a computer
vision dataset including massive indoor scenes. Existing HOA-based methods 41 4 leveraged 5 vision tasks of this dataset to evaluate MTG. We also follow the setups in [41 and denote the preprocessed dataset as Taskonomy-55 2) ETTml [66] is electric load dataset with 7 time series. We follow the multi-variate time-series forecasting problem in [46) and regard the forecasting for each series task. Accordingly, we denote this dataset as ETTm1-7. 3) MIMIC-III (17] is healthcare database with rich electronic health records. Previous studies [30 constructed tens of clinical prediction tasks and found that simply performing MTL on these tasks did not always bring performance gains. We select 27 crucial clinical tasks to build the MIMIC-III-27 dataset.
Transferring Gains & Grouping Selection Section 3.1 introduces how to collect ground-truth transferring gains and how to perform the final grouping selection given transferring gains (or predictions). Here we make some detailed specifications for each MTL dataset. For 5 vision tasks on Taskonomy-5, we follow 44 use the loss value as the metric and thus calculate the transferring gain as the relative reduction of the loss. For 7 forecasting tasks on ETTm1-7, we instantiate the transferring gain as the relative reduction of the mean absolute error. While for 27 clinical prediction tasks on MIMIC-III-27, all of which adopt AUROC as the metric, we define the transferring gain as the relative improvement of AUROC. Note that all these transferring gains are collected on the validation set, and we conduct the MTL procedure for each case with different random seeds. Only after conducting the grouping selection based on transferring gains, we report the performance on the test set for selected groupings. Moreover, we collect all ground-truth transferring gains for 31 task combinations on Taskonomy-5 and 127 task combinations on ETTm1- -7. However, 27 tasks on MIMIC-III-22 produce over one hundred million task combinations, so we randomly sample 3, 000 combinations to represent the whole space. All these MTL procedures cost thousands of GPU hours in total, and we will release the collected meta datasets for future research.
Hyper-parameters of MTG-Net We the same cyper-parameters for all MTL datasets. To be specific. we set the dimension of task embeddings as = 64 and stack self-attention encoding 25 to prioritize the selection of task combinations
layers 45). As for Algorithm| I we set with large gains. Besides, we leverage a dynamic strategy to schedule v). At an early stage when [Ctrasn| <= N + 1, we set 1) as 1 to frequently updating MTG-Net to pursue more effective selections. When (trainn > N + 1, we set as N to reduce the number of updating for MTG-Net to further improve efficiency. K is the hyper-parameter deciding the total number of meta-training samples, which is specified along with each figure. Moreover, we repeat the meta training and inference of Net with five random seeds and report the average results eliminate the effects of randomness.
Baselines We compare MTG-Net with four baselines. 1) Oracle denotes the grouping solution based on all ground-truth transferring gains on the validation set, which forms an upper bound of all approximation methods. 2) Random denotes randomly selecting group of task combinations satisfying a given budget. Since this solution has high variance, we conduct one million random trials and report the average performance, as did in [11. 3) HOA [4] conducted ( MTL procedures on pairwise combinations to collect ground - truth pairwise gains and then estimated all rest hoder gains via the average of corresponding pairwise gains. 4) TAG [14] attempted to improve the efficiency of HOA by obtaining an approximation to the pairwise gains via customized MTL procedure on all tasks, which involved N times extra forward and backward processes per each gradient updating to collect pairwise affinities on the fly.
4.2 Main Experiments
We present our main evaluation results for all three datasets in Figure 2. For each dataset, we include two configurations of K for MTG-Net: K = 1 and K = .N11 As illustrated in Section 3.4 our active learning strategy requires O(KN) times MTL procedures. Therefore, these two types of K roughly correspond to the computational cost required by TAG and HOA, respectively.
In the left side of Figure 2. we show the final performance (on the test set) of selected groupings under different budgets. As the budget increases, meaning we have more combinations for task to choose as the affiliated group, the final performance should also increase for a well-behaved
7

---

20%
Parformance
The Distrbuzion Esimution Erors
The buien Gains
-B-
Drece
HTG--tet (P-2)
A1 Palwiis
Taskonomy
18%
HOA
HTG-Met (P-1)
33
17%
6%
0.1
-oc MITI-MMe Met 22 POA
0.0
(0%,,5%
33%
23
(4%,100) (U%)
-0%% 40%
-0% 22% 10% E. 23% 20% 20% 40% 50%
Budget Performnce vss. Budget
Groundauth Gain
Grrund-tuth Can
28%
The Distiturion stiatiin from
1x0
The Dihh button Gaiss
MTGG Net (C 33+ NTG Ven (.
(C+3)
35%
0.25
HOA
00=1)
3.25
0.20
1.00
0:35
0.30
0.32
0.85
6.25
0.00
(-20%] (00% (
(49.13%)
(138)
www.gehhar wikgg//hattssshhhhhhhhp. Lof gifgof-ggl wo
Budge:
Grordenuhh Gain
Cimuns truth Can The butonn Gains
Perfomance Budget
The Distrbutin Estiation Emoos
Ied
1075
Not (6633)
7.5%
55%
03
3.00
MM
HTC Not (S-13))
0.
Numbarre
0:75
NTC (6wl)
-85
7.55
Ran33m
TAC
0.0
0:01
(10%)-24
Budget
Ground-uuth Gain
CroundAruth Gain
Figure 2: We show the final performance of selected groupings under different budgets (left column), the distribution of gain estimation errors (middle column), and the distribution of ground-truth gains (right column) Taskonomy-5 ETTm1-7, and HIMIC-III-22
MTG solution. However, we observe some fluctuation and even degradation effects. There are two reasons: 1) inaccurate estimations of transferring gains largely misguide the grouping selection, and the grouping errors become much more evident as the budget increases; 2) we use the transferring gains collected on the validation not completely consistent with the actual performance gains on the test set. The latter reason also apply Oracle, but its performance increases with the budget at most time. Thus, we can conclude that the degradation effect influenced by the latter reason is limited, and it is reliable to attribute the final grouping performance to the quality of the estimations for transferring gains.
guide the grouping selection, but these gains are
Furthermore, we can find that HOA and TAG produce poor grouping decisions on ETTm1-7 and MIMIC-III-27, while HOA obtain pretty well grouping performance Taskonomy-5, and TAG can produce reasonable performance when the budget is small. The key reason is that the number of task combinations increases exponentially the number of tasks increases, as result, HOA and TAG fail to provide accurate estimations for massive high-order task combinations solely based on the information of pairwise combinations. As for Taskonomy-S there large transferring gains covered by pairwise combinations, so HOA can obtain reasonable grouping performance. However, since TAG introduces another approximation step predict pairwise affinities, its grouping performance suffers more degradation than HOA as the budget increases. In contrast, we can see that MIG-Net behaves much better on all cases. Specifically, when we set K N-1, which means we only have
the ground-truth gains on N(N-1) + task combinations, MTG-Net is able to produce high-quality groupings that are very close to the ones derived by Oracle. In addition to the grouping performance, also show the estimation errors of transferring gains (calculated on the validation set) on all task combinations in the middle column of Figure 2 and plot the distribution of ground-truth transferring gains in the right side. These auxiliary information aligns with the analyses mentioned above and further reveal the drawbacks of the HOA assumption that maps pairwise gains to high-order ones.
Given the superiority of MTG-Net over existing solutions, we further conduct experiments on the much challenging MIMIC-III-2 benchmark to verify two critical factors that drive the success of MTG-Net. One is the number of meta-training samples. Figure 3al includes the performance comparisons of multiple MTG-Net counterparts with different K. Here we conduct multiple runs for each counterpart and plot the average performance as well as the standard deviation (shadow
8

---

10%
Parformance Budget
Performance
Average Gan
Oracle
Dracle
K=13
Activv Ka13
Random K-13
K-1
Random
5%
13
Budget
Rudger
(a) The impact of K.
(b) Active selection of Cerain v.s. random selection
Figure 3: Different configurations of MTG-Net on MIMIC-III-27.
Tasko nomy- Normal
ETTm3-7 High
ETTm1-7
MIMiC-II1-27 congenital
MIMIC-I11-27
circulatory
201-
1.2
20-
MTGANt,
-102
-100
0
-302
-202
200
203
25
MTG Net. K
102
-002
-200
-132
300
-20
MTG Net. K=
102
102
102
-000
102
-202
-200
-200
230
Figure We select five tasks from different datasets and visualize the latent structures transferring relationships discovered by NTG-Net. For each task, we aggregate its encoded embeddings (from H,) for all relevant task combinations and project them into a swo-dimensional space via t-SNE (44) The upper two rows only include task combinations on Ctrain for two K's. The bottom row includes all combinations covering this task. We color each combination by its ground-truth gain for this task (red: positive; blue: negative; the deeper the color depth, the larger the gain (absolute value)).
area). We can clearly see that adding eeta-training samples brings prominent improvements in the grouping performance for large budgets. While for small budgets, even K can produce reasonably good performance. This observation can help us to balance efficiency and effectiveness flexibly according to specific practical scenarios. Besides, the other crucial factor is the active learning strategy illustrated in Algorithm|] [ Figure 3b includes the ablation study on the importance of active learning. We can observe that matter K is large (13) small (1), the active selection of meta samples produces remarkable improvements over the random selection.
4.3 Visualization Analyses
Figure visualizes some latent structures of transferring relationships discovered by MTG-Net. By inspecting the upper two rows, we can clearly see that MTG-Net discovers more comprehensive latent
structure as the active learning strategy progressively increases [Ctroin] from N + to N(N-1) + 1. From the bottom two rows, we can observe that the latent structure discovered on Ctroin is roughly consistent with the overall structure C. This observation intuitively reveals that the transferring relationships indeed lie in some low-dimensional manifolds, and our active learning strategy is effective selecting those crucial combinations, which can function as the landmarks of latent structure. Moreover, by comparing different latent structures across different tasks, we can observe that no matter how the MTL. scenario and the number of task combinations vary, MTG-Net can still provide reasonable approximation to the underlying latent structure.

---

Conclusion and Future Work
This paper introduces an efficient and effective approach for MTG, which is based on a new meta- learning formulation task combinations. The key takeaway is that matter how large the space of task combinations is, the relationships between task combinations and performance gains lie in some low- diw-dimeniional manifolds. Our experiments across diversified MTL scenarios demonstrate the practicability of this hypothesis. This is why we can make predictions for massive unseen task combinations with only ground-truth gains of this work is to benefit a wide range of real-world applications [6] 19 36 31. in which people have
few actively selected combinations. The direct impact
multiple learning tasks but do not know how to organize them into different groups effectively.
Moreover, we note that there are some valuable future research directions given the contribution of this work, demonstrating that the transferring relationships of predefined MTL procedure across different task combinations could be meta learned. First, when the data or the model architectures changed, the transferring effects among tasks could also distinctly change. Both HOA [41] and us have observed these phenomena, which implies that multi-tast transferring relationships may be function of the data, the model, and some other factors (such as optimization algorithms). Thus how to enable effective meta learning on task combinations across different configurations of MTL procedures is valuable research question be answered. Besides, this work only considers the transferring relationships among fixed set of tasks, while in practice we would always encounter new tasks. Accordingly, how enable incremental meta learning or endow MIG-Nes Net with the zero-shot capability is worthy of great research attention.
References
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta- learning. In ICCV, 2019.
[2] Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zeullemoyer and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In EMNLP, 2021.
[3] Rie Kubota Ando, Tong Zhang, and Peter Bartlett. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 2005.
[4] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, 2016.
[5] Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. Machine learning, 2008.
[6] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In ICLR, 2022.
[7] Jonathan Baxter. A model of inductive bias learning. JAIR, 2000.
[8] Richard Caruana. Multitask learning: A knowledge-basee source of inductive bias. In ICML, 1993.
[9] Richard Caruana. Multitask learning. Machine learning, 1997.
[10] Xiaohan Chen, Siyu Tang, Krikamol Muandet, et al. Mate: Plugging in model awareness to task embedding for meta learning. In NeurIPS, 2020.
[11] Ronan Collobert and Jason Weston. unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008.
10

---

[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICI.R 2021.
Cross-lingual parameter sharing in neural network parser. In ACL, pages 445-850, 2015. [13] Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing:
[14] Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In NeurlPS, 2021.
[15] Pengsheng Guo, Chen- Lee, and Daniel Ulbricht. Learning to branch multi-task learning. In ICML, 2020.
[16] Siyu Huang, Xi Li, Zhi- Qi Cheng, Zhongfei Zhang, and Alexander Hauptmann. GNAS: greedy neural architecture search method for multi aulti-attribute learning, In ACM-MM, 2018.
[17] Alistair EW Johnson, Tom Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 2016.
[18] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 2021.
[19] Dmitry Kalashnikov. Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2 104.08212, 2021.
[20] Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In ICML, 2011.
[21] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, 2015.
[22] Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multi-task learning. In ICML, 2012.
[23] Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta reinforcement learning with task embedding and shared policy. In IJCAI, 2019.
[24] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, Nature, 2015.
[25] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In NeurlPS, 2019.
[26] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018
[27] Mingsheng Long, Zhangjic Cao, Jianmin Wang, and Philip S Yu. Learning multiple tasks with multilinear relationship networks. In NeurIPS, 2017.
[28) Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In CVPR, 2017.
[29] Pingchuan Ma. Tao Du and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task learning. In JCML, 2020.
[30] Mattew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzyeh Ghassemi. comprehensive evaluation of multi-task learning and multi-task pre- pre-training on ehr time-series data. arXiv preprint arXiv:2007.10185 2020.

---

[31] Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzyeh Ghassemi. comprehensive ehr timeseries pre-training benchmark. In CHII, 2021.
[32] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations words and phrases and their compositionality. In NewrIPS, 2013.
[33] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stited networks for multi-task learning. CVPR, 2016.
[34] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, 2017.
[35] Michael Prince. Does active learning work? review of the research. Journal of engineering education, 2004.
[36] Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprint 2015.
[37] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[38] Shiv Kumar Saini, Sunny Dhamnani, Akil Arif Ibrahim, and Prithviraj Chavan. Multiple treatment effect estimation using deep generative model with task embedding. In WWW, 2019.
[39] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In NeurlPS, 2018.
[40] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
[41] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? ICML, 2020.
[42] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. In NeurlPS, 2020.
[43] Sebastian Thrun and Lorien Pratt. Learning learn. Springer Science & Business Media, 2012.
[44] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[46] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In NeurIPS, 2021.
[47] Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In ICLR, 2019.
[48] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurIPS, 2020.
[49] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
[50] Yu Zhang and Qiang Yang. A survey multi-task learning. TKDE, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [Yes] See the extra experiments provided in the appendix.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yHgmcNLX2-", "reviewer": "Reviewer_ZASQ", "summary": "The paper introduces a meta-learning method to learn to predict the transfer gains for each task grouping in a Multi-Task Learning problem. This is based on the assumption that, while the number of task combinations (datapoints in this meta-learning problem) is huge, the data lies in a low-dimensional manifold (manifold hypothesis) and thus can be learned. Since the meta-training set needs to be small (computational cost), the authors also apply active learning to help select the most informative datapoints (groupings) to train.", "questions": "See above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The idea of the paper is clear. The manifold assumption also seems reasonable, which suggest that we can do better than heuristic estimations of the transfer gain (via learning to predict it).\n- The paper obtain good performance when compared with heuristic approaches such as HOA or TAG.\n\n**Weaknesses and Questions**\n- One of my main concerns is the computational cost of MTL procedure. The authors mentioned that they consider $k=1$ and $k=(n-1)/2$ so that the number of MTL combinations is roughly equal to TAG and HOA. However, for example, the combinations in HOA only consist of 2 tasks, so might be much less expensive than a combination considered in MTL-Net. Therefore, the evaluation may still be unfair. We should really consider the total computational cost here, instead of the number of task combinations.\n\n- If my understanding is correct, the task embedding (taken out from an embedding table) is independent of the task data. I wonder if we can additionally pass (part of) the task data to a set embedding network to get the task embedding. It might open the door to generalizing to some unseen (but related) tasks at test time. \n\n- Also I found the indicator of transfer gain to be inconsistent in the experiments. For some experiments it is the loss, for some others it is the AUC or the mean absolute error. I am a bit skeptical about this since this might have been cherry-picked. Why don't we use a consistent category across the experiments? \n\n- The authors only present experiments with a hard sharing case of MTL. It is natural to wonder if the method works for soft-sharing of the parameters. Is it the case that we can predict better the transfer gain for hard sharing only?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Rqdt7d8VRrS", "reviewer": "Reviewer_FUf3", "summary": "This paper researches on the task grouping problem in Multi-task Learning (MTL). Task grouping matters MTL’s performance and has gained much attention recently. This paper proposes a novel method to address the task grouping problem, which formulate  a meta-learning problem for task grouping. The proposed method has achieved superior performance.", "questions": "What’s the performance of the active learning strategy in the early epochs?", "limitations": "The technical contribution of this paper is limited for it just utilizes the well-known meta learning frame-work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n\n1.The proposed method is experimentally verified to be effective on various applications.\n2.The paper has conducted extensive experiments .\n3.The paper is easy to follow.\n\nCons\n\n1.The technical contribution of this paper is limited for it just utilizes the well-known meta learning frame-work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "IRz-3J3c4Xl", "reviewer": "Reviewer_q6pk", "summary": "The paper proposes a method for multi-task grouping, which is to find the best sets of multi-task networks that covers the given target tasks. Specifically, the paper trains a model that predicts performance gains as a function of task combinations. Based on this prediction, best task combinations that maximizes the total performance gain can be obtained for given N tasks and budget B through the grouping selection algorithm. To save the training cost, the paper proposes an active learning algorithm that selects task combinations to use for training based on the predicted performance gains. Experiments show that the proposed method outperforms existing MTG methods and predicts total performance gains reasonably well on three MTL datasets (Taskonomy-5, ETTm1-7, MIMIC-III-27).", "questions": "- It would be interesting to see which task combinations are selected at each step of active learning. How similar they are for each task and do their sizes change over iterations? Also, it would be interesting to see the average train/test error of gain predictions over the iterative training combination sets selection procedure, how it improves and what is the final error.\n- How sensitive the method is to the different design choices of MTG-Net (initialization, number of layers, feature dimension, etc)? Since it is trained using only a small sets of (task combination, accuracy gains) pairs, the MTG-Net design can affect a lot on the prediction accuracy of the performance gains. For example, at the first round of MTG-Net training (k=1 in Algorithm1), the MTG-Net used to sample meta-training set is trained using only one (task combination, accuracy gain)-pair. I am curious how much they affect the results.\n- How does the method work for different hyperparameter choices (D, alpha, eta)? For example, alpha is currently set to 25, which I guess will greedily choose the task combinations with best gain for each task. This makes sense to me because these task combinations with extreme gains likely serve as anchors and help interpolating gains for other combinations. At the same time, selecting diverse combinations could be more sample-efficient when training MTG-Net. How important the current hyperparameter choices are and what happens they are set to other values?\n- It would be interesting to see if task relations are model-agnostic. Will the gain estimation obtained from smaller MTL backbone generalizes to larger MTL backbones? This may reduce the training cost.\n- Which task prediction is selected at test time if that task is included in several task combinations and there are multiple predictions?\n", "limitations": "The authors discussed the limitations in the paper. I do not see potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper proposes an interesting approach for the multi-task grouping problem. Compared to [41] which also takes into account the compute budget constraint, this paper does not consider the compute budget in a strict manner, however, the computational complexity is reduced from O(2^n) to O(n^2).\n\n1. Strengths\n- The key finding of the paper is that the relationships between task combinations and performance gains lie in some low-dimensional space and can be approximated reasonably well by the proposed self-attention encoder architecture. I think this is an original finding and also provides good insights.\n- Unlike existing works which approximated relations of task combinations using pairwise task relations, this paper proposes to predict the task gains as functions of higher order task combinations directly using the self-attention encoder and task representations. I think this design is novel and reasonable.\n- The paper extensive experiments on three datasets, ranging from 5 to 27 tasks. Ground truth performance gains are collected and will be released, which I believe will be useful for future MTL research.\n\n2. Weaknesses\n- I think one weakness is the high computational cost. The method requires multi-task network training for O(NK) task combinations. Though it is much smaller than the number of all possible combinations, however, this could be still expensive especially when the number of tasks is large or datasets are large.\n- Analysis of the effect of some design choices (hyperparameters, MTL model architecture, MTG model design) is missing.\n- As also mentioned in the paper, the prediction accuracy is less reliable with smaller number of meta-training data selection. Since MTG-Net is trained using only a small sets of (task combination, accuracy gains) pairs, the MTG-Net design (initialization, architectures) can affect a lot on the prediction accuracy of the performance gains. For example, at the first round of MTG-Net training (k=1 in Algorithm1), the MTG-Net used to sample meta-training set is trained using only one (task combination, accuracy gain)-pair.\n- It is good that the paper takes into account the budget B, which is defined as the allowed number of task groups. However, the actual resource usage  (e.g. inference time, memory), which are of more interest in practice, may not be proportional to this budget B because the multi-task network for each group may have different resource usages.\n- Experiments look relatively weak in terms of dataset size and task diversity. It would be nice to have experiments for datasets with larger number of data points and tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HQ9wRwGJTHv", "reviewer": "Reviewer_E1on", "summary": "This paper studies the problem of task grouping by using an efficient and effective meta-learning framework. The authors propose to use a meta-learning algorithm, which predicts the per-task performance gains of multi-task learning over single-task learning for any combination, to find optimal task combinations. The key insight is the low-rank hypothesis on task combination space. Based on the hypothesis, the authors develop MTG-Net, which uses an active learning strategy to select meta-learner’s training examples. Experiment results conducted on multiple multitask datasets show significant improvement over state-of-the-art task grouping algorithms. \n\n", "questions": "Q1  Can a similar hypothesis from ENAS[1]  can be used to further save cost by sharing the parameters of the pair of models for measuring gains?\nQ2 How the proposed active learning framework compares with the existing AutoML framework, which models the meta-learner as an RL algorithm?\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper is well written and easy to follow. \n2. The approach of introducing a meta-learner to learn task grouping strategies is novel and reasonable.\n3. Multiple MTL datasets are used to evaluate the proposed method.\n\nWeaknesses\n1. On the efficiency side, even though an active learning algorithm is introduced to reduce the number of samples needed to train the meta-learner, the cost can still be large for training models and measuring the gains. I am wondering if a similar hypothesis from ENAS[1] can be used to further save cost by sharing the parameters of the pair of models for measuring gains.\n2.  There exists some related work [2] in AutoML that also leverages the low-rank hypothesis to learn AutoML algorithms based on task-task relationships. Even though the application is slightly different, the novelty of building meta-learned based on low-rank hypothesis is limited.\n3.  Even though the author compared it with SOTA for task grouping. I am wondering how the proposed active learning framework compares with the existing AutoML framework, which models the meta-learner as an RL algorithm[1].\n\n[1] Pham, Hieu, et al. \"Efficient neural architecture search via parameters sharing.\" International conference on machine learning. PMLR, 2018.\n[2]  Yang, Chengrun, et al. \"OBOE: Collaborative filtering for AutoML model selection.\" Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0LuOi2ObgZ", "reviewer": "Reviewer_QRvu", "summary": "This paper introduces a simple yet effective meta-learning framework for multi-task grouping. The main goal is to avoid computing the performance gains of multi-task learning (MTL) over single-task learning (STL) for all possible task combinations, which is intractable for a large number of tasks. During meta-training, a mapping function between task combinations and performance gains is learned, where the meta-training set is selected using an active learning strategy to keep it small in size while still being effective. During meta-testing, the performance gains of the rest of the task combinations are predicted using the learned mapping function. Finally, the task groupings are selected based on the meta-training (ground-truth) and meta-testing (predicted) performance gains. Experiments on 3 datasets demonstrate the effectiveness of the proposed method.", "questions": "I think this is a rather decent submission with a novel idea and convincing experimental results. Apart from the whopping computational cost of conducting the MTL procedures and perhaps the difficulty to scale to deeper models, I only have one minor suggestion for the authors to improve the paper.\n\n- The proposed active learning strategy assigns sample weights to prioritize task combinations with larger (absolute) performance gains. According to my understanding, here the performance gains are computed using the learned MTG-Net (instead of ground-truth). Is there a rationale behind why doing so (i.e., prioritizing task combinations with larger predicted gains) is more helpful for learning MTG-Net?\n\nI also identify the following typos.\n- Line 196, “work tokens” should be “word tokens”.\n- Line 198, the times sign $\\times$ in the shape of $\\mathbf{X}_i$ is written as an $x$.\n- Line 205, the shape of $H_i$ should be $|C_i| \\times D$.\n- In Figure 1, the STL procedure in the left subfigure, you may change the title to \"for $t^i_j \\in C_i$\" to keep the notation consistent with your text in line 132.\n- In Figure 2, performance vs budget for Taskonomy-5, the oracle line is missing.\n", "limitations": "The authors have pointed out the limitations in lines 353-356.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed framework to learn the mapping between task combinations and performance gains is novel.\n- The contribution is quite significant as based on their experiments, the proposed framework significantly improves over the pairwise non-parametric approaches under similar computational costs. The visualization also helps to reveal why it works.\n- The paper is well-written and very easy to follow.\n- Most of the presentations are clear.\n\nWeaknesses\n- Certain parts of the paper lack clarity (see suggestions below).\n- The idea of selecting task groupings based on performance gains is very costly in nature as it involves a lot of MTL training. Given that the experiments cost “thousands of GPU hours in total” (line 268), the experimental results might be difficult to reproduce.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept