Which Tasks Should Be Learned Together in Multi-task Learning?
Authors: Trevor Standley
Amir R. Zamir
Dawn Chen
Leonidas Guibas
Jitendra Malik
Silvio Savarese
Venue:  (2020)
Keywords: multi-task learning
Computer Vision

Abstract: Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using 'multi-task learning'. This saves computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We systematically study task cooperation and competition and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.


Under review as a conference paper at ICLR 2020
WHICH TASKS SHOULD BE LEARNED TOGETHER IN MULTI-TASK LEARNING?
Anonymous authors Paper under double-blind review
ABSTRACT
Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi- task learning. This saves computation inference time only single network needs to be evaluated. Unfortunately, this often leads to inferior overall perfor- mance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We systematically study task cooperation and competition and propose a framework for assigning tasks to few neural networks such that cooperating tasks are computed by the same neural network, while com- peting tasks are computed by different networks. Our framework offers time- accuracy trade-off and can produce better accuracy using less inference time than not only single large multi-task neural network but also many single-task net- works.
INTRODUCTION
Many applications, especially robotics and autonomous vehicles, are chiefly interested in using multi-task learning to reduce the inference time and computational complexity required to estimate many characteristics of visual input. For example, an autonomous vehicle may need to detect the location of pedestrians, determine per-pixel depth, and predict objects' trajectories, all within tens of milliseconds. In multi-task learning, multiple tasks are solved at the same time, typically with a single neural network. In addition reduced inference time, solving set of tasks jointly rather than independently can, in theory, have other benefits such as improved prediction accuracy, increased data efficiency, and reduced training time.
Unfortunately, the quality of predictions are often observed to suffer when a network is tasked with making multiple predictions. This because learning objectives can have complex and unknown dynamics and may compete. In fact, multi-task performance can suffer so much that smaller inde- pendent networks are often superior (as we will see in the experiments section). We refer any situation in which the competing priorities of the network cause poor task performance as crosstalk.
On the other hand, when task objectives do not interfere much with each other, performance on both tasks can be maintained or even improved when jointly trained. Intuitively, this loss or gain of quality seems to depend on the relationship between the jointly trained tasks.
Prior work has studied the relationship between tasks for transfer learning (Zamir et al. (2118). However, we find that transfer relationships are not highly predictive of multi-task relationships. In addition to studying multi-task relationships, we attempt to determine how to produce good pre- diction accuracy under a limited inference time budget by assigning competing tasks to separate networks and cooperating tasks to the same network.
More concretely, this leads to the following problem: Given a set tasks, T, and a computational budget b (e.g., maximum allowable inference time), what is the optimal way to assign tasks to networks with combined cost < such that a combined measure of task performances is maximized?
To this end, we develop computational framework for choosing the best tasks to group together in order to have a small number of separate deep neural networks that completely cover the task set and that maximize task performance under given computational budget. We make the intriguing

---

Under review as a conference paper at ICLR 2020
Semante Segmentation
Depih Eximation
Surface No
Discarded
Prediction
Surfaco Normal
Discarded
Prediztion
Desection
Diraction
Suriace
Prediction
Recodar
Decoder
Dacandes
Decarned
Decoder
Decoder
Deooder
Shane Rap Iigrosmaaion
Stanel radiianii
Encoder
Encoder
Encodar
Sized
B
Input Image
Figure 1: Given five tasks to solve, there are many ways that they be split into task groups for multi- task learning. How do we find best one? We propose a computational framework that, for instance, suggests the following grouping to achieve the lowest total loss, using a computational budget of 2.5 units: train network A to solve Semantic Segmentation, Depth Estimation. and Surface Normal Prediction: train network B solve Keypoint Detection, Edge Detection, and Surface Normal Prediction; train network C with a less computationally expensive encoder to solve Surface Normal Prediction alone; including Surface Normals as an output in the first two networks were found advantageous for improving the other outputs, while the best Normals were predicted by the third network. This task grouping outperforms all other feasible ones, including learning all five tasks in one large network using five dedicated smaller networks.
observation that the inclusion of an additional task in a network can potentially improve the accuracy of the other tasks, even though the performance of the added task might be poor. This can be viewed as regularizing guiding the loss of one task by adding curriculum learning or network regularization Bengio et al. (2009). Achieving this, of course, de- pends picking the proper regularizing task our system can take advantage of this phenomenon, as schematically shown in Figure [
additional loss, as often employed in
outline framework for systematically assigning tasks to networks in order to achieve the best total prediction accuracy with a limited inference-time budget. We then analyze the resulting accuracy and show that selecting the best assignment of tasks to groups is critical for good performance. Secondly, in Section 6 we analyze situations in which multi-task learning helps and when doesn't, t, quantify the compatibilities of var- ious task combinations for multi-task learning, compare them to the transfer learning task affinities, and discuss the implications. Moreover, we analyze the factors that influence multi-task affinities.
This paper has main contributions. In Section
PRIOR WORK
Multi-Task Learning: See Ruder (2017) > for good overview of multi-task learning. The authors identify two clusters of contemporary techniques that we believe cover the space well, hard param- eter sharing and soft parameter sharing. In brief, the primary difference between the majority of the existing works and our study is that we wish to understand the relationships between tasks and find compatible groupings of tasks for any given set of tasks, rather than designing neural network architecture to solve a particular fixed set of tasks well.
A known contemporary example of hard parameter sharing in computer vision is UberNet (Kokki- nos] (2017)). The authors tackle 7 computer vision problems using hard parameter sharing. The authors focus on reducing the computational cost of training for hard parameter sharing, but expe- rience a rapid degradation in performance as more tasks are added to the network. Hard parameter sharing is also used many other works such as (Thrun] (1996); Caruana (1997); Nekrasov et al. (2018) Dvornik et al. (2017): Kendall et al.] (2018 8); Bilen & Vedaldi (2016); Pentina & Lampert 2017 ; Doersch Zisserman 2017);Zamin et al. (2016); Long et al. (2017): Mercier et al. (2018);
d. Miranda et al. (2012); Zhou et al. (2018); Rudd et al. (2016)).
Other works, such as (Sener & Koltun (208)) and (Chen et al. (2018b)), aim to dynamically re- weight each task's loss during training. The former work finds weights that provably lead to Pareto-optimal solution, while the latter attempts to find weights that balance the influence of each task on network weights. Finally, (Bingel & Søgaard (2017)) studies task interaction for NLP.
In soft or partial parameter sharing, either there is a separate set of parameters per task, or signif- icant fraction of the parameters unshared. The models are tied together either by information sharing or by requiring parameters to be similar. Examples include (Dai et al. (2016): Duong et al.
2015): Misra et al. (2016): Tessler al. (2017); Yang & Hospedales 2017) et al. 2017)).
2

---

Under review as a conference paper at ICLR 2020
The canonical example of soft parameter sharing can be seen in (Duong al. (2015) The authors are interested in designing a deep dependency parser for languages such as Irish that do not have much treebank data available. They tie the weights of two networks together by adding an L2
distance penalty between corresponding weights and show substantial improvement.
Another example of soft parameter sharing is Cross-stitch Networks (Misra al. (2166). Starting with separate networks for two tasks, the authors add 'cross-stitch units' between them, which allow each network peek the other network's hidden layers. This approach reduces but does not eliminate task interfearence, and the overall performance is less sensitive to the relative loss weights.
Unlike our method, none of the aforementioned works attempt to discover good groups of tasks to train together. Also, soft parameter sharing does not reduce inference time, major goal of ours.
Transfer Learning: Transfer learning (Pratt ); Helleputte & Dupont 2009); Silver Bennett
2008);
Finn et al. 2016);J0 Mihalkova et al. (2007); Niculescu-Mizii & Caruana (2007): Luo et al.
2017 Razavian et al.] (2014); Pan & Yang (2010); Mallya & Lazebnik (2018); Fernando et al.
2017
Rusu et al. (2016) is similar to multi task learning in that solutions are learned for multiple
tasks. Unlike multi-task learning, however, transfer learning methods often assume that a model for source task is given and then adapt that model to target task. Transfer learning methods generally neither seek any benefit for source tasks nor reduction in inference time as their main objective.
Neural Architecture Search (NAS): Many recent works search the space of deep learning architec-
tures to find ones that perform well (Zoph & Le 2017 Liu et al. 2018 Pham et al. 2018 Xie et al., 2019 Elsken et al. 2019 Zhou et al.] 2019; Baker et al. [011]] Real et al. 2018). This related to our work as we search the space of task groupings. Just as with NAS, the found task groupings often perform better than human-engineeree ones.
Our work is most related to Taskonomy (Zamir et al. (2018), ), where the au- thors studied the relationships between visual tasks for transfer learning and introduced dataset with over 4 million images and corresponding labels for 26 tasks. This was followed by a number of recent works, which further analyzed task relationships (Pal & Balasubramanian (2019); Dwivedi & Roig. (oigg((011)) 2 Achille et al. (019)); Wang et al. (2019)) for transfer learning. While they extract rela- tionships between these tasks for transfer learning. we are interested in the multi-tas learning set- ting. Interestingly, we find notable differences between transfer task affinity and mulli--ask affinity. Their method also differs in that they are interested in labeled-data efficiency and not inference- -ime efficiency. Finally, the transfer quantification approach taken by Taskonomy (readout functions) is only capable of finding relationships between the high-level bottleneck representations developed for each task, whereas structural similarities between tasks at all levels are potentially relevant for multi-task learning.
Task Relationships:
TASK GROUPING FRAMEWORK
Our goal is to find an assignment of tasks networks that results in the best overall loss. Our strategy is to select from a large set of candidate networks to include in our final solution.
We define the problem as follows: We want to minimize the overall loss on a of tasks T {41, l2.,.., { given a limited inference time budget, b, which is the total amount of time we have to complete all tasks. Each neural network that solves some subset of T and that could potentially be a part of the final solution is denoted by n. It has an associated inference time cost, Cn. and a loss for each task, L(n, 42) (which is oo for each task the network does not attempt to solve). A solution S is set of networks that together solve all tasks. The computational cost of solution cost(S) (S) Enes Ox- The loss of solution on a task, L(S,t), the lowest loss on that task among the solution's 's networks]] C(S,t)) = minnes L(n, t.). The overall performance for a solution is L(S) Σn,T (1)
We want to find the solution with the lowest overall loss and a cost that is under our budget, S, = argmin Sooatt L(S).
In principle, it may be possible to create even hetter-performing ensemble when multiple networks solve the same task, though we do not explore this.
3

---

than tasks will perform based Under review as a conference paper at ICLR 2020
3.1 WHICH CANDIDATE NETWORKS TO CONSIDER?
For given task set T, wish determine not just how well each pair of tasks performs when trained together, but also how well each combination of tasks performs together so that we can capture higher-order task relationships. To that end, our candidate set of networks contains all 2/ possible groupings: (IT) networks with one task. (22 networks with two tasks. (1z) networks with three tasks, etc. For the five tasks we use in our experiments, this is 31 networks, of which five are single-task networks.
The size of the networks is another design choice, and to somewhat explore its effects we also include single task networks each with half of the computational cost of a standard network. This brings our total up to 36 networks.
3.2 NETWORK SELECTION
Consider the situation in which we have an initial candidate set Co {m1, f2, Rm of fully- trained networks that each solve some subset of our task set T. Our goal is to choose subset of Co that solve all the tasks with total inference time under budget and the lowest overall loss. More formally, want to find solution S,
[egminngCCrroot(5555 ((S).
It can be shown that solving this problem is NP-hard in general (reduction from SET-COVER). How- ever, many techniques exist that can optimally solve most reasonably-sized instances of problems like these in acceptable amounts of time. All of these techniques produce the same solutions. We chose to use a baanch-and-bound-lik algorithm for finding our optimal solutions (shown as Al- gorithm [ in the Appendix), but in principle the exact same solutions could be achieved by other optimization methods, such as encoding the problem as binary integer program (BIP) and solving it in a way similar to Taskonomy (Zamir al. (2018)).
Most contemporary MTL works use fewer than unique task types, but in principal, the NP-hard nature of the optimization problem does limit the number of candidate solutions that can be consid- like approach requires
ered. However, using synthetic inputs, we found that our branch-and- less time than network training for all 2/77 1 + [T| candidates for fewer than ten tasks. Scaling beyond that would require approximations stronger optimization techniques.
3.3 APPROXIMATIONS FOR REDUCING TRAINING TIME COMPLEXITY
This section describes two techniques for reducing the training time required to obtain a collection of networks input to the network selection algorithm. Our goal is to produce task groupings with results similar to the ones produced by the complete search, but with less training time burden. Both techniques involve predicting the performance of network without actually training it to convergence. The first technique involves training each of the networks for a short amount of time, and the second involves inferring how networks trained on how networks trained on two tasks perform.
3.3.1 EARLY STOPPING PRIOR TO CONVERGENCE
We found a moderately high correlation (Pearson's 0.49) between the validation loss of our neural networks after pass through just 20% of our data and the final test loss of the fully trained networks. This implies that the task relationship trends stabilize early. We fine that we can get decent results by running network selection on the lightly trained networks, and then simply training the chosen networks to convergence.
For our setup, this technique reduces the training time burden by about 20x over fully training all candiate networks and would require fewer than 150 GPU hours to execute. This is only 35% training-time overhead. Obviously, this technique does come with prediction accuracy penalty. Because the correlation between early network performance and final network performance is not perfect, the decisions made by network selection works are trained to convergence. We call this approximation the Early Stopping Approximation (ESA) and present the results of using this technique in Section
longer guaranteed to be optimal once net-
4

---

Under review as a conference paper at ICLR 2020
3.3.2 PREDICT HIGHER- FROM LOWER- -EEEER
Do the performances of a network trained with tasks and B, another trained with tasks and C, and third trained with tasks B and C' tell us anything about the performance of network trained on tasks A, B, and C? As it turns out, the answer is yes. Although this ignores complex task interactions and nonlinearities, simple average of the first-order networks' accuracies was good indicator of the accuracy of a higher-order network. Experimentally, this prediction strategy has an average max ratio error of only 5.2% our candidate networks.
Using this strategy, we can predict the performance of all networks with three or more tasks using the performance of all of the fully trained two task networks. First, simply train all networks with two or fewer tasks to convergence. Then predict the performance of higher-order networks. Finally, run network selection on both groups.
With our setup (see Section A. this strategy saves training time by only about 50%, compared with 95% for the early stopping approximation, and it still comes with a prediction quality penalty. However, this technique requires only quadratic number of networks to be trained rather than an exponential number, and would therefore win out when the number of tasks is large.
We call this strategy the Higher Order Approximation (HOA), and present its results in Section 5
EXPERIMENTAL SETUP
We perform our evaluation using the Taskonomy dataset (Zamir et al. (2018). which is currently the largest multi-task dataset in vision with diverse tasks. The data was obtained from 3D scans of about 600 buildings. There are 4,076,375 examples, which we divided into 3,974, 199 training instances, 52,000 validation instances, and 50,176 test instances. There was no overlap in the buildings that appeared in the training and test sets. All data labels were normalized (? = 0, a = 1).
Our framework is agnostic to the particular set of tasks. We have chosen to perform the study using five tasks in Taskonomy: Semantic Segmentation, Depth Estimation, Surface Normal Prediction, Keypoint Detection, and Edge Detection, so that one semantic task, two 3D tasks, and two 2D tasks are included. These tasks were chosen to be representative of major task categories, but also to have enough overlap in order to test the hypothesis that similar tasks will train well together. Cross- entropy loss was used for Semantic Segmentation, while an L1 loss was used for all other tasks.
Network Architecture: The proposed framework can work with any network architecture. In our experiments, all of the networks used standard encoder-decoder architecture with modified Xcep- tion (Chollet (2017)) encoder. Our choice of architccture is not critical and was chosen for reason- ably fast inference time performance. The Xception network encoder was simplified to have 17 layers and the middle flow layers were reduced to having 512 rather than 728 channels. All max- pooling layers were replaced by 2 x 2 convolution layers with stride of 2 (similar to Chen et al. (2018))) ). The full-size encoder had about 4 million parameters. All networks had an input image size of 256x256. We measure inference time in units of the time taken to do inference for one of our full-size encoders. We call this a Standard Network Time (SNT). This corresponds to 2.28 billion multiply-adds and about ms/image on a single Nvidia RTX 2080 Ti.
Our decoders were designed to be lightweight and have four transposed convolutional layers (Noh al. (2015) and four separable convolutional layers (Chollet (2017)) Every decoder has about 116,000 parameters. All training was done using PyTorch (Paszke et al. (2017)) with Apex for fp16 acceleration (Micikevicius et al. (2017).
Trained Networks: As described in Section 3.1 we trained 31 networks with full sized encoders and standard decoders. 26 were multi-task networks and 5 were single task networks. Another five single-task networks were trained, each having half-size encoder and a standard decoder. These 36 networks were included in network optimization as Co. 20 smaller, single-task networks of various sizes were also trained to be used in the baselines and the analysis of Section 6 but not used for network selection. In order to produce our smaller models, we shrunk the number of channels in every layer of the encoder such that it had the appropriate number of parameters and flops.
The training loss we used was the unweighted mean of the losses for the included tasks. Networks were trained with initial learning rate of 0.2, which was reduced by half every time the training
5

---

Under review as a conference paper at ICLR 2020
loss stopped decreasing. Networks were trained until their validation loss stopped improving, typ- ically requiring only 4-8 passes through the dataset. The network with the highest validation loss (checked after each epoch of 20% of our data) was saved.
The performance scores used for network selection were calculated on the validation set. We com- puted solutions for inference time budgets from to increments of 0.5. Each solution chosen was evaluated on the test set.
4.1 BASELINES
We compare our results with conventional methods, such as five single-task networks and a single network with all tasks trained jointly.
We also compare with two multi-task methods in the literature. The first one is Sener & Koltun (2018). We found that their algorithm under-weighted the Semantic Segmentation task too aggres- sively, leading to poor performance on the task and poor performance overall compared to a simple sum of task losses. We speculate that this is because semantic segmentation" loss behaves differ- ently from the other losses. Next we compared to GradNorm (Chen et al. (2018b)). The results here were also slightly worse than classical MTI. with uniform task weights. In any event, these techniques are orthogonal to ours and can be used in conjunction for situations in which they lead to better solutions than simply summing losses.
Finally, we compare results to two control baselines illustrative of the importance of making good choices about which tasks to train together, 'Random' and 'Pessimal." 'Random' is solution consisting of valid random task groupings that solve five tasks. The reported values average thousand random trials. 'Pessimal' is a solution in which we choose the networks that lead to the worst overall performance, though the solution's 's performance each task is still the best among its networks.
the
Each baseline was evaluated with multiple encoder sizes that all models' results could be com- pared at many inference time budgets.
5 TASK GROUPING EVALUATION
Optimal Grouping
Early Stopping Approximation
Higher Order Approximation
Pessimal Grouping
Figure 2: The task groups picked by each of our techniques for integer budgets between 1 and 5, Networks are shown as O (full-size) o thalf-size). Networks are connected to the tasks for which they compute predictions. s: Semantic Segmentation, d:
Depth Estimation, n: Surface Normal Prediction, k: Keypoint Detection, e: Edge Detection. Dotted edges represent unused decoders. For example, the highlighted solution consists of two half-size net- works and full-size network. work solves Depth Estimation, Surface Normal Pre- diction, and Keypoint Detection. One half-size net- work solves Semantic Segmentation and the other solves Edge Detection. The total loss for all five tasks is 0.455. The groupings for fractional budgets are shown in the appendix.
Inference Time Budget
SNT
Tetel Lews a G-too
Total LonN BAAR
Tetnl Leess 0LANS
Teaa Lood 0.500
V Tetel Lows CAS
full-size net-
Teli 0A5
Tetal Lown 0.49
Tad Looe 0.500
I
SNT
Figure 2 shows the task groups that were chosen for each technique, and Figure 3 shows the per- formance of these groups along with those of our baselines. We can see that each of our methods outperforms our traditional baselines for every computational budget.
When the computational budget is only SNT, all our methods must select the same model-a traditional multi-task network with SNT encoder and five decoders. This strategy outperforms GradNorm, [Sener& Koltun (2018), and individual training. However, solutions that utilize multiple networks outperform this traditional strategy for every budget > 1.5-better performance can always be achieved by grouping tasks according to their compatibility.

---

Under review as a conference paper at ICLR 2020
Plant
Bed
Tablle
Figure 4: Qualitative results for our baselines (left) and our techniques (right). All solutions allowed 2.5 SNT.
When the computational budget is ef- fectively unlimited (5 SNT), our optimal method picks five networks, each of which is used to make predictions for a sep-
Performance vs Compute
Total Loss (lower is better)
Mora: Natwar Chaice
0.525
Vulrisa
arate task. However, three of the net- works are trained with three tasks each,
Ra sdee tuck
Tive hndepadent
while only two are trained with one task
0.475
Netwesk
each. This shows that the representations learned through multi-task learning were found to be best for three of our tasks (s. d, and e), whereas two of our tasks (n and k) are best solved individually.
55A (1
0.425
Optimal Newwrk
1
1.5
2
2.5
3
3.5
4.5
5
Inference Time Cost
Figure: 3: The performance/inferrnce time trade-off for var- ious methods. Data presented tabularly in Table
We also see that our optimal technique us- ing 2.5 SNT and our Higher Order Ap- proximation using 3.5 SNT can both outperform five individual networks (which uses 5 SNT).
In order to determine how these task groupings generalize to other architectures, we retrained best solution for 3 SNT using resnet18 (He et al. (2016)). The results in Table [ suggest that good task groupings for one architecture are likely to be good in another, though to lesser extent. Task affinities seem to be some- what architecture-dependeer
resnet18 8
Total Loss
All-in-one (triple-size resnet18)
0.50925
Five Individual (resnet18s .6-size each)
0.53484
nKE, SDn, N (3 standard resnell8's) s)
0.50658
Table I:
The performance of our best 3 SNT solution found using Xception but evaluated on ResNet18.
so for the very best results, task selection must be run for each architecture choice.
Figure 4 allows qualitative comparison between our methods and our baselines. We can see clear visual issues with each of our baselines that are not present in our methods. Both of our approximate methods produce predictions similar to the optimal task grouping.
6 ANALYSES OF TASK RELATIONSHIPS
The data generated by the above evaluation presents an opportunity analyze how tasks interact in multi-task setting, and allows to compare with some of the vast body of research in transfer learning, such as Taskonomy (Zamir et al. (2018).

---

Under review as a conference paper at ICLR 2020
Relative Performance On
SemSeg
Depth
Normals
Keypoints
Edges
Average
Traned With
SomSog
-5.41%
11 29%
-4.32%
-34,64%%
-13.92%
Depth
4.17%
-3.55%
3.49%
3.76%
1.97%
Normals
8.50%
2.48%
1.37%
12.33%
6.17%
Keypoints
4.82%
1.38%
-0.02%
-5.26%
0,23%
Fddees
1.37%
Average
5.14%
-0.62%
-4.82%
0.48%
-5.95%
-1.15%
Table 2: The first-order multi-task learning relationships between tasks. The table lists the performance every task trained pair with every other task. For instance, when Depth trained with SemSeg. SemSeg performs 4.17% better than when SemSeg is trained alone on a half-size network.
Depth
Normals Keypoints
Edges
Depth Normals Keypoints
Edges
SemSeg
-0.62%
-1.9%
0.25%
15.78%
SemSeg
1.740
1.828
0.723
0.700
Depth
-0.54%
2.43%
1.42%
Depth
1.915
0.406
0.468
Normals
0.67%
3.95%
Normals
0.089
0.1
Keypoints
-1.95%
Keypoints
0.232
Table 3: The multi-task learning affinity between pairs of tasks. These values show the average change in the performance of two tasks when trained as pair, relative when they are trained separately.
The transfer learning affinities between pairs of tasks according to the authors of Taskonomy (aaiir et al. (2018)). Forward and backward transfer afhnitics are averaged.
Table 4:
Multi-Ta Affinity vs Transfer Affinity 4.00%
In order to determine the between task affin- ity for multi-task learning, we took the average of our first-order relationships matrix (Table 2} and its transpose. The result is shown in Ta-
Multi-Tash Miii--askk Affinity
ble 3, The pair with the highest affinity by this metric are Surface Normal Prediction and 2D Edge Detection. Our two 3D tasks, Depth Esti-
0.00%
mation and Surface Normal Prediction, do not score highly on this similarity metric. This con- trasts with the findings for transfer learning in Taskonomy (Table 4 in which they have the highest affinity. Our two 2D tasks also do not score highly. We speculate that the Normals task naturally preserves edges, while Depth and Normals (for example) don't 't add much training signal to each other. See Section A.3 for more on factors that influence multi-task affinity.
-2.00%
Transfer Affinity
Figure Task affinities multi-task learning vs. transfer learning. The correlation (Pearson's 's r) is 0.54, 0.13. One outlier is removed.
Figure 5 depicts the relationship between transfer learning affinities and multi-task affinities, which surprisingly seem be negatively correlated in our high-data scenario. This suggests that it might be better train dissimilar tasks together. This could be because dissimilar tasks able provide stronger and more meaningful regularization. More research is necessary to discover when and this correlation and explanation hold.
7 CONCLUSION
We describe the problem of task compatibility as it pertains to multi-task learning. We provide an algorithm and computational framework for determining which tasks should be trained jointly and which tasks should be trained separately. Our solution can take advantage of situations in which joint training is beneficial to some tasks but not others in the same group. For many use cases, this framework is sufficient, but it can be costly at training time. Hence, we offer two strategies for coping with this issue and evaluate their performance. Our methods outperform single-task networks, multi-task network with all tasks trained jointly, well other baselines. Finally, use this opportunity to analyze how particular tasks interact in a multi-tase setting and compare that with previous results transfer learning task interactions.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pictro Perona. Tesk Task Embedding for Meta-Learning. arXiv e-prints, art. arXiv: 1902.00355 February 2019.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network archi- tectures using reinforcement learning. International Conference on Learning Representations, 2017.
Yoshua Bengio, Jerôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009.
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 235- 243. Curran Associates, Inc., 2016. URL http:://appee: nips cttps.ccc/aaprr/
6393 integrated perception inteppaotdd-rrrpptioo with recurrent wwaprrenttuuttti multi task neural newsrr-etttotoosss
pdf
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 164-169, Valencia, Spain, April A0sociasooittion for Computational Linguistics. URLhttps: www aclweb.
org/anthol 2ttpenthhhoogy/11722222222
Rich Caruana. Multitask learning. Machine Learning, 28)))11-1)))1111111111 15 1997, ISSN 1573-0565, doi: 10.1023/A410073379007744 /ttp://////////...oo./////////10
Liang-Chich Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Computer Vision ECCV 2018- 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Part VII, pp. 833 851, 201 8a. doi: 1000000977830000-0120000 org 10 100.1100//90 10010000077000000000-----------0
.49. URL https://doi.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, pp. 793-802, 2018b. Utt/http::/prroceeiiii v8C/chent 8a. html
p.rsss/
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
B. C. d. Miranda, multi-objective optimization approach with meta-learning for svm parameter selection. In 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2909-2914, Oct 2012. doi: 10. LLL.110C..2222126663333333
PrudIncio, A. C. P. L. F. Carvalho, and Soares. Combining
Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3150- 3158, 2016.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In International Conference on Computer Vision, 2017.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 845-850, July 2015.

---

Under review as a conference paper at ICLR 2020
Nikita Dvomik, Konstantin Shmelkov, Julien Mairal, and Cordelia Schmid. BlitzNet: A real-time deep network for scene understanding. In IEEE International Conference on Computer Vision (ICCV), 2017.
K. Dwivedi and G. Roig. Representation similarity analysis for efficient task taxonomy and transfer learning. In CVPR. IEEE Computer Society, 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi- neural architecture search via lamarckian evolution. International Conference on Learning Representations, 2019. URL
net/ forum? id=ByME42AqK7
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, 0os//701108734, 2017. URL
08734
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pp. 512-519. IEEE, 2016.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1109/CVPR.2016.999
Thibault Helleputte and Pierre Dupont. Feature selection by transfer learning with linear regularized models. In Wray Buntine, Marko Grobelnik, Dunja Mladenić, and John Shawe- Taylor (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 533-547, Berlin, Heidelherg, 2009. Springer Berlin Heidelberg. ISBN 978-3-642-041800
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp.
2017.
doi: 10. 10.1109/CVPR.2217..777 URL https:: https://ddci.ooo orc.ooorg110.. http://021..00111111101001
CVPR. 2017.51
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018.
Mingsheng Long, ZHANGJE CAO, Jianmin Wang, and Philip S Yu. Learning multiple tasks with multilinear relationship networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1594-1603, Curran Associates, Inc., 2017. URL http: httt:::/paaera...m http://ppprs.iii..ccoppp www.: pdf
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogério Schmidt Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classi- fication. In CVPR, pp. 1141-1140. IEEE Computer Society, 2017.
Zelun Luo, Yuliang Zou, Judy Hoffman, and Li F Fei-Fei. Label efficient learning of transferable representations acrosss domains and tasks. In Advances in Neural Information Processing Sys- tems, pp. 164-176, 2017.
Arun Mallya and Svetlana Lazebnik. Piggyback: Adding multiple tasks single, fixed network by learning to mask. CoRR, abs/1881.066111 2018. URL / wwwpvvooo//bb//.../00
06519 g]
10

---

Under review as a conference paper at ICLR 2020
Quentin Mercier, Fabrice Poirion, and Jean-Antoine Dsidri. A stochastic multiple gradi- ent descent algorithm. European Journal of Operational Research, 271(3):808 817, 2018. doi: 10.1016/j.joo.20188000.0 v27
URL
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training, arXiv preprint 170.0777,0 2017.
Lilyana Mihalkova, Tuyen Huynh, and Raymond Mooney. Mapping and revising markov logic networks for transfer learning. In AAAI, volume pp. 608-614, 2007.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hehert. Cross-stitch networks for multi-task learning. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 3994-4003, 2016.
Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, and Ian D. Reid, Real-time joint semantic segmentation and depth estimation using asymmetric annotations. CoRR, (bs//809.04766. 2018. URL http: aawiv. org/ abs abs/1809 04766
Alexandru Niculescu-Mizil and Rich Caruana. Inductive transfer for bayesian network structure learning. In Artificial Intelligence and Statistics, pp. 339-346, 2007.
H. Noh, S. Hong, and B. Han. Learning deconvolution network for semantic segmentation. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1520-1528, Dec 2015. doi: 10. 1.11099/C.22015..720
Arghya Pal and Vineeth N Balasubramanian. Zero-shot task transfer, 2019.
S. J. Pan and Q Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22110):3355110
Oct 2010. ISSN 1041-4347. doi: 10. 10.1109/KDEE2009.100
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-WW W, 2017.
Anastasia Pentina and Christoph H. Lampert. Multi-task learning with labeled and unlabeled tasks.
In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2807-2816, International Convention Centre, Sydney, Australia, 06-11 11 Aug 2017. PMLR. URL http:// proceedings mlr. wwwpsppesssppppppt httpa/pppeeee//o/eeeeeaaaaaaaaaa ht.ml
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architec- war.emmrrr..
ture search via parameters sharing. 2018. URL
phaml Ba/ Baa.aallppppmaamm 8aa. ppd.
L. Pratt. Discriminabilit--asdee transfer between neural networks. In J. Hanson, J. D. Cowan, and C. I.. Giles (eds.), Advances in Neural Information Processing Systems 5, pp. 204-20 211. Morgan-Kann Kaufaann, 1993. URL http::/ pttpp//eas..ippssspppppppeeee
641 discriminabiln
based transfer betweetwwwwee............e..ee.e newral hew.:eeeeeeeeeeeeeeeeeeeeeeeeeee neural--etworss.p
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn fea- tures off-the-shelf: An astounding baseline for recognition. In Proceedings of the 2014 IEEE Conference Computer Vision and Pattern Recognition Workshops, CVPRW '14 pp. 512- 519, Washington, DC, USA. 2014. IEEE Computer Society. ISBN 978-1-4799-430-11 doi: N0.1109/CVPR.22114.333 URL http://ox htt.:/xxx.i...... org www..sstaaffooo.o www.ssaa.....do.o www. orgg/ 10. 1109/C9/ 2014.131
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. 2018. URL
wwwpaxwwwwwwog///////////// 548. .ddf

---

Under review as a conference paper at ICLR 2020
Ethan M. Rudd, Manuel Günther, and Terrance E. Boult. MOON: A mixed objective optimization network for the recognition of facial attributes. In ECCV (5), volume 9909 Lecture Notes in Computer Science, pp. 19-35. Springer, 2016.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. abs/1706.05098, 2017. |atp::://axxx abs 706. 05098
CoRR,
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/ 111166666.667,, 2016. http:://xxx/... 1ttp///xx....rr/a////666.66... 04671
abs/1606.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimiza- tion. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 525-536. Curran Associates, Inc., 2018. URL http::/ nips hwwp.cc/ppps... pdf
Daniel L Silver and Kristin P Bennett. Guest editors introduction: special issue on inductive transfer learning. Machine Learning, 73(3):215-220, 2008
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierar- chical approach to lifelong learning in minecraft. In Proceedings of the Thirty- First AAAI Con- ference on Artificial Intelligence, AAAI17, pp. 1553-1561. AAAI Press, 2017. URL http:// d1 acm. erg. stanford...........mmmmo. cm. oc 1 c org/ ei tat on cmfm d= 3298465
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In Advances in Neural Information Processing Systems, pp. 640-646, The MIT Press, 1996.
Aria Y Wang. Leila Wehbe, and Michael Tarr. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. BioRxiv, pp. 708016, 2019.
Peng Wang, Xiaohui Shen, Bryan Russell, Scott Cohen, Brian Price, and Alan L Yuille. Surge: Surface regularized geometry estimation from single image. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 172-180. Curran Associates, Inc., 2016. URL http: nips. cc/paperrcppppp/ 6502 hrw.ee/eww.............a.aaaaaaa pdf
image.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference Learning Representations, 2019. URL https://
openreview wwtpeviiwwwweeeeeeeeeeeeeeeeeeee wttee//ooonnnddyylleeeeee
Yongxin Yang and Timothy Hospedales. Trace norm regularised deep multi-task learning. In 5th International Conference on Learning Representations Workshop, 2017.
Amir R. Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, and Silvio Savarese. Generic 3d representation via pose estimation and matching. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 535-553, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46487-9.
Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattern- affinitvee propagation across depth, surface normal and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
D. Zhou, J. Wang, B. Jiang, H. Guo, and Y. Li. Multi-task milti-view learning based on cooperative multi-objective optimization. IEEE Access, 6:99475-19477, 2018. ISSN 2169-3536. doi: 10. 1109/ACCESS.201.227778
12

---

Under review as a conference paper at ICLR 2020
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. BayesNAS: A Bayesian approach for neural architecture search, In Kamalika Chaudhuri and Ruslan Salakbutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7603-7613, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL. http::/proceeet ngs m1 www.s w0tp://v9///oo1111 9e ht.1
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. 2017. URL
ht.tps:/arxiv.com wwwp:iiivvrrgggaae http:///aaaxi..orrra/////11111..
A APPENDIX
A.1 NETWORK SELECTION ALGORITHM
Algorithm Get Best Networks
Input:
Cr, rumning set of candidate networks, each with associated cost E R and perfor- mance score for each task the network solves. Initially, C, = Co
Input: S, Co. running solution, initially Ø
Input: b, E IR, the remaining time budget, initially
1: function GETBESSNNTTORRSSCC,C 6,)
2:
C, - FILLEERCC,, S,, b,) SORT(C,,, Best +- S,
3:
> Most promising networks first
4:
5:
for n E C, do
6:
C, + C, \ S, S,
> is set subtraction.
7:
8:
b.tbb b, -
9:
Child GETBESTNETWORKSCC. b,
10:
Best - BETTER(Best. Child)
11:
return Best
12:
function FILTER Sr. b,)
13:
Remove networks from C, with en > by.
14:
Remove networks from C, that cannot improve S,'s performance any task.
15:
return C₁
16:
function BETTER(S S2) if C(S)) < ((22) then
17:
return S₁
19:
else
return S₂
Algorithm D chooses the best subset of networks in our collection, subject to the inference time budget constraint. The algorithm recursively explores the space of solutions and prunes branches that cannot lead to optimal solutions. The recursion terminates when the budget is exhausted, at which point C, becomes empty and the loop body does not execute.
The sorting step on line requires heuristic upon which to sort. We found that ranking models based on how much they improve the current solution, S, works well. It should be noted that this algorithm always produces optimal solution, regardless of which sorting heuristic is used. However, better sorting heuristics reduce the running time because subsequent iterations will more readily detect and prune portions of the search space that cannot contain optimal solution. In our setup, we tried variants of problems with tasks and 36 networks, and all of them took less than a second to solve.
The definition the BETTER() function is application-specife For our experiments, we prefer networks that have the lowest total loss across all five tasks. Other applications may have hard
13

---

Under review as a conference paper at ICLR 2020
performance requirements for some of the tasks, and performance one of these tasks cannot be sacrificed in order to achieve better performance on another task. Such application-spen -speciif con- straints can be encoded in BETTER().
14

---

Under review as a conference paper at ICLR 2020
A.2
ANALYSIS ON SMALLER TASK SETS
Average of five runs with 4 tasks each
Single Traditional Multi-task Network
Average Total Loss
0.4
0.39
Independent Networks
0.38
Prediction with 20% Data Pass (ours)
0.37
Optimal Network Choice (ours)
0.36
2
4
Figure 6: Our experiments re-rum all 4-task subsets, then averaged.
In order to determine how well network selection works for different task sets, re-ran network selection on all five 4-task subsets of our task set. The performance average of all sets is shown in Figure 6 We see that techniques generalize least subsets of our studied tasks.
15

---

Under review as a conference paper at ICLR 2020
A.3 DEPTH AND NORMALS MORE TYPICAL SETTING
The finding that Depth and Normals don't cooperate counter much of the multi- task learning literature such as Wang et al.)
100k training instances Depth Test Loss Normals Test Loss
Depth Alone
0.265
Normals Alone
0.1398
2016 al. (2018), and Zhang al. 2019). However, the majority of these works use training sets with fewer than 100k instances, while we use nearly 4 mil- lion training instances. Table 5 shows the loss obtained on our setup when we limit to only 100k training instances. The fact that task affinities can change depending the amount of available training data demonstrates the necessity of using an empirical approach like ours for finding task affinities and groupings.
Joint Depth + Normals
0.2525
0.1319
Table 5: Positive task affinity between depth and normals in low data setting.
A.4
TABULAR DATA
Ours Optimal
Single 20% pass 3.3.1
Higher Order 3.3.2
SDNKE
SDNKE
SDNKE
1.5
DNKE, S
SDNK, E
DNKE, S
2
nKE. SDN nKE, SDn, N nKE, SDn, N
SDke, NKE
DNK, E, S
2.5
SDke, nKE, N
DNK, E. Sn
SDne, sdke, NKE
DNK, E. Sn
3.5
nKE, Snk, Dnk, N nKE, Snk, Dnk, nKE, Snk, Dak, N nkE, Snk, Dak, N. K
SDne, sdke, nKE, N SDne, sdke, nKE, N sDne, sdke, nKE, N, Snk sDne, sdke, nKE, N, Snk
DnK. E, Sn, N
Sn, DK, E, N
4.5
Sn, E. K, Dn, N
Sn, E, K, Dn, N
Table 6: The task groups picked by each of our techniques for every budget choice between 1 and Networks are shown as a list of letters corresponding to each task the network contains. S: Semantic Segmentation, D): Depth Estimation, N: Surface Normal Prediction, K: Keypoint Detection, E: Edge Detection. Capital letters denote that solution used that network's prediction for that task. Half-sized networks shown red.
Time Budget
1
1,5
2
2.5
3.5
4
4.5
5
Sener al.
0.562
0.556
0.551
0.547
GradNorm 0.515
0.500
Pessimal Grouping
0.503
0.503
0.503
0.503
0.503
0.502
0.499
0.496
0.495
Traditional MTL
0.503
0,492
0.487
0.488
Random Groupings
0.503
0.483 0.475 0.471 0.467
0.464
0.462
0.460
0.459
Independent
0.515
0.501
0.477 0.465
0.454
0.448
Ours (ESA) 3
0,503 0.487
0.467 0.461
0.457
0.451
0.451
0,447
0.447
Ours (HOA) 3.3.2
0.503
0.461 0.455
0.451 0.449
0.445
0.444
0.445
0.442
Ours Optimal
0.503 0.461 0.452 0.446 0.442 0.439 0.436 0.436 0.435
Table 7: The total test set loss on all five tasks for each method under each inference time budget. Lower is better. The data is the same as in Figures 3 and
16

---

Under review as a conference paper at ICLR 2020
SemSeg Depth Normals Keypoints
Edges
S
0.08039
NNNN
0.1695
0.08591
K
0.0895
E
0.02783
SD
0.07858
0.1833
SN
0.074
0.0997
SK
0.07722
0.09718
SE
0.07897
0.04462
DN
0.1695
0.09275
DK
0.1706
0.09318
DE
0.1748
0.03192
NK
0.08968
0.09181
NE
0.09358
0.02908
KE
0.09185
0.03488
SDN
0.07498
0.1698
0.09575
SDK
0.07699
0.1782
0.09704
SDE
0.07893
0.1863
0.04559
SNK
0.0722
0.09919
0.0961
SNE
0.07222
0.0982
0,03689
SKE
0.0766
0.09342
0.03508
DNK
0.1654
0.09358
0.09253
DNE
0.1708
0.09396
0.03286
DKE
0.1793
0.09073
0.02937
NKE
0.09626
0.09024
0.02609
SDNK
0.07762
0.1822
0.09869
0.1015
SDNE
0.07576
0.1735
0.09718
0.04513
SDKE
0.0795
0.1797
0.09272
0.04141
SNKE
0.07369
0.09944
0.09697
0.03312
DNKE
0.1708
0.09392
0.09334
0.02803
SDNKE
0.07854
0.1864
0.1
0.09814
0.04453
Table 8: The validation set performance of our 31 networks each task that they solve. Tasks are named to contain a letter for each task that they solve. S: Semantic Segmentation, D: Depth Estimation, N: Surface Normal Prediction, K: Keypoint Detection, E: Edge Detection.
17

---

Under review as a conference paper at ICLR 2020
SemSeg Depth Normals Keypoints Edges
0.07662
0.1696
0.08555
K
0.08847
0.0275
SD
0.07419
0.1831
SN
0.07084
0.0994
SK
0.07369
0.09601
SE
0.07504
0.044
DN
0.1694
0.09249
DK
0.1713
0.08882
DE
0.1753
0.03145
NK
0.08934
0.09077
NE
0.09327
0.02865
KE
0.09077
0.0344
SDN
0.07193
0.17
0.09544
SDK
0.07311
0.1785
0.09591
SDE
0.07617
0.1865
0.04474
SNK
0.06933
0,09966
0.09302
SNE
0.06859
0.09796
0.03625
SKE
0.07323
0.09232
0.03463
DNK
0.1658
0.09318
0.09143
DNE
0.1706
0.09362
0.03239
DKE
0.1795
0.08968
0.02887
NKE
0.09596
0.08921
0.02566
SDNK
0.07338
0.1826
0.09836
0.1003
SDNE
0.07249
0.1739
0.09689
0.04441
SDKE
0.07634
0.1801
0.09157
0.04097
SNKE
0.07111
0.09941
0.09464
0.03328
DNKE
0.1704
0,09356
0.09226
0.02768
SDNKE
0.07603
0.186
0.09976
0.09704
0.04395
Table 9: The test set performance of our 31 networks on each task that they solve.
18


Reviews:
[{"review_id": "HkxHDaSaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission studies how to group tasks to train together to find a better runtime-accuracy trade-off with a single but large multi-task neural model. The authors perform an extensive empirical study in the Taskonomy dataset. I like Section 6.1 in particular, which shows the difference between multi-task learning and transfer learning. \n\nMy main concern is that the competition between different tasks may stem from the limited capacity of the model during training. It might be possible that with enough parameters, the competing tasks become compatible. If I were the authors, I would first train a bigger model on multiple tasks and then distill it into a smaller one, which does not increase inference time. \n\nFurthermore, the paper has more than 8 pages. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgJ83u0KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on how to partition a bunch of tasks in several groups and then it use multi-task learning to improve the performance.  The paper makes an observation that multi-task relationships are not entirely correlated to transfer relationships and proposes a computational framework to optimize the assignment of tasks to network under a given computational budget constraint. It experiments on different combinations of the tasks and uses two heuristics to reduce the training overheads, early stopping approximation and higher order approximation. \n\nPlease see the detained comments as follows:\n1. The experiments are based on the assumptions that the network structures (how parameters are shared across tasks) are fixed. From my perspective, understanding how to optimize the parameters sharing across two tasks should be the first step to study how to optimally combine the training of tasks. Otherwise, different parameter sharing structures across tasks may lead to different conclusions.\n\n2. It requires optimization of the article structure. E.g., algorithm 1 is important and should be in the main context.\n\n3. It is also related to neural architecture search and it requires some discussions.\n\n4. The paper is over-length. \n\n5. A lot of typos. \n\nNits:\nPage 1: vide versa -> vice versa\n\nPage 3: two networsk -> two networks\n\nPage 6: budge ?? 1.5\n\nPage 7: overlap between lines (under figure 3)\n\nPage 8: half segmented sentence in section 6.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "ByeKMG-W5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper works on the problem if training a set of networks to solve a set of tasks. The authors try to discover an optimal task split into the networks so that the test performances are maximized given a fixed testing resource budget. By default, this requires searching over the entire task combination space and is too slow. The authors propose two strategies for fast approximating the enumerative search. Experiments show their searched combinations give better performance in the fixed-budget testing setting than several alternatives.\n\n+ This paper works on a new and interesting problem. Training more than one networks for a few tasks is definitely a valid idea in real applications and related to broad research fields.\n+ The baseline setup is comprehensive. The difference between optimal, random, and worst clearly shows this problem worths effort for research.\n- I believe this problem setup requires a larger task set. In the paper the authors manually picked 5 tasks. It seems straightforward for a human to manually group them together: segmentation and edge/ surface + depth/ keypoint for 3 networks. It is unclear how better a network can do than a human in one minute or should we expect learning the task split is better than manual design.\n- Both technical contributions in Section 3.3 look straightforward. Given the good performance in Figure3, it is fine.\n- I am confused by the comparison to Sener and Koltun. How do you change the inference budget for them? If it is changing the number of channels for a single network, I believe it can be improved more.\n\nOverall I believe this is a good paper to open interesting research direction with solid baselines. I am happy to accept this paper and see more exciting future works in this direction.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: An approach to make multi-task learning is presented, based on the idea of assigning tasks through the concepts of cooperation and competition. 

The main idea is well-motivated and explained well. The experiments demonstrate that the method is promising. However, there are a few  concerns regarding fundamental aspects, such as: how are the decisions affected by the number of parameters? Could ad-hoc algorithms with human in the loop provide the same benefit, when the task-set is small? More importantly, identifying task groups for multi-task learning is an idea presented in prior work, e.g. [1,2,3]. This important body of prior work is not discussed at all in this paper.

[1] Han and Zhang. "Learning multi-level task groups in multi-task learning"
[2] Bonilla et al. "Multi-task Gaussian process prediction"
[3] Zhang and Yang. "A Survey on Multi-Task Learning"

Decision: Reject