Which Tasks Should Be Learned Together in Multi-task Learning?

1 INTRODUCTION . Many applications , especially robotics and autonomous vehicles , are chiefly interested in using multi-task learning to reduce the inference time and computational complexity required to estimate many characteristics of visual input . For example , an autonomous vehicle may need to detect the location of pedestrians , determine a per-pixel depth , and predict objects ’ trajectories , all within tens of milliseconds . In multi-task learning , multiple tasks are solved at the same time , typically with a single neural network . In addition to reduced inference time , solving a set of tasks jointly rather than independently can , in theory , have other benefits such as improved prediction accuracy , increased data efficiency , and reduced training time . Unfortunately , the quality of predictions are often observed to suffer when a network is tasked with making multiple predictions . This is because learning objectives can have complex and unknown dynamics and may compete . In fact , multi-task performance can suffer so much that smaller independent networks are often superior ( as we will see in the experiments section ) . We refer to any situation in which the competing priorities of the network cause poor task performance as crosstalk . On the other hand , when task objectives do not interfere much with each other , performance on both tasks can be maintained or even improved when jointly trained . Intuitively , this loss or gain of quality seems to depend on the relationship between the jointly trained tasks . Prior work has studied the relationship between tasks for transfer learning ( Zamir et al . ( 2018 ) ) . However , we find that transfer relationships are not highly predictive of multi-task relationships . In addition to studying multi-task relationships , we attempt to determine how to produce good prediction accuracy under a limited inference time budget by assigning competing tasks to separate networks and cooperating tasks to the same network . More concretely , this leads to the following problem : Given a set of tasks , T , and a computational budget b ( e.g. , maximum allowable inference time ) , what is the optimal way to assign tasks to networks with combined cost≤ b such that a combined measure of task performances is maximized ? To this end , we develop a computational framework for choosing the best tasks to group together in order to have a small number of separate deep neural networks that completely cover the task set and that maximize task performance under a given computational budget . We make the intriguing Encoder Shared Representation Decoder Decoder Semantic Segmentation Depth Estimation Encoder Shared Representation DecoderDecoder Edge Detection Keypoint Detection Half Sized Encoder Decoder Surface Normal Prediction A B C Input Image Discarded Decoder Discarded Surface Normal Prediction Discarded Decoder Discarded Surface Normal Prediction Figure 1 : Given five tasks to solve , there are many ways that they can be split into task groups for multitask learning . How do we find the best one ? We propose a computational framework that , for instance , suggests the following grouping to achieve the lowest total loss , using a computational budget of 2.5 units : train network A to solve Semantic Segmentation , Depth Estimation , and Surface Normal Prediction ; train network B to solve Keypoint Detection , Edge Detection , and Surface Normal Prediction ; train network C with a less computationally expensive encoder to solve Surface Normal Prediction alone ; including Surface Normals as an output in the first two networks were found advantageous for improving the other outputs , while the best Normals were predicted by the third network . This task grouping outperforms all other feasible ones , including learning all five tasks in one large network or using five dedicated smaller networks . observation that the inclusion of an additional task in a network can potentially improve the accuracy of the other tasks , even though the performance of the added task might be poor . This can be viewed as regularizing or guiding the loss of one task by adding an additional loss , as often employed in curriculum learning or network regularization Bengio et al . ( 2009 ) . Achieving this , of course , depends on picking the proper regularizing task – our system can take advantage of this phenomenon , as schematically shown in Figure 1 . This paper has two main contributions . In Section 3 , we outline a framework for systematically assigning tasks to networks in order to achieve the best total prediction accuracy with a limited inference-time budget . We then analyze the resulting accuracy and show that selecting the best assignment of tasks to groups is critical for good performance . Secondly , in Section 6 , we analyze situations in which multi-task learning helps and when it doesn ’ t , quantify the compatibilities of various task combinations for multi-task learning , compare them to the transfer learning task affinities , and discuss the implications . Moreover , we analyze the factors that influence multi-task affinities . 2 PRIOR WORK . Multi-Task Learning : See Ruder ( 2017 ) for a good overview of multi-task learning . The authors identify two clusters of contemporary techniques that we believe cover the space well , hard parameter sharing and soft parameter sharing . In brief , the primary difference between the majority of the existing works and our study is that we wish to understand the relationships between tasks and find compatible groupings of tasks for any given set of tasks , rather than designing a neural network architecture to solve a particular fixed set of tasks well . A known contemporary example of hard parameter sharing in computer vision is UberNet ( Kokkinos ( 2017 ) ) . The authors tackle 7 computer vision problems using hard parameter sharing . The authors focus on reducing the computational cost of training for hard parameter sharing , but experience a rapid degradation in performance as more tasks are added to the network . Hard parameter sharing is also used in many other works such as ( Thrun ( 1996 ) ; Caruana ( 1997 ) ; Nekrasov et al . ( 2018 ) ; Dvornik et al . ( 2017 ) ; Kendall et al . ( 2018 ) ; Bilen & Vedaldi ( 2016 ) ; Pentina & Lampert ( 2017 ) ; Doersch & Zisserman ( 2017 ) ; Zamir et al . ( 2016 ) ; Long et al . ( 2017 ) ; Mercier et al . ( 2018 ) ; d. Miranda et al . ( 2012 ) ; Zhou et al . ( 2018 ) ; Rudd et al . ( 2016 ) ) . Other works , such as ( Sener & Koltun ( 2018 ) ) and ( Chen et al . ( 2018b ) ) , aim to dynamically reweight each task ’ s loss during training . The former work finds weights that provably lead to a Pareto-optimal solution , while the latter attempts to find weights that balance the influence of each task on network weights . Finally , ( Bingel & Søgaard ( 2017 ) ) studies task interaction for NLP . In soft or partial parameter sharing , either there is a separate set of parameters per task , or a significant fraction of the parameters are unshared . The models are tied together either by information sharing or by requiring parameters to be similar . Examples include ( Dai et al . ( 2016 ) ; Duong et al . ( 2015 ) ; Misra et al . ( 2016 ) ; Tessler et al . ( 2017 ) ; Yang & Hospedales ( 2017 ) ; Lu et al . ( 2017 ) ) . The canonical example of soft parameter sharing can be seen in ( Duong et al . ( 2015 ) ) . The authors are interested in designing a deep dependency parser for languages such as Irish that do not have much treebank data available . They tie the weights of two networks together by adding an L2 distance penalty between corresponding weights and show substantial improvement . Another example of soft parameter sharing is Cross-stitch Networks ( Misra et al . ( 2016 ) ) . Starting with separate networks for two tasks , the authors add ‘ cross-stitch units ’ between them , which allow each network to peek at the other network ’ s hidden layers . This approach reduces but does not eliminate task interfearence , and the overall performance is less sensitive to the relative loss weights . Unlike our method , none of the aforementioned works attempt to discover good groups of tasks to train together . Also , soft parameter sharing does not reduce inference time , a major goal of ours . Transfer Learning : Transfer learning ( Pratt ( 1993 ) ; Helleputte & Dupont ( 2009 ) ; Silver & Bennett ( 2008 ) ; Finn et al . ( 2016 ) ; Mihalkova et al . ( 2007 ) ; Niculescu-Mizil & Caruana ( 2007 ) ; Luo et al . ( 2017 ) ; Razavian et al . ( 2014 ) ; Pan & Yang ( 2010 ) ; Mallya & Lazebnik ( 2018 ) ; Fernando et al . ( 2017 ) ; Rusu et al . ( 2016 ) ) is similar to multi-task learning in that solutions are learned for multiple tasks . Unlike multi-task learning , however , transfer learning methods often assume that a model for a source task is given and then adapt that model to a target task . Transfer learning methods generally neither seek any benefit for source tasks nor a reduction in inference time as their main objective . Neural Architecture Search ( NAS ) : Many recent works search the space of deep learning architectures to find ones that perform well ( Zoph & Le , 2017 ; Liu et al. , 2018 ; Pham et al. , 2018 ; Xie et al. , 2019 ; Elsken et al. , 2019 ; Zhou et al. , 2019 ; Baker et al. , 2017 ; Real et al. , 2018 ) . This is related to our work as we search the space of task groupings . Just as with NAS , the found task groupings often perform better than human-engineered ones . Task Relationships : Our work is most related to Taskonomy ( Zamir et al . ( 2018 ) ) , where the authors studied the relationships between visual tasks for transfer learning and introduced a dataset with over 4 million images and corresponding labels for 26 tasks . This was followed by a number of recent works , which further analyzed task relationships ( Pal & Balasubramanian ( 2019 ) ; Dwivedi & Roig . ( 2019 ) ; Achille et al . ( 2019 ) ; Wang et al . ( 2019 ) ) for transfer learning . While they extract relationships between these tasks for transfer learning , we are interested in the multi-task learning setting . Interestingly , we find notable differences between transfer task affinity and multi-task affinity . Their method also differs in that they are interested in labeled-data efficiency and not inference-time efficiency . Finally , the transfer quantification approach taken by Taskonomy ( readout functions ) is only capable of finding relationships between the high-level bottleneck representations developed for each task , whereas structural similarities between tasks at all levels are potentially relevant for multi-task learning . 3 TASK GROUPING FRAMEWORK . Our goal is to find an assignment of tasks to networks that results in the best overall loss . Our strategy is to select from a large set of candidate networks to include in our final solution . We define the problem as follows : We want to minimize the overall loss on a set of tasks T = { t1 , t2 , ... , tk } given a limited inference time budget , b , which is the total amount of time we have to complete all tasks . Each neural network that solves some subset of T and that could potentially be a part of the final solution is denoted by n. It has an associated inference time cost , cn , and a loss for each task , L ( n , ti ) ( which is ∞ for each task the network does not attempt to solve ) . A solution S is a set of networks that together solve all tasks . The computational cost of a solution is cost ( S ) = ∑ n∈S cn . The loss of a solution on a task , L ( S , ti ) , is the lowest loss on that task among the solution ’ s networks1 , L ( S , ti ) = minn∈S L ( n , ti ) . The overall performance for a solution is L ( S ) = ∑ ti∈T L ( S , ti ) . We want to find the solution with the lowest overall loss and a cost that is under our budget , Sb = argminS : cost ( S ) ≤b L ( S ) . 1In principle , it may be possible to create an even better-performing ensemble when multiple networks solve the same task , though we do not explore this .

Summary: This paper works on the problem if training a set of networks to solve a set of tasks. The authors try to discover an optimal task split into the networks so that the test performances are maximized given a fixed testing resource budget. By default, this requires searching over the entire task combination space and is too slow. The authors propose two strategies for fast approximating the enumerative search. Experiments show their searched combinations give better performance in the fixed-budget testing setting than several alternatives.