Safe Multi-Task Learning

1 INTRODUCTION . Multi-Task Learning ( MTL ) ( Caruana , 1997 ; Zhang & Yang , 2021 ) , which aims to improve the generalization performance of multiple learning tasks by sharing knowledge among those tasks , has attracted much attention in recent years . Compared with single-task learning , it not only improves the performance but also reduces the training and inference time . Though MTL has demonstrated its usefulness in many applications , MTL can not guarantee to improve the performance of all the tasks when compared with single-task learning . As empirically observed in ( Lee et al. , 2016 ; Guo et al. , 2020 ; Sun et al. , 2020 ; Standley et al. , 2020 ) , when learning on multiple tasks together , each of some existing MTL models can achieve better performance on some of the tasks than a single-task model but perform worse on the other tasks . Such phenomenon is defined as the negative sharing phenomenon here , which is similar to the ‘ negative transfer ’ phenomenon ( Wang et al. , 2019 ) in transfer learning ( Yang et al. , 2020 ) but with some difference in the ways of knowledge transfer and sharing in those two learning paradigms as discussed later . One reason for the occurrence of negative sharing is that there are unrelated tasks among tasks in investigation , making jointly learning these tasks impair the performance of some tasks . To the best of our knowledge , there is little work to study the negative sharing problem for MTL . In this paper , we firstly give a formal definition for negative sharing occurred in MTL . Then we formally define an ideal and also basic situation of MTL , safe multi-task learning , where an MTL model performs no worse than its single-task counterpart on each task . Hence , safe multi-task learning means that there is no negative sharing occurred . According to the definition of MTL ( Caruana , 1997 ; Zhang & Yang , 2021 ) , we can see that every MTL model should achieve safe multi-task learning . Otherwise , single-task learning is more preferred than MTL , since an unsafe MTL model may bring the risk of worsening the generalization performance of some or even all the tasks . Moreover , we formally define η-partially safe multi-task learning as a loose version of safe multi-task learning to allow the MTL model to perform worse than its single-task counterpart on tasks with a proportion no larger than η . To achieve safe multi-task learning , we propose a Safe Multi-Task Learning ( SMTL ) model . Specifically , given m learning tasks , the SMTL model consists of a public encoder shared by all the tasks , m private encoders for the m tasks , m gates for the m tasks , and m private decoders for the m tasks . Hence each task has a private encoder , a gate , and a private decoder . The gate of each task is responsible of learning how to linearly combine the public encoder and the corresponding private encoder for the downstream private decoder . To reduce the storage cost during the inference stage , we propose a lite version of SMTL via the Gumbel-softmax trick ( Jang et al. , 2017 ; Maddison et al. , 2017 ) to enforce each gate to choose either the public encoder or private encoder . Moreover , to study the impact of different locations of the gates , we propose variants of the SMTL model , which place the gates after the decoders . Furthermore , we analyze the SMTL model from the perspectives of generalization bound and optimization . Experiments on several MTL benchmark datasets demonstrate the effectiveness of the proposed methods . The main contributions of this paper are summarized as follows . • We provide formal definitions for MTL , including negative sharing , safe multi-task learning , and η-partially safe multi-task learning . • To achieve safe multi-task learning , we propose a simple and effective SMTL model . Built on the SMTL model , we propose its variants . • We conduct extensive experiments to demonstrate the superiority of the proposed methods over state-of-the-art methods . 2 RELATED WORK . MTL has been extensively studied in recent years ( Evgeniou & Pontil , 2004 ; Zhang & Yeung , 2010 ; Kumar & Daume III , 2012 ; Zhang et al. , 2021 ; Guo et al. , 2021 ) . How to design a good network architecture for MTL is an important issue . The most popular model is the multi-head hard sharing architecture ( Caruana , 1997 ; Zhang et al. , 2014 ; Long & Wang , 2015 ; Liu et al. , 2015 ; Ruder et al. , 2019 ) , which shares the first several layers among all the tasks and allow the subsequent layers to be specific to different tasks . Then , to better handle task relationships , different MTL architectures have been proposed . For example , Misra et al . ( 2016 ) propose a cross-stitch network to learn to linearly combine hidden representations of different tasks . Liu et al . ( 2019 ) propose a Multi-Task Attention Network ( MTAN ) , which consists of a shared network and an attention module for each task so that both shared and task-specific feature representations can be learned via the attention mechanism . Gao et al . ( 2019 ) propose a Neural Discriminative Dimensionality Reduction ( NDDR ) layer to enable automatic feature fusing at every layer from different tasks . Sun et al . ( 2020 ) propose an Adaptive Sharing ( AdaShare ) method to learn the sharing pattern through a task-specific policy that selectively chooses which layers to be executed for each task . Guo et al . ( 2020 ) propose an algorithm to learn where to share or branch within a network for MTL . Cui et al . ( 2021 ) propose an Adaptive Feature Aggregation ( AFA ) layer , where a dynamic aggregation mechanism is designed to allow each task to adaptively determine the degree of the knowledge sharing between tasks . All the existing works do not study how to achieve safe multi-task learning , which is the focus of this paper . 3 SAFE MULTI-TASK LEARNING . In this section , we first formally define some terminologies for MTL . Then we introduce the proposed SMTL method . Moreover , we propose some variants of the SMTL model . Finally , we provide some analyses for SMTL . 3.1 DEFINITIONS . Definition 1 ( Negative Sharing ) . For an MTL model which is trained on multiple learning tasks jointly , if its generalization performance on some tasks is inferior to the generalization performance of the corresponding single-task model which is trained on each task separately , then negative sharing occurs . Remark 1 . Negative sharing occurs when some tasks are totally or partially irrelevant to other tasks . In this case , manually enforcing all the tasks to have some forms of sharing will impair the performance of some or even all the tasks . In Definition 1 , the MTL model and the singletask model usually have similar architectures as totally different architectures may bring additional confounding factors . Moreover , negative sharing is similar to negative transfer ( Wang et al. , 2019 ) in transfer learning ( Yang et al. , 2020 ) . However , knowledge transfer in transfer learning is directed as it is from a source domain to a target domain , while knowledge sharing in MTL is among all the tasks , making it undirected . From this perspective , negative sharing is different from negative transfer . Definition 2 ( Safe Multi-Task Learning ) . When no negative sharing occurs for an MTL model on a dataset , this MTL model is said to achieve safe multi-task learning on this dataset . The ideal situation for an MTL model is to achieve safe multi-task learning . However , not all the MTL methods can achieve safe multi-task learning and hence we define η-partially safe multi-task learning , which can be viewed as a loose version of safe multi-task learning . Definition 3 ( η-Partially Safe Multi-Task Learning ) . Given multiple learning tasks in a dataset , ηpartially safe multi-task learning ( 0 ≤ η ≤ 100 ) indicates that on about η percentage of tasks , the generalization performance of an MTL model is no worse than that of its single-task counterpart . When η equals 100 , η-partially safe multi-task learning becomes safe multi-task learning . When η is equal to 0 , the MTL model performs worse than its single-task counterpart in all the tasks . 3.2 SMTL . With m learning tasks { Ti } mi=1 , our goal is to design a model that can achieve safe multi-task learning . To achieve this goal , we propose the SMTL model , which is introduced in the following . Without loss of generality , we consider the case that different tasks share the input data or equivalently each data point has an output for each task . As shown in the left figure of Figure 1 , the SMTL model can be divided into four parts : a public encoder fS shared by all the tasks , m private encoders { ft } mt=1 for m tasks , m gates { gt } mt=1 for m tasks , and m private decoders { ht } mt=1 for m tasks . For task t , its model consists of the public encoder fS , the private encoder ft , the gate gt , and the private decoder ht , where fS and ft are combined by gt . Specifically , given a data point x , the gate gt in task t receives two inputs : fS ( x ) and ft ( x ) , and outputs gt ( fS ( x ) , ft ( x ) ) , which is fed into ht to obtain the final prediction ht ( gt ( fS ( x ) , ft ( x ) ) ) , which is used to define a loss for x . Here the gate gt is to determine the contributions of fS and ft . Ideally , when task t is unrelated to other tasks , gt should choose ft only . On another extreme where all the tasks have the same data distribution , all the tasks should use the same model and hence gt should choose fS only . On cases between the two extremes , gt can combine fS and ft in proportion . To achieve the aforementioned effects , we use a simple convex combination function for gt as gt ( fS ( x ) , ft ( x ) ) = αtfS ( x ) + ( 1− αt ) ft ( x ) , ( 1 ) where αt ∈ [ 0 , 1 ] defines the weight of fS ( x ) and is a learnable parameter . When αt equals 0 , only the private encoder ft will be used , which corresponds to the unrelated case . When αt is equal to 1 , only the public encoder fS will be used , which is corresponding to the case that all the tasks follow the same distribution . When αt is between 0 and 1 , fS and ft are combined with proportions αt and 1− αt , respectively , where αt can be adaptively learned to minimize the training loss on task t. Thus , the entire objective function of SMTL is formulated as min Θ∈C 1 mn n∑ i=1 m∑ t=1 Lt ( yit , ht ( gt ( fS ( xi ) , ft ( xi ) ) ) ) , ( 2 ) where xi denotes the ith data point , yit denotes the label of xi in task t , n denotes the total number of data points in the training dataset , Θ includes all the parameters in fS , { ft } mt=1 , { gt } mt=1 , and { ht } mt=1 , C = { Θ|0 ≤ αt ≤ 1 , ∀t } denotes the feasible set for Θ , and Lt denotes the loss function for task t ( e.g. , the pixel-wise cross-entropy loss for semantic segmentation , L1 loss for depth estimation , and element-wise dot product loss for surface normal prediction ) . To see why the proposed SMTL model could achieve safe multi-task learning , we compare SMTL and the corresponding Single-Task Learning ( STL ) model which consists of a private encoder ft and a private decoder ht . It is easy to see that SMTL can reduce to the STL model for some or even all the tasks by making the gates of those tasks choose the corresponding private encoders ( i.e. , setting αt ’ s of those tasks to 0 ) . So if a task is unrelated to other tasks , the SMTL model can use the gating mechanism to separate this outlier task from other tasks , which may help achieve safe multi-task learning . Moreover , we can show that the training loss of the SMTL model is no larger than the average of that of the STL model on each task . To see that , it is easy to show that the STL model for task t can be represented as ht ( g0t ( ∅ , ft ( x ) ) ) , where g0t denotes the gate of task t with αt as 0 and ∅ denotes a null network . As g0t is a feasible gate for SMTL , after sufficient training , we probably have min Θ∈C 1 mn n∑ i=1 m∑ t=1 Lt ( yit , ht ( gt ( fS ( xi ) , ft ( xi ) ) ) ) ≤ min Θ′ 1 mn n∑ i=1 m∑ t=1 Lt ( yit , ht ( g0t ( ∅ , ft ( xi ) ) ) ) , where Θ′ includes parameters in { ft } mt=1 and { ht } mt=1 in STL models for all the tasks . This inequality shows the benefit of the proposed SMTL model . Even though the training loss is not a tight estimation of the generalization loss , we think that it is an indicator to reflect the generalization performance of the two models . In Section 4.4 , we study to use a bi-level formulation to learn the gates as hyperparameters on a validation set and its performance is comparable with the SMTL model based on problem ( 2 ) , which proves the usefulness of the indication of the training loss . Similarly , we can show that the training loss of the SMTL model is no larger than that of the DMTL model ( a.k.a . the multi-head hard sharing network ) which consists of a shared encoder by all the tasks and private decoders for m tasks . To see this , the DMTL model for task t can be represented as ht ( g1t ( fS ( x ) , ∅ ) ) , where g1t denotes the gate of task t with αt as 1 . As g1t is a feasible gate for SMTL , it is easy to get that the training loss of the SMTL model after sufficient training is lower than that of the DTML model , which could be one reason for the phenomenon that SMTL outperforms DMTL in our experiments .

Summary: The paper propose a simple method for safe multi-task learning where there is no negative transfer or "negative sharing" among tasks. It jointly trains shared encoder, task-specific (private) encoders, gate, and decoder. The gate computes importance for each output of shared and task specific encoder and combine the outputs with simple convex combination.