Year: 2021

Abstract:
In recent years, Multi-Task Learning (MTL) attracts much attention due to its good performance in many applications. However, many existing MTL models cannot guarantee that its performance is no worse than its single-task counterpart on each task. Though this phenomenon has been empirically observed by some works, little work aims to handle the resulting problem, which is formally defined as negative sharing in this paper. To achieve safe multi-task learning where no negative sharing occurs, we propose a Safe Multi-Task Learning (SMTL) model, which consists of a public encoder shared by all the tasks, private encoders, gates, and private decoders. Specifically, each task has a private encoder, a gate, and a private decoder, where the gate is to learn how to combine the private encoder and public encoder for the downstream private decoder. To reduce the storage cost during the inference stage, a lite version of SMTL is proposed to allow the gate to choose either the public encoder or the corresponding private encoder. Moreover, we propose a variant of SMTL to place all the gates after decoders of all the tasks. Experiments on several benchmark datasets demonstrate the effectiveness of the proposed methods.

heading: 1 INTRODUCTION
text: Multi-Task Learning (MTL) (Caruana, 1997; Zhang & Yang, 2021), which aims to improve the generalization performance of multiple learning tasks by sharing knowledge among those tasks, has attracted much attention in recent years. Compared with single-task learning, it not only improves the performance but also reduces the training and inference time. Though MTL has demonstrated its usefulness in many applications, MTL cannot guarantee to improve the performance of all the tasks when compared with single-task learning. As empirically observed in (Lee et al., 2016; Guo et al., 2020; Sun et al., 2020; Standley et al., 2020), when learning on multiple tasks together, each of some existing MTL models can achieve better performance on some of the tasks than a single-task model but perform worse on the other tasks. Such phenomenon is defined as the negative sharing phenomenon here, which is similar to the ‘negative transfer’ phenomenon (Wang et al., 2019) in transfer learning (Yang et al., 2020) but with some difference in the ways of knowledge transfer and sharing in those two learning paradigms as discussed later. One reason for the occurrence of negative sharing is that there are unrelated tasks among tasks in investigation, making jointly learning these tasks impair the performance of some tasks.
To the best of our knowledge, there is little work to study the negative sharing problem for MTL. In this paper, we firstly give a formal definition for negative sharing occurred in MTL. Then we formally define an ideal and also basic situation of MTL, safe multi-task learning, where an MTL model performs no worse than its single-task counterpart on each task. Hence, safe multi-task learning means that there is no negative sharing occurred. According to the definition of MTL (Caruana, 1997; Zhang & Yang, 2021), we can see that every MTL model should achieve safe multi-task learning. Otherwise, single-task learning is more preferred than MTL, since an unsafe MTL model may bring the risk of worsening the generalization performance of some or even all the tasks. Moreover, we formally define η-partially safe multi-task learning as a loose version of safe multi-task learning to allow the MTL model to perform worse than its single-task counterpart on tasks with a proportion no larger than η.
To achieve safe multi-task learning, we propose a Safe Multi-Task Learning (SMTL) model. Specifically, given m learning tasks, the SMTL model consists of a public encoder shared by all the tasks, m private encoders for the m tasks, m gates for the m tasks, and m private decoders for the m
tasks. Hence each task has a private encoder, a gate, and a private decoder. The gate of each task is responsible of learning how to linearly combine the public encoder and the corresponding private encoder for the downstream private decoder. To reduce the storage cost during the inference stage, we propose a lite version of SMTL via the Gumbel-softmax trick (Jang et al., 2017; Maddison et al., 2017) to enforce each gate to choose either the public encoder or private encoder. Moreover, to study the impact of different locations of the gates, we propose variants of the SMTL model, which place the gates after the decoders. Furthermore, we analyze the SMTL model from the perspectives of generalization bound and optimization. Experiments on several MTL benchmark datasets demonstrate the effectiveness of the proposed methods.
The main contributions of this paper are summarized as follows.
• We provide formal definitions for MTL, including negative sharing, safe multi-task learning, and η-partially safe multi-task learning.
• To achieve safe multi-task learning, we propose a simple and effective SMTL model. Built on the SMTL model, we propose its variants.
• We conduct extensive experiments to demonstrate the superiority of the proposed methods over state-of-the-art methods.

heading: 2 RELATED WORK
text: MTL has been extensively studied in recent years (Evgeniou & Pontil, 2004; Zhang & Yeung, 2010; Kumar & Daume III, 2012; Zhang et al., 2021; Guo et al., 2021). How to design a good network architecture for MTL is an important issue. The most popular model is the multi-head hard sharing architecture (Caruana, 1997; Zhang et al., 2014; Long & Wang, 2015; Liu et al., 2015; Ruder et al., 2019), which shares the first several layers among all the tasks and allow the subsequent layers to be specific to different tasks. Then, to better handle task relationships, different MTL architectures have been proposed. For example, Misra et al. (2016) propose a cross-stitch network to learn to linearly combine hidden representations of different tasks. Liu et al. (2019) propose a Multi-Task Attention Network (MTAN), which consists of a shared network and an attention module for each task so that both shared and task-specific feature representations can be learned via the attention mechanism. Gao et al. (2019) propose a Neural Discriminative Dimensionality Reduction (NDDR) layer to enable automatic feature fusing at every layer from different tasks. Sun et al. (2020) propose an Adaptive Sharing (AdaShare) method to learn the sharing pattern through a task-specific policy that selectively chooses which layers to be executed for each task. Guo et al. (2020) propose an algorithm to learn where to share or branch within a network for MTL. Cui et al. (2021) propose an Adaptive Feature Aggregation (AFA) layer, where a dynamic aggregation mechanism is designed to allow each task to adaptively determine the degree of the knowledge sharing between tasks. All the existing works do not study how to achieve safe multi-task learning, which is the focus of this paper.

heading: 3 SAFE MULTI-TASK LEARNING
text: In this section, we first formally define some terminologies for MTL. Then we introduce the proposed SMTL method. Moreover, we propose some variants of the SMTL model. Finally, we provide some analyses for SMTL.

heading: 3.1 DEFINITIONS
text: Definition 1 (Negative Sharing). For an MTL model which is trained on multiple learning tasks jointly, if its generalization performance on some tasks is inferior to the generalization performance of the corresponding single-task model which is trained on each task separately, then negative sharing occurs. Remark 1. Negative sharing occurs when some tasks are totally or partially irrelevant to other tasks. In this case, manually enforcing all the tasks to have some forms of sharing will impair the performance of some or even all the tasks. In Definition 1, the MTL model and the singletask model usually have similar architectures as totally different architectures may bring additional confounding factors. Moreover, negative sharing is similar to negative transfer (Wang et al., 2019)
in transfer learning (Yang et al., 2020). However, knowledge transfer in transfer learning is directed as it is from a source domain to a target domain, while knowledge sharing in MTL is among all the tasks, making it undirected. From this perspective, negative sharing is different from negative transfer. Definition 2 (Safe Multi-Task Learning). When no negative sharing occurs for an MTL model on a dataset, this MTL model is said to achieve safe multi-task learning on this dataset.
The ideal situation for an MTL model is to achieve safe multi-task learning. However, not all the MTL methods can achieve safe multi-task learning and hence we define η-partially safe multi-task learning, which can be viewed as a loose version of safe multi-task learning. Definition 3 (η-Partially Safe Multi-Task Learning). Given multiple learning tasks in a dataset, ηpartially safe multi-task learning (0 ≤ η ≤ 100) indicates that on about η percentage of tasks, the generalization performance of an MTL model is no worse than that of its single-task counterpart.
When η equals 100, η-partially safe multi-task learning becomes safe multi-task learning. When η is equal to 0, the MTL model performs worse than its single-task counterpart in all the tasks.

heading: 3.2 SMTL
text: With m learning tasks {Ti}mi=1, our goal is to design a model that can achieve safe multi-task learning. To achieve this goal, we propose the SMTL model, which is introduced in the following.
Without loss of generality, we consider the case that different tasks share the input data or equivalently each data point has an output for each task. As shown in the left figure of Figure 1, the SMTL model can be divided into four parts: a public encoder fS shared by all the tasks, m private encoders {ft}mt=1 for m tasks, m gates {gt}mt=1 for m tasks, and m private decoders {ht}mt=1 for m tasks. For task t, its model consists of the public encoder fS , the private encoder ft, the gate gt, and the private decoder ht, where fS and ft are combined by gt. Specifically, given a data point x, the gate gt in task t receives two inputs: fS(x) and ft(x), and outputs gt(fS(x), ft(x)), which is fed into ht to obtain the final prediction ht(gt(fS(x), ft(x))), which is used to define a loss for x.
Here the gate gt is to determine the contributions of fS and ft. Ideally, when task t is unrelated to other tasks, gt should choose ft only. On another extreme where all the tasks have the same data distribution, all the tasks should use the same model and hence gt should choose fS only. On cases between the two extremes, gt can combine fS and ft in proportion. To achieve the aforementioned effects, we use a simple convex combination function for gt as
gt(fS(x), ft(x)) = αtfS(x) + (1− αt)ft(x), (1)
where αt ∈ [0, 1] defines the weight of fS(x) and is a learnable parameter. When αt equals 0, only the private encoder ft will be used, which corresponds to the unrelated case. When αt is equal to 1, only the public encoder fS will be used, which is corresponding to the case that all the tasks follow the same distribution. When αt is between 0 and 1, fS and ft are combined with proportions αt and 1− αt, respectively, where αt can be adaptively learned to minimize the training loss on task t. Thus, the entire objective function of SMTL is formulated as
min Θ∈C
1
mn n∑ i=1 m∑ t=1 Lt(yit, ht(gt(fS(xi), ft(xi)))), (2)
where xi denotes the ith data point, yit denotes the label of xi in task t, n denotes the total number of data points in the training dataset, Θ includes all the parameters in fS , {ft}mt=1, {gt}mt=1, and {ht}mt=1, C = {Θ|0 ≤ αt ≤ 1,∀t} denotes the feasible set for Θ, and Lt denotes the loss function for task t (e.g., the pixel-wise cross-entropy loss for semantic segmentation, L1 loss for depth estimation, and element-wise dot product loss for surface normal prediction).
To see why the proposed SMTL model could achieve safe multi-task learning, we compare SMTL and the corresponding Single-Task Learning (STL) model which consists of a private encoder ft and a private decoder ht. It is easy to see that SMTL can reduce to the STL model for some or even all the tasks by making the gates of those tasks choose the corresponding private encoders (i.e., setting αt’s of those tasks to 0). So if a task is unrelated to other tasks, the SMTL model can use the gating mechanism to separate this outlier task from other tasks, which may help achieve safe multi-task learning. Moreover, we can show that the training loss of the SMTL model is no larger than the average of that of the STL model on each task. To see that, it is easy to show that the STL model for
task t can be represented as ht(g0t (∅, ft(x))), where g0t denotes the gate of task t with αt as 0 and ∅ denotes a null network. As g0t is a feasible gate for SMTL, after sufficient training, we probably have
min Θ∈C
1
mn n∑ i=1 m∑ t=1 Lt(yit, ht(gt(fS(xi), ft(xi)))) ≤ min Θ′ 1 mn n∑ i=1 m∑ t=1 Lt(yit, ht(g0t (∅, ft(xi)))),
where Θ′ includes parameters in {ft}mt=1 and {ht}mt=1 in STL models for all the tasks. This inequality shows the benefit of the proposed SMTL model. Even though the training loss is not a tight estimation of the generalization loss, we think that it is an indicator to reflect the generalization performance of the two models. In Section 4.4, we study to use a bi-level formulation to learn the gates as hyperparameters on a validation set and its performance is comparable with the SMTL model based on problem (2), which proves the usefulness of the indication of the training loss.
Similarly, we can show that the training loss of the SMTL model is no larger than that of the DMTL model (a.k.a. the multi-head hard sharing network) which consists of a shared encoder by all the tasks and private decoders for m tasks. To see this, the DMTL model for task t can be represented as ht(g1t (fS(x), ∅)), where g1t denotes the gate of task t with αt as 1. As g1t is a feasible gate for SMTL, it is easy to get that the training loss of the SMTL model after sufficient training is lower than that of the DTML model, which could be one reason for the phenomenon that SMTL outperforms DMTL in our experiments.

heading: 3.3 LITE SMTL
text: The SMTL model is computationally efficient during both the training and inference stages, but it requires to keep all the private encoders as well as the public encoder at the inference stage. In some applications with limited storage resource such as edge computing, we hope to eliminate some private encoders that have small contributions to the corresponding tasks to reduce the storage cost. To achieve that, we propose a lite version of SMTL called L-SMTL. Specifically, the L-SMTL model will enforce each gate to choose either the public encoder fS or the corresponding private encoder, which is equivalent to forcing each αt to be either 1 or 0. Thus, as shown in the right figure of Figure 1, at the inference stage, the unchosen private encoders can be thrown away and we do not need to store parameters in them. However, the objective function of the L-SMTL model, which is similar to problem (2) by replacing the constraint αt ∈ [0, 1] with αt ∈ {0, 1}, is non-differentiable as each αt is binary valued.
To optimize the non-differentiable objective function in L-SMTL, we adopt the Gumbel-softmax trick (Jang et al., 2017; Maddison et al., 2017). Specifically, if task t chooses the public encoder
with the probability αt, it can be cast as sampling α̃t from a Bernoulli distribution B(pt,1) with probability pt,1 = αt to assign α̃t the value of 1 (i.e., using the public encoder for task t) and with probability pt,0 = 1 − αt to assign α̃t the value of 0 (i.e., using the private encoder for task t). However, since the sampling process is still non-differentiable, the Gumbel-softmax trick is used to reparameterize αt. We first use an equivalent formulation for sampling α̃t based on the Gumbel-Max trick (Gumbel, 1948) as
α̃t = arg maxk∈{0,1}(bk + log pt,k), (3)
where bk (k = 0, 1) is drawn from a Gumbel distribution Gumbel(0, 1) independently (i.e., bk = − log(− log(u)) where u is sampled from a uniform distribution U(0, 1)). Then we use the softmax function to approximate the arg max function and define
α̂t = exp((b1 + log pt,1)/τ)∑
k∈{0,1} exp((bk + log pt,0)/τ) , (4)
where τ is a temperature parameter. It is easy to show that α̂t = α̃t when τ → 0 and α̂t = 12 when τ → ∞. Thus, we use a small value of τ to make a sharp distribution for α̂t. In implementations as shown in (Jang et al., 2017), the arg max function in Eq. (3) is used in the forward pass and the softmax function in Eq. (4) is used in the backward pass to approximate true gradients.

heading: 3.4 VARIANT OF SMTL AND L-SMTL
text: To study the impact of the position of gates in SMTL, we introduce a variant of SMTL called SMTLc, where all the gates are placed after all the decoders. Similarly, the SMTLc model also has a lite version called L-SMTLc. An illustration for the SMTLc and L-SMTLc models is shown in Figure 2 in Appendix C.
Specifically, the SMTLc model can be divided into five parts, including a public encoder fS shared by all the m tasks, m public decoders {hS,t}mt=1 for the m tasks, m private encoders {ft}mt=1 for the m tasks, m private decoders {ht}mt=1 for the m tasks, and m gates {gt}mt=1 for the m tasks. For task t, a data point x is fed into the public encoder fS and the public decoder hS,t to get an output oS , and it is also fed into the private encoder ft and the private decoder ht to get another output ot. Then the gate gt will adaptively combine the two outputs to obtain the final output, i.e., ŷt = αtoS + (1− αt)ot. Similar to the SMTL model, when task t is unrelated to other tasks, ideally the SMTLc model can only choose the private encoder ft and the private decoder ht with a zero αt. Hence, the SMTLc model can achieve safe multi-task learning when the negative sharing occurs. Moreover, when 0 < αt < 1, the combination of the public component consisting of the public encoder and decoder and the private component consisting of the private encoder and decoder may act in a way similar to ensemble learning, which may help improve the generalization performance. Moreover, similar to the SMTL model, the training loss of SMTLc could be lower than those of the STL and DMTL models. Built on the SMTLc model, the L-SMTLc model approximately learns binary-valued {αt} via the Gumbel-softmax trick.

heading: 3.5 ANALYSIS
text: We analyze the generalization bound for the SMTL method. We consider a general case that different tasks can have different data distributions. The probability measure for the data distribution in task t is denoted by µt and the data in all the tasks take the form of (X̄, Ȳ) ∼ ∏m t=1(µt)
n, where X̄ = (X1, . . . ,Xm), Xt = (x1t , . . . ,x n t ) and Ȳ denotes the label of X̄. Here we consider the encoders f1, . . . , fm, fS : X → RP as mapping functions and define ϕ(x) = (f1(x), . . . , fm(x), fS(x)) : X → RP (m+1). The functions ϕ and ht are assumed to be chosen from hypothesis classes F and H, respectively. Note that we only use ft and fS for the representation in task t. For the ease of analysis, we define a weight vector βt ∈Mt for task t, whereMt = {βt ∈ Rm+1+ | βt,t+βt,m+1 = 1, βt,i = 0 if i 6= t and i 6= m + 1}, and βt,i represents the ith entry of βt. Thus, for given αt, the corresponding βt satisfies βt,t = 1−αt and βt,m+1 = αt. Therefore, the risk of task t in SMTL can be written as the expectation E(x,y)∼µt [Lt(y, ht(βTt ϕ(x))]. We denote by E the average expected risk of all the tasks. Then, the minimal risk is defined as
E∗ = min ht∈H,βt∈Mt,ϕ∈F E = min ht∈H,βt∈Mt,ϕ∈F
1
m m∑ t=1 E(x,y)∼µtLt(y, ht(β T t ϕ(x))).
Then we obtain a generalization bound for the SMTL method with the proof in Appendix A. Theorem 1 (Generalization bound). Assume that Lt(·, ·) ∈ [0, 1] for t = 1, . . . ,m is 1-Lipschitz w.r.t the second argument, and the function ϕ in F is M -Lipschitz continuous. Then for (X̄, Ȳ) ∼∏m t=1(µt) n, with probability at least 1− δ, we have
E − E∗ ≤ C1MG(F(X̄)) + minz∈F(X̄) G(H′(z))
mn + 2QC2 supϕ∈F ‖ϕ(X̄)‖ n √ m +
√ 8 ln(4/δ)
mn , (5)
where C1 and C2 are two constants, F(X̄) = {(ϕ(xit)) : ϕ ∈ F}, ‖ · ‖ denotes the `2 norm, H′ = {z ∈ RmnP (m+1) 7→ ht(βTt z) : ht ∈ H, βt ∈ Mt}, G(·) denotes the Gaussian average, Qt is defined as
Qt = sup z 6=z̃∈RnP (m+1)
1
‖z − z̃‖E suph∈H,β∈Mt n∑ i=1 γi(h(β T zi)− h(βT z̃i)),

heading: Q = max1≤t≤mQt, and γ is a vector of independent standard normal variables.
text: In the generalization bound (5), the first term of the right-hand side can be regarded as the cost of estimating the feature map ϕ, the second term corresponds to the cost of estimating task-specific functions βt and ht, and the third term defines the confidence of the bound. The convergence rate of this bound is O( 1√
mn ), which is as tight as typical generalization bounds for MTL such as (Maurer
et al., 2016). Moreover, in Appendix B, we discuss some necessary condition for the optimal αt being 0 or 1.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: Experiments are conducted on four MTL benchmark datasets, including CityScapes (Cordts et al., 2016), NYUv2 (Silberman et al., 2012), PASCAL-Context (Mottaghi et al., 2014), and Taskonomy (Zamir et al., 2018). Detailed introductions of the four datasets are put in Appendix D.1.
The baseline methods in comparison include the Single-Task Learning (STL) that trains each task separately, the DMTL model which adopts the multi-head hard sharing architecture, Cross-stitch network (Misra et al., 2016), MTAN (Liu et al., 2019), NDDR-CNN (Gao et al., 2019), AdaShare (Sun et al., 2020), and AFA (Cui et al., 2021). For fair comparison, we use the same backbone for all the models in comparison.
For each task in the benchmark datasets, we use one or more evaluation metrics to thoroughly evaluate the performance. The detailed introduction of each evaluation metric is put in Appendix D.3. To better show the comparison between each method and STL, we report the relative performance of each method over STL in terms of the jth evaluation metric on task t as ∆t,j = (−1)pt,j (Mt,j − STLt,j), where for a method M, Mt,j denotes its performance in terms of the jth evaluation metric for task t, STLt,j is defined similarly, pt,j equals 1 if a lower value represents a better performance in terms of the jth metric in task t and 0 otherwise. So positive relative performance indicates better performance than STL, which is shown in green in the following tables, while worse performance corresponding to negative relative performance is shown in red. The overall relative improvement of a method M over STL is defined as ∆I = 1m ∑m t=1 1 mt ∑mt j=1 ∆t,j STLt,j
, where mt denotes the number of evaluation metrics in task t. Moreover, to measure the safeness of each method, the estimation η̂ of η in the definition of η-partially safe multi-task learning (i.e., Definition 3) for a method M is computed as η̂ = 1m ∑m t=1 1 mt ∑mt j=1 δ(∆t,j)× 100, where δ(x) is a step function that outputs 0 when x < 0 and otherwise 1.
We use the Deeplab-ResNet (Chen et al., 2017) with atrous convolutions, a popular architecture for pixel-wise prediction tasks, as encoders and the ASPP architecture (Chen et al., 2017) as decoders. We adopt the ResNet-50 for the CityScapes and NYUv2 datasets to implement the the DeeplabResNet, and use the smaller ResNet-18 for the larger PASCAL-Context and Taskonomy datasets for training efficiency. We use the cross-entropy loss for the semantic segmentation, human parts segmentation and saliency estimation tasks, the cosine similarity loss for the surface normal prediction task, and the L1 loss for other tasks. For optimization, we use the Adam method (Kingma & Ba, 2014) with the learning rate as 10−4. All the experiments are conducted on Tesla V100 GPUs.
4.2 EXPERIMENTAL RESULTS Table 1: Performance of various models on the CityScapes validation dataset. ↑ (↓) indicates the higher (lower) the result, the better the performance. The green color indicates that the corresponding method performs better than the STL method and the red color indicates oppositely.
Method Segmentation Depth ∆I ↑ η̂ ↑ mIoU ↑ Pix Acc ↑ Abs Err ↓ Rel Err↓
STL 67.48 91.00 0.0139 46.2507
DMTL −0.08 −0.08 −0.0003 +0.8245 −0.0014 25 Cross-stitch +0.53 +0.29 +0.0004 +1.8261 +0.0198 100 MTAN +1.49 +0.59 +0.0003 +2.4999 +0.0260 100 NDDR-CNN +0.54 +0.25 +0.0002 +1.3845 +0.0138 100 AdaShare +0.68 +0.20 −0.0005 −20.651 −0.1175 50 AFA +1.44 +0.52 −0.0019 −0.9136 −0.0323 50
SMTL +1.50 +0.62 +0.0005 +3.3886 +0.0346 100 L-SMTL +1.17 +0.51 +0.0001 +3.2747 +0.0252 100
SMTLc +1.62 +0.63 +0.0008 −1.9361 +0.0117 75 L-SMTLc +1.18 +0.38 +0.0008 +1.5060 +0.0279 100 Tables 1-4 show the performance of all the models in comparison on the four benchmark datasets. On the CityScapes dataset, the proposed SMTL, L-SMTL, L-SMTLc, and some baseline methods (i.e., Cross-stitch, MTAN, and NDDR-CNN) all achieve safe multi-task learning (i.e., η̂ = 100), which indicates that their performance is better than that of the STL model in all tasks. In addition, the proposed SMTL model achieves the best overall relative improvement ∆I , which demonstrates its effectiveness. On the NYUv2 and PASCAL-Context datasets, none of the baselines can achieve safe multi-task learning, but the proposed methods (i.e., SMTL, SMTLc, and L-SMTLc) can achieve that, which again shows the effectiveness of the proposed methods. Though the proposed L-SMTL method does not achieve safe multi-task learning on these two datasets, it still achieves a better η̂ than baseline methods on the NYUv2 dataset and a comparable η̂ on the PASCAL-Context dataset. On the PASCAL-Context dataset, the overall relative improvement of all the baselines are negative, while all the proposed methods achieves positive ∆I ’s, which shows the superiority of the proposed methods. On the Taskonomy dataset, only the Cross-stitch network and the proposed SMTL, SMTLc, and L-SMTLc methods can achieve safe multi-task learning. According to results shown in Table 4, we can see that the AFA method achieves the best ∆I because it has the largest improvements on the keypoint detection and edge detection tasks, but it does not achieve safe multitask learning, while the proposed methods can achieve that.
For the proposed methods, we can see that the lite versions (i.e., L-SMTL and L-SMTLc) perform comparable with SMTL and SMTLc, respectively, which suggests that the elimination strategy works well on the four datasets. By comparing SMTL with SMTLc, it seems that the performance is not so sensitive to the two positions of the gates, and similar observations hold for the L-SMTL and L-SMTLc methods. Moreover, the proposed SMTL and L-SMTLc methods can achieve safe multi-task learning on all the four datasets, while the proposed L-SMTL and SMTLc methods fail to achieve this. This observation may suggest the SMTL and L-SMTLc methods are more preferred than others.
4.3 ANALYSIS ON LEARNED {αt}
We record the learned {αt} of the proposed models in Table 5. According to results for the SMTL and SMTLc methods, we can see that some αt’s are closed to 0.5, which means in those cases, the public encoder and the private encoder are both important to the entire model. Thus, only using the public encoder (i.e., DMTL) and only using the private encoder (i.e., STL) cannot achieve good
performance, while the proposed models can take the advantages of these two methods to achieve better performance in most cases. Moreover, some of the learned αt’s have relatively small values (i.e., values smaller than 0.3), which are shown in box in Table 5. These small values indicate that for the surface normal prediction task on the NYUv2, PASCAL-Context, and Taskonomy datasets, the public encoder is relatively unimportant, which may imply that the surface normal prediction task is not strongly related to other tasks on these datasets. On the other hand, this observation may explain why DMTL is much worse than STL and why the proposed SMTL method has good performance on these datasets (refer to Tables 2-4).
For the L-SMTL and L-SMTLc methods, when the learned αt is larger than 0.5, the corresponding task t will choose to use the public component and otherwise choose the corresponding private component at the inference stage. According to Table 5, we can see that in some cases, all the tasks will choose private components on some datasets (i.e., both methods on the CityScapes and Taskonomy datasets, and L-SMTL on the NYUv2 dataset) and other cases are mixed in that some tasks choose the public component and other tasks choose private components. Interestingly, for both methods, the surface normal prediction task chooses to use private components on the NYUv2, PASCAL-Context, and Taskonomy datasets, which corresponds to the small values for αt in the SMTL method.

heading: 4.4 EXPERIMENTS ON BI-LEVEL FORMULATION
text: In this section, we use a bi-level formulation for the SMTL model to study the effects of different formulations. Specifically, the original training dataset is divided into two parts, including a training set with ntr data points and a validation set with nval data points. The training set is used to learn parameters in the public encoder, m private encoders, andm private decoders, which corresponds to the lower-level subproblem in the following problem (6). Parameters in the m gates are viewed as hyperparameters and the validation set is used to learn them, which is corresponding to the upperlevel subproblem in problem (6). The objective function of SMTL under the bi-level formulation is formulated as
min {gt}
1
mnval nval∑ i=1 m∑ t=1 Lt(ỹit, h∗t (gt(f∗S(x̃i), f∗t (x̃i))))
s.t. f∗S , {f∗t }, {h∗t } = arg min fS ,{ft},{ht}
1
mntr ntr∑ i=1 m∑ t=1 Lt(ȳit, ht(gt(fS(x̄i), ft(x̄i)))), (6)
where x̄i denotes the ith data point in the training set, ȳit denotes the corresponding label of x̄i in task t, and x̃i as well as ỹit is defined similarly in the validation set. Then we conduct experiments on the CityScapes and NYUv2 datasets to compare the performance of SMTL under different formulations. According to experimental results shown in Table 6, we can see that, on the CityScapes dataset, the learned {αt} by the bi-level formulation (i.e., problem (6)) is similar to that by the singlelevel formulation (i.e., problem (2)), thus the performance has no much difference. However, on the NYUv2 dataset, SMTL with the bi-level formulation learns a large αt for the surface normal prediction task, which makes its performance inferior to SMTL with the single-level formulation in some tasks (i.e., depth estimation and surface normal prediction). One reason is that the surface normal prediction task is not so strongly related to other tasks that learning them together may impair not only its own performance but also the performance of other tasks. Moreover, it is well known that the complexity of solving a bi-level optimization problem is much higher than that of solving the corresponding single-level optimization problem and so experiments on larger datasets (i.e., PASCAL-Context and Taskonomy) based on problem (6) are too computational demanding to be conducted. Hence, the single-level formulation of the SMTL model (i.e., problem (2)) is preferred as it is both effective and efficient. For other variants of SMTL, we have similar observations so that we do not report the results for them.

heading: 5 CONCLUSION
text: In this paper, to study the problem of safe multi-task learning, we propose a simple and effective SMTL method that can automatically learn to combine encoders via a gating mechanism. To reduce the storage cost, we design lite SMTL by learning a binary gate. Furthermore, we study to place the gates after the decoders. Extensive evaluations demonstrate the effectiveness of the proposed methods. In the future work, we are interested in identifying the location for the gates via neural architecture search.
ETHICS STATEMENT
We have read the Code of Ethics and ensure that this work follows it. No human subject is involved in this work and all the datasets used in experiments are publicly available such that they do not contain any personally identifiable information or offensive content. Hence, this work has no potentially negative societal impact.
REPRODUCIBILITY STATEMENT
For theoretic results, assumptions used have been fully stated and the complete proof is included in the Appendix. We include all the code and instructions in the supplementary material so that this work can be reproduced. The implementation details are provided in Section 4.1 and Appendix.

heading: A PROOF OF THEOREM 1
text: In this section, we analyze the generalization bound for the SMTL method.
To analyze the generalization bound for SMTL, we introduce a useful theorem in terms of Gaussian averages (Bartlett & Mendelson, 2002; Maurer et al., 2016). Theorem 2. Let G be a class of functions Ψ : X → [0, 1]T , and µ1, ..., µm be the probability measure on X with X̄ = (X1, ...,Xm) ∼ ∏m t=1(µt) n where Xt = (x1t , . . . ,x n t ). Let Z be the random set {(Ψt(xit)) : Ψ ∈ G} and γ be a vector of independent standard normal variables. Then for all Ψ ∈ G, with probability at least 1− δ in X̄, we have
1
m ∑ t ( Ex∼µt [Ψt(x)]− 1 n ∑ i Ψt(x i t) ) ≤ √ 2πG(Z) mn + √ 9(ln(2/δ)) 2mn ,
where G(Z) = E[supz∈Z 〈γ, z〉] is the Gaussian average of the set Z.
Based on Theorem 2, in the following section, we give the proof of Theorem 1. Proof. By Theorem 2, with probability at least 1 − δ in (X̄, Ȳ) ∼ ∏m t=1(µt)
n, for all ht ∈ H, αt ∈Mt and ϕ ∈ F , we have
E − 1 mn ∑ ti Lt(yit, ht(βTt ϕ(xit))) ≤ √ 2πG(S) mn +
√ 9(ln(2/δ))
2mn , (7)
where S = {(Lt(yit, ht(βTt ϕ(xit)))) : ht ∈ H, αt ∈Mt, ϕ ∈ F} andG(S) is the Gaussian average of the set S. Then by the Lipschitz property of Lt and Slepian’s Lemma (Ledoux & Talagrand, 2013), we have G(S) ≤ G(S′), where S′ = {(ht(βTt ϕ(xit))) : ht ∈ H, αt ∈Mt, ϕ ∈ F}.
Note that the input data X̄ ∈ Xmn, and hence F(X̄) ⊆ RmnP (m+1) is defined as F(x̄) = {(ϕ(xit)) : ϕ ∈ F}. We define a class of functions H′ = {z ∈ RmnP (m+1) 7→ ht(βTt z) : ht ∈ H, βt ∈ Mt}. Then we have S′ = H′(F(X̄)). By using Theorem 2 in (Maurer, 2016), we obtain the following inequality
G(S′) ≤ C1L(H′)G(F(X̄)) + C2D(F(X̄))Q(H′) + min z∈F(X̄) G(H′(z))
where C1 and C2 are two constants, L(H′) represent the Lipschitz constant of the functions in H′ , D(F(x̄)) = 2 supϕ∈F ‖ϕ(X̄)‖ denotes the Euclidean diameter of the set F(X̄), and
Q(H′) = sup z 6=z̃∈RmnP (m+1)
1
‖z − z̃‖ E sup ψ∈H′ 〈γ, ψ(z)− ψ(z̃)〉 .
Let z, z̃ ∈ RmnP (m+1), where z = (zit) with zit ∈ RP (m+1) and z̃ = (z̃it) with z̃it ∈ RP (m+1). Then for any functions ψ ∈ H′, we have
E sup ψ∈H′ 〈γ, ψ(z)− ψ(z̃)〉 = E sup ht∈H,βt∈Mt ∑ ti 〈 γti, ht(β T t z i t)− ht(βTt z̃it) 〉 =
m∑ t=1 E sup h∈H,βt∈Mt n∑ i=1 γi(h(β T t z i t)− h(βTt z̃it))
≤ √ m  m∑ t=1 ( E sup h∈H,βt∈Mt n∑ i=1 γi(h(β T t z i t)− h(βTt z̃it)) )21/2
≤ √ m ( m∑ t=1 Q2max n∑ i=1 ‖zit − z̃it‖2 )1/2 ≤ √ mQmax‖z − z̃‖,
where Qmax = max1≤t≤mQt. Therefore, Q(H′) ≤ √ mQmax. Moreover, we have
‖ψ(z)− ψ(z̃)‖2 = ∑ ti (ht(β T t z i t)− ht(βTt z̃it))
≤M2 ∑ ti ‖βTt zit − βTt z̃it‖2 ≤M2‖βTt ‖2‖z − z̃‖2
where the first inequality is due to the Lipschitz property and the second inequality is due to the Cauchy-Schwarz inequality. Note that ‖βt‖ ≤ 1, hence L(H′) ≤M . Then, we have
G(S) ≤ G(S′) ≤ C1MG(F(X̄)) + 2C2 √ mQmax sup
ϕ∈F ‖ϕ(X̄)‖+ min z∈F(X̄) G(H′(z)). (8)
Let ϕ∗, h∗1, ..., h ∗ m, β ∗ 1 , ..., β ∗ m be the minimizer in E∗, then we have
E − E∗ = ( E − 1
mn ∑ ti Lt(yit, ht(βTt ϕ(xit)))
) + ( 1
mn ∑ ti Lt(yit, h∗t (β∗Tt ϕ∗(xit)))− E∗ )
+
( 1
mn ∑ ti Lt(yit, ht(βTt ϕ(xit)))− 1 mn ∑ ti Lt(yit, h∗t (β∗Tt ϕ∗(xit)))
) , (9)
where the first term can be bounded by substituting inequality (8) into (7) and the second term can be regarded as mn random variables Lt(yit, h∗t (β∗Tt ϕ∗(xit))) with values in [0, 1]. By using Hoeffding’s inequality, with probability at least 1− δ, we have
1
mn ∑ ti Lt(yit, h∗t (β∗Tt ϕ∗(xit)))− E∗ ≤ √ ln(1/δ) 2mn .
The last term is non-positive due to the definition of minimizers. Therefore, we have
E − E∗ ≤ C1MG(F(X̄)) + minz∈F(X̄)G(H′(z))
mn + 2QC2 supϕ∈F ‖ϕ(X̄)‖ n √ m +
√ 8 ln(4/δ)
mn .
B NECESSARY CONDITION FOR OPTIMAL αt
We analyze the conditions that model parameters except {αt} satisfy when the optimal αt equals 0 or 1 for each task. By taking the cross-entropy loss as an example, we have following result. Theorem 3. When Lt is the cross-entropy loss of task t that is formulated as Lt = − 1n ∑n i=1(y i t) T log ht(αtfS(xi) + (1 − αt)ft(xi)) where yit denotes the one-hot label vector, if α∗t is a local minimum of Lt over αt, then we have n∑ i=1 (yit) T ft(xi)(fS(xi)− ft(xi)) ht(ft(xi)) ≤ 0, if α∗t = 0,
n∑ i=1 (yit) T fS(xi)(fS(xi)− ft(xi)) ht(fS(xi)) ≥ 0, if α∗t = 1,
Proof. Fix αt and define the function Ht( ) = Lt(α∗t + (αt − α∗t )), which is continuously differentiable in an open interval containing [0, 1]. By using the chain rule to differentiate Ht, we have
0 ≤ lim →0
Lt(α∗t + (αt − α∗t ))− Lt(α∗t )
= dHt(0) d = ∂Lt(α∗t ) ∂αt (αt − α∗t )
where the inequality follows from the assumption that α∗t is a local minimum.
If α∗t = 0, then αt − α∗t ≥ 0. To satisfy ∂Lt(α∗t ) ∂αt (αt − α∗t ) ≥ 0, we have ∂Lt(α∗t ) ∂αt ≥ 0.
If α∗t = 1, then αt − α∗t ≤ 0. To satisfy ∂Lt(α∗t ) ∂αt (αt − α∗t ) ≥ 0, we have ∂Lt(α∗t ) ∂αt ≤ 0.
Based on the formulation of Lt, we can compute ∂Lt(α ∗ t ) ∂αt as ∂Lt(α ∗ t ) ∂αt = − 1n ∑n i=1 (yit) T (α∗t fS(xi)+(1−α ∗ t )ft(xi))(fS(xi)−ft(xi)) ht(α∗t fS(xi)+(1−α∗t )(ft(xi))) .
Then we have n∑ i=1 (yit) T ft(xi)(fS(xi)− ft(xi)) ht(ft(xi)) ≤ 0, if α∗t = 0,
n∑ i=1 (yit) T fS(xi)(fS(xi)− ft(xi)) ht(fS(xi)) ≥ 0, if α∗t = 1,
in which we reach the conclusion.
Theorem 3 provide a necessary condition for the optimal αt being 0 or 1. Such analysis can easily be extended to other loss functions such as the square loss, and we omit it due to similar proofs.
C ILLUSTRATION OF THE SMTLc AND L-SMTLc MODELS
Similar to the SMTL and L-SMTL models, an illustration of the SMTLc and L-SMTLc models is shown in Figure 2.

heading: D EXPERIMENTAL SETUP
text: D.1 DETAILS OF DATASETS
CityScapes. The CityScapes dataset (Cordts et al., 2016) consists of high resolution outside streetview images, which contains 2, 975 images for training and 500 images for validation. This dataset contains 19 classes for pixel-wise semantic segmentation, together with ground-truth inverse depth labels. By following (Liu et al., 2019), we evaluate the performance on the 7-class semantic segmentation and depth estimation tasks. All training and validation images are resized to 128× 256.
NYUv2. The NYUv2 dataset (Silberman et al., 2012) consisting of RGB-D indoor scene images has 795 images for training and 654 images for validation. We evaluate the performance on three learning tasks: 13-class semantic segmentation, depth estimation, and surface normal prediction. By following (Liu et al., 2019), all the training and validation images were resized to 288× 384. PASCAL-Context. The PASCAL-Context dataset (Mottaghi et al., 2014) is an annotation extension of the PASCAL VOC 2010 challenge and it contains 4, 998 images for training and 5, 105 images for validation. We evaluate the performance on four learning tasks: 21-class semantic segmentation, 7-class human parts segmentation, saliency estimation, and surface normal estimation, where the last two tasks are generated by (Maninis et al., 2019).
Taskonomy. The Taskonomy dataset (Zamir et al., 2018) which contains over 4.5 million indoor images from over 5, 000 buildings with 26 tasks. By following (Standley et al., 2020), we sample five learning tasks, including 17-class semantic segmentation, depth estimation, keypoint detection, edge detection, and surface normal prediction. Furthermore, we select 5 building images (i.e., “allensville”, “collierville”, “mifflinburg”, “noxapater”, and “onaga”) from the standard tiny benchmark as our dataset, which contains 13, 286 images for training and 3, 794 images for validation.
D.2 SETTINGS OF BATCH SIZE
The settings of the batch size for all the models on different datasets are shown in Table 7.
D.3 EVALUATION METRICS
On the PASCAL-Context dataset, by following (Maninis et al., 2019), the semantic segmentation is evaluated by the mean Intersection over Union (mIoU) and on the other three datasets, by following (Sun et al., 2020), this task is additionally evaluated by the Pixel Accuracy (Pix Acc). For the depth estimation task, the absolute error (Abs Err) and relative error (Rel Err) are used as the evaluation metrics. For the surface normal prediction task, the mean and median angle distances between the prediction and ground truth of all pixels are used as measures. For this task, the percentage of pixels, whose prediction is within the angles of 11.25◦, 22.5◦, and 30◦to the ground truth, is used as another measure. For the keypoint detection and edge detection tasks, the absolute error (Abs Err) is used as the evaluation metric. For the human parts segmentation task, the mIoU is used as the measure. For the saliency estimation task, the mIoU and max F-measure (maxF) are adopted as the evaluation metrics.

heading: E COMPARISON OF TRAINING TIME
text: The training time per epoch for all the models on the CityScapes dataset is recorded in Table 8. According to the results, the training time of the proposed SMTL and L-SMTL methods is comparable with all the baseline methods, implying that the proposed SMTL and L-SMTL methods are as efficient as baseline methods. The training time of the proposed SMTLc and L-SMTLc methods is a bit longer due to additional public decoders introduced.

Review summaries:
The paper propose a simple method for safe multi-task learning where there is no negative transfer or "negative sharing" among tasks. It jointly trains shared encoder, task-specific (private) encoders, gate, and decoder. The gate computes importance for each output of shared and task specific encoder and combine the outputs with simple convex combination. 

This paper presents a multi-task learning approach that avoids negative sharing in training deep neural networks.  A novel network architecture is proposed, which consists of a public encoder shared by all the tasks, private encoder for each task, and a gate for each task to combine encoded features from public and private encoders. Their experiment results indicate the proposed approach is effective on image recognition related tasks.

This paper focuses on the negative sharing problem in multi-task learning, which has not been studied sufficiently in existing work. The authors propose the Safe Multi-Task Learning (SMTL) model and several of its variants to avoid negative sharing and achieve safe multi-task learning. Both theoretical analysis and comprehensive experimental results are provided to demonstrate the effectiveness of the proposed method.

This paper aims to solve the negative sharing problem, which is defined as that a multi-task learning model has inferior performance than single-task learning on some tasks, for multi-task learning. To address negative sharing, Safe Multi-Task Learning (SMTL) model is proposed. The model combines hard-sharing model and single-task learning together and is expected to achieve performance that not inferior than single-task learning.