Understanding and Improving Information Transfer in Multi-Task Learning
Authors: Sen Wu
Hongyang R. Zhang
Christopher Ré
Venue:  (2020)
Keywords: Multi-Task Learning

Abstract: We investigate multi-task learning approaches that use a shared feature representation for all tasks. To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task. We study the theory of this setting on linear and ReLU-activated models. Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtained a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT LARGE using our alignment method. We also design an SVD-based task re-weighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.

Published as a conference paper at ICLR 2020
UNDERSTANDING AND IMPROVING INFORMATION TRANSFER IN -ULTI-TASK LEARNING
Sen Wu"
Hongyang R. Zhang' University of Pennsylvania
Christopher Ré Stanford University
Stanford University
ABSTRACT
We investigate multi-task learning approaches that shared feature represen- tation for all tasks. To better understand the transfer of task information, we study architecture with a shared module for all tasks and separate output module for each task. We study the theory of this setting on linear and ReLU-activated models. Our key observation that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning. We show that mis- alignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoreti- cal insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning the GLUE benchmark and sentiment analysis tasks; for example, we obtain a 2.35% GLUE score average improvement on 5 GLUE tasks over BERTLARGE using our alignment method. We also design an SVD-baser task reweighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.
INTRODUCTION
Multi-task learning has recently emerged as powerful paradigm in deep learning to obtain lan- guage (Devlin et al. (2018); Liu et al. (2019a:b)) and visual representations (Kokkinos (2017)) from large-scale data. By leveraging supervised data from related tasks, multi-task learning approaches reduce the expensive cost of curating the massive per-task training data sets needed by deep learning methods and provide a shared representation which is also more efficient for learning over multiple tasks. While in some cases, great improvements have been reported compared to single-task learning (McCann et al. (2018)), practitioners have also observed problematic outcomes, where the perfor- mances of certain tasks have decreased due to task interference (Alonso and Plank (2016); Bingel and Søgaard (2017)). Predicting when and for which tasks this occurs is challenge exacerbated by the lack of analytic tools. In this work, we investigate key components to determine whether tasks interfere constructively or destructively from theoretical and empirical perspectives. Based on these insights, we develop methods to improve the effectiveness and robustness of multi-task training.
There has been large body of algorithmic and theoretical studies for kernel-based multi-task learn- ing, but less is known for neural networks. The conceptual message from the earlier work (Bax- ter (2000); Evgeniou and Pontil (2004); Micchelli and Pontil (2005); Xue et (2007)) show that multi-task learning is effective over "similar" tasks, where the notion of similarity is based on the single-task models (e.g. decision boundaries are close). The work on structural correspondence learning (Ando and Zhang (2005); Blitzer et al. (2006)) uses alternating minimization to learn shared parameter and separate task parameters. Zhang and Yeung (2014) use a parameter vector for each task and learn task relationships via le regularization, which implicitly controls the capacity of the model. These results are difficult to apply to neural networks: it is unclear how to reason about neural networks whose feature space is given by layer-wise embeddings.
To determine whether two tasks interfere constructively or destructively, we investigate an architec- ture with a shared module for all tasks and separate output module for each task (Ruder (2017)). See Figure for an illustration. Our motivating observation is that in addition to model similarity which affects the type of interference, task data similarity plays a second-order effect after control- ling model similarity. To illustrate the idea, we consider three tasks with the same number of data
Equal contribution. Correspondence (Senwu.hongyanggchriimne @cs stanford.edu

---

Published as a conference paper at ICLR 2020
More Points
Fewer points
Task training set
Task
Task
3
Task-Spocific Modules (A)
Tesk Validation Set
Shared Modules
Task no
Task
Taak
x
%
X
X
X2
Acc.: 94.8
Acc.: 98.0
ACc.: 90LB
Multiplc Tasks (X)
STL: Task 1
MTL: Task & Task
MTL: Task Task 3
Figure I: An illustration of the multi-task learning architecture with a shared lower module B and task-specific modules (A: [A.
Figure 2: Positive vs. Negative transfer is affected by the data not just the model. See lower right-vs-mid. Task 2 and have the same model (dotted lines) but different data distribu- tions. Notice the difference of data in circled areas.
samples where task 2 and 3 have the same decision boundary but different data distributions (see Figure 2 for an illustration). We observe that training task with task 2 or task 3 can either im- prove or hurt task I's performance, depending on the amount of contributing data along the decision boundary! This observation shows that by measuring the similarities of the task data and the models separately, we can analyze the interference of tasks and attribute the cause more precisely.
Motivated by the above observation, study the theory of multi-task learning through the shared module in linear and Rel.U-activated settings. Our theoretical contribution involves three compo- nents: the capacity of the shared module, fask covariance, and the per-task weight of the training procedure. The capacity plays a fundamental role because, if the shared module's capacity is too large, there is interference between tasks; if it is too small, there Then, we show how to determine interference by proposing a more fine-grain ane-ggrined notion called task covariance which can be used to measure the alignment of task data. By varying task covariances, we observe both positive and negative transfers from one task to another! We then provide sufficient conditions which guarantee that ciently many data points from the contributor task. Finally, we study how to assign per-task weights for settings where different tasks share the same data but have different labels.
be destructive interference.
task transfer positively to another task, provided with suffi-
Experimental results. Our theory leads to the design of two algorithms with practical interest. First, we propose align the covariances of the task embedding layers and present empirical evalu- ations on well-known benchmarks and tasks. On tasks from the General Language Understanding Evaluation (GLUE) benchmark (Wang et (2018b)) trained with the BERTLARGE model by De- vlin et al. (2018), our method improves the result of BERTLARGE by a 2.35% average GLUE score, which the standard metric for the benchmark. Further, we show that our method is applicable to transfer learning settings; observe up to 2.5% higher accuracy by transferring between six sentiment analysis tasks using the LSTM model of Lei et al. (2018).
Second, we propose SVD-based task reweighting scheme to improve multi-task training for set- tings where different tasks have the same features but different labels. On the ChestX 14 dataset, compare our method to the unweighted scheme and observe improvement of 0.4% AUC score on average for all tasks In conclusion, these evaluations confirm that our theoretical insights are applicable to broad range of settings and applications.
THREE COMPONENTS OF MULTI-TASK LEARNING
We study multi-task learning (MTL) models with a shared module for all tasks and separate output module for each task. We ask: What are the key components to determine whether or not MTL is better than single-task learning (STL)? In response, we identify three components: model capacity, task covariance, and optimization scheme. After setting up the model, we briefly describe the role of model capacity. We then introduce the notion of task covariance, which comprises the bulk of the section. We finish by showing the implications of our results for choosing optimization schemes.
2

---

Published as a conference paper at ICLR 2020
2.1 MODELING SETUP
We are given tasks. Let m, denote the number of data samples of task i. For task i, let X; E Rm denote its covariates and let y/4 R denote its labels, where d is the dimension of the data. We have assumed that all the tasks have the same input dimension d. This is not a restrictive assumption and typically satisfied, e-g- for word embeddings on BERT, or by padding zeros to the input otherwise. Our model assumes the output label is 1-dimensional. We can also model multi-label problem with types of labels by having & tasks with the same covariates but different labels. We consider an MTL model with shared module B E _dd and separate output module A; E Rº for task i, where r denotes the output dimension of B. See Figure for the illustration. We define the objective of finding MTL model as minimizing the following equation over B and the A, s:
f(A1, Ax: B)
Σ L Lg(X v.),
where L is loss function such as the squared loss. The activation function g R is applied on every entry of X,B. In equation 1, all data samples contribute equally. Because of the differences between tasks such as data size, it is natural to re-weight tasks during training:
A B) = Σ L()
()
This setup is abstraction of the hard parameter sharing architecture (Ruder (2017)). The shared module provides a universal representation (e-g.. an L.TTM for encoding sentences) for all tasks. Each task-specific module A, is optimized for its output. We focus two models follows.
The single-task linear model. The labels y of each task follow a linear model with parameter 0 E R4: y -00 + Every entry of follows the normal distribution (O, o²) with variance o², The function ( XB. This is well-studied setting for linear regression (Hastie et al. (2005)).
The single-task task ReLU model. Denote by ReLU(z) max(z, 0) for any x R. We will also consider a non-linear model where X goes through the ReLU activation function with E and Re + which applics the ReLU activation on X0 entrywise. The encoding
/ E :2 function g( X B) then maps to R)).
Positive vs. negative transfer. For source task and target task, say the source task transfers positively to the target task, if training both through equation improves over just training the target task (measured on its validation set). Negative transfer is the converse of positive transfer.
Problem statement. Our goal is to analyze the three components to determine positive vs. negative transfer between tasks: model capacity (r), task covariances ([X; x.) and the per-task weights 1/ (fax), ). We focus regression tasks under the squared loss but also provide synthetic exper-
iments on classification tasks to validate our theory.
Notations. For matrix X, its column span the set of all linear combinations of the column vectors of X. Let X denote its pseudoinverse. Given u, v E IRd, cos(u, u) is equal to uT TTo) //[[]]]] /|]])-
2.2 MODEL CAPACITY
We begin by revisiting the role of model capacity, i.e. the output dimension of B (denoted by r). We show that as rule of thumb, should be smaller than the sum of capacities of the STL modules.
Example. Suppose we have linear regression tasks using the squared loss, equation becomes:
A(A,, A B) = Σ X.BAA, 1
(3)
The optimal solution of equation for task i is 0 = xx) y; E Rd. Hence a capacity of suffices for each task. We show that if r > k, then there is transfer between any two tasks.
3

---

Published as a conference paper at ICLR 2020
Source target (Task 1):
Same covariance (Source: Task
Difi. covariance (Source: Task 3)
Target task's Perf.: MTL STL
Positive Transter
Positive Transfer
Positive Transfer
0.1
0.1
0-
-0.1
Negative Transter
Negative Transfer
Negative Transfer
-0.2
-02
2000
4000
8000
8000
2000
4000
8000
8000
2000
4000
6000
8000
# Data samples of source task
Data samples of source task
# Data samples of source task
(a) Linear regression tasks
(b) Logistic classification tasks
(c) ReLU regression tasks
Figure 3: Performance improvement of a target task (Task by MTL with a source task vs. STL. Red: positive transfer when the source is Task 2, which has the same covariance matrix with target. Green: negative (to positive) transfer when the source is Task 3, which has a different covariance from the target, its # of samples increases. See the example below for the definition of each task.
Proposition 1. Let r 2 k. There exists an optimum B and {AT of equution 3 where B A 0, for all i 1.2....,4
To illustrate the idea, as long as B contains {0:) in column span, there exists A such that A 0,. which is optimal for equation 3 with minimum error. But this means transfer among any two tasks. This can hurt generalization if task has limited data, in which case its STL solution overfits training data, whereas the MTL solution can leverage other tasks' data to improve generalization. The proof Proposition 1 and its extension to ReLU settings are in Appendix A.1.
Algorithmic consequence. The implication is that limiting the shared module's capacity is nec- essary to enforce information transfer. If the shared module is too small, then tasks may interfere negatively with each other. But if it is too large, then there may be transfer between tasks. In Sec- tion 3.3, we verify the need to carefully choose model capacity on a wide range of neural networks including CNN, LSTM and multi-layer perceptron.
2.3 TASK COVARIANCE
To show how to quantify task data similarity, we illustrate with two regression tasks under the linear limit the
model without noise: y1 = 0 and y/2 X202. By Section 2.2, it is necessary capacity of the shared module to enforce information transfer. Therefore, we consider the case of r Hence, the shared module B is now d-dimensional vector, and A₁, A2 are both scalars.
A natural requirement of task similarity is for the STL models to be similar, i.c. |cos(01, 1 to be large. To see this, the optimal STL model for task is (X X1) -XTT 31 01 Hence if [cos(0], 02)l is 1, then tasks 1 and 2 can share model B E Rdd which is either 01 or - -1. The scalar A₁ and As can then transform B to equal to 0, and 0z.
Is this requirement sufficient? Recall that in equation 3, the task data X₁ and X₂ are both multiplied by B. If they are poorly "aligned" geometrically, the performance could suffer. How do we formal- ize the geometry between task aligmment? In the following, we show that the covariance matrices of X, and X2, which we define to be X X1 and X, X2, captures the geometry. be close to I to examine the effects of task covariances. In Appendix A.2. we fix task covariances to examine the effects of model cosine similarity. Concretely, equation 3 reduces to:
fix coss(0,, , 02) to
X,B
X²B
max ((B) = |X vI) + BER
|X, v2)
(4)
where we apply the first-order optimality condition on A₁ and A2 and simplify the equation. Specif- ically, we focus on a scenario where task 1 is the source and task 2 is the target. Our goag is to negatively in MTL. Determining the
determine when the source transfers the target positively type of transfer from task 2 to task can be done similarly. Answering the question boils down to studying the angle cosine similarity between the optimum of equation 4 and 02.
Example. In Figure 3, we show that by varying task covariances and the number samples, we can observe both positive and negative transfers. The conceptual message is the same Figure 2;
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 Covariance alignment for multi-task training
Require: Task embedding layers X1 [em1x xd X2 [e ....,, Xx E R shared module B Parameter: Alignment matrices Ri, R2...., Rs E gaxd and output modules A2.42.... . E R I: Let Z, = Rt, for 1 I i k.
Consider the following modified loss (with B being fixed):
f(A1,..., As; Ri,..., Rs) = Ei. B)) Ui) = Ei= Rg vk) Minimize f by alternatively applying gradient descent update on A, and Rú, given a sampled data batch from task i. Other implementation details are described in Appendix B.3.
2:
describe the data generation process in more detail. We use 3 tasks and measure the type of transfer from the souree to the target. The r-axis is the number of data samples from the source. The y-axis is the target's performance improvement measured on its validation set between MTL minus STL.
Data generation. We have (co, (2)) Re 1 (say 0.96). For i 21, let R2 m. denote a random Gaussian matrix drawn from (0, 1). Let Si, S2 S (1,2,..... d) be two disjoint sets of size d/10. For 1, 2. let D; be diagonal matrix whose entries are equal
large value K (e.g. 100) for coordinates in S; and 1 otherwise. Let Qi C Radd denote an orthonormal matrix, i.c. Q Q equal the identity matrix, orthogonalized from a random Gaussian matrix.
Then, we define the 3 tasks as follows. (i) Task 1 (target): X1 = Rı Q1 D1 and y1 = X101. (ii Task 2 (source task for red line): X2 R2Qi and y/2 X202. (ii) Task (source task for green line): X3 R3Q2 D2 and y/3 = X302. Task 1 and 2 have the same covariance matrices but task and 3 have different covariance matrices. Intuitively, the signals of task 1 and 3 lie in different subspaces, which arise from the difference in the diagonals of D, and the orthonormal matrices.
Analysis. Unless the source task has lots of samples to estimate 02, which is much more than the samples needed to estimate only the coordinates of Si, the effect of transferring to the target is small. We observe similar results for logistic regression tasks and for ReL.U-activate regression tasks.
Theory. We rigorously quantify how many data points is needed to guarantee positive transfer. The folklore in MTL is that when a source task has a lot of data but the related target task has limited data, then the source can often transfer positively to the target task. Our previous example shows that by varying the source's number of samples and its covariance, we can observe both types of transfer. How much data do we need from the source to guarantee positive transfer the target? We show that this depends on the condition numbers of both tasks' covariances.
Theorem 2 (informal). For = 1,2, let y; = X,0; + E, denote two linear regression tasks with parameters 02 E Rd and m, number of samples. Suppose that each of the source task X, is drawn independently from a distribution with covariance Σ₁ C Rddd and bounded l2-norm. Let (x)))si((,,, With high probability, when my is at least on the order of (k²(2)n -x (x())) 119221?)//t we have
s( (22 )in (2)) and assume that S 1/3. Denote by (B* Ai, Ab) the optimal MTL solution.
1
|
|B* A 0211/110221 S Gc +
(5)
1- 3c
|2202||
Recall that for matrix X, x(X) denotes its condition number. Theorem 2 quantifies the trend in Figure 3, where the improvements for task 2 reaches the plateau when my becomes large enough.
The parameter here indicates how similar the two tasks are. The smaller sin(O1, 02) is, the smaller e is. As an example, if sin(01,0) 02) I 5/rz(X, for some ó, then equation 5 is at most (()) + ||e2l//||Xx@2|| The formal statement, its proof and discussions the assumptions are deferred to Appendix A.2.2.
The ReL.U model. We show similar result for the ReL.U model, which requires resolving the chal- lenge of analyzing the ReLU function. We use a geometric characterization for the ReI.U function under distributional input assumptions by Du et al. (2017). The result deferred to Appendix A.2.3.
'The estimation error of 02 is upper bounded by task 2's signal to noise ratio dence arises because the linear component A fits the projection of y/2 to B² So even if 1² is equal to 0z, there could still be an estimation error out of As, which cannot be estimated from task 1 data.
This depen-
5

---

Published as a conference paper at ICLR 2020
Algorithm 2 An SVD-based task re- re-weighting scheme
Input: tasks: (X, E ([erxd ,I"")): rank parameter (L,
Output: A weight vector: {an, aa,....... Onk)
I: Let 0; X Vi.
2: Ur, Dr, V. SVD, (01, 02,..., 0z), i.c. the best rank-r approximation the 0:
3: ae = |10 U-11, for i = 1.
Algorithmic consequence. An implication of our theory is covariance alignment method to im- prove multi-task training. For the i-th task, we add an alignment matrix R, before its input X; passes through the shared module B. Algorithm shows the procedure.
We also propose metric called covariance similarity score to measure the similarity between two tasks. Given X1 R xd and X2 E Rmaxx,, we measure their similarity in three steps: (a) The covariance matrix is X1 X1. (b) Find the best rank-71 approximation to be U1.r, Di,r, UT,,, where is chosen to contain 99% of the singular values. (c) Apply step (a).(b to X2, compute the score:
[[U1,,, U2 1/2 101,r, 1/2 ll
Covariance similarity score
2,r2
(6)
1122.r, D 2/2 ll.
The nice property of the score is that is invariant to rotations of the columns of X₁ and X2.
2.4 OPTIMIZATION SCIIEME
Lastly, consider the effect of re- re-weighting the tasks (or their losses in equation 2). When does weighting the tasks help? In this part, we show a use case for improving the robustness of multi-task training the presence of label noise. The settings involving label noise arise when some tasks only have -upeevisee labels, which have been studied before in the literature (e.g. Mintz al. (2009); Pentina and Lampert (2017)). We start by describing motivating example.
Consider two tasks where task 1 is y1 = X0 and task 2 is 3/2 = X0 + 82- we train the two tasks together, the error E2 will add noise to the trained model. However, by up weighting task 1, we reduce the noise from task 2 and get better performance. To rigorously study the effect of task weights, we consider setting where all the tasks have the same data but different labels. This setting arises for example in multi-label image tasks. We derive the optimal solution in the linear model.
Proposition 3. Let the shared module have capacity r k. Given i tasks with the same covariates X C Rm but different labels by. Let X be full rank and U DVT be its SVD. Let Q,Q, be the best rank-r approximation to Σ aUU yoy, U. Let B pdxr be optimal solution for the re-weighted loss. Then the column span of B" is equal to the column span of (X) -)-VDDQ..
We can also extend Proposition 3 to show that all local minima of equation are global minima in the linear setting. We leave the proof to Appendix A.3. We remark that this result does not extend to the non-linear ReLU setting and leave this for future work.
Based on Proposition 3, we provide rigorous proof of the previous example. Suppose that X is full rank, (X) X))X((a)), an y2]) [o.00, 0220 + ox2(X X)e). Hence, when we increase a1 cOs( 0) increases closer to 1.
Algorithmic consequence. Inspired by our theory, we describe a re-weighting scheme in the pres- ence of label noise. We compute the per-task weights by computing the SVD over X yı. for 1 i & k. The intuition that the label vector of a task y; is noisy, then the entropy of ys is small. Therefore, does this. where the weight of a task is calculated by its projection into the principal r directions. See Algorithm 2 for the description.
would like to design procedure that removes the noise. The SVD procedure

---

Published as a conference paper at ICLR 2020
Shared Modules (LSTMBBERT)
Zi
E
Task Embeddings
Figure 4: Illustration of the covariance alignment module on task embeddings.
EXPERIMENTS
We describe connections between our theoretical results and practical problems of interest. We show three claims on real world datasets. (i) The shared MTL module is best performing when its capacity smaller than the total capacities of the single-task models. (ii) Our proposed covariance alignment method improves multi-task training six sentiment analysis tasks. Our method can be naturally extended to transfer learning settings and validate this well. (iii) Our SVD-based reweighed scheme is more robust than the standard unweighted scheme on multi-label image classification tasks in the presence of label noise.
variety of settings including the GLUE benchmarks and
3.1 EXPERIMENTAL SETUP
Datasets and models. We describe the datasets and models we use in the experiments.
GLUE: GLUE is natural language understanding dataset including question answering, sentiment analysis, text similarity and textual entailment problems. We choose LERTLARGE as our model, which is a 24 layer transformer network from Devlin al. (2018). We use this dataset evaluate how Algorithm works on the state-of-the-art BERT model.
Sentiment Analysis: This dataset includes six tasks: movie review sentiment (MR), sentence sub- jectivity (SUBJ), customer reviews polarity (CR), question type (TREC), opinion polarity (MPQA), and the Stanford sentiment treebank (SST) tasks.
For each task, the goal is to categorize sentiment opinions expressed in the text. We use an embed- ding layer (with GloVe embeddings") followed by an LSTM layer proposed by Lei et al. (2018)².
ChestX-ray14, This dataset contains 1 11, 120 frontal-view X-ray images and each image has up to 14 diseases. This a 14-task multi-label image classification problem. We use the CheXNet model from Rajpurkar et al. (2017), which is a 121-layer convolutional neural network on all tasks.
For all models, we share the main module across all tasks (BERT for GLUE, LSTM for sentiment analysis, CheXNet for ChestX-ray 14) and assign separate regression or classification layer on top of the shared module for each tasks.
Comparison methods. For the experiment on multi-task training, we compare Algorithm 1 by training with our method and training without Specifically, we apply the alignment procedure on the task embedding layers. See Figure 4 for an illustration, where F, denotes the embedding of task i, R₁ denotes its alignment module and Z; = E, is the rotated embedding.
For transfer learning, we first train an STL model on the source task by tuning its model capacity (e.g. the output dimension of the LSTM layer). Then, we fine-tune the STL model on the target task for 5-10 epochs. To apply Algorithm 1, we add an alignment module for the target task during fine-tuning.
For the experiment reweighted schemes, compute the per-task weights as described in Algo- rithm 2. Then, we reweight the loss function as in equation 2. We compare with the reweighting techniques of Kendall et al. (2018). Informally, the latter uses Gaussian likelihood to model classi-
chtt:://nlp.stanforr wwwpss/fooor.uu/utttaaaaeeeeeeee wwwpd//d//wwdddeeeeeee hodddaat/wordvvcs/lloe.... 66B.r We also tested with multi-layer perceptron and CNN. The results similar (cf. Appendix B.5).

---

Published as a conference paper at ICLR 2020
COLA-
-0.6
4.3
-2.4
1
LSTM
Baseling Covariance Aligmment
5.8
0.90
Accuracy
ONLI
4.3
5.8
0.1
0.7
0.85
-2.4
-1.9
0.1
SST
2.7
0.7
1.1
0.75
COLA
NRPC
ONLI
RTE
SST
CR MPQA MR TREC
(a) MTL on GLUE over 10 task pairs
(b) Transfer learning on six sentiment analysis tasks
Figure 5: Performance improvements Algorithm by aligning task embeddings.
fication outputs. The weights, defined as inversely proportional the variances of the Gaussian, optimized during training. We also compare with the unweighted loss (cf. equation 1) as baseline.
Metric. We measure performance the GLUE benchmark using standard metric called the GLUE score, which contains accuracy and correlation scores for each task.
For the sentiment analysis tasks, measure the accuracy of predicting the sentiment opinion.
For the image classification task, we measure the area under the curve (AUC) score. We run five different random seeds to report the average results. The result of MTL experiment is averaged over the results of all the tasks, unless specified otherwise.
For the training procedures and other details the setup, refer the reader to Appendix B.
3.2 EXPERIMENTAL RESULTS
We present use cases of our methods on open-source datasets. We expected see improvements via our methods in multi-task and other settings, and indeed we saw such gains across variety of tasks.
Improving multi-task training. We apply Algorithm on five tasks (CoLA, MRPC, QNLI, RTE, SST-2) from the GLUE benchmark using state-of-the-art language model BERT
We train the output layers { and the alignment layers } using our algorithm. We compare the average performance over all five tasks and find that method outperforms BERT by 2.35% average GLUE score for the five tasks. For the particular setting training two tasks, our method outperforms BERTLARGE 7 of the 10 task pairs. See Figure 5a for the results.
Improving transfer learning. While our study has focused on multi-tase tast- learning, transfer learning is naturally related goal and find that our method is also useful in this case. We validate this by training an LSTM on sentiment analysis. Figure 5b shows the result with SST being the source task and the rest being the target task. Algorithm 1 improves accuracy four tasks by up to 2.5%.
Reweighting training for the same task covariates. We evaluate Algorithm on the ChestX-ray 14 dataset. This setting satisfies the assumption of Algorithm 2, which requires different tasks have the same input data. Across all 14 tasks, we find that our reweighting method improves the technique of Kendall al. (2018) by 0. 1%1% AUC score. Compared training with the unweighted loss, our method improves performance by 0.4% AUC score over all tasks.
3.3 ABLATION STUDIES
Model capacity. We verify our hypothesis that the capacity of the MTL model should not exceed the total capacities of the STL model. We show this on an LSTMM model with sentiment analysis tasks. Recall that the capacity of an LSTM model is its output dimension (before the last classification layer). We train an MTL model with all tasks and vary the shared module's capacity to find the 500. Similarly we train an STL model for each task and find the optimum.
optimum from
In Figure I, we find that the performance of MTL peaks when the shared module has capacity 100. This is much smaller than the total capacities of all the STL models. The result confirms that
8

---

Published as a conference paper at ICLR 2020
Table 1: Comparing the model capacity between MTL and STL.
Figure 6:
Covariance similarity score vs.
performance improvements from alignment.
Performance Improvements
Task
STL
MTL
0.04
LSTM
Covariance Alignment
Cap.
Acc.
Cap.
Acc.
SST
200
82.3
90.8
0.02
Baseline
MR
76.4
CR
73.2
78.7
100
SUBJ
200
91.5
89.5
THEC SUBJ
Cit Mil
MPQA
500
86.7
87.0
TREC
100
85.7
78.7
0.1
0.2
0.3
0.4
0.5
Overall
1205
82.6
100
85.1
Covariance similarity score
constraining the shared module's capacity is crucial to achieve the ideal performance. Extended results on CNN/MLP to support our hypothesis are shown in Appendix B.5.
Task covariance. We apply our metric of task covariance similarity score from Section 2.3 to provide in-depth study of the covariance alignment method. The hypothesis that: (a) aligning the covariances helps, which we have shown in Figure 5a; (b) the similarity score between two tasks increases after applying the alignment. We verify the hypothesis on the sentiment analysis tasks. We use the single-task model's embedding before the LSTM layer to compute the covariance.
First, we measure the similarity score using equation between all six single-task models. Then, for each task pair, we train an MTLL model using Algorithm 1. We measure the similarity score on the trained MTL model. Our results confirm the hypothesis (Figure 6): (a) we observe increased accuracy on 13 of 15 task pairs by up to 4.1%; (b) the similarity score increases for all 15 task pairs.
Optimization scheme. We verify the robustness of Algorithm After selecting two tasks from the ChestX-ray 14 dataset, we test our method by assigning random labels to 20% of the data on one task. The labels for the other task remain unchanged.
On 10 randomly selected pairs, our method improves over the unweighted scheme by average 1.0% AUC score and the techniques of Kendall et al. (2018) by an average 0.4% AUC score. We include more details of this experiment in Appendix B.5.
4 RELATED WORK
There has been large body of recent work on using the multi-task learning approach to train deep neural networks. Liu et al. (2019a); McCann et al. (2018) and subsequent follow-up work show titt-off-thearr results the GLUE benchmark, which inspired our study of an abstraction of the MTL model. Recent work of Zamir et al. (2018); Standley et al. (2019) answer which visual tasks to train together via a heuristic which involves intensive computation. We discuss several lines of studies related to this work. For complete references, we refer the interested readers to the survey of Ruder (2017), 7): Zhang and Yang (2017) and the surveys on domain adaptation and transfer learning by Pan and Yang (2009); Kouw (2018) for references.
Theoretical studies of multi-task learning. Of particular relevance this work those that study the theory of multi-task learning. The earlier works of Baxter (2000); Ben-David and Schuller (2003) are among the first formally study the importance of task relatedness for learning multiple tasks. See also the follow-up work of Maurer (2006) which studies generalization bounds of MTT..
closely related line of work to structural learning subspace selection, i.e. how to select common subspace for multiple tasks. Examples from this line work include Obozinski et al. (2010); Wang et al. (2015); Fernando et al. (2013); Elhamifar al. (2015). Evgeniou and Pontil (2004); Micchelli and Pontil (2005) study formulation that extends support vector machine to the multi- task setting. See also Argyriou et al. (2008); Pentina et al. (2015); Pentina and Ben-David (2015); Pentina and Lampert (2017) that provide more refined optimization methods and further study. The work of Ben-David et al. (2010) provides theories to measure the differences between source and target tasks for transfer learning in different model setup. Khodak al. (2019); Kong et al. (2020);

---

Published as a conference paper at ICLR 2020
Du et al. (2020) consider the related meta learning setting, which is in spirit online setting of multi-task learning.
Our result on restricting the model capacities for multi-task learning is in contrast with recent the- oretical studies on paraammmettrze models (e.g. Li et al. (2018); Zhang et al. (2019a); Bartlett al. (2020)), where the model capacities are usually much larger than the regime we consider here. would be interesting to better understand multi-task learning in the context of over wver-parametrized models with respect to other phenomenon such double descent that has been observed in other contexts (Belkin et al. (2019)).
Finally, Zhang et al. (2019b); Shui al. (2019) consider multi-task learning from the perspective of adversarial robustness. Mahmud and Ray (2008) consider using Kolmogorov complexity measure the effectiveness of transfer learning for decision tree methods.
Hard parameter sharing vs soft parameter sharing. The architecture that we study in this work also known as the hard parameter sharing architecture. There is another kind of architecture called soft parameter sharing. The idea is that each task has its own parameters and modules. The relationships between these parameters are regularized in order to encourage the parameters to be similar. Other architectures that have been studied before include the work of Misra et al. (2016), where the authors explore trainable architectures for convolutional neural networks.
Domain adaptation. Another closely related line of work is on domain adaptation. The acute reader may notice the similarity between our study in Section 2.3 and domain adaptation. The cru- cial difference here is that we are minimizing the multi-task learning objective, whereas in domain adaptation the objective is typically to minimize the objective on the target task. See Ben-David et al. (2010); Zhang et al. (2019b) and the references therein for other related work.
Optimization techniques. Guo et al. (2019) use ideas from the multi- bandit literature to de- velop method for weighting each task. Compared their method, our SVD-based method is con- ceptually simpler and requires much less computation. Kendall et al. (2018) derive a weighted loss schme by maximizing a Gaussian likelihood function. Roughly speaking, each task is reweighted by 1/a" where a is the standard deviation of the Gaussian and a penalty of log a is added to the loss. The values of {o are also optimized during training. The exact details can be found in the paper. The very recent work of Li and Vasconcelos (2019) show empirical results using similar idea of covariance normalization on imaging tasks for cross-domain transfer.
CONCLUSIONS AND FUTURE WORK
We studied the theory of multi-task learning in linear and ReI.U-activated settings. We verified our theory and its practical implications through extensive synthetic and real world experiments.
Our work opens up many interesting future questions. First, could we extend the guarantees for choosing optimization schemes to non-linear settings? Second, a limitation of our SVD-based op- timization scheduler is that only applies to settings with the same data. Could we extend the method for heterogeneous task data? More broadly, we hope our work inspires further studies to better understand multi-task learning in neural networks and to guide its practice.
Acknowledgements. Thanks to Sharon Y. Li and Avner May for stimulating discussions dur- ing early stages of this work. We are grateful the Stanford StatsML group and the anony- mous referees for providing helpful comments that improve the quality of this work. We grate- fully acknowledge the support of DARPA under Nos. F587501720095 (D3M), 1A86501827865 (SDH), and FA86501827882 (ASED); NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCFI566078 (Volume to Velocity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP. Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC. Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Er- icsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss Re, and members of the Stanford DAWN project: Teradata, Facebook, Google, Ant Fi- nancial, NEC, VMWare, and Infosys. H. Zhang is supported in part by Gregory Valiant's ONR YIP award (#I 704417). The experiments are partly run on Stanford's SOAL cluster. 5 The U.S. Govern- ment is authorized reproduce and distribute reprints for Governmental purposes notwithstanding
s htap::/Shharrd..om/ooolllllsae
10

---

Published as a conference paper at ICLR 2020
any copyright notation thereon. Any opinions, findings, and conclusions recommendations pressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA. NIH, ONR, or the U.S. Government.
REFERENCES
Héctor Martínez Alonso and Barbara Plank. When is multitask learning effective? semantic quence prediction under varying data conditions. arXiv preprint arXiv: 2016.
Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov):1817-1853 2005.
Andreas Argyriou, Andreas Maurer, and Massimiliano Pontil. An algorithm for transfer learning in heterogeneous environment. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 71-85. Springer, 2008
Maria-Florina Balcan, Yingyu Liang, David P Woodruff, and Hongyang Zhang. Matrix completion and related problems via strong duality. 9th Innovations in Theoretical Computer Science Conference (ITCS 2018), 2018.
Peter L. Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 2020.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 16632))15549-15888
2019.
Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. Learn- ing Theory and Kernel Machines, pages 567-580. Springer, 2003.
Shai Ben- Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning. 151111): 2010.
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. arXiv preprint arXiv: 100..0:770000000 08303, 2017.
John Blitzer, Ryan McDonald, and Fernando Pereira. Domain adaptation with structural correspon- dence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120-128. Association for Computational Linguistics, 2006.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18 10.04005 2018.
Simon S Du, Jason D Lee, Yuandong Tian, Barnabas Poczos, and Aarti Singh. Gradient de- scent learns one-hirden-llyer cnn: Don't be afraid of spurious local minima. arXiv preprint arXiv:1712.00779 2017.
Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.094322 2020.
Ehsan Elhamifar, Guillermo Sapiro, and S Shankar Sastry. Dissimilarity- based sparse subset selee- tion. IEEE transactions pattern analysis and machine intelligence, 38(1 (8(11))1122-2197 2015.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 109-117. ACM, 2004.
Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised visual do- main adaptation using subspace alignment. In Proceedings of the IEEE international conference computer vision, pages 2960-2967, 2013.

---

Published as a conference paper at ICLR 2020
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. Autosem: Automatic task selection and mixing in multi-task learning. arXiv preprint 1rXiv:1904.01133, 2019.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2):33-8,, 2005.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 167-177. ACM, 2004.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 7482-7491, 2018.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Provable guarantees for gradient- based meta-learning arXiv preprint
2019.
Yoon Kim. Convolutional neural networks for sentence classification.
arXiv preprint
arXiv: 1408. 5882, 2014.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 6129-6138, 2017.
Weihao Kong, Raghav Somani, Zhao Song, Sham Kakade, and Sewoong Oh. Meta-learning for mixed linear regression. arXiv preprint arXiv:2002.08930 2020.
Wouter M Kouw. An introduction to domain adaptation and transfer learning. arXiv preprint arXiv: 1806, 2018
Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly par- allelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470-4481, 2018.
Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics- 1, pages 1-7. Association for Computational Linguisties, 2002.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in wer-parameterrzze matrix sensing and neural networks with quadratic activations. In Conference On Learning The- ory, pages 2-47, 2018
Yunsheng Li and Nuno Vasconcelos. Efficient multi-domain learning by covariance normaliza- tion. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 5424 5433, 2019.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv: 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019b.
MM Mahmud and Sylvian Ray. Transfer learning using kolmogorov complexity: Basic theory and empirical evaluations. In Advances in neural information processing systems, pages 985-992, 2008.
Pasin Manurangsi and Daniel Reichman. The computational complexity of training relu (s). arXiv preprint 1xX.:18100002200 2018
Andreas Maurer. Bounds for linear multi-task learning. Journal of Machine Learning Research, 17-1): 2006.
12

---

Published as a conference paper at ICLR 2020
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language
decathlon: Multitask learning as question answering. arXiv preprint arXivv:806. 08730, 2018.
Charles A Micchelli and Massimiliano Pontil. Kernels for multi learning. In Advances in neural information processing systems, pages 921 1218 2005.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference Natural Language Processing of the AFNLP: Volume 2- 2-Volume 2, pages 1003-1011. Association for Computational Linguistics, 2009.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994-4003 2016.
Guillaume Obozinski, Ben Taskar, and Michael Jordan. Joint covariate selection and joint subspace selection for multiple classification problems. Statistics and Computing, 20(2):231-25 2010.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):335-1135, 2009.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics, page 271. Association for Computational Linguistics, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for com- putational linguistics, pages 115-124. Association for Computational Linguistics, 2005.
Anastasia Pentina and Shai Ben-David. Multi-task and lifelong learning of kernels. In International Conference on Algorithmic Learning Theory, pages 194-208 Springer, 2015.
Anastasia Pentina and Christoph H Lampert. Multi-task learning with labeled and unlabeled tasks. In Proceedings of the 34th International Conference on Machine Learning- Voluue 70, pages 2807-2816. JMLR. org, 2017.
Anastasia Pentina, Viktoriia Sharmanska, and Christoph H Lampert. Curriculum learning of multi- ple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5492-5500. 2015.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, al. Chexnet: Radiologist-level pneumo- nia detection chest x-rays with deep learning, arXiv preprint arXiv:1711.0522. 2017.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1505007706.05098, 2017.
Changjian Shui, Mahdieh Abbasi, Louis-Emile Robitaille, Boyu Wang, and Christian Gagné. A principled approach for learning task similarity in multitask learning. arXiv preprint arXiv:1903.09 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pages 1631-1642, 2013.
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? arXiv preprint arXiv: 1905.07553, 2019.
Joel Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1 1-230,, 2015.
13

---

Published as a conference paper at ICLR 2020
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, 2018a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0746/) 2018b.
Xiaosong Wang. Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
Yu Wang, David Wipf, Qing Ling, Wei Chen, and Ian James Wassell. subspace segmentation. 2015.
Multi-task learning for
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3): 2005.
Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classifica- tion with dirichlet process priors. Journal of Machine Learning Research, 8(Jan):35-63 63, 2007.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 3712-3722, 2018.
Hongyang Zhang, Vatsal Sharan, Moses Charikar, and Yingyu Liang, Recovery guarantees for quadratic tensors with limited observations. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2019a.
Yu Zhang and Qiang Yang. A survey multi-task learning. arXiv preprint arXh::1707.08111 2017.
Yu Zhang and Din-Yan Yeung, A regularization approach to learning task relationships in multitask learning. ACM Transactions on Knowledge Discovery from Data (TKDD), 8(3):12, 2014.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I Jordan. Bridging theory and algorithm for domain adaptation. arXiv preprint 4904v: 2019b.
14

---

Published as a conference paper at ICLR 2020
A MISSING DETAILS OF SECTION 2
We fill the missing details left from Section 2. In Section A.1, we provide rigorous arguments regarding the capacity of the shared module. In Section A.2, we fill in the details left from Section 2.3, including the proof of Theorem 2 and its extension to the ReLU model. In Section A.3, we provide the proof of Proposition on the task reweighting schemes. We first describe the notations.
Notations. We define the notations to be used later on. We denote f(x) S g(x) if there exists an absolute constant such that fx) I Cg(x). The big-O notation f(z) ((g(z)) means that fz) S g(x).
Suppose A Rmx" then (A) denotes its largest singular value and Amii (A) denotes its minfm, m, n/-th largest singular value. Alternatively, we have Amin (A) =in=:||=||=1 Az]l Let <(A) = Amax(() (A)/ Amin denote the condition number of A. Let Id denotes the identity matrix. Let Uf denote the Ponro-e pseudo-inverse of the matrix U. Let || denote the Euclidean norm for vectors and spectral norm for matrices. Let || || denote the Frobenius norm of matrix. Let (A =) (r B) denote the inner product of two matrices.
The sine function define as sin(u, v) = v1 - cos(u, u, v)², where we assume that sin(u, v) 2 which is without loss of generality for our study.
A.1 MISSING DETAILS OF SECTION 2.2
We describe the full detail to show that our model setup captures the phenomenon that the shared module should be smaller than the of capacities of the single-task models. We state the fol- lowing proposition which shows that the quality of the subspace in equation determines the performance of multi-task learning. This supplements the result of Proposition 1.
Proposition 4. In the optimum of f(-) (equation 1), each A; selects the vector within the column span of gø (X.) to minimize L(v, M.). As a corollary, in the linear setting, the optimal B can be achieved at rotation matrix B* C Raxr by maximizing
Σ (B X X. vo. X,).
Furthermore, any B* which contains {0 in its column subspace optimal. In particular, for such a B", there exists {AT} so that 3 A 0, for all 1 < sist k.
Proof. Recall the MTL objective in the linear setting from equation 3 as follows:
min f(A), A2.... As; B) = Σ h
(X,BA, v.)² -
Note that the linear layer A; can pick any combination within the subspace B. Therefore, we could assume without loss of generality that B is rotation matrix. i.e. B B = Id. After fixing B, since objective f(-) is linear in A; for all i, by the local optimality condition, we obtain that
A = (BT X XTB)BBBBBB x, yi
Replacing the solution of A; to f(-), we obtain an objective over B.
&
M(B)
Ix,B(B X,B) B vil..-
Next, note that
|X X X,B)*B B x voll. Trly X,B(BTL X X.BBBBBB
=(B(B) X X,B) x. yor. x.),
where we used the fact that A A'AA- = A for A B X X,B in the first equation. Hence we have shown equation? 7.
15

---

Published as a conference paper at ICLR 2020
For the final claim, long B* contains {0.) in its column subspace, then there exists A such that B A 0. The B and {A1)/=1 are optimal solutions because each 03 is an optimal solution for the single-task problem.
The above result on linear regression suggests the intuition that optimizing an MTL model reduces to optimizing over the span of B. The intuition can be easily extended to linear classification tasks as well as mixtures of regression and classification tasks.
Extension to the ReLU setting. If the shared module's capacity is larger than the total capacities of the STL models, then we can put all the STL model parameters into the shared module. As in the linear setting, the final output layer A, can pick out the optimal parameter for the i-th task. This remains an optimal solution to the MTL problem in the ReLU setting. Furthermore, there is no transfer between any two tasks through the shared module.
A.2 MISSING DETAILS OF SECTION 2.3
A.2.1 THE EFFECT OF COSINE SIMILARITY
We consider the effect of varying the cosine similarity between single task models in multi-task learning. We first describe the following proposition to solve the multi-task learning objective when the covariances of the task data are the same. The idea is similar to the work of Ando and Zhang (2005) and we adapt here for our study.
Proposition 5. Consider the reweighted loss of equation with the encoding function being linear, where the weights are {ax)/ Suppose the task features of every task have the same covariance: X. X; for all 1 < k. Let Σ = VVV be the singular vector decomposition (SVD) of Σ. Then the optimum of [-) in equation 3 is achieved at:
B B*=VD-
where C is the best rank-r approximation subspace of Ei O.U y:u, U; and X; = U.DVT is the SVD of X,, for each sis k.
As corollary, denote by A1. .2..... dE as the singular values of D D -VVVV a you. X; in decreasing order. Then the difference between an MTL model with hidden dimension and the all the single task models is bounded by Σt Eimr41 X2.
Proof. Note that 8" is obtained by maximizing
Σ CBB XTB ax, vun, X,,)
Let C = DVT B. Clearly, there is one to one mapping between and C. And we have V VD-CC, Hence the above is equivalent to maximizing over C' C Rddr with
h Σ
D-
Σ
axx you! X; VD
=(C(CTT))-CT,,
Σ U. yo..U. U.).
Note that C(C C)- is projection matrix onto a subspace of dimension Hence the maximum (denote by C*) is attained at the best rank-r approximation subspace of El_ U,T you U.-
To illustrate the above proposition, consider simple setting where X; is identity for every S i < k. and y4 e; i.c. the i-th basis vector. Note that the optimal solution for the i-th task is (X.x,) - T y = y:- Hence the optimal solutions are orthogonal to each other for all the tasks, with X; = 1 for all 1 < < k. And the minimum STL emror is zero for all tasks.
16

---

Published as a conference paper at ICLR 2020
Consider the MTL model with hidden dimension By Proposition 5, the minimum MTL error is achieved by the best rank-r approximation subspace to Σ X you, X; the optimum as Br The MTL error is:
Σ y.y
Denote
yov. B; k
Different data covariance. We provide upper bounds on the quality MTL solutions for different
data covariance, which depend on the relatedness of all the tasks. The following procedure gives the precise statement. Consider regression tasks with data (i)) Let 0; (X, Xx) X, y denote the optimal solution of each regression task. Let W S Rdxk denote the matrix where the i-th column is equal to 0. Consider the following procedure for orthogonalizing W for 1 S i S k.
a) Let W E IRd denote the vector which maximizes Σ
u1)² over B E Rd. X
X.
b) Denote by Xy = Σラ= WX, U;))
x, R,W;
c) For each 1 i k. project X, X,W off from every column of X,. Go to Step a).
Proposition 6. Suppose that S d. Let B* denote the optimal MTL solution of capacity r in the shared module. Denote by OPT = E [XX X, 9ill3). Then h(B²) z
OPT Ei-r41 As-
Proof. It suffices to show that OPT is equal to Σ A. The result then follows since h(B*) is less than the error given by Wi,..., Wg, which equal to OPT Eiert Av.
A.2.2 PROOF OF THEOREM 2
We fill in the proof of Theorem 2. First, we restate the result rigorously as follows.
1,2, let (X ys) E (Iem xd Rm))) denote two linear regression tasks with parameters 0g E Rd. Suppose that each row of X is drawn independently from distribution with covariance Σi C BRdxd and bounded 12-momm VL. Assume that 0, Σ, = w.l.o.g.
Theorem 2. For
Let E (in())) f2), 1/3) denote the desired error margin. Denote by (B*, Ai, A2) the optimal MTL solution. With probability 1 - 55 over the randomness of (X1, when
L 1|
log d 5
x(2)))>(X2)
x² (2(2))) of log !)
mi Z max
1921
X (E,)
e
we have that I A
0211/11021 < 6c
3- |2211/1
We make several remarks to provide more insight on Theorem 2.
Theorem guarantees positive transfers in MTL, when the source and target models are close and the number of source samples large. While the intuition folklore in MTI., we provide formal justification in the linear and ReL.U models to quantify the phenomenon.
The error bound decreases with c, hence the smaller e is the better. On the other hand, the required number of data points my increases. Hence there is a trade-off between accuracy and the amount of data.
is assumed be at most 1/3. This assumption arises when we deal with the label noise task 2. there is no noise for task 2, then this assumption is not needed. If there is noise for task 2, this assumption is satisfied when sin(O,, 02) less than 1/(3x(X2).. In synthetic experiments, we observe that the dependence on sf (22 and sin(Or, 02) both arise in the performance of task 2, cf. Figure 3 and Figure respectively.
The proof of Theorem 2 consists of two steps.
17

---

Published as a conference paper at ICLR 2020
a) We show that the angle between B* and 0 will be small. Once this is established, get a bound on the angle between B and 0g via the triangle inequality.
b) We bound the distance between 3* Ag and 02. The distance consists two parts. One part comes from B', i.e. the angle between B and 02. The second part comes from A2, i.e. the estimation error of the norm of 02, which involves the signal to noise ratio of task two.
We first show the following geometric fact, which will be used later in the proof.
Fact 7. Let o, b E Ied denote two unit vectors. Suppose that X E IR has full column rank with condition number denoted by K x(X). Then we have
sin(Xa, Xb)
Isin(a, b]|-
Proof. Let X UDVT be the SVD of X. Since has full column rank by assumption, we have X X XXT = Id. Clearly, have sin Xb) din a, DV b). Denote by a' V and 1/ = b. We also have that a' and d' are both unit vectors, and sin(a', b') sin(a, b). Let AI...., Aa denote the singular values of X. Then,
Ei= APa.b!
sin' (Dn', Db') =1
EE X2o,²) (Σi=) \96{?)
Eighosa (a o,,b)² EE Xta(²)
(Ei=1 xz0.²)
A
Σ (a/bi a,bb)²
X
1st
6
d(b)))
sin (a',b')
This concludes the proof.
We first show the following Lemma, which bounds the angle between B" and 02
Lemma 8. In the setting of Theorem 2, with probability - over the randomness of task one, we have that
[sin( 020] S sin(O1, 02) c/x(22))
X,B" Proof. We note that h(B*) 2 /1r112 by the optimality of B* Furthermore, (Tx U2) S 1112|12. Hence we obtain that
X, B,B-I 91)² 33) 2 1ll1 12 11
For the left hand side,
B'
X, B²
01)2
X101 =1)²
|x,
B²
B₁B²
X₁B* X₂0i) |
X,B*
.201)² +
|X, 51) +
51)
|x,
Note that the second term is chi-squared random variable with expectation of Hence it is bounded by of 2/o/ } with probability least 8. Similarly, the third term bounded by 2||X;0 log with probability
Therefore, we obtain the following:
|X,0 12 cos cos2 (X, X20,) Z || 2 || (of + 20, 2 (x,111111)) log
18

---

Published as a conference paper at ICLR 2020
Note that
1 11 || 1((
2
| 11 211x1011111 log
Therefore,
|X cos (XIB", 2 ||x101112 1//2/1 - (o] + 30.11 log
112211
40
sin B,, S
+
|X
||x0011
112el12
40
401 log
sin 01) S (X1)
(by Lemma 7)
0|
|X
By matrix Bernstein inequality (see e.g. Tropp et al. (2015)), when mi 2 10 log / (E1), we have that:
X X X1 Ei (m1)
m1
Hence we obtain that (X,) S (x,), and 1| 2 my 0 0/0/1//2 2 m1/2 (where we assumed that 0f 2101 1). Therefore,
lual mf/4
40 log mm /2
sBn²(B",0,) S 3x(EI)
which most 6² (X2) by our setting of mi. Therefore, the conclusion follows by triangle inequality (noting that both e and sin(O), 02) are less than 1/2).
Based on the above Lemma,
ready to prove Theorem 2.
Proof of Theorem 2. Note that in the MTL model, after obtaining B", we then solve the linear layer for each task. For task 2, this gives weight value A == (X20, 32) [|= Thus the regression coefficients for task 2 is B" A5. For the rest of the proof, focus bounding the distance between B* A and 02. By triangle inequality,
((xBB",e))
(XoB* X202)
|B" A - 0211 <
||021|
+ ||B*|0021 02l1
(8)
1+2
1x2B+112
Note that the second term of equation is equal to
(X, X2(02 B+)
<x(2)) 100 1B02|11*1| 11.
1x
The first term of equation 8 is bounded by
e ||
||2||||021 [|02|[+)]|
(9)
|x202
(x2
Lastly, we have that
|0 ||02| =| cos(B',000) 02)) 2110211 sin S 02)
By Lemma 8, we have
[sin(B", (an,(BB') (2,)) S sin(01, 02) + c/k(X2)
19

---

Published as a conference paper at ICLR 2020
Therefore, conclude that equation 9 is at most
| ||02]
| [x202
(x2)
182 (in 1. 02) V2cd (X22) 162
||22|| 021
|x202
3cAmiu
X221
|| ||0211
3c
||2202
||
Thus equation 8 is at most the following.
|22
+ V2(x(X2) + sin((",
1 3c
||x282l
|52| |l
+ 6c
3c
||22l
Hence we obtain the desired estimation error B.A..
A.2.3 EXTENSION TO THE RELU MODEL
In this part, we extend Theorem 2 the ReLU model. Note that the problem is reduced to the following objective.
g (B) ReL.U(X))) |ReLLUXX 01)²
ReI.U(X2
[eeLUXX,B]] U2)
(10)
BER
We make crucial assumption that task I's input X, follows the Gaussian distribution. Note that making distributional assumptions is necessary because for worst-case inputs, even optimizing single ReLU funetion under the squared loss is NP-hard (Manurangsi and Reichman (2018)). We state our result formally as follows.
Theorem 9. Let (X1,31) E (R xd R".) and (X2, 3/2 E (Rmaxd, (0maxd denote two tasks. Suppose that each row of X1 drawn from the standard Gaussian distribution. And = ag ReI.U(X,0;) + & are generated via the ReLU model with 0,,0, E Rd Let E [a, ReLU((X,0)) = for every 1 S S my without loss of generality, and let of denote the variance of every entry =1
Suppose that e 2 sin(01,02) 02)00,2))((11) xx)).. Denote by (B*, Ai; At) the optimal MTL solution of equa- tion 10. With probability 1 5 over the randomness of (X1, when
(dlogd 1 + log d), llvel1?) (=22 (22
c² )
mi Z max
have that the estimation error is at most:
sin(B* 01) sin(O1, 02) ((c/k((2)), [Ag -2 O(c)
1
||
02
(1 Oc))) 02 (alUU(|222))
Proof. The proof follows similar structure to that of Theorem 2. Without loss of generality, we can assume that 01, 0g are both unit vectors. We first bound the angle between B* and 01.
By the optimality of B", we have that:
ReI.U(X,B*)) |eLLUXX 11) ReLUU(X,))
(| 01)2 ||
From this we obtain:
Be
df
ReLU((XBB'))
|Rel.UXX,
2a [R)
+ 40 - (eL
8 (11)
20

---

Published as a conference paper at ICLR 2020
Note that each entry of ()LU((1011 is truncated Gaussian random variable. By the Hoeffding bound, with probability 8 we have
m1
M1
log 8'
As for (R, Re we will use an epsilon-net argument over B to show the concentration. For fixed B*, we note that this is sum of independent random variables that are all bounded within Olog = with probability 1 Denote by é the angle between 8 and 0,, standard geometric fact states that (see e.g. Lemma I of Du et al. (2017)) for random Gaussian vector x E IR"
E ReLUUx B²) ReLU(x To.)] cos +
cos (tan の)
g())
2x
2
Therefore, by applying Bernstein's inequality and union bound, with probability 1f have:
1
log m.
(ReLU(XXB") ReLU(X(0)) m2g(0))2 s 2 mig(o) log 1) 3
log
By standard arguments, there exists a set (d(d) unit vectors S such that for any other unit vector there exists å S such that ||u - all S min(1/d², //ddd,///"((x))))) (X2)). By setting 1j d-O(d) and take union bound over all unit vectors in S, we have that there exists i E S satisfying B ill s min(1/dP, e² e2/ / (X2)) and the following:
(ReLU(XX ReLU(X101) mmg((o)) E /midlogd + dlogg
I 2m 1am (X2)
(by our setting of m))
where ø' is the angle between ú and 0j. Note that (ReLU(X,0) ReLU(X, ReLU(X(0) 1) S 1| B*)l [|ReLU(X(0))]
s (x2) 0(m1)
Together we have shown that (ReI.U(X, B*), ReILU(X(1)) mmg(0') < 2 2 0(m).
Combined with equation 11, by our setting of my, it is not hard to show that
g(0') 2 1 -ce nk² (X2)).
Note that
1 1-g(Ø')
cos Φ cos Ø (tan Ø d²)
cos Φ' 2sin 1
< 2 (X2), 2
which implies that sin? 0' S e 2/12 (X2) (since cos o' 2 0.9). Finally note that || B' s c² //x)(X2). hence
| B3 2(1 - cos ú1 B*)) 2 sin B*). Overall, conclude that sin , 01) < O(c/x(X2)) Hence
sin 1 02) s sin(O, 02) + O(c/n(X2)).
For the estimation of 02, we have (ReLU(X, U2)
(ReLU(XpB*).er E2)
02
|ReLU(X2B*)IP
BR) ReLU(X22') - ReLU(X202))
t02
ReeLU x
The first part is at most
||al||
|
(X2LU
(Rel.U(X2003)) - ReI.U(XX2 - ReI.U(X2
1
||2
I- [ReLU(X202]
Similarly, show that the second part is most (c)). Therefore, the proof is complete.
21

---

Published as a conference paper at ICLR 2020
A.3 PROOF OF PROPOSITION 3
In this part, we present the proof of Proposition 3. In fact, we present a more refined result, by showing that all local minima global minima for the reweighted loss in the linear case.
I(A1, A2. As; Σ as X, -
(12)
The key is to reduce the MTL objective f(.) to low rank matrix approximation, and apply recent results by Balcan et al. (2018) which show that there is no spurious local minima for the latter problem.
Lemma 10. Assume that XJ = az with os > 0 for all Sii S Then all the local minima of f(Ai,..., As; B) are global minima of equation 3.
Proof. We first transform the problem from the space of B to the space of C. Note that this is without loss of generality, since there is to one mapping between B and C with C DV B. In this case, the corresponding objective becomes the following.
g(A Ag; B)
04 IIUCCA, - Vill?
i=1
&
Σ
Σ aq [[((a)A)) valu, 91ll 11ll++++
||0]] vxl13)
The latter expression is constant. Hence it does not affect the optimization solution. For the former, denote by A E r as stacking the /a;A,'s 's together column-wise. Similarly, denote by Z E Rddk as stacking Va,U, y: together column- wiise. Then minimizing g(-) reduces solving low rank matrix approximation: ||C'A - 112
By Lemma 3.1 of Balcan et al. (2018), the only local minima of ||CA- - 2 are the ones where C'A is equal to the best rank-r approximation of Z. Hence the proof is complete.
D
Now we are ready to prove Proposition 3.
Proof of Proposition 3. By Proposition 5, the optimal solution of B for equation 12 is VD times the best rank-r approximation to a,U y:u, U, where we denote the SVD of X as UDV denote by Z
Denote by Q, the best rank-r approximation to U ZZ U, where (Vrrh, va23y,..... yonyk) as stacking the & vectors to a d by k matrix. Hence the result of Proposition 5 shows that the optimal solution 8* is VD-1Q,, which is equal to (X xx
By Proposition the optimality of 8* is the same up to transformations on the column space. Hence the proof is complete.
To show that all local minima are also equal to (X- 'XQ", we can simply apply Lemma 10 and Proposition 3.
Remark. This result only applies to the linear model and does not work on ReLU models. The ques- tion of characterizing the optimization landscape in non-linear ReLU models is not well-understor based on the current theoretical understanding of neural networks. We leave this for future work.
22

---

Published as a conference paper at ICLR 2020
B SUPPLEMENTARY EXPERIMENTAL RESULTS
We fill in the details left from our experimental section. In Appendix B.1, review the datasets used in our experiments. In Appendix B.2, we describe the models we use on each dataset. In Appendix B.3, describe the training procedures for all experiments. In Appendix B.4 and Ap- pendix B.5, we show extended synthetic and real world experiments to support our claims.
B.1 DATASETS
We describe the synthetic settings and the datasets Sentiment Analysis, General Language Under- standing Evaluation (GLUE) benchmark, and ChestX-ray14 used in the experiments.
Synthetic settings. For the synthetic experiments, we draw 10,000 random data samples with di- mension d = 100 from the standard Gaussian (0, 1) and calculate the corresponding labels based on the model descrihed in experiment. We split the data samples into training and validation sets with 9,000 and 1,000 samples in each. For classification tasks, we generate the labels by applying a sigmoid function and then thresholding the value to binary labels at 0.5. For ReLU regression tasks, we apply the ReLU activation function on the real-valued labels. The number of data samples used in the experiments varies depending on the specification. Specifically, for the task covariance experiment of Figure 3, we task I's data with my 9, 000 training data and vary task 2's data under three settings: (i) same rotation Q = Q2 but different singular values Di K D2; (ii) same
singular values D1 = D2 but random rotations Q + Q2.
Sentiment analysis. For the sentiment analysis task, the goal is to understand the sentiment opinions expressed in the text based on the context provided. This is a popular text classification task which is usually formulated as multi-label classification task over different ratings such as positive (+1), negative (-1), or neutral (0). We use six sentiment analysis benchmarks in our experiments:
Movie review sentiment (MR): In the MR dataset (Pang and Lee (2005)), each movie re- view consists of single sentence. The goal is detect positive vs. negative reviews.
Sentence subjectivity (SUBJ): The SUBJ dataset is proposed in Pang and Lee (2004) and the goal is classify whether a given sentence is subjective or objective.
Customer reviews polarity (CR): The CR dataset (Hu and Liu (2004)) provides customer reviews of various products. The goal categorize positive and negative reviews.
Question type (TREC): The TREC dataset is collected by Li and Roth (2002). The aim is to classify a question into question types.
Opinion polarity (MPQA): The MPQA dataset detects whether opinion is polarized or not (Wiebe et al. (2005)).
Stanford sentiment treebank (SST): The SST dataset, created by Socher et al. (2013), is an extension the MR dataset.
The General Language Understanding Evaluation (GLUE) benchmark. GLUE is a collection of NLP tasks including question answering, sentiment analysis, text similarity and textual entail- ment problems, The GLUE benchmark is state-of-the-art MTL benchmark for both academia and industry. We select five representative tasks including CoLA, MRPC, QNLI, RTE, and SST-2 to validate our proposed method. We emphasize that the goal of this work is not to come up with state-of-the-as -hrtt result but rather to provide insights into the working of multi-task learning. It is conceivable that our results can be extended to the entire dataset as well. This is left for future work. More details about the GLUE benchmark can be found in the original paper (Wang et al. (2018a)).
ChestX-ray14. The ChestX-rayl- 14 dataset (Wang et al. (2017)) is the largest publicly available chest X-ray dataset. It contains 112, 120 frontal-view X-ray images of 30,805 unique patients. Each image contains up to 14 different thoracic pathology labels using automatic extraction methods on radiology reports. This can be formulated as a 14-task multi-label image classification problem. The ChestX-ray Max-aayy444 14 dataset is representative dataset in the medical imaging domain well in computer vision. We use this dataset to examine our proposed task reweighting scheme since satisfies the assumption that all tasks have the same input data but different labels.
23

---

Published as a conference paper at ICLR 2020
B.2 MODELS
Synthetic settings. For the synthetic experiments, we use the linear regression model, the logistic regression model and one-layer neural network with the ReLU activation function.
Sentiment analysis. For the sentiment analysis experiments, we consider three different models including multi-layer perceptron (MLP), LSTM, CNN:
For the MLP model, we average the word embeddings of a sentence and feed the result into a two layer perceptron, followed by classification layer.
For the LSTM model, we use the standard one-layer single direction LSTM as proposed by Lei al. (2018), followed by classification layer.
For the CNN model, we use the model proposed by Kim (2014) which uses one convolu- tional layer with multiple filters, followed by ReLU layer, max-pooling layer, and classi- fication layer. We follow the protocol of Kim (2014) and set the filter size as (3, 5.
We use the pre-trained GLoVe embeddings trained on Wikipedia 2014 4 and Gigaword corpora 6 We fine-tune the entire model in our experiments. In the multi-task learning setting, the shared modules include the embedding layer and the feature extraction layer (i.e. the MLP, LSTM, or CNN model). Each task has its separate output module.
GLUE. For the experiments on the GLUE benchmark, we use a state-of-the-an language model called BERT (Devlin et al. (2018)). For each task, we add a clasification/regressinn layer on top as our model. For all the experiments, we use the LARGE uncased model, which a 24 layer network as described in Devlin et al. (2018). For the multi-task learning setting, we follow the work of Liu et al. (2019a) and use BERTLARGE as the shared module.
ChestX-ray14. For the experiments the ChestX-ray14 14 dataset, use the DenseNet model proposed by Rajpurkar et al. (2017) as the shared module, which is a 121 layer network. For each task, we use separate classification output layer. We use the pre-trained model? in our experiments.
B.3 TRAINING PROCEDURES
In this subsection, we describe the training procedures for our experiments.
Mini-batch SGD. We describe the details of task data sampling in our SGD implementation.
For tasks with different features such as GLUE, we first divide each task data into small batches. Then, we mix all the batches from all tasks and shuffle randomly. During every epoch, a SGD step is applied on every batch over the corresponding task. If the current batch is for task then the SGD is applied on A,, and possibly R3 or B depending on the setup. The other parameters for other tasks are fixed.
For tasks with the same features such as ChestX-ray 14, the SGD is applied on all the tasks jointly to update all the A, and B together.
Synthetic settings. For the synthetic experiments, we do grid search over the learning rate from (le - 4, le 3, le 2, le }} and the number of epochs from {10, 20, 30, 40, 50). We pick the best results for all the experiments. We choose the learning rate to be 1e 3, the number of epochs to be 30, and the batch size to be 50. For regression task, report the Spearman's 's correlation score For classification task, we report the classification accuracy.
Sentiment analysis. For the sentiment analysis experiments, we randomly split the data into train- ing, dev and test sets with percentages 80%, 10%, and 10% respectively. We follow the protocol of Lei et al. (2018) to set up our model for the sentiment analysis experiments.
The default hidden dimension of the model (c.g. LSTM) is set to be 200, but we vary this parameter for the model capacity experiments. We report the accuracy score on the test set as the performance metric.
24

---

Published as a conference paper at ICLR 2020
MTL performance over STL
MTL performance over STL
(Speaman correlation)
0
(Accuracy)
-0.1
-0.2
-0.2
-0.4
0
0.5
1.0
0.5
1.0
Cosine distance between tasks
Cosine distance between tasks
(a) Linear regression tasks
(b) Logistic classification tasks
MTL performance over STL
0.02
MTL performance over STL
0.05
(Spearman correlation)
(Acouracy)
-0.02
-0.04
-0.05
0.5
1.0
0.5
1.0
Cosine distance between tasks
Cosine distance between tasks
(c) Regression tasks with ReLU non-linearity
(d) Classification tasks with ReL.U non-linearity
Figure 7: Comparing MTL model performance over different task similarity. For (a) and (c), MTL trains two regression tasks; For (b) and (d), MTL trains two classification tasks. For regression tasks, use spearman correlation use accuracy as the metric. We report the average model performance over two tasks. The r-axis denotes the cosine distance, i.e.
model performance indicator. For classification tasks,
02).
GLUE. For the GLUE experiments, the training procedure is used the alignment modules and the output modules. Due to the complexity of the BERT module, which involves 24 layers of non-linear transformations.
We fix the BERT LARGE module during the training process to examine the effect of adding the alignment modules to the training process. In general, even after fine-tuning the BERT, module on a set tasks, is always possible to add our alignment modules and apply Algorithm 1.
For the training parameters, apply grid search tune the learning rate from (2c- 5, 3e-5, le-5) and the number of epochs from (2.3. 5, 10). We choose the learning rate to be 2c-5, the number of epochs to 5, and with batch size 16 for all the experiments.
We use the GLUE evaluation metric (cf. Wang et al. (2018b)) and report the scores on the develop- ment the performance metric.
ChestX-ray14. For the ChestX-ray 14 experiments, we use the configuration suggested by Rajpurkar al. (2017) and report the AUC score on the test after fine-tuning the model for 20 epochs.
B.4 EXTENDED SYNTHETIC EXPERIMENTS
Varying cosine similarity on linear and ReLU models. We demonstrate the effect of cosine simi- larity in synthetic settings for both regression and classification tasks.
Synthetic tasks. We start with linear settings. We generate 20 synthetic task datasets (either for regression tasks, classification tasks) based on data generation procedure and vary the task simi- larity between task and task We run the experiment with a different dataset pairs (dataset 1 and dataset i).
25

---

Published as a conference paper at ICLR 2020
Sim. 0.9
Sim. 0.5 Sim. 0.3
Sim. 0.1
Sim. 0.9
Sim. 0.5 Sim. 0.3
Sim. 0.1
Performance Improvements
Sim. 0.7
Performance Improvements
Sim. 0.7
0.2
0.1
0
-0.1
-0.2
-0.2
2000
4000
6000
8000
2000
4000
6000
8000
# Data size
# Data size
(a) Regression tasks with non-lincane con-lincarity
(b) Classification tasks with non non- lincarity
Figure 8: The performance improvement on the target task (MTL minus STL) by varying the cosine similarity of the two tasks' STL models. We observe that higher similarity between the STL models leads to better improvement on the target task.
Target task's Perf.: MTL- STL
Target task's Perf: MTL STL
Positive Transfer
Positive Transfer
0.2
Baseline
Algorithm
0.1
0.1
Baseline
Algorithm
-0.1
-0.1
Negative Transfer
Negative Transfer
-0.2
-0.2
2000
4000
6000 8000
8000 # Data samples of source task
2000
4000
6000
# Data samples of source task
(a) Linear regression tasks
(b) Regression tasks with ReLU activation
Figure 9: Comparing Algorithm to the baseline MTL training the synthetic example in Section 2.3. Algorithm corrects the negative transfer phenomenon observed in Figure 3.
After generating the tasks, we compare the performance gap between MTL. and STL model.
Results. From Figure 7a and Figure 7a, we find that for both regression and classification settings, with the larger task similarity the MTL outperforms more than STL model and the negative transfer could occur if the task similarity is too small.
ReLU settings. We also consider a ReLU-activated model. We use the same setup as the linear setting, but apply a ReLU activation to generate the data. Similar results are shown in Figure 7c, 7d.
Higher rank regimes for ReLU settings. We provide further validation of our results on ReLU- activated models.
Synthetic tasks. this synthetic experiment, there are two sets of model parameters O C Rdxx and O2 C Raxr (d 100 and 10). O is a fixed random rotation matrix and there are 100 data points for task 1. Task 2's model parameter is O = a@; + +(1 a)O', where is also fixed rotation matrix that is orthogonal to θ1. Note that is the cosine valuc/similarity of the principal angle between O and ④,
We then generate X1 C Rm Xd and X2 Rd from Gaussian. For each task, the labels 8/ = ReL, + Es, where e E IR" is the all ones vector and E; is a random Gaussian noise.
MTL with ReLU activations and capacity H 10 to co-train the two tasks. The goal is to see how different levels of or similarity affects the transfer from task two to task one. Note that this setting parallels the ReI.U setting of Theorem but applies rank 5.
Given the two tasks,
26

---

Published as a conference paper at ICLR 2020
0.9
MTL-Avg. peak
Accuracy
0.8
MTL
STIL
0.7
STL-UUBJ
STL-PPA
STL-TREC
LSTM
0.6
100
200
300
400
500
Model capacity
Figure 10: Cross validation to choose the best performing model capacity for each model.
0.80
MVL-Avg peak
MTL--ugg peak
MTL-Avg peak
Accurady
0.78
STL-SST peak
Acouracy
0.85
STL-SST
STL-SST
Acouracy
0.85
STL-SST peak
STL-MR peak
0.80
ATL STL-SST STI-MR
0.80
STL-MR CNN
STL-MR peak
LSTM
0.75
100
200
300
400
500
500
1000
100
200
300
400
500
capacity
Model capacity
Model capacity
Figure 11: Validation MLP, CNN and LSTM models for sentiment analysis tasks.
Results. In Figure 8 show that the data size, the cosine similarity between the STL solutions and alignment of covariances continue affect the rate of transfer the new settings. The study shows that our conceptual results are applicable to wide range of settings.
Evaluating Algorithm on linear and R-Atvvctiede models. We consider the synthetic ex- ample in Section 2.3 to compare Algorithm and the baseline MTL training. Recall that in the example, when the source and target tasks have different covariance matrices, MTL causes negative transfer the target task. Our hypothesis in this experiment is to show that Algorithm can correct the misalignment and the negative transfer.
Synthetic tasks. We evaluate both linear and ReLU regression tasks. The linear case follows the example in Section 2.3. For the ReLU case, the data is generated according to the previous example.
Results. Figure 9 confirms the hypothesis. We observe that Algorithm corrects the negative transfer the regime where the source task only has limited amount of data. Furthermore, Algorithm matches the baseline MTL training when the source task has sufficiently many data points.
B.5 EXTENDED ABLATION STUDIES
Cross validation for choosing model capacities. We provide a cross validation experiment to indicate how choose the best performing model capacities in Figure 1. This is done on the six sentiment analysis tasks trained with an LSTM layer.
In Figure 10, vary the model capacities to plot the validation accuracies of the MTL model trained with all six tasks and the STL model for each task. The result complements Table in Section 3.3.
Choosing model capacities for CNN and MIP. Next verify result model capacities for CNN and MLP models. We select the SST and MR datasets from the sentiment analysis tasks for this experiment. We train all three models CNN, MLP and LSTM by varying the capacities.
Results. From Figure 11 we observe that the best performing MTL model capacity is less than total best performing model capacities of STL model all models.
27

---

Published as a conference paper at ICLR 2020
The effect of label noise Algorithm 2. To evaluate the robustness of Algorithm 2 in the presence of label noise, we conduct the following experiment. First, we subsample 10% the ChestX-ray -ray 14 dataset and select two tasks from it. Then, randomly pick one task to add 20% of noise to its labels by randomly flipping them with probability 0.5. We compare the performance of training both tasks using reweighting scheme (Algorithm 2) the reweighting techniques of Kendall et al. (2018) and the unweighted loss scheme.
Results. On 10 randomly chosen task pairs, our method improves over the unweighted training scheme by 1.0% AUC score and 0.4% AUC score over Kendall et al. (2018) averaged over the 10 task pairs. Figure shows example task pairs from our evaluation.
Unweighted loss
Kendall et al.
Algorithm
0.8
MTL performance
0.7
0.6
0.5
0.4
Consolidation Consolidation Cardiomegaly
Consolidation Infiltration
Edema Atelectasis
Fibrosis Consolidation
Edema
Figure 12: Comparing Algorithm 2 to the unweighted scheme and Kendall et al. (2018).
28


Reviews:
[{"review_id": "rkl9a2T3KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzed the principles for a successful transfer in the hard-parameter sharing multitask learning model. They analyzed three key factors of multi-task learning on linear model and relu linear model: model capacity (output dimension after common transformation), task covariance (similarity between tasks) and optimization strategy (influence of re-weighting algorithm), with theoretical guarantees. Finally they evaluated their assumptions on the state-of-the-art multi-task framework (e.g GLUE,CheXNet), showing the benefits of the proposed algorithm.\n\nMain comments:\n\nThis paper is highly interesting and strong. The author systematically analyzed the factors to ensure a good multi-task learning. The discovering is coherent with with previous works, and it also brings new theoretical insights (e.g. sufficient conditions to induce a positive transfer in Theorem 2). The proof is non-trivial and seems technically sound.\n\nMoreover, they validated their theoretical assumptions on the large scale and diverse datasets (e.g NLP tasks, medical tasks) with state-of-the-art baselines, which verified the correctness of the theory and indicated strong practical implications.\n\nMinor comments:\nThe main message of the paper is clear but some parts still confuse me:\n1. I suggest the author to merge the Figure 3 and Data generation (Page 4) part for a better presentation. e,g which “diff.covariance” is task 3 or 4 ?  And why we use different rotation matrix Q_i ? \n\n2. In algorithm 1 (Page 5) , I suggest the author use a formal equation (like algorithm 2) instead of descriptive words.\n     -- Step 2, I have trouble in understading this step.\n     -- Step 3, how to jointly minimize R_1,\\dots, R_k, A_1, \\dots, A_k ? we use loss (3)  or other losses ?\n     -- I suggest that the author release the code for a better understanding.\n\n3. For theorem 2, can we find some “optimal” c to optimize the right part ? Since 6c + \\frac{1}{1-3c}\\frac{\\epsilon}{\\X_2\\theta_2} might be further optimized \n\n4. In section 3.3. (Figure 6)  of the real neural network, the model capacity is the dimension of Z or simply the dimension before last fc-layer ?\n\n5. Some parts in the appendix can be better illustrated:\n    (a) I am not clear how proposition 4 can derive proposition 1.\n    (b) Page 15, proving fact 8: last line \\frac{1}{k^4}sin(a^{prime},b^{prime}) should be \\frac{1}{k^4}sin^{2}(a^{prime},b^{prime}). \n\n\nOverall I think it is a good work with interesting discoverings for the multi-task learning. I think it will potentially inspire the community to have more thoughts about the transfer learning. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlRuaviKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to improve the multi-task learning from both theoretical and experimental viewpoints. More specifically, they study an architecture where there is a shared model for all of the tasks and a separate module specific to each task. They show that data similarity of the tasks, measured by task covariance is an important element for the tasks to be constructive or destructive. They theoretically find a sufficient condition that guarantee one task can transfer positively to the other; i.e. a lower bound of the number of data points that one task has to have. Consequently, they propose an algorithm which is basically applying a covariance alignment method to the input. \nThe paper is well-written, and easy to follow. \nPros:\nA new theoretical analysis for multi-task learning, which can give insight of how to improve it through data selection.\nThey empirically show that their algorithm improves the multi-task learning on average by 2.35%. \n\nCons:\nThere is not much of novelty in the algorithm and architecture. Their method is very similar to domain adaptation but for multi-learning setting.\nIn the Theorem 2, they have assumed parameter c <= 1/3. They have not provided any insight of how much restrictive this assumption is.  \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJeNT1T3YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission investigates multitask learning (MTL) and develops new theories around MTL with linear models and linear+ReLU. In the experimental section, the authors improve performance in sentiment analysis on subtasks of the GLUE benchmark (building on BERT - highly non-linear neural network) and show a SVD-based task loss reweighting scheme on an multi-label image classification dataset. \n\nThe submission is overall well written though some paragraphs (2.1-2.3, in particular the example section) would benefit from additional effort towards clearer sentences. One issue with the submission is that there is a significant gap between the theory and experimental sections as theory only covers linear models and the experiments don’t include linear models and purely focus on deep networks. The benefits of a bottleneck in multitask learning are well known (based empirical results). However, it is helpful that the additional theoretical results (given strong assumptions) provide some grounding. \nWhile the model with non-linear activation is mentioned at places, nearly all theorems rely on the linear model instead such that it might make sense to either work towards generalising the theorems or emphasising that most only apply to linear models.\n\nAdditional assumptions (1D labels, same input dimensionality across all tasks) should be emphasised to clarify limitations of all derivations. Where previous work addressed model similarity it often looks at models in the context of existing datasets (i.e. taking the data into account to describe boundaries etc) such that the emphasised novelty at looking at data similarity is to be taken with a grain of salt.\n\nOverall, the paper contributes to the conversation around multitask learning but would benefit from comparing again external work on multitask learning (e.g. see under minor) and from bridging between theory and experiments (e.g. experiments with the models described in the theory section - linear/ReLU).\n\nMinor:\n- y is used as label and as data terminology at different parts of the text. \n- the model in the first set of experiments has lower capacity than most models individually, suggesting that the capacity should be smaller even for individual tasks to prevent overfitting.\n- An ablation over model capacities is mentioned but missing for 3.3\n- comparison against existing multitask loss weighting techniques should be performed [1]\n\n\n[1] Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics\nAlex Kendall, Yarin Gal, Roberto Cipolla 2017\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Many existing approaches in multi-task learning rely on intuitions about how to transfer information. This paper, instead, tries to answer what does "information transfer" even mean in this context. Such ideas have already been presented in the past, but the approach taken here is novel, rigorous and well-explained.

The reviewers agreed that this is a good paper, although they wished to see the analysis conducted using more practical models. 

For the camera ready version it would help to make the paper look less dense.

Decision: Accept (Poster)