$\alpha$VIL: Learning to Leverage Auxiliary Tasks for Multitask Learning
Authors: Rafael Kourdis
Gabriel Gordon-Hall
Philip John Gorinski
Venue:  (2021)
Keywords: multitask learning
meta-optimization
deep learning

Abstract: Multitask Learning is a Machine Learning paradigm that aims to train a range of (usually related) tasks with the help of a shared model. While the goal is often to improve the joint performance of all training tasks, another approach is to focus on the performance of a specific target task, while treating the remaining ones as auxiliary data from which to possibly leverage positive transfer towards the target during training. In such settings, it becomes important to estimate the positive or negative influence auxiliary tasks will have on the target. While many ways have been proposed to estimate task weights before or during training they typically rely on heuristics or extensive search of the weighting space. We propose a novel method called $\alpha$-Variable Importance Learning ($\alpha$VIL) that is able to adjust task weights dynamically during model training, by making direct use of task-specific updates of the underlying model's parameters between training epochs. Experiments indicate that $\alpha$VIL is able to outperform other Multitask Learning approaches in a variety of settings. To our knowledge, this is the first attempt at making direct use of model updates for task weight estimation.

Under review as a conference paper at ICLR 2021
A VIL: LEARNING TO LEVERAGE AUXILIARY TASKS FOR MULTITASK LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Multitask Learning is Machine Learning paradigm that aims to train range of (usually related) tasks with the help of a shared model. While the goal is often to improve the joint performance of all training tasks, another approach is focus on the performance auxiliary data from which possibly leverage positive transfer towards the target during training. In such settings, it becomes important to estimate the positive or negative influence auxiliary tasks will have on the target. While many ways have been proposed to estimate task weights before or during training they typi- cally rely on heuristics or extensive search of the weighting space. We propose a novel method called a- Variable Importance Learning (a VIL) that is able to adjust task weights dynamically during model training, by making direct use of task- specific updates of the underlying model's parameters between training epochs. Experiments indicate that VVLL is able to outperform other Multitask Learning approaches in variety of settings. To knowledge, this is the first attempt at making direct use of model updates for task weight estimation.
specific target task, while treating the remaining ones as
INTRODUCTION
In Machine Learning, we often encounter tasks that are at least similar, if not even almost iden- tical. For example, in Computer Vision, multiple datasets might require object segmentation or recognition (Deng et al. 2009; LeCun al. 1998 Lin et al. (2144) whereas in Natural Language Processing, tasks can deal with sentence entailment (De Marneffe et al. (019) or paraphrase recog- nition (Quikk et al., (004), both of which share similarities and fall under the category of Natural Language Understanding.
Given that many such datasets accessible to researchers, naturally emerging question is whether we can leverage their commonalities in training setups. Multitask Learning (Caruana 1993) is a Machine Learning paradigm that aims to address the above by training group sufficiently similar tasks together. Instead of optimizing each individual task's 's objective, a shared underlying model is fit as maximize a global performance measure, for example a LeNet-like architecture (LeCun et al. 1998) for Computer Vision, or Transformer based encoder (Vaswani et al., (017)) 7) for Natural Language Processing problems. For a broader perspective of Multitask Learning approaches, we refer the reader to the overviews of Ruder (2017); Vandenhende et al. 2020))
In this paper we introduce VII,,, an approach to Multitask Learning that estimates individual task weights through direct, gradient-based metaoptimization specific model updates. To our knowledge, this the first attempt to leverage task-specific model deltas, that is, realized differences of model parameters before and after task's training steps, to directly optimize task weights for target task-oriented multitask learning. We perform initial experi- ments multitask setups in two domains, Computer Vision and Natural Language Understanding, and show that our method is able to successfully learn good weighting of classification tasks.
weighted accumulation of task-
RELATED WORK
Multitask Learning (MTL) can be divided into techniques which aim to improve a joint performance metric for a group of tasks (Caruana 1993), and methods which use auxiliary tasks to boost the performance of single targel task (Caruana 1998. Bingel & Søgaard 2017).

---

Under review as a conference paper at ICLR 2021
Some combinations of tasks suffer when their model parameters are shared, phenomenon that has been termed negative transfer. There have been efforts to identify the cause of negative transfer. Du et al. 2018) use negative cosine similarity between gradients heuristic for determining negative transfer between target and auxiliary tasks. Yu et al. (2200) suggest that these conflicting gradients detrimental training when the joint optimization landscape has high positive curvature and there is a large difference in gradient magnitudes between tasks. They address this by projecting task gradients onto the normal plane if they conflict with each other. [W al. (2020) hypothesize that the degree of transfer between tasks is influenced by the alignment of their data samples, and
propose an algorithm which adaptively aligns embedded inputs. [Sener & Koltun (2018) avoid the issue of negative transfer due to competing objectives altogether, by casting MTL as Multiobjective Optimization problem and searching for a Pareto optimal solution.
In this work, we focus on the target task approach to Multitask Learning, tackling the problem of auxiliary task selection and weighting to avoid negative transfer and maximally utilize positively re- lated tasks. Auxiliary tasks have been used to improve target task performance in Computer Vision, Reinforcement Learning (Jaderberg et al. 2016). and Natural Language Processing (Collobert et al.) 2011]. They commonly selected based on knowledge about which tasks should be beneficial to each other through the insight that they utilize similar features to the target task (Caruana 1998), or are grouped empirically (Søgaard & Goldberg 2016). While this may often result in successful task selection, such approaches have some obvious drawbacks. Manual feature-based selection re- quires the researcher to have deep knowledge of the available data, an undertaking that becomes ever more difficult with the introduction of more datasets. Furthermore, this approach is prone to failure when it comes Deep Learning, where model behaviour does not necessarily follow human intu- ition. Empirical task selection, e.g., through trialling various task combinations, quickly becomes computationally infeasible when the number of tasks becomes large.
Therefore, in both approaches to Multitask Learning (optimizing either a target task using auxiliary data or a global performance metric), automatic task weighting during training can be beneficial for optimally exploiting relationships between tasks.
To this end, Guo et al. (2019) use a two-staged approach; first, a subset of auxiliary tasks which are most likely to improve the main task's validation performance is selected, by utilizing Multi- Armed Bandit, the estimates of which are continuously updated during training. The second step makes use of Gaussian Process to infer mixing ratio for data points belonging to the selected tasks, which subsequently are used to train the model.
A different approach by Wang et al. 2020) aims to directly differentiate at cach training step the model's 's validation loss with respect to the probability of selecting instances of the training data (parametrised by scorer network). This approach is used in multilingual translation by training the scorer to output probabilities for all of the tasks' training data. However, this method relies on noisy, per step estimates of the gradients of the scorer's parameters as well as the analytical derivation of it depending on the optimizer used. Our method in comparison is agnostic to the optimization procedure used.
Most similarly to our method, Sivasankaran et al. 2017) recently introduced Discriminative Im- portance Weighting for acoustic model training. In their work, the authors train a model on the CHiME-3 dataset (Barker et al., 2015), adding 6 artificially perturbed datasets as auxiliary tasks. Their method relies on estimating model performances on the targeted validation data when train- ing tasks in isolation, and subsequently using those estimates proxy to adjust individual task weights. Our method differs from this approach by directly optimizing the target validation loss with respect to the weights applied to the model updates originating from training each task.
3 Q- VARIABLE IMPORTANCE LEARNING
The target taik-triented approach to Multitask Learning can be defined as follows. A set of classi- fication tasks T = {f1, f2. ... } are given, each associated with training and validation datasets, Dfrain and DdtT as well as a target task t' T. We want to find weights W = {W1, W2, Wn) capturing the importance of each task such that training the parameters 0 of Deep Neural Network on the weighted sum of losses for each task maximizes the model's 's performance on the target task's 's development set:
2

---

Under review as a conference paper at ICLR 2021
0" = arg min
Wi Σ
- (Di 0) s.b. L (Dir,, 0*) min L' (Dite,
(1)
2=0
where L (Dt;, 0) is defined the average loss over all data points in the dataset Dr; for network parameters computed using the appropriate loss function for task t; (in our case the standard cross entropy loss):
L (Du, iDe.l
Σ f(ark Uk; (sk, yk) E De,
The introduction of task weights in Equation I aimms scaling the tasks' model updates in way that positive transfer towards the target is exploited and negative transfer avoided. It is crucial therefore have efficient and reliable way of estimating the influence of tasks the target's performance, and of adjusting the weights accordingly during model training.
To this end, introduce a- a-Variable Importance Learning (aVVLL, novel method for target task- oriented Multitask training, outlined in Algorithm | a VIL introduces a number of additional pa- rameters a-variables into the model, which are associated with the actually realized ssec-specific model updates.
Algorithm 1: The a- Variibel Importance Learning algorithm.
Data: Model parameters 0; a set of tasks = {tl..... in target task t"; training data Dirain for each task ty development data D for the target task; maximum number of training epochs E; ratio p of tasks' training data to sample per epoch; number of a tuning steps
Result: updated parameters 0, optimized for performance on t"
W (we, = 4 T // initialize all task weights to
for
1...8 do
for t4 T do
Di L Dirain
// sample task lask-specific data
0,, + arg ming
E C (Di,, // task's model update starting at
wt,
8, 0r. 0
// task - speci fic we ght update, optimizing wrt. paramaters on 6 for s 1...s do
{ar, (2.,.,..... am)) + arg (ar L(D)ev 0 + 015 +... (()))
10
1 + 0 aj +... a W (wr, + (an, 1) f, ET]
During training, our approach first performs weighted task-specific model updates on a proportion of the available training data for each individual task, starting from the current model parameters. It collects the resulting model deltas, i.e., the differences between the model's parameters before and after the singletask update and resets the model. After this delta collection phase, the optimal mixing factors, that is, {a1, {2,,a2,... ....... azz) } of the model updates are found, such that the parameters resulting from the interpolation scaled task-specific ò's minimize the loss on the target task's development data.
The a parameters can be optimized through any type optimization method however, since our models end to end differentiable, can backpropagate directly and use gradient descent.
Once we have found the optimal mixing ratio of task updates, we write the new state back to the model, and update the task weights subject the optimized a parameters.
The task weight update rule (line 11 in Algorithm D combined with the weighted task- ssecifc model updates (line 5) tries capture the intuition that if task update up- down-scaled in the a-tuning stage, we likely want to update the parameters more/less for this task, during the next delta collection phase.
3

---

Under review as a conference paper at ICLR 2021
3624
0 0123
Figure I: Examples images of digits found in MNIST (left) and the super-imposed digits present in MultiMNIST (right). Gold labels for the MultiMNIST examples for top-left/bottommrige -ight respec- tively are 3/5, 1/6, 2/2, 6/7.
EXPERIMENTS
To test the efficacy of VII., we apply it in two domains, Computer Vision (CV) and Natural Lan- guage Processing (NLP). We perform experiments on multitask version of the MNIST dataset (LeCun et al. 1998 and on number of well-established Natural Language Understanding (NLU) tasks. In all scenarios, evaluate a VIL against baselines that perform single task and standard multitask learning, as well as against a strong target task-oriented approach.
4.1 COMPUTER VISION
As the first benchmarking domain for VVL we chose Computer Vision, as Multitask Learning has a longstanding tradition in this field, with a variety of existing datasets. For our experiments, we use (Sener& & Koltun (2018)s 's variation of MultiMNIST, itself introduced in (Sabuu et al.) (2017) as an augmentation of the well-established MNIST dataset. In MNIST, the task is to classify a given hand-drawn digit (Figure I left). For MultiMNIST, two digits are overlaid, the first shifted to the top left and the second to the bottom right (Figure I right). The two tasks are to recognize each of the super-imposed digits. The resulting MultiMNIST dataset contains total of 10.000 test and 60,000 training instances, of which we sample 10,000 at random to use for validation.
For all Multitask experiments on MultiMNIST, we use classification architecture similar to Sener & Koltun (018)) as depicted in Figure 2 The model comprises shared convolutional encoder, where the first convoluational layer has filters with kernel size 5 and the second uses the same kernel size but 20 filters. Both max pooling layers are of size 2x2, and the shared fully connected layer is of dimensionality 320x50. The encoded images are passed into task-specific heads of size 50x10, used to classify the top-left and nottom-right digit respectively.
We compare VIL to a single task baseline, where we remove one of the classification heads from the model, as well standard multitask baseline in which both heads are trained jointly, each of which receives the shared encoder output for a given image and updates the model to solve its specific task, averaging their updates for the image. For the standard multitask baseline, two snapshots of the model during training, each of which performs best on either the top-left or the bottom-right digit on the development sets. For V'LLL, predefine either one of these tasks as the target and try to optimize its performance using the other task as auxiliary data. We also compare our method to the Discriminative Importance Weighting (DIW) approach of Sivasannaaaan et (201)) which provides a very strong target taik-ori optimization method. Their approach is relatively similar our own however, contrast to a VII., DIW collects unweighted single-task updates, and evaluates each individual task on the specified target task. It then performs actual weighted multitask updates an inner loop, evaluates the jointly updated model, and adjusts individual task weights with respect to the difference in evaluation results, after which the model is reset and re-trained with
Fully Connected
Fully Connected
Conv
MaxPool
ReLU
Conv
MaxPool
ReLU
28x28
Fully Connected
Figure General Multitask model architecture for MultiMNIST experiments.
4

---

Under review as a conference paper at ICLR 2021
Development Set Accuracy
Task 1 (Top-Left Digit)
Task 2 (Bottom-Right Digit)
Min
Max
Mean
Std.
Min
Max
Mean
Std.
Singletask
96.12
97.25
96.73
0.30
94.95
96.48
95.57
0.46
Multitask
95.99
96.71
96.37
0.20
94.43
95.54
95.00
0.33
DIW
95.94
97.14
96.84
0.27
94.83
96.32
95.60
0.45
aVIL
96.18
97.24
96.91
0.25
94.78
96.22
95.89
0.30
Test Set Accuracy
Singletask
95.90
96.89
96.44
0.35
93.97
95.48
94.83
0.44
Multitask
95.40
96.29
95.92
0.24
94.34
95.02
94.67
0.23
DIW 95.67
96.85
96.46
0.29
94.09
95.52
94.93
0.42
VVIL
95.74
96.95
96.56
0.32
94.13
95.58
95.16
0.33
Table 1: Model classification accuracy and standard deviation over 20 random seeds, on Mul- tiMNIST development and test sets, for single task and standard multitask baselines, Discriminative Importance Weighting, and a VIL. Results in bold indicate best overall approach.
the new weights, until improvement over the previous iteration's 's target task performance has been achieved.
For all experiments, use batch size of 256 and SGD the model optimizer with learning rate set to 0.05, momentum to 0.9. Dropout is not used. We train on the entire MultiMNIST training set before evaluating (single task, multitask) or weight tuning on the development data (DIW, VVL... For a VIL, we set the number of tuning steps 8= 10, and use SGD with learning rate set to 0.005 and momentum 0.5 as the meta-optimizer tune the parameters. The delta collection weights, Wt, as well as the task importance weights of DIW are clamped to [10-5 -0000-000 I Similarly. we set an early stopping criterion for the re-weighting loop of DIW to break after 10 epochs without improving over the previous performance We train all models for 100 episodes. keeping the best- performing snapshots with respect to the development accuracy and target task?] We average over 20 random seeds, and report in Table [ the minimum, maximum and mean classification accuracy on the MultiMNIST development and test sets, as well as the models' standard deviation.
We can see from Table T the second task, classifying the bottom-right digit, seems to be some- what harder for the model to learn than the first, as reflected by the worse model performance in all settings for this task. Furthermore, when training in the standard multitask setting, model perfor- mance actually decreases for both tasks. This indicates that in MultiMNIST, there exists negative transfer between the two tasks. We can also see that both target task-oriented Multitask approaches are able to rectify the negative transfer problem, and in fact even improve over the models trained on the single tasks in isolation.
Across the board, a VIL achieves the best overall mean accuracy, on both tasks' development and test sets, while keeping comparably low standard deviation. Crucially, a VII not only brings Multitask performance back to the single task level, but outperforms the single task baseline, as well as the DIW target task-oriented training approach.
Figure SecureBshone shows the a-parameters over the course of training (left) and the corresponding normalized weights of the two tasks (right). The target task t" set to task As expected, due the existence of negative transfer between the tasks, the algorithm initially weights model updates originating from the main task and auxiliary task with on > and <2<1 respectively, quickly driving the task weights down. Finally, a's 's settle at z 1, as task 2 has been essentially excluded from the training.
In practice, training with even a slightly negative loss weight causes parameters to massively overshoot. II their original paper, the authors mention such early stopping criterion however, we found that training can enter an infinite loop without it.
For standard multitask training, we save 5 snapshots of the model, one for each best performance the individual tasks
5

---

CoPA: This task provides premise ("My body cast a shadow over the grass."), "), two alternative choices ("h sun was rising."/"The "hhe grass cut." "), and relation (" ("cause") as input. The model's task to determine whether choicel or choice2 is the more likely continuation of the premise, given Under review as a conference paper at ICLR 2021
1.0
Task --- Task
Normatzze Weight
140
0-values
8.90
06
Task Task
0.50
0.85
02
0.50
20
00
100
Epoch
Epoch
Figure 3: a parameter values (left) and task-specific weights (right) over the course of a VVL training on MultiMNIST VII. Task is the target task.
4.2 NATURAL LANGUAGE UNDERSTANDING
We also test VVL in the domain Natural Language Processing. In particular, we interested in the learning of Natural Language Understanding (NLU) tasks.
The field of NLU has recently regained traction research area, with the introduction of the Trans- former architecture /asswani et al. (2017) and subsequent advent of very large language models such as BERT (Devlin al. 2019) and similar models. These very deep models are trained on plethora of data, and (at least seem to) incorporate large amount of linguistic knowledge (Tenney et al., 2019 Staltūnatt lacobacci 2020), which makes them ideal for downstream tasks like NLU.
Natural Language Understanding comprises wide variety of established datasets and tasks, each dealing with different aspects of the broader field. This provides rich and interesting resource for Multitask Learning research, NLU tasks are at least at first glance related in the kind of linguistic knowledge they require. They therefore lend themselves being trained jointly, yet tasks might in reality make use of different aspects of the underlying shared model, leading potentially to negative transfer.
For first test of a VIL in the NLU domain, we limit our experiments to 5 commonly used tasks that are also represented in the GLUE and SuperGLUE (Wang et al. 2019a b) research benchmarks: BammitmentBank (De Marneffe et al. 2019 CB), Choice of Plausible Alternatives (Roemmele
et al. 2011 CoPA), Microsoft Research Paraphrase Corpus (Quirk et al. 2004] MRPC), Recog- nizing Textual Entailment (Dagan al. 2006 Bar Haim al. [006)] Giampiccolo al.; 2007] Bentivogli et al.) 2009 RTE), and Winograd Natural Language Inference (WNLI), itself based on
the Winograd Schema Challenge (Levesque et al. 2011) Brief descriptions and examples for each of the tasks are given below. The tasks were chosen for their relatively small data sizes, to allow for faster experimentation.
CommitmentBank: CB is a three-way classification task, where model inputs consist of a premise text of one or more sentences, c.g. "B: Oh, well that's good. A: but she really doesn's Nobody thought she would adjust", and a hypothesis like "she would adjust" which can either be entailed in the premise, contradict it, or be neutral. CB the smallest of our NLU tasks, and contains 250 examples of training data, as well as 56 for validation and 250 for testing.
the relation. The CoPA dataset provides 400 training well as 100 and 500 validation and test instances.
This is paraphrase recognition task, in which the model is shown two sentences, and to classify whether or not they are paraphrases. MRPC constitutes our largest dataset for NLU experiments, with 3668 training and 408 validation examples, with 1 1725 for testing.
MRPC:
Recognizing Textual Entailment: RTE is a two-way classification task, where given a premise, for example "Things are easy when you't re big Japan." and a hypothesis, such " "You 're big

---

Under review as a conference paper at ICLR 2021
Classifiation Head Task 1
Classifiation Head Task 2
Classifiation Head Task
RoBERto Encoder
Encoder Layer
Encoder Layer 2
Encoder Layer
W.
W2
Wn
Woi
Wozz Wass
Win
</5>
Figure 4: General Multitask model architecture for NLU experiments.
in Japan.", the task is correctly classify whether the former entails the latter. The RTE data is our second larges dataset, and contains 2490 instances for training, as well as 277 and 3000 for validation and testing respectively.
WNLI: This task is about classifying Natural Language Inference. The input consists of a short text ("I stuck a pin through a carrot. When I pulled the pin out, it had a hole.") ") and follow-up sentence ( ("The carrot had a hole."), and the model has to correctly classify whether the sentence can be inferred from the preceding text. WNLI comprises training, validation, and test sets containing 635, 71, and 146 instances, respectively.
To address the Multitask learning problem on these 5 NLU tasks, similar to the MultiMNIST image classification setup above, we employ an architecture of joint encoder and task task-specific heads which depicted in Figure 4 We use a RoBERTa model (Liu et al.) 2019) as the underlying shared encoder, and prepare the data for all tasks such that is suitable as Transformer encoder input. For our experiments, we employ the pre-trained RoBERTàpas: encoder provided by Huggingfacq") which consists of 12 encoder layers, and outputs 768-din -f8-dimensional embedding vector for each input token. On top of the shared encoder we add one classification head for each the 5 NLU tasks. Each head takes as input the final RoBERTa encoding of the start-of-sequent layer of dimensionality 768x256, ReLU, followed by another linear layer into the classification space (2 output units for CoPA, MRPC, RTE, and WNLI, 3 output units for CB).
token < <S>, and employs linear
As for the Computer Vision experiments above, we perform experiments on the 5 NLU tasks with standard single task and multitask baselines, Discriminative Importance Weighting. and a VIL. All experiments use the same model architecture, except in the single task setup where all heads but that for the target task are disabled. We use AdamW (Loshchilov & Hutter (017) as the optimizer for the base model, with learning rate of 5e-0, c= Ic-", and weight decay of 0.01. For DIW, we use a weight update learning rate of 0.1, and we use SGD with a learning rate of 0.001 and momentum of 0.5 for a- cevariable optimization. Parallel to the previous experiments, we use 10 update steps for a VIL, and an early stopping patience of 10 for DIW. Other than for MultiNNIST, where both tasks share common input, each NLU task comes with distinct model inputs and outputs, well as with datasets of very different sizes. We therefore do not wait for an entire episode before model evaluation, but instead increase the evaluation frequency. This is necessary as of the tasks tend to overshoot their optimal point during training, and we might miss the true hest model performance if we wait for entire episode finish. To increase the evaluation frequency, epoch 25% of the respective training sets, train the model on batches of this quarter of the total data, and evaluate. For DIW and a VIL we also use this 25% split for weight adjustment. The batch size is set to 8 for all tasks, and we train for a total of 20 epochs, keeping the best performing snapshot per task.
sample each
https://hugginggface co/ robe rta base
7

---

Under review as a conference paper at ICLR 2021
CB
CoPA
MRPC
RTE
WNLI
Singletask
90.18
84.0
56.00
47.6
90.01
87.8
80.23
73.1
55.99
65.1
Multitask
95.98
94.0
68.00
60.0
89.83
87.8
80.05
75.3
56.69
65.1
DIW
98.66
93.6
64.25
56.4
90.69
88.2
81.14
75.1
59.15
62.3
aVIL
97.77
92.8
67.33
59.0
89.22
88.2
81.05
75.8
57.75
65.1
Table 2: Average classification accuracy on the tested NLU development datasets over 4 random seeds per task (left column per task), as well GLUE/ / SuperGLUE test accuracy for ensembles (right column), for single task and standard multitask baselines, Discriminative Importance Weight- ing, and a VIL. Results in bold indicate best overall training approach. For space constraints, we show only average development accuracy and final enemble test scores.
Table 2 summarizes the results of the NLU experiment, with model accuracies on the tasks' de- velopment sets over 4 random seeds. To obtain test set accuracies, we submit for each method the predictions of ensembles of the trained models to the GLUE and SuperGL.UE benchmarks
Even though these experiments were carried out on very limited number of tasks and random seeds, and should accordingly be interpreted with care, it is clear that Discriminative Importance Weight- ing constitutes a very strong Multitask Learning system for comparison, especially with respect to development set performance.
This should of little surprise: First, DIW does optimize its task weights directly based on the target task's development accuracy, re- re-weighting and re-training at each epoch until highest score is found. This in contrast to a VIL, which performs predetermined number of optimization steps the average development set loss, and just interpolation of model updates per epoch. Second, as there is no perfect correlation between a lower average loss and higher absolute accuracy decreasing the first slightly not guaranteed to immediately increase the second VIL might not find a new best development accuracy each epoch.
However, the picture changes when considering actual performance unseen test data. While DIW almost consistently performs best on the development data, VVLL actually outperforms it on the final test scores. For 3 out of the tested NLU tasks, the VVL ensembles are ranked first on test (shared with DIW on MRPC, and singletask and standard multitask on WNLI). For one more dataset (CoPA), a VIL comes second, trailing the best system by just point.
We conjecture that DIW is more prone to overfitting than a VIL, precisely because of DIW's heavy reliance on tuning on dev multiple times each epoch until finding a new best performance. This might be less severe in settings where training, development, and test datasets are both large and sufficiently similar, as is the case in the MultiMNIST experiment above where no such performance discrepancies seem manifest. However, in the NLU domain with less data and potentially large difference between training, development, and test examples, overfitting constitutes a more severe problem, which VIL seems be able to better avoid.
5 CONCLUSIONS AND FUTURE WORK
In this work have introduced VIL, novel algorithm for target task-oriented multitask training- a VIL uses tpsc-fpe weights which are tuned via metaoptimization of additional model param- eters, with respect to target task loss. Experiments in two different domains, Computer Vision and NLP, indicate that a VIL is able to successfully learn good task weights, and can lead to increased target-task performance over singletask and standard multitask baselines, well strong target task-oriented optimization approach.
aVIL's formulation very flexible and allows for many variations in its (meta)optimizzaioon ap- proach. In the future, we would like to experiment more with different ways to optimize a param- eters than standard SGD. Also, VVL does not currently perform joint optimization iteration after its estimation task and re-weighting, which could lead to further performance gains.
httpps:/ http::///gglubbbeccmar..kkk https://super.ghh http://ppper...uuueeechmmmmr www.gggwwweencme..cccmm
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
J. Barker, R. Marxer, E. Vincent, and Watanabe. The Third 'CHiME" Speech Separation and Recognition Challenge: Dataset, Task and Baselines. In JEEE Workshop Automatic Speech Recognition and Understanding (ASRU), pp. 504-511, 2015.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL. recognizngg textual entailment challenge. 2009.
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 2, Short Papers, pp. 164-169, Valencia, Spain, April 2017. Association for Computational Linguistics.
Linguitsso
Rich Caruana. Multitask Learning, pp. 95-133. Springer US, Boston, MA, 1998. ISBN 978-1- 4615-5529-2. doi: 10.
Richard Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pp. 41-48. Morgan Kaufmann, 1993.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493 2537, 2011.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, pp- 177-190. Springer. 2006.
De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: In- vestigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung 23. Data can be found at
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019.
Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Razvan Pascanu, and Balaji Lakshmi-- narayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:181..02224, 2018.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third recognzzin textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguistics, 2007.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. AutoSeM: Automatic task selection and mixing in multi-task learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3520-3531, Minneapolis, Minnesota, June 2019, Association for Com- putational Linguistics. doi: 10.18653/v1/N11-133 1355. URL https: www. wwwwwwebb... .wweeb.orgr...
9 1355
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 161 1.05997, 2016.

---

Under review as a conference paper at ICLR 2021
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 96(11):2278-233 1998.
Hector J Levesque, Emest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv preprint /97.1 2019.
lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1.0051001,,01,, 2017.
Chris Quirk, Chris Brockett, and William B Dolan. Monolingual machine translation for paraphrase generation. In Proceedings of the 2004 conference on empirical methods natural language processing, pp. 142, 2004.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 20/1 AAAI Spring Symposium Series, 2011.
Sebastian Ruder. An Overview of Multi-Task Learning in Deep Neural Networks. arXiv preprint arXiv:1706.05090 05098, 2017.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objectivy optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Jesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 527-538. Curran Associates, Inc., 2018.
Sunit Sivasankaran, Emmanuel Vincent, and Irina Illina. Discriminative Importance Weighting of Augmented Training Data for Acoustic Model Training. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4885-4889. IEEE, 2017.
Anders Søgaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 231-235, 2016.
leva Staliūnaité and Ignacio lacobacci. Compositional and lexical semantics in roberta, bert and distilbert: A case study on coqa, 2020.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint FrxXiv:1905.09900 2019.
Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Revisiting Multi-Task Learning in the Deep Learning Era. arXiv preprint arXiv:2004 13379, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGI. UUE stickier benchmark for general-purper language understanding systems. arXiv preprint 1905.00537, 2019a.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-tase benchmark and analysis platform for natural language understanding. 2019b. In the Proceedings of ICLR.
10

---

Under review as a conference paper at ICLR 2021
Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards, 2020.
Sen Wu, Hongyang R. Zhang, and Christopher R. Understanding and improving information transfer in multi-task learning. In International Conference on Learming Representations, 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.
Gradient surgery for multi-task learning. arXiv preprint arXiv:200100678) 2020.


Reviews:
[{"review_id": "y6RkFNGKTm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel multi-task learning method which adjusts task weights dynamically during training, by exploiting task-specific updates of the model parameters between training epochs. Specifically, the proposed model takes the differences between the model’s parameters before and after the singletask update, after that the mixing factors of the model updates are found based on the differences to minimize the loss on the target task’s development data. Empirical studies are performed on tasks of computer vision and natural language understanding.\n\nThe paper is well written and easy to follow, the authors summarize the related work in a clear manner. The proposed methodology is intuitive and well-motivated, in the meantime, it is flexible and can be generalized to other variations in terms of models and tasks. \n\nMy major concern about the paper include the following:\n1)\tAlthough the proposed method is intuitive and straightforward, it would be necessary to provide some theoretical justification or a formal analysis of the proposed methodology.\n\n2)\tConsidering the lack of a theoretical justification, the experimental results are not convincing enough to justify the proposed method. The baselines chosen include standard multi-task learning and one single task-oriented approach, which is somewhat limited. Even so, on both of the computer vision and natural language understanding tasks, the proposed method doesn’t consistently outperform the baselines in most cases. The authors did provide sufficient analysis, nevertheless, it doesn’t justify the effectiveness of the algorithm. \n\nBased on the concerns above, the paper can be improved from both the theoretical and empirical perspectives. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "he0-t4t4_YF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents an algorithm for multitask learning that learns task weights via an EM-like approach that alternates between updating the model parameters (using task weights) and updating the task weights (using current model parameters, based on the target task development set). \n\nExperiments: They compare against single task training, standard multitask training (though this isn’t described very clearly but roughly is training jointly on tasks), and another method for learning task weights, Discriminative Importance Weighting (DIW).\nThey present experiments on MultiMNIST, where the two tasks are to predict the task in the top left and bottom right of two superimposed digits. The proposed algorithm has the beast mean performance, but results are within a standard deviation of the baselines. They also present experiments on 5 NLU tasks (CommitmentBank, COPA, MRPC, RTE, WNLI) with the same baselines. The results on these tasks are mixed, with all multitask methods outperforming single task training (except on WNLI, which is a bit degenerate).\n\nOverall, this paper needs a bit more work. The proposed is quite ad hoc, and with little justification, it’s not clear why we should be doing any of the things the algorithm proposes. For example, in line 11 of Algorithm 1, I don’t understand the intuition arbitrarily subtracting 1 from all weights. From a novelty perspective, I’m not convinced the proposed method is different enough from existing methods. Dynamic task weighting is not particularly new (e.g. the baseline method, DIW, they compare against), and their method starts to look a lot like meta-learning of task weights (like MAML [1], [2], or [3]). The results from the experiments are not convincing to me. On MNIST, the results between all methods are fairly close together, and on the NLU tasks, there’s no clear best algorithm.\n\nAdditional notes and questions\n1. For the “standard multitask baseline”, are the tasks balanced in size? Do you deterministically train on a batch from both or is it stochastic? This is mostly relevant for the NLU tasks, which have fairly different sizes.\n2. On MNIST, given that the algorithm sets the weight of one task to 1.0 and the other to 0.0, why is this algorithm outperforming single-task training?\n3. On a similar note, it'd be nice to see a sanity check experiment that the learned weights are sensible (e.g. one task has random labels) or an examples of where the learned weights are binary.\n4. I appreciate that the authors report min/max/mean/std of 20 runs on MNIST. It would be nice to see the standard deviations for the NLU tasks for consistency and given the fact that the standard deviations on the MNIST task were important in differentiating significant differences. Similarly, it would be nice to see how the task weights evolve.\n\nStyle notes\n* Huggingface Transformers now has a citation\n* Multitask Learning; Computer Vision; Natural Language Processing/Understanding: lowercase\n* “Singletask” should probably be hyphenated or two words.\n* “10.000” → “10,000”\n* Table 2 could really use headers over the two columns within each task.\n\n[1] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. \"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.\" ICML. 2017.\n[2] Shu, Jun, et al. \"Meta-weight-net: Learning an explicit mapping for sample weighting.\" Advances in Neural Information Processing Systems. 2019.\n[3] Wang, Xinyi, Yulia Tsvetkov, and Graham Neubig. \"Balancing training for multilingual neural machine translation.\" arXiv preprint 2004.06748 (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "33OOEexVtIv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new approach for multi-task learning that estimates the individual task weights through gradient-based meta-optimization on a weighted accumulation of task-specific model updates. Evaluations are performed in a multi-task learning setup on tasks related to computer vision (Multi-MNIST) and natural language understanding (tasks from GLUE and SuperGLUE).\n\nPros:\n1) The paper is easy to follow. \n\n2) Empirical evaluation is performed on vision and NLU domains.\n\nCons:\n1) I am not completely convinced with the proposed alpha-Variable Importance Learning algorithm. It is not very clear in the discussion how the alpha is different from task-specific weights. For example, in algorithm-1, if you replace deltas in line-10 with line-6, then there is no need to have separate alphas and task-specific weights, where line-9 can calculate the task-specific weights directly. \n\n2) In general, for a multi-task setup, I would expect to show the multi-task learning with multiple auxiliary tasks (that’s the main motivation of this paper as well). However, the choice of the experimental setup is convincing, especially for the vision domain there is only one auxiliary task. \n\n3) Both results in Table-1 and Table-2 suggest that the proposed algorithm is not superior over the baselines and previous approaches. The improvements are minor and sometimes lower, and I believe most of the results fall within the statistically insignificant range. \n\n\nOverall: \nI think the paper can be made stronger with more thorough discussion on the algorithm and its properties. Further, the experimental results suggest that the proposed algorithm performs more or less similar to previous methods. Hence, there is a lot of scope for further improvement and I would suggest rejecting this paper. I would also suggest the authors to perform more experiments and ablations. \n\n\nQuestions:\n1) How is the alpha different from task-specific weights. Please discuss more on this. In algorithm-1, if you replace deltas in line-10 with line-6, then there is no need to have separate alphas and task-specific weights?\n\n\n2) Please provide statistical significant scores for all the results. \n\n3)  What's the reason behind choosing a multi-MNIST dataset with only one auxiliary task? Aren’t there other datasets in a MTL setup with more auxiliary tasks? \n\n4) Table-2 results for the development set are based on the average of multiple runs, but for test you reported the ensemble, so why don’t you report ensemble for the development set as well?\n\n5) Can you also present some ablations/discussion on the learned importance of an auxiliary task (based on task-specific weight over the training trajectory) vs. any intuitive reason that makes sense of this importance of the axillary task for a given primary task? If there is not such correlation, then also it's good to discuss. \n\nOther comments: \n\n1) Please try to expand the introduction section. \n\n2) Please provide some more ablations. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes \alphaVIL, a method for weighting the task-specific losses in a multi-task setting in order to optimize the performance on a particular target task. The idea is to first collect gradient updates for the model based on all the separate tasks, and then re-weight those updates in order to optimize the loss on a held-out development set for the target task. In practice, this meta-optimization is performed with gradient descent. Experiments on multi-MNIST and several tasks that are part of GLUE and SuperGLUE show that \alphaVIL is close in performance to a baseline multitask method and discriminative importance weighting.

Strengths:
- The idea is intuitively appealing. Directly reweighting tasks as a meta-optimization step is straightforward and appears to not be proposed previously in the literature.
- The paper is clear in its presentation.

Weaknesses:
- The reviewers agree that the main weakness is that the experimental results do not show that \alphaVIL offers any substantial benefits over existing methods. On the multi-MNIST task, while \alphaVIL tends to have the highest mean performance, the difference is small (less than a standard deviation). On the GLUE/SuperGLUE tasks, it outperforms other methods on only 1 out of 10 experiments. There are also no confidence intervals/standard deviations provided to assess the significance of the results.
Decision: Reject