Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data
Authors: Jonathan Pilault
Amine El hattami
Christopher Pal
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-Task Learning
Adaptive Learning
Transfer Learning
Natural Language Processing
Hypernetwork

Abstract: Multi-Task Learning (MTL) networks have emerged as a promising method for transferring learned knowledge across different tasks. However, MTL must deal with challenges such as: overfitting to low resource tasks, catastrophic forgetting, and negative task transfer, or learning interference. Often, in Natural Language Processing (NLP), a separate model per task is needed to obtain the best performance. However, many fine-tuning approaches are both parameter inefficient, i.e., potentially involving one new model per task, and highly susceptible to losing knowledge acquired during pretraining. We propose a novel Transformer based Hypernetwork Adapter consisting of a new conditional attention mechanism as well as a set of task-conditioned modules that facilitate weight sharing. Through this construction, we achieve more efficient parameter sharing and mitigate forgetting by keeping half of the weights of a pretrained model fixed. We also use a new multi-task data sampling strategy to mitigate the negative effects of data imbalance across tasks. Using this approach, we are able to surpass single task fine-tuning methods while being parameter and data efficient (using around 66% of the data). Compared to other BERT Large methods on GLUE, our 8-task model surpasses other Adapter methods by 2.8% and our 24-task model outperforms by 0.7-1.0% models that use MTL and single task fine-tuning. We show that a larger variant of our single multi-task model approach performs competitively across 26 NLP tasks and yields state-of-the-art results on a number of test and development sets.

Published conference paper at ICLR 2021
CONDITIONALLY ADAPTIVE MULTI-TASK LEARNING: IMPROVING TRANSFER LEARNING IN NLP USING FEWER PARAMETERS & LESS DATA
Jonathan Pilault", Amine El hattami Christopher Pal1.2.3
Polytechniqqee Montreal & Mila, "Element AI, CCanada CIFAR Al Chair [oontthan. painnlli,lltl amine... eehattami, christopper pal @eo ca
ABSTRACT
Multi-Task Learning (MTL) networks have emerged promising method for transferring learned knowledge across different tasks. However, MTL must deal with challenges such as: overfitiing to low resource tasks, catastrophic forgetting, and negative task transfer, or learning interference. Often, Natural Language Processing (NLP), a separate model per task is needed to obtain the best perfor- mance. However, many fine-tuning approaches are both parameter inefficient, i.c., potentially involving one new model per task, and highly susceptible to losing knowledge acquired during pretraining. We propose novel Transformer based Adapter consisting of new conditional attention mechanism as well as set of task cask-conditioned modules that facilitate weight sharing. Through this construction, we achieve more efficient parameter sharing and mitigate forgetting by keeping half of the weights of a pretrained model fixed. We also pling strategy to mitigate the negative effects of data imbalance across tasks. Using this approach, we are able to surpass single task fine-tuning methods while being parameter and data efficient (using around 66% of the data for weight updates). Compared to other BERT Large methods on GLUE, our 8-task model surpasses other Adapter methods by 2.8% and our 24-task model outperforms by 0.7-1.0% models that use MTL. and single task fine-tuning. We show that a larger variant of our single multi-task model approach performs competitively across 26 NLP tasks and yields state-of of-the-ar art results on a number of test and development sets. Our code is publicly available at htthtps://giiie
multi-task data
com Coom/CAMT//CA-.. MTL
INTRODUCTION
The introduction of deep, contextualized Masked Language Models (MLM trained on massive amounts of unlabeled data has led to significant advances across many different Natural Language Processing (NLP) tasks (Peters al. 2018:) Liu et al. (019a). Much of these recent advances can be attributed to the now well-known BERT approach (Devlin et al 2018). Substantial improvements over previous tate-offthheart results on the GLUE benchmark (Wang et al.) (2018) have been obtained by multiple groups using BERT models with task specific tne-ttning. The EEER-variant- fine-tuning" formula has continued to improve over time with newer work constantly pushing the state-of -free-of-hh-aaae -rtt-art forward on the GLUE benchmark. The use of single neural architecture for multiple NLP tasks has shown promise long before the current wave of BERT inspired methods (Collobert & Weston 2008) and recent work has argued that autoregressive language models (ARLMs) trained on large-scale datasets such as the GPT family of models (Radford et al.) 2018). are in practice multi-task learners (Brown al. (2020). However, even with MLMs and ARLMs trained for multi-tasking, single task fine-tuning is usually also employed to achieve state-of-ther -ottt--rtttttttt performance on specific tasks of interest. Typically this fine-tuning process may entail: creating task-specific fine-tuned model (Devlin al. 2018). training specialized model components for task-specific predictions (Houlsby et al. 2019) or fine-tuning a single multi-task architecture (Liu et al. 2019b).
"Joint first-authors
For reader convenience, all acronyms this paper summarized in seetion A.I of the Appendix.

---

Published a conference paper at ICLR 2021
Logerd
Single-task tne-tuning overall pretrained model parameters may have other issues. Recent analy- ses of such MLM have shed light on the linguistic knowledge that is captured in the hidden states and
Gecoder
Decoder
Dacoder
ned
attention maps (Clark et al. 2019b, [Tenney et al.
Samp data
Embeodinngs
Conditoo N Nigitran
Prefrained 1-6
Pretrel red Thansommr layers 6-12 Ireiedd Adapter
2019a; Merchant et al. 2020). Particularly, BERT
Mulllask
has middle Transformer (Vaswami et al. (0177) lay- ers that are typically the most transferable to a
Uncertainty
Sampling
downstream task (Liu et al., 2211a). The model
proxies the steps the traditional NLP pipeline in a localizable way (Tenney et al., 2019a) - with basic syntactic information appearing earlier in the network. while high-level semantic information ap- pearing in higher-level layers. Since pretraining is usually done on large-scale datasets, it may be use- ful, for variety of downstream tasks, conserve that knowledge. However, single task fine-tuning causes catastrophic forgetting of the knowledge learned during MLM (Howard & Ruder 2018). To preserve knowledge, freezing part of pretrained network and using Adapters for new tasks have shown promising results (Houlsby et al. 2019).
Task Embedd ng Layer
Figure 1: CA-MTL base architecture with our sampling algorithm. Each task has its own decoder. The input embedding layer and the lower Transformer layers are frozen. The up- per Transformer layer and Conditional Alignment module are modulated with the task embedding.
uncertainty-
Inspired by the human ability to transfer learned knowledge from one task to another new task, Multi-Task Learning (MTL) in general sense (Caruanal 1997] Rajjurkar et al. [2016b] Ruder]] (2017) has been applied in many fields outside of NLP. Caruana ( (1993) showed that a model trained in a multi-task manner can take advantage of the inductive transfer hetween tasks, achieving a better generalization performance. MTL has the advantage of compuuaaiional/storagg efficiency (Zhang & Yang 2017). training models in multi-tas, setting is a balancing act; particularly with datasets that have different: (a) dataset sizes, (b) task difficulty levels, and (c) different types of loss functions. In practice, learning multiple tasks once is challenging since negative transfer (Wang et al. 2019a)). task interference (Wu et al., 2020; Yu et al.) 2020) and catastrophic forgetting (Serrà et al.] 2018)) can lead to worse data efficiency, training stability and generalization compared to single task fine-tuning,
Using Conditionally Adaptive Learning, we seek to improve pretraining knowledge retention and multi-task inductive knowledge transfer. Our contributions are the following:
A new task conditioned Transformer that adapts and modulates pretrained weights (Section 2.1). A novel way to prioritize tasks with an uncertainty based multi-task data sampling method that helps halance the sampling of tasks to avoid catastrophic forgetting (Section 2.2).
Our Conditionally Adaptive Multi-Tass Learning (CA-MTL) approach is illustrated in Figure 1 To the best of our knowledge, our work is the first to explore the use of a latent representation to modularize and adapt pretrained architectures. Further, we believe our work is also the first to examine uncertainty sampling for large-scale multi-task learning in NLP. We show the efficacy of CA-MTL by: (a) testing on 26 different tasks and (b) presenting state-of-the-ant results on a number of test sets as well as superior performance against both single-task and MTL baselines. Moreover, we further demonstrate that method has advantages over (c) other adapter networks, and (d) other MTL sampling methods. Finally, we provide ablations and separate analysis of the MT-Uncertainty Sampling technique in section 4.1 and of cach component of the adapter in 4.2
tasks
METHODOLOGY
This section organized according to the two main MTL problems that we will tackle: (1) How to modularize a pretrained network with latent task representations? (2) How to balance different tasks in MTL? We define each task as: J, 2 (p:(y,xx, 2), &,, C. (x:)), where z, is task i's learnable shallow embedding, &; is the task loss, and (x,) is the empirical distribution of the training data pair y.). for E {1,..., T) and 7' the number of supervised tasks. The MTL objective is:
min 6(x),01.....
Lalfeter.o, (x)
(1)

---

Published conference paper at ICLR 2021
where is the predictor function (includes encoder model and decoder heads), o(z) are learnable generated weights conditioned on z, and 0; are task-specific parameters for the output decoder heads. z is constructed using an embedding lookup table.
2.1 TASK CONDITIONED TRANSFORMER
Our task conditioned Transformer architecture is based on one simple concept. We either add conditional layers modulate existing pretrained weights using task representation by extending Feature Wise Linear Modulation (Perez et al.) 2018)) functions in several ways depending on the Transformer layer. We define our framework below.
Definition 1 (Conditional Weight Transformations). Given neural network weight matrix W, we compute transformations of the form ((Wz.) = w(z:)W + 8,(2z). where "it and B; are learned functions that transform the weights based on a learned vector embedding zi. for task i.
Definition 2 (Conditionally Adaptive Learning). In our setting, Conditionally Adaptive Learning is the process of learning a set of @s for the conditionally adaptive modules presented below along with a set of task embedding vectors z, for 'T tasks, using a multi-task loss (see equation!! D.
In the subsections that follow: We introduce new Transformer Attention Module using block- diagonal Conditional Attention that allows the original query-key based attention account for task-specific biases (section 2.1.1). We propose a new Conditional Alignment method that aligns the data of diverse tasks and that performs better than its unconditioned and higher capacity predecessor (section 2.1.2). We adapt layer normalization statistics specific tasks using new Conditional Layer Normalization module (section[2.133 We add a Conditional Bottleneck that facilitates weight sharing and task-specific information flow from lower layers (section 2.1.4). In our experiments we provide ablation study of these components (Table M examining performance in terms of GLUE scores.
2.1.1 CONDITIONAL. ATTENTION
MelAdd
Sortmax
MaiMMi
Given d, the input dimensions, the query Q the key K, and the value V as defined in Vaswani et al. (2117). we redefine the attention operation:
OK vd
Conditional Allention
Scaled Dot
Attention V,K.V.z) v,)) softmax M(z,)
V
Product
N
Query
Key
Value
(z,) (z)). A,(2;) = (z)) + (z(,,)
Figure 2:
Conditional Attention Module
where + is the direct sum operator (see section A.6 N
is the number of block matrices A, ((L/N)x(L/V)) along the diagonal of the attention matrix, is the input sequence, M (z,) = diag( ( A.y) is a block diagonal conditional matrix. Note that A, is constructed using L/N trainable and randomly initialized L/N dimensional vectors. Iearnable weight
While the original attention matrix depends on the hidden states h. (z.)
matrix that only depends on the task embedding z, E R.d. 74 3r : Rd + R /Na are Feature Wise Linear Modulation (Perez et al., (2018) functions. We also experimented with full-block Conditional Attention E IR LR_X Not only did it have 2 more parameters compared to the block-diagonal variant, but also performed sigmificantly worse on the GLUE development set (see FBA variant in Table [T] is possible that GLUE tasks derive a certain benefit from localized attention that is a consequence M (z.). With M (2,), each element in a sequence can only attend to other elements in its subsequence of length L/N. N. In our experiments we used N d/I L. The full Conditional Attention mechanism
used in our experiments is illustrated in Figure 2
2.1.2 CONDITIONAL ALIGNMENT
Wu et al. (000) showed that in MTL having T separate alignment modules Ri...., Ri increases BERT LARGE avg. scores on five GLUE tasks (CoLA, MRPC. QNLI, RTE, SST-2) by 2.35%. Inspired by this work, we found that adding task conditioned alignment layer between the input embedding

---

Published conference paper at ICLR 2021
layer and the first BERT Transformer layer improved multi-task model performance. However, instead of having T separate alignment matrices R; for each ' task, one alignment matrix ft is generated as a function of the task embedding 2i. As in Wu et al. (2020), we tested this module on the same five GLUE tasks and with BERT Enabling task conditioned weight sharing across covariance alignment modules allows to outperforms BERTLARGE by 3.61 61%. This is 1.26 % higher than having T separate alignment matrices. Inserting R into BERT, yields the following encoder function f:
Σ go. (())))
R(z,) = Rrs(z,) + (.(zz)
(2)
where X; E I is the layer input, go. is the decoder head function for task i with weights 8,, E the frozen BERT embedding layer, 1 the BERT Transformer layers and /2 the linear weight matrix of a single task conditioned alignment matrix. Yi, .r R + RR are Feature Wise Linear Modulation functions.
2.1.3 CONDITIONAL LAYER NORMALIZATION (CLN)
We extend the Conditional Batch Normalization idea from de Vries et al. (207)) to Layer Normaliza- tion (Ba et al. 2016). For task Tú, i E (1,..... .1.....7..
(a: 14 (z.)) + (z.).
7;(2.) = 7 (z)) + 3
(3)
where h; is the CLN output vector, a; are the preceding layer activations associated with task i, / and are the mean and the variance of the summed inputs within each layer as defined in Ba et al. (2016). Conditional Layer Normalization is initialized with BERT's Layer Normalization affine transformation weights and bias 7 and 3 from the original formulation: h = 1 o (a (4) * + 3'. During training, the weight and bias functions of (+) and (+(+)) are always trained, while the original Layer Normalization weight may be kept fixed. This module
added account for task specific rescaling of individual training cases. Layer Normalization normalizes the inputs across features. The conditioning introduced in equation 2.1.3 allows us to modulate the normalization's output based on a task's latent representation.
2.1.4 CONDITIONAL BOTTLENECK
m)
b)
Add CUN
We created task conditioned two layer feed-forward bot- tleneck layer (CFF up/down in Figure The conditional bottleneck layer follows the same transformation as in equation 2 The module in Figure 3h is added to the top most Transformer layers of CA-MCT CA-MTERASH and uses a CLN. For CA-MTLLARGE this module is the main building block of the skip connection added alongside all Transformer layers seen in Figure 3p. The connection at layer j takes in the matrix sum of the Transformer layer output at and the previous connection's output at j 1. The Con- flow
CIT dbwn
Add & CLN
Adsi & CUN
Formard
Adl
Mull Hood Attettion
MUSS Hoad
Figure 3: a) Conditional Bottleneck for CA-MTLEASE. b) Conditional Bottleneck for CA-MTLIARGE
ditional bottleneck allows lower layer information upwards depending on the task. Our intuition for intro- ducing this component is related to recent studies (Tenney) al. 2019a) that showed that the "most important layers for given task appear at specific positions". As with the other modules described so far, each task adaptation is created from the weights of a single shared adapter that is modulated by the task embedding.
2.2 MULTI-TASK UNCERTAINTY SAMPLING
MT- UTT-nnertainnyy Sampling is task selection strategy that is inspired by Active Learning techniques. Our algorithm | outlined in the Appendix, Section A.2 Similar Active Learning, our algorithm first evaluates the model uncertainty. MT-Unecrtainty Sampling uses Shannon Entropy, an uncertainty measure, to choose training examples by first doing forward pass through the model with b T input samples. For output classification prediction with C, possible classes and probabilities

---

Published a conference paper at ICLR 2021
(p........Pa the Shannon Entropy H;, for task J, and i E (l....,T) T), our uncertainty measure ((x are given by:
C H = Hilfocnr = E
Ux,+) = Hil (x))
Pe log Po:
(4)
H H'
A = max (E
H. max
H
H = C
log
(5)
C
XEX.
where FT is the average Shannon Entropy across samples of task t, Hy, the Shannon entropy of choosing classes with uniform distribution and H, the maximum of each task's average entropy over b samples. Hi is normalizing factor that accounts for differing number of prediction classes (without the normalizing factor Hi, tasks with a binary classification C; = 1 were rarely chosen). Further, to limit high entropy outliers and to favor tasks with highest uncertainty. we normalize with H. The measure in eq. alllows Algorithm to choose b samples from b x T candidates train the model.
3 RELATED WORK
Multi-Tasking in NLP. To take advantage of the potential positive transfer of knowledge from one task to another, several works have proposed carefully choosing which tasks to train as an intermediate step in NLP before single task fine-tuning (Bingel & Søgaard [017]] Kerinec et al.) 2018: Wang et al. 2019a Standley et al.] 2019; Pruksachatkun et al. [020; Phang et 2018)) The intermediate tasks are not required to perform well and are not typically evaluated jointly. In this work, all tasks are trained jointly and all tasks used evaluated from single model. In Natural Language Understanding (NLU), it is still the case that to get the best task performance one often needs a separate model per task (Clark et al. 2019c McCann et al. 2018). At scale, Multilingual NMT systems (Aharomi et al. 2019) have also found that MTL model performance degrades as the number of tasks increases. We notice a similar trend in NLU with our bascline MTL model. Recently, approaches in MTL have tackled the problem by designing task specific decoders on top of shared model (Liu et al. 2019b) or distilling multiple single-task models into one (Clark et al.] ( Nonetheless, such MTL approaches still involves single task fine-tuning. In this paper, we show that it is possible to achieve high performance in NLU without single task fine-tuning.
Adapters. Adapters are trainable modules that are attached in specific locations of pretrained network. They provide another promising avenue limit the number of parameters needed when confronted with a large number of tasks. This approach useful with pretrained ML.M models that have rich linguistic information (Tenney et al. (209bb Clark et al. 2019h Liu et all 2019a] TTnney al. (019a)) Recently, Houlsby et al. (2019) added an adapter fine-tuning the layer norms and adding feed forward bottlenecks in every Transformer layer. However, such methods adapt each task individually during the fine-tuning process, Unlike prior work, our method harnesses the vectorized representations of tasks modularize a single pretrained model across all tasks. (Stickland et al. (2019) and Tay et al. (2020) also mix both MTI. and adapters with BERT and T5 encoder-decoder (Raffel et al. (2099) respectively by creating local task modules that are controlled by global task agnostic module. The main drawback is that new set of non-shared parameters must be added when a new task is introduced. CA-MTL. shares all parameters and is able to re-modulate existing weights with a new task embedding vector.
pretrained BERT model by
Active Learning, Task Selection and Sampling. Our sampling technique is similar to the ones found in several active learning algorithms (Chen al., 2006) that based Shannon entropy estimations. Reichart et (2008) and Ikhwantri et al. (2018) examined Multi- Task Active Learning (MTAL). a technique that chooses one informative sample for T' different learners (or models) for each T' tasks. Instead we choose 7' tasks samples for one model. Moreover, the algorithm weights each sample by the corresponding task score, and the Shannon entropy is normalized to account for various losses (see equation 5). Also, our algorithm is used in a large scale MTL setup (*> 2 tasks). Recently, Glover & Hokamp (0019 explored task selection in MTL using learning policies based on counterfactual estimations (Charles et al. (013)). However, such method considers only fixed stochastic parameterized policies while our method adapts its selection criterion based on model uncertainty throughout the training process.

---

Published conference paper at ICLR 2021
4 EXPERIMENTS AND RESULTS
We show that our adapter of section 2 achieve parameter efficient transfer for 26 NLP tasks. Our implementation of CA-MTL is based on HuggingFace (Wolf et al. (2019). Hyperparameters and our experimental set-up outlined in A.3 To preserve the weights of the pretrained model, CA-MTL's bottom half Transformer layers are frozen in all experiments (except in section 4.4). We also tested different layer freezing configurations and found that freezing half the layers worked best on average (see Section A.8)
4.1 MULTI-TASK UNCERTAINTY SAMPLING
0.62
Our MT- MT-Uncertainty sampling strategy, from section
0.53
2.2 compared to other task selection schemes; a) Counterfactual b) Task size c) Random. We used a BERT (no adapters) on 200k iterations and with the same hyperparameters as in Glover & Hokamp (2019). For more information on Counterfactual task selection, we invite the reader to consult the full expla- nation in Gloover&&& & Hokamp (2019). For tasks and the dataset D) for tasks i E (I..... ."), we rewrite the definitions of Random Trand and Task size Ttaak sampling:
8.74
0.73
Canntord
Rades
0.88
Figure 4: MT--nccrraintyy vs. other task sam- pling strategies: median dev set scores on 8 GLUE tasks and using BERT haSe, Data for the Counter- factual and Task Size policy T[aakk (eq, 6 is from Glover & Hokamp 2019).
T Σ
Trand 1/T, T[task] = |Da]
D
Figure we see from the results that MT- Uncertainty converges faster by reaching the 80% average GLUE score line before other task sampling methods. Further, MT-Uncertainty maximum score on 200k iterations is at 82.2, which is 1.7% higher than Counterfactual sam- pling. The datasets the GLUE benchmark offers wide range of dataset sizes. This is useful to test how MT-Uncertainty manages jointly trained low resource task (CoLA) and high resource task (MNLI). Figure 5 explains how catastrophic forgetting is curtailed by sam- pling tasks before performance drops. With Wrand, all of CoI.A's tasks are sampled by it- eration 500, at which point the larger MNLI dataset overtakes the learning process and CoLA's dev set performance starts to diminish. On the other hand, with MT-Uncertainty sampling, CoLA is sampled whenever Shannon entropy is higher than MNLI's. s. The model first assesses uncertain samples using Shannon Entropy then decides what data necessary to train on. This process allows lower resource tasks to keep performance steady. We provide evidence in Figure (f A.2 that MT-Uncertainty is able to manage task difficulty - by choosing the most difficult tasks first.
onimpy
C.LA des aoee
Cul.A tain catingyy
0.5
01.5
0.6
0.1
0.4
0.0
0.0
500
5000
10000
310
5000
10000
Trin incratiss
Train dxcratioe
Randam
MT-Uncemainiy
Figure 5: CoLA/MNLI Dev scores and Entropy for Trand (left) and MT-Uncertainty (right).
Table 1: Model ablation study" on the GLUE dev
4.2 ABLATION AND MODULE ANALYSIS
set. All models have the bottom half layers frozen.
In Table we present the results of an ablation study determine which elements of CA-MTL BASE had the largest positive gain on average GLUE scores. Starting from a MTL BERT baseline trained us- ing random task sampling (Trand). Apart for the Conditional Adapter, each module as well as MT- Uncertainty lift overall performance and reduce vari- ance across tasks, Please note that also included
Model changes
Avg
Task o
% data
GLUE GLUE
used
BERTBASE MTL (Trena)
80.61
14.41
100
Conditional Attention
82.41
10.67
100
Conditional Adapter
82.90
11.27
100
CA and CLN
83.12
10.91
100
MT- MTT-nncertiyyy (CA- CAA-MLLaERT-AAEE)
84.03
10.02
66.3
CA=Conditional Aligument, CCN=Conditoonal Layer Nomml ization, Task a=scores standard devistion acrovy tirks.

---

Published a conference paper at ICLR 2021
accuracy/Fi scores for QQP. MRPC and Pearson/ Spearman correlation for STS-B to calculate score standard deviation Task a. Intuitively, when negative task transfer occurs between two tasks, either (1) task interference is bidirectional and scores are both impacted, or (2) interference is unidirec- tional and only one score is impacted. We calculate Task to characterize changes in the dynamic range of performance across multiple tasks. We do this to asses the degree to which performance improvements distributed across all tasks only subsets of tasks. As can see from Table [ Conditional Attention, Conditional Alignment, Conditional Layer Normalization, MT-Uncertainty play roles in reducing Task o and increasing performance across tasks. This provides partial evidence of CA-MTL's ability to mitigating negative task transfer.
We show that Conditional Alignment can learn to capture covariate distribution differences with task embeddings co-learned from other adapter compo- nents of CA-MTL. In Figure 6 we arrive at similar conclusions as Wu et al. (2020) who proved that neg- ative task transfer is reduced when task covariances are aligned. The authors provided "covariance simi- larity score" to gauge covariance alignment. For task i and j with my and my data samples respectively, and given d dimensional inputs to the first Trans-
0153
pum)
55 score improvament
DallA (ovum)
0.00
VILLI ICA MTLI
pm))
VEPC (oumm
-025
CDP (CAMIT)))
.008
STS-B (CA-MIT)))
0.1
0.2
0.3
0.4
Avg. Covariance Similarity Score
former layer X; E Rm xd and X, E IRm, xd, we rewrite the steps to calculate the covariance similarity score between task i and j: (a) Take the covariance matrix X; X,, (b) Find its best rank-r's approxima- tion U,,, Di.,, U.T,, where is chosen to contain 99% of the singular values. (c) Apply steps (a), (b) to Xj. and compute the covariance similarity score CovSima.y:
Figure 6: Task performance vs. avg. covariance similarity scores (eq. 7 for MTL and CA
ConSima. = II(Un,, D. Usrr, D 1r
ConSim; T-1
Σ SAi
ConSimas
(7)
[1Ur D. 1U D
Since we are training models with T tasks, we take the average covariance similarity score ConSimg between task i and all other tasks. We measure ConSii; using equation / between 9 single-task models trained score on the MTL trained BERT BASE baseline, e.g., CoLA (MTL), or CAAMTLLERE=aSSE: model, e.g.. MNLI (CA-MTL), Our score improvement measure is the % difference between a single task model and MTL or CA-MTL on the particular task. We find that covariance similarity increases for 9 tasks and that performance increases for 7 out 9 tasks. These measurements confirm that the Conditional Alignment is able to align task covariance, thereby helping alleviate task interference.
individual GLUE tasks. For each task in Figure 6 we measure the similarity
4.3 JOINTLY TRAINING ON 8 TASKS: GLUE
In Table 2 we evaluate the performance of CA-MTL against single task fine-tuned models, MTL well as the other BERT-based adapters on GLUE, As Houlshy et al. (201)), MNLI and M are treated as separate tasks. Our results indicate that CA-MTI. outperforms both the BASE adapter,
Table 2: Adapters with layer freezing vs. ST/T GLUE test Fl scores are reported for QQP/MRPC.
Spearman's 's correlation for STS-B, accuracy on the matched/mismatch sets for MNLI, Matthew's correlation for Co and accuracy for other tasks. * Individual scores not available. ST=Single Task, MTI.=Multtask, g.c.=
greater or equal Results from: Devlin
2018
Sticklannd et al. 2019). (Houlsby et al. (2019)
Total paroms
Trained
GLUE
Method
Typo
parmmsttakk
g.e. ST
CoLA
MNLI
MRFC
QNLI
QQP
RTB S81-2
STS-8
Avg.
Base Models
Test
Results
BERTHAMT BPR T
ST
9.0%
100%
52.1
84,6783,4
88.9
90.5
71.2
66.4
93.5
85.8
79.6
MTT.
1.0x
11.1%
2
51.2
84.083.4
86.7
89.3
70.8
26.6
93.4
83.6
79.9
PALs+Ameea Samp. 2
MTL
1.1xx
12.5%
51.2
84.3/53.5
88.7
21.5
76.0
80.4
CA-Millyaraast tours)
MIL
1.12x
5.6
5
53.1
85.9058
83.6
90.5
69.2
76.4
93.2
85.3
80.9
Large Models
Test Server Results
BERTLAKEE
ST
9.0%
100%
60.5
86.7785.9
89.3
12.7
72.1
70.1
94.9
86.5
82,1
Adapters 256
ST
1.3x
3.6%
3
39.5
84.9/85.1
89.5
90.7
71.8
71.5
94.0
86.9
80.0
CaLAmTLan (ours)
MIL
111x
5.6%
3
59.5
85.9785.4
89.3
92.6
71.4
79.0
94.7
87.7
82.8

---

Published conference paper at ICLR 2021
PAnS+Anneal Sampling (Stickland et al., 2019), and the LARGE adapter, Adapters 256 (Houlsby et al, 2019). Against single task (ST) models, CA-MTL 1.3% higher than BART. with 5 out 9 tasks equal or greater performance, and 0.7% higher than BERT LARGe, with 3 out 9 tasks equal or LARGs
greater performance. ST models, however, need models or close to 9x more parameters for all 9 tasks. We noted that CA-MTLBEKKHARGGEE 's average score is driven by strong RTE scores. While RTE benefits from MTL, this behavior may also be side effect of layer freezing. In [T we see that CA-MTL has gains over ST on more and more tasks as we gradually unfreeze layers.
4.4 TRANSFER TO NEW TASKS
Table 3: Domain adantation results on dov, sets for BASE models. Liu et al.
2019b),
"Jiang ct al.
2020)
In Table we examine the ability of our method to quickly adapt to new tasks. We performed domain adaptation on SciTail (Koot et al.] 2018) and SNLI (Bowman et al. (015) datasets, using CA-MTLBASt model trained on GLUE and a new linear decoder head. We
SciTul
SNLI
data used
0.15
1%
10%
100%
0.1
1%
10%
1010
BERTRASE MT DNN
51.2
82.2
90.5
94.3
52.5
78.1
86.7
91.0
81.9
88.3
91.1
95.7
81.9
88.3
91.1
95.7
MT. DNNSAAR
82.3
88.6
91.3
96.1
82.7
86.0
88.7
91.6
CA-M
83.2
88.7
91.4
95.6
82.8
86.2
88.0
91.5
tested several pretrained and randomly initialized task embeddings in a zero-shot setting. The com-
plete set of experiments with all task embeddings can be found in the Appendix, Section [A.4] We then selected the best task embedding for our results T STS-B and MRPC MTL-trained task embeddings performed best on Sciiail and SNLI respectively. CA-MTT- DASE has faster adapta- tion than MT-DNNsMART (Jiang al. (2020) evidenced by higher performances in oowrresouece regimes (0.1% %% and 1% of the data). When trained on the complete dataset, CA-MTLBERR-BASE is on par with MT-DNN SMART. Unlike MT-DNN however, we do not add context from a semantic similarity model MT-DNN is built off HNN (He et al. 2019). Nonetheless, with a larger model, CA-MTL surpasses MT-DNN SMART on the full SNLI and SciTail datasets Table 6
4.5 JOINTLY TRAINING ON 24 TASKS: GLUE/SUPER- GLUE, MRQA AND WNUT2017
Effects of Scaling Task Count. In Figure we
GLUE (9 tasks)
MRRG (15 tasks)
+Super-QLUE (23 tasks)
continue to test if CA-MTL mitigates lask in-
94.0
terference by measuring GLUE average scores
when progressively adding 9 GLUE tasks, 8
78.5
80.0
80.8
Super-GLUE tasks (Wang et al.
2019b).
6
76.1
75.7
MRQA tasks (Fisch et al.j 2019).
Tasks are
71.8
described in Appendix section A.9. The results
show that adding 23 tasks drops the performance of our baseline MTL BERTBASE (Frand). MTL. BERT increases by 4.3% when adding MRQA but, with 23 tasks, the model performance drops by 1.8%. The opposite is true when CA-MTL modules are integrated into the model. CA-MTL continues to show gains with large number of tasks and surpasses the haseline MTI. model by close to 4% when trained on 23 tasks.
66
BERT basaline
CA-MTL Samp.
Un-Uneert.
Figure 7: Effects of adding more datasets avg GLUE scores. Experiments conducted on 3 epochs. When 23 tasks trained jointly, performance of CA-MTLaERTaASE continues improve.
24-task CA-MTL. We jointly trained large MTL baselines and CA-MTL models on GLUE/Super GLUE/MRQA and Named En- tity Recognition (NER) WNUT2017 (Derczyn- ski et al. (017). Since some dev. set scores are not provided and since RoBERTa results were reported with median score over 5 random seeds, we our own single seed ST/MTL. baselines (marked "Relmp") for a fair compar- ison. The dev. set numbers reported in Liu al. (2019c) are displayed with our baselines in Table 9 Results are presented in Table [
Table 4: 24-task CA-MTI. vs. ST and vs. 24-task MTI. with frozen layers on GLUE, SuperGLUE, MRQA and NER development sets. ST=Single Task, -ull:Multiask, g.e.= greater or equal to. Details in section A.5
Tisk Grouping
tusks
Total
Model
GLUR SuperGi MRQA NER
Avg
eg..T
Parana
BERT-LARGE models
STautre
84.5
68.9
79.7
54.1
76.8
24x
Milaa
83.2
72.1
77.8
42.2
76.4
9/24
1x
MTT
74.1
79.5
49.0
79.1
17/24
1.12x
RobERTo / ARGE models
STadte
88.2
76.5
83.6
57.8
81.9
24x
MTT CA MIL
86.0
78.6
80.7
49.3
80.7
7/24
1x
80.0
82.4
83.1
15/24
We notice in Table 4 that even for large models, CA-MTL provides large gains in performance on average over both ST and MTL models. For the BERT based models, CA-MTI. provides 2.3% gain

---

Published a conference paper at ICLR 2021
over ST and higher scores on 17 out 24 tasks. For RoBERTa based models, CA-MTL provides 1.2% gain over ST and higher scores on 15 out 24 tasks. We remind the reader that this is achieved with single model. Even when trained with 16 other tasks, it is interesting to note that the MTL. baseline perform better than the ST baseline on Super GLUE where most tasks have a small number of samples. Also, we used NER to test if we could still outperform the ST baseline on a token-level task, significantly different from other tasks. Unfortunately, while CA-MTL performs significantly better than the MTL baseline model, CA-MTL had not yet overfit on this particular task and could have closed the gap with the ST baselines with more training cycles.
Comparisons with other methods. In Table [ CA-MTLBERT is com- pared to other Large BERT based methods that either use MTL + ST, such as MT-DNN (Liu et al.] 2019b).
Table 5: Our 24-task CA-MTL vs. other large models on GLUE FI is reported for QQPMMPCC Spearman's 's corr. for STS-B, Matthew's s corr. for CoLA and accuracy for other tasks. *Split not available. **uess intermediate task fine-tuning + ST.
intermediate tasks + ST, such as
Model
CoLA
MNLI
MRPC QNLI QQP RTE SST-2
GLUE tasks
STS-B B
Avg
STILTS (Phang et al. 2018) or MTL
BERF-LARGE bused models on Dev ser.
MI-DNN
63,5
$7.1886.2
91.0
92.9
89.2
83.4
94.3
90.6
85.6
model distillation
ST, such
STILIS
62.1
86.1
92.3
83.4
93.2
90.8
85.9
BAM! (Clark et al. c019c). Our method scores higher than MT-DNN on 5 of 9 tasks and by 1.0 % on avg. Against STILTS, CA-MTI. realizes a 0.7 % avg. score gain, surpassing scores on 6 of tasks. We also show
RAM!
61.8
87.0
92.5
82.8
93.6
89.7
MTL
63.8
$6.3/86.0
92.9
84.5
90.3
RoBERTa LARGE based models on Test sct
RoBERTA with
67.8
91.0/90.8
91.6
95.4
74.0
87.9
97.5
92.5
87.3
Ensemble
CA-MTL
62.2
89.0/88.4
92.0
89.8
85.7
that CA-MTL-23ERT is within only 1.6 % a RoBERTa ensemble of models per task and that uses intermediate tasks. Using our 24-task CA-MTI. large CoBERTa-baase model, we report NER F1 scores on the WNUT2017 test set in Ta.le We compare our result with RoBERTa and XLM-RLARGE (Nguyen al. 2020) the current state-of-the-an (SOTA). Our model outperforms XLM-RI.ARGE by 1.6%, reaching new state-of-the-art. Using domain adaptation as described in Section 4.4 report results the SciTail test set in Table and SNLI test set in Table For SeiTail, our model matches the current SOTA ALUM (Liu et al. 2020), a RoBERTa large based
SOTA@ALLUM
model that additionally uses the SMART (Jiang et al. 2020) fine-tuning method. For SNL.I, our model outperforms SemBert, the current SOTA
Table 6: CA-MTL test performance vs. SOTA.
(b) SciTail
% Acc
(c) SNLI
% Acc
(a) T207
F1
MT-DNN
94.1
MT-DNN
91.6
RoBERTa, ARGE
56.9
ALUMR
96.3
MT-DNNSMART
91.7
XLM-R
57.1
ALUMR
96.8
SemBERT
91.9
CA-MTI Routatn (ours)
58.0
CA-MTL LROBERIZ (ours)
96.8
CA-MTI ROBERD (ours)
92.1
5 CONCLUSION
We believe that our experiments here have helped demonstrate the potential of task conditioned adaptive learning within single model that performs multiple tasks. In large-scale 24-task NLP experiment, CA-MTL outperforms fully tuned single task models by 2.3% for BERT Large and by I 1.2% for RoBERTa Large using 1.12 times the number of parameters, while single task fine-tuning approach requires 24 separately tuned single task models or 24 times the number of parameters. When a BERT vanilla MTL model sees its performance drop as the number of tasks increases, CA-MTL scores continue to climb. Performance gains are not driven by a single task as it is often the case in MTL. Each CA-MTL module that adapts a Transformer model is able to reduce performance variances between tasks, increasing average scores and aligning task covariances. This evidence shows that CA- CA- is able to mitigate task interference and promote more efficient parameter sharing. We showed that MT- MT-Uncertainty is able to avoid degrading performances of low resource tasks. Tasks are sampled whenever the model sees entropy increase, helping avoid catastrophic forgetting. Overall, CA-MTL offers a promising avenue to dynamically adapt and modularize knowledge embedded in large monolithic pretrained models. Extending such ideas will be an objective for future work.
on 09/27/2020
http:://tf..suaafoottddd/////ssn 09/27/2120

---

Published conference paper at ICLR 2021
ACKNOWLEDGMENTS
This research supported by the Canada CIFAR AT Chairs Program, NSERC and PROMPT. Exper- iments in this article were conducted with Compute Canada and MILL computational infrastructure and we thank them for their support. We would like to thank Colin Raffel, Sandeep Subramanian. and Nicolas Gontier for their useful feedback and the anonymous reviewers for helpful comments, discussions and suggestions.
REFERENCES
Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers),
pp. 3874-3884, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
doi: 10. 1/.5///1//19111/33 1388. URL. https: /www ac web. org. anttool ogy/ /N1 9- 10099-1330 1388
Lei Jimmy Ba. Jamie Ryan Kiros, and Geoffrey E. Hinton, Layer normalization. CoRR, abs/1607.064500, 2016. URL
06450
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 18-48, 2009.
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 164-169, Valencia. Spain. April 2017. Association for Computational Linguistics. URL https: www. wwwweb. org
antholocy/88772022
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. 2015.
Tom B Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. Language models are few-shot learners. arXiv, pp. arXiv-2005, 2020.
Rich Caruana. Multitask learning. Mach. Learn 28(1):41-75 July 1997. ISSN 0885-6125. doi: 10073//6007777900777 /ttp: 1ttp://0000..1111111777773700000
Richard Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference Machine Learning, pp. 41-48. Morgan Kaufmann, 1993.
Daniel Cer, Mona Diab, Encko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evallaation (SemEval-2017), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18655/v1/$17-2001 URL www www.wwww....................ww.w orq/ aathologg/S117-2000
Denis Charles, Max Chickering, and Patrice Simard. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14:3207-3260. November 2013.
Jinying Chen, Andrew Schein, Lyle Ungar, and Martha Palmer. An empirical study of the behavior of active learning for word sense disambiguation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pp. 127, New York City, USA.
June 2006. Association for Computational Linguistics. URL /ttp:://wwwww..o.. www. www.bb.wwwbboo. anthology/N06-111
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. CoRR, abs/1711.0225: 2017. URL. http::////// //ap: httt::://axxxi..or//ab//////////
wttp/bs////////// 02257
10

---

Published conference paper at ICLR 2021
Christopher Clark, Kenton Lee, Ming-Woi Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2922-29222222 2936, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10.18653/ vI/N19-1300 Utthttps:: petetps://wwitr ac w web. wtg.ee/ooog//aoooogo//////o// /N19 1300
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NIP,, pp. 276-286, Florence, Italy, August 2019b. Association for Computational Linguistics. doi: 10.1 10.18653/v 18853// 16633/1//19-4828. URL https 7/ www.acllweb..og//ntt.ooo.ooo
4828
Kevin Clark, Minh-Thang Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoe V. Le. Bam! bom-again multi-task networks for natural language understanding. CoRR, abs/1997,,4889, 2019c. BRLphtt://aaae:
org/ abs///ab///// 1907. 04829
Edward Collins, Nikolai Rozanov, and Bingbing Zhang. Evolutionary data measures: Understanding the difficulty of text classification tasks. In Proceedings of the 22nd Conference on Compulational
Natural Language Learning. pp. 380-391, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/vl/KI8. 1037. Utt/ttts:: www wwwweb.
Bathology/K18 1037
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, pp. 160-167, 2008. URL https://doi
org/10. 11450111115/11111555 1390177
de Marneffe. Mandy Simons, and Judith Tonhauser. The commitmentann:: Investi- gating projection naturally occurring discourse. Proceedings Sinn und Bedeutung, 23(2):107- httttt//t:////// un uni konstanz
124, Jul. 2019. doi: 18. /0L111/4//u//211.223222222
de/ be/sub/inn http/nndex..oo httpe//ppppppbbbaattccc/////////
Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.). Advances in Neural Information Processing Systems 30, pp. 6594 6604. Curran Associates, Inc., 2017. URL http::/ .nis cc/paper/ www.ualiinnneaaaaa..aa wwweeaalllllaaalaaaaaaaaaaaaaaaa wtwppy/wwisaal-popssssse..
by language...a......
Leon Derezynski. Eric Nichols Marieke van Erp, and Nut Limsopatham. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 140-147, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/11W11744411
URLhttps: www. aelweb.orgg
aathology//i W1 717-4418
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, 180//880,04405, 2018. URI.
http:///rxive. org/ abs/1810 04805
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). 2005. URI.
https:: www. aclweb.
2atthologyy000-5022
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs/ 704.05177,
2017. URL http: /ttp/v//aavvvv/vaa///a org/ abs / 1701.00000000 05179
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering. pp. 1 1-13, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1e..1855/1w/////... 1/D19-58011111111111111111111111 URL https:
7/ www. acl www.www...wo........... ott.ee.rg//aatttollg////11//111

---

Published conference paper at ICLR 2021
John Glover and Chris Hokamp, Task selection policies for multitask learning. CoRR, 2019. URL btttp::/arxivv.org// wttp://///////oo///////// 06214
Andrew Gordon, Zornitsa Kozareva, and Mclissa Roemmele. SemEval-2012 task 7: Choice of plausi- ble alternatives: An evaluation of commonsense causal reasoning. In "SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main confer- ence and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Se- mantic Evaluation (SemEval 2012). pp. 394 398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL http:::::: www. aclweb. org/ http:/aathoooooyaaaaa
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioriti- zation for multitask learning. In Proceedings of the Europeun Conference Computer Vixion (ECCV), September 2018.
Pengcheng He, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. hybrid neural network model for commonsense reasoning. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pp. 13-21, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-6o0
URL www aclweb org/
anthology D1 9-9--65 6002
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parfmeter-efffiiient transfer learning for NLP. CoRR, abt//190.000551 2019. URI.htt:: http:/ppp//aaiw.oo//////////e///
902. 00751
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 239, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1031. URL httpps: www www.wwwwww.w.......ww..w..www..w wwwpeeebbrgg/nnie org/anthology P18-1031
Fariz Ikhwantri, Samuel Louvan, Kemal Kurniawan, Bagas Abisena, Valdi Rachman, Alfan Farizki Wicaksono, and Rahmad Mahendra. Multi-task active learning for neural semantic role labeling on low resource conversational corpus. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, pp. 43-50, 2018.
Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regular- ized optimization. In Procedingss of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2177-2190, Online, July 2020. Association for Computational Linguistics. doi: 00/6655/1//020..oo acl- main.197. URL https: www. aclweb. orq http///ataaooo//////22222 acl ww..innn.1..11 197]
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1601 1611. Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10.18653/ v1/717-1147. URL http:::p//........... www. acl crg/ attp//aatthoo.ooo//11111111
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907. 1997.1029,, 2019. URL
www./org/ss....... /197 10529
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty weigh losses for scene geometry and semantics. CoRR, 1hs//7005.07 07115, 15, 2017. URR.httpp:/arxxv.oor
0bs/1705.077
Emma Kerinec, Chloé Braud, and Anders Søgaard, When does deep multi-task learning work for loosely related document classification tasks? In Proceedings the 2018 EMNLP Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP, pp. 1-8, Brussels, Belgium. November 2018 Association for Computational Linguistics. doi: 10.18653/v1/WIs
URL
https: / www. acl www..org//nthoooog///oo/o//////e

---

Published conference paper at ICLR 2021
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceed- ings of the 2018 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long Papers), pp. 252-25222222 252-262,, New Orleans, Louisiana, June 2018, Association for Computational Linguistics. doi: 10. 10.18653/11/N18- 1023. URLlhttps: / www. ac web. org http///ohhhooooooo// 1023
Tushar Khot, A. Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In AAAI, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,, 2015.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alherti, Danielle Epstein, Mllia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov, Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Hector J. Levesque. The winograd schema challenge. In AAAI Spring Symposium: Logical Formal-
izations of Commonsense Reasoning, AAAL, 2011. URL hhtt:://dbbpp wni-pp.uniiteer. /e
ab/conf/ /ttaa/asaaaaaaa///aaaaaa hatp::/aaaaaaaa0111-ammaaaaeeaee
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Marthew E. Peters, and Noah A. Smith. Linguistic
knowledge and transferability of contextual representations. CoRR, abs/1903.08855, 2019a. URL
http: //tp/vaar/ivvvvvv wwtp:o/gg////ooooo /wtps////13... 100.500..5555
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. CoRR, abs/1901.115000 2019b. /RLphtt////xxxi..o
abs/1901 11504
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language models, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1669,, 2019c. http:::/aaxw...rgg////p
907. 11692
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning question answering. arXiv preprint I 086600830, 2018.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to bert embeddings during tine-tuning' arXiv preprint arXiv:2004. /4448, 2020.
Dat Quoe Nguyen, Thanh Vu, and Anh Tuan Nguyen. Bertweet: A pre-trained language model for english tweets. arXiv preprint arXiv:2005. 10200, 2020.
Hao Peng, Roy Schwartz, Dianqi Li, and Noah A. Smith. A mixture of h heads is better than h heads. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
Online, July 2020. Association for Computational Linguistics. doi: 10. 1e..8865//1/222...s acl-main.587. URL httpst:// //wp:////wwwwww////// awwwwb.ww...... org/ www.oorgggnnooo///ooooooooo////o acl main. 587
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville, Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. CoRR, abs/ 08949, 2018. URL http: 77 rx v. wwtp://///ab///oeeooese 08949
13

---

Published conference paper at ICLR 2021
Jason Phang. Thibault Févry, and Samuel Bowman. Sentence encoders on STILTss: Supplementary training on intermediate labeled-data tasks. CoRR, abs/ 2018. URL http:://aaa/ii lorg/bs 1811. 01 01088
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 67-81, Brussels, Belgium. October-November 2018. Association for Computational Linguistics. doi: 18653/v3 I/DI8 1007. URLhttps www. http://wwwwwwlll.cc... org/
BrrcologyyDDB-I00
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R Bowman. Intermediate-tane transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint arXi::2005.0060 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Πlya Sutskever. Improving language under- standing by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods Natural Language Processing, pp.
Austin, Texas, November 2016a. Association for Computational Linguistics. doi: 10.18653/v1/D16- 1264, URL [tL[attps:/ www aclweb. org/
anthology/D1e 1264
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016b. Association for Computational Linguistics. doi: 10. 18653v/ 1/D16-126/
Roi Reichart, Katrin Tomanek, Udo Hahn, and Ari Rappoport. Multi-task active learning for linguistic annotations. In Proceddings of ACI-088 HLT, pp. 861-869, 2008
Sebastian Ruder. An overview of multi-task learning in deep neural networks. ArXiv, abs/ a 1706.05098. 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-ohjective optimization. CoRR, abs/1810.04450 2018. URL http:/ttpp//aaxx...rro/////
04650
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, pp. 4555-4564, 2018. URL httpp:/7
proceedings www.m...pree.. press/ v80/ serral materral8a... html
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp- 1642, Seattle, Washington, USA, October 2013, Association for Computational Linguistics. URL
Christopher Potts. Recursive deep models for semantic compositionality
org/ http////totooooooo11111100
Trevor Standley, Amir Roshan Zamir. Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? CoRR, abs/1905.07553.
2019. URL http: //tpvv/aavivvvv/aa org/ abs 1905.07553 07553
Asa Cooper Stickland, lain Murray, someone, and someone. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. volume 97 of Proceedings of Machine Learning Research, pp. 5995, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http::/proceet ngs wtt.lir.prsss//////
and 9a. html

---

Published conference paper at ICLR 2021
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da- Da-Cheng Juan. Hypergrid: Efficient multi-task transformers with grid-wise decomposable hyper projections. arXiv preprint arXiv:2007. 7,0891, 2020.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. CoRR. abs/1905.05950 2019a. wttlhttpp//axxlv..rr///////
905. 05950
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. CoRR, abs/1905.06316, 2019b. URLp://///arx:
org/ abs///////// / 1905. 06316
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop Representation Learning for NLP. pp. 191-200, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10:16633/1/W11--22233, URL ht tps 77 wwwp://wwwww...e////totooooooooo
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, 2017. URL.
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR,
03762
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE; A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP:: Analyyyin and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 8153166551l/// 1/518//4/ URL
ac web. wwg antto www.anowooooyooo.ooooooo ogy/w18 5446
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. Can you tell me how get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019a.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general purpose language understanding systems. CoRR, abs/1905.00S. 000537, 2019b. URL. general-purpose /
1 1905_ 00537
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. CoRR, abs/1805. 12471, 2018. URt.ttt::///////// http:://axxvvvovo org/ abs 1100111011.00 12471
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pp- 1112-11233 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653//1/N18-111 URL htLhttss
/ 7 www. aclweb. org/ ant /N1 8-1101
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: tfate-of-theeart natural language processing. CoRR, abs/1910.03771, 2019. URL 03771
http://arxiv..rg/ hwtpr/g/b////1//
Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference on Learning Representations, 2020. URL
https://openrevieew wtt.://w..ee...eeeeeeeeeeee Stl.mmmmmmdmyy10mmm000m
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2369-2380, Brussels, Belgium, putational Linguistics. doi: 10.18653/v//D18. 1259. onthollocy/D18-1255
2018. Association for Com- URL https: / www. ac wwwweb.oorg org/
15

---

Published conference paper at ICLR 2021
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. CoRR, 100/ 12885, 2018.
12885
Yu Zhang and Qiang Yang. A survey on multi task learning. CoRK, htts//777.00 14, 2017. URL http://arx: org/ abs//a/1/101 110170117110.

---

Published conference paper at ICLR 2021
A
APPENDIX
SUMMARY OF ACRONYMS
Acronyms of datasets and descriptions can he found below in section A.9
Table
acronyms used this puper.
Acronym
Description
ARLM
Autniegresseve Language Modals
Coeditional Adaptive Multi Learning:
CFF
Cooditional Feed Forward: a feed forward laver modulated by conditioning vector
Conditional Layer Normalizaioon
FDM
Evolutionary Data Mensures (Callins et al. 40T a task difficutty estimate
GLUE
General Language Understanling Evaleance Wang al. (018) benchmark with muliiple dalasets
QA
Question Answering
MT
Multi Task
Leariing:
informative BERTDevvine Devlin al. (2015) is example of un MLM
mullippe
modelss
MLM
Masked Language Model: BERT
Multi- Täsk Learning: "learning tasks paralla While using shared reprosentation (Caruane 1997
MRQA
Machine Reading Question Answering Fisch el. 2019) benchmark with mulapie danasets
NER
Named Eaiity Racognition
Narural anguage Processing
SOTA
State of the art
Task finc-tuning: weights typically updated
ST-A
ST with Adapter modules: one adopter pee task is trained and pretrained weights are optionally updated
A.2
UNCERTAINTY SAMPLING: ALGORITHM AND ADDITIONAL RESULTS
Algorithm I: Multi-task Uncertainty Sampling
Input: Training data D, for task & [1...., T}: batch size b; C possible output classes for task f == fo(x,),, our model with weights @, 0,; Output: E' multi-task batch of size b
B+0 0
for + to T do
Generate X₁ (xt,1 (...,.... 21,b3 lid De
for t 1 to do
He,i D. Pa(f pa(f(ar),) log po pe(f(zea))
Entropy of each sample
end
Compute Hl, + Exex, Hiss
Average entropy for task
10
Compute Hi + - -c, O, log
C₁
b. Max entropy (uniform distribution)
12
B BUx, and D2 Dr X₁ if D = then
Reload Dr end
15
16
for i 1 to do
Compute: U4,s Ho.i/ Hi end
Uncertainty normalized with entropy
end
20
Compute A maxxe(1.....?? [Hly]
Entropy task with highest average entropy
a
Update Ue, +- Ue./H
& Normalize each sample's uncertainty measure D samples w/ highest uncertainty Return: With B", solve eq. [ with gradient descent; updated model f
12 B' + top_b((fl,,t [1,.... TJ, [1,....,}}})

---

Published conference paper at ICLR 2021
An advantage of our MT- MT-Uncertainty Sampling approach its ability to manage task difficulty. This is highlighted in Figure 8 In this experiment, we estimated task difficulty using the Evolutionary Data Measures (EDM proposed by Collins et al. (2018). The task difficulty estimate relies on multiple dataset statistics such as the data size, class diversity, class balance and class interference. Interestingly, estimated task difficulty correlates with the first instance that the selection of a specific task occurs. Supposing that QNLI is an outlier, notice that peaks in the data occur whenever tasks are first selected by MT Uncertainty sampling. This process follows the following order: MNLI 2. CoLA 3. RTE 4. QQP 5. MRPC 6.SST-2, which is the order from highest task difficultyy to lowest task difficulty using EDMM As opposed to Curriculum I Learning (Bengio et | 2000)) MT-Uncertainty dynamically prioritizes the most difficult tasks, As also discovered in MTL vision work (Guo al., 2018), this type of prioritization on more difficult tasks may explain MT- Un-eertaittyy's ' improved performance over other task selection methods. In MTL, heuristics to balance tasks during training is typically done by weighting each task's loss differently. We see here how MT-Uncertainty is able prioritize task difficulty.
82-
28
Task Dilliculty
24-
MNLI 4.2
<smmpps
QNLI 3.8
20-
CoLA 3.7
16-
RTE
MRPC 3.5
QQP: 3.5
SST-2 3.2
500
1000
1500
2000
2500
3000
Heraaion
Figure 8: Task composition of MT- sampling and estimated task difficulty using EDM: number of training samples per task each iteration for batch size of 32. The occurrence of first peaks and estimated difficulty follow the same order: From highest lowest: MNLI CoLA > RIE QQP MRPC SST-2.
While the EDM difficulty measure is shown to correlate well with model performance, lacks precision. As reported in Collins et al. (2018), the average score achieved on the Yahoo Answers dataset 69.99 99.9% and its difficulty 4.51. The average score achieved on Yelp Full is 56.8%. 13.1% less than Yahoo Answers and its difficulty is 4.42. The authors mention that "This indicates that the difficulty measure in its current incarnation may be more effective at assigning class of difficulty to datasets, rather than a regression-like value".
A.3 OTHER RELATED WORK
Multi-Tasking in NLP and other fields. MTL weight sharing algorithms such as Mixture-of-Experr (MoF) have found success in NI.P (Lepikhin et al.] 2020). CA-MTI. can complement MoE. since the Transformers multi-headed attention can be seen as form of MoE (Peng al. 2020). In Vision,
MTL can also improve with optimization (Sener & Koltun 2018 or gradient- approaches (Chen et al.
2017 Yu et al. 2020).
Active Learning, Task Selection and Sampling. [khwantri et al. (2018) examined multi-task active learning for neural semantic role labeling in low resource setting, using entity recognition as the sole auxiliary task. They used uncertainty sampling for active learning and found that 12% less data could be used compared passive learning. |Reichrtt et al. (2008) has examined different active learning techniques for the two task annotation scenario, focusing on named entity recognition and syntactic parse tree annotations. In contrast, here we examine the larger scale data regime, the modularization of multi-task neural architecture, and the many task (* 2) setting among other differences. Other than MTAL (Reichart al., 2008 Ikhwantri et al. 2018). Kendall et al. (2017) leveraged model uncertainty to balance MTL losses but not to select tasks as is proposed here.
18

---

Published conference paper at ICLR 2021
A.4 SERO-SHOT RESULTS ON SCITAIL AND SNLI
Before testing models on domain adaptation in section 4.4 we ran zero-shot evaluations on the development set of SciTail and SNLL. Rablenootlines 8-task CA MTA-MLBEETT--SS 's zero-shot transfer abilities when pretrained on GLUE with our MTL approach. We expand the task embedding layer accommodate extra task and explore various embedding initialization. We found that reusing STS-B and MRPC task embeddings worked best for SciTail and SNLI respectively.
Table 8: CA-MTI is flexible and extensible to new tasks. However, CA-MTI. is sensitive to the new task's embedding. We tested multiple task embeddings that worked best on either SciTail or SNLI by checking performance in zero shot setting using 0% of the data.
Initialization of
Scifail
SNLI
task embedding layer
0% of data
0% of data
CoLA's embeddings
43.0
34.0
MNLI's embeddings
33.0
MRPC"s emheddings
34.5
45.5
STS-B's embeddings
46.9
33.2
SST-2's embeeldings
34.2
QQP's embeddings
31.7
37.3
QNLI's embeddings
38.0
RTE's embeddings
32.3
40.6
WNLI's embeddings
29.0
30.4
Average
28.7
37.7
Random initialization
46.8
34.0
Xavier initialization
29.8
37.6
A.5 MORE EXPERIMENTAL DETAILS
We used batch size of 32 and seed of 12 in all experiments. We used Adam (Kingma & Ba 2015)
the optimizer with a learning rate of 2c-5. We applied learning rate decay with warm up over the first 10% of the training steps. Unless otherwise specified, we used 5 epochs, a seed of 12 and sequence length of 128. Additional details are outlined in section. Our data prepossessing and linear decoder heads are the same as in Devlln al. (2018). We used the same dropout rate of 0.1 in all layers. To run our experiments, we used either four NVIDIA P100 GPU for base models or four NVIDIA V100 GPU for larger ones. We did not perform parameter search. We do not ensemble of models or sasc-sp tricks (Devlin al. 2018 [Lu et al. 20196 Clark et al., 2019c). All models are either 12 Transformer layers for BASE and 24 Transformer layers for LARGE. Apart from CA-MTI, models trained in multi-task learning (BERT RoBERTa without adapters) used random task sampling. For Table and Figure all BERT-based model have half their layers frozen (untrained) for a fair comparison of ablation results. For the 24-task MTL and CA-MTL models in Tables and 5 we increased the input sequence length to 256 and used 8 epochs.
A.6 THE DIRECT SUM OPERATOR
In section 2. used the direct operator This operation allows create block diagonal matrix. The direct sum a matrix E R" Raxm and B E R results a matrix of size (m + p) x (re + q), defined as:
G11
ain
0
B
A 0
Om
ayn
0
B
0
0
by
b
bp
6pg

---

Published conference paper at ICLR 2021
A.7 BASELINES AND OTHER EXPERIMENTAL RESULTS
In this section, we present our baseline results for BERT, RoBERTa, CA-MTL as well as other models. Our single task results (ST) that we ran ourselves surpass other paper's s reported scores in Table Liu al. (019c) reports random seed median scores for RoBERTa. However, RoBERTa ST baseline matches or surpasses the original paper's 's scores 4 out times on the development set when scores are comparable (QQP F1 and STS B spearman are not reported).
Table 9: Fl scores are reported for QQP/MRPC, Spearman's correlation for STS-B ITS-B, accuracy the matched/mismatch sets for MNLL, Matthew's correlation for CoLA and accuracy for other tasks. ST=Single Task, MTL=Multitakk. *QNLI v1 (we report v2) #*F1
Spearman's correlation is not reported. *- Unknow
random seeds. Results from: Stickland al. 2019
Liu et al.
2019b)
[Phnn al. (2018)
Liu et (019c).
Total
Trained
GLUE MRPC QNLI QOP RTE SST STS.B Avg
parama
prrms/tsk
CoLA
MNLI
Modells
Dev Results
PAL:+ Samp,"
1.13%
12.5%
81.70
task CA MTLarrass (ouns)
1.12x
82,7/83.1
91.9
BEKT
LARGE Models
Dev set Results
ST BERI-LARGE
9X
100%
64.5
86. 86.7655.9 9
89.3
92.7*
89.3
70.1
94,9
86.5
84.0
LARTE ST BERI-LARGEE (ours)
9x
36.2/86.2
92.5
9x
100%
63.6
86.5066.0
91.4
91.0
88.5
70.2
94,7
88.2
84.5
24-rask CA. MTT LEKTLARCE (ours)
1.2x
5.6%
63.8
86.386600
93.4
88,1
84.5
94.5
90.3
86.6
RoBERIa LARGE Models
Dev set Resuits
POBERTa-LARGEE
(Median 5 runsy
9x
100%
68.0
90.2
90.9
94.7
86.6
96.4
RoBERRa-L (ours)
29..008.9
94.8
84.6
96.4
91.7
24-task CAA-ITLa ARRE (ours)
1.12X
5.6%
69.7
89.4099.
95.9
94.9
8888
91.0
96.2
91.0
89,4
A.8 SOME RESULTS ON LAYER FREEZING AND WITH FULL BLOCK ATTENTION.
All experiments in this section were run for only 5 epochs. exclusively on the GLUE dataset for the large based 8-task CA-MTL model. Results in Table reveal that performance tends to decrease. However, since we wanted to preserve as much pretrained knowledge as possible, we chose to keep at least 50% of layers frozen. While this has slightly lowered our performance 9 GLUE tasks, believe that keeping much of the original pretrained weights is beneficial when increasing total number of tasks in MTL to or more tasks. However, we did not explore this hypothesis more.
freeze more layers,
Table 10: 8-task CA-MTLEE BErtaLAGe (sce section 4.3 for various layer freezing configurations. FI scares are reported for QQP/MRPC, Spearman's correlation for STS-B, accuracy on the matched/mismatch sets for MNLI, Matthew's s correlation for C and accuracy for other tasks, FRA Full Block Attention
% Irozen layers
4 tasks
GLUE QOP
Method
ST CaLA
QNLI
SST-2
STS-R Avg
LARGE Models
Dey set Resalts
ST BERT-LARGE (ours)
0%
63.6
86.50660
91.4
91.0
88.5
76.
93.1
88.2
84.3
CT-
0%
60.2
86.2/86.0
92.0
91.5
88.7
76.3
93.3
89.5
84.9
CA-MIL
25%
63.7
1/85
89.1
91.2
88.6
79.7
92.9
88.5
85.1
CA-MIL
50%
63.2
85 55.5588.5
91.8
90.9
88.3
81.4
93.0
90.1
85.5
CA-MTL
81.7/81.1
85.8
88.6
A.9 DATASET DESCRIPTION
The datasets that were used for the domain adaptation experiments were Sci jointly trained a CA-MTL MoTTLooBKKKAARGE model on 9 GLUE tasks, 8 Super-GLUE tasks, 6 MRQA tasks, and on WNUT201 Derezynski al. 2017).
3 and SNL We 6
All GLUE tasks are binary classification, except STS-B (regression) and MNLI (three classes). We used the same GLU data preprocessing as in Devlin et al. (2018).
https:/allenaioorr/duuusiiiiil Leaderboard be found at: http////ww..eoooooooooooooooo/nn
http:://upprggllbbbecmmmak...oo com/tasss
http:://hhhu..ooommmmmmt///////h 200
20

---

Published conference paper at ICLR 2021
Table 11: GLUE (Wang et al. 2018)
dataset description.
References: Warstadtt ct al.
2018
Socher et al. (2013).
Dolan Brocketi
2005)
Cer et al.
2017).
Williams et al. (018).
Wang et al
2018).
Levesque (201
Acronym
Corpus
Train
Task
Domain
CoLA
Corpus of Linguistic Acceptability
8.5K
acceptability
miscellaneous
SST-22
Stanford Sentiment Treebank
67K
sentiment detection
movie reviews
MRPC
Microsoft Research Paraphrase Corpus
3.7K
paraphrase detection
STS-B*
Semantic Textual Similarity Benchmark
7K
textual similarity
miscellaneous
QQP
Quora Question Pairs
364K
paraphrase detection
online QA
MNLI
Multi- Celti-GGereee NLI
393K
inference
miscellaneous news. Wikipedia
RTE
Recognition Textual Entailment
2.5K
inferenceeentailmmet
WNLI
Winograd NLI
634
coreference
fiction books
Table 12: SuUe Wane et al.
2019b dataset description. References: Clarr et al (011%) Marneffe
ct al
2019).
Gordon et al. 2012))
Khashabi et al. 2018
Zhang et al. (0018).
"Wann et al. (219b))
Poliak
et al.
2018
(011))
Acronym BoolQ'
Corpus
TTain
Task
Domain
Boolenn Questions
9.4K
acceptability
Google queries. Wikipedia miscellaneous
CB
CommitmentBank
250
sentiment detection
COPA"
Choice Plausible Alternatives
paraphrase detection
blogs, encyclopodia miscellancous
MultiC
Multi-Sentence Reading Comprchension
5.1K
textual similarity
ReCoRD
Reading Comprehension
101K
paraphrase detection
news
and Commonsense Reasoning
KTE
Recognition Textual Entailment
2.5K
inference
Wikipedia WordNet, VerbNet
WiC?
Wond-in-Context
6K
word sense disambiguation coreference resolution
WSC
Winograd Schema Challenge
Table
MRQA Fisch et al.
2019) dataset description. References:
Rajpurkar et al. 201 Trischler et al.
2017)
Joshi al. 2017))
Dunn et al.
2017
Yang et al. 2018)) wiatkowwkii et al, 2019)
Acronym
Corpus
Train
Task
Domain
SQuAD'
Stanford QA Dataset
86.6K
crowdsourced questions
Wikipedia
NewsQA?
NewsQA
74.2K
crowdsourced questions
news
TriviaQA
TriviaQA
61.7K
trivia QA
web snippets web snippets Wikipedia
SearchQA
SearchQA
117.4K
Jeopardy QA
HotpotQA 5
HotpotQA
72.9K
crowdsourced questions
Natural Questions
Natural Questions
104.7K
search logs
Wikipedia
SuperGLUE has a more diverse task format than GLUE, which is mostly limited to sentence and sentence-pair classification. We follow the same preprocessing procedure in Wang et al. (20 All tasks are binary classification tasks, except CB (three classes). Also, WiC and WSC span based classification tasks. We used the same modified MRQA dataset and preprocessing steps that were used in Joshi et al | (019). All MRQA tasks are span prediction tasks which seeks to identify start and end tokens an answer span in the input text.
Table 14: SNLI Bowman
2015)
SeiTail Khot
2018) datasets description.
Acronym Corpus
Train
Task
Domain
SNLI
Stanford Natural Language Inference
550.2k
inference
numan-writeen English sentence pairs
SciTail
Science and Entailment
23.5K
entailment
Science question answering
SNLI is a natural inference task where we predict three classes. Examples are: Entailment, Contradiction, and Neutral (irrelevant). SeiTail a textual entailment dataset. The hypotheses in SciTail created from multiple-choie science exams and the answer candidates (premise) are extracted from the web using information retrieval tools. SciTail is binary true/false classification tasks that seeks to predict whether the premise entails the hypothesis. The two datasets are used only for domain adaptation in this study (sce section A.4 for the details of our approach).
target labels
21


Reviews:
[{"review_id": "lqqHbSYexrB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: After seeing the changes the authors have made, I have slightly increased my score. Thanks!\n\nThe authors investigate how to use multi-task training on top of a pretrained model for a variety of unrelated tasks from the GLUE benchmark. Their goal is to reduce the amount of parameters and data needed, while improving (or maintaining) state-of-the-art performance.\n\nThe main contribution are the following innovations: a special attention mechanism called block-diagonal conditional attention, a set of modules for adaptation of a pretrained model, and an uncertainty-based multi-task data sampling method. The latter aims at avoiding catastrophic forgetting, while the former avoid having to share all (or no) parameters.\n\nExperiment are performed on the GLUE benchmark. The authors already published their code in an anonymous GitHub repository (that's great!).\n\nOverall, I like the idea of the paper: it is important to reduce the parameters needed for sets of tasks, to enable NLP models to be deployed in a larger variety of settings, and reducing the amount of data needed. Also, the authors did a great job in terms of conducting evaluations from different angles. For instance, they ask how different amounts of data influence model behavior or if their data sampling method can react to task difficulty.\n\nUnfortunately, I think this particular paper might need some more work before being ready for publication. I will discuss my concerns one-by-one in detail:\n- Most importantly, the evaluation is confusing. The authors switch between using BERT_base and RoBERTa_base without being very clear about when and why, e.g., in Table 2 they have both models, but only in different sections. This makes a direct comparison impossible. Also in Table 2, they claim that they outperform other approaches. However, they compare to BERT models and build themselves on RoBERTa_base; how are the results meaningful if they use a stronger model to start with? (And the comparison to their BERT_base-based model yields roughly equal scores.) I compared their numbers explicitly to Liu et al. (2019), and RoBERTa_base outperforms their approach on nearly all tasks (and on average). I would argue that this is not a fair comparison: did the authors intentionally choose weak baselines? The same applies to Table 3: it is unclear to me why or how the baseline T5 model has been chosen. I do understand that the main point is the reduction of the amount of parameters (per task), but this doesn't mean that the evaluation should paint a wrong picture.\n- Additionally, in my opinion, the authors are misrepresenting prior work when saying in line 163 that the \"MTL approach has not yet been successful in NLP\". The authors refer to the fact that MTL can (and often does!) lead to performance decrease for individual tasks. However, this is not the case, for instance, for massively multilingual MT (see the reference below for an example). Granted, the final effect of MTL depends on task similarities, but that's probably the same for the proposed approach.\n- Major contributions of the work should be described in the main paper. Therefore, it's not ideal that the uncertainty sampling algorithm has been moved to the appendix.\n\nAdditionally, there are some minor things I would add or improve:\n- I would add references to multi-task training on different languages (e.g., Task 1 is translation from EN to FR and Task 2 is translation from EN to DE). For instance: Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. NAACL. (And the references therein.)\n- The paper would benefit from more proofreading. For instance, line 94: \"task latent\" -> \"latent task\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "lneOlScwTCc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new transformer model framework for multitask learning on NLP tasks. To deal with challenges in multitask learning/co-training, the authors proposed five improvements, including modifications on the transformer layers with task conditioning and uncertainty sampling. In the experiments, the authors showed that the proposed model can outperform full fine-tuned BERT large model with less parameters (adding some parameters on a single co-trained model), and with less training data. \n\nThis paper is very well written and easy to follow. It is also a very meaningful application as pretrain-finetuning based approach could be both sub-optimal and ineffiicent. \n\nThe proposed model structure and sampling techniques suits well for multitask learning based solutions. Having soft-parameter sharing conditioned by task embeddings while freezing part of pre-trianed parameters seems reasonable and efficient. The uncertainty based sampling approach is also very interesting and could potentially benefit other multitask learning problems in general. \n\nThe authors also conducted experiments on various types of NLP tasks including GLUE and Super-GLUE, compared with baseline methods such as pretrain finetuning BERT model, adapter based approach, and T5.\n\nHowever, I think the paper could improve in the following aspects.\nA1. In-depth connection to different multitask learning challenges. This paper proposed 5 modeling components. Intuitively, they all target at solving multitask learning challenges mentioned earlier in this paper, e.g., capacity balancing on multiple tasks, catastrophic forgetting, negative task transfer and interference. I think more in-depth discussion on how each of the 5 modeling improvements solve what aspects of those challenges can singficanlty improve the impact of this paper. \n\nFor example, the conditional alignment part, inspired by Wu et al. (2020), could have some similar discussion related to the covariate shifts among different tasks. Uniquely, it is less clear such alignment matrix can learn to capture covariate distribution difference with task embeddings co-learned from other model improvement components introduced in this paper. \n\nSimilarly, it would be interesting to see discussions on how the uncertainty sampling can improve what aspects in multitask learning. Comparing to many sampling tricks (e.g., discussed in T5), would uncertainty sampling dynamically improve task capacity balancing? Either empirical analysis or theoretical justification could improve this subsection. \n\nIt would also be helpful to understand how these 5 model improvements interact with each other, both intuitively and empirically.  \n\nA2. Might need to compare and/or include some existing work. I see two lines of related work are missing. \n\n(1) There are similar research in NLP mutltiask learning that adopts mixture-of-expert based model architecture to do soft-parameter sharing, for example:\n[1] https://arxiv.org/pdf/2006.16668.pdf\n[2] https://arxiv.org/pdf/2007.05891.pdf\nAnd routing/mixute-of-expert based task conditioning is somewhat popular in other domains. It might be interesting to compare, or to point out the modeling/motivation differences of the proposed work compared to those approaches.\n\n(2) Dynamic reweighting based optimization algorithms, for example:\n[1] https://arxiv.org/abs/1711.02257\n[2] https://arxiv.org/abs/1705.07115\n[3] https://arxiv.org/abs/1810.04650\nI think this line of work might be less popular in NLP based tasks but overall would be meaningful to be included in a discussion.\n\na minor comments:\nIt might be good to explicitly mention each of the proposed improvements in figure 1, for example, conditional attention, conditional layer norm, ..\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iGoWRlRYTXQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores a collection of strategies to improve multitask learning and bring performance on par with single task training. The strategies build on and show good awareness of existing work and achieve positive overall results on GLUE, SuperGLUE, and MRQA tasks. Abolation experiments demonstrate the value of the individual components being proposed. The results are particularly impressive given the small number of additional parameters introduced into the model and in the context of prior work that has shown mixed results from multitask training.\n\nHowever, despite the strong results, I still have mixed feelings about this paper. The presentation follows the format of introducing a fairly large number of new methods followed by experiments that directly attempt to exceed the current SoTA. The analysis of the contribution of the individual methods, while greatly appreciated, still almost seems like an after thought. To be a strong accept for ICLR or another similar venue, it think the techniques explored in this paper deserve an alternative treatment that focus on the contribution of the individual methods first and then concludes with their combination. With this presentation, it would be good to include results for any non-standard design choices. For example, on line 110, you have \"We also experimented with full-block 110\nConditional Attention ∈ R L×L, Not only did it have N2 more parameters compared to the block-diagonal variant\". I'd like to see the numbers that support this claim.\n\nI also found some of the comparisons in the paper somewhat confusing. For example, Table 2 compares a BERT baseline to the authors proposed bag of methods, dubbed CA-MTL, combined with a RoBERTa model. If these numbers are to be meaningful, it would be good to include a RoBERTa baseline in the tables.\n\nFinally, one of the original motivations for multitask training was that it might allow us to make use of less data for each individual task. While it's great that the paper shows good performance on GLUE using a fraction of the available training data, and good domain adaption with varying amounts of training data, it would be desirable to dig a little deeper. It would be interesting to see something like Table 1 for other tasks being explore in the paper of perhaps using the whole GLUE benchmark (e.g., using less than 64% of the training data and finding the knee where performance drops off).", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZLlS_WgGyHH", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: see comment below for rationale behind change from 5 to 7.\n\n## Claimed Contribution:\n\nThis paper seeks to develop a multi-task learning (MTL) strategy that performs competitively with single-task (ST) fine-tuning despite having fewer parameters per task. To this end, this work introduces several task-specific modules components and an uncertainty-weighted sampling strategy.\n\tSome of the task-specific module components are similar to the Adapters line of work, with a key difference being that the adapters are trained in a MTL setting versus in ST finetuning. The authors also introduce latent representations of tasks that feed into these different modules. The sampling strategy is well motivated and credibly improves results.\n\tOverall, this MTL approach can(*) match the average performance of single-task finetuning without having to keep a separate copy of the parameters for each task.\n\n(*) see caveats in Cons.\n\n### Pros \n\n* Paper is well-written, concepts are introduced clearly, related work section is well-done.\n* Comparisons are made to relevant related work, including relatively recent papers.\n* Ablations in Table 5 are very useful and convincing.\n* Method perform well compared to alternatives in the Adapters world.\n* Figure 5 is also helpful to understand the benefits of incremental tasks.\n* The sampling method is interesting and compelling, and appendix A2 gives additional interesting results. This strikes me as an important contribution that deserves to be explored more widely, including outside of the Adapters context. \n\n\n### Cons\n\n* Paper tends to overclaim / have some comparisons that don’t have the necessary context. When correcting for those claims, the results are awash with its ST-tuned alternatives and worse than the MTL-then-ST counterparts. In particular:\n  * The abstract claim “With our base model, we attain 2.2% higher performance compared to a full fine-tuned BERT large model on the GLUE benchmark, adding only 5.6% more trained parameters per task” has a  typo that makes it less impressive, as shown in Table 2. This gap is for a full fine-tuned BERT base model, not large. The large BERT model outperforms the CA-MTL base ones by 0.3. In addition, the authors compare a BERT checkpoint to a Roberta one, which feels like something that should be mentioned upfront. \n  * I am not a fan of only exposing the #params per task in Table 3 and 4. I would like to also see the row on #parameters of the model on each task (~220M for T5, ~370 for CA-MTL). It is worth mentioning in Table 3 that you are comparing a BERT-large encoder only model to a ~T5-base encoder decoder. The choice of SpanBERT in Table 4 is also suboptimal, as you are comparing a BERT checkpoint (albeit with an additional QA focused pre-training step) to a Roberta one.\n  * Table 4 compares models derived from BERT checkpoints to yours that is derived from a Roberta one, this should be emphasized more clearly as it explains some of the score discrepancy.\n* There are no comparisons to single-task finetuned Roberta models. On the Dev set, these outperform by 0.7 on Glue for the Base models (see https://arxiv.org/pdf/1907.11692.pdf Table 8), though the story varies considerably by task. \n* There is little mention made of the discrepancies across tasks. For the same base model, CA-MTL underperforms on many tasks, recovering most of the points on RTE, a task that is known for performing well when doing MTL with MNLI (see for instance STILTs, Phang et al, 2018, who find a double digit point gain on Dev when using a BERT model finetuned on MNLi before finetuning on RTE). I think wider discussion of the discrepancy across task is warranted. Overall, it seems like CA-MTL can capture part of the gains of MTL+ST tuning but still underperform ST tuning on tasks where little is gained by doing MTL.\n* The STILTs paper ( https://arxiv.org/abs/1811.01088 ) seems relevant at least in the related work, esp. considering you are citing some of its follow-up papers. It might also be worth comparing to models that do intermediate fine-tuning.\n* There is little discussion of when this would be advantageous. Since the results are often lower than the ST and MTL-then-ST counterparts, it would be worth emphasizing that the lower disk space requirements are key in mobile/on-device contexts.\n* One redeeming remark for some of those points: the papers this method compares to often explore a wider hyperparam grid for each task, while this method does not.\n\n### Recommendation\n\nOverall, I think this is an interesting approach. However, some of the claimed results are not as strong when put in the proper context, which the authors sometime fail to do. As such, I think this paper is a slight reject in its current state. With a fairer presentation of the comparisons it might be a slight accept.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3GUZBRSG95", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary and Key Contributions:**\n\nThis paper proposes the use of task embeddings for a novel conditional attention mechanism and various task-conditioned modules. The paper also proposes the use of an entropy-based multi-task uncertainty sampling to automatically determine how to sample the data for training the MT architecture. The paper demonstrates the effectiveness of the proposed architecture and sampling through multiple comprehensive experiments and ablations/analyses.\n\n**Positives:**\n\n* The paper proposes several modifications to the transformer architecture that enable leveraging a learned task-specific embedding, namely conditional attention, conditional alignment, conditional layer norm and conditional adapter. \n* The paper proposes the use of the entropy-based MT-Uncertainty Sampling to choose which training examples to use, in a manner that accounts for some of the subtleties (that were perhaps not immediately obvious, but clearly visible in hindsight), such as accounting for the differing numbers of classes in each task.\n* Most of the modifications proposed are clean, well motivated and make a lot of intuitive sense. The overall approach also seems to work well without the need for extensive hyperparameter tuning.\n* Strong, thorough experimental validation of the proposed techniques. Evaluations on GLUE, Super-GLUE show the proposed method often outperforming (or at least performing comparably to) several strong baselines, both MT-based and single model/fine-tuned, and with fewer parameters. The paper's proposed approach sets a new state-of-the-art performance on 3 tasks (WNUT-2017, SciTail, SNLI).\n* Good ablations of the utility of the different task embedding-grounded modules, on the performance of MT-Uncertainty sampling compared to other methods, how MT-Uncertainty sampling tends to choose most difficult tasks first while also avoiding catastrophic forgetting, and of transferring to new tasks or progressively adding in additional tasks.\n\n**Suggestions:**\n* The paper claims to attain 2.2% higher performance compared to a full fine-tuned BERT large model (abstract line 16). However, it seems like the paper compares a CA-MTL-RoBERTa-Base model to a BERT-base baseline. This is a somewhat unfair comparison, and it might be better to either clarify or to compare the baseline to the CA-MTL-BERT-Base model (which based on Table 2 would still be a rather impressive 1.3% better while making the comparison fairer).\n* While I understand that space is limited, Section 2.1 might have benefited from explaining FiLM, its use here and some intuition, to help keep the paper more self-contained.\n\n**Questions/Clarifications:**\n* (A5 Table 11) On SciTail, it appears that Random Init performs almost as well as STS-B in the zero-shot setting, which is rather surprising. Do you have any intuition/analysis as to why this might be the case?\n* (line 104-105) How exactly was the block matrix A_n obtained? And does $\\bigoplus$ represent the diag operator?\n* (line 95) Does $p_i(\\mathbf{y_i}|\\mathbf{x_i},\\mathbf{z_i})$ represent the model prediction score, i.e., $f_{\\phi(\\mathbf{z_i}), \\theta_i}(\\mathbf{x_i})$?\n* In Table 5, the total data used was 66.3% (or 64.6%). How was the amount of data to be used determined? Was a stopping point determined based on the GLUE dev set?\n* The paper refers to \"assemble methods\" (line 267, Table 4 left header, line 657). Was it perhaps meant to say \"ensemble\"?\n\n=================================\n\n**Update:**\n\nThank you for you responses and clarifications to mine and the other reviewers' comments. Thank you also for baking so much of the reviewers' feedback into your latest draft, in particular by improving how the comparisons are presented in the abstract and the rest of the paper, by updating Table 2 to make it clearer and cleaner, and by adding experiments and analyses in Section 4. As a suggestion pertaining to the latest draft, I agree with R5 that a consequence of the additions to Section 4 have made it a little crammed, although I certainly understand that the constrained spaced forced the authors to make this trade-off. Overall, I continue to maintain that this is a good paper with novel ideas that are well-justified by experimental results and analyses, and would like to reiterate that I believe that this work is a clear accept.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their constructive engagement in the review process - this paper clearly benefitted from your prompt attention to initial weaknesses - and I thank the reviewers for their excellent, detailed, careful reviews.

This paper presents methods that allow a multi-task learning (MTL) system to performs competitively against single-task (ST) fine-tuning despite using fewer parameters and less data per task. This is a great goal, which disrupts the currently most pursued approach of ST fine-tuning and beats aiming for fully single model like BAM!

Pros

* The paper presents a number of useful methods, some novel like conditional attention mechanisms, for improving MTL. There is a lot in this paper.
* Clean, motivated, intuitive model modifications
* Comprehensive experiments
* Generally well-written paper, fairly comprehensive discussion of related work

Cons

The initial paper had a number of weaknesses, the most consistently mentioned being that the results presented tend to overclaim and to confuse (flipping between BERT and RoBERTa, etc.). The authors have done a good job revising the paper to address these concerns but should certainly remember these points in producing the final version.

The paper is a bit of a grab bag of small contributions that together help for MTL, but which isn't as strong a story as one big novel idea carefully presented and evaluated. While this paper has a ton of work behind it, and a lot of good stuff in it, I think this does somewhat weigh against the paper being selected for spotlight presentation.

Requests for the final version:

Further small clean up: e.g., still one "STITLS" to become "STILTS", some cleaning up could be done in the references where "stilts" is lowercase, there isn't capitalization after punctuation in the title of either Clark et al. paper, and the de Marneffe et al. paper, Fisch et al. paper and others lack information on where the work appeared).

Despite the improvements, I think more can and should be done to make the paper clearer and better laid out. Here are a few suggestions:

* More consistency in labeling the contributions might be possible. I'm imagining that they might be able to be consistently labeled 1 to 5, rather than being 1 to 5 in subsections of section 2, but 1 to 3 with bulleted sub items in the introduction, and several of them (a) through (c) in Fig 1.
* Although the reviewers generally said the paper was clear and well-written, I still feel that section 2.1 is harder to get than it should be – and indeed you resorted to pasting PyTorch code in this discussion to help us! While people often say that a figure should be standalone, maybe that doesn't really make sense when it's an inline figure like this, and you'd be better off explaining things well once (mainly in the text) rather than badly twice?!? I.e., just make the Fig 2 caption: "Figure 2: Conditional attention". I still think it's harder to get what the equations are doing than it should be, because there is inadequate text explaining the equations and the ideas motivating them. Although you now sort of sneak in to the text that ⨁ means diag, you still don't explicitly say so. I think having even 2 more lines of text at your disposal could really help if well deployed.
* As R5 observes, the compactification in pages 7-9 just gets kind of ridiculous. I get that you're trying to deal with page breaks and to fit a lot in etc., but it just makes no sense when the tables embedded into paragraphs on p9 don't even belong with the corresponding paragraph! How might this be fixed? It's difficult, since you do just have a lot of material. The best idea I can come up with would be to shorten the main paper related work section to only 1 paragraph that discusses things at a very high level, and to put your detailed related work (even adding a few more of the things reviewers suggest like MT, etc.) to the Appendix. This might give you another half page in the main paper to make things more sensible.


Decision: Accept (Poster)