Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
Authors: Lucio M. Dery
Paul Michel
Ameet Talwalkar
Graham Neubig
Venue: ICLR 2022 Poster (2022)
Keywords: pre-training
multitask learning
meta-learning
deeplearning
end-task aware training
NLP

Abstract: In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continued-pretraining variant are end-task agnostic: they rarely, if ever, exploit knowledge of the target task. We study replacing end-task agnostic continued training of pre-trained language models with end-task aware training of said models. We argue that for sufficiently important end-tasks, the benefits of leveraging auxiliary data in a task-aware fashion can justify forgoing the traditional approach of obtaining generic, end-task agnostic representations as with (continued) pre-training. On three different low-resource NLP tasks from two domains, we demonstrate that  multi-tasking the end-task and auxiliary objectives results in significantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al. (2020).
We next introduce an online meta-learning algorithm that learns  a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on end-task performance and data efficiency.

Published as a conference paper at ICLR 2022
SHOULD WE BE Pre-TRAINING ? EXPLORING END-TASK AWARE TRAINING IN LIEU OF CONTINUED PRE-TRAINING
Lucio M. Dery', Paul Michel', Ameet Talwalkari.3 Graham Neubig'
Carnegie Mellon University, ENS PSL University, 3 Determined Al
LderySandrew. andrew cmu.edu, mmiche131411@gmaati talwalkar wtt.kkkkarcmumdii.. gneubigRcs. cdu.edu.
ABSTRACT
In most settings of practical concern, machine learning practitioners know in ad- vance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continued- pretraining variant are end-task agnostic: they rarely, ever, exploit knowledge of the target task. We study replacing end-task agnostic continued training of pre- trained language models with end-task aware training of said models. We argue that for sufficiently important end-tasks, the benefits leveraging auxiliary data in task-aware fashion can justify forgoing the traditional approach of obtain- ing generic, end-task agnostic representations as with (continued) pre-training. On three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in significantly bet- ter downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan al. (2020). We next introduce an online meta-learning algorithm that learns set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on end- task performance and data efficiency.
INTRODUCTION
T
The increasingly popular pre-training paradigm Dai & Le, 2015| Devlin et al., 2018, Gururangan et al. 2020) involves first training generalist model copious amounts of easy-to-obtain data, e.g. raw text data NLP, and then using this model to initialize training a wide swath downstream tasks. Generalist models like BERT (Devlin al., 2018), RoBERTa (Liu al. 2019). and GPT-3 (Brown et al. 2020) have a strong appeal; few institutions with significant resources in- cur the cost of training these large models whilst the rest of the research community enjoys significant per- formance improvement at minimal computational over- head. However, the advantages of initializing down- stream task from a generalist model are not guaranteed. Previous work has shown that the benefits pre-training depend heavily on the degree of domain overlap between the end-task data and the massive, heterogenous data on which the generalist model was trained (Beltagy al. 2019
end-task aware training
pre-training+ fine-tuning
Pre-training trains on auxil- iary task Taux before fine-tuning on pri- mary task T*. End-task aware training optimizes both Taux and T simultane- ously and can find better minima since optimization is informed by the end-task.
Figure 1:
Gururangan et al. 2020).
Notably, Gururangan et al. 20200 have demonstrated the benefits of continued pre-training gen- eralist models using data that is similar that of the end-task. Their approach is formalized into two classes: Domain Adaptive Pre-training (DAPT) and Task Adaptive Pretraining (TAPT) where further stages of pre-training of generalist models conducted on domain- and task-specific data,

---

Published as a conference paper at ICLR 2022
respectively. DAPT and TAPT exploit the fact that we often know the end-task beforehand, and so we can make specific choices about our pre-training regimen to improve end-task performance.
However, in both pre- pre-training for generalist models and continued pre-training, the training proce- dure itself does not explicitly incorporate the end-task objective function.. Because of this, practi- tioners have to be careful with their choice of auxiliary tasks, the order in which they are trained on, and the early-stopping criteria for each pre- training stage so as to actually achieve good downstream end-task performance (Gururangan et al. 2020; Dery et al., 2021). In the absence of principled criteria to make these difficult design choices, it is common to instead resort to the computationally demanding heuristic of pre-training on as much data as possible for as long as possible.
In this paper, we raise the following question: "In settings where we have particular end-task in mind, should we be pre-training at all?". We define pre-training as any form of task-agnostic training that a model undergoes before it is finally fine-tuned on the end-task of interest. As first milestone in addressing the larger question posed above, we explore the ubiquitous continued pre- training setting (Gururangan et al. [2020] AAhhaaayyan et al. [022]. Specifically, our paper questions the wisdom of having disjoint further pre-training then fine-tuning steps on a generalist model. In response, we advocate for an alternative approach in which we directly introduce the end-task objective of interest into the learning process. This results in a suite of end-task aware methods called TARTAN (end-Task AwaRe TrAiniNg). Our formulations incorporate both unsupervised auxiliary objectives traditionally used in NL.P pre-training (such as masked language modeling as in Devlin et al. (2018)) and the end-task objective, followed by an optional fine-tuning step on the end-task. We motivate TARTAN experimentally in the continued pre- pre-training setting and based on this, we make the following contributions to the literature on leveraging auxiliary tasks and data:
In lieu of standard end-task agnostic continued pre-training, we suggest introducing the end-task objective into the training process via multi-task learning (Caruana 1997 Ruder 2017). We call this procedure Multi-Tasking TARTAN is a simple yet surprisingly effeetive alternative to aask-agnottic pre- tre-training. In Section
aad-Task_a AwaRe TrAiniNg (MT-TARTAN) (Section B.1). MT-
5 demonstrate that MT-TARTAN significantly improves performance and data efficiency over Gururangan et al. (2000'ss 's results. It also obviates the need for fickle hyper-parameter tuning through direct optimization of validation performance.
To allow more fine- tinengrained control of the end-task over the auxiliary tasks, in Section 3.2 we present an online meta-learning algorithm that learns adaptive multi-task weights with the aim of improving final end-task performance. Our META- learning end-Task AwaRe TrAiniNg (META- TARTAN) allows us to robustly modulate between multiple objectives and further improves per- formance over MT-TARTAN. naive implementation of a sub-optimal algorithm that ignores all tasks except the end-task. We trace this problem to the use of a single model training head for computing both the end-task training loss and meta-objective (end-task validation loss). To guard against this pathological solution, we introduce a separate we justify this
based on first-order meta-learning analysis results in
model head for computing the meta-objective. In Section 3.3
fix and validate its practical efficacy in Section 5
Our results suggest that TARTAN may be an attractive alternative to the continued pre-training paradigm, and further research into the place of pre-training in end-task aware settings is warranted.
2 FORMALIZING
AND CONTINUED PRE- TRA-TIGINGNG
Consider a dataset D = [(1 Be)semm)) consisting of m labelled examples. We define a task as an objective function and dataset pair: T = (C(.). D).. Mj is a model parameterized by The objective function C(y,, (z.)) evaluates how well a model prediction Me(z,) fits the true label yı. such as cross-entropy loss in the case of classification. Note that the task dataset, D, is typically decomposed into the sets (Dirain Vval Disst). ). Dtrain is the set of examples used for model training whilst Deest is used for final task evaluation. The validation set, Dval, is typically used for model selection but it is also frequently used in meta-learning to define the oeta- Loa!
Given a specific end-task T",, our aim is to improve performance on T (as measured by the model loss on D).st) by leveraging auxiliary tasks T Taux = (Ti,...; Tn). Note that we do not particularly
2

---

Published as a conference paper at ICLR 2022
care about the performance of any of the tasks in Taux- We are willing to sacrifice performance Taux it improves performance on T"
From the perspective of model architecture, there several ways leverage Taux. We focus the simple but widely-used parameter sharing setting. Here, all tasks share model body body but each task T, has its own head ø¹ for prediction. We denote the head belonging to T as d'. Thus (booyy' (øt...., ,,", and Oodyy is reusable across new tasks.
0
d')]
2.1 PRE-TRAINING
Pre-trane Pre-training is when model is first trained on Taux before performing final fine-tuning phase on 7" The motivation behind pre-training that learning Taux first hopefully captures relevant information that can be utilized during training of T". This desire has led to the proliferation of generalist pre-trained models like BERT (Devlin et al. 2018). RoBERTa (Liu et al. 2019) and GPT- 3 (Brown et al., 2020) that have been trained copious amounts of data. Generalist models have been widely successful at improving downstream task performance when used as initilization.
We can formalize the pre-training procedure follows:
Oo argming
Σ T.T
CT. (0)
In Equation we seek point 0g that achieves minimal loss on the tasks in Taux. We hope that Oo will be a good starting point for gradient descent on T Let g(%)) represent the set of end-points of stochastic gradient descent on an initialization, Og. Stochastic gradient descent from the same initialization can produce different end- points due to differences in hyper-paramette parameters like learning rate, batch size and order, well as regularization strength. We can write the fine-tuning phase as:
0" = argmin a(oo)) (T-(())
Note that pre-training is end-task agnostic: the pre-training Equation occurs entirely before train- ing on the end-task Equation and does not explicitly incorporate the end-task objective, T Since there is awareness of the end-task during pre-training it is important to carefully choose Taux so that pre-training actually results in improved performance on T" (Wang et al., 2018a). For text data, past work has found left-to-righh language modeling (Peters et 2017)) and masked language modeling (MLM) (Devlin et al. 2018) to be good choices to include in Taxx-
2.2 CONTINUED PRE-TRAINING
Recent work (Beltagy et al.) 2019 Gururangan et al. 2020 Lee et al.] 2020) showed that down- stream performance on T" can be improved by further adapting generalist models via continued pre-training on a more relevant set of auxiliary tasks. This is equivalent to sequentially performing multiple steps of Equation] [ with different Tauxs before finally performing Equation 2 T Domain and Task Adaptive Pre-training Gururangan et al. (2020) present Domain Adaptive Pre- Training (DAPT) and Task Adaptive Pre-Training (TAPT) as methods for continued pre-training. During DAPT, generalist model is further pre-trained on an unsupervised objective with large amounts of data from the same domain as the end-task. TAPT also pre-trains with the same unsu- pervised objective as DAPT, but the actual dataset of the end-task. Gururangan et al. (0200) find that performance can be further improved by chaining objectives, DAPT first, followed by TAPT.
Though TAPT and DAPT do not directly incorporate the end-task objective during training, it still indirectly informs both the choice pre- pre-training data and the order in which the pre-training tasks trained on. Below, explore stronger versions of this influence.
END-TASK AWARE TRAINING (TARTAN)
In this section, we argue for the end-task to added directly into the training process to create explicit interactions between T and T Taux-
3

---

Published as a conference paper at ICLR 2022
3.1 END-TASK AWARE TRAINING VIA MULTI-TASKING (MT- TARTAN)
We propose to directly incorporate knowledge of the end-task by multi-tasking 7 together with Taux, before optionally fine-tuning on 7 exclusively. To this end, introduce set of task weights
w = (w=, W1," W(TAux) satisfying w² + Σi Wi = 1, to modulate between the different losses.
Our new formulation is:
0o = argming Linta (0, w) = argming (0) + (w.LT:(0)
(3)
Here, Equation replaces Equation | and can be followed by the optional fine-tuning stage of Equation 2 Note that this formulation fixes the tasks weights w throughout the training process. We call this formulation End-task Aware Training via Multi-tasking (MT-TARTAN) because we introduce the end-task directly into the training procedure, and do so by multi-tasking it with Taux-
MT-TARTAN allows us to prioritize performance on T in several ways. First, we can weight the end-task higher than all the other auxiliary tasks. Also, during training, we can monitor Zr on the end-task validation set and early stop when it plateaus; even if the auxiliary tasks have not yet converged. This is not possible during standard pre-training because we do not train T" and so it performs at random before we actually start fine-tuning. Early stopping on T" can represent signif- icant computational savings over end-task agnostic pre-training when the savings in data-efficiency supercede the extra overhead of end-task aware gradient descent steps.
3.2 END-TASK AWARE TRAINING VIA META-LEARNING (META- TARTAN)
MT-TARTAN, DAPT and TAPT, all share the same drawback: they implicitly assume that the aux- iliary tasks have static importance to the end-task over the lifetime of its training, either by being end-task agnostic (DAPT and TAPT) or by having static task weights (MT-TARTAN). With MT- TARTAN, an additional drawback noted by Wang et al. ( Yu et al. (020)) is that multi-tasking can negatively impact task performance compared vate the formulation of an adaptive algorithm that can mitigate the negative influence of some tasks whilst responding the changing relevance of auxiliary tasks over the lifetime of end-task training.
isolated training. These shortcomings moti-
As they stand, the pre-training equation pair (Equations I 2 and the MT-TARTAN pair (Equations are decoupled. The inner-level variables of the pre-tra pre-training phase do not depend on the outer- level variables of the fine-tuning phase. Thus the equation pairs are typically solved sequentially. We propose to tightly couple Equations [ and 3 by formulating jointly learning w and On as a bi-level optimization problem. A bi-level formulation allows to leverage meta-learning (Schmidhuber) 1995) techniques to learn adaptive task weights which capture variable auxiliary task importance whilst mitigating the contribution of harmful tasks. We propose meta-learning algorithm in the mold of Model Agnostic Meta-Learning (MAML) (Finn et al., 2017) to learn task weights. As a bi-level problem, this can be formulated as
0,w""= = argmin g(06). w) (0)
(4)
where
(w" LT- +
0u argming (f, w) argming
Σ wiCr, (0)
(5)
T:ET.ux oux
We want to jointly learn w, with 0g, such that taking gradient descent step modulated by leads to improvement in end-task generalization. We use performance on the end-task validation set (Dy)l) as meta-objective to train w. Performance on Diaa serves as stand-in for end-task generalization performance whilst also naturally capturing the asymmetrical importance ?'
Our joint descent algorithm proceeds as follows. At each timestep I, we hold the task weights fixed and update 0, based on (O,,ota(OO,, w). We then proceed update w via gradient descent on the end-task validation loss at 0,+1. For this, we derive an approximation for V Cy (0.+1, w) below:
(0e+1(()(w)) = Cyal 0g B(w VLT + Σ W. V.T..)
T R Lyal (8:) - -3 10" VL + Σ w; W.VLn VyCY )
4

---

Published as a conference paper at ICLR 2022
We can take the gradient of the above first-order approximation w.r. individual weight wy. This tells us how to update W; to improve the meta-objective.
a @Cyal (Or+1 -((LT)) (VLTT -V(VT.)) (V (V (
Cyal ([0hody (()))
(6)
Ow,
In Equation we explicitly specify |Abody: d'I, because computing losses on T depend on only these parameters. Lr, depends solely on [Obody, Ø¹), but we leave this out to avoid notation clutter.
Our analysis above is similar to that Lin et al. (2019) with one key difference: we learn a weighting for the main task w² too. This ability to directly modulate T allows us to capture the fact that at certain stages in training, auxiliary tasks may have greater impact on end-task generalization than the end-task's own training data. This choice also allows us to control for over-fitting and the influence of bad (mislabelled noisy) training data.
3.3 INTRODUCING A SEPARATE CLASSIFICATION HEAD FOR META-LEARNIH
Observe that from Equation 6 updates for w * w" involve gradients computed from different model heads of and ø' whilst for w. we are taking the dot product of gradients from the same end-task head Φ'. As we will show empirically in Section 5.4 computing weight updates this way creates strong bias towards the primary task, causing w" to rail towards whilst the other weights dampen to O, which may be sub-optimal in the long run.
Intuitively, this short-horizon (greedy) (Wu et al. 2018) behavior makes sense: the quickest way to make short-term progress (improve (B(+1) )
is to descend solely on T More formally, the greedy approach arises because we derive V w. Cyal (0s+1) in Equation as proxy for the gradient at 0", the outer-loop end-point in Equation meta-learning literature (Finn al. 2017 Liu et al. 2018 Nichol et al. 2018) because it is compu- tationally infeasible to train model to convergence every time we wish to compute V W. (0*).
Variations of this substitution are common in the
To remedy the greedy solution, instead of estimating VaLr- and Vo from the same classifica- tion head (Equation 6). we introduce special head 6* for computing the mota-objective. Specif- ically, instead of trying to compute 8", we approximate by fixing the body of the network Bbody and training the randomly initialized head o² to convergence data. We do this every time we wish to estimate W.; Cr Cyal (0"). Introducing o² eliminates the strong positive bias on w* and enables us to compute better proxy for the meta-gradient 0*:
subset of the end-task training
d (f" (w)) = (VoLr.) (Voooa" (looody: o")())
Ow,
Equation 7 represents cimple-to-implement alternative to Equation We provide more de- tailed justification for Equation 7 in Appendix [A.]] In Section 5.4 we empirically validate that the transition from Equation 6 improves performance whilst mitigating pathological solutions. Our approach of creating ø² For approximating the meta-objective (down-stream validation perfor- mance) is inspired by |Mtt et al. (2018), who use similar technique construct meta-objective for evaluating the quality of unsupervised representations.
Please see Algorithm in Appendix A.3 for details about META-TARTAN
4 EXPERIMENTAL SETUP
Setting Though our algorithms and methodology can be directly applied to both continued pre- training (Section 2.2) and pre-training from scratch (Section 2.I1 of generalist models, we focus on the former scenario. This is because the continued pre-training setting is more common amongst everyday practitioners it is less computationally demanding. It thus lends itself more easily to exploration under a realistic computational budget. In Appendix A.4 we show that end-task aware training from scratch is viable by studying a simple computer vision setting. Concurrent work by Yao et al. (2021) shows that from-scratch end-task aware training for NLP problems is viable.
"ooee will he released https:/ggthuu..coo///de///rAAA
5

---

Published as a conference paper at ICLR 2022
In keeping with previous work (Devlin et al. 2018 Gururangan al. 2020), focus on Taux as a set of MLM tasks on varied datasets. In the case of DAPT and our end-task task aware variants of it, Taux is an MLM task with data from the domain of the end-task. For TAPT. Taux is an MLM task with data from the end-task itself. DAPT, TAPT and DAPT+ TAPT (chained pre-training with DAPT followed by TAPT) will serve our baseline continued pre- pre-training approaches. We will compare these baselines to their end-task aware variants that use MT-TARTAN and META-TARTAN
Datasets Our experiments focus on two domains: computer science (CS) papers and biomedical (BIOMED) papers. We follow Gururangan et al. (2220) and build our CS and BIOMED domain data from the S2ORC dataset (Lo et al., 2019). We extract 1.49M full text articles to construct our CS corpus and 2.71M for our BIOMED corpus. Under both domains, our end-tasks are low-resource classification tasks. Using low-resource tasks allows us to explore a setting where pre-training can have a significant impact. Under the CS domain, we consider two tasks: ACL-ARC (Jurgens et al., 2018 and SCIERC (Luan et al. 2018). ACL-ARC a 6-way citation intent classification task with 1688 labelled training examples. For SCIERC, the task is to classify the relations between entities in scientific articles. This task has 3219 labelled examples as training data. We choose CHEMPROT Kringelum al., 2016) the classification task from the BIOMED domain. This task has 4169 labelled training examples and the goal is to classify chemical chemical-protein interactions. More details of these datasets can be found in Table of Gururanaan al. (2020). Gururangan et al. (2020) evaluate against all 3 tasks and their available code served as a basis on which we built MT-TARTAN and META-TARTAN.
Model Details We use a pre-trained RoBERTabas Liu et al. 2019) as the shared model base and implement each task as separate multi-layer perceptron (MLP) head on top of this pre-trained base. As in Devlin et al. (2018), we pass the [CLS] token embedding from RoBERTabaee to the MLP for classification.
Training Details For DAPT and TAPT, we download the available pre-trained model bases pro- vided by Gururangan et al. (2020). To train thier corresponding classification heads, we follow the experimental setup described in Appendix B of ofrurrnggn et al. (2020).
Performing end-task task aware training introduces a few extra parampaaaeetess. We fix the other hyper- parameters to those used in Gururangan et al. (2020). MT-TARTAN and META- META-TARTAN introduce joint training of a classification head for the end-task T We experiment with batch sizes of 128, 256 and 512 for training this head. We try out learning rates in the set {10-3, 10 10-5) and drop out rates of {0.1, 0.3). For META-TARTAN since we are now learning the task weights, w, test out task weight learning rates in {10-1, 1,5 x 10 2 10-3 1.102)) 2). Note that for all MT- TARTAN experiments we use equalized task weights IT |Ta I+T- A small grid-search over a handful of weight configurations did not yield significant improvement over the uniform task weighting. We use the Adam optimizer (Kingma Ba 2014) for all experiments.
As mentioned in section 3.3 we train as separate mesa-classiiication head, 6", to estimate the valida- tion meta- mnta-gradientss To estimate o², we use batch sizes of (16,3) 32} samples from T train set. We regularize the meta-head with l2 weight decay and set the decay constant to 0.1. We use a learning rate 10-3 to learn the meta-head. We stop training o² after 10 gradient descent steps.
5 RESULTS AND DISCUSSION
In this section, we will discuss the results of comparing our models against DAPT and TAPT base- lines Broadly, we demonstrate the effectiveness of end-task awareness as improving both perfor- mance and data-efficiency.
"Our results are slightly different from those presented Table of Gururangan et al. (020)) terms of absolute values but the trends observed there still hold here. We attribute these differences to (1) minor implementation differences, and (2) averaging performance over ten seeds instead of five as used in the original paper in order to more strongly establish statistical significance. We observe slightly lower performance on ACI-ARCC and SCIERC tasks due to these changes and higher performance CHEMPROT.

---

Published as a conference paper at ICLR 2022
Domain
Task
RoBERTa
TAPT
MT-TARTAN
META-TARTAN
CS
ACL-ARC
66.033.55
67.744.68
70.48
70.08
SCIERC
77.962.96
79.531.98
80.81u.74
81.48u.82
BIOMED
CHEMPROT
82.10g.98
82.170.65
84.290.63
84.499.50
Table 1: Comparison of our end-task aware approaches to RoBERTa and TAPT. All end-task aware approaches use TAPT as the auxiliary task. Reported results are test macro-F1, except for CHEMPROT, for which we report micro-F1, following BellowyggBelltgyy et (2019). We average across 10 random seeds, with standard deviations as subscripts. Statistically sigmificant performance (p-value from permutation test 0.05), is boldfaced. See A.2 A.5 for more details about this table
5.1
END-TASK AWARENESS IMPROVES OVER TASK- AAS-AGNOOTIC PRE-TRAINING
Table 1 compares TAPT to its end-task aware variants. As in Gururangan al. (020), we observe that performing task adaptive pre-training improves upon just fine-tuning RoBERTa. However, note that introducing the end-task by multi-tasking with the TAPT MLM objective leads significant improvement in performance. This improvement is consistent across the 3 tasks we evaluate against. We find that both MT-TARN and
achieve similar results in this setting.
5.2 END-TASK AWARENESS IMPROVES DATA-EFFICIENCY
Gururangan et al. (2020) train DAPT on large amounts of in-domain data to achieve results com- petitive with TAPT. They use 7.55 billion tokens for the BIOMED domain and 8. 10 billion for the CS domain. This is average
10 x the size of the training data of end-tasks of interest. The large amount of data required to train a competitive DAPT model represents a significant com- putational burden the every-day practitioner. This begets the question: are such large amounts of auxiliary data necessary for achieving good downstream performance? To answer this, we train DAPT and its TARTAN version on variable amounts of data for both SCIERC and ACL-ARC tasks.
Task
02
75-
8.
78
DART
i2. 70
DAPT
NT-TARTAN
META TARTAN w DAFT
65
META TARTAN DAPT
102
10a
10a
10a
10"
10a
10t
10'
10
16"
Ausllary data size as fraction of labelled traring data
Ausiliiar data size as fraction of labelled training ditta
Compared to DAPT, TARTAN makes more efficient use of data. Large standard deviations are a result of the heterogeneity of the domain data used and the fact that our tasks are low-resource.
Figure 2:
TARTAN is more data-efficient than DAPT In Figure 2 we focus on training on a small fraction of available domain data {10%, 10t) } x |Train| for the DAPT auxiliary task. Full domain data is n' 10 x |TTriin]. This relatively low auxiliary data regime represents a realistic setting that is akin to those encountered by everyday practitioners who are likely to be computationally constrained. As can be seen in F on the ACL-ARC task, -ARTANN matches the performance of DAPT when the sizes of the domain data and end-task data are of the same order (10%). At this data size, META-TARTAN supersedes DAPT on the SCIERC task. When trained on 10x more auxiliary data, META-TARTAN supersedes DAPT in performance on both tasks. On the ACL-ARC task, META- TARTAN achieves 71.194.88. which is close to DAPT's performance of 72.49 using more than 10yx auxiliary data. These results indicate that end-task awareness improve data-efficiency and in this case, improvements are on the order of 1000x.
Domain
Task
DAPT
DAPT+TAPT MI-TARTAN META-TARTAN
CS
ACL-ARC
68.602.62
69. 69.125.76
71.58
71.05..37
SCIERC
76.44
77.621.38
81.021.44
8 81.41
BIOMED CHEMPROT
80.760.54
78.22n.74
83.770.60
83.38p.89
Table 2: We use n = 10 x [rain] a small fraction the full domain data which is > 10 x |rain].
TARTAN methods are trained on both DAPT and TAPT. We average performance across 10 seeds.
Statistically significant performance is boldfaced. See A.2 A.6 for more details about this table.
7

---

Published as a conference paper at ICLR 2022
TARTAN is more data-efficient than DAPT+TAPT Table 2 compares DAPT and DAPT+TAPT (DAPT followed by TAPT) to *-TARTAN which multi-task DAPT. TAPT and the end-task. MT- TARTAN and META-TARTAN significantly outperform DAPT and DAPT+TAPT in 2 of the tasks whilst giving higher average performance in the ACL-ARC task. We thus conclude that end-task awareness allows to get greater performance boost out of the same amount of data.
We explore the data efficiency of TARTAN methods even further by comparing the rel- atively data-poor versions of MT-TARTAN and META-TARTAN above (n 10 x (Trainn) to the DAPT and DAPT+TAPT variants trained on all the available domain data (n' 10c [Train|). We can see from Table 3 that for the CS domain, our end-task aware variants come close to (ACL-ARC) and even supersede (SCIERC) the end-task agnos- tic variants though trained with R 1000 less data. For BIOMED domain (CHEMPROT task), increasing the amount of data drastically improves the performance of end-task ag- nostic variants compared to MT-TARTAN and META-TARTM, trained on much less data. Zhang et al. (2020) show that different tasks ex- hibit sigmoid curves in terms of how much pre- training data is required to achieve good results be- fore performance levels off. We contextualize Ta- bles 2 and 3 within said work and posit that the CHEMPROT task intrinsically requires much more data (compared to our other tasks) before perfor- mance begins to improve appreciably.
Task ACL-ARC SCIERC CHEMPROT
DAPT fall
+ +TAPT full
72.493.28
73.791.75
79.972.11
80.001.0s
86.541.05
87.24g.81
Table 3: DAPT and DAPT+TAPT runs on all domain data available.
5.3
META-TARTAN MORE EFFECTIVELY UTILIZES DATA OVER MT-TARTAN
AUXILIARY
We have seen that leveraging TAPT (Ta- ble [ and 2 leads MT-TARTAN and META-TARTAN to perform similarly. The advantage of learning adaptive weights becomes pronounced in the DAPT only setting. Whilst TAPT uses the end-task's own training data for masked language modelling, DAPT uses heterogeneous domain data whose impact on the end-task performance is less clear. Notice from thhat when required to rely solely on domain data for auxiliary tasking, META-TARTAN im- proves performance MT-TARTAN. We attribute META-TARTAN improvement over MT- TARTAN to its ability to more flexibly adapt to incoming data of variable utility to the end-task.
TARTAN
ACL-ARC
SCIERC
CHEMPROT
MT META
69.276.96
81.53p.99
80.263.79
71.194.88
82.31o.75 82.081.19
Table 4: All methods use only DAPT as auxiliary task. We use 10 x |Train|. We report averages across
random seeds. Best average task performance is bolded.
5.4
TASK WEIGHTING STRATEGIES DISCOVERED BY META-LEFARNING
META-TARTAN Weights: Separate Head
NETA-TARTAN Weights (Train Head Meta
0.55
ACL-ARC
0.55
0.50
Tesk
0.50
ACL CRC
DAPT-MLM
0.45
1.0
2.0
2.5
Iteration
Iteration
Figure 3: Having separate classification head for computing meta-gradients is important. Using the same head as when training up- -p-wiights the end-task and under-utilizes auxiliary tasks.
To illustrate the importance of the separate classification head Ø³ for computing the meta-signal for the task weights (described in Section 3.3), we run META-TARTAN experiments with ACL-ARC as the end-task and DAPT as the auxiliary task. We compare using either a separate (ő*) ) or the same (ő') classification head for calculating the meta-gradient. Figure 3 plots the task weightings learned in each setting during training. We can clearly see that using a separate head counteracts the patho- all tasks that are not T* and as a result, improves performance: a delta of 1.7 F1 points in this case. The strategy discovered by TAAATTAAAAA presents an in- teresting contrast to classical pre-training: whilst the initial phase of classical pre-training involves
logical solution of
8

---

Published as a conference paper at ICLR 2022
ACLARC
NETA-TARTAN SCIERC
NETA-TARTAN CHEMPROT
0.45
TAPE-MLM ACL ARRC
50.40
1adk wegg
0.40
DAPT-NL TAPT-MLN CHEMPROT
0.35
TAPT-MIN SCIERC
0030
0.30
025
1.5
Iteration
Iteration
herstion
Figure 4: The meta-leamed task weightings show similar trajectories across different end-tasks.
solely the auxiliary task, early in training, META-TARTAN up-weights the auxiliary task but does not fully zero out the end-task. Later in training, we see leveling off of weights instead of railing the end-task to as in classical pre-training.
Next, we plot a similar graph for using both DAPT and TAPT across three tasks in Figure 4 From the figure, it is apparent that META-TARE TARTAN discovers similar task-weighting strategies across different end-tasks. This suggests that the MLM objective and
strategy for learning task weights are generic enough to induce similar behaviours across tasks. In general, DAPT is significantly up-weighted compared to the end-task and TAPT. Note that the TAPT ACL-ARC task weights (Figure[4) has the same approximate trajectory as ACL-ARC task weight in Figure It seems important to assign high weight the task data (Figure 3 but not necessarily all of it needs to go to the actual task loss (Figure 4). We hypothesize that the diversity in the domain data counteracts overfitting to the end-task data and results in DAPT being up- weiiteg
RELATED WORK
Multi-task learning can be traced back to seminal work by Caruana (1955). Caruana (9977 and has since been the subject of a flourishing literature, recent surveys of which can be found in Ruder (2017) or Zhang & Yang (222)) In NLP, while initial work from Collobert & Weston (2088 already showed the benefits of multi-task learning, it has only recently become a central topic in the field, with the advent of multi-task benchmarks (Wang et al. 2018b McCann et al. 2018).
Pre-training is where a machine learning model is first trained on a generic, data-rich task before being fine-tuned an end-task. In NLP this practice dates back to the use of pre-trained word embeddings (Turian et al.) 2010 Mikolov et al. 2013) and later pre-trained encoders (Kiros et al. 2013 Dai & Le| 2015). Peters et al. (2018) and Howard & Ruder (2018) heralded renaissance of
pre- pre-traine training before BERT (Devlin et al. 2018)) and its many offshoots (Liu et al.] 2019 Yang et al.)
2019 Lewis al. 2019 cemented
the de facto standard for modern NLP.
Meta-leaming dates back to early work from Schmidhuber 1995); ; Thrun 1998). More relevant
to our work is gradieet-bbsed meta-learning for solving bivle optimization problems, first pop- ularized by FFin et al. (2017) and followup work (Nichol et al. for few-shot learning. This method has transferred variety of applications such as architecture search (Liu et al., 2018) and model poisoning (Kurita et al. 2020).
2018; Rajeswaran et al. (0019)
7 CONCLUSION
We have advocated for paradigm shift in the way we approach pre-training. We have moti- vated making pre-training more end-task aware when the end-task is known in advance. Our work introduced two novel end-task aware training algorithms: End-task Aware Training via Multi- tasking (MT-TARTAN) and End-task Aware Training via Meta-learning (META-TARTAN), In Sec- tion 5 we demonstrated the ability of our proposed algorithms to improve performance and data- efficiency over their end-task agnostic counterparts.
This work suggests several promising directions for future work. Instead of learning coarse task level weights, can further performance improvements be achieved via finer anrrngrained example level weighting as in Wang et al. (2020) ? Can meta-learning algorithms like META-TAM TARTAN enable more effective utilization of previously discarded (Aroct-Ouellette & Rudzicz 2020) pre-training auxiliary tasks like Next Sentence Prediction (NSP) (Devlin et al. 2018)? We hope this work spurs conversation around these questions and many more.

---

Published as a conference paper at ICLR 2022
8 ACKNOWLEDGEMENTS
This work was supported in part by DSO National Laboratories, an ENS-CFM Data Science Chair, DARPA FA875017C01411 the National Science Foundation grants I1S170112,, I18188017, USS244613 and I1S-2112471, an Amazon Web Services Award, a Facebook Faculty Research Award, funding from Booz Allen Hamilton Inc., and Block Center Grant. Any opinions, find- ings and conclusions or recommendations expressed in this material are those of the author(s) and do necessarily reflect the views of any of these funding agencies.
ETHICS STATEMENT
Our work introduces new algorithms but leverages pre-existing datasets and models. Overall, this work inherits some of the risk of original work upon which it is implemented. Algorithms for contin- ued training such TAPT and DAPT necessitate per-task training of unsupervised objectives which result in corresponding green-house emissions due to energy consumption (Strubell et al., 2019). However, as shown in Sections 3 [ our new compute-efficien algorithms greatly increase the data efficiency of these algorithms, reducing these harms well the various harms associated with labor for data-collection (Jo & Gebru 2020). Also, since our work is set in the context of pre-existing datasets and models (Section we recognize that any ethical issues that have been vealed in these (such as bias (Bender et al. 2021) or privacy leakage (Carlini et al. (021)) ))) may also propagate models trained using work, and mitigation strategies such (Schick et al. (202T); Liang et al. (2021) may be necessary. Finally, there is potential risk in HETA-TARTAN that lever- aging validation for defining the meta-objective could amplifying bias that exists in this data split, although this is done indirectly through task weighting and hence we believe that this risk is small.
10 REPRODUCIBILITY STATEMENT
We pledge release the source-code for this project improve the ease of reproducibility of our results by the NLP and machine learning communities. In Section we have specified details about datasets, training regimes and models to allow anyone who wishes to reproduce our results without original source code to do Our discussion of the algorithmic and evaluation details can be
found in Appendices/A A.I A.3 and A.2
As we noted in we build off Gururangan et al. (2220)s 's
implementations which can be found at https///stttbbbooom//mmmmmmoomot
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlmmoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. arXiv preprint 0rXiv:2101.11038 2021.
Stéphane Aroca-Ouellette and Frank Rudzicz. On Losses for Modern Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4970-4981 Online, November 2020. Association for Computational Linguisties. doi: 10. 10.118533vv120000 hwwp:/eelll200mmomp/-mmomo main.403. URL httph::pp///.. http://wwwwwwww.bboooooo wwwpebb..rg/ntthoooogg. 2020 emn waaannl-mmmm... 403
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: pretrained language model for scientific text. arXiv preprint arXiv: 1903. 10676, 2019.
Emily M Bender, Timnit Gebru, Angelina McMillan- Majoor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voon Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2021.
Rich Caruana. Learning many related tasks at the same time with backpropagation. In Advances in neural information processing systems, pp. 657-664, 1995.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, 2008.
Andrew M Dai and Quoc Le. arXiv:1511.01453 2015.
Semi-supervised sequence learning.
arXiv preprint
Lucio M. Dery, Yann Dauphin, and David Grangier. Auxiliary task update decomposition the good, the bad and the neutral. In International Conference on Learning Representations, 2021.
/tRtsttpss://
ptntttss://opnreeieeee net. / forum? 1id1 1 wwtp:/iwwwweeeeeeeeeemmeeeeeeeee
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1383-1392, 2018.
statistical significance in natural language processing. In Proceedings the
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Phillip I Good.
Permutation, parametric and bootstrap tests of hypotheses: practical guide to resampling methods for testing hypotheses. 2005.
Suchin Gururangan, Ana Marasovie, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.1090 2020.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers). pp. 328-339 2018.

---

Published as a conference paper at ICLR 2022
Eun Seo Jo and Timnit Gebru. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 306-316, 2020.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Com- putational Linguistics, 6:391-406, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. C. Cortes, Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. URL httpp:// / neur cttppp.cc///pppe//////eeee
.pff
Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. Chemprot-3.0: a global chemical biology diseases mapping. Database, 2016, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pre-trained models. arXiv preprint arXiv:2004.06666 2020.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1334):12344224,, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461, 2019.
Paul Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understand- ing and mitigating social biases in language models. In International Conference Machine Learning, pp. 6565-6576. PMLR, 2021.
Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urta- sun, and Richard Zemel. Reviving and improving recurrent pack-propagation In International Conference on Machine Learning, pp. 3082-3091. PMLR, 2018.
Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806. 1,000550 2018.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan Weld. S2ore: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782. 2019.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp- 1540-1552. PMLR. 2020.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of en- tities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv: 1808 09602, 2018.
12

---

Published as a conference paper at ICLR 2022
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint
2018.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning up- date rules for unsupervised representation learning. arXiv preprint 1804::800400000 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc., 2013. URL https: httttps::/ppoeeeeeeteeee htwce////s...o.ceeggg.....s.. www.:/diiiinnnnnnngggge neurips. ppf
Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. arXiv preprint arXiv:2007.0269) 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1803. 2018.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pupers), pp. 2227-2237, 2018.
Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi-supervised sequence tagging with bidirectional language models. arXiv preprint arXiv:1705.00100. 2017.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. arXiv preprint 190,000009 2019.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. arXiv preprint arXiv:2103.00453 2021.
Jürgen Schmidhuber. On learning how to learn learning strategics. 1995.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint
2019.
Sebastian Thrun. Lifelong learning algorithms. In Learning learn, pp. 181-209. Springer. 1998.
Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Associ- ation for Computational Linguistics, pp. Computational Linguistics. URL
Uppsala, Sweden, July 2010. Association for wwwp:/ooow..rgp111110000000
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, et al. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. arXiv preprint arXiv: /8/2. 10860, 2018a.
Alex Wang, Amanpreet Singh, Julian Michael. Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 07461, 2018b.
Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In International Conference on Ma- chine Learning, pp. 9983-9993 PMLR, 2020.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding neg- ative transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pp. 1 12293-11302 2019.
13

---

Published as a conference paper at ICLR 2022
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXivv preprint 1rXiv:7906.08237, 2019.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NIp from scratch without large- scale pretraining: A simple and efficient framework. arXiv preprint arXiv:2111.04130 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782 2020.
Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. When do you need billions of words of pretraining data? arXiv preprint arXiv:2011.049/4 2020.
Yu Zhang and Qiang Yang. A survey multi-tast task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 JUSTIFYING THE INTRODUCTION OF A META-HEAD
we start with the closed form solution for W., (0*) and then introduce approximations in order to produce Equation 7
Proof. To arrive at Equation 7
First, note that
dCyal (0 (w))
(Vo Co (0" (w)) T
(vm,
[Chain rule]
(8)
Ow,
To get V my 0 (w) invoke the Cauchy Implicit Function Theorem (IFT) with Lorraine et al. 2020
Navon et al. (2020): Liao et al. (2018):
0 (w)
V (Q
V Vw. VoLiota(f((w))
[IFT]
Vwz Vo w" LT. + Σ
wiLT, (0 (w))
T.ET
VoLr((0(w)) ((L))
[only terms with w/ survive]
Bringing it all together, we get:
(O )[F()()))))))
(V
(9)
Ow;
Computing V Ww, Cio (0") from Equation computationally unwieldy since we would only have to optimize 0 to convergence for every step of wy but we would also have to invert the Hessian of typically large model. Our middle ground between Equations 2 and 6 (Equation makes use of the following approximations:
We approximate the inverse Hessian with the identity. This approximation not new; we follow previous work like Lorraine et al. (2020) 3) who explore the use of this approximation because of computational efficiency.
II- V--V (0 )
Vi (0'
lim j=0
We are assuming the contribution of terms with > 0 are negligible.
Instead training the whole network to convergence, at each time-step. we fix the body of the network and train special head ø" convergence data. We then use [d ø"] as a proxy for 0". This a computationally feasible work- around to training all of 0 to convergence get a single step gradient estimate. Especially in the continued pre-training setting where pre-trained generalist model like BERT is used Obody- this approximation is reasonable. To our knowledge, this approximation.
small batch of end-task training
the first suggest
V VoLyal (0*) VoLyal (10body d'I)
cual
Above, we have approximated 0- = [obody; d*}. Since o² is only used to evaluate end-task (I*) validation data, it means remains unchanged with respect to the training data for task T, Thus Lr, (Obodyy (Ø".... d²))) = Vo ((Obady; Φ'1) = VoLr.
Bringing it all together, we get Equation 7 repeated here:
a (0= (w))
(VoLs.) T (VoLye! ((ooody
(o o²(())
Ow;
15

---

Published as a conference paper at ICLR 2022
A.2 CALCULATING P- VALUES FROM PERMUTATION TEST
We used the permutation test Good 2005 Dror et al. 2018) to test for statistical significance. For each test, we generate 10000 permutations calculate significance level. This is sufficient to converge to stable p-value without being a computational burden. We chose this over the common student t-test because
1. We have only 10 runs per algorithm and permutation tests are more robust low sample size
2. Permutation test is assumption free. Student t-tests assume that the samples are normally distributed
3. Permutation test robust to variance in the samples, so even though error-bars can overlap, we still establish significant differences in the samples. Variance in results is expected due to small dataset sizes of end-taske
A.3 ALGORITHM FOR META-TARTAN
Algorithm 1: End-task Aware Training via Meta-learning (META-TARE -AARAAN
Require: 7 Taux: End-task, Set of auxiliary pre-training tasks
Require: 1), 31, S2: Step size hyper-parameter
Initialize
Pre-trained RoBERTa as shared network body, Boody
Task weightings: w", W/ Tan +1
Randomly initialize:
end-task head o'
meta head for end-task as o²
task head, &², for each T. E T aux
while not done do
Big
train // Sample a batch from end-task
(O, BT) // Get end-task grads
To g
V
V
Lo. 9
V V
CT, (0, 0., Ba) // Get task grads. Wie B, ~i
// Learn a new meta head
B B2, 0,0) // B T train Imeta + VaLr- (0,6" (Ba)) // Bra T
6² +
11 Update task weightings
w² f- w² + cos(gmetaa 9h)
wi 5- wi + cos g6)
// Update task parameters
a', "r,aa...,, A.T.,,, = softmax(ur (w,, W1...., W Twma!)
Update Obody + Obody - B1 (1 gØ + Σ, aggh)
Update Φi + Φi B296) ) (o' - Φ6 329
end Result: : 0,6²
A.4 VISION EXPERIMENTS
We validate that the gains from end-task Aware Training not siloed to only learning from text. We conduct an experiment comparing end-task aware training on images to its end-task agnostic variant. We use the Cifar100 dataset (Krizhevsky et al.) 2009). We use the Medium-Sizee Sized Mammals superclass (one the 20 coarse labels) as our main task whilst the other 19 super classes are used auxiliary data. Our primary task is thus 5-way classification task of images different types of
16

---

Published as a conference paper at ICLR 2022
Method
Medium- Mammals
Regular (Task-Agnostic Pre-training
46.72.2
MT-TARTAN
51.31.2
META-TARTAN
52.33.8
Table We report averages across random seeds. Best average task accuracy is bolded. medium- sized mammals whilst whilst the remaining 95 classes are grouped into a single auxiliary task.
As can be seen from Table [ being end-task aware improves over task agnostic pre- pre-training. We find that, again, when auxiliary task consist of solely domain data and task data, META- TARTAN performs better than MI-TARTAN (as measured by averaged performance).
A.5
FULL TAPT TABLE WITH SIGNIFICANCE LEVELS
We repeat Table and provide details about levels of statistical signifance.
Task
TAPT
MT-TARTAN
p-values
META-TARIAN
p-values
ACL-ARC
67.74g.
70.484.42
0.040
70.08
0.069
SCIERC
79.531.0
80.81o.74
0.038
81.480.s
0.005
CHEMPROT
82.17g.065
84. 84.299.65
0.000
84.49h.60
0.000
Table 6: Significance levels as computed from the permutation test. All p-valuss are relative to
the TAPT column. Statistically significant performance(p-valuee from permutation test < 0.05), boldfaced
Task HYPER-PARTISAN
TAPT
META-TARTAN
p values
93.392.26
96.841.72
0.003
Table 7: Additional results for HYPERPARTISAN task. This is binary, partisanship classification task with 515 labeled training examples.
A.6 FULL DAPT/DAPT+TAPT TABLE
We repeat Table
and provide details about levels of statistical signifance.
Task
DAPT
DAPT+TAPT
MT-TARTAN
p-values
META-TARTAN
p-values
ACL-ARC
68. 68.602.0
69.12p.76
71.58
0.110
71.05
0.174
SCIERC
76.441.19
77.621.38
81.021.24
0.000
81.411.70
0.000
CHEMPROT
80.76p.54
78.22o.74
83.770.60
0.000
83.38o.89
0.000
Table 8: Duplicate of Table Significance levels computed from the permutation All
p-values are relative to max DAPT, DAPT *TATT.. Statistically significant performance(p- value from permutation test < 0.05), is boldfaced
A.7 FAQ
1. What settings TARTAN algorithms designed for?
TARTAN algorithms specialize auxiliary objectives to a particular end-task. This comes at risk of losing the generic representations afforded by generalist pre-trained models. Thus if a practitioner has sufficiently important end-task where obtaining improved end-task performance is paramount over generic representations, then TARTAN is viable option.
2. When do we get computational savings from META-TARTAN?
MT-TARTAN does not add any extra overhead compared to pre-train then fine-tune ap- proaches. META-TARTAN however, adds extra overhead per gradient descent step due to computing meta- gradienaas However, as shown Section 5 we are able to get several orders of magnitude improvement in data-efficiency from applying the method. In general,
17

---

Published as a conference paper at ICLR 2022
for the tasks experimented with, we find that the savings in data-efficiency superseded the extra per-timestep meta-learning overhead.
3. When should we use META-TARTAN over MT-TARTAN?
In +TAPT settings (Tables 5 we observe that META-TARTAS and MT-TARTAN per- form similarly. We attribute this to the strength of TAPT-MLM objective. We were pleas- antly surprised that the two methods performed comparatively this setting but in hind- sight, we appreciate the insight that went into designing TAPT-MLM as an objective which makes it a strong baseline. In other settings with less carefully designed auxiliary objec- tives and data (which can potentially be detrimental the end-task) we expect META- TARTAN to perform better. Section 5.3 provides evidence of this.
18


Reviews:
[{"review_id": "zfPwpGCM6z", "reviewer": "Reviewer_uWQC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Formally formularize different configurations of pre-training, finetuning, end-task-aware multitasking or meta learning setups. These formulations are clean and make the difference between the various setup clear.The paper is well-written and easy to understand.\n2. The set-up is clean and the experiments are done thoroughly. There are several interesting observations. For example, the observation of task weighting strategies over time from the meta-learning setup is interesting. It shows that in the beginning the end-task is included but with a smaller weight while in the later phase the end-task is upweighted but the auxiliary tasks still receives small weights, instead of being all zeros.\n\nWeaknesses:\n1. Intuitively, it is not surprising that the proposed method performs better than other end-task agnostic pretraining. While the paper argues that the end-task aware setting, when done correctly, will save computational power, it does not consider the comparisons when there are multiple end-tasks involved. It is understood that a shared pretrained model has the advantage of quick iterations on various end-tasks. When there are a large number of end-tasks involved, it become daunting to train/pre-train large models for each end-task.\n\nThough I also acknowledge that the end-task aware approach proposed by the paper is a good middle ground when there is a specific end task and medium amount of resources are available, which makes training useful large models more accessible, since the end-task aware method are more data-efficient (resource-efficient.)", "summary_of_the_paper": "This paper considers the common setup of pretraining and finetuning paradigm and argues that an end-task aware setting, either by multitasking or an additional meta learning that learns the weights between various auxiliary tasks and the end task, is superior than end-task agnostic pretraining.\n\nThe paper provides formal formulations of the various set-ups. Propose an interesting strategy for the meta-learning setup (Section 3.3).", "summary_of_the_review": "The paper provides supporting evidence that end-task aware training can achieve a better performance while being more data-efficient. The paper is well-written and easy to follow. It provides an alternative to the pretraining + finetuning setup and their observations that end-task aware approaches are more data-efficient might make training large models more accessible to entities with slightly less computational resources.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8HHViCEIym9", "reviewer": "Reviewer_ZMKG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper focuses on the continued pre-training setting and it proposes a multi-task end-task aware training method (MT-TARTAN) and a meta-learning variant to achieve better performance than the pre-training + fine-tuning paradigm. \n\nThe idea to compare co-training with pre-training-then-fine-tuning is interesting and a promising direction. The proposed method is well motivated to adapt weights during training for the auxiliary tasks and the domain task. The analysis about the tasks weight is interesting in figure 3 and 4, which shows some guidance on how to choose training tasks and dynamically changing the weights during MTL.\n\nThere’re a few minor concerns about the paper:\n\n1. Lacking comparisons with other multi-task learning (MTL) work about weight selection\nOne of the main contributions is to use meta-learning for choosing the weights of each pre-training task. There's a line of work regarding weight selection for MTL. How is meta learning compared to these methods in terms of performance, training speed, etc?\n\n[1] Guo, Michelle et al. “Dynamic Task Prioritization for Multitask Learning.” ECCV (2018).\n\n[2] Kendall, Alex et al. “Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018): 7482-7491.\n\n2. The setting studied here is kind of limited. Only focusing on low-resource classification tasks. The experiments cover three datasets and show mixed results. \nThe META-TARTAN doesn’t show much improvement without OOD auxiliary data.\n- According to Table 1, the META learning version seems only slightly on two tasks while worse on the other for TAPT. Is there any explanation here?\n- It’s more convincing to add more fine-tuning datasets and analyze the effect of dataset size and final performance for the proposed method.\n\n3. One merit about DAPT is that for similar tasks belonging to the same domain, you only need to train DAPT once and then do TAPT. On the other hand, the proposed method always needs to mix in pre-training data to fine-tuning tasks. Given this, TARTAN also introduces overhead and it’s not 100% true that DAPT is more data efficient given that you might only need to go through it once. Not to mention many domains actually have plenty of unlabeled corpus.\n\nA few more questions and comments:\n1. What are the number of auxiliary tasks? Seems like for DAPT, TAPT and DAPT+TA, there’s only one auxiliary task? The MLM on either the domain corpus or the target corpus? If that’s the case, then it seems to be too few tasks to be worth selecting from.\n2. How hard is it to extend the proposed method to generation tasks instead of discriminative tasks?\n3. It’s until the experimental section that I realize the paper focuses on continued pre-training rather than pre-training from scratch. I guess it requires some revision to make that clearer at the beginning of the paper.\n\n", "summary_of_the_paper": "The author argues that direct training on both pre-training task and fine-tuning task would lead to better end-task performance. The key is to incorporate the end task objective function to the pre-training stage.\n", "summary_of_the_review": "The MTL f design to combine pre-training tasks and fine-tuning tasks in continued pre-training stage is well motivated. However the experimental setting is a bit limited and the results are also mixed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2zTspBhxFau", "reviewer": "Reviewer_nvTM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and well structured.\n\nI believe the question that the authors actually address, whether auxiliary tasks should be used separately or in conjunction with the main task, is important, and their results should be of interest to the community.\n\nHowever, I think the general framing of their paper in abstract / introduction is misleading. At no point do they train a model from scratch (i.e. without pre-training) with their proposed methods. They do justify this with the high cost of pre-training and the convenient availability of pre-trained models, which ironically would be my main criticisms of actually foregoing generic pre-training. So although they raise the question whether pre-training is necessary, they then don’t actually compare against a model that is not pre-trained. Rather, they show that after pre-training it might not be necessary to further pre-train on large amounts of data just for domain adaptation.\n\nI think the paper would be much stronger if they did not defer their main question (“Should we be pre-training?”) to future work, but rather tested their method with the typical MLM auxiliary task on a newly initialized Transformer model.\n", "summary_of_the_paper": "The paper makes the argument that generic pre-training (on auxiliary tasks) is inferior to task-specific pre-training. The authors argue that the final task should be learned together with the auxiliary tasks in a multi-task setup, which they call MT-TARTAN. They also propose a meta-learning algorithm META-TARTAN that uses meta-learning to mitigate the potential impact of updates from auxiliary tasks that detract from the main task.\n\nThe authors back this up through experiments. They consider three tasks on CS and Biomed papers, where they train 1) the main task on top of a RoBERTa checkpoint 2) the main task on top of a pre-trained model from Gururangan et al. (2020) and 3) the main task mixed-in with auxiliary tasks from (2) on top of a RoBERTa checkpoint. They observe that their approaches (3) outperform (1) and (2), both in terms of model accuracy and data efficiency. In a setting where the auxiliary tasks are potentially more noisy, they also observe an advantage of META-TARTAN over MT-TARTAN.\n\nOverall their work heavily references Gururangan et al. (2020), who show that adapting a generic model to the task domain through more task-specific pre-training can improve model performance. They make it directly comparable by using their models and a subset of tasks considered in that paper.", "summary_of_the_review": "I think the paper raises an interesting question, but then only addresses a different, but related question. The results should still be of interest to the community. I think the paper would be much stronger if the authors actually ran experiments that forego pre-training entirely and instead use their mixed-training approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-uwpkCKQCOx", "reviewer": "Reviewer_HAVV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The authors advocate for / introduce end-task aware (continue) pre-training, which is a new and practical problem setting for NLP practitioners.\n* The authors overcomes the pathological solution problem in their META-TARTAN methods by introducing a separate classification head $\\phi^{*}$. Given that meta-learning is known to be unstable and have optimization challenges, the authors' observation and solution is helpful.\n* Improved performance (with significance test) and data efficiency compared to DAPT and TAPT proposed in Gururangan et al. 2020.\n\nWeakness:\n* The description of the META-TARTAN method is not clear enough. \n  * A figure illustrating the data, parameters and optimization steps for META-TARTAN would be very helpful. (e.g., illustrating the relation of $\\theta_{body}, \\phi^1, \\phi^2, ..., \\phi^{'}$, how meta-objective is computed using weights $w$ and parameters $\\theta$, where does the separate classification head come in, ...)\n  * I get to understand the method better (e.g., whether $\\phi^*$ is re-initialized at every time stamp $t$, whether $\\phi^{'}$ is still being optimized when $\\phi^{*}$ appears) only after seeing Algorithm 1 in the appendix. If space allows, please enrich the Sec 3.2-3.3 or move the algorithm to the main text.\n* Though the authors put lots of efforts in META-TARTAN, it appears that META-TARTAN is comparable with MT-TARTAN in most cases. META-TARTAN seems to be better in the DAPT-only setting (Sec 5.3); however since we're doing _task-aware_ pre-training, TAPT can be easily achieved in this case. I wonder what is the practical utility of META-TARTAN.\n* The paper needs more thorough discussion on computation costs. Computation savings is claimed to be one major advantage of TARTAN. However the comparison is made according to training iterations/steps, while one step in META-TARTAN is much more computationally-expensive than one step in DAPT or TAPT. Please take this into consideration when discussion computation savings.\n\nQuestions and further discussion:\n* I'm not sure if involving the validation set $D_{T^*}^{val}$ in computing and optimizing the meta-objective is a fair practice. DAPT/TAPT only access it for model selection or early stopping; while META-TARTAN can use it to update meta-parameters $w$ and indirectly influence the training for model parameters $\\theta$. Therefore META-TARTAN has more advantage in terms of data usage; this should be discussed in the paper.\n* The authors mentioned \"an optional fine-tuning step\" on the end task in the introduction. Does the results in the paper include this step or not? \n* I wonder why in Figure 3 DAPT-MLM is gradually down-weighted, but in Figure 4 DAPT-MLM is up-weighted. Seems to be contradicting. What does Figure 3 look like if we have more iterations?\n\n---\nThanks the authors for their hard work!", "summary_of_the_paper": "In the paper the authors propose TARTAN, methods to enable end-task aware pre-training. MT-TARTAN simply combines the pre-training objectives and the end-task objective as multi-task learning. META-TARTAN learns a set of weights for the pre-training objective and the end-task objective, using meta-learning. MT-TARTAN and META-TARTAN shows improved performance and data efficiency in a set of three low-resource text classification tasks. ", "summary_of_the_review": "Strength: problem setting is practical and interesting; good performance and improved data-efficiency.\nWeakness: lack of visualization and clear description for the method; comparison in computation cost and data usage is questionable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors argue in favor of task-aware continued pretraining and demonstrate through experiments that using objectives based on the end-task during continued pretraining help in improving downstream performance. 

The reviewers generally appreciated the motivation, the formal treatment of the topic and the thoroughness in the experiments. There were some concerns about (i) positioning of the paper (pretraining as opposed to continued pre-training) (ii) thorough comparison with other MTL frameworks (iii) evaluating on more datasets (iv) cost of continued pretraining for each task v) the benefit of META-TARTAN over MT-TARTAN only in specific  settings and (vi) lack of surprise/novelty in the results. 

IMO, the authors have adequately addressed ALL the above concerns raised by the reviewers. Further, despite the above concerns, all reviewers agree that the problem is well motivated and of interest to the community and most aspects of this work are thorough. The findings will be useful and may spawn other work in this area.
Decision: Accept (Poster)