Year: 2022

Abstract:
In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continuedpretraining variant are end-task agnostic: they rarely, if ever, exploit knowledge of the target task. We study replacing end-task agnostic continued training of pretrained language models with end-task aware training of said models. We argue that for sufficiently important end-tasks, the benefits of leveraging auxiliary data in a task-aware fashion can justify forgoing the traditional approach of obtaining generic, end-task agnostic representations as with (continued) pre-training. On three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in significantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al. (2020). We next introduce an online meta-learning algorithm that learns a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on endtask performance and data efficiency.

text: In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continuedpretraining variant are end-task agnostic: they rarely, if ever, exploit knowledge of the target task. We study replacing end-task agnostic continued training of pretrained language models with end-task aware training of said models. We argue that for sufficiently important end-tasks, the benefits of leveraging auxiliary data in a task-aware fashion can justify forgoing the traditional approach of obtaining generic, end-task agnostic representations as with (continued) pre-training. On three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in significantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al. (2020). We next introduce an online meta-learning algorithm that learns a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on endtask performance and data efficiency.
1 INTRODUCTION
The increasingly popular pre-training paradigm (Dai & Le, 2015; Devlin et al., 2018; Gururangan et al., 2020) involves first training a generalist model on copious amounts of easy-to-obtain data, e.g. raw text data in NLP, and then using this model to initialize training on a wide swath of downstream tasks. Generalist models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020) have a strong appeal; a few institutions with significant resources incur the cost of training these large models whilst the rest of the research community enjoys a significant performance improvement at minimal computational overhead. However, the advantages of initializing a downstream task from a generalist model are not guaranteed. Previous work has shown that the benefits of pre-training depend heavily on the degree of domain overlap between the end-task data and the massive, heterogenous data on
which the generalist model was trained (Beltagy et al., 2019; Gururangan et al., 2020).
Notably, Gururangan et al. (2020) have demonstrated the benefits of continued pre-training of generalist models using data that is similar to that of the end-task. Their approach is formalized into two classes: Domain Adaptive Pre-training (DAPT) and Task Adaptive Pretraining (TAPT) where further stages of pre-training of generalist models are conducted on domain- and task-specific data,
respectively. DAPT and TAPT exploit the fact that we often know the end-task beforehand, and so we can make specific choices about our pre-training regimen to improve end-task performance.
However, in both pre-training for generalist models and continued pre-training, the training procedure itself does not explicitly incorporate the end-task objective function. Because of this, practitioners have to be careful with their choice of auxiliary tasks, the order in which they are trained on, and the early-stopping criteria for each pre-training stage so as to actually achieve good downstream end-task performance (Gururangan et al., 2020; Dery et al., 2021). In the absence of principled criteria to make these difficult design choices, it is common to instead resort to the computationally demanding heuristic of pre-training on as much data as possible for as long as possible.
In this paper, we raise the following question: “In settings where we have a particular end-task in mind, should we be pre-training at all?”. We define pre-training as any form of task-agnostic training that a model undergoes before it is finally fine-tuned on the end-task of interest. As a first milestone in addressing the larger question posed above, we explore the ubiquitous continued pretraining setting (Gururangan et al., 2020; Aghajanyan et al., 2021). Specifically, our paper questions the wisdom of having disjoint further pre-training then fine-tuning steps on a generalist model. In response, we advocate for an alternative approach in which we directly introduce the end-task objective of interest into the learning process. This results in a suite of end-task aware methods called TARTAN (end-Task AwaRe TrAiniNg). Our formulations incorporate both unsupervised auxiliary objectives traditionally used in NLP pre-training (such as masked language modeling as in Devlin et al. (2018)) and the end-task objective, followed by an optional fine-tuning step on the end-task. We motivate TARTAN experimentally in the continued pre-training setting and based on this, we make the following contributions to the literature on leveraging auxiliary tasks and data:
• In lieu of standard end-task agnostic continued pre-training, we suggest introducing the end-task objective into the training process via multi-task learning (Caruana, 1997; Ruder, 2017). We call this procedure Multi-Tasking end-Task AwaRe TrAiniNg (MT-TARTAN) (Section 3.1). MTTARTAN is a simple yet surprisingly effective alternative to task-agnostic pre-training. In Section 5, we demonstrate that MT-TARTAN significantly improves performance and data efficiency over Gururangan et al. (2020)’s results. It also obviates the need for fickle hyper-parameter tuning through direct optimization of validation performance. • To allow more fine-grained control of the end-task over the auxiliary tasks, in Section 3.2, we present an online meta-learning algorithm that learns adaptive multi-task weights with the aim of improving final end-task performance. Our META-learning end-Task AwaRe TrAiniNg (METATARTAN) allows us to robustly modulate between multiple objectives and further improves performance over MT-TARTAN . • A naive implementation of META-TARTAN based on first-order meta-learning analysis results in a sub-optimal algorithm that ignores all tasks except the end-task. We trace this problem to the use of a single model training head for computing both the end-task training loss and meta-objective (end-task validation loss). To guard against this pathological solution, we introduce a separate model head for computing the meta-objective. In Section 3.3, we justify this simple-to-implement fix and validate its practical efficacy in Section 5.
Our results suggest that TARTAN may be an attractive alternative to the continued pre-training paradigm, and further research into the place of pre-training in end-task aware settings is warranted.

heading: 2 FORMALIZING PRE-TRAINING AND CONTINUED PRE-TRAINING
text: Consider a dataset D = {(xi, yi)i∈[m]} consisting of m labelled examples. We define a task as an objective function and dataset pair: T = {L(·), D}. Mθ is a model parameterized by θ. The objective function L(yi,Mθ(xi)) evaluates how well a model prediction Mθ(xi) fits the true label yi, such as cross-entropy loss in the case of classification. Note that the task dataset, D, is typically decomposed into the sets (Dtrain, Dval, Dtest). Dtrain is the set of examples used for model training whilst Dtest is used for final task evaluation. The validation set, Dval, is typically used for model selection but it is also frequently used in meta-learning to define the meta-objective – Lval. Given a specific end-task T ∗, our aim is to improve performance on T ∗ (as measured by the model loss on DtestT∗ ) by leveraging auxiliary tasks Taux = {T1, . . . , Tn}. Note that we do not particularly
care about the performance of any of the tasks in Taux. We are willing to sacrifice performance on Taux if it improves performance on T ∗.
From the perspective of model architecture, there are several ways to leverage Taux. We focus on the simple but widely-used parameter sharing setting. Here, all tasks share a model body θbody but each task Ti has its own head φi for prediction. We denote the head belonging to T ∗ as φ′. Thus θ = [ θbody; ( φ1, . . . , φn, φ′ )] and θbody is reusable across new tasks.

heading: 2.1 PRE-TRAINING
text: Pre-training is when a model is first trained on Taux before performing a final fine-tuning phase on T ∗. The motivation behind pre-training is that learning Taux first hopefully captures relevant information that can be utilized during training of T ∗. This desire has led to the proliferation of generalist pre-trained models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and GPT3 (Brown et al., 2020) that have been trained on copious amounts of data. Generalist models have been widely successful at improving downstream task performance when used as initilization.
We can formalize the pre-training procedure as follows:
θ0 = argminθ ( ∑ Ti∈Taux LTi(θ) )
(1)
In Equation 1, we seek a point θ0 that achieves minimal loss on the tasks in Taux. We hope that θ0 will be a good starting point for gradient descent on T ∗. Let g(θ0) represent the set of end-points of stochastic gradient descent on an initialization, θ0. Stochastic gradient descent from the same initialization can produce different end-points due to differences in hyper-parameters like learning rate, batch size and order, as well as regularization strength. We can write the fine-tuning phase as:
θ∗ = argmin{θ ∈ g(θ0)} LT∗(θ) (2)
Note that pre-training is end-task agnostic: the pre-training Equation 1 occurs entirely before training on the end-task Equation 2, and does not explicitly incorporate the end-task objective, T ∗. Since there is no awareness of the end-task during pre-training it is important to carefully choose Taux so that pre-training actually results in improved performance on T ∗ (Wang et al., 2018a). For text data, past work has found left-to-right language modeling (Peters et al., 2017) and masked language modeling (MLM) (Devlin et al., 2018) to be good choices to include in Taux.

heading: 2.2 CONTINUED PRE-TRAINING
text: Recent work (Beltagy et al., 2019; Gururangan et al., 2020; Lee et al., 2020) showed that downstream performance on T ∗ can be improved by further adapting generalist models via continued pre-training on a more relevant set of auxiliary tasks. This is equivalent to sequentially performing multiple steps of Equation 1, with different Taux, before finally performing Equation 2 on T ∗. Domain and Task Adaptive Pre-training Gururangan et al. (2020) present Domain Adaptive PreTraining (DAPT) and Task Adaptive Pre-Training (TAPT) as methods for continued pre-training. During DAPT, a generalist model is further pre-trained on an unsupervised objective with large amounts of data from the same domain as the end-task. TAPT also pre-trains with the same unsupervised objective as DAPT, but on the actual dataset of the end-task. Gururangan et al. (2020) find that performance can be further improved by chaining objectives, DAPT first, followed by TAPT.
Though TAPT and DAPT do not directly incorporate the end-task objective during training, it still indirectly informs both the choice of pre-training data and the order in which the pre-training tasks are trained on. Below, we explore stronger versions of this influence.

heading: 3 END-TASK AWARE TRAINING (TARTAN)
text: In this section, we argue for the end-task to be added directly into the training process to create explicit interactions between T ∗ and Taux.

heading: 3.1 END-TASK AWARE TRAINING VIA MULTI-TASKING (MT-TARTAN)
text: We propose to directly incorporate knowledge of the end-task by multi-tasking T ∗ together with Taux, before optionally fine-tuning on T ∗ exclusively. To this end, we introduce a set of task weights w = (w∗, w1, · · · , w|Taux|) satisfying w∗ + ∑ i wi = 1, to modulate between the different losses. Our new formulation is:
θ0 = argminθ Ltotal(θ,w) = argminθ ( w∗LT∗(θ) + ∑ i wiLTi(θ) )
(3)
Here, Equation 3 replaces Equation 1 and can be followed by the optional fine-tuning stage of Equation 2. Note that this formulation fixes the tasks weights w throughout the training process. We call this formulation End-task Aware Training via Multi-tasking (MT-TARTAN) because we introduce the end-task directly into the training procedure, and do so by multi-tasking it with Taux.
MT-TARTAN allows us to prioritize performance on T ∗ in several ways. First, we can weight the end-task higher than all the other auxiliary tasks. Also, during training, we can monitor LT∗ on the end-task validation set and early stop when it plateaus; even if the auxiliary tasks have not yet converged. This is not possible during standard pre-training because we do not train T ∗ and so it performs at random before we actually start fine-tuning. Early stopping on T ∗ can represent significant computational savings over end-task agnostic pre-training when the savings in data-efficiency supercede the extra overhead of end-task aware gradient descent steps.

heading: 3.2 END-TASK AWARE TRAINING VIA META-LEARNING (META-TARTAN)
text: MT-TARTAN, DAPT and TAPT, all share the same drawback: they implicitly assume that the auxiliary tasks have static importance to the end-task over the lifetime of its training, either by being end-task agnostic (DAPT and TAPT) or by having static task weights (MT-TARTAN). With MTTARTAN, an additional drawback noted by Wang et al. (2019); Yu et al. (2020) is that multi-tasking can negatively impact task performance compared to isolated training. These shortcomings motivate the formulation of an adaptive algorithm that can mitigate the negative influence of some tasks whilst responding to the changing relevance of auxiliary tasks over the lifetime of end-task training.
As they stand, the pre-training equation pair (Equations 1, 2) and the MT-TARTAN pair (Equations 2, 3) are decoupled. The inner-level variables of the pre-training phase do not depend on the outerlevel variables of the fine-tuning phase. Thus the equation pairs are typically solved sequentially. We propose to tightly couple Equations 2 and 3 by formulating jointly learning w and θ0 as a bi-level optimization problem. A bi-level formulation allows us to leverage meta-learning (Schmidhuber, 1995) techniques to learn adaptive task weights which capture variable auxiliary task importance whilst mitigating the contribution of harmful tasks. We propose a meta-learning algorithm in the mold of Model Agnostic Meta-Learning (MAML) (Finn et al., 2017) to learn task weights. As a bi-level problem, this can be formulated as :
θ∗,w∗ = argmin{θ ∈ g(θ0), w} LT∗(θ) (4) where
θ0 = argminθ Ltotal(θ,w) = argminθ ( w∗LT∗(θ) + ∑ Ti∈Taux wiLTi(θ) )
(5)
We want to jointly learn w, with θ0, such that taking a gradient descent step modulated by w leads to improvement in end-task generalization. We use performance on the end-task validation set (DvalT∗ ) as a meta-objective to train w. Performance on DvalT∗ serves as a stand-in for end-task generalization performance whilst also naturally capturing the asymmetrical importance of T ∗.
Our joint descent algorithm proceeds as follows. At each timestep t, we hold the task weights fixed and update θt based on ∇θLtotal(θt,w). We then proceed to update w via gradient descent on the end-task validation loss at θt+1. For this, we derive an approximation for∇wLvalT∗ (θt+1,w) below:
LvalT∗ (θt+1(w)) = LvalT∗ ( θt − β ( w∗∇LT∗ + ∑ i wi∇LTi ))
≈ LvalT∗ (θt)− β ( w∗∇LT∗ + ∑ i wi∇LTi )T ∇LvalT∗ (θt)
We can take the gradient of the above first-order approximation w.r.t an individual weight wi. This tells us how to update wi to improve the meta-objective.
∂LvalT∗ (θt+1(w)) ∂wi
≈ −β ( ∇LTi )T (∇LvalT∗ (θt)) = −β(∇LTi)T (∇LvalT∗ ([θbody, φ′]t)) (6) In Equation 6, we explicitly specify [ θbody, φ ′] t
because computing losses on T ∗ depend on only these parameters. LTi depends solely on [ θbody, φ i ] t but we leave this out to avoid notation clutter.
Our analysis above is similar to that of Lin et al. (2019) with one key difference: we learn a weighting for the main task w∗ too. This ability to directly modulate T ∗ allows us to capture the fact that at certain stages in training, auxiliary tasks may have greater impact on end-task generalization than the end-task’s own training data. This choice also allows us to control for over-fitting and the influence of bad (mislabelled or noisy) training data.

heading: 3.3 INTRODUCING A SEPARATE CLASSIFICATION HEAD FOR META-LEARNING
text: Observe that from Equation 6, updates forw 6= w∗ involve gradients computed from different model heads φi and φ′ whilst for w∗, we are taking the dot product of gradients from the same end-task head φ′. As we will show empirically in Section 5.4, computing weight updates this way creates a strong bias towards the primary task, causing w∗ to rail towards 1 whilst the other weights dampen to 0, which may be sub-optimal in the long run.
Intuitively, this short-horizon (greedy) (Wu et al., 2018) behavior makes sense: the quickest way to make short-term progress (improve LvalT∗ (θt+1)) is to descend solely on T ∗. More formally, the greedy approach arises because we derive ∇wiLvalT∗ (θt+1) in Equation 6 as a proxy for the gradient at θ∗, the outer-loop end-point in Equation 4. Variations of this substitution are common in the meta-learning literature (Finn et al., 2017; Liu et al., 2018; Nichol et al., 2018) because it is computationally infeasible to train a model to convergence every time we wish to compute∇wiLvalT∗ (θ∗).
To remedy the greedy solution, instead of estimating ∇θLT∗ and ∇θLvalT∗ from the same classification head (Equation 6), we introduce a special head φ∗ for computing the meta-objective. Specifically, instead of trying to compute θ∗, we approximate it by fixing the body of the network θbody and training the randomly initialized head φ∗ to convergence on a subset of the end-task training data. We do this every time we wish to estimate∇wiLvalT∗ (θ∗). Introducing φ∗ eliminates the strong positive bias on w∗ and enables us to compute a better proxy for the meta-gradient at θ∗:
∂LvalT∗ (θ∗(w)) ∂wi
≈ ( ∇θLTi )T (∇θLvalT∗ ([θbody;φ∗]t)) (7) Equation 7 represents a simple-to-implement alternative to Equation 6. We provide a more detailed justification for Equation 7 in Appendix A.1. In Section 5.4, we empirically validate that the transition from Equation 6 to 7 improves performance whilst mitigating pathological solutions. Our approach of creating φ∗ for approximating the meta-objective (down-stream validation performance) is inspired by Metz et al. (2018), who use a similar technique to construct a meta-objective for evaluating the quality of unsupervised representations.
Please see Algorithm 1 in Appendix A.3 for details about META-TARTAN.

heading: 4 EXPERIMENTAL SETUP
text: Setting1 Though our algorithms and methodology can be directly applied to both continued pretraining (Section 2.2) and pre-training from scratch (Section 2.1) of generalist models, we focus on the former scenario. This is because the continued pre-training setting is more common amongst everyday practitioners as it is less computationally demanding. It thus lends itself more easily to exploration under a realistic computational budget. In Appendix A.4, we show that end-task aware training from scratch is viable by studying a simple computer vision setting. Concurrent work by Yao et al. (2021) shows that from-scratch end-task aware training for NLP problems is viable.
1Code will be released at https://github.com/ldery/TARTAN
In keeping with previous work (Devlin et al., 2018; Gururangan et al., 2020), we focus on Taux as a set of MLM tasks on varied datasets. In the case of DAPT and our end-task aware variants of it, Taux is an MLM task with data from the domain of the end-task. For TAPT, Taux is an MLM task with data from the end-task itself. DAPT, TAPT and DAPT+TAPT (chained pre-training with DAPT followed by TAPT) will serve as our baseline continued pre-training approaches. We will compare these baselines to their end-task aware variants that use MT-TARTAN and META-TARTAN.
Datasets Our experiments focus on two domains: computer science (CS) papers and biomedical (BIOMED) papers. We follow Gururangan et al. (2020) and build our CS and BIOMED domain data from the S2ORC dataset (Lo et al., 2019). We extract 1.49M full text articles to construct our CS corpus and 2.71M for our BIOMED corpus. Under both domains, our end-tasks are low-resource classification tasks. Using low-resource tasks allows us to explore a setting where pre-training can have a significant impact. Under the CS domain, we consider two tasks: ACL-ARC (Jurgens et al., 2018) and SCIERC (Luan et al., 2018). ACL-ARC is a 6-way citation intent classification task with 1688 labelled training examples. For SCIERC, the task is to classify the relations between entities in scientific articles. This task has 3219 labelled examples as training data. We choose CHEMPROT (Kringelum et al., 2016) as the classification task from the BIOMED domain. This task has 4169 labelled training examples and the goal is to classify chemical-protein interactions. More details of these datasets can be found in Table 2 of Gururangan et al. (2020). Gururangan et al. (2020) evaluate against all 3 tasks and their available code served as a basis on which we built MT-TARTAN and META-TARTAN.
Model Details We use a pre-trained RoBERTabase (Liu et al., 2019) as the shared model base and implement each task as a separate multi-layer perceptron (MLP) head on top of this pre-trained base. As in Devlin et al. (2018), we pass the [CLS] token embedding from RoBERTabase to the MLP for classification.
Training Details For DAPT and TAPT, we download the available pre-trained model bases provided by Gururangan et al. (2020). To train thier corresponding classification heads, we follow the experimental setup described in Appendix B of Gururangan et al. (2020).
Performing end-task aware training introduces a few extra hyper-parameters. We fix the other hyperparameters to those used in Gururangan et al. (2020). MT-TARTAN and META-TARTAN introduce joint training of a classification head for the end-task T ∗. We experiment with batch sizes of 128, 256 and 512 for training this head. We try out learning rates in the set {10−3, 10−4, 10−5} and drop out rates of {0.1, 0.3}. For META-TARTAN since we are now learning the task weights, w, we test out task weight learning rates in {10−1, 5 × 10−2, 3 × 10−2, 10−2}. Note that for all MTTARTAN experiments we use equalized task weights 1|Taux|+1 . A small grid-search over a handful of weight configurations did not yield significant improvement over the uniform task weighting. We use the Adam optimizer (Kingma & Ba, 2014) for all experiments.
As mentioned in section 3.3, we train as separate meta-classification head, φ∗, to estimate the validation meta-gradients. To estimate φ∗, we use batch sizes of {16, 32} samples from T ∗’s train set. We regularize the meta-head with l2 weight decay and set the decay constant to 0.1. We use a learning rate 10−3 to learn the meta-head. We stop training φ∗ after 10 gradient descent steps.

heading: 5 RESULTS AND DISCUSSION
text: In this section, we will discuss the results of comparing our models against DAPT and TAPT baselines.2 Broadly, we demonstrate the effectiveness of end-task awareness as improving both performance and data-efficiency.
2Our results are slightly different from those presented in Table 5 of Gururangan et al. (2020) in terms of absolute values but the trends observed there still hold here. We attribute these differences to (1) minor implementation differences, and (2) averaging performance over ten seeds instead of five as used in the original paper in order to more strongly establish statistical significance. We observe slightly lower performance on ACL-ARC and SCIERC tasks due to these changes and higher performance on CHEMPROT.

heading: Domain Task RoBERTa TAPT MT-TARTAN META-TARTAN

heading: 5.1 END-TASK AWARENESS IMPROVES OVER TASK-AGNOSTIC PRE-TRAINING
text: Table 1 compares TAPT to its end-task aware variants. As in Gururangan et al. (2020), we observe that performing task adaptive pre-training improves upon just fine-tuning RoBERTa. However, note that introducing the end-task by multi-tasking with the TAPT MLM objective leads to a significant improvement in performance. This improvement is consistent across the 3 tasks we evaluate against. We find that both MT-TARTAN and META-TARTAN achieve similar results in this setting.

heading: 5.2 END-TASK AWARENESS IMPROVES DATA-EFFICIENCY
text: Gururangan et al. (2020) train DAPT on large amounts of in-domain data to achieve results competitive with TAPT. They use 7.55 billion tokens for the BIOMED domain and 8.10 billion for the CS domain. This is on average over 104× the size of the training data of our end-tasks of interest. The large amount of data required to train a competitive DAPT model represents a significant computational burden to the every-day practitioner. This begets the question: are such large amounts of auxiliary data necessary for achieving good downstream performance? To answer this, we train DAPT and its TARTAN version on variable amounts of data for both SCIERC and ACL-ARC tasks.
TARTAN is more data-efficient than DAPT In Figure 2, we focus on training on a small fraction of available domain data n = {100, 101}× |Train| for the DAPT auxiliary task. Full domain data is n′ ≈ 104×|Train|. This relatively low auxiliary data regime represents a realistic setting that is akin to those encountered by everyday practitioners who are likely to be computationally constrained. As can be seen in Figure 2, on the ACL-ARC task, META-TARTAN matches the performance of DAPT when the sizes of the domain data and end-task data are of the same order (100). At this data size, META-TARTAN supersedes DAPT on the SCIERC task. When trained on 10×more auxiliary data, META-TARTAN supersedes DAPT in performance on both tasks. On the ACL-ARC task, METATARTAN achieves 71.194.88, which is close to DAPT’s performance of 72.493.28 using more than 103× auxiliary data. These results indicate that end-task awareness can improve data-efficiency and in this case, improvements are on the order of 1000×.

heading: Domain Task DAPT DAPT+TAPT MT-TARTAN META-TARTAN
text: TARTAN is more data-efficient than DAPT+TAPT Table 2 compares DAPT and DAPT+TAPT (DAPT followed by TAPT) to *-TARTAN which multi-task DAPT, TAPT and the end-task. MTTARTAN and META-TARTAN significantly outperform DAPT and DAPT+TAPT in 2 of the tasks whilst giving higher average performance in the ACL-ARC task. We thus conclude that end-task awareness allows us to get a greater performance boost out of the same amount of data.
We explore the data efficiency of TARTAN methods even further by comparing the relatively data-poor versions of MT-TARTAN and META-TARTAN above (n = 10 × |Train|) to the DAPT and DAPT+TAPT variants trained on all the available domain data (n′ ≈ 104 × |Train|). We can see from Table 3 that for the CS domain, our end-task aware variants come close to (ACL-ARC) and even supersede (SCIERC) the end-task agnostic variants though trained with ≈ 1000× less data. For BIOMED domain (CHEMPROT task), increasing the amount of data drastically improves the performance of end-task agnostic variants compared to MT-TARTAN and META-TARTAN trained on much less data.
Zhang et al. (2020) show that different tasks exhibit sigmoid-like curves in terms of how much pretraining data is required to achieve good results before performance levels off. We contextualize Tables 2 and 3 within said work and posit that the CHEMPROT task intrinsically requires much more data (compared to our other tasks) before performance begins to improve appreciably.

heading: 5.3 META-TARTAN MORE EFFECTIVELY UTILIZES OUT-OF-DISTRIBUTION AUXILIARY
text: DATA OVER MT-TARTAN
TARTAN ACL-ARC SCIERC CHEMPROT MT 69.270.96 81.530.99 80.263.79 META 71.194.88 82.081.19 82.310.75
heterogeneous domain data whose impact on the end-task performance is less clear. Notice from Table 4 that when required to rely solely on domain data for auxiliary tasking, META-TARTAN improves performance over MT-TARTAN. We attribute META-TARTAN’s improvement over MTTARTAN to its ability to more flexibly adapt to incoming data of variable utility to the end-task.

heading: 5.4 TASK WEIGHTING STRATEGIES DISCOVERED BY META-LEARNING
text: To illustrate the importance of the separate classification head φ∗ for computing the meta-signal for the task weights (described in Section 3.3), we run META-TARTAN experiments with ACL-ARC as the end-task and DAPT as the auxiliary task. We compare using either a separate (φ∗) or the same (φ′) classification head for calculating the meta-gradient. Figure 3 plots the task weightings learned in each setting during training. We can clearly see that using a separate head counteracts the pathological solution of down-weighting all tasks that are not T ∗ and as a result, improves performance: a delta of 1.7 F1 points in this case. The strategy discovered by META-TARTAN presents an interesting contrast to classical pre-training: whilst the initial phase of classical pre-training involves
solely the auxiliary task, early in training, META-TARTAN up-weights the auxiliary task but does not fully zero out the end-task. Later in training, we see leveling off of weights instead of railing the end-task to 1 as in classical pre-training.
Next, we plot a similar graph for using both DAPT and TAPT across our three tasks in Figure 4. From the figure, it is apparent that META-TARTAN discovers similar task-weighting strategies across different end-tasks. This suggests that the MLM objective and META-TARTAN’s strategy for learning task weights are generic enough to induce similar behaviours across tasks. In general, DAPT is significantly up-weighted compared to the end-task and TAPT. Note that the TAPT + ACL-ARC task weights (Figure 4) has the same approximate trajectory as ACL-ARC task weight in Figure 3. It seems important to assign high weight to the task data (Figure 3) but not necessarily all of it needs to go to the actual task loss (Figure 4). We hypothesize that the diversity in the domain data counteracts overfitting to the end-task data and results in DAPT being up-weighted.

heading: 6 RELATED WORK
text: Multi-task learning can be traced back to seminal work by Caruana (1995), Caruana (1997), and has since been the subject of a flourishing literature, recent surveys of which can be found in Ruder (2017) or Zhang & Yang (2021). In NLP, while initial work from Collobert & Weston (2008) already showed the benefits of multi-task learning, it has only recently become a central topic in the field, with the advent of multi-task benchmarks (Wang et al., 2018b; McCann et al., 2018).
Pre-training is where a machine learning model is first trained on a generic, data-rich task before being fine-tuned on an end-task. In NLP this practice dates back to the use of pre-trained word embeddings (Turian et al., 2010; Mikolov et al., 2013) and later pre-trained encoders (Kiros et al., 2015; Dai & Le, 2015). Peters et al. (2018) and Howard & Ruder (2018) heralded a renaissance of pre-training before BERT (Devlin et al., 2018) and its many offshoots (Liu et al., 2019; Yang et al., 2019; Lewis et al., 2019) cemented it as the de facto standard for modern NLP.
Meta-learning dates back to early work from Schmidhuber (1995); Thrun (1998). More relevant to our work is gradient-based meta-learning for solving bi-level optimization problems, first popularized by Finn et al. (2017) and followup work (Nichol et al., 2018; Rajeswaran et al., 2019) for few-shot learning. This method has transferred to a variety of applications such as architecture search (Liu et al., 2018) and model poisoning (Kurita et al., 2020).

heading: 7 CONCLUSION
text: We have advocated for a paradigm shift in the way we approach pre-training. We have motivated making pre-training more end-task aware when the end-task is known in advance. Our work introduced two novel end-task aware training algorithms: End-task Aware Training via Multitasking (MT-TARTAN) and End-task Aware Training via Meta-learning (META-TARTAN). In Section 5, we demonstrated the ability of our proposed algorithms to improve performance and dataefficiency over their end-task agnostic counterparts.
This work suggests several promising directions for future work. Instead of learning coarse task level weights, can further performance improvements be achieved via finer-grained example level weighting as in Wang et al. (2020)? Can meta-learning algorithms like META-TARTAN enable more effective utilization of previously discarded (Aroca-Ouellette & Rudzicz, 2020) pre-training auxiliary tasks like Next Sentence Prediction (NSP) (Devlin et al., 2018)? We hope this work spurs conversation around these questions and many more.

heading: 8 ACKNOWLEDGEMENTS
text: This work was supported in part by DSO National Laboratories, an ENS-CFM Data Science Chair, DARPA FA875017C0141, the National Science Foundation grants IIS1705121, IIS1838017, IIS2046613 and IIS-2112471, an Amazon Web Services Award, a Facebook Faculty Research Award, funding from Booz Allen Hamilton Inc., and a Block Center Grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.

heading: 9 ETHICS STATEMENT
text: Our work introduces new algorithms but leverages pre-existing datasets and models. Overall, this work inherits some of the risk of original work upon which it is implemented. Algorithms for continued training such as TAPT and DAPT necessitate per-task training of unsupervised objectives which result in corresponding green-house emissions due to energy consumption (Strubell et al., 2019). However, as shown in Sections 3 and 5, our new compute-efficient algorithms greatly increase the data efficiency of these algorithms, reducing these harms as well as the various harms associated with labor for data-collection (Jo & Gebru, 2020). Also, since our work is set in the context of pre-existing datasets and models (Section 4), we recognize that any ethical issues that have been revealed in these (such as bias (Bender et al., 2021) or privacy leakage (Carlini et al., 2021)) may also propagate to models trained using our work, and mitigation strategies such as Schick et al. (2021); Liang et al. (2021) may be necessary. Finally, there is a potential risk in META-TARTAN that leveraging a validation set for defining the meta-objective could amplifying bias that exists in this data split, although this is done indirectly through task weighting and hence we believe that this risk is small.

heading: 10 REPRODUCIBILITY STATEMENT
text: We pledge to release the source-code for this project to improve the ease of reproducibility of our results by the NLP and machine learning communities. In Section 4, we have specified details about datasets, training regimes and models to allow anyone who wishes to reproduce our results without our original source code to do so. Our discussion of the algorithmic and evaluation details can be found in Appendices A.1, A.3 and A.2. As we noted in 4, we build off of Gururangan et al. (2020)’s implementations which can be found at https://github.com/allenai/dont-stop-pretraining.

heading: A APPENDIX

heading: A.1 JUSTIFYING THE INTRODUCTION OF A META-HEAD
text: Proof. To arrive at Equation 7 we start with the closed form solution for ∇wiLvalT∗ (θ∗) and then introduce approximations in order to produce Equation 7. First, note that :
∂LvalT∗ (θ∗(w)) ∂wi =
( ∇θLvalT∗ (θ∗(w)) )T( ∇wiθ∗(w) ) [Chain rule] (8)
To get ∇wiθ∗(w) we invoke the Cauchy Implicit Function Theorem (IFT) as with Lorraine et al. (2020); Navon et al. (2020); Liao et al. (2018):
∇wiθ∗(w) = [ ∇2θLtotal(θ∗(w)) ]−1[ ∇wi∇θLtotal(θ∗(w)) ] [IFT]
= [ ∇2θLtotal(θ∗(w)) ]−1[ ∇wi∇θ ( w∗LT∗(θ∗(w)) + ∑ Ti∈Taux wiLTi(θ∗(w)) )]
= [ ∇2θLtotal(θ∗(w)) ]−1[ ∇θLTi(θ∗(w)) ] [Only terms with wi survive]
Bringing it all together, we get :
∂LvalT∗ (θ∗(w)) ∂wi =
( ∇θLvalT∗ (θ∗(w)) )T([ ∇2θLtotal(θ∗(w)) ]−1[ ∇θLTi(θ∗(w)) ]) (9)
Computing ∇wiLvalT∗ (θ∗) from Equation 9 is computationally unwieldy since we would not only have to optimize θ to convergence for every step of wi but we would also have to invert the Hessian of a typically large model. Our middle ground between Equations 9 and 6 (Equation 7) makes use of the following approximations:
• We approximate the inverse Hessian with the identity. This approximation is not new; we follow previous work like Lorraine et al. (2020)(Table 3) who explore the use of this approximation because of computational efficiency.[
∇2θLtotal(θ∗(w)) ]−1
= lim i→∞ i∑ j=0 ( I−∇2θLtotal(θ∗(w)) )j ≈ I
We are assuming the contribution of terms with i > 0 are negligible.
• Instead of training the whole network to convergence, at each time-step, we fix the body of the network and train a special head φ∗ to convergence on a small batch of end-task training data. We then use [θbody;φ∗] as a proxy for θ∗. This is a computationally feasible workaround to training all of θ to convergence to get a single step gradient estimate. Especially in the continued pre-training setting where a pre-trained generalist model like BERT is used as θbody, this approximation is reasonable. To our knowledge, we are the first to suggest this approximation.
∇θLvalT∗ (θ∗)→ ∇θLvalT∗ ([θbody;φ∗])
• Above, we have approximated θ∗ = [θbody;φ∗]. Since φ∗ is only used to evaluate end-task (T ∗) validation data, it means θ remains unchanged with respect to the training data for task Ti. Thus ∇θLTi([θbody; ( φ∗, . . . , φi ) ]) = ∇θLTi([θbody;φi]) = ∇θLTi(θ)
Bringing it all together, we get Equation 7, repeated here:
∂LvalT∗ (θ∗(w)) ∂wi
≈ ( ∇θLTi )T (∇θLvalT∗ ([θbody;φ∗]t))

heading: A.2 CALCULATING P-VALUES FROM PERMUTATION TEST
text: We used the permutation test (Good, 2005; Dror et al., 2018) to test for statistical significance. For each test, we generate 10000 permutations to calculate significance level. This is sufficient to converge to a stable p-value without being a computational burden. We chose this over the common student t-test because :
1. We have only 10 runs per algorithm and permutation tests are more robust at low sample size
2. Permutation test is assumption free. Student t-tests assume that the samples are normally distributed
3. Permutation test is robust to variance in the samples, so even though error-bars can overlap, we still establish significant differences in the samples. Variance in our results is expected due to small dataset sizes of end-tasks.

heading: A.3 ALGORITHM FOR META-TARTAN
text: Algorithm 1: End-task Aware Training via Meta-learning (META-TARTAN) Require: T ∗,Taux: End-task, Set of auxiliary pre-training tasks Require: η, β1, β2: Step size hyper-parameters Initialize :
Pre-trained RoBERTa as shared network body, θbody Task weightings: w∗, wi = 1|Taux|+1
Randomly initialize : end-task head as φ′ meta head for end-task as φ∗ task head, φi, for each Ti ∈ Taux while not done do B∗tr ∼ T ∗train // Sample a batch from end-task
g∗θ , g ∗ φ ← [ ∇θ,∇φ′ ]( LT∗(θ, φ′, B∗tr) ) // Get end-task grads
giθ, g i φ ← [ ∇θ,∇φi ]( LTi(θ, φi, Bi) ) // Get task grads. ∀i ∈ [n], Bi ∼ Ti
// Learn a new meta head φ∗ ← estimate meta head(B∗tr, β2, θ, φ∗) // B∗tr ∼ T ∗train g∗meta ← ∇θLT∗(θ, φ∗, B∗val) // B∗val ∼ T ∗val // Update task weightings w∗ ← w∗ + η cos(g∗meta, g∗θ) wi ← wi + η cos(g∗meta, giθ) // Update task parameters α∗, α1, . . . , α|Taux| = softmax(w ∗, w1, . . . , w|Taux|)
Update θbody ← θbody − β1 ( α∗g∗θ + ∑ i αig i θ ) Update ( φi ← φi − β2giφ ) , ( φ′ ← φ′ − β2g∗φ
) end Result : θ, φ′
A.4 VISION EXPERIMENTS
We validate that the gains from end-task Aware Training are not siloed to only learning from text. We conduct an experiment comparing end-task aware training on images to its end-task agnostic variant. We use the Cifar100 dataset (Krizhevsky et al., 2009). We use the Medium-Sized Mammals superclass (one of the 20 coarse labels) as our main task whilst the other 19 super classes are used as auxiliary data. Our primary task is thus a 5-way classification task of images different types of
medium-sized mammals whilst whilst the remaining 95 classes are grouped into a single auxiliary task.
As can be seen from Table 5, being end-task aware improves over task agnostic pre-training. We find that, again, when our auxiliary task consist of solely domain data and no task data, METATARTAN performs better than MT-TARTAN (as measured by averaged performance).

heading: A.5 FULL TAPT TABLE WITH SIGNIFICANCE LEVELS
text: We repeat Table 1 and provide details about levels of statistical signifance.

heading: Task TAPT MT-TARTAN p−values META-TARTAN p−values

heading: Task TAPT META-TARTAN p−values

heading: A.6 FULL DAPT/DAPT+TAPT TABLE
text: We repeat Table 3 and provide details about levels of statistical signifance.

heading: A.7 FAQ
text: 1. What settings are TARTAN algorithms designed for? TARTAN algorithms specialize auxiliary objectives to a particular end-task. This comes at a risk of losing the generic representations afforded by generalist pre-trained models. Thus if a practitioner has a sufficiently important end-task where obtaining improved end-task performance is paramount over generic representations, then TARTAN is a viable option.
2. When do we get computational savings from META-TARTAN? MT-TARTAN does not add any extra overhead compared to pre-train then fine-tune approaches. META-TARTAN however, adds extra overhead per gradient descent step due to computing meta-gradients. However, as shown in Section 5 we are able to get several orders of magnitude improvement in data-efficiency from applying the method. In general,
for the tasks we experimented with, we find that the savings in data-efficiency superseded the extra per-timestep meta-learning overhead.
3. When should we use META-TARTAN over MT-TARTAN? In +TAPT settings (Tables 1, 3), we observe that META-TARTAN and MT-TARTAN perform similarly. We attribute this to the strength of TAPT-MLM objective. We were pleasantly surprised that the two methods performed comparatively in this setting but in hindsight, we appreciate the insight that went into designing TAPT-MLM as an objective which makes it a strong baseline. In other settings with less carefully designed auxiliary objectives and data (which can potentially be detrimental to the end-task) we expect METATARTAN to perform better. Section 5.3 provides evidence of this.

Review summaries:
This paper considers the common setup of pretraining and finetuning paradigm and argues that an end-task aware setting, either by multitasking or an additional meta learning that learns the weights between various auxiliary tasks and the end task, is superior than end-task agnostic pretraining.  The paper provides formal formulations of the various set-ups. Propose an interesting strategy for the meta-learning setup (Section 3.3).

In the paper the authors propose TARTAN, methods to enable end-task aware pre-training. MT-TARTAN simply combines the pre-training objectives and the end-task objective as multi-task learning. META-TARTAN learns a set of weights for the pre-training objective and the end-task objective, using meta-learning. MT-TARTAN and META-TARTAN shows improved performance and data efficiency in a set of three low-resource text classification tasks. 

The paper makes the argument that generic pre-training (on auxiliary tasks) is inferior to task-specific pre-training. The authors argue that the final task should be learned together with the auxiliary tasks in a multi-task setup, which they call MT-TARTAN. They also propose a meta-learning algorithm META-TARTAN that uses meta-learning to mitigate the potential impact of updates from auxiliary tasks that detract from the main task.  The authors back this up through experiments. They consider three tasks on CS and Biomed papers, where they train 1) the main task on top of a RoBERTa checkpoint 2) the main task on top of a pre-trained model from Gururangan et al. (2020) and 3) the main task mixed-in with auxiliary tasks from (2) on top of a RoBERTa checkpoint. They observe that their approaches (3) outperform (1) and (2), both in terms of model accuracy and data efficiency. In a setting where the auxiliary tasks are potentially more noisy, they also observe an advantage of META-TARTAN over MT-TARTAN.  Overall their work heavily references Gururangan et al. (2020), who show that adapting a generic model to the task domain through more task-specific pre-training can improve model performance. They make it directly comparable by using their models and a subset of tasks considered in that paper.