AANG : Automating Auxiliary Learning
Authors: Lucio M. Dery
Paul Michel
Mikhail Khodak
Graham Neubig
Ameet Talwalkar
Venue: ICLR 2023 notable top 25% (2023)
Keywords: auxiliary learning
automl
natural language processing
meta-learning
algorithmic stability
multitask learning

Abstract: Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whilst much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand-design. Intuition for how and when these objectives improve end-task performance has also had limited theoretical backing. In this work, we present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. Next, we theoretically formalize widely-held intuitions about how auxiliary learning improves generalization on the end-task. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task.
With natural language processing (NLP) as our domain of study, we demonstrate that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP end-tasks.

Published as a conference paper at ICLR 2023
AANG: AUTOMATING AUXILIARY LEARNING
Lucio M. Dery'* Paul Michel Mikhail Khodak!
Graham Neubig 1
Ameet Talwalkar
Carnegie Mellon University
ENS PSL University
Hewlett Packard Enterprise
ABSTRACT
Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whilst much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand- design. Intuition for how and when these objectives improve end-task performance has also had limited theoretical backing. In this work, we present an approach for automatically generating a suite auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. Next, theoretically formalize widely-held intuitions about how auxiliary learning improves generalization on the end-task. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task. With natural language processing (NLP) as our domain of study, we demonstrate that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP tasks
INTRODUCTION
The auxiliary learning paradigm, where augment primary objective with extra learning signals boost end-task performance, is a staple of many machine learning (ML) domains. In natural language processing (NLP), well known models like SpanBERT (Joshi al., 2020) and RoBERTa (Liu et al. 2019b) are trained on
Objective
Data (D)
Transform (?)
Represcutation (R)
Output (C)
BERT
Out-of-dommin
BERT-Op
Bidirectional
Denoise Token
TAPT
Task data
DERT-OP
Didirectional
Denoise Token
DAPT
In-lomain
BERT-OP
Ridirectional
Donaioo Trdken
REMO
Out-of-domsin
No-Op
Laft-to-Right and Right to Lefft
Next Tiken
GPT
Out-of-domain
No-Op
Left-To- Right
Next Tiken
NL.Net
Out-of-domain
No-Op
Random factorized
Next Token
Electea
Neural LM Data
Replace
Hidirectional
Real Syathooii
We present the decomposition of some auxiliary masked language modelling objectives in NLP within our framework.
Figure 1:
(MLM) auxiliary objectives (Devlin al., 2018) before fine-tuning on the end-task. And for speech processing and reinforcement learning (RL), Oord et al. (2018) introduced the popular contrastive predictive coding objective which achieved state of the performance in many settings when multi-tasked with the end-task. Despite these successes and many more, research into devising such objectives has progressed in very local, et al.] 2020 Grill et al.) 20200 Chen et al. (2020). Auxiliary objectives are constructed by hand-design and without much overarching structure, relying on the experience and intuition of a select group of researchers versed at making appropriate design choices. Unfortunately, this status-quo not only creates a technical barrier of entry for exploring auxiliary objectives in new domains but also, by virtue of its incremental nature, limits the rate at which new objectives are discovered and investigated.
manner (Raffel et al.) 2019 Clark
To address the above challenges, this paper presents framework for automatically generating and utilizing a large set of candidate auxiliary objectives. Our framework is seeded by the following key observation: leading auxiliary objectives across multiple domains can be viewed as making different design decisions within 4 stage pipeline: Input Data (D) + Input Transformation (T) +
"Corresonndence to lddery@andrew.cmm..cci
Code available http://gthicccoolleyyy//mmmmmnit

---

Published as a conference paper at ICLR 2023
Model Representation (R) - Output (O), For instance, in RL, common auxiliary objective is to predict the environment's 's forward dynamics (Agrawal et al. 2016: Hafner et al. 2019). To construct this objective, the current task state-action pair (D) 18 corrupted (7) and then passed through the
model to produce a latent representation (R) which is finally used to predict the next state (O). Similarly, in NLP, the XL.Net (Yang et al. 2019) objective-whic which performs language modelling on randomly lactorized permutation of the input-can can be written within our taxonomy as {D= O = Next Token). These two examples (along
Out-of-Domain, T No-op, R
with others listed in Figure D fall within class we term named objectives: objectives that have been previously proposed in the auxiliary learning literature.
Decomposing named objectives within our taxonomy provides a unified view of the auxiliary learning landscape. From this vantage point, it becomes clear that there are many unexplored combinations of the various primitives used across named objectives. This presents
Data (D)
Transform (7)
Representation (R)
Output (0)
Out-of-lommnn In-domain
No-Op
Bidirectional
Next Token
Replace
Left-to- Right
Real &yytth
Task data
Mask Volsing emhords
Right-to-Latt Rand. factorized
Denoise Token
Noural LM Data
TE-DF
TAPT (Taek data + BERT-OP + Bidirectional + Denoise Token) GPT (Out-of-domain No-Op Left-to-Right Next Token;
Now-Obj-- (Task data + BERT-OP - Left-lo-Right + Denoise Token)
New-Obj: (In-domain No-Op Random Factorized - TF-IDF)
simple formula for automatically generating a large set of candidate objectives: take the cartesian product of the design decisions across given stages (Figure 2 Using this compositional process,
Figure 2:
Our framework in the context of NLP. We decompose named objectives within our four staged taxonomy: {DD, T., R. O). By taking the cartesian product of choices across stages, we reproduce named objectives and discover new ones.
not only can we reconstruct existing named objectives, we can also generate new combinations. This overcomes the tedium of implementing each objective independently since we can just reuse a small set of simple stage-wise primitives.
Generating a large set of objectives raises the natural question of how to efficiently select the most helpful ones for given end task. Instead of leaving this to practitioner intuition, we develop principled guidelines to address this question by theoretically studying the impact of auxiliary learning on a particular end-task. Specifically, using arguments based on algorithmic stability (Hardt et al. 2016 Bousquet & Elisseeff 2002), we derive end-task generalization error bounds that are dependent on
the choice of auxiliary task. This contributes to existing theory (Saunshi al. 2020 Xie al. 2021 on how auxiliary learning impacts the end-task by suggesting a new candidate mechanism: auxiliary learning results in more stable optimization end-points in the sense of Bousquet & Elisseeff (2002). which in theory improves generalization of the final model.
Guided by our theory, we introduce AANG (Automating Auxiliary LeariiNG,, an efficient, structure- aware algorithm for adaptively combining set of related objectives to improve generalization on specific end-task. AANG incorporates the following prescriptions from our theory: (i) auxiliary tasks that are more similar to the end-task are desirable. Given a set of objectives, AANG learns adaptive weights to bring the composite objective closer to the end-task; (ii) general, more auxiliary data better. AANG maximizes the effective amount of data used in training by using all the generated objectives instead of taking tpsc-fpec subsets.
To empirically validate our method for automatically generating and utilizing auxiliary objectives, we experiment on five NLP tasks. We do so in the widely-used setting of continued pre- training (Gururangan et al. 2020 Aghajanyan et al. 2021 Dery et al. 2021b Zhang et al. (022), where a model trained with a single auxiliary objective on large-scale data is further trained on end-task related data. Without introducing any external data or architectural modifications, variants of AANG outperform strong and widely used baselines in 4 out of 5 tasks. AANG achieves an average improvement of 4.2% over standard fine-tuning of RoBERTa across our chosen tasks. We believe our results will spur further research into exploring automating auxiliary learning across a variety of settings. Notably, while we focus on NLP when discussing the space of auxiliary objectives (Section 3 and in our empirical evaluation (Section 6). our theoretical results (Section 4 and AANG itself are domain-agnoss
?Our ideas could be applied to domains like RI. or computer vision (CV), where similar dissection of existing objectives can be performed.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
To properly scope this work, we define auxiliary learning as training a model on alternative objectives with the goal of improving performance on some primary end-task. Auxiliary learning is an
instantiation of transfer learning (Caruana 1997 Baxter 2000, Ruder et al. 2019). It covers
the pretrain-then-Nee etrnnnn-tthe---nntuueee paradigm (Huh et al. 2016 Devlin et al. 2018: Schneider et al. 2019
Gururangan et al., 2020) as well as end-task aware multitasking approaches (Lin et al. 2019: Dery
et al. 202 b). Whilst auxiliary objectives may be meta-learned (Liu et al. [2019a] Navon et al.
2020 for simplicity since incorporating these would require further complication of our design space such objectives are out of the scope of this paper.
This work bears many parallels to the area of neural architecture search (NAS) (tanley & Mikkkllanen 2002 Zoph & Le 2016; Roberts et al. 202T). Whilst we seek to automate auxiliary learning, the objective of NAS is to automate the discovery of the right neural architecture given a specific end-task. Search spaces of candidate architectures are created by taking the cartesian product of architecture design choices across the depth of the network. The design of suitable architectural
search spaces for a variety of settings has been an active area of research (Tan & Le 2019 Howard iet al. Dao et al. 2020 Roberts et al. (021). To develop AANG, we borrow ideas from the NAS literature on efficient algorithms for sifting through spaces of architectures. Mirroring the popular differentiable NAS method DARTS Liu et al.] (2018), we perform continuous relaxation over the search space of objectives, allowing for efficient search by gradient descent. We also use a factored approach to model relationships between objectives that share primitives. This is inspired by
recent work on stochastic-relaxatio weight sharing (Dong & Yang 2019 Li et al. 2020).
As theoretical contribution, this work derives an end-task aware generalization error bound for auxiliary learning. Our bound is built on that of Hardt et al. (2016), who derive generalization bounds for parametric models trained with stochastic gradient descent (SGD). To derive their bounds, they leverage the concept of algorithmic stability introduced by |Bousquet & Elisseeff (2002). Informally, a randomized algorithm is uniformly stable if changing single training data point in the given samples does not change its end-point too much. Said change is characterized as the average difference in predictions between the two learned models. Stability implies generalization in expectation (Hardt
et al. 2016 Kuzborskij & Lampert 2018).
AUTOMATICALLY GENERATING AUXILIARY OBJECTIVES
To begin, we take a high-level view of the landscape of named objectives. Using running examples from NLP, we propose the following coarse structure for the sequence of choices made in the hand-design of auxiliary objectives:
Data, D: Auxiliary objective pipelines begin with a choice of input data. Here, options can range from heterogeneous out-of-domain data (Radford et al.] 2019). in-domain data with respect to the final end-task (Beltagy et al. (211)) or the task data itself (Gururangan et al., 2020). It may even include data outside the modality of the end-task.
T: Many auxiliary objectives are self-supervised with respect to their input data. They corrupt or transform the input and then reconstruct it in whole or part. For example, input text tokens can be masked, replaced or deleted. Operations can also be aggregated in BERT-Op: mask 80% of selected tokens and randomly replace 50% of the remaining Devlin et al. (2018); Liu et al. (2019b).
Representation, R: After transformation. representations of the input data can be computed from given model in different ways. A chosen token's 's representation can depend on only its left context (Left-to-Right (Radford et al., 2018)) or its right context (Right-to-Leff) (Peters et al.. 2018). It could also depend on the representations of a randomly selected permutation of other tokens (Random Factorized) Yang et al. (2019).
4. Output, O: Finally, representations obtained from the previous stage are fed into loss function producing a final output. The choice of output loss is usually coupled with the choice of transformation made in stage 2. Choices include but are not restricted to denoising tokens, predicting the next token or predicting the TF-IDF (Term Frequency- Inverse Document Frequency) of token.
3

---

Published as a conference paper at ICLR 2023
The above taxonomy {D T R
O} is expansive enough to cover range of named auxiliary objectives of interest in NLP (Figure For example, we can write any member of the GPT series (Radford et al.) 2018 2019 Brown et al. 2020) which perform left-to-right language modelling on out-of-domain data as (D= Out-of-Domain, 7 = =No-op, R= Left-To-Righh To-Right, O = Next Token). We summarize the pre-existing choices within each design stage obtain unique set of options. For example, we can reduce the set of model representation types used by the objectives enumerated in Figure to to the unique set R = (Bi-directional, LeighTo-Right, Right-To-Left, Having summarized the list of primitives within each stage, simple formula for generating space of auxiliary objectives becomes apparent: take the cartesian product of the design choices at each stage (see Figure 2). In general, given an instance of our taxonomy, we can construct a space of objectives A = 7 R x O of size |A| S [D] x [T] x [R] |O). Consider New Obj, from Figure This previously unexplored objective can be obtained by combining the special masking operation from BERT (BERT-Op) with computing model representations based on left-to-right causal masking as in GPT. In fact, this objective proved one of the most useful ones in our experiments below (see Figure
mandom-Factorizedt
Our framework also allows us to reason about whole families of objectives, F, by thinking in terms of design stages and choices. For example, given particular end-task E with input text Ep, we can create a family of objectives based solely on task data by fixing to that option in our input data stage; we call this family FD-EE JD=Ep Ep not only includes pre-existing TAPT Aururannan eet al. (2020) but also unexplored objectives like task-data dependent variants of XINET, ELMO etc. Auxiliary learning with FD=E> can be seen as relaxed form of data augmentation which we dub task augmentation. Whilst data augmentation requires applying transformations that preserve the data-point's 's label, task augmentation has no such restriction and thus offers greater flexibility in terms of specifying (T, R. O}. We can also reason about expanding particular stages to include new primitives. Any supervised loss can be added to the output stage, O allowing us to potentially explore auxiliary objectives based on supervised signals like NER or POS tagging (Carreras et al.) 2003] Charniak] 1997p. A special example setting O to the end-task supervised output Eg. This FD=EB
leads to Er which is a subset of FD=ED =O=Eo includes many objectives like predicting the end-task signal from corrupted input data. In Section 6 we will introduce search space of objectives that leverages task augmentation.
4 THE IMPACT OF AUXILIARY LEARNING ON END-TASK GENERALIZATION
In this section, we relieve reliance on practitioner intuition by deriving a set of guiding principles on how to effectively utilize the automatically generated objectives from Section
Auxiliary learning influences the end-task through both training and generalization error. Previous theory has largely focused on characterizing the impact on end-task training error. Liu et al. (02T)) for example, show that end-task agnostic pre-training can create a performance gap in training error compared to training with the end-task alone. The size of this gap depends on how dissimilar the pre-training auxiliary objective is from the end-task. They introduce the following assumption (which we will borrow) to formalize their notion of task similarity:
Assumption A.1: Let fe represent the end-task objective and fo be the auxiliary objective. There exists A 2 0 such that f. (0) V (0)] < A Y 0.
Note that 9 represents all the parameters of the model. Smaller A implies fa is more similar to the primary task fe. Liu et al. 202T bound the end-task agnostic training error gap to be logarithmic in A.
Unlike training error, end-task generalization eтror has gone unstudied in the auxiliary learning setting. Bounding the generalization error not only adds
theoretical understanding of the impact of auxiliary learning but also provides insights to guide algorithm design. To arrive at a bound, we adapt the technique Hardt et al. (016)) who derive a generalization bound on training with only the end-task via stochastic gradient descent. We consider the end-task aware setting where the end-task multi-tasked with the auxiliary objective. This setting has recently been shown to improve end performance over the pretrain- pretrain-then-fee
paradigm (Dery et al. 2021a Yao et al. (021).
Auxiliary learning with Dynamic Sampling: We are given an auxiliary objective fa(::2) E [0. 1] with N₂ samples S, = (zi...., )N.) from the distribution D.. fa can either be single objective or
3 Athoughh this taxonomy is quite expansive, it obviously does not consider other elements of objective creation such as choice of model architecture, optimizer settings, etc.
4

---

Published as a conference paper at ICLR 2023
weighted linear combination of objectives: fa f== Ex w fa. At any iteration of SGD, sample a choice of the end-task function fe or the auxiliary objective fo according to the probabilities Xe- Aa E [0, 1] Åe Aa 1. Given the chosen objective, sample data-point and perform stochastic gradient descent based on the sampled data-point, We now present our bound in the setring described.
Theorem 4.1 (Auxiliary learning with Dynamic Sampling). Assume that fe (e)) fal; (122z))) za) E [0.1] are both L-Lipschitz with Be and Sm-shooth loss functions respectively. Consider that have N' = Ne + Na total samples where Je and fa have Ne and No samples respectively. re No is the fraction of the available data represented by the end-task. Suppose that we stochastic gradient descent for T steps with monotonically non-increasing step sizes Or S & by dynamically sampling the tasks according de and Xa. Then, with respect to Sex the generalization error is bounded by:
?
Egen
(A) 1-)
Where
(1)
N
Te
Here 3r = min( 3o) and X the weighting of the fanction with smaller smoothness.
Proof. See Appendix D for full proof and Appendix F for more discussion
As detailed inspection of the proof will show, we derive Equation [ by appealing to algorithmic
stability (Bousquet & Elisseeff 2002, Hardt et al., 2016 Kuzborskij & Lampert 2018) (Section
2) To our knowledge, ours is the first work to present an algorithmic stability view to formally explain how auxiliary learning influences end-task performance. Equation [uufaaess the following prescriptions about learning with auxiliary tasks
P1 Smaller A improves Egen. This implies that the more similar the auxiliary objective is to the end-task (under Assumption A. the lower the generalization error.
P2 Larger N' leads to smaller Egen Since usually have a fixed amount of task data No. we can increase N' by adding more auxiliary data Na.
5 END-TASK AWARE SEARCH OF STRUCTURED OBJECTIVE SPACES
Guided by Section 4 we build a practical
Algorithm AANG
method for exploring a set of objectives, A.
Input: Search Space A Factor vectors- - (W-AIl W WT, W wo) End-task E, End-task weight- Ae
Whilst the dynamic sampling setting described in Section 4 is amenable to theoretical consideration, we make few practical changes to it. First, instead of performing alternating gradient descent by sampling fa according to Xe, Aa we instead use them multitask weights and perform joint training. Joint training has been found produce superior results compared to alternating optimization when leveraging auxiliary objectives (Aghajanyan et al. 2021I. We perform gradient descent on the following total loss which interpolates between the end-task and the auxiliary loss Liotal XcLE (1 Xe) Here, K. is
Initial Model Params 0, R D
repeat
Sample batch of n objectives KC" ~ A
Weighting of objectives in K"
Construct w"
for to do (d, L, o)
[CT] stages
w² xexp (WAIl WW7 W +W7 +0R
w + w
end for
Get losses from batches of data LA(K"" wx) = En wL
Lootal AcLE + (1-A1)) (1-Ae)
a chosen subset of A.
Get gradients and update factors
Second, as indicated in Section 4 given KG, we can write the set as single objective Sa = ΣkEk w= fk. By Prescription we want to choose { such that fa has a small A with the end-task fe- We would also like to set >x such that the bound on égon is minimized. Whilst a closed form exists for the optimal weightings de {w* it depends on variables like { (8+), L that are hard to estimate.
0141, WV
META-TARTAN(O, E, Liota))
Update (W-ll, WF,w", W WO using W Update d. using Vx.
until done
Return: 0r
"This holds fixed y which we achieve hy adjusting de account for introducing more auxiliary data.
5

---

Published as a conference paper at ICLR 2023
We therefore propose to learn λe. {w*] in online, data-driven way. To do this, we build on top of the META-TAREE TARTAN algorithm proposed by Dery et al. (202lb). META-TARTHN meta-learning algorithm that learns adaptive weights for different auxiliary tasks in a way that prioritizes end-task generalization. It learns ) by minimizing the loss on the end-task validation
is a
set: Fry - -(V )" (VoLg.). This corresponds to learning {w²} such that (Vefa) (Vofe) is maximized. This minimizes one of the terms that contributes to A and thus attempts to fulfil Prescription PI] We can similarly learn de to minimize the end-task validation loss. For a more detailed discussion of META-TARTAN please see Appendix [B]
2CB
So far, we have introduced independent weights, fw"), for each objective. This is sufficient in the case of unrelated objectives. However, the objectives in A share underlying structure. We recognize this by using factored approach to model each w² We introduce a factor vector for each of the 4 stages introduced in Section 3 W E R Pi WT E R R/7,, W E R [ and W E R R This ties together the weights of objectives that share primitives in common. To capture the fact that an objective can be more than the sum of it parts, we also introduce an independent weight for each objective WAlll E R|P|x|/|X [R|x|O|, Consider the objective k which is generated by T, R, E O), its weighting is computed as
the composition of the operations {d E D, : w x exp (W (d.t.r.o) + W2 + W.T + W + Wo). Our factored approach not only allows us to share information between objectives but it also allows to analyze which stages and primitives are most important to a particular end-task after training is completed (Section 77
Prescription P2 from Section 4 advocates for introducing as much auxiliary data possible. As such, instead of fixing to a specific subset throughout training for a particular end-task, we propose to utilize all the objectives in A. This also avoids the combinatorial explosion that comes with exploring subsets of A at time. | can be large and descending on all of A at once can be computationally prohibitive, As an efficient work around, at each training step, we sample a subset of A for execution with META-TARTAN. Our samples are drawn from all of A so any objective can get used at any timestep. Because we model each = via a factored approach, even if an objective not sampled its weight is implicitly updated. Our approach is reminiscent of
weight sharing (Pham al., 2018: Dong & Yang. 2019 [i et al., 2020) where sampled architectural primitives result in updates to shared model weights which can be used by other primitives that are not sampled.
We coalesce all the ideas we have introduced so far into Algorithm which we dub AANG (Automated Auxiliary LearniNG). At high-level, given an end-task E:
1. We generate space of auxiliary objectives A by leveraging the taxonomy discussed in Section 3 A may contain auxiliary tasks that can improve our performance on E.
2. We leverage MAML-style (Finn al. (017) meta-learning to adaptively weight the objectives in A based on measuring each objective's 's influence on E's s validation set loss.
3. We make our algorithm scalable by sub-sampling the tasks A. By exploiting the underlying structure of the objectives in A via a factored approach to modeling task weights, we reduce the impact of the inexact sub-sampling
6 EXPERIMENTAL SETTING
Our exploration of auxiliary learning has made the following transitions from the status-quo: manual to automated, single task to multitask, end-task agnostic to end-task aware. In this section, we set up experiments to validate these deviations from the standard.
We focus on continued pre-training (Gururangan et al. 2020 Aghajanyan et al. 202T). In this setting, we perform further auxiliary learning on an already pre- trained model. We favor this setting over pre-training from scratch (Liu et al., 2019b; Yang et al. 2019) not only because it is a more computationally feasible arena for experimentation but also because is more relevant to modern ML systems where building upon pre-trained models is the norm (Qiu et al. 2020 Du et al.) 2020). Model Details and Datasets: We use a pre-trained RoBERTapass (Liu et al. 2019b) as the shared model base. We implement each auxiliary objective as a separate head on top of this shared base. For classification based objectives, the output head is a 2-layer multi-layer perceptron (MLP) that receives representations for the special classification token [CLS] (Devlin et al. 2018) from RoBERTapaee: For sequence generation objectives, we make a copy of the pre- trairrd output layer of RoBERTa for each task. Table 4 in Appendix C provides details of the 5 datasets used.

---

Published as a conference paper at ICLR 2023
All datasets are low-resource classification tasks. Not only are these datasets more amenable to meta-learning from a computational standpoint, but low-resource tasks also benefit the most from auxiliary learning. We also choose these tasks because they feature in previous work which use as baselines (Gururangan et al., 2020) Dery et al. 2021b)
Baselines and Search Spaces: The following methods are end-task agnostic baselines. By end-task agnostic, we mean that these do not multitask with the end-task. Finetuning on the end-task occurs after training on the auxiliary objective.
1. RoBERTa (Liu et al., 2019b): We simply finetune a pre-trained RoBERTabase on the end-task. TAPT (Gururangan et al., 2020): Continue training RoBERTanas on masked language modelling on end-tast task data itself before finetuning on the end-task.
The following named objectives are end-task aware baselines that use META-TARTAN (Dery et al. 2021b) but utilize only auxiliary task. Each auxiliary objective is multi-tasked with the end-task.
1. GPT-style: We perform end-task aware training with a denoising auxiliary objective based on left-to-right causal masking for computing representations. {I End-task data, T No-op, R=
I Left-Tto O = Denoise Token
XLNET-style: This a denoising auxiliary objective that uses randomized masking for computing representations.. {I = End-task data, T = No-op, R = Random-factorized, O Denoise Token]. 3. BERT-style TAPT: Denoising inputs corrupted via BERT-Op: 80% masking and 10% random replacement. {I End-task data, T BERT-Op, R Bi-directional, O Denoise Token). Please note that this baseline is equivalent to META-TATEE TARTAN as introduced in Dery et al. (2021b).
Table T details the search spaces that we evaluate against the above baselines. This is by means the most encompassing search space but we leave more expansive space design to future work. Please note that all tasks within AANG-TD, and those with {I End-task] in AANG-TD+ED, are instantiations of task augmentation as introduced in Section 3
Table 1: AANG-TD (task data) has 24 objectives and based on only end-task data. AANG-TD+ED (task data + external data) has 40 objectives and uses both end-task and in-domain data.
T
R
TD
End-tusk
BERT-op
Bi-directional Left-to Right
Denoise Token
Musk
End-task
TD+ED
End-task
Replace
Right to Left
In-Domain data
No-op
Random Random-Factorian
Training Details: : Please see Appendix D for more details about hyper-paramett configurations.
7 RESULTS AND DISCUSSION
In this section, we experimentally validate our case for automating the creation of auxiliary objectives and using them in an end-task aware multitask fashion.
7.1 GOING A LONG WAY WITHOUT EXTERNAL DATA
We first consider the setting where we rely solely on end-task data (task augmentation), and work with the AANG-TD search space. This search space has 24 objectives. Table that automatically generating auxiliary objectives from only task data and using them appropriately is productive. End-task awareness is key: From Table 2 methods that are end-task aware result in over 1.12% average improvement over those that are end-task agnostic even under the most generous comparison (GPT-style 79.84% task-agnostic TAPT 78.72%). Knowing the end-task means that at each iteration, AANG can make informed gradient updates by adapting task weights so the resulting auxiliary task better aligns with the end-task (Prescription (P]). Amongst the single task objectives, BERT-style performs best. We posit that this is because RoBERTa was trained from scratch on a similar objective and this objective represents minimal shift in training distributions.
Adaptive multi-task auxiliary learning improves performance: We compare single-task end-task aware auxiliary learning to its multitask variant. Table 2 shows that multitasking our different types of language modelling tasks results in improved average performance over using the tasks individually (8 12% for the BERT-style and 81.55% for combining the three single task objectives). We get our best performance when we multitask 24 auxiliary objectives automatically generated with our framework using AANG-TD. Boosting the number of objectives from to 24 resulted in a 0.66% improvement in average performance across tasks. This is in line with Prescription P2 from Section since we are increasing the effective amount of auxiliary data. We further posit that introducing more auxiliary objectives also serves to implicitly regularize the end-task during training.
7

---

Published as a conference paper at ICLR 2023
Table 2: Our framework and AANG on tasks using only task data. Without using any external data, we are able to get significant average performance improvement over baselines. Superscripts are p-values from paired I-tests (best multitask versus best single-task).
Task Adaptive
Mothod
CS
BIOMED
NEWS
STANCE
ACL-ARC
SCIERC
CHEMPROT
I.PARTISAN
SE-2016-6
AVG
RoBERTa TAPT
66 083
77. 77.96ig.s6
82.10g
83.39k
70.37..51
77.97
79. 13
82.17a.6
83
70.74;
78. 72
[OURS] Static Muliiask-TD
60
83.37o.5
83.12
97. 97.95g.7
71.022,.4
Yes
X. GPT-style
67 222.44
81 81.622.84
83.29,,a1
96.41o.
70.87
79.94
Y. XLNET-avle
69.76x.
81.81o x
83,39
96.41
71.18p
80.51
Z. BERT-style (Dery 2021b
70 088amu
81. $1.448
84,49%,
(0000)
96,84
72.70,,
81.12
[OURS] AANG-[X-Y1Z]
71 71.51
82.89g
83.68
96-92
72.77
81.55
[OURS] AANG-TD
24
73.26)
(0:25)
82.388
(0.27)
83.9
98.46%
(0.14)
72.46
82 21
7.2 INTRODUCING EXTERNAL DATA
Results on ACL-ARC with external
76
For the ACL-ARC task, we experiment with introducing auxiliary tasks based on external data. AANG-TD+ED has 40 tasks, 16 of which are based on domain data. We introduce CS domain data (from the S2ORC dataset (Lo) et al. 2019) that is n = 10x the size of the task data. From Figure 3 we see that AANG-TD+ED makes better use of domain-data than doing end-task aware training using only
74
(0.129)
72
Average
70
68
BERT-style objective with task (TAPT) and domain-data (DAPT) jointly as in Dery et al. (2021b). However, AANG-TD+ED (73.70) does not sigmificanlly improve over AANG-TD (73.26) on the ACL-ARC task (Figure (3). This might seem odds with Prescription P2 since the TD+ED search space introduces more data. However, note that the AANG search algorithm is approximate and as such, with a larger search space, it can be harder find composite tasks with a small A as suggested by Prescription PT We posit that we need more external data than see marked improvements to offset our inexact search of the space of composite functions. However, such scales are outside our computational budget.
(0.998)
64
62
AANG-TD+ED
DAPT TAPT [Dery et al, 2021b] Finetune RoBERTa
Figure 3: AANG effectively leverages out-of-task data. P-values (in brackets) are comparisons to (Dery et al. 2021b)
10x order to
7.3 WHY DOES AANG WORK? ?
To better understand why our auxiliary learning pipeline improves end-task performance, we perform multiple ablations under AANG-TD.
Static versus Dynamic Weighting: We ablate the impact of using static task weights throughout training, as against adaptive task weights. Just as with AANG, we sub-sample tasks from the search space at every iteration (n is cross-validated exactly as AANG tasks weight is initialized to and this remains unchanged throughout training. This is the Static Multitask-T TD baseline in Tabig2] AANG-TD improves upon the static multitask bascline by over 1.1% average. With adaptive weighting, AANG down- objectives that are harmful to the end-task whilst up-weighting relevant ones (Prescription PT]. However, using static weightings is more compute friendly since we do not have calculate taighteight meta-gradee cetta-graiienss. This compute- vs-performance trade-off is left for practitioners to resolve based on their available resources.
Table DD). Each sampled
Impact of number of sampled objectives: Due to computational constraints, A AANG sub-samples the set of generated objectives. Whilst this sampling can result in approximation error when inferring task weightings, can also introduce stochasticity which can help regularize the learned model. From Table (Appendixx A) we find that for some tasks (ACL-ARC and SCIERC) sampling larger number of tasks helps. 20-6016- and CHEMPRÖT on the other hand benefit from smaller number of sampled tasks. Our recommendation is that the number of sampled tasks be cross-validated on a per-task basis. Learned task weight trajectories: AANG learns interesting trajectories for weighting design stage primitives. From Table 2 the fact that AANG-TD roughly matches the best single task performance (72.46 versus 72.70g.60 for BERT-style) on the SE-2016-6 task suggests that it may be learning to mostly up- up-weight this task. Figure 4 provides evidence of this. For the SE-2016-6 task (row 1), composing the highest weighted primitive from each stage [BERT o None o DENOISE] results in BERT-style, the best single task objective. Figure 4 also shows that AANG adapt to overfitting.
8

---

Published as a conference paper at ICLR 2023
5E-201666
1.05
O- 68-20164
BEAT
Av
0.75
1.00
fles
0.50
To
BENOSE 50-20064
0.25
0.25
0.35
400
600
800
0.00
250
800
200
400
400 Itoratiion
600
800
toration
Iteration
Reration
1.05
ERC
0.6
SCIERC
0.25
1.00
Nome
0.20
Mire
SCIERC
0.22
Tigh-TT-uott
0.95
0.15
1,000
1500
0
500
10000
1500
500
1000
1500
500
1000
1500
Figure 4: Learned trajectories for AANG-TD for run instances of SE-2016-6 and SCIERC tasks.
The vertical black lines indicate the point of best validation set performance. AANG responds to over-fitting by down-weighting objectives based on the output loss being over-fit Thus, after several iterations, the objective that dominates when the validation performance is at its highest (black vertical line) gets down-weighted in response to it becoming saturated.
What tasks are important and when they are important? We study which tasks are most highly weighted early in training (first 10% of learning trajectory) and later in training (last 50%). We aggregate statistics across 3 datasets. Note that early in training, objectives based on the self-
Tasks highest average weight during first training
highest average weight during training
R
None
LanhTh-Right
Dencien
Mask
RandiFool
Tosk
Ber--p
Mask
Mask
Lath---llihht
Lafh-T--light
Nona
Denciss
Danciss
Dancien
Replace
Replace
Bert-ap
Rand-Fact
Rand-Fact
None
Task
Tesk
Nons
Rand-Fact
Mask
Bert-op
Nona
Danciss
Replaca
Bant-Fact
Danciss
None
Rand-Fact
Danciss
Replace
Letl-To-llght
Dancise
Rand-Foot
Denotsc
Reptace
Band-Fact
Task
LefhTo-Right
Replace
Right Tipht-ToLLatt
Dancise
Rand-Fact
Denotse
Bertoo
Bettop
Right---LLatt
Rand-Fact
Denoise
Danciss
None
Nons
Right-TT-Lett
Lan-Te-Right
Denatee
Denates
Bertap Rand-Fact
Tusk
Repisce
Righe-To-Lot
Denolse
Mask None
Task
Bert-op
None
Denalse
Mask
Rand-Fact
None
Rand-Fiu
Danobe
0.0
Prastion Plana
Figure 5: Top ranked objectives (averaged weight) early in training (left) and later in training (right)
supervised output O {DENOISE) are highly weighted but later, objectives based on supervised signal, O [Task} play a larger role. AANG rediscovers the common practice of training on self-supervised objectives before introducing supervised ones. It also interesting to note that many newly generated objectives (outside of the 3 named single task baselines in T2 such as simple input reconstruction were discovered have relevant impact on end-tasks. This means AANG can automatically surface new, previously unexplored objectives relevant the end-task.
LIMITATIONS AND CONCLUSION
Our work has some limitations that leave for future work. First, because AANG relies meta-learning, it presents extra compute burden over simple multitasking. This is because, we have to independently compute meta-gradients for each auxiliary task thus requiring (n)) forward-backwardd
operations for n sampled tasks compared to (1) for static multitasking. In Table we show that static Multitask-TV method outperforms all other ton-ttas--aappit methods by 2.4% and is thus viable alternative when runtime is a signficant constraint. Secondly, AANG as presented an approximate algorithm primarily due sub-sampling the space of tasks. Thus mentioned in Section 7.2 we do not get as much gain as desired when our search space becomes larger. We leave finding an efficient exact search algorithm for future exploration.
This paper presents a procedure for automating the creation of auxiliary objectives. We showed, theoretically, how auxiliary learning impacts end-task generalization. This resulted prescriptions that informed the design of AANG, an algorithm to search the space of generated objectives in an end-task aware multitask fashion. Our experiments show that AANG is promising first step in automating auxiliary learning.

---

Published as a conference paper at ICLR 2023
9 ACKNOWLEDGEMENTS
This work was supported in part by DSO National Laboratories, an ENS-CFM Data Science Chair, DARPA FA875017C0141, the National Science Foundation grants IIS17011 IIS 1881888017. ISS2046613 13 and ISS-2112471 an Amazon Web Services Award, a Facebook Faculty Research Award, funding from Booz Allen Hamilton Inc., and a Block Center Grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies. We are grateful for helpful feedback from Uri Alon, Patrick Fernandes, Joon Sik Kim, Han Guo, Victor Akinwande and Clara Na.
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. arXiv preprint arXiv:2101.1103: 1038, 2021.
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Beltagy, Arman Cohan, and Kyle Lo. Scibert: Pretrained context enw.e// for scientific text. CoRR, abs/1903. 10676, 2019. UBL.ittte//arrvvvvorrr//b/// / 10676
Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, ftss/2005.14166 2020. URL httpsst::: /ttp://aaiivvi.o org abs 2005. 141 14165
Xavier Carreras, Lluís Märquez, and Lluís Padró. A simple named entity extractor using adaboost. In Proceedings of the seventh conference on Natural language learning HLT-NAACI. 2003, pp. 152-155, 2003.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Eugene Charniak. Statistical techniques for natural language parsing. Al magazine, 18(4):33-33 1997.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1697-1607. PMLR, 2020.
Kevin Clark, Minh- Thah-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Tri Dao, Nimit S Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher Ré. Kaleidoscope: An efficient, learnalle representation for all structured linear maps. arXiv preprint arXiv:2012.14966) 2020.
Lucio M Dery, Yann Dauphin, and David Grangier. Auxiliary task update decomposition: The good, the bad and the neutral. arXiv preprint 108.v:2108./1344 2021a.
Lucio M Dery, Paul Michel, Ameet Talwalkar, and Graham Neubig. Should we be pre-trate pre-training? an argument for end-task aware training alternative. arXiv preprint arXiv:2109.07433 2021b.
10

---

Published as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480. 2018.
Ming-Wei
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1761-1770, 2019.
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau, Self-training improves pre-training for natural language understanding. arXiv preprint arXiv:2010.02194, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agmosticc meta-learing for fast adaptation of deep networks, 2017. URL http::://aa......g////a///////
03400
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-212844 2020.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004. 109::4 2020.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning. pp. 2555-2565, PMLR, 2019.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. International Conference on Machine Learning, pp. 1225-1234. PMLR, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2019.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608 08614, 2016.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. Measuring the evolution of scientific field through citation frames. Transactions of the Association for Computational Linguisticy, 6:391-406 2018
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. SemEval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop Semantic Evaluation, pp. 829-839, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/ S19-2145. URL https:::///// httptppss//cccaaccoooooo.o..////
Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. Chemprot-3.0: global chemical biology diseases mapping. Database, 2016, 2016.
IIja Kuzborskij and Christoph Lampert. Data-dependent stability of stochastic gradient descent. In International Conference Machine Learning, pp. 2824. PMLR, 2018.
Liam Li, Mikhail Khodak, algorithms for neural architecture search. arXiv preprint arXiv:2004007802, 2020.
Balcan, and Ameet Talwalkar. Geometry- aware gradient
Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2023
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, arXiv preprint arXiv:1806.0900 2018.
Shikun Liu, Andrew Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning, arXiv preprint arXiv:1901.068 190100833,, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1699, 2019b.
Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Antoni B. Chan, and Rong Jin. Improved fine-tuning by leveraging pre-training data: Theory and practice. CoRR, abs/2111.12292, 2021. URL
abs/2111. 12292
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld. S2ore: The semantic scholar open research corpus. arXiv preprint 1rXXiv://1.0077600 2019.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017. URL https
//arxiv. abs http://b//7111.1011111
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv:1808.09602. 2018.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. SemEval- 2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 31-41, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 11855//11/11-11110
htL/https://ast httptpsss//aallaahllollog///////
org/S16 1003
Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. arXiv preprint arXiv:2007.02693 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:180..03744 2018.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. CoRR, abs/1802..0535, 2018. URL
http: wrg. abs /ttbb////22.000 053665
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095-4104 PMLR. 2018.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pp- 1-26, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre- treitrannng. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arxiv: 10683, 2019.
Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Ré, and Ameet Talwalkar. Rethinking neural operations for diverse tasks. arXiv preprint arXiv:2103.15798 2021.
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Tutorials, pp. 15-18, 2019.
12

---

Published as a conference paper at ICLR 2023
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03668, 2020.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv: 05862, 2019.
Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114, 14. PMLR, 2019.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. arXiv preprint wwxiv:22/1.0000 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NIp from scratch without large-scale pretraining: A simple and efficient framework. arXiv preprint arXiv: 2021.
Tong Zhang, Peng Gao, Hao Dong, Yin Zhuang, Guanqun Wang, Wei Zhang, and He Chen. Consecutive pretraining: A knowledge transfer learning strategy with relevant unlabeled data for remote sensing domain. arXiv preprint urXiv:2207.33860 2022.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1.0055,, 2016.
13

---

Published as a conference paper at ICLR 2023
A
MORE ABLATION TABLES
Table Varying number sampled objectives per-iteration.
Task
tasks
6 tasks
ACL.ARC
72.112.12
73. 73.261.32
SCIERC
82.351.76
82.981.52
SF-2016-6
72.461.65
72.460.90
CHEMPROT
83.91q.32
83.699.98
H.PARTISAN
98. 08.46p.0
97.95p.73
B DISCUSSION OF META-TARTAN (DERY ET AL.,
2021B
TARTANTARANN (Dery et al. 202 is MAML style (Finn et al. 2017) meta-learning algorithm
that learns to adaptively weight a given set of tasks based on their influence on the end-task validation performance. META-TARTAN achieves this by formulating the following bi-level optimization problem:
0*, w argmin s(%)), w) LE(
where
0g argming (0, w) argmin, w0 LO) +
Σ TRA
WiCT, (0)
Note that the end-task and is the set of auxiliary tasks.
Since the above bi-level problem is difficult to solve directly, Dery et al. (202 relax the problem and into an alternating optimization problem where task weights are updated based on 1 1-step improvement to the validation performance of the end-task
a (0++1(w)) = -VVL..))) (g
(4)
Dw, To prevent the above relaxation from finding the trivial solution of just upweigting solely the end-task. Dery et al. (222lb) introduce special dev-head which they use for estimating the meta-gradient
i d (0'(w)
(VVoLL. (Vo ((Obodyy Φ") o)){))
(5)
Ow;
Where of is the special dev-head and Obody is the body of the model. For even more details about META-TARTAN, please see Section 3 of Dery et al. (2021b).
Though we leverage MET-TARTAN, compared to Dery et al.) (2021b), we make three distinct contributions to the field of auxiliary learning. We list them below
Novel Problem Formulation: As far as we are aware of, we are the first to formulate the problem of automated auxiliary learning. Specifically, presented approach for automatically constructing a suite of auxiliary objectives based on existing objectives. Please note that Dery et (202 perform auxiliary learning with only the DAPT/TAPT variants of the BERT objective. They effectively assume that the search space of objectives (the they explore) is given sefore-hand. Our approach automatically creates the search space.
2. Theoretical Novelty: To the best of knowledge,
the first work to provide exploration of why auxiliary learning improves primary task performance via algorithmic stability. Dery et al. (002 in introducing META-TARTAN do attempt give theoretical characterization of why the algorithm improves end-task performance.
3. Algorithm Improvements to META-TARTAS presented in Dery al. (2021b) used with only 2 auxiliary tasks. When scaling to more tasks, using META-TARTAN naively becomes computationally prohibitive. Specifically, search space of N tasks, META-TARTAN requires O(N) order computation per step.
Please note that META-TARA as
14

---

Published as a conference paper at ICLR 2023
We improve upon this by introducing the task sub-sampling of (k < N) which reduces the compute overhead to O(k). To account for the impact sampling as an approximation, introduced the factorised modelling of task weights which allows sharing of information between auxiliary tasks that might themselves be related.
C DATASET DETAILS
Table 4: Specifications of datasets used to evaluate our methods.
Domain
Task
Label Type
Train Size
Dev Size
Test Size
Classes
Metric
BIOMED
CHEMPROT Kringelum
2016
relation classilication
4169
2427
3469
13
Accuracy
CS
SCIERCI Taai Luah all 2018
relation classification
32.19
455
974
STANCE
SE-20164 6-0 Mohammad
2016
detection
2497
1249
Accuracy
CS
ACL-ARC Jurglls et m
2013
citation intent
1688
114
139
6
F1
H.PARTISAN Kiesel
partisanship
Accuracy
D MORE TRAINING DETAILS
We run each
configuration across 3 seeds {0, 1, 2}. We use a batch size of 128 for all end-tasks tasks except H.PARTISAN where use batch size of 64. The auxiliary task bateh-size, aux is shared across all the n sub-sampled auxiliary objectives according to the objective's weight.
We use the AdamW optimizer (Loshchilov & Hutter 2017), with weight decay of 0.01 for all experiments.
Table 5: AANG-TD specific Hyper-parameter
Values
Description
aux.Ir
1.0 0.1
Learning rate for factor vectors (W-AIl W WT, W- woj Learning rate for primary task weighting Ae
soptIr
0.1,
nconf subsamp
3,6
Number of sub-sampled auxiliary tasks.
learning rate
le-3, lc-4
Learning rate used for further training RoBERTabase
aux_bsz
256
Batch size of for auxiliary objectives
Table 6: AANG-TD+ED specific Hyper-parameters
Hyper-parameter
Values
Description
aux_Ir
1.0. O.5, 0.1
Learning rate factor vectors (W-All W WT, w woj Learning rate for primary task weighting X Number of sub-sampled auxiliary tasks.
sopt_Ir
nconf subsa_su
6, 12, 24
learning rate
lc-4
Learning rate used for further training of RoBERTabase Batch size of for auxiliary objectives
aux_bsz
1024
Table 7: META-TARTAN Hyper-parameters for single task auxiliary tasks
Hyper-parameter
Values
Description
sopt_r
1.0, 0.1, 0.01
Learning rate for primary task weighting Xe Learning rate used for further training of RoBERTasaae
learning rate
1e-3, Ic-4, 5e-5
META-TARTHS introduces dev-head which is trained sporadically during training for estimating the meta vetta-gradienss We use the following parameters for training this dev-head: we sample 32 examples (8 examples in the of H.PARTISAN) and perform full batch gradient descent with
15

---

Published as a conference paper at ICLR 2023
learning rate of le-2 for 10 iterations. The dev-head is trained with the Adam optimizer with weight decay set to 0.1.
We copy the end-task agnostic baseline results from (Dery et al. 2021b) when available. We the hyper-parameter specified for TAPT in Gururangan et al. (2020) to train for the SE-2016-6 task.
All models were trained one of two types of gpus: NVIDIA A100 or NVIDIA A6000. All models fit within a single gpu. We used gradient accumulation to expand the effective batch sizes used for experiments.
E GENERALIZATION ERROR BOUND FOR END-TASK AWARE TRAINING
E.1 DEFINITIONS
Definition E.1. A fanctio,, f - R is L-Lipschitz if Vu, dom(f):
Il((w) - s(v))l s Lilu wll
Note that L-Lipschitz implies bounded gradients.
/Vf(w)] < Ve Definition E.2. A functin, f : + R smoohh Vu, E Q: [V - vf(v) V - Bl - wll
Definition E.3, An update rule, G is o-bounded if:
supwER
|| - (())ll o
Consider the following general setting. There an unknown distribution De over examples from some space Z. We receive sample $ (=1,..., 2N. of N₂ examples drawn i.i.d. from D,.- Our goal is to find model te that parameterizes the function fe, with small population risk defined as: Definition E.4. Population Risk
R|w] Ez-D. So(w;
Definition E.S. Empirical Risk Since we have a finite number of samples, we can only compute the empirical risk which is
Rs[w] = No se((;;
Let A be a potentially randomized algorithm (such as Stochastic Gradient Descent) that is a function of the S such that w A(S). Definition E.6. Generalization Error Egen (A, Ne)
Egen Ne) Es.A [R(A(S)) - R[A(S)]]
Definition F.7. Uniform Stability
A randomized algorithu A is c-uniformly stable if for all data sets S, E 2, |S| = = = N₂ such that S and S differ in at most example, have
sup =) - =)] Se
Here, the expectation is taken only over the internal randomness of A. We will denote by Caab (A, Ne) the infimum over all e for which the above holds.
E.2 RELEVANT THEOREMS
Theorem E.1 (Uniform Stability implies Generalization in expectation). Let Algorithm A be uniformly stable. Then,
Cyen (A, Ne) Es.A - R[A(S)]]
(A,)
For fill proof Theorem 2.2 Hardi et al. 2016
16

---

Published as a conference paper at ICLR 2023
Theorem E.2 (Stochastic Gradient Method is stable), Assume that fe(;2) [0, 1] is L-Lipschitz and Be- smooth loss function for every 2, Suppose that we run SGM for T steps with monotonically non-increasing step sizes 04 S 4. Then, SGM has uniform stability with
1
(2c1.) T N₂
Esgm
where q == Bec
We can simplify this to only terms involving T and Ne
T c8
Exgm
Ne
Proof. For the full proof, see Theorem 3.12 of Hardt et al. (2016)
E.3 GROWTH FUNCTIONS
Lemma E.3 (Growth Recursion Under Dynamic Sampling). We consider the Stochastic Gradient
update rule G Ω +Ω:
Gy(w) = w aVS(w)
Fix arbitrary sequence of updates Gf..... Gtr and another C' G'zz- Let wo w/ be starting point in P given that f Q + R and define
8, = En [llu: w/ll]
where Wt, w are defined recursively through
W; = (w:-1) w/= Gz,((//))) t20
Then have the recurrence relation
80 =
min {(1 + a + ad +LL, a1+a + 228)))8&
Gj, =G,,
8.41 S
8, + 20,
Gf G₁ a-bounded
Note that Py is distribution over the support {
,f) according to probabilities (A1, A2 A + =
1}. [/, }2] have smoothness B1, 32 respectively.
17

---

Published as a conference paper at ICLR 2023
Proof. The second bound on 6G is taken directly from Lemma 2.5 of Hardt et al. 2016). We now derive the first-half of the first bound
5.41 Pi....I-
[1 - w/+11]
X1 G'y (w/)1 >21[G Gp (w((i
Aillur aall--aff uny
aVf -
+ -Vf - wa + a
f.....PPP [llw - w/lll + all fI..
\1 - - 1 1
v (
(Triangle Inequality used for above step)
= + fI..P.. (n (
(Without Loss of Generality, let (22)
de + A......PPPP /1/31/wu2 w/ll + >21lV V (w(wi)
(Smoothness)
& + ad13110 ad2E, f1 Si ( -V ((a))]
(Triangle Inequality)
8 ( (w)) P((n))
(1 (1+1)1))0 0d2
(add zero)
(1+ ad
(v)) (
vf(w)))
(Triangle Inequality)
a1+))13))0 ad2
A A 1 V
Using Assumption A.1
(1+ aA2
A [ [V52(()
+
Triangle Inequality
{1+ (A 2L) L-Lipschitz function
To obtain the second half of the first bound:
d.41 E
[||W++1 - w4+1l]
E
A1 (wf()l + X211G -
dullur (vr) + avvf(( + A2llwr - -aff( - u. + a (
$ E |lll - ww'll + a.E A1 \1 (w)) (w:)) + v
(Triangle Inequality used for above step)
A1silllw - will + A2/3211u2 - will
S 6t + a D.E....PPPPP
(Smoothness)
|lwe w/ll|
+ 322.3 /.. |10e - w/ll
5, + 1d11,111 (..
de + a()1 + \2
a1+ + )232))6
E.4 STABILITY OF DYNAMIC SAMPLING
We repeat the description of our Auxiliary Learning with Dynamic Sampling Setting here for ease of access.
18

---

Published as a conference paper at ICLR 2023
Setting: We are given auxiliary objective (= E [0, 1] with Na samples Sa (21..... ................................ 2N,) )
from the distribution Da. At any iteration of SGD, we sample a choice of either the end-task function fe or the auxiliary objective fa according the probabilities Aes X₂ Ae Aa 1. Given the chosen objective, we sample a data-point and perform stochastic gradient descent (SGD) based on the sampled data-point.
An equivalent way to instantiate this procedure to create SA by drawing N' N₂ + N₂ total samples from the end-task and auxiliary task according Px. S'A then created by replacing end-task sample in SA. At each step, a sample is drawn from distribution: 2i, 2g ~ PSA: Psy and gradient step taken on the function corresponding to the the sample was drawn from.
Lemma E.4 (Stability dynamic sampling). We denote the outputs T steps of SGM on SA and S'A with the dynamically sampled functions, as wr and w/y respectively. Then, for every be Ze and every to > O, under both the random update rule and the random permutation rule, we have
2to N
fe [uTr 2) ffe (e(pr: 2)] <
sup fe(w; ze) + LB[5T]o0. = 0]
w,ze
Where N' Ne + No and 7
N' No
Proof. Let 3 = 16,,, = 0] denote the event that 6ip- = O. We have
[(fe(wr:z) - fe(w<;=)) = ((E]=([[e(wrr=)) fe =e(w(r:)]1E]]
+ JE [((](((r:)) (e(1ms=)
< (w]]c( (Awwr==)|13 + P(E9)
sup fe(w;za ze)
w.ee
because is nen-negative
$ <L 11 + P(E)) sup Ze)
wi
because is I-Lipschitz
We now proceed to bound P(E)). Let i. [N'] denote the position in which SA, differ and consider the random variable assuming the index of the first time step in which SGM uses the example 2 Note that when / > to. then must have that drg since the two samples are identical up until this point.
- + 0} < P( to)
Using the selection rule specified above (sample either fer fa according to the probabilities de Ao. and then sample uniformly from the selected task data) we have that
Acto
Ytc Ne N
PfI < to)
Σ
P(I to)
Σ
(ee
Ne
Theorem E.5 (Stability Bound Dynamic Sampling). Assume that fel; ze), ), Sal: =a) E [0, 1] are L- L-Lipschitz and Be and Smoshh loss functions. Consider that we have N' Ne + Na total samples where fo and fa have N. and N, samples respectively. Suppose that run SGM for steps with monotonically non-increasing step sizes 04 & by dynamically sampling the tasks according to de and >a- Then, with respect to Jc, SGM has uniform stability with:
c8
Estab S (1+ cB
L2
2011
7T
Iled
pLe
AD
No
Where
7
Ne
Given that 3 min Ba) and X" the corresponding weighting of the function with smaller smoothness.
Depending on which one gives tighter bound the pair (B.p) p) can be
(8, p) 3", (² 2T.))
or
(3,p)² (Nei3e + Ba, 0)
19

---

Published as a conference paper at ICLR 2023
When (3, gives the tighter bound, simplify to:
77 N
Egen
T+cx+ #
As presented in Section
Proof. Let SA, S be two sample of size N' Ne + Na as described in lemma E.4 Consider the gradient updates ff....,, and G',,... G induced by running SGM on samples S and S
respectively. Let wr and wip denote the corresponding outputs of SGM. By lemma
we have: :
ytu N7
[esr(w;; Se(wor =)
sup fa(w; ze)
LE 0]
8
w..3e
Let , (6, = 0j. We will bound V as function of to and then minimize for to. Note the following
At any step i, with probability (1 N ), the sample selected is the same in both SA and
Shx In this case Gf. Gy and we use the corresponding expansivity rule from lemma E.4 This gives:
8-1 S min {11 3x)"3"6o+ oz (A 2L), (1 + (NeBee + )r.
Where 8r min (Be, 8a) and X is the corresponding weighting of the function with smaller smoothness. To avoid deriving the bound independently for each case, we perform a variable substituation that captures the cases:
8e+1 S (1 + 3, 5. + top
3 (X*"*, XeBe AaPaa and p {(1 - X (A) 23). We can present the final bound in terns of these variables which can be substituted depending on the minimizer.
With probability N7 the selected example different. Note that in this case, we know that we are evaluating the end-task function fe. We use that both G5, and G',, are (a, a2 L)- bounded according to lemma E.3 since fe is L-LLipscitz.
Combining the above we have
V S (1- ) (1+ (15) U + amp)
7 (w, + +2a4L) N'
N' + +(1 2 No (1 + a0.
7
1 L
N' + aal
N )
1+ (1 )aa8) 1/ (27 + (N' -7)p)
N'
< 1+ (1 No [3 1. c(2yL + (N' - 7)p)
2))
tN'
S (1 71) 3 V
1(27 (N' 7)p)
1.
INO
We use 1 x < exp(x) Vr
Sexp (1 No. / V + cp
2
t
Where p == (2 + (N' - 7)p)
20

---

Published as a conference paper at ICLR 2023
We unwind the recurrence until Vto 0.
T
T
TT
S
Σ
II
exp
((1
Nz
N
c3
k
IN
cp
t=
k=(+1
T
T
cp
exp (1
7
Σ
tN
No
1=++1
T
cp
(1- N
(
L'
exp
ce log
1=
cpT=0(1-
)r)
T
t
cf
)
(10)
N'
+=
We can upper bound the sum over t with an integral + drop negative terms
cδ(1
cp
T
N1
N:)
to
p
T
cß(1
x))
3(N' - 7)
oB
p
T
3(N'
Plugging this bound back into Equation 8 and using the fact that fe E
[0, 1]:
c3
yoo
Lp
T
E fe(wr;;) 2) - fe(w((;)) <
+
(1I)
N
3(N
7)
40
We let 9" c5, we can minimize the R.H.S by setting:
Ncp री.
To
to
7(N'
(7)
Plugging this in gives us
(wrr - fel fe(w
(1+ 1
cg
N' L. + (N' (N'--1))
65+7
(TT)
11e3
c3
N'
(N' -7)
(12)
1
(
22,
6011
(TT)
1
N' - pTic)
c3
N
Recall that:
B = (x* Aege + Na
p= (11 (' 2L),
We can choose whichever of the pairs for B, p that minimizes the bound:
F DISCUSSION OF GENERALIZAAION ERROR BOUNDS
F.1 WHAT DOES THEOREM E.5 SAY.
We consider the setting where
B = 3'
P P= (1- (1 2T.
21

---

Published as a conference paper at ICLR 2023
Assuming the p term dominates Equation 12 in this setting is
GB 1+00
YT
Auxdyn
auxdyn stab
1t
- (1- + 2L)
gen
(3,p)
N
77
This is Equation from Section
4
N
(13)
In going from the first line the second we consider the setting where A > 2L. This a case where the auxiliary task is sufficiently different from the primary task. Some observations about this setting:
1. Smaller A implies auxiliary task is similar to main task and leads to improving the bound.
2. Dependence of the bound on N' is a bit more nuanced. Note that increasing N' increases Y unless we reduce X appropriately. Remember that Xe is the rate at which sample the primary task. Thus, if we add more auxiliary data but still sample the primary task at the original rate, then are effectively ignoring the extra auxiliary data.
3. It might be tempting to assume that we can get arbitrary improvements in this setting by setting A = 0. However, note that whilst this might reduce the generalization error, means that training error
sceing of the end-task which would result in large increase in the
4. Note that (3 = X < 3c) always. So we get improvements on the dependence on T compared to Theorem
5. We can optimize des minimize stab
auxdyn
22


Reviews:
[{"review_id": "tje-89VC61", "reviewer": "Reviewer_pGCt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The pretrain-(continued-pretrain)-finetune paradigm has been de-facto in recent NLP state-of-the-arts. However, the construction of auxiliary objectives still need carefully design by experts. This paper proposes to automatically generate a suite of auxiliary objectives. Specifically, they provide theoretical bound for the generalization error in auxiliary learning, and design an efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task.", "summary_of_the_review": "To the best of my knowledge, this is the first work that propose to automatically generate auxilliary objectives. Moreover, it provides good results both theoretically and practically. ", "strength_and_weaknesses": "**Strength**\n* This is the first work that propose to automatically generate auxilliary objectives\n* The experimental resutls seems promising\n\n\n**Weaknesses**\n* The proposed method presents (potentially heavy) extra computational cost\n* The proposed method only run experiments on continued pretraining setting and the datasets are relatively small.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "t6aUJ9bXlR", "reviewer": "Reviewer_KWWF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach using meta-learning to learn to weight multiple auxiliary losses for any given target task. They first define a space of auxiliary losses that contains popular self-supervised methods like BERT and XLNet as special cases. Then they propose to learn the weighting over a given space of such losses using the META-TARTAN algorithm from previous work. Approach is evaluated on 5 classification tasks where the dynamic weight learning on auxilary losses improves over a multi-task baseline on 4 out 5 datasets by.", "summary_of_the_review": "I am not entirely convinced of the utility of this method and it lacks some important baseline comparisons (see weaknesses above).", "strength_and_weaknesses": "Strengths\n- An approach to dynamically learn auxiliary loss weighting that could be useful for finetuning pre-trained models on specific tasks.\n- Empirical results include statistical significance, and interesting analysis of how weights vary during training.\n\nWeaknesses\n- The paper was a bit tough to follow. The setting up of the auxiliary loss space was more detailed than it needed to be. In particular, I do not find this setup to be specially illuminating, or inspiring new losses just by its construction (as one would not trivially expect every possible choice of loss in this space to be useful). It would have been better to provide more details in sections 4/5/6. The paper also assumes knowledge of META-TARTAN paper and heavily relies on it.\n- Differences from the META-TARTAN paper are not clear and are not explicitly mentioned. It should be an important baseline but it seems it is not compared.\n- It is not clear if the multi-task baseline is trained in the same way, i.e. with all the 24 losses in the auxiliary space and updating based on a random subset of 24 losses (which introduces some regularization from overfitting). Improvements over the existing multi-task results appear minor.\n- When scaling on the number of auxiliary objectives, the authors find no improvement. This is disappointing as one would expect that trivially taking the cartesian product of the space of choices will yield a large number of uninteresting losses and the approach is incapable of pruning them out.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SJPfJDrfXD", "reviewer": "Reviewer_rBEM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes AANG to automatically generate a suite of auxiliary objectives for end tasks. AANG builds a unified framework/pipeline that consists of several parts: input data, input transformation, model representation, and output. AANG uses algorithmic stability to search for the best combination in each part. The experiments are conducted on the NLP domain, and the results demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "I think this paper is novel and soild enough. It can meet the acceptance line of ICLR.", "strength_and_weaknesses": "This paper proposes AANG to automatically generate a suite of auxiliary objectives for end tasks. AANG builds a unified framework/pipeline that consists of several parts: input data, input transformation, model representation, and output. AANG uses algorithmic stability to search for the best combination in each part. The experiments are conducted on the NLP domain, and the results demonstrate the effectiveness of the proposed method. \n\nEven though many techniques used in this paper are proposed by other works, this paper delicately combines them.\n\nThe paper presents a solid theoretical analysis of the proposed algorithm.\n\nIt would be better to extend and deepen the experiment to non-NLP tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JWinE9G6na", "reviewer": "Reviewer_oyPz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studied the problem of how to take advantage of auxiliary loss functions in continuously adapting a pretrained language model. The author proposed to automatically pick auxiliary loss functions from a search space. The loss functions are reweighted during the learning process based on how they can help the downstream task. Author conducted experiments in four datasets to demonstrate the effectiveness of the AANG algorithm. Ablation study also shows that the algorithm for updating the weight for each loss function is effective and can pick auxiliary losses that are mostly related to the downstream problem.\n", "summary_of_the_review": "Voted for weak rejection due to concern in experimental evaluation, and the computational complexity of the AANG algorithm. The problem of automatically search for auxiliary learning tasks that are helpful for adapting the model is novel.\n", "strength_and_weaknesses": "Strength: \n1. The paper studied a novel problem: trying to automatically pick auxiliary loss functions from a large search-space.\n2. Experiments show that AANG is able to outperform baselines (Table 2). Figure 4 also shows that AANG is able to identify the important loss functions during the training procedure, and the weighted + META-Train variant outperforms assigning equal weights to each loss function.\n\nWeakness:\n1. The computational complexity of AANG is higher due to the meta-learning phase.\n2. The reviewer feels that the author needs to compare with advanced pretraining + distillation algorithms, like \"[NeurIPS2020] MINILM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers\". In the paper, the author only compared with GPT / XLNet / BERT objectives. However, these objectives may not be suitable for continuous pretraining and the author needs to compare with stronger baselines.\n3. The final language model adapted via AANG is applicable to a single specific downstream applications. The author may need to study if the final backbone model adapted via AANG can also generalize to other tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%