Weighted Training for Cross-Task Learning
Authors: Shuxiao Chen
Koby Crammer
Hangfeng He
Dan Roth
Weijie J Su
Venue: ICLR 2022 Oral (2022)
Keywords: Cross-task learning
Natural language processing
Representation learning

Abstract: In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks. We show that TAWT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees. The effectiveness of TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning.

Published as a conference paper at ICLR 2022
WEIGHTED TRAINING FOR CROSS-TASK LEARNING
Shuxiao Chen
Koby Crammer
University of Pennsylvania shuxiaoce wharton. wwwnn.een.. edu
The Technion
www./iiieteccion.....m ac. i1
Hangfeng He University of Pennsylvania sangfengeseas upenn. edu
Dan Roth
University of Pennsylvania
danroth@seas. upenn edu
Weijie J. Su University of Pennsylvania suwewharton, upenn edu
ABSTRACT
In this paper, introduce Target-Award Weighted Training (TAWT), weighted training algorithm for cross-task learning based on minimizing a representation- based task distance between the source and target tasks. We show that TANT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretie theoretic guarantees. The effectiveness of TAWT corroborated through extensive experiments with BERT on four sequence tagging tasks natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation- based task distance allows one to reason a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning
INTRODUCTION
The state-of-the-art (SOTA) models in real-world applications rely increasingly on the usage of weak
supervision signals (Pennington al., 2014 Devlin al., 2019 Liu al. 2019). Among these, cross-task signals are one of the most widely-used weak signals (Zamir et al. 2018 McCann et al.) 2018). Despite their popularity, the benefits of cross-tar pooss-task signals are not well understood from theoretical point of view, especially in the context of deep learning (He et al. 2021 Neyshabur et al. 2020), hence impeding the efficient usage those signals. Previous work has adopted representation learning as framework to understand the benefits of cross-task signals, where knowledge transfer is achieved by learning a representation shared across different tasks (Baxter) 2000 Maurer et al., 2016 Tripuraneni et al. 2020 202T Du et al. 2021). However, the existence of shared representation is often too strong an assumption in practice. Such an assumption also makes it difficult to reason about several critical aspects of cross-task learning, such as the quantification of the value of the source
data and the impact of fine-tuning (Kalan & Fabian 2020
Chua et al. 2021).
In this paper, we propose Target-Aware Weighted Training (TAWT), weighted training algorithm for efficient cross-task learning. The algorithm can be easily applied to existing cross-task learning paradigms, such pre-training and joint training, to boost their sample efficiency by assigning adaptive (i.e., trainable) weights on the source tasks or source samples. The weights are determined in theoretically principled way by minimizing a representation--bsee task distance between the
source and target tasks. Such a strategy is in sharp contrast to other weighting schemes common machine learning, such as importance sampling domain adaptation (Shimodaira) 2000 Cortes
et al. 2010 Jiang & Zhai 2007).
OOr code is publicly available at http:p///aggoomp.or//////////p/
ew/963

---

Published as a conference paper at ICLR 2022
The effectiveness of TAWT is verified via both theoretical analyses and empirical experiments. Using empirical process theory, we prove a non non- -on-smmptoticc generalization bound for TAWT. The bound is a superposition of two vanishing terms and a term depending the task distance, the latter of which is potentially negligible due to the re-weighting operation. We then conduct comprehensive experiments on four sequence tagging tasks in NLP. part-of-speech (PoS) tagging. chunking, predicate detection, and named entity recognition (NER). We demonstrate that TART further improves the performance of BERT (Devlin et al. 2019 in both pre-training and joint training for cross-task learning with limited target data, achieving an average absolute improvement of 3.1% on the performance.
As a byproduct, we propose a representation based task distance that depends on the quality of representations for each task, respectively, instead of assuming the existence of single shared repre- sentation among all tasks. This finer-graines notion of task distance enables a better understanding of cross-task signals. For example, the teassenttttio-bsseee task distance gives an interpretable measure of the value of the source data on the target task based on the discrepancy between their optimal representations. Such measure is more informative than measuring the difference between tasks via the discrepancy of their task -peecifi functions (e.g. linear functions) as done in previous theoretical frameworks (Tripuraneni et al., 2020). Furthermore, the representation-bun baaodd task distance clearly conveys the necessity of fine-tuning: this distance is non-zero, then fine-tuning the representation becomes necessary the representation learned from the source data does not converge the optimal target representation.
Finally, compare our work with some recent attempts in similar directions. Liu al. (020) analyze the benefits of transfer learning by distinguishing source-specific features and transferable features in the source data. Based on the two types of features, they further propose a meta representation learning algorithm to encourage learning transferable and generalizable features. Instead of focusing on the distinction between two types of features. our algorithm and analyses are based on the representation-basee based task distance and are thus different. Chua et al. (202T) present a theoretical framework for analyzing representations derived from model agnostic meta-learning (Finn et al., 2017p, assuming all the tasks use approximately the same underlying representation. In contrast, we do not impose any a priori assumption on the proximity among source and target representations, and our algorithm seeks for weighting scheme to maximize the proximity. Our work is also different from task weighting in curriculum learning. This line of work tends to learn suitable weights in the stochastic policy to decide which task to study next in curriculum learning (Graves et al.] (017)). while TAWT aims to learn better representations by assigning more suitable weights on source tasks. Compared to heuristic weighting strategies in multi-task learning (Gong al. 2019 Zhang & Yang 202T) we aim to design a practical algorithm with theoretical guarantees for cross-task learning.
2
TAWT: TARGET-AWARE WEIGHTED TRAINING
2.1 PRELIMINARIES
Suppose have T source tasks, represented by collection of probability distributions (D,}7- on the sample space X x V, where C Rd is the feature space and y C E is the label space. For classification problems, we take y to be a finite subset of E. We have single target task, whose probability distribution is denoted as Do. For the 1-th task, where = 0, 1,..., T, we observe n. i..d. samples S, = { )o)))= from Dr. Typically, the number of samples from the target task, no. is much smaller than the samples from the source tasks, and the goal is to use samples from source tasks to aid the learning of the target task.
Let & be a collection of representations from the feature space X' to some latent space 2 C R". We refer & as the representation class. Let F be a collection of mask-specific functions from the latent space Z to the label space y The complexity of the representation class & is usually much larger (i.e., more expressive) than that of the task-specific function class F.
Given bounded loss function é 3 x y + [0, 11, the optimal pair of representation and task-specific function of the t-th task is given by
(ot, E argmin &EE
Le(de, ft,, Li(dr, fi) = Exx,r)_,, or)))
(2.1)
Note that in general, the optimal representations of different tasks are different. For brevity, all proofs for the theory part deferred to the Appx.
2

---

we have Published as a conference paper at ICLR 2022
2.2 DERIVATION OF TAWT
Under the assumption that the optimal representations {6z Y are similar, a representation learned using samples only from the source tasks would perform reasonably well the target task. Conse- quently, we can devote Big samples from the target task to learn only the task specific function. This a much easier task, since the complexity of JF is typically much smaller than that of Φ.
This discussion leads to simple yet immensely popular two-step procedure as follows (Trppuaaeeni al. 2020 Du al. 202T). First, solve weighted empirical risk minimization problem with respect to the source tasks:
(o ( E argmin
Σ fo. (e
Le fi) = 1 Σ e(fe o d(zu), vu). (2.2)
where E A T-1 is user-specified vector lying in the T-dimensional probability simplex (i.e., En Wt 1 and w, 2 0, V1 t T). In the second stage, we freeze the representation o, and seek the task-specific function that minimizes the empirical risk with respect to the target task:
fo E argmin Lu(o, fo).
(2.3)
foeT
In practice, we can allow á to slightly vary (e.g., via fine-tuning) to get a performance boost. In the two-step procedure (.2) 2.33) the weight vector usually taken to be a hyperparameter and is fixed during training. Popular choices include the uniform weights (i.e., Wy 1/7") or weights
2019:Jonnso proportional to the sample sizes (i.e., wi n// Er= Re) (Liu al. 2019 Johnson & Khoshgoftaar 2019). This reveals the purget-agnostic nature of the two-step procedure (.2 (2.3). the weights stay the same regardless the level of proximity between the source tasks and the target task.
Consider the following thought experiment: if we know a priori that the first source task D1 is closer (compared to other source tasks) to the target task Do. then we would expect a better performance by raising the importance of 'D1 i.e., make W1 larger. This thought experiment motivates target-awes aware procedure that adaptively adjusts the weights based on the proximity of source tasks to the target. A natural attempt for developing such task-aware procedure is a follows:
T Σ
Lo(g, fo) NeEF WEA
min
subject to & E argmin min (W,
(OPT1)
vee (fiCT
That is, we seek for the best weights w such that solving 2.2 with this choice of w would lead to the lowest training error when we subsequently solve (2.
Despite its conceptual simplicity, the formulation OPTT is complicated constrained optimization problem. Nevertheless, we demonstrate that is possible to transform it into an unconstrained form for which a customized gradient-based optimizer could be applied. To do so, we let (0" {sr) }) be any representation and task-specific functions that minimizes ΣT WrCr(0 f() over & and (f+} C F. Equivalently, d minimizes Σ wi minserr Lo J2) over & E t. With such notations, we re-write (OPT)
min JEF
Lo(0", fo).
(OPT2)
The gradient of the above objective with respect to the task-specific function, V LoCo0 ff). is easy to calculate via pack-propagation. The calculation of the gradient with respect the weights requires more work, as 8t is an implicit function of w. By the chain rule, we have Jws [V [VoLo(0", fo)]_ a Bu, Ge Since d" is minimizer of + Σt We ming,rr Le(o.
8 Lo(de fo) =
F(d w) O, Yo T-1
F(ò. V
wh min fieF L,
(2.4)
By implicit function theorem, if .(:.) is everywhere differentiable and the matrix F wa/ is invertible for any (db, w) near some (Ø), w) satisfying F(ő, w) 0, then conclude that the
3

---

Published as a conference paper at ICLR 2022
Algorithm 1: Target-Aware Weighted Training (TAWT)
Input: Datasets (St)?=o.
Output: Final pair of representation and task-specific function (ő. fo) for the target task. Initialize parameters w ~ .60 E 2, ( C F:: for = O,..., K do
Starting from (Ø* [ )[-), run few steps of SGD get (Ø (1_+1, (1+
Use the approximate gradient V, Eo( fu) to run few steps of SGD from fo to get fot1;
Run one step of approximate mirror descent (2.7 2.8 from w to get w
end
return / = 6* fu fo
map w + d² is locally well-defined function near W. and the derivative of this map given by -
a Ow S0
|F [FF(o.w) w)
(2Fro Owr
(F (F(()w)) w)
(2.5)
06
To simplify the above expression, note that under regularity conditions, we can regard V VoLi(é, f as a sub- cubb-gradent of the map é + mins,eF L,(0, ft). This means that we can write F(ø", w)
En W. VoLi(0" ff)) Plugging this expression back to 2.5) and recalling the expression for @Cold" fo)/ derived via the chain rule, get
T
Owr Lo(d", fo) =
- -VoLo000 fo)]
Σ Civigiil f2) " IVoLi00, sr').
(2.6)
Now that we have the expressions for the gradients of (d0", fo) with respect to So and w, we can solve (OPT2) via a combination of alternating minimization and mirror descent. To be more specific, suppose that at iteration k, the current weights, representation, and task-specific functions are w øf, and (sr o respectively. At this iteration, we conduct the following three steps:
1. Freeze wh Starting from (Øt (1T)T))) run few steps of SGD on the objective function (0. (fiTTn) + Et w Le, to get (ah+), (f/+1)T mation of (o"", {S IT=)):
) which is regarded approxi-
2. Freeze (d (T Approximate the gradient V Ca(out .o) by V Co So). Us- ing this approximate gradient, run few steps of SGD from f to get fhtt;
3. Freeze (0++1 ( )Z Approximate the partial derivative eoo , // by
T
= -V fot
[ wi W. s1
VV (o sbttt). (2.7)
Then run one step of mirror descent (with step size n²*) from wt get wtl;
w expl 11n 9
x+1
(2.8)
Σr we exp( 9
We use mirror descent in (2.8). as it is canonical generalization of Euclidean gradient descent to gradient descent on the probability simplex (Beck & Teboulle 2003). Note that other optimization methods, such as projected gradient descent, rather intuitive explanation. Note that 9 is weighted dissimilarity measure between the gradients
also be used here. The update rule (2.8) has
VaCo and V. This can further be regarded as crude dissimilarity measure between the optimal representations the target task and the 1-th source task. The mirror descent updates W, along the direction where the target task and the 1-th source task are more similar. The overall procedure is summarized in Algorithm]
A faithful implementation of the above steps would require costly evaluation of the inverse of the Hessian matrix Ei wi V V2L. (d++1, ( R.xr. In practice, we can bypass this step by
4

---

Published as a conference paper at ICLR 2022
replacing the Hessian-inverrs -neeiee-eeeeeeeeeeedd weighted dissimilarity measure 2.7 with consine-similarity similarity-basee dissimilarity measure (see Section 4 for details).
The previous derivation has focused weighted pre-training, i... the target data is not used when defining the constrained set in (OPTT. It can be modified, mutatis mutandis, to handle weighted joint-training, where change (OPTT) to
T wi (t).
Lo(d, fo) f. WEAT
min
subject to Φ E argmin min WE (
(2.9)
Compared to (OPTT). we now also use the data from the target task when learning the representation s, and thus there is an extra weight Wp on the target task. The algorithm can also be easily extended to handle multiple target tasks or to put weights on samples (as opposed to putting weights on tasks). The algorithm could also be applied to improve the efficiency of learning from cross-domain and cross-lingual signals, and we postpone such explorations for future work.
THEORETICAL GUARANTEES
3.1 A REPRESENTATION BASED TASK DISTANCE
In this subsection, we introduce a representation- task distance, which will be crucial in the theoretical understanding of TAWT. To start with, let us define the representation and task-specific functions that are optimal in an "w-weighted sense" follows:
T Σ wi Jt.
argmin (ER.
(3.1)
Intuitively, (f(t) are optimal on the w-eeggtted source tasks when only a single representation is used. Since d² may not be unique, introduce the function space d" C to collect all di's that satisfy (3.I) To further simplify the notation, we let LJ (Ø) == minj,eF (r( which stands for the risk mncrred by the representation on the t-th task. With the foregoing notation,
write
T4 argminge Et= we wr (Ø). The definition of the task distance is given below.
Definition 3.1 (Representation: based task distance). The representation- task distance between the w-weighted source tasks and the target task defined as
dist Du.D Do) = sup (3(00) C7(()) t=1
(3.2)
where the supremum is taken over any T" satisfying
3.1 and ob is the optimal target representation.
If all the tasks share the same optimal representation, then above distance becomes exactly zero. Under such an assumption, the only source of discrepancy among tasks arises from the difference in their task-speeific functions. This can be problematic in practice, as the task-speeific functions alone are usually not expressive enough to describe the intrinsic difference among tasks. In contrast, we relax the shared representation assumption substantially by allowing the optimal representations to differ and the distance to be non-zero.
The above notion of task distance also allows us to reason about certain important aspects of cross-task learning. For example, this task distance is asymmetric, capturing the asymmetric nature of cross-task learning. Moreover, if the task distance is non-zero, then fine-tuning the representation becomes necessary, because solving (2.2) alone would gives representation that does not converge to the correct target oi. In addition, an empirical illustration of the representation-bee based task distance can be found in Fig. in Appx.
B
This task distance can be naturally estimated from the data by replacing all population quantities with their empirical version. Indeed, minimizing the estimated task distance over the weights is equivalent to the optimization formulation (OPTT) of our algorithm (see Appx. A. for a detailed derivation). This observation gives alternative and theoretically principled derivation of TAWT.
This type of approximation is common and almost necessary, such MAMI. Finn al. 2017
5

---

Published as a conference paper at ICLR 2022
3.2 PERFORMANCE GUARANTEES FOR TAWT
To give theoretical guarantees for TAWT, we need few standard technical assumptions. The first one concerns the Lipschitzness of the function classes, and the second one controls the complexity of the function classes via uniform entropy (Wellner & van der Vaart 2013) as follows.
Assumption A (Lipschitzness). The loss function E y x y - 00, 1] is Le-Lipschitz in the first argument, uniformly over the second argument. Any F is Ly Lipschitt w.r.f. the E2 norm.
Assumption B (Uniform entropy control of function classes). There exist Cp > 0,V0 > 0, such that for any probability measure Qx on X Rd, we have
( (Qxx)::) (C+/E)"²,
0,
(3.3)
where N ((x()))))))))))) is the (²x) covering number of & (i.e., the minimum number of L2 (Qxx)
balls with radius e required cover 4). In parallel, there exist CF > 0, Mf O, such that for any probability measure Q2 on Z S R", we have
(JF; (22)): s (Cz/e)"², is the (Q2z) covering number of F.
Ve >0 0,
(3.4)
where
Uniform entropy generalizes the notion of Kapnik-Chervonenk' dimension (Vapnik 2013) and allows us to give a unified treatment of regression and classification problems. For this reason, function classes satisfying the above assumption are also referred to as "VC-type classes" in the literature (Koltchinski) (006). In particular, if each coordinate of & has VC-subgraph dimension c( then (3.3) is satisfied with Ve = (rr c( (()) (recall that is the dimension of the latent space Z). Similarly, if F has VC-subgraph dimension c(F), then 3.4 is satisfied with VF = (c((F))
The following definition characterizes how "transferable" representation e is from the w- w-weighted source tasks to the target task.
Definition 3.2 (Transferability). representation & is (p, Cpn-transfeerable from w-weighted
source tasks to the target lask, if there exists p 0, Cp O Ouch that for any T E $w we have
E we (1 - 1/2
(6(0) - -5(0m) S Cp
(3.5)
Intuitively, the above definition says that relative to T"h the risk of ó on the target task can be controlled by polynomial of the average risk of á on the source tasks. This can be regarded as adaptation the notions of transfer exponent and relative signal exponent (Hanneke & Kpotufe 2019) Cai & Wei| (022)) originated from the transfer learning literature the representation learning setting. This can also be seen as a generalization of the task diversity assumption (Tripuraneni et al. 2020: Du et al. 2021) to the case when the optimal representations { 12" do not coincide. In addition, Tripuraneni et al. (2020) and Du et al. (2021) proved that p = 1 in certain simple models and under some simplified setting.
In this part, we prove a non-asymptotic generalization bound for TAWT. The fact that the weights are learned from the data substantially complicates the analysis. To proceed further, we make a few simplifying assumptions. First, we assume that the sample sizes of the source data are relatively balanced: there exists an integer n such that n, (n)) for any t (1....,,T] T). Meanwhile, instead of directly analyzing (OPTI), we focus on its sample split version. In particular, we let B U B2 be partition (1,...,7To) J. where |Bi] = O2) Define
2(1) (é. = B1
1
e(fo o o(zo)) vor), i
Σ
(o. =
1 Σ e(fo o(xor), you) sEB2
B2
We first solve OPTI restricted to the first part of the data:
T wi (W. (3.6)
( E
argmin min 201) (b, fo)
subject to e argmin min weo
Σ
de wEtWEATTL T-1 JoEF foEF
(fi)J
For vector-valued functions + their (Qx)) distance is (ʃ [[]((>) - (x)) (xxx))))()))

---

Published as a conference paper at ICLR 2022
Then, proceed by solving
fo E argmin 2 fo).
(3.7)
foEF
Such sample splitting ensures the independence of @ and the second part of target data B2, hence allowing for more transparent theoretical analyses. Such strategies common in statistics and econometrics literature when the algorithm has delicate dependence structures (see, e.g., (Hansen 2000; Cheroozuukov al. 2018)) We emphasize that sample splitting conducted only for theoretical convenience and is not used in practice.
The following theorem gives performance guarantees for the sample split version of TAWT.
Theorem 3.1 (Performance of TANT with sample splitting). Let (ō. fo) be obtained via solving 3.6 (..7p. Ler Assumptions and B hold. In addition, assume that the learned weights satisfy w Wg == {w E A : S wi/We < 3, Vt & {'}, where B 2 1 is an absolute constant. Fix E 1). There exists constant = C(L,, Ly, Co. Cr) > such that the following holds: if for any weights w E W9 and any representation & a C3 Vlug log o 1/n/T + (uF + log /m/m- neighborhood of [u4 there exists a specific o²e E 1h00 such that ee (p. Cp) -rttsssasfee,,ee, then there exists another C' CLe, LF, Cs. CF, Co, p) such that with probability at least 1 6, we have
VF + 1og(1/6)) (
+ 8/11000 + +log(1/5) vy + log T nT
Lolo, Fo) - Cold, f6) s sc'
no
dist W Dr. Do).
(3.8)
The upper bound in (3.8) is superposition of three terms. Let us disregard the log(1/8) terms for now and focus on the dependence on the sample sizes and problem dimensions. The first term, which scales with Vy /tnp corresponds to the error of learning the task-specific function in the target task. This is unavoidable even if the optimal representation Φ is known.
The second term that scales with [Pp + T2>>// (nT))) characterizes the error of learning the imperfect representation d" from the source datasets and transferring the knowledge to the target task. Note that this term is typically much smaller than ((vg + ///)/ng, the error that would have been incurred when learning only from the target data, thus illustrating the potential benefits of representation learning. This happens because nT is typically much larger than nog-
The third term is precisely the task distance introduced in Definition 3.1 The form of the task distance immediately demonstrates the possibility of "matching" d" x Φ via varying the weights w, under which case the third term would be negligible compared to the former two terms. For example, in Appx. A.2
we give a sufficient condition for exactly matching --"= Φ
The proof of Theorem 3.1 based on empirical process theory. Along the way, we also establish an interesting result on multi-task learning, where the goal is to improve the average performance for all tasks instead of target tasks (see Lemma A. in Appx. A). The current analysi, sis can also be extended to cases where multiple target tasks are present.
4 EXPERIMENTS
In this section, we verify the effectiveness of TAWT in extensive experiments using four NLP tasks, including PoS tagging, chunking, predicate detection, and NER. More details are Appx. B
Experimental settings. In our experiments, we mainly use two widely-used NLP datasets, Ontontes 5.0 (Hovy et al., 2006) and CoNLL-2000 (Tjong Kim Sang & Buchholz 2000). Ontonotes 5.0 contains annotations for PoS tagging, predicate detection, and NER, and CoNLL-2000 is shared task for chunking. There are about 116K sentences, 16K sentences, and 12 sentences in the training, development, and test sets for tasks in Ontonotes 5.0. As for CONLL-2000, there are
A representation is in an e-neighhorhond of :-' if ST Li (1)))
7

---

Published as a conference paper at ICLR 2022
Target Task
PoS
Chunking
Predicate Detection
NER
Avg
Learning Paradigm
Single-Task Task Learning
34.37
43.05
66.26
33.20
44.22
Pre- Pre-Training
49.43
73.15
74.10
41,22
59.48
Weighted Pre- Trainingn
51.17 *
73,41
75.77 **e
46.23 ***
61.64
Joint Training
53,83
75.58
75.42 75.98 * 6-4
43.50
62.08
Weighted Joint Training
57.34 4.4
77.78 ***
53.44 4**
66.14
Normalized Joint Training
84.14
88,91
77.02
61.15
77.80
Weighted Normalized Joint Training
86.07 4.40
90.62 4:0.6
76.67
63.44 ***
79.20
The benefits of weighted training for different learning paradigms under different settings. There are four tasks in total, PoS tagging, chunking. predicate detection, and NER. For each setting, we choose one task as the target task and the remaining three tasks as source tasks. We randomly choose 9K training sentences for each source task respectively, because the training size of the chunking dataset is 8936. As for the target task, we randomly choose 100, 100, 300, 500 training sentences for PoS tagging, chunking, predicate detection, and NER respectively. based on the difficulty of tasks. Single-task learning denotes learning only with the small target data. *** indicates the p-value of the paired sampled t-test smaller than 0.001.
Table 1:
about 9K sentences and 2K sentences in the training and test sets. As for the evaluation metric, we use accuracy for PoS tagging, span-level Fl for chunking, word-level Fl for predicate detection, and span-level F1 for NER. We use BERT Devlin et al., 2019) as our basic model in our main experiments. Specifically, we use the pre-trained sesessensitive BERT-base PyTorch implementation (Wolf et al. 2020), and the common hyperparameters for BERT. In the BERT, the task-specific function is the last-layer linear classifier, and the representation model is the remaining part. As for cross-task learning paradigms, we consider two popular learning paradigms, pre-training, and joint training. Pre-training first pre-train the representation part on the source data and then fine-tune the whole target model on the target data. Joint training uses both source and target data to train the shared representation model and task -pecificc functions for both source and target tasks at the same time. As for the multi-task learning part in both pre-training and joint training, we adopt the same multi-task learning algorithm as in MT-DNN (Liu et al. 2019). More explanation on the choice of experimental settings can be found in Appx. C
Settings for weighted training. For scalability, all the experiments, we approximate of in Eq. (.7) by -c x sim(V (o similarity between two vectors. Note that this type of approximation is common and almost necessary, such as MAML (Finn et al. 2017). For weighted joint training, we choose pre-training, we choose the best e among [0.3, 1, 10, 30, 100), because the cosine similarity between the pre- vre-training tasks and the target task is small in general. In practice, we further simplify the computation of V Ce fb+1) the average loss of a randomly sampled subset of the training set instead of the average loss of the simply set the size of the randomly sampled subset of the training set as 64, though a larger size is more beneficial in general. In our experiments, we choose 1/* 1.0 in the mirror descent update 2.. It is worthwhile to note that there no need to tune any extra hyperparameters for weighted joint training, though we believe that the performance of TAWT can be further improved by tuning extra hyper parameters, such as the learning rate n?"
V VoLe(ght1, s(++))) where sim(-,-) denotes the cosine
1. For weighted
(t 0, l...., T)) in Eq. 2.7 by computing the gradient over
whole training set as in Eq. (2.2). In our experiments,
Results. The effectiveness of TAWT is first demonstrated for both pre-training and joint training on four tasks with quite a few training examples in the target data, as shown in Table [I] Experiments with more target training examples and some additional experiments can be found in Table and Table in in Appx. B Finally, we note that TANT can be easily extended from putting weights on tasks to putting weights on samples (see Fig. 2 in Appx. B).
Normalized joint training. Inspired by the final task weights learned by TANT (see Table 6 in Appx. B. we also experiment with normalized joint training. The difference between joint training and normalized joint training lies in the initialization of task weights. For (weighted) joint training, the weights on tasks are initialized to be wt- = ne/ Ei- ny (i.e, the weights on the loss of each
'hhile BERT is no longer the SOTA model, all SOTA models are slight improvements of BERT, experiments are done with highly competitive models.
8

---

Published as a conference paper at ICLR 2022
2.5
Imorovementtn 1N6
Improvement
1.5
0.6
2
500
1000
4000
8000
-05
2000 4000
training of target datsset
trainig target
(a) The impact of the ratio between the training size the dataset and the target dataset.
(b) The impact the training size of the target dataset (Ratio=8).
(b) The impact of the training size the target dataset (Ratio=10).
Figure I: Analysis of the weighted training algorithms. We analyze the impact of two crucial factors on the improvement of the weighted training algorithms: the ratio between the training sizes of the source and target datasets, and the training size of the target dataset. In this figure, we use NER as the target task and source tasks include PoS tagging and predicate detection. In the first subfigure, we keep the training size of the target tasks as 500 and change the ratio from 1 to 128 on a log scale. In the second (third) subfigure, we keep the ratio 8 (10) and change the training size of the target dataset from 500 to 8000 on a log scale. The corresponding improvement from the normalized joint training the weighted normalized joint training is shown.
example will be uniformi), whereas for the (weighted) normalized joint training, the weights tasks are initialized to be w, = 1/TT (i.e., the weight on the loss of each example will be normalized by the sample sizes). Note that the loss for each task already normalized by the sample size in our theoretical analysis (see Eq. 2.2). As a byproduct, we find that normalized joint training is much better than the widely used joint training when the source data much larger than the target data. In addition, TANT can still be used to further improve normalized joint training. The corresponding results can be found in Table [ addition, we find that dynamic weights might be a better choice in weighted training compared to fixed weights (see Table 7 in Appx. B
Analysis. Furthermore, we analyze two crucial factors (i.e., the ratio between the training sizes of the source and target datasets, and the training size of the target dataset) that affect the improvement of TART in Fig. In general, we find that TART is more beneficial when the performance of the base model is poorer, either because of smaller-sized target data due a smaller ratio between the source data size and the target data size. More details are Table 4 and T Appx. B
DISCUSSION
In this paper, we propose a new weighted training algorithm, TANT. to improve the sample efficiency of learning from cross-task signals. TANT adaptively assigns weights tasks or samples in the source data to minimize the representation-basee algorithm is an easy-to-use plugin that can be applied existing cross-task learning paradigms, such as pre-training and joint training, without introducing too much computational overhead and hyperparameters tuning. The effectiveness of TAWT is further corroborated through theoretical analyses and empirical experiments. To the best of our knowledge, TART is the first weighted algorithm for cross-task learning with theoretical guarantees, and the proposed representation-barg task distance also sheds light on many critical aspects of cross-task learning.
task distance between source and target tasks. The
Limitations and Future Work. There are two main limitations in our work. First, although we gave an efficient implementation of weighted version of TAWT, an efficient implementation of sample-weighted version of TAWT is still lacking, and we leave it for future work. Second, we are not aware of any method that can efficiently estimate the representation-based task distance without training the model, and we plan to work more on this direction. In addition, we also plan to evaluate TAWT in other settings, such as cross doms-n and cross-lingual settings; in more general cases, such as multiple target tasks in the target data; and in other tasks, such as language modeling, question answering, sentiment analysis, image classification, and object detection.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This material is based upon work supported by the US Defense Advanced Research Projects Agency (DARPA) under contracts FA8750-19-2-0200 and 191 DMS-1847415 and an Alfred Sloan Research Fellowship. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
NSF through CAREER
REFERENCES
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):67-175,, 2003.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
T Tony Cai and Hongji Wei. Transfer learning for nonparametric classification: Minimax rate and adaptive classifier. The Annals of Statistics, 49(1):100-128, 2021.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal,
2018.
Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 34, 2021.
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In Nips, volume 10, pp. 442-450. Citeseer, 2010.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Simon Shaolci Du, Wei Hu, Sham M. Kakade, Jason D. Lec, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021.
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndi- rango, Gokce Keskin, and Oguz H Elibol. A comparison of loss weighting strategies for multi task learning in deep neural networks. IEEE Access, 7:141627-141632. 2019.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference machine learning, pp. 1311-1320. PMLR, 2017.
Steve Hanneke and Samory Kpotufe. On the value of target data in transfer learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Bruce E Hansen. Sample splitting and threshold estimation. Econometrica, 68(3):575-60) 2000.
Hangfeng He, Mingyuan Zhang, Qiang Ning, and Dan Roth. Foreseeing the benefits incidental supervision. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7782-1800, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 150.v0/503.0255 2015.
10

---

Published as a conference paper at ICLR 2022
Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pp. 57-60, 2006.
Jing Jiang and ChengXiang Zhai, Instance weighting for domain adaptation in nip. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 264-271, 2007.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
MM Kalan and Z Fabian. Minimax lower bounds for transfer learning with linear and one-hidden layer neural networks. Neural Information Processing Systems, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Vladimir Koltchinskii. Local rademacher complexities and oracle inequalities in risk minimization. The Annals of Statistics, (4(6):2593-2656 2006.
Hong Liu, Jeff Z HaoChen, Colin Wei, and Tengyu Ma. Meta-learning transferable representations with single target domain. arXiv preprint arXiv:2011.01418, 2020.
Xiaodong Liu, Pengeheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pp. 4487-4496, 2019.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. The Journal of Machine Learning Research, 17(1):2853-288) 2016.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint
2018.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What being transferred in transfer learning? In Advances in Neural Information Processing Systems, volume 33, pp. 512-523, 2020.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Erik F Tjong Kim Sang and Sabine Buchholz. Introduction the conll-2000 shared task: chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning- Volume 7, pp. 127-132. 2000.
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in Neural Information Processing Systems, volume 33, pp. 7852-7862, 2020.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-le of linear representations. In International Conference on Machine Learning, pp. 14434-10443, PMLR, 2021.
Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer science & business media, 2013.
Jon Wellner and Aad van der Vaart. Weak convergence and empirical processes: with applications to statistics. Springer Science & Business Media, 2013.
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierrie Cistac, Morgan Funtowicz. Joe Davison, and Sam Shleifer. Transformers: of-te-of- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.

---

Published as a conference paper at ICLR 2022
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018.
Yu Zhang and Qiang Yang. A survey on multi-task learning, IEEE Transactions on Knowledge and Data Engineering, 2021.
12

---

Published as a conference paper at ICLR 2022
A OMITTED PROOFS
We start by introducing some notations. For set S, we let g be its indicator function and we use #S and |S| interchangeably to denote its cardinality. For two positive sequences {a, } and {br J. we write On S bn or = (b,,) to denote lim an/aa//b < oo, and we let Z bn or denote bn & On- Meanwhile, the notation On x by or on (bb,) means an S by and On 2 b, simultaneously. For a vector we let ||ж|| denote its E norm. In this section, we treat Le, Lx, Cs. Cf as absolute constants and we hide the dependence on those parameters in our theoretical results. The exact dependence on those parameters can be easily traced from the proofs.
an (bbn,)
A.1 TAWT AND TASK DISTANCE MINIMIZATION
we estimate (6) = min foF Lo(Ø, fo) by minser Lo(d, (1, fo) and T E argmin sEN E W. minf,e (e( fi) by the argmin of ET Wi minser Lo fi), then overall, the quantity (3y) can be estimated by
min Co(d, fo) JOEF
subject to ø E argmin
Σ We We fc).
Thus, minimizing the estimated task distance over the weights (note that (0b*)) ) is constant) is equivalent to the optimization formulation (OPTT) of our algorithm.
To further relate TAWT with task distance minimization, provided analysis of the two-step procedure (.2
2.3 with fixed weights. The next theorem gives the corresponding performance
guarantees.
Theorem A.1 (Performance of the two-step procedure with fixed weights). Lei (5, Fo) be obtained via solving 2.2
(.33 with fixed w. Let Assumptions A and Bnd hold. Fix any ō E (0, 1) and define No = (Ei wf// /n,) /n.) There exists a constant = C(L,,L,,CC Cy) > O such that the following holds: for any representation & in CVlu + +VF 8og neighborhood of Jull there exists a specific Tre E &hw such that is (p. Cy)-transferable, then there exists another
C² CL Ly, Co. Cr, CpsP) such that with probability at least 1 5, we have
Lo(o, (o 56) S cc
cc(("
uy "ys+loge +log(1/0))
Ub + Tuy log(1/8) 20
+
ng T
No
dist
Σ
DrD,, Do)
(A.1)
Proof. See Appendix A.3
Note that the task distance naturally appears in the upper bound above. This theorem can be regarded predecessor of Theorem 3.1. and refer readers Section 3.2 for detailed exposition on the meaning of each term the upper bound.
A.2 A SUFFICIENT CONDITION FOR EXACTLY MATCHING T" = 65
Proposition A.1 (Weighting and task distance minimization). Suppose that for every à E &, there exists two source tasks 1 S f1, to < T such that Li, (6) < (Ø))) < Lt (Ø). Then there exists choice of weights w. possibly depending on 6, such that dist WiD,, Do) = 0.
Proof. By construction, it suffices show the existence of some w such that
T
Σ
Ve T.
(A.2)
'Recall that representation & in e-neighborhood of H0 ET (i( -I(0") Se.
13

---

Published as a conference paper at ICLR 2022
By assumption, fix any Φ E @ find f1.t2 I T such that Li, (0) - Cb(0) I Ci,(0) (0). We set wy = for any t & (f1.t2). If 27, (ő) = ()( = Li (0). then any choice of wi, Wt2 will suffice. Otherwise, we let
(6) - C)()) Why (0) Li (Ø)
Lb(0) LT, (0) Lig Li (+)
W1z
It straightforward to check that such choice of w indeed ensures
The proof is concluded.
A.3 PROOF OF THEOREM A.1
We start by stating two useful lemmas.
Lemma A.1 (Error for learning the imperfect representation from source data). Under the setup of
Theorem A.1 there exists constant C1 C1 (ILe, Ly, Co. Cr) > 0 such that for any E (0, 1), have
Σ wr - Ci
V4
Tuy log(1/6)
(A.3)
No
with probability at least 1-
Proof. See Appendix A.3.1
Lemma A.2 (Error for learning the task- function from target data). Under the setup of
Theorem A.1 there exists constant C2- C2(Le, Cx)
such that for any E (0, 1), we have
uy + log (1/6)
Colo, fo) (o) S C2
(A.4)
no
with probability at least
Proof. See Appendix A.3.2
To prove Theorem A.I we start by writing
Co fo) Lo f6) Lo(ō, Fo) - ( + ( (30 + - (6)
-
Suppose the high probability events in Lemmas A. and A.2 hold. Then can bound Lo Fo) (Ø) by (A.4). Mcanwhile, since Ø is in CV (2/ + Tws + log 6 1110)/////0 1/ neighborhood of th can invoke the transferability assumption to get
1/P C,a.le. ( Uo +
(uo+ +Tv Tuy + log(1/0) 1/2p
-
()( S C,
Σ (n
No
Moreover, we have the trivial bound that (D())) ( Lb (06) I supz Li(dw) - Lo(())). Assembling the three bounds above gives the desired result.
A.3.1 PROOF OF LEMMA A.
We start by writing
Σ we ( -
(T4)
T
wi (o, P() - L,
T
Σ
(1( - 2, P.) + 2. f.) - 2,(3",17"
+ + 2,130, - L.(30,
t=1
14

---

Published as a conference paper at ICLR 2022
T
$ W/
>>
- ft) + 2,00 - Lo(0", (io) ,p,)
1=1
f1( - Lo fi) + 2,00", ,
sup
wi
&EDAAACCF
T
St) - {(o o (( Mes + E(Se o g5 (2is), yrs) - Le(0 ;
sup (ER.LA)
W1
where the first inequality is by L7 ming.e Le Jt) and the second inequality is by the fact that (0, (ff))())) is a minimizer of 2.2). To simplify notations, let zu: (a Mes) and let the right-hand side above be G(((z,)). Fix two indices < t' S T. 1 < br S ne, and let {Zt)) J. be the source
datasets formed by replacing ăr, with some Ey (Ze Wr..)) ) E X x V. Since { and { differ by only one example, we have
WE
(.( fc) 8(fe ((xa), ua) + E(fe o² (*a), Mo) + (((3m,
The
Σ
(Cr fu) ( (.. 9.. + E(fe (0 (1 + Cy 18,
We
iti ne
we
(v) Aife (oltess)). + e(Fv o o000 (ae Cv (.3 fii)
nr
G((Zú))
(o( ((fr o (( ((Zr..)). Cu(00 Fit [(fr :h (xv 1.00 ve
ny
+ +Lor(0,fe) Sc) o ( - (Co (2) e(fe N0 (oe De
4ww f2y
(A.5)
where the last inequality is by the fact that the loss function is bounded in [0. 1]. Taking the supremum
over E T, (f+} C F at both sides, get G([z,)) } G( }) S 4wy /ng. A symmetric
argument shows that the reverse inequality, namely G( G( (xx)) S 4wv /ne, is also true. That is, we have shown
4w
[G((za)) G((za))
ne
This means that we can invoke McDiarmid's inequality get
2ea
G(((z))) - E[GG((=))) z I exp
E
16 /n?
for any & > 0, or equivalently
log(1/5) Nw
G((=+)) < 22V2.
2v2.
(A.6)
with probability at least 1 - ô for any ò E (0, To bound the expectation term, we use a standard symmetrization argument (see, e.g., Lemma 11.4 (Boucheron et al. 2013)) to get
T
(-e(fio((zu) yud ya) +((f)o(((w)) mul) ) Ett
Wi
/ NN.E[G((Za)) < 2 No
sup GEO
where the expectation is taken over the randomness in both the source datasets (z,,) ) and the i.idd sym- metric Rademacher random variables {et). Consider the function space 3 = ((), (fe) Φ. (fe) C F). Let
T
Mg == / No
Eti
Wt.
I o ((zu).
g == (ò. (ft)%) E Eg
1s
15

---

Published as a conference paper at ICLR 2022
be the empirical process indexed by the function space J. Conditional on the randomness in the data ( this a Rademacher process with sub-Gaussian increments:
X²
log Be (M, Ma3 < (g.
ZAA 0,g (o. (.. (o) E
2
where the pseudometric
d (g. No ΣΣ o(())))))) mai e(Te ((x)) ()s)))))))))))))))))))))) mai)
Now Σ
n
Thus, we can invoke Dudley's 's entropy integral inequality (see, c.g., Corollary 13.2 in (Boucheron al. 2013)) to get
L/" VlogN
E[sup Mg getd
1 (2x))
where 0 = 120, 1), metric Taking expectation over the randomness in {zu), we get
}), and N(9; d; E) is the e-covering number of & with respect to the pseudo-
B[G([z+)) S No /"
(og djee
(A.7)
We now bound the covering number of T. To do so, we define
n2e
Smm
N
8
R4.
The
where 6w,, is a point mass at Ft.. Let (Φ()) .(..) C + be an E-covering of & with respect to
(Q)), where N₂ = -N(;; e)).. This means that for any & E Φ there exists j {l...., Ne} such that
T Nw (o)(()) - (60) (xn))l < s².
|| 1 10) == ΣΣ
W
t=1
Now for each j E {I,..., No} and E {I...., T). let (F((.1) ,..., (...,,,(.. } be an s-covering of F with respect to (Q)) where #C is the pushforward of Q: by Φ6), and N = N(F; ((0))))))) has no dependence on due to the uniform entropy control from Assumption
This means that for any Sa E F,j (1,..., Ne). there exists k: ".....,N(), such that
I1.a - (
(())))))))))))))))))))))))))))))
((a (000 (2ns) E?
[1 =
Now, let us fix (0, (ff)TT E G. By construction, we can find j E {L..... v(.)) } for any 1 1st T such that
and ke
1 112 SE.
llfe ((iRt)) f(.nk) 122 Se, V1 ST (A.8)
Thus, we have
d2 (ø, (f ( (s((n))
me T No
Clfe (( - ((F(C)))) (ZW)) vi)) 2
ne
f(oo(au) f(iik) o (6(())))) (oxil) 2
T L
f(o((z)) - fc )())((=))) + fo o (x() - -(lite) o (
16

---

Published as a conference paper at ICLR 2022
T N -
T
[7 - (x,)))
+2I
Σ
wi 1
Σ
St (10
sl 2 (222))
nt
R4.
1=
2=1
i=1
2
- (2 + 22
Σ
Ilff f(3
12 (+())
nt
t=1
211 + 1)E2.
This yields
N(9:d: V2LL/ 133 1.) { (0l)) ( S
N. 1 S SN NN
= No (N()))
C4
CC
Tvr
E
from which we get
Nog.W9;::s S uo log(oo log 2(L3 + 1)) + Tuz 1o + 1)) + (vs + Tux) log( (1/e)
S (vo + Tv=) + log(1/=))
Plugging the above inequality to A.7 we get
(1+ //'' Vlog(1/EJde)
TvF
B[G({2+))] 2 N_ Vvo Tuy
No
The proof is concluded by plugging the above inequality to (A.6).
A.3.2 PROOF OF LEMMA A.2
Since @ is obtained from the source datasets {S is independent the target data Sp. Through- out the proof, we condition on the randomness in the source datasets, thus effectively treating Φ as fixed. Let fu,z E argmin foer Co(ō, f). We start by writing
1o) 6 Lo(d, Fn) Eo(a, Jo) Eo( Zo(0, Sa.z) f So,a) - Co
S Co Fo) Zo(o, fo) + fo.)) Co
s sup Lo fo) 26,, fo) 2. fo,ō) Lolo, FoEF
The right-hand side above is an empirical process indexed by fo E F. Using similar arguments as those appeared in the proof of Lemma A.I we have
Vs + log (1/))
sup Lo(ã, 5o) - Jo) + Eo(0, Lo(a, (o,))
JoEF
74
with probability at least 8-8, 8,
which is exactly the desired result.
A.4 PROOF OF THEOREM
The proof bears similarities to the proof of Theorem A.1 with additional complications in ensuring a uniform control over the learned weights. Let fo,ō E argmin Lo fo). Note that
fo) Lo(ds (
Col Fo) E fo) (2) fo. - -22
(6. So,5)
+ (02) (6, (o,)) Lo 1o,5) (/(D) L L2( -
17

---

Published as a conference paper at ICLR 2022
fo) - (e, 22 Lo(o. fo( Lo(00) -g/(0)) 2)(())
(A.9)
Let ft E argminn,ee2 fEF Le(o. ft). We then have
r
-
(o -0 12')
Σ we
(2(0.))) - -
(po)
Ft) 2. - 2. - $,00,170 130)
We
1=1
T
2
sup
Σ
wi (((..)))
- 1,000 - Lo (o(0m, f+i)
(E WEW
t=1
Let zu
(Eti yti), and let {Zis} be the source datasets formed by replacing Zuri, with Se Let
T
G((zn))) =
sup
Σ We
(c((0.))
ft) Lo fo) + 2, R,) Li ((0)
sE CF WEW,
Then conducting a similar calculation to what led to (A.5). we get
Awiy 8
(G((z)))-G((E))))) G((E+)))
ny
nT
where the last inequality is by the fact that w E W3 implies wt S B/T for any 1 S T. Now, invoking McDiarmid's 's inequality, we get
( S EG([Z=)) o BV
log(1/0) nf
(A.10)
with probability at least 1 8. Now, standard symmetrization argument plus application of Dudley's entropy integral bound (similar to what led to (A.7) gives
/ :(9:
B[G([za)) (nT (nT
(A.11)
where now & := ((0) ( (11) w) E $ (1)) C FF,WEE, and
da
(0) (.1 w) (6, ()))
nT
wi o 3r ((ar)))
82
nf
[( (()oo(x)))))) vi) e(Je a(
nf
T
- -5) T(xx)) sra)
+
32
12
((froolan) vn) ((Fo o H(
+ 1 W||
where the last inequality is by n4 n, B/T for any 1 1st5 T and 3 2 1. This means that can construct a Ce-covering of by the following two steps (where is an absolute constant only depending on I and I.)): (1) cover the space {(A, (f()=)) by the same construction AA.8)
18

---

Published as a conference paper at ICLR 2022
(2) construct an coverrng of Wa with at most (cT/e)" many points, where is an absolute constant. Overall, construct Cs covering y with (C+/s) (Cz/s)Pur (cT/E)² many points. Hence, we have
log N d):)
S (v+ + Tvz) + + T(1 log T log((//e)
S (vo (log(1o +
Plog T.
where the last inequality is by 2 1. Plugging this inequality back to A.I0 and A.11 we get
Σ - - - PV
T
Vip + log(1/0)
MF + log
nf
with probability at least 6. This means that under this high probability event, we can invoke the transferability assumption to conclude the existence of a specific @o with
( + log(1/0) (og(/1))) nT
MF + log T
1/20
c 2 CpB3/P.
Recalling A.9 we arrive at
Lo(ō, Fo) Lo
S Lo fo) (6, + 26 (o. Lo,0 sup Lo(00) (7(0))
50
+ + CosV/e. (vo (vb
+ log T 1/20
nf
n
I sup Joet
Colo,fo) (6, fo) + (22 (o. fo.z) Lold, fo,6)
+ sup Ca - -(((d))
VF + log T' +o(c,glle. (mD + Log(11/8) ( nT
1/20
n1
with probability 1 - 5. Since @ is independent of the second batch of target data {< Yoi :1 B2) and [Ba x no. nearly identical argument as that appeared in the proof of Lemma A.2 gives
foEF (olā. (12) (6, + (o. - Lol--oooooooo Lo(
log 1/8)
sup
no
with probability at least 5. We conclude the proof by invoking union bound.
B EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
Target Task
PoS
Predicate Detection
NER
Avg
Learning Paradigm
Single-Tash Learning
85.06
71.51
$4.96
70.51
Pre-Training
88.66
78.57
58.22
75.15
Weighted Pre- TeeaTrinn
89.31 #**
79.21 #*
60.09
76.20
Joint Training
87.23
74.90
59.55
73.89
Weighted Joint Training
90.71 a**
76.29
64.49
77.16
Table Compared to Table more training examples are considered here. Here we only consider the three tasks in Ontonotes 5.0, i.e., PoS tagging, predicate detection, and NER. For each setting, choose one task as the target task and the remaining two tasks as source tasks. We randomly choose 20K training sentences for each source task respectively. As for the target task, we randomly choose 500, 600, 1000 training sentences for PoS tagging, predicate detection, and NER respectively.
In this section, we briefly highlight some important settings in our experiments. More details can be found in our released code. It usually costs about half an hour to run the experiment for each setting (e.g. number in Table T GeForce RTX 2080 GPU.
19

---

Published as a conference paper at ICLR 2022
Setting
Poš NER
Poš Chunking NER
PoS Chunking Predicate Detection
Avg
Leurning Paradigm
Task Learning
68.47
68.47
78.00
71.65
Joint Training
65.70
67.12
79.81
70.38
Weighted Joint Training
69.58
69.73
80.38
73.23
Table 3: Additional experiments to illustrate the benefits of weighted joint training compared to joint training. For each setting, the task in the bold text is the target task and the remaining tasks source tasks. The improvement from weighted training in the joint training paradigm under various settings indicates the effectiveness of TAWT.
Ratio
1
2
16
32
64
128
Avg
Learning Paradign
Single Task Learning
30.08
30.08
30.08
30.08
30.08
30.08
30.08
30.08
30.08
Normalized Joint Training
38.35
36.53
42.37
54.18
62.08
63.58
62.85
62.33
52.81
Weighted Normalized Joint Training
42.31
42.13
51.64
59.08
63.46
64.86
64.08
63.65
56.40
Upper Bound
62.69
70.31
75.65
79.39
81.59
83.19
84.18
Table 4: The performance for the analysis the ratio between the training sizes of the source and target datasets in Fig. [T] In this part, we use NER as the target task and source tasks include PoS tagging and predicate detection. We keep the training size of the target task as 500 and change the ratio from 1 to 128 on a log scale. Single-task learning denotes learning only with the small target data, and upper bound denotes learning with the target data that has the same size as the overall size of all datasets in cross-task learning. "-" means that we do not have enough training data for the upper bound in that setting. The sampling strategy for training examples we used for the analysis a little different from that used in the experiments in Table I so that the performance of single-task learning for the same setting can be little different.
Data. In our experiments. we mainly use two widely-used NLP datasets, Ontontes 5.0 (Hovy et al. 2006) and CoNLL-2000 (Tjong Kim Sang & Buchholz 2000). Ontonotes 5.0 is a large multilingual richly annotated corpus covering a lot of NLP and we use the corresponding English annotations for three sequence tagging tasks, PoS tagging, predicate detection, and NER. CoNLL-2000 is shared task for another sequence tagging task, chunking. There are about 116K sentences, 16K sentences, and 12K sentences in the training. development, and test sets for tasks in Ontonotes 5.0. The average sentence length for sentences in Ontonotes 5.0 is about 19. As for CoNLL-2000, there are about 9K sentences and 2K sentences in the training and test sets. The corresponding average sentence length is about 24.
Tasks. We consider four common sequence tagging tasks in NLP, PoS tagging, chunking, predicate detection, and NER. PoS tagging aims to assign a particular part of speech, such as nouns and verbs, for each word in the sentence. Chunking divides sentence into syntactically related non- overlapping groups of words and assigns them with specific types, such as noun phrases and verb phrases. Predicate detection aims find the corresponding verbal nominal predicates for each sentence. NER seeks to locate and classify named entities mentioned in each sentence into pre-defined categories such as person names, organizations, and locations. Based on the above datasets, there are 50 labels in PoS tagging, 23 labels in chunking, 2 labels in predicate detection, and 37 labels in NER. As for the evaluation metric, we use accuracy for PoS tagging, span-level F1 for chunking, token-level Fl for predicate detection, and span-level Fl for NER.
The model. We use BERT as our basic model our main experiments. Specifically, we use the pre- trained case-sensitive BERT-base PyTorch implementation (Wolf et al., 2020). We use the common parameter settings for BERT. Specifically, the max length is 128, the batch size is 32, the epoch number is 4, and the learning rate is 5e In the BERT model, the task-specific function is the last-layer linear classifier, and the representation model is the remaining part.
'Specifically, in the main experiments, we first randomly sample 9K sentences and then randomly sample a specific number sentences (e.g. 500 sentences for NER) among the 9K sentences for the training set of the target task. In the analysis, we first randomly sample 100K sentences and then randomly sample a specific number of sentences (e.g. 500 sentences for NER) among the 100K sentences for the training set of the target task. We use different sampling strategy for training sentences in main experiments and analysis, simply because the largest training size of the data considered in the two situations different.
20

---

Published as a conference paper at ICLR 2022
Target Size
500
1000
2000
4000
8000
Avg
Learning Paradigm
Single-Tase Task Learning
30.08
54.26
68.05
74.77
79.20
61.27
Normalized Joint Training (Ratio=8)
54.18
64.69
71.33
76.95
80.37
69,50
Weighted Normalized Joint Training (Ratio=8)
59.08
66.00
72.30
76.95
80.64
70,99
Upper Bound
79.39
81.49
83.53
84,26
Normalized Joint Training (Ratio=10)
58.06
65.94
71.48
77.33
80.34
70.63
Weighted Normalized Joint Training (Ratio=10)
60.96
67.45
72.52
77.44
79.94
71.66
Upper Bound
79.70
82.32
83.39
84.84
Table 5: The performance for the analysis on the training size of the target dataset in Fig. 1 In this part, we use NER as the target task and source tasks include PoS tagging and predicate detection. We keep the ratio between the training sizes of the source and target datasets as 8 (10) and change the training size of the target dataset from 500 8000 on log scale. Single-task learning denotes learning only with the small target data, and upper bound denotes learning with the target data that has the same size as the overall size of all datasets in cross-task learning. means that do not have enough training data for the upper bound in that setting.
Target Tank
PoS
Chunking (0.32, 0.37)
Predicate Detection
NER
Leamiing Panudige
Weighted Pre Training
10.68, 0.31)
(0.33, 0:32)
(0.92, 0.08, 0.04)
Weighted Joint Training
10.04, 0,04, 0.03)
(0.04, 0.04, 0.05)
(0.04, 0.044 0.05)
(0.04, 0.04, 0.05)
0,89
0,87
0.87
0,87
Weighted Normalized Joint Training
(0.0005, 0,00004, 0.00154)
(0.0005, 0.0004)
(00022 0.002, 0.002)
(0.003, 0.003,, 0.003) 0.990
0.9986
0.9987
0,995
Table 6:
The final learned weights on tasks in the experiments in Table For each setting, the final weights on three source tasks are represented by a three-element tuple. The source tasks are always organized in the following order: PoS tagging. chunking, predicate detection, and NER. For example, the tuple (0.68, 0.01, 0.31) for the target task Pos tagging in the weighted pre- pre-training indicate that the final weights on the three source tasks (chunking, predicate detection, NER) are 0.68, 0.01, and 0.31 respectively, As for TANT in the joint training, we have extra weight on the target task, which is bold in the second line.
Cross-task learning paradigms. In our experiments, consider two cross-task learning paradigms, pre-training, and joint training. Pre-training first pre-trains the representation part on the source data and then fine-tunes the whole target model on the target data. Joint training uses both source and target data to train the shared representation model and task-specific functions for both source and target tasks the same time. As for the multi-task learning part in both pre- pre-training and joint training. we udopt the same multi-task learning algorithm as in MT-DNN (Liu et al.) 2019
Experiments with more target training examples. Compared to the main experiments in Sec. we further experiment with more training examples in the target data for three tasks in Ontonotes 5.0. Without chunking, we also use more training examples in the source tasks. As shown in Table 2 we can see that TAWT is still beneficial even with more training examples, though the relative improvement of TANT is smaller compared to that with fewer training examples.
Additional experiments to illustrate the benefits of TAWT. In this part, we conduct additional ex- periments with TANT on the joint training with fewer source tasks compared to the main experiments in Sec. 4 For simplicity, we make the target task bold to distinguish it from source tasks for each setting. For example, in the setting of PoS + Chunking NER, NER is the target task and the other two tasks source tasks. For PoS NER, we randomly sample 20K training sentences for PoS tagging and 2K training sentences for NER. For PoS + Chunking + NER, we randomly sample 20K, 9 and 2K training sentences for PoS tagging, chunking, and NER respectively. As for PoS + Chunking + Predicate Detection, we randomly sample 20K, 9K, and 2K training sentences for PoS tagging, chunking, and predicate detection respectively. As shown in Table 3, we can see that TAWT is still beneficial under these diverse settings, which is complement for our main experiments
in Sec. 4
WWe choose 9K training sentences for chunking because the training size of the chunking dataset is 8936.
21

---

Published as a conference paper at ICLR 2022
Target Task
PoS
Chunking
Predicate Detection
NER
Avg
Learning Paradigm
Weighted Pre- Training (fxee weights)
49.17
73.05
74.26
39.84
59.08
Weighted Pre- rre-riining (dynamic weights)
$1.17
73.41
75.77
46.23
61.64
Weighted Normalized Joint Training (lixed weights)
86.93
90.12
74.90
63.60
78.89
Weighted Normalized Joint Training (dynamic weights)
86.07
90.62
76.67
63.44
79.20
Table 7: Comparison between dynamic weights and fixed final weights. There are four tasks in total, PoS tagging, chunking, predicate detection, and NER. For each setting, we choose one task as the target task and the remaining three tasks are source tasks.
44
Single Task Learning
Single Task Learring
Pre-Traning
Join Training
43
Weighted-See Sample Pre-Trairing
Weighted-Samph Joint Training.
Perforaance (99
42
42
41
Performance
41
40
40
39
38
Figure 2: Extension to weighted-sample training. The weighted training algorithm can be easily extended from the weights on tasks to the weights samples. As for the experiments weighted- sample training, we use the PoS tagging on entity words as the source task and named entity classification entity words as the target task. Note that the settings for the weighted-sampe training are quite different from those for the weighted-task training in the remaining parts because the weighted-sample training is much more costly compared the weighted-task training.
Analysis of some crucial factors. As shown in Fig. I analyze two crucial factors that affect the improvement of TAWT. First, in general, we find that the improvement of TAWT in normalized joint training is larger when the ratio between the training sizes of the source and target dataset is smaller, though the largest improvement may not be achieved at the smallest ratio. Note that the improvement vanishes with a large ratio mainly because the baseline (normalized joint training) already good enough with large ratio. Second, the improvement from TAWT will decrease with the increase of the training size of the target data. In a word, TAWT is more beneficial when the performance of the base model is poorer, either with smaller target data with smaller ratio. The specific performance for experiments in our analysis on two crucial factors for TANT, the ratio between the training sizes of the source and target datasets, and the training size of the target dataset, can be found in Table and Table (e
Dynamic weights analysis. As for experiments in Table there are four tasks in total, PoS tagging, chunking, predicate detection, and NER. For each setting, we choose one task as the target task and the remaining three tasks are source tasks. The learned final weights To better understand our algorithm, we compare TAWT with dynamic weights and TAWT with fixed final weights. TANT with dynamic weights is the default setting for our algorithm, where weights on tasks for each epoch are different. As for TAWT with fixed weights, we simply initialize the weights as the final weights learned by our algorithm and fix the weights during the training. As shown in Table 7 we find that TAWT with dynamic weights (the default one) is slightly better than TANT with final fixed weights. It indicates that fixed weights might not a good choice weighted training, because the importance of difffrent source tasks may change during the training. In other words, it might be better to choose the weighted training with dynamic weights, where the weights automatically adjusted based on the state of the trained model.
tasks are shown Table 6
Extension to the weighted-sample training, As shown in Fig. 2 TAWT can be easily extended from the weights on tasks to the weights samples. We can that TAWT based veighted-sampl training is also beneficial for both pre-training and joint training. Because the weighted-samppe training is much more costly compared the weighted-task training, we choose much simpler
22

---

Published as a conference paper at ICLR 2022
70
STL
LTL with flp==0.0
60
LTL with flp=00.2
Performance (%)
LTL with f0p
50
LTL with f.p
LTL with f0p==088
LTL with flpp=1.0
40
30
20
10
10
102
103
10
The training size of the target task
Figure 3: Illustration for the task distance. We find that the source data is more beneficial when the task distance between the source data and the target data is smaller or the size of the target data is smaller. In this part, we keep the training size of the source task as 10000 and change the training size of the target task from 10 to 10000 in a log scale. STL denotes the single-task learning only with the target data. LTL denotes the learning to learn paradigm, where first learn the representations in the source data and then learn the task-specific function in the target data. For the learning to learn paradigm, we consider the source task with different flip rates from 0.0 to 1.0, where the flip rate is an important factor in generating the source data and lower flip rate indicates a smaller task distance between the source data and the target data.
setting here. In this part, we use the PoS tagging on entity words the source task and named entity classification on entity words as the target task. Note that both tasks we considered here are word-level classification tasks, i.e. predicting the label for given word in named entity. We still use the Ontontes 5.0 (Hovy et al. 2006) as our dataset. There are 50 labels in PoS tagging on entity words, and 18 labels in named entity classification on entity words. There are 37534 examples in the development set and 23325 examples in the test set for both source and target tasks. As for the training set, we randomly sample 1000 examples for the source task and 100 examples for the target task. As for the model, we use two-layer NNs with 5-gram features. The two-layer NNs have hidden size of 4096, ReLU non-lineat activation, and cross- cross-entropy loss. As for the embeddings, we use 50 dimensional Glove embeddings (Pennington al. 2014). The majority baseline for named entity classification on entity words on the test data is 17%. As for training models, the size of the training batch is 128, the optimizer is Adam (Kingma & Ba 2015) with learning rate 3e-1, and the number of training epochs is 200. As for updating weights on samples, we choose to update the weights every 5 epoch with the mirror descent in Eq. 2.8 and thus 40 updates on weights in total. In
this part, we approximate the inverse of the Hessian matrix ( EE wi V3L f
) in Eq. 2.7 by constant multiple of the identity matrix as in MAML (Finn et al. 2017). and choose the constant as 5. According to our experiments, the results of this approximation are similar to those of the approximation that we used in Sec. 4 The corresponding results are shown in Fig. 2 In the future, we also plan to group the instances and give each group a weight rather than each sample.
Settings for simulations on task distance. In this part, we first randomly generate 10K examples D. where the dimension of the inputs are 1000 and the corresponding labels are in 0- - 9 (10 classes). For each dimension of the input, is randomly sampled from a uniform distribution over |-0.5, 0.5). For each flip rate 9% E J0, 1]. we randomly choose 9% of the data and replace their labels with random label uniformly sampled from the 10 classes. The data with flip rate 4% is denoted as D.% (including Do.o for the original dataset). For each dataset D D9%. we train 2-layer NNs Mas with 100% accuracy and almost zero training loss on the dataset. The 2-layer NNs can then be used to generate data for the task T3% We use the To.0 the target task and change 9% from 0.0 to 1.0 as
In NER, we have 37 labels because each type of entity have two variants of labels (B/T for heside/inside) and one more extra-label O for non-entity words also considered.
23

---

Published as a conference paper at ICLR 2022
the source task. Based on the process of the data generation, we can see that the flip rate 9% plays an important role in the intrinsic difference between the source task 79% and the target task To.o- In general, we can expect that a smaller flip rate 9% indicates smaller task distance between the source task 75% and the target task Tn.-- This data generation process is inspired by the teacher-student network in (Hinton et al.) 2013). As shown in Fig. [3 we keep the training size of the source task as 10000 and change the training size of the target task from 10 to 10000 in log scale. For simplicity, we use the same architectures for both the teacher network that we used for generating data and the student network that we used for learning the data, i.e., a two-layer neural network with a hidden size of 4096, an input size of 1000, and an output size of 10. As for training. the size of the training batch is 100, the optimizer is Adam (Kingma & Ba 2015) with learning rate 3e -4 and the number of training epochs 100.
Details on using existing assets. As for the corresponding code and pre-trained models for BERT, we directly download it from the Github of huggingface whose license is Apache-2.0. and more details can be found in hhtpps:: / thub. com/ auggiingfaaee/trassssoooooo
As for the Ontonotes 5.0, we obtain the data from LDC. The corresponding license and other details can be found in https::
http://caaalll cat awt.taloo..o. Idc. upenn edu/ Ltt//DDC201111 11112, As for the CoNLL-2000 shared task, we download the data directly from the website and more details are in https: 7/www. c cip.s.ppooooccoo.... wantwerpen........ www.be/ccooolooooo 10tml10000000000000
C ON THE CHOICE OF EXPERIMENTAL SETTINGS
In this part, we clarify the choice of experimental settings based on the following perspectives.
TAWT can also be applied to cross-domain or cross-lingual signals. Compared to the above two types of signals, we think cross-task signals are more widespread and more difficult to be used efficiently. Therefore, in our experiments, we choose the most difficult and crucial signals, cross-task signals, to verify the effectiveness of TAWT. Another reason is that we didn't 't find any existing weighted training algorithms with theoretical guarantees for cross-task learning, while weighted training is common for domain adaptation, such as importance sampling.
Signal selection:
The choice of similar domains:
We choose to select tasks with similar domains. Ideally, TAWT can be used for cross-task signals in different domains languages. However, the gap between different domains or different languages can make cross-task learning more complicated. We instead consider cross-task learning in similar domains of English to simplify the settings. But similar domains didn't t mean that datasets overlap because we randomly sample sentences for different tasks independently without repetition, causing a small ratio of overlap in our case.
Area selection: We choose to experiment with NI. tasks because we think cross-task signals are more common in the NLP area and NLP tasks are more diverse compared to tasks in other areas.
Dataset selection: We choose Ontonotes 5.0 as main dataset because it is widely used, and provides large-scale expert annotations (on 2.9 million words) for a wide range of NLP tasks. This enables us to focus on learning with various tasks in similar domains. As for CoNLL-2000, add it mainly because we want to analyze the impact of chunking on NER.
Task selection:
On the one hand, sequence tagging is more challenging than classification tasks. On the other hand, the evaluation of sequence tagging is more reliable, compared with generative tasks. Another reason is that Ontonotes 5.0 mainly cover sequence tagging tasks.
We choose to use BERT in our main experiments because BERT is widely used and all SOTA models are slight improvements of BERT. For weighted sample training, instead choose two-layer NNs with 5-gram features because it is simple and fast. Similar results could be shown even if the model is more complex, but exhaustive experimentation is not our goal.
Model selection:
The choice of the low-resource setting: There is an intrinsic trade-off between the base performance of single-task learning and the relevant improvement of cross-task learning. Specifically, if the base performance of single-task learning is good enough, adding cross-task signals can introduce extra noise compared to its information. In our experiments, we simply consider a simple low-resource setting where few target examples are available. Actually, we can still see the effectiveness of TAWT in cross-task learning as long as the base performance is not too high, but the relative improvement of TAWT will be smaller compared to that in the low-resource setting.
24


Reviews:
[{"review_id": "_skTszPNpyz", "reviewer": "Reviewer_jFGp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed method is novel and well-motived.\n- The authors provide a theoretical guarantee of the proposed method.\n- Empirical results indicate that the proposed model greatly outperforms the vanilla methods in both pre-training and joint learning. The authors also open-source their implementation.\n\n\nWeaknesses\n- Add a section that introduces \"Related Works\" would be very helpful for readers to understand the background and the novelty of this work. For example, what're the most popular/advanced methods in cross-task learning? How is this method related to and different from them? What's the relationship between cross-task learning, joint learning, and multi-task learning?\n- The empirical evidence would be much more convincing if authors compare their model with other methods besides the vanilla pre-training and joint training. Many existing works that utilize different strategies to calculate the weight of different task has been introduced in the area of multi-task learning. Although this work is designed for \"cross-task learning\" instead of \"multi-task learning\", the existing methods can be straightforwardly used to address the same problem. So it would be great if the authors can either compare the proposed methods with existing works or explain why other existing works cannot be directly adapted to this problem. There are many great survey papers that summarize the related works (e.g., [A Comparison of Loss Weighting Strategies for Multi task Learning in Deep Neural Networks](https://ieeexplore.ieee.org/document/8848395) and [A Survey on Multi-Task Learning](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9392366&tag=1). )", "summary_of_the_paper": "This paper introduces a novel cross-task training method that weights different tasks by optimizing a representation-based task distance between the source and target tasks. They provide both theoretical guarantee and empirical evidence to corroborate the proposed method.", "summary_of_the_review": "This is a good paper with novel ideas, but it can be further improved by adding more backgrounds, related works, and comparisons with other state-of-the-art methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S6Xp_BWbpSL", "reviewer": "Reviewer_HcJj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Review note: As noted in the initial review assessment, I am less comfortable with the mathematics underlying section 3 and Appendix A. I will focus more on the empirical results and I hope other reviewers will help review those. This explains the lower confidence score. \n\nPros:\n\n- The method is well-motivated. It is clear what the method is aiming for and how it differs from existing approaches. The added theoretical guarantees are welcome.\n- The experimental setup is convincing and the results confirm the soundness of the approach. The settings are well described and I appreciated Section B.1. describing some of the rationale behind these choices.\n- The writing is very good.\n- The appendix is very rich in additional interesting experiments.\n\nCons:\n\n- Dynamic weights analysis not present in main text:\n\nTAWT is more complex than prior methods. Seeing the importance of dynamic weights throughout training (Table 8) is very relevant as it shows that even a strong choice of initial weights for $\\omega$ (e.g through hyperparam opt) may not be as good as TAWT.\nCurrently, this feels like it is missing from the main body and deserves to be emphasized earlier. \n\n- Weight initialization for joint training\nMore of a suggestion than a con: For joint training, I believe it would be better to use normalized joint training as the baseline. Indeed, initialization with task weights that depend on the sample size is a stronger baseline and more closely aligned with approaches used in practice. \nAn added benefit is that table 3 could then be removed (since it’d be the later rows of Table 1 and Table 2). The non normalized results could still be kept in appendix for completeness. This would also allow Table 8 and its discussion to be moved into the main body.\n\n- Minor typos: The writing is good overall, but there are still some minor typos:\ne.g: p20 Pre-training first pre-train**s** and then fine-tune**s**\n", "summary_of_the_paper": "This paper introduces Target-Aware Weighted Training (TAWT), a cross-task learning algorithm. In the two-step procedure popular for cross-task training, the weight vector $\\omega$ defining the relative weight of each source task is usually exogenous (i.e: a hyperparameter). In contrast, in TAWT it is part of the optimization procedure, with the weights depending on the proximity (throughout training) of each source task to the target task (hence the “target aware” name).\n\nThe authors first derive an algorithm that enables learning when $ \\omega $ is made endogenous in the case of (weighted) pre-training. They then show how it can naturally be extended for weighted joint-training by considering the source tasks as one of the source tasks we learn representations from. They then provide theoretical performance guarantees for TAWT.\n\nThe authors apply TAWT on 4 NLP tasks, using BERT as their base model. They evaluate their method on both full-data and limited-data settings on each target task, using the 3 other tasks as source tasks. They show that TAWT-pretraining and TAWT-joint training significantly outperform their unweighted counterparts across tasks. These improvements are more marked when target task data is scarce.\n\nThey also show that weight initialization can be particularly important when the different source datasets have different # examples (e.g: when training jointly on data abundant source tasks and the data-scarce target task). This specific finding is not very novel, but it is great to see TAWT-training also helping when initialization weights are chosen correctly. \n\nCritically, the authors also show the importance of varying task weights throughout training. Indeed, fixing the task weights to be the final weights from TAWT leads to worse results than training with TAWT. This suggests that it is important that task-weights vary throughout the training process.\n", "summary_of_the_review": "A rare mix of well-motivated method, theoretical guarantees, solid experimental work and convincing results. There are minor presentation improvements to be done but I think this is a strong paper otherwise. As noted earlier, I am not able to review carefully Sec 3/Appendix A so this the caveat to my review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CvFchBmxC_Z", "reviewer": "Reviewer_xsx7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new weighted training algorithm, TAWT, to learn the task-aware weights on tasks for better using the cross-task signals.  The author theoretically and empirically verified the effectiveness of TAWT. \nStrength:\n1. The paper was clearly written and give a good formal definition for the problem of weighted training.\n2. The paper can bring a new research interest for multi-task learning and transfer learning. \n3. The experiment designs are very good,which show the affects on different training data size, impact of the ratio between source tasks training samples and target task training samples, etc.\n\nWeakness:\n1. The paper introduces a representation-based task distance, but this distance is neglected in the analysis of Eq.(3.8). I think it could not be negligible. Otherwise, the upper bound become task-agnostic. \n2. The used four NLP tasks are closely related.  It's better to add an irrelevant task, such as sentiment analysis, to show the effect of weighted training.\n3. An illustration of weight vector should be provided.\n\nOthers:\n1. It's interesting to extend this idea for incorporating some unsupervised tasks, such as Masked language model.\n", "summary_of_the_paper": "The paper proposes a new weighted training algorithm, TAWT, to learn the task-aware weights on tasks for better using the cross-task signals.  The author theoretically and empirically verified the effectiveness of TAWT. \nThe paper can bring a new research interest for multi-task learning and transfer learning.", "summary_of_the_review": "The paper proposes a new weighted training algorithm, TAWT, to learn the task-aware weights on tasks for better using the cross-task signals.  The proposed method is quite effective for transfer learning and multi-task learning on small target data. \nThe weighted training is very important but there lacks good work in this direction. Therefore, this paper is great to give an attempt for task-aware weighted training.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1JfTzZ_Zf2e", "reviewer": "Reviewer_wLuT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed this paper a great deal. The presentation is very clean and the underlying algorithm and theory is interesting. The results look quite promising for the method.\n\nA few comments:\n\n[1] Are there connections to work that learns data weighting using multi-arm bandits? Specifically:\n\nAutomated Curriculum Learning for Neural Networks\nAlex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, Koray Kavukcuoglu\n\nIt would be interesting to discuss connections to this work - the ideas are different but there does seem to be some connection.\n\n[2] The choice of mirror descent is not critical, correct? For example projected gradient methods could be used instead? It would be useful to note this.\n\n[3] Given Assumption B, it seems clear that \\phi must be a vector, correct? This is a minor thing, but I think that was not specified earlier in the paper?\n\n[4] Definition 3.2 (transferability) could benefit from much more explanation. I’m also not sure if it’s quite correct as written. Should there be a \\forall quantifier, for example \\forall \\phi \\in \\Phi before Eq. 3.5? Also I’m not sure how this equation implies that the loss is “controlled by a polynomial…”. Finally, is there a reason that the expression (\\sum …) is positive? And if it can become negative, how can we take a power to 1/p? Finally, I have very little intuition about what values p will take in practice, or what p intuitively corresponds to.\n", "summary_of_the_paper": "The paper discusses an approach that learns to weight data from different tasks in pretraining or mutli-task learning. It also gives a VC/empirical-processes style analysis, which gives guarantees for the algorithm and insights about sample complexity. Finally it describes experiments on a number of NLP problems.\n", "summary_of_the_review": "An interesting algorithm, theory, and results. A few parts of the paper need clarification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to learn the task-specific weights in pretraining or mutli-task learning. It provides theoretical guarantees to the algorithm, as well as strong empirical results on several NLP problems. All the reviewers agreed that the work is interesting and the paper is well written. During the discussion period, the authors committed to address in the revised version (relatively minor) concerns raised by reviewers, including providing additional clarifications and additional comparisons to related methods. Overall, this is a strong paper that merits an acceptance.
Decision: Accept (Oral)