Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
Authors: Seanie Lee
Hae Beom Lee
Juho Lee
Sung Ju Hwang
Venue: ICLR 2022 Poster (2022)
Keywords: multilingual language model
gradient alignment

Abstract: Multilingual models jointly pretrained on multiple languages have achieved remarkable performance on various multilingual downstream tasks. Moreover, models finetuned on a single monolingual downstream task have shown to generalize to unseen languages. In this paper, we first show that it is crucial for those tasks to align gradients between them in order to maximize knowledge transfer while minimizing negative transfer. Despite its importance, the existing methods for gradient alignment either have a completely different purpose, ignore inter-task alignment, or aim to solve continual learning problems in rather inefficient ways. As a result of the misaligned gradients between tasks, the model suffers from severe negative transfer in the form of catastrophic forgetting of the knowledge acquired from the pretraining. To overcome the limitations, we propose a simple yet effective method that can efficiently align gradients between tasks. Specifically, we perform each inner-optimization by sequentially sampling batches from all the tasks, followed by a Reptile outer update. Thanks to the gradients aligned between tasks by our method, the model becomes less vulnerable to negative transfer and catastrophic forgetting. We extensively validate our method on various multi-task learning and zero-shot cross-lingual transfer tasks, where our method largely outperforms all the relevant baselines we consider.

Published as a conference paper at ICLR 2022
SEQUENTIAL REPTILE: INTER-TASK GRADIENT ALIGNMENT FOR MULTILINGUAL LEARNING
Seanie Lee Hae Beom Lee'; Juho Leel.2,
Sung Ju Hwang!
KAIST", AITRICS", South Korea
{Isnfamily02, haebeom.lee juholee, @jivangs2jekaist.cc
ABSTRACT
Multilingual models jointly pretrained on multiple languages have achieved re- markable performance on various multilingual downstream tasks. Moreover, models finetuned on a single monolingual downstream task have shown to gen- eralize to unseen languages. In this paper, first show that it is crucial for those tasks to align gradients between them in order to maximize knowledge transfer while minimizing negative transfer. Despite its importance, the existing methods for gradient alignment either have a completely different purpose, ignore inter-task alignment, or aim to solve continual learning problems in rather inefficient ways. As a result of the misaligned gradients between tasks, the model suffers from se- vere negative transfer in the form of catastrophic forgetting of the knowledge ac- quired from the pretraining. To overcome the limitations, we propose a simple yet effective method that can efficiently align gradients between tasks. Specifically, we perform each onner-optimization by sequentially sampling batches from all the tasks, followed by Reptile outer update. Thanks to the gradients aligned between tasks by our method, the model becomes less vulnerable to negative transfer and catastrophic forgetting. We extensively validate our method on various multi-task learning and zero-shot cross- -ongull transfer tasks, where method largely out- performs all the relevant baselines we consider.
INTRODUCTION
Multilingual language models (Devlin al., 2019; Conneau & Lample, 2019; Conneau al., 2020; Liu et al., 2020; Lewis et al., 2020a; Xue et al., 2021) have achieved impressive performance on a variety of multilingual natural language processing (NLP) tasks. Training model with multiple languages jointly can be understood as multi-task learning (MTL) problem where each language serves as distinct task to be learned (Wang al., 2021). The goal of MTL is to make use of relatedness between tasks to improve generalization without negative transfer (Kang et al., 2011; Kumar Daumé III, 2012; Lee et al., 2016; Wang al.. 2019b; 2020b). Likewise, when we train with downstream multilingual MTL objective, we need to maximize knowledge transfer between the languages while minimizing negative transfer between them. This is achieved by developing an effective MTL strategy that can prevent the model from memorizing task-specific knowledge not casily transferable across the languages.
Such MTL problem is highly related to the gradient alignment between the tasks, especially when we finetune a well- pretrrrerdd model like multilingual BERT (Devlin et al., 2019). We see from the bottom path of Fig. 1 that the cosine similarity between the task gradients (gradients of MTL losses individually computed for each task) tend to gradually decrease as we finetune the model with the MTL objective. It means that the model gradually starts memorizing task-specific (or language- specific) knowledge not compatible across the languages, which can cause a negative transfer from one language another. In case of finetuning the well-pretrained model, we find that it causes catastrophic forgetting of the pretrained knowledge. Since the pretrained model is the fundamen- tal knowledge shared across all NLP tasks, such catastrophic forgetting can severely degrade the performance of all tasks. Therefore, we want our model to maximally retain the knowledge of the pretrained model by finding a good trade-off between minimizing the downstream MTL loss
"qqull contribution

---

Published as a conference paper at ICLR 2022
Better trade-off hlw MTL loss vs.
egg.) Seq. Repéle
BERT Pretrained Model
egg) Etny-stoppiing
e.). Base MTL
Repilee
High Cos, Sim. High MTL Loss
PCGrad
Worse trado.off
RecAdam
Low Cos. Sim. Low MTL Loss
Figure 1: Concepts. Black arrows denote finetuning processes. The darker the part of the arrows, the lower the MTL loss. Upper and bottom path shows better and worse trade-off, respectively. Colored arrows denote task gradients. Blue and red color shows high and low cosine similarity, respectively. We demonstrate this concept with the actual experimental results in Fig. 7a.
and maximizing the cosine similarity between the task gradients, as illustrated in the upper path of Fig. In this paper, we aim to solve this problem by developing an MTL method that can efficiently align gradients across the tasks while finetuning the pretrained model.
There has been a seemingly related observation by Yu et al. (2020) that the conflict (negative co- sine similarity) between task gradients makes it hard to optimize the MTI. objective. They propose to manually alter the direction of task gradients whenever the task gradients conflict to each other. However, their intuition is completely different from ours. They manually modify the task gradi- ents whenever the gradient conflicts happen, which leads to more aggressive optimization of MTL objective. In case of finetuning a -reettart model, we find that it simply leads to catastrophic forgetting. Instead, we aim to make the model converge to a point where task gradients are naturally aligned, leading to less aggressive optimization of the MTL objective (See the upper path of Fig. 1).
Then a natural question is we can alleviate the catastrophic forgetting with early stopping. Our observation is that whereas early stopping can slightly increase cosine similarity some extent, it is not sufficient to find a good trade-off hetween minimizing MTL objective and maximizing cosine similarity to improve generalization (See Fig. 1). means that we may need either an implicit or ex- plicit objective for gradient alignment between tasks. Also, Chen et al. (2020) recently argue that we can mitigate catastrophic forgetting by adding & regularization
AdamW optimizer (Loshchilov & Hutter, 2019). They argue that the resultant optimizer penalizes & distance from the pretrained model during the finetuning stage. However, unfortunately we find that their method is not much effective in preventing catastrophic forgetting in the experimental setups we consider.
On the other hand, Reptile (Nichol et al., 2018) implicitly promotes gradient alignment between mini-batches within a task. Reptile updates a shared initial parameter individually for each task, such that the task gradients are not necessarily aligned across the tasks. In continual learning area, MER (Riemer et al., 2019) and La-MAML (Gupta et al., 2020) propose to align the gradients be- tween sequentially incoming tasks order to maximally share the progress on their objectives. However, they focus continual learning problems, they require explicit memory buffers to store previous task examples and align gradients with them, which is complicated and costly. Fur- ther, their methods rather inefficient in that the inner-optimization is done with batch size set to 1, which takes significantly more time than usual batch-wise training. Therefore, their methods are not straightforwardly applicable to multilingual MTL problems we aim to solve in this paper.
In this paper, we show that when we finetune well-pretrained model, it is sufficient to align gra- dients between the currently given downstream tasks in order retain the pretrained knowledge, without accessing the data used for pretraining or memory buffers. Specifically, during the finetun- ing stage, we sequentially sample
from all the downstream tasks at hand to perform single inner-optimization, followed by Reptile outer update. Then, we can efficiently align the gradients between tasks based on the implicit dependencies between the inner-update steps. This procedure, which we call Sequential Reptile, is simple yet effective method that can largely im- prove the performance of various downstream multilingual tasks by preventing negative transfer and catastrophic forgetting in an efficient manner. We summarize our contributions as follows.
We show that when finetuning well-pretrained model, gradients not aligned between tasks can cause negative transfer and catastrophic forgetting of the knowledge acquired from the pretrain- ing.
2

---

Published as a conference paper at ICLR 2022
To solve the problem, propose Sequential Reptile, simple yet effective MTL method that efficiently aligns gradients between tasks, thus prevents negative transfer and catastrophic forget- ting.
We extensively validate our method on various MTL and zero-shot cross-lingual transfer tasks, including question answering, named entity recognition and natural language inference tasks, in which our method largely outperforms all the baselines.
RELATED WORKS
Multi-task Learning The goal of MTL. to leverage relatedness between tasks for effective knowledge transfer while preventing negative interference between them (Zhang & Yeung, 2010; Kang et al., 2011; Lee et al., 2016). GradNorm (Chen et al., 2018) tackles task imbalance problem by adaptively weighting each task loss. Another line of literature propose to search Pareto optimal solutions which represent trade-offs between the tasks (Sener & Koltun, 2018; Lin et al., 2019). Re- cently, Yu et al. (2020) and Wang al. (2021) propose manually resolve the conflict between task gradients to more aggressively optimize the given MTL objective. However, their goal is completely different from ours because here we focus on preventing negative transfer by finding a model that can naturally align the task gradients without such manual modifications.
Multilingual Language Model Training a multilingual language model is typical example of multi-task learning. Most the previous works focus on jointly pretraining a model with hundreds of languages to transfer common knowledge between the languages (Devlin et al., 2019; Conneau & Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Lewis et al., 2020a; Xue et al., 2021). Some literature show the limitation of jointly training the model with multilingual corpora (Arivazhagan et al., 2019; Wang et al., 2020b). Several follow-up works propose to tackle the various accom- panying problems such as post-hoc alignment (Wang et al., 2019c; Cao et al., 2019), data balanc- ing (Wang et al., 2020a) and loss curvature-aware optimization to improve the performance of low resource languages (Li & Gong, 2021). In this paper. we focus on how finetune well pretrained multilingual language model by preventing catastrophic forgetting of the pretrained knowledge.
Zero-shot Cross Lingual Transfer Zero-shot gross-lingual transfer is train model with mono- lingual labeled data and evaluate it on some unseen target languages without further finetuning the model on the target languages. Nooralahzadeh al. (2020) utilize meta-learning learn how to transfer knowledge from high resource languages to low resource ones. Hu et al. (2021) and Pan al. (2021) leverage a set of paired sentences from different languages minimize the distance between the representation of the paired sentences. Instead, we partition the monolingual labeled data into groups and consider them as a set of tasks for multi-task learning.
train the model and
APPROACH
The goal of multi-task learning (MTL) is to estimate model parameter & that can achieve good performance across all the given 7 tasks, where each task f. = 1,..., We learn é by minimizing the sum of task losses.
....,,7 ' has task-specific data D,-
min
(I)
where 12))) is regularization term and is an associated coefficient.
Reptile We briefly review Reptile (Nichol et al., 2018), an efficient first-order meta-learning method suitable for large-scale learning scenario. We show that Reptile has approximate learning objective of the form in Eq. 1. Although Reptile is originally designed for learning a shared initial- ization, we can use the initialization é for actual predictions without any adaptation (Riemer al., 2019). Firstly, given a set of T tasks, we individually perform the task-specific optimization from G. Specifically, for each task = 1..... T, we perform the optimization by sampling mini-batches B(() "("),..,,!! BB(K) from task data D, and taking gradient steps with them.
0(h) = (k- ac(0 1) B(())
0 - =
(2)
80 (--1)
3

---

Published as a conference paper at ICLR 2022
Ø
4 update
Task update
Task update
Base MTL
Reptile
Sequential Reptle
Figure 2: Comparison between the methods.
for k 1,... K, where an inner-learning rate and o(4)) denotes the task-specific parameter of task t evolved from & by taking & gradient steps. After performing K gradient steps for all T tasks, we meta-update & as follows:
Φ 0
T Σ (G,(0), where (Q) = (K)
(3)
where 7/ denotes outer-learning rate. Nichol al. (2018) show that expectation of (G) over the random sampling of batches, which is the meta eeta-gradient of task t evaluated at ø, can be approximated as follows based on Taylor expansion:
K Σ
K
ac(6: B(a)) OΦ
OL(6;
[M)
C(o: B
(4)
OΦ
2
R=
where () denotes a dot product. We can see that [o) approximately minimizes the task- specific loss (first term) and maximizes the inner-products between gradients computed with differ- ent batches (second term). The inner-learning rate controls the trade-off between them. However, the critical limitation is that the dot product does not consider aligning gradients computed from different tasks. This is because each inner-learning trajectory consists of the batches B²") ",......"" sampled from the same task data Dr.
BB
3.1 SEQUENTIAL REPTILE
In order to consider gradient alignment across lasks well, we propose to let the inner-learning tra- jectory consist of mini- mini-batches randomly sampled from all tasks, which we call Sequential Reptile.
Unlike Reptile where we run T task-specific inner-learning trajectory (in parallel), single learning trajectory responsible for all tasks (See Figure 2). Specifically, for each inner-step k, we randomly sample task index l E {1....,7) and a corresponding mini-batch B(*). and then sequentially update ((k) follows.
have
g(h) = - (1(0
((0) = Φ.
where t4 ~ Cat(pI......
(5)
0g(k-1)
Cat(pi.... .... PT) is categorical distribution parameterized by PI,...,PT, the probability of each task to be selected. For example, we can let & (N,)* where N, is the number of training instances for task and y is some constant. After K gradient steps with Eq. 5, update Φ follows
7) MG((), where MG(
g(K)
(6)
Again, based on Taylor expansion, have the following approximate form for expectation of the meta-gradient (G over the random sampling of tasks (See derivation in Appendix C).
K
KK1 a LD
ac(o: B(t)) do
ac(o: B(.") )
E [G] ~
E
M
C(o: (()
(7)
d6
do
Note that the critical difference of Eq. 7 from Eq. 4 is that the dot product between the two gradients computed from the different tasks, te and t,. Such inter-task dependency appears as we randomly sample batches from all the tasks sequentially and compose single learning trajectory with them. As a result, Eq. 7, or Sequential Reptile promotes the gradient alignments between the tasks, pre- venting the model from memorizing language specific knowledge. It also means that the model can find a good trade-off between minimizing the MTL objective and inter-task gradient alignment, thereby effectively preventing catastrophic forgetting of the knowledge acquired from pretraining.
4

---

Published as conference paper at ICLR 2022
15
15
-1810
10
15
-10
-5
30
15
-1910
5
10
-10
-5
10
15
(a) MTL
(b) GradNorm
(c) PCGrad
(d) Grad Vac
20
20
70-
1.5
15
15
16
10
10
04
02
5
0.
-0
-04
-5
-5
-5
0.
1900
110
110
(h) Cos. sim. btw task gradients.
(c) RecAdam
(f) Reptile
(g) Seq. Reptile
Figure 3: (a)~(g) Loss surface and learning trajectory of each method. (h) Heatmap shows average pair-wise
cosine similarity between the task gradients.
4 EXPERIMENTS
We first verify our hypothesis with synthetic experiments. We then validate our method by solving multi-task learning (MTL) and zero-shot cross-lingual transfer tasks with large-scale real datasets.
Baselines We compare our method against the following relevant baselines.
1. STL:
Single Task Learning (STL) model trained on each single language.
MTL: Base MTL model of which objective Eq. without the regularizer (()).
3. GradNorm (Chen et al., 2018): This model tackles task imbalance problem in MTL. It prevents
the training from being dominated by a single task by adaptively weighting each task gradient.
4. PCGrad (Yu et al., 2020): This model aims to optimize MTL objective more aggressively by resolving gradient conflicts. Specifically, it projects task gradient onto the other task gradients if the inner product between them is negative.
Grad (Wang al., 2021): Similarly PCGrad, this model alters the task gradients to match the empirical moving average of cosine similarity between the task gradients.
RecAdam (Chen et al., 2020): model trained with RecAdam optimizer to prevent catas- trophic forgetting by penalizing & distance from the the pretrained model.
Reptile (Nichol al., 2018): A first-order meta-learning method suitable for large-scale learn- ing scenario. Unlike our method, Reptile performs ontrroopimizationn individually for each task. Sequential Reptile: Our method that can align gradients across the tasks by composing the inner-learning trajectory with all the tasks.
4.1 SYNTHETIC EXPERIMENTS
We first verify our hypothesis with the following synthetic experiments. We define three local optima an = (0, 10), 22 = (0, 0), and (10,0) in 2-dimensional space. Then, we define three tasks as minimizing each of the following loss functions w.r.t Φ ?
(.(0) 200 exp (0.2 | x; 12))),, for i= 1,2,3.
MTL objective is defined Σ L.(d), which optimize from the initialization 5).,5).
Results and analysis Fig. shows the MTL loss surface and the learning trajectory of each method. We observe that except for Reptile and Sequential Reptile, all the other baselines con- verge to one the MTL local minima, failing to find a reasonable solution that may generalize better across the tasks. While Reptile avoid such minimum, the resultant solution has very low cosine similarity (See Fig. 3h) because does not enforce gradient alignments between tasks. On the other hand, Figure 3h shows that Sequential Reptile tends to find reasonable trade-off

---

Published as a conference paper at ICLR 2022
between minimizing MTL loss and maximizing cosine similarity, thanks to the implicit enforcement of inter-task gradient alignment in Eq.
Table 1: Fl and EM score the TYDI-QA dataset for QA. The hest result for multilingual models is marked with bold while the underline denotes the best result among all the models including the monolingual model.
Question Answering (FIEMM
Method
ar
en
ko
le
Avp.
STL
60.5/65.9
70.9/584
72.07 59.2
76.2 63.7
82.7 70.6
61.0/508
73.4/ 56.5
78.4/ 70.1
81.1/66, /66.4
75.1 /62.4
MIL
79.7/64.7
74.7/640
72.8/6111
82.9/713
64.0/534
80.572.5 57.11
82.5/67.9
RecAdam
79.0/63.7
72.5762-4
73.5762.8
76.96119
82.1/72.2
61.6/51.5
53.9157.5
$04/72.9
82.8/68.1
76.2/61.4
GradNorm
78.8/62.
72,5/61.9
73.4/606
78.5
83.7/74.3
66.2/539
74.5157.7
70.7/72.7
83.3/ 68.9
76.8161.3
PCGrad
79.9/65.0
72.6/615
74.6/62.3
75.366.7
82.7/73.0
65.9/56.1
80.6733 74.2/57.3
82.5/68.1
76.8761.7
GradVac
71.5/592
73.0/612
63.8/334
30.6 72.6
82.2 67.5
76.3/
Replile
79.8765.2
75.1 64.1
62.8
78.8 65.3
83.8/73.6
65.7/55.9
75.5/
$1.47 72.9
83.1 / 68.5
77.5/65.2
Roptlle
$1.2/ 66.7
73.9762.6
76.7/653 65.2
79.4 66.3
84.9/747 74.7
68.0/58.
76.8 59.2
$2.9/ $2.9/7488
83.0 68.6
78.5 66.3
Table 2: score on WikiAnn dataset for NER. The best result multilingual models is marked with bold while the underline denotes the best result among all the models including the monolingual model.
Named Entity Recognition (FI)
Method
de
en
es
hi
jv
kk
mr
my
SW
te
tl
yo
Avg.
STL
90.3
85.0
92.0
89.7
59.1
88.5
89.4
61.7
90.7
80.1
96.3
77.7
83.3
MTL
83.4
77.8
87.6
82.3
77.7
87.5
82.2
75.7
87.5
78.8
83.5
90.8
82,9
RecAdam
84.5
80.0
88.5
82.7
85.3
88.5
84.4
70.3
89.0
81.6
87.7
91.6
84.5
GradNorm
83.6
77.5
87.3
82.8
78.3
87.8
81.3
73.5
85.4
78.9
83.6
91.4
82.6
PCGrad
83.8
78.5
88.1
81.7
79.7
87.8
81.7
74.4
85.9
78.4
85.7
92.3
83.1
GradVac
83.9
79.5
88.3
81.8
80.6
87.5
82.2
73.9
87.9
79.4
87.9
93.0
83.8
Reptile
85.9
82.4
90.0
86.3
81.3
81.4
86.8
61.8
90.6
72.7
92.8
93.0
83.7
Seq.Reptile
87.4
83,9
90.8
88.1
85.2
89.4
88.9
76.0
91.5
82.5
94.7
92.5
87.5
4.2 MULTI-TAST LEARNING
We next verify our method with large-scale real datasets. We consider multi-task learning tasks such as multilingual Question Answering (QA) and Named Entity Recognition (NER). For QA, we use "Gold passage" of TYDI-QA (Clark et al., 2020) dataset where a QA model predicts start and end position of answer from a paragraph for a given question. For NER, we use WikiAnn dataset (Pan et al., 2017) where a model classifies each token of a sentence into three classes. We consider cach language as a distinct task and train MTL models.
Implementation Details For all the experiments, we use multilingual BERT (Devlin et al., 2019) base model as backbone network. We fintune it with AdamW (Loshchilov & Hutter, 2019) opti- mizer, setting the inner-learning rate a to 3 10-5 We use batch size 12 for QA and 16 for NER, respectively. For our method, we set the outer learning rate ry to 0.1 and the number inner-steps K to 1000. Following Wang et (2021), for all the baselines, we sample eight tasks proportional to Pe x (N,) where N, is the number of training instances for task t. For our Sequential Reptile, we set the parameter of the categorical distribution in Eq. 5 to the same Pt.
Results We compare our method, Sequential Reptile against the baselines QA and NER tasks. Table shows the results of QA task. We see that our method outperforms all the baselines including STL most of the languages. Interestingly, all the other baselines but ours underperform STL Arabic language which contains the largest number of training instances (2 ~ 3 times larger than the other languages. See Appendix B for more information about data statistics). It implies that the baselines suffer from negative transfer while ours is relatively less vulnerable. We can observe essentially the same tendency for NER task, which is a highly imbalanced such that the low-resource languages can have around 100 training instance, while the high-resource languages can have around 5,000 to 20,000 examples. Table shows the results NER task. We see that all the baselines but ours highly degrade the performance means that they fail to address the imbalance problem properly and suffer from severe negative transfer. Even GradNorm is not effective in our experiments, which is developed to tackle the task imbalance problem by adaptively scaling task losses.
high resource languages de, en, es, hi, mr and tl, which

---

Published as a conference paper at ICLR 2022
Seg Reptile
Seg Reptie
Reptile
Seg Reptle
NTL
NTL
PCorad
PCGrad
PCGrd
PCGrd
Gradvac
Gradvec
Gradvix
GredNorm
Gradkom
Reptile
Reptie
RepUll
RecAdam
RecAdam
ReoMdam
Reckdom
SNEP
SWaP
04
04
21
60
0.2
56
0.
02
2.4
00
(a) TYDI-QA
(b) NER
(c) SQuAD
(d) XNLI
Figure 4: Average pair-wise cosine similarity between the gradients computed from different tasks.
MLM loss languages
MLM loss on unseen languages
distance from pretrained BERT
BERT
Seq.Reptile
Reptiee
Reptile
NTL
MTL
MTL
PCGrad
PCGrad
PCGrad
Graovac
Gradvoc
GradNorm
GradNorm
GradNorm
Reptile
Reptie
Reptile
RecAdam
RecAdam
RecAdam
15
10
2.0
30
40
(a)
(c)
Figure 5: (a, b) Average MLM loss on (a) seen and (b) unseen languages from Common Crawl dataset. We mask 15% tokens of sentences from Common Crawl dataset, which preprocessed and provided by Wenzek al. (2020), and compute the masked language modeling loss (MLM), which reconstruction loss of masked sentences. (c) & distance between the finetuned models and the initially pretrained BERT.
Analysis We next analyze the source of performance improvements. Our hypothesis is that our method can effectively filter out language specific knowledge when solving the downstream tasks, which prevents negative transfer and helps retain linguistic knowledge acquired from the pretraining.
Table evaluate the model, trained with TYDI-QA dataset, on five unseen languages from MI.QA dataset.
TYDI-QA MLQA (FI/EM)
Method
de
hi
vi
zh
Avg.
MTL
50.6/35.8
54.3 35.4
35.4
45.0
31.2
53.7 34.8
52.5 32.0
51.2/ 33.8
RecAdam
/
52.8/
41.7 19.7/29.0
52.2 34.6
/
49.2 33.0
GradNorm
51.7 36.6
54.9/36.3
44.4/30.4
55.3/37.1
52.9/
51.8/34.5
PCGrad
/6.5
/3.2
44.1/31.2
54.4/
52.4/31.5
51.1 34.2
GradVac
50.0/35..
53.0/35.2
41.6/28.6
52.9/34.9
51.3/31.2
49.8 33.1
Reptile
/8.2
56.1/38..
45.9
56.9
53.9/33.2
53.0 35.9
Reptiee
57.6 38.9
47.7 33.7
58.1/39.2
55.1/34.7
54.4/37.0
Firstly, we quantitatively measure the cosine similarity between the gradients computed from dif- ferent tasks in Fig. 4a (QA) and Fig. 4b (NER). We from the figure that Sequential Reptile shows the highest cosine similarity as expected, due to the approximate learning objective in Eq. 7. Such high cosine similarity between the task gradients implies that the model has captured the com- mon knowledge well transferable across the languages. This is in contrast to the other basclines whose task gradients can have even negative cosine similarity with high probability. Such different gradient directions mean that the current model has memorized the knowledge not quite transferable across the languages, which can cause negative transfer from one language to others.
As result of such negative transfer, we see from Fig. 5a and Fig. 5b that the baselines suffer from catastrophic forgetting. In those figures, high MLM losses on seen (Fig. 5a) and unseen languages (Fig. 5b) mean that the models have forgotten how to reconstruct the masked sentences, which is the original training objective of BERT. the other hand, our method shows relatively lower MLM loss. demonstrating its effectiveness in preventing negative transfer and thereby alleviating the catas- trophic forgetting. We further confirm this tendency in Fig. 5c by measuring the & distance from the initially pretrained BERT model. We that the distance is much shorter for our method than the baselines, which implies that ours can better retain the linguistic knowledge than the baselines.
We can actually test if the finetuned models have acquired knowledge transferable across the lan- guages by evaluating on some unseen languages without further finetuning. In Table 3, we test the finetuned QA models five unseen languages from MI.QA dataset (Lewis al., 2020b). Again,

---

Published as a conference paper at ICLR 2022
Table 4: Zero-shot cross lingual transfer from English (SQuAD) to six unseen languages (MLQA).
SQuAD - MIQA (FI/EM)
Method
de
hi
vi
Avg.
MTL
48.0 29.9
59.1 44.6
64.2 64.2/46.2
44.6/29.0
56.8 37.8
55.2/34.7
54.6 / 37,0
RecAdam
/
58.8 44.0
64.3/
4
57.9/
54.8/
54.7 / 37.0
GradNorm
48.7/31.
59.6/4466
64.8/46.3
47.2/31.2
57.2/37.8
55.4 55.0/33.9
PCGrad
/
59.2/44.1
65.4/46.1
41.0/26.0
57.4/7.4
54.6/34.0
54.4 36.2
GradVac
28.3/28.3
58.2/43.4
63.9/45.9
42,4/27.9
56.3
53.1 53.0/
SWEP
49.5/31.0
60.5/460
65.0/47.3
47.6/319
57.9
56.2/38.5 /3.9
Reptile
46.8/
58.7/44.4
65.1/47.5
41.5//27.
/6.1
/3.9
53.6 36.8
Seq.Reptile
54.8/34.3
60.7/45.9
67.1 48.77
33.6
60.7 40.8
57.3/36.5
58.2 39.3
Table 5: Zero shot cross lingual transfer from English (MNLI) fourteen unseen languages (XNLI).
MNLI + XNLI (Accuracy)
Method
bg
el
hi
th
vi
Avg.
MTL
65.1
68.8
71.3
66.4
74.3
72.6
59.2
68.8
50.1
52.8
61.8
57.9
69.4
69.0
64.8
RecAdam
63,5
66.9
69.4
65.2
72.7
72.2
58.8
67.3
49.9
51.9
61.0
56.5
67.9
67.7
63.6
GradNorm
64.0
68.2
70.6
67.0
74.1
72.9
58.8
68.0
48.9
53.2
61.0
56.8
70.3
69.3
64.5
PCGrad
64.0
68.7
69.8
66.5
74.3
71.8
59.4
68.3
51.0
53.1
60.7
57.4
69.7
69.3
64.5
GradVac
62.3
67.8
69.2
65.9
72.6
72.2
59.6
67.1
51.1
52.9
61.7
56.4
68.8
68.0
63.9
SWEP
64.8
69.4
70.5
67.1
74.8
74.4
58.7
69.7
49.2
53.7
60.2
57.1
69.8
68.4
64.8
Reptile
63.3
66.6
69.4
64.9
72.6
71.3
58.3
66.9
46.4
47.5
58.6
55.9
68.6
62.6
Rep.Reptile
67.2
69.3
71.9
67.8
75.1
74.1
60.6
69.5
51.2
55.1
63.8
59.1
70.8
69.6
66.9
Sequential Reptile outperforms all the baselines. The results confirm that the gradient alignment between tasks is key for obtaining common knowledge transferable across the languages.
4.3 ZERO-SHOT CROSS LINGUAL TRANSFER
We next validate our method on zero-shot cross lingual transfer task, motivated from the previous results that our method can learn common knowledge well transferable across the languages. We train a model only with English annotated data and evaluate the model on target languages without further finetuning on the target datasets. To utilize the methods of MTL, we cluster the data into four groups with Gaussian mixture model. We focus on QA and natural language inference (NLI) task. For QA, we train the model on SQuAD (Rajpurkar et al., 2016) dataset and evaluate the model on six languages from MLQA dataset (Lewis et al., 2020b). For NLI, we use MNLI (Williams et al., 2018) dataset source training dataset and test the model on fourteen languages from XNLI (Conneau et al., 2018) as a target languages.
Baselines As well as the baselines from the previous experiments, additionally include SWEP (Lee et al., 2021). It learns to perturb word embeddings and uses the perturbed input as extra training data, which is empirically shown to be robust against out-of distribution data.
Implementation Detail We finetune multilingual BERT-base model with AdamW. For QA, we the same hyperparameter in multi-task learning experiment. For NLI, use batch size 32 and choose the learning rate 3 10-5 or 5 10-5 with AdamW optimizer based on the performance on the validation set. For our model, we set the outer learning rate 17 to 0.1 and the number inner steps K to 1000. In order to construct multiple tasks from a single dataset, we cluster concatenation of questions and paragraphs from SQuAD or sentences from MNLI into four groups Following Aharoni & Goldberg (2020), we encode the paragraphs or sentences into hidden representations with pretrained multilingual BERT. Then, we reduce the dimension of hidden representations with PCA and run Gaussian mixture model to partition them into disjoint four clusters. Since the number of training instances are almost evenly distributed for each task, we sample mini-batches from all the four tasks for all the basclines and set Pt 1/4 for model.
Results and Analysis The results of zero-shot cross lingual transfer show essentially the same tendency as those of multi-task learning in the previous subsection. Firstly, our model largely out- performs all the baselines for QA and NLI tasks as shown in Table 4 and 5. To see where the performance improvements come from, we compute the average pairwise cosine similarity between the gradients computed from different tasks in Fig. 4c and 4d. Again, Sequential Reptile shows much higher similarity than the baselines, which implies that method can effectively filter out
8

---

Published as a conference paper at ICLR 2022
BERT
See Reptie
Seg Reptil
Ropticc
Seq Reptilc
PCGrad
PCCrad
PCGrad
PCGrad
Gradvac
Gradvac
Gradhac
Gradvac
GradNom
ooomm
GradNom
GradNorm
Reptile
Reptil
Reptil
Reptili
RecAdam
RecAdam
RecAdam
SWEP
SWEP
SWEP
SWEP
10
0.6
2.5
5.0
75
26
20
(a) QA
(b) NLI
(c)
QA
Figure 6: (a.b) Masked Language Modeling (MLM) loss. (c,d) & distance from the pretrained BERT model.
0.8
Similarity
(EM)
Exact Match (EM)
0.6
Fineturing
Cozine Simil
8.55
0.50
64
0.4
62
Reptile
Sequential Reptile
0.40
60
Sequential Reptile
50K Gradient Computations
Training
Learning Rate
(a)
(b)
(c)
Figure 7: (a) Trade-off shown in Fig. 1: average cosine similarity between task gradients MTL training loss. (b) Efffect the strength of gradient alignment: Average cosine similarity hetween task gradients and test performance (EM) vs. inner-learning rate. (c) Computational efficiency: Test performance (EM) vs. the cumulative count of (inner-) gradient steps used for training.
task-specific knowledge incompatible across the languages and thereby prevent negative transfer. As result, Fig. 6a and show that our method can better retain the linguistic knowledge obtained from the pretraining in terms of relatively lower MLM loss. Further, Fig. 6c and 6d confirm this tendency with shorter lo distance from the initially pretrained BERT model.
4.4 FURTHER ANALYSIS
(a) Trade-off shown in Fig. 1: In Fig. 7a, MTL loss and cosine similarity decrease as we finetune the model from top-right to the bottom-left corner. Meanwhile, Sequential Reptile shows much higher cosine similarities between task gradients the points of similar MTL losses, achieving better trade-off than Reptile. explains why simple early stopping cannot outperform Sequential Reptile (see Fig. 7c) that directly enforces gradient alignment across tasks.
Then the next question is, how can we further control the trade-off to maximize performance? According Eq. 7, the gradient alignment by increasing or decreasing the inner-learning rate a, respectively. Fig. 7b control the cosine similarity by varying a is around 3. 10 which is indeed the most commonly used value for finetuning the BERT model.
(b) Effect of the strength of gradient alignment:
strengthen weaken
shows that while
expected, the best- pest-performing
(c) Computational efficiency: Lastly, may suspect training Sequential Reptile takes signifi- cantly longer wall clock time because inner steps are not parallelizable as Reptile (See Fig. 2). This is not true. Fig. 7c shows that whereas base MTT. requires around 40K gradient computations to achieve 64 EM score, Sequential Reptile requires only around 15K. As a result, although we run Sequential Reptile with a single GPU at time, the wall-clock time becomes even comparable the base MTL that we run in parallel with 8 GPUs. Please see wall clock comparison on Appendix D.
CONCLUSION
We showed that when finetuning a well-pretrained language model, is important align gradients between the given set of downstream tasks to prevent negative transfer and retain linguistic knowl- edge acquired from the pretraining. We proposed simple yet effective method aligning gradients between tasks with efficiency. Specifically, instead of performing multiple inner-optimizationn sep- arately for each task, we performed a single inner-optine omerroptimmzzttin by sequentially sampling batches from all the tasks, followed by Reptile outer update. We extensively validated the efficacy of our method on various MTL and zero-shot gross-lingus transfer tasks, where ours largely outperformed all the baselines considered.

---

Published as a conference paper at ICLR 2022
Acknowledgements This work was supported by Institute of Information & communications Tech- nology Planning & Evaluation (IITP) grant funded by the Korea govermmen((MSII) (No.2019-0- Artificia Intelligence Graduate School (rogTam(KAIST)) the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Govern- ment MSIT (NRF-2018K) 1220111R1AA1109900
Samsung Research Funding Center of Samsung Electron- ics (No. SRFC-ITI502-51), Samsung Electronics (IO201214) 24-0011488100 145-01), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govern- Ment(MSIT) (No. 1021-0-02068, Artificial Intelligence Innovation Hub), the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021R) AFFFA11116655 and KAIST-NAVER Hypercreative AI Center.
REFERENCES
Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 7747-7763, 2020.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv: 1900::107.,0019, 2019.
Steven Cao, Nikita Kitaev, and Dan Klein. Multilingual alignment of contextual word representa- tions. In International Conference Learning Representations, 2019.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7870- 7881, 2020.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pp. 794-803. PMLR, 2018
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski. Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-secking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454 470, 2020.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059-7069, 2019.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475 2475, 2018
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudbary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsuper- vised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- crarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Lhok-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. Explicit alignment objectives for multilingual bidirectional encoders. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3633-3643, 2021.
Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. ICML, 2011.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC), 2011.
Alex Krizhevsky et Learning multiple layers of features from tiny images. arXiv preprint, 2009.
Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multi-task learning. In Proceedings of the 29th International Coference on International Conference on Machine Learn- ing. pp. 1723-1730, 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. arXiv preprint, 2015.
Giwoong Lee, Eunho Yang, and Sung Hwang. Asymmetric multi-task learning based on task relat- edness and loss. In International conference on machine learning, pp. 230-238. PMLR, 2016.
Scanie Lee, Minki Kang, Juho Lee, and Sung Ju Hwang. Learning to perturb word embeddings for ditt----isttibution QA. Proceedings of the 59th Anmual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Papers). Virtual Event, August 1-6, 2021, pp- 5583-5595. Association for Computational Linguisties, 2021.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang. and Luke Zettle- moyer. Pre-training via paraphrasing. Advances in Neural Information Processing Systems, 33, 2020a.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7315-7330, 2020b.
Xian Li and Hongyu Gong. Robust optimization for multilingual translation with imbalanced data. arXiv preprint arXiv:210..07639 2021
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in neural information processing systems, 32: 1200012070, 2019.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans- actions of the Association for Computational Linguistics, 8:726-742, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- fine-grained visual classification of aircraft. arXiv preprint 15555 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. arXiv preprint, 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arxiv: 1803.00 029999 2018
11

---

Published as a conference paper at ICLR 2022
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. Zero-shot cross-lingual transfer with meta learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4547-4562, 2020.
Lin Pan, Chung- Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. Multilingual bert post-pretrainnng alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp-
210-219, 2021.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1946-1958, 2017.
Jonathan Pilault, Amine El hattami, and Christopher Pal. Conditionally adaptive multi-task learn- ing: Improving transfer learning in NLP using fewer parameters & less data. In International Conference on Learning Representations, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representations, 2019.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 525-536, 2018.
Jaewoong Shin, Haebeom Lee, Boqing Gong, and Sung Ju Hwang. Large-scale meta-learning with continual trajectory shifting. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, Proceedings of Machine Learning Research. PMLR, 2021.
Samuel L. Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regular- ization in stochastic gradient descent. In International Conference on Learning Representations, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. arXiv preprint, 2011.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019a.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural ma- chine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8526-8537, 2020a.
Zirui Wang. Zihang Dai, Barnabás Póezos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11293-11302. Computer Vision Foundation IEEE, 2019b.
Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime G Carbonell. Cross-lingual alignment vs joint training: A comparative study and a simple unified framework. In International Conference on Learning Representations, 2019c.
Zirui Wang, Zachary C Lipton. and Yulia Tsvetkov. On negative interference in multilingual lan- guage models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4438 4450, 2020b.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and im- proving multi-task optimization in massively multilingual models. In International Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2022
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4003-4012, 2020.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 11 122-1122, 2018.
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the- art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint /wXXvv/708.00744:: 2017.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 483-498, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33, 2020.
Yu Zhang and Dit- Yan Yeung. A convex formulation for learning task relationships in multi-task learning. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, pp. 733-742, 2010.
13

---

Published as a conference paper at ICLR 2022
A ALGORITHM
We provide the pseudocode for Sequential Reptile described in section 3.1:
Algorithm Sequential Reptile
1:
Input: pretrained language model parameter Φ, a set of task-specific data {Di..... Drh, prob- ability vector "P......PT" PP) size a, outer-step size 1).
for categorical distribution, the number of inner steps K, inner step- 2: while not converged do
3: ((0) -
4: for k
K do 5: Sample task t ~ Cat(pi......
6: Sample mini-batch (!( from the dataset Dr
((k) g(k-11) ac(0(k-1); (k)) 00(k-1)
7:
8:
end for
9:
MG( = 0 Q(R)
10: ① - Ø 17 (G
end while
Table 6: The number of train/validation instances for each language from TYDI-QA dataset.
Split
ar
bn
en
id
ru
sw
te
Total
Train
14,805
2,390
3,696
6.855
5,702
1,625
6490
2,755
5,563 828
49.881
Val.
1,034
153
449
819
587
306
914
504
5,594
Table
The number of train/validation instances for each language from WikiAhn dataset.
Split
de
en
es
hi
jv
kk
mr
my
sw
te
yo
Total
Train
20,000
20,017
20,000
5,001
100
1,000
5,000
106
1,000
10,000
100
100
82,424
Val.
10,000
10.003
10,000
1.000
100
1,000
1,000
1,000
1,000
1,000
36,316
B DATASET
TYDI-QA (Clark et al., 2020) It is multilingual question answering (QA) dataset covering 11 languages, where a model retrieves passage that contains answer to the given question and find start and end position of the answer in the passage. Since focus extractive QA tasks, use "Gold Passage" in which ground truth paragraph containing answer to the question is pro- vided. Since some of existing tools break due the lack of white spaces for Thai and Japanese, the creators of the dataset does not provide the Gold Passage of those two languages, Following the conventional preprocessing for QA, we concatenate question and paragraph and tokenize it with BertTokenizerr (Devlin et al., 2019) which is implemented in transformers library (Wolf et al., 2020). We split the tokenized sentences into chunks with overlapping words. In Table we provide the number of preprocessed training and validation instances for each language.
WikiAhn (Pan et al., 2017) It a multilingual NER dataset which is automatically constructed from Wikipedia. We provide the number of instances for training and validation in Table 7.
XNLI (Conneau et al., 2018) It is the dataset for multilingual NLI task, which consists of 14 languages other than English. Since it targets for zero-shot cross-lingual transfer, there no training instances. It provides 7,500 human annotated instances for each validation and test set.
'ttpps: //gi thub. com/ passage_bast basel 1 i me/READMM md
resea rah reherrrrrchhdadaarseeaayydididd /bl ob/ hb/p:tter/mo/llll.oo
14

---

Published as a conference paper at ICLR 2022
Table 8: The number of train/validation instances for each language from MLQA dataset.
Split
ar
de
en
cs
hi
zh
Total
Val.
517
512
1,148
500
507
511
504
4,199
Test
5,335
4,517
11,590
5,253
4,918
5,495
5,137
42,245
MLQA (Lewis al., 2020b) is the dataset zero-shot multilingual question answering task. As XNLI dataset, it only provides only validation and test set for 6 languages other than English. In Table we provide data statistics borrowed from the original paper (Lewis et al., 2020b)
Common Crawl 100 (Conneau et al., 2020; Wenzek et al., 2020) It multilingual corpus sisting of more than 100 language that is used for pretraining XLM-R (Conneau et al., 2020) model. We download the preprocessed corpus provided by Wenzek et al. (2020). We sample 5,000 in- stances for each language and evaluate masked language modeling loss.
C IMPLICIT GRADIENT ALIGNMENT OF SEQUENTIAL REPTILE
In this section, we provide derivation implicit gradient alignment of Sequential Reptile in the equation 7. Firstly, we define the following terms from Nichol et al. (2018).
(00) Φ
(initial point)
(8)
ac(0(K 1); B(
(gradient obtained during SGD with mini-batch B(())
91 00(k-1)
(9)
((k) g agg
(sequence of parameter vectors)
(10)
OC(0(K-1)) 11;BB())))
(3(1))
(gradient at initial point with mini-batch with B(())
(11)
T L(o(k--1) 1); (()))
H
(Hessian initial point)
(12)
O62
where denotes learning rate for inner optimization. la ..y...,T'. 3 is task index and B(() is a mini-batch sampled from the task tę-
Corollary. After K steps of inner-updates, expectation of Ek Ite over the random sampling of tasks approximates the following:
K
K
K k-1
gbb
Je. Dt.
(13)
k=1
k=
where (,, denotes dot-product in Rd
Proof. We borrow the key idea from the theorem of Reptile (Nichol al., 2018) that task specific cnmer-optimization of Reptile implicitly maximizes inner products between mini-batch gradients within task. First, we approximate the gradient 9t, as follows.
gr, = aC(o: B
8 LL B()
(0(k) Φ) + +([10(()) -113)
d6
d
k-
= D.,
aHl 1k
91, + +
==1
k-1
= D.h
all i
Tt, +O(a3)
1=1
htttp://data.ttar stt.ittttttttttttttooooooo
15

---

Published as a conference paper at ICLR 2022
70
68
Exact Match
Exact Match
66
64
55
MTL
62
Sequential Reptile
50
60
10a
10 Inner Steps
10a
Wall Clock (Hour)
(a)
(b)
Figure 8: (a) Computational efficiency: Test performance (EM) vs. wall clock. (b) Effect of the the number of inner steps: Test performance (EM) vs the number of inner steps for Sequential Reptile.
After K gradient steps. we approximate the expectation of the meta-gras reta-gradient MG( over the task ti...., tk, where Ey ~ Cat(p)...... Pr) for k 1....,K K.
E [G[(0]]
Σ
9tk
2 E
Σ 9t.
H It
E
91
E
H
Mon
k=1;=1
2
Kgeu 9i,) Ob
E8
91
E
2
k=1j=1
K
(ghn 91)) ao
E
9t
2
1=
k=
K
K Æ-1
E
9tc
a
Σ
(gte 96,)
2 dΦ
km
k=15=1
K
K
ac(o; B(()))
Oc(6; B
((())
C(o;
8bb
OΦ
1=1
k=1
K
C(o: B(())
K
aL(h: ((())
ac(o:
Btg
Bli)
E
06
2
06
Similarly, Smith et al. (2021) show implicit regularization of SGD, After one epoch of SGD, it implicitly biases model to minimize difference between gradient of each example and full batch gradient. As result, it approximately aligns gradient of each instance with the full-batch gradient.
D FURTHER ANALYSIS
Computational efficiency In Figure Sa, we plot test Exact Match score as a function of wall clock time. Although training Sequential Reptile is not parallelizable, it shows tolerable computational efficiency compared to MTL model. For MTL, it takes about hour and 15 minutes while Sequential Reptile takes hour and 45 minutes to reach 64 EM score.
Inner steps As shown in Figure 8b, Sequential Reptile is robust to the number of steps for the inner optimization. It shows consistent performance with little variance.
16

---

Published as a conference paper at ICLR 2022
Table 9: We train MTL models on tasks from GLUE dataset report their performance.
GLUE
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
MTL
80.82
83,47
82.10
90.50
90.16
68.23
90.13
89.73
84,39
RecAdam
81.78
82.88
79.16
90.00
90.35
71.84
91.20
89.12
84.54
Reptile 82.07
78.81
81.30
89.29
87.57
72.56
88.76
88.55
83.61
Seq. Reptile
82.35
83.02
83.30
90.50
89.40
73.28
92.31
89.40
85.44
Table 10: We finetune MTL models with pretrained ResNet18 backbone on 8 image classification tasks.
Image Classification
Method
TIN-1
TIN-2
CIFAR100
Dogs
Aircraft
CUB
F-MNIST
SVHN
Avg.
MTL
56.26
$3.40
54.43
33.44
44.97
29.48
90.10
87.70
56.22
Reptile
23.56
23.28
9.64
12.20
10.02
6.61
60.87
20.86
20.88
Seq. Reptile
58.12
56.83
57.46
38.05
59.55
35.96
87.44
88.86
60.28
E ADDITIONAL EXPERIMENTS
In order to show that our model Sequential Reptile is generally applicable to various multi-task learning problems, additionally perform experiments monolingual text classification and im- age classification.
Text classification Following (Pilault et al., 2021), we train BERT base model on text classifica- tion tasks - CoLA, MNLI, MRPC, QNLI,QQP, RTE, SST2 and one text similarity score regression -STSB from GLUE dataset (Wang et al., 2019a). We share the BERT encoder across all the task and add linear layer on top of the encoder for each task. For Reptile, we use learned initialization with each task specific head for prediction test time. For STSB task, we use Pearson correlation coefficient to measure the performance of the baselines and ours. For the other task, we evaluate all the models with accuracy. As shown in Table 9, Sequential Reptile outperforms the Reptile and MTL tasks - CoLA, MRPC, RTE, SST2 and shows comparable performance on the other tasks.
Image classification Following the experimental setup from Shin et al. (2021), we use datasets - -Tiny- -Tiny-ImageNet (Le & Yang, 2015), CIFAR100 (Krizhevsky et al., 2009), Stanford Dogs (Khosla et al., 201 I), Aircraft (Maji et al., 2013), CUB (Wah et al., 201 1), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011). 1). We class-wisely divide Imny-mageNNet into two splits which are denoted as TIN-1 and TIN-2, respectively. and consider each split as distinct task, which results in total 8 tasks for multi-task learning image classification. We finetune ResNet18 (He et al., 2016) which is pretrained on ImageNet (Deng et al., 2009) with a randomly initialized linear classifier for each task. For all the models, the pretrained ResNet shared across tasks. As the previous experiments, use shared initialization of Reptile with task specific linear classifiers. We measure accuracy of each image classification task and report the average score.
As shown in Table 10, Sequential Reptile outperforms MTL and Reptile with large margin other than Fashion MNIST (F-MNIST). We observe that Reptile fails on image classification, which is contrast to text classification tasks where it shows reasonable accuracy compared to MTL model. We conjecture that Reptile needs adaptation to the target task since the set of image datasets are more heterogeneous than the set of text datasets.
F VISUALIZATION OF LEARNING TRAJECTORY
In Figure 9, 10, and 11, visualize learning trajectory of the MTL models and cosine similarity between task gradients with three different initialization as described in section 4.1. The similar pattern holds for all different initialization. All the MTL baselines except Reptile fall into of local optima. On the other hand, Sequential Reptile avoid such local minima while maximizing cosine similarities of task gradients.
17

---

Published as a conference paper at ICLR 2022
20
20
20
15
15
15
15
10
-5
-10
20
40
20
15
15
15
Aupsk Oun
19
-2
-5
-5
-18
10
-5
-10
Figure 9
13
20
3
-10
-10
-10
-10
10
OuD
10
02
-2
Figure 10
5
5
-15
-10
-10
-10
(8)
(b)
(c)
(d)
Oun
12
13
10
-10
(h)
(f)
Figure 11: (a)~(g) Loss surface learning trajectory of method. (h) Heatmap shows average pair-wise cosine similarity between the task gradients.
18


Reviews:
[{"review_id": "r7cg8EuMOHf", "reviewer": "Reviewer_ZkcB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The hypothesis is very interesting: is the negative transfer actually caused by the misalignments of task gradients? The paper provides some empirical evidences showing that such correlation (may or may not in causality relationship though) did exist, through a large number of synthetic and multilingual experiments. \n- The proposed sequential reptile algorithm is simple and effective. \n- The experiments are pretty extensive and show many interesting insights, which also verifies the effectivenesses of the proposed method. I think the following insights can be important:\n\na). Both reptile and sequential reptile show better generalizability across different tasks of the final optimized model.\n\nb). By implicitly adding the task gradient alignment, the resulted finetuned MTL model show less forgetting of the pretrained knowledge, and higher cosine similarities across tasks.\n\nc) The task gradient alignment does have correlation with the MTL task model performance, though it's still not sure if they are in causality relationship.\n\nWeaknesses:\n- The paper's findings would be much more generalizable if the experiments are also performed in the MTL setting of a single language, i.e., in a standard MTL setting.\n- There are still a few issues in the experiments:\n\na) In the synthetic experiment, instead of only showing the loss landscape of a single initialization point, it would be more convincing if such patterns hold for random initialization.\n\nb) In Table 1&2, the average performance is calculated as the mean performance across different languages. Such average doesn't consider the differences of the number of train/eval examples for different languages. Typically, we observed most MTL methods hurt for high-resource languages while improve on low-resource language. Simply averaging across different languages lead to a discrepancy between MTL model training and results reporting, as the loss is optimized as the sum of the example losses in training time. Therefore, simply averaging the performance across languages can be a misleading performance report as the actual empirical effectiveness per example may differ. But I guess this issue exists in most MTL papers. \n\nc) In Table 5 & 6, it would be good to include the single-task learning (STL) performance on zero-shot cross-lingual experiments. \n\nd) More experiments on some hyper-parameters could lead to better understanding of the algorithms, especially on inner and outer learning rates and inner gradient steps. \n", "summary_of_the_paper": "The paper proposed an optimization method for training multi-task learning models by increasing the task alignments. Their key assumption is that standard MTL can result in catastrophic forgetting of pretrained knowledge and lead to local task optimum in the finetuning stages. To avoid this, the paper proposed a method to jointly optimize the task losses and gradient alignments between tasks. Their key contributions are:\n- The paper empirically examined the question of whether worse alignments of task gradients can lead to negative transfer, though extensive experiments on multilingual tasks.\n- Based on such observation, the paper proposes a simple and effective optimization method for MTL that jointly optimizes the task losses and gradient alignments between tasks.", "summary_of_the_review": "Overall, I think the paper's merits outweigh its drawbacks. It provides some important questions and findings to multi-task learning. Therefore I would recommend accept the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gJbZJ5QCR47", "reviewer": "Reviewer_vCuD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n1.The paper is well-organized and clearly presented.\n\n2.The proposed Sequential Reptile is very simple and extensive experiments show that Sequential Reptile can significantly outperform baseline methods.\n\nWeaknesses\n\n1.Negative transfer and catastrophic forgetting are the common issues in the transfer learning field, which are not specific for multilingual learning. Besides, the proposed Sequential Reptile method is also general (simply modify the task sampling strategy to a sequential way during the inner optimization), which also has not special designs for the multilingual setting. I am curious why the authors choose this setting instead of more general transfer learning settings to verify the effectiveness of the proposed method.\n\n2.The authors claim that finetune a model with MTL objective will gradually enforce the model to memorize task-specific knowledge. Did the \nauthors compare with some transfer learning methods like multi-task adversarial training to learn task-invariant representations?\n\n3.The experiments only focus on two tasks, i.e., QA and NER. It would be better to verify on more multilingual tasks, e.g., XGLUE and Xtreme.\n\n4.Are the improvements statistically significant?\n", "summary_of_the_paper": "This paper proposes Sequential Reptile, a meta-learning method to perform inter-task gradient alignment to alleviate negative transfer and catastrophic forgetting. Empirically, experiments on both multi task learning and zero-shot cross-lingual transfer settings for QA and NER tasks demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "I carefully review this paper. I would vote for marginally below the acceptance threshold and make the final decision after seeing the rebuttal from authors .", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "73qUbbCmCrA", "reviewer": "Reviewer_8YbF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The experimental results are strong and very promising. On all settings considered in this paper, the proposed method significantly outperforms all methods. If this can generalize to other MTL problems, the method has a good potential for the MTL community.\n2. The paper provides plenty of experimental evidence to demonstrate the effectiveness of the method. These results also reveal its behavior in the multilingual setting that can be valuable for future research (although I do find some of them to be a bit hard to understand, please see below).\n\nCons:\n1. The novelty of the method is somewhat limited. The proposed modification is just a simple change to the original reptile. In addition, the resulting SR algorithm is actually very similar to a trivial MTL algorithm such that its inner loop is essentially a naive MTL algorithm without within-batch mixing, while the outer loop can be seen as a form of regularization. So indeed the method is a combination of naive MTL with regularization, as suggested in Eq (1). This, however, seems to deviate from the original design principle of Reptile.\n2. The intuition behind the method and the source of improvement is not clear to me. This is particularly important given my first point that the method is quite similar to existing methods at the first glance yet shows superiority. There are several points that need better clarification: (1) why are you considering Reptile specifically in the first place? what is the intuition of applying a few-shot learning method here? (2) the intuition behind SR is 'to consider gradient alignment across tasks as well'. While it does sound natural to do so, I find it hard to understand the overall framework with this modification. The original reptile (as well as MAML) aims to find a good initialization for all tasks and therefore considers the bi-level optimization setup which contains task-specific inner loop and task-universal outer loop. Here, this small modification of SR actually fundamentally changes this intuition and set task-universal goals for both inner and outer loops. This does not make sense to me and I wish to get more intuition on this point. (3) The source of improvement is said to be 'our method can effectively filter out language specific knowledge when solving the downstream tasks, which prevents negative transfer and helps retain linguistic knowledge acquired from the pretraining.' But why is that case? The results have shown that the resulting model is closer to the pretrained mBERT. But there is nothing specifically designed for SR to do so. The outer loop can be a potential cause yet regular reptile also has this step (while being much less similar in terms of L2 distance). I think it will be helpful if more analysis can be included to show the source of improvement.\n\nMinor issues/questions/suggestions:\n1. It seems to me that adding regularization towards the original mBERT is helpful for the performance. Can we add it for other baselines considered (or perhaps other related techniques introduced in [1,2,3])? Would that improve their performance? \n2. How about using within-batch mixing for the proposed method? How would that compare against the current version?\n\n[1] Noise Stability Regularization for Improving BERT Fine-tuning. Hua et al., 2021.\n[2] Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models. Leet et al., ICLR 2020.\n[3] SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. Jiang et al., ACL 2020.", "summary_of_the_paper": "The paper presents a new method for multi-task learning (MTL), namely sequential reptile (SR). The proposed method aims to align task gradients and mitigate negative interference among tasks during the finetuning process of multilingual BERT. Specifically, different from a regular Reptile which performs the inner loop on a single (sampled) task, the modified version does so on a batch of tasks, sampled according to some prior distributions. The paper argues that this simple modification can promote gradient alignment naturally. Empirically, the proposed SR method outperforms all baselines considered significantly, while using similar or even less computational resources.", "summary_of_the_review": "Overall, this paper presents a simple method with strong empirical improvements, yet the intuition and source of improvement can be somewhat confusing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0ztepnZgbTX", "reviewer": "Reviewer_gNwa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n- This paper tackles a problem which is less understood nor solved. The experiments results demonstrate the empirical significance of the work.\n- The method is simple and the authors provide insights into the problem and the methods, i.e. the adaptation of Reptile.\n- The experiments design are reasonable, specifically the zero-shot crosslingual transfer experiments are interesting.\n- The overall writing is clear and the content flows well.\n\nWeakness:\n- One major drawback of the paper is that it mixes two problems, i.e. catastrophic forgetting (of knowledge in pretrained model) vs. negative transfer between tasks (languages) during finetuning. It seems the authors focus on solving the second problem, but they also hypothesize that solving the second problem would improve the first problem as a by-product. For example, the paper made several claims such as \"As a result of such negative transfer, we see from Fig. 5a and Fig. 5b that the baselines suffer from catastrophic forgetting.\". However, such hypothesis (that the former caused the latter) was never verified. \n- Another limitation is the incompleteness in experimentation. For example, the authors use BERT base, which is a relative small model and draw conclusions such as \"It implies that the baselines suffer from negative transfer while ours is relatively less vulnerable.\". Several work has shown that negative transfer is related to model capacity, e.g. capacity bottleneck has been well studied in MTL literature and specifically in multilingual translation [1]. Therefore, I'd like to see experiments with larger models to verify that this method is still as effective in mitigating negative transfer as was observed in small models.\n- Related to the above point, it's unclear whether some of the worse performance from baselines are due to specific experiment setup which puts those approaches at disadvantage. For example, the authors shows that \"We see that all the baselines butours highly degrade the performance on high resource languages\" but the experiment was conducted in a manner where such observation may be expected, i.e. the authors adjust the sampling distribution to pt∝(Nt)^1/5, which upsamples low resource while downssamples high resource, that is, it intrinsically hurts high resource languages' performance.\n\n\nThe technical details in several places are not clear or lack of justification:\n1. In Eq. 1, the MTL loss is defined as the sum of single task losses. This definition is quite limited since minimizing this loss may not be the objective of multi-task learning. A more general MTL objective has been studied as a Pareto front of task losses [2].\n2. This work points out the importance of optimization setting which leads to different learning trajectories. However, the experiments were done with a specific set of hyperparameter, e.g. the authors chose 8 tasks (languages) in inner loop and sampled them with a specific distribution, the batch sizes were chosen differently for different tasks (e.g. QA vs. NLI). However, the author did not provide empirical justification that how senstive is the method to the choice of those hyperparameters.\n3. In Fig 5.c, Why RecAdam has the largest L2 distance given that it has explicit objective to regularize this metric?\n4. The clustering preprocessing in zero-shot crosslingual transfer experiments are very unclear. Why do you need to do the clustering to form tasks instead of using language as tasks as is in other experiments?\n5. Fig 7.c does not exactly capture the training overhead from the proposed method. Could you provide a wall clock based comparison (which is more precise) as you mentioned in the text?\n6. Related work misses relevant work on the optimization methods of multilingual training, e.g. [3] and [4].\n\n[1] Lepikhin, Dmitry, et al. \"GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.\" International Conference on Learning Representations. 2020.\n[2] Lin, Xi, et al. \"Pareto multi-task learning.\" Advances in neural information processing systems 32 (2019): 12060-12070.\n[3] Wang, Xinyi, Yulia Tsvetkov, and Graham Neubig. \"Balancing Training for Multilingual Neural Machine Translation.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n[4] Li, Xian, and Hongyu Gong. \"Robust Optimization for Multilingual Translation with Imbalanced Data.\" arXiv preprint arXiv:2104.07639 (2021).", "summary_of_the_paper": "This paper proposed a new training method to improve gradients alignment between languages (tasks) in multilingual finetuning of pretrained language models. The method is based on an existing algorithm, Reptile, which was originally developed for meta learning. The authors adapted it for multi-task learning and verified its effectiveness in multilingual finetuning.", "summary_of_the_review": "This paper proposes a simple method to improve multilingual finetuning of pretrained language models. The proposed solution and its effectiveness indicated by the experiments results has potential for enough technical significance. However, the current version of the paper has room for improvement in terms of experiment completeness in order to draw conclusion on the practical value of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a gradient alignment approach to alleviate negative transfer and catastrophic forgetting for multitask and cross lingual learning. Experiments on many domains and datasets demonstrate the efficacy of the proposed approach.

All reviewers agree that the simplicity of the proposed method is a strength of the paper and the experiments are promising. They have suggestions to improve the experiments section, which I believe the authors have addressed in their rebuttal by adding GLUE, image classification, and statistical significance tests, among others. 

I recommend accepting this paper for ICLR.
Decision: Accept (Poster)