Year: 2022

Abstract:
Multilingual models jointly pretrained on multiple languages have achieved remarkable performance on various multilingual downstream tasks. Moreover, models finetuned on a single monolingual downstream task have shown to generalize to unseen languages. In this paper, we first show that it is crucial for those tasks to align gradients between them in order to maximize knowledge transfer while minimizing negative transfer. Despite its importance, the existing methods for gradient alignment either have a completely different purpose, ignore inter-task alignment, or aim to solve continual learning problems in rather inefficient ways. As a result of the misaligned gradients between tasks, the model suffers from severe negative transfer in the form of catastrophic forgetting of the knowledge acquired from the pretraining. To overcome the limitations, we propose a simple yet effective method that can efficiently align gradients between tasks. Specifically, we perform each inner-optimization by sequentially sampling batches from all the tasks, followed by a Reptile outer update. Thanks to the gradients aligned between tasks by our method, the model becomes less vulnerable to negative transfer and catastrophic forgetting. We extensively validate our method on various multi-task learning and zero-shot cross-lingual transfer tasks, where our method largely outperforms all the relevant baselines we consider.

heading: 1 INTRODUCTION
text: Multilingual language models (Devlin et al., 2019; Conneau & Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Lewis et al., 2020a; Xue et al., 2021) have achieved impressive performance on a variety of multilingual natural language processing (NLP) tasks. Training a model with multiple languages jointly can be understood as a multi-task learning (MTL) problem where each language serves as a distinct task to be learned (Wang et al., 2021). The goal of MTL is to make use of relatedness between tasks to improve generalization without negative transfer (Kang et al., 2011; Kumar & Daumé III, 2012; Lee et al., 2016; Wang et al., 2019b; 2020b). Likewise, when we train with a downstream multilingual MTL objective, we need to maximize knowledge transfer between the languages while minimizing negative transfer between them. This is achieved by developing an effective MTL strategy that can prevent the model from memorizing task-specific knowledge not easily transferable across the languages.
Such MTL problem is highly related to the gradient alignment between the tasks, especially when we finetune a well-pretrained model like multilingual BERT (Devlin et al., 2019). We see from the bottom path of Fig. 1 that the cosine similarity between the task gradients (gradients of MTL losses individually computed for each task) tend to gradually decrease as we finetune the model with the MTL objective. It means that the model gradually starts memorizing task-specific (or languagespecific) knowledge not compatible across the languages, which can cause a negative transfer from one language to another. In case of finetuning the well-pretrained model, we find that it causes catastrophic forgetting of the pretrained knowledge. Since the pretrained model is the fundamental knowledge shared across all NLP tasks, such catastrophic forgetting can severely degrade the performance of all tasks. Therefore, we want our model to maximally retain the knowledge of the pretrained model by finding a good trade-off between minimizing the downstream MTL loss
∗Equal contribution
and maximizing the cosine similarity between the task gradients, as illustrated in the upper path of Fig. 1. In this paper, we aim to solve this problem by developing an MTL method that can efficiently align gradients across the tasks while finetuning the pretrained model.
There has been a seemingly related observation by Yu et al. (2020) that the conflict (negative cosine similarity) between task gradients makes it hard to optimize the MTL objective. They propose to manually alter the direction of task gradients whenever the task gradients conflict to each other. However, their intuition is completely different from ours. They manually modify the task gradients whenever the gradient conflicts happen, which leads to more aggressive optimization of MTL objective. In case of finetuning a well-pretrained model, we find that it simply leads to catastrophic forgetting. Instead, we aim to make the model converge to a point where task gradients are naturally aligned, leading to less aggressive optimization of the MTL objective (See the upper path of Fig. 1).
Then a natural question is if we can alleviate the catastrophic forgetting with early stopping. Our observation is that whereas early stopping can slightly increase cosine similarity to some extent, it is not sufficient to find a good trade-off between minimizing MTL objective and maximizing cosine similarity to improve generalization (See Fig. 1). It means that we may need either an implicit or explicit objective for gradient alignment between tasks. Also, Chen et al. (2020) recently argue that we can mitigate catastrophic forgetting by adding `2 regularization to AdamW optimizer (Loshchilov & Hutter, 2019). They argue that the resultant optimizer penalizes `2 distance from the pretrained model during the finetuning stage. However, unfortunately we find that their method is not much effective in preventing catastrophic forgetting in the experimental setups we consider.
On the other hand, Reptile (Nichol et al., 2018) implicitly promotes gradient alignment between mini-batches within a task. Reptile updates a shared initial parameter individually for each task, such that the task gradients are not necessarily aligned across the tasks. In continual learning area, MER (Riemer et al., 2019) and La-MAML (Gupta et al., 2020) propose to align the gradients between sequentially incoming tasks in order to maximally share the progress on their objectives. However, as they focus on continual learning problems, they require explicit memory buffers to store previous task examples and align gradients with them, which is complicated and costly. Further, their methods are rather inefficient in that the inner-optimization is done with batch size set to 1, which takes significantly more time than usual batch-wise training. Therefore, their methods are not straightforwardly applicable to multilingual MTL problems we aim to solve in this paper.
In this paper, we show that when we finetune a well-pretrained model, it is sufficient to align gradients between the currently given downstream tasks in order to retain the pretrained knowledge, without accessing the data used for pretraining or memory buffers. Specifically, during the finetuning stage, we sequentially sample mini-batches from all the downstream tasks at hand to perform a single inner-optimization, followed by a Reptile outer update. Then, we can efficiently align the gradients between tasks based on the implicit dependencies between the inner-update steps. This procedure, which we call Sequential Reptile, is a simple yet effective method that can largely improve the performance of various downstream multilingual tasks by preventing negative transfer and catastrophic forgetting in an efficient manner. We summarize our contributions as follows.
• We show that when finetuning a well-pretrained model, gradients not aligned between tasks can cause negative transfer and catastrophic forgetting of the knowledge acquired from the pretraining.
• To solve the problem, we propose Sequential Reptile, a simple yet effective MTL method that efficiently aligns gradients between tasks, thus prevents negative transfer and catastrophic forgetting.
• We extensively validate our method on various MTL and zero-shot cross-lingual transfer tasks, including question answering, named entity recognition and natural language inference tasks, in which our method largely outperforms all the baselines.

heading: 2 RELATED WORKS
text: Multi-task Learning The goal of MTL is to leverage relatedness between tasks for effective knowledge transfer while preventing negative interference between them (Zhang & Yeung, 2010; Kang et al., 2011; Lee et al., 2016). GradNorm (Chen et al., 2018) tackles task imbalance problem by adaptively weighting each task loss. Another line of literature propose to search Pareto optimal solutions which represent trade-offs between the tasks (Sener & Koltun, 2018; Lin et al., 2019). Recently, Yu et al. (2020) and Wang et al. (2021) propose to manually resolve the conflict between task gradients to more aggressively optimize the given MTL objective. However, their goal is completely different from ours because here we focus on preventing negative transfer by finding a model that can naturally align the task gradients without such manual modifications.
Multilingual Language Model Training a multilingual language model is a typical example of multi-task learning. Most of the previous works focus on jointly pretraining a model with hundreds of languages to transfer common knowledge between the languages (Devlin et al., 2019; Conneau & Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Lewis et al., 2020a; Xue et al., 2021). Some literature show the limitation of jointly training the model with multilingual corpora (Arivazhagan et al., 2019; Wang et al., 2020b). Several follow-up works propose to tackle the various accompanying problems such as post-hoc alignment (Wang et al., 2019c; Cao et al., 2019), data balancing (Wang et al., 2020a) and loss curvature-aware optimization to improve the performance of low resource languages (Li & Gong, 2021). In this paper, we focus on how to finetune a well pretrained multilingual language model by preventing catastrophic forgetting of the pretrained knowledge.
Zero-shot Cross Lingual Transfer Zero-shot cross-lingual transfer is to train a model with monolingual labeled data and evaluate it on some unseen target languages without further finetuning the model on the target languages. Nooralahzadeh et al. (2020) utilize meta-learning to learn how to transfer knowledge from high resource languages to low resource ones. Hu et al. (2021) and Pan et al. (2021) leverage a set of paired sentences from different languages to train the model and minimize the distance between the representation of the paired sentences. Instead, we partition the monolingual labeled data into groups and consider them as a set of tasks for multi-task learning.

heading: 3 APPROACH
text: The goal of multi-task learning (MTL) is to estimate a model parameter φ that can achieve good performance across all the given T tasks, where each task t = 1, . . . , T has task-specific data Dt. We learn φ by minimizing the sum of task losses.
min φ T∑ t=1 L(φ;Dt) + λΩ(φ) (1)
where Ω(φ) is a regularization term and λ ≥ 0 is an associated coefficient. Reptile We briefly review Reptile (Nichol et al., 2018), an efficient first-order meta-learning method suitable for large-scale learning scenario. We show that Reptile has an approximate learning objective of the form in Eq. 1. Although Reptile is originally designed for learning a shared initialization, we can use the initialization φ for actual predictions without any adaptation (Riemer et al., 2019). Firstly, given a set of T tasks, we individually perform the task-specific optimization from φ. Specifically, for each task t = 1, . . . , T , we perform the optimization by sampling mini-batches B(1)t , . . . ,B (K) t from task data Dt and taking gradient steps with them.
θ (0) t = φ, θ (k) t = θ (k−1) t − α
∂L(θ(k−1)t ;B (k) t )
∂θ (k−1) t
(2)
for k = 1, . . . ,K, where α is an inner-learning rate and θ(k)t denotes the task-specific parameter of task t evolved from φ by taking k gradient steps. After performing K gradient steps for all T tasks, we meta-update φ as follows:
φ← φ− η · 1 T T∑ t=1 MGt(φ), where MGt(φ) = φ− θ(K)t (3)
where η denotes an outer-learning rate. Nichol et al. (2018) show that expectation of MGt(φ) over the random sampling of batches, which is the meta-gradient of task t evaluated at φ, can be approximated as follows based on Taylor expansion:
E [MGt(φ)] ≈ ∂
∂φ E  K∑ k=1 L(φ;B(k)t )− α 2 K∑ k=1 k−1∑ j=1 〈 ∂L(φ;B(k)t ) ∂φ , ∂L(φ;B(j)t ) ∂φ 〉 (4) where 〈·, ·〉 denotes a dot product. We can see that E[MGt(φ)] approximately minimizes the taskspecific loss (first term) and maximizes the inner-products between gradients computed with different batches (second term). The inner-learning rate α controls the trade-off between them. However, the critical limitation is that the dot product does not consider aligning gradients computed from different tasks. This is because each inner-learning trajectory consists of the batches B(1)t , . . . ,B (K) t sampled from the same task data Dt.

heading: 3.1 SEQUENTIAL REPTILE
text: In order to consider gradient alignment across tasks as well, we propose to let the inner-learning trajectory consist of mini-batches randomly sampled from all tasks, which we call Sequential Reptile.
Unlike Reptile where we run T task-specific inner-learning trajectory (in parallel), now we have a single learning trajectory responsible for all T tasks (See Figure 2). Specifically, for each inner-step k, we randomly sample a task index tk ∈ {1, . . . , T} and a corresponding mini-batch B(k)tk , and then sequentially update θ(k) as follows.
θ(0) = φ, θ(k) = θ(k−1) − α ∂L(θ(k−1);B(k)tk )
∂θ(k−1) , where tk ∼ Cat(p1, . . . , pT ) (5)
Cat(p1, . . . , pT ) is a categorical distribution parameterized by p1, . . . , pT , the probability of each task to be selected. For example, we can let pt ∝ (Nt)q whereNt is the number of training instances for task t and q is some constant. After K gradient steps with Eq. 5, we update φ as follows
φ← φ− η ·MG(φ), where MG(φ) = φ− θ(K) (6) Again, based on Taylor expansion, we have the following approximate form for expectation of the meta-gradient MG(φ) over the random sampling of tasks (See derivation in Appendix C).
E [MG(φ)] ≈ ∂ ∂φ E  K∑ k=1 L(φ;B(k)tk )− α 2 K∑ k=1 k−1∑ j=1 〈 ∂L(φ;B(k)tk ) ∂φ , ∂L(φ;B(j)tj ) ∂φ 〉 (7) Note that the critical difference of Eq. 7 from Eq. 4 is that the dot product between the two gradients is computed from the different tasks, tk and tj . Such inter-task dependency appears as we randomly sample batches from all the tasks sequentially and compose a single learning trajectory with them. As a result, Eq. 7, or Sequential Reptile promotes the gradient alignments between the tasks, preventing the model from memorizing language specific knowledge. It also means that the model can find a good trade-off between minimizing the MTL objective and inter-task gradient alignment, thereby effectively preventing catastrophic forgetting of the knowledge acquired from pretraining.

heading: 4 EXPERIMENTS
text: We first verify our hypothesis with synthetic experiments. We then validate our method by solving multi-task learning (MTL) and zero-shot cross-lingual transfer tasks with large-scale real datasets. Baselines We compare our method against the following relevant baselines. 1. STL: Single Task Learning (STL) model trained on each single language. 2. MTL: Base MTL model of which objective is Eq. 1 without the regularizer Ω(φ). 3. GradNorm (Chen et al., 2018): This model tackles task imbalance problem in MTL. It prevents
the training from being dominated by a single task by adaptively weighting each task gradient. 4. PCGrad (Yu et al., 2020): This model aims to optimize MTL objective more aggressively by
resolving gradient conflicts. Specifically, it projects a task gradient onto the other task gradients if the inner product between them is negative.
5. GradVac (Wang et al., 2021): Similarly to PCGrad, this model alters the task gradients to match the empirical moving average of cosine similarity between the task gradients.
6. RecAdam (Chen et al., 2020): A model trained with RecAdam optimizer to prevent catastrophic forgetting by penalizing `2 distance from the the pretrained model.
7. Reptile (Nichol et al., 2018): A first-order meta-learning method suitable for large-scale learning scenario. Unlike our method, Reptile performs inner-optimization individually for each task.
8. Sequential Reptile: Our method that can align gradients across the tasks by composing the inner-learning trajectory with all the tasks.

heading: 4.1 SYNTHETIC EXPERIMENTS
text: We first verify our hypothesis with the following synthetic experiments. We define three local optima x1 = (0, 10), x2 = (0, 0), and x3 = (10, 0) in a 2-dimensional space. Then, we define three tasks as minimizing each of the following loss functions w.r.t φ ∈ R2.
Li(φ) = −200 · exp (0.2 · ‖φ− xi‖2) , for i = 1, 2, 3. MTL objective is defined as ∑3 i=1 Li(φ), which we optimize from the initialization (20, 5).
Results and analysis Fig. 3 shows the MTL loss surface and the learning trajectory of each method. We observe that except for Reptile and Sequential Reptile, all the other baselines converge to one of the MTL local minima, failing to find a reasonable solution that may generalize better across the tasks. While Reptile can avoid such a minimum, the resultant solution has very low cosine similarity (See Fig. 3h) because it does not enforce gradient alignments between tasks. On the other hand, Figure 3h shows that our Sequential Reptile tends to find a reasonable trade-off
between minimizing MTL loss and maximizing cosine similarity, thanks to the implicit enforcement of inter-task gradient alignment in Eq. 7.

heading: 4.2 MULTI-TASK LEARNING
text: We next verify our method with large-scale real datasets. We consider multi-task learning tasks such as multilingual Question Answering (QA) and Named Entity Recognition (NER). For QA, we use “Gold passage” of TYDI-QA (Clark et al., 2020) dataset where a QA model predicts a start and end position of answer from a paragraph for a given question. For NER, we use WikiAnn dataset (Pan et al., 2017) where a model classifies each token of a sentence into three classes. We consider each language as a distinct task and train MTL models.
Implementation Details For all the experiments, we use multilingual BERT (Devlin et al., 2019) base model as a backbone network. We fintune it with AdamW (Loshchilov & Hutter, 2019) optimizer, setting the inner-learning rate α to 3 · 10−5. We use batch size 12 for QA and 16 for NER, respectively. For our method, we set the outer learning rate η to 0.1 and the number inner-steps K to 1000. Following Wang et al. (2021), for all the baselines, we sample eight tasks proportional to pt ∝ (Nt)1/5 where Nt is the number of training instances for task t. For our Sequential Reptile, we set the parameter of the categorical distribution in Eq. 5 to the same pt.
Results We compare our method, Sequential Reptile against the baselines on QA and NER tasks. Table 1 shows the results of QA task. We see that our method outperforms all the baselines including STL on most of the languages. Interestingly, all the other baselines but ours underperform STL on Arabic language which contains the largest number of training instances (2 ∼ 3 times larger than the other languages. See Appendix B for more information about data statistics). It implies that the baselines suffer from negative transfer while ours is relatively less vulnerable. We can observe essentially the same tendency for NER task, which is a highly imbalanced such that the low-resource languages can have around 100 training instance, while the high-resource languages can have around 5,000 to 20,000 examples. Table 2 shows the results of NER task. We see that all the baselines but ours highly degrade the performance on high resource languages — de, en, es, hi, mr and tl, which means that they fail to address the imbalance problem properly and suffer from severe negative transfer. Even GradNorm is not effective in our experiments, which is developed to tackle the task imbalance problem by adaptively scaling task losses.
Analysis We next analyze the source of performance improvements. Our hypothesis is that our method can effectively filter out language specific knowledge when solving the downstream tasks, which prevents negative transfer and helps retain linguistic knowledge acquired from the pretraining.
Firstly, we quantitatively measure the cosine similarity between the gradients computed from different tasks in Fig. 4a (QA) and Fig. 4b (NER). We see from the figure that our Sequential Reptile shows the highest cosine similarity as expected, due to the approximate learning objective in Eq. 7. Such high cosine similarity between the task gradients implies that the model has captured the common knowledge well transferable across the languages. This is in contrast to the other baselines whose task gradients can have even negative cosine similarity with high probability. Such different gradient directions mean that the current model has memorized the knowledge not quite transferable across the languages, which can cause negative transfer from one language to others.
As a result of such negative transfer, we see from Fig. 5a and Fig. 5b that the baselines suffer from catastrophic forgetting. In those figures, high MLM losses on seen (Fig. 5a) and unseen languages (Fig. 5b) mean that the models have forgotten how to reconstruct the masked sentences, which is the original training objective of BERT. On the other hand, our method shows relatively lower MLM loss, demonstrating its effectiveness in preventing negative transfer and thereby alleviating the catastrophic forgetting. We further confirm this tendency in Fig. 5c by measuring the `2 distance from the initially pretrained BERT model. We see that the distance is much shorter for our method than the baselines, which implies that ours can better retain the linguistic knowledge than the baselines.
We can actually test if the finetuned models have acquired knowledge transferable across the languages by evaluating on some unseen languages without further finetuning. In Table 3, we test the finetuned QA models on five unseen languages from MLQA dataset (Lewis et al., 2020b). Again,
Sequential Reptile outperforms all the baselines. The results confirm that the gradient alignment between tasks is key for obtaining common knowledge transferable across the languages.

heading: 4.3 ZERO-SHOT CROSS LINGUAL TRANSFER
text: We next validate our method on zero-shot cross lingual transfer task, motivated from the previous results that our method can learn common knowledge well transferable across the languages. We train a model only with English annotated data and evaluate the model on target languages without further finetuning on the target datasets. To utilize the methods of MTL, we cluster the data into four groups with Gaussian mixture model. We focus on QA and natural language inference (NLI) task. For QA, we train the model on SQuAD (Rajpurkar et al., 2016) dataset and evaluate the model on six languages from MLQA dataset (Lewis et al., 2020b). For NLI, we use MNLI (Williams et al., 2018) dataset as a source training dataset and test the model on fourteen languages from XNLI (Conneau et al., 2018) as a target languages.
Baselines As well as the baselines from the previous experiments, we additionally include SWEP (Lee et al., 2021). It learns to perturb word embeddings and uses the perturbed input as extra training data, which is empirically shown to be robust against out-of distribution data.
Implementation Detail We finetune multilingual BERT-base model with AdamW. For QA, we use the same hyperparameter in multi-task learning experiment. For NLI, we use batch size 32 and choose the learning rate 3 · 10−5 or 5 · 10−5 with AdamW optimizer based on the performance on the validation set. For our model, we set the outer learning rate η to 0.1 and the number inner steps K to 1000. In order to construct multiple tasks from a single dataset, we cluster concatenation of questions and paragraphs from SQuAD or sentences from MNLI into four groups. Following Aharoni & Goldberg (2020), we encode the paragraphs or sentences into hidden representations with pretrained multilingual BERT. Then, we reduce the dimension of hidden representations with PCA and run Gaussian mixture model to partition them into disjoint four clusters. Since the number of training instances are almost evenly distributed for each task, we sample mini-batches from all the four tasks for all the baselines and set pt = 1/4 for our model.
Results and Analysis The results of zero-shot cross lingual transfer show essentially the same tendency as those of multi-task learning in the previous subsection. Firstly, our model largely outperforms all the baselines for QA and NLI tasks as shown in Table 4 and 5. To see where the performance improvements come from, we compute the average pairwise cosine similarity between the gradients computed from different tasks in Fig. 4c and 4d. Again, Sequential Reptile shows much higher similarity than the baselines, which implies that our method can effectively filter out
task-specific knowledge incompatible across the languages and thereby prevent negative transfer. As a result, Fig. 6a and 6b show that our method can better retain the linguistic knowledge obtained from the pretraining in terms of relatively lower MLM loss. Further, Fig. 6c and 6d confirm this tendency with shorter `2 distance from the initially pretrained BERT model.

heading: 4.4 FURTHER ANALYSIS
text: (a) Trade-off shown in Fig. 1: In Fig. 7a, MTL loss and cosine similarity decrease as we finetune the model from top-right to the bottom-left corner. Meanwhile, Sequential Reptile shows much higher cosine similarities between task gradients at the points of similar MTL losses, achieving better trade-off than Reptile. It explains why simple early stopping cannot outperform Sequential Reptile (see Fig. 7c) that directly enforces gradient alignment across tasks.
(b) Effect of the strength of gradient alignment: Then the next question is, how can we further control the trade-off to maximize performance? According to Eq. 7, we can strengthen or weaken the gradient alignment by increasing or decreasing the inner-learning rate α, respectively. Fig. 7b shows that while we can control the cosine similarity by varying α as expected, the best-performing α is around 3 ·10−5, which is indeed the most commonly used value for finetuning the BERT model. (c) Computational efficiency: Lastly, one may suspect training Sequential Reptile takes significantly longer wall clock time because inner steps are not parallelizable as in Reptile (See Fig. 2). This is not true. Fig. 7c shows that whereas base MTL requires around 40K gradient computations to achieve 64 EM score, Sequential Reptile requires only around 15K. As a result, although we run Sequential Reptile with a single GPU at a time, the wall-clock time becomes even comparable to the base MTL that we run in parallel with 8 GPUs. Please see wall clock comparison on Appendix D.

heading: 5 CONCLUSION
text: We showed that when finetuning a well-pretrained language model, it is important to align gradients between the given set of downstream tasks to prevent negative transfer and retain linguistic knowledge acquired from the pretraining. We proposed a simple yet effective method aligning gradients between tasks with efficiency. Specifically, instead of performing multiple inner-optimizations separately for each task, we performed a single inner-optimization by sequentially sampling batches from all the tasks, followed by a Reptile outer update. We extensively validated the efficacy of our method on various MTL and zero-shot cross-lingual transfer tasks, where ours largely outperformed all the baselines we considered.
Acknowledgements This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-000075, Artificial Intelligence Graduate School Program(KAIST)), the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R1A5A1059921), Samsung Research Funding Center of Samsung Electronics (No. SRFC-IT1502-51), Samsung Electronics (IO201214-08145-01), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021R1F1A1061655), and KAIST-NAVER Hypercreative AI Center.

heading: A ALGORITHM
text: We provide the pseudocode for Sequential Reptile described in section 3.1:
Algorithm 1 Sequential Reptile 1: Input: pretrained language model parameter φ, a set of task-specific data {D1, . . . ,DT }, prob-
ability vector (p1, . . . , pT ) for categorical distribution, the number of inner steps K, inner stepsize α, outer-step size η.
2: while not converged do 3: θ(0) ← φ 4: for k = 1 to k = K do 5: Sample a task tk ∼ Cat(p1, . . . , pT ) 6: Sample a mini-batch B(k)tk from the dataset Dtk 7: θ(k) ← θ(k−1) − α ∂L(θ(k−1);B(k)tk )
∂θ(k−1)
8: end for 9: MG(φ) = φ− θ(K)
10: φ← φ− η ·MG(φ) 11: end while

heading: B DATASET
text: TYDI-QA (Clark et al., 2020) It is multilingual question answering (QA) dataset covering 11 languages, where a model retrieves a passage that contains answer to the given question and find start and end position of the answer in the passage. Since we focus on extractive QA tasks, we use “Gold Passage” 1 in which ground truth paragraph containing answer to the question is provided. Since some of existing tools break due to the lack of white spaces for Thai and Japanese, the creators of the dataset does not provide the Gold Passage of those two languages. Following the conventional preprocessing for QA, we concatenate a question and paragraph and tokenize it with BertTokenizer (Devlin et al., 2019) which is implemented in transformers library (Wolf et al., 2020). We split the tokenized sentences into chunks with overlapping words. In Table 6, we provide the number of preprocessed training and validation instances for each language.
WikiAhn (Pan et al., 2017) It a multilingual NER dataset which is automatically constructed from Wikipedia. We provide the number of instances for training and validation in Table 7.
XNLI (Conneau et al., 2018) It is the dataset for multilingual NLI task, which consists of 14 languages other than English. Since it targets for zero-shot cross-lingual transfer, there is no training instances. It provides 7,500 human annotated instances for each validation and test set.
1https://github.com/google-research-datasets/tydiqa/blob/master/gold_ passage_baseline/README.md
MLQA (Lewis et al., 2020b) It is the dataset for zero-shot multilingual question answering task. As XNLI dataset, it only provides only validation and test set for 6 languages other than English. In Table 8, we provide data statistics borrowed from the original paper (Lewis et al., 2020b)
Common Crawl 100 (Conneau et al., 2020; Wenzek et al., 2020) It is multilingual corpus consisting of more than 100 language that is used for pretraining XLM-R (Conneau et al., 2020) model. We download the preprocessed corpus 2 provided by Wenzek et al. (2020). We sample 5,000 instances for each language and evaluate masked language modeling loss.
C IMPLICIT GRADIENT ALIGNMENT OF SEQUENTIAL REPTILE
In this section, we provide derivation of implicit gradient alignment of Sequential Reptile in the equation 7. Firstly, we define the following terms from Nichol et al. (2018).
θ(0) = φ (initial point) (8)
gtk = ∂L(θ(k−1);B(k)tk )
∂θ(k−1) (gradient obtained during SGD with mini-batch B(k)tk ) (9)
θ(k) = θ(k−1) − αgtk (sequence of parameter vectors) (10)
gtk = ∂L(θ(k−1);B(k)tk )
∂φ (gradient at initial point with mini-batch with B(k)tk ) (11)
Htk = ∂2L(θ(k−1);B(k)tk )
∂φ2 (Hessian at initial point) (12)
where α denotes a learning rate for inner optimization. tk ∈ {1, . . . , T} is a task index and B(k)tk is a mini-batch sampled from the task tk.
Corollary. After K steps of inner-updates, expectation of ∑K k=1 gtk over the random sampling of tasks approximates the following:
E [ K∑ k=1 gtk ] ≈ K∑ k=1 gtk − α 2 K∑ k=1 k−1∑ j=1 〈 gtk , gtj 〉 (13)
where 〈·, ·〉 denotes a dot-product in Rd.
Proof. We borrow the key idea from the theorem of Reptile (Nichol et al., 2018) that task specific inner-optimization of Reptile implicitly maximizes inner products between mini-batch gradients within a task. First, we approximate the gradient gtk as follows.
gtk = ∂L(φ;B(k)tk ) ∂φ + ∂2L(φ;B(k)tk ) ∂φ2 (θ(k) − φ) +O(‖θ(k) − φ‖22)
= gtk − αHtk k−1∑ j=1 gtj +O(α 2) = gtk − αHtk k−1∑ j=1 gtj +O(α 2)
2http://data.statmt.org/cc-100/
After K gradient steps, we approximate the expectation of the meta-gradient MG(φ) over the task t1, . . . , tK , where tk ∼ Cat(p1, . . . , pT ) for k = 1, . . . ,K.
E [MG(φ)] = E [ K∑ k=1 gtk ] ≈ E  K∑ k=1 gtk − α K∑ k=1 k−1∑ j=1 Htkgtj  = E
[ K∑ k=1 gtk ] − E α 2  K∑ k=1 k−1∑ j=1 Htkgtj + K∑ k=1 k−1∑ j=1 Htjgtk  = E
[ K∑ k=1 gtk ] − E α 2 K∑ k=1 k−1∑ j=1 ∂ 〈 gtk , gtj 〉 ∂φ  = E
 K∑ k=1 gtk − α 2 K∑ k=1 k−1∑ j=1 ∂ 〈 gtk , gtj 〉 ∂φ  = E
 K∑ k=1 gtk − α 2 ∂ ∂φ  K∑ k=1 k−1∑ j=1 〈 gtk , gtj 〉 = E  ∂ ∂φ  K∑ k=1 L(φ;B(k)tk )− α 2 K∑ k=1 k−1∑ j=1 〈 ∂L(φ;B(k)tk ) ∂φ , ∂L(φ;B(j)tj ) ∂φ
〉 = ∂
∂φ E  K∑ k=1 L(φ;B(k)tk )− α 2 K∑ k=1 k−1∑ j=1 〈 ∂L(φ;B(k)tk ) ∂φ , ∂L(φ;B(j)tj ) ∂φ 〉
Similarly, Smith et al. (2021) show implicit regularization of SGD. After one epoch of SGD, it implicitly biases a model to minimize difference between gradient of each example and full batch gradient. As a result, it approximately aligns gradient of each instance with the full-batch gradient.

heading: D FURTHER ANALYSIS
text: Computational efficiency In Figure 8a, we plot test Exact Match score as a function of wall clock time. Although training Sequential Reptile is not parallelizable, it shows tolerable computational efficiency compared to MTL model. For MTL, it takes about 1 hour and 15 minutes while Sequential Reptile takes 1 hour and 45 minutes to reach 64 EM score.
Inner steps As shown in Figure 8b, Sequential Reptile is robust to the number of steps for the inner optimization. It shows consistent performance with little variance.

heading: E ADDITIONAL EXPERIMENTS
text: In order to show that our model Sequential Reptile is generally applicable to various multi-task learning problems, we additionally perform experiments on monolingual text classification and image classification.
Text classification Following (Pilault et al., 2021), we train BERT base model on 7 text classification tasks — CoLA, MNLI, MRPC, QNLI,QQP, RTE, SST2 and one text similarity score regression — STSB from GLUE dataset (Wang et al., 2019a). We share the BERT encoder across all the task and add linear layer on top of the encoder for each task. For Reptile, we use learned initialization with each task specific head for prediction at test time. For STSB task, we use Pearson correlation coefficient to measure the performance of the baselines and ours. For the other 7 task, we evaluate all the models with accuracy. As shown in Table 9, Sequential Reptile outperforms the Reptile and MTL on 4 tasks – CoLA, MRPC, RTE, SST2 and shows comparable performance on the other tasks.
Image classification Following the experimental setup from Shin et al. (2021), we use 7 datasets — Tiny-ImageNet (Le & Yang, 2015), CIFAR100 (Krizhevsky et al., 2009), Stanford Dogs (Khosla et al., 2011), Aircraft (Maji et al., 2013), CUB (Wah et al., 2011), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011). We class-wisely divide Tiny-ImageNet into two splits which are denoted as TIN-1 and TIN-2, respectively, and consider each split as a distinct task, which results in total 8 tasks for multi-task learning image classification. We finetune ResNet18 (He et al., 2016) which is pretrained on ImageNet (Deng et al., 2009) with a randomly initialized linear classifier for each task. For all the models, the pretrained ResNet is shared across tasks. As the previous experiments, we use shared initialization of Reptile with task specific linear classifiers. We measure accuracy of each image classification task and report the average score.
As shown in Table 10, Sequential Reptile outperforms MTL and Reptile with large margin other than Fashion MNIST (F-MNIST). We observe that Reptile fails on image classification, which is contrast to text classification tasks where it shows reasonable accuracy compared to MTL model. We conjecture that Reptile needs adaptation to the target task since the set of image datasets are more heterogeneous than the set of text datasets.
F VISUALIZATION OF LEARNING TRAJECTORY
In Figure 9, 10, and 11, we visualize learning trajectory of the MTL models and cosine similarity between task gradients with three different initialization as described in section 4.1. The similar pattern holds for all different initialization. All the MTL baselines except Reptile fall into one of local optima. On the other hand, Sequential Reptile avoid such local minima while maximizing cosine similarities of task gradients.

Review summaries:
The paper proposed an optimization method for training multi-task learning models by increasing the task alignments. Their key assumption is that standard MTL can result in catastrophic forgetting of pretrained knowledge and lead to local task optimum in the finetuning stages. To avoid this, the paper proposed a method to jointly optimize the task losses and gradient alignments between tasks. Their key contributions are: - The paper empirically examined the question of whether worse alignments of task gradients can lead to negative transfer, though extensive experiments on multilingual tasks. - Based on such observation, the paper proposes a simple and effective optimization method for MTL that jointly optimizes the task losses and gradient alignments between tasks.

The paper presents a new method for multi-task learning (MTL), namely sequential reptile (SR). The proposed method aims to align task gradients and mitigate negative interference among tasks during the finetuning process of multilingual BERT. Specifically, different from a regular Reptile which performs the inner loop on a single (sampled) task, the modified version does so on a batch of tasks, sampled according to some prior distributions. The paper argues that this simple modification can promote gradient alignment naturally. Empirically, the proposed SR method outperforms all baselines considered significantly, while using similar or even less computational resources.

This paper proposed a new training method to improve gradients alignment between languages (tasks) in multilingual finetuning of pretrained language models. The method is based on an existing algorithm, Reptile, which was originally developed for meta learning. The authors adapted it for multi-task learning and verified its effectiveness in multilingual finetuning.