Less-forgetting Multi-lingual Fine-tuning
Authors: Yuren Mao
Yaobo Liang
Nan Duan
Haobo Wang
Kai Wang
Lu Chen
Yunjun Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-lingual Language  Models
Multi-lingual Fine-tuning
Less-forgetting

Abstract: Multi-lingual fine-tuning (MLF), which fine-tunes a multi-lingual language model (MLLM) with multiple source languages, aims to gain good zero-shot performance on target languages. In MLF, the fine-tuned model tends to fit the source languages while forgetting its cross-lingual knowledge obtained from the pre-training stage. This forgetting phenomenon degenerates the zero-shot performance of MLF, which remains under-explored. To fill this gap, this paper proposes a multi-lingual fine-tuning method, dubbed Less-forgetting Multi-lingual Fine-tuning (LF-MLF). In LF-MLF, we cast multi-lingual fine-tuning as a constrained optimization problem, where the optimization objective is to minimize forgetting, and constraints are reducing the fine-tuning loss. The proposed method has superior zero-shot performance; furthermore, it can achieve the Pareto stationarity. Extensive experiments on Named Entity Recognition, Question Answering and Natural Language Inference back up our theoretical analysis and validate the superiority of our proposals.

Less-Forgetinng Multi-Lingual Fine- Tuning
Yuren Mao!
Yaobo Liang
Nan Duan
Haobo Wang'
Kai Wang"
Lu Chen'
Yunjun Gao
Zhejiang University
Microsoff Research Asia
Shanghai Jiao Tong University.
Cyuren. mao, wanghaobo, luchen, (rnnchnngaay]]ooeo. gaoyj] edu. en {yalia, nanduan)@miico crosoft.. com, waaiajjtu. edu.
Abstract
Multi-lingual fine-tuning (MLF), which fine-tunes multi-lingual language model (MLLM) with multiple source languages, aims to gain good zero-shot performance target languages. In MLF, the fine-tuned model tends to fit the source languages while forgetting its cross-lingual knowledge obtained from the pre-training stage. This forgetting phenomenon degenerates the zero-shot performance of MLF, which remains explorex To fill this gap, this paper proposes a multi- Manlti-linnuall fine- tuning method, dubbed Less fers-forge Multi-lingual Fine-tuning (LF-MLF). In LF-MLF, we cast multi-lingual fine-tuning as a constrained optimization problem, where the optimization objective is minimize forgetting, and constraints reducing the fine-tuning loss. The proposed method has superior zero-shot perfor- mance; furthermore, it can achieve the Pareto stationarity. Extensive experiments Named Entity Recognition, Question Answering and Natural Language Inference back up our theoretical analysis and validate the superiority of our proposals.
Introduction
Multi-lingual Language Models (MLLMs), which are pre-trained on multiple languages, have the cross-lingual generalization ability. By fine-tuning MLLMs on source languages, we can expect promising zero-shot performance on target languages M 2 In literature, most of the works focus on mono-lingual fine-tuning, which fine-tunes a multilingual model on only one source language (typically English). However, in practice, available, e.g., in name entity recognition, it is easy to obtain labeled data for English, German, French and so on. this background, multi-lingual fine-tuning (MLF), which fine-tunes an MLLM on multiple source languages, becomes an emerging research topic.
common that there are multiple source languages
In the fine-tuning procedure of MIF, each source language has language-specific gradient descent direction. Combining these gradient descent directions together, we can find set of gradient descent directions common for all the source languages. However, in this set, some directions lead MLLM to fit the source languages too much and forget its cross-lingual knowledge obtained from the pre- training stage. This forgetting phenomenon can degenerate the zero-shot performance of MLF. Thus, how to find common gradient descent direction that benefits both the fine- line-tuning performance and zero-shot performance a important issue MLF, which remains under-explored
To address this issue, this paper proposes to find a less-forgetting descent direction, which prevents a MILM from forgetting the cross-lingual generalization ability and is common descent direction for the source languages. To find this less-forgetting descent direction, we cast MLF as a constrained
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

optimization problem. The optimization objective is to minimize the forgetting of a MLLM's cross- lingual generalization ability, and the constraint is that the direction should be descent direction common to all the source languages. Correspondingly, we propose a novel algorithm, namely Less- forgetting Multi-lingual Fine-tuning (I.F-MLF), to solve this constrained optimization problem and obtain the less-forgetting descent direction.
For LF-MLF, we have conducted both theoretical and experimental analysis. Theoretical results demonstrate that LF-MLF can effectively reduce the forgetting of multi-lingual models in fine- tuning; besides, LF-MLF can achieve the Pareto stationarity. The superiority of LF-MLF has been experimentally verified on three kind of tasks (i.e., Named Entity Recognition, Question Answering and Natural Language Inference). Furthermore, we also conduct experimental analysis on several factors that impact the performance of multi-lingual fine-tuning (e.g.. the impact of the number of the fine-tuning languages), which have the potential to forge new trends in multi-lingual fine-tuning.
2 Related Works
Multi-lingual Language Models (MLLMs) are pre-trained using tons of unlabeled data from multiple languages and expected to own the cross- tross-lingual zero-shot transfer ability. This ability can facilitate the low resource languages processing with knowledge transferred from the high resource languages (2). Several MLLMs have been proposed, such as mBERT [3] and XLM-R []] Given a MLLLM, most of the existing works analyze its cross-lingual zero-shot transfer ability by observing the model's s zero-shot performance on the target languages after mono-lingual fine-tuning [5] 7]. Few works [ [9] have investigated the multi-lingual fin-tuning setting, but they do not consider the forgetting of MLLMs' cross-lingual generalization ability. To fill this gap, this paper focuses on the forgetting problem in multi-lingual fine-tuning.
Continual Learning (CL) studies the problem of learning a model on sequential tasks without forgetting knowledge obtained from the preceding tasks [TO. In CL, reducing forgetting the major challenge. Lots of methods have been proposed to reduce forgetting, such as GEM [II), EWC [12] and IMM 13: : however, these methods typically focus on the single-objective optimization setting, where there is only one learning objective. None of them considers the multi-objective optimization setting, where multiple learning objectives involve. Nevertheless, multi-linge fine-tuning involves a multi-objective optimization problem, which aims to optimize the fine-tuning performance on all the including languages. Thus, existing CL methods are not suitable for multi-linguu fine-tuning.
Multi-task Learning (MTL), which simultaneously learns multiple tasks, aims to achieve proper performance on all included tasks. Recently, various MTL methods 14 15 16
18
19
20 have
been proposed. Among them, PCGrad [19] and Gradient Vaccine 20 have achieved the state-of- the-art performance. Regarding the source languages of MLF as MTL tasks, MTL methods can be used in MLF. However, MTL methods just focus on the performance of including source languages (tasks), while they do not consider the performance of target languages. It will bring inferior zero-shot performance on the target languages. For example, PCGrad and Gradient Vaccine just focus on finding a proper common descent direction for the source languages, but the target languages are not considered in these methods. Thus, they cannot expect to achieve promising zero-shot performance. By contrast, the proposed LF-MLF finds the common descent direction that benefit the zero-shot performance. Overall, LF-MLF have more advantages over the MTL methods in MLF.
3 Less-forgetting Multi-lingual Fine-tuning
The goal of munti-linngul fine-tuning is twofold: firstly, achieving proper fine-tuning performance on the source languages, and secondly, achieving proper zero-shot performance on the target languages. For the first goal, it is necessary to avoid the model from forgetting the cross-lingual knowledge obtained in the pre-training phase. For the second goal, we face a multi-objective optimization problem, where the losses for the source languages should be jointly optimized. In this section, we firstly give a formal definition of the multi- multi-lingual fine-tuning and then propose method (dubbed LF-MLF) that can simultaneously achieve the above two goals.
2

---

3.1 Problem Definition of Multi-lingual Fine-tuning
Given a multi colti-lingual language model pp. which is pre-trained using large amounts of unlabeled data from language set Sp. we aim to fine-tune this model on a downstream task and achieve proper performance. For the downstream task, we have task-specific labeled training data from set of source languages Sy, where S C Sp. After fine-tuning, 0p has been updated as 0,. Assume there are languages source languages set Sy. For each language, the task-specific loss denoted as (fy). Under the gradient descent paradigm of deep learning, in the hth iteration of multi multi-lingual fine-tuning, the model is updated by
T 02 = 0k-1 L 1m1
V (0x-)))). Wi
(1)
where 7f is the learning rate, Σl= u
and wt 2 0,t {T. In Eq. 1
- -E /t (fx(()) is the convex hull of the gradient descent directions of the included lan- guages, which contains feasible gradient descent directions for the multi-lingual fine-tuning.
Furthermore, denote the pre-training loss of 8p and f as (0pp and Lp(0r) respectively. According to the classic definition of forgetting [21] we define the forgetting between 8p and fy Fp = [p,( - (8p)). which quantifies the performance degeneration of f on the multi-lingun pre- training task (e.g., masking tokens). With the above notation, the two goals for multi-lingual fine-tuning can be formally defined as: (1) minimizing Fp. and (2) minimizing (L, (0))).
3.2 Upper Bound the Forgetting in Multi-lingual Fine-tuning
In fine-tuning, we cannot directly compute the forgetting Fy. Alternatively, we propose to minimize the forgetting by minimizing its upper bound. In Theorem give a upper bound for the forgetting of multi-lingual fine-tuning. The assumptions in Theorem are widely accepted 21 22 X ensure proper performance.
and can
Theorem 1. Assume L,(0g) can be approximated by its second order Taylor expansion and 6, is minima w.r.t the pre-training loss. Then, we have
P3
Apl
wi
LL
h
k=1
where Xp is the maximum eigenvalue of 2 (Op), and K is the number of iterations of fine-tuning.
3.3 forgettgg Multi-lingual Fine-tuning
In multi-lingual fine-tuning, we not only need to keep the multi-lingual memory the pre- trained model but also need to achieve proper fine-tuning performance, i.e., minimizing F3 and (0,) Lr (r(0 (0,) together. Achieving these two goals at same time is challenging and remains unexplored. In this section, propose novel method, dubbed Less-forgettnng Multi-lingual Fine-tuning (LF-MLF), to jointly achieve these two goals.
Specifically, we minimizing F, by tightening its upper bound introduced in Theorem 1 Besides, minimizing { [L(( (0)).... ..,, Lr-(6) (0r) is a multi-objecctvve optimization (MOO) problem. We solve this MOO problem by the means of multi-objective gradient descent.
According to Theorem
in the h iteration,
<
ApT
W (V
Ap VLL
Σ
w
(3)
2
where VL= -ーー W/ (. and 0o = 0
To reduce fogetting, in the tthh iteration, propose minimize Fk's upper bound as in Eq.
4
min wutt
VL +
w Le (0c
3

---

0p
Uniform-MLF
LF-MLF
01
(L((0))
0
-V4,(0)
Piz(01)
Except for reducing forgetting, we also need decrease the task-specific losses of the fine-tuned languages, i.,, move toward the objective of minimizing {L1(#)), (0,)) Thus, in the 4t iteration, we need to update the model in a descent direction common all the source languages. Figure 1: Toy example: comparison the model update procedures of LF-MLF and Uniform-MLF:
which is the principle of multi-objective gradient descent. Under this principle, the update direction should not have obtuse angle with the gradient direction on each language as Eq. 5
w Lo. (I..(00-))) 0, for LE {I,...
(5)
To reduce fogetting and minimizing the downstream task losses, we combine Eq. with Eq. and propose formulate the less-forgetting multi-lingual fine-tuning as the following constrained optimization problem.
Problem 1.
min w
VL Σt / VI,
s.t.
w V Lr (0x 1))
(0n- 0, (1..,,
(6)
w =1 w E,
1=
To illustrate the rationale of this constrained optimization problem, we give toy example in Figure 1 In this example, we compare LF-MLF with Uniform-MLF method which adopts the uniform weights across the training procedure (i.e., w = 1for {1...., T} and k E (1, K}). LF-MLF chooses the common descent direction that minimizes the distance between the original MLLM 0p and the updated MLLM 0₂ at each update step. Obviously, LF-MLF has shorter distance than the Uniform-MLF method, which illustrates that LF-MLF can slow the forgetting of original MLLM down. Furthermore, to solve Problem T we formulate it as the following quadratic programming problem with variables and 27 + 1 constraints.
Problem 2.
min w s.t.
Gw. + (ww (w
bj w 2 20 f ΣT wh =1 w 0, E {l....,
[V where G VL00--1VL(0--), () L2(xx1)) LT(Ox-1)] VL(B--1) and =
Problem can be effectively solved. This paper adopts interior-point solver 244. Let wi be the solution of the above quadratic programming problem. In our proposed LF-MLE, the update rule is
02 = 0k-1 (L)))))
(8)
Overall, the challenging multi-lingual fine-tuning problem is transferred to a simple quadratic programming problem. The detailed steps of LF-MLF can be referred Algorithm

---

Algorithm I: Less-forgetting Multi-lingual Fine-tuning (LF-MLF)
Input: Pre-trained model Op. Number of iterations K. for <- to K do
i == then
Solve Problem. with c = [0, o........ 0J and obtain solution wf.
01 0p
Σ 1=1 w V L, (8p).
else
Solve Problem. and obtain solution wz. 0g 02- 2 LV,
end if
if the feasible set of Problem. is empty then
break.
end if end for return 0K-
4 Theoretical Analysis
In less-forgetting multi--inguut fine-tuning, explicitly set the goal, minimizing the upper bound, as the optimization objective. Therefore, the less-forgetting property of the proposed LF-MLF method is straightforward. By contrast, the target of minimizing (0,), LT(()) is achieved by the means of putting constraints on the update direction, which forces the update direction to be the common decreasing direction for all the fine-tuned languages. Its effectiveness seems not obvious. To make it clear, in this section, we conduct theoretical analysis on LF-MLF's fine-tuning performance.
In multi-lingual fine-tuning, we aim to jointly fine-tune several languages, namely minimizing the loss vector L (0y), (0,),, is a multi-objective optimization problem, where the optimal solutions are Pareto optimal in Definition
the Pareto optimality w.r. the optimization objective ming L(0) is defined as follows: (1) A solution 0 dominates solution 0 if (0) S (,) for all t E T and L(0) # L(0) and (2) A solution 0* is deemed Pareto optimal if there exists no solution that dominates 0"
Definition (Pareto Optimality). For L(0) (()))
However, in deep learning, the parameter spaces highly non-convex. To the best of knowledge, there is no method that can learn a Pareto optimal deep neural network model. Settling for less, the best we can do is achieve the Pareto stationarity in Definition Pareto stationarity necessary condition of achieving Pareto optimality.
Definition (Pareto Stationarity). For L(8*) = stationary the design-point ( iff there exists convex combination of the gradient-vectors, ET We )O")), that is equal zero:
(L HTT is said to be Pareto-
T
T
(L) = 0;
Σ w =
w k 2 0, LE (1......
In Theorem 2 and we propose that LF-MLF can achieve Pareto stationarity, which theoretically verifies that our proposed method achieve proper performance on the fine-tuning languages.
Theorem 2. Let H. be Hilbert space finite dimension N. Let L.(Os) (1 < T S N) be T smooth functions of the vector 0g E H. and 6e a particular admissible design-point. Let wi be the solution of Problem and descent direction VL ET= (w,)'VV. Then, either VL Z, and [L, [L.()), .... Lr (T)) pareto stationary at f. VL Ø and -V is descent direction common all {L, (0x))
Theorem 3. LF-MLF can stop after finite number of iterations if Pareto stationary point is reached. Otherwise, If the sequence of iterates {0x km of the L.FMLF is infinite, admits a weakly convergent subsequence.
5

---

Experiments
In this section, we perform experimental studies on three downstream tasks- Named Entity Recog- nition (NER), Question Answering (QA) and Natural Language Inference (NLI) respectively to evaluate the performance of our proposed LF-MLF and verify our theoretical analysis.
5.1 Experimental Setup
Datasets: In our experiments, we adopt the NER (25). TyDiQA-GoldP [26] and XNLI [27 datasets for NER, QA and NLI respectively from the XTREME benchmark [8. The details of these datasets are introuced in the supplementary material.
Baselines: We compare proposed LF-MLF with the following baselines: (1) Single-lingual Fine- tuning (SLF); fine-tuning on each language independently; (2) Uniform-MLFF fine-tuning on the included languages simultaneously by means of uniformly weighted sum of the language-specific gradient descent directions, i.e., wf Gradients (PCGrad) method proposed by [9 multi-lingual fine-tuning: (4) Grad using
constantly; (3) PCGrad-MLF: using the Project Conflicting
the Gradient Vaccine (Grad Vac) method proposed by 20 on multi-lingual fine-tuning.
Experimental Settings: In our experiments, we generally fine-tune XLM-RoBERTT models (base- sized model) [28] with a training batch size of 32, and Adam [29] used as the optimizer. As to epochs.
adopt the default setting of the XTREME benchmark, that are 10, 3, 5 for NER, TyDiQA and XNLI respectively. Besides, the learning rates are selected from {1e 5, 2e 5, 5e 5). {le 5, 2e 5, 5e 5) and {5e 6, le 5, 2e 5) with grid search for NER, TyDiQA and XNLI respectively. All the results are averaged over runs.
the numbers of
5.2 Performance of Multi-lingual Fine-tuning
Conducting experiments NER, TyDiQA and XNLI, we compare LF-MLF with the baselines the zero-shot performance and fine-tuning performance respectively. For each dataset, we choose two three source languages for fine-tuning according the following rules: (1) belongs to the Top 10 righ-resource languages w.r. Wikipedias; (2) the higher-resource the better. For NER and XNLI, en, de, fr are chosen, while en, chosen for TyDiQA
5.2.1 Zero-shot Performance
To evaluate the zero-shot performance of LF-MLF and the baselines, we record the F1 score perfor- mance (mean and standard deviation) over the target languages that have not been fine-tuned (i.e.. the other 45 languages except en, de. fr in NER, the other 7 languages except en, ru in TyDiQA and the other 10 languages except en, de, fr in XNLI). Furthermore, pairwise t-tests at 0.05 significance level are conducted based on the three runs. The results are reported in Table and for NER, TyDiQA and XNLI respectively.
Besides, to comprehensively evaluate the supe- riority of LF-MLF, further utilized Fried- man test as the statistical test to analyze the rela- tive performance among the compared methods across the three applications. At 0.05 signifi- cance level, the Friedman statistics 5.5, and critical is 4.76. Thus, at 0.05 significance level, the null hypothesis of indistinguishable performance of LF-MTF among all compared methods is clearly rejected. Subsequently, we employ the Bonferroni-Dume test as the post-hoc test by regarding LF-MLF as the control approach, [igurerrepone the CD diagrams at 0.1 significance level, where the average ranks of the compared approaches is marked along the axis. From this figure, we can see that LF-MTF achieves highly superior results to other baseline methods.
GradVac-MLF PCGard MLF
Uniform MLF
Figure 2: CD diagram of Bonferroni-Deee test.
5.2.2 Fine-tuning Performance
Except the zero-shot performance, we also care about the fine-tuning performance, namely the F1- score on the languages that have been fine-tuned. The results of fine-tuning performance are reported in Table 4 and for NER, TyDiQA and XNLI respectively. From these results, we can see that:

---

Method
en de Uniform-MLF PCGrad MLF GradVac-MLF
en-SLF
de-SLF
fi-SLF
LF-MLF
Zero-Shot 60.05+0.188 18 18.53++0.19 63.97=0.58 67.45+0.33e
67.40+0.32 67.57+40.25 68.50++..11
Table 1: Zero-shot performance (FI score (mean=std)) on NER. addition, indicates that LF-MLF is statistically superior to the comparing method (pairwise t-test at 0.05 significance level).
Method
en-SLF
ru-SLF
Uniform-MLF
PCGrad-MILF GradVae-MLF
LF-MLF
Zero-Shot 55.0510.677
60.66.10.34
62.51
61.7410.51
61.3610.36 63.03.10.31
Table 2: Zero-shot performance (FI score (tt))) on TyDiQA. addition, indicates that
LF-MLF is statistically superior to the comparing method (pairwise t-test at 0.05 significance level).
Method
en-SLF
de-SLF
fr-SLF
Uniform-MI.F PCGrad-MI.F GradVac-MLFF LF-MIF
Zero-Shot 72.46:20.39 74.26:10.38 74.01=0.114 75.17=0.12+ 74.87+0.21 74.67:40.14 75.56:10115
Table 3: Zero-shot performance (FI score (mean=sstd) XNLL In addition, indicates that LF-
MLF is statistically superior to the comparing method (pairwise t-test at 0.05 significance level).
Method
en-SLF
de-SIF
fr-SLF
Uniform-ML.F PCGrad-MLF GradVac-MLF
LF-MLF
82.59-0.114
8.15
81.76-10.08
82.0710.188
82.75:10.10
88.78
87.90=10.03
87.13=00055
87.30+0.13
88.92:10.09
fr Table 4: Fine-tuning performance (FI score (mean=ssd)) NER.
89.75:10009
89.6811.117
88.85=0.19
89.7010.33
89.15:0.26
Method
en-SLF
ru-SLF
Uniform-MLF PCGrad-MLF
GradVac-MLF
LF-MLF
66.03-0031
68.71+0.55 72.43-10.08
69.28+0.66
67.26:10.38 71.87:10.15
69.31+0.41
70.89
71.76:10.19
71.66:10.21
Table 5: Fine-tuning performance (Fl score (mean on TyDiQA-GoldP.
Method
en-SLF
de-SLF
fr-SLF
Uniform-MLF PCGrad-MLF GradVac-MLF
LF-MLF
en
84.29-10.35
8..9311003
84.75-0.21
84.5940.11
84.8111.114
de
78 78.9813.23
79.92=10007
79.48=0.18
79.96#0.12 80.43+00.6
80.01
fr
79.80++0.36
80.634..111
79.98-0.12
80.83--0.13
Table 6: Fine-tuning performance (F1 score (mean=istd)) XNLI.
(1) the multi-lingual methods do not clearly outperform single-lingual methods on corresponding languages; (2) LF-LLF's performance is comparable to the baselines, and there is no baseline method dominate LF-MLF. PCGrad-MLF and GradVac-MLFF have inferior performance comparing with Uniform-MLF and LF-MLF. It because that PCGrad-MIL G.F and Grad VacdVaccMLL MLF cannot guarantee to find Pareto stationary point, while Uniform-MLF and LF-MLF can find Pareto stationary point. verifies the theoretical analysis proposed in Section 4
that LF-MLF has superior zero-shot performance and comparable fine-tuning performance. verifies that LF-MLF has superior lingull transfer ability. The superiority comes from LF-MLF's less-forge property, because such property enables LF-MLF to remember more about the target languages.
Overall, from section 5 5.2.1 and 5.2.2
5.3 Impact of the Number of the Source Languages
In the last section, it is concluded that multi-lingual fine-tuning can significantly improve the zero-shot performance. Then, it natural ask whether the zero-shot performance can be further improved when more languages have been nuned? In this section, experimentally present that the answer
7

---

10 lang
UniformMUL:
LF-MLF:
66, Tr.
73
74 44
F-MEF::
73.59
lang
tang
18.
1
UnI
e0. de. Tr 65. (6.1
6 lang
Unfoom-MLF
eh, d6, Tr. nu, 6s.ja
70.81
5 lang
Uniform MLF:
LE MLF:
ch. dc. Tr. nu,
de, fr.
67.31
68.87
LF-UU:
en. de, fr.
60.96
lang
UnI
de, fr,
68. 19
lang
LF-MLF:
dr.
do,
69.54
Uniform-MLF
2 lang
LF MLF
Average score over 38 lanuages
Figure 3: Zero-shot performance w.r.f different number of fine-tuning languages.
1.05
Re ative Average F1
1.0
Uniform MLF
LF MLF
0.95
2
10
Number of Languages
Figure 4: Relative fine-tuning performance w.r.t different number of fine-tuning languages.
is yes. Besides, we also investigate the impact of the number of source languages on the fine-tuning performance. Our experiments are conducted on the NER dataset, and the source languages are selected from the Top 10 high-resource languages w.r.t Wikipedias, i.e., de, fr; ru, es, ja, it, zh, ko, ar. By contrast, the other thirty eight languages are used as target languages. the experiments, increase the source languages from Top 2 to Top 10 and record the changing of zero-shot and fine-tuning performance.
5.3.1 Impact on the Zero-shot Performance
Figure illustrates the change of MLF's zero-shot performance along with the increasing number of source languages. From this figure, we can conclude that (1) generally speaking, the zero-shot performance of a multi-lingual model continuously improves with the increasing of the number of source languages; (2) LF-MLF consistently outperforms Uniform-Ne when the number of source languages changes.
5.3.2 Impact the Fine-tuning Performance
To evaluate the fine-tuning performance over different number of source languages, we present metric, dubbed average relative F1 score, which measures the average relative improvement of multi- lingual methods's 's fine-tuning performance comparing with the single-lingual fine-tuning. Specifically, let St be the Fl score of the lingle-ligual fine-tuning language and let STz be the Fl score of the multi-lingual fine-tuning on language Assume there are T fine-tuning languages, the average relative Fl score is defined as Rr = + ET 8t
ST.
Figure 4 demonstrates the change of fine-tuning performance along with the increasing number of source languages. From this figure, we can see that (1) the fine-tuning performance just slightly fluctuates when the number of source languages changes; (2) LF-MLF and Uniform-MLFF do not evidently outperform single-lingual fine-tuning, and they have similar average relative Fl score over different number of source languages.

---

Gemmanic
Indo-tranian
Balte-Slavic
Italic
4 Comoinations of
Commaneiions of
Germanic
87.5
77.8
69.4
77.1
Comminattoos of
Avera P1 Score
85
Italic
83.5
91.5
74.7
84.2
80
01000 80.0L
03.00
03.00
20:42
Indo dda-iranian 70.4
68.8
92.5
71.4
PU
75.00
70
Balto-Slavic 79.2
79.5
70.2
91.5
70
Am
(a) Cross Language Branches
(b) Different Combinations of Language Families
Figure 5: Impact of the diversity on the fine-tuning languages' branches for LF-MLE.
Germanic
Indo-tranian
Balte-Slavic
Italic
4 Comainenions of
4 Comdinations of
Germanic
88.5
77.7
67.1
77.1
Combinwiiom of
Awerase 11 scote
adela
21.
92.01
Itailc
82.5
91.5
71.7
84.1
66.9
In Indo-Iranian 66.5
66.7
93.4
69.5
75
h
(1)
70
Balto-Slavic 79.1
78.8
70.6
91.5
70
(a) Cross Language Branches
(b) Different Combinations of Language Families
Figure 6: Impact of the diversity on the fine-tuning languages' branches for Uniform-ML.R
Language Family
Germanic
Italic
Indo Iranian
bn, hi Balto-Slavis
Fine-tuning Candidates
en, de
fr, it
bg, pl, uk
Test Set for Zero-shot
nl
pt, es
mr,
Table 7: Data allocation for the language branches.
5.4 Impact of the Diversity of Language Branches Covered by the Source Languages
After investigating the impact of the number source languages, we further explore whether the diversity of language branches covered by the source languages impacts the zero-shot performance. In this section, we conduct experiments on the different branches of European language family based on the NER dataset. Specifically, choose sixteen languages, which belong four different language branches (Germanic, Italic, Indo-Iranian, Balto-Slavic) and each branch has four languages, from the NER dataset.
Firstly, we investigate the cross-language- maaissaaagageeraeehhes branches transfer performance, namely fine-tuning on the four languages of a branch and observe the zero-shot performance another branch, such as fine-tuning on Germanic
test zero-shot performance on Italic. The experimental results are illustrated in Fig- (a) and Fig. (a) for L.-MLF and Uniform-ML.F respectively. In these figures, the vertical axis represents the source. These results present that: (1) the cross- lesssllnggage-brang branchhs transfer performance of each pair of branches is asymmetric; (2) LF-MLF has better maass-languge--raneeee transfer ability than Uniform-MLE which experimentally explains the superiority of LF-MLF and further verifies LF-MLF's s less forgetting property.
Next, we explore the impact of language branches' are used as candidates for fine-tuning (i.e., source languages), while the other eight languages are used to test the zero-shot performance. The detailed allocation of these languages are reported Table Fixing the number of fine-tuning languages as four, we test the language branches' combinations that the source languages cover two branches, three branches and four branches respectively, see that:
diversity of the source languages. Eight languages
and the results are illustrated in Fig. (b) and Fig. 6 (b). From these figures,

---

(1) the language branches' diversity of the fine-tuning languages does not have clear impact on the zero-shot performance. which means that covering more language branches might not lead to better zero-shot performance; (2) LF-MLF generally outperforms Uniform-ML.F on most of the combinations.
Conclusion
This paper researches novel fine-tuning setting for multi-lingual language models, namely multi-lingual fine-tuning; furthermore, we propose novel multi-lingual fine-tuning method, dubbed Less-forgetting Multi-lingual Fine-tuning (LF-MLF). Comparing with its mono-lingual fine-tuning counterparts, multi-lingual fine-tuning can achieve superior zero-shot performance; moreover, our proposed LF-MLF outperforms the baseline multi-lingual fine-tuning methods. Researching on mono-line lingual fine fin-tuning provides several inspirations and has the potential to forge new trends in multi-linguu learning research. This paper only focuses on the multi-lingus fine-tume tine-tuiig scenario. In the future, we will extend our method to more general scenarios, such as dess-forgetting multi-task learning.
Acknowledgements
This work was supported in part by the National Key Research and Development Program of China under Grant No. 2021 DFC330033,,,, the NSFC under Grants No. (62025206, 61972338, and 62102351), and Ningbo Science and Technology Special Projects of China under Grant No. 2021Z019.
References
[1] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Translation artifacts in cross-lingual transfer learning. In EMNLP, 2020.
[2] Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. A primer on pretrained multilingual language models. CoRR, 2021.
[3] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[4] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek. Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- rros--inguall representation learning at scale. In ACL, 2020.
[5] Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. Exploring fine-tuning techniques for pre-trained cross-lingual models via continual learning. CoRR, 2020.
[6] Yuxuan Wang, Wanxiang Che, Jiang Guo. Yijia Liu, and Ting Liu. Cross-lingual BERT transformation for zero-shot dependency parsing. In EMNLP-IJCNLP, 2019.
[7] Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross-lingual structure in pretrained language models. In ACL, 2020.
[8] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin John- son. XTREME: A massively multilingual multi-task benchmark for evaluating ling-lln generalisation. In ICML, 2020.
[9] David Ifeoluwa Adelani, Jesujoba O. Alabi, Angela Fan, and et.ll A few thousand translations go a long way! leveraging pre-trained models for african news translation. NAACL, 2022.
[10] Zhiyuan Chen and Bing Liu. Lifelong Machine Learning, Second Edition. Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers, 2018.
[11] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In NewrIPS, 2017.
10

---

[12] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwingg et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11 114(13):351-3352, 2017.
[13] Sang- Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS, 2017.
[14] Ozan Sener and Vladlen Koltun. Multi-task learning multi-objectivv optimization. In NeurlPS, 2018.
[15] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In ICML, 2020.
[16] Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learning. CoRR, 2020.
[17] Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task learning. In ICML, 2020.
[18] Yuren Mao, Shuang Yun, Weiwei Liu, and Bo Du. Tchebycheff procedure for multi-task text classification. In ACL, 2020.
[19] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurlPS, 2020.
[20] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: improving multi-task optimization in massively multilingual models. ICI.R, 2021.
Investigating and
[21] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Un- derstanding the role of training regimes in continual learning. In NeurIPS, 2020.
[22] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In AISTATS, 2020.
[23] James Kirkpatrick, Razvan Pascanu, and et.al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 1 (14(13):3521-4 -3526, 2017.
[24] M Schmidt, D Kim, and $ Sra. of book: Optimization for machine learning. MIT Press, 2011.
[25] Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. ACL, 2017.
[26] Jonathan H. Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. Tydi QA: A benchmark for information-sseking question answering in typologically diverse languages. Trans. Assoc. Compul. Linguistics, 8:454-470, 2020.
[27] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. In EMNLP, 2018.
[28] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. CoRR, 2019.
[29] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, 2017.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "vaJc0v6Nga_", "reviewer": "Reviewer_2LmV", "summary": "This paper presents a way of learning from multilingual annotated data. They focus on the zero-shot transfer to low-resource languages for tasks like NER, QA and NLI.  The authors claim that for this setting they need to optimize the model for two new objectives: one that minimizes the forgetting of the low-resource languages from pretraining, and one that ensures that when fine-tuning on the different multi-lingual annotated datasets that the descent direction between them is shared or is in common. They formulate this as a constrained optimization problem and call their method Less-forgetting Multi-lingual Fine-tuning (LF-MLF). They show consistent but not very large improved performance on NER, QA and NLI and look at where labelled data from more languages helps - and unsurprisingly it does. \n", "questions": "For Figure 3 and 4 please make sure that you specify if the x or y axis is the source or target.\n", "limitations": "This work does not discuss its limitations which is a limitation. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\nThey show consistent improvements averaged over up to 45 zero-shot language directions over three different tasks which is quite extensive. \n\nWeaknesses\n\nIt is not clear from the paper how their approach is different from the baseline methods. The authors should explain the most relevant baselines and describe how they are different: Project Conflicting Gradients method and Gradient Vaccine.  This is a major flaw in the paper. \n\nThey have limited the usefulness of the methods to a very narrow, specific use-case: multilingual and zero-shot. The less-forgetting direction could in theory be useful for monolingual fine-tuning as well as multilingual fine-tuning. The combined descent could be useful for transfer learning with labelled data too - not just zero-shot. Why did the authors not mention this or do any experiments along these lines? \n\nIt is probably incorrect to state line 75: \"MTL methods are not suitable for multi-lingual fine-tuning\" as the authors use such methods as baselines in the paper, even though they do not describe them and how they are different to the LF-MLF. \n \nThe authors do not discuss the size of labelled data for each of the multilingual cases or discuss how this affects the results. Eg. in Table 2 en, de, fr, ru etc. as performance on fine-tuning just on Russian gives better performance that just training on English which does not make sense. \n\nThe amount of improvement is not great. The simple uniform multilingual fine tuning method is the second best method and LF-MLF is ahead of it by only 0.5-1 point. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OVkTHmUrfTl", "reviewer": "Reviewer_7aAk", "summary": "The paper proposes an investigation of how to fine-tune a model with multilingual data effectively with a theoretical and empirical evidence. They proposed a method to avoid catastrophic forgetting by using interior-point solver to optimize the objective function based on the theoretical work. The authors claim that the method is able to reach the Pareto stationary -- no more changes occurred by optimizing multiple objectives by applying optimization. They show the superiority of the approaches in Named Entity Recognition, Question Answering and Natural Language Inference, and several languages.\n", "questions": "- Did you run any significant tests or compute the std of all results (for all runs)? The scores for LF-MLF are similar to Uniform-MLF for some tasks and settings.\n- I am curious about the running time of the optimization. Does this method efficient in practice?", "limitations": "There is no specific limitation section provided in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Interesting work with a good theoretical foundation\n- The task is very useful for learning multilingual models while optimizing the model to reach closer to the upper bound\n\nWeaknesses:\n- The paper is not easy to follow. The paper method is unclear until I carefully read the paper several times. I would say the paper would be easier to follow if they put more details on the Algorithm and the introduction. For example, what method does the paper use to optimize the weights, not just pointing to the equation. And, it would be great to add a short description of the method in the introduction so that the first readers can understand the proposed approach.\n- The method is not well-motivated with a clear hypothesis. Therefore, I cannot find the rationale why the LF-MLF is useful in the abstract and introduction.\n\nTypo:\n- In conclusion, Learning => learning ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2P4UaMjJ7n5", "reviewer": "Reviewer_R8FH", "summary": "This paper proposes a method for maintaining multilinguality under fine-tuning when a set of languages is used for fine-tuning. The core idea is to maintain crosslingual generalization by finding updates that are deviating not too much from the original pretrained model and its loss, and those that many languages benefit from. The proposed loss is and its optimization are theoretically motivated and defined, and then evaluated on downstream tasks of the XTREME benchmark with XLM RoBERTa, where a few high-resource languages are chosen for fine-tuning, and the remaining languages are used for zero-shot evaluation.   \n\n[NOTE: presentation and overall score updated after author rebuttal]", "questions": " [NOTE: all well addressed in revised version]\n1. Please discuss [26] and [27] in Related Work as well, as baselines are based on them, and the differences should be discussed.\n2. How are mini-batches represented in the loss definition? And how are they composed (mixed across languages or only one language? This might make a different for optimization. \n3. How are the \"heads\" defined in Section 3.2? Why can one not use the pretraining head after having fine-tuned the rest of the parameters to get an estimate of L_p?\n4. Figure 2 is not readable in black and white / grayscale.\n\n", "limitations": "Limitations are not addressed, except for that in some scenarios the proposed method does not outperform all other methods. Please discuss in which scenarios the method might not work, or where the assumption that being close to the original pretraining model/objective is not sufficient/desirable for downstream crosslingual generalization.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem setting of zero-shot generalization *after* fine-tuning is novel and offers impactful applications.\n2. The experimental results are largely convincing.\n3. The detailed impact studies (Section 5.3 and following) are insightful and thorough.\n\nWeaknesses: [NOTE: all well addressed in revised version]\n1. The paper misrepresents multilingual fine-tuning (\"multilingual fine-tuning, which has not been explored before\" l.34; \"the fine-tuning scenario where multiple source languages are involved in fine-tuning, namely multilingual fine-tuning, remains unexplored.\" l.59). It has been explored before, most prominently in the paper for the XTREME benchmark (https://arxiv.org/abs/2003.11080) that this very paper is also evaluating on. \"translate-train\" and \"in-language multi-task\" (from XTREME) yielded gains over monolingual fine-tuning, and should be discussed, in particular in relation to the Uniform MLF baseline presented in this work. Another work that builds on multilingual fine-tuning is for example https://arxiv.org/abs/2205.02022 (NAACL 2022). What hasn't been studied in this setting afaik is the problem of forgetting for some languages, fine-tuning was either done on all languages of interest or only on English.\n2. The abstract and introduction are not well representing the focus of the remaining sections, as it does not introduce and motivate the forgetting problem, that the proposed method is mainly developed for and the title is derived from. \n3. A comparison to all-language fine-tuning should be included as an upper bound, since training data is available for at least the NER task. \n4. Significance tests should be performed, since some differences are sometimes small (and averages across multiple languages) and might not be strong enough to conclude superiority of one model over the other. At least standard deviation could be reported as well (since it's 3 runs).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept