Robust Optimization for Multilingual Translation with Imbalanced Data
Authors: Xian Li
Hongyu Gong
Venue: NeurIPS 2021 Poster (2021)
Keywords: multilingual
optimization
machine translation
multi-task learning

Abstract: Multilingual models are parameter-efficient and especially effective in improving low-resource languages by leveraging crosslingual transfer. Despite recent advance in massive multilingual translation with ever-growing model and data, how to effectively train multilingual models has not been well understood. In this paper, we show that a common situation in multilingual training, data imbalance among languages, poses optimization tension between high resource and low resource languages where the found multilingual solution is often sub-optimal for low resources. We show that common training method which upsamples low resources can not robustly optimize population loss with risks of either underfitting high resource languages or overfitting low resource ones. Drawing on recent findings on the geometry of loss landscape and its effect on generalization, we propose a principled optimization algorithm, Curvature Aware Task Scaling (CATS), which adaptively rescales gradients from different tasks with a meta objective of guiding multilingual training to low-curvature neighborhoods with uniformly low loss for all languages. We ran experiments on common benchmarks (TED, WMT and OPUS-100) with varying degrees of data imbalance. CATS effectively improved multilingual optimization and as a result demonstrated consistent gains on low resources ($+0.8$ to $+2.2$ BLEU) without hurting high resources. In addition, CATS is robust to overparameterization and large batch size training, making it a promising training method for massive multilingual models that truly improve low resource languages. 

Robust Optimization for Multilingual Translation with Imbalanced Data
Xian Li and Hongyu Gong
Facebook Al [xianl, hygong) @fb. com
Abstract
Multilingual models are parameter-efficiet and especially effective in improving low-resource languages by leveraging crosslingual transfer. Despite recent advance in massive multilingual translation with ever-growing model and data, how to effectively train multilingual models has not been well understood. In this paper, we show that common situation in multilingual training, data imbalance among languages, poses optimization tension between high resource and low resource languages where the found multilingual solution is often sub-optimal for low resources. We show that common training method which upsamples low resources can not robustly optimize population loss with risks of either underfitting high resource languages or overfitting low resource ones. Drawing on recent findings on the geometry of loss landscape and its effect on generalization, principled optimization algorithm, Curvature Aware Task Scaling (CATS), which adaptively rescales gradients from different tasks with meta objective of guiding multilingual training to low-curvature neighborhoods with uniformly low loss for all languages. We ran experiments OPUS-100) with varying degrees data imbalance. CATS effectively improved multilingual optimization and a result demonstrated consistent gains low resources (+0.88 to +222 BLEU) without hurting high resources. In addition, CATS is robust to overparameterrization and large batch size training, making it promising training method for massive multilingual models that truly improve low resource languages.
propose a
common benchmarks (TED, WMT and
Introduction
Multilingual models have received growing interest in natural language processing (NLP) 10 26 2 [9 (58. The task of multilingual machine translation aims to have one model which can translate between multiple languages pairs, which reduces training and deployment cost by improving parameter efficiency. It presents several research questions around crosslingual transfer learning and multi-task learning (MTL) [23 1 27 45| 22).
Recent progress in multilingual sequence modeling, with multilingual translation as a representative application, been extending the scale of massive multilingual learning, with increasing number of
languages [2] 10 58) the amount of data 2 [2), as well as model size (27 [3]. Despite the power-law scaling of (English-only) language modeling loss with model, data and compute [25]] it has been found that multilingual models do not always benefit from scaling up model and data size, especially multilingual machine translation multiple languages even after exploiting language proximity with external linguistic knowledge [12 51).
There has been limited understanding the optimization aspects of multilingual models. Multilingual training is often implemented as monolithic with data from different languages simply combined.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Challenges were observed when training with imbalanced data 2 26 which is common for multilin- gual NLP as only few languages are rich in training data (high resource languages) while the rest of the languages in the world has zero or low training data (low resource languages) (24). This has been mostly treated as a "data problem" " with a widely used work-around of upsampling low resource
languages' data to make the data more balanced 2 26 TO (8).
In this paper, we fill the gap by systematically study the optimization of multilingual models in the task of multilingual machine translation with Transformer architecture. Our contribution is twofold.
First, we reveal the optimization tension between high resource and low resource languages where low resources' performance often suffer. This had been overlooked in multilingual training but have important implications for achieving the goal of leveraging crosslingual transfer to improve low resources. We analyze the training objectives of multilingual models and identify an important role played by local curvature of the loss landscape, where "sharpness" causes interference among languages during optimization. This hypothesis is verified empirically, where we found optimization tension between high and low resource languages. They compete to update the loss landscape during early stage of training, with high resource ones dominating the optimization trajectory during the rest of training. Existing approaches such upsampling low resources implicitly reduce this tension by augmenting training distribution towards more uniform. We show that this approach is not robust to different data characteristics, where it suffers from either overfitting low resources or underfitting high resources.
Second, we propose principled training algorithm for multilingual models to mitigate such tension and effectively improve all languages. Our algorithm explicitly learn the weighting of different languages' gradients with a meta-objective of guiding the optimization to "flatter" neighborhoods with uniformly low loss (Curvature-A wware Task Scaling, CATS). Compared to static weighting implied by sampling probabilities of the data distribution, our method effectively reduced the optimization tension between high and low resource languages and improves the Pareto front of generalization. On common benchmarks of multilingual translation, CATS consistently improves low resources in various data conditions, +0.8 BLEU on TED (8 languages. 700K sentence pairs), +2.2 BLEU on WMT (10 languages, 30M sentence pairs), and +1.3 BLEU on OPUS- (100 languages, 50M sentence pairs) without sacrificing performance on high resources. Furthermore, CATS can effectively leverage model capacity. yielding better generalization in overparameterized models. The training algorithm is conceptually simple and efficient to apply to massive multilingual settings, making it suitable approach for achieving equitable progress in NLP for every language.
2 Related Work
Multilingual Learning and Multi-task Learning. Massive multilingual models have been gaining increasing research interest as a result of recent advancement model and data scaling [17 10 23
19 58 However, multilingual models are often trained using the same monolithic optimization models. To deal with imbalanced training data, upsampling
as is used in training single-language low resource languages (and downsampling high resource languages) was first proposed in massive multilingual machine translation in order to improve low resource performance 23 It was adopted
in recent state-of of-tthe-rt multilingual pretraining [10 34 58).
Although recent work has looked into maximizing positive transfer across languages by learning parameter-sharing sub-networks conditional on languages [28,3]] few prior work had looked into the blackbox of multilingual optimization. Relevant efforts focus on dynamic data selection, such as adaptive scheduling [22 and MultiDDS (Differentiable Data Selection). which dynamically selects training examples based on gradient similarity with validation set [53]. Although they share the same motivation of treating multilingual training multi-objective optimization problem, data selection introduces additional computation and slows down training.
This work also adds to a growing interest in addressing interference in multilingual models, known as "the curse of multilinguality" [2] [0]. which was initially hypothesized as "capacity bottleneck" in general multi-task learning [.] Existing work have mainly focused on model architecture, e.g. via manually engineered or learnt parameter sharing across languages based on language
proximity 61 47 46). Gradient Vaccine is recent work addressing interference from an optimization
perspective by de-conflicting gradients via projection [54], a general approach MTL. 60). Although
confficting gradients are also examined in this work, they are used as evaluation metrics, where

---

show that regularizing local curvature can prevent conflicting gradients from happening in the first place. Overall, we provide new understanding of interference multilingual tasks by pointing out the optimization tension between high and low resources, which had not been studied before.
Multilingual training is instance of multi-task learning (MTL) 36
50
35
Task
balancing has been studied from both architecture and optimization perspectives
33
6
48
60)
Among them, GradNorm 66] and MGDA [48] closely related, where the GradNorm adjust gradient norms to be uniform across tasks, and MGDA adapts gradient-base multi-objective optimization to modern neural networks with efficient training.
Sharpness of the Loss Landscape and Generalization. Our analysis and proposed method are closely related to recent findings on the generalization of neural networks and optimization behaviors including geometric properties of loss landscape during training. It was observed that the "sharpness" of the loss landscape grows rapidly especially during the early phase of training by examining relevant metries such as largest eigenvalues of Hessian, spectral norm of empirical Fisher Information matrix, etc. 8 20]. Gradient alignment, measured as cosine similarity of gradients computed on different training examples, indicates the covariance of gradients, which was shown be related to generalization performance 21 (32. Our work applies those insights from optimization literature analyze the training dynamics of multilingual learning with the Transformer architecture. A closely-related concurrent work [4] has verified the effectiveness of regularizing loss sharpness in improving generalization, but single-task setting on computer vision benchmarks.
Demystifying Optimization Challenges in Multilingual Training
Notations. In multilingual translation task, typically train model with training data from set of languages N = (41.... J. and measure its generalization performance on a set of evaluation languages M. We use "language" and "task" interchangeably throughout the paper. We introduce the following notations:
Dr = ((af") ((())) refers to the set of labeled examples of language Ln A prominent property of multilingual translation task is the highly imbalanced data, i.e. the distribution of |D,)l denoted as P.D.J. is usually heavy-tailed, with few high resource (HiRes) languages, and large number of low resource (LoRes) ones 22 55
Let 0 E Rdd be the parameters of the multilingual model.
C., (, [L v)] as the expected loss as measured on language n, with token-level cross entropy loss L(ÿ. y) E. v/ log g. where V is the vocabulary the target language(s) which the model translates to.
Denote V as gradient, 2 as Hessian H, tr(.) as trace, and ||-1|2 as Fuclidean (12) norm.
3.1 Optimization Objective
Our analysis starts by taking a closer look at the training and generalization objective of mul- tilingual models. Generalization performance is measured weighted by their importance Wn. LMult = I WnCCn (z, y) For multilingual transla- tion, W,, M n = I..M indicating that all translation directions have equal importance. This population loss is usually optimized by minimizing empirical risk, i.e. a training loss Exalts := mine DM anCn( 0), where 0n corresponds to the weight of language n's loss during training.
set of evaluation languages M
We point out that as an important optimization choice, a = {a,, is either left implicit or chosen heuristically in existing work of multilingual training. A common choice for resembles the sampling probability of language Ln in mini-batch according training data distribution P((D..)) with a temperature hyperparameter T. For example, T corresponds to sampling proportional to training data sizes []..l....(D) DD I]: T = oo corresponds to uniformly sampling across languages. For most multilingual tasks, training data is highly imbalanced with a few high resource languages accounting for majority of the training data and long tail of low resource languages. Practitioners manually tune I to make minimizing training loss Lxulti a proxy of optimizing the population loss
3

---

Mull Massive multilingual translation and pretraining with real- datasets of 100 languages usually upsample low resource languages (c.g. T performance 2 34 10 58)
5) improve the averaged generalization
3.2 Local Curvature and Robust Multi-task Optimization
L2
We analyze how local curvature affects optimization
()(),)
E
efficiency of multilingual training. Task interference has been observed to pose optimization challenges in general multi-task learning and continual learning, where improvement in one task hurts performance on 60
C1
(1
02
01
In.
another task 45
On
In the context of training multilingual models, we define interference from the optimization perspective, as the change of empirical risk Ci as a result of an update to dominated by empirical risk minimiza- tion for another language's s loss L2. In Figure D we illustrate how curvature of the loss landscape affects interference similar to the analysis which has been done in continual learning [38]. After a gradient step driven by V 22. the shared model parameters moved from of to 0z. The corresponding changes in L1, i.e.
Figure 1: Left: Change of one language's loss Li after shared parameters being up- dated to 02 driven by VL2 from another lan- guage is affected by the curvature of the loss landscape around previous critical point 01. Right: Illustration of the proposed algorithm, Curvature A Aware Task Scaling (CATS), to learn task weighting re-scaling on such that the combined gradients will guide the opti- mization trajectory low curvature region (pointed by the green arrow).
L1 (02) L1 (01) is affected by the curvature.
I(C,, - L. (02) Li R 102-07 01) TV)
(0)) 01
This relationship is also summarized in Eq. (T). We make an assumption that in small neighborhood of 0j. the loss surface is almost convex 7 [6]. Then we can apply a second-order Taylor expansion of Li (02) in Eq. (2) and derive the connection between [[],, L2) and the Hessian H(0) V²
which indicates the local curvature of loss surface.
1 (02 (5) V (V (1)) 05)
(12) (0T) + (0 05) (01)
(2)
Since (LT( = 0 at critical point for L1, the major contributing factor to interference is local curvature, measured by the spectral norm of H(7)). ).)) as well I We hypothesize that the optimization tension defined above affects low resource languages more than high resource ones, assuming that low-resource tasks have lower sample complexity, and thus are more likely reach local minima during early stage of training.
Algorithm Curvature Aware Task Scaling (CATS).
as the magnitude of parameter update |02
Input: mini-batch B with K languages; number of inner loop updates m. Output: converged multilingual model O.
1: Initialize 0, 1, A O.
2: while not converged do for from 1 4: for n = 1, K do
do
3.3 Meta-learn a with curvature regularization
5:
Compute In- = V (B,)
Motivated by previous section's 's analysis, pro- pose principled optimization procedure for mul- tilingual translation with a meta objective of reg- ularizing local curvature of the loss landscape of shared parameters O. We explicitly learn the task weighting parameters
F.. clone(gn)
7:
O² = clone((x..)
end for Update O with Σ EnIn end for
9:
10:
= [o,, mo.... so as to minimize the trace norm of empirical Fisher informa-
11:
Compute V Lmmemta VaLmotay ac-
cording to Eq. (4) using Ho for V VoL...
tion (FF))= = Exz,,) [I) Σ> m an VoC., which
12:
Update a with gradient descent
an approximation tr (H) as was proposed in opti-
13:
Update À gradient ascent
mization literature 220 52 To leverage distributed
14:
end while
training in modern deep learning, we estimate a from mini-batch B with K languages as is shown in Eq.
K
K
min 0200000
Σ
On V Co..(())
s.t.
Σ n=1
an 1, a, 2 0, Vn
(3)
72

---

We treat solving Eq. 3 along with minimizing empirical risk Luulti multi-objective optimization problem, and optimizing the corresponding Lagrangian follows:
The overall training objective as
CoNTs(0. Σ Janhn - dale - On VoLn VoLn
1)2
An (a -c) (4)
We learn both model parameters 0 task weighting parameters simultancously with bi-level optimization, where update 0 in the inner loop and update a and Lagrangian multipliers À = [der A,, AI,, in the outer loop. We refer proposed algorithm CATS (Curvature-Awar Task Scaling) with the detailed training procedure described in Algorithm
Experiments
4.1 Experimental setup
Datasets. We experiment on three public benchmarks of multilingual machine translation with varying character- isties of imbalanced data as shown Table [ They are representative in terms of the number of languages (W), total volume of training data [D,,] measured as the number of sentence pairs in millions (M), and the entropy of data distribution Hy,,, indicating the degree of data imbalance. For example, distributions with multiple high resource languages are covered by experiments on the TED and OPUS100 datasets, while experiments WMT dataset cover unique scenario of extremely skewed distribution with one high resource language and a long tail of low resource ones, which is not uncommon in real world applications. Additional details and statisties of the datasets are provided in Appendix
[Dx]
H
TED 153
8
0.8
0.71
WMT 34
10
31
0.21
OPUS- 100
62
92
55
4.2
Table 1: Description of datasets and char- acteristics of data imbalance in three representative multilingual translation benchmarks.
Models. We use the Transformer architecture, and the same model size and configuration as were used corresponding baselines. For experiments on TED, the model is a 6-layer encoder-decoder, 512 hidden dimension, 1024 FFN dimension, and 4 attention heads [53] For experiments on OPUS-
100, the model is the Transformer-base configuration as used in 622 We use Transformer-bass for WMT experiments. We use the same preprocessed data by the MultiDDS baseline authors 53 and followed the same procedure to preprocess OPUS-100 data released by the baseline 662 All
models are trained with the same compute budget for comparison. We provide detailed training hyperparameters in Appendix
Baselines. We compare to strong baselines used in state-of-the -frth-rrr multilingual translation and relevant approaches in generic multi-task learning:
Proportional sampling. This is straightforward yet common approach used in practice by training
combined training data from all languages [62, which corresponds to a, = p T 1.
Upsampling low resources T 5. This has been adopted in state-of-the-art multilingual transform-
ers as is discussed in Section 3.1
recently proposed method to balance training losses in multilingual training. It learns to select training examples among different languages based gradient similarity with validation set. Although it does not address the optimization challenges studied in this work, shares the same interest of improving generalization performance and utilizing training examples a dynamic and adaptive fashion.
MultiDDS 53
We also compare to closely related approach proposed for general multi-task learning. The key difference is that the objective in GradNorm rescale each task's gradients to be closer to the average gradients norm (i.e. only based on VI,), while the objective in our approach is to minimize the second-order derivative 42 which corresponds to the curvature of the loss landscape.
GradNorm 6]:
5

---

a=p,,T= T=
a=pt,T=3 5=5
CATS
LoRes
HiRes
165
161
Amaa
cos(g Sto)
02
0.
Early stage
Late stage
Figure 2: Local curvature measured by top eigenvalue (Top) and gradient direction similarity (Bottom) of multilingual training with high-resource (HiRes) and low resource (LoRes) languages measured on TED corpus. To control for other factors such as language proximity, both high resource (Russian) and low resource (Belarusian) are chosen be from the same language family. We compare the proposed optimization method CATS with existing approaches of setting a, based on empirical distribution of training examples, i.e. On O P/D,I= and upsampling with a temperature hyperparameter T. T = oo corresponds to equal weighting (uniform sampling). In the beginning of training (Left), HiRes and LoRes competes to increase the sharpness the loss landscape, with HiRes dominating the optimization trajectory during the rest of the training (Right) and their gradients almost orthogonal. Our proposed method (CATS a) effectively reduced local curvature and improved gradients alignment.
Evaluation of Optimization. For the analysis of loss landscape, we look into the top eigenvalue (Amaz) of H computed from examples in mini-batch with the power iteration method described in [59] for each language. To evaluate optimization efficiency, we analyze gradient similarity using
the same metrics introduced in [60]. At each training step, with 91 VL1, 92 vL2 computed from training examples of two languages, we measure the following metrics as indicators of gradient similarity:
Gradient direction similarity (alignment): cos (()) .2) where & is the angle between gi and g2.
Gradient magnitude (L.2 norm) similarity: ( 92)
211g 9al2 1lan 1 19
Evaluation of Generalization. We verify whether improved optimization leads to better general- ization. We report both token-level loss (negative log- ligllioel,oood, NLL,) and BLEU scores (1) on hold-out datasets. we choose the best checkpoint by validation perplexity and only use the single best model without ensembling.
We conduct most experiments in multilingual one-to-many task it is challenging task than many-to-one, and represents the core optimization challenges underlying many-to-many task. Additional experiments with many-to-one are provided in Appendix C
4.2 Results
4.2.1
Robust Optimization
M2O,
0.03
O2M,HILo,
Abrasive Gradients between HiRes and LoRes.
0..02
HiM.D
First, we illustrate the optimiza- tion tension between high resource lan- guages and low resource languages (HiLo)). We examine abrasive gradients in Figure 3. which shows fine-gra fine-grained view of gra- dients similarity in terms of both direction (left) and magnitude (right) for different parameters in the Transformer architecture: encoder self attention (E.SA), encoder feed- forward (E.FC), decoder self-attention (D.SA), decoder encoder attention (D.EA), and decoder feed- forward (D.FC). We can see that gradients are more similar when training with balanced data, i.e. HiHi and LoLo,, compared to HiLo for two fundamental multilingual translation tasks one-to-many (02M) and many-to-one (M20), indicating that the amount of training data less a cause of the problem compared to the distribution of training data across languages. Furthermore, contrary to
HIMI
0..01
-0.7
O2M.
0..00
ESAEFCDSAD.BADFE
ESAEFCD.SAGEADRC
Figure Gradients similarity of different Transformer parameters (I-axis) in common multilingual translation tasks (y-axis), measured as gradients direction similarity (Left), and gradients norm similarity (Right).

---

E.SA
LoRes
HiRes
coSig: 92)
E.FC
NLL LOSS
2.0
baseline
LayerNorm
D.SA
train
valld
-929
D.FC
2.
-1.6
D.LN
0
50 100 150 200 250 300
0
50 100 150 200 250 300
O2M,HILo
02M,Looo
O2M,HIHI
N20,HILo
updates (k)
updates (k)
(a) Gradient alignment (g-axis) for Transformer param-(h) Stable training without LayerNorm parameters and eters across common multilingual translation tasks (r- mitigaatnnn LoRes overfitting.
axis).
Figure 4: Gradients alignment between HiRes and LoRes have high variance for Layer normalization parameters (LayerNorm) both encoder (E.LN) and decoder (D.LN), which exacerbate overfitting of LoRes. CATS allows stable training without LayerNorm parameters and reduces generalization gap for LoRes.
common heuristics which use language family information to determine which languages share parameters [12), language proximity, i.e. related (R) vs. diverse (D), has smaller effect gradient similarity compared to data imbalance.
Next, we look into optimization tension through the lens of curvature. Figure 2 (top) plots the top eigenvalues Amaz when training a multilingual model in the HiLo setting. We can see that with uniform sampling (equal weighting, T oo), LoRes updates the shared parameter space in the direction towards higher curvature during the early stage of training, while HiRes dominates the optimization trajectory (with the LoRes having negative top eigenvalue) for the rest of the training. We found that common heuristics of T = reduces the early-stage sharpness although the loss landscape is still dominated by HiRes. In contrast, the proposed optimization approach (CATS o.,) ) mitigated the abrasive optimization and increased gradients alignment as is illustrated Figure (bottom).
x=pt,T= T=5
a= pt, T=1
CATS a
Training without layer normalization.
train
Valid
We take closer look at gradients align- ment for different types of parameters in
LoRes
HiRes
2.6
NLL LOSS
the Transformer architecture. We found
2.4
that gradient alignments for layer normal-
22
2.0
ization parameters (LayerNorm, LN) are
1.0
much noisier than other parameters as is shown in Figure 4a Extensive work has provided empirical and theoretical results of LayerNorm be being critical for training stability of Transformers [3] 57 [56]. Our work is the first to show that is a double- edged sword in multilingual training and exacerbates the tension between HiRes and LoRes, which competes to set the gain and bias parameters in LayerNorm, where the final weights highly depend on training data distributions. However, simply removing the gain and bias parameters causes train- ing instability [32]. In Figure 4b) we show that with CATS optimization in place, Layer- Norm can be removed yet stable training can still be achieved without further adjusting other hyperparameters such as decreasing learning rate, etc. As result, the general- ization gap (difference between training and validation loss) for low resource is greatly reduced possibly due to that the learnt Layer- Norm parameters would otherwise be biased towards HiRes.
10 updates (k)
20
30
5
10
15
20
25
30
35
updates (k)
9.0
LOSS
4.3
3.
60 20
updates (k)
updates (k)
Figure 5: Train and validation loss (token-level nega- tive log-likelihood, N) for low resource (Left) and high resource (Right) from the same multilin- gual model. We ance with the temperature hyperparameter is not robust changing data distribution PIp. TED (Top) and WMT (Bottom). For highly imbalanced WMT dataset, the common practice of upsampling low re- source (T = 5) improves LoRes but at the cost of underfitting HiRes, while leads to LoRes overfitting for less imbalanced dataset (TED). Datasets character- isties are described in Table
that addressing data imbal-
7

---

Related
Diverse
LoRes
HiRes
Avg.
LoRes
HiRes
Avg.
aze
bel
slk
glg
por
rus
All
Lo
bos
fra 5.6 9.8 18.8 25.3 134 174 192 205
mar hin mkd ell bul
kor
All
Lo
[Pal (K)
5.9 4.5 61.5 10.0 185 208 103 182
= 00
5.4 9.6 24.6 22.5 19.9 22.2 15.9
19.9
15.5
12.7
10.8 14.8 22.7 30.
37.2
16.9
22.1
15.3
T=
5.5 10.44.4.2 39.0 19.6 22.5 16.0
19.9
15.6
13.4
10.4
14.7 23.8
30.4
32.3
37.8
17.6
22.5
15.6
T=1
5,, 22.6 20.1 21.6 16.7
20.0
15.6
13.0
11.4
25.0
15.0
34.0
40.5
18.8
23.5
15.7
GradNorm
5.3 9.8 24.5 22.6 20.1 21.9 15.4
19.8
15.6
13.7
14.4114443222
29.0
35.1
16.3
21.6
15.6
MultiDDS
6.6 12.4 20.6 21.7 15.3
11.6
17.3
15.3
14.0
4.8
25.4
27.8 29.6
7.0
18.2
14.0
CATS
11.7 24.7 40.6 20.6 22.6 27.7 16.3*
12.1 11.9 15.6 24.7 23.3 35.7 41.8 19.3 24.3*
16.1
Table Comparison of CATS with common methods used in multilingual training. We evaluate on the 8-language TED benchmark with related (top) and diverse (bottom) languages which help verify performance while controlling optimization difficulty due to language proximity. We compare to static weighting with manually tuned temperatures T = 1, 5, 100()o), and dynamic weighting such 6 and MultiDDS [53]. Results are BLEU scores on test sets per language and the average BLEU score (t) across all languages (All) and low resource languages (Lo). Multilingual training which achieves the best average BLEU bold and the strongest baseline approach is annotated with underscore. indicates the improvements are statistically significant with 0.05.
as GradNorm
Robust to different training data distribu-
27
25
a=p,.T=1
tions P/D- We show that the improved op-
22
timization reflected in generalization perfor-
@ 17
20
a=pi.
mance. In Figure 5] we report the training and validation loss of HiRes and LoRes languages throughout training on two representative bench- marks: TED (top) and WMT (bottom) with dif- ferent degrees of data imbalance. We can see that the optimal temperature hyperparameters T vary given the distinct distributions of training data P/D. these two benchmarks. For exam- ple, on WMT dataset where the training data distribution across languages is more skewed, upsampling LoRes (T = 5, oc) is beneficial, as was observed in other large-scale multilingual
15
CATS a
HiRes
10
LoRes
50 100 150 200 250 300
updates (k)
Figure 6: CATS is very effective in highly imbal- anced datasets (WMT), where common sampling approaches T = 1,5 sacrifice either low resources (LoRes) or high resources (HiRes). CATS signif- icantly improves generalization on LoRes while demonstrating better sample efficiency.
datasets 22 10 34. However, T 5 easily leads to overfitting for LoRes more balanced dataset such as the TED dataset. Compared to existing approaches of using hyperparameter T, CATS is more robust principled way address varying distributions of training data. It does not need manual tuning of hyperparameter nor suffer from overfitting or underfitting. On both benchmarks, it consistently improves LoRes without sacrificing HiRes.
4.2.2 Generalization
LoRes
MidRes HiRes
Avg.
We report translation qual- ity on three representa- tive benchmarks whose de- tailed characteristics are de- scribed in Table
kk
vi
tr 0.2
ja
nl
it
ar 0.3
TO 0.6
hi
de 27.8
All
Lo
DD.| (M)
0.1 0.1
0.2
0.2
0.3
0.8
T
1.1 27.3
13.5
13.3
28.6
25.2
14.0
14.0
29.8
14.1
26.8
19.4
17.6
T = 5
1.2
31.3
15.5
14.9
30.9
30
16.9
31.1
14.7
22.3
20.9
20.1
CATS a
2.0
33.1
19.2
16.7
33.3
32.1
19.6
35,4
18.0
25.9
23.5*
22.3*
TED. Table 2 summa- rizes translation quality im- provement on the TED cor- pus. To control for lan- guage similarity, we eval- uate on 8-language benchmark with both Related (4 LoRes and 4 HiRes from the same language family) and Di- verse (4 LoRes and HiRes without shared linguistic properties) settings as used in (53). We found simple mixing (T" 1) is a very strong bascline and complex training methods such as GradNorm
Table 3: Detailed performance on the WMT benchmark. Compared to the strongest baselines (T' 5), CATS drastically improves low and mid resource languages. Multilingual training which achieves the best BLEU is in bold and the best baseline annotated with underscore.
8

---

and MultiDDS do not generalize well (TED is a smaller dataset and more prone to overfitting). CATS achieves +0.4 0.7 BLEU on average for LoRes and +0.7 0.8 BLEU overall.
WMT. On a larger and more imbalanced 10-language WMT datset with 30.5M sentence pairs, found regularizing local curvature with meta learnt a (CATS) is more beneficial. CATS improved low and mid resource languages across the board as is shown in Table Compared to the strongest baseline (I" = 5), CATS achieved +-2 average BLEU for low resource languages and +2.6 BLEU across all languages. Furthermore, CATS more sample efficient as shown Figure
OPUS-100. Finally, test the scalability of CATS in massive multilingual setting using the OPUS- 100 benchmark [62] Results summarized in Table Con-
LoRes MidRes HiRes All
DD, W
100K
1M
sistent with previous results on OPUS- 100 [62 61 as well
29
45
as other massive multilingual model in (2], upsampling low resource with T' = 5 improves BLEU scores for low resource but at the cost of accuracy for high resource ones, The trade-off is the opposite for proportional sampling (T 1). CATS achieves the the best of both worlds, espe- cially +1.3 BLEU on low resource, +1.4 BLEU on mid resource, and +1.0 BLEU on high resource compared to the strong baseline of T = 5. We also compare to recent state-of-the model, conditional language-specifin rout- ing (CLSR) [61], which adds additional language specific parameters ( (157M parameters in total compared to 110M parameters in our experiments). By improving optimization without increasing model capacity, CATS still outperforms CLSR on low resource by +0.9 BLEU.
=
16.4
22.8
22.1
21.2
T= 5
26.8
24.6
18.9
22.3
CATS
28.1*
26.0
19.9
23.4*
61]
27.2
26.3
21.7
23.3
Table 4: Performance on OPUS-100 CATS can easily apply training the scale of ~ 100 languages and improves low resources.
5 Analysis
Robust to overparameterization. Scaling up model size has been of central interest in recent development of massive multilingual models such as GShard and Switch
Transformer with trillions of parameters (27 [3. Train-
ing overparameterized models a shorter amount of time have shown to be more efficient than training smaller model for longer time 229 39 255. Therefore, we are interested in understanding CATS' performance in train- ing overparameterized models. Figure 7 plots change in generalization (measured by BLEU score difference) as increase model capacity. CATS can effectively benefit from larger model capacity, especially in the overaparame- terized regime (300M parameters for TED dataset) where performance begins to degradate with standard training.
a= <=pt,T p=
HiRes
CATS a
LoRes
100 200
300
# params (M)
Figure CATS improves generalization in overparameterized models, while stan- dard approach suffers from overfitting
2.0
T=S
a=pi. T=1
Performance with large batch size. Large batch size is an effective way to speed up training without sacrific-
1.5
CATS
BLEU
1.0
ing performance [8] 49]. However, heterogeneous train- ing data do not necessarily benefit from large batch size training[37] This is a challenge for multilingual training as is shown in Figure 8 where increasing batch size hurts generalization for LoRes with a common practice of up- sampling (T 5). This is likely due to LoRes is prone to overfitting (illustrated in Figure 5) and larger batch size exacerbates it. The same batch size without upsampling leads to improved generalization (T = 1). In comparison, CATS can effectively benefit from larger batch size due its adaptive rescaling of gradients.
0.5
0.0
-0.5
LoRes
HiRes
Figure 8: CATS is robust in large batch size training (4 batch size from 33K to 131 tokens).

---

Ablation layer normalization. To further un- derstand the effect from removing layer normal- ization which mitigates LoRes overfitting as was shown in Section 4.2.1 we provide an ablation study on its role when combined with CATS. We ran experiments on TED Diverse dataset with the strongest baseline (T' 1). In Table we can see that CATS brings additional +0.5 BLEU to low resources top of the +0.3 BLEU im- provement from removing layer normalization parameters (-LN).
LoRes
HiRes
Avg.
bos
mar
mkd]
ell
bul
kor
All
Lo
Bascline 24.3 10.7 22.9 38.0 39.1 40.5 19.1 22..5 22.8 -LN only 24.4 10.6 23,7 33,9 38.4 39.5 40.9 19.5 28.9 23.1 38.4
CATS 11.6 11.3 24.2 34,1 39.1 40.1 41.2 19.6,29.3 23.6
Table 5: Ablation study of the effectiveness of CATS when combined with removing layer nor- malization parameters (-LN),
6 Conclusion
In this work, look under the hood of monolithic optimization of multilingual models. We unveil optimization challenges arising from imbalanced training data, where low resource languages have been sub-optimally optimized. We proposed principled optimization algorithm for multilingual training with adaptive gradient rescaling of different languages where the scaling is learnt with a meta-objective of guiding the optimization solutions with low local curvature.
We evaluated the proposed method on three representative benchmarks (TED, WMT and OPUS100) which cover wide range of imbalanced data distributions commonly seen in real word multilingual datasets. Compared to existing methods of simply mixing the data or manually augmenting the data distribution to be more "balanced" with temperature hyperparameter, the proposed training method demonstrates robust optimization and consistently improves generalization for low resource languages. Further analysis shows that the proposed approach is suitable for the realistic training settings of large-scale multilingual learning such as overparameterized models, large batch size, highly imbalanced data, well large number of languages etc., paving the way for advancing massive multilingual models which truly benefit low resource languages.
Broader Impact. Recent progress in NLP enabled by scaling up model size and data is widening the gap of technology equity between high resource languages (such as English) and low resource languages. Multilingual model is a promising approach to close this gap. However, current language agnostic multilingual models does not effectively improve low resources for various reasons to be understood. Our investigation in optimization is step towards building truly inclusive multilingual models.
This work has several limitations which hope to extend in future work. We did not run experi- ments on translation between non-Eno non-English directions including zero-shot. The absolute size of the overparameterized model in our analysis is relatively small compared to the state-of- state-of-the-ar of model scaling of billions of parameters. As an language generation application, machine translation model could produce unsafe output or hallucination.
10

---

References
[1] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3874-3884, 2019.
[2] Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neu- ral machine translation in the wild: Findings and challenges. arXiv preprint 1XXix1907.00009 2019.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization. arXiv preprint 2016.
[4] Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
[5] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
[6] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803. PMLR, 2018.
[7] Anna Choromanska, Mikacl Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pages 192-204. PMLR, 2015,
[8] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gra- dient descent on neural networks typically occurs at the edge of stability. arXiv preprint WvXi::2103.00055 2021.
[9] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th international conference on Machine learning, pages 160-167, 2008
[10] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. 2019.
[11] Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2051-2060, 2017.
[12] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El- Ei-Kish Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, al. Beyond english- centric multilingual machine translation. arXiv preprint arXiv:2010.1 111.2, 2020.
[13] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021.
[14] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[15] Stanislav Fort, Pawel Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: new perspective on generalization in neural networks. arXiv preprint 1eXXv:/901.094 2019.
[16] Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv: 14Xi::4412.6544, 2014.
[17] Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. arXiv preprint (XXi::800.00689) 2018.

---

[18] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer. generalize better: closing the generalization gap large batch training of neural networks. arXiv preprint 17Xiv:1705.08741, 2017.
[19] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin John- son. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411-4421. PMLR, 2020.
[20] Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo Kerg, Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. arXiv preprint 20Xiv:2012.141995 2020.
[21] Stanislaw Jastrzebski, Maciej Szymozak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2019.
[22] Sébastien Jean, Orhan Firat, and Melvin Johnson. Adaptive scheduling for multi-task learning. arXiv preprint arXiv: I 1909.06434, 2019.
[23] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google's multilingual neural machine translation system: Enabling sero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351, 2017.
[24] Pratik Joshi, Sehastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint 2000: 2020.
[25] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
[26] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:190..07290 2019.
[27] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint
(00:2000666668, 2020.
[28] Xian Li, Asa Cooper Stickland, Yuqing Tang, and Xiang Kong. Deep transformers with latent depth. Advances in Neural Information Processing Systems, 33, 2020.
[29] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gon- zalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference Machine Learning, pages 5958-5958889 PMLR, 2020.
[30] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi-task learning. arXiv preprint 1xXiv:191..12855 2019.
[31] Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. Learning language specific sub-network for multilingual machine translation. arXiv preprint arXiv:2105.094 2021.
[32] Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding why neural networks generalize well through gsnr of parameters. In International Conference on Learning Representations, 2019.
[33] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1871-1880, 2019.
[34] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
12

---

[35] Kevin Lu, Aditya Grover. Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 2021.
[36] Minh-Thang Luong, Quoe V Le. Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence sequence learning. arXiv preprint arXiv:/511.06114 14, 2015.
[37] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv: 1822.066, 2018.
[38] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning, arXiv preprint arXiv:2006.06955, 2020.
[39] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv:1912.0229,, 2019.
[40] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.010 01038, 2019.
[41] Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? arXiv preprint arXiv: 01502, 2019.
[42] Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.000 2018.
[43] Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? arXiv preprint 180.006304 2018.
[44] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swictojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 69933. IEEE, 2020.
[45] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1706. 2017.
[46] Devendra Singh Sachan and Graham Neubig. Parameter sharing methods for multilingual self-attentional translation models. arXiv preprint 1809.08 2018.
[47] Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. Multilingual unsupervised nmt using shared encoder and language-specifie decoders. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083-3089, 2019.
[48] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. arXiv preprint
2018
[49] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't 't decay the learning rate, increase the batch size. arXiv preprint arXiv: 2017.
[50] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng. Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv: / /904.09223, 2019.
[51] Yuqing Tang. Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.0044 00401, 2020.
[52] Valentin Thomas, Fabian Pedregosa, Bart Merriënboer, Pierre-Antoina Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. International Conference on Artificial Intelligence and Statistics, pages 3503-3513, PMLR, 2020.
13

---

[53] Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neu- ral machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8537,,53,, 2020.
[54] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874, 2020.
[55] Shijie Wu and Mark Dredze. Are all languages created equal in multilinguul bert? arXiv preprint arXiv:2005.09093 2020.
[56] Ruibin Xiong, Yunchang Yang, Di Hc, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In International Conference on Machine Learning, pages 10524- 10533. PMLR, 2020.
[57] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. arXiv preprint arXiv: /91 1.07013, 2019.
[58] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text trans- former. arXiv preprint arXiv:2010.11934 2020.
[59] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. arXiv preprint arXiv: 102.02180,.08241, 2.
[60] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33, 2020.
[61] Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-specific ca-pacity for multilingual translation. In International Conference on Learning Representations, 2021.
[62] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.1 1867, 2020.
14


Reviews:
[{"review_id": "wPNCc0Y3HvZ", "reviewer": "Reviewer_SuVh", "summary": "This work looks at the optimization of multilingual NMT models. Typically, the data across different language pairs is highly skewed and prior work either upsamples low-resource languages such that it's uniform or uses a temperature (T=5) to upsample lowRes and downsample HiRes. This work proposes a new optimization algorithm called CATS which encourages solutions with low local curvature. The proposed technique has been shown to be effective when compared to 3 temperature-based baselines (T=1,5,infinity) and two dynamic methods multiDDS and GradNorm. Experiments are performed on 3 public datasets: TED, WMT and OPUS-100. Robustness of the technique to overparameterization and to large batches has been studied.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- The paper approaches the optimization problem from a novel standpoint and establishes that low local curvature affects optimization. Then it proposes an algorithm that jointly also learns weights \\alpha which regularizes local curvature of the loss landscape.\n\nClarity:\n- For the most part, the paper is clearly written. However, there are a lot of minor mistakes. The figure numbers are mixed up and the paper could have been more polished.\n\nPositives:\n- The paper proposes a novel optimization algorithm called CATS which alleviates the lowres/highres interference problem in multilingual NMT.\n- Results on 3 standard benchmarks against many strong baselines show that the proposed approach leads to improves across lores, midres and hires language-pairs.\n\nNegatives:\n- The paper should have had a dedicated section on the complexity of the proposed approach. This has largely been ignored. How much extra time does the additional optimization add to training? How stable is the proposed approach? \n- There are a lot of presentation mistakes which could have been easily avoided and a bit more care in organizing figures and tables better could have helped the reader.\n\nMinor comments:\n- L74: I think the idea of upsampling low-resource languages was already proposed in [20].\n- L125: $N$ should be \\mathcal{N}?\n- L249-251: \"Contrary to popular belief...\" can you please cite prior work that states that language similarity has this effect?\n- L241 Figure 5b shows... Do you mean Figure 3? \n- Figure 4 I believe the legend should say T=infinity and not T=1.\n- L279 \"empirical an\" --> \"empirical and\"\n- Section 4, why was the T=infinity baseline not evaluated for WMT and OPUS-100 corpora?\n- L367 \"principle optimization..\" --> \"principled..\"?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "cE1dccCG9r", "reviewer": "Reviewer_V2N9", "summary": "This paper theoretically motivates and then implements a curvature-based, online, per-language learning rate adjustment to improve the performance of low-resource languages in multilingual machine translation (MT) models. Experiments in three scenarios varying in multilinguality and data size show that the technique works consistently, outperforming a widely-adopted temperature-based sampling baseline. Through a number of analysis and ablation experiments, they are able to provide insight into how and why the optimization technique works.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper takes a step toward solving an important problem (balancing languages in multilingual MT) using a well-motivated technique influenced by recent advances in the optimization literature. It has a very good balance of clear motivation, theoretical background, empirical results and analysis. The method may be too complex to see wide adoption without a clever or clear open source implementation (the baseline, after all, is very very easy to build), but the ideas and the connections to the optimization literature are likely to be influential regardless. The paper definitely has more than enough technical content to warrant a conference paper, and is bordering on a journal paper. I learned a lot and earmarked many other papers from the substantial bibliography to follow up on.\n\nIn terms of weaknesses, the paper is very dense. In particular, the analysis sections require a lot of effort to understand what was done and why. I spent much more time than I would have liked staring at legends, captions (sometimes multiple captions per figure!), and cross-referencing with the text to try to figure out what’s going on. Both halves of Figure 5 are particularly confusing. However, this time was generally worth the effort.\n\nSmall concerns:\n\nThe paper is littered with small disfluencies (particularly confusion of singular and plural nouns and corresponding agreement), but I found it did not hinder my understanding.\n\nI found the definition in Section 3 of p_{|D_n|} confusing, but its intended meaning quickly became clear after seeing its use in the (familiar) temperature sampling baseline.\n\nIn the first paragraph of section 4.2.1 when Figure 5b is referenced, I’m 95% sure Figure 5a is what was intended.\n\nAlso in 4.2.1, I found the phrase “data volume” ambiguous and confusing. I guess you mean the total amount of training data for all languages? As opposed to the amount of data for any one language?\n\nIn general, the relationship between CATS (the proposed method) and layer norm is not clear in the main text. Figure 5b and the “Training without layer normalization” paragraph, together with the “Ablation layer normalization” paragraph seem to suggest that layer normalization is always switched off for CATS throughout all of the paper’s main results, but this is never explicitly stated in the main text. Likewise, in Figure 5b, it isn’t clear what the baseline system is (is it CATS or T=5 or T=1?). Presumably, the w/o layer norm line is just the baseline (whatever it is) without layer norm.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is handled nicely in the appropriate section.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "P4xAmYGWBp_", "reviewer": "Reviewer_kfHY", "summary": "This paper studies the problem of robustness optimization in multilingual machine translation models. In response to this problem, the author empirically found that in the loss optimization of multilingual machine translation models under unbalanced data scenarios, \"sharpness\" of local curvature in the loss landscape causes interference among languages, which caused instability in training. Inspired by the robust optimization work of Multi-task Learning, the paper proposes the Curvature Aware Task Scaling (CATS) optimization algorithm, which alleviates the phenomenon that different translation directions competitive update the loss landscape during the early stage of training in multi-language translation training, while high resource ones dominating the optimization trajectory during the rest of training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed CATS optimization method has achieved better results than the multi-objective optimization methods: GradNorm and MultiDDS.\n2. The paper has done a very good analysis and visualization of the \"sharpness\" of local curvature in the loss landscape for the multilingual machine translation model in the unbalanced data scenario, making the research problem very intuitively presented.\n3. This work also conducts experimental exploration on more practical scenarios: overparameterization and large batch size, which makes the proposed method more practicable. \n\nWeakness:\n1. The unbalanced data scenario has not been properly explored by experiments. Under what circumstances can it be counted as an unbalanced data scenario, and what is the data ratio? Therefore, the experiments should not pay more attention to one given setting like TED, WMT, etc., but should construct unbalanced scenarios of different ratios by sampling data in one setting like WMT to verify this important issue.\n2. There is a lack of a reasonable ablation study on the upsampling parameter T, so we cannot confirm whether the oversampling overfit phenomenon will occur, and to what extent will the upsampling reach.\n3. Some baselines are missing in the experimental comparison, such as 1) giving different weights to the loss of unbalanced translation pairs so that in the later stages of training, there will be no situation where rich-resource pairs dominate the training loss; 2) the use of low-resource language pairs further finetune the multilingual model and use the method like R3F to maintain the generalization ability of the model.\n4. In some low-resource language translations from 1.2->2.0, although the improvement of 0.8 can be claimed, it is insignificant in a practical sense.\n\nMissing References:\n1. Aghajanyan, Armen, et al. \"Better Fine-Tuning by Reducing Representational Collapse.\" International Conference on Learning Representations. 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "JFpFXyQrWQB", "reviewer": "Reviewer_aPhj", "summary": "Paper presents a curvature aware bi-level optimization algorithm applied to multilingual/multi-task machine translation. The proposed algorithm is designed to encourage the loss is reduced along the directions of low-curvature (with the implicit inductive bias of flat minimas being better at generalization). Akin to \"Sharpness-Aware Minimization (Foret et al. 2020)\" but in a multi-task environment. Experiments on three different datasets tests the effectiveness of the proposed method (called Curvature Aware Task Scaling - $CATS$). Results show clear trend that the proposed method is highly effective at improving the quality of low-resource languages. Further analysis provided to understand the tradeoffs, sensitivity and dynamics of the algorithm. \n\n- Making use of curvature information during the optimization of multi-task models for translation is novel and expected to be impactful in the long run (while the paper is not demonstrating convincing empirical results, it is showing the potential of using higher order signals during optimization, which I found to be an important contribution).", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "At high level paper proposes to learn to weight/scale the gradients of different languages/tasks in a bi-level optimization setup. The signal used for scaling the gradients is coming from the curvature (flatness) of the loss landscape, where the inductive bias is to guide the optimization towards flatter regions of the loss landscape. \n\nAuthors found that optimization tension or the tug-of-war between high and low resource languages is due to the observation that they compete to update the loss landscape during early stage of training, with high resource ones dominating the optimization trajectory during the rest of training, and connect this to the existing methods (like upsampling low-resource languages). \n\nPaper provides a new perspective on the problem of interference in multilingual models which I found novel.\n\n**Questions**\n- What is the number of inner optimization steps `m`\n- Figure 1(right) is not clear, how do the colored arrows relate to the (left)? Perhaps add some additional comments to the caption and the main text. \n- Figure 3 (left) the range of similarities are quite narrow, $\\pm 0.01$, I'm a bit reluctant to make claims about gradient similarities (in terms of direction) given the range of the Figure, color-coding is misleading. But I agree that the available signal is somewhat supporting the argument \"gradients are more similar when training with balanced data\".  \n- What is the inner optimization step rule, `Adam`, `SGD` or else?\n- What is the value of the epsilon in $Eq.4$ ?\n- In Section 3 you mention that you *only* report one-to-many results, but in $Fig.3,5$ you mention many-to-one. I couldn't get the motivation, could you please clarify?\n- Section 4.2.1 refers to $Fig.5$ but i guess it should be citing to $Fig.3$ right?\n- What is the method you used to compute the top-eigenvalues of the Hessian? $Fig.2$\n- You have a section detailing the use of layer-norm and its impact on the gradient alignment. What is the architecture you are using in the first place? Pre-norm Transformer or post-norm?\n- When you use $CATS\\alpha$ how do we choose the sampling ratios? Also in the appendix you mention that you mix K=4 (TED&WMT) and K=8 (OPUS100), what is the rationale behind these, and did you measure the sensitivity of $CATS\\alpha$ for K. \n- If we use T=5 and $CATS\\alpha$ would that yield us the best adaptive schedule? Could you please elaborate on this?\n- When you are plotting the NLL ($Fig. 4,5$), are you using $Eq.4$ ? What is the value we are seeing in the plots? (Asking because there is almost $1.0$ diff between different methods vs $CATS\\alpha$)\n- In WMT experiments, only German is added to the mix and is considered to be high-resource language. While it is the highest resource language in the mix (28M examples), it is still dwarfed by languages in the WMT (Czech, Chinese, French, Spanish ...) in terms of the amounts of data. I'm curious why you chose to add only 1 high resource language in the mix? Considering the gap between temperature based sampling and $CATS\\alpha$ in $Table.3$ I suspect the ranking would change if you were to add more high resource languages. Given that the community is mixing high and low resource languages in a more uniform way (ie. by not including only one high resource language but as many as available), do you think the choice of languages is introducing a selection bias and skewing the results? I'm very much okay with the framing and main contribution as boosting the quality for low-resource languages btw. \n- Experimental results does not support your argument on \"without hurting high resources (line 18, line 64)\". The only realistic high-resource setting in all your experiments (TED, WMT, OPUS) is the WMT English->German direction, and with the experimental results you present in $Table 3$ (T=1 26.8 BLEU vs $CATS\\alpha$ 25.9 BLEU) we cannot support the argument (also in $Table.4$ if we \"assume\" language pairs that have 1M training examples being \"high-resource\", final results are contradicting with the argument 22.1 vs 19.9 BLEU). Please revise accordingly (I think low-resource boost with this principled approach is still a valuable contribution, but we should be cautious about not overclaiming). \n- Baselines: I was puzzled by the motivation behind the selected baselines (Table 2). GradNorm is a method that has never been applied to the domain of interest here. If you would like to compare your method with a solid baseline exploiting the gradient directionality during optimization, it would be better to compare it against \"Gradient Vaccine\". Similarly, MultiDDS is making use of the validation set information, which is making it hard to compare since its inductive bias is quite different. I think the closest approach to your proposed method is Foret et al. 2020 \"Sharpness-Aware Minimization for Efficiently Improving Generalization\", which is also not applied to multilingual translation (but along the lines of using GradNorm as a baselines). I'd really like to see a section discussing what are we trying to measure by the selected \"baselines\".\n\n\n**About the background and related work:**\n-  2nd paragraph of your intro: [1] and [2] are not multilingual translation papers. While they are related work, they do not support the argument you are trying to make in the paragraph (which grounds the motivation). Please either remove the unrelated papers from the motivation (which will help the reader to locate the relevant information) or change the framing of the paragraph to include NLP in general. \n- Again in the 2nd paragraph, papers [10, 24, 46] are not providing the evidence you are using to motivate the manuscript. The benefits on some dimensions start to diminish, but some others keep improving with more scale and data (eg. the quality on (extremely) low-resource languages start to diverge and regress - but this only supports the argument on scaling model, ). Rather than motivating it along the line of deficiency of one particular branch in the multilingual NMT (data, scale, # languages), you can still motivate your work since optimization is a canonical branch encapsulating all. In short, I don't think you need to motivate your work which improves the optimization angle by showing the deficiency of other branches, but rather by showing the lack of compelling approaches that address optimization in particular (which the 3rd paragraph does nicely).\n- Please credit [34] for the section 3.2 where both the motivational figure, equations and the narrative are very much alike with aligned flow, and perhaps point the reader to [34] for more details. \n- On Interference, or \"the curse of multilinguality\", paper [8] rebrands the interference problem, which was studied and introduced by [2] with \"capacity bottleneck\". It is not clear that the problem is exclusively related to the \"multilinguality\", as it could be observed under several other scenarios like multi-domain, multi-view, multi-class problems (in fact the capacity bottleneck is introduced in the seminal paper by Caruana \n on multi-task learning where the abstraction is over tasks, not languages). I believe this will make the framing of the paper more generic and applicable beyond \"multilingual\" translation domain. If you don't want to abstract it further, please either be more specific under multilingual translation context, or apply proper credit assignment if you would like to highlight when and under which context it was introduced. \n- One thing that made me frown a few times while reading the paper was the way paper uses the previous work (citation rationale). Not all the citations are really supporting the argument the authors are trying to make, they sometimes appear to be remotely related or not related at all (we can of course bundle all of them under NLP applications if we wanted to). Please do another pass with this in mind and clean the citations. I hope the authors will improve this point. \n\n**Typos**\n- line 190: \"pubic\" --> public\n- Sub-captions of Figure 5 are hard to decouple between (a) and (b). Could you please use some more spacing?\n- line 264: temperate -> temperature\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of training multilingual machine translation model in presence of imbalanced data, which is common in multilingual parallel corpus. The method is based on empirical observation that in multilingual MT training, "sharpness" of local curvature in the loss landscape causes interference among languages and instability. The paper then propose CATS, a curvature-based, online, per-language learning rate adjustment to improve the performance of low-resource languages in multilingual machine translation (MT) models.
Experiments in three scenarios  varying in multilinguality and data size (WMT-10, TED, OPUS-100) show that the technique works consistently, outperforming a widely-adopted temperature-based sampling baseline. Through a number of analysis and ablation experiments, they are able to provide insight into how and why the optimization technique works.
Experiments can be improved with more analysis (e.g. upsampling T). The paper also lacks comparison with some recent strong baselines for multilingual machine translation. The authors may consider moving some results and analysis from appendix to the main content. The overall writing, including figures, tables and captions, could be improved as suggested by all reviewers. 



Decision: Accept (Poster)