Examining Scaling and Transfer of Language Model Architectures for Machine Translation
Authors: Biao Zhang
Behrooz Ghorbani
Ankur Bapna
Yong Cheng
Xavier Garcia
Jonathan Shen
Orhan Firat
Venue: ICLR 2022 Submitted (2022)
Keywords: language modeling
machine translation
prefixlm
causallm
model scaling
zero-shot transfer

Abstract: Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs. In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic variations of data conditions and model sizes. Our results show that: (i) Different LMs have different scaling properties, where architectural differences often have a significant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices, including causal masking and language-modeling objectives for the source sequence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, but improve greatly on zero-shot directions by facilitating the reduction of off-target translations.

Under review as a conference paper at ICLR 2022
EXAMINING SCALING AND TRANSFER OF LANGUAGE MODEL ARCHITECTURES FOR MACHINE TRANSLA- TION
Anonymous authors Paper under double-blind review
ABSTRACT
Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concate- nated sequences single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing. In machine trans- lation, EncDec has long been the favoured approach, but with few studies inves- tigating the performance of LMs. In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic vari- ations of data conditions and model sizes. Our results show that: (i) Different LMs have different scaling properties, where architectural differences often have a significant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices. including causal masking and language-modelinn quence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, but improve greatly on zero-shot directions by facilitating the reduction of off-target translations.
objectives for the source se-
INTRODUCTION
The popularity of large, general-purpose text generation models has skyrocketed recent years due to their outstanding performance across wide range of natural language processing tasks (Brown et al., 2020; Raffel et al., 2020: Liu et al., 2020; Xue et al., 2021). These studies, under large-scale pretraining and also model scaling, show the promise of moving to unified neural architectures and that dropping various taskispecific inductive biases improves model generalization and/or perfor- mance (Devlin et al., 2019; Kaplan et al., 2020). In neural machine translation (NMT), essential task-specific inductive bias is to separately handle source sentence understanding and target sentence generation with the encoder-decoder paradigm (EncDec). Although such bias significantly benefits translation (Vaswani et al.. 2017: Chen et al., 2018; Aharoni et al., 2019; Barrault et al., 2020: Ansari al., 2020), some recent work shows insights challenging it, for example, aggressively simplifying the decoder yields little to no compromise on translation quality (Kasai et al., 2021). This thereby inspires the question how the removal of this separation, i.e. using a single unified module for both encoding and decoding (LMs), works for translation, and whether we can get any benefits out of that.
Although some studies reported promising translation quality with LMs (He et al., 2018; Wang et al., 2021), they compare models under merely one configuration (model size), neglecting that neural models follow scaling laws (Kaplan et al., 2020; Ghorbani et al., 2021; Gordon et al., 2021) where the impact of each added parameter on model performance might vary across different models. How the inductive biases of LMs and EncDec impact the model's performance and the amounts of training data are intriguing yet missing in the literature.
increase their size
In this paper, explore various configurations of LM architectures for translation as in Figure 1, and compare them with EncDec on model scaling and cross-lingual transfer. We conduct a system- atic study under variety of data conditions, tasks (bilingual, multilingual and zero-shot) and model

---

Under review as a conference paper at ICLR 2022
Attention Mask
Source MLE Loss (optioal)
Target MLEE Loss
Xi Xi Ka V
Xo
Top Enceding
Feed-Forward Sublayer
PrefixLM
CasuelLM
Self-Attention
Self-Attention
Masks in Encoder- Encoder-Decoder Model
LayeWWan Freoding
X Xi
Xa
<B Viy:
Takeo Embeddlings
Xi
X2
Xs
<S>
OOO
Position Embeddingg
2
mask
Figure I: Illustration for translation translation-orientedd oriented language models. X and denote source and target input, respec- tively. To enable translation, we adapt the LM self-attention mask to either the PrefixLM mask or CausalLM mask (top right), where filled black circles indicate disallowed attention. We also explore top-only encoding (Top Encoding) for PrefixI.M which feeds the final-layer source encedings to generation similar EncDec, rather than layer-wise coordinated encodings (He et al., 2018). Masks of EncDec shown in the bottom right for comparison.
capacities. We examine architectural design choices associated with LMs, including causal masking (CausalLM) vs. full-visible masking (PrefixLM) for source sequences, layer-wise coordination (He al., 2018) vs. final-layer source encodings (TopOnly) for target sequence generation, increasing LM depth vs. width, and also the effect of adding source language modeling loss for CausalL.M.
We evaluate how these architectural priors affect translation quality increase the number of parameters available to the model, under a diverse set of bilingual translation settings and also (mas- sively) multilingual settings, with special focus transfer low-resource languages and zero-shot directions. Our main findings are summarized below:
LMs show different scaling properties compared to EncDec. The architectural differences become less important models scale, measured by reduced quality gap against EncDec, regardless of the language similarities, training data conditions and evaluation settings.
PrefixLM variants often outperform their CausalLM counterparts: increasing LM depth benefits the translation task more than increasing the width; and adding source-side lan- guage modeling objective to CausalLM doesn't 't affect translation quality.
Cross-lingual transfer also benefits from model scaling, where EncDec almost always dom- inates the quality Pareto frontier on supervised directions while zero-shot translation favors PrefixLM LMs. PrefixLM LMs significantly reduce off-target translations.
Although LMs could reach or even surpass the translation quality of EncDec, they still lag far behind EncDec with respect to computational efficiency as measured in FLOPs.
2 RELATED WORK
Using language models in the task of translation has long history, particularly in the of statis- tical machine translation (SMT) where LM was used as separate yet crucial component ensuring the fluency of generation (Stolcke, 2002: Heafield, 2011; Koehn, 2010). With neural networks, NMT unified those isolated SMT components including LM under the encoder-decoder formula- tion (Kalchbrenner & Blunsom, 2013; Cho et al., 2014: Sutskever et al., 2014; Bahdanau et al., 2015), which makes use of separate modules to process input and output. Further studies exploring architectural modifications by using LM alone as translation model, nevertheless, got much less attention. He et al. (2018) proposed layer-wise coondination between encoder and decoder with tied weights, where each decoder layer attends to its corresponding encoder layer at the same depth opposed to the conventional method of attending the top-most encoder representations. Later,
Also known unidirectional vs bidirectional language modelling, where in the unidirectional case token representation takes into account only the preceding tokens and their representations, but the hidirectional case takes into account both preceding and following tokens in sequence.
2

---

Under review as a conference paper at ICLR 2022
Objective
Structure
Sre-Sre
Parameter
Model
log P(X)
log X))
Layer- Wise
TopOnly
Mask
Sharing
EncDec
Full
X
PrefixLM
Full
-TopOnly
Full
CausalLM
Causal
+TgtOnly
Causal
Table 1: Comparison different models. X/Y: source/target input. Layer-Wise: layer-wise coordination; TopOnly: use topmost-layer source encodings; Sre-Src Mask: the intra-source masking schema, either fully visible (Full) causal (Causal); Parameter Sharing: share parameters between source and target.
Fonollosa al. (2019) extended it with locality constraint. Dong et al. (2019) explored LMs for sequence generation under large-scale pretraining. Despite reporting promising results, these prior studies either focus only on bilingual tasks or do not consider the scaling properties of the models, leaving the picture incomplete: how the findings will change languages benefit from/interfere each other as the architectural priors (inductive biases) change.
scale the models and how the
Neural models follow some scaling laws. Kaplan et al. (2020) reported the test cross-entropy loss of LMs can be formulated as a power-law scaling function of either model size (excluding embed- ding parameters) dataset size. Later on, researchers examined and confirmed such findings across different domains, including vision modeling (Zhai et al., 2021), knowledge transfer from pretrain- ing (Hernandez al., 2021), autoregressive generative modeling (Henighan al., 2020), and neural machine translation (Gordon et al., 2021; Ghorbani et al., 2021), to name a few. We find it essential study the scaling behavior of new architectures and approaches given the recent evidence on the emergent properties of the models at scale (Brown et al., 2020).
Another critical component in machine translation the number languages being considered with the models, which is the very focus of multilingual NMT (Firat et al., 2016). Cross-lingual transfer multilingual NMT often results from parameter sharing across languages, which bene- fits low-resource languages and also enables zero-shot translation (Johnson et al., 2017), although the quality on zero-shot directions is largely hindered by the off-target translation problem (Ari- vazhagan et al., 2019; Zhang et al., 2020). The structure of LMs further encourages parameter sharing, offering a chance to improve the transfer while magnifying the problem interference incgative-transter (Wang et al., 2020; Zhang et al., 2021). Very recently, Wang et al. (2021) an- alyzed the cross-lingual transfer behavior CausalL.,, and reported encouraging zero-shot per- formance. However, did not observe the same results likely because of data sampling, model architecture and optimization differences which zero-shot transfer is sensitive to.
3 LANGUAGE MODEL ARCHITECTURES FOR TRANSLATION
In this section, we briefly review EncDec and then present LM architectures for translation based on Transformer (Vaswani al., 2017). Table compares different models. Given source sequence X of length X and its target translation Y of length (Y). EncDec performs translation via the following structure:
x = FFN o SAtt (x Y = FFN o CAtt o SAtt (Y'-1, -y)-),x+).
(1)
where denotes the layer index and indicates consecutive sublayers. X E x and Y! E | are the layer representations of the source and target sequence respectively, with model dimension of d. The first input layer (X", Y is the summation of token embeddings and their positional encodings. We drop all the layer normalization and residual connections in our formulations for brevity.
The encoder is a stack L layers, each of which includes a multi-head self-attention sublayer (SAtt) followed by feed-forward sublayer (FFN). SAtt in the encoder is bidirectional with full- visible masking that has full visibility to all source tokens, preceding and following. Its final-layer representations XL are fed the decoder, which shares similar structure the encoder but with
3

---

Under review as a conference paper at ICLR 2022
#Samples (Sources)
Experiments
Dataset
Train
Dev
Test
BIL MUL
WMT14 En-De
4.5M
3000 (WMT13)
3003 (WMT14)
WMT14 En-Fr
41M
3000 (WMT13)
3003 (WMT144
1997 (WMT19, SO)
WMT19 En-Zh
26M
3981 (WMT18)
2000 (WMT19 TO)
4927/1997
(Web/WMT19, SO)
Web En-De
2B
7927 (Web)
6000/2000 (Web/WMT19, TO)
Table 2: Statistics of different datasets. M/B: million/billion; SO/TO: source mource-originalttrge -originaa test sets; Web: in-house web-crawled datasets; MUT: the data is used for bilingual/multilingaal experiments.
an additional (multi-head) cross-attention sublayer (CAtt). Unlike encoder, SAtt in the decoder is unidirectional with causal masking, where attention following tokens is disabled (masked). CAtt can always access all source inputs, though. Note we set the encoder and decoder depth equally to L, and d" denote the intermediate dimension of FFN. EncDec is often optimized with the target translation objective based on Y
EacDDe (X,Y) Y) L - log P(Y|X,YL)
(2)
Rather than separately modeling source and target sequences, LM handles both with single module:
[X Y' FFN SAtt ([x'-1,
-1 Y .M).
(3)
where M {0, 1 1 is the attention mask that controls the information flow within the concatenated sequences ((. Two LM variants explored by changing the structure of mask M, PrefixLM and CausalLM.
PrefixLM merges different modules EncDec, trained with CGT Its attention mask
M (i,j) = 1, if i 2 j or < [X] otherwise O, combines the encoder/decoder self-attention mask and the cross-attention mask of EncDec.
(4)
CausalLM, by contrast, strict LM that applies causal masking to both sequences: CausaLLM (i,j) = 1, if 2 j; otherwise O.
(5)
Apart from C C also includes the source-side language modeling loss for training:
[Causall M (X, Y) = LSRC + LGT log P log P(YIX, Y
To improve our understanding LMs for translation, we further incorporate two extensions:
PrefixLM TopOnly
The model defined Equation performs attention over the source and target sequence within the same layer. In contrast, EncDec always uses the topmost-layer source encodings for translation. We mimic this with TopOnly extension by feeding top- layer encodings, i.c. X instead of X each attention sublayer. operates the same as EncDec but with the parameters of encoder and decoder tied.
CausalLM + TgtOnly The inclusion the source-side objective enriches CausalLM's learning signal and encourages the model to absorb source language characteristics. However, requires and occupies part of modeling capacity, which might negatively affect translation. To offset this impact, we add the TgtOnly extension that optimizes CausalLM with the tar- get translation objective CTGT alone, which also aligns better with EncDec and PrefixLM.
SETUP
Model Setting We use Transformer for experiments. By default, we adopt the base setting, with d 512, dt = 2048 and 8 attention heads. We also work with the Transformer big setting where each ppprm-pareeeter above doubled. Training and inference details are in Appendix A.
Note that, in implementation we still separate source and target positions shown in Figure 1.

---

Under review as a conference paper at ICLR 2022
Orclec
2.83
3.65
Caua Wells cusssiee peoeepsiie
Test PPL
Test
2.60
3.36
Cuaaal le po
2.40
3.14
0.08
0.19
0.43
0.97
2.22
0.08
0.19
0.43
0.97
2.22
PParams
KIU
#Params
K10
(a)
Log-Perplexity for En-+Fr
(h) Log-Perplexity for En 7nh
40.54
29.51
36.62
Test BLEU
Test BLEU
33.08
22.58
1
Peety debe
29.89
19.75
Crea Milee
Orana leo Chasal pporycoo
27.00
17.27
0.97
0.43 PParams
2.22
Params
(c) BLEU for En-+Fr
(d) BLEU for En-7Zh
Figure 2: Fitted scaling curves (lop) and BLEU scores (bottom) for different models on WMT14 En-Fr (left) and WMT19 En-Zh (right) tasks. Top: dashed and solid fitted curves are for LM + Deep and LM + Wide, respectively. We represent the EncDec scaling with bold solid curve. Bottom: dashed curve denotes the BLEU scores of EncDec function of model parameters for reference. Markers in circles for CausalLM variants. Models are trained in Transformer base setting. Best seen in color.
Datasets and Evaluation We use WMT14 English- French (En-Fr), WMT14 English-Germat (En-De), WMT19 English-Chinese (En-Zh) and an in-house web-crawled (Web) En-De dataset for experiments, whose statisties are summarized Table We also report results on OPUS- 100 (Zhang et al., 2020), massively multilingual corpus containing 100 languages. All datasets are reepprocssed with byte pair encoding (Sennrich et al., 2016, BPE) implemented by Sentence- Piece (Kudo & Richardson, 2018). We set the BPE vocabulary size to 32K by default. We report test oog-perplexity score (PPL. for scaling study particularly and also show SacreBLEU (Post, 2018)
5 EXPERIMENTS FOR MODEL. SCALING
Kaplan et al. (2020) reported the model performance can described with a power-law, with respect to its parameters, as below:
No N
((N)
Loc,
where C(N) fits test PPL, and N denotes the number of parameters. No is a constant used for numerical stability which obtained from I-layer EncDee model. a, P, Coo are fitted parameters, and we mainly analyze the estimated scaling exponent and the irreducible loss Loc for different models.
The way of increasing model parameters varies for the same model and also across different models. We perform scaling firstly for EncDec by changing the depth (from to 26 layers, equally for its encoder and decoder) while keeping the other paper-parameters intact following Ghorbani et al. (2021). We then align the scaling settings of LM with its EncDec counterpart in term of model parameters through increasing either its depth width:
LM + Deep adds parameters via stacking more Transformer layers, which was also used in previous studies (He et al., 2018; Wang al., 2021).
Signature:
+ummrefs. | 3.4v.1.5.1
5

---

Under review as a conference paper at ICLR 2022
Estimated Scaling Expanent
Estimated inreducible Loss
1,0
2.40
7
2.35
FetaliTrchee Dre
2.30
0.8
2.25
2.20
2.15
2.10
All
Sic OrlgTgt Orig
Short
Medium
Long
All
Src Orig Tgt Orig Short
Medium
Long
Figure 3: Fitted scaling exponent (p. left) and irreducible loss (Coz, right) over different evaluation settings on WMT14 En-Fr (En-Fr). All: the whole test set; Src Orig, Tgr Orig: source-original and target-original test set, respectively; Short, Medium, Long: shortest, medium and longest <376 samples from the test set, respectively.
LMM + Wide, instead, grows the model width. We choose to enlarge the feed-forware di- mension from dt to 3d Note other strategies for width scaling possible and many, but exploring them is resource-consumingg and beyond the scope of our paper.
We distinguish data-limited regime from model size-limited regime for model scaling (Bahri et al., 2021), where the former has relatively fewer training samples than model parameters thus likely suffers from overfitting (e.g. with WMT14 En-Fr and WMT19 En-Zh), while the latter has enough samples for model fitting (e.g. with Web En-De).
5.1 SCALING IN DATA-LIMITED REGIME
Architectural difference matters most when the model is at a small scale. Figure 2 summa- rizes the scaling results on WMT14 En-Fr and WMT19 En-Zh. When there are fewer parameters, the model with inductive biases favoring translation will achieve better quality. Such inductive bias includes 1) allowing the full visibility to the source input as in PrefixLM rather than causal masking; 2) using topmost-layer source encodings for translation (TopOnly) rather than layer-wise coordinated encodings; 3) deeper LMs (Deep) rather than wider; and 4) training LMs without source-side lan- guage modeling loss (TgrOnly). The fact that LM + Deep outperforms LM + Wide shows that not only the number of parameters matters, but also the way parameters are added. This aligns with the previous findings: deeper models apply more non-linear operations and induce more abstract repre- sentations, which often helps translation (Wang al., 2019). This also applies to TopOnly. Most of these findings are consistent across different languages and evaluation metrics (Figure 2a-2d).
Different models show different sealing properties, but the gap narrows at scale. The impact of added parameters on translation performance differs across different models. The LMs perform- ing poorly at small scale often gain more from the increased capacity. For instance, the difference between LM + Deep and LM- + Wide almost disappears at the end, resonating with the optimal depth-vs.-width theory (Levine et al., 2020). We observe that PrefixL.M and EncDec converge to a similar region, followed by CausalLM TggOnlly while CausalLM still retains clear gap against the others. This performance gap on WMT19 En-Zh is smaller, mainly because of model overfitting. BLEU scores in Figure 2c and 2d also show such trend, although the relationship between BLBU and PPL is non-trivial (Ghorbani et al., 2021). These tell us that the success of architectural modifi- cations on small-scale models might not transfer to large-scal scale settings, and that comparing different models under one model configuration might result in incomplete and misleading conclusions. Note we also observe reduced gap when considering the number of layers (see Figure in the Appendix).
Do sentence length and originality of test samples affect scaling properties? Not much! We further test how the scaling changes across different evaluation settings, and show the results on WMT14 En-Fr in Figure 3. The scaling exponent changes marginally over different settings (often less than 0.05), suggesting that the scaling curves are quite similar in these settings (see Figure 8, 10, 11 in Appendix), although sentences of different originalities differ largely in style and natural-
'By default, we use PrefixIM (CausalLMM to refer to all PrefixI.M variants (CausalL.M) M variants). We adopt the italic form to denote a specific variant.

---

Under review as a conference paper at ICLR 2022
3.93
Orclec
2.83
3.64
Test PPL
Test PPL
2.60
3.36
Chaaal le p20
2.40
3.14
2.21
0.15
0.30
0.63
1.30
2.69
0.15
0.30
0.63
1.30
2.69
#FLOPS
x10
#FLOPS
X100
(a) En-+Fr
(b) En En-+7h
Figure 4: Fitted scaling curves for different models on WMT14 En-Fr and WMT19 En-Zh in term of FLOPs.
(src original)
(tgt original)
2.67
2.81
Puta
2,51
2.64
Peaf sottooonyyoo
Test PPL
Caarsi Liicee
Chasel Jecen
Clarsi Lel piocy Dewp
Chasal peec His lcall Lan
2.37
Test
2.47
Hibe Scall
2.23
2.10
0.33
0.74
1.70
3.89
8.88
0.33
0.74
1.70 &Params
3.89
8,88
#Params
KIU
K10
Figure 5: Fitted scaling curves for different models on Web En-De (En ste/tg: source/target; Web: in-domain evaluation set. Models are trained in the Transformer big setting.
ness (Graham al., 2020; Freitag al., 2020). The estimated irreducible loss shows that target- original parallel sentences are harder to model than the source-original ones, and that translating medium-length sentences is much easier. The loss ranking of different models changes little over these settings, supporting PrefixLM and EneDec generally more than CausalLM.
Do LMs deal with computational efficiency better than EncDec? No! The FLOPs results in Figure 4 show that EncDec demands generally less computation than LM, but the gap narrows scale. Note LM doesn't save computations. By contrast, to perform similarly to EneDee, LM often goes wider deeper, which even deteriorates its running (training and decoding) efficiency. Besides, EncDee allows arbitrary decoders, e.g. shallow decoders for faster inference, which is non- feasible for LMs. Figure 4 also shows that adding the source-side loss hurts CausalLM's efficiency,
5.2 SCALING IN MODEL SIZE-LIMITED REGIME
Figure 5 shows the in-domain scaling performance on Web En-De. Overall, we observe similar scaling patterns discovered above, and such pattern transfers to out-of-domain evaluation, FLOPs and BLEU scores. More results are available in the Appendix (Figure 12, 13 and 14).
6 EXPERIMENTS FOR CROSS-LINGUA, TRANSFER
Based on the literature (Wang et al., 2020; Zhang et al., 2021), sharing capacity across languages could encourage knowledge transfer but might also gain the risk of negative interference. In this section, we further compare different models but on multilingual many-to-many translation. To en- able multilingual NMT, we append a target language tag to each source sentence following Johnson et al. (2017). We perform over-sampling to balance the training data with temperature of T T =. 5.
Do LMs facilitate the transfer to low-resource languages? Not really! We start with multilin- gual translation for WMT En En-De/Fr/Zh, and regard En-De a relatively low-resource language pair. One reason behind the popularity of multilingual NMT is its transfer capability to low lowrresourc lan- guages. We analyze this transfer behavior for LMs and explore transfer (to De) from similar (Fr) and

---

Under review as a conference paper at ICLR 2022
Many-to- Many (EnDeFr)
Many (EnDeZh)
30.50
30.50
Test BLEU (Enee De)
28.50
28.50
Test BLEU
26.50
26.50
Podtate
Feof restrs--ee Feshsst-eeorn
Crank
22.50
0.35
0.68
1.01 #Params
1.35
1,68
0.35
0.68
1.01 #Params
1.35
1.68
x12
Figure 6: Cross-lingus transfer results (average BLEU scores) for different models from high-resource lan- guages to the low-resource one (En-De) under different model sizes on WMT datascts. Average is performed over En and De evaluation. Left: multilingual En-De-Fr system; Right: multilingual En En-De-Zh system. Both systems many-to-many models. Models trained in the Transformer base setting.
In-Domain Do Fr
Out-of-Domain Fr
Out-of-Domain 2h De/Fr
27.00
Test BUEU
Ckaer iee wwwpprie-eee
15.75
15.75
15.75
10.12
10:12
10.12
4.50
4.50
4.50
0.35
0.68
1.01
1.35
1.68
0.35
0.58
1.01
1.35
1.68
0.35
0.68
1.01
1.35
1.68
aParams
#Params
4Params
Out-of-Domein
3.65
3.65
3.65
3.35
3.35
3.35
Test PPL
3.05
3.05
3.05
2.75
2.75
Robacl nuddry
2.45
2.45
2.45
0.35
1.01
1.68
1.35
1.68
1.35
PParams
x20
=100
K12"
Transtation Langiage Accurity
In-Domain De Fr
Out-or-Demain Do
Out Out Domatnn Zh De/Fr
75.0
75.0
75.0
Node
4
50.0
$0.0
50.0
25.0
25.0
25.0
03.35
0.68
1.01
1.35
1.68
0.0
0.35
0.68
1,01
1.35
1.68
0.
0.35
0.68
1.01
1.35
1.68
#Params
Paramms
Params
x13
Figure Zero-shot transfer results of different models for multilingual many-to-mant modeling on four languages (En-De-Fr-Zh) under different model sizes. Top: average BLEU scores; Middle: average PPL scores; Bottom: average translation language accuracy scores. In-domain: WMT test set; Out-of-domain: in-house sport-domain test sets.
distant (Zh) languages separately. Figure 6 shows the results. PrefixLM produces comparable results to EncDec, while CausalLM lags far behind, and the incorporation of source-side objective actually hurts translation. Overall, we observe that EncDec almost dominates the transfer performance un- der different model sizes, regardless of language similarity. Similar results are also observed for low-resource high-resource transfer (see Figure 15 in the Appendix).
Do LMs benefit zero-shot transfer? PrefixLM does! We further test how LMs perform zero- shot translation. We use the newstest2019 De-Fr test set as the in-domain zero-shot eval set, and internal sports-domain N-way test set for De-Fr-Zh (2000 samples) as the out-of-domain eval
8

---

Under review as a conference paper at ICLR 2022
En-+XX
XX-+En
Zero-Shot
Model
High
Med
Low
All
High
Med
Low
All
BLEU
ACC
EncDec
25.8
32.4
31.9
29.2
31.4
34.3
35.0
33.1
4.80
24.21
PrefixLM
-0.34
-0.21
-0.82
-0.41
-0.27
-0.74
-1.59
-0.70
7.95
41.46
TopOnly
-0.01
-0.14
-1.79
-0.44
-0.07
-0.71
-1.43
-0.57
6.59
39.06
D
CausalLM
-4.51
-8.18
-12.9
-7.47
-5.18
-10.1
-13.0
-8.38
4.10
25.60
TggOnly
-0.83
-0.78
-1.40
-0.93
-1.27
-1.81
-2.43
-1.69
7.34
39.62
PrefixLM
-0.71
-0.75
-2.02
-1.01
-0.77
-0.88
-0.68
-0.78
7.44
38.60
+ 'TopOnly
-0.40
-0.37
-0.66
-0.45
-0.47
-0.50
-1.41
-0.69
6,92
37.69
W
CausalLM .4.25
-7.58
-12.2
-7.03
-5.05
-9.88
-13.3
-8.32
4.49
28.08
TgrOnly
-1.29
-1.27
-0.82
-1.18
-1.88
1.96
-2.04
-1.4
5.53
29.75
Table Translation quality of different models for En -nx,XX XX XX and zero-shot language pairs on OPUS- 100. Models are trained in the Transformer big setting. aligned with 14-layer EncDec, containing about 412M parameters (excluding embedding and softmax layers). During training, we perform oversampling with tem- perature of 5. We list average BLEU for High, Med, Low and All language groups. We also show average BLEUU and translation language accuracy (ACC) for zero-shot test sets. D) L.Ms Deep; W: LMs Wide.
set. Figure 7 shows the results. Scaling improves knowledge transfer for almost all models, while PrefixLM performs surprisingly well on zero-shot directions. In most settings, PrefiXLM surpasses EncDec significantly with respect to BLEU, and such superiority is more obvious out-of-domain evaluation and for distant language pairs.
Nevertheless, find that PrefiXLM usually underperforms EncDec in terms of PPL. In other words, EneDec still possesses the best fitting ability on zero-shot language pairs. Results on translation language accuracy explains this mismatch: compared to EncDee, PrefixLM drastically reduces off- target translation a bottleneck of zero-shot translation (Zhang et al., 2020). This also suggests that EncDec suffers from serious searching errors during inference (Stahlberg & Byrne, 2019), which the inductive biases of PrefixLM help.
In addition, observe benefits from CausalLM zero-shot translation, with without the source-side language modeling objective. This finding disagrees with that of Wang et al. (2021), which ascribe to various differences model, data and optimization. Note that Wang et al. (2021) adopted more aggressive data oversampling, didn't 't consider distant languages, proposed dedicated optimization with the source-side loss, used a different way to count model parameters, and designed different language tags for multilingual translation that could greatly affect zero-shot results (Wu et al., 2021). We leave the study of these differences the future.
How do LMs perform on massively multilingual translation? We further examine the seala- bility of LMs with respect to the number of languages, and experiment massively multilingual translation using OPUS- 100. We enlarge the BPE size to 64K to handle multilingual lexicons. Fol- lowing Zhang et al. (2020), low-resource (Low, <0.1M), and medium-resource (Med, others) groups, and report average scores for each group. Table 3 summarizes the results. EncDec outperforms LMs supervised direc- tions, with larger gap on low-resource languages and for XX-En translation. By contrast, LMs, particularly PrefixLM, perform better on zero-shot directions, with improved translation language accuracy. Overall, PrefixLM outperforms CausalLM, and also performs comparably to EncDec on supervised directions (often BLEU on average), echoing with our above findings.
divide the test language pairs into high-resource (High, >0.9M),
7 CONCLUSION AND FUTURE WORK
In this paper, we revisited language model architectures for machine translation in the perspective of model scaling and cross-lingual transfer. Extensive experiments show that LMs often have dif- ferent scaling properties where the impact of architectural differences gradually reduces as model scales up, and that LMs often deliver better zero-shot transfer than its EncDec counterpart with improved off-target translation although its cinssllingual transfer supervised directions is subop- timal. PrefixLM, the one with full visibility to the source input, shows consistent superiority to its CausalL M counterpart, and performs similarly well to EncDec across different settings especially

---

single setting Under review as a conference paper at ICLR 2022
when paired with deep modeling. These findings show that while current product offerings for major language pairs or small on-device models should continue using EncDec, LMs can be an effective architecture for giant multilingual models with zero-shot transfer primary focus.
We notice that the performance gap caused by architectural differences gradually disappears as the model size increases. This has several implications for future researches: 1) Comparing NMT mod- els in one model setting is not enough, particularly with the widely adopted 6-layer Transformer base setting, because of the scaling property difference. We believe the best practice should portray the whole scaling picture for model comparison. 2) Just like NMT models optimized for high-resoure translation transfer poorly low-resource scenarios, many models developed in the past with claims outperforming Transformer might not transfer to large-scale model settings. These studies ideally should revisited in the face of model scaling. 3) When models are compared
due to shortage of resources, best practice should state clearly which model size the conclusion is based on. 4) PrefixLM matches the performance of EncDec at large scale, and delivers promis- ing zero-shot transfer, which deserves more efforts study. This is also encouraging signal for model unification, i.c. supporting disparate tasks with a large enough yet single model (including the translation tasks), which is exciting future direction.
We also notice that LMs underperform EncDec significantly on computational efficiency. In the future, will put more efforts developing more efficient networks for LM scaling. We are also interested in exploring the applicability of translation-opimmized LMs to downstream monolingual and tross-lingual tasks.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine transla- tion. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pa- pers), pp. 3874-3887 Minneapolis, Minnesota, June 2019. Association for Computational Lin- guisties. doi: 10.18633/1//N11-13888. URL https:/ /wwww wwwweb.www....... org/ borrgg/nthooogy.. 1388.
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondiej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Nichues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alexander Waibel, and Changhan Wang. FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN. In Proceedings of the 17th International Confer- ence on Spoken Language Translation, pp. 1-34, Online, July 2020. Association for Compu- tational Linguistics. doi:
URL https: / www. www.aclwet..tccc aclweb. /
anthology/20200 iwslt-1.1.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roce Aharoni, Melvin Johnson, and Wolf- gang Macherey. The missing ingredient in zero-shot neural machine translation. CoRR, abs/1903.07091, 2019. URL http: http://axxivvv.oo.. wwwp/r///////ooeeeeeeeeeeeeeeeee abs/1993. 07091.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR), volume abs/ Abs/1449.0447/ September 2015.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. ArXiv, abs/2102.06701, 2021.
Loic Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Koemi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2020 con- ference on machine translation (WMT20). In Proceedings of the Fifth Conference on Machine Translation, pp. 1-55, Online, November 2020. Association for Computational Linguistics. URL https: /ataa://aaaaaaii http://aaaaaatttagaaaaaaa org/ 2020. wmt-0..wwtt11..... wmt-1.1.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https: httts:://proceedingg 1 1457c0d6bfcb44 96741 wb.baacllllloooooo..oooooooaoaaa 42fffa88cc12ff4a-aaaaaaa Paper pdf.
neurips...... htapps.cc/pappe wttpccppp/p////////eeeeeppp 20/0//////ee////////////// file/
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Fos- ter, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszko- reit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Proceedingy of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 76-86, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 1008. URL https: peatttss//accaaahhonn /awcaaaaanttoooooo.aaoaa .rrg 1008.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
11

---

Under review as a conference paper at ICLR 2022
the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 1111655//1/N19-143. URL https: //aclanthol ogy. org/N1 9- 42.3.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buus Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: http:////cccceeeeppeeeooo neurips. wwtppppee/l19//1i....... Paperr pdf.
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine transla- tion with a shared attention mechanism. arXiv preprint arXiv: 2016.
José A. R. Fonollosa, Noe Casas, and Marta Ruiz Costa-jussà. Joint source-target self attention with locality constraints. ArXiv, hass/1905.06596 2019.
Markus Freitag, David Grangier, and Isaac Caswell. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 61-71, Online, November 2020. Association for Computational Linguistics. doi: 00.18663//1/2020emmmp---m 2020. emnlp-main.5.
URL https: http://aaaaattttt httt:pss//aallantoooogooo org/
B. Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier García, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. ArXiv, abs/2109.07740, 2021.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan, Data and parameter scaling laws for neural machine translation. In ACL Rolling Review May 2021, 2021. StLnttts://ppereeee net/ ferr/f i 1d=IdA IKA 7MLx
Yvette Graham, Barry Haddow, and Philipp Kochn. Statistical power and translationese in machine translation evaluation. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 72-81, Online, November 2020. Association for Computa- tional Linguistics. doi: 111866//112220.mmmppppa org/2020. moanlp-maing main. 6.
URL https: /atpaa/ttaaaoggy....o///.
Tianyu He. Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tin-Yan Liu. Layer- wise coordination between encoder and decoder for neural machine translation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https::/ppoceeedeeee http:///ooceeeein.eeeee. neurips cc/ppaper/218//ille/ 4 1fb8a7a22a82:300 (at.22222222000222eeeeeeeeeeeeee Papeeeafeeccceeecooepaaa..e.rrre
Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 187-197, Edinburgh, Scotland, July 201 As- sociation for Computational Linguistics. URL ht http:///aaaaatltoooo/oo .rg.//r//// wogg///--1122211
T. J. Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. ArXiv, abs/2010.147011 2020.
Danny Hernandez, Jared Kaplan, T. J. Henighan, and Sam McCandlish. Scaling laws for transfer. ArXiv, abs/2102.01295. 2021.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. doi: 10.1 1661/a/l a 00065. URL https: /wwww/wwww. ac web. org/ wwtaal -
12

---

Under review as a conference paper at ICLR 2022
Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700-17000 1009, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: //aacla viclanthoolgge wgt.oggggg..11111111111//
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arX1v:2001.08361, 2020.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. Deep encoder, shal- low decoder: Reevaluating non-autoregressive machine translation. In International Confer- ence on Learning Representations, 2021. URL https:/ /ttsss////peeeieeeeeeee net forum?id=
Philipp Koehn. Statistical Machine Translation. Cambridge University Press, USA, 1st edition, 2010. ISBN 0521874157.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D188 2012 URL https:::pp..... : www. acl web. org httpaaatloooo//////0///
Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. Limits depth efficiencies of self-attention. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22640-22651. Curran As- sociates, Inc., 2020, URL https: nttss:ss: http://pcoceeeenn....e/ nwwrigs......................... cc Paper.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans- actions of the Association for Computational Linguistics, 8:726-745 2020.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octo- ber 2018, Association for Computational Linguistics. doi: 10.18653/11/111-63111 URL https: https///alaaathhhlaagoee
W18 / 6319.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: / ///rr...../ wwt.oo/pppppp////....
html.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P1G-111 1162. URL https: / wwww welweb.
org/ http//atthoooy/11111111
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 4594-4604. PMLR, 10-15 Jul 2018. URL https://///. /ttp::://p/ccceeeee ngs. ml Cr shazeer1 Ba. html.
Felix Stahlberg and Bill Byrne On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3356 3362, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1188///11/D9911111 URL https: http:::s//aclaantholooooooo / 10t.1/999y..rg/1--13333
Andreas Stolcke. Srilm extensible language modeling toolkit. In INTERSPEECH, 2002.
13

---

Under review as a conference paper at ICLR 2022
Iya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 1104-3112. Curran Associates, Inc., 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems 30, volume 30, pp. http://papers n nips. wwtpppppp/ppp.ettpet.ctcecte.ett / 81 Bi-attent oo--oo--- s- a you--ou-eeeee pdf.
Curran Associates, Inc., 2017. URL
Qiang Wang. Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 1810-1822, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/1186/////11991176 URL https: //aa/aaaaa/ //aclanthologg ogy org wtgg//a--//1/.oee 9-1176
Shuo Wang. Zhaopeng Tu, Zhixing Tan, Wenxuan Wang, Maosong Sun, and Yang Liu. Language models are good translators. ArXiv, abs/2106. 13627, 2021.
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual mod- els: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP), pp. 4438 -4500, Online, November 2020. Association for Computational Linguisties. doi: 10. 00.18653/1//222..comm www.:p-0emmmlmmmmmme m5in.359. URL ht.t www.///ww...cccooc.......o...o.. web. org/ aatetntoolllyooo
emn 1 1.. 359.
Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. Language tags matter for zero-shot neu- ral machine translation. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pp. 3001-3007, Online, August 2021. Association for Computational Linguis- tics. doi: 101186553/1/2222 1.2021...niings-a..l. findings-acl.264. URL https: //taaaaaaaatttoooo.aaaaaaaoealle org/2021. findings-acl. 264.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational LinguattiiooaaLLnnniiiisisii doi: 10.18653/v1/2011 wtwa:///2a21/11mmmmmmeaa .naacl-main.41 URL httpttss: //aclanthology org/
main.41.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers.
ArXiv, abs/2106 04560, 2021.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1038-1639, Online, July 2020. Association for Computational Linguistics. doi:
aac.:/.n...immi.n11.... main. URL https:// 148.
acl aclweb. org/
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations, 2021. URL / //openreview net /et//ooumm// -iriiiijjj00 Wj4DDoDuyCE.
14

---

Under review as a conference paper at ICLR 2022
A MODEL TRAINING AND INFERENCE
We update model parameters via Adafactor (Shazeer & Stern, 2018) with label smoothing of value 0.1, and scheduled learning rate of warmup steps 40K. We apply dropout of 0.1 residuals, feed- forward activations and attentions. We employ the post-norm Transformer by default; for some exceptional cases (often with deep models where training unstable) we use the pre-norm one instead. Batch size is set to about 128K tokens. We train models for up to IM steps on different tasks, except Web En-De where 500K steps is used. We average 10 checkpoints for evaluation. For bilingual experiments, these checkpoints are selected according to the dev set performance; for multilingual experiments, we use the last 10 checkpoints. Beam search used for inference, with beam size of 8 and length penalty of 0.5.
B MORE EXPERIMENTAL RESULTS
3.16
4.07
PPL
cwseeleee pdid bcal Lae
Test PPL
Teas
2.43
3.19
2.22
2.94
0.19
0.43
0.97
0.97
2,22
#Params
#Params
(a) En FF
(b) En -Zh
Figure 8: Fitted scaling curves for different models on WMT14 En-Fr and WMT19 En-Zh on the longest sentence group. We rank our test set according to source sentence length, and then split it into 8 disjoint groups This shows the results on the longest group.
3.07
3.93
3.65
Partaal Ioporyeeeg
Tet PPL
Test PPL
Freet poors-ile
2.40
3.14
2.11
2.91
#Layers
#Layers
(a) En +Fr
(b) En +Zh
Figure 9: Fitted scaling curves for different models on WMT14 En-Fr and WMT19 En-Zh with respect to the number of layers. Note under the same number of layers, LM Deep has much fewer parameters than EncDec and LM + Wide. The performance gap also narrows as model scales
C RELATIVE VS. ABSOLUTE TRANSFER RESULTS
15

---

Under review as a conference paper at ICLR 2022
Sec original
tgt original
2.98
3.16
Prefoc
Prefoe
2.75
Freti
2.91
Test 9PL
2.54
Fitedd goorleiica
2.68
Causa giori-nnoa
2.34
2.47
2.16
2.27
0.08
0.19
0.43
0.97
2,22
0.08
0.19
0.43
0.97
2.22
#Params
x20
x201
(a) En-+Fr
original
tgt original
3.93
3.89
Porta
3.65
Frefe Soginycleg
3.58
Teat PPL
On tekero
Ok tse prorie
3.38
3.30
Fiteed ca aguas
Hiti Sca hgiis
2.91
2.80
0.08
0.19
0.43 #Params
0.97
2.22 K30'
0.08
0.19
0.43 #Params
0.97
2.22
K10
(b) En
Figure 10: Fitted scaling curves for different models on WMT14 En-Fr and WMT19 En-Zh evaluated on source original and target original test sets.
40.64
41.29
37.27
38.25
Test BLEU
BLEU
Brove
35.43
fee Maic
Frefts Mevita
31.36
32.82
Frett Aet DED
Chall CA-ANCA
28.76
10
15
20
25
30
35
40
45
10
15
20
25
30
35
40
45
@Source
(a) En-+Fr
Ball
27.51
28.92
Dep Canel
Tex BLEU
Cuael to pelets
26.34
Pelu Pel baphy
21.78
23.99
19.37
21.05
10
15
20
25
35
40
45
10
15
20
25
30
35
45
@Source Tokens
@Source Tokens
En
Figure 11: BLEU scores for different models on WMT14 En-Fr and WMT19 Ea-Zh as function of source sentence length. Left: models aligned with 6-layer EncDec; Right: models aligned with 14-layer EncDec.
16

---

Under review as a conference paper at ICLR 2022
WMT (src original)
WMT (tg original)
2.85
3.04
Peaf AiNE
2.64
2.80
Praf FLN RLN
all Dore
Test PPL
Tutthey Doic Hiw Sraling
2.44
2.58
2.25
2.38
2.08
2.19
0.33
0.74
1.70
3.89
8,88
0.33
0.74
1.70
3.89
8.08
Params
K10
@ Params
Web (arc original)
Web (tot original)
2.67
Ordwec
Oritee
meto Wells
Peaf tu willd
2.51
2.64
Test PPL
Conral Jain
Con M03
2.37
Conrel (Arralory
Test
2.47
Connal Ml MrTtrtry
2.23
2.31
2.10
2.17
0.33
0.74
1.70
3.89
8.88
0.33
0.74
1.70
3.89
8,88
x10'
x1o0
Figure 12: Fitted scaling curves for different models on Web En-De (En sre/tgt: source/target; WMT: out-of-domain evaluation Web: in-domain evaluation Models trained in the Transformer big setting.
WMT (snc original)
WMT (ttt original)
2.85
3.04
2.64
pel syoler Coss to
2,80
Pell MANI
Crasal Mel Coa al tuthhyy Dooe
Test P9L
2.44
Test
2.58
2.25
2.38
2.08
2.19
0.42
0.92
2.02
4.44
9.74
0.42
0.92
2.02
4.44
9.74
#FLOPs
#FLOPs
Web (src original)
Web (tgt original)
2.67
2.81
the mataieelaa nota ruph sranil ald San
teete net aivills Hot aares tii
2.51
2.64
Test PPL
Can al M0
2.37
dll futhwy Aie
Test
al Mituthyy
2.23
2.10
0.42
0.92
2.02
4.44
9.74
0.42
0.92
2.02 #FLOPS
4.44
9.74
#FLOPS
x12
x1pfi
Figure 13: Fitted scaling curves for different models on Web En-De (En->De) in terms of FLOPs. Models are trained in the Transformer big setting.
17

---

Under review as a conference paper at ICLR 2022
WMT (src original)
WMT originall
38.68
38.23
35.58
34.87
Test BLEU
32.72
Oritec
Test
31.80
Distee
meta Wells
mett is wills
30.09
29.01
net we
Clasal Je0
Con (a
Canual Mi
0.33
0.74
1.70
3.89
8.68
0.74
1.70
3.89
8,88
K10
xzot
Web (src original)
Web (tot original)
38.66
38.60
36.73
35.94
Test BLEU
Test BLEU
34.90
33.47
Heaf ninic
retl da to
33.16
Claril Li-ce
31.17
Chasal Jec
Claial sabiiy--ea M4Tar/ww
31.51
29.02
0.33
0.74
1.70 #Params
3.89
8,88
0.33
0.74
1.70 PPorams
3.89
8,88
K10
Figure 14: BLEU scores different models on Web En-De (En++De) as a function model parameters. Models are trained in the Transformer big setting.
Many-to-Many Model (EnDaFr)
Many-to-Mary Model (EnDeZh)
24.42
Test BLEU Fe)
Tost BLEU (Ene 2h)
34.65
Enclic
20.95
Owe
32.95
Craud
19.22
Exautii
Craud Thr
31.24
0.35
1.01 #Paroms
17.48
0.35
1.68
#Porams
Figure 15: Cross-lingual transfer results (average BLEU scores) for different models from the low-resource language (En-De) to high-resource directions under different model sizes on WMT datasets. Average is per- formed over Ens+Fr/Zh. Left: multilingual En-De-Fr system; Right: multilingual En En-De-Ze De-Zh system.
18

---

Under review as a conference paper at ICLR 2022
Many-to Many Model (EnDeFr)
Many Modal (EnDeZh)
38.53
28.66
BLEU Fr)
Abso ltte BLEU (En Zh)
24.75
33.09
22.80
Da Lit Bees
oy Dum
31.27
20.84
1.35
#Params
#Paaams
Many-to-Many Model (EnDeFr)
Many-to-Many Model (EnDeZh)
-0.90
1.42
Relative BLEU (En- Fm
Rolative BLEU (En 2h)
-1.43
0.21
caa M Maayeyy
Cacarl
Conit Sytes
-1.95
-1.00
-2.21
-3.00
-3.42
0.35
0.68
1.01
1.35
1.68
0.35
0.68
1.01
1.35
1.68
#Params
x10*
#Porams
x10
Figure 16: Absolute (top) and relative (bottom) transfer results of different models for En -Fr and En -Zh under different models sizes WMT datasets. Left: multilingual En-De-Fr system; Right: multilingual En- De-Zh system. Relative score computed by comparing multilingual model and its corresponding bilingual counterpart. Overall, there no clear pattern supporting that LMs encourage knowledge transfer better than EncDec.
19


Reviews:
[{"review_id": "x-gj0d9f6o5", "reviewer": "Reviewer_aP6J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a paper on investigating the gaps of information in a relatively new area of research in machine translation. The approach, experiment setup, and evaluation methods are reasonable and provide confident insight into the questions that the authors ask. \n\nHere are a number of questions for the authors:\n\n- Your TopOnly setting is supposed to mimic how the encoder-decoder model uses the topmost-layer source encodings for translation. This setting indeed shows improvements and achieves lower perplexity on almost all language pairs. One can interpret that two fundamentally different architectures are still bound by similar features. Did you investigate why the topmost-layer is the most informative way (at least in the tried approaches) of transferring information from the source to the target generation? Following on that, looks like that the difference between the _Deep_ and _TopOnlyDeep_ variations is very small. What is your understanding on the difference of information captured by deeper _and_ multiple layers versus deeper and topmost layer? Did you compare _Deep_ and _TopOnly_ setups? This question is mostly about the PrefixLM model. \n\n-  You mention that the _TgtOnly_ setting is supposed to enrich the CasualLM model with additional information. However it\nrequires and occupies part of modeling capacity. Can you explain what you mean by occupying the modeling capacity?\n\n- You conclude in one of the sections that \"When there are fewer parameters, the model with inductive biases favoring translation will achieve better quality\" and follow it with four observations two of which I want to discuss here. It is not entirely clear to me why \"deeper LMs (_Deep_) rather than wider\" and \"training LMs without source-side language modeling loss (_TgtOnly_)\" are both considered as high inductive biases. For the former, how is deeper models have more inductive biases than wider models? For the later, wouldn't the other way around be correct?\n\n- You observe that the PrefixLM model performs surprisingly well on zero-shot directions. Do you have any intuitions or have you looked into finding _why_ that's the case?\n\n- According to your experiments, it looks like CausalLM (unidirectional) model is not performing well, or at least as good as PrefixLM or EncoderDecoder models, in almost any settings. This is intuitive because the model has access to less information at each step. I'm wondering if you _can_ think of a scenario where such a model will be more effective.\n\n- Since this is an examination work on multiple translation models and configurations, it would have been nice to provide a deeper analysis on the linguistic aspects of the experiments as well: Language pairs with different characteristics (morphologically rich languages for instance) work differently with different models. How different variations of the same model perform on different classes of languages is a significant question to investigate.\n\n\n\n", "summary_of_the_paper": "This paper provides an in-depth examination of language model based translation models and the impact of various parameters in the learning process. \n\nThey pick the \"classic\" encoder-decoder based translation model as the baseline and propose several variations of language model based translation models to compare. This includes the two main categories of unidirectional and bidirectional language modelling of the source language. On top of that, they look into various setups such as which layers to use to pass the source encodings to the target generation phase of the translation, whether to share parameters between source and target, and changing the depth (number of transformer layers) and the width (feed forward dimension).\n\nThey evaluate on in-house and publicly available datasets and investigate how different setups compare in the supervised translation task, zero-shot learning, low-resource vs. high-resource language pairs, and bilingual to multilingual translation. \n", "summary_of_the_review": "I find this paper an interesting read. The comprehensive study into different variations sheds light on effectiveness of language model based translation models in different settings. The approach, experiment setup, and evaluation methods are reasonable and provide confident insight into the questions that the authors ask. \nThis is valuable in the progress of research in developing new translation models. \nOne downside of this work is that there is no novel approach proposed in this paper, as well as a more comprehensive study into existing models and combining various setups is more useful to the research community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q-RsjLgnDtB", "reviewer": "Reviewer_vGrd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nThe authors delivered these findings via extensive experiments on different language pairs, data scales, model sizes, and translation task settings (bilingual, multilingual, zero-shot, etc.), so these findings have good reliability and generalization. Some of the findings are inspiring to the follow-up research on this topic.\n\nWeaknesses: \nSome experiments require more careful design to make findings more reliable. Such as, in Figure 2(d), the Test Bleu of EndDec, CasualLM-Deep, and Prefix-TopOnly-Deep decrease as the parameters increase in the interval of #Params 0.19-0.43E8. This phenomenon is not in line with the general trend, also the intuition. I suspect that the experimental settings could be improved. Othervise, the authors need to give a reasonable explanation.\n\nThere is another question I would like to promote a discussion. That is how to measure the transferring capabilities. The conclusions drawn from Figure 6 is that LMs do not really facilitate the transfer to low-resource languages, because LM-based models do not outperform EncDec model in all #params scales. However, we cannot see the improvement each model gain through transferring over the baseline. These relative improvements over the baseline (not the absolute scores) may be more appropriate to measure the transferring capabilities.", "summary_of_the_paper": "There are more and more work using language model to conduct translation between languages. This paper examines scaling and transferring laws when implementing translation with LM architecture through extensive experiments. Some of these findings are straightforward and intuitive but not experimentally verified before, and some are not so obvious. The latter is especially important for the community. These findings include: LMs are generally inferior to the EncDec architecture in terms of MT performance; it does not have the advantage of speed either; while on the zero-shot transferring aspect, LMs often performs better than EncDec models; different LM architectures have different scaling properties, among which architecture differences usually have a significant impact on the performance of small-scale models, but the performance gap narrows as the number of parameters increases, and so on. We can find more in the paper.\n", "summary_of_the_review": "This paper has conducted extensive experiments to examine the scaling and transferring laws of LMs for machine translation and has concluded several interesting findings which could be inspiring to the future work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZuM4hIEX36", "reviewer": "Reviewer_AF5b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. The empirical evaluations of this paper are comprehensive and solid, the results and findings are interesting. \n2. The paper is well-organized and clearly written. \n\n\n**Weaknesses**\n\n1. Despite good empirical efforts, the motivation of this paper seems somewhat unclear. Given that the encoder-decoder paradigm dominantly governs machine translation (also from the experiment part of this paper that LMs underperform EncDec most of the time.), for what reason should we need to consider a shift to a unified language model for such a seq2seq task? If the zero-shot transfer is the case, why not directly fix the off-target issue for EncDec and preserve the good of translation performance, instead of changing the paradigm? I feel like the authors didn't convey an incentive for this. \n2. The paper conducted extensive experiments to show how scaling affects LMs for MT, however, few suggestions based on the findings are given for future development of MT. \n3. The setting of CausalLM seems a bit weird that a unidirectional encoding behavior makes obviously no sense. \n\n**Questions**\n1. Section 5 seems to mainly examine p and L_{\\inf} In equation (7), whereas \\alpha remains undetermined. How was the value of \\alpha determined for diagrams in Figure 3? Get fitted from Figure 2 I guess? \n", "summary_of_the_paper": "This paper empirically investigates the scaling and transfer of Transformer language model architectures (i.e., a unified LM architecture rather than encoder-decoder models) for machine translation. The paper examines choices of architectural designs wrt data scales and model sizes and evaluates by performance (BLEU) on bilingual, multilingual and zero-shot translation, as well as the power-law (Kaplan+20) of neural language models.  Experiments and analyses are extensive and findings are interesting. \n\n", "summary_of_the_review": "This paper is a good empirical work investigating the unified LM for machine translation and shows some good findings. However, the motivation yet remains unclear, plus few insights are delivered. I feel like the paper is not bad in terms of empirical study, but feel a bit not interesting regarding the bar of ICLR. I tend to consider it as a borderline paper but are willing to change my mind if I did miss something upon the author response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y_8uTqRWR_U", "reviewer": "Reviewer_tsgN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an OK paper. The writing and presentation are clear, the experiments are relatively thorough, the related works are discussed comprehensively, the results are convincing, the claims are well supported by the results.\n\nHowever, after a careful reading of the paper, I still could hardly find enough insights from the paper. ***How could the findings benefit the community?*** The design principle of EncDec-based NMT is the sub-optimal performance/efficiency of LM-based NMT, and this paper just says: \"yes, the intuition is true\", which is not exciting at all. One insight is the zero-shot performance of multilingual NMT, however, I do not think there are many differences between 4.80 BLEU scores and 7.95 BLEU scores, where are too low to make sense.\n\nIt would be nice if the authors could discuss the applicability of the findings in the author response and the future version. And better yet, directly utilizing the findings to enhance existing NMT systems. Furthermore, the paper topic is too narrow, it would be much better to extend to other language generations tasks, like dialogue and QA (not a weakness but a suggestion). \n\n", "summary_of_the_paper": "This paper gives a quite thorough comparison between language-model-based (LM) and encoder-decoder-based (EncDec) architectures for neural machine translation (NMT). The investigated LMs are varied: different masking strategies, different origins of source features, and so on. The authors conducted experiments on representative bilingual and multilingual benchmarks, summarizing some findings based on the experimental results. ", "summary_of_the_review": "This is an OK paper but the current version is not exciting, which could hardly attract attention from the community. The authors should answer the question: how could the paper findings benefit the community?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has conducted extensive experiments to examine the scaling and transferring laws of LMs for machine translation and has concluded several interesting findings which could be inspiring to the future work.

The main concerns from reviewers are that the novelty of this paper is not enough. In addition, the experiments are not well-designed and the clarity of this paper can be further improved. We hope the reviews can help authors improve their paper.
Decision: Reject