Examining Scaling and Transfer of Language Model Architectures for Machine Translation

1 INTRODUCTION . The popularity of large , general-purpose text generation models has skyrocketed in recent years due to their outstanding performance across a wide range of natural language processing tasks ( Brown et al. , 2020 ; Raffel et al. , 2020 ; Liu et al. , 2020 ; Xue et al. , 2021 ) . These studies , under large-scale pretraining and also model scaling , show the promise of moving to unified neural architectures and that dropping various task-specific inductive biases improves model generalization and/or performance ( Devlin et al. , 2019 ; Kaplan et al. , 2020 ) . In neural machine translation ( NMT ) , one essential task-specific inductive bias is to separately handle source sentence understanding and target sentence generation with the encoder-decoder paradigm ( EncDec ) . Although such bias significantly benefits translation ( Vaswani et al. , 2017 ; Chen et al. , 2018 ; Aharoni et al. , 2019 ; Barrault et al. , 2020 ; Ansari et al. , 2020 ) , some recent work shows insights challenging it , for example , aggressively simplifying the decoder yields little to no compromise on translation quality ( Kasai et al. , 2021 ) . This thereby inspires the question how the removal of this separation , i.e . using a single unified module for both encoding and decoding ( LMs ) , works for translation , and whether we can get any benefits out of that . Although some studies reported promising translation quality with LMs ( He et al. , 2018 ; Wang et al. , 2021 ) , they compare models under merely one configuration ( model size ) , neglecting that neural models follow scaling laws ( Kaplan et al. , 2020 ; Ghorbani et al. , 2021 ; Gordon et al. , 2021 ) where the impact of each added parameter on model performance might vary across different models . How the inductive biases of LMs and EncDec impact the model ’ s performance as we increase their size and the amounts of training data are intriguing yet missing in the literature . In this paper , we explore various configurations of LM architectures for translation as in Figure 1 , and compare them with EncDec on model scaling and cross-lingual transfer . We conduct a systematic study under a variety of data conditions , tasks ( bilingual , multilingual and zero-shot ) and model capacities . We examine architectural design choices associated with LMs , including causal masking ( CausalLM ) vs. full-visible masking ( PrefixLM ) for source sequences,1 layer-wise coordination ( He et al. , 2018 ) vs. final-layer source encodings ( TopOnly ) for target sequence generation , increasing LM depth vs. width , and also the effect of adding source language modeling loss for CausalLM . We evaluate how these architectural priors affect translation quality as we increase the number of parameters available to the model , under a diverse set of bilingual translation settings and also ( massively ) multilingual settings , with a special focus on transfer to low-resource languages and zero-shot directions . Our main findings are summarized below : • LMs show different scaling properties compared to EncDec . The architectural differences become less important as models scale , measured by reduced quality gap against EncDec , regardless of the language similarities , training data conditions and evaluation settings . • PrefixLM variants often outperform their CausalLM counterparts ; increasing LM depth benefits the translation task more than increasing the width ; and adding a source-side language modeling objective to CausalLM doesn ’ t affect translation quality . • Cross-lingual transfer also benefits from model scaling , where EncDec almost always dominates the quality Pareto frontier on supervised directions while zero-shot translation favors PrefixLM LMs . PrefixLM LMs significantly reduce off-target translations . • Although LMs could reach or even surpass the translation quality of EncDec , they still lag far behind EncDec with respect to computational efficiency as measured in FLOPs . 2 RELATED WORK . Using language models in the task of translation has a long history , particularly in the era of statistical machine translation ( SMT ) where LM was used as a separate yet crucial component ensuring the fluency of generation ( Stolcke , 2002 ; Heafield , 2011 ; Koehn , 2010 ) . With neural networks , NMT unified those isolated SMT components including LM under the encoder-decoder formulation ( Kalchbrenner & Blunsom , 2013 ; Cho et al. , 2014 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , which makes use of separate modules to process input and output . Further studies exploring architectural modifications by using LM alone as a translation model , nevertheless , got much less attention . He et al . ( 2018 ) proposed layer-wise coordination between encoder and decoder with tied weights , where each decoder layer attends to its corresponding encoder layer at the same depth as opposed to the conventional method of attending the top-most encoder representations . Later , 1Also known as unidirectional vs bidirectional language modelling , where in the unidirectional case a token representation takes into account only the preceding tokens and their representations , but the bidirectional case takes into account both preceding and following tokens in a sequence . Fonollosa et al . ( 2019 ) extended it with locality constraint . Dong et al . ( 2019 ) explored LMs for sequence generation under large-scale pretraining . Despite reporting promising results , these prior studies either focus only on bilingual tasks or do not consider the scaling properties of the models , leaving the picture incomplete : how the findings will change as we scale the models and how the languages benefit from/interfere each other as the architectural priors ( inductive biases ) change . Neural models follow some scaling laws . Kaplan et al . ( 2020 ) reported the test cross-entropy loss of LMs can be formulated as a power-law scaling function of either model size ( excluding embedding parameters ) or dataset size . Later on , researchers examined and confirmed such findings across different domains , including vision modeling ( Zhai et al. , 2021 ) , knowledge transfer from pretraining ( Hernandez et al. , 2021 ) , autoregressive generative modeling ( Henighan et al. , 2020 ) , and neural machine translation ( Gordon et al. , 2021 ; Ghorbani et al. , 2021 ) , to name a few . We find it essential to study the scaling behavior of new architectures and approaches given the recent evidence on the emergent properties of the models at scale ( Brown et al. , 2020 ) . Another critical component in machine translation is the number of languages being considered with the models , which is the very focus of multilingual NMT ( Firat et al. , 2016 ) . Cross-lingual transfer in multilingual NMT often results from parameter sharing across languages , which benefits low-resource languages and also enables zero-shot translation ( Johnson et al. , 2017 ) , although the quality on zero-shot directions is largely hindered by the off-target translation problem ( Arivazhagan et al. , 2019 ; Zhang et al. , 2020 ) . The structure of LMs further encourages parameter sharing , offering a chance to improve the transfer while magnifying the problem of interference ( negative-transfer ) ( Wang et al. , 2020 ; Zhang et al. , 2021 ) . Very recently , Wang et al . ( 2021 ) analyzed the cross-lingual transfer behavior of CausalLM , and reported encouraging zero-shot performance . However , we did not observe the same results likely because of data sampling , model architecture and optimization differences which zero-shot transfer is sensitive to . 3 LANGUAGE MODEL ARCHITECTURES FOR TRANSLATION . In this section , we briefly review EncDec and then present LM architectures for translation based on Transformer ( Vaswani et al. , 2017 ) . Table 1 compares different models . Given a source sequence X of length |X| and its target translation Y of length |Y | , EncDec performs translation via the following structure : Xl = FFN ◦ SAtt ( Xl−1 ) , Yl = FFN ◦ CAtt ◦ SAtt ( Yl−1 , XL ) , ( 1 ) where l denotes the layer index and ◦ indicates consecutive sublayers . Xl ∈ R|X|×d and Yl ∈ R|Y |×d are the layer representations of the source and target sequence respectively , with a model dimension of d. The first input layer ( X0 , Y0 ) is the summation of token embeddings and their positional encodings . We drop all the layer normalization and residual connections in our formulations for brevity . The encoder is a stack of L layers , each of which includes a multi-head self-attention sublayer ( SAtt ) followed by a feed-forward sublayer ( FFN ) . SAtt in the encoder is bidirectional with fullvisible masking that has full visibility to all source tokens , preceding and following . Its final-layer representations XL are fed to the decoder , which shares a similar structure to the encoder but with an additional ( multi-head ) cross-attention sublayer ( CAtt ) . Unlike encoder , SAtt in the decoder is unidirectional with causal masking , where attention to following tokens is disabled ( masked ) . CAtt can always access all source inputs , though . Note we set the encoder and decoder depth equally to L , and use dff to denote the intermediate dimension of FFN . EncDec is often optimized with the target translation objective based on YL : LEncDec ( X , Y ) = LTGT = − logP ( Y |X , YL ) . ( 2 ) Rather than separately modeling source and target sequences , LM handles both with a single module : [ Xl , Yl ] = FFN ◦ SAtt ( [ Xl−1 , Yl−1 ] , M ) , ( 3 ) where M ∈ { 0 , 1 } ( |X|+|Y | ) × ( |X|+|Y | ) is the attention mask that controls the information flow within the concatenated sequences ( [ · , · ] ) .2 Two LM variants explored by changing the structure of mask M , PrefixLM and CausalLM . PrefixLM merges different modules of EncDec , trained with LTGT . Its attention mask MPrefixLM ( i , j ) = 1 , if i ≥ j or j ≤ |X| ; otherwise 0 , ( 4 ) combines the encoder/decoder self-attention mask and the cross-attention mask of EncDec . CausalLM , by contrast , is a strict LM that applies causal masking to both sequences : MCausalLM ( i , j ) = 1 , if i ≥ j ; otherwise 0 . ( 5 ) Apart from LTGT , CausalLM also includes the source-side language modeling loss for training : LCausalLM ( X , Y ) = LSRC + LTGT = − logP ( X|XL ) − logP ( Y |X , YL ) . ( 6 ) To improve our understanding of LMs for translation , we further incorporate two extensions : PrefixLM + TopOnly The model defined in Equation 3 performs attention over the source and target sequence within the same layer . In contrast , EncDec always uses the topmost-layer source encodings for translation . We mimic this with TopOnly extension by feeding toplayer encodings , i.e . XL instead of Xl−1 , to each attention sublayer . It operates the same as EncDec but with the parameters of encoder and decoder tied . CausalLM + TgtOnly The inclusion of the source-side objective enriches CausalLM ’ s learning signal and encourages the model to absorb source language characteristics . However , it requires and occupies part of modeling capacity , which might negatively affect translation . To offset this impact , we add the TgtOnly extension that optimizes CausalLM with the target translation objective LTGTC alone , which also aligns better with EncDec and PrefixLM .

Summary: This paper empirically investigates the scaling and transfer of Transformer language model architectures (i.e., a unified LM architecture rather than encoder-decoder models) for machine translation. The paper examines choices of architectural designs wrt data scales and model sizes and evaluates by performance (BLEU) on bilingual, multilingual and zero-shot translation, as well as the power-law (Kaplan+20) of neural language models.  Experiments and analyses are extensive and findings are interesting.