Examining Scaling and Transfer of Language Model Architectures for Machine Translation

1 INTRODUCTION . The popularity of large , general-purpose text generation models has skyrocketed in recent years due to their outstanding performance across a wide range of natural language processing tasks ( Brown et al. , 2020 ; Raffel et al. , 2020 ; Liu et al. , 2020 ; Xue et al. , 2021 ) . These studies , under large-scale pretraining and also model scaling , show the promise of moving to unified neural architectures and that dropping various task-specific inductive biases improves model generalization and/or performance ( Devlin et al. , 2019 ; Kaplan et al. , 2020 ) . In neural machine translation ( NMT ) , one essential task-specific inductive bias is to separately handle source sentence understanding and target sentence generation with the encoder-decoder paradigm ( EncDec ) . Although such bias significantly benefits translation ( Vaswani et al. , 2017 ; Chen et al. , 2018 ; Aharoni et al. , 2019 ; Barrault et al. , 2020 ; Ansari et al. , 2020 ) , some recent work shows insights challenging it , for example , aggressively simplifying the decoder yields little to no compromise on translation quality ( Kasai et al. , 2021 ) . This thereby inspires the question how the removal of this separation , i.e . using a single unified module for both encoding and decoding ( LMs ) , works for translation , and whether we can get any benefits out of that . Although some studies reported promising translation quality with LMs ( He et al. , 2018 ; Wang et al. , 2021 ) , they compare models under merely one configuration ( model size ) , neglecting that neural models follow scaling laws ( Kaplan et al. , 2020 ; Ghorbani et al. , 2021 ; Gordon et al. , 2021 ) where the impact of each added parameter on model performance might vary across different models . How the inductive biases of LMs and EncDec impact the model ’ s performance as we increase their size and the amounts of training data are intriguing yet missing in the literature . In this paper , we explore various configurations of LM architectures for translation as in Figure 1 , and compare them with EncDec on model scaling and cross-lingual transfer . We conduct a systematic study under a variety of data conditions , tasks ( bilingual , multilingual and zero-shot ) and model capacities . We examine architectural design choices associated with LMs , including causal masking ( CausalLM ) vs. full-visible masking ( PrefixLM ) for source sequences,1 layer-wise coordination ( He et al. , 2018 ) vs. final-layer source encodings ( TopOnly ) for target sequence generation , increasing LM depth vs. width , and also the effect of adding source language modeling loss for CausalLM . We evaluate how these architectural priors affect translation quality as we increase the number of parameters available to the model , under a diverse set of bilingual translation settings and also ( massively ) multilingual settings , with a special focus on transfer to low-resource languages and zero-shot directions . Our main findings are summarized below : • LMs show different scaling properties compared to EncDec . The architectural differences become less important as models scale , measured by reduced quality gap against EncDec , regardless of the language similarities , training data conditions and evaluation settings . • PrefixLM variants often outperform their CausalLM counterparts ; increasing LM depth benefits the translation task more than increasing the width ; and adding a source-side language modeling objective to CausalLM doesn ’ t affect translation quality . • Cross-lingual transfer also benefits from model scaling , where EncDec almost always dominates the quality Pareto frontier on supervised directions while zero-shot translation favors PrefixLM LMs . PrefixLM LMs significantly reduce off-target translations . • Although LMs could reach or even surpass the translation quality of EncDec , they still lag far behind EncDec with respect to computational efficiency as measured in FLOPs . 2 RELATED WORK . Using language models in the task of translation has a long history , particularly in the era of statistical machine translation ( SMT ) where LM was used as a separate yet crucial component ensuring the fluency of generation ( Stolcke , 2002 ; Heafield , 2011 ; Koehn , 2010 ) . With neural networks , NMT unified those isolated SMT components including LM under the encoder-decoder formulation ( Kalchbrenner & Blunsom , 2013 ; Cho et al. , 2014 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , which makes use of separate modules to process input and output . Further studies exploring architectural modifications by using LM alone as a translation model , nevertheless , got much less attention . He et al . ( 2018 ) proposed layer-wise coordination between encoder and decoder with tied weights , where each decoder layer attends to its corresponding encoder layer at the same depth as opposed to the conventional method of attending the top-most encoder representations . Later , 1Also known as unidirectional vs bidirectional language modelling , where in the unidirectional case a token representation takes into account only the preceding tokens and their representations , but the bidirectional case takes into account both preceding and following tokens in a sequence . Fonollosa et al . ( 2019 ) extended it with locality constraint . Dong et al . ( 2019 ) explored LMs for sequence generation under large-scale pretraining . Despite reporting promising results , these prior studies either focus only on bilingual tasks or do not consider the scaling properties of the models , leaving the picture incomplete : how the findings will change as we scale the models and how the languages benefit from/interfere each other as the architectural priors ( inductive biases ) change . Neural models follow some scaling laws . Kaplan et al . ( 2020 ) reported the test cross-entropy loss of LMs can be formulated as a power-law scaling function of either model size ( excluding embedding parameters ) or dataset size . Later on , researchers examined and confirmed such findings across different domains , including vision modeling ( Zhai et al. , 2021 ) , knowledge transfer from pretraining ( Hernandez et al. , 2021 ) , autoregressive generative modeling ( Henighan et al. , 2020 ) , and neural machine translation ( Gordon et al. , 2021 ; Ghorbani et al. , 2021 ) , to name a few . We find it essential to study the scaling behavior of new architectures and approaches given the recent evidence on the emergent properties of the models at scale ( Brown et al. , 2020 ) . Another critical component in machine translation is the number of languages being considered with the models , which is the very focus of multilingual NMT ( Firat et al. , 2016 ) . Cross-lingual transfer in multilingual NMT often results from parameter sharing across languages , which benefits low-resource languages and also enables zero-shot translation ( Johnson et al. , 2017 ) , although the quality on zero-shot directions is largely hindered by the off-target translation problem ( Arivazhagan et al. , 2019 ; Zhang et al. , 2020 ) . The structure of LMs further encourages parameter sharing , offering a chance to improve the transfer while magnifying the problem of interference ( negative-transfer ) ( Wang et al. , 2020 ; Zhang et al. , 2021 ) . Very recently , Wang et al . ( 2021 ) analyzed the cross-lingual transfer behavior of CausalLM , and reported encouraging zero-shot performance . However , we did not observe the same results likely because of data sampling , model architecture and optimization differences which zero-shot transfer is sensitive to . 3 LANGUAGE MODEL ARCHITECTURES FOR TRANSLATION . In this section , we briefly review EncDec and then present LM architectures for translation based on Transformer ( Vaswani et al. , 2017 ) . Table 1 compares different models . Given a source sequence X of length |X| and its target translation Y of length |Y | , EncDec performs translation via the following structure : Xl = FFN ◦ SAtt ( Xl−1 ) , Yl = FFN ◦ CAtt ◦ SAtt ( Yl−1 , XL ) , ( 1 ) where l denotes the layer index and ◦ indicates consecutive sublayers . Xl ∈ R|X|×d and Yl ∈ R|Y |×d are the layer representations of the source and target sequence respectively , with a model dimension of d. The first input layer ( X0 , Y0 ) is the summation of token embeddings and their positional encodings . We drop all the layer normalization and residual connections in our formulations for brevity . The encoder is a stack of L layers , each of which includes a multi-head self-attention sublayer ( SAtt ) followed by a feed-forward sublayer ( FFN ) . SAtt in the encoder is bidirectional with fullvisible masking that has full visibility to all source tokens , preceding and following . Its final-layer representations XL are fed to the decoder , which shares a similar structure to the encoder but with an additional ( multi-head ) cross-attention sublayer ( CAtt ) . Unlike encoder , SAtt in the decoder is unidirectional with causal masking , where attention to following tokens is disabled ( masked ) . CAtt can always access all source inputs , though . Note we set the encoder and decoder depth equally to L , and use dff to denote the intermediate dimension of FFN . EncDec is often optimized with the target translation objective based on YL : LEncDec ( X , Y ) = LTGT = − logP ( Y |X , YL ) . ( 2 ) Rather than separately modeling source and target sequences , LM handles both with a single module : [ Xl , Yl ] = FFN ◦ SAtt ( [ Xl−1 , Yl−1 ] , M ) , ( 3 ) where M ∈ { 0 , 1 } ( |X|+|Y | ) × ( |X|+|Y | ) is the attention mask that controls the information flow within the concatenated sequences ( [ · , · ] ) .2 Two LM variants explored by changing the structure of mask M , PrefixLM and CausalLM . PrefixLM merges different modules of EncDec , trained with LTGT . Its attention mask MPrefixLM ( i , j ) = 1 , if i ≥ j or j ≤ |X| ; otherwise 0 , ( 4 ) combines the encoder/decoder self-attention mask and the cross-attention mask of EncDec . CausalLM , by contrast , is a strict LM that applies causal masking to both sequences : MCausalLM ( i , j ) = 1 , if i ≥ j ; otherwise 0 . ( 5 ) Apart from LTGT , CausalLM also includes the source-side language modeling loss for training : LCausalLM ( X , Y ) = LSRC + LTGT = − logP ( X|XL ) − logP ( Y |X , YL ) . ( 6 ) To improve our understanding of LMs for translation , we further incorporate two extensions : PrefixLM + TopOnly The model defined in Equation 3 performs attention over the source and target sequence within the same layer . In contrast , EncDec always uses the topmost-layer source encodings for translation . We mimic this with TopOnly extension by feeding toplayer encodings , i.e . XL instead of Xl−1 , to each attention sublayer . It operates the same as EncDec but with the parameters of encoder and decoder tied . CausalLM + TgtOnly The inclusion of the source-side objective enriches CausalLM ’ s learning signal and encourages the model to absorb source language characteristics . However , it requires and occupies part of modeling capacity , which might negatively affect translation . To offset this impact , we add the TgtOnly extension that optimizes CausalLM with the target translation objective LTGTC alone , which also aligns better with EncDec and PrefixLM .

Summary: This paper provides an in-depth examination of language model based translation models and the impact of various parameters in the learning process.   They pick the "classic" encoder-decoder based translation model as the baseline and propose several variations of language model based translation models to compare. This includes the two main categories of unidirectional and bidirectional language modelling of the source language. On top of that, they look into various setups such as which layers to use to pass the source encodings to the target generation phase of the translation, whether to share parameters between source and target, and changing the depth (number of transformer layers) and the width (feed forward dimension).  They evaluate on in-house and publicly available datasets and investigate how different setups compare in the supervised translation task, zero-shot learning, low-resource vs. high-resource language pairs, and bilingual to multilingual translation.