Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation
Year: 2023

Abstract:
This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST). Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default. However, Xu et al. (2019) has revealed that PreNorm carries the risk of overfitting the training data. Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST. Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points. We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm. This study highlights the need for careful consideration of the LayerNorm setting for ZST.

heading: 1 Introduction
text: Multilingual neural machine translation (MNMT) enables translation between unseen language pairs, i.e., zero-shot translation (ZST) (Johnson et al., 2017; Firat et al., 2017). Prior studies have explored techniques such as language tags (Wu et al., 2021), residual connections (Liu et al., 2021), and novel training objectives (Al-Shedivat and Parikh, 2019; Pham et al., 2019; Arivazhagan et al., 2019; Gu et al., 2019; Zhu et al., 2020; Zhang et al., 2020; Wang et al., 2021; Yang et al., 2021) for improving ZST. They primarily used the Transformer architecture (Vaswani et al., 2017), which has two variations depending on the position of layer normalization (LayerNorm) (Ba et al., 2016), namely, PreNorm (applied at the input of layers) (Baevski and Auli, 2019) and PostNorm (applied after residual connections), as shown in Fig. 1. As previous studies showed that PreNorm can result in more stable training and faster convergence compared to
PostNorm for MNMT (Xiong et al., 2020), most ZST works (Pham et al., 2019; Wu et al., 2021; Liu et al., 2021) use PreNorm as the default setting following those MNMT studies. However, Xu et al. (2019) revealed that PreNorm carries the risk of overfitting the training data. We thus hypothesize that in a multilingual scenario, PreNorm may overfit supervised directions and have poor ZST generalizability. We systematically explore PreNorm and PostNorm’s effect on ZST to verify this.
Using the OPUS, IWSLT, and Europarl datasets and a total of 54 ZST directions, we show that PostNorm consistently outperforms PreNorm by up to 12.3 BLEU points. Following previous work, we also evaluate different language tag (Wu et al., 2021) and residual connection (Liu et al., 2021) settings, as they have been shown to impact ZST but we observe that PostNorm continues to be superior thereby lending credibility to our hypothesis.
1300
To better understand the performance differences, we introduce a novel analysis approach called layer-wise language recognition (LLR), which tracks the off-target rates for each encoder and decoder layer by training token-level classifiers to recognize the source or target language. This analysis shows that PreNorm is more sensitive to language tag settings than PostNorm, negatively impacting ZST performance. Additionally, by examining the unraveled view of PreNorm (Fig. 1) inspired by Veit et al. (2016), we reveal structural flaws in PreNorm for ZST. Our analysis demonstrates that the order of LayerNorm and selfattention/feed-forward network in PreNorm is the main factor affecting its ZST performance.
Given the prevalent use of PreNorm as the default setting in ZST baselines and frameworks such as Fairseq (Ott et al., 2019)1 and Tensor2Tensor (Vaswani et al., 2018), our study emphasizes the importance of careful consideration in the LayerNorm setting for ZST.

heading: 2 Background: LayerNorm
text: LayerNorm (Ba et al., 2016) normalizes the input x by zero-centering and scaling to have a unit standard deviation, followed by an additional trainable transformation, including a gain and bias adjustment. Specifically, it is formulated as:
LayerNorm(x) = x−E(x)√
V(x) · g + b, (1)
where g and b are trainable gain and bias. E and V indicate expectation and variance. LayerNorm is commonly used in two positions in the Transformer, as shown in Fig. 1. PostNorm, which is the originally proposed setting of the Transformer (Vaswani et al., 2017), involves applying LayerNorm after each sub-module (i.e., selfattention or feed-forward network) and residual connections. PreNorm (Baevski and Auli, 2019), on the other hand, involves applying LayerNorm directly before each sub-module and is known to stabilize Transformer training. While variants of Transformer LayerNorm like RMSNorm (Zhang and Sennrich, 2019) have been proposed, the vanilla PreNorm and PostNorm are still the most widely adopted settings in current multilingual NMT literature. Therefore, we only focus on PreNorm and PostNorm in this work.
1https://github.com/facebookresearch/fairseq/ tree/main/examples/multilingual
Datasets Languages Nzero Strain Arch.
OPUS ar, de, en,
30 12.00M base fr, nl, ru, zh
IWSLT en, it, nl, ro 6 1.38M base Europarl de, en, es, fr, nl 12 15.78M big

heading: 3.1 Experimental Settings

heading: 3 Experiments and Results
text: (2) S-ENC-T-DEC or T-ENC: Source language tag on the encoder-side and target language tag on the decoder-side; or only target language tag on the encoder-side. Wu et al. (2021) showed that this setting impacts ZST for Transformer with PreNorm. (3) w/ or w/o Res.: With the residual connection for self-attention in the middle (4th) encoder layer or not. Liu et al. (2021) revealed that “w/o Res.” improves ZST for the model trained with PreNorm. We experiment this with different LayerNorm settings as this may reduce the potential of overfitting on supervised directions, then further impacts ZST, which aligns with our hypothesis.
The settings above lead to eight different combinations, shown in Table 2 (#1 - #8). Additional training details are in Appendix A.

heading: 3.2 Main Results
text: We evaluate ZST systems using SacreBLEU (Post, 2018) and off-target rates. We report in Table 2 BLEU scores for both zero-shot and supervised directions. For ZST, we also present pivot-based translation results as a reference. Implementation details of evaluation can be found in Appendix B. Our findings are as follows: PreNorm vs. PostNorm: We find that PostNorm consistently yields better BLEU scores than PreNorm for ZST across various language tag and residual connection settings, while their performance is comparable for supervised directions. Impact of Language Tag and Residual Connection: We observe that using the “T-ENC” language tag and “w/ Res.” improves ZST performance for IWSLT, which aligns with the findings of Wu et al. (2021) and Liu et al. (2021). Nevertheless, the best performance is achieved using “w/ Res.” for Post-
Norm with “S-ENC-T-DEC” and “T-ENC” tags for OPUS and Europarl, respectively (#2 and #4). Given that Wu et al. (2021) and Liu et al. (2021) used PreNorm as the default setting (#2, #4, #6 and #8 are unreported results in their work), our results emphasize the need to consider PostNorm as the default setting for ZST, while the language tag and residual connection settings have less impact. Off-target Rates: Off-target rates help understand the different BLEU score gaps between PreNorm and PostNorm, which ranges from 0.5 to 12.3 BLEU points. For PreNorm and PostNorm with the “T-ENC” language tag (#3, #4, #7, and #8), they have similar off-target rates, with a discrepancy ranging from −0.61% to 2.02%, which results in narrow BLEU score gaps, ranging from 0.5 to 1.8 points. However, for PreNorm and PostNorm with the “S-ENC-T-DEC” language tag (#1, #2, #5, and #6), the off-target rates show a more considerable discrepancy, ranging from 5.40% to 54.23%, resulting in BLEU score gaps from 1.7 to 12.3 points. Further analysis of the nature of Transformer hidden states in the next section explores the reason for these different off-target rates in translations.

heading: 3.3 Tracking Off-targets within Transformer
text: We probe the language independence of hidden states to track off-targets within Transformer and reveal the differences between PreNorm and PostNorm. In previous work, language independence was primarily analyzed using either SVCCA (Raghu et al., 2017) or language classification accuracy (LCA) (Liu et al., 2021). However, we provide evidence in Appendix C that SVCCA, which measures the cosine similarity between hidden states, are not suitable for ZST systems. In-
stead, LCA trains a classifier to inspect the hidden states on top of the encoder, but it does not simulate the training of a ZST system, which may introduce bias in the analysis for ZST.3 In this work, we propose a novel approach for ZST based on LCA: LLR tailors classifiers for each layer to recognize the source or target language. We train a tokenlevel linear classifier for each layer to utilize hidden states in each layer as features to identify the source or target language. We use hidden states obtained by feeding sentence pairs in supervised directions to simulate the training of ZST. We then test each layer’s classifer’s ability to recognize the source or target language for supervised or zeroshot directions. This approach enables the trained classifier to best represent the language recognition ability of hidden states in a ZST system.
We train two types of linear classifiers for each encoder and decoder layer. One is for recognizing the source language, and the other is for the target language. Each linear classifier is a linear transformation from the dimension of the hidden states (512 or 1, 024) to the number of source or target languages (e.g., 7 for OPUS). We use the validation set of all supervised directions to obtain the hidden
3Liu et al. (2021) regulate the output language via a decoder-side language tag, hence analyzing only the encoder states poses no issues as the target language tag does not impact them. Nevertheless, with other language tag settings such as S-ENC-T-DEC and T-ENC, employed in this study, we require a method to obtain hidden states properly, given their impact on hidden states.
state of each token in each layer and set their source language tag or target language tag as the gold labels. Note that the decoder hidden state of each token in each layer is obtained auto-regressively without teacher-forcing. We train each classifier for 3 epochs4 with a learning rate of 1e-3 and a batch size of 64 sentences. For inference, we utilize the test sets of all supervised or zero-shot directions for computing the LLR results for corresponding directions, respectively.
The LLR results for #1 and #2 in Table 2 are presented in Fig. 2. First, we find that the encoder and decoder hidden states are highly correlated with the target and source languages, respectively, for supervised directions (L1 to L6 of Pre/Post-Tgt and L7 to L12 of Pre/Post-Src of 3 upper sub-figures), which may impact the generalizability for ZST. Second, we see that the encoder hidden states of PostNorm are less dependent on the source language than PreNorm (L6 of Pre/Post-Src of 3 lower subfigures). Third, we observe that the hidden states in all the decoder layers of PostNorm are more dependent on the target language and less on the source language than PreNorm (L7 to L12 of 3 lower subfigures). The latter two points contribute to the observed gaps in off-target rates between PreNorm and PostNorm. Conclusions for #5 and #6 with the “S-ENC-T-DEC” tag are identical (Appendix G).
4The classifier can fully converge within 3 epochs as the classifier is lightweight that only contains a small number of parameters.
For systems using “T-ENC,” we find that the LLR are similar between PreNorm and PostNorm (Appendix G) and attribute the BLEU score gaps to translation quality (i.e., adequacy and fluency).

heading: 3.4 Unraveling Structural Flaws of PreNorm
text: We investigate the structural differences between PreNorm and PostNorm to explain the observed differences in hidden states for models trained with the “S-ENC-T-DEC” tag. Inspired by Veit et al. (2016), we present an “unraveled view” for PreNorm, which decomposes the residual connections by the summation of several sub-networks, as shown in Fig. 1 (paths with different colors indicate sub-networks). However, this is not applicable to PostNorm, as LayerNorm is located after residual connections. Based on this analysis, the structural characteristic of PreNorm is: (1) Shallow Sub-network Nature: PreNorm includes shallow sub-networks, such as the embedding layer output fed through encoder layers without any operation except for the final LayerNorm (red path in Fig. 1), but PostNorm does not. (2) LayerNorm Before SA/FFN: In PreNorm, LayerNorm is placed directly before the self-attention (SA) or feed-forward module (FFN) within the residual connection module.
To analyze the impact of these structural characteristics on the generalizability of PreNorm in ZST, we swap the order of LayerNorm and SA/FFN within the residual connection module (SwapPreNorm), while keeping the shallow sub-network nature of PreNorm. Refer to Appendix D for specific illustrations of Swap-PreNorm. The results, presented in Fig 3, show that PreNorm can be significantly improved through Swap-PreNorm, with Swap-PreNorm approaching the performance of PostNorm. This demonstrates that ZST is more
sensitive to the position of LayerNorm in PreNorm than its shallow sub-network nature.

heading: 4 Conclusion
text: In this paper, we comprehensively explored the effects of LayerNorm on ZST performance. Our results demonstrate that PostNorm consistently outperforms PreNorm for ZST, regardless of the language tag and residual connection settings used. Through in-depth analysis of off-target rates and structural flaws in the PreNorm model, we were able to identify the underlying factors that contribute to the performance discrepancy. Our study suggests that care should be taken when selecting the LayerNorm setting for ZST in future research.
Limitations
According to us there are 3 limitations of our work which will be addressed in future work.
• The impact of LayerNorm, language tags, and residual connection settings on ZST was analyzed in this study. However, other factors, such as the number of layers of the Transformer model, may also have an effect and should be further investigated.
• Our conclusions were based on overall scores across all ZST directions. Further examination of how LayerNorm impacts specific language pairs is necessary.
• We explored the setting of LayerNorm for ZST systems trained from scratch. Exploration of how the LayerNorm setting of multilingual pre-trained models such as mBART (Liu et al., 2020) impacts the finetuning for ZST will be needed.
Ethical Considerations
In this study, we utilized only publicly accessible datasets for model training. Though our experiments focused on neural machine translation models, it is worth noting that these models may produce biased translations. Although this can be mitigated through a debiasing filtering process, it is beyond the scope of this work. Regarding the composition of this paper, only Grammarly5 was utilized for grammar correction, and there is no originally machine-generated text in the paper.
5https://app.grammarly.com/

heading: Acknowledgements
text: This work was supported by JSPS KAKENHI Grant Number 22KJ1843.

heading: A Training Details
text: For data preprocessing, we utilize jieba6 for Chinese segmentation and Moses7 (Koehn et al., 2007) for tokenization of other languages. After applying BPE, we obtain vocabularies with sizes of 66, 158, 40, 100, and 50, 363 for OPUS, IWSLT, and Europarl, respectively. For multilingual training, we do not apply oversampling as the data size for each language pair is comparable. The maximum sentence length is set to 256. We train Transformer models using Fairseq8 and set the dropout rate to 0.1, 0.4, and 0.3 for each dataset. Adam (Kingma and Ba, 2015) is used as the optimizer with a learning rate of 5e-4, 1e-3, and 5e-4 for each dataset, and 4, 000 warm-up steps are employed. We train the Transformer-base model using 4 32G V100 GPUs and the Transformer-big model using 8 32G V100 GPUs with the batch size of 4, 096 tokens. Additionally, we employ mixed precision training (Micikevicius et al., 2018) to accelerate the training process. We train each dataset for 200, 100, and 400 epochs, respectively.

heading: B Evaluation Details
text: For OPUS, we use the test sets following (Zhang et al., 2020), while for IWSLT and Europarl, we choose the test sets following (Wu et al., 2021). We select the checkpoint with the lowest validation loss for evaluation. The inference is performed on the trained models using a beam size of 5. For calculating SacreBLEU,9 we utilize the “zh” tokenization mode for Chinese, and the “13a” tokenization mode for other languages. We use the model of setting #410 (Table 2) for pivot-based translation. To calculate the off-target rates, we utilize the language identification tool provided by FastText (Joulin et al., 2016).11 Our experiment has revealed that this tool is slightly more accurate than another tool called “langdetect,”12 as it can achieve
6https://github.com/fxsjy/jieba 7https://github.com/moses-smt/mosesdecoder 8https://github.com/facebookresearch/fairseq 9https://github.com/mjpost/sacrebleu
10We use this setting as it achieves the best performance for supervised directions, as shown in Table 2.
11https://fasttext.cc/docs/en/ language-identification.html
12https://github.com/Mimino666/langdetect
Zero-shot Supervised
PreNorm 9.8 33.8 PostNorm 17.5 33.8 PreNorm w/o Enc-Last 11.2 33.7

heading: C Discussion about SVCCA score
text: Norm is more language-agnostic and thus has a higher ZST BLEU score in Table 3, aligning with the results found in Wu et al. (2021) and Liu et al. (2021). However, “PreNorm w/o Enc-Last” shows an extremely high SVCCA score on top of the encoder, whereas its ZST BLEU performance is significantly lower than PostNorm by 6.3 BLEU points. This reveals the significant inconsistency between the SVCCA score and the performance of ZST models. Therefore, it is crucial to carefully consider how to leverage SVCCA for ZST analysis in the future.
On the other hand, our proposed LLR score is consistent with the ZST BLEU score, as shown in Fig. 5. Specifically, we observe the lowest LLR score on top of the encoder of PostNorm for the source language and the highest LLR scores in all the decoder layers, which aligns with its best ZST performance among the three systems.

heading: D Swap-PreNorm
text: Fig. 6 illustrates the implementation of SwapPreNorm, which incorporates LayerNorm following the SA/FFN layers within the residual connection block. Compared with PostNorm, SwapPreNorm alters the order of LayerNorm and residual connections. As depicted in the unraveled view of Swap-PreNorm in Fig. 6, it preserves the shallow sub-network characteristics of PreNorm, which is the main difference compared with PostNorm.

heading: E LayerNorm without Trainable Parameters
text: Xu et al. (2019) demonstrated that the overfitting issue of PreNorm can be alleviated by removing the trainable parameters of LayerNorm (LayerNormsimple). We apply this technique to our ZST experimental settings to investigate the overfitting state of PreNorm and PostNorm. PreNorm and PostNorm after applying this technique are denoted as PreNorm-simple and PostNorm-simple. As reported in Table 4, the results indicate that PreNormsimple and PostNorm-simple outperform their respective original versions in supervised directions, which aligns with the findings of Xu et al. (2019). Additionally, we observe comparable or better BLEU scores for PreNorm-simple than PreNorm (except for #7 on Europarl), indicating that the original PreNorm had low generalizability for ZST. For PostNorm-simple, we observe significant improvement only for #4 on OPUS, which suggests the superior generalizability of the original PostNorm for ZST. Despite this improvement, PreNorm-simple still underperforms PostNorm, highlighting the severe overfitting problem of the original PreNorm.

heading: F Details of the LLR Results
text: We show the LLR results of #3 - #8 (Table 2) for ZST and supervised directions in Fig. 7.

heading: G Details of the Main Results
text: We report the specific BLEU score for each translation direction and each random seed in Ta-
bles 5, 6, 7, 8, 9, and 10.13 In addition to BLEU scores, we present model-based evaluation results obtained using BLEURT (Sellam et al., 2020)14 in Table 11. The results trend is consistent with those obtained from BLEU scores.
13Refer to details of setting random seeds in PyTorch at https://pytorch.org/docs/stable/notes/randomness. html.
14https://github.com/google-research/bleurt
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations"
3 A2. Did you discuss any potential risks of your work? Section "Ethical Considerations"
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
3 A4. Have you used AI writing assistants when working on this paper? Only Grammarly was utilized for grammar correction, and there is no originally machine-generated text in the paper.
B 3 Did you use or create scientific artifacts? Section 2.1 and 2.2 Appendix A and B
3 B1. Did you cite the creators of artifacts you used? Section 2.1 and 2.2 Appendix A and B
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A and B
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Following previous work of neural machine translation, we directly used OPUS, IWSLT, and Europarl datasets without further checking filtering in order to conduct fair comparisons.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2.1 and Appendix A and B
C 3 Did you run computational experiments? Section 2.1, 2.2, 2.3, and 2.4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 2.1 and Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 2.1 Appendix A and B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 2.2, 2.3, and 2.4 Appendix C, E, F, and G
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2.1 Appendix A and B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, it is crucial to carefully consider how to leverage SVCCA for ZST analysis in the future. On the other hand, our proposed LLR score is consistent with the ZST BLEU score, as shown in Fig. 5. Specifically, we observe the lowest LLR score on top of the encoder of PostNorm for the source language and the highest LLR scores in all the decoder layers, which aligns with its best ZST performance among the three systems