Learning Language-Specific Layers for Multilingual Machine Translation
Year: 2023

Abstract:
Multilingual Machine Translation promises to improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (e.g., avoiding losing gender and formality information when translating through English). On the downside, adding more languages reduces model capacity per language, which is usually countered by increasing the overall model size, making training harder and inference slower. In this work, we introduce Language-Specific Transformer Layers (LSLs), which allow us to increase model capacity, while keeping the amount of computation and the number of parameters used in the forward pass constant. The key idea is to have some layers of the encoder be source or target language-specific, while keeping the remaining layers shared. We study the best way to place these layers using a neural architecture search inspired approach, and achieve an improvement of 1.3 CHRF (1.5 SPBLEU) points over not using LSLs on a separate decoder architecture, and 1.9 CHRF (2.2 SPBLEU) on a shared decoder one.

heading: 1 Introduction
text: Multilingual Neural Machine Translation (MNMT) has received much attention from the Machine Translation community in recent years (Johnson et al., 2017; Aharoni et al., 2019; Freitag and Firat, 2020; Zhang et al., 2020; Fan et al., 2021; Yang et al., 2021; Tran et al., 2021). This interest is based on the many advantages it provides:
Scalability Instead of having one model per language pair, a single model suffices, significantly reducing maintenance efforts as well as the combined model size across all languages.
Inference Speed and Less Error Cascading Due to the availability of data, most production sys-
tems are English-centric, meaning translation between two non-English languages naïvely involves translating twice (i.e. pivoting), once to English, and once from English. This approach increases latency and contributes to error cascades, since the translation to or from English usually implies information loss, e.g. missing gender or formality distinctions that do not exist similarly in English.
Low-Resource Improvements Having a single model capable of handling multiple languages, means it can generalize across language boundaries and utilize characteristics of closely related translation directions to improve the translation quality for low-resource language-pairs (i.e. knowledge transfer). Although achieving good zero-shot translation quality remains a challenging task, MNMT has been shown to help (Johnson et al., 2017).
Despite the above advantages, training high quality multilingual models is a challenging task: as more languages are added, the more they compete for the model’s parameters (Sachan and Neubig, 2018). A common solution is to increase the model size, but blindly doing so comes with its own troubles, as training becomes harder, inference slower, and the storage requirements increase, which makes them challenging to deploy to portable devices.
In this work, our goal is to increase the model capacity per language pair, while at the same time, letting the model share knowledge between languages, and without increasing the inference cost. To this end, and combined with the observation from Kudugunta et al. (2019) that the translation process in Transformer models starts in the top encoder layers, we propose an architecture with shared and language-specific weights. Figure 2 shows one such architecture, where layers 3 and 41 are source language-specific, layers 13, 14, and 15 are target language-specific, the remaining layers are shared across all languages, and the decoder is
1Throughout the paper, we use layer indices starting at 1.
14767
also target language-specific. For the non-shared layers, we propose using Language-Specific Transformer Layers (LSLs), illustrated in Figure 1b. Quite simply, LSLs are a combination (i.e., a dictionary) of regular Transformer layers (Figure 1a), where the sub-layer used depends on the chosen language. We consider two cases: source-indexed LSLs, and target-indexed LSLs, distinguished by whether we use the source or the target language to select the appropriate sub-layer.
The main contributions of this work are:
1. We propose a way to increase the model capacity per language, without changing the inference speed.
2. We show that the model benefits from having both language-specific and shared com-
ponents, as well as from having source and target language-specific components.
3. We propose a technique to aid in learning the best architecture, rather than relying purely on manual trial-and-error.

heading: 2 Related Work
text: There exists a vast literature investigating parameter sharing mechanisms for MNMT. Particularly relevant is the shared-encoder, separate-decoder architecture proposed by Dong et al. (2015) which we use as the base for some of our experiments.
Several works analyze which weights should be shared between languages (Sachan and Neubig, 2018; Blackwood et al., 2018; Platanios et al., 2018; Zhu et al., 2020; Wang et al., 2019, 2018). Regardless, most closely related to the presented work are the studies by Zhang et al. (2021) and Purason and Tättar (2022). Zhang et al. (2021) propose adding Conditional Language-Specific Routing (CLSR) layers inside the encoder and decoder Transformer layers. They learn to mix between language-specific and shared weights, and do this on a word by word basis. Our approach does not use learned routing but uses the same components for the whole sentence per language-pair, instead of computing a mixed representation. We also do not add extra parameters to the layer, meaning we have the same inference time complexity as regular Transformer layers. The approach in Purason and Tättar (2022) is similar to ours in the sense that they use language-specific Transformer layers on the encoder side, and also look into sharing weights on a language-family basis. In contrast to our approach,
they focus on source-indexed language-specific layers, while we investigate selecting the layers based on the source or the target language. Besides, we propose a systematic method for deciding which layers to share, and which to be language specific.
Connection to Adapter Layers Adapter Layers (Houlsby et al., 2019; Bapna and Firat, 2019; He et al., 2022) are a lightweight technique to finetune a pre-trained encoder model by injecting taskspecific sub-modules into the existing architecture. In contrast, LSLs are designed to be trained from scratch, and replace shared by language-specific components, rather than adding new ones, keeping the overall computational costs constant.
Connection to Mixture-of-Experts LSLs enable the introduction of source- and target-specific parameters in the encoder and increase the model capacity, while at the same time keeping the inference cost and effective parameter count for the forward-pass constant (see Figure 1). As such, they are similar in nature to sparsely activated mixture-of-experts layers (MOEs, Shazeer et al., 2017; Roller et al., 2021; Lepikhin et al., 2021) but with the important differences that 1) there is no need for learning a balanced routing module; 2) sub-layer utilization is enforced by design, which tends to be a problem for MOE layers (Dua et al., 2022); and 3) sentences are always routed to the same conditional compute based on the indexinglanguage, enabling smaller binaries for on-device downloading of model weights as well as consecutive downloads to extend the on-device capabilities to new languages. In fact, Kudugunta et al. (2021) have shown that the final encoder MOE layers also learn target language-specific utilization where a subset of experts is used when translating e.g. X→EN. However, since it is commonly not strictly enforced, downloading all experts is required, increasing the download size for end users.

heading: 3 Methods
text: In this section we describe our proposed LanguageSpecific Transformer Layer, as well as a way to select whether to use shared or language-specific weights for each layer.

heading: 3.1 Language-Specific Transformer Layer
text: The idea of LSLs is simple: instead of sharing the same parameters across all languages, have the weights for the layer be language-specific as illus-
trated in Figure 1. LSLs are composed of one “regular” Transformer encoder layer per language. The input is routed to the appropriate sub-layer depending on the source or target language, and at any time only one of the sub-layers is used. Simply replacing all layers in the Transformer with LSLs would significantly increase the number of parameters, and reduce the sharing between languages. For example, if all LSLs are indexed by the source (or target) language it would be identical to a “separate encoder separate decoder” architecture. Instead, we propose a mix of LSLs and regular Transformer layers, which allows the model to learn languagespecific and shared weights. See Figure 2 for one such architecture. A sample implementation for FAIRSEQ (Ott et al., 2019) is given in Appendix A.

heading: 3.2 Learning the Architecture
text: Intuitively, we expect the bottom layers of the encoder to require more source language knowledge, while the top ones should already capture target language information as found by Kudugunta et al. (2019). This observation motivates using sourceindexed LSLs in the bottom encoder layers, targetindexed LSLs in the top ones, and keeping the remaining layers shared as illustrated in Figure 2. This type of reasoning quickly gets out of hand, as the number of possible architectures is exponential in the numbers of layers. To avoid having to manually select which layers should be shared, and which should be source- or target-indexed LSLs, we propose a Neural Architecture Search (Elsken et al., 2019) inspired approach.
For each layer in the encoder, we learn a shared layer as well as one LSL, which can be source- and target-indexed, and 3 scalar mixing weights:
hi = w shared i · layersharedi (hi−1) +
wsrci · LSLi(hi−1, src) + (1) wtgti · LSLi(hi−1, tgt) ,
where hi−1 and hi are the outputs of layers i− 1 and i, respectively, and wsharedi +w src i +w tgt i = 1. LSLi(hi−1, src) means we select the LSL weights by the source language, while LSLi(hi−1, tgt) corresponds to using the target weights.
As there is no constraint on the mixing weights, other than that they sum to 1 and are non-negative2, the model is incentivized to use all the sub-layers,
2We implement this constraint by applying the softmax function to the 3 scalar parameters.
resulting in a huge increase in the number of parameters. If we have L different languages, then each layer will have as many parameters as L+ 1 “regular” Transformer layers.3 The amount of computation increases by a factor of 3, as we compute three intermediate representations: a shared one, one using the source language sub-layer, and another using the target language sub-layer, which we then mix according to Equation (1).
To keep the inference time unaffected and the model size reasonable, only one of the components should be used, i.e., the mixing weights should be sparse. In this work, we propose a simple but effective approach: for each layer, we select the component with the largest converged weight. For example, if the largest weight for layer i is wtgti , then layer i will be a target-indexed LSL. After selecting the architecture, we train it from scratch.

heading: 3.3 Dense Pre-training
text: Inspired by Dua et al. (2022), we found that initializing all encoder weights (both shared and LSLs) from a pre-trained architecture consisting only of “regular” Transformer layers helped achieve better performance. In our experiments, we copy the pretrained weights from the respective layers to the language-specific modules for initialization. The pre-trained weights come from our baseline architectures, shared-encoder models with only “regular” Transformer Layers. We use the separate decoder baseline’s weights for the separate decoder models (e.g., LSL-NAS), and the shared decoder baseline’s weights for the shared decoder models (e.g., LSL-NAS-SD). This procedure has multiple advantages: 1) It maximizes cross-lingual transfer by training a general representation across languages first and minimizes language interference during fine-tuning; 2) It mitigates under-trained languagespecific components for low-resource languages as they usually see significantly less data and the naïve approach of training with higher sampling temperatures typically degrades performance on high resource languages (Arivazhagan et al., 2019; Wang et al., 2020); and 3) it improves convergence speed for architectures with LSLs.

heading: 4 Results
text: In the following, we will describe our experiments and discussion regarding the effectiveness of LSLs.
3Plus, of course, the mixing weights, but they amount to only 3 extra parameters per layer.

heading: 4.1 Experimental Setup
text: Data In our experiments, we focus on the following 10 languages: German (DE), English (EN), Spanish (ES), French (FR), Italian (IT), Japanese (JA), Korean (KO), Portuguese (PT), Swahili (SW), and Chinese (ZH). We collect data for these languages from the WMT21 news translation task sources (composed of Europarl v10, ParaCrawl v7.1, ParaCrawl v8, Common Crawl, News Commentary v16, Wiki Titles v3, UN Parallel Corpus V1.0, Tilde Rapid, WikiMatrix, Back-translated news, Japanese-English Subtitle Corpus, The Kyoto Free Translation Task Corpus, and TED Talks) as well as Opus-100 (Zhang et al., 2020), Tatoeba (Tiedemann, 2012), and CCMatrix (Schwenk et al., 2021). We deduplicate the data, and preprocess it using the M2M-100 (Fan et al., 2021) scripts.4 The final dataset sizes can be seen in Appendix B.
Since CCMatrix is a large yet low quality data source, we found it helpful to downsample it relative to the other sources using temperature sampling. For more details, see Appendix B.
Evaluation For evaluation, we use the dev and devtest splits of the Flores-101 dataset (Goyal et al., 2022) as our validation and test sets, respectively. Except when stated otherwise, the reported numbers are on the test set. We report both CHRF (Popović, 2015) and SPBLEU (Goyal et al., 2022), a SENTENCEPIECE-based BLEU computed using the Flores-101 tokenizer, with sacreBLEU5 version 2.3.1. The evaluation signatures are nrefs:1 | case:mixed | eff:no | tok:flores101 | smooth:exp for SPBLEU, and nrefs:1 | case:mixed | eff:yes | nc:6 | nw:0 | space:no for CHRF. All our results are from a single training run of each architecture, and we perform statistical significance tests using paired bootstrap resampling (Koehn, 2004). We run the significance tests for CHRF for all language directions, using a significance level of 5%. We also provide COMET scores (Rei et al., 2020)6 for selected models in Appendix G.
Tokenization We use SENTENCEPIECE (Kudo and Richardson, 2018), with a vocabulary size of 250k, and a character coverage of 0.9995. We balance the data for SENTENCEPIECE training by ran-
4https://github.com/facebookresearch/fairseq/ tree/main/examples/m2m_100
5https://github.com/mjpost/sacrebleu 6Obtained with wmt20-comet-da from version 1.1.2.
domly sampling 1.5M sentences per language.
Tagging We found it helpful to make the model aware of the corpus by training with corpus labels. Similarly to NLLB Team et al. (2022), we add a tag (e.g. <HQ> or <LQ>) to the beginning of the source sentence, so that the model can learn to distinguish between higher quality (WMT21, Opus-100, and Tatoeba) and lower quality examples (CCMatrix). During inference, we always use the high quality (<HQ>) tag. Additionally, we append source and target language tags to the end of the sentence.
Architecture In our experiments, we use a deep encoder, shallow decoder architecture (Kasai et al., 2021) with 16 encoder layers and 3 decoder layers. We share token embeddings between the encoder, decoder, and output layer (Press and Wolf, 2017). In our experiments we consider two kinds of models: those with target language-specific decoders, following Dong et al. (2015), on which we conduct most of our experiments, and those with a shared decoder. The encoder is always shared, with the exception of the LSLs. In the baseline models, the encoder consists only of “regular” Transformer Layers, and so it is fully shared. In this work, we only consider adding LSLs to the encoder. In preliminary experiments with LSLs in the decoder, our selection criteria picked target-specific LSLs for all decoder layers, effectively choosing a separate decoder architecture. We tried different placements of the layers in the decoder, but did not achieve any improvements. We leave a deeper analysis to future work.
Hyperparameters All experiments are implemented using FAIRSEQ (Ott et al., 2019). We use ADAM (Kingma and Ba, 2015) for optimization, due to its robustness (Schmidt et al., 2021) and popularity, with a learning rate of 0.0004. We train for 150k steps, by which point our models had converged, with 4000 warm-up steps, and an inverse square root learning rate scheduler (Vaswani et al., 2017). Due to the abundance of data, adding regularization in the form of dropout or weight decay did not help in our initial experiments, so we do not use any regularization in the remaining experiments. The layer and embedding sizes are 512, the hidden size of the feed-forward layers is 2048, and we use 8 attention heads. All models are trained using fp16 (Ott et al., 2018).

heading: 4.2 Architecture Search
text: As described in Section 3.2, we train a separatedecoder model where all encoder layers are a mix of shared, source, and target weights. This architecture used a total of 804 million (M) parameters, and achieved a score of 46.6 CHRF points (27.4 SPBLEU), averaged over all language pairs. We plot the mixing coefficients of the model in Figure 3, averaged over 3 runs.
We can see clear trends here: the model gives a higher weight to the source-specific sub-layers near the bottom of the encoder, while the target-specific sub-layers get a higher weight near the top. This is in line with previous studies as lower encoder layers usually capture low-level information about the source (Tenney et al., 2019), while the top encoder layers are known to already capture target language information (Kudugunta et al., 2019). Interestingly, the mixing coefficients for the shared weights are relatively stable across layers, making them dominant for the middle layers of the model.
Taking the argmax of the mixing coefficients, we select the architecture in Figure 2, where layers 3 and 4 are source-indexed LSLs7, layers 13, 14, 15 are target-indexed LSLs, and the remaining layers are “regular” Transformer encoder layers (Figure 1a). From here onward, we will refer to this architecture as LSL-NAS. We use the architecture selection method only to select the architecture, and the selected architecture is trained from scratch (not pruned) in the upcoming experiments. To
7For these layers there is some uncertainty in the source weights, but they are the largest weights by a small margin. Performance is improved by selecting source layers, as can be attested by comparing to LSL (SRC=∅ & TGT={13, 14, 15}).
simplify the text, we will also use the notation LSL (SRC={1, 2} & TGT={15, 16}) to refer to an architecture with source-indexed LSLs in layers 1 and 2, and target-indexed LSLs in layers 15 and 16.

heading: 4.3 Learned Architecture Comparison
text: In Table 1, we compare our baseline separate decoder architecture (with a fully shared 16 layer encoder) with the learned architecture from the architecture search (LSL-NAS), and additional variants. We share CHRF and SPBLEU scores averaged over all language pairs, as well as the number of total (|θ|) and effective (|θeff|) parameters used during inference for each architecture. For the baseline models, |θ| and |θeff| differ due to the separate decoders. For an accurate comparison of CPU and GPU speed, see Appendix H.
Our learned architecture (LSL-NAS in Table 1) achieves a 0.9 CHRF (1.2 SPBLEU) improvement over the baseline, which we can be further increased to 1.3 with dense pre-training, reaching a total of 46.8 CHRF (27.5 SPBLEU). These improvements are statistically significant (p < 0.05) for all but 6 of the 90 translation directions. In Table 2, we summarize the averaged results for translating to and from each language, i.e. X→ DE is the average CHRF score for translating into German from all other languages. For the full results (per language pair) on the validation and test sets, see Appendix C. Our approach gives substantial gains for both high resource languages, such as
English and German, which improve by more than 1 CHRF point, as well as lower resource, such as Korean, with close to 2 CHRF points improvement for both directions, or Swahili, which improves by over 1.5 CHRF points in both directions. Although the effective number of parameters is the same for this architecture and our baseline (186M), it can be argued that this comparison is unfair, since our model is bigger. To alleviate this concern, and to show that the gains we achieve are not just due to the higher parameter count, but rather, the better way we allocate the extra parameters, we trained three bigger baselines: with hidden sizes of 640, 704, and 768. As expected, these models also show an improvement over the original baseline, but even the biggest model, with a total of 509M parameters (15% more than ours) and a higher inference cost than our method, is not able to match our performance (only 46.5 CHRF and 27.3 SPBLEU).
Adapters Following Philip et al. (2020), we insert one adapter block after each Transformer layer. In our experiments, inserting adapters into a pretrained model either provided no improvement over training from scratch or suffered from numerical instabilities, even after tuning the initialization gain (Houlsby et al., 2019). For this reason we report numbers for models trained from scratch, similar to Baziotis et al. (2022). Since our models have separate decoders, we inserted adapters only on the encoder. For completeness, results using adapters on the decoder are reported in Appendix D.
We consider two kinds of adapters: source language adapters (Language Adapters ENC), following Philip et al. (2020), or source language adapters in the bottom half of the encoder and target language language adapters in the top half (Language Adapters ENC (SRC+TGT)). We show the results for different bottleneck dimensions (128, 256, and 512) in Table 1. Our proposal of using source and target adapters on the encoder outperforms using only source adapters (for the same model size). The best performing model, Language Adapters ENC (SRC+TGT), achieves a score of 46.1 CHRF points, 0.3 (0.7) points lower than our model without (with) dense pre-training.These improvements are statistically significant (p < 0.05) for 38 (62) of the 90 translation directions. Results for languagepair adapters (Bapna and Firat, 2019) are shown in Appendix D, but they lag behind language adapters.
Importance of Bottom and Top Shared Layers LSL-NAS uses two shared layers on the bottom and one shared layer on the top of the encoder. In Table 3, we analyze the effect of removing these layers, i.e., moving the LSLs up or down. When comparing SPBLEU there is a small drop when removing either the top shared layer (row “LSL (SRC={3, 4} & TGT={14, 15, 16})”) or the bottom-most shared layer (row “LSL (SRC={2, 3} & TGT={13, 14, 15})”), but the difference is negligible when comparing CHRF. In fact the difference is only statistically significant for 15 of the 90 translation directions. When removing the bottom shared layers (row “LSL (SRC={1, 2} & TGT={13, 14, 15})”) or all the shared layers (row “LSL (SRC={1, 2} & TGT={14, 15, 16})”), there is a bigger difference, but it is only statistically significant for less than 1/3 of the translation directions, mostly low resource pairs including either Swahili or Korean. For an analysis regarding the number of LSLs, please refer to Appendix E.
Alternative Configurations Additionally, we look at different configurations of LSLs. In particular, we compare using only source-specific layers LSL (SRC={3, 4} & TGT=∅) or target-specific layers LSL (SRC=∅ & TGT={13, 14, 15}) in Table 1. In both cases, the configuration is worse than LSLNAS, thus showing the importance of having both source and target-specific layers. For completeness,
row LSL (SRC={13, 14, 15} & TGT={3, 4}) shows the opposite of our configuration (i.e., swapping the source and target layers), with considerably degraded performance, showing that the position of source-specific and target-specific languages is very important. In particular, it shows that forcing the model to learn source-specific representations at higher encoder layers and target language representations on the lower layers hinders learning.
Layer Component Ablation We analyze the effect of using full language-specific layers (LSL), with having only language-specific feed-forward (LS-FFN), or only language-specific attention (LSATTENTION) on the LSL-NAS architecture in Table 4. We observe a small degradation of 0.1 CHRF (0.4 SPBLEU) when switching from LSL to LS-FFN, which is statistically significant for 35/90 translation directions, and a degradation of 0.5 CHRF (0.7 SPBLEU) when switching to LSATTENTION, which is significant for 49 directions. These results imply that both the language-specific feed-forward and attention are important, with the biggest contribution coming from the feed-forward part, where most of the parameters are located.

heading: 4.4 Shared Decoder
text: So far we have focused on the separate-decoder architecture. In this section, we turn to a shared-
decoder setup (see Table 5). As in Section 4.2, we ran an architecture search experiment and selected the following architecture: LSL (SRC={4} & TGT={12, 13, 14, 15, 16}), or LSL-NAS-SD for short. The mixing weights follow a trend similar to Figure 3. With the shared decoder, we benefit from placing more target-specific layers at the top of the encoder. Our intuition is that these layers compensate the lack of a separate decoder.
As in Section 4.3, we compare against shared decoder baseline models (i.e., without LSLs) of increasing sizes, as well as models with Adapter Blocks. For the latter, we insert one block after each Transformer layer, both on the encoder and the decoder. Following Philip et al. (2020), we insert source adapters on the encoder, and target adapters on the decoder. As expected, shared-decoder models perform worse than their separate-decoder models, which have a higher parameter count. Despite this, our proposed architecture, LSL-NAS-SD, outperforms the remaining models by a wide margin, and is even better than the separate-decoder baseline (26.0 SPBLEU). The improvements of our LSL-NAS-SD model with pre-training over the shared decoder baseline are statistically significant for 86/90 translation directions. The improvements over the best adapter model (bottleneck size 512) are significant for 76/90 directions.
We also show the performance for LSL (SRC={4} & TGT={13− 16}), an architecture similar to LSL-NAS-SD, but with one less targetspecific LSL. This architecture performs worse than our selection, but has fewer parameters, which might make it a preferable candidate for deployment. This highlights a limitation of our selection approach: it does not take model complexity (i.e., model size) into account. We tried adding a prior on the mixing weights to make LSLs more costly than shared layers, but obtained mixed results, and we leave further investigation to future work.

heading: 4.5 Zero-shot Translation
text: In the previous experiments, we used training data for all language directions. We now consider a different scenario: we limit our training data to English directions (i.e., X-EN and EN-X) and languages in the same language group8. We then evaluate our models on zero shot performance for the directions between groups.
8We consider 3 groups: European, CJK, and Swahili. We use data where both the source and target languages are in the same group.
In our initial experiments, separate decoder models performed poorly on zero-shot directions, so we focused our evaluation on shared decoder models. Table 6 shows the zero-shot results for 3 architectures: the shared decoder baseline, the best performing (shared decoder) adapter model (Shared Decoder Adapters 512), and LSL-NAS-SD. Our approach gives improvements for most zero-shot directions, except when translating into SW. Translating from SW works well, though. Our intuition is that this degradation is caused by the SW targetspecific LSLs being overfitted to EN, and thus failing to transfer to other languages. In LSL-NASSD, the top 5 encoder layers are target LSLs, and in the zero-shot scenario, the SW layers are only trained for EN-SW, which is relatively small. Indeed, if we exclude the * →SW pairs, both the overall and the zero-shot average scores increase.

heading: 5 Conclusion
text: In this work, we studied how to increase the capacity of MNMT models using LSLs. We showed that LSLs are effective at increasing the model capacity per language, while keeping the computation
requirements constant. We proposed a method for selecting the placement of LSLs, and showed the importance of having shared as well as source and target language-specific parameters on the encoder.
Limitations
In this work, we focused our exploration of LSLs on the encoder. Although we ran some initial explorations on the decoder side, further investigation is needed. Another venue for research is how LSLs affect language expansion. Since our approach tries to limit the language-specific weights to just a few layers, in theory, it should be possible to add new languages by only expanding and training the LSLs. However, blindly doing so might not work well and the interactions between languages from different families needs further studying. Lastly, it is unclear whether our argmax approach to selecting where to place LSLs is optimal, how dataset dependent it is, and if there exist alternative approaches that can lead to better results. The fact that it does not take model complexity (i.e., model size) into account can be a disadvantage in practice.
Ethics Statement
Our work uses existing datasets, so it inherits some of the risks associated with them, namely gender bias (Cho et al., 2019), or privacy leakage (Carlini et al., 2021), and mitigation strategies such as Vanmassenhove et al. (2018) may be necessary. However, replacing bilingual translation systems with multilingual systems should help reduce gender bias caused by pivoting through English. Another consideration is the energy consumption for model training, which results in green-house emissions (Strubell et al., 2019). Our proposed architectures result in smaller (and faster to train) models, than similarly-performing baselines, increasing the efficiency of translation systems.

heading: Acknowledgements
text: We would like to thank Sarthak Garg, Luke Carlson, António V. Lopes, and Matthias Sperber for their comments and suggestions, which significantly improved the final work.

heading: A LSLs in FAIRSEQ
text: Listing 1 shows our implementation of LSL in FAIRSEQ. The implementation is straightforward, and consists of a dictionary that selects the appropriate language depending on the lang_pair attribute, which FAIRSEQ dynamically sets, and is guaranteed to match that of the input.

heading: B Dataset sizes
text: For most language pairs, CCMatrix is the largest data source, and it is also the lowest quality one. To compensate for this quality imbalance, we apply temperature sampling (Arivazhagan et al., 2019) to balance the different sources, using a temperature of 5, which worked well in our experiments. In our initial experiments, we considered two approaches to apply this temperature re-sampling: either upsampling the higher quality sources (WMT21, Opus-100, and Tatoeba), or downsampling CCMatrix. The results between these two approaches were similar, and since the downsampling runs were faster and more stable, we used the downsampling for all our experiments. To avoid discarding too much data, we capped the maximum downsampling to a factor of 10.
Table 7 shows the number of sentence pairs for each language direction, after de-duplication, cleaning, and downsampling CCMatrix.

heading: C Full results
text: Table 8 shows the CHRF scores on the Flores-101 test set for all language directions, of both our shared-encoder, separate-decoder baseline model and our proposed LSL-NAS architecture with pretraining. Statistically non-significant results (p ≥ 0.05) are marked with † (in total 6 of the 90 language pairs). The results on the validation set can be found in Table 9.

heading: D Results on different configurations
text: In Table 10 we show the results of further experiments with Adapter Blocks. Besides encoder source language adapters (Language Adapters ENC) and source adapters in the bottom half of the encoder and target adapters in the top half (Language Adapters ENC (SRC+TGT), we include source adapters on the encoder and target adapters on the decoder (Language Adapters ENC+DEC, like Philip et al. (2020), and language-pair adapters
on the encoder (Bapna and Firat, 2019) (LanguagePair Adapters ENC). Our proposed architecture, LSL-NAS, outperforms all other techniques while introducing no extra computation at inference time (i.e., it keeps |θeff| constant).

heading: E Number of LSLs
text: We look at the effect of changing the number of LSLs, illustrated in Figure 4. To this end, we change the number of LSLs from 0 to 16, in increments of 2, and, for each point, we place an additional LSL on the bottom and on the top of the encoder, using the source and target languages to index them, respectively. For example, 4 LSLs corresponds to LSL (SRC={1, 2} & TGT={15, 16}). We see that adding more LSLs helps performance, but only up to a point (in this case, 4 layers), and that afterwards, performance degrades, except for an outlier at 12 LSLs. This implies that while the language-specific layers boost performance, having shared layers is crucial for knowledge transfer.

heading: F Per Language results
text: In Table 11, we show aggregate scores for each language group: European (DE, EN, ES, FR, IT, PT), CJK (ZH, JA, KO), and SW (isolated, since it is the only language in its family). Here, we see a similar trend, with our approach showing clear improvements both within groups, and between different groups.

heading: G COMET results
text: We show COMET, CHRF, and SPBLEU scores, averaged over all language pairs in Table 12. We show the scores for the baseline (i.e., non-LSL), our LSL model, and the best Adapter model for both the separate decoder and the shared decoder architectures. In all metrics, our proposed architectures outperform the remaining models.
H Inference Speed
We report the inference times for the various architectures we considered in Table 13. We report tokens/second on the DE-EN test set9, averaged over 5 runs. Our latency measurements were collected using a single NVIDIA V100 GPU (Speed GPU) or a single-threaded Intel Xeon Platinum 8275CL CPU @ 3.00GHz (Speed CPU), both with batch
9We repeated these measurements for language pairs, such as EN-ZH, with similar results.
size of 1, which faithfully captures the inference on a deployed neural machine translation model. As expected, the latency of shared decoder models is the same as that of similar separate decoder models (since only one of the decoders is used at inference time) so, for succinctness, we only report the separate decoder numbers.
A couple of comments regarding the Adapter models: 1) we do not report speed numbers for the “Language Adapters ENC (SRC+TGT)” as the architecture is the same as “Language Adapters ENC”; 2) inference speed does not change significantly when adding encoder adapters, but only when adding adapters to the decoder.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We have a "Limitations" section without section number, present after conclusion.
3 A2. Did you discuss any potential risks of your work? We have an "Ethics Statement" section without section number, present after conclusion.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract has no section number. Introduction is section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We share code in Appendix A, and list the datasets we use in section 4.1.
3 B1. Did you cite the creators of artifacts you used? We use various datasets, and cite in section 4.1.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We use the splits provided by the dataset creators, and mention this fact in the paper (section 4.1).
C 3 Did you run computational experiments? Section 4.
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We provide parameter counts (section 4), as well as the inference infrastructure (Appendix H) and inference speed (Appendix H).
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.1.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4. We provide average scores (and say so in the text), and did significance tests for the results.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We tried adding a prior on the mixing weights to make LSLs more costly than shared layers, but obtained mixed results, and we leave further investigation to future work