Gradient-based Gradual Pruning for Language-Specific Multilingual Neural Machine Translation
Authors: Dan He
Minh-Quang PHAM
Thanh-Le Ha
Marco Turchi
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual neural machine translation
language-specific sub-network extraction
model pruning
gradual pruning schedule
gradient-based pruning criterion
TL;DR: Applying gradient-based gradual pruning for language-specific multilingual neural machine translation.

Abstract: Multilingual neural machine translation (MNMT) offers the convenience of translating between multiple languages with a single model. However, MNMT often suffers from performance degradation in high-resource languages compared to bilingual counterparts. This degradation is commonly attributed to parameter interference, which occurs when parameters are fully shared across all language pairs. In this work, to tackle this issue we propose a gradient-based gradual pruning technique for MNMT. Our approach aims to identify an optimal sub-network for each language pair within the multilingual model by leveraging gradient-based information as pruning criterion and gradually increasing the pruning ratio as schedule. Our approach allows for partial parameter sharing across language pairs to alleviate interference, and each pair preserves its unique parameters to capture language-specific information. Comprehensive experiments on IWSLT and WMT datasets show that our approach yields a notable performance gain on both datasets.

Gradient-based Gradual Pruning for Language-Specific Multilingual Neural Machine Translation
Dan He, Minh- Quang PHAM, Thanh-Le Ha, Marco Turchi
Zoom Video Communications
(dan. he, mwnmquang.nnn.. pham, thanhle.. ha, marco. turchi Jezoom.
Abstract
Despite these benefits, MNMT remains challeng- ing as its performance degrades compared to bilin- gual counterparts in high-resource languages (Ari- vazhagan et al., 2019). Previous studies (Wang et al., 2020; Shaham et al., 2023) attribute this degradation to parameter interference: each lan- guage is unique and therefore has distinct re- quirements from the model parameters to capture language-specifie information, and fully sharing parameters causes negative interactions across lan- guage pairs as languages compete for model capac- ity. Although naively increasing model size may alleviate parameter interference, large models often suffer from parameter inefficiency and overfitting (Zhang et al., 2020; Arivazhagan et al., 2019).
Multilingual neural machine translation (MNMT) offers the convenience of translating between multiple languages with single model. However, MNMT often suffers from performance degradation in high-resource languages compared to bilingual counterparts. This degradation is commonly attributed to parameter interference, which occurs when parameters are fully shared across all language pairs. In this work, tackle this issue propose a gradient-based gradual pruning technique for MNMT. Our approach aims to identify an optimal sub-network for each language pair within the multilingual model by leveraging gradient-based information pruning criterion and gradually increasing the pruning ratio schedule. Our approach allows for partial parameter sharing across language pairs to alleviate interference, and each pair preserves its unique parameters capture language-specifse information. Comprehensive experiments on IWSLT and WMT datasets show that our approach yields notable performance gain on both datasets.
To address the parameter interference issue more effectively, researchers have explored various meth- ods to allocate language-specing parameters cap- ture unique linguistic information for each lan- guage pair. One line of work focused on intro- ducing extra parameters to the model. For instance. adapter-basee based approaches (Bapna and Firat, 2019; Philip et al., 2020; Zhu et al., 2021; Baziotis et al., 2022) inject lightweight language-specing mod- ules into the shared model. Another well-known method (Dong et al., 2015) utilizes shared en- coder but different decoder for each target lan- guage. Despite being effective, these methods can become parameter-ineffccient as the number of lan- guages in the multilingual model grows.
1 Introduction
In recent years, neural machine translation (NMT) based on the transformer architecture has achieved great success and become the dominant paradigm for machine translation (Sutskever et al.. 2014; Vaswani et al., 2017). Multilingual neural machine translation (MNMT), which learns a unified model to translate between multiple languages, has at- tracted growing attention in the NMT area (Ha et al., 2016; Johnson et al., 2017). The reasons are: 1) From a practical perspective, it significantly reduces the training and inference cost and simpli- fies deployment in production: 2) Utilizing data from multiple language pairs simultaneously can potentially help improve the translation quality of low-resource or even zero-resource language pairs by transferring knowledge across languages.
Another line of work focused on extracting a sep- arate sub-network within the multilingual model for each language pair. In these approaches, each sub-network preserves exclusive parameters to cap- ture the language -anguage-specffic parameters shared with other languages (Xie et al., 2021; Lin al., 2021; Wang and Zhang, 2022: Pham et al., 2022), A recent work (Lin et al., 2021) shows promising results by first training multi- lingual model that covers all language pairs, then fine-tuning the trained model on each pair, and
specific features and has some

---

the end, extracting the sub-networks via magnitude pruning in one step. Although this paradigm is intu- itive and straightforward. it can be suboptimal due to the following limitations: (1) Magnitude prun- transfer learning context (Sanh et al., 2020), potentially af- fecting NMT systems in the multilingual scenario. (2) Pruning in one single step after fine-tuning can lead to the removal of crucial weights resulting in lower performance.
2 Related Work
Standard multilingual neural machine translation systems translate between multiple languages with a unified model. The model can be jointly trained on multiple language pairs by prepending a special token to the source sentence, informing the model about the desired target language (Johnson et al., 2017). Although fully sharing parameters across languages and joint training can enhance knowl- edge transfer, MNMT suffers from parameter in- terference and the lack of language-specific param- eters for capturing sagguage-speciigg information, resulting in performance degradation, especially in high-resource language pairs. Various previous works have explored the idea of partially sharing parameters across languages while allowing for language-specifie parameters. Sachan and Neu- big (2018) investigates different parameter sharing strategies. Blackwood al. (2018) compares sev- eral methods of designing sanguage--speifiii atten- tion module. Zhang al. (2021) studies when and where language-specific capacity matters.
ing is demonstrated to be ineffective
This work aims to mitigate the parameter in- terference issue in MNMT without adding ex- tra parameters and overcome the aforementioned two limitations in Lin et al. (2021). We pro- pose gradient- gradient-based gradual pruning for language- specific MNMT. More specifically, multilingual base model is first trained, and subsequently, the model is simultancously fine-tuned and pruned on each language pair. Conducting finetuning and pruning concurrently allows the model to adapt and optimize the parameters while reducing the model size. Instead of the widely used magnitude scores, we opt for the gradient-basee scores as the prun- ing criterion. The percentage of pruned weights is gradually increased from zero to the target level throughout the pruning process. By optimizing the pruning criterion and schedule, we strive to iden- tify optimal sub-networks and limit the parameter interference. Lastly, the resulting sub-networks are integrated into the MNMT model through a final training phase in a language-aware manner.
Additionally, another widely recognized tech- nique, adapter-based, has attracted substantial in- terest in recent years. Adapter- approaches (Bapna and Firat, 2019; Zhu al., 2021: Baziotis et al., 2022) inject additional lightweight language- specific modules for different language pairs to capture language-specifie information. Although effective, these methods increase the parameters for each language pair and thus result in an infer- ence speed decrease. On the contrary, our work introduces extra parameters to the model and thus has negligible to impact the inference speed. For more details, see Appendix E.
A large set of experiments is conducted on IWSLT and WMT datasets, showing that our ap- proach leads to substantial performance gain of 2.06 BLEU on IWSLT and 1.41 BLEU WMT. Our contributions can be summarized as follows:
To avoid the inference speed decrease, several works (Dong et al., 2015; Purason and Tättar, 2022; Pfeiffer et al., 2022; Pires et al., 2023) explore de- signing language-aware modules inside the MNMT model. This way, the model can capture language- specific information without sacrificing the infer- ence speed. For instance, Dong et al. (2015) em- ploys a single shared encoder across all languages but a unique decoder for each target language. However, the model can suffer from the param- eter exploration issue as the number of languages in the multilingual model increases. In contrast, our approach maintains consistent number of total pa- rameters, irrespective of the number of languages. Recent works allocate language-specine param- eters by extracting a unique sub-network for each
Our method leads to significant boost in medium- and high-resource languages and also reasonable improvement in low- resource languages, suggesting its effective- ness alleviating parameter interference.
We provide comprehensive study of vari- ous pruning criteria and schedules in search- ing optimal sub-networks in the multilingual translation scenario.
We provide additional analyses of our method by studying the contribution of various sub- layers to the overall performance and explor- ing the relationship between language-specific sub-networks and language families.

---

language pair within the multilingual model, which avoids introducing additional parameters the model. Different approaches are explored for sub- network extraction, such as Taylor expansion (Xie et al., 2021), parameter differentiation (Wang and Zhang, 2022), and model pruning (Lin ct al., 2021). In this paper, we focus model pruning because it showed to be more effective and results in promis- ing performance.
Algorithm I Gradient- based Gradual pruning for
MNMT
Input: N bilingual corpora data Dall
(Da--t1- Deg-tt::.. -g. beg
language
pairs
pairs {8l t1, 12, 8N tNN
// Phase pretrain multilingual model Training a multilingual model hase: 9g
// Phase extract sub-networt each pair for each pair 8, if step < T2 then
pairs
// Finetune without pruning Fine-tune and keep prune_ratio=
In machine learning, model pruning is widely used to remove redundant weights from neu- ral network while preserving important ones to maintain accuracy (Han et al., 2015, 2016; Fran- kle and Carbin, 2019; Liu et al., 2019; Sun et al.. 2020). Two key components of model pruning are the pruming criterion, which determines the rela- tive importance of weights and which weights to prune, and the pruning schedule, which defines the strategy how the target pruning ratio is achieved throughout the pruning process. In terms of prun-
else if step < (Ti + +T) then
10:
Calculate gradient-based importance scores as shown in Eq. (2), and rank the scores.
11:
Calculate the current pruning ratio according to Eq. (4) and prune the weights with the lowest scores accordingly.
12:
else
13:
Finetune with target ratio until converge.
14:
end if
15: Extract pruning mask Mo.-rt;
16:
Extract sub-network 0,,-11; Oo M.,
17:
end for
18:
// Phase structure- joint training
19:
6 (f.g-ty- Beg--ta--.. -
ing criteria, magnitude pruning (Han et al., 2015,
20: while fAi not converge do for each pair 8; in pairs do
21:
2016), which removes weights with low absolute values, is the most widely used method for weight pruning. A recent work on large language models (Sanh et al., 2020) proposed movement pruning. which scores weights by using the accumulated product of weight and gradient and removes those weights shrinking toward zero. We refer to this approach as gradient-based pruning in this work. Regarding pruning schedules, two commonly con- sidered options are one-shot pruning (Frankle and Carbin, 2019) and gradual pruning (Zhu and Gupta, 2017). While one-shot pruning removes the de- sired percentage of weights in a single step after the completion of finetuning, gradual pruning incre- mentally removes weights, starting from an initial pruning ratio (often 0) and gradually progressing towards the target pruning ratio.
Further training 0or-re, on Dac-tte 23: end for
22:
24: end while
tuning, allowing the model to self-correct and re- cover from previous choices. To the best of our knowledge, this is the first study to explore the ef- fectiveness of gradient-based gradual pruning in the multilingual translation context.
Methodology
Our approach includes three main phases. In the first one (Phase 1 in Algorithm 1), multilingual model is initially trained using the parameter log- likelihood loss (see Sec. 3.1). Subsequently, the multilingual base model is simultaneously fine- tuned and pruned on each language pair (Phase 2 in Algorithm 1). In this phase, we adopt the gradient- based pruning criterion as the scoring function for each weight and the gradual pruning schedule to identify sub-network masks for language pairs (see Sec. 3.2). The extracted masks are then jointly used during the final training (Phase 3 in Algorithm 1) as described in Sec. 3.3.
In MNMT, the most similar technique our ap- proach for extracting nebnnetwrrss through model pruning has been proposed by Lin et al. (2021). In their work, sub-networks are searched through mag- nitude one-shot pruning, which means the model is prumed based on the magnitude of weights in single step after the completion of fine-tuning, Dif- ferent from their method, we opt for the gradient- based pruning criterion, which is proved to be more transfer learn-
3.1 Multilingual Neural Machine Translation
effective than magnitude pruning ing context. Besides, in contrast to pruning one single step (Lin et al., 2021), ), we gradually increase the ratio from zero to the target value during fine-
In this work, we adopt the multilingual Transformer (Vaswani et al., 2017) as the backbone of our ap- proach. Following Lin et al. (2021), we use uni- fied model for multilingual NMT by adding two

---

special language tokens to indicate source and tar- get languages. Given a set of N bilingual corpora Dau = (Da1 D D22- ............. D.. ...DaN-tt' ), the multi- lingual model is jointly trained over the set of all N parallel training corpora. The objective to minimize the parameter log-likelihood of the target sentence given the source sentence over all corpora. The training loss is formulated as follows:
Weights with scores among the lowest v% are assigned a value of 0 in the binary mask matrix and pruned, while the other weights are assigned a value of in the mask and retained.
We generate mask for each matrix that is tar- geted for pruning in the model and extract unique lub-network for each language pair: 0 80
Ms, M Where C denotes the Hadamard product. 8 + t, is the language pair, Ms:-,, {0, 1 1) represents the mask of the entire model for pair
N
LMT
log logp(u,slu..s,,..
8 + 4, 0g denotes the initial model, and 0 represents the corresponding sub-network for the pair s; - t4.
where X; (Ii,, Fi.2,..., Fi.1) and Y (3.,1 Vi,2, ..., J) represent the source and target sentences of one sentence pair in the parallel corpus De,-st, respectively, with source sentence length I and target sentence length J, and special tokens omitted. The index of the current target word is denoted by j. which ranges from I to the sentence length J. 0 represents the model parameters.
Gradual pruning schedule. In this work, the prun- ing ratio (v% in Eq. (3)) is gradually increased from the target value Rp through a three-stage process, similar to Zhu and Gupta (2017). In the first stage spanning Ti training steps, the model remains unpruned with a pruning ratio of O. In the second stage, which lasts for T2 training steps, the pruning ratio gradually increases from 0 to the predefined threshold Rp- In the third stage, the pruning ratio remains constant at the target pruning ratio Rg. This strategy is formalized as follows:
3.2 Identify Sub-networks Via Pruning
Once the MNMT model is trained, sub-networks are identified by applying our pruning approach. Gradient-based pruning criterion. Inspired by Sanh et al. (2020), we first learn an importance score for each weight in the weight matrix targeted for pruming, and then prune the model based on these importance scores during the simultaneous finetuning and pruning process. The importance scores can be represented as follows !:
0
t <<Ti Ti
Rs
=
Rp (1
(1-
t-7; T2
Yi <t <t) + +7))
Pp
otherwise
(4)
where t represents the current training step, Rs represents the pruning ratio at step f, R2 represents the preset target pruning ratio, Ti and T2 represent the total steps of stage and stage 2, respectively.
s TTT
-
aL
(K)
w
(2)
aw
where OL is the gradient of loss L with respect aW., to W., generic weight matrix W C RM of the model, T denotes the number of performed gra- dient updates, $S7 denotes the importance score of weight W after T updates.
3.3 Joint Training
Once the sub-networks 0 1,..., N for all language pairs are obtained, the multilingual base model 0g is further trained with language-awan data batching and structure-aware model updating. For this purpose, batches of each language pair are randomly grouped from the language-specinn bilingual corpus (Lin et al., 2021). Given a pair 8; - tú batches Ba for this pair are randomly grouped from the bilingual corpus De; xt. Impor- tantly, each batch contains samples from single language pair, which differs from standard mul- tilingual training where each batch can contain fully random sentence pairs from all language pairs. The model is iteratively trained on batches of all language pairs until convergence. During back-
After scoring each weight using Eq. (2) and rank- ing the score values, we prune the weights having importance scores among the v% (pruning ratio) lowest, regardless of the absolute score values. To this end, binary mask matrix M E {0, MxN based on the importance scores is calculated as:
Mi = fo 1,
Srj among v% lowest scores
otherwise
For detailed information please refer to Appendix G.

---

propagation, only parameters in the sub-network of the corresponding language pair are updated. Dur- ing inference, the sub-network corresponding to the specific language pair is utilized to generate predictions. Notably, the sub-networks of all lan- guage pairs are accommodated in a single model, without introducing any additional parameters.
we opt for Transformer-small following Wu et al. (2019). For the WMT experiment, we choose Transformer-base For more training details please refer to Appendix B. To have fair comparison with Lin et al. (2021), our method applied to two linear sub-layers: attention and feed fcrw-forward."
4.3 Terms of Comparison
4 Experiment settings
We compare our method with two well-known and adopted technologies: the adapter-based (Bapna et al., 2019) and the method utilizing a shared en- coder but separate decoders (Dong al., 2015).
4.1 Data
We run our experiments using two collections of datasets (IWSLT and WMT) having different lan- guage coverage and training data sizes. Due to its limited dimensions, the IWSLT data is used to run a deeper analysis of our method while WMT is used to further simulate real-world imbalanced dataset scenarios.
Adapten Adapter.256, and Adapter:512 Inject unique lightweight adapter to the shared model for each language pair with vary- ing bottleneck dimensions 128, 256, and 512, respectively (Bapna et al., 2019). Following Pires et al. (2023), we train the entire model, including all adapters, from scratch for train- ing stability.
IWSLT. We collect 8 English-centric language pairs for a total of 9 languages from IWSL12014 (Cettolo al., 2014), with corpus size ranging from 89K to 169K, as shown Appendix A.1. We first tokenize data with moses scripts (Koehn et al., 2007)?, and further learn shared byte pair encod- ing (BPE) (Sennrich et al., 2016), with vocabu- lary size of 30K, to preprocess data into sub-word units. To balance the training data distribution, low-resource languages are oversampled using temperature of T=2 (Arivazhagan al., 2019). WMT. We collect another dataset comprising 19 languages in total with 18 languages to-and-fron from English from previous years' WMT (Barrault et al., 2020). The corpus sizes range from very low- resource (Gu, 9K) to high-resource (Fr, 37M). More detailed information about the train, dev, and test datasets is listed in Appendix A.2. We apply the shared byte pair encoding (BPE) algorithm us- ing SentencePiece (Kudo and Richardson, 2018) to preprocess multilingual sentences, with vo- cabulary size of 64K. Since the data is extremely imbalanced, we apply oversampling with a larger temperature of T=5. We categorize the language pairs into three groups based on their corpus sizes: low-resource (< IM), medium-resouree (> 1M and <1 <10M), and high-resource (> 10M).
SepaDec Use shared encoder for all lan- guage pairs and a separate decoder for each target language (Dong et al., 2015).
As described in Section 3, our GradientGradual approach employs a gradient- based pruming crite- rion, and pruning occurs during finetuning, with the pruning ratio gradually increasing. To investi- gate the effectiveness of our pruning strategy for sub-network extraction in multilingual translation, further compare our method with three pruning- based variants by combining different pruning cri- teria and schedules.
MagniuudeOneshot (LaS8) Extract sub- networks through pruning: the pruning cri- terion is the magnitude values of the weights; the pruning is performed in one step after the completion of finetuning (Lin et al., 2021).
MagnitudGrradual Extract sub-networks through pruning: the pruning criterion is the magnitude values of weights; the pruning ratio increases gradually during finetuning.
GradientOneshot Extract sub-networks through pruning: the pruning criterion is
4.2 Model Settings
our experiments, we adopt models of different sizes to adjust for the variation in dataset sizes. Given the smaller size of the IWSLT benchmark,
"Although our analysis limited to attention and feed- forward sub-layers, our algorithm can be used to prune other components the model, e.g. the embedding matrixes.
"In the original paper, this approach is referred to as LaSs. In the following sections of this paper, we will use Magni- IndeOneshot for simplicity.
http://twwu...cmm/mmmmsss/////s/ http///ssserssrppt////oeeee////e perl

---

Lang
PI
He
NI
De
It
Es
Average
Size
89K
128K
140K
144K
153K
160K
167K
169K
Baseline
17.54
17.25
21.31
29.39
32.11
30.09
30.02
36.18
26.74
Adapter. 128
+0.20
+1.46
+1.32
+1.56
+1.62
+1.34
+1.36
+1.82
+ +.33
256
+0.06
+1.44
+1.53
+1.56
+1.43
+1.52
+1.53
+1.70
+1.34
111
-0.23
+1.39
+1.61
+2.09
+1.67
+1,81
+2.01
+2.01
+1.54
SepaDec
-0.38
+0.79
+0.88
+0.84
+0.75
+1.23
+1.12
+0.73
+0.74
MagnitudeOneshot (LaSS)
+0.04
+0.26
+0.68
+0.65
+0,56
+0,49
+0.72
+0.60
+0,50
Gradien (Ours)
+1.13
+0.67
+2.24
+2.8
+2.56
+2.28
+2.19
+2.60
+2.06
MagnitudeGraddal
+0.43
+0.20
42.22
+2.36
+2.07
+1.92
+2.02
+2.24
+1.68
GradientOmeshot
+0.73
+0.32
+2.08
+2.44
+2.14
+2.08
+1.88
+2.14
+1.73
Table 1; Average En ++ X BLEU score gain of the sub-network extraction methods on the IWLST dataset.
gradient-basede pruning is performed in one
most languages. The consistent improvements in average scores confirm the necessity of language- specific parameters in the MNMT model. Subse- quently, we report the results of the pruning-busee approach MugnitudeOneshot, which extracts sub- networks for language pairs through magnitude pruning in one step. MagnitudeOnesshot obtains 0.50 BLEU score gain, demonstrating the po- tential of designing language speciifc parameters and mitigating parameter interference without in- creasing the parameter count. Furthermore, proposed GradientGrrdnaa approach, which lever- ages gradient- based information as the pruning cri- terion and gradually increases the pruning ratio from to the target value, delivers a substantial im- provement of 2.06 BLEU scores over the baseline model and achieves the best performance among all approaches, suggesting the effectiveness of our method in mitigating parameter interference in the MNMT model.
step after the completion of finetuning.
4.4 Evaluation
We report the tokenized BLEU (Papineni et al., 2002) individual languages the IWSLT dataset, and average tokenized BLEU of low-, medium-, and high-resouree groups on the WMT dataset using the SacreBLEU tool (Post, 2018). We provide detailed results on the WMT dataset in Ap- pendix D.I. . In addition, we show the results win ratio based on tokenized BLEU in Appendix D.2. and the results of COMET (Rei et al., 2020) and chrl' (Popović, 2015) scores in Appendix D.3.
5 Experiment Results on IWSLT
This section shows the results of our approach on the IWSLT dataset, along with comparative anal- ysis against the MNMT baseline and prior research works. We also provide a comprehensive analysis our method from various perspectives.
5.2 Pruning criteria and schedules
5.1 Main Results
In the last two rows of Table I we further ex- plore the individual impact of two key factors that contribute to the performance improvement sepa- rately, namely the gradient-based pruning criterion and gradual pruning schedule. MagnitudeGradual leads 1.68 BLEU score improvement over the baseline model and 1.18 BLEU score improve- ment over Magnitudeennesho separately. Gradi- entOneshot leads to a 1.73 BLEU score gain over baseline and 1.23 BLEU score gain over Mag- ninudeOneshot. The results demonstrate that both the muddeent-based pruning criterion and gradual pruning schedule have the potential to significantly improve multilingual translation performance when implemented separately and surpass the perfor- mance of MagnitudeOneshan Nevertheless, the maximum average improvement obtained when the gradient-basee pruning criterion and gradual
In Table 1, we first report the performance of multilingual baseline, adapter-based approaches (Adapter. 128, Adapter:26,, and Adapter.5 with bottleneck dimensions 128, 256, and 512) and SepaDec method, with a dedicated decoder for each target language. On average, Adapter.128, 2da and Adapter.512 achieve 1.33, 1.34, and 1.54 BLEU score gains, respectively. Sepa- Dec clearly underperforms adapter-bart based methods yet exhibits a smaller gain of 0.74 BLEU score points the baseline. The above approaches alleviate the parameter interference issue by in- troducing additional sanguage-speciffs parameters to the model and outperform the baseline across
"Additional analysis of parameter count, disk storage, and inference speed among different approaches is provided in Appendix E.

---

tial, perform ablation experiments by applying our approach the attention and feed-forward sub- layers separately. The results in Table show that applying our approach to feed-forward sub-layers yields a limited average performance gain (+0.49 BLEU) while applying it to attention sub-layers leads to notable gain (+1.22 BLEU), suggest- ing parameters in attention sub-layers are more language-specifie. This finding aligns with the pre- vious work (Clark et al., 2019), which shows that specific attention heads specialize in distinct as- peets of linguistic syntax. Given the unique syn- tax patterns different languages, parameters attention sub-layers are possibly more language- specific and suffer more severe parameter inter- ference. Consequently, applying our approach to attention sub-layers yields notable gain. How- ever, the largest average performance gain (+2.06 BLEU) is achieved when applying our approach to both attention and feed-forward sub-layers. Our results suggest that parameter interference exists in both sub-layers, but more severe in attention sub-layers.
86.5
GradlentGracuul (Durs)
26.0
NagrluxteSrodual
GradientOr estce
25.5
WagnttunOneshottio (LaSS) Bascline
0.3
0.4
0.5
0.6
0.9
Pruring Ratoo
Figure Average BLEU score of all language pairs across 4 methods with pruning ratio [0.1:0.9].
pruning schedule combined, demonstrated in our proposed GradienGGradual approach.
5.3 Robustness with respect to pruning ratios Results presented in Table demonstrate that all methods with different pruning criteria and sched- ules lead to varying levels of performance improve- ment. To gain more comprehensive understand- ing of the relationship between performance and pruning ratios, we further visualize the average per- formance of all language pairs across the 4 meth- ods with different pruning ratios. Results in Figure show that the optimal performance of these methods is obtained at slightly different pruning ratios. In addition, although MagnitudeGradual and GradientOneshot yield a considerably higher performance gain (around 1.7 BLEU) than Magni- tudeOneshot (0.5 BLEU) at their optimal pruning ratios, they both show instability across pruning ratios. Specifically, GradientOneshot suffers from significant performance drop from middle high pruning ratios, and MagninudeGradual experiences an unexpected performance drop at the specific pruning ratio 0.6. In contrast, Gradien(Gradual demonstrates a more robust behavior across a wide range of pruning ratios, consistently outperforming the other methods except for MagnitudeGradue at very high pruning ratio of 0.9. These results further verify the effectiveness and robustness of Gradien approach.
5.5 Similarity Scores and Phylogenetic Tree To gain deeper insights into the effectiveness and interpretability of our method and to evaluate its capability extract high-quality language-specinn nub-nesworks, compute the similarity of masks obtained with our approach, and use these simi- larities reconstruct the phylogenetic trees of lan- guages. We present the results of En - language pairs in this section and the results of XnEnn which exhibit similar patterns as En-X in Appendix F. Similarity scores are determined by the propor- tion of shared parameters between two language pairs (Lin et al., 2021), which can be obtained by dividing the number of shared "1" values in two masks by the number of "I in the first mask, as illustrated in the equation below:
M1 C M2
Sim Sim(M1, M2)
|M1||
where - is Lo norm, M1 and M2 represent binary masks of two language pairs, | @ M2 | represents the number of shared 1, i.c., the num- ber of shared parameters, in these two language pairs. Intuitively, languages within the same family share a higher linguistic similarity, implying an in- creased likelihood shared parameters and higher similarity scores. Conversely, languages that are
5.4 Which sub-layer matters?
In this work, our approach is applied attention and feed-forward sub-layers. To better understand where the parameter interference more severe and where inguage--peciffe parameters are essen-

---

Lang
Fa
Pi
He
NI
De
Average
Size
89K
128k
140K
144K
153K
160K
167K
169K
Baseline
17.34
17.25
21.31
29.39
32.11
30.09
30.02
36.18
26.74
Attn
+0.45
+0.09
+1.39
+1.95
+1.33
+1.57
+1.58
+1.22
+0.1
-0.01
+0.79
+0.74
+0.41
+0.63
+0.71
+0.54
+0.49
AttFFr
+1.13
+0.67
+2.24
+2.8
+1.13
+0.67
+2.24
+2.8
+2.06
Table 2: Sub-layer Ablation Results. Baseline denotes the multilingual Transformer model. Attn denotes applying our approach to attention sub-layers only. Ff denotes the approach applied to the feed-forward sub-layers only. AttnFf represents applying our approach to both the attention and feed-forward sub-layers.
Fa
Ar
11.2
51.2
50.0
49.
55
He
54
48.4
Es
512
51.4
56.4
50.5
48.0
47.9
48.5
Pl
50.9
51.0
10.4
50.5
47.6
47.2
48.0
48,0
50
De
60.4
hc
Figure 3: The built tree positively correlates with lan- guage families. and Es are in the Romance branch; De and NI are in the Germanic branch. Fa, Ar, and He are similar Afro-Asiatic languages; with Fa and Ar written in Arabic script.
Figure 2: Similarity scores (%) of En-X language pairs.
linguistically distant from one another tend to pos- sess more distinct language-specifie characteristics, which implies lower similarity scores.
(Makarenkov and Leclere, 1999) using the T-REX tool (Boc et al., 2012) Figure 3 demonstrates the constructed phylogenetic tree is highly aligned with the language families shown in Table 14 in Appendix F, confirming the capability of our ap- proach extract high-quality sanguage-speciffe sub-networks.
Figure 2 shows the similarity scores of En-+X language pairs. Italian (It) and Spanish (Es) ob- tained the highest score, followed by German (De) and Dutch (NI). The third highest score is obtained by Arabic (Ar) and Farsi (Fa). In addition, we no- ticed that within the European or Afro-Asiatic lan- guage groups, similarity scores between languages are relatively high, while lower scores often observed when comparing a language from the Eu- ropean group to one from the Afro-Asiatic group. The results demonstrate that similarity scores ob- tained with our method are highly positively corre- lated with language family clustering (see Table 14 in Appendix F). This implies the capability of our GradienGGradual approach to generate high-quality language-speciin sub-network.
6 Experiment Results on WMT
On the highly imbalanced WMT dataset with the Transformer-bas model, we compare our method with the multilingual baseline and the prior work MagnitudeOneshot, which shares the same phi- losophy
method: both our approach and MagnitudeOneshot introduce no additional param- eters to the model and have no negative impact on the inference speed. The results Table show that our method outperforms both the baseline and the MagnitudeOneshho across low-, medium-, and high- nigh-resource language pairs, reconfirming the ef- fectiveness of our approach. More specifically, we observe average improvement of 0.91 BLEU on low-resource, 1.56 BLEU on medium-resource and 1.67 BLEU on high-resource language pairs
Furthermore, to quantify the proportion of monguage-speciffe parameters and construct a phy- logenetic tree of languages, we compute lan- guage distance scores as: Dis(M1, M2) = Sim (M1, M2).
With these scores calculated between every two language pairs, the phylogenetic tree constructed according to a weighted least-squares criterion
hhttp://wwe trex. uqam. sa/index.phppe ht/.ieex.hggaactionntre. php?action=trex

---

Model
Low
Med
High
All
Limitations
Baseline
14.01
17.54
24.16
18.76
MagnitudeOneshot
+0.70
+0.85
+0,99
+0.85
Modelling
GradienGradual (Ours)
+0.91
+1.56
+1.67
+1.41
In this work, we explore approach in the English-centric multilingual machine translation setting. However, we believe that the effective- ness of our method extends to the real-world non- English-centric scenario. We will explore this di- rection in future work.
Table 3: Average BLEU improvement of Low (< IM),
Medium (2 1M and 10M) and High (210M) resource language pairs over baseline on WMT dataset.
over the baseline. These results shed light on three aspects, Firstly, the performance improvement be- comes larger when the number training resources increases, which validates findings in previous re- search works, suggesting that high-resource lan- guage pairs tend to be more negatively affected by the parameter interference issue in a unified multilingual model. As result, mitigating the interference issue can result in more notable en- hancements for high-resource languages. Secondly, while previous works suggest that low-resource languages often benefit from knowledge transfer, our results suggest that parameter interference also harms the performance of low-resource languages. By mitigating the interference issue, the perfor- mance of low-resource languages can also be im- proved. Finally, the consistent improvement of our method compared Magnitude Oneshot across lan- guage groups of all resource sizes suggests that our gradient-basee gradual pruning approach more effective in identifying optimal sub-networks and mitigating the parameter interference issue in the multilingual translation scenario.
Regarding model capacity, we adopt Transformer-small and Transformer-bass for IWSLT and WMT experiments, respectively, to reduce the training cost. Given the extensive coverage of languages in the WMT dataset, using an even larger model like Transformer-big (Vaswani et al., 2017) may further boost the overall translation performance, especially high-resource directions, but we do not expect any difference in the relative quality of the tested methods (GradientGradual > MagnitudeOOnshh Baseline). This has been confirmed when moving from Transformer-small to Transformer-bare in our experiments.
Another limitation of our work that the current version our algorithm applies the same pruning ratio to all language pairs and, hence, prunes the same percentage of parameters for each language. However, language pairs with different sizes of available training data may have different optimal pruning ratios. For example, high-resource lan- guage pairs may need smaller pruning ratios and preserve sufficient parameters to process and cap- ture more complex information in the abundant data. To potentially improve the gains and have a method able to differently behave with differ- ent data conditions, future work could explore a method to automatically identify appropriate prun- ing ratios for different languages.
7 Conclusion
a standard MNMT model, the parameters are shared across all language pairs, resulting in the parameter interference issue and compromised per- formance. In this paper, we propose gradient-based gradual pruning for multilingual translation to iden- tify optimal sub-networks and mitigate the interfer- ence issue. Extensive experiments on the IWSLT and WMT datasets show that our method results in large gains over the normal MNMT system and yields better performance and stability than other approaches. Additionally, we observe that the inter- ference issue can be more severe in attention sub- layers and it is possible to reconstruct reliable phylogenetic tree of languages using the language- specific sub-networks generated by our approach. All the experiments confirm the effectiveness of our approach and the need for better training strategies to improve MNMT performance.
Training
In this work, we aim to search for sub-networt for each language pair within the multilingual model to mitigate the parameter interference is- sue and improve the performance. While our ap- proach avoids introducing additional parameters, the 3 3-phase training process (training base model, scarching for sub-networks, and joint training) introduces additional complexity the training pipeline compared the standard end-to-end multi- lingual model training, and demands computational resources for the sub-network searching phase.

---

References
Kevin Clark, Urvashi Khandelwal. Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Association for Com- putational Linguistics.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. CoRR, 0ts//997.050011
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multi- ple language translation. In Proceedings of the S3rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence Natural Language Processing (Volume 1: Long Papers), pages 1723-1732, Beijing, China. As- sociation for Computational Linguisties.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. CoRR, abs/ htts/190..000 90098.877.
Ankur Bapna and Orhan Firat. 2019. Simple, scal- able adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 1538- 1548, Hong Kong, China. Association for Computa- tional Linguistics.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Thanh-Le Ha, Jan Niehues, and Alex Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. In Proceedings of the 13th International Conference on Spoken Language Translation, Seattle, Washington D.C. International Workshop on Spoken Language Translation.
Loic Barrault, Magdalena Biesialska, Ondfej Bo- jar, Marta R. Costa-jussă, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Had- dow, Mathhis Huck, Eric Joanis, Tom Koemi, Philipp Koehn, Chi-kiu Lo, Nikola J.jubeKić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of the Fifth Conference Machine Translation, pages 1-55, Online. Association for Computational Linguis- tics.
Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings.
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances Neural In- formation Processing Systems, volume 28. Curran Associates, Inc.
Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. 2022. Multilingual machine transla- tion with hyper-adapters.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computa- tion. CoRR, 0bs/1308.34333
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's s multilingual neural machine translation system: En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351.
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. 2018. Multilingual neural machine translation with task-specific attention. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 31 12-3122, Santa Fe, New Mexico, USA. Association for Computational Linguisties.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Alix Boc, Alpha Boubacar Diallo, and Vladimir Makarenkov. 2012. T-rex: a web server for inferring, validating and visualizing phylogenetic trees and networks. Nucleic acids research, 40 W5533 W579.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch Marcello Federico, Nicola Bertoldi Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondfej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine tramslation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions,
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign. In Proceed- ings of the 1 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2-17, Lake Tahoe, California.

---

pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Maja Popović. 2015. chrF: character n n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece:
A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations. pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Learning language specific sub-network for
Taido Purason and Andre Tättar. 2022. Multilingual neural machine translation with the right amount of sharing. In Proceedings of the 23rd Annual Con- ference of the European Association for Machine Translation, pages 91-100, Ghent, Belgium. Furo- pean Association for Machine Translation.
multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 293 305, Online. Association for Computational Linguisties.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019. Rethinking the value of network pruning. In ICLR.
Vladimir Makarenkov and Bruno Leclere. 1999. An algorithm for the fitting of tree metric according to a weighted least-squares criterion. Journal of classi- fication, 16(1):3-20 -66
Devendra Sachan and Graham Neubig. 2018. Parame- ter sharing methods for multilingual self attentonali translation models. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pages 261-271, Brussels, Belgium. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine tramslation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Sys- tems, volume 33, pages 20378 20389. Curran Asso- ciates, Inc.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Minh-Quang Pham, François Yvon, and Josep Crego. 2022. Latent group dropout for multilingual and multidomain machine translation. In Findings of the A Association for Computational Linguistics: NAACL 2022, pages pages 2469-2481, Seattle, United States. Asso- ciation for Computational Linguistics.
Uri Shaham, Maha Elbayad. Vedanuj Goswami, Omer Levy, and Shruti Bhosale. 2023. Causes and cures for interference in multilingual translation. In Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 15849-15863, Toronto, Canada, Association for Computational Linguistics.
Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465-4470, Online. Association for Computational Linguistics.
Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 8936-8943.
Ilya Sutskever, Oriol Vinyals, and Quoe V Lee 2014. Se- quence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Telmo Pessoa Pires, Robin M. Schmidt, Yi-Hsiu Liao, and Stephan Peitz. 2023. Learning language-specinn layers for multiinnuua machine translation.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz
A.I IWSLT dataset
Table 4 provides detailed information about the IWLST data.
Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
ISO
L.anguage
Family
Script
Size (K)
Qian Wang and Jiajun Zhang. 2022. Parameter differen- tiation based multilingual neural machine translation. Proceedings of the AAAI Conference Artificial Intelligence, 36(10): 100011144--1444
Farsi
Iramian
Arabic
Arabic
Arabic
Arabic
140
he Hehrew
Semitie
Hebrew
Zirui Wang. Zachary C. Lipton, and Yulia Tsvetkow. 2020. On negative interference in multilingual mod- els: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 4438-4450, Online. Association for Computa- tional Linguistics.
Italian
Romance
Latin
167
es
Spanish
Romance
Latin
169
pl
Polish Slavic
Latin
128
nl Dutch
Germanic
Latin
153
de German
Germanic
Latin
160
Felix Wu, Angela Fan, Alexci Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. Interna- tional Conference on Learning Representations.
Table 4: Statistics IWSLT data with similar languages organized into groups.
A.2 WMT dataset
Wanying Xie, Yang Feng, Shuhao Gu, and Dong Yu. 2021. Importance- based neuron allocation for multi- lingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5725-5737, Online. Association for Computational Linguistics.
Table 5 provides detailed information about the WMT data.
B Model details
In this section, we provide detailed model informa- tion in our experiments.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share not? learning to schedule language-specific capacity for multilingual transla- tion. In International Conference on Learning Rep- resentations.
IWSLT. Given the small scale of the data in the IWSLT dataset, we adopt the Transformer-small architecture with 4 attention heads: L 6, d 512,
Thead 4 and dg = 1024.
WMT. For the WMT experiment, we adopt a Transformer-base architecture with attention heads: L= 6, 512, Theea 8 and dg 2048.
Biao Zhang, Philip Williams, Tvan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639, Online. Association for Computational Linguis- tics.
C Training details
As shown in section 3, our approach includes three phases: training a multilingual base model (Phase 1), identifying sub-networks through prun- ing (Phase 2), and joint training (Phase 3). In this section, we provide the details of the hyperparame- ters of these 3 phases in our experiments.
Michael Zhu and Suyog Gupta. 2017. prune, or not to prune: exploring the efficacy of pruning for model compression.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingx- Wang, and Lei Li. 2021. Counter-interference adapter multilingual machine wanslation. Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 2812-2823 Punta Cana. Dominican Republic. Association for Computational Linguistics.
C.1 IWSLT
In Phase 1, we train the multilingual base model with the same set of hyper-parameters as in Lin et al. (2021). More specifically, optimize parameters with Adam (Kingma and Ba, 2015) 0.9, /32 0.98), learning rate schedule of (5e-4,4k), dropout of 0.1 and label smoothing of 0. The max tokens per batch is set to 262144. The maximum update number is set to 160K with a checkpoint saved
A Datasets details
this section, we provide detailed data informa- tion our experiments.

---

ISO
Language
Family
Script
Train
Valid
Test
Size
gu
Gujarati
Indo-Aryan
Gujarati
WMT19
newsdev19
newstest19
7.5K
ta
Tamil
Dravidian
Tamil
WMT20
newsdev20
newstest20
605K
kk
Kazakh
Turkic
Cyrillic
WMT19
newsdev19
newstest19
102K
tr
Turkish
Turkic
Latin
WMT16
ncwsdev16
newstest16
198K
Romanian
Romance
Latin
WMT16
ncwsdev16
newstest16
596K
es
Spanish
Romance
Latin
WMT13
newstest12
newstest13
12.9M
fr
French
Romance
Latin
WMT14
newstest13
newstest
37M
ps
Pashto
Iranian
Arabic
WMT20
wikipedia
newstest20
IM
Farsi
Uralic
Latin
WMT16
newstest15
newstest16
2M
Iv
Latvian
Baltic
Latin
WMT17
newsdev17
newstest17
2.1M
et
Estonian
Uralic
Latin
WMT18
newsdev18
newstest18
2.IM
Lithuanian
Baltic
Latin
WMT19
newsdev19
newstest19
3.4M
Russian
Slavic
Cyrillie
WMT16
newstest
newstest16
1.4M
CS
Czech
Slavic
Latin
WMT14
newstest13
newstest14
11M
pl
Polish
Slavic
Latin
WMT20
newsdov20
newstest20
11M
ja
Japanese
Japonic
Kanji; Kana
WMT20
newsdev20
newstest20
16.5M
zh
Chinese
Chinese
Chinese
WMT17
newsdev17
newstest17
11.9M
de
German
Germanic
Latin
WMT16
newstest13
newstest14
4.4M
Table 5: Statistics of WMT data with similar languages organized into groups.
Lang
Gu 7.5K
Kk
Ir
Ro
Ta 605K
Average
Size
102K
198K
596K
En
translation
Baseline
17.8
14.1
13.34
MagnitudeOneshot
+0.1
+1.3
+1.7
+0.8
+3.1
+1.40
GradientGradual (Ours)
0.22
+1.4
+1.7
+1.9
+6.0
+2.16
-> En translation
Baseline
0.3
9.5
21.3
26.6
15.7
14.68
MagnitudeOneshot
+0.4
-0.5
- 0.4
+0,2
+0.3
0.00
GradientGradual (Ours)
0.0
-25
0.2
+0.8
+0.2
-0.34
Table BLEU score over baseline each pair the low-resource group (< IM) on the WMT dataset.
every 500 updates, and the patience for early stop training is set to 30. In Phase 2, we set the max tokens to 16384, and dropout 0.3. The training steps of 3 stages set to 4K, 36K, and 40K. The best performance of the final model is achieved with a pruning ratio of settings are as same as Phase In Phase 3, we keep the same settings as Phase 1, except we apply masks on the model.
number is set to 600K with a checkpoint saved every 1K updates, and the patience for early stop training is set 30. In Phase 2, the max tokens per batch set 20K, 40K, 80K, and 160K for languages with training data sizes >00K, >10KK, >IM, and >0MM. The training steps of stages are set to 4K, 16K, and 20K. The best performance the final model achieved with pruning ratio of 0.2 in this phase. In Phase 3, we keep the same settings as Phase 1, except we apply masks on the model.
this phase. The other
C.2 WMT
In Phase 1, the parameters al. (2021). We train the multilingual base model
same in Lin
D Additional evaluation results
D.1 Detailed BLEU scores WMT dataset
with Adam (Kingma and Ba. 2015) (31 0.9, 82 0.98), a learning rate schedule of (5e-4,4k), dropout 0.1 and label smoothing 0.1. The max tokens per batch set to 524288. The maximum update
Considering the relatively extensive number of lan- guages involved in the WMT dataset, we present av- erage scores for low-, medium-, and high-resource

---

Lang
Ps
Fi
Lv
Ft
De 4.4M
Average
Size
IM
2M
2.1M
2.1M
2.3M
3.4M
En
translation
Baseline
4.1
14.7
14.1
15.1
22.2
8.6
19.9
14,10
MagnitudeOnessot
+0.4
+1.8
+1.1
+1.3
+2.4
+0.1
+2.0
+1.30
GradientGradual (Ours)
+2.6
+2.3
+2.0
+2.3
+3.5
0.0
+2.9
+2.23
+ En translation
Baseline
9.8
22.1
18.8
22.4
28.7
17.1
27.9
20.97
MagnitudeOneshot
0.0
+0.4
+0.4
+0.5
+0.5
+0.3
+0.7
+0.40
GradientGradual (Ours)
+1.1
+0.7
+0.6
+1.2
+1.1
+0.3
+1.3
+0.90
Table 7: BLEU score over baseline of each pair in the medium-resoure group (IM-10M) on the WMT dataset.
Lang
Pl
Zh
Es
Fr 37M
Average
Size
11M
11M
11.9M
12.9M
16.5M
En + X translation
Baseline
19.4
15.9
22.7
29.4
20.0
32.6
23.33
MagnitudeOneshot
+1.9
+1.6
+0.7
+1.1
+1.2
+1.8
+1.38
GradientGradual (Ours)
+2.7
+1.6
+3.0
+2.0
+2.3
+2.50
+2.35
X - En translation
Baseline
28.7
24.8
15.5
31.4
15.8
33.7
24.98
MagnitudeOneshot
0.5
+0.8
+1.7
+0.4
+0.8
+0.4
+0.60
GradientGradual (Ours)
-0.2
+0.7
+2.5
+1.1
+1.2
+0.6
+0.98
Table 8: BLEU score over baseline of each pair in the high-resource group (>10M) on the WMT dataset.
Lang
Low
Med
High
All
Model
COMET
chrF
WR (%)
60
86
83
78
Baseline
0.773
51.2
Adapter.128
0.785
52.5
Table 9: Average win ratios of Low (< 1M), Medium (2 1M and 10M) and High (210M) resource lan- guage pairs of our Gradientrrduual approach the Onsshtt approach
Adapter.256
0.785
52.2
Adapter.512
0.785
52.3
SepaDec
0.774
51.6
MagnitudeOneshot
0.775
51.5
groups to offer an overview of performance in dif- ferent data situations Table For the detailed results, we provide BLEU scores of languages in low, medium, and high resource groups in Table 6, Table 7, and Table 8, respectively.
GradientGradual (Ours)
Table 10: Average COMET chrF scores across language pairs of various approaches the IWSLT dataset.
D.2 Win ratio results
D.3 COMET and chrF scores
To gain further insights into the performance of our approach, we present the results of win ratio (WR) based on tokenized BLEU, which denotes the per- centage of languages where our approach outper- forms the other. We choose the MagninudeOneshot (LaSS) as a strong baseline to compare with. Table shows that our GradientGradual approach out- performs MagnitudeOneshot across all languages on the IWSLT dataset, i.e., the win ratio is 100%. Table 9 presents the win ratio (WR) results on the imbalanced WMT dataset. The results show that our approach achieves win ratios of 60%. 86% and 83% on low-, medium-, and high-resource lan- guages, further verifying the superiority of our ap- proach across different data sizes and especially on medium-, and high-resource languages.
We present the averaged COMET and chrl scores of all language pairs the IWLST dataset in Ta- ble 10 and the WMT dataset in Table 11. More specifically, on the IWLST dataset, we report the scores for the multilingual baseline and various existing approaches proposed in prior works, i.e., the Adapter-based approach with different bottle- neck dimensions: Adapter. 1 128, Adapter:256, and Adapter:512: SepaDec, MagninueeOneshot, as well as GradientGradual method. On the WMT dataset, show the scores of the multilingual baseline, Magnindeenesshot, and our approach. On both the IWISS and the WMT datasets, our method outperforms other approaches on COMET and chrF metrics.

---

Model
COMET
chrF
Model
Ratio
Speed GPU
Speed CPU
Buseline
0.733
44.4
Baseline
237.56 + 2.70
59.98 + 2.73
MagniuueOneeshtt
0.747
45.4
GradientGradual 0.1
235.23 a 3.65
59.57 :: 0.77
GradientGradual (Ours)
0.755
45.9
GradientGradual 0.2
233.93 + 3.28
58.81 + 4.27
GradientGrudual 0.3
235.71 # 1.92
58.26 + 3.34
Table 11: Average COMET and chrF scores across all language pairs of various approaches on the WMT dataset.
GradientGradual 0.4
2
58.04 + 2.93
GradienGradual 0.5
235.94 2.08
59.37 = 1.73
GradienGradual 0.6
238.76
58.94 + 3.28
Model
BL.UU
[1]]
[aul
Disk storage
GradienGradual 0.7
240.10 # 2.62
60.17 + 2.70
Baseline
26.74
78M
78M
298M
GradientGradual 0.8
242.55 + 2.47
60.58 2.37
Ad
28.07
79M
103M
397M
Gradien/Grudual 0.9
243,23
61.56 + 3.99
Adapter:256
28.08
8IM
128M
493M
Table 13: Tokens/second comparison of our approach against the baseline on the IWSLT De->En test set.
12
28.28
84M
179M
685M
SepaDec
27.48
78M
478M
1.8G
MagnindeOneshot
27.24
78M
78M
497M
age for storing indices. However, the most signifi- cant disk storage requirement is from SepaDec due to the largest number of total parameters.
GradientGradual 28.80
78M
78M
497M
Table 12: Comparison of performance, parameter count, and disk storage among different approaches.
Inference speed. Adapter-based approaches in- troduce additional parameters during inference. A larger bottleneck dimension of the adapter leads to greater expansion of parameters, consequently causing a more considerable inference speed de- crease. SepuDec, on the other hand, maintains the same number of parameters as the baseline during inference and, therefore, has insignificant to no im- pact on inference speed. Below, we demonstrate that our Gradien(Gradual method, has negligible effect inference speed with detailed GPU and CPU speed results.
E Additional metrics
In this section, we compare our method with var- ious approaches from the perspectives of total model parameter counts (10au)), parameter counts of individual languages ((Ot)) during inference, disk storage, inference speed, and the average BLEU scores of all language pairs as performance. We present the primary results in Table 12 and detailed inference speed results of our method in Table 13, The results are obtained with the Transformer-sman architecture on the IWS dataset.
In our approach, the sub-network mask matrices are represented as binary matrices and are obtained during training and directly applied to Attention and Feedforward sub-layers during inference. Com- pared to the compute-intensive multiplication be- tween dense weight matrices, multiplying binary mask matrix with weight matrix on some sub- layers introduces minimal overhead and, there- fore, has a negligible impact on inference speed. In addition, zero elements in the masked weight matrix could result in faster inference speed due to the possibility of avoiding unnecessary arithmetic operations. We report tokens/second on the IWSLT De-=En test set in Table 13. The batch size is al- ways set to and the result is averaged over runs. Speed GPU is measured using a single NVIDIA 100 GPU and Speed CPU is measured using a single-threaded Intel(R) Xcon(R) Gold 6330 CPU 2.00GHz.
Parameter count. As shown in Table 12, adapter- based approaches increase both the total model parameter counts and the parameter counts of indi- vidual languages. SepaDec keeps parameter counts of individual languages constant but leads to the most significant total model parameter count due separate decoders. In contrast, without intro- ducing additional parameters, MagninudeOneshor and our GradientGradual approach maintain the same parameter counts for both the overall models and individual languages as the multilingual NMT baseline model.
Disk requirement. Similar to the parameter count analysis, adapter-based approaches cause different levels of disk storage increase for added parameters depending on the bottleneck dimensions. Magni- OndeOne and our GradientGradual approach also require a moderate amount of extra disk stor-
Thhe analysis also applies to MagnitudeOneshot.

---

While the best performance of our approach is achieved with pruning ratio set to 0.6, as shown in Figure 1, we provide the inference speed results with the pruning ratio ranging from 0.1 to 0.9 to offer a more comprehensive analysis of inference speed. Although Speed GPU and CPU are differ- ent in absolute value. they show a similar pattern across different pruning ratios, and we take the Speed GPU as example analyze. The results indicate an improvement in the inference speed of our approach when compared with the multilingual NMT bascline within the pruning ratio range of 0.6 to 0.9, and the highest inference speed is obtained when pruning ratio is set to 0.9. We attribute this speed improvement to the avoidance of unneces- sary operations with element O. Additionally, the inference speed is slightly slower than the base- line, with pruning ratio smaller than 0.6, but not statistically significantly different.
62
62.5
19.0
57.9
56.5
55.0
61
62.5
59.3
59.3
58.2
56.6
552
58.9
59.1
5933
61.4
57.7
55.7
54.9
55.9
59
$9.0
55.0
558
56.5
56.6
55.7
55.9
55.1
58.1
56.3
-05
1a
55.0
552 54.9 15.1
53.9
58.1
56.0
-54
fu
56.8
56.9
55.9 56.0
55.1
58.3
56.0
05
de
ar
Figure 4: Similarity scores (%) of X-+En language pairs. The scores are represented on both the x-axis and y-axis.
Fa
Ar
Performance. Our approach outperforms SepaDe by 1.53 BLEU scores and surpasses the Adapter- based approaches with 128, 256, and 512 bottle- neck dimensions by 0.72, 0.73, and 0.51 BLEU scores, respectively. Additionally, compared to MagnitadeOneshot, the work most similar ours, our approach outperforms by 1.53 BLEU scores.
He
De
NI
It
Es
Pl
Based on the comprehensive analysis above, our approach is particularly advantageous when priori- tizing performance and inference speed is crucial, and some additional disk requirements consid- ered acceptable.
Figure Phylogenetic Tree built with X-=En distance scores obtained in our method. The built tree shows strong positive correlation with language familics. It. Es, De, NI, and Pl are European languages written in Latin; With It, and Es in the Romance branch; De and NI the Germanic branch. Fa, Ar, and He are similar Afro-Asiatis languages; with Fa and Ar written
F Similarity Scores and Phylogenetic Tree of X- language pairs
Table 14 reports detailed language family infor- mation for the languages in the IWSLT dataset. In particular, it includes the cluster, branch, and script of each language. Languages belonging to the same cluster and branch are expected to be lin- guistically closer to each other and have relatively high similarity scores.
European language, and Farsi (fa), an Afro-Asiatio language. Figure 5 shows the corresponding phylo- genetic tree obtained with distance scores.
G Gradient-based pruning criterion
In this paper, we apply gradient-based pruning to MNMT. This approach uses gradient-based in- formation to identify the language-specifee sub- networks. More specifically, to identify which weights to prune in a given weight matrix W, scoring matrix S and a binary mask matrix M are introduced in association with the weight matrix. Each parameter in the score matrix is intended capture and learn the importance of the correspond- ing weight, and each element in the binary mask is assigned value of either 0 or I according to whether the corresponding weight is pruned or
Figure 4 shows the similarity scores of X->En language pairs. The results demonstrate that the similarity scores of languages belonging to the same cluster are relatively high. Italian (It) and Spanish (Es) obtained the highest score, followed by German (De) and Dutch (NI). The third highest score is obtained by Arabic (Ar) and Hebrew (He). Besides, the similarity scores of languages between two distinct language clusters are relatively low, with the lowest score obtained by Dutch (NI), a

---

ISO
Language
Cluster
Branch
Script
be calculated as in Eq. (5) and (6), respectively.
fa
Farsi
Afro-Asiate
Iranian
Arabic
at. aL da: Ot
ar
Arabic
Aro-Asiatic
Semitic
Arabic
0Sk.j da, SSi.j
da;
W
Wija
(5)
he Hebrew
Afro-Asiatic
Semitie
Hehrew
it Italian
Earopean
Romance
Latin
AL aL da: OL Maj
es
Spanish
European
Romance
Latin
(6)
Da. aWij da:
pl
Polish
European
Slavic
Latin
al
Dutch
European
Germanic
Latin
From Eq. (6), we derive OL aL
da: aW., M..j By
de German
European
Germanic
Lain
omitting the binary mask term Mij as in Sanh et al. (2020), we obtain aL at 1 Inserting 8e, aw., 2j" into Eq. (5) yields ast
Table 14: Language clustering of languages in IWSLT
the obtained result of @L da, ar. Wiz..... Therefore, the gradient of L aw,, 2
aL
retained. Weights with relatively low scores in the score matrix are considered less important and assigned value of in the binary mask matrix. Score parameters are learned and updated itera- tively during the training process. The scores of all the weights, both pruned and retained weights, are updated. The updating of scores change the relative importance of different weights and affect their score distribution. This process enables the model to self-correct by allowing pruned weights to come back.
with respect to St.j can be represented as;j aw,; Wa.- The importance score Sij after gradient updates can be represented as:
AL
$ = 0i Σ
aL
W!
aWiJ
t<T
where 7 denotes the number of gradient updates, ag is the learning rate during training process. In our method, specific percentage of weights is pruned based on the distribution of importance score values, regardless of the absolute score val- ues. The learning rate Oj, which remains constant across all score parameters, does not impact the distribution and can be disregarded for simplicity without affecting the pruning outcome, as shown in Eq. (2).
In the following demonstration, we will show how the learned scores are based on gradient infor- mation, as depicted in Eq. (2). In the forward pass of the training process, the masking step, where the output is if the input (in this context, the score) is above threshold and otherwise, is performed after the linear operation. The output of the linear operation and masking can be calculated as a; Ek W.,.Mr, During backpropagation, the gradients learnable parameters are computed to update these parameters and facilitate the learning process. However, the masking step, introduces non-differentiable behavior at the threshold point. Besides, the constant output of or 0 results in gradient of 0 everywhere is defined. This can lead the so-called "vanishing gradient" issue, which arises when the gradients become very small or vanish at some point during backpropagation. As a result, the flow useful gradient information is hindered, making it difficult to train the model effectively. Thanks to Bengio et al. (2013), we mitigate this issue by employing straight- throughh estimator. More specifically, during backpropaga- tion, the masking step ignored and the gradient after the masking step flows "straight-throughh" to the step before the masking step. As a result, the gradient of loss L with respect to S.y and W.y can


Reviews:
[{"review_id": "PxDYjdDaYL", "reviewer": "Reviewer_ro9B", "summary": "", "questions": "- could the authors add another baseline which uses bilingual models? i.e., training a bilingual model for each language pair. This baseline can be used to answer how much the proposed approach can fix the parameter interference problem for high-resource language pairs, e.g., ideally the proposed approach should outperform the bilingual baselines on all the language pairs.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ro9B, \n\nWe are extremely grateful for your valuable feedback on our paper, and we will address each of your concerns and questions in detail below. \n\n>1. the proposed approach has multiple steps which may be hard to be used or followed by future work;\n>\nThank you for your input on the training processing. In this work, we aim to explore the most effective approach for mitigating the parameter interference issue in the multilingual NMT model and improving the model performance. We acknowledge that the three-phase training process might introduce \nadditional complexity to the training pipeline compared to the standard MNMT training. Moving forward, we plan to explore the possibility of making the training process end-to-end in our future work.\n\n\n>2. the general direction is not novel, i.e., extract sub-layers for bilingual language pairs;\n>\n Thanks for sharing your perspective on the novelty of our work. We understand that extracting sub-networks for bilingual language pairs is not a new concept. However, the critical challenge of pinpointing essential parameters for each language pair remains relatively underexplored. \n\nIn addition, although pruning is a commonly employed technique in machine learning, there is a significant lack, or complete absence, of exploration regarding effective pruning in multilingual neural machine translation. The most widely used pruning criterion and schedule can be ineffective for this specific task. For instance, we observed a high overlap of retained weights across different language pairs when utilizing magnitude pruning for sub-network extraction. Consequently, the distinct and conflicting requirements on the extensively shared parameters of various language pairs persist, thus limiting the effectiveness of parameter interference mitigation. \n\nBy researching pruning criteria and schedules within multilingual neural machine translation, we aspire to shed light on exploring the most effective approach in mitigating parameter interference across languages while not introducing additional parameters to the model for future MNMT research work.\n\n>3. could the authors add another baseline which uses bilingual models? i.e., training a bilingual model for each language pair. This baseline can be used to answer how much the proposed approach can fix the parameter interference problem for high-resource language pairs, e.g., ideally the proposed approach should outperform the bilingual baselines on all the language pairs.\n>\n\nThank you for your suggestion. Following your advice, we conducted experiments using bilingual models, and we hope this can offer insights into the performance comparison of utilizing a bilingual model against our method with the multilingual model.\nDue to the limited time for the author response, we selected one language from each of the low-, medium-, and high-resource groups to conduct bilingual experiments. The sizes of our chosen languages fall in the middle range of each respective group’s size spectrum. We conducted bilingual experiments in En->X and X->En translation directions and present the average En<->X BLEU scores for Turkish (Tr), Estonian(Et), and Chinese(Zh) below.\n\nTable 1: The average BLEU scores of En <-> Tr (198K), En <-> Et (2.1 M), and  En <-> Zh (17.8M).\n| Lang | Tr |  Et  | Zh | \n|------------------------ |------------|----------|----------|\n| Bilingual                          | 11.9 | 18.55 | **21.9** |\n| MNMT baseline              | 19.55 | 18.75 |19.1 |\n| LaSS                                | 20.2 | 19.65 | 20.3| \n| GradientGradual (Ours) | **20.3** | **20.5** | 21.85 | \n\n**MNMT baseline versus Bilingual:** The results show that the MNMT baseline significantly outperforms the bilingual model in the low-resource language by 7.65 (19.55 vs. 11.9) BLEU scores, achieves similar performance to the bilingual model (18.75 vs. 18.55) in the medium-resource language and experiences a considerable decrease of 2.8 (19.1 vs. 21.9) BLEU scores in the high-resource language. This observation aligns with our experience that while low-resource languages often benefit from knowledge transfer in MNMT, high-resource languages can suffer from performance degradation due to the lack of language-specific parameters and the issue of parameter interference resulting from full parameter sharing in MNMT.\n\n**LaSS and GradientGradual (Ours)  versus  Bilingual:** In the low-resource language, both LaSS and our approach significantly outperform the bilingual model, with 8.3 (20.2 vs. 11.9) BLEU scores and 8.4 (20.3 vs. 11.9) BLEU scores, separately. In the medium-resource language,  while LaSS exhibits a performance gain of 1.1 (19.65 vs. 18.55) BLEU scores,  our approach achieves a larger improvement of 1.95 (20.5 vs. 18.55) BLEU scores. Notably, LaSS underperforms the baseline in the high-resource language by a notable margin of 1.6 (20.3 vs. 21.9)  BLEU scores. In contrast, our approach achieves similar performance to the bilingual model (21.85 vs. 21.9) in the high-resource language. \n \n**Summarization:** Our method shows a larger improvement over the Bilingual models than the MNMT baseline and LaSS in low- and medium-resource languages. Furthermore, while the MNMT baseline and LaSS experience performance degradation compared to the Bilingual models in the high-resource language, our approach demonstrates almost the same performance (21.85 vs. 21.9), alleviating the typical problem of performance degradation for high-resource languages in the MNMT model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the improvements are promising;\n- the paper is well written;", "reasons_to_reject": "- the proposed approach has multiple steps which may be hard to be used or followed by future work;\n- the general direction is not novel, i.e., extract sub-layers for bilingual language pairs;", "questions_to_authors": "- could the authors add another baseline which uses bilingual models? i.e., training a bilingual model for each language pair. This baseline can be used to answer how much the proposed approach can fix the parameter interference problem for high-resource language pairs, e.g., ideally the proposed approach should outperform the bilingual baselines on all the language pairs.", "paper_topic_and_main_contributions": "The paper tries to improve multilingual NMT models by addressing the parameter interference problem using a  gradient-based gradual pruning approach.\n\nMain contributions:\n- significant improvements in BLEU;\n- ablation study on pruning criteria and schedules;\n- ablation study on  the contribution of various sub-layers;", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ruPDtNd2tT", "reviewer": "Reviewer_KqQ9", "summary": "", "questions": "Could you provide the detailed results for every language pair on IWLST and WMT? Table 1 provides only the average en<->x BLEU score. Table 3  doesn't even give the BLEU score for every language pair.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KqQ9,\n\nWe highly appreciate your feedback on our paper, and we address your concerns and questions below.\n\n>1. The novelty is limited. This work is similar with the previous work (Lin et al,2021). Please provide the detailed results.\n>\n\n**Regarding novelty:** Thanks for your feedback and your insight is highly appreciated.\nRegarding novelty, as noticed by reviewer 29ZV, our approach is unique in exploring the effectiveness of various pruning criteria and schedules within the context of multilingual neural machine translation. This aspect has received limited attention in LaSS (Lin et al. 2021) and remains entirely unexplored in other prior work. We kindly invite you to check our detailed explanation of a similar question posed by the reviewer ro9B.\n\n**Regarding the detailed results:** Thank you for your careful review and valuable feedback. Considering the relatively extensive number of languages involved in the wmt dataset, we have adopted a strategy similar to several prior research studies by presenting average scores for low-, mid-, and high-resource groups. This way, it offers an overview of performance in different data situations. However, your feedback is greatly appreciated, and we accordingly provide detailed scores below. We first report the averaged performance results of En->X and X->En for low- medium- and high-resourced languages  (Tables 1, 4, and 7) and then present the performance splitting En->X and X->En in all the data conditions (Tables 2, 3, 5, 6, 8, and 9). The results confirm the findings we report in the paper, where the gain over the baseline and LaSS improve by increasing the data conditions. Interestingly, we also observed that the major gains our approach and LaSS have over the baseline are in the En->X directions. This suggests that the multilingual decoder benefits more from the application of subnetworks. This aspect has not been noticed in previous papers. \n \n\n(1) Low-resource languages: Gu (7.5 K), Kk (102 K), Tr (198 K), Ro (596K) and Ta (605 K)\n\nTable 1: The average En <-> X performance for low-resource languages:\n|Lang                           | Gu |  Kk  | Tr | Ro |  Ta  | average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|\n| Baseline                          | 0.75| 6.00 | 19.55|28.85|14.90|14.01|\n|MagnitudeOneshot            | 0.25| 0.40 | 0.65  |0.50  |1.70| 0.70|\n|GradientGradual (Ours) | -0.10| -0.55 | 0.75  |1.35  |3.10| **0.91**|\n\n\nTable 2: En->X performance for low-resourced languages:\n|Lang                           | Gu |  Kk  | Tr | Ro |  Ta  | average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|\n|Baseline|1.20|2.50|17.80|31.10|14.10|13.34|\n|MagnitudeOneshot|0.10|1.30|1.70|0.80|3.10|1.40|\n|GradientGradual (Ours)(ours)|-0.20|1.40|1.70|1.90|6.00|**2.16**|\n\nTable 3: X->En performance for low-resourced languages:\n| Lang                          | Gu |  Kk  | Tr | Ro |  Ta  | average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|\n|Baseline|0.30|9.50|21.30|26.60|15.70|14.68|\n|MagnitudeOneshot|0.40|-0.50|-0.40|0.20|0.30|**0.00**|\n|GradientGradual (Ours)|0.00|-2.50|-0.20|0.80|0.20|-0.34|\n\n(2) Medium-resource languages: Ps(1M), Fi(2M), Lv(2.1M), Et(2.1M), Ru(2.3M), Lt(3.4M) and De(4.4M)\n\nTable 4: The average En <-> performance for medium-resource languages:\n| lang                     |Ps|Fi|Lv|Et|Ru|Lt|De|average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|-------|-------|\n|Baseline|6.95|18.40|16.45|18.75|25.45|12.85|23.90|17.54|\n|MagnitudeOneshot|0.20|1.10|0.75|0.90|1.45|0.20|1.35|0.85|\n|GradientGradual (Ours)|1.85|1.50|1.30|1.75|2.30|0.15|2.10|**1.56**|\n\nTable 5: The En->X  performance for medium-resource languages:\n| lang                     |Ps|Fi|Lv|Et|Ru|Lt|De|average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|-------|-------|\n| Baseline|4.10|14.70|14.10|15.10|22.20|8.60|19.90|14.10|\n| MagnitudeOneshot|0.40|1.80|1.10|1.30|2.40|0.10|2.00|1.30|\n| GradientGradual (Ours)|2.60|2.30|2.00|2.30|3.50|0.00|2.90|**2.23**|\n\n\nTable 6: The X->En performance for medium-resource languages:\n| lang                     |Ps|Fi|Lv|Et|Ru|Lt|De|average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|-------|-------|\n|Baseline|9.80|22.10|18.80|22.40|28.70|17.10|27.90|20.97|\n|MagnitudeOneshot|0.00|0.40|0.40|0.50|0.50|0.30|0.70|0.40|\n|GradientGradual (Ours)|1.10|0.70|0.60|1.20|1.10|0.30|1.30|**0.90**|\n\n(3) High-resource languages: Cs(1M), Pl(2M), Es(2.1M), Ja(2.1M), Zh(2.3M) and Fr(3.4M)\n\nTable 7: The average En <-> X performance for high-resource languages:\n| lang                     |Cs|Pl|Es|Ja|Zh|Fr|average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|-------|\n|Baseline|24.05|20.35|30.40|17.90|19.10|33.15|24.16|\n|MagnitudeOneshot|0.70|1.20|0.75|1.00|1.20|1.10|0.99|\n|GradientGradual (Ours)|1.25|1.15|1.55|1.75|2.75|1.55|**1.67**|\n\nTable 8: The En->X  performance for high-resource languages:\n| lang                     |Cs|Pl|Es|Ja|Zh|Fr|average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|-------|\n|Baseline|19.40|15.90|29.40|20.00|22.70|32.60|23.33|\n|MagnitudeOneshot|1.90|1.60|1.10|1.20|0.70|1.80|1.38|\n|GradientGradual (Ours)|2.70|1.60|2.00|2.30|3.00|2.50|2.35|\n\nTable 9: The X->En performance for high-resource languages:\n| lang                     |Cs|Pl|Es|Ja|Zh|Fr|average|\n|--------------------------------|------------  |----------|----------|----------|----------|----------|-------|\n|Baseline|28.70|24.80|31.40|15.80|15.50|33.70|24.98|\n|MagnitudeOneshot|-0.50|0.80|0.40|0.80|1.70|0.40|0.60|\n|GradientGradual (Ours)|-0.20|0.70|1.10|1.20|2.50|0.60|**0.98**|\n\n\n\n\n>2. Could you provide the detailed results for every language pair on IWLST and WMT? Table 1 provides only the average en<->x BLEU score. Table 3 doesn't even give the BLEU score for every language pair.\n>\nThank you for your feedback. We provide the detailed scores in the above section and will include them in the appendix section of our paper soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written and easy to follow.\nThis paper provided the analysis to show the impact of four pruning methods in the evaluation of model performance.", "reasons_to_reject": "The novelty is limited. This work is similar with the previous work (Lin et al,2021).\nPlease provide the detailed results.\n", "questions_to_authors": "Could you provide the detailed results for every language pair on IWLST and WMT? Table 1 provides only the average en<->x BLEU score. Table 3  doesn't even give the BLEU score for every language pair.", "paper_topic_and_main_contributions": "Due to parameter interference when parameters are fully shared across all language pairs, the traditional multilingual neural machine translation often suffers from performance degradation in high-resource languages compared to bilingual counterparts. To tackle this issue, they propose a gradient-based gradual pruning technique for MNMT. Compared with the previous similar work (Lin et al,2021), they use gradient-based scores instead of magnitude scores as  pruning criterion and use gradual pruning instead of one-shot pruning as pruning schedule. Experiments on IWSLT and WMT datasets show the effectiveness of this method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SEfgXvut4O", "reviewer": "Reviewer_29ZV", "summary": "", "questions": "**Question A**: The additional sub-network search procedure e.g. Phase 2 + 3 imposes computational overhead during training that could be used differently e.g. training the baselines for longer. I'm curious how that will affect the comparison?\n\n**Question B**: Will code be released for the experimental setup? Specifically, Phase 2 and Phase 3 would be interesting to the community I believe", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 29ZV,\n\nThank you for your careful review of our paper. We greatly appreciate your feedback and address your concerns and questions below.\n\n>**1.  [large]: Besides LaSS the presented method is not compared to any other method that has been proposed for MNMT. I'd appreciate a more thorough comparison to existing literature. Specifically, I'd recommend to compare against LSL-NAS + dense pre-training from [1], X-MOD from [2], or Adapter-based approaches, similar to what is done in [1]. All of these share similar efficiency advantages to what is presented here since they make use of sparsity.**\n>\n\nThank you for your constructive insights. The first paper you mentioned from ACL 2023 is intriguing. However, it became publicly available on ACL around July, whereas we submitted our paper in June. We will soon expand our related work section to mention it and also the second paper you pointed out.\n\nHowever, in this study, we aim to prevent the inclusion of any trainable parameters in the model and avoid inference speed slowdown. Thus, we chose the method LaSS, which shares a similar philosophy, for comparison. It is essential to highlight that the mask matrices in our method are obtained based on the specific pruning criterion and schedule. They are not learnable parameters.\n\nDesigning language-specific parameters by adding extra parameters to the model could lead to two issues:  1) an unfair comparison when compared with techniques and models with fewer parameters. 2) a decrease in inference speed. More specifically, Adapter-based approaches commonly introduce learnable parameters for each language pair, thus increasing the parameter count for the entire model of all languages and the parameter count for individual language pairs, causing inference speed decrease. Designing distinct layers/modules for different languages in the two mentioned papers maintains a constant size for each pair and has no/negligible impact on inference speed. However, it still leads to the augmentation of the total model parameters.    \n\n \n\n\n> **2. [medium]: While BLEU is still widely used, there are now better metrics to use for Machine Translation that correlate better with human judgment [3], specifically I'd recommend including chrF and COMET scores.**\n>\n\nThanks for your feedback on the evaluation metric for Machine Translation. While COMET has gained recognition for its robust correlations with human judgments, recent research (Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET) shows that it can be unreliable in identifying specific critical errors, such as deviations in entities and numbers. As a result, we have opted to utilize the (most) widely employed evaluation metric BLEU, combined with the win ratio, to evaluate the performance of different methods. \nNonetheless, we greatly value your recommendation and fully agree that considering COMET holds value. We report the COMET scores and chrF scores on both the IWSLT dataset and WMT dataset below. The results in Table 1 and Table 2 below confirm the improvements of our approach over the baseline and MagnitudeOneshot (LaSS) approach reported in Tables 1 and 3 in the paper. We will upload these results in the final version of the paper.\n\nTable 1 IWSLT chrF and COMET performance:\n| | chrF |  COMET  | \n|------------------------ |------------|----------|\n| Baseline                                      | 51.2 | 0.773 |\n|MagnitudeOneshot (LaSS)             | 51.5 | 0.775 |\n| GradientGradual (Ours)            | **52.9** | **0.788** | \n\n\nTable 2  WMT chrF and COMET performance:\n| | chrF |  COMET  | \n|------------------------ |------------|----------|\n| Baseline                                      | 44.4 | 0.737 |\n|MagnitudeOneshot (LaSS)             | 45.4 | 0.747 |\n| GradientGradual (Ours)            | **45.9** | **0.755** | \n\n\n\n>**3. [medium]: Since sub-layers are extracted for specific language pairs, it would be interesting to see how that affects zero-shot performance. I'm assuming it will be very detrimental to that since it is unclear which subnetwork to use so I'd consider this a limitation of this approach over the dense baseline.**\n>\n\nThanks for sharing your input on zero-shot. We argue that our method has the capability to surpass the performance of the dense baseline. It is demonstrated by Lin et al. (2021) that by combining the encoder mask of X -> En and the decoder mask of En -> Y for X -> Y translation direction,  LaSS obtains consistent gains over the multilingual baseline in all language pairs for zero-shot evaluation. Meanwhile, as you described in the review feedback, our approach improves two weaknesses in Lin et al. (2021), and the ablation study on the pruning ratio further verifies that our method is more effective. Therefore, our approach has a strong probability of outperforming the dense baseline. Due to the constraints of paper dimensions and the limited time for the author response, we decided to leave the exploration of our approach in this direction to future work.\n\n\n>**4. [large]: …With a high pruning ratio, we can resort to only storing the active weight indices but scaling this to hundreds of languages seems sub-optimal given the quadratic growth. Since the proposed approach is motivated for \"resource-constrained scenarios\" (l. 161), I fail to see how this is more resource efficient in terms of disk space than training sparse lower-dimensional language-specific modules (which will scale linearly in L(latic form) and still maintain a similar efficient inference costs and memory consumption). As a result: I believe a more thorough analysis regarding disk requirements and inference speed is required across different methods.**\n>\n\n\n\nThanks for your feedback. We appreciate your input. We clarify the disk requirements and the inference speed with detailed evaluation results below.\n\n**Response to the definition of “resource-constrained”:** By \"resource-constrained\", we mainly refer to scenarios where researchers do not have enough data and computational resources to train the big model when a significant number of additional language-specific parameters are introduced. For instance, the design of separate decoders for each language pair following the conventional transformer structure, with an equal number of encoder and decoder layers, will potentially require more data and computational resources to successfully train all the parameters. We will improve the clarity of this specific sentence to avoid potential confusion.\n\n\n**Response to disk requirement:**\nRegarding disk requirements, we would like to clarify the following points.\nOur approach is not applied to any bias parameters and layer normalization layers. Assuming the size of each mask in our approach is |$\\theta_m$|, |$\\theta_m$|is supported to be less than (|$\\theta_0$|-|$\\theta_{embed}$|). Moreover, as you mentioned, we can exclusively store indices of active weights,  reducing the size to less than |$\\theta_m$|/2. Additionally, compared to the extra parameters introduced to the model in other studies, typically stored in float32 format and requiring 4 bytes per element, each element in our approach requires only 1 byte. \nFurther, in this work, we focus on English-centric multilingual translation and leave the exploration of M2M to future work. However, for the M2M scenario, our strategy will not be using $L*(L-1)$ masks for $L$ languages. Instead, we would utilize a single encoder mask for all the language pairs with the same source language and a single decoder mask for all pairs with the same target language, which equals requiring $L$ distinct full masks in terms of disk storage requirement (a full mask contains an encoder mask and a decoder mask). An alternative option is to search for encoder and decoder masks on a language-family basis, suggesting less than $L$ different full masks are required. Further investigation is needed for a comprehensive understanding, and we would like to extend our effort in this direction in future work, as mentioned in our limitation section. With the clarifications above, the impact of our approach on disk space is manageable, even in the M2M scenario.\n\n\n\n\n\n**Regarding inference speed, we provide clarification below, accompanied by detailed results, to demonstrate that our approach has a negligible to no impact on inference speed.**\n\nThe subnetwork mask matrices are represented as binary matrices and are obtained during training and directly applied to Attention and Feedforward sub-layers during inference. Compared to the compute-intensive multiplication between dense weight matrices (typically in float32 or float16), multiplying a binary mask matrix with a weight matrix on some sub-layers introduces a  minimal overhead and, therefore, has a negligible impact on inference speed. \nIn addition, zero elements in the masked weight matrix could result in faster inference speed due to the possibility of avoiding unnecessary arithmetic operations; for example, $x+0$ is always $x$, and $x*0$ is always 0. Furthermore, our approach provides the potential for making the most of model sparsity and further boosting inference speed through advanced techniques, such as Sparse Multiplication Optimization.\n\nFollowing your suggestion, we present the inference speed results of the MNMT baseline and our method without applying additional techniques such as Sparse Multiplication for a fair comparison. \n\nWe report tokens/second on the de-en test set with Transformer-small(details shown in Appendix B Model details section). Following the inference speed evaluation setup in Pires et al. (2023), the batch size is always set to 1 and the result is averaged over 5 runs. Speed GPU is measured using a single NVIDIA A100 GPU and   Speed CPU is measured using a single-threaded Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz.\n\nWhile the best performance of our approach is achieved with pruning_ratio set to 0.6, as shown in Figure 1 in the paper, we provide the inference speed results with the pruning ratio ranging from 0.1 to 0.9. We hope this could offer a more comprehensive analysis of inference speed. Although Speed GPU  and CPU are different in absolute value, they show a similar pattern across different pruning ratios, and we take the Speed GPU as an example to analyze. \nThe results indicate an improvement in the inference speed of our approach when compared with the multilingual baseline within the pruning_ratio range of 0.6 to 0.9, and the highest inference speed is obtained when pruning_ratio is set to 0.9. We attribute this speed improvement to the avoidance of unnecessary operations with element 0, as explained above. Additionally, the inference speed is slightly slower than the MNMT baseline, with pruning_ratio smaller than 0.6, but not statistically significantly different.\n\n| Model                              |Speed GPU (tokens/s)   |Speed CPU (tokens/s) |\n|------------------------ |------------|----------|\n| Multilingual Baseline               |237.56 ± 2.70   |59.98 ± 2.73 |\n| GradientGradual( pruning_ratio=0.1) |235.23 ± 3.65   |59.57 ± 0.77 |\n| GraientGradua( pruning_ratio=0.2)   |233.93 ± 3.28   |58.81 ± 4.27 |\n| GraientGradua( pruning_ratio=0.3)   |235.71 ± 1.92   |58.26 ± 3.34 |\n| GraientGradua( pruning_ratio=0.4)   |236.35 ± 2.71   |58.04 ± 2.93 |\n| GraientGradua( pruning_ratio=0.5)   |235.94 ± 2.08   |59.37 ± 1.73 |\n| GraientGradua( pruning_ratio=0.6)   |238.76 ± 2.49   |58.94 ± 3.28 |\n| GraientGradua( pruning_ratio=0.7)   |240.10 ± 2.62   |60.17 ± 2.70 |\n| GraientGradua( pruning_ratio=0.8)   |242.55 ± 2.47   |60.58 ± 2.37 |\n| GraientGradua( pruning_ratio=0.9)   |243.23 ± 2.89   |61.56 ± 3.99 |\n\n\n\n**Regarding your questions**\n> **Question A: The additional sub-network search procedure e.g. Phase 2 + 3 imposes computational overhead during training that could be used differently e.g. training the baselines for longer. I'm curious how that will affect the comparison?**\n>\n\nThe multilingual baseline model serves as the initialization for the final model. The baseline and the final model can indeed exhibit different absolute performances depending on the duration of baseline training. With the same baseline(initialization), the quality of the final model relies on the capability of extracting high-quality sub-networks for language pairs. The robustness results in section 5.3 and the similarity scores in section 5.5 in the paper confirmed the effectiveness and robustness of our approach compared to MagnitudeOneshot. Therefore, we believe that as long as the baseline model is the same, the relative quality of the final models remains the same regardless of the duration of the baseline training (GradientGradual > MagnitudeOneshot > Baseline).\n\n\nSpecifically, in our experiment, we configured the hyper-parameter “max_update” to a sufficiently large value to avoid underfitting and employed the early stop technique by configuring the hyper-parameter “patience” to 30 to prevent overfitting. We aimed to train an optimal baseline, serving as the foundation for the final model for both GradientGradual and MagnitudeOneshot. In this way, the final model can potentially achieve peak performance. This configuration simulates the most likely scenario used to achieve the best MNMT model in real-world applications.\n\n>**Question B: Will code be released for the experimental setup? Specifically, Phase 2 and Phase 3 would be interesting to the community I believe**\n>\n\nWe appreciate your interest, and we fully recognize the value of sharing code with the community. We are actively working to explore the possibility of releasing both the code and experimental setups.\n\n\n**Regarding your feedback on Grammar Style and Presentation improvement**\n\n> **Algorithm 1 l.3 missing space before $\\theta_0$ and other feedback.**\n>\n\nThanks for your careful review. We will correct each Grammar issue you pointed out and improve our presentation soon in our paper. We will also extend our related work section to mention the missing references you pointed out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Improves two weaknesses in Lin et al. (2021) i.e. one-shot + magnitude pruning and clearly differentiates the novelty of the presented work\n- Very well written and easy to follow the main idea and experiments\n- Interesting ablation on the pruning ratio which verifies that gradual gradient pruning is very effective", "reasons_to_reject": "- **[large]**: Besides LaSS the presented method is not compared to any other method that has been proposed for MNMT. I'd appreciate a more thorough comparison to existing literature. Specifically, I'd recommend to compare against LSL-NAS + dense pre-training from [1], X-MOD from [2], or Adapter-based approaches, similar to what is done in [1]. All of these share similar efficiency advantages to what is presented here since they make use of sparsity.\n- **[medium]**: While BLEU is still widely used, there are now better metrics to use for Machine Translation that correlate better with human judgment [3], specifically I'd recommend including chrF and COMET scores.\n- **[medium]**: Since sub-layers are extracted for specific language pairs, it would be interesting to see how that affects zero-shot performance. I'm assuming it will be very detrimental to that since it is unclear which subnetwork to use so I'd consider this a limitation of this approach over the dense baseline.\n- **[large]**: The proposed method requires one sub-network mask $\\mathbf{M}\\_{s\\_i \\rightarrow t\\_i }$ per language-pair supported in the MNMT model. In the many-to-many (M2M) scenario, this will require $L * (L-1)$ masks (where $L$ is the number of languages supported in the model). Assuming we apply the proposed approach for all non-embedding parameters in the model, each mask will have the size $|\\theta_0| - |\\theta_\\{embed\\}|$ for a total of $L * (L-1) * (|\\theta_0| - |\\theta_\\{embed\\}|)$ entries that will require significant disk space to store during inference. With a high pruning ratio, we can resort to only storing the active weight indices but scaling this to hundreds of languages seems sub-optimal given the quadratic growth. Since the proposed approach is motivated for \"resource-constrained scenarios\" (l. 161), I fail to see how this is more resource efficient in terms of disk space than training sparse lower-dimensional language-specific modules (which will scale linearly in $L$ and still maintain a similar efficient inference costs and memory consumption). **As a result: I believe a more thorough analysis regarding disk requirements and inference speed is required across different methods.**", "questions_to_authors": "**Question A**: The additional sub-network search procedure e.g. Phase 2 + 3 imposes computational overhead during training that could be used differently e.g. training the baselines for longer. I'm curious how that will affect the comparison?\n\n**Question B**: Will code be released for the experimental setup? Specifically, Phase 2 and Phase 3 would be interesting to the community I believe", "paper_topic_and_main_contributions": "The presented paper proposes a gradient-based gradual pruning approach for multilingual machine translation that will extract language-pair specific subnetworks to boost translation quality. Additionally, different sub-network extraction techniques are compared and  the overlap in sub-networks is analyzed showing that sub-network overlap is correlated with language similarity.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- [1]: [Learning Language-Specific Layers for Multilingual Machine Translation](https://aclanthology.org/2023.acl-long.825) (Pires et al., ACL 2023)\n- [2]: [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://aclanthology.org/2022.naacl-main.255) (Pfeiffer et al., NAACL 2022)\n- [3]: [Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust](https://aclanthology.org/2022.wmt-1.2) (Freitag et al., WMT 2022)", "Typos_Grammar_Style_and_Presentation_Improvements": "- Algorithm 1 l.3 missing space before $\\theta_0$\n- Please include all figures as vector graphics so they look good in print and scale properly e.g. Figure 1, 2, 3\n- Some of the cited bibliography entries have published version, e.g. Han et al., 2015 was published at ICLR 2016. Please go through your cited sources and ensure to credit the published version."}}]

Author rebuttal:

Meta-review: The overall excitement of the reviewers was ambivalent, but the active author response discussion period showed a lot more positive sentiment from the reviewers on the quality of the work especially with the addition of the extra experimental results provided by the authors and the extra context and thoughtful reasoning provided to answer the questions raised by the reviewers.

While the overall assessment is higher than the numerical scores assigned by the reviewers, this is mainly due to the extra information that was provided during the author response, so it is very important that all of the extra experimental results and clarifications are included as part of the paper by the authors. The additional context provided by the information in the author response is a big reason for the final evaluation of this submission in this meta-review.
Decision: Accept-Main