Condensing Multilingual Knowledge with Lightweight Language-Specific Modules
Authors: Haoran Xu
Weiting Tan
Shuyue Stella Li
Yunmo Chen
Benjamin Van Durme
Philipp Koehn
Kenton Murray
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual Machine Translation
Lightweight
Language interference
Distillation
TL;DR: We propose lightweight language-specific parameters to boost multilingual performance and parameter efficiency, and propose fuse distillation to further improve the efficiency of inference and model serialization.

Abstract: Incorporating language-specific (LS) modules or Mixture-of-Experts (MoE) are proven methods to boost performance in multilingual model performance, but the scalability of these approaches to hundreds of languages or experts tends to be hard to manage. We present Language-specific Matrix Synthesis (LMS), a novel method that addresses the issue. LMS utilizes parameter-efficient and lightweight modules, reducing the number of parameters while outperforming existing methods, e.g., +1.73 BLEU over Switch Transformer on OPUS-100 multilingual translation. Additionally, we introduce Fuse Distillation (FD) to condense multilingual knowledge from multiple LS modules into a single shared module, improving model inference and storage efficiency. Our approach demonstrates superior scalability and performance compared to state-of-the-art methods.

Condensing Multilingual Knowledge with Lightweight Language-Specific Modules
Haoran Xu, Weiting Tan*, Shuyue Stella Li*, Yunmo Chen*, Benjamin Van Durme, Philipp Koehn, Kenton Murray
Johns Hopkins University (hxu64, wtan yunmo, ,hi, kenton juu.. silu.edi. edu
Abstract
3.0
Incorporating kanguage-ppecific (LS) modules or Mixture-of-Expprtt (MoE) are proven methods boost performance multilingual model performance, but the scalability of these approaches to hundreds of languages experts tends to be hard manage. We present sanguage-specifee Matrix Synthesis (LMS),, novel method that addresses issue. LMS utilizes parameter-efficiett and lightweight modules, reducing the number parameters while outperforming existing methods, c.g., +1.73 BLEU over Switch Transformer on OPUS-100 multilingual translation. Additionally, we introduce Fuse Distillation (FD) to condense multilingual knowledge from multiple LS modules into single shared module, improving model inference and storage efficiency. Our approach demonstrates superior scalability and performance compared to state-of-the-art methods.'
2.5-
2.0
ABLEU
15-
Switch Transforme
0.5
25
50
75
100
125
150
175
Figure 1: We show the BLEU gains between the LMS method and the Switch Transformer the model's parameters increase in multilingual translation ablation study. The LMS method notably outperforms the Switch Transformer with similar extra LS (expert) parameter counts, achieving comparable performance even with four to five times fewer parameters.
1 Introduction
difficult to manage and sometimes impractical when working with a large variety of languages. This is because the fundamental element forming LS or MoE modules is typically the full-rank weight matrix derived from a densely connected layer, which causes rapid increase in the number of parameters with a large number of languages experts.
Multilingual models confer the benefit of facilitating cross-lingual learning: however, they also grapple with the issue of language interference (Conneau et al., 2020; Wang et al., 2020a; Shaham et al., 2022). Recent studies aim to alleviate negative language interference through the introduction of language-specific (LS) modules (Zhang et al., 2020; Fan et al., 2020; Zhang et al., 2021; Fan et al., 2021; Pires al., 2023). In this setup, each language batch is processed through its designated module rather than shared module. Although this approach is promising and barely inflates the number of FLOPs like Mixture-of: Experts (MoE) (Shazeer et al., 2017; Lepikhin et al., 2021),² the number of parameters becomes
In this paper, we first scrutinize the parameter efficiency of language-specific modules from the perspective of using fewer parameters. Consequently, necessary question arises (RQI): can we approximate the original dense weight matrix using substantially fewer parameters? To answer this question, we propose novel and parameter-effiient method, Language-Specific
* Equal contribution
computational cost may only come from communication among devices (such as ALLToALL) or gate routing.
'Werelease our code at: https: //gs thwb.bb.ttttttt.ee/eee/e.eb... hwtpgg.ttt......eeeeeeeeeeeeeeee con/fel ixxu/ LM.
AAthhough MoE employs routing mechanism to keep the number of experts smaller than the number of languages, the parameter cost remains substantial.
'Fach pass through the model utilizes only the corresponding language-specific component. The additional

---

Matrix Synthesis (LMS), which can achieve similar performance switch transformer even with three four times smaller LS parameters (as shown in Figure 1).
the Language-specific Matrix Synthesis (LMS) method form low-rank matrices to approximate the full-rank ones. This is inspired by the concept of "intrinsic dimension" " in pre-trained language models (Aghajanyan et al., 2021; Hu et al., 2021) and "intrinsic rank" in trainable matrices, lcading to the idea that features are learned in subspace. Specifically, shown in Figure 2, our LS matrix is derived from the multiplication of an L. 'vertical' matrix with an LS 'flat' matrix. Formally speaking, let W E R be weight matrix in the model and we want to build parallel LS matrices which have the same size. Hence, for each language 4, E {1,2,..., L with L being the number of languages, there exists an LS vertical matrix Wh Rexd and an LS flat matrix W E R (d min(r,e)) that full-rank matrix. Here, we propose two synthesis methods: language-wist and pair-wise synthesis.
Then, we further investigate parameter efficiency from the perspective of knowledge density in each LS module. Given recent discoveries that the performance improvement of sparsely activated models diminishes with an increase in the number of experts (Hoffmann et al., 2022; Gao al., 2022; Xu et al., 2023), we hypothesize that knowledge in these experts (or LS modules) is over-estimated. Hence, we propose another question (RQ2): Could a single shared module encapsulate the same level of knowledge as language-specific modules? In introduce the Fuse Distillation (FD) method to examine the feasibility of condensing the multilingual knowledge into a single module.
addressing this question,
approximate the
Our main contributions are summarized as follows:
We propose the parameter-effiiient and lightweight LMS method, which substantially outperforms previous LS methods or MoE with fewer than or the same number of parameters, c.g., +1.73 BLEU over Switch Transformer on OPUS-100 multilingual translation.
he
Pair-Wise Synthesis
Language-Wise Synthesis
We introduce FD to condense multilingual knowledge from LS modules into a shared module. FD is able to use only 2M more parameters (1% increase) to achieve the 65% of performance gains from Switch Transformer which use 760M more parameters (314% increase) during inference.
ne
r
LMS and FD show strong generalization performance among multiple tasks, including multilingual machine translation (MMT) (Zhang et al., 2020), multilingual named-entity recognition (MNER) (Pan et al., 2017), and multilingual question answering (MQA) (Artetxe et al., 2020).
Wen-he E Brxc
Wen E TC
The difference between pair- and language- wise synthesis. Language-wist wise synthesis constructs a low-rank matrix using both vertical and matrices derived from the same language. Conversely, pair- wise synthesis formulates the matrix by combining the vertical matrix from the source language with the flat matrix from the target language.
Figure 2:
2
Lightweight LS Modules
In this section, we address RQI by constructing LS modules with significantly fewer parameters.
Language- Wise Synthesis Most multilingual tasks, such conventional multilingual question- answering, are characterized by a language- monolithic nature: a single example only pertains to a single language, and examples from different
2.1 Language-Specific Matrix Synthesis
Language-specific modules are typically composed of linear projections, whose weights are full- rank matrices previous studies. We propose

---

languages build the multilingual data. Under such circumstances, naive way assemble language-specific matrix for a given language, I, is straightforwardly using its corresponding vertical and flat matrices, such that W = w
3
Can We Fuse Multilingual Knowledge in A Single Module?
In this section, we introduce Fuse Distillation (FD) and use preliminary experiment to answer RQ2: whether we can condense the multilingual knowledge from language- modules into single module.
Pair-Wiss Synthesis Cross-lingual tasks like MMT can also be accomplished using language- wise synthesis, wherein the encoder uses the source language matrix and the decoder uses the target language matrix. However, we posit that this is not the optimal strategy for MMT tasks due to the lack of learning bilingual information. Motivated by this, we introduce a pair-wise synthesis method to accommodate the bilingual context in each example in MMT. In this strategy. the srnguage-speciing matrix is composition of the vertical matrix from the source language 4 and the flat matrix from the target language ly: W4 Ml = W w/y. The difference between the language-wise and pairwise synthesis approaches is depicted in Figure 2. In Section 5, we will demonstrate that the pair-wise synthesis upproach is more effective.
3.1 Fuse Distillation
Let us first consider a language- (or task-) level MoE (Kudugunta al., 2021), where we replace a single FFN layer with FFN modules. is the number of languages, as defined previously. The slight difference from the original design is we discard the routing gate and make each expert language-specific, i.e., an expert only serves batches in its corresponding language. Given recent findings that model improvements diminish with an increasing number of experts (Hoffmann et al., 2022; Gao et al., 2022; Xu al., 2023), we hypothesize that information contained in experts is sparse and can be condensed into a shared module. To fuse knowledge from L FFN layers to the shared one, we propose the following training scheme and name this method Fuse Distillation:
After deriving language-specific matrix, incorporate it into the original full-rank matrix, as opposed to performing an isolated forward pass of the model like MoE and conventional LS methods. This approach stems from our hypothesis that the employment of low-rank matrices alone may not sufficiently facilitate the learning of features. Therefore, given input I3 associated with source language and target language l; (4 and lj are the same for language- monolithic tasks), our modified forward pass yields the output Fo:
We first add an additional shared FFN parallel to an existing model with L FFN layers as shown in Figure During training, each batch undergoes two forward passes and one backward pass. In the first forward pass, the batch is processed through its language-specifie FPN module; in the second pass, the batch is routed through the shared FFN. To fuse the language- speuaage-peccficcc knowledge contained within the L FFN modules into the shared FFN module, distillation loss between the outputs from the two forward passes is also incorporated:
To = (W + W4 = (W + W/'Wy)aa (I)
2.2 Where to Implement?
We primarily focus on incorporating language- specific matrices generated using the LMS method into the linear projection of each feedforward network (FFN) layer in every transformer layer. Recall from earlier that r and e are the number rows and columns in the matrix, and L is the number of languages. Thus, the total number of language-specing parameters added is given by 21. N d (c + r). where N represents the number of layers. We also conduct an ablation study examine the performance when implementing LMS in attention layers in Section 6. For initialization, we employ a random Gaussian distribution for vertical matrices and zeros for flat matrices suggested by Hu et al. (2021).
f (gl()(()) p&)-
(2)
where pt denotes the probability output for the LS pass, and ps represents the shared pass output. The function g(-) signifies that gradients will not be traced back, so only the shared module learns from LS modules but LS ones do not learn from this loss. The backward pass also involves optimizing the model by minimizing the Cross-Entroopy loss (CE) between the target and predicted values (the regular training loss). Thus, the total loss is:
L - - (CE() pi) + CE p&)) + L (3)
2
where y denotes gold labels.

---

Then, during the inference stage, we discard the LS modules. The model only forward passes the shared FFN for inference. To evaluate whether the shared FFN has effectively learned all LS information, we conduct a comparison between its results and those obtained via the routing through LS modules instead.
L
Pr
Ps
X N
he
shared
3.2 Prelininary Experiments
Our preliminary experiments are conducted under three settings:
LayerNorm
(1) Naive MMT: A basic multilingual translation model is trained without any modifications.
Multi-head Attention
(2) FD: This setting utilizes our proposed fuse distillation method.
LayerNorm
(3) FD-LS: We train the model with the FD method, but during the inference stage, the input is processed through its language-specific FFN module instead of the shared module the original language-level MoE did.
Training Forward Pass:
Infarance Forward Pars:
We carry out our experiments using the IWSLT benchmarks, focusing on the many-to-many translation model paradigm. Following Lin et al. (2021); Xu et al. (2022), we collect 8 English- centric language pairs from the IWSLT' 14 dataset, with sizes ranging from 89K to 169K sentences. We train all methods with the same number of steps and leave detailed training settings in Appendix A. We report sacreBLEU scores (Papineni et al., 2002; Post, 2018) with the FLORES-200 tokenizer (NLLB Team et al., 2022).
We utilize a language-leve MoE architecture verify the feasibility of fusing multilingual knowledge from all language-specific modules into single shared module. During training, each batch goes through the LS module in the first forward pass and goes through the shared module in the second pass. Then, we conduct distillation between two outputs to condense the knowledge into the shared module. For inference, we discard the LS module and only use the shared module.
Figure 3:
needed by utilizing the Linguage-speciffe matrix synthesis method during inference, so we then attempt to incorporate the FD method within LMS. Similar to Section 3.1, apart from the LS vertical and flat matrices, we introduce shared vertical and flat matrices, denoted as W/shired and Wahured, respectively. To employ the fuse distillation method, each batch is required to undergo two forward passes. The initial pass navigates through the LS matrix W + W while the subsequent pass traverses the shared matrix W + Wishaned Wyshaee, These two passes generate two respective outputs, Pt and p.. Given the common parameter W shared across both paths, utilize symmetric KL divergence (Jiang et al., 2020) for distillation, as opposed the traditional KI divergence:
3.3 Results and Analysis
Overview results of these 4 settings are shown in Table 1. The reported scores are the average of both xx-te and en-*xx directions. As anticipated, after applying languag--ppecific modules for each FFN layer, FD-LS has considcrable enhancements the naive MMT (+1.50 BLEU gains). Importantly, after discarding LS modules, FD only performs slightly worse than FD-LSS (+1.17 vs. +1.50) with much fewer parameters for inference (48M vs. 149M). This observation underscores the feasibility of condensing multilingual knowledge into a single FFN module, thereby reducing the need of large number of LS parameters for inference.
4 Combining LMS and FD
1 C/a (KL p&) KLL Pt). (4)
We have shown the success of multilingual information condensation by fuse distillation. We are interested in further reducing the parameters
Thus, the backward pass optimizes both the standard prediction loss and the fuse distillation

---

#purams
Methods
de
he
it
n1
pl
avg.
Training
Inference
Naive MMT
25.03
32.59
39.98
18.76
33.39
34.00
36.71
22.37
30.35
48M
FD
+1.01
+1.15
+1.43
+0.64
+1.44
+1.19
+1.22
+1.22
+1.17
161M
48M
FD-LS
+1.30
+1.45
+1.72
+0.77
+2.08
+1.48
+1.41
+1.73
+1.50
161M
149M
Table 1: Average BLEU on IWSLT 14 many-to-many translation. Our proposed FD able to fuse the majority of knowledge into single module (+1.17 vs. +1.50) with the same parameters as the naive model during inference.
Method
Illustration
Complexity for Training
Complexity for Inferance
Conventional L5 Module
xx
O(L o(L-r-e)
62.9M
O(L
62.9M
full rank
Mixture Ex-experts
x
O(E (T
33.6M
O(E c)
33.6M
LMS lours)
xx
O(L d (r c))
2.5M
O(L d- (r++ e))
2.5M
shared full-rank mairis
Conventional IS Module 1
lours)
o((L + -)-
67.1M
0(r-c)
4.2M
Small
low rank matris
x
(ours)
O(LL 1) d c));
2.6M
O(d c))
0.2M
Figure 4: Suppose we incorporate additional language-specific (LS) linear projections into layer. We compare the space complexity of the extra LS parameters (or experts) needed across all methods for both training and inference phases. Let's denote = 15 as the number of languages, = 4096 as the output dimension, dimension,
1024 as the input represents the number of experts for Mixture-of-Exye ofeexerts (MoE), and 32 signifies the rank for low-rank matrices. The number adjacent to the dashed line is the number of parameters calculated based on the given sample numbers. In this case, one can observe that the language-Speciff Matrix Synthesis (L.M) requires a significantly lower quantity of LS parameters compared to other methods during training, and distillation (FD) demands substantially reduced number of additional parameters during the inference stage.
loss.
batches (i.e., a batch only containing sentences one language or one language direction) and only activate the corresponding LS module.
In Figure 4, we provide a comprehensive comparison of space complexity for generating extra LS (or expert) modules, among conventional LS modules, Mfxture-of of and our proposed methods demonstrate
5.1 Baselines
methods. Notably, substantial reductions in parameter usage during both training and inference.
We compare our approaches against two strong baselines that incorporate additional parameters mitigate language interference.
5 Experiments
CLSR: The first baseline is Conditional Language-Specifie Routing (CLSR) (Zhang al., 2021). which employs LS linear projections following FEN or attention layer. Following their best settings, we set the budget p = 0.3 for LS routing. The original setting used shared LS projections across all encoder or decoder sublayers. We also consider a non-shared version, where each sublayer has its own LS projection, and denote it as CLSR*
We evaluate our LMS and LMS+FD methods using three tasks: MMT, MNER, and MQA. Similar to Section 3.2, have two routing options for the LMS+FD method during inference time: 1) evaluating the model by passing the shared route (denoted as LMM+FFD-Share, the default setting), or 2) passing the language-specific module (denoted as LMS+FD-LS). We present results for both routes to show the performance difference between using the condensed module and the original LS modules. Considering the computational cost for MMT, we run all methods once with the same random seed. For the other two tasks, we run experiments with different random seeds and report the average scores. For ease of implementation, we build homogeneous
Switch Transformer: We also consider Switch Transformer (Fedus et al., 2021) as the second strong baseline, which uses similar FLOPss as our methods.3 5 We 16 experts for every two layers
"This does not apply Switch Transformer. *The design of the Switch Transformer, which employs top-1 routing, hears similarity to our model in that it processes

---

#params
Methods
de
fa
it
pl
avg.
Training
Inference
Naive MMT
25.03
32.59
39.98
18.76
33.39
34,00
36.71
22.37
30.35
48M
48M
Switch Transforner
+0.28
+0.40
+0.45
+0.04
+0.60
+0.59
+0.34
+0.67
+0.42
149M
149M
CLSR
+0.00
+0.48
+0.51
-0.23
+0.31
+0,50
+0,42
+0.30
+0.28
53M
53M
CLSR*
+0.66
+0.87
+1.16
+0.53
+0.99
+1.00
+0.87
+0.94
+0.88
105M
105M
LMS, lang-wise
+0.48
+0.53
+0.88
+0.83
+0.86
+0,91
+0.81
+0.91
+0.78
58M
58M
LMS
+0.87
+1.08
+1.04
+0.62
+1.37
+1.20
+1.04
+1.16
+1.05
58M
58M
LMS+FD-Share
+0.82
+0.93
+1.06
+0.34
+1.23
+0.92
40.87
+0.83
+0.88
60M
49M
LMS+FD-LS
+1.23
+1.34
+1.44
+0.77
+1.51
+1.36
+1.24
+1.15
+1.26
60M
58M
Table 2: Overall BLEU results of on 1WSLT"14 many-to-many translation. LMS outperforms all baselines. At inference, FMS+FD-Share utilizes extra IM parameters exceed baselines that enlarge the model size times.
Methods
-XXXX
XX-2e0
#params
high
med
low
all
WR (5)
high
med
low
WR (5)
Training
Inference
Naive MMT
23.89
31.17
29.76
27.37
29.40
31.85
31.49
30.60
242M
242M
Switch Transformer
+1.87
+3.29
+3.51
+2.66
100
+1.18
+1.15
-0.31
+0.84
83
1002M
1002M
CLSR
+0.02
+0.00
+0.01
+0.02
52
+1.33
+2,00
+2.71
+1.83
91
443M
443M
LMMS, lang-wise, 64
+2.12
+2.28
+1.77
+2.09
95
+1.85
+2.34
+2.30
+2.09
94
989M
989M
LMS, d = 64 LMS+FD Share,, d 64
+3.60
+3.82
+3.32
+3.60
99
+2.75
+3.74
+4.16
+3.35
95
989M
989M
+0.49
+0.75
+1.29
+0.74
88
+0.64
+1.52
+2.08
+1.22
98
996M
250M
LMS+FD-LS,
+1.72
+2.03
+2.60
+2.01
100
+1.64
+2,82
+4.03
+2.52
99
996M
996M
LMS, d' 16
+2.45
+2.62
+2.56
+2.53
99
+1.75
+2.68
+3,40
+2.39
96
429M
429M
LMS+FD-SHR d' 16
+0.54
+1.13
+2.20
+1.09
94
+0.81
+1.26
+1.85
+1.17
94
431M
244M
LMS+FD-LS, d 16
+1.28
+1.84
+2.74
+1.77
100
+1.35
+2,25
+3.53
+2.10
100
431M
431M
LMS,
+1.72
+2.05
+2.31
+1.95
99
+1.33
+1.80
+1.71
+1.55
93
289M
289M
Table 3: BLEU scores on OPUS-100 many-to-mant translation. LMS with average. LMS+FD-Shan with d = 16 uses 1% more parameters, and achieves 65% BLEU gains averaged by all directions, compared to the Switch Transformer which uses 3 314% more parameters.
outperforms baselines on
with gate balance loss with weight of 0.01.
lightweight LS model construction. The default LMS setting for MMT tasks is pair-wise unless otherwise specified. We discuss more training details Appendix A.
5.2 Multilingual Machine Translation
Data and Training settings We concentrate on the many-to-many translation setting, with results reported from two benchmarks. The first the English-centric IWSLT' 14 dataset, as aforementioned in Section 3.2. Additionally, examine the OPUS-100 dataset (Zhang et al., 2020), which encompasses 100 languages in total, including 94 development/test language pairs. We preprocess the data by sentencepiece (Kudo and Richardson, 2018), establishing a vocabulary size of 32K for the IWSLT' 14 dataset and 64K for the OPUS- 100 dataset. We utilize transformersnall and transformengig for IWSLT' 14 and OPUS-100, respectively. We fix the training steps for all methods for a fair comparison. For IWSLT' we use 32 as the rank for low-rank matrices. For OPUS-100, we consider three settings: (i) d = 64 to match the parameter size of the Switch Transformer, (ii) = 16 to match the parameter size of CLSR, and (iii) 4 for very
Evaluation We report results in terms sacreBLEU (Post, 2018), tokenized by FLORES- 200 tokenizer (NLLB Team et al., 2022), and win ratio (WR) (Zhang et al., 2020) which is the proportion of language pairs which our method beats the baseline. For IWSLT' 14, we report the scores averaged by xx-*en and en-+xx directions. For OPUS-100, we split the 94 test language pairs into three groups based on their training data size suggested by Zhang et al. (2020): high-resource (> 0.9M, 45 languages), low-resource (< M. 21 languages) and medium-resoune languages), and report the averaged scores in each category. We use beam search with width of 5 and use length penalty of 1.
(others, 28
LMS performance: Light and Effective LS Module The primary results for IWSLT" 14 and OPUS-100 are presented in Table and Table 3, respectively. In the IWSLT' 14 dataset, LMS
through a single module in each expert layer.

---

significantly surpasses both the Switch Transformer and CLSR, despite having considerably fewer parameters. For OPUS-100, our methods and the baselines are evaluated with approximately equal extra parameters (e.g., 1002M in the Switch Transformer and 989M in LMS with 64). Compared with the gains from Switch transformer (+2.66 for en-*xx and +0.84 for xx->en), our pair- wise LMS method achieves substantially higher gains (+3.60 and +3.35). Similarly, our LMS method also outperforms CLSR (+0.02 and +1.83) with a comparable number extra parameters. These results show the strong parameter efficiency of LMS for the MMT tasks. With merely 47M parameters (d = 4), our LMS method matches the Switch Transformer's performance for en->xx and the CLSR's performance for xx-ten
Sampled Language
avg.
(%)
eparams
vi
Tra.
Inf.
LMS
+3.61
440.88
+0.55
96
340M
340M
LMS+FD Share
+3.22
+0.45
+0.33
88
343M
273M
LMS+FD-LS
100
340M
Table 4: The overall MNER results (FI score) between baseline and our three proposed methods.
5.3 Multilingual Named-Entity Recognition
Data and Settings We evaluate our methods on Wikiann Named-Entity Recognition (Pan al., 2017) dataset. We randomly select 24 languages to conduct experiments. The model architecture is based on pre-trained XL-M- attached with feed-forward token-level classifier. We set the dropout rate as 0.1 and run 20 epochs for all methods. We set d = 32 for low-rank matrices and report FI scores.
Language- Mangu Wise or Pair-Wise' We compare language- and pair-wise synthesis in both IWSLT'14 and OPUS-100 = 64) datasets. On average, pair-wise synthesis outperforms language- 0.27 BLEU points on IWSLT'14 14 (+1.05 vs. +0.78). Moreover, the pair-wise method (+3.60 and +3.35) also shows superior performance on the OPUS- 100 dataset compared with the hunguage-wise one (+2.09 and + 2.09). Notably, pair-wise synthesis with d 16 surpassed the performance of language-wise synthesis with d 64, even though the latter has times more extra parameters. Hence, this discovery strongly advocates for the use of pair-wise wise synthesis over the language-wise approach.
Results The overall results are shown in Table 4. When applying LMS to each FFN layer for 24 languages, the model size increases by only 70M, while yielding a 0.55 Fl improvement. After implementing LMS+FD, the performance improves by 0.67 with the LS route and achieves 0.33 gain with the shared route, which requires only an additional 3M parameters. Full results are shown in Appendix B.
wise synthesis by
5.4 Multilingual Question Answering
Data and Settings We pick 6 languages from TyDiQA (Typologically Diverse Question Answering)-God Passage to conduct the MQA experiments (Artetxe et al., 2020). Following Xu and Murray (2022), the representations of subwords in XLM-Rbase are input to a span classification head; linear layer computing the answer's start and end. We set 32 for low-rank matrices, dropout rate = 0.1, and run 20 epochs.
FD performance: Can FD Fuse 95 Languages? On the IWSLT'14 14 8-language MMT dataset, we observe negligible differences between LMS and LMS+FD (+1.05 vs. +0.88), suggesting successful condensation of information from various language-specific modules into the shared module. In the 95-lange (94 languages plus English) scenario of OPUS-100, FD with dimensionality of 16 utilizes only an additional 2M parameters (less than 1% increase compared to the 242M naive model) to attain 65% of the performance improvements from Switch Transformer (+1.13 vs. +1.75 average), which requires 760M additional parameters (a 314% increase). While FD may not condense all multilingual information due restricted parameter capacity, its parameter efficiency is commendable.
The overall results are shown in Table 5. Upon the application of LMS and LMS+FD, all methods exhibit improved performance with slight increase in parameters. Notably, LMS+FD- Share outperforms LMS+FD-LS. This suggests that FD may be more effective in fusing knowledge when the number of languages is relatively small. Full results are shown in Appendix C.
Results

---

Sampled Language
epurams
avg.
Methods
avg. BLEU
WR (%)
#params
bn
Sw
Tra.
Inf.
Naive MQA
77.69
270M
270M
Naive MMT
28.05
61M
LMS
40.59
+0.93
+0.58
50
287M
287M
LMS. IIn only (default)
+2.10
100
80M
LMS+TD Share
+1.39
+0.32
+1.22
100
290M
273M
LMS, only
+1.32
100
77M
LMS+RD-LS
+1.26
200M
287M
LMS, att+ffn
+2.14
100
96M
Table 5: The overall MQA results (F1 score) between baseline and our three proposed methods.
Table 6: The average BLEU gains with three different
LMS designs with a fixed rank = 20.
6 Ablation Study
parameters. Moreover, applying LMS to both FFN and attention layers results in a marginal improvement over its application solely to FFN layers. This outcome suggests that LS information is primarily situated in FFN layers, aligning with the previous findings of Wang et al. (2020b).
6.1 Is LMS Parameter-Efticientt
Here, examine the parameter efficiency of the LMS method, i.e., whether an increase in extra parameters yields proportional enhancement in model performance. We conduct experiments with d ranging from 4 to 60 in increments of to observe the resulting performance variations. For comparison, we examine the Switch Transformer with 4, 8, 12, 16 experts to assess its parameter efficiency. We focus on the MMT task using the OPUS-100 dataset. Due computational demands, we limit experiments to randomly selected 15 languages from OPUS- 100, designated as OPUS- 15. We leave training details in Appendix D.
Related Work
Saneuige-Speciffi Modules To mitigate language interference, previous studies incorporate language-specific modules into models, such as additional language-aware linear projections (Zhang et al., 2020; Fan et al., 2020; Zhang et al., 2021; Fan et al., 2021), LS layer normalization (Zhang et al., 2020). Feed-Forward Networks (Kwon and Chung, 2023), or even entire language- dependent transformer layers (Escolano et al., 2021; Wang and Zhang, 2022: Pires al.. 2023). Similar to LS modules, are also able to reduce language interference (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al.. 2021; Xu et al., 2023). However, the parameter count of LS (or expert) drastically increases when scaling to numerous languages. Zhang et al. (2021) address this issue by sharing all L.S modules across all encoder decoder layers. However, this does not fundamentally resolve the problem, given that the complexity of constructing modules remains unaltered and that different layers may need to learn varying types of LS information.
We report the average BLEU gains over all translation directions in Figure 1. The plot reveals that the LMS curve is steeper compared to that of the Switch Transformer, indicating higher parameter efficiency for method, i.e., it achieves greater model performance with fewer additional parameters. Compared with a 16-expert Switch Transformer. LMS with 52 yields similar performance by using 3.7 times smaller parameters (51M vs. 189M). Numeric results are in Appendix E.
ceature-of-Experrt (MoE)
6.2 Applying LMS to The Attention Layer In our default design, the LMS is solely applied FFN layers. We are interested in assessing the potential benefits of extending LMS the attention layer (in each K. Q. V, output projection). We consider three model variants: (1) LMS applied only to FFN layers (default design), (2) LMS applied only to the attention layers, and (3) LMS applied to both FFN and attention layers. We conduct experiments on OPUS-15, with fixed rank value of d 20.
Lightweight Modules Our proposed techniques draw inspiration from another research line, lightweight fine-tuning, wherein the model undergoes fine-tuning on a parameter subset significantly smaller than that of the original model, such as prefix tuning (Li and Liang, 2021), prompt tuning (Lester tuning (Wang et al., 2023), LoRA (Hu al., 2021). In the multilingual machine translation setting, previous studies use language-pal adapters (Bapna and Firat, 2019) to fine-tune a specific
We show the averaged BLEU of all translation directions of the three designs in Table 6. LMS applied only attention layers yields inferior performance compared to LMS applied only to FFN layers with a similar number of extra
al., 2021). multitask prompt

---

direction. This approach also extends to language- wise adapters (Philip al., 2020), language- family adapters (Chronopoulou et al., 2023), hyper- adapters (Baziotis et al., 2022) to facilitate the cross-lingual learning. In light of the efficient lightweight modules, we propose LMS to help LS modules scale to hundreds of languages.
differences were observed; the discrepancy was less than 1 second per 100 updates. Additionally, a potentially effective strategy to enhance multilingual information encapsulation in FD could involve using a larger shared module relative to other lightweight LS modules. This could be an intriguing avenue for future research.
8 Conclusion
Acknowledgements
The construction of language-specific modules (or experts) using full-rank matrices tends be parameter-intensive and inefficient, especially as the number of languages (or experts) increases. To address this, we have introduced the Sppeciee-Speecffc Matrix Synthesis (LMS) method that approximates the original full-rank matrix. Notably, pair-wise synthesis, a variant of the LMS methods, exhibits commendable performance in MMT tasks. Further, have proposed the Fuse Distillation (FD) approach to condense multilingual information into a shared module, thereby further diminishing parameter requirements during inference. Our methods outperform CLSR and Switch Transformer in MMT tasks and also demonstrate their effectiveness in MNER and MQA tasks.
We thank anonymous reviewers for their insightful feedback. We also extend our gratitude to Lingfeng Shen, Hieu Hoang, Young Jin Kim, Hany Hassan Awadalla, Stephen Rawls, and Amr Sharaf for their valuable suggestions. This work was supported in part by IARPA BETTER (#2019-1905 1600005). The views and conclusions contained in this work are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, or endorsements of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work is also supported in part by an Amazon Initiative for Artificial Intelligence (AI2AT) Faculty Research Award.
Limitations
One limitation of our LMS method is that it necessitates the construction of homogeneous batches, i.e., batches containing sentences exclusively in one language or language direction. However, this limitation could potentially be addressed by implementing ALLToALL communications amongst devices, a strategy that is already widely employed in Mixture of Experts (MoE) models (Lepikhin et al., 2021), which is topic we intend to explore in future research. In each forward pass of an FFN layer. we need an additional step multiply two small matrices, creating the low-rank large matrix. The additional cost of this operation is negligible, as the computational complexity of the FLOPs/tok for Feedforward linear projection, given an input dimension e and output dimension is Orr c), while the complexity for constructing the low-rank matrix with rank d is O(d (r + c)). For example. in our ablation study, when
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine- tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 7319-7328.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874-3884. Minneapolis, Minnesota. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingus Transfcrability of Monolingual Representations. Proceedings of ACI. 2020.
2048, e 512. and 20, the difference in computational 2048x512 & 20 times less. In load can be 2512
Ankur Bapma and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th
terms of actual training time, no significant

---

International Joint Conference on Natural Language Processing (EMNLP-HJCN)) pages 1538-1544 1548, Hong Kong, China. Association for Computational Linguisties.
Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems.
Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. 2022. Multilingual machine translation with hyper-adapters. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1170-1185, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Edward Hu, Phillip Wallis, Zeyuan Allen- Yuanzhi Li. Shean Wang, Lu Wang. Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Alexandra Chronopoulou, Dario Stojanovski. and Alexander Fraser. 2023. Language-family adapters for low-resource multilingual neural machine translation. In Proceedings of the The Sixth Workshop on Technologies for Machine Translation of Low-Resouree Languages (LoResMT 2023), pages 59-72, Dubrovnik, Croatia, Association for Computational Linguistics.
Haoming Jiang, Pengcheng Hc. Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART:
Robust and efficient fine-tuning for pre- trained natural language models through principled regularized optimization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 2177-2190, Online, Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization. arXiv preprint arXiv:1412.6980
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Carlos Escolano, Marta Costa-jussà, José A. R. Fonollosa, and Mikel Artetxe. 2021. Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 944-948, Online. Association for Computational Linguistics.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. 2021. Beyond distillation: Task-level mixture-of-expers for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3577-3599.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed Kiskky,, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2020. Beyond english-centric multilingual machine translation. arXiv preprint Axiv:2010.11125.1
Yoohwan Kwon and Soo- Whan Chung. 2023. Mole: Mixture of language experts for multi-lingual automatic speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. [GS)hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El- El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary. et al. 2021. Beyond english-centric multilingual machine translation. The Journal of Machine Learning Research, (2(1):4839-44) 4886.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for garameter-efficiee prompt tuning. arXiv preprint arXiv:2104 100.009.....
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
Xiang Lisa Li and Percy Liang. 2021. trefi-ttuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on
Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong-Yi -ii Lu, and Ji-Rong Wen. 2022. Parameter-effiiient mixture-of-expee of-exessss architecture for pre-trained language models. arXiv preprint 00311:2203.01104.

---

Natural Language Processing (Volume 1: Long Papers), pages 4582-4597 Online. Association for Computational Linguistics.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-effiieen transfer learning. In The Eleventh International Conference on Learning Representations.
Zchui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 293-305, Online. Association for Computational Linguistics.
Zirui Wang. Zachary C. Lipton, and Yulia Tsvetkov. 2020a. On negative interference in multilingual models: Findings and meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438 4438-4450, Online, Association for Computational Linguistics.
Marta R NLLB Team, Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.044 04672.
Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov. 2020b. On negative interference in multilingual models: Findings and a meta-learning treatment. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 4438 4450.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross- lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1953 1958, Vancouver, Canada. Association for Computational Linguistics.
Haoran Xu, Maha Elbayad, Kenton Murray, Jean Maillard, and Vedanuj Goswami. 2023. Towards being parameter-effiiient: A stratified sparsely activated transformer with dynamic capacity, arXiv preprint 2rX5v:2305.02176.
Haoran Xu, Philipp Koehn, and Kenton Murray. 2022. The importance of being parameters: An intra- distillation method for serious gains. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 170-183, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jerin Philip, Alexandre Berard, Matthis Gallé, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465- 4470.
Haoran Xu and Kenton Murray. 2022. Por qué não utiliser alla språk? mixed training with gradient optimization few-shot cross-lingual transfer. Findings of the Association for Computational Linguistics: NAACL 2022, pages 2043 2059, Seattle, United States. Association for Computational Linguistics.
Telmo Pessoa Pires, Robin M Schmidt, Yi-Hsiu Liao, and Stephan Peitz. 2023. Learning language-speciff layers for multilingual machine translation. arXiv preprint arXiv:2305.022: 02666..
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share not? learning schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations.
Matt Post. 2018 A call for clarity in reporting bleu scores. WMT 2018, page 186.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628 1639, Online. Association for Computational Linguistics.
Uri Shaham, Maha Elbayad, Vedanuj Goswami. Omer Levy, and Shruti Bhosale. 2022. Causes and cures for interference in multilingual translation. arXiv preprint arxiv:2212.07530.
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixtureoof-experts layer. In International Conference on Learning Representations.
Qian Wang and Jiajun Zhang. 2022. Parameter differentiation based multilingual neural machine translation. Proceedings of the AAAI Conference on Artificial Inteligenne, 10, pages 11440-1 1144.

---

A Training Details for IWSLT'14 and OPUS-100
We preprocess the data by sentencepiece (Kudo and Richardson, 2018), establishing vocabulary size of 32K vocabulary. we fix the training steps at 50K with 8K warm-up steps for all methods, with a batch size of 4096 tokens. We employ the transformeTpase model (with an FFN dimension of 2048 and an embedding dimension of 512) for training the OPUS-15 dataset. The other settings the same Appendix A.
To balance the training data, we also over-sample low-resource languages with temperature of T (Aharoni et al., 2019) for the OPUS-100 data and preprocess the data by sentencepiece (Kudo and Richardson, 2018), establishing a vocabulary size of 32K for the IWSLT 14 dataset and 64K for the OPUS-100 dataset. We pre-pend a special language id symbol at the beginning of the source sentence to indicate the target language. We build homogeneous batches (i.e., batch only containing sentences in one language direction) and only activate the corresponding language-specific matrix. We set the dropout rate as 0.1 for both datasets. For the IWSLT' 14 dataset, we fix the training steps at 150K with 8K warm-up steps for all methods, with a batch size of 4096 tokens. For OPUS, we fix the training steps at 100K with 8K warm-up steps for all methods, with batch size of 4096 tokens but accumulating gradients times. We train all models on RTX 6000 GPUs. For the IWSLT' 14 dataset, we employ the gransformeTgmall model (with an FFN dimension of 1024 and an embedding dimension of 51 512), while the transformerbig model (with an FFN dimension of 4096 and an embedding dimension of 1024) is utilized for training the OPUS-100 dataset. The maximum learning rate 0.0005. The optimizer is Adam (Kingma and Ba, 2014) with
for the IWSLT' 14 data. We
E Numeric Results for The Ablation Study
Figure 1 shows the averaged BLEU over all directions. Here, We show the detailed numeric results in Figure 9.
learning rate scheduler and weight decay of We use beam search with width of and use length penalty of
B Full Results for MNER
We show the full results of MNER in Table 7.
C Full Results for MQA
We show the full results of MQA in Table
D Training Details for The Ablation Study
We randomly pick 15 languages from the OPUS- 100 data to build a smaller 15-language data (OPUS-15) for the ablation study: eu, pt. bg. sk, zh. sl, de, hr, nb, ga, rw, as, fy, mr, se. We conduct the ablation study under the many-to-many translation settings. To balance the training data, we sample the data with a temperature of = 5.

---

Methods
pt
mss
al
qu
1e
vi
mny
Naive NER
90.12
92.56
94.7
91.59
88.25
89.64
76.79
82.42
92.60
73.22
96.65
90.47
90.63
LMS
90.47
92.76
94.87
92.95
88.45
89.62
80.4
83.15
92.88
75,92
97.00
90.69
90.87
LMS-FD-Share
90.67
92.79
94.91
92.29
87.98
89.74
80.01
82.61
93.05
96.84
90.61
91.24
T.MS-FD-I.S
90.90
93.15
95.13
93.05
88.25
89.87
80.75
83.33
93.17
74.04
96.94
90.78
91.54
tr
zb
bu
m1
id
el
he
avg.
WR (%)
Naive NER
94.90
92.17
93.49
77.26
92.06
93.24
92.18
93.64
92.01
86.23
83.97
89.20
LMS
95.01
92.42
93.75
77.32
92.71
93.56
92.46
93.84
92.07
86.59
84.20
89.75
96%
MAS-FD-Share
94.88
92.31
93.65
77.78
92.39
93.40
92.41
93.79
92.07
85.67
84.33
89.53
88
MMS-FD--LS
95.03
92.63
93.83
77.99
92.67
93.75
92.67
94.02
92.22
86.88
84.35
89.87
100%
Table 7: Full results for the NMER task. We report scores.
Methods
bn
en
id
ko
SW
avg.
Naive MQA
77.69
70.36
78.26
83.00
61.60
80.97
75.31
LMS
77.1
71.7
78.18
82.76
63.70
81.90
75.89
LMS+FD-LS
78.95
73.47
78.80
84.27
61.90
81.35
76.46
LMS+FD-Share
79.08
73.44
78.86
84.34
62.15
81.29
76.53
Table 8: Full results for the MQA task. We report Fl scores.
Methods
en-xX
XX XX-+en
extra #params
high
med
low
all
WR (96)
high
med
low
all
WR (%)
Training
Naive MMT
20.94
42.3
22.72
26.99
25.45
37.25
27.95
29.1
Switch Transformer, E
21.94
45.00
25.76
28.85
100
26.21
39.35
29.12
30.30
100
38M
Switch Transformer, E
22.36
45.11
27.47
29.45
100
26.37
40.02
29.26
30.59
93
88M
Switch Transformer,
22.66
27.19
40.32
29.55
138M
Transformer, E
23.05
46.25
28.61
30.35
100
40.33
30.31
31.12
100
189M
LMS, d
21.61
40.55
24.24
27.19
87
26.16
38.52
29.21
30.07
100
4M
LMS, d 12
22.20
44.10
25.12
28.63
100
26.56
39. 40
28.65
30.40
100
12M
LMS, d 20
22.57
45.19
25.85
29.26
100
26.86
39.89
30.34
31.03
100
20M
LMS, d 28
22.82
43.56
26. 13
29.01
93
27.07
39.88
30.27
31.13
100
28M
LMS. d' 36
23.10
43.89
26.3
29.28
93
27.24
40.07
30.31
31.27
100
36M
LMS. d 44
23.32
43.61
26.52
29.37
93
27.30
40.53
30.81
31.53
100
43M
LMS, d 52
23.36
45.05
26.65
29.80
93
27.36
40.75
30.72
31.60
100
51M
LMS, d 60
23.50
45.63
26.94
30.09
100
27.51
40.88
31,20
31.81
100
59M
Table 9: The numeric results for the Figure


Reviews:
[{"review_id": "FxxaQIBi0A", "reviewer": "Reviewer_fwLV", "summary": "", "questions": "* I am curious to see the comparison of this approach with other computationally-light methods such as Adapter https://github.com/adapter-hub/adapter-transformers \n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your recognition and positive support! \n\n>I am curious to see the comparison of this approach with other computationally-light methods such as Adapter https://github.com/adapter-hub/adapter-transformers\n\nWe greatly appreciate your recognition and positive support! Thanks for your suggestions! For a more comprehensive understanding, we've included additional experiments focused on the conventional language-specific (LS) sequential LoRA, ensuring they too are trained from scratch in alignment with LMS. The training and evaluation dataset is IWSLT’14 dataset. Our method also substantially outperforms LS-LoRA. \n\n|   Methods  |     ar    |     de    |     es    |     fa    |     he    |     it    |     nl    |     pl    |    avg.   |\n|:----------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|  Naive MMT |   25.03   |   32.59   |   39.98   |   18.76   |   33.39   |   34.00   |   36.71   |   22.37   |   30.35   |\n|   LS-LoRA  |   +0.32   |   +0.15   |   +0.32   |   +0.38   |   +0.64   |   +0.44   |   +0.12   |   +0.74   |   +0.40   |\n| LMS (ours) | **+0.87** | **+1.08** | **+1.04** | **+0.62** | **+1.37** | **+1.20** | **+1.04** | **+1.16** | **+1.05** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is simple yet smart and intuitive, which is supported by the strong performance in the desired downstream task. Beside the performance, it scales to a number of languages, and even propose a technique to pack all the knowledge into a single module, to further reduce the complexity. It's well grounded and solid study.", "reasons_to_reject": "I'm mostly convinced that the paper should get accepted, as there is nothing to complain.", "questions_to_authors": "* I am curious to see the comparison of this approach with other computationally-light methods such as Adapter https://github.com/adapter-hub/adapter-transformers \n\n", "paper_topic_and_main_contributions": "The paper is proposing a new approach for multilingual translation by introducing language specific module, which achieves the state-of-the-art in the multilingual benchmark. Aside from the superior performance, it has a better scalability when it is compared with other architecture such as SwitchFormer, even if they both use a mixture of sub-modules.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0P1oepcpIg", "reviewer": "Reviewer_Khj5", "summary": "", "questions": "A) In line 139 you define W_{l_i -> l_j} = W_v^{l_i} W_f{l_j}. Why not the reverse other, ie, W_v{l_j} W_f^{l_i}? If the idea is to go from language i to language j, then presumably the input is in language i and the order I proposed seems more \"natural\". Did you try this alternative? If not, why?\nA.1) Slightly related: you never state whether you use row or column vectors. Looking at equation 1, it seems you use the latter, but you should state this clearly.\nB) Regarding equation 2, I assume you compute it just at the output layer (and not at each layer), right?\nB.1) If you're using g(.) as a stop_gradient function, shouldn't the formula be: KL(g(p_l) || p_s) instead?\nC) On equation 3, isn't the first term frozen during the fuse distillation? What is trained and what is fixed? It is not entirely clear from the paper where you're updating the weights at each point in time.\nD) In Table 1 you show the naive model, FD, and FD-LS. Where is just LS? This question applies to the rest of the paper. You never show LS alone, but always FD-LS.\nE) During FD-LS, are the LS weights trained, or just the shared weights? I assume they are, because you say in section 3.2 that \"during the inference stage the input is processed through its LS FFN module\", and it would make sense otherwise.\nF) In equation 4, you propose using the Jensen-Shannon divergence (I'm pretty sure you can find some older paper to cite rather than Jiang et al 2020). What is the reason for this? Did you try the regular KL divergence and it didn't work?\nG) In lines 344-345 you say you \"fix the training step for all methods for a fair comparison\". Are all models trained to convergence? Otherwise I'm not convinced the comparison is fair.\nI) In line 451 you say \"we set d=32\". Similar on line 497, \"d=20\". How did you pick this number?\nJ) I don't think you have enough evidence to say on lines 457-459 that \"FD may be more effective in fusing knowledge when the number of languages is relatively small\", since you're comparing different tasks. Is there other evidence?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n### **Regarding your concern about the Method name**:\n>The authors call their method \"Language-Specific Matrix Synthesis\", but their method is essentially LoRA for multilingual settings.\n\nWe are grateful for your feedback and would like to provide further clarification. Firstly, LoRA and LMS differ fundamentally in their approaches. While LoRA serves as a general lightweight fine-tuning method for pre-trained models, LMS ( and the whole model) is explicitly trained from scratch to encapsulate LS information. Secondly, we acknowledge and value the contributions of LoRA. As evidence, we have cited the foundational paper for LoRA (line 178) as well as the work on LoRA itself (line 533). It's essential to note that LMS introduces specific nuances and modifications tailored for multilingual contexts, making it more than a mere variant of LoRA. Recognizing the need for transparency and clear articulation, we commit to ensuring our revised manuscript accurately depicts the relationship between our method and LoRA, highlighting the unique aspects of our approach without any unintentional overstatement.\n\n### **Regarding your concern about Baselines**\n>The paper claims their method \"significantly outperforms previous language-specific or mixture of experts methods\" (lines 77-78), yet they don't compare with any of those methods.\n\nWe deeply appreciate your insightful feedback! Both the CLSR and our method aim towards a shared goal: leveraging language-specific parameters to counteract negative language interference. The CLSR stands as a robust, widely-validated, and foundational method in language-specific models, rendering it a quintessential baseline in our view. Similarly, the Switch transformer serves as a foundational Mixture-of-Experts (MoE) model upon which many MoE models are structured. We respectfully disagree with the point that the chosen baselines are meaningless. In Section 7, we extensively reference related work, underscoring our regard for their contributions. Admittedly, drawing comparisons across every method would be challenging, leading us to opt for two standard methods as baselines. \n\nNonetheless, to underscore the prowess of our approach and follow the reviewer’s suggestion, we benchmark our model against two cutting-edge MoE models: NLLB-EOM and NLLB-CMR [1]. Notably, both these models amplify the computational burden at the FFN layer due to their utilization of top-2 gating, with NLLB-CMR also leveraging one more FFN layer during training. Our LMS method, in contrast, outperforms these state-of-the-art MoE techniques while consuming fewer computational resources and parameters:\n| Methods            | en-xx, high  | en-xx, med | en-xx, low | en-xx, all | xx-en, high | xx-en, med | xx-en, low | xx-en, all |\n|--------------------|:------------:|:----------:|:----------:|:----------:|:-----------:|:----------:|:----------:|:----------:|\n| Naive MMT          |     23.89    |    31.17   |    29.76   |    27.37   |    29.40    |    31.85   |    31.49   |    30.60   |\n| Switch transformer |     +1.87    |    +3.29   |    +3.51   |    +2.66   |    +1.18    |    +1.15   |    -0.31   |    +0.84   |\n| NLLB-EOM           |     +2.35    |    +3.59   |  **+4.1**  |    +3.11   |    +1.83    |    +1.84   |    +1.95   |    +1.85   |\n| NLLB-CMR           |     +2.43    |    +3.81   |    +4.01   |    +3.19   |    +1.93    |    +1.84   |    +1.62   |    +1.83   |\n| LMS, d=64 (ours)   |   **+3.60**  |  **+3.82** |    +3.32   |  **+3.60** |  **+2.75**  |  **+3.74** |  **+4.16** |  **+3.35** |\n\nReference:\n\n[1] NLLB Team, Costa-jussà MR, Cross J, Çelebi O, Elbayad M, Heafield K, Heffernan K, Kalbassi E, Lam J, Licht D, Maillard J, Sun A. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. 2022 Jul 11.\n\n### **Regarding your question A**\n>Why not the reverse other, ie, W_v{l_j} W_f^{l_i}?\n\nThanks for your interesting suggestions!  The primary difference that the reverse order $W_v{l_j}W_f^{l_i}$ will make is that the matrix originally serving the language direction $i\\rightarrow j$ will serve $j\\rightarrow i$. We would expect that they have a similar performance, so we do not further dig into the reverse order.\n\n### **Regarding your question B**\n>I assume you compute it just at the output layer (and not at each layer), right? \n\nYes, we are computing at the output (final) layer rather than each layer. The outputs of the middle layers are vectors that cannot be used by computing KL divergence. We appreciate for pointing out the ambiguity and will provide a clearer elaboration on this in the revised manuscript.\n\n>If you're using g(.) as a stop_gradient function, shouldn't the formula be: KL(g(p_l) || p_s) instead\n\nThanks for pointing this out! Yes, the stop gradient function should be on the left side as you indicated! \n\n### **Regarding your question C**\n>On equation 3, isn't the first term frozen during the fuse distillation? What is trained and what is fixed?\n\nWe appreciate the opportunity to clarify the components of Equation 3. Within the equation, the first two items in the bracket $(CE(y ∥ p_l)$ and $CE(y ∥ p_s)$ are not frozen because they are regular machine translation losses to learn how to translate. We also need to learn the original task while distilling the model. The third item $L_{fd}$ is used for learning from LS components to shared components as we call fuse distillation. All terms in Equation 3 are trained concurrently.\n\n### **Regarding your question D**\n>In Table 1 you show the naive model, FD, and FD-LS. Where is just LS?\n\nWe appreciate your insightful suggestion. The primary focus of Section 3 is to elucidate the feasibility of condensing multilingual knowledge into a shared module, thus the concise presentation. However, for clarity and completeness, we present the LS results. It can be observed that our FD approach performs similarly to (even slightly outperforms) the basic LS method.\n\n|  Methods  |    ar    |     de    |     es    |     fa    |     he    |     it    |     nl    |     pl    |    avg.   |\n|:---------:|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Naive MMT |   25.03  |   32.59   |   39.98   |   18.76   |   33.39   |   34.00   |   36.71   |   22.37   |   30.35   |\n| LS        |   +1.06  |   +1.12   |   +1.43   |   +0.87   |   +1.10   |   +1.11   |   +1.28   |   +1.02   |   +1.13   |\n| FD        |   +1.01  |   +1.15   |   +1.43   |   +0.64   |   +1.44   |   +1.19   |   +1.22   |   +1.22   |   +1.17   |\n| FD-LS     | **+1.3** | **+1.45** | **+1.72** | **+0.77** | **+2.08** | **+1.48** | **+1.41** | **+1.73** | **+1.50** |\n\n### **Regarding your question E**\n>During FD-LS, are the LS weights trained, or just the shared weights?\n\nWe appreciate the opportunity to provide further clarification. Both LS and shared weights are trained.  FD is trained by two forward passes, one with LS modules and one with shared modules, and by one backward pass with regular MT loss and a distillation loss. As indicated in line 230, FD-LS is exactly the model trained by FD method, but the only difference is how to inference. Evaluation of FD only utilizes shared modules but evaluation of FD-LS only uses LS modules.\n\n### **Regarding your question F**\n>you propose using the Jensen-Shannon divergence (I'm pretty sure you can find some older paper to cite rather than Jiang et al 2020). you can find some older paper to cite rather than Jiang et al 2020). What is the reason for this?\n\nPlease allow us to correct that symmetric KL divergence is not the same as the Jensen-Shannon divergence. The definition of JS divergence is $JS(p|q)=\\frac{1}{2} (KL(p|m) + KL(q|m))$, where $m=\\frac{p+q}{2}$. However, symmetric $KL=\\frac{1}{2}*(KL(p|q) + KL(q|p))$. The pioneering work by SMART (Jiang et al 2020) [1] introduced the symmetric KL and employed it within the smoothness-inducing adversarial regularizer, a process necessitating two forward passes. Following in their footsteps, previous studies like R3F [2] and THOR [3] that utilize two passes have also cited SMART for their adoption of symmetric KL.  We wish to duly acknowledge and honor these foundational contributions, hence our citation of the work.\n\n>Did you try the regular KL divergence and it didn't work?\n\nThanks for your interesting question. We appreciate the insightful query. While we have not experimented with the standard KL divergence, we anticipate its potential efficacy. Given the common parameter $W$ involves both LS and shared paths, our intention was to learn $W$ symmetrically, leading us to opt for symmetric KL over the conventional KL, primarily due to the asymmetry inherent in the latter. Moreover, as illuminated by preceding works such as [1,2,3], symmetric KL has demonstrated superior performance over simple KL in similar contexts.\n\nReference:\n\n[1] Jiang H, He P, Chen W, Liu X, Gao J, Zhao T. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020 Jul (pp. 2177-2190).\n\n[2] Aghajanyan A, Shrivastava A, Gupta A, Goyal N, Zettlemoyer L, Gupta S. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2008.03156. 2020 Aug 6.\n\n[3] Zuo S, Liu X, Jiao J, Kim YJ, Hassan H, Zhang R, Gao J, Zhao T. Taming Sparsely Activated Transformer with Stochastic Experts. International Conference on Learning Representations 2021 Oct 6.\n\n### **Regarding your question G**\n>\"fix the training step for all methods for a fair comparison\". Are all models trained to convergence?\n\nWe appreciate the opportunity to provide further clarity on this matter. Employing a fixed number of training steps is a standard approach in multilingual machine translation, as evidenced by a plethora of prior studies, including [1,2,3,4] (see reference below). We opted for this approach for several reasons:\n* Scale of Model and Data: Given the substantial size of our model and data, pinpointing the exact moment of \"convergence\" based on the rise of evaluation loss becomes challenging. Specifically, our most extensive model encompasses 1B+ parameters, coupled with a data size of 55M. Training such a model for weeks to discern convergence signals would be impractical.  Nevertheless, we've designated a rational 100K training step, which has shown to be sufficient for the model to deliver robust performance [2]. Importantly, we apply this same constraint across all models, ensuring an equitable comparison, as also advocated by preceding works.\n* Composite Evaluation Loss: Multilingual machine translation inherently involves multi-task learning, where the evaluation loss aggregates results from all the 100 languages. Continuing training may lead performance on some languages to improve while others may degrade and get worse,  making it challenging to determine a universally \"converged\" point.\n* Reasonable settings: Based on empirical evidence and guidelines from previous works like [2], we've configured our training settings — 100K steps for a 65K batch size and 150K steps for a 4K batch size in OPUS and IWSLT datasets, respectively. These settings have been verified to achieve robust and good results.\n\nReference:\n\n[1] Zhang B, Williams P, Titov I, Sennrich R. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867. 2020 Apr 24.\n\n[2] NLLB Team, Costa-jussà MR, Cross J, Çelebi O, Elbayad M, Heafield K, Heffernan K, Kalbassi E, Lam J, Licht D, Maillard J, Sun A. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. 2022 Jul 11.\n\n[3] Xu H, Maillard J, Goswami V. Language-Aware Multilingual Machine Translation with Self-Supervised Learning. arXiv preprint arXiv:2302.05008. 2023 Feb 10.\n\n[4] Kim YJ, Awan AA, Muzio A, Salinas AF, Lu L, Hendy A, Rajbhandari S, He Y, Awadalla HH. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.10465. 2021 Sep 22.\n\n### **Regarding your question I**\n> \"we set d=32\". Similar on line 497, \"d=20\". How did you pick this number?\n\nThank you for your insightful feedback. These numbers were selected based on empirical observations from our primary MMT tasks,  where these values were observed to introduce only a minimal increase in parameters while maintaining substantial improvements. Although we acknowledge that these hyperparameters were not exhaustively tuned, they serve to ensure a consistent and meaningful comparison. For instance, both the LMS with ffn-only and attn-only utilize the same d=20.\n\n### **Regarding your question J**\n>I don't think you have enough evidence to say on lines 457-459 that \"FD may be more effective in fusing knowledge when the number of languages is relatively small\", since you're comparing different tasks. Is there other evidence?\n\nThank you for the question. We do have other evidence and will clarify it more succinctly in the paper. When exploring multilingual machine translation in the context of a 15-language setup (refer to Table 2), the performance disparity between LMS-FD-Share and LMS-FD-LS is notably narrower (+0.88 vs. +1.26) in comparison to the 100-language configuration presented in Table 3 (+0.74 vs. +3.60 with rank=64). This observation suggests that within the 15-language framework, the information encapsulated within the shared module aligns more closely with that of the LS modules. Moreover, intuitively, with fewer languages in play, the FD approach should find it easier to encapsulate less multilingual knowledge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea of using LoRA to tune the FFNs to be language specific is interesting. The proposed approach for distillation is also interesting.", "reasons_to_reject": "* The authors call their method \"Language-Specific Matrix Synthesis\", but their method is essentially LoRA for multilingual settings. I still think it's interesting, but this attempt to make it sound more novel than it really is is a negative point in my opinion.\n* The paper claims their method \"significantly outperforms previous language-specific or mixture of experts methods\" (lines 77-78), yet they don't compare with any of those methods. There is plenty of related work that is cited on section 7, but no comparisons were made. While it is interesting that the authors test their method on different tasks, the lack of any meaningful baseline is a serious omission.\n\nI do think the idea is interesting and that this can be an interesting paper, but as it stands I strongly believe it would benefit from further work.", "questions_to_authors": "A) In line 139 you define W_{l_i -> l_j} = W_v^{l_i} W_f{l_j}. Why not the reverse other, ie, W_v{l_j} W_f^{l_i}? If the idea is to go from language i to language j, then presumably the input is in language i and the order I proposed seems more \"natural\". Did you try this alternative? If not, why?\nA.1) Slightly related: you never state whether you use row or column vectors. Looking at equation 1, it seems you use the latter, but you should state this clearly.\nB) Regarding equation 2, I assume you compute it just at the output layer (and not at each layer), right?\nB.1) If you're using g(.) as a stop_gradient function, shouldn't the formula be: KL(g(p_l) || p_s) instead?\nC) On equation 3, isn't the first term frozen during the fuse distillation? What is trained and what is fixed? It is not entirely clear from the paper where you're updating the weights at each point in time.\nD) In Table 1 you show the naive model, FD, and FD-LS. Where is just LS? This question applies to the rest of the paper. You never show LS alone, but always FD-LS.\nE) During FD-LS, are the LS weights trained, or just the shared weights? I assume they are, because you say in section 3.2 that \"during the inference stage the input is processed through its LS FFN module\", and it would make sense otherwise.\nF) In equation 4, you propose using the Jensen-Shannon divergence (I'm pretty sure you can find some older paper to cite rather than Jiang et al 2020). What is the reason for this? Did you try the regular KL divergence and it didn't work?\nG) In lines 344-345 you say you \"fix the training step for all methods for a fair comparison\". Are all models trained to convergence? Otherwise I'm not convinced the comparison is fair.\nI) In line 451 you say \"we set d=32\". Similar on line 497, \"d=20\". How did you pick this number?\nJ) I don't think you have enough evidence to say on lines 457-459 that \"FD may be more effective in fusing knowledge when the number of languages is relatively small\", since you're comparing different tasks. Is there other evidence?", "paper_topic_and_main_contributions": "In this work, the authors propose a method similar to LoRA (under a different name) on the MLPs of transformer models to build language-specific modules. They also propose a method to distill these language-specific modules into a single, shared module to save model size. They then evaluate their methods on 3 tasks: NER, MT, and QA.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 40: \"becasue\" -> \"because\".\n* Line 109: \"martices\" -> \"matrices\".\n* Line 193: add \"an existing model with\" after \"shared FFN parallel to\".\n* Line 263: \"need\" -> \"needed\".\n* Line 280: \"Equation 3\" -> \"Equation 4\"?\n* In the text (page 7) you refer to the pair-wise results on Table 3, but the table doesn't mention which results are pair-wise. I suppose it's the 5th result from the top?\n* Line 344: \"training step\" -> \"training steps\".\n* Line 883: \"Figure 1\" -> \"Table 9\".\n* Table 9 caption: \"for the Figure 1\" -> \"for Figure 1\"."}}, {"review_id": "owFPRACujx", "reviewer": "Reviewer_LVVn", "summary": "", "questions": "- I am not sure I understand the difference between the LMS and LMS-FD-LS results. If I understand correctly LMS-FD-LS settings, you add KL-divergence loss that tries to bring close the output of LMS layers and new shared layer. However gradient updates from such loss are only applied to shared layer and not LMS layers. Therefore, according to my understanding, this should start from existing LMS model where you add shared layer (randomly initialized), and only shared layer is updated further with distillation procedure defined in the paper. However, according to the results this is not the case, since the results reported for LMS are different from the results LMS-FD-LS. Does this mean that you do not initatialize model LMS-FD-LS from LMS, but train it from scratch? Could you please bring more training details for these different settings?\n\n- I do not understand what does WR(%) refers to in tables 3 and 7\n\n- I am not sure I understand how do you compute training/inference parameters: is it the amount of total parameters you load in memory? In my understanding, when you know which language/laguage pair you are dealing with, one would only need to load language-specific modules for those languages, and therefore inference  (for specified language) should require less parameters compared to total amount of training parameters?\n\n- I think it would be useful to report total amount of flops in training and inference. That would bring additinal axis of \"efficiency\" comparison between different models \n\n- it would be interesting to report full results decomposed per language pair somewhere (in the appendix, or on separate document) for better", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n### **Regarding your concern about reproducibility**\n>do you strat from pretrained model and add LMS layers on top of it or do you train everything from scratch?\n\nThanks for the detailed question! We start training everything from scratch! We note that this is in contrast to other work in the field and is one of the novel aspects of our proposed method.\n\n>Do LMS-FD-* models start from LMS model or do you train LS and shared modules simultaneously?\n\nFor the LMS-FD-* models, we simultaneously train the LS and shared modules, as delineated in Equation 3. We appreciate the feedback and will provide a clearer elaboration on this in the revised manuscript. **Regarding the other hyperparameters, they are comprehensively detailed in Appendix A and D to facilitate reproduction**. Furthermore, we plan to release the code following the final decision on the paper.\n\n### **Regarding your concern about computing inference parameters**\n>I do not have full understanding about how do authors compute inference parameters.\n\nWe appreciate your insightful feedback. We agree that the number of parameters required by LMS-FD-LS is equivalent to those of LMS-FD-Share if and only if evaluated in a single translation direction. Nonetheless, our work involves translating in multiple directions. Therefore, we would like to define the “number of inference parameters used by LMS-FD-*” as the total number of parameters needed to complete all required translations. In practical scenarios, this refers to the number of parameters loaded into memory at a given time, because it is impractical to reload a model for each translation direction due to the time cost and its infeasibility for production. We will clarify this in the paper.\n\n### **Regarding your concern about full picture of efficiency**\n>inference speed, the amount of flops (in training and inference) would provide additional and important view to get a full picture.\n\nWe appreciate your valuable suggestions. For clarity, we want to remind the reviewer that we discussed inference speed and Flops/tok in the Limitations section (lines 569-583). In light of your feedback, we will highlight it on the main pages with an extra space. It's worth noting that our training and inference speed aligns closely with that of the naive MMT and top-1 MoE training (Please see detailed numbers in “**Regarding your question 4**” below). Moreover, our method boasts a reduced parameter count and enhanced performance.\n\n### **Regarding your concern about CLSR and PEFT baselines**\n>I am not sure the choice of CLSR as a baseline. There are many PEFT methods these days that could serve as a reasonable baseline for such approach\n\nWe are grateful for your constructive feedback. It's important to highlight that the central aim of our paper is to introduce lightweight language-specific modules that counteract negative language interference, **specifically in models that are trained from scratch**. While CLSR aligns with our goal and employs an adaptive routing algorithm to LS modules (which we view as a standard baseline), PEFT is tailored for fine-tuning pre-trained models, serving a different purpose. While we strive to maintain clarity and brevity by not delving deeply into yet-unproposed baselines (e.g., LS prefix tuning in MMT and models entirely trained from scratch), we prioritize existing baselines that resonate with our core goals. Nevertheless, we do explore potential integrations of PEFT and LS modules when training from scratch:\n\n* LoRA: LoRA closely mirrors our LS component, employing two low-rank matrices to forge a larger matrix. In essence, the composition of language-wise LMS can be perceived as a parallel LoRA. For a more comprehensive understanding, we've included additional experiments focused on the conventional language-specific sequential LoRA, ensuring they too are trained from scratch in alignment with LMS. The training and evaluation dataset is IWSLT’14 dataset.\n\n|   Methods  |     ar    |     de    |     es    |     fa    |     he    |     it    |     nl    |     pl    |    avg.   |\n|:----------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Naive MMT  |   25.03   |   32.59   |   39.98   |   18.76   |   33.39   |   34.00   |   36.71   |   22.37   |   30.35   |\n| LS-LoRA    |   +0.32   |   +0.15   |   +0.32   |   +0.38   |   +0.64   |   +0.44   |   +0.12   |   +0.74   |   +0.40   |\n| LMS (ours) | **+0.87** | **+1.08** | **+1.04** | **+0.62** | **+1.37** | **+1.20** | **+1.04** | **+1.16** | **+1.05** |\n\n* Prompt tuning and prefix tuning: In our MMT training approach, a special token is typically prefixed to source sentences, signaling the target language. This process in naive MMT can be equated to a length-1 LS prompt tuning. Prefix tuning bears a resemblance to prompt tuning. However, the LS parameters in these methods are often too limited to encapsulate translation or multilingual information, making them less prevalent in translation tasks.\n\n### **Regarding your question 1:**\n>Does this mean that you do not initatialize model LMS-FD-LS from LMS, but train it from scratch?\n\nWe appreciate the reviewer's questions! Indeed, the LMS-FD-LS is not initialized from LMS; both are trained from scratch. Given that LMS-FD incorporates an additional distillation loss and undergoes another pass, we deemed it appropriate to train LMS-FD from scratch as well. Notably, LMS-FD possesses an additional (language-shared) low-rank matrix compared to LMS. Consequently, the training strategy for LMS-FD encompasses regular MT training complemented by distillation between the LS low-rank matrix and the shared low-rank matrix.\n\n### **Regarding your question 2**\n>I do not understand what does WR(%) refers to in tables 3 and 7\n\nPlease allow us to clarify it more clearly: WR is the proportion of language pairs on which our method beats the baseline, as we mentioned in lines 358-360. For example, if we beat the baseline 99 out of 100 languages, the WR is 99%. We appreciate the feedback and will provide a clearer elaboration on this in the revised manuscript.\n\n### **Regarding your question 3**\n>I am not sure I understand how do you compute training/inference parameters\n\nThanks for your questions!  Please refer to our answers in “Regarding your concern in computing inference parameters\n\n### **Regarding your question 4**\n>I think it would be useful to report total amount of flops in training and inference. \n\nThanks for your suggestions!  Please refer to our answers in “Regarding your concern in full picture of efficiency”. We previously noted that the FLOPs/tok and time costs are similar to those of the naive MMT and top-1 MoE. For a comprehensive understanding, we provide a detailed breakdown of the FLOPs/tok for all experiments (primary results in OPUS-100) in Table 3:\n\n| Methods            | en->xx avg. | xx->en avg. | FLOPs/tok, train | FLOPs/tok, infer |\n|--------------------|:-----------:|:-----------:|:----------------:|:----------------:|\n| Naive MMT          |    27.37    |    30.60    |       506M       |       506M       |\n| Switch transformer |    +2.66    |    +0.84    |       506M       |       506M       |\n| CLSR               |    +0.02    |    +1.83    |       556M       |       556M       |\n| LMS, d=64          |    +3.60    |    +3.35    |      507.6M      |      507.6M      |\n| LMS-FD-Share,d=64  |    +0.74    |    +1.22    |      507.6M      |      507.6M      |\n| LMS-FD-LS, d=64    |    +2.01    |    +2.52    |      507.6M      |      507.6M      |\n| LMS, d=16          |    +2.53    |    +2.39    |      506.4M      |      506.4M      |\n| LMS-FD-Share, d=16 |    +1.09    |    +1.17    |      506.4M      |      506.4M      |\n| LMS-FD-LS, d=16    |    +1.77    |    +2.10    |      506.4M      |      506.4M      |\n| LMS, d=4           |    +1.95    |    +1.55    |      506.1M      |      506.1M      |\n\n### **Regarding your question 5**\n>it would be interesting to report full results decomposed per language pair somewhere (in the appendix, or on separate document) for better \n\nWe deeply appreciate your insightful suggestions. To address this, we will include the complete results for each language pair in the appendix of the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-  Both techniques proposed in the paper seem interesting. I like the idea of LMS, and language-pair LMS looks an interesting. \n- Idea of the fused distillation is very interesting as well. Even though I am not fully convinced by the LMS-FD-Shared results, the fact that it outperforms NMT-Naive model is very interesting, and I believe different directions could be explored to further improve the results", "reasons_to_reject": "- The training details of different variants are not fully documented and would harm reproducibility , eg. \n1) do you strat from pretrained model and add LMS layers on top of it or do you train everythin from scratch?\n2) Do LMS-FD-*  models start from LMS model or do you train LS and shared modules simultaneously?\n\n- I do not have full understanding about how do authors compute inference parameters. Eg. we could distinguish the case of language-specifc from multilingual inference. In the case when we know the language of interest (mandatory when we rely on LMS) we could load only the parameters specific to this language(s) and it would mean that we would have much less parameters for inference than for training: thus in my understanding LMS-FD-LS should have the same amount of inference parameters as LMS-FD-Share . Similarly, CLSR should only activate language-specific modules, and have less parameters at inference that at training. But this doesn't seem to be the case in the tables reported by authors. Do they consider multilingual inference? Some clarifications would help. \n\n-  Moreover, if the authors wanted to reinforce their argument about efficiency, only counting the amount of the parameters doesn't give the full picture (parameter count only address memory efficiency): inference speed, the amount of flops (in training and inference) would provide additional and important view to get a full picture. \n\n- I am not sure the choice of CLSR as a baseline. There are many PEFT methods these days that could serve as a reasonable baseline for such approach (eg. laguage-specific or language-pair specific adapter layers seems like a very natural baseline to me, and should probably be comparable in terms of parameter efficiency )", "questions_to_authors": "- I am not sure I understand the difference between the LMS and LMS-FD-LS results. If I understand correctly LMS-FD-LS settings, you add KL-divergence loss that tries to bring close the output of LMS layers and new shared layer. However gradient updates from such loss are only applied to shared layer and not LMS layers. Therefore, according to my understanding, this should start from existing LMS model where you add shared layer (randomly initialized), and only shared layer is updated further with distillation procedure defined in the paper. However, according to the results this is not the case, since the results reported for LMS are different from the results LMS-FD-LS. Does this mean that you do not initatialize model LMS-FD-LS from LMS, but train it from scratch? Could you please bring more training details for these different settings?\n\n- I do not understand what does WR(%) refers to in tables 3 and 7\n\n- I am not sure I understand how do you compute training/inference parameters: is it the amount of total parameters you load in memory? In my understanding, when you know which language/laguage pair you are dealing with, one would only need to load language-specific modules for those languages, and therefore inference  (for specified language) should require less parameters compared to total amount of training parameters?\n\n- I think it would be useful to report total amount of flops in training and inference. That would bring additinal axis of \"efficiency\" comparison between different models \n\n- it would be interesting to report full results decomposed per language pair somewhere (in the appendix, or on separate document) for better", "paper_topic_and_main_contributions": "This paper addresses the problem of \"efficient multilingual models\". First, it proposes a parameter efficient alternative to language adapters, called Language Specific Matrxi Synthesis (LMS). The idea of LMS is similar to LoRA adapters: the weight matrix is decomposed into a product of 2 low-rank (language-specific) matrices.  LMS can be either language-specific (eg. for English both matrices would be English-specific), or language-pair specific (eg. en-fr would be a composition of English and French matrices). Another technique proposed in this work is Fuse distillation: it aims at distilling all the knowledge from language-specific modules into a single shared module. This is done by bringing representations of shared module close to the repreresentations done by language-specific modules, everythin is trained jointly. At inference, one can either discard language-specific module and only use shared module (LMS-FD-shared) or discard shared module and use only language-specific modules at inference (LMS-FD-LS). \nExperiments are performed with multlingual NMT (IWSLT or OPUS-100 datasets), and multilingual NER and multilingual Question Answering. For translation task language-pair specific LMS outperforms Language-specific LMS. Interestingly LMS-FD-Shared also outperforms Naive NMT (but performs worse than LMS or LMS-FD-LS). This is an interesting result to me, given that in the end, once language specific layers are discarded, the model is not very different from the original Naive NMT baseline in terms of parameter count/architecture. But it looks like that such a fused distillation does help , probably acting as a regularizer during multilingual training.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 40: typo in becasue --> because"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new way to enhance the efficiency of multilingual neural machine translation (MNMT). First, it introduces an efficient alternative to language adapters, termed Language Specific Matrix Synthesis (LMS). Second, it presents Fuse Distillation (FD), which aims to distill all the knowledge encapsulated in language-specific modules into a single shared module. The experimental results show that LMS helps reduce the parameter count without compromising performance. Furthermore, FD reduces the parameter count while also improving MNMT, as it essentially serves as a regularization method.

Pros:
- I agree with the reviewers regarding this work's innovative approach to achieving both efficiency and effectiveness in MNMT. The concepts presented are intriguing, and the results demonstrate notable improvements in both performance and efficiency.
- The paper is well written, making it easy to comprehend and potentially straightforward to replicate. It includes comprehensive and thorough analyses.

Cons:
- While there are minor details that require refinement in the final version, they have all been addressed well in the rebuttal phase.
Decision: Accept-Main