Multilingual Lottery Tickets to Pretrain Language Models
Authors: Jaeseong Lee
seung-won hwang
Venue: EMNLP 2023 Findings (2023)
Keywords: lottery ticket hypothesis
negative interference
zero-shot neural architecture search
multilingual pretrained language model

Abstract: The curse of multilinguality in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages, especially when the capacity is limited. While increasing the capacity may appear intuitive for overcoming this curse,
it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping capacity per each language more or less the same. Specifically, we first scale the model to reduce interference, then search for a per-language subnetwork, or a lottery ticket, with comparable performance to the full model. According to lottery ticket hypothesis, this scale-then-find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters to keep the parameters reduced. Finally, to avoid the cost of multiple retraining for searching multilingual tickets, we explore zero-shot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the inference cost of models for each languages, while boosting the performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is publicly available.

Multilingual Lottery Tickets to Pretrain Language Models
Jaeseong Lee and Seung-won Hwang"
Computer Science and Engineering, Seoul National University jsleelldi. ac.uk, seungwonh ac. kc.
Abstract
tween languages (Wang et al., 2020b) has been observed, known as curse of multilingualitty (Con- neau et al., 2020).
The curse multilinguality in training multi- lingual pretrained language models (mPLMs) refers to the negative interference between lan- guages, especially when the capacity is limited. While increasing the capacity may appear in- tuitive for overcoming this curse, it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping the capacity per each language more less the same. Specifically, we first scale the model to reduce interference, then search for a per- language subnetwork, comparable performance to the full model. Ac- cording to lottery ticket hypothesis, this seale- then-find-tie find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters keep the parameters reduced. Fi- nally, avoid the cost of multiple retraining for searching multilingual tickets, we explore zero- shot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the infer- ence cost of models for each languages, while boosting performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is pub- licly available.
For the problem of signals from languages inter- fering with each other, a naive solution would be increase the capacity per each language (Conneau et al., 2020; Pfeiffer et al., 2022). For example, Figure 1b illustrates adding additional parameters per each language to the shared model. This design intuitively improves the average performance of each language and allows the gradient conflict to be alleviated, but with the cost of enlarging per- language parameter size, which affects both the training and inference costs of the model.
lottery ticket, with
Our distinction is keeping the per-language pacity similar reach the same goal. To achieve multiple competing goals -improving the perfor- mance and mitigating the gradient conflict, without increasing per-language capacity- we invite the lot- tery ticket hypothesis (Frankle and Carbin, 2019). Lottery ticket hypothesis claims that dense mod- els contain subnetworks -called "winning tickets"- whose performance is at least similar to the full model. This hypothesis is empirically shown to be true for popular language model architec- tures (Prasanna et al., 2020; Chen et al., 2020a; Zheng et al., 2022).
Our key idea is search tickets per each lan- guage, i.c., multilingual tickets, that can achieve all of the competing goals, by scaling the model and then searching for such tickets. The parameters of each language will be redistributed in the scaled model to keep the per-language capacity unaffected by scaling. To illustrate, in Figure Ic, each of / and 12 maintains the same capacity before (Figure la) and after scaling (Figure Ic) as 9 parameters. This will improve the performance, since the perfor- mance from tickets will be similar to the scaled model, by the lottery ticket hypothesis. Moreover, disseminating the parameters will also ii) mitigate the negative interference.
1 Introduction
Multilingual pretrained language models (mPLMs) (Devlin al., 2019; Conneau et al., 2020) have become the de-facto standard for multilingual tasks. These models, by mapping multiple languages through shared parameters (Figure la), may benefit from positive transfer between languages (Pires et al., 2019; Conneau and Lample, 2019).
However, when more languages are covered, or parameters are limited, negative interference be-
"Corresponding author

---

params used by/ 4
params used by la
(a) mBERT
(b) increasing per-lang capacity
(c) keeping per-lang capacity (ours)
Figure I: Comparison between various multilingual pretraining methods. (a): The na've multilingual pretraining. where all the languages share all parameters. However it suffers negative interference between languages. (b): Increasing per-language capacity. (c): Keeping per-language capacity, while mitigating the interference between languages.
However, finding multilingual tickets in scaled model incurs prohibitive cost -even to find sin- gle ticket, Frankle and Carbin (2019) train model multiple times. To overcome, problem as neural architecture search (NAS) (Zoph and Le, 2017; Liu et al., 2019) per each language. Particularly, we explore recently emerging zero- shot NAS (Abdelfattah et al., 2021; Javaheripi et al., 2022; Shu et al., 2022), which aims to search archi- tecture without training cost. We discriminate the most appropriate zero-shot NAS method to remove the burden of finding multilingual tickets.
task performance while keeping the capacity and computational complexity.
Our code publicly available.
interpret this
Preliminaries
2.1 Negative Interference in mPLM
Pretraining an mPLM utilizing corpora of multi- ple N languages aims to leverage positive cross- lingual transfer (Pires et al., 2019; Devlin et al., 2019; Conncau and Lample, 2019). However, Wang et al. (2020b) unveil negative interference in mPLMs.
Finally, verify claims, devise metric measure the negative interference during mul- tilingual pretraining. Our measurements indicate that multilingual tickets decrease negative interfer- ence. Our experiments show that multilingual tick- ets we found increase task performance while main- taining the capacity as expected. In our qualitative analysis, the locality of subnetworks preserves the linguistic similarity within the same family well. Our contributions can be summarized as follows:
Specifically, they measure the interference from bilingual pretraining (N = 2), using cosine similar- ity between gradients (Yu et al., 2020), originally devised measure the interference from multitask training. First, the total loss from multitask train- ing L(0) can be denoted sum of each task loss (#). Then the total gradient can be decomposed as follows:
VC(0) = v Σ Ci(0) = Σ (i)))
(1)
We propose a novel method to alleviate nega- tive interference during multilingual pretrain- ing: Searching multilingual tickets.
where (0) (O). They consider two tasks and j interfere with each other if the cosine similar- ity between 9: and 9; low. Wang et al. (2020b) use this metric to measure the interference in bilin- gual pretraining as follows:
We explore the most appropriate zero-shot NAS method to remove the cost of finding multilingual tickets.
gc(0) cos (0). (2))
(2)
Experiments show that multilingual tickets do alleviate negative interference, increasing the

---

where 91 and 92 are gradients from each language,
manually-designed neural architectures, they were extremely computationally expensive -for example, AmoebaNet needed days on 450 enterprise GPUs. Therefore, reducing the search cost attracted keen research interests (Pham et al., 2018; Liu et al., 2019). Recently, zero-shot NAS (Abdelfattah et al., 2021; Mellor et al., 2021: Javaheripi et al., 2022) emerged, whose goal is to make the search cost almost negligible. We study how leverage zero- shot NAS for our purpose of finding subnetworks per each language.
In this measure. the inference increases as ge(0)
becomes lower, which is maximized when gc(0)
is 1,, indicating 91 (f) and g2(f) are in opposite directions, canceling the updates for each other. Wang et al. (2020b) reveal that gc(0) is lower in bilingual pretraining (NV 2) compared with monolingual pretraining (N = 1). implying nega- tive interference happens during multilingual pre- training.
Our strategy is finding a lottery ticket for scaling up and reducing interference, shown in the follow- ing subsection.
Proposed Method: Multilingual Lottery Tickets with Zero-Shot NAS
2.2 Lottery Ticket Hypothesis
This section presents our method, scaling the model and then finding tickets m, per each language 4 ($3.1), leveraging zero-shot NAS techniques ($3.2).
Lottery ticket hypothesis (Frankle and Carbin, 2019) states that every dense model contains some subnetwork whose performance is at least similar to the dense model. Formally, given the initial parameter of the dense model 0, let p(0) be the performance of the network. The Lottery ticket hypothesis is denoted as follows:
3.1 Seale-then-Scare Multilingual Tickets
Lower Interference, Higher Performance Al- lowing more space for each language to operate without interfering with each other has been shown to be beneficial in previous studies (Conneau al., 2020; Pfeiffer et al., 2022). However, such a change would increase the total parameter size, which in turn increases both training and inference costs.
3m E {0,1] 10,11dimm s.t. p(m + )0) 2 p(0)
(3)
where m @ 0 denotes the pruned subnetwork, and dim 0 denotes the dimensionality of 0. They named m as "wiining ticket"2 of the given dense network. Frankle and Carbin (2019) find the ticket by it- erative train-and-prune In each stage, they train the model starting from the subnetwork from the previous stage, and prune some parameters with the least magnitudes. They repeat this expensive process for multiple stages to get the final ticket m, known as iterative magnitude pruning.
Our distinction is first to scale the baseline model, specifically by increasing the number of layers, from the initial model 7o (3 in Figure la) to 77, (4 in Figure Ic). We then redistribute the per-language parameters, by finding subnetworks maintaining the initial per-language parameter size. However, would scaling the model from to O' then finding the lottery ticket m' from e' perform better than p(0)? To answer this question, we rein- terpret Equation 3. Since scaling (He et al., 2016; ?; OpenAl, 2023) is a trustworthy method to improve performance, or, p(0') > p(0), it has the effect of raising a lower bound performance of p(m' @ 0'). In conclusion, once we successfully identify the per-language tickets from the scaled model, not only will the interference be alleviated, but the performance will also be enhanced. To ensure sim- ilar per-language capacity and computational com- plexity during the search procedure, it is crucial to carefully design the search space to apply neural architecture search (NAS) techniques.
Our key idea is finding tickets per each language, i.e., finding subnetwork m, @ for every language 4 E f4..... [N}. However, iterative magnitude pruning N ~ 100 languages in our target prob- lem of multilingual pretraining incurs a prohibitive cost. Thus we formulate our target search as zero-cost Neural Architecture Search (NAS) per each language, removing the cost of searching the architectures.
2.3 (Zero-shot) Neural Architecture Search
NAS (Zoph and Le, 2017: Zoph et al., 2018) aims to automatically search for superior architectures in the given search space (Elsken et al., 2019). While the seminal works, NASNet (Zoph et al.. 2018) or AmoebaNet (Real et al., 2019), outperformed the
NAS Search Space Our goal in finding multilin- gual tickets is to keep computational complexity
For simplicity, we will abbreviate as "ticket".

---

and capacity, rather unaffected by scaling. First, regarding computational complexity for matrix multiplication, we mask rows or columns from key and query matrices K, Q, respectively, for pruning subnetwork. When masking the entire row of K (and a column Q) reduces the cost of matrix multiplication, we constrain masking rules to favor those with lower multiplication costs. Similarly, when mask the rows of value matrix V and columns of the following linear layer Wo or, for the rows of W1 and columns of 22. The total masked capacity is constrained to be similar to the baseline model.
input-invariance, they use the input filled with 1s, which is counter-intuitive our purpose of finding different structures per different lan- guage inputs. We thus resort to adaatie zero-shot NAS methods.
(b) Transformer-friendly Our search space is based transformers (Vaswani et al., 2017) 7). However, some NAS methods (Mellor et al., 2021; Lin al., 2021a) rely on particular tributes of CNNs, since the NAS methods mainly developed from searching CNN ar- chitectures. For example, JACOV_COV (Mel- lor et al., 2021) assumes the network uses ReLU (Fukushima, 1969), which is typically true for CNNs. However NLP models based on transformers architecture (Devlin al.. 2019; Brown al., 2020; OpenAI, 2023) mostly use GeLU (Hendrycks and Gimpel, 2016) instead of Such approaches are not practical for our goal.
Formally, given input hú, we mask the layers to get the output ho as follows:
h1 smaxx (Mg Qua) Kh₂)²
Mah Vh,)
Vd
h22 (No Wo'" h1 bo
h3 LNo(ho h1)
h4 = (Nw W " H(w W1h3 + sw" b1) + b2 ho (h2 + h4)
To this end, we choose SNIP (Lee et al., 2019) as the most appropriate zero-shot NAS method for searching multilingual tickets. SNIP calculates the score of the subnetwork as follows:
where represents a scalar product along rows, smax denotes the softmax function, and o is the activation function.
Second, regarding controlling capacity, we let the NAS algorithm search through the various can- didate tickets of m = Pag Moi Mws where is the concatenation. To maintain capacity and compu- tational complexity, we set r /m/o// dim (m) Rs no/ns, where llo denotes the number of non-zero components.
OC
S(0) = Σ S(OR) Σ &
(4)
00p
Note that this equation the same as impor- tance (Molchanov et al., 2019), when we consider measuring the scores of m only (Michel et al., 2019). Importantly, this score relies on the given inputs, which satisfies (a). Moreover, SNIP (or, importance) has use cases in transformer architec- tures (Michel et al., 2019; Prasanna al., 2020), which supports (b).
3.2 Zero-Shot NAS
Choosing Zero-Shot NAS On the defined search space, we will search for tickets while minimizing the search cost. For this goal, need to choose specific zero-shot NAS algorithm. To narrow it down, we describe the characteristics of the zero- shot NAS method we need.
Searching Multilingual Tickets with SNIP With the NAS search space defined, and the zero- shot NAS chosen, we will search tickets per each language 4, i.c., determine the subnetwork m, @ 0 with the highest SNIP score (Eq. 4). Formally, we need to maximize as follows:
Input-adaptive Our goal is adapting subnet- work my languages. However, recent zero-shot NAS methods (Tanaka et al., 2020; Javaheripi et al., 2022; Zhou et al., 2022; Shu et al., 2022; Sun et al., 2022), such as Syn- flow or TF-TAS, aims to find a task-specific structure that remains invariant for inputs. For
ac Ōmik
max S(m, +.)
8) mmxx
Σ k
mi,k
(5)
Since S(m;,k is always non-negative, and mi, E {0, 1}. maximizing the Σ. S(Ox) is set- ting m,,k = when S(m,,) is within the top r%
FFor simplicity, we describe with the transformer layer with single-head attention. We similarly generalize to the transformer layers with multi-head attention.

---

of the SNIP values.* Therefore, once collect
4000%
the gradients 8C with the initial weight 0 and 8m,, given input data from 4, we can easily decide the ticket m,.
3500%
3000%
2500%
2000%
1500%
3.3 On Stably Measuring Negative Interference
1000%
500%
Though commonly used measurement of negative interference is cosine similarity between gradients (Eq. 2), denoted as gc (orange) in Figure 2, its measurement is highly variant, the metric change is significant over training steps.
-50%%
gc
(ours)
Figure 2: Relative difference of metrics for negative interference per every 10 steps in mBERT pretraining. Our proposed metric (red) much stabler than cosine similarity between gradients (orange).
Such a variance makes comparing the reported performance at the given step highly unreliable. We thus propose to compare a cumulative metric: We evaluate the cosine similarity between the cu- mulated parameter updates by each language. For- tunately in our case, since each batch consists of a single language, we can easily decompose the total updates from languages. Formally, suppose the language ((t) used in each step 6. Then we define our metric to evaluate negative interference as follows:
Are the searched multilingual tickets better than random tickets?
RQ2:
Do multilingual tickets mitigate the neg- ative interference?
RQ3:
RQ4:
Is our method better than input- invariant NAS method?
T u,(()) Σ ((1(1) 1;)
How much more computations would be needed to match our improvement with naive mBERT?
RQ5:
(6)
(0:)(Or) cos(u: (0()), Ug (0r))
4.1 Experimental Settings
where ((((0) = 4) is if and only (()) = 4, and & the parameter update from step t. We regard negative interference between language I; and 1 is larger as (fr) is smaller.
Unlabeled Datasets and Languages for Pretrain- ing We utilize Wikipedia dumps of the same lan- guages mBERT (Devlin et al., 2019) used. We extract Wikipedia articles using WIKEXTRACTOR. To split the sentences, 2020) for Thai (th), and TRANKIT (Nguyen et al., 2021) for Hebrew (he), Armenian (hy), and Arabic (ar). We also use Arabic version of TRANKIT for azb, ka, my, new, pnb. Otherwise, we split the sentences using MOSES (Koehn al., 2007).
Our proposed metric compares accumulated effect of the influences, which is more tolerant to steps: In Figure 2, we show the relative difference cumulative metric uc (red) is much stabler than ge (orange). Thus, we will use metric uc in the following experimental section.
use STANZA (Qi et al.,
4 Experiments
Task Datasets and Languages for Evaluation We focus on evaluating in-language performance using XTREME benchmarks (Hu et al., 2020; Ruder et al., 2021). Since we focus on in langage performance, deal with the NER and POS tasks. which are the only tasks available for tens of lan- guages in the benchmarks. Moreover, the languages with sufficient amount of train and test data, for reliable evaluation. 6 These results evaluating 42 languages over language families
In this section, tackle the following research questions:
RQ1: Do multilingual tickets improve perfor- mance?
restrict
*We normalize the SNIP score by layers, similar to the normalization importance score by Michel (2019). We choose the top % values after collecting SNIP values over all languages.
"Note that the positive transfer would be larger as (frr)(f)) hecomes larger, following the convention of (Wang et al., 2020bb.
WWe omit the language from the given task, the train or test data size is not larger than 100.

---

en
pl
ja
uk
ar
pt
mBERT
80.39
86.30
87.6
88.0
85.35
89.31
88. 46
88. 12
61
45
87
68
89 99
84.34
89.35
90.43
90.08
random tickets (%2 = 14)
80.35
86.31 88.11
88.86
85.26
89,30
88.50
88.35
61.19
88.04
89.79
84.59
89.22
90.14
90.23
random tickets (%, = 16)
80.22
86.04
87.80
88.81
85.21
89.03
88.07
88.05
61.03
87.78
89.84
84.07
89.34
90.54
90.10
multilingual tickets (n. 14)
80.56
86.27
87.90
88.91
85.17
89.33
79 88.37
61.54
88.15
89.93
84.53
89.22
90.26
90.20
multilinguul tickets 16)
81.05
86.75
88.51
89.30
85.92
89.70
89.01
88.62
61.68
88.24
90.59
85.10
89.89
90.81
90.54
bg
kk
et
el
ax
MBERT
82,89 88.1 16 89.98 89,53
91 48
87.99
90.61
79.03
89.95
81 23 88.29
87.04
85. 85.97
84.22
91.88
random tickets (%2 = 14)
82.68 88.31 89.87 89,91
91.54
88.05
91 14
78.88
89.73
80.86
88.37 86.85
85.75
83.28
91.77
random tickets (n. 16)
82.80 88.0 01.01 89.85 89.81
91.63
88.05
90.22 78
89.95
81 89 88 28
87.04
85,
84,18
90.86
multilingual tickets (1, 14)
83.35 88.61 90.01
90.12
91.97
88.31
90.20
79.57
89.89
82.69
88.69
87.05
86,39
84.40
91.51
multilingual tickets 16)
83.66
88.90 90.40
90.46
91.89
88.63
92.04
80.13
90.77
82.06
89.09
87.77
86.99
85.86
91.61
ka
mr
te
SW
11
zh
avg
FLOPS(M)
mBERT
80.69
76.37 50,78
79.06
91.17
84,96
18,19
76.48
72.81
85.32
92.59
72.73
84.21
341.7
random tickets (%2 = 14)
79.64 76.10 49.42
77.55
89.77
83.67
75.35
76.02
69.76
86.51
92.50
72.32
83.92
338.6
rundom tickets (n.. 16)
80.45
75.72
48.20
78.45
90.34
83,90
76,59
76.00
70.70
86.57
91.96
73.06
83,92
341.3
multilingual tickets 14)
81.05
76.36
47.18
78.00
90.33
85.67
78.81
76.82
73.09
86.02
93.16
72.60
84.31
335.5
multilingual tickets 16)
81.72 77.06
52.34
79.38
90.99 86.34
79.89
76.70
72.73 86.90
92.93 73.25
84.91
337.7
Table I: Fl scores on NER task over 42 languages. We average the score over 5 runs
and isolate: af, ar, az, bg, bn, de, el,, en, es, et, eu, fa, fi, fr, he, hi, hu, id, it, ja, ka, kk, ko, It, ml, mr, ms, nl, pl, pt, ro, ru, sw, ta te, th, tl, tr, uk, ur, vi, zh.
Comparisons
We compare the following mod-
cls:
mBERT: Naive multilingual pretraining with the baseline dense model architecture.
random tickets:
Multilingual pretraining uti- lizing the randomly selected tickets on the defined search space.
Implementation Details The baseline dense model follows the BERT architecture (Devlin et al., 2019). We use 70 = 12 layers, with 768 hidden units where the intermediate linear layer W1 ex- pands the dimension to 3072. We scale the model so that 7)s becomes 14 or 16. When searching for multilingual tickets, we accumulate gradients from 2.5M tokens to calculate the SNIP values (Eq. 4). This takes only about 2-3 minutes 1 RTX 3090 per language.
multilingual tickets: Multilingual pretraining with the multilingual tickets found on the de- fined search space.
4.2 Experimental Results
RQI: Effectiveness of Multilingual Tickets The results presented Tables 1 and 2 demon- strate the significant computational efficiency of our proposed multilingual ticket variants compared to other approaches. For example, multilingual tickets (1/- = 16) outperforms mBERT, while re- quiring fewer FLOPss for inference.
To pretrain the model, we follow the default setting described in Devlin et al. (2019). We over- sample low-resource languages with an exponential smoothing factor of 0.7. We use learning rate of le-4, batch size of 256, and update for 1M steps. We use sequence length of 128 for 90% of the updates and 512 for the last 10%. Pretraining is conducted on TPUv3-8.
RQ2: Effectiveness of Chosen Zero-Shot NAS To demonstrate the efficacy of our chosen zero- shot NAS approach, we compare it with randomly selected tickets from the same search space. Ta- ble and 2 highlight the substantial performance improvement achieved by our cost-free search method; For instance, random tickets (7+ = 16) even suffer performance degradation when com- pared to mBERT, while our multilingual tickets outperform
Our evaluation settings largely follow the XTREME benchmarks (Hu et al., 2020; Ruder et al., 2021). We fine-tune the pretrained mod- els with batch size of 32 and a learning rate of 2e-5. We generally fine-tune them for two epochs. But for NER, since the training dataset for some languages is scarce, we ensure to update parame- ters for least 2500 iterations. We run 5 times per each language, and report the average score over all languages. Fine-tuning is conducted on RTX 3090.
RQ3: Alleviated Interference by multilingual tickets To confirm that the discovered multilin- gual tickets mitigate negative interference, we com- pare the average uc (Eq. 7) values. In a controlled

---

pt
1d
94.86 97.08
96.25
97
93.33
66
97.77
87
84,
95
40
82
93.18
94.65
77.78
76.71 91.93
tickets (1), 14)
94.85 97.08
95.94
95.
97.76
87.33
93
77.66
76.62
random tickets (1). 16)
94.94 97.06
98.18
97.62
95.31
95.69
97.75
86.93
85.25
95 48 82.62 93
94.61
77.67
76.57 92.02
multilingual tickets (17,
14)
94.90
97.08
96.26
97.64
95.75
97.87
87.31
85
93
79
77.78
76.77 92.26
multilingual tickets (1). 16)
94.91
97.09 95.67 tr
96.43
97.73
95.32
95.78
97.91
87.34
85
95.80
93.24 84.08
95.24
77.92
76.98 92.39
Tо
GU
bg
et
ta
af
avg
FLOPS(M)
MBERT
75.86 94. 10 96.35 89.55 88.73 97.80 94.62 97.19 89.57 91.15 90..3 83,40 96 70 87 87.82
91.23
341.7
random tickets (1), = 14)
76.23 93.70 96.29 90.1 88.51 97 94 94, 96.77 89 68 90. 90.52 82, 88 96 73 87 87.93
91.17
338.6
random tickets (1), = 16)
76.10 94.20 96.40 90.20 88 82 97.90 94.79 97.05 90.05 90.80 90.56 82, 70 88.04
91.24
341.3
multilingual tickets (1), 14)
76.42 94.01 96.42 90.41 88.79 98. 10 94, 98 97.01 90.04 90.78 90..92
82,91 96
88 26
91.38
335.5
multilingual tickets (1) 16)
76.69 94.50 96.48 90.58 89.66 98.05 95.05 97.27 90.70 91.09 91.00 82.83 96.97 88.28
91.55
337.7
Table 2: Fl scores POS task over 31 languages. We average the score 5
NER
POS
FLOPS
mBERT 84.21
91.23
342
mBERT scaled
84.85
91.36
372
multilingual tickets
84.91
91.55
338
Table 4: Comparison between multilingual tickets and scaled mBERT. We report the averaged NER and POS Fl score over all languages.
2
7
9
10
12
loyer
muttilinual tickets
MIBERT
Figure 3: Multilingual tickets alleviates the negative interference, compared with mBERT. x-axis: layer id, y- axis: metric uc ($3.3) indicating how much interference is mitigated.
tude pruning, the input-invariant zero-shot NAS method. This highlights that input-adaptive NAS is essential.
RQ5: Computational Efficiency of Our Method To estimate our improvement as computational cost, compare with scaled version of naïvely pretrained mBERT. We scale hidden units by 10%. Table shows that even if we add 10% of FLOPs to the baseline, multilingual tickets outperforms it.
input NER POS FLOPS
magnitude pruning invariant 84.73 91.31
342
SNIP (ours) adaptive 84.91 91.55
338
Table 3: Comparison between multilingual tickets with input-invariant zero-shot NAS (magnitude pruning) and input-adaptive zero-shot NAS (SNIP). We report the averaged NER and POS Fl score over all languages.
4.3 Analysis: Ticket Similarity and Language Relatedness
We further analyze the effectiveness of our mul- tilingual tickets and provide deeper insights into their characteristics. Our hypothesis is that the multilingual tickets capture language relatedness, as related languages are less prone negative in- terference (Wang et al., 2020b) and benefit more from the positive transfer (Pires al., 2019; Khem- chandani et al., 2021; Muller et al., 2021). The redistribution strategy of parameters (Figure 1c), effective, should thus favor sharing parameters among related languages.
experiment, search for the multilingual tick- ets on the baseline model without scaling. With a setting of r = 0.85, we calculate the average ue for each layer at 10K update steps, considering all languages.
As depicted in Figure 3, the ue values of the mul- tilingual tickets are higher compared to mBERT, indicating a reduction in negative interference.
RQ4: Superiority of Input-Adaptive NAS We emphasize that the selection of input-adaptive NAS is important. We establish another comparison of multilingual tickets built with representative amput-invariat zero-shot NAS method, magnitude pruning (Frankle al., 2021). We 7/s as 16 for both methods.
To investigate whether our multilingual tickets exhibit such characteristics, we project the found tickets for each language using UMAP (McInnes et al., 2018). For comparison, we also project ran- domly selected tickets. We identify language fam- ilies using the Glottolog database (Hammarström et al., 2021) to list related languages.
Table 3 shows that SNIP, the input-adaptive zero-shot NAS method, outperforms the magni-

---

new
ba
ga
ml
ko
sh
lnn
e hi
fr
sk
my
kkc
no
ben
he
Afro-tsiatic
Austroasiatic
Artificial Language
atlantic-Congo
RP
th
bpy
ka
Austronesian
Ima
Besque (Isolate)
fa
ni
est
Dravidian
ST
bn
ap
IE_Armenic
da
scn
bg
en
18, Balto-Slavic
Teltic
IE_ Germanic
Graeco Phryygann
Imo
an
UT
ammb
mg
It_ltalic
bayn
gu
dg
ban
lb
nl
SCD
en
Japonic
Kartvelian
mieh
Korcanic
bg
hiuk
kg
PPs
km29mm
Rinww
kk
azb
sumin
ceb
adf
ragle
Sino-Tibetan
Mongolic-Khiine Khipii
Nukht-Daghestarian
sk
cs
te
Tei- Kadei
Turkic
Uralic
Figure 4: Random tickets (top) and the found multilingual tickets (bottom) projected via UMAP. Each color corresponds to a specific language family. Multilingual tickets from same language families are similar to each other.
Figure 4 illustrates the results, where our multi- lingual tickets (bottom) show similarity among lan- guages from the same language family (dots with the same color). This is in stark contrast to the ran- dom tickets (top). For example, in our multilingual tickets, the tickets of the artificially constructed languages Volapük (vo) and Ido (io) close to each other, while they are positioned at the left and right edges in the random tickets. Similarly, natu- ral languages belonging to the Germanic language family exhibit similar characteristics.
cally learn language relatedness with only a few unlabeled language tokens and negligible computa- tional cost. This finding greatly benefits multilin- gual pretraining.
Related Work
5.1 Subnetworks for mPLMs
Several previous works have explored the of language-specific subnetworks in mPLM. For the translation task, Lin et al. (2021b) identify language-speciff subnetwork based on the magni- tude of neurons after some update, then further fine- tune it with the data of specific language pairs. Xie et al. (2021) investigate important neurons using importance score, then follow similar approach. However, all of these assume that the negative in- terference incurred in the pretraining stage can be alleviated in a post-hoc manner, which is chal- lenged by Pfeiffer et al. (2022). Our distinction eradicating the interference from the pretraining stage, such that the post-hoc methods can be ap- plied complementarily to our approach. The closest work to ours is S3Net (Lu et al., 2022), which uti- lizes language-specific subnetworks to pretrained mPLMs for automatic speech recognition (ASR). However it requires significant computational cost to find such subnetworks, whereas our method re- moves the need for expensive subnetwork search using zero-shot NAS.
One might question the Balto-Slavic languages counter-example, as they appear in two clus- ters at the bottom of Figure However, with a more fine-grat fine-grained taxonomy, we can further ver- ify that the multilingual tickets correlate with lin- guistic genealogy. The Balto-Slavic languages on the left side belong to the East Slavic or Fastern South Slavic languages, while the languages on the right side belong to the West Slavic, Western South Slavie, and Baltic languages. One exception is Ser- bian (sr), which belongs to Western South Slavic but is closer to the left cluster. We hypothesize that since Serbian shares dialects, such as Torlak, with some Eastern South Slavic languages (Kortmann and van der Auwera, 2011), 1), it is positioned closer to the left cluster rather than other Western South Slavic languages.
In summary, our multilingual tickets automati-

---

5.2
NAS for Pretrained Language Models
which may require additional research to sophisti- cate qualitative analysis.
After the success of automatically searched neural networks (Zoph et al., 2018: Real et al., 2019) out- performing the hanually-designae researchers have also applied NAS for NLP mod- els (So al., 2019; Wang et al., 2020a; So al.. 2021; Gao et al.. 2022: Javaheripi al., 2022). While most of these works focus on optimizing models for a single language, DARTS-AST (Chen et al., 2020b) leverages a successful NAS method, DARTS (Liu et al., 2019), to automatically search multilingual ASR model for four languages. Sim- ilarly, Tsai et al. (2020) perform NAS to find shared architecture for multilingual corpora. In contrast, our approach leverages NAS techniques address negative interference by searching for language-specific tickets rather than focusing on the shared model part, as overlooked previous works. Moreover, to the best of our knowledge, we are the first to explore zero-shot NAS methods for mPLMs
Acknowledgements
architectures,
This research was partially supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program 11PP-222-2200000011111 vised by the IITP (Institute for Information & Com- munications Technology Planning & Evaluation). This research was also partially supported by a gift from Google. We would also like to thank Google's s TPU Research Cloud (TRC) program for providing Cloud TPUs.
super-
References
Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. 2021. Zero- cost proxies for lightweight (NAS). In International Conference on Learning Representations.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
6 Conclusion
This paper studied the challenge by balancing the conflicting goals of reducing negative interference while maintaining a similar per-language capac- ity. We proposed a Scale-then-See Seeeerchhh approach, of searching for per-language subnetworks, or lottery tickets, from a scaled model, that improves perfor- mance without increasing per-language capacity. We keep the cost of finding such tickets negligi- ble, by exploring zero-shot NAS method. Our results show that ours reduces negative interference as expected, and the tickets discovered qualitatively preserve linguistic relatedness.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang. Zhangyang Wang. and Michael Carbin. 2020a. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems, volume 33, pages 15834-15846. Curran Associates, Inc.
Limitation
Generalization to Unseen Languages Our re- search primarily focuses on the effectiveness of multilingual tickets and their impact on reducing interference. However, the performance and gen- cralizability of our approach to languages unseen during the pretraining stage, may be limited. Fur- ther investigation and adaptation of the method specifically for resource-poo settings are neces- sary.
Yi-Chen Chen, Jui- Yang Hsu, Cheng- Lee, and Hung-yi Lee. 2020b. DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recog- nition and Adaptation. In Interspeech 2020, pages 1803-1807. ISCA.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised Cross Cross-lingual lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Language Relatedness While we use language relatedness to qualitatively analyze multilingual tickets found, this notion may not cap- ture fine-grained variations in language relatedness,

---

Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Mojan Javaheripi, Gustavo Henrique de Rosa, Sub- habrata Mukherjee, Shital Shah. Tomasz Lukasz Re- liga, Caio Cesar Teodoro Mendes, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. 2022. Lite- TransformerSearch: Training- free neural architecture search for efficient language models. In Advances in Neural Information Processing Systems.
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conferencee on Natural Language Processing ( (Vol- 1: Long Papers), pages 1312-1323, Online. As- sociation for Computational Linguistics.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2021. Pruming neural net- works at initialization: Why are we missing the mark? International Conference on Learning Representa- tions.
Kunihiko Fukushima. 1969. Visual Feature Extraction by Multilayered Network of Analog Threshold Ele- ments. IEEE Transactions on Systems Science and Cybermetics, 5(4):322-333.
Bernd Kortmann and Johan der Auwera, editors. 201 I. The Languages and Linguistics of Europe A Comprehensive Guide. De Gruyter Mouton.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. 2019. SNIP: SINGLE SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIV- ITY. In International Conference on Learning Rep- resentations.
Jiahui Gao, Hang Xu, Han Shi, Xiaozhe Ren, Philip I., H. Yu, Xiaodan Liang, Xin Jiang, and Zhenguo L.. 2022. AutoBERT-Zero: Evolving BERT Backhone from Scratch. Proceedings of the AAAI Conference on Artificial Intelligence, (arteeeee,6110))) (6(10):10663-100
Ming Lin, Pichao Wang. Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. 2021a. Zen-NAS: A Zero-Shot NAS for High- High-Performance Image Recognition. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 347-356.
Harald Hammarström. Robert Forkel, Martin Haspel- math, and Sebastian Bank. 2021. Glotollo/glootoooo:: Glottolog database 4.5.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770- 778.
Zehui Lin, Liwei Wu, Mingxuan Wang. and Lei Li. 202 Learning Language Specific Sub-network for Multilingual Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 293-305, Online. Association for Computational Linguistics.
Dan Hendryeks and Kevin Gimpel. 2016. sian error linear units (gelus). arXiv preprint
Gaus-
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 441 1-- 44211 PMLR.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. In 7th In- ternational Conference on Learning Representations, ICLR 2019.
Yizhou Lu, Mingkun Huang, Xinghua Qu, Pengfei Wei, and Zejun Ma. 2022. Language Adaptive Cross- Lingual Speech Representation Learning with Spanse

---

Sharing Sub-Networks. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6882-6886.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How Multilingual is Multilingual BERT? In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4996- 5001, Florence, Italy. Association for Computational Linguistics.
Leland MeInnes, John Healy, and James Melville. 2018. Uniform manifold approximation and pro- jection for dimension reduction. arXiv preprint arXiv:1802.03226.
Umap:
Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT Plays the Lottery, All Tickets Are Win- ning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3208-3229 Online. Association for Computational Linguistics.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. 2021, Neural Architecture Search without Training. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7588-7598, PMLR.
Peng Qi, Yuhao Zhang. Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Hu- man Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance Estima- tion for Neural Network Pruning. In Proceedings of the LEEE/CVF Conference Computer Vision and Puttern Recognition, pages 11264-11272.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2019. Regularized Evolution for Image Classifier Architecture Search. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4780- 4789.
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When Being Un- seen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 448-462, Online. Association for Computa- tional Linguistics.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Sid- dhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 102 10-455555 Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Vey- sch, and Thien Huu Nguyen. 2021, Trankit: A Light- Weight Transformer-based Toolkit for Multilingual Natural Language Processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: System Demonstrations, pages 80-90, Online, Association for Computational Linguistics.
Yao Shu, Shaofeng Cai, Zhongxiang Dai, Beng Chin Ovi, and Bryan Kian Hsiang Low, 2022. NASI: Label- and data-agnostic neural architecture search at initialization. In International Conference on Learn- ing Representations.
David So. Quoc Le, and Chen Liang. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Pro- ceedings of Machine Learning Research, pages 5877- 5886. PMLR
OpenAl. 2023. GPT-4 Technical Report.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the Curse Multilinguality by Pre-training Modular Transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-344 -495, Seattle, United States. Association for Computational Lin- guistics.
David So, Wojeiech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. 2021. Searching for efficient transformers for language modeling. In Thirty-Fitth Conference on Neural Information Pro- cessing Systems.
Zhenhong Sun, Ming Lin, Xiuyu Sun, Zhiyu Tan, Hao Li. and Rong Jin. 2022. MAE-DET: Revisiting Max- imum Entropy Principle Zero-Shot NAS for Effi- cient Object Detection. In Proceedings of the 39th International Conference Machine Learning, vol- ume 162 of Proceedings of Muchine Learning Re- search, pages 208 20-00226. PMLR.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018 Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning, vol- ume 80 of Proceedings of Machine Learning Re- search, pages 4004-4104 PMLRR

---

Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. 2020. Pruning neural networks with- out any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Sys- lems, volume 33, pages 6377-6389. Curran Asso- ciates, Inc.
on Computer Vision and Pattern Recognition, pages 10894-10880 10903.
Barret Zoph and Quoe Le. 2017. Neural architecture search with reinforcement learning. In International Conference on Learning Representations.
Henry Tsai, Jayden Ooi, Chun-Sung Ferng, Hyung Won Chung, and Jason Riesa. 2020. Finding Fast Trans- formers: One-Shot Neural Architecture Search by Component Composition. arXiv:2008.06800 06808 [cs, stat].
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. 2018. Learning Transferable Archi- tectures for Scalable Image Recognition. In Pro- ceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 8697-8710.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020a. HAT: Hardware-Awand Transformers for Efficient Natural Language Processing. In Proceedings of the 58th Annual Meeting of the Association for Compu- Online. Asso- ciation for Computational Linguistics.
tational Linguistics, pages
Zirui Wang. Zachary C. Lipton, and Yulia Tsvetkov. 2020b. On Negative Interference Multilingual Models: Findings and A Meta-Learning Treatment. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438-4450, Online. Association for Computa- tional Linguistics.
Wanying Xie, Yang Feng, Shuhao Gu, and Dong Yu. 2021. Importance-bastt based Neuron Allocation for Mul- tilingual Neural Machine Translation. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Natural Language Processing (Volume I: Long Papers), pages 5725-5737, Online. Association for Computational Linguistics.
Joint Conference
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine. Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. In Proceed- ings of the 34th International Conference Neu- ral Information Processing Systems, NIPS'20, pages 5824-5836, Red Hook, NY, USA. Curran Associates Inc.
Rui Zheng, Bao Rong, Yuhao Zhou, Di Liang, Sirui Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing Huang. 2022. Robust Lottery Tickets for Pre- Pre-trained Language Models. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2211- 2224, Dublin, Ireland. Association for Computational Linguistics.
Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong 2022. Training-Free Free Transformer Architecture Search. In Proceedings of the IEEE/CVE Conference


Reviews:
[{"review_id": "v1aelDeSNW", "reviewer": "Reviewer_TbBM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged that you find our paper to be novel, exciting and successful. We also point out your concerns are already addressed in the submitted draft, as below:\n\nC1. “The method in this paper may face challenges when dealing with more languages or tasks” Results we reported in the draft were **in the largest-scale setting (103 languages) reported in the literature** (i.e., most of existing works use less than 103), using the widely adopted XTREME-R benchmark for large-scale multilingual evaluation.\n\nC2. “The method mainly focuses on the effect of mitigating negative interference, but the positive transfer needs to be discussed” Positive transfer **is already discussed** in our manuscript. i) Figure 4 shows that our tickets preserve linguistic similarity, which is a critical signal for positive transfer, as suggested in previous works (line 416). ii) Our proposed metric ‘uc’ considers positive transfer also. High value of uc of ours (Figure 3) implies that positive transfer is large. uc measures both negative and positive transfer in one equation, rooted from the convention of previous works (Wang et al. 2020), and we will revise to emphasize this point in the camera ready.\n\nZ. Wang, Z. C. Lipton, and Y. Tsvetkov, “On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment,” in 2020 EMNLP", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel approach is proposed to address the multilingual curse in multilingual pre-trained language models, namely mitigating negative noise by searching for multilingual tickets.\n2. Successfully exploited the zero-shot neural architecture search (NAS) method to reduce the cost of searching for multilingual tickets.\n3. Experimental results show that the method can effectively mitigate negative interference and improve task performance while maintaining capacity and computational complexity.", "reasons_to_reject": "1. The method in this paper may face challenges when dealing with more languages or more complex tasks, and further verification of its effectiveness in large-scale scenarios is required.\n2. The experiments in this paper mainly focus on the effect of mitigating negative interference, but the impact on positive transfer has not been fully discussed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the problem of multilingual curse in multilingual pretrained language models (mPLM), that is, the negative interference between different languages under limited capacity. The authors propose a novel approach to mitigate negative noise by searching for multilingual tickets while keeping the capabilities of each language similar. To reduce the cost of searching for multilingual tickets, the authors explore a zero-shot neural architecture search (NAS) approach. Experimental results demonstrate that the approach can mitigate negative perturbations and improve task performance while maintaining capacity and computational complexity.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MGNphQObyM", "reviewer": "Reviewer_spkU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation. We will make sure to thoroughly revise to reflect all reviewer comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-", "reasons_to_reject": "-", "questions_to_authors": "", "paper_topic_and_main_contributions": "-", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ajxki36FMr", "reviewer": "Reviewer_kJa9", "summary": "", "questions": "- Is the proposed metric for gradient interference just a batched version of `gc` to reduce noise?\n- Is the scaled-up model trained from scratch, or do you use the first 12 layers fixed from mBERT? Figure 1 might be a bit misleading if it's from scratch.\n- Did you compare your NAS approach with something like (non-iterative) magnitude pruning? If so, how does it perform?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you find our work is computationally-cheap, and provides interesting analysis. We will reflect your suggestions (updating figure, adding reference) in the final version. We address your specific questions below.\n\nC1-i. “The idea of training large and then compressing is generally not novel (Zhuohan et al. 2020)” Our contribution is not following the old idea of training large and compressing (LeCun et al. 1989, Han et al. 2016). Our aim is reducing interference between languages, by providing tickets differently per language.\n\nC1-ii. “Is the cost of pre-training from the beginning much lower than finding language-specific subnetworks with a relatively cheap approach?” Comparing with the methods applied after pre-training challenged by previous work, for failing to remove the negative interference incurred in the pretraining stage (line 473).\nOur additional experiment shows that performing magnitude pruning applied after pretraining, though cheap, performs poorly (ex. F1 avg : NER -22, POS -18 from ours). \nMoreover, magnitude pruning with pretraining (and 10% further training compared to ours) is both more expensive and less effective (ex. F1 avg: NER - 2, POS -1 from ours).\nWe thus focus on finding subnetworks before pre-training (We report magnitude pruning before pretrain in our answer below)\n\nC2. Q3. “More baselines, such as magnitude pruning, would be desirable” We reiterate that our aim is redistributing the parameters differently per language to avoid negative interference. Therefore, we stated in section 3.2 that generating the same ticket per language (as in magnitude-based pruning) is not desirable. As the table below reveals, ours **outperforms** magnitude pruning (Frankle et al. 2021). We highlight that POS performance of magnitude pruning is on par with mBERT, while ours improves with a notable amount.\n\n|                   | POS   | NER   | FLOPs(M) |\n| ----------------- | ----- | ----- | ------ |\n| mBERT             | 91.23 | 84.21 | 342    |\n| magnitude pruning | 91.31 | 84.73 | 342    |\n| ours              | 91.55 | 84.91 | 338    |\n\nC3. “The improvement over baseline is quite marginal” Our improvement, when translated into additional computational overhead baselines have to pay, is not marginal– For example. mBERT, even after trained with 10% more FLOPs than ours, performs far below than ours, in our additional evaluation reported below (limited by rebuttal period; We will report more extensive results in the revised draft). \n\n|              | POS   | FLOPs (M) |\n| ------------ | ----- | ------ |\n| mBERT        | 91.23 | 342    |\n| mBERT scaled | 91.36 | 372    | \n| ours         | 91.55 | 338    |\n\nQ1. “Is the proposed metric just a batched version of gc?” No. We compare the parameter updates, not the gradients. In this way, we can exactly compare the effect of each language to the optimization result.\n\nQ2. “Is the scaled-up model trained from scratch, or do you use the first 12 layers from the mBRET?” We trained from scratch.\n\nY. LeCun, J. Denker, and S. Solla, “Optimal Brain Damage,” in NIPS 1989.\nS. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding.,” in ICLR 2016.\nJ. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin, “Pruning neural networks at initialization: Why are we missing the mark?,” in ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A computationally-cheap appraoch for finding sub-networks could be interesting for the community, as e.g., methods based on iterative magnitude pruning, while having good performance, are quite expensive.\n- The analysis on the overlap of similar languages' subnetworks is quite interesting and supports previous works that found language-neutral subnetworks for multiLMs.", "reasons_to_reject": "- The idea of training large and then compressing is generally not novel. It has been investigated by [1], which they also specifically used cheap methods for doing the compression. Moreover, when scaling up the LM, it has to be pretrained from the beginning which is quite computationally-expensive. Is the cost of pre-training much lower than finding language-specific subnetworks with a relatively cheap approach?\n- pruning with a ratio of 12/14 or even 12/16 can even be done with simple magnitude-pruning (without losing considerable performance), so we don't really know the significance of the proposed approach compared to trivial baseline methods. It would be nice if there was a comparison to some baseline methods, rather than only \"random tickets\".\n- The improvement over the baseline method at the end is quite marginal, and the final tickets are also having the same #parameters as the original model, so the only real benefit would be a slight decrease in gradient interference (which doesn't really translate into better empirical performance for e.g. low/high-resource languages).\n\n\n[1] Li, Zhuohan, et al. \"Train big, then compress: Rethinking model size for efficient training and inference of transformers.\"", "questions_to_authors": "- Is the proposed metric for gradient interference just a batched version of `gc` to reduce noise?\n- Is the scaled-up model trained from scratch, or do you use the first 12 layers fixed from mBERT? Figure 1 might be a bit misleading if it's from scratch.\n- Did you compare your NAS approach with something like (non-iterative) magnitude pruning? If so, how does it perform?\n\n", "paper_topic_and_main_contributions": "This paper tries to address the curse of multi-linguality in multiLMs, while maitaining the same #parameters and #FLOPS as the baseline model. They do by first scaling up the language model (compared to the mBERT model), and then searching for language-specific lottery tickets of the scaled model. They constrain the final lottery tickets to have the same number of parameters as the baseline model, which is mBERT in this paper. They propose a computationally-cheap appraoch for finding lottery tickets based on zero-shot neural architecture search (NAS). They perform NAS by computing SNIP scores for language-specific inputs and then keeping the parameters with highest SNIP values. They show that the resulting lottery tickets perform better than both the baseline model (i.e., mBERT), as well as random tickets. (although the improvement is marginal).\nThey also propose a method for measuring gradient interference in the resulting model and show that the resulting tickets interfere less than the baseline model.\nFinally they perform an analysis on the resulting lottery tickets and show that close languages (i.e., languages in the same language family) have more overlapping sub-networks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Ansell, Alan, et al. \"Composable sparse fine-tuning for cross-lingual transfer.\" arXiv preprint arXiv:2110.07560 (2021).\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission deals with language model pre-training. 

The reviewers have praised the paper's approach to the task in hand as computationally efficient but they have also identified a number of shortcomings described in detail in the reviews.
Decision: Accept-Findings