FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
Authors: Konstantin Dobler
Gerard de Melo
Venue: EMNLP 2023 Main (2023)
Keywords: model transfer
multilingual
crosslingual
TL;DR: Embedding initialization for better cross-lingual specialization of pretrained multilingual language models

Abstract: Using model weights pretrained on a high-resource language as a warm start can reduce the need for data and compute to obtain high-quality language models for other, especially low-resource, languages.
However, if we want to use a new tokenizer specialized for the target language, we cannot transfer the source model's embedding matrix.
In this paper, we propose FOCUS - **F**ast **O**verlapping Token **C**ombinations **U**sing **S**parsemax, a novel embedding initialization method that effectively initializes the embedding matrix for a new tokenizer based on information in the source model's embedding matrix.
FOCUS represents newly added tokens as combinations of tokens in the overlap of the source and target vocabularies.
The overlapping tokens are selected based on semantic similarity in an auxiliary static token embedding space.
We focus our study on using the multilingual XLM-R as a source model and empirically show that FOCUS outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER).
We publish our model checkpoints and code on GitHub.

FocUs: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
Konstantin Dobler and Gerard de Melo Hasso Plattner Institute / University of Potsdam (konstantin.cm dobler, gerard. derrrddeemlo))hhp.. de
Abstract
Find similar tokens in target and pettainss socababury overla
Initialie weighted mosn from pretzained embleddiigge
Using model weights pretrained resource language as warm start can reduce the need for data and compute obtain high- quality language models for other, especially low-resouree, languages. However, we want new tokenizer specialized for the tar- get language, we cannot transfer the source model's embedding matrix. In this paper, we propose FocUs Fast Overlapping Token Combinations Using Sparsemax, a novel em- bedding initialization method that initializes the embedding matrix effectively for new source tokenizer based on information model's embedding matrix. FOCUS represents newly added tokens as combinations of tokens in the overlap of the source and target vocab- ularies. The overlapping tokens are selected based on semantic similarity in an auxiliary static token embedding space. We focus study using the multilingual XLM-R source model and empirically show that Focus outperforms random initialization and previous work in language modeling and on range of downstream tasks (NLI, QA, and NER). We publish our checkpoints and code on GitHub.'
high-
Sel weights accurlinng to distanoes
Target Vocabulary (EastText Ecabedding)
Pretruined Vocabalary
(Pretrained Model Embedding)
Figure Illustration of FOCUS's initialization strategy for embeddings of new tokens (blue dot): Find similar tokens (orange dots) an auxiliary fastText embedding space; then initialize the new token as their weighted mean in the pretrained embedding space.
trained on high-resource languages. Crosslingual transfer directly copies the pretrained weights the Transformer layers to the target language model. Subsequently, the model is further adapted to the target language by continued pretraining on unla- beled target language text using the original self- supervised pretraining objective. This sort of train- ing regimen is also known as language adaptive pretraining (LAPT; Chau et al., 2020).
However, the pretrained model's embedding ma- trix cannot be directly transferred if we use a new tokenizer for the target language (Artetxe et al.. 2020; de Vries and Nissim, 2021). Using ap- propriate tokenizers has been shown to be impor- tant for the model's performance on downstream tasks (Rust et al., 2021) and is crucial if the source and target language use different scripts.
1 Introduction
Research on large language models advancing rapidly with powerful new models being published at a break-neck pace (e.g., Zeng et al., 2022a; Le Scao et al., 2022; Touvron et al., 2023). Al- though multilingual models have been released, many of the world's languages are not covered. Multilingual models have also been shown to have subpar performance on under-resourced lan- guages (Wu and Dredze. 2020). Therefore, it is cru- cial to develop methods that harness these advances and make them available for further languages, es- pecially low-resource ones.
We present FOCUs, an embedding initialization method that allows us to transfer information from the source model's pretrained embedding matrix to new embedding matrix for the target language's 's tokenizer. FOcUS is illustrated in Figure 1. The key idea is use overlapping tokens between both tokenizers anchor points and represent new tar- get language tokens as weighted mean of over- lapping tokens' embeddings. This enables us initialize the new embedding matrix in the same semantic space as the pretrained embedding ma-
A promising line of work in this regard focuses on crosslingual transfer of Transformer models pre-
chttps://gitthe httpe//hw..mm//osttttttnnnaele/e

---

trix. We empirically show in extensive experiments across a range of different high-resource and low- resource target languages that FOcUS outperforms various strong baselines both in language modeling as well as on downstream tasks (Natural Language Inference, Question Answering, and Named Entity Recognition).
tasks in extensive experiments using XLM-R as a source model on range of high- and low- resource languages.
We further show that FOCUS is effective also when the target language was not part of the source models' pretraining or only a minimal vocabulary overlap is available.
In our experiments, focus the multilingual XLM-R (Conneau et al., 2020) as a source model and specialize it for single language. Focus is particularly well-positioned to take advantage of multilingual source models due to their larger vo- cabulary and the fact that they have already been pretrained to certain extent on many potential tar- get languages. Additionally, we show that Focus still improves significantly over random initializa- tion even if only minimal vocabulary overlap is available.
FOCUS
Our goal to initialize embeddings for tokens a new, language-specific target vocabulary in the same semantic space as the source model's em- beddings. In this study, we mainly focus on the multilingual source model XLM-R although Fo- CUS can in principle also be applied to monolin- gual source models. We copy all embeddings of shared tokens between source and target tok- enizer for our new embedding matrix. If the target language was already part of the source model's 's pretraining corpus, this takes advantage of target language tokens with pretrained embeddings in the source model's tokenizer. In any case, we take advantage of shared named entities, symbols, num- bers, punctuation, and shared words resulting from code-switching between the target and pretrained vocabularies. Additional target language tokens not present in the source model are represented a linear combination of embeddings of semanti- cally similar shared tokens. Unlike previous work on embedding initialization, this requires neither bilingual dictionaries nor an alignment of embed- ding spaces across different languages (Minixhofer et al., 2022; Zeng al., 2022b). Next, we formally describe Focus.
In previous work, a common approach has been to adapt multilingual models to target languages while simply keeping or extending the original vo- cabulary (Wang et al., 2019; Chau et al., 2020; Wang et al., 2020; Chau and Smith, 2021). When extending the vocabulary, FOCUS can also be ap- plied to initialize embeddings just for the new to- kens. However, advocate considering the set- ting of full vocabulary replacement. Only a fraction the multilingual vocabulary is actually used for any single language, so by fully replacing the large multilingual vocabulary with a language-specific smaller vocabulary we can enable faster training times and smaller models. XLM-R's vocabulary has 250k tokens, and replacing this with language- specific 50k token vocabulary reduces the model size quite dramatically, by over 55%. In our ex- periments, training with a language-specifte 50k token vocabulary is 40% faster than extending the original 250k token vocabulary. We summarize the contributions of our paper as follows:
Details of FOCUS. We obtain input source vocabulary 7 with pretrained embeddings E and a target vocabulary //' with embeddings E', which we seek to initialize. The target vocabulary V is obtained by training a tokenizer on monolingual text in the target language. We use è and è to denote embeddings for individual tokens in ES and E'. respectively. We denote the set of overlapping tokens as 0 V AV For each overlapping token we can copy the pretrained embedding over into our target embedding matrix:
We propose FOCUS, a novel embedding initial- ization method that effectively transfers knowl- edge from a pretrained embedding matrix to one for new, language-specific tokenizer.
We empirically verify the effectiveness of Fo- CUS for language modeling and downstream
"We study minimal overlap consisting exclusively of tokens that are numbers, punctuation, and whitespace. 'Comparing 278 million parameters at 1.1 GB 124 million parameters at R GB.
Yo E D
T - こ.
'Comparing a 50k token vocabulary to an extended 273k token vocabulary with HuggingFace takenizers and tr ansformers on two Nvidia A100 80GB GPUs.
'In Section 4, we also conduct experiments in setting with almost no vocahulary overlap. showing that FocUs can he used to transfer a model to previously unseen languages.

---

Note that we make an assumption here: tokens that are part of the overlap O have sufficiently similar semantics in our source and target vocabularies. For multilingual source models, we can exploit already existing tokens from the target language. Otherwise, this will obviously not always be the case', but through common named entities, code- switched tokens, symbols, numbers, and punctu- ation this assumption will hold reasonably often. We provide an in-depth analysis in Appendix B. Finding an initialization in the same semantic space as the pretrained embeddings is not as easy for the set of non-overlapping ("additional") tokens A = Vi O. To initialize embeddings for the addi- tional tokens, we first train auxiliary embeddings X for all target tokens V (i.e., both O and A.). In experiments,
We then initialize the target embeddings for each additional token as weighted mean over pre- trained embeddings of the overlapping tokens from E,, with the weights given by w.. Due to sparsemax, most of the elements in each Uे will be zero, Note that we use the pretrained embed- dings E. instead of the auxiliary embeddings X. as only the pretrained embeddings are in the same semantic space the rest of the transferred Trans- former layers. Therefore:
Va EaEA A: Ey
Σ OED
Va, Eo.
(6)
Summary. FOCUS uses cheap and fast-to-train static embeddings for tokens in the target vocab- ulary to select semantically similar overlapping tokens for each additional target token. The pre- trained embeddings of the overlapping tokens are then used to initialize embeddings for the additional target tokens. In Appendix B, we provide further implementation details as well as detailed analy- sis of the different types of overlapping tokens we encountered in our experiments.
apply fastText unlabeled target language data pre-tokenized with the target tokenizer for Vt, Individual embeddings in X are denoted by H. Next, we compute the pairwise co- sine similarities between the auxiliary embeddings 3. of tokens in and O so that for any a C A::
ca = [sim (a, 01), ..., sim (a, on))
(2)
where oj is the overlapping token at index i and:
3 Experimental Setup
z.a Fo
sim (a, 0)
(3)
Tol
We perform experiments using XLM-R as our mul- tilingual source model, due its popularity and widespread use. We use the base variant for all experiments. Our language-specific tokeniz- ers are trained in the same way as XLM-R for comparability, specifically SentencePiece tokeniza- tion (Kudo and Richardson, 2018) with the Uni- gram algorithm (Kudo, 2018). We use Hugging- Face tokenizers and a vocabulary size of 50k tokens for all languages.
We convert the similarity scores č. to weights by applying sparsemax (Martins and Astudillo, 2016) over each e.. Sparsemax is a sparse variant of softmax that assigns zero probability mass to low- probability elements, which has previously been used by Tran (2020) in a similar setting. Using sparsemax has the advantage of being able to dy- namically accommodate different degrees of skew in the similarity distribution. In some cases we might have only one or two very similar tokens, in other cases, we might have significantly more. Accordingly, the weights w. of the overlapping tokens are:
3.1 Baselines
To evaluate FOCUS, we compare against multiple strong baselines for embedding initialization well as other methods of adapting XLM-R to target language. We always transfer all layers of XLM-R, except for the embedding. Minixhofer et al. (2022) already demonstrate the superiority of this over random initialization of all weights, so we do not compare against the weak baseline of training model completely from scratch.
W. sparsemax C.) = argmin ||
ča (4)
with A denoting the bility simplex, i.e.,
proba-
A = ( E R
TT 0}.
(5)
"Consider words known as false friends: words with the same spelling different meanings across languages.
XLM-R with the original vocabulary. We re- port results of using XLM-R off-the-shelf without
"In principle, we could instead also obtain token embed- dings from already pretrained fastText word embeddings fol- lowing Minixhofer et al. (2022). We show in Section 4 that training TastText directly at the token level provides a better initialization.
8 "A of October 2023, XLM-R has 12.2 million downloads per month on the HuggingFace Hub.

---

language-adaptive pretraining (LAPT) as well as after adapting XLM-R the target language with the original vocabulary kept as-is.
Language
Dataset Size (GB)
German
18 GB
Arabic
5,4 GB
Random Initialization. For vocabulary replace- ment with language-specitne tokenizer and ran- dom embedding initialization, we copy the orig- inal pretrained embeddings following Zoph al. (2016). This randomly maps pretrained embed- dings to tokens in the new vocabulary and per- formed slightly better than other types of random initialization in preliminary experiments." In this case, we also consider the variant of training just the embeddings for an additional 20% of training steps before unfreezing the rest of the network (called 2-STAGE-LAPHY
Kiswahili
0.3 GB
Hausa
0.06 GB
isiXhosa
0.03 GB
Table 1: Size of datasets from CC100 used for LAPT.
the pretrained vocabulary before applying WECH- SEL, which we dub WECHSELEN- This yields an initialization method similar the mixture map- ping method proposed by Wang et al. (2019).
Vocabulary Extension. We also run experiments with vocabulary extension following Wang et al. (2020) by extending with the top 30k tokens of the language-specific tokenizer as well as using Focus to initialize embeddings for the extended tokens.
De Vries and Nissim (2021) note that this allows the new embeddings to adapt the transferred Transformer layers pre- vent catastrophic forgetting. Therefore, this strong baseline is trained 20% longer than other methods.
WECHSEL. We additionally compare against us- ing WECHSEL (Minixhofer et al., 2022) to initial- ize the embedding matrix for the language-specific tokenizer. WECHSEL is method for embedding initialization originally designed for transferring monolingual source models. It relies on aligning pretrained word embeddings for the source and target languages using the Orthogonal Procrustes method (Schönemann, 1966) with bilingual dictio- naries as seed data. Then, each source and target token is embedded into the same semantic space using the out-of-vocabulat method of fastText, re- sulting in aligned static token embeddings for both languages.
3.2 Language-Adapten Pretraining (LAPT) For LAPT, we use the same self-supervised Masked Language Modeling (MLM) objective as in the original pretraining of XLM-R. We use the CC100 corpus to obtain unlabeled text in our target lan- guages, which was also already used for the pre- training of XLM-R (Conneau et al., 2020). There- fore, do not introduce any new unseen data. We show dataset sizes for our target languages in Ta- ble 1. We use the same hyperparameters for all languages, as detailed in Appendix A. In particular, we use a batch size of 128 with chunked sequences of 256 tokens and train our models on 50 million samples (resulting in total of 12.8 billion training tokens and 390k optimizer steps).
To faithfully apply WECHSEL with multilin- gual source model, we would need to provide a word embedding space for all the languages that are part of the multilingual models' pretraining cor- pus. Also, gathering bilingual dictionarics from all source languages to the target language would become a challenge. Instead, upply WECHSEL as-is using only pretrained English fastText word embeddings for the source model. This effectively assumes that all pretrained source token embed- dings are English, which is a rough but not entirely unreasonable assumption given the predominance of English over other languages in the pretraining corpus of XLM-R. We can further commit to this assumption by deleting all non-English tokens from
3.3 Evaluation
We also evaluate models downstream tasks in their respective target languages. We perform downstream task evaluation on five languages: Ger- man, Arabic, Kiswahili, isiXhosa, and Hausa. They were chosen to provide a mix of high-, medium- and low-resource languages, typological and script diversity while satisfying the practical constraints of available evaluation datasets. We refer to Ger- as high-resource, Arabic and Kiswahili medium-resource, and isiXhosa and Hausa as low- resource languages.
We use the translated training sets of XNLI (Con- neau et al., 2018) to evaluate Natural Language Inference (NLJ) in the translate-train setting. To
"We refer to this as random initialization. Random map- ping performed slightly hetter than initialization from a normal or uniform distribution.

---

evaluate Question Answering, we use German- QuAD (for German, Möller al., 2021) and Ty- DiQA GoldP (for Swahili and Arabic; Clark et al.. 2020). We perform Named Entity Recognition (NER) experiments using the balanced train-dev- test split of WikiANN (Rahimi et al., 2019; Pan et al., 2017). Additionally, we evaluate NER for German the GermEva12014 dataset (Benikova al., 2014) and for Swahili, Hausa, and Xhosa using M (Adelani et al., 2022). If there is no dedicated dev split, we construct our own with a random sample 10% of the train- ing data. We perform model selection on the dev split and report the selected checkpoint's result on the test set. We report accuracy for XNLI and F1- scores otherwise. We run all experiments five times with different random seeds and report the mean and standard deviation. Hyperparameters for all evaluation tasks are given in Appendix A.
ing. Naturally, the effect of initialization is less pronounced the longer we train the models. We have deliberately constructed a difficult evaluation with our long training regime of 12.8 billion tokens. In settings where less compute is available, Focus may be even more beneficial.
The improved effectiveness on the pretraining objective also translates to gains in downstream tasks, as reported in Table 2 and Table 3. Fo- CUS initialization outperforms random initializa- tion across all downstream tasks and languages (except for Arabic TyDiQA). WECHSEL also im- proves over random initialization, but FOCUS ob- tains superior results. FOCUS also be applied for vocabulary extension instead of vocabulary re- placement. Here, we see less of an improvement over the random initialization baseline. This could be due to the smaller impact of Focus, since only a relatively small percentage of the large extended vocabulary is affected.
Furthermore, we evaluate the initialization per- formance of FOCUS without further training mea- sured by the MLM loss on a held out set on five additional very low-resource languages (Scottish Gaelic, Luxembourgish, Cebuano, Samoan, and Hmong). For these languages, we use mC4 (Raf- fel et al., 2020) and OSCARv23.01 (Abadji et al., 2022) as additional data sources for unlabeled text.
Vocabulary Extension or Replacement? We find that vocabulary extension generally performs worse on downstream tasks than keeping the origi- nal vocabulary. This finding is in line with results reported by Ebrahimi and Kann (2021) on a set of 30 typologically diverse languages. Prior stud- ies proposing vocabulary extension (Chau et al., 2020; Wang al., 2020: Chau and Smith, 2021) used mBERT and were motivated by the possibility of out-of-vocabulary (OOV) tokens. For XLM-R using SentencePicce with 100% character set cov- erage or byte-level tokenizers, OOV tokens can always be represented at the character or byte level. Therefore, the benefits of vocabulary extension might be less pronounced in these cases because the OOV problem is less relevant to begin with.
4 Results
We present downstream task results for NLI and QA in Table 2 and for NER in Table 3. In the fol- lowing, discuss various aspects of these results. In Figure 2, we show loss curves on a held out set when adapting XLMM-R with custom tokenizers. Table 4, we report the masked language mod- eling (MLM) loss of various methods right after initialization (no further training performed).
On average, when combined with Focus ini- tialization, vocabulary replacement outperforms both vocabulary extension and keeping the origi- nal vocabulary. Nevertheless, keeping the original vocabulary intact proves to be strong baseline and for the high-resource language German even outperforms vocabulary replacement with Focus. However, vocabulary replacement paired with Fo- cus performs better on medium- and low-resouree languages, results in smaller models, and is thus faster to train.
Effectiveness of FOcUS. Table 4 shows the effec- tiveness of Focus initialization for vocabulary re- placement. Directly after initialization without fur- ther training, FOCUS significantly outperforms all other initialization methods. In Figure 2, we show loss curves over the course of language-adaptive pretraining (LAPT). For random initialization, we only plot the second stage after the embeddings have already been trained for additional 20% of total training steps. FOCUS yields lower loss than random initialization even at the end of training, despite random initialization having been trained for more steps in total. WECHSEL starts off worse than FOCUS but catches up over the course of train-
Now-Resouree Languages. Focusing on lesser- resourced languages, FocUs outperforms random initialization and LAPT with the original vocabu-

---

XNLI (translate-traing
GermanQuAD/ Q TyDiQA
Method
German
Arahic
Kiswahili
Avg-
German
Arahic
Kiswahili
Avg.
XL.M-R (original vocab)
-off-the-shelf
78.8 = 0.3
74.8 0.4
69.1 0.4
74.2
71.3 0.4
78.4 0.9
73.9 = 1.4
74.5
LLAPT
78.9 = 0.4
75.1 = 0.6
72.4 0.4
75.5
70.5 0.8
78.9 0.5
75.8 1.0
75.0
XLM-R (replaced vocab)
Random LAPT?
77.6 = 0.4
74.6
0.3
74.5
69.1 0.7
79.3 0.6
74.2 1.0
74,2
WECHSELEN + LAPT
77.7 0.5
75.4
0.2
75.0
71.0 = 0.4
=
75.2
75.2
WECHSEL + LAPT
78.2 = 0.2
76.0 0.2
72.3
75.5
70.5 0.5
79.4 0.9
75.5 = 1.5
75.1
FOCUS LAPT
78.3 +: 0.6
76.5 0.4
72.9 0.5
75.9
71.3 0.2
79.1 0.4
76.5 + 1.5
75.6
XLM-R (extended vocab)
Random LAPT
77.7 0.6
75.2 0.6
71.8
74.9
69.8 0.6
77.7
76.3
74.6
FOCUS + LAPT
78.0 = 0.4
75.5 = 0.4 72.1 + 0.2
75.2
69.5
03
77.8 1.0
77.0 + 0.6
74.7
Table 2: Results on Natural Language Inference and Question Answering tasks, Details on the datasets used for evaluation are given in Section 3.3. We bold the best result each section and underline the overall best result. LAPT is short for anguaie--daative pretraining; perform LAPT for 50 million samples unlabeled target texts. t; For random initialization, we train just the embeddings for an additional 20% of training steps before full LAPT to create a stronger baseline.
WikiANN
4ermEval14 MasakhaNERy2
Method
German
Arabic
Kiswahili
Avg.
German
Kiswahili
Hausa
isiXhosa
Avg.
-off-th-sshelf
86.3 0.2
85.7 1. 0.3
86.6 0.5
86.2
85.6 0.3
92.0 1.1.1
84.2 0.5
85.5 0.3
87.1
-LAPT
86.7
87.1
86.9
86.9
86.8 0.2
92.5 0.2
85.6 0.4
88.3 0.2
88.3
XLM-R (replaced vocab)
Random LAPT!
86.0 +: 0.1
87.5 +: 0.1
85.8
86.4
85.9 0.3
92.3 0.2
85.0 +
87.4 0.2
87.8
WECCSEL LAPT
86.4
87.8
86.6
87.0
86.4
92.3
WECUSEL + LAPT
86.5 1. 0.2
87.9 0.0.3
87.4 1 1.6
87.3
86.7 1.0.1
92.2 1.1.
FOCUS + LAPT
86.6 1: 0.2
87.9 0.1.1
86.9 1. 0.4
87.1
86.6 0.0
92.6 1.0.1
86.0 0.4
88.5 1 0.4
88.4
XLM-R (extended vocah)
Random LAPT
85.6 0.2
85.2 +: 0.3
86.2 +: 0.7
85.6
85.4 0.3
92.0 0:2.2
84.1
0.2
87.2
0.4
87.5
FOcUS + I.PPT
86.0 + 0.1
85.3 + 0.3
86.2 + 0.3
85.8
85.6 0.2
92.1
+.0.2
84.9
0.4
87.7
0.3.
87.9
Table 3: Results on Named Entity Recognition (NER) tasks. Details the datasets used for evaluation are given in Section 3.3. We bold the best result in each section and underline the overall best result. t; For random initialization, train just the embeddings for additional 20% of training steps before full LAPT to create stronger baseline. Languages not covered by the pretrained fastText word embeddings used by WECHSEL.
German
Arubie
Kiewahii
WECISSL
RaNDoM
WICHEEL
RANDOM
4.0-
BANDOM
Wrenes
Focus
4.0-
Focus
Focus
2.5
3.0
3.0-
2.5x
2.0
2,5
2.0
1.5
10
20
30
10
20
.raining peoggrede
(En million somples weeet)
kmining progres (in million sumpple w000)
Immmiin progres ii million eamppee m0c0)
Figure 2: Masked Language Modeling (MLM) loss of different methods for vocabulary replacement over the course of further pretraining (LAPT), evaluated on a held out set. The first data point is logged at 1 million samples. For random initialization, we plot only the second stage, i.c., after already training just the embeddings for 10 million samples. This allows compare Focus and WECHSEL embedding initialization directly with gradient descent training of the embeddings.

---

Part of XI-M-R's pretraining
Not Part of XLM-R's pretraining
Data source
CC100
OSCARv23.01
mC4
Initialization 1
German
Arabic
Kiswahili
isiXhosa
Hausa
Scott. Gaelic
Luxembourgish
Cebuano
Samoan
Hmong
Random
24.0
24.1
24.2
25.5
23.7
22.8
24.4
22.5
21.9
22.8
Focus
4,0
5.2
4,8
7.6
6.0
6.1
8.2
6.33
4.9
5.7
Word-fastTest
4.3
5.5
6.0
7.5
8.7
6.7
Symbolic Overlap
10.6
10.6
10.7
10.4
10.4
10.4
10.2
9.7
8.4
8.1
WECHSEL
8.3
9.8
11.2
11.
10.9
10.1
Table 4: MLM loss on a held-out set immediately after initialization (no training performed) with full vocabulary replacement. We use the same vocabulary for methods in a single language. Symbolic Overlap restricts overlapping tokens numbers, punctuation, or whitespace. Word-fastText uses WECHSBL'S method of turning pretrained word embeddings into token embeddings instead of our proposed directly trained token-level embeddings. -1: Languages are not covered by the pretrained fastText word embeddings used by WECHSEL.
lary NER for Hausa and isiXhosa. Furthermore, we report the Masked Language Modeling loss di- rectly after initialization on a number of very low- resource languages in Table We see that across all languages, including the very low-resource ones, FOCUS achieves the best results. FOCUS also pro- vides a good initialization even when the target language was not part of the source model's pre- training.
and others). For Arabic and German, Focus with only symbolic overlapping tokens performs slightly worse than WECHSEL. In practice however, we will generally have numerous further overlapping tokens such as named entities and code-switched tokens. This is demonstrated by our results for Luxembourgish, Cebuano, Samoan, and Hmong all languages that XLM-R and XLM-R's tokenizer were not pretrained Here, using the full overlup outperforms using only symbols, suggesting more beneficial overlapping tokens beyond the ones in- cluded in our symbolic overlap. Overall, these results show that FOCUS can provide a good initial- ization even when the target language was not part of the source model's pretraining.
In low-resource settings, key advantage of FOCUS is that we only need unlabeled target lan- guage data to train our auxiliary embeddings - a resource already needed for LAPT in any case. Un- like WECHSEL, no bilingual dictionary is required. the quality and coverage of which might also be insufficient in low-resource settings. Some low- resource languages, such Hausa and isiXhosa, are also not covered by WECHSEL'S source of pre- trained word embeddings." 10
Auxiliary Embeddings. WECHSEL proposes method to pretrained word-level fastText em- beddings to obtain token-level embeddings. We propose directly train token-level fastText em- beddings. In Table 4, we additionally show Fo- CUS's 's initialization performance when using the WECHSEL-style method to obtain token-level fast- Text embeddings (Word-fastTexxs We see that us- ing our directly trained token-level fastText embed- dings results in better initialization for low- and high-resource languages.
Effect of Vocabulary Overlap. Naturally, the quality and quantity of overlapping tokens influ- ences the success of Focus. To investigate this, we conducted empirical analyses in two settings: using the full overlap and using only overlapping tokens that are symbols, numbers, or punctuation (Symbolic Overlap). Full overlap can take advan- tage of the source model's multilingual pretraining if the target language or a closely related language were part of the pretraining corpus. In any case, however, symbolic tokens such as whitespace, num- bers, and punctuation should generally be available, allowing transfer a model any language. In Table 4, we show that even when using only symbolic overlapping tokens, FOcUS outperforms WecHSel on medium to low-resource languages (e.g., Scottish Gaelic, Luxembourgish, Kiswahili,
W WECHSELEN- On average, WECHSEL actually fares slightly better than WECHSELEN, although WECHSELen also improves over random initializa- tion. For WECHSELEN. we followed Wang et al. (2019) in selecting English tokens in XLM-R's original vocabulary by taking the overlap with language-specific English tokenizer's vocabu- lary. Due the substantial presence of English in XLM-R's original vocabulary, this may have been too restrictive, excluding too many potentially useful tokens.

---

5
Related Work
embeddings in the multilingual source model. For these tokens, we can copy their pretrained embed- dings as a gold standard. Obtaining a different initialization is likely to lead to a worse result. Fo- CUS is well-positioned to take advantage of these pretrained embeddings of target language tokens. Additionally, WECHSEL requires bilingual die- tionary as additional resource seed the embed- ding space alignment. For low-resource languages, such a bilingual dictionary might of lower qual- ity or not available. FocUs does not require bilin- gual dictionaries as an additional resource.
We now discuss further related work apart from the studies introduced in Section 1.
Language Adaptive Pretraining (LAPT). Alabi al. (2022) adapted XLM-R to up to 20 African languages at the same time instead of specializing on a single language. Ebrahimi and Kann (2021) and Wang et al. (2022) used resources with much higher language coverage than web web-scraped mono- lingual texts (the Bible and lexicons, respectively) to adapt pretrained multilingual models to unseen languages. Muller al. (2021) transliterated un- seen languages into Latin script to improve the results when using an existing pretrained vocabu- lary.
Other Embedding Initialization Methods. In concurrent work, Ostendorff and Rehm (2023) pro- pose a similar method to Focus that initializes an embedding matrix for new vocabulary based combinations of overlapping tokens with pre- trained embedding matrix, but use the embedding layer of smaller pretrained Transformer model in- stead of static fastText embeddings as an auxiliary embedding space. However, their study only pro- vides results on the high-resource language German as a target language and they do not consider BERT- style source models. If no smaller pretrained Trans- former model with the desired tokenizer is avail- able, training one from seratch comes with much higher computational cost than training the fastText embeddings for Focus. Zeng et al. (2022b) create a new vocabulary and embedding matrix for the target language by translating tokens in the source vocabulary with bilingual dictionaries.
Adapters. In contrast to approaches changing all pretrained model weights, Pfeiffer et al. (2020) in- troduce additional adapter modules and only these new weights are changed. This is more parameter- efficient than full model adaptation, but gradients still need to be backpropagated throughout the model until the first adapter (Rücklé et al., 2021). Also, adapters introduce additional computational cost at inference time.
Bilingual Embedding Alignment. Vernikos and Popescu-Belin Beeiss (2021) propose SMALA to calculate a mapping between embedding spaces for two lan- guages to find semantically similar tokens across languages. They also experiment with initializing the remaining tokens based on this cross-lingual mapping. WECHSEL (Minixhofer al., 2022) aligns word embeddings from two different lan- guages. Such aligmments operate under the as- sumption of near-isomorphism berween embedding spaces of different languages (Vulić et al., 2020), i.e., that they share a similar geometric structure. Recent studies have challenged this assumption, especially for language pairs with typological (Sø- gaard et al., 2018; Patra al., 2019; Ormazabal et al., 2019) and resource (Vulić et al., 2020; Fu et al., 2020) differences. This is especially detri- mental the case of language model transfer, as we usually transfer from a high-resource language such as English to less-resourced languages with potentially different typology. FOCUS does not require the alignment of embedding spaces.
Conclusion
We propose FOcus, a novel embedding initializa- tion method for the monolingual specialization of language models with a tonguage-specinieg tokenizer. FOCUS uses the vocabulary overlap between source and target languages to effectively transfer the pre- trained embeddings to the new target tokenizer's embedding matrix. In series of experiments across diverse set of languages and several differ- ent tasks, we show that Focus outperforms other available embedding initialization methods without requiring additional resources like bilingual dictio- naries. FOCUS can provide a good initialization even if only a minimal vocabulary overlap is avail- able and when the target language has not been part of the source model's pretraining. We release our code and model checkpoints GitHub. !1
For multilingual source models, WECHSEL also disregards a valuable resource at our disposal: tar- get language tokens that already have pretrained
"https:/ / thub. wwt.//oosstaaiinjjdoleee/ffooooo

---

Limitations
Acknowledgements
The authors acknowledge the financial support by the German Federal Ministry for Education and Research (BMBF) through the project 4KI- Servicezentrum Berlin Brandenburg (011S22092). We also thank the reviewers for their helpful com- ments.
We evaluate FOCUS only for BERT-like Trans- former models. In principle, the method is applica- ble to any model that uses a vocabulary, tokenizer, and embedding matrix. In future work, we hope to investigate the use of FOCUS on GPT decoder mod- els, as explored by Ostendorff and Rehm (2023).
We conduct downstream task evaluations for NLI, QA, and NER on German, Arabic, and Swahili. For the low-resource languages Xsosa and Hausa, we conduct downstream task experi- ments for NER. This provides good mix of dif- ferent levels of available resources, scripts, and ty- pology. However, further evaluations on languages covering more scripts and languages that were not part of the source models' pretraining are needed to substantiate the effectiveness of FOcUS in these set- tings. All our chosen languages have monolingual texts available for further pretraining. As Wang et al. (2022) note, this is not the case for many other low-resource languages. Since further pre- training target language data is key component of our model adaptation strategy, the applicability of Focus is also limited in this regard, although such data can in some cases also be synthesized.
References
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards cleaner document- oriented multilingual crawled corpus. In Proceedings of the Thirteenth Language Resources and Evalua- tion Conference, pages 4344-4355, Marseille, France. European Language Resources Association.
David Adelani, Graham Neubig, Sebastian Ruder. Shruti Rijhwani, Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sihanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor. Fatoumata Ka- bore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh- Buabeng, Victoire Memdjokam Koagne, Allah- sera Auguste Tapo, Tehogo Macuewa, Vukosi Mari- vate, Mboning Tehiaze Elvis. Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende: Neo Lerato Mokono, Ig- natius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme. Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4588-4508, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Ethics Statement
In this work, we conduct the main part of our down- stream task experiments on German, Arabic, and Swahili. These choices stem from our desire to pro- vide practically useful ideas that reflect the current availability of models and to conduct experiments on downstream tasks such as question answering. NLI, and named entity recognition, for which we need relevant ground truth data.
Jesujoba O. Alabi, David Ifcoluwa Adelani, Marius Moshach, and Dietrich Klakow. 2022. Adapting pre- trained language models to African languages via multilingual adaptive fine-tuning, In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336-4349, Gycongju, Republic of Korea. International Committee on Computational Linguistics.
Finally, researchers and practitioners need to be cognizant of the fact that adopting existing mono- lingual or even multilingual models as a starting point instead of training new models from scratch can lead to remnant biases towards the original pretraining data. Hence, there is risk that a model adopts certain forms of behavior that reflect other languages and cultures than that of the lan- guage community one targeting. Also, web-scale datasets used for pretraining such as CC100 might contain personal and sensitive information. Such behavior needs to be assessed very carefully before any deployment of the models.
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics.
Darina Benikova, Chris Biemann, and Marc Reznicek. 2014. NoSta-D named entity annotation for German:

---

Guidelines and dataset. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (IREC'144), pages 2524-2531, Reyk- javik, Iceland. European Language Resources Asso- ciation (ELRA).
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model 1600 languages. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference Nutu- ral Language Processing (Volume I: Long Papers), pages 4555-45615 Online. Association for Computa- tional Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-14 146.
Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324-1334, Online. Association for Computational Linguistics.
William Falcon, The Py Torch Lightning team, and Open Source Contributors. 2019. Py PyTorch Lightning.
Zuohui Fu, Yikun Xian, Shijie Geng, Yingqiang Ge, Yuting Wang, Xin Dong, Guang Wang. and Gerard de Melo. 2020. ABSent: Cross-lingual sentence rep- resentation mapping with bidirectional GANs. In Proceedings of the 34th AAAI Conference on Artifi- cial Intelligence (AAAI 2020). AAAI Press.
Ethan C. Chau and Noah A. Smith. 2021. Specializing multilingual language models: An empirical study. In Proceedings of the Ist Workshop on Multilingual Representation Learning, pages 51-61, Punta Cana, Dominican Republic. Association for Computational Linguisties.
HuggingFace. 2021. Huggingface Tokenizers.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: benchmark for setormatioon-seeking question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguisties, 8:454-470
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Taku Kudo and John Richardson. 2018 Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyamov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Teven Le Seao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsa- har, Niklas Muennighoff, Jason Phang, Ofir Press. Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Lau- nay, and Iz Beltagy. 2022. What language model to train if you have million GPU hours? In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 765-782, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Wietse de Vries and Malvina Nissim. 2021. As good new. how successfully recycle English GPT-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 836-846, Online. Association for Computational Linguistics.
Quentin Lhoest, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Mario Šaško, Yacine Jernite, Abhishek Thakur, Lewis Tunstall, Suraj Patil, Mariama Drame, Julien Chaumond, Julien Plu, Joe Davison, Simon Brandeis, Victor Sanh, Teven Le Scao, Kevin Canwen Xu, Nicolas Patry, Steven Liu, Angelina McMillan-Major, Philipp Schmid, Syl- vain Gugger, Nathan Raw, Sylvain Lesage, Anton Lozhkov, Matthew Carrigan, Théo Matussière, Lean- dro von Werra, Lysandre Debut, Stas Bekman, and
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages

---

Clément Delangue. 2021. Datasets: A Community Library for Natural Language Processing. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, pages 175-184. Association for Computa- tional Linguistics.
Xiaoman Pan, Boliang Zhang, Jonathan May. Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Pross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- lion for Computational Linguistics (Volume I: Long Pupers), pages 1946-1958, Vancouver, Canada. As- sociation for Computational Linguisties.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buce E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
André F. T. Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg. Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1 184-193, Florence. Italy. Association for Computational Linguistics.
Benjamin Minixhofer, Fabian Paischer, and Navid Rek- absaz. 2022. WFCCHSBL; Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3992-4006, Seattle, United States. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An Adapter -Base Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7654-7673, Online. Association for Computa- tional Linguistics.
Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving non- English question answering and passage retrieval. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering. pages 42-50, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Lim- its of Transfer Learning with Unified Text-to-Text Transformer.
Afshin Rahimi, Yuan Li, and Trevor Cohn, 2019. Mas- sively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 151-164, Florence. Italy. Association for Computational Linguisties.
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being un- seen from mBERT's is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 448-462, Online. Association for Computa- tional Linguisties.
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers). pages 3118-3135, Online. Association for Computational Linguistics.
Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. 2019. Analyzing the limi- tations of cross-lingual word embedding mappings. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 4990- 4995, Florence, Italy, Association for Computational Linguistics.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer. Nils Reimers, and Iryna Gurevych. 2021. Adapterdrop: On the efficiency of adapters in transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing.
Malte Ostendorff and Georg Rehm. 2023. Efficient language model training through cross-lingual and progressive transfer learning.

---

Peter Schönemann. 1966.
generalized solution of the orthogonal procrustes problem. Psychometrika, (100)
2020. Transformers: State-of-the-Ar Natural Lan- guage Processing. pages 38-45. Association for Computational Linguistics.
Anders Søgaard. Sebastian Ruder, and Ivan Vulić, 2018. On the limitations of unsupervised bilingual dictio- nary induction. In Proceedings the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 778-788, Melbourne, Australia. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Com- putational Linguistics.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuayi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022a. Glm-130b: An open bilingual pre-trained model.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Qingcheng Zeng. Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, and Jie Yang. 2022b. Greenplm: Cross-lingual pre-trained language models conversion with (almost) no cost.
Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models.
Barret Zoph. Deniz Yuret, Jonathan May, and Kevin Kuight. 2016. Transfer learning for low resource neu- machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1568-1575 Austin, Texas. Association for Computational Linguisties.
Giorgos Vernikos and Andrei Popescu-Beils Belis. 2021. Sub- word mapping and anchoring across languages. In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages (633-2647, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Ivan Vulié, Sebastian Ruder, and Anders Søgaard. 2020. Are all good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3178-3192 Online. Association for Computa- tional Linguistics.
Hai Wang. Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual model with vocabulary expansion. In Proceedings of the 23rd Conference Computational Natural Lan- guage Learning (CoNLLL), pages 316-327, Hong Kong. China. Association for Computational Lin- guistics.
Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2022. Expanding pretrained models to thousands more languages via lexicon-base based adaptation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 863-877, Dublin, Ireland. Association for Computational I Linguistics.
Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low- resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649-2656, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Can- wen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.

---

A Hyperparameters and Experiment Details
Hyper-parameter
Value
peak learning rate learning rate schedule learning rate warmup batch size sequence length
5 x 10-5
We conducted all experiments heterogeneous compute cluster with Nvidia V100 32GB, A 40GB, A100 80GB, and A6000 48GB GPUs. De- pending on availability we used one, two, or four GPUs for our experiments and adjusted the batch size per device so that we retain the same ef- fective batch size. Depending on total model size, we also used gradient accumulation with smaller batch sizes to fit the model into GPU memory. We used PyTorch (Paszke al., 2019)
constant
5 million samples
128
256
gradient clipping
1.0
Adam e
x 10-8
Adam 31
0.9
Adam $2
0.999
training samples
50 million
resulting train steps
390625
and pytorch-lightning (Falcon al., 2019) as
well as the HuggingFace transformers (Wolf
Table 5: Hyper-parameters for further pretraining on target language data.
et al., 2020), tokenizers (HuggingFace, 2021)
and datasets (Lhoest et al., 2021) libraries. We
used the fp16 mixed precision training imple- mented by pytorch-lightning
using the original XLM-R tokenizer. Motivated by this, we turned to OSCARv23.01 (for Cebunao and Luxembourgish, since it does not contain Cor- sican). We filter each dataset based the qual- ity warnings header, footer, and noisy provided by OSCARv23.01. Training a tokenizer on this data yielded the expected results. For Hmong and Samoun, we did not observe such degraded tok- enization training on mC4. Nevertheless, these cor- pora (and all web-crawled corpora of low-resource languages) can also be expected to be noisy.
Further pretraining. We used the same hyper- parameters for all target languages, as detailed in Table 5. We trained for a total of 50 million samples with batches of 128 sequences of 256 to- kens. This results in total of 390,625 optimizer steps (weight updates). We used A opti- mization (Loshchilov and Hutter, 2019) as imple- mented in torch. optim!2 and a linear learning rate warmup for 5 million samples (39,062 opti- mizer steps) followed by constant learning rate at 5 x 10-5, We used constant schedule to allow for more flexible experimentation regarding the to- tal number of training steps and to ensure that the impression of converged loss curves is not false positive induced by a decaying learning rate. We also conducted preliminary experiments using a cosine learning rate schedule and did not observe a significant difference. We used the CC100 dataset in line with its intended use for the pretraining of language models. For German, training does not even complete a full epoch.
Downstream tasks. We detail our hyperparame- ters for all downstream tasks in Table 6. We largely followed default values of finetuning scripts pro- vided by Huggingface!4 but adjusted the training epochs depending dataset size, added linear learning rate warmup for 10% of total training steps, and adjusted the batch size based on used GPU memory per task. Additionally, we used a 2 x 10-5 peak learning rate for all non-QA tasks. We repeated each experiment five times with the random seeds (1, 2,,,,,55 and report the mean and standard deviation across runs. For XNLI, report accuracy, for TyDiQA, GermanQuAD, WikiANN, MasakhaNERv2, and GermEval14, we report the Fcoor.
Datasets for low-resource languages. Compar- ing OSCARv23.01 and mC4 data source for low-resource languages, we observe that for Cor- ! Cebuano and Luxembourgish, the qual- ity in mC4 is quite poor. Training a tokenizer on these datasets results in a tokenizer that, on aver- age, encodes fewer characters per token than when
Tokenizer training. Our language-specific tok- enizers were trained in the same way XLM-R for comparability, specifically SentencePiece tok- enization (Kudo and Richardson, 2018) with the Unigram algorithm (Kudo, 2018). We used Hug-
""Since we do not use weight decay. this is equivalent to using Adam (Kingma and Ba, 2015).
15 "Results not included in the paper as the language is not provided by OSCARv23.01.
"i thub. www.wwgg httpr//aeeetaasffeee//ereeeeee/e hamples/pytorca

---

gingFace tokenizers and a vocabulary size of 50k tokens for all languages. The resulting vocabular- ies contain a large amount (roughly 10k tokens) of emojis and Chinese, Japanese, and Korean single characters, which is an artifact of SentencePiece s parameter (which defaults to 100%). Characters are included in the vocab- ulary even if they appear only once in the large amount of noisy web-scraped training documents. This effectively means that our language-specitin vocabularies are roughly 10k tokens smaller in prac- tice, as such single characters rarely occur in the training data. In practice, one may wish to tune the aeeractee_coverrreep carefully based on the requirements of the target language if smaller model is desired.
words being interspersed within native sentences. which also contributes to shared semanties. A con- siderable share of tokens is also made up of names, named entities, symbols, numbers, and punctuation. While these are not exclusive to any particular lan- guage, they are likely to possess the same semantics across languages, making them good overlapping tokens. We report the number of overlapping to- kens for languages used during training in Table 8. Additionally, we manually classified a random sample of 500 overlapping tokens for German and report the results in Table 7. The overlap is cal- culated between XLM-R's s original tokenizer and our newly trained, language-specing German one. For this analysis, excluded the noisy single- character tokens mentioned in Appendix A. We conclude that a considerable share of the overlap- ping tokens for German does indeed possess sim- ilar semantics in the pretrained and new vocabu- laries. For less-resourced languages than German that were still part of the multilingual models' pre- training, we can expect fewer overlapping tokens that are directly part of the target language. High- resource languages have larger share of language- specific tokens in the vocabulary of XLM-R. How- ever, for languages with an uncommon or unique script, tokens are more likely to be exclusive to the target language. During the pretraining of XLM-R, low-resource languages are also oversampled (Con- neau et al., 2020). Therefore, tokens that are shared between low and high-resource languages are more likely also have the low-resource language mantics encoded in their embeddings than would otherwise be the case.
B Further details for FOcUS
FastText training. To obtain static token embed- dings for Focus, we train fastText embeddings on tokenized target language training data. We mostly used default mensionality to 300, as is commonly done in the literature (Mikolov al., 2013; Bojanowski et al., 2017). We ran the training for three epochs. On German, due to its corpus size, we ran only a single epoch.
but increased the di-
Additionally, we set a minCount of 10 for to- kens during fastTText training to filter out very rare tokens. These rare tokens are initialized from a nor- mal distribution with mean and standard deviation per dimension calculated from the source embed- ding matrix, done by WECHSEL for tokens that have no subwords in the pretrained word embed- ding. Setting minCount also helps with filtering the noisy single characters that are part of our tokeniz- ers due to SentencePiece's character_coveret parameter.
Overlaps between different tokenizers. In gen- eral, we only consider tokens as overlapping if they are an exact match (including case and the "be- ginning of word" (Bow) signifier. However, for tokens that only consist of numbers, punctuation, or whitespace, we implement fuzzy matching where we disregard the case and the Bow signifier.
Vocabulary Overlap. FocuS relies on overlap- ping tokens between the new and pretrained vocab- ularies. Ideally, an overlapping token would have the same semantics in the target language vocabu- lary and in the pretrained vocabulary. If the target language was already part of the pretraining, this is most obviously true for (sub-) that only occur in the target language. Differences in script pe- culiarities of the target language (such German umlauts and other language-speciff accented char- acters) help facilitate such occurrences. In many languages, especially online, there is widespread code-switching with English, leading to English
A peculiarity of calculating token overlaps be- tween different kinds of tokenizers is the repre- sentation of tokens that are Bow tokens and non- ASCIT characters. For example, the HuggingFace implementation of Byte-Level BPE uses G as a pre- fix for Bow tokens, whereas XLM-R's tokenizer uses To complicate things, BERT's tokenizer WordPiece (Devlin et al., 2019) prefixes tokens that are not Bow with ##. Also, Byte-level BPE represents NSn-ASCII characters in tokens differ-

---

Hlyper-parameter
XNLI
QA
WikiANN
GermEval14
Masa
epochs
2
5*
5#
25
25
peak learning rate
2 x 10
5 10-6
5 X 10-5
5 X 10
2 x 10
Ir schedule
linear
linear
linear
linear
linear
Ir warmup ratio
10%
10%
10%
10%
10%
batch size
128
64
128
128
128
sequence length
256
384
256
256
256
gradient clipping
1.0
1.0
1.0
1.0
1.0
Adam 6
x 10
x 10
x 10
1 x 10
1 x 10
Adam 51
0.9
0.9
0.9
0.9
0.9
Adam f32
0.999
0.999
0.999
0.999
0.999
for our downstream tasks. *: After observing high variance due to smaller training set sizes, adjusted the number of epochs for Swahili to 25 for QA (on TyDiQA GoldP) and NER (on WikiANN). We evaluate every 5% of total training steps and report the hest checkpoint's results on the test set.
Table
Category
Share
Examples
Symbols Numbers
9%
3.5, 1919, 3500, ...,;.),, :-), !!!, 1%, [6] BlackBerry, Oscar, Messi, JavaScript Bewerber, Wahl, günstig. fallen Smoothie, FAQ, Backup, Settings ik, Kri, kub, rez, zy, BF, oka
Names & Entities
10%
German (sub-)words
46%
English & Code-switched
18%
Not assignable
17%
Table 7: Investigation of overlapping tokens on German. The evaluation was conducted manually by one of the authors on a random sample of 500 overlapping tokens. the examples, leading spaces are omitted and we further exclude the "noisy" single-character tokens mentioned in Appendix A.
ently than XLM-R's tokenizer. In our experiments in this paper. we only use the XLM-R tokenizer. which also matches the source model's s tokenizer, and therefore avoid these problems. However, a cor- rect canonicalization of tokens to common form is crucial to enable FOCUS when the tokenizers of source and target model do not match. We imple- ment such canonicalization method for common tokenizers and release it as part of our ready-to-use implementation of Focus.!5
Tokens in Overlap
Language
minCount = 10
Full
German
14,485
18,986
Arabic
10,658
13,996
Swahili
10,443
12,353
Hausa
1 11,481
14,806
isiXhosa
6,222
8,333
Table 8: Number of tokens in the overlap between language-specific and XLM-R's original vocabulary. For learning fasttext embeddings with FocUs, we set minCount = 10, which filters out very rare and noisy tokens.


Reviews:
[{"review_id": "hfauFxdlak", "reviewer": "Reviewer_EJyH", "summary": "", "questions": "1. Could you give more details about how you apply fastText to the unlabeled target language data?\n2. What if a word in the target language cannot be found in fastText vocabulary, how do you handle this?\n3. Is there any reason you choose fastText rather than other static word embeddings such as GloVe and word2vec?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our paper.\n\nRegarding your questions:\n\n1. To obtain token-level embeddings for a target language, we first tokenize the entire unlabeled language corpus using the language-specific tokenizer. The result is written to disk, with whitespace as a delimiter between individual tokens (although this is an implementation detail of fastText). Then, we apply the standard fastText training procedure. Note that tokenizing first is crucial to obtain token embeddings rather than word embeddings. We provide some further implementation details in Appendix B.\n\n2. In our experiments, we train language-specific tokenizers on the same corpus that we use for fastText training, so we do not have this problem. However, some tokens might only occur very infrequently, so they will not obtain a reliable fastText embedding. In these cases, we will initialize them from a normal distribution with the mean and standard deviation of the source embeddings.\n\n3. FOCUS is not at all limited to using fastText token embeddings. However, fastText is more recent than word2vec or GloVE and has also been used by previous work such as Minixhofer et al. 2022. We will also correct the spelling of \"fasttext\" to \"fastText\".\n\nWe hope that our answers are helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and the idea is straightforward. The authors use the vocabulary overlap between source and target languages to effectively transfer the pre-trained embeddings to the new target tokenizer's embedding matrix, and without requiring either bilingual dictionaries or an alignment of embedding spaces for different languages. ", "reasons_to_reject": "From my point of view, I do not have reason to reject it.", "questions_to_authors": "1. Could you give more details about how you apply fastText to the unlabeled target language data?\n2. What if a word in the target language cannot be found in fastText vocabulary, how do you handle this?\n3. Is there any reason you choose fastText rather than other static word embeddings such as GloVe and word2vec?", "paper_topic_and_main_contributions": "This paper proposes FOCUS, which is an embedding initialization method for the monolingual specialization of language models with a language-specific tokenizer. The authors conduct experiments on NLI, QA and NER tasks and show that this method is able to effectively initialize the embedding matrix for a new tokenizer based on information from the source model's embedding matrix. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "fasttext --> fastText"}, "extra_text": {}}, {"review_id": "HVNl03vOB6", "reviewer": "Reviewer_edfE", "summary": "", "questions": "A. In L167, you mention that overlap stems from code-switching. When applying FOCUS to a language with the unseen script do you always apply code-switching or just rely on code-switched words naturally occurring in the data? \n\nB. What do you mean by “always transferring the Transformer layers of XLM-R”? That in all examined models, encoder layers are copied from the original model?\n\nC. In “Random Initialization,” how is embedding mapping conducted when the sizes of source and target vocabularies are different? Are source embeddings subsampled? Do you consider a case where the source vocabulary would be larger than the target vocabulary?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our paper.\n\n---\nRegarding your comment about the role of LAPT:\n\nIndeed, further language-adaptive pretraining (LAPT) is important and FOCUS will not work well without it. However, in Tables 2 and 3, we can draw a direct one-to-one comparison of FOCUS initialization vs. random initialization with LAPT (2-Stage-Lapt) for adapting a model with vocabulary replacement. In this comparison, we clearly see that FOCUS obtains better results, in most cases outside the margin of error. When comparing FOCUS (vocabulary replacement) to LAPT with the original vocabulary (line 2 in the Tables), it is important to keep in mind that the model with FOCUS has far fewer parameters due to the replaced vocabulary and is faster and more memory-efficient for training and inference. Still, FOCUS also obtains better results.\n\n---\nRegarding your questions:\n\nA: We do not add any code-switching but simply take advantage of code-switching if it naturally occurs in the target language data (so if it is very uncommon in a particular target language, we cannot leverage it). However, even without any code-switching and a new, unseen script, FOCUS can also take advantage of named entities, numbers, symbols, and punctuation.\n\nB: Yes, that understanding is correct. We will update the paper to make it more clear.\n\nC: In the experiments reported in the paper, the source vocabulary is always larger than the target vocabulary, in which case we copy the first n embeddings from the source until the target embeddings are filled. If we have a smaller source vocabulary, we can initialize the added embeddings from a normal distribution with mean and standard deviation of the source embeddings, similar to how we deal with target tokens that do not have an auxiliary embedding.\n\n---\nFurthermore, we will incorporate your suggestions on improving the paper's presentation.\nWe hope that our answers are helpful in addressing all of your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method brings overall improvement in downstream tasks and reduces the size of the model leading to faster inference and training on in-language data via the LAPT method.\n\nThe introduced method is simple and clearly described. FOCUS requires just an unlabeled corpus of in-language data.\n\nThe paper presents an in-depth analysis of the strengths of the introduced method. It is compared with recent strong baselines.  \n", "reasons_to_reject": "The results in downstream tasks show that FOCUS with LAPT is further than the standard deviation above the original model with LAPT or WECHSEL with LAPT. This strongly suggests that most lifting is done by LAPT, and FOCUS does not offer a significant improvement. Figure 2 shows that good embedding initialization can lead to faster learning on in-language data, yet even for low-resource languages, the difference between FOCUS+LAPT and LAPT is small.\n\nThe analysis was performed only on one pre-trained language model (XLM-R). I’m afraid that the improvement stemming from better embedding initialization can be even smaller for bigger-scale models.\n", "questions_to_authors": "A. In L167, you mention that overlap stems from code-switching. When applying FOCUS to a language with the unseen script do you always apply code-switching or just rely on code-switched words naturally occurring in the data? \n\nB. What do you mean by “always transferring the Transformer layers of XLM-R”? That in all examined models, encoder layers are copied from the original model?\n\nC. In “Random Initialization,” how is embedding mapping conducted when the sizes of source and target vocabularies are different? Are source embeddings subsampled? Do you consider a case where the source vocabulary would be larger than the target vocabulary?\n", "paper_topic_and_main_contributions": "The paper introduces a method of adapting the embedding layer (and associated vocabulary) of multilingual language models for language-specific downstream tasks.\n\nThe method is based on averaging existing learned embeddings of tokens in monolingual vocabulary to obtain better representation for specific languages. The main strength of the method is that it increases the model’s performance on downstream tasks (QA, XNLI, NER)  while parameter count by 55%, thus increasing training/inference speed by 40%.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I find the paper well organized. I have just a few suggestions to move some pieces of information to help the reader better understand the contributions and methodology.\n\nL102: Ad: Reduction of the model size. I suggest underscoring this fact and reiterate it in the experimental setting.\n\nL290 Ad: WECHSEL_EN. I was wondering how the English tokens were identified. The answer came only at the end of the result section (in L492). I suggest bringing this explanation here (maybe in the form of a footnote).\n"}}, {"review_id": "MQZpRPxZTC", "reviewer": "Reviewer_s1L6", "summary": "", "questions": "- I didn’t understand Figure 1, what are the different points, what are the dashed lines, what do orange and yellow colors signify? In general, I think it’d be helpful to have a Formal Definitions sections where it’s clearly defined what’s the source model, what’s the target, etc.\n\n- I wonder if comparing loss across different tokenizers is a valid comparison (Table 4), can you please elaborate on how this is done? I think that this correlates with perplexity, which will be biased towards smaller vocabularies as they’re inherently less surprising.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review and will address your suggestions on our paper’s presentation.\n\n---\nRegarding your comments:\n\nWe state: “Without further training, FOCUS significantly outperforms all other initialization methods”. This refers to the results reported in Table 4. Here, we can clearly see that FOCUS performs much better than other initialization methods **without further training**.\n\nThe results on downstream tasks (Tables 2 & 3) and loss curves (Figure 2) are reported after further training. Indeed, we do see that the gap between FOCUS and other methods is smaller here (also acknowledged in the paper, e.g. l. 376ff). However, we would like to point out the following:\n* Naturally, the effect of initialization is less pronounced, the longer one trains a model. We have deliberately constructed a difficult evaluation with our long training regime of 12.8 billion tokens.\n* For the baseline of random initialization of a new embeddings matrix, we train just the embeddings for an additional 20% longer (which is not done for FOCUS). Nevertheless, we find that FOCUS obtains better results and in most cases also outside the margin of error. Note that random initialization is denoted as 2-Stage-Lapt in Tables 2 & 3 to signify the training regime with an additional 20% training. We will clarify that this uses random initialization.\n* When comparing FOCUS to LAPT of the original model (without vocabulary replacement), it is important to remember that the vocabulary replacement used with FOCUS yields a smaller model with fewer parameters and enables faster training and inference as additional benefits. Also, FOCUS still performs better.\n* In summary: Adapting a multilingual model using the original vocabulary is a strong baseline. However, especially for low-resource languages and crucially for languages that use a different script, customizing the vocabulary for the target language can be helpful. When using vocabulary replacement, we do see that on average, FOCUS improves over random initialization and also over WECHSEL (without requiring a bilingual dictionary).\n\n---\nRegarding your questions:\n\nFigure 1 is a high-level schematic of FOCUS: For each target token we want to initialize (blue), we select a set of similar other tokens (orange, connected by dashed lines). We colored the tokens that are “not similar” as yellow. Indeed, we sacrificed some clarity for brevity of presentation in Figure 1 and will work to improve this tradeoff.\n\nYes, comparing losses between different tokenizers would be problematic because in our case perplexity = exp(loss). In Table 4, for each language, all compared methods use the same tokenizer.\n\n---\nWe thank you again for your helpful comments and hope that we have answered your questions sufficiently well for you to consider raising your scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Interesting and simple approach for a timely problem - how to infer initializations for new tokens when adapting multilingual PLMs to a specific language and domain.", "reasons_to_reject": "* The performance seems rather similar to the baseline. While this isn't necessarily a problem, it is not discussed and explored in the paper. Conversely, it states that \"Without further training, FOCUS significantly outperforms all other initialization methods\", but I couldn't find what kind of stat. significance performance tests were performed, as the performance difference in most tables seem to be within the margin of error.", "questions_to_authors": "- I didn’t understand Figure 1, what are the different points, what are the dashed lines, what do orange and yellow colors signify? In general, I think it’d be helpful to have a Formal Definitions sections where it’s clearly defined what’s the source model, what’s the target, etc.\n\n- I wonder if comparing loss across different tokenizers is a valid comparison (Table 4), can you please elaborate on how this is done? I think that this correlates with perplexity, which will be biased towards smaller vocabularies as they’re inherently less surprising.", "paper_topic_and_main_contributions": "Edit following author reponse:\n------\nI thank the authors for their detailed comments, which answer my questions, and I raise my score accordingly. I hope that the clarification here are added to the paper.\n------\n\n\nThe paper proposes a method for monolingual token initialization from multilingual language models. This is achieved by training small-scale monolingual LMs, and then expressing tokens which are only in the monolingual model as a linear combination of known embeddings in the multilingual LM. This approach is tested both in intrinsic LM performance, as well as in downstream tasks.\n\nAlthough I like the approach, and I think it's very clearly described and it's well motivated, I was disappointed by the performance. Unless I missed something, I think that in most of the tested settings, the difference from simpler baselines is well within the error boundary.\n\nWhile I don't think that performance on its own is the only measure by which to review a paper, I do think that in this case it's (1) much of the paper premise seems predicated on this approach being useful for empirical reasons, and (2) I didn't find the paper acknowledging that this is the case, and trying to provide insights into why this may happen. \n\nIf that's indeed the case, I think that the paper would benefit from a deeper look into why the performance is similar, e.g., by examining error patterns via manual evaluation. I think that would have helped me learn more from this paper.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I think that the paper is superfluous with footnotes. Consider omitting in a few\n- The mathematical font for symbols used in Section 2 are nonstandard I think."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to adapt the embedding layer of multilingual language models for language-specific downstream tasks by averaging existing embeddings of tokens in the target language vocabulary to obtain better initialization. The approach seems to lead to pretty significant improvements while reducing parameter count.
Decision: Accept-Main