SlowGuess/NAACL_2022_OCR huggingface dataset:
Benjamin Minixhofer<sup>1</sup> and Fabian Paischer<sup>2,3</sup> and Navid Rekabsaz<sup>1,3</sup>
$^{1}$ Institute of Computational Perception, Johannes Kepler University Linz
$^{2}$ Institute for Machine Learning, Johannes Kepler University Linz
3ELLIS Unit Linz and LIT AI Lab
{benjamin.minixhofer,navid.rekabsaz}@jku.at
Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method - called WECHSEL - to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.
Large LMs based on the Transformer architecture (Vaswani et al., 2017) have become increasingly popular since GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) were introduced, prompting the creation of many large LMs pretrained on English text (Yang et al., 2019; Clark et al., 2020; Lewis et al., 2020; Ram et al., 2021). There is a tendency towards training larger and larger models (Brown et al., 2020; Fedus et al.,
2021) while the main focus is on the English language. Recent work has called attention to the costs associated with training increasingly large LMs, including environmental and financial cost (Strubell et al., 2019; Bender et al., 2021). If training large LMs for English is already costly, it is prohibitively expensive to train new, similarly powerful models to cover other languages.
One approach to address this issue is creating massively multilingual models (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021) trained on a concatenation of texts in many different languages. These models show strong natural language understanding capabilities in a wide variety of languages, but suffer from what Conneau et al. (2020) call the curse of multilinguality: beyond a certain number of languages, overall performance decreases on monolingual as well as cross-lingual tasks. Consistent with this finding, Nozza et al. (2020) observe that monolingual LMs often outperform massively multilingual models. This might be attributed to superior quality of monolingual tokenizers over their multilingual counterparts (Rust et al., 2021). It is thus desirable to train monolingual models in more languages. Training monolingual models in non-English languages is commonly done by training a new model with randomly initialized parameters (Antoun et al., 2020; Louis, 2020; Martin et al., 2020; Rekabsaz et al., 2019). However, to train a model with capabilities comparable to that of an English model in this way, presumably a similar amount of compute to what was used to train the English model would be required.
To address this issue, we introduce WECHSEL, $^{1}$  a novel method to transfer monolingual language models to a new language. WECHSEL uses multilingual static word embeddings between the source language and the target language to initialize model parameters. WECHSEL first copies all inner (non
Word Embeddings Can Help Initialize Subword Embeddings in a new Language.
Human Language Technologies, pages 3992 - 4006
embedding) parameters of the English model, and exchanges the tokenizer with a tokenizer for the target language. Next, in contrast to prior work doing random initialization (de Vries and Nissim, 2021), the token embeddings in the target language are initialized such that they are close to semantically similar English tokens by mapping multilingual static word embeddings to subword embeddings. The latter step is particularly important considering that token embeddings take up roughly  $31\%$  of the parameters of RoBERTa (Liu et al., 2019) and roughly  $33\%$  of the parameters of GPT2 (Radford et al., 2019). Intuitively, semantically transferring embeddings instead of randomly initializing one third of the model should result in improved performance. Our parameter transfer provides an effective initialization in the target language, requiring significantly fewer training steps to reach high performance than training from scratch. As multilingual static word embeddings are available for many languages (Bojanowski et al., 2017), WECHSEL is widely applicable.
We conduct our experiments on RoBERTa and GPT-2 as representative models of encoder and decoder language models, respectively. We transfer the English RoBERTa model to four languages (French, German, Chinese and Swahili), and the English GPT-2 model to the same four plus another four very low-resource languages (Sundanese, Scottish Gaelic, Uyghur and Malagasy). We evaluate the transferred RoBERTa models on Named Entity Recognition (NER), and Natural Language Inference (NLI) tasks in the respective languages. The transferred GPT-2 models are evaluated in terms of Language Modelling Perplexity (PPL) on a held-out set. We compare WECHSEL with randomly initialized models (denoted as FullRand), as well as the recently proposed TransInner method which only transfers the inner (non-embedding) parameters (de Vries and Nissim, 2021). All mentioned models are trained under the same conditions (around 4 days on a TPUv3-8). We also compare our model with models of comparable size trained from scratch under significantly larger training regimes, in particular CamemBERT (Martin et al., 2020) (French), GBERT<sub>Base</sub> (Chan et al., 2020) (German), and BERT<sub>Base</sub>-Chinese (Devlin et al., 2019).
Results show that models initialized with WECHSEL outperform randomly initialized models and models initialized with TransInner across
all languages and all tasks, for both RoBERTa and GPT-2. In addition, strong performance is reached at a fraction of the training steps of other methods. Our contribution is summarized as follows.
- We propose WECHSEL, a novel method for transferring monolingual language models to a new language by utilizing multilingual static word embeddings between the source and the target language.
- We show effective transfer of RoBERTa and GPT-2 using WECHSEL to four and eight languages, respectively, achieved after minimal training effort.
- We release more effective GPT-2 and RoBERTa models than previously published non-English models, achieved under our more efficient training setting. Our code and models are publicly available at github.com/ cpjku/wechsel.
In the following, we review related work in Section 2. We introduce the WECHSEL method in Section 3, followed by explaining the experiment setup in Section 4. We show and discuss results in Section 5.
Large Language Models. Training Language Models is usually done in a self-supervised manner i.e. deriving labels from the training text instead of needing explicit annotations. One optimization objective is Masked Language Modelling (Devlin et al., 2019, MLM), where randomly selected tokens in the input are replaced by a special [MASK] token, and the task is to predict the original tokens. Another common objective is Causal Language Modelling (CLM), where the task is to predict the next token. These two objectives highlight a fundamental distinction between language models: models can be trained as encoders (e.g. with MLM) or as decoders (e.g. with CLM).
Instead of words, the vocabulary of recently proposed language models commonly consists of subwords (Clark et al., 2020; Liu et al., 2019; Devlin et al., 2019).
Multilingual representations. There has been a significant amount of work in creating multilingual static word embeddings. A common method is learning embeddings from scratch using data
in multiple languages (Luong et al., 2015; Duong et al., 2016). Alternatively, multilinguality can be achieved by aligning existing monolingual word embeddings using a bilingual dictionary, so that the resulting embeddings share the same semantic space (Xing et al., 2015; Joulin et al., 2018). Recent studies improve on this by reducing (or completely removing) the need for bilingual data (Artetxe et al., 2017, 2018; Lample et al., 2018).
Beside static word embeddings, multilinguality is also well studied in the area of contextualized representations. One approach to learn multilingual contextualized representations is through training a model on a concatenation of corpora in different languages. Some models created based on this approach are mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) and mT5 (Xue et al., 2021), trained on text in 104, 100, and 101 languages, respectively. As shown by Pires et al. (2019), a multilingual model such as mBERT can enable cross-lingual transfer by using task-specific annotations in one language to fine-tune the model for evaluation in another language. Despite the benefits, recent studies outline a number of limitations of massively multilingual LMs. Wu and Dredze (2020) empirically show that in mBERT "the  $30\%$  languages with least pretraining resources perform worse than using no pretrained language model at all". Conneau et al. (2020) report that beyond a certain number of languages in the training data, the overall performance decreases on monolingual as well as cross-lingual tasks. These studies motivate our work on introducing an efficient approach for creating effective monolingual LMs for more languages.
Cross-lingual transfer of monolingual LMs. Studies in this area can be divided into two categories:
- Bilingualization of a monolingual LM is concerned with extending a model to a new language while preserving its capabilities in the original language. Artetxe et al. (2020) approach this problem by replacing the tokenizer and relearning the subword embeddings, while freezing other (non-embedding) parameters. Such a model becomes bilingual, since the initial tokenizer and embeddings can be used for tasks in the source language, while the new tokenizer and embeddings can be used for tasks in the target language. Thus, a model can be finetuned on annotated task data in
the source language, and then zero-shot transferred to the target language. Tran (2020) follow a similar approach, while instead of randomly initializing embeddings, they utilize static word embeddings to initialize embeddings in the target language close to semantically similar English tokens. They then continue training the model on an English text corpus as well as on the target language in order to preserve model capabilities in English.
- Creating a new monolingual LM in the target language is, in contrast, concerned with transferring a model from a source to a target language without the necessity to preserve its capabilities in the source language. Zoph et al. (2016) and Nguyen and Chiang (2017) show that cross-lingually transferring a machine translation model can improve performance, especially for low-resource languages. Zoph et al. (2016) use embeddings of random tokens in the original vocabulary to initialize token embeddings in the new vocabulary, while Nguyen and Chiang (2017) utilize vocabulary overlap between the source and target language. More recently, de Vries and Nissim (2021) follow a similar approach to the one of Artetxe et al. (2020) for transferring a GPT-2 model to a new language. de Vries and Nissim (2021) add an additional step, where they train the entire model for some amount of steps to allow adapting to the target language beyond the lexical level. We refer to the method of de Vries and Nissim (2021) as TransInner and consider it as a baseline in our experiments.
Our WECHSEL method belongs to the second category. WECHSEL can be seen as an extension to the method proposed by Tran (2020) with the goal of creating a new monolingual LM instead of bilingualizing the LM. This allows removing the constraints imposed by the need to preserve the model's capabilities in the source language. In addition, we generalize the semantic subword mapping done by Tran (2020) to consider an arbitrary number of semantically similar subword with an arbitrary temperature. We are the first to show that a cross-lingually transferred model can outperform monolingual models which have been trained extensively from scratch in the target language, while requiring substantially less computational resources.
To initialize the model in the target language, we copy the inner (non-embedding) parameters from the source model. Our goal, then, is given the tokenizer  $T^s$  in the source language with vocabulary  $\mathbb{U}^s$ , the corresponding token embeddings  $E^s$ , and a tokenizer  $T^t$  in the target language with vocabulary  $\mathbb{U}^t$ , to find a good initialization of the embeddings  $E^t$  by using  $E^s$ . To this end, we use existing bilingual word embeddings enriched with subword information, containing a set of words and subword n-grams in the source and target language and their aligned vectors. We denote the set of words and n-grams in the source and target language as  $\mathbb{V}^s$  and  $\mathbb{V}^t$  respectively, and the aligned static embeddings as  $W^s$  and  $W^t$ . In Appendix D we consider an alternative method if no subword information is available in the bilingual word embeddings.
First, independently for both languages, we compute static subword embeddings for tokens in the tokenizer vocabulary in the same semantic space as the static word embeddings (Section 3.1). This results in subword embeddings  $\mathbf{U}^s$  and  $\mathbf{U}^t$  for the source and target language, respectively. Next, we use  $\mathbf{U}^s$  and  $\mathbf{U}^t$  to compute the semantic similarity of every subword in  $\mathbb{U}^s$  to every subword in  $\mathbb{U}^t$ . Using these semantic similarities, we initialize the embeddings in  $\mathbf{E}^t$  through a convex combination of embeddings in  $\mathbf{E}^s$  (Section 3.2). By applying WECHSEL, the vectors of  $\mathbf{E}^t$  are in the same semantic space as  $\mathbf{E}^s$ , where a subword in the target language is semantically similar to its counterpart(s) in the source language. These steps are summarized in Figure 1 and explained in more detail in the following.
3.1 Subword Embedding Computation
The process of mapping word embeddings to subword embeddings is done individually for the source and the target language. Given a tokenizer  $T$  with vocabulary  $\mathbb{U}$  and embeddings  $W$ , the goal is to find subword embeddings  $U$  for subwords in  $\mathbb{U}$  in the same semantic space as  $W$ . To this end, we decompose subwords in  $\mathbb{U}$  into n-grams and compute the embedding by taking the sum of the embeddings of all occurring n-grams, equivalent to how embeddings for out-of-vocabulary words are computed in fastText (Bojanowski et al., 2017).
$$
\boldsymbol {u} _ {x} = \sum_ {g \in \mathbb {G} ^ {(x)}} \boldsymbol {w} _ {g}
$$
Figure 1: Summary of our WECHSEL method. We show inputs, intermediate results and outputs.
where  $\mathbb{G}^{(x)}$  is the set of n-grams occurring in the subword  $x$  and  $\pmb{w}_{g}$  is the embedding of the n-gram  $g$ . Subwords in which no known n-gram occurs are initialized to zero.
3.2 Subword similarity-based Transfer
Applying the previous step to both source and target language results in the subword embeddings  $\mathbf{U}^s$  and  $\mathbf{U}^t$  over the subword vocabularies  $\mathbb{U}^s$  and  $\mathbb{U}^t$ , respectively. Our aim is to leverage these embeddings to find an effective transformation from  $\mathbf{E}^s$  to  $\mathbf{E}^t$ . We first compute the cosine similarity of every subword  $x \in \mathbb{U}^t$  to every subword  $y \in \mathbb{U}^s$ , denoted as  $s_{x,y}$ .
$$
s _ {x, y} = \frac {\boldsymbol {u} _ {x} ^ {t} \boldsymbol {u} _ {y} ^ {s T}}{\| \boldsymbol {u} _ {x} ^ {t} \| \| \boldsymbol {u} _ {y} ^ {s} \|}
$$
We now exploit these similarities to initialize embeddings in  $\pmb{E}^{t}$  by a convex combination of embeddings in  $\pmb{E}^{s}$ . In particular, each subword embedding in  $\pmb{E}^{t}$  is defined as the weighted mean of the  $k$  nearest embeddings in  $\pmb{E}^{s}$  according to the similarity values. The weighting is done by a softmax of the similarities with temperature  $\tau$ .
$$
\boldsymbol {e} _ {x} ^ {t} = \frac {\sum_ {y \in \mathcal {J} _ {x}} \exp (s _ {x , y} / \tau) \cdot \boldsymbol {e} _ {y} ^ {s}}{\sum_ {y ^ {\prime} \in \mathcal {J} _ {x}} \exp (s _ {x , y ^ {\prime}} / \tau)}
$$
where  $\mathcal{I}_x$  is the set of  $k$  neighbouring subwords in the source language. Subword embeddings for which  $\pmb{U}^{t}$  is zero are initialized from a random normal distribution  $\mathcal{N}(\mathbb{E}[E^s],\mathrm{Var}[E^s])$ .
We evaluate our method by transferring the English RoBERTa (Liu et al., 2019) and the English GPT-2 model (Radford et al., 2019) to French, German, Chinese and Swahili. We refer to these languages as medium-resource languages. In addition, we study the benefits of our method on four low-resource languages, namely Sundanese, Scottish Gaelic, Uyghur and Malagasy.
We evaluate WECHSEL-RoBERTa by fine-tuning on XNLI (Conneau et al., 2018), and on the balanced train-dev-test split of WikiANN (Rahimi et al., 2019; Pan et al., 2017) to evaluate NLI and NER performance, respectively. The hyperparameters used for fine-tuning are reported in Appendix B. GPT-2 is evaluated by Perplexity (PPL) on a held-out set from the same corpus on which the model was trained on. Due to the difficulty of extrinsic evaluation on low-resource languages, we only train GPT-2 models in these languages, and evaluate their performance intrinsically via Language Modelling Perplexity on a held-out set. We use the pretrained models RoBERTaBase with 125M parameters, and the small GPT-2 variant with 117M parameters provided by HuggingFace's Transformers (Wolf et al., 2020) in all experiments.
Since under limited training regimes such as ours, using a smaller corpus does not in general degrade performance (Martin et al., 2020), we use a subset of 4GiB from the OSCAR corpus for German, French and Chinese. For the other languages, we use data from the CC-100 corpus (Conneau et al., 2020) which contains 1.6GiB, 0.1GiB, 0.1GiB, 0.4GiB and 0.2GiB for Swahili, Sundanese, Scottish Gaelic, Uyghur and Malagasy, respectively. To obtain aligned word embeddings between the source and the target language we use monolingual fastText word embeddings² (Bojanowski et al., 2017). We align these embeddings using the Orthogonal Procrustes method (Schönemann, 1966; Artetxe et al., 2016) with bilingual dictionaries from MUSE³ (Conneau et al., 2017) for French, German and Chinese and a bilingual dictionary from FreeDict⁴ (Bánski and Wojtowicz, 2009) for Swahili. For the low-resource languages, we use bilingual dictionaries scraped from Wiktionary.⁵
Model	Tokens trained on	Factor
WECHSEL-RoBERTa	65.5B	1.0x
TransInner-RoBERTa	65.5B	1.0x
FullRand-RoBERTa	65.5B	1.0x
CamemBERT	419.4B	6.4x
GBERTBase	255.6B	3.9x
BERTBase-Chinese	131.1B	2.0x
Table 1: Tokens trained on in the target language between our models and previous monolingual models.
We choose temperature  $\tau = 0.1$  and neighbors  $k = 10$  for WECHSEL by conducting a parameter search over a grid with varying values for  $k$  and  $\tau$  using linear probes (Appendix A). We train tokenizers in the target languages using a vocabulary size of 50k tokens and byte-level BPE (Radford et al., 2019). After applying WECHSEL, we continue training RoBERTa on the MLM objective and GPT-2 on the CLM objective. We compare against two baseline methods.
- TransInner: Randomly initializing  $\boldsymbol{E}^{t}$  while transferring all other parameters from the English model as in de Vries and Nissim (2021). After training only embeddings for a fixed amount of steps while freezing other parameters, the entire model is trained for the remaining steps. In preliminary experiments reported in Appendix E, we compare the method by Zoph et al. (2016) with TransInner, observing superior performance of TransInner, so we choose TransInner as the baseline for crosslingual transfer in all our experiments.
- FullRand: Training from scratch in the target language, as is commonly done when training BERT-like or GPT-like models in a new language (Antoun et al., 2020; Louis, 2020; Chan et al., 2020; Martin et al., 2020).
All models are trained for 250k steps with the same hyperparameters across all languages (reported in Appendix B). Training one model takes around 4 days on a TPUv3-8. For WECHSEL and FullRand we use a learning rate (LR) schedule with linear warmup from zero to the peak LR for the first  $10\%$  of steps, followed by a linear decay to zero. For TransInner, we perform two warmup phases from zero to peak LR, once for the first  $10\%$  of steps for training embeddings only, then again for the remaining steps while training the entire model.
In addition to the mentioned baselines trained under this setting, we compare the results of
$^{2}$ https://fasttext.cc
4https://freedict.org
5available at github.com/cpjku/wechsel
Lang	Model	Score@0	Score@25k	Score@250k	Score (more training)
NLI	NER	Avg	NLI	NER	Avg	NLI	NER	Avg	NLI	NER	Avg
French	WECHSEL-RoBERTa	78.25	86.93	82.59	81.63	90.26	85.95	82.43	90.88	86.65	-	-	-
TransInner-RoBERTa	60.86	69.57	65.21	65.49	83.82	74.66	81.75	90.34	86.04	-	-	-
FullRand-RoBERTa	55.71	70.79	63.25	69.02	84.24	76.63	75.28	89.30	82.29	-	-	-
CamemBERT	-	-	-	-	-	-	-	-	-	80.88	90.26	85.57
XLM-RBase	-	-	-	-	-	-	-	-	-	79.25	89.48	84.37
German	WECHSEL-RoBERTa	75.64	84.53	80.08	81.11	89.05	85.08	81.79	89.72	85.76	-	-	-
TransInner-RoBERTa	58.51	65.23	61.87	64.78	82.05	73.42	80.75	89.30	85.02	-	-	-
FullRand-RoBERTa	54.82	66.84	60.83	68.02	81.53	74.77	75.48	88.36	81.92	-	-	-
GBERTBase	-	-	-	-	-	-	-	-	-	78.64	89.46	84.05
XLM-RBase	-	-	-	-	-	-	-	-	-	78.58	88.76	83.67
Chinese	WECHSEL-RoBERTa	63.23	72.79	68.01	77.19	79.07	78.13	78.32	80.55	79.44	-	-	-
TransInner-RoBERTa	46.95	69.06	58.01	52.96	73.35	63.16	76.99	80.00	78.49	-	-	-
FullRand-RoBERTa	44.24	57.95	51.09	58.34	64.84	61.59	71.38	78.35	74.86	-	-	-
BERTBase-Chinese	-	-	-	-	-	-	-	-	-	76.55	82.05	79.30
XLM-RBase	-	-	-	-	-	-	-	-	-	76.41	78.36	77.38
Swahili	WECHSEL-RoBERTa	60.28	74.38	67.33	73.87	87.63	80.75	75.05	87.39	81.22	-	-	-
TransInner-RoBERTa	54.67	64.46	59.56	58.85	80.27	69.56	74.10	87.05	80.57	-	-	-
FullRand-RoBERTa	50.59	62.35	56.47	63.79	83.49	73.64	70.34	87.34	78.84	-	-	-
XLM-RBase	-	-	-	-	-	-	-	-	-	69.18	87.37	78.28
Table 2: Results from fine-tuning RoBERTa models. We report accuracy for NLI on XNLI and micro F1 score for NER on WikiANN. Results are averaged over 3 runs. We report scores before training ( $\mathbf{Score}@\mathbf{0}$ ), after  $10\%$  of steps ( $\mathbf{Score}@\mathbf{25k}$ ) and after training ( $\mathbf{Score}@\mathbf{250k}$ ). We also report results from fine-tuning prior monolingual models and XLM-R ( $\mathbf{Score}(\text{more training})$ ), all trained on more tokens than our models. For each language, the best results in every column are indicated with underlines. The overall best results including the comparison with existing monolingual/multilingual models of comparable size are shown in bold.