SlowGuess/NAACL_2021_OCR huggingface dataset:
Xinyi Wang<sup>1</sup> Sebastian Ruder<sup>2</sup> Graham Neubig<sup>1</sup>
$^{1}$ Language Technology Institute, Carnegie Mellon University
$^{2}$ DeepMind
xinyiw1@cs.cmu.edu, ruder@google.com, gneubig@cs.cmu.edu
Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary. However, standard heuristic algorithms often lead to sub-optimal segmentation, especially for languages with limited amounts of data. In this paper, we take two major steps towards alleviating this problem. First, we demonstrate empirically that applying existing subword regularization methods (Kudo, 2018; Provilkov et al., 2020) during fine-tuning of pre-trained multilingual representations improves the effectiveness of cross-lingual transfer. Second, to take full advantage of different possible input segmentations, we propose Multi-view Subword Regularization (MVR), a method that enforces the consistency between predictions of using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark (Hu et al., 2020) show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms.
Multilingual pre-trained representations (Devlin et al., 2019; Huang et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020) are now an essential component of state-of-the-art methods for cross-lingual transfer (Wu and Dredze, 2019; Pires et al., 2019). These methods pretrain an encoder by learning in an unsupervised way from raw textual data in up to hundreds of languages which can then be fine-tuned on annotated data of a downstream task in a high-resource language, often English, and transferred to another language. In order to encode hundreds of languages with diverse vocabulary, it is standard for such multilingual models to employ a shared subword vocabulary jointly learned on the
en	excitement	fr	excita/tion
de	Auf/re/gung	pt	excita/çã­o
el	εν/θ/ούσι/ασμός	ru	BOJIH/ēHNé
Table 1: XLM-R segmentation of "excitement" in different languages. The English word is not segmented while the same word in other languages is over-segmented. A better segmentation would allow the model to match the verb stem and derivational affix across languages.
multilingual data using heuristic word segmentation methods based on byte-pair-encoding (BPE; Sennrich et al., 2016) or unigram language models (Kudo and Richardson, 2018) (details in §2). However, subword-based preprocessing can lead to sub-optimal segmentation that is inconsistent across languages, harming cross-lingual transfer performance, particularly on under-represented languages. As one example, consider the segmentation of the word "excitement" in different languages in Tab. 1. The English word is not segmented, but its translations in the other languages, including the relatively high-resourced French and German, are segmented into multiple subwords. Since each subword is mapped to a unique embedding vector, the segmentation discrepancy—which generally does not agree with a language's morphology—could map words from different languages to very distant representations, hurting cross-lingual transfer. In fact, previous work (Conneau et al., 2020; Artetxe et al., 2020) has shown that heuristic fixes such as increasing the subword vocabulary capacity and upsampling low-resource languages during learning of the subword segmentation can lead to significant performance improvements.
Despite this, there is not much work studying or improving subword segmentation methods for cross-lingual transfer. Bostrom and Durrett (2020) empirically compare several popular word segmentation algorithms for pretrained language models of a single language. Several works propose to use different representation granularities, such as
1Code for the method is released here: https://github.com/cindyxinyiwang/multiview-subword-regularization
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473-482 June 6-11, 2021. ©2021 Association for Computational Linguistics
phrase-level segmentation (Zhang and Li, 2020) or character-aware representations (Ma et al., 2020) for pretrained language models of a single high-resource language, such as English or Chinese only. However, it is not a foregone conclusion that methods designed and tested on monolingual models will be immediately applicable to multilingual representations. Furthermore, they add significant computation cost to the pretraining stage, which is especially problematic for multilingual pretraining on hundreds of languages. The problem of suboptimal subword segmentation has drawn more attention in the context of neural machine translation (NMT). Specifically, subword regularization methods have been proposed to improve the NMT model of a single language pair by randomly sampling different segmentations of the sentences during training (Kudo, 2018; Provilkov et al., 2020). However, these methods have not been applied to multilingual NMT or pretrained language models and it is similarly not clear if they are useful for cross-lingual transfer.
In this paper, we make two contributions to close this gap. First, we perform the first (to our knowledge) empirical examination of subword regularization methods on a variety of cross-lingual transfer tasks from the XTREME benchmark (Hu et al., 2020). We demonstrate that despite its simplicity, this method is highly effective, providing consistent improvements across a wide variety of languages and tasks for both multilingual BERT (mBERT; Devlin et al., 2019) and XLM-R (Conneau et al., 2020) models. Analysis of the results shows that this method is particularly effective for languages with non-Latin scripts despite only being applied during English fine-tuning.
Further, we posit that naively applying probabilistic segmentation only during fine-tuning may be sub-optimal as it creates a discrepancy between the segmentations during the pretraining and fine-tuning stages. To address this problem, we propose Multi-view Subword Regularization (MVR; Fig. 1), a novel method—inspired by the usage of consistency regularization in semi-supervised learning methods (Clark et al., 2018; Xie et al., 2018)—which utilizes both the standard and probabilistically segmented inputs, enforcing the model's predictions to be consistent across the two views. Such consistency regularization further improves accuracy, with MVR finally demonstrating consistent gains of up to 2.5 points over the standard
Figure 1: Fine-tuning models using MVR on data  $(x^{*},y^{*})$
practice across all models and tasks. We analyze the sources of the improvement from consistency regularization and find that it can be attributed to both label smoothing and self-ensembling.
2 Background: Subword Segmentation
Here, we first discuss two common deterministic segmentation methods based on byte pair encoding (BPE) and unigram language models (ULM), discuss their probabilistic variants, and explain how to incorporate them in training.
2.1 Deterministic Segmentation
The most widely used subword segmentation methods first estimate a segmentation model from the training corpus in an unsupervised fashion. They then produce a segmentation  $\widehat{x}$  of the input  $x^{*}$  under the estimated segmentation model  $P(x)$ :
$$
\widehat {x} = \underset {x \in S (x ^ {*})} {\operatorname {a r g m a x}} P (x)
$$
Here  $S(x^{*})$  is the set of all possible segmentations, and  $P(x)$  is the likelihood of a given segmentation. Note that  $\widehat{x}$  is deterministically selected for each input  $x^{*}$ .
Byte-pair encoding (BPE) The popular BPE algorithm (Sennrich et al., 2016) initializes the vocabulary with individual characters and initially represents each word as a sequence of characters. It then counts the most frequent character token bigrams in the data, merges them into a new token, and adds the new token to the vocabulary. This process is done iteratively until a predefined vocabulary size is reached.
To segment a word, BPE simply splits the word into character tokens, and iteratively merges adjacent tokens with the highest priority until no merge operation is possible. That is, for an input  $x^{*}$ , it assigns segmentation probability  $P(\hat{x}) = 1$  for the sequence  $\hat{x}$  obtained from the greedy merge operations, and assigns other possible segmentations a probability of 0.
Notably, a variant of this method (Schuster and Nakajima, 2012) is used for the mBERT embedding model (Devlin et al., 2019).
Unigram language model (ULM) The ULM method (Kudo and Richardson, 2018) starts from a reasonably large seed vocabulary, which is iteratively pruned to maximize the training corpus likelihood under a unigram language model of the subwords until the desired vocabulary size is reached.
During segmentation, ULM decodes the most likely segmentation of a sentence under the estimated language model using the Viterbi algorithm. This method is used in the XLM-R cross-lingual embeddings (Conneau et al., 2020).
2.2 Probabilistic Segmentation
As explained in §1, one drawback of both word segmentation algorithms is that they produce a deterministic segmentation for each sentence, even though multiple segmentations are possible given the same vocabulary. In contrast, Kudo (2018) and Provilkov et al. (2020) have proposed methods that enable the model to generate segmentations probabilistically. Instead of selecting the best subword sequence for input  $x^{*}$ , these method stochastically sample a segmentation  $x^{\prime}$  as follows:
$$
x ^ {\prime} \sim P ^ {\prime} (x) \text {w h e r e} P ^ {\prime} (x) \propto \left\{ \begin{array}{l} P (x) \text {i f} x \in S (x ^ {*}) \\ 0 \text {o t h e r w i s e} \end{array} \right.
$$
Here we briefly introduce these two methods.
BPE-dropout This method is used together with the BPE algorithm, randomly dropping merge operations with a given probability  $p$  while segmenting the input data (Provilkov et al., 2020).
ULM-sample As the ULM algorithm relies on a language model to score segmentation candidates for picking the most likely segmentation, Kudo (2018) propose to sample from these segmentation candidates based on their language model scores.
2.3 Subword Regularization (SR)
Subword regularization (Kudo, 2018) is a method that incorporates probabilistic segmentation at training time to improve the robustness of models to different segmentations. The idea is conceptually simple: at training time sample different segmentations  $x'$  for each input sentence  $x^*$ . Previous works (Kudo, 2018; Provilkov et al., 2020) have demonstrated that subword regularization using both BPE-dropout and ULM-sampling are effective at improving machine translation accuracy,
Figure 2: Percentage of words with different number of segments from different languages.
particularly in cross-domain transfer settings where the model is tested on a different domain than the one on which it is trained.
3 Subword Regularization for Cross-lingual Transfer
While sub-optimal word segmentation is a challenge in monolingual models, it is an even bigger challenge for multilingual pretrained models. These models train a shared subword segmentation model jointly on data from many languages, but the segmentation can nonetheless be different across languages, stemming from two main issues. First, the granularity of segmentation differs among languages, where the segmentation model tends to over-segment low-resource languages that do not have enough representation in the joint training data (Ács, 2019). Fig. 2 shows the distribution of words from languages from different language families based on the number of subwords they are split into. We can see that the majority of English words are not segmented at all, while many languages only have less than half of the words unsegmented. Notably, even though Burmese (my) is a language with little inflectional morphology, almost a quarter of the words are segmented into more than nine subwords. Second, the segmentation might still be inconsistent between different languages even if the granularity is similar, as explained in Tab. 1. For example, neither the English word "excitement" nor the same word in French "excita/tion" are overly segmented, but segmenting the English word into "excite/ment" would allow the model to learn a better cross-lingual alignment.
Despite these issues, few methods have tried to address this subword segmentation problem for multilingual pretrained models. Chau et al. (2020)
2We use Pan et al. (2017)'s named entity recognition test data with mBERT's tokenizer.
propose to adapt a pretrained multilingual model to a new language by augmenting the vocabulary with a new subword vocabulary learned on the target language, but this method might not help for languages other than the target language it adapts to. Chung et al. (2020) propose to separately construct a subword segmentation model for each cluster of related languages for pretraining the multilingual representations. However, directly modifying the word segmentation requires retraining large pretrained models, which is computationally prohibitive in most cases.
In this paper, we instead propose a more efficient approach of using probabilistic segmentation during fine-tuning on labeled data of a downstream task. As mismatch in segmentation is one of the factors harming cross-lingual transfer, we expect a model that becomes more robust to different varieties of segmentation in one language will be more accommodating to differing segmentations in other languages during inference. Despite the simplicity of this method it is, as far as we are aware, unattested in the literature, and we verify in § 5.3 that it significantly improves the cross-lingual transfer performance of multilingual pretrained models.
4 Multi-view Subword Regularization
Previous attempts at SR have mainly applied it to models trained from scratch for tasks such as MT. However, the situation is somewhat different when fine-tuning pre-trained representations, in which case the original pre-trained models are generally not trained on sampled segmentations. This discrepancy between the segmentation of the English labeled data and the segmentation of English monolingual data during pretraining might hurt the ability of the model to take full advantage of the parameters learned during the pretraining stage. To reduce this pretraining-fine-tuning discrepancy, we propose Multi-view Subword Regularization (MVR), a method for learning from multiple segmented versions of the same data and enforcing the consistency of predictions over different segmentations.
Given the input  $\widehat{x}_i$  tokenized with the deterministic segmentation such as BPE, and  $x_{i}^{\prime}$ , the same input tokenized with the corresponding probabilistic segmentation algorithm such as BPE-dropout,
the objective for MVR has three components
$$
\begin{array}{l} J (\theta) = \sum_ {i = 1} ^ {n} \left[ - \frac {1}{2} \underbrace {\log p _ {\theta} \left(y _ {i} \mid \widehat {x} _ {i}\right)} _ {\text {D e t . S e g C r o s s E n t}} - \frac {1}{2} \underbrace {\log p _ {\theta} \left(y _ {i} \mid x _ {i} ^ {\prime}\right)} _ {\text {P r o b . S e g C r o s s E n t}} \right. \\ + \lambda \underbrace {D \left(p _ {\theta} \left(y _ {i} \mid \widehat {x} _ {i}\right) \mid \mid p _ {\theta} \left(y _ {i} \mid x _ {i} ^ {\prime}\right)\right)} _ {\text {C o n s i s t e n c y l o s s}} ] \tag {1} \\ \end{array}
$$
1. A cross-entropy loss using the standard deterministic segmentation. This loss acts on data whose segmentation is consistent with the segmentation seen during pretraining. It thus maximizes the benefit of pretrained representations.
2. A cross entropy loss using probabilistic segmentation. It allows the model to learn from different possible segmentations of the same input.
3. A distance term  $D(\cdot ||\cdot)$  between the model prediction distributions over the two different versions of the input. We use KL divergence as the distance metric and a hyperparameter  $\lambda$  to balance the supervised cross-entropy losses and the consistency loss. Minimizing the distance between the two distributions enforces the model to make consistent predictions under different input segmentations, making it robust to sub-optimal segmentation of multilingual data.
Flattening the prediction The benefit of consistency regularization might be limited if the model prediction becomes overly confident on certain classes, especially when the number of output classes is large. Inspired by a similar technique in knowledge distillation (Hinton et al., 2014), we can use a softmax temperature  $\tau$  to flatten the prediction distribution when calculating the consistency loss. Specifically, the distance loss between two prediction distributions in Eq. 1 can be written as  $D(p_{\theta}^{\mathrm{flat}}(y_i|\widehat{x}_i) || p_{\theta}(y_i|x_i'))$ , where
$$
p _ {\theta} ^ {\text {f l a t}} \left(y _ {i} \mid \widehat {x} _ {i}\right) = \frac {\exp \left(z _ {y}\right) / \tau}{\sum_ {y ^ {\prime}} \exp \left(z _ {y ^ {\prime}}\right) / \tau} \tag {2}
$$
and  $z_{y}$  is the logit for output label  $y_{i}$ . Normally  $\tau$  is set to 1, and a higher  $\tau$  makes the probability distribution more evenly distributed over all classes. In our experiments, we find that  $\tau = 1$  works well for most of the tasks and  $\tau = 2$  works slightly better for tasks that have larger output label spaces.
<sup>3</sup>As in semi-supervised learning (Clark et al., 2018), we expect our method to also be effective when applied to unlabeled data, e.g. using target language adaptation (Pfeiffer et al., 2020), which we leave for future work.
Efficiency At inference time, we simply use the model prediction based on the input tokenized by deterministic segmentation only. Therefore, our method does not add additional decoding latency. MVR needs about twice the fine-tuning cost compared to the baseline. However, compared to pretraining and inference usage of a model, fine-tuning is generally the least expensive component.
5.1 Training and evaluation
We evaluate the multilingual representations using tasks from the XTREME benchmark (Hu et al., 2020), focusing on the zero-shot cross-lingual transfer with English as the source language. We consider sentence classification tasks including XNLI (Conneau et al., 2018) and PAWS-X (Yang et al., 2019), a structured prediction task of multilingual NER (Pan et al., 2017), and question-answering tasks including XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al., 2020).
We evaluate on both the mBERT model which utilizes BPE to tokenize the inputs, and the XLM-R models which uses ULM segmentation. To replicate the baseline, we follow the hyperparameters provided in the XTREME codebase<sup>4</sup>. Models are fine-tuned on English training data and zero-shot transferred to other languages. We run each experiment with 5 random seeds and record the average results and the standard deviation.
SR We use BPE-dropout (Provilkov et al., 2020) for mBERT and ULM-sample (Kudo, 2018) for XLM-R models to do probabilistic segmentation of the English labeled data. BPE-dropout sets a dropout probability of  $p \in [0,1]$  for the merge operations, where a higher  $p$  corresponds to stronger regularization. ULM-sample utilizes a sampling temperature  $\alpha \in [0,1]$  to scale the scores for segmentation candidates, and a lower  $\alpha$  leads to stronger regularization. We select the  $p$  and  $\alpha$  values based on the model performance on the English dev set of the NER task and simply use the same values across all other tasks. We set  $p = 0.1$  for BPE-dropout and  $\alpha = 0.6$  for ULM-sample.
MVR We select the hyperparameters for MVR using the English dev set performance
on the NER task. MVR works slightly better by using stronger regularization than SR, likely because using inputs deterministically segmented by the standard algorithm can balance the negative impact of bad tokenization by sampling from a more diverse set of segmentation candidates. We use  $\lambda = 0.2, p = 0.2$  for mBERT and  $\lambda = 0.6, \alpha = 0.2$  for XLM-R. We use prediction temperature  $\tau = 2$  for the question-answering tasks XQuAD and MLQA for the XLMR mdoels, and simply use  $\tau = 1$  for all other tasks. Further analysis of hyperparameters on the performance of MVR can be found in § A.1.
We compare performance of SR, MVR and the baseline for all models in Tab. 2, focusing on the average performance on all languages for each task. Our baseline numbers match or exceed the benchmark results in Hu et al. (2020) for both mBERT and XLM-R large (Hu et al. (2020) do not include results for XLM-R base) on almost all tasks.
Applying SR on English significantly improves other languages SR is surprisingly effective for mBERT—it is comparable to the baseline on XNLI and significantly improves over the baseline for the rest of the four tasks. However, the gains are less consistent for XLM-R models. For both XLM-R base and large, SR leads to improvements on the NER task and the PAWS-X classification task, but is mostly comparable to the baseline for the rest of the three tasks. SR performs better for mBERT likely because the vocabulary of mBERT is more imbalanced than that of XLM-R; it thus benefits more from the regularization methods. mBERT relies on BPE, which could be worse than ULM at tokenizing subwords into morphologically meaningful units (Bostrom and Durrett, 2020). Furthermore, mBERT has only 100K words in the vocabulary while XLM-R has a much larger vocabulary of 250K.
MVR consistently improves over SR For mBERT, it leads to improvements of over 1 to 2 points over the baseline for all tasks. It is also very effective for the XLM-R models. For both the XLM-R base and the stronger XLM-R large models, MVR improves over 1 point over the baseline on the NER task and the two classification tasks. On the question-answering tasks, MVR delivers strong improvements for the XLM-R base model while the improvements on the XLM-R
4https://github.com/google-research/xtreme
Model	Method	Avg.	XNLI	PAWS-X	NER	XQuAD	MLQA
Metrics	Acc.	Acc.	F1	F1/EM	F1/EM
mBERT	Hu et al. (2020)	67.1	65.4	81.9	62.2	64.5 / 49.4	61.4 / 44.2
Baseline (ours)	67.3	66.5±0.4	83.1±0.4	61.5±0.7	64.7±0.2 / 49.8±0.4	60.9±0.4 / 43.8±0.5
SR	68.0	66.4±0.2	85.0±0.3	62.2±0.6	64.7±0.3 / 50.0±0.3	61.5±0.3 / 44.4±0.3
MVR	68.8	67.2±0.3	85.6±0.3	62.7±0.4	66.3±0.2 / 51.7±0.2	62.2±0.2 / 45.3±0.1
XLM-R base	Baseline (ours)	71.1	74.4±0.2	84.3±0.7	60.6±0.6	70.9±0.3 / 54.9±0.5	65.5±0.3 / 47.7±0.2
SR	71.4	74.4±0.7	85.5±0.5	61.0±0.6	70.9±0.3 / 55.7±0.2	65.4±0.1 / 47.5±0.1
MVR	72.3	75.3±0.3	86.3±0.6	61.8±0.3	71.6±0.5 / 56.5±0.4	66.4±0.5 / 48.5±0.4
XLM-R large	Hu et al. (2020)	75.8	79.2	86.4	65.4	76.6 / 60.8	71.6 / 53.2
Baseline (ours)	76.1	80.3±0.4	86.9±0.5	63.6±0.3	77.0±0.2 / 61.7±0.3	72.8±0.2 / 54.5±0.1
SR	76.5	80.1±0.5	87.3±0.4	65.5±0.6	77.2±0.2 / 62.0±0.2	72.5±0.1 / 54.0±0.2
MVR	77.2	81.3±0.1	88.2±0.2	66.0±0.7	77.6±0.2 / 62.5±0.4	72.8±0.2 / 54.5±0.1
Table 2: Average performance and standard deviation of different methods for mBERT, XLM-R base and XLM-R large models. SR is especially effective for mBERT. MVR leads to significant further improvements across all models and tasks.
large model is slightly smaller. It has around 0.5 point improvement on XQuAD and has the same performance on MLQA. MVR leads to more improvements on XQuAD, probably because it has a more diverse set of languages that potentially have more sub-optimal subword segmentation. The consistent gains on both mBERT and XLM-R show that MVR is a general and flexible method for a variety of pretrained multilingual models based on different segmentation methods.
5.4 Effect of each loss component