SlowGuess/NAACL_2021_OCR huggingface dataset:
In this section, we verify the effectiveness of the three loss components in MVR by removing each of them from the objective. The ablation results on mBERT for all tasks are listed in Tab. 3. Removing any of the three loss components hurts the model performance by about the same amount for most of the tasks. For the question answering tasks, however, removing the cross-entropy loss on the deterministically segmented inputs reduces the model performance by almost half. This is likely because under this setting, the model only learns to locate exact spans for inputs tokenized by BPE-dropout, while we use the standard BPE to segment the inputs at test time.
6 Analyses
In this section, we perform several analyses to better understand the behavior and root causes of the accuracy gains realized by our method.
6.1 Effect on over-segmentation
In this section, we analyze the effect of our methods on languages and words with different subword segmentation granularity. We focus on the NER
task because it contains a diverse set of over 40 languages. We calculate the average number of subword pieces in a language, and plot the gains over the baseline for these languages with respect to their average subwords in Fig. 3. To visualize the relationship between the two values, we also fit a trend line and record its coefficient for each method in the legend. We consider three methods for mBERT: SR, MVR without consistency loss, and the full MVR. The trend line for MVR has a positive coefficient, indicating that it improves more on languages that are more overly segmented. Removing the consistency loss tends to hurt more for these languages. SR, on the other hand, does not tend to favor languages with more subword segmentation.
Next, we bucket all the words together based on how many subwords they are segmented into, and compare the performance of our methods for each word bucket. We use the XLM-R model and plot the results in Fig. 4. SR brings slightly more improvements on average for words that are split into 4 or more pieces for the large model. MVR outperforms SR for all categories, especially for difficult words that are segmented into 5 or more subwords.
Gains on Latin vs. non-Latin script In addition, it is notable that we fine-tune the model using labeled data from English, a Latin script language, while the non-Latin scripted languages might have larger segmentation and vocabulary discrepancies from English. We thus also plot the score improvements of both SR and MVR over the baseline for languages with and without Latin script in Fig. 6. We use a lighter shade to represent improvements for Latin-script languages and a darker shade for languages with non-Latin scripts. Across all the
Method Metrics	Avg.	XNLI Acc.	PAWS-X Acc.	NER F1	XQuAD F1/EM	MLQA F1/EM
MVR	68.8	67.2±0.3	85.6±0.3	62.7±0.4	66.3±0.2 / 51.7±0.2	62.2±0.2 / 45.3±0.1
- Det. Seg CrossEnt	52.8	66.7±0.5	85.5±0.2	62.4±0.7	25.0±8.2 / 15.6±6.7	24.3±8.6 / 13.1±6.4
- Prob. Seg CrossEnt	67.7	66.7±0.6	85.0±0.3	62.3±0.7	64.0±0.3 / 48.7±0.4	60.3±0.4 / 43.1±0.3
- consistency loss	68.2	66.5±0.7	85.3±0.3	62.3±0.6	65.2±0.2 / 50.2±0.4	61.7±0.1 / 44.5±0.2
Table 3: Effect of removing each loss component on mBERT.
Figure 3: mBERT gains over the NER baseline by average word pieces of a language. MVR tends to benefit oversegmented languages more.
Figure 4: XLM-R large gains over the NER baseline for words with increasing number of subword pieces.
Figure 5: mBERT improvements over the NER baseline for examples with different entropy. Consistency loss helps examples with higher entropy more.
Figure 6: Gains over the baseline for languages with Latin vs. non-Latin script. Both SR and MVR improve more for non-Latin languages.
Figure 7: Increase in distance to the ensemble distribution by removing the consistency loss on the NER task. The languages are labeled based on the method with closer distance to the ensemble distribution. The consistency loss shifts model prediction closer to the ensemble distribution.
tasks, both SR and MVR generally have larger improvements on languages with non-Latin script. MVR, which is represented by blue shades, generally outperforms SR for both the Latin and non-Latin scripted languages across all models. While SR sometimes underperforms the baseline on Latin scripted languages, especially for XLM-R models, MVR delivers consistent improvements over the baseline across both types of languages. Overall, MVR achieves the largest improvements over SR for languages with non-Latin scripts.
6.2 Effect of consistency loss
One of the novel components of MVR is the consistency loss between two different segmentations of the input. In this section we analyze two hypotheses about the source of benefit provided thereby.
Label smoothing The first hypothesis is that the consistency loss may be able to mitigate overconfident predictions by calibrating the two output distributions against each other. This effect is similar to label smoothing (Szegedy et al., 2015; Yuan et al., 2020), which softens the one-hot target label by adding a loss of uniform distribution over all class labels and has proven helpful across a wide variety of models. To measure this, we plot the F1 improvement on the NER task for examples categorized by increasing predictive entropy in Fig. 5. MVR leads to more improvements on examples with higher entropy, or those that the model is more uncertain about, indicating that MVR is indeed helping the model improve on examples where it is not confident.
Figure 8: Gains of MVR and SR for English. While SR harms the performance on English, MVR generally improves it.
Ensemble effect The second hypothesis is that the consistency loss could regularize the model to be closer to the ensemble of models trained on standard deterministically segmented inputs and probabilistically segmented inputs. To verify this hypothesis, we first calculate the ensembled prediction probability of the baseline and the SR models for each language. Then we compare the KL divergence between this ensemble distribution and MVR with or without the consistency loss. In Fig. 7, we plot this KL divergence difference between the MVR without consistency loss and the full MVR for each language in NER. For most of the languages, the full MVR has lower KL divergence with the ensemble distribution, which indicates that the consistency loss trains the model to be closer to the ensemble of two inputs.
6.3 MVR also improves English
Although SR improves the model performance averaged over all languages, surprisingly it can hurt the performance on English, the language we use for fine-tuning. Fig. 8 shows the improvement on English over the baseline for both SR and MVR, and notably English performance decreases for all tasks on mBERT. MVR, on the other hand, generally brings improvements for English across both mBERT and XLM-R large models. This is likely because MVR also utilizes English inputs with standard segmentation, the method used at pretraining time, which allows it to take full advantage of the information encoded during pretraining.
Several works propose to optimize subword-sensitive word encoding methods for pretrained language models. Ma et al. (2020) uses convolutional neural networks (Kim, 2014) on characters to calculate word representations. Zhang and Li (2020) propose to add phrases into the vocabulary
for Chinese pretrained language models. However, they focus on improving the vocabulary of pretrained representations of a single language, and they require modification to the model pretraining stage. Chung et al. (2020) propose to cluster related languages together and run subword vocabulary construction on each language cluster when constructing vocabularies for mBERT. Their method is also applied at the pretraining stage and could be combined with our method for potential additional improvements.
Our method is also related to prior work that optimize word representations for NMT and language modeling. Character level embeddings have been utilized instead of subword segmentation for NMT (Cherry et al., 2018; Lee et al., 2017; Ataman and Federico, 2018) and language modeling (Kim et al., 2016; Jozefowicz et al., 2016). Wang et al. (2019) propose a multilingual word embedding method for NMT that relies on character n-gram embedding and a latent semantic embedding shared between different languages. Ataman and Federico (2018) show that character n-gram based embedding performs better than BPE for morphologically rich languages. He et al. (2020) propose to learn the optimal segmentation given a subword vocabulary for NMT.
Our method is inspired by semi-supervised learning methods that enforce model consistency on unlabeled data. Several self-training methods utilize unlabeled examples to minimize the distance between the model predictions based on the unlabeled example and a noised version of the same input (Miyato et al., 2017b,a; Xu and Yang, 2017; Clark et al., 2018; Xie et al., 2018). Xu and Yang (2017) use knowledge distillation on unlabeled data to adapt models to a new language. Clark et al. (2018) propose to mask out different parts of the unlabeled input and encourage the model to make consistent prediction given these different inputs. These methods all focus on semi-supervised learning, while our method regulates model consistency to mitigate the subword segmentation discrepancy between different languages.
We believe that the results in this paper convincingly demonstrate that standard deterministic subword segmentation is sub-optimal for multilingual pretrained representations. Even incorporating simple methods for subword regularization such as
BPE-dropout at fine-tuning can improve the cross-lingual transfer of pretrained models, and our proposed Multi-view Subword Regularization method further shows consistent and strong improvements over a variety of tasks for models built upon different subword segmentation algorithms. Going forward, we suggest that some variety of subword regularization, MVR or otherwise, should be a standard component of the fine-tuning of pre-trained representations that use subword segmentation.
The first author XW is supported by the Apple PhD fellowship. This project is made possible by the computing resources from the Pittsburgh Supercomputing Center. The authors would like to thank Adhi Kuncoro for his comments regarding the draft of the paper.
Judit Ács. 2019. Exploring bert's vocabulary.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingual Transferability of Monolingual Representations. In ACL.
Duygu Ataman and Marcello Federico. 2018. Compositional representation of morphologically-rich input for neural machine translation. ACL.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of EMNLP.
Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In *Findings of EMNLP* 2020.
Colin Cherry, George Foster, Ankur Bapna, Orhan First, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. CoRR.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In EMNLP.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In EMNLP.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In NeurIPS.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In EMNLP, pages 2475–2485.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2020. Dynamic programming encoding for subword segmentation in neural machine translation. In ACL, pages 3042-3051, Online. ACL.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In ICML.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhoun. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In EMNLP.
Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. *Arxiv*.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. AAAI.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In EMNLP.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. TACL.
Patrick Lewis, Barlas Oğuz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating Cross-lingual Extractive Question Answering. In ACL.
Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Charbert: Character-aware pre-trained language model. In COLING.
Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2017a. Adversarial training methods for semi-supervised text classification. In *ICLR*.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2017b. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. In IEEE Transactions on Pattern Analysis and Machine Intelligence.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In ACL, pages 1946-1958.
Jonas Pfeiffer, Ivan Vuli, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. In Proceedings of EMNLP 2020.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In ACL, Florence, Italy.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In ACL.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In ICASSP.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. In CVPR.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. 2019. Multilingual neural machine translation with soft decoupled encoding. In ICLR.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In EMNLP.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. 2018. Unsupervised data augmentation for consistency training. In EMNLP.
Ruochen Xu and Yiming Yang. 2017. Cross-lingual distillation for text classification. In ACL, Vancouver, Canada.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In EMNLP, pages 3685-3690.
Li Yuan, Francis E.H.Tay, Guilin Li, Tao Wang, and Jiashi Feng. 2020. Revisit knowledge distillation: a teacher-free framework. In CVPR.
Xinsong Zhang and Hang Li. 2020. Ambert: A pretrained language model with multi-grained tokenization. In *arxiv*.