Entropy-guided Vocabulary Augmentation of Multilingual Language Models for Low-resource Tasks
Year: 2023

Abstract:
Multilingual language models (MLLMs) like mBERT promise to extend the benefits of NLP research to low-resource languages (LRLs). However, LRL words are under-represented in the wordpiece/subword vocabularies of MLLMs. This leads to many LRL words getting replaced by UNK, or concatenated from morphologically unrelated wordpieces, leading to low task accuracy. (Pre)-training MLLMs after including LRL documents is resource-intensive in terms of both human inputs and computational resources. In response, we propose EVALM (entropy-based vocabulary augmented language model), which uses a new task-cognizant measurement to detect the most vulnerable LRL words, whose wordpiece segmentations are undesirable. EVALM then provides reasonable initializations of their embeddings, followed by limited fine-tuning using the small LRL task corpus. Our experiments show significant performance improvements and also some surprising limits to such vocabulary augmentation strategies in various classification tasks for multiple diverse LRLs, as well as code-mixed texts. We will release the code and data to enable further research1.

heading: 1 Introduction
text: It is common practice to start with a multilingual language model (MLLM) like mBERT2 or XLM-R (Conneau et al., 2020), which has been pre-trained with large multilingual corpora, and fine-tune theMLLM for diverse downstream tasks. Although MLLMs support many low-resource languages (LRLs), closer inspection of these MLLMs reveals that the portion of vocabulary allotted to LRLs can be orders of magnitude smaller than that allotted to high-resource languages (HRLs) such as English (Table 1).
1https://github.com/NLPatCNERG/EVALM 2https://github.com/google-research/bert/
blob/master/multilingual.md
Due to this imbalance, sometimes an LRL word may not be possible to segment into wordpieces as per the MLLM vocabulary, leading to the LRL word being conflated with the UNK (unknown) token. An even more insidious situation is that the MLLM vocabulary has enough (over-fragmented) wordpieces to assemble almost any LRL word (thereby dodging the obvious UNK alert), but the embeddings of these wordpieces collide with unrelated usage in HRLs, and/or are so sparsely trained that contextual aggregations fail to yield satisfactory LRL word embeddings which may lead to poor LRL task performance. On the other hand, significant human and computational investments are needed to create task-specific LRL corpora that are large enough to augment and retrain theMLLM vocabulary. In this work, we address the setting where a MLLM (that is presumably deficient in LRL coverage) must be minimally fine-tuned after modest modification to its wordpiece vocabulary, guided by specific LRL tasks. We design a measure of damage to an LRL word, caused by wordpiece fragmentation, based on a suitably defined notion of entropy of the word and constituent wordpieces, with respect to the LRL task. This measure then guides the selection of LRL words with which the vocabulary should be augmented. Subsequently, we propose various ways to initialize the embeddings of these newly-introduced words, including
8619
using information from the LRL itself, to ‘importing’ information from HRLs. We call the resulting system EVALM (entropy-based vocabulary augmented language model). We study the effect of EVALM on an existing MLLM during the fine-tuning stage for various downstream classification tasks covering multiple LRLs and also a code-mixed language. Our study shows that, for most of the datasets, EVALM’s vocabulary augmentation strategy helps improve LRL task performance by greater margins than recent best practices (Hong et al., 2021; Hofmann et al., 2022). A detailed analysis of successes and failures delineates the perimeter of EVALM’s capabilities and guides our design choices.

heading: 2 Related Work
text: Continued pre-training (Tai et al., 2020; Ebrahimi and Kann, 2021; Wang et al., 2020; Chau et al., 2020) with or without vocabulary augmentation of existing LMs like monolingual BERT, multilingual BERT (mBERT), XLM-R, etc., proves beneficial for improving domain and languagespecific performances over various tasks. Some works (Ruzzetti et al., 2021; Yu et al., 2021) focus on rare/OOV words. Liu et al. (2021) propose an embedding generator module in the pretrain-finetune pipeline to resolve vocabulary gaps. Adaptors (Sachidananda et al., 2021; Moon and Okazaki, 2020; Hofmann et al., 2021) are also showing promising outcomes in LRL modeling. Chung et al. (2020) explore multilingual vocabulary generation from language clusters. Minixhofer et al. (2021) transfer English LMs to new languages without expensive computation. Hofmann et al. (2022) propose a simple algorithm which modifies the tokenization process to preserve the morphological structure of a word. Others (Wang et al., 2019; Hong et al., 2021) focus on embedding initialization for newly added vocabulary words which are word fragments, which is also among our concerns.

heading: 3 Our system: EVALM
text: EVALM has three key components. The purpose of the first component (Section 3.1) is to identify (based on only the train fold) a subset of vulnerable LRL words whose assembly from wordpieces is likely to distort the embedding informationmade available to LRL labeling tasks. The second component (Section 3.2) comprises various possible
Algorithm 1 LRL vocabulary selection. Inputs: • C-class LRL task training corpusD, • MLLM tokenizer T • word frequency threshold θ • entropy reduction threshold γ • maximum size of augmentation set Vnew
1: W ← all words from corpusD 2: S ← ∪w∈W T (w) 3: compute n(w, c) for all LRL words w ∈W, c ∈ C 4: compute n(s, c) for all wordpieces s ∈ S, c ∈ C 5: compute p(c|w), p(c|s), H(w), H(s) as described 6: candidates = ∅ 7: for each LRL word w ∈W do 8: compute average wordpiece entropyHS(w) =∑ s∈T (w) H(s)/|T (w)| 9: compute word frequency n(w) = ∑ c n(w, c)
10: compute∆H(w) = HS(w)−H(w) HS(w) 11: features of w are ⟨n(w), |T (w)|,∆H(w)⟩ 12: if n(w) ≥ θ and∆H(w) ≥ γ then 13: add the feature triple to candidates 14: sort candidates in decreasing∆H
Output: Prefix of candidates of specified size as Vnew
policies to initialize the embeddings of the newlyintroduced LRL words. In the third component, as in AVocaDo (Hong et al., 2021), we prevent overfitting to a small LRL task corpus by regularizing embeddings of corresponding wordpieces of each sentence obtained by the pre- and postaugmentation MLLM tokenizers.

heading: 3.1 Vulnerable LRL word selection
text: Weneed a computationally efficient, task-sensitive surrogate of the value of introducing an LRL word into the wordpiece vocabulary. (Here we augment the vocabulary with whole LRL words, blocking their fragmentation entirely. More clever sharing of fragments is left for future work.) Suppose LRL word w is not in the MLLM vocabulary; w is fragmented into wordpiece sequence T (w) = s1, . . . , sT by the MLLM tokenizer T . The LRL task has C class labels. A specific label is denoted c ∈ [C] = {1, . . . , C}. The counts of w and constituent wordpieces st in each class c are denoted n(w, c) and n(st, c). Based on these counts, we define the following multinomial distributions: p(c|•) = n(•, c)/∑c′ n(•, c′) (1) where • = w, st, etc. Based on this we define the entropy H(•) = −∑c p(c|•) log p(c|•) (2) Suppose H(w) is small. This means w is potentially a good feature for the LRL task. Now suppose a wordpiece st has large H(st). That means st is being shared across other words that are dis-
tributed more evenly across classes. If this is the case for most st, then fragmentation of w may be a serious problem. To combine information from all wordpieces, we average their entropies, and use the relative increase in entropy, going from LRL word to wordpieces, as one signal for the danger of fragmenting w. As an example, suppose the word 'धरम' (religion) occurs ten times in a threeclass sentiment analysis dataset with the class distribution of ‘positive‘, ‘neutral‘, and ‘negative‘ as (1,1,8). Its wordpieces have class distributions 'ध' (100,85,80), '##र' (130,235,250), and '##म' (130,90,125). Then as per equation 2,H('धरम') = 0.639, H('ध') = 1.094, H('##र') = 1.062, and H('##र') = 1.086. The average wordpiece entropy is HS('धरम') = 1.094+1.062+1.0863 = 1.081, and the percentage of entropy reduction from average wordpiece to word entropy is about 41%. We also retain two simpler signals: the number of fragments |T (w)|, and the frequency of w in the LRL task corpus. LRL words are sorted on the amount of entropy decrease and the top LRL words proposed for vocabulary augmentation. We remove words with very low frequency and retain a prefix of specified size to obtain Vnew, the LRL words to be added to the MLLM vocabulary. Algorithm 1 shows a high-level pseudocode.

heading: 3.2 Embedding initialization
text: Herewe describe the different ways to initialize the embeddings of newly-added LRL words. InitLRL: The embedding of the newlyintroduced LRL word is initialized using other LRL wordpieces already in the MLLM dictionary. Suppose we add Bengali word 'হাসপাতাল', (‘hospital’ in English). Suppose the existing MLLM tokenizer splits it into ['হ', '##◌াস', '##প', '##◌াত', '##◌াল']. Then we initialize the embedding of 'হাসপাতাল' with the average of the existing MLLM embeddings of the fragments. InitHRL: Here we translate 'হাসপাতাল' to English (‘hospital’), tokenize it using T , and take the average embedding of the tokens in the list. InitMix: We use the average of InitLRL and InitHRL embeddings. InitRand: We randomly initialize the embeddings of the newly-added words. It is challenging to learn good contextual embedding for words in Vnew due to very small taskspecific training data compared to the MLLM pretraining corpus. Therefore, we found it neces-
sary to apply some regularization to avoid overfitting during fine-tuning. Let T , T ′ be the initial and final MLLM tokenizers. For a particular sentence S = w1, w2, ..., wI with words wi, we will get two different tokenizations; these will generally lead to different contextual embeddings E = (e1, . . . , eK) and E′ = (e′1, . . . , e′L); generallyK ̸= L. We average-pool these to get vectors e, e′ which a final layer uses for the classification task, with losses ℓT and ℓT ′ . We also use (e+e′)/2 for a third classification, with loss ℓmix. The overall training loss is ℓT + ℓT ′ + ℓmix, where ℓT and ℓmix are expected to reduce overfitting.

heading: 4 Experiments

heading: 4.1 Datasets and evaluation metric
text: We experiment with six short multi-class text classification tasks covering four Indian languages and a Hindi-English code-mixed dataset. We show the details of the datasets in Tables 2 and 6. We use mBERT as theMLLM and report macro-F1 (we report the accuracymetric in Appendix B). Details of model hyperparameters are present in Appendix C.

heading: 4.2 Quantitative results
text: In Figure 1, we plot macro-F1 against the extent of vocabulary augmentation. Green, orange, and blue lines show the performance with InitLRL, InitHRL, and InitMix initialization, respectively. Corresponding colored bands show 1-standard deviation spreads. Vnew helps: For all tasks, including Vnew is better than baseline MLLM, and the gap is usually significant. This shows that even minimal training of newly added LRL tokens that used to be UNK or over-fragmented helps improve performance. More augmentation̸⇒larger lift: We expected that larger Vnew would monotonically improve performance, but this was not universally the case. Inclusion of non-informative words, as we grow Vnew (∆H decreases with high variance as shown in Appendix B Figure 3), maybe a reason.
Initialization does not matter much: Although there are cases where InitHRL or InitMix performs better than InitLRL, we did not find significant performance difference between different embedding initialization of new LRL words. Transfer of embeddings from a well-represented HRL is the likely reason. We also check the performance by randomly initializing the Vnew words and find, for almost all the cases, random initialization performance, both for macro-F1(in Figure 1) and accuracy(in Appendix B Figure 2), is lesser compared to InitHRL, InitLRL, or InitMix. It suggests meaningful initialization helps.
Comparison with recent approaches: We compare EVALM with AVocaDo (Hong et al., 2021)
keeping Vnew comparable in size. Table 4 shows that AVocaDo leads to performance degradation for all LRL datasets. The lack of domainspecificity for our datasets may be why AVocaDo’s performance dropped. We also compare with FLOTA (Hofmann et al., 2022) in Figure 1. For all datasets except GLUECoS Hi-En codemix dataset, EVALM performs better than FLOTA. A possible explanation is that mBERT vocabulary already includes many English as well as Hindi words, which helps FLOTA better compose embeddings of morphological components of English and Hindi words compared to other Indian languages.
Regularization helps: Table 5 shows that EVALM with AVocaDo-style regularization performs better than without it, for all datasets. Cases where EVALMhurts: The samples in Table 3 show that EVALM generally helps by spotting words important for predicting the correct class. This is shown in the first two examples, where the added vocabulary (∆H=100%) tipped the prediction toward the gold label. But the last two examples show cases where for a word, the train and test set frequency distribution among target classes are different. As a consequence, these words may become misleading at test time.

heading: 5 Conclusion
text: We have proposed a simple and effective method to augment an MLLM wordpiece vocabulary with LRL words that are important for LRL classification tasks. Our study, involving several Indian languages, shows a consistent positive impact of vocabulary augmentation and fine-tuning. We find more augmentation does not guarantee performance improvement, and different embedding initialization fails to show significant performance differences among themselves. We also show that regularization is crucial to prevent overfitting new LRL word embeddings during fine-tuning. We have limited the augmentation to whole LRL words, and a judicious selection of LRL wordpieces may improve performance. We also want to extend to other target tasks (especially language generation) and a more diverse set of LRLs.

heading: 6 Limitations
text: While EVALM demonstrates that vocabulary augmentation with LRL task performance as objective requires different priorities from vocabulary augmentation for improving representation for its
own sake, our work opens up several avenues for exploration. Our understanding of the potential conflict between fidelity of LRL word representation from wordpieces and LRL task class discrimination requirements remains far from complete, particularly when we extend from sequence-tosingle-label applications to sequence labeling (as in POS and NER tagging) and further to sequenceto-sequence applications (such as translation). Perhaps, further experiments with mBERT and other MLLMs will further our understanding of these trade-offs. While initializing an LRL word embedding using InitHRL or InitMix, we depend on automatic machine translation, which can be errorprone. Ranking by ∆H and picking a prefix fails to discount informative but correlated features. A more sophisticated formulation of loss of information owing to fragmentation, taking multiple LRL words into account simultaneously, may alleviate this problem. In the short term, these two limitations may deserve closer scrutiny.

heading: A Discussion on vulnerable LRL words
text: We discuss some natural ideas to determine LRL words vulnerable to improper wordpiece segmentation. UNK and fragment counts: A natural impulse may be to augment the vocabulary with all LRL words (in the task’s train fold) that cannot be assembled from wordpieces in the original vocabulary (i.e., those that become UNK tokens). This is neither necessary nor sufficient. Many UNK words may offer little signal toward labelling. As an example, suppose the Bengali word 'েদায়া' (translate to ‘prayer‘ in English) split to a single [‘[UNK]’] token after passing through mBERT tokenizer can be helpful for sentiment analysis classification task. But the word 'েজলায়' (translate to ‘in the district‘ in English) also split to a single [’[UNK]’] token but might not carry any particular signal for the sentiment classification. On the other hand, simply adding all LRL characters as ‘wordpieces’ precludes UNKs entirely but by no means assures us that the LRL words thus assembled will obtain contextual embeddings of good quality. The word 'ভালবাসা' (translate to ‘love‘ in English) splits to ['ভ', '##◌াল', '##বা', '##সা'], where all these word fragments do not carry any semantic meaning in Bengali. Contextual embedding distortion: Another natural idea is to ask if embeddings of wordpieces assembled into the LRL word can be combined by the (typically transformer-like) MLLM network into a good-quality embedding for the LRL word. This can be ascertained only if we have access to a reference embedding for the LRL word, which can be obtained only after introducing the LRL word into the vocabulary and re-training the MLLM! Another problem with this approach is that it is not guided by the impact of the distortion of embedding of a LRL word on end-task accuracy.

heading: B Supplementary results
text: In Figure 2, we report the accuracy with vocab augmentation under different embedding initialization techniques for all the datasets.

heading: C Experimental Settings
text: In all experiments, we trained the models on a single NVIDIA RTX A6000 with 48GB of memory. We implemented all models with PyTorch using the Transformers library from Huggingface. Our model has ∼29M trainable parameters, and it takes 10-45 minutes to train, depending on the size of the datasets.
C.1 Hyperparameters We search for the best hyperparameters manually based on the macro F1 scores. These parameter values are listed in Table 8.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6 Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and section 1 introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4 experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix C Experimental Settings
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix C Experimental Settings
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 experiments
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: More clever sharing of fragments is left for future work. ) Suppose LRL word w is not in the MLLM vocabulary; w is fragmented into wordpiece sequence T (w) = s1,. , sT by the MLLM tokenizer T. The LRL task has C class labels. A specific label is denoted c ∈ [C] = {1,. , C}. The counts of w and constituent wordpieces st in each class c are denoted n(w, c) and n(st, c). Based on these counts, we define the following multinomial distributions: p(c|•) = n(•, c)/∑c′ n(•, c′) (1) where • = w, st, etc. Based on this we define the entropy H(•) = −∑c p(c|•) log p(c|•) (2) Suppose H(w) is small. This means w is potentially a good feature for the LRL task. Now suppose a wordpiece st has large H(st). That means st is being shared across other words that are dis-
tributed more evenly across classes. If this is the case for most st, then fragmentation of w may be a serious problem. To combine information from all wordpieces, we average their entropies, and use the relative increase in entropy, going from LRL word to wordpieces, as one signal for the danger of fragmenting w. As an example, suppose the word 'धरम' (religion) occurs ten times in a threeclass sentiment analysis dataset with the class distribution of ‘positive‘, ‘neutral‘, and ‘negative‘ as (1,1,8). Its wordpieces have class distributions 'ध' (100,85,80), '##र' (130,235,250), and '##म' (130,90,125). Then as per equation 2,H('धरम') = 0. 639, H('ध') = 1. 094, H('##र') = 1. 062, and H('##र') = 1. 086. The average wordpiece entropy is HS('धरम') = 1. 094+1. 062+1. 0863 = 1. 081, and the percentage of entropy reduction from average wordpiece to word entropy is about 41%. We also retain two simpler signals: the number of fragments |T (w)|, and the frequency of w in the LRL task corpus. LRL words are sorted on the amount of entropy decrease and the top LRL words proposed for vocabulary augmentation. We remove words with very low frequency and retain a prefix of specified size to obtain Vnew, the LRL words to be added to the MLLM vocabulary. Algorithm 1 shows a high-level pseudocode