SlowGuess/ACL_2020_OCR huggingface dataset:
5.1 Language Identification
Approach 1. We establish three strong baselines using a vanilla ELMo (Exp 1.1), ELMo combined
with BLSTM and CRF (Exp 1.2) as suggested by Peters et al. (2018), and a multilingual BERT (Exp 1.3) provided by Devlin et al. (2019). We experiment with frozen weights for the core parameters of ELMo and BERT, but we find the best results when the full models are fine-tuned, which we report in Table 2.
Approach 2. In the second set of experiments, we add the components of our mechanism upon ELMo combined with BLSTM and CRF (Exp 1.2). We start by replacing the max-pooling operation with the attention layer at every individual n-gram order in Exp 2.1. In Exp 2.2, we incorporate the position information. The third experiment, Exp 2.3, adds the hierarchical attention across all n-gram order vectors. It is worth noting that we experiment by accumulating consecutive n-gram orders, and we find that the performance stops increasing when  $n > 3$ . Intuitively, this can be caused by the small size of the datasets since n-gram features of greater order are infrequent and would require more data to be trained properly. We apply our mechanism for n-gram orders in the set  $\{1, 2, 3\}$ , which we report in Table 2.
Approach 3. For the third set of experiments, we focus on emphasizing the morphological clues extracted by our mechanism (Exp 2.3). First, in Exp 3.1, we concatenate the enhanced character n-grams with their corresponding word representation before feeding the input to the CRF layer. In
8038
POS System	Dev F1	Test F1
ML-BERT	86.84	84.70
ELMo + BLSTM + CRF	87.42	88.12
Prev. SOTA (Singh et al., 2018)	-	90.20
Architecture: CS-ELMo + BLSTM + CRF
Exp 4.1: No CS knowledge	87.02	87.96
Exp 4.2: CS knowledge frozen	89.55	89.92
Exp 4.3: CS knowledge trainable	90.37	91.03
Exp 3.2, we add the secondary task over the previous experiment to force the model to predict the simplified LID labels by only using the morphological clues (i.e., no context is provided). Finally, in Exp 3.3, we add static word embeddings that help the model to handle social media style and domain-specific words.
We achieve the best results on Exp 3.3, which outperforms both the baselines and the previous state of the art on the full LID label scheme (see Table 2). However, to compare with other work, we also calculate the average of the weighted F1 scores over the labels lang1 and lang2. Table 3 shows a comparison of our results and the previous state of the art. Note that, for Spanish-English and Hindi-English, the gap of improvement is reasonable, considering that similar gaps in the validation experiments are statistically significant. In contrast, in the case of Nepali-English, we cannot determine whether our improvement is marginal or substantial since the authors only provide one decimal in their scores. Nevertheless, Al-Badrashiny and Diab (2016) use a CRF with hand-crafted features (Al-Badrashiny and Diab, 2016), while our approach does not require any feature engineering.
5.2 POS Tagging and NER
We use LID to adapt the English pre-trained knowledge of ELMo to the code-switching setting, effectively generating CS-ELMo. Once this is achieved, we fine-tune the model on downstream NLP tasks such as POS tagging and NER. In this section, our goal is to validate whether the CS-ELMo model can improve over vanilla ELMo, multilingual BERT, and the previous state of the art for both tasks. More specifically, we use our best architecture (Exp 3.3) from the LID experiments 1) without the code-switching adaptation, 2) with the code-switching
Table 4: The F1 scores on POS tagging for the Hindi-English dataset. CS knowledge means that the CS-ELMo architecture (see Figure 2A) has been adapted to code-switching by using the LID task.
NER System	Dev F1	Test F1
ML-BERT	61.11	64.56
ELMo + BLSTM + CRF	59.91	63.53
Best at CALCS (Trivedi et al., 2018)	-	63.76
Prev. SOTA (Winata et al., 2019)	-	66.63
Architecture: CS-ELMo + BLSTM + CRF
Exp 5.1: No CS knowledge	62.59	66.30
Exp 5.2: CS knowledge frozen	64.39	67.96
Exp 5.3: CS knowledge trainable	64.28	66.84
Table 5: The F1 scores on the Spanish-English NER dataset. CS knowledge means that the CS-ELMo architecture (see Figure 2A) has been adapted to code-switching by using the LID task.
adaptation and only retraining the inference layer, and 3) with the code-switching adaptation and retraining the entire model.
POS tagging experiments. Table 4 shows our experiments on POS tagging using the Hindi-English dataset. When we compare our CS-ELMO + BLSTM + CRF model without CS adaptation (Exp 4.1) against the baseline (ELMo + BLSTM + CRF), the performance remains similar. This suggests that our enhanced n-gram mechanism can be added to ELMo without impacting the performance even if the model has not been adapted to CS. Slightly better performance is achieved when the CS-ELMo has been adapted to code-switching, and only the BLSTM and CRF layers are retrained (Exp 4.2). This result shows the convenience of our model since small improvements can be achieved faster by leveraging the already-learned CS knowledge while avoiding to retrain the entire model. Nevertheless, the best performance is achieved by the adapted CS-ELMO + BLSTM + CRF when retraining the entire model (Exp 4.3). Our results are better than the baselines and the previous state of the art.
Interestingly, our model improves over multilingual BERT, which is a powerful and significantly bigger model in terms of parameters. Our intuition is that this is partly due to the word-piece tokenization process combined with the transliteration of Hindi. The fact that we use the multilingual version of BERT does not necessarily help to handle transliterated Hindi, since Hindi is only present in BERT's vocabulary with the Devanagari script. Indeed, we notice that in some tweets, the original number of tokens was almost doubled by the greedy tokenization process in BERT. This behavior tends to degrade the syntactic and semantic
8039
Figure 3: Visualization of the tri-gram attention weights for the 2016 Spanish-English LID dataset. The boxes contain the tri-grams of the word below them along with the right  $(\checkmark)$  or wrong  $(X)$  predictions by the model.
information captured in the original sequence of tokens. In contrast, ELMo generates contextualized word representations out of character sequences, which makes the model more suitable to adapt to the transliteration of Hindi.
NER experiments. Table 5 contains our experiments on NER using the 2018 CALCS Spanish-English dataset. Exp 5.1 shows that the enhanced n-gram mechanism can bring improvements over the ELMo + BLSTM + CRF baseline, even though the CS-ELMo has not been adapted to the code-switching setting. However, better results are achieved when the CS-ELMo model incorporates the code-switching knowledge in both Exp 5.2 and 5.3. Unlike the POS experiments 4.2 and 4.3, fixing the parameters of CS-ELMo model yields better results than updating them during training. Our intuition is that, in the NER task, the model needs the context of both languages to recognize entities within the sentences, and having the code-switching knowledge fixed becomes beneficial. Also, by freezing the CS-ELMo model, we can accelerate training because there is no backpropagation for the CS-ELMo parameters, which makes our code-switching adaptation very practical for downstream tasks.
Position embeddings. Localizing n-grams within a word is an important contribution of our method. We explore this mechanism by using our fine-tuned CS-ELMo to predict the simplified LID labels on the validation set from the secondary task (i.e., the predictions solely rely on morphology) in two scenarios. The first one uses the position embeddings corresponding to the actual place of the character n-gram, whereas the second one chooses position embeddings randomly. We notice a consistent decay in performance across the language pairs, and a
variation in the confidence of the predicted classes. The most affected language pair is Spanish-English, with an average difference of 0.18 based on the class probability gaps between both scenarios. In contrast, the probability gaps in Hindi-English and Nepali-English are substantially smaller; their average differences are 0.11 and 0.09, respectively.
Position distribution. Considering the previous analysis and the variations in the results, we gather insights of the attention distribution according to their n-gram positions (see position-aware attention in Section 3.1). Although the distribution of the attention weights across n-gram orders mostly remain similar along the positions for all language pairs, Spanish-English has a distinctive concentration of attention at the beginning and end of the words. This behavior can be caused by the differences and similarities between the language pairs. For Spanish-English, the model may rely on inflections of similar words between the languages, such as affixes. On the other hand, transliterated Hindi and Nepali tend to have much less overlap with English words (i.e., words with few characters can overlap with English words), making the distinction more spread across affixes and lemmas.
Attention analysis. Figure 3 shows the tri-gram attention weights in the Spanish-English LID dataset. The model is able to pick up affixes that belong to one or the other language. For instance, the tri-gram -ing is commonly found in English at the end of verbs in present progressive, like in the word coming from the figure, but it also appears in Spanish at different places (e.g., ingeniero) making the position information relevant. On the contrary, the tri-grams aha and hah from the figure do not seem to rely on position information because the attention distribution varies along the words. See more examples in Appendix E.
8040
Error analysis. Morphology is very useful for LID, but it is not enough when words have similar spellings between the languages. We inspect the predictions of the model, and find cases where, for example, miserable is gold-labeled as ambiguous but the model predicts a language (see the top-right tweet in Figure 3). Although we find similar cases for Nepali-English and Hindi-English, it mostly happens for words with few characters (e.g., me, to, use). The model often gets such cases mistrailed due to the common spellings in both languages. Although this should be handled by context, our contribution relies more on morphology than contextualization, which we leave for future work.
We present a transfer learning method from English to code-switched languages using the LID task. Our method enables large pre-trained models, such as ELMo, to be adapted to code-switching settings while taking advantage of the pre-trained knowledge. We establish new state of the art on LID for Nepali-English, Spanish-English, and Hindi-English. Additionally, we show the effectiveness of our CS-ELMo model by further fine-tuning it for NER and POS tagging. We outperform multilingual BERT and homologous ELMo models on Spanish-English NER and Hindi-English POS tagging. In our ongoing research, we are investigating the expansion of this technique to language pairs where English may not be involved.
This work was supported by the National Science Foundation (NSF) on the grant #1910192. We thank Deepthi Mave for providing general statistics of the code-switching datasets and Mona Diab for insightful discussions on the topic.
Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2018. Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 138-147, Melbourne, Australia. Association for Computational Linguistics.
Mohamed Al-Badrashiny and Mona Diab. 2016. LILI: A Simple Language Independent Approach for Language Identification. In Proceedings of COLING
2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1211-1219, Osaka, Japan. The COLING 2016 Organizing Committee.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Kelsey Ball and Dan Garrette. 2018. Part-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language Identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3084-3089, Brussels, Belgium. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135-146.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392. Association for Computational Linguistics.
Björn Gambäck and Amitava Das. 2014. On Measuring the Complexity of Code-Mixing. In Proceedings of the 11th International Conference on Natural Language Processing, Goa, India, pages 1-7.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. CoRR, abs/1705.03122.
Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Naman Jain and Riyadh Ahmad Bhat. 2014. Language Identification in Code-Switching Scenario. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 87-93, Doha, Qatar. Association for Computational Linguistics.
8041
Deepthi Mave, Suraj Maharjan, and Thamar Solorio. 2018. Language Identification and Analysis of Code-Switched Social Media Text. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 51-61, Melbourne, Australia. Association for Computational Linguistics.
Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the Second Shared Task on Language Identification in Code-Switched Data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40-49, Austin, Texas. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A Universal Part-of-Speech Tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2089-2096, Istanbul, Turkey. European Language Resources Association (ELRA).
Holger Schwenk. 2018. Filtering and Mining Parallel Data in a Joint Multilingual Space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228-234, Melbourne, Australia. Association for Computational Linguistics.
Holger Schwenk and Matthijs Douze. 2017. Learning Joint Multilingual Sentence Representations with Neural Machine Translation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 157-167, Vancouver, Canada. Association for Computational Linguistics.
Holger Schwenk and Xian Li. 2018. A Corpus for Multilingual Document Classification in Eight Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Kushagra Singh, Indira Sen, and Ponnurangam Kumaraguru. 2018. A Twitter Corpus for Hindi-
English Code Mixed POS Tagging. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, pages 12-17, Melbourne, Australia. Association for Computational Linguistics.
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W. Black. 2019. A Survey of Code-switched Speech and Language Processing. CoRR, abs/1904.00784.
Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the First Shared Task on Language Identification in Code-Switched Data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62-72, Doha, Qatar. Association for Computational Linguistics.
Victor Soto and Julia Hirschberg. 2018. Joint Part-of-Speech and Language ID Tagging for Code-Switched Data. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 1-10, Melbourne, Australia. Association for Computational Linguistics.
Shashwat Trivedi, Harsh Rangwani, and Anil Kumar Singh. 2018. IIT (BHU) Submission for the ACL Shared Task on Named Entity Recognition on Code-switched Data. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 148-153, Melbourne, Australia. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Changhan Wang, Kyunghyun Cho, and Douwe Kiela. 2018. Code-Switched Named Entity Recognition with Embedding Attention. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 154-158, Melbourne, Australia. Association for Computational Linguistics.
Genta Indra Winata, Zhaojiang Lin, and Pascale Fung. 2019. Learning Multilingual Meta-Embeddings for Code-Switching Named Entity Recognition. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 181-186, Florence, Italy. Association for Computational Linguistics.
Genta Indra Winata, Chien-Sheng Wu, Andrea Madotto, and Pascale Fung. 2018. Bilingual Character Representation for Efficiently Addressing Out
of-Vocabulary Words in Code-Switching Named Entity Recognition. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 110-114, Melbourne, Australia. Association for Computational Linguistics.
Zeynep Yirmibesoglu and Gulsen Eryigit. 2018. Detecting Code-Switching between Turkish-English Language Pair. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 110-115, Brussels, Belgium. Association for Computational Linguistics.
Appendix for “From English to Code-Switching: Transfer Learning with Strong Morphological Clues”
A Language Identification Distributions
Table 6 shows the distribution of the language identification labels across the CALCS datasets.
Labels	Nep-Eng	Spa-Eng	Hin-Eng
lang1	71,148	112,579	84,752
lang2	64,534	119,408	29,958
other	45,286	55,768	21,725
ne	5,053	5,693	9,657
ambiguous	126	404	13
mixed	177	54	58
fw	0	30	542
unk	0	325	17
We notice that the CALCS datasets have monolingual tweets, which we detail at the utterance-level in Table 7. We use the information in this table to measure the rate of code-switching by using the Code-Mixed Index (CMI) (Gambäck and Das, 2014). The higher the score of the CMI, the more code-switched the text is. We show the CMI scores in Table 8.
Table 6: Label distribution for LID datasets.
Labels	Nep-Eng	Spa-Eng	Hin-Eng
code-switched	9,868	8,733	3,237
lang1	1,374	8,427	3,842
lang2	1,614	7,273	298
other	11	697	44
B Parts-of-Speech Label Distribution
Table 9 shows the distribution of the POS tags for Hindi-English. This dataset correspond to the POS tagging experiments in Section 5.2.
Table 7: Utterance level language distribution for language identification datasets.
Corpus	CMI-all	CMI-mixed
Nepali-English 2014	19.708	25.697
Spanish-English 2016	7.685	22.114
Hindi-English 2018	10.094	23.141
Table 8: Code-Mixing Index (CMI) for the language identification datasets. CMI-all: average over all utterances in the corpus. CMI-mixed: average over only code-switched instances.
POS Labels	Train	Dev	Test
X	5296	790	1495
VERB	4035	669	1280
NOUN	3511	516	1016
ADP	2037	346	599
PROPN	1996	271	470
ADJ	1070	170	308
PART	1045	145	23
PRON	1013	159	284
DET	799	116	226
ADV	717	100	204
CONJ	571	77	161
PART_NEG	333	43	92
PRON_WH	294	39	88
NUM	276	35	80
C Named Entity Recognition Label Distribution
Table 10 shows the distribution of the NER labels for Spanish-English. This dataset corresponds to the NER experiments in Section 5.2.
Table 9: The POS tag distribution for Hindi-English.
NER Classes	Train	Dev	Test
person	6,226	95	1,888
location	4,323	16	803
organization	1,381	10	307
group	1,024	5	153
title	1,980	50	542
product	1,885	21	481
event	557	6	99
time	786	9	197
other	382	7	62
NE Tokens	18,544	219	4,532
O Tokens	614,013	9,364	178,479
Tweets	50,757	832	15,634
Table 10: The distribution of labels for the Spanish-English NER dataset from CALCS 2018.
D Hyperparameters and Fine-tuning
We experiment with our LID models using Adam optimizer with a learning rate of 0.001 and a plateau learning rate scheduler with patience of
Figure 4: Visualization of the attention weights at the tri-gram level for the Hindi-English 2018 dataset on the LID task. The boxes contain the tri-grams of the word below them. We also provide the predicted label by the model, and whether it was correct or wrong.
5 epochs based on the validation loss. We train our LID models using this setting for 50 epochs. For the last block of experiments in Table 2, we use a progressive fine-tuning process described below.
Fine-tuning. We fine-tune the model by progressively updating the parameters from the top to the bottom layers of the model. This avoids losing the pre-trained knowledge from ELMo and smoothly adapts the network to the new languages from the code-switched data. We use the slanted triangular learning rate scheduler with both gradual unfreezing and discriminative fine-tuning over the layers (i.e., different learning rates across layers) proposed by Howard and Ruder (2018). We group the non-ELMo parameters of our model apart from the ELMo parameters. We set the non-ELMo parameters to be the first group of parameters to be tuned (i.e., parameters from enhanced character ngrams, CRF, and BLSTM). Then, we further group the ELMo parameters as follows (top to bottom):
1. the second bidirectional LSTM layer,
2. the first bidirectional LSTM layer,
3. the highway network,
4. the linear projection from flattened convolutions to the token embedding space,
5. all the convolutional layers, and 6) the character embedding weights.