SlowGuess/ACL_2020_OCR huggingface dataset:
Simran Khanuja $^{1}$  Sandipan Dandapat $^{2}$  Anirudh Srinivasan $^{1}$  Sunayana Sitaram $^{1}$  Monjit Choudhury $^{1}$
<sup>1</sup> Microsoft Research, Bangalore, India
2 Microsoft R&D, Hyderabad, India
{t-sikha, sadandap, t-ansrin, sunayana.sitaram, monojitic} @microsoft.com
Code-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on cross-lingual and multilingual tasks. We present an evaluation benchmark, GLUECoS, for code-switched languages, that spans several NLP tasks in English-Hindi and English-Spanish. Specifically, our evaluation benchmark includes Language Identification from text, POS tagging, Named Entity Recognition, Sentiment Analysis, Question Answering and a new task for code-switching, Natural Language Inference. We present results on all these tasks using cross-lingual word embedding models and multilingual models. In addition, we fine-tune multilingual models on artificially generated code-switched data. Although multilingual models perform significantly better than cross-lingual models, our results show that in most tasks, across both language pairs, multilingual models fine-tuned on code-switched data perform best, showing that multilingual models can be further optimized for code-switching tasks.
Code-switching, or code-mixing, is the use of more than one language in the same utterance or conversation and is prevalent in multilingual societies all over the world. It is a spoken phenomenon and is found most often in informal chat and social media on the Internet. Processing, understanding, and generating code-mixed text and speech has become an important area of research.
Recently, contextual word embedding models trained on a large amount of text data have shown state-of-the-art results in a variety of NLP tasks. Models such as BERT (Devlin et al., 2018) and its multilingual version, mBERT, rely on large
amounts of unlabeled monolingual text data to build monolingual and multilingual models that can be used for downstream tasks involving limited labelled data. (Wang et al., 2018) propose a Generalized Language Evaluation Benchmark (GLUE) to evaluate embedding models on a wide variety of language understanding tasks. This benchmark has spurred research in monolingual transfer learning settings.
Data and annotated resources are scarce for code-switched languages, even if one or both languages being mixed are high resource. Due to this, there is a lack of standardized datasets in code-switched languages other than those used in shared tasks in a few language pairs. Although models using synthetic code-switched data and cross-lingual embedding techniques have been proposed for code-switching (Pratapa et al., 2018a), there has not been a comprehensive evaluation of embedding models across different types of tasks. Furthermore, there have been claims that multilingual models such as mBERT are competent in zero-shot cross lingual transfer and code-switched settings. Though comprehensively validated by (Pires et al., 2019) in the case of zero-shot transfer, the probing in code-switched settings was limited to one dataset of one task, namely POS Tagging.
To address all these issues and inspired by the GLUE (Wang et al., 2018) benchmark, we propose GLUECoS, a language understanding evaluation framework for Code-Switched NLP. We include five tasks from previously conducted evaluations and shared tasks, and propose a sixth, Natural Language Inference task for code-switching, using a new dataset<sup>1</sup>(Khanuja et al., 2020). We include tasks varying in complexity ranging from word-level tasks [Language Identification (LID); Named Entity Recognition (NER)], syntactic tasks [POS
we use a subset of the original corpus as available to us at the time of experimentation
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575-3585 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tagging], semantic tasks [Sentiment Analysis; Question Answering] and finally a Natural Language Inference task. Where available, we include multiple datasets for each task in English-Spanish and English-Hindi. We choose these language pairs, not only due to the relative abundance of publicly available datasets, but also because they represent variations in types of code-switching, language families, and scripts between the languages being mixed. We test various cross-lingual and multilingual models on all of these tasks. In addition, we also test models trained with synthetic code-switched data. Lastly, we fine-tune the best performing multilingual model with synthetic code-switched data and show that in most cases, its performance exceeds the multilingual model, highlighting that multilingual models can be further optimized for code-switched settings.
The main contributions of our work are as follows:
- We point out the lack of standardized datasets for code-switching and propose an evaluation benchmark GLUECoS, which can be used to test models on various NLP tasks in English-Hindi and English-Spanish.
- In creating the benchmark, we highlight the tasks that are missing from code-switched NLP and propose a new task, Natural Language Inference, for code-switched data.
- We evaluate cross-lingual and pre-trained multilingual embeddings on all these tasks, and observe that pre-trained multilingual embeddings significantly outperform cross-lingual embeddings. This highlights the competence of generalized language models over cross lingual word embeddings.
- We fine-tune pre-trained multilingual models on linguistically motivated synthetic code-switched data, and observe that they perform better in most cases, highlighting that these models can be further optimized for code-switched settings.
The rest of the paper is organized as follows. We relate our work to prior work to situate our contributions. We introduce the tasks and datasets used for GLUECoS motivating the choices we make. We describe the experimental setup, with details of the models used for baseline evaluations. We present the results of testing all the models on the
benchmark and analyze the results. We conclude with a direction for future work and highlight our main findings.
2 Relation to prior work
The idea of a generalized benchmark for code-switching is inspired by GLUE (Wang et al., 2018), which has spurred research in Natural Language Understanding in English, to an extent that a set of harder tasks have been curated in a follow-up benchmark, SuperGLUE (Wang et al., 2019) once models beat the human baseline for GLUE. The motivation behind GLUE is to evaluate models in a multi-task learning framework across several tasks, so that tasks with less training data can benefit from others. Although our current work does not include models evaluated in a multi-task setting, we plan to implement this in subsequent versions of the benchmark.
There have been shared tasks conducted in the past as part of code-switching workshops co-located with notable NLP conferences. The first and second workshops on Computational Approaches to Code Switching (Diab et al., 2014, 2016) conducted a shared task on Language Identification for several language pairs (Solorio et al., 2014; Molina et al., 2016). The third workshop (Aguilar et al., 2018) included a shared task on Named Entity Recognition for the English-Spanish and Modern Standard Arabic-Egyptian Arabic language pairs(Aguilar et al., 2019).
The Forum for Information Retrieval Evaluation (FIRE) aims to meet new challenges in multilingual information access and has conducted several shared tasks on code-switching. These include tasks on transliterated search, (Roy et al., 2013; Choudhury et al., 2014) code-mixed entity extraction (Rao and Devi, 2016) and mixed script information retrieval (Sequiera et al., 2015; Banerjee et al., 2016). Other notable shared tasks include the Tool Contest on POS Tagging for Code-Mixed Indian Social Media at ICON 2016 (Jamatia et al., 2016), Sentiment Analysis for Indian Languages (Code-Mixed) at ICON 2017 (Patra et al., 2018) and the Code-Mixed Question Answering Challenge (Chandu et al., 2018a).
Each of the shared tasks mentioned above attracted several participants and have led to follow up research in these problems. However, all tasks have focused on a single NLP problem and so far, there has not been an evaluation of models across
several code-switched NLP tasks. Our objective with proposing GLUECoS is to address this gap, and determine which models best generalize across different tasks, languages and datasets.
3 Tasks and Datasets
Some NLP tasks are inherently more complex than others - for example, a Question Answering task that needs to understand both the meaning of the question and answer, is harder to solve by a machine than a word-level Language Identification task, in which a dictionary lookup can give reasonable results. Some datasets and domains may contain very little code-switching, while others may contain more frequent and complex code-switching. Similar languages, when code-switched, may maintain the word order of both languages, while other language pairs that are very different may take on the word order of one of the languages. With these in mind, our choice of tasks and datasets for GLUE-CoS are based on the following principles:
- We choose a variety of tasks, ranging from simpler ones, on which the research community has already achieved high accuracies, to relatively more complex, on which very few attempts have been made.
- We desire to evaluate models on language-pairs from different language families, and on a varied number of tasks, to enable detailed analysis and comparison. This led us to choose English-Hindi and English-Spanish, as we found researched upon datasets for almost all tasks in our benchmark for these language pairs.
- English and Spanish are written in the Roman script, while English-Hindi datasets can contain Hindi words written either in the original Devanagari script, or in the Roman script, thus adding script variance as an additional parameter to analyse upon.
- We include multiple datasets from each language pair where available, so that results can be compared across datasets for the same task.
Due to the lack of standardized datasets, we choose to create our own train-test-validation splits for some tasks. Also, we use an off-the-shelf transliterator and language detector, where necessary, details of which can be found in Appendix A. Table
1 shows all the datasets that we use, with their statistics, while Table 2 shows the code-switching statistics of the data in terms of standardized metrics for code-switching (Gambäck and Das, 2014; Guzmán et al., 2017). Briefly, the code-mixing metrics include :
Code-Mixing Index (CMI): The fraction of language dependent tokens not belonging to the matrix language in the utterance.
- Average switch-points (SP Avg): The average number of intra-sentential language switchpoints in the corpus.
- Multilingual Index (M-index): A word-count-based measure quantifying the inequality of distribution of language tags in a corpus of at least two languages.
- Probability of Switching (I-index): The proportion of the number of switchpoints in the corpus, relative to the number of language-dependent tokens.
- *Burstiness*: The quantification of whether switching occurs in bursts (randomly similar to a Poisson process), or has a more periodic character.
- Language Entropy (LE): The bits of information needed to describe the distribution of language tags.
- Span Entropy (SE): The bits of information needed to describe the distribution of language spans.
In cases where the datasets have been a part of shared tasks, we report the highest scores obtained in each task as the State Of The Art (SOTA) for the dataset. However, note that we report this to situate our results in context of the same, and these cannot be directly compared, since each task's SOTA is obtained by varied training architecture, suited to perform well in one particular task alone.
3.1 Language Identification (LID)
Language Identification is the task of obtaining word-level language labels for code-switched sentences. For English-Hindi we choose the FIRE 2013 (FIRE LID) dataset originally created for the transliterated search subtask (Roy et al., 2013). The test and development sets provided contain word-level language tagged sentences. For training we
English-Hindi
Corpus	Sent (Train)	Sent (Dev)	Sent (Test)	Sent (All)
Fire LID (D)	2631	500	406	3537
UD POS (D)	1384	215	215	1814
FG POS (R)	2104	263	264	2631
IIITH NER (R)	2467	308	309	3084
SAIL Sentiment (R)	10080	1260	1261	12601
QA (R)	250	-	63	313
NLI (R)	1040	130	130	1300
English-Spanish
Corpus	Sent (Train)	Sent (Dev)	Sent (Test)	Sent (All)
EMNLP 2014	10259	1140	3014	14413
Bangor POS	2192	274	274	2758
CALCS NER	27366	3420	3421	34208
Sentiment	1681	211	211	2103
Table 1: Corpus Statistics. (R) and (D) indicates Hindi written in Roman and Devanagari script, respectively
English-Hindi
Corpus	CMI	SP Avg	M-index	I-index	Burstiness	LE	SE
Fire LID	78.26	4.47	0.39	0.33	-0.42	0.86	1.02
UD POS	136	4.98	0.46	0.39	-0.25	1.35	1.47
FG POS	68	5.5	0.4	0.34	-0.43	0.87	1.05
IIITH NER	133	11.39	0.64	0.53	-0.26	1.28	1.36
SAIL Sentiment	72.8	5.07	0.02	0.32	-0.32	0.87	1.17
QA	142.28	3.96	0.81	0.5	-0.4	0.89	1.09
NLI	149.95	66.74	0.44	0.63	-0.2	1.53	1.39
English-Spanish
Corpus	CMI	SP Avg	M-index	I-index	Burstiness	LE	SE
EMNLP 2014	33.46	2.86	0.33	0.29	-0.34	0.79	1.1
Bangor POS	123.06	1.67	0.32	0.27	-0.35	0.82	1.06
CALCS NER	94.52	3.17	0.004	0.31	-0.42	0.75	1.02
Sentiment	110.56	4.13	0.15	0.27	-0.21	0.79	1.42
Table 2: Code-switching Statistics
use a POS tagging dataset (Jamatia et al., 2016) which also contains language labels.
For English-Spanish we choose the dataset in (Solorio et al., 2014), provided as part of the LID shared task at EMNLP 2014. We report the highest score obtained for SPA-EN (Solorio et al., 2014) as the SOTA for this task.
3.2 Part of Speech (POS) tagging
POS tagging includes labelling at the word level, grammatical part of speech tags such as noun, verb, adjective, pronoun, prepositions etc. For English-Hindi, we use two datasets. The first is the code-switched Universal Dependency parsing dataset provided by (Bhat et al., 2018) (UD POS). This corpus contains a transliterated version, where Hindi
is in the Roman script, and also a corrected version in which Hindi has been manually converted back to Devanagari. We report the highest score obtained by (Bhat et al., 2018) as the SOTA for this task.
The second English-Hindi dataset we use was part of the ICON 2016 Tool Contest on POS Tagging for Code-Mixed Indian Social Media Text (Jamatia et al., 2016) (FG POS). We report the highest score obtained by (Anupam Jamatia, 2016)- (report communicated directly by authors) as the SOTA for this task.
For English-Spanish, of the two corpora utilised in (AlGhamdi et al., 2016), we choose the Bangor Miami corpus (Bangor POS) owing to the larger size of the corpus. We report the highest score
obtained by (AlGhamdi et al., 2016) as the SOTA for this task.
3.3 Named Entity Recognition (NER)
NER involves recognizing named entities such as person, location, organization etc. in a segment of text. For English-Hindi we use the Twitter NER corpus provided by (Singh et al., 2018) (IIITH NER). We report the highest score obtained by (Singh et al., 2018) as the SOTA for this task.
For English-Spanish, we use the Twitter NER corpus provided as part of the CALCS 2018 shared task on NER for code-switched data (Aguilar et al., 2019) (CALCS NER). We report the highest score obtained by (Winata et al., 2019) as the SOTA for this task.
3.4 Sentiment Analysis
Sentiment analysis is a sentence classification task wherein each sentence is labeled to be expressing a positive, negative or neutral sentiment.
For English-Hindi we choose the sentiment annotated social media corpus used in the ICON 2017 shared task; Sentiment Analysis for Indian Languages (SAIL) (Patra et al., 2018). This corpus is originally language tagged at the word level with Hindi in the Roman script. We report the highest score obtained for HI-EN (Patra et al., 2018) as the SOTA for this task.
For English-Spanish we choose the sentiment annotated Twitter dataset provided by (Vilares et al., 2016) which we split into an 8:1:1 train:test:validation split ensuring sentiment distribution. (Vilares et al., 2016) report an average F1 score of 58.9 on the same dataset, while (Pratapa et al., 2018b) report an F1 of 64.6 on the same, which we report as the SOTA for this dataset. We are not aware of future work done on this dataset.
3.5 Question Answering (QA)
Question Answering is the task of answering a question based on the given context or world knowledge. We choose the dataset provided by (Chandu et al., 2018a) which contains two types of questions for En-Hi, one with context (185 article based questions) and one containing image based questions (774 questions). For the image based questions we use the  $DrQA$ -Document Retriever module<sup>2</sup> to extract the most relevant context from Wikipedia. Since it is a code-switched dataset, context could
not be extracted for all questions. We obtain a final dataset having 313 (question-answer-context) triples.
3.6 Natural Language Inference (NLI)
Natural Language Inference is the task of inferring a positive (entailed) or negative (contradicted) relationship between a premise and hypothesis. While most NLI datasets contain sentences or images as premises, the code-switched NLI dataset we use contains conversations as premises, making it a conversational NLI task (Khanuja et al., 2020). Since this is a new dataset, we report our number as the SOTA for this task.
We use standard architectures for solving each of the tasks mentioned above (Refer to Appendix B). We experiment with several existing cross lingual word embeddings that have been shown to perform well on cross lingual tasks. We also experiment with the Multilingual BERT (mBERT) model released by (Devlin et al., 2018). In a survey on cross lingual word embeddings, (Ruder et al., 2017) establish that various embedding methods optimize for similar objectives given that the supervision data involved in training them is similar. Based on this, we choose the following representative embedding methods that vary in the amount of supervision involved in training them.
4.1 MUSE Embeddings
We use the MUSE library<sup>3</sup> to train both supervised and unsupervised word embeddings. The unsupervised word embeddings are learnt without any parallel data or anchor point. It learns a mapping from the source to the target space using adversarial training and (iterative) Procrustes refinement (Conneau et al., 2017). The supervised method leverages a bilingual dictionary (or identical character strings as anchor points), to learn a mapping from the source to the target space using (iterative) Procrustes alignment.
4.2 BiCVM Embeddings
This method, proposed by (Hermann and Blunsom, 2014), leverages parallel data, based on the assumption that parallel sentences are equivalent in meaning and subsequently have similar sentence
$^{2}$ https://github.com/facebookresearch/DrQA
<sup>3</sup>https://github.com/facebookresearch/MUSE
representations. We use the BiCVM toolkit<sup>4</sup> to learn these embeddings. The parallel corpus we use for English-Spanish consists of 4.5M parallel sentences from Twitter. For English-Hindi, we make use of an internal parallel corpus consisting of roughly 5M parallel sentences.
4.3 BiSkip Embeddings
This method makes use of parallel corpora as well as word alignments to learn cross-lingual embeddings. (Luong et al., 2015) adapt the skip-gram objective originally proposed by (Mikolov et al., 2013) to a bilingual setting wherein a model learns to predict words cross-lingually along with the monolingual objectives. We make use of the fast align toolkit<sup>5</sup> to learn word alignments given parallel corpora and use the BiVec toolkit<sup>6</sup> to learn the final BiSkip embeddings given the parallel corpora and the word alignments. The parallel corpora utilised to learn these are the same as those used to learn the BiCVM embeddings.
4.4 Synthetic Data (GCM) Embeddings
We also experiment with skip-gram embeddings learnt from synthetically generated code-mixed data as proposed by (Pratapa et al., 2018b). We make use of the fasttext library to learn the skipgram embeddings. For English-Spanish, we obtain data from (Pratapa et al., 2018a) which consists of 8M synthetic code-switched sentences. For English-Hindi, we generate synthetic data from the IITB parallel corpus. We sample from the generated sentences obtained using Switch Point Fraction (SPF), as described in (Pratapa et al., 2018a), to obtain a GCM corpus of roughly 10M sentences.
4.5 mBERT
Multilingual BERT is pre-trained on monolingual corpora of 104 languages and has been shown to perform well on zero-shot cross-lingual model transfer and code-switched POS tagging (Pires et al., 2019). Specifically, we use the bert-base-multilingual-cased model for our experiments.
4.6 Modified mBERT
(Sun et al., 2019) show that fine-tuning BERT with in-domain data on language modeling improves
performance on downstream tasks. On similar lines, we fine-tune the mBERT model with synthetically generated code-switched data (gCM) and a small amount of real code-switched data (rCM), on the masked language modeling objective. The training curriculum we use in fine-tuning this model is similar to as proposed by (Pratapa et al., 2018a), which has been shown to improve language modeling perplexity. Although we train on real code-mixed data, it accounts for a small fraction (less than  $5\%$ ) of the total code-mixed data used. Refer to Appendix C for training details.
Tables 3-8 show the results of using the embedding techniques described above for each task and dataset. mBERT provides a large increase in accuracy as compared to cross-lingual techniques, and in most cases, the modified mBERT technique performs best. We do not experiment with baseline or cross-lingual embedding techniques for NLI, since we find that mBERT surpasses the other techniques for all other tasks. For NLI, as in the other cases, we find that modified mBERT performs better than mBERT. We hypothesize that this happens because code-switched languages are not just a union of two monolingual languages. The distributions and usage of words in code-switched languages differ from their monolingual counterparts, and can only be captured with real code-switched data, or synthetically generated data that closely mimics real data.
(Glavas et al., 2019) point out how all cross-lingual word embedding methods optimize for bilingual lexicon induction. Each model is trained using different language pairs and different training and evaluation dictionaries, leading to it overfitting to the task it is optimizing for and failing in other cross-lingual scenarios. Also, the loss function in training cross-lingual word embeddings has a component where  $w1$  in one language predicts the context of its aligned word  $w2$  in the other language. However, in the case of code-switching,  $w1$  appear
The original task was language tagging and transliteration of Hindi words in the Roman script, while we report LID results for Hindi in Devanagari. An accuracy of 99.0 was obtained on the original subtask(Roy et al., 2013)
10We create our own test split from the training data, since the test data is not publicly available
11The original dataset contains multiple code-mixed pairs and there exists no language based segregation of the results. Since we only choose the EN-HI examples we report this as N/A
<sup>4</sup>https://github.com/karlmoritz/bicvm/
$^{5}$ https://github.com/clab/fast_align
$^{6}$ https://github.com/lmthang/bivec
$^{7}$ https://fasttext.cc/
<sup>8</sup>http://www.cfilt.iitb.ac.in/iitb_parallel/
Data	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
FIRE En-Hi	93.21	94.53	94.92	93.98	N/A9
BiCVM	GCM	mBERT	Mod. mBERT
95.24	93.64	95.87	96.6
EMNLP En-Es	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
92.95	92.86	93.39	92.79	94.0
BiCVM	GCM	mBERT	Mod. mBERT
91.47	92.42	95.97	96.24