SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3: LID results (F1)
Data	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
UD En-Hi	77.49	78.06	77.88	77.43	90.53*
BiCVM	GCM	mBERT	Mod. mBERT
77.49	77.84	87.16	88.06
FG En-Hi	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
60.88	60.76	60.59	60.4	80.810
BiCVM	GCM	mBERT	Mod. mBERT
60.2	61.03	63.42	63.31
Bangor En-Es	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
88.78	88.65	88.82	89.2	95.39*
BiCVM	GCM	mBERT	Mod. mBERT
87.46	89.37	93.33	93.62
Table 4: POS results (F1/  ${}^{ * }$  Accuracy)
Data	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
IIITH En-Hi	71.52	71.48	72.15	72.13	78.14
BiCVM	GCM	mBERT	Mod. mBERT
71.55	72.37	74.96	78.21
CALCS En-Es	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
47.9	53.74	54.17	52.98	69.17
BiCVM	GCM	mBERT	Mod. mBERT
51.6	53.57	59.69	61.77
Table 5: NER results (F1)
Data	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
SAIL En-Hi	50.44	48.37	51.27	48.84	56.9
BiCVM	GCM	mBERT	Mod. mBERT
49.56	50.01	58.24	59.35
Sentiment En-Es	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
50.62	58.73	58.44	60.4	64.6
BiCVM	GCM	mBERT	Mod. mBERT
62.62	62.89	66.03	69.31
Table 6: Sentiment Analysis results (F1)
Data	Baseline	Unsup. MUSE	Sup. MUSE	BiSkip	SOTA
QA En-Hi	61.39	56.11	62.78	65.56	N/A11
BiCVM	GCM	mBERT	Mod. mBERT
62.33	62.78	71.96	68.01
Table 7: QA results (F1)
Data	mBERT	Mod. mBERT	SOTA
NLI En-Hi	61.09	63.1	63.1
Table 8: NLI results (Accuracy)
ing in the context of  $w2$  may not be natural. This clearly highlights the need to learn cross-lingual embeddings keeping code-mixed language processing as an optimization objective.
The results using mBERT cannot be directly compared to the cross-lingual models because of the difference in the magnitude of data involved in training. Also, due to the fact that mBERT is trained on 104 languages together, with massive amounts of data for a large number of epochs, it learns several common features better providing for a well represented common embedding space. The training data used for training the cross-lingual embeddings is restricted to Twitter and query logs, while mBERT is trained on the entire wiki dump.
Overall, the cross-lingual and mBERT models perform better for English-Spanish as compared to English-Hindi. This could be due to several reasons.
- English and Spanish are similar languages, with both mostly retaining individual word order while code-switching, which is not the case for English and Hindi.
- Romanized Hindi does not use standardized spellings, and errors made by the translitator could have influenced the results.
- We use Twitter and social media data to train cross lingual word embeddings for English-Spanish which are similar in domain to the task datasets, while we use the IITB and query-based parallel corpora for English-Hindi which is generic in domain, constrained by the available resources at hand.
We find that for most tasks, modified mBERT performs better than mBERT. In cases where this is not true (QA En-Hi; FG En-Hi), the difference in accuracy between the two models is small. This could be attributed to errors made by the transliterator or corpus differences, but in general we observe that the modified En-Hi mBERT model does not significantly outperform the base mBERT model. Given the promising results obtained by modified mBERT, it would be interesting to pre-train a language model for code-switched data which is
trained on the monolingual corpora of languages involved and fine-tuned on GCM as proposed, to compare against fine-tuning mBERT itself, which is trained on multiple languages.
We find that accuracies vary across tasks in the GLUECoS benchmark, and except in the case of LID, code-switched NLP is far from solved. This is particularly stark in the case of Sentiment and NLI, which are three and two way classification tasks respectively. Modified mBERT performs only a little over chance, which shows that we are still in the early days of solving NLI for code-switched languages, and also indicates that our models are far from truly being able to understand code-switched language.
In this paper, we introduce the first evaluation benchmark for code-switching, GLUECoS. The benchmark contains datasets in English-Hindi and English-Spanish for six NLP tasks - LID, POS tagging, NER, Sentiment Analysis, Question Answering and a new code-switched Natural Language Inference task. We test various embedding techniques across all tasks and datasets and find that multilingual BERT outperforms cross-lingual embedding techniques on all tasks. We also find that for most datasets, a modified version of mBERT that has been fine-tuned on synthetically generated code-switched data with a small amount of real code-switched data performs best. This indicates that while multilingual models do go a long way in solving code-switched NLP, they can be improved further by using real and synthetic code-switched data, since the distributions in code-switched languages differ from the two languages being mixed.
In this work, we use standard architectures to solve each NLP task individually and vary the embeddings used. In future work, we would like to experiment with a multi-task setup wherein tasks with less training data can significantly benefit from those having abundant labelled data, since most code-switched datasets are often small and difficult to annotate. We experiment with datasets having varied amounts of code-switching and from different domains and show that some tasks, such as
3582
LID and POS tagging are relatively easier to solve, while tasks such as QA and NLI have low accuracies. We would like to add more diverse tasks and language pairs to the GLUECoS benchmark in a future version.
All the datasets used in the GLUECoS benchmark are publicly available, and we plan to make the NLI dataset available for research use. We hope that this will encourage researchers to test multilingual, cross-lingual and code-switched embedding techniques and models on this benchmark.
Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2019. Named entity recognition on code-switched data: Overview of the calcs 2018 shared task. arXiv preprint arXiv:1906.04138.
Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Thamar Solorio, Mona Diab, and Julia Hirschberg. 2018. Proceedings of the third workshop on computational approaches to linguistic code-switching. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching.
Fahad AlGhamdi, Giovanni Molina, Mona Diab, Thamar Solorio, Abdelati Hawwari, Victor Soto, and Julia Hirschberg. 2016. Part of speech tagging for code switched data. EMNLP 2016, page 98.
Amitava Das Anupam Jamatia. 2016. Task report: Tool contest on pos tagging for code-mixed indian social media (facebook, twitter, and pinterest) text @ icon 2016. Unpublished.
Somnath Banerjee, Kunal Chakma, Sudip Kumar Naskar, Amitava Das, Paolo Rosso, Sivaji Bandyopadhyay, and Monjit Choudhury. 2016. Overview of the mixed script information retrieval (msir) at fire-2016. In Forum for Information Retrieval Evaluation, pages 39-49. Springer.
Irshad Ahmad Bhat, Riyadh Ahmad Bhat, Manish Shrivastava, and Dipti Misra Sharma. 2018. Universal dependency parsing for Hindi-english code-switching. arXiv preprint arXiv:1804.05868.
Khyathi Chandu, Ekaterina Loginova, Vishal Gupta, Josef van Genabith, Günter Neumann, Manoj Chinnakotla, Eric Nyberg, and Alan W Black. 2018a. Code-mixed question answering challenge: Crowdsourcing data and techniques. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 29-38.
Khyathi Chandu, Thomas Manzini, Sumeet Singh, and Alan W. Black. 2018b. Language informed modeling of code-switched text. In Proceedings of the Third Workshop on Computational Approaches
to Linguistic Code-Switching, pages 92-97, Melbourne, Australia. Association for Computational Linguistics.
Monojit Choudhury, Gokul Chittaranjan, Parth Gupta, and Amitava Das. 2014. Overview of fire 2014 track on transliterated search. Proceedings of FIRE, pages 68-89.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Mona Diab, Pascale Fung, Mahmoud Ghoneim, Julia Hirschberg, and Thamar Solorio. 2016. Proceedings of the second workshop on computational approaches to code switching. In Proceedings of the Second Workshop on Computational Approaches to Code Switching.
Mona Diab, Julia Hirschberg, Pascale Fung, and Thamar Solorio. 2014. Proceedings of the first workshop on computational approaches to code switching. In Proceedings of the First Workshop on Computational Approaches to Code Switching.
Björn Gambäck and Amitava Das. 2014. On measuring the complexity of code-mixing. In Proceedings of the 11th International Conference on Natural Language Processing, Goa, India, pages 1-7. CiteSeer.
Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (properly) evaluate crosslingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. arXiv preprint arXiv:1902.00508.
Gualberto A Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for modeling code-switching across corpora. In *INTERSPEECH*, pages 67-71.
Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 58-68, Baltimore, Maryland. Association for Computational Linguistics.
Anupam Jamatia, Björn Gambäck, and Amitava Das. 2016. Collecting and annotating indian social media code-mixed corpora. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 406-417. Springer.
Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, and Monjit Choudhury. 2020. A new dataset for natural language inference from code-mixed conversations.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The iit bombay english-hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151-159.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Giovanni Molina, Nicolas Rey-Villamizar, Thamar Solorio, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, and Mona Diab. 2016. Overview for the second shared task on language identification in code-switched data. EMNLP 2016, page 40.
Braja Gopal Patra, Dipankar Das, and Amitava Das. 2018. Sentiment analysis of code-mixed indian languages: An overview of sail_code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502.
Adithya Pratapa, Gayatri Bhat, Monjit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018a. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1543-1553.
Adithya Pratapa, Monojit Choudhury, and Sunayana Sitaram. 2018b. Word embeddings for code-mixed language processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067-3072.
Pattabhi RK Rao and Sobha Lalitha Devi. 2016. Cmeeil: Code mix entity extraction in indian languages from social media text@ fire 2016-an overview. In FIRE (Working Notes), pages 289-295.
Shruti Rijhwani, Royal Sequiera, Monjit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on twitter with a novel generalized word-level language detection technique. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1971-1982.
Rishiraj Saha Roy, Monjit Choudhury, Prasenjit Majumder, and Komal Agarwal. 2013. Overview of the fire 2013 track on transliterated search. In Post-Proceedings of the 4th and 5th Workshops of the Forum for Information Retrieval Evaluation, page 4. ACM.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2017. A survey of cross-lingual word embedding models. arXiv preprint arXiv:1706.04902.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
Royal Sequiera, Monojit Choudhury, Parth Gupta, Paolo Rosso, Shubham Kumar, Somnath Banerjee, Sudip Kumar Naskar, Sivaji Bandyopadhyay, Gokul Chittaranjan, Amitava Das, et al. 2015. Overview of fire-2015 shared task on mixed script information retrieval. In FIRE Workshops, volume 1587, pages 19-25.
Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. Named entity recognition for Hindi-english code-mixed social media text. In Proceedings of the Seventh Named Entities Workshop, pages 27-35.
Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62-72.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? arXiv preprint arXiv:1905.05583.
David Vilares, Miguel A Alonso, and Carlos Gomez-Rodriguez. 2016. En-es-cs: An english-spanish code-switching twitter corpus for multilingual sentiment analysis. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 4149-4153.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, and Pascale Fung. 2019. Hierarchical meta-embeddings for code-switching named entity recognition. arXiv preprint arXiv:1909.08504.
For each dataset wherein the training, development and test splits are not provided, we create balanced custom splits in an 8:1:1 ratio.
For En-Hi datasets, where the corpus is originally in the Roman script and is not language tagged, we use the LID tool provided by (Rijhwani et al., 2017) to obtain language tags.
In cases where language tags are provided, we convert Roman Hindi words to Devanagari using an off-the-shelf translitator.
We conduct each experiment for 5 random seed values and report the average of the results obtained.
B.1 Word-Level Tasks
For the word level tasks including Language Identification, Named Entity Recognition and Part of Speech tagging we make use of the sequence labeler. This implements a BiLSTM with a CRF layer on the top as described in (Lample et al., 2016). We use the adadelta optimizer with a learning rate of 1.0, dropout of 0.5 and a batch size of 32. We run the model for a maximum of 20 epochs and stop if the validation accuracy on the best_model_selector hyperparameter shows no improvement for 5 epochs continually. The best_model_selector hyperparameter is the F1 score. The dimension of the word embeddings is 300.
We make use of the transformers library $^{13}$  for the mBERT experiments. We use the AdamW optimizer with a learning rate of 5e-5, epsilon of 1e-8, and a batch size of 32, as suggested by (Devlin et al., 2018). We train for 5 epochs.
B.2 Sentence-Level Tasks
For the sentence level tasks (Sentiment Classification) we implement a BiLSTM with one hidden layer of dimension 256. We apply a dropout of 0.5, and use the Adam optimizer with a 0.001 learning rate and 1e-8 epsilon value. We use a batch size of 64 and train for a maximum of 15 epochs stopping if the validation accuracy continually drops for 3 epochs. The dimension of the word embeddings is 300.
We make use of the transformers library for the mBERT experiments. We use the AdamW opti
mizer with a learning rate of 5e-5, epsilon of 1e-8, and a batch size of 32, as suggested by (Devlin et al., 2018). We train for 5 epochs.
B.3 Sentence-Pair Tasks
For the embedding evaluations on the QA task, we make use of the BiDAF architecture $^{14}$  as proposed in (Seo et al., 2016). We keep the default training hyperparameters which include a learning rate of 0.5, a batch size of 1, training epochs as 5, a maximum context length of 400 tokens and a maximum question length of 50 tokens.
We make use of the SQuAD training script and the XNLI training script of the Transformers library with its default hyperparameters for the mBERT experiments.
C BERT LM fine-tuning
We take the bert model released by Google (bert-base-multilingual-cased) and fine-tune it for masked language modeling on 2 types of code-mixed datasets.
We use a curriculum wherein the model is first trained on generated code-mixed data (gCM) for 10 epochs and then on real code-mixed data (rCM) for 10 epochs.
For English-Hindi, the details of the datasets are as follows:
- 2M gCM sentences generated from the parallel corpus by Kunchukuttan et al. (2018)
- 93k rCM sentences from the corpora by Chandu et al. (2018b)
For English-Spanish, the details of the datasets are as follows:
- 8M gCM sentences generated from the corpus by Rijhwani et al. (2017)
93k rCM sentences from the corpus by Rijhwani et al. (2017)
$^{12}$ https://github.com/marekrei/sequence-labeler/tree/484a6beb1e2a2ccaac74ce717b1ee30c79fc8d8
$^{14}$ https://github.com/ElizaLo/Question-Answering-based-on-SQuAD