SlowGuess/ACL_2020_OCR huggingface dataset:
Gustavo Aguilar and Thamar Solorio
Houston, TX 77204-3010
{gaguilaralas, tsolorio}@uh.edu
Linguistic Code-switching (CS) is still an understudied phenomenon in natural language processing. The NLP community has mostly focused on monolingual and multi-lingual scenarios, but little attention has been given to CS in particular. This is partly because of the lack of resources and annotated data, despite its increasing occurrence in social media platforms. In this paper, we aim at adapting monolingual models to code-switched text in various tasks. Specifically, we transfer English knowledge from a pre-trained ELMo model to different code-switched language pairs (i.e., Nepali-English, Spanish-English, and Hindi-English) using the task of language identification. Our method, CS-ELMo, is an extension of ELMo with a simple yet effective position-aware attention mechanism inside its character convolutions. We show the effectiveness of this transfer learning step by outperforming multilingual BERT and homologous CS-unaware ELMo models and establishing a new state of the art in CS tasks, such as NER and POS tagging. Our technique can be expanded to more English-paired code-switched languages, providing more resources to the CS community.
Although linguistic code-switching (CS) is a common phenomenon among multilingual speakers, it is still considered an understudied area in natural language processing. The lack of annotated data combined with the high diversity of languages in which this phenomenon can occur makes it difficult to strive for progress in CS-related tasks. Even though CS is largely captured in social media platforms, it is still expensive to annotate a sufficient amount of data for many tasks and languages. Additionally, not all the languages have the same incidence and predominance, making annotations impractical and expensive for every combination
Hindi-English Tweet
Original: Keep calm and keep kaam se kaam !!other #office #tgif #nametag #buddha_ne #SouvenirFromManali #keepcalm
English: Keep calm and mind your own business !!!
Nepali-English Tweet
Original: Youtu ne ma live re ,other chalcha ki vanni aash garam !other Optimistic .other
English: They said Youtube live, let's hope it works! Optimistic.
Spanish-English Tweet
Original: @MROlvera06other @T11gReother go too cavenders_ne y también ve a @ElToroBoots_ne other
English: @MROolvera06 @T11gRe go to cavenders and also go to @ElToroBoots
Figure 1: Examples of code-switched tweets and their translations from the CS LID corpora for Hindi-English, Nepali-English and Spanish-English. The LID labels ne and other in subscripts refer to named entities and punctuation, emojis or usernames, respectively (they are part of the LID tagset). English text appears in italics and other languages are underlined.
of languages. Nevertheless, code-switching often occurs in language pairs that include English (see examples in Figure 1). These aspects lead us to explore approaches where English pre-trained models can be leveraged and tailored to perform well on code-switching settings.
In this paper, we study the CS phenomenon using English as a starting language to adapt our models to multiple code-switched languages, such as Nepali-English, Hindi-English and Spanish-English. In the first part, we focus on the task of language identification (LID) at the token level using ELMo (Peters et al., 2018) as our reference for English knowledge. Our hypothesis is that English pre-trained models should be able to recognize whether a word belongs to English or not when such models are fine-tuned with code-switched text. To accomplish that, we introduce
8033
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8033-8044 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
CS-ELMo, an extended version of ELMo that contains a position-aware hierarchical attention mechanism over ELMo's character n-gram representations. These enhanced representations allow the model to see the location where particular n-grams occur within a word (e.g., affixes or lemmas) and to associate such behaviors with one language or another. With the help of this mechanism, our models consistently outperform the state of the art on LID for Nepali-English (Solorio et al., 2014), Spanish-English (Molina et al., 2016), and Hindi-English (Mave et al., 2018). Moreover, we conduct experiments that emphasize the importance of the position-aware hierarchical attention and the different effects that it can have based on the similarities of the code-switched languages. In the second part, we demonstrate the effectiveness of our CS-ELMo models by further fine-tuning them on tasks such as NER and POS tagging. Specifically, we show that the resulting models significantly outperform multilingual BERT and their homologous ELMo models directly trained for NER and POS tagging. Our models establish a new state of the art for Hindi-English POS tagging (Singh et al., 2018) and Spanish-English NER (Aguilar et al., 2018).
Our contributions can be summarized as follows: 1) we use transfer learning from models trained on a high-resource language (i.e., English) and effectively adapt them to the code-switching setting for multiple language pairs on the task of language identification; 2) we show the effectiveness of transferring a model trained for LID to downstream code-switching NLP tasks, such as NER and POS tagging, by establishing a new state of the art; 3) we provide empirical evidence on the importance of the enhanced character n-gram mechanism, which aligns with the intuition of strong morphological clues in the core of ELMo (i.e., its convolutional layers); and 4) our CS-ELMo model is self-contained, which allows us to release it for other researchers to explore and replicate this technique on other code-switched languages.
Transfer learning has become more practical in the last years, making possible to apply very large neural networks to tasks where annotated data is limited (Howard and Ruder, 2018; Peters et al.,
2018; Devlin et al., 2019). CS-related tasks are good candidates for such applications, since they are usually framed as low-resource problems. However, previous research on sequence labeling for code-switching mainly focused on traditional ML techniques because they performed better than deep learning models trained from scratch on limited data (Yirmibesoglu and Eryigit, 2018; Al-Badrashiny and Diab, 2016). Nonetheless, some researchers have recently shown promising results by using pre-trained monolingual embeddings for tasks such as NER (Trivedi et al., 2018; Winata et al., 2018) and POS tagging (Soto and Hirschberg, 2018; Ball and Garrette, 2018). Other efforts include the use of multilingual sub-word embeddings like fastText (Bojanowski et al., 2017) for LID (Mave et al., 2018), and cross-lingual sentence embeddings for text classification like LASER (Schwenk, 2018; Schwenk and Li, 2018; Schwenk and Douze, 2017), which is capable of handling code-switched sentences. These results show the potential of pre-trained knowledge and they motivate our efforts to further explore transfer learning in code-switching settings.
Our work is based on ELMo (Peters et al., 2018), a large pre-trained language model that has not been applied to CS tasks before. We also use attention (Bahdanau et al., 2015) within ELMo's convolutions to adapt it to code-switched text. Even though attention is an effective and successful mechanism in other NLP tasks, the code-switching literature barely covers such technique (Sitaram et al., 2019). Wang et al. (2018) use a different attention method for NER, which is based on a gated cell that learns to choose appropriate monolingual embeddings according to the input text. Recently, Winata et al. (2019) proposed multilingual meta embeddings (MME) combined with self-attention (Vaswani et al., 2017). Their method establishes a state of the art on Spanish-English NER by heavily relying on monolingual embeddings for every language in the code-switched text. Our model outperforms theirs by only fine-tuning a generic CS-aware model, without relying on task-specific designs. Another contribution of our work are position embeddings, which have not been considered for code-switching either. These embeddings, combined with CNNs, have proved useful in computer vision (Gehring et al., 2017); they help to localize non-spatial features extracted by convolutional networks within an image. We apply the same prin
<sup>1</sup>Note that there are more than two labels in the LID tagset, as explained in Section 4.
<sup>2</sup>http://github.com/RiTUAL-UH/cs_elmo
8034
ciple to code-switching: we argue that character n-grams without position information may not be enough for a model to learn the actual morphological aspects of the languages (e.g., affixes or lemmas). We empirically validate those aspects and discuss the incidence of such mechanism in our experiments.
ELMo is a character-based language model that provides deep contextualized word representations (Peters et al., 2018). We choose ELMo for this study for the following reasons: 1) it has been trained on a large amount of English data as a general-purpose language model and this aligns with the idea of having English knowledge as starting point; 2) it extracts morphological information out of character sequences, which is essential for our case since certain character n-grams can reveal whether a word belongs to one language or another; and 3) it generates powerful word representations that account for multiple meanings depending on the context. Nevertheless, some aspects of the standard ELMo architecture could be improved to take into account more linguistic properties. In Section 3.1, we discuss these aspects and propose the position-aware hierarchical attention mechanism inside ELMo. In Section 3.2 and Section 3.3, we describe our overall sequence labeling model and the training details, respectively.
3.1 Position-Aware Hierarchical Attention
ELMo convolves character embeddings in its first layers and uses the resulting convolutions to represent words. During this process, the convolutional layers are applied in parallel using different kernel sizes, which can be seen as character n-gram feature extractors of different orders. The feature maps per n-gram order are max-pooled to reduce the dimensionality, and the resulting single vectors per n-gram order are concatenated to form a word representation. While this process has proven effective in practice, we notice the following shortcomings:
1. Convolutional networks do not account for the positions of the character n-grams (i.e., convolutions do not preserve the sequential order), losing linguistic properties such as affixes.
2. ELMo down-samples the outputs of its convolutional layers by max-pooling over the feature maps. However, this operation is not ideal
to adapt to new morphological patterns from other languages as the model tends to discard patterns from languages other than English.
To address these aspects, we introduce CS-ELMo, an extension of ELMo that incorporates a position-aware hierarchical attention mechanism that enhances ELMo's character n-gram representations. This mechanism is composed of three elements: position embeddings, position-aware attention, and hierarchical attention. Figure 2A describes the overall model architecture, and Figure 2B details the components of the enhanced character n-gram mechanism.
Position embeddings. Consider the word  $\mathbf{x}$  of character length  $l$ , whose character n-gram vectors are  $(x_{1}, x_{2}, \ldots, x_{l - j + 1})$  for an n-gram order  $j \in \{1, 2, \ldots, n\}$ . The n-gram vector  $x_{i} \in \mathbb{R}^{c}$  is the output of a character convolutional layer, where  $c$  is the number of output channels for that layer. Also, consider  $n$  position embedding matrices, one per n-gram order,  $\{\mathbf{E}_1, \mathbf{E}_2, \ldots, \mathbf{E}_n\}$  defined as  $\mathbf{E}_j \in \mathbb{R}^{(k - j + 1) \times e}$  where  $k$  is the maximum length of characters in a word (note that  $l \leq k$ ),  $e$  is the dimension of the embeddings and  $j$  is the specific n-gram order. Then, the position vectors for the sequence  $\mathbf{x}$  are defined by  $\mathbf{p} = (p_1, p_2, \ldots, p_{l - j + 1})$  where  $p_i \in \mathbb{R}^e$  is the  $i$ -th vector from the position embedding matrix  $\mathbf{E}_j$ . We use  $e = c$  to facilitate the addition of the position embeddings and the n-gram vectors. Figure 2B illustrates the position embeddings for bi-grams and tri-grams.
Position-aware attention. Instead of down-sampling with the max-pooling operation, we use an attention mechanism similar to the one introduced by Bahdanau et al. (2015). The idea is to concentrate mass probability over the feature maps that capture the most relevant n-gram information along the word, while also considering positional information. At every individual n-gram order, our attention mechanism uses the following equations:
$$
\begin{array}{l} u _ {i} = v ^ {\top} \tanh  \left(\mathrm {W} _ {x} x _ {i} + p _ {i} + b _ {x}\right) (1) \\ \alpha_ {i} = \frac {\exp (u _ {i})}{\sum_ {j = 1} ^ {N} \exp (u _ {j})}, \quad \text {s . t .} \sum_ {i = 1} \alpha_ {i} = 1 (2) \\ z = \sum_ {i = 1} \alpha_ {i} x _ {i} (3) \\ \end{array}
$$
$^{3}$ ELMo has seven character convolutional layers, each layer with a kernel size from one to seven characters  $(n = 7)$ .
4ELMo varies the output channels per convolutional layer, so the dimensionality of  $\mathbf{E}_j$  varies as well.
8035
Figure 2: A) The left figure shows the overall model architecture, which contains CS-ELMo followed by BLSTM and CRF, and a secondary task with a softmax layer using a simplified LID label set. The largest box describes the components of CS-ELMo, including the enhanced character n-gram module proposed in this paper. B) The right figure describes in detail the enhanced character n-gram mechanism inside CS-ELMo. The figure shows the convolutions of a word as input and a single vector representation as output.
where  $\mathrm{W}_x \in \mathbb{R}^{a \times c}$  is a projection matrix,  $a$  is the dimension of the attention space,  $c$  is the number of channels for the n-gram order  $j$ , and  $p_i$  is the position embedding associated to the  $x_i$  n-gram vector.  $v \in \mathbb{R}^a$  is the vector that projects from the attention space to the unnormalized scores, and  $\alpha_i$  is a scalar that describes the attention probability associated to the  $x_i$  n-gram vector.  $z$  is the weighted sum of the input character n-gram vectors and the attention probabilities, which is our down-sampled word representation for the n-gram order  $j$ . Note that this mechanism is used independently for every order of n-grams resulting in a set of  $n$  vectors  $\{z_1, z_2, \ldots, z_n\}$  from Equation 3. This allows the model to capture relevant information across individual n-grams before they are combined (i.e., processing independently all bi-grams, all tri-grams, etc.).
Hierarchical attention. With the previous mechanisms we handle the problems aforementioned. That is, we have considered positional information as well as the attention mechanism to down-sample the dimensionality. These components retrieve one vector representation per n-gram order per word. While ELMo simply concatenates the n-gram vectors of a word, we decide to experiment with another layer of attention that can prioritize n-gram vectors across all the orders. We use a similar formulation to Equations 1 and 3, except that we do not have  $p_i$ , and instead of doing the weighted sum, we concatenate the weighted inputs. This concatenation keeps the original dimensionality expected
in the upper layers of ELMo, while it also emphasizes which n-gram order should receive more attention.
3.2 Sequence Tagging
We follow Peters et al. (2018) to use ELMo for sequence labeling. They reported state-of-the-art performance on NER by using ELMo followed by a bidirectional LSTM layer and a linear-chain conditional random field (CRF). We use this architecture as a backbone for our model (see Figure 2A), but we add some modifications. The first modification is the concatenation of static English word embeddings to ELMo's word representation, such as Twitter (Pennington et al., 2014) and fastText (Bojanowski et al., 2017) embeddings similar to Howard and Ruder (2018) and Mave et al. (2018). The idea is to enrich the context of the words by providing domain-specific embeddings and subword level embeddings. The second modification is the concatenation of the enhanced character ngram representation with the input to the CRF layer. This emphasizes even further the extracted morphological patterns, so that they are present during inference time for the task at hand (i.e., not only LID, but also NER and POS tagging). The last modification is the addition of a secondary task on a simplified language identification label scheme (see Section 4 for more details), which only uses
5The LID label set uses eight labels (lang1, lang2, ne, mixed, ambiguous, fw, other, and unk), but for the simplified LID label set, we only consider three labels (lang1, lang2 and other) to predict only based on characters.
8036
the output of the enhanced character n-gram mechanism. Intuitively, this explicitly forces the model to associate morphological patterns (e.g., affixes, lemmas, etc.) to one or the other language.
3.3 Multi-Task Training
We train the model by minimizing the negative log-likelihood loss of the CRF classifier. Additionally, we force the model to minimize a secondary loss over the simplified LID label set by only using the morphological features from the enhanced character n-gram mechanism (see the softmax layer in Figure 2A). The overall loss  $\mathcal{L}$  of our model is defined as follows:
$$
\mathcal {L} _ {\text {t a s k} _ {t}} = - \frac {1}{N} \sum_ {i} ^ {N} y _ {i} \log p (y _ {i} | \Theta) \tag {4}
$$
$$
\mathcal {L} = \mathcal {L} _ {\text {t a s k} _ {1}} + \beta \mathcal {L} _ {\text {t a s k} _ {2}} + \lambda \sum_ {k} ^ {| \Theta |} w _ {k} ^ {2} \tag {5}
$$
where  $\mathcal{L}_{task_1}$  and  $\mathcal{L}_{task_2}$  are the negative log-likelihood losses conditioned by the model parameters  $\Theta$  as defined in Equation 4.  $\mathcal{L}_{task_1}$  is the loss of the primary task (i.e., LID, NER, or POS tagging), whereas  $\mathcal{L}_{task_2}$  is the loss for the simplified LID task weighted by  $\beta$  to smooth its impact on the model performance. Both losses are the average over  $N$  tokens. The third term provides  $\ell_2$  regularization, and  $\lambda$  is the penalty weight.
Language identification. We experiment with code-switched data for Nepali-English, Spanish-English, and Hindi-English. The first two datasets were collected from Twitter, and they were introduced at the Computational Approaches to Linguistic Code-Switching (CALCS) workshops in 2014 and 2016 (Solorio et al., 2014; Molina et al., 2016). The Hindi-English dataset contains Twitter and Facebook posts, and it was introduced by Mave et al. (2018). These datasets follow the CALCS label scheme, which has eight labels: lang1 (English), lang2 (Nepali, Spanish, or Hindi), mixed, ambiguous, fw, ne, other, and unk. We show the distribution of lang1 and lang2 in Table 1. Moreover, we add a second set of labels using a simplified LID version of the original CALCS label set. The simplified label set uses lang1,
Corpus	Split	Posts	Tokens	Lang1	Lang2
Nep-Eng	Train	8,494	123,959	38,310	51,689
Dev	1,499	22,097	7,173	9,008
Test	2,874	40,268	12,286	17,216
Spa-Eng	Train	11,400	139,539	78,814	33,709
Dev	3,014	33,276	16,821	8,652
Test	10,716	121,446	16,944	77,047
Hin-Eng	Train	5,045	100,337	57,695	20,696
Dev	891	16,531	9,468	3,420
Test	1,485	29,854	17,589	5,842
Table 1: The distribution of the LID datasets according to the CALCS LID label set. The label lang1 refers to English and lang2 is either Nepali, Spanish or Hindi depending on the corpus. The full label distribution is in Appendix A.
lang2, and other. We use this 3-way token-level labels in the secondary loss of our model where only morphology, without any context, is being exploited. This is because we are interested in predicting whether a word's morphology is associated to English more than to another language (or vice versa), instead of whether, for example, its morphology describes a named entity (ne).
Part-of-speech tagging. Singh et al. (2018) provide 1,489 tweets (33,010 tokens) annotated with POS tags. The labels are annotated using the universal POS tagset proposed by Petrov et al. (2012) with the addition of two labels: PART_NEG and PRON_WH. This dataset does not provide training, development, or test splits due to the small number of samples. Therefore, we run 5-fold cross validations and report the average scores.
Named entity recognition. We use the Spanish-English NER corpus introduced in the 2018 CALCS competition (Aguilar et al., 2018), which contains a total of 67,223 tweets with 808,663 tokens. The entity types are person, organization, location, group, title, product, event, time, and other, and the labels follow the BIO scheme. We used the fixed training, development, and testing splits provided with the datasets to benchmark our models.
Importantly, Hindi and Nepali texts in these datasets appear transliterated using the English alphabet (see Figure 1). The lack of a standardized transliteration process leads code-switchers to employ mostly ad-hoc phonological rules that conveniently use the English alphabet when they write in social media. This behavior makes the automated processing of these datasets more challenging be
6While Equation 4 is formulated for a given sentence, in practice  $N$  is the number of tokens in a batch of sentences.
<sup>7</sup>We exclude the CRF parameters in this term.
8037
Exp ID	Experiment	Nepali-English	Spanish-English	Hindi-English
Dev	Test	Dev	Test	Dev	Test
Approach 1 (Baseline models)
Exp 1.1	ELMo	96.192	95.700	95.508	96.363	95.997	96.420
Exp 1.2	ELMo + BLSTM + CRF	96.320	95.882	95.615	96.748	96.545	96.717
Exp 1.3	ML-BERT	95.436	96.571	96.212	96.212	95.924	96.440
Approach 2 (Upon Exp 1.2)
Exp 2.1	Attention on each n-gram	96.413	96.771	95.952	96.519	96.579	96.069
Exp 2.2	Position-aware attention on each n-gram	96.540	96.640	95.994	96.791	96.629	96.141
Exp 2.3	Position-aware hierarchical attention	96.582	96.798	96.072	96.692	96.705	96.186
Approach 3 (Upon Exp 2.3)
Exp 3.1	Concatenating character n-grams at the top	96.485	96.761	96.033	96.775	96.665	96.188
Exp 3.2	Adding simplified LID (secondary) task	96.612	96.734	96.051	96.932	96.565	96.215
Exp 3.3	Adding static word embeddings	96.879	97.026	96.757	97.532	96.776	97.001
Comparison: Previous best published results
Mave et al. (2018)	-	-	96.510	97.060	96.6045	96.840
Table 2: The results of incremental experiments on each LID dataset. The scores are calculated using the weighted F-1 metric across the eight LID labels from CALCS. Within each column, the best score in each block is in bold, and the best score for the whole column is underlined. Note that development scores from subsequent experiments (e.g., Exp 2.2 and 2.3) are statistically significant with  $p < 0.02$ .
Corpus	LID System	Lang1	Lang2	WA F1
Spa-Eng	Al-Badrashiny and Diab	88.6	96.9	95.2
Jain and Bhat	92.3	96.9	96.0
Mave et al.	93.184	98.118	96.840
Ours (Exp 3.3)	94.411	98.532	97.789
Hin-Eng	Mave et al.	98.241	95.657	97.596
Ours (Exp 3.3)	98.372	95.750	97.718
Nep-Eng	Al-Badrashiny and Diab	97.6	97.0	97.3
Ours (Exp 3.3)	98.124	95.170	97.387
Table 3: Comparison of our best models with the best published scores for language identification. Scores are calculated with the F1 metric, and WA F1 is the weighted average F1 between both languages.
cause it excludes potentially available resources in the original scripts of the languages.
We describe our experiments for LID in Section 5.1, including insights of the optimized models. In Section 5.2, the optimized LID models are further fine-tuned on downstream NLP tasks, such as NER and POS tagging, to show the effectiveness of our preliminary CS adaptation step. We test for statistical significance across our incremental experiments following Dror et al. (2018), and we report p-values below 0.02 for LID. We discuss hyperparameters and fine-tuning details in Appendix D.