Venue: NAACL 2018

# Universal Dependency Parsing for Hindi-English Code-switching

Irshad Ahmad Bhat

LTRC, IIIT-H,

Hyderabad, India

irshad.bhat@iiit.ac.in &Riyaz Ahmad Bhat

Interaction Labs,

Bangalore, India

rbhat@interactions.com &Manish Shrivastava

LTRC, IIIT-H,

Hyderabad, India

m.shrivastava@iitit.ac.in &Dipti Misra Sharma

LTRC, IIIT-H,

Hyderabad, India

dipti@iiit.ac.in

###### Abstract

Code-switching is a phenomenon of mixing grammatical structures of two or more languages under varied social constraints. The code-switching data differ so radically from the benchmark corpora used in NLP community that the application of standard technologies to these data degrades their performance sharply. Unlike standard corpora, these data often need to go through additional processes such as language identification, normalization and/or back-transliteration for their efficient processing. In this paper, we investigate these indispensable processes and other problems associated with syntactic parsing of code-switching data and propose methods to mitigate their effects. In particular, we study dependency parsing of code-switching data of Hindi and English multilingual speakers from Twitter. We present a treebank of Hindi-English code-switching tweets under Universal Dependencies scheme and propose a neural stacking model for parsing that efficiently leverages part-of-speech tag and syntactic tree annotations in the code-switching treebank and the preexisting Hindi and English treebanks. We also present normalization and back-transliteration models with a decoding process tailored for code-switching data. Results show that our neural stacking parser is 1.5% LAS points better than the augmented parsing model and our decoding process improves results by 3.8% LAS points over the first-best normalization and/or back-transliteration.

## 1 Introduction

Code-switching1 (henceforth CS) is the juxtaposition, within the same speech utterance, of grammatical units such as words, phrases, and clauses belonging to two or more different languages [20]. The phenomenon is prevalent in multilingual societies where speakers share more than one language and is often prompted by multiple social factors [46]. Moreover, code-switching is mostly prominent in colloquial language use in daily conversations, both online and offline.

Footnote 1: Code-mixing is another term in the linguistics literature used interchangeably with code-switching. Both terms are often used to refer to the same or similar phenomenon of mixed language use.

Most of the benchmark corpora used in NLP for training and evaluation are based on edited monolingual texts which strictly adhere to the norms of a language related, for example, to orthography, morphology, and syntax. Social media data in general and CS data, in particular, deviate from these norms implicitly set forth by the choice of corpora used in the community. This is the reason why the current technologies often perform miserably on social media data, be it monolingual or mixed language data [9, 7, 23, 24, 16]. CS data offers additional challenges over the monolingual social media data as the phenomenon of code-switching transforms the data in many ways, for example, by creating new lexical forms and syntactic structures by mixing morphology and syntax of two languages making it much more diverse than any monolingual corpora [23]. As the current computational models fail to cater to the complexities of CS data, there is often a need for dedicated techniques tailored to its specific characteristics.

Given the peculiar nature of CS data, it has been widely studied in linguistics literature [14, 20, 46], and more recently, there has been a surge in studies concerning CS data in NLP as well [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 183, 185, 187, 188, 189, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 222, 213, 214, 215, 216, 217, 218, 229, 220, 221, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 282, 284, 286, 287, 288, 289, 291, 285, 286, 288, 289, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 323, 335, 336, 337, 341, 342, 343, 344, 345, 346, 347, 348, 358, 360, 361, 362, 363, 364, 365, 366, 367, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 388, 389, 390, 391, 392, 303, 304, 305, 306, 307, 309, 311, 323, 338, 389, 393, 394, 395, 396, 397, 398, 399, 400, 41, 42, 435, 446, 447, 45, 46, 47, 48, 490, 42, 449, 41, 44, 491, 44, 492, 42, 44, 44, 493, 44, 44, 494, 45, 495, 496, 497, 498, 500, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 52, 54, 57, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 78, 79, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 82, 84, 86, 88, 89, 91, 83, 85, 87, 89, 92, 86, 89, 93, 94, 95, 96, 97, 98,

2017; Guzman et al., 2017, and others). Besides the individual computational works, a series of shared-tasks and workshops on preprocessing and shallow syntactic analysis of CS data have also been conducted at multiple venues such as Empirical Methods in NLP (EMNLP 2014 and 2016), International Conference on NLP (ICON 2015 and 2016) and Forum for Information Retrieval Evaluation (FIRE 2015 and 2016). Most of these works have attempted to address preliminary tasks such as language identification, normalization and/or back-transliteration as these data often need to go through these additional processes for their efficient processing. In this paper, we investigate these indispensable processes and other problems associated with syntactic parsing of code-switching data and propose methods to mitigate their effects. In particular, we study dependency parsing of Hindi-English code-switching data of multilingual Indian speakers from Twitter. Hindi-English code-switching presents an interesting scenario for the parsing community. Mixing among typologically diverse languages will intensify structural variations which will make parsing more challenging. For example, there will be many sentences containing: (1) both SOV and SVO word orders2, (2) both head-initial and head-final genitives, (3) both prepositional and postpositional phrases, etc. More importantly, none among the Hindi and English treebanks would provide any training instance for these mixed structures within individual sentences. In this paper, we present the first code-switching treebank that provides syntactic annotations required for parsing mixed-grammar syntactic structures. Moreover, we present a parsing pipeline designed explicitly for Hindi-English CS data. The pipeline comprises of several modules such as a language identification system, a back-transliteration system, and a dependency parser. The gist of these modules and our overall research contributions are listed as follows:

Footnote 2: Order of Subject, Object and Verb in transitive sentences.

* back-transliteration and normalization models based on encoder-decoder frameworks with sentence decoding tailored for code-switching data;
* a dependency treebank of Hindi-English code-switching tweets under Universal Dependencies scheme; and
* a neural parsing model which learns POS tagging and parsing jointly and also incorporates knowledge from the monolingual treebanks using neural stacking.

## 2 Preliminary Tasks

As preliminary steps before parsing of CS data, we need to identify the language of tokens and normalize and/or back-transliterate them to enhance the parsing performance. These steps are indispensable for processing CS data and without them the performance drops drastically as we will see in Results Section. We need normalization of non-standard word forms and back-translation of Romanized Hindi words for addressing out-of-vocabulary problem, and lexical and syntactic ambiguity introduced due to contracted word forms. As we will train separate normalization and back-transliteration models for Hindi and English, we need language identification for selecting which model to use for inference for each word form separately. Moreover, we also need language information for decoding best word sequences.

### Language Identification

For language identification task, we train a multilayer perceptron (MLP) stacked on top of a recurrent bidirectional LSTM (Bi-LSTM) network as shown in Figure 1.

We represent each token by a concatenated vector of its English embedding, back-translated Hindi embedding, character Bi-LSTM embedding and flag embedding (English dictionary flag and word length flag with length bins of 0-3, 4-6, 7-10, and 10-all). These concatenated vectors are passed to a Bi-LSTM network to generate a sequence of

Figure 1: Language identification network



hidden representations which encode the contextual information spread across the sentence. Finally, output layer uses the feed-forward neural network with a softmax function for a probability distribution over the language tags. We train the network on our CS training set concatenated with the data set provided in ICON 20153 shared task (728 Facebook comments) on language identification and evaluate it on the datasets from Bhat et al. (2017). We achieved the state-of-the-art performance on both development and test sets Bhatt et al. (2017). The results are shown in Table 1.

Footnote 3: [http://ltrei.iiit.ac.in/fcon2015/](http://ltrei.iiit.ac.in/fcon2015/)

### Normalization and Back-transliteration

We learn two separate but similar character-level models for normalization-cum-transfiteration of noisy Romanized Hindi words and normalization of noisy English words. We treat both normalization and back-transliteration problems as a general sequence to sequence learning problem. In general, our goal is to learn a mapping for non-standard English and Romanized Hindi word forms to standard forms in their respective scripts. In case of Hindi, we address the problem of normalization and back-translation of Romanized Hindi words using a single model. We use the attention-based encoder-decoder model of Luong Luong et al. (2015) with global attention for learning. For Hindi, we train the model on the transliteration pairs (87,520) from the Libindic transliteration project4 and BrahmiNet Kunchukuttan et al. (2015) which are further augmented with noisy transliteration pairs (1,75,668) for normalization. Similarly, for normalization of noisy English words, we train the model on noisy word forms (4,29,715) synthetically generated from the English vocabulary. We use simple rules such as dropping non-initial vowels and replacing consonants based on their phonological proximity to generate synthetic data for normalization. Figure 2 shows some of the noisy forms generated from standard word forms using simple and finite rules which include vowel elision (please\({}_{\blacksquare}\) pls), interchanging similar consonants and vowels (cousin\({}_{\blacksquare}\) couzin), relacing consonant or vowel clusters with a single letter (Twitter\({}_{\blacksquare}\) Twiter), etc. From here onwards, we will refer to both normalization and back-transliteration as normalization.

Footnote 4: [https://github.com/libindic/indic-trans](https://github.com/libindic/indic-trans)

At inference time, our normalization models will predict the most likely word form for each input word. However, the single-best output from the model may not always be the best option considering an overall sentential context. Contracted word forms in social media content are quite often ambiguous and can represent different standard word forms. For example, noisy form 'pt' can expand to different standard word forms such as 'put', 'pit', 'pat', 'pot' and 'pet'. The choice of word selection will solely depend on the sentential context. To select contextually relevant forms, we use exact search over n-best normalizations from the respective models extracted using beam-search decoding. The best word sequence is selected using the Viterbi decoding over \(b^{n}\) word sequences scored by a trigram language model. \(b\) is the size of beam-width and \(n\) is the sentence length. The language models are trained on the monolingual data of Hindi and English using KenLM toolkit Heafield et al. (2013). For each word, we extract five best normalizations (\(b\)=5). Decoding the best word sequence is a nontrivial problem for CS data due to lack of normalized and back-translated CS data for training a language model. One obvious solution is to apply decoding on individual language fragments in a CS sentence Dutta et al. (2015). One major prob

Figure 2: Synthetic normalization pairs generated for a sample of English words using hand crafted rules.

 

lem with this approach is that the language models used for scoring are trained on complete sentences but are applied on sentence fragments. Scoring individual CS fragments might often lead to wrong word selection due to incomplete context, particularly at fragment peripheries. We solve this problem by using a 3-step decoding process that works on two separate versions of a CS sentence, one in Hindi, and one in English. In the first step, we replace first-best back-transliterated forms of Hindi words by their translation equivalents using a Hindi-English bilingual lexicon.5 An exact search is used over the top '\(5\)' normalizations of English words, the translation equivalents of Hindi words and the actual word itself. In the second step, we decode best word sequence over Hindi version of the sentence by replacing best English word forms decoded from the first step by their translation equivalents. An exact search is used over the top '\(5\)' normalizations of Hindi words, the dictionary equivalents of decoded English words and the original words. In the final step, English and Hindi words are selected from their respective decoded sequences using the predicted language tags from the language identification system. Note that the bilingual mappings are only used to aid the decoding process by making the CS sentences lexically monolingual so that the monolingual language models could be used for scoring. They are not used in the final decoded output. The overall decoding process is shown in Figure 3.

Footnote 5: An off-the-shelf MT system would have been appropriate for this task, however, we would first need to adapt it to CS data which in itself is a non-trivial task.

Both of our normalization and back-transliteration systems are evaluated on the evaluation set of Bhat et al. (2017). Results of our systems are reported in Table 3 with a comparison of accuracies based on the nature of decoding used. The results clearly show the significance of our 3-step decoding over first-best and fragment-wise decoding.

## 3 Universal Dependencies for Hindi-English

Recently Bhat et al. (2017) provided a CS dataset for the evaluation of their parsing models which they trained on the Hindi and English Universal Dependency (UD) treebanks. We extend this dataset by annotating 1,448 more sentences. Following Bhat et al. (2017) we first sampled CS data from a large set of tweets of Indian language users that we crawled from Twitter using Tweepy6-a Twitter API wrapper. We then used a language identification system trained on ICON dataset (see Section 2) to filter Hindi-English CS tweets from the crawled Twitter data. Only those tweets were selected that satisfied a minimum ratio of 30:70(%) code-switching. From this dataset, we manually selected 1,448 tweets for annotation. The selected tweets are thoroughly checked for code-switching ratio. For POS tagging and dependency annotation, we used Version 2 of Universal dependency guidelines De Marneffe et al. (2014),

Figure 3: The figure shows a 3-step decoding process for the sentence “_Yar cn anyone tel me k twitr account bnd ksy krty hn plz” (Friend can anyone tell me how to close twitter account please)._



while language tags are assigned based on the tag set defined in [18, 20]. The dataset was annotated by two expert annotators who have been associated with annotation projects involving syntactic annotations for around 10 years. Nonetheless, we also ensured the quality of the manual annotations by carrying an inter-annotator agreement analysis. We randomly selected a dataset of 150 tweets which were annotated by both annotators for both POS tagging and dependency structures. The inter-annotator agreement has a 96.20% accuracy for POS tagging and a 95.94% UAS and a 92.65% LAS for dependency parsing.

We use our dataset for training while the development and evaluation sets from Bhat et al. (2017) are used for tuning and evaluation of our models. Since the annotations in these datasets follow version 1.4 of the UD guidelines, we converted them to version 2 by using carefully designed rules. The statistics about the data are given in Table 3.

## 4 Dependency Parsing

We adapt Kiperwasser and Goldberg (2016) transition-based parser as our base model and incorporate POS tag and monolingual parse tree information into the model using neural stacking, as shown in Figures 4 and 6.

### Parsing Algorithm

Our parsing models are based on an arc-eager transition system [23]. The arc-eager system defines a set of configurations for a sentence w1,...,wn, where each configuration C = (S, B, A) consists of a stack S, a buffer B, and a set of dependency arcs A. For each sentence, the parser starts with an initial configuration where S = {ROOT}, B = [w1,...,wn] and A = \(\emptyset\) and terminates with a configuration C if the buffer is empty and the stack contains the ROOT. The parse trees derived from transition sequences are given by A. To derive the parse tree, the arc-eager system defines four types of transitions (\(t\)): Shift, Left-Arc, Right-Arc, and Reduce.

We use the training by exploration method of Goldberg and Nivre (2012) for decoding a transition sequence which helps in mitigating error propagation at evaluation time. We also use pseudo-projective transformations of Nivre and Nilsson (2005) to handle a higher percentage of non-projective arcs in the CS data (\(\sim\)2%). We use the most informative scheme of head+path to store the transformation information.

### Base Models

Our base model is a stack of a tagger network and a parser network inspired by stack-propagation model of Zhang and Weiss (2016). The parameters of the tagger network are shared and act as a regularization on the parsing model. The model is trained by minimizing a joint negative log-likelihood loss for both tasks. Unlike Zhang and Weiss (2016), we compute the gradients of the log-loss function simultaneously for each training instance. While the parser network is updated given the parsing loss only, the tagger network is updated with respect to both tagging and parsing losses. Both tagger and parser networks comprise of an input layer, a feature layer, and an output layer as shown in Figure 4. Following Zhang and Weiss (2016), we refer to this model as stack-prop.

Figure 4: POS tagging and parsing network based on stack-propagation model proposed in [20].

\begin{table}
\begin{tabular}{|c|c c|c c c c c|} \hline Data-set & Sentences & Tokens & Hi & En & Ne & Univ & Arco \\ \hline Train & 1,448 & 20,203 & 8,363 & 8,270 & 698 & 2,730 & 142 \\ Dev & 225 & 3,411 & 1,549 & 1,300 & 151 & 379 & 32 \\ Test & 225 & 3,295 & 1,465 & 1,283 & 168 & 349 & 30 \\ \hline \end{tabular}
\end{table}
Table 3: Data Statistics. Dev set is used for tuning model parameters, while Test set is used for evaluation.

 

Tagger network:The input layer of the tagger encodes each input word in a sentence by concatenating a pre-trained word embedding with its character embedding given by a character Bi-LSTM. In the feature layer, the concatenated word and character representations are passed through two stacked Bi-LSTMs to generate a sequence of hidden representations which encode the contextual information spread across the sentence. The first Bi-LSTM is shared with the parser network while the other is specific to the tagger. Finally, output layer uses the feed-forward neural network with a softmax function for a probability distribution over the Universal POS tags. We only use the forward and backward hidden representations of the focus word for classification.

Parser Network:Similar to the tagger network, the input layer encodes the input sentence using word and character embeddings which are then passed to the shared Bi-LSTM. The hidden representations from the shared Bi-LSTM are then concatenated with the dense representations from the feed-forward network of the tagger and passed through the Bi-LSTM specific to the parser. This ensures that the tagging network is penalized for the parsing error caused by error propagation by back-propagating the gradients to the shared tagger parameters [23]. Finally, we use a non-linear feed-forward network to predict the labeled transitions for the parser configurations. From each parser configuration, we extract the top node in the stack and the first node in the buffer and use their hidden representations from the parser specific Bi-LSTM for classification.

### Stacking Models

It seems reasonable that limited CS data would complement large monolingual data in parsing CS data and a parsing model which leverages both data would significantly improve parsing performance. While a parsing model trained on our limited CS data might not be enough to accurately parse the individual grammatical fragments of Hindi and English, the preexisting Hindi and English treebanks are large enough to provide sufficient annotations to capture their structure. Similarly, parsing model(s) trained on the Hindi and English data may not be able to properly connect the divergent fragments of the two languages as the model lacks evidence for such mixed structures in the monolingual data. This will happen quite often as Hindi and English are typologicalls very diverse (see Figure 5).

As we discussed above, we adapted feature-level neural stacking [24, 16] for joint learning of POS tagging and parsing. Similarly, we also adapt this stacking approach for incorporating the monolingual syntactic knowledge into the base CS model. Recently, Wang et al. wang_etal_2017 used neural stacking for injecting syntactic knowledge of English into a graph-based Singlish parser which lead to significant improvements in parsing performance. Unlike Wang et al. wang_etal_2017, our base stacked models will allow us to transfer the POS tagging knowledge as well along the parse tree knowledge.

As shown in Figure 6, we transfer both POS tagging and parsing information from the source

Figure 5: Code-switching tweet showing grammatical fragments from Hindi and English.

Figure 6: Neural Stacking-based parsing architecture for incorporating monolingual syntactic knowledge.

 

model trained on augmented Hindi and English data. For tagging, we augment the input layer of the CS tagger with the MLP layer of the source tagger. For transferring parsing knowledge, hidden representations from the parser specific Bi-LSTM of the source parser are augmented with the input layer of the CS parser which already includes the hidden layer of the CS tagger, word and character embeddings. In addition, we also add the MLP layer of the source parser to the MLP layer of the CS parser. The MLP layers of the source parser are generated using raw features from CS parser configurations. Apart from the addition of these learned representations from the source model, the overall CS model remains similar to the base model shown in Figure 4. The tagging and parsing losses are back-propagated by traversing back the forward paths to all trainable parameters in the entire network for training and the whole network is used collectively for inference.

## 5 Experiments

We train all of our POS tagging and parsing models on training sets of the Hindi and English UD-v2 treebanks and our Hindi-English CS treebank. For tuning and evaluation, we use the development and evaluation sets from Bhat et al. (2017). We conduct multiple experiments in gold and predicted settings to measure the effectiveness of the sub-modules of our parsing pipeline. In predicted settings, we use the POS taggers separately trained on the Hindi, English and CS training sets. All of our models use word embeddings from transformed Hindi and English embedding spaces to address the problem of lexical differences prevalent in CS sentences.

### Hyperparameters

Word RepresentationsFor language identification, POS tagging and parsing models, we include the lexical features in the input layer of our neural networks using 64-dimension pre-trained word embeddings, while we use randomly initialized embeddings within a range of \([-0.1,+0.1]\) for non-lexical units such as POS tags and dictionary flags. We use 32-dimensional character embeddings for all the three models and 32-dimensional POS tag embeddings for pipelined parsing models. The distributed representation of Hindi and English vocabulary are learned separately from the Hindi and English monolingual corpora. The English monolingual data contains around 280M sentences, while the Hindi data is comparatively smaller and contains around 40M sentences. The word representations are learned using Skip-gram model with negative sampling which is implemented in word2vec toolkit Mikolov et al. (2013). We use the projection algorithm of Artetxe et al. (2016) to transform the Hindi and English monolingual embeddings into same semantic space using a bilingual lexicon (\(\sim\)63,000 entries). The bilingual lexicon is extracted from ILCI and Bojar Hindi-English parallel corpora Jha (2010); Bojar et al. (2014). For normalization models, we use 32-dimensional character embeddings uniformly initialized within a range of \([-0.1,+0.1]\).

Hidden dimensionsThe POS tagger specific Bi-LSTMs have 128 cells while the parser specific Bi-LSTMs have 256 cells. The Bi-LSTM in the language identification model has 64 cells. The character Bi-LSTMs have 32 cells for all three models. The hidden layer of MLP has 64 nodes for the language identification network, 128 nodes for the POS tagger and 256 nodes for the parser. We use hyperbolic tangent as an activation function in all tasks. In the normalization models, we use single layered Bi-LSTMs with 512 cells for both encoding and decoding of character sequences.

LearningFor language identification, POS tagging and parsing networks, we use momentum SGD for learning with a minibatch size of 1. The LSTM weights are initialized with random orthonormal matrices as described in Saxe et al. (2013). We set the dropout rate to 30% for POS tagger and parser Bi-LSTM and MLP hidden states while for language identification network we set the dropout to 50%. All three models are trained for up to 100 epochs, with early stopping based on the development set.

In case of normalization, we train our encoder-decoder models for 25 epochs using vanilla SGD. We start with a learning rate of \(1.0\) and after 8 epochs reduce it to half for every epoch. We use a mini-batch size of 128, and the normalized gradient is rescaled whenever its norm exceeds 5. We use a dropout rate of 30% for the Bi-LSTM.

Language identification, POS tagging and parsing code is implemented in DyNet Neubig et al. (2017) and for normalization without decoding, we use Open-NMT toolkit for neural machine translation Klein et al. (2017). All 

the code is available at [https://github.com/irshadbhat/nsdp-cs](https://github.com/irshadbhat/nsdp-cs) and the data is available at [https://github.com/CodeMixedUniversalDependencies/UD_Hindi_English](https://github.com/CodeMixedUniversalDependencies/UD_Hindi_English).

## 6 Results

In Table 4, we present the results of our main model that uses neural stacking for learning POS tagging and parsing and also for knowledge transfer from the Bilingual model. Transferring POS tagging and syntactic knowledge using neural stacking gives 1.5% LAS7 improvement over a naive approach of data augmentation. The Bilingual model which is trained on the union of Hindi and English data sets is least accurate of all our parsing models. However, it achieves better or near state-of-the-art results on the Hindi and English evaluation sets (see Table 5). As compared to the best system in CoNLL 2017 Shared Task on Universal Dependencies Zeman et al. (2017); Dozat et al. (2017), our results for English are around 3% better in LAS, while for Hindi only 0.5% LAS points worse. The CS model trained only on the CS training data is slightly more accurate than the Bilingual model. Augmenting the CS data to Hindi-English data complements their syntactic structures relevant for parsing mixed grammar structures which are otherwise missing in the individual datasets. The average improvements of around \(\sim\)5% LAS clearly show their complementary nature.

Footnote 7: The improvements discussed in the running text are for the models that are evaluated in auto settings.

Table 6 summarizes the POS tagging results on the CS evaluation set. The tagger trained on the CS training data is 2.5% better than the Bilingual tagger. Adding CS training data to Hindi and English train sets further improves the accuracy by 1%. However, our stack-prop tagger achieves the highest accuracy of 90.53% by leveraging POS information from Bilingual tagger using neural stacking.

Pipeline vs Stack-propTable 7 summarizes the parsing results of our pipeline models which use predicted POS tags as input features. As compared to our stack-prop models (Table 4), pipeline models are less accurate (average 1% LAS improvement across models) which clearly emphasizes the significance of back-propagating the parsing loss to tagging parameters as well.

Significance of normalizationWe also conducted experiments to evaluate the impact of normalization on both POS tagging and parsing. The results are shown in Table 8. As expected, tagging and parsing models that use normalization without decoding achieve an average of 1% improvement over the models that do not use normalization at all. However, our 3-step decoding leads to higher gains in tagging as well as parsing accuracies. We achieved around 2.8% improvements in tagging and around 4.6% in parsing over the models that use first-best word forms from the normalization models. More importantly, there is a mod

\begin{table}
\begin{tabular}{|c|c c|c c|} \hline Model & \multicolumn{2}{c|}{Gold (LID+TRN)} & Auto (LID+TRN) \\ \cline{2-5}  & Pipeline & SP & Pipeline & SP \\ \hline \hline Bilingual & 88.36 & 88.12 & 86.71 & 86.27 \\ \hline CS & 90.32 & 90.38 & 89.12 & 89.19 \\ \hline Augmented & 91.20 & 91.50 & 90.02 & 90.20 \\ \hline Neural Stacking & **91.76** & **91.90** & **90.36** & **90.53** \\ \hline Bhat et al. (2017) & 86.00 & & 85.30 \\ \hline \end{tabular}
\end{table}
Table 6: POS tagging accuracies of different models on CS evaluation set. SP = stack-prop.

\begin{table}
\begin{tabular}{|c|c c|c c|} \hline Model & \multicolumn{2}{c|}{Gold (LID+TRN)} & Auto (LID+TRN) \\ \cline{2-5}  & UAS & LAS & UAS & LAS \\ \hline \hline Bilingual & 75.26 & 65.41 & 73.29 & 63.18 \\ \hline CS & 76.69 & 66.90 & 75.84 & 64.94 \\ \hline Augmented & 80.39 & 71.27 & 78.95 & 69.51 \\ \hline Neural Stacking & **81.50** & **72.44** & **80.23** & **71.03** \\ \hline Bhat et al. (2017) & 74.16 & 64.11 & 66.18 & 54.40 \\ \hline \end{tabular}
\end{table}
Table 4: Accuracy of different parsing models on the test set using predicted language tags. normalized/back-transliterated words and predicted POS tags. POS tags are predicted separately before parsing. In Neural Stacking model, only parsing knowledge from the Bilingual model is transferred.

\begin{table}
\begin{tabular}{|c|c c|c c|c c|} \hline \multirow{2}{*}{Model} & \multicolumn{4}{c|}{Popeline} & \multicolumn{4}{c|}{Stack-prop} \\ \cline{2-7}  & Gold POS & \multicolumn{4}{c|}{Auto POS} \\ \cline{2-7}  & UAS & LAS & POS & UAS & LAS & POS & UAS & LAS \\ \hline Hindi & 95.66 & 93.08 & 97.52 & 94.08 & 90.69 & **97.65** & **94.36** & **91.02** \\ \hline English & 89.95 & 87.96 & 95.75 & 87.71 & 84.59 & **95.80** & **88.30** & **85.30** \\ \hline \end{tabular}
\end{table}
Table 5: POS and parsing results for Hindi and English monolingual test sets using pipeline and stack-prop models.

 

erate drop in accuracy (1.4% LAS points) caused due to normalization errors (see results in Table 4 for gold vs auto normalization).

**Monolingual vs Cross-lingual Embeddings** We also conducted experiments with monolingual and cross-lingual embeddings to evaluate the need for transforming the monolingual embeddings into a same semantic space for processing of CS data. Results are shown in Table 9. Cross-lingual embeddings have brought around \(\sim\)0.5% improvements in both tagging and parsing. Cross-lingual embeddings are essential for removing lexical differences which is one of the problems encountered in CS data. Addressing the lexical differences will help in better learning by exposing syntactic similarities between languages.

## 7 Conclusion

In this paper, we have presented a dependency parser designed explicitly for Hindi-English CS data. The parser uses neural stacking architecture of Zhang and Weiss (2016) and Chen et al. (2016) for learning POS tagging and parsing and for knowledge transfer from Bilingual models trained on Hindi and English UD treebanks. We have also presented normalization and back-transliteration models with a decoding process tailored for CS data. Our neural stacking parser is 1.5% LAS points better than the augmented parsing model and 3.8% LAS points better than the one which uses first-best normalizations.

## References

* Artetxe et al. (2016) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. pages 2289-2294.
* Bhat et al. (2017) Irshad Bhat, Riyaz A. Bhat, Manish Shrivastava, and Dipti Sharma. 2017. Joining hands: Exploiting monolingual treebanks for parsing of code-mixing data. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_. Association for Computational Linguistics, Valencia, Spain, pages 324-330.
* Hindi-English and Hindi-only Corpus for Machine Translation. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafin Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_. European Language Resources Association (ELRA), Reykjavik, Iceland.
* Cetinoglu et al. (2016) Ozlem Cetinoglu, Sarah Schulz, and Ngoc Thang Vu. 2016. Challenges of computational processing of code-switching. In _Proceedings of the Second Workshop on Computational Approaches to Code Switching_. Association for Computational Linguistics, Austin, Texas, pages 1-11.
* Chanu et al. (2017) Khyathi Raghavi Chandu, Manoj Chinnakotla, Alan W Black, and Manish Shrivastava. 2017. Webshoth: A code mixed factoid question answering system for web. In _International Conference of the Cross-Language Evaluation Forum for European Languages_. Springer, pages 104-111.
* Chen et al. (2016) Hongshen Chen, Yue Zhang, and Qun Liu. 2016. Neural network for heterogeneous annotations. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 731-741.
* De Marneffe et al. (2014) Marie-Catherine De Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D Manning. 2014. Universal stanford dependencies: A cross-linguistic typology. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation_. volume 14, pages 4585-92.
* Dozat et al. (2017) Timothy Dozat, Peng Qi, and Christopher D Manning. 2017. Stanford's graph-based neural dependency parser at the conll 2017 shared task. _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ pages 20-30.
* Dutta et al. (2015) Sukanya Dutta, Tista Saha, Sommath Banerjee, and Sudip Kumar Nakar. 2015. Text normalization in code-mixed social media text. In _Proceedings of the

\begin{table}
\begin{tabular}{|c|c|c c|} \hline System & POS & UAS & LAS \\ \hline \hline No Normalization & 86.98 & 76.25 & 66.02 \\ \hline First Best & 87.74 & 78.26 & 67.22 \\ \hline
3-step Decoding & **90.53** & **80.23** & **71.03** \\ \hline \end{tabular}
\end{table}
Table 8: Impact of normalization and back-transliteration on POS tagging and parsing models.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Embedding & POS & UAS & LAS \\ \hline \hline Monolingual & 90.07 & 79.46 & 70.53 \\ \hline Crosslingual & **90.53** & **80.23** & **71.03** \\ \hline \end{tabular}
\end{table}
Table 9: Impact of monolingual and cross-lingual embeddings on stacking model performance.

 

2nd International Conference on Recent Trends in Information Systems (ReTIS)_.
* Gimpel et al. [2011] Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2_. Association for Computational Linguistics, pages 42-47.
* Goldberg and Nivre [2012] Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In _Proceedings of the 24th International Conference on Computational Linguistics_. pages 959-976.
* Gumperz [1982] John J Gumperz. 1982. _Discourse strategies_, volume 1. Cambridge University Press.
* Guzman et al. [2017] Gaulberto Guzman, Joseph Ricard, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for modeling code-switching across corpora. _Proceedings of Interspeech 2017_ pages 67-71.
* Haefield et al. [2013] Kenneth Haefield, Ivan Pouzyrevsky, Jonathan H Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In _ACL (2)_. pages 690-696.
* Jamatia et al. [2015] Anupam Jamatia, Bjorn Gamback, and Amitava Das. 2015. Part-of-speech tagging for code-mixed english-hindi twitter and facebook chat messages. page 239.
* Jha [2010] Girish Nath Jha. 2010. The TDIL program and the Indian language corpora initiative (ILCI). In _Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC 2010)_. European Language Resources Association (ELRA).
* Joshi et al. [2016] Aditya Joshi, Ameya Prabhu, Manish Shrivastava, and Vasudeva Varma. 2016. Towards sub-word level compositions for sentiment analysis of hindi-english code mixed text. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLING 2016 Organizing Committee, Osaka, Japan, pages 2482-2491.
* Kiperwasser and Goldberg [2016] Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional lstm feature representations. _Transactions of the Association for Computational Linguistics_ 4:313-327.
* Klein et al. [2017] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In _Proc. ACL_. [https://doi.org/10.18653/v1/P17-4012](https://doi.org/10.18653/v1/P17-4012).
* Kong et al. [2014] Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, and Chris Dyer. 2014. A dependency parser for tweets. In _Proceedings of Conference on Empirical Methods In Natural Language Processing (EMNLP)_. volume 1001, page 1012.
* Kunchukuttan et al. [2015] Anoop Kunchukuttan, Ratish Puduppuly, and Pushpak Bhattacharyya. 2015. Brahmi-net: A transliteration and script conversion system for languages of the indian subcontinent.
* Luong et al. [2015] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. _arXiv preprint arXiv:1508.04025_ .
* Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_ .
* Myers-Scotton [1995] Carol Myers-Scotton. 1995. _Social motivations for codeswiching: Evidence from Africa_. Oxford University Press.
* Neubig et al. [2017] Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. _arXiv preprint arXiv:1701.03980_ .
* Nivre [2003] Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In _Proceedings of the 8th International Workshop on Parsing Technologies (IWPT)_.
* Nivre and Nilsson [2005] Joakim Nivre and Jens Nilsson. 2005. Pseudo-projective dependency parsing. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_.
* Owoputi et al. [2013] Olutobi Owoputi, Brendan OConnor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In _Proceedings of NAACL-HLT_. pages 380-390.
* Podleck [1980] Shana Poplack. 1980. Sometimes ill start a sentence in spanish y termino en espanol: toward a typology of code-switching1. _Linguistics_ 18(7-8):581-618.
* Rijhwani et al. [2017] Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on twitter with a novel generalized word-level language detection technique. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. volume 1, pages 1971-1982.
* Rudra et al. [2017] Koustav Rudra, Shruti Rijhwani, Rafiya Begum, Karlka Bali, Monojit Choudhury, and Niloy Ganguly.

 

2016. Understanding language preference for expression of opinion and sentiment: What do hindinguish speakers do on twitter? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 1131-1141. [https://aclweb.org/anthology/D16-1121](https://aclweb.org/anthology/D16-1121).
* Saxe et al. (2013) Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. _arXiv preprint arXiv:1312.6120_ .
* hindi-english code-mixed social media text. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 1340-1345.
* Solorio et al. (2014) Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steve Bethard, Mona Diab, Mahmoud Gonheim, Abdelati Hawwari, Fahad AlGhamadi, Julia Hirshberg, Alison Chang, and Pascale Fung. 2014. Overview for the first shared task on language identification in code-switched data. In _Proceedings of the First Workshop on Computational Approaches to Code-Switching. EMNLP 2014, Conference on Empirical Methods in Natural Language Processing, Octobe, 2014, Doha, Qatar_.
* Solorio and Liu (2008a) Thamar Solorio and Yang Liu. 2008a. Learning to predict code-switching points. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 973-981.
* Solorio and Liu (2008b) Thamar Solorio and Yang Liu. 2008b. Part-of-speech tagging for english-spanish code-switched text. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1051-1060.
* Vyas et al. (2014) Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014. Pos tagging of english-hindi code-mixed social media content. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. volume 14, pages 974-979.
* Wang et al. (2017) Hongmin Wang, Yue Zhang, GuangYong Leonard Chan, Jie Yang, and Hai Leong Chiueu. 2017. Universal dependencies parsing for colloquial singaporean english. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, pages 1732-1744.
* Zeman et al. (2016) Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmada, Anna Nedduzhko, Silvie Cinkova, Jan Hajic jr, Jaroslavacva, Vaclav Kettnerova, Zdenka Uresova, Jenna Kanerva, Stina Ojala, Anna Missilia, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sangunetti, Maria Simi, Hiroshi Kanayama, Valeria de Paixa, Kira Droganova, Hector Martinez Alonso, Cagf Coltekin, Ulmut Sulubackat, Hans Uszkoreit, Vivien Macketanz, Alyoshe Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkhaky, Zhuoran Yu, Emily Pitler, Saran Lertpratidt, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadova, Esha Banerjee, Rui Manurung, Antonio Stella, Astuso Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitasiro, and Josie Li. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. Association for Computational Linguistics, Vancouver, Canada, pages 1-19.
* Zhang and Weiss (2016) Yuan Zhang and David Weiss. 2016. Stackpropagation: Improved representation learning for syntax. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 1557-1566.

 

Supplemental Material