Venue: EMNLP 2017

**All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media**

\({}^{1}\)**Jasabanta Patro, \({}^{2}\)Bidisha Samanta, \({}^{2}\)Saurabh Singh,**

\({}^{2}\)**Abhipsa Basu,\({}^{2}\)Prithwish Mukherjee, \({}^{3}\)Monojit Choudhury, \({}^{4}\)Animesh Mukherjee**

\({}^{1,2,4}\) Indian Institute of Technology Kharagpur, India - 721302

\({}^{6}\)Microsoft Research India, Bangalore - 560001

\({}^{1}\)jasabantapatro@iitkgp.ac.in, \({}^{3}\)monojitc@microsoft.com, \({}^{4}\)animeshm@cse.iitkgp.ernet.in

###### Abstract

In this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman's correlation values, our methods perform more than two times better (\(\sim 0.62\)) in predicting the borrowing like-liness compared to the best performing baseline (\(\sim 0.26\)) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88% of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus indicating a huge scope for improvement of automatic language identification systems.

## 1 Introduction

In social media communication, multilingual people often switch between languages, a phenomenon known as _code-switching_ or _codemixing_[1]. This makes _language identification and tagging_, which is perhaps a prerequisite for almost all other language processing tasks that follow, a challenging problem [9]. In code-mixing people are subconsciously aware of the foreign origin of the code-mixed word or the phrase. A related but linguistically and cognitively distinct phenomenon is _lexical borrowing_ (or simply, _borrowing_), where a word or phrase from a foreign language say \(L_{2}\) is used as a part of the vocabulary of native language say \(L_{1}\). For instance, in Dutch, the English word "sale" is now used more frequently than the Dutch equivalent "uitverkoop". Some English words like "shop" are even inflected in Dutch as "shoppen" and heavily used. While it is difficult in general to ascertain whether a foreign word or phrase used in an utterance is borrowed or just an instance of code-mixing [1], one tell tale sign is that only proficient multilinguals can code-mix, while even monolingual speakers can use borrowed words because, by definition, these are part of the vocabulary of a language. In other words, just because an English speaker understands and uses the word "tortilla" does not imply that she can speak or understand Spanish. A borrowed word from \(L_{2}\) initially appears frequently in speech, then gradually in print media like newspaper and finally it loses its origin's identity and is used in \(L_{1}\) resulting in an inclusion in the dictionary of \(L_{1}\)[11, 12]. Borrowed words often take several years before they formally become part of \(L_{1}\) dictionary. This motivates our research question "is early-stage automatic identification of likely to be borrowed words possible?". This is known to be a hard problem because (i) it is a socio-linguistic phenomenon closely related to acceptability and frequency, (ii) borrowing is a dynamic process; new borrowed words enter the lexicon of a language as old words, both native and borrowed, might slowly fade away from usage, and (iii) it is a population level phenomenon that necessitates data from a large portion of the population unlike standard natural language corpora that typically comes from a very small set of authors. Automatic identification of borrowed words in social media content (SMC) can improve language tagging by recommending the tagger to tag the language of the borrowed words as \(L_{1}\) instead

of \(L_{2}\). The above reasons motivate us to resort to the social media (in particular, Twitter), where a large population of bilingual/multilingual speakers are known to often tweet in code-mixed colloquial languages Carter et al. (2013); Solorio et al. (2014); Vyas et al. (2014); Jurgens et al. (2017); Rijhwani et al. (2017). We designed our methodology to work for any pair of languages \(L_{1}\) and \(L_{2}\) subject to the availability of sufficient SMC. In the current study, we consider Hindi as \(L_{1}\) and English as \(L_{2}\).

The main stages of our research are as follows:

_Metrics to quantify the likeliness of borrowing from social media signals_: We define three novel and closely similar metrics that serve as social signals indicating the likeliness of borrowing. We compare the likeliness of borrowing as predicted by our model and a baseline model with that from the ground truth obtained from human judges.

_Ground truth generation_: We launch an extensive survey among \(58\) human judges of various age groups and various educational backgrounds to collect responses indicating if each of the candidate foreign word is likely borrowed.

_Application_: We randomly selected some words that have a high, low and medium borrowing likeliness as predicted by our metrics. Further, we randomly selected one tweet for each of the chosen words. The chosen words in almost all of these tweets have \(L_{2}\) as their language tag while a majority of the surrounding words have a tag \(L_{1}\). We asked expert annotators to re-evaluate the language tags of the chosen words and indicate if they would prefer to switch this tag from \(L_{2}\) from \(L_{1}\).

Finally, our key results are outlined below:

1. We obtained the Spearman's rank correlation between the ground-truth ranking and the ranking based on our metrics as \(\sim 0.62\) for all the three variants which is more than double the value (\(\sim 0.26\)) if we use the most competitive baseline Bali et al. (2014) available in the literature.

2. Interestingly, the responses of the judges in the age group below \(30\) seem to correspond even better with our metrics. Since language change is brought about mostly by the younger population, this might possibly mean that our metrics are able to capture the early signals of borrowing.

3. Those users that mix languages the least in their tweets present the best signals of borrowing in case they do mix the languages (correlation of our metrics estimated from the tweets of these users with that of the ground truth is \(\sim 0.65\)).

4. Finally, we obtain an excellent re-annotation accuracy of 88% for the words falling in the surely borrowed category as predicted by our metrics.

## 2 Related work

In linguistics _code-mixing_ and _borrowing_ have been studied under the broader scope of language change and evolution. Linguists have for a long time focused on the sociological and the conversational necessity of borrowing and mixing in multilingual communities (see Auer (1984) and Muysken (1996) for a review). In particular, Sankoff et al. (1990) describes the complexity of choosing features that are indicative of borrowing. This work further showed that it is not always true that only highly frequent words are borrowed; nonce words could also be borrowed along with the frequent words. More recently, Nzai et al. (2014) analyzed the formal conversation of Spanish-English multilingual people and found that code mixing/borrowing is not only restricted to daily speech but is also prevalent in formal conversations. Hadei (2016) showed that phonological integration could be evaluated to understand the phenomenon of word borrowing. Along similar lines, Sebonde (2014) showed morphological and syntactic features could be good indicators for numerical borrowings. Senaratne (2013) reported that in many languages English words are likely to be borrowed in both formal and semi-formal text.

**Mixing in computer mediated communication and social media**: Sotillo (2012) investigated various types of code-mixing in a corpora of 880 SMS text messages. The author observed that most often mixing takes place at the beginning of a sentence as well as through simple insertions. Similar observations about chat and email messages have been reported in Bock (2013); Negron (2009). However, studies of code-mixing with Chinese-English bilinguals from Hong Kong Li (2009) and Macao San (2009) brings forth results that contrast the aforementioned findings and indicate that in these societies code-mixing is driven more by linguistic than social motivations.

Recently, the advent of social media has immensely propelled the research on code-mixing and borrowing as a dynamic social phenomena. Hidayat (2012) noted that in Facebook, users mostly preferred inter-sentential mixing and showed that 45% of the mixing originated from 

real lexical needs, 40% was used for conversations on a particular topic and the rest 5% for content clarification. In contrast, Das and Gamback (2014) showed that in case of Facebook messages, intra-sentential mixing accounted for more than half of the cases while inter-sentential mixing accounted only for about one-third of the cases. In fact, in the First Workshop on _Computational Approaches to Code Switching_ a shared task on code-mixing in tweets was launched and four different code-mixed corpora were collected from Twitter as a part of the shared task Solorio et al. (2014). Language identification task has also been handled for English-Hindi and English-Bengali code-mixed tweets Das and Gamback (2013). Part-of-speech tagging have been recently done for code-mixed English-Hindi tweets Solorio and Liu (2008); Vyas et al. (2014).

**Diachronic studies**: As an aside, it is interesting to note that the availability of huge volumes of timestamped data (tweet streams, digitized books) is now making it possible to study various linguistic phenomena quantitatively over different timescales. For instance, Sagi et al. (2009) uses latent semantic analysis for detection and tracking of changes in word meaning, whereas Frermann and Lapata (2016) presents a Bayesian approach for the same problem. Peirsman et al. (2010) presents a distributed model for automatic identification of lexical variation between language varieties. Bamman and Crane (2011) discusses a method for automatically identifying word sense variation in a dated collection of historical books Mitra et al. (2014) presents a computational method for automatic identification of change in word senses across different timescales. Cook et al. (2014) presents a method for novel sense identification of words over time.

Despite these diverse and rich research agendas in the field of code-switching and lexical dynamics, there has not been much attempt to quantify the likeliness of borrowing of foreign words in a language. The only work that makes an attempt in this direction is Bali et al. (2014), which is described in detail in Sec 3.1. One of the primary challenges faced by any quantitative research on lexical borrowing is that borrowing is a social phenomenon, and therefore, it is difficult to identify suitable indicators of such a lexical diffusion process unless one has access to a large population-level data. In this work, we show for the first time how certain simple and closely related signals encoding the language usage of social media users can help us construe appropriate metrics to quantify the likeliness of borrowing of a foreign word.

## 3 Methodology

In this section, we present the baseline metric and propose three new metrics that quantify the likeliness of borrowing.

### Baseline metric

_Baseline metric_ - We consider the \(log(\frac{F_{L_{2}}}{E_{L_{1}}})\) value proposed in Bali et al. (2014) as the baseline metric. Here \(F_{L_{2}}\) denotes the frequency of the \(L_{1}\) transliterated form of the word \(w\) in the standard \(L_{1}\) newspaper corpus. \(F_{L_{1}}\), on the other hand, denotes the frequency of the \(L_{1}\) translation of the word \(w\) in the same newspaper corpus. For our experiments discussed in the later sections, both the transliteration and the translation of the words have been done by a set of volunteers who are native \(L_{1}\) speakers. The authors in Bali et al. (2014) claim that the more positive the value of this metric is for a word \(w\), the higher is the likeliness of its being borrowed. The more negative the value is, the higher are the chances that the word \(w\) is an instance of code-mixing.

_Ranking_ - Based on the values obtained from the above metric for a set of target words, we rank these words; words with high positive values feature at the top of the rank list and words with high negative values feature at the bottom of the list. For two words having the same \(log(\frac{F_{L_{2}}}{F_{L_{1}}})\) value, we resolve the conflict by assigning each of these the average of their two rank positions. In a subsequent section, we shall compare this rank list with the one obtained from the ground truth responses.

### Proposed metric

In this section, we present three novel and closely related metrics based on the language usage patterns of the users of social media (in specific, Twitter). In order to define our metrics, we need all the words to be language tagged. The different tags that a word can have are: \(L_{1}\), \(L_{2}\), _NE_ Named Entity) and _Others_. Based on the word level tag, we also create a tweet level tag as follows:

1. \(L_{1}\): Almost every word (\(>90\%\)) in the tweet is tagged as \(L_{1}\).
2. \(L_{2}\): Almost every word (\(>90\%\)) in the tweet is tagged as \(L_{2}\) 

3. \(CML_{1}\): Code-mixed tweet but majority (i.e., \(>50\%\)) of the words are tagged as \(L_{1}\).
4. \(CML_{2}\): Code-mixed tweet but majority (i.e., \(>50\%\)) of the words are tagged as \(L_{2}\).
5. _CMEQ_: Code-mixed tweet having very similar number of words tagged as \(L_{1}\) and \(L_{2}\) respectively.
6. _Code Switched_: There is a trail of \(L_{1}\) words followed by a trail of \(L_{2}\) words or vice versa.

Using the above classification, we define the following metrics:

_Unique User Ratio_ (\(UUR\)) - The Unique User Ratio for word usage across languages is defined as follows:

\[UUR(w)=\frac{U_{L_{1}}+U_{CML_{1}}}{U_{L_{2}}} \tag{1}\]

where \(U_{L_{1}}\) (\(U_{L_{2}}\), \(U_{CML_{1}}\)) is the number of unique users who have used the word \(w\) in a \(L_{1}\) (\(L_{2}\), \(CML_{1}\)) tweet at least once.

_Unique Tweet Ratio_ (\(UTR\)) - The Unique Tweet Ratio for word usage across languages is defined as follows:

\[UTR(w)=\frac{T_{L_{1}}+T_{CML_{1}}}{T_{L_{2}}} \tag{2}\]

where \(T_{L_{1}}\) (\(T_{L_{2}}\), \(T_{CML_{1}}\)) is the total number of \(L_{1}\) (\(L_{2}\), \(CML_{1}\)) tweets which contain the word \(w\).

_Unique Phrase Ratio_ (\( UPR\)) - The Unique Phrase Ratio for word usage across languages is defined as follows:

\[UPR(w)=\frac{P_{L_{1}}}{P_{L_{2}}} \tag{3}\]

where \(P_{L_{1}}\) (\(P_{L_{2}}\)) is the number of \(L_{1}\) (\(L_{2}\)) phrases which contain the word \(w\). Note that unlike the definitions of \(UUR\) and \(UTR\) that exploit the word level language tags, the definition of \( UPR\) exploits the phrase level language tags.

_Ranking_ - We prepare a separate rank list of the target words based on each of the three proposed metrics - \(UUR\), \(UTR\) and \( UPR\). The higher the value of each of this metric the higher is the likeliness of the word \(w\) to be borrowed and higher up it is in the rank list. In a subsequent section, we shall compare these rank lists with the one prepared from the ground truth responses.

## 4 Experiments

In this section we discuss the dataset for our experiments, the evaluation criteria and the ground truth preparation scheme.

### Datasets and preprocessing

In this study, we consider code-mixed tweets gathered from Hindi-English bilingual Twitter users in order to study the effectiveness of our proposed metrics. The native language \(L_{1}\) is Hindi and the foreign language \(L_{2}\) is English. To bootstrap the data collection process, we used the language tagged tweets presented in Rudra et al. (2016). In addition to this, we also crawled tweets (between Nov 2015 and Jan 2016) related to 28 hashtags representing different Indian contexts covering important topics such as sports, religion, movies, politics etc. This process results in a set of 811981 tweets. We language-tag (see details later in this section) each tweet so crawled and find that there are 3577 users who use mixed language for tweeting. Next, we systematically crawled the time lines of these 3577 users between Feb 2016 and March 2016 to gather more mixed language tweets. Using this two step process we collected a total of 1550714 distinct tweets. From this data, we filtered out tweets that are not written in romanized script, tweets having only URLs and tweets having empty content. Post filtering we obtained 725173, tweets which we use for the rest of the analysis. The datasets can be downloaded from [http://cnerg.org/borrow](http://cnerg.org/borrow)

**Language tagging**: We tag each word in a tweet with the language of its origin using the method outlined in Gella et al. (2013). _Hi_ represents a predominantly Hindi tweet, _En_ represents a predominantly English tweet, _CMH_ (_CME_) represents code-mixed tweets with more Hindi (English) words, _CMEQ_ represents code-mixed tweets with almost equal number of Hindi and English words and _CS_ represents code-switched tweets (the number and % of tweets in each of the above six categories are presented in the supplementary material). Like the word level, the tagger also provides a phrase level language tag. Once again, the different tags that an entire phrase can have are: _En_, _Hi_ and _Oth_ (Other). The metrics defined in the previous section are computed using these language tags.

**Newspaper dataset for the baseline**: As we had discussed in the previous section, for the construction of the baseline ranking we need to resort to counting the frequency of the foreign words (i.e., English words) and their Hindi translations in a newspaper corpus as has been outlined in Bali et al. (2014). For this purpose, we use the FIRE 

dataset built from the Hindi Jagaran newspaper corpus1 which is written in Devanagari script.

Footnote 1: Iagaran corpus: [http://fire.irsi.res.in/fire/static/data](http://fire.irsi.res.in/fire/static/data)

### Target word selection

We first compute the most frequent foreign (i.e., English words) in our tweet corpus. Since we are interested in the frequency of the English word only when it appears as a foreign word we do not consider the (i) _Hi_ tweets since they do not have any foreign word, (ii) _En_ tweets since here the English words are not foreign words and the (iii) code-switched tweets. Based on the frequency of usage of English as a foreign word, we select the top 1000 English words. Removal of stop words and text normalization leaves beyond 230 nouns (see supplementary material for the list of words).

**Final selection of target words**: In language processing, context plays an important role in understanding different properties of a word. For our study, we also attempt to use the language tags as features of the context words for a given _target word_. Our hypothesis here is that there should exist classes of words that have similar context features and the likelihood of being borrowed in each class should be different. For example, when an English word is surrounded by mostly Hindi words it seems to be more likely borrowed. We present two examples in the box below to illustrate this.

[background=lines]Example I: @*****Welcome. _Film jaroor dekhma. Nahit to injection ready hai. Example II: @*****Trust @*****Kuch to ache se karo sirji... Har jagah bhaqte rehna_ is not a good _thing_.

In _Example I_ the English word "film" is surrounded by mostly Hindi words. On the other hand, in _Example II_ the English word "thing" is surrounded mostly by English words. Note that the word "film" is very commonly used by Hindi monolingual speakers and is therefore highly likely to have been borrowed unlike the English word "thing" which is arguably an instance of mixing. This socio-linguistic difference seems to be very appropriately captured by the language tag of the surrounding words of these two words in the respective tweets. Based on this hypothesis we arrange the 230 words into contextually similar groups (see supplementary material for the grouping details). Finally, using the baseline metric \(log(\frac{F_{E}}{F_{H}})\) (\(E\): English, \(H\): Hindi), we proportionately choose words from these groups as follows: _Words with very high or very low values of \(log(\frac{F_{E}}{F_{H}})\)_ (\(hlws\)) - we select words having the highest and the lowest values of \(log(\frac{F_{E}}{F_{H}})\) from each of the context groups. This constitutes a set of 30 words. Note that these words are baseline-biased and therefore the metric should be able to discriminate them well. _Words with medium values of \(log(\frac{F_{E}}{F_{H}})\)_ (\(mws\)) - we selected 27 words having not so high and not so low \(log(\frac{F_{E}}{F_{H}})\) at uniformly at random. _Full set of words_ (\(full\)) - Thus, in total we selected 57 target words for the purpose of our evaluation. We present these words in the box below.

[background=lines]Baseline-biased words - 'thing', 'way', 'woman', 'press', 'wrong', 'well', 'matter', 'reason', 'question', 'guy', 'moment', 'week', 'luck', 'president', 'body', 'job', 'car', 'god', 'gift', 'status', 'university', 'lyrics', 'road', 'politics', 'parliament', 'review', 'scene', 'seat', 'filt', 'degree', 'Randomly selected words - 'people', 'play', 'house', 'service', 'rest', 'boy', 'month', 'money', 'cool', 'development', 'group', 'friend', 'day', 'performance', 'school', 'blue', 'room', 'interview', 'share', 'request', 'profile', 'college', 'star', 'class', 'superstar', 'petrol', 'uncle'

## 4 Evaluation criteria

We present a four step approach for evaluation as follows. We measure (i) how well the \(UUR\), \(UTR\) and \(UPR\) based ranking of the \(hlws\) set, the \(mws\) set and the \(full\) set correlate with the ground truth ranking (discussed in the next section) in comparison to the rank given by the baseline metric, (ii) how well the different rank ranges obtained from our metric align with the ground truth as compared to the baseline metric, (iii) whether there are some systematic effects of the age group of the survey participants on the rank correspondence, (iv) how our metrics if computed from the tweets of users who (a) rarely mix languages, (b) almost always mix languages and (c) are in between (a) and (b), align with the ground truth.

**Rank correlation**: We measure the standard Spearman's rank correlation (\(\rho\)) [18] pairwise between rank lists generated by (i) \(UUR\) (ii) \(UTR\) (iii) \(UPR\) (iv) baseline metric and the ground truth.

We shall describe the next four measurements 

taking \(UUR\) as the running example. The same can be extended verbatim for the other two similar metrics.

* (i) surely borrowed (SB) containing top 20% words from each list, (ii) likely borrowed (LB) containing the next 20% words from each list, (iii) borderline (BL) constituting the subsequent 20% words from each list, (iv) likely mixed (LM) comprising the next 20% words from each list and (v) surely mixed (SM) having the last 20% words from each rank list. Therefore, we have three sets of five buckets, one set each for \(UUR\), the ground truth and the baseline based rank list.

Next we calculate the bucket-wise correspondence between (i) the \(UUR\) and the ground truth set and (ii) the baseline and the ground truth set in terms of standard _precision_ and _recall_ measures. For our purpose, we adapt these measures as follows.

\(G\): ground truth bucket set, \(B_{b}\): baseline bucket set, \(U_{b}\): \(UUR\) bucket set;

\(BS\in\{B_{b},U_{b}\}\), \(T\) (type of bucket) = {SB, LB, BL, LM, SM};

\(b_{t}\) = words in type \(t\) bucket from \(BS\), \(g_{t}\) = words in type \(t\) bucket from \(G\), \(t\in T\);

\(tp_{t}\) (no. of true positives) = \(|b_{t}\cap g_{t}|\), \(fp_{t}\) (no. of false positives) = \(|b_{t}-g_{t}|\), \(tn_{t}\) (no. of true negatives) = \(|g_{t}-b_{t}|\);

Bucket-wise _precision_ and _recall_ therefore:

\(precision(b_{t})=\frac{tp_{t}}{f_{p_{t}}+tp_{t}}\); \(recall(b_{t})=\frac{tp_{t}}{tn_{t}+tp_{t}}\)

For a given set, we obtained the overall _macro precision_ (_recall_) by averaging the _precision_ (_recall_) values over the five buckets. For a given set, we also obtained the overall _micro precision_ by first adding the true positives across all the buckets and then normalizing by the sum of the true and the false positives over all the buckets. We take an equivalent approach for obtaining the _micro recall_.

**Age group effect**: Here we construct two ground truth rank lists one using the responses of the participants with age below 30 (young population) and the other using the responses of the rest of the participants (elderly population). Next we repeat the above two evaluations considering each of the new ground truth rank lists.

* (i) High (users who have more than 20% of tweets as code-mixed), (ii) Mid (users who have 7-20% of their tweets as code-mixed, and (iii) Low (users who have less than 7% of their tweets as code-mixed). We create three \(UUR\) based rank lists for each of these three user categories and respectively compare them with the ground truth rank list.

### Ground truth preparation

Since it is very difficult to obtain a suitable ground truth to validate the effectiveness of our proposed ranking scheme, we launched an online survey to collect human judgment for each of the 57 _target words_.

* (i) a Hindi sentence with the _target word_ as the only English word, (ii) the same Hindi sentence in (i) but with the _target word_ replaced by its Hindi translation and (iii) none of the above two options. There were no time restrictions imposed while gathering the responses, i.e., the volunteers theoretically had unlimited time to decide their responses for each _target word_.

Footnote 2: Survey portal [https://goo.gl/forms/L0kJmBBNMrkj0jA53](https://goo.gl/forms/L0kJmBBNMrkj0jA53)

**Language preference factor** For each _target word_, we compute a _language preference factor_ (\(LPF\)) defined as \((Count_{En}-Count_{Hi})\), where \(Count_{Hi}\) refers to the number of survey participants who preferred the sentence containing the Hindi translation of the _target word_ while \(Count_{En}\) refers to the number of survey participants who preferred the sentence containing the _target word_ itself. More positive values of \(LPF\) denotes higher usage of _target word_ as compared to its Hindi translation and therefore higher likeliness of the word being borrowed.

**Ground truth rank list generation** We generate the ground truth rank list based on the \(LPF\) score of a _target word_. The word with the highest value 

of \(LPF\) appears at the top of the ground truth rank list and so on in that order. Tie breaking between _target words_ having equal \(LPF\) values is done by assigning average rank to each of these words.

* (i) young population (age \(<30\)) and (ii) elderly population (age \(\geq 30\)). For each group we then construct a separate \(LPF\) based ranking of the _target words_.

## 5 Results

### Correlation among rank lists

The Spearman's rank correlation coefficient (\(\rho\)) of the rank lists for the \(hlws\) set, the \(mws\) set and the \(full\) set according to the baseline metric, \(UUR\), \(UTR\) and \( UPR\) with respect to the ground truth metric \(LPF\) are noted in table 1. We observe that for the \(full\) set, the \(\rho\) between the rank lists obtained from all the three metrics \(UUR\), \(UTR\), and \( UPR\) with respect to the ground truth is \(\sim 0.62\) which is more than double the \(\rho\) (\(\sim 0.26\)) between the baseline and the ground truth rank list. This clearly shows that the proposed metrics are able to identify the likeliness of borrowing quite accurately and far better than the baseline. Further, a remarkable observation is that our metrics outperform the baseline metric even for the \(hlwes\) set that is baseline-biased. Likewise, for the \(mws\) set, our metrics outperform the baseline indicating a superior recall on arbitrary words. The ranking of the \(full\) set of words obtained from the ground truth, the baseline and the \(UUR\) metric is available in the supplementary material.

We present the subsequent results for the \(full\) set and the \(UUR\) metric. The results obtained for the other two metrics \(UTR\) and \( UPR\) are very similar and therefore not shown.

### Rank list alignment across rank ranges

The number of _target words_ falling in each bucket across the three rank lists are the same and are noted in table 2. Thus, the _precision_ and _recall_ as per the definition are also the same. The bucketwise _precision/recall_ for the baseline and \(UUR\) with respect to the ground truth are noted in table 3. We observe that while in the SB bucket both the baseline and \(UUR\) perform equally well, for all the other buckets \(UUR\) massively outperforms the baseline. This implies that for the case where the likeliness of borrowing is the strongest, the baseline does as good as \(UUR\). However, as one moves down the rank list, \(UUR\) turns out to be a considerably better predictor than the baseline. The overall _macro_ and _micro precision/recall_ as shown in table 4 further strengthens our observation that \(UUR\) is a better metric than the baseline.

### Age group based analysis

As already discussed earlier, we split the ground truth responses based on the age group of the survey participants. We split the responses into two groups - (i) young population (age \(<30\)) and (ii) elderly population (age \(\geq 30\)) so that there are almost equal number of responses in both the groups (see supplementary material for the exact distribution).

**Rank correlation**: The Spearman's rank correlation of \(UUR\) and the baseline rank lists with these

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline Rank-List\({}_{1}\) & Rank-List\({}_{2}\) & \(p-hws\) & \(\rho-mws\) & \(\rho-full\) \\ \hline \(UUR\) & Ground truth & **0.67** & **0.64** & **0.62** \\ \hline \(UTR\) & Ground truth & **0.66** & **0.63** & **0.63** \\ \hline \(UTR\) & Ground truth & **0.66** & **0.64** & **0.62** \\ Baseline & Ground truth & 0.49 & 0.14 & 0.26 \\ \hline \end{tabular}
\end{table}
Table 1: Spearman’s rank correlation coefficient (\(\rho\)) among the different rank lists. Best result is marked in bold.

\begin{table}
\begin{tabular}{|c|c|c|} \hline Backet type & _pre./inc._ Baseline & _prec./rec._\(UUR\) \\ \hline
**SB** & **0.27** & **0.27** \\ \hline LB & **0.09** & **0.18** \\ \hline BL & **0.08** & **0.33** \\ \hline LM & **0.18** & **0.36** \\ \hline SM & **0.33** & **0.50** \\ \hline \end{tabular}
\end{table}
Table 2: Number of words falling in each bucket of three bucket sets.

\begin{table}
\begin{tabular}{|c|c|c|} \hline Measure & Baseline & \(UUR\) \\ \hline \hline _Micro pre. prec. proc._ & 0.19 & **0.33** \\ \hline _Micro pre. proc._ & 0.19 & **0.33** \\ \hline \end{tabular}
\end{table}
Table 4: Overall _macro_ and _micro precision/recall_. Best results are marked in bold.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Backet type & _pre./inc._ Baseline & _prec./rec._\(UUR\) \\ \hline
**SB** & **0.27** & **0.27** \\ \hline LB & **0.09** & **0.18** \\ \hline BL & **0.08** & **0.33** \\ \hline LM & **0.18** & **0.36** \\ \hline SM & **0.33** & **0.50** \\ \hline \end{tabular}
\end{table}
Table 3: Bucket-wise _precision/recall_. Best results are marked in bold.

 

two ground truth rank lists are shown in table 5. Interestingly, the correlation between \(UUR\) rank list and the young population ground truth is better than the elderly population ground truth. This possibly indicates that \(UUR\) is able to predict recent borrowings more accurately. However, note that the \(UUR\) rank list has a much higher correlation with both the ground truth rank lists as compared to the baseline rank list.

**Rank ranges**: Table 6 shows the bucket-wise _precision_ and _recall_ for \(UUR\) and the baseline metrics with respect to two new ground truths. For the young population once again the number of words in each bucket for all the three sets is the same thus making the values of the _precision_ and the _recall_ same. In fact, the _precision/recall_ for this ground truth is exactly same as in the case of the original ground truth.

In contrast, when we consider the ground truth based on the responses of the elderly population, the number of words across the different buckets are different across the three sets. In this case, we observe that the _precision/recall_ values are better for the \(UUR\) metric in SB, LB and LM buckets.

Finally, the overall _macro_ and _micro precision_ and _recall_ for both the age groups are noted in table 7. Once again, for both the young and the elderly population based ground truths, the _macro_ and _micro precision_ and _recall_ values for the \(UUR\) metric are higher compared to that of the baseline.

### Extent of language mixing

As mentioned earlier, we divide the set of 3577 users into three categories. The Spearman's correlation between \(UUR\) and the ground truth for each of these buckets are given in table 8. As we can see, for Low bucket the \(\rho\) value is maximum. This points to the fact that the signals of borrowing is strongest from the users who rarely mix languages.

## 6 Re-annotation results

In order to conduct the re-annotation experiments we performed the following. To begin with, we ranked all the 230 English nouns in non-increasing order of their \(UUR\) values. We then randomly selected 20 words each having (i) high \(UUR\) (top 20%) values (call \(TOP\)), (ii) low \(UUR\) (bottom 20%) values (call \(BOT\)), and (iii) middle \(UUR\) (middle 20%) values (call \(MID\)). This makes a total of 60 words. Using this word list we extracted one tweet each that contained the (foreign) word from this list along with all other words in the tweet tagged in Hindi (\(H_{all}\)). We similarly prepared another such list of 60 words and extracted one tweet each in which most of the other words were tagged in Hindi (\(H_{most}\)).

We presented the selected words and the corresponding tweets to a set of annotators and asked them to annotate these selected words once again. Over all the words, we calculated the mean (\(\mu_{E\to H}\)) and the standard deviation (\(\sigma_{E\to H}\)) of the fraction of cases where the annotators altered the tag of the selected word from English to Hindi. The average inter-annotator agreement Fleiss [19] for our experiments was found to be as high as 0.64. For the words in the \(TOP\) list, the fraction of times the tag is altered is **0.91 (0.85)** with an inter-annotator agreement of 0.84 (0.80) for the

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|} \hline Bucker & Young & Young & Elder- & Elder- & Elder- \\ type & Baseline & \(UUR\) & Baseline/UUR & baseline & \(UUR\) \\ \cline{2-6}  & _pr_ & _p!_ & _p!_ & _p!_ & _r!_ \\ \hline \hline SB & **0.27** & **0.27** & 0.27 & **0.36** & 0.25 & **0.33** \\ \hline LB & 0.09 & **0.18** & 0.09 & **0.18** & 0.08 & **0.17** \\ \hline BL & 0.08 & **0.33** & **0.16** & 0.08 & **0.28** & 0.14 \\ \hline LM & 0.18 & **0.36** & 0.18 & **0.45** & 0.14 & **0.35** \\ \hline SM & 0.33 & **0.5** & **0.41** & 0.25 & **0.41** & 0.25 \\ \hline \end{tabular}
\end{table}
Table 6: Bucket-wise _precision_ (_p_)/_recall_ (_r_) for \(UUR\) and the \(baseline\) metrics for the two new ground truths. Best results are marked in bold.

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline Backer & Number of users & \(\rho\) \\ \hline High & 302 & 0.52 \\ Mid & 744 & 0.60 \\ \hline Low & 2531 & **0.45** \\ \hline \end{tabular}
\end{table}
Table 8: Spearman’s correlation between \(UUR\) and the ground truth in the different user buckets. Best results are marked in bold.

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline Rank-List\({}_{1}\) & Rank-List\({}_{2}\) & \(\rho\) \\ \hline Baseline & Ground-truth-Young & 0.26 \\ \hline \(UUR\) & Ground-truth-Yung & **0.62** \\ \hline Baseline & Ground-truth-Elder & 0.27 \\ \hline \(UUR\) & Ground-truth-Elder & **0.53** \\ \hline \end{tabular}
\end{table}
Table 5: Spearman’s rank correlation across the two age groups. Best results are marked in bold.

 

\(H_{all}\) (\(H_{most}\)) category. In other words, on average, in as high as 88% of cases the annotators altered the tags of the words that are highly likely to be borrowed (i.e., \(TOP\)) in a largely Hindi context (i.e., \(H_{all}\) or \(H_{most}\)). Table 9 shows the fractional changes for all the other possible cases. An interesting observation is that the annotators rarely flipped the tags for the words in the \(BOT\) list (i.e., the sure mixing cases) in either of the \(H_{all}\) or the \(H_{most}\) contexts. These results strongly support the inclusion of our metric in the design of future automatic language tagging tasks.

## 7 Discussion and conclusion

In this paper, we proposed a few new metrics for estimating the likliness of borrowing that rely on signals from large scale social media data. Our best metric is two-fold better than the previous metric in terms of the accuracy of the prediction. There are some interesting linguistic aspects of borrowing as well as certain assumptions regarding the social media users that have important repercussions on this work and its potential extensions, which are discussed in this section.

**Types of borrowing**: Linguists define broadly three forms of borrowing, (i) _cultural_, (ii) _core_, and (iii) _therapeutic_ borrowings. In _cultural borrowing_, a foreign word gets borrowed into native language to fill a lexical gap. This is because there is no equivalent native language word present to represent the same foreign word concept. For instance, the English word 'computer' has been borrowed in many Indian languages since it does not have a corresponding term in those languages3. In _core borrowing_, on the other hand, a foreign word replaces its native language translation in the native language vocabulary. This occurs due to overwhelming use of the foreign word over native language translation as a matter of prestige, ease of use etc. For example, the English word 'school' has become much more prevalent than its corresponding Hindi translation '_vidyalaya_' among the native Hindi speakers. Finally, therapeutic borrowing refers to borrowing of words to avoid taboo and holonomy in the native language. In this paper, although we did not perform any category based studies, most of our focus was on core borrowing.

Footnote 3: Words for “computer” were coined in many Indian languages through morphological derivation from the term for ”compute”, however, none of these words are used in either formal or informal contexts.

**Language of social media users**: We assumed that if a user is predominantly using Hindi words in a tweet then the chances of him/her being a native Hindi speaker should be high, since, while the number of English native speakers in India is 0.02%, the number of Hindi native speakers is 41.03%4. This assumption has also been made in earlier studies Rudra et al. (2016). Note that even if a user is not a native Hindi speaker but a proficient (or semi-proficient) Hindi speaker, the main results of our analysis should hold. For instance, consider two foreign words 'a' and 'b'. If 'a' is frequently borrowed in the native language, then the proficient speaker would also tend to borrow 'a' similar to a native speaker. Even if due to lack of adequate native vocabulary, the non-native speaker borrows the word 'b' in some cases, these spurious signals should get eliminated since we are making an aggregate level statistics over a large population.

Footnote 4: [http://bit.ly/2ufOvea](http://bit.ly/2ufOvea)

**Future directions**: It would be interesting to understand and develop theoretical justification for the metrics. Further, it would be useful to study and classify various other linguistic phenomena closely related to core borrowing, such as: (i) _loanword_, where a form of a foreign word and its meaning or one component of its meaning gets borrowed, (ii) _calques_, where a foreign word or idiom is translated into existing words of native language, and (iii) _semantic loan_, where the word in the native language already exists but an additional meaning is borrowed from another language and added to existing meaning of the word.

Finally, we would also like to incorporate our findings into other standard tasks of multilingual IR and multilingual speech synthesis (for example to render the appropriate native accent to the borrowed word).

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline Word-rank & Context & \(\mu_{E-II}\) & \(\sigma_{E-II}\) \\ \hline \(TOP\) & \(H_{all}\) & **0.91** & 0.15 \\ \hline \(TOP\) & \(H_{med}\) & **0.85** & 0.23 \\ \hline \(MID\) & \(H_{all}\) & 0.58 & 0.28 \\ \hline \(MID\) & \(H_{max}\) & 0.61 & 0.34 \\ \hline \(BOT\) & \(H_{all}\) & 0.13 & 0.18 \\ \hline \(BOT\) & \(H_{meas}\) & 0.16 & 0.21 \\ \hline \end{tabular}
\end{table}
Table 9: Re-annotation results 



## References

* Auer (1984) P. Auer. 1984. _Bilingual Conversation_. John Benjamins.
* Bali et al. (2014) K. Bali, J. Sharma, M. Choudhury, and Y. Vyas. 2014. "i am borrowing ya mixing?" an analysis of english-h indi' code mixing in facebook. In _First workshop on Computational approaches to code-switching, EMNLP_, page 116.
* Bamman and Crane (2011) David Bamman and Gregory Crane. 2011. Measuring historical word sense variation. In _Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries_, pages 1-10. ACM.
* Barman et al. (2014) Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. _EMNLP 2014_, 13.
* Bock (2013) Z. Bock. 2013. Cyber socialising: Emerging genres and registers of intimacy among young south african students. _Language Matters: Studies in the Languages of Africa_, 44(2):68-91.
* Carter et al. (2013) S. Carter, W. Weerkamp, and M. Tsagkias. 2013. Micro-blog language identification: Overcoming the limitations of short, unedited and idiomatic text. _Language Resources and Evaluation_, 47(1):195-215.
* Cook et al. (2014) Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel word-sense identification. In _COLING_, pages 1624-1635.
* Das and Gamback (2013) A. Das and B. Gamback. 2013. code-mixing in social media text: The last language identification frontier? _TAL_, 54(3):41-64.
* Das and Gamback (2014) A. Das and B. Gamback. 2014. Identifying languages at the word level in code-mixed indian social media text. In _ICON_, pages 169-178.
* Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. _Psychological bulletin_, 76(5):378.
* Frermann and Lapata (2016) Lea Frermann and Mirella Lapata. 2016. A bayesian model of diachronic meaning change. _Transactions of the Association for Computational Linguistics_, 4:31-45.
* Gella et al. (2013) S. Gella, J. Sharma, and K. Bali. 2013. Query word labeling and back transliteration for indian languages: Shared task system description. _FIRE Working Notes_, 3.
* Hadei (2016) M. Hadei. 2016. Single word insertions as code-switching or established borrowing? _International Journal of Linguistics_, 8(1):14-25.
* Hidayat (2012) T. Hidayat. 2012. An analysis of code switching used by facebookers (a case study in a social network site). In _BA thesis_. English Education Study Program, College of Teaching and Education (STKIP), Bandung, Indonesia.
* Jurgens et al. (2017) David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. _ACL 2017_.
* Li (2009) D. C. S. Li. 2009. Cantonese-english code-switching research in hong-kong: a y2k review. _World Englishes_, 19(3):305-322.
* Mitra et al. (2014) Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That's sick dude!: Automatic identification of word sense change across different timescales. _arXiv preprint arXiv:1405.4392_.
* Muysken (1996) P. Muysken. 1996. Code-switching and grammatical theory. In _One speaker, two languages: Crossdisciplinary perspectives on code-switching_, pages 177-198. Cambridge University Press.
* Myers-Scotton (2002) C. Myers-Scotton. 2002. _Contact linguistics: Bilingual encounters and grammatical outcomes_. Oxford University Press.
* Negron (2009) R. G. Negron. 2009. Spanish-english code-switching in email communication. _Langage@Internet_, 6(3).
* Nzai et al. (2014) V. E. Nzai, Y-L. Feng, M. R. Medina-Jimenez, and J. Ekiaka-Ohazamemago. 2014. Understanding code-switching & word borrowing from a pluralistic approach of multilingualism.
* Peirsman et al. (2010) Yves Peirsman, Dirk Geeraerts, and Dirk Speelman. 2010. The automatic identification of lexical variation between language varieties. _Natural Language Engineering_, 16(4):469-491.
* Rijhwani et al. (2017) Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on twitter with a novel generalized word-level language detection technique. _ACL 2017_.
* Rudra et al. (2016) Koustav Rudra, Shruti Rijhwani, Rafiya Begum, Kalika Bali, Monojit Choudhury, and Niloy Ganguly. 2016. Understanding language preference for expression of opinion and sentiment: What do hindinguish speakers do on twitter? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1131-1141.
* Sagi et al. (2009) Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2009. Semantic density analysis: Comparing word meaning across time and phonetic space. In _Proceedings of the Workshop on Geometrical Models of Natural Language Semantics_, pages 104-111. Association for Computational Linguistics.
* San (2009) H. K. San. 2009. Chinese-english code-switching in blogs by macao young people. In _MSc. thesis_. University of Edinburgh.
* Sankoff et al. (1990) D. Sankoff, S. Poplack, and S. Vannirarajan. 1990. The case of the nonce loan in tamil. _Language variation and change_, 2(01):71-101

R. Y. Sebonde. 2014. Code-switching or lexical borrowing: Numerals in chasu language of rural tanzania. _Journal of Arts and Humanities_, 3(3):67.
* Senaratne [2013] C D. W. Senaratne. 2013. Borrowings or code mixes: The presence of lone english nouns in mixed discourse.
* Solorio et al. [2014] T. Solorio, E. Blair, S. Maharajan, S. Bethard, M. Diab, M. Gohneim, A. Hawwari, F. AlGhamdi, J. Hirschberg, A. Chang, and P. Fung. 2014. Overview for the first shared task on language identification in code-switched data. In _EMNLP First Workshop on Computational Approaches to Code Switching_, pages 62-72.
* Solorio and Liu [2008] T. Solorio and Y. Liu. 2008. Part-of-speech tagging for english-spanish code-switched text. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1051-1060. Association for Computational Linguistics.
* Sotillo [2012] S. Sotillo. 2012. Ehhhh utede hacen plane sin mi???:@ im feeling left out :( form, function and type of code switching in sms texting. In _ICAME_, pages 309-310.
* Thomason [2003] S. G. Thomason. 2003. Contact as a source of language change. _The handbook of historical linguistics_, pages 687-712.
* Vyas et al. [2014] Y. Vyas, S. Gella, J. Sharma, K. Bali, and M. Choudhury. 2014. POS tagging of english-hindi code-mixed social media content. In _EMNLP_, volume 14, pages 974-979.
* Zar [1972] J. H. Zar. 1972. Significance testing of the spearman's rank correlation coefficient. _Journal of the American Statistical Association_, 67(339):578-580.