Venue: EMNLP 2017

# Distinguishing Japanese Non-standard Usages from Standard Ones

 Tatsuya Aoki\({}^{\dagger}\)  Ryohei Sasano\({}^{\ddagger}\)   Hiroya Takamura\({}^{\dagger}\)    Manabu Okumura\({}^{\dagger}\)

\({}^{\dagger}\)Department of Information and Communications Engineering, Tokyo Institute of Technology

\({}^{\ddagger}\)Graduate School of Informatics, Nagoya University

{aoki,takamura,oku}@lr.pi.titech.ac.jp

sasano@i.nagoya-u.ac.jp

###### Abstract

We focus on non-standard usages of common words on social media. In the context of social media, words sometimes have other usages that are totally different from their original. In this study, we attempt to distinguish non-standard usages on social media from standard ones in an unsupervised manner. Our basic idea is that non-standardness can be measured by the inconsistency between the expected meaning of the target word and the given context. For this purpose, we use context embeddings derived from word embeddings. Our experimental results show that the model leveraging the context embedding outperforms other methods and provide us with findings, for example, on how to construct context embeddings and which corpus to use.

+
Footnote †: _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2323–2328 Copenhagen, Denmark, September 7–11, 2017. ©2017 Association for Computational Linguistics

## 1 Introduction

On social media such as Twitter, we often find posts that are difficult to interpret without prior knowledge on non-standard usage of words. For example, consider the following Japanese sentence1:

Footnote 1: This is interlinear-gloss text representation. POSS, NOM, PRS respectively represent the possessive case, nominative case, and present tense. The third line is the standard translation of the Japanese sentence.

\begin{tabular}{l l l} \#\

of the 40 words were computer terms, another 10 were company/service names, and the remaining 20 were other Internet slang. For each of the 40 target words, we found 100 tweets that contained the target word. Here, we used Twitter as the source for examples since there are many non-standard usages on it. To segment tweets into words, we used the Japanese morphological analyzer MeCab3 with the standard IPA standard dictionary.4

Footnote 3: [http://taku910.github.io/mecab/](http://taku910.github.io/mecab/)

Footnote 4: [https://ja.osdn.net/projects/ipadic/](https://ja.osdn.net/projects/ipadic/)

Next, we asked two human annotators to judge whether the usage of the target word in each tweet is standard, non-standard, a named entity, or undecidable. We excluded tweets which at least one annotator judged as undecidable (96 tweets).5 Cohen's kappa of the annotations for the remaining 3,904 tweets was 0.808. We further excluded tweets which at least one annotator judged as containing a named entity (772 tweets) in order to focus the dataset on our main purpose.6

Footnote 5: The undecidable tweets are meaningless expressions such as the emotion “(* ’ ’’)”, which includes the word “\(j\bar{l}\)”.

Finally, to create a final dataset, we selected from the remaining 3,132 tweets the 2,973 tweets that are judged as standard by both annotators or as non-standard by them. The selected 2,973 tweets are equivalent to 94.9% of the entire set of tweets, which suggests that human can reliably distinguish non-standard usages from standard ones. The statistics of the final dataset are shown in Table 2.

## 3 Methodology

Our basic idea for distinguishing word usages is that if a word is used in a non-standard manner, the context words around it will tend to differ from standard context words. To implement this idea, we employed word embeddings. Below, we review the Skip-gram model used for obtaining the word embeddings in Section 3.1 and present our method in Section 3.2.

### Skip-gram

Skip-gram Mikolov et al. (2013) is widely used for obtaining word embeddings. Given a sequence of words \(w_{1}\), \(w_{2}\), ..., \(w_{T}\) as training data, Skip-gram maximizes the likelihood \(\frac{1}{T}\sum_{t=1}^{T}\sum_{-m\leq i\leq m,i\neq 0}\log p(w_{t+i}|w_{ t})\), where \(m\) is the window size. \(w_{t+i}\) is a context word nearby \(w_{t}\). \(p(w_{k}|w_{t})\) is given by

\[p(w_{k}|w_{t})=\frac{\exp(v_{w_{k}}^{IN}\cdot v_{w_{k}}^{OUT})}{\sum_{w=1}^{W }\exp(v_{w_{k}}^{IN}\cdot v_{w}^{OUT})},\]

where \(W\) is the vocabulary size of the training data. Skip-gram learns a model predicting context words using word embeddings \(v^{IN}\) and \(v^{OUT}\), which are called _input_ embedding and _output_ embedding respectively.

The embeddings are learned in such a way that \(v_{w_{t}}^{IN}\cdot v_{w_{k}}^{OUT}-\log\sum_{w=1}^{W}\exp(v_{w_{t}}^{IN}\cdot v _{w}^{OUT})\) increases if word \(w_{t}\) occurs near \(w_{t}\) in the training corpus. As a result, \(v_{w_{t}}^{IN}\cdot v_{w_{k}}^{OUT}\) tends to be large for such words and small for word pairs that do not co-occur in the training corpus. We exploited this tendency for recognizing non-standard usages; if the dot-product between the embeddings of the target word and the context words is small, it should indicate a non-standard usage, on the condition that the embeddings have been learned on a general balanced corpus where words correspond to their standard meanings in most cases.

\(v^{IN}\) is widely used as a _word embedding_ in many studies, while \(v^{OUT}\) has not been in the limelight; only a few researchers have examined the effectiveness of \(v^{OUT}\)Mitra et al. (2016); Press and Wolf (2017). In recent studies, embeddings \(v^{IN}\) are usually used for measuring the similarity between words. However, given the characteristics described in the previous paragraph and SGNS's equivalence with shifted positive pointwise mutual information Levy and Goldberg (2014), if we want to measure to what extent word \(w_{t}\) tends to co-occur with \(w_{k}\) in the training data, then we should use the similarity of \(v_{w_{t}}^{IN}\cdot v_{w_{k}}^{OUT}\), instead of \(v_{w_{t}}^{IN}\cdot v_{w_{k}}^{IN}\).

In this study, we show the importance of using \(v^{OUT}\) in a task where we need to see if a word matches its context.

\begin{table}
\begin{tabular}{c|r r} \hline \multirow{2}{*}{ Category} & \multicolumn{2}{c}{Usage} \\ \cline{2-3}  & standard & non-standard \\ \hline \hline Computer terms & 416 & 234 \\ Company/Service names & 440 & 252 \\ Other Internet slang & 817 & 814 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the dataset.

 



### Distinguishing Non-standard Usages from Standard Ones

Following the idea described in Section 3.1, we propose a method for distinguishing non-standard usages from standard ones by leveraging word embeddings. An overview of our method is shown in Figure 1. We use Skip-gram with Negative Sampling (SGNS) Mikolov et al. (2013) for obtaining the word embeddings.

Given a target word \(w_{t}\) and its context \(\mathbf{w}_{c}\) as input, we calculate the following weighted average of scaled dot-products as a measure of standardness:

\[\frac{\sum_{w_{j}\in\mathbf{w}_{c}}\sigma(v^{IN}_{w_{t}}\cdot v^{OUT}_{w_{j}}) \times\alpha_{w_{j}}}{\sum_{w_{j}\in\mathbf{w}_{c}}\alpha_{w_{j}}}, \tag{1}\]

where \(v^{IN}_{w_{t}}\) is the input embedding for the target word \(w_{t}\) and \(v^{OUT}_{w_{i}}\) is the output embedding for the context word \(w_{j}\). \(\alpha_{w_{j}}\) is a non-negative weight for the word \(w_{j}\), and \(\sigma\) is the sigmoid function used for scaling dot-products into a range from 0 to 1. Although the values of \(\alpha_{w_{j}}\) are arbitrary, we will use the values given by the training algorithm used in word2vec7 and gensim Rehurek and Sojka (2010), popular tools for obtaining word embeddings. In their training of word embeddings, context words that are closer to the target word are weighted higher.8 We therefore set \(\alpha_{w_{j}}\) to be \(m+1-d_{w_{j}}\), where \(m\) is the window size and \(d_{w_{j}}\) is an integer that represents the distance between \(w_{j}\) and the target word. Hence, this is a decaying weighting. In contrast, with uniform weights, we set \(\alpha_{w_{j}}\) to be 1 for all \(w_{j}\) in the context. We call the score of Equation (1) _standardizedness_. If the standardness is low, our method regards the instance as non-standard; otherwise, our method regards it as standard. We should note again that, in our method, word embeddings should be learned on a general balanced corpus that is different from the domain of the target instances.

Footnote 7: code.google.com/archive/p/word2vec/

Footnote 8: This weighting scheme is mentioned in Levy et al. (2015).

## 4 Experiment

### Methods for Comparative Evaluation

Our model has three characteristics: (input and output) word embeddings, decaying weights, and a general balanced corpus. We evaluated each of these characteristics in a task distinguishing non-standard usages from standard ones.

First, we verified the effectiveness of the input and output embeddings. We tested a method in which only input embeddings are used to calculate the similarity: the cosine similarity between \(v^{IN}_{w_{t}}\) and \(v^{IN}_{w_{j}}\) instead of \(\sigma(v^{IN}_{w_{t}}\cdot v^{OUT}_{w_{j}})\), which is a similar framework to that of previous work Neelakantan et al. (2014); Gabriel et al. (2016). We then tested a method based on the positive pointwise mutual information (PPMI) Levy et al. (2015); Hamilton et al. (2016). Here, suppose that \(M\) is a matrix in which each element is a PPMI of words \(w_{i}\) and \(w_{j}\). \(v^{IN}_{w_{t}}\cdot v^{OUT}_{w_{j}}\) in Equation (1) is replaced with the (_t,j_)-element of the low-rank approximation of \(M\) obtained through singular value decomposition (SVD). We refer to this model as SVD.



Next, we replaced the decaying weights \(\alpha\) with uniform weights to examine the impact of decaying weights.

Finally, we conducted experiments with different training corpora to examine the impact of the balanced corpus. We used four corpora as training data for obtaining word embeddings. These corpora are described in Table 2.

### Experimental Settings

In the training of the word embeddings, we set the window size to 5, and the dimensions of the word embeddings to 300. We regarded the words with frequency counts of 5 or less in the training data as unknown words and replaced those words with "<unk>". We used gensim[14] as an implementation of SGNS, where we set the number of negative samples to 10. We used the code provided by Levy et al. levy:2015:EMNLP as the SVD implementation. For the evaluations, we ranked test instances in ascending order of standardness score and evaluated the ranking in terms of the area under the ROC curve (AUC)[6].

### Results

Table 3 shows the AUC for each model.13 First, we examined the impact of the choice of training corpus for obtaining word embeddings. The models with BCCWJ are constantly better than those with other corpora, although BCCWJ is smaller than the others (Table 2). This result suggests that use of a balanced corpus is crucial in our method for this task.

Footnote 9: The Balanced Corpus of Contemporary Written Japanese [7].

Footnote 10: Japanese sentences are collected using the method described in [8].

Next, we examined the impact of context embeddings. Table 3 shows that our model (SGNS IN-OUT) with BCCWJ achieved the best AUCs (.875 and .870), better than the AUCs of SGNS IN-IN with BCCWJ (.846 and .837). This result suggests that input embeddings should be used in combination with output embeddings for the task of judging whether a word matches its context or not. Table 3 also shows that SGNS-based models are better than SVD-based models.

As we discussed in Section 3.2, we used two weighting schemes for each model. Although the AUC of each decaying weight model is larger than that of the corresponding uniform weight model, the differences were not statistically significant.

## 5 Related Work

The previous studies focused on distinguishing non-standard usages that are multi-word expressions or idiomatic expressions [15; 16]. The task of this research is similar to new sense detection [11]. Our research target includes jargon, whose actual meaning is difficult to infer without specific knowledge about its usage [10]. Recent studies in computational linguistics have used word embeddings and other techniques to capture various semantic changes in words, such as diachronic changes, geographical variations, and sentiment changes [14; 15; 16; 17; 18; 19].

A few researchers have exploited output embeddings for natural language applications such as document ranking [14] and improving language models [20].

\begin{table}
\begin{tabular}{c|l|r|r} \hline Corpus & Description & \#word & \#token \\ \hline \hline BCCWJ9  & Japanese Balanced & 131,913 & 1.1b \\  & Corpus & & \\ \hline Web10  & Sentences randomly picked from the Web & & 6.0b \\ \hline Wikipedia11  & Japanese Wikipedia & 1,081,154 & 8.9b \\ \hline Newspaper12  & Japanese Newspapers & 1,204,914 & 15.0b \\ \hline \end{tabular}
\end{table}
Table 2: Description of corpora.

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline corpus & SGNS IN-OUT & SGNS IN-IN & & \multicolumn{2}{c}{SVD} \\ \cline{2-7}  & decay & uni & decay & uni & decay & uni \\ \hline \hline BCCWJ & **.875** & **.870** & 846 & .837 & .821 & .813 \\ Web & 846 & **.842** & 817 & 807 & .771 & 765 \\ Wikipedia & .827 & .821 & .824 & .805 & .739 & .732 \\ Newspaper & .844 & .839 & .825 & .810 & .770 & .764 \\ \hline \end{tabular}
\end{table}
Table 3: Area under the ROC curve (AUC) in usage classification task for each model.

 



## 6 Conclusion

We presented a model that uses context embeddings to distinguish Japanese non-standard usages from standard ones on social media. Our experimental results show that our model is better than the other models tested. They indicate the importance of context embeddings. To sum up, to distinguish non-standard usage, (1) using a balanced corpus as training data for obtaining word embeddings is crucial, (2) exploiting context embeddings derived from _input_ and _output_ word embeddings of SGNS achieves the best AUC, and (3) decaying weights have little impact on performance.

We are interested in expanding our method for detecting words that have non-standard usages. We are also interested in finding the meanings of the detected non-standard usages.

## References

* Cook et al. (2014) Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel word-sense identification. In _Proceedings of COLING '14_, pages 1624-1635.
* Davis and Goadrich (2006) Jesse Davis and Mark Goadrich. 2006. The relationship between precision-recall and roc curves. In _Proceedings of ICML '06_, pages 233-240.
* Eisenstein et al. (2010) Jacob Eisenstein, Brendan O'Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In _Proceedings of EMNLP '10_, pages 1277-1287.
* Frermann and Lapata (2016) Lea Frermann and Mirella Lapata. 2016. A bayesian model of diachronic meaning change. _Transactions of the Association for Computational Linguistics_, 4:31-45.
* Gharbieh et al. (2016) Waseem Gharbieh, Bhavsar Virendra, and Paul Cook. 2016. A word embedding approach to identifying verb-noun idiomatic combinations. In _Proceedings of the 12th Workshop on Multiword Expressions_, pages 112-118.
* Hamilton et al. (2016) William L. Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016. Inducing domain-specific sentiment lexicons from unlabeled corpora. In _Proceedings of EMNLP '16_, pages 595-605.
* Huang and Riloff (2010) Ruihong Huang and Ellen Riloff. 2010. Inducing domain-specific semantic class taggers from (almost) nothing. In _Proceedings of ACL '10_, pages 275-285.
* Kawahara and Kurohashi (2006) Daisuke Kawahara and Sadao Kurohashi. 2006. Case frame compilation from the web using highperformance computing. In _Proceedings of LREC '06_, pages 1344-1347.
* Kiela and Clark (2013) Douwe Kiela and Stephen Clark. 2013. Detecting compositionality of multi-word expressions using nearest neighbours in vector space models. In _Proceedings of EMNLP '13_, pages 1427-1432.
* Kulkarni et al. (2015) Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In _Proceedings of WWW '15_, pages 625-635.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In _Proceedings of NIPS '14_, pages 2177-2185.
* Levy et al. (2015) Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. _Transactions of the Association for Computational Linguistics_, 3:211-225.
* Li and Sporleder (2010) Linlin Li and Caroline Sporleder. 2010. Linguistic cues for distinguishing literal and non-literal usages. In _Proceedings of COLING '10_, pages 683-691.
* Maekawa et al. (2010) Kikuo Maekawa, Makoto Yamazaki, Takehiko Maruyama, Masaya Yamaguchi, Hideki Ogura, Wakako Kashino, Toshinobu Ogiso, Haane Koiso, and Yasuharu Den. 2010. Design, compilation, and preliminary analyses of balanced corpus of contemporary written Japanese. In _Proceedings of LREC '10_, pages 1483-1486.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proceedings of NIPS '13_, pages 3111-3119.
* Mitra et al. (2016) Bhaskar Mitra, Eric T. Nalisnick, Nick Craswell, and Rich Caruana. 2016. A dual embedding space model for document ranking. _arXiv: 1602.01137_.
* Mitra et al. (2014) Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That's sick dude!: Automatic identification of word sense change across different timescales. In _Proceedings of ACL '14_, pages 1020-1029.
* Neelakantan et al. (2014) Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient nonparametric estimation of multiple embeddings per word in vector space. In _Proceedings of EMNLP '14_, pages 1059-1069.
* Press and Wolf (2017) Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In _Proceedings of EACL '17_, pages 157-163.
* Rehurek and Sojka (2010) Radim Rehurek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In _Proceedings of Workshop on NewChallenges for NLP Frameworks_, pages 45-50.
* Salehi et al. (2015) Bahar Salehi, Paul Cook, and Timothy Baldwin. 2015. A word embedding approach to predicting the compositionality of multiword expressions. In _Proceedings of NAACL-HLT '15_, pages 977-983

Aleksandr Sboev. 2016. The sources of new words and expressions in the Chinese internet language and the ways by which they enter the internet language. In _Proceedings of PACLIC '16_, pages 355-361.
* Yang and Eisenstein (2016) Yi Yang and Jacob Eisenstein. 2016. Overcoming language variation in sentiment analysis with social attention. _arXiv:1511.06052_.