Venue: NAACL 2016

# Improve Chinese Word Embeddings by Exploiting Internal Structure

Jian Xu, Jiawei Liu, Liangang Zhang, Zhengyu Li, Huanhuan Chen

Department of Computer Science, University of Science and Technology of China, China {jianxu1,ustcljw,liangang, lzy0503}@mail.ustc.edu.cn

hchen@ustc.edu.cn

Corresponding author

###### Abstract

Recently, researchers have demonstrated that both Chinese word and its component characters provide rich semantic information when learning Chinese word embeddings. However, they ignored the semantic similarity across component characters in a word. In this paper, we learn the semantic contribution of characters to a word by exploiting the similarity between a word and its component characters with the semantic knowledge obtained from other languages. We propose a similarity-based method to learn Chinese word and character embeddings jointly. This method is also capable of disambiguating Chinese characters and distinguishing non-compositional Chinese words. Experiments on word similarity and text classification demonstrate the effectiveness of our method.

## 1 Introduction

Distributed representations of knowledge has received wide attention in recent years. Researchers have proposed various models to learn it at different granularity levels. Distributed word representations, also known as word embeddings, were learned in Rumelhart et al. (1988); Bengio et al. (2006); Mnih and Hinton (2009); Mikolov et al. (2013a). Larger granularity levels than words have also been investigated, including phrase level Socher et al. (2010); Zhang et al. (2014); Yu and Dredze (2015), sentence level Le and Mikolov (2014); Socher et al. (2013); Kalchbrenner et al. (2014); Kiros et al. (2015), and document level Le and Mikolov (2014); Hermann and Blunsom (2014); Srivastava et al. (2013).

For language like Chinese, some smaller units than word also provide rich semantic information. For example, Chinese characters in word, Chinese radicals in character. These internal structures have been proved to be useful for Chinese word and character embeddings Chen et al. (2015); Li et al. (2015). Chen et al. (2015) took Chinese characters in a word into account when modeling the semantic meaning of the word. They proposed a character-enhanced word embeddings model (CWE) by adding the embedding of component characters in a word with the same weight to the word embedding. However, the internal characters in a Chinese word have different semantic contributions to its meaning. Take Chinese word "\(\frac{\text{th}}{\text{th}}\)" (frog) as an example. The character "\(\frac{\text{th}}{\text{th}}\)" (blue or green) is to decorate character "\(\frac{\text{th}}{\text{th}}\)" (frog). It is obvious that the latter character contributes more than the former one to the word meaning. In Li et al. (2015), they proposed a component-enhanced Chinese character embeddings model based on the feature that most Chinese characters are mono-semantic compounds. They considered characters and bi-characters as the basic embedding units. However, some bi-characters are meaningless, and may not form a Chinese word. These bi-characters may undermine embeddings of others.

This paper, motivated by Chen et al. (2015), exploits the internal structures of Chinese word, namely the Chinese characters. We propose a method to calculate the semantic contribution of characters to a word in a cross-lingual manner. The basic

idea is that the semantic contribution of Chinese characters in most Chinese words can be learned from their translations in other languages. Such as the word "\(\cancel{F}\)" we mentioned above. The word embeddings of other languages are used to calculate semantic contribution of characters to the word they compose. Moreover, Chinese character-s are more ambiguous than words. To tackle this problem, multiple-prototype character embeddings is proposed. Different meanings of characters will be represented by different embeddings. Our contributions can be summarized as follows:

1. We provide a method to calculate the semantic contribution of Chinese characters to the word they compose with English translation. Compared with English, there are fewer human-made resources to supervise the learning process of Chinese word and character embeddings. While translation resources are always easy to be accessed on the Internet.
2. We propose a novel way to disambiguate Chinese characters with translating resources. There are some limitations in existing cluster-based algorithms Huang et al. (2012); Neelakantan et al. (2015); Chen et al. (2015). They either fixed the number of clusters or proposed a nonparametric way to learn it for each word. However, the number of clusters for words varies a lot. For nonparametric method, different hyperparameters have to be tune to control the number of clusters for different datasets.
3. We provide a method to distinguish whether a Chinese word is semantically compositional automatically. Not all Chinese words exhibit semantic compositions from their component characters. For example, entity names, transliterated words like "\(\not\Downarrow\) %" (sofa), single-morpheme multi-character words like "\(\cancel{H}\)" (wander). In Chen et al. (2015), they performed part-of-speech tagging to identify entity names. The transliterated words are tagged manually, which requires human work and need to be updated when new words are created.

The evaluations on word similarity, text classification, Chinese characters disambiguation, and qualitative analysis of word embeddings demonstrate the effectiveness of our method.

## 2 Related Work

### Word2vec

Word2vec Mikolov et al. (2013a) is an algorithm to learn distributed word representations using a neural language model. Word2vec has two models, the continuous bag-of-words model (CBOW) and the skip-gram model. In this paper, we propose a new model based on the CBOW, hence we focus attention on it. CBOW aims at predicting the target word given context words in a slide window. Given a word sequence \(D=\{x_{1},x_{2},\ldots,x_{T}\}\), the objective of CBOW is to maximize the average log probability

\[L=\frac{1}{T}\sum_{i=1}^{T}\log p(x_{i}|x_{i-j}^{i+j}), \tag{1}\]

where \(x_{i-j}^{i+j}\) is the context words centered at \(x_{i}\), \(p(x_{i}|x_{i-j}^{i+j})\) is defined as:

\[\frac{\exp({v_{x_{i}}^{{}^{\prime}}}^{T}\sum_{-j\leq k\leq j,k\neq 0}v_{ x_{i+k}})}{\sum_{x=1}^{W}\exp({v_{x}^{{}^{\prime}}}^{T}\sum_{-j\leq k\leq j,k \neq 0}v_{x_{i+k}})}, \tag{2}\]

where \(v_{x_{i}}\) and \(v_{x_{i}}^{{}^{\prime}}\) are the input and output vector representations of word \(x_{i}\). Since the size of English vocabulary \(W\) may be up to \(10^{6}\) scale, hierarchical softmax and negative sampling Mikolov et al. (2013b) are applied during training to learn the model efficiently. However, using CBOW to learn Chinese word embeddings directly may have some limitations. It fails to capture the internal structure of words. In Botha and Blunsom (2014); Luong et al. (2013); Trask et al. (2015); Chen et al. (2015), they demonstrated the usefulness to exploit the internal structure of words, and proposed some morphological-based methods. For example, Chen et al. (2015) exploit the internal structure in Chinese words.

### The CWE model

The basic idea of CWE is that both external context words and internal component characters in words provide rich information in modeling the semantic meaning of the target word. In CWE, they learned word embeddings with its component characters embeddings. Let \(C\) denotes the Chinese characters set, and the word \(x_{t}\) in context \(x_{i-j}^{i+j}\) is composed by 

several characters in \(C\), let \(x_{t}=\{c_{1},c_{2},\ldots,c_{N_{t}}\}\), \(c_{k}\) denotes the \(k\)-th character in \(x_{t}\),

\[\widehat{v}_{x_{t}}=v_{x_{t}}+\frac{1}{N_{t}}\sum_{k=1}^{N_{t}}v_{c_{k}}, \tag{3}\]

where \(\widehat{v}_{x_{t}}\) is the modified word embedding, \(N_{t}\) denotes the number of Chinese characters in \(x_{t}\). To address the issue of ambiguity in Chinese characters, they proposed several approaches for multiple-prototype character embeddings: position-based, cluster-based, nonparametric methods, and position-cluster-based character embeddings. These methods are denoted as CWE+P, CWE+L, CWE+N, CWE+LP respectively. However, this model has some limitations. The internal characters are of the same contribution to the semantic meaning of the word in CWE, which is not the case for most Chinese words.

## 3 Methodology

Our method can be described as three stages:

* **Obtain translations of Chinese words and characters** Chinese words segmentation tool is used to segment words in Chinese corpus. Then we use an online English-Chinese translation tool to translate all the Chinese characters and segmented words.
* **Perform Chinese character sense disambiguation** We train an English corpus with CBOW to get English word embeddings. Then, we merge some meanings of Chinese characters with small difference, and disambiguate the meanings of characters in words by computing the similarity between their English translation words.
* **Learn word and character embeddings with our model** Based on the character sense disambiguation process, we modify the objective of CWE to learn Chinese word and character embeddings. Then we analyse the complexity of our model briefly.

### Obtain translations of Chinese words and characters

We use segmentation tools to segment words in Chinese training corpus, and perform part-of-speech tagging to recognize all the entity names. Since entity name words do not exhibit semantic compositions, they are identified as non-compositional words. We count the times of characters appearing in different words. Words with Chinese characters rarely combined with other characters are classified as single-morpheme multi-character words and identified as non-compositional.

Then programming interface of online translation tool is used to translate Chinese words and characters into English. For non-compositional Chinese words, they are not included in the translation list. Table 1 shows the English meanings of Chinese word "X", "Y" and their component characters "X" and "X", "Y" and "X".

### Perform Chinese character sense disambiguation

We train an English corpus with CBOW to get English word embeddings. Then, the meanings of characters with small difference are merged.

In Table 1, we observe that the difference between some meanings of character "X" is very small, some of them differ only in their part-of-speech. In Chinese, the same characters and words are used in different part-of-speech but express the same semantic meaning. Hence these meanings are merged as one semantic meaning. Let \(\mathrm{Sim}(\cdot)\) denotes the function to calculate the similarity between meanings of Chinese words and characters, we use cosine distance as the distance metric. The \(i\)-th and \(j\)-th meanings of Chinese character \(c\) are \(c^{i}\) and \(c^{j}\). Their similarity is defined as:

\[\begin{array}{rl}\mathrm{Sim}(c^{i},c^{j})=\max(cos(v_{x_{m}},v_{x_{n}})),\\ s.t.&x_{m}\in\mathrm{Trans}(c^{i}),x_{n}\in\mathrm{Trans}(c^{j}),\\ &x_{m},x_{n}\notin\text{stop\_words}(\mathrm{en}),\end{array} \tag{4}\]

where \(\mathrm{Trans}(c^{i})\) denotes the English translation words set of \(c^{i}\), \(\text{stop\_words}(\mathrm{en})\) denotes the stop words in English, \(x_{m}\) and \(x_{n}\) are not in these stop words. For example, the Chinese word "X" in Table 1, \(c_{2}\) denotes the second character 

"\(\frac{c^{i}_{2}}{K}\)" in the word. \(\mathrm{Trans}(c_{2}^{3})\) is the third translation English words set of character "\(\frac{c}{K}\)", which is \(\{pleasure,enjoyment\}\). Therefore \(x_{m}\) can be pleasure or enjoyment here.

If the \(\mathrm{Sim}(c^{i},c^{j})\) is above a threshold \(\delta\), then they are merged as one semantic meaning. For simplicity, we use the union of English translation words set. One character may be translated into several English words. We may average all the translation word embeddings and then compute the similarity, or select t the maximum value of the similarity between all English word pairs. In our experiments, maximum method works better.

Finally, we perform Chinese character sense disambiguation. In Chinese, characters may have multiple meanings, but for a certain word, their meanings are determined. For exmaple, the word "\(\frac{\text{``$\times$}}{\text{FI}}\)", the English translation is music. For character "\(\frac{\text{``$\times$}}{\text{FI}}\)", the first translation "music" matches the meaning of the word. For character "\(\frac{\text{``$\times$}}{\text{FI}}\)", the best match is the first translation "sound". For transliterated word like "\(\frac{\text{``$\times$}}{\text{F}}\)", the English translations are sofa and setete, neither sofa nor setete have high similarity with English translation words of character "\(\frac{\text{``$\times$}}{\text{F}}\)" and character "\(\frac{\text{``$\times$}}{\text{F}}\)". Formally, if \(\max(\mathrm{Sim}(x_{t},c_{k}))>\lambda,c_{k}\in x_{t}\), then \(x_{t}\) is identified as compositional word, and belongs to the compositional set \(\mathrm{COMP}\). For compositional words, we build a set

\[F=\{(x_{t},s_{t},n_{t})\mid x_{t}\in\mathrm{COMP}\}, \tag{5}\]

where

\[\begin{split}& s_{t}=\{\mathrm{Sim}(x_{t},c_{k})\mid c_{k}\in x_{t }\},\\ & n_{t}=\{\underset{i}{\max}\,\mathrm{Sim}(x_{t},c_{k}^{i})\mid c_{ k}\in x_{t}\}\end{split} \tag{6}\]

For example, the word "\(\frac{\text{``$\times$}}{\text{FI}}\)" is defined as ("\(\frac{\text{``$\times$}}{\text{FI}}\)", \(\{\mathrm{Sim}(\text{``$\times$}}{\text{FI}}\)", "\(\frac{\text{``$\times$}}{\text{FI}}\)"), \(\mathrm{Sim}(\text{``$\cdot$}{\text{FI}}\)", "\(\frac{\text{``$\times$}}{\text{FI}}\)"), \(\mathrm{1}\),\(\mathrm{1}\)) in \(F\).

### Learn word and character vectors with SCWE

The internal characters in a word make different contributions to its semantic meaning. However, in Chen et al. (2015), the contribution of component characters to the semantic meaning of word are treated equally. They add character embeddings to the word embeddings with the same weight, which may undermine the quality of word embeddings. Based on this point, we propose a similarity-based character-enhanced word embedding model, which takes the contribution of characters into account. We name it SCWE for ease of reference in the later part. The architecture of CWE and SCWE are shown in Fig. 1.

**Similarity-Based Character-Enhanced word Embedding** In the character sense disambiguation stage, we build a set \(F\), which contains compositional words, the similarity between words and its component characters, and the meaning order number of characters in the word. Suppose \(x_{t}\) in \(W\) is a

\begin{table}
\begin{tabular}{p{48.4pt}|l} \hline Word & English Explanation \\ \hline 叏 & music; \\ \hline � & (\(\nexists\)) sound; (\(\nexists\)) news, tidings; (\(\nexists\)) tone; (\(\nexists\)) a surname; \\ \hline � & N. (\(\nexists\)) music; (\(\nexists\)) a surname; (\(\nexists\)) pleasure, enjoyment; JJ. (\(\nexists\)) happy,glad,joyful,cheerful; V. (\(\nexists\)) enjoy, be glad to, love, find pleasure in; (\%) laugh, he amused; RB. (\(\nexists\)) gladly, happily, willingly; \\ \hline � & sofa, setete; \\ \hline � & N. (\(\nexists\)) sand; (\(\nexists\)) granulated, powdered; (\(\nexists\)) a surname; JJ. (\(\nexists\)) (of voice) hoarse, husky; \\ \hline � & N. (\(\nexists\)) hair; V. (\(\nexists\)) send out, distribute, deliver; (\(\nexists\)) launch, discharge, shoot, emit; (\(\exists^{o}\)) produce, generate, come into existence; (\(\nexists\)) express, utter; (\(\nexists\))✗, \(\nexists\)) expand, develop; (\(\nexists\))✗, \(\nexists\)) flourish; (\(\nexists\))✗, \(\nexists\)) spread out, disperse, diffuse; etc.; \\ \hline \end{tabular}
\end{table}
Table 1: English Translation of Chinese words and characters in ICIBA. V., N., J.J., RB. denote their verb, noun, adjective and adverb meaning respectively. Different meanings of word and character are separated by semicolon.

 

compositional word, in SCWE,

\[\widehat{v}_{x_{t}}=\frac{1}{2}\big{\{}v_{x_{t}}+\frac{1}{N_{t}} \sum_{k=1}^{N_{t}}\mathrm{Sim}(x_{t},c_{k})v_{c_{k}}\big{\}} \tag{7}\]

To deal with ambiguity problem of Chinese characters, we propose multiple-prototype character embeddings and denote it as SCWE+M model. Since the meaning of a character is determined in a given word, we utilize the information provided by the last element in set \(F\), and use different character embeddings for different meanings of characters. Then, in SCWE+M,

\[\widehat{v}_{x_{t}}=\frac{1}{2}\big{\{}v_{x_{t}}+\frac{1}{N_{t}} \sum_{k=1}^{N_{t}}\mathrm{Sim}(x_{t},c_{k})v_{c_{k}^{i}}\big{\}} \tag{8}\]

**Complexity analysis** We analyze the complexities of CBOW, CWE, SCWE and SCWE+M. Let \(S\) denotes the size of corpus, \(|W|\) denotes the size of vocabulary, \(|C|\) denotes the number of Chinese characters in corpus. And \(d\) is the dimensions of Chinese word and character embeddings, \(k\) is the context window size, \(f\) is the time spend in computing hierarchical softmax or negative sampling, \(n\) is the average number of characters in a Chinese word, \(m\) is the average meaning number of Chinese characters. The results are shown in Table 2.

In Chinese, most of words are composed by t-wo Chinese characters, and the meaning number of commonly used characters are usually less than five. Moreover, according to _CJK Unified Ideographs1_, the total number of Chinese characters is 20913, the commonly used characters are less than 10000. Therefore, our model is competitive to other methods in model parameters and computational complexity.

Footnote 1: [https://en.wikipedia.org/wiki/CJK_Unified_Ideographs](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs)

Footnote 2: [http://download.wikipedia.com/enwiki/](http://download.wikipedia.com/enwiki/)

Footnote 3: [http://download.wikipedia.com/zhwiki/](http://download.wikipedia.com/zhwiki/)

Footnote 4: [https://github.com/NLPchina/ansj_seg](https://github.com/NLPchina/ansj_seg)

## 4 Experiments and Analysis

### Experiments Settings

We select _English Wikipedia Dump_2 to train English word embeddings with CBOW, and set dimensions to 200. For Chinese word embeddings, we select _Chinese Wikipedia Dump_3 to train character and word embeddings. Before training, pure digits and non-Chinese characters are removed. We use an open-source Chinese segment tool called _ANSJ4_ to

Figure 1: Architecture of models. The left is CWE and right is SCWE.

 

segment words in corpus. ANSJ is a java implementation of ICTCLAS (Institute of Computing Technology, Chinese Lexical Analysis System). It can process about one million words in a second, and get up to 96 percent accuracy in segmentation task. The part-of-speech tagging and name entity recognition tasks are also done in this process. We select _ICIBA5_ as English-Chinese translation tool, which provides us with an application programming interface. CBOW and CWE are used as baseline methods. Context window size is set as 5 and both Chinese word and character embeddings are set as 100 dimension. After some cross validation steps, our threshold \(\delta\) and \(\lambda\) are set as 0.5 and 0.4 in character disambiguation process. The influence of \(\lambda\) and \(\delta\) is report in the later part.

Footnote 5: [http://www.iciba.com/](http://www.iciba.com/)

### Word Similarity

Word similarity is a task to compute semantic relatedness between given word pairs. The relatedness between word pairs have been scored by human in advance. The correlation between model results and human judgement can be used to evaluate the performance of models. In this paper, wordsim-240 and wordsim-296 Jin and Wu (2012) are used as evaluation datasets. The Spearman's rank correlation Myers et al. (2010) is applied to compute the correlation. The experimental results are summarized in Table 3.

We observe that on wordsim-240, SCWE and SCWE+M outperform the baseline methods, which indicates the effectiveness of exploiting the internal structure. On dataset wordsim-296, we can see that CBOW, CWE, SCWE perform similarly. This may be explained by some highly ambiguous Chinese characters in this dataset. In SCWE and CWE, representing these ambiguous characters with the same embeddings may undermine word embeddings. Therefore, SCWE+M achieves a better performance by applying multiple-prototype character embeddings.

### Text Classification

In this experiment, we use _Fudan Corpus6_ as datasets, which contains 20 categories of documents, including economy, politics, sports and etc.. The number of documents in each category ranges from 27 to 1061. To avoid imbalance, we select 10 categories and organize them into 2 groups. One group is named _Fudan-large_ and each category in this group contains more than 1000 documents. The other is named _Fudan-small_ and each category contains less than 100 documents. In each category, 80 percent of documents are used as training set, the rest are used as testing set to evaluate the performance. The detailed information for two datasets are reported in Table 4.

Footnote 6: [http://www.datatang.com/data/44139](http://www.datatang.com/data/44139)

Similar to the way we deal with Chinese training corpus, pure digits and non-Chinese characters are removed and ANSJ is used to do word segmentation on these datasets. The publish information of each document is removed. We represent each document t by averaging word embeddings in the document. The classifiers are trained using LIBLINEAR packagefan et al. (2008) with the embeddings obtained from different methods. The performance of each method is evaluated by predicting accuracy on testing set. Experiment results are given in Table 5.

It is observed that our methods outperform the baseline methods on both datasets. This can be explained that the semantic relatedness of a word with the component characters which have more contribution to its semantic meaning is strengthen in our methods. Such as, in sports documents, the word

\begin{table}
\begin{tabular}{l|c c|c} \hline Fudan-large & Size & Fudan-small & Size \\ \hline Environment & 1218 & Education & 59 \\ Agriculture & 1022 & Philosophy & 44 \\ Economy & 1601 & Transport & 58 \\ Politics & 1025 & Medical & 52 \\ Sports & 1254 & Military & 75 \\ \hline \end{tabular}
\end{table}
Table 4: 2 groups datasets of text classification, the first column denotes the category of documents and the second denotes number of documents in each category.

\begin{table}
\begin{tabular}{c|c|c} \hline Model & wordsim-240 & wordsim-296 \\ \hline CBOW & 51.78 & 60.82 \\ \hline CWE & 52.57 & 60.36 \\ \hline SCWE & 54.92 & 60.85 \\ SCWE + M & **55.10** & **62.86** \\ \hline \end{tabular}
\end{table}
Table 3: Evaluation on wordsim-240 and wordsim-296 

"X\({}^{+}\)" (ball) is used frequently. For Chinese words like "X\({}^{-}\)" (basketball) and "Y\({}^{-}\)" (tennis), the character "X\({}^{-}\)" contributes more to their semantic meaning than other characters. Therefore, they lie closer to character "X\({}^{-}\)" in embedding space obtained by our model than CBOW and CWE, and tend to form a cluster in embedding space.

### Multiple Prototype of Chinese Charaters

To tackle the ambiguity of Chinese characters, we propose multiple-prototype character embeddings. To evaluate the effectiveness of our method, we use PCA to conduct dimensionality reduction on word and character embeddings. The results are illustrated in Fig 2. We take 3 different meanings of Chinese characters "X\({}^{-}\)X\({}^{-}\)" and "Y\({}^{-}\)", and 2 of their top-related words as examples. The character followed by a digit it denotes the \(i\)-th meanings of it.

We can observe that characters and words, which have similar meanings are gathered together. For example, "X\({}^{-}\)X\({}^{-}\)", "X\({}^{-}\)X\({}^{-}\)X\({}^{-}\)" and "X\({}^{-}\)X\({}^{-}\)X\({}^{-}\)" are all related to the light. Thus, they get closer in the embedding space.

We also develop a dataset to compare our method with the disambiguation methods in Chen et al. (2015). We select some ambiguous Chinese characters, and then use _online Xinhua Dictionary7_ as our standard to disambiguate the words that contain these ambiguous characters. Each word is assigned a number according to their explanation in the dictionary. We use KNN as classifier to evaluate all the methods. The results are shown in Table 7. It is observed that our method outperforms the methods proposed in Chen et al. (2015).

Footnote 7: [http://xh.5156edu.com/](http://xh.5156edu.com/)

\begin{table}
\begin{tabular}{c|p{84.1pt}} \hline Characters & Words \\ \hline X\({}^{-}\)X\({}^{ 



### Qualitative analysis of word embeddings

In this part, we take two Chinese words as examples, and list their nearest words to examine the quality of word embeddings obtained by CWE and SCWE. The results are shown in Table 8. We can observe the most similar words return by CWE and SCWE both tend to share common characters with the given word. In CWE, characters with little semantic contribution to the word may undermine the quality of word embeddings. For example, the character "\(\overline{\text{F}}^{*}\)" in word "\(\overline{\text{F}}^{*}\)". The semantic relatedness of words with character "\(\overline{\text{F}}^{*}\)" to the given word are overestimated in CWE. In our model, by calculating the semantic contribution of internal characters to the word, we alleviate this misjudgement greatly, which demonstrates the effectiveness of our model.

### Parameter Analysis

In this part, the influence of parameters on our model is investigated. The parameters include the compositional word similarity threshold \(\lambda\), character disambiguation threshold \(\delta\).

**Compositional word similarity** To investigate how \(\lambda\) influence the process of non-compositional word detection, we build a word list of transliterated words manually, which consists of 161 words. Then 161 of most frequent semantic compositional words with more than one Chinese characters are added to the list in the corpus. In Table 9, the performance of our method in classifying transliterated words when \(\lambda\) ranges from 0.25 to 0.55 are reported. From Table 9, we can observe as \(\lambda\) increases, more compositional words will be classified as non-compositional words, while transliterated words are more likely to be classified correctly. Our method achieves best F-Score when \(\lambda=0.4\).

**Character disambiguation threshold** In Table 10, we show the performance of our model in disambiguating Chinese characters. We adopted the same datasets in Section 4.4 with different \(\delta\). From Table 1, we can observe some meanings of a character are very close, therefore, a high \(\delta\) are adopted in our model. When \(\delta=0.5\), our model gets the best result in our dataset.

## 5 Conclusion

In this paper, we exploit the internal structure in Chinese words by learning the semantic contribution of internal characters to the word. We propose a method to improve Chinese word and character embeddings with a similarity-based character-enhanced word embeddings model. Ambiguity problem of Chinese characters can also be tackled in our method. Moreover, we build a way to classify whether a Chinese word is compositional

\begin{table}
\begin{tabular}{c|c} \hline Parameter \(\delta\) & Precision \\ \hline
0.35 & 87.5 \\ \hline
0.40 & 89.0 \\ \hline
0.45 & 89.5 \\ \hline
0.50 & **91.1** \\ \hline
0.55 & 89.9 \\ \hline
0.60 & 89.5 \\ \hline
0.65 & 88.5 \\ \hline \end{tabular}
\end{table}
Table 10: Evaluation accuracies (%) on ambiguous characters when \(\lambda\) ranges from 0.35 to 0.65.

\begin{table}
\begin{tabular}{l|p{142.3pt}|p{142.3pt}} \hline Words & CWE & SCWE \\ \hline  & [] (green snake) & [] \\  & [] (blue crab) & (Rana catesbeiana) \\ (frog) & [] \\  & [] (green pepper) & [] \\  & [] \\  & [] \\  & [] (xaba catesbeiana) & [] \\ \hline  & [] \\  & [] \\  & [] \\  & [] \\  & [] \\  & [] \\ \hline  & [] \\  & [] \\  & [] \\  & [] \\ \hline \end{tabular}
\end{table}
Table 8: Nearest words example of Chinese words.

\begin{table}
\begin{tabular}{c|c|c|c} \hline Parameters \(\lambda\) & Precision & Recall & F-Score \\ \hline
0.25 & 97.0 & 60.9 & 74.8 \\ \hline
0.30 & 96.5 & 68.9 & 80.4 \\ \hline
0.35 & 94.6 & 75.8 & 84.2 \\ \hline
0.40 & 92.0 & 78.9 & **85.0** \\ \hline
0.45 & 88.9 & 80.1 & 84.3 \\ \hline
0.50 & 84.0 & 84.5 & 84.2 \\ \hline
0.55 & 82.5 & 85.1 & 83.8 \\ \hline \end{tabular}
\end{table}
Table 9: Precision, recall, F-score of transliterated words when \(\lambda\) ranges from 0.25 to 0.55

automatically, which requires to be labelled manually in CWE. We argue that our method may be used to improve word embeddings of other language whose internal structure is similar to Chinese. The code and datasets we use is available at: [https://github.com/JianXul23/SCWE](https://github.com/JianXul23/SCWE).

## Acknowledgement

This work is supported by NSFC grants 91546116 and 61511130083.

## References

* [Bengio et al.2006] Yoshua Bengio, Holger Schwenk, Jean-Sebastien Senecal, Frederic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In _Innovations in Machine Learning_, pages 137-186. Springer.
* [Botha and Blunsom2014] Jan A Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. _arXiv preprint arXiv:1405.4273_.
* [Chen et al.2015] Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. _In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI)_.
* [Fan et al.2008] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Len Lin. 2008. Liblinear: A library for large linear classification. _The Journal of Machine Learning Research_, 9:1871-1874.
* [Hermann and Blunsom2014] Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. _arXiv preprint arXiv:1404.4641_.
* [Huang et al.2012] Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In _In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_, pages 873-882. Association for Computational Linguistics.
* [Jin and Wu2012] Peng Jin and Yunfang Wu. 2012. Semeval-2012 task 4: evaluating chinese word similarity. In _In Proceedings of the Sixth International Workshop on Semantic Evaluation_, pages 374-377. Association for Computational Linguistics.
* [Kalchbrenner et al.2014] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. _arXiv preprint arXiv:1404.2188_.
* [Kiros et al.2015] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Advances in Neural Information Processing Systems_, pages 3276-3284.
* [Le and Mikolov2014] Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. _arXiv preprint arXiv:1405.4053_.
* [Li et al.2015] Yanran Li, Wenjie Li, Fei Sun, and Sujan Li. 2015. Component-enhanced chinese character embeddings. _arXiv preprint arXiv:1508.06669_.
* [Luong et al.2013] Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. _CoNLL-2013_, 104.
* [Mikolov et al.2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* [Mikolov et al.2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* [Mnih and Hinton2009] Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. In _Advances in neural information processing systems_, pages 1081-1088.
* [Myers et al.2010] Jerome L Myers, Arnold Well, and Robert Frederick Lorch. 2010. _Research design and statistical analysis_. Routledge.
* [Neelakantan et al.2015] Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2015. Efficient non-parametric estimation of multiple embeddings per word in vector space. _arXiv preprint arXiv:1504.06654_.
* [ Rumelhart et al.1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1988. Learning representations by back-propagating errors. _Cognitive modeling_, 5:3.
* [Socher et al.2010] Richard Socher, Christopher D Manning, and Andrew Y Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In _In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop_, pages 1-9.
* [Socher et al.2013] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _In Proceedings of the conference on empirical methods in natural language processing (EMNLP)_, volume 1631, page 1642. Citeseer 

* [Srivastava et al.2013] Nitish Srivastava, Ruslan R Salakhutdinov, and Geoffrey E Hinton. 2013. Modeling documents with deep boltzmann machines. _arXiv preprint arXiv:1309.6865_.
* [Trask et al.2015] Andrew Trask, David Gilmore, and Matthew Russell. 2015. Modeling order in neural word embeddings at scale. _arXiv preprint arXiv:1506.02338_.
* [Yu and Dredze2015] Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. _Transactions of the Association for Computational Linguistics_, 3:227-242.
* [Zhang et al.2014] Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, and Chengqing Zong. 2014. Bilingually-constrained phrase embeddings for machine translation. In _In Proceedings of the 52th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics_.