Chinese Embedding via Stroke and Glyph Information: A Dual-channel View

Abstract: Recent studies have consistently given positive hints that morphology is helpful in enriching word embeddings. In this paper, we argue that Chinese word embeddings can be substantially enriched by the morphological information hidden in characters which is reflected not only in strokes order sequentially, but also in character glyphs spatially. Then, we propose a novel Dual-channel Word Embedding (DWE) model to realize the joint learning of sequential and spatial information of characters. Through the evaluation on both word similarity and word analogy tasks, our model shows its rationality and superiority in modelling the morphology of Chinese.

Word embeddings are fixed-length vector representations for words BIBREF0 , BIBREF1 . In recent years, the morphology of words is drawing more and more attention BIBREF2 , especially for Chinese whose writing system is based on logograms.
UTF8gbsn With the gradual exploration of the semantic features of Chinese, scholars have found that not only words and characters are important semantic carriers, but also stroke feature of Chinese characters is crucial for inferring semantics BIBREF3 . Actually, a Chinese word usually consists of several characters, and each character can be further decomposed into a stroke sequence which is certain and changeless, and this kind of stroke sequence is very similar to the construction of English words. In Chinese, a particular sequence of strokes can reflect the inherent semantics. As shown in the upper half of Figure FIGREF3 , the Chinese character “驾" (drive) can be decomposed into a sequence of eight strokes, where the last three strokes together correspond to a root character “马" (horse) similar to the root “clar" of English word “declare" and “clarify".
Moreover, Chinese is a language originated from Oracle Bone Inscriptions (a kind of hieroglyphics). Its character glyphs have a spatial structure similar to graphs which can convey abundant semantics BIBREF4 . Additionally, the critical reason why Chinese characters are so rich in morphological information is that they are composed of basic strokes in a 2-D spatial order. However, different spatial configurations of strokes may lead to different semantics. As shown in the lower half of Figure 1, three Chinese characters “入" (enter), “八" (eight) and “人" (man) share exactly a common stroke sequence, but they have completely different semantics because of their different spatial configurations.
In addition, some biological investigations have confirmed that there are actually two processing channels for Chinese language. Specifically, Chinese readers not only activate the left brain which is a dominant hemisphere in processing alphabetic languages BIBREF5 , BIBREF6 , BIBREF7 , but also activate the areas of the right brain that are responsible for image processing and spatial information at the same time BIBREF8 . Therefore, we argue that the morphological information of characters in Chinese consists of two parts, i.e., the sequential information hidden in root-like strokes order, and the spatial information hidden in graph-like character glyphs. Along this line, we propose a novel Dual-channel Word Embedding (DWE) model for Chinese to realize the joint learning of sequential and spatial information in characters. Finally, we evaluate DWE on two representative tasks, where the experimental results exactly validate the superiority of DWE in capturing the morphological information of Chinese.
Traditional methods on getting word embeddings are mainly based on the distributional hypothesis BIBREF9 : words with similar contexts tend to have similar semantics. To explore more interpretable models, some scholars have gradually noticed the importance of the morphology of words in conveying semantics BIBREF10 , BIBREF11 , and some studies have proved that the morphology of words can indeed enrich the semantics of word embeddings BIBREF12 , BIBREF13 , BIBREF2 . More recently, Wieting et al. wieting2016charagram proposed to represent words using character n-gram count vectors. Further, Bojanowski et al. bojanowski2017enriching improved the classic skip-gram model BIBREF0 by taking subwords into account in the acquisition of word embeddings, which is instructive for us to regard certain stroke sequences as roots in English.
The complexity of Chinese itself has given birth to a lot of research on Chinese embedding, including the utilization of character features BIBREF14 and radicals BIBREF15 , BIBREF16 , BIBREF17 . Considering the 2-D graphic structure of Chinese characters, Su and Lee su2017learning creatively proposed to enhance word representations by character glyphs. Lately, Cao et al. cao2018cw2vec proposed that a Chinese word can be decomposed into a sequence of strokes which correspond to subwords in English, and Wu et al. wu2019glyce designed a Tianzige-CNN to model the spatial structure of Chinese characters from the perspective of image processing. However, their methods are either somewhat loose for the stroke criteria or unable to capture the interactions between strokes and character glyphs.
As we mentioned earlier, it is reasonable and imperative to learn Chinese word embeddings from two channels, i.e., a sequential stroke n-gram channel and a spatial glyph channel. Inspired by the previous works BIBREF14 , BIBREF18 , BIBREF4 , BIBREF19 , we propose to combine the representation of Chinese words with the representation of characters to obtain finer-grained semantics, so that unknown words can be identified and their relationship with other known Chinese characters can be found by distinguishing the common stroke sequences or character glyph they share.
UTF8gbsn Our DWE model is shown in Figure FIGREF9 . For an arbitrary Chinese word INLINEFORM0 , e.g., “驾车", it will be firstly decomposed into several characters, e.g., “驾" and “车", and each of the characters will be further processed in a dual-channel character embedding sub-module to refine its morphological information. In sequential channel, each character can be decomposed into a stroke sequence according to the criteria of Chinese writing system as shown in Figure FIGREF3 . After retrieving the stroke sequence, we add special boundary symbols INLINEFORM1 and INLINEFORM2 at the beginning and end of it and adopt an efficient approach by utilizing the stroke n-gram method BIBREF3 to extract strokes order information for each character. More precisely, we firstly scan each character throughout the training corpus and obtain a stroke n-gram dictionary INLINEFORM3 . Then, we use INLINEFORM4 to denote the collection of stroke n-grams of each character INLINEFORM5 in INLINEFORM6 . While in spatial channel, to capture the semantics hidden in glyphs, we render the glyph INLINEFORM7 for each character INLINEFORM8 and apply a well-known CNN structure, LeNet BIBREF20 , to process each character glyph, which is also helpful to distinguish between those characters that are identical in strokes.
After that, we combine the representation of words with the representation of characters and define the word embedding for INLINEFORM0 as follows: DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are compositional operation. INLINEFORM6 is the word ID embedding and INLINEFORM7 is the number of characters in INLINEFORM8 .
According to the previous work BIBREF0 , we compute the similarity between current word INLINEFORM0 and one of its context words INLINEFORM1 by defining a score function as INLINEFORM2 , where INLINEFORM3 and INLINEFORM4 are embedding vectors of INLINEFORM5 and INLINEFORM6 respectively. Following the previous works BIBREF0 , BIBREF21 , the objective function is defined as follows: DISPLAYFORM0 
where INLINEFORM0 is the number of negative samples and INLINEFORM1 is the expectation term. For each INLINEFORM2 in training corpus INLINEFORM3 , a set of negative samples INLINEFORM4 will be selected according to the distribution INLINEFORM5 , which is usually set as the word unigram distribution. And INLINEFORM6 is the sigmoid function.
We download parts of Chinese Wikipedia articles from Large-Scale Chinese Datasets for NLP. For word segmentation and filtering the stopwords, we apply the jieba toolkit based on the stopwords table. Finally, we get 11,529,432 segmented words. In accordance with their work BIBREF14 , all items whose Unicode falls into the range between 0x4E00 and 0x9FA5 are Chinese characters. We crawl the stroke information of all 20,402 characters from an online dictionary and render each character glyph to a 28 INLINEFORM0 28 1-bit grayscale bitmap by using Pillow.
We choose adagrad BIBREF23 as our optimizing algorithm, and we set the batch size as 4,096 and learning rate as 0.05. In practice, the slide window size INLINEFORM0 of stroke INLINEFORM1 -grams is set as INLINEFORM2 . The dimension of all word embeddings of different models is consistently set as 300. We use two test tasks to evaluate the performance of different models: one is word similarity, and the other is word analogy. A word similarity test consists of multiple word pairs and similarity scores annotated by humans. Good word representations should make the calculated similarity have a high rank correlation with human annotated scores, which is usually measured by the Spearman's correlation INLINEFORM3 BIBREF24 .
The form of an analogy problem is like “king":“queen" = “man":“?", and “woman" is the most proper answer to “?". That is, in this task, given three words INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , the goal is to infer the fourth word INLINEFORM3 which satisfies “ INLINEFORM4 is to INLINEFORM5 that is similar to INLINEFORM6 is to INLINEFORM7 ". We use INLINEFORM8 BIBREF0 and INLINEFORM9 function BIBREF25 to calculate the most appropriate word INLINEFORM10 . By using the same data used in BIBREF14 and BIBREF3 , we adopt two manually-annotated datasets for Chinese word similarity task, i.e., wordsim-240 and wordsim-296 BIBREF26 and a three-group dataset for Chinese word analogy task.
We use gensim to implement both CBOW and Skipgram and apply the source codes pulished by the authors to implement CWE, JWE, GWE and GloVe. Since Cao et al. cao2018cw2vec did not publish their code, we follow their paper and reproduce cw2vec in mxnet which we also use to implement sisg BIBREF21 and our DWE. To encourage further research, we will publish our model and datasets.
UTF8gbsn The experimental results are shown in Table TABREF11 . We can observe that our DWE model achieves the best results both on dataset wordsim-240 and wordsim-296 in the similarity task as expected because of the particularity of Chinese morphology, but it only improves the accuracy for the family group in the analogy task.
Actually, it is not by chance that we get these results, because DWE has the advantage of distinguishing between morphologically related words, which can be verified by the results of the similarity task. Meanwhile, in the word analogy task, those words expressing family relations in Chinese are mostly compositional in their character glyphs. For example, in an analogy pair “兄弟" (brother) : “姐妹" (sister) = “儿子" (son) : “女儿" (daughter), we can easily find that “兄弟" and “儿子" share an exactly common part of glyph “儿" (male relative of a junior generation) while “姐妹" and “女儿" share an exactly common part of glyph “女" (female), and this kind of morphological pattern can be accurately captured by our model. However, most of the names of countries, capitals and cities are transliterated words, and the relationship between the morphology and semantics of words is minimal, which is consistent with the findings reported in BIBREF4 . For instance, in an analogy pair “西班牙" (Spain) : “马德里" (Madrid) = “法国" (France) : “巴黎" (Paris), we cannot infer any relevance among these four words literally because they are all translated by pronunciation.
In summary, since different words that are morphologically similar tend to have similar semantics in Chinese, simultaneously modeling the sequential and spatial information of characters from both stroke n-grams and glyph features can indeed improve the modeling of Chinese word representations substantially.
In this article, we first analyzed the similarities and differences in terms of morphology between alphabetical languages and Chinese. Then, we delved deeper into the particularity of Chinese morphology and proposed our DWE model by taking into account the sequential information of strokes order and the spatial information of glyphs. Through the evaluation on two representative tasks, our model shows its superiority in capturing the morphological information of Chinese.

Figure and table captions:
Figure 1: The upper part is an example for illustrating the inclusion relationship hidden in strokes order and character glyphs. The lower part reflects that a common stroke sequence may form different Chinese characters if their spatial configurations are different.
Figure 2: An illustration of our Dual-channel Word Embedding (DWE) model.
Table 1: Performance on word similarity and word analogy task. The dimension of embeddings is set as 300. The evaluation metric is ρ for word similarity and accuracy percentage for word analogy.

Questions about this paper:
How much data do they use to train the embeddings?

Answers:
11,529,432 segmented words and 20,402 characters