Introduction
Word embeddings have been widely used for natural language processing (NLP) tasks BIBREF0 . However, the large word vocabulary makes word embeddings expensive to train. Some people argue that we can model languages at the character-level BIBREF1 . For alphabetic languages such as English, where the characters are much fewer than the words, the character embeddings achieved the state-of-the-art results with much fewer parameters.
Unfortunately, for the other languages that use non-alphabetic systems, the character vocabulary can be also large. Moreover, Chinese and Japanese, two of the most widely used non-alphabetic languages, especially contain large numbers of ideographs: hanzi of Chinese and kanji of Japanese. The character vocabulary can be as scalable as the word vocabulary (e.g., see the datasets introduced in Section SECREF11 ). Hence the conventional character embedding-based method is not able to give us a slim vocabulary on Chinese and Japanese.
For convenience, let us collectively call hanzi and kanji as Chinese characters. Chinese characters are ideographs composed with semantic and phonetic radicals, both of which are available for character recognition, and the semantic information may be embedded in Chinese characters by the semantic radicals BIBREF2 . Besides, though the character vocabulary is huge, the number of the radicals is much fewer. Accordingly, we explored a model that represents the Chinese characters by the sequence of the radicals. We applied our proposed model to sentiment classification tasks on Chinese and Japanese and achieved the follows:
Methodology
The architecture of our proposed model is as shown in Fig. FIGREF3 . It looks similar to the character-aware neural language model proposed by BIBREF1 , but we represent a word by the sequence of radical embeddings instead of character embeddings. Besides, unlike the former model, there are no highway layers in the proposed model, because we find that highway layers do not bring significant improvements to our proposed model (see Section SECREF31 ).
Representation of Characters: Sequences of Radical-level Embeddings
For every character, we use a sequence of INLINEFORM0 radical-level embeddings to represent it. They are not treated as in a bag because the position of each radical is related to how it is informative BIBREF3 . For a Chinese character, it is the sequence of the radical embeddings. When it comes to the other characters, including kanas of Japanese, alphabets, digits, punctuation marks, and special characters, it is the sequence comprised of the corresponding character embedding and INLINEFORM1 zero vectors. We zero-pad the radical sequences of all the characters to align the lengths.
From Character Radicals to Word Features: CNN Encoder
CNNs BIBREF4 have been used for various NLP tasks and shown effective BIBREF0 , BIBREF5 , BIBREF1 . For NLP, CNNs are able to extract the temporal features, reduce the parameters, alleviate over-fitting and improve generalization ability. We also take advantage of the weight sharing technology of CNNs to learn the shared features of the characters.
Let INLINEFORM0 be the radical-level vocabulary that contains Chinese character radicals, kanas of Japanese, alphabets, digits, punctuation marks, and special characters, INLINEFORM1 be the matrix of all the radical-level embeddings, INLINEFORM2 be the dimension of each radical-level embedding. We have introduced that each character is represented by a sequence of radical-level embeddings of length INLINEFORM3 . Thus a word INLINEFORM4 composed of INLINEFORM5 characters is represented by a matrix INLINEFORM6 , each column of which is a radical-level embedding.
We apply convolution between INLINEFORM0 and several filters (convolution kernels). For each filter INLINEFORM1 , we apply a nonlinear activation function INLINEFORM2 and a max-pooling on the output to obtain a feature vector. Let INLINEFORM3 be the stride, INLINEFORM4 be the window, INLINEFORM5 be the hidden weight of a filter, respectively. The feature vector of INLINEFORM6 obtained by INLINEFORM7 is given by: DISPLAYFORM0
where INLINEFORM0 is the convolution operator. The pooling window is INLINEFORM1 to obtain the most important information of word INLINEFORM2 .
We have two kinds of filters: (1) the filters with stride INLINEFORM0 to obtain radical-level features; (2) the filters with stride INLINEFORM1 to obtain character-level features.
After the max-pooling layer, we concatenate and flatten all of the outputs through all of the filters as the feature vector of the word. Let INLINEFORM0 be the number of the output channel of each filter. If we use totally INLINEFORM1 filters, each output of which is INLINEFORM2 , the output feature of INLINEFORM3 is INLINEFORM4 . Here, we assume that the number of the output channels of every filter is the same, but we tailor it for each filter in the experiments following BIBREF1 .
From Word Features to Document Features: Bi-directional Long Short-term Memory RNN Encoder
An RNN is a kind of neural networks designed to learn sequential data. The output of an RNN unit at time INLINEFORM0 depends on the output at time INLINEFORM1 . Bi-directional RNNs BIBREF6 are able to extract the past and future information for each node in a sequence, have shown effective for Machine Translation BIBREF7 and Machine Comprehension BIBREF8 .
A Long Short-term Memory (LSTM) BIBREF9 Unit is a kind of unit for RNN that keeps information from long range context. We use a bi-directional RNN of LSTM to encode the document feature from the sequence of the word features.
An LSTM unit contains a forget gate INLINEFORM0 to decide whether to keep the memory, an input gate INLINEFORM1 to decide whether to update the memory and an output gate INLINEFORM2 to control the output. Let INLINEFORM3 be the output of a LSTM unit at time INLINEFORM4 , INLINEFORM5 be the candidate cell state at time INLINEFORM6 , INLINEFORM7 be the cell state at time INLINEFORM8 . They are given by: DISPLAYFORM0
where, INLINEFORM0 and INLINEFORM1 are the element-wise sigmoid function and multiplication operator.
Our proposed model contains two RNN layers that read document data from different directions. Let INLINEFORM0 be a document composed of INLINEFORM1 words. One of the RNN layers reads the document from the first word to the INLINEFORM2 th word, the other reads the document from the INLINEFORM3 th word to the first word. Let INLINEFORM4 be the final output of the former RNN layer and INLINEFORM5 be the final output of the latter. We concatenate INLINEFORM6 and INLINEFORM7 as the document feature. After that, we apply an affine transformation and a softmax to obtain the prediction of the sentiment labels: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 is the concatenation operator. INLINEFORM2 is the estimated label of the document, INLINEFORM3 is one of the labels in the label set INLINEFORM4 .
We minimize the cross entropy loss to train the model. Let INLINEFORM0 be the set of all the documents, and INLINEFORM1 be the true label of document INLINEFORM2 , the loss is given by: DISPLAYFORM0
Datasets
In the experiments, we used a Chinese dataset and a Japanese dataset. We used the publicly available Ctrip review data pack for Chinese. They are comprised of travel reviews crawled from ctrip.com . We used a subset of 10,000 reviews in the pack. We randomly select 8,000 and 2,000 from it for training and test, respectively. The Japanese dataset is provided by Rakuten, Inc. It contains 64,000,000 reviews of the products in Rakuten Ichiba . The reviews are labeled with 6-point evaluation of 0-5. We labeled the reviews with less than 3 points as the negative samples, and the others as the positive samples. We randomly chose 10,000 reviews to align the size of the Chinese datasets, 8,000 and 2,000 from it for training and test, respectively.
The detailed information of the datasets is shown in Table TABREF15 . The character vocabularies are as scalable as the word vocabularies but the radical-level vocabularies are much smaller. The character vocabulary of the Rakuten dataset is even larger than its own word vocabulary. 94% of the Rakuten dataset are Chinese characters while the Ctrip dataset contains 74% Chinese characters. Chinese characters account for fewer percentage in Ctrip data, probably because the Ctrip data is not well stripped.
Baselines
We compared the proposed model with the follows:
The character-aware neural language model BIBREF1 : It is an RNN language model that takes character embeddings as the inputs, encodes them with CNNs and then input them to RNNs for prediction. It achieved the state-of-the-art as a language model on alphabetic languages. We let it predict the sentiment labels instead of words.
Bi-directional RNN BIBREF6 with word embeddings: It is a classical bi-directional RNN classifier, basic but effective. We also employed LSTM for it, and input the word embeddings.
Hierarchical attention networks BIBREF10 : It is the state-of-the-art RNN-based document classifier. Following their method, the documents were segmented into shorter sentences of 100 words, and hierarchically encoded with bi-directional RNNs.
FastText BIBREF11 : It is the state-of-the-art baseline for text classification, which simply takes n-gram features and classifies sentences by hierarchical softmax. We used the word embedding version but did not use the bigram version because the other models for comparison do not use bigram inputs.
Hyperparameters
The setup of the hyperparameters in our experiments is shown in Table TABREF22 . They were tuned on the development set of 4,000 reviews, 2,000 from another subset in the public Ctrip data pack, and the other 2,000 randomly chosen from the review data of Rakuten Ichiba. We aligned the sizes of the feature vectors of the words and the documents in different models for a fair comparison. All the embeddings are initialized randomly with the uniform distribution.
All of the models were trained by RMSprop BIBREF12 with mini-batches of 100 samples. The learning rate and decay term were set as 0.001 and 0.9 respectively, also tuned on the development set.
Text Preprocess
We segmented the documents into words by Jieba and Juman++ BIBREF13 , respectively for Chinese and Japanese. We zero-padded the length of the sentences, words, and radical sequences of the characters as 500, 4 and 3, respectively.
We split the Chinese characters in CJK Unified Ideographs of ISO/IEC 10646-1:2000 character set, until there is no component can be split further, according to CHISE Character Structure Information Database . Then the Chinese character is represented by the sequence of the radicals from the left to the right, the top to the bottom as shown in Fig. FIGREF26 . The sequences are zero-padded to the same length. For an unknown Chinese character not in the set, we treat it as a special character.
Results
The number of parameters, test accuracy, and cross entropy loss of each model are as shown in Fig. FIGREF28 . The proposed model has 13% fewer parameters than the character embedding-based model, 91% and 82% fewer parameters than the word embedding-based models for Ctrip dataset and Rakuten dataset, respectively. The accuracy is statistically the same as the character embedding-based model, approximately 98% of the word embedding-based model. The losses of the models are also close. The hierarchical attention networks and fastText achieved approximately 11% and 19% lower loss on Ctrip dataset. But on Rakuten dataset whose percentage of Chinese characters is higher, the differences between them and the proposed model drops to 0% and 9% respectively.
The Proposed Model Is the Most Cost-effective
The performance of the proposed model is not significantly different from the character embedding-based baseline, and very close to the word embedding-based baselines, with a smaller vocabulary and fewer parameters. It indicates that radical-embeddings are at least as effective as the character-embeddings for Chinese and Japanese, but require less space. It suggests that for Chinese and Japanese, the radical embeddings are more cost-effective than the character embeddings.
The CNN Encoder Is Efficient
Even though the character vocabulary is as scalable as the word vocabulary on Chinese and Japanese, the character embedding-based method with CNN encoder can still reduce approximately 90% and 80% parameters for the Chinese and Japanese datasets, respectively. The CNNs allow low-dimension inputs, and share weights in the procedure of encoding the intpus to the high-dimension word features. It is probably the reason that it can save parameters although the sizes of the vocabularies are similar.
Highway Layers Are Not Effective For Us
BIBREF1 reported that the highway networks BIBREF14 are effective for RNN language models. A highway layer is tailored to adaptively switch between a full-connected layer and a “highway" that directly outputs the input. We also studied that whether it is effective for our proposed model in the sentiment classification task. Following BIBREF1 , we attempted to input the flattened concatenated output of the max-pooling layer to a highway layer that employs ReLU before we input it to RNN. The change of the performance is as shown in Fig. FIGREF32 .
We observed no significant improvement. Probably for two-class sentiment classification, a full-connected layer with ReLU is not necessary between the CNN encoder and the bi-directional RNN encoder, hence the highway network learned to pass the inputs directly to the outputs all the time.
Related Works
The computational cost brought by the large word vocabulary is a classical problem when neural networks are employed for NLP. In the earliest works, people limited the size of the vocabulary, which is not able to exploit the potential generalization ability on the rare words BIBREF15 . It has made people explore alternative methods for the softmax function to efficiently train all the words, e.g., hierarchical softmax BIBREF16 , noise-contrastive estimation BIBREF17 and negative sampling BIBREF18 . However, the temporal complexity of the softmax function is not the only thing suffering the high-dimension vocabulary. Scalable word vocabulary leads to a large embedding layer, hence huge neural network with millions of parameters, which costs quite a few gigabytes to store. BIBREF19 proposed a convolutional neural network (CNN) that takes characters as the input for text classification and outperforms the previous models for large datasets. They showed the character-level CNNs are effective for text classification without the need for words. BIBREF1 introduced a recurrent neural network (RNN) language model that takes character embeddings encoded by convolutional layers as the input. Their model has much fewer parameters than the models using word embeddings, and reached the performance of the state-of-the-art on English, and outperformed baselines on morphologically rich languages. However, for Chinese and Japanese, the character vocabulary is also large, and the character embeddings are blind to the semantic information of the radicals.
Conclusion and Outlook
We have proposed a model that takes radicals of characters as the inputs for sentiment classification on Chinese and Japanese, whose character vocabulary can be as scalable as word vocabulary. Our proposed model is as powerful as the character embedding-based model, and close to the word embedding-based model for the sentiment classification task, with much smaller vocabulary and fewer parameters. The results show that the radical embeddings are cost-effective for Chinese and Japanese. They are useful for the circumstances where the storage is limited.
There are still a lot to do on radical embeddings. For example, a radical may be related to the meaning sometimes, but express the pronunciation at other times. We will work on dealing with such phenomena for machine learning in the future.
The authors would like to thank Rakuten, Inc. and the Advanced Language Information Forum (ALAGIN) for generously providing us the Rakuten Ichiba data.

Question: what are the state of the art models?
Answer: The character-aware neural language model BIBREF1, Hierarchical attention networks BIBREF10, FastText BIBREF11 

character-aware neural language model BIBREF1, Hierarchical attention networks BIBREF10, FastText BIBREF11

The character-aware neural language model, Hierarchical attention networks, FastText