Character-Based Text Classification using Top Down Semantic Model for Sentence Representation

Abstract: Despite the success of deep learning on many fronts especially image and speech, its application in text classification often is still not as good as a simple linear SVM on n-gram TF-IDF representation especially for smaller datasets. Deep learning tends to emphasize on sentence level semantics when learning a representation with models like recurrent neural network or recursive neural network, however from the success of TF-IDF representation, it seems a bag-of-words type of representation has its strength. Taking advantage of both representions, we present a model known as TDSM (Top Down Semantic Model) for extracting a sentence representation that considers both the word-level semantics by linearly combining the words with attention weights and the sentence-level semantics with BiLSTM and use it on text classification. We apply the model on characters and our results show that our model is better than all the other character-based and word-based convolutional neural network models by \cite{zhang15} across seven different datasets with only 1\% of their parameters. We also demonstrate that this model beats traditional linear models on TF-IDF vectors on small and polished datasets like news article in which typically deep learning models surrender.

Recently, deep learning has been particularly successful in speech and image as an automatic feature extractor BIBREF1 , BIBREF2 , BIBREF3 , however deep learning's application to text as an automatic feature extractor has not been always successful BIBREF0 even compared to simple linear models with BoW or TF-IDF feature representation. In many experiments when the text is polished like news articles or when the dataset is small, BoW or TF-IDF is still the state-of-art representation compared to sent2vec or paragraph2vec BIBREF4 representation using deep learning models like RNN (Recurrent Neural Network) or CNN (Convolution Neural Network) BIBREF0 . It is only when the dataset becomes large or when the words are noisy and non-standardized with misspellings, text emoticons and short-forms that deep learning models which learns the sentence-level semantics start to outperform BoW representation, because under such circumstances, BoW representation can become extremely sparse and the vocabulary size can become huge. It becomes clear that for large, complex data, a large deep learning model with a large capacity can extract a better sentence-level representation than BoW sentence representation. However, for small and standardized news-like dataset, a direct word counting TF-IDF sentence representation is superior. Then the question is can we design a deep learning model that performs well for both simple and complex, small and large datasets? And when the dataset is small and standardized, the deep learning model should perform comparatively well as BoW? With that problem in mind, we designed TDSM (Top-Down-Semantic-Model) which learns a sentence representation that carries the information of both the BoW-like representation and RNN style of sentence-level semantic which performs well for both simple and complex, small and large datasets.
Getting inspiration from the success of TF-IDF representation, our model intends to learn a word topic-vector which is similar to TF-IDF vector of a word but is different from word embedding, whereby the values in the topic-vector are all positives, and each dimension of the topic-vector represents a topic aspect of the word. Imagine a topic-vector of representation meaning $[animal, temperature, speed]$ , so a $rat$ maybe represented as $[0.9, 0.7, 0.2]$ since $rat$ is an animal with high body temperature but slow running speed compared to a $car$ which maybe represented as $[0.1, 0.8, 0.9]$ for being a non-animal, but high engine temperature and fast speed. A topic-vector will have a much richer semantic meaning than one-hot TF-IDF representation and also, it does not have the cancellation effect of summing word-embeddings positional vectors $([-1, 1] + [1, -1] = [0, 0])$ . The results from BIBREF5 show that by summing word-embedding vectors as sentence representation will have a catastrophic result for text classification.
Knowing the topic-vector of each word, we can combine the words into a sentence representation $\tilde{s}$ by learning a weight $w_i$ for each word ${v_i}$ and do a linear sum of the words, $\tilde{s} = \sum _i {w_i}\tilde{v_i}$ . The weights $w_i$ for each word in the sentence summation is learnt by recurrent neural network (RNN) BIBREF6 with attention over the words BIBREF7 . The weights corresponds to the IDF (inverse document frequency) in TF-IDF representation, but with more flexibility and power. IDF is fixed for each word and calculated from all the documents (entire dataset), however attention weights learned from RNN is conditioned on both the document-level and dataset-level semantics. This sentence representation from topic-vector of each word is then concatenated with the sentence-level semantic vector from RNN to give a top-down sentence representation as illustrated in Figure 2 .
TDSM is a framework that can be applied to both word-level or character-level inputs. Here in this paper, we choose character-level over word-level inputs for practical industry reasons.
In industry applications, often the model is required to have continuous learning on datasets that morph over time. Which means the vocabulary may change over time, therefore feeding the dataset by characters dispel the need for rebuilding a new vocabulary every time when there are new words.
Industry datasets are usually very complex and noisy with a large vocabulary, therefore the memory foot-print of storing word embeddings is much larger than character embeddings.
Therefore improving the performance of a character-based model has a much larger practical value compared to word-based model.
There are many traditional machine learning methods for text classification and most of them could achieve quite good results on formal text datasets. Recently, many deep learning methods have been proposed to solve the text classification task BIBREF0 , BIBREF9 , BIBREF10 .
Deep convolutional neural network has been extremely successful for image classification BIBREF11 , BIBREF12 . Recently, many research also tries to apply it on text classification problem. Kim BIBREF10 proposed a model similar to Collobert's et al. BIBREF13 architecture. However, they employ two channels of word vectors. One is static throughout training and the other is fine-tuned via back-propagation. Various size of filters are applied on both channels, and the outputs are concatenated together. Then max-pooling over time is taken to select the most significant feature among each filter. The selected features are concatenated as the sentence vector.
Similarly, Zhang et al. BIBREF0 also employs the convolutional networks but on characters instead of words for text classification. They design two networks for the task, one large and one small. Both of them have nine layers including six convolutional layers and three fully-connected layers. Between the three fully connected layers they insert two dropout layers for regularization. For both convolution and max-pooling layers, they employ 1-D filters BIBREF14 . After each convolution, they apply 1-D max-pooling. Specially, they claim that 1-D max-pooling enable them to train a relatively deep network.
Besides applying models directly on testing datasets, more aspects are considered when extracting features. Character-level feature is adopted in many tasks besides Zhang et al. BIBREF0 and most of them achieve quite good performance.
Santos and Zadrozny BIBREF15 take word morphology and shape into consideration which have been ignored for part-of-speech tagging task. They suggest the intra-word information is extremely useful when dealing with morphologically rich languages. They adopt neural network model to learn the character-level representation which is further delivered to help word embedding learning.
Kim et al. BIBREF16 constructs neural language model by analysis of word representation obtained from character composition. Results suggest that the model could encode semantic and orthographic information from character level.
 BIBREF17 , BIBREF7 uses two hierarchies of recurrent neural network to extract the document representation. The lower hierarchical recurrent neural network summarizes a sentence representation from the words in the sentence. The upper hierarchical neural network then summarizes a document representation from the sentences in the document. The major difference between BIBREF17 and BIBREF7 is that Yang applies attention over outputs from the recurrent when learning a summarizing representation.
Attention model is also utilized in our model, which is used to assign weights for each word. Usually, attention is used in sequential model BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 . The attention mechanism includes sensor, internal state, actions and reward. At each time-step, the sensor will capture a glimpse of the input which is a small part of the entire input. Internal state will summarize the extracted information. Actions will decide the next glimpse location for the next step and reward suggests the benefit when taking the action. In our network, we adopt a simplified attention network as BIBREF22 , BIBREF23 . We learn the weights over the words directly instead of through a sequence of actions and rewards.
Residual network BIBREF3 , BIBREF24 , BIBREF25 is known to be able to make very deep neural networks by having skip-connections that allows gradient to back-propagate through the skip-connections. Residual network in BIBREF3 outperforms the state-of-the-art models on image recognition. He BIBREF24 introduces residual block as similar to feature refinement for image classification. Similarly, for text classification problem, the quality of sentence representation is also quite important for the final result. Thus, we try to adopt the residual block as in BIBREF3 , BIBREF24 to refine the sentence vector.
The entire model has only 780,000 parameters which is only 1% of the parameters in BIBREF0 large CNN model. We used BiLSTM BIBREF27 with 100 units in both forward and backward LSTM cell. The output from the BiLSTM is 200 dimensions after we concatenate the outputs from the forward and backward cells. We then use attention over the words by linearly transform 200 output dimensions to 1 follow by a softmax over the 1 dimension outputs from all the words. After the characters to topic-vector transformation by FCN, each topic vector will be 180 dimensions. The topic-vectors are then linearly sum with attention weights to form a 180 dimensions BoW-like sentence vector. This vector is further concatenate with 200 dimensions BiLSTM outputs. The 380 dimensions undergo 10 blocks of ResNet BIBREF25 plus one fully connected layer. We use RELU BIBREF29 for all the intra-layer activation functions. The source code will be released after some code refactoring and we built the models with tensorflow BIBREF30 and tensorgraph .
Unlike word-embedding BIBREF26 , topic-vector tries to learn a distributed topic representation at each dimension of the representation vector, which thus allows the simple addition of the word-level topic-vectors to form a sentence representation. Figure 1 illustrates how topic-vector is extracted from characters in words using FCN (Fully Convolutional Network). In order to force word level representation with topic meanings, we apply a sigmoid function over the output from FCN. Doing so, restrain the values at each dimension to be between 0 and 1, thus forcing the model to learn a distributed topic representation of the word.
Forming a sentence representation from words can be done simply by summing of the word-embeddings which produce catastrophic results BIBREF5 due to the cancellation effect of adding embedding vectors (negative plus positive gives zero). Or in our model, the summing of word-level topic-vectors which give a much better sentence representation as shown in Table 3 than summing word-embeddings.
Sentence vector derived from summing of the word topic-vectors is equivalent to the BoW vectors in word counting, whereby we treat the prior contribution of each word to the final sentence vector equally. Traditionally, a better sentence representation over BoW will be TF-IDF, which gives a weight to each word in a document in terms of IDF (inverse document frequency). Drawing inspiration from TF-IDF representation, we can have a recurrent neural network that outputs the attention BIBREF7 over the words. And the attention weights serve similar function as the IDF except that it is local to the context of the document, since the attention weight for a word maybe different for different documents while the IDF of a word is the same throughout all documents. With the attention weights $w_i$ and word topic-vector $\tilde{v}_i$ , we can form a sentence vector $\tilde{s}_{bow}$ by linear sum 
$$\tilde{s}_{bow} = \sum _i w_i \tilde{v}_i$$   (Eq. 10) 
With the neural sentence vector that derived from BoW which captures the information of individual words. We can also concatenate it with the output state from the RNN which captures the document level information and whose representation is conditioned on the positioning of the words in the document.
$$&\tilde{s}_{t} = RNN(\tilde{v}_{t}, \tilde{s}_{t-1}) \\
&\tilde{s}_{pos} = \tilde{s}_T \\

&\tilde{s} = \tilde{s}_{bow} \oplus \tilde{s}_{pos}$$   (Eq. 12) 
 where $T$ is the length of the document, and $\oplus $ represents concatenation such that $|\tilde{s}| = |\tilde{s}_{bow}| + |\tilde{s}_{pos}|$ . The overall sentence vector $\tilde{s}$ will then capture the information of both the word-level and document-level semantics of the document. And thus it has a very rich representation.
We used Bi-directional LSTM (BiLSTM) BIBREF27 as the recurrent unit. BiLSTM consist of a forward LSTM (FLSTM) and a backward LSTM (BLSTM), both LSTMs are of the same design, except that FLSTM reads the sentence in a forward manner and BLSTM reads the sentence in a backward manner. One recurrent step in LSTM of Equation 12 consists of the following steps 
$$\tilde{f}_t &= \sigma \big (\mathbf {W}_f (\tilde{s}_{t-1} \oplus \tilde{v}_t) + \tilde{b}_f \big ) \\
\tilde{i}_t &= \sigma \big (\mathbf {W}_i (\tilde{s}_{t-1} \oplus \tilde{v}_t) + \tilde{b}_i\big ) \\
\tilde{C}_t &= \tanh \big (\mathbf {W}_C(\tilde{s}_{t-1}, \tilde{v}_t) + \tilde{b}_C\big ) \\
\tilde{C}_t &= \tilde{f}_t \otimes \tilde{C}_{t-1} + \tilde{i}_t \otimes \tilde{C}_t \\
\tilde{o}_t &= \sigma \big (\mathbf {W}_o (\tilde{s}_{t-1} \oplus \tilde{v}_t) + \tilde{b}_o \big ) \\
\tilde{s}_t &= \tilde{o}_t \otimes \tanh (\tilde{C}_t)$$   (Eq. 14) 
 where $\otimes $ is the element-wise vector multiplication, $\oplus $ is vector concatenation similarly defined in Equation . $\tilde{f}_t$ is forget state, $\tilde{i}_t$ is input state, $\tilde{o}_t$ is output state, $\tilde{C}_t$ is the internal context which contains the long-short term memory of historical semantics that LSTM reads. Finally, the output from the BiLSTM will be a concatenation of the output from FLSTM and BLSTM 
$$\tilde{f}_t &= \text{FLSTM}(\tilde{v}_t, \tilde{s}_{t-1}) \\
\tilde{b}_t &= \text{BLSTM}(\tilde{v}_t, \tilde{s}_{t+1}) \\
\tilde{h}_{t} &= \tilde{f}_t \oplus \tilde{b}_t$$   (Eq. 15) 
Here, the concatenated output state of BiLSTM has visibility of the entire sequence at any time-step compared to single-directional LSTM which only has visibility of the sequence in the past. This property of BiLSTM is very useful for learning attention weights for each word in a document because then the weights are decided based on the information of the entire document instead of just words before it as in LSTM.
We use the standard benchmark datasets prepare by BIBREF0 . The datasets have different number of training samples and test samples ranging from 28,000 to 3,600,000 training samples, and of different text length ranging from average of 38 words for Ag News to 566 words in Sogou news as illustrated in Table 1 . The datasets are a good mix of polished (AG) and noisy (Yelp and Amazon reviews), long (Sogou) and short (DBP and AG), large (Amazon reviews) and small (AG) datasets. And thus the results over these datasets serve as good evaluation on the quality of the model.
In this paper, we take 128 ASCII characters as character set, by which most of the testing documents are composite. We define word length as 20 and character embedding length as 100. If a word with characters less than 20, we will pad it with zeros. If the length is larger than 20, we just take the first 20 characters. We set the maximum length of words as the average number of words of the documents in the dataset plus two standard deviation, which is long enough to cover more than 97.5% of the documents. For documents with number of words more than the preset maximum number of words, we will discard the exceeding words.
We select both traditional models and the convolutional models from BIBREF0 , the recurrent models from BIBREF7 , BIBREF17 as baselines. Also in order to ensure a fair comparison of models, such that any variation in the result is purely due to the model difference, we compare TDSM only with models that are trained in the same way of data preparation, that is the words are lowered and there are no additional data alteration or augmentation with thesaurus. Unfortunately, BIBREF7 , BIBREF17 recurrent models are trained on full text instead of lowered text, so their models may not be objectively compared to our models, since it is well known from BIBREF28 that different text preprocessing will have significant impact on the final results, Zhang's result shows that a simple case lowering can result up to 4% difference in classification accuracy. Despite this, we still include the recurrent models for comparison, because they provide a good reference for understanding time-based models on large datasets of long sentences.
is the standard word counting method whereby the feature vector represents the term frequency of the words in a sentence.
is similar to BoW, except that it is derived by the counting of the words in the sentence weighted by individual word's term-frequency and inverse-document-frequency BIBREF31 . This is a very competitive model especially on clean and small dataset.
is derived from clustering of words-embeddings with k-means into 5000 clusters, and follow by BoW representation of the words in 5000 clusters.
is CNN model on word embeddings following BIBREF0 , to ensure fair comparison with character-based models, the CNN architecture is the same as Lg. Conv and Sm. Conv with the same number of parameters.
from BIBREF17 is basically a recurrent neural network based on LSTM and GRU BIBREF32 over the words in a sentence, and over the sentences in a document. It tries to learn a hierarchical representation of the text from multi-levels of recurrent layers.
from BIBREF7 is basically similar to LSTM-GRNN except that instead of just learning the hierarchical representation of the text directly with RNN, it also learns attention weights over the words during the summarization of the words and over the sentences during the summarization of the sentences.
are proposed in BIBREF0 , which is a CNN model on character encoding and is the primary character-based baseline model that we are comparing with.
Table 3 shows the comparison results of different datasets with different size, different sentence length, and different quality (polished AG news vs messy Yelp and Amazon reviews).
From the results, we see that TDSM out-performs all the other CNN models across all the datasets with only 1% of the parameters of Zhang's large conv model and 7.8% of his small conv model. Since these results are based on the same text preprocessing and across all kinds of dataset (long, short, large, small, polished, messy), we can confidently say that TDSM generalizes better than the other CNN models over text classification. These results show that a good architecture design can achieve a better accuracy with significantly less parameters.
Character-based models are the most significant and practical model for real large scale industry deployment because of its smaller memory footprint, agnostic to changes in vocabulary and robust to misspellings BIBREF16 . For a very long time, TF-IDF has been state-of-art models especially in small and standardized datasets. However because of its large memory footprint and non-suitability for continuous learning (because a new vocabulary has to be rebuilt every once in awhile when there are new words especially for data source like Tweeter), it was not an ideal model until character-based models came out. From the results, previous character-based models are generally better than TF-IDF for large datasets but falls short for smaller dataset like AG news. TDSM successfully close the gap between character-based models and TF-IDF by beating TF-IDF with 1% better performance. The results also confirm the hypothesis that TDSM as illustrated in Figure 2 which contains both the BoW-like and sentence-level features, has the best of the traditional TF-IDF and the recent deep learning model, is able to perform well for both small and large datasets.
From the results, we also observe that TDSM improves over other character-based models by a big margin of 3% for Lg. Conv and 5.7% for Sm. Conv on the AG dataset. But the improvement tails off to only 0.5% for Amazon reviews when the dataset size increases from 120,000 to 3.6 million. This is probably because TDSM has reached its maximum capacity when the dataset gets very large compared to other character-based models which have 100 times the capacity of TDSM.
For Yelp Full, we observe that the hierarchical recurrent models LSTM-GRNN and HN-ATT performs about 10% points better than TDSM but drops to only 3% for Amazon Full. This may be partly due to their data being prepared differently from our models. This can also be due to the structure of these hierarchical recurrent models which has two levels of recurrent neural networks for summarizing a document, whereby the first level summarizes a sentence vector from the words and the second level summarizes a document vector from the sentences. So these models will start to perform much better when there are alot of sentences and words in a document. For Yelp Full, there are of average 134 words in one document and Amazon Full has about 80 words per document. That's why the performance is much better for these recurrent models on Yelp than on Amazon. However, these hierarchical recurrent models will be reduce to a purely vanilla RNN for short text like AG News or Tweets with a few sentences, and under such circumstances its result will not be much different from a standard RNN. Nevertheless, LSTM-GRNN or HN-ATT does indicate the strength of RNN models in summarizing the sentences and documents and deriving a coherent sentence-level and document-level representation.
From the results, we see a strong promise for TDSM as a competitive model for text classification because of its hybrid architecture that looks at the sentence from both the traditional TF-IDF point of view and the recent deep learning point of view. The results show that this type of view can derive a rich text representation for both small and large datasets.

Figure and table captions:
Figure 1: Illustration of transformation from character-level embeddings to word-level topic-vector. The transformation is done with fully convolutional network (FCN) similar to (Long et al., 2015), each hierarchical level of the FCN will extract an n-gram character feature of the word until the word-level topic-vector.
Figure 2: TDSM: Illustration of how topic-vector of the words are combined together into a sentence representation. Note that for the actual model, we are using BiLSTM for extracting positional features. In this diagram, in order to present our idea in a neater manner, we demonstrate with a vanilla RNN.
Table 1: Statistics of datasets.
Table 2: Fully-Convolutional Network from characters to topic-vector. The first convolutional layer has kernel size of (100, 4) where 100 is the embedding size over 4-gram character as shown in Figure 1.
Table 3: Comparison results on accuracy for various models. Lg w2v Conv and Sm. w2v Conv is CNN on word embedding. Lg. Conv and Sm. Conv is CNN on character embedding. LSTM-GRNN and HN-ATT are different species of recurrent neural networks on words and sentences. Unfortunately, these two RNN models did not use the same text preprocessing technique as other models, so their models may not be objectively comparable to Zhang’s or our model, because it is well known that (Zhang et al., 2015; Uysal & Gunal, 2014), the difference in text preprocessing will have a significant impact on the final accuracy. However, these RNN models are still a good reference for our understanding of time-based models on large datasets of long sentences.
Table 4: Number of parameters for different models

Questions about this paper:
What other non-neural baselines do the authors compare to? 

Answers:
bag of words, tf-idf, bag-of-means