End-to-End Multi-View Networks for Text Classification

Abstract: We propose a multi-view network for text classification. Our method automatically creates various views of its input text, each taking the form of soft attention weights that distribute the classifier's focus among a set of base features. For a bag-of-words representation, each view focuses on a different subset of the text's words. Aggregating many such views results in a more discriminative and robust representation. Through a novel architecture that both stacks and concatenates views, we produce a network that emphasizes both depth and width, allowing training to converge quickly. Using our multi-view architecture, we establish new state-of-the-art accuracies on two benchmark tasks.

State-of-the-art deep neural networks leverage task-specific architectures to develop hierarchical representations of their input, with each layer building a refined abstraction of the layer that came before it BIBREF0 . For text classification, one can think of this as a single reader building up an increasingly refined understanding of the content. In a departure from this philosophy, we propose a divide-and-conquer approach, where a team of readers each focus on different aspects of the text, and then combine their representations to make a joint decision.
More precisely, the proposed Multi-View Network (MVN) for text classification learns to generate several views of its input text. Each view is formed by focusing on different sets of words through a view-specific attention mechanism. These views are arranged sequentially, so each subsequent view can build upon or deviate from previous views as appropriate. The final representation that concatenates these diverse views should be more robust to noise than any one of its components. Furthermore, different sentences may look similar under one view but different under another, allowing the network to devote particular views to distinguishing between subtle differences in sentences, resulting in more discriminative representations.
Unlike existing multi-view neural network approaches for image processing BIBREF1 , BIBREF2 , where multiple views are provided as part of the input, our MVN learns to automatically create views from its input text by focusing on different sets of words. Compared to deep Convolutional Networks (CNN) for text BIBREF3 , BIBREF0 , the MVN strategy emphasizes network width over depth. Shorter connections between each view and the loss function enable better gradient flow in the networks, which makes the system easier to train. Our use of multiple views is similar in spirit to the weak learners used in ensemble methods BIBREF4 , BIBREF5 , BIBREF6 , but our views produce vector-valued intermediate representations instead of classification scores, and all our views are trained jointly with feedback from the final classifier.
Experiments on two benchmark data sets, the Stanford Sentiment Treebank BIBREF7 and the AG English news corpus BIBREF3 , show that 1) our method achieves very competitive accuracy, 2) some views distinguish themselves from others by better categorizing specific classes, and 3) when our base bag-of-words feature set is augmented with convolutional features, the method establishes a new state-of-the-art for both data sets.
The MVN architecture is depicted in Figure FIGREF1 . First, individual selection vectors INLINEFORM0 are created, each formed by a distinct softmax weighted sum over the word vectors of the input text. Next, these selections are sequentially transformed into views INLINEFORM1 , with each view influencing the views that come after it. Finally, all views are concatenated and fed into a two-layer perceptron for classification.
Each selection INLINEFORM0 is constructed by focusing on a different subset of words from the original text, as determined by a softmax weighted sum BIBREF8 . Given a piece of text with INLINEFORM1 words, we represent it as a bag-of-words feature matrix INLINEFORM2 INLINEFORM3 . Each row of the matrix corresponds to one word, which is represented by a INLINEFORM4 -dimensional vector, as provided by a learned word embedding table. The selection INLINEFORM5 for the INLINEFORM6 view is the softmax weighted sum of features: DISPLAYFORM0 
 where the weight INLINEFORM0 is computed by: DISPLAYFORM0 DISPLAYFORM1 
 here, INLINEFORM0 (a vector) and INLINEFORM1 (a matrix) are learned selection parameters. By varying the weights INLINEFORM2 , the selection for each view can focus on different words from INLINEFORM3 , as illustrated by different color curves connecting to INLINEFORM4 in Figure FIGREF1 .
Having built one INLINEFORM0 for each of our INLINEFORM1 views, the actual views are then created as follows: DISPLAYFORM0 
 where INLINEFORM0 are learned parameter matrices, and INLINEFORM1 represents concatenation. The first and last views are formed by solely INLINEFORM2 ; however, they play very different roles in our network. INLINEFORM3 is completely disconnected from the others, an independent attempt at good feature selection, intended to increase view diversity BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . Conversely, INLINEFORM4 forms the base of a structure similar to a multi-layer perceptron with short-cutting, as defined by the recurrence in Equation EQREF7 . Here, the concatenation of all previous views implements short-cutting, while the recursive definition of each view implements stacking, forming a deep network depicted by horizontal arrows in Figure FIGREF1 . This structure makes each view aware of the information in those previous to it, allowing them to build upon each other. Note that the INLINEFORM5 matrices are view-specific and grow with each view, making the overall parameter count quadratic in the number of views.
The final step is to transform our views into a classification of the input text. The MVN does so by concatenating its view vectors, which are then fed into a fully connected projection followed by a softmax function to produce a distribution over the possible classes. Dropout regularization BIBREF13 can be applied at this softmax layer, as in BIBREF14 .
The MVN's selection layer operates on a matrix of feature vectors INLINEFORM0 , which has thus far corresponded to a bag of word vectors. Each view's selection makes intuitive sense when features correspond to words, as it is easy to imagine different readers of a text focusing on different words, with each reader arriving at a useful interpretation. However, there is a wealth of knowledge on how to construct powerful feature representations for text, such as those used by convolutional neural networks (CNNs). To demonstrate the utility of having views that weight arbitrary feature vectors, we augment our bag-of-words representation with vectors built by INLINEFORM1 -gram filters max-pooled over the entire text BIBREF14 , with one feature vector for each INLINEFORM2 -gram order, INLINEFORM3 . The augmented INLINEFORM4 matrix has INLINEFORM5 rows. Unlike our word vectors, the 4 CNN vectors each provide representations of the entire text. Returning to our reader analogy, one could imagine these to correspond to quick ( INLINEFORM6 ) or careful ( INLINEFORM7 ) skims of the text. Regardless of whether a feature vector is built by embedding table or by max-pooled INLINEFORM8 -gram filters, we always back-propagate through all feature construction layers, so they become specialized to our end task.
The Stanford Sentiment Treebank contains 11,855 sentences from movie reviews. We use the same splits for training, dev, and test data as in BIBREF7 to predict the fine-grained 5-class sentiment categories of the sentences. For comparison purposes, following BIBREF14 , BIBREF15 , BIBREF16 , we train the models using both phrases and sentences, but only evaluate sentences at test time.
We initialized all of the word embeddings BIBREF17 , BIBREF18 using the publicly available 300 dimensional pre-trained vectors from GloVe BIBREF19 . We learned 8 views with 200 dimensions each, which requires us to project the 300 dimensional word vectors, which we implemented using a linear transformation, whose weight matrix and bias term are shared across all words, followed by a INLINEFORM0 activation. For optimization, we used Adadelta BIBREF20 , with a starting learning rate of 0.0005 and a mini-batch of size 50. Also, we used dropout (with a rate of 0.2) to avoid overfitting. All of these MVN hyperparameters were determined through experiments measuring validation-set accuracy.
The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table TABREF11 . The results indicate that the bag-of-words MVN outperforms most methods, but obtains lower accuracy than the state-of-the-art results achieved by the tree-LSTM BIBREF21 , BIBREF22 and the high-order CNN BIBREF16 . However, when augmented with 4 convolutional features as described in Section SECREF9 , the MVN strategy surpasses both of these, establishing a new state-of-the-art on this benchmark.
In Figure FIGREF12 , we present the test-set accuracies obtained while varying the number of views in our MVN with convolutional features. These results indicate that better predictive accuracy can be achieved while increasing the number of views up to eight. After eight, the accuracy starts to drop. The number of MVN views should be tuned for each new application, but it is good to see that not too many views are required to achieve optimal performance on this task.
To better understand the benefits of the MVN method, we further analyzed the eight views constructed by our best model. After training, we obtained the view representation vectors for both the training and testing data, and then independently trained a very simple, but fast and stable Naïve Bayes classifier BIBREF23 for each view. We report class-specific F-measures for each view in Figure FIGREF13 . From this figure, we can observe that different views focus on different target classes. For example, the first two views perform poorly on the 0 (very negative) and 1 (negative) classes, but achieve the highest F-measures on the 2 (neutral) class. Meanwhile, the non-neutral classes each have a different view that achieves the highest F-measure. This suggests that some views have specialized in order to better separate subsets of the training data.
We provide an ablation study in Table TABREF14 . First, we construct a traditional ensemble model. We independently train eight MVN models, each with a single view, to serve as weak learners. We have them vote with equal weight for the final classification, obtaining a test-set accuracy of 50.2. Next, we restrict the views in the MVN to be unaware of each other. That is, we replace Equation EQREF7 with INLINEFORM0 , which removes all horizontal links in Figure FIGREF1 . This drops performance to 49.0. Finally, we experiment with a variant of MVN, where each view is only connected to the most recent previous view, replacing Equation EQREF7 with INLINEFORM1 , leading to a version where the parameter count grows linearly in the number of views. This drops the test-set performance to 50.5. These experiments suggest that enabling the views to build upon each other is crucial for achieving the best performance.
The AG corpus BIBREF3 , BIBREF0 contains categorized news articles from more than 2,000 news outlets on the web. The task has four classes, and for each class there are 30,000 training documents and 1,900 test documents. A random sample of the training set was used for hyper-parameter tuning. The training and testing settings of this task are exactly the same as those presented for the Stanford Sentiment Treebank task in Section SECREF10 , except that the mini-batch size is reduced to 23, and each view has a dimension of 100.
The test errors obtained by various methods are presented in Table TABREF16 . These results show that the bag-of-words MVN outperforms the state-of-the-art accuracy obtained by the non-neural INLINEFORM0 -gram TFIDF approach BIBREF3 , as well as several very deep CNNs BIBREF0 . Accuracy was further improved when the MVN was augmented with 4 convolutional features.
In Figure FIGREF17 , we show how accuracy and loss evolve on the validation set during MVN training. These curves show that training is quite stable. The MVN achieves its best results in just a few thousand iterations.
We have presented a novel multi-view neural network for text classification, which creates multiple views of the input text, each represented as a weighted sum of a base set of feature vectors. These views work together to produce a discriminative feature representation for text classification. Unlike many neural approaches to classification, our architecture emphasizes network width in addition to depth, enhancing gradient flow during training. We have used the multi-view network architecture to establish new state-of-the-art results on two benchmark text classification tasks. In the future, we wish to better understand the benefits of generating multiple views, explore new sources of base features, and apply this technique to other NLP problems such as translation or tagging.

Figure and table captions:
Figure 1: A MVN architecture with four views.
Table 1: Accuracies on the Stanford Sentiment Treebank 5-class classification task; except for the MVN, all results are drawn from (Lei et al., 2015).
Figure 2: Accuracies obtained by varying the number of views.
Table 3: Error rates on the AG News test set. All results except for the MVN are drawn from (Conneau et al., 2016)
Figure 3: Class-specific F-measures obtained by Naı̈ve Bayes classifiers built over different views.
Table 2: Ablation experiments on the Stanford Sentiment Treebank test set
Figure 4: Accuracies and cost on the validation set during training on the AG News data set.

Questions about this paper:
what state of the accuracy did they obtain?
what models did they compare to?
which benchmark tasks did they experiment on?

Answers:
51.5
High-order CNN, Tree-LSTM, DRNN, DCNN, CNN-MC, NBoW and SVM 
 They used Stanford Sentiment Treebank benchmark for sentiment classification task and   AG English news corpus for the text classification task.