Abstract: We propose MVCNN, a convolution neural network (CNN) architecture for sentence classification. It (i) combines diverse versions of pretrained word embeddings and (ii) extracts features of multigranular phrases with variable-size convolution filters. We also show that pretraining MVCNN is critical for good performance. MVCNN achieves state-of-the-art performance on four tasks: on small-scale binary, small-scale multi-class and largescale Twitter sentiment prediction and on subjectivity classification.

Introduction: Different sentence classification tasks are crucial for many Natural Language Processing (NLP) applications. Natural language sentences have complicated structures, both sequential and hierarchical, that are essential for understanding them. In addition, how to decode and compose the features of component units, including single words and variable-size phrases, is central to the sentence classification problem. In recent years, deep learning models have achieved remarkable results in computer vision BIBREF0 , speech recognition BIBREF1 and NLP BIBREF2 . A problem largely specific to NLP is how to detect features of linguistic units, how to conduct composition over variable-size sequences and how to use them for NLP tasks BIBREF3 , BIBREF4 , BIBREF5 . socher2011dynamic proposed recursive neural networks to form phrases based on parsing trees. This approach depends on the availability of a well performing parser; for many languages and domains, especially noisy domains, reliable parsing is difficult. Hence, convolution neural networks (CNN) are getting increasing attention, for they are able to model long-range dependencies in sentences via hierarchical structures BIBREF6 , BIBREF5 , BIBREF7 . Current CNN systems usually implement a convolution layer with fixed-size filters (i.e., feature detectors), in which the concrete filter size is a hyperparameter. They essentially split a sentence into multiple sub-sentences by a sliding window, then determine the sentence label by using the dominant label across all sub-sentences. The underlying assumption is that the sub-sentence with that granularity is potentially good enough to represent the whole sentence. However, it is hard to find the granularity of a “good sub-sentence” that works well across sentences. This motivates us to implement variable-size filters in a convolution layer in order to extract features of multigranular phrases. Breakthroughs of deep learning in NLP are also based on learning distributed word representations – also called “word embeddings” – by neural language models BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . Word embeddings are derived by projecting words from a sparse, 1-of- $V$ encoding ( $V$ : vocabulary size) onto a lower dimensional and dense vector space via hidden layers and can be interpreted as feature extractors that encode semantic and syntactic features of words. Many papers study the comparative performance of different versions of word embeddings, usually learned by different neural network (NN) architectures. For example, chen2013expressive compared HLBL BIBREF9 , SENNA BIBREF2 , Turian BIBREF13 and Huang BIBREF14 , showing great variance in quality and characteristics of the semantics captured by the tested embedding versions. hill2014not showed that embeddings learned by neural machine translation models outperform three representative monolingual embedding versions: skip-gram BIBREF15 , GloVe BIBREF16 and C&W BIBREF3 in some cases. These prior studies motivate us to explore combining multiple versions of word embeddings, treating each of them as a distinct description of words. Our expectation is that the combination of these embedding versions, trained by different NNs on different corpora, should contain more information than each version individually. We want to leverage this diversity of different embedding versions to extract higher quality sentence features and thereby improve sentence classification performance. The letters “M” and “V” in the name “MVCNN” of our architecture denote the multichannel and variable-size convolution filters, respectively. “Multichannel” employs language from computer vision where a color image has red, green and blue channels. Here, a channel is a description by an embedding version. For many sentence classification tasks, only relatively small training sets are available. MVCNN has a large number of parameters, so that overfitting is a danger when they are trained on small training sets. We address this problem by pretraining MVCNN on unlabeled data. These pretrained weights can then be fine-tuned for the specific classification task. In sum, we attribute the success of MVCNN to: (i) designing variable-size convolution filters to extract variable-range features of sentences and (ii) exploring the combination of multiple public embedding versions to initialize words in sentences. We also employ two “tricks” to further enhance system performance: mutual learning and pretraining. In remaining parts, Section "Related Work" presents related work. Section "Model Description" gives details of our classification model. Section "Model Enhancements" introduces two tricks that enhance system performance: mutual-learning and pretraining. Section "Experiments" reports experimental results. Section "Conclusion" concludes this work.

Question: How much gain does the model achieve with pretraining MVCNN?
Answer: 0.8 points on Binary; 0.7 points on Fine-Grained; 0.6 points on Senti140; 0.7 points on Subj