Venue: NAACL 2016

# MGNC-CNN: A Simple Approach to Exploiting

Multiple Word Embeddings for Sentence Classification

 Ye Zhang\({}^{1}\) &Stephen Roller\({}^{1}\) &Byron C. Wallace\({}^{2}\)

\({}^{1}\)Department of Computer Science

The University of Texas at Austin

{yezhang,roller}@cs.utexas.edu &\({}^{2}\)iSchool

The University of Texas at Austin

byron.wallace@utexas.edu

###### Abstract

We introduce a novel, simple convolution neural network (CNN) architecture - _multi-group norm constraint CNN (MGNC-CNN)_ - that capitalizes on multiple sets of word embeddings for sentence classification. MGNC-CNN extracts features from input embedding sets independently and then joins these at the penultimate layer in the network to form a final feature vector. We then adopt a group regularization strategy that differentially penalizes weights associated with the subcomponents generated from the respective embedding sets. This model is much simpler than comparable alternative architectures and requires substantially less training time. Furthermore, it is flexible in that it does not require input word embeddings to be of the same dimensionality. We show that MGNC-CNN consistently outperforms baseline models.

## 1 Introduction

Neural models have recently gained popularity for Natural Language Processing (NLP) tasks Goldberg (2015); Collobert and Weston (2008); Cho (2015). For sentence classification, in particular, Convolution Neural Networks (CNN) have realized impressive performance Kim (2014); Zhang and Wallace (2015). These models operate over _word embeddings_, i.e., dense, low dimensional vector representations of words that aim to capture salient semantic and syntactic properties Collobert and Weston (2008).

An important consideration for such models is the specification of the word embeddings. Several options exist. For example, Kalchbrenner et al. Kalchbrenner et al. (2014) initialize word vectors to random low-dimensional vectors to be fit during training, while Johnson and Zhang Johnson and Zhang (2014) use fixed, one-hot encodings for each word. By contrast, Kim Kim (2014) initializes word vectors to those estimated via the word2vec model trained on 100 billion words of Google News Mikolov et al. (2013); these are then updated during training. Initializing embeddings to pre-trained word vectors is intuitively appealing because it allows transfer of learned distributional semantics. This has allowed a relatively simple CNN architecture to achieve remarkably strong results.

Many pre-trained word embeddings are now readily available on the web, induced using different models, corpora, and processing steps. Different embeddings may encode different aspects of language Pado and Lapata (2007); Erk and Pado (2008); Levy and Goldberg (2014): those based on bag-of-words (BoW) statistics tend to capture associations (_doctor_ and _hospital_), while embeddings based on dependency-parses encode similarity in terms of use (_doctor_ and _surgeon_). It is natural to consider how these embeddings might be _combined_ to improve NLP models in general and CNNs in particular.

**Contributions.** We propose MGNC-CNN, a novel, simple, scalable CNN architecture that can accommodate multiple off-the-shelf embeddings of variable sizes. Our model treats different word embeddings as distinct groups, and applies CNNs independently to each, thus generating corresponding feature vectors (one per embedding) which are then

concatenated at the classification layer. Inspired by prior work exploiting regularization to encode structure for NLP tasks [23, 14], we impose different regularization penalties on weights for features generated from the respective word embedding sets.

Our approach enjoys the following advantages compared to the only existing comparable model [60]: (i) It can leverage diverse, readily available word embeddings with different dimensions, thus providing flexibility. (ii) It is comparatively simple, and does not, for example, require _mutual learning_ or _pre-training_. (iii) It is an order of magnitude more efficient in terms of training time.

## 2 Related Work

Prior work has considered combining latent representations of words that capture syntactic and semantic properties [57], and inducing multi-modal embeddings [8] for general NLP tasks. And recently, Luo et al. luoEtAl14 proposed a framework that combines multiple word embeddings to measure text similarity, however their focus was not on classification.

More similar to our work, Yin and Schutze yinSchutze15 proposed _MVCNN_ for sentence classification. This CNN-based architecture accepts multiple word embeddings as inputs. These are then treated as separate 'channels', analogous to RGB channels in images. Filters consider all channels simultaneously. MVCNN achieved state-of-the-art performance on multiple sentence classification tasks. However, this model has practical drawbacks. (i) MVCNN requires that input word embeddings have the same dimensionality. Thus to incorporate a second set of word vectors trained on a corpus (or using a model) of interest, one needs to either find embeddings that happen to have a set number of dimensions or to estimate embeddings from scratch. (ii) The model is complex, both in terms of implementation and run-time. Indeed, this model requires pre-training and mutual-learning and requires days of training time, whereas the simple architecture we propose requires on the order of an hour (and is easy to implement).

## 3 Model Description

We first review standard one-layer CNN (which exploits a single set of embeddings) for sentence classification [69], and then propose our augmentations, which exploit multiple embedding sets.

**Basic CNN.** In this model we first replace each word in a sentence with its vector representation, resulting in a sentence matrix \({\bf A}\in\mathbb{R}^{s\times d}\), where \(s\) is the (zero-padded) sentence length, and \(d\) is the dimensionality of the embeddings. We apply a convolution operation between linear filters with parameters \({\bf w}_{1},{\bf w}_{2},...,{\bf w}_{k}\) and the sentence matrix. For each \({\bf w}_{i}\in\mathbb{R}^{h\times d}\), where \(h\) denotes 'height', we slide filter \(i\) across \({\bf A}\), considering 'local regions' of \(h\) adjacent rows at a time. At each local region, we perform element-wise multiplication and then take the element-wise sum between the filter and the (flattened) sub-matrix of \({\bf A}\), producing a scalar. We do this for each sub-region of \({\bf A}\) that the filter spans, resulting in a _feature map_ vector \({\bf c}_{i}\in\mathbb{R}^{(s-h+1)\times 1}\). We can use multiple filter sizes with different heights, and for each filter size we can have multiple filters. Thus the model comprises \(k\) weight vectors \({\bf w}_{1},{\bf w}_{2},...{\bf w}_{k}\), each of which is associated with an instantiation of a specific filter size. These in turn generate corresponding feature maps \({\bf c}_{1},{\bf c}_{2},...{\bf c}_{k}\), with dimensions varying with filter size. A _1-max pooling_ operation is applied to each feature map, extracting the largest number \(o_{i}\) from each feature map \(i\). Finally, we combine all \(o_{i}\) together to form a feature vector \({\bf o}\in\mathbb{R}^{k}\) to be fed through a softmax function for classification. We regularize weights at this level in two ways. (1) _Dropout_, in which we randomly set elements in \({\bf o}\) to zero during the training phase with probability \(p\), and multiply \(p\) with the parameters trained in \({\bf o}\) at test time. (2) An l2 norm penalty, for which we set a threshold \(\lambda\) for the l2 norm of \({\bf o}\) during training; if this is exceeded, we rescale the vector accordingly. For more details, see [26].

**MG-CNN.** Assuming we have \(m\) word embeddings with corresponding dimensions \(d_{1},d_{2},...d_{m}\), we can simply treat each word embedding independently. In this case, the input to the CNN comprises multiple sentence matrices \({\bf A}_{1},{\bf A}_{2},...{\bf A}_{\mathbf{m}}\), where each \({\bf A}_{l}\in\mathbb{R}^{s\times d_{l}}\) may have its own width \(d_{l}\). We then ap 

ply different groups of filters \(\{\mathbf{w}_{1}\},\{\mathbf{w}_{2}\},...\{\mathbf{w}_{m}\}\) independently to each \(\mathbf{A}_{l}\), where \(\{\mathbf{w}_{l}\}\) denotes the set of filters for \(\mathbf{A}_{l}\). As in basic CNN, \(\{\mathbf{w}_{l}\}\) may have multiple filter sizes, and multiple filters of each size may be introduced. At the classification layer we then obtain a feature vector \(\mathbf{o}_{l}\) for each embedding set, and we can simply concatenate these together to form the final feature vector \(\mathbf{o}\) to feed into the softmax function, where \(\mathbf{o}=\mathbf{o_{1}}\oplus\mathbf{o_{2}}...\oplus\mathbf{o_{m}}\). This representation contains feature vectors generated from all sets of embeddings under consideration. We call this method _multiple group CNN_ (MG-CNN). Here groups refer to the features generated from different embeddings. Note that this differs from 'multi-channel' models because at the convolution layer we use different filters on each word embedding matrix independently, whereas in a standard multi-channel approach each filter would consider all channels simultaneously and generate a scalar from all channels at each local region. As above, we impose a max l2 norm constraint on the final feature vector \(\mathbf{o}\) for regularization. Figure 1 illustrates this approach.

**Mgnc-Cnn**. We propose an augmentation of MG-CNN, _Multi-Group Norm Constraint CNN_ (MGNC-CNN), which differs in its regularization strategy. Specifically, in this variant we impose _grouped_ regularization constraints, independently regularizing subcomponents \(\mathbf{o}_{l}\) derived from the respective embeddings, i.e., we impose separate max norm constraints \(\lambda_{l}\) for each \(\mathbf{o}_{l}\) (where \(l\) again indexes embedding sets); these \(\lambda_{l}\) hyper-parameters are to be tuned on a validation set. Intuitively, this method aims to better capitalize on features derived from word embeddings that capture discriminative properties of text for the task at hand by penalizing larger weight estimates for features derived from less discriminative embeddings.

## 4 Experiments

### Datasets

**Stanford Sentiment Treebank** Stanford Sentiment Treebank (SST) [30]. This concerns predicting movie review sentiment. Two datasets are derived from this corpus: (1) **SST-1**, containing five classes: _very negative_, _negative_, _neutral_, _positive_, and _very positive_. (2) **SST-2**, which has only two classes: _negative_ and _positive_. For both, we remove phrases of length less than 4 from the training set.1**Subj**[56]. The aim here is to classify sentences as either _subjective_ or _objective_. This comprises 5000 instances of each. **TREC**[18]. A question classification dataset containing six classes: _abbreviation_, _entity_, _description_, _human_, _location_ and _numeric_. There are 5500 training and 500 test instances. **Irony**[79]. This dataset contains 16,006 sentences from reddit labeled as ironic (or not). The dataset is imbalanced (relatively few sentences are ironic). Thus before training, we under-sampled negative instances to make classes sizes equal. Note that for this dataset we report the Area Under Curve (AUC), rather than accuracy, because it is imbalanced.

Figure 1: Illustration of MG-CNN and MGNC-CNN. The filters applied to the respective embeddings are completely independent. MG-CNN applies a max norm constraint to \(\mathbf{o}\), while MGNC-CNN applies max norm constraints on \(\mathbf{o_{1}}\) and \(\mathbf{o_{2}}\) independently (group regularization). Note that one may easily extend the approach to handle more than two embeddings at once.

 



### Pre-trained Word Embeddings

We consider three sets of word embeddings for our experiments: (i) **word2vec\({}^{2}\)** is trained on 100 billion tokens of Google News dataset; (ii) **GloVe**[4]3 is trained on aggregated global word-word co-occurrence statistics from Common Crawl (840B tokens); and (iii) **syntactic** word embedding trained on dependency-parsed corpora. These three embedding sets happen to all be 300-dimensional, but our model could accommodate arbitrary and variable sizes.

Footnote 3: [https://code.google.com/p/word2vec/](https://code.google.com/p/word2vec/)

Footnote 3: [http://nlp.stanford.edu/projects/glove/](http://nlp.stanford.edu/projects/glove/)

We pre-trained our own syntactic embeddings following Levy and Goldberg (2014). We parsed the ukWaC corpus Baroni et al. (2009) using the Stanford Dependency Parser v3.5.2 with Stanford Dependencies Chen and Manning (2014) and extracted (word, relation+context) pairs from parse trees. We "collapsed" nodes with prepositions and notated inverse relations separately, e.g., "dog barks" emits two tuples: _(barks, nsubj_dog)_ and _(dog, nsubj\({}^{-1}\)__barks)_. We filter words and contexts that appear fewer than 100 times, resulting in \(\sim\)173k words and 1M contexts. We trained 300d vectors using word2vecf4 with default parameters.

Footnote 4: [https://bitbucket.org/yoavgo/word2vecf/](https://bitbucket.org/yoavgo/word2vecf/)

### Setup

We compared our proposed approaches to a standard CNN that exploits a single set of word embeddings Kim (2014). We also compared to a baseline of simply concatenating embeddings for each word to form long vector inputs. We refer to this as Concatenation-CNN _C-CNN_. For all multiple embedding approaches (C-CNN, MG-CNN and MGNC-CNN), we explored two combined sets of embedding: word2vec+Glove, and word2vec+syntactic, and one three sets of embedding: word2vec+Glove+syntactic. For all models, we tuned the l2 norm constraint \(\lambda\) over the range \(\{\frac{1}{3},1,3,9,81,243\}\) on a validation set. For instan

tiations of MGNC-CNN in which we exploited two embeddings, we tuned both \(\lambda_{1}\), and \(\lambda_{2}\); where we used three embedding sets, we tuned \(\lambda_{1},\lambda_{2}\) and \(\lambda_{3}\).

We used standard train/test splits for those datasets that had them. Otherwise, we performed 10-fold cross validation, creating nested development sets with which to tune hyperparameters. For all experiments we used filters sizes of 3, 4 and 5 and we created 100 feature maps for each filter size. We applied 1 max-pooling and dropout (rate: 0.5) at the classification layer. For training we used back-propagation in mini-batches and used AdaDelta as the stochastic gradient descent (SGD) update rule, and set mini-batch size as 50. In this work, we treat word embeddings as part of the parameters of the model, and update them as well during training. In all our experiments, we only tuned the max norm constraint(s), fixing all other hyperparameters.

### Results and Discussion

We repeated each experiment 10 times and report the mean and ranges across these. This replication is important because training is stochastic and thus introduces variance in performance Zhang and Wallace (2015). Results are shown in Table 1, and the corresponding best norm constraint value is shown in Table 2. We also show results on _Subj_, _SST-1_ and _SST-2_ achieved by the more complex model of Yin and Schutze (2015) for comparison; this represents the state-of-the-art on the three datasets other than TREC.

We can see that **MGNC-CNN and MG-CNN always outperform baseline methods (including C-CNN), and MGNC-CNN is usually better than MG-CNN**. And on the _Subj_ dataset, MG-CNN actually achieves slightly better results than Yin and Schutze (2015), with far less complexity and required training time (MGNC-CNN performs comparably, although no better, here). On the TREC dataset, the best-ever accuracy we are aware of is 96.0% Mou et al. (2015), which falls within the range of the result of our MGNC-CNN model with three word embeddings. On the _irony_ dataset, our model with three embeddings achieves 4% improvement (in terms of AUC) compared to the baseline model.

On _SST-1_ and _SST-2_, our model performs slightly worse than Yin and Schutze (2015). However, we again note that their performance is achieved using a much more complex model which involves pretraining and mutual-learning steps. This model takes days to train, whereas our model requires on the order of an hour.

We note that the method proposed by Astudillo et al. (2015) is able to accommodate multiple embedding sets with different dimensions by projecting the original word embeddings into a lower-dimensional space. However, this work requires training the optimal projection matrix on laebled data first, which again incurs large overhead.

Of course, our model also has its own limitations: in MGNC-CNN, we need to tune the norm constraint hyperparameter for all the word embeddings. As the number of word embedding increases, this will increase the running time. However, this tuning procedure is embarrassingly parallel.

## 5 Conclusions

We have proposed MGNC-CNN: a simple, flexible CNN architecture for sentence classification that can exploit multiple, variable sized word embeddings. We demonstrated that this consistently achieves better results than a baseline architecture that exploits only a single set of word embeddings, and also a naive concatenation approach to capitalizing on multiple embeddings. Furthermore, our results are comparable to those achieved with a recently proposed model Yin and Schutze (2015) that is much more complex. However, our simple model is easy to implement and requires an order of magnitude less training time. Furthermore, our model is much more flexible than previous approaches, because it can accommodate variable-size word embeddings.

#### Acknowledgments

This work was supported in part by the Army Research Office (grant W911NF-14-1-0442) and by The Foundation for Science and Technology, Portugal (grant UTAP-EXPL/EEIESS/0031/2014). This work was also made possible by the support of the Texas Advanced Computer Center (TACC) at UT Austin 



## References

* [1]R. F Astudillo, S. Amir, W. Lin, M. Silva, and I. Trancoso (2015) Learning word representations from scarce and noisy data with embedding subspaces. In Proceedings of Association for Computational Linguistics, pp. 1074-1084. Cited by: SS3.1.
* [2]M. Baroni, S. Bernardini, A. Ferraresi, and E. Zanchetta (2009) The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation43 (3), pp. 209-226. External Links: ISSN 0021-909X, Link, Document Cited by: SS1, SS2.
* [3]E. Bruni, G. Boleda, M. Baroni, and N. Tran (2012) Distributional semantics in technicolor. In Proceedings of Association for Computational Linguistics, pp. 136-145. Cited by: SS1.
* [4]D. Chen and C. Manning (2014) A fast and accurate dependency parser using neural networks. In Proceedings of Empirical Methods in Natural Language Processing, pp. 740-750. Cited by: SS1.
* [5]K. Cho (2015) Natural language understanding with distributed representation. arXiv preprint arXiv:1511.07916. Cited by: SS1.
* [6]R. Collobert and J. Weston (2008) A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the International Conference on Machine learning, pp. 160-167. Cited by: SS1.
* [7]K. Erk and S. Pado (2008) A structured vector space model for word meaning in context. In Proceedings of Empirical Methods in Natural Language Processing, pp. 897-906. Cited by: SS1.
* [8]Y. Goldberg (2015) A primer on neural network models for natural language processing. arXiv preprint arXiv:1510.00726. Cited by: SS1.
* [9]R. Johnson and T. Zhang (2014) Effective use of word order for text categorization with convolutional neural networks. arXiv preprint arXiv:1412.1058. Cited by: SS1.
* [10]N. Kalchbrenner, E. Grefenstette, and P. Blunsom (2014) A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188. Cited by: SS1.
* [11]Y. Kim (2014) Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. Cited by: SS1.
* [12]O. Levy and Y. Goldberg (2014) Dependency-based word embeddings. In Proceedings of Association for Computational Linguistics, pp. 302-308. Cited by: SS1.
* [13]X. Li and D. Roth (2002) Learning question classifiers. In Proceedings of the International Conference on Computational Linguistics, pp. 1-7. Cited by: SS1.
* [14]Y. Luo, J. Tang, J. Yan, C. Xu, and Z. Chen (2014) Pre-trained multi-view word embedding using two-side neural network. In Conference on Artificial Intelligence, Cited by: SS1.
* [15]T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean (2013) Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111-3119. Cited by: SS1.
* [16]L. Mou, H. Peng, G. Li, Y. Xu, L. Zhang, and Z. Jin (2015) Discriminative neural sentence modeling by tree-based convolution. In Proceedings of Empirical Methods in Natural Language Processing, pp. 2315-2325. Cited by: SS1.
* [17]S. Pado and M. Lapata (2007) Dependency-based construction of semantic space models. Computational Linguistics33 (2), pp. 161-199. Cited by: SS1.
* [18]B. Pang and L. Lee (2004) A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of Association for Computational Linguistics, pp. 271. Cited by: SS1.
* [19]J. Pennington, R. Socher, and C. D. Manning (2014) Glove: global vectors for word representation. In Proceedings of the Empirical Methods in Natural Language Processing, pp. 1532-1543. Cited by: SS1.
* [20]R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of Empirical Methods in Natural Language Processing, pp. 1631-1642. Cited by: SS1.
* [21]T. Van de Cruys, Thierry Poibeau, and Anna Korhoen (2011) Latent vector weighting for word meaning in context. In Proceedings of Empirical Methods in Natural Language Processing, pp. 1012-1022. Cited by: SS1.
* [22]P. C. Wallace, D. Kook Choe, L. Kertz, and E. Charniak (2014) Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of Association for Computational Linguistics, pp. 512-516. Cited by: SS1.
* [23]A. Vaswani, V. Nair, Q. Shazeer, and B. Usai (2015) Attention-driven neural networks for sentence classification. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 1-10. Cited by: SS1.
* [24]W. Yin and H. Schutze (2015) Multichannel variable-size convolution for sentence classification. In Proceedings of the Conference on Computational Natural Language Learning, pp. 204-214. Cited by: SS1.
* [25]D. Yogatama and Noah Smith (2014) Making the most of bag of words: sentence regularization with alternating direction method of multipliers. In Proceedings of the International Conference on Machine Learning, pp. 656-664. Cited by: SS1.
* [26]Y. Zhang and B. C. Wallace (2015) A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820.