Adaptive Multi-Compositionality for Recursive Neural Models with Applications to Sentiment Analysis

Recursive neural models have achieved promising results in many natural language processing tasks. The main difference among these models lies in the composition function, i.e., how to obtain the vector representation for a phrase or sentence using the representations of words it contains. This paper introduces a novel Adaptive Multi-Compositionality (AdaMC) layer to recursive neural models. The basic idea is to use more than one composition functions and adaptively select them depending on the input vectors. We present a general framework to model each semantic composition as a distribution over these composition functions. The composition functions and parameters used for adaptive selection are learned jointly from data. We integrate AdaMC into existing recursive neural models and conduct extensive experiments on the Stanford Sentiment Treebank. The results illustrate that AdaMC significantly outperforms state-of-the-art sentiment classification methods. It helps push the best accuracy of sentence-level negative/positive classification from 85.4% up to 88.5%. Introduction Recursive Neural Models (RNMs), which utilize the recursive structure of the input (e.g., a sentence), are one family of popular deep learning models. They are particularly effective for many Natural Language Processing (NLP) tasks due to the compositional nature of natural language. Recently, many promising results have been reported on semantic relationship classification (Socher et al. 2012), syntactic parsing (Socher et al. 2013a), sentiment analysis (Socher et al. 2013b), and so on. The main difference among RNMs lies in the semantic composition method, i.e., how to obtain the vector representation for a phrase or sentence using the representations of words and phrases it contains. For instance, we can compute the word vector for the phrase “not good” with the vectors of the words “not” and “good”. For many tasks, we even need to obtain the vector representations for sentences. The composition algorithm becomes the key to make the vector representations go beyond words to phrases and sentences. ⇤Contribution during internship at Microsoft Research. Copyright c 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. There have been several attempts in literature to address the semantic composition for RNMs. Specifically, RNN (Socher et al. 2011) uses a global matrix to linearly combine the elements of vectors, while RNTN (Socher et al. 2013b) employs a global tensor to model the products of dimensions. Sometimes it is challenging to find a single powerful function to model the semantic composition. Intuitively, we can employ multiple composition functions, instead of only using a single global one. Instead of finding more complex composition functions, MV-RNN (Socher et al. 2012) assigns matrices for every words to make the compositions specific. However, the number of composition matrices is the same as vocabulary size, which makes the number of parameters quite large. It is easy to overfit the training data and difficult to be optimized. Moreover, MVRNN needs another global matrix to linearly combine the composition matrices for phrases, which still makes these compositions not specific. In order to overcome these shortcomings and make the compositions specific, it is better to use a certain number of composition functions, and embed the role-sensitive (linguistic and semantic) information into word vectors to adaptively select these compositions rather than concrete words. The example “not (so good)” in sentiment analysis illustrates this point. To obtain the polarity of this phrase, we firstly combine the words “so” and “good”, then combine the “not” and “so good”. Specifically, the first combination is a strengthen composition which makes the sentiment polarity stronger, and the second step is a negation composition which negates the positive polarity to negative. In this paper, we introduce a novel Adaptive MultiCompositionality (AdaMC) method for RNMs. AdaMC consists of more than one composition functions, and adaptively selects them depending on the input vectors. The model learns to embed the semantic categories of words into their corresponding word vectors, and uses them to choose these composition functions adaptively. Specifically, we propose a parametrization method to compute the probability distribution for every function given the child vectors. We also introduce a hyper-parameter to model the adaptive preferences over the different composition functions and show three special cases of AdaMC. By adjusting this hyperparameter, there is a continuous transition between these three special cases. Moreover, all these composition functions and how to select them are automatically learned from Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence