CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model

1 INTRODUCTION . Word embeddings are perceived as one of the most impactful contributions from unsupervised representation learning to natural language processing from the past few years ( Goth , 2016 ) . Word embeddings are learned once on a large-scale stream of words . A key benefit is that these pre-computed vectors can be re-used almost universally in many different downstream applications . Recently , there has been increasing interest in learning universal sentence embeddings . Perone et al . ( 2018 ) have shown that the best encoding architectures are based on recurrent neural networks ( RNNs ) ( Conneau et al. , 2017 ; Peters et al. , 2018 ) or the Transformer architecture ( Cer et al. , 2018 ) . These techniques are , however , substantially more expensive to train and apply than word embeddings ( Hill et al. , 2016 ; Cer et al. , 2018 ) . Their usefulness is therefore limited when fast processing of large volumes of data is critical . More efficient encoding techniques are typically based on aggregated word embeddings such as Continuous Bag of Words ( CBOW ) , which is a mere summation of the word vectors ( Mikolov et al. , 2013a ) . Despite CBOW ’ s simplicity , it attains strong results on many downstream tasks . Using sophisticated weighting schemes , the performance of aggregated word embeddings can be further increased ( Arora et al. , 2017 ) , coming even close to strong LSTM baselines ( Rücklé et al. , 2018 ; Henao et al. , 2018 ) such as InferSent ( Conneau et al. , 2017 ) . This raises the question how much benefit recurrent encoders actually provide over simple word embedding based methods ( Wieting & Kiela , 2019 ) . In their analysis , Henao et al . ( 2018 ) suggest that the main difference may be the ability to encode word order . In this paper , we propose an intuitive method to enhance aggregated word embeddings by word order awareness . The major drawback of these CBOW-like approaches is that they are solely based on addition . However , addition is not all you need . Since it is a commutative operation , the aforementioned methods are not able to capture any notion of word order . However , word order information is crucial for some tasks , e.g. , sentiment analysis ( Henao et al. , 2018 ) . For instance , the following two sentences yield the exact same embedding in an addition-based word embedding aggregation technique : “ The movie was not awful , it was rather great. ” and “ The movie was not great , it was rather awful. ” A classifier based on the CBOW embedding of these sentences would inevitably fail to distinguish the two different meanings ( Goldberg , 2017 , p. 151 ) . To alleviate this drawback , Rudolph & Giesbrecht ( 2010 ) propose to model each word as a matrix rather than a vector , and compose multiple word embeddings via matrix multiplication rather than addition . This so-called Compositional Matrix Space Model ( CMSM ) of language has powerful theoretical properties that subsume properties from vector-based models and symbolic approaches . The most obvious advantage is the non-commutativity of matrix multiplication as opposed to addition , which results in order-aware encodings . In contrast to vector-based word embeddings , there is so far no solution to effectively train the parameters of word matrices on large-scale unlabeled data . Training schemes from previous work were specifically designed for sentiment analysis ( Yessenalina & Cardie , 2011 ; Asaadi & Rudolph , 2017 ) . Those require complex , multi-stage initialization , which indicates the difficulty of training CMSMs . We show that CMSMs can be trained in a similar way as the well-known CBOW model of word2vec ( Mikolov et al. , 2013a ) . We make two simple yet critical changes to the initialization strategy and training objective of CBOW . Hence , we present the first unsupervised training scheme for CMSMs , which we call Continual Multiplication Of Words ( CMOW ) . We evaluate our model ’ s capability to capture linguistic properties in the encoded text . We find that CMOW and CBOW have properties that are complementary . On the one hand , CBOW yields much stronger results at the word content memorization task . CMOW , on the other hand , offers an advantage in all other linguistic probing tasks , often by a wide margin . Thus , we propose a hybrid model to jointly learn the word vectors of CBOW and the word matrices for CMOW . Our experimental results confirm the effectiveness of our hybrid CBOW-CMOW approach . At comparable embedding size , CBOW-CMOW retains CBOW ’ s ability to memorize word content while at the same time improves the performance on the linguistic probing tasks by 8 % . CBOW-CMOW outperforms CBOW at 8 out of 11 supervised downstream tasks scoring only 0.6 % lower on the tasks where CBOW is slightly better . On average , the hybrid model improves the performance over CBOW by 1.2 % on supervised downstream tasks , and by 0.5 % on the unsupervised tasks . In summary , our contributions are : ( 1 ) For the first time , we present an unsupervised , efficient training scheme for the Compositional Matrix Space Model . Key elements of our scheme are an initialization strategy and training objective that are specifically designed for training CMSMs . ( 2 ) We quantitatively demonstrate that the strengths of the resulting embedding model are complementary to classical CBOW embeddings . ( 3 ) We successfully combine both approaches into a hybrid model that is superior to its individual parts . After giving a brief overview of the related work , we formally introduce CBOW , CMOW , and the hybrid model in Section 3 . We describe our experimental setup and present the results in Section 4 . The results are discussed in Section 5 , before we conclude . 2 RELATED WORK . We present an algorithm for learning the weights of the Compositional Matrix Space Model ( Rudolph & Giesbrecht , 2010 ) . To the best of our knowledge , only Yessenalina & Cardie ( 2011 ) and Asaadi & Rudolph ( 2017 ) have addressed this . They present complex , multi-level initialization strategies to achieve reasonable results . Both papers train and evaluate their model on sentiment analysis datasets only , but they do not evaluate their CMSM as a general-purpose sentence encoder . Other works have represented words as matrices as well , but unlike our work not within the framework of the CMSM . Grefenstette & Sadrzadeh ( 2011 ) represent only relational words as matrices . Socher et al . ( 2012 ) and Chung & Bowman ( 2018 ) argue that while CMSMs are arguably more expressive than embeddings located in a vector space , the associativeness of matrix multiplication does not reflect the hierarchical structure of language . Instead , they represent the word sequence as a tree structure . Socher et al . ( 2012 ) directly represent each word as a matrix ( and a vector ) in a recursive neural network . Chung & Bowman ( 2018 ) present a two-layer architecture . In the first layer , pre-trained word embeddings are mapped to their matrix representation . In the second layer , a non-linear function composes the constituents . Sentence embeddings have recently become an active field of research . A desirable property of the embeddings is that the encoded knowledge is useful in a variety of high-level downstream tasks . To this end , Conneau & Kiela ( 2018 ) and Conneau et al . ( 2018 ) introduced an evaluation framework for sentence encoders that tests both their performance on downstream tasks as well as their ability to capture linguistic properties . Most works focus on either i ) the ability of encoders to capture appropriate semantics or on ii ) training objectives that give the encoders incentive to capture those semantics . Regarding the former , large RNNs are by far the most popular ( Conneau et al. , 2017 ; Kiros et al. , 2015 ; Tang et al. , 2017 ; Nie et al. , 2017 ; Hill et al. , 2016 ; McCann et al. , 2017 ; Peters et al. , 2018 ; Logeswaran & Lee , 2018 ) , followed by convolutional neural networks ( Gan et al. , 2017 ) . A third group are efficient methods that aggregate word embeddings ( Wieting et al. , 2016 ; Arora et al. , 2017 ; Pagliardini et al. , 2018 ; Rücklé et al. , 2018 ) . Most of the methods in the latter group are word order agnostic . Sent2Vec ( Pagliardini et al. , 2018 ) is an exception in the sense that they also incorporate bigrams . Despite also employing an objective similar to CBOW , their work is very different to ours in that they still use addition as composition function . Regarding the training objectives , there is an ongoing debate whether language modeling ( Peters et al. , 2018 ; Ruder & Howard , 2018 ) , machine translation ( McCann et al. , 2017 ) , natural language inference ( Conneau et al. , 2017 ) , paraphrase identification ( Wieting et al. , 2016 ) , or a mix of many tasks ( Subramanian et al. , 2018 ) is most appropriate for incentivizing the models to learn important aspects of language . In our study , we focus on adapting the well-known objective from word2vec ( Mikolov et al. , 2013a ) for the CMSM . 3 METHODS : CBOW AND CMOW . We formally present CBOW and CMOW encoders in a unified framework . Subsequently , we discuss the training objective , the initialization strategy , and the hybrid model . 3.1 TEXT ENCODING . We start with a lookup table for the word matrices , i.e. , an embedding , E ∈ Rm×d×d , where m is the vocabulary size and d is the dimensionality of the ( square ) matrices . We denote a specific word matrix of the embedding by E [ · ] . By ∆ ∈ { ∑ , ∏ } we denote the function that aggregates word embeddings into a sentence embedding . Formally , given a sequence s of arbitrary length n , the sequence is encoded as ∆ni=1E [ si ] . For ∆ = ∑ , the model becomes CBOW . By setting ∆ = ∏ ( matrix multiplication ) , we obtain CMOW . Because the result of the aggregation for any prefix of the sequence is again a square matrix of shape d × d irrespective of the aggregation function , the model is well defined for any non-zero sequence length . Thus , it can serve as a general-purpose text encoder . Throughout the remainder of this paper , we denote the encoding step by encE∆ ( s ) : = flatten ( ∆ni=1E [ si ] ) , where flatten concatenates the columns of the matrices to obtain a vector that can be passed to the next layer . 3.2 TRAINING OBJECTIVE . Motivated by its success , we employ a similar training objective as word2vec ( Mikolov et al. , 2013b ) . The objective consists of maximizing the conditional probability of a word wO in a certain context s : p ( wO | s ) . For a word wt at position t within a sentence , we consider the window of tokens ( wt−c , . . . , wt+c ) around that word . From that window , a target word wO : = { wt+i } , i ∈ { −c , . . . , +c } is selected . The remaining 2c words in the window are used as the context s. The training itself is conducted via negative sampling NEG-k , which is an efficient approximation of the softmax ( Mikolov et al. , 2013b ) . For each positive example , k negative examples ( noise words ) are drawn from some noise distribution Pn ( w ) . The goal is to distinguish the target word wO from the randomly sampled noise words . Given the encoded input words enc∆ ( s ) , a logistic regression with weights v ∈ Rm×d2 is conducted to predict 1 for context words and 0 for noise words . The negative sampling training objective becomes : log σ ( vTwO enc E ∆ ( s ) ) + k∑ i=1 Ewi∼Pn ( w ) [ log σ ( −vTwi enc E ∆ ( s ) ) ] ( 1 ) In the original word2vec ( Mikolov et al. , 2013a ) , the center word wO : = wt is used as the target word . In our experiments , however , this objective did not yield to satisfactory results . We hypothesize that this objective is too easy to solve for a word order-aware text encoder , which diminishes incentive for the encoder to capture semantic information at the sentence level . Instead , we propose to select a random output word wO ∼ U ( { wt−c , . . . , wt+c } ) from the window . The rationale is the following : By removing the information at which position the word was removed from the window , the model is forced to build a semantically rich representation of the whole sentence . For CMOW , modifying the objective leads to a large improvement on downstream tasks by 20.8 % on average , while it does not make a difference for CBOW . We present details in the appendix ( Section B.1 ) .

Summary: The authors propose CMOW, an extension of the CBOW model that allows the model to capture word order. Instead of each word being represented as a vector, words are represented by matrices. They extend the CBOW objective to take into account word order by replacing the averaging of vectors to create the context with matrix multiplication (a non-commutative operation). This is the first time this model has been applied in a large scale unsupervised setting. They are able to do this using their objective and an initialization strategy where the matrix embeddings are set to the identity matrix with some Gaussian noise added.