CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model

1 INTRODUCTION . Word embeddings are perceived as one of the most impactful contributions from unsupervised representation learning to natural language processing from the past few years ( Goth , 2016 ) . Word embeddings are learned once on a large-scale stream of words . A key benefit is that these pre-computed vectors can be re-used almost universally in many different downstream applications . Recently , there has been increasing interest in learning universal sentence embeddings . Perone et al . ( 2018 ) have shown that the best encoding architectures are based on recurrent neural networks ( RNNs ) ( Conneau et al. , 2017 ; Peters et al. , 2018 ) or the Transformer architecture ( Cer et al. , 2018 ) . These techniques are , however , substantially more expensive to train and apply than word embeddings ( Hill et al. , 2016 ; Cer et al. , 2018 ) . Their usefulness is therefore limited when fast processing of large volumes of data is critical . More efficient encoding techniques are typically based on aggregated word embeddings such as Continuous Bag of Words ( CBOW ) , which is a mere summation of the word vectors ( Mikolov et al. , 2013a ) . Despite CBOW ’ s simplicity , it attains strong results on many downstream tasks . Using sophisticated weighting schemes , the performance of aggregated word embeddings can be further increased ( Arora et al. , 2017 ) , coming even close to strong LSTM baselines ( Rücklé et al. , 2018 ; Henao et al. , 2018 ) such as InferSent ( Conneau et al. , 2017 ) . This raises the question how much benefit recurrent encoders actually provide over simple word embedding based methods ( Wieting & Kiela , 2019 ) . In their analysis , Henao et al . ( 2018 ) suggest that the main difference may be the ability to encode word order . In this paper , we propose an intuitive method to enhance aggregated word embeddings by word order awareness . The major drawback of these CBOW-like approaches is that they are solely based on addition . However , addition is not all you need . Since it is a commutative operation , the aforementioned methods are not able to capture any notion of word order . However , word order information is crucial for some tasks , e.g. , sentiment analysis ( Henao et al. , 2018 ) . For instance , the following two sentences yield the exact same embedding in an addition-based word embedding aggregation technique : “ The movie was not awful , it was rather great. ” and “ The movie was not great , it was rather awful. ” A classifier based on the CBOW embedding of these sentences would inevitably fail to distinguish the two different meanings ( Goldberg , 2017 , p. 151 ) . To alleviate this drawback , Rudolph & Giesbrecht ( 2010 ) propose to model each word as a matrix rather than a vector , and compose multiple word embeddings via matrix multiplication rather than addition . This so-called Compositional Matrix Space Model ( CMSM ) of language has powerful theoretical properties that subsume properties from vector-based models and symbolic approaches . The most obvious advantage is the non-commutativity of matrix multiplication as opposed to addition , which results in order-aware encodings . In contrast to vector-based word embeddings , there is so far no solution to effectively train the parameters of word matrices on large-scale unlabeled data . Training schemes from previous work were specifically designed for sentiment analysis ( Yessenalina & Cardie , 2011 ; Asaadi & Rudolph , 2017 ) . Those require complex , multi-stage initialization , which indicates the difficulty of training CMSMs . We show that CMSMs can be trained in a similar way as the well-known CBOW model of word2vec ( Mikolov et al. , 2013a ) . We make two simple yet critical changes to the initialization strategy and training objective of CBOW . Hence , we present the first unsupervised training scheme for CMSMs , which we call Continual Multiplication Of Words ( CMOW ) . We evaluate our model ’ s capability to capture linguistic properties in the encoded text . We find that CMOW and CBOW have properties that are complementary . On the one hand , CBOW yields much stronger results at the word content memorization task . CMOW , on the other hand , offers an advantage in all other linguistic probing tasks , often by a wide margin . Thus , we propose a hybrid model to jointly learn the word vectors of CBOW and the word matrices for CMOW . Our experimental results confirm the effectiveness of our hybrid CBOW-CMOW approach . At comparable embedding size , CBOW-CMOW retains CBOW ’ s ability to memorize word content while at the same time improves the performance on the linguistic probing tasks by 8 % . CBOW-CMOW outperforms CBOW at 8 out of 11 supervised downstream tasks scoring only 0.6 % lower on the tasks where CBOW is slightly better . On average , the hybrid model improves the performance over CBOW by 1.2 % on supervised downstream tasks , and by 0.5 % on the unsupervised tasks . In summary , our contributions are : ( 1 ) For the first time , we present an unsupervised , efficient training scheme for the Compositional Matrix Space Model . Key elements of our scheme are an initialization strategy and training objective that are specifically designed for training CMSMs . ( 2 ) We quantitatively demonstrate that the strengths of the resulting embedding model are complementary to classical CBOW embeddings . ( 3 ) We successfully combine both approaches into a hybrid model that is superior to its individual parts . After giving a brief overview of the related work , we formally introduce CBOW , CMOW , and the hybrid model in Section 3 . We describe our experimental setup and present the results in Section 4 . The results are discussed in Section 5 , before we conclude . 2 RELATED WORK . We present an algorithm for learning the weights of the Compositional Matrix Space Model ( Rudolph & Giesbrecht , 2010 ) . To the best of our knowledge , only Yessenalina & Cardie ( 2011 ) and Asaadi & Rudolph ( 2017 ) have addressed this . They present complex , multi-level initialization strategies to achieve reasonable results . Both papers train and evaluate their model on sentiment analysis datasets only , but they do not evaluate their CMSM as a general-purpose sentence encoder . Other works have represented words as matrices as well , but unlike our work not within the framework of the CMSM . Grefenstette & Sadrzadeh ( 2011 ) represent only relational words as matrices . Socher et al . ( 2012 ) and Chung & Bowman ( 2018 ) argue that while CMSMs are arguably more expressive than embeddings located in a vector space , the associativeness of matrix multiplication does not reflect the hierarchical structure of language . Instead , they represent the word sequence as a tree structure . Socher et al . ( 2012 ) directly represent each word as a matrix ( and a vector ) in a recursive neural network . Chung & Bowman ( 2018 ) present a two-layer architecture . In the first layer , pre-trained word embeddings are mapped to their matrix representation . In the second layer , a non-linear function composes the constituents . Sentence embeddings have recently become an active field of research . A desirable property of the embeddings is that the encoded knowledge is useful in a variety of high-level downstream tasks . To this end , Conneau & Kiela ( 2018 ) and Conneau et al . ( 2018 ) introduced an evaluation framework for sentence encoders that tests both their performance on downstream tasks as well as their ability to capture linguistic properties . Most works focus on either i ) the ability of encoders to capture appropriate semantics or on ii ) training objectives that give the encoders incentive to capture those semantics . Regarding the former , large RNNs are by far the most popular ( Conneau et al. , 2017 ; Kiros et al. , 2015 ; Tang et al. , 2017 ; Nie et al. , 2017 ; Hill et al. , 2016 ; McCann et al. , 2017 ; Peters et al. , 2018 ; Logeswaran & Lee , 2018 ) , followed by convolutional neural networks ( Gan et al. , 2017 ) . A third group are efficient methods that aggregate word embeddings ( Wieting et al. , 2016 ; Arora et al. , 2017 ; Pagliardini et al. , 2018 ; Rücklé et al. , 2018 ) . Most of the methods in the latter group are word order agnostic . Sent2Vec ( Pagliardini et al. , 2018 ) is an exception in the sense that they also incorporate bigrams . Despite also employing an objective similar to CBOW , their work is very different to ours in that they still use addition as composition function . Regarding the training objectives , there is an ongoing debate whether language modeling ( Peters et al. , 2018 ; Ruder & Howard , 2018 ) , machine translation ( McCann et al. , 2017 ) , natural language inference ( Conneau et al. , 2017 ) , paraphrase identification ( Wieting et al. , 2016 ) , or a mix of many tasks ( Subramanian et al. , 2018 ) is most appropriate for incentivizing the models to learn important aspects of language . In our study , we focus on adapting the well-known objective from word2vec ( Mikolov et al. , 2013a ) for the CMSM . 3 METHODS : CBOW AND CMOW . We formally present CBOW and CMOW encoders in a unified framework . Subsequently , we discuss the training objective , the initialization strategy , and the hybrid model . 3.1 TEXT ENCODING . We start with a lookup table for the word matrices , i.e. , an embedding , E ∈ Rm×d×d , where m is the vocabulary size and d is the dimensionality of the ( square ) matrices . We denote a specific word matrix of the embedding by E [ · ] . By ∆ ∈ { ∑ , ∏ } we denote the function that aggregates word embeddings into a sentence embedding . Formally , given a sequence s of arbitrary length n , the sequence is encoded as ∆ni=1E [ si ] . For ∆ = ∑ , the model becomes CBOW . By setting ∆ = ∏ ( matrix multiplication ) , we obtain CMOW . Because the result of the aggregation for any prefix of the sequence is again a square matrix of shape d × d irrespective of the aggregation function , the model is well defined for any non-zero sequence length . Thus , it can serve as a general-purpose text encoder . Throughout the remainder of this paper , we denote the encoding step by encE∆ ( s ) : = flatten ( ∆ni=1E [ si ] ) , where flatten concatenates the columns of the matrices to obtain a vector that can be passed to the next layer . 3.2 TRAINING OBJECTIVE . Motivated by its success , we employ a similar training objective as word2vec ( Mikolov et al. , 2013b ) . The objective consists of maximizing the conditional probability of a word wO in a certain context s : p ( wO | s ) . For a word wt at position t within a sentence , we consider the window of tokens ( wt−c , . . . , wt+c ) around that word . From that window , a target word wO : = { wt+i } , i ∈ { −c , . . . , +c } is selected . The remaining 2c words in the window are used as the context s. The training itself is conducted via negative sampling NEG-k , which is an efficient approximation of the softmax ( Mikolov et al. , 2013b ) . For each positive example , k negative examples ( noise words ) are drawn from some noise distribution Pn ( w ) . The goal is to distinguish the target word wO from the randomly sampled noise words . Given the encoded input words enc∆ ( s ) , a logistic regression with weights v ∈ Rm×d2 is conducted to predict 1 for context words and 0 for noise words . The negative sampling training objective becomes : log σ ( vTwO enc E ∆ ( s ) ) + k∑ i=1 Ewi∼Pn ( w ) [ log σ ( −vTwi enc E ∆ ( s ) ) ] ( 1 ) In the original word2vec ( Mikolov et al. , 2013a ) , the center word wO : = wt is used as the target word . In our experiments , however , this objective did not yield to satisfactory results . We hypothesize that this objective is too easy to solve for a word order-aware text encoder , which diminishes incentive for the encoder to capture semantic information at the sentence level . Instead , we propose to select a random output word wO ∼ U ( { wt−c , . . . , wt+c } ) from the window . The rationale is the following : By removing the information at which position the word was removed from the window , the model is forced to build a semantically rich representation of the whole sentence . For CMOW , modifying the objective leads to a large improvement on downstream tasks by 20.8 % on average , while it does not make a difference for CBOW . We present details in the appendix ( Section B.1 ) .

Summary: The paper presents new training schemes and experiments for a matrix-multiplicative variant of CBOW. This variant is called a CMSM (Yessenalina and Cardie, 2011; Asaadi and Rudolph, 2017) which swaps the bag of vectors to a product of square matrices for encoding context to incorporate word ordering. It seems this model has not been trained successfully before (at least with a simple approach) due to the vanishing gradient problem.