Efficient Vector Representation for Documents through Corruption

Abstract: We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a simple average of word embeddings. It ensures a representation generated as such captures the semantic meanings of the document during learning. A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero. Doc2VecC produces significantly better word embeddings than Word2Vec. We compare Doc2VecC with several state-of-the-art document representation learning algorithms. The simple model architecture introduced by Doc2VecC matches or out-performs the state-of-the-art in generating high-quality document representations for sentiment analysis, document classification as well as semantic relatedness tasks. The simplicity of the model enables training on billions of words per hour on a single machine. At the same time, the model is very efficient in generating representations of unseen documents at test time.

Text understanding starts with the challenge of finding machine-understandable representation that captures the semantics of texts. Bag-of-words (BoW) and its N-gram extensions are arguably the most commonly used document representations. Despite its simplicity, BoW works surprisingly well for many tasks BIBREF0 . However, by treating words and phrases as unique and discrete symbols, BoW often fails to capture the similarity between words or phrases and also suffers from sparsity and high dimensionality.
Recent works on using neural networks to learn distributed vector representations of words have gained great popularity. The well celebrated Word2Vec BIBREF1 , by learning to predict the target word using its neighboring words, maps words of similar meanings to nearby points in the continuous vector space. The surprisingly simple model has succeeded in generating high-quality word embeddings for tasks such as language modeling, text understanding and machine translation. Word2Vec naturally scales to large datasets thanks to its simple model architecture. It can be trained on billions of words per hour on a single machine.
Paragraph Vectors BIBREF2 generalize the idea to learn vector representation for documents. A target word is predicted by the word embeddings of its neighbors in together with a unique document vector learned for each document. It outperforms established document representations, such as BoW and Latent Dirichlet Allocation BIBREF3 , on various text understanding tasks BIBREF4 . However, two caveats come with this approach: 1) the number of parameters grows with the size of the training corpus, which can easily go to billions; and 2) it is expensive to generate vector representations for unseen documents at test time.
We propose an efficient model architecture, referred to as Document Vector through Corruption (Doc2VecC), to learn vector representations for documents. It is motivated by the observation that linear operations on the word embeddings learned by Word2Vec can sustain substantial amount of syntactic and semantic meanings of a phrase or a sentence BIBREF5 . For example, vec(“Russia”) + vec(“river”) is close to vec(“Volga River”) BIBREF6 , and vec(“king”) - vec(“man”) + vec(“women”) is close to vec(“queen”) BIBREF5 . In Doc2VecC, we represent each document as a simple average of the word embeddings of all the words in the document. In contrast to existing approaches which post-process learned word embeddings to form document representation BIBREF7 , BIBREF8 , Doc2VecC enforces a meaningful document representation can be formed by averaging the word embeddings during learning. Furthermore, we include a corruption model that randomly remove words from a document during learning, a mechanism that is critical to the performance and learning speed of our algorithm.
Doc2VecC has several desirable properties: 1. The model complexity of Doc2VecC is decoupled from the size of the training corpus, depending only on the size of the vocabulary; 2. The model architecture of Doc2VecC resembles that of Word2Vec, and can be trained very efficiently; 3. The new framework implicitly introduces a data-dependent regularization, which favors rare or informative words and suppresses words that are common but not discriminative; 4. Vector representation of a document can be generated by simply averaging the learned word embeddings of all the words in the document, which significantly boost test efficiency; 5. The vector representation generated by Doc2VecC matches or beats the state-of-the-art for sentiment analysis, document classification as well as semantic relatedness tasks.
Text representation learning has been extensively studied. Popular representations range from the simplest BoW and its term-frequency based variants BIBREF9 , language model based methods BIBREF10 , BIBREF11 , BIBREF12 , topic models BIBREF13 , BIBREF3 , Denoising Autoencoders and its variants BIBREF14 , BIBREF15 , and distributed vector representations BIBREF8 , BIBREF2 , BIBREF16 . Another prominent line of work includes learning task-specific document representation with deep neural networks, such as CNN BIBREF17 or LSTM based approaches BIBREF18 , BIBREF19 .
In this section, we briefly introduce Word2Vec and Paragraph Vectors, the two approaches that are most similar to ours. There are two well-know model architectures used for both methods, referred to as Continuous Bag-of-Words (CBoW) and Skipgram models BIBREF1 . In this work, we focus on CBoW. Extending to Skipgram is straightforward. Here are the notations we are going to use throughout the paper:
Several works BIBREF6 , BIBREF5 showcased that syntactic and semantic regularities of phrases and sentences are reasonably well preserved by adding or subtracting word embeddings learned through Word2Vec. It prompts us to explore the option of simply representing a document as an average of word embeddings. Figure FIGREF9 illustrates the new model architecture.
Similar to Word2Vec or Paragraph Vectors, Doc2VecC consists of an input layer, a projection layer as well as an output layer to predict the target word, “ceremony” in this example. The embeddings of neighboring words (“opening”, “for”, “the”) provide local context while the vector representation of the entire document (shown in grey) serves as the global context. In contrast to Paragraph Vectors, which directly learns a unique vector for each document, Doc2VecC represents each document as an average of the embeddings of words randomly sampled from the document (“performance” at position INLINEFORM0 , “praised” at position INLINEFORM1 , and “brazil” at position INLINEFORM2 ). BIBREF25 also proposed the idea of using average of word embeddings to represent the global context of a document. Different from their work, we choose to corrupt the original document by randomly removing significant portion of words, and represent the document using only the embeddings of the words remained. This corruption mechanism offers us great speedup during training as it significantly reduces the number of parameters to update in back propagation. At the same time, as we are going to detail in the next section, it introduces a special form of regularization, which brings great performance improvement.
Here we describe the stochastic process we used to generate a global context at each update. The global context, which we denote as INLINEFORM0 , is generated through a unbiased mask-out/drop-out corruption, in which we randomly overwrites each dimension of the original document INLINEFORM1 with probability INLINEFORM2 . To make the corruption unbiased, we set the uncorrupted dimensions to INLINEFORM3 times its original value. Formally, DISPLAYFORM0 
Doc2VecC then defines the probability of observing a target word INLINEFORM0 given its local context INLINEFORM1 as well as the global context INLINEFORM2 as DISPLAYFORM0 
Here INLINEFORM0 is the length of the document. Exactly computing the probability is impractical, instead we approximate it with negative sampling BIBREF1 . DISPLAYFORM0 
here INLINEFORM0 stands for a uniform distribution over the terms in the vocabulary. The two projection matrices INLINEFORM1 and INLINEFORM2 are then learned to minimize the loss: DISPLAYFORM0 
Given the learned projection matrix INLINEFORM0 , we then represent each document simply as an average of the embeddings of the words in the document, DISPLAYFORM0 
We are going to elaborate next why we choose to corrupt the original document with the corruption model in eq.( EQREF10 ) during learning, and how it enables us to simply use the average word embeddings as the vector representation for documents at test time.
We approximate the log likelihood for each instance INLINEFORM0 in eq.( EQREF13 ) with its Taylor expansion with respect to INLINEFORM1 up to the second-order BIBREF26 , BIBREF27 , BIBREF28 . Concretely, we choose to expand at the mean of the corruption INLINEFORM2 : INLINEFORM3 
where INLINEFORM0 and INLINEFORM1 are the first-order (i.e., gradient) and second-order (i.e., Hessian) of the log likelihood with respect to INLINEFORM2 . Expansion at the mean INLINEFORM3 is crucial as shown in the following steps. Let us assume that for each instance, we are going to sample the global context INLINEFORM4 infinitely many times, and thus compute the expected log likelihood with respect to the corrupted INLINEFORM5 . INLINEFORM6 
The linear term disappears as INLINEFORM0 . We substitute in INLINEFORM1 for the mean INLINEFORM2 of the corrupting distribution (unbiased corruption) and the matrix INLINEFORM3 for the variance, and obtain DISPLAYFORM0 
As each word in a document is corrupted independently of others, the variance matrix INLINEFORM0 is simplified to a diagonal matrix with INLINEFORM1 element equals INLINEFORM2 . As a result, we only need to compute the diagonal terms of the Hessian matrix INLINEFORM3 .
The INLINEFORM0 dimension of the Hessian's diagonal evaluated at the mean INLINEFORM1 is given by INLINEFORM2 
Plug the Hessian matrix and the variance matrix back into eq.( EQREF16 ), and then back to the loss defined in eq.( EQREF13 ), we can see that Doc2VecC intrinsically minimizes DISPLAYFORM0 
Each INLINEFORM0 in the first term measures the log likelihood of observing the target word INLINEFORM1 given its local context INLINEFORM2 and the document vector INLINEFORM3 . As such, Doc2VecC enforces that a document vector generated by averaging word embeddings can capture the global semantics of the document, and fill in information missed in the local context. The second term here is a data-dependent regularization. The regularization on the embedding INLINEFORM4 of each word INLINEFORM5 takes the following form, INLINEFORM6 
where INLINEFORM0 prescribes the confidence of predicting the target word INLINEFORM1 given its neighboring context INLINEFORM2 as well as the document vector INLINEFORM3 .
Closely examining INLINEFORM0 leads to several interesting findings: 1. the regularizer penalizes more on the embeddings of common words. A word INLINEFORM1 that frequently appears across the training corpus, i.e, INLINEFORM2 often, will have a bigger regularization than a rare word; 2. on the other hand, the regularization is modulated by INLINEFORM3 , which is small if INLINEFORM4 . In other words, if INLINEFORM5 is critical to a confident prediction INLINEFORM6 when it is active, then the regularization is diminished. Similar effect was observed for dropout training for logistic regression model BIBREF27 and denoising autoencoders BIBREF28 .
We evaluate Doc2VecC on a sentiment analysis task, a document classification task and a semantic relatedness task, along with several document representation learning algorithms. All experiments can be reproduced using the code available at https://github.com/mchen24/iclr2017
We compare against the following document representation baselines: bag-of-words (BoW); Denoising Autoencoders (DEA) BIBREF14 , a representation learned from reconstructing original document INLINEFORM0 using corrupted one INLINEFORM1 . SDAs have been shown to be the state-of-the-art for sentiment analysis tasks BIBREF29 . We used Kullback-Liebler divergence as the reconstruction error and an affine encoder. To scale up the algorithm to large vocabulary, we only take into account the non-zero elements of INLINEFORM2 in the reconstruction error and employed negative sampling for the remainings; Word2Vec BIBREF1 +IDF, a representation generated through weighted average of word vectors learned using Word2Vec; Doc2Vec BIBREF2 ; Skip-thought Vectors BIBREF16 , a generic, distributed sentence encoder that extends the Word2Vec skip-gram model to sentence level. It has been shown to produce highly generic sentence representations that apply to various natural language processing tasks. We also include RNNLM BIBREF11 , a recurrent neural network based language model in the comparison. In the semantic relatedness task, we further compare to LSTM-based methods BIBREF18 that have been reported on this dataset.
For sentiment analysis, we use the IMDB movie review dataset. It contains 100,000 movies reviews categorized as either positive or negative. It comes with predefined train/test split BIBREF30 : 25,000 reviews are used for training, 25,000 for testing, and the rest as unlabeled data. The two classes are balanced in the training and testing sets. We remove words that appear less than 10 times in the training set, resulting in a vocabulary of 43,375 distinct words and symbols.
Setup. We test the various representation learning algorithms under two settings: one follows the same protocol proposed in BIBREF8 , where representation is learned using all the available data, including the test set; another one where the representation is learned using training and unlabeled set only. For both settings, a linear support vector machine (SVM) BIBREF31 is trained afterwards on the learned representation for classification. For Skip-thought Vectors, we used the generic model trained on a much bigger book corpus to encode the documents. A vector of 4800 dimensions, first 2400 from the uni-skip model, and the last 2400 from the bi-skip model, are generated for each document. In comparison, all the other algorithms produce a vector representation of size 100. The supervised RNN-LM is learned on the training set only. The hyper-parameters are tuned on a validation set subsampled from the training set.
Accuracy. Comparing the two columns in Table TABREF20 , we can see that all the representation learning algorithms benefits from including the testing data during the representation learning phrase. Doc2VecC achieved similar or even better performance than Paragraph Vectors. Both methods outperforms the other baselines, beating the BOW representation by 15%. In comparison with Word2Vec+IDF, which applies post-processing on learned word embeddings to form document representation, Doc2VecC naturally enforces document semantics to be captured by averaged word embeddings during training. This leads to better performance. Doc2VecC reduces to Denoising Autoencoders (DEA) if the local context words are removed from the paradigm shown in Figure FIGREF9 . By including the context words, Doc2VecC allows the document vector to focus more on capturing the global context. Skip-thought vectors perform surprisingly poor on this dataset comparing to other methods. We hypothesized that it is due to the length of paragraphs in this dataset. The average length of paragraphs in the IMDB movie review dataset is INLINEFORM0 , much longer than the ones used for training and testing in the original paper, which is in the order of 10. As noted in BIBREF18 , the performance of LSTM based method (similarly, the gated RNN used in Skip-thought vectors) drops significantly with increasing paragraph length, as it is hard to preserve state over long sequences of words.
Time. Table TABREF22 summarizes the time required by these algorithms to learn and generate the document representation. Word2Vec is the fastest one to train. Denoising Autoencoders and Doc2VecC second that. The number of parameters that needs to be back-propagated in each update was increased by the number of surviving words in INLINEFORM0 . We found that both models are not sensitive to the corruption rate INLINEFORM1 in the noise model. Since the learning time decreases with higher corruption rate, we used INLINEFORM2 throughout the experiments. Paragraph Vectors takes longer time to train as there are more parameters (linear to the number of document in the learning set) to learn. At test time, Word2Vec+IDF, DEA and Doc2VecC all use (weighted) averaging of word embeddings as document representation. Paragraph Vectors, on the other hand, requires another round of inference to produce the vector representation of unseen test documents. It takes Paragraph Vectors 4 minutes and 17 seconds to infer the vector representations for the 25,000 test documents, in comparison to 7 seconds for the other methods. As we did not re-train the Skip-thought vector models on this dataset, the training time reported in the table is the time it takes to generate the embeddings for the 25,000 training documents. Due to repeated high-dimensional matrix operations required for encoding long paragraphs, it takes fairly long time to generate the representations for these documents. Similarly for testing. The experiments were conducted on a desktop with Intel i7 2.2Ghz cpu.
Data dependent regularization. As explained in Section SECREF15 , the corruption introduced in Doc2VecC acts as a data-dependent regularization that suppresses the embeddings of frequent but uninformative words. Here we conduct an experiment to exam the effect. We used a cutoff of 100 in this experiment. Table TABREF24 lists the words having the smallest INLINEFORM0 norm of embeddings found by different algorithms. The number inside the parenthesis after each word is the number of times this word appears in the learning set. In word2Vec or Paragraph Vectors, the least frequent words have embeddings that are close to zero, despite some of them being indicative of sentiment such as debacle, bliss and shabby. In contrast, Doc2VecC manages to clamp down the representation of words frequently appear in the training set, but are uninformative, such as symbols and stop words.
Subsampling frequent words. Note that for all the numbers reported, we applied the trick of subsampling of frequent words introduced in BIBREF6 to counter the imbalance between frequent and rare words. It is critical to the performance of simple Word2Vec+AVG as the sole remedy to diminish the contribution of common words in the final document representation. If we were to remove this step, the error rate of Word2Vec+AVG will increases from INLINEFORM0 to INLINEFORM1 . Doc2VecC on the other hand naturally exerts a stronger regularization toward embeddings of words that are frequent but uninformative, therefore does not rely on this trick.
In table TABREF24 , we demonstrated that the corruption model introduced in Doc2VecC dampens the embeddings of words which are common and non-discriminative (stop words). In this experiment, we are going to quantatively compare the word embeddings generated by Doc2VecC to the ones generated by Word2Vec, or Paragraph Vectors on the word analogy task introduced by BIBREF1 . The dataset contains five types of semantic questions, and nine types of syntactic questions, with a total of 8,869 semantic and 10,675 syntactic questions. The questions are answered through simple linear algebraic operations on the word embeddings generated by different methods. Please refer to the original paper for more details on the evaluation protocol.
We trained the word embeddings of different methods using the English news dataset released under the ACL workshop on statistical machine translation. The training set includes close to 15M paragraphs with 355M tokens. We compare the performance of word embeddings trained by different methods with increasing embedding dimensionality as well as increasing training data.
We observe similar trends as in BIBREF1 . Increasing embedding dimensionality as well as training data size improves performance of the word embeddings on this task. However, the improvement is diminishing. Doc2VecC produces word embeddings which performs significantly better than the ones generated by Word2Vec. We observe close to INLINEFORM0 uplift when we train on the full training corpus. Paragraph vectors on the other hand performs surprisingly bad on this dataset. Our hypothesis is that due to the large capacity of the model architecture, Paragraph Vectors relies mostly on the unique document vectors to capture the information in a text document instead of learning the word semantic or syntactic similarities. This also explains why the PV-DBOW BIBREF2 model architecture proposed in the original work, which completely removes word embedding layers, performs comparable to the distributed memory version.
In table 5, we list a detailed comparison of the performance of word embeddings generated by Word2Vec and Doc2VecC on the 14 subtasks, when trained on the full dataset with embedding of size 100. We can see that Doc2VecC significantly outperforms the word embeddings produced by Word2Vec across almost all the subtasks.
For the document classification task, we use a subset of the wikipedia dump, which contains over 300,000 wikipedia pages in 100 categories. The 100 categories includes categories under sports, entertainment, literature, and politics etc. Examples of categories include American drama films, Directorial debut films, Major League Baseball pitchers and Sydney Swans players. Body texts (the second paragraph) were extracted for each page as a document. For each category, we select 1,000 documents with unique category label, and 100 documents were used for training and 900 documents for testing. The remaining documents are used as unlabeled data. The 100 classes are balanced in the training and testing sets. For this data set, we learn the word embedding and document representation for all the algorithms using all the available data. We apply a cutoff of 10, resulting in a vocabulary of size INLINEFORM0 .
Table TABREF29 summarizes the classification error of a linear SVM trained on representations of different sizes. We can see that most of the algorithms are not sensitive to the size of the vector representation. Doc2Vec benefits most from increasing representation size. Across all sizes of representations, Doc2VecC outperform the existing algorithms by a significant margin. In fact, Doc2VecC can achieve same or better performance with a much smaller representation vector.
Figure FIGREF30 visualizes the document representations learned by Doc2Vec (left) and Doc2VecC (right) using t-SNE BIBREF32 . We can see that documents from the same category are nicely clustered using the representation generated by Doc2VecC. Doc2Vec, on the other hand, does not produce a clear separation between different categories, which explains its worse performance reported in Table TABREF29 .
Figure FIGREF31 visualizes the vector representation generated by Doc2VecC w.r.t. coarser categorization. we manually grouped the 100 categories into 7 coarse categories, television, albums, writers, musicians, athletes, species and actors. Categories that do no belong to any of these 7 groups are not included in the figure. We can see that documents belonging to a coarser category are grouped together. This subset includes is a wide range of sports descriptions, ranging from football, crickets, baseball, and cycling etc., which explains why the athletes category are less concentrated. In the projection, we can see documents belonging to the musician category are closer to those belonging to albums category than those of athletes or species.
We test Doc2VecC on the SemEval 2014 Task 1: semantic relatedness SICK dataset BIBREF33 . Given two sentences, the task is to determine how closely they are semantically related. The set contains 9,927 pairs of sentences with human annotated relatedness score, ranging from 1 to 5. A score of 1 indicates that the two sentences are not related, while 5 indicates high relatedness. The set is splitted into a training set of 4,500 instances, a validation set of 500, and a test set of 4,927.
We compare Doc2VecC with several winning solutions of the competition as well as several more recent techniques reported on this dataset, including bi-directional LSTM and Tree-LSTM trained from scratch on this dataset, Skip-thought vectors learned a large book corpus BIBREF34 and produced sentence embeddings of 4,800 dimensions on this dataset. We follow the same protocol as in skip-thought vectors, and train Doc2VecC on the larger book corpus dataset. Contrary to the vocabulary expansion technique used in BIBREF16 to handle out-of-vocabulary words, we extend the vocabulary of the learned model directly on the target dataset in the following way: we use the pre-trained word embedding as an initialization, and fine-tune the word and sentence representation on the SICK dataset. Notice that the fine-tuning is done for sentence representation learning only, and we did not use the relatedness score in the learning. This step brings small improvement to the performance of our algorithm. Given the sentence embeddings, we used the exact same training and testing protocol as in BIBREF16 to score each pair of sentences: with two sentence embedding INLINEFORM0 and INLINEFORM1 , we concatenate their component-wise product, INLINEFORM2 and their absolute difference, INLINEFORM3 as the feature representation.
Table TABREF35 summarizes the performance of various algorithms on this dataset. Despite its simplicity, Doc2VecC significantly out-performs the winning solutions of the competition, which are heavily feature engineered toward this dataset and several baseline methods, noticeably the dependency-tree RNNs introduced in BIBREF35 , which relies on expensive dependency parsers to compose sentence vectors from word embeddings. The performance of Doc2VecC is slightly worse than the LSTM based methods or skip-thought vectors on this dataset, while it significantly outperforms skip-thought vectors on the IMDB movie review dataset ( INLINEFORM0 error rate vs INLINEFORM1 ). As we hypothesized in previous section, while Doc2VecC is better at handling longer paragraphs, LSTM-based methods are superior for relatively short sentences (of length in the order of 10s). We would like to point out that Doc2VecC is much faster to train and test comparing to skip-thought vectors. It takes less than 2 hours to learn the embeddings on the large book corpus for Doc2VecC on a desktop with Intel i7 2.2Ghz cpu, in comparison to the 2 weeks on GPU required by skip-thought vectors.
We introduce a new model architecture Doc2VecC for document representation learning. It is very efficient to train and test thanks to its simple model architecture. Doc2VecC intrinsically makes sure document representation generated by averaging word embeddings capture semantics of document during learning. It also introduces a data-dependent regularization which favors informative or rare words while dampening the embeddings of common and non-discriminative words. As such, each document can be efficiently represented as a simple average of the learned word embeddings. In comparison to several existing document representation learning algorithms, Doc2VecC outperforms not only in testing efficiency, but also in the expressiveness of the generated representations.

Figure and table captions:
Figure 1: A new framework for learning document vectors.
Table 1: Classification error of a linear classifier trained on various document representations on the Imdb dataset.
Table 2: Learning time and representation generation time required by different representation learning algorithms.
Table 3: Words with embeddings closest to 0 learned by different algorithms.
Figure 2: Accuracy on subset of the Semantic-Syntactic Word Relationship test set. Only questions containing words from the most frequent 30k words are included in the test.
Table 4: Top 1 accuracy on the 5 type of semantics and 9 types of syntactic questions.
Table 5: Classification error (%) of a linear classifier trained on various document representations on the Wikipedia dataset.
Figure 3: Visualization of document vectors on Wikipedia dataset using t-SNE.
Figure 4: Visualization of Wikipedia Doc2VecC vectors using t-SNE.
Table 6: Test set results on the SICK semantic relatedness task. The first group of results are from the submission to the 2014 SemEval competition; the second group includes several baseline methods reported in (Tai et al., 2015); the third group are methods based on LSTM reported in (Tai et al., 2015) as well as the skip-thought vectors (Kiros et al., 2015).

Questions about this paper:
How do they determine which words are informative?

Answers:
Informative are those that will not be suppressed by regularization performed.