Title: Efficient Vector Representation for Documents through Corruption
Authors: Minmin Chen
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Semi-Supervised Learning

Abstract: We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a simple average of word embeddings. It ensures a representation generated as such captures the semantic meanings of the document during learning. A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero. Doc2VecC produces significantly better word embeddings than Word2Vec. We compare Doc2VecC with several state-of-the-art document representation learning algorithms. The simple model architecture introduced by Doc2VecC matches or out-performs the state-of-the-art in generating high-quality document representations for sentiment analysis, document classification as well as semantic relatedness tasks. The simplicity of the model enables training on billions of words per hour on a single machine. At the same time, the model is very efficient in generating representations of unseen documents at test time.


TL;DR: a simple document representation learning framework that is very efficient to train and test

Review discussion:
- Were the word embeddings for the Word2Vec+ baselines only trained on the training data? Did you try training them on a much bigger (unlabeled) corpus?
- Why did you exclude the RNN-LM baseline for the document classification task?
Thanks!
Thank you for your questions. For both tasks, a bigger unlabeled set is used to learn representation for all the representation learning methods.  A classifier is then trained using the learned representation on the training data to perform either sentiment analysis or document classification.
RNN-LM builds one language model  per class, each one taking long time to train. Its performance is not as competent as the others in the sentiment analysis task. Due to time limit, I omitted this method for the multi-class document classification task.
Have you analyzed the effect of document length on the resulting embeddings, e.g. does the norm decrease with length? Does performance depend strongly on document length?
Thank you for your questions.
Indeed, the norm of the learned document embedding decreases with document length. I added a figure in the appendix to demonstrate this effect on the embeddings learned on the Imdb dataset. I did not find the performance in particular depends on the length of the document. The classification error on the different document length buckets are comparable.
This paper proposes learning document embeddings as a sum of the constituent word embeddings, which are jointly learned and randomly dropped out ('corrupted') during training.
While none of the pieces of this model are particularly novel, the result is an efficient learning algorithm for document representation with good empirical performance.
Joint training of word and document embeddings is not a new idea, nor is the idea of enforcing the document to be represented by the sum of its word embeddings (see, e.g. '“The Sum of Its Parts”: Joint Learning of Word and Phrase Representations with Autoencoders' by Lebret and Collobert).
Furthermore, the corruption mechanism is nothing other than traditional dropout on the input layer. Coupled with the word2vec-style loss and training methods, this paper offers little on the novelty front.
On the other hand, it is very efficient at generation time, requiring only an average of the word embeddings rather than a complicated inference step as in Doc2Vec.
Moreover, by construction, the embedding captures salient global information about the document -- it captures specifically that information that aids in local-context prediction. For such a simple model, the performance on sentiment analysis and document classification is quite encouraging.
Overall, despite the lack of novelty, the simplicity, efficiency, and performance of this model make it worthy of wider readership and study, and I recommend acceptance.
This paper presents a framework for creating document representations.
The main idea is to represent a document as an average of its word embeddings with a data-dependent regularization that favors informative or rare words while forcing common words to be close to 0.
Experiments on sentiment analysis and document classification show that the proposed method has the lowest error rates compared to baseline document embedding methods.
While I like the motivation of finding the best way to encode a document into a vector, the paper does not offer significant technical contributions.
Most of the techniques are not new, and the main selling point is the simplicity and speed of the proposed method.
For this reason, I would like to see good results for more than two tasks to be convinced that this is the best way to learn document representations.
For RNN-LM, is the LM trained to minimize classification error, or is it trained  as a language model? Did you use the final hidden state as the representation, or the average of all hidden states?
One of the most widely used method to represent documents now is to have a bidirectional LSTM and concatenate the final hidden states as the document representation.
I think it would be useful to know how the proposed method compares to this approach for tasks such as document classification or sentiment analysis.
Dear reviewer, thank you for your constructive feedback.
Indeed our main goal is to come up with a simple and efficient framework for generating document representations. I would like to argue that simplicity does not deny originality. The reason we can simply average word embeddings at test time to form document representation is because of the new model architecture proposed, which represents documents with corrupted average of word embeddings at learning time, and learns the document embedding with word embeddings together. The corruption at learning time enables fast learning (comparing to [2][3]), as well as a data-dependent regularization. As far as I know, it is the first do so.
I believe the paper contains quite thorough analyses of the proposed work on the sentiment analysis and document classification tasks. I would like to see the community start exploring and benefiting from this simple idea, while we work on testing it on more tasks.
For RNN-LM, we used the implementation provided by the author, which was tested on the same dataset in their 2015 ICLR submission [1]. It builds two language models, one for the positive class and one for negative. It then computes the probability of each LM generating the document and assigns the one with higher score as the prediction.
I included skip-thought vectors as another baseline in the manuscript thanks to the feedback from another reviewer. The encoder and decoder in the method are constructed from gated RNN. The method produces two models, uni-skip and bi-skip. Among them, the bi-skip is a bi-directional model that generates one forward and one backward encoding of the document.  Its performance is not satisfactory on this dataset, and it takes long time to test due to the high-dimensional encoders used.
Again thanks for your feedback and please let me know if you have other questions.
[1] Mesnil, Grégoire, et al. "Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews." arXiv preprint arXiv:1412.5335 (2014).
[2] Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. Improving word representations via global context and multiple word prototypes. In ACL, pp. 873–882, 2012.
[3] Lebret, Rémi, and Ronan Collobert. "" The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders." arXiv preprint arXiv:1506.05703 (2015).
Dear reviewer,
I added another dataset to the draft (appendix): SemEval 2014 Task 1 semantic relatedness SICK dataset, as several of the recent works the reviewers pointed out have been reported on this dataset.
Despite its simplicity, Doc2VecC significantly out-performs the winning solutions of the competition and several baseline methods, noticeably the dependency-tree RNNs introduced in [1], which relies on additional dependency parsers to compose sentence vectors from word embeddings. The performance of Doc2VecC is comparable (slightly worse) than the LSTM based methods or skip-thought vectors on this dataset.
On the other hand, it is much faster to train and test. As reported in the original paper, training of the skip-thought vector models on the book corpus dataset takes around 2 weeks on GPU.
In contrast, it takes less than 2 hours to learn the embeddings for Doc2VecC on a desktop with Intel i7 2.2Ghz cpu.
I also provided more insight on why Skip-thought vectors did not perform well on the movie review dataset in Section 4.2 (Accuracy).
I would greatly appreciate it if you could take another look at revisions and provide me with some feedbacks.
[1] Socher, Richard, et al. "Grounded compositional semantics for finding and describing images with sentences." Transactions of the Association for Computational Linguistics 2 (2014): 207-218.
Dear reviewer,
I have added more experiments (a new dataset and comparison to LSTM-based methods) and explanations to the manuscript regarding some of the concerns you pointed out. I believe that the additional information provided could worth a second view and would be very much interested in an open discussion to find out if there are any remaining unfavorable factors.
Thank you again for your time.
The new results addressed some of my concerns so I have updated my review score accordingly.
Dear reviewer,
Thank you for taking time to read the paper again and update the review.
Minmin
This paper discusses a method for computing vector representations for documents by using a skip-gram style learning mechanism with an added regularizer in the form of a global context vector with various bits of drop out.
While none of the individual components proposed in this paper are new, I believe that the combination in this fashion is.
Further, I appreciated the detailed analysis of model behaviour in section 3.
The main downside to this submission is in its relative weakness on the empirical front. Arguably there are more interesting tasks than sentiment analysis and k-way classification!
Likewise, why waste 2/3 of a page on t-sne projections rather than use that space for further analysis?
While I am a bit disappointed by this reduced evaluation and agree with the other reviewers concerning soft baselines, I think this paper should be accepted: it's an interesting algorithm, nicely composed and very efficient, so it's reasonable to assume that other readers might have use for some of the ideas presented here.
Dear reviewers,
Thank you for your feedback. The updated manuscript included skip-thought as another baseline method. We will test this idea on more datasets, in particular the ones experimented in Skip-thought vectors in the submission.
Dear reviewers,
I added another dataset to the draft (appendix): SemEval 2014 Task 1 semantic relatedness SICK dataset, as several of the recent works the reviewers pointed out have been reported on this dataset.
Despite its simplicity, Doc2VecC significantly out-performs the winning solutions of the competition and several baseline methods, noticeably the dependency-tree RNNs introduced in [1], which relies on additional dependency parsers to compose sentence vectors from word embeddings. The performance of Doc2VecC is comparable (slightly worse) than the LSTM based methods or skip-thought vectors on this dataset.
On the other hand, it is much faster to train and test. As reported in the original paper, training of the skip-thought vector models on the book corpus dataset takes around 2 weeks on GPU.
In contrast, it takes less than 2 hours to learn the embeddings for Doc2VecC on a desktop with Intel i7 2.2Ghz cpu.
I also provided more insight on why Skip-thought vectors did not perform well on the movie review dataset in Section 4.2 (Accuracy).
I would greatly appreciate it if you could take another look at revisions and provide me with some feedbacks.
[1] Socher, Richard, et al. "Grounded compositional semantics for finding and describing images with sentences." Transactions of the Association for Computational Linguistics 2 (2014): 207-218.
Extended the paper with experiments on the word relationship dataset, showing Doc2VecC generates better word embeddings in comparison to Word2Vec or Paragraph Vectors.

Meta-review:
role: PC
data: comment: The introduced method for producing document representations is simple, efficient and potentially quite useful. Though we could quibble a bit that the idea is just a combination of known techniques, the reviews generally agree that the idea is interesting.
 
 Pros:
 + interesting and simple algorithm
 + strong performance
 + efficient
 
 Cons:
 + individual ideas are not so novel
 
 This is a paper that will be well received at a poster presentation.

Decision: Accept (Poster)