Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper discusses a method for computing vector representations for documents by using a skip-gram style learning mechanism with an added regularizer in the form of a global context vector with various bits of drop out. While none of the individual components proposed in this paper are new, I believe that the combination in this fashion is. Further, I appreciated the detailed analysis of model behaviour in section 3.

The main downside to this submission is in its relative weakness on the empirical front. Arguably there are more interesting tasks than sentiment analysis and k-way classification! Likewise, why waste 2/3 of a page on t-sne projections rather than use that space for further analysis?

While I am a bit disappointed by this reduced evaluation and agree with the other reviewers concerning soft baselines, I think this paper should be accepted: it's an interesting algorithm, nicely composed and very efficient, so it's reasonable to assume that other readers might have use for some of the ideas presented here.
review_title: Simple idea, nicely composed

final_score: rating: null
review_content: - Were the word embeddings for the Word2Vec+ baselines only trained on the training data? Did you try training them on a much bigger (unlabeled) corpus?
- Why did you exclude the RNN-LM baseline for the document classification task?

Thanks!
review_title: pre review questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a framework for creating document representations. 
The main idea is to represent a document as an average of its word embeddings with a data-dependent regularization that favors informative or rare words while forcing common words to be close to 0. 
Experiments on sentiment analysis and document classification show that the proposed method has the lowest error rates compared to baseline document embedding methods. 

While I like the motivation of finding the best way to encode a document into a vector, the paper does not offer significant technical contributions.
Most of the techniques are not new, and the main selling point is the simplicity and speed of the proposed method. 
For this reason, I would like to see good results for more than two tasks to be convinced that this is the best way to learn document representations.  
For RNN-LM, is the LM trained to minimize classification error, or is it trained  as a language model? Did you use the final hidden state as the representation, or the average of all hidden states?
One of the most widely used method to represent documents now is to have a bidirectional LSTM and concatenate the final hidden states as the document representation. 
I think it would be useful to know how the proposed method compares to this approach for tasks such as document classification or sentiment analysis.
review_title: review

final_score: rating: 7: Good paper, accept
review_content: This paper proposes learning document embeddings as a sum of the constituent word embeddings, which are jointly learned and randomly dropped out ('corrupted') during training. While none of the pieces of this model are particularly novel, the result is an efficient learning algorithm for document representation with good empirical performance.

Joint training of word and document embeddings is not a new idea, nor is the idea of enforcing the document to be represented by the sum of its word embeddings (see, e.g. '“The Sum of Its Parts”: Joint Learning of Word and Phrase Representations with Autoencoders' by Lebret and Collobert). Furthermore, the corruption mechanism is nothing other than traditional dropout on the input layer. Coupled with the word2vec-style loss and training methods, this paper offers little on the novelty front.

On the other hand, it is very efficient at generation time, requiring only an average of the word embeddings rather than a complicated inference step as in Doc2Vec. Moreover, by construction, the embedding captures salient global information about the document -- it captures specifically that information that aids in local-context prediction. For such a simple model, the performance on sentiment analysis and document classification is quite encouraging.

Overall, despite the lack of novelty, the simplicity, efficiency, and performance of this model make it worthy of wider readership and study, and I recommend acceptance.
review_title: Interesting corruption mechanism for document representation

final_score: rating: null
review_content: Have you analyzed the effect of document length on the resulting embeddings, e.g. does the norm decrease with length? Does performance depend strongly on document length?
review_title: Effect of document length

Meta-review: The introduced method for producing document representations is simple, efficient and potentially quite useful. Though we could quibble a bit that the idea is just a combination of known techniques, the reviews generally agree that the idea is interesting.
 
 Pros:
 + interesting and simple algorithm
 + strong performance
 + efficient
 
 Cons:
 + individual ideas are not so novel
 
 This is a paper that will be well received at a poster presentation.
Decision: Accept (Poster)