Venue: ICLR 2018 conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This paper proposes using CNNs with a skip-gram like objective as a fast way to output document embeddings and much faster compared to skip-thought and RNN type models.

While the problem is an important one, the paper only compares speed with the RNN-type model and doesn't make any inference speed comparison with paragraph vectors (the main competing baseline in the paper). Paragraph vectors are also parallelizable so it's not obvious that this method would be superior to it. The paper in the introduction also states that doc2vec is trained using localized contexts (5 to 10 words) and never sees the whole document. If this was the case then paragraph vectors wouldn't work when representing a whole document, which it already does as can be seen in table 2.

The paper also fails to compare with the significant amount of existing literature on state of the art document embeddings. Many of these are likely to be faster than the method described in the paper. For example:


Arora, S., Liang, Y., & Ma, T. A simple but tough-to-beat baseline for sentence embeddings. ICLR 2017.
Chen, M. Efficient vector representation for documents through corruption. ICLR 2017.

review_title: No comparison against recent SOTA in text representation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new model for the general task of inducing document representations (embeddings). The approach uses a CNN architecture, distinguishing it from the majority of prior efforts on this problem, which have tended to use RNNs. This affords obvious computational advantages, as training may be parallelized. 

Overall, the model presented is relatively simple (a good thing, in my view) and it indeed seems fast. I can thus see potential practical uses of this CNN based approach to document embedding in future work on language tasks. The training strategy, which entails selecting documents and then indexes within them stochastically, is also neat. Furthermore, the work is presented relatively clearly. That said, my main concerns regarding this paper are that: (1) there's not much new here, and, (2) the experimental setup may be flawed, in that it would seem model hyperparams were tuned for the proposed approach but not for the baselines; I elaborate on these concerns below.

Specific comments:
---
- It's hard to tease out exactly what's new here: the various elements used are all well known. But perhaps there is merit in putting the specific pieces together. Essentially, the novelty is using a CNN rather than an RNN to induce document embeddings. 

- In Section 4.1, the authors write that they report results for their after running "parameter sweeps ..." -- I presume that these were performed on a validation set, but the authors should say so. In any case, a very potential weakness here: were analagous parameter sweeps for this dataset performed for the baseline models? It would seem not, as the authors write "the IMDB training data using the default hyper-parameters" for skip-thought. Surely it is unfair comparison if one model has been tuned to a given dataset while others use only the default hyper-parameters? 

- Many important questions were left unaddressed in the experiments. For example, does one really need to use the gating mechanism borrowed from the Dauphin et al. paper? What happens if not? How big of an effect does the stochastic sampling of document indices have on the learned embeddings? Does the specific underlying CNN architecture affect results, and how much? None of these questions are explored. 

- I was left a bit confused regarding how the v_{1:i-1} embedding is actually estimated; I think the details here are insufficient in the current presentation. The authors write that this is a "function of all words up to w_{i-1}". This would seem to imply that at test time, prediction is not in fact parallelizable, no? Yet this seems to be one of the main arguments the authors make in favor of the model (in contrast to RNN based methods). In fact, I think the authors are proposing using the (aggregated) filter activation vectors (h^l(x)) in eq. 5, but for some reason this is not made explicit. 

Minor comments:

- In Eq. 4, should the product be element-wise to realize the desired gating (as per the Dauhpin paper)? This should be made explicit in the notation.

- On the bottom of page 3, the authors claim "Expanding the prediction to multiple words makes the problem more difficult since the only way to achieve that is by 'understanding' the preceding sequence." This claim should either by made more precise or removed. It is not clear exactly what is meant here, nor what evidence supports it.

- Commas are missing in a few. For example on page 2, probably want a comma after "in parallel" (before "significantly"); also after "parallelize" above "Approach".

- Page 4: "In contrast, our model addresses only requires" --> drop the "addresses". 
review_title: Review of "Learning Document Embeddings With CNNs"

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper uses CNNs to build document embeddings.  The main advantage over other methods is that CNNs are very fast.

First and foremost I think this: "The code with the full model architecture will be released … and we thus omit going into further details here."  is not acceptable.  Releasing code is commendable, but it is not a substitute for actually explaining what you have done.  This is especially true when the main contribution of the work is a network architecture.  If you're going to propose a specific architecture I expect you to actually tell me what it is.

I'm a bit confused by section 3.1 on language modelling.  I think the claim that it is showing "a direct connection to language modelling" and that "we explore this relationship in detail" are both very much overstated.  I think it would be more accurate to say this paper takes some tricks that people have used for language modelling and applies them to learning document embeddings.

This paper proposed both a model and a training objective, and I would have liked to see some attempt to disentangle their effect.  If there is indeed a direct connection between embedding models and language models then I would have also expected to see some feedback effect from document embedding to language modeling.  Does the embedding objective proposed here also lead to better language models?

Overall I do not see a substantial contribution from this paper. The main claims seem to be that CNNs are fast, and can be used for NLP, neither of which are new.

review_title: An okay paper that fails to document its contribution

Meta-review: there are two separate ideas embedded in this submission; (1) language modelling (with the negative sampling objective by mikolov et al.) is a good objective to use for extracting document representation, and (2) CNN is a faster alternative to RNN's, both of which have been studied in similar contexts earlier (e.g., paragraph vectors, CNN classifiers and so on, most of which were pointed out by the reviewers already.) Unfortunately reading this manuscript does not reveal too clearly how these two ideas connect to each other (and are separate from each other) and are related to earlier approaches, which were again pointed out by the reviewers. in summary, i believe this manuscript requires more work to be accepted.
Decision: Reject