Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The paper is mostly compared against the full model, without binarization. Have you tried a simple baseline such that 1) first learn the network; 2) make an unsupervised learning for binarization; and 3) optionally, fine-tuning of the last layer?
review_title: Comparison with simple baseline

final_score: rating: 6: Marginally above acceptance threshold
review_content: The method in this paper introduces a binary encoding level in the PV-DBOW and PV-DM document embedding methods (from Le & Mikolov'14). The binary encoding consists in a sigmoid with trained parameters that is inserted after the standard training stage of the embedding.
 
For a document to encode, the binary vector is obtained by forcing the sigmoid to output a binary output for each of the embedding vector components. The binary vector can then be used for compact storage and fast comparison of documents.
 
Pros:
 
- the binary representation outperforms the Semantic hashing method from Salakhutdinov & Hinton '09
 
- the experimental approach sound: they compare on the same experimental setup as Salakhutdinov & Hinton '09, but since in the meantime document representations improved (Le & Mikolov'14), they also combine this new representation with an RBM to show the benefit of their binary PV-DBOW/PV-DM
 
Cons:
 
- the insertion of the sigmoid to produce binary codes (from Lin & al. '15) in the training process is incremental
 
- the explanation is too abstract and difficult to follow for a non-expert (see details below)
 
- a comparison with efficient indexing methods used in image retrieval is missing. For large-scale indexing of embedding vectors, derivations of the Inverted multi-index are probably more interesting than binary codes. See eg. Babenko & Lempitsky, Efficient Indexing of Billion-Scale Datasets of Deep Descriptors, CVPR'16
 
Detailed comments:
 
Section 1: the motivation for producing binary codes is not given. Also, the experimental section could give some timings and mem usage numbers to show the benefit of binary embeddings
 
figure 1, 2, 3: there is enough space to include more information on the representation of the model: model parameters + training objective + characteristic sizes + dropout. In particular, in fig 2, it is not clear why "embedding lookup" and "linear projection" cannot be merged in a single smaller lookup table (presumably because there is an intermediate training objective that prevents this).
 
p2: "This way, the length of binary codes is not tied to the dimensionality of word embeddings." -> why not?
 
section 3: This is the experimental setup of  Salakhutdinov & Hinton 2009. Specify this and whether there is any difference between the setups.
 
"similarity of the inferred codes": say here that codes are compared using Hamming distances.
 
"binary codes perform very well, despite their far lower capacity" -> do you mean smaller size than real vectors?
 
fig 5: these plots could be dropped if space is needed.
 
section 3.1: one could argue that "transferring" from Wikipedia to anything else cannot be called transferring, since Wikipedia's purpose is to include all topics and lexical domains
 
section 3.2: specify how the 300D real vectors are compared. L2 distance? inner product?
 
fig4: specify what the raw performance of the large embedding vectors is (without pre-filtering with binary codes), or equivalently, the perf of (code-size, Hamming dis) = (28, 28), (24, 24), etc.

review_title: A few comments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes a model that can learn short binary codes via paragraph vectors to allow fast retrieval of documents. The experiments show that this is superior to semantic hashing. The approach is simple and not very technically interesting. For a code size of 128, the loss compared to a continuous paragraph vector seems moderate.

The paper asks the reader to refer to the Salakhutdinov and Hinton paper for the baseline numbers but I think they should be placed in the paper for easy reference. For simplicity, the paper could show the precision at 12.5%, 25% and 50% recall for the proposed model and semantic hashing. It also seems that the semantic hashing paper shows results on RCV2 and not RCV1. RCV1 is twice the size of RCV2 and is English only so it seems that these results are not comparable. It would be interesting to see how many binary bits are required to match the performance of the continuous representation. A comparison to the continuous PV-DBOW trained with bigrams would also make it a more fair comparison.

Figure 7 in the paper shows a loss from using the real-binary PV-DBOW. It seems that if a user needed high quality ranking after the retrieval stage and they could afford the extra space and computation, then it would be better for them to use a standard PV-DBOW to obtain the continuous representation at that stage.

Minor comments:
First line after the introduction: is sheer -> is the sheer
4th line from the bottom of P1: words embeddings -> word embeddings
In table 1: What does code size refer to for PV-DBOW? Is this the number of elements in the continuous vector?
5th line from the bottom of P5: W -> We
5th line after section 3.1: covers wide -> covers a wide

review_title: Nice method for obtaining binary codes from paragraph vectors

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method to represent text documents and paragraphs as short binary codes to allow fast similarity search and retrieval by using hashing techniques. The real-valued paragraph vectors by Le & Mikolov is extended by adding a stochastic binary layer on top of the neural network architecture. Two methods for binarizing the final activations are compared: (1) simply adding noise to sigmoid activations to encourage discritization. (2) binarizing the activations in the forward pass and keeping them real-valued in the backward pass (straight-through estimation). The paper presents encouraging results by using straight-through estimation on 20 newsgroup and RCV1 text datasets by using 128 and 32 bit binary codes.

On the plus side, the application presented in the paper is interesting and important. The exposition of the paper is clean and clear. However, the novelty of the approach is limited from a machine learning standpoint. The literature on binary hashing beyond semantic hashing and Krizhevsky's binary autoencoders in 2011 is not explained. An important baseline is missing where real-valued paragraph vectors are learned first, and then converted to binary codes using off-the-shelf hashing methods (e.g. random projection LSH by Charikar, BRE by Kulis & Darrell, ITQ by Gong & Lazebnik, MLH by Norouzi & Fleet, etc.)

Given the lack of novelty and the missing baseline, I do not recommend this paper in its current for publication in the ICLR conference's proceeding. Moving forward, this paper may be more suitable for NLP conferences as it is more on the applied side.

More comments:
- I believe from an practical perspective it may be easier to first learn real-valued paragraph vectors and then quantize them for indexing. That said, an end-to-end approach as proposed in this paper may perform better. I would like to see an empirical comparison between the proposed end-to-end approach and a simpler two stage quantization method suggested here.
- See "Estimating or Propagating Gradients Through Stochastic Neurons" By Bengio et al - discussing straight through estimation and some other alternatives.
- The paper argues that the length of binary codes cannot be longer than 32 bits because longer codes are not suitable for document hashing. This is not quite right given multi-probe hashing mechanisms, for example see "Mult-index Hashing" by Norouzi et al.
- See "Hashing for Similarity Search: A Survey" by Wang et al. for a survey of related work on binary hashing and quantization. You seem to ignore the extensive work done on binary hashing.

review_title: Review

Meta-review: This paper proposes to binarize Paragraph Vector distributed representations in an end-to-end framework. Experiments demonstrate that this beats autoencoder-based binary codes. However, the performance is similar to using paragraph vectors followed by existing binarization techniques, failing to show an advantage of training end-to-end. Therefore, the novel contribution seems too limited for publication.
Decision: Reject