Title: Binary Paragraph Vectors
Authors: Karol Grzegorczyk
Marcin Kurdziel
Venue: ICLR 2017
Keywords: Natural language processing
Transfer Learning

Abstract: Recently Le & Mikolov described two log-linear models, called Paragraph Vector, that can be used to learn state-of-the-art distributed representations of documents. Inspired by this work, we present Binary Paragraph Vector models: simple neural networks that learn short binary codes for fast information retrieval. We show that binary paragraph vectors outperform autoencoder-based binary codes, despite using fewer bits. We also evaluate their precision in transfer learning settings, where binary codes are inferred for documents unrelated to the training corpus. Results from these experiments indicate that binary paragraph vectors can capture semantics relevant for various domain-specific documents. Finally, we present a model that simultaneously learns short binary codes and longer, real-valued representations. This model can be used to rapidly retrieve a short list of highly relevant documents from a large document collection.

TL;DR: Learning short codes for text documents with Binary Paragraph Vectors.

Review discussion:
The paper is mostly compared against the full model, without binarization. Have you tried a simple baseline such that 1) first learn the network; 2) make an unsupervised learning for binarization; and 3) optionally, fine-tuning of the last layer?
Dear Reviewer,
Thank you for your helpful comment.
Following your suggestion, we experimented with two baseline approaches, where binary codes were inferred from real-valued paragraph vectors using: a) an autoencoder with sigmoid coding layer (and Krizhevsky's binarization) and b) Gaussian-Bernoulli RBM. We focused on networks with the number of parameters similar to that of Binary PV-DBOW models. The results from these experiments are included as Table 2 in an updated version of our manuscript (uploaded today) and discussed in the paragraph above the table. Our results show no benefit from using this two unsupervised models for binarization.
In addition to the experiments described above, we also used an autoencoder and an RBM to construct binary codes from 300-dimensional paragraph vectors. We did not observe clear improvement from this approach, even though it uses 10x more parameters than Binary PV models.
Specifically, with the autoencoder we observed lower MAP on both test sets. The RBM improved MAP on TNG, but on RCV it yielded performance similar to Binary PV-DBOW.
This work proposes a model that can learn short binary codes via paragraph vectors to allow fast retrieval of documents. The experiments show that this is superior to semantic hashing. The approach is simple and not very technically interesting. For a code size of 128, the loss compared to a continuous paragraph vector seems moderate.
The paper asks the reader to refer to the Salakhutdinov and Hinton paper for the baseline numbers but I think they should be placed in the paper for easy reference. For simplicity, the paper could show the precision at 12.5%, 25% and 50% recall for the proposed model and semantic hashing. It also seems that the semantic hashing paper shows results on RCV2 and not RCV1. RCV1 is twice the size of RCV2 and is English only so it seems that these results are not comparable. It would be interesting to see how many binary bits are required to match the performance of the continuous representation. A comparison to the continuous PV-DBOW trained with bigrams would also make it a more fair comparison.
Figure 7 in the paper shows a loss from using the real-binary PV-DBOW. It seems that if a user needed high quality ranking after the retrieval stage and they could afford the extra space and computation, then it would be better for them to use a standard PV-DBOW to obtain the continuous representation at that stage.
Minor comments:
First line after the introduction: is sheer -> is the sheer
4th line from the bottom of P1: words embeddings -> word embeddings
In table 1: What does code size refer to for PV-DBOW? Is this the number of elements in the continuous vector?
5th line from the bottom of P5: W -> We
5th line after section 3.1: covers wide -> covers a wide
Dear Reviewer,
Thank you for your review.
As you suggested, a comparison against semantic hashing was added to Figure 4.
Regarding your remark on the Reuters corpus used in the semantic hashing paper, Salakhutdinov & Hinton mistakenly identified it as RCV2 in the conference version of their paper. Journal version of the paper (published in 2009, and cited in out work) states that they use RCV1-v2 (which is a cleansed version of RCV1). This also agrees with the size of the corpus reported in both conference and journal versions. In our experiments we also use version 2 of RCV1. We've added that detail to the paper.
Regarding your remark that it would be interesting to see how many bits are required to match the performance of the continuous representation, we were not able to exactly match the performance of real-valued representation with binary codes (by simply increasing code size). One strength of PV networks is that they are in fact simple, yet quite effective, log-linear models. In Binary PV we must add a binarized sigmoid non-linearity, which can make training harder.
As your requested, a comparison against the continuous PV-DBOW with bigrams was added to Table 1 and in Figure 4.
Furthermore, we extended Table 4 with comparison between Real-Binary PV results (variant B therein) and an alternative approach, where filtering is carried out with plain Binary PV-DBOW codes and ranking is carried out using standard paragraph vectors (variant C in Table 4). As could be expected, variant C yielded higher NDCG.
However, we point out that this retrieval strategy requires a model (PV-DBOW) with approximately 10 times more parameters than Real-Binary PV-DBOW. We added a text in the paragraph discussing these results, where we point to a possible improvement in retrieval precision at the cost of using a bigger model.
Thanks for the additions to the tables and figures.
I will be keeping my previous rating.
This paper presents a method to represent text documents and paragraphs as short binary codes to allow fast similarity search and retrieval by using hashing techniques. The real-valued paragraph vectors by Le & Mikolov is extended by adding a stochastic binary layer on top of the neural network architecture. Two methods for binarizing the final activations are compared: (1) simply adding noise to sigmoid activations to encourage discritization. (2) binarizing the activations in the forward pass and keeping them real-valued in the backward pass (straight-through estimation). The paper presents encouraging results by using straight-through estimation on 20 newsgroup and RCV1 text datasets by using 128 and 32 bit binary codes.
On the plus side, the application presented in the paper is interesting and important. The exposition of the paper is clean and clear.
However, the novelty of the approach is limited from a machine learning standpoint. The literature on binary hashing beyond semantic hashing and Krizhevsky's binary autoencoders in 2011 is not explained. An important baseline is missing where real-valued paragraph vectors are learned first, and then converted to binary codes using off-the-shelf hashing methods (e.g. random projection LSH by Charikar, BRE by Kulis & Darrell, ITQ by Gong & Lazebnik, MLH by Norouzi & Fleet, etc.)
Given the lack of novelty and the missing baseline, I do not recommend this paper in its current for publication in the ICLR conference's proceeding. Moving forward, this paper may be more suitable for NLP conferences as it is more on the applied side.
More comments:
- I believe from an practical perspective it may be easier to first learn real-valued paragraph vectors and then quantize them for indexing.
That said, an end-to-end approach as proposed in this paper may perform better. I would like to see an empirical comparison between the proposed end-to-end approach and a simpler two stage quantization method suggested here.
- See "Estimating or Propagating Gradients Through Stochastic Neurons" By Bengio et al - discussing straight through estimation and some other alternatives.
- The paper argues that the length of binary codes cannot be longer than 32 bits because longer codes are not suitable for document hashing. This is not quite right given multi-probe hashing mechanisms, for example see "Mult-index Hashing" by Norouzi et al.
- See "Hashing for Similarity Search: A Survey" by Wang et al. for a survey of related work on binary hashing and quantization. You seem to ignore the extensive work done on binary hashing.
Dear Reviewer,
Thank you for your review.
We extended our experimental results with comparison against random hyperplane projection LSH and iterative quantization. These results were added to Table 2 and are discussed in the paragraph above the table.
In short, our results show no benefit from using a separate algorithm for binarization.
As you suggested, we also evaluated several gradient estimators for stochastic binary neurons (SBN), namely:
- plain straight-through (ST) estimator (dSBN(a)/da = 1),
- modified ST estimator (dSBN(a)/da = sigma'(a)),
- slope-annealed variants of the above two (proposed by Chung et al. in "Hierarchical Multiscale Recurrent Neural Networks").
The best results in this settings were obtained with slope-annealed variant of the plain ST estimator.
Specifically, we observed better top-hits precision on 20NG dataset (compared to deterministic Krizhevsky's binarization).
However, this did not translate to an improved MAP, due to lower precision at higher recall levels. We did not observe any improvement over Krizhevsky's method on the larger RCV1 dataset.
Therefore we left Krizhevsky's method as our binarization approach.
Regarding the length of binary codes, in the original version of our paper we focused on indexing directly with short codes. To compare with semantic hashing, we also reported results for 128-bit codes. As you pointed out, codes longer then 32 bits are feasible, e.g. by using Norouzi et al. multi-index hashing.
However, with longer codes we also need to use larger search radii. Norouzi et al report search radii of around 15-bits for 128-bit codes (assuming 10-NN lookup over a 1B database). This gives radius/code_size of approx. 0.11, at which point Norouzi's method scales with the square root of the database size. Summing up, we reworded parts of our paper where we discuss the code size, to not exclude codes larger than 32-bits.
I really appreciate the new changes to the paper and the new results in Table 2 discussing the two-stage approach (first learning real-valued paragraph vectors and then quantizing them).
Given the results in Table 2 though, I do not see a clear benefit in favor of your direct approach over the combination of paragraph vectors and ITQ, especially given the simplicity of this two-stage process. You should change the tone of the paper to clearly state that based on some metrics the two-stage process performs better and it may be preferred given its simplicity and ease of use. This will have an impact on the story and the conclusions.
I keep my previous rating.
Dear Reviewer,
Thank you for your comment.
Regarding the simplicity of end-to-end training (i.e. Binary PV) vs. a two-stage approach (first infer real-valued paragraph vectors and then binarize with, e.g., ITQ), we would argue that the end-to-end training is simpler. From our experience, the training and inference time in Binary PV models is almost identical to that of the original paragraph vectors. We then obtain binary codes with no additional steps.
Also, Binary PV are simple to implement - extending PV models to binary variants requires just adding a sigmoid layer with rounding.
The method in this paper introduces a binary encoding level in the PV-DBOW and PV-DM document embedding methods (from Le & Mikolov'14). The binary encoding consists in a sigmoid with trained parameters that is inserted after the standard training stage of the embedding.
For a document to encode, the binary vector is obtained by forcing the sigmoid to output a binary output for each of the embedding vector components. The binary vector can then be used for compact storage and fast comparison of documents.
Pros:
- the binary representation outperforms the Semantic hashing method from Salakhutdinov & Hinton '09
- the experimental approach sound: they compare on the same experimental setup as Salakhutdinov & Hinton '09, but since in the meantime document representations improved (Le & Mikolov'14), they also combine this new representation with an RBM to show the benefit of their binary PV-DBOW/PV-DM
Cons:
- the insertion of the sigmoid to produce binary codes (from Lin & al. '15) in the training process is incremental
- the explanation is too abstract and difficult to follow for a non-expert (see details below)
- a comparison with efficient indexing methods used in image retrieval is missing. For large-scale indexing of embedding vectors, derivations of the Inverted multi-index are probably more interesting than binary codes. See eg. Babenko & Lempitsky, Efficient Indexing of Billion-Scale Datasets of Deep Descriptors, CVPR'16
Detailed comments:
Section 1: the motivation for producing binary codes is not given.
Also, the experimental section could give some timings and mem usage numbers to show the benefit of binary embeddings
figure 1, 2, 3: there is enough space to include more information on the representation of the model: model parameters + training objective + characteristic sizes + dropout.
In particular, in fig 2, it is not clear why "embedding lookup" and "linear projection" cannot be merged in a single smaller lookup table (presumably because there is an intermediate training objective that prevents this).
p2: "This way, the length of binary codes is not tied to the dimensionality of word embeddings." -> why not?
section 3: This is the experimental setup of  Salakhutdinov & Hinton 2009. Specify this and whether there is any difference between the setups.
"similarity of the inferred codes": say here that codes are compared using Hamming distances.
"binary codes perform very well, despite their far lower capacity" -> do you mean smaller size than real vectors?
fig 5: these plots could be dropped if space is needed.
section 3.1: one could argue that "transferring" from Wikipedia to anything else cannot be called transferring, since Wikipedia's purpose is to include all topics and lexical domains
section 3.2: specify how the 300D real vectors are compared. L2 distance? inner product?
fig4: specify what the raw performance of the large embedding vectors is (without pre-filtering with binary codes), or equivalently, the perf of (code-size, Hamming dis) = (28, 28), (24, 24), etc.
Dear Reviewer,
Thank you for your review.
We agree that indexing data structures are important parts of information retrieval systems.
However, in this work we are mainly interested in learning good binary representation of text. Such representation can be used for document retrieval (and indeed, this is a proxy to the code quality in our paper) but can also serve other purposes: e.g. can be used for fast comparison of hashed data items or fast duplicate detection.
Also, while we do not discuss efficiency of binary codes in our work, this has been looked at by Salakhutdinov & Hinton in their semantic hashing paper (section 4.2). We do not rule out investigating indexing methods for paragraph vectors in future research.
We added information regarding training loss (which is a sampled softmax) on the figures depicting our models. The linear projection is not merged with embedding lookup, because in Real-Binary model we want to explicitly infer both short binary code and high-dimensional real-valued representation.
In Binary PV-DM model we may want to use a different number of dimensions for word vectors and binary codes (e.g. to infer short binary codes).
When the context passed to the sigmoid layer is a concatenation of word vectors and (pre-sigmoid) document vectors, their dimensionality does not need to match. This wouldn't be the case if, instead, we used a sum of relevant vectors as the context passed to the sigmoid non-linearity.
Regarding the experimental setup, we added a sentence at the end of 2nd paragraph of section 3, explaining that we use the same benchmark datasets and quality measures that were used in semantic hashing paper. We also explained in the caption of Table 4 that 300D real-valued vectors are compared using cosine similarity.
Furthermore, Table 4 now also reports performance of high-dimensional embeddings (from Real-Binary model) without pre-filtering.
Regarding the transfer learning, despite its wide scope, Wikipedia follows specific style and language, which is not representative for many other domains (e.g. newsgroup posts). In the same spirit networks pre-trained on ImageNet are frequently fine-tuned for domain-specific classification.
Thanks for the updated paper.
I am keeping my previous rating.
We have just uploaded an updated version of our paper. Below we list the most important changes. We will then follow with responses to the reviews.
1) AnonReviewer3 pointed out that, apart from the Krizhevsky's binarization, one can also employ stochastic neurons in the coding layer.
When comparing these two approaches we carried out a more thorough investigation of the dropout in the coding layers. In the initial experiments we employed small, 10% dropout.
However, larger dropout rates (up to 50%) turned out to improve performance on the validation sets (in both binary and real-valued models). We therefore updated final results to reflect the dropout rates selected in validation experiments.
2) We extended the comparison with baseline methods by adding results for two hashing techniques, namely random hyperplane projection and iterative quantization.
3) Visualization of Binary PV codes (Figure 5 in the previous version of the paper) along with the corresponding text was moved to an appendix.

Meta-review:
role: PC
data: comment: This paper proposes to binarize Paragraph Vector distributed representations in an end-to-end framework. Experiments demonstrate that this beats autoencoder-based binary codes. However, the performance is similar to using paragraph vectors followed by existing binarization techniques, failing to show an advantage of training end-to-end. Therefore, the novel contribution seems too limited for publication.

Decision: Reject