Title: Compact Embedding of Binary-coded Inputs and Outputs using Bloom Filters
Authors: Joan Serrà
Alexandros Karatzoglou
Venue: ICLR 2017
Keywords: Applications
Deep learning
Unsupervised Learning

Abstract: The size of neural network models that deal with sparse inputs and outputs is often dominated by the dimensionality of those inputs and outputs. Large models with high-dimensional inputs and outputs are difficult to train due to the limited memory of graphical processing units, and difficult to deploy on mobile devices with limited hardware. To address these difficulties, we propose Bloom embeddings, a compression technique that can be applied to the input and output of neural network models dealing with sparse high-dimensional binary-coded instances. Bloom embeddings are computationally efficient, and do not seriously compromise the accuracy of the model up to 1/5 compression ratios. In some cases, they even improve over the original accuracy, with relative increases up to 12%. We evaluate Bloom embeddings on 7 data sets and compare it against 4 alternative methods, obtaining favorable results. We also discuss a number of further advantages of Bloom embeddings, such as 'on-the-fly' constant-time operation, zero or marginal space requirements, training time speedups, or the fact that they do not require any change to the core model architecture or training configuration.

TL;DR: Bloom embeddings compactly represent sparse high-dimensional binary-coded instances without compromising accuracy

Review discussion:
As far as I can tell, this paper is proposing to apply multiple hashes and take the union, as opposed to taking a single hash of input or output values.
While the intuition about bloom filters is nice, at the end of the day, this technique is just producing a smaller, but more dense, binary vector, is that correct?
Also, what is the computational complexity of decoding the output? The bloom filter can tell you quickly if a given element is in the set.
But for producing a sparse output it looks like the suggested procedure is just to iterate over all possible outputs...which defeats the purpose of a fast sparse coding of the output.
Can you discuss in more detail also how this will affect neural models that learn embeddings of the input? A straightforward application of your method would suggest that you represent each input as a sum of embeddings indexed by random hashes. Is that correct?  Is there anything further going on here?
Thanks for your questions.
1) Yes, the produced vector has k times more ones than the original one. The crucial point here is that the embedding dimension m is smaller than the original d, thus considerably lowering the space requirements of the model.
2) The computational complexity of decoding the output is O(dk), k<<d, typically k={3,4,5} as used in the paper. Notice however that the output of the models after the softmax is not sparse, and one has to iterate over all the outputs, even in the uncompressed case (for instance, to sort or to find the maximum, thus leading to an O(d) process).
Moreover, in the case of Bloom Embeddings, such decoding is only needed at prediction time, but not at training time.
Finally, the driving motivation of our paper is space efficiency, with affordable timings as a byproduct.
3) Yes. As embeddings can be viewed as an intermediate layer between the encoding of the items and the actual model, our approach yields to embeddings that are distributed across k vectors.
Moreover, the total space of the embedding is compressed by a factor of d/m.
Thank you for this very interesting paper.
While Bloom filters have already been used to encode outputs in multilabel classification, the paper makes a more general contribution by using it for inputs and outputs, and in particular gives a method to apply bloom filters for ranking tasks.
Some questions regarding the experiments:
1) As far as I understood the "dimensionality ratio" is the compression applied to both inputs and outputs, as far as this is applicable (e.g., I suppose input compression is not performed for PTB).
In case compression was applied to both (e.g., recommendation), did you try different compression ratios in the input ?
2) regarding the use of bloom filters for input encoding: bloom filters do not encode counts of objets, but some extensions do.
In particular for recommendation tasks with count data, to what extent the suppression of the counts affects performance ?
3) Some Deep Learning libraries have layers specialized for sparse inputs (e.g., SparseLinear in torch, which behaves exactly like a Linear layer).
When timings are reported, do they use such implementations ?
In particular, the Bloom filter approach for the inputs leads to inputs that are *less* sparse (but in smaller dimension), so I think it is important on the exact implementation when reporting timings.
4) When using bloom filters on the output, I suppose the baseline uses a sofmax activation layer.
However, since all your evaluations are in terms of ranking and not in terms of perplexity/log-likelihood, a pairwise ranking loss could be used instead of softmax+cross-entropy, making it possible to severely subsample the negative classes during training (each epoch would essentially cost about the number of positive labels, even though this doesn't say anything about the overall training time nor on the final accuracy). Did you experiment with such losses ?
Thank you for the questions and the interest in the paper. And thank you for pointing out the multilabel classification case. Certainly we missed to include it in the related work section. After some search, we found the work by Cisse et al (NIPS 2013). Was that the work being referred to?
1) Input compression is applied to all data sets without exception. Output compression is applied to all data sets except CADE, which is a 12-class data set. The compression ratios (m/d) reported in the paper are both applied to inputs and outputs (except the CADE case, where it is only applied to the input).
2) The only considered data sets that originally had count data were the CADE and the MSD data sets, which we reduced to a binary indicator of present/not present and like/not like, respectively. As it is reported, we obtain state-of-the-art (or slightly better) results using this trick. In general, many recommendation/collaborative filtering (CF) methods discard count data.
Of course, using the actual count is a signal that can improve performance, as it imposes a type of more fine-grained relevance. Count data can be incorporated in CF models in several ways, e.g., using the counts for ranking in the loss function, using them as weights in a weighted least squares model (Wu et al, 2016), or modeling the data with poisson models (Gopalan et al, 2014). In our work, we focused on the most common setting of binary relevance (with no grades), as mostly all the data we used was of binary relevance (or ratings that were binarized), but it is definitely possible to extend the Bloom compression to count data. One option would be to use some kind of counting Bloom filters, which would most likely help in improving the recommendation accuracy in these scenarios.
3) In the experiments we used Keras and Theano. We did not use any sparse layers, and the type of implementation was the same for both compressed and uncompressed networks. We believe that the timing experiments will be quite similar even if sparse layers are used at the input. Basically, in the case of sparse layers, we would have O(c) complexity for the input which, using BE, becomes O(ck), with k typically equal to 3 or 4 and, in many cases, k<c. In that case, the timings would be still dominated by the output layer (O(d) or O(m), d>m>c>=k).
Moreover, our main motivation in the paper is fundamentally to reduce the space of the models. Good timings are a byproduct of our approach, but not the driving idea of the paper.
4) We did not.
Although we plan to do so in an extension of this work, we expect though to get quite similar results in terms of relative performance (between compressed and uncompressed models). One can expect a 1-3% performance improvement overall by using a ranking loss function (see Hidasi et al, 2016). One could also use a similar subsampling trick on a softmax-type loss as shown in the paper by Ji et al (ICLR 2016). Note though that using an uncompressed model with the subsampling trick, one would still have to store the full output matrix. Interestingly, after the reconstruction of the original space through the proposed Bloom Embedding, we can still use a ranking-based subsampling loss.
The authors propose a simple scheme based on Bloom filters to generate embeddings for inputs and outputs.  This reduces memory while introducing limited computational overhead, and it is simple enough to implement that it can easily be added to any practitioner's toolbox.
Pros:
- Can be applied to practically any model, either at the input or hte output.
- Method is very straightforward: apply multiple hashes, instead of single one. The algorithm for decoding is nice too.
Cons:
- The paper is a bit difficult to read; the idea is really simple so it doesn't seem like it warrants such a complex description.
- The novelty of the approach is a bit limited since, as other reviewers have mentioned, Bloom filters are in use in a lot of areas including multi-label classification.
This seems like it would be a good short paper. There's a lot of well fleshed out experiments, but the core of the paper is a bit incremental.
Thank you for your review. We'll try to simplify the description of the method in Section 3.
The paper presents the idea of using Bloom filter encodings on the input features and the output layer of deep network to reduce the size of the network. Bloom-filter like encodings were proposed in Shi et al. (JMLR 2009) "Hash Kernels for Structured Data" (see Theorem 2 and its proof), whereas it was proposed to encode the outputs in the context of multilabel classification in Cisse et al. (NIPS 2013) "Robust Bloom Filters for Large MultiLabel Classification Tasks". The paper still joins both ideas together, applies it to ranking problems, and presents extensive experiments in the context of deep neural networks and language modelling.
The main motivation of the paper is the reduction of model size for deep neural networks. There is a whole body of literature on this topic,  that is not mentioned at all in the paper, where the baselines are based on weight quantization (with an available implementation in TensorFlow https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantization). More recent methods are e.g.,
1) the model compression approach of https://arxiv.org/abs/1510.00149
2) training with integer/binary weights https://arxiv.org/abs/1511.00363
Overall, the advantage of the Bloom filter approach is its simplicity and its wide applicability -- it could, as well, be used in conjunction with weight quantization and other compression methods. The main merits of the paper is to present extensive experiments on how well the vanilla Bloom filter approach can perform, but overall the novelty is fairly limited.
Thank you for your review. In the next iteration of the paper we will include and comment the Shi et al. (JMLR 2009) and Cisse et al. (NIPS 2013) references.
In addition, we will add the suggested "Tensorflow" and "integer/binary weights" references to our related work section.
Regarding the "model compression approach" reference, we'd like to note that we already cite it in Section 2, together with https://arxiv.org/abs/1511.06530.
Description:
This paper aims at compressing binary inputs and outputs of neural network models with unsupervised "Bloom embeddings".
The embedding is based on Bloom filters: projecting an element of a set to different positions of a binary array by several independent hash functions, which allows membership checking with no missed but with possibly some false positives.
Inputs and outputs are assumed to be sparse. The nonzero elements of an input are then simply encoded by Bloom filters onto the same binary array. The neural network is run with the embedded inputs.
Desired outputs are assumed to be a softmax-type ranking of different alternatives. a sort of back-projection step is needed to recover a probability ranking of the desired ground truth alternatives from the lower-dimensional output. For each ground-truth class, this is simply approximated as a product of the output values at the Bloom-filtered hash positions of that class.
The paper simply applies this idea, testing it on seven data sets. Scores and training times are compared to the baseline networks without embeddings. Comparison embedding methods are mostly very traditional (hashing trick, error-correcting output codes, linear projection by canonical correlation analysis) but include one recent pairwise mutual information based approach.
Evaluation:
It is hard to see a lot of novelty in this paper. The Bloom filters are an existing technique, which is applied very straightforwardly here. The back-projection step of equation 2 is also a straightforward continuous-valued variant of the Bloom-filter membership test.
The way of recovering outputs is heuristic, since the neural network inbetween the embedded inputs and outputs is not really aware that the outputs will be run through a further back-projection step.
In the comparisons of Table 3, only two embedding dimensionalities are used for each data set. This is insufficient, since it leaves open the question whether other methods could get improved performance for higher/lower embeddings, relative to the proposed method. (In appendix B, Figure 4, authors do compare their method to a variant of it for many different embedding dimensionalities; why not do this for all comparison methods too?)
Overall, this seems for the most part too close to off-the-shelf existing embedding to be acceptable.
Minor points:
As the paper notes, dimensionality reduction of inputs by various techniques is common. The paper lists some simple embeddings such as SVD based ones, CCA etc.
but a more thorough review of other approaches including the vast array of nonlinear dimensionality reduction solutions should be mentioned.
The experiments in the paper seem to have an underlying assumption that inputs and outputs need to have the same type of embedding dimension. This seems unnecessary.
Thanks for the (somewhat delayed) review. The main critic seems to be summarized in the phrase: "It is hard to see a lot of novelty in this paper. The Bloom filters are an existing technique, which is applied very straightforwardly here." That is a surprising way to criticize the work, since many advances in deep learning come about through the application of a relatively simple method to a network or model.
For instance, batch-normalization is the application of z-score normalization to each layer in the network, dropout is the application of sampling to the units of each layer in the network, embedding is adding a dense layer between the network and the input, residual learning is adding the input to the layer directly to the next layer, and much more. In that respect, we are somewhat surprised to see that the simplicity of our approach is seen as its main weakness.
Our approach is simple but yields very good results and we are not aware of any "off-the-shelf existing embedding" that provides all the benefits of Bloom embeddings (such as they can be applied to both inputs outputs, minimal computation overhead, increase in performance compared to alternatives or uncompressed networks in many cases, etc.). We would be grateful if the reviewer could point out any such "off-the-shelf existing embedding" with similar properties.
Regarding studying more embedding dimensionalities for competing approaches, we do not do so because the majority of such approaches do not scale in terms of time and space. Consider, for instance, the time required to perform an SVD decomposition from a 70K-by-70K sparse matrix (MSD data set) into 30K dimensions and the space to store the resultant 30K-by-30K dense projection matrix.
Moreover, considering different input and output projections would result in an unattainable number of experiments (InputDim \times OutputDim \times #Datasets (\times #Approaches)), which is somewhat prohibitive given our GPU resources. Note though that, even without optimizing for the 'right' input/output embedding dimensionality, our method still performs very well.
Updated PDF with suggested references.

Meta-review:
role: PC
data: comment: The authors present a simple scheme based on Bloom filters to generate embeddings for inputs and outputs. On the one hand, the scheme is simple, it reduces memory while introducing limited computational overhead. On the other hand, reviewers were concerned with the limited novelty of the approach, which diminishes quite a bit its impact. Overall, this paper is just a pinch too borderline.

Decision: Reject