Variational Hashing-based Collaborative Filtering with Self-Masking
Authors: Casper Hansen
Christian Hansen
Jakob Grue Simonsen
Stephen Alstrup
Christina Lioma
Venue:  (2020)
Keywords: hashing
collaborative filtering
information retrieval
supervised learning

Abstract: Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes. A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user. 
To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent. This allows a binary user-level importance weighting of each item without the need to store additional weights for each user. We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG. We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance.

Under review as a conference paper at ICLR 2020
VARIATIONAL HASHING-BASED FILTERING WITH SELF-MASKING
COLLABORATIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very ef- ficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes. A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user. To this end, we pro- pose end-to-end trainable variational hashing- based collaborative filtering ap- proach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent. This allows binary user-level importance weighting of each item without the need to store additional weights for each user. We experimentally evaluate our approach against state-of- tfteeof-the--a fi-the-art art base- lines on 4 datasets, and obtain significant gains of up to 12% in NDCG, We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance.
INTRODUCTION
Collaborative filtering (Herlocker et al.) 1999) is an integral part of personalized recommender sys- tems and works by modelling user preference on past item interactions to predict new items the user may like (Sarwar et al. 2001). Early work is based on matrix factorization approaches (Koren et al., 2009) that learn a mapping to a shared m-dimensional real-valued space between users and items, such that user-item similarity can be estimated by the inner product. The purpose of hashing- based collaborative filtering (Liu et al., 2014) is the same as traditional collaborative filtering, but allows for fast similarity searches to massively increase efficiency (e.g.. realtime brute-force search in bil- lion items (Shan et al. 2018) This is done by learning semantic hash functions that map users and items into binary vector representations (hash codes) and then using the Hamming distance (the sum of differing bits between two hash codes) to compute user-item similarity. This leads to both large storage reduction (floating point versus binary representations) and massively faster computation through the use of the Hamming distance.
One problem with hashing-basee based collaborative filtering is that each bit is weighted equally when computing the Hamming distance. This is problem because the importance of each bit in an item hash code might differ between users. The only step towards addressing this problem has been to associate a weight with k-bit blocks of each hash code (Liu et al. 2019). However, smaller values of k lead increased storage cost, but also significantly slower computation due to the need of com- puting multiple weighted Hamming distances. To solve this problem, without using any additional storage and only a marginal increase in computation time, we present Variational Hashing- based collaborative filtering with Self-Masking (VaHSM-CF). VaHSM-CF is our novel variational deep learning approach for hashing-based collaborative filtering that learns hash codes optimized for self- masking. Self-masking is novel technique that we propose in this paper for user-level bit- bit-weighting on all items. Self leff-masking modifies item hash codes by applying an AND operation between an item and user hash code, before computing the standard Hamming distance between the user and self- masked item hash codes. Hash codes optimized with self- masking represent which bit bit-dimensions encode properties that important for the user (rather than bitwise -1/1 preference towards each

---

Under review as a conference paper at ICLR 2020
property). In practice, when ranking set of items for specific user, self self-masking ensures that only bit differences on bit-dimensions that are equal to 1 for the user hash code are considered, while ignoring the ones with a -1 value, thus providing user-level bitwise binary weigthing. Since self- masking is applied while having the user and item hash codes in the lowest levels of memory (i.e., register), it only leads
very marginal efficiency decrease.
We contribute (i) a new variational hashing- based collaborative filtering approach, which is opti- mized for (ii) a novel self-masking technique, that outperforms state-of-the-art baselines by up to 12% in NDCG across 4 different datasets, while experimentally yielding less than 4% runtime over- head compared to the standard Hamming distance. We publicly release the code for our model, as well as an efficient implementation of the Hamming distance with self solf-masking"
RELATED WORK
We focus on collaborative filtering with explicit feedback, which assumes that users and items are related via user-specified rating: the task to rank pool of pre-selected items. This is different from implicil feedback, where the task is to estimate the pool of items that are of interest to the user. Matrix factorization is one of the most successful collaborative filtering methods (Koren et al.] 2009), but to reduce storage requirements and speed up computation, hashing- based collaborative filtering has been researched. For hashing-basee hash codes (as opposed to real-valued vectors), such that the highly efficient Hamming distance (as opposed to the inner product) can be used for computing user-item similarities.
methods the users and items are represented as binary
Two-stage approaches. Early hashing-based collaborative filtering methods include two stages: First, real-valued user and item vectors are learned, and then the real-valued vectors are transformed into binary hash codes. ZZhang et al. (2014) employ matrix factorization initially, followed by a bi- nary quantization of rounding the realev vectors, while ensuring that the hash code is preference preserving of the observed user-item ratings using their proposed Constant Feature Norm constraint. Zhou & Zha (2012) and Liu et al.] (2014) both explore binary quantization strategies based on or- thogonal rotations of the real-valued vectors, which share similarities with Spectral Clustering YYu) & Shi 2003). However, the two-stage approaches often suffer from large quantization errors (Zhang al. 2016: Liu al. 2019), because the hash codes are not learned directly, but rather based on different quantization procedures.
Learned hashing approaches. Zhang et al. (2016) propose Discrete Collaborative Filtering (DCF), which is a binary matrix factorization approach that directly learns the hash codes using relaxed in- teger optimization, while enforcing bit balancing and decorrelation constraints. Extensions of DCF have focused on incorporating side-infone inle-nfformation (e.g., reviews associated with rating) (Lian et al.) 1017|iu et al. [2188] Zhang et al., (219)) and have been redesigned for implicit feedback signals (Zhang et al. 2017). More recent work addresses the problem that hashing-based based collaborative fil- tering methods have reduced representational power compared to real-valued vectors, but increasing the hash code dimensionality to match the amount of bits used in the real-valued case hurts model generalization (Liu et al., 2019). To address this, [iu et al.] (2019) propose Compositional Cod- ing for Collaborative Filtering (CCCF), which is broadly similar to learning compositional codes for (word) embedding compression (Chen et al.] 2018) [hu & Nakayama| 2018). CCCF is a hy- brid approach that combines hash codes and real-valued weights: each hash code is split into k blocks of r bits each, and each block is associated with a real-valued scalar indicating the weight of the block. The distance between two CCCF hash codes is then computed as a weighted sum of the Hamming distances of the individual blocks, where each weight is the product of each block's 's weight. The problem with this approach is that each block requires an individual Hamming distance computation, as well as floating point multiplications of the block weights. In fact, the CCCF block construction no longer allows for highly efficient Boolean operations because the distance computa- tion is weighted by each block's 's weight. Another problem with CCCF is that it drastically increases storage requirements by needing to store the real-valued weights for all blocks in a hash code.
In contrast to CCCF, our proposed variational hashing- collaborative filtering with self- masking solves the same problems, as it effectively allows to disable unimportant bits - corre- sponding 1-bit block size with 0/1 weights without needing to store any additional weights 'The code is available at anoonymm zed- for- submi ssion
2

---

Under review as a conference paper at ICLR 2020
vectors. Additionally, after having applied the self- user-item similarity still be computed using only a single Hamming distance on the two hash codes.
3 HASHING-BASED COLLABORATIVE FILTERING
Hashing-based Collaborative filtering aims to learn binary user and item representations (called hash codes), such that the distance between the representations indicates how well user likes item i. In practice, the Hamming distance is used due to its fast hardware-level implementation. Formally, we learn 2n {-1,1} and 2; E {-1, 13mh, where m is the number of bits in the hash code, which is typically chosen to fit into a machine word. The preference of user u for item i is specified by the rating Ru,, E f1, 2, 3..,, K}. where K is the maximum rating, such that the Hamming distance between 2, and 2; is low when Ru, is high. Computing the Hamming distance computation is extremely efficient due to fast hardware-level implementation of the Boolean operations, as
Hamming 2,) SUM XOR 2:)
where SUM is computed fast on hardware using the popent instruction. Given and set of items, the integer-valued Hamming distances can be linear-time sorted using e.g. radix sort (because Hamming distances are bounded in [0, ml) in ascending order to create ranked list based user preference (Shan et al. 2018).
3.1 SELF-MASKING
The Hamming distance assigns equal weights to all bits, but in reality bit importance might differ among users. For example, if consider each bit encoding of specific property of an item (e.g., a movie being a thriller), then the weight of each property would be dependent on each user's preference. However, since the hash codes are binary, it is not possible weights without using more storage and computation time due to no longer operating on binary val- ues. In fact, existing method even allows disabling specific bits for certain users (corresponding to the case of 0 preference weight). We next present a solution for the latter problem, which en- codes the importance of each bit directly into the user hash code, and therefore does not require any additional storage. We define the Hamming distance with self-masking
encode such preference
Hamming Seltfmna (Zus z;) = SUM XOR (z+ AND Z.1
(2)
sell-musking
We first apply an AND operation between the user and item hash codes, and then compute the Ham- ming distance between that and the user hash code. This fundamentally changes the purpose of the user hash code: instead of encoding a positive or negative preference for property, it encodes which properties important to the user (-1's from the user hash code are copied to the item due the AND operation). This allows the model to disable unimportant bits on a user-level, meaning that we enable the model produce user-specific item representations while still only storing single hash code for each user and item respectively. Self-masking requires an additional Boolean operation, but since this is applied once the hash codes are already placed in the lowest levels of memory (i.e., register), it only leads to marginal decrease in efficiency (see Section 4.6 for an empirical analysis).
3.2 VARIATIONAL HASHING-BAN -BASED COLLABORATIVE FILTERING
To derive a variational setup for hashing based collaborative filtering, we define the likelihood of user, u, and the likelihood of item as the product over the likelihoods of the observed user specified ratings:
p(u) II
Po Po RRn,))
SEI
(())
II WEU
Po (R...)
where 9 are the parameters of the (neural network) model. This formulation enforces a dual sym- metric effect of users being defined by all their rated items, and items being defined by the ratings
3

---

Under review as a conference paper at ICLR 2020
provided by all the users. To maximize the likelihood of all observed items and users, need to maximize the likelihood of the observed ratings Po i...). Note that instead of maximizing the consider the log likelihood derive the objective below. We assume that the
raw likelihood, likelihood of a rating, (Ra,),,, is conditioned on two latent vectors, a user hash code žu, and an item hash code 2;- To obtain the hash codes of the user and item, assume that 2u and 2; each sampled by repeating m Bernoulli trials, which have equal probability of sampling -1 and 1. This gives us the following log likelihood, which we wish maximize:
log (R.,)) log
Σ
po Ro zi 2i)(2z))(z.)
(5)
RE 1,1
-.6(-1.1)
The latent vectors 2n and the user and item respectively. To do this we first multiply and divide with the approximate posterior distributions (0(=)l), q (=)
and item hash code, and should therefore be conditioned on
go(zz got (zulu))
log (a.s) = log
Σ 4 Zu 1,1
Ra, zi 2.) Po
zn)
(6)
go (=;1)) (u)
where y and are the parameters of the approximate posteriors. We can now rewrite to an expecta- tion and apply Jensen's 's inequality to obtain a lower bound on the log likelihood:
log po (Ry,,) 2 (go(s log [ou Zur 21) p(z;) p(zu)
2o (=(=i (1) Qei (Zu un)
=Bag(sl log [po (Ra, Eus 2;)] + log ((:) - log (o
+lgp((z) log go (zulu))
Since 2, and z, will be sampled independently, then (0(z)l)) and (zulll are independent and can rewrite to the variational lower bound:
log po (R.,s) 2 (z.l))))) loglpe (2us 2]]]
(z(((())) (L(gx(zulu)) p(zu))
where KL(..) is the Kullback-Leible divergence. Thus, to maximize the expected log likelihood of the observed rating, we need to maximize the conditional log likelihood of the rating, while min- imising the KL divergence between the approximate posterior and prior distribution of the two latent vectors. Maximizing the expected conditional log likelihood can be considered reconstruction term of the model, while the KI. divergence can be considered as regularizer.
Next present the computation of the approximate posterior distributions (2,()) and (Zull)) (Section 3.3 and the conditional log likelihood of the rating po (Zu,ilzus z:) (Section 3.4)
3.3 COMPUTING THE APPROXIMATE POSTERIOR DISTRIBUTIONS
The approximate posterior distributions can be seen as two encoder functions modelled through a neural network by considering the functions as embedding layers. Each encoder function maps either user or an item to a hash code. Next, we focus on the derivation of the encoder function for the user, as they are both computed in the same way. The probability of the j bit is given by:
d (Eulu)) = (B())
where E!(") is the j' t'th entry learned real-valuee embedding E for user u, and is the sigmoid function. The j'th bit is then given by:
( = [o( - (
(10)
where ( is either chosen stochastically by sampling م from a uniform distribution in the interval [0,1], or chosen deterministically to be 0.5, which can be used for evaluation to obtain fixed hash codes. Note that μ is sampled for each bit. As the sampling is non-differentiable, a straight- estimator (Bengio al. (013)) is used for backpropagation.
4

---

Under review as a conference paper at ICLR 2020
0.3
0.9
affine
0.6
-Bernoull
Har. Dist.
transformation
user U
MSE
0.3
0.8
R
0.6
item
-Bernoulli
AND
Self-masked item
0.9
0.7
Figure 1: Model overview: The m-dimensional user and item sampling probabilities are learned and then sampled by repeating m Bernoulli trials. The AND operation denotes the self-masking. The model is optimized using MSE between an affine transformation of the Hamming distance and the observed rating.
3.4 COMPUTING THE CONDITIONAL LOG LIKELIHOOD
The conditional log likelihood can be considered reconstruction of the rating, given the user and item hash codes. We model the observed ratings as a ground truth rating with additive standard nor- mally distributed noise, which is then discretized to the observed categorical rating. The conditional log likelihood can then be computed as:
po CeuRullae 24) = Ni f(zu: x2)
(11)
where f(zu, 2:) is function that reconstructs the rating given the user and item hash codes. Max- imising the log likelihood, log po Ru., Euz 2;), corresponds to minimising the mean squared error (MSE) between R,,,; and [ 2.). which is done for training the model. Existing work on hashing- based collaborative filterig (Liu et al. 2019 Zhang et al., (016)) also employs a MSE objective, and thus implicitly make the same normal distribution assumption done in this work.
We define the reconstruction function to the self-masking Hamming distance from Eq.
f(a 2:) = g (Hamming (Zu: 2;))
(12)
where is a fixed affine transformation that maps the interval of the Hamming distance to the interval of the ratings, such that the minimum and maximum of the Hamming distance correspond to the minimum and maximum of the ratings. The model is now fully differentiable and can be trained end-to-end using backpropagation, such that the network is able optimize the hash codes directly for self-masking A depiction of the model is provided in Figure It should be noted that while variational autoencoders are generative models, we not explicitly utilize this in our model, and are primarily concerned with the reconstruction of the observed ratings.
EXPERIMENTAL EVALUATION
4.1 DATASETS AND EVALUATION METRICS
We evaluate 4 publicly available datasets commonly used in prior work (Zhang et al., 2016 Liu
eet al., 2019) Zhang et al., 2017; Liu et al. 2018 Lian et al.) (017) and summarized in Table
Specifically, we use: two movie rating datasets, Movielens IM (ML-IM) and Movielens 10M (ML-10M); a Yelp dataset with ratings of e.g., restaurant and shopping malls") and book rating dataset from Amazor He & McAuley 2016). Similarly Rendle et al. (2009). we filter the data
such that all users and items have at least 10 ratings. Following Zhang et al. (2016), for each user
5

---

Under review as a conference paper at ICLR 2020
Dataset
#Rarings
#fims
#Users
Density
Range of ratings
ML-IM
998,539
3,260
6,040
5.07%
1-5
ML- 10M
9,995,471
9,708
69,878
1.7%%
1-10
Yelp
602,517
14,873
22,087
0.18%
1-5
Amazon
4,701,968
128,939
158,650
0.02%
1-5
Table Dataset statistics. Density: % ratings relative to the total amount of user-item combinations.
50% of the ratings are used for testing, 42.5% are used for training, while the last 7.5% are used for validation.
We evaluate our method, VaHSM-CF, and all baselines (see Section 4.2) using Normalised Dis- counted Cumulative Gain (NDCG) (Järvelin & Kekäläinen 2000), which is often used to evaluate recommender systems with non-binary ratings (or relevance values). We use the average NDCG at cutoffs (2, 6, 10) over all users and report the average for each cutoff value.
4.2 BASELINES
We use as baselines the state-of-the-art methods for hashing-based collaborative filtering (see Sec- tion 2). standard matrix factorisation, and two different strategies for binarising the output of the matrix factorisation We include the standard matrix factorization as reference traditional real-valued collaborative filtering baseline, in order to highlight the performance gap between the hashing-based approaches and real-valued approach.
DCF Zhang et al. 2016) learns user and item hash codes through a binary matrix factorization solved as relaxed integer problem.
CCCF (Liu et al. 2019) learns hash codes consisting of k blocks, where each block has r bits. A floating point weight associated with each block for computing user-item similarities weighted sum of block-level Hamming distances. In the original paper, the floating point weights are not counted towards the amount of bits used. thus leading to an unfair advantage. For a fair comparison, we count each floating point weight as 16 bits in the following experimental comparison.
MF Koren et al. 2009) the classical matrix factorization based collaborative filtering method, where latent real-valued vectors are learned for users and items. We set the latent dimension to be the same as the number of bits used in the hashing-
approaches.
MF mean and MF median are based on MF, but use either each dimension's 's mean median for do- ing the binary quantization to bits (Zhang al., 2010). We include these to highlight the large quantization loss occurring when the hash codes are not learned directly.
VaH-CF is our proposed method without self-masking. We use show the effect of neural variational hashing-bast collaborative approach without self-masking.
4.3 TUNING
We train both VaHSM-CF and VaH-CF using the Adam optimizer (Kingma & Ba 2014), and tune the learning rate from the set (0.005, 0.001, 0.0005), where 0.001 is always chosen across all data sets. The batch size is chosen from the set {100, 200, 400, 800), where 400 was always chosen. To reduce over-fitting, Gaussian noise added to the ratings during training, as noise injection has been found beneficial in multiple domains for variational neural models (Sohn et al. 2015). For the noise injection, we initially set the variance of the Gaussian to I and reduce by factor of 1 - 10-4 every iteration during training. Our model is implemented using the Tensorflow Python library (Abadi et al. 2016), and all experiments are run on Titan X GPUs.
All hyper parameters for the baselines are tuned using the same set of possible values as in the original papers. For the CCCF baseline, we consider block sizes of (8, 16, 32, 64) and each floating
All hyperparameterss are tuned on the validation data as described in the original papers.
https: / / thub. com/ 31 4010440111400010
Provided as baseline the CCCF h

---

Under review as a conference paper at ICLR 2020
32 bits
ML-1M
ML-T0M
Yelp
Amazon
0.3635
0.5806
0.6022
0.3788
0.4190
D.4665
0,6003
0.7192
0.770S
0.223]
0.8011
0.8376
0.5689
0.5792
0.6021
0.3793
0.4190
D.4064
0,0038
0.7194
0.7717
0.2198
0.7927
0.8348
DCF
0.7088
0.5275
0.5618
0.6588
0,7642
0.8080
COCP
0.6507
0.6768
0.7003
0.5227
0.558
0.5982
0.6417
0,7506
0.7978
VaH-CF
0.6755
0.6916
0.7177
0.5322
0.5745
0.6137
0.6668
0.7694
0.8124
0.7795
0.8419
0.872
VaHSM-CI
0.7362
0.730M
0.7408
0.5555
0.5892
0.6349
0.7424
0181166
0,8517
0.8081
0.8618'
0.8874
MF (real-valued)
0.7375
0.7567
0.7502
0.5915
0.6107
0.6428
0.7553
0,8288
0.8564
0,8185
0.8686
0.8929
6f biss
ML-IM
ML-10M
Yolp
Amazzon
Naco67
soroos
snctiee
sncocm
0.3714
0.5826
0.6069
0.3825
0.4211
0:4675
0.0050
0.7195
0.7716
0.7238
0.8007
0.8371
0.6049
0.3835
0.4217
0.7230
0,7752
0.7205
0.7983
0.8354
ICF
0.6922
0.2043
0.7212
03528
0.5221
0.6193
0.6708
0.7725
0.8145
0.7788
0.8418
0.3710
COCF
0,6862
0.7008
0.7230
0,530
0,5755
D66127
0.6528
0.7584
0,8039
VaH-CF
0.7011
0.712
0.7310
0.5621
0.5960
D.6322"
0.769
0.775
0.8178
0.7810
0.8166
0.8733
VaHIMM-CT
0.720M
0.7277
0.7441
0.5815'
0.6022
0.6399
0.75IT
0.8224
0,8533
0.8162
0,8663
0.8911
MI (real-valued)
0.744
0.2438
0.7552
0.5905
0.6101
0.0423
0.7541
0.8348
0.8573
0.8235
0.8711
0.8948
Table 2: NDCG@K of our method (VaHSM-CF) against baselines using hash codes of length 32
and 64 bits. The missing results for CCCF on Amazon are due to the model requiring more than 128 GB of RAM. Statistically significant improvements using a paired two tailed t-test the 0.05 level, compared to the best existing hashing-based baseline (DCF), are indicated by *
point weight counts for 16 bits. We try all possible combinations that fit within the bit budget, and if a single block is chosen, then the weight is not included in the bit calculation.
4.4 RESULTS
The results are shown in Table 2 for hash code lengths of 32 and 64 bits, as these correspond to common machine word sizes. The highest NDCG per column is shown in bold (among hashing- based baselines), and results statistically significantly better than the best hashing-based baseline (DCF), using a paired two tailed t-test the 0.05 level, are indicated by an asterisk The Amazon results for CCCF are not included, as the released implementation requires excessive amounts of RAM (> 128GB) on this dataset due to the large amount of items and users.
Our proposed VaHSM-CF significantly outperforms all hashing bashing-based baselines across all datasets by up to 12%. Our VaH-CF without self-masking second best (up to 2% better than the hashing- based baselines), although on ML-IM the VaH-CF results are not statistically significantly bet- ter than DCF. This highlights the benefit of modelling hashing-basee collaborative filtering with a variational deep learning based framework, which is notably different from existing hashing-bart based methods based on dicrete matrix factorization solved as relaxed integer problems. Most importantly however, it shows the significant improvement self-masking brings to hashing- -based collaborative filtering. We observe that the top 3 baselines (including our VaH-CF) generally obtain similar scores, which highlights the difficulty of improving performance without changing how the hash codes are used (as done by self-masking).
The real-valued MF baseline outperforms all the hashing-based approaches, which is to be expected as the representational power of of 32/64 floating point numbers are notably higher than 32/64 bits. However, our VaHSM-CF bridges large part of the gap between existing hashing-based methods and MF, such that the NDCG difference in most cases is below 0.01. Additionally, the large performance decrease by both the mean or median rounding shows the large quantization error obtained if the hash codes are not learned directly, as done by our and the existing hashing-based approaches.
4.5 MODEL ANALYSIS
How self-masking influences the convergence rate of the model. Tigure2a/shove the convergence rate for the ML-1M dataset with and without self-masking We see that training with self-masking significantly improves the convergence rate compared to the model without self-masking. Since the time for a single epoch is approximately the same with and without self malf-nasking, we conclude that self-mar masking not only improves NDCG, but also reduces training time by a very large margin.
7

---

Under review as a conference paper at ICLR 2020
Convergence plet for MI-1M
NDCG@10 with varyling stochastic and deterministic evaluationvtranning
Stoc Exol Stoc.rron
Stoc Exol Det.Tein
Evol StaciTain
Des Eval Det Deah
var1380n ND152g10
0.7
0.82
0.80
0.5
3.5
335
30
100
150 200 0.1 MM-1M
sports
MI-10M
Yelp
Amrwon
(a)
(b)
Figure 2: 2a shows convergence using the validation NDCG@ 10 on ML-IM, where self-masking significantly speeds up convergence. We observe the same trend the other datasets (see Appendix [. [6] Shows the test NDCG@ 10 when varying whether hash codes are sampled stochastically or deterministically while training and for evaluation. For example, Eval + Stoc. Train corresponds to deterministic sampling of hash codes for evaluation, while sampling stochastically when training the model.
Convergence plots for the remaining datasets are shown in the Appendix, where we observe the same trend.
Stochastic or deterministic sampling. We investigate the effect of the sampling strategy for the hash codes (see Eq. [0p during training and evaluation. The sampling can either be deterministic (1(55) 0.5) or stochastic (pl1) sampled uniformly at random from [0, 1],, and does not have to be the same for training and evaluation. Figure 2b shows the performance for these 4 configura- tions across all datasets. We see that stochastic training with deterministic evaluation performs the best, while deterministic training and deterministic evaluation perform second best. As expected, stochastic sampling at evaluation performs significantly worse than deterministic sampling, as every item has a very small probability of being sampled such that it has small Hamming distance to user, even though it has a low rating (and vice versa for highly rated items).
4.6 RUNTIME ANALYSIS
Self-masking has an additional cost added to the standard Hamming distance, due to the additional AND operation between the user and item hash codes (see Eq. T and 2 We now investigate the actual runtime cost associated with this modification.
We implement both the Hamming distance and Hamming distance with self-masking efficiently in Cn a machine with a 64 bit instruction set. test environment was made with 64 bit hash codes for 100,000 users and 100,000 items. For each user, the distances were computed to all items using both the Hamming distance and Hamming distance with self self-masking. We measure the actual time taken for computing the distances to all items from each user, and report the average over 50 repeated runs. All experiments are run on a single thread T0 with all users and items loaded in RAM. The code was compiled with the highest optimization level, and utilizing all optimization flags applicable to the hardware. We verified the produced assembler code used the efficient popent instruction.
The mean experiment time was 8.0358s when using the Hamming distance, and 8.3506s when using the Hamming distance with self-masking. Thus, self-masking only adds a runtime overhead of
3.91% compared using the standard Hamming distance. As in this setup we are only computing the distances, this can be seen as an upper bound of the actual overhead in complete system, as the remaining operations (e.g., sorting) would be the same with and without self-masking. Thus, this provides a good trade-off compared to the large performance gains it yields. Note that the measured times are for the total 10 distance computations, highlighting the scalability of hashing- based methods to datasets of massive scale. For comparison, if the experiment is repeated with computing
We used Intel(R) Xcon(R) CPU E5-2670 v2 @ 2.50GHz.
8

---

Under review as a conference paper at ICLR 2020
the dot product of floating point vectors of size 64, then the computation time is 648.5632s, thus close to 80x slower.
CONCLUSION
We proposed an end-to-end trainable variational basshing-based collaborative filtering method, which optimizes hash codes using a novel modification to the Hamming distance, which we call self- masking. The Hamming distance with self-masking first creates a modified item hash code, by applying AND operation between the user and item hash codes, before computing the Hamming distance. Intuitively, this can be seen as ignoring user-specified bits when computing the Hamming distance, corresponding to applying a binary importance weight to each bit, but without using more storage and only a very marginal runtime overhead. We verified experimentally that our model outperforms state-of-the-an baselines by up 12% in NDCG different cutoffs, across 4 widely used datasets. These gains come at a minimal cost in recommendation time (self-masking only increased computation time by less than 4%).
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In Proceedings the USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 265-283, 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Ting Chen, Martin Renqiang Min, and Yizhou Sun. Learning k-way d-dimensional discrete codes for compact embedding representations. In Proceedings of the International Conference on Machine 2018.
Learning, pp.
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the International Conference on World Wide Web, pp. 507-517, 2016.
Jonathan L Herlocker, Joseph A Konstan, Al Borchers, and John Riedl. An algorithmic framework for performing collaborative filtering. Proceedings of the International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 230-237, 1999.
Kalervo Järvelin and Jaana Kekäläänen. IR evaluation methods for retrieving highly relevant docu- ments. In Proceedings of the International ACM SIGIR Conference on Reseurch and Development in Information Retrieval, pp. 41-48, 2000.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2014.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, August 2009.
Defu Lian, Rui Liu, Yong Ge, Kai Zheng, Xing Xie, and Longbing Cao. Discrete content-aware ma- trix factorization. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 325-334, 2017.
Chenghao Liu, Tao Lu, Xin Wang, Zhiyong Cheng, Jianling Sun, and Steven C.H. Hoi. Com- positional coding for collaborative filtering. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 145-154, 2019.
Han Liu, Xiangnan He, Fuli Feng, Liqiang Nie, Rui Liu, and Hanwang Zhang. Discrete factoriza- tion machines for fast baaed recommendation. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3449-3455, 2018
Xianglong Liu, Junfeng He, Cheng Deng, and Bo Lang. Collaborative hashing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2139-21444 2146, 2014.

---

Under review as a conference paper at ICLR 2020
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thies Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pp. 452 461, 2009.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the International Conference on World Wide Web, pp. 285-295, 2001.
Ying Shan, Jie Zhu, JC Mao, et al. Recurrent binary embedding for gpu-enabled exhaustive re- trieval from billion-scale semantic vectors. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 2170-2179, 2018.
Raphael Shu and Hideki Nakayama. Compressing word embeddings via deep compositional code learning. In Proceedings of the International Conference Learning Representations, 2018.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, pp. 3483-3491, 2015.
Stella X. Yu and Jianbo Shi. Multiclass spectral clustering. In Proceedings of the IEEE International Conference Computer Vision Volume 2, pp. 313-319, 2003.
Dell Zhang, Jun Wang, Deng Cai, and Jinsong Lu. Self-taught hashing for fast similarity search. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 18-25, 2010.
Hanwang Zhang, Fumin Shen, Wei Liu, Xiangnan He, Huanbo Luan, and Tat-Seng Chua. Discrete collaborative filtering. In Proceedings of the International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 325-334, 2016.
Yan Zhang, Defu Lian, and Guowu Yang. Discrete personalized ranking for fast collaborative tering from implicit feedback. In Proceedings of the Thirty-First AAAI Conference Artificial Intelligence, 2017.
Yujia Zhang, Jun Wu, and Haishuai Wang. Neural binary representation learning for large-scale collaborative filtering. IEEE Access, 7:60752-60763, 2019.
Zhiwei Zhang, Qifan Wang, Lingyun Ruan, and Luo Si. Preference preserving hashing for efficient recommendation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 183-192, 2014.
Ke Zhou and Hongyuan Zha. Learning binary codes for collaborative filtering. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp- 498-506, 2012.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1
CONVERGENCE PLOTS
Convergence plots for Yelp, Amazon, and ML-10M are shown in Figure We observe similar trend to ML-IM in Figure 2a where the self-masking leads to notably faster rate of convergence.
Convergence plst
Convergence plat
0.962
Sell-masking
0.M
Sell-makking
mel-manknn
0.933
0.50
>altrrbn MB-24013
0.912
0.78
Convergence plot for ML-J0M
0.04
vMl Si-Sttin N0.00010
0.02
0.78
Sell-
0.76
50
1.50
230
300
6pcchs
Figure 3: Convergence plot for Yelp, Amazon, and 10M.
11


Reviews:
[{"review_id": "SJgFWI0B9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed an end-to-end variational hashing-based collaborative filtering scheme with self-masking to solve the efficiency of large-scale recommender systems. It mainly targets at addressing a problem in existing hashing-based collaborative filtering methods, where each bit is equally weighted in the Hamming distance computing process. To this end, the presented hashing scheme develops a self-masking technique to encode which bits are important to the user. The comparative experiments on several datasets demonstrate the superior performance of the presented hashing method.\n\nThe idea is interesting in the sense that an efficient self-masking technique is proposed to generate user-adaptive hash codes, by differentiating the importance of binary bits, for a fast recommendation system. However, the issues with the experimental results make me inclined to reject this paper.\n\nIn Table 2, the reported results of MF and its variants are lower than those of DCF. These results are very unreasonable. I have also conducted many similar experiments and the obtained results are not consistent with the results this paper reported.\n\nIn addition, in the original paper, CCCF is reported to achieve the superior performance over DCF. However, in this paper, the results are also inconsistent.\n\nTo sum up, the reported results in the paper are not convincing to me. I would doubt that there is no much accuracy improvement compared against the state-of-the-art methods.\n\nFinally, as I understand, using variational hashing to maximize the likelihood of all observed items and users will inevitably increase the training time. The presented experiments, however, include no results on the efficiency comparison.\n\nIf there are no advantages on the recommendation accuracy and efficiency, the motivations of this designed hashing method are vague.\n\nBased on the above reasons, I tend to reject this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyevKp7TFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author proposes a variational encoder (VAE) based approach to perform hashing-based collaborative filtering, which focuses on the weighting problem of each hash-code bit by applying the \"self-masking\" technique. This proposed technique modifies the encoded information in hash codes and avoids the additional storage requirements and floating point computation, while preserving the efficiency of bit manipulations thanks to hardware-based acceleration. An end-to-end training is achieved by resorting the well-known discrete gradient estimator, straight-through (ST) estimator. \n\nStrength:\nThe idea of employing the discrete VAE framework to perform hashing-based collaborative filtering is interesting. From the perspective of applications, I think it is somewhat novel.\n\nThe experimental results demonstrate the performance superiorities of the self-masking hashing-based collaborative filtering method. \n\n\n\nWeakness:\n\nI have concerns over using the VAE to model the ratings of each user and item pairs here. Essentially, you are building a VAE for every rating value R_u, i.  Although the parameters are shared, the VAE’s have their own prior. For generative models like VAE, they need to learn from a lot of data, not just one data point. From the perspective of generating data looking similar to the training data, I don’t think your model have learned anything. Only the reconstruction part is important to your model. \n\nThe idea of using discrete VAE for hashing tasks has been explored before, see “NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing”.  It employed a very similar idea, although it is not used for CF, but for hashing directly. The novelties of the model is limited.\n\nSince ratings are essentially ordinal data, using Gaussian distribution to model the rating data may be not appropriate.\n\nThe whole paper, especially the model, is not presented well. The model is not presented in a rigorous way, and some sentences in this paper are difficult to follow.\n\nThe runtime analysis is not sufficient. In addition to comparing with the methods using hamming distance, we also want to see the advantages of the hamming based method over the real-value based method on speed acceleration.\n\n\nOther question:\n\nTo realize the self-masking role, the paper proposed to use the function f(z_u, z_i)=g(Hamming_self-mask(z_u, z_i)), Equ. 12, and demonstrate the effectiveness of using this function by experiments. Since the necessity of using self-masking is not very convincing, I doubt whether this self-masking function f(z_u, z_i) is indispensable. Maybe, if some other functions that takes z_u and z_i as input are used, better results may be observed.  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lGdIJEKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study a classical (and well-studied) problem called rating prediction from a new perspective, i.e., learning binary vector representations of the users' and items' latent representations for efficiency. In particular, the authors introduce a personalized self-masking shown in Eq.(2) and in Figure 1 (the 'AND' operation) in order to improve the previous hashing-based collaborative filtering methods without increasing the time cost much.\n\nEmpirical studies on four public datasets show the effectiveness of the proposed model, i.e., variational hashing-based collaborative filtering with self-masking (VaHSM-CF).\n\nSome comments/suggestions:\n\n1 The studied problem (i.e., rating prediction) is well studied in the community of recommender systems, and there are many more accurate algorithms than the basic MF algorithm used in the empirical studies. I thus suggest the authors to include more such algorithms though the focus of this paper is for efficiency (but the authors also claim the accuracy of the proposed model).\n\n2 The authors are suggested to give a brief explanation on choosing those baseline methods in the context of other hashing-based CF methods.\n\n3 Some details are missing, e.g., the number of latent dimensions in MF. And some presentation in the parameter setting are not clear, e.g., 'is chosen consistently across all data sets', 'was consistently chosen'. Does 'consistently' means 'exactly the same'?\n\nMinors:\nThere are some typos: 'a user, u, and', 'of e.g., restaurant and shopping malls', etc.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a clear consensus amongst reviewers that the paper should not be accepted. This view was not changed by the rebuttal. Thus the paper is rejected. 
Decision: Reject