Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: In this paper, the authors proposed an end-to-end variational hashing-based collaborative filtering scheme with self-masking to solve the efficiency of large-scale recommender systems. It mainly targets at addressing a problem in existing hashing-based collaborative filtering methods, where each bit is equally weighted in the Hamming distance computing process. To this end, the presented hashing scheme develops a self-masking technique to encode which bits are important to the user. The comparative experiments on several datasets demonstrate the superior performance of the presented hashing method.

The idea is interesting in the sense that an efficient self-masking technique is proposed to generate user-adaptive hash codes, by differentiating the importance of binary bits, for a fast recommendation system. However, the issues with the experimental results make me inclined to reject this paper.

In Table 2, the reported results of MF and its variants are lower than those of DCF. These results are very unreasonable. I have also conducted many similar experiments and the obtained results are not consistent with the results this paper reported.

In addition, in the original paper, CCCF is reported to achieve the superior performance over DCF. However, in this paper, the results are also inconsistent.

To sum up, the reported results in the paper are not convincing to me. I would doubt that there is no much accuracy improvement compared against the state-of-the-art methods.

Finally, as I understand, using variational hashing to maximize the likelihood of all observed items and users will inevitably increase the training time. The presented experiments, however, include no results on the efficiency comparison.

If there are no advantages on the recommendation accuracy and efficiency, the motivations of this designed hashing method are vague.

Based on the above reasons, I tend to reject this paper.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The author proposes a variational encoder (VAE) based approach to perform hashing-based collaborative filtering, which focuses on the weighting problem of each hash-code bit by applying the "self-masking" technique. This proposed technique modifies the encoded information in hash codes and avoids the additional storage requirements and floating point computation, while preserving the efficiency of bit manipulations thanks to hardware-based acceleration. An end-to-end training is achieved by resorting the well-known discrete gradient estimator, straight-through (ST) estimator. 

Strength:
The idea of employing the discrete VAE framework to perform hashing-based collaborative filtering is interesting. From the perspective of applications, I think it is somewhat novel.

The experimental results demonstrate the performance superiorities of the self-masking hashing-based collaborative filtering method. 



Weakness:

I have concerns over using the VAE to model the ratings of each user and item pairs here. Essentially, you are building a VAE for every rating value R_u, i.  Although the parameters are shared, the VAE’s have their own prior. For generative models like VAE, they need to learn from a lot of data, not just one data point. From the perspective of generating data looking similar to the training data, I don’t think your model have learned anything. Only the reconstruction part is important to your model. 

The idea of using discrete VAE for hashing tasks has been explored before, see “NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing”.  It employed a very similar idea, although it is not used for CF, but for hashing directly. The novelties of the model is limited.

Since ratings are essentially ordinal data, using Gaussian distribution to model the rating data may be not appropriate.

The whole paper, especially the model, is not presented well. The model is not presented in a rigorous way, and some sentences in this paper are difficult to follow.

The runtime analysis is not sufficient. In addition to comparing with the methods using hamming distance, we also want to see the advantages of the hamming based method over the real-value based method on speed acceleration.


Other question:

To realize the self-masking role, the paper proposed to use the function f(z_u, z_i)=g(Hamming_self-mask(z_u, z_i)), Equ. 12, and demonstrate the effectiveness of using this function by experiments. Since the necessity of using self-masking is not very convincing, I doubt whether this self-masking function f(z_u, z_i) is indispensable. Maybe, if some other functions that takes z_u and z_i as input are used, better results may be observed.  

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors study a classical (and well-studied) problem called rating prediction from a new perspective, i.e., learning binary vector representations of the users' and items' latent representations for efficiency. In particular, the authors introduce a personalized self-masking shown in Eq.(2) and in Figure 1 (the 'AND' operation) in order to improve the previous hashing-based collaborative filtering methods without increasing the time cost much.

Empirical studies on four public datasets show the effectiveness of the proposed model, i.e., variational hashing-based collaborative filtering with self-masking (VaHSM-CF).

Some comments/suggestions:

1 The studied problem (i.e., rating prediction) is well studied in the community of recommender systems, and there are many more accurate algorithms than the basic MF algorithm used in the empirical studies. I thus suggest the authors to include more such algorithms though the focus of this paper is for efficiency (but the authors also claim the accuracy of the proposed model).

2 The authors are suggested to give a brief explanation on choosing those baseline methods in the context of other hashing-based CF methods.

3 Some details are missing, e.g., the number of latent dimensions in MF. And some presentation in the parameter setting are not clear, e.g., 'is chosen consistently across all data sets', 'was consistently chosen'. Does 'consistently' means 'exactly the same'?

Minors:
There are some typos: 'a user, u, and', 'of e.g., restaurant and shopping malls', etc.


review_title: Official Blind Review #3

Meta-review: There was a clear consensus amongst reviewers that the paper should not be accepted. This view was not changed by the rebuttal. Thus the paper is rejected. 
Decision: Reject