Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ICLR review

In this paper, the authors propose to tackle the multivariate spatial point process model with a variational inference approach. The variational inference is implemented with MLP (amortized inference). In experiments, the results show that the proposed approach outperforms VAE-based collaborative filtering on Gowalla datasets and MovieLens datasets.

My overall judgement of this paper leans to acceptation. However, by doing a quick comparison with Liang's VAE-CF paper. The proposed algorithm seems to be an extension or modification of Liang's framework. Therefore I guess the main contribution of this paper is Eq. (9). If so, I'm expecting more detailed comparison with VAE-CF in the paper (not only quantitive evaluation).

Questions:

- In Liang's paper, I see they report the evaluation results on both ML-1M and ML-20M dataset. However, in this paper, the results are reported on ML-100K and ML-1M. Why not evaluate the model on the bigger ML-20M dataset?

- In Liang's paper and also He's paper (Neural Collaborative Filtering), they report NDCG@10 on ML-1M dataset. The NCF approach reaches a NDCG@10 of 0.426, Multi-DAE (denoising version of Liang's model) reaches 0.446. You are reporting NDCG@100, which is a different measure. However, the numbers in Table 4 seem to be lower than what I expected. Can you compare with them in the same condition?
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The work describes an application of a spatial point process for solving problems with missing data. The authors introduce a novel method based on a non-parametric definition of point intensities for the multivariate case. The method incorporates VAE framework to effectively handle missing points via smooth intensity estimation and enjoys amortized inference for efficient computations and quick prediction generation. Using a sequence of mild assumptions, the authors show connection to a popular VAE-based collaborative filtering model, which turns out to be a special case of their approach.
 

This is a rigorous study providing theoretically justified evidence on the effectiveness of the proposed approach. Apart from the issues in the last part of experiments with classical collaborative filtering task (which will be detailed below), the work presents a solid research. I would therefore vote for accepting it.


The text is well structured, and all key points are clearly explained. The problem solved by the authors is well described, and the motivation for this work is convincing. The way point process theory is applied constitutes a rigorous probabilistic approach. The authors convincingly justify the need for all approximations and simplifications made in the model. One of key results making the entire model feasible is supported by the corresponding theorem proved by the authors. I haven’t carefully verified all the derivations, though.

 
My major concern is related to the last part with experiments on the Movielens data. As the authors state, “applications without explicit spatial information, we embed each event into a latent space as a vector.” “No spatial information” is exactly the case with the standard collaborative filtering task, which the authors attempt to solve. This leads to an introduction of an additional model like GNN, which is unrelated to the main approach. As GNN is involved it’s not immediately obvious that the improvement over standard VAE architecture, observed in the experiments on ML-100K and ML-1M, is due to a better point process modelling.  No evidence is provided to argue that this is not simply due to a good compression or a good data preprocessing achieved by a GNN architecture itself. Therefore, the results on a pure recommender systems part are not convincing. What would happen if GNN was trained and fed into another (simpler) algorithm? Maybe a simple KNN based algorithm would produce comparable or even better results? As indicated by the work of [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”, VAE-CF (along with several other recently proposed neural network-based methods) is inferior to even properly tuned kNN-based models. I would not be surprised, if a kNN model trained on GNN output would produce even better results than the proposed VAE-SPP.

Another related question is how incorporating GNN affects the training time? Is it comparable to that of VAE-CF or is it much worse? Computational performance is an important part in making practical decisions and should be also considered.

Furthermore, both ML datasets used for tests are too small and not very representative to make any generalized conclusions. Even on a larger ML-20M dataset an optimal SVD-based model can be trained within several minutes on a standard CPU on a laptop (according to my experiments, VAE-CF would take at least twice longer on Tesla K80). Therefore, it can hardly be considered a realistic example. In practice, there could be millions and hundreds of millions of items. The authors even mention it in the in the introduction, using it as a vehicle to motivate their approach. However, computing similarities between that many entities can be a laborious task on its own, which adds an extra layer of complexity and again is not directly related to the main approach. It can easily become a bottleneck or make further computations inefficient. More efficient similarity computations may in contrast reduce the resulting accuracy.
The issue can get even worse, because, unlike classical MF methods, there’s still no proper support for sparse operations in NN frameworks. In the VAE framework it means that, during the training, user batches will be converted into dense arrays and may become inefficient to work with in terms of memory and CPU utilization (a few non-zero entries vs. hundreds of millions of explicitly stored zeroes).
In spite of all this, I’d also suggest rephrasing “We validate these beneﬁts through extensive experiments” as it sounds a bit exaggerated (if we are considering real recommender systems applications). I agree that the proposed approach is potentially applicable in real cases for recommender systems, however, there’s still not enough evidence for this. In fact, I don’t even think that completely removing the part with ML-100K and ML-1M datasets would make the whole work any worse. Clearly stating the region of applicability of the proposed approach would be enough. Right now some statements in this section in contrast are raising concerns rather than convincing the reader. The wording should be at least changed, so that readers do not get an impression that the case with classical CF task is solved purely by the proposed VAE-SPP approach.

Other remarks to help improve the text:
1) “… points are more likely to … form clusters than the simple Poisson process …” the sentence seems to be inconsistent.
2) “The generative process of our model can be described as follow:” -> … as follows:
3) Page 4, last paragraph, line 6 – shouldn’t the upper bound for summation be N_u instead of just N?

References:
Dacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. "Are we really making much progress? A worrying analysis of recent neural recommendation approaches." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a VAE model for spatial point processes. The model generalizes the kernel density-based intensity and applies variational inference. The model is applied to synthetic datasets, a location-based social network dataset, and a recommender system dataset. 

The paper is well motivated and clearly written. I found the probabilistic modeling interesting. My major concern is that, with added complexity, the experimental results suggest that VAE-SPP is not significantly better than the existing VAE-CF on most tasks.

For the MovieLens task, it seems that the GNN is an important component in the pipeline, but no further detail is provided about it, including the network architecture. The authors might also want to provide the definitions of NDCG@k and Recall@k, at least in the appendix.

---

I appreciate the author's detailed response and updated paper. I have changed my rating.
review_title: Official Blind Review #2

Meta-review: This paper presents a novel VAE-based model for multivariate spatial point process which can realize efficient inference by amortization and handle missing points via smooth intensity estimation. Authors also provide interesting theoretical analysis to connect their method to a popular VAE-based collaborative filtering method.
Overall, all reviewers appreciate the methodological and theoretical contributions of the paper. During the reviewer discussion, one reviewer decided to update to the score to Weak Acceptance. There are still some concerns regarding experimental validation, I think the paper provides enough theoretical contribution to the community and would like to recommend acceptance. 
Decision: Accept (Poster)