Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a probabilistic tensor factorization model for streaming data. The model uses:

* neural networks to learn richer factors,
* spike-and-slab prior on NN weights
* and online moment-matching for posterior inference.

Major comments:

1. Overall, the novelty of this work is minimal. There are existing works in literature on deep factorization models e.g. for recommender systems [Xue, et al. 2017]. It should be clarified how this work stands out.

2. Authors posit that existing tensor factorizations only conduct multilinear decompositions, yet to make inference of the proposed SPIDER model tractable, they use extensive linearization relaxations. How is the model capacity to learn non-linear relationships in data preserved despite these simplifications?

3. It has been claimed that stochastic gradient methods with re-parametrization trick perform poorly for parameter inference in this setting. It would be enlightening to see this is experimentally validated, as the SGD is the most successful optimization scheme for NN methods.

4. How does the spike-and-slab prior perform compared to other regularization methods such as drop-out? 

5. The message of experiments section is not quite coherent. What is the trade-off between performance accuracy and computational complexity of the proposed framework?

review_title: Review for Streaming Probabilistic Deep Tensor Factorization

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##################################################

After author feedback:

The authors have addressed my comments, though it is impossible to evaluate what the authors promise to do in future work. My evaluation remains unchanged. 

 
##################################################

Summary:
The paper proposes a Bayesian neural network model for tensor factorization, with particular focus on streaming data. The method features spike-and-slab prior to avoid overfitting. The main computational challenge is how to handle the streaming data in the posterior inference steps. The authors propose an approximate algorithm via moment matching. The method shows promising results in simulation.

##################################################

Pros:
1. The proposed method is novel and mostly described and motivated in a clear fashion. It is an interesting addition to the current literature of Bayesian methods for streaming data.
2. The predictive performance, as shown by the authors, show significant improvements over the alternatives.

##################################################

Cons:
While I like the overall method, I have some questions about the inferential procedure. While it seems to work in the examples chosen by the authors, readers could benefit more from a clearer presentation of the methods, especially in terms of the computation and approximation accuracy trade-off.

1.  There are many (usually first order) approximation steps in the inference procedure. Can you quantify how these approximations affect the predictive performance and comment more on the validity of the Bayesian algorithm? To be more specific, I would like to see more justification or discussion in using the first order approximation of f_w(x_{i_n}) in both the binary and continuous case when calculating the normalizing constant Z_n. It is unclear how the approximation affects the estimation of Z_n after integration, and then in turn affects the posterior distribution. It also makes me wonder if the second-order Tayler expansion leads to no benefit in performance (claimed after equation 12) could be due to how f_w is approximated? The current justifications offered above equation 9 is useful in intuitively understand the considerations of the authors, yet not fully satisfactory why that is the right thing to do. In general, it is difficult for readers to see how the several approximations involved in the inference step change the posterior distribution computed versus the original target.

2. One thing that is missing is how the spike-and-slab selection indicators S are updated. Point mass spike are notoriously difficult to sample.  Is that the case here as well?

##################################################

Minor points:
1. Caption of figure 1 is cut out.
2. I can see the authors want to fit the length to 8 pages. Consider shotern the background section. Some of the notations can be introduced only when describing the proposed method.
review_title: Review of "STREAMING PROBABILISTIC DEEP TENSOR FACTORIZATION"

final_score: rating: 5: Marginally below acceptance threshold
review_content: 

Update: The authors addressed part of my concerns. For the factor estimation, the proposed method relies on first order approximations while learning the posterior of the factors; however, the approximation error does not enter into the posterior.  The approximation also raises concerns regarding the convergence of the algorithm. Overall, I think the approach is promising, but some justification of the quality of the approximation is needed. Thus, I tend to keep my rating.

##############

The authors propose a streaming approach to tensor factorization with Bayesian neural networks. The problem is to factorize a three-way tensor with Gaussian noise. The proposed approach combines a Bayesian neural network (BNN) whose output predicts the entries of the tensor and the streaming variational Bayes (SVB) for incremental posterior updates. In addition, a spike-and-slab prior is placed on the BNN weights to encourage sparsity as well as prevent overfitting.  The authors performed empirical studies on four real datasets, DBLP, Anime, ACC, and MovieLen1M, where improved prediction accuracy of the tensor entries are reported.

From the empirical evaluation, the objective of the proposed approach is to ensure that the output of the BNN matches the tensor entries. However, this does not necessarily guarantee the correctness of the recovered factor matrices (input to the BNN). I feel that the problem setting of the paper is somehow different from the standard CP factorization setting where the (unique) factors are of primary interest. It would be good to add some discussions about the correctness and/or uniqueness of the uncovered factors.

Strengths:
- The idea of introducing a spike-and-slab prior in the factorization is interesting.
- Using SVB for online posterior updates is computationally practical.

Weaknesses:
- The presentation could be improved — it is not very clear how the factor matrices are estimated.
- Factorization using the proposed method is not unique (up to rescaling and permutation of the factor matrices). 

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################

After author feedback:
Thanks for the detailed feedback from the authors. Most of my concerns have been addressed and I will keep my scores unchanged. Please add the additional information in the feedback to the final version.

##########################

Summary:

This paper proposes a streaming probabilistic deep tensor factorization model, called SPIDER, to solve the tensor factorization problem under streaming setting with deep factorization parameterized by Bayesian neural networks. To encourage sparsity and prevent overfitting, a spike-and-slab prior is used for the weights of the neural network layers. Learning algorithms are also derived based on the expectation propagation framework where the posterior of the network output is approximated using multivariate Delta's method and moment matching. Experimental results show improvement against existing tensor factorization models.


##########################

Pros:
 
1. Although there has been some work on streaming or online tensor factorization, the (Bayesian) deep factorization has never been addressed in the streaming setting, which is also quite an important problem to study.
2. The incorporation of spike-and-slab prior for sparsity is good for many practical applications.
3. Overall, the paper is organized and easy to follow.
4. The experiment section is well structured and multiple real-world large datasets are used.


##########################

Cons/Questions: 

1. Does the algorithm converge? It would be great to see any theoretical or empirical analysis on convergence.
2. Two important baselines NeuralCP and CoSTCo are missing. As they are very recent and state-of-the-art models in neural tensor factorization, it would be great to see comparisons against them.
3. It is unclear from the experimental section that how many parameters does each model have. Can the authors provide a table or figure about the number of parameters of each model?
4. One of the most important properties of CP and Tucker decomposition is their interpretability. I am wondering if any interpretation of the factors can be carried out for the proposed model.
5. The caption of Fig. 1 seems unfinished or blocked by Fig 2.
review_title: Reasonable model and good experimental results

Meta-review: The paper proposes a Bayesian neural network model for tensor factorization, with particular focus on streaming data. The key contribution is the streaming posterior inference of the deep TF models.  The combinations of online tensor factorization, Bayesian NN with sparsity priors, posterior inference is new and interesting.  However, there are many approximation steps, and the quality of the approximation and convergence of algorithm are not well justified.   
Decision: Reject