Streaming Probabilistic Deep Tensor Factorization
Authors: shikai fang
Zheng Wang
Zhimeng pan
Ji Liu
Shandian Zhe
Venue:  (2021)
Keywords: Probabilistic Methods
online learing
tensor factorization

Abstract: Despite the success of existing tensor factorization methods, most of them conduct a multilinear decomposition, and rarely exploit powerful modeling frameworks, like deep neural networks, to capture a variety of complicated interactions in data. More important, for highly expressive, deep factorization, we lack an effective approach to handle streaming data, which are ubiquitous in real-world applications. To address these issues, we propose SPIDER, a Streaming ProbabilistIc Deep tEnsoR factorization method. We first use Bayesian neural networks (NNs) to construct a deep tensor factorization model. We assign a spike-and-slab prior over the NN weights to encourage sparsity and prevent overfitting. We then use Taylor expansions and moment matching to approximate the posterior of the NN output and calculate the running model evidence, based on which we develop an efficient streaming posterior inference algorithm in the assumed-density-filtering and expectation propagation framework. Our algorithm provides responsive incremental updates for the posterior of the latent factors and NN weights upon receiving new tensor entries, and meanwhile select and inhibit redundant/useless weights. We show the advantages of our approach in four real-world applications.

Under review as a conference paper at ICLR 2021
STREAMING PROBABILISTIC DEEP TENSOR FACTOR- IZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the success of existing tensor factorization methods, most of them conduct a multilinear decomposition, and rarely exploit powerful modeling frameworks, like deep neural networks, to capture a variety of complicated interactions in data. More important, for highly expressive, deep factorization, we lack an effective approach to handle streaming data, which ubiquitous in real-world applications. To address these issues, we propose SPIDER, a Streaming Probabilistlc Deep (EnsoR factorization method. We first use Bayesian neural networks (NNs) construct a deep tensor factorization model. We assign a spike-and-slab prior over each NN weight to encourage sparsity and to prevent overfitting. We then use mul- tivariate Delta's 's method and moment matching to approximate the posterior of the NN output and calculate the running model evidence, based on which we develop efficient streaming posterior inference algorithm in the assumed-deensty-illee and expectation propagation framework. Our algorithm provides responsive cremental updates for the posterior of the latent factors and NN weights upon receiving new tensor entries, and meanwhile select and inhibit redundantusseless weights. We show the advantages of our approach in four real-world applications.
Introduction
Tensor factorization is fundamental tool for multiway data analysis. While many tensor factorization
methods have been developed (Tucker 1966 Harshman, 1970; Chu & Ghahramani 2009 Kang et al. 2012 Choi & Vishwanathan] (014). most of them conduct mutilinear decomposition and are incapable of capturing complex, nonlinear relationships in data. Deep neural networks (NNs) are class of very flexible and powerful modeling framework, known be able to estimate all kinds of complicated (e.g., highly nonlinear) mappings. The most recent work (Liu et al. 2018 2019) have attempted to incorporate NNs into tensor factorization and shown promotion of the performance, in spite of the risk of overfitting the tensor data that are typically sparse.
Nonetheless, critical bottleneck for NN based factorization is the lack of effective approaches for streaming data. In practice, many applications produce huge volumes of data at a fast pace (Du al. 2018). It is extremely costly to run the factorization from scratch every time when a new set of entries. Some privacy- -emanddin applications (e.g., SnapChat) even forbid us from revisiting the previously seen data. Hence, given new data, we need an effective way to update the model incrementally and promptly.
receive
general and popular approach is streaming variational Bayes (SVB) (Broderick et al. 2013), which integrates the current posterior with the new data, and then estimates a variational approximation as the updated posterior. Although SVB has been successfully used to develop the sfate-of -ht multilinear streaming factorization (Du et al. 2018). it does not perform well for (deep) NN based factorization. Due to the nested linear and nonlinear coupling of the latent embeddings and NN weights, the variational model evidence lower bound (ELBO) that SVB maximizes is analytically intractable and we have to seck for stochastic optimization, which is unstable and hard to diagnose the convergence. Consequently. the posterior updates often unreliable and inferior, and in turn hurt the subsequent updates, leading to poor model estimations finally.
To address these issues, we propose SPIDER, streaming probabilistic deep tensor factorization method that not only exploits ÑN's expressive power to capture intricate relationships. but also provides efficient, high-quality posterior updates for streaming data. Specifically, we first use Bayesian

---

Under review as a conference paper at ICLR 2021
neural networks to build deep tensor factorization model, where the input is the concatenation of the associated factors in each tensor entry and the NN output predicts the entry value. To reduce the risk of overfitting, we place spike-and-slab prior over each NN weight to encourage sparsity. For streaming inference, we use multivariate Delta's 's method (Bickel & Doksum 2015) that employs a first-order Taylor expansion of the NN output analytically compute its moments, and match the moments to obtain the its current posterior and the running model evidence. We then use back-propagalie to calculate the gradient of the log evidence, with which we match the moments and update the posterior of the embeddings and NN weights in the -eeuumddddesitt---ee filltti-ffitngggggg (Boyen & Koller 1998) framework. Finally, after processing all the newly received entries, we update the spike-and-slan prior approximation with expectation propagation (Minka 2001a) to select and inhibit redundant/useless weights. this way, the incremental posterior updates are deterministic, reliable and efficient.
For evaluation, we examined SPIDER on four real-world large-scale applications, including both binary and continuous tensors. We compared with the state-of-the- streaming tensor factorization algorithm (Du et al. 2018) based on multilinear form, and streaming nonlinear factorization methods implemented with SVB. In both running and final predictive performance, our method consistently outperforms the competing approaches, mostly by a large margin. The running accuracy of SPIDER is also much more stable and smooth than the SVB based methods.
2 Background
Tensor Factorization. We denote a K K-mode tensor by y E Rdix...xxx where mode k includes dy nodes. We index each entry by tuple i (ik). which stands for the interaction of the corresponding K nodes. The value of entry i is denoted by y/. To factorize the tensor, we represent all the nodes by K latent embedding matrices U = (U'.... .U.., UK}, J. where each U = -uf.....,uu u 1 is of size dg X The and each u², is the embedding vector of node j in mode k. The goal is to use 2 to recover the observed entries in y. To this end, the classical Tucker factorization (Tucker, 1966) assumes y = W x U X2 ... Xx UK where W E X is parametric tenor and X the mode-k tensor matrix multiplication (Kolda 2006), which resembles the matrix-matrix multiplication. If we set all r and W to be diagonal, Tucker factorization becomes CANDECCMP//ARAFAC (CP) factorization (Harshman) 1970). The clement-wise form is = En j-1 II _ = (u,, o o... aar 11, where o is the Hadamard (element-wise product and 1 the vector filled with ones. We can estimate the embeddings li by minimizing a loss function, e.g., the mean squared error in recovering the observed elements in y.
Streaming Model Estimation. A general and popular framework for incremental model estimation streaming variational (aye)(SVB) (Broderick et al.) 2013), which is grounded on the incremental version of Bayes' rule,
pl oiu U Uneww) & p Pou))) (
where O are the latent random variables in the probabilistic model we interested in, Dola all the data that have been seen so far, and Dnew the incoming data. SVB approximates the current posterior
p( ( (aaa) with variational posterior yenr When the new data arrives, SVB integrates gear (0) with the likelihood of the new data to obtain an unnormalized, blending distribution,
(0)) Sear ((Dee (Dnww (
(2) Doww). To
which can be viewed as approximately proportional to the joint distribution p(0, Doss U D
conduct the incremental update, SVB uses p(0) to construct a variational BLBO WWinwwiihht et al. 2008), O)) = EEg [log (p(0)/((0)), and maximizes the ELBO to obtain the updated posterior, q" arrgmax, L(q). This equivalent to minimizing the Kullback-Leible (KL) divergence between q and the normalized p(0). We then set gour = q" and prepare the update for the next batch of new data. At the beginning (when we do not receive any data), we set qer = p(0), the original prior in the model. For efficiency and convenience, factorized variational posterior (0) = II, ((,)) is usually adopted to fulfill cyclic, flosed-form updates. For example, the -fate-of-the-- arr streaming tensor factorization, POST (Du et al.) 20018), uses the CP form to build a Bayesian model, and applies SVB to update the posterior of the embeddings incrementally when receiving new tensor entries.
3 Bayesian Deep Tensor Factorization
Despite the elegance and convenience of the popular Tucker and CP factorization, their multilinear form can severely limit the capability of estimating complicated, highly nonlinear/nonssationayy
2

---

Under review as a conference paper at ICLR 2021
relationships hidden in data. While numerous other methods have also been proposed. e.g.. (Chu & Ghahramami] 2009: Kang et al. 2012] Choi & Vishwanathan 2014). most of them are still inherently based the CP or Tucker form. Enlightened by the expressive power of (deep) neural networks (Goodfellow et al. 2016), we propose a Bayesian deep tensor factorization model to overcome the limitation of traditional methods and flexibly estimate all kinds of complex relationships.
Specifically, for each tensor entry i, we construct an input X; by concatenating all the embedding vectors associated with i, namely, x; [(u1,)² ..... (u (xx) 'JT. We assume that there is unknown
mapping between the input embeddings x; and the value of entry i, ʃ R RE - R, which reflects the complex interactions/relationsippps between the tensor nodes in entry Note that CP factorization uses multilinear mapping. We use M-layer neural network (NN) to model the mapping f. which are parameterized by M weight matrices W = {W [W..... 3M Each W.. is Vm x (Vm-1 1) where Vn and V are the widths for layer m and
1, respectively; Vi = Σk Ph is the input dimension and VM = 1. We denote the output in each hidden layer m by hm (1 S m I -1) and define hy = x; We compute each h. = o (W mmmmmmm: 1; /V1m/Vmm1 + append a constant
where (.) is nonlinear activation function, e.g., ReLU and tanh. Note that
feature to introduce the bias terms in the linear transformation. namely the last column in each Wm. For the last layer, we compute the output by fw (x;) W (1 the output, sample the observed entry value 1/1 via a noisy model. For continuous data, use a Gaussian noise model, ((vi = (xi).
(;;1] 1]/ /Va/VVVs-11 + 1. Given
where r is the inverse noise variance. We further assign r a Gamma prior, p(T) = Gamma bo). For binary data, we use the Probit model, p(yi(4) + ((2y) (xi)(x))) where (-) the cumulative density function (CDF) of the standard normal distribution.
Despite their great flexibility, NNs take the risk of overfitting. The larger a network, Lee, with more weight parameters, the easier the network overfits the data. In order to prevent overfitting, we assign a spike-and-slab prior (Ishwaran et al., 20055 Titsias & Lazaro-Gredilla 201T) over each NN weight to sparsify and condense the network. Specifically, for each weight Wmit binary selection indicator Smjt from p(5mis (Po) Bern(smjal Po) Po weight is then sampled from
W mljt, we first sample
smit (1 po) 5mat, The
pl p(umjjt Smnjt) 8mjta (wmje 0,, a + (1 Smjt) (WWmjr)
(3)
where 5(-) is the Dirac-delta function. Hence, the selection indicator 8mjt determines the type of prior over Unjtt 8mjjt is 1, meaning the weight is useful and active, we assign flat Gaussian prior with variance o (slab component); if otherwise Smjt is namely the weight is useless and should be deactivated, we assign spike prior concentrating on 0 (spike component).
Finally, we place a standard normal prior over the embeddings 24, Given the set of observed tensor entries D = (M......, Viy J. the joint probability of our model for continuous data is
M Vin
Vn 1+1
p(u, W,S,)
II
II
Bern (erje
oB) (1 (mmj)6((mms
m=1j=1
O, ToCnaaaa Gaaaaaaaaao bo) II N W(., (4)
K
II
where S {8mst), and for binary data is
Vm Vm 1+1
puu, W,S) II Bern(5nglaa)(mms) 0, + (1 smme))
m=1
[=1
du
N ((22n. 1)fw(x).))
(5)
4 Streaming Posterior Inference
We now present our streaming model estimation algorithm. In general, the observed tensor entries assumed to be streamed in a sequence of small batches, [B1, .... Different batches do not have to include the same number of entries. Upon receiving each batch B, we aim to update the posterior distribution of the embeddings U, the inverse noise variance (for continuous data),
3

---

Under review as a conference paper at ICLR 2021
the selection indicators S and the neural network weights W, without re-accessing the previous batches (By}zet While we can apply SVB, the variational ELBO that integrates the current posterior and the new entry batch will be analytically intractable. Take binary tensors an example. Given a new entry batch Bi, the EBLO constructed based on the blending distribution (see (2) is L KT. ( S, [gauur (M, S, W)) EngB, Byy [og ((2g,.. nested, nonlinear coupling of the embeddings (in each x1,) and NN weights W in calculating fw(x,,). ). the expectation terms in L are intractable, without any closed form. Obviously, the same conclusion applies to the continuous data. Therefore, to maximize L so as to obtain the updated posterior, we have to use stochastic gradient descent (SGD), typically with the re- re-parameterization trick (Kingma & Welling 2013). However, without the explicit form of L, it is hard to diagnosis the convergence of SGD - may stop at a place far from the (local) optimums. Note that we cannot use hold-out data or cross-validation to monitor/control the training because we cannot store or revisit data. The inferior posterior estimation in one batch can in turn influence the posterior updates in the subsequent batches, and finally result in very poor model estimation.
()xx(xxt,)) Due to the
4.1 Online Moment Matching for Posterior Update
(ADF) framework (Boyen & Koller 1998), which can be viewed as an online version of expectation propagation (EP) (Minka 2001a), a general approximate Bayesian inference algorithm. ADF also based on the incremental version of Bayes' rule (see (T). It uses a distribution in the exponential family (Wainwright et al. 2008) to approximate the current posterior. When the new data arrive, instead of maximizing a variational ELBO, ADF projects the (unnormalized) blending distribution 2 to the exponential family to obtain the updated posterior. The projection is done by moment matching, which essentially is minimize (T((p(0) /Z]lg(()) where Z is the normalization constant. For illustration, suppose we choose (0) to be fully factorized Gaussian distribution, (()) = II, (02,) = II, (1( vz). To update each q(f)), we compute the first and second moments of f w.r.t p(0), and match a Gaussian distribution with the same moments, namely, 14 = (0,) and ny Varp(0,) (0)) - (O,)
To address these problems, we exploit the assumed -eeeeedddnnntt--tteeeeee
For our model, we use fully factorized distribution in the exponential family to approximate the current posterior. When a new batch of data B, are received, we sequentially process each observed entry, and perform moment matching to update the posterior of the NN weights W and associated embeddings. Specifically, let us start with the binary data. We approximate the posterior with
M V.n V +1
K de
Teur (W,,.S) = IIII II ==
Bern(SmjalPmmgh W (Wmjt (Wijt//mmtt (mjtt Umjtt
(u Wage, vxze). j=jji
IIIIII
1=1
Given each entry in in the new batch, we construct the blending distribution, P(W,US)) x yeur (w, 2,,) ((29.. - -)))w(x),)) To obtain its moments, we consider the normalizer, iee. the model evidence under the blending distribution.
Zn = Jcur W 4, S) ((2)s. - (x)
(6)
Under the Gaussian form, according to Minka 2001b), posterior of each NN weight Wijjt and each embedding element associated with i,
compute the moments and update the
{ {u
l by
p = Olog Zn 0u
2 log Zn Zn)² 2 0²[( -'= v² =-=
a log Zm De
where μ and o are the current posterior mean and variance of the corresponding weight or embedding element. Note that since the likelihood does not include the binary selection indicators S, their moments are the same as those under feur and we do not need to update their posterior.
However, a critical issue is that due to the nonlinear coupling of the UI and W in computing the NN output (w,, ). the exact normalizer is analytically intractable. To overcome this issue, we consider approximating the current posterior of (w,),,, first. We use multivariate Delta's 's method (Oehlert 1992. Bickel & Doksum 2015) that expands the NN output at the mean of W and U,
(x R. fr[w] (E(x,,))
g. (n.. E(n..)
where the expectation is under (eur()). nn = vee(W U x;,). gn. = V Vfw (x;,. In... Note that Xi, is the concatenation of the embeddings associated with i,. The rationale of the approximation
4

---

Under review as a conference paper at ICLR 2021
(8) is that the NN output is highly nonlinear and nonconvex U and W. Hence, the scale of the output change rate (i.e., gradient) can be much larger than the scale of the posterior variances of W and 21 which are (much) smaller than prior variance 1 (see 4 and ( Therefore, we can ignore the second-order term and just use the first-order Taylor approximation. We have also tried the second-order expansion, which, however, is unstable and does not improve the performance.
Based on (8). we can calculate the first and second moments of fw((x,),
On = gun (x1.) Rs /E[W] (E(x,,.), 8n. = Var qua (x1,) R g,, diaag(m)))
(9)
where each [yll5- = Varg... (17,)))- Due to the fully factorized posterior form, we have cov(n.) diag (.,). Note that all the information in the Gaussian posterior (i.e., mean and variance) of W and UI have been integrated to approximate the moments and posterior of the NN output. Now we use moment matching to approximate the current (marginal) posterior of the NN output by qeur (fw(x1.)) = -N(f((x,,
3a). Then we compute the running model evidence (6) by
On (w.... 1 (22 - (x,,(xx,,,)) 1)] = E (/ .x. 1) ()(00(22))))) ((2th.. - (xt,)()))))
NN Bx) ) (2yr.. -oo.on--.f..ffff 1)f.)df. = o6 (2 - 1)ag
(10)
Bn
where we redefine fo = (xt.).) for simplicity. With the nice analytical form, we can immediately apply 77) to update the posterior for W and the associated embeddings in i,. In light of the NN structure, the gradient can be efficiently calculated via back-propagation, which can be automatically done by many deep learning libraries.
For continuous data, we introduce a Gamma posterior for the inverse noise variance, (eur(T) = Gamma(r]a, (r|aab, b). in addition to the fully factorized posterior for W, U and as in the binary case. After we use (8) and (9) to obtain the posterior of the NN output, we derive the running model evidence by
2, yeu (w.u.s (N (wr. [)] (agu(S)) (W,
~ E (n) / ([Jolan, Wn, 1S, T
-o,r= ')asa] = F (Win,, lan, 3n + +)
(11)
Next, we use the first-order Taylor expansion again, at the mean of to approximate the Gaus- sian term inside the expectation, A N Ja 3r + 1) v N la 3n +
( + (r
Eqg. (T),,, lnn, 3n + / -+ Taking expectation over the Taylor expansion gives
2n RJ N( la, 3n + Bg.. = 3n b/a).
(12)
We now can use to update the posterior of W and the embeddings associated with the entry. While we can also use more accurate approximations, e.g., the second-order Taylor expansion quadrature, we found empirically our method achieves almost the same performance.
To update (T), we consider the blending distribution only in terms of the NN output fo and so we have p(fo:T) x Gear ( (f) gear( (T (T) (yn,,, (o,. - = (f an, 3n, Gammaa(Ja,, (),((,, Io., Then we follow (Wang & Zhe (019) to first derive the conditional moments and then approximate the expectation of the conditional moments to obtain the moments. The details are given in the supplementary material. The updated posterior is given by (T) Gamma(7|a" bb"), where ] ((m. - on)² + Bn).
a"
a a and b* = b
4.2 Prior Approximation Refinement
Ideally, at the beginning (i.e, when we have not received any data), we should set Jeur to the prior of the model (see 44) and (5)) This is feasible for the embeddings 41 selection indicators S and the inverse noise variance t (for continuous data only), because their Gaussian, Bernoulli and Gamma priors are all members of the exponential family. However, the spike-and-slab prior for each NN weight Wnje (see (3) a mixture prior and does not belong to the exponential family. Hence, we introduce an approximation term,
p( (0mjelsmph X A(Wmjtt Smjt) =
(C(mssalll()))))))) hww.ee/eeennnnn. Wmngas unnje)
(13)
where X means "approximately proportional to" and c(x) = 1/(1 + exp(-x)). At the beginning, we initialize ofagt of and pengee be random number generated from a standard Gaussian distribution
5

---

Under review as a conference paper at ICLR 2021
truncated in i-ou. we initialize Pmjt = 0. Obviously, this is a very rough approximation. If we only execute the standard ADF to continuously integrate new entries to update the posterior (see 77), the prior approximation term will remain the same and never be changed. However, the spike-and-slab prior is critical to sparsify and condense the network, and an inferior approximation will make it noneffective at all. To address this issue, after we process all the entries in the incoming batch, we use EP to update/improve the prior approximation term (13), with which to further update the posterior of the NN weights. In this way, as we process more and more batches of the observed
tensor entries, the prior approximation becomes more and more accurate, and thereby can effectively inhibit/deactivate the redundant or useless weights on the fly. The details of the updates are provided
in the supplementary material. Finally, we summarize our streaming inference in Algorithm in the supplementary material.
4.3 Algorithm Complexity
The time complexity of our streaming inference O(NV + E ders) where V is the total number of weights in the NN. Therefore, the computational cost is proportional to N, the size of the streaming batch. The space complexity is O(V + k= R, including the storage of the posterior for the embeddings U, NN weights W and selection indicators S, and the approximation term for the spike-and-slab prior.
5 Related Work
Classical CP (Harshman 1970) and Tucker (Tucker 1966)
factorization are multilinear and there-
fore are incapable of estimating complex, nonlinear relationships in data. While numerous other
approaches have been proposed (Shashua Hazan 2005 Chu & Ghahramani 2009 Sutskever
et al. 2009; Hoff 2011 Kang et al., [012] Yang & Dunson 2013 Choi & Vishwanathan 2014
Hu et al. 2015 Rai et al. (015), most of them are still based on the CP or Tucker form. To over- come these issues, recently, several Bayesian nonparametric tensor factorization models (Xu et al., 2012, Zhe et al. 2015| 2016) were proposed to estimate the nonlinear relationships with Gaussian processes (Rasmussen & Williams, 2006). The most recent work, NeurICP (Liu et al.) (0118) and CoSTCo (Liu et al. 2019) have shown the advantage of NNs in tensor factorization. CoSTCo also concatenates the embeddings associated with each entry construct the input and use the NN output to predict the entry value; but to alleviate overfitting, CoSTCo introduces two convolutional layers to extract local features and then feed them into dense layers. By contrast, with the spike-and-slab prior and Bayesian inference, we found that our model can also effectively prevent overfitting, without the need for extra convolutional layers. NeuralCP uses two NNs, one to predict the entry value, the other the (log) noise variance. Hence, NeuralCP only applies to continuous data. Our model can be used for both continuous and binary data. Finally, both CoSTCo and NerualCP are trained with stochastic optimization, need to pass the data many times (epochs), and hence cannot handle streaming data.
Expectation propagation (Minka 200 is an approximate Bayesian inference algorithm that general- (ADF) (Boyen & Koller 1998) and (loopy) belief propagation (Murphy) term to approximate the prior and likelihood of each data point, and cyclically updates each approximation term via moment matching. ADF can be considered as applying EP in a model including only one data point. Because ADF only maintains the holistic posterior, without the need for keeping individual approximation terms, it is very appropriate for streaming learning. EP can meet a practical barrier when the moment matching is intractable. To address this problem. Wang & Zhe (2019) proposed conditional EP that uses conditional moment matching, quadrature and Taylor approximations to provide a high-quality, analytical solution. Based on EP and ADF; Herbnaneez-Lobtoo & Adams (2015) proposed probabilistic propapropag,tion, batch inference algorithm for Bayesian neural networks. PBP conducts ADF to pass the dataset many times and re-update the prior approximation after each pass. A key difference from our work is that PBP conducts a forward, layer by layer moment matching, to approximate the posterior of each hidden neuron the network, until reaches the output. The computation limited to fully connected, fred-forward networks and ReLU activation function. By contrast, our method computes the moments of the NN output via Delta's method (i.e., Taylor expansions) and does not need to approximate the posterior of the hidden neurons. Therefore, our method is free to use any NN architecture and activation function. Note that murti-variate Delta's 's method was also used in Laplace's 's approximation (MacKay) 1992) and nonjconju variational inference Wang & Bleil (2013). Furthermore, we employ spike-and priors over the NN weights to control the complexity of the model and to prevent overfitting in the streaming inference.
izes assumed-density-flleee
et al. 1999). EP employs an exponential-famin

---

Under review as a conference paper at ICLR 2021
6 Experiment
6.1 Predictive Performance
Datasets. We examined SPIDER on four real-world, large-scale datasets. (1) DBLP (Du et al. 2018
a binary tensor about bibliography relationships (author conference, keyword), of size 10, 000 x 200 10, 000, including 0.001% nonzero entries. (2) Anime(https:/ /wwww//wwwwww.. http://ss//www..kggll.l./ com/
a two-mode tensor depicting binary (user, anime) preferences. The tensor contains 3,0, 160 observed entries, of size 25, 838 x 460. (3) ACC (Du et al. 2018), continuous tensor representing the three-way interactions (user, action, file), of size 3, 000 x 150 x 30, 000, including 0.9% nonzero entries. (4) MovieLen M /ttp::://grrup20
CooperUni runinnnnnnnnnnn an recommendat ionatiios--oe cons-databasse
org/ datasets htt.//moilelleee size 6, 040 x 3, 706, consisting of (user, movie) ratings. We have 1, 000, 209 observed entries.
two-mode continuous tensor of
Competing methods. We compared with the following basclines. (1) POST (Du et al. 2018). the -fate-of-the-arr probabilistic streaming tensor decomposition algorithm based on the CP model. It uses streaming variational Bayes (SVB) (Broderick et al., 2013) perform mean-field posterior updates upon receiving new entries. (2) SVB-DTF, SVB based deep tensor factorization.
(3) SVB-GPTE. the streaming version of the Gaussian process(GP) based nonlinear tensor factorization (Zhe et al. 2016). implemented with SVB. Note that similar to NNs, the ELBO in SVB for GP factorization is intractable and we used stochastic optimization. (4) SS-GPTF, the streaming GP factorization implemented with the recent streaming sparse GP approximations (Bui et al. 2017). It uses SGD to optimize another intractable ELBO. (5) CP-WOPT (Acar et al.] 1a), a scalable static CP factorization algorithm implemented with gradient-based optimization.
Parameter Settings. We implemented our method, SPIDER with Theano, and SVB-DTF, SVB/SS- GPTF with TensorFlow. For POST, we used the original MATLAB implementation (https) //github. cechub.coooa// y srisshaiinn wwwaaddu/PSSSS For SVB/SS-GPTE, we set the number of pseudo inputs to 128 in their sparse approximations. We used Adam (Kingma & Ba 2014) for the stochastic optimization in SVB-DTF and SVB/SS-GPTE, where we set the number of epochs to 100 in process- ing each streaming batch and tuned the learning rate from {10 5, 5 x 10 10 10-4. 4.3 x 10 10-4,5 X 10-1, 10-3,13000 3 x 10-3, 5 x 10-3 10-2). For SPIDER and SVB-DTF, We used a 3-layer NN, with 50 nodes in each hidden layer. We tested ReLU and tanh activations.
We first evaluated the prediction accuracy after all the (accessible) entries are processed. To do so, we sequentially fed the training entries into every method, each time small batch. We then evaluated the predictive performance on the test entries. We examined the nott-mea--suuardd-ero (RMSE) and area under ROC curves (AUC) for continuous and binary data, respectively. We ran the static factorization algorithm CP-WOPT on the entire training set. On DBLP and ACC, we used the same split of the training and test entries in (Du et al. 2018), including 320K and 1M training entries for DBLP and ACC respectively, and 100K test entries for both. On Anime and Movielen we randomly split the observed entries into 90% for training and 10% for test. For both datasets, the number of training entries is around one million. For each streaming factorization approach, we randomly shuffled the training entries and then partitioned them into a stream of batches. On each dataset, we repeated the test for 5 times and calculated the average of R.MSSS/AUCs and their standard deviations. For CP-WOPT, we used different random initializations in each test.
We conducted two groups of evaluations. In the first group, we fixed the batch size to 256 and examined the predictive performance with different ranks (or dimensions) of the embedding vectors, (3, 5, 8,8 10). In the second group, we fixed the rank to 8, and examined how the prediction accuracy varies with the size of the streaming batches, {25 27, 28, 22).. The results are reported in Fig. As we can see, in both settings, our method SPIDER (with both lanh and ReLU) consistently outperforms all the competing approaches in all the cases and mostly by large margin. First, SPIDER significantly improves upon POST and CP-WOPT the streaming and static multilinear factorization, confirming the advantages of the deep tensor factorization. It worth noting that CP- WOPT performed much worse than POST on ACC and MovieLen/M, M, which might be due to the poor local optimums CP-WOPT converged to. Second, SVB/SS-GPTF are generally far worse than our method, and in many cases even inferior to POST (see Fig. [p. c, and f). SVB-DTF are even worse. Only on Fig. Dr. the performance of SVB-DTF is comparable to or better than CP-WOPT, and in all the other cases, SVB-DTF is much worse than all the other methods and we did not show its curve in the figure (similar for CP- CP-WOPT in Fig. [g). Those results might be due to the inferior/unreliable stochastic posterior updates. Lastly, although both SPIDER-Recd and SPIDER- -tah outperform all the baselines, SPIDER-Rei Rel.U is overall better than SPIDER-tanh (especially Fig. Ig)
7

---

Under review as a conference paper at ICLR 2021
SPIDER-ReLU SPIDER-aanh
SVB-DTF-ReLU
POST SS-GPTF
SVB-GPTF CP-WOPT
SVB-DTF-tanh
0.8
0.9
AUC
0.85
AUC
0.7
RMSE
0.55
RMSE
0.8
0.45
0.95
0.75
0.6
0.35
0.9
3
10
5
8
10
3
5
10
3
10
Rank
Rank
Rank
Rank
(a) DBLP
(b) Anime
(c) ACC
(d) MovieLenIM
0.8
0.9
AUC
RMSE
0.4
RMSE
0.85
0.7
0.38
0.95
0.8
0.35
0.75
0.6
0.9
2627 2
29
2
2
2
2
2
29
28
2
2
Batch Size
Batch Size
Batch Size
Batch Size
(e) DBLP
(f) Anime
(g) ACC
(h) MovieLenIM
Figure I: Predictive performance with different ranks (top row) and streaming batch sizes (bottom row). In the top row, the streaming bath size is fixed to 256; in the bottom row, the rank is fixed to 8. The results
0.8
0.9
AUC
0.7
RMSE
0.5
RMSE
0.8
0.95
0.6
0.4
0.7
0.5
0.34
0.9
500
1000
1000 2000
0
2000
4000
1500
3000
Number of Batches
Number of Batches
Number of Batches
Number of Batches
(a) DBLP (r= 3)
(b) Anime (r=3) 3)
(c) ACC (r= 3)
(d) MovieLenIM (r= 3)
0.8
0.9
0.5
AUC
0.7
RMSE
AMSE
0.8
0.95
0.6
0.4
0.7
0.5
0.34
0.9
500
1000
1000 2000
2000
4000
1500
3000
Number of Batches
Number of Batches
Number of Batches
Number of Batches
(e) DBLP D
(f) Anime (r= 8)
(g) ACC (r= 8)
(h) MovieLenIM (r= 8)
Figure 2: Running prediction accuracy along with the number of processed streaming batches. The batch size
was fixed to 256.
6.2 Prediction On the Fly
Next, we evaluated the dynamic performance. We randomly generated stream of training batches from each dataset, upon which we ran each algorithm and examined the prediction accuracy after processing each batch. We set the batch size to 256 and tested with rank r and
The running RMSH/AUC of each method are reported in Fig. [ Note that some curves are missing or partly missing because the performance the corresponding methods are much worse than all the other ones. In general, nearly all the methods improved the prediction accuracy with more and more batches, showing increasingly better embedding estimations. However, SPIDER always obtained the best AUC/RMSE on the fly. except at the beginning stage on Anime and MovieLen/ (r = 8). In addition, the trends of SPIDER and POST are much smoother than that of SVB-DTF and SVB/SS-GPTF, which might again because the stochastic updates in the latter methods are unstable and unreliable. Note that Fig. pp. SVB-DTF has running AUC steadily around 0.5, implying that SVB actually failed to effectively update the posterior. Finally, in the supplementary material, provide the running time, and showcase the sparse posteriors of the NN weights learned by SPIDER.
7 Conclusion
We have presented SPIDER, streaming probabilistic deep tensor factorization approach, which can effectively leverage neural networks to capture complicated relationships for streaming factorization. Experiments real-world applications have demonstrated the advantage of SPIDER.
8

---

Under review as a conference paper at ICLR 2021
References
E. Acar, D. M. Dunlavy, T. G. Kolda, and M. Morup. Scalable tensor factorizations for incomplete data. Chemometrics and Intelligent Laboratory Systems, 106(1):41-56, 201 la.
Evrim Acar, Daniel Dunlavy, Tamara Kolda, and Morten Morup. Scalable tensor factorizations for incomplete data. Chemometries and Intelligent Laboratory Systems, 106(1):41-56, 201 lb.
Peter Bickel and Kjell A Doksum. Mathematical statistics: basic ideas and selected topics, volume volume 117. CRC Press, 2015.
Xavier Boyen and Daphne Koller. Tractable inference for complex stochastic processes. In Proceedings of the Fourteenth conference on Uncertainty artificial intelligence, pp. 33-42, 1998.
Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia Wilson, and Michael I Jordan. Stream- ing variational bayes. In Advances in neural information processing systems, pp. 1727-1735, 2013.
Thang Bui, Cuong Nguyen, and Richard E Turner. Streaming sparse gaussian process approxima- tions. In Advances in Neural Information Processing Systems, pp. 3299-3307, 2017.
Joon Hee Choi and S Vishwanathan. Dfacto: Distributed factorization of tensors. In Advances in Neural Information Processing Systems, pp. 1304, 2014.
Wei Chu and Zoubin Ghahramani. Probabilistic models for incomplete multi mulei-dimensional arrays. AISTATS, 2009.
Yishuai Du, Yimin Zheng, Kuang-chih Lee, and Shandian Zhe. Probabilistic streaming tensor decomposition. In 2018 IEEE International Conference Data Mining (ICDM), pp. 99-108.
IEEE, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: 7/www. //ww.deeplearinngooo.. org
R. A. Harshman. Foundations of the PARAFAC procedure: Model and conditions for cexplanatoy"mmlliinodee multi-mode factor analysis. UCLA Working Papers in Phonetics, 16:1-84, 1970.
an" "eeeplannatrr"
José Miguel Hemández--Lobat and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Learning, pp. 1861-1869, 2015.
P.D. Hoff. Hierarchical multilinear models for multiway data. Computational Statistics & Data Analysis, 55:3, 2011. ISSN 0167-9473. URL http:tttp//// stat. wttp://ww.tttt......an.tgt..ttot
edu/hoff/ http:/oof/Cooo//haee
2011 cada
Changwei Hu, Piyush Rai, and Lawrence Carin. Zero-truncated poisson tensor factorization for massive binary tensors. In UAI, 2015.
Hemant Ishwaran, Sunil Rao, et Spike and slab variable selection: frequentist and bayesian strategies. The Annals of Statistics, 33(2):730-773, 2005.
U Kang. Evangelos Papalexakis, Abhay Harpale, and Christos Faloutsos. Gigatensor: scaling tensor analysis up by 100 times-algorithms and discoveries. In Proceedings of the 18th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 316 316-324. ACM, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1411 141.,6992,690, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Tamara Gibson Kolda. Multilinear operators for higher-order decompositions, volume United States. Department of Energy, 2006.

---

Under review as a conference paper at ICLR 2021
Bin Liu, Lirong He, Yingming Li, Shandian Zhe, and Zenglin Xu. Neuralep: Bayesian multiway data analysis with neural tensor decomposition. Cognitive Computation, 1066):0511-061,, 2018.
Hanpeng Liu, Yaguang Li, Michael Tsang, and Yan Liu. Costco: A neural tensor completion model for sparse tensors. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 324-334, 2019.
David JC MacKay. The evidence framework applied to classification networks. Neural computation, 4(5):720-73,, 1992.
Thomas P Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pp. 362-369, 2001a.
Thomas Peter Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001b.
Kevin P Murphy, Yair Weiss, and Michael I Jordan. Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pp. 467-46 475. Morgan Kaufmann Publishers Inc.. 1999.
Gary W Oehlert. note on the delta method. The American Statistician, 46(1):27-29, 1992.
Piyush Rai, Yingjaan Wang, Shengbo Guo, Gary Chen, David Dunson, and Lawrence Carin. Scalable Bayesian low-rank decomposition of incomplete multiway tensors. In Proceedings of the 31th International Conference on Machine Learning (ICML), 2014.
Piyush Rai, Changwei Hu, Matthew Harding, and Lawrence Carin. Scalable probabilistic tensor factorization for binary and count data. In IJCAL, 2015.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Amnon Shashua and Tamir Hazan. Non -egatiie tensor factorization with applications to statistics and computer vision. In Proceedings of the 22th International Conference Machine Learning (ICML), pp. 792-799, 2005.
Ilya Sutskever, Joshua B Tenenbaum, and Ruslan R Salakhutdinov. Modelling relational data using bayesian clustered tensor factorization. In Advances in neural information processing systems, pp. 1821-1828, 2009.
Michalis K Titsias and Miguel Lázaro-Gredilla. Spike and slab variational inference for multi-task and multiple kernel learning. In Advances in neural information processing systems, pp. 2339-23322220 2347, 2011.
Ledyard Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31: 279-311, 1966.
Martin Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1 1(1-2):1-305, 2008.
Chong Wang and David M Blei. Variational inference in nonconjugate models. Journal of Machine Learning Research, 1405p13:100-1033, 2013.
Zheng Wang and Shandian Zhe. Conditional expectation propagation. In UAI, pp. 6, 2019.
Zenglin Xu, Feng Yan, and Yuan Qi. Infinite Tucker decomposition: Nonparametric Bayesian models for multiway data analysis. In Proceedings of the 29th International Conference on Machine Learning (ICML), 2012.
Y. Yang and D.B. Dunson. Bayesian conditional tensor factorizations for high-dimensional classifica- tion. Journal of the Royal Statistical Society B, revision submitted, 2013.
Shandian Zhe, Zenglin Xu, Xingi Chu, Yuan Qi, and Youngja Park. Scalable nonparametric multiway data analysis. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, pp. 1125 11 113, 2015.
10

---

Under review as a conference paper at ICLR 2021
Shandian Zhe, Kai Zhang, Pengyuan Wang, Kuang-chih Lee, Zenglin Xu, Yuan Qi, and Zoubin Ghahramani. Distributed flexible nonlinear tensor factorization. In Advances in Neural Information Processing Systems, pp. 2016.


Reviews:
[{"review_id": "lewXU7bGuDM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nUpdate: The authors addressed part of my concerns. For the factor estimation, the proposed method relies on first order approximations while learning the posterior of the factors; however, the approximation error does not enter into the posterior.  The approximation also raises concerns regarding the convergence of the algorithm. Overall, I think the approach is promising, but some justification of the quality of the approximation is needed. Thus, I tend to keep my rating.\n\n##############\n\nThe authors propose a streaming approach to tensor factorization with Bayesian neural networks. The problem is to factorize a three-way tensor with Gaussian noise. The proposed approach combines a Bayesian neural network (BNN) whose output predicts the entries of the tensor and the streaming variational Bayes (SVB) for incremental posterior updates. In addition, a spike-and-slab prior is placed on the BNN weights to encourage sparsity as well as prevent overfitting.  The authors performed empirical studies on four real datasets, DBLP, Anime, ACC, and MovieLen1M, where improved prediction accuracy of the tensor entries are reported.\n\nFrom the empirical evaluation, the objective of the proposed approach is to ensure that the output of the BNN matches the tensor entries. However, this does not necessarily guarantee the correctness of the recovered factor matrices (input to the BNN). I feel that the problem setting of the paper is somehow different from the standard CP factorization setting where the (unique) factors are of primary interest. It would be good to add some discussions about the correctness and/or uniqueness of the uncovered factors.\n\nStrengths:\n- The idea of introducing a spike-and-slab prior in the factorization is interesting.\n- Using SVB for online posterior updates is computationally practical.\n\nWeaknesses:\n- The presentation could be improved — it is not very clear how the factor matrices are estimated.\n- Factorization using the proposed method is not unique (up to rescaling and permutation of the factor matrices). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V7AlN2kkDGj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##################################################\n\nAfter author feedback:\n\nThe authors have addressed my comments, though it is impossible to evaluate what the authors promise to do in future work. My evaluation remains unchanged. \n\n \n##################################################\n\nSummary:\nThe paper proposes a Bayesian neural network model for tensor factorization, with particular focus on streaming data. The method features spike-and-slab prior to avoid overfitting. The main computational challenge is how to handle the streaming data in the posterior inference steps. The authors propose an approximate algorithm via moment matching. The method shows promising results in simulation.\n\n##################################################\n\nPros:\n1. The proposed method is novel and mostly described and motivated in a clear fashion. It is an interesting addition to the current literature of Bayesian methods for streaming data.\n2. The predictive performance, as shown by the authors, show significant improvements over the alternatives.\n\n##################################################\n\nCons:\nWhile I like the overall method, I have some questions about the inferential procedure. While it seems to work in the examples chosen by the authors, readers could benefit more from a clearer presentation of the methods, especially in terms of the computation and approximation accuracy trade-off.\n\n1.  There are many (usually first order) approximation steps in the inference procedure. Can you quantify how these approximations affect the predictive performance and comment more on the validity of the Bayesian algorithm? To be more specific, I would like to see more justification or discussion in using the first order approximation of f_w(x_{i_n}) in both the binary and continuous case when calculating the normalizing constant Z_n. It is unclear how the approximation affects the estimation of Z_n after integration, and then in turn affects the posterior distribution. It also makes me wonder if the second-order Tayler expansion leads to no benefit in performance (claimed after equation 12) could be due to how f_w is approximated? The current justifications offered above equation 9 is useful in intuitively understand the considerations of the authors, yet not fully satisfactory why that is the right thing to do. In general, it is difficult for readers to see how the several approximations involved in the inference step change the posterior distribution computed versus the original target.\n\n2. One thing that is missing is how the spike-and-slab selection indicators S are updated. Point mass spike are notoriously difficult to sample.  Is that the case here as well?\n\n##################################################\n\nMinor points:\n1. Caption of figure 1 is cut out.\n2. I can see the authors want to fit the length to 8 pages. Consider shotern the background section. Some of the notations can be introduced only when describing the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "F7kOQSi1Hb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a probabilistic tensor factorization model for streaming data. The model uses:\n\n* neural networks to learn richer factors,\n* spike-and-slab prior on NN weights\n* and online moment-matching for posterior inference.\n\nMajor comments:\n\n1. Overall, the novelty of this work is minimal. There are existing works in literature on deep factorization models e.g. for recommender systems [Xue, et al. 2017]. It should be clarified how this work stands out.\n\n2. Authors posit that existing tensor factorizations only conduct multilinear decompositions, yet to make inference of the proposed SPIDER model tractable, they use extensive linearization relaxations. How is the model capacity to learn non-linear relationships in data preserved despite these simplifications?\n\n3. It has been claimed that stochastic gradient methods with re-parametrization trick perform poorly for parameter inference in this setting. It would be enlightening to see this is experimentally validated, as the SGD is the most successful optimization scheme for NN methods.\n\n4. How does the spike-and-slab prior perform compared to other regularization methods such as drop-out? \n\n5. The message of experiments section is not quite coherent. What is the trade-off between performance accuracy and computational complexity of the proposed framework?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2b7CWUjFnpx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################\n\nAfter author feedback:\nThanks for the detailed feedback from the authors. Most of my concerns have been addressed and I will keep my scores unchanged. Please add the additional information in the feedback to the final version.\n\n##########################\n\nSummary:\n\nThis paper proposes a streaming probabilistic deep tensor factorization model, called SPIDER, to solve the tensor factorization problem under streaming setting with deep factorization parameterized by Bayesian neural networks. To encourage sparsity and prevent overfitting, a spike-and-slab prior is used for the weights of the neural network layers. Learning algorithms are also derived based on the expectation propagation framework where the posterior of the network output is approximated using multivariate Delta's method and moment matching. Experimental results show improvement against existing tensor factorization models.\n\n\n##########################\n\nPros:\n \n1. Although there has been some work on streaming or online tensor factorization, the (Bayesian) deep factorization has never been addressed in the streaming setting, which is also quite an important problem to study.\n2. The incorporation of spike-and-slab prior for sparsity is good for many practical applications.\n3. Overall, the paper is organized and easy to follow.\n4. The experiment section is well structured and multiple real-world large datasets are used.\n\n\n##########################\n\nCons/Questions: \n\n1. Does the algorithm converge? It would be great to see any theoretical or empirical analysis on convergence.\n2. Two important baselines NeuralCP and CoSTCo are missing. As they are very recent and state-of-the-art models in neural tensor factorization, it would be great to see comparisons against them.\n3. It is unclear from the experimental section that how many parameters does each model have. Can the authors provide a table or figure about the number of parameters of each model?\n4. One of the most important properties of CP and Tucker decomposition is their interpretability. I am wondering if any interpretation of the factors can be carried out for the proposed model.\n5. The caption of Fig. 1 seems unfinished or blocked by Fig 2.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a Bayesian neural network model for tensor factorization, with particular focus on streaming data. The key contribution is the streaming posterior inference of the deep TF models.  The combinations of online tensor factorization, Bayesian NN with sparsity priors, posterior inference is new and interesting.  However, there are many approximation steps, and the quality of the approximation and convergence of algorithm are not well justified.   
Decision: Reject