SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shikai Fang
Kahlert School of Computing The University of Utah shikai.fang@utah.edu
Xin Yu
Kahlert School of Computing  
The University of Utah  
yuxwind@gmail.com
Kahlert School of Computing The University of Utah shiboli.cs@gmail.com
Zheng Wang
Kahlert School of Computing  
The University of Utah  
u1208847@utah.edu
Robert M. Kirby
Kahlert School of Computing  
The University of Utah  
kirby@cs.utah.edu
Shandian Zhe*
Kahlert School of Computing  
The University of Utah  
zhe@cs.utah.edu
Practical tensor data is often along with time information. Most existing temporal decomposition approaches estimate a set of fixed factors for the objects in each tensor mode, and hence cannot capture the temporal evolution of the objects' representation. More important, we lack an effective approach to capture such evolution from streaming data, which is common in real-world applications. To address these issues, we propose Streaming Factor Trajectory Learning (SFTL) for temporal tensor decomposition. We use Gaussian processes (GPs) to model the trajectory of factors so as to flexibly estimate their temporal evolution. To address the computational challenges in handling streaming data, we convert the GPs into a state-space prior by constructing an equivalent stochastic differential equation (SDE). We develop an efficient online filtering algorithm to estimate a decoupled running posterior of the involved factor states upon receiving new data. The decoupled estimation enables us to conduct standard Rauch-Tung-Striebel smoothing to compute the full posterior of all the trajectories in parallel, without the need for revisiting any previous data. We have shown the advantage of SFTL in both synthetic tasks and real-world applications. The code is available at https://github.com/xuangu-fang/Streaming-Factor-Trajectory-Learning.
Tensor data is common in real-world applications. For example, one can extract a three-mode tensor (patient, drug, clinic) from medical service records and a four-mode tensor (customer, commodity, seller, web-page) from the database of an online shopping platform. Tensor decomposition is a fundamental tool for tensor data analysis. It introduces a set of factors to represent the objects in each mode, and estimate these factors by reconstructing the observed entry values. These factors can be viewed as the underlying properties of the objects. We can use them to search for interesting structures within the objects (e.g., communities and outliers) or as discriminate features for predictive tasks, such as personalized treatment or recommendation.
Real-world tensor data is often accompanied with time information, namely the timestamps at which the objects of different modes interact to produce the entry values. Underlying the timestamps can be rich, valuable temporal patterns. While many temporal decomposition methods are available,
most of them estimate a set of static factors for each object — they either introduce a discrete time mode [Xiong et al., 2010, Zhe et al., 2016a] or inject the timestamp into the decomposition model [Zhang et al., 2021, Fang et al., 2022, Li et al., 2022]. Hence, these methods cannot learn the temporal variation of the factors. Accordingly, they can miss important evolution of the objects' inner properties, such as health and income. In addition, practical applications produce data streams at a rapid pace [Du et al., 2018]. Due to the resource limit, it is often prohibitively expensive to decompose the entire tensor from scratch whenever we receive new data. Many privacy-protecting applications (e.g., SnapChat) even forbid us to preserve or re-access the previous data. Therefore, not only do we need a more powerful decomposition model that can estimate the factor evolution, we also need an effective method to capture such evolution from fast data streams.
To address these issues, we propose SFTL, a Bayesian streaming factor trajectory learning approach for temporal tensor decomposition. Our method can efficiently handle data streams, with which to estimate a posterior distribution of the factor trajectory to uncover the temporal evolution of the objects' representation. Our method never needs to keep or re-visit previous data. The contribution of our work is summarized as follows.
- First, we use a Gaussian process (GP) prior to sample the factor trajectory of each object as a function of time. As a nonparametric function prior, GPs are flexible enough to capture a variety of complex temporal dynamics. The trajectories are combined through a CANDECOMP/PARAFAC (CP) [Harshman, 1970] or Tucker decomposition [Tucker, 1966] form to sample the tensor entry values at any time point.
- Second, to sidestep the expensive covariance matrix computation in the GP, which further causes challenges in streaming inference, we use spectral analysis to convert the GP into a linear time invariant (LTI) stochastic differential equation (SDE). Then we convert the SDE into an equivalent state-space prior over the factor (trajectory) states at the observed timestamps. As a result, the posterior inference becomes easier and computationally efficient.
- Third, we take advantage of the chain structure of the state-space prior and use the recent conditional expectation propagation framework [Wang and Zhe, 2019] to develop an efficient online filtering algorithm. Whenever a collection of entries at a new timestamp arrives, our method can efficiently estimate a decoupled running posterior of the involved factor states, with a Gaussian product form. The decoupled Gaussian estimate enables us to run standard RTS smoothing [Särkkä, 2013] to compute the full posterior of each factor trajectory independently and in parallel, without revisiting any previous data. Our method at worst has a linear scalability with respect to the number of observed timestamps.
We first evaluated our method in a simulation study. On synthetic datasets, SFTL successfully recovered several nonlinear factor trajectories, and provided reasonable uncertainty estimation. We then tested our method on four real-world temporal tensor datasets for missing value prediction. In both online and final predictive performance, SFTL consistently outperforms the state-of-the-art streaming CP and Tucker decomposition algorithms by a large margin. In most cases, the prediction accuracy of SFTL is even higher than the recent static decomposition methods, which have to pass through the dataset many times. Finally, we investigated the learned factor trajectories from a real-world dataset. The trajectories exhibit interesting temporal evolution.
Tensor Decomposition. We denote an  $M$ -mode tensor by  $\mathcal{Y} \in \mathbb{R}^{d_1 \times \dots \times d_M}$ , where each mode  $m$  has  $d_m$  dimensions, corresponding to  $d_m$  objects. Each tensor entry is indexed by a tuple  $\ell = (\ell_1, \ldots, \ell_M)$ , and the value is denoted by  $y_\ell$ . For decomposition, we introduce a set of latent factors  $\mathbf{u}_j^m \in \mathbb{R}^{R_m}$  to represent each object  $j$  in mode  $m$  ( $1 \leq m \leq M$ ). One most popular tensor decomposition model is the CANDECOMP/PARAFAC (CP) decomposition [Harshman, 1970], which sets  $R_1 = \ldots = R_M = R$ , and uses the following element-wise form,  $y_\ell \approx \mathbf{1}^\top (\mathbf{u}_{\ell_1}^1 \circ \ldots \circ \mathbf{u}_{\ell_M}^M) = \sum_{r=1}^{R} \prod_{m=1}^{M} u_{\ell_m,r}^m$ , where  $\circ$  is the element-wise product. Another commonly used model is Tucker decomposition [Tucker, 1966],  $y_\ell \approx \operatorname{vec}(\mathcal{W})^\top (\mathbf{u}_{\ell_1}^1 \otimes \ldots \otimes \mathbf{u}_{\ell_M}^M) = \sum_{r_1=1}^{R_1} \ldots \sum_{r_M=1}^{R_M} [w_{\mathbf{r}} \cdot \prod_{m=1}^{M} u_{\ell_m,r_m}^m]$ , where  $\mathcal{W} \in \mathbb{R}^{R_1 \times \dots \times R_M}$  is the tensor-core parameter,  $\operatorname{vec}(\cdot)$  is the vectorization,  $\otimes$  is the Kronecker product, and  $\mathbf{r} = (r_1, \ldots, r_M)$ .
Gaussian Process (GP)s are nonparametric function priors. For a function  $f(\mathbf{x})$ , if we place a GP prior,  $f \sim \mathcal{GP}(0, \kappa(\mathbf{x}, \mathbf{x}'))$ , it means  $f(\cdot)$  is sampled as a realization of the GP with covariance function  $\kappa$ , which is often chosen as a kernel function. The GP prior only models the correlation between the function values, namely,  $\operatorname{cov}(f(\mathbf{x}), f(\mathbf{x}')) = \kappa(\mathbf{x}, \mathbf{x}')$ , and does not assume any parametric form of the function. Hence, GPs are flexible enough to estimate various complex functions from data, e.g., from multilinear to highly nonlinear. The finite projection of the GP is a Gaussian distribution. That is, given an arbitrary collection of inputs  $\{\mathbf{x}_1, \ldots, \mathbf{x}_N\}$ , the corresponding function values  $\mathbf{f} = [f(\mathbf{x}_1), \ldots, f(\mathbf{x}_N)]^\top$  follow a multi-variate Gaussian prior distribution,  $p(\mathbf{f}) = \mathcal{N}(\mathbf{f}|\mathbf{0}, \mathbf{K})$  where  $\mathbf{K}$  is the covariance matrix and each  $[\mathbf{K}]_{mn} = \kappa(\mathbf{x}_m, \mathbf{x}_n)$ .
3 Bayesian Temporal Tensor Decomposition with Factor Trajectories
In real-world applications, tensor data is often associated with time information, namely, the timestamps at which the objects of different modes interact to generate the entry values. To capture the potential evolution of the objects' inner properties, we propose a Bayesian temporal tensor decomposition model that can estimate a trajectory of the factor representation. Specifically, for each object  $j$  in mode  $m$ , we model the factors as a function of time,  $\mathbf{u}_j^m : [0,\infty] \to \mathbb{R}^R$ . To flexibly capture a variety of temporal evolution, we assign a GP prior over each element of  $\mathbf{u}_j^m(t) = [u_{j,1}^m(t), \dots, u_{j,R}^m(t)]^\top$ , i.e.,  $u_{j,r}^m(t) \sim \mathcal{GP}(0,\kappa(t,t')) (1 \leq r \leq R)$ . Given the factor trajectories, we then use the CP or Tucker form to sample the entry values at different time points. For the CP form, we have
$$
p \left(y _ {\ell} (t) \mid \mathcal {U} (t)\right) = \mathcal {N} \left(y _ {\ell} (t) \mid \mathbf {1} ^ {\top} \left(\mathbf {u} _ {\ell_ {1}} ^ {1} (t) \circ \dots \circ \mathbf {u} _ {\ell_ {M}} ^ {M} (t)\right), \tau^ {- 1}\right), \tag {1}
$$
where  $\mathcal{U}(t) = \{\mathbf{u}_j^m (t)\}$  includes all the factor trajectories, and  $\tau$  is the inverse noise variance, for which we assign a Gamma prior,  $p(\tau) = \mathrm{Gam}(\tau |\alpha_0,\alpha_1)$ . For the Tucker form, we have  $p(y_{\ell}(t)|\mathcal{U}(t),\mathcal{W}) = \mathcal{N}(y_{\ell}(t)|\mathrm{vec}(\mathcal{W})^\top (\mathbf{u}_{\ell_1}^1 (t)\otimes \ldots \otimes \mathbf{u}_{\ell_M}^M (t)),\tau^{-1})$  where we place a standard normal prior over the tensor-core,  $p(\mathrm{vec}(\mathcal{W})) = \mathcal{N}(\mathrm{vec}(\mathcal{W})|\mathbf{0},\mathbf{I})$ . In this work, we focus on continuous observations. It is straightforward to extend our method for other types of observations.
Suppose we have a collection of observed entry values and timestamps,  $\mathcal{D} = \{(\ell_1,y_1,t_1),\ldots ,(\ell_N,y_N,t_N)\}$  where  $t_1\leq \dots \leq t_N$ . We denote the sequence of timestamps when a particular object  $j$  of mode  $m$  participated in the observed entries by  $s_{j,1}^{m} < \ldots < s_{j,c_{j}^{m}}^{m}$  where  $c_{j}^{m}$  is the participation count of the object. Note that it is a sub-sequence of  $\{t_n\}$ . From the GP prior, the values of each  $u_{j,r}^{m}(t)$  at these timestamps follow a multi-variate Gaussian distribution,  $p(\mathbf{u}_{j,r}^{m}) = \mathcal{N}(\mathbf{u}_{j,r}^{m}|\mathbf{0},\mathbf{K}_{j}^{m})$  where  $\mathbf{u}_{j,r}^{m} = [u_{j,r}^{m}(s_{j,1}^{m}),\ldots ,u_{j,r}^{m}(s_{j,c_{j}^{m}}^{m})]^{\top 2}$  and  $\mathbf{K}_j^m$  is the covariance/kernel matrix computed at these timestamps. The joint probability with the CP form is
$$
\begin{array}{l} p (\{\mathbf {u} _ {j, r} ^ {m} \}, \tau , \mathbf {y}) = \prod_ {m = 1} ^ {M} \prod_ {j = 1} ^ {d _ {m}} \prod_ {r = 1} ^ {R} \mathcal {N} (\mathbf {u} _ {j, r} ^ {m} | \mathbf {0}, \mathbf {K} _ {j} ^ {m}) \cdot \operatorname {G a m} (\tau | \alpha_ {0}, \alpha_ {1}) \\ \cdot \prod_ {n = 1} ^ {N} \mathcal {N} \left(y _ {n} \mid \mathbf {1} ^ {\top} \left(\mathbf {u} _ {\ell_ {n 1}} ^ {1} \left(t _ {n}\right) \circ \dots \circ \mathbf {u} _ {\ell_ {n M}} ^ {M} \left(t _ {n}\right)\right), \tau^ {- 1}\right). \tag {2} \\ \end{array}
$$
The joint probability with the Tucker form is the same except that we use the Tucker likelihood instead and multiply with the prior of tensor-core  $p(\mathcal{W})$ .
While this formulation is straightforward, it can introduce computational challenges. There are many multi-variate Gaussian distributions in the joint distribution (2), i.e.,  $\{\mathcal{N}(\mathbf{u}_{j,r}^{m}|\mathbf{0},\mathbf{K}_{j}^{m})\}$ . The time and space complexity to compute each  $\mathcal{N}(\mathbf{u}_{j,r}^{m}|\mathbf{0},\mathbf{K}_{j}^{m})$  is  $\mathcal{O}\left(\left(c_j^m\right)^3\right)$  and  $\mathcal{O}\left(\left(c_j^m\right)^2\right)$ , respectively. With the increase of  $N$ , the appearance count  $c_{j}^{m}$  for many objects can grow as well, making the computation cost very expensive or even infeasible. The issue is particularly severe when we handle streaming data — the number of timestamps grows rapidly when new data keeps coming in, so does the size of each covariance matrix.
3.1 Equivalent Modeling with State-Space Priors
To sidestep expensive covariance matrix computation and ease the inference with streaming data, we follow [Hartikainen and Särkkä, 2010] to convert the GP prior into an SDE via spectral analysis.
For convenience, we abuse the notation a little bit: we denote by  $\mathbf{u}_{j,r}^{m}(\cdot)$  trajectory function and by  $\mathbf{u}_{j,r}^{m}$  the values of the trajectory function at the observed timestamps.
We use a Matérn kernel  $\kappa_{\nu}(t,t^{\prime}) = a\frac{(\sqrt{2\nu}\Delta)^{\nu}}{\Gamma(\nu)2^{\nu - 1}} K_{\nu}\left(\frac{\sqrt{2\nu}}{\rho}\Delta\right)$  where  $\Gamma (\cdot)$  is the Gamma function,  $\Delta = |t - t^{\prime}|$ ,  $a > 0$ ,  $\rho >0$ ,  $K_{\nu}$  is the modified Bessel function of the second kind, and  $\nu = p + \frac{1}{2}$ $(p\in \{0,1,2,\ldots \})$  as the GP covariance. Via the analysis of the power spectrum of  $\kappa_{\nu}$ , we can show that if  $f(t)\sim \mathcal{GP}(0,\kappa_{\nu}(t,t'))$ , it can be characterized by a linear time-invariant (LTI) SDE, with state  $\mathbf{z} = (f,f^{(1)},\dots ,f^{(p)})^\top$  where  $f^{(k)}\triangleq \mathrm{d}^k f / \mathrm{d}t^k$
$$
\frac {\mathrm {d} \mathbf {z}}{\mathrm {d} t} = \mathbf {A} \mathbf {z} + \boldsymbol {\eta} \cdot \beta (t), \tag {3}
$$
where  $\beta (t)$  is a white noise process with diffusion  $\sigma^2$
$$
\mathbf {A} = \left( \begin{array}{c c c c} 0 & 1 & & \\ & \ddots & \ddots & \\ & & 0 & 1 \\ - c _ {0} & \ldots & - c _ {p - 1} & - c _ {p} \end{array} \right), \quad \boldsymbol {\eta} = \left( \begin{array}{c} 0 \\ \vdots \\ 0 \\ 1 \end{array} \right).
$$
Both  $\sigma^2$  and  $\mathbf{A}$  are obtained from the parameters in  $\kappa_{\nu}$ . Due to the space limit, we leave the detailed derivation in Appendix (Section A). The LTI-SDE is particularly useful in that its finite set of states follow a Gauss-Markov chain, i.e., the state-space prior. Given arbitrary  $t_1 < \ldots < t_L$ , we have
$$
p (\mathbf {z} (t _ {1}), \dots , \mathbf {z} (t _ {L})) = p (\mathbf {z} (t _ {1})) \prod_ {k = 1} ^ {L - 1} p (\mathbf {z} (t _ {k + 1}) | \mathbf {z} (t _ {k})),
$$
where  $p(\mathbf{z}(t_1)) = \mathcal{N}(\mathbf{z}(t_1)|\mathbf{0},\mathbf{P}_\infty)$ ,  $p(\mathbf{z}(t_{k + 1})|\mathbf{z}(t_k)) = \mathcal{N}(\mathbf{z}(t_{k + 1})|\mathbf{F}_k\mathbf{z}(t_k),\mathbf{Q}_k)$ ,  $\mathbf{P}_{\infty}$  is the stationary covariance matrix computed by solving the matrix Riccati equation [Lancaster and Rodman, 1995],  $\mathbf{F}_n = \exp (\Delta_k\cdot \mathbf{A})$  where  $\Delta_{k} = t_{k + 1} - t_{k}$ , and  $\mathbf{Q}_k = \mathbf{P}_{\infty} - \mathbf{A}_k\mathbf{P}_{\infty}\mathbf{A}_k^\top$ . Therefore, we do not need the full covariance matrix as in the standard GP prior, and the computation is much more efficient. The chain structure is also convenient to handle streaming data as we will explain later.
We therefore convert the GP prior over each factor trajectory  $u_{j,r}^{m}(t)$  into an LTI-SDE. We denote the corresponding state by  $\mathbf{z}_{j,r}^{m}(t)$ . For example, if we choose  $p = 1$ , then  $\mathbf{z}_{j,r}^{m}(t) = [u_{j,r}^{m}(t); \mathrm{d}u_{j,r}^{m}(t)/\mathrm{d}t]$ . For each object  $j$  in mode  $m$ , we concatenate all its trajectory states into one,  $\mathbf{z}_j^m (t) = [\mathbf{z}_{j,1}^m (t);\dots ;\mathbf{z}_{j,R}^m (t)]$ . Then on all of its timestamps  $s_{j,1}^{m} < \ldots < s_{j,c_{j}^{m}}^{m}$ , we obtain a state-space prior
$$
p \left(\mathbf {h} _ {j, 1} ^ {m}\right) = \mathcal {N} \left(\mathbf {h} _ {j, 1} ^ {m} \mid \mathbf {0}, \overline {{\mathbf {P}}} _ {\infty}\right), \quad p \left(\mathbf {h} _ {j, k + 1} ^ {m} \mid \mathbf {h} _ {j, k} ^ {m}\right) = \mathcal {N} \left(\mathbf {h} _ {j, k + 1} ^ {m} \mid \overline {{\mathbf {F}}} _ {j, k} ^ {m} \mathbf {h} _ {j, k} ^ {m}, \overline {{\mathbf {Q}}} _ {j, k} ^ {m}\right), \tag {4}
$$
where  $\mathbf{h}_{j,k}^{m}\triangleq \mathbf{z}_{j}^{m}(s_{j,k}^{m})$ ,  $\overline{\mathbf{P}}_{\infty} = \mathrm{diag}(\mathbf{P}_{\infty},\dots ,\mathbf{P}_{\infty})$ ,  $\overline{\mathbf{F}}_{j,k}^{m} = \mathrm{diag}\left(\mathbf{F}_{j,k}^{m},\dots ,\mathbf{F}_{j,k}^{m}\right)$ ,  $\mathbf{F}_{j,k}^{m} = e^{(s_{j,k + 1}^{m} - s_{j,k}^{m})\mathbf{A}}$ ,  $\overline{\mathbf{Q}}_{j,k}^{m} = \mathrm{diag}\left(\mathbf{Q}_{j,k}^{m},\dots ,\mathbf{Q}_{j,k}^{m}\right)$ , and  $\mathbf{Q}_{j,k}^{m} = \mathbf{P}_{\infty} - \mathbf{F}_{j,k}^{m}\mathbf{P}_{\infty}\left(\mathbf{F}_{j,k}^{m}\right)^{\top}$ .
The joint probability of our model with the CP form now becomes
$$
\begin{array}{l} p (\{\mathbf {h} _ {j, k} ^ {m} \}, \tau , \mathbf {y}) = p (\tau) \prod_ {m = 1} ^ {M} \prod_ {j = 1} ^ {d _ {m}} p (\mathbf {h} _ {j, 1} ^ {m}) \prod_ {k = 1} ^ {c _ {j} ^ {m} - 1} p (\mathbf {h} _ {j, k + 1} ^ {m} | \mathbf {h} _ {j, k} ^ {m}) \\ \cdot \prod_ {n = 1} ^ {N} \mathcal {N} \left(y _ {n} \mid \mathbf {1} ^ {\top} \left(\mathbf {u} _ {\ell_ {n 1}} ^ {1} \left(t _ {n}\right) \circ \dots \circ \mathbf {u} _ {\ell_ {n M}} ^ {M} \left(t _ {n}\right)\right), \tau^ {- 1}\right). \tag {5} \\ \end{array}
$$
Note that in the likelihood, each  $\mathbf{u}_{\ell_{nm}}^m (t_n)(1\leq j\leq M)$  is contained in a corresponding state vector  $\mathbf{h}_{\ell_{nm},k}^{m}$  such that  $s_{\ell_{nm},k}^{m} = t_{n}$  (by definition, we then have  $\mathbf{h}_{\ell_{nm},k}^{m} = \mathbf{z}_{\ell_{nm}}^{m}(t_{n})$ ). The joint probability with the Tucker form is similar, which we omit to save the space.
4 Trajectory Inference from Streaming Data
In this section, we develop an efficient, scalable algorithm for factor trajectory estimation from streaming data. In general, we assume that we receive a sequence of (small) batches of observed tensor entries,  $\{\mathcal{B}_1,\mathcal{B}_2,\ldots \}$ , generated at different timestamps,  $\{t_1,t_2,\dots \}$ . Each batch  $\mathcal{B}_n$  is generated at timestamp  $t_n$  and  $t_n < t_{n + 1}$ . Denote by  $\mathcal{D}_{t_n}$  all the data up to timestamp  $t_n$ , i.e.,  $\mathcal{D}_{t_n} = \mathcal{B}_1\cup \ldots \cup \mathcal{B}_n$ . Upon receiving  $\mathcal{B}_{n + 1}$ , we intend to update our model without revisiting  $\mathcal{D}_{t_n}$
Figure 1: A graphical representation of our factor trajectory learning, from which we can see  $\{\Theta_{n + 1},\mathcal{B}_{n + 1}\}$  are independent to  $\mathcal{D}_{t_n}$  conditioned on  $\Theta_{n}$  and the noise inverse variance  $\tau$  , namely,  $\Theta_{n + 1},\mathcal{B}_{n + 1}\perp \mathcal{D}_{t_n}|\Theta_n,\tau$
to provide the trajectory posterior estimate,  $\{p(\mathbf{u}_j^m (t)|\mathcal{D}_{t_{n + 1}})|\forall t\geq 0,1\leq m\leq M,1\leq j\leq d_m\}$  where  $\mathcal{D}_{t_{n + 1}} = \mathcal{D}_{t_n}\cup \mathcal{B}_{n + 1}$
To this end, we first observe that the standard state-space model with a Gaussian likelihood has already provided a highly efficient streaming inference framework. Denote by  $\mathbf{x}_n$  and  $\mathbf{y}_n$  the state and observation at each step  $n$ , respectively. To handle streaming observations  $\{\mathbf{y}_1,\mathbf{y}_2,\ldots \}$ , we only need to compute and track the running posterior  $p(\mathbf{x}_n|\mathbf{y}_{1:n})$  upon receiving each  $\mathbf{y}_n$ , where  $\mathbf{y}_{1:n}$  denotes the total data up to step  $n$ . This is called Kalman filtering [Särkkä, 2013], which only depends on the running posterior at step  $n - 1$ , i.e.,  $p(\mathbf{x}_{n - 1}|\mathbf{y}_{1:n - 1})$ , and is highly efficient. After all the data is processed (suppose it stops at step  $N$ ), we can use Rauch-Tung-Striebel (RTS) smoother [Särkkä, 2013] to efficiently compute the full posterior of each state,  $p(\mathbf{x}_n|\mathbf{y}_{1:N})$ , from backward, which does not need to re-access any previous observations (see Section B in Appendix).
However, one cannot apply the above framework outright to our model, since the tensor decomposition likelihood of each observed entry couples the states of multiple factor trajectories, see (1) — which correspond to the participated objects at different modes. That means, the factor-state chains of different objects are dynamically intertwined through the received data. The multiplicative form of these states in the likelihood render the running posterior of each trajectory intractable to compute, not to mention running RTS smoother. To address this challenge, we take advantage of the chain structure and use the recent conditional Expectation propagation (CEP) framework [Wang and Zhe, 2019] to develop an efficient online filtering algorithm, which approximates the running posterior of the involved factor states as a product of Gaussian. Thereby, we can decouple the involved factor state chains, and conduct standard RTS smoothing for each chain independently.
Specifically, denote the sequence of timestamps when each object  $j$  of mode  $m$  has showed up in the data stream up to  $t_n$ , by  $s_{j,1}^m < s_{j,2}^m < \ldots < s_{j,c_{j,n}^m}^m$  where  $c_{j,n}^m$  is the object's appearance count up to  $t_n$ . Denote by  $\mathcal{I}_n^m$  the indexes of all the objects of mode  $m$  appearing in  $\mathcal{B}_n$ . Hence, for every object  $j \in \mathcal{I}_n^m$ , we have  $s_{j,c_{j,n}^m}^m = t_n$ , and  $\mathbf{h}_{j,c_{j,n}^m}^m \triangleq \mathbf{z}_j^m(t_n)$  is the factor state of the object at  $t_n$ . Upon receiving each  $\mathcal{B}_n$ , we intend to approximate the running posterior of all the involved factor states and noise inverse variance  $\tau$  with the following decoupled form,
$$
p \left(\tau , \left\{\mathbf {h} _ {j, c _ {j, n} ^ {m}} ^ {m} \mid j \in \mathcal {I} _ {n} ^ {m} \right\} _ {1 \leq m \leq M} \mid \mathcal {D} _ {t _ {n}}\right) \approx q (\tau \mid \mathcal {D} _ {n}) \prod_ {m = 1} ^ {M} \prod_ {j \in \mathcal {I} _ {n} ^ {m}} q \left(\mathbf {h} _ {j, c _ {j, n} ^ {m}} ^ {m} \mid \mathcal {D} _ {t _ {n}}\right), \tag {6}
$$
where  $q(\tau |\mathcal{D}_{t_n}) = \mathrm{Gam}(\tau |a_n,b_n)$ , and  $q(\mathbf{h}_{j,c_{j,n}^m}^m |\mathcal{D}_{t_n}) = \mathcal{N}(\mathbf{h}_{j,c_{j,n}^m}^m |\widehat{\boldsymbol{\mu}}_{j,c_{j,n}^m}^m, \widehat{\mathbf{V}}_{j,c_{j,n}^m}^m)$ . To this end, let us consider given the approximation at  $t_n$ , how to obtain the new approximation at  $t_{n + 1}$  (i.e., upon