SlowGuess/NeurIPS_2023_OCR huggingface dataset:
receiving  $\mathcal{B}_{n + 1})$  in the same form of (6). To simplify the notation, let us define the preceding states of the involved factors by  $\Theta_{n} = \{\mathbf{h}_{j,c_{j,n}^{m}}^{m}|j\in \mathcal{I}_{n + 1}^{m}\}_{m}$ , and the current states by  $\Theta_{n + 1} = \{\mathbf{h}_{j,c_{j,n + 1}^{m}}^{m}|j\in \mathcal{I}_{n + 1}^{m}\}_{m}$ . First, due to the chain structure of the prior over each  $\{\mathbf{h}_{j,c_{j,k}^{m}}^{m}|k = 0,1,2,\ldots \}$ , we can see that conditioned on  $\{\Theta_n,\tau \}$ , the current states  $\Theta_{n + 1}$  and the new observations  $\mathcal{B}_{n + 1}$  are independent of  $\mathcal{D}_{t_n}$ . This is because in the graphical model representation,  $\{\Theta_n,\tau \}$  have blocked all the paths from the old observations  $\mathcal{D}_{t_n}$  to the new state and observations [Bishop, 2007]; see Fig. 1 for an illustration. Then, we can derive that
$$
\begin{array}{l} p (\Theta_ {n + 1}, \Theta_ {n}, \tau | \mathcal {D} _ {t _ {n + 1}}) \propto p (\Theta_ {n + 1}, \Theta_ {n}, \tau , \mathcal {B} _ {n + 1} | \mathcal {D} _ {t _ {n}}) = p (\Theta_ {n}, \tau | \mathcal {D} _ {t _ {n}}) p (\Theta_ {n + 1}, \mathcal {B} _ {n + 1} | \Theta_ {n}, \tau , \mathcal {D} _ {t _ {n}}) \\ = p \left(\Theta_ {n}, \tau \mid \mathcal {D} _ {t _ {n}}\right) p \left(\Theta_ {n + 1} \mid \Theta_ {n}\right) p \left(\mathcal {B} _ {n + 1} \mid \Theta_ {n + 1}, \tau\right), \tag {7} \\ \end{array}
$$
where  $p(\Theta_n,\tau |\mathcal{D}_{t_n})$  is the running posterior at  $t_n$
$$
p(\Theta_{n + 1}|\Theta_{n}) = \prod_{m = 1}^{M}\prod_{j\in \mathcal{I}_{n + 1}^{m}}p(\mathbf{h}_{j,c_{j,n + 1}^{m}}^{m}|\mathbf{h}_{j,c_{j,n}^{m}}^{m}),
$$
each  $p(\mathbf{h}_{j,c_{j,n + 1}^m}^m |\mathbf{h}_{j,c_{j,n}^m}^m)$  is a conditional Gaussian distribution defined in (4), and  $p(\mathcal{B}_{n + 1}|\Theta_{n + 1},\tau) = \prod_{(\ell ,y)\in \mathcal{B}_{n + 1}}\mathcal{N}\left(y|\mathbf{1}^\top \left(\mathbf{u}_{\ell_1}^1 (t_{n + 1})\circ \dots \circ \mathbf{u}_{\ell_M}^M (t_{n + 1})\right),\tau^{-1}\right).$  Since  $p(\Theta_n,\tau |\mathcal{D}_{t_n})$  takes the form of (6), we can analytically marginalize out each  $\mathbf{h}_{j,c_{j,n}^m}^m\in \Theta_n,$  and obtain
$$
\begin{array}{l} p \left(\Theta_ {n + 1}, \tau \mid \mathcal {D} _ {t _ {n + 1}}\right) \propto \operatorname {G a m} \left(\tau \mid a _ {n}, b _ {n}\right) \prod_ {m = 1} ^ {M} \prod_ {j \in \mathcal {I} _ {n + 1} ^ {m}} \mathcal {N} \left(\mathbf {h} _ {j, c _ {j, n + 1} ^ {m}} ^ {m} \mid \widehat {\boldsymbol {\mu}} _ {j, c _ {j, n + 1} ^ {m}}, \widehat {\mathbf {V}} _ {j, c _ {j, n + 1} ^ {m}} ^ {m}\right) \tag {8} \\ \cdot \prod_ {(\boldsymbol {\ell}, y) \in \mathcal {B} _ {n + 1}} \mathcal {N} \left(y | \mathbf {1} ^ {\top} \left(\mathbf {u} _ {\ell_ {1}} ^ {1} (t _ {n + 1}) \circ \dots \circ \mathbf {u} _ {\ell_ {M}} ^ {M} (t _ {n + 1})\right), \tau^ {- 1}\right). \\ \end{array}
$$
If we view the R.H.S of (8) as a joint distribution with  $\mathcal{B}_{n + 1}$ , then our task amounts to estimating the posterior distribution, i.e., the L.H.S of (8). The product in the CP likelihood (and also Tucker likelihood) renders exact posterior computation infeasible, and we henceforth approximate
$$
\mathcal {N} \left(y | \mathbf {1} ^ {\top} \left(\mathbf {u} _ {\ell_ {1}} ^ {1} \left(t _ {n + 1}\right) \circ \dots \circ \mathbf {u} _ {\ell_ {M}} ^ {M} \left(t _ {n + 1}\right)\right)\right) \stackrel {{\infty}} {{\sim}} \prod_ {m = 1} ^ {M} \mathcal {N} \left(\mathbf {u} _ {\ell_ {m}} ^ {m} \left(t _ {n + 1}\right) | \boldsymbol {\gamma} _ {\ell_ {m}} ^ {m}, \boldsymbol {\Sigma} _ {\ell_ {m}} ^ {m}\right) \operatorname {G a m} (\tau | \alpha_ {\ell}, \omega_ {\ell}) \tag {9}
$$
where  $\infty$  means approximately proportional to. To optimize these approximation terms, we use the recent conditional Expectation propagation (CEP) framework [Wang and Zhe, 2019] to develop an efficient inference algorithm. It uses conditional moment matching to update each approximation in parallel and conducts fixed point iterations, and hence can converge fast. We leave the details in the Appendix (Section C). Once it is done, we substitute the approximation (9) into (8). Then the R.H.S of (8) becomes a product of Gaussian and Gamma terms over each state and  $\tau$ . We can then immediately obtain a closed-form estimation in the form as (6). At the beginning, when estimating  $p(\Theta_1,\tau |\mathcal{D}_{t_1})$ , since the preceding states  $\Theta_0 = \emptyset$ , we have  $a_{n} = \alpha_{0}$ ,  $b_{n} = \alpha_{1}$ ,  $\widehat{\boldsymbol{\mu}}_{j,c_{j,n + 1}^{m}}^{m} = \mathbf{0}$ , and  $\widehat{\mathbf{V}}_{j,c_{j,n + 1}^{m}}^{m} = \overline{\mathbf{P}}_{\infty}$  in (8), which is the prior of each  $\mathbf{h}_{j,c_{j,1}^{m}}^{m}$  and  $\tau$  (see (4)).
In this way, we can continuously filter the incoming batches  $\{\mathcal{B}_1,\mathcal{B}_2,\ldots \}$ . As a result, for the factor state chain of every object  $j$  in every mode  $m$ , along with each timestamp  $s_{j,k}^{m}$ , we can online estimate and track a running posterior approximation  $\{q(\mathbf{h}_{j,k}^{m}|\mathcal{D}_{t_{s_{j,k}^{m}}})|k = 1,2,\ldots \}$ , which is a Gaussian distribution. Hence, we can run the standard RTS smoother, to compute the full posterior of every factor state, with which we can compute the posterior of the trajectory at any time point  $t$  [Bishop, 2006]. Our method is summarized in Algorithm 1.
Algorithm Complexity. The time complexity of our algorithm processing a batch  $\mathcal{B}_n$  is  $\mathcal{O}(|\mathcal{B}_n|R^3)$  where  $|\cdot |$  is the size. The time complexity of RTS smoother for a particular object  $j$  in mode  $m$  is  $\mathcal{O}(R^3 c_{j,N}^m)$ , where  $N$  is the total number of timestamps. The space complexity of our algorithm is  $\mathcal{O}(\sum_{m = 1}^{M}\sum_{j = 1}^{d_m}c_{j,N}^m R^2)$ , which is to track the running posterior of the factor state at each appearing timestamp for every object. Since  $c_{j,N}^m\leq N$ , the complexity of our algorithm is at worst linear in  $N$ .
Many tensor decomposition methods have been developed, such as [Yang and Dunson, 2013, Rai et al., 2014, Zhe et al., 2015, 2016b,a, Tillinghast et al., 2020, Pan et al., 2020b, Fang et al., 2021a,b,
Algorithm 1 Streaming Factor Trajectory Learning (SFTL)
1: Input: kernel hyper-parameters  $a, \rho, \nu = p + \frac{1}{2}$  ( $p \in \{0, 1, 2, \ldots\}$ )  
2:  $n \gets 0$   
3: while Receiving a new batch of entreis  $\mathcal{B}_{n+1}$  do  
4: if  $n = 0$  then  
5: Set  $a_n = a_0, b_n = b_0, \hat{\mu}_{j,c_{j,n+1}}^m = 0$ , and  $\widehat{\mathbf{V}}_{j,c_{j,n+1}}^m = \overline{\mathbf{P}}_\infty$  in (8).  
6: Goto 9.  
7: end if  
8: Retrieve the involved preceding factor states  $\Theta_n = \{\mathbf{h}_{j,c_{j,n}}^m | j \in \mathcal{T}_{n+1}^m\}_m$  and their running posterior,  $p(\Theta_n, \tau|\mathcal{D}_{t_n}) \approx \mathrm{Gam}(\tau|a_n, b_n) \prod_{m=1}^{M} \prod_{j \in \mathcal{T}_{n+1}^m} \mathcal{N}(\mathbf{h}_{j,c_{j,n}}^m | \widehat{\boldsymbol{\mu}}_{j,c_{j,n}}^m, \widehat{\mathbf{V}}_{j,c_{j,n}}^m)$ .  
9: According to (8) and (9), use conditional Expectation Propagation to calculate the running posterior of the current factor states,  $p(\Theta_{n+1}, \tau|\mathcal{D}_{t_{n+1}}) \approx \mathrm{Gam}(\tau|a_{n+1}, b_{n+1}) \prod_{m=1}^{M} \prod_{j \in \mathcal{T}_{n+1}^m} \mathcal{N}(\mathbf{h}_{j,c_{j,n+1}}^m | \widehat{\boldsymbol{\mu}}_{j,c_{j,n+1}}^m, \widehat{\mathbf{V}}_{j,c_{j,n+1}}^m)$ .  
10: if Needed then  
11: Run RTS smoothing on any factor state chain  $\{\mathbf{h}_{j,k}^m | k = 1, 2, \ldots\}$  of interest.  
12: end if  
13:  $n \gets n+1$   
14: end while  
15: Run RTS smoothing for every factor state chain  $\{\mathbf{h}_{j,k}^m | k = 1, 2, \ldots\}$ .  
16: Return:  $\{q(\mathbf{h}_{j,k}^m |\mathcal{D}) | k = 1, 2, \ldots\}_{1 \leq m \leq M, 1 \leq j \leq d_m}$ ,  $q(\tau|\mathcal{D})$ , where  $\mathcal{D}$  is all the data received.
Tillinghast and Zhe, 2021, Tillinghast et al., 2022, Fang et al., 2022, Zhe and Du, 2018, Pan et al., 2020a, Wang et al., 2020, Pan et al., 2021, Wang et al., 2022]. For temporal decomposition, most existing methods augment the tensor with a discrete time mode to estimate additional factors for time steps, e.g., [Xiong et al., 2010, Rogers et al., 2013, Song et al., 2017, Du et al., 2018, Ahn et al., 2021]. The most recent works have conducted continuous-time decomposition. Zhang et al. [2021] used polynomial splines to model a time function as the CP coefficients. Li et al. [2022] used neuralODE [Chen et al., 2018] to model the entry value as a function of latent factors and time point. Fang et al. [2022] performed continuous-time Tucker decomposition, and modeled the tensor-core as a time function. To our knowledge, [Wang et al., 2022] is the first work to estimate factor trajectories. It places a GP prior in the frequency domain, and samples the factor trajectories via inverse Fourier transform. It then uses another GP to sample the entry values. While successful, this method cannot handle streaming data, and the black-box GP decomposition lacks interpretability.
Current Bayesian streaming tensor decomposition methods include [Du et al., 2018, Fang et al., 2021a, Pan et al., 2020b, Fang et al., 2021b], which are based on streaming variational Bayes [Broderick et al., 2013] or assumed density filtering (ADF) [Boyen and Koller, 1998]. ADF can be viewed as an instance of Expectation Propagation (EP) [Minka, 2001] for streaming data. EP approximates complex terms in the probability distribution with exponential-family members, and uses moment matching to iteratively update the approximations, which essentially is a fixed point iteration. To address the challenge of intractable moment matching, Wang and Zhe [2019] proposed conditional EP (CEP), which uses conditional moment matching and Taylor expansion to compute the moments for factorized approximations. The theoretical guarantees and error bound analysis for EP and ADF have been studied for a long time, such as [Boyen and Koller, 1998, Dehaene and Barthelme, 2015, 2018]. The most recent work [Fang et al., 2022] also uses SDEs to represent GPs and CEP framework for inference, but their GP prior is placed on the tensor-core, not for learning factor trajectories, and their method is only for static decomposition, and cannot handle streaming data.
We first conducted a simulation study, for which we simulated a two-mode tensor, with two nodes per mode. Each node is represented by a time-varying factor:  $u_1^1(t) = -\sin^3(2\pi t)$ ,  $u_2^1(t) = (1 - \sin^3(\frac{1}{2}\pi t))\sin^3(3\pi t)$ ,  $u_1^2(t) = \sin(2\pi t)$ , and  $u_2^2(t) = -\cos^3(3\pi t)\sin(3\pi t)\sin(2\pi t)$ . Given
(a)  $u_{1}^{1}(t)$
(b)  $u_2^1 (t)$
(c)  $u_1^2 (t)$
Figure 2: The learned factor trajectories from the synthetic data. The shaded region indicates the posterior standard deviation.
(d)  $u_2^2 (t)$
RMSE	FitRecord	ServerRoom	BeijingAir-2	BeijingAir-3
Static	PTucker	0.656 ± 0.147	0.458 ± 0.039	0.401 ± 0.01	0.535 ± 0.062
Tucker-ALS	0.846 ± 0.005	0.985 ± 0.014	0.559 ± 0.021	0.838 ± 0.026
CP-ALS	0.882 ± 0.017	0.994 ± 0.015	0.801 ± 0.082	0.875 ± 0.028
CT-CP	0.664 ± 0.007	0.384 ± 0.009	0.64 ± 0.007	0.815 ± 0.018
CT-GP	0.604 ± 0.004	0.223 ± 0.035	0.759 ± 0.02	0.892 ± 0.026
BCTT	0.518 ± 0.007	0.185 ± 0.013	0.396 ± 0.022	0.801 ± 0.02
NONFAT	0.503 ± 0.002	0.117 ± 0.006	0.395 ± 0.007	0.882 ± 0.014
THIS-ODE	0.526 ± 0.004	0.132 ± 0.003	0.54 ± 0.014	0.877 ± 0.026
Stream	POST	0.696 ± 0.019	0.64 ± 0.028	0.516 ± 0.028	0.658 ± 0.103
ADF-CP	0.648 ± 0.008	0.654 ± 0.008	0.548 ± 0.015	0.551 ± 0.043
BASS-Tucker	0.976 ± 0.024	1.000 ± 0.016	1.049 ± 0.037	0.991 ± 0.039
SFTL-CP	0.424 ± 0.014	0.161 ± 0.014	0.248 ± 0.012	0.473 ± 0.013
SFTL-Tucker	0.430 ± 0.010	0.331 ± 0.056	0.303 ± 0.041	0.439 ± 0.019
MAE
Static	PTucker	0.369 ± 0.009	0.259 ± 0.008	0.26 ± 0.006	0.263 ± 0.02
Tucker-ALS	0.615 ± 0.006	0.739 ± 0.008	0.388 ± 0.008	0.631 ± 0.017
CP-ALS	0.642 ± 0.012	0.746 ± 0.009	0.586 ± 0.056	0.655 ± 0.018
CT-CP	0.46 ± 0.004	0.269 ± 0.003	0.489 ± 0.006	0.626 ± 0.01
CT-GP	0.414 ± 0.001	0.165 ± 0.034	0.55 ± 0.012	0.626 ± 0.011
BCTT	0.355 ± 0.005	0.141 ± 0.011	0.254 ± 0.007	0.578 ± 0.009
NONFAT	0.341 ± 0.001	0.071 ± 0.004	0.256 ± 0.004	0.626 ± 0.007
THIS-ODE	0.363 ± 0.004	0.083 ± 0.002	0.345 ± 0.004	0.605 ± 0.013
Stream	POST	0.478 ± 0.014	0.476 ± 0.023	0.352 ± 0.022	0.486 ± 0.095
ADF-CP	0.449 ± 0.006	0.496 ± 0.007	0.385 ± 0.012	0.409 ± 0.029
BASS	0.772 ± 0.031	0.749 ± 0.01	0.934 ± 0.037	0.731 ± 0.02
SFTL-CP	0.242 ± 0.006	0.108 ± 0.008	0.15 ± 0.003	0.318 ± 0.008
SFTL-Tucker	0.246 ± 0.001	0.216 ± 0.034	0.185 ± 0.029	0.278 ± 0.011
Table 1: Final prediction error with  $R = 5$ . The results were averaged from five runs.
these factors, an entry value at time  $t$  is generated via  $y_{(i,j)}(t) \sim \mathcal{N}\left(u_i^1(t) u_j^2(t), 0.05\right)$ . We randomly sampled 500 (irregular) timestamps from [0, 1]. For each timestamp, we randomly picked two entries, and sampled their values accordingly. Overall, we sampled 1,000 observed tensor entry values.
We implemented SFTL with PyTorch [Paszke et al., 2019]. We used  $\nu = \frac{3}{2}$  and  $a = \rho = 0.3$  for the Matérn kernel. We streamed the sampled entries according to their timestamps, and ran our streaming factor trajectory inference based on the CP form. The estimated trajectories are shown in Fig. 2. As we can see, SFTL recovers the ground-truth pretty accurately, showing that SFTL has successfully captured the temporal evolution of the factor representation for every node. It is interesting to observe that when  $t$  is around 0, 0.5 and 1, the posterior standard deviation (the shaded region) increases significantly. This is reasonable: the ground-truth trajectories overlap at these time points, making it more difficult to differentiate/estimate their values at these time points. Accordingly, the uncertainty of the estimation increases. In Section D of Appendix, we further provide the root-mean-square error (RMSE) in recovering the four trajectories, and sensitivity analysis of the kernel parameters.
(a) FitRecord
(b) ServerRoom
(c) BeijingAir-2
Figure 3: Online prediction error with the number of processed entries  $(R = 5)$ .
(d) BeijingAir-3
6.2 Real-World Applications
Next, we examined SFTL in four real-world datasets: FitRecord, ServerRoom, BeijingAir-2, and BeijingAir-3. We tested 11 competing approaches. We compared with state-of-the-art streaming tensor decomposition methods based on the CP or Tucker model, including (1) POST [Du et al., 2018], (2) BASS-Tucker [Fang et al., 2021a] and (3) ADF-CP [Wang and Zhe, 2019], the state-of-the-art static decomposition algorithms, including (4) P-Tucker [Oh et al., 2018], (5) CP-ALS and (6) Tucker-ALS [Bader and Kolda, 2008]. For those methods, we augment the tensor with a time mode, and convert the ordered, unique timestamps into increasing time steps. We also compared with the most recent continuous-time decomposition methods. (7) CT-CP [Zhang et al., 2021], (8) CT-GP, (9) BCTT [Fang et al., 2022], (10) THIS-ODE [Li et al., 2022], and (11) NONFAT [Wang et al., 2022], nonparametric factor trajectory learning, the only existing work that also estimates factor trajectories for temporal tensor decomposition. Note that the methods 4-11 cannot handle data streams. They have to iteratively access the data to update the model parameters and factor estimates. The details about the competing methods and datasets are provided in Appendix (Section E).
For all the competing methods, we used the publicly released implementations of the original authors. The hyper-parameter setting and turning follows the original papers. For SFTL, we chose  $\nu$  from  $\left\{\frac{1}{2}, \frac{3}{2}\right\}$ ,  $a$  from [0.5, 1] and  $\rho$  from [0.1, 0.5]. For our online filtering, the maximum number of CEP iterations was set to 50 and the tolerance level to  $10^{-4}$ . For numerical stability, we re-scaled the timestamps to [0, 1]. We examined the number of factors (or factor trajectories)  $R \in \{2, 3, 5, 7\}$ .
Final Prediction Accuracy. We first examined the final prediction accuracy with our learned factor trajectories. To this end, we followed [Xu et al., 2012, Kang et al., 2012], and randomly sampled  $80\%$  observed entry values and their timestamps for streaming inference and then tested the prediction error on the remaining entries. We also compared with the static decomposition methods, which need to repeatedly access the training entries. We repeated the experiment five times, and computed the average root mean-square-error (RMSE), average mean-absolute-error (MAE), and their standard deviations. We ran our method based on both the CP and Tucker forms, denoted by SFTL-CP and SFTL-Tucker, respectively. We report the results for  $R = 5$  in Table 1. Due to the space limit, we leave the other results in the Appendix (Table 4,5, and 6). As we can see, SFTL outperforms all the streaming approaches by a large margin. SFTL even obtains significantly better prediction accuracy than all the static decomposition approaches, except that on Server Room, SFTL is second to THIS-ODE and NONFAT. Note that SFTL only went through the training entries for once. Although NONFAT can also estimate the factor trajectories, it uses GPs to perform black-box nonlinear decomposition and hence loses the interpretability. Note that NONFAT in most case also outperforms the other static decomposition methods that only estimate time-invariant factors. The superior performance of SFTL and NONFAT shows the importance of capturing factor evolution.
Online Predictive Performance. Next, we evaluated the online predictive performance of SFTL. Whenever a batch of entries at a new timestamp has been processed, we examined the prediction accuracy on the test set, with our current estimate of the factor trajectories. We repeated the evaluation for five times, and examine how the average prediction error varies along with the number of processed entries. We show the results for  $R = 5$  in Fig. 3, and the others in Appendix (Fig. 5, 6, and 7). It is clear that SFTL in most cases outperforms the competing streaming decomposition algorithms by a large margin throughout the course of running. Note that the online behavior of BASS-Tucker was quite unstable and so we excluded it in the figures. It confirms the advantage of our streaming trajectory learning approach — even in the streaming scenario, incrementally capturing the time-variation of the factors can perform better than updating fixed, static factors. To confirm the advantage of SFTL in computational efficiency, we report the running time in Section G of Appendix.
(e)  $u_{2,2}^{2}(t)$
(f)  $u_{2,3}^{2}(t)$
(d)  $u_{2,1}^{2}(t)$
(g)  $u_{3,1}^{3}(t)$
Figure 4: The learned factor trajectories of object 1, 2, 3 in mode 1, 2, 3, respectively, from ServerRoom data. The shaded region shows one standard deviation and the dashed line indicates the biggest timestamp in the data.
(h)  $u_{3,2}^{3}(t)$
(i)  $u_{3,3}^{3}(t)$
Investigation of Learning Results. Finally, we investigated our learned factor trajectories. We set  $R = 3$  and ran SFTL-CP on ServerRoom. In Fig. 4, we visualize the trajectory for the 1st air conditioning mode, the 2nd power usage level, and the 3rd location. The shaded region indicates the standard deviation, and the red dashed line the biggest timestamp in the data. First, we can see that the posterior variance of all the trajectories grow quickly when moving to the right of the red line. This is reasonable because it is getting far away from the training region. Second, for each object, the first and second trajectories (1st and 2nd column in Fig. 4) exhibit quite different time-varying patterns, e.g., the local periodicity in  $u_{1,2}^{1}(t)$  and  $u_{2,2}^{2}(t)$ , the gradual decreasing and increasing trends in  $u_{3,1}^{3}(t)$  and  $u_{3,2}^{3}(t)$ , respectively, which imply different inner properties of the object. Third, it is particularly interesting to see that the third trajectory for all the objects appear to be close to zero, with relatively large posterior variance all the time. This might imply that two factor trajectories have been sufficient to represent each object. Requesting for a third trajectory is redundant. More important, our model is empirically able to detect such redundancy and returns a zero-valued trajectory.
We have presented SFTL, a probabilistic temporal tensor decomposition approach. SFTL can efficiently handle streaming data, and estimate time-varying factor representations. On four real-world applications, SFTL achieves superior online and final prediction accuracy.
This work has been supported by MURI AFOSR grant FA9550-20-1-0358 and NSF CAREER Award IIS-2046295.
Yanqing Zhang, Xuan Bi, Niansheng Tang, and Annie Qu. Dynamic tensor recommender systems. Journal of Machine Learning Research, 22(65):1-35, 2021.
Shandian Zhe and Yishuai Du. Stochastic nonparametric event-tensor decomposition. In Advances in Neural Information Processing Systems, pages 6856-6866, 2018.
Shandian Zhe, Zenglin Xu, Xinqi Chu, Yuan Qi, and Youngja Park. Scalable nonparametric multiway data analysis. In Artificial Intelligence and Statistics, pages 1125-1134. PMLR, 2015.
Shandian Zhe, Yuan Qi, Youngja Park, Zenglin Xu, Ian Molloy, and Suresh Chari. Dintucker: Scaling up Gaussian process models on large multidimensional arrays. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016a.
Shandian Zhe, Kai Zhang, Pengyuan Wang, Kuang-chih Lee, Zenglin Xu, Yuan Qi, and Zoubin Ghahramani. Distributed flexible nonlinear tensor factorization. Advances in neural information processing systems, 29, 2016b.
A Spectral Analysis and LTI-SDE
We consider the Matérn kernel family,
$$
\kappa_ {\nu} \left(t, t ^ {\prime}\right) = a \frac {\left(\frac {\sqrt {2 \nu}}{\rho} \Delta\right) ^ {\nu}}{\Gamma (\nu) 2 ^ {\nu - 1}} K _ {\nu} \left(\frac {\sqrt {2 \nu}}{\rho} \Delta\right), \tag {10}
$$
where  $\Delta = |t - t'|, \Gamma(\cdot)$  is the Gamma function,  $a > 0$  and  $\rho > 0$  are the amplitude and length-scale parameters, respectively,  $K_{\nu}$  is the modified Bessel function of the second kind,  $\nu > 0$  controls the smoothness. Since  $\kappa_{\nu}$  is a stationary kernel, i.e.,  $\kappa_{\nu}(t,t') = \kappa_{\nu}(t - t')$ , according to the Wiener-Khinchin theorem [Chatfield, 2003], if
$$
f (t) \sim \mathcal {G P} (0, \kappa_ {\nu} (t, t ^ {\prime})),
$$
the energy spectrum density of  $f(t)$  can be obtained by the Fourier transform of  $\kappa_{\nu}(\Delta)$ ,
$$
S (\omega) = a \frac {2 \sqrt {\pi} \Gamma \left(\frac {1}{2} + \nu\right)}{\Gamma (\nu)} \alpha^ {2 \nu} \left(\alpha^ {2} + \omega^ {2}\right) ^ {- \left(\nu + \frac {1}{2}\right)}, \tag {11}
$$
where  $\omega$  is the frequency, and  $\alpha = \frac{\sqrt{2}\nu}{\rho}$ . We consider the commonly used choice  $\nu = p + \frac{1}{2}$  where  $p \in \{0, 1, 2, \ldots\}$ . Then we can observe that
$$
S (\omega) = \frac {\sigma^ {2}}{(\alpha^ {2} + \omega^ {2}) ^ {p + 1}} = \frac {\sigma^ {2}}{(\alpha + i \omega) ^ {p + 1} (\alpha - i \omega) ^ {p + 1}}, \tag {12}
$$
where  $\sigma^2 = a\frac{2\sqrt{\pi}\Gamma(p + 1)}{\Gamma(p + \frac{1}{2})}\alpha^{2p + 1}$ , and  $i$  indicates an imaginary number. We expand the polynomial
$$
(\alpha + i \omega) ^ {p + 1} = \sum_ {k = 0} ^ {p} c _ {k} (i \omega) ^ {k} + (i \omega) ^ {p + 1}, \tag {13}
$$
where  $\{c_k|0\leq k\leq p\}$  are the coefficients. From (12) and (13), we can construct an equivalent system to generate the signal  $f(t)$ . That is, in the frequency domain, the system output's Fourier transform  $\widehat{f} (\omega)$  is given by
$$
\sum_ {k = 1} ^ {p} c _ {k} (i \omega) ^ {k} \widehat {f} (\omega) + (i \omega) ^ {p + 1} \widehat {f} (\omega) = \widehat {\beta} (\omega), \tag {14}
$$
where  $\widehat{\beta}$  is the Fourier transform of a white noise process  $\beta(t)$  with spectral density (or diffusion)  $\sigma^2$ . The reason is that by construction,  $\widehat{f}(\omega) = \frac{\widehat{\beta}(\omega)}{(\alpha + i\omega)^{p+1}}$ , which gives exactly the same spectral density as in (12),  $S(\omega) = |\widehat{f}(\omega)|^2$ . We then conduct inverse Fourier transform on both sides of (14) to obtain the representation in the time domain,