SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {k = 1} ^ {p} c _ {k} \frac {\mathrm {d} ^ {k} f}{\mathrm {d} t ^ {k}} + \frac {\mathrm {d} ^ {p + 1} f}{\mathrm {d} t ^ {p + 1}} = \beta (t), \tag {15}
$$
which is an SDE. Note that  $\beta(t)$  has the density  $\sigma^2$ . We can further construct a new state  $\mathbf{z} = (f, f^{(1)}, \ldots, f^{(p)})^\top$  (where each  $f^{(k)} \triangleq \mathrm{d}^k f / \mathrm{d}t^k$ ) and convert (15) into a linear time-invariant (LTI) SDE,
$$
\frac {\mathrm {d} \mathbf {z}}{\mathrm {d} t} = \mathbf {A} \mathbf {z} + \boldsymbol {\eta} \cdot \beta (t), \tag {16}
$$
For a concrete example, if we take  $p = 1$  (and so  $\nu = \frac{3}{2}$ ), then  $\mathbf{A} = [0,1; -\alpha^2, -2\alpha]$ ,  $\pmb{\eta} = [0;1]$ , and  $\sigma^2 = 4a\alpha^3$ .
The LTI-SDE is particularly useful in that its finite set of states follow a Gauss-Markov chain, namely the state-space prior. Specifically, given arbitrary  $t_1 < \ldots < t_L$ , we have
$$
p (\mathbf {z} (t _ {1}), \ldots , \mathbf {z} (t _ {L})) = p (\mathbf {z} (t _ {1})) \prod_ {k = 1} ^ {L - 1} p (\mathbf {z} (t _ {k + 1}) | \mathbf {z} (t _ {k})),
$$
where  $p(\mathbf{z}(t_1)) = \mathcal{N}(\mathbf{z}(t_1)|\mathbf{0},\mathbf{P}_\infty)$ ,  $p(\mathbf{z}(t_{k + 1})|\mathbf{z}(t_k)) = \mathcal{N}(\mathbf{z}(t_{k + 1})|\mathbf{F}_k\mathbf{z}(t_k),\mathbf{Q}_k)$ ,  $\mathbf{P}_{\infty}$  is the stationary covariance matrix computed by solving the matrix Riccati equation [Lancaster and Rodman, 1995],  $\mathbf{F}_n = \exp (\Delta_k\cdot \mathbf{A})$  where  $\Delta_{k} = t_{k + 1} - t_{k}$ , and  $\mathbf{Q}_k = \mathbf{P}_{\infty} - \mathbf{A}_k\mathbf{P}_{\infty}\mathbf{A}_k^\top$ . Therefore, we do not need the full covariance matrix as in the standard GP prior, and the computation is much more efficient. The chain structure is also convenient to handle streaming data as explained in the main paper.
Note that for other type of kernel functions, such as the square exponential (SE) kernel, we can approximate the inverse spectral density  $1 / S(\omega)$  with a polynomial of  $\omega^2$  with negative roots, and follow the same way to construct an LTI-SDE (approximation) and state-space prior.
B RTS Smoother
Consider a standard state-space model with state  $\mathbf{x}_n$  and observation  $\mathbf{y}_n$  at each time step  $n$ . The prior distribution is a Gauss-Markov chain,
$$
p \left(\mathbf {x} _ {n + 1} \mid \mathbf {x} _ {n}\right) = \mathcal {N} \left(\mathbf {x} _ {n + 1} \mid \mathbf {A} _ {n} \mathbf {x} _ {n}, \mathbf {Q} _ {n}\right),
$$
$$
p (\mathbf {x} _ {0}) = \mathcal {N} (\mathbf {x} _ {0} | \mathbf {m} _ {0}, \mathbf {P} _ {0}).
$$
Suppose we have a Gaussian observation likelihood,
$$
p (\mathbf {y} _ {n} | \mathbf {x} _ {n}) = \mathcal {N} (\mathbf {y} _ {n} | \mathbf {H} _ {n} \mathbf {x} _ {n}, \mathbf {W} _ {n}).
$$
Then upon receiving each  $\mathbf{y}_n$ , we can use Kalman filtering to obtain the exact running posterior,
$$
p \left(\mathbf {x} _ {n} \mid \mathbf {y} _ {1: n}\right) = \mathcal {N} \left(\mathbf {x} _ {n} \mid \mathbf {m} _ {k}, \mathbf {P} _ {k}\right),
$$
which is a Gaussian. After all the data has been processed — suppose it ends after step  $N$  — we can use Rauch-Tung-Striebel (RTS) smoother [Särkkä, 2013] to efficiently compute the full posterior of each state from backward, which does not need to re-access any data:  $p(\mathbf{x}_n|\mathbf{y}_{1:N}) = \mathcal{N}(\mathbf{x}_n|\mathbf{m}_n^s,\mathbf{P}_n^s)$ , where
$$
\mathbf {m} _ {n + 1} ^ {-} = \mathbf {A} _ {n} \mathbf {m} _ {n}, \quad \mathbf {P} _ {n + 1} ^ {-} = \mathbf {A} _ {n} \mathbf {P} _ {n} \mathbf {A} _ {n} ^ {\top} + \mathbf {Q} _ {n},
$$
$$
\mathbf {G} _ {n} = \mathbf {P} _ {n} \mathbf {A} _ {n} ^ {\top} [ \mathbf {P} _ {n + 1} ^ {-} ] ^ {- 1},
$$
$$
\mathbf {m} _ {n} ^ {s} = \mathbf {m} _ {n} + \mathbf {G} _ {n} \left(\mathbf {m} _ {n + 1} ^ {s} - \mathbf {m} _ {n + 1} ^ {-}\right),
$$
$$
\mathbf {P} _ {n} ^ {s} = \mathbf {P} _ {n} + \mathbf {G} _ {n} \left[ \mathbf {P} _ {n + 1} ^ {s} - \mathbf {P} _ {n + 1} ^ {-} \right] \mathbf {G} _ {n} ^ {\top}. \tag {17}
$$
As we can see, the computation only needs the running posterior  $p(\mathbf{x}_n|\mathbf{y}_{1:n}) = \mathcal{N}(\cdot |\mathbf{m}_n,\mathbf{P}_n)$  and the full posterior of the next state  $p(\mathbf{x}_{n + 1}|\mathbf{y}_{1:N}) = \mathcal{N}(\cdot |\mathbf{m}_{n + 1},\mathbf{P}_{n + 1})$ . It does not need to revisit previous observations  $\mathbf{y}_{1:N}$
C Details about Online Trajectory Inference
In this section, we provide the details about how to update the running posterior according to (8) and (9) (in the main paper) with the conditional EP (CEP) framework [Wang and Zhe, 2019].
C.1 EP and CEP framework
We first give a brief introduction to the EP and CEP framework. Consider a general probabilistic model with latent parameters  $\theta$ . Given the observed data  $\mathcal{D} = \{\mathbf{y}_1, \dots, \mathbf{y}_N\}$ , the joint probability distribution is
$$
p (\boldsymbol {\theta}, \mathcal {D}) = p (\boldsymbol {\theta}) \prod_ {n = 1} ^ {N} p \left(\mathbf {y} _ {n} \mid \boldsymbol {\theta}\right). \tag {18}
$$
Our goal is to compute the posterior  $p(\pmb{\theta}|\mathcal{D})$ . However, it is usually infeasible to compute the exact marginal distribution  $p(\mathcal{D})$ , because of the complexity of the likelihood and/or prior. EP therefore seeks to approximate each term in the joint probability by an exponential-family term,
$$
p \left(y _ {n} \mid \boldsymbol {\theta}\right) \approx c _ {n} f _ {n} (\boldsymbol {\theta}), \quad p (\boldsymbol {\theta}) \approx c _ {0} f _ {0} (\boldsymbol {\theta}), \tag {19}
$$
where  $c_{n}$  and  $c_{0}$  are constants to ensure the normalization consistency (they will get canceled in the inference, so we do not need to calculate them), and
$$
f _ {n} (\boldsymbol {\theta}) \propto \exp \left(\lambda_ {n} ^ {\top} \phi (\boldsymbol {\theta})\right), (0 \leq n \leq N)
$$
where  $\lambda_{n}$  is the natural parameter and  $\phi (\pmb {\theta})$  is sufficient statistics. For example, if we choose a Gaussian term,  $f_{n} = \mathcal{N}(\pmb {\theta}|\pmb {\mu}_{n},\pmb{\Sigma}_{n})$ , then the sufficient statistics is  $\phi (\pmb {\theta}) = \{\pmb {\theta},\pmb {\theta}\pmb{\theta}^{\top}\}$ . The moment is the expectation of the sufficient statistics.
We therefore approximate the joint probability with
$$
p (\boldsymbol {\theta}, \mathcal {D}) = p (\boldsymbol {\theta}) \prod_ {n = 1} ^ {N} p (\mathbf {y} _ {n} | \boldsymbol {\theta}) \approx f _ {0} (\boldsymbol {\theta}) \prod_ {n = 1} ^ {N} f _ {n} (\boldsymbol {\theta}) \cdot \text {c o n s t .} \tag {20}
$$
Because the exponential family is closed under product operations, we can immediately obtain a closed-form approximate posterior  $q(\pmb{\theta}) \approx p(\pmb{\theta}|\mathcal{D})$  by merging the approximation terms in the R.H.S. of (20), which is still a distribution in the exponential family.
Then the task amounts to optimizing those approximation terms  $\{f_n(\pmb {\theta})|0\leq n\leq N\}$ . EP repeatedly conducts four steps to optimize each  $f_{n}$
- Step 1. We obtain the calibrated distribution that integrates the context information of  $f_{n}$ ,
$$
q ^ {\backslash n} (\boldsymbol {\theta}) \propto \frac {q (\boldsymbol {\theta})}{f _ {n} (\boldsymbol {\theta})},
$$
where  $q(\pmb{\theta})$  is the current posterior approximation.
- Step 2. We construct a tilted distribution to combine the true likelihood,
$$
\widetilde {p} (\boldsymbol {\theta}) \propto q ^ {\setminus n} (\boldsymbol {\theta}) \cdot p (\mathbf {y} _ {n} | \boldsymbol {\theta}).
$$
Note that if  $n = 0$ , we have  $\widetilde{p}(\boldsymbol{\theta}) \propto q^{n}(\boldsymbol{\theta}) \cdot p(\boldsymbol{\theta})$ .
- Step 3. We project the tilted distribution back to the exponential family,
$$
q ^ {*} (\boldsymbol {\theta}) = \operatorname * {a r g m i n} _ {q} \mathrm {K L} (\widetilde {p} \| q)
$$
where  $q$  belongs to the exponential family. This can be done by moment matching,
$$
\mathbb {E} _ {q ^ {*}} [ \phi (\boldsymbol {\theta}) ] = \mathbb {E} _ {\widetilde {p}} [ \phi (\boldsymbol {\theta}) ]. \tag {21}
$$
That is, we compute the expected moment under  $\widetilde{p}$ , with which to obtain the parameters of  $q^{*}$ . For example, if  $q^{*}(\theta)$  is a Gaussian distribution, then we need to compute  $\mathbb{E}_{\widetilde{p}}[\theta]$  and  $\mathbb{E}_{\widetilde{p}}[\theta \theta^{\top}]$ , with which to obtain the mean and covariance for  $q^{*}(\theta)$ . Hence we obtain  $q^{*}(\theta) = \mathcal{N}(\theta | \mathbb{E}_{\widetilde{p}}[\theta], \mathbb{E}_{\widetilde{p}}[\theta \theta^{\top}] - \mathbb{E}_{\widetilde{p}}[\theta] \mathbb{E}_{\widetilde{p}}[\theta]^{\top})$
- Step 4. We update the approximation term by
$$
f _ {n} (\boldsymbol {\theta}) \approx \frac {q ^ {*} (\boldsymbol {\theta})}{q ^ {\backslash} (\boldsymbol {\theta})}. \tag {22}
$$
In practice, EP often updates all the  $f_{n}$ 's in parallel, and uses damping to avoid divergence. It iteratively runs the four steps until convergence. In essence, this is a fixed point iteration to optimize a free energy function (a mini-max problem) [Minka, 2001].
The critical step in EP is the moment matching (21). However, in many cases, it is analytically intractable to compute the moment under the tilted distribution  $\widetilde{p}$ , due to the complexity of the likelihood. To address this problem, CEP considers the commonly used case that each  $f_{n}$  has a factorized structure,
$$
f _ {n} (\boldsymbol {\theta}) = \prod_ {m} f _ {n m} \left(\boldsymbol {\theta} _ {m}\right), \tag {23}
$$
where each  $f_{nm}$  is also in the exponential family, and  $\{\pmb{\theta}_m\}$  are mutually disjoint. Then at the moment matching step, we need to compute the moment of each  $\pmb{\theta}_m$  under  $\widetilde{p}$ , i.e.,  $\mathbb{E}_{\widetilde{p}}[\phi(\pmb{\theta}_m)]$ . The first key idea of CEP is to use the nested structure,
$$
\mathbb {E} _ {\widetilde {p}} [ \phi (\boldsymbol {\theta} _ {m}) ] = \mathbb {E} _ {\widetilde {p} (\boldsymbol {\theta} _ {\backslash m})} \mathbb {E} _ {\widetilde {p} (\boldsymbol {\theta} _ {m} | \boldsymbol {\theta} _ {\backslash m})} [ \phi (\boldsymbol {\theta} _ {m}) ], \tag {24}
$$
where  $\theta_{\backslash m} = \theta \backslash \theta_{m}$ . Therefore, we can first compute the inner expectation, i.e., conditional moment,
$$
\mathbb {E} _ {\widetilde {p} \left(\boldsymbol {\theta} _ {m} \mid \boldsymbol {\theta} _ {\backslash m}\right)} [ \phi (\boldsymbol {\theta} _ {m}) ] = \mathbf {g} (\boldsymbol {\theta} _ {\backslash m}), \tag {25}
$$
and then seek for computing the outer expectation,  $\mathbb{E}_{\widetilde{p} (\theta_{\setminus m})}\big[\mathbf{g}(\theta_{\setminus m})\big]$ . The inner expectation is often easy to compute (e.g., with our CP/Tucker likelihood). When  $f_{n}$  is factorized individually over each element of  $\theta$ , this can always be efficiently and accurately calculated by quadrature. However, the outer expectation is still difficult to obtain because  $\widetilde{p} (\theta_{\setminus m})$  is intractable. The second key idea of CEP is that since the moment matching is also between  $q(\theta_{\setminus m})$  and  $\widetilde{p} (\theta_{\setminus m})$ , we can use the current marginal posterior to approximate the marginal titled distribution and then compute the outer expectation,
$$
\mathbb {E} _ {\widetilde {p} (\boldsymbol {\theta} _ {\backslash m})} [ \mathbf {g} (\boldsymbol {\theta} _ {\backslash m}) ] \approx \mathbb {E} _ {q (\boldsymbol {\theta} _ {\backslash m})} [ \mathbf {g} (\boldsymbol {\theta} _ {\backslash m}) ]. \tag {26}
$$
If it is still analytically intractable, we can use the delta method [Oehlert, 1992] to approximate the expectation. That is, we use a Taylor expansion of  $\mathbf{g}(\cdot)$  at the mean of  $\theta_{\backslash m}$ . Take the first-order expansion as an example,
$$
\mathbf {g} (\boldsymbol {\theta} _ {\backslash m}) \approx \mathbf {g} \left(\mathbb {E} _ {q (\boldsymbol {\theta} _ {\backslash m})} [ \boldsymbol {\theta} _ {\backslash m} ]\right) + \mathbf {J} \left(\boldsymbol {\theta} _ {\backslash m} - \mathbb {E} _ {q (\boldsymbol {\theta} _ {\backslash m})} [ \boldsymbol {\theta} _ {\backslash m} ]\right)
$$
where  $\mathbf{J}$  is the Jacobian of  $\mathbf{g}$  at  $\mathbb{E}_{q(\theta_{\backslash m})}[\theta_{\backslash m}]$ . Then we take the expectation on the Taylor approximation instead,
$$
\mathbb {E} _ {q (\boldsymbol {\theta} \backslash m)} [ \mathbf {g} (\boldsymbol {\theta} \backslash m) ] \approx \mathbf {g} \left(\mathbb {E} _ {q (\boldsymbol {\theta} \backslash m)} [ \boldsymbol {\theta} \backslash m ]\right). \tag {27}
$$
The above computation is very convenient to implement. Once we obtain the conditional moment  $\mathbf{g}(\pmb{\theta}_{\backslash m})$ , we simply replace the  $\pmb{\theta}_{\backslash m}$  by its expectation under current posterior approximation  $q$ , i.e.,  $\mathbb{E}_{q(\pmb{\theta}_{\backslash m})}[\pmb{\theta}_{\backslash m}]$ , to obtain the matched moment  $\mathbf{g}(\mathbb{E}_{q(\pmb{\theta}_{\backslash m})}[\pmb{\theta}_{\backslash m}])$ , with which to construct  $q^{*}$  in Step 3 of EP (see (21)). The remaining steps are the same.
C.2 Running Posterior Update
Now we use the CEP framework to update the running posterior  $p(\Theta_{n+1}, \tau | \mathcal{D}_{t_{n+1}})$  in (8) via the approximation (9). To simplify the notation, let us define  $\mathbf{v}_{l_m}^m \triangleq \mathbf{u}_{\ell_m}^m(t_{n+1})$ , and hence for each  $(\ell, y) \in \mathcal{B}_{n+1}$ , we approximate
$$
\mathcal {N} \left(y | \mathbf {1} ^ {\top} \left(\mathbf {v} _ {\ell_ {1}} ^ {1} \circ \dots \circ \mathbf {v} _ {\ell_ {M}} ^ {M}\right), \tau^ {- 1}\right) \approx \prod_ {m = 1} ^ {M} \mathcal {N} \left(\mathbf {v} _ {\ell_ {m}} ^ {m} \mid \boldsymbol {\gamma} _ {\ell_ {m}} ^ {m}, \boldsymbol {\Sigma} _ {\ell_ {m}} ^ {m}\right) \operatorname {G a m} (\tau | \alpha_ {\ell}, \omega_ {\ell}). \tag {28}
$$
If we substitute (9) into (8), we can immediately obtain a Gaussian posterior approximation of each  $\mathbf{v}_{\ell_m}^m$  and a Gamma posterior approximation of the noise inverse variance  $\tau$ . Then dividing the current posterior approximation with the R.H.S of (28), we can obtain the calibrated distribution,
$$
q ^ {\backslash \ell} (\mathbf {v} _ {\ell_ {m}} ^ {m}) = \mathcal {N} (\mathbf {v} _ {\ell_ {m}} ^ {m} | \boldsymbol {\beta} _ {\ell_ {m}} ^ {m}, \boldsymbol {\Omega} _ {\ell_ {m}} ^ {m}),
$$
$$
q ^ {\backslash \ell} (\tau) = \operatorname {G a m} \left(\tau \mid \alpha^ {\backslash \ell}, \omega^ {\backslash \ell}\right), \tag {29}
$$
where  $1 \leq m \leq M$ . Next, we construct a tilted distribution,
$$
\widetilde {p} \left(\mathbf {v} _ {\ell_ {1}} ^ {1}, \dots , \mathbf {v} _ {\ell_ {M}} ^ {M}, \tau\right) \propto q ^ {\backslash \boldsymbol {\ell}} (\tau) \cdot \prod_ {m = 1} ^ {M} q ^ {\backslash \boldsymbol {\ell}} \left(\mathbf {v} _ {\ell_ {m}} ^ {m}\right) \cdot \mathcal {N} \left(y | \mathbf {1} ^ {\top} \left(\mathbf {v} _ {\ell_ {1}} ^ {1} \circ \dots \circ \mathbf {v} _ {\ell_ {M}} ^ {M}\right), \tau^ {- 1}\right). \tag {30}
$$
To update each  $\mathcal{N}(\mathbf{v}_{\ell_m}^m |\gamma_{\ell_m}^m,\Sigma_{\ell_m}^m)$  in (28), we first look into the conditional tilted distribution,
$$
\widetilde {p} \left(\mathbf {v} _ {\ell_ {m}} ^ {m} \mid \mathcal {V} _ {\ell} ^ {\backslash m}, \tau\right) \propto \mathcal {N} \left(\mathbf {v} _ {\ell_ {m}} ^ {m} \mid \beta_ {\ell_ {m}} ^ {m}, \boldsymbol {\Omega} _ {\ell_ {m}} ^ {m}\right) \cdot \mathcal {N} \left(y \mid \left(\mathbf {v} _ {\ell_ {m}} ^ {m}\right) ^ {\top} \mathbf {v} _ {\ell} ^ {\backslash m}, \tau^ {- 1}\right) \tag {31}
$$
where  $\mathcal{V}_{\ell}^{\backslash m}$  is  $\{\mathbf{v}_{\ell_j}^j |1\leq j\leq M,j\neq m\}$ , and
$$
\mathbf {v} _ {\ell} ^ {\backslash m} = \mathbf {v} _ {\ell_ {1}} ^ {1} \circ \dots \circ \mathbf {v} _ {\ell_ {m - 1}} ^ {m - 1} \circ \mathbf {v} _ {\ell_ {m + 1}} ^ {m + 1} \circ \dots \circ \mathbf {v} _ {\ell_ {M}} ^ {M}.
$$
The conditional tilted distribution is obviously Gaussian, and the conditional moment is straightforward to obtain,
$$
\mathbf {S} \left(\mathbf {v} _ {\ell m} ^ {m} \mid \mathcal {V} _ {\ell} ^ {\backslash m}, \tau\right) = \left[ \boldsymbol {\Omega} _ {\ell m} ^ {m} ^ {- 1} + \tau \mathbf {v} _ {\ell} ^ {\backslash m} \left(\mathbf {v} _ {\ell} ^ {\backslash m}\right) ^ {\top} \right] ^ {- 1}, \tag {32}
$$
$$
\mathbb {E} \left[ \mathbf {v} _ {\ell_ {m}} ^ {m} \mid \mathcal {V} _ {\ell} ^ {\backslash m}, \tau \right] = \mathbf {S} \left(\mathbf {v} _ {\ell_ {m}} ^ {m} \mid \mathcal {V} _ {\ell} ^ {\backslash m}, \tau\right) \cdot \left(\Omega_ {\ell_ {m}} ^ {m} ^ {- 1} \boldsymbol {\beta} _ {\ell_ {m}} ^ {m} + \tau y \mathbf {v} _ {\ell} ^ {\backslash m}\right), \tag {33}
$$
where  $\mathbf{S}$  denotes the conditional covariance. Next, according to (27), we simply replace  $\tau$ ,  $\mathbf{v}_{\ell}^{\backslash m}$ , and  $\mathbf{v}_{\ell}^{\backslash m}\left(\mathbf{v}_{\ell}^{\backslash m}\right)^{\top}$  by their expectation under the current posterior  $q$  in (32) and (33), to obtain the moments, i.e., the mean and covariance matrix, with which we can construct  $q^{*}$  in Step 3 of the EP framework. The computation of  $\mathbb{E}_q[\tau]$  is straightforward, and
$$
\mathbb {E} _ {q} [ \mathbf {v} _ {\ell} ^ {\backslash m} ] = \mathbb {E} _ {q} [ \mathbf {v} _ {\ell_ {1}} ^ {1} ] \circ \ldots \circ \mathbb {E} _ {q} [ \mathbf {v} _ {\ell_ {m - 1}} ^ {m - 1} ] \circ \mathbb {E} _ {q} [ \mathbf {v} _ {\ell_ {m + 1}} ^ {m + 1} ] \circ \ldots \circ \mathbb {E} _ {q} [ \mathbf {v} _ {\ell_ {M}} ^ {M} ],
$$
$$
\mathbb {E} _ {q} [ \mathbf {v} _ {\boldsymbol {\ell}} ^ {\backslash m} (\mathbf {v} _ {\boldsymbol {\ell}} ^ {\backslash m}) ^ {\top} ] = \mathbb {E} _ {q} [ \mathbf {v} _ {\ell_ {1}} ^ {1} (\mathbf {v} _ {\ell_ {1}} ^ {1}) ^ {\top} ] \circ \ldots \circ \mathbb {E} _ {q} [ \mathbf {v} _ {\ell_ {m - 1}} ^ {m - 1} (\mathbf {v} _ {\ell_ {m - 1}} ^ {m - 1}) ^ {\top} ]
$$
$$
\circ \mathbb {E} _ {q} \left[ \mathbf {v} _ {\ell_ {m + 1}} ^ {m + 1} \left(\mathbf {v} _ {\ell_ {m + 1}} ^ {m + 1}\right) ^ {\top} \right] \circ \dots \circ \mathbb {E} _ {q} \left[ \mathbf {v} _ {\ell_ {M}} ^ {M} \left(\mathbf {v} _ {\ell_ {M}} ^ {M}\right) ^ {\top} \right].
$$
Similarly, to update  $\mathrm{Gam}(\alpha_{\ell},\omega_{\ell})$  in (28), we first observe that the conditional titled distribution is also a Gamma distribution,
$$
\widetilde {p} (\tau | \mathcal {V} _ {\ell}) \propto \operatorname {G a m} (\tau | \widetilde {\alpha}, \widetilde {\omega}) \propto \operatorname {G a m} (\tau | \alpha^ {\setminus \ell}, \omega^ {\setminus \ell}) \mathcal {N} (y | \mathbf {1} ^ {\top} \mathbf {v} _ {\ell}, \tau^ {- 1}), \tag {34}
$$
where  $\mathbf{v}_{\boldsymbol{\ell}} = \mathbf{v}_{\ell_1}^1\circ \dots \circ \mathbf{v}_{\ell_M}^M$  , and
$$
\widetilde {\alpha} = \alpha^ {\backslash \ell} + \frac {1}{2},
$$
$$
\widetilde {\omega} = \omega^ {\setminus \ell} + \frac {1}{2} y ^ {2} + \frac {1}{2} \mathbf {1} ^ {\top} \mathbf {v} _ {\ell} \mathbf {v} _ {\ell} ^ {\top} \mathbf {1} - y \mathbf {1} ^ {\top} \mathbf {v}. \tag {35}
$$
Since the conditional moments (the expectation of  $\tau$  and  $\log \tau$ ) are functions of  $\alpha$  and  $\omega$ , when using the delta method to approximate the expected conditional moment, it is equivalent to approximating the expectation of  $\widetilde{\alpha}$  and  $\widetilde{\omega}$  first, and then use the expected  $\widetilde{\alpha}$  and  $\widetilde{\omega}$  to recover the moments. As a result, we can simply replace  $\mathbf{v}_{\ell}$  and  $\mathbf{v}_{\ell} \mathbf{v}_{\ell}^{\top}$  in (35) by their expectation under the current posterior, and we obtain the approximation of  $\mathbb{E}_q[\widetilde{\alpha}]$  and  $\mathbb{E}_q[\widetilde{\omega}]$ . With these approximated expectations, we then construct  $q^{*}(\tau) = \mathrm{Gam}(\tau |\mathbb{E}_{q}[\alpha],\mathbb{E}_{q}[\omega])$  at Step 3 in EP. The remaining steps are straightforward. The running posterior update with the Tucker form likelihood follows a similar way.
D More Results on Simulation Study
D.1 Accuracy of Trajectory Recovery
We provide the quantitative result in recovering the factor trajectories. Note that there is only one competing method, NONFAT, which can also estimate factor trajectories. We therefore ran our method and NONFAT on the synthetic dataset. We then randomly sampled 500 time points in the domain and evaluate the RMSE of the learned factor trajectories for each method. As shown in Table 2, the RMSE of NONFAT on recovering  $u_{1}^{1}(t)$  and  $u_{1}^{2}(t)$  is close to SFTL, showing NONFAT achieved the same (or very close) quality in recovering these two trajectories. However, on  $u_{2}^{1}(t)$  and  $u_{2}^{2}(t)$ , the RMSE of NONFAT is much larger, showing that NONFAT have failed to capture the other two trajectories. By contrast, SFTL consistently well recovered them.
D.2 Sensitive Analysis on Kernel Parameters
To examine the sensitivity to the kernel parameters, we used the synthetic dataset, and randomly sampled 100 entries and new timestamps for evaluation. We then examined the length-scale  $\rho$  and
u1(1)(t)	u2(1)(t)	u12(t)	u22(t)
SFTL	0.073	0.082	0.103	0.054
NONFAT	0.085	0.442	0.096	0.443
Table 2: RMSE in recovering trajectories on the simulation data.
ρ	0.1	0.3	0.5	0.7	0.9
Matérn-1/2	SFTL-CP	0.091	0.064	0.059	0.056	0.057
SFTL-Tucker	0.060	0.055	0.056	0.056	0.057
Matérn-3/2	SFTL-CP	0.062	0.061	0.074	0.093	0.112
SFTL-Tucker	0.061	0.059	0.078	0.101	0.129
(a) Prediction RMSE with  $a = 0.3$  and varying  $\rho$ .
a	0.1	0.3	0.5	0.7	0.9
Matérn-1/2	SFTL-CP	0.056	0.064	0.057	0.059	0.063
SFTL-Tucker	0.065	0.055	0.054	0.055	0.055
Matérn-3/2	SFTL-CP	0.072	0.061	0.063	0.060	0.059
SFTL-Tucker	0.098	0.059	0.064	0.062	0.061
(b) Prediction RMSE with  $\rho = 0.3$  and varying  $a$ .
Table 3: Sensitive analysis of amplitude  $a$  and length-scale  $\rho$  on synthetic data.
amplitude  $a$ , for two commonly-used Matérn kernels: Matérn-1/2 and Matérn-3/2. The study was performed on SFTL based on both the CP and Tucker forms. The results are reported in Table 3. Overall, the predictive performance of SFTL is less sensitive to the amplitude parameter  $a$  than to the length-scale parameter  $\rho$ . But when we use Matérn-1/2, the performance of both SFTL-CP and SFTL-Tucker is quite stable to the length-scale parameter  $\rho$ . When we use Matérn-3/2, the choice of the length-scale is critical.
E Real-World Dataset Information and Competing Methods
We tested all the methods in the following four real-world datasets.
- FitRecord<sup>3</sup>, workout logs of EndoMondo users' health status in outdoor exercises. We extracted a three-mode tensor among 500 users, 20 sports types, and 50 altitudes. The entry values are heart rates. There are 50K observed entry values along with the timestamps.
- ServerRoom<sup>4</sup>, temperature logs of Poznan Supercomputing and Networking Center. We extracted a three-mode tensor between 3 air conditioning modes  $(24^{\circ}, 27^{\circ}$  and  $30^{\circ})$ , 3 power usage levels  $(50\%, 75\%, 100\%)$  and 34 locations. We collected 10K entry values and their timestamps.
- BeijingAir-2 $^5$ , air pollution measurement in Beijing from year 2014 to 2017. We extracted a two-mode tensor (monitoring site, pollutant), of size  $12 \times 6$ , and collected 20K observed entry values (concentration) and their timestamps.
- BeijingAir-3, extracted from the same data source as BeijingAir-2, a three-mode tensor among 12 monitoring sites, 12 wind speeds and 6 wind directions. The entry value is the PM2.5 concentration. There are 15K observed entry values at different timestamps.
We first compared with the following state-of-the-art streaming tensor decomposition methods based on the CP or Tucker model. (1) POST [Du et al., 2018], probabilistic streaming CP decomposition via mean-field streaming variational Bayes [Broderick et al., 2013] (2) BASS-Tucker [Fang et al., 2021a] Bayesian streaming Tucker decomposition, which online estimates a sparse tensor-core via a spike-and-slab prior to enhance the interpretability. We also implemented (3) ADF-CP, streaming CP