SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$^{3}$ https://sites.google.com/eng.ucsd.edu/fitrec-project/home  
 $^{4}$ https://zenodo.org/record/3610078#%23.Y8SYt3bMJGi  
 $^{5}$ https://archive.ics.uci.edu/ml/datasets/Beijing+Multi-Site+Air-Quality+Data
decomposition by combining the assumed density filtering and conditional moment matching [Wang and Zhe, 2019].
Next, we tested the state-of-the-art static decomposition algorithms, which have to go through the data many times. (4) P-Tucker [Oh et al., 2018], an efficient Tucker decomposition algorithm that performs parallel row-wise updates. (5) CP-ALS and (6) Tucker-ALS [Bader and Kolda, 2008], CP/Tucker decomposition via alternating least square (ALS) updates. The methods (1-6) are not specifically designed for temporal decomposition and cannot utilize the timestamps of the observed entries. In order to incorporate the time information for a fair comparison, we augment the tensor with a time mode, and convert the ordered, unique timestamps into increasing time steps.
We then compared with the most recent continuous-time temporal decomposition methods. Note that none of these methods can handle data streams. They have to iteratively access the data to update the model parameters and factor estimates. (7) CT-CP [Zhang et al., 2021], continuous-time CP decomposition, which uses polynomial splines to model a time-varying coefficient  $\lambda$  for each latent factor, (8) CT-GP, continuous-time GP decomposition, which extends [Zhe et al., 2016a] to use GPs to learn the tensor entry value as a function of the latent factors and time  $y_{\ell}(t) = g(\mathbf{u}_{\ell_1}^1,\dots ,\mathbf{u}_{\ell_K}^K,t)\sim \mathcal{GP}(0,\kappa (\cdot ,\cdot))$ , (9) BCTT [Fang et al., 2022], Bayesian continuous-time Tucker decomposition, which estimates the tensor-core as a time-varying function, (10) THIS-ODE [Li et al., 2022], which uses a neural ODE [Chen et al., 2018] to model the entry value as a function of the latent factors and time,  $\frac{\mathrm{dy}_{\ell}(t)}{\mathrm{dt}} = \mathrm{NN}(\mathbf{u}_{\ell_1}^1,\dots ,\mathbf{u}_{\ell_K}^K,t)$  where NN is short for neural networks. (11) NONFAT [Wang et al., 2022], nonparametric factor trajectory learning, the only existing work that also estimates factor trajectories for temporal tensor decomposition. It uses a bi-level GP to estimate the trajectories in the frequency domain and applies inverse Fourier transform to return to the time domain.
F More Results about Prediction Accuracy
We report for  $R = 2$ ,  $R = 3$  and  $R = 7$ , the final prediction error (after the data has been processed) of all the methods in Table 4, Table 5, and Table 6, respectively. We report for  $R = 2$ ,  $R = 3$  and  $R = 7$ , the online predictive performance of the streaming decomposition approaches in Fig. 5, Fig. 6, and Fig. 7, respectively.
(c) BeijingAir-2
Figure 5: Online prediction error with the number of processed entries  $(R = 2)$
(c) BeijingAir-2
(d) BeijingAir-3
Figure 6: Online prediction error with the number of processed entries  $(R = 3)$
G Running Time
As compared with static (non-streaming) methods, such as BCTT, our method is faster and more efficient. That is because whenever new data comes in, the static methods have to retrain the model from scratch and iteratively access the whole data accumulated so far, while our method only performs
RMSE	FitRecord	ServerRoom	BeijingAir-2	BeijingAir-3
Static	PTucker	0.606 ± 0.015	0.757 ± 0.36	0.509 ± 0.01	0.442 ± 0.142
Tucker-ALS	0.914 ± 0.01	0.991 ± 0.016	0.586 ± 0.016	0.896 ± 0.032
CP-ALS	0.926 ± 0.013	0.997 ± 0.016	0.647 ± 0.041	0.918 ± 0.031
CT-CP	0.675 ± 0.009	0.412 ± 0.024	0.642 ± 0.007	0.832 ± 0.035
CT-GP	0.611 ± 0.009	0.218 ± 0.021	0.723 ± 0.01	0.88 ± 0.026
BCTT	0.604 ± 0.019	0.715 ± 0.352	0.504 ± 0.01	0.799 ± 0.027
NONFAT	0.543 ± 0.002	0.132 ± 0.002	0.425 ± 0.002	0.878 ± 0.014
THIS-ODE	0.544 ± 0.005	0.142 ± 0.004	0.553 ± 0.015	0.876 ± 0.027
Stream	POST	0.705 ± 0.013	0.767 ± 0.155	0.539 ± 0.01	0.695 ± 0.135
ADF-CP	0.669 ± 0.033	0.764 ± 0.114	0.583 ± 0.07	0.54 ± 0.045
BASS-Tucker	1 ± 0.016	1 ± 0.016	1.043 ± 0.05	0.982 ± 0.058
SFTL-CP	0.437 ± 0.014	0.18 ± 0.019	0.323 ± 0.019	0.462 ± 0.009
SFTL-Tucker	0.446 ± 0.024	0.276 ± 0.031	0.344 ± 0.031	0.417 ± 0.035
MAE
Static	PTucker	0.416 ± 0.005	0.388 ± 0.152	0.336 ± 0.004	0.271 ± 0.053
Tucker-ALS	0.676 ± 0.008	0.744 ± 0.01	0.408 ± 0.008	0.669 ± 0.02
CP-ALS	0.686 ± 0.011	0.748 ± 0.009	0.454 ± 0.057	0.691 ± 0.016
CT-CP	0.466 ± 0.005	0.295 ± 0.029	0.49 ± 0.006	0.642 ± 0.02
CT-GP	0.424 ± 0.006	0.155 ± 0.012	0.517 ± 0.01	0.626 ± 0.01
BCTT	0.419 ± 0.015	0.534 ± 0.263	0.343 ± 0.003	0.579 ± 0.018
NONFAT	0.373 ± 0.001	0.083 ± 0.001	0.282 ± 0.002	0.622 ± 0.006
THIS-ODE	0.377 ± 0.003	0.097 ± 0.003	0.355 ± 0.008	0.606 ± 0.015
Stream	POST	0.485 ± 0.008	0.564 ± 0.091	0.368 ± 0.008	0.517 ± 0.123
ADF-CP	0.462 ± 0.022	0.574 ± 0.073	0.401 ± 0.029	0.415 ± 0.038
BASS	0.777 ± 0.039	0.749 ± 0.01	0.871 ± 0.125	0.727 ± 0.029
SFTL-CP	0.248 ± 0.005	0.126 ± 0.007	0.199 ± 0.005	0.311 ± 0.004
SFTL-Tucker	0.25 ± 0.01	0.203 ± 0.032	0.218 ± 0.02	0.261 ± 0.023
Table 4: Final prediction error with  $R = 2$ . The results were averaged from five runs.
RMSE	FitRecord	ServerRoom	BeijingAir-2	BeijingAir-3
Static	PTucker	0.603 ± 0.045	0.677 ± 0.129	0.464 ± 0.012	0.421 ± 0.074
Tucker-ALS	0.885 ± 0.007	0.989 ± 0.014	0.559 ± 0.017	0.863 ± 0.032
CP-ALS	0.907 ± 0.015	0.993 ± 0.016	0.594 ± 0.031	0.901 ± 0.03
CT-CP	0.666 ± 0.008	0.5 ± 0.2	0.641 ± 0.006	0.819 ± 0.019
CT-GP	0.606 ± 0.008	0.217 ± 0.025	0.749 ± 0.014	0.895 ± 0.054
BCTT	0.576 ± 0.015	0.358 ± 0.082	0.454 ± 0.011	0.829 ± 0.028
NONFAT	0.517 ± 0.002	0.129 ± 0.002	0.408 ± 0.005	0.877 ± 0.014
THIS-ODE	0.528 ± 0.005	0.132 ± 0.002	0.544 ± 0.014	0.878 ± 0.026
Stream	POST	0.706 ± 0.034	0.741 ± 0.161	0.518 ± 0.016	0.622 ± 0.123
ADF-CP	0.641 ± 0.009	0.652 ± 0.012	0.542 ± 0.012	0.518 ± 0.003
BASS-Tucker	1.008 ± 0.017	1 ± 0.016	1.035 ± 0.038	0.99 ± 0.034
SFTL-CP	0.434 ± 0.014	0.178 ± 0.006	0.288 ± 0.017	0.454 ± 0.011
SFTL-Tucker	0.418 ± 0.01	0.289 ± 0.096	0.314 ± 0.049	0.41 ± 0.013
MAE
Static	PTucker	0.392 ± 0.009	0.323 ± 0.053	0.307 ± 0.005	0.197 ± 0.029
Tucker-ALS	0.648 ± 0.012	0.743 ± 0.008	0.39 ± 0.008	0.651 ± 0.018
CP-ALS	0.666 ± 0.013	0.746 ± 0.01	0.415 ± 0.022	0.676 ± 0.021
CT-CP	0.462 ± 0.005	0.348 ± 0.141	0.489 ± 0.006	0.632 ± 0.015
CT-GP	0.419 ± 0.005	0.158 ± 0.022	0.544 ± 0.012	0.627 ± 0.015
BCTT	0.392 ± 0.004	0.267 ± 0.067	0.299 ± 0.006	0.607 ± 0.027
NONFAT	0.355 ± 0.001	0.078 ± 0.001	0.265 ± 0.003	0.622 ± 0.006
THIS-ODE	0.363 ± 0.004	0.083 ± 0.002	0.348 ± 0.006	0.603 ± 0.009
Stream	POST	0.482 ± 0.022	0.54 ± 0.102	0.351 ± 0.009	0.442 ± 0.109
ADF-CP	0.445 ± 0.006	0.5 ± 0.009	0.381 ± 0.006	0.393 ± 0.009
BASS	0.822 ± 0.024	0.749 ± 0.009	0.919 ± 0.041	0.73 ± 0.018
SFTL-CP	0.246 ± 0.005	0.121 ± 0.003	0.176 ± 0.006	0.305 ± 0.006
SFTL-Tucker	0.24 ± 0.002	0.18 ± 0.042	0.196 ± 0.03	0.263 ± 0.011
Table 5: Final prediction error with  $R = 3$ . The results were averaged from five runs.
RMSE	FitRecord	ServerRoom	BeijingAir-2	BeijingAir-3
Static	PTucker	0.603 ± 0.045	0.677 ± 0.129	0.464 ± 0.012	0.421 ± 0.074
Tucker-ALS	0.826 ± 0.003	0.983 ± 0.016	0.586 ± 0.018	0.825 ± 0.026
CP-ALS	0.878 ± 0.012	0.994 ± 0.013	0.897 ± 0.215	0.863 ± 0.024
CT-CP	0.663 ± 0.008	0.384 ± 0.008	0.64 ± 0.007	0.818 ± 0.019
CT-GP	0.603 ± 0.006	0.381 ± 0.303	0.766 ± 0.016	0.904 ± 0.046
BCTT	0.498 ± 0.011	0.194 ± 0.017	0.368 ± 0.01	0.813 ± 0.028
NONFAT	0.497 ± 0.003	0.128 ± 0.002	0.394 ± 0.004	0.88 ± 0.013
THIS-ODE	0.138 ± 0.003	0.554 ± 0.016	0.878 ± 0.027
Stream	POST	0.675 ± 0.012	0.707 ± 0.14	0.519 ± 0.017	0.738 ± 0.068
ADF-CP	0.652 ± 0.01	0.646 ± 0.008	0.548 ± 0.012	0.552 ± 0.026
BASS-Tucker	0.604 ± 0.043	0.493 ± 0.071	0.391 ± 0.005	0.634 ± 0.083
SFTL-CP	0.424 ± 0.006	0.166 ± 0.013	0.256 ± 0.013	0.481 ± 0.006
SFTL-Tucker	0.448 ± 0.009	0.406 ± 0.052	0.249 ± 0.017	0.432 ± 0.019
MAE
Static	PTucker	0.353 ± 0.005	0.305 ± 0.042	0.248 ± 0.004	0.32 ± 0.038
Tucker-ALS	0.6 ± 0.002	0.737 ± 0.009	0.392 ± 0.011	0.619 ± 0.015
CP-ALS	0.64 ± 0.009	0.745 ± 0.008	0.593 ± 0.121	0.637 ± 0.015
CT-CP	0.459 ± 0.005	0.27 ± 0.003	0.488 ± 0.005	0.626 ± 0.012
CT-GP	0.412 ± 0.004	0.282 ± 0.23	0.557 ± 0.009	0.628 ± 0.01
BCTT	0.342 ± 0.005	0.157 ± 0.015	0.234 ± 0.005	0.581 ± 0.022
NONFAT	0.335 ± 0.002	0.077 ± 0.002	0.256 ± 0.003	0.627 ± 0.005
THIS-ODE	0.362 ± 0.002	0.089 ± 0.002	0.357 ± 0.007	0.603 ± 0.013
Stream	POST	0.461 ± 0.008	0.518 ± 0.087	0.357 ± 0.011	0.558 ± 0.058
ADF-CP	0.451 ± 0.006	0.489 ± 0.009	0.384 ± 0.014	0.411 ± 0.025
BASS	0.745 ± 0.026	0.749 ± 0.01	0.903 ± 0.044	0.721 ± 0.038
SFTL-CP	0.243 ± 0.003	0.111 ± 0.008	0.159 ± 0.004	0.323 ± 0.003
SFTL-Tucker	0.253 ± 0.004	0.273 ± 0.033	0.144 ± 0.008	0.273 ± 0.016
(d) BeijingAir-3
Figure 7: Online prediction error with the number of processed entries  $(R = 7)$
incremental updates and never needs to revisit the past data. To demonstrate this point, we compared the training time of our method with BCTT on BeijingAir2 dataset. All the methods were run on a Linux workstation. From Table 7, we can see a large speed-up of our method with both the CP and Tucker form. The higher the rank  $(R)$ , the more significant the speed-up.
Table 6: Final prediction error with  $R = 7$ . The results were averaged from five runs.
R=2	R=3	R=5	R=7
SFTL-CP	27.1	27.2	28.5	29.1
SFTL-Tucker	32.3	35.6	43.2	59.3
BCTT	49.5	56.1	72.1	136.7
Table 7: Running time in seconds on BeijingAir2 dataset.