SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The time complexity of the proposed objective is  $\mathcal{O}(\nu B(DRH + LH^2))$ , where  $B$  is the batch size,  $\nu$  is the number of conditional noises,  $H$  is the number of hidden units per layer,  $L$  is the number of layers and  $D$  is the tensor order. The time complexity of our model is  $\nu$  times greater than traditional TDs, since we need to compute forward passes for  $\nu$  particles. However, as we only use small networks, the computational speed is still very fast (See Appendix C.3 for an illustration).
Traditional tensor decompositions (TDs) are based on multi-linear contraction rules, such as CP [15], Tucker [42], tensor networks [29, 51] and their variations [19, 6]. In this paper, we mainly focus on probabilistic TDs, which extend traditional methods by providing uncertainty estimates about both observations and latent factors [32, 49, 50, 26, 38]. These models build directed mapping from latent factors to tensor entries using multi-linear contraction rules, resulting in limited flexibility when dealing with complex datasets. An alternative approach involves replacing the multi-linear relations with non-linear ones. [5, 45, 52] introduced the use of tensor-variate Gaussian processes (GPs) for achieving nonparametric factorization. [53] further expanded on this concept by incorporating a GP prior on the function that maps latent factors to tensor entries, resulting in a nonparametric TD for sparse tensors. GP-based TDs are further extended using hierarchical priors [40], stochastic processes [43, 8]. Despite the success of GP-based TDs, nonparametric approaches can encounter computational challenges and may still lack sufficient flexibility. Recently, neural networks (NNs) are also applied to TDs. [25] suggested the utilization of convolutional NNs to map latent factors to tensor entries. Besides, [7] built a hierarchical version of the Tucker model and introduced non-linear mappings within each hierarchy of latent factors. To mitigate overfitting, [39] suggested the adoption of deep kernels in GP-based TD rather than using NNs directly. On the other hand, [9] proposed to use Bayesian NN with spike-and-slab prior to prevent from overfitting and obtain probabilistic estimates. More recently, [24] adopt neural ODEs to capture dynamic tensor trajectories. Other works regarding more flexible exponential families [13] or mixture of Gaussians [12] employ linear structures. While all these methods using directed mapping from latent factors to tensor entries, our model is fundamentally different from them, in that we construct much more flexible undirected probabilistic model of TD that can deal with diverse distributions and structures.
Another related direction is the energy-based model (EBM). To address the intractable pdf of EBMs, various training methods have been proposed, including contrastive divergence [CD, 14], score matching [SM, 16], noise-contrastive estimation [NCE, 11]. CD requires large steps of Monte Carlo samples, which can be computationally expensive for high-dimensional tensors. SM cannot handle discrete data, and learning latent variables with SM requires complex bi-level optimization [2]. Therefore, we focus on NCE in this paper. Learning energy-based TD is even more challenging because it involves multiple coupled latent factors that cannot be analytically marginalized. [33] proposed VNCE to handle unnormalized models with latent factors. We enhance their algorithm by using conditional noises [3] to improve the learning efficiency. One fundamental distinction between our model and traditional EBMs is that, through TD construction, we only need to learn one-dimensional distributions for each scalar entry, instead of the original high-dimensional tensor. Hence, our model avoids performance degradation when learning high-dimensional data using NCE.
We demonstrate the proposed energy-based TD (EnergyTD) on synthetic data and several real-world applications. All the experiments are conducted on a Linux workstation with Intel Xeon Silver 4316 CPU, 256GB RAM and NVIDIA RTX A5000 GPUs (24GB memory each). The code is implemented based on PyTorch 1.12.1 [30]. More experimental details can be found in the appendix. The code is available at https://github.com/taozerui/energy_td
5.1 Simulation study
Tensors with non-Gaussian distributions Traditional TDs commonly assume that tensor entries follow a Gaussian distribution. However, in real-world applications, we often encounter highly complex distributions. In this experiment, we evaluate the capability of our model to learn distributions that deviate from the Gaussian assumption.
We consider a two-mode tensor of shape  $I \times I$ , where we set  $I = 8$ . Firstly, two latent factors of shape  $I \times R$  are generated, where the rank  $R$  is set to 5. Then, conditioned on the latent factors, we generated tensor observations from particular distributions. For each entry, we generate  $N = 200$  samples. Three types of distributions are considered: (1) Beta distribution; (2) Mixture of Gaussians (MoG) and (3) Exponential distribution. For Beta distribution, we generate latent factors from uniform distribution  $Z^{i=1,2} \stackrel{\mathrm{id}}{\sim} Uni(0.0,1.1)$ . Then, we sample the observed tensor from Beta
(a) Beta distribution
(b) Mixture of Gaussians
(c) Exponential distribution
Figure 1: Simulation results for different distributions. The blue line is the ground truth pdf. The yellow line is the kernel density estimation (KDE) plot of observed samples. The red line is the GPTF estimation, which is a Gaussian pdf. The green line is our method, computed by evaluating the unnormalized pdf on grids and calculating the partition function using Gaussian quadrature.
(a)  $\mathrm{MR} = 0.3$
Figure 2: Simulation results for continuous-time tensor decomposition. The blue regions are observed and the red regions are missing. The trajectories of ground truth, BCTT, NONFAT and our model are drawn in black, blue, red and green lines, respectively.
(b)  $\mathrm{MR} = 0.1$
(c) Fully observed
distribution  $x_{ij} \stackrel{\mathrm{iid}}{\sim} \text{Beta}((\mathbf{Z}^1 \mathbf{Z}^{2,\tau})_{ij}, 1.2)$ . For MoG distribution, we draw latent factors from uniform distribution  $Z^{i=1,2} \stackrel{\mathrm{iid}}{\sim} Uni(0.0, 1.0)$ . The tensor entries are then drawn from MoG  $x_{ij} \stackrel{\mathrm{iid}}{\sim} 0.6 \cdot \mathcal{N}(\cos((\mathbf{Z}^1 \mathbf{Z}^{2,\tau})_{ij}), 0.1^2) + 0.4 \cdot \mathcal{N}(\sin((\mathbf{Z}^1 \mathbf{Z}^{2,\tau})_{ij}), 0.25^2)$ . For Exponential distribution, we generate latent factors from uniform distribution  $Z^{i=1,2} \stackrel{\mathrm{iid}}{\sim} Uni(0.0, 1.0)$ . The tensor entries are then sampled from Exponential distribution  $x_{ij} \stackrel{\mathrm{iid}}{\sim} \text{Exp}((\mathbf{Z}^1 \mathbf{Z}^{2,\tau})_{ij})$ . We compare with GP tensor factorization [GPFT, 53], which assumes the entries follow Gaussian distribution.
The results of probability density function (pdf) estimation are presented in Fig. 1. We display the learned pdf of one single tensor entry. This reveals that GPTF is limited to capturing only the 1st moments (mean values) and overlooks higher-order information of more complex distributions. Our model exhibits greater flexibility in handling non-Gaussian distributions.
Continuous-time tensors We then consider a dynamic tensor, where each entry is a time series. We follow similar setting with [9] and use the same data size with the previous simulation, i.e., a two-mode tensor of shape  $8 \times 8$  with each entry being a time series of length 200. We firstly generate latent factors of shape  $8 \times 2$ , with each rows drawn from  $z_i^1 \sim \mathcal{N}([0, 2], 2 \cdot I)$  and  $z_i^2 \sim \mathcal{N}([1, 1], 2 \cdot I)$ . Then we generate  $N = 200$  observed entries from time  $t \in [0, 1]$ . The tensor entries are computed by  $x_i(t) = \sum_{r_1=1}^{2} \sum_{r_2=1}^{2} z_{i_1r_1}^1 z_{i_2r_2}^2 \omega_{r_1r_2}(t)$ , where  $\omega_{11}(t) = \sin(2\pi t), \omega_{12}(t) = \cos(2\pi t), \omega_{21}(t) = \sin^2(2\pi t)$  and  $\omega_{22}(t) = \cos(5\pi t) \sin^2(5\pi t)$ . Finally, the data are normalized to range  $[0, 2]$ . The synthetic data consist of low-frequency trends and high-frequency fluctuations. Apart from fully observed case, we also test with missing rates (MR)  $10\%$  and  $30\%$ . In specific, for each entry, we randomly select a starting time and set the following consecutive  $10\%$  or  $30\%$  time stamps as missing.
We compare with two methods that are designed for dynamic tensors, the Bayesian continuous-time Tucker decomposition [BCTT, 8] and the nonparametric factor trajectory learning [NONFAT, 43].
Table 1: Sparse tensor completion
Alog	RMSE	MAE
Rank 3	Rank 5	Rank 8	Rank 10	Rank 3	Rank 5	Rank 8	Rank 10
CP-WOPT	1.486 ± 0.282	1.386 ± 0.043	1.228 ± 0.063	1.355 ± 0.079	0.694 ± 0.098	0.664 ± 0.018	0.610 ± 0.027	0.658 ± 0.026
GPTF	0.911 ± 0.008	0.867 ± 0.008	0.878 ± 0.009	0.884 ± 0.009	0.511 ± 0.005	0.494 ± 0.004	0.530 ± 0.004	0.554 ± 0.006
HGP-GPTF	0.896 ± 0.011	0.867 ± 0.009	0.850 ± 0.011	0.844 ± 0.006	0.479 ± 0.007	0.473 ± 0.003	0.474 ± 0.004	0.480 ± 0.004
POND	0.885 ± 0.010	0.871 ± 0.013	0.858 ± 0.009	0.857 ± 0.011	0.463 ± 0.004	0.454 ± 0.005	0.444 ± 0.005	0.443 ± 0.006
CoSTCo	0.999 ± 0.007	0.936 ± 0.017	0.930 ± 0.024	0.909 ± 0.014	0.523 ± 0.006	0.481 ± 0.007	0.514 ± 0.031	0.481 ± 0.008
EnergyTD	0.864 ± 0.011	0.835 ± 0.011	0.840 ± 0.013	0.833 ± 0.016	0.450 ± 0.006	0.433 ± 0.006	0.424 ± 0.005	0.409 ± 0.004
ACC
CP-WOPT	0.533 ± 0.039	0.592 ± 0.037	0.603 ± 0.028	0.589 ± 0.022	0.138 ± 0.004	0.147 ± 0.005	0.148 ± 0.003	0.147 ± 0.004
GPTF	0.367 ± 0.001	0.357 ± 0.001	0.359 ± 0.001	0.368 ± 0.001	0.152 ± 0.002	0.150 ± 0.001	0.167 ± 0.002	0.182 ± 0.001
HGP-GPTF	0.355 ± 0.001	0.344 ± 0.001	0.341 ± 0.001	0.338 ± 0.001	0.125 ± 0.003	0.129 ± 0.001	0.139 ± 0.000	0.145 ± 0.002
CoSTCo	0.385 ± 0.003	0.376 ± 0.018	0.363 ± 0.004	0.348 ± 0.002	0.117 ± 0.004	0.137 ± 0.020	0.107 ± 0.004	0.101 ± 0.004
EnergyTD	0.348 ± 0.005	0.336 ± 0.004	0.328 ± 0.003	0.328 ± 0.003	0.110 ± 0.008	0.101 ± 0.006	0.094 ± 0.006	0.101 ± 0.009
BCTT treats Tucker core tensors as functions of time and NONFAT treats all GPTF factors as time series. Unlike BCTT with GP prior on the time domain, NONFAT uses GP prior on the frequency domain through inverse Fourier transform of original time series.
Fig. 2 displays the completion results. The learned trajectory of a single tensor entry is plotted. Higher missing rates result in the inability of BCTT to capture accurate trajectories, particularly in missing regions. NONFAT achieves more stable predictions, yet it tends to favor over-smoothed trajectories while disregarding high-frequency fluctuations. This behavior may be attributed to its unique construction, which introduces a GP prior in the frequency domain. Our utilization of flexible neural networks allows us to adapt to complex situations encompassing both low-frequency and high-frequency information.
5.2 Tensor completion
We evaluate our model on two sparse tensor and two dynamic tensor completion applications. For real datasets, the energy function can be difficult to learn, when the pdf has a very sharp curve. Motivated by the idea of noise-perturbed score estimation [35], we add small i.i.d. Gaussian noises on the data during the training of EnergyTD as a form of smoothing technique. The results are reported on clean test data. For EnergyTD, we use MAP estimates as described in Section 3.1.
5.2.1 Sparse tensor completion
We test our model on two sparsely observed tensors: (1)  $Alog$ , a file access log dataset [52] of shape  $200$  users  $\times 100$  actions  $\times 200$  resources with about  $0.33\%$  nonzero entries; (2)  $ACC$ , a three-way tensor generated from a code repository management system [52] of shape  $3k$  users  $\times 150$  actions  $\times 30k$  resources with about  $0.009\%$  nonzero entries. We use the same dataset split as in [52] and report the 5-fold cross validation results.
Competing methods We compare with five baselines: (1) CP-WOPT [1], CP decomposition with stochastic optimization; (2) GPTF [53], a GP-based tensor factorization using stochastic variational inference; (3) HGP-GPTF [40], a GPTF equipped with hierarchical Gamma process prior; (4) POND [39], a probabilistic non-linear TD using deep kernels with convolutional NNs (CNNs); (5) CoSTCo [25], a non-linear TD that uses CNNs to map latent factors to tensor entries. CP-WOPT is provided in Matlab Tensor Toolbox [1]. We implement GPTF based on PyTorch by ourselves and use official implementations for HGP-GPTF $^2$ , POND $^3$  and CoSTCo $^4$ .
Experimental settings and results We set batch size 1000 and run 1000 epochs for  $Alog$ , 100 epochs for  $ACC$ . For our model, we use Adam [17] optimizer. Learning rates of all models are chosen from  $\{1e-2, 1e-3, 1e-4\}$ . For all methods, we evaluate with rank  $R \in \{3, 5, 8, 10\}$ . All methods are evaluated by 5 runs with different random seeds.
$^{2}$ https://github.com/ctilling/SparseTensorHGP
<sup>3</sup>https://github.com/ctilling/POND
<sup>4</sup>https://github.com/USC-Melady/KDD19-CoSTCo
Table 2: Continuous-time tensor completion
Air	RMSE	MAE
Rank 3	Rank 5	Rank 8	Rank 10	Rank 3	Rank 5	Rank 8	Rank 10
CTCP	1.020 ± 0.002	1.022 ± 0.002	1.022 ± 0.002	1.022 ± 0.002	0.784 ± 0.002	0.785 ± 0.002	0.787 ± 0.002	0.787 ± 0.002
CTGP	0.475 ± 0.000	0.463 ± 0.000	0.459 ± 0.000	0.458 ± 0.000	0.318 ± 0.000	0.304 ± 0.000	0.301 ± 0.000	0.299 ± 0.000
CTNN	1.013 ± 0.001	1.005 ± 0.005	0.999 ± 0.009	1.013 ± 0.002	0.780 ± 0.001	0.777 ± 0.003	0.776 ± 0.003	0.780 ± 0.001
NNDTN	0.377 ± 0.004	0.364 ± 0.002	0.334 ± 0.004	0.328 ± 0.004	0.247 ± 0.003	0.239 ± 0.002	0.217 ± 0.003	0.212 ± 0.004
NONFAT	0.339 ± 0.002	0.335 ± 0.002	0.351 ± 0.005	0.342 ± 0.002	0.224 ± 0.002	0.219 ± 0.001	0.228 ± 0.003	0.223 ± 0.001
THIS-ODE	0.569 ± 0.001	0.566 ± 0.004	0.542 ± 0.005	0.541 ± 0.002	0.415 ± 0.002	0.409 ± 0.004	0.395 ± 0.004	0.391 ± 0.001
EnergyTD	0.302 ± 0.008	0.291 ± 0.006	0.300 ± 0.012	0.283 ± 0.004	0.184 ± 0.006	0.177 ± 0.003	0.172 ± 0.006	0.184 ± 0.003
Click
CTCP	2.063 ± 0.009	2.020 ± 0.025	2.068 ± 0.012	2.009 ± 0.023	1.000 ± 0.009	0.977 ± 0.021	1.005 ± 0.010	0.969 ± 0.012
CTGP	1.424 ± 0.002	1.423 ± 0.004	1.404 ± 0.004	1.392 ± 0.002	0.880 ± 0.003	0.877 ± 0.003	0.856 ± 0.002	0.849 ± 0.001
CTNN	1.820 ± 0.005	1.820 ± 0.005	1.820 ± 0.005	1.820 ± 0.005	1.077 ± 0.027	1.053 ± 0.012	1.083 ± 0.016	1.071 ± 0.024
NNDTN	1.418 ± 0.005	1.409 ± 0.004	1.407 ± 0.002	1.410 ± 0.004	0.858 ± 0.002	0.856 ± 0.002	0.859 ± 0.003	0.863 ± 0.006
NONFAT	1.400 ± 0.008	1.411 ± 0.006	1.365 ± 0.004	1.351 ± 0.002	0.853 ± 0.004	0.873 ± 0.004	0.832 ± 0.004	0.812 ± 0.002
THIS-ODE	1.421 ± 0.004	1.413 ± 0.002	1.408 ± 0.002	1.395 ± 0.003	0.836 ± 0.004	0.836 ± 0.003	0.832 ± 0.002	0.829 ± 0.003
EnergyTD	1.396 ± 0.003	1.385 ± 0.003	1.356 ± 0.001	1.357 ± 0.001	0.777 ± 0.003	0.775 ± 0.003	0.772 ± 0.002	0.773 ± 0.001
The completion results are shown in Table 1. The mean and standard deviation of the root mean square error (RMSE) and mean absolute error (MAE) are reported. Results from POND are not included for ACC due to the slow code execution and the substantial memory requirements for this dataset. Our model outperforms the baselines in nearly all cases on both RMSE and MAE. Moreover, the improvement is significant  $(p < 0.05)$  for most cases. All non-linear methods exhibit substantial superiority over CP-WOPT, thereby highlighting the advantage of employing more flexible structures. Furthermore, the enhanced performance of our model may be attributed to the utilization of more flexible undirected probabilistic distributions. It is worth noting that our model, despite employing MLP layers, outperforms POND and CoSTCo, which utilize convolutional layers. We believe that our model can be further improved by designing more suitable network architectures.
5.2.2 Continuous-time tensor completion
In this subsection, we evaluate our model on two continuous-time tensor datasets: (1) Air, the Beijing air quality dataset [47] of shape  $12$  sites × 6 pollutants with about  $1 \times 10^{4}$  observations at different time stamps; (2) Click, an ads click dataset [43] of shape 7 banner positions × 2842 site domains × 4127 mobile APPs with about  $5 \times 10^{4}$  entries at different time stamps. We use the same dataset split as in [43] and report the 5-fold cross validation results.
Competing methods We compare with (1) Nonparametric factor trajectory learning [NONFAT, 43], which is the SOTA method for continuous-time tensor completion, and other baselines including: (2) Continuous-time CP [CTCP, 48], which uses polynomial splines to model dynamics of CP coefficients; (3) Continuous-time GP (CTGP), which extends GPTF [53] by injecting time stamps into GP kernels; (5) Continuous-time NN decomposition (CTNN), which directly uses time stamps as inputs in CoSTCo [25] to learn continuous dynamics; (5) Discrete-time NN decomposition with non-linear dynamics [NNDTN, 43], which employs RNN dynamics for time steps; (6) Tensor high-order interaction learning via ODEs [THIS-ODE, 24], where continuous-time trajectories of tensor entries are captured by neural ODEs. For baseline models (1-5), we use the official implementation<sup>5</sup> provided by [43]. For (6) THIS-ODE, its official implementation<sup>6</sup> is used.
Experimental settings and results We set batch size 128 and run 400 epochs for  $Air$ , 200 epochs for Click. Notably, in the case of THIS-ODE, we only run 200 epochs for  $Air$  and 75 epochs for Click due to the slow speed of ODE solvers (See Appendix C.3 for an illustration). For our model, we use Adam [17] optimizer with learning rate chosen from  $\{1e - 2, 1e - 3\}$ . The network architecture is described in Section 3.1. For baseline models, provided settings in their codebases are used. For all methods, we evaluate with rank  $R \in \{3, 5, 8, 10\}$ . All methods are evaluated by 5 runs with different random seeds.
<sup>5</sup>https://github.com/wzhut/NONFAT
$^{6}$ https://github.com/shibOli/THIS-ODE
Table 2 presents the completion results. It should be noted that the results presented here differ from those reported in [43] as we adhere to the standard definition of RMSE and MAE (see appendix for detail). Our model surpasses the baseline methods in almost all cases for both RMSE and MAE, with statistically significant improvements  $(p < 0.05)$  observed in most cases. Particularly, we observe that the improvements in MAE are notably more significant. One possible reason is that NONFAT is trained implicitly by minimizing the square loss (with regularization) as it adopts Gaussian assumption about the data. However, our model does not make such assumptions about the data distribution and the loss function. Hence, it can adapt to the data distribution more flexibly. Additionally, we observe that directly injecting time stamps into neural networks, as done in CTNN, is ineffective, thus highlighting the advantage of our model in learning more informative structures.
We introduce an innovative approach to undirected probabilistic tensor decomposition (TD), characterized by its exceptional flexibility in accommodating various structures and distributions. Specifically, our model integrates deep EBMs in TD to relax both structural and distributional assumptions, enabling it to handle complex real-world applications. To efficiently learn the doubly intractable pdf, we derive a VCNCE objective that is the upper bound of the CNCE loss. Experimental results demonstrate that our model can handle diverse distributions and outperforms baseline methods in multiple real-world applications. One limitation is that our final loss function is not a fully variational upper bound of CNCE, since we have to use importance samples to approximate the pdf of noise samples in Eq. (7). In the future, we aim to derive a fully variational bound as in [46]. Finally, we did not delve into the interpretability of learned factors in this work. However, exploring the interpretability of these factors represents a promising avenue for future research in the realm of tensor decompositions.
Zerui Tao was supported by the RIKEN Junior Research Associate Program. This work was supported by the JSPS KAKENHI Grant Numbers JP20H04249, JP23H03419.
[1] Brett W Bader and Tamara G Kolda. Efficient matlab computations with sparse and factored tensors. SIAM Journal on Scientific Computing, 30(1):205-231, 2008.
[2] Fan Bao, Chongxuan Li, Kun Xu, Hang Su, Jun Zhu, and Bo Zhang. Bi-level score matching for learning energy-based latent variable models. Advances in Neural Information Processing Systems, 33:18110-18122, 2020.
[3] Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. In International Conference on Machine Learning, pages 726–734. PMLR, 2018.
[4] Rong Chen, Dan Yang, and Cun-Hui Zhang. Factor models for high-dimensional tensor time series. Journal of the American Statistical Association, 117(537):94-116, 2022.
[5] Wei Chu and Zoubin Ghahramani. Probabilistic models for incomplete multi-dimensional arrays. In Artificial Intelligence and Statistics, pages 89-96. PMLR, 2009.
[6] Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, Danilo P Mandic, et al. Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions. Foundations and Trends® in Machine Learning, 9(4-5):249–429, 2016.
[7] Jicong Fan. Multi-mode deep matrix and tensor factorization. In international conference on learning representations, 2021.
[8] Shikai Fang, Akil Narayan, Robert Kirby, and Shandian Zhe. Bayesian continuous-time tucker decomposition. In International Conference on Machine Learning, pages 6235-6245. PMLR, 2022.
[44] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681–688, 2011.
[45] Zenglin Xu, Feng Yan, and Yuan Qi. Infinite tucker decomposition: nonparametric bayesian models for multiway data analysis. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1675-1682, 2012.
[46] Christopher Zach. Fully variational noise-contrastive estimation. In Image Analysis: 23rd Scandinavian Conference, SCIA 2023, Sirkka, Finland, April 18–21, 2023, Proceedings, Part II, pages 175–190. Springer, 2023.
[47] Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality improvement in beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):20170457, 2017.
[48] Yanqing Zhang, Xuan Bi, Niansheng Tang, and Annie Qu. Dynamic tensor recommender systems. The Journal of Machine Learning Research, 22(1):3032-3066, 2021.
[49] Qibin Zhao, Liqing Zhang, and Andrzej Cichocki. Bayesian cp factorization of incomplete tensors with automatic rank determination. IEEE transactions on pattern analysis and machine intelligence, 37(9):1751-1763, 2015.
[50] Qibin Zhao, Liqing Zhang, and Andrzej Cichocki. Bayesian sparse tucker models for dimension reduction and tensor completion. arXiv preprint arXiv:1505.02343, 2015.
[51] Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decomposition. arXiv preprint arXiv:1606.05535, 2016.
[52] Shandian Zhe, Zenglin Xu, Xinqi Chu, Yuan Qi, and Youngja Park. Scalable nonparametric multiway data analysis. In Artificial Intelligence and Statistics, pages 1125-1134. PMLR, 2015.
[53] Shandian Zhe, Kai Zhang, Pengyuan Wang, Kuang-chih Lee, Zenglin Xu, Yuan Qi, and Zoubin Ghahramani. Distributed flexible nonlinear tensor factorization. Advances in neural information processing systems, 29, 2016.
Firstly, we give the definition of  $f$ -divergence.
Definition 1 (f-divergence) The f-divergence between two probability density functions (pdf)  $p$  and  $q$  is defined as,
$$
\mathbb {D} _ {f} (p \| q) = \mathbb {E} _ {q} \left[ f \left(\frac {p}{q}\right) \right],
$$
where  $f:[0,\infty)\to \mathbb{R}$  is a convex function and  $f(1) = 0$
As shown in [33], since partition functions for  $\phi (x,m;\theta)$  and  $\phi (x;\theta)$  are the same, we have the following factorization,