Title: Undirected Probabilistic Model for Tensor Decomposition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tensor decomposition

tensor completion

probabilistic methods
Abstract: Tensor decompositions (TDs) serve as a powerful tool for analyzing multiway data. Traditional TDs incorporate prior knowledge about the data into the model, such as a directed generative process from latent factors to observations. In practice, selecting proper structural or distributional assumptions beforehand is crucial for obtaining a promising TD representation. However, since such prior knowledge is typically unavailable in real-world applications, choosing an appropriate TD model can be challenging. This paper aims to address this issue by introducing a flexible TD framework that discards the structural and distributional assumptions, in order to learn as much information from the data. Specifically, we construct a TD model that captures the joint probability of the data and latent tensor factors through a deep energy-based model (EBM). Neural networks are then employed to parameterize the joint energy function of tensor factors and tensor entries. The flexibility of EBM and neural networks enables the learning of underlying structures and distributions. In addition, by designing the energy function, our model unifies the learning process of different types of tensors, such as static tensors and dynamic tensors with time stamps. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability function. To efficiently train the model, we derive a variational upper bound of the conditional noise-contrastive estimation objective that learns the unnormalized joint probability by distinguishing data from conditional noises. We show advantages of our model on both synthetic and several real-world datasets.

Undirected Probabilistic Model for Tensor Decomposition

Abstract

Tensor decompositions (TDs) serve as a powerful tool for analyzing multiway data. Traditional TDs incorporate prior knowledge about the data into the model, such as a directed generative process from latent factors to observations. In practice, selecting proper structural or distributional assumptions beforehand is crucial for obtaining a promising TD representation. However, since such prior knowledge is typically unavailable in real-world applications, choosing an appropriate TD model can be challenging. This paper aims to address this issue by introducing a flexible TD framework that discards the structural and distributional assumptions, in order to learn as much information from the data. Specifically, we construct a TD model that captures the joint probability of the data and latent tensor factors through a deep energy-based model (EBM). Neural networks are then employed to parameterize the joint energy function of tensor factors and tensor entries. The flexibility of EBM and neural networks enables the learning of underlying structures and distributions. In addition, by designing the energy function, our model unifies the learning process of different types of tensors, such as static tensors and dynamic tensors with time stamps. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability function. To efficiently train the model, we derive a variational upper bound of the conditional noise-contrastive estimation objective that learns the unnormalized joint probability by distinguishing data from conditional noises. We show advantages of our model on both synthetic and several real-world datasets.

1 Introduction

Tensor decompositions (TDs) serve as powerful tools for analyzing high-order and high-dimensional data, aiming to capture the inter-dependencies among different modes by utilizing multiple latent factors. TDs have demonstrated remarkable success in various machine learning tasks, including data imputation [53, 9], factor analysis [4], time-series forecasting [27], model compression [28, 41], generative models [10, 20] among others.

Existing TDs typically incorporate predefined directed graphical models into the generative process. These models specify the priors of latent factors and the conditional probabilities of observations, following specific contraction rules associated with the latent factors. Traditional contraction rules predominantly employ multi-linear products, like CP [15], Tucker [42], tensor train [29] and other variants [19, 6]. However, selecting an appropriate contraction rule for specific datasets is often challenging in real-world applications. Recent research, known as tensor network structure search [TNSS, 22, 23], has demonstrated that selecting an appropriate TN contraction rule significantly enhances the factorization performance. Another promising approach involves learning non-linear mappings from the data, utilizing techniques like nonparametric models [5, 45, 53] and deep neural networks

[25, 9]. Empirical results demonstrate that non-linear TDs exhibit superior performance compared to traditional multi-linear TDs in various applications, attributed to their enhanced expressive power.

Despite the success of non-linear TDs in reducing structural assumptions, they often rely on simplistic distributional assumptions. Typically, a specific directed graphical model is adopted to model the generative process from latent factors to tensor entries, represented as p ( x ) = ∫ p ( x | z ) p ( z ) d z , where z denotes tensor factors and x represents observations. Additionally, the distributions are usually selected from exponential families for tractability, such as Gaussian and Bernoulli distributions. For instance, a Gaussian prior can be assigned to latent factors, and observed entries can be modeled using Gaussian distribution [32, 49] or Gaussian process [45, 53]. However, these prior assumptions regarding the probabilistic model can introduce model bias and reduce the effectiveness of TD models. In real-world applications, the latent factors might originate from unknown distributions, and the observations can exhibit complex multi-modal generative processes. Without knowing the underlying generative process, these simplistic assumptions can lead to inaccurate estimations.

To address these issues, this paper proposes to construct an undirected graphical model of TD. More specifically, a TD model that captures the joint probability of the data and latent tensor factors is constructed through a deep energy-based model (EBM), represented as p ( x, z ) ≈ exp( -f ( x, z )) . Neural networks (NNs) are then employed to parameterize the joint energy function f ( x, z ) . The flexibility of EBM and NNs facilitates the learning of underlying structures and distributions. Furthermore, our model unifies the learning process in the presence of side information, such as dynamic tensors with time stamps, by designing the energy function. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability density function (pdf). For efficient model training, we derive a variant of conditional noise-contrastive estimation [3] algorithm that learns the unnormalized joint probability by distinguishing data from conditional noises. The proposed model offers several advantages: (1) it features a flexible structure that can adapt to different distributions; (2) the undirected nature allows us to learn more general correlations than traditional directed TDs; (3) it can handle diverse tasks and encode auxiliary information by adjusting the energy function.

Experiments are conducted on synthetic and real-world datasets to showcase the advantages of our model. Through simulation studies, we demonstrate the capability of our model to handle data generated from diverse distributions, in contrast to traditional Gaussian-based models that yield unfaithful and biased estimates. Subsequently, experiments are performed on multiple real-world datasets to evaluate sparse and continuous-time tensor completion. Our model outperforms various baselines across multiple metrics and settings, highlighting the generality of the proposed model.

2 Backgrounds

Notations We adopt similar notations with [19]. Throughout the paper, we use lowercase letters, bold lowercase letters, bold capital letters and calligraphic bold capital letters to represent scalars, vectors, matrices and tensors, e.g. , x, x , X and X . Tensors refer to multi-way arrays which generalize matrices. For a D -order tensor X ∈ R I 1 ×···× I D , we denote its ( i 1 , . . . , i D ) -th entry as x i .

2.1 Tensor decomposition

Given a D -order tensor X ∈ R I 1 ×···× I D , tensor decomposition (TD) aims to factorize X into D smaller latent factors Z d =1 ,...,D ∈ R I d × R d by using some predefined tensor contraction rules. The classical Tucker decomposition [42] assumes X = W × 1 Z 1 × 2 · · ·× D Z D , where W ∈ R R 1 ×···× R D is the coefficient and × d denotes the matrix-tensor contraction [19]. Equivalently, each entry can be written as x i = ∑ R 1 r 1 =1 · · · ∑ R D r D =1 w r 1 ...r D z 1 i 1 r 1 · · · z D i D r D , where the tuple ( R 1 , . . . , R D ) is the Tucker rank of tensor X . The latent factors Z d can capture information of each tensor mode and W represents the weight of each factors. CP decomposition [15] is a restricted form of Tucker by assuming W is super-diagonal, i.e. , x i = ∑ R r =1 w r z 1 i 1 r · · · z D i D r , where we simplify w r = w r...r . In this paper, we focus on probabilistic version of TDs, which serves as generalizations of traditional ones. The standard approach is to formulate TDs as a directed graphical model, p ( X ) = ∫ p ( X | Z ) p ( Z ) d Z , where Z denotes { Z 1 , . . . , Z D } for simplicity. For continuous data, the p ( X | Z ) is usually assumed to be Gaussian and TDs are used to parameterize the mean of corresponding Gaussian distribution [32, 49, 50].

Despite the elegant form of these multi-linear contraction rules, they have limited flexibility that can be mitigated by extending TDs to their non-linear counterparts. We can think of TD as a function that maps the multiway latent factors to tensor entries. One extension is to add Gaussian process (GP) priors on the function to obtain a nonparametric model, which resembles a GP latent variable model. In particular, [53] proposed to stack the latent factors as m i = [ z 1 i 1 , . . . , z D i D ] ∈ R DR and then assign a GP prior on the functional mapping. In specific, for continuous data, it assumes x i ∼ N ( µ i , σ 2 ) , where the mean function is a GP µ i = f ( m i ) ∼ GP (0 , k ( m i , · )) associated with kernel k ( · , · ) . Since GP has large computational complexity and designing the kernel requires ad-hoc expert domain knowledge, [25] proposed to parameterize the function using neural networks (NNs), x i ∼ N ( µ i , σ 2 ) where µ i = f NN ( m i ) . However, NNs easily overfit due to the high-dimensional and sparse nature of tensor data. To address this issue, [9] proposed to use Bayesian NN with spike-and-slab prior for sparse weights. All these models are based on directed graphical model, that assumes there exists a direct mapping from the latent factors to tensor entries and use simplistic distributions.

2.2 Energy-based model

Energy-based model [EBM, 21] is a class of undirected probabilistic models, which uses an energy function to characterize the data distribution. Given observed data x , the basic idea of EBM is to approximate the data distribution by a Boltzmann distribution, p data ( x ) ≈ exp( -f ( x ; θ )) Z ( θ ) , where Z ( θ ) = ∫ exp( -f ( x ; θ )) d x is the normalization constant (a.k.a., the partition function) and f ( x ; θ ) is the energy function. One classical example of EBM is the restricted Boltzmann machine (RBM) where the energy function has bi-linear form for tractability. In deep EBMs, the energy function is typically parameterized by deep neural networks. The main difficulty for training EBMs is to deal with the intractable normalization constant [36]. There are several ways to train EBMs, including contrastive divergence [14], score matching [16], noise-contrastive estimation [NCE, 11] and so on. In this paper, we focus on NCE, due to its efficiency and ability of tackling different data types.

We denote the unnormalized pdf as ϕ ( x ; θ ) = exp( -f ( x ; θ )) . NCE consider the normalization constant Z ( θ ) as a trainable parameter. However, maximum likelihood estimation (MLE) does not work for this case since Z ( θ ) can be arbitrarily small and the log-likelihood goes to infinity. Instead, the NCE can be obtained by maximizing the following objective,

where x denotes observed data and y denotes noises generated from some known distribution p n , ν is the ratio between noise and sample sizes, i.e. , ν = # y/ # x . And h ( · ) can be regarded as a classifier that distinguish data from noises, defined as follows, h ( u ; θ ) = ϕ ( u ; θ ) ϕ ( u ; θ )+ νp n ( u ) . It has been shown that NCE is consistent with MLE [11].

Although the noise distribution is essential for training efficiency, the selection of noises is currently limited to heuristics. A common intuitive is that the noise should be similar with the data. Indeed, [3] proposed to use conditional noises y ∼ p c ( y | x ) and minimizing the following loss function,

3 Proposed model

3.1 Energy-based tensor decomposition

Even though many non-linear tensor decompositions (TD) have been proposed to enhance flexibility, existing methods typically adopt simplistic distributions such as Gaussian. This can be problematic for complex real-world data. To address the issue, we propose energy-based tensor decomposition (EnergyTD), by integrating EBMs in TD framework. Given an orderD tensor X of shape I 1 × · · · × I D , we aim to factorize it into D smaller latent factors Z d ∈ R I d × R , ∀ d = 1 , . . . , D . We denote the latent factor associated with the ( i 1 , . . . , i D ) -th entry as m i = [ z 1 i 1 , . . . , z D i D ] ∈ R DR , where z d i d ∈ R R is the i d -th row of Z d . Unlike traditional directed TDs trying to parameterize the conditional expectation E [ x i | m i ] = f ( m i ) , we model the joint distribution using an EBM,

ϕ ( u 2 ; θ ) p c ( u 1 | u 2 )

where f ( · , · ; θ ) is the energy function and Z ( θ ) = ∫ exp( -f ( x i , m i ; θ )) d x i d m i is the partition function to make it a valid pdf. We further assume the joint probability of all entries are independent, i.e. , p ( X , m ) = ∏ i ∈ Ω p ( x i , z 1 i 1 , . . . , z D i D ) , where Ω denotes the set of observed entries. This is a standard setting in TDs and the dependence of tensor entries can be captured by sharing latent factors.

The expressive nature of the energy function enables us to easily handle diverse data types. For example, we can deal with discrete data by plugging one-hot codings into Eq. (2) to represent categorical probabilities. Additionally, the flexibility of NNs allows us to model tensors with side information, where each tensor entry incorporates additional features [34]. Specifically, in this paper, we focus on a particular case of dynamic tensors with continuous time stamps [48]. In this case, we consider an observed tensor as a time series X t , where the time stamp t is continuous, and each entry i has its own specific time stamp t i . To model the tensor time series, we assume that each entry follows the same distribution and construct the time-dependent energy function, p ( x i , m i ; θ, t i ) ∝ exp( -f ( x i , m i , t i ; θ )) , where the time stamp t i is considered as an auxiliary feature. The flexibility of NNs allows this function to learn general patterns across continuous time stamps. Experimental results demonstrate that this simple treatment can achieve good performances.

Network architecture The network architecture plays a crucial role in learning accurate probabilistic manifolds. Specifically, we define the energy function as f ( x i , m i ) = g 1 ( g 2 ( g 3 ( x i ) , g 4 ( m i ))) , where g 3 and g 4 are MLP layers that encode information from x i and m i , respectively. g 2 is a summation or concatenation layer that induce coupling between tensor values and latent factors, and g 1 is the output layer. Although we currently utilize only MLPs, it is worth noting that convolutional architectures, as demonstrated by [39, 25], can also be employed, which is a topic for future research. To handle dynamic tensors, we incorporate an extra sinusoidal positional encoding layer [37] denoted as g 5 ( t ) to capture temporal information. This embedding utilizes random Fourier features as proposed by [31]. Consequently, the energy function can be expressed as f ( x i , m i , t i ) = g 1 ( g 2 ( g 3 ( x i ) , g 4 ( m i ) , g 5 ( t i ))) . This architecture is commonly employed to capture temporal information and has been demonstrated to effectively learn high-frequency information when combined with MLPs [37].

Posterior sampling A significant application of TDs is to estimate the posterior of missing entries. Unlike traditional TDs, direct predictions cannot be obtained even after learning the latent factors due to the utilization of an undirected probabilistic model. Instead, we need to seek for sampling methods of p ( x i | m i ) . One choice is score-based samplers by utilizing the score function ▽ x i log p ( x i | m i ) = ▽ x i log p ( x i , m i ) p ( m i ) = -▽ x i f ( x i , m i ) , such as Langevin dynamics [44]. Score-based samplers are not suitable for handling discrete data. However, in our case, we model the one-dimensional pdf for each entry, enabling us to directly sample the discrete data. Consequently, for continuous data, the use of grid search is a viable approach to obtain maximum a posteriori (MAP) estimations.

3.2 Learning objective

Despite the flexibility of the proposed model in Eq. (2), obtaining maximum likelihood estimation (MLE) becomes doubly intractable, as both the partition function Z ( θ ) and the marginal distribution p ( x i ) are intractable. Therefore, the CNCE loss Eq. (1) cannot be directly applied. In this section, we extend the variational approach [33] to construct a upper bound that addresses the challenge posed by intractable marginal distributions.

Denote the unnormalized pdf as ϕ ( x i , m i ; θ ) = exp( -f ( x i , m i ; θ )) and the unnormalized marginal pdf as ϕ ( x i ; θ ) = ∫ ϕ ( x i , m i ; θ ) d m i . For clarity, we omit the index i in the subsequent context of this subsection. We follow the idea of CNCE [3] to distinguish data x from conditional noises y ∼ p c ( y | x ) . Firstly, Eq. (1) can be rewritten as

where

However, for our problem, the unnormalized marginal probability ϕ ( x ; θ ) is unknown. An additional variational distribution q ( m ; φ ) is used to approximate the true posterior p ( m | X ; θ ) . Note that in TDs where the data size is static, there is no need for amortized inference, which is different

from previous ones like [2]. Equipped with the variational distribution, the unnormalized marginal distribution can be computed using importance sampling,

Plugging Eq. (5) into Eq. (4), we have

Since Eq. (3) is a convex function w.r.t. r ( x, y ; θ ) , plugging Eq. (6) into Eq. (3) and applying the Jensen's inequality, we have the upper bound,

Following [33], we have the theorem about the tightness of the bound.

Theorem 1 The difference between the VCNCE loss Eq. (7) and CNCE loss Eq. (1) is the expectation of the f -divergence,

The proof can be found in appendix. Based on the theorem, we have the following corollaries to justify the optimization process.

Corollary 1 When q ( m ; φ ) equals to the true posterior, the CVNCE bound is tight, i.e.,

Corollary 2 The following two optimization problems are equivalent,

In practice, we need to seek for the sampled version of Eq. (7). Supposing we have N observed samples { x i } N i =1 , ν noises { y i ,j } ν j =1 for each sample x i and using importance sampling for ϕ ( y ; θ ) , the sampled objective function is,

Specifically, we formulate q ( m ; φ ) as a diagonal Gaussian and use reparameterization trick [18] to compute the expectation. When dealing with continuous data, we typically select conditional Gaussian noises, represented as p c ( y | x ) = N ( y | x, σ 2 ) . This choice entails only one hyperparameter σ that needs to be tuned. Another benefit is the symmetry of the conditional distribution for Gaussian noise, expressed as p c ( y | x ) = p c ( x | y ) . Hence, the objective function can be further reduced. For binary or categorical data, such symmetric noises can also be derived [3].

The time complexity of the proposed objective is O ( νB ( DRH + LH 2 )) , where B is the batch size, ν is the number of conditional noises, H is the number of hidden units per layer, L is the number of layers and D is the tensor order. The time complexity of our model is ν times greater than traditional TDs, since we need to compute forward passes for ν particles. However, as we only use small networks, the computational speed is still very fast (See Appendix C.3 for an illustration).

4 Related work

Traditional tensor decompositions (TDs) are based on multi-linear contraction rules, such as CP [15], Tucker [42], tensor networks [29, 51] and their variations [19, 6]. In this paper, we mainly focus on probabilistic TDs, which extend traditional methods by providing uncertainty estimates about both observations and latent factors [32, 49, 50, 26, 38]. These models build directed mapping from latent factors to tensor entries using multi-linear contraction rules, resulting in limited flexibility when dealing with complex datasets. An alternative approach involves replacing the multi-linear relations with non-linear ones. [5, 45, 52] introduced the use of tensor-variate Gaussian processes (GPs) for achieving nonparametric factorization. [53] further expanded on this concept by incorporating a GP prior on the function that maps latent factors to tensor entries, resulting in a nonparametric TD for sparse tensors. GP-based TDs are further extended using hierarchical priors [40], stochastic processes [43, 8]. Despite the success of GP-based TDs, nonparametric approaches can encounter computational challenges and may still lack sufficient flexibility. Recently, neural networks (NNs) are also applied to TDs. [25] suggested the utilization of convolutional NNs to map latent factors to tensor entries. Besides, [7] built a hierarchical version of the Tucker model and introduced non-linear mappings within each hierarchy of latent factors. To mitigate overfitting, [39] suggested the adoption of deep kernels in GP-based TD rather than using NNs directly. On the other hand, [9] proposed to use Bayesian NN with spike-and-slab prior to prevent from overfitting and obtain probabilistic estimates. More recently, [24] adopt neural ODEs to capture dynamic tensor trajectories. Other works regarding more flexible exponential families [13] or mixture of Gaussians [12] employ linear structures. While all these methods using directed mapping from latent factors to tensor entries, our model is fundamentally different from them, in that we construct much more flexible undirected probabilistic model of TD that can deal with diverse distributions and structures.

Another related direction is the energy-based model (EBM). To address the intractable pdf of EBMs, various training methods have been proposed, including contrastive divergence [CD, 14], score matching [SM, 16], noise-contrastive estimation [NCE, 11]. CD requires large steps of Monte Carlo samples, which can be computationally expensive for high-dimensional tensors. SM cannot handle discrete data, and learning latent variables with SM requires complex bi-level optimization [2]. Therefore, we focus on NCE in this paper. Learning energy-based TD is even more challenging because it involves multiple coupled latent factors that cannot be analytically marginalized. [33] proposed VNCE to handle unnormalized models with latent factors. We enhance their algorithm by using conditional noises [3] to improve the learning efficiency. One fundamental distinction between our model and traditional EBMs is that, through TD construction, we only need to learn one-dimensional distributions for each scalar entry, instead of the original high-dimensional tensor. Hence, our model avoids performance degradation when learning high-dimensional data using NCE.

5 Experiments

We demonstrate the proposed energy-based TD (EnergyTD) on synthetic data and several real-world applications. All the experiments are conducted on a Linux workstation with Intel Xeon Silver 4316 CPU, 256GB RAM and NVIDIA RTX A5000 GPUs (24GB memory each). The code is implemented based on PyTorch 1.12.1 [30]. More experimental details can be found in the appendix. The code is available at https://github.com/taozerui/energy_td

5.1 Simulation study

Tensors with non-Gaussian distributions Traditional TDs commonly assume that tensor entries follow a Gaussian distribution. However, in real-world applications, we often encounter highly complex distributions. In this experiment, we evaluate the capability of our model to learn distributions that deviate from the Gaussian assumption.

We consider a two-mode tensor of shape I × I , where we set I = 8 . Firstly, two latent factors of shape I × R are generated, where the rank R is set to 5 . Then, conditioned on the latent factors, we generated tensor observations from particular distributions. For each entry, we generate N = 200 samples. Three types of distributions are considered: (1) Beta distribution; (2) Mixture of Gaussians (MoG) and (3) Exponential distribution. For Beta distribution, we generate latent factors from uniform distribution Z i =1 , 2 iid ∼ Uni (0 . 0 , 1 . 1) . Then, we sample the observed tensor from Beta

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

2.0

Density

(a) Beta distribution

-1.0

0.0

1.0

2.0

0.0

1.0

2.0

3.0

Density

(b) Mixture of Gaussians

-1.0

0.0

1.0

2.0

3.0

0.0

0.5

1.0

1.5

2.0

Density

Truth

Obs.

GPTF

EnegyTD

(c) Exponential distribution

0.0

0.2

0.4

0.6

0.8

1.0

Time

0.0

0.5

1.0

1.5

2.0

(a) MR

= 0

.

3

0.0

0.2

0.4

0.6

0.8

1.0

Time

0.0

0.5

1.0

1.5

2.0

(b) MR

= 0

.

1

0.0

0.2

0.4

0.6

0.8

1.0

Time

0.0

0.5

1.0

1.5

2.0

Train

Test

Truth

BCTT

NONFAT

EnergyTD

(c) Fully observed

distribution x ij iid ∼ Beta (( Z 1 Z 2 , ⊺ ) ij , 1 . 2) . For MoG distribution, we draw latent factors from uniform distribution Z i =1 , 2 iid ∼ Uni (0 . 0 , 1 . 0) . The tensor entries are then drawn from MoG x ij iid ∼ 0 . 6 · N (cos(( Z 1 Z 2 , ⊺ ) ij ) , 0 . 1 2 ) + 0 . 4 · N (sin(( Z 1 Z 2 , ⊺ ) ij ) , 0 . 25 2 ) . For Exponential distribution, we generate latent factors from uniform distribution Z i =1 , 2 iid ∼ Uni (0 . 0 , 1 . 0) . The tensor entries are then sampled from Exponential distribution x ij iid ∼ Exp (( Z 1 Z 2 , ⊺ ) ij ) . We compare with GP tensor factorization [GPFT, 53], which assumes the entries follow Gaussian distribution.

The results of probability density function (pdf) estimation are presented in Fig. 1. We display the learned pdf of one single tensor entry. This reveals that GPTF is limited to capturing only the 1st moments (mean values) and overlooks higher-order information of more complex distributions. Our model exhibits greater flexibility in handling non-Gaussian distributions.

Continuous-time tensors We then consider a dynamic tensor, where each entry is a time series. We follow similar setting with [9] and use the same data size with the previous simulation, i.e. , a two-mode tensor of shape 8 × 8 with each entry being a time series of length 200 . We firstly generate latent factors of shape 8 × 2 , with each rows drawn from z 1 i ∼ N ([0 , 2] , 2 · I ) and z 2 i ∼ N ([1 , 1] , 2 · I ) . Then we generate N = 200 observed entries from time t ∈ [0 , 1] . The tensor entries are computed by x i ( t ) = ∑ 2 r 1 =1 ∑ 2 r 2 =1 z 1 i 1 r 1 z 2 i 2 r 2 ω r 1 r 2 ( t ) , where ω 11 ( t ) = sin(2 πt ) , ω 12 ( t ) = cos(2 πt ) , ω 21 ( t ) = sin 2 (2 πt ) and ω 22 ( t ) = cos(5 πt ) sin 2 (5 πt ) . Finally, the data are normalized to range [0 , 2] . The synthetic data consist of low-frequency trends and high-frequency fluctuations. Apart from fully observed case, we also test with missing rates (MR) 10% and 30% . In specific, for each entry, we randomly select a starting time and set the following consecutive 10% or 30% time stamps as missing.

We compare with two methods that are designed for dynamic tensors, the Bayesian continuous-time Tucker decomposition [BCTT, 8] and the nonparametric factor trajectory learning [NONFAT, 43].

BCTT treats Tucker core tensors as functions of time and NONFAT treats all GPTF factors as time series. Unlike BCTT with GP prior on the time domain, NONFAT uses GP prior on the frequency domain through inverse Fourier transform of original time series.

Fig. 2 displays the completion results. The learned trajectory of a single tensor entry is plotted. Higher missing rates result in the inability of BCTT to capture accurate trajectories, particularly in missing regions. NONFAT achieves more stable predictions, yet it tends to favor over-smoothed trajectories while disregarding high-frequency fluctuations. This behavior may be attributed to its unique construction, which introduces a GP prior in the frequency domain. Our utilization of flexible neural networks allows us to adapt to complex situations encompassing both low-frequency and high-frequency information.

5.2 Tensor completion

We evaluate our model on two sparse tensor and two dynamic tensor completion applications. For real datasets, the energy function can be difficult to learn, when the pdf has a very sharp curve. Motivated by the idea of noise-perturbed score estimation [35], we add small i.i.d. Gaussian noises on the data during the training of EnergyTD as a form of smoothing technique. The results are reported on clean test data. For EnergyTD, we use MAP estimates as described in Section 3.1.

5.2.1 Sparse tensor completion

We test our model on two sparsely observed tensors: (1) Alog , a file access log dataset [52] of shape 200 users × 100 actions × 200 resources with about 0.33% nonzero entries; (2) ACC , a three-way tensor generated from a code repository management system [52] of shape 3 k users × 150 actions × 30 k resources with about 0.009% nonzero entries. We use the same dataset split as in [52] and report the 5-fold cross validation results.

Competing methods We compare with five baselines: (1) CP-WOPT [1], CP decomposition with stochastic optimization; (2) GPTF [53], a GP-based tensor factorization using stochastic variational inference; (3) HGP-GPTF [40], a GPTF equipped with hierarchical Gamma process prior; (4) POND [39], a probabilistic non-linear TD using deep kernels with convolutional NNs (CNNs); (5) CoSTCo [25], a non-linear TD that uses CNNs to map latent factors to tensor entries. CP-WOPT is provided in Matlab Tensor Toolbox [1]. We implement GPTF based on PyTorch by ourselves and use official implementations for HGP-GPTF 2 , POND 3 and CoSTCo 4 .

Experimental settings and results We set batch size 1000 and run 1000 epochs for Alog , 100 epochs for ACC . For our model, we use Adam [17] optimizer. Learning rates of all models are chosen from { 1e -2 , 1e -3 , 1e -4 } . For all methods, we evaluate with rank R ∈ { 3 , 5 , 8 , 10 } . All methods are evaluated by 5 runs with different random seeds.

The completion results are shown in Table 1. The mean and standard deviation of the root mean square error (RMSE) and mean absolute error (MAE) are reported. Results from POND are not included for ACC due to the slow code execution and the substantial memory requirements for this dataset. Our model outperforms the baselines in nearly all cases on both RMSE and MAE. Moreover, the improvement is significant ( p < 0.05) for most cases. All non-linear methods exhibit substantial superiority over CP-WOPT, thereby highlighting the advantage of employing more flexible structures. Furthermore, the enhanced performance of our model may be attributed to the utilization of more flexible undirected probabilistic distributions. It is worth noting that our model, despite employing MLP layers, outperforms POND and CoSTCo, which utilize convolutional layers. We believe that our model can be further improved by designing more suitable network architectures.

5.2.2 Continuous-time tensor completion

In this subsection, we evaluate our model on two continuous-time tensor datasets: (1) Air , the Beijing air quality dataset [47] of shape 12 sites × 6 pollutants with about 1 × 10 4 observations at different time stamps; (2) Click , an ads click dataset [43] of shape 7 banner positions × 2842 site domains × 4127 mobile APPs with about 5 × 10 4 entries at different time stamps. We use the same dataset split as in [43] and report the 5-fold cross validation results.

Competing methods We compare with (1) Nonparametric factor trajectory learning [NONFAT, 43], which is the SOTA method for continuous-time tensor completion, and other baselines including: (2) Continuous-time CP [CTCP, 48], which uses polynomial splines to model dynamics of CP coefficients; (3) Continuous-time GP (CTGP), which extends GPTF [53] by injecting time stamps into GP kernels; (5) Continuous-time NN decomposition (CTNN), which directly uses time stamps as inputs in CoSTCo [25] to learn continuous dynamics; (5) Discrete-time NN decomposition with non-linear dynamics [NNDTN, 43], which employs RNN dynamics for time steps; (6) Tensor highorder interaction learning via ODEs [THIS-ODE, 24], where continuous-time trajectories of tensor entries are captured by neural ODEs. For baseline models (1-5), we use the official implementation 5 provided by [43]. For (6) THIS-ODE, its official implementation 6 is used.

Experimental settings and results We set batch size 128 and run 400 epochs for Air , 200 epochs for Click . Notably, in the case of THIS-ODE, we only run 200 epochs for Air and 75 epochs for Click due to the slow speed of ODE solvers (See Appendix C.3 for an illustration). For our model, we use Adam [17] optimizer with learning rate chosen from { 1e -2 , 1e -3 } . The network architecture is described in Section 3.1. For baseline models, provided settings in their codebases are used. For all methods, we evaluate with rank R ∈ { 3 , 5 , 8 , 10 } . All methods are evaluated by 5 runs with different random seeds.

Table 2 presents the completion results. It should be noted that the results presented here differ from those reported in [43] as we adhere to the standard definition of RMSE and MAE (see appendix for detail). Our model surpasses the baseline methods in almost all cases for both RMSE and MAE, with statistically significant improvements ( p < 0.05) observed in most cases. Particularly, we observe that the improvements in MAE are notably more significant. One possible reason is that NONFAT is trained implicitly by minimizing the square loss (with regularization) as it adopts Gaussian assumption about the data. However, our model does not make such assumptions about the data distribution and the loss function. Hence, it can adapt to the data distribution more flexibly. Additionally, we observe that directly injecting time stamps into neural networks, as done in CTNN, is ineffective, thus highlighting the advantage of our model in learning more informative structures.

6 Conclusion

We introduce an innovative approach to undirected probabilistic tensor decomposition (TD), characterized by its exceptional flexibility in accommodating various structures and distributions. Specifically, our model integrates deep EBMs in TD to relax both structural and distributional assumptions, enabling it to handle complex real-world applications. To efficiently learn the doubly intractable pdf, we derive a VCNCE objective that is the upper bound of the CNCE loss. Experimental results demonstrate that our model can handle diverse distributions and outperforms baseline methods in multiple real-world applications. One limitation is that our final loss function is not a fully variational upper bound of CNCE, since we have to use importance samples to approximate the pdf of noise samples in Eq. (7). In the future, we aim to derive a fully variational bound as in [46]. Finally, we did not delve into the interpretability of learned factors in this work. However, exploring the interpretability of these factors represents a promising avenue for future research in the realm of tensor decompositions.

Acknowledgments

Zerui Tao was supported by the RIKEN Junior Research Associate Program. This work was supported by the JSPS KAKENHI Grant Numbers JP20H04249, JP23H03419.

References

Brett W Bader and Tamara G Kolda. Efficient matlab computations with sparse and factored tensors. SIAM Journal on Scientific Computing , 30(1):205-231, 2008.

Fan Bao, Chongxuan Li, Kun Xu, Hang Su, Jun Zhu, and Bo Zhang. Bi-level score matching for learning energy-based latent variable models. Advances in Neural Information Processing Systems , 33:18110-18122, 2020.

Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. In International Conference on Machine Learning , pages 726-734. PMLR, 2018.

Rong Chen, Dan Yang, and Cun-Hui Zhang. Factor models for high-dimensional tensor time series. Journal of the American Statistical Association , 117(537):94-116, 2022.

Wei Chu and Zoubin Ghahramani. Probabilistic models for incomplete multi-dimensional arrays. In Artificial Intelligence and Statistics , pages 89-96. PMLR, 2009.

Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, Danilo P Mandic, et al. Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions. Foundations and Trends® in Machine Learning , 9(4-5):249-429, 2016.

Jicong Fan. Multi-mode deep matrix and tensor factorization. In international conference on learning representations , 2021.

Shikai Fang, Akil Narayan, Robert Kirby, and Shandian Zhe. Bayesian continuous-time tucker decomposition. In International Conference on Machine Learning , pages 6235-6245. PMLR, 2022.

Shikai Fang, Zheng Wang, Zhimeng Pan, Ji Liu, and Shandian Zhe. Streaming bayesian deep tensor factorization. In International Conference on Machine Learning , pages 3133-3142. PMLR, 2021.

Ivan Glasser, Ryan Sweke, Nicola Pancotti, Jens Eisert, and Ignacio Cirac. Expressive power of tensor-network factorizations for probabilistic modeling. Advances in neural information processing systems , 32, 2019.

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 297-304. JMLR Workshop and Conference Proceedings, 2010.

Zhi Han, Yao Wang, Qian Zhao, Deyu Meng, Lin Lin, Yandong Tang, et al. A generalized model for robust tensor factorization with noise modeling by mixture of gaussians. IEEE transactions on neural networks and learning systems , 29(11):5380-5393, 2018.

Kohei Hayashi, Takashi Takenouchi, Tomohiro Shibata, Yuki Kamiya, Daishi Kato, Kazuo Kunieda, Keiji Yamada, and Kazushi Ikeda. Exponential family tensor factorization for missingvalues prediction and anomaly detection. In 2010 IEEE International Conference on Data Mining , pages 216-225. IEEE, 2010.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

Frank L Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics , 6(1-4):164-189, 1927.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review , 51(3):455-500, 2009.

Maxim Kuznetsov, Daniil Polykovskiy, Dmitry P Vetrov, and Alex Zhebrak. A prior of a googol gaussians: a tensor ring induced prior for generative models. Advances in Neural Information Processing Systems , 32, 2019.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Chao Li and Zhun Sun. Evolutionary topology search for tensor network decomposition. In International Conference on Machine Learning , pages 5947-5957. PMLR, 2020.

Chao Li, Junhua Zeng, Zerui Tao, and Qibin Zhao. Permutation search of tensor network structures via local sampling. In International Conference on Machine Learning , pages 1310613124. PMLR, 2022.

Shibo Li, Robert Kirby, and Shandian Zhe. Decomposing temporal high-order interactions via latent ODEs. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12797-12812. PMLR, 2022.

Hanpeng Liu, Yaguang Li, Michael Tsang, and Yan Liu. Costco: A neural tensor completion model for sparse tensors. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 324-334, 2019.

Zhen Long, Ce Zhu, Jiani Liu, and Yipeng Liu. Bayesian low rank tensor ring for image recovery. IEEE Transactions on Image Processing , 30:3568-3580, 2021.

Jacob Miller, Guillaume Rabusseau, and John Terilla. Tensor networks for probabilistic sequence modeling. In International Conference on Artificial Intelligence and Statistics , pages 3079-3087. PMLR, 2021.

Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems , 28, 2015.

Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing , 33(5):2295-2317, 2011.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Piyush Rai, Yingjian Wang, Shengbo Guo, Gary Chen, David Dunson, and Lawrence Carin. Scalable bayesian low-rank decomposition of incomplete multiway tensors. In International Conference on Machine Learning , pages 1800-1808. PMLR, 2014.

Benjamin Rhodes and Michael U Gutmann. Variational noise-contrastive estimation. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2741-2750. PMLR, 2019.

Qingquan Song, Hancheng Ge, James Caverlee, and Xia Hu. Tensor completion algorithms in big data analytics. ACM Transactions on Knowledge Discovery from Data (TKDD) , 13(1):1-48, 2019.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 2021.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

Zerui Tao, Xuyang Zhao, Toshihisa Tanaka, and Qibin Zhao. Bayesian latent factor model for higher-order data. In Asian Conference on Machine Learning , pages 1285-1300. PMLR, 2021.

Conor Tillinghast, Shikai Fang, Kai Zhang, and Shandian Zhe. Probabilistic neural-kernel tensor decomposition. In 2020 IEEE International Conference on Data Mining (ICDM) , pages 531-540. IEEE, 2020.

Conor Tillinghast, Zheng Wang, and Shandian Zhe. Nonparametric sparse tensor factorization with hierarchical gamma processes. In International Conference on Machine Learning , pages 21432-21448. PMLR, 2022.

Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Compressing recurrent neural network with tensor train. In 2017 International Joint Conference on Neural Networks (IJCNN) , pages 4451-4458. IEEE, 2017.

Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika , 31(3):279-311, 1966.

Zheng Wang and Shandian Zhe. Nonparametric factor trajectory learning for dynamic tensor decomposition. In International Conference on Machine Learning , pages 23459-23469. PMLR, 2022.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688, 2011.

Zenglin Xu, Feng Yan, and Yuan Qi. Infinite tucker decomposition: nonparametric bayesian models for multiway data analysis. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1675-1682, 2012.

Christopher Zach. Fully variational noise-contrastive estimation. In Image Analysis: 23rd Scandinavian Conference, SCIA 2023, Sirkka, Finland, April 18-21, 2023, Proceedings, Part II , pages 175-190. Springer, 2023.

Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality improvement in beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 473(2205):20170457, 2017.

Yanqing Zhang, Xuan Bi, Niansheng Tang, and Annie Qu. Dynamic tensor recommender systems. The Journal of Machine Learning Research , 22(1):3032-3066, 2021.

Qibin Zhao, Liqing Zhang, and Andrzej Cichocki. Bayesian cp factorization of incomplete tensors with automatic rank determination. IEEE transactions on pattern analysis and machine intelligence , 37(9):1751-1763, 2015.

Qibin Zhao, Liqing Zhang, and Andrzej Cichocki. Bayesian sparse tucker models for dimension reduction and tensor completion. arXiv preprint arXiv:1505.02343 , 2015.

Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decomposition. arXiv preprint arXiv:1606.05535 , 2016.

Shandian Zhe, Zenglin Xu, Xinqi Chu, Yuan Qi, and Youngja Park. Scalable nonparametric multiway data analysis. In Artificial Intelligence and Statistics , pages 1125-1134. PMLR, 2015.

Shandian Zhe, Kai Zhang, Pengyuan Wang, Kuang-chih Lee, Zenglin Xu, Yuan Qi, and Zoubin Ghahramani. Distributed flexible nonlinear tensor factorization. Advances in neural information processing systems , 29, 2016.

A Proof of Theorem 1

Firstly, we give the definition of f -divergence.

Definition 1 ( f -divergence) The f -divergence between two probability density functions (pdf) p and q is defined as,

where f : [0 , ∞ ) → R is a convex function and f (1) = 0 .

As shown in [33], since partition functions for ϕ ( x, m ; θ ) and ϕ ( x ; θ ) are the same, we have the following factorization,

The difference between the two objective becomes,

where

with κ xy = ϕ ( x ; θ ) p c ( y | x ) ϕ ( y ; θ ) p c ( x | y ) . It is straightforward to verify that f (1) = 0 . The derivatives of f is

Since κ and u are positive, f is a convex function. Therefore, f satisfy the requirements of f -divergence.

B Proof of Corollaries 1 and 2

Corollary 1 is a straightforward consequence of Theorem 1. Since the f -divergence becomes zero if and only if the two distributions are identical, we have,

Moreover, since the f -divergence is positive and Theorem 1, we have

Then, plugging the optimal distribution gives the tight bound, we have,

C Experimental details

C.1 Simulation study

Tensors with non-Gaussian distributions For both GPTF and our model, we set batch size to 1000 and run 500 epochs with Adam optimizer. The initial learning rate is 1e -3 and subsequently reduced

by 0.3 at 60% , 75% and 90% of the maximum epochs. Moreover, the rank is set to 3 for both models. For GPTF, radial basis function (RBF) kernel with band width 1.0 is used, where 100 inducing points is adopted for approximation. For the conditional distribution p ( x i | m i ) = N ( x i | f ( m i ) , σ 2 ) in GPTF, σ is fixed and chosen as the sample standard variance. For our model, we use 5 hidden layers of width 64 for both g 1 , g 3 and g 4 defined in Section 3. g 2 is a summation layer. We use ELU activation for non-linearity. For the VCNCE loss, the conditional noise distribution is set as p c ( y | x ) = N ( y | x, 0 . 3 2 ) and ν = 10 noise samples are used for each data point.

Continuous-time tensors The data sizes and optimization parameters are the same with the previous simulation. The rank of all models are set to 3. For NONFAT, 100 inducing points are used to approximate the kernel function. We run the NONFAT model for 5000 epochs because we find that the algorithm converges very slowly. Other hyper-parameters are chosen by their default settings. For BCTT, we do not modify their code and settings. For our model, we use 3 hidden layers of length 64 with ELU activation. The conditional noise distribution in the VCNCE loss is set to p c ( y | x ) = N ( y | x, 1) and ν = 20 noise samples are used for each datum.

C.2 Tensor completion

For all datasets, when training our model, we scale the data to [0 , 1] based on the training data. For testing, we multiply the scale statistic computed by the training data and evaluate the performance on the original domain. We do not employ such data normalization for baselines models, because that will influence their default settings.

C.2.1 Sparse tensor completion

For both Alog and ACC , the batch size is set to 1000. We run 1000 epochs for Alog and 100 epochs for ACC due to their different sample numbers. For Alog dataset, we add i.i.d. Gaussian noises from N (0 , 0 . 05 2 ) during training, while for ACC , the standard variance is set to 0 . 02 . The Adam optimizer is used with learning rate chosen from { 1e -2 , 1e -3 , 1e -4 } . We also use gradient clip with maximum infinity norm of 2.0 for training stability. Moreover, we use learning rate scheduler by reducing the initial learning rate by 0.3 at 40%, 60%, and 80% of the total iterations. For both datasets, we use 2 hidden layers of length 50 with ELU activation for g 1 , g 3 and g 4 for our model. For the VCNCE loss, we set ν = 20 noise samples with noise variance tuned from { 0 . 3 2 , 0 . 5 2 , 0 . 8 2 , 1 . 0 2 } . In practice, we find that the noise variance is influential to the final performance, even we are using conditional noises. However, with VCNCE, there is only one hyper-parameter for the noise distribution. While for CNCE, one may need to tune both mean and variance of the noise.

C.2.2 Continuous-time tensor completion

For Air and Click datasets, we set batch size to 128 . We run 400 epochs for Air and 200 epochs for Click due to their different data sizes. For Alog dataset, we add i.i.d. Gaussian noises from N (0 , 0 . 05 2 ) during training, while for ACC , the variance is set as 0 . 15 2 . To encode the temporal information into the energy function, we use the sinusoidal positional encoding, as described in Section 3. Other settings are the same with Appendix C.2.1.

It should be noted that we use the standard definition of root mean square error (RMSE) and mean absolute error (MAE), namely,

where x i is the ground truth and ˆ x i is the estimate. Therefore, the results are different from those presented in [43], where the authors used relative versions of RMSE and MAE,

We modify the evaluation part of their code 7 and report the results.

C.3 Computational time

The time complexity of our model is ν times greater than traditional TDs, since we need to compute forward passes for ν particles. However, as we only use small networks, the computational speed is still very fast. To illustrate, we compare the runtime performance of several baselines and our model on a single RTX A5000 GPU. We conduct tests on the Air dataset, with a batch size of 128 and tensor rank of 5. The reported running time is the average of the first 10 epochs. For our model, we set ν = 20 . The default settings are used for other baselines. Table 3 lists the computational time of CTGP, NNDTN, NONFAT and THIS-ODE, all of which perform better than other baselines. The results show that our model achieves better performances within a reasonable time, especially compared to THIS-ODE.

C.4 Ablation study on the objective function

We conduct an additional ablation study to show the advantage of VCNCE over the variational noisecontrastive estimation [VNCE, 33] objective. The main difference between the VNCE and VCNCE is that VNCE uses noises from a fixed Gaussian distribution, e.g. , y ∼ p n ( y ) = N ( y | µ, σ 2 ) , while VCNCE uses conditional noises, e.g. , y ∼ p c ( y | x ) = N ( y | x, σ 2 ) . Hence, these two strategies yield different objective functions. The objective function of VNCE is defined as

where p n ( · ) is the fixed noise distribution. For VNCE, choosing inappropriate noise distributions may result in bad performances.

We test the proposed model on the Air dataset, training on the VCNCE loss and VNCE loss, respectively. We set the batch size to 128 and run 400 epochs. Adam optimizer with initial learning rate 1e -2 is adopted. The initial learning rate is subsequently reduce by 0.3 at 20% , 50% and 80% of the total epochs. For VNCE, we set µ = 0 , which is a common practice in relevant literature. To show how the noise variance affects the learning process, we test different noise variances, e.g. , σ ∈ { 0 . 3 , 0 . 5 , 0 . 7 } for both VNCE and VCNCE. Other settings are the same with Appendix C.2.2.

Fig. 3 depicts the RMSE and MAE on the test data when optimizing VNCE and VCNCE objective functions. We test five runs, plot mean values in lines and standard deviations in shadowed areas. It is shown that VCNCE gets better and more stable results on both RMSE and MAE.

0

100

200

300

400

Epoch

0.3

0.5

0.7

RMSE

0

100

200

300

400

Epoch

0.3

0.5

0.7

RMSE

0

100

200

300

400

Epoch

0.3

0.5

0.7

RMSE

0

100

200

300

400

Epoch

0.2

0.3

0.4

0.5

MAE

VNCE

VCNCE

(a) Noise

σ

= 0

.

3

0

100

200

300

400

Epoch

0.2

0.3

0.4

0.5

MAE

(b) Noise

σ

= 0

.

5

0

100

200

300

400

Epoch

0.2

0.3

0.4

0.5

MAE

(c) Noise

σ

= 0

.

7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new Probabilistic Tensor Decomposition (TD) method. By modeling the joint probability of the data and latent tensor factors using an Energy Based approach, they make no (possibly restrictive) structural and distributional assumptions on the generative process linking latent and observations. The model is trained using an upper bound of the Conditional Noise-Contrastive Estimation Loss. It is then evaluated on simulations using non Gaussian generative models and on tensor completion task on real world datasets.\n\nSTRENGTHS:\nThe possibility to bypass the specification of both structural and distributional dependencies is very interesting. It has many applications beyond the tensor completion tasks, for example when analyzing datasets with complicated dependencies on significantly simpler latent variables. \n\nThe simulated datasets used in the paper are simple but illustrate this point well. \n\nResults on real world tensor completion tasks are convincing.\n\nWEAKNESSES:\nMajor Points:\n\n1) The code is not provided. \n\n2) In the introduction, it would be worth motivating the use of probabilistic methods over traditional ones.\n\n3) I did not find the Energy Based Model section (2.2) very well written. In particular, it is not clear how to choose the conditional $p_c(y|x)$  to improve efficiency. \n\n4) The method is only evaluated on tensor completion tasks. Do the inferred factors have any explanatory value ? How robust is the decomposition ?\n\n5) There is no discussion on the compute time of the proposed method.\n\nMinor Points:\n\n1) l.43 It is worth mentioning that Generalized CP decomposition exist but that most of them do not allow a probabilistic treatment of the data. Moreover, beyond Gaussian and Binomial, tractable decomposition have been developed for Negative Binomial Distributions.\n\n2) l.93 I think the authors implicitly assume $R_d = R$ for all $d$ without explicitly saying it.\n\n3) Experimental details in the main paper and supplementary are not consistent (experiment 1: main paper $R=5$, supplementary $R=3$).\n\nQUESTIONS:\n1) The paper only discusses 2 and 3-way tensors. How does the method scale with higher order tensors ?  \n\n2) Beside Tensor completion, an interesting aspect of TD methods is often to explain a dataset $\\mathcal{X}$ using simpler yet unobserved factors $\\mathcal{Z}$. How robust are the inferred factors depending on the initialization of the model ? Can you measure it using e.g. a similarity metric and compare it to existing methods ?\n\n3) The authors argue that the use of overly simple generative model $p(x|z)$ can bias the TD decomposition. They provide convincing illustration on their toydataset but it would be interesting to better understand why it is the case on the tested the real world dataset.  \n\n4) The approach developed by the authors seems extremely flexible. Yet, by allowing almost arbitrary complex link between the latent and the observation, one might fear that the inferred latent distribution is \"non-unique\", which might hinder the interpretation of the discovered factors. Can you say more about identifiability ?\n\nLIMITATIONS:\nThere is no discussion on the limitation of the method when the posterior $p(z|x)$ itself is non Gaussian. The approximation $q(m; \\phi)$ is set to be multivariate normal with diagonal covariance, and, so, it cannot model, for example, multi-modal distribution on $z$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

comments: SUMMARY:
This paper proposes a probabilistic tensor decomposition model called EnergyTD that integrates a deep energy-based model (EBM) in tensor decomposition. The EnergyTD does not model the values in a tensor as the conditional probability conditioned on the latent factors based on the predefined models but models them as the joint probability of the data and the latent factors. It provides a learning algorithm that extends the CNCE loss using the variational approach. The experimental results show that EnergyTD outperforms conventional methods in the tensor decomposition tasks assuming diverse distributions including continuous-time tensors, tensor completion, and continuous-time tensor completion.

STRENGTHS:
This paper proposes a probabilistic tensor decomposition model that integrates EBM in tensor decomposition along with a theoretical guarantee. The experimental results show that the proposed method outperforms conventional methods in the tensor decomposition tasks assuming diverse distributions. This paper is well-organized and easy to follow.

WEAKNESSES:
The time/space complexity of the proposed method is only poorly discussed, and there is no evaluation regarding the computational cost. This is one of the major concerns in the tensor decomposition research community.

A minor problem: The "difficultly" in line 109 should be "difficulty".

QUESTIONS:
What is the time/space complexity of the proposed method? How is the proposed method computationally effective compared to the conventional methods?

LIMITATIONS:
As the authors describe in the Conclusion, there is room to improve theoretical analysis. Moreover, the architecture of the network structure should be discussed more in the future.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses the Energy Based Model (EBM) framework to capture the joint probability of the data and latent tensor factors to learn as much information from data as possible, which discards the structural and distributional assumptions and thus avoid picking an inappropriate TD model.  To further flexibly learn the unnormalized probability function, the authors derive a variational upper bound of the conditional noise-contrastive estimation objective.  The main contribution lies in inserting the EBM framework into modeling probabilistic Tensor Decompositions.\n\nSTRENGTHS:\nPros:\n\n1. This paper first introduces undirect graphic models into modeling probabilistic tensor decomposition. In detail, the authors consider latent factor matrics as latent factors in the Variational Noise Contrastive Estimation framework, leading to flexible modeling of joint distribution over observations. To the best of my knowledge, this is interesting and novel to the Tensor Decomposition community.\n\n2. This paper is well-written and the technological parts are sound and valid.\n\nWEAKNESSES:\nCons:\n\n1. If what I understand is correct,  one important goal of this paper is to find the $\\theta$ and $p_{\\theta}(x)$. The authors use the framework of variational noise contrastive estimation to obtain the unnormalized function $\\phi(x;\\theta)$. However, it seems to be easier to directly model $\\phi(x;\\theta)$ and thus minimize $\\mathcal{L}_{CNCE}(\\theta)$. Furthermore, it seems that these $\\textbf{m}$s are unnecessary and can have arbitrary dimensions.\n\n2. Most technical contributions are in the form of a combination of existing theory and framework. This actually decreases this paper's contributions.\n\nQUESTIONS:\nQuestions:\n\n1. As mentioned in the last part, I am confused about why not directly model $\\phi(x;\\theta)$ and $p_{\\theta}(x)$.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an innovative approach for non-linear tensor decomposition. It utilizes the deep energy-based model (EBM) to model the joint energy function of tensor observations and latent factors. This design enables a more flexible decomposition without the need for explicitly defining the interaction between the latent factor and the commonly used Gaussian prior. Furthermore, the expressive nature of the energy function allows for the modeling of side information, such as timestamps.\n\nSTRENGTHS:\nThe paper presents a clear motivation and introduces innovative and sophisticated solutions. To my knowledge, this is the first work to adopt Evidence-Based Medicine (EBM) for tensor decomposition without the need for an explicit design of the decomposition form.\n\nFurthermore, the adaptability of the proposed method for dynamic tensors is impressive. The experimental setup is robust and comprehensive, demonstrating the thoroughness of the research.\n\nWEAKNESSES:\n1. Instead of using NON-FAT or BCTT for experiments on dynamic tensors' decomposition and visualization, it would be more appropriate to choose THIS-ODE[1] as the baseline. While NON-FAT and BCTT are designed for modeling temporal dynamics of factors in a tensor, THIS-ODE aligns with the proposed model by having an entry-wise dynamic design.\n \n2. It appears that the objective function involves entry-wise importance sampling, and completing the posterior will require another round of entry-wise sampling. I am unsure if this will lead to training and inference costs becoming problematic, especially in scenarios with a large number of entries. It would be beneficial to have a more in-depth discussion about the scalability aspect.\n\n\n[1] Decomposing Temporal High-Order Interactions via Latent ODEs, Shibo Li, Robert M. Kirby, and Shandian Zhe, The 39th International Conference on Machine Learning, 2022\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}