SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zerui Tao $^{1,2}$  Toshihisa Tanaka $^{1,2}$  Qibin Zhao $^{2,1*}$
zerui.tao@riken.jp tanakat@cc.tuat.ac.jp qibin.zhao@riken.jp
$^{1}$ Tokyo University of Agriculture and Technology  $^{2}$ RIKEN AIP
Tensor decompositions (TDs) serve as a powerful tool for analyzing multiway data. Traditional TDs incorporate prior knowledge about the data into the model, such as a directed generative process from latent factors to observations. In practice, selecting proper structural or distributional assumptions beforehand is crucial for obtaining a promising TD representation. However, since such prior knowledge is typically unavailable in real-world applications, choosing an appropriate TD model can be challenging. This paper aims to address this issue by introducing a flexible TD framework that discards the structural and distributional assumptions, in order to learn as much information from the data. Specifically, we construct a TD model that captures the joint probability of the data and latent tensor factors through a deep energy-based model (EBM). Neural networks are then employed to parameterize the joint energy function of tensor factors and tensor entries. The flexibility of EBM and neural networks enables the learning of underlying structures and distributions. In addition, by designing the energy function, our model unifies the learning process of different types of tensors, such as static tensors and dynamic tensors with time stamps. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability function. To efficiently train the model, we derive a variational upper bound of the conditional noise-contrastive estimation objective that learns the unnormalized joint probability by distinguishing data from conditional noises. We show advantages of our model on both synthetic and several real-world datasets.
Tensor decompositions (TDs) serve as powerful tools for analyzing high-order and high-dimensional data, aiming to capture the inter-dependencies among different modes by utilizing multiple latent factors. TDs have demonstrated remarkable success in various machine learning tasks, including data imputation [53, 9], factor analysis [4], time-series forecasting [27], model compression [28, 41], generative models [10, 20] among others.
Existing TDs typically incorporate predefined directed graphical models into the generative process. These models specify the priors of latent factors and the conditional probabilities of observations, following specific contraction rules associated with the latent factors. Traditional contraction rules predominantly employ multi-linear products, like CP [15], Tucker [42], tensor train [29] and other variants [19, 6]. However, selecting an appropriate contraction rule for specific datasets is often challenging in real-world applications. Recent research, known as tensor network structure search [TNSS, 22, 23], has demonstrated that selecting an appropriate TN contraction rule significantly enhances the factorization performance. Another promising approach involves learning non-linear mappings from the data, utilizing techniques like nonparametric models [5, 45, 53] and deep neural networks
[25, 9]. Empirical results demonstrate that non-linear TDs exhibit superior performance compared to traditional multi-linear TDs in various applications, attributed to their enhanced expressive power.
Despite the success of non-linear TDs in reducing structural assumptions, they often rely on simplistic distributional assumptions. Typically, a specific directed graphical model is adopted to model the generative process from latent factors to tensor entries, represented as  $p(x) = \int p(x \mid z)p(z)\mathrm{d}z$ , where  $z$  denotes tensor factors and  $x$  represents observations. Additionally, the distributions are usually selected from exponential families for tractability, such as Gaussian and Bernoulli distributions. For instance, a Gaussian prior can be assigned to latent factors, and observed entries can be modeled using Gaussian distribution [32, 49] or Gaussian process [45, 53]. However, these prior assumptions regarding the probabilistic model can introduce model bias and reduce the effectiveness of TD models. In real-world applications, the latent factors might originate from unknown distributions, and the observations can exhibit complex multi-modal generative processes. Without knowing the underlying generative process, these simplistic assumptions can lead to inaccurate estimations.
To address these issues, this paper proposes to construct an undirected graphical model of TD. More specifically, a TD model that captures the joint probability of the data and latent tensor factors is constructed through a deep energy-based model (EBM), represented as  $p(x,z) \approx \exp(-f(x,z))$ . Neural networks (NNs) are then employed to parameterize the joint energy function  $f(x,z)$ . The flexibility of EBM and NNs facilitates the learning of underlying structures and distributions. Furthermore, our model unifies the learning process in the presence of side information, such as dynamic tensors with time stamps, by designing the energy function. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability density function (pdf). For efficient model training, we derive a variant of conditional noise-contrastive estimation [3] algorithm that learns the unnormalized joint probability by distinguishing data from conditional noises. The proposed model offers several advantages: (1) it features a flexible structure that can adapt to different distributions; (2) the undirected nature allows us to learn more general correlations than traditional directed TDs; (3) it can handle diverse tasks and encode auxiliary information by adjusting the energy function.
Experiments are conducted on synthetic and real-world datasets to showcase the advantages of our model. Through simulation studies, we demonstrate the capability of our model to handle data generated from diverse distributions, in contrast to traditional Gaussian-based models that yield unfaithful and biased estimates. Subsequently, experiments are performed on multiple real-world datasets to evaluate sparse and continuous-time tensor completion. Our model outperforms various baselines across multiple metrics and settings, highlighting the generality of the proposed model.
Notations We adopt similar notations with [19]. Throughout the paper, we use lowercase letters, bold lowercase letters, bold capital letters and calligraphic bold capital letters to represent scalars, vectors, matrices and tensors, e.g.,  $x$ ,  $\mathbf{x}$ ,  $\mathbf{X}$  and  $\mathbf{X}$ . Tensors refer to multi-way arrays which generalize matrices. For a  $D$ -order tensor  $\mathcal{X} \in \mathbb{R}^{I_1 \times \dots \times I_D}$ , we denote its  $(i_1, \ldots, i_D)$ -th entry as  $x_i$ .
2.1 Tensor decomposition
Given a  $D$ -order tensor  $\mathcal{X} \in \mathbb{R}^{I_1 \times \dots \times I_D}$ , tensor decomposition (TD) aims to factorize  $\mathcal{X}$  into  $D$  smaller latent factors  $\mathbf{Z}^{d=1, \ldots, D} \in \mathbb{R}^{I_d \times R_d}$  by using some predefined tensor contraction rules. The classical Tucker decomposition [42] assumes  $\mathcal{X} = \mathcal{W} \times_1 \mathbf{Z}^1 \times_2 \dots \times_D \mathbf{Z}^D$ , where  $\mathcal{W} \in \mathbb{R}^{R_1 \times \dots \times R_D}$  is the coefficient and  $\times_d$  denotes the matrix-tensor contraction [19]. Equivalently, each entry can be written as  $x_i = \sum_{r_1=1}^{R_1} \dots \sum_{r_D=1}^{R_D} w_{r_1 \dots r_D} z_{i_1 r_1}^1 \dots z_{i_D r_D}^D$ , where the tuple  $(R_1, \ldots, R_D)$  is the Tucker rank of tensor  $\mathcal{X}$ . The latent factors  $\mathbf{Z}^d$  can capture information of each tensor mode and  $\mathcal{W}$  represents the weight of each factors. CP decomposition [15] is a restricted form of Tucker by assuming  $\mathcal{W}$  is super-diagonal, i.e.,  $x_i = \sum_{r=1}^{R} w_r z_{i_1 r}^1 \dots z_{i_D r}^D$ , where we simplify  $w_r = w_{r \dots r}$ . In this paper, we focus on probabilistic version of TDs, which serves as generalizations of traditional ones. The standard approach is to formulate TDs as a directed graphical model,  $p(\mathcal{X}) = \int p(\mathcal{X} | Z) p(Z) \, \mathrm{d}Z$ , where  $Z$  denotes  $\{Z^1, \ldots, Z^D\}$  for simplicity. For continuous data, the  $p(\mathcal{X} | Z)$  is usually assumed to be Gaussian and TDs are used to parameterize the mean of corresponding Gaussian distribution [32, 49, 50].
Despite the elegant form of these multi-linear contraction rules, they have limited flexibility that can be mitigated by extending TDs to their non-linear counterparts. We can think of TD as a function that maps the multiway latent factors to tensor entries. One extension is to add Gaussian process (GP) priors on the function to obtain a nonparametric model, which resembles a GP latent variable model. In particular, [53] proposed to stack the latent factors as  $\pmb{m}_{\mathrm{i}} = [z_{i_1}^1,\dots,z_{i_D}^D ]\in \mathbb{R}^{DR}$  and then assign a GP prior on the functional mapping. In specific, for continuous data, it assumes  $x_{\mathrm{i}}\sim \mathcal{N}(\mu_{\mathrm{i}},\sigma^{2})$ , where the mean function is a GP  $\mu_{\mathrm{i}} = f(\pmb {m}_{\mathrm{i}})\sim \mathcal{GP}(0,k(\pmb {m}_{\mathrm{i}},\cdot))$  associated with kernel  $k(\cdot ,\cdot)$ . Since GP has large computational complexity and designing the kernel requires ad-hoc expert domain knowledge, [25] proposed to parameterize the function using neural networks (NNs),  $x_{\mathrm{i}}\sim \mathcal{N}(\mu_{\mathrm{i}},\sigma^{2})$  where  $\mu_{\mathrm{i}} = f_{\mathrm{NN}}(\pmb {m}_{\mathrm{i}})$ . However, NNs easily overfit due to the high-dimensional and sparse nature of tensor data. To address this issue, [9] proposed to use Bayesian NN with spike-and-slab prior for sparse weights. All these models are based on directed graphical model, that assumes there exists a direct mapping from the latent factors to tensor entries and use simplistic distributions.
2.2 Energy-based model
Energy-based model [EBM, 21] is a class of undirected probabilistic models, which uses an energy function to characterize the data distribution. Given observed data  $x$ , the basic idea of EBM is to approximate the data distribution by a Boltzmann distribution,  $p_{\mathrm{data}}(x) \approx \frac{\exp(-f(x;\theta))}{Z(\theta)}$ , where  $Z(\theta) = \int \exp (-f(x;\theta)) \, \mathrm{d}x$  is the normalization constant (a.k.a., the partition function) and  $f(x;\theta)$  is the energy function. One classical example of EBM is the restricted Boltzmann machine (RBM) where the energy function has bi-linear form for tractability. In deep EBMs, the energy function is typically parameterized by deep neural networks. The main difficulty for training EBMs is to deal with the intractable normalization constant [36]. There are several ways to train EBMs, including contrastive divergence [14], score matching [16], noise-contrastive estimation [NCE, 11] and so on. In this paper, we focus on NCE, due to its efficiency and ability of tackling different data types.
We denote the unnormalized pdf as  $\phi (x;\theta) = \exp (-f(x;\theta))$ . NCE considers the normalization constant  $Z(\theta)$  as a trainable parameter. However, maximum likelihood estimation (MLE) does not work for this case since  $Z(\theta)$  can be arbitrarily small and the log-likelihood goes to infinity. Instead, the NCE can be obtained by maximizing the following objective,
$$
\mathcal {L} _ {\mathrm {N C E}} (\theta) = \mathbb {E} _ {x} \log h (x; \theta) + \nu \mathbb {E} _ {y} \log (1 - h (y; \theta)),
$$
where  $x$  denotes observed data and  $y$  denotes noises generated from some known distribution  $p_n$ ,  $\nu$  is the ratio between noise and sample sizes, i.e.,  $\nu = \#y / \#x$ . And  $h(\cdot)$  can be regarded as a classifier that distinguish data from noises, defined as follows,  $h(u; \theta) = \frac{\phi(u; \theta)}{\phi(u; \theta) + \nu p_n(u)}$ . It has been shown that NCE is consistent with MLE [11].
Although the noise distribution is essential for training efficiency, the selection of noises is currently limited to heuristics. A common intuitive is that the noise should be similar with the data. Indeed, [3] proposed to use conditional noises  $y \sim p_c(y \mid x)$  and minimizing the following loss function,
$$
\mathcal {L} _ {\mathrm {C N C E}} (\theta) = 2 \mathbb {E} _ {x y} \log [ 1 + \exp (- G (x, y)) ], \tag {1}
$$
where  $G(u_{1},u_{2};\theta) = \log \frac{\phi(u_{1};\theta)p_{c}(u_{2}|u_{1})}{\phi(u_{2};\theta)p_{c}(u_{1}|u_{2})}$  with  $y$  drawn from  $p_c(y\mid x)$ .
3 Proposed model
3.1 Energy-based tensor decomposition
Even though many non-linear tensor decompositions (TD) have been proposed to enhance flexibility, existing methods typically adopt simplistic distributions such as Gaussian. This can be problematic for complex real-world data. To address the issue, we propose energy-based tensor decomposition (EnergyTD), by integrating EBMs in TD framework. Given an order-  $D$  tensor  $\mathcal{X}$  of shape  $I_{1} \times \dots \times I_{D}$ , we aim to factorize it into  $D$  smaller latent factors  $\pmb{Z}^{d} \in \mathbb{R}^{I_{d} \times R}, \forall d = 1, \ldots, D$ . We denote the latent factor associated with the  $(i_{1}, \ldots, i_{D})$ -th entry as  $\pmb{m}_{\mathrm{i}} = [z_{i_{1}}^{1}, \ldots, z_{i_{D}}^{D}] \in \mathbb{R}^{DR}$ , where  $\pmb{z}_{i_d}^d \in \mathbb{R}^R$  is the  $i_d$ -th row of  $\pmb{Z}^{d}$ . Unlike traditional directed TDs trying to parameterize the conditional expectation  $\mathbb{E}[x_{\mathrm{i}} \mid m_{\mathrm{i}}] = f(m_{\mathrm{i}})$ , we model the joint distribution using an EBM,
$$
p \left(x _ {\mathbf {i}}, \boldsymbol {m} _ {\mathbf {i}}; \theta\right) = \frac {\exp \left(- f \left(x _ {\mathbf {i}} , \boldsymbol {m} _ {\mathbf {i}} ; \theta\right)\right)}{Z (\theta)}, \tag {2}
$$
where  $f(\cdot, \cdot; \theta)$  is the energy function and  $Z(\theta) = \int \exp(-f(x_{\mathbf{i}}, m_{\mathbf{i}}; \theta)) \, \mathrm{d}x_{\mathbf{i}} \, \mathrm{d}m_{\mathbf{i}}$  is the partition function to make it a valid pdf. We further assume the joint probability of all entries are independent, i.e.,  $p(\boldsymbol{\mathcal{X}}, \boldsymbol{m}) = \prod_{\mathbf{i} \in \Omega} p(x_{\mathbf{i}}, z_{i_1}^1, \ldots, z_{i_D}^D)$ , where  $\Omega$  denotes the set of observed entries. This is a standard setting in TDs and the dependence of tensor entries can be captured by sharing latent factors.
The expressive nature of the energy function enables us to easily handle diverse data types. For example, we can deal with discrete data by plugging one-hot codings into Eq. (2) to represent categorical probabilities. Additionally, the flexibility of NNs allows us to model tensors with side information, where each tensor entry incorporates additional features [34]. Specifically, in this paper, we focus on a particular case of dynamic tensors with continuous time stamps [48]. In this case, we consider an observed tensor as a time series  $\mathcal{X}_t$ , where the time stamp  $t$  is continuous, and each entry  $\mathbf{i}$  has its own specific time stamp  $t_{\mathbf{i}}$ . To model the tensor time series, we assume that each entry follows the same distribution and construct the time-dependent energy function,  $p(x_{\mathrm{i}},m_{\mathrm{i}};\theta ,t_{\mathrm{i}})\propto \exp (-f(x_{\mathrm{i}},m_{\mathrm{i}},t_{\mathrm{i}};\theta))$ , where the time stamp  $t_\mathrm{i}$  is considered as an auxiliary feature. The flexibility of NNs allows this function to learn general patterns across continuous time stamps. Experimental results demonstrate that this simple treatment can achieve good performances.
Network architecture The network architecture plays a crucial role in learning accurate probabilistic manifolds. Specifically, we define the energy function as  $f(x_{\mathrm{i}},m_{\mathrm{i}}) = g_1(g_2(g_3(x_{\mathrm{i}}),g_4(m_{\mathrm{i}})))$ , where  $g_{3}$  and  $g_{4}$  are MLP layers that encode information from  $x_{\mathrm{i}}$  and  $m_{\mathrm{i}}$ , respectively.  $g_{2}$  is a summation or concatenation layer that induce coupling between tensor values and latent factors, and  $g_{1}$  is the output layer. Although we currently utilize only MLPs, it is worth noting that convolutional architectures, as demonstrated by [39, 25], can also be employed, which is a topic for future research. To handle dynamic tensors, we incorporate an extra sinusoidal positional encoding layer [37] denoted as  $g_{5}(t)$  to capture temporal information. This embedding utilizes random Fourier features as proposed by [31]. Consequently, the energy function can be expressed as  $f(x_{\mathrm{i}},m_{\mathrm{i}},t_{\mathrm{i}}) = g_{1}(g_{2}(g_{3}(x_{\mathrm{i}}),g_{4}(m_{\mathrm{i}}),g_{5}(t_{\mathrm{i}})))$ . This architecture is commonly employed to capture temporal information and has been demonstrated to effectively learn high-frequency information when combined with MLPs [37].
Posterior sampling A significant application of TDs is to estimate the posterior of missing entries. Unlike traditional TDs, direct predictions cannot be obtained even after learning the latent factors due to the utilization of an undirected probabilistic model. Instead, we need to seek for sampling methods of  $p(x_{\mathrm{i}} \mid m_{\mathrm{i}})$ . One choice is score-based samplers by utilizing the score function  $\nabla_{x_{\mathrm{i}}} \log p(x_{\mathrm{i}} \mid m_{\mathrm{i}}) = \nabla_{x_{\mathrm{i}}} \log \frac{p(x_{\mathrm{i}}, m_{\mathrm{i}})}{p(m_{\mathrm{i}})} = -\nabla_{x_{\mathrm{i}}} f(x_{\mathrm{i}}, m_{\mathrm{i}})$ , such as Langevin dynamics [44]. Score-based samplers are not suitable for handling discrete data. However, in our case, we model the one-dimensional pdf for each entry, enabling us to directly sample the discrete data. Consequently, for continuous data, the use of grid search is a viable approach to obtain maximum a posteriori (MAP) estimations.
3.2 Learning objective
Despite the flexibility of the proposed model in Eq. (2), obtaining maximum likelihood estimation (MLE) becomes doubly intractable, as both the partition function  $Z(\theta)$  and the marginal distribution  $p(x_{\mathbf{i}})$  are intractable. Therefore, the CNCE loss Eq. (1) cannot be directly applied. In this section, we extend the variational approach [33] to construct a upper bound that addresses the challenge posed by intractable marginal distributions.
Denote the unnormalized pdf as  $\phi (x_{\mathbf{i}},m_{\mathbf{i}};\theta) = \exp (-f(x_{\mathbf{i}},m_{\mathbf{i}};\theta))$  and the unnormalized marginal pdf as  $\phi (x_{\mathbf{i}};\theta) = \int \phi (x_{\mathbf{i}},m_{\mathbf{i}};\theta)\mathrm{d}m_{\mathbf{i}}$ . For clarity, we omit the index  $\mathbf{i}$  in the subsequent context of this subsection. We follow the idea of CNCE [3] to distinguish data  $x$  from conditional noises  $y\sim p_c(y\mid x)$ . Firstly, Eq. (1) can be rewritten as
$$
\mathcal {L} _ {\mathrm {C N C E}} (\theta) = 2 \mathbb {E} _ {x y} \log [ 1 + 1 / r (x, y; \theta) ], \tag {3}
$$
$$
r (x, y; \theta) = \frac {\phi (x ; \theta) p _ {c} (y \mid x)}{\phi (y ; \theta) p _ {c} (x \mid y)}. \tag {4}
$$
However, for our problem, the unnormalized marginal probability  $\phi(x; \theta)$  is unknown. An additional variational distribution  $q(m; \varphi)$  is used to approximate the true posterior  $p(m \mid \mathcal{X}; \theta)$ . Note that in TDs where the data size is static, there is no need for amortized inference, which is different
from previous ones like [2]. Equipped with the variational distribution, the unnormalized marginal distribution can be computed using importance sampling,
$$
\phi (x; \theta) = \int \frac {\phi (x , \boldsymbol {m} ; \theta) q (\boldsymbol {m} ; \varphi)}{q (\boldsymbol {m} ; \varphi)} d \boldsymbol {m} = \mathbb {E} _ {q (\boldsymbol {m}; \varphi)} \left[ \frac {\phi (x , \boldsymbol {m} ; \theta)}{q (\boldsymbol {m} ; \varphi)} \right]. \tag {5}
$$
Plugging Eq. (5) into Eq. (4), we have
$$
r (x, y; \theta) = \frac {\mathbb {E} _ {q (\boldsymbol {m} ; \varphi)} [ \phi (x , \boldsymbol {m} ; \theta) / q (\boldsymbol {m} ; \varphi) ] p _ {c} (y \mid x)}{\phi (y ; \theta) p _ {c} (x \mid y)}. \tag {6}
$$
Since Eq. (3) is a convex function w.r.t.  $r(x,y;\theta)$ , plugging Eq. (6) into Eq. (3) and applying the Jensen's inequality, we have the upper bound,
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C N C E}} (\theta) = 2 \mathbb {E} _ {x y} \log [ 1 + 1 / r (x, y; \theta) ] \\ \leq 2 \mathbb {E} _ {x y} \mathbb {E} _ {q (\boldsymbol {m}; \varphi)} \log \left[ 1 + \frac {\phi (y ; \theta) p _ {c} (x \mid y) q (\boldsymbol {m} ; \varphi)}{\phi (x , \boldsymbol {m} ; \theta) p _ {c} (y \mid x)} \right] \triangleq \mathcal {L} _ {\mathrm {V C N C E} (\theta , \varphi)}. \tag {7} \\ \end{array}
$$
Following [33], we have the theorem about the tightness of the bound.
Theorem 1 The difference between the VCNCE loss Eq. (7) and CNCE loss Eq. (1) is the expectation of the  $f$ -divergence,
$$
\mathcal {L} _ {\mathrm {V C N C E}} (\theta , \varphi) - \mathcal {L} _ {\mathrm {C N C E}} (\theta) = \mathbb {E} _ {x y} [ \mathbb {D} _ {f _ {x y}} (p (\boldsymbol {m} \mid x; \theta) \| q (\boldsymbol {m}; \varphi)) ],
$$
where  $f_{xy}(u) = \log (\frac{\kappa_{xy} + u^{-1}}{\kappa_{xy} + 1})$  with  $\kappa_{xy} = \frac{\phi(x;\theta)p_c(y|x)}{\phi(y;\theta)p_c(x|y)}$ .
The proof can be found in appendix. Based on the theorem, we have the following corollaries to justify the optimization process.
Corollary 1 When  $q(m; \varphi)$  equals to the true posterior, the CVNCE bound is tight, i.e.,
$$
\mathcal {L} _ {\mathrm {V C N C E}} = \mathcal {L} _ {\mathrm {C N C E}} \Longleftrightarrow q (\boldsymbol {m}; \varphi) = p (\boldsymbol {m} \mid x; \theta).
$$
Corollary 2 The following two optimization problems are equivalent,
$$
\min  _ {\theta} \mathcal {L} _ {\mathrm {C N C E}} (\theta) = \min  _ {\theta} \min  _ {q (\boldsymbol {m}; \varphi)} \mathcal {L} _ {\mathrm {V C N C E}} (\theta , \varphi).
$$
In practice, we need to seek for the sampled version of Eq. (7). Supposing we have  $N$  observed samples  $\{x_{\mathrm{i}}\}_{\mathrm{i} = 1}^{N}$ ,  $\nu$  noises  $\{y_{\mathrm{i},j}\}_{j = 1}^{\nu}$  for each sample  $x_{\mathrm{i}}$  and using importance sampling for  $\phi (y;\theta)$ , the sampled objective function is,
$$
\mathcal {L} _ {\mathrm {V C N C E}} (\theta , \varphi) = \frac {2}{\nu N} \sum_ {\mathrm {i} = 1} ^ {N} \sum_ {j = 1} ^ {\nu} \mathbb {E} _ {q \left(\boldsymbol {m} _ {\mathrm {i}}; \varphi\right)} \log \left[ 1 + \frac {\mathbb {E} _ {q \left(\boldsymbol {m} _ {\mathrm {i}} ; \varphi\right)} \left[ \frac {\phi \left(y _ {\mathrm {i} , j} , \boldsymbol {m} _ {\mathrm {i}} ; \theta\right)}{q \left(\boldsymbol {m} _ {\mathrm {i}} ; \varphi\right)} \right] p _ {c} \left(x _ {\mathrm {i}} \mid y _ {\mathrm {i} , j}\right) q \left(\boldsymbol {m} _ {\mathrm {i}} ; \varphi\right)}{\phi \left(x _ {\mathrm {i}} , \boldsymbol {m} _ {\mathrm {i}} ; \theta\right) p _ {c} \left(y _ {\mathrm {i} , j} \mid x _ {\mathrm {i}}\right)} \right].
$$
Specifically, we formulate  $q(m; \varphi)$  as a diagonal Gaussian and use reparameterization trick [18] to compute the expectation. When dealing with continuous data, we typically select conditional Gaussian noises, represented as  $p_c(y \mid x) = \mathcal{N}(y \mid x, \sigma^2)$ . This choice entails only one hyperparameter  $\sigma$  that needs to be tuned. Another benefit is the symmetry of the conditional distribution for Gaussian noise, expressed as  $p_c(y \mid x) = p_c(x \mid y)$ . Hence, the objective function can be further reduced. For binary or categorical data, such symmetric noises can also be derived [3].