Bayesian Regularization of Latent Representation
Authors: Chukwudi Paul Obite
Zhi Chang
Keyan Wu
Shiwei Lan
Venue: ICLR 2025 Poster (2025)
Keywords: Dimensionality Reduction
Latent Variable Models
Representation Complexity Regularization
Variational Inference
Generative Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel class of latent variable models based on the recently introduced Q-exponential process to effectively learn latent data structures with greater flexibility in managing representation complexity while enhancing interpretability.

Abstract: The effectiveness of statistical and machine learning methods depends on how well data features are characterized. Developing informative and interpretable latent representations with controlled complexity is essential for visualizing data structure and for facilitating efficient model building through dimensionality reduction. Latent variable models, such as Gaussian Process Latent Variable Models (GP-LVM), have become popular for learning complex, nonlinear representations as alternatives to Principal Component Analysis (PCA). In this paper, we propose a novel class of latent variable models based on the recently introduced Q-exponential process (QEP), which generalizes GP-LVM with a tunable complexity parameter, $q>0$. Our approach, the \emph{Q-exponential Process Latent Variable Model (QEP-LVM)}, subsumes GP-LVM as a special case when $q=2$, offering greater flexibility in managing representation complexity while enhancing interpretability. To ensure scalability, we incorporate sparse variational inference within a Bayesian training framework. We establish connections between QEP-LVM and probabilistic PCA, demonstrating its superior performance through experiments on datasets such as the Swiss roll, oil flow, and handwritten digits.

Published as a conference paper at ICLR 2025
BAYESIAN REGULARIZATION OF LATENT REPRESEN- TATION
Chukwudi Paul Obite
Zhi Chang
Keyan Wu Shiwei Lart
School of Mathematical & Statistical Sciences, Arizona State University 901 S Palm Walk, Tempe, AZ. 85287, USA
ABSTRACT
The effectiveness of statistical and machine learning methods depends on how well data features characterized. Developing informative and interpretable la- tent representations with controlled complexity essential for visualizing data structure and for facilitating efficient model building through dimensionality re- duction. Latent variable models, such as Gaussian Process Latent Variable Mod- els (GP-LVM), have become popular for learning complex, nonlinear represen- tations as alternatives to Principal Component Analysis (PCA). In this paper, we propose a novel class of latent variable models based on the recently introduced Q-exponential process (QEP), which generalizes GP-LVM with tunable com- plexity parameter, q > O. Our approach, the @-exponential Process Latent Vari- able Model (QEP-LVM), subsumes GP-LVM special case when q 2, of- fering greater flexibility in managing representation complexity while enhancing interpretability. To ensure scalability, we incorporate sparse variational inference within a Bayesian training framework. We establish connections between QEP- LVM and probabilistic PCA, demonstrating its superior performance through ex- periments on datasets such as the Swiss roll, oil flow, and handwritten digits.
Dimensionality Reduction, Latent Variable Models, Representation Complexity Regu- larization, Variational Inference, Generative Models.
Keywords:
INTRODUCTION
In various learning tasks involving high-dimensional data, it is crucial to effectively learn a repre- sentation of the data that lies in low-dimensional subspace. Such latent representation is often the key in unsupervised learning to understand the complex data structure which is infeasible to visu- alize in the original space. What more, reducing data low-dimensional latent space is also the core to build efficient models in supervised learning. Principal component analysis (PCA) (Pearson) 1901 Jolliffe 1986 Dolliffe Cadimal (0166) has long been used as classic technique factor analysis. It is later cast into probabilistic latent variable model (LVM) with linear (Tipping &
Bishop 1999 Minka 2000) or nonlinear (Schölkopf et al. 1998 mapping from the latent space
to the data feature space.
propose to employ Gaussian process (GP) (Rasmussen & Williams, 2005) as a prior for the mapping in LVM. Such GP-LVM views data as functional outputs of latent variables and can be regarded as a linear or nonlinear probabilistic PCA through linear more general kernel respectively. Titsias & Lawrence (2010) then develop Bayesian version of GP-LVM and an efficient variational Bayes inference with sparse approximation through inducing points (Titsias 2009). Since then has been extensively applied in neuroscience (Gundersen et al. 2021). bioinformatics (Ahmed et al. 2018), and robotics (Delgado-Guende
In the probabilistic PCA literature, Lawrence (2003 2005
et al.) 2020), etc. More recent development extends GP-LVM M discriminative classification (Urtasun & Da 2007), Gaussian mixture models (Nickisch & Rasmussen 2010)) deep probabilistic models (Damianou Lawrence 2013), inverse problems (Atkinson & Zabaras 2019), and longitudinal
modeling (Le & Honavar
2020
"slan @asu.edu

---

Published as a conference paper at ICLR 2025
All the aforementioned works are based on GP which tends to be over-smooth for inhomogeneous data with abrupt changes or sharp contrast. To address this issue, a new stochastic process named Q-Exponential process (Q-EP) (Li et al. 2023) has been proposed based Lg penalty regularize the modeling effect through a parameter
which also embraces GP as a special case for q = 2. In this paper, aim to port such flexibility in regularization L.M. More specifically, we replaces GP in GP-LVM (Lawrence) 20003 with Q-EP to propose a novel class of LVMs named @-exponential Process Latent Variable Models (QEP-LVM) parameterized by the regularization parameter q 0. We establish its connection to the probabilistic PCA and provide explicit formula for the maximum likelihood estimator of the latent variable singular values. We also develop the Bayesian QEP- LVM and adopt variational inference as Titsias & Lawrencel (2010) with sparse approximation via inducing points (Titsias) 2009). To derive the evidence lower bound (ELBO) for the variational form of Bayesian QEP-IVM, we adopt the approach by Hennsman et al. (2015) to directly compute the variational distribution of latent function instead of using variational calculus (Titsias & Lawrence, 2010). Though not straightforward in the Q-EP setting, tractable form of ELBO can be obtained with the help of Jensen's 's inequality.
We demonstrate the regularization effect through the parameter 9 > O the latent representations learned with QEP-LVM some of which exhibit controlled compactness or enhanced interpretability compared with GP-LVM (g 2) (See Section 4). It mains an open question determine the optimal g² automatically based on the given data. We therefore adopt a Bayesian approach by imposing appropriate priors on g to obtain an optimal choice. With chosen regularization parameters, we also investigate the generative classification models built on fitted LVMs and show that GP-LVM often sub-optimal, especially for inhomogeneous data. All the numerical examples have been efficiently implemented in GPyTorch (Gardner et al. 2018).
Connection to existing works on non-Gaussian LVMs Our proposed QEP-LVM directly gener- alizes GP-LVM (Tissia& & Lawrence 20100) with flexible regularization on the learned latent repre- sentations. There are also some works regarding non-Gaussian LVMs falling in the same category of QEP-LVM. Palmer al. (2005) theoretically characterize variational representation of LVMs with non non-Gaussian priors constructed as a supremum or an integral over the scale parameter Gaussian densities. Kleppe & Skaug (2008) build and fit non- non-Gaussian LVMs via the moment-genentt generating function. Salehkaleybar et al. (2020); Xie et al. (2022); Chen et al. (2022) consider linear causal LVMs with non-Gaussian noises. Gundersen et al. (2021): Zhang al. (2023): Li et al. (2024) propose non-Gaussian LVMs with random Fourier features in the Karhunen-
Larhunen-Loéve Loéve expansion of GP tailored to non-G non-Gassiian data. The non- aon-Gausianity of these LVMs is constructed by certain com- position of GPs. On the contrary, the QEP-LVM is, if not the first, developed directly based non non-Gaussian stochastic process. Our proposed work has multi-fold contributions to the literature of LVM:
1. We propose novel LVM based on Q-EP which learns latent data representation with flex- ible complexity.
2. We establish the connection between QEP-LVM and nonlinear probabilistic PCA.
3. We demonstrate the regularization effect of Bayesian QEP-LVM which enables enhanced latent representation learning compared with GP-LVM.
The rest of the paper is organized as follows. Section reviews Q-EP as flexible prior for Bayesian multi-output regression. We then introduce QEP-LVM VM as a nonlinear probabilistic PCA and the variational inference for Bayesian QEP-LVM in Section 3 effect of QEP-LVM via parameter > 0 and demonstrate that the optimal 9 leads to better latent representation compared with that obtained by GP-LVM (g 2). Finally, we conclude in Section 5 with discussion on some future directions.
the regularization
2 BACKGROUND: BAYESIAN MODELS WITH Q-EXPONENTIAL PRIORS
2.1 Q-EXPONENTIAL DISTRIBUTION AND PROCESS
Motivated by Lg regularization, Li et al. 2023) propose the multivariate q-exponential distribution
that is exchangable and consistent with respect to marginalization.
2

---

Published as a conference paper at ICLR 2025
Definition 1. A multivariate g-exponential distribution for random vector R., denoted as q-ED (p,, C) has the following density
r² 2 2
p(ulp, C, =2xx)
2,(1-11 exp
r(u) (u- (u p.).
Remark 1. The above density (log-convex, heavy-tailed around mean for 0 < < 2 and log- concave, light-tailed for q > 2), when taken negative logarithum, is dominated by some weighted Lg norm of μ., i.e. ]r = 2 μi pl From the optimization perspective, q-EDN. when used as prior, imposes Lg regularization in obtaining the maximum posteriori (MAP).
Suppose function u(x) observed at locations, 21,-.. IN E D C R. Li et al. (002))
define the following g-exponential process (Q-EP) based on scaled q-exponential distribution q-ED (0, C) assumed for the random vector u (u(x1), ,u(xN)).
Definition (centered) g-exponential process u(x) with kernel C, q-EP(O, C). is a collection (u(x1),..-uuf u(xx)), follows scaled mul- tivariate g-exponential distribution q-ED" (0, C), i.e., N- qOEDx(O, C), where C [C(x;, NN)]NNN IFC= II then is said to be marginally identical but uncorrelated (miuu..
of random variables such that any finite set,
Remark 2. If q = 2, q-ED (H, C) reduces NN C) and q-EP(O, C) becomes GP(O,C)
When [1,2), 2), q-EP(O, C) lends flexibility to modeling functional data with more regularization than GP (See Figure for the regularization effect of q).
For multiple Q-EPs, (u1(z), ,uD(x)). we define multi-output (multivariate) Q-EP through ma- which forms a vector by concatenating its
trix vectorization, vec(UNxD) = [u]..... ub. columns. Note, the component processes are often assumed uncorrelated, rather than independent.
Definition 3. A multi-outpul (multivariate) g-exponential process, u(-) (ui()),... ,up()).
each u.(+) ~ C,) is said to have association C, if at any finite locations, x = (In)nz vec((u[(x),".. "'x((x())))xx)) ~ g-EDD C₁ 0 Cz), where we have (x(x) = (=("))" for 1...... a [u "MD(xx)= and C2 Fm NxN.. We denote u q-EP(,, C.). In particular, the component processes are m.uu if ID.
2.2 BAYESIAN REGRESSION WITH Q-EP PRIORS
Given data x [ and y {ym N 1= for the generic Bayesian regression model:
fxx) t e e~ q- E), ~~ C-E((,CC)
(I)
we have tractable posterior (predictive) distribution similar GP regression as in Theorem 3.5 of Li et al. (203)).
Denote X .x Nxq. F
[X((); ... fn [X) NXD and [y yolNX D. With m.i.u. Q-EP priors as in Definition imposed on f := (Si, SD), now we consider the following multivariate regression problem:
NXD
3
likelihood
vec (cc(Y)|Fo EDvo & E),
(2)
prior on latent function f CP
Noticing that = e with vec(e) ~ G-ED(O, Ip & )), we can find the marginal of Y based on the property of q-ED (Fang & Zhang 1990) follows:
marginal likelihood
vec ree(YY)X (O. & (C+E))
(3)
In the following, we view Y as the only data and X is merely regarded as the latent variable. Usually it is assumed Q < D in the latent representation learning. Different from standard regression problems, a latent variable model (LVM) seeks to solve X in (2) or equivalently (3) for given data Y either deterministically or probabilistically (assuming proper prior on X).
3

---

Published as a conference paper at ICLR 2025
3 Q-EP LATENT VARIABLE MODEL
GP-LVM is introduced by Lawrence 2003 2005) as an unsupervised learning method for dimen- sionality reduction. It can be interpreted as nonlinear extension of probabilistic PCA (Tipping & Bishop 1999, Minka 2000). The GP can be replaced by Q-EP to impose more regularization on the latent space, and hence we propose Q-EP based LVM (QEP-LVM).
3.1 QEP-LVM AS A NON-LINEAN PROBABILISTIC PCA
For convenience, we set E = = in the following. Let us start with linear latent function XW in the model 2 with W E BR&D The original probabilistic PCA (Tipping & Bishop 1999
is formulated by integrating out the latent variable X: p(Y W 3) = [YYY,, W, 3)p(X)d d with prior p( X) = (X; 0, IN)) The resulting marginal likelihood, p(Y(W, 3) = O, (WTW W+
I-II)) Iv). maximized for the solution W*, which spans the principal space of the data Y.
Lawrence 2003) forms the dual problem of probabilistic PCA by marginalizing the weight pa- rameter W as p X 3) = J p X W, p)p( WJdW with prior p( (W) = O, IQI and optimizing the resulting X .) = NYY; 0, D [XXT+ + ))) for the optimum, X which related to the sigendeccompositionn of empirical covariance matrix YY Lawrence 2003 2005) replace the above linear kernel with general ones, c.g. K = - K as in 3), to learn nonlinear latent reduction X, essentially defining an LVM with GP mapping (GP-LVM).
Now we impose a m.i.u. Q-EP prior on vec W) q-ED(O, a "I@p) that induces the following prior on the latent function F:
vec(F) (In @ Xjvee(W) (W) ~ q-ED(O, a 'I, xxx).
The following marginal likelihood of Y X can be obtained similar to which defines stochastic mapping from the latent space of X to the data space of Y:
vec | B ~ q-ED(O, ID @ K), k= - + IN-
(4)
Similarly as GP-LVM, 3 or 4 with general kernel defines an LVM with Q-EP mapping (QEP- LeM). r = vec
(In ) K)- -vecv Y) tr(k-1yYT) The log-likelihood (4) is
D
ND
g log (Y)
L = log
K +
2
2
2
The following theorem states that the maximum likelihood estimator (MLE) for X is equivalent to the solution for the dual probabilistic PCA (Tipping & Bishop 1999, Minka 2000).
Theorem 3.1. Suppose YY has cigen-decomposition UAUT with A being the diagonal ma- NN trix with eigenvalues {A:) Then the MLE for (5) is
X = UQLV, L --didaa(///a
- (1))) c(g) = -D'- 2(D (q N)N
where UQ is an N x Q matrix with the first Q eigen-vectors in U, V is an arbitrary Q x Q orthogonal matrix, and a b= min(a, b).
Proof. See Appendix A
Remark 3. When q = 2, the singular values of the latent variable X reduce to 4 aa(A 3-1 corresponding to those for GP-LVM, though they were mistakenly stated as 1 (a-1(); (A in Lawrence (2003). When >> O varies, it regularizes the singular values (hence the latent space spuced by X) through c(q), whose properties are illustrated in Figure A.1
To detect complicated nonlinear latent representation, we consider the following automatic relevance determination (ARD) squared exponential (SE) kernel k(-, .) (Titsias Lawrence 2010):
1 (x,, xm) diag(y) x
[>,,, Xm )1 N 1((X,, xm
exp
(6)
2
Then the locale of X can be determined by optimizing 5 with respect to X. The full solution to QEP-LVM also involves optimizing the kernel parameters (a, 7). Other kernels may lead to slightly different results, but the numerical conclusion (Section 4 is robust to the choice of kernels.
4

---

Published as a conference paper at ICLR 2025
3.2 BAYESIAN QEP-LVM
In this section we introduce Bayesian QEP-LVM and develop variational inference as Bayesian GP- LVM (Titsias & Lawrence 2010). Compared with the optimization method (Lawrence) 2003), the Bayesian training procedure is robust to over-fitting and can automatically determine the dimension- ality of the nonlinear latent space (Titsias & Lawrence 2010) e.g. by thresholding the correlation length Y. The derivation of variational Bayes for QEP-LVM is much more involved because the log-likeliing ligliike (5) no longer presented as quadratic form of data. Yet an appropriate evidence lower bound (ELBO) can still be obtained with Jensen's 's inequality.
In addition the likelihood model (4) we imposes prior on the latent variable X and consider the following Bayesian QEP-LVM:
marginal likelihood:
vec () X ~ q-ED(O, @ K),
prior on latent variable: vec( q-ED(O, IvQ).
We use variational Bayes to approximate the posterior distribution p( p(Y(X))(XX with the variational distribution using uncorrelated q-ED:
variational distribution for latent variable ((X) q-ED( ((..)()))))),
where each covariance S, is of size Q x Q and can be chosen as diagonal matrix for convenience.
Due to the equality of log-evidence, log Y (L X (Ip(X) + L(q(X)), minimizing the
KL divergence is equivalent to maximizing the ELBO (4q (X)) follows:
logp(Y Y) 2 L(o(X)) = ((X) log ( dX L(((X)) (T(((())lp(X))
(X)
( Y dX =: (logp(Y(X)) is intractable and where the first term (X)( = ((X) lo hence difficult to bound.
3.2.1 LOWER BOUND FOR THE MARGINAL LIKELIHOOD
To address such intractability issue and to speed up the computation, sparse variational approxima- tion Titssas 2009) is adopted by introducing a set of inducing points X E Mx with their function
values U (X). fn [X)] ] E MMD Hence the marginal likelihood pXY defined in (5)
can be augmented to the following joint distribution each being q-ED:
X( p(Y(F))(F]U, (UU())
where we have vec (O)( EDD ID KMM) and the conditional distribution
~ (eeevveeKKy)) M K U), In @ (KNN - KN K KaN)).
(7)
The inducing points X regarded variational parameters and hence they are dropped from the following probability expressions. We then approximate p(F, UJX) xx p(F]U, X)p(U) with q( U) [F[ (U) in another variational Bayes follows:
/ U) log p FFF U. X)p(U)
logp(Y(X)
dFJU
q(FUU)
(8)
/ (U))U)((U)dUloopp)
((U)) ((U)
U(U) du.
Different from Titsias
2009); Titsias & Lawrence 2010) using the variational calculus, (SVGP
Hensman et al. 2015 compute the marginal likelihood ELBO through the variational distribution of latent function F. Instead of the variational free form, follow Hensman et al. (2015) the
variational distribution for of the following format conjugate to p( (U]
variational distribution for inducing values : ((U) ~ C-D(
(EE)))
where M the variational mean. Noticing that U follows a conditional g-exponential 7 we can obtain the variational distribution of F, (F), by marginalizing U out similarly as in
variational distribution for latent function : qF)) =
= P(FIU))(U)dJ) [UJJUU/DCFFUUUUUU ~ EDED
(Ke Ip & (KNN KNMK K + MM KNNH),

---

Published as a conference paper at ICLR 2025
Therefore, the variational lower bound of the marginal likelihood becomes
logp( (Y)())())))))))))))))))>()) 2 (log ()))) KL(q(U)) (U)).
Denote by logp(YY(F)
( F)), where po(r) := DN log 3 ND 1) log r 2 is convex for q E (0. and r(Y, F) vec(Y - (FF)F) F IND) FY F) is quadratic form of random variable Y. Therefore, by Jensen's inequality, we can bound from below
F) = Btr((Y
(log p( Y g)) (po(r((, F))) 2 (o((((((,,)))))))))))))))) F))o()))
where we calculate the expectation of the quadratic form FFY,F) as
())(F) =r(Y, KNNK Mar Mar 3Dr 1-D (NNN KNAs MM MMN
+ 3
Σ d=1
MAs Ea
MA Mar KMN).
Denote by XX = ((log Y(F)) ((1)))(())) Then by Jensen's 's inequality again we have
1(Y,X) 2 ()(((r(X,)))ut))aaap (4()))))) =: h² XY, Define 4/n tr((KN))(()). V (KNM)/(X)) and V22 (K 1KMNKNN (NN))N))( Further
calculate the expectations of quadratic terms similarly
((r(Y, ()) ()) - -D tr(K
M2)l
Σ
ir(K
MM EoK
MM 42)
MM
+ +r(Y, W, MMM M) + Str(M K (M2 - V. K M).
We compute the lower bounds for two K-L divergence terms KLu ( (P))() and KLx := (L(g(X)|p(()) by similar argument with Jensen's 's inequality and expectation of quadratic forms. Details are left Appendix B.2 for interested readers.
3.2.2 SUMMARY OF ELBO
Denote by p(r; E, D)
D) == D log
Σ
+
ND 2
(2 - 1) logr r We summarize the final ELBO
L (q(x)) as follows:
log 2((((X)) / ((X))(UU)p(FU)))) X) log
P F (PUU)() daUUx (X)
2x)((())) = h"(Y, - KLU - KLX, NYY,X)) ; IIN D),
ry -Y V, Ky M) + (tr(M K
MaS (T22 K
+ 3D tK 22) + Σ
tr(K Tos V2)
(9)
d=1
D
KTU
1
log Σg +
tr(M MM M) + Σ
ir( EirEEaksta)) KMM,
2
d=1
d=1
1
-KL
log S 40 tr(H 1) + Σ tr (tr(S.):IN.)))
2
where th = tr =t(((KNN)(xi)) v (KK=M (KNM))(X); and v = MK NNa) 9(x) x()) (Appendix B.I.
Remark 4. The variational soluttao qXX) can be obtained by maximizing the ELBO ( with respect
the variational parameters (H,, { X, M, (Ea)) and kernel parameters (ax, B, B,7).
Remark 5. When q = 2, s(r; 2, D) = - log 2'
f/r with r r(Y, v K -1
MM M) becomes
the log-density of matrix normal MM K
Ma M, IN, Then the ELBO g
duces to Equation (7) of (SVGP Hensman al. 2015) with extra term (tr(MTT.) MM (V2-
V V, W 1K M), not computed when falling back GP-LVM. The computational complex-
ity, M22M,, remains the same SVGP and GP-LVM (Tissias & Lawrence) 2010).

---

Published as a conference paper at ICLR 2025
3.3 WHAT q?
While the parameter > 0 regularizes the learned latent representations (See Section 4. it remains question automatically search for the optimal q", rather than manually pick one. Therefore, we impose a Gamma prior, q ~ [(ao: Po), and jointly optimize its posterior for the optimal q".
Note, the target function, ELBO 9 involves q only through the function sp, which is concave in Le = - (3 log r)² < 0). Therefore, if we block-update parameters in the overall optimization procedure, maximizing ELBO with respect to should return unique solution, though the joint densities could be rather complicated (See Figure C.T. In the numerical experiments (Section we adopt Onp 4 and So 2 to make the prior mass concentrated in (). Figure B. investigates op as function of g that plays key role in finding the optimal q².
3.4 PREDICTION
With Bayesian QEP-LVM, we can make predictions for test data y.. As in Bayesian GP-LVM Titsias & Lawrence 2010), we can compute the predictive density p(y, (Y) for some test data y..
Given the latent variables X obtained for the training data Y and new test latent variable x,, the predictive density can be written and approximated as follows
ply Y) S p(y,, xXX,x.)p( x.), )XXx.. exp (e((( ((g())))
(10)
I J
where C is the ELBO as in (9) Specifically, train Bayesian QEP-LVM based on Y to get (()())). In the testing stage, we append y. to Y on which we train another Bayesian QEP- LVM with augmented variational distribution qX, x.) q-ED([u, pm], diag (.,, to ob- tain (()(X.
The predictive density (T)) can be utilized to build generative models to classify labels. Suppose in addition to data Y, we have labels t [ falling in K categories. Let ( [yvl4 = k).
Then we train K LVMs each based on one of ( 1m and consider the following classifier:
f.
arg max ply, Y (())p(t p(t, k).
This will be further investigated in Section 4.2 and Section 4.3
4 NUMERICAL EXPERIMENTS
In this section, we demonstrate the regularization effect of QEP-LVM on latent representation learn- ing through the parameter q > and compare it with GP-LVM (q = 2). We investigate the latent representations of multiple datasets learned by QEP-LVMs with changing y 0 and observe the regularization effect that smaller y tends to contract the latent space. Compared with GP-LVM as QEP-LVM for fixed = 2, optimizing the parameter q often leads superior latent repre- sentation with enhanced interpretability. Throughout this section, we use the kernel (6) and the Gamma prior T44, 2) if varying q 0 in the Bayesian framework. Since autoencoder is equiva- lent to PCA when all of its activation functions are linear, we also include the probabilistic version, variational autoencoder (VAE) (Kingma & Welling 2014), for comparison. All the examples implemented in GorTorch (Gardner et al. 2018)) and the computer codes are publicly available at com/ htmm/lanzithininn Regggg/Reegeeppeeeee
4.1 SwISS ROLL
First we consider the Swiss roll dataset (Marsland 2014) usually used in manifold learning. Illus- trated in upper left panel of Figure [ this dataset consists of a 3d point cloud resembling the food with the same name. Because the 1000 points mainly sit curved surface, we aim to learn 2d latent subspace with GP-LVM and QEP-LVM respectively. As shown in the lower left panel of Figure I. PCA returns result seemingly by compressing the 3d cloud from above (along axis). We then train VAE with 5 dense layers and the resulting 2d latent representation identified by the two largest variances of latent distribution resembles that of PCA. QEP-LVMs are trained for
7

---

Published as a conference paper at ICLR 2025
WIE
2.0 (aassiinn)
Laken
Letes
7CA
Strlances iiteee dea Button
Inverse lengthscale temee
Inserse Lengthsrale of tomel
Inwerse longthorale of teme!
300
XB
Figure I: Latent representation of Swiss roll dataset. Upper left: 3d cloud of 1000 points; lower left: PCA in 2d principal space; right 4 columns: 2d latent representations (upper row) by VAE and QEP-LVMs with q 1.0, 1.5 and 2.0 (GP-LVM) showing regularization effect via the parameter and the corresponding variances of latent distribution (VAE) and inverse length-scales ordered on the r-axes (lower row). Colors are used aid visualization but not for training.
1.39
Inversia sungttiscale kernel
Neg. ELBO LOSN
Postartur
275
625
6.8
600
1.
850
4003
6059
Figure 2:
Latent representation of Swiss roll dataset output by QEP-LVM with optimal q" 1.39 (red line in the rightmost panel) found in the Bayesian framework.
different gs with 25 inducing points. The upper row of right three panels in Figure [ compares the latent representations output by QEP-LVMs with = 1.0, 1.5 and 2.0 (GP-LVM) which all appear as projections from outward view (along axis). As q decreases from 2.0 to 1.0, the learned la- tent representations contract towards axes, verifying its regularization effect. The latent result with q 1.5 is the best representing the latent "roll" structure among these plots. Most of the inverse length-scales (7) of the kernel in the lower row imply 2 dominant dimensions (used to plot 2d latent representations), indicating intrinsic dimensionality for this dataset.
Next, we let q > 0 be a random variable imposed with a Gamma prior and jointly optimize q with other parameters in the QEP-LVM. Figure C.I plots the pairwise densities of q and length-scale I = 1/r,, which highlights the difficulty of joint optimization (different scales, complex landscape, etc). As shown in the rightmost panel of Figure 2 the optimal regularization parameter is attained at 9- = 1.39. The resulting latent representation, as in the leftmost panel of Figure 2 reflects a clear latent "roll" structure.
We dig hole in the 3d point cloud along the curved surface to make a "Swiss hole" dataset in Appendix C.I We also include latent representations by Isomap and t-SNE Figure C.2 for com- parison. Then we repeat the same experiments for such dataset and present similar results in Figure C,3 and Figure C.4 Again QEP-LVM with regularization parameter around 1.5 outputs latent rep- resentation having "roll" with "hole" structure better than GP-LVM and VAE.
4.2
OIL FLOW
Next, we demonstrate the behavior of QEP-LVM and contrast it with GP-LVM using the canon- ical multi-phase oil-flow dataset (Bishop & James 1993) that consists of 1000 observations (12-
8

---

Published as a conference paper at ICLR 2025
2 (iGreesbn))
9.3
3:35
1-10
Laton omn
Lolove 2012
Loton
Veriences of etent distribution
Inverse Lengtrocale of kemel
Inverse Langthscale kerne
Inverse Lengthocale of kemel
Inverse Langthscale of kerre
05
Figure 3: Latent representation of oil flow dataset. Upper: 2d latent representations by VAE and QEP-LVMs with q = 2.0 (Gaussian), 1.5, 1.15 and 1.0 showing a regularization effect via the parameter q. Lower: the corresponding variances of latent distribution and inverse length-scales 7y. Colors for points label three classes but are not used for training. Gray-scale colors indicate uncertainty approximated by the variational distribution.
dimensional) belonging three known classes corresponding to different phases of oil-flow. This dataset is also used in Lawrence (2003): Titsias & Lawrence (2010). We train the models only on the 12 covariates (no labels used) with the same settings as the above example in Section 4.1 The upper row of Figure 3 visualizes 2d slices of the latent spaces by VAE (identified by the largest two latent variances) and by QEP-LVMS corresponding to the most dominant latent dimensions (identified by the largest two inverse length-scales 7) with The gcay-scale color indicates the uncertainty (computed from S, in ((X)). Again we observe the regularization effect by the parameter > 0 of QEP-LVM in latent representation learning: smaller leads to more regularization on the learned representations and hence yields more aggregated clus- ters, as illustrated by the green class. In the lower of Figure 3 except VAE, all the QEP-LVM models unanimously identify 3 dominant latent dimensions despite their orders.
2.0 (Gaussian), 1.5, 1.15 and 1.0 respectively.
We also vary q > with the same Gamma prior and obtain the optimal q 1.15. The results are shown in Figure [.6). With each trained QEP-LVM, we utilize the predictive density to build a generative classifier described Scction 3.4 Then we compare these classifiers on common test dataset and repeat the experiments for 1O different random seeds. Table [ summarizes the performance in terms of accuracy (ACC), area under ROC curve (AUC), adjusted rand index (ARI) score, normalized mutual information (NMI) score, log predictive probability (LPP) values, and running time (per class). The models with y = 1.0 and 1.15 are generally better than the other two with y 1.5 and 2.0 (Gaussian) probably because the former two have labeled points more separated from each class in the latent subspace. The best classifier (ACC/AUC) coincides with the one for optimal q-' 1.15. Note that LPPs not comparable with each other since they from different models with different likelihoods. Though not used in training, the class labels can be identified as cluster assignments. Then QEP-LVM with optimal ARI and NMI scores which are metrics for evaluating clustering algorithms.
1.15 also attains the highest
Table 1: Classification on flow based on learned latent representation: accuracy (ACC), area under ROC curve (AUC), adjusted rand index (ART) score, normalized mutual information (NMI) score, log predictive probability (LPP) values, and running time (per class) by various Bayesian QEP-LVMs. Result in each cell are averaged over 10 experiments with different random seeds; values after = standard errors of these repeated experiments.
Model (2)
ACC
AUC
ARI
NMI
LPP
time/class
1.0
0.70 + 0.04
0.03
0.35
0.05
0.36 0.05
-10.42 0.42
211.92 12.53
1.15
0.73 + 0.04
0.89 0.02
0.38 + 0.05
0.39 0.05
-9.21 0.25
212.30
13.17
1.5
0.67 + 0.05
0.83 0.03
0.30 + 0.07
0.33 0.06
-7.70 -.700+0.22
0.25
206.91
13.58
2.00 2.0(Gausiian)
0.68 + 0.05
0.83 = 0.03
0.34 + 0.07
0.37 + 0.07
-8.78
6.78::00 0.19
198.47 + 14.42

---

Published as a conference paper at ICLR 2025
VAE
2.0 (Gaussian)
9=155
50
20
20
Latent dim
Lanenn dim
20
-10
-20
Latent 20
-30
-20
20
Figure 4: Latent representations of MNIST database by VAE (left), GP-LVM (middle), QEP-LVM with q 1.5 (right). For the convenience of visualization, 10-dimensional latent spaces learned by these algorithms are projected to 2-d subspace by t-SNE respectively.
4.3 MNIST
Lastly, we consider the MNIST database (Lecun et al. 1998) consisting of 60, 000 training and 10, 000 testing handwritten digits of size 28 x 28. We suppress the labels and use these images alone to train QEP-LVMS for
1.5 and 2.0 (Gaussian) with 128 inducing points. We set the latent dimension to 10 and use t-SNE to project latent spaces by VAE, GP-LVM and QEP-LVM (q 1.5) to 2d subspaces in Figure 4 for better visualization. Unlike VAE generating unstruc- tured latent representation with all digits mixed up, outputs by LVMs are much more interpretable. While GP-LVM has segregated clusters of digits and 7 respectively, QEP-LVM has more concen- trated clusters with each further apart from others, despite confusion-prom groups 3-5-8 and 4-9-(7). Figure C.7 compares the pairwise (10 digits divided into 5 groups of confusing pairs) latent rep- resentations by GP-LVM (lower) and QEP-LVM (q 1.5, upper). QEP-LVM (g = 1.5) has latent representations better separated in the pairs of (0, 6), (2, 3) and (5, 8) compared with GP-LVM. Seen from Figure(C.8 QEP-LVM (q- = 1.5) generates sample digits of 3, 5 clearer than GP-LVM. Table compares the generative classifiers constructed as in Section 3.4 QEP-LVM (q 1.5) outperforms GP-LVM (g = 2.0) in terms of ACC and AUC. If we view the class labels (not used in training) as cluster assignments, QEP-LVM (q- 1.5) also has better ARI and NMI scores for clustering.
Table Classification on handwritten digits (MNIST) based on learned latent representation: accuracy (ACC), area under ROC curve (AUC), adjusted rand index (ARI) score, normalized mutual information (NMI) score, log predictive probability (LPP) values, and running time (per class) by various Bayesian QEP-LVMs. Results in each cell averaged over 10 experiments with different random seeds; values after + are standard errors of these repeated experiments.
Model (g)
ACC
AUC
ARI
NMI
LPP
lime/class
1.5 (Gaussian)
0,973 + 0.012
0.986 + 0.0065
0,943 + 0.024
0.952 + 0.020
-26453.5 + 481.9
129.91 + 4.60 127.18
0.923
0.025
0.938
0.019
-279414.
5184.6
5 CONCLUSION
In this paper, we introduce a novel Bayesian latent variable model based on the recently proposed Q- exponential process (Li et. 2023) (QEP-LVM) for latent representation learning. Q-EP empowers the LVM with flexible regularization that controls the complexity of the leamned latent representa- tions often with improved interpretability compared with GP-LVM, a special case of QEP-LVM for q 2. The theoretic connection between QEP-IVM and probabilistic PCA has been established. Using three examples, we demonstrate the advantage of the proposed methodology terms of qual- ity of learned latent representation and quantitative performance the derived generative models.
remains unknown how to efficiently deal with missing data, one of the possible defects of the pro- posed QEP-LVM. Data imputation based the variational distribution for latent function, qff, (X). has no tractable form when integrating with respect to (X) as for GP-LVM (Equation (21) in Titsias & Lawrence (2100). Perhaps Monte Carlo approximation is unavoidable. Another interesting di- rection is to generalize QEP-LVM to deep Q-EP as deep GP (Damianou & Lawrence 2013), which will treat in separate paper.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
SL is supported by NSF grant DMS-2134256.
REFERENCES
Sumon Ahmed, Magnus Rattray, and Alexis Boukouvalas. GrandPrix: scaling up the Bayesian GPLVM for single-cell data. Bioinformatics, 35(1):47-54, 07 2018. ISSN 1367-4803.
Steven Atkinson and Nicholas Zabaras. Structured bayesian gaussian process latent variable model: Applications to data-driven dimensionality reduction and high-dimensional inversion. Journal of Computational Physics, 383:166 66-195, 2019. ISSN 0021-9991.
C.M. Bishop and G.D. James. Analysis of multiphase flows using dual-energy gamma densito- metry and neural networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 327(():580-57) 593, 1993. ISSN 0168-9002.
Zhengming Chen, Feng Xie, Jie Qiao, Zhifeng Hao, Kun Zhang, and Ruichu Cai. Identification of linear latent variable model with arbitrary distribution. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6):6350-6377, June 2022. ISSN 2159-5399..
Andreas Damianou and Neil D. Lawrence. Deep Gaussian processes. In Carlos M. Carvalho and Pradeep Ravikumar (eds.), Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Research, pp. 207- 215, Scottsdale, Arizona, USA, 29 Apr-01 May 2013. PMLR.
Juan Antonio Delgado-Guuerren, AdriĀ Colomā@, and Carme Torras Sample-efficient robot mo- tion learning using gaussian process latent variable models. In 2020 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 314-320, 2020.
K. Fang and Y.T. Zhang. 11803871765122
Generalized Multivariate Analysis. Science Press, 1990. ISBN
Jacob Gardner, Geoff Pleiss, Kilian Q Weinherger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matris gaussian process inference with gpu acceleration. In S. Bengio, H. Wal- lach, H. Larochelle, Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Gregory Gundersen, Michael Zhang, and Barbara Engelhardt. Latent variable modeling with ran- dom features. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings The 24th Interna- tional Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 1333-1341. PMLR, 13-15 Apr 2021.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Eighteenth International Conference Artificial Intelligence and Staristies, volume 38 of Pro- ceedings of Machine Learning Research, pp. 351-360, San Diego, California, USA, 09-12 May 2015. PMLR.
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25 (2):328-373, February 2013. ISSN 1530-888X.
I. T. Jolliffe. Principal Component Analysis. Springer Series in Statistics. Springer- 1986. ISBN 0387954422.
Ian T. Jolliffe and Jorge Cadima. Principal component analysis: review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sci- ences, (7412065)201100002 April 2016. ISSN 1471-2962..
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations. International Conference on Learning Representations, 2014.
11

---

Published as a conference paper at ICLR 2025
Tore Selland Kleppe and Hans J. Skaug. Building and fitting non con-gaussian latent variable models via the moment moment-generating generating function. Scandinavian Journal of Statistics, 35(4):664 676, 2008.
Neil Lawrence. Gaussian process latent variable models for visualisation of high dimensional data. In S. Thrun, L. Saul, and B. Schölkopf (eds.), Advances in Neural Information Processing Sys- tems, volume 16. MIT Press, 2003.
Neil Lawrence. Probabilistic non-lines principal component amalysis with gaussian process latent variable models. Journal of Machine Learning Research, 6(60):1783-18)) 2005.
Thanh Le and Vasant Honavar. Dynamical gaussian process latent variable model for representation learning from longitudinal data. In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, volume 21 of FODS -20, pp. 183-188. ACM, October 2020.
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6(11))227822322 1998.
Shuyi Li, Michael O'Connor, and Shiwei Lan. Bayesian learning via q-exponential process. In Proceedings of the 37th Conference on Neural Information Processing Systems. NeurIPS, 12 2023.
Ying Li, Zhidi Lin, Feng Yin, and Michael Minyi Zhang. Preventing model collapse in Gaussian process latent variable models. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 28278-28308. PMLR, 21-27 Jul 2024.
Marsland. Machine Learning: An Algorithmic Perspective, chapter 6. CRC Press, edition, 2014. ISBN 9781420067194.
Thomas Minka. Automatic choice of dimensionality for pea. In Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
Hannes Nickisch and Carl Edward Rasmussen. Gaussian Mixture Modeling with Gaussian Process Latent Variable Models, pp. 272-282. Springer Berlin Heidelberg, 2010. ISBN 9783642159862.
Rok Pahič, Zvezdan Lončarević, Andrej Gams, and Aleš Ude. Robot skill learning in latent space of deep autoencoder neural network. Robotics and Autonomous Systems, 135:103690, 2021. ISSN 0921-8890.
Jason Palmer, Kenneth Kreutz-Delgado, Bhaskar Rao, and David Wipf. Variational em algorithms for non-gaussian latent variable models. In Y. Weiss, B. Schölkopf, and J. Platt (eds.), Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005.
Karl Pearson. Liii on lines and planes of closest fit to systems of points in space. The London, Ed- inburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, November 1901. ISSN 1941-5990.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Muchine Learning. The MIT Press, 2005.
Saber Salehkaleybar, AmirEmad Ghassami, Negar Kiyavash, and Kun Zhang. Learning linear non- gaussian causal models in the presence of latent variables. Journal of Machine Learning Research, 21(39):1-24, 2020.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robers Müller. Nonlinear component analysis as kernel eigenvalue problem. Neural Computation, 1295): 1319, 1998.
Michael E. Tipping and Christopher M. Bishop. Probabilistic Principal Component Analysis. Jour- nal of the Royal Statistical Society Series B: Statistical Methodology. 61(3):611666 09 1999. ISSN 1369-7412.
12

---

Published as a conference paper at ICLR 2025
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
Michalis Titsias and Neil D. Lawrence. Bayesian gaussian process latent variable model. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Con- ference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 851, Chia Laguna Resort, Sardínia, Italy, 13-15 May 2010. PMLR.
Raquel Urtasun and Trevor Darrell. Discriminative gaussian process latent variable model for clas- sification. In Proceedings of the 24th international conference Machine learning, ICML -07 & ILP -07, pp. 927-934. ACM, June 2007.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-Gaussian latent hierarchical structure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 24370-24387. PMLR, 17-23 Jul 2022.
Michael Minyi Zhang, Gregory W. Gundersen, and Barbara E. Engelhardt. Bayesian non-linear latent variable modeling via random fourier features. 06 2023.
13

---

Published as a conference paper at ICLR 2025
Supplement Document for "Bayesian Regularization of Latent Representation"
A PROOFS
Proof of Theorem 3.1
The gradients of log-likelihood
S with respect to K can be found as
OL
D
N
1 r
K-
1)
r
K KY'
(11)
OK
2
2
2
22
The MLE for X can be found by setting
aL
2a
-1
0L
0. which leads to
0x
5K
2 - 1) -- 2 D-- x ND
(12)
Now suppose we have the formal solution for (12 as ULV, where is an x Q matrix whose only nonzero entries {4] are the main diagonal be determined. Based the formal solution of X, we have
DAQ == U(a ULa LL + 'DU,, r(Y) tr(K (YY)) Σ DN 11 12+
Denote by h(r) - ND - + 3r We substitute the above quantities into 12 and get
ULV UA 'LLT + ---())-
which reduces to
= A1 1 + 13--111)4,00) i.I....,AA
Let l(r) with e to be determined. Assume 4; 0. Then solve 4; -a(c))-8-1))
This yields c D(D Q). Hence
h(()) = N 2 - 1) (DD A))
And solves e == (DD 2(DAQ) +g 2)
Hence the proof is completed.
N=1000, Dw200,
N=10, Dw100,
N=1000, Q=D/20
850
500
35-
3.0
400
400
40
23
425 8.50
6.5
Figure A.1: Function e(g) Theorem 3.1 that regularizes the singular values of latent variable X.
Left two panels: c(g) for fixed N, D, Q: right two panels: c(q) for fixed N and Q D/10.
14

---

Published as a conference paper at ICLR 2025
B VARIATIONAL BAYES FOR QEP-LVM
B.1 EXPECTED KERNEL TERMS
We compute the tr((KNN))(0x)) V1 (K (KNM)) and W22 (KMMNKNM))(X follows:
/ (Q) Il: S., )
1/6
(V))mm =
k: (xn, Zm )- x [nn, Sn,dx
2 mame
k(Xn, Zm Yk(zm/ X. (D Sn)dxa.
With ARD SE kernel 6 we have thp = Na While the integration in Vi and V2 is intractable general, we can compute them using Monte Carlo approximation. Alternatively, we approximate
V 1)mm a- exp
(Xx, 2m)
diag(>)) Zm ((xx)))
a exp
[(m mm) " d'dddgg)) km) (r(ding(7)S..))
1 Σ (m Zm) ) diag(7) zm)) + an(diag(7)S,)
V mm a -² exp
2 m=mmm
If use the ARD linear form, k(x, x') xx diag(7)x', then we have
VO (r(diag(7) + +,,)): (V1)am = ph cliag(7) 2mm 1 mmmm = I diag(t)) + S.) diag(7)zzm"
B.2 LOWER BOUND FOR THE K-I. DIVERGENCE ADDED TERMS
We also need compute the K-L divergence KLu (L(((U)))(()):
KLU = /
log (og)) - q logg) = -2 (U) - (log p ())))))
Denote by r = vec (U M) -vec) M). Then log (U)
vec
- 2 DD log Eal + MD
2 ( 1) logr
1r², From (Proposition A.1. of Li et al. 2023) we
know that r (M Therefore
MD q -1)
2 H(x (H(x2(D)) MD
(U)
log En +
Σ
D
log Egl + 2
MD
(1-
MD
+ log 2T
MD
+ (1
MD 2
+
MD
+
MD
2
2
2
2
2
d=1
Denote by (r(() - log KM + MD
(1 1) log 1 Then by Jensen's inequality
2
(logp(U)))(u) = ( -- ())) 2 (p (rr((ttU" MM U))
KMM
(tr(U MM ()) = tr(M K K M) + Σ tr(EE
Therefore we have the K-L divergence added term KLu bounded by
D
KLu 2 KLL = 2
Σ
log Ea + (P(((t(UU K K ))
15

---

Published as a conference paper at ICLR 2025
Lastly, we bound the K-L divergence term kLx := ([[[((()|p(X)) by similar argument:
N
KLx 2 -KL = 2
Σ
log |Sm| + (xxp((r(x'x))_xpm)
n=l
where (r) == NQ
(2 1) logr r and (tr(( , = tr(p (0) + Σ n=1 tr((S,).
r-100, D-100
r-0.1, D-1000
D-100
D-1000
191
-10
130
-150
12"
logils
250
10y
10P
-30
10-1
10"
-30
10
460
3.0
Figure B.1: aogllikelihod function p(g). Left two panels: (p()) for fixed r, D; right two panels: (Q)) for fixed D.
16

---

Published as a conference paper at ICLR 2025
C MORE NUMERICAL RESULTS
C.1 SWISS HOLE
Figure C.I: Pairwise densities between regularization parameter q and kernel length-scales (1,, 12, 1g) in Swiss roll (left) and Swiss hole (right) problems.
Isomap
E-NNE
Isomap
ESNE
Seg
Lsantam
Lasaet am
Latant dm
Figure C.2: Latent representation of Swiss roll (left two) and Swiss hole (right two) by Isomap and t-SNE respectively.
(Goussian)
dot 1
Loterddi
Litent Brn.
Vartanoes Ictent ditribution
Inverse Longthscale
Langthocall lamel
kimm!
02
33
Figure C.3: Latent representation of Swiss hole dataset. Upper left: 3d cloud of 1000 points; lower left: PCA in 2d principal space; right 3 columns: 2d latent representations (upper row) by VAE and QEP-LVMs with 4 = 1.0, 1.5 and 2.0 (GP-LVM) showing regularization effect via the parameter and the corresponding variances of latent distribution (VAE) and inverse length-scales y (lower row). Colors are used to aid visualization but not for training.
17

---

Published as a conference paper at ICLR 2025
1:37
Inverse Lengthscale kernel
Neg. ELBO Loss
Postento
035
Hy
6:11
Laterd tim
20
Figure C.4: Latent representation of Swiss hole dataset output by QEP-IVM with optimal g" = 1.37 (red line in the rightmost panel) found in the Bayesian framework.
C.2 ROBOT LEARNING
In this section, we consider the robot learning problem in Pahic et al. (202T). Robots can improve their performance by repeating desired behavior and updating the learned skill representation. So it important to obtain the latent space of motor skills on which statistical and reinforcement learning be applied optimize robot's behavior. We focus the task of robotic ball throwing target, which is realized by a 7 degree of freedom (DOF) arm Mitsubishi PA-10. Three DOFs of the robot, which contribute its motion in the saggital plane, are used for the ball throwing. Dynamic movement primitives (DMPs) are used to represent smooth robot motion (Jjspeert et al. 2013) and with N 20 DMP weights for each DOF we get 60 weights the DMP parameter space. Here we test PCA, VAE, beta- VAE, and QEP-LVMS in learning the latent representation of motion skills.
In this dataset. there are no labels. We compare the latent spaces generated by different algorithms in Figure C.5 We set latent dimension to 10 and visualize the 2d subspaces projected by t-SNE algorithm. VAE and QEP-LVM (g = 1.0) output more structured latent spaces than others. con- trast, the latent representations by beta- VAE and GP-LVM appear more dispersed and unstructured. Among all the algorithms, only beta-VA and QEP-LVM (g = 1.0) correctly identify the intrinsic dimensionality 3 (DOFs) of the latent space.
Mausienn
Lakare 2012
2012
Laterctio
Latendcte
Labent der
dialiuuten
IVESE Lengthscil lernel
Ieveric Lergt stae
800
Figure C.5: Latent representation of robot learning dataset. Upper: 2d latent representations output by PCA, VAE, Meta-VAE, GP-LVM and QEP-LVM (q = 1.0). Lower: latent dimensions indicated by dominant eigenvalues (PCA), variances of latent distribution (VAEs), and inverse congth-scales, QBP-LVMS. For the convenience of visualization, 10-dimensional latent spaces learned by these algorithms are projected to 2-d subspace by (-SNE respectively.
C.3 OIL FLOW
C.4 MNIST
18

---

Published as a conference paper at ICLR 2025
subepare
lengthorale
Neg.
30
Latent dim 1
Figure C.6: Latent representation of oil flow dataset output by QEP-LVM with optimal q" = 1.57 (red line in the rightmost panel) found in the Bayesian framework.
[0.6]
[1.7]
[2:3]
of
Labert din
-00
-10 60 25
zo 25 2d lasent of [5.8]
latent
2d latent
2d latent
2d [4.9]
LAtent 2
3
Latent
Latent dim
Latent dim
Figure C.7: Pairwise latent representations of MNIST database by QEP-LVMs with q 1.5 (upper)
and = 2.0 (Gaussian, lower).
2
/
3
4
5
6
7
8
9
0
/
2
3
4
S
6
9
8
9
Figure C.8: Sample MNIST digits by QEP-LVMs with q 1.5 (upper two rows) and q 2.0 (Gaussian, lower two rows).
19


Reviews:
[{"review_id": "Yw2FVYE62L", "reviewer": "Reviewer_mVse", "summary": "This paper extends Gaussian Process Latent Variable Models (GP-LVM) by replacing the Gaussian Process with a more general Q-exponential process, thereby obtaining a series of new LVMs, Q-exponential Process LVMs (QEP-LVM). The author points out the relationship between QEP-LVM and non-linear probabilistic PCA, derives the ELBO of Bayesian QEP-LVM, and introduces a mechanism to optimize the ‘q’ of q-exponential distribution. Experiments prove the effectiveness of the proposed QEP-LVM.", "questions": "In the visualization of the latent representation in Figure 3, it seems that the separation of points is not as good as in Figure 2 when q=1.5. Is this an illusion? If not, what is the reason for it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation of this paper is intuitive and reasonable.\n2.\tThe research on the proposed QEP-LVM is very solid. Not only does it reveal the relationship between QEP-LVM, GP-LVM and non-linear PCA, but it also derives the tractable ELBO of Bayesian QEP-LVM. In addition, a Bayesian approach is developed to obtain the optimal q.\n3.\tThis paper is clearly written.", "weaknesses": "1.\tThe paper includes numerous visualization experiments; however, some, like Figure 5, do not reveal significant performance differences between QEL-LVM and GP-LVM. It is recommended to incorporate more numerical comparisons. For instance, for the cluster formed by the two numbers in Figure 5, utilize evaluation metrics from clustering algorithms to assess and compare the proposed QEP-LVM and GP-LVM.\n2.\tIt appears that the proposed QEP-LVM requires additional training during the testing phase to compute the ELBO. How long does this training take, and will it impact the practicality of QEP-LVM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "phEh4XfOwm", "reviewer": "Reviewer_u49y", "summary": "This paper proposes a model that extends the Gaussian Process Latent Variable Model (GP-LVM) to a non-Gaussian framework by employing the Q-exponential process (QEP). Specifically, the authors replace the Gaussian log-likelihood’s $ l_2 $ norm component with an $ l_q $ norm for $q > 0$, introducing an alternative to Gaussian assumptions. They derive both Maximum Likelihood Estimation (MLE) and variational Bayesian inference methods for this model. Experimental evaluations are conducted on the Swiss roll, oil flow, and MNIST datasets, and the results are compared with those of the GP-LVM.", "questions": "For more detailed points, please refer to the weaknesses section above.\n\nMinor Comments:\n- The notation $ q-ED $ could be improved as it might be misinterpreted as \"q minus ED.\" A clearer notation, such as $ q_{ED} $, might enhance readability.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clarity**: The paper is well-organized and clearly written, making it accessible for readers.\n- **Contextualization**: The authors provide a thorough review of related work.\n- **Theoretical Derivation**: While I haven't exhaustively verified here, the paper appears to present a sound derivation of the ELBO within the variational Bayesian framework.", "weaknesses": "- **Significance of Contribution**: The experiments focus on datasets with limited scale and relatively artificial characteristics, which limits the persuasiveness of the findings. The results do not convincingly demonstrate a substantial improvement over the GP-LVM. Testing the proposed method on larger datasets or more practical tasks (e.g., robotics) could improve the impact of the results.\n- **Comparison with Other Methods**: The experimental comparisons are restricted to the GP-LVM. To better assess the advantages of the QEP-LVM, it would be valuable to include comparisons with other methods suited for non-Gaussian or manifold data, such as Isomap for Swiss roll or neural networks for classification tasks like MNIST and oil flow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnSMImTVyJ", "reviewer": "Reviewer_CSd1", "summary": "The paper introduces a generalisation of the GP-LVM with the novel Q-Exp process. The training ELBO is derived and the performance is shown in multiple experiments to be improved upon GP-LVM.\n\n\nAfter the first rebuttal answer, I have raised my score by one point.", "questions": "- Definition 1: \n  - Can you give an intuition on how the q-exp distribution is different from the Gaussian potentially with examples? How does the distribution change with $q$?\n  - What values can $q$ take in theory and what values are suitable in practice? You mainly focus on $0<q<2$ in the experiments, why is that?\n- Remark 1: \"more regularization\". What form does the regularization take and where do I see this in the mathematical expressions? How does it connect to other commonly known forms of regularization?\n- Section 2.2: can you clarify what the latent variables are here? With $Q\\ll D$, I assume that $f$ is the latent but in (1) we have $y=f(x)+\\varepsilon$. Then, in line 184 $X$ is the latent variable. This is confusing. Please clarify. \n- Section 2: Can you give a brief overview on GP-LVM? This can help to compare your development more directly and see the differences.\n- Line 161: How does replacing the GP with Q-EP impose more regularization?\n- Remark 2: \n  - How does $q>0$ regularize the singular values? Which singular values do you mean explicitly?\n  - $c(q)$ is not defined?\n- Figure 1: \n  - What is the colour scale?\n  - In the rightmost plot what is the artifact in the bottom left corner?\n- Line 220: \"can automatically determine the dimensionality of the nonlinear latent space\". Can you explicitly clarify how this is achieved and show this in experiments? Currently, this claim is not fully supported.\n- Line 261: \"in two stages\". Please clarify which stages.\n- Line 264: What is $M$?\n- Section 3.2.2: \n  - The summary in section 3.2.2 is highly appreciated but can you additionally provide an algorithmic overview of how the model is trained and how it is used during inference?\n  - What is the computational complexity of your method and how does it compare to GP-LVM? How does it scale larger sample sizes? What does this imply in practice for the use of your algorithm compared to GP-LVM? Please discuss.\n  - What are the hyperparameters, e.g. $\\beta$ and how are they chosen, what effect do they have, and how robust is the training towards those hyperparameters? Please discuss and if possible show in experiments.\n- Remark 4: With $q=2$ you claim that the Gaussian case is recovered. But here you show that there is an extra term. What am I missing or where does this discrepancy arise from? This would affect all your experiments since then $q=2$ is not exactly the GP-LVM case anymore.\n- Line 363: \"superior\" in what sense?\n- Line 377: \"best\" in what sense?\n- Line 377: \"regularization effect\", can you explicitly state how we can see this? Does this mean an axis alignment effect similar to Lasso regularization?\n- Figure 2 (bottom): What is the x-axis of the right plots, $q$?\n- Figure 3: Why is the 2d latent subspace so different with $q=1.57$ to the one in Figure 2 with $q=1.5$? How is the representation with $q=1.57$ good, the colours of the datapoint are overlapping?\n- Table 2: The results of the Gaussian case are within the standard errors (at least in terms of AUC). Please highlight and discuss that.\n- Line 528: The connection to probabilistic PCA is not made explicit. Please work that out to support your claim here and from the abstract.\n\nSmaller details:\n- Citations need to be revisited:\n  - Line 42: Schö[l]kopf. Please check all citations for correctness.\n  - Line 82: Lowercase the citation KLEPPE & SKAUG\n  - E.g. Line 141: Citations that are used within the text should not contain brackets such as \"(Li et al., 2023, in Theorem 3.5) prove...\".\n  - E.g. Shuyi Li, ... Bayesian Learning via q-exponential process, has the NeurIPs and arXiv reference.\n- Wording needs improving in some places:\n  - Line 161: \"[The] GP...\"\n  - Line 411: \"of [the] kernel\"\n- Introduce non-standard notation for easy understanding:\n  - Line 179: $\\bigotimes$\n  - Line 196: $\\wedge$\n- Table 1/2: Why is the caption font size smaller?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "Disclaimer: I am not familiar with QEP and am only partially familiar with GP-LVM. Thus my judgement is at times more superficial.\n\n- The generalisation of GP-LVM offers more flexibility and improved performance\n- While I have not managed to check all the mathematical details, they seem to be well executed.\n- Experiments in small-scale toy problems demonstrate the performance of the method.", "weaknesses": "- The main weakness for me lies in the presentation which is hard to follow and at places unclear. See the questions below for more details on this point.\n- Some claims such as the regularization effect and the connection to probabilistic PCA are not sufficiently worked out and need improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cp9eNyrzBI", "reviewer": "Reviewer_th3c", "summary": "This paper extends the GP-LVM to a more general form, known as QEP-LVM, which regularizes the model using an Lq penalty. It establishes the equivalence between GP-LVM and a specific case of QEP-LVM. The experimental results conducted on three real-world datasets demonstrate the good performance of the proposed approach.", "questions": "1. What is the practical meaning of q? Also, the author mentioned that it's based on the Lq penalty, but I'd like more explanation, like why q ranges from 1 to 2.\n2. Does the choice of kernel impact the performance of the QEP-LVM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well-organized and easy to read.", "weaknesses": "1. It would be beneficial to include the time complexity analysis and a comparison of the running times between GP-LVM and QEP-LVM, especially for a relatively large data set -- MNIST.\n2. For me, it represents a natural combination of QEP and LVM. While the creativity is incremental, I would like to see more applications in other areas where Gaussian processes are typically utilized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)