Multi-View Oriented GPLVM: Expressiveness and Efficiency
Authors: Zi yang
Ying Li
Zhidi Lin
Michael Minyi Zhang
Pablo M. Olmos
Venue: NeurIPS 2025 poster ()
Keywords: Multi-view representation learning
Kernel learning
Random Fourier feature
Primary area: probabilistic_methods

Abstract: The multi-view Gaussian process latent variable model (MV-GPLVM) aims to learn a unified representation from multi-view data but is hindered by challenges such as limited kernel expressiveness and low computational efficiency. To overcome these issues, we first introduce a new duality between the spectral density and the kernel function. By modeling the spectral density with a bivariate Gaussian mixture, we then derive a generic and expressive kernel termed Next-Gen Spectral Mixture (NG-SM) for MV-GPLVMs. To address the inherent computational inefficiency of the NG-SM kernel, we propose a random Fourier feature approximation. Combined with a tailored reparameterization trick, this approximation enables scalable variational inference for both the model and the unified latent representations. Numerical evaluations across a diverse range of multi-view datasets demonstrate that our proposed method consistently outperforms state-of-the-art models in learning meaningful latent representations.

Multi-Viee Oriented GPLVM: Expressiveness and Efficiency
Zi Yang"
Ying Li
School of Computing and Data Science
School of Computing and Data Science University of Hong Kong 1 1ynn1i98@oonnect hku. hk
University of Hong Kong
ziyang20238comeect hku. hk
Zhidi Lin
Michael Minyi Zhangi
School of Computing and Data Science
School Computing and Data Science
University of Hong Kong
University of Hong Kong mzhang 180nku. hk
1 Ainzhidi017@mmai..com
Pablo M. Olmos
Department of Signal Theory and Communications
Universidad Carlos III de Madrid
pamartineing.
werinnn..cmm.e.m uc3m.es
Abstract
The multi-view Gaussian process latent variable model (MV-GPLVM) aims to learn unified representation from multi-view data but is hindered by challenges such limited kernel expressiveness and low computational efficiency. To overcome these issues, we first introduce a new duality between the spectral density and the kernel function. By modeling the spectral density with bivariate Gaussian mixture, we then derive generic and expressive kernel termed Next-Gen Spectral Mixture (NG-SM) for MV-GPLVMs. To address the inherent computational inefficiency the NG-SM kernel, we design a new form random Fourier feature approx- imation. Combined with tailored reparameterization trick, this approximation enables scalable variational inference for both the model and the unified latent rep- resentations. Numerical evaluations across diverse range of multi-view datasets demonstrate that our proposed method consistently outperforms state-of-the-un models in learning meaningful latent representations.
Introduction
Multi-view representation learning aims to construct a unified latent representation by integrating multiple modalities and aspects of the observed data [1, 2]. The learned representation captures inter-view correlations within observations. By sharing information between each view of the data, we can obtain a much richer latent representation of the data compared to modeling each view independently which is crucial for handling complex datasets [3-5]. For example, modeling video data which involves both visual frames and audio signals [6], or developing clinical diagnostic systems that incorporates the patients' various medical records [7].
The paradigm of multi-view learning first emerged in early works leveraging techniques such as canonical correlation analysis (CCA) [8] and its kernelized extensions [9, 10] to jointly model
"Equal contribution
Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: An overview of relevant MV-LVMs, where extend advanced GPLVMs to the multi-view case by prefixing them with MV. Additionally, N represents the total numher of observations, M denotes the observation dimensions, V refers to the number of views, refers to number inducing points, indicates the number of GP layer, and L indicates the dimension of random features.
Scalable model
Highly expressive kernel
Probubilistic mapping
Bayesian inference of latent variables
Computational complexity
Model
Reference
MVAE
X
[11]
MM-VAE
[12, 16]
MV-GPLVM
O(N
[22]
MV-GPIYM-SVV
O
[23]
MV-RELVM
X
/
(INEZV)
[24]
MV-DGPLVM
X
/
OUINUI
[14]
MV-ARFIVM
/
O(NTAV)
[21]
NG-MVLVM
/
O(NEAV)
This work
correlated data. However, these methods are limited their ability to capture latent representations advanced approaches have become
in complex datasets. To address this limitation, standard in multi-view learning: neural network-basee -bsse methods, exemplified by multi-view variational autoencoders (MV-VAEs) [11, 12], and Gaussian process-based methods, represented by multi-view Gaussian process latent variable models (MV-GPLVMs) [13, 14].
MV-VAFs incorporate various view-specific variational autoencoders (VAHs) to address multi-view representation learning [11, 12, 15-17]. However, they suffer from posterior collapse-an an inherent issue in VAEs [18]-wher) the encoder collapses to the prior distribution over the latent variables, thereby failing to capture the underlying structure of the data. Several hypotheses have been proposed to explain posterior collapse. A well-known contributing factor is the overfitting of the decoder network [19, 20). One promising direction to mitigate this issue is to introduce regularization in the function space of the decoder, which may help the latent variable model learn more informative representations.
The implicit regularization imposed by the Gaussian process (GP) prior helps prevent MV-GPLVMs from severe overfitting [21]. However, MV-GPIVME often lack the kernel flexibility required to model complex representations [13], and are computationally intensive to train, especially on large- scale multi-view datasets [14]. To overcome these challenges, we propose an expressive and efficient multi-view oriented GPLVM. Our contributions are:
We establish novel duality between the spectral density and the kernel function, deriving the expressive and generic Next-Gen Spectral Mixture (NG-SM) kemel, which, by modeling spectral density dense Gaussian mixtures, can approximate any continuous kernel with arbitrary precision given enough mixture components. Building on this, we design a novel MV-GPLVM for multi-view scenarios, capable of capturing the unique characteristics of each view, leading to an informative unified latent representation.
To enhance the computational efficiency, we design a unique unbiased random Fourier features (RFF) approximation for the NG-SM kernel that is differential w.r.t. kernel hyper- parameters. By integrating this RFF approximation with an efficient two-step reparameteri- zation trick, we enable efficient and scalable learning of kernel hyperparameters and unified latent representations within the variational inference framework [15], making the proposed model well-suited for multi-view scenarios.
We validate our model on a range cross-domain multi-view datasets, including syn- thetic, image, text, and wireless communication data. The results show that our model consistently outperforms various state-of-the-e (SOTA) MV- VAEs, MV-GPLVMs, and multi-view extensions of SOTA GPLVMs in terms of generating informative unified latent representations.
2 Background
Gaussian Processes. Probabilistic models like the Gaussian process latent variable model (GPLVM) [25] introduces regularization effect via GP-distributed prior that helps prevent overfitting and

---

thus improves generalization from limited samples. A GP f(-) is defined as a real-valued stochastic process defined over the input set X RD, such that for any finite subset of inputs (x,, (x, C X. the random variables f = (f(x,) ) J
n== follow a joint Gaussian distribution [26]. A common prior
choice for a GP-distributed function is:
f (( 0, K),
where K denotes the covariance/kermel matrix evaluated on the finite input X with the kernel function 1(x1:22), i.e., [K] = (x,, Xy), E (1....N).
Consequently, the GPLVM has laid the groundwork for several advancements in multi-view represen- lation learning. One of the early works by Li et al. [13] straightforwardly assumes that each view in observations is projection from shared latent space using GPLVM, referred as multi-view GPLVM (MV-GPLVM).
Multi- Gaussian Process Latent Variable Model. A MV-GPLVM assumes that the relation- ship between each view E (1, ..., V) of observed data Y Y .... "......"" V",,, e. Nx xM, and the
shared/unified latent variables X x=xx1,x,, X2,... ..... XN] E RNxD is modeled by a GP. That is, in each dimension m E (L, ...,...,,. M,) and view v, MV-GPLVM is defined as follows:
y m.
f (X) ~ (N)
f% ( ~ N',, X, NOO,I),
(2)
where of represents the noise variance and f. (X)
= [f (x1) ... fm (xN)) ERN. The stationary
radial basis function (RBF) typically the 'default' choice of the kernel function. Due to the conjugacy between the Gaussian likelihood and the GP, we can integrate out each (.) and get the marginal likelihood formed as:
y.m ~ (O, +T
Based on the marginal likelihood and the prior of X, the maximum a posteriori (MAP) estimation of X can be obtained, with O(N3) computational complexity due to the inversion of the kernel matrix. Eleftheriadis et al. [27] extend the MV-GPLVM with a discriminative prior over the shared latent space for adapted to classification tasks. Later, Sun et al. [14] incorporate deep GPs [28] into MV-GPLVM for model flexibility, named MV-DGPLVM. However, existing MV-GPLVMs still fall short in handling practical multi-view datasets, that are often large-scale and exhibit diverse patterns across views. This limitation arises from either (1) the high computational complexity of fitting a deep GP or (2) limited kernel expressiveness caused by the stationary assumption. Particularly, stationary kernel may fail to model imput-varying correlations [29], especially in domains like video dynamics.
analysis and clinical diagnosis that exhibit complex,
To address those issues, we review the recent work of GPLVM [21, 23, 24], that we may extend to deal with the multi-view scenario (See detailed comparisons in Table 1). One potential solution is the RddvseddRFLVM (named ARFLVM for short) [21]. This method integrates the stationary spectral mixture (SM) kernel to enhance kernel flexibility in conjunction with scalable random Fourier feature (RFF) kernel approximation. However, it remains limited by the stationary assumption.
Random Fourier Features. Bochner's theorem [30] states that any continuous stationary kernel and its spectral density p(w) are Fourier duals, i.e., k(x; -x2) = I p(w) exp (iw (X1 (xx) dw.
Built upon this duality, Rahimi and Recht [31] approximate the stationary kernel k(x, - X2) using an unbiased Monte Carlo (MC) estimator with L/2 spectral points (w(()) L/2 sampled from p(w). given by
k(x1 - xx) R x(x1) ((x2),
where the random feature
((x) / cos 2TTx ( sin 2TTxTT w w./l//2]] R-4.
Here the superscript 1 L/2 indicates that the cosine sine function is repeated L/2 times, with each element corresponding to the one entry of (w(1)) / Consequently, the kernel matrix K can be approximated by (xx)((x)) with & (x) = (é(x1)i... "(xx)] By employing the RFF approximation, the kernel matrix inversion can be computed using the Woodbury matrix identity [32],
thereby reducing the computational complexity (2
"Ettending other likelihoods is straightforward. guided by the GP literature [23, 24].
3

---

3 Methodology
In $ 3.1, we introduce our expressive kernel function for multi-view representation learning. In $ 3.2, we propose a sampling-base variational inference algorithm to efficiently learn both kernel hyperparameters and latent representations. The scalable RFF approximation and the associated efficient reparameterization trick are detailed in $ 3.3.
3.1 Next-Gen SM (NG-SM) Kernel
As mentioned in 2, limited kernel expressiveness in the MV-GPLVM may hinder its ability to capture informative latent representations, potentially neglecting crucial view-specific characteristics like time-varying correlations in the data'. This problem necessitates a more flexible kernel design. The bivariate SM kernel (BSM) is one notable development for improving kernel flexibility [29, 33, 34]. Grounded in the duality that relates any continuous kernel function to dual density from the generalized Bochner's 's theorem [35], this method first models the dual density using a mixture of eight bivariate Gaussian components and then transforms into the BSM kemel. By removing the restrictions of stationarity, the generalized Bochner's 's theorem enables the BSM kernel to capture vire-varying correlations [29].
However, the BSM kernel faces several limitations imposed by the generalized Bochner's 's theorem: (I) To guarantee the positive semi-definite (PSD) spectral density, the two variables in the bivari- ate Gaussian must have equal variances, which limits the flexibility of the Gaussian mixture and consequently reduces the kernel's 's expressiveness. (2) According to the duality in the generalized Bochner's s theorem, it is not possible to derive closed form expression of random features', such as Φ(x) in Eq. (4). Thus, the inversion of the BSM kernel matrix retains high computational complexity, rendering it unsuitable for multi-vien -iew datasets. To address the limitations of the BSM kemel, we propose the new NG-SM kernel-a a more flexible and RFF-admissible alternative. Its formulation enabled by a novel duality, stated in Theorem 1.
Theorem 1 (Universal Bochner's Theorem). complex-valued bounded continuous kernel l (X1; X2) on TR. is the covariance function of mean square continuous complex-valued random process on IRD D if and only if
4 /
l exp(iw] i x2) + cxp(iw, x1 -w]] x22 iw]xx2) +
1s xx11,x))
exp(iw, x1 iw, x2) + exp( (iw, x1 iw. x22 (dw,,dd))
where u is the Lebesgue-Stienn measure associated with some function p (W1, w2). When W1 W2. this theorem reduces to Bochner's theorem.
Proof. The proof can be found in App. B.2.
The duality established in Theorem I implies that bivariate spectral density entirely determines the properties of a continuous kernel function. this sense, we propose the underlying bivariate spectral density by Gaussian mixture:
Pogsm (W1, W2)
Sq (W1, W2)
(5)
9=1
with each symmetric density Sy (W1, W2)
1 s, (W), W2) - .
((1)
(w W/2/
M4g
Σ Σ
+ 1 N ((w))
Mngl Mup
Σ
Σ
(6)
in order to explore the space of continuous kernels. To simplify the notation, we omit the index from the sub-matrices Σ = Σn = diag(o )2), and Σ. diag(a,.) diag((,z)) where q1: of E R and the scalar Pa denotes the correlation between W1 and w2. These components
4 'For an exploration of the impact of limited kernel expressiveness manifold learning, see $ 4.1.
Seee App. B.I for more detailed discussion.

---

0% ngsm
(w,²)² (w
y.m
X
L/2
M
V
Figure 1: Probabilistic graphical model of our proposed method. Here, Pigam denotes the parameters of Pasam
collectively form the covariance matrix of the q-th bivariate Gaussian component. Furthermore, the vectors M4g1 and Ma2 TR. constitute the mean of the q-th bivariate Gaussian component.
Based on Theorem 1, we derive the NG-SM kernel, fngsm(X1:1 x2) (see Eq. (40) in App. B.3), with kernel hyperparameters
ugaam {ags Myl:Hug2 o q22 P
As the PSD assumption relaxed and the mixtures Gaussians become dense [36], the duality ensures that the NG-SM kernel can approximate any continuous kernel arbitrary well. Next, we will demonstrate that general closed-form RFF approximation can be derived for all kernels based on our duality, which we specify for the NG-SM kernel below.
Theorem 2. Let o(x) be the randomized feature map of Kingnn (x1, (x), defined as follows:
(1:L/2)T
(1:L/2)T
1 2L
+cos
W2
ERL
sin
(1:T/2)T
+sin
W
(1:1./2TT
where the verticully stacked vectors {[w() W2 1 LR22222 are independent and identically
(1)
18)
distributed (ii.d) random vectors drawn from the spectral density Prgsm( W2). Then, the unbiased estimator of the kernel kugam(Xs (x,, X2) using RFFs is given by:
Kngam (x1,X2) xe ((x1) (o21))((())) ((x2).
(8)
Proof. The proof can found App. C.I.
By integrating this unbiased estimator into the framework of MV-GPLVM, we derive the next-gen multi-view GPLVM:
5. NO Th + +o,1), (w)))y, (("))"((w))) (w"))">p"' Prgsm (wf, w>), Xn (0.1),
(9)
where the random feature matrix for each view is denoted t [o(x] 1)=k) i)....... :(xx)) o(xx)] with
the superscript of each feature map omitted for simplicity of notation. The probabilistic graphical model is shown in Figure 1, Furthermore, collectively denote o² {o {o and spectral points W A (W"J
Je=1 with each
W ([ (0): (w DD E R4/2X22
(10)
The following subsections will illustrate how to infer the parameters of the NG-MVLVM through unified sampling-bur based variational inference framework [15].
3.2 Sampling- Variational Inference
We employ variational inference [37] to jointly and efficiently estimate the posterior and hyperparam- eters (0 o²). Variational inference reformulates Bayesian inference task as a deterministic optimization problem by approximating the true posterior p( W, X|Y) using surrogate distri- bution, (W, X), indexed by variational parameters &. The variational parameters are typically estimated by maximizing the evidence lower bound (ELBO) which is equivalent minimizing the
5

---

Kullback-Leible (KL) divergence (L W. x W. W()) between the surrogate and the true posterior.
To construct the ELBO for NG-MVLVM, we first obtain joint distribution of the model:
P PY,X,W) = x P XY" W")
(11)
and then define the variational distributions q( W. X)
(Xx W) a g(X)p(W) x II P(W"),
(12)
um
where I parameters. The variational distribution ((W) is set to its prior ((W) as this is essentially equal to assuming that (() follows a Gaussian mixtures (the justification for this choice of approximation is in App. C.3). Consequently, the optimization problem is:
(xx, pb, S,), and & _ (m, ERD, S,, ERDDDDJ J==1 are the free variational
max (X.W) log (XX IT V=1 P W P X, W 0
((X) II -= (WW)
Σ
B [logp( (Y X, W (X)((X))I((
(a): recomstruction
regularization
which jointly learns the model hyperparameters A and infers the variational parameters &. Term (a) encourages reconstructing the observations using any samples drawn from gXX, W) and term (b) serves as regularization to prevent from deviating significantly from the prior distribution.
To address this optimization problem using the sampling baaedd-bbddddd variational inference [15], we first derive the analytical form of term (b). the KL divergence between two Gaussian distributions, and expand term (a), approximating the expectation via MC estimation,
ΣΣ Mr
log.N (y"m 0, ($" 1 +00
(13)
#=
where I denotes the number of differentiable MC samples drawn from ((X) and p( () with respect to and & (see the further computational details in App. C.2). Then, modern optimization techniques, such as Adam [38], can be directly applied to solve the problem. However, this raises question: How can differentiable MC samples be efficiently generated from the mixture bivariate Gaussian distribution, p(W"), that implicitly involves discrete variables?
3.3 Sampling in Mixture Bivariate Gaussians
In other words, it is essential to both generate differentiable samples from the mixture bivariate Gaussian and ensure high sampling efficiency, which particularly beneficial in the multi-vier case. However, the typical sampling process hinders us from achieving this goal [39, 40]. A primary difficulty stems from first generating an index q from the discrete distribution, P(q)
ag/ Σل= aj.q L, I..,, Q, as it is inherently con-reparamettrizable w.r.t. the mixture weights. Although the Gumbel-SSffmal method provides an approximation, it remains unstable and highly sensitive to hyperparameter choices [41]. Additionally, directly performing joint sampling from s,(W1, W2) (see Eq. (6)) incurs a computational complexity of O(SDS), further complicating the sampling process. Next, we will demonstrate how to address both issues. To simplify the notation, we omit the superscript v in this section.
1) Two-step reparameterization trick
Applying the reparameterization trick on a multivariate normal distribution requires computing the Cholesky decomposition of the full covariance matrix, incurring a computational cost of O(8D3) [39]. To alleviate this computational burden, we propose the two-step reparameterization trick as follows, which reducing sampling complexity to O(D).

---

Proposition (Two-step reparameterization trick), We can sample w1 ww2 from bivariate Gaussian distribution (W(, W2) using the following steps:
(1)
1. W1 Mal + 1g1
2. W2 (1) = H4p + Pa Po(0 (wi ( M21) /1 12 ez.
where Eg, E1 are standard normal random variables and o and represents element-wise multiplica- tion and division, respectively.
Proof. The proof and complexity analysis can be found in App. C.4.
2) Unbiased differential RFF estimator
wq2 gaalig=1l=s> g=1.1=1 Let spectral points W å {w Q2142 be sampled from p (W) = I I Sq (w1, w2)
using the two-step reparameterization trick. Inspired by previous work [21, 42], we first use the spectral points from the q-th mixture component to construct the following feature map:
(x(xx)
Vag o(x;
(w
1 L/2 I1=1' {w 02 1t=1
L/2
(14)
Based on the feature maps {4p (x)) 1- can formulate our RFF estimator for the NG-SM kernel as follows.
Theorem 3. Stacking the feature maps sp (x),) 1, .,.., Q, yields the final form of the RFF approximation for the NG-SM kernel, (xx):
4 (x) = (xx (2((x)
..o.)((x.x)))) EQQ..
(15)
Built upon this mapping, our unbiased RFF estimator for the NG-SM kernel is reformulated as follows:
Ep(w) [x((1)] [x(x))] = -aags (x1, (x1,
(16)
Proof.
The proof can be found in App. C.5.
Moreover, given inputs X and the RFF feature map (xx)), establish the approximation error bound for the NG-SM kernel gram matrix approximation, Kngam 2 &nggsm (X) (X) below,
where + uggmm X)= = 46 (xi)...... (p (xx)) E N
Theorem 4. Let c= (E9=1 02) then for a small <> O, the approximation error between the true NG-MM kernel matrix K Kugam and its RFF approximation Kngam bounded follows:
3
P [||
K Kngsm
26) SN)xp
2NC (6 K Kugsm/ ll +NCCVQ-
where ||2 is the matrix spectral norm.
Proof. The proof can be found in App. C.6.
The feature map xx(x) not only provides theoretical guarantees for the approximation but also eliminates the need to generate differential samples from the mixture bivariate Gaussian distribution by directly introducing differentiability into the optimization objective w.r.t. the mixture weights ag-
Furthermore, the two-step reparameterization trick uses the correlations between W1 and w2, for efficient sampling. Consequently, the aforementioned optimization problem is solvable with standard algorithms like Adam [15]. We summarize the pseudocode in Algorithm which illustrates the implementation of our proposed method, named Next-Gen Multi- View Latent Variable Model (NG- MVLVM).
7

---

Algorithm 1: NG-MVLVM: Next-Gen Multi-View Latent Variable Model
Input: Dataset Y; Maximum iterations T. Initialize: Iteration count f t= 0, model hyperparameters 0, and variational parameters &.
3 while & < 'T' or Not Converged do
Sample X from x(X) = II==1-(xx [4,, Sn) using the reparameterization trick.
For each view, sample w from p (W"))= 11 I1 L/2 8y (Wi, w²) via the two-step
reparameterization trick.
For each view, construct & ngsam X) using the sampled X and W
Evaluate data reconstruction term of ELBO through Eq. (13).
Evaluate regularized term of ELBO analytically.
Maximize ELBO and update 0, & using Adam [38].
10
Increment + 1.
Output: 0. E.
4
Experiments
We demonstrate the superior performance our model across multiple cross-domain multi-view datasets, including synthetic data ($ 4.1), image-text data ($ 4.2), and wireless communication data ($ 4.3). Additional experimental details are provided in App. D, including benchmark implementations (App. D.2) and hyperparameter selection (App. D.3). We further evaluate the representation learning capability on single-view real-world datasets (App. D.4.1), 1), and demonstrate robustness in multi-view settings (App. D.4) and data reconstruction tasks (App. D.5), through supplementary simulations.
4.1 Synthetic Data
We firstly demonstrate the impact of kernel expressiveness on manifold learning and highlight the expressive power of the proposed NG-SM kernel. To this end, synthesized two datasets using a two-view MV-GPLVM with the S-shape X, based on different kernel configurations: (1) both views using the stationary RBF kernel, and (2) view using the RBF kernel and the other using the non-stationary Gibbs kernel [26] (see further details in App. D.1.I). For benchmark methods, we use the MV-DGPLVM [14] and the multi-view extension of the SOTA ARFLVM [21], namely, MV-ARFLVM.
The manifold learning and kernel learning results across different methods for the two datasets presented in Figure 2. The results for MV-ARFLVM indicate that the model fails to capture the non-stationary features of one view, then significant distortions arise in the unified latent variables. In turn this degrades the model's ability to learn stationary features from other views. In contrast, both the latent variables and kernel matrices learned by our method are closer to the ground truth compared to the benchmark methods, especially in non-stationary kernel setting. Thus, these results demonstrate the importance of kernel flexibility for the MV-GPLVM.
For the MV-DGPLVM, we select the best latent representations from all GP layers and plot the corresponding kernel Gram matrix. However, the kernel learning result may be less meaningful, as the flexibility of model stems from using deep GPs rather than the kernel choice. The deep architecture of MV-DGPLVM exhibits notable capability to capture non-stationary components, yielding a relatively "high-quality" latent representations. However, due to the large number of parameters and the complex hierarchical structure [43], MV-DGPLVM often suffers from overfitting and computational inefficiency, still resulting in inferior performance compared to our method.
4.2 Multi-View Image and Text Data
We further demonstrate our model's ability learn unified latent representations various multi- view image and text datasets, while maintaining high computational efficiency. Specifically, fol- lowing the setting of Wu and Goodman [11], construct multi-view settings by pairing each single-view dmtgset-images (MNIST, YALE, CIFAR), text (NEWSGROUPS), and structured data
'Code is publicly available https: / tttbgtttttuuttteeggeeeeeeeeggett http://www..cm//yyyayaa//////a//

---

S SXX S
OURS
Ground
MV-INGPLM
Truth
We FTime (seconds)
1200-
MV-OGPLVM MNKE
S
1000
OURS
800
Sapnsion
600
MV ARFUMM
400
200
MY.
DGPS/M
100
200
300
400
500
600
760
Sample Size (N)
Figure 2: Comparison of learned latent variables and kernel matrices with the ground truth. Each dashed box shows latent variables (left) and kernel matrices for the two views (middle and right). Left: Both views use RBF Right: View uses RBF, view 2 uses Gibbs.
Figure 3: The wall-time (in seconds) for training on MNIST across different benehmark methods with vary- ing dataset size N. Solid lines indicate the average over five runs; shaded regions represent the standard deviation.
Table 2:
Classification accuracy, expressed percentage (%), ), is evaluated by using KNNN and SVM clas- sifiers with five-fold cross- cross- validation.. Mean and standard deviation the accuracy is computed over five experiments and the top-performing result for each metric is in bold.
DATASET
BRIDGESS
CIEAR
MNISTT
NEWGROUPS
YALE
METRIC
KNN
SVM
KNN
SVM
KNN
SVM
KNN
SVM
KNN
SVM
OURS
1.47
1.76
3602
0.96
K2.72
0,00
838T
1.44
MYARRFLVM
86.11
35.21
1.58
10000
78366
MV-DGPLVM
11.66
10.54
24.31
8:07
2683
27.04
60.14
2.31
3602
53.97
34.58
MYAF
23:36 + 283
71.25
25.3
31.74
2:23
55.18
13581
4.18
74.91
66.13 2.34
MMVAR
7301 +
72.04
1.4
99.07
05-11
1988
38.86
2.70
41.35
MYNCIPIVM
80.72 + 71.58
29.14 2007
1.73
$
199.55
35.25
30.37 + 1.36
23.01
(BRIDGES)" its corresponding label as an additional view. In addition to SOTA MV-GPLVM variants used in $ 4.1, we also compare MV-GPIVM with the Gibbs kernel (MV-NGPLV M) and the SOTA MV-VAE variants: MVAE [11] I] and MMVAE [12]. After inferring the unified latent variable X, we perform five-fold cross -ross-validation using two types of classifiers: K-nearest neighbor (KNN) and support vector machine (SVM). As summarized in Table 2, we report the mean and standard deviation of classification accuracy of various classifiers across multiple datasets. Additionally, Figure illustrates the model fitting wall-time on the MNIST dataset with respect to dataset size.
From those results, we can see that our method demonstrates superior performance in estimating the latent represntations while maintaining computational scalability. For the MV-VAE variants, the inferior performance can be attributed to: (1) optimizing the huge number of neural network parameters leads model instability, and (2) the posterior collapse issue in the VAE results in an uninformative latent space. Both factors partly due to overfitting, problem that can be naturally addressed by the MV-GPLVM variants.
Consequently, the performance of the MV-GPLVM variants is generally superior to that of the MV- VAE, though this often comes with increased computational cost an issue our method effectively mitigates. Specifically, the performance improvement of our method compared to MV-ARFLVM and MV-NGPLVM is due our proposed NG-SM kernel. MV-DGPLVM not only exhibits instability but also incurs extremely high computational costs, which can be attributed to its large number of parameters.
4.3 Wireless Communication Data
For further evaluations, we test the model in channel compression task. Specifically, we generate wireless communication channel datasets using a high-fidelity channel simulator QUAsi Determin- istic Radlo channel GenerAtor (QUADRIGA)* The environment considered consists of a base station (BS) with 32 antennas serving 10 single-antene user equipments (UEs), each moving at a speed of 30 km/h. We sample 1, 000 complex-valuedt channel vectors for each UE at intervals of 2.5
7 "See detailed dataset descriptions in App. D.1.2.
'https//qqudrriaa channel-modeee

---

SWW
0.014
KNN
0.100
0.006
40
2
Accuracy (%)
OURB
MV-ARFLVM
MV-DGPLVM
MVAE
MMVAE
Figure 4: (Left) Classification accuracy expressed as percentage (%). is evaluated using KNN, Logistic Regression (I.R) and SVM classifiers with five-fold cross-validation. Mcan and standard deviation are computed over five experiments. (Right) The average mean squared error (MSF) of the reconstructed channel data compared to the original channel data across different latent variable dimensions five experiments.
ms. The real and imaginary parts of each complex channel vector are treated as two distinct views, with the identifier of the UE as the label, resulting two-view dataset" with N 10, 000 and Mg = 32 for e = 1,
To reduce communication overhead, the UE typically transmits compressed channel vector the BS instead of the full vector. This compressed vector must retain sufficient information to accurately reconstruct the ground-truth channel vectors. We use the unified latent variables as the compressed channel vector and evaluate both their classification performance and reconstruction capability. The mean and standard error of the classification accuracy, along with the mean square between the reconstructed and ground-truth channel vectors, are shown in Figure 4. The results indicate that the KNN accuracy achieved by our method consistently surpasses that of competing methods, while the MSE is consistently lower. These findings highlight the superior performance of our model channel compression tasks compared other benchmark approaches.
(MSE)
Conclusion
This paper introduces NG-MVLVM, novel model designed to address two critical challenges in multi-view representation learning with MV-GPLVMs: limited kernel expressiveness and compu- tational inefficiency. Specifically, first establish duality between spectral density and kernel function, yielding a versatile kernel capable of modeling the non-stationarity in multi-view datasets. We then show that with the proposed RFF approximation and efficient sampling method, the infer- ence model and latent representations can be effectively performed within variational inference framework. Experimental validations demonstrate that model, NG-MVLVM. outperforms state-of-the-re methods such as MV-VAE and MV-GPLVM in providing informative unified latent representations across diverse cross-domain multi-view datasets.
Acknowledgments
The contribution of Michael Zhang was supported by the HKU Seed Fund for PI Research Basic Research #2402101367.
Pablo M. Olmos was supported by the Comunidad de Madrid IND2024/TIC-34728, IDEA-CM project (TEC-2024/COM-99)
the ELLIS Unit Madrid (European Laboratory for Learning and Intelligent Systems), the 2024 Leonardo Grant for Scientific Research and Cultural Creation from the BBVA Foundation, and by projects MICIU/AE1/10. 3039/501 10001 1033/FEDER and UE (PID2024- 157856NB-100 CARTESIAN, P120001-123318200-8--00 EPICENTER).
See App. D.1.3 for more details regarding the wireless communication scenario settings.
10

---

References
[1] Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learning. IEEE Transactions on Knowledge and Data Engineering, E (111)) 2018.
[2] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International Conference on Machine Learning, 2015.
[3] Changqing Zhang, Huazhu Fu, Qinghua Hu, Xiaochun Cao, Yuan Xie, Dacheng Tao, and Dong Xu. Generalized latent multi-view subspace clustering, IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(1):86 990669999222 2018.
[4] Qi Wei, Haoliang Sun, Xiankai Lu, and Yilong Yin. Self-filterig; A noise-aware sample selection for label noise with confidence penalization. In European Conference on Computer Vision, 2022.
[5] Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention Siamese networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[6] Tanveer Hussain, Khan Muhammad, Weiping Ding, Jaime Lloret, Sung Wook Baik, and Victor Hugo C. de Albuquerque. A comprehensive survey of multi-view video summarization. Pattern Recognition, 109:107567, 2021.
[7] Ye Yuan, Guangxu Xun, Kebin Jia, and Aidong Zhang. A multi-view deep learning framework for EEG seizure detection. IEEE Journal of Biomedical and Health Informaties, 23(1):83-94 2018.
[8] Harold Hotelling. Relations between two sets of variates. In Breakthroughs Statistics: Methodology and Distribution, pages 162-190. Springer, 1992.
[9] Francis R. Bach and Michael Jordan. Kernel independent component analysis. Journal of Machine Learning Research, 3(Jul):1-48, 2002.
[10] David R. Hardoon, Sandor Szedmak, and John Shawe-Taylor Canonical correlation analysis:
An overview with application to learning methods. Neural Computation, 16( 16(12):2639-2664 2004.
[11] Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. In Advances in Neural Information Processing Systems, 2018.
[12] Yuxin Mao, Jing Zhang, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Multimodal variational auto-encoder based audio-visum segmentation. In IEEE/CVF International Conference on Computer Vision, pages 954 954-965,727 2023.
[13] Jinxing Li, Bob Zhang, and David Zhang. Shared autoencoder Gaussian process latent variable model for visual classification. IEEE Transactions on Neural Networks and Learning Systems, 29(9):4272-42) 4286, 2017.
[14] Shiliang Sun, Wenbo Dong, and Qiuyang Liu. Multi-view representation learning with deep Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12):4453 4468, 2020.
[15] Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. International Conference Learning Representations, 2013.
[16] Yuge Shi, Brooks Paige, Philip Torr, al. Variational mixture-of-experts autoencoders for multi-modal deep generative models. In Advances in Newral Information Processing Systems, 2019.
[17] Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. Multi-VAE: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In Proceedings of the IEEF/CVE international conference on computer vision, pages 9234-9243, 2021.

---

[18] Yixin Wang, David M. Blei, and John P. Cunningham. Posterior collapse and latent variable non-identifiaiiityy. Advances in Neural Information Processing Systems, 2021.
[19] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from continuous space. In SIGNLL Conference on Compulational Natural Language Learning, 2016.
[20] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in Neural Information Processing Systems, 2016.
[21] Ying Li, Zhidi Lin, Feng Yin, and Michael Minyi Zhang. Preventing model collapse in Gaussian process latent variable models. In International Conference on Machine Learning, 2024.
[22] Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-view learning overview: Recent progress and new challenges. Information Fusion, 38:43-54, 2017.
[23] Vidhi Lalchand, Aditya Ravuri, and Neil D. Lawrence. Generalised GPLVM with stochastic variational inference. In International Conference on Artificial Intelligence and Statistics, 2022.
[24] Michael Minyi Zhang, Gregory W. Gundersen, and Barbara E. Engelhardt. Bayesian non-linear latent variable modeling via random Fourier features. arXiv preprint arXiv:2336..055 083552 2023.
[25] Neil D. Lawrence. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Learning Research, 6(1 2005.
[26] Christopher K.I. Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
[27] Stefanos Eleftheriadis, Ognjen Rudovic, and Maja Pantic. Shared Gaussian process latent variable model for multi-view facial expression recognition. In International Symposium on Visual Computing, pages 527-538. Springer, 2013.
[28] Andreas C. Damianou and Neil D. Lawrence. Deep Gaussian processes. In International Conference on Artificial Intelligence and Statistics, 2013.
[29] Sami Remes, Markus Heinonen, and Samuel Kaski, Non-stationary spectral kernels. In Advances in Neural Information Processing Systems, 2017.
[30] Salomon Bochner al. Lectures on Fourier integrals, volume 42. Princeton University Press, 1959.
[31] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 2007.
[32] Max A. Woodbury. Inverting modified matrices. Department of Statistics, Princeton University, 1950.
[33] Kai Chen, Twan van Laarhoven, and Elena Marchiori. Compressing spectral kernels in Gaussian process: Enhanced generalization and interpretabilityy Pattern Recognition, page 110642, 2024.
[34] Kai Chen, Twan van Laarhoven, and Elena Marchiori. Gaussian processes with skewed Laplace spectral mixture kernels for long-ten term forecasting. Machine Learning. 110:2213-205 2238, 2021.
[35] Akiva M. Yaglom. Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results. Springer, 1987.
[36] Kostantinos N. Plataniotis and Dimitris Hatzinakos. Gaussian mixtures and their applications to signal processing. Advanced Signal Processing Handbook, pages 89-124, 2000.
[37] David M Blei, Alp Kucukelbir, and Jon D McAuliffe, Variational inference: A review for statisticians. Journal of the American Statistical Association, 1 12((118:8859 877, 2017.
[38] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference for Learning Representations, 2014.
12

---

[39] Shakir Mohamed, Mihaela Rosca, Michael Figumov, and Andriy Mnih. Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
[40] Alex Graves. Stochastic backpropagation through mixture density distributions. arXiv preprint arXiv:1607.05690, 2016.
[41] Andres Potapczynski, Gabriel Loaiza-Ganem, Ganem, and John P. Cunningham. Invertible Gaussian reparameteriization: Revisiting the Humbel-softmast Advances in Neural Information Processing Systems,
2020.
[42] Yohan Jung, Kyungwoo Song, and Jinkyoo Park. Efficient approximate inference for stationary kernel on frequency domain. In International Conference on Machine Learning, 2022.
[43] Matthew M. Dunlop, Mark A. Girolami, Andrew M. Stuart, and Aretha L. Teckentrup. How deep are deep Gaussian processes? Journal of Muchine Learning Research, 19(54):1-46, 2018.
[44] Andrew Wilson and Ryan Adams, Gaussian process kernels for pattern discovery and extrapo- lation. In International Conference on Machine Learning, 2013.
[45] Lves-Laurent Kom Samo and Stephen Roberts. Generalized spectral kernels. arXiv preprint arXiv:1506.022338 2015.
[46] Jean-Francois Ton, Seth Flaxman, Dino Sejdinovic, and Samir Bhatt. Spatial mapping with Gaussian processes and nonstationary Fourier features. Spatial Statistics, 18:59-78, 2018.
[47] Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1-230, 2015.
[48] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3):37-52, 1987.
[49] Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: Experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pages 108-122, 2013.
[50] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan):933-102, 2003.
[51] Mukund Balasubramanian and Eric L Schwartz. The Isomap algorithm and topological stability. Science, 295(5552):7-7, 2002.
[52] Prem Gopalan, Jake M. Hofman, and David Blei. Scalable recommendation with hierarchical Poisson factorization. In Conference on Uncertainty in Artificial Intelligence, 2015.
[53] Michalis Titsias and Neil Lawrence. Bayesian Gaussian process latent variable model. In International Conference on Artificial Intelligence and Statistics, 2010.
[54] He Zhao, Piyush Rai, Lan Du, Wray Buntine, Dinh Phung, and Mingyuan Zhou. Variational autoencoders for sparse and overdispersed discrete data. In International Conference on Artificial Intelligence and Statistics, 2020.
[55] Gökcen Eraslan, Lukas M. Simon, Maria Mircea, Nikola S. Mueller, and Fabian J. Theis.. Single- cell RNA-seq denoising using deep count autoencoder. Nature Communications, 10(1):390, 2019.
[56] Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In IEEE/CVF International Conference on Computer Vision, 2023.
[57] Gregory Gundersen, Michael Zhang, and Barbara Engelhardt. Latent variable modeling with random features. In International Conference on Artificial Intelligence and Statisties, pages 1333-1344 1341. PMLR, 2021.
13

---

[58] Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
[59] Andreas C. Damianou, Carl Henrik Ek, Michalis K. Titsias, and Neil Lawrence. Manifold relevance determination. In International Conference on Machine Learning, pages 145-152, 2012.
[60] Andreas C. Damianou, Neil D. Lawrence, and Carl Henrik Ek. Multi-view learning as a nonparametric nonlinear inter-battery factor analysis. Journal Machine Learning Research, 22(86):1-51, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state our main contributions in the abstract and provide point-by- point claboration of these contributions in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in App. E. Computational efficiency and scalability are also analyzed in the experimental section ($ 4.2) and in App. D.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: All theoretical results in the paper are presented with elearly stated assumptions. Formal proofs are provided in App. B and C, and are properly referenced in the main text. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide full code in the supplemental material and list all hyperparameter settings of experiments in App. D.3, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the code in the supplemental material and clearly indicate the sources of all datasets and benchmark implementations in App. D.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
onPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cctpublic//eeeccccccccccccccecct habbiccggiidss//// /Coo//eoeeemeeeee cestsubmmssiooon swiiisssisiogoooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant experimental details-include data processing (App. D.1), hyperparameter settings (App. D.3), and resources of benchmark methods (App. D.2) are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification; We report mean and standard deviation over five runs for each experiment, and compare and analyze the performance and variability across different methods in $ 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details the compute resources are provided in App. D.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / htt///eurripcc.o httpssccccppbcccccceeeecec ittpppiiiiitiiiiiiiiieiiieeeee
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics and confirm that our research complies fully with its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As this work focuses theoretical and methodological advances in kernel lcarning and multi-view representation learning, without direct application or deployment, did not include discussion societal impacts.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use widely adopted public datasets and baselines with no known misuse risks, and do not release any high-risk models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and baseline implementations are properly cited, and detailed usage informations and sources are provided App. D.I and App. D.2.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will include our model and code in the supplemental material at submission time, along with clear documentation in App. D.3 and App. D.2 support reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used for proofreading and grammar checking, not for core research development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: pettps:/neeteee //nerrurpp.... for what should or should not be described.
21

---

APPENDICES
A Notation Table
23
Next-Gen SM kernel and Universal Bochner's Theorem
23
B.1 Bivariate SM kernel
23
B.2 Proof of Theorem
25
B.3 Derivation of Next-Gen SM kemne
26
Auto-differentiable Next-Gen SM Kernel using RFF Approximation
28
C.I Random Fourier feature for Next-Gen SM Kernel
28
C.2 ELBO Derivation and Evaluation
29
C,3 Treating W variationally
30
C,4 Proof of Proposition
31
C.5 Proof of Theorem 3
32
C.6 Proof of Theorem
D Experimental Details
37
D.I Dataset Description
37
D.1.1 Synthetic Data
37
D.1.2 Real-World Data
37
D.1.3 Channel Data
38
D.2
Benchmark Methods
38
D.3
Hyperparameter Settings
40
D.4
Additional Experiments
41
D.4.1
Single-View Data
D.4.2
Multi-View Synthetic Data
D.4.3
Multi View MNIST
41
D.4.4
Experiments More Challenging Multi- Multi-View Datasets
42
D.4.5
Visualization of Learned Representation
D.5 Data Reconstruction..
.ew.sstttiio....................
43
D.5.1 Image Data Reconstruction
43
D.5.2
Missing Data Imputation
43
Limitation
43
22

---

A Notation Table
We summarize key symbols and their corresponding dimensions used throughout the paper to improve clarity.
Table 3: Summary of Notations
Symbol
Description
Dimension
N
Number of observations
scalar
M
Dimensionality of observation Number of views
scalar
V
scalar
Dimensionality of latent space
scalar
U
Number of inducing points
scalar
Number GP layers (for DGPLVM) Dimensionality of random features Latent representation
scalar
L
scalar
x
Y
Observation view Observation vector in view v, dimension
x M,
y.m.
N x 1
(w)
(X)
Latent function for view v, dimension m
N xx1
K
Kernel matrix for view v Spectral points of view v
Nx N
w(
L/2 2D
I-th sampled spectral point for view (first component)
D
d
Noise variance for view V Mixture weight of q-th Gaussian component
scalar
Ogq
scalar
P
1g11 H22
Mean vectors of q-th bivariate Gaussian
0 11
q2
Variance vectors of q-th bivariate Gaussian
Pg
Correlation coefficient in q-th component
scalar
Set of kernel hyperparameters Set of variational parameters
B Next-Gen SM kernel and Universal Bochner's Theorem
B.1 Bivariate SM kernel
The development of the spectral mixture (SM) kernel is based on the fundamental Bochner's theorem [44], which suggests that any stationary kernel and its spectral density are Fourier duals. However, the stationarity assumption limits the kernel's learning capacity, especially when dealing with multi-view datasets, where some views may exhibit non-stationary characteristics. model non-stationarity, the Bivariate SM (BSM) kernel was introduced in [29, 45], based on the following generalized Bochner's 's theorem:
Theorem 5 (Generalized Bochner's Theorem). A complex-vatie valued bounded continuous kernel X2) on R² is the covariance function of a mean square continuous complex-valued random process RRP and only if
/ exppi (w[xi ww (2xx2)) (dw1,
(X1, x2)
(17)
where m is the Lebesgue-Stieltjes measure associated with some positive semi-definite (PSD) function S (W1, W2).
According to Theorem 5, approximate the function S(w1, W2), and implement the inverse Fourier transform shown Eq. (17) to obtain the associated kernel function. The bivariate spectral mixture (BSM) kernel adopts this concept and approximates S(w1, Wa) as bivariate Gaussian mixture as follows:
SW1, (w)
Σ Sg (We1, Wq2).
(18)
23

---

where Sy (Wo1. W42) represents the q-th component of bivariate Gaussian distribution, formally defined as [29]:
We1
M4g
diag(o Pq diag(o diag((o.))
Pq diag(o,)) diag(a (p))))
N
Wyz
/
diag( (9)))
My E (Mul
A E
(19)
Here [M My2 2 represents:
[111 (y2), (pat Mot1) (1u22 1422), (1ug2 pgl);
(20)
(- M4l. M22)) (- My1 -Hy1, (-1422 - 1422; (-M42---411) (b41)-
The terms 0i1 and 0 E R represent the variances of the q-th bivariate Gaussian component, while Py denotes the correlation between Wg1 and Wa2; the vectors M41 and Ma2 E IR 2 specify the means of the g-th Gaussian component. The mixture weight is denoted by ogg and Q is the total number of mixture components. Taking the inverse Fourier transform, can obtain the following kernel function:
(-1 1 - Σn E (x((x) ( Vy (x2)
h xx),
Σ ag exp qm
- (-2
(21)
where
coss T + cos [1
/2111 sin (f111)) + sin (4g2
1 (x)x)
ER
and & = (X1, X2) E 2D22
is a stacked vector.
The BSM kernel overcomes the stationarity limitation in the SM kernel. However, it still has two major issues.
First, the assumption of identical variance of w, and W2 limits the approximation flexibility of the mixture of Gaussian, which in turn diminishes the generalization capacity of the kernel.
Second, the RFF kernel approximation technique cannot be directly applied, as the closed- form expression of the feature map is hard to derive (see explaination below).
The first limitation arises from the requirement that the spectral density must be PSD. To ensure the symmetry of the BSM kernel function (i.e., k(x1, x2) = k(x2, x1)), the BSM kernel assumes that T1 = ,2, Vg.
In addition, we highlight the challenges of deriving a closed-form feature map for RFF when directly applying Theorem 5. According to Theorem 5, the kernel can be approximated via MC as follows:
1: x2) = exp(i(wT w/ x2]x)) (3)))((dw,,w:))
1
En (exp i( (WT x1 w. x2)))
L/2
(22)
exp [(wl (1) X1 -w/ x2)
~
2L
1=1
L/2
Σ 1=1
(cos( (w( (1)T (1(1 x1) cos w (OT x2) + sin (w]] (T)
x1 sin w T x3
x3))
2L
Since w and W2 are distinct, it is hard to derive a closed-form RFF approximation for k(xi, X2). More specifically, it is challenging explicitly define dww, (-), (ww)))) the feature maps of the kernel function, such that
k(x1, X2) Rs Φw, (x1)T @ww (x2) = Pw, (X2) (ww (x1)
(23)
Thus, the inversion of the BSM kernel matrix retains high computational complexity, rendering unsuitable for mieti-view data.
24

---

Remark 1. To enhance the kernel capacity, this paper proposes the Universal Bochner's Theorem (Theorem 1) and the NG-MM kernel. The main contribution of Theorem is that it relaxes the PSD assumption of the spectral density, thus the induced NG-SM kernel can miligate the constraint of identical spectral variance.
To derive a closed-form feature map for any kernel, one potential approach is to decompose the spectral density 5 (W1,, W2) into some density functions pl Wi; W2) [45, 46], such as:
Remark 2.
1 1(w1, w2) = - (D((W, W2) + PW2,, w1) + p(w: (8 - wi) + p(w2)6(W, - w2)), (24)
where p(w1) and p(W2) are the marginal distributions of p(W1. W2). and 6(x) denotes the Dirac delta funnction. Subsequently, MC integration can be applied to S(W1, W2) to derive the closed-form feature map, see details in Appendix C.I.
B.2 Proof of Theorem
(=) ) Suppose there exists a continuous kernel xx) on R... By the Theorem 5, this kernel can be represented as:
k(x1, x2) I exp (i(w] x1 -w x2)))) m(dwi,dwa).
where m is the Lebesgue-Stiellee measure associated with some PSD function S(w1, W2) of bounded variation.
To ensure that the kernel function is exchangeable and PSD, we design the spectral density S(Wi, W2) as follows:
5(W1, (p(w1, w2) p(w2, w1) p( (0W)))(W2 wi) p ( w2)), (25)
where ((x) represents the Dirac delta function, and is certain density function that can be decomposed from S as Eq, (25). Additionally, p(w1) and p(w2) are the marginal distributions of p(wI, w2).
The resulting kernel & (x1,x2)
1
(f exp (i (wi x1 w x2) (W1, w2) dwi dww+ / exp (i (w] -w, x2)) p (w2, wi) dw2 dwi
4
exp (iw] T (x1- [ (x)) (WI) 6(w2 wi) dwi+ / eep (iw (x1- p (W2) 5(wi w2) dw2
+
1
(fexp exp (i (w1 xi - wx x2)) u(dwi, dw2) / eex (i (i(w]x) -.w w]xx)) u(dw1,
4
+ f exp (i (((w[Tx-w]xxo) wwxxx)) u(dw1)
/ exp (i (i(wix wx)x2) u(dws)).
(26)
where u is the Lebesgue-Stiele Stiellees measure associated with the density function p(W1, W2).
Note that we can disregard the functions in the last two terms of the expression, as the integrands in these terms depend solely on or w2. Consequently, we can only integrate over the single variable, while setting the other variable to be equal to the one being integrated.
Finally, we can express the kernel:
1 (exp (i (w] XI w2]x)) +exp (i (i(wIxx xi wi x2))
ks (xi, xx), x2)
w]xx))
(27)
+ exp (iw] (x1 - x2)) +exp
- (x (x)) dww))
which is the expression shown Theorem
(=) Given the function:
1 t (x1, X2) = 1 /
/ (exp(iw]] [iw]:i/ixiiwxxe (exx))+>xxpxxxx +exp(iw] x1 -iw] x2)
(28)
+ exp(iw] (ici:w/[xx--- (x)))))) (aoxx))))xxppp (ixp (x1 -x2)) )u(dw),
25

---

we have the condition that:
S(w1, W2) = (p(W1, (1, W2) + p(W2, w1) + p(w (WW)) wi) + p(w2)8(WW - -22),
(29)
a PSD function. We aim to demonstrate that Xy) is a valid kernel function. Specifically, need to show that is both symmetric and PSD.
First, it is straightforward to show that x2) kxx,, x1), confirming that & is symmetric.
The next step establish that l is PSD. Consider that k(x1, x2)
(f xI x2)]xx))p((()) w2) ww -// w2) dwidws
-
+ / -xxp(((3>x w2) dwwdww - w2) dwidwe
1 4
(ʃ /exp(i(w] [exp(((w]x)-ww<x))) (w: w2) dw1
dwi /dw2
exp(i(w] xx:xxxxw., w. x2) w2, w1) dwidwe + xx -we) dw dww
/ exp(i(w] x1 -w
wx)x)) x2)) (p(w), w2) (ww (w)
dwi dw2
/ exp(i(w)
md)
(30)
where m is the Lebesgue-Stieltjes measure associated with the PSD density function S (W1, W2).
Thus, by Theorem 5, k(W1, W2) is PSD.
B.3 Derivation of Next-Gen SM kernel
The BSM kernel based on Theorem is constrained by the requirement identical variances for w and W2, and is incompatible with the RFF approximation technique. In this section, we derive the NG-SM kernel based on Theorem 1, which effectively resolves these limitations.
The spectral density of the NG-SM kernel is designed as:
Pogsm (W1, W2) Σ Sq (Wi, W2).
(31)
with each
W1
pgg1 M4p
Σ Σ
((w))
(Wi
(ggl
Σ
Σ
8g (Wi, W2)
2 .V
2 N
14p
De
Σn
(32)
We simplify the notation by omitting the index from the sub-matrices ΣS = diag(o )
Σg ? and E. Pg diag( diag(a,2), where of 2 RD and Pg represents the correlation between w and w2. These terms together define the covariance matrix for the q-th bivariate Gaussian component. Additionally, the vectors phy and plg RD serve the mean of the q-th bivariate Gaussian component.
Relying on Theorem 1, we derive the NG-SM kernel kingan (x1, X2)
26

---

1
Pogam (Wi; w2) exxp(iwTxx iww x2) + exp(iw] x1 -iw x2)
4
+ eexp(iw] xi - -iw]xe exp x1 iw. x2) dw1dw2
1 Q
0ig / 2 (f) w2) + ww. w2)) exp(iwixx - ing x2)
4 8=1
+exp(iws x1 i x2) + exp(iw] x1 iw xx) exp((ww - -iw, x2) dwpdww,
where
((w)
w2 ((w)
W1
M4y1 Map
Σ Dg
& (wI, w2) =N
(33)
Σ
We focus solely on the real part of the kernel function. Since the real part of the integrand is cosine function, and both D, and the cosine function are even functions, we can therefore simplify the expression follows.
(wi, w2) exp(iww]x - -iw x2) + exp(iw, x1 -iwi x2)
Ogg
g
+ eexp(iw]xx - xs) + exp(iw]xx iw? iw]x2) dw,dww
1 2,(W1, W2) [xp(iw] x1 .w: x2) dw,
Og
1=1
Term (1)
(, wi) exp(ii iw x2 )
Term (2)
+ L (W1) exp(iw] x1 iw] x2) dw1 + (y(W2) exp(iw x1 i x22 dw
Term (3)
Term
(34)
where by (W1) and By (w2) are marginal distributions of (y(1,, W2). Next, we will derive the closed forms for each term. First, Term (1) is given by:
Term (1) = / N(w (w)
Mgg Eg) ew &dw
/ Texp
(w -myg ( " (w mg) w x) dw
(2 E
1
1 j Din un dw (35)
w "n + w T (x + En (n)
exp
(2T)²²
exp
(X
Eg (17)
En (x+ + Eg (17) exp - E 'g
= =exp
T x + u x
where we defined x (ixi, ixx) and w = (W1, W2). In addition, ,a (1411 M922) and
Σ Σ
E, =
Σ
Σg
27

---

The first term of the kernel mixture is then given by:
Term (2) = exp
x T +
M4g x
2
cx
[x] Σ1x1 2x T Dxx2 + x2 Σ2x²)
exp (i (11 - 11)2x2))
(36)
(x] Σ1x1 2x Σx + x² Σ2x²) cos (HTTi M.zx2).
exp
2
By swapping X1 and X2 in Term (1), the closed form of Term (2) can be easily obtained as below:
1
Term (2) = exp -
[x
Ex + +x +xx cos (M11 - M222x)
(37)
Term (3) of the kernel is then given by:
Term (3) f NN (W1
M9l, E exp(iw]] (x1 - x2)
1
/ exp
1 (W1 (11))a)) ET (wi 1uy1) + iw] (x1 x2) dwi
(2
2
1
1
1
exp
w Σ 'w1 WT (i(x) x2) + Ei m31)
2 Hul E Mai dwi
(2
2
exp
(i(x1- -2) + ET pby1) Ei (i(x) x22)+ + (a)+Eii'n ET(M41) exp
- Σj Meg
((x) x2) EE x2) + M ixx xxx
cxp
1
2 (xi x2)
TE (x) - x2)) exp (iteg) (xx, xx2))
exp
1
cxp
2 (x1 x2) En (x1 x2)) cos (H (x1 x2))
(38)
The 4'th term of the kernel can be derived in manner similar the 3'rd term.
Term (4) = exp (-2
2 (x1
T - xxi) cos (HTg(( - -x2))
(39)
Thus, NG-SM kernel takes the form:
?
(x] Σ1x1 - -2x, Exx2 + x3 Σxxe) ) cos (HTii² - 1T3222
4(X1, X2)
Σ
0g
exp
2
-=1
+ exp
1
(x) Σ1x² 2x Exxx + xx E2x1) cos (HtT 1
T
2
1
+ exp
(x1 X2) Σ
_ (x1 - xx)) cos (u (x1 x22)
2
+ exp
(x) x2)
Ex(x; -xx) cos (1 xx))
2
(40)
C
Auto-differentiable Next-Gen SM Kernel using RFF Approximation
C.1 Random Fourier feature for Next-Gen SM Kernel
Our proposed Theorem 1 establishes the following duality: is (x1,X2)
E, (cxp(i(w xx wx +cxp (i(w,) x1 -w] x²)) -w]xx))
4
+ exx((((w] I X1 -wi x²)) +exp( x - wwxx)))
28

---

By estimating this expectation with the MC estimator using spectral points {w("; 10 w/ 2 sampled from p(W1, W2), we can drive
L/2
k (x1, X2)
~
Σ
exp
((w]] (TT
X1 -w/ w
T!) x2)] + exp i(w2 (1)
X1 w (1)
x2)
2L
1=1
+ exp
i(w1
(1)
-wI w((xx)) x2)) + exp (((W?" (1)
X1 - w/
(D) (1) xxe1
L/2
cos
w| x1
cos
ww
(1) x2 + cos
w| (1) x1
cos (ill !!)T X2)
2L
1=1
(41)
+ cos
W2
10T
X1)
000
W
(0)
x2)
+
cos
22
(() X1
cOS
(1)
+ sin
w (1)
sin
w (1)
x2
sin
W
sin
13
+ sin
W2
(1)T
x1
sin
W (1)
x2
+
sin
W2 (l))
x1
sin
W2
(1)
X2
x(x1) TT Tx((2)
where
(1:L/2)T
(1:L/2)
1
W
+008
W2
(x) = 2L
ERL.
(42)
sin (1:L/2)T
(1:1/2)T
+sin
W2
Here the superscript : L/2 indicates that the cosine plus cosine or sine plus sine function is repeated L.22 times, with each element corresponding to the one entry of {w(") w [1=1 the spectral density as Prgsan (W1, W2) (Eq. (31)), the estimator of the kernel kingsm (X1, X2) can be formulated as:
5./2 If we specify
(X1,
Kingsm (X1, (2) ze ((x))* ((x2),
(43)
where random features ((x1) and o(x2) are constructed using spectral points sampled from Pngsm| (W1, W2).
C.2 ELBO Derivation and Evaluation
The term (a) of ELBO is handled numerically with MC estimation as below:
M
(a)
E (x,W)
ply mm [X, W!]
(44a)
m=1
M. Nyx"" m. 00, ugam + ININ)
log.Nyy
(44b)
where denotes the number of MC samples drawn from (X, W). Additionally, R(((t) ( (() is the NG-SM kernel gram matrix approximation, where D" NxL.
The term (b) of ELBO can be evaluated analytically due to the Gaussian nature of the distributions.
More specific, have
N
(b) =
(45a)
n=1
tr((S.) + pb, pn, log [S.l D]. -
(45b)
2
where represents the dimensionality of x,, and S,, commonly assumed to diagonal matrix. Consequently, the ELBO can be expressed follows:
29

---

P(Y,X: X; W) αX, W)
ELBO B)((.W)
log p(X) 1=1 P W (( Y X, )
W)
q(x) P (W")
Bith [ogp( Y X, W
(L(((X)lp(X)
(a): reconstruction
(b): rogularization
N
log (y
K 0,
agem + o I''N
tr(Sw)) + 14, pnn - -log |S,l D)
M
N
log 2
log Kula + ooIN
T 2 2m:
ugoan + o?lx)
8:m
2
2
m=1
1=1
(r(S.)
pTt pn
log [S,1 1n11-D]
When >> I,, both the determinant and the inverse of K (e)(()) by the following two lemma [26].
Kggsn + o I can be computed efficiently
Lemma 1. Suppose A is an invertible n-by-n matrix and U, V are n-by-m matrices. Then the following determinant equality holds.
AUVV Im + V A U |A].
Lemma 2 (Woodbury matrix identity). Suppose A is an invertible n-by-n matrix and U, V are n-by-m matrices. Then
(A+UVT)" - A UU + V VU--VV.
We can reduce the computational complexity for evaluating the ELBO from the original O(N3) to ON as below:
1
K vlt) ngai
+
+oolN| = 2N
??
I +
b )())
(46)
x
( ngam
+ ooINN
IN
$ (i) [I + ( "0000
$ "()
(47)
o
C.3 Treating W variationally
Alternatively, we define the variational distributions
(X, W) (XX),
where the variational distribution q( (W:n) m)) is also bivariate Gaussian mixture that parameterized
by 77. By combining these variational distributions with the joint distribution defined in Eq. (11), we derive the following ELBO:
P(Y, X, W)
ELBO= = (x(X)
((: (X)
log p(X) IT" w== P W Ongam (P (X Wo)
W)(
XX)
D W
Σ Eyt (Y X.W (L(g(X))llg IP(
KL(g W: (W; W 0
(48)
[a]:
(b): regularization of X
(c): regulatization
where we redefine the prior distribution p( ()) as p( W; 0ggsm) to maintain notational consistency. When maximizing the ELBO, we obtain o( n: = p( W; Ongam), as the optimization variable Ongan only affects term e. Consequently, term (c) becomes zero, and Eq. (48) aligns with the optimization objective outlined in the main text.
30

---

C.4 Proof of Proposition 1
Proof. The proposed two-step reparameterization trick leverages sequential simulation, where waz is first sampled from p(Wa1), followed by W22 drawn from the conditional distribution p(W Wg1).
1) Sample from p(We1): Given that Wa1 follows normal distribution (H14 diag(d)))) (1)), we can directly use the reparameterization trick to sample from it, i.c.
(1)
q1 pa1 Jgl 61;
where 61 ~ I..
2) Sample from (W Given w 91 we use the given correlation parameter Py and a new
standard normal variable to generate w (1) 92
M4p + Pg ot o (w 1411 + / 11 P2 o 62.
W
Now, we need to proof that the generated w (1) and with follow the bivariate Gaussian distribution ql q2
Sy (i, w2). To this ends, we compute the mean and variance of w!tl (!) and the covariance between
gl and (1) below.
Mean of
E w 22) E M42 + 02 (w 1bg1 ) + /1 11- P2 o 62
Since B[E2] 0 and E[w (1)
Mig. we have:
E w q2
= uup + P, (B(w mai + V 1ō o BBE22 = 1ug.
(1) 42
Variance of
Var w (2)) Var (prraa o (w (411) V 11 02 2
Since w (1) ql 141 and 62 are independent, and Var(E2) I, we have:
(1) Var( w 92 2 (a ,1 Var( (w (1) + (1 - -1)) = 16 17)0 =0²²
Covariance between w !1) q1 and w (1) 92.
Cov(w (1) gl w Cov plt + ogt o,,, 72 (w Ma1) + v 03 66
(49)
Since Mg11 and p42 are constants, the covariance only depends on og c, and Og 0 C1 /1 P2age o C2. Thus we can reformulate Eq. (49) as
C0 Pq 2061 11 2202)) = (ov po
(o 62)
Given that and E2 are independent, COv(EI, 62) = O, and Cov(EI, = I, this covariance is equal to:
o1P 02 o I 041 -2 0 o 09..1 022-
Therefore, the generated will and w !1)
42 follow the bivariate Gaussian distribution (W1, W2).
The two-step reparameterization trick simplifies the sampling process of the bivariate Gaussian distribution. Specifically, the traditional sampling method [39] requires Cholesky decomposition of the 2D x 2D covariance matrix, resulting in a computational complexity of O(DD²) method achieves computational complexity of O(D) for sampling from pl (WW).. while sampling from yl (Wya/ (/1))) also maintains a complexity of (D).. This results in a total computational cost of O(D).
contrast, our
31

---

C.5 Proof of Theorem 3
Proof. With the RFF feature map defined in Eq. (15), we can express the inner product of the feature maps follows:
L/2
(x1) T ((())((x)))))))))))) = Σ
0g
Σ
A
(50)
2L
9=1
1=1
where,
1 = cos
w
91
(1)
Xi
cos
(1)T w ql
X2)
+ cos
w [1]
ql
xi
0o8
(TT
m X2)
(!)
cos
w 42
X1
COS
w y1
X2
cos
w 42 X1
cos
W
X2)
(51)
gl sin
(
sin
9l (i)
X2
+ sin
w (1)
91
X1
sin
92
(1)T
X2
sin
(1)T g2 X1
sin
w (i)T
X2 + sin
w (1)
X1
sin
w
(!) 92
X2
where {w1, E/? are independently and identically distributed (i.i.d)) spectral points drawn from the density function 5, (W1, w2) using the two step reparamcterization trick. Taking the expectation
with respect to Prgsm (W1, w2) = I 1./2
8q (W1, w2), we obtain:
Q
L/2
E Progin (w: we))
10 (x1) T 4 (xx)
= E Eppegan (Wi, w2)
Σ
A0
2L
9=1
1=1
Q
Lf2 Σ
Σ
Og B. "(wa 1:6/22, wao////,waa//// 1:k/2)
10 4
(linearity of expectation)
2L
q=1
(52)
Q
04g Eg, (w,(w,,ws) w2) [Ag)
(i.i.d. of w()))
1=1
Q
aqp (ss w2) [exp(i(wix] w2xx)) + exp(i(w2x] wix!])
+ exp(i - -wix))) + exp(i(w2x] -
(Euler's identity)
aghi (x1, X2)
kagam (0ggm(X1,300 X2).
(NG-SM kernel definition)
(53)
Thus, we conclude that yp (x1) (x2) provides an unbiased estimator for the NG-SM kernel.
C.6 Proof of Theorem
Proof. We primarily rely on the Matrix Bernstein inequality [47] establish Theorem with the proof outline depicted in Figure 5.
Lemma MMatix Bernstein Inequality). Consider a finite sequence (E,) of independent, random, Hermitian matrices with dimension N. Assume that
E E 0 and |E, S H for each index i,
32

---

Step 1: Decompese the difference Kngan Kagses into a sum
independent, random, Hermitian matriocs Rn!")
Step 4: Prove Thearem 4 using the Lemma 3.
Step 2: Derive the upper bound of the spectral norm |E.
Step 3: Compute the upper bound of the variance of the sum of of E
Figure 5: Flowchart for proving Theorem 4
where || ||2 denotes the matrix spectral norm. Introduce the random matrix E E, E,, and let
(E) be the matrix variance statistic of the sum:
||
Σ E E
Then we have
E [||a] S 2e( log N
I. log N.
(54)
Furthermore, for all 0.
-22/2 P B 2 c S N. exp v(E) + HE/3
(55)
Proof. The proof of Lemma can be found in Theorem 6.6.1 in [47].
Step 1: With the constructed NG-SM kernel matrix approximation,
K Kugam
2 nugam (X) 4 ngam (X)_ where the random feature matrix 4 2ggam (X) = (9 (x.)....,. (xN)] E
R. XQL we have the following approximation error matrix:
E agsm
K ngsmm
(56)
We are going to show that E can be factorized as
Q L/2
E
Σ>
M
E 4
(57)
where E.( is a sequence of independent, random, Hermitian matrices with dimension N.
Sample qli wyz from Sy (W1, W2). and show that
())
(1)
1./2
KK nggmmm h.
Og
A (h,g),
2T.
9=1
whereA
(1)
(h,9)
cos
w 7l
(T)
X/
cos
w
(!)
cos
(1) w Xx, cos
(1jT w/2 xz)
(T) (1) xx. cos w/yl Xgg cos (wilT
w (
((()) (
sin (1! x
sin (!]) Xg
sin (ill) x sin w x,,
cos
Xn.
cos
92
xg)
T
w (!)T
w (TT
Xn sin w (1; X
+ sin
w
42
X/
sin
xg
+ sin
(58)
33

---

Thus, we have K ngem 3, we have that
= -E
L1/2 2g Ag Based on this factorization and Eq. (52) in Proposition 21=1 2I
L/2
K nggam
2L E
Therefore, the approximation error matrix can be factorized as E
EL'
E
where
qp=l
E
0g A
(59)
2T.
sequence of independent, random, Hermitian matrices with dimension N that satisfy the condition of E E(E!!1 0.
Step 2:
We can bound the
E
ll
by following.
| 1|2
0g
E
A
(60a)
2T
4
9
$
0g
11)
+
E
A
(triangle inequality)
(60b)
2L
2
$ 0g
44)
+ E
(,00
(Jensen's 's inequality)
(60c)
2L
C
(8N + 8N)
(60d)
2L
C'
8CN
16N
(60e)
2L
where C== / 2 og and
96
cos
w ( X1)
COS
(w(() (!)
9k XNN
ERR
96
(0)
sin
w qki
(L)
X1
.....
sin
w (1)
gk
X.
E R
A
(1)
(I)T
(1)
(()
(1) (
(1) (()T
(1) ((T
(1) (TT
(1) (() (1)
(!) (()
9l ql
90l 8 9l
C42
42
+ 8 92 8
gl 92
+ 8q1 92
+
c2 Cgl
8 gl
(61)
We are interested in bounding the spectral norm of A where
1
(1) (I)T
(TT
( (
(1) (I)T
! (
(0) (()
1 (I)
(1)
11) (()
1 (1T
92 22
22 92 c 92
g2 c1
92
gg and function. Thus, we have: ||c ll S vN, || |/s lla < VN.
(2) ( qh are N-dimensional vectors, where each element is given by a cosine or sine
For any vector a, b E RN, the spectral norm of ab T satisfies: ||ab 112 = |a||2 ||b|22_ Note
that each term in A!") is of the form abT. Therefore, since ||a||2 < /N and |bbl VN as shown earlier, we have: ||ab ||2 s N, for all terms in A!
Since A( is the sum of 8 such terms, we apply the subadditivity of the spectral norm:
| 2 S Σ
ba,b 12 SNN
The inequality (C.6) serves as valid upper bound. However, not necessarily tight, since spectral norm of sum of rank-one matrices is typically less than the sum of their individual norms unless the corresponding vectors are perfectly aligned. At last, this upper bound leads directly the inequality in Eq. (60d).
34

---

Step 3:
We first have the following bound:
4L2
( 2 E [[ao))]]
(E [441 2
E
(62a)
4
<
E
(62b)
E
Σ
Cokss Gqkz Coks Syks
( (1) (
+ gig gkk
(8) (
2kk 8
gki
1U) (1)
qke
8
gkzz
( (1) (1)
9kk Saks
+
+s
gki
8
gkz
TT
8
ahk & (
(1)
gki
k
(62c)
E
Σ
(cyg2 Cgk3
(
c
gk. ggk
UT
+ +
(1)
gkz
gkg
)
gk.
(T)
gki
+
gkz
(TT (1)
Cgka
(1) ( (s
Cikks
+ Cgk
$ (
gka
k
(62d)
Σ
E
(
Cgkag/ gik
( (1)
Cyke
(
+ +(s (gggk
yky
gkk
UT
+ +(*
Cykg gka
(T)
(s
gki Syke
(1) (
+
ggg uyka
(1)
(U)))
11)
k
(62e)
NE
Ggki Cake
(1)
(
+
gki
8 (
gkk
+I (s
TTT Cikio (s
Coke + Cyks 9aks 5)
(()
(1)
(62f)
k
ANE [100
Σ k
E
(sgka
gks
(1)
(s
gli Sika
11 TT
(0) + Caki 8gka 222T))
(62g)
where Σx - Ek =
and the notation A < B denotes that B - A is PSD matrix. To ensure the validity of the inequality in Eq. (62b), it is necessary to show that (E(4!"))
a positive semi-definite (PSD) matrix.
Since A is a real symmetric matrix, its expectation [[A4)) is also real symmetric. Therefore, we can apply the following result:
Lemma 4 (Square of a Real Symmetric Matrix is PSD). Let M E Ru be real symmetric matrix. Then its square M² ;= MM is positive semi-definite, i.e., M č 0.
Proof. Since M is symmetric, it admits an eigendecomposition of the form M UA where U E Raxn is an orthogonal matrix and A E Rx is a real diagonal matrix of eigenvalues. Then,
M = MM = UAU UUUT = UAUT,
where A is the diagonal matrix of squared eigenvalues of M. Since the square of any real number is non-negative, all eigenvalues of 2 are non-negative. Therefore, M² is PSD.
Applying Lemma 4 to (EAA))) we conclude that (B(A))))* is PSD. This confirms that the inequality in Eq. (62b) holds.
The inequality in Eq. (62f) holds because
NE
(1)
(O)T + syhi 9yka
(1) (1)T] - E (CohT (1) (1) (B)
Caks )
+ (s (1) (
(sgka syks gkg s UT
gki gkk
skk
= E
gka skg Jons Saka
(OT (1) (1) (()
+ (Cyyk Gaks ) gaka
T (1)
(63)
N Σ
(1) (T) ()) TT
(cos WTT w (T) xx) N
due to Cg Caka sky 9yk
is a PSD matrix.
35

---

E Ele B[ ) as
Then we are able bound the variance,
L/2
B
4=1 1=1
Q L/2
2
ANE
+
E
gk2
(T)
Skk
(s
ggg Syka
(T
+ 99
gkk
(T)
1=1
1m1
k
2C
L
Σ ag
N
40
Σ
E
(s
gk2
(
Coks
(s
Soke
1TT
+ copes
11) (
skks
2C'
L
N |||Kngmml 112
16
Qig
E
(5
Cokig
())
(
(1) (U)
Sake
+c 8
Sake
(1)T
(triangle inequality)
2C
L
N |KK mgaam 112
16
0gy E
S gkz
( (1)
gkcs
(s
(T
gkk
+
+ggls 222 |1(]
(1)
(Jensen's inequality)
2C'
L
N W||Knne 11
+
16
4
2
N
2
a
E
|||
(8
Saka
(T)
b111)
(Iee (1)
cxkg) s
N
2
k1=
ky=1
2CN
(1K
ll2
+
1 N
16C/Q
16
2
(64)
where the last incquality is because that
(8 qkg Sgks qbg
(i)
(1)
qke
sup ||0|| 11
0 T ghg
UT Sake
+
(CT cby oahs )0
S SN, (65)
and Ed ag S CVQ by the Cauchy- inequality.
Step 4: We can now apply the derived upper bounds, given by Eqs. (60) and (64), to and v(E)
in Lemma 3,
-3e
K Kagam Kngsm
S N exp
Nexp
(66)
2NC [6| K ngsm llz + +NCC
which completes the proof of Theorem
36

---

D Experimental Details
All experiments were conducted cloud server equipped with 2 NVIDIA Tesla V100 GPUs (16GB memory each), a 10-core Intel Xcon Platinum 81xx series CPU, 64GB RAM, and 200GB storage.
D.1 Dataset Description
This section presents detailed overview of the datasets used in our experiments, covering the generation process of synthetic data, the description of real-world data, and the applied preprocessing steps.
D.1.1 Synthetic Data
The datasets are generated using 2-view MV-GPLVM with an S-shaped latent variable, employing two different kernel configurations: (1) both views use the RBF kernel, and (2) one view uses the RBF kernel while the other uses the Gibbs kernel. Detailed descriptions of the kernels are provided below.
RBF Kernel:
The kernel function expressed as:
X2
(x1,, x2) = to exp
20?
where l, = 1 denotes the outputscale, and & = represents the lengthscale.
Gibbs Kernel: As non-stationary kernel, the kernel function is formulated
26x, lx2
[x x2|1
k(x1, X2) =
exp
x. + %2
X2
In this context, &x dynamic length scales derived from the positions of the input point x, specifically
defined
Ex = cxp (-0.5 ||x().
D.1.2 Real-World Data
Table 4: Description of real-world datasets. SAMPLES (N)
DATASET
DIMEENSIONS (M)
BRIDGES
214
4
CIFAR
60,000
x
R-CIFAR
2,000
MNIST
70,000
x
R-MNIST
1,000
NEWSGROUPS
2752
19
YALE
165
1850
CHANNEL
1,000
64
BRENDAN
2,000
x
The detailed preprocessing steps and descriptions of the real-world datasets are provided below. For convenience, key information about the datasets is summarized in Table 4.
This dataset is collection of data documenting the daily bicycle counts crossing four East River bridges New York City (Brooklyn, Williamsburg, Manhattan, and Queensboro). We classify the data into weekdays and weekends, treating these binary labels. This classification aims to examine the variations in bicycle counts on weekdays versus weekends, exploring whether significant differences exist in the traffic patterns between the two.
1) BRIDGES:
This dataset comprises 60,000 color images with a resolution of 32x32 pixels, categorized into 10 distinct classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship. and truck. These images were further resized from 32 pixels to 20 20 pixels and converted to grayscale for training.
2) CIFAR:
10 ntttps:/data, cityofnewyork. us/ http.//aaatttttttt////oooo...oa Come Counts - for- East- BivetRiier--a River-Bridgen Bridges/ qua4-p9ug
37

---

3) R-CIFAR:
We sample this dataset from the CIFAR dataset, specifically selecting five categories: airplane, automobile, bird, cat, and deer. For each category, 400 images are sampled, and each 32x32 pixel image is converted into 20x20 pixel image.
4) MNIST: It is a classic handwritten digit recognition dataset. It consists of 70,000 grayscale images with resolution of 28 28 pixels, divided into 60,000 training samples and 10,000 testing samples. Each image represents a handwritten digit ranging from 0 to 9.
5) R-MNIST: We select 1,000 randomly handwritten digit images from the classic MNIST dataset.
6) NEWSGROUPS:
It is dataset for text classification, containing articles from multiple newsgroups" We restrict the vocabulary words that appear within document frequency range of 10% to 90%. For our analysis, we specifically select text from three classes: comp.sys.mace www.mmachaaawrre... sci.med, and alt.atheism.
The Yale Faces Dataset consists of face images from 15 different individuals, captured under various lighting conditions, facial expressions, and viewing angles.
7) YALE:
8) BRENDAN: The dataset contains 2,000 photos of Brendan's face
9) MMMMN--SSSVI -VHN: Two paired views: MNIST digits (784 dimensions) and SVHN digits (3072 dimensions), each belonging to one of 10 classes.
Extracted from IMDb, where each movie is represented by a keyword vector (1878 dimensions) and an actor vector (1398 dimensions)* The dataset consists of 617 movies categorized into 17 genre classes.
10) MOVIES:
11) CORA: A citation network dataset in which each document is described by two views: a bag-of words content vector (1433 dimensions) and a citation structure vector (2708 dimensions). It contains 2708 documents spanning 7 research topic classes
D.1.3 Channel Data
To evaluate model performance under realistic wireless environments, we simulate channel data using the QUAsi Deterministic Radlo channel GenerAtor (QuaDRiGa)'s 16 widely-used geometry- based stochastic channel simulator. The parameter settings used in our experiments are listed in Table 5.
The simulation scenario emulates a typical urban communication setting, where the user equipment (UE) moves at a constant speed 30 km/h. QuaDRiGa accounts for both the speed and the direction of movement, which significantly influence the channel behavior. Specifically, the relative motion between the UE and the base station (BS) introduces Doppler shifts and varying path loss. As the UE moves through the environment, the geometry between the transmitter, receiver, and scatterers changes over time, leading to dynamic variations in multipath components such as delays, angles, and Doppler frequencies.
is worth noting that no explicit additive noise (e.g.. Gaussian noise) introduced in the simulation. Instead, the randomness in the channel arises naturally from QuaDRiGa's 's stochastic modeling of multipath fading, spatial non-stationarity, and time dynamics. These controlled yet realistic variations ensure consistency across different simulation runs while providing sufficient complexity to evaluate the robustness of the proposed model.
D.2 Benchmark Methods
We provide description of the benchmark methods as follows:
1) PCA: A classical linear dimensionality reduction method that projects data orthogonal components maximizing variance.
2) LDA: A supervised linear method that finds projections maximizing class separability.
" "nttp://quone. http:://gooe.ccmmm"iiiiiiee/ ommm""jaoan/2/Neearruuppp
Pnttp://vision. ucd.. see.dddec/cccatett//aal--------- database .mat
Wnttps://lig
Btteppp://ig--membressttt membres.immg..t//rriii
html
/httpp://lig Dtttppa://iig-mebbreesss wwmbmes..mag..ee.eee..
haabbrrrs.iag..ffrriimmaa//tttt html
16 http:////gadrrrgaaammmmmomaooo// de
38

---

Table 5: Parameters settings of QUADRIGA,
PARAMETER DESCRIPTION
VALUES
Overall Setup
samples
# user equipment (UE)
10
réceive antennas
Moving speed (km/h)
30
Proportion indoor
Time sampling interval (scconds)
Se-3
Total duration sampling (scconds)
Channel type
3GPP 3D___DMMMMMM
Channel Configuration
frequency (lLz)
1.8499
Use random initial phase
False
geametric polarization
Use spherical waves
False
progress
Base Station (BS) Anfenna Configuration
# vertical elements per antenna
# berizontal elements per
# rows in the antenna array
colurna
antenna array
Electrical tilt angle (degrees)
transmit antennas
32
Antemna Configuration
antenma amay
subcarriers
Subcarrier spacing (Hz)
lo6
# loops (simulations)
Minimum distance from BS (meters)
Maximum UE distance from BS (meters)
300
Layout Configuration
# base stations
Base station position (x, y. (meters)
0, 30)
UE movement length (meters)
41.667
3) ISOMAP: nonlinear manifold learning method that preserves geodesic distances between data points.
hierarchical poisson factorization model for probabilistic matrix factorization. commonly used in recommendation systems.
4) HPF:
5) BGPLVM: bayesian formulation of GPLVM that places a prior over latent variables and infers their posterior using variational inference.
6) GPLVM-SVI: scalable GPLVM variant using stochastic variational inference to handle large datasets.
7) VAE: A deep generative model that learns latent representations via amortized variational inference.
8) NBVAE: A probabilistic model for sparse, overdispersed count data, combining Gaus- sian process prior with a negative binomial likelihood for flexible, non-linear modeling.
9) DCA:
The deep count autoencoder is designed denoise single-cell RNA sequencing data by accounting for count distribution, overdispersion, and sparsity, using a zero-inflated negative binomial noise model.
10) CVQ-VAE: The clustering vector quantised variational aautoencoder is an enhanced VQ- VAE model designed prevent codebook collapse by utilizing an online clustered codebook.
11) RFLVM: GPLVM extension using random feature approximation for scalable inference.
12) DGPLVM: A deep extension of GPLVM that stacks multiple GP layers to learn hierar- chical nonlinear mappings from latent space, typically trained via variational inference.
13) ARFLVM: An advised GPLVM using kernel learning and latent regularization for flexible representation learning.
Multi-modal VAE using a product-offexperts (PoE) to infer shared latent representations from multiple modalities.
14) MVAE:
39

---

Table 6: Descriptions of henchmark methods.
METHOD
REFERENCE
IMPLEMENTATION CODE
[48]
Using scikit-less library [49]
LDA
[50]
Using the acik15-learn library [49]
ISOMAP
[51]
Using sciktt-leern library [49]
HPF
[52]
https: h /ttp//gbtb/bb.///////
BGPLVM
[53]
httpa: //gtt//ttttt ctn/coo////aaee
GPLVM-SVI
[23
https: //gtthub con/ www/v//0////////// 21300//eeeeeeeiiiiee 0erp////eeeeral11eerrrr
VAE
[15]
httpa://g1tte www.t.thb........ con/ http:///torrrrraaaaaaaa//////aaa
NBVAE
[54]
httpa: :/ //tgt//gbbbb.////// wtt.///aaaaeeeeeecttee///////e/e
DCA
[55]
httpe: stape://e //github. con/ wom/theeeeab//ee
CVQ-VAE
[56]
httpe: hts:://gitubb con/ Lyadonzbeeng/CQQ-WWW
RFLVM
124, 57]
Ostpe://gitan com/ gwgunsereeen//flln
DGPLVM
[58]
bttpa:/ www.b...g......oo/oogo/o/tt....t
Coo//CL--a- SXL
ARFLVM
121
hitpe: //ggthub.....t/////gg com/ http////dd...//a/iidddd///
MVAE
[11]
httpa:/ githhb www.///w///////occcee//eee/ee//e http:///s//mmiiinnaaaa-eeeeeeeee
MMVAE
112 161
httpe: pttps://gittae tht.b//ggttttttg///gg com/ wwtp//ppaaaaPPaaa//////--------.
Table 7: Default Hyperparameter Settings.
PARAMETER DESCRIPTION
VALLES
NG-SM Kernel Sctup
Mixture densities (Q)
2
Dim. of random feature (L/2)
50
Dim. of latent space (D)
2
Oplimizer Setup (Adam)
Leurning rale
0,01
Beta
(0.9, 0.99)
(0.9,0.99)
# Iterations
10000
21.23
27.44
65.26
66.20
26.18
0,8821
0.60
0.4866
0.1650
0. 0.9637
0,9894
22.93
34.98
75.67
122.09
192.46
0,90300
0.9999
0.9174
0.9947
0.
5.9152
2-
24.50
58.74
112.49
205.00
595.12
50
0,990
0,9749
0.9945
0.9935
0.1931
0.9494
0.9112
0,9043
6.9153
0.9941
0.9952
0.5439
0.915
0.98
2-
34.00
$3.03
282.36
382.19
442.31
647.36
0.9736
6.9969
0.9365
0.9931
0.5962
0.9106
R-
48.47
120.96
277.32
451.15
$22.76
020.11
200
100
0,4877
6.9445
0.4941
0.9932
0.9463
0.9896
6.975
70.02
231.12
435.04
570.43
555.22
1401.74
10
Figure Heatmap wall-time time (seconds) latent manifold learning.
Figure 6: Heatmap of R² in latent manifold learning.
15) MMVAE: Mixture-of-experts (MoE) based VAE that balances shared and private repre- sentations for multi-modal learning.
Table 6 provides the references and implementation details for each benchmark method, aiming to enhance reproducibility.
D.3 Hyperparameter Settings
Figure 1l depicts the latent manifold learning outcomes of NG-RFLVM for varying values of Q and L on synthetic single-view view data. Figure 6 shows heatmap of R scores, quantifying the similarity between the learned and ground truth latent variables, with values closer indicating better aligmment. Figure 7 displays heatmap of wall-time across varying values of Q and L. These results demonstrate that larger values of Q and L typically improve model performance, albeit at the cost of higher computational complexity. To balance computational efficiency with latent representation quality, we select Q = 2 and 50. The default hyperparameter settings are summarized in Table 7.
40

---

Table 8: Classification accuracy (%) is evaluated by fitting KNN classifier (A: 1) with five-fold cross- validation. Mean and standard deviation computed over five experiments, and the top performance in
bold.
DATASET
PCA
LDA
ISOMAP
HPF
BGPLVM
GPLVM-SVI
DGPLYM
R-CIFAR
26.78
0.22
22.78
0.66
27.23
0.66
20.80
0.64
27.13
1.46
25.12
1.24
28.16
1.36
R. R-MNIST
36.56
1.23
23.38
2.69
+4,45
2.14
31.19 4.02
56.75 -. 3.37
34.41 5.48
74.82
2.24
NEWSCROUPS
19.24
0.52
39.14 1.89
39.79 1.04
33.44 2: 1.91
38.53 =: 1.05
37.84
1.88
57.63
2.48
YALE
1.96
3.65
9.177-1.59
DATASET
DCA
CVQ-VAE
BRIDGES
75.15 -: 1.63
75.85 3.88
70.21 3,63
68.86 1: 1,38
$4.61 3.95
84,64 1,54
$5.30 1.27
R-MNIST
64.37
28.15
17.19
12.87
0.53
5.53
74.59 1.52
0.59
NEWSGROUPS
18.52 0.39
39 1.00
39.97
1.41
35.60 1.96
41.35 0.95
41.82 0.75
40 40.10 1.48
YALE
61.16 = 2.04
45.00 4,68
28.49
5.40
33.89 = 0.28
65.37
6.79
76.56 1.02
76.60
1.96
D.4
Additional Experiments
To explore more general scenarios, we first conduct single-vies experiments validate our model's representation capability (App. D.4.1), and then extend the evaluation to more complex multi-view settings, including synthetic data (App. D.I.I), multi-view MNIST with a large number of views (App. D.4.3) and more challenging multi-view datasets (App. D.4.4). Finally, we provide qualitative visualization of the learned representations (App. D.4.5).
D.4.1 Single- View Data
In this section, we first examine the following single-view data types: images (R-MNIST, YALE, R-CIFAR), text NNEWSGROUPH computational costs of RFLVM, dataset sizes for CIFAR and MNIST are reduced and denoted with the prefix 'R-' (see details in App. D.1.2).
and structured data (BRIDGES). To accommodate the high
The results in Table present the mean and standard deviation of classification accuracy from five-fold -ross-validated K-nearest neighbor (KNN) classification, performed on the learned latent variables for each dataset and method.
Based on this experiment, our method capable of capturing informative latent representations across various datasets due its superior modeling and learning capacities. The inferior performance of classic approaches is primarily due to their insufficient learning capacity. The GPLVM variants perform slightly inferior our approach, primarily due the assumption of kernel stationarity.
While DGPLVM addresses the modeling limitation by incorporating deep structures, its performance is hindered by more complex model optimization processes [43]. Similarly, VAE-based methods inherently suffer from posterior collapse, making them prone to generating uninformative latent representations, partly due overfitting [19, 20].
D.4.2 Multi-View Synthetic Data
As the number of views increases, Figures 8a and 8b present the unified latent representations generated by our method and the corresponding Rt2 scores, respectively. These results show that with more views, the unified latent representations learned by our model progressively align more closely with the ground truth.
D.4.3 Multi-View MNIST
We generated a four-view dataset derived from the MNIST dataset using rotation operation, as illustrated on the left-hand side Figure alongside reconstruction results obtained with our method. The right-hand side of Figure displays the KNN accuracy evaluated using the latent variables learned by various methods on MV-MNIST. These results highlight that our approach not only achieves superior performance in downstream classification tasks but also effectively reconstructs data form each view through the shared latent space.
Comparison benchmarks include various classic dimensionality reduction approaches [9. 48. 50. 52). GPLVM MM-based approaches [21, 23, 24, 53J. and VAE- based models [15, 46-56). See further details in Table 6.
41

---

0.95
0.950
0.925
2.
Boundfrnd
wrbske
0.300
(a) Latent manifold learning results with multiple views.
(b) R² against the number views. Figure 8: Comparison of latent manifold learning and R2 against the number of views.
Figure (Left) MV-MNIST reconstruction (Right) Classification accuracy (%) evaluated using KNN classifier with five-fold cross-validation. Mean and standard deviation of the accuracy is computed over five experiments.
View 1
MWAE
Shared Lantnt
MV-0GPLVM
Space
View
KNN
Table 9: Classification accuracy (%) evaluated by KNN (A: = 1) and NN classifiers with five-fold cross- validation. Mean and standard deviation are computed over five experiments; the performance is in bold.
MNIST-SVHN
MOVIES
CORA
MODEL
KNN
NN
KNN
NN
KNN
NN
OURS
93.13 0.84
96.86 0.69
20.64 0.57
43.44
2.35
46.13 0.51
57.81
0.58
MV-ARFLVM
91.16 1.16
94.62
1.56
19.44
1.00
41.71
0.49
43.61
0.58
56.16
0.74
MV-DGPLVM
85.67 1 0.72
72.72
1.18
15.15
0.55
38.70
0.21
25.17 2.87
30.65
1.95
MVAE
80.23 1.02
88.21
0.94
14.32 0.68
40.78 1.26
38.85
0.23
34.29
1.63
D.4.4 Experiments on More Challenging Multi- View Datasets
In this section, we evaluate our method on more challenging multi-view datasets, including MNIST-SVHN, MOVIES, and CORA
We evaluate the models through five-fold cross-validation with two types of classifiers: KNN and neural network (NN). The averaged classification accuracy and its standard deviation across different datasets are reported in Table 9. The results show that our method learns high-quality latent representations. MVAE variants perform worse due large parameterization and posterior collapse, which lead to uninformative latents and overfitting. MV-GPLVM models alleviate these issues and generally surpass MVAE, though often higher cost. Our approach mitigates this overhead while maintaining accuracy, with gains over MV-NGPLVM stemming from the NG-SM kemel. In contrast, MV-DGPLVM unstable and computationally prohibitive due its heavy parameterization.
D.4.5 Visualization of Learned Representation
We visualize the clustering structure of the learned representation to enhance interpretability. Since the latent space is set to 2D by default, we directly visualize it for BRIDGES, MNIST, and NEWS- GROUPS in multi-view setting in Figure 10, and observe that clearer cluster structure corresponds to higher classification accuracy.
See App. D.1.2 for detailed dataset descriptions.
42

---

Figure 10: Visualization of learned representations of BRIDGES, MNIST, and NEWSGROUPS datasets in multi-view setting.
MNIST
BRIDGES
NEWSGROUPS
Latert Dinz
Lakent Ole
Lateed Drve
Latent
Table 10: Quantitative comparison of reconstruction quality on image- -ased datasets, measured by MSE, Each entry reports the mean and standard deviation over five runs. The best performance is highlighted in bold.
MODEL
MNIST
CIFAR
MNIST-SVHI (MNIST view)
MNIST-SVHN (SVHN view)
OURS
0.023 0.0004
0.021
0.0002
0,019 0.0002
0.040 + 0.0003
MV-ARFLVM
0.025 0.0002
0.022 0.0008
0,020 0.0016
0,041 0.0016
MV-DGPLVM
0.064 + 0.0031
0.044 + 0.0009
0.024 = 0.0013
0.057 = 0.0054
MVAR
0.077 + 0.0040
0.031 + 0.0010
0.031 + 0.0021
0.044 + 0.0015
D.5
Data Reconstruction
In this section, we evaluate our model on data reconstruction tasks, including image reconstruction (App. D.5.I) and missing data imputation (App. D.5.2).
D.5.1 Image Data Reconstruction
We evaluate the reconstruction capability of our model on image-based datasets. We first assess single-view reconstruction quality on MÑIST and CIFAR, where each model reconstructs images from their respective latent representations. We then extend the evaluation to the multi-view setting using the MNIST-SVIH dataset: from the learned shared latent variables, we reconstruct both the MNIST and SVHN views. Reconstruction quality is measured using the MSE. Each model is trained and evaluated over five independent runs, and we report the averaged MSE along with the corresponding standard deviation in Table 10.
Our method consistently achieves the lowest MSE across all datasets, indicating superior reconstruc- tion quality compared with existing baselines. These results demonstrate the effectiveness of approacc capturing complex image structures while maintaining robustness across both single- and multi-view settings.
D.5.2 Missing Data Imputation
In this section, we evaluate our model's capability for missing data imputation using the single-view datasets MNIST and BRENDAN. We randomly set various proportions of the observed data Y to zero (denoted as Y obb),, ranging from 0%% to 60%. Using the incomplete datasets Y Yots- our model estimates the underlying latent variable X, and the missing values are imputed as Y miss = E[Y miss X, Yous)- Figures 12 and 13 illustrate the reconstruction tasks on the MNIST and BRENDAN datasets with varying proportions of missing values, showing superior ability of our model to restore missing pixels." 19
E Limitation
IIn the future, we particularly interested in expressing a factorized latent space where each view is paired with an additional private space, alongside shared space to capture unaligned variations across different views [59, 60].
43

---

Figure 11: Latent manifold learning results with different Q and L.
41
68
0:4
60
S
While the proposed NG-MVLVM framework effectively captures informative unified latent rep- resentations, there remains for further enhancement. In particular, the current model does not explicitly account for cross-view dependencies, which may limit its performance in scenarios with structured dependencies across views. As future direction, plan incorporate cross-view interaction terms to further enhance the model's capacity for multi-view representation learning.
44

---

10% missing
30% missing
miss kng
4
7
9
7
4
7
9
7
C
6
C
6 O
6
2 3
9
train data
9
reconstruction
6
9
Figure 12: MNIST reconstruction task.
Ground Truth
0% micsing
10% missing
30% missing
80% micssing
train data
reconstruction
Figure 13: BRENDAN reconstruction task.
45


Reviews:
[{"review_id": "xhJiff2zSx", "reviewer": "Reviewer_ATeZ", "summary": "This paper proposes a novel and well-motivated multi-view representation learning method by addressing two key limitations of the existing MV-GPLVM model: the lack of kernel expressiveness and poor scalability.", "questions": "1)\tWhile this paper claims that the proposed NG-SM kernel is capable of approximating any continuous kernel via Bochner duality and Gaussian mixture spectral density modeling, the theoretical guarantee is only loosely discussed. \n2)\tAlthough the method aims to learn informative shared latent variables across views, the paper does not offer enough interpretability or visualization analysis (e.g., t-SNE, clustering structure, or downstream task results). \n3)\tThe paper mentions that the complexity is reduced to O(NL²) using RFF-based kernel approximation, but it lacks empirical runtime comparisons with other methods.", "limitations": "1)\tWhile this paper claims that the proposed NG-SM kernel is capable of approximating any continuous kernel via Bochner duality and Gaussian mixture spectral density modeling, the theoretical guarantee is only loosely discussed. \n2)\tAlthough the method aims to learn informative shared latent variables across views, the paper does not offer enough interpretability or visualization analysis (e.g., t-SNE, clustering structure, or downstream task results). \n3)\tThe paper mentions that the complexity is reduced to O(NL²) using RFF-based kernel approximation, but it lacks empirical runtime comparisons with other methods.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The authors propose a new Next-Gen Spectral Mixture (NG-SM) kernel, derived through a duality between the kernel and its spectral density modeled via bivariate Gaussian mixtures. To make the model scalable, they further design an efficient random Fourier feature approximation combined with a reparameterization strategy. The resulting model enables efficient variational inference and achieves strong performance across multiple heterogeneous multi-view datasets.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the invaluable feedback and comments. Below are our detailed responses.\n\n>C1: While this paper claims that the proposed NG-SM kernel is capable of approximating any continuous kernel via Bochner duality and Gaussian mixture spectral density modeling, the theoretical guarantee is only loosely discussed.\n\nThank you for raising this point and we agree that a rigorous proof is necessary. Accordingly, we have included a complete proof in the revised paper and have provided a brief sketch below for your reference. \n\n**Step 1:**  \n   \nAs established by our universal Bochner’s theorem, any continuous  kernel $k_{t}(x_1,x_2)$ can be represented as:\n   $$\n   k_{t}(x_1, x_2) = \\frac{1}{4} \\int \\Gamma(x_1, x_2; w_1, w_2) \\, p_{t}(w_1, w_2) \\, dw_1 dw_2,\n   $$\n   where $p_{t}(w_1,w_2)$ is the  spectral density of the target kernel and $\\Gamma$ is a bounded sum of exponential terms.\n\n**Step 2:**   \n\n   It is known that Gaussian mixtures are dense in $L_1(\\mathbb{R}^{2d})$ [1]. Thus, for any $\\varepsilon > 0$, there exists a $Q$-component Gaussian mixture $p_{ngsm}$ such that\n   $$\n   \\| p_{ngsm} - p_{t} \\|_{L_1} < \\varepsilon,\n   $$\n   provided $Q$ is sufficiently large. \n\n**Step 3:**  \n\n   We define the approximate kernel:\n   $$\n   k_{ngsm}(x_1, x_2) = \\frac{1}{4} \\int \\Gamma(x_1, x_2; w_1, w_2) \\, p_{ngsm}(w_1, w_2) \\, dw_1 dw_2.\n   $$\n   The difference between the target and approximated kernels is:\n   $$\n   k_{t}(x_1, x_2) - k_{ngsm}(x_1, x_2) \n   = \\frac{1}{4} \\int \\Gamma(x_1, x_2; w_1, w_2) \\, [p_{t}(w_1, w_2) - p_{ngsm}(w_1, w_2)] \\, dw_1 dw_2.\n   $$\n   Taking the absolute value and applying the triangle inequality:\n   $$\n   |k_{t}(x_1, x_2) - k_{ngsm}(x_1, x_2)|\n   \\le \\frac{1}{4} \\int |\\Gamma(x_1, x_2; w_1, w_2)| \\, |p_{t} - p_{ngsm}| \\, dw_1 dw_2.\n   $$\n   Since each exponential term in $\\Gamma$ has modulus 1 (i.e., $|e^{i\\theta}| = 1$), and $\\Gamma$ is a sum of 4 such terms, we have $|\\Gamma(x_1, x_2; w_1, w_2)| \\le 4$. Thus,\n\n   $$\n   |k_{t}(x_1, x_2) - k_{ngsm}(x_1, x_2)| \\le 1/4 \\cdot 4  \\| p_{t} - p_{ngsm} \\|_{L_1}\n   $$\n\n   This implies that the kernel approximation error is bounded by the $L_1$ distance between $p_t$ and $p_{ngsm}$, uniformly over all input $(x_1, x_2)$.\n\n**Conclusion**  \n\nAs $Q \\to \\infty$, $p_{ngsm} \\to p_{t}$ in $L_1$, implying $k_{ngsm} \\to k_{t}$ uniformly. Therefore, NG-SM can approximate any continuous kernel arbitrarily well. \n\n[1] N. Kostantinos. *Gaussian mixtures and their applications to signal processing*. In *Advanced Signal Processing Handbook: Theory and Implementation for Radar, Sonar, and Medical Imaging Real Time Systems*, 2000.\n\n>C2: Although the method aims to learn informative shared latent variables across views, the paper does not offer enough interpretability or visualization analysis (e.g., t-SNE, clustering structure, or downstream task results).\n\nWe agree that visualization and interpretability analysis is important. Thus, we  already included both qualitative and quantitative evidence in the original submission: Section 4.1 (Figure 1) visualizes the learned 2D latent variables and compares them with the ground-truth S-curve; Sections 4.2 and 4.3 evaluate downstream tasks using KNN, SVM, and other classifiers on multiple datasets. \n\nFollowing your suggestion, we further visualize the clustering structure of the learned representation to enhance interpretability. Since the latent space is 2D, we directly visualize it for MNIST and CIFAR, and observe that the clearer cluster structure corresponds to higher classification accuracy - consistent with the results in Section 4.2.\n\n>C3: The paper mentions that the complexity is reduced to O(NL²) using RFF-based kernel approximation, but it lacks empirical runtime comparisons with other methods.\n\nWe appreciate the reviewer’s suggestion. As shown in Figure 2 (Section 4.2), we have included runtime evaluations on the MNIST dataset, demonstrating the efficiency of our RFF-based kernel approximation. However, since the analysis is embedded within the overall performance discussion and Figure 2 is not explicitly referenced in the runtime-related text, this result may appear unclear. To address this, we will revise the experimental section to explicitly refer to Figure 2 and better highlight the runtime advantage in practice."}, {"review_id": "iOz2suwl71", "reviewer": "Reviewer_7AbS", "summary": "In this work, the authors introduce a new multi-view Gaussian process variable model that incorporates a novel Next-Gen Spectral Mixture (NG-SM) kernel. This approach aims to address the limitations of kernel expressiveness and reduce the high computational complexity associated with the previous bivariate Spectral Mixture (BSM) kernel. The authors conduct experiments using synthetic data, (image, label) pairs, and wireless communication data. Their proposed method demonstrates state-of-the-art performance in terms of classification accuracy, training time, and data reconstruction.", "questions": "- Can the authors utilize neural network-based classifiers for evaluating classification accuracy results? \n- Why did the authors decrease the image size of the original datasets? How does the approach adjust to larger images? \n- How do the latent space sizes of the different models compare? Do they have similar dimensions? \n- For the wireless communication data, what do the ten different classes represent?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The authors introduce a novel duality between the spectral density and the kernel function in the context of the multi-view Gaussian process latent variable model. The new kernel enhances both the expressiveness and computational efficiency compared to previous work. \n\n- The paper is well-written, accompanied by detailed supplementary material and shared code. The authors conduct experiments on a variety of datasets, reporting a range of results that include classification accuracy, training times, and data reconstruction capabilities. \n\n- The proposed approach achieves state-of-the-art performance when compared to prior Gaussian process latent variable models.\n\n**Weaknesses**\n- While the approach advocates for learning a model using multimodal data, the chosen experiments do not include sufficiently challenging multimodal datasets. In the (image, label) experiments, the use of multi-view techniques primarily serves as dataset augmentation since the new views are just rotations of the original images. Additionally, it is unclear to the reviewer whether the model is supposed to reconstruct the class labels for the images. Examples of more challenging multimodal datasets can be found in reference [1].\n\n- **Clarity** The reviewer believes that including diagrams illustrating the dependencies of the random variables, both overall and for each experimental setup, would enhance the understanding of your work. Although the authors provide experimental details in the supplementary materials, it remains unclear to the reviewer what Y represents for each dataset.\n\n- For the image-based datasets, a quantitative analysis of reconstruction quality is missing. How do the different approaches compare concerning data reconstruction?\n\n[1] Shi, Yuge, Brooks Paige, and Philip Torr. \"Variational mixture-of-experts autoencoders for multi-modal deep generative models.\" Advances in neural information processing systems 32 (2019).", "was_revised": true, "final_justification": "After reviewing the authors' response along with the other reviews, they addressed my concerns by providing additional experiments and evaluations, stating they would update the submitted manuscript accordingly.\n\nFor the above reasons the reviewer increases its pre-rebuttal score.", "rebuttal": "We thank the reviewer for the valuable feedback. We have addressed the raised questions and suggestions, with most of the changes made in the experimental section. Below, we summarize the key additions to the experiments:\n\n1. **New Datasets:**  \n   Added experiments on more challenging multimodal datasets, including MNIST–SVHN, MOVIES, and CORA.\n\n2. **New Metrics:**  \n   Used neural network-based classifiers (3-layer MLP) for evaluating classification accuracy.\n\n3. **Image Reconstruction Experiments:**  \n  Reported MSE-based reconstruction results and added cross-view reconstruction tasks.  \n\nBelow, we provide point-by-point responses to each of your comments.\n\n>C1: While the approach advocates for learning a model using multimodal data, the chosen experiments do not include sufficiently challenging multimodal datasets. In the (image, label) experiments, the use of multi-view techniques primarily serves as dataset augmentation since the new views are just rotations of the original images. Additionally, it is unclear to the reviewer whether the model is supposed to reconstruct the class labels for the images. Examples of more challenging multimodal datasets can be found in reference [1].\n\n>C4: Can the authors utilize neural network-based classifiers for evaluating classification accuracy results?\n\nWe thank the reviewer for the constructive feedback. To address the concerns about multimodal challenges and evaluation, we have:\n- Evaluated on more challenging datasets (MNIST-SVHN [1], MOVIES, and CORA), where our method maintains strong performance (see table below); \n- Added neural network-based classifiers (3-layer MLP) as an additional evaluation metric, further validating our approach (see table below);  \n- Included cross-view reconstruction tasks (e.g., MNIST → SVHN) to better assess representation learning. Due to image upload limitations, the visual results cannot be shown here but will be included in the revised version. \n\n\n| Metric | Dataset     | MNIST-SVHN     | MOVIES         | CORA           |\n|--------|-------------|----------------|----------------|----------------|\n| KNN    | **OURS**    | **93.13 ± 0.84** | **20.64 ± 0.57** | **46.13 ± 0.51** |\n| KNN    | MV-ARFLVM   | 91.16 ± 1.16    | 19.44 ± 1.00   | 43.61 ± 0.58   |\n| KNN    | MV-DGPLVM   | 85.67 ± 0.72    | 15.15 ± 0.55   | 25.17 ± 2.87   |\n| KNN    | MVAE        | 80.23 ± 1.02    | 14.32 ± 0.68   | 38.85 ± 0.23   |\n| NN     | **OURS**    | **96.86 ± 0.69** | **43.44 ± 2.35** | **57.81 ± 0.58** |\n| NN     | MV-ARFLVM   | 94.62 ± 1.56    | 41.71 ± 0.49   | 56.16 ± 0.74   |\n| NN     | MV-DGPLVM   | 72.72 ± 1.18    | 38.70 ± 0.21   | 30.65 ± 1.95   |\n| NN     | MVAE        | 88.21 ± 0.94    | 40.78 ± 1.26   | 34.29 ± 1.63   |\n\n**Remark**:  We also provide a quantitative analysis of reconstruction quality on MNIST-SVHN in our response to Comment C3. \n\n**The new datasets used here are:**\n\n   - *MNIST–SVHN:*  Two views: Paired MNIST (784-dim) and SVHN (3072-dim) digits (10 classes).\n\n   - *MOVIES:*  Extracted from IMDb, each movie is represented by a keyword vector (1878-dim) and an actor vector (1398-dim). The dataset contains 617 movies categorized into 17 genre classes.\n\n   - *CORA:*  Each document has two views: a bag-of-words content vector (1433-dim) and a citation structure vector (2708-dim). The dataset includes 2708 documents across 7 research topic classes.\n\n**Clarification**:\n\n   - Section 4.2 evaluates classification on learned latent variables, not label reconstruction. These additions will be incorporated into the revision.\n\n>C2: Clarity. The reviewer believes that including diagrams illustrating the dependencies of the random variables, both overall and for each experimental setup, would enhance the understanding of your work. Although the authors provide experimental details in the supplementary materials, it remains unclear to the reviewer what Y represents for each dataset.\n\nWe appreciate the reviewer’s constructive suggestion. To improve clarity, we will add a **probabilistic graphical model** of NG-MVLVM in the revision to explicitly illustrate the dependencies between random variables.  \n\n**Clarify the role of $Y$:**  Across all experiments, $Y$ represents the observed multi-view data (e.g., images, text, or graphs), while our model infers a unified latent representation from these inputs. \n\n>C3: For the image-based datasets, a quantitative analysis of reconstruction quality is missing. How do the different approaches compare concerning data reconstruction? \n\nWe appreciate the reviewer’s suggestions. We now include a quantitative comparison of reconstruction quality on image-based datasets, using **MSE** as the reconstruction metric. \n\n\n | Model        | MNIST            | CIFAR            | MNIST–SVHN (MNIST view) | MNIST–SVHN (SVHN view) |\n |--------------|------------------|------------------|--------------------------|-------------------------|\n | **OURS**     | **0.023 ± 0.0004**   | **0.021 ± 0.0002**   | **0.019 ± 0.0002**           | **0.040 ± 0.0003**          |\n | MV-ARFLVM    | 0.025 ± 0.0002   | 0.022 ± 0.0008   | 0.020 ± 0.0016           | 0.041 ± 0.0016          |\n | MV-DGPLVM    | 0.064 ± 0.0031   | 0.044 ± 0.0009   | 0.024 ± 0.0013           | 0.057 ± 0.0054          |\n | MVAE         | 0.077 ± 0.0040   | 0.031 ± 0.0010   | 0.031 ± 0.0021           | 0.044 ± 0.0015          |\n\nFrom the table, our method consistently achieves lower MSE values compared to other baselines on all image-based datasets, demonstrating better reconstruction quality. \n\n>C5: Why did the authors decrease the image size of the original datasets? How does the approach adjust to larger images?\n\nWe appreciate the reviewer's question regarding image size adjustments. \n\nTo clarify, the size reduction was applied only in the single-view experiments (Appendix 4.1) when comparing with RFLVM. This was necessary because RFLVM's MCMC sampling becomes computationally prohibitive for full-size images, as noted in our paper. \n\nHowever, in the main experiments (Section 4.2), we used full datasets, and all benchmarks reported in Section 4.2 can handle the full-size data without reduction.\n\n>C6: How do the latent space sizes of the different models compare? Do they have similar dimensions?\n\nAll models are evaluated with the same latent space dimensionality, which is set to 2 by default to ensure a fair comparison.\n\n>C7: For the wireless communication data, what do the ten different classes represent?\n\nThe ten classes correspond to ten different user equipments (UEs), each following a distinct trajectory. Within each class, the samples are the channel matrices observed at different time steps as the UE moves."}, {"review_id": "svH12GS7LA", "reviewer": "Reviewer_oW4K", "summary": "This paper proposes a novel NG-MVLVM model aimed at enhancing kernel expressiveness and scalable inference. The key contributions include a new duality between spectral density and kernel function, and a scalable random Fourier feature approximation combined with a reparameterization trick. Empirical validation showing superior performance over state-of-the-art multi-view learning methods.", "questions": "1. Is the bound of Theorem 4 tight and does it play an important role?\n2. Are there guidences or requirements when choosing the kernel?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed NG-SM kernel addresses limited kernel expressiveness, lack of flexibility, and high computational complexity of previous BSM kernels.\n2. The duality between spectral density and kernel function provides a principled foundation for kernel design.\n3. Experiments are conducted on diverse datasets to demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n1. It's hard to identify the contributions over previous work. The authors are suggested to summarize how each component differs from them.\n2. Despite the huge amount of experiments, he paper does not fully analyze the individual contributions of the NG-SM kernel and the random Fourier feature approximation through ablation studies. The terms \"kernel expressiveness\", \"time-varying correlations\" are abstract and hard to quantify.  \n3. The claimed efficiency is not empirically validated with runtime comparisons against other MVC methods.", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge and appreciate the reviewer’s detailed feedback. Please see our responses addressing each comment below.\n\n>C1: It's hard to identify the contributions over previous work. The authors are suggested to summarize how each component differs from them.\n\nWe have summarized the comparison with previous work in **Table 1 (page 2 of the paper)**. To highlight our contributions more clearly, we extract the key content of Table 1 below:\n\n\n | Model          | Scalable model | Highly expressive kernel  |\n |----------------|----------------|---------------------------|\n | MVAE           | √              | -                         |\n | MM-VAE         | √              | -                         |\n | MV-GPLVM       | ×              | ×                         |\n | MV-GPLVM-SVI   | √              | ×                         |\n | MV-RFLVM       | ×              | ×                         |\n | MV-DGPLVM      | ×              | ×                         |\n | MV-ARFLVM      | √              | ×                         |\n | **NG-MVLVM**   | **√**          | **√**                     |\n\nFrom this table, our contributions can be summarized as:  \n\n1. **Expressiveness:** We propose a **Next-Gen Spectral Mixture (NG-SM) kernel** that models the spectral density as a Gaussian mixture, making it a highly expressive kernel capable of capturing complex data structures. Building on this, we design **NG-MVLVM**, which learns a more informative shared latent representation across views.  \n\n2. **Efficiency:** We develop an **unbiased RFF-based approximation** for the NG-SM kernel that is differentiable with respect to kernel parameters, allowing us to use variational inference while reducing training cost. Combined with a two-step reparameterization trick, our approach further lowers computational complexity and enables efficient, scalable learning for multi-view data.\n\n>C2: Despite the huge amount of experiments, the paper does not fully analyze the individual contributions of the NG-SM kernel and the random Fourier feature approximation through ablation studies. The terms \"kernel expressiveness\", \"time-varying correlations\" are abstract and hard to quantify.\n\nWe appreciate the reviewer’s comment. First, we note that the NG-SM kernel and the RFF approximation target different aspects of our model: **NG-SM** focuses on enhancing **expressiveness**, aiming to learn a more informative shared latent representation across views, while **RFF** improves **efficiency** by reducing computational complexity.\n\nFurthermore, we demonstrate  the individual contributions of each component in our model through our experiments. In Section 4.2, we compare our model against baselines using various kernels (e.g., SM kernel in MV-ARFLVM, Gibbs kernel in MV-NGPLVM), where our model with NG-SM kernel consistently achieves better results, demonstrating its stronger expressiveness and improved manifold learning. \nFigure 2 in Section 4.2 illustrates the model fitting wall-time on the MNIST dataset as the dataset size increases, highlighting the efficiency gains achieved by the RFF-based approximation.\n\nFinally, we will provide clearer definitions of key terms in the revision. For example:\n- **Kernel expressiveness:** Refers to how flexible a kernel function is in modeling various data patterns. A highly expressive kernel can approximate complex, nonlinear relationships, periodic behaviors, or multi-scale structures in the data.  \n\n- **Time-varying correlations:** Means the similarity between two points is influenced not only by their distance, e.g., $\\|x_1 - x_2\\|$, but also by their specific locations $x_1$ and $x_2$ in the input domain. For example, two points with the same distance apart may have high correlation in one region but low correlation in another, depending on the local data characteristics.\n\n\n\n>C3: The claimed efficiency is not empirically validated with runtime comparisons against other MVC methods.\n\nThank you for your suggestion. We note that Figure 2 in Section 4.2 have presented the runtime on the MNIST dataset, where we compare our method against other multi-view baselines (e.g., MV-DGPLVM, MVAE). The results clearly demonstrate the efficiency of our model relative to the other Gaussian process-based models. And we assume that \"MVC\" here refers to **multi-view learning methods**, which include the baselines we have compared against in our experiments.\n\n\n>C4: Is the bound of Theorem 4 tight and does it play an important role?\n\n**Tightness**: The bound in Theorem 4 is considered reasonably tight because it correctly captures the convergence behavior of the RFF approximation. \n\n   - Specifically, the term $L$ appears in the numerator of the exponent, $\\exp \\left(- \\frac{3c^2 L}{\\cdots}\\right),$\nwhich implies that the probability of the approximation error exceeding $\\epsilon$ decays exponentially as the number of random features $L$ increases. \n  - This aligns with the result that the expected RFF approximation error decreases at a rate of $O(1/\\sqrt{L})$ [1], because the exponential tail bound guarantees that if we set $\\epsilon \\propto 1/\\sqrt{L}$, the probability of the error exceeding $\\epsilon$ decreases rapidly with L, which is consistent with the same convergence rate.\n\n**Importance**: The bound provides a crucial theoretical guarantee for the unbiased and differentiable RFF estimator, ensuring that the error between the RFF-based kernel and the true NG-SM kernel matrix diminishes as $L$ increases.\n\n[1] Rahimi, A., and Recht, B. (2008). *Random Features for Large-Scale Kernel Machines*. In *Advances in Neural Information Processing Systems (NeurIPS)*, 1177–1184.\n\n\n\n>C5: Are there guidences or requirements when choosing the kernel?\n\nOur NG-SM kernel is a universal kernel, meaning it can theoretically approximate any continuous kernel function. Unlike traditional GP models, our approach does not require selecting a predefined kernel, as the kernel is learned through inference of the model parameters. This is particularly advantageous since, as suggested by the question, the ``correct'' kernel is often unknown a priori.\n\nBy learning the kernel directly from the data, our method can adaptively capture the underlying structure and dependencies without manual specification. As demonstrated in our experiments, this flexibility allows the NG-SM kernel to model diverse and complex data patterns effectively. Thus, we recommend the NG-SM kernel as a default choice in most applications where kernel flexibility is desired.\n\n>C6: The full phrase of the abbreviation RFF should appear at its first use on line 48\n\nThank you for pointing this out. We have revised the manuscript to include the full phrase **“Random Fourier Features (RFF)”** at its first occurrence on line 48."}]

Author rebuttal:

Meta-review: This paper proposes a multi-view GPLVM with a new Next-Gen Spectral Mixture kernel and scalable RFF-based inference. It tackles the issues of limited expressiveness and high cost, showing strong results across several datasets. All the reviewers agree that the paper is technically solid. Overall, the work is solid and would make a useful contribution.
Decision: Accept (poster)