Isometric Quotient Variational Auto-Encoders for Structure-Preserving Representation Learning
Authors: In Huh
changwook jeong
Jae Myung Choe
Young-Gu Kim
Dae Sin Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
auto-encoders
geometry
symmetry
TL;DR: We propose a simple yet effective auto-encoding framework that can learn the underlying symmetry and geometry in data manifolds simultaneously.

Abstract: We study structure-preserving low-dimensional representation of a data manifold embedded in a high-dimensional observation space based on variational auto-encoders (VAEs). We approach this by decomposing the data manifold $\mathcal{M}$ as $\mathcal{M} = \mathcal{M} / G \times G$, where $G$ and $\mathcal{M} / G$ are a group of symmetry transformations and a quotient space of $\mathcal{M}$ up to $G$, respectively. From this perspective, we define the structure-preserving representation of such a manifold as a latent space $\mathcal{Z}$ which is isometrically isomorphic (i.e., distance-preserving) to the quotient space $\mathcal{M} / G$ rather $\mathcal{M}$ (i.e., symmetry-preserving). To this end, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQVAEs), that can extract the quotient space from observations and learn the Riemannian isometry of the extracted quotient in an unsupervised manner. Empirical proof-of-concept experiments reveal that the proposed method can find a meaningful representation of the learned data and outperform other competitors for downstream tasks.

Isometric Quotient Variational Auto-Encoders for Structure-Preserving Representation Learning
In Huh Changwook Jeong?,, Jae Myung Choe', Young-Gu Kim', Dae Sin Kim
CSE Team, Innovation Center, Samsung Electronics Graduate School of Semiconductor Materials and Devices Engineering, UNIST
in, www.hhhhhssuug..... com beeggook.eeoggunnsst.. ac.kr
Abstract
We study structure-preservinn low-dimensional representation of data manifold embedded in high-dimensional observation space based variational auto- encoders (VAEs). We approach this by decomposing the data manifold M as M = M/G x G. where G and M/G are group of symmetry transformations and a quotient space of M up to G. respectively. From this perspective, we define the structure- presurringg representation of such manifold which is isometrically isomorphic (i.e., Distance-preserte M/G rather .M (i.c., remmeetr--rreervinge auto-encoding framework, named isometric quotient VAFS (IQVAEs). that can extract the quotient space from observations and learn the Riemannian isometry of the extracted quotient in an unsupervised manner. Empirical proof-of-concep experiments reveal that the proposed method can find meaningful representation of the learned data and outperform other competitors for downstream tasks.
latent space Z to the quotient space To this end, propose novel
Introduction
There has been a common consensus that natural image datasets form a low-dimensional manifold M embedded in a high-dimensional observation space Rd, i.e. dim(M) < d (6). From this perspective, a good neural network is a mapping function that can recognize the underlying structure of the data manifold well (25 A question that arises is what structures should be represented from the data.
For unsupervised learning task, especially in generative modeling. if we suppose the data manifold to be Riemannian one, then preserving geometric structures in the representation space is a key To make things more explicit, if Z is a latent representation of M, then Z should be isometrically isomorphic to M in the sense of the Riemannian isometry. This means that an infinitesimal distance on the data manifold should be preserved in the latent space as well, promoting geodesic distances in the latent space that accurately reflect those in the data space. Recently, several papers S (4 25 have suggested using the concept of Riemannian isometry for deep generative models, such variational auto-encoders (VAEs) 22 and generative adversarial networks (GANs) 14, to obtain more meaningful and relevant representations. These concepts have been applied to improve latent space interpolations [30] or clustering [55 of data manifolds.
consideration g 25
However, the Riemannian isometry does not tell the whole story of the data structure, particularly in the case of vision datasets. One common property of natural images is that there are symmetry transformations that preserve the semantic meaning of the given image. For example, if one applies rotational transformation to certain image, then its semantic meaning, e.g., its label, does not change. In this situation, what one really wants to do is represent the inherent geometry of the manifold, i.c., the geometry up to such symmetry transformations 229
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The concept of the inherent geometry can be formalized by using the notion of the principal bundle M .M/G x G, fiber bundle that consists of the group of symmetry transfor- mations and quotient space M / as the fiber and base spaces, respectively [16 26). In this case, the inherent geometry of M up to G is indeed determined by its quotient M/G solely; the quotient formulates invariant structure of the data, thus a measure on M/G gives a mea- sure M that is invariant to the actions of G. Therefore, one should find the Riemannian isom- etry of M/G rather than M for geometric- meaningful representation of the dataset (see Fi Nevertheless, the best of our knowl- edge, there has been a lack of studies, at least in the field of generative models, that integrate both the concepts of the quotient space and its Riemannian geometry.
dx
x
82
&.
XIG
Hg
dz = c dxoG
Figure 1: We denote X' C IRd as an extrin- sie view of the data manifold M. consists as X' = X/G x G where X/G is quotient space of X and G group of symmetry trans- formations that naturally act on X, respectively. From this perspective, the structure- resentation 2 should preserve all geodesic dis- tances of X/G (dx/a(--)) rather than them of X' (dx(x, i.c., d2 (Z), 22) c dx/G(x1, X2). 1 X/G. Z1, Z2 2, where dz(:;)) is distance on 2. Such a space preserves all the essential geometry and symmetry of the data.
rep-
In this paper, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQ- VAEs), that can extract the quotient space from observations and learn its Riemannian isometry in an unsupervised manner, allowing for a cor- rect representation of the data manifold and accurate measurement of distances between samples from the learned representation. The proposed IQVAE model can be applied to a variety of practical tasks such as downstream clustering and out-of-distribution (OoD) detection. We evaluate the model's performance using three datasets: rotated MNIST 44 mixed-type wafer defect maps MixedWMM38 and cervical cancer cell images (SIPaKMeD)
42
32
In summary. our contribution is threefold:
We propose novel approach for structure data manifold as a principal bundle, and formulate it as an unsupervised learning task that finds Riemannian isometry for the quotient of the data manifold.
structure-preseeving preserving representation learning by viewing
We introduce practical method, called the IQVAE, which learn such structure- preserving representation in an unsupervised manner by using the auto-encoding framework.
We demonstrate through experimental evaluations on various datasets that our proposed method can find useful representations and outperform other competing methods for down- stream tasks such as classification, clustering, and OoD detection.
2 Related Works
2.1 G-Invariance and Quotient Space Learning
Quotient space learning highly related to the notion of the G-invariance. Data augmentation the most common approach to deal with the G-invariance of label distributions [18. To achieve such symmetry more explicitly, various G-invariant and G-equivariant architectures have been proposed:
[10 has proposed G-convolution beyond the conventional translational equivariance of convolutional neural networks. [12] has generalized further for Lic groups. G-orbit pooling, proposed by (23), can be applied to both arbitrary discrete groups and continuous groups by discretizing them. Some research has attempted to discover the underlying invariances within data 4 46 31 (3). These approaches can be integrated as major component of auto-encoders to represent the quotient symmetry of the data manifold, as we will discuss later Section 3.3
More relevant to our work, 37 has proposed using the consistency regularization for VAEs to encourage consistency the latent variables of two identical images with different poses. can be considered as the unsupervised learning version of the data augmentation and does not guarantee the explicit G-invariance of the regularized latent space. (5) has disentangled image content from pose variables, e.g., rotations or translations under the VAE framework. However, it has some limitations,

---

such as the requirement of using specific decoder, the spatial transformer network [20]. It also has problem that it cannot handle arbitrary groups beyond the rotation and translation. [TT has dealt with more complicated SO(3) group under the VAE framework. [29 has proposed the quotient auto-encoder (QAE) that can learn the quotient space of observations with respect to given arbitrary group, extending early works in quotient images [34 41. However, it is deterministic model and therefore cannot generate new samples easily. In addition, all the above-mentioned studies have not considered the Riemannian geometry of the learned latent representation.
2.2 Rimeannian Geometry of Generative Models
Recent studies have proposed view the latent space of generative models as Riemannian manifold 79 In this perspective, the metric tensor of the latent space is given by pullback metric induced from the observation space through the generator function. In line with this theoretical perspective, the flat manifold VAE (FMVAE) [8] and its variant [25] 15 have been proposed to regularize the generator function to be Riemannian isometry by matching the pullback metric with the Euclidean one. The regularized latent space preserves geodesic distances of the observation space, making a more geometrically meaningful representation that can be useful. However, the FMVAE and its variants do not take into account the underlying group symmetry, which can lead to inaccurate estimation of distances between data points. For example, they could estimate a non-zero distance between two identical images with different rotational angles, which is unsuitable for clustering tasks.
3 Method
Suppose x E X' C. RR is an observation where X' is an extrinsic view of M, i... the data manifold is realized via mapping representation where n < d. In addition, suppose G to be a group (of symmetry transformations) that its group elements g E G naturally act on X' by a left group action a G X' - X. We will denote agg, x) as x. Typical examples of G include the special orthogonal group SO(2), group of 2-dimensional rotations that can also naturally act on d-dimensional images by rotating them.
M -> Rd such that X' x (M), and E 2 C R is its latent
3.1 Auto-Encoders
The auto-encoder framework consists of two parameterized neural networks, an encoder p4 : R R" and a decoder H&g : R" Rdd Vanilla auto-encoders find low-dimensional compression of observations by minimizing the following reconstruction loss:
E (x) [AA((O, Φ; x)] (xx |||x Me pe p [x)
(I)
where px (x) is the data distribution. The expectation over (x) can be computed via Monte Carlo (MC) estimation with finite samples X = {x as (x) (O di x)] ~ (1/N) Σ |x, N& o (xx Unless otherwise mentioned, we will omit the expectation over px((x) for brevity.
Although sufficiently deep auto-encoders can provide effective latent compression [X( = Z C R of high-dimensional data, they tend to overfit and the learned latent manifolds are often inaccurate (25 i.e., they cannot consider the underlying geometry of data. This can lead to problems such as incorrect latent interpolation or poor performance in downstream tasks, addition to the absence of the ability to generate new samples.
3.2 Variational Auto-Encoders
The VAE [2 is a stochastic auto-encoding architecture belonging to the families of deep generative models. Contrary to vanilla auto-encoders, the VAE framework consists of two parameterized distributions (z x) and p& P=(x(22 (z) where the former and the latter are variational posterior and likelihood, respectively. VAES try to maximize the marginal log-likelihood of observations by optimizing the following evidence lower bound:
log Po (x) 2 Ex [log Po (x(x)] - DRi (p((i)i))ll & LO,E((, di x),
(2)
where Pz (z) the latent prior and (x
(..)) :) is the Kullback-Leibles (KI.) divergence, pz(z) is often given by a normal distribution. (z) 0, I,) where 0 is zero vector and I, is X n identity matrix.
3

---

For vanilla VAEs, qo xx) is chosen as multivariate Gaussian distribution with a diagonal covariance
matrix N ( (x x), ( (x))) and represented by a neural network encoder (H4. σa) : Rd
In Ry. A latent variable sampled by using the reparameterization trick [ + (x) @
where ~ (e(E O, I.). Although Po (x (z) is chosen depending on the modeling of the data, it is often
taken simple distribution such Gaussian with fixed variance, NN ( (z), _L), represented
by neural network decoder M R" - Rd. In this case, the VAE objective 2 can be seen as
regularized version of ( given by sum of the stochastic outo-encoding reconstruction term xx e)
CAB (0, Φi p;x,
||x Hol (x) + ae (x) @ e)ll2 and 3-weighted KL divergence LKL (K x, ()
(D (z) (x(()) diag o (I)
I, as follows:
CvvE Φ: x; 3) ((t) (C, Φ; c)] + (x, B).
Practically, the expectation over ov term is estimated via single-point MC when the mini-batch size is sufficiently large [22, When 3 - 0 and o3 - O, VAEs reduce to vanilla auto-encoders.
By minimizing 3 VA can learn the probabilistic process on the smooth latent space [3 that can easily generate plausible new samples by sampling a latent variable and decoding it, i.c., P (2) x X'. However, VAEs, like vanilla auto-encoders, do not ensure that the learned latent representation preserves the crucial information about the symmetry and geometry of the data manifold.
3.3 Quotient Auto-Encoders
QAE 29 is modification of the deterministic auto-encoding framework that can find the quotient of the observation set //G by replacing (T) as the following quotient reconstruction loss:
( Φ: x, G) inf ||g x Pg (x) 9EG
As shown in 44, QAEs aim to minimize the set distance between the auto-encoded sample fs (00 (x) and G-orbit + {g xlg E G) of the given rather than x itself. It is noteworthy that G-orbit loses the information of G for a given x, which is the case of the natural quotient for x up to G. Thus, QAE can learn the quotient space of observations directly: for the ideal case of QABs, i.e., when LOAR(O. d; x, G) - O, the following property holds:
P& (x) He po(x x & &. Vgg E G,
which means that the image Ms u (X) A X does not contain any information on g. In addition, QAEs typically adopt G-invariant architectures such G-orbit pooling (23] (see Section B of Supplementary Material (SM) for details) for the encoder part to explicitly guarantee the following G-invariant latent representation:
16 xg 2, E
where p I2d + R" is the G-invariant encoder. It worth mentioning that these encoder architectures are unable to learn meaningful representation of the data when using a vanilla auto- encoding framework, as the reconstruction loss (I) cannot be reduced enough; on the other hand, QAEs can use the explicitly G-invariant encoding architectures easily by taking advantage of the infimum in 4). This allows for both X and p (X) Ž to possess the invariant property up to G.
3.4 Isometric Quotient Variational Auto-Encodesd
Although QAEs with G-invariant encoders extract the underlying symmetry of observations, they still have the following drawbacks. First, QAEs are deterministic and may overfit to limited observations, similar to vanilla auto-encoders. Second, the Riemannian geometry of the extracted quotient manifold may not be preserved in the latent space. To address these issues, we present a stochastic version of QAEs called QVAEs, and further improve them with a novel framework for isometric learning called IQVAEs.
QVAEs. Inspired by the connection among [ 3 and 4). we propose the stochastic version of QAEs (QVAEs) which minimizes the following stochastic quotient objective:
LoVAE(O, x. B, G) A E.-N(() [Q x; G)] + (0: B,G),
(5)

---

where the first term is defined as LoAE(O, 0; G) - inf xx ns ( ( (x) + og (x) @ c)
and the second term is CKL x, G) - 3DRL (N(2lug (x), diaglo llzz We denote
(ng :M0000 :g0) Rd R R2 for the G-invariant multivariate Gaussian encoder.
QVAEs are able to generate new samples from the quotient space, unlike QAEs. It means that QVAEs can approximate the quotient space of the data manifold X/G _ & as a form of the immersed manifold pal 2) = X where is G-invariant latent representation. This leads us to introduce the concept of Riemannian isometry as geometric regularization for QVAEs.
Isometric regularziation to quotient spaces. Before introducing the isometric regularization of the proposed QVAE, we review some basic concepts of the Riemannian geometry 2 25
Definition 3.1. (Riemannian manifolds) n-dimensional Riemannian manifold X is a n- dimensional differentiable manifold that is equipped with symmetric positive-definte metric lensor H. (x) E RTx" (n X n matrix). It defines local smooth inner product on the tangent space TxX' as (ux, Vx) Hx))xx with tangent vectors Ux Vx Tx each X. allowing us to measure the distance and angles between tangent vectors at each point.
Definition 3.2. (Riemannian distances) Suppose X' is a connected Riemannian manifold equipped with Riemannian metric Hx(x) at each point x E X. A length of a curve on X' between two points Xy and X1 is defined as:
Lhxxl = //' dt dt (fx(tttt), (x())))
where 7x : 10 + X' is curve on X' that travels from X -0) to x1 (1) and (x)() = (t) E tt is the velocity of the curve. The curve minimizing L is called a geodesic. The Riemannian distance is given by the length of the geodesic, i.e., dx (Xo, x1) = inl, L[xx]-
Definition 3.3. (Immersions) Let be an open set RR, smooth map μ - Rd (n d) is called an immersion if its pushforward djux T + (T()))(()) is injective for all E 2. Then, (22) is called an immersed submanifold.
Definition 3.4. (Geometry of immersed submanifolds) Suppose μ Z - Rd is an immersion onto submanifold X C Fdd The Euclidean inner product in K a proper inner product of tangent vectors of X. Thus, X' is a Riemannian manifold equipped with a Riemannian metric H. (x) = La.
i.e. (ux, Vx H2x = I.ILavx u for tangent vectors Ux Vx E Txd' at each x X. Definition 3.5. (Pullback metric) Suppose μ 2 - Rdd is an immersion onto Riemannian submanifold X' C IRd equipped with a Riemannian metric Ia. Then, for tangent vectors Ux. Vx E TxX' each uzz) X 2(Z), the following holds:
(ux, Vx)H, u vx
Idpay, (ddp)) A (2) = (U,, Vz)HH,
Therefore, Z can be viewed Riemannian manifold equipped with Hz,(z) called a pullback metric. In addition, because Z R and X c R., the pushforward duz is equal to the standard Jacobian [z) of μ4, i.c., (z) J (z) Definition 3.6. (Isometric immersion) Suppose is a Riemannian manifold equipped with a Riemannian metric Hz at each z E Z. A smooth immersion μ 2 - Rd onto X' C IR is Riemannian isometry if the metric (z) equal to the pullback metric (z), i.e., Hz(z) = [z) for every z E Z. In this case, the length of curves on X' is equivalent with that on Z:
[[xx]= = [(" di (1x( f. di (T/TH)x())
f( di /
= =f. d = = S/' dt ((22()), = L(72),
where 72 : [0, 1] Z and 7x [0, X' are respectively curves from 72(()) to = (1) and from Xg = 7x to X1 = 7x when Xy = p(zo), x1 = u(zz), (1) TT 2, and
accordingly (t) E (2)
With some mild assumptions 36). a sufficiently well-trained decoder us 2 + Rd of the QVAE can be viewed smooth immersion onto the quotient of X X/G.. Then, from Definition 3.6
5

---

the following regularization should be added to 5 to make the decoder be a Riemannian isometry between the latent representation E and quotient X:
L0s Φ x, H 2. >) AE Ag (0)) Mo (2) Mug
H F.
(6)
where Ju (2) is Jacobian of Mgg at 2, A is a pypereparameter, and H (2) is a metric tensor of å.
There are two things that should be clarified for practical computation of (6) The first aspect to consider is the selection of the Riemannian metric H2( A commonly used and partic- ularly useful choice of H (2) is the scaled Euclidean metric in R", i.e., H H Cn = eL,, where e is a constant. This is because it is the case that using the Euclidean distance on the latent representation d (2o, 21)
Algorithm IQVAES
Input: data {x]] hyper-parameters (3, A) group G, G-
invariant encoders (10 og)) ) decoder Há
Initialize Φ. C..
while training do
Sample {ar ~ [0, 1] {a ~ (o.
Compute {ug, 0b {ug (10(xe) (x.) ( (x,))
Sample {z: ;= [u] + o Ö De:
Shuffle {z, N shuffle ((zi))e((()))) )i=1)
Augment [7, J.a [(1 as 2z + ME/j=N
Compute COAE = Ex= mingeo 11g 11g*x: (z)
V(2 - T (2] - 2n) can pre- serve the geodesic distance quotient manifold d (Xg. x1) up to The constant can be regarded as either pre-defined typer-parameter or a learnable one. More generally, one can use any parameterized sym- metric positive-definite matrix C, for H 2 (Z). Note that such matrices can be achieved from an arbitrary learnable parameterized n x n matrix M, by using the Cholesky decomposition as follows:
Compute CKL = > (Ka(N dinglo)) I.))
the
Compute {J 1.N {Jmm [
Compute Liso = Em [I(J Ag) 'J Calle
Optimize (CoAE + 3Lx. + /Csoo/NN 0,
end while
(7) where L, is a real lower triangular matrix with all positive diagonal entries and is a small non- zero constant. In this case, using the Mahalanobis distance on the latent space dg 2,) =
C (L,,), L,, lower [M,] diag[M,] + diag [M,] 12 + e'In:
V(2, Cn 20) preserves the geodesic distance of the quotient space of the observation manifold
81).
The second aspect to consider is the sampling from (z It can be considered Gaussian vicinal distribution for finite observation samples that smoothly fills the latent space where data is missing. In addition to that, following [8) we use the mix-up vicinal distribution (39] to effectively regularize the entire space of interest to be isometric. As a result, the tractable form of (6) is equal to
[J] (fäl (xi,, C1,5)) pe (x.,) c1,)) C,
(8)
1g( (xx,,) + (x @ a.j- Practically, it is computed by sampling mini-batch of latent variables, i.e., { (x)(x))) + og (x:) O I shuffling it for / = shufflel ((2,) and mixing-up them.
where fø (xx.j, c.y) (1 - a)², + a2, is the latent mix-up for 2,, mg
G
IQVAE. We define the IQVAE as class of QVAEs whose objective function is given by the sum of the variational quotient objective LOVAE (5) and Riemannian isometry Liso (8) as follows:
(9) The proposed IQVAE optimization procedure is summarized in Algorithm || (see Section A of SM for additional tips). In the IQVAE, we typically set C, = ILI, For clarity, we will refer to the IQVAE with (7) as IQVAE-M, to distinguish it from the basic version.
LIOVAE (0, Φ; x. 3, X, G) & LovAE(O, d; x, 3, G) + Ciso (0, Φb C,:
4 Experiments
We compared our proposed QVAE and IQVAE with six different competitors: the auto-encoder (AE), VAE, 3-VAE [1)) consistency regularized VAE (CRVAE) [37] QAE (29], and FMVAE (8]. We
used the same convolutional architecture hyper-parameters, optimizer 21). and training scheme for
We assume the straight path between 20 and éi is an on-manifold path for every pair 20, 21 E 2. In g) we temporarily retrieve the expectation on observations to make things more explicit.
'We used G-orbit pooling (see Section B of SM for details). which can achieve G G-invariance in combination with arbitrary convolutional models, for encoders of QAE, QVAE, and IQVAE.

---

Table 1: The averaged clustering ARIs and classification test accuracies for (2T (2otater (left) and rotttt (right) MNIST across various models, each repeated five times.
METHOD
K-MEANS
GMM
SVM
RF
METHOD
M-MEANS
GMM
SVM
RF
AE
21.8
29.5+22
86.4
86.2.10.5
AE
10.0
12.5420
68.3=26 71.2
VAE
23.1=05
28.4
87.4
87.440.4
VAE
11.4=1.
14.7#1.
72.8=1.
73.3+0.
8-VAE
21.5=110 26.5+0a
88.5
88.5+005
8-VAE
10.1
11,9+1.1
72.5=0.5
73.5+0.
CRVAE
34.9=1.
37.4+2.4
93.0=0.2
93.0+0..
CRVAE
26.1
33.4236
78.3=24
80.3+20
QAE
45.2.44.
65.543.6
93.1=0.3
92.7.
QAE
36.9
52.943.1
91.8=06
91.7+0.
FMVAE
19.0+1.2
25.742.
85.6=0.5
85.
FMVAE
10.1 11,9410
67.6=0.7
69.240.
QVAE
59.3+6.5
72.645.2
93.0=0.4
42.5
QVAE
53.9
66,346.5
93.0=0.1
92.5
IQVAE
72.8=29
74.4
93.2=0.2
92.9
IQVAB
70.9
72.9
-2.9-0a3 92.7
all models. Implementation details can be found in Section D of SM. After training, we encoded all samples and extracted their latent representations across models. Then, we conducted latent- based downstream clustering tasks by using the simple k-means and Gaussian mixture model (GMM) [33] Similarly, we conducted downstream classification tasks by using the support vector machine (SVM)
17 and random forest (RF) classifiers. We used the same hyper-parameters for downstream models regardless of the benchmarked latent representations. Then, we computed the adjusted Rand index and test classification accuracy for clustering and classification models, respectively.
(ARI) 38
4.1 Rotated MNIST
Rotated MNIST consists of randomly rotated images of handwritten digits. It has the rotational group sym- metry of G SO(2). We padded the original MNIST images to size 32 32 for improved usability. Then, we generated yo-rotated MNIST by randomly rotating each sample in the MNIST with uniform distribu- tion of [- 2]. We considered both o = 2n /3 and و 2m cases (see Section C of SM for ex- amples). We used 60,000 samples for training and 10,000 samples for testing.
Predicted
Preducted
Figure 2: Clustering confusion matrices of
(left) QVAE and (right) IQVAE for y 2m.
Basic evaluation. For quantitative analysis, we summarize downstream task performances across the models in Table [1] It clearly shows the proposed QVAE and IQVAE outperform other competitors by large margin because they consider the group symmetry explicitly. Moreover, the IQVAE shows greatly improved clustering performances even over the QVAE when م 2n. We deduce that the Rieman- nian isometry helps the IQVAE recognize a meaning- ful distance relationship between controversial samples, e.g., digits and 9, as depicted in Figure For qualitative analysis, we also visualize the learned latent representations for the test dataset of
Figure 3: UMAP visualizations of (left) VAE and (right) IQVAE for ip 2
the rotated MNIST by using the uniform manifold approximation and projection (UMAP) 28 (see
Figure 3 and Section E of SM for all models). It further supports that the IQVAE succeeds in finding meaningful representations of the dataset.
Reconst, Original
Aligned reconstructions. Figure 4 shows six im- ages of digit 7 and their reconstructions based on the proposed IQVAE demonstrating that the IQVAE can reconstruct rotated digits in the aligned frame thanks to learning the quotient up rotations.
Figure 4: Six images of digit 7 and their re- constructions based the proposed IQVAE.
Better interpolation. Figure compares linear interpolations between digits and 7 for the VAE, QVAE, and IQVAE. is conducted by parameterizing a linear path as 2(() z()) + (z(10) 20)) (1/10) and then reconstructing x()) = ns (2(1)) for i E {0, 1,..., 10). As shown Figure (a), the VAE shows meaningless interpolation from = 5 to = 9 (see Section F of SM for other examples).
7

---

(b)
WA
Volume measure
600
QNAE IQVAE
QVAE
IOVAE
t=0
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
t=9
t=
Figure 5: (a) Linear interpolations between digits and for the VAE. QVAE, and IQVAE. (b) The geometric volume measures along each linear path of the QVAE and IQVAE.
While the QVAE shows more convincing interpolation compared to the VAE, it also shows abrupt changes at 7 and 8. The IQVAE shows the smoothest interpolation between digits 1 and 7 compared to the other models owing to the isometry regularization. The smooth interpolation of the IQVAE is also quantitatively confirmed in Figure (b) which compares the geometric volume measures along each linear path of the QVAE and IQVAE. The volume measure is computed via V det. ue (z(1))J,a fhe (z(()) and can viewed as local infinitesimal volume ratio between the latent and observation spaces. The volume measure of the QVAE shows clear peak near 8, while that of the IQVAE is nearly constant.
Evidence for isometry. To demonstrate that the learned decoding function of IQVAEs up- holds the desired Riemannian isometry, we evaluated the condition numbers of the pull-
2.0
Condttin number
3.5
20- VAE
back metric tensors and the Riemannian vol- ume measures for all test samples across mod-
els. The condition number is defined the ratio Amax/ Aminn where Amax and Amin respec- tively represent the maximum and minimum eigenvalues of the pull-back metric, denoted as Pe (z)) All computed volume measures are normalized by their average values. Therefore, condition numbers and normalized volume measures approaching 1.0, accompanied by minimal variances, indicate that the learned latent repre- sentation is isometrically isomorphic to Euclidean space. In other words, the learned pull-back metric aligns with the Euclidean metric. As shown in Figure 6 the IQVAE presents condition numbers and volume measures close to 1.0, exhibiting trivial variances compared to other models.
CHAE
OVAE
WAE
QVRE
IQVAE
Figure 6: (left) Condition numbers and (right) vol- ume measures for the -ottttd MNIST for the VAE, QVAE, and IQVAE.
J
Sample efficiency comparison. Table 2 com- pares the sample efficiency IQVAE and the most comparable baseline, QAE, in terms of classification accuracy. The results demonstrate the effectiveness of the proposed method as IQ- VAE shows more robust performance with re- spect variations in the training sample size.
Table 2: Classification test accuracies for 2m- rotated MNIST with varying training sample sizes.
SaMPLE SIZE QAE SVM IQVAE + SVM
5,000
83.9
87.61066
10,000
85.5
89.8+10
60.000
91.8406
92.940.3
Test-time orbit augmentation. The current QVAE and IQVAE require more computational cost than vanilla VAE due to the G-orbit pooling encoders, which augment the number of obser- vations [G] times larger by expanding a data point x G-orbit G It might be bot- tleneck, especially when training the proposed model with larger-scale datasets. To resolve this issue, we suggest the test-time orbit augmen- tation strategy, which involves using a coarse step size when discretizing the group parame- ters (e.g., rotation angle) during training, and then switching finer step size during the inference phase. Table compares wall-clock training
Table 3: Training wall-clock time per epoch and clustering ARIs of VAEs, QVAEs, and IQVAEs. We used a single V100 32GB GPU.
METHOD
TIME [s]
k-MEANS
GMM
VAE
1.12
11,4112
14.7116
QVAE (36:36)
10.17
53.9
66.3465
QVAE (12-12)
4.06
50.8+26
59.2+1.
QVAE (12-36)
4.06
54.6428
63.8439
TQ IQV (36-36)
13.22
70.942.
72.9+1.5
IQVAE (12-12)
6.11
59.6427
60.9
IQVAE (12-26)
6.11
64.3+21
65.0+2.

---

times and downstream task performances for rr-rotteed MNIST datasets for the VAE, QVAE, and IQVAE. The former and latter numbers in parentheses for QVAEs and IQVAEs in the table indicate discretized group sizes used in the training and inference phases respectively. The results show that the proposed technique can scale the training time without severe performance degradation.
Effect of mix-up. We employed the mix-up technique (8) to regularize the entire latent space. By augmenting the latent space with the mix-up, it is possible to smoothly fill gaps in the latent space regions where data might be missing. This becomes particularly crucial when the number of available training samples is limited. To val- idate this, we trained IQVAEs both with and without the mix-up technique in scenarios: one with a training sample size of 60,000 and the other with 10,000. Subsequently, we computed the condition numbers of the pull-back metric tensors and mean-normalized volume measures for test samples. Figure 7 shows that using the mix-up reduces the variances of both condition numbers and volume measures for pull-back metric tensors, demonstrating its efficacy in regularizing the entire latent space. Furthermore, the impact of the mix-up approach becomes more noticeable when the training sample size smaller.
Nomratzus valume measure
1.5
Condition
0%
1.0
0.5
(S0K)
(634)
130C
1000
(600)
(G3K)
[20K]
(10Q
Figure 7: (left) Condition numbers of the pullback
metrics and (right) volume measures of 27r-rotated MNIST test samples, with and without the mix-up for IQVAEs. The numbers in parentheses indicate the training sample size.
4.2 MixedWM38
MixedWM38 is dataset containing 38,015 52
Table 4: Clustering ARIs and classification test
52 semiconductor wafer maps, separated into
F-scores for MixedWM38 (repeated five times).
38 defect categories including 1 normal type, 8
single-defect types, and 29 mixed-defect types
METHOD k-MEANS
GMM
SVM
RF
42
(see Section C of SM for examples). We
AE
18.710s
22.020.
81.7106
83.8
assume rotation and reflection (i.e., G = O(2)) symmetries are present in the MixedWM38. We used 30,720 samples for training and 7,295 sam- ples for testing.
VAE
21.31007
24.920.6
82.510.3
84.
8- AVE
21.310. 27.1
82.5106
84.5
CRVAE
21.4+06 24.620.
82.34a9
84.9=1.
QAE
22.7*0. 30.721.
84.440.
85.1202
FMVAE
17.6404
25.8207
81.4402
83.1204
QVAE
23.3
31.140.5
85.0
85.1
Basic evaluation. We performed an experi- ment on the MixedWM38 dataset, similar to the one previously conducted on rotated MNIST. Unlike the rotated MNIST, the MixedWM38 dataset requires multi-label classification task, which is a key difference between the two experiments. Thus, we use the F-score as a classification accuracy for this problem. Table 4 summarizes the downstream task performances of eight models the MixedWM33 dataset (see Section E of SM for UMAP visualizations). As shown in the table, the proposed methods have a higher performance on the downstream task than the other models.
IQVAE
27.54.0.
35.3
84.4+0.
85.6=0.3
OoD evaluation. We further demonstrate the effectiveness of the proposed IQVAE in detecting OoD samples, important task in the semiconductor engineering field. In order to mimic a real-world setting, divided the 7,015 single-defect type data (8 classes: C, D, FI., ER, I., NF, S, R; see Section C of SM) into two sets: C-EL-L-S the in-distribution set, and D-ER- NF-R as the OoD set. Then, we randomly rotated all samples the C-EL-L-S subset, and considered the rotated samples an in-distribution set, aligning with the nature of the semiconduc- tor manufacturing process 43 Using auto-encoding models trained on the original C-EL-L-S data, we formulated an OoD detection task, the goal of which to differentiate between samples from the rotated C-EL-L-S set (in-distribution) and samples from the D-ER-NF-R set (OoD). We assessed the standard distance- based OoD detection: it labels instance OoD if its latent vector is farther than pre-defined threshold from any
Table 5: AUCROCs and AUPRCs for MixedWM38 across the models (repeated five times).
METHOD AUCROC
AUPRC
AE 47.3436
27.2-33
VAE
62.4+3.1
34.3=24
3-VAE
66.1
41.4=68
CRVAE
80.546.1
60.4+10.2
QAE
90.2+3.7
81.8-34
FMVAE
69.8+5.6
42.5
QVAE
90.3+1.1
79.5
IQVAE
93.820.4
88.3=17

---

class-conditee conditionl mean latent vector of learned in nn-distribution sets, in terms of the Mahalanobis
distance 24 (see Section of SM for details). Table[]showe the comparison of competing models using standard metrics (AUCROC and AUPRC) for threshold-based Ool detection, demonstrating that the proposed IQVAE excels in the Ool tasks as accurately learns the data structure.
4.3 SIPaKMeD
The SIPaKMeD dataset includes 4,049 single- cell Pap smear images for cervical cancer diag- nosis and is split into 5 classes [32 (see Section Cf SM for examples). We assume 0(2) sym- metry for the SIPaKMeDD dataset. We resized the original SIPaKMeD images size 32 32 for better usability. In accordance with the orig- inal data splitting, 3,549 samples were used for training and 500 samples for testing.
Table 6: Clustering ARIs and classification test accuracies for SIPaKMed (repeated five times).
METHOD M-MEANS
GMM
SVM
RF
AE
33.4=4.0
35.621.0
78.6
76. 1..
VAE
33.9=1.5
34.0=1.7
79.4
76.5
8-VAE
34.4=1.
33.7
79.1
76.641.
CRVAE
24.9-3.6
31.941.3
77.1
80.641.1
QAE
40.9 41.1#29
80.5
81.610.7
FMVAE
39.6-0.3
37.9
80.0
75.74:1.2
IQVAE-M. As did with rotated MNIST, we conducted the same experiment on the SIPaKMeDD dataset. Furthermore, also as- sessed the proposed IQVAE-M that incorporates
QVAE
40.6=12
40.622.
82.2208
81.7:109
IQVAR
39.3=04
44.7
82.720.
80.9
IQVAE-M
43.0=1.1
46.5222
84.32007
81.6
learnable flexible Riemannian metric as described in (7) and utilizes the latent Mahalanobis distance when computing the radial basis function (RBF) ker- nel K : R" x x + R of SVMs as K ((zo, 21) = exp (-(z) ) 2 C,, (z) zo)). To quantitatively analyze the learned representations, we present the downstream task performance of the competing models on the MIPaKMeeD dataset in Table [ (see Section E of SM for UMAP visualizations). As
shown in Table the proposed methods exhibit superior performance compared to other models"
5 Conclusion and Limitation
We have proposed and demonstrated the effectiveness of IQVAE, a simple yet effective approach that maintains symmetry and geometry in data. However, our work has two main limitations as follows.
Predefined groups of symmetry transformations. We assume the group structure of given dataset is known in advance. This group invariance is represented using the quotient auto-encoding framework with a Riemannian isometry. However, several recent papers have delved into learning an unknown group directly from data. For example, (35] presents novel neural network that identifies bispectral invariants. [3I] employs Lie algebra to find underlying Lie group invariances. [46] tackles the disentanglement challenge with VAEs, aiming to learn the latent space in terms of peeepareeeter subgroups of Lie groups. Integrating these methods with our IQVAEs learning approach could be a promising direction for future research.
Euclidean metric tensors. We focus on cases where the metric tensor of the observation space is Euclidean. If the intrinsic dimensionality of the image manifold is significantly lower than that of the ambient observation space, the Euclidean metric of the observation space can well serve as a Riemannian metric for the intrinsic geometry of the image manifold. This reasoning supports use of the pullback metric as metric tensor for the latent space. However, when constructing a latent representation that effectively addresses specific task, a specialized metric tensor might be more appropriate than the standard Euclidean metric of the ambient space. For instance, when tasks involve capturing subtle variations in a local image patch, the pullback metric derived from the entire observation dimensions may not provide the most efficient geometry. In such cases, a specialized metric structure better suited for capturing these local variations should be considered. In this context, a task-specific metric can be determined using prior knowledge about the tasks [27]. Alternatively, IQVAEs can sunesulpervs learn this specialized metric with minimal labeled data, leveraging metric learning concepts 3. This approach holds significant potential for future research, as it allows the model to tailor its representation to the specific requirements of the task, thereby improving overall performance.
'When using the Euclidean RBF (C., = I.)) for SVMs of the IQVAE-M. MM the accuracy decreases to 81.9; it shows the usefulness of the learned metric and the correspouding Mahalanobis RBF.
10

---

Acknowledgments and Disclosure of Funding
Changwook Jeong sincerely acknowledges the support from Samsung Electronics Co., Ltd (10221027- 03271-01, 10230220-00500-001) and is further supported by the Technology Innovation Program (20020803, RS-2023-00233956 and Korea Institute for Advancement of Technology (KIAT) grant (P0023703), both funded by the MOTIE, Korea (1415180307, (415187475) the National Research funded by the MSIT, Korea, the IC Design Education Center (IDEC), Korea, and the research project funds (1.220125.01, 1.230063.01) and supercomputing center of UNIST.
Foundation of Korea (NRF) grant (RS-2023-00257666
References
[1] Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: the curvature of deep generative models. In International Conference on Learning Representations, 2018.
[2] Georgios Arvanitidis, Soren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. In International Conference on Artificial Intelligence and Statistics, pages 631-639. PMLR, 2021.
[3] Aurélien Bellet, Amaury Habrard, and Mare Sebban. Metric learning. Springer, 2022.
[4] Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems, 13:17605-17616, 2020.
[5] Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentangling image content from translation and rotation with spatial-VAE, Advances in Neural Information Processing Systems, 32, 2019.
[6] Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disentanglement perspective. IEEE Transactions Neural Networks and Learning Systems, 27(10): 2008, 2015.
[7] Clément Chadebec and Stéphanie Allassonnière. A geometric perspective on variational autoencoders. arXiv preprint a0Xiv:2209007370, 2022.
[8] Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with VAEs. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2020.
[9] Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metries for deep generative models. In International Conference on Artificial Intelligence and Statistics, pages 1540-1550. PMLR, 2018
[10] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning, pages 2990-2999. PMLR, 2016.
[11] Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding. In ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2021.
[12] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance Lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3176-3176. PMLR, 2020.
[13) Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations, 2020.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.

---

[15] Amos Gropp, Matan Atzmon, and Yaron Lipman. Isometric autoencoders, arXiv preprint SkXiv:2006.092880 2020.
[16] Jihun Ham and Daniel D Lee. Separating pose and expression in face images: manifold learning approach. Neural Information Processing-Letten and Reviews, 114:991- 2007.
[17] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-25) 1998.
[18] A Herrandez-Garcia, P König, and TC Kietzmann. Learning robust visual representations using data augmentation invariance. 2019 Conference on Cognitive Computaatioal Neuroscience, pages 47. [SI: sn]. 2019.
[19] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAF: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations. 2017.
[20] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems, 28, 2015.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[22] Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
[23] Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Mare Pollefeys. Ti-pooling: wransformation-invariinn pooling for feature learning in convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 289-297, 2016.
[24] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distributinn samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[25] Yonghyeon Lee, Sangwoong Yoon, Minjun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. In International Conference on Learning Representations, 2022.
[26] Jongwoo Lim, Jeffrey Ho, Ming-Hsuan Yang, Kuang-chih Lee, and David Kriegman. Image clustering with metric, local linear structure, and affine symmetry. In European Conference on Computer Vision, pages 456-468. Springer, 2004.
[27] Laura Manduchi, Kieran Chin-Cheong, Holger Michel, Sven Wellmann, and Julia Vogt. Deep conditional Gaussian mixture model for constrained clustering. Advances in Neural Information Processing Systems, 34:1 1304-11314, 2021.
[28] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
[29] Eloi Mehr, André Lieutier, Fernando Sanchez Bermudez, Vincent Guitteny, Nicolas Thome, and Matthieu Cord. Manifold learning quotient spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9165-9174, 2018
[30] Mike Yan Michelis and Quentin Becker. On linear interpolation in the latent space of deep generative models. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning, 2021.
[31] Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold Smeulders. LieGG: Studying learned Lie group generators. Advances Neural Information Processing Systems, 35:25212- 25223, 2022.
12

---

[32] Marina E Plissiti, Panagiotis Dimitrakopoulos, Giorgos Sfikas, Christophoros Nikou, O Krikoni, and Antonia Charchanti. SIPaKMeD: A new dataset for feature and image based classification of normal and pathological cervical cells in Pap smear images. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 3144-3148. IEEE, 2018
[33] Douglas A Reynolds. Gaussian mixture models. Encyclopedia of Biometrics, 741(659-663), 2009.
[34] Tammy Riklin-Raviv and Amnon Shashua. The quotient image: Class based recognition and synthesis under varying illumination conditions. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Patlern Recognition (Cat. No PR00149), volume 2, pages 566-571. IEEE, 1999.
[35] Sophia Sanborn, Christian A Shewmake, Bruno Olshausen, and Christopher J Hillar. Bispectral neural networks. In International Conference on Learning Representations, 2023.
[36] Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The Riemannian geometry of deep generative models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pages 315-323, 2018.
[37] Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34: 14:11293-12295 2021.
[38] Douglas Steinley. Properties of the Hubert-Arable adjusted Rand index. Psychological methods, 933):886 2004.
[39] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najañi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages
PMLR, 2019.
[40] Binxu Wang and Carlos R Ponce. A geometric analysis of deep generative image models and its applications. In International Conference on Learning Representations, 2021.
[41] Haitao Wang, Stan Li, and Yangsheng Wang. Generalized quotient image. In Proceedings of the 2004 IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II-II. IEEE, 2004.
[42] Junliang Wang, Chuqiao Xu, Zhengliang Yang, Jie Zhang, and Xiaoou Li. Deformable convolu- tional networks for efficient mixed-type wafer defect pattern recognition. IEEE Transactions on Semiconductor Manufacturing, 33(4):587-596, 2020.
[43] Rui Wang and Nan Chen. Wafer map defect pattern recognition using rotation-invarian features. IEEE Transactions on Semiconductor Manufacturing, (3()):9966004 2019.
[44] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 249-858,70 2018
[45] Tao Yang, Georgios Arvanitidis, Dongmei Fu, Xiaogang Li, and Søren Hauberg. Geodesic clustering in deep generative models. arXiv preprint arXiv:1800 04747, 2018
[46] Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative Lie group VAE for disentanglement learning. In International Conference on Machine Learning, pages 12924-12999 12934. PMLR, 2021.
13


Reviews:
[{"review_id": "BIJNcFDcJQ", "reviewer": "Reviewer_AAhr", "summary": "This paper proposes an autoencoder framework with additional features of (simultaneously) (1.) An invariance/equivariance of the latent space to a chosen group symmetry (2.) a near local isometry of the latent space. The authors build on the Quotient Autoencoder (QAE) framework that accounts for the group symmetry by modifying it into a probabilistic formulation of a variational autoencoder (called QVAE) and then further adding an isometry condition to the latent space by forcing the metric (computed via jacobians) to be close to a desirable form like identity - IQVAE. \n\nExperiments are demonstrated on the image datasets like Rotated MNIST, MixedWM38, and 3SIPaKMeD. Overall, the results are favorable and marginally outperform all conceptual baselines like AE, VAE, QAE, FMVAE, etc.  \n", "questions": "- More fundamentally, what is the difference between the isometry assumption applied to the decoder (like this paper) in comparison to the encoder? \n\n- What is the importance of the shuffle and augment in Algorithm 1?  ", "limitations": "I do not see any direct societal limitations. See Weaknesses for technical limitations. \n\nOverall, I am voting for a borderline reject at this point. The paper is compiled ok, and reasonably motivated - it is interesting to combine symmetry and isometry into an autoencoder framework. However, both the concept and the results do not lead to a strong take-home message, especially in the midst of the many AE frameworks proposed in prior work. \n\n", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper has been compiled well, with decent evaluations and visualizations demonstrating the positive aspects of the contributions of the IQVAE.\n- In particular, I found Table 2 and Figure 5 to be interesting experiments: showing performance with limited data and a more meaningful interpolation of the proposed IQVAE\n", "weaknesses": "- I do not get the impression that this paper makes any fundamentally new observation/contribution. The central premise is to combine already known components of QAE, VAE, and Iso-VAE [6, 22] etc - leading to the QVAE and IQVAE. More significantly, the results are decent but not to the extent of making a strong impression, thereby justifying the interest in investigating the combination. \n- An important reference must be included, compared, and commented on:   Gropp, Amos, Matan Atzmon, and Yaron Lipman. \"Isometric autoencoders.\" arXiv preprint arXiv:2006.09289 (2020).\n- I do not see IRVAE [22] being amongst the baseline evaluations. I understand the symmetry aspect is not a part of this paper. Still, it would be nice to clearly highlight the relative importance of symmetry, isometry, and probabilistic (i.e. like a VAE) components of the autoencoder approach - either implicitly or explicitly using an ablation study. Right now, despite the comparisons in Tables 1, 4, and 5 the message is not clear.  \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper, and the constructive comments. Below are our responses to your questions, along with the global response.\n\n***\n\n**Q1. Combination of already known components.**\n\nWe would like to emphasize that our approach of combining both symmetry and geometry in a unified manner is a natural yet principled direction. Previous works, however, have not paid sufficient attention to this integration. As we explained in Section 2 of the manuscript, symmetry-aware works (such as QAEs or CRVAEs) have not adequately considered the underlying geometric distance structure, while geometry-aware works (such as FMVAEs or IRVAEs) have not incorporated the underlying group symmetry. As a result, previous methods do not perform well even for simple rotated MNIST with SO(2) group symmetry. For instance, for the clustering, QAEs record ARI of 36.9, FMVAEs record ARI of 10.1, and the proposed IQVAEs record ARI of 70.9 (please see Table 1 in the manuscript for more details).\n\nWe identify a critical reason for the failure of previous methods, which stems from their insufficient attention to the complete structure-preserving representation of the data manifold. We believe that our proposed framework addresses this issue and offers a substantial step toward addressing more challenging data equipped with complicated geometry and group symmetry. \n\n***\n\n**Q2. Important reference Gropp et al.**\n\nThank you for bringing the important reference [1] to our attention. We fully agree that the reference is relevant to our work, as it deals with the isometry regularization for auto-encoders to preserve a meaningful distance relationship between samples. We appreciate your suggestions, and we will include this reference in Section 2.2 of the revised manuscript.\n\nThe core concept of [1] is similar to that of FMVAEs and IRVAEs, and it still has the limitation of not considering the underlying group symmetry of data. Nonetheless, [1] exhibits the following two intriguing features:\n\n(1) Unlike FMVAEs and IRVAEs, it takes into account the (pseudo) isometry of the encoder.\n\n(2) It proposes a stochastic objective that encourages Jacobian orthogonality (isometry) more cost-effectively.\n\nRegarding case (1), we will provide a related discussion in the response of Q4. For case (2), it can be applied from the perspective of enhancing the learning efficiency of IQVAE. We will discuss this matter in the revised manuscript.\n\n[1] Gropp et al. Isometric Autoencoders. arXiv 2020.\n\n***\n\n**Q3. IRVAE baseline.**\n\nIRVAEs, as briefly mentioned in Q2, share a limitation with FMVAEs, as they do not take into account the underlying group symmetry explicitly. Consequently, this may result in similar performance outcomes for both models. However, we fully agree with your comment that the comparison with IRVAEs are important for the completeness of the paper as pointed out. Therefore, we evaluated IRVAEs on the fully-rotated MNIST dataset as a preliminary example. As shown in the Table A of the attached PDF file in the global response, it is evident that IRVAEs show comparable performance with FMVAEs. We will include the evaluation results of IRVAEs, including the above experiment, in the revised manuscript.\n\n***\n\n**Q4. Isometry assumption applied to the decoder.**\n\nThe isometry assumption applied to the decoder is particularly important for IQVAEs, because it is desired to match the pullback metric of the latent space $\\mathcal{Z}$ with the (Euclidean) metric of the quotient space $\\mathcal{X}/G$, rather than that of the observation space $\\mathcal{X}$ itself, to prevent from estimating a non-zero distance between two identical images with different poses. It means that the smooth map between $\\mathcal{Z}$ and $\\mathcal{X}/G$ should be isometry. Note that the encoder and decoder parts of IQVAEs are maps $\\mathcal{X} \\to \\mathcal{Z}$ and $\\mathcal{Z} \\to \\mathcal{X}/G$, respectively. As a result, regularizing the decoder rather than encoder is natural for IQVAEs. We will add this discussion in the revised manuscript.\n\n***\n\n**Q5. Importance of the shuffle and augment.**\n\nPlease refer to our response to Q1 of the global rebuttal.\n"}, {"review_id": "trjehN3aJn", "reviewer": "Reviewer_kuEf", "summary": "The authors proposed isometric quotient VAEs (IQVAEs), to extract the quotient space and learn the Riemannian isometry. Empirical proof-of-concept experiments are conducted to demonstrate the performance of the method.", "questions": "1. The details of the downstream classification tasks are not clear enough. Line 260, ``tasks by using the support vector machine (SVM)\n261 [14] and random forest (RF) classifiers\". The author needs to provide the parameters so that others can reproduce the results.\n2. The authors need to analyze the results of experiments, such as Table 4 and Table 5.\n3. The format of references needs to be standardized.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "A new Auto-Encoders approach is proposed and its effectiveness is demonstrated to some extent.", "weaknesses": "However, some technical details are not clearly described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper, and the constructive comments. Below are our responses to your questions.\n\n***\n\n**Q1. Details of the downstream classification tasks.**\n\nFor the downstream classification model, we utilized the default hyper-parameters provided by scikit-learn. We will state this in the revised manuscript.\n\nWe further conducted additional evaluations by varying the regularization strength ($C$) for the SVM model and the number of trees ($n_{estimators}$) for the RF model, to check the influence of the hyper-parameters of downstream classifiers. As shown in Figure A of the attached PDF file in the global response, we found (1) the proposed QVAE and IQVAE consistently outperform the other baselines, and (2) classification accuracy remains relatively consistent throughout the hyper-parameter variations, especially for QVAE and IQVAEs. It further supports the efficiency of the proposed framework in terms of the robustness to the hyper-parameter variations. We will add this experimental result with related discussion in the revised manuscript.\n\n***\n\n**Q2. Analyzing the results of experiments.**\n\nTable 4 and Table 5 in our manuscript respectively summarizes the downstream task and OoD detection performances for MixedWM38. From line 329 to line 364, we analyzed the used dataset, experimental procedure, and evaluation results.\n\nWe would like to enhance the analysis of the experimental results according to your suggestion. If there are specific analysis results you would like to see, please do not hesitate to let us know. We value your input and strive to make our work as rigorous and impactful as possible.\n\n***\n\n**Q3. Format of references.**\n\nBased on your suggestion, we will standardize the format of references more consistently. We will use the format provided by the NeurIPS 2023 example paper.\n"}, {"review_id": "aYanBrufGe", "reviewer": "Reviewer_Va6B", "summary": "This paper introduces a new VAE with the modified objective; they call it improved Quotient Variational Auto-Encoder or IQVAE. This new structure-preserving VAE incorporates an additional regularization term, the Riemannian isometry loss, to optimize the decoder. This new term enables the decoder to preserve Riemannian isometry while mapping latent vectors into images. Working in an unsupervised manner, IQVAE offers enhanced flexibility for dealing with real-world image tasks such as classification and clustering. The effectiveness of the approach is demonstrated through experiments on various datasets and tasks, where some of them show improved results.", "questions": "\n-- The use of Riemannian geometry in the latent is fine but it will be better to relate to a larger question: What is the natural Riemannian structure of images associated with certain types of image sets? Say all views of an object, or all illumination images of an object under a single pose. Does this approach help answer such questions? ", "limitations": "\nThe paper discusses some limitations -- the choice of Euclidean metric in the image space and the knowledge of the symmetry group. Perhaps they can also discuss the issues arising from 3D rotations of the objects (not just the 2D rotations as used in the paper). ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n-- The intuition of the proposed method is well justified and holds promise. Unlike previous VAEs, it emphasizes preserving the structure between the latent space and image space, which has not been well studied in the previous literature.\n\n--  The mathematical concepts underlying Riemannian geometry are explained.\n\n--  Experiments are conducted across multiple datasets and tasks. When compared with six different baselines, some results show promising initial improvements.", "weaknesses": "\n--  The evaluation was conducted on the Rotated MNIST dataset, belonging to the rotation group $SO(2)$. I wonder whether this framework would also be effective for the $SO(3)$ group, which represents a more challenging space for generative models, especially in producing smooth interpolations.\n\n--  Figure 5(a) compares interpolation results. However, the differing start and end points across various methods raise questions about the fairness of the comparison.\n\n--  Some of the baseline VAEs used for comparison, such as AE, VAE, $\\beta$-VAE, are somewhat outdated, dating back to around 2016. Against recent baselines like CRVAE and QVAE, the proposed framework does not always give significant improvements, as shown in Tables 1 and 6 (with some results even worse than CRVAE or QVAE). Given that classification tasks are typically sensitive to model tuning, a minor increase in performance (1 percent) should not be considered as superior.\n\n-- The authors should consider comparisons with some of the latest baselines on Riemannian VAEs, like \\href{https://proceedings.neurips.cc/paper_files/paper/2022/file/7bf1dc45f850b8ae1b5a1dd4f475f8b6-Paper-Conference.pdf}{A Geometric Perspective on Variational Autoencoders}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper, and the constructive comments. Below are our responses to your questions, along with the global response.\n\n***\n\n**Q1. SO(3) group beyond SO(2).**\n\nWe would like to emphasize that we have conducted experiments with two datasets (MixedWM38 and SIPaKMeD) that exhibit both rotational and reflectional group symmetry in addition to the rotated MNIST which has SO(2) group symmetry. We will make the necessary revisions to our manuscript to clarify these points more explicitly.\n\nWe agree with your comment that considering more complicated 3-dimensional group actions such as SO(3) and SE(3) is an important direction relevant to our work. However, we would like to emphasize that our focus in this work is two-fold: (1) to formulate a novel framework that can consider both the symmetry and geometry of the data manifold simultaneously, i.e., structure-preserving approach, and (2) to validate the proposed framework for image dataset which naturally exhibit 2-dimensional group action.\n\nWe would like to mention that even for the simple rotated MNIST with SO(2), previous methods do not perform very well. For instance, for the clustering, CRVAEs record ARI of 26.1, QAEs record ARI of 36.9, while the proposed IQVAEs record ARI of 70.9. Please see Table 1 of our manuscript for more details. Our analysis reveals a critical reason for their failure, which stems from their insufficient attention to the complete structure-preserving representation of the data manifold. We also demonstrate that this limitation can be addressed using the proposed framework, and we believe that it is a substantial step toward addressing more challenging data equipped with complicated geometry and group symmetry.\n\nFollowing your comment, we will revise the manuscript to provide more details on our contribution and limitations regarding 3-dimensional group actions. We appreciate your feedback and thank you for bringing this important point to our attention.\n\n***\n\n**Q2. Interpolation comparison.**\n\nWe would like to mention that in Figure 5 all models interpolate between the same starting and ending samples (the first ‘1’ and ‘7’ samples, respectively). The reason they appear differently is that (1) VAEs do not learn the SO(2)-invariance, and (2) in the case of QVAEs and IQVAEs, they generates quotient samples with SO(2)-invariance, yet up to the canonical frame (i.e., angle). Nevertheless, we agree that this comparison, as you pointed out, is limited to a specific case and might not be entirely fair. Therefore, we conducted an additional experiment that can provide a more general evidence that the proposed IQVAEs indeed learn the isometric embedding and can interpolate between arbitrary samples smoothly (please refer to our response to Q7 of Reviewer nvnJ for detailed explanation between the isometry, volume preservation, and smooth interpolation). Please refer to our response to Q2 of the global rebuttal for the experimental setting and results.\n\n***\n\n**Q3. Minor increase in classification tasks.**\n\nWe would like to respectfully assert that the proposed IQVAE outperforms the baseline QAE, especially in terms of the fully unsupervised clustering tasks. As we mentioned above, the IQVAE outperforms the competitors, including QAE, by a significantly large margin for the fully-rotated MNIST example, as indicated by an ARI of 70.9 for the IQVAE + k-means, representing a margin of +34.0 over that of the QAE + k-means. Please see Table 1 in our manuscript for more details. In addition, our results show that the IQVAEs (or IQVAE-M) consistently exhibit better performances than competitors for real-world examples such as MixedWM38 and SIPaKMeD. For more details, please refer to Tables 1, 4 and 6 of our manuscript.\n\nRegarding the performance gain of IQVAEs for classification tasks, we agree that it may not be as significant as that for clustering tasks, especially where the sample size is sufficiently large. However, when the sample size is limited, the proposed IQVAEs outperform the QAE baseline even for the classification task with larger margin (~ 4%) as shown in Table 2 of our manuscript. This result has been acknowledged as impressive by Reviewer AAhr. We believe this evidence supports the superiority of our proposed IQVAE in the classification task as well.\n\n***\n\n**Q4. Important reference Chadebec et al.**\n\nThe reference you pointed out [1] is definitely relevant to our work, because it considers the Riemannian geometry of the VAE model. More specifically, [1] proposes a geometry-aware sampling technique that leverages the covariance matrix of the learned encoding distribution as a metric tensor for the latent space. By employing this sampling approach, the vanilla VAE demonstrates improved capability in generating more realistic and plausible samples with ease.\n\nDespite its relevance to our work, we would like to mention that comparison with [1] is not straightforward because they do not modify the learning objective or architecture of the vanilla VAE, and rather introduce the covariance-based Riemannian metric and corresponding sampling technique for the vanilla VAE. As a result, without considering the covariance metric tensor explicitly, the learned latent representation of [1] is identical to that of the vanilla VAEs. It should be noted that conventional downstream models such as k-means cannot consider such a specialized metric structure. Consequently, applying [1] directly to downstream tasks would yield same results with the vanilla VAE. Modulating the downstream model to accommodate special metrics can be possible, yet beyond the scope of the goal of this paper. We will discuss this matter in the revised manuscript.\n\n[1] Chadebec et al. A Geometric Perspective on Variational Autoencoders. NeurIPS 2022.\n\n***\n\n**Q5. Natural Riemannian structure.**\n\nPlease refer to our response to Q3 of the global rebuttal.\n"}, {"review_id": "8iGikZxNjB", "reviewer": "Reviewer_nvnJ", "summary": "This paper considers representation learning and generative models that can learn symmetry and isometric information about the data manifold. Some types of data, especially image data, can have semantic meanings that are invariant under some group of symmetry transformations. Past work has explored models that can learn representations that are invariant or equivariant under such symmetries. This paper proposes to also preserve distance information by learning a representation that is isometrically isomorphic to the data manifold. Two methods based on VAEs are discussed that purport to learn representations that respect symmetry and/or distance structure. These methods are studied in several experiments against notionally similar benchmarks.", "questions": "1. When computing Jacobians for IQVAE, is the approximation in Appendix A used, or are they computed exactly?\n\n2. In Figure 3 TSNE visualizations are shown, and the authors state that they show more \"meaningful representations\" for IQVAE. TSNE visalizations are inherently qualitative and their 2d nature means they do not capture much information from the relatively higher dimensional latent representations. Can the authors justify why they think TSNE shows that IQVAE learns more meaningful representations, and the mechanism by which TSNE could identify the isometrically isomorphic embeddings that IQVAE purportedly learns?\n\n3. In Figure 4 IQVAE is shown to have aligned reconstructions, but this would also be expected of other approaches that learn group invariances, including several of the paper's benchmarks. The difference of IQVAE to other approaches is in its distance preservation, which is not demonstrated in Figure 4. Can the authors give their rationale for showing only IQVAE here, or provide comparisons to the benchmarks?\n\n4. In Figure 5 linear interpolations are shown. Can the authors explain why they believe the symmetry or distance-aware representations from QVAE and IQVAE should lead to smooth interpolations, and why smooth interpolations are considered a good thing? The statement \"The IQVAE shows the smoothest interpolation ... owing to its isometry regularization.\" is not sufficiently explained by the theory.\n\n5. In the tables, what do the plus-minus numbers represent (std dev, std err, something else)?\n\n6. In the tables of results authors apply bolding on the best results inconsistently. In particular, many numbers are not bolded despite them being within statistical uncertainty of the best results. For instance in Table 1 left, column SVM, all of CRVAE, QAE, QVAE, and IQVAE are within error of the best result, but only QAE and IQVAE are bolded which seems misleading. Can the authors explain their decisions or correct this throughout the paper?\n\nMinor:\n\nIn L16, there is more recent research that the authors may consider referencing, which provides more evidence that natural image datasets agree with the manifold hypothesis:\n\n[A] Pope et al. 'The Intrinsic Dimension of Images and Its Impact on Learning' ICLR 2021\n\n[B] Brown et al. 'Verifying the Union of Manifolds Hypothesis for Image Data' ICLR 2023\n\nIn Eqn 6, does the F indicate Frobenius norm? This notation is not described.\n\nIn line 243, can the authors provide more explanation on what the \"mix-up vicinal distribution\" is and why it is related to isometries?\n\nThere are some grammatical errors that do not detract from the clarity of the work (non-exhaustive list):\n\nL8 rather -> rather than\n\nL53 rather than them of -> maybe \"rather than those of\"", "limitations": "Some limitations are discussed, but only briefly, and no real indication of how to surmount them is proposed. Broader impacts of the work are not discussed.\n\n---- \nSummary of Discussion: I have read all reviews and rebuttals for this submission.\n\nTo summarize the discussion, the authors have provided a considerable amount of additional evidence supporting their method in their rebuttals, especially around my main concern about IQVAEs learning isometric embeddings. I agreed with many of the points in Reviewer AAhr's review, especially their W1, and point that the \"concept and the results do not lead to a strong take-home message\" in the submitted version of the paper. The biggest outstanding concern of mine is then around novelty, see Weakness: Originality in my review.\n\nI believe that the authors will be able to improve their paper for a final submission, and make it more clear what the roles of symmetry, isometry, and probability are both in their methods but also experimental results.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Thank you to the authors for sharing their interesting research! The provided review of Riemannian geometry is useful, as some readers will not be familiar with these topics.\n\nOriginality: Previous work has studied symmetry in representations, and others have studied preserving distance information, but relatively little has been done to respect both aspects simultaneously. The IQVAE training method proposal to directly regularize the Jacobian $J^TJ$ term is interesting.\n\nQuality: The quality of background and method explanation is good. Relevant benchmarks are used in the experiments. Code and adequate reproducibility information is provided.\n\nClarity: The overall clarity of the paper is good.\n\nSignificance: The chosen topic, learning representations and generative models that more accurately reflect the manifold structure of data, is important and it is encouraging to see the authors' efforts in this space.", "weaknesses": "Originality: The QVAE loss function proposed in eqn 5 is a very straightforward combination of the VAE and QAE approaches.\n\nQuality: The evidence provided that IQVAE learns isometric embeddings is weak - most of the experiments rely on evaluation metrics on downstream tasks, and use higher performance as evidence that the method is working, but such results do not sufficiently address the claim that isometric embeddings are learned. The only evidence that directly targets distance information seems to be Figure 5b.\n\nClarity: Because some advanced mathematics is used, some readers may find the methods difficult to follow, but the authors have tried to make that section self-contained with relevant definitions. See questions below for clarifications on some experimental results.\n\nSignificance: Wide applicability of the approach is limited by the need to know the symmetry group of the data manifold a priori, and similarly for the metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper, and the constructive comments. Below are our responses to your questions, along with the global response.\n\n***\n\n**Q1. QVAE loss.**\n\nQVAEs, a probabilistic auto-encoder that can learn group invariance, might be seemed to be a simple extension of QAEs. However, it is essential to highlight their importance extends further as they serve as a bridge to reach IQVAEs, a novel approach that combines both symmetry and geometry in a unified manner. While it is a principled direction, as you pointed out, previous works have not paid sufficient attention to this integration.\n\n***\n\n**Q2. Evidence for isometry.**\n\nWe agree with your comment that there is no direct evidence that IQVAEs learn the isometric embedding, except for a specific case shown in Figure 5 which might not be entirely general. Therefore, we conducted an additional experiment that can provide stronger evidence that IQVAEs indeed learn isometry. Please refer to our response to Q2 of the global rebuttal for details.\n\n***\n\n**Q3. Limitation on known groups and metrics.**\n\nPlease refer to our responses to Q3 and Q4 of the global rebuttal.\n\n***\n\n**Q4. Jacobian approximation.**\n\nWe used the first-order approximation in Appendix A when estimating Jacobian. We will state this in the revised manuscript.\n\nAn important question is how the trade-off between the estimation quality of the Jacobian and the computational cost in the objective affects the performance of IQVAEs. To check this point, we conducted an experiment to evaluate the estimation quality of the Jacobian using the approximation. The result is shown in Figure D of the attached PDF file in the global response, where we present the R2 scores between the true pullback metric and its approximation as a function of the number of epochs. It demonstrates the approximation can effectively mimic the true metric with R2 greater than ~0.97.\n\nFurthermore, we compared the performance of IQVAEs trained with true and approximated Jacobians for the rotated MNIST dataset, as presented in Table B of the attached PDF. The results show that the first-order approximation is a good alternative to the true Jacobian. We will include these results in the revised manuscript.\n\n***\n\n**Q5. t-SNE.**\n\nWe consider the presence of a clustered label distribution on t-SNE as evidence of a meaningful representation due to two reasons: (1) the label information is invariant under SO(2) transformations, and (2) images with similar appearances are assigned identical labels, while those with dissimilar appearances receive distinct labels. Thus, the learned representation that preserves the symmetry and geometry is expected to exhibit well-clustered label distributions. As shown in the t-SNE plot of Figure 3, the t-SNE of IQVAEs shows a clear clustered structure according to digit labels, while VAEs do not. As a result, we conclude IQVAEs demonstrate more meaningful representations.\n\nNevertheless, we agree with your comment that t-SNE primarily preserves local distance information and may not capture the global features of the data. It raises the concern that t-SNE does not ensure IQVAEs learn the isometric embedding, as you correctly pointed out. Thus, we provide UMAP [1] visualization in Figure E of the attached PDF file in the global response, along with the quantitative evidence discussed in Q2. UMAP is known to better preserve the global structure of data compared to t-SNE. As shown in Figure E, IQVAE once again exhibits a distinct cluster structure compared to other baselines. We will add this result in the revised manuscript.\n\n[1] McInnes et al. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv 2018.\n\n***\n\n**Q6. Aligned reconstructions.**\n\nAs you pointed out, the aligned reconstruction is one of the advantages, yet not a unique feature of the IQVAEs. We will state it explicitly. \n\n***\n\n**Q7. Smooth interpolations.**\n\nIQVAEs learn the isometry, which means that the pullback metric tensor $\\mathbf{J}^T \\mathbf{J}$ in the latent space is regularized to be an identical matrix $\\mathbf{I}$ (up to constant $c^2$). It means that their volume measures are constant, as both theoretically ($\\sqrt{\\det{\\mathbf{J}^T \\mathbf{J}}} = c \\cdot \\sqrt{\\det{\\mathbf{I}}} = c$) and empirically (please refer to Q2) guaranteed. Thus, IQVAEs ensure that the decoding function from the latent space to the (quotient) observation space is volume-preserving. This means that there is no stretching or distortion of the embeddings, avoiding sudden changes in the images as the latent space varies gradually. Consequently, IQVAEs enable smooth interpolations of images between different latent points.\n\nThe smooth interpolation is important in our paper, because it serves as qualitative evidence for the volume-preserving property of IQVAEs. In addition, more generally in the context of generative models, the smooth interpolation might be good because it encourages the latent space continuity and semantic meaning preservation [2]. We will add this discussion in our revised manuscript.\n\n[2] Berthelot et al. Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer. ICLR 2019.\n\n***\n\n**Q8. Tables.**\n\nWe used standard deviation for all plus-minus numbers. We will state this in the revised manuscript. In addition, we will revise the tables to ensure consistent criteria for bolding.\n\n***\n\n**Q9. Important references.**\n\nThank you for bringing to our attention recent references [3, 4]. We will include them in our revised manuscript.\n\n[3] Pope et al. The Intrinsic Dimension of Images and Its Impact on Learning. ICLR 2021. \n\n[4] Brown et al. Verifying the Union of Manifolds Hypothesis for Image Data. ICLR 2023.\n\n***\n\n**Q10. Mix-up.**\n\nPlease refer to our response to Q1 of the global rebuttal.\n\n***\n\n**Q11. Others.**\n\nWe will add the definition of the Frobenious norm $|| \\cdot ||_{F}$ and correct typos.\n"}]

Author rebuttal:
Dear Reviewers,

We would like to express our heartfelt appreciation for taking the time to review our paper. Your insightful comments and suggestions have been tremendously valuable to us. Below are our responses to some of the frequently asked questions you've shared.

***

**Q1. Mix-up.**

We used the mix-up (shuffle and augment) technique in order to regularize the entire space of interest (i.e., the whole latent space $\mathcal{Z}$) to be isometrically isomorphic to the quotient space. By augmenting the latent space with the mix-up, one can smoothly fill the latent space where data is missing. It can be particularly important when the amount of available training samples is limited.

To support this claim, we conducted an additional experiment that evaluates the effect of the mix-up approach as shown in Figure B of the attached PDF file in the global response. We trained IQVAEs with and without the mix-up augmentation, and computed condition numbers of the pullback metric tensors ($\lambda_{max} / \lambda_{min}$, where $\lambda_{max}$ and $\lambda_{min}$ are respectively the maximum and minimum eigenvalues for the pullback metric $\mathbf{J}^T \mathbf{J}$, when $\mathbf{J}$ is Jacobian of the decoding function) and (mean-normalized) Riemannian volume measures ($\sqrt{\det{\mathbf{J}^T \mathbf{J}}}$) for the test samples. Condition numbers and normalized volume measures near 1.0 with small variances indicate the learned latent representation is isometrically isomorphic. We evaluated both cases with a training sample size of 60,000 and 10,000. 

As shown in Figure B, the mix-up approach helps the learned latent space of the IQVAE to be more isometric. In addition, the impact of the mix-up approach becomes more pronounced when the training sample size is reduced. We will add this experimental result with related discussion in the revised manuscript.

***

**Q2. Evidence for isometric embedding.**

To support our claim that IQVAEs learn the isometric embedding, we evaluated the condition numbers of the pullback metric tensors ($\lambda_{max} / \lambda_{min}$, where $\lambda_{max}$ and $\lambda_{min}$ are respectively the maximum and minimum eigenvalues for the pullback metric $\mathbf{J}^T \mathbf{J}$, when $\mathbf{J}$ is Jacobian of the decoding function) and (mean-normalized) Riemannian volume measures ($\sqrt{\det{\mathbf{J}^T \mathbf{J}}}$) for all test samples, across models. Condition numbers and normalized volume measures near 1.0 with small variances indicate the learned latent representation is isometrically isomorphic, thus maintains the volume-preservation.

As shown in Figure C in the attached PDF in the global response, IQVAEs show condition numbers and volume measures near 1.0, which indicate the learned latent representation is desired isometric one. This result can be considered as a generalized version of the analysis reported in Figure 5 (b) in the manuscript. We will add this experimental result in the revised manuscript.

***

**Q3. Natural Riemannian metric and beyond.**

The manifold hypothesis suggests that image data exists as a manifold embedded in a high-dimensional space. When the intrinsic dimensionality of the image manifold is much lower than the observation space, the Euclidean metric of the observation space can serve as a suitable Riemannian metric for the intrinsic geometry of the image manifold. It allows us to capture local distances and angles effectively. Utilizing the induced metric from the ambient Euclidean space facilitates the analysis and modeling of image manifolds within VAEs. It is the reason why we use the pullback metric as a metric tensor for the latent space.

However, when constructing a latent representation for solving a specific task effectively, it might be preferable to use a specialized metric tensor rather than the standard Euclidean metric of the ambient space. For example, when dealing with tasks that require capturing small variations in a local image patch, the pullback metric induced from the entire observation dimensions might not be the most effective geometry. Instead, it is more suitable to consider a specialized metric structure that can efficiently capture such local variations.

In this context, a task-specific metric can be derived using prior knowledge about the tasks. Alternatively, IQVAEs can learn this specialized metric in a semi-supervised manner with very limited labeled data, utilizing the concept of metric learning. This approach holds significant potential for future research, as it enables the model to adapt its representation to the specific requirements of the task, improving overall performance. We will discuss this matter more explicitly in the revised manuscript.

***

**Q4. Limitation on known groups.**

We assume the underlying group structure is known in advance, and represent such a group invariance with a Riemannian isometry by using the auto-encoding framework. On the other hand, there are some recent papers that to learn an unknown group from data. For example, [1] proposes a novel neural network that learns the bispectral invariants. [2] introduces Lie algebra to find underlying Lie group invariances. [3] solves the disentanglement problem with VAEs that can learn the latent space as a form of one-parameter subgroups of Lie groups. Combining these approaches with our proposed learning approach of IQVAEs would be a promising direction for future research. We will elaborate our limitation on known group and the future research direction more explicitly, in the revised manuscript.

[1] Sanborn et al. Bispectral Neural Networks. ICLR 2023.

[2] Moskalev et al. LieGG: Studying Learned Lie Group Generators. NeurIPS 2022.

[3] Zhu et al. Commutative Lie group VAE for disentanglement learning. ICML 2021.


Meta-review: The paper proposes a new variational auto-encoder training object for capturing data submanifold structures and invariance in the data such as rotations. The performance is tested in three different data sets and shows on-par or improved generation quality.

Initial reviews on the paper are diverging. The authors' response made a notable improvement on the paper and directly addressed criticisms raised on the first version (Reviewer AAhr, nvnJ). Importantly, the primary motivation of the invariance property is tested in the added experiments, such as measuring isometry with condition numbers.

The contribution in the paper is a combined effect: a design on the architecture and losses adapted from existing ones plus experimental results demonstrate some improved performance. To validate a fair contribution, careful revision should be made including articulating design novelties and incorporating the new experimental results.
Decision: Accept (poster)