Title: Isometric Quotient Variational Auto-Encoders for Structure-Preserving Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: representation learning

auto-encoders

geometry

symmetry
Abstract: We study structure-preserving low-dimensional representation of a data manifold embedded in a high-dimensional observation space based on variational auto-encoders (VAEs). We approach this by decomposing the data manifold $\mathcal{M}$ as $\mathcal{M} = \mathcal{M} / G \times G$, where $G$ and $\mathcal{M} / G$ are a group of symmetry transformations and a quotient space of $\mathcal{M}$ up to $G$, respectively. From this perspective, we define the structure-preserving representation of such a manifold as a latent space $\mathcal{Z}$ which is isometrically isomorphic (i.e., distance-preserving) to the quotient space $\mathcal{M} / G$ rather $\mathcal{M}$ (i.e., symmetry-preserving). To this end, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQVAEs), that can extract the quotient space from observations and learn the Riemannian isometry of the extracted quotient in an unsupervised manner. Empirical proof-of-concept experiments reveal that the proposed method can find a meaningful representation of the learned data and outperform other competitors for downstream tasks.

Isometric Quotient Variational Auto-Encoders for Structure-Preserving Representation Learning

Abstract

We study structure-preserving low-dimensional representation of a data manifold embedded in a high-dimensional observation space based on variational autoencoders (VAEs). We approach this by decomposing the data manifold M as M = M /G × G , where G and M /G are a group of symmetry transformations and a quotient space of M up to G , respectively. From this perspective, we define the structure-preserving representation of such a manifold as a latent space Z which is isometrically isomorphic (i.e., distance-preserving) to the quotient space M /G rather M (i.e., symmetry-preserving). To this end, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQVAEs) , that can extract the quotient space from observations and learn the Riemannian isometry of the extracted quotient in an unsupervised manner. Empirical proof-of-concept experiments reveal that the proposed method can find a meaningful representation of the learned data and outperform other competitors for downstream tasks.

1 Introduction

There has been a common consensus that natural image datasets form a low-dimensional manifold M embedded in a high-dimensional observation space R d , i.e., dim ( M ) ≪ d [6]. From this perspective, a good neural network is a mapping function that can recognize the underlying structure of the data manifold well [25]. A question that arises is what structures should be represented from the data.

For unsupervised learning task, especially in generative modeling, if we suppose the data manifold to be a Riemannian one, then preserving geometric structures in the representation space is a key consideration [1, 8, 25]. To make things more explicit, if Z is a latent representation of M , then Z should be isometrically isomorphic to M in the sense of the Riemannian isometry. This means that an infinitesimal distance on the data manifold should be preserved in the latent space as well, promoting geodesic distances in the latent space that accurately reflect those in the data space. Recently, several papers [8, 40, 25] have suggested using the concept of Riemannian isometry for deep generative models, such as variational auto-encoders (V AEs) [22] and generative adversarial networks (GANs) [14], to obtain more meaningful and relevant representations. These concepts have been applied to improve latent space interpolations [30] or clustering [45] of data manifolds.

However, the Riemannian isometry does not tell the whole story of the data structure, particularly in the case of vision datasets. One common property of natural images is that there are symmetry transformations that preserve the semantic meaning of the given image. For example, if one applies a rotational transformation to a certain image, then its semantic meaning, e.g., its label, does not change. In this situation, what one really wants to do is represent the inherent geometry of the manifold, i.e., the geometry up to such symmetry transformations [29].

The concept of the inherent geometry can be formalized by using the notion of the principal bundle M = M /G × G , a fiber bundle that consists of the group G of symmetry transformations and a quotient space M /G as the fiber and base spaces, respectively [16, 26]. In this case, the inherent geometry of M up to G is indeed determined by its quotient M /G solely; the quotient formulates an invariant structure of the data, thus a measure on M /G gives a measure on M that is invariant to the actions of G . Therefore, one should find the Riemannian isometry of M /G rather than M for a geometricmeaningful representation of the dataset (see Figure 1). Nevertheless, to the best of our knowledge, there has been a lack of studies, at least in the field of generative models, that integrate both the concepts of the quotient space and its Riemannian geometry.

In this paper, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQVAEs) , that can extract the quotient space from observations and learn its Riemannian isometry in an unsupervised manner, allowing for a cor-

G

G

Rd

2

Rn

dx

Z1

Z

Pe

X.

dxIG

XIG

a

+

5/Xp.0 = Zp

rect representation of the data manifold and accurate measurement of distances between samples from the learned representation. The proposed IQVAE model can be applied to a variety of practical tasks such as downstream clustering and out-of-distribution (OoD) detection. We evaluate the model's performance using three datasets: rotated MNIST [44], mixed-type wafer defect maps (MixedWM38) [42] and cervical cancer cell images (SIPaKMeD) [32]. In summary, our contribution is threefold:

We propose a novel approach for structure-preserving representation learning by viewing a data manifold as a principal bundle, and formulate it as an unsupervised learning task that finds a Riemannian isometry for the quotient of the data manifold.

We introduce a practical method, called the IQVAE, which can learn such a structurepreserving representation in an unsupervised manner by using the auto-encoding framework.

We demonstrate through experimental evaluations on various datasets that our proposed method can find useful representations and outperform other competing methods for downstream tasks such as classification, clustering, and OoD detection.

2 Related Works

2.1 G -Invariance and Quotient Space Learning

Quotient space learning is highly related to the notion of the G -invariance. Data augmentation is the most common approach to deal with the G -invariance of label distributions [18]. To achieve such a symmetry more explicitly, various G -invariant and G -equivariant architectures have been proposed; [10] has proposed G -convolution beyond the conventional translational equivariance of convolutional neural networks. [12] has generalized it further for Lie groups. G -orbit pooling, proposed by [23], can be applied to both arbitrary discrete groups and continuous groups by discretizing them. Some research has attempted to discover the underlying invariances within data [4, 46, 31, 35]. These approaches can be integrated as a major component of auto-encoders to represent the quotient symmetry of the data manifold, as we will discuss later in Section 3.3.

More relevant to our work, [37] has proposed using the consistency regularization for VAEs to encourage consistency in the latent variables of two identical images with different poses. It can be considered as the unsupervised learning version of the data augmentation and does not guarantee the explicit G -invariance of the regularized latent space. [5] has disentangled image content from pose variables, e.g., rotations or translations under the V AE framework. However, it has some limitations,

such as the requirement of using a specific decoder, the spatial transformer network [20]. It also has a problem that it cannot handle arbitrary groups beyond the rotation and translation. [11] has dealt with more complicated SO(3) group under the VAE framework. [29] has proposed the quotient auto-encoder (QAE) that can learn the quotient space of observations with respect to a given arbitrary group, extending early works in quotient images [34, 41]. However, it is a deterministic model and therefore cannot generate new samples easily. In addition, all the above-mentioned studies have not considered the Riemannian geometry of the learned latent representation.

2.2 Rimeannian Geometry of Generative Models

Recent studies have proposed to view the latent space of generative models as a Riemannian manifold [1, 9, 7]. In this perspective, the metric tensor of the latent space is given by a pullback metric induced from the observation space through the generator function. In line with this theoretical perspective, the flat manifold VAE (FMVAE) [8] and its variant [25, 15] have been proposed to regularize the generator function to be a Riemannian isometry by matching the pullback metric with the Euclidean one. The regularized latent space preserves geodesic distances of the observation space, making it a more geometrically meaningful representation that can be useful. However, the FMVAE and its variants do not take into account the underlying group symmetry, which can lead to inaccurate estimation of distances between data points. For example, they could estimate a non-zero distance between two identical images with different rotational angles, which is unsuitable for clustering tasks.

3 Method

Suppose x ∈ X ⊂ R d is an observation where X is an extrinsic view of M , i.e., the data manifold is realized via a mapping π : M → R d such that X = π ( M ) , and z ∈ Z ⊂ R n is its latent representation where n ≪ d . In addition, suppose G to be a group (of symmetry transformations) that its group elements g ∈ G naturally act on X by a left group action α : G ×X → X . We will denote α ( g, x ) as g ∗ x . Typical examples of G include the special orthogonal group SO(2), a group of 2-dimensional rotations that can also naturally act on d -dimensional images by rotating them.

3.1 Auto-Encoders

The auto-encoder framework consists of two parameterized neural networks, an encoder µ θ : R d → R n and a decoder µ ϕ : R n → R d . Vanilla auto-encoders find a low-dimensional compression of observations by minimizing the following reconstruction loss:

where p X ( x ) is the data distribution. The expectation over p X ( x ) can be computed via Monte Carlo (MC) estimation with finite samples X = { x i } N i =1 as E p X ( x ) [ L AE ( θ, ϕ ; x ) ] ≈ (1 /N ) ∑ N i =1 ∥ x i -µ ϕ ◦ µ θ ( x i ) ∥ 2 2 . Unless otherwise mentioned, we will omit the expectation over p X ( x ) for brevity.

Although sufficiently deep auto-encoders can provide effective latent compression µ θ ( X ) = Z ⊂ R n of high-dimensional data, they tend to overfit and the learned latent manifolds are often inaccurate [25], i.e., they cannot consider the underlying geometry of data. This can lead to problems such as incorrect latent interpolation or poor performance in downstream tasks, in addition to the absence of the ability to generate new samples.

3.2 Variational Auto-Encoders

The VAE [22] is a stochastic auto-encoding architecture belonging to the families of deep generative models. Contrary to vanilla auto-encoders, the VAE framework consists of two parameterized distributions q θ ( z | x ) and p ϕ ( x | z ) where the former and the latter are variational posterior and likelihood, respectively. VAEs try to maximize the marginal log-likelihood of observations by optimizing the following evidence lower bound:

where p Z ( z ) is the latent prior and D KL ( · , · ) is the Kullback-Leibler (KL) divergence. p Z ( z ) is often given by a normal distribution N ( z | 0 , I n ) where 0 is a zero vector and I n is n × n identity matrix.

For vanilla V AEs, q θ ( z | x ) is chosen as a multivariate Gaussian distribution with a diagonal covariance matrix N ( z | µ θ ( x ) , diag [ σ 2 θ ( x )]) and represented by a neural network encoder ( µ θ , σ θ ) : R d → R n × R n + . A latent variable is sampled by using the reparameterization trick z = µ θ ( x ) + σ θ ( x ) ⊙ ϵ where ϵ ∼ N ( ϵ | 0 , I n ) . Although p ϕ ( x | z ) is chosen depending on the modeling of the data, it is often taken as a simple distribution such as a Gaussian with fixed variance, N ( x | µ ϕ ( z ) , β I d ) , represented by a neural network decoder µ ϕ : R n → R d . In this case, the V AE objective (2) can be seen as a regularized version of (1) given by a sum of the stochastic auto-encoding reconstruction term L AE ( θ, ϕ ; x , ϵ ) = ∥ x -µ ϕ ( µ θ ( x ) + σ θ ( x ) ⊙ ϵ ) ∥ 2 2 and β -weighted KL divergence L KL ( θ ; x , β ) = βD KL ( N ( z | µ θ ( x ) , diag [ σ 2 θ ( x )]) ∥N ( z | 0 , I n )) as follows:

Practically, the expectation over N ( ϵ ) term is estimated via a single-point MC when the mini-batch size is sufficiently large [22]. When β → 0 and σ 2 θ → 0 , VAEs reduce to vanilla auto-encoders.

By minimizing (3), VAEs can learn the probabilistic process on the smooth latent space [13] that can easily generate plausible new samples by sampling a latent variable and decoding it, i.e., µ ϕ ( Z ) ≈ X . However, VAEs, like vanilla auto-encoders, do not ensure that the learned latent representation preserves the crucial information about the symmetry and geometry of the data manifold.

3.3 Quotient Auto-Encoders

QAE [29] is a modification of the deterministic auto-encoding framework that can find the quotient of the observation set X /G by replacing (1) as the following quotient reconstruction loss:

As shown in (4), QAEs aim to minimize the set distance between the auto-encoded sample µ ϕ ◦ µ θ ( x ) and G -orbit G ∗ x = { g ∗ x | g ∈ G } of the given x rather than x itself. It is noteworthy that G -orbit loses the information of G for a given x , which is the case of the natural quotient for x up to G . Thus, QAE can learn the quotient space of observations directly: for the ideal case of QAEs, i.e., when L QAE ( θ, ϕ ; x , G ) → 0 , the following property holds:

which means that the image µ ϕ ◦ µ θ ( X ) ≜ ˆ X does not contain any information on g . In addition, QAEs typically adopt G -invariant architectures such as G -orbit pooling [23] (see Section B of Supplementary Material (SM) for details) for the encoder part to explicitly guarantee the following G -invariant latent representation:

where µ G θ : R d → R n is the G -invariant encoder. It is worth mentioning that these encoder architectures are unable to learn a meaningful representation of the data when using a vanilla autoencoding framework, as the reconstruction loss (1) cannot be reduced enough; on the other hand, QAEs can use the explicitly G -invariant encoding architectures easily by taking advantage of the infimum in (4). This allows for both ˆ X and µ G θ ( X ) ≜ ˆ Z to possess the invariant property up to G .

3.4 Isometric Quotient Variational Auto-Encoders

Although QAEs with G -invariant encoders can extract the underlying symmetry of observations, they still have the following drawbacks. First, QAEs are deterministic and may overfit to limited observations, similar to vanilla auto-encoders. Second, the Riemannian geometry of the extracted quotient manifold may not be preserved in the latent space. To address these issues, we present a stochastic version of QAEs called QVAEs, and further improve them with a novel framework for isometric learning called IQVAEs.

QVAEs. Inspired by the connection among (1), (3), and (4), we propose the stochastic version of QAEs (QVAEs) which minimizes the following stochastic quotient objective:

where the first term is defined as L QAE ( θ, ϕ ; x , ϵ, G ) ≜ inf g ∈ G ∥ g ∗ x -µ ϕ ( µ G θ ( x ) + σ G θ ( x ) ⊙ ϵ ) ∥ 2 2 and the second term is L KL ( θ ; x , β, G ) ≜ βD KL ( N (ˆ z | µ G θ ( x ) , diag [ σ G θ ( x )] 2 ) ∥ p Z (ˆ z ) ) . We denote ( µ G θ , σ G θ ) : R d → R n × R n + for the G -invariant multivariate Gaussian encoder.

QVAEs are able to generate new samples from the quotient space, unlike QAEs. It means that QVAEs can approximate the quotient space of the data manifold X /G ≜ ˆ X as a form of the immersed manifold µ ϕ ( ˆ Z ) ≈ ˆ X where ˆ Z is a G -invariant latent representation. This leads us to introduce the concept of Riemannian isometry as a geometric regularization for QVAEs.

Isometric regularziation to quotient spaces. Before introducing the isometric regularization of the proposed QVAE, we review some basic concepts of the Riemannian geometry [2, 25].

Definition 3.1. (Riemannian manifolds) A n -dimensional Riemannian manifold X is a n -dimensional differentiable manifold that is equipped with a symmetric positive-definite metric tensor H X ( x ) ∈ R n × n ( n × n matrix). It defines a local smooth inner product on the tangent space T x X as ⟨ u x , v x ⟩ H X = u T x H X ( x ) v x with tangent vectors u x , v x ∈ T x X at each x ∈ X , allowing us to measure the distance and angles between tangent vectors at each point.

Definition 3.2. (Riemannian distances) Suppose X is a connected Riemannian manifold equipped with a Riemannian metric H X ( x ) at each point x ∈ X . A length of a curve on X between two points x 0 and x 1 is defined as:

where γ X : [0 , 1] → X is a curve on X that travels from x 0 = γ X (0) to x 1 = γ X (1) and ˙ γ X ( t ) = ∂ t γ X ( t ) ∈ T γ X ( t ) X is the velocity of the curve. The curve minimizing L is called a geodesic. The Riemannian distance is given by the length of the geodesic, i.e., d X ( x 0 , x 1 ) = inf γ X L [ γ X ] .

Definition 3.3. (Immersions) Let Z be an open set in R n . A smooth map µ : Z → R d ( n < d ) is called an immersion if its pushforward dµ z : T z Z → T µ ( z ) µ ( Z ) is injective for all z ∈ Z . Then, µ ( Z ) is called an immersed submanifold.

Definition 3.4. (Geometry of immersed submanifolds) Suppose µ : Z → R d is an immersion onto a submanifold X ⊂ R d . The Euclidean inner product in R d is a proper inner product of tangent vectors of X . Thus, X is a Riemannian manifold equipped with a Riemannian metric H X ( x ) = I d , i.e., ⟨ u x , v x ⟩ H X = u T x I d v x = u T x v x for tangent vectors u x , v x ∈ T x X at each x ∈ X .

Definition 3.5. (Pullback metric) Suppose µ : Z → R d is an immersion onto a Riemannian submanifold X ⊂ R d equipped with a Riemannian metric I d . Then, for tangent vectors u x , v x ∈ T x X at each x = µ ( z ) ∈ X = µ ( Z ) , the following holds:

Therefore, Z can be viewed as a Riemannian manifold equipped with H µ ( z ) called a pullback metric. In addition, because Z ⊂ R n and X ⊂ R d , the pushforward dµ z is equal to the standard Jacobian J µ ( z ) of µ , i.e., H µ ( z ) = J T µ ( z ) J µ ( z ) .

Definition 3.6. (Isometric immersion) Suppose Z is a Riemannian manifold equipped with a Riemannian metric H Z ( z ) at each z ∈ Z . A smooth immersion µ : Z → R d onto X ⊂ R d is a Riemannian isometry if the metric H Z ( z ) is equal to the pullback metric H µ ( z ) , i.e., H Z ( z ) = J T µ ( z ) J µ ( z ) for every z ∈ Z . In this case, the length of curves on X is equivalent with that on Z :

where γ Z : [0 , 1] →Z and γ X : [0 , 1] →X are respectively curves from z 0 = γ Z (0) to z 1 = γ Z (1) and from x 0 = γ X (0) to x 1 = γ X (1) , when x 0 = µ ( z 0 ) , x 1 = µ ( z 1 ) , ˙ γ Z ( t ) ∈ T γ Z ( t ) Z , and accordingly ˙ γ X ( t ) ∈ T µ ( γ Z ( t )) µ ( Z ) .

With some mild assumptions [36], a sufficiently well-trained decoder µ ϕ : ˆ Z → R d of the QVAE can be viewed as a smooth immersion onto the quotient of ˆ X = X /G . Then, from Definition 3.6,

the following regularization should be added to (5) to make the decoder be a Riemannian isometry between the latent representation ˆ Z and quotient ˆ X :

where J µ ϕ (ˆ z ) is Jacobian of µ ϕ at ˆ z , λ is a hyper-parameter, and H ˆ Z (ˆ z ) is a metric tensor of ˆ Z .

There are two things that should be clarified for practical computation of (6). The first aspect to consider is the selection of the Riemannian metric H ˆ Z (ˆ z ) . A commonly used and particularly useful choice of H ˆ Z (ˆ z ) is the scaled Euclidean metric in R n , i.e., H ˆ Z (ˆ z ) = C n = c 2 I n where c is a constant. This is because it is the case that using the Euclidean distance 1 on the latent representation d ˆ Z (ˆ z 0 , ˆ z 1 ) = √ (ˆ z 1 -ˆ z 0 ) T I n (ˆ z 1 -ˆ z 0 ) can preserve the geodesic distance on the quotient manifold d ˆ X (ˆ x 0 , ˆ x 1 ) up to c . The constant c can be regarded as either a pre-defined hyper-parameter or a learnable one. More generally, one can use any parameterized sym-

Algorithm 1 IQVAEs

Input: data { x i } N i =1 , hyper-parameters ( β, λ ), group G , G -invariant encoders ( µ G θ , σ G θ ), decoder µ ϕ Initialize θ , ϕ , C n while training do Sample { α i ∼ [0 , 1] } N i , { ϵ i ∼ N ( 0 , I ) } N i Compute { µ i θ , σ i θ } N i = { µ G θ ( x i ) , σ G θ ( x i ) } N i Sample { z i } N i =1 = { µ i θ + σ i θ ⊙ ϵ i } N i =1 Shuffle { z j } N j =1 = shuffle ( { z i } N i =1 ) Augment { ˜ z i } N i =1 = { (1 -α i ) z i + α i z j } i = j = N i = j =0 Compute L QAE = ∑ N i =1 min g ∈ G ∥ g ∗ x i -µ ϕ ( z i ) ∥ 2 2 Compute L KL = ∑ N i =1 D KL ( N ( µ i θ , diag [ σ i θ ] 2 ) ∥N ( 0 , I n )) Compute { J i µ ϕ } N i =1 = { J µ ϕ (˜ z i ) } N i Compute L ISO = ∑ N i =1 ∥ ( J i µ ϕ ) T J i µ ϕ -C n ∥ F Optimize ( L QAE + β L KL + λ L ISO ) /N w.r.t θ, ϕ end while metric positive-definite matrix C n for H ˆ Z (ˆ z ) . Note that such matrices can be achieved from an arbitrary learnable parameterized n × n matrix M n by using the Cholesky decomposition as follows:

where L n is a real lower triangular matrix with all positive diagonal entries and ε is a small nonzero constant. In this case, using the Mahalanobis distance on the latent space d ˆ Z (ˆ z 0 , ˆ z 1 ) = √ (ˆ z 1 -ˆ z 0 ) T C n (ˆ z 1 -ˆ z 0 ) preserves the geodesic distance of the quotient space of the observation manifold d ˆ (ˆ x 0 , ˆ x 1 ) .

X

The second aspect to consider is the sampling from q θ (ˆ z | x ) . It can be considered as Gaussian vicinal distribution for finite observation samples that smoothly fills the latent space where data is missing. In addition to that, following [8], we use the mix-up vicinal distribution [39] to effectively regularize the entire space of interest to be isometric. As a result, the tractable form of (6) is equal to 2 :

where f α θ ( x i,j , ϵ i,j ) = (1 -α )ˆ z i + α ˆ z j is the latent mix-up for ˆ z i,j = µ G θ ( x i,j ) + σ G θ ( x i,j ) ⊙ ϵ i,j . Practically, it is computed by sampling a mini-batch of latent variables, i.e., { ˆ z i } N i =1 = { µ G θ ( x i ) + σ G θ ( x i ) ⊙ ϵ i } N i =1 , shuffling it for { ˆ z j } N j =1 = shuffle ( { ˆ z j } N i =1 ) , and mixing-up them.

IQVAE. We define the IQVAE as a class of QVAEs whose objective function is given by the sum of the variational quotient objective L QVAE (5) and Riemannian isometry L ISO (8) as follows:

The proposed IQVAE optimization procedure is summarized in Algorithm 1 (see Section A of SM for additional tips). In the IQV AE, we typically set C n = c 2 I n . For clarity, we will refer to the IQV AE with (7) as IQVAE-M, to distinguish it from the basic version.

4 Experiments

We compared our proposed QVAE and IQVAE with six different competitors: the auto-encoder (AE), VAE, β -VAE [19], consistency regularized VAE (CRVAE) [37], QAE [29], and FMVAE [8]. We used the same convolutional architecture 3 , hyper-parameters, optimizer [21], and training scheme for

all models. Implementation details can be found in Section D of SM. After training, we encoded all samples and extracted their latent representations across models. Then, we conducted latent-based downstream clustering tasks by using the simple k -means and Gaussian mixture model (GMM) [33]. Similarly, we conducted downstream classification tasks by using the support vector machine (SVM) [17] and random forest (RF) classifiers. We used the same hyper-parameters for downstream models regardless of the benchmarked latent representations. Then, we computed the adjusted Rand index (ARI) [38] and test classification accuracy for clustering and classification models, respectively.

4.1 Rotated MNIST

Rotated MNIST consists of randomly rotated images of handwritten digits. It has the rotational group symmetry of G = SO(2). We padded the original MNIST images to size 32 × 32 for improved usability. Then, we generated φ -rotated MNIST by randomly rotating each sample in the MNIST with a uniform distribution of [ -φ/ 2 , φ/ 2] . We considered both φ = 2 π/ 3 and φ = 2 π cases (see Section C of SM for examples). We used 60,000 samples for training and 10,000 samples for testing.

Basic evaluation. For quantitative analysis, we summarize downstream task performances across the models in Table 1. It clearly shows the proposed QVAE and IQVAE outperform other competitors by a large margin because they consider the group symmetry explicitly. Moreover, the IQVAE shows greatly improved clustering performances even over the QVAE when φ = 2 π . Wededuce that the Riemannian isometry helps the IQVAE recognize a meaning-

0

970

12

2

3

0

0

949

13

2

11

1

0

1113

1

1

2

0

Q

12

6

0

1

0

127

2

1

0

0

3

2

２３4５

16

8

909

6

2

3

34

20

1

2

-

7

23

855

8

44

2

0

19

63

11

2

3

16

744

2

13

2

4

222

2

3

1

11

5

852

41

7

5

80

8

Actual

0

0

0

1

814

6

46

30

22

60

4

0

28

1

1

822

18

2

4

6

100

5

0

0

20

16

755

33

15

33

15

5

-

2

11

2

68

7

714

34

16

15

23

6

44

2

5

9

1

33

588

66

6

171

6

一

4

47

0

18

0

1

801

23

1

63

7

1

14

16

4

17

2

12

941

4

17

7

3

41

8

0

31

5

2

873

3

62

8

1

6

1

4

13

12

10

3

921

3

8

2

9

3

59

18

17

0

0

842

24

9

6

0

0

2

6

16

385

1617

558

9

6

20

0

10

7

12

30

2

11

116

0

1

2

345

6

7

8

6

0

1

2

3

45

6

7

8

6

Predicted

Predicted

ful distance relationship between controversial samples, e.g., digits 6 and 9, as depicted in Figure 2. For qualitative analysis, we also visualize the learned latent representations for the test dataset of the rotated MNIST by using the uniform manifold approximation and projection (UMAP) [28] (see Figure 3, and Section E of SM for all models). It further supports that the IQV AE succeeds in finding meaningful representations of the dataset.

Aligned reconstructions. Figure 4 shows six images of digit 7 and their reconstructions based on the proposed IQVAE demonstrating that the IQVAE can reconstruct rotated digits in the aligned frame thanks to learning the quotient up to rotations.

Original

Reconst.

Better interpolation. Figure 5 compares linear interpolations between digits 1 and 7 for the V AE, QVAE, and IQVAE. It is conducted by parameterizing a linear path as z ( t ) = z (0) + ( z (10) -z (0)) · ( t/ 10) and then reconstructing x ( t ) = µ ϕ ( z ( t )) for t ∈ { 0 , 1 , ..., 10 } . As shown in Figure 5 (a), the VAE shows a meaningless interpolation from t = 5 to t = 9 (see Section F of SM for other examples).

(a)

(b)

VAE

QVAE

measure

600

IQVAE

QVAE

1

1

1

1111

400

Volume

200

IQVAE

0

0

2

4

6

8

10

t=0

t = 1

t =2

t=3

t = 4

t =5

t = 7

t=8

t=9

t =10

t

While the QVAE shows a more convincing interpolation compared to the VAE, it also shows abrupt changes at t = 7 and t = 8 . The IQVAE shows the smoothest interpolation between digits 1 and 7 compared to the other models owing to the isometry regularization. The smooth interpolation of the IQVAE is also quantitatively confirmed in Figure 5 (b) which compares the geometric volume measures along each linear path of the QVAE and IQVAE. The volume measure is computed via √ det J T µ ϕ ( z ( t )) J µ ϕ ( z ( t )) and can be viewed as a local infinitesimal volume ratio between the latent and observation spaces. The volume measure of the QVAE shows a clear peak near t = 8 , while that of the IQVAE is nearly constant.

Evidence for isometry. To demonstrate that the learned decoding function of IQVAEs upholds the desired Riemannian isometry, we evaluated the condition numbers of the pullback metric tensors and the Riemannian volume measures for all test samples across models. The condition number is defined as the ratio λ max /λ min, where λ max and λ min respectively represent the maximum and minimum eigenvalues of the pull-back metric, denoted as J T µ ϕ ( z ) J µ ϕ ( z ) . All computed volume measures

Normalized volumemeasure

30

2.0

Conditionnumber

1.5

20

1.0

10

0.5

0.0

VAE

QVAE

IQVAE

VAE

QVAE

IQVAE

are normalized by their average values. Therefore, condition numbers and normalized volume measures approaching 1.0, accompanied by minimal variances, indicate that the learned latent representation is isometrically isomorphic to Euclidean space. In other words, the learned pull-back metric aligns with the Euclidean metric. As shown in Figure 6, the IQV AE presents condition numbers and volume measures close to 1.0, exhibiting trivial variances compared to other models.

Sample efficiency comparison. Table 2 compares the sample efficiency of IQVAE and the most comparable baseline, QAE, in terms of classification accuracy. The results demonstrate the effectiveness of the proposed method as IQVAE shows more robust performance with respect to variations in the training sample size.

Test-time orbit augmentation. The current QVAE and IQVAE require more computational cost than vanilla V AE due to the G -orbit pooling encoders, which augment the number of observations | G | times larger by expanding a data point x as a G -orbit G ∗ x . It might be a bottleneck, especially when training the proposed model with larger-scale datasets. To resolve this issue, we suggest the test-time orbit augmentation strategy, which involves using a coarse step size when discretizing the group parameters (e.g., rotation angle) during training, and

then switching to a finer step size during the inference phase. Table 3 compares wall-clock training

times and downstream task performances for 2 π -rotated MNIST datasets for the VAE, QVAE, and IQVAE. The former and latter numbers in parentheses for QVAEs and IQVAEs in the table indicate discretized group sizes used in the training and inference phases respectively. The results show that the proposed technique can scale the training time without severe performance degradation.

Effect of mix-up. We employed the mix-up technique (8) to regularize the entire latent space. By augmenting the latent space with the mix-up, it is possible to smoothly fill gaps in the latent space regions where data might be missing. This becomes particularly crucial when the number of available training samples is limited. To validate this, we trained IQVAEs both with and without the mix-up technique in two scenarios: one with a training sample size of 60,000 and the other with 10,000. Subsequently, we computed the condition numbers of the pull-back metric tensors and mean-normalized volume measures

for test samples. Figure 7 shows that using the mix-up reduces the variances of both condition numbers and volume measures for pull-back metric tensors, demonstrating its efficacy in regularizing the entire latent space. Furthermore, the impact of the mix-up approach becomes more noticeable when the training sample size is smaller.

4.2 MixedWM38

MixedWM38 is a dataset containing 38,015 52 × 52 semiconductor wafer maps, separated into 38 defect categories including 1 normal type, 8 single-defect types, and 29 mixed-defect types [42] (see Section C of SM for examples). We assume rotation and reflection (i.e., G = O(2)) symmetries are present in the MixedWM38. We used 30,720 samples for training and 7,295 samples for testing.

Basic evaluation. We performed an experiment on the MixedWM38 dataset, similar to the

one previously conducted on rotated MNIST. Unlike the rotated MNIST, the MixedWM38 dataset requires a multi-label classification task, which is a key difference between the two experiments. Thus, we use the F-score as a classification accuracy for this problem. Table 4 summarizes the downstream task performances of eight models on the MixedWM38 dataset (see Section E of SM for UMAP visualizations). As shown in the table, the proposed methods have a higher performance on the downstream task than the other models.

OoD evaluation. We further demonstrate the effectiveness of the proposed IQVAE in detecting OoD samples, an important task in the semiconductor engineering field. In order to mimic a real-world setting, we divided the 7,015 single-defect type data (8 classes: C, D, EL, ER, L, NF, S, R; see Section C of SM) into two sets: C-EL-L-S as the in-distribution set, and D-ERNF-R as the OoD set. Then, we randomly rotated all samples in the C-EL-L-S subset, and considered the rotated samples an in-distribution set, aligning with the nature of the semiconductor manufacturing process [43]. Using auto-encoding models trained on the original C-EL-L-S data, we formulated an OoD detection task, the goal of which is to differentiate between samples from the rotated C-EL-L-S set (in-distribution) and

samples from the D-ER-NF-R set (OoD). We assessed the standard distance-based OoD detection: it labels an instance as OoD if its latent vector is farther than a pre-defined threshold from any

class-conditional mean latent vector of learned in-distribution sets, in terms of the Mahalanobis distance [24] (see Section G of SM for details). Table 5 shows the comparison of competing models using standard metrics (AUCROC and AUPRC) for threshold-based OoD detection, demonstrating that the proposed IQVAE excels in the OoD tasks as it accurately learns the data structure.

4.3 SIPaKMeD

The SIPaKMeD dataset includes 4,049 singlecell Pap smear images for cervical cancer diagnosis and is split into 5 classes [32] (see Section C of SM for examples). We assume O(2) symmetry for the SIPaKMeD dataset. We resized the original SIPaKMeD images to size 32 × 32 for better usability. In accordance with the original data splitting, 3,549 samples were used for training and 500 samples for testing.

IQVAE-M. As we did with rotated MNIST, we conducted the same experiment on the SIPaKMeD dataset. Furthermore, we also as-

sessed the proposed IQVAE-M that incorporates a learnable flexible Riemannian metric as described in (7) and utilizes the latent Mahalanobis distance when computing the radial basis function (RBF) kernel K : R n × R n → R of SVMs as K ( z 0 , z 1 ) = exp ( -( z 1 -z 0 ) T C n ( z 1 -z 0 ) ) . To quantitatively analyze the learned representations, we present the downstream task performance of the competing models on the SIPaKMeD dataset in Table 6 (see Section E of SM for UMAP visualizations). As shown in Table 6, the proposed methods exhibit superior performance compared to other models 4 .

5 Conclusion and Limitation

We have proposed and demonstrated the effectiveness of IQVAE, a simple yet effective approach that maintains symmetry and geometry in data. However, our work has two main limitations as follows.

Predefined groups of symmetry transformations. We assume the group structure of a given dataset is known in advance. This group invariance is represented using the quotient auto-encoding framework with a Riemannian isometry. However, several recent papers have delved into learning an unknown group directly from data. For example, [35] presents a novel neural network that identifies bispectral invariants. [31] employs Lie algebra to find underlying Lie group invariances. [46] tackles the disentanglement challenge with VAEs, aiming to learn the latent space in terms of one-parameter subgroups of Lie groups. Integrating these methods with our IQVAEs learning approach could be a promising direction for future research.

Euclidean metric tensors. We focus on cases where the metric tensor of the observation space is Euclidean. If the intrinsic dimensionality of the image manifold is significantly lower than that of the ambient observation space, the Euclidean metric of the observation space can well serve as a Riemannian metric for the intrinsic geometry of the image manifold. This reasoning supports our use of the pullback metric as a metric tensor for the latent space. However, when constructing a latent representation that effectively addresses a specific task, a specialized metric tensor might be more appropriate than the standard Euclidean metric of the ambient space. For instance, when tasks involve capturing subtle variations in a local image patch, the pullback metric derived from the entire observation dimensions may not provide the most efficient geometry. In such cases, a specialized metric structure better suited for capturing these local variations should be considered. In this context, a task-specific metric can be determined using prior knowledge about the tasks [27]. Alternatively, IQVAEs can semi-supervisedly learn this specialized metric with minimal labeled data, leveraging metric learning concepts [3]. This approach holds significant potential for future research, as it allows the model to tailor its representation to the specific requirements of the task, thereby improving overall performance.

Acknowledgments and Disclosure of Funding

Changwook Jeong sincerely acknowledges the support from Samsung Electronics Co., Ltd (IO22102703271-01, IO230220-05060-01) and is further supported by the Technology Innovation Program (20020803, RS-2023-00231956) and Korea Institute for Advancement of Technology (KIAT) grant (P0023703), both funded by the MOTIE, Korea (1415180307, 1415187475), the National Research Foundation of Korea (NRF) grant (RS-2023-00257666) funded by the MSIT, Korea, the IC Design Education Center (IDEC), Korea, and the research project funds (1.220125.01, 1.230063.01) and supercomputing center of UNIST.

References

Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations , 2018.

Georgios Arvanitidis, Soren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. In International Conference on Artificial Intelligence and Statistics , pages 631-639. PMLR, 2021.

Aurélien Bellet, Amaury Habrard, and Marc Sebban. Metric learning . Springer, 2022.

Gregory Benton, Marc Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems , 33:17605-17616, 2020.

Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentangling image content from translation and rotation with spatial-V AE. Advances in Neural Information Processing Systems , 32, 2019.

Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disentanglement perspective. IEEE Transactions on Neural Networks and Learning Systems , 27(10):1997-2008, 2015.

Clément Chadebec and Stéphanie Allassonnière. A geometric perspective on variational autoencoders. arXiv preprint arXiv:2209.07370 , 2022.

Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with V AEs. In International Conference on Machine Learning , pages 1587-1596. PMLR, 2020.

Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. In International Conference on Artificial Intelligence and Statistics , pages 1540-1550. PMLR, 2018.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning , pages 2990-2999. PMLR, 2016.

Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding. In ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models , 2021.

Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to Lie groups on arbitrary continuous data. In International Conference on Machine Learning , pages 3165-3176. PMLR, 2020.

Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations , 2020.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Amos Gropp, Matan Atzmon, and Yaron Lipman. Isometric autoencoders. arXiv preprint arXiv:2006.09289 , 2020.

Jihun Ham and Daniel D Lee. Separating pose and expression in face images: a manifold learning approach. Neural Information Processing-Letters and Reviews , 11(4):91-100, 2007.

Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications , 13(4):18-28, 1998.

AHernandez-Garcia, P König, and TC Kietzmann. Learning robust visual representations using data augmentation invariance. In 2019 Conference on Cognitive Computational Neuroscience , pages 44-47. [Sl: sn], 2019.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. β -VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , 2017.

Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems , 28, 2015.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 , 2013.

Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Marc Pollefeys. Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 289-297, 2016.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems , 31, 2018.

Yonghyeon Lee, Sangwoong Yoon, Minjun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. In International Conference on Learning Representations , 2022.

Jongwoo Lim, Jeffrey Ho, Ming-Hsuan Yang, Kuang-chih Lee, and David Kriegman. Image clustering with metric, local linear structure, and affine symmetry. In European Conference on Computer Vision , pages 456-468. Springer, 2004.

Laura Manduchi, Kieran Chin-Cheong, Holger Michel, Sven Wellmann, and Julia Vogt. Deep conditional Gaussian mixture model for constrained clustering. Advances in Neural Information Processing Systems , 34:11303-11314, 2021.

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software , 3(29):861, 2018.

Eloi Mehr, André Lieutier, Fernando Sanchez Bermudez, Vincent Guitteny, Nicolas Thome, and Matthieu Cord. Manifold learning in quotient spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9165-9174, 2018.

Mike Yan Michelis and Quentin Becker. On linear interpolation in the latent space of deep generative models. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning , 2021.

Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold Smeulders. LieGG: Studying learned Lie group generators. Advances in Neural Information Processing Systems , 35:2521225223, 2022.

Marina E Plissiti, Panagiotis Dimitrakopoulos, Giorgos Sfikas, Christophoros Nikou, O Krikoni, and Antonia Charchanti. SIPaKMeD: A new dataset for feature and image based classification of normal and pathological cervical cells in Pap smear images. In 2018 25th IEEE International Conference on Image Processing (ICIP) , pages 3144-3148. IEEE, 2018.

Douglas A Reynolds. Gaussian mixture models. Encyclopedia of Biometrics , 741(659-663), 2009.

Tammy Riklin-Raviv and Amnon Shashua. The quotient image: Class based recognition and synthesis under varying illumination conditions. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149) , volume 2, pages 566-571. IEEE, 1999.

Sophia Sanborn, Christian A Shewmake, Bruno Olshausen, and Christopher J Hillar. Bispectral neural networks. In International Conference on Learning Representations , 2023.

Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The Riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 315-323, 2018.

Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems , 34:12943-12954, 2021.

Douglas Steinley. Properties of the Hubert-Arable adjusted Rand index. Psychological methods , 9(3):386, 2004.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David LopezPaz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , pages 6438-6447. PMLR, 2019.

Binxu Wang and Carlos R Ponce. A geometric analysis of deep generative image models and its applications. In International Conference on Learning Representations , 2021.

Haitao Wang, Stan Z Li, and Yangsheng Wang. Generalized quotient image. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. , volume 2, pages II-II. IEEE, 2004.

Junliang Wang, Chuqiao Xu, Zhengliang Yang, Jie Zhang, and Xiaoou Li. Deformable convolutional networks for efficient mixed-type wafer defect pattern recognition. IEEE Transactions on Semiconductor Manufacturing , 33(4):587-596, 2020.

Rui Wang and Nan Chen. Wafer map defect pattern recognition using rotation-invariant features. IEEE Transactions on Semiconductor Manufacturing , 32(4):596-604, 2019.

Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 849-858, 2018.

Tao Yang, Georgios Arvanitidis, Dongmei Fu, Xiaogang Li, and Søren Hauberg. Geodesic clustering in deep generative models. arXiv preprint arXiv:1809.04747 , 2018.

Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative Lie group VAE for disentanglement learning. In International Conference on Machine Learning , pages 12924-12934. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers representation learning and generative models that can learn symmetry and isometric information about the data manifold. Some types of data, especially image data, can have semantic meanings that are invariant under some group of symmetry transformations. Past work has explored models that can learn representations that are invariant or equivariant under such symmetries. This paper proposes to also preserve distance information by learning a representation that is isometrically isomorphic to the data manifold. Two methods based on VAEs are discussed that purport to learn representations that respect symmetry and/or distance structure. These methods are studied in several experiments against notionally similar benchmarks.\n\nSTRENGTHS:\nThank you to the authors for sharing their interesting research! The provided review of Riemannian geometry is useful, as some readers will not be familiar with these topics.\n\nOriginality: Previous work has studied symmetry in representations, and others have studied preserving distance information, but relatively little has been done to respect both aspects simultaneously. The IQVAE training method proposal to directly regularize the Jacobian $J^TJ$ term is interesting.\n\nQuality: The quality of background and method explanation is good. Relevant benchmarks are used in the experiments. Code and adequate reproducibility information is provided.\n\nClarity: The overall clarity of the paper is good.\n\nSignificance: The chosen topic, learning representations and generative models that more accurately reflect the manifold structure of data, is important and it is encouraging to see the authors' efforts in this space.\n\nWEAKNESSES:\nOriginality: The QVAE loss function proposed in eqn 5 is a very straightforward combination of the VAE and QAE approaches.\n\nQuality: The evidence provided that IQVAE learns isometric embeddings is weak - most of the experiments rely on evaluation metrics on downstream tasks, and use higher performance as evidence that the method is working, but such results do not sufficiently address the claim that isometric embeddings are learned. The only evidence that directly targets distance information seems to be Figure 5b.\n\nClarity: Because some advanced mathematics is used, some readers may find the methods difficult to follow, but the authors have tried to make that section self-contained with relevant definitions. See questions below for clarifications on some experimental results.\n\nSignificance: Wide applicability of the approach is limited by the need to know the symmetry group of the data manifold a priori, and similarly for the metric.\n\nQUESTIONS:\n1. When computing Jacobians for IQVAE, is the approximation in Appendix A used, or are they computed exactly?\n\n2. In Figure 3 TSNE visualizations are shown, and the authors state that they show more \"meaningful representations\" for IQVAE. TSNE visalizations are inherently qualitative and their 2d nature means they do not capture much information from the relatively higher dimensional latent representations. Can the authors justify why they think TSNE shows that IQVAE learns more meaningful representations, and the mechanism by which TSNE could identify the isometrically isomorphic embeddings that IQVAE purportedly learns?\n\n3. In Figure 4 IQVAE is shown to have aligned reconstructions, but this would also be expected of other approaches that learn group invariances, including several of the paper's benchmarks. The difference of IQVAE to other approaches is in its distance preservation, which is not demonstrated in Figure 4. Can the authors give their rationale for showing only IQVAE here, or provide comparisons to the benchmarks?\n\n4. In Figure 5 linear interpolations are shown. Can the authors explain why they believe the symmetry or distance-aware representations from QVAE and IQVAE should lead to smooth interpolations, and why smooth interpolations are considered a good thing? The statement \"The IQVAE shows the smoothest interpolation ... owing to its isometry regularization.\" is not sufficiently explained by the theory.\n\n5. In the tables, what do the plus-minus numbers represent (std dev, std err, something else)?\n\n6. In the tables of results authors apply bolding on the best results inconsistently. In particular, many numbers are not bolded despite them being within statistical uncertainty of the best results. For instance in Table 1 left, column SVM, all of CRVAE, QAE, QVAE, and IQVAE are within error of the best result, but only QAE and IQVAE are bolded which seems misleading. Can the authors explain their decisions or correct this throughout the paper?\n\nMinor:\n\nIn L16, there is more recent research that the authors may consider referencing, which provides more evidence that natural image datasets agree with the manifold hypothesis:\n\n[A] Pope et al. 'The Intrinsic Dimension of Images and Its Impact on Learning' ICLR 2021\n\n[B] Brown et al. 'Verifying the Union of Manifolds Hypothesis for Image Data' ICLR 2023\n\nIn Eqn 6, does the F indicate Frobenius norm? This notation is not described.\n\nIn line 243, can the authors provide more explanation on what the \"mix-up vicinal distribution\" is and why it is related to isometries?\n\nThere are some grammatical errors that do not detract from the clarity of the work (non-exhaustive list):\n\nL8 rather -> rather than\n\nL53 rather than them of -> maybe \"rather than those of\"\n\nLIMITATIONS:\nSome limitations are discussed, but only briefly, and no real indication of how to surmount them is proposed. Broader impacts of the work are not discussed.\n\n---- \nSummary of Discussion: I have read all reviews and rebuttals for this submission.\n\nTo summarize the discussion, the authors have provided a considerable amount of additional evidence supporting their method in their rebuttals, especially around my main concern about IQVAEs learning isometric embeddings. I agreed with many of the points in Reviewer AAhr's review, especially their W1, and point that the \"concept and the results do not lead to a strong take-home message\" in the submitted version of the paper. The biggest outstanding concern of mine is then around novelty, see Weakness: Originality in my review.\n\nI believe that the authors will be able to improve their paper for a final submission, and make it more clear what the roles of symmetry, isometry, and probability are both in their methods but also experimental results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new VAE with the modified objective; they call it improved Quotient Variational Auto-Encoder or IQVAE. This new structure-preserving VAE incorporates an additional regularization term, the Riemannian isometry loss, to optimize the decoder. This new term enables the decoder to preserve Riemannian isometry while mapping latent vectors into images. Working in an unsupervised manner, IQVAE offers enhanced flexibility for dealing with real-world image tasks such as classification and clustering. The effectiveness of the approach is demonstrated through experiments on various datasets and tasks, where some of them show improved results.\n\nSTRENGTHS:\n-- The intuition of the proposed method is well justified and holds promise. Unlike previous VAEs, it emphasizes preserving the structure between the latent space and image space, which has not been well studied in the previous literature.\n\n--  The mathematical concepts underlying Riemannian geometry are explained.\n\n--  Experiments are conducted across multiple datasets and tasks. When compared with six different baselines, some results show promising initial improvements.\n\nWEAKNESSES:\n--  The evaluation was conducted on the Rotated MNIST dataset, belonging to the rotation group $SO(2)$. I wonder whether this framework would also be effective for the $SO(3)$ group, which represents a more challenging space for generative models, especially in producing smooth interpolations.\n\n--  Figure 5(a) compares interpolation results. However, the differing start and end points across various methods raise questions about the fairness of the comparison.\n\n--  Some of the baseline VAEs used for comparison, such as AE, VAE, $\\beta$-VAE, are somewhat outdated, dating back to around 2016. Against recent baselines like CRVAE and QVAE, the proposed framework does not always give significant improvements, as shown in Tables 1 and 6 (with some results even worse than CRVAE or QVAE). Given that classification tasks are typically sensitive to model tuning, a minor increase in performance (1 percent) should not be considered as superior.\n\n-- The authors should consider comparisons with some of the latest baselines on Riemannian VAEs, like \\href{https://proceedings.neurips.cc/paper_files/paper/2022/file/7bf1dc45f850b8ae1b5a1dd4f475f8b6-Paper-Conference.pdf}{A Geometric Perspective on Variational Autoencoders}\n\nQUESTIONS:\n-- The use of Riemannian geometry in the latent is fine but it will be better to relate to a larger question: What is the natural Riemannian structure of images associated with certain types of image sets? Say all views of an object, or all illumination images of an object under a single pose. Does this approach help answer such questions?\n\nLIMITATIONS:\nThe paper discusses some limitations -- the choice of Euclidean metric in the image space and the knowledge of the symmetry group. Perhaps they can also discuss the issues arising from 3D rotations of the objects (not just the 2D rotations as used in the paper).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors proposed isometric quotient VAEs (IQVAEs), to extract the quotient space and learn the Riemannian isometry. Empirical proof-of-concept experiments are conducted to demonstrate the performance of the method.

STRENGTHS:
A new Auto-Encoders approach is proposed and its effectiveness is demonstrated to some extent.

WEAKNESSES:
However, some technical details are not clearly described.

QUESTIONS:
1. The details of the downstream classification tasks are not clear enough. Line 260, ``tasks by using the support vector machine (SVM)
261 [14] and random forest (RF) classifiers". The author needs to provide the parameters so that others can reproduce the results.
2. The authors need to analyze the results of experiments, such as Table 4 and Table 5.
3. The format of references needs to be standardized.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an autoencoder framework with additional features of (simultaneously) (1.) An invariance/equivariance of the latent space to a chosen group symmetry (2.) a near local isometry of the latent space. The authors build on the Quotient Autoencoder (QAE) framework that accounts for the group symmetry by modifying it into a probabilistic formulation of a variational autoencoder (called QVAE) and then further adding an isometry condition to the latent space by forcing the metric (computed via jacobians) to be close to a desirable form like identity - IQVAE. \n\nExperiments are demonstrated on the image datasets like Rotated MNIST, MixedWM38, and 3SIPaKMeD. Overall, the results are favorable and marginally outperform all conceptual baselines like AE, VAE, QAE, FMVAE, etc.\n\nSTRENGTHS:\n- Overall, the paper has been compiled well, with decent evaluations and visualizations demonstrating the positive aspects of the contributions of the IQVAE.\n- In particular, I found Table 2 and Figure 5 to be interesting experiments: showing performance with limited data and a more meaningful interpolation of the proposed IQVAE\n\nWEAKNESSES:\n- I do not get the impression that this paper makes any fundamentally new observation/contribution. The central premise is to combine already known components of QAE, VAE, and Iso-VAE [6, 22] etc - leading to the QVAE and IQVAE. More significantly, the results are decent but not to the extent of making a strong impression, thereby justifying the interest in investigating the combination. \n- An important reference must be included, compared, and commented on:   Gropp, Amos, Matan Atzmon, and Yaron Lipman. \"Isometric autoencoders.\" arXiv preprint arXiv:2006.09289 (2020).\n- I do not see IRVAE [22] being amongst the baseline evaluations. I understand the symmetry aspect is not a part of this paper. Still, it would be nice to clearly highlight the relative importance of symmetry, isometry, and probabilistic (i.e. like a VAE) components of the autoencoder approach - either implicitly or explicitly using an ablation study. Right now, despite the comparisons in Tables 1, 4, and 5 the message is not clear.\n\nQUESTIONS:\n- More fundamentally, what is the difference between the isometry assumption applied to the decoder (like this paper) in comparison to the encoder? \n\n- What is the importance of the shuffle and augment in Algorithm 1?\n\nLIMITATIONS:\nI do not see any direct societal limitations. See Weaknesses for technical limitations. \n\nOverall, I am voting for a borderline reject at this point. The paper is compiled ok, and reasonably motivated - it is interesting to combine symmetry and isometry into an autoencoder framework. However, both the concept and the results do not lead to a strong take-home message, especially in the midst of the many AE frameworks proposed in prior work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}