Regularized Autoencoders for Isometric Representation Learning
Authors: Yonghyeon Lee
Sangwoong Yoon
MinJun Son
Frank C. Park
Venue: ICLR 2022 Poster (2022)
Keywords: Autoencoders
Manifold Learning
Regularization
Geometry
Distortion

Abstract: The recent success of autoencoders for representation learning can be traced in large part to the addition of a regularization term.
Such regularized autoencoders ``constrain" the representation so as to prevent overfitting to the data while producing a parsimonious generative model. A regularized autoencoder should in principle learn not only the data manifold, but also a set of geometry-preserving coordinates for the latent representation space; by geometry-preserving we mean that the latent space representation should attempt to preserve actual distances and angles on the data manifold. In this paper we first formulate a hierarchy for geometry-preserving mappings (isometry, conformal mapping of degree $k$, area-preserving mappings). We then show that a conformal regularization term of degree zero -- i.e., one that attempts to preserve angles and relative distances, instead of angles and exact distances -- produces data representations that are superior to other existing methods. Applying our algorithm to an unsupervised information retrieval task for CelebA data with 40 annotations, we achieve 79\% precision at five retrieved images, an improvement of more than 10\% compared to recent related work. Code is available at https://github.com/Gabe-YHLee/IRVAE-public.

Published as a conference paper at ICLR 2022
REGULARIZED AUTOENCODERS FOR ISOMETRIC REPRESENTATION LEARNING
Yonghyeon Lee', Sangwoong Yoon', Minjun Son', Frank C. Park
1 Department of Mechanical Engincering, Scoul National University
2 Saige Research {yhlee, swyoon, wtw.:o/sssssoaooooioiooooooo ac.. fcpêanu.ac. kr
ABSTRACT
The recent success of autoencoders for representation learning can be traced in large part to the addition of a regularization term. Such regularized autoencoders "constrain" the representation ducing a parsimonious generative model. A regularized autoencoder should in principle learn not only the data manifold, but also set of geometry-preserving coordinates for the latent representation space; by geometry-preserving we mean that the latent space representation should attempt preserve actual distances and angles on the data manifold. In this paper we first formulate hierarchy for seometry-preserviie
to prevent overfitting the data while pro-
mappings (isometry, conformal mapping of degree k, area- preserving mappings). We then show that conformal regularization term of de- gree zero ii.., one that attempts to preserve angles and relative distances, instead of angles and exact distances produces data representations that are superior other existing methods. Applying our algorithm to an unsupervised information retrieval task for CelebA data with 40 annotations, five retrieved images, an improvement of more than 10% compared to recent re- lated work. Code is available
achieve 79% precision
INTRODUCTION
Learning good representation for high-dimensional data is one of the most fundamental problems machine learning. good representation should capture essential information about the data in parsimonious manner while filtering out all non-essential variations. Clearly what is "essential" or "non-cssential" depends heavily on the end task [Tschannen et al. 2018), and numerous criteria have been proposed for range of contexts, e.g., disentanglement (Higgins et al. 2017 Zhao et al. 2017) clustering (Makhzani et al., 2015). sparsity (Lewicki & Sejnowski| 2000), hierarchy (Gulrajani et al, 2017). and isometric embedding (McQueen et al. 166erfirn
al. 2020).
In this paper we take the view that the essential information best captured by the geometry of the data. More specifically, adopt the manifold hypothesis as point of departure, and further argue that a good representation should also preserve the geometry of the data manifold. That is, nearby points on the manifold should have representations in the latent representation space that are also nearby, and angles and volumes should be preserved as much as possible when moving between the data manifold and its representation space. To find such a representation, is important to (i) learn the correct low-dimensional data manifold, and (ii) to find an optimal set of latent space coordinates that preserves the geometry of the learned data manifold.
A primary reason that autoencoders are widely used for unsupervised representation learning is that they can learn both the manifold and the latent space coordinates simultaneously during the training phase. Vanilla autoencoders trained purely to minimize reconstruction loss tend to overfit, and the learned manifolds are often inaccurate. As demonstrated
(Rifai et al. 2011 Kingma & Welling 2014: Makhzani et al.) 2015 Tolstikhin et al. 2018 Lee al. 2021). by augmenting the reconstruction loss with a regularization term, manifold learning performance of autoencoders can be significantly enhanced. However, little if any consideration has been given to the concurrent problem of learning a set of latent space coordinates that preserves the geometry of the data manifold (with the exception of FMVAE (Chen et al. 2020), which discuss further below).

---

Published as a conference paper at ICLR 2022
The main contribution of this paper is regularized autoencoder that simultaneously lears the data manifold and geometry-preserving set of latent space coordinates. For this purpose, we first formulate hierarchy of geometry coometty-peeeervngg preserving mappings. At the top of the hierarchy isometries, which preserve distances and angles, followed by conformal maps, which preserve angles, and then area-preser maps. Of particular relevance this paper the conformal maps, which we further stratify into degree k conformal maps, with k = O corresponding to scaled isometries, i.c., maps that preserve angles and distances up to some scale factor.
Based on this hierarchy of mappings, we then formulate a corresponding hierarchy of regularization criteria for training the autoencoder. One of the important findings of our study that somewhat counterintuitively. using regularization term that measures the nearness to an isometry is in fact detrimental; such regularization term overly constrains the mapping, resulting struction loss whose effects cannot be mitigated even with adjustments to the regularization term weight. Rather, using less stringent regularization term is more helpful. our examples the degree zero conformal maps, scaled isometries, seem to offer the best balance between reconstruction accuracy and model parsimony.
higher recon-
We note that if the exact data manifold were known in advance, then the scale factor could be pre- computed, e.g., to make the latent space and the data manifold have the same volume, in which case it would make sense to
normalized version of the isometry measure regularization term. When using an autoencoder for representation learning, however, the data manifold not known a priori but rather learned together with the latent space representation. The effective alternative, we argue, is to learn this scale together with the manifold and the latent space representation. The FMVAE of (Chen al. 2020) is fact the first work adopt this approach, but the measure does not adequately capture the nearness to scaled isometry, and is also not coordinate-invariant, limiting its performance as we show later in our experiments.
Once the data manifold and an initial set of geometry srinnetry-preervvngg latent space coordinates are learned, is possible further "flatten" the latent space, by adding a postprocessing step that leads to an even more isometric set of coordinates. Specifically, we use an invertible neural network model to map the pre-trained latent space more isometric representation space, without incurring any further losses in reconstruction accuracy.
Experiments on diverse image and motion capture data confirm that, compared to existing related methods, our geometrically regularized autoencoder produces more isometric representations of the data while incurring only minimal losses in reconstruction accuracy. In particular, information retrieval task experiments conducted with CelebA data show that data similarity measurements per- formed in our representation space lead to significantly improved levels of retrieval performance.
Our specific contributions can be summarized follows:
We define a family of soordinate-invarinn regularization terms that measure how close the de-
coder is to being scaled isometry;
We propose an isometric regularization method for autoencoders that learns both the data mani- fold and set of in reconstruction accuracy;
latent space coordinates, all while incurring minimal losses
We propose postprocessing flattening technique that learns a more isometric representation space without further losses in reconstruction accuracy.
2 A HIERARCHY OF
PRESERVING MAPPINGS
This section introduces a hierarchy of geometry--reserving mappings between two Riemannian manifolds. Let M be a Riemannian manifold of dimension with local coordinates E R and Riemannian metric G(z) E mxm and A be a Riemannian manifold of dimension n with local coordinates mapping, represented in local coordinates by the italic symbol f R - R.. Its differential is 8f denoted by the Jacobian matrix (z(x) = as (z) E Raxm.
R and Riemannian metrie H(x) Rax= Let f M N be smooth
Intuitively, isometry is mapping between two spaces that preserves distances and angles every- where. For a linear mapping between two vector spaces equipped with inner products, an isometry preserves the inner product everywhere. In the case of mapping between Riemannian manifolds,
2

---

Published as a conference paper at ICLR 2022
f: M - . is an isometry if
VE R
(1)
Sometimes, requiring a map be an isometry can be overly restrictive; preserving only angles may be sufficient. A conformal map is mapping that preserves angles but not necessarily distances. Mathematically, f M + N conformal (or angle-presers preeeerrss))))) if
((=) = c(>)J)(z)* (((())) V2 E R."
(2)
for some positive function c MM - R. The positive function is called the conformal factor.
Conformal maps can be further categorized by the polynomial degree of c(z). A conformal map of degree zero, i.e., one in which c(=) is constant, sits one level below the isometric mapping and is defined formally as any mapping for which positive scalar constant satisfying
G(z) = cJr(2) ((f()))J((z) V2 E R"
(3)
can found. Such a map not only preserves angles but also scaled distances; for this reason we shall also refer to a degree conformal map scaled isometry. Area-preservine maps also can be placed within this hierarchy, but for the purposes of this paper our focus will be exclusively on isometric and conformal mappings.
3 A COORDINATE-INVARRAN RELAXED DISTORTION MEASURE
The goal of this section is to design coordinate-invariant functional F that measures the proximity of the mapping f: M + / to a scaled isometry. Section 3.1 shows how to construct coordinate- invariant functionals for smooth mapping between two Riemannian manifolds, while Section 3.2 introduces simple technique to define coordinate-invariant distortion measures that measure how close the mapping is to being an isometry. Section 3.3 defines a family of coordinate-invarian relaxed distortion measures that measure how close the mapping to being a scaled isometry.
In order to extend the discussion of traditional distortion measures to more general cases (we will later use a probability measure), we consider a positive measure v on .M absolutely continuous to the Riemannian volume form the spaces of interest will then be limited to the support of y rather than the entire manifold M.
3.1 COORDINATE- INVARIANT CEVADINATT-INVARINNT FUNCTIONALS ON RIEMANNIAN MANIFOLDS
functionals for smooth mapping f: M - N (Jang et al. 2020). At a point z E M, consider the characteristic values of the pullback metric J HJ,, relative to the metric G of M, i.e., the m eigenvalues Å (1 Am (m) of ( ( (z)) Apart from their order, these eigenvalues are intrinsically asso- ciated with J3. H and G, i.e., they are invariant under coordinate transformations?
We begin this section by reviewing how construct coordinate-invarian
Let S(A1, Am) be any symmetric function (i.e., a function whose value invariant with respect to permutations of its arguments) of the eigenvalues. Then the integral
Is(f) /
Am dv())
is an intrinsic quantity, i.c., coordinate-invariant.
In this paper we introduce a new family of coordinate-invariant functionals that can be used to formulate measures for scaled isometries in a more natural way. Consider symmetric function S such that the above integral Is(f) # O, and let S'(A,...,,Am. be any symmetric function. The following integral
S. S'( (2)2)
Am(2))
dr(z)
(5)
Is(f)"
Is(
is then an intrinsic quantity, since the cigenvalues and Is(f) are both intrinsic quantities.
"Given mcasurable space M, a measure is absolutely continuous to the Riemannian volume measure Vol such that dVVol22)= = dee G(z) dz, if Vol(A) 0 implics ((A) 0 for any measurable set C M. Figenvalues of J? HJ remain the under coordinate transformations ++ ((2), (0)).
3

---

Published as a conference paper at ICLR 2022
3.2 DISTORTION MEASURES OF ISOMETRY
Recall that f: M + A is local isometry at if Ax(=) = for Ví, and a global isometry if f local isometry everywhere (Jang al. 2020). In this section we introduce simple technique to define family of distortion measures that measure the proximity to an isometry. Consider any convex function h R + 00) such that h(1) O, (A(A) = 0 iff À 1, and define a symmetric function S(A,...., Am) = Di=1 h(A.). Then the foordinate-invartn functional
/ h(),(z) dv(z)
(6)
is global measure of distortion (restricted to the support of v). Since the integrand in the above functional locally measures the deviation of the mapping f from an isometry, its integral also serves as global measure of distortion. Popular choices include h(A) = (1 - X)2 and h(A) (lo)(
3.3 A RELAXED DISTORTION MEASURE AND SCALED ISOMETRY
The goal of this section is to design a family of coordinate-invariane functionals F that measure how far the mapping : M + N is from being a scaled isometry (over the support of v). We refer to these as relaxed distortion measures, the sense that larger set of mappings are minimizers the relaxed distortion measure than the original distortion measure.
Given mappings f, f, : M + N (denoted f and f' in local coordinates) with respective Jacobians Jr. Jr, the desired properties of F are as follows:
1. F()) 2 0;
2. F(f) 0 if and only if (2) for vi, V2 Supp(v), and for some 0;
3. F()) = F(') if HJ for Vz E Supp(y), and for some c >> 0,
where {²(2) denotes the eigenvalues of (()))((((( and Supp(v) is the support of v. The first condition is to ensure minimum of zero, while the second condition is to make any scaled isometry (restricted to the support of v) be minimizer. Although the first and second conditions are sufficient to use F(f) as a measure of the proximity of f to a scaled isometry, we impose an additional third condition to make the measure more natural in the following sense: since the measure should not priori favor particular scale for the pullback metric, if the pullback metrics are equivalent up to some scale, then these should be treated equivalently.
Our core idea for constructing such measure is to use the newly proposed family of coordinate- invariant functionals 5 with the technique used in 6 as follows:
(x(2)
F(f) = Σ h( JM S(A)(2), .... Am(2) du(z) dv(2)
1=1
where h is some convex function and S some symmetric function. This functional automatically satisfies the first condition. To satisfy the second and third conditions, the symmetric function S must satisfy some further conditions:
Proposition 1 The functioaal in 0 satisfies the second and third conditions if v is a finite measure, S(k),, ..., kdm) = k.S (x1, ..., Am). and S(1, ..., 1) 1/v(M). Proof: See Appendix A.
ISOMETRIC REPRESENTATION LEARNING
We now introduce a regularized autoencoder with practical form of the relaxed coordinnte-invariin distortion measure. We also describe the additional postproccssing step for further flattening the latent space. Finally, we introduce some relevant implementation details.
4.1 ISOMETRIC REGULARIZATIO WITH THE RELAXED DISTORTION MEASURE
Denote a parametric encoder function by 9s R + R. and decoder function by fo R + R.". We consider the latent space R. with coordinates z assigned with the identity metric G(z) I and
4

---

Published as a conference paper at ICLR 2022
the data space R with coordinates assigned with the metric (x). We denote the data distribution in I by x ~ PD and the distribution of the encoded data in R by 2 ~ Po where & = (x(x).
First, to construct relaxed distortion measure of fe. need to select positive measure Con- sidering y/ as a probability measure Pa and replacing the integrals SM dv(z) (7) by expectations Pg get the following expression:
(1(2) E Px [S(A1 A (2)) )],
F(fo; Pa) = B
where (z) are the eigenvalues of the pullback metric JT. (2) (())(((())
Then, we propose regularized autoencoder where the loss function consists of the following two terms i) loss function [(0, Φ) for manifold learning (e.g., reconstruction error) and ii) regularization term F(f; Pa) for learning a scaled isometric decoder:
min [(,,0) oFoo Po).
Training an autoencoder to minimize referred to as Isometric Regularization (IR).
In practice, the computation of and prok-propag through F(Jo: : Pa) that includes the computa- tion of the entire Jacobian of fe requires sufficient memory and computational cost. Instead, with an appropriate choice of convex function h and symmetric function S, the measure F(fe; Po) can be efficiently estimated by using the more easily computed Jacobian- vector and vector-Jacobian products and Hutchinson's stochastic trace estimator (Hutchinson 1989):
Proposition 2 Let ((A) = (1 2)2 and S(A1,...,AAm) E \a/m, then
m
1) = m² E.P P.
Tr (H)()))
F(fo: = EsNPa
m, (10)
Pe |E; \r(2)/m]
Ex (o)))))))
where >=(2) means the eigenvalues of Holz) :=, (T)
Proof: See Appendix A.
4.2 LATENT SPACE FLATTENING
In the isometric regularization approaches, autoencoders are trained to minimize both manifold learning loss and geometric regularization term; hence there exists inherent tradeoff. We in- troduce a postprocessing step to further flatten the pre-trained latent space to a more isometric rep- resentation space without incurring any further losses in reconstruction accuracy.
Given a trained encoder 9o DRD - RT and trained decoder So : m + RD, consider an in- vertible map i: I + Rm such that '' = i(z) and the composition fo o i-1 Rh + RD. The map transforms the pre-trained latent space to reconstruction accuracy. This gives an additional degree of freedom to find a more isometric repre- sentation space. Based on this idea, we formulate the latent space flattening problem as follows:
set of latent coordinates without affecting the
min F(Jooi P!) + B
(11)
where 2' ~ P. is the distribution of the encoded data in T (&' = i go (x)) and the second term the regularization term with coefficient B added to prevent i(2) from diverging. In this paper, we use the invertible deep neural network RealNVP (Dinh al. 2017 for i.
4.3 IMPLEMENTATION DETAILS
We now explain some relevant implementation details of our approach. We use a Gaussian encoder (=(=))) and decoder pe (z]z), and treat the mean of (2(22) as (z). We use the negative evidence lower bound for L(,, Φ) from the VAE with the unit Gaussian prior distribution (Kingma & Welling 2014]. We assume the identity metric for the data space, i.c., (xx = I. We use the particular combination of h and S described in Proposition 2 In cases when the latent space dimension is sufficiently small, it becomes feasible to compute the full metric and use other combinations. As an example, when the dimension is two, we another popular choice h(X) (log()))
5

---

Published as a conference paper at ICLR 2022
Augmented Distribution: The relaxed distortion measure is defined as the expectation over the encoded data distribution P. However, the influence of the measure is then limited to regions where data is available; in practice, data augmentation can resolve this issue. Following the FMVAE (Chen et al. 2020), we use the modified mix-up data-augmentation method with v) >> O, where P is aug- mented by = 8z1 (1 =) such that 2; P, 2, where 5 is uniformly sampled from [-7,, 1 1+ n/.. Sampling from the augmented latent space data distribution is denoted as ~ Pz.
The pseudocode is available in Appendix B.
EXPERIMENTS
Throughout this section use isometric regularization VAE (Kingma & Welling 2014 Rezende & Mohamed| 2015) which we refer to as the Isometrically Regularized VAE (IRVAE). We then train the Flattening Module (FM) in a post hoc manner, which we denote by IRVAE FM. In Se. we show that (i) IRVAE can effectively learn the isometric representation with a minimal loss in reconstruction accuracy compared vanilla VAE and FMVAE (Chen al. 2020), and (ii) IRVAE + FM leads to a more isometric representation without any loss in reconstruction accuracy. In Section 5.2 through an unsupervised human face retrieval task, we show that measuring data similarity in our isometric representation spaces significantly improves retrieval performance. A mathematical comparison of IRVAE with FMVAE is provided in Appendix C, with experimental details given in Appendix D.
5.1 ISOMETRIC REPRESENTATION WITH MINIMAL LOSS IN RECONSTRUCTION ACCURACY
We first introduce some evaluation metrics that measure (i) the accuracy of the learned manifolds, and (ii) how isometric the latent space is (i.e., how close the pullback metric (z) == (2)
is to {clle (0, o)))). These metrics computed over the test datasets.
The accuracy of the learned manifolds measured in terms of the mean square reconstruction errors. To evaluate M(2,, we introduce two different metrics that are scale-invariant (i.e.. (z) and (=) have the same values). First, the Variance of the Riemannian metric ( (VoR) is defined
as the mean square distance from M(2) to M E2-Pz P. [M]]] where we use the dfffine-ivaaiann Riemannian distance d, i.c., (A, = Σx= (log AB (1B-11 ( (Pennec et al. 2006: Lee & Park, the Mean Condition Number (MCN) of the Riemannian metric, where the
2018). Second, condition number of (z) is the ratio between the maximum eigenvalue and minimum eigenvalue. Both metrics are computed by sampling sufficiently large number of points from Pz.
VoR measures how the Riemannian metric is spread out from M, which becomes zero when the metric is constant for all in the support of Pz. While VoR measures the spatial variance of the metric in 2, MCN measures how much the metric is isotropic. MCN becomes one when M(2) x for all 2 in the support of Pz-
For IRVAE and FMVAE, there are natural tradcoff's between MSE versus VoR and MCN. If we use higher weights a for the regularization terms, the representation becomes more isometric with losses in reconstruction accuracy. In order to compare the algorithms in regularization coefficient a-invariant manner, we compare the tradeoff curves, i.e., MSE as function of VoR and MCN obtained by using varying regularization coefficients
Sections 5.1.1 and 5.1.2 Juse the image and CMU motion capture data, respectively, with two- dimensional latent spaces. Additional experimental results including experiments on more diverse image data (MNIST, FMNIST, SVHN, CIFAR-1O) are provided in Appendix E.
5.1.1 MNIST
Figurel shows tradeoff curves of FMVAE and IRVAE trained on MNIST data with two-dimensional latent spaces, some example reconstructed images, and latent space representations with equidis- tance plots. We note that the tradeoff curves of IRVAE (orange) are below those of FMVAE (blue), meaning that the IRVAE learns more isometric representations at any level of reconstruction accu- racy. Cases A and B produce good reconstruction results, but non-homogeneous and non-isotropic

---

Published as a conference paper at ICLR 2022
equidistance plots since lower regularization coefficients used. For cases F, I, opposite results are obtained.
Input
0
2
1
1
6
7
0008
0033
reconstruction
0
2
2
9
4
M56
0.26
0035
0 2
6434
2
2
2011
20n1
35
10
2
5
6
7
9
VoR (log)
MCN (log)
IRVAE
Figure I: Top: The MSE and VoR, MCN tradeoff curves, and some example reconstructed images produced by IRVAE trained with various regularization coefficients. Bottom: Two-dimensional la- tent space representations with some equidistance plots whose centers consist of randomly selected data point 2e from each class for
The equidistance plots are =2](=-2e) Jf (Z))) ze) for k 0. (The more homogeneous and isotropic, the better.)
Figure shows tradcoff curves and latent space representations for IRVAE + FM (under the same experimental setting above). IRVAE + FM results in more isometric representations than IRVAE, with no losses in MSE. In particular, for cases and B, which have the lowest MSE but the highest VoR and MCN, the flattener significantly lowers both VoR and MCN. These improvements can be qualitatively seen by comparing the equidistance plots of A and B in Figure T and Figure 2
IRVAE
FM
0138
2018
llog]
(log)
Figure 2: Tradeoff curves for FMVAE, IRVAE, and IRVAE + FM, and two-dimensional latent space representations with some equidistance plots (under the same experimental setting Figure [].
5.1.2 CMU MOTION CAPTURE DATA
We use subset of the CMU motion cap- ture data by selecting four classes: walk- ing, jogging, balancing, and punching, with two-dimensional latent spaces. The motion data in each class is a sequence pose data, where each pose data is repre- sented by a 50-dimensional vector of joint angles. We 2,000 pose data items for training, valida- tion, and test, respectively.
24
MSE
13.
35
00
80
VoR
MCN
FMVAE
IRVAE
IRVAE + FM
total of 10,000, 2,000 and
Figure The tradeoff curves for FMVAE, IRVAE, IR- VAE + FM trained with the pose data.

---

Published as a conference paper at ICLR 2022
Figure 3 shows the tradeoff curves for FMVAE, IRVAE, and IRVAE + FM. The results of VAE are omitted from the figure since the MSE, VoR. and MCN are too large to be plotted together with the other models (MSE: 1.72x10 VoR: 4.86, MCN: 20.3). The tradcoff curves for IRVAE are far below compared to those for FMVAH, while the IRVAE + FM results in a more isometric representation without any loss in MSE.
Figure shows the latent space representations with equidistance ellipses. For a fair comparison, select models with similar reconstruction accuracy (the selected regularization coefficients are marked by the black circles in Figure 3 We observe that the ellipses for IRVAE and IRVAE + FM are much more homogeneous (blue) than those for VAE and FMVAE.
VAE
FMVAE
IRVAE
IRVAE + FM
condition numbor numbar(elllpps (ellippz color)
walking
balancing
jonging
punching
Figure 4: Latent spaces and equidistance ellipses for VAE, FMVAE, IRVAE, and IRVAE + FM (the redder the ellipse, the larger the condition number).
5.2 UNSUPERVISED HUMAN FACE RETRIEVAL
We also demonstrate the effectiveness of our approach to image-to-image retrieval of human faces. We consider the retrieval of face images that have a queried set of attributes, in which the query is also given as set of face images that share some visual attributes. For example, if a user provides a set of photos that contain smiling faces but differ in other attributes, retrieval algorithm is expected to return a set of smiling face images from the database. In this setting, a user communicates with a retrieval system by providing examples, avoiding the dependence on human language.
We focus in particular on an unsupervised scenario in which a retrieval algorithm built without any annotation. Unsupervised representation learning methods are trained on CelebA (Liu et al. 2013), which contains 182,637 training images and 19,962 test images. Each image in CelebA has 64x64x3 pixels and contains an aligned human face with binary annotations on 40 attributes. The attribute annotations are only used to evaluate retrieval performance.
We refer to attributes that query images have in common as query attributes. We experiment with different settings in which there is a single query attribute (single attribute retrieval; SAR) and also with two query attributes (double attribute retrieval; DAR). We use all images that have given query attributes in the training set as query images, and retrieve the top K similar images from the test set based on the summed cosine similarity to the query images.
The quality of a retrieval is measured using precision at K (P@K) for = 1,5, 10. 20. P@K K denotes the ratio of images that have query attributes among the K retrieved images. For SAR, P@K K are averaged over the 40 attributes. For DAR, there are 780 possible combinations of two attributes. We select a combination of attributes if the images that have those attributes account for more than one percent and less than fifty percent of the total test data; thus P@ K are averaged over 487 selected combinations.
Retrieval is performed in the latent space of the representation learning algorithms. Algorithms to be compared include (i) unsupervised representation learning methods VAE (Kingma & Welling 2014). FMVAE (Chen et al.) 2020). IRVAE (ours), and IRVAE + FM (ours), whose latent space dimensions are 128, (ii) a neural network (ResNet-50) pre- pre-trained on ImageNet (He et al.) 2016). and (iii) supervised learning method (binary relevance) (Read et al. 2011). The performance of
8

---

Published as a conference paper at ICLR 2022
the pre-trained network and the supervised learning method serve lower and upper bounds for the unsupervised methods, respectively.
Table 1: Precision at K retrieved images, i.e., P@K, averaged over the attributes. The best results among unsupervised methods are colored red, while the second best results are colored blue.
Method
P@1 P@5
P@10
P@20
P@1
P@5
P@10
P@20 20
Single Attribute Retrieval
Double Attribute Retrieval
VAE
60.0
60.5
61.0
58.4
39.8
38.7
37.0
34.7
FMVAE
67.5
66.0
64.0
62.6
41.5
39.1
38,8
36.8
IRVAE (ours)
82.5
72.0
71.5
69.0
56.1
50.8
47.2
44.2
IRVAE FM (ours)
75.0
79.0
75.3
74.1
57.3
54.0
51.4
49,4
Pre-trained (ResNet-50)
35.0
28.5
23.5
23.9
18.3
11.0
8.7
9.2
Supervised (BR)
87.5
82.5
80.3
80.8
66.9
65.2
63.7
62.6
Table lists the retrieval performance for the various cases. IRVAE outperforms FMVAE, which in turn outperforms VAE. FM improves the performance of IRVAE for most cases with only one exception, P@I score in SAR. More detailed results on SAR are included in Appendix F. Some example image retrieval results are visualized in Figure 5
Wearing Hat
Bald
Bangs
VAE
FMVAE
IRVAE
IRVAE +
FM
Black Hair + Mouth Slightly Open
Smiling + Wearing Hat
Eyeglasses Male
VAE
FMVAE
IRVAE
IRVAE + FM
Figure 5: Some example image retrieval results (top 5 images). Common attributes of query image sets are written above the figures. Higher rank images are located left.
CONCLUSION
We have formulated the problem of learning the manifold simultancously with a set of optimal latent space coordinates that preserve the geometry of the learned manifold. We have introduced hierarchy of geometry- mappings between two Riemannian manifolds (e.g., isometry, conformal mapping of degree k, area area-presg mappings) and defined family of coordinate- invariant relaxed distortion measures that measure the proximity of the mapping to a scaled isometry (i.e., conformal mapping of degree 0). Finally, two algorithms, isometric regularization and latent space flattening, have been proposed. We have verified the efficacy of our methods with diverse image and motion capture data, and through human face retrieval task with CelebA data.
We believe the algorithm can be further enhanced in a number of different ways. The current im- plementation of IRVAE and IRVAE + FM assumes the identity metric in the ambient data space, ie., H(x) = I. Although this is a reasonable choice in a fully unsupervised setting, we think that domain- sppciffc knowledge a few labels can be leveraged define better H(x). In addition, in- stead of the mix-up data augmentation method used in this paper, developing a principled approach for defining Pz will be interesting future research direction.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have included complete proofs of the Propositions in Appendix A. Also, we have included experiment settings in detail as much as possible in Appendix D and E such as the number of training/validationnness splits of datasets, preprocessing methods, neural network architectures, and hyper-parameters used in model training (e.g., batch size, number of epochs, learning rate, etc).
ACKNOWLEDGMENTS
This work was supported in part by SRRC NRF grant NA16R1A5A1988477 HTTP-MSIT Grant 2021- 0-02068 (SNU AT Innovation Hub), SNU-IAMD, SNU BK21+ Program in Mechanical Engineering, and the SNU Institute for Engineering Research.
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3598)11798-1828, 2013.
Nutan Chen, Alexej Klushyn, Francesco Ferroni. Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with vaes. International Conference on Machine Learming, pp. 1587-1596. PMLR, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp- 2180-2188, 2016.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In Proceedings of the 5th International Conference Learning Representations (ICLR), 2017.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. In Proceedings of the 5th International Conference Learning Representations (ICLR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a con- strained variational framework. In Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications Coatistics-Simalant and Computation, 1856 1989.
Cheongjae Jang, Yung-Kyun Noh, and Frank Chongwoo Park. A riemannian geometric framework for manifold learning of non-euclidean data. Advances Data Analysis and Classification, pp- 1-27, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks.
AIChE journal, 37(2):233-24 223, 1991.
Taeyoon Lee and Frank C Park. A geometric algorithm for robust multibody inertial parameter identification. JEEE Robotics and Automation Letters, (03:2455-24622 2018.
Yonghyeon Lee, Hyeokjun Kwon, and Frank Park. Neighborhood reconstructing autoencoders. Advances in Neural Information Processing Systems, 34, 2021.
10

---

Published as a conference paper at ICLR 2022
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in neural information processing systems, pp. 777-784, 2005.
Michael S Lewicki and Terrence J Sejnowski. Learning overcomplete representations. Neural
computation, 12(2):337-365, 2000.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Alireza Makhzani. Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.0564 2015.
James McQueen, Marina Meila, and Dominique Perrault-loncas. Nearly isometric embedding by relaxation. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 1639-26447 2016.
Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A riemannian framework for tensor computing. International Journal of computer vision, (66(1):11666 2006.
Erez Peterfreund, Ofir Lindenbaum, Felix Dietrich, Tom Bertalan, Matan Gavish, Ioannis G Kevrekidis, and Ronald R Coifman. Local conformal autoencoder for standardized data coor- 2020.
dinates. Proceedings of the National Academy of Sciences,
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-label classification. Machine learning, 85(3):333-359 2011.
Danilo Jimenez. Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. In Joint European conference on machine learning and knowledge discovery in databases, pp. 645-6666 Springer, 201 20
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Icml, 201 lb.
I Tolsttkkiin, O Bousquet, S Gelly, and B Schölkopf. Wasserstein auto-encoders. In 6th International Conference on Learning Representations (ICLR 2018). OpenReview. net, 2018
Michael Tschannen, Olivier Bachem, and Mario Lucic Recent advances in autoencoder-based representation learning. arXiv preprint arXiv: 101..0112000055 2018.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational
autoencoders. arXiv preprint arXiv: 1706.0226, 2017.

---

Published as a conference paper at ICLR 2022
APPENDIX
A PROOF OF PROPOSITIONS
Proof (Proof of Proposition] 1 For sufficiency (+) of the second condition, suppose F(J) = O.
Then, denoting by c JM (2), ..., Am (2))
dv(2)).
Ai(z), h Σ 1=1
)= =0 42 E Supp(v).
Since h(A) 2 for AA E R. h(A(2)/c) 0 for Vi Supp(r). Since IA(A) 0 iff X 1, X1(2) = cfor Vi, V2 E Supp(v).
For necessity (+) of the second condition, suppose >=(2) cfor Vi, V2 Supp(v), then
Xx(2) JM (1) Am (=)) du(z) JM c) dx(z)
cJm du(z)
D(M) J' du(z)
1 for Ve Supp(v).
Then consequently,
Σ M F() - M -
A122) SM S(AI (2)) du(z) dulz)
- JM
Σ
dv)=))))>>)))))))))))())))))dd 0.
For the third condition, suppose J JJJ cJ HJ in Supp(v) for some > 0. Denote the eigenvalues of Jブ HJJG and J HJ by >[ff] and A4g9] where Ailf] 2 ... 2 Am If] and A1[g) 2 ... 2 Am(g), [g), respectively. Then, obviously, Aiff cArlgl Vi. Then consequently.
F()) |
Σ
m
M
IM S(A((())()), Am [f]())) du(z) ) du(z)
x=)
Σ h
JMC c. S(h1 ...,, Am [=)) du(z) du(z2)
(x)(g(=))
2x)
hl JM S(A1lo((2),
(2)) du(z) du(z) = =F(g).
Proof 2 (Proof of Proposition 2 Note that Tr(Ho(z)) = E, \x(z) and Tr(H?(2)) = E.
and denote by E LE-PP [Tr (1)()))))) Then
m
(x(2)
m
mdx(2)
E
Es
1)21 = E EeN
1)21
E Eg Po 1E: {x(2mm]]
E
m
2 X7(z)
2mda(2)
EE Ps
+ +1]
E2
E
m? (²(2))
E
F22
E E_~Pa
TT [T(H)
m
Eg-P, (x)))
m.
12

---

Published as a conference paper at ICLR 2022
B PSEUDOCODE
In this section, we provide pytorch style pseudocode for our Isometric Regularization and Flattening Module.
B.1 ISOMETRIC REGULARIZATION AUTOENCODERS
In the main manuscript, we mainly study the isometric regularization effect Variational Autoen- coder (VAE). VAE is just one possible choice; our method is straightforwardly applicable to other types of autoencoders. We consider deterministic encoder and decoder in the following pseu- docode.
>* assume
data_space dimension) 2: latent value w/ size (batch_size) latent_ sepeet___pc__________ dimension)
input data w/ size Dbatch_size, data space
encoder: torch.an. MoMouule (o.g., 2 encoder (x)
decoder: torch. nn Module (o.g., x decoder (2)
eta: mixup parameter
def relaxed saxeedddsttortinn tettstrrrttoommsuureeeeeee measure (decoder, 2)
bs, z_dim z.size()
10
torch. randn (bs, 2_dim)
12
Jv torch, autograd. functienal, (ttetieenn...yyeeeeeeeepeeee (decoder, 2, v=v)
13
TrG torch, sum (Jv++2, dim-1) ()an
14
functicnal JTJV - terch. iunttrad. vippaiiipppaa.. (decoder, 2, v-JV)
TrG2 torch. (JTJJV+ dim-1) ()
TrG2/ (TrG**2)
17
18
def iscmetric_ httpretti___ggglariaatttttttet encoder, decoder, eta): (x)
19
bs, 2_dim s.size
21
22
sample z_augmented from P_
23
2_permuted z [torch. randperm (bs)
al 1ipha_samples (torch. rand (ba, 1) (1 2veta) eta)
augmented alpha samples (1 alpha samples) ez permuted
26
27
compute relaxed distortion measure
28
return relaxed_distorrtinn_aaa (decodees (decoder, 2_augmented)
B.2 FLATTENING MODULE
rrrin In addition to the above, we assume
2_: now latent value w/ size (batch_size, latent_spacc_dimenee
batteening_modulee torch. nn Module (e.g., 2 mlodtening_modulle (2)),
invertible ..
def Elattening_loss encoder, decoder, flattening_module, eta):
= wtlatteeinngnooeeei module (encoder )x)
bs, d_m s_seize
10
2_permuted "_(torch.randpemn (bs)
11
alpha_samples (torch. rand (bs, 1) * (1 + 2-eta) eta)
augmented = alpha samples + (1 - alpha samples) permuted
14
def func (z_):
15
z = inverse oal_ooff_afllle_eee httee_offofffiigggeeee flattening_modulefz
16
return decoder (2)
18
return relaxed distortion measurr (fune, a_agmeeted)
13

---

Published as a conference paper at ICLR 2022
C COMPARISON TO OTHER REGULARIZATION APPROACHES THAT USE THE JACOBIAN.
There have been several works regularized autoencoders that the Jacobian of the encoder or decoder. In the following discussion, let ((x) be an encoder and f(z) be a decoder, and denote the Jacobian of g(x) by (z) and the Jacobian of f2) by yz)... Let ||-|| denote the Frobenius norm. The latent space dimension is denoted by m. PD is the data distribution and Pz is the (augmented) encoded data distribution.
The Contractive Autoencoder (CAE) attempts to enhance robustness of representation by penalizing the Jacobian norm of the encoder function (Rifai et al. 201Tb). Mathematically. the regularization term is
R Pg [|] (+)
The Contractive Autoencoder with Hessian regularization (CAE+H) introduced long after in Rii- fai et al. 2011a) has regularization term
Eg- En-PPIIISS + J -x+
where 6 is small value and 7 controls the balance between the two terms. The second order regularization using the Hessian penalizes curvature, and thus favors smooth manifolds. These approaches mainly aim to learn robust representations by finding smooth encoders that have small Jacobian and Hessian norms.
More recently, a regularization method that finds a geometry preserving mapping has been intro- duced (Chen et al.) 2020), called the Flat Manifold Variational Autoencoder (FMVAE). This is the closest work to our paper, and similarly aims to find a scaled isometry. However, the regularization term introduced in FMVAE has several limitations compared to ours.
We first note that geometric objects that we want to preserve, such as length, angle, and volume, are all coordinate invariant concepts, which require coordinate invariant formulation from the start. If one uses coordinate-varian measure, then there is no guarantee that it will work effectively for different choices of coordinates. Also, there no priori way of knowing which coordinates best.
In this regard, we first show that the regularization term defined in FMVAE is coordinate- invariant, Let M be a Riemannian manifold of dimension m with local coordinates E R= and Riemannian metric G(z) mm and A be Riemannian manifold of dimension with local coordinates z E R and Riemannian metric Hxx) E R=*n Let f M - N be smooth mapping, represented in local coordinates by the italic symbol f Rm - R". Let Jy be the Jacobian of f and Pz be the probability distribution expressed in Rmm. Let be positive measure on M.
The regularization term defined in FMVAE is then
(()) -
For simplicity, con- the input manifold z' Az and output manifold x' = Br. Then the function f is transformed to f'(x') Bf(A-1="), G(z) is transformed to
where || || is the Frobenius norm and c J- (TTT(
sider pair of linear coordinate transformations
G'(=))== A A A and (x) is transformed H² : B Bz) Then, after some calculations, the above regularization term is transformed to
|A -(((()))((())))))
where è' d= -Sm m (T((A" the A-T and A-1 multiplied at sides are not canceled; we can see that this is not coordinate-invariant
Recall that the Frobenius norm ||||
In addition to this, perhaps more direct reason that the FMVAE does not perform as well as our method can be explained as follows. We remark that one of the desired properties of the scaled isometry measure the third condition in Section 3.3: given two mappings f and f' from M to (f))) for some 0 for all 2, then the scaled isometry measure should be the same (that is, the metric should not distinguish between and f").
N, if Jf (f)
14

---

Published as a conference paper at ICLR 2022
This makes the measure more natural, in the sense that it does not a priori favor a particular scale for the pullback metric; the pullback metrics are equivalent up to some scale, then these should be treated the same.
The regularization term introduced in FMVAE can also be viewed as a scaled isometry mea- sure. However, it fails satisfy the all-important third property. Even though the pullback met- rics of f and f' are equivalent up to some scalar multiplication, i.e., J7 (z)((( = (())()))(((()) for some > 0 for all 2, the with the "smaller" Jacobian (i.c., smaller norm) achieves a lower value of the regularization term. As result, the FMVAE favors mapping with a smaller Jacobian, which can be detrimental to learning accurate data manifolds.
On the other hand, our relaxed distortion measures are defined in goordinate-invarte way. Further, as we have shown in our main manuscript, the measure does not favor particular seale of the pullback metric. We believe, because of these characteristics, IRVAE can outperform FMVAE.
D EXPERIMENTAL DETAILS
D.1 SECTION 5.1.1
Dataset: We use MNIST dataset. The training, validation, and test data are 50,000, 10,000, and 10,000, respectively.
Network Architecture: For both FMVAE, IRVAE, we use fully-connected neural networks that have four hidden layers with ReLU activation functions (256 nodes for each layer are used) for en- coder and decoder. The output activation functions are linear and sigmoid for encoder and decoder, respectively. For FM, we use the RealNVP model of depth and length 512.
Other Details: For FMVAE and IRVAE, the batch size is 100, the number of training epochs is 300, the learning rate is 0.0001, and 7) 0.2. For FM, the batch size is 100, the number of training epochs is 100, the learning rate is 0.0001, 3 = O, and 17 = 0.2. We use Adam optimizer. Validation sets are used to determine optimal models during training.
D.2 SECTION 5.1.2
Dataset: We use CMU motion capture dataset. The training, validation, and test data are 10,000, 2,000, and 2,000, respectively.
Preprocessing: In the pre-processing step, the position and orientation of human body center (root) are removed. In addition, the joint angles that have nearly no movements (clavicles, fingers) are removed as done in FMVAE (Chen et al., 2020), and result, each pose data is expressed 50-dimensional vector.
Network Architecture:
fully-connected neural networks that have two hidden layers with ReLU activation functions (512 nodes for each layer are used) for encoder and decoder. The output activation functions are linear and tanh for encoder and decoder, respectively. For FM, we use the RealNVP model of depth 8 and length 512.
For VAE, FMVAE, and IRVAE,
Other Details: For VAE, FMVAE and IRVAE, the batch size is 100, the number of training epochs is 300, the learning rate is 0.0001. = 0.2 for FMVAE and IRVAE. For FM, the batch size is 100, the number of training epochs 300, the learning rate 0.0001, 3 = 0, and = 0.2. We use Adam optimizer. Validation sets are used to determine optimal models during training.
D.3 SECTION 5.2
Dataset: The number of training, validation, and test data are 162700, 19937, and 19962.
Network Architecture: We denote Con2d layer by Con2d (input channel, output channel, kernel size, stride, padding) and Cony Transpose2d layer by Conv (imput channel, output channel, kernel size, stride, padding).
For VAE, FMVAE, and IRVAE, encoders are Convolutional Neural Networks with the following architecture: i) Cony2d (3, 128, 5, 2, 0), ii) Conv2d (128, 256, 5, 2, 0), iii) Conv2d (256, 512, 5, 2,
15

---

Published as a conference paper at ICLR 2022
0), iv) Conv2d (512, 1024, 5, 2. 0), v) Conv2d (1024, 256, 1, 0) with ReLU hidden layer activa- tion functions, and decoders are Convolutional Neural Networks with the following architecture: i) ConvT2d (128, 1024, 8, 1, 0), ii) ConvT2d (1024, 512, 4. 2, 1), iii) ConvT2d (512, 256. 4. 2. 1), iv) ConvT2d (256, 128, 4, 2, 1), and v) Cony (128, 3, 1, 1, 0) with ReLU hidden layer activation functions. The output activation functions are linaer and tanh for encoder and decoder, respectively.
For FM, we use the ReaINVP model of depth and length 512.
For BR, output by its norm) at last (this addition improves the retrieval performance, very significantly).
the same encoder network as VAE, but add normalizing layer (i.e., divide the
For ResNet-50, we use the pre-trained model on ImageNet.
Other Details: For FMVAE and IRVAE, the batch size is 100, the number of training epochs is 100, the learning rate is 0.0001, and 7) 0.2. For FM, the batch size is 100, the number of training epochs is 100, the learning rate is 0.00001, and 17 0.2. We use Adam optimizer. For BR, the batch size is 100, the number of training epochs is 100, the learning rate is 0.001 (binary cross entropy loss is used). Validation sets are used to determine optimal models during training.
For FMVAE, we test the following regularization coefficients a E {0.1, 1, 10, 100} and re- port the best results at a = 1. For IRVAE, we test the following regularization coefficients E {1, 10, 100, 1000} and report the best results at a 10. For FM, we use the stabilizing term with 3 = {10, 100} and report the best results at B = 100; and it is empirically observed that using small learning rate is very important.
Cosine similarity is used as similarity measure except for the ResNet-50 that uses Euclidean metric (we have tested both cosine similarity and Fuclidean metric for each learned representation, and found that cosine similarity mostly works better except the ResNet-50).
E ADDITIONAL EXPERIMENTAL RESULTS
E ADVANTAGES OF ISOMETRIC REGULARIZATION FROM A GENERATIVE PERSPECTIVE
Autoencoders are not only useful for representation learning, but also can be used for realistic data generation. In this section, we show advantages of our isometric regularization from generative perspective with the following experiments: (i) modulation in the latent space and (ii) linear inter- polation in the latent space. We use the CMU motion capture data with the fully connected neural network as described in D.2. We use the latent space of dimension 8.
Latent Space Modulation Figure eight generated poses for VAE and IRVAE obtained by the following procedure: (i) encode the original pose in the latent space, (ii) translate the encoded latent value along each latent space axis, and (iii) decode the translated values back to the data space. In case of VAE, translations along the latent space axes except 22, žg axes do not generate new poses. In particular, the pose changes really a lot along the 22 axis and yet just a little along the 26 axis. This shows that the trained decoder in VAE is far from the isometry. In contrast, with the isometric regularization in IRVAE, each translated latent value along each latent space axis generates different pose. Unlike VAE, diverse poses are generated, and in other words, more disentangled representation is obtained. This property of the isometric decoder has a great advantage (Bengio) 2013 Chen et al. 2016).
al.
Latent Space Interpolation : Figure 7 shows sequences of poses generated from the latent space linear interpolants between walking pose and balancing pose. In case of VAE, punching poses suddenly appear in the interpolation of the walking and balancing poses. In contrast, IRVAE does not produce such poses irrelevant to the two given start and end poses. The IRVAE produces more smoothly varying poses than VAF, and the pose interpolation obtained by the IRVAE is closer to the geodesic interpolation along the pose data manifold.
E.2 DIVERSE IMAGE DATA WITH HIGHER LATENT SPACE DIMENSIONS
We provide additional results with diverse image data (MNIST, FMNIST, SVHN, CIFAR-1O) using various latent space dimensions. Figure & shows that the tradeoff curves of IRVAE are located
16

---

Published as a conference paper at ICLR 2022
VAE
Translation along each latent axis
IRVAL
Original Pose (Walking)
Translation Axes
24
Z2
Zg
24
25
26
Z7
Za
Figure 6: The original pose is encoded in the latent space, then the encoded latent value is translated along each latent space axis (21, 22, 2g). The translated latent values are decoded back to generate new eight pose for each model VAE and IRVAE. The translated distances are proportional the standard deviations of the encoded training data.
Walking
Linewr Interpolation
Balancing
VAE
IRVAE
Figure 7: Pose interpolations between a walking pose and balancing pose with linear interpolations in the latent spaces. The red box indicates suddenly appeared punching poses.
lower than those of FMVAE and ii) the FM learns more isometric representations without losses in MSEs.
Dataset:
The number of training, validation, test data is 50,000, 10,000, 10,000 for MNIST, FM- NIST, SVHN, and 45000, 5000, 10,000 for CIFAR-10.
Network Architecture: Let denotes the latent space dimension. For VAE, FMVAE, IRVAE, encoders are Convolutional Neural Networks with the following architecture: i) Conv2d (3, 32, 2, 0), ii) Conv2d (32, 64, 4, 2, 0), iii) Conv2d (64, 128, 4, 2, 0), iv) Conv2d (128, 256, 2, 2, 0), v) Conv2d (256, 2m, 1, 1, 0) with ReLU hidden layer activation functions, and decoders are Convolutional Neural Networks with the following architecture: i) ConvT2d (m, 256, 8, 0) ii) ConvT2d (256, 128, 4, 2, 1) iii) ConvT22 (128, 64, 4, 2, 1) iv) ConvT2d (64, 3, 1, 1, O) with ReLU hidden layer activation functions. The output activation functions are linear and sigmoid for encoder and decoder, respectively.
For FM, we the RealNVP model of depth and length 512.
Other Details: For FMVAE and IRVAE, the batch size is 100, the number of training epochs is 100, the learning rate is 0.0001, and 7) = 0.2. For FM. the batch size is 100, the number of training epochs is 100, the learning rate is 0.00001, and n7 = 0.2. We use Adam optimizer. Validation sets used to determine optimal models during training.
E.3 ABLATION STUDY ON MIXUP PARAMETER
In the isometric regularization (and flattening module), we use the augmented encoded data distribu- tion Pz whose sampling procedure is given follows: (i) encode data F,, Zy to 2,, 2j by the encoder (and by the invertible map composed with the encoder) and (ii) compute a new latent value & with dz, 0)2z for 5 uniformly sampled form [-7, 17], where 1) is the mixup parameter.
17

---

Published as a conference paper at ICLR 2022
MNIST (latent dim
FMNIST (latent dim 16)
SVHN (latent dim 64)
CIFAR1O (latent dim 128)
0.25
0300
60239
03370
00025
0.13
00110
0.00
0.000
14
180
200
750
VoR
V6R
VoR
V6R
01010
6.00
0.133
0005
00375
0000
03154
0312
00029
600
30
MCN
MCN
MCN
MCN (lop)
-MWWE
IRWAS
IKWN +M
Figure 8: The tradeoff curves of FMVAE, IRVAE, and IRVAE FM.
Then, our relaxed distortion measure is defined over the support of Pz, i.e.
Egn Ps
E [I
TE EE EE (x) 1)²],
Xx(z)
where f is the decoder, Jy is the Jacobian of f, and A (=) are eigenvalues of (z)). The mixup augmentation technique is used to extend the influence isometric regularization an area where data does not exist.
In this section, we visually analyze the effects of the mixup augmentations in a range of mixup pa- rameters We use the CMU data with fully connected networks described in D.2 and 2-dimensional latent spaces.
Figure 9 shows the latent space data distributions with some equidistance ellipses (the colors of the ellipses represent the condition numbers; redder-the-bigger) In no mixup case, there are many un-isotropic red ellipses in regions between data points, meaning that they are not isometrically reg- ularized. Mixup augmentation makes it possible to regularize even regions between data. When the mixup parameter 77 O, only the area covered by data interpolation is regularized. By increasing the mixup parameter y,, the outer area, which is covered by data extrapolation, begins to be reg- ularized. Although using bigger eta seems be unconditionally good because the wider area is regularized, there is indeed tradeoff as we can see from Figure 9 When the mixup parameter is big as y 1, some inner area (i.e., area covered by data interpolation) is sufficiently regu- larized because of the cost of regularizing the outer domain. We use a balanced value 1/ 0.2 for all experiments conducted in this paper (for all algorithms IRVAE, FM and FMVAE).
walking
balancing
Jogging
punching
No Mixup
Mixup 2)
Mixup 17
Mixup
Mixup r 1.0
Figure 9: The effect of mixup augmentation with varying mixup parameters
E.4 COMPUTATIONAL. SPEED
In this section, we provide per-epoch runtimes of VAE, FMVAE, IRVAE, and FM. We use the MNIST data and same experiment settings in D.I. We use the GeForce RTX 3090 for GPU
18

---

Published as a conference paper at ICLR 2022
sources. 2 shows the per-epoch runtimes. The difference in computation times arises from the computations of the regularization terms. The IRVAE takes a little longer than the FMVAE because, while the regularization term in FMVAE requires compute the Jacobian vector product only, the regularization term in IRVAE requires to compute both the Jacobian vector product and vector Ja- cobian product. In the case of FM, it takes much longer because the invertible neural network architecture is relatively heavier. Fortunately, in training of FM that uses the pre-trained IRVAE, empirically observed that only few epochs are required for convergence, probably because the IRVAE already had learned isometric representations to certain degree.
Table Averages and standard deviations the per-epoch runtimes. 50,000 MNIST image training data used with 100 batch size. For VAE, FMVAE, IRVAE, we use the 4 layers of fully connected neural networks, and for FM, we use the RealNVP of depth 8.
VAE
FMVAE
IRVAE
FM
per-epoch runtime
1.91 0.0818 2.80 0.0755 8 2.81 0.0869
19.9 0.235 s
E.5 ISOMETRIC REGULARIZATION FOR OTHER AUTOENCODERS
Our isometric regularization algorithm can be used with other types of autoencoders although we focus on the Variational Autoencoder (VAE) in the main manuscript. In this section, verify the effects of the isometric regularization with more diverse autoencoder methods: Vanilla Autoencoder (AE) (Kramer 1991). Wasserstein Autoencoder (WAE) (Tolstikhin et al. 2018). and Denoising Autoencoder (DAE) (Vincent et al. 2010
We use the MNIST data and same experiment settings in D.1. For DAE, the Gaussian noise with standard deviation of 0.01. For WAE, we use the maximum mean discrepancy, median heuristic for bandwidth selection, and the regularization coefficient 0.001.
Figure 10 shows the effects of the isometric regularization to diverse autoncoders (AE, VAE, DAE, WAE). Regardless of the autoencoder types, we consistently observe that the added isometric reg- ularization terms effectively separate data of different classes (even without major tuning of the regularization coefficient o.).
AE
VAE
DAE
WAE
Without Regularization
With Isometric Regularization
Figure 10: Isometric regularization with diverse autoencoder methods. The more homogeneous and isotropic equidistance plots are, the more isometric the representations are.
19

---

Published as a conference paper at ICLR 2022
E.6 ISOMETRY VS SCALED ISOMETRY
In this section, show empirical results that support claim in the introduction that finding scaled isometry is better than finding strict isometry.
Whereas an isometry exactly preserves angles and distances, scaled isometry preserves angles and scaled distances. At first this may seem a superficial difference why not simply choose length scales that the two spaces have the same scales? but the reason this difference is consequential for our problem is that we do not a priori have a precise charcterization (and hence their relative scales) of the two spaces. We are effect discovering the manifold structure of the data space while constructing a latent space representation for the data manifold at the same time.
To characterize the above more precisely, let f(z) be decoder, 22(2) be the Jacobian of f(z), and (2(2) be the augmented encoded data distribution. Let (= be eigenvalues of Jf (z)J} Let (2)) be the latent space metric, and (z) I. Recall that the isometry ff needs satisfy (=) G(2) for all in the support of Pz. whereas the scaled isometry needs to satisfy (z) cG(z) for some e > and all the support Pz. To find a strict isometry, we need set scalar parameter k that determines the latent space metric, i.e., G(z) = kI, in advance, whereas, to find scaled isometry, we just need to set G(z) T as done in the paper.
To find the isometry, we add the following distortion measure as regularization term:
&BE-PZ (()())))))
where k is the pre-defined parameter and is the regularization coefficient. We need to search for the optimal parameter in this case.
In contrast, do not need such parameter & to find scaled isometry. We just need to add the following relaxed distortion measure as a regularization term:
(x)2 B {x) - 1)3],
aEgoP
where a is the regularization coefficient.
Figure 11 shows the tradeoff curves (lower-the-better) obtained by using (i) the relaxed distortion measure (for finding a scaled isometry) or (ii) the distortion measure with a range of user-defined parameters & (for finding strict isometry). For the strict isometry learning case, it can be seen that the performance peaks when the value of & is at an appropriate level that is neither too high nor too low. In this experiment, the optimal lies in between 10 and 1000 and can be experimentally found only by performing a finer grid search. The tradeoff curves obtained by using the relaxed distortion measure are always located lower than the tradeoff curves obtained by using the distortion measures. From this experimental result, we conclude that it is better to find a scaled isometry than strict isometry.
6000
Isometry (k = 0.1)
Isometry (k 1)
0050
Isometry (k 10)
MSE
106
8045
Isometry (k = 100)
Isometry (k 1000)
6032
Isometry (k = 10000)
Scaled Isometry
VoR (log)
MCN (log)
Figure 11: Tradeoff curves obtained by changing the regularization coefficients a (lower-the-better)
20

---

Published as a conference paper at ICLR 2022
E.7 MULTIPLE TIMES RUN TO COMPUTE AVERAGES AND STANDARD DEVIATIONS
is important to judge whether the experimental result is a result of randomness or not. In this section, we take the same experiments done in Figure and of the main manuscript with MNIST and CMU data multiple times, and report the averaged tradeoff curves and standard deviations of the measured metrics.
Figure 12 shows the results, where the standard deviations computed over multiple experiments with different random seeds are visualized as ellipses. The horizontal axes lengths the ellipses represent the standard deviations of the VoR and MCN, and the vertical axes lengths of the ellipses represent the standard deviations of the MSE. It correct to use the standard errors see the standard deviations of the sample means, i.e., standard deviations divided by the square root of the number of experiments, but we use the standard deviations since the standard errors are too small to visualize. The sizes of the ellipses are rally small, which provides solid evidence that IRVAE outperforms FMVAE.
MNIST (Figure 1)
40
40
FMVAE
30
30
(averaged tradeoff curve)
MSE
36-
36
IRVAE (averaged tradeoff curve)
38
10'7
33
10a
10'
2x
Standard
VoR (log)
MCN (log)
deviations of the measured metrics
CMU (Figure 3)
1e-4
28
2.8
24
2.6
244
2.4
MSE
22-
22
20-
2.-
18
01
0.3
1.5
VoR
MCN
Figure 12: Averaged tradeoff curves and standard deviations represented as ellipses for MNIST and CMU experiments in Figure/ and 3 of the main manuscript (20 times run). We wanted to draw ellipses by using the standard errors, but they were too small to visualize. Even standard deviations really small.
E.8 SENSITIVITY ANALYSIS OF THE LATENT SPACE DIMENSION
In general, when training an autoencoder, the most important parameter is the latent space dimen- sion. Although there have been some works on estimating the intrinsic dimension of the data dis- tribution (Levina Bickel 2005). given complex nigh-dimensional data, the intrinsic dimension estimation problem is very challenging; in practice, appropriate value is often found, empiri- cally. Throughout literature, there are some frequently-used choices for well-known standard image datasets. For relatively simple datasets such the MNIST and FMNIST, latent space dimensions 2 32 are often used, and for more complex and higher dimensional datasets such as the CIFARIO and CELEBA, latent dimensions 64 512 are often used.
In this section, using the MNIST image data with the fully connected network described in D.1 (trained for 100 epochs), provide analysis of the isometric regularization in autoencoders with multiple latent space dimensions. Table 3 shows averages and standard deviations of the condition numbers of the pullback metrics J7 (=)J (f the decoder and J, is the Jacobian of computed
21

---

Published as a conference paper at ICLR 2022
with the test data. Both the averages and standard deviations should be small if (F = cl for some > 0 for all encoded test data
In the case of VAE, it can be seen that both the average and standard deviation of the condition number sharply increase as the latent dimension increases. On the other hand, IRVAE appears to be able to effectively find an isometric representation even the dimension of the latent space increases. In the case of 16-dimensional latent space, the condition number of IRVAE is about 32 which is not small enough, and this result may imply that the true intrinsic dimension of the MNIST data manifold is lower than 16.
In summary, our isometric regularization can effectively find more isometric representations than vanilla autoencoders regardless of the latent space dimensions. When the dimension of the latent space is too large, it may not be possible to obtain sufficiently isometric representation space. This This seems because the dimension of the ground true manifold is much smaller than the selected latent space dimension.
Table 3: Averages and standard deviations of the condition numbers of the pullback metrics for VAE and IRVAE with respect the latent space dimension. The lower the average and standard deviations are, the more isometric the latent is.
Latent Space Dimension
VAE
IRVAE
2
5.39 + 4.72
1.41 0.55
8
85.55 + 85,69
1.87 + 0.25
16
1635018.8$ + 1789834.25
37.28 42.12
22

---

Published as a conference paper at ICLR 2022
F DETAILED RESULTS FOR SINGLE ATTRIBUTE RETRIEVAL
Table 4 shows the detailed retrieval results (P@ 10 of the single attribute retrieval). Our algorithms (IRVAE and IRVAE + FM) have much higher performance than FMVAE and VAE, and even show performance close to the supervised method (BR).
Table 4: Single attribute retrieval results of Precision at 10, @@ 10 (for total 40 attributes). The best results among unsupervised methods colored red, and the best results among all six methods marked bold.
IRVAE (ours)
IRVAE FM
Pre-trained (ResNet-50)
Supervised
Attributes
VAE
FMVAE
(ours)
(BR)
Young
0.8
0.7
0.8
0.7
Wearing Necktie
0.4
0.7
0.6
0.7
0.1
0.5
Wearing Necklace
0.2
0.1
0.3
0.2
0.1
0.1
Wearing Lipstick
1
0.9
1
1
0.4
1
Wearing Hat
0.4
0.7
0.8
0.9
0
1
Wearing Earrings
0.5
0.5
0.7
0.6
0.4
1
Wavy Hair
0.8
0.9
0.9
0.9
0,3
0.9
Straight Hair
0.9
0.2
0.7
0.7
0.2
0.5
Smiling
1
1
1
1
0.7
1
Sideburns
0.4
0.7
0.9
0.2
0.6
Rosy Cheeks
0,6
0.5
0.8
0.7
0
1
Receding Hairline
0.3
0.8
0.6
0.5
0.8
Pointy Nose
0.5
0.6
0.4
0.8
0.2
0.8
Pale Skin
0.3
0.8
0.7
0.7
0.1
0.8
Oval Face
0.5
0.6
0.5
0.4
0.5
0.6
No Beard
0.8
1
1
1
0.7
1
Narrow Eyes
0.5
0.1
0.3
0.5
0.1
0.7
Mustache
0.2
0.3
0.3
0.4
0.1
0.3
Mouth Slightly Open
1
1
1
0.6
1
1
Male
1
1
0.4
1
High Chockbones
1
1
0.9
1
0.4
1
Heavy Makeup
1
0.8
1
1
0.3
1
Gray Hair
0.5
0.7
0.7
0.8
0
0.7
Goatee
0.5
0.6
0.7
0.9
0
0.7
Eyeglasses
1
0.9
1
1
0.1
1
Double Chin
0.4
0.3
0.6
0.8
0.1
1
Chubby
0.3
0.1
0.3
0.4
0
0.9
Bushy Eyebrows
0.6
0.5
1
1
0.3
0.9
Brown Hair
0.5
0.3
0.8
0.6
0.2
0.8
Blurry
0
0.1
0.2
0.1
0.1
0.4
Blond Hair
1
1
1
0.1
0.9
Black Hair
0.4
0.9
0.9
1
0.2
0.8
Big Nose
0.4
0.6
0.5
0.5
0.1
0.6
Big Lips
0.4
0.6
0.2
0.5
0.3
0.3
Bangs
1
0.8
1
1
0
1
Bald
0.3
0.4
0.6
0.8
0
0.9
Bags Under Eyes
0.5
0.4
0.6
0.7
0.4
0.7
Attractive
0.9
0.9
0.5
1
Arched Eyebrows
0.8
0.7
0.7
0.8
0.2
0.9
o'lock Shadow
0.7
0.5
0.7
0.6
0.3
1
Number of REDS
14
15
18
25
Number of BOLDS
12
15
15
19
0
28
Average Precision
61.0
64.0
71.5
75.3
23.5
80.3
23


Reviews:
[{"review_id": "mNj8JNH9Z9s", "reviewer": "Reviewer_5xzF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear and well-motivated. The problem of introducing geometry properties to the latent space is challenging and not much explored. The proposed regularization method is convincing and well justified. The idea of coordinate invariant functional is novel (for the best of my knowledge) and theoretically sound. \nThe paper is well organized. The state of the art is complete. The description of method is clear. It would be nice to see a graphical scheme of the method architecture to give a visual explanation to the meaning of the involved functions and their role to move from one space to the other.\nExperiments are convincing. The experiments on the most simpler scenarios like MNIST and CMU motion capture are useful to understand the behavior of the proposed approach. The experiment on the most challenging dataset, i.e., unsupervised human face retrieval is important to appreciate the improvement of the proposed approach against other methods.    \nThere are some important points that authors should clarify:\n-In the introduction, authors claimed that using a less stringent regularization term on isometry is more helpful, but than is not clear to identify other parts of the paper that support this claim. Please clarify with more examples and details. \n-It is not clear the role of the stabilizing latent space flattening described in Sec 4.3. If this term is relevant it should be described in Eq. (11).\n-Although it is already clear that the proposed method is better than FMVAE, authors should explain more details on that, especially in highlighting the importance of a coordinate-invariant strategy.\n-h in Eq. (7) is different than h in Sec. 4.2, perhaps it would be better to change notation…,\n-It would be nice to see the benefit of the proposed method for data generation. Usually to evaluated the regularity of the latent space,  the generation of data in between two samples is visually shown. Is it possible to show something similar to appreciate the benefit of the geometric constraints?\n-Is the code available? More details on the used software and performance in terms of speed of method will be appreciated. \n", "summary_of_the_paper": "The paper proposes a new regularization approach to introduce geometric constraint to the latent space of an autoencoder. A hierarchy for geometry-preserving mappings is formulated to clarify how strong this constraint can be defined. In particular, authors focused on scaled isometries, i.e., maps that preserve angles and distances up to some scale factor. This scale is learnt together with the manifold and the latent space representation during the training of the autoencoder. This approach was proposed also in (Chen et al., 2020) for the so called FMVAE, but here a new coordinate-invariant regularization term is introduced that measure how close the decoder\nis to being a scaled isometry. Finally, a post-processing flattening procedure is introduced to further improve the geometry properties of the latent space. \nSeveral results have been reported on different applicative scenarios showing a clear improvement of the proposed approach in comparison with other methods of the state of the art .      \n", "summary_of_the_review": "\nThe proposed method is novel and convincing.\nThe paper is well organized and clear.\nExperiments shown that the proposed approach clearly overcome other methods.  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T1ALZFbeYFG", "reviewer": "Reviewer_YhcR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe idea of studying geoemetry-preserving regularization while learning representations is a challenging and an important topic with a large potential impact.\n\nCons:\n\n1) In section 4.2 the paper talks about using a seperate invertible mapping from R^m -> R^m that essentially looks at not affecting the reconstruction accuracy and only improves the isometric representation. I am not sure why this can not be already captured by the original decoder that takes an input point in R^n-> R^m\n\n2) The experiments show that the isometric regularization and FM leads to marginal or no loss in reconstruction accuracy. In the implementation H(x) is assumed to be identity and this is a strong assumption. As the authors acknowledge, it would be more useful to use domain-specific knowledge. However, by assuming identity for H(x) the regularization gets closer to Jacobian regularization that is already a well studied problem:\n\n  Rifai, S., Dauphin, Y. N., Vincent, P., Bengio, Y., and Muller, X. The manifold tangent classifier. In Advances in Neural Information Processing Systems, pp. 2294–2302, 2011a. \n\n   Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., and Glorot, X. Higher order contractive auto-encoder. In \nECML-PKDD, pp. 645–660. Springer, 2011b.\n\n   Hoffman et al. Robust Learning with Jacobian Regularization, 2019.\n\n3) The visualization in Fig. 1 does not clearly show the improvement using the isometric regularization approach shown in the paper. \n\n4) There should be some ablation on the mixup regularization that is used in this paper. It is not clear whether the improvement is mainly from mixup or the proposed regularization.\n\n5)  The paper does not look at sufficient strong baselines for representational learning methods. It only uses VAE and FMVAE. ", "summary_of_the_paper": "This paper studies the aspect of preserving geometry on the learned latent space representations. In particular, the paper looks at a hierarchy of geometry-preserving mapping (isometry, conformal mapping of degree k, area preserving mapping, etc.). Existing popular methods such as SimCLR pays limited attention to preserving geometric relationships. The paper shows that the mapping that preserves angles and relative distances is better than the ones that preserve angles and absolute distances. The main contribution of this paper is to propose a representational learning technique that uses a reconstruction loss and an isometric regularization term. This allows us to learn embeddings that satisfy isometric properties while not suffering from any reconstruction loss or just marginal reconstruction loss. The comparison is done with VAE (Kingma & Welling, 2014) and FMVAE (Chen et al. 2020). Experiments are shown on CelebA with 40 annotations.", "summary_of_the_review": "Overall, the paper addresses an important problem but there are many concerns with the experiments and the formulation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mqex2fDcRxr", "reviewer": "Reviewer_i9fR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper shares an insight that whenever the mapping between latent space and data space in a generative model (e.g. VAE) approximates an isometry (or a scaled isometry, as proposed), one observes better results in a range of experiments. I am missing the key motivation behind this, perhaps also due to the paper itself mentioning (beginning of page 2) that this kind of requirement is too much to ensure good reconstruction. What is the rationale behind this? One unclear aspect to me springs from the fact that, in general, an isometry does not even exist between arbitrary manifolds, for example, one can not isometrically embed a sphere into a Euclidean space of any dimension. Why should a quasi-isometry lead to better generative models? Why isn't a homeomorphism enough?\n\n- What I think is really missing, is a so called \"killer application\" that would convincingly show the benefits of the proposed regularization. In particular, it is important to demonstrate that isometric representation learning is relevant, and that it can lead to significant improvements in many contexts. By contrast, the current experiments show some good results on simple data. Without a strong motivation, or a strong argument in support of the proposed method, it is hard to judge how potentially impactful it can be within the broader research field.\n\n- I would expect the choice of dimensions D and m to be important, since the distortion measure can vary greatly among different embeddings. How were these parameters chosen? I suggest to include some sensitivity experiments showing the influence of these dimensions on the final results, together with a discussion on their effect. Figure 6 in the supplementary goes in this direction, but I am missing an interpretation of these plots. Similarly, I could not find a sensitivity experiment showing the influence of parameter k (degree of the conformal mapping).\n\n- What is the consequence of choosing the identity metric for the data space?\n\n- The proposed measure (eq.7) is novel to my knowledge and is well described.\n\n- The manuscript is very pleasant to read, it is fluent, solid, and clear from start to end. The mathematics are sound and accessible. The writing is sharp.\n\n- The idea behind the paper is novel to my knowledge, but I found a few related works in recent conferences, see e.g. \"LIMP: Learning Latent Shape Representations with Metric Preservation Priors\" (ECCV 2020), it might be worth checking these out.\n\nA few side-comments:\n\n- It would be nice, although not strictly necessary, to visually illustrate the difference between scaled isometries of different order, perhaps by texture mapping a checkerboard pattern between 2D surfaces embedded in 3D if at all possible.\n\n- How costly is the flattening step (solving eq.11)?\n\n- How should the equidistance plot be read? The more isotropic, the better?\n\n- The paper mentions a few time some \"popular\" choices for the h() function. To what literature does the popularity refer to?\n\n- There is an incomplete sentence in page 5 \"inherent tradeoff We introduce\".\n\n- At the end of page 5, \"the the\".", "summary_of_the_paper": "The paper makes two main contributions. The first is the introduction of a new regularizer term for the VAE loss, ensuring a (scaled) isometry between the learned latent space and the (typically unknown) data space. The second is a post-processing \"flattening\" step to improve the isometry constraint by directly operating on the latent space, while leaving the reconstruction error untouched. The resulting pipeline seems effective, as showcased on a selection of experiments over standard datasets.", "summary_of_the_review": "A well-written manuscript with a practical implementation of an isometric representation learning pipeline, which is nonetheless counterbalanced by unclear motivations and not very impressive results. Might still have an impact on a certain community, but a potentially broader impact is not clear at this stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5scvzWXHZD", "reviewer": "Reviewer_ZGHS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nI appreciate the theoretical motivation and foundation for the method proposed. Its justification and development were clear enough to follow. The experimental results in an unsupervised image retrieval task show a noticeable improvement in performance measured by multiple metrics when compared to a strong baseline. Furthermore, the tradeoffs between achieving a scaled isometry and balancing the reconstruction error are on the side of the proposed method, as evaluated on image and motion capture data.\n\n**Weaknesses:**\n\nWhile I mentioned most experimental results are convincing, I still think the paper has some considerable flaws to be addressed.\n\nFirst, the experimental results do not seem averaged over multiple random seeds. Additional information such as standard deviation, would be helpful in assessing how stable the training is and how reliable are those results for someone attempting to reproduce the experiments. The same applies to the curves shown in Figures 1 and 2, where the difference to the baseline is marginal. I see this as critical for the paper to be accepted.\n\nSecond, there seems to be a conflict between the KL-divergence regularization of a VAE w.r.t. changes introduced to the latent space caused by the newly introduced regularization scheme. I believe there should be a discussion on this, since knowing $P_Z$ seems to be necessary and without the KL term it is hard to say it follows a standard Gaussian.\n\nThird, I missed a discussion with other perspectives, such as from [1], where instead of changing the latent space that can be learned, one considers different ways of \"navigating\" it as is.\n\n**Questions for the authors:**\n\n- I found the first listed contribution to be problematic, which says the authors define a family of coordinate-invariant regularization terms. It strikes me as straightforward since it seems to be a by-product of the necessary mathematical foundation for the method. The family of functions itself was not really explored, just one specific case. Could you make it clearer how novel is that family of regularization terms w.r.t. existing work on this?\n- Regarding the generative performance of the models in question. While I understand this might not be the main goal of the paper, the methods used as starting point (VAE) do present the characteristic of being able to generate new data points. For instance, because we change the structure of the latent space, how does this affect the generative aspect of the model?\n- Why is it the case that IRVAE outperforms IRVAE+FM w.r.t. to P@1 by so much, but this does not happen in any other case?\n- What are the actual impacts of the additional computational costs of the regularization procedure, in terms of total training time compared to not using them?\n\n**Additional minor comments:**\n- Section 2 seemed more like a subsection, as it only deals with introducing some concepts and relating them to each other, without carrying any major point on its own.\n- There are some problems with notation, specifically the function h which seems to have multiple meanings (c.f. Eq. (6) and Eq. (11))\n- It is also usual to also credit [2] for VAEs.\n- Section 5.1: what is considered a sufficiently large number of samples for computing MCN and VoR?\nFigures 1 and 2 show categorical information (classes of data points) using a continuous color scale. While this does not affect interpretation, I believe a categorical color scale would improve readability.\n- References seem to need review. For instance, the first and fifth are from ICML 2020 and ICLR 2017, respectively, but this information is missing.\n\n[1] Arvanitidis, Georgios, Lars Kai Hansen, and Søren Hauberg. \"Latent Space Oddity: on the Curvature of Deep Generative Models.\" International Conference on Learning Representations. 2018.\n\n[2] Rezende, Danilo, and Shakir Mohamed. \"Variational inference with normalizing flows.\" International conference on machine learning. PMLR, 2015.\n", "summary_of_the_paper": "The paper introduces a regularization scheme that constrains autoencoders to produce latent spaces which are isometrically closer to the input space, up to a certain scale factor, that is, a latent space whose pullback metric is closer to a scaled identity matrix. The paper further shows both (i) the tradeoff between two surrogate metrics for isometry and reconstruction accuracy, showing how much reconstruction error is affected by increasingly stricter regularization and (ii) that the representations learned by a variational autoencoder with such a regularizer results in better performance on a retrieval task when compared to existing baselines.", "summary_of_the_review": "The paper follows a solid theoretical foundation using (scaled) isometry between input and latent space as a principle for designing a regularizer to learn meaningful representations with autoencoders. Experimental results show improvements over relevant baseline methods in an unsupervised retrieval task and as tradeoffs between reconstruction accuracy and surrogate metrics for how close the learned latent space is to an (scaled) isometry. These positive aspects, however, are overshadowed by key major issues: (i) results are not averaged over multiple random seeds, which makes it difficult to assess the stability of the regularizer in training; (ii) and missing discussion of conflicting requirements of the KL divergence term with the one introduced, making the method analysis seem too superficial.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0uv9jZUw4ko", "reviewer": "Reviewer_MiWx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address an important problem in unsupervised learning, namely, manifold learning using a NN. The method proposed in the paper extends two recent works by Chen et al. and Jang et al. They propose an Isometric Regularization that is realized based on Hutchinson’s stochastic trace estimator to enable efficient estimation of the gradients. The English level is good, but the description of the method is hard to follow. It would be helpful if the authors could add a pseudocode of the proposed methodology. The authors demonstrate that the proposed method leads to several benefits compared to the existing method. However, I have several concerns about the proposed method. Specifically, my major comments are:\n-The method proposed here is a variational autoencoder; why is it presented in the introduction as an autoencoder?\n-On the same point, the variational part is not well justified; I had to read Chen et al. to understand why the variational part is included.\n-I am not convinced that the method leads to any benefit on MNIST. The MSE gain compared to FMVAE is really marginal and could result from non-optimal hyperparameter tuning. \n-How are the hyperparameters of all methods tuned?\n-If I understand correctly, the latent dimension used in the paper is 2; can you use a higher dimension? I would expect that for MNIST and other image datasets, a higher latent dimension is required.\n-The equidistance ellipses are not well defined, are these computed in the ambient space and plotted in the latent space? If this is the case, I am not sure that preserving large Euclidean distances is “good”. In high dimensional space, we should mostly trust small distances and not large ones. Therefore, these ellipses should be related to a small distance in the ambient space. The scale of these ellipses is not provided.\n-The improvements in the retrieval task do seem substantial; this supports the effectiveness of the method. However, the way the hyperparameters of all methods are tuned is not explained.\nMinor comments:\n-”Learning the data manifold”- this term is not well defined in the paper. Perhaps the authors mean “preserve the data manifold”. This is not clear; learning the manifold suggests learning its underlying parametric representation. \n -Before the words “We introduce” on page 4, a period or comma is missing; also, the sentence does not flow.\n-Figures 1 and 2, the MSE vs. MCN and MSE vs. VoR from both of these plots are redundant; you could only leave the ones in figure 2; the information is redundant.\n-What are the colors in the embedding? The labels? If so, the separation is not improved between classes.\n-Please mention in the caption of figure 2 on what data is this experiment evaluated.\n-The condition number calculation for figure 4 is not clearly explained; please add some intuition.\nThe following works have very similar ideas and should be cited:\n\n\n[1] McQueen, J., et al.  (2016). Nearly isometric embedding by relaxation. Advances in Neural Information Processing Systems, 29, 2631-2639.‏\n[2] Peterfreund, Erez, et al. Local conformal autoencoder for standardized data coordinates. Proceedings of the National Academy of Sciences, 2020, 117.49: 30918-30927.‏", "summary_of_the_paper": "The authors propose a new type of regularized autoencoder (actually variational autoencoder) that is designed to preserve the geometry of the data. They demonstrate that preserving angles and relative distances lead to an improved representation of the data. Specifically, they add a regularization and explore the tradeoff between reconstruction and geometry preservation. They further propose a scheme to flatten the latent representation in a postprocessing fashion.", "summary_of_the_review": "To summarize, the authors address an important problem and propose a new method for learning a low-dimensional representation from high-dimensional data. They rely on the manifold assumption and propose a new regularized to a variational autoencoder. The results demonstrate that the method leads to improvement in retrieval tasks. However, the method has some limitations (namely the low dimensional embedding dimension), and the experiments are not backed up by a rigorous scheme for hyperparameter tuning. For these reasons and the marginal improvements demonstrated in figures 1-3, I recommend a weak rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension to learning a representation: it motivates, proposes and evaluates a new regularizer term that promotes smoothness via enforcing the representation to be geometry-preserving (isometry, conformal mapping of degree k). Comparisons with a standard VAE and FMVAE (Chen et al. 2020) are shown and experiments are provided on CelebA with several different attributes as target classification tasks. 

The paper has received extensive reviews and the authors have successfully answered most of the concerns raised, mostly regarding comparisons to other techniques that try to introduce a regularization based on the properties of the Jacobian of the decoder network. 
The appendix has been extended as a result of the rebuttal and the paper could be accepted.

Notes: 
I find the formulation based on the notion of the isometric decoder somewhat surprising as the encoder is a key object of interest that controls the nature of the representation. The authors should clarify the assumption 3 in 3.3 better by the consideration of potentially $dim(z) << dim(x)$, how the isometry of the decoder effects the encoder, 

Additionally, for the latent space flattening an ablation using SVD (merely a linear mapping for $i(\cdot)$) could be considered.

Reviewer ZGHS has noted that they raise their grade to 6 in their comment, but this is still not currently reflected.
Decision: Accept (Poster)