Regularized Autoencoders for Isometric Representation Learning

The recent success of autoencoders for representation learning can be traced in large part to the addition of a regularization term . Such regularized autoencoders “ constrain ” the representation so as to prevent overfitting to the data while producing a parsimonious generative model . A regularized autoencoder should in principle learn not only the data manifold , but also a set of geometry-preserving coordinates for the latent representation space ; by geometry-preserving we mean that the latent space representation should attempt to preserve actual distances and angles on the data manifold . In this paper we first formulate a hierarchy for geometry-preserving mappings ( isometry , conformal mapping of degree k , areapreserving mappings ) . We then show that a conformal regularization term of degree zero – i.e. , one that attempts to preserve angles and relative distances , instead of angles and exact distances – produces data representations that are superior to other existing methods . Applying our algorithm to an unsupervised information retrieval task for CelebA data with 40 annotations , we achieve 79 % precision at five retrieved images , an improvement of more than 10 % compared to recent related work . 1 INTRODUCTION . Learning a good representation for high-dimensional data is one of the most fundamental problems in machine learning . A good representation should capture all essential information about the data in a parsimonious manner while filtering out all non-essential variations . Clearly what is “ essential ” or “ non-essential ” depends heavily on the end task ( Tschannen et al. , 2018 ) , and numerous criteria have been proposed for a range of contexts , e.g. , disentanglement ( Higgins et al. , 2017 ; Zhao et al. , 2017 ) , clustering structure ( Makhzani et al. , 2015 ) , sparsity ( Lewicki & Sejnowski , 2000 ) , hierarchy ( Gulrajani et al. , 2017 ) , and isometric embedding ( McQueen et al. , 2016 ; Peterfreund et al. , 2020 ) . In this paper we take the view that the essential information is best captured by the geometry of the data . More specifically , we adopt the manifold hypothesis as our point of departure , and further argue that a good representation should also preserve the geometry of the data manifold . That is , nearby points on the manifold should have representations in the latent representation space that are also nearby , and angles and volumes should be preserved as much as possible when moving between the data manifold and its representation space . To find such a representation , it is important to ( i ) learn the correct low-dimensional data manifold , and ( ii ) to find an optimal set of latent space coordinates that preserves the geometry of the learned data manifold . A primary reason that autoencoders are widely used for unsupervised representation learning is that they can learn both the manifold and the latent space coordinates simultaneously during the training phase . Vanilla autoencoders trained purely to minimize reconstruction loss tend to overfit , and the learned manifolds are often inaccurate . As demonstrated in ( Rifai et al. , 2011b ; Kingma & Welling , 2014 ; Makhzani et al. , 2015 ; Tolstikhin et al. , 2018 ) , by augmenting the reconstruction loss with a regularization term , manifold learning performance of autoencoders can be significantly enhanced . However , little if any consideration has been given to the concurrent problem of learning a set of latent space coordinates that preserves the geometry of the data manifold ( with the exception of FMVAE ( Chen et al. , 2020 ) , which we discuss further below ) . The main contribution of this paper is a regularized autoencoder that simultaneously learns the data manifold and a geometry-preserving set of latent space coordinates . For this purpose , we first formulate a hierarchy of geometry-preserving mappings . At the top of the hierarchy are isometries , which preserve distances and angles , followed by conformal maps , which preserve angles , and then area-preserving maps . Of particular relevance to this paper are the conformal maps , which we further stratify into degree k conformal maps , with k = 0 corresponding to scaled isometries , i.e. , maps that preserve angles and distances up to some scale factor . Based on this hierarchy of mappings , we then formulate a corresponding hierarchy of regularization criteria for training the autoencoder . One of the important findings of our study is that somewhat counterintuitively , using a regularization term that measures the nearness to an isometry is in fact detrimental ; such a regularization term overly constrains the mapping , resulting in a higher reconstruction loss whose effects can not be mitigated even with adjustments to the regularization term weight . Rather , using a less stringent regularization term is more helpful . In our examples the degree zero conformal maps , or scaled isometries , seem to offer the best balance between reconstruction accuracy and model parsimony . We note that if the exact data manifold were known in advance , then the scale factor could be precomputed , e.g. , to make the latent space and the data manifold have the same volume , in which case it would make sense to use a normalized version of the isometry measure as regularization term . When using an autoencoder for representation learning , however , the data manifold is not known a priori but rather learned together with the latent space representation . The more effective alternative , we argue , is to learn this scale together with the manifold and the latent space representation . The FMVAE of ( Chen et al. , 2020 ) is in fact the first work to adopt this approach , but the measure does not adequately capture the nearness to a scaled isometry , and is also not coordinate-invariant , limiting its performance as we show later in our experiments . Once the data manifold and an initial set of geometry-preserving latent space coordinates are learned , it is possible to further “ flatten ” the latent space , by adding a postprocessing step that leads to an even more isometric set of coordinates . Specifically , we use an invertible neural network model to map the pre-trained latent space to a more isometric representation space , without incurring any further losses in reconstruction accuracy . Experiments on diverse image and motion capture data confirm that , compared to existing related methods , our geometrically regularized autoencoder produces more isometric representations of the data while incurring only minimal losses in reconstruction accuracy . In particular , information retrieval task experiments conducted with CelebA data show that data similarity measurements performed in our representation space lead to significantly improved levels of retrieval performance . Our specific contributions can be summarized as follows : • We define a family of coordinate-invariant regularization terms that measure how close the decoder is to being a scaled isometry ; • We propose an isometric regularization method for autoencoders that learns both the data manifold and a set of geometry-preserving latent space coordinates , all while incurring minimal losses in reconstruction accuracy ; • We propose a postprocessing flattening technique that learns a more isometric representation space without further losses in reconstruction accuracy . 2 A HIERARCHY OF GEOMETRY-PRESERVING MAPPINGS . This section introduces a hierarchy of geometry-preserving mappings between two Riemannian manifolds . Let M be a Riemannian manifold of dimension m with local coordinates z ∈ Rm and Riemannian metric G ( z ) ∈ Rm×m , and N be a Riemannian manifold of dimension n with local coordinates x ∈ Rn and Riemannian metric H ( x ) ∈ Rn×n . Let f : M → N be a smooth mapping , represented in local coordinates by the italic symbol f : Rm → Rn . Its differential is denoted by the Jacobian matrix Jf ( z ) : = ∂f∂z ( z ) ∈ R n×m . Intuitively , an isometry is a mapping between two spaces that preserves distances and angles everywhere . For a linear mapping between two vector spaces equipped with inner products , an isometry preserves the inner product everywhere . In the case of a mapping between Riemannian manifolds , f : M → N is an isometry if G ( z ) = Jf ( z ) TH ( f ( z ) ) Jf ( z ) ∀z ∈ Rm . ( 1 ) Sometimes , requiring a map f to be an isometry can be overly restrictive ; preserving only angles may be sufficient . A conformal map is a mapping that preserves angles but not necessarily distances . Mathematically , f : M → N is conformal ( or angle-preserving ) if G ( z ) = c ( z ) Jf ( z ) TH ( f ( z ) ) Jf ( z ) ∀z ∈ Rm , ( 2 ) for some positive function c : M → R. The positive function is called the conformal factor . Conformal maps can be further categorized by the polynomial degree of c ( z ) . A conformal map of degree zero , i.e. , one in which c ( z ) is constant , sits one level below the isometric mapping and is defined formally as any mapping f for which a positive scalar constant c satisfying G ( z ) = cJf ( z ) TH ( f ( z ) ) Jf ( z ) ∀z ∈ Rm ( 3 ) can be found . Such a map not only preserves angles but also scaled distances ; for this reason we shall also refer to a degree zero conformal map as a scaled isometry . Area-preserving maps also can be placed within this hierarchy , but for the purposes of this paper our focus will be exclusively on isometric and conformal mappings . 3 A COORDINATE-INVARIANT RELAXED DISTORTION MEASURE . The goal of this section is to design a coordinate-invariant functional F that measures the proximity of the mapping f : M → N to a scaled isometry . Section 3.1 shows how to construct coordinateinvariant functionals for a smooth mapping between two Riemannian manifolds , while Section 3.2 introduces a simple technique to define coordinate-invariant distortion measures that measure how close the mapping is to being an isometry . Section 3.3 defines a family of coordinate-invariant relaxed distortion measures that measure how close the mapping is to being a scaled isometry . In order to extend the discussion of traditional distortion measures to more general cases ( we will later use a probability measure ) , we consider a positive measure ν on M absolutely continuous to the Riemannian volume form1 , the spaces of interest will then be limited to the support of ν rather than the entire manifold M . 3.1 COORDINATE-INVARIANT FUNCTIONALS ON RIEMANNIAN MANIFOLDS . We begin this section by reviewing how to construct coordinate-invariant functionals for a smooth mapping f : M → N ( Jang et al. , 2020 ) . At a point z ∈ M , consider the characteristic values of the pullback metric JTf HJf relative to the metric G of M , i.e. , the m eigenvalues λ1 ( z ) , ... , λm ( z ) of Jf ( z ) TH ( f ( z ) ) Jf ( z ) TG−1 ( z ) . Apart from their order , these eigenvalues are intrinsically associated with JTf HJf and G , i.e. , they are invariant under coordinate transformations 2 . Let S ( λ1 , ... , λm ) be any symmetric function ( i.e. , a function whose value is invariant with respect to permutations of its arguments ) of the m eigenvalues . Then the integral IS ( f ) : = ∫ M S ( λ1 ( z ) , ... , λm ( z ) ) dν ( z ) ( 4 ) is an intrinsic quantity , i.e. , coordinate-invariant . In this paper we introduce a new family of coordinate-invariant functionals that can be used to formulate measures for scaled isometries in a more natural way . Consider a symmetric function S such that the above integral IS ( f ) ̸= 0 , and let S′ ( λ1 , ... , λm ) be any symmetric function . The following integral ∫ M S′ ( λ1 ( z ) IS ( f ) , ... , λm ( z ) IS ( f ) ) dν ( z ) ( 5 ) is then an intrinsic quantity , since the eigenvalues and IS ( f ) are both intrinsic quantities . 1Given a measurable space M , a measure ν is absolutely continuous to the Riemannian volume measure Vol such that dVol ( z ) = √ detG ( z ) dz , if Vol ( A ) = 0 implies ν ( A ) = 0 for any measurable set A ⊂ M. 2Eigenvalues of JTf HJfG −1 remain the same under coordinate transformations z 7→ ϕ ( z ) , x 7→ ψ ( x ) .

Summary: The paper makes two main contributions. The first is the introduction of a new regularizer term for the VAE loss, ensuring a (scaled) isometry between the learned latent space and the (typically unknown) data space. The second is a post-processing "flattening" step to improve the isometry constraint by directly operating on the latent space, while leaving the reconstruction error untouched. The resulting pipeline seems effective, as showcased on a selection of experiments over standard datasets.