Regularized Autoencoders for Isometric Representation Learning

The recent success of autoencoders for representation learning can be traced in large part to the addition of a regularization term . Such regularized autoencoders “ constrain ” the representation so as to prevent overfitting to the data while producing a parsimonious generative model . A regularized autoencoder should in principle learn not only the data manifold , but also a set of geometry-preserving coordinates for the latent representation space ; by geometry-preserving we mean that the latent space representation should attempt to preserve actual distances and angles on the data manifold . In this paper we first formulate a hierarchy for geometry-preserving mappings ( isometry , conformal mapping of degree k , areapreserving mappings ) . We then show that a conformal regularization term of degree zero – i.e. , one that attempts to preserve angles and relative distances , instead of angles and exact distances – produces data representations that are superior to other existing methods . Applying our algorithm to an unsupervised information retrieval task for CelebA data with 40 annotations , we achieve 79 % precision at five retrieved images , an improvement of more than 10 % compared to recent related work . 1 INTRODUCTION . Learning a good representation for high-dimensional data is one of the most fundamental problems in machine learning . A good representation should capture all essential information about the data in a parsimonious manner while filtering out all non-essential variations . Clearly what is “ essential ” or “ non-essential ” depends heavily on the end task ( Tschannen et al. , 2018 ) , and numerous criteria have been proposed for a range of contexts , e.g. , disentanglement ( Higgins et al. , 2017 ; Zhao et al. , 2017 ) , clustering structure ( Makhzani et al. , 2015 ) , sparsity ( Lewicki & Sejnowski , 2000 ) , hierarchy ( Gulrajani et al. , 2017 ) , and isometric embedding ( McQueen et al. , 2016 ; Peterfreund et al. , 2020 ) . In this paper we take the view that the essential information is best captured by the geometry of the data . More specifically , we adopt the manifold hypothesis as our point of departure , and further argue that a good representation should also preserve the geometry of the data manifold . That is , nearby points on the manifold should have representations in the latent representation space that are also nearby , and angles and volumes should be preserved as much as possible when moving between the data manifold and its representation space . To find such a representation , it is important to ( i ) learn the correct low-dimensional data manifold , and ( ii ) to find an optimal set of latent space coordinates that preserves the geometry of the learned data manifold . A primary reason that autoencoders are widely used for unsupervised representation learning is that they can learn both the manifold and the latent space coordinates simultaneously during the training phase . Vanilla autoencoders trained purely to minimize reconstruction loss tend to overfit , and the learned manifolds are often inaccurate . As demonstrated in ( Rifai et al. , 2011b ; Kingma & Welling , 2014 ; Makhzani et al. , 2015 ; Tolstikhin et al. , 2018 ) , by augmenting the reconstruction loss with a regularization term , manifold learning performance of autoencoders can be significantly enhanced . However , little if any consideration has been given to the concurrent problem of learning a set of latent space coordinates that preserves the geometry of the data manifold ( with the exception of FMVAE ( Chen et al. , 2020 ) , which we discuss further below ) . The main contribution of this paper is a regularized autoencoder that simultaneously learns the data manifold and a geometry-preserving set of latent space coordinates . For this purpose , we first formulate a hierarchy of geometry-preserving mappings . At the top of the hierarchy are isometries , which preserve distances and angles , followed by conformal maps , which preserve angles , and then area-preserving maps . Of particular relevance to this paper are the conformal maps , which we further stratify into degree k conformal maps , with k = 0 corresponding to scaled isometries , i.e. , maps that preserve angles and distances up to some scale factor . Based on this hierarchy of mappings , we then formulate a corresponding hierarchy of regularization criteria for training the autoencoder . One of the important findings of our study is that somewhat counterintuitively , using a regularization term that measures the nearness to an isometry is in fact detrimental ; such a regularization term overly constrains the mapping , resulting in a higher reconstruction loss whose effects can not be mitigated even with adjustments to the regularization term weight . Rather , using a less stringent regularization term is more helpful . In our examples the degree zero conformal maps , or scaled isometries , seem to offer the best balance between reconstruction accuracy and model parsimony . We note that if the exact data manifold were known in advance , then the scale factor could be precomputed , e.g. , to make the latent space and the data manifold have the same volume , in which case it would make sense to use a normalized version of the isometry measure as regularization term . When using an autoencoder for representation learning , however , the data manifold is not known a priori but rather learned together with the latent space representation . The more effective alternative , we argue , is to learn this scale together with the manifold and the latent space representation . The FMVAE of ( Chen et al. , 2020 ) is in fact the first work to adopt this approach , but the measure does not adequately capture the nearness to a scaled isometry , and is also not coordinate-invariant , limiting its performance as we show later in our experiments . Once the data manifold and an initial set of geometry-preserving latent space coordinates are learned , it is possible to further “ flatten ” the latent space , by adding a postprocessing step that leads to an even more isometric set of coordinates . Specifically , we use an invertible neural network model to map the pre-trained latent space to a more isometric representation space , without incurring any further losses in reconstruction accuracy . Experiments on diverse image and motion capture data confirm that , compared to existing related methods , our geometrically regularized autoencoder produces more isometric representations of the data while incurring only minimal losses in reconstruction accuracy . In particular , information retrieval task experiments conducted with CelebA data show that data similarity measurements performed in our representation space lead to significantly improved levels of retrieval performance . Our specific contributions can be summarized as follows : • We define a family of coordinate-invariant regularization terms that measure how close the decoder is to being a scaled isometry ; • We propose an isometric regularization method for autoencoders that learns both the data manifold and a set of geometry-preserving latent space coordinates , all while incurring minimal losses in reconstruction accuracy ; • We propose a postprocessing flattening technique that learns a more isometric representation space without further losses in reconstruction accuracy . 2 A HIERARCHY OF GEOMETRY-PRESERVING MAPPINGS . This section introduces a hierarchy of geometry-preserving mappings between two Riemannian manifolds . Let M be a Riemannian manifold of dimension m with local coordinates z ∈ Rm and Riemannian metric G ( z ) ∈ Rm×m , and N be a Riemannian manifold of dimension n with local coordinates x ∈ Rn and Riemannian metric H ( x ) ∈ Rn×n . Let f : M → N be a smooth mapping , represented in local coordinates by the italic symbol f : Rm → Rn . Its differential is denoted by the Jacobian matrix Jf ( z ) : = ∂f∂z ( z ) ∈ R n×m . Intuitively , an isometry is a mapping between two spaces that preserves distances and angles everywhere . For a linear mapping between two vector spaces equipped with inner products , an isometry preserves the inner product everywhere . In the case of a mapping between Riemannian manifolds , f : M → N is an isometry if G ( z ) = Jf ( z ) TH ( f ( z ) ) Jf ( z ) ∀z ∈ Rm . ( 1 ) Sometimes , requiring a map f to be an isometry can be overly restrictive ; preserving only angles may be sufficient . A conformal map is a mapping that preserves angles but not necessarily distances . Mathematically , f : M → N is conformal ( or angle-preserving ) if G ( z ) = c ( z ) Jf ( z ) TH ( f ( z ) ) Jf ( z ) ∀z ∈ Rm , ( 2 ) for some positive function c : M → R. The positive function is called the conformal factor . Conformal maps can be further categorized by the polynomial degree of c ( z ) . A conformal map of degree zero , i.e. , one in which c ( z ) is constant , sits one level below the isometric mapping and is defined formally as any mapping f for which a positive scalar constant c satisfying G ( z ) = cJf ( z ) TH ( f ( z ) ) Jf ( z ) ∀z ∈ Rm ( 3 ) can be found . Such a map not only preserves angles but also scaled distances ; for this reason we shall also refer to a degree zero conformal map as a scaled isometry . Area-preserving maps also can be placed within this hierarchy , but for the purposes of this paper our focus will be exclusively on isometric and conformal mappings . 3 A COORDINATE-INVARIANT RELAXED DISTORTION MEASURE . The goal of this section is to design a coordinate-invariant functional F that measures the proximity of the mapping f : M → N to a scaled isometry . Section 3.1 shows how to construct coordinateinvariant functionals for a smooth mapping between two Riemannian manifolds , while Section 3.2 introduces a simple technique to define coordinate-invariant distortion measures that measure how close the mapping is to being an isometry . Section 3.3 defines a family of coordinate-invariant relaxed distortion measures that measure how close the mapping is to being a scaled isometry . In order to extend the discussion of traditional distortion measures to more general cases ( we will later use a probability measure ) , we consider a positive measure ν on M absolutely continuous to the Riemannian volume form1 , the spaces of interest will then be limited to the support of ν rather than the entire manifold M . 3.1 COORDINATE-INVARIANT FUNCTIONALS ON RIEMANNIAN MANIFOLDS . We begin this section by reviewing how to construct coordinate-invariant functionals for a smooth mapping f : M → N ( Jang et al. , 2020 ) . At a point z ∈ M , consider the characteristic values of the pullback metric JTf HJf relative to the metric G of M , i.e. , the m eigenvalues λ1 ( z ) , ... , λm ( z ) of Jf ( z ) TH ( f ( z ) ) Jf ( z ) TG−1 ( z ) . Apart from their order , these eigenvalues are intrinsically associated with JTf HJf and G , i.e. , they are invariant under coordinate transformations 2 . Let S ( λ1 , ... , λm ) be any symmetric function ( i.e. , a function whose value is invariant with respect to permutations of its arguments ) of the m eigenvalues . Then the integral IS ( f ) : = ∫ M S ( λ1 ( z ) , ... , λm ( z ) ) dν ( z ) ( 4 ) is an intrinsic quantity , i.e. , coordinate-invariant . In this paper we introduce a new family of coordinate-invariant functionals that can be used to formulate measures for scaled isometries in a more natural way . Consider a symmetric function S such that the above integral IS ( f ) ̸= 0 , and let S′ ( λ1 , ... , λm ) be any symmetric function . The following integral ∫ M S′ ( λ1 ( z ) IS ( f ) , ... , λm ( z ) IS ( f ) ) dν ( z ) ( 5 ) is then an intrinsic quantity , since the eigenvalues and IS ( f ) are both intrinsic quantities . 1Given a measurable space M , a measure ν is absolutely continuous to the Riemannian volume measure Vol such that dVol ( z ) = √ detG ( z ) dz , if Vol ( A ) = 0 implies ν ( A ) = 0 for any measurable set A ⊂ M. 2Eigenvalues of JTf HJfG −1 remain the same under coordinate transformations z 7→ ϕ ( z ) , x 7→ ψ ( x ) .

Summary: This paper studies the aspect of preserving geometry on the learned latent space representations. In particular, the paper looks at a hierarchy of geometry-preserving mapping (isometry, conformal mapping of degree k, area preserving mapping, etc.). Existing popular methods such as SimCLR pays limited attention to preserving geometric relationships. The paper shows that the mapping that preserves angles and relative distances is better than the ones that preserve angles and absolute distances. The main contribution of this paper is to propose a representational learning technique that uses a reconstruction loss and an isometric regularization term. This allows us to learn embeddings that satisfy isometric properties while not suffering from any reconstruction loss or just marginal reconstruction loss. The comparison is done with VAE (Kingma & Welling, 2014) and FMVAE (Chen et al. 2020). Experiments are shown on CelebA with 40 annotations.