Invertible Manifold Learning for Dimension Reduction

1 INTRODUCTION . In real-world scenarios , it is widely believed that the loss of data information is inevitable after dimension reduction ( DR ) , though the goal of DR is to preserve as much information as possible in the low-dimensional space . In the case of linear DR , compressed sensing ( Donoho , 2006 ) breaks this common sense with practical sparse conditions of the given data . In the case of nonlinear dimension reduction ( NLDR ) , however , it has not been clearly discussed , e.g . what is the structure within data and how to maintain these structures after NLDR ? From the perspective of manifold learning , the manifold assumption is widely adopted , but classical manifold based DR methods usually fail to yield good results in the many practical case . Therefore , what is the gap between theoretical and real-world applications of manifold based DR ? Here , we give the first detailed discussion of these two problems in the context of manifold learning . We think that a good low-dimensional representation should preserve the topology and geometry of input data , which require the NLDR transformation to be homeomorphic . Thus , we propose an invertible NLDR process , called inv-ML , combining sparse coordinate transformation and local isometry constraint which preserve the property of topology and geometry , to explain the information-lossless NLDR in manifold learning theoretically . We instantiate inv-ML as a neural network called i-ML-Enc via a cascade of equidimensional layers and a linear transform layer . Sufficient experiments are conduct to validate invertible NLDR abilities of i-ML-Enc and analyze learned representations to reveal inherent difficulties of classical manifold learning . Topology preserving dimension reduction . To start , we first make out the theoretical definition of information-lossless DR on a manifold . The topological property is what is invariant under a homeomorphism , and thus what we want to achieve is to construct a homeomorphism for dimension reduction , removing the redundant dimensions while preserving invariant topology . To be more specific , f : Md0 → Rm is a smooth mapping of a differential manifold into another , and if f is a homeomorphism ofMd0 intoMd1 = f ( Md0 ) ⊂ Rm , we call f is an embedding ofMd0 into Rm . Assume that the data set X = { xj |1 ≤ j ≤ n } sampled from the compact manifoldMd1 ⊂ Rm which we call the data manifold and is homeomorphic toMd0 . For the sample points we get are represented in the coordinate after inclusion mapping i1 , we can only regard them as points from Euclidean space Rm without any prior knowledge , and learn to approximate the data manifold in the latent space Z . According to the Whitney Embedding Theorem ( Seshadri & Verma , 2016 ) , Md0 is can be embedded smoothly into R2d by a homeomorphism g. Rather than to find the f−1 : Md1 →Md0 , our goal is to seek a smooth map h : Md1 → Rs ⊂ R2d , where h = g ◦ f−1 is a homeomorphism ofMd1 intoMd2 = h ( Md1 ) and d ≤ s ≤ 2d m , and thus the dim ( h ( X ) ) = s , which achieves the DR while preserving the topology . Owing to the homeomorphism h we seek as a DR mapping , the data manifoldMd1 is reconstructible viaMd1 = h−1 ◦ h ( Md1 ) , by which we mean h a topology preserving DR as well as information-lossless DR. Geometry preserving dimension reduction . While the topology of the data manifoldMd1 can be preserved by the homeomorphism h discussed above , it may distort the geometry . To preserve the local geometry of the data manifold , the map should be isometric on the tangent space TpMd1 for every p ∈ Md1 , indicating that dMd1 ( u , v ) = dMd2 ( h ( u ) , h ( v ) ) , ∀u , v ∈ TpM d 1 . By Nash ’ s Embedding Theorem ( Nash , 1956 ) , any smooth manifold of class Ck with k ≥ 3 and dimension d can be embedded isometrically in the Euclidean space Rs with s polynomial in d. Noise perturbation . In the real-world scenarios , sample points are not lied on the ideal manifold strictly due to the limitation of sampling , e.g . non-uniform sampling noises . When the DR method is very robust to the noise , it is reasonable to ignore the effects of the noise and learn the representation Z from the given data . Therefore , the intrinsic dimension of X is approximate to d , resulting in the lowest isometric embedding dimension is larger than s . 2 RELATED WORK . Manifold learning . Most classical linear or nonlinear DR methods aim to preserve the geometric properties of manifolds . The Isomap ( Tenenbaum et al. , 2000 ) based methods aim to preserve the global metric between every pair of sample points . For example , McQueen et al . ( 2016 ) can be regarded as such methods based on the push-forward Riemannian metric . For the other aspect , LLE ( Roweis & Saul , 2000 ) based methods try to preserve local geometry after DR , whose derivatives like LTSA ( Zhang & Zha , 2004 ) , MLLE ( Zhang & Wang , 2007 ) , etc . have been widely used but usually fail in the high-dimensional case . Recently , based on local properties of manifolds , MLDL ( Li et al. , 2020 ) was proposed as a robust NLDR method implemented by a neural network , preserving the local geometry but abandoning the retention of topology . In contrast , our method takes the preservation of both geometry and topology into consideration , trying to maintain these properties of manifolds even in cases of excessive dimension reduction when the target dimension s′ is smaller than s. Invertible model . From AutoEncoder ( AE ) ( Hinton & Salakhutdinov , 2006 ) , the fundamental neural network based model , having achieved DR and cut information loss by minimizing the reconstruction loss , some AE based generative models like VAE ( Kingma & Welling , 2014 ) and manifold-based NLDR models like TopoAE ( Moor et al. , 2020 ) has emerged . These methods can not avoid information loss after NLDR , and thus , some invertible models consist of a series of equidimensional layers have been proposed , some of which aim to generate samples by density estimation through layers ( Dinh et al. , 2015 ) ( Dinh et al. , 2017 ) ( Behrmann et al. , 2019 ) , and the other of which are established for other targets , e.g . validating the mutual information bottleneck ( Jacobsen et al. , 2018 ) . Different from methods mentioned above , our proposed i-ML-Enc is a neural network based encoder , with NLDR as well as maintaining structures of raw data points based on manifold assumption via a series of equidimensional layers . Compressed sensing . The JohnsonLindenstrauss Theorem ( Johnson & Lindenstrauss , 1984 ) provides the lower bound of target dimension for linear DR with the pairwise distance loss . Given a small constant ∈ ( 0 , 1 ) and n samples { xi } ni=1 in Rm , a linear projection W : Rm → Rs , s > O ( logm 2 ) can be found , which embeds samples into a s-dimensional space with ( 1 + ) distortion of any sample pairs ( xi , xj ) . It adopts a prior assumption that the given samples in high-dimensional space have a relevant low-dimensional structure constraint which can be maintained by keeping the pairwise distance . Further , compressed sensing ( CS ) provides strict sparse conditions of linear DR with great probability to recover the compressed signal , which usually cooperates with sparse dictionary learning ( Hawe et al. , 2013 ) . The core of CS is Restricted Isometry Property ( RIP ) condition , which reads ( 1− ) ‖x1 − x2‖2 ≤ ‖W ( x1 − x2 ) ‖2 ≤ ( 1 + ) ‖x1 − x2‖2 , ( 1 ) where ∈ ( 0 , 1 ) is a rather small constant and W is a linear measurement of signal x1 and x2 . Given a signal x ∈ Rm with s-sparse representation α = Φx on an m-dimensional orthogonal basis Φ , α can be recovered from the linear measurement y = Wα with great probability by the sparse optimization if Wm×s satisfies the RIP condition : arg minã ||α̃||0 , s.t . y = Wα̃ . The linear measurement is rewritten as y = ΨΦα = Ψx where Ψ is a low-dimensional orthogonal basis and Φ can be found by the nonlinear dictionary learning . Some reconstructible CS-based NLDR methods ( Wei et al. , 2015 ) ( Wei et al. , 2019 ) are proposed , which are achieved by preserving local geometry on AE-based networks , but usually with unsatisfying embedding qualities . 3 PROPOSED METHOD . We will specifically discuss the proposed two-stage invertible NLDR process inv-ML as the first stage in Sec 3.1 , in which a s-dimensional representation is learned by a homeomorphism transformation while keeping all topological and geometric structure of the data manifold ; then give applicable conditions in real-world scenarios as the second stage in Sec 3.2 , in which the dimension is further compressed to s′ . We instantiate the proposed inv-ML as a neural network i-ML-Enc in Sec 3.3 . 3.1 TOPOLOGY AND GEOMETRY PRESERVATION . Canonical embedding for homeomorphism . To seek the smooth homeomorphism h , we turn to the theorem of local canonical form of immersion ( Mei , 2013 ) . Let f : M→N an immersion , and for any p ∈ M , there exist local coordinate systems ( U , φ ) around p and ( V , ψ ) around f ( p ) such that ψ ◦ f ◦ φ−1 : φ ( U ) → ψ ( V ) is a canonical embedding , which reads ψ ◦ f ◦ φ−1 ( x1 , x2 , · · · , xd ) = ( x1 , x2 , · · · , xd , 0 , 0 , · · · , 0 ) . ( 2 ) In our case , letM = Md2 , and N = Md1 , any point z = ( z1 , z2 , · · · , zs ) ∈ Md1 ⊂ Rs can be mapped to a point in Rm by the canonical embedding ψ ◦ h−1 ( z1 , z2 , · · · , zs ) = ( z1 , z2 , · · · , zs , 0 , 0 , · · · , 0 ) . ( 3 ) For the point z is regarded as a point in Rs , φ = I is an identity mapping , and for h = g ◦ f−1 is a homeomorphism , h−1 is continuous . The Eq . ( 3 ) can be written as ( z1 , z2 , · · · , zs ) = h ◦ ψ−1 ( z1 , z2 , · · · , zs , 0 , 0 , · · · , 0 ) = h ( x1 , x2 , · · · , xm ) . ( 4 ) Therefore , to reduce dim ( X ) = m to s , we can decompose h into ψ and h ◦ ψ−1 , by firstly finding a homeomorphic coordinate transformation ψ to map x = ( x1 , x2 , · · · , xm ) into ψ ( x ) = ( z1 , z2 , · · · , zs , 0 , 0 , · · · , 0 ) , which is called a sparse coordinate transformation , and h ◦ ψ−1 can be easily obtained by Eq . ( 3 ) . We denote h ◦ ψ−1 by h0 and call it a sparse compression . The theorem holds for any manifold , while in our case , we aims to find the mapping of X ⊂ Rm into Rs , so the local coordinate systems can be extended to the whole space of Rm . Local isometry constraint . The prior local isometry constraint is applied under the manifold assumption , which aims to preserve distances ( or some other metrics ) locally so that dMd1 ( u , v ) = dMd2 ( h ( u ) , h ( v ) ) , ∀u , v ∈ TpM d 1 .

Summary: In this paper, the authors propose a novel manifold learning method, via adding a locally isometric smoothness constraint, which preserves topological and geometric properties of data manifold. Empirical results demonstrate the efficacy of their approach. The authors also show that the reliability of tangent space approximated by its local neighborhood is essential to the success of manifold learning approaches.