Manifold Learning and Alignment with Generative Adversarial Networks

We present a generative adversarial network ( GAN ) that conducts manifold learning and alignment ( MLA ) : A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information . Our main idea is to exploit the powerful abstraction ability of encoder architecture . Specifically , we define multiple generators to model multiple manifolds , but in a particular way that their inverse maps can be commonly represented by a single smooth encoder . Then , the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space . In experiments with MNIST , 3D-Chair , and UT-Zap50k datasets , we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores . Furthermore , by virtue of the abstractive modeling , we show that our model can generate data from an untrained manifold , which is unique to our model . 1 INTRODUCTION . Real-world datasets generally have multi-manifold structure . Smoothly varying features such as the size or the pose of an animal introduce smooth manifold structure , while the existence of discrete features such as species makes the structure not a single manifold but multiple disconnected manifolds ( e.g. , one manifold per species ) ( Khayatkhoei et al. , 2018 ) . Importantly , these manifolds are correlated to each other in their structure since the transformation rules according to the size ( stretching the foreground patch ) or the pose ( rotating the patches of body parts ) are similar even for different species . Hence , our desire is to build a model that learns not only the multi-manifold structure but also the correlations between the manifolds . As for the learning of manifold structure , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been known for their remarkable performance . In particular , recently proposed GAN models have successfully demonstrated multi-manifold learning by using multiple generator networks ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) or giving a mixture density on the latent space ( Xiao et al. , 2018 ; Gurumurthy et al. , 2017 ) . However , none of these models have taken the correlations between the manifolds into account . On the other hand , a field of research named manifold alignment ( Ham et al. , 2003 ) has its primary concern in learning the correlations between the manifolds . In particular , unsupervised manifold alignment ( Cui et al. , 2014 ; Wang & Mahadevan , 2009 ) aims to learn plausible correspondences between the data points of different manifolds , without any ground-truth correspondence information is given . However , most existing methods focus only on finding the correspondences , not on learning the structure of the manifolds itself . In this work , we propose a model that performs both of the tasks , which we call as manifold learning and aligning GAN ( MLA-GAN ) . MLA-GAN is similar to the GANs with multiple generators ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) , but the generators are restricted such that their inverse maps can be represented by a single deep encoder . Due to the abstracting property of deep encoders , this restriction guides the inverse maps to learn semantically similar mappings , and the model achieves a semantically plausible manifold alignment . The main contributions of this work can be summarized as follows . • The MLA problem : Up to our knowledge , MLA-GAN is the first model that performs both multi-manifold learning and alignment with real-world datasets . It shows state-ofthe-art FID scores in data generation and plausible alignments with disentangled features . • Generalization to an untrained manifold : Due to the abstracting nature of an encoder , the smooth features can be inferred even for data lying on an untrained manifold . Exploiting this , one can generate other data points lying on this manifold . • Easily applicable : Due to the new regularizer we propose , the inverse maps used for restricting the generators do not need to be actually computed . This makes MLA-GAN easily applicable to any existing GAN architectures . 2 MANIFOLD LEARNING AND ALIGNMENT WITH GAN . 2.1 BACKGROUNDS . Multi-Manifold Learning GANs Conventional GANs ( Goodfellow et al. , 2014 ) model data by transforming a latent-space distribution p ( z ) with a generating map fG : Z → X . If fG is smooth and injective1 , the image of fG is a smooth manifold M ⊂ X , thus the data are modeled as they are lying on this manifold . To deal with multiple manifolds , recent extensions ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) employ multiple generating maps { f ( i ) G } Ai=1 where each f ( i ) G accounts for the i-th manifold M ( i ) . Unsupervised Manifold Alignment Most existing methods formulate the manifold alignment problem as finding embeddings of manifolds into a common low-dimensional space Y . This set of embeddings { r ( i ) E : M ( i ) → Y } i essentially defines an alignment of manifolds ; data points that are mapped to the same point in Y correspond to each other . To achieve a plausible alignment , the embeddings are trained to minimize several loss terms guiding geometrical similarity ( see Sec . 3 ) . 1Although these conditions are not guaranteed for general neural networks , they are approximately met or observed in practice ( Shao et al. , 2017 ) . Additionally , the support of p ( z ) needs to be a completely separable Hausdorff space , but this condition is met for the usual choices of p ( z ) ( e.g. , unimodal Gaussian ) . 2.2 MAIN IDEA . To model multiple manifolds , we use multiple generating maps { f ( i ) G : Z → M ( i ) } i as with the aforementioned GAN models . Interestingly , the inverses2 of these maps { ( f ( i ) G ) −1 : M ( i ) → Z } i can be regarded as a set of embeddings ( where Y is replaced by Z ) , which means that the manifold alignment is automatically defined as we define the manifolds . However , this alignment is not at all semantically plausible unless properly regulated . Here , we regulate it by restricting the generating maps such that their inverses to be commonly represented by a single encoder hE : X → Z . Formally , we restrict the generating maps as follows : { f ( i ) G } i : A set of functions ( Z →M ( i ) ⊂ X ) s.t . ∃hE ∈ H satisfying ( f ( i ) G ) −1 ( x ) = hE ( x ) ∀i , ∀x ∈M ( i ) . ( 1 ) Here , H is a certain set of functions ( to be clearer soon ) , and hE being an element of H is an important constraint . Without this constraint , one can always find hE satisfying the above condition , for any sets of { f ( i ) G } i , because hE is merely a union of functions { ( f ( i ) G ) −1 } i extending the domain from ⋃ iM ( i ) ⊂ X to X . However , if H is , for example , a set of smooth functions and hE ∈ H , then { ( f ( i ) G ) −1 } i will be restricted to be similar to each other and the alignment will become more plausible . In our model , H is defined as a set of functions represented by deep encoder networks . We explain the intuition behind this design in what follows . Intuition Consider a dataset containing images of cats and dogs . It is not difficult to imagine a deep encoder network that takes an image from this dataset and outputs the size and pose regardless of the species . At least , we could have trained this encoder easily if the size and pose labels were given . Also , even if we train the encoder using the cat data only , we still expect it works fairly well for the dogs due to the powerful abstraction and generalization abilities of deep encoders . We want to exploit these abilities of deep encoders for aligning the manifolds . If we let H be a set of deep encoders , its element hE , once well-trained , will abstract the species away and outputs the size and pose . This gives a nice manifold alignment , where a cat and a dog images with the same size and pose correspond to each other . Remaining Question Is H being a set of deep encoders enough to obtain the abstraction and generalization ? An answer to this question is related to how deep networks generalize so well despite the large number of parameters they have . Several works conjecture that it is due to the SGD ( Zhang et al. , 2016 ; Brutzkus et al. , 2018 ) , but this is not a fully developed theory yet . At least , we can assume that deep encoders learn simpler , generalization-friendly maps among others , based on various cases ( e.g. , VGG-Net ( Simonyan & Zisserman , 2014 ) , Fast R-CNN ( Girshick , 2015 ) ) . 2.3 MLA-GAN . What remains is to implement the generating maps with deep neural networks . The main challenge is to make the deep networks comply with the constraint given in Eq . 1 . In this section , we explain how this challenge can be achieved in the proposing MLA-GAN model in detail . We first focus on a single linear layer and prove the following ( detailed in Proposition 1 ) : If linear generating maps are defined with the same weight and the same tangential components of bias , their inverses can always be represented by a single linear encoder . Thus , the maps comply with Eq . 1 . Then , to realize the above , we introduce a new form of regularizer . After , we move on to the entire network and derive MLA-GAN by applying the same principles as the linear layer . 2.3.1 SINGLE LINEAR LAYER . Consider a set of linear generating maps { f ( i ) G : Z →M ( i ) ⊂ X } i ; each of the maps is defined as f ( i ) G ( z ) : = U ( i ) z + a ( i ) , 2With the same assumption as Footnote 1. where U ( i ) ∈ RdX×dZ ( dX > dZ ) and a ( i ) ∈ RdX are the weight and bias parameters respectively . We assume U ( i ) is a full-rank matrix ( rank-dZ ) such that f ( i ) G is injective ( see Footnote 1 ) . Then , the inverse ( f ( i ) G ) −1 : M ( i ) → Z can be derived as ( f ( i ) G ) −1 = ( U ( i ) ) + ( x ( i ) − a ( i ) ) ( 2 ) where ( U ( i ) ) + : = ( ( U ( i ) ) > U ( i ) ) −1 ( U ( i ) ) > denotes the pseudo-inverse of U ( i ) . As noted in Eq . 1 , our desire is to restrict the inverse maps { ( f ( i ) G ) −1 } i such that they can be represented by a single encoder hE : X → Z . One simple way to achieve this is to use the following proposition . Proposition 1 . If the linear generating maps { f ( i ) G ( z ) } i are restricted to have the same weight U and to have the same tangential components of bias a‖ ( i.e. , a ( i ) = a‖ + a ( i ) ⊥ ) , then their inverses { ( f ( i ) G ) −1 } i can be represented by a single linear encoder hE ( x ) : = W > x + b where W = U ( U > U ) −1 and b = −W > a‖ . Proof . Note the bias a ( i ) can be decomposed into the tangential component a ( i ) ‖ and the normal component a ( i ) ⊥ to the column space of U > . Substituting the condition a ( i ) = a‖+a ( i ) ⊥ and the same weight U , Eq . 2 is written as ( f ( i ) G ) −1 = U+x− U+ ( a‖ + a ( i ) ⊥ ) = U+x− U+a‖ = W > x+ b . Bias Regularizer When implementing Proposition 1 , making { U ( i ) } i the same is as trivial as setting the same weight U for all f ( i ) G , but making { a ( i ) ‖ } i the same is not trivial since the tangential direction keeps changing while training . One solution would be to use a regularizer minimizing the sum of the variance : trace ( cov ( a ( i ) ‖ ) ) . However , computing this term is intractable due to the inversion ( U > U ) −1 inside of a ( i ) ‖ = U ( U > U ) −1U > a ( i ) . Theorem 1 . The following inequality holds : trace ( cov ( U > a ( i ) ) ) ≥ 1 dz H ( { λk } dzk=1 ) trace ( cov ( a ( i ) ‖ ) ) where { λk } dzk=1 denotes the eigenvalues of U > U and H ( · ) denotes harmonic mean . Proof . See Appendix B . As the harmonic mean in Theorem 1 is constant from the perspective of a ( i ) ‖ , we can minimize the original term by minimizing the upper bound instead . With an additional log function to match the scale due to the dimensionality , we propose the upper bound as a regularizer to make a ( i ) ‖ the same : Regularizer : Rbias = log ( trace ( cov ( U > a ( i ) ) ) ) . ( 3 )

Summary: The paper suggests performing simultaneous manifold learning and alignment by multiple generators that share common weight matrices and a constructed inverse map that is instantiated by a single encoder. The method utilizes a special regularizer to guide the training. It has been empirically well tested on a multi-manifold learning task, manifold alignment, feature disentanglement, and style transfer.