Deep Quotient Manifold Modeling

1 INTRODUCTION . Real-world data are usually considered to involve a multi-manifold structure by having discrete features as well as continuous features ; continuous features such as size or location induce a smooth manifold structure in general , whereas discrete features such as digit-class or a new object in the background induce disconnections in the structure , making it a set of disjoint manifolds instead of a single ( Khayatkhoei et al. , 2018 ) . While this multiplicity makes modeling data a difficult problem , recently proposed deep generative models showed notable progresses by considering each manifold separately . Extending the conventional models by using multiple generators ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) , discrete latent variables ( Chen et al. , 2016 ; Dupont , 2018 ; Jeong and Song , 2019 ) , or mixture densities ( Gurumurthy et al. , 2017 ; Xiao et al. , 2018 ; Tomczak and Welling , 2018 ) , they exhibit improved performances in image generations and in learning high-level features . There are , however , two additional properties little considered by these models . First , since discrete features are both common and combinatorial , there can be exponentially many manifolds that are not included in the dataset . For example , an image dataset of a cat playing around in a room would exhibit a simple manifold structure according to the locations of the cat , but there are also numerous other manifolds derivable from it via discrete variations—such as placing a new chair , displacing a toy , turning on a light or their combinations—that are not included in the dataset ( see Fig . 1 ) . Second , while the manifolds to model are numerous considering such variations , they usually have the same generic structure since the underlying continuous features remain the same ; regardless of the chair , toy , or light , the manifold structures are equally due to the location of the cat . Considering these properties , desired is a model that can handle a large number of resembling manifolds , but the aforementioned models show several inefficiencies . They need proportionally many generators or mixture components to model a large number of manifolds ; each of them requires much data , only to learn the manifolds having the same generic structure . Moreover , even if they are successfully trained , new discrete changes are very easy to be made , yet they can not generalize beyond the trained manifolds . In this paper , we propose quotient manifold modeling ( QMM ) —a new generative modeling scheme that considers generic manifold structure independent of discrete features , thereby deriving efficiency in modeling and allowing generalization over untrained manifolds . QMM outwardly follows the multi-generator scheme ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) ; but it involves a new regularizer that enforces encoder compatibility—a condition that the inverse maps of the generators to be presented by a single deep encoder . Since deep encoders usually exhibit good generalizability , this condition not only makes a generic structure be shared among the generators but also makes it generalizable to untrained manifolds . In particular , it induces a generalizable equivalence relation between data , and the manifold structure of out-of-sample data can be derived by taking the quotient of this relation , hence the name QMM . Since the implementation of QMM is essentially adding a regularizer , it can be easily applied to existing deep generative models such as generative adversarial networks ( GANs ; ( Goodfellow et al. , 2014 ) ) , variational auto-encoders ( VAEs ; ( Kingma and Welling , 2013 ) ) , and their extensions . We demonstrate that these QMM-applied models not only show better FID scores but also show good generalizations . Our contributions can be summarized as follows : • We propose QMM , a new generative modeling scheme that considers generic manifold structure , thereby allowing generalizations over untrained manifolds . • We derive a regularizer enforcing encoder compatibility , an essential condition for QMM . • We show that GANs and VAEs implementing QMM show superior FID scores and generalize across different datasets . 2 BACKGROUND . 2.1 MANIFOLD MODELING IN GANS AND VAES . While generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and variational autoencoders ( VAEs ) ( Kingma and Welling , 2013 ) are two different strands of models , they have the same data-modeling scheme that leads to a manifold structure ( though VAEs involve stochasticity ) . They model data x ∈ X as a transformation of a low-dimensional latent code z ∈ Z via a generative ( decoding ) map fG : Z → X , which makes every datum they consider lie on a subspace M = fG ( Z ) ⊂ X . Since fG can be assumed to be smooth and injective in practice ( Shao et al. , 2017 ) , M accords with the mathematical definition of a smooth manifold . But to deal with multi-manifold data , these models need to approximate disconnections in the structure with low densities . This requires them to have a highly nonlinear fG , which is difficult to learn and often leads to either a low-quality model or a mode collapse ( Khayatkhoei et al. , 2018 ) . 2.2 MULTI-MANIFOLD EXTENSIONS . To better model the multi-manifold structure , several studies proposed extended GAN and VAE models that consider each of the manifolds separately . According to which component is extended , the approaches can be broken down into the below three . While these approaches have advantages in dealing with multi-manifold data , they still show limiting performance in learning generic structure and do not allow generalization over untrained manifolds . 1 ) Multi-generators—x ( i ) = f ( i ) G ( z ) ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) . In this approach , each manifold is modeled by a separate generator . The generators are usually independent , but some models design them to share the weight parameters in a subset of the layers . This in part contributes to the learning of a generic structure , but lacks theoretical grounds and shows inferior performances ( see Appendix F ) . 2 ) Mixture density—x ( i ) = fG ( z ( i ) ) , where z ( i ) ∼ p ( i ) ( Gurumurthy et al. , 2017 ; Xiao et al. , 2018 ; Tomczak and Welling , 2018 ) . In this approach , each manifold is modeled by a separate mode of the latent distribution . While the modes outwardly share the generator , the actual mappings are effectively different from each other as they reside in different regions in Z . 3 ) Discrete latent variables—x ( i ) = fG ( [ z ; d ] ) ( Chen et al. , 2016 ; Dupont , 2018 ; Jeong and Song , 2019 ) . In this approach , discrete random variables are explicitly defined and concatenated to the continuous variable . Since discrete information is slowly blended in layer by layer , it can learn the generic structure to some degree , but not as clear ( see Table 1 ) . 3 QUOTIENT MANIFOLD MODELING ( QMM ) . QMM inherits the multi-generator scheme ( Khayatkhoei et al. , 2018 ; Ghosh et al. , 2017 ; Hoang et al. , 2018 ) , but involves an additional regularizer enforcing the encoder compatibility . Leaving the regularizer for the next section , we first explain the role of this compatibility as binding the generative maps . Then , we see how a plausible equivalence relation can be defined using a deep encoder . Lastly , we explain how a new manifold can be obtained by taking the quotient of the relation . 3.1 ENCODER COMPATIBILITY . Definition 1 . LetH be a set of encoding maps ( X → Z ) that can be represented by a deep encoder . We say that generative maps { f ( i ) G : Z →M ( i ) ⊂ X } Ai=1 have encoder compatibility if there exists hE ∈ H satisfying ( f ( i ) G ) −1 ( x ) = hE ( x ) for all x ∈M ( i ) and i . With this condition satisfied , the generative maps { f ( i ) G } i are no longer independent to each other but share a single X ↔ Z translation rule represented by the deep encoder hE ∈ H. However , this binding is meaningful only when H has a certain property ; otherwise , hE is just an extension of functions { ( f ( i ) G ) −1 } i giving no useful signal to { f ( i ) G } i . In practice , H indeed involves an important property that its elements—deep encoders—have good generalizability . Having numerous parameters , deep encoders could overfit data , but in practice they find the smoothest function exhibiting generalizability . For example , if a deep encoder is trained on images illustrated in Fig . 1 to output the position of the cat , we expect it would work fairly well even after we place a vase or turn on a light , generalizing over the discrete changes of the room condition . While this generalizing property is not fully understood to date ( there are several compelling theories ( Zhang et al. , 2016 ; Brutzkus et al. , 2018 ) ) , it has been consistently demonstrated by many deep recognition models ( e.g. , VGG-Net ( Simonyan and Zisserman , 2014 ) , Fast R-CNN ( Girshick , 2015 ) ) . In this regard , we assume its continuity and generalizability in developing our model , and verify them later from the experiments . 3.2 EQUIVALENCE RELATION AND QUOTIENT MANIFOLDS . Putting the generalizability of hE and the compatibility ( f ( i ) G ) −1 = hE together , we expect hE to output the continuous features z given data x . Then , there is a naturally induced equivalence relation ( called the kernel of hE ) x ( 1 ) ∼hE x ( 2 ) ⇐⇒ hE ( x ( 1 ) ) = hE ( x ( 2 ) ) , which effectively groups data having the same continuous features together regardless of the manifolds they are lying on . This can be seen more concretely in Fig . 1 . Assuming the model is well trained , it should have captured the crucial continuous feature—the location of the cat—in Z . Due to the encoder compatibility , such a feature should be encoded in Z smoothly , meaning that hE outputs the location information of the cat in a consistent and generalizable way , if not exactly calibrated . Given this , images having the cat on the same location , regardless of the chair , will have the same hE ( x ) , thus being equivalent under ∼hE ( ones on a green dotted curve ) . Since hE can generalize , the equivalence relation is consistent for data lying on an untrained manifold as well ( ones with the vase , drawn in purple ) . Now that the data lying on different manifolds are made equivalent under ∼hE , manifold structures are described as quotients of this relation . In implementations , taking the quotient is the same as taking the orthogonal directions to the equivalence-relation contours . In Fig . 1 , we can see that M ( 1 ) and M ( 2 ) , the manifolds that are included in the dataset , are already orthogonal to the contours formed from hE . When given an untrained image ( shown in purple ) , we can obtain the new manifold just by following the orthogonal directions . It will be explained later that this manifold can be described by a new generator , whose bias parameters are optimized for the manifold to pass through the given image ( see Sec . 5 ) .

Summary: The author extends generative models with multi-generators by restricting the generators to share weights and all bias to be regularized in order to enforce that the inverse maps of the generators can be represented by a single encoder. The regularizer proposed minimizes an upper bound the the sum of the bias variances. This extension is evaluated on a set of visual datasets (MNIST, 3DChair and UT-Zap50k) with respect to density estimation (evaluated with the FID score) and disentanglement (evaluated with their own disentanglement score).