CIGMO: Learning categorical invariant deep generative models from grouped data

1 INTRODUCTION . In everyday life , we see objects in a great variety . Categories of objects are numerous and their shape variations are tremendously rich ; different views make an object look totally different ( Figure 1 ( A ) ) . Recent neuroscientific studies have revealed how the primate brain organizes representation of complex objects in the higher visual cortex ( Freiwald & Tsao , 2010 ; Srihasam et al. , 2014 ; Bao et al. , 2020 ) . According to these , it comprises multi-stream networks , each of which is specialized to a particular object category , encodes category-specific visual features , and undergoes multiple stages with increasing view invariance . These biological findings inspire us a new form of learning model that has multiple modules of category-specific invariant feature representations . More specifically , our goal is , given an image dataset of general objects , to learn a generative model representing three latent factors : ( 1 ) category ( e.g. , cars , chairs ) , ( 2 ) shape ( e.g. , particular car or chair types ) , and ( 3 ) view ( e.g. , 3D orientation ) . A similar problem has been addressed by recent disentangling models that discover complex factors of input variations in a way invariant to each other ( Tenenbaum & Freeman , 2000 ; Kingma et al. , 2014 ; Chen et al. , 2016 ; Higgins et al. , 2016 ; Bouchacourt et al. , 2018 ; Hosoya , 2019 ) . However , although these models can effectively infer a category or shape factor separately from a view factor , these typically can not capture the structure in general object images that the diversity of shapes is much larger across categories than within a category . In this study , we propose a novel model called CIGMO ( Categorical Invariant Generative MOdel ) , which can learn to represent all the three factors ( category , shape , and view ) at once only with weak supervision . Our model has the form of mixture of deep generative models , where the mixture components correspond to categories and each component model gives a disentangled representation of shape and view for a particular category . We develop a learning algorithm based on variational autoencoders ( VAE ) method ( Kingma & Welling , 2014 ) that does not use explicit labels , but uses only grouping information that links together different views of the same object ( Bouchacourt et al. , 2018 ; Chen et al. , 2018 ; Hosoya , 2019 ) . Using two image datasets of 3D objects ( one derived from ShapeNet ( Chang et al. , 2015 ) ) , we demonstrate that CIGMO can solve multiple unconventional visual tasks on objects that are unseen during training . These include invariant clustering , i.e. , clustering of objects regardless of the view , one-shot classification , i.e. , object recognition given one example per class , and other various feature manipulations using the disentangled representation . Quantitative comparison indicates that our model often outperforms many existing approaches including state-of-the-art methods . Our key contributions are ( 1 ) development of the new deep generative model CIGMO , together with the VAE-based weakly supervised learning algorithm , ( 2 ) experiments of the model on two datasets with quantitative comparisons that show performance advantages over several existing models , and ( 3 ) empirical exposition of the importance of separate representation of category , shape , and view factors in modeling general object images . 2 RELATED WORK . The present work is closely related to recently proposed disentangling models for discovering mutually invariant factors of variation in the input . In one direction , some models have used unsupervised learning with certain constraints on the latent variable , though these seem to be effective only in limited cases ( Higgins et al. , 2016 ; Chen et al. , 2016 ) . Thus , more practical approaches have made use of explicit labels , such as semi-supervised learning for a part of dataset ( Kingma et al. , 2014 ; Siddharth et al. , 2017 ) or adversarial learning to promote disentanglement ( Lample et al. , 2017 ; Mathieu et al. , 2016 ) . However , labels are often expensive . To find out a good compromise , weaker forms of supervision have been investigated . One such direction is group-based learning , which assumes inputs with the same shape to be grouped together ( Bouchacourt et al. , 2018 ; Chen et al. , 2018 ; Hosoya , 2019 ) . In particular , our study here is technically much influenced by Group-based VAE ( GVAE ) ( Hosoya , 2019 ) in the algorithm construction ( Section 3 ) . However , these existing group-based methods are fundamentally limited in that the factors that can be separated are two—a group-common factor ( shape ) and an instance-specific factor ( view ) —and there is no obvious way to extend it to more than two . Thus , our novelty here is to introduce a mixture model comprising multiple GVAE models ( each with shape and view variables ) so that fitting the mixture model to a grouped dataset can give rise to the third factor , categories , as mixture components . In Section 4 , we examine the empirical merits of this technique in several tasks . Note that grouping information can most naturally be found in temporal data ( like videos ) since the object identity is often stable over time , cf . classical temporal coherence principle ( Földiák , 1991 ) . Indeed , some weakly supervised disentangling approaches have explicitly used such temporal structure ( Yang et al. , 2015 ) . Some recent work has used deep nets for clustering of complex object images . The most typical approach is a simple combination of a deep generative model ( e.g. , VAE ) and a conventional clustering method ( e.g. , Gaussian mixture ) , although such approach seems to be limited in capturing large object view variation ( Jiang et al. , 2017 ) . A latest approach proposes a feedforward approach that takes pairs of image data , similarly to ours , and maximizes the mutual information between the categorical posterior probability distributions for such paired image ; this has shown remarkable clustering performance on natural images under various view variation ( Ji et al. , 2019 ) . In Section 4 , we experimentally compare their method with ours . Note , however , that these methods are specialized to clustering and throw away all information other than the category . 3 CIGMO : CATEGORICAL INVARIANT GENERATIVE MODEL . 3.1 MODEL . In our framework , we assume a grouped dataset D = { ( x ( n ) 1 , . . . , x ( n ) K ) | x ( n ) k ∈ R D , n = 1 , . . . , N } ( 1 ) where each data point is a group ( tuple ) ofK data instances ( e.g. , images ) ; we assume independence between groups but not instances within a group . For a data group ( x1 , . . . , xK ) , we consider three types of hidden variables : category c ∈ { 1 , . . . , C } , shape z ∈ RM , and views y1 , . . . , yK ∈ RL ( eliding the superscript ( n ) for brevity ) , where the category and shape are common for the group while the views are specific to each instance . We consider the following generative model ( Figure 1 ( B ) ) : p ( c ) = πc ( 2 ) p ( z ) = NM ( 0 , I ) ( 3 ) p ( yk ) = NL ( 0 , I ) ( 4 ) p ( xk|yk , z , c ) = ND ( fc ( yk , z ) , I ) ( 5 ) for c = 1 , . . . , C and k = 1 , . . . , K. Here , fc is a decoder deep net defined for each category c and πc is a category prior with ∑C c=1 πc = 1 . In the generative process , first , the category c is drawn from the categorical distribution ( π1 , . . . , πC ) , while the shape z and views yk are drawn from standard Gaussian priors . Then , each data instance xk is generated by the decoder deep net fc for category c applied to the group-common shape z and the instance-specific view yk ( added with Gaussian noise of unit variance ) . In other words , the decoder fc generates different data instances in a group from the same shape and different views . Having defined a mixture of deep generative models as above , we expect that , after fitting it to a view-grouped object image dataset , object categories will arise as mixture components and category-specific shapes and views will be represented in each component model . 3.2 LEARNING . We construct a learning algorithm following the VAE approach ( Kingma & Welling , 2014 ) . As the most important step , we specify inference models to encode approximate posterior distributions ( Figure 1 ( C ) ) . First , we estimate the posterior probability for category c as follows : q ( c|x1 , . . . , xK ) = 1 K K∑ k=1 u ( c ) ( xk ) ( 6 ) Here , u is a classifier deep net that takes an individual instance x and outputs a probability distribution over the categories ( ∑C c=1 u ( c ) ( x ) = 1 ) , similarly to ( Kingma et al. , 2014 ) . We then take the average over the instance-specific probability distributions and use it as the group-common distribution . This is a simple approach to make the instance-specific distributions converge to equal values , i.e. , u ( x1 ) ≈ u ( x2 ) . This is an adaptation of a key technique of GVAE used for computing the group-common shape representation ( Hosoya , 2019 ) ; see below . Then , given the estimated category c , we infer each instance-specific view yk from the input xk as follows : q ( yk|xk , c ) = NL ( gc ( xk ) , diag ( rc ( xk ) ) ) ( 7 ) where gc and rc are encoder deep nets that are defined for each category c to specify the mean and variance , respectively . To estimate shape z , we compute the following : q ( z|x1 , . . . , xK , c ) = NM ( 1 K K∑ k=1 hc ( xk ) , 1 K K∑ k=1 diag ( sc ( xk ) ) ) ( 8 ) Here , again , encoder deep nets hc and sc are defined for each category c. These compute the mean and variance , respectively , of the posterior distribution for the individual shape for each instance xk . Then , the group-common shape z is obtained as the average over all the individual shapes . In this way , again , the instance-specific shape representations are expected to converge to an equal value in the course of training , i.e. , hc ( x1 ) ≈ hc ( x2 ) ( Hosoya , 2019 ) . Note that the way the view and shape are inferred here is mostly borrowed from Hosoya ( 2019 ) . For training , we define the following variational lower bound of the marginal log likelihood for a data point : L ( φ ; x1 , . . . , xK ) = Lrecon + LKL ( 9 ) where Lrecon = Eq ( y1 , ... , yK , z , c|x1 , ... , xK ) [ K∑ k=1 log p ( xk|yk , z , c ) ] ( 10 ) LKL = −DKL ( q ( y1 , . . . , yK , z , c|x1 , . . . , xK ) ‖p ( y1 , . . . , yK , z , c ) ) ( 11 ) with the set φ of all weight parameters in the classifier , encoder , and decoder deep nets . We compute the reconstruction term Lrecon as follows : Lrecon = C∑ c=1 q ( c|x1 , . . . , xK ) Eq ( y1 , ... , yK , z|x1 , ... , xK , c ) [ K∑ k=1 log p ( xk|yk , z , c ) ] ( 12 ) ≈ C∑ c=1 q ( c|x1 , . . . , xK ) K∑ k=1 log p ( xk|yk , z , c ) ( 13 ) where we approximate the expectation using one sample z ∼ q ( z|x1 , . . . , xK , c ) and yk ∼ q ( yk|xk , c ) for each k , but directly use the probability value q ( c|x1 , . . . , xK ) for c. The KL term LKL is computed as follows : LKL = −DKL ( q ( c|x1 , . . . , xK ) ‖p ( c ) ) − C∑ c=1 q ( c|x1 , . . . , xK ) K∑ k=1 DKL ( q ( yk|xk , c ) ‖p ( yk ) ) − C∑ c=1 q ( c|x1 , . . . , xK ) DKL ( q ( z|x1 , . . . , xK , c ) ‖p ( z ) ) ( 14 ) Finally , our training procedure is to maximize the lower bound for the entire dataset with respect to the weight parameters : φ̂ = argmaxφ 1 N ∑N n=1 L ( φ ; x ( n ) 1 , . . . , x ( n ) K ) . A diagrammatic outline of the algorithm is given in Figure 2 .

Summary: This paper proposes a categorical invariant generative model (CIGMO) from a set of 2D images that tries to disentangle the factors of data category, intra-category geometry, and rendering viewpoint. CIGMO trains a VAE with a hierarchical graphical model that explicitly factors out the three components by design. Experiments on two datasets (ShapeNet rendered images, MultiPie face images) show that the proposed method can discover the concept of data category without using explicit supervision. It also supports feature manipulation over the geometry and viewpoint factors.