CIGMO: Learning categorical invariant deep generative models from grouped data

1 INTRODUCTION . In everyday life , we see objects in a great variety . Categories of objects are numerous and their shape variations are tremendously rich ; different views make an object look totally different ( Figure 1 ( A ) ) . Recent neuroscientific studies have revealed how the primate brain organizes representation of complex objects in the higher visual cortex ( Freiwald & Tsao , 2010 ; Srihasam et al. , 2014 ; Bao et al. , 2020 ) . According to these , it comprises multi-stream networks , each of which is specialized to a particular object category , encodes category-specific visual features , and undergoes multiple stages with increasing view invariance . These biological findings inspire us a new form of learning model that has multiple modules of category-specific invariant feature representations . More specifically , our goal is , given an image dataset of general objects , to learn a generative model representing three latent factors : ( 1 ) category ( e.g. , cars , chairs ) , ( 2 ) shape ( e.g. , particular car or chair types ) , and ( 3 ) view ( e.g. , 3D orientation ) . A similar problem has been addressed by recent disentangling models that discover complex factors of input variations in a way invariant to each other ( Tenenbaum & Freeman , 2000 ; Kingma et al. , 2014 ; Chen et al. , 2016 ; Higgins et al. , 2016 ; Bouchacourt et al. , 2018 ; Hosoya , 2019 ) . However , although these models can effectively infer a category or shape factor separately from a view factor , these typically can not capture the structure in general object images that the diversity of shapes is much larger across categories than within a category . In this study , we propose a novel model called CIGMO ( Categorical Invariant Generative MOdel ) , which can learn to represent all the three factors ( category , shape , and view ) at once only with weak supervision . Our model has the form of mixture of deep generative models , where the mixture components correspond to categories and each component model gives a disentangled representation of shape and view for a particular category . We develop a learning algorithm based on variational autoencoders ( VAE ) method ( Kingma & Welling , 2014 ) that does not use explicit labels , but uses only grouping information that links together different views of the same object ( Bouchacourt et al. , 2018 ; Chen et al. , 2018 ; Hosoya , 2019 ) . Using two image datasets of 3D objects ( one derived from ShapeNet ( Chang et al. , 2015 ) ) , we demonstrate that CIGMO can solve multiple unconventional visual tasks on objects that are unseen during training . These include invariant clustering , i.e. , clustering of objects regardless of the view , one-shot classification , i.e. , object recognition given one example per class , and other various feature manipulations using the disentangled representation . Quantitative comparison indicates that our model often outperforms many existing approaches including state-of-the-art methods . Our key contributions are ( 1 ) development of the new deep generative model CIGMO , together with the VAE-based weakly supervised learning algorithm , ( 2 ) experiments of the model on two datasets with quantitative comparisons that show performance advantages over several existing models , and ( 3 ) empirical exposition of the importance of separate representation of category , shape , and view factors in modeling general object images . 2 RELATED WORK . The present work is closely related to recently proposed disentangling models for discovering mutually invariant factors of variation in the input . In one direction , some models have used unsupervised learning with certain constraints on the latent variable , though these seem to be effective only in limited cases ( Higgins et al. , 2016 ; Chen et al. , 2016 ) . Thus , more practical approaches have made use of explicit labels , such as semi-supervised learning for a part of dataset ( Kingma et al. , 2014 ; Siddharth et al. , 2017 ) or adversarial learning to promote disentanglement ( Lample et al. , 2017 ; Mathieu et al. , 2016 ) . However , labels are often expensive . To find out a good compromise , weaker forms of supervision have been investigated . One such direction is group-based learning , which assumes inputs with the same shape to be grouped together ( Bouchacourt et al. , 2018 ; Chen et al. , 2018 ; Hosoya , 2019 ) . In particular , our study here is technically much influenced by Group-based VAE ( GVAE ) ( Hosoya , 2019 ) in the algorithm construction ( Section 3 ) . However , these existing group-based methods are fundamentally limited in that the factors that can be separated are two—a group-common factor ( shape ) and an instance-specific factor ( view ) —and there is no obvious way to extend it to more than two . Thus , our novelty here is to introduce a mixture model comprising multiple GVAE models ( each with shape and view variables ) so that fitting the mixture model to a grouped dataset can give rise to the third factor , categories , as mixture components . In Section 4 , we examine the empirical merits of this technique in several tasks . Note that grouping information can most naturally be found in temporal data ( like videos ) since the object identity is often stable over time , cf . classical temporal coherence principle ( Földiák , 1991 ) . Indeed , some weakly supervised disentangling approaches have explicitly used such temporal structure ( Yang et al. , 2015 ) . Some recent work has used deep nets for clustering of complex object images . The most typical approach is a simple combination of a deep generative model ( e.g. , VAE ) and a conventional clustering method ( e.g. , Gaussian mixture ) , although such approach seems to be limited in capturing large object view variation ( Jiang et al. , 2017 ) . A latest approach proposes a feedforward approach that takes pairs of image data , similarly to ours , and maximizes the mutual information between the categorical posterior probability distributions for such paired image ; this has shown remarkable clustering performance on natural images under various view variation ( Ji et al. , 2019 ) . In Section 4 , we experimentally compare their method with ours . Note , however , that these methods are specialized to clustering and throw away all information other than the category . 3 CIGMO : CATEGORICAL INVARIANT GENERATIVE MODEL . 3.1 MODEL . In our framework , we assume a grouped dataset D = { ( x ( n ) 1 , . . . , x ( n ) K ) | x ( n ) k ∈ R D , n = 1 , . . . , N } ( 1 ) where each data point is a group ( tuple ) ofK data instances ( e.g. , images ) ; we assume independence between groups but not instances within a group . For a data group ( x1 , . . . , xK ) , we consider three types of hidden variables : category c ∈ { 1 , . . . , C } , shape z ∈ RM , and views y1 , . . . , yK ∈ RL ( eliding the superscript ( n ) for brevity ) , where the category and shape are common for the group while the views are specific to each instance . We consider the following generative model ( Figure 1 ( B ) ) : p ( c ) = πc ( 2 ) p ( z ) = NM ( 0 , I ) ( 3 ) p ( yk ) = NL ( 0 , I ) ( 4 ) p ( xk|yk , z , c ) = ND ( fc ( yk , z ) , I ) ( 5 ) for c = 1 , . . . , C and k = 1 , . . . , K. Here , fc is a decoder deep net defined for each category c and πc is a category prior with ∑C c=1 πc = 1 . In the generative process , first , the category c is drawn from the categorical distribution ( π1 , . . . , πC ) , while the shape z and views yk are drawn from standard Gaussian priors . Then , each data instance xk is generated by the decoder deep net fc for category c applied to the group-common shape z and the instance-specific view yk ( added with Gaussian noise of unit variance ) . In other words , the decoder fc generates different data instances in a group from the same shape and different views . Having defined a mixture of deep generative models as above , we expect that , after fitting it to a view-grouped object image dataset , object categories will arise as mixture components and category-specific shapes and views will be represented in each component model . 3.2 LEARNING . We construct a learning algorithm following the VAE approach ( Kingma & Welling , 2014 ) . As the most important step , we specify inference models to encode approximate posterior distributions ( Figure 1 ( C ) ) . First , we estimate the posterior probability for category c as follows : q ( c|x1 , . . . , xK ) = 1 K K∑ k=1 u ( c ) ( xk ) ( 6 ) Here , u is a classifier deep net that takes an individual instance x and outputs a probability distribution over the categories ( ∑C c=1 u ( c ) ( x ) = 1 ) , similarly to ( Kingma et al. , 2014 ) . We then take the average over the instance-specific probability distributions and use it as the group-common distribution . This is a simple approach to make the instance-specific distributions converge to equal values , i.e. , u ( x1 ) ≈ u ( x2 ) . This is an adaptation of a key technique of GVAE used for computing the group-common shape representation ( Hosoya , 2019 ) ; see below . Then , given the estimated category c , we infer each instance-specific view yk from the input xk as follows : q ( yk|xk , c ) = NL ( gc ( xk ) , diag ( rc ( xk ) ) ) ( 7 ) where gc and rc are encoder deep nets that are defined for each category c to specify the mean and variance , respectively . To estimate shape z , we compute the following : q ( z|x1 , . . . , xK , c ) = NM ( 1 K K∑ k=1 hc ( xk ) , 1 K K∑ k=1 diag ( sc ( xk ) ) ) ( 8 ) Here , again , encoder deep nets hc and sc are defined for each category c. These compute the mean and variance , respectively , of the posterior distribution for the individual shape for each instance xk . Then , the group-common shape z is obtained as the average over all the individual shapes . In this way , again , the instance-specific shape representations are expected to converge to an equal value in the course of training , i.e. , hc ( x1 ) ≈ hc ( x2 ) ( Hosoya , 2019 ) . Note that the way the view and shape are inferred here is mostly borrowed from Hosoya ( 2019 ) . For training , we define the following variational lower bound of the marginal log likelihood for a data point : L ( φ ; x1 , . . . , xK ) = Lrecon + LKL ( 9 ) where Lrecon = Eq ( y1 , ... , yK , z , c|x1 , ... , xK ) [ K∑ k=1 log p ( xk|yk , z , c ) ] ( 10 ) LKL = −DKL ( q ( y1 , . . . , yK , z , c|x1 , . . . , xK ) ‖p ( y1 , . . . , yK , z , c ) ) ( 11 ) with the set φ of all weight parameters in the classifier , encoder , and decoder deep nets . We compute the reconstruction term Lrecon as follows : Lrecon = C∑ c=1 q ( c|x1 , . . . , xK ) Eq ( y1 , ... , yK , z|x1 , ... , xK , c ) [ K∑ k=1 log p ( xk|yk , z , c ) ] ( 12 ) ≈ C∑ c=1 q ( c|x1 , . . . , xK ) K∑ k=1 log p ( xk|yk , z , c ) ( 13 ) where we approximate the expectation using one sample z ∼ q ( z|x1 , . . . , xK , c ) and yk ∼ q ( yk|xk , c ) for each k , but directly use the probability value q ( c|x1 , . . . , xK ) for c. The KL term LKL is computed as follows : LKL = −DKL ( q ( c|x1 , . . . , xK ) ‖p ( c ) ) − C∑ c=1 q ( c|x1 , . . . , xK ) K∑ k=1 DKL ( q ( yk|xk , c ) ‖p ( yk ) ) − C∑ c=1 q ( c|x1 , . . . , xK ) DKL ( q ( z|x1 , . . . , xK , c ) ‖p ( z ) ) ( 14 ) Finally , our training procedure is to maximize the lower bound for the entire dataset with respect to the weight parameters : φ̂ = argmaxφ 1 N ∑N n=1 L ( φ ; x ( n ) 1 , . . . , x ( n ) K ) . A diagrammatic outline of the algorithm is given in Figure 2 .

Summary: This work proposes a probabilistic model which disentangles view, class and shape attributes explicitly (it does not rely on an emergent phenomena of disentangled factors in the latent space). In comparison to a similar approach, GVAE, CIGMO additionally disentangles the content factor into category and shape factors. Obtained results are reasonable and show advantage of the proposed method over related approaches.