Adversarially Learned Mixture Model

1 INTRODUCTION . Semi-supervised or unsupervised representation learning enables the utilization of all available data when tackling problems where there are little or no labeled examples . This is a common scenario in many applications of machine learning , such as medical image analysis , where it is reinforced by the expense of obtaining expert labeled examples . Moreover , machine-learned representations are more likely to be used for subsequent tasks if they are interpretable and meaningful . Deep generative modelling is a suitable approach to this problem , as derived models have been shown capable of learning from both labeled and unlabeled examples , embedding data according to desired latent variable distributions , and producing realistic data examples generated from samples of those latent variables . The Generative Adversarial Network ( GAN ) has recently emerged as a powerful framework for modeling complex data distributions without having to approximate intractable likelihoods . In the formulation by Goodfellow et al . ( 2014 ) , a GAN consists of two networks : a generator G that is trained to yield unique samples from the data distribution , and a discriminator D that is trained to distinguish between generated and true data samples . Dumoulin et al . ( 2016 ) and Donahue et al . ( 2017 ) have proposed the ALI and BiGAN models that add an inference process , i.e. , the ability to map data samples to points in the latent space , to the GAN framework . A second generator for inference , or encoder , is added to the original GAN generator and the discriminator is adapted for the two-dimensional space of data inputs and latent representations . A variant of the resulting model is also introduced by Dumoulin et al . ( 2016 ) for conditional data generation , but still assumes that the class of the data is always observed , as inference of categorical variables is not included . Adversarial approaches for the inference of both continuous and categorical variables are actively researched . Chen et al . ( 2016 ) introduce a hybrid adversarial method that is capable of modelling both continuous and categorical latent variables for unsupervised clustering and feature disentanglement . Another hybrid adversarial method is introduced by Makhzani et al . ( 2016 ) where adversarial objectives on continuous and categorical latent variables are optimized for unlabeled examples and categorical cross entropy on categorical variables is optimized for labeled examples . Li et al . ( 2017 ) and Deng et al . ( 2017 ) point toward fully adversarial semi-supervised classification using inferred categorical variables by introducing a “ three player ” adversarial game , but stop short by adding auxiliary “ collaborative ” objectives . In each of these methods , it is assumed that categorical and continuous latent variables are independently distributed . This independence assumption results in discontinuities in the latent space between categories , which removes the notion of inter-categorical proximity . Another notable family of generative models , Variational Autoencoders ( VAEs ) , maximize the posterior distribution of latent representations given the data instead of using an adversarial approach . As VAEs integrate inference , semi-supervised classification can be performed by conditioning the continuous latent variable of the VAE on the class label ( Kingma et al. , 2014 ; Dilokthanakul et al. , 2016 ; Maaløe et al. , 2017 ) . However , the quality of VAE results depend on the expressiveness of the inference distribution and every time the assumptions about the inference or data distributions are changed a new objective function needs to be derived . In this way , variational optimization is not as versatile as adversarial training . We present the Adversarially Learned Mixture Model ( AMM ) . The AMM is , to our knowledge , the first generative model inferring both continuous and categorical latent variables to perform either unsupervised or semi-supervised clustering of data using a single adversarial objective . This is enabled , in part , by explicitly modelling the dependence between continuous and categorical latent variables , which eliminates discontinuities between categories in the latent space . Semi-supervised clustering and classification is enabled by a simplified formulation of the “ three player game ” , presented by Li et al . ( 2017 ) . In this paper we show that the AMM achieves an unsupervised clustering error rate of 3.32 % and 20.4 % on the MNIST ( LeCun & Cortes , 2010 ) and SVHN datasets ( Netzer et al. , 2011 ) respectively , and that a semi-supervised extension , SAMM , achieves a classification error rate of 5.60 % on the SVHN dataset . To support the reproducibility of the methodology and experiments presented , a public version of the code will be made available . 2 METHOD . 2.1 PRELIMINARIES . The ALI and BiGAN models are trained by matching two joint distributions of images x∈RD and their latent code z ∈RL . The two distributions to be matched are the inference distribution q ( x , z ) and the synthesis distribution p ( x , z ) , where , q ( x , z ) = q ( x ) q ( z |x ) , ( 1 ) p ( x , z ) = p ( z ) p ( x |z ) . ( 2 ) Samples of q ( x ) are drawn from the training data and samples of p ( z ) are drawn from a prior distribution , usuallyN ( 0,1 ) . Samples from q ( z |x ) and p ( x |z ) are drawn from neural networks that are optimized during training . Dumoulin et al . ( 2016 ) show that sampling from q ( z |x ) =N ( µ ( x ) , σ2 ( x ) I ) is possible by employing the reparametrization trick ( Kingma & Welling , 2013 ) , i.e . computing z=µ ( x ) +σ ( x ) , ∼N ( 0 , I ) , ( 3 ) where is element wise vector multiplication . A conditional variant of ALI has also been explored by Dumoulin et al . ( 2016 ) where an observed class-conditional categorical variabley has been introduced . The joint factorization of each distribution to be matched are : q ( x , y , z ) = q ( x , y ) q ( z |y , x ) , ( 4 ) p ( x , y , z ) = p ( y ) p ( z ) p ( x |y , z ) . ( 5 ) Samples of q ( x , y ) are drawn from the data . Samples of p ( z ) are drawn from a continuous prior on z , and samples of p ( y ) are drawn from a categorical prior on y , both of which are marginally independent . Samples from q ( z | y , x ) and p ( x | y , z ) are drawn from neural networks that are optimized during training . In the following sections we present graphical models for q ( x , y , z ) and p ( x , y , z ) that build off of conditional ALI . Where conditional ALI requires the full observation of categorical variables , the models we present will account for both unobserved and partially observed categorical variables . We finally show how they can be optimized using a single adversarial objective . 2.2 ADVERSARIALLY LEARNED MIXTURE MODEL . The AMM is an adversarial generative model for deep unsupervised clustering of data . Figure 1 presents an overview of the model . Like conditional ALI , a categorical variable is introduced to model the labels . However , the unsupervised setting now requires a different factorization of the inference distribution in order to enable inference of the categorical variable y , namely : q1 ( x , y , z ) =q ( x ) q ( y |x ) q ( z |x , y ) , ( 6 ) or q2 ( x , y , z ) =q ( x ) q ( z |x ) q ( y |x , z ) . ( 7 ) Samples of q ( x ) are drawn from the training data , and samples from q ( y |x ) , q ( z |x , y ) or q ( z |x ) , q ( y |x , z ) are generated by neural networks . The reparametrization trick is not directly applicable to discrete variables and multiple methodologies have been introduced to approximate categorical samples ( Jang et al. , 2016 ; Maddison et al. , 2017 ) . We follow Kendall & Gal ( 2017 ) and sample from q ( y |x ) by computing hy ( x ) = µy ( x ) +σy ( x ) , ∼N ( 0 , I ) , ( 8 ) y ( x ) = softmax ( hy ( x ) ) . ( 9 ) Then , we can sample from q ( z |x , y ) by computing z ( x , hy ( x ) ) =µz ( x , hy ( x ) ) +σz ( x , hy ( x ) ) , ∼N ( 0 , I ) . ( 10 ) A similar sampling strategy can be used to sample from q ( y |x , z ) in ( 7 ) . The factorization of the synthesis distribution p ( x , y , z ) also differs from conditional ALI : p ( x , y , z ) =p ( y ) p ( z |y ) p ( x |y , z ) . ( 11 ) The product p ( y ) p ( z | y ) can be conveniently given by a mixture model . Samples from p ( y ) are drawn from a multinomial prior , and samples from p ( z | y ) are drawn from a continuous prior , for example , N ( µy=k,1 ) . Samples from p ( z |y ) can alternatively be generated by a neural network by again employing the reparameterization trick . Namely , z ( y ) =µ ( y ) +σ ( y ) , ∼N ( 0 , I ) . ( 12 ) This approach effectively learns the parameters ofN ( µy=k , σy=k ) . 2.2.1 ADVERSARIAL VALUE FUNCTION . We follow Dumoulin et al . ( 2016 ) and define the value function that describes the unsupervised game between the discriminatorD and the generatorG as : min G max D V ( D , G ) =Eq ( x ) [ log ( D ( x , Gy ( x ) , Gz ( x , Gy ( x ) ) ) ) ] +Ep ( y , z ) [ log ( 1−D ( Gx ( y , Gz ( y ) ) , y , Gz ( y ) ) ) ] = ∫∫∫ q ( x ) q ( y |x ) q ( z |x , y ) log ( D ( x , y , z ) ) dxdydz + ∫∫∫ p ( y ) p ( z |y ) p ( x |y , z ) log ( 1−D ( x , y , z ) ) dxdydz . ( 13 ) There are four generators in total : two for the encoderGy ( x ) andGz ( x , Gy ( x ) ) , which map the data samples to the latent space ; and two for the decoderGz ( y ) andGx ( y , Gz ( y ) ) , which map samples from the prior to the input space . Gz ( y ) can either be a learned function , or be specified by a known prior ( see Algorithm 1 for a detailed description of the optimization procedure ) . Algorithm 1 AMM training procedure using distributions ( 6 ) and ( 11 ) . θGy ( x ) , θGz ( x , Gy ( x ) ) , θGz ( y ) , θGx ( y , Gz ( y ) ) , θD . Initialize AMM parameters while not done do x ( 1 ) , ... , x ( M ) ∼q ( x ) . Sample from data and priors y ( 1 ) , ... , y ( M ) ∼p ( y ) z ( j ) ∼p ( z |y=y ( j ) ) , j=1 , ... , M x̃ ( j ) ∼p ( x |y=y ( j ) , z=z ( j ) ) , j=1 , ... , M . Sample from conditionals ỹ ( i ) ∼q ( y |x=x ( i ) ) , i=1 , ... , M z̃ ( i ) ∼q ( z |x=x ( i ) , y= ỹ ( i ) ) , i=1 , ... , M ρ ( i ) q ←D ( x ( i ) , ỹ ( i ) , z̃ ( i ) ) , i=1 , ... , M .Compute discriminator predictions ρ ( j ) p ←D ( x̃ ( j ) , y ( j ) , z ( j ) ) , j=1 , ... , M LD←− 1M ∑M i=1log ( ρ ( i ) q ) − 1M ∑M j=1 log ( 1−ρ ( j ) p ) .Compute discriminator losses LGx ( y , Gz ( y ) ) =LGz ( y ) ←− 1M ∑M i=1log ( ρ ( i ) p ) .Compute x generator losses LGy ( x ) =LGz ( x , Gy ( x ) ) ←− 1M ∑M i=1log ( 1−ρ ( i ) q ) .Compute y and z generator loss θD←θD−∇θDLD .Update discriminator parameters θGx ( y , Gz ( y ) ) ←θGx ( y , Gz ( y ) ) −∇θGx ( y , Gz ( y ) ) LGx ( y , Gz ( y ) ) .Update generator parameters θGz ( y ) ←θGz ( y ) −∇θGz ( y ) LGz ( y ) θGy ( x ) ←θGy ( x ) −∇θGy ( x ) LGy ( x ) θGz ( x , Gy ( x ) ) ←θGz ( x , Gy ( x ) ) −∇θGz ( x , Gy ( x ) ) LGz ( x , Gy ( x ) )

Summary: The paper uses Generative Adversarial Networks (GAN) for unsupervised and semi-supervised clustering. Neural network based generators are used for sampling using a mixture model. The parameters of the generators are optimised during training against a discriminator that tries to distinguish between generated distributions. Experimental results on MNIST and SVHN datasets are given to motivate their models.